Attack of the Vocaloids

Marrying music and mathematics for The Spectator, 3 August 2019

In 1871, the polymath and computer pioneer Charles Babbage died at his home in Marylebone. The encyclopaedias have it that a urinary tract infection got him. In truth, his final hours were spent in an agony brought on by the performances of itinerant hurdy-gurdy players parked underneath his window.

I know how he felt. My flat, too, is drowning in something not quite like music. While my teenage daughter mixes beats using programs like GarageBand and Logic Pro, her younger brother is bopping through Helix Crush and My Singing Monsters — apps that treat composition itself as a kind of e-sport.

It was ever thus: or was once 18th-century Swiss watchmakers twigged that musical snuff-boxes might make them a few bob. And as each new mechanical innovation has emerged to ‘transform’ popular music, so the proponents of earlier technology have gnashed their teeth. This affords the rest of us a frisson of Schadenfreude.

‘We were musicians using computers,’ complained Pete Waterman, of the synthpop hit factory Stock Aitken Waterman in 2008, 20 years past his heyday. ‘Now it’s the whole story. It’s made people lazy. Technology has killed our industry.’ He was wrong, of course. Music and mechanics go together like beans on toast, the consequence of a closer-than-comfortable relation between music and mathematics. Today, a new, much more interesting kind of machine music is emerging to shape my children’s musical world, driven by non-linear algebra, statistics and generative adversarial networks — that slew of complex and specific mathematical tools we lump together under the modish (and inaccurate) label ‘artificial intelligence’.

Some now worry that artificially intelligent music-makers will take even more agency away from human players and listeners. I reckon they won’t, but I realise the burden of proof lies with me. Computers can already come up with pretty convincing melodies. Soon, argues venture capitalist Vinod Khosla, they will be analysing your brain, figuring out your harmonic likes and rhythmic dislikes, and composing songs made-to-measure. There are enough companies attempting to crack it; Popgun, Amper Music, Aiva, WaveAI, Amadeus Code, Humtap, HumOn, AI Music are all closing in on the composer-less composition.

The fear of tech taking over isn’t new. The Musicians’ Union tried to ban synths in the 1980s, anxious that string players would be put out of work. The big disruption came with the arrival of Kyoko Date. Released in 1996, she was the first seriously publicised attempt at a virtual pop idol. Humans still had to provide Date with her singing and speaking voice. But by 2004 Vocaloid software — developed by Kenmochi Hideki at the Pompeu Fabra University in Barcelona — enabled users to synthesise ‘singing’ by typing in lyrics and a melody. In 2016 Hatsune Miku, a Vocaloid-powered 16-year-old artificial girl with long, turquoise twintails, went, via hologram, on her first North American tour. It was a sell-out. Returning to her native Japan, she modelled Givenchy dresses for Vogue.

What kind of music were these idoru performing? Nothing good. While every other component of the music industry was galloping ahead into a brave new virtualised future — and into the arms of games-industry tech — the music itself seemed stuck in the early 1980s which, significantly, was when music synthesizer builder Dave Smith had first come up with MIDI.

MIDI is a way to represent musical notes in a form a computer can understand. MIDI is the reason discrete notes that fit in a grid dominate our contemporary musical experience. That maddenning clockwork-regular beat that all new music obeys is a MIDI artefact: the software becomes unwieldy and glitch-prone if you dare vary the tempo of your project. MIDI is a prime example (and, for that reason, made much of by internet pioneer-turned-apostate Jaron Lanier) of how a computer can take a good idea and throw it back at you as a set of unbreakable commandments.

For all their advances, the powerful software engines wielded by the entertainment industry were, as recently as 2016, hardly more than mechanical players of musical dice games of the sort popular throughout western Europe in the 18th century.

The original games used dice randomly to generate music from precomposed elements. They came with wonderful titles, too — witness C.P.E. Bach’s A method for making six bars of double counterpoint at the octave without knowing the rules (1758). One 1792 game produced by Mozart’s publisher Nikolaus Simrock in Berlin (it may have been Mozart’s work, but we’re not sure) used dice rolls randomly to select beats, producing a potential 46 quadrillion waltzes.

All these games relied on that unassailable, but frequently disregarded truth, that all music is algorithmic. If music is recognisable as music, then it exhibits a small number of formal structures and aspects that appear in every culture — repetition, expansion, hierarchical nesting, the production of self-similar relations. It’s as Igor Stravinsky said: ‘Musical form is close to mathematics — not perhaps to mathematics itself, but certainly to something like mathematical thinking and relationship.’

As both a musician and a mathematician, Marcus du Sautoy, whose book The Creativity Code was published this year, stands to lose a lot if a new breed of ‘artificially intelligent’ machines live up to their name and start doing his mathematical and musical thinking for him. But the reality of artificial creativity, he has found, is rather more nuanced.

One project that especially engages du Sautoy’s interest is Continuator by François Pachet, a composer, computer scientist and, as of 2017, director of the Spotify Creator Technology Research Lab. Continuator is a musical instrument that learns and interactively plays with musicians in real time. Du Sautoy has seen the system in action: ‘One musician said, I recognise that world, that is my world, but the machine’s doing things that I’ve never done before and I never realised were part of my sound world until now.’

The ability of machine intelligences to reveal what we didn’t know we knew is one of the strangest and most exciting developments du Sautoy detects in AI. ‘I compare it to crouching in the corner of a room because that’s where the light is,’ he explains. ‘That’s where we are on our own. But the room we inhabit is huge, and AI might actually help to illuminate parts of it that haven’t been explored before.’

Du Sautoy dismisses the idea that this new kind of collaborative music will be ‘mechanical’. Behaving mechanically, he points out, isn’t the exclusive preserve of machines. ‘People start behaving like machines when they get stuck in particular ways of doing things. My hope is that the AI might actually stop us behaving like machines, by showing us new areas to explore.’

Du Sautoy is further encouraged by how those much-hyped ‘AIs’ actually work. And let’s be clear: they do not expand our horizons by thinking better than we do. Nor, in fact, do they think at all. They churn.

‘One of the troubles with machine-learning is that you need huge swaths of data,’ he explains. ‘Machine image recognition is hugely impressive, because there are a lot of images on the internet to learn from. The digital environment is full of cats; consequently, machines have got really good at spotting cats. So one thing which might protect great art is the paucity of data. Thanks to his interminable chorales, Bach provides a toe-hold for machine imitators. But there may simply not be enough Bartok or Brahms or Beethoven for them to learn on.’

There is, of course, the possibility that one day the machines will start learning from each other. Channelling Marshall McLuhan, the curator Hans Ulrich Obrist has argued that art is an early-warning system for the moment true machine consciousness arises (if it ever does arise).

Du Sautoy agrees. ‘I think it will be in the world of art, rather than in the world of technology, that we’ll see machines first express themselves in a way that is original and interesting,’ he says. ‘When a machine acquires an internal world, it’ll have something to say for itself. Then music is going to be a very important way for us to understand what’s going on in there.’

100 minutes of immersive terror

I interviewed Carl Guyenette, the creative intelligence behind The War Of The Worlds: The Immersive Experience, for New Scientist, 23 May 2019.

It’s six years since the Martian invaders succumbed to a microbial infection, leaving us once again in possession of our planet. Carl Guyenette has repaired to The Spirit of Man to raise a glass to Earth’s victory, and to take stock of his new production, a 100 minute-long immersion in Jeff Wayne’s The War of the Worlds.

The pub, which is part of the set and an integral part of the show, comes with its own meticulous backstory. On its walls, animated paintings record famous scenes from humanity’s first interstellar conflict. Remnants of Martian technology loom over the patrons. The effect is amusing for the first few minutes, but the aura of threat is unmistakable: pleasingly, the guts from one of the invaders’ war machines turn out to have been re-engineered to dispense gin.

Wayne’s musical retelling of H G Wells’s sci-fi shocker was released as a double album in 1978, and remains a hit, having sold over 2.5 million copies in the UK alone. There have been spin-offs a-plenty: video games, DVDs, stage shows, live tours. Nothing quite like this, though: “When I’ve been trying to explain this show to people,” says Guyenette, “I say it’s like walking into a cinema, except that once you’re there, you just keep on walking, into the screen. Into the movie itself.”

The full effect of Guyenette’s experiment in “layered reality” can only really be experienced at first hand. Nothing stays still, and neither does the audience, as it moves in groups of a dozen through over 2000 square metres of unlikely theatrical space – two floors of the old Metal Exchange in the City of London.

Visiting this venue mid-development, it had looked like somebody’s open-plan office: bad fluorescent lighting, grey carpet tiles; bins full of sandwich-shop litter; plastic water bottles in cardboard trays; laptops everywhere. Now, as the cast and crew set about unkinking the show’s phenomenally complicated logistics, the space is coming alive, fully dressed in both real and virtual light. Everything trembles. Everything moves, especially the air. Everything has a temperature. Everything has a smell.

Some of the experiences on offer in this show use VR headsets. Others use projection mapping. Some involve puppetry. Almost all manage to work in one of eight different holographic effects. Reality intrudes on the virtual world in unsettling and shocking ways. Things grab you – things you had thought were only in the headset. In VR, meanwhile, figures that seem to be fellow theatre-goers are plucked into the sky by Martian harvesting machines, their eyes meeting those of the participants (thanks to a neat eye-tracking algorithm) as they rise and perish.

Carl Guyenette talks about how he created the show.

New Scientist: What do they call you here?

Carl Guyenette: My job description’s a nuisance. When I called myself the CTO, the technologists on the show insisted I was actually the creative director. Then the creative people told me I’m a technologist. What I actually do is bring things together and makes new things out of them. So I suppose I’m an inventor.

NS: How did you come to work in theatre?

CG: I studied computer science, then joined the film visual effects industry, compositing for big Hollywood films. From there I moved on to making creative technological applications for the British Museum and other venues and festivals. I worked on Viens!, a virtual-reality piece by Michel Reilhac, which then went to Sundance and Cannes. This shot me into the centre of things. And now with the production company dotdotdot I’m trying to bring new media technologies and general audiences together through immersive theatrical experiences like this one. Not that we’ve worked at quite this scale before.

NS: Which of these new media are making the biggest impact on live performance at the moment? 

CG: Projection mapping is really interesting. There are systems now that will project images and textures over objects even as you move them. This is edging us towards VR experiences that won’t require us to wear headsets. And there are domes which you can projection-map from the inside which give you immersive video experiences. There’s a massive one that is going up in Madison Square Garden in 2020 which has a capacity of around 20,000 people: that’s going to be fun!

NS: How did you select the technologies for War of the Worlds?

Stability was essential. Because we’re splitting the audience up into groups of a dozen, and leading them from set to set, and from experience to experience, we’re effectively putting on 70 shows a day. The bottom line is, you want to be using kit that doesn’t break or fall over, so we’re using the HTC Vive Pro. We try out more exotic machinery in our prototyping and experimental work — everything from Hololens to Magic Leap, which I’d dearly like to use in a theatrical setting. But augmented reality systems are still a generation behind VR in terms of stability.

NS: Even with a workhorse VR platform, you’ve been able to mix the real and the virtual in clever ways. Was achieving that mix always an important aspect of the production?

CG: More important for us was to make sure that the technologies worked well with the storytelling. At one point we place our audience in a small boat and set them afloat on a computer-generated sea. The graphics are just one element to the experience. The mechanisms that move the boat, the breeze, the drop in temperature: these elements are just as important. And timing’s the most vital element of all, not just to provide seamless experiences, but also to give the audience breathing space between experiences.

NS: A lot of the technology you’re using is old…

CG: I wanted this show to be an homage to old media: Pepper’s Ghost illusions, and zoopraxiscopes, pyrotechnics and animatronics. It’s a show set over a hundred years ago, after all, at the birth of photography and cinema. In The War of the Worlds, all these technologies feel new.

NS: VR was said to be a medium that would isolate us from each other but you’ve used it to create a social experience. Is this the future of VR?

CG: I think there’s still money to be made from the home VR market. But building something big, in a spacious venue, layering technologies together so you can let audiences do things they couldn’t do anywhere else, means that you can also add a social dimension to the experience. There are not many places where you can be with 12 people in the same room in VR, firing cannon at Martian invaders, fighting off tentacles, befriending and losing people as you struggle through a besieged city.

Ushering in the End Times at London’s Barbican Hall

LCO_Barbican_311018_244

Mark Allan / Barbican

Listening to the London Contemporary Orchestra for New Scientist, 1 November 2018

On All Hallow’s Eve this year, at London’s Barbican Hall, the London Contemporary Orchestra, under the baton of their co-artistic director Robert Ames, managed with two symphonic pieces to drown the world and set it ablaze in the space of a single evening.

Giacinto Scelsi’s portentously titled Uaxuctum: The legend of the Maya City, destroyed by the Maya people themselves for religious reasons, evoked the mysterious and violent collapse of that once thriving civilisation; the second piece of the evening, composer and climate activist John Luther Adams’s Become Ocean, looked to the future, the rise of the world’s oceans, and good riddance to the lot of us.

Lost Worlds was a typical piece of LCO programming: not content with presenting two very beautiful but undeniably challenging long-ish works, the orchestra had elected to play behind a translucent screen onto which were projected the digital meanderings of an artistically trained neural net. Twists of entoptic colour twisted and cavorted around the half-seen musicians while a well-place spotlight, directly over Ames’s head, sent the conductor’s gestures sprawling across the screen, as though ink were being dashed over all those pretty digitally generated splotches of colour.

Everything, on paper, pointed to an evening that was trying far too hard to be avant garde. In the execution, however, the occasion was a triumph.

The idea of matching colours to sounds is not new. The painter Wassily Kandinsky struggled for years to fuse sound and image and ended up inventing abstract painting, more or less as a by-product. The composer Alexander Scriabin was so desperate to establish his reputation as the founder of a new art of colour-music, he plagiarised other people’s synaesthetic experiences in his writings and invented a clavier à lumières (“keyboard with lights”) for use in his work Prometheus: Poem of Fire. “It is not likely that Scriabin’s experiment will be repeated by other composers,” wrote a reviewer for The Nation after its premiere in New York in 1915: “moving-picture shows offer much better opportunities.” (Walt Disney proved The Nation right: Fantasia was released in 1937.)

Now, as 2018 draws to a close, artificial intelligence is being hurled at the problem. For this occasion the London-based theatrical production company Universal Assembly Unit had got hold of a recursive neural net engineered by Artrendex, a company that uses artificial intelligence to research and predict the art market. According to the concert’s programme note, it took several months to train Artrendex’s algorithm on videos of floods and fires, teaching it the aesthetics of these phenomena so that, come the evening of the performance, it would construct organic imagery in response to the music.

LCO_Barbican_311018_156

Mark Allan / Barbican

While never obscuring the orchestra, the light show was dramatic and powerful, sometimes evoking (for those who enjoy their Andrei Tarkovsky) the blurriness of the clouds swamping the ocean planet Solaris in the movie of that name; then at other moments weaving and flickering, not so much like flames, but more like the speeded-up footage from some microbial experiment. Maybe I’ve worked at New Scientist too long, but I got the distinct and discomforting impression that I was looking, not at some dreamy visual evocation of a musical mood, but at the the responses of single-celled life to desperate changes in their tiny environment.

As for the music – which was, after all, the main draw for this evening – it is fair to say that Scelsi’s Uaxuctum would not be everyone’s cup of tea. For a quick steer, recall the waily bits from 2001: A Space Odyssey. That music was by the Hungarian composer György Ligeti, who was born about two decades after Scelsi, and was — both musically and personally — a lot less weird. Scelsi was a Parisian dandy who spent years in a mental institution playing one piano note again and again and Uaxuctum, composed in 1966, was such an incomprehensibly weird and difficult proposition, it didn’t get any performance at all for 21 years, and no UK performance at all before this one.

John Luther Adams’s Become Ocean (2013) is an easier (and more often performed) composition – The New Yorkermusic critic Alex Ross called it “the loveliest apocalypse in musical history”. This evening its welling sonorities brought hearts into mouths: rarely has mounting anxiety come wrapped in so beautiful a package.

So I hope it takes nothing away from the LCO’s brave and accomplished playing to say that the visual component was the evening’s greatest triumph. The dream of “colour music” has ended in bathos and silliness for so many brilliant and ambitious musicians. Now, with the judicious application of some basic neural networking, we may at last be on the brink of fusing tone and colour into an art that’s genuinely new, and undeniably beautiful.

Keaton Henson: The mechanisms of empathy

A conversation with Keaton Henson for New Scientist, 16 July 2018

What inspired your new composition Six Lethargies?

Keaton Henson     Sad songs are something we all understand. I wondered if, instead of bringing people to tears, which can be quite cathartic, I could give them a direct musical experience of my anxiety disorder. When I used to perform live, I would distract myself from my anxiety by watching my audience – this group of 3000 strangers – and how they reacted to certain chord changes and certain inflections in my voice. You can really feel this happening. I became fascinated by the mechanisms of empathy.

And music is one of those mechanisms?

KH     For sure. Every culture we know of dances around a fire. Our heartbeats sync up, we all follow this one rhythm, and we feel the tribe unite. If I explain my break-up in words, say, you will be able to understand to a degree what I’m going through. But if I write a piece of music and play it to you, you might just start crying, and that’s totally incredible because I’m not giving you any framework. I’m not necessarily reminding you of something from your past. It’s purely those patterns that are bringing you to tears.

KH     The Britten Sinfonia are performing a piece in six movements, and five of these movements simply explain how I feel. But there’s also a movement that’s designed to elicit those feelings in the Barbican Hall audience, which is where Brendan comes in.

Brendan Walker    I’m best known for my work helping design roller-coasters. More generally, I’m playing with the synergies between bodily rhythms and patterns in nature that have an emotional impact. Think, for example, about breathing rhythm, heartbeat, and why we find calm in the sound of waves crashing on a beach. For Six Lethargieswe’re gathering electrodermal activity data from a portion of the audience. The electrical conductivity of the skin is the physiological trait most closely associated with the state of anxiety and the one that’s most easily decoded.

KH     Brendan’s kit is set up so that a tiny pore on someone’s fingertip will control the hall’s huge lighting rig, in real time. It’s an amazing thing, and very beautiful. It can be a very uniting and comforting thing to be surrounded by people. At the same time, being surrounded by people you don’t know is a perfect breeding ground for anxiety. The more anxious you become, the more you feel, “Oh God, I’m anxious again, and everyone will notice!” Well, we’re going to be projecting people’s anxiety through the entire lighting rig of the Barbican Hall. This perfectly represents what anxiety is like.

2nd

And the more anxious the audience is, the more anxious you’re going to make them…

BW     Yes. In the movement I’m working on, we’re not just trying to communicate. We’re trying to actually elicit a state of anxiety. We’re talking about having quiet rooms and ways to extract people if they feel panicked at any stage.

KH     I’m hoping that Friday’s performance at the Barbican will be the first of many. We’re interested in trying different things for each show, including varying the type of data we gather, and who we choose to gather data from.

How much research went into this piece?

KH     In particular I went to Canada to meet with a cognitive neuroscientist called Jessica Grant who studies the relationship between music, rhythm and emotion. But I’m a massive science nerd, and I’m wary of crossing too far into the realm of research. I wanted to use scientific thought and theory to help express what I’m feeling. I didn’t want *Six Lethargies* to become manipulative or sterile.

How did you go about composing Six Lethargies?

KH     I kept asking myself, what’s the exact opposite of what I’m trying to achieve? It’s probably baroque music, because that’s all about resolving tension, again and again. It delivers these constant hits of relief. I don’t want to give too much away about the show, but a lot of it is going to be about what people think they’re going to get next — and what I can do to stop them getting it.

You could simply not turn up…

KH     I’ve given myself certain limitations! For instance, I’m composing purely for string orchestra – believe me, you can do some really weird stuff with strings. And Six Lethargies is a tonal composition. Atonal music is everyone’s go-to method for inducing anxiety. But I’m a singer-songwriter. I write pop songs. I work with intervals and scales. I decided I would try to make an anxious piece while hitting all the proper notes.

“Proper” for whom?

KH     Music is built out of the melody of speech, and the way our speech patterns convey emotion. We assume Western music is a sort of universally understood music that can convey emotion intuitively to all cultures, and as it turns out we’re not altogether wrong. Pretty much everyone around the world will hear the Moonlight Sonata and think, Wow, Beethoven must have been really sad when he wrote that.

Do people expect anxiety to sound a certain way?

KH     A lot of people have been talking to me about Bernard Herrmann’s theme music for the film Psycho. And, naturally, I’ve avoided any suggestion of that in this project. I want to avoid anything that people might expect to hear. Anxiety is all about not knowing what is going to happen next.

Is that what it’s like to have your anxiety disorder?

KH     In horror movies, when the terrifying thing bursts out of the door, you’re given this horrible fright which lasts a millisecond and is immediately followed by a sigh of relief. You’re pulling on a string and then releasing it. For me, that tension is never released. It’s like an infinite rollercoaster, just building up, and up, and the higher it gets, the more you realise the drop is going to be very steep indeed…

Music and art stop dementia from stealing everything we cherish

A choir and a De Kooning show inspired this piece for New Scientist, 4 January 2018

This year, stalked by insomnia, I have been going to bed with Melvyn Bragg. More precisely, I have been putting myself to sleep with podcasts of Bragg’s BBC Radio 4 programme In Our Time, the best cultural contexter money can’t buy. On its website there are quizzes to check how much you remember, though I rarely score more than 4 out of 12.

Podcasts, the St John’s College reading list, Wagner’s Ring cycle: I’ve been pouring culture down my throat the way the Danaids filled their bath, and to about the same effect.

According to Greek legend, 49 of King Danaus’s 50 daughters were mariticidal, and condemned to fill a leaky bath in hell, and their lot is an apt metaphor for the human condition. However much we fill our lives, our lives still dribble away. We experience, we learn – but we also forget. Finally, we die.

No wonder death is terrifying. It’s not just me that will perish on my deathbed. With me, a whole world will gutter out.

One day in October, circumstances conspired to bring me a little comfort. The Wellcome Trust invited me along to a rehearsal of Singing with Friends, a community choir for families living with dementia, led by the Wigmore Hall in partnership with Westminster Arts.

Wellcome’s interdisciplinary research group Created Out of Mind are trying to understand (and, where possible, quantify) the therapeutic properties of the arts in dementia care and the care of older people generally.

Paul Camic, a psychologist at Canterbury Christ Church University, Kent, UK, was there to talk me through the research.

Musically inclined readers may already be familiar with the idea that we recall best and most reliably the music we first heard between the ages of 15 and 21.

This choir’s weekly rehearsals (four-part harmonies from a standing start, public performances announced, and a great deal of mutual mickey-taking) reveal something that for my money is much more exciting.

Apparently, musically inclined people are more than capable of continuing their musical education, and achieving command of new material, in even quite advanced stages of dementia.

Is there a general truth to be drawn here? That same morning saw me visiting Skarstedt Gallery in central London, and an exhibition of late canvases by Willem de Kooning.

The American abstract-expressionist’s celebrated and long career ended in an Alzheimer’s diagnosis, and wags in search of a cheap laugh have long suggested that his last, ever-more minimal canvases reflected his mental deterioration.

Face to face with this work, however, it’s clear that de Kooning was developing and exploiting new vocabularies of form, depth and colour right up to his retirement at the age of 86.

Such stories take nothing away from the misery of dementia. I recall vividly, at the Wigmore Hall, the wife of a former jazz musician explaining how her husband, after hours of skilful extemporising around a new theme, would fail to remember how or where to go to the toilet.

Then again, she also told me that attending Singing with Friends was like “coming out”; that for her husband and for the choir as a whole, excellence was still achievable; and that its attainment was all too rarely recognised by a squeamish wider world.

Which is why I’m here typing that most unfashionable thing: a story with a moral. The bath is leaking. But then, the bath was always leaking. Deal with it. Keep going with the buckets. “Pour, pour, against the draining of the bath,” as Dylan Thomas didn’t say.