absolutely love to thank CMJ as well as Microsoft for making this all possible.

Also we’d like to begin by introducing everybody else up on stage here. We have Charlie Whitney,

Dave Rife, and Kamil Nawratil. I almost got it.

Listen. This is who we are. Thought it would be great to give you guys a little bit of

background on the project. I am the founding partner of Listen. We are a creative and strategic

group just down the street.

And most of our work is focused, and many of the projects we do, are focused on the

intersection of music and marketing. The two projects that we’re going to talk about

today are part of an ongoing partnership that we have with Microsoft.

Delqa is the first project we’re going to look at, that we worked on with Matthew Dear.

And the second, something that’s actually launching this week, was a live show visual

system for Neon Indian.

When we think about music and technology and we think about how technology has really over

history shaped how musicians are able to express themselves and how audiences receive music

and then how we also define what is music.

Technology definitely plays a role in that. And this is true now more than ever, and is

core to the work that we’re doing with Microsoft.

Specifically our work involves developing and concepting ideas and projects like this,

so we can bring together musicians with technologists and talented folks like the people who are

here on stage today.

Delqa is a project here, we’re going to watch a video in just a minute which will

help set it up, and then Dave and Charlie are going to talk a bit about, uh, get under

the hood a little bit on this project.

But it was a project, it was an installation that went up at the New Inc. space, at the

New Museum, over the summer. And you know, I think I’m going to let the video speak

for itself.

Technology’s allowing for this openness. It’s this living, breathing sound sculpture.

You can play with it, and tweak it.

I wouldn’t make the song that I made for DELQA at any other point in my career or my

life because it only works for a space like that.

To have that opportunity, and to meet wonderful new people and learn from them and see what

they’re doing and get ideas that I wouldn’t have come up with, I mean, that’s the best

. . . that’s art.

We saw this project as an opportunity to create something that had never been done, pulling

together an amazing artist with a team of amazing artists using Microsoft’s technology.

Seeing the guys break it apart, I’m seeing the guts of the system. It’s a lot of math

and a lot of numbers. These guys are smart; they’re really smart.

The big idea that we were exploring was being able to step inside of Matthew’s music.

This piece is really about creating a sonic environment. So one of the ways that we’re

going to achieve that is by installing 40 loud speakers throughout the space.

Those will all be linked together into a spatial audio system that allows for a three-dimensional

field of sound.

You’ll be inside of all these different musical ideas that make up one beautiful composition.

We were able to leverage the Microsoft Kinect’s ability to track people’s movements spatially

through this experience.

The Kinect is everywhere in this thing. You’re almost always covered by a Kinect. We specifically

sourced this fabric material. It was opaque to the depth cameras, so we can actually read

the surface of the fabric, but it was transparent to the colored infrared cameras, so we can

actually see through it.

As people push on the walls and displace the membranes, they’re actually able to change

the quality of the music.

Different areas of the installation affect the music in different ways. Sometimes a user

will be changing the actual part that’s playing. Sometimes they’ll be introducing

new parts into the music.

We wanted every interaction to feel very powerful for the audience. So, if we take the drum

part for example, as they press in, the rhythmic density of it becomes much more complex, but

we’re also changing the tamboral qualities of those synthesized drum sounds.

All of these parameters allow the audience to have a multidimensional influence across

the composition.

So you could visit this piece on a number of occasions and you’ll hear something different

each time. It would never be the same if you went there twice.

Our desire was to create a new type of architecture, influenced by how we experience sound.

Doing interactive design is really about creating a magical experience, and having really powerful

things like a Kinect that help you create those experiences.

There’s so much that we can do with a project like this.

The more accessible these technologies become, the more people are able to use them and do

surprising things with them.

People want to make music, people want to be part of the music making process. Technology

allows that.

Your mind is totally free to wander. There’s no right or wrong answer. It’s just however

far you want to push it.

To be able to bend music, make music, and put that in different worlds, that’s what

makes this fun.

So, uh…that should help to give it a little bit of context around the project, what we

were up to and how that all came together. Now I think I’ll hand it over to Dave to

talk a little bit about the composition and the sound and light and even a bit more.

Check, check, check, we’re here. So I’m glad that video played. I can kind of be done,

now. But no, we’ll go a little bit under the hood – [To Kamil] yeah, you can go ahead.

I lead a company with Gabe Liberti, who is out there somewhere. Where are you Gabe? Raise

your hand so everyone knows—that’s Gabe Liberti!

We have a creative studio called Dave and Gabe. We do a lot of sound and light work

with immersive and interactive environments. I have a background in architectural acoustics,

and, how sound behaves in space.

Gabe has a really strong background in production, and you can see why we ended up doing a lot

of the work on this project.

If you go to the next slide, you’ll see some of our work. It all involves interaction,

it, most of it, involves sound and lights. Go ahead, go to the next one, oh good that

was out…

I think one of the first things that we talked about as a team, that entire team that you

saw in the video, was, what is this interaction going to feel like?

Like, how are you going to trigger these different moments in the sound, and how are you going

to create this composition based on what Matt already wrote.

And so, we really loved the idea of something being physical. Actually like physically touching

something, something like an interface, but something more architectural.

So we had these two different types of meshes. One of them was kind of “spandexey” like

this, and one was more like a net.

And as you pushed into it, you would change the composition, you would change different

parts of this music, and so based on how you’re pushing like this, and also where you are

in the space, like if you’re pushing on this side or that side, you would trigger

different sounds and different effects.

You saw in the video -- this is how it kind of works. So that’s great. We can go to

the next one. But very tactile, was what we were going with.

And then, at the same time, we were working with Matt on the composition. So Matt totally

got it right away. He met with the whole team and totally understood this would be rad as

an environment. I’ll create some music.

But how does it work in terms of making that music interactive? Because typically you’re

listening just over headphones or out of your speakers and you’re just taking it in as

the listener.

But what we were all going after was putting you inside of his music, so pulling his music

apart and surrounding you with it, but also giving you as a listener agency to change

what’s going on, to manipulate the sounds that are happening. [To Kamil] No, not yet

not yet.

So, what you see here is the Ableton session. So Matt created his composition in Ableton,

and then we all shared that session together.

And we worked with him to take specific tracks, so you’re seeing six different tracks here,

so six different parts of the music.

But we write multiple clips per track. Right? So there’s different energies within each

one of those tracks.

There’s different kind of sounds that are happening within that, but it’s all aligned

to one type of sound, so like, one might be a high hat, one might be a kick drum, one

might be arpeggios, that’s all kind of in the vertical there, and then within each one

of those are different sounds, so that when you push into the netting, you can trigger

those different sounds. [To Kamil] So if you go to the next one we should see that.

This is kind of the way that we made up the media nodes that were synthesizing some of

the drum sounds, so you see, as we push in, you get more rhythmically complex, a little

bit more dense patterns, and then as you push out all of that goes back away… next slide

And so I don’t know – go ahead and hit it again so it plays the movie – so you

can kind of see right in the middle, um, there’s a kick selector, and then this is the Microtonic

VSC. Right in the middle, you’ll see, right around here…somebody is starting to push

in to the music and you see that little arrow clicking down on the different tunes, and

you hear the different complexity that’s happening within the music.

I think it’s a little out of sync with the visuals, but you can also see within the Microtonic,

there’s different filters happening, and that’s all based on the real time information

about where people are pushing as well.

So this is a custom tool that we wrote within Max for Live that Yotam Mann, that you saw

in the video, wrote. In order to be able to make this music more interactive, and not

just like a static copy of what’s going on.

And then so now we have the interaction that we want to do, and we have the music and the

music is flexible. So now we wanted to reproduce it over a huge audio system that’s all around

the listener.

And so we had 44 loud speakers. And the point of this was not for loudness—it’s not

to make the thing really loud—it’s so that we can pull the tracks apart. We can

take the stems out, and give each its own place to be. And we can also give all of the

stems, or some of the stems, we give them trajectories, so there’s sounds that are

moving around the space all around you, and there’s some sounds that are moving based

on what you’re doing within the architecture.

So we laid it out in six different zones—A, B, C, D—so A, B, and C and D were kind of

four different instruments, and then E and F were when you climbed up into the middle

of the netting, you’d get this kind of drone type thing happening around your ears if you

were there, and if you weren’t there it wouldn’t be there. Go ahead.

And then this is kind of what it looked like in one zone. So we had ten loud speakers for

each of those quadrants, and some of the sounds would be really diffuse and be happening all

around you if you weren’t pushing into the net, and then if you pushed into the net it

would go local to where you were.

So you could really like spatialize where the sound was happening just based on what

you were doing, on top of all of the compositional stuff that was happening. And you already

saw that. I think that this is just a really funny clip because Gabe is in slow mo with

a speaker on his head. I had to show that, sorry. Keep going, oh there it is again. Wonderful.

And then in terms of the spatialization we used MaxMSP. We had some of the tracks within

the piece. These were not so much interactive, but they were always moving around you.

So what you’re looking at here on the right side of the screen is like a top down view

of all the loud speakers in the space, so you’re looking at the floor.

And then those green dots are actually sound sources, so that’s like particular parts

of his track that are always like slowly rotating in the space, and then we had these high hats,

which is kind of that crazy one, randomly going all around, and that ended up feeling

like it was like a fly kind of going all around your ear. It was super cool.

And this is kind of what it looked like to mix the piece. It’s really interesting to

mix a piece over a large spatial system like this, and also interactive, so we had to set

up in front of each one of the quadrants and get the local mix right and then sit and listen

to the overall piece and get the overall thing right, so it’s a ton of variables, but that’s

kind of what it looked like.

And then you’ll notice here, go ahead, that there’s lighting in the space too that was

also reactive to what you were doing in the space and it was also tied to the music, so

Ableton was kind of driving some of the lighting as well.

So what you’re seeing here is again a planned view overhead of the lighting system, a whole

bunch of DMX-addressable wash lighting that we hit the netting with. Go ahead

And then we used TouchDesigner to drive that. So we had these different color palettes within

touch for each part of the song. There were kind of three parts of the song in the timeline.