Professor Gil Weinberg: "Robotic Musicianship at Georgia Tech" | Talks at Google

Subtitles section Play video

MALE SPEAKER: Welcome.
It's my pleasure to introduce Gil Weinberg.
He's a professor at Georgia Tech for the Center of Music
and Technology.
And I went to Georgia Tech a long, long time ago
and we didn't have a music department back then.
So when I visited recently, the folks I was meeting with in ECE
sent me over to meet Gil and he's
doing some really cool stuff with interactive computing
and musicianship.
So building machines that you can interact with,
not just from a dialogue perspective,
like the kind of thing I'm thinking about,
but more from a music perspective
where you want to jam and do some jazz
improvisation with a robot.
And he's going to talk about that in many other things
today.
So welcome.
GIL WEINBERG: Thank you.
[APPLAUSE]
Thank you.
Thank you for coming.
So I'm going to talk about three or four main projects.
Only three of them I showed in these slides.
The last one is a surprise.
We did the project which we just finished and I have some very,
very fresh new slides.
The reason I started to be interested
in robotic musicianship is that I'm
a musician before I became interested
in computation or in robotics, I was a musician and still am.
And I was always fascinated by playing in a group.
By being constantly ready to change what I'm doing.
By trying to build on what other people
are playing in real time.
Improvising in a group, a visual cues, auditory cues, of course.
So when I started to be interested in robotics,
I wanted to capture this experience.
Before I show you some of the efforts that I did,
maybe I'll show a short clip of me playing with a trumpet
player and trying to see the kind of experiences
that I was trying to recreate.
[VIDEO PLAYBACK]
[JAZZ MUSIC]
[END PLAYBACK]
So I think you've seen a lot of eye contact,
trying to build a motif that I'm not sure what is going to be
and trying to create something interesting with them
back and forth.
And the first robots that I tried to develop it
was building on these ideas.
But what I wanted to do is to have the robot understand music
like humans do.
The big idea that I started with,
was to create robots that listen like humans,
but improvise like machines.
Because I felt that if I want to push what music
is about through new, novel improvisation
algorithms and new acoustic playing,
I first have to have connection between humans and robots,
and that's why there is the listening like human part.
Only then would I be able to start to make the robot
play like a machine in order to create this kind of connection
and relationship.
I'll start with something very simple
that probably many of you are familiar with.
If I want a robot to understand me,
maybe the first simple thing that I can make him do
is understand the beat of the music that I play.
And here we use auto-correlation and self-similarity algorithms.
This is a piece from Bach.
You see the time is both on the x and on the y.
And you see that it's symmetric and by comparing
the [INAUDIBLE] to the algorithm you
can try to find sections that are similar
and detect the beat from that.
But what we try to do is to actually have it in real time.
And you see here my student Scott Driscoll
used this algorithm based on Davies and Plumbley
from Queen Mary.
And you see that's it becomes much more
sophisticated because it's not just analyzing the beat in Bach
or in the Beatles-- Scott is playing in real-time so he
is trying to get the beat, but then with Haile,
start to play with it.
Scott is trying to fit what he is
doing to what Haile-- Haile is a robot-- and back and forth.
So it's a little more complicated you see.
Sometimes they escape from the beat and get back to the beat.
And here's a short example.
Got the beat.
Now Scott will start faster and slower.
Faster , got it.
So as you can see, it loses it, it gets it back.
I think in a second, it will play slower which
shows how the system fails.
The next thing that I was trying to get
is to have the robot understand other things at a high level
musically.
Not just the beat but concepts that we humans
understand such as stability and similarity.
And basically we had a huge database
of rhythm generated almost randomly-- with some rules,
some stochastic rules.
And then we had a coefficient for stability and similarity.
Whenever Haile listened to a particular beat
there's some settings and coefficient
for stability and similarities.
At some point the robot actually decided by itself.
First the human on the side can change the similarity
and stability and create some output to bring rhythm back.
And this similarity is based on Tanguiane from 1993,
basically looking at the overlapping
onset between beats.
I can get more into this if you want, maybe later.
And this is as similarity algorithm
based on Desain and Honing.
[INAUDIBLE] between adjunct intervals
is what set how stable the rhythm is.
This is based on music perception studies
that I've been doing.
And basically, there is a mathematical procedure
where you compare each one of the notes
to the note that comes after it.
And at some point, after giving preference to one and two,
which are stable, you can get for every particular rhythem--
for example, this one, the quarter quarter,
two-eighths quarter-- a particular onset stability
by combining all of the ratios and getting something stable.
And here's a short example of Scott
playing with Haile and Haile trying
to understand the stability of Scott's rhythms.
And then based on a curve of similarity,
starting most similar then going less similar.
And basically a human put a curve.
But I can see a scenario where Haile could come up
with a curve by itself, trying to start with something
that Scott understands.
Scott is playing seven quarters, and then slowly
introduce new ideas.
And you can see how Scott actually listens to the robot
and at some point, building on what the robot is doing.
So Haile is building on what Scott is doing, obviously,
by looking at the stability and similarity.
But at some point, Scott is almost inspired.
Maybe inspired is too big of a word,
but that's a goal that Scott will come up with an idea
that he would come up with if he played with humans.
[VIDEO PLAYBACK]
[DRUMMING]
That's almost a new idea.
Scott is building on it.
[END PLAYBACK]
And this is a darbuka drum player concert.
You will see how the professional darbuka
player, actually from Israel, is kind of surprised.
But I think his facial gestures were interesting for me
because I think he was surprised for the better.
And at some point you'll see how all of us are playing
and Haile tries to get to beat.
So we combine the stability similarity
and beat detection into a drum circle.
[VIDEO PLAYBACK]
[DRUMMING]
This is call and response.
Later it will be simultaneously.
[DRUMMING]
And now, what it does, it listened to these two drummers,
and tricked the pitch from one drummer,
and the rhythm from another.
And the other arm, most from pitch
and the timbre of the two players.
[END PLAYBACK]
So we played the rhythm that one player played,
and the pitch-- well, it's not really pitch, it's in the drum,
but this is lower and this is higher next to the rim,
and tried to create something that is really
morphing between these two.
Again, things that humans cannot do and maybe shouldn't do,
but here something interesting can come up.
Another thing that I was interested in is polyrhythm.
It's very easy for a robot to do things that humans cannot.
Sometimes I'll ask my student to clap.
I will not ask you to clap.
I'll give you an example.
I think there is two main reasons here.
[SPEAKING RHYTHM]
This is nine.
[SPEAKING RHYTHM]
I don't ask you to clap but sometimes I would.
It was [SPEAKING RHYTHM] seven.
And then I asked my students to do
the nine in one hand and the seven
in another hand which I will definitely not ask you.
But see how Haile here captured--
decided to record the rhythm.
So it records the nine.
He choose the nine and the seven, and at some point
he introduced them in polyrhythmic, interesting
rhythms.
[VIDEO PLAYBACK]
[DRUMMING]
And we add more and more rhythms over it.
[END PLAYBACK]
And I don't know if know, but Pat Metheny had a project
that he used robots in.
He came to our lab and I explained it to him,
I showed it to him.
I said, this is something no humans can do.
And he said, sure, my drummers can do it.
And he can also do the four with his leg, and another three
with another leg.
So I think everyone can do it except maybe Pat Metheny's
drummer.
And here is something that's at the end of concert,
it's obviously, you know, fishing for cheers.
We just have a little short MIDI file where
we playing with it together.
Unison always works, so I'll just play this.
We grabbed the nine and seven.
[VIDEO PLAYBACK]
[DRUMMING]
And that's from a performance in Odense in Denmark.
They had a robot festival.
[END PLAYBACK]
So the next project was Shimon.
And actually I have an story about this
because I put a video of Haile and Scott playing,
the first one, on my website.
It was before YouTube, or before I knew about YouTube.
And someone grabbed it and put it on YouTube,
and then CNN saw it and they asked to come and do a piece.
And when they put a piece, the next day
I got an email from the NSF, from the NSF director
who said we saw your piece, please submit a proposal
and continue to develop that.
Rarely happens.
Never happened since.
I tried.
I put so many videos--
And this is the next robot that we came up
which adds multiple things.
The main thing that it adds is the concept of pitch.
It plays a marimba.
And the second aspect it adds-- we're
talking about the personal connection, gestures,
visual cues-- is the head.
And many people ask me, why the head?
It doesn't play any music.
In a second I'll show some of the utilization
of what the head does.
And of course it has a camera.
That's the first question you get
about any head, robotic head.
So the name is Shimon, and here is the first example
of my post-doc, Guy Hoffman.
He's now a professor in Israel.
[VIDEO PLAYBACK]
[PLAYING PIANO]
It shows how a headless Shimon responds
to dynamics, to expression, and to improvise.
Look at Guy, how he looks at him as
if his kid did something nice.
[END PLAYBACK]
[LAUGHTER]
So when Guy played fast, the robot responded.
Simple, just tempo detection.
And then loud and louder, and tried
to get some understanding of tonality
and of giving the right codes.
And we had some studies done about it.
If you remember the first code was [SPEAKING CODE]
Three codes.
And we tried to see if, by looking at the robot,
at the gestures, with the arms before the head,
if the synchronization between the human and the robot
is better.
And you see that the first code is maybe easier.
The second code, very difficult to synchronize
in terms of milliseconds.
The third code, you kind of adjust yourself.
So it's a little better.
And we have multiple conditions, one only visual, one audio,
and one with synthesized sound.
And you can see that the visual is better
because it's less milliseconds of delay
between the human and the robot in both of the conditions,
precise and imprecise.
Especially in imprecise, you'll see it's better.
For imprecise, the robot could play 50 milliseconds,
before or after.
And just looking at the robot helped
it become a much more synchronized,
tight performance.
And it works much better at slower tempos, less than 100
BPM as in more than that.
Then we added some other aspects.
We said, OK, we have a head, let's do what the head can do.
Actually we didn't think of this way.
We thought of what the head can do and then built the head,
but for the story.
And what it can do is multi-modal detection.
We had a camera there and we tried
to detect the beats of the hip hop artist
to coordinate with the beat detection of the music.
And you see how the multi-modal detection work.
And also just to help them feel together--
and this is Guy Hoffman's work-- and develop gestures
that will make everyone feel that they are
synchronized with each other.
[VIDEO PLAYBACK]
[MUSIC PLAYING]
So that's just the audio.
[MUSIC PLAYING]
Now double-checking with the beat of the head.
And of course blinking never hurts.
The blinking is random.
[END PLAYBACK]
All right.
OK.
So that was a little bit about listening like a human
and maybe grooving like a human.
And then I started to think more about creativity, or machine
creativity and what it means.
And actually, this is the first time I've showed it.
I hope it's not streamed, I think it's not streamed.
This is a TED Ed video that I just finished doing.
I didn't put in online yet, so it's a part of it.
I didn't show it yet.
But it shows one of the ideas I had
for creating machine creativity using genetic algorithms.
And the idea was that if have a population of motifs, maybe
random motifs, and then I have a set of mutations
and cross-breeding between these motifs,
and have a fitness function that is based on human's melody,
then I can create new, interesting music
that has some aesthetics for humans in them.
Because it's easy to create melodies
that were never created before.
Most of them will not be listenable.
The question is, how do you make meaningful and valuable
melodies?
And in the video, you'll see a little reference
to the Lovelace Test, which probably some of you
know about maybe I'll just say that the Lovelace Test is based
on Ad Lovelace, the first programmer,
or is considered to be the first programmer,
trying to-- instead of intelligence like the Turing
test, try to determine when the robot is creative.
And she said something like, a robot will never be creative
before it generates ideas that its own programmers cannot
understand how they came to be.
And then in 2001 there was a group of scientists,
some of them continued to develop [INAUDIBLE] that
came up with this Lovelace test, which is mostly
a theoretical thought experiment of creating something
like that.
But even if we do something like that,
I didn't think that the music that would come be valuable.
And I'll show this little TED Ed clip
and see how I went about it.
[VIDEO PLAYBACK]
-So how can evolution make a machine musically creative?
Well, instead of organisms, can start
with an initial population of musical phrases.
[MUSIC PLAYING]
GIL WEINBERG: They put the drum beat.
It's not mine.
-And a basic algorithm that mimics reproduction
and random mutations by switching some parts,
combining others, and replacing random notes.
Now that we have a new generation of phrases,
we can apply selection using an operation called a fitness
function.
Just as biological fitness is determined
by external, environmental pressures,
our fitness function can be determined
by an external melody, chosen by human musicians or music fans
to represent the ultimate beautiful melody.
[MUSIC PLAYING]
GIL WEINBERG: Chopin.
-The algorithm can then compare between our musical phrases
and that beautiful melody, and select only the phrases
that are most similar to it.
Once the least similar sequences are weeded out,
the algorithm can re-apply mutation and recombination
to what's left, select the most similar or fitted ones, again
from the new generation and repeat for many generations.
[MUSIC PLAYING]
The process that got us there has so much randomness
and complexity built in, that the result
might pass the Lovelace Test.
More importantly, thanks to the presence of human aesthetic
in the process, we'll theoretically
generate melodies we would consider beautiful.
GIL WEINBERG: All right.
[END PLAYBACK] So here's an example
of how this is used in real-time.
Chopin is not [INAUDIBLE] with fitness function
as you see here.
But actually, the saxophone player and myself
playing piano-- I programmed Haile with multiple motifs
and every time it listened to one of our musical motifs,
it used dynamic time warping to see
what's most similar and come up with a variation.
And it stopped after something like 50 generations
because, of course, if it ran more than 50
it would become exactly what we played.
So we just stopped one time to create something
that is similar to what we did, but a little different
and based on the, let's call it, original DNA of the population
that we started with.
And [INAUDIBLE] Haile will never use it with Shimon.
This is Haile when we first tried to have Haile with pitch.
So the whole idea of creating rich, acoustic sound
is not there.
We put the little toy vibraphone below Haile.
So excuse the quality of the sound.
[VIDEO PLAYBACK]
[MUSIC PLAYING]
And listen to how all three of us
are building on each other's ideas.
Saxophone, piano, and robot.
[MUSIC PLAYING]
[END PLAYBACK]
Yeah.
It's difficult to evaluate it because of quality of sound.
But that was the idea of trying to put it
in a real-time scenario.
Another idea where I tried to have
it play more like a machine was style morphing.
And as you can understand by now, I play jazz, I like jazz,
I like Monk on the left and Coltrane on the right.
And I thought, wouldn't it be awesome
if we tried to morph between Coltrane and Monk
and create something that they themselves will never
be able to create.
They cannot morph into each other.
And maybe something interesting will come up
that is just a combination of both their styles.
And we used HMMs here, basically analyzing
a lot of improvisation by both of these musicians
and trying to see what the likelihood is that at any given
note, what the next note would be, what the next rhythm would
be, just a second order.
And then with an iPhone I can play a little melody.
[VIDEO PLAYBACK]
And this is a little app that we built regardless.
And at some point when the melodies that I play,
of course it can be piano too, is ready,
I will send it to Shimon, and then I
will be able to morph between my own melody, Monk's
style, Coltrane's style and with two styles with saw
a different kind of balances between these two
improvisations and myself.
[MUSIC PLAYING]
If you know Monk's syncopation in clusters.
If you know Coltrane, you know the crazy, fast licks.
And this is somewhere in between.
And of course, one doesn't play saxophone over piano,
so it doesn't sound like any of them.
[END PLAYBACK]
But at least the notes are based on the improvisation.
Some other quick examples of things that we tried.
Here we had the connect because we
wanted to get more than just what
this simple camera in the head had.
And we tried to have motion-based improvisation.
You see the Atlanta symphony percussionist, Thomas Sherwood,
doing some tai chi-like movement and trying
to have Shimon improvise.
And actually Tom is playing an electronic marimba
and Shimon has a system in which sometimes it will
repeat what is listened to.
Sometimes it's [INAUDIBLE], sometimes
it will introduce new ideas, a stochastic system that's
supposed to inspire Tom to play differently
than he would otherwise.
[VIDEO PLAYBACK] So that's the tai chi part.
[MUSIC PLAYING]
And it wasn't easy for it to repeat, exactly this pattern.
It's very limited with the four arms and where they can be.
I'll show some path planning strategies later.
So not every motif was captured perfectly.
And now it starts to introduce new motifs.
It will repeat and then create some kind
of a musical interaction.
[END PLAYBACK]
This is my student, Ryan Nikolaidis, the entire motif.
[VIDEO PLAYBACK]
And Shimon is developing this motif.
This is part of the US Science and Technology Fair
in Washington.
[MUSIC PLAYING]
[END PLAYBACK]
And for something else, some of them just, you know,
they love dubsteps.
And the machine learning maybe interesting or not,
but dubsteps are there.
So it's just a dubstep piece that one of our students
developed for Shimon.
[VIDEO PLAYBACK]
[MUSIC PLAYING]
So it can do all kinds of stuff.
[END PLAYBACK]
Here we used vision-- and you see this green ball
on the conductor's baton-- and tried
to combine the input from conductor
and from the audio analysis from the orchestra to improvise,
based on the chords that the orchestra is playing.
[VIDEO PLAYBACK] That's the Georgia Tech student orchestra.
[MUSIC PLAYING]
[END PLAYBACK]
All right.
And I think this is my favorite clip.
This actually was a PSA of Georgia Tech for three or four
years.
You know that we come from Georgia
where dueling banjos are a big thing, at least in some movies.
So we tried to do dueling marine buzz if you like,
and present kind of the dueling aspect and Shimon
with the gestures-- and this is again,
Guy Hoffman's work-- project some personality.
Hopefully better than the movie.
[VIDEO PLAYBACK]
[CALL AND RESPONSE MUSIC PLAYING]
[END PLAYBACK]
I think I have two more clips before we
go to the next project.
I just got the request, how about having a robot band?
They listen to each other and create something.
I'm sure robotics people here are being asked about,
will robots replace humans often?
And in my case it's really painful sometimes
because they come to me and tell me, how dare you?
Music, really?
Like, OK cars, OK cleaning carpets,
but you're going to take human from us
and let the robots play it.
So I never tried, I'm not interested in having robots
playing together by themselves.
It's only and always have been about inspiring human
and creating something interesting for humans.
And here you can see a quick clip
where you see bit detection and [INAUDIBLE] improvisation
between two students and two robots.
[VIDEO PLAYBACK]
[MUSIC PLAYING]
He's not so good with the beat, by the way,
so-- strictly for Haile to detect it.
But it got it.
And now it start to improvise with the other arm.
What you hear here is a student that we don't
see, playing the synthesizer.
[MUSIC PLAYING]
And now Shimon, before we had a head,
actually we used someone else's CGI head.
Andrea Tomaz from Georgia Tech.
I'm dropping back and forth in time here.
[END PLAYBACK]
All right.
So that's a robot-robot interaction for you.
I promised to talk a little bit about path playing.
There are multiple ways in which these forums
can be next to the piano or next to the marimba.
In this case I would just put a piano.
And we have state machines that represent each one
off these states.
Given any melody, what would be the most reasonable path
to take?
And the inputs that we have is a curve
that humans play for tension-- melodic tension
or harmonic tension, or rhythmic stability.
And I'll show it here.
This is some the pitch or note density.
And using the Viterbi algorithm, actually [INAUDIBLE] Viterbi,
we're trying reduce the space and try
to come up a reasonable path to create improvisation,
following these curves.
So it's basically developing, as I say,
an embodied musical mind, where not only of the music
it has to play but also it's own body.
The whole idea of embodiment is a bit part
that robot musicians have and that software musicians
or machine musicians don't have.
Only robotic musicianship has.
So we are trying to come up with these algorithms, and actually,
my student, Mason Bretan programmed really extensive
jazz theory into Shimon.
So now it can listen to chords that Mason, in a second
will play, and then try to improvise,
using this both music and physical limitations
that he has using Viterbi.
He tried some other, but the Viterbi worked pretty well
and this is his improvisation.
This clip actually went viral kind of for our fill.
There were around 50k viewers last month.
[VIDEO PLAYBACK]
[MUSIC PLAYING]
Mason fed all the chords offline.
All the calculation of what's going to be play
was done offline.
[MUSIC PLAYING]
[END PLAYBACK]
So I don't know if it's got [INAUDIBLE] quality
but it's not bad.
I was happy.
All right, so we all think it's very important, the path
planning and creativity and so on.
But it's important to realize that this can
be taken in a different light.
And I think if you're being made fun of for what you do,
this guy is probably the best guy to make fun of you.
Some people tell me, don't show it,
but I think-- I'm proud of being made fun of by Stephen Colbert.
[VIDEO PLAYBACK]
-Jazz robots, which combines two of the biggest
threats to our nation-- jazz and robots.
He's been programmed to dig it too.
[LAUGHTER]
Bobbing his head to the beat and checking in
with this fellow players.
Meaning that long after the robot uprising kills
all humans on earth, there will still
be someone on the planet pretending to enjoy jazz.
[LAUGHTER]
[END PLAYBACK]
OK.
Shimi was an effort to take Shimon and make
it a little smaller.
Actually it almost became a product.
There was a failed Kickstarter there.
And it focused on the gestures, not so much
from the acoustic reproduction of sound,
which is part of the reason that I went into robotics because I
kind of grew tired by sound from speakers, no matter
how great your hifi system is.
I was missing the acoustic sound which
is one of the reasons I went to robotics.
So here we don't have acoustic sound,
but we have a bunch of other applications.
We call it a robot companion that can understand music.
It can connect to your Spotify.
It can dance, it can do the bit detection.
It has a set of algorithms to get data from the audio
and respond accordingly.
And here's a clip.
[VIDEO PLAYBACK]
[MUSIC COLDPLAY, "VIVA LA VIDA"]
GIL WEINBERG: In this case it takes the music from the phone,
but we also have an application that takes it from Spotify.
And the camera and the phone can see if you don't like it.
And better to say no, because the music is loud.
[TAPPING "DROP IT LIKE IT'S HOT"]
[MUSIC SNOOP DOGG, "DROP IT LIKE IT'S HOT"]
GIL WEINBERG: [INAUDIBLE] speaker dock.
You can see the dancing speaker dock.
[MUSIC SNOOP DOGG, "DROP IT LIKE IT'S HOT"]
[END PLAYBACK]
Here is a couple of other applications, some using
speech, and some using audio analysis.
That stuff is audio analysis.
Looking for a beat in what Mason is playing.
[VIDEO PLAYING]
[PLAYING GUITAR]
GIL WEINBERG: No, there wasn't anything there.
[PLAYING GUITAR]
You get it.
It has a system of gestures.
[END PLAYBACK]
GIL WEINBERG: And here, using Google Voice.
Mason is connecting to Spotify.
[VIDEO PLAYBACK]
-Look at me.
Can you play something happy?
GIL WIENBERG: And we have some algorithm
to determine happy, disco, rock.
[HAPPY MUSIC PLAYING]
-Can you play disco?
[MUSIC BEE GEES, "SATURDAY NIGHT LIVE"]
GIL WIENBERG: Of course this.
So a different set gestures for different genres.
All right.
[END PLAYBACK]
Annie Zhang, another student, worked on the Shimi band,
trying to create collaborations between three Shimis.
And she looked at a lot of [INAUDIBLE], music
[INAUDIBLE] elements you would get from the audio.
Anything for the beat detection to the beat to the arm
is for the energy.
MFCC for vocalist to see if there's vocals there or melody.
And then this is her interface.
And she then came up from the low level features,
detecting some high level features,
beats, energetics, melodics, vocalness,
percussiveness of the sound.
And then she actually went and studied [INAUDIBLE] system
and tried to create a dancing grammar
for all of these gestures that will connect
between how Shimi is dancing.
And here are some of her ideas for that.
And a whole system of different ways in which three Shimis can
dance together-- solo, having the wave, two
in synchronicity from both sides, and the main one
is different.
So a full system that later allowed her to have this.
[VIDEO PLAYBACK]
[MUSIC PLAYING]
And every time it moved different, the gestures.
[END PLAYBACK]
And even in a very noisy environment in a demo--
[VIDEO PLAYBACK]
[MUSIC PLAYING]
[END PLAYBACK]
Here too, we try to have an embodied musical mind
and have an understanding of what are the important music
notes in a piece.
Tonally, maybe they're more stable,
maybe it's the end of a phrase that goes up and so on.
And then combine it with some understanding
of what are the positions in which what
can move to particular places, being
aware of its own limitation in terms
of the time of the gestures.
So here you see the red dots that
represents a particular end point
in gestures that try to be based on the important notes
melodically.
Again, the combination of being aware of the embodiment
and of musical theory and creating a system
that, I think Mason here put some titles
that you can read as he plays.
[VIDEO PLAYBACK]
[PLAYING GUITAR]
[END PLAYBACK]
And as you can see, Mason is extremely talented,
and he is looking for an internship this summer.
[LAUGHTER]
Just to end the Shimi part-- maybe
I'll just play this little clip.
I couldn't make a Shimi without letting it dance to this song.
[VIDEO PLAYBACK]
[MUSIC STAR WARS, "CANTINA SONG"]
[END PLAYBACK]
And that's the last project that we just finished last year.
I actually took a break.
I took a sabbatical year off.
And just when I was about to go to the sabbatical
I got an email from Jason Barnes.
He's an amputee drummer.
He lost his arm, now it would be like three years ago.
When we met it was two.
And he saw some of the videos online
and he was really devastated.
There's a clip online.
You can see him talking about how he thought he had nothing
to live for if he cannot drum when he lost his arm when he
was electrocuted in some weird accidents on a roof.
And then he saw these video and he emailed me and asked me,
can you use some of the technology
to build me a robotic arm?
And, of course, that was very intriguing for me.
What he really wanted was to be able to control the grip.
Because he had the arm up all the way up until here so
he can still hit.
But for a drummer, it's very important for them
to have control of the grip, tight or loose
to create different kinds of drum strokes.
And that was very interesting for me.
And the people I usually collaborate
with couldn't come through, so I-- I
don't know if anyone here is from Meka,
but I did contact Meka who built this arm.
And that was great but because I'm interested in improvisation
and creativity, I really wanted to have this aspect
of my research in this arm.
So I asked Jason if he minded that we had two
different sticks, not only one.
One was all about grip, so you can have your expression
and play it as he wants.
The other one actually would listen and improvise
and will listen to the music and listen to other musicians
and try to maybe inspire you to create something else
and become better or try to create music that he
would never create otherwise.
Let me show you a couple of clips
first of the basic interaction of just controlling the group.
And what we discovered after Phillip from Meka
built the arm is that we don't need just to hold the stick,
we can actually use it to generate hits.
And after some work on noise reduction,
we were able to some pretty quick responses,
very low latency so Jason could just
flex his muscle, in the muscle and actually create
a beat, in addition to using his own arm.
So there's some aspect of a cyborg
there because part of the gesture
is his own body and part of it is electronically
based on his muscle and maybe other muscles,
not only the forearm muscle.
So actually, this took eight months.
It was amazingly fast, again, thanks to Meka.
And this is the first time Jason tried it.
MALE SPEAKER: Next.
Next.
Next.
GIL WIENBERG: And that's where we really
got-- I didn't expect that, that we can create 20 hertz.
And you know, my favorite polyrhythm games.
I'll show more of the musical context later.
But let's first-- see, here we didn't
have the EMG working yet.
It's a little knob that changes between tight and loose,
but just to see the effect.
This was a tight hit.
[VIDEO PLAYBACK]
[DRUMMING]
Assuming the holding of the stick tight,
this is a loose hit.
[DRUMMING]
And this is a role tight.
[DRUMMING]
[END PLAYBACK]
Many more expression than he had before.
He couldn't do all of this.
And actually, he was in tears the first time.
He said, I didn't play this well since I lost my arm.
Which was the first time I actually worked with people.
And it was pretty powerful.
But, here, you can see the second stick.
The kick is very short.
Look here, we have an accelerometer
and when he wants-- only when he wants,
he can lift the arm quickly and the second stick will pop up.
It will be really fast.
OK.
And then we can start with the improvisation
and the creativity, the machine creativity.
Before we do that we can just have a computer
generate strikes.
So nothing to do with him.
Even though it's part of his body,
suddenly someone else is generating hits for him.
And even Pat Metheny's drummer cannot do.
Not with one arm.
And this is, again, playing with roll,
with tumble, because it's so fast it becomes actually color
and not even rhythm.
And this is what I call cyborg drumming where
you combine some of the computer generated hits
with his own hits.
He decides where to put the arm, where not, what to choose.
Create a human machine interaction
and musical collaboration.
And at some point-- actually I wrote an NSF proposal
about it-- we want to create some human brain machine
interaction.
But now some of the most sophisticated
stuff we're doing with pedals.
With his leg, it controls dynamics and expressive
expression elements.
And when this first clip went online,
it also kind of became viral.
And I got a lot of emails and one of them
was from a guy called Tom Leighton who actually
played with Richard Feynman.
I didn't know that Richard Feynman was a drummer,
playing hand drums.
And he said, I have a bunch of recordings from Richard Feynman
and I would love for you to play with it
and see if you could do something with it.
And of course I jumped on this opportunity.
And what we did here is we put some of the rhythm
that Richard Feynman played in his arm,
so he had a chance to play with Richard Feynman in a weird way.
And first you'll see him hovering over the drum,
just listening to the rhythm, and then he
starts to decide what drum, when he whats to lift the symbol,
and so on so on.
So it created all kinds of interaction
with someone who is not with us anymore.
And I think it also was a powerful experience for him.
[VIDEO PLAYBACK]
[DRUMMING]
That's the rhythm by itself, just hovering.
And now he adds to it with the other arm.
[DRUMMING]
Now he starts to decide where to put it.
He's hovering but now we drew this
as part of his gestures and movements.
[END PLAYBACK]
Here we have analysis of the chords
and we try to create similar connections,
a very intimate connection between the guitar player,
Andrew, and Jason.
Different chord that he plays.
We analyze the chord and change the rhythms
that Jason is playing.
So suddenly he has to respond, not only
to the arm that is generated by the computer
or generated by his muscle, but generated or controlled
by a different musician.
And you see the whole eye contact and the whole,
trying to create something together,
happening between them two.
[VIDEO PLAYBACK]
[DRUMMING AND GUITAR]
It was one of the first times that he
did it and it's not tight, but it becomes tighter
with every rehearsal and every performance.
[DRUMMING AND GUITAR]
[END PLAYBACK]
And here he is, in context, trading four for those of you
who play jazz.
You play four bars and then someone else
improvises four bars and back and forth.
Here, the arm listens to the improvisation for four bars
and then Jason gets to improvise.
And the arm is responding, improvising, [INAUDIBLE]
classic manipulation of what it heard in the four bars before.
[VIDEO PLAYBACK]
[JAZZ MUSIC]
[END PLAYBACK]
And I have longer clips for all of this online.
This is the latest project.
I just received this videos from Deepak, my student.
Just yesterday he just finished doing that.
And if you may have noticed, a lot
of the expressions that if you are a drummer
you probably know that you can hit in much more versatile ways
than just a hit or not a hit.
There's a lot of multiple bounces
and a lot of expression in it.
Here is Deepak who is a drummer.
My students need to be musicians,
and in this case roboticist.
Actually he was just accepted to a robotics program
at Northwestern.
The first master student from music technologies
that actually got into a robotics program.
It shows the different kind of drumming techniques
that he has.
[VIDEO PLAYBACK]
So all of this richness and expression, we couldn't have.
So Deepak had to go and learn PID control, which
again, is not one of the classes that we
offered in music technology.
[END PLAYBACK]
And he tried to come up with a model for the talk,
taking into account gravity.
And something he had to know, it's a very complex system.
The bounciness of the drum, for example,
is not being taken into account.
But he tried to put everything in simulation.
And with some coefficient of restitution
here is 0.85 just as a constant.
And he tried to create all kind of bounce behaviors.
Some of them are just based on humans and bouncing balls kind
of techniques.
But some of them are actually bounces that humans will not
create or are not likely to.
You see this is much faster.
At its end, it's lifting for the next hit.
And here you have a high to low force profile you see.
This hit is softer than the next one.
So you really have to change-- it's time varying talk changes.
And he put it into a model and created a PID control.
And this is the first clip.
He just did it yesterday morning and sent it to me
just before I left.
[VIDEO PLAYBACK]
So hopefully this will be embedded into some
of the new performances.
And we're going to France to perform in May.
Hopefully Jason can play a much more wide range of expression.
This is some of the future works that the NSF proposal that I
have wrote is going to address.
Am I'm working with a neuroscientist
from Northwestern, [INAUDIBLE].
And we are trying to get some understanding
about anticipation.
Because there are latencies in the system,
and no one isn't focusing on volition detection, which
is basically-- your brain is shooting,
sometimes seconds before your motor cortex
actually knows that you're about to hit.
In a way, you know you're going to do something
before you know you're going to do something,
or a part of your brain knows about it.
I tried to get this data.
We did some experiments with 36 EEG electrodes,
and there maybe some way to get information
before the hit is going to be made and then
better anticipation algorithms.
So if he hits with both hands, or hits
with both times-- this hand hits by itself,
it will be on time because we have some information.
We did some machinery to detect different kind of strokes
and we saw that the sum-- it's promising.
It might be tricky and we have some outliers,
but we can pretty much detect single strokes
versus double strokes, which is something else that we
can try to implement.
And this is one way it will probably never look like.
But this picture is in the proposal.
Justin is an example.
The trick here, when you create such cyborgs
is how not to interfere with what you're doing anyhow.
But as you can see, I'm jumping into third arm scenarios.
So half of the proposal would be working with Jason
on his own arm, creating this.
But the second part is, let's try, available to anyone,
whether you are disabled or abled, and have a third arm.
And the promise, the spiel is that if we
have anticipation algorithms and timing and synchronization,
that works in music-- and music is a very, very time demanding
and medium.
You know, you hear five milliseconds
that what you heard is two.
So if we can pass synchronization in music,
we can anticipate and help in all kind of scenarios--
on operation rooms or fixing a space station with a third arm.
So how to put it on the body in a way
that you can be completely-- first,
the cognitive load, how to control it.
So we thought about the brain, the organ of muscles
that we can still try to hit.
But the muscles are used for legs for the drums
and for arms for the drums.
This is a different kind of design for the future brain
machine control.
And I'll be happy to go for questions right now.
Thank you very much.
[APPLAUSE]
AUDIENCE: I think you alluded to this,
but can you say more about why you
think the physical manifestation of a striking thing
is important.
Are there other artists actually getting information
from the motion of the robot or is it just for looks.
GIL WIENBERG: So one study showed that it
helped with synchronization.
That's the graphs that I showed when you see the arm moving,
when you see the beat.
You'll better synchronize your gestures with it.
I'm a big believer in embodiment.
I believe that that's why we go to concerts as opposed
to listening to music at home.
We want to see how people interact, not only by music
but by gestures.
A drummer and a guitar will end the song together.
And there's a lot about interacting with gestures
and being aware of your body as a musician that I think you
cannot do without it.
Of course, all of it can two iPhones that play this music.
But I think we're going to miss the whole musical experience.
It's something really deep and strong in embodiment
in the musical experience.
AUDIENCE: So two sort of related questions, if you don't mind.
The first is, have you tried sort
of persisting some information across several sessions for any
of these to try to have the robot develop
some kind of style more incrementally,
and sort of lock into certain patterns
that it prefers systematically over longer periods of time?
GIL WIENBERG: I thought about it.
We didn't get to it.
It's a great idea because musicians--
that's what they do.
There's length, there's lifelong learning.
And we're doing just small episodes.
There was a student that was interested in starting to do
but then he decided that a master's is enough.
He doesn't need a PhD.
But it's a great suggestion and it's definitely
something important for music.
AUDIENCE: And then related to that,
some composers or musicians have--
you talked about embodiment-- certain sort
of different bodily features like, I don't know,
someone like Django Reinhardt in jazz had only two fingers
to use on the guitar.
Rachmaninoff had very big hands.
So have you thought about sort of creating robots
that have different variations on what kind of motion
is available to them and seeing how that guides things, or sort
of, how more constraint can change?
GIL WIENBERG: So that's the latest work by Mason
with the path planning.
And he tried to come up with a system.
His PhD thesis will come up with a model of other musicians.
That's one of his main goals to continue
to develop the field of robotic musicianship.
Other people who build robots will
be able to use this kind of path planning ideas
and put as a coefficient the different limitations
that they have.
AUDIENCE: OK.
Thank you.
GIL WIENBERG: Yeah.
AUDIENCE: There was only one instance
where you mentioned a robot knowing
anything context-specific or genre-specific.
For most of the examples, it was instead,
everything that the robot should know
about how to generate the content
comes from the content it's receiving.
But then there was one where one of your students
programmed all of these jazz theories into it.
Do you have an aversion to telling
a robot genre-specific or context-specific information?
Or do you want it to be as blank of a slate as possible?
GIL WIENBERG: I don't have aversion for anything.
I come from jazz.
Jazz is about improvisation.
A lot of what I'm trying to do is [INAUDIBLE] improvisation.
So it doesn't necessarily fit maybe other genres.
Like with classical, yes, Bach can be improvised
and you can have all kind of counterpoints
based on improvisation.
But I can see a scenario when you're trying
to put classical theory in it.
I less see how it's going to be used for improvisation.
I think that it should be very interesting, yes.
AUDIENCE: Do you think you would ever do it negatively and say,
OK, well, these things, if you ever come across this,
this is not idiomatic?
You should avoid it?
I'm thinking, for example, in jazz of course,
but also for the hand drums, like, I don't know anything
about that style of music but would
you ever want to say like, in whatever mutation
or combination that comes up, this particular pattern
is seen as ugly?
GIL WIENBERG: I don't know if ugly, but--
AUDIENCE: Or not--
GIL WIENBERG: Not part of the grammar.
Yes, we have it in the jazz system.
AUDIENCE: OK.
Cool.
GIL WIENBERG: And I think it will
be interesting to try to see what happens when
we go to other genres, with maybe a different kind
of tonal theory.
AUDIENCE: I have a question back to the pathing analysis.
I'm curious if you have a measure of how much less you
are able to do when you do have these physical constraints,
versus if you could have the machine improvise it and watch
other people, and it would just have to produce tones.
Is there some sort of measure as to the limitations,
qualitatively versus what it could do, if it didn't have
the physical limitations of forearms that
can't cross over one another?
GIL WIENBERG: I think we can deliver--
I don't have a measure.
But I think a measure can be developed
and I think a measure can also be used for future robot
designers to build robots that have less of these limitations.
And when we came up with the four arms,
that this arm can never play this octave, we knew that.
And we knew that, in a way, it might play faster than human
and create 16 chords simultaneously which humans
cannot do.
But it will also be more limited than humans.
And I think when I play piano, I do path planning in a way.
If I don't have the fingers, if I just
to look at notes with the fingers,
I constantly, probably, do something
similar to Viterbi's beam search job algorithm or something.
Creating a measure in terms of coming up with different kind
of chords that you can or cannot play--
different kind of speed of phrases that you can or cannot,
you know.
They're what, if you noticed, one for the black keys,
for the accidentals and one line for-- so the limitation just
there and see if you can play clusters.
Can it play a cluster of C C sharp D E flat E?
It can't.
We don't have a measure to really come up
with a coefficient, but I think it's a good idea.
I think all of you have a lot of good ideas.
AUDIENCE: Thank you.
AUDIENCE: First, awesome talk.
Thank you very much.
The question I have has to do with,
kind of, wanting to play with these robots myself.
There's a bit of a bottleneck in that
your team is doing the hardware and also all the software.
Have you thought about moving towards a more
platform-based approach where other labs could write code
against your robot and be able to try it out?
GIL WIENBERG: That's a good one.
And by the way I'd like to give credit when credit is due.
My students are mostly computer science students.
The platform itself-- I have colleagues
come from the MIT Media Lab.
Guy Hoffman designed and Rob [INAUDIBLE]
designed the robots and Meka designed the latest one.
So this is not our research.
Probably the Meka robot is the best candidate
to become a platform.
The others are too flaky, and too specific.
But part of what I'm interested in
is also the art and the design and the coolness
of creating a new robot.
So it's just because I love it.
But I think it can be a good idea to try to, for the arm,
for example, to try to make a platform,
create code, create, I don't know, an API for other people
to try to play with.
Definitely.
I think this field is pretty new and there's
all kinds of [INAUDIBLE].
At some point, if enough people think it's important,
maybe we will use a robotic operating system.
But for now, we just focus on creating cool musical one-offs.
AUDIENCE: Thanks.
AUDIENCE: In the physical gesture robot,
you showed a picture of a score where
you had analyzed important points in the melody where
you might end a gesture.
But it was taking as input, raw audio.
Is that right?
As opposed to a MIDI file or a digital encoded score.
GIL WIENBERG: The input was a chord
that Mason and played from a MIDI piano.
So it got the symbolic notes from the MIDI.
AUDIENCE: Oh, OK.
And he was playing a guitar I thought at one point as well?
GIL WIENBERG: Yes, that was different.
The guitar was only played with shimmies.
Piano plays with Shimon.
The guitar for Shimi to depict the right--
and yes, it did repeat-- actually, as I mentioned,
this is offline.
Mason played this melody, it did it offline
and then it recreated it.
AUDIENCE: Oh, OK.
OK.
Thank you.
And one other thing.
About the auto-correlation beat detection.
Does that infer any time signature information?
Or how good is that at detecting strange time signatures?
If at all?
GIL WIENBERG: It's not as good.
Four quarters is much easier.
A big estimation, so it pretty much
focused on regular time, three, four.
Others would be more difficult.
AUDIENCE: Thank you.
MALE SPEAKER: I had one question.
I'm a classical musician and this is about online learning.
I mean, yeah, online real-time learning from the instructor.
So as a classical musician, you sit down, you have a lesson
and you have a instructor that teaches you,
based on what they're hearing, giving you feedback
and then they correct or improve your playing right then.
I don't know anything about instruction
for jazz improvisation.
I guess I've seen a bit of that.
Wynton Marsalis has a program right now
that you can watch where he's working with some young jazz
musicians and he's giving them feedback.
A lot of it's about communication and listening.
Have you thought about that sort of thing?
Not so much-- what you're showing here
is the actual improvisation, but if you wanted to give feedback
to the robot and say, you know, hey, why don't you try this.
Or that's a little bit too busy.
You maybe should be kind of simpler at this time
and then add some more color later or something.
I don't know.
GIL WIENBERG: The same students that
decided that MS is fine were thinking about learning.
Starting from nothing and learning getting some input
and reinforced.
And then develop a style that will develop over time.
We were thinking about.
We didn't get into it.
But yes, in jazz too, I think similar to classical music,
that's how it works.
First production of the sound.
You know, when I first played piano,
it was a lot of hits on my fingers and holding turtles.
If I have some more students using
learning-- not in the machinery applications of it,
but actually your robot that knows nothing and starting
to teach it slowly.
It can be very interesting.
Maybe very interesting new music perhaps that
wouldn't come this way.
MALE SPEAKER: OK, thank you very much.
That was awesome.
GIL WIENBERG: All right.
Thank you.
[APPLAUSE]