Thank you for having me many missile.

Livia.

Jack, my talk is entitled the browser as a modular, networked video synthesizer.

Um, hopefully by the end of the talk, it'll be clear or more clear what I mean by that, um, everything I'm going to be talking about is online.

Um, at this website, hydra at editor dot Glitch stopped me.

Um, and it's sort of an ongoing set of research and experimentation and creation that I've been I've been working on for the last year and 1/2.

Um and so Hydra is It's an investigation of collaboration in the creation of life visuals, possibilities of future pure technology on the Web feedback loops.

And it's also a platform for life coding visuals in JavaScript.

Um, and so one of the inspirations for this this project is a project from 1977 called the Satellite Arts Project.

My Kit Galloway and show Cherie Rubino.

It's, um, and in the seventies they got access to a satellite and were able to stream video live between two places at once, and this is in the seventies, you know, before the Internet, before skype and and they started experimenting with this just sort of streaming video and putting images on top of each other, mixing them together, overlaying them and and seeing what?

What happened when they started experimenting with this?

Um, and one of the things they talked about with their project is this marks the first time that the geographically dispersed electronic image was conduct contextualized as a live immersive place where artists and others could convene and co create together on a scale that could be as culturally inclusive is desired.

And, um, I want to focus on the live immersive place piece of this because, um, I think all they're doing, they're sending data between two places, right?

And they're talking about how, just by putting these two images on top of each other and seeing them live, that it's creating this space that's not there and it's not there.

It's sort of some third, third space that the combination of these two spaces, um and I think now with the web we talk about, we use this kind of metaphor of a place a lot.

Um, like a website.

It's a site.

It has an address.

You go to the address.

Um, you're at the website.

And, um, it might just seem kind of like nit picky.

But it's also it's kind of talking about a way that were engaging with these technologies.

Um, and but what's really happening when I go to a website, you know, my computer is sending a message to a server somewhere the server is responding with with a message, and so I'm not anywhere when I go to a website, but But we have these sort of ideas about how we talk about the Web and how we talk about how we engage with the Web, that to me it's just sort of interesting to be aware of them and start to dissect them.

Um, and especially just in the seventies, this was like, Well, what do you mean?

We can see live video from two different places at once.

And what what are the possibilities of this?

And I think sometimes when when technology just comes out is when people are often the most experimental with it, and so fast forward to now.

There's this protocol called Web RTC riel time Web streaming that lets you, um, the browser protocol for streaming video, audio and data directly between clients.

And so what?

One of the interesting things that Web R T C is that although you need a signaling server to sort of set up initial handshakes between clients, once clients are connected, they're basically just sending video or audio or data between one another.

And about four or five years ago I started experimenting with this, this protocol making sort of little video chat applications, and it's sort of I was just kind of amazed that that it's not that hard in JavaScript to, like, send video everywhere.

Um and so So I'm still kind of interested in this protocol, and I think lately there's been a lot of exciting developments with this.

Such a CZ Web torrent like bit torrents, running on the Web or bigger browser and different sort of puta Pierre Peer to peer applications of this is saying, What happens if we don't necessarily depend on a server for everything around on this one?

Centralization for everything on DSO.

So this this diagram kind of gets throwing around a lot, but on the left is like a really centralized network and and towards the middle of more decentralized network and on the right of more distributed network.

And one thing that Web RTC enables is possibly Maur things like on the right.

Um, but one thing that's been sort of interesting to me is seeing a lot of feud.

If your technology that often although it's, ah, use it, that it's it has the opportunity to connect people like the diagram on the right.

Often what happens is sort of replicating essential as database such a cz with Bitcoin or something you're you're creating a centralized database of transactions.

And so, uh, everyone still sort of received the same information.

And in some ways it's recreating the experience of Essential Eyes database, Um, and so experimenting with this video streaming, I got interested in sort of different ways of, of thinking, of using this kind of client side peer to peer network.

And so one of the things that really inspired me sending video around the Internet is analog video synthesizers.

And so, um, what's an analog video synthesizer?

This is an example.

Also from the seventies.

It's the sand in image processor, created in 1971 by Daniel Sanden, and basically each of these boxes either generates a Siri's of electrical signals or transforms a series of electrical signals.

And when the when the signal gets to the the screen, it creates a pattern, and then the order in which you connect the box is sort of, um, creates different possibilities for modifying and changing the image.

And so what I was interested in is is kind of sort of seeing similarities between this thing where we can send lots of video streams around.

And this thing over here, which is sending lots of signals around and the way in the order in which which were sending them, can create different creative possibilities.

Um, and this is really I had no, like specific goal in mind, but this is sort of like, Oh, I just want to play with these Web protocols.

And it's not in this case, not really thinking of a website as a place or a sight, but thinking of it as a flow of information.

Um, in this case, sort of a signal flow.

And so So this is sort of was like a nen initial experiment that I did with this, which is basically, um, here have a bunch of windows connected via Web ROTC.

In this case, they're all on the same computer.

But they could easily have been, you know, one in one country, one in different country, connected over the Web and, um, here on the left and creating an image.

And I'm sending it as a video to hear and combining it with this one.

And then I'm sending other videos.

Uh, here's the Web Kim and then combining it with what's on the bottom.

Right?

Um, and again, I was just sort of started.

Expect playing around with these things and as they didn't exactly know what Michael's were with it.

I started life coding in the Consul just to sort of experiment and be ableto live, change.

Change these these things.

And it's sort of like it's both a visual experiment, but also sort of a conceptual experiment of what happens if we connect in these ways.

And how can we maybe collaborate or something?

Um, so So this is an initial experiment.

Um, and then it turns out it's not that practical.

The time If you're on the same computer to be sending, uh, video streams, it's like, takes a lot of processing power.

If you're If you're just on the same computer.

And so, um, this experiment led thio a node module called RTC Patch Bay That just for me made it easy Thio to send these video streams, um, and sort of reroute them.

And then, um, I started making this sort of in browser editor to play with this Go to right now.

Let's see.

Um, So, um and so this this platform called called hydra that I've been working on it lets you send video streams to other people who are connected.

But it also lets you kind of play with this way of making images directly within the browser.

And so, um, I'm gonna actually clear the canvas right now, And, um, so drawing from analog synthesizer world, one of the basic things that you can make in hydra is an oscillator.

But then when you just right oscillator, you don't actually see anything.

And so then to connect it to actually see it, you have to connect it to an output.

And so, um, when a name doing control enter to execute this coat.

And so here have an oscillator and I'm connecting it to an output they sort of use functions to correspond to this idea of the analog modular synthesizer on and then chaining the functions as sort of the patch cables connecting the different pieces of of the analog modular synthesizer.

And then the parameters here can correspond.

Sort of like turning the knobs off of the analog synthesizer working, um, change, change certain parameters.

Um And so here here, I'm I am changing the frequency of the oscillator.

How fast it's moving in, how separated the colors are.

Um, And then I can add more more chained functions onto this.

So more transformations of the signal.

Six.

Similar to the boxes in the modular synthesizer, Um, and so it can keep adding, There's oscillator, roti.

And now I could add a kaleidoscope.

Um, change them.

Repeat the kaleidoscope.

And so one of the things I'll suffer with this that's interesting to me is, um, it's It's sort of a lot of the ways that we make graphics on the computer use also different analogies.

For example, Um, canvas.

It's like the idea that you have a canvas in your drawing things onto the canvas are using Microsoft Word or something.

This idea that you have a document in your typing text onto it.

And all of these technologies kind of create a framework for thinking about making things using the computer.

Um and so instead of using either the idea of a canvas where you're drawing shapes or using a three D rendering thing like three gs where you have a camera and a mash and lights.

Now, this stuff, um, here, I'm really interested in this idea of signal flow and using signal flow as a way to to create images.

Um, and all of this this each function corresponds to ah g L a cell shader function and a shader is a program that runs on the graphics card, and it basically, um, is given a coordinate and outputs a color.

And it's sort of most basic sense on DSO.

So each each of these functions either transforms a coordinate to a color, transforms a color or transforms a coordinate space and then adding them together kind of lets you experiment with these things.

And, of course, the order that thing's air in matters.

So like, if I put another rotate here than, can it start rotating afterwards?

Um, and I could also, instead of using an oscillator is a source I could use for eating them.

For example, I could use ah, camera of the source.

Um and so, uh, it's kind of dark, but that's all of you.

Uh, and so I could I can also sort of a play apply transformations to the celibacy.

Um, yeah, it's a little It's a little dyke.

I'm gonna switch to my camera, so we will see about her.

There we go.

Um, and, uh, a flight transformations, Um and so, no, we just sort of have one source and one output I've shown you and I could It could execute different code to sort of switch back between the two.

Um, but we thought ho the whole analog synthesizer.

There's all these options for, like, patching things into each other on DSO.

So in this system, in order to be able to do that actually have for sort of virtual outputs, um, with which, with different names.

And so I could send the oscillator instead of to the first screen.

I could send it to the second screen, which, in this case has a name.

No one.

And I'm gonna simplify this a little.

Here So So here I am sending the camera to the to the first output Ozio and I'm sending the oscillator to the second output.

Um, And then what I could do is actually blend Blend these together so I could take 00 and blend it with one and send it out tow to here and so here and blending the two together.

And if I change one of them, it changes all of them again.

This is all running directly on the graphics card with through mgl and shakers.

Um, and and simple if you're familiar with photo shop, our video editing software There's different, um, ways of compositing pixels together.

So here I'm taking the difference between the colors.

If you think of each color as a number between zero and one RGB channels and then you can do different sort of arithmetic.

Um, but then another thing that I borrowed from analog synthesizer is not just combining colors together, but it's actually using the colors of one thing to affect the coordinates of another thing.

And so here, um, here it's using the oscillator to sort of work, Um, the the initial image and, um, this I'm all doing this sort of in separate function changed.

But I could also do it.

Ah, all sort of together.

And so it could just here say, I'm gonna take this and modulate it with an oscillator.

Um, I'll just show here, um, and I could, I don't know, put another oscillator on top here and just keep keep kind of chain chaining all these things together.

And it creates a really complicated patterns really quickly.

And this this is just really fun for me.

Um And then so, uh, using different Web a PS Also, you can not just use the camera as in as in, um, input, but also lots of other things.

Oh, and this is one of the things that Dan Sand and said about this sand, an image processor in the seventies is the modules are designed to maximize the possibility of interconnection, thereby maximizing the number of possible modifications of image.

So that's like when I'm thinking about the functions that are there, it's how can these functions really a maximize the possibilities of the way that you can You can mix them together s so this is using the screen itself as an input these air sort of experiments that I've done using this this idea of the video synthesizer, Uh, this is using my twitter feed as an input using the desktop capture a p I you can use any window as an input.

Um, this is using Google maps, says as an input and sort of experimenting with it.

And this is using a live stream of an aquarium and the live stream of ah of Times Square and then live coating them together.

Um, and this is sort of going back to the the satellite arts project of just sort of combining to live things.

And what, Er, what kind of possibilities air ideas about those two spaces arrive arise by putting them on top of each other.

And so another thing that, ah, when I clear this year, that is really exciting.

For me to experiment with this is is video feedback or sort of using the output of something as the input to itself or, in other words, sort of Rikers in, um and so here.

Let's see.

So here here is, um, the camera, um, I'm actually gonna switch cameras here to the other one.

Um, and So So the basics of video feedback is just pointing a camera at its own screen.