And hello.

Today I'll be talking to you about the case for augmented reality on the web.

I'm Marley Rafson.

I'm a software engineer at Google on the web XR team.

My Twitter handle is mprafson.

Love to talk letter.

The two things today with the web and augmented reality.

I believe that the combination of these two things, the whole is greater than the sum

of its parts situation.

I think that the web will gain from augmented reality, and things that augmented reality

will certainly gain from the web.

So, to begin, let's start by getting everyone on the same page and let's define augmented

reality.

It's the mixing of computergenerated content with the real world.

To me, we can now add contextualized information into the world around us.

So, for example, you're trying to make a coffee at the coffee machine, and you have the instructions

overlaid on top of the coffee machine.

And implication of augmented reality is you can interact with 3D content in a 3D space.

So, the canonical example would be you're trying to buy a piece of furniture and you're

able to put it into your room before you buy it.

Let's look at what augmented reality looks like today.

Today this is smartphone AR.

This is run using a smart phone and basically has some understanding of the world around

it allowing virtual content to interact with that world too.

We have augmented reality on headsets.

Your hands would be free, and you would be free to interact with the content using your

hands.

And the third example that I think is less talked about, but I find compelling is audiobased

augmented reality.

You're seeing Bose frames.

It's a pair of sunglasses that have speakers that shoot audio into your ears.

With this example, with audio AR, you would have that experience where you're at a coffee

machine and you don't know what to do.

But instead of seeing the information, it's read off to you.

And it I would also like to say if you hear me say AR, that's just short for augmented

reality.

So, this talk, as I mentioned, it's also about the web.

Let's talk about where those two things intersect.

That's what we'll call the immersive web.

And the immersive web is the use of augmented reality, virtual reality and other 3D technologies

on the web.

It's this whole spectrum of things.

It can be more augmented reality than virtual reality and doesn't quite matter.

And you'll hear me say XR.

And that really just stands for wild card reality.

So, it could be virtual, it could be augmented reality and some combination of the two.

So, today what the immersive web looks like is something like this.

We have seen progressive enhancements of sites we use every day.

The left two examples, Facebook 3D posts and they're incorporated directly into the newsfeed.

So, that first example is a 3D model that you can interact with on the newsfeed.

The second is using portrait mode depth information to delightfully play with these photos with

a bit of depth added.

And the third example is from Wikipedia who about a year ago started supporting 3D models

in their articles.

So, if you're not a large company and you don't have a whole engineering team dedicated

to building this model viewing experience, there's a solution to that now.

So, recently Google, a team at Google open sourced a web component for viewing models.

It's called Model viewer.

It's fantastic.

Using HTML, you can add into your side around it.

That's an example with the astronaut which we know and love.

And something that's great about the modelviewer team is they're thinking about accessibility

early.

So, they've already incorporated alt text into the component.

So, let's say you have these 3D models on your website.

How do you actually get it into augmented reality?

So, right now what we've seen is implementations of native system viewers that allow for augmented

reality.

This is not running in a web browser, but this is slick and tight integration with the

web.

So, you would have this search result, so it's integrated into search with Google.

So, you can tap the button and then it intents into this native viewer and then you can place

it in augmented reality is the same exists on iOS and it's similarly intense.

And so, what I just said there was that we're using native viewers to do augmented reality.

But we would like to do that in the browser itself.

That's where the WebXR device API comes in.

It exposes lowlevel sensory information.

So, the camera pose, if you can place an object on the floor.

And that's under development.

It's open so we can all contribute to it.

And you can find that on GitHub.

And with those examples that I just shared, that's all talking about bringing immersive

technologies into the web.

But we can also bring the web into immersive technologies.

Two examples of XR browsers.

And you will see the Helio browser placing images into the world around you and then

Firefox Reality, which is a VR browser.

So, I think the immersive web today is fantastic.

I think it's adding information, it's making it more delightful to browse the web.

But when we're developing these technologies, it's important to see where we're going next

and to start thinking about that earlier rather than later.

So, when I think about a future of augmented reality, I start by thinking about how can

this assist me?

So, I think about maybe I'm leaving work one day and I'm wearing these lightweight AR goggles,

I'm walking down the street.

I'm looking for dinner.

So, as I look at the restaurants on the street, I'm seeing Yelp data populating with the stars

rating and the types of menu.

And maybe I also want to see photos of the food.

So, Instagram information's coming too.

And I like Pokemon.

So, maybe while I'm doing all of this, I'm catching Pokemon, playing Pokemon Go.

And when I get to the train station, I'll have information populated for me telling

me which train to take so I can get home fastest.

And as I mentioned earlier, I probably don't want to be restricted to doing this on just

one headset.

Maybe I want headphones another day from a different company.

Who knows?

Maybe I just want to use my smartphone.

So, when I'm thinking of this vision of augmented reality going forward, I start to see some

interesting things and patterns that I would like to call out here.

So, the first is that is inherently uses web content.

I want access to all the things that I already use when I'm browsing the web.

Another thing is it's this really interesting composition of different types of information.

So, that could be models or 2D pages.

And we're going to have to interact with them in a way that's esthetic and delightful and

still makes me want to use it.

And onethird trend that I start to see it we're really interacting with the world around

us.

I could be places information on to a building or on to a tree or on passing by cars.

And so, when I start to think about this vision that I want and where we are today, which

is mostly placing 3D models, I think there is a lot of work to go.

And where I see a lot of that work coming into play is with a user agent.

So, the definition from W3C of a user agent is a user agent is any software that retrieves,

renders and facilitates end user interaction with web content.

So, today that's usually just your browser.

So, you get a lot of things when you're browsing using any of the modern browsers.

It could be Chrome, it could be Firefox.

It helps you do safe browsing.

It renders the HTML pages for you and things like that.

So, to understand how a user agent will play out with this added dimension for augmented

reality, let's start to break down this definition of user agent.

So, I'll go in reverse order.

Start with a user agent facilitates end user interaction with web content.

So, what I believe is that the user should always be in control of how information is

presented to them.

So, on a modern web browser, that's preferences like default text sizing.

That's contrast ratios.

And Chrome extensions and things like that.

But when you think about augmented reality, we have this whole new dimension.

It's really immersive.

It's, you know?

And so, we need to start to think about other ways that we want to have safe browsing and

we want the user to be in control.

One example, think about physical proximity.

You have content that's being rendered possibly close to you and I should be able to say I'm

not comfortable with anything rendering closer than 5 meters away from me.

Or I think it's compelling to think about this in terms of sound.

I probably would never want to allow someone to whisper into my ear as I'm walking down

the street.

so, a lot of these considerations come into play when we're talking about augmented reality.

And in order to accommodate these things for the years, we're going to need an intervention

point for the information.

So, the user agent knows what's being asked of it to do this.

If we think of declarative content that we know and love, that would be HTML and CSS.

I see this future where there's declarative content that you're able to add into the world

around you so you could specify things like I would like float left and pin to a building.

Or you could say, I want the depth to be 5 meters from the user.

But it's up to the user agent to decide whether or not it can actually honor that request.

And it's in that way that it can advocate for the user.

So, another thing that declarative content offers is that is provides semantic understanding

for the user agent.

So, what that means is that the user agent has a view into what's being asking to rendered

or presented to the use.

I find this example extremely lie lighting of what I'm talking about.

If we look at the left side, you're seeing the web.

If given this canvas to render the object like on the web today, given an XY location,

you can only see the color.

I see a black pixel, a white pixel, a gray pixel.

But nothing more than that.

But with semantic understanding of what's actually in the scene, we have a much more

rich understanding of the model the and the content that we're placing.

So, if you look at the right side, you can see I'm looking at the windshield.

And it has a material that's class.

And it's transparent.

You can also see things like the tire and stuff like that.

And so, you can imagine with screenreaders or accessible or just translating this into

soundbased AR, you can actually talk about what the user is seeing.

So, you could say I see a van with black tires and white car metal paint.

So, the next part of the user agent that I'll talk about is rendering.

So, the user agent is response for rendering to the browser.

So, modern web rendering emergencies are amazing.

They're fantastic.

That's blank, echo, WebKit, things like that.

When you think about what they asked to do, it's mostly 2D and it's textbased.

They're fantastic at rendering pages like the verge at 60 frames per second.

And some of you might know and some of you might not, but the web already does performant

3D rendering.

That's great and provided to the user through a Canvas element and uses WebGL to create

those graphics.

What that might look like today is this, this is not augmented or virtual reality.

But it's just rendering in a web page using a canvas and WebGL.

So, for the rendering needs of augmented reality, thinking back to the vision is this complex

composition and layout dynamics of content.

So, we're gonna probably need to build a rendering engine that can handle that from the ground

up.

Whereas the modern rendering engines were optimized for 2D.

And so, even though I just said that we probably want this authorized from the ground up and

that we don't want to use Canvas elements, we tried it.

So, my team experimented on this internal prototype, air web view.

It's a set of libraries that enable AR through native apps and web view.

It's a lightweight browse their you can embed into a native app.

The technical understanding of what we've done is that we've web rendered and using

that to do than we have tracking, AR Core and ARKit.

And we combine that with a natively rendered camera feed.

Let's look at what that looks like.

You look at the example, we have a stormtrooper and it's convincingly rendered into the space