CHRIS KELLEY: Thank you so much for joining us.

My name is Chris.

I'm a designer and prototyper working

on immersive prototyping at Google,

and I'm joined by Ellie and Luca.

And today, we're going to talk about exploring AR interaction.

It's really awesome to be here.

We explore immersive computing through rapid prototyping

of AR and VR experiments.

Often, that's focused on use case exploration or app ideas.

We work fast, which means we fail fast,

but that means that we learn fast.

We spend a week or two on each prototyping sprint,

and at the end of the sprint, we end

with a functional prototype starting

from a tightly scoped question.

And then we put that prototype in people's hands

and we see what we can learn.

So this talk is going to be about takeaways we have

from those AR explorations.

But first, I want to set the table a little bit

and talk about what we mean when we say augmented reality.

When a lot of people think about AR,

the first thing they think about is bringing virtual objects

to users in the world.

And it is that.

That's part of it.

We call this the out of AR.

But AR also means more than that.

It means being able to understand the world visually

to bring information to users, and we call this understanding

the in of AR.

Many of the tools and techniques that

were created for computer vision and machine learning perfectly

complement tools like ARCore, which is Google's AR

development platform.

So when we explore AR, we build experiences

that include one of these approaches or both.

So this talk is going to be about three magic powers

that we've found for AR.

We think that these magic powers can help you build better AR

experiences for your users.

So we're going to talk about some prototypes that we've

built and share our learnings with you

during each of these three magic power areas during the talk.

First, I'll talk to you about context-driven superpowers.

That's about how we can combine visual and physical

understanding of the world to make magical AR experiences.

Then Ellie will talk to you about shared augmentations.

And this is really all about the different ways

that we can connect people together in AR,

and how we can empower them just by putting them together.

And then Luca will cover expressive inputs.

This is about how AR can help unlock

authentic and natural understanding for our users.

So let's start about context-driven superpowers.

What this really means is using AR technologies

to deeply understand the context of a device,

and then build experiences that directly leverage that context.

And there's two parts to an AR context.

One is visual understanding, and the other

is physical understanding.

With ARCore, this gives your phone

the ability to understand and sense

its environment physically.

But through computer vision and machine learning,

we can make sense of the world visually.

And by combining these results, we

get an authentic understanding of the scene,

which is a natural building block of magical AR.

So let's start with visual understanding.

The prototyping community has done some awesome explorations

here, and we've done a few of our own

that we're excited to share.

To start, we wondered if we could

trigger custom experiences from visual signals in the world.

Traditional apps today leverage all kinds of device

signals to trigger experiences.

GPS, the IMU, et cetera.

So could we use visual input as a signal as well?

We built a really basic implementation of this concept.

This uses ARCore and the Google Cloud Vision

API that detects any kind of snowman

in the scene, which triggers a particle system that

starts to snow.

So through visual understanding, we

were able to tailor an experience to specific cues

in the environment for users.

This enables adaptable and context aware applications.

So even though this example is a simple one,

the concept can be extended so much further.

For example, yesterday we announced the augmented images

API for ARCore.

So if you use this, you can make something

like an experience that reacts relative to device movement

around an image in the scene, or even

from a known distance to an object in the world.

If you think this concept is interesting,

I highly recommend checking out the AR VR demo tent.

They have some amazing augmented images demos there.

The next thing we wanted to know is

if we could bridge the gap between digital and physical,

and, for example, bring some of the most delightful features

of e-readers to physical books.

The digital age has brought all kinds of improvements

to some traditional human behaviors,

and e-readers have brought lots of cool new things to reading.

But if you're like me, sometimes you just

missed the tactility in holding a great book in your hands.

So we wanted to know if we could bridge that gap.

In this prototype, users highlight a passage or word

with their finger and they instantly

get back a definition.

This is a great example of a short-form-focused interaction

that required no setup for users.

It was an easy win only made possible

by visual understanding.

But as soon as we tried this prototype,

there were two downfalls that we noticed,

and they became immediately apparent when we used it.

The first is that it was really difficult to aim your finger

at a small moving target on a phone,

and maybe the page is moving as well,

and you're trying to target this little word.

That was really hard.

And the second was that when you're highlighting a word,

your finger is blocking the exact thing

that you're trying to see.

Now, these are easily solvable with a follow-up UX iteration,

but they illustrate a larger lesson.

And that's that with any kind of immersive computing,

you really have to try it before you can judge it.

An interaction might sound great when you talk about it

and it might even look good in a visual mock,

but until you have it in your hand

and you can feel it and try it, you're

not going to know if it works or not.

You really have to put it in a prototype

so you can create your own facts.

Another thing we think about a lot

is, can we help people learn more effectively?

Could we use AR to make learning better?

There's many styles of learning, and if you

combine these styles of learning,

it often results in faster and higher-quality learning.

In this prototype, we combined visual, oral, verbal,

and kinesthetic learning to teach people how

to make the perfect espresso.

The videos explain--

I'm sorry.

We placed videos around the espresso machine

in the physical locations where that step occurs.

So if you were learning how to use the grinder,

the video for the grinder is right next to it.

Now, for users to trigger that video,

they move their phone to the area

and then they can watch the lesson.

That added physical component of the physical proximity

of the video and the actual device

made a huge difference in general understanding.

In our studies, users who had never used an espresso machine

before easily made an espresso after using this prototype.

So for some kinds of learning, this

can be really beneficial for users.

Now, unfortunately for our prototype,

one thing that we learned here was

that it's actually really hard to hold your phone

and make an espresso at the same time.

So you need to be really mindful of the fact

that your users might be splitting

their physical resources between the phone and the world.

And so as it applies to your use case,

try building experiences that are really

snackable and hands-free.

Speaking of combining learning and superpowers together,

we wondered if AR could help us learn

from hidden information that's layered in the world

all around us.

This is a prototype that we built

that's an immersive language learning app.

We showed translations roughly next to objects of interest

and positioned these labels by taking a point cloud

sample from around the object and putting the label sort

of in the middle of the points.

Users found this kind of immersive learning really fun,

and we saw users freely exploring

the world looking for other things to learn about.

So we found that if you give people

the freedom to roam and tools that are simple and flexible,

the experiences that you build for them

can create immense value.

We now have physical understanding.

This is AR's ability to extract and infer

information and meaning from the world around you.

When a device knows exactly where it is, not only in space,

but also relative to other devices,

we can start to do things that really

feel like you have superpowers.

For example, we can start to make

interactions that are extremely physical, natural,

and delightful.

Humans have been physically interacting

with each other for a really long time,

but digital life has abstracted some of those interactions.

We wondered if we could swing the pendulum

back the other direction a little bit using AR.

So in this prototype, much like a carnival milk bottle game,

you fling a baseball out of the top of your phone

and it hits milk bottles that are shown on other devices.

You just point the ball where you want to go, and it goes.

We did this by putting multiple devices

in a shared coordinate system, which

you could do using the new Google Cloud Anchors API

that we announced for ARCore yesterday.

And one thing you'll notice here is

that we aren't even showing users past their camera.

Now, we did that deliberately because we really

wanted to stretch and see how far we

could take this concept of physical interaction.

And one thing we learned was that once people learned

to do it, they found it really natural

and actually had a lot of fun with it.

But almost every user that tried it had to be not only

told how to do it, but shown how to do it.

People actually had to flip this mental switch

of the expectations they have for how a 2D smartphone

interaction works.

So you really need to be mindful of the context that people

are bringing in and the mental models they have

for 2D smartphone interactions.

We also wanted to know if we could help someone visualize

the future in a way that would let them make better decisions.

Humans pay attention to the things that matter to us.

And in a literal sense, the imagery

that appears in our peripheral vision

takes a lower cognitive priority than the things

we're focused on.

Would smartphone AR be any different?

In this experiment, we overlaid the architectural mesh

of the homeowner's remodel on top of the active construction

project.

The homeowner could visualize in context

what the changes to their home was going to look like.