I'm Oliver

And this talk really is it is about user experience, but it's much more about

Finding good applications for machine learning in the browser and I want to kick this off because I have zero slides

So I want to kick this off with a stone joke

Hopefully no one knows it already. So this is about user experience. And oh sorry, this is about user experience in the way

It actually works. So hope for the internet works here. So this is what you the user do

So it doesn't work quite

Obviously guys wrong

Fixing it. This is I mean

Because you're laughing. I know you have the same feeling as me. Yeah, if you have watch people using user

So same thing fixing it

Again, yes

So that's the joke that I stole but obviously it worked

Because I think it only works if you have some sort of experience like this you think so hard about you user

Interface and then that's what people end up doing

They will figure out how to how to deal with it, but maybe not them sense you expected it

So this talk is about the same thing

Don't blame me for the for the styling

This is my weak spot. Oh wait

so, um

In a typical user interface. So this regard those buttons those are real buttons the the buttons down here. I'm not real buttons

so the buttons are just here to

make a point so typically

In a very classic

user interface the buttons they will light up when you hover over them to give people the the

certainty when you click now, you will click this button, so

What I had in mind was would it be at all possible to highlight the buttons before you click them?

So would it be possible that the machine learns? Which button are you going to click now on next and

You may ask what could be the applications of this and I've talked to quite some user experience people that said

Nothing or less and then they thought for a little bit longer. So maybe I asked you so I get something back here

Why would even do that? I mean highlight the button before the user clicks on it

Predicting which button is it going to be any ideas any takers? Yes

Perfect so about flows so actually I haven't implemented this but with the buttons up there

that's a horrible user interface because you need to click those buttons in certain order and

Some orders are totally don't make any sense you get weird error messages

This would be my next step like learn how to actually use those buttons up there because they're they're complex

But I haven't implemented this yet. So what?

What would be another reason why would you why would you do that? Yes

Accessibility yes, so make it easier. So that was also my fault. Yes. You have another point. Yes in the back

Sorry

Yes preload resources that was also something that I came up with

someone said maybe there so I don't know a server less function behind that button and you want to start it up because you think

Oh, the user is going to click on that button soon. You better have this thing ready or

Preload the next page or someone

I've written down some stuff. Wait. Oh

yeah, someone came up with the idea when there's artificial reality and it's always

Pretty hard to decide which detail are you going to show and then if you can predict where the user is going to look next?

then you can preload that stuff and

Again I only implemented the easiest word because I just wanted to know it's this at all possible

Could you even do that? And the easiest version that I implemented is just what you said

I highlighted the button that I think the users going to click. So, um

There could be heuristics to do this you could just measure I don't know

It's the user close to a button like this

and if the if it's reasonably close then you would highlight that button which I don't do now and

You could also check its the mouse closer to this button

And so they and it might all work, but it might not match the individual style

That someone has that used this user interface

for example

my I always use the trackpad and it's

extremely sensitive because I don't move much and that might mean if I want to hit this button I might

Briefly go over it and then go back

That means depending on the user that you're having and depending on the machine that you're using and how is it set up?

The mouse movements might be completely different. So I'm pretty sure

In my I'm not sure that I'm pretty sure a machine might identify me just by my mouse movements

Out of all of the people in the on somebody's nodding. I haven't made the experience but my mouse

Mouse movements are so weird. Italy makes people really nervous

angry

so

Having said that so what I came up with is that I said, maybe it's possible to learn that

Those movements using machine gun and

If you want to do that, the the crucial part is you need data

So the typical machine learning that we do you need to have data and if you don't have data

Probably not going to work, but fortunately I can generate data on the fly and I'm doing it like this

So once I hover over button, I generate a data set and typically the user would also need to click that button

But it would take me forever to generate just a few data samples if I really clicked on that button

So for the sake of this demo, I'm just counting hovering over button is a click

this is this is just fake or hacked but that's the only hack part and I'm doing this to

quickly

Get some data. Yeah, and what kind of data would you I mean, I'm

Generating data. What kind of data would you would you collect what would you do?

I mean if you were to predict the button I'm going to click you solve those movements. What would you recall?

Yes

Proximity you could record that I did not do this. What else?

Yes

Yes, that's what I did so I recorded XY of the mouse I recorded the Delta between the previous

Movement because I just get this from the mouse movie event. I just get this for free and

delta-t exactly what you say so and

There's also something really special about this. I recalled a sequence of

So to say time stamps and I can show these

If I find them on local storage, there you go, so I stole them local storage and

They are they give me a second here

And that's how they look like so this why hopefully it's big enough is it big enough for the back

Yeah good. So the why it's just an idea of the button and two means this one I

Think so. This one should be

One and this might be zero or maybe this is one. This is two this is three, but I'm not quite sure but why represents?

The button and I

Made this already complicated so x and y don't mean positions on the screen, but they mean the input sequence

This is x and y is the button that I'm going to click so

this is the input sequence of

Mouth positions you see this is position X this is position Y

Delta X Delta Y that means difference from the previous mouse position and this is the time step or delta T

So how much time how many milliseconds past from this event to the other?

And sometimes it's very similar and sometimes it's different

so that's my input and

I

Did some pre-processing on that because I was considering

how many of them would I really need like maybe 10 maybe 100 because I didn't know I

Cut all these sequences into parts and trained all of this into the model

So but you could like for example

What would he suggest that you could have taken anything else and if it works you find out by experiment?

That's how it works in machinery. So you make an assumption you try the model and then you see if it works and

well, this might work at see so I have the data now and now I

have this

This collection of buttons and I'm the only person in the world right now who knows how to

Use this thing because it's a horrible user interface giving a talk but user interfaces and then they playing horribly. I

Apologize, but that's the way it is. Now I can train on that data

So I collected some data and I collected some data before you see I have

369 dataset that means um, I don't know my wave down here from over from outside to down here. I

Did some clicks to have some sample data and if you're at all into machine learning having?

370 samples for three categories is nothing

Tippity people won't even start they will say this is not going to work. Well

But maybe does so I'm clicking this train model now, this is all

I've no fall back. That's why I'm having sweaty palms here. So this looks good

reasonably good

this is using tensorflow, Jess and

Tensorflow jess comes with a very nice brutalization library. I did not write this code

So this stuff that comes with tensorflow Jess and it shows you something and hopefully something good

Oh and shows me something good, but I didn't know before I kicked off this thing. So it looks good

What's most interesting is these?

Two curves down here. That's why I'm doing it now

and

These curves tell you something about how well the training process

Proceeds I will show you some code later

But let's save on the conceptual level for a while

And we concentrate on that blue curve and the blue curve describes over time

How well is the machine learning a model able to reproduce the data that that I gave to it?

so I've had in the mouse movement sequences and

as a prediction the button that is going to be clicked and now the machine learning tries to

learn which sequence

Spits out which bar and

the higher the blue line gets the better it is able to do that and this is around 70 percent or

0.67 that means of the 70

Sorry off of all the mouse movements that I did 70 percent of them. The model is able to predict correctly

so I made some movements to the to the button and

That's pretty good 70 percent on that

pretty astonishing

But what's much more astonishing is the orange line because that is data points that the model has never seen that means

20% which I decided deliberately I've taken off from the

Start now it looks good. I've taken off from from the training data, and I'm just using it

To

Constantly check how well the model is doing on something it has never seen and is anyone at all familiar with machine learning anyone?

When we say it doesn't look I mean would it would you have expected that

Who would have expected that?

Maybe you didn't expect that because I did make myself quite clear, but this is a pretty outstanding

Result so it's not something you could expect from very small training data training on this machine is a very powerful

GPU but it's still a laptop

Something like this. Typically. I don't know. It wouldn't perform that well. I'm not quite sure

Why does this is so welcome to the world of machinery, you make an experiment. You can't explain it just works and

that's my world at least javascript is also my world but also that so um

before we try that and I'm hope you saw I just tried trade this online we can get some more information and

This is also built into the visualization tools of tensorflow

and I'm always so happy if you guys don't see any red stuff in the console because it's it seems to work quite well and

This is a while. I'm hiding this for a second because what you see now, it's a very powerful tool

It's called the confusion matrix. I don't think it's only used in machine learning but in

Machine learning it actually is so what can you see on?

The y-axis that is this axis here

you see the real buttons that the model should predict because it knows those are the real buttons that means

left button middle button right button and on

The x-axis you see the prediction of the model and this is useful

Because in the perfect world and in the model that works perfectly you would expect all the clicks to be on that diagonal here

But you don't see that you see something else. So what do you see? So this means the real

value that the model should predict is left and

The prediction actually also is left that's good. But then this here

The predictor should be left, but it was middle

22 quite a few and then the prediction should be left but it is right and

Then for all the three buttons so for the middle button

we see something that at least from my perspective was expected because the middle button is so close to the other buttons that

You get a lot of confusion

Even though it is the middle button you get a lot of confusion for either left or the right button

so that's also totally expected and

Finally for the right button that's also something that's slowly expected and also something that I would have expected for the left button

So it's actually the right button, but sometimes you confuse it with the middle button, but only rarely with the left

And this is because of the way I created those silly mouse movements

But it's actually pretty good even without trying the model. I now know that this model might be good. Um,

There are some problems with it first problem

This is only on the data the model had we still have no idea

is it going to perform well in real life I mean and

will I do the same mouse movements I make a little bit interesting because I'm still I still also don't know but it could it's