So yeah, it's got a fancy name. It sounds cool. It is quite cool.

Let's imagine you're running a game, right?

I don't do that as often as I'd like anymore.

But- but- maybe you're pushing your graphics card right to the limit of where it- where it's happy, right?

You've got a 4K monitor, you're pushing a lot of pixels to the screen,

the game is a big game with lots of- lots of effects.

The problem is that then your frame rate's going to start suffering.

So maybe what we could do is run at a lower frame rate,

which is going to be that much easier for your graphics card, and then use deep learning

to recreate that 4K image - and if the deep learning is good enough, you won't be able to tell the difference.

So the most recent generation of Nvidia graphics cards have these Tensor Cores on board, right?

Tensor Cores are basically very quick matrix multiplication circuitry.

Matrix multiplication comes up a lot in deep learning, which is kind of what these were designed around.

But it has some applications to games, a little bit.

I'm going to betray my lack of knowledge about modern games, but I don't really know much about them.

I play some games, but very easy ones.

So, I- you know, I'm not running games at 4K and worrying about framerates, right?

But some people do concern themselves with these things and they spend a lot of money on these graphics cards,

and they want them to run as well as they can.

The problem is that maybe some game comes out,

and it has a huge demand on your- on your GPU, right? For every pixel in your game.

The GPU's gotta work out which triangles in the world

it needs to - you know - it needs to render what color they're going to be,

it's got to include lighting and shadows,

you know, blurs and - you know - depth of field effects, like I was talking about last- last video.

You know, this takes a long time and the more pixels you use, the worse it gets.

OFF-SCREEN: And motion blur, of course?

And motion blur, I know people love motion blur. I myself can't get enough of it.

What do you do about this? One thing you do is you just make your graphics cards faster, right?

This is something that happens every generation, not just on NVIDIA cards, but - you know - all graphics cards, and that helps a lot.

All right, but the problem is we've had a jump from 1080P

To 4Kand that is not a slight increase in number of pixels

It's four times the number of pixels which on a simple level means four times the amount of work right to render

This this screen and you've got it all so when we talk about things like super so

Something sometimes we're looking at a pixel more than once right and that means it's getting really slow

So what Nvidia have tried to do here is

Say well

Maybe we can we can we can save quite a lot of performance by let's say running our game at 1080p, right?

But if you up sample that to 4k, maybe it won't look very good because basically it's just going to be blurred

so

maybe we could do a slightly more a

slightly smarter up sampling using these sort of deep learns to prepare solution techniques as I understand it even modern TVs these days do

Scaling and they're scale things up and blu-ray recorders DVD recorders are always

You know had an element of doing this scaling is this just a more advanced version?

Yeah, and some TVs have starting to bring in as far as I know

Deep learned, you know smart AI driven up sampling right the idea

Is that so what happens when you run a game is you've give adding all these effects on top of one another to?

Get a good performance

now if your performance if your frame rate starts to drop

What you're going to do probably is either drop your resolution or if you don't want to drop your resolution because you don't like it

You start to remove some of the effects, right?

So you drop your shaders down from high to?

Something else and that will just reduce the amount of work per pixel and if slightly increase your frame rate

But some people don't want to do this

Right, they spent a lot of money on their computer and they want to run it on full graphics

So but maybe a game has come out. There's just really really demanding

There's kind of two problems we want to solve here one

Is this problem of aliasing, right which is but if you rent if you rasterize a scene with triangles

They don't always line up exactly where the pixels are. So you get a kind of jagged edge, right?

That's one problem, which doesn't look very nice and there are techniques to get around this the other problem

Is this issue of resolution, right? If you drop down from 4k to 1080p you gain months, you know gone four times faster

That's a huge benefit. Right? If you could do that without noticing and the difference in appearance

Well, that's a winner right that's going to be great. And then you can start putting it even even more

Computational time on shader effects and things so but yes, you're running out a lower resolution

But those pixels are better had a good deal time spent on them. They look really nice

What are the problems we have? Is this problem called aliasing right now?

I will just talk about this very briefly because it's not really what this video is about

But if you're valuing an image and your triangle Falls like this, then this pixel actually isn't all of this object here

Maybe this whole dish is dark and this object is light. It's not all of this object

It's like 70% dark and 30% light now. The problem is that there's no way of doing this

So if you saw and pull this pixel here this pixel here and this pixel here. You're going to get an edge that goes light

Dark light like this, right and that looks ugly

So what you'd use is a technique usually called some multisample

Anti-aliasing where essentially you take multiple readings from this pixel like these four here or something more dense than that

And there's lots of different ways to do this and then you have those

values and the nice thing is then you've got three readings of dark one reading of light and you come out with a reading of

About 75% dark and you get a smooth edge

This is one thing that graphics cards do to try and make things look a little bit better if you turn off all these

Anti-aliasing approaches, then what you'll get is your your core jaggedy lines?

It'll run nice and quickly. If you're sampling four times per pixel, that could be a four-fold decrease in speed

Right and that has a performance hit unless your graphics card is just amazing. Mine is not that's one problem, right?

the other problem is but you know when you go up to 4k

It's just four times the number of pixels whatever you're doing per pixel is

Multiplied by four these four times in our pixels that you know, that's a huge problem. So if you're running

4k and four samples for example per pixel

That's a lot more computation

but if you were just running without anti-aliasing on

1080p and so you inevitably have to drop down somebody's setting so that you can get a good framerate for your game

And it doesn't look as nice. So one option is just to make the graphics cost faster, right?

This can't always, you know, this isn't the answer to everything and they get more more expensive, right?

That's also a problem

these new cards have in them these tentacles which are very good specific operations namely matrix multiplications and

So there is a chance of what we could do is use a deep network to kind of just clean this image up for us

Very quickly before it's presented to the screen and that will give us a nice presentation for our image without the huge

Performance hit of doing all this anti-aliasing and all of this

massive resolution, right

So what board is speaking it works like this your computer?

will render a raw frame with

Aliasing and all the kind of problems that it has and this may or may not be at the same resolution as your monitor

It will then be passed through a deep network

which is not very deep because

this has got to happen pretty quickly which utilizes the tensor cause of these new graphics cards to work very very quickly and that will

Produce your knives 4k with anti-aliasing shot

Which theoretically looks really really nice the way they train this network when they take the lower resolution alias version and the higher resolution

Anti-alias version is they're going to train it basically to try and create the image on a pixel level as closely as possible

But they'll also add additional parameters like that. It looks perceptually nice high

So basically perceptual loss functions which try and make things look aesthetically closer to something right now

Different loss functions are going to have different effects. So we trying out all these different loss functions

They might even use adversarial loss functions which are these adversarial networks of what Mars talked about, right?

There's loads of different ways to train these and how you do that. It's going to influence the actual result you get

All right, because it's not worth going to work perfectly. So there's kind of two answer answer questions here

I mean firstly does this work

Right and and the other might personally

I don't know because I don't haven't tried this right but I think it the results vary right? That's certainly true

But also how do we train this new your network, right?

Because what you don't want to have happen right is like the fate you're unlocking the phone thing

What was it unlocking your face with your phone if you to unlock a face of your phone?

You don't want users to have to do this, right?

This is something for NVIDIA to do if they're gonna you know

Silvus and make money off this technology and that's exactly what happens

sometimes shortly before games release the game developers will send an early copy to a midea an

Nvidia will start generating training data and train a neural network to do this process to take an image, but isn't quite as nice

It's got aliasing it's lower resolution and perform this up sampling smart up sampling up to 4k

anti-aliased

right

that's the idea and they do this by generating essentially a

Perfect representation of the game using 64 samples per pixel anti-aliasing, right?

So that is for every pixel they do 64 samples from that pixel instead of just one really nice

sixty-four times slower than normal and then they take that as their

Output and the input is just serve all frame with no anti-aliasing at all

Maybe lower resolution and they train this network to take the raw frame and output the 64

Samples per pixel really nice frame, right?

And so really what it comes down to is whether in practice this works

Right and the answer I think is probably some of the time yes some of the time no

This is true of most deep learning right people don't tend to say this as much as they should but you know

will it generalize if you take

10 million frames of battlefield 5 and train this network on them to get as close to this output as possible and

Then you generate the 10 million from one. Don't even want the next frame, right?

if you generate the next frame, will it have as good a

Performance on that unseen frame and the answer is usually pretty good, but it won't be ever be perfect

right, especially if you're going from 1080p to 4k

so I think NVIDIA kind of made the point here that actually this is about when you're running at the very top end of your

graphics cards capability and so in some sense

They're not talking about people who are barely struggling to run the game at 1080p

You should already barely run the game at 4k and then maybe this will make it look slightly nicer first kind of two ways of

Doing this one is you take a 4k input and you use this to perform

Anti-aliasing and the other is you take a low resolution input and you use this to perform both

Anti-aliasing and up sampling and that's a harder job to do because if you imagine that you've got a 1080p saw

Then what actually you're going to have is a series of pixels like this and you've got to come up with all of these

Pixels in between right and this is just like increasing the size of an image enhance enhance by you know, will it work?

I don't know. It's going to be better than

Violent bicubic up sampling, right?

Because it's going to be bearing in mind this local area. It's going to say well look there's an edge coming down here

So this needs to be sharper, this doesn't need to be as sharp things like this

But this is not an easy problem to solve and you know by Nvidia's own admission. This is an ongoing process

they continually train these networks on a supercomputer and then

You know, hopefully they get better and better we shall see right

well

One thing I think is quite interesting is that it means that essentially a deep network is part of your game experience

On your GPU and so the weights for this network

The parameters of this network are actually going to be shipped out with drivers, which I think is quite neat, right?

So you're no longer just getting graphics drivers which have performance optimizations for games?

and of course the hardware the hardware control software, you've also got this network weights being shipped around by they're quite big so

you know and

So that's why you get limited support for games early on because they're training these maybe they haven't gotten early copy of the game

right

So it's it's it's down to Nvidia to just take these games render these super high resolution

64 times amazing scenes and train these networks strikes me that even if it's been trained

Running a network a deep network is not computationally cheap though. Is it is it worth it?

I suppose what's the trade-off so I mean, I guess that's the question people are asking at the moment, right?