transformers this architecture that uses attention to give

Unprecedented ly good performance on sort of language modeling tasks and some other tasks as well

but when were looking at language modeling and that was in preparation to make a video about

GPG 2, which is this very giant language model that has been there was recently

Well, it was recently not released actually by open AI the way that they generated the data set for this is pretty cool

to get enough text they went to Reddit and

They pulled every website that is linked to from reddit. Do we have any idea of how many days lots?

Literally, everything was everything that had more than three karma

I think or maybe more than two karma something like that like

Anything that had somebody had thought to post around it and at least two or three people who had thought was good enough to upload

They scraped the text from that. It's pretty much just a transformer. It's not the the

Architecture is not especially novel. They haven't done any like amazing new

new discovery, but

What they realized was?

Transformers it seems like

the more data you give them the better they do and the bigger you make them the better they do and

Everything that we built up until this point is clearly not

Like we haven't hit the limits of what this can do

We they thought we think we're probably

Bottle necked on data and maybe network size

So what happens if we'd like to turn that 211 what happens if we just give this all?

The data and make a really big one. It makes sense to talk about the acronym right so it's a generative pre-training

Transformer so generative same as generative adversarial network. It generates outputs to generate samples

Your pre-trained is this thing. I was talking about all of the different things

You can use a language model for right you can do you can do translation. You can try and resolve ambiguities

You can do summarization. You can answer questions. You can use the probabilities for augmenting other systems

So yeah, there's a bunch of different benchmarks for these different tasks

that you might want your language model to do and

This is what we talked about in the grid worlds video of having these like standardized problems with standardized metrics and standardized data sets

So that if you're comparing two different methods, you know that you're actually comparing apples to apples

And this is like very important it gives you numbers on these things. It's often quite difficult

Expected to like you're generating samples of text and it's like how plausible is this text? How realistic does it look like?

How do you put a number on that it's kind of difficult. So there's all of these standardized metrics and

the thing that

People came to realize which actually I mean I say that as though it's like some amazing discovery

It's fairly obvious. If you train your system in a like an unsupervised way on a large corpus of just general English text and

then you take that and

Train that with the data from this benchmark or the data from that benchmark

You can like fine-tune it so you start with something which has like a decent

Understanding of how English works more or less and then you say now I'm going to give you these

Samples for like question answering or I'm going to build a system using that to solve to go for this benchmark

So it's pre trained you start with something. That's like a general-purpose language model and then you from that a

Fine-tuned it to whichever

Actual benchmark or problem you're trying to solve

and this

Can give you better performance than to starting from nothing and training to each of the benchmarks from scratch

make sense

and so

The point of the GPT 2 paper the thing that makes it cool is they said okay if we make a really huge one

What if we?

don't

Fine tune it at all

What if we just make a giant model and then just try and run it on the benchmarks without messing with it?

Without showing it any of their specialized data for that benchmark. Just the raw

general-purpose language model, how does that perform and it turns out

surprisingly well, so this is a

Very very large data set for text

It's about 40 gigabytes

which

Actually doesn't sound like very much but like for text text that's insane, right? It's

somebody said that this was the size of

Google's entire index of the Internet in 98

So like it's yeah, it's a lot of text

and they trained it on that and they ended up with a

1.5 billion parameter model, but which is like a previous state of the art system was 345 million

This is 1.5 billion

So they've just made the thing much much bigger and it performs really well some of their samples that they published quite

captured the public imagination

You could say and now that we've talked a little about the problems that

Neural networks or any language model really?

Has with a long term dependency

we can now realise just how impressive these samples are because when you look at them as a you know,

If you look at them uninitiated, you're like yeah, that's pretty realistic

It seems to like make sense and it's cool. But when you look at it knowing how language models work, it's like

very impressive the the coherence and the

Consistency and the long-range dependencies so we can look at this one that got everybody's attention the unicorns one

right

So they prompted it with in a shocking finding scientists discovered a herd of unicorns

living in a remote previously unexplored valley in the Andes Mountains

Even more surprising to the researchers was the fact that the unicorns spoke perfect English

And from there you then say you go to your language model gbgt, and you say given that we started with this

What's the next word and what's the word after that and so on?

So it goes on the scientist named the population after their distinctive horn of its unicorn

These four horned silver white unicorns were previously unknown to science

We do have a clue here as a human being unicorns for horned doesn't quite make sense

But nonetheless we're going okay

Now after almost two centuries the mystery of what sparked this odd phenomenon is finally solved. Dr

Budetti Jorge Jorge Perez

Jo are G an evolutionary biologist from the University of La Paz

This is impressive because we've mentioned the Andes Mountains in our prompt and so now it's saying okay

This is clearly, you know in a shocking finding. This is a science press release news article

It's seen enough of those because it has every single one that was ever linked to from reddit, right?

So it knows how these go it knows. Okay third paragraph

This is when we talk about the scientist, we interview the scientist, right? Okay

First word of the scientist paragraph, dr. Obviously, right because this is the now we're in the name of the scientist

What name are we going to give?

It needs to be a name

conditioning on the fact that we have the Andes Mountains

So we need to get where we're in South America

The name probably should be Spanish or maybe Portuguese

So we get we get dr. Perez here

And then evolutionary biologist makes sense because we're talking about animals

from the University of La Paz again

This is the first sentence like when you have that first clause that introduces the scientist you always say where they're from

So we say from the University of and then university names tend to be the name of a city

What's the city where we have the Andes Mountains, so we're going to Bolivia lapaz. Perfect

And the thing that's cool about this is it's remembered all of these things that were quite a long time ago several sentences ago

Well, it hasn't remembered them. It's paid attention to them across that distance, which is impressive

But also this is encoding a bunch of understand understanding a bunch of information about the real world

Right all that was given all it knows is statistical relationships between words, but the way that it comes out to us

Is that it knows?

Where the Andes Mountains are what kind of names people in that area have what their cities are what the universities are all of those

Facts about the real world because in order to have a really good language model it turns out you have to kind of implicitly encode

information about the world because

We use language to talk about the world and knowing what's likely to come next

Requires actual real world understanding and that's something that we see in some of the other

Things that they got it to do you can see the real world understanding coming through

Let's keep going

University of a person several companions were exploring the Andes Mountains when they found a small valley with no other animals or humans peres see

We're hanging on to him. Yep. We're referring to him again

but now we've changed it to be just the surname because that's the

format that people use in news articles Peres noticed that the valley had what appeared to be a natural fountain surrounded by two peaks of

Rock and silver snow presently others, then ventured further into the valley a round about here in our article

We should have a quote from the scientist right quote

By the time we reached the top of one peak the water looked blue with some crystals on top and we're talking about this fountain

I guess it's natural fountain. We're referring back to the previous int. It's like everything is

Relying on in contingent on earlier parts of the text while examining there by snipped paragraph while examining these bizarre

Creatures the scientists discovered that the creatures also spoke some fairly regular English know when I read that I like, okay

this is now unusually good because that's the second sentence of the lead right where six paragraphs in and

It knows about this point. I've covered the first sentence of this

initial paragraph

now it's time to talk about this second sentence of the lead even more surprising to the research of us of the fact that they

spoke English and

It completely ignored the speaking English part until it got to the part of the news article where that comes in

You've gone six whole paragraphs

the idea of

Accurately remembering that the unicorn speak perfect

English is like that's very impressive to me and then it goes into its gets a little bit unhinged

Starts talking about it's likely that the only way of knowing for sure if unicorns are indeed

The descendants of a lost alien race is through DNA. That's read it really

Well, it's not actually stuff on reddit. It's stuff linked to from reddit. But yeah, this is this is news articles men

They seem to be able to communicate in English quite well

Which I believe is a sign of evolution or at least a change in social organization said the scientist

That's his evolutionary biology there. Right? Right, right. Yeah, we know here's an evolutionary biologist. So so the the

coherence of this text is

really dependent on its ability to

Condition what it's generating on

Things that it's generated a long time ago

So yeah

So it can generate really nice news articles and it can generate all kinds of text things that it anything that is

Sufficiently well represented in the original data set. So that's GPG - it's a really

Unusually powerful and like versatile

language model that can do all of these different natural language processing

Tasks without actually being trained specifically on those tasks

It's really and that's that's why it's impressive

It's not that it's a it's a brand new architecture or a brand new approach or whatever

It's just when you make these things really huge and give them tremendously large amounts of data

The results are really impressive

In the original data set. So it will it will write you the Lord of the Rings fan fiction

It will write you cake recipes if we're like, there's all kinds of examples of different samples. Here's a recipe for

Some kind of peppermint chocolate cake and it's got a bunch of different