Google got obliterated by Microsoft's blitzkrieg attack in the great AI war of 2023.

GPT-4 captured the zeitgeist of the artificial intelligence age we just entered.

And things got so bad for Google that people unironically started using Bing.

But the war is just getting started and just yesterday, Google unleashed its highly anticipated Gemini model that beats GPT-4 on nearly every benchmark.

It is December 7th 2023, and you are watching the Code Report.

Gemini first became known to the public earlier this year at Google IO when Sundar explained it like this.

You've been applying AI; to make AI; rigorously tested; AI; AI.

Gemini is a multimodal large language model that will replace LaMDA and PaLM2 like GPT-4.

It's multimodal which means it's not only trained on text but also sound, images and video.

Google's demo is absolutely insane.

It can recognize what's going on in a video feed and respond in real time.

Like this guy draws a duck, then the AI tells him it's a duck.

It is a duck.

Like holy fuck, and it can do that in multiple languages.

鴨子

What's really crazy though is that it can keep track of things in an ongoing video feed.

Like it plays the game of find the ball under the cup and even after the cups are scrambled up, it still knows where the ball is.

And it can even do connect the dots, which makes my five-year-old obsolete.

It also does multimodal outputs like it can generate images on the fly like Stable Diffusion and can even generate music based on a prompt.

And not just text to audio but image to audio.

How about some 80s hair metal?

It's an anything-to-anything model.

It's also good at logic and spatial reasoning.

Using these two pictures, it's able to tell you which car will go faster based on the aerodynamics of the vehicle.

In the future, a civil engineer will be able to just take a picture of some land, then the AI can instantly generate some blueprints for a bridge.

So software engineers aren't the only type of engineers becoming obsolete.

Although I do of course have some more bad news for programmers.

Google also unveiled AlphaCode 2, which performs better than 90% of competitive programmers.

And we're talking about programmer solving highly complex abstract problems like you might find on Codeforces competitions.

Like any good programmer, AlphaCode 2 can break down problems into smaller problems using techniques like dynamic programming.

Now all these demos look really amazing at first glance, but is this all just a marketing sleight of hand from Google?

Well, currently, Gemini comes in three sizes: tall, grande and venti.

The smallest version is designed to be embedded on devices like Android phones.

While the Pro version is your more general purpose model.

While Ultra is like the Magnum XL of the Gemini family and the one that's blowing everybody's minds.

If you're in the United States, you can actually use Gemini right now in the Bard chatbot.

However, it's using Gemini Pro, the midrange version.

Bard is way better than it was six months ago and it's still extremely fast,

but after using it for a few minutes, it's pretty obvious that it's not quite as good as GPT-4 Pro.

But GPT-4 is nervous about Gemini Ultra.

When I asked about it, it started throwing mad shade at itself and then before it finished, Sam Altman pulled a plug, giving me this network error.

When it comes to benchmarks, Gemini Pro underforms GPT-4 in most situations, but Gemini Ultra outperforms it on almost every single category.

Most notably, it's the first model ever to outperform human experts on massive multitask language understanding, which is typically a multiple-choice test over a wide array of subjects.

Kind of like the SATs but for AI.

What's hella surprising though is that Gemini Ultra underperforms GPT-4 on the HellaWwag benchmark,

it's designed to evaluate common sense natural language by having the AI finish a sentence that's often vague and ambiguous.

For example, a man watches a Fireship video and afterwards feels blank.

It's a job that's really easy for humans to do and a very important benchmark, because When an AI can't do this well, it doesn't feel very humanlike.

In GPT-4, I can write a vague prompt filled with typos, and somehow it almost always seems to know what I'm talking about.

The fact that GPT-4 is doing so much better on HellaSwag is hella concerning to say the least.

But another interesting thing to know from the technical paper is how they train this beast.

They use their newly unveiled version 5 Tensor Processing Units, which are deployed in SuperPODs of 4096 chips.

Each SuperPOD has a dedicated optical switch which allows data to transfer quickly between the pods to train in parallel.

Then they can dynamically reconfigure into 3D tours topologies.

In other words, they can shape shift into donuts to reduce the latency between ships.

And the scale of Gemini Ultra is so large that they had to communicate between multiple data centers.

The paper also describes the training data set which basically includes everything you can find on the internet, including web pages and YouTube videos as well as scientific papers and books.

They filter for quality, then use reinforcement learning through human feedback to fine tune the quality and avoid hallucinations.

Overall, Gemini looks amazing on paper but prepare to be disappointed.

The Nano and Pro models will be available on Google Cloud on December 13th,

but the Gemini Ultra Pro Max won't be available until next year until additional safety tests are done and it reaches 100% on the Hella woke benchmark.

This has been the Code Report.

Thanks for watching and I will see you in the next one.