Okay so first of all China has a lot of disadvantages in competing with the U.S.

首先，中國在與美國競爭時有很多劣勢。

Number one is the fact that they don't get access to all the hardware that we have access to here.

第一，他們無法使用我們這裡的所有硬件。

So they're kind of working with lower-end GPUs than us.

是以，他們使用的 GPU 比我們低端。

It's almost like working with the previous generation GPUs crappily.

這幾乎就像是在蹩腳地使用上一代 GPU。

So and the fact that the bigger models tend to be more smarter naturally puts them at a disadvantage.

是以，更大的機型往往更聰明，這自然使它們處於劣勢。

But the flip side of this is that necessity is the mother of invention.

但反過來說，必要是發明之母。

Because they had to go figure out workarounds, they actually ended up building something a lot more efficient.

因為他們不得不去想變通的辦法，所以實際上他們最終建造的東西效率要高得多。

It's like saying, hey look you guys really got to get a top-notch model and I'm not going to give you resources.

這就好比說，嘿，你們真的需要一個頂級模特，但我不會給你們資源。

I mean figure out something, right?

我的意思是想辦法，對吧？

Unless it's impossible, unless it's mathematically possible to prove that it's impossible to do so, you can always try to like come up with something more But that is likely to make them come up with a more efficient solution than America.

除非這是不可能的，除非在數學上可以證明這是不可能的，否則你總可以試著想出一些更有可能讓他們想出比美國更有效解決方案的辦法。

And of course they've open sourced it so we can still adopt something like that here.

當然，他們已經將其開源，所以我們仍然可以在這裡採用類似的東西。

But that kind of talent they're building to do that will become an edge for them over time, right?

但他們正在培養的這種人才，隨著時間的推移，會成為他們的優勢，對嗎？

The leading open source model in America is Meta's Lama family.

美國領先的開源模式是 Meta 的喇嘛家族。

It's really good.

真的很不錯。

It's kind of like a model that you can run on a computer.

它有點像一個可以在電腦上運行的模型。

But even though it got pretty close to GPT-4 and Asana at the time of its release, the model that was closest in quality was a giant 405B, not the 70B that you could run on your computer.

不過，儘管它在發佈時已經非常接近 GPT-4 和 Asana，但在品質上最接近的型號是巨大的 405B，而不是可以在電腦上運行的 70B。

And so that was still not a small, cheap, fast, efficient, open source model that rivaled the most powerful closed models from Okinawa and Tropic.

是以，這仍然不是一個小型、廉價、快速、高效的開源模型，無法與沖繩和熱帶最強大的封閉模型相媲美。

Nothing from America.

沒有來自美國的東西。

Nothing from Mistral either.

Mistral 也沒有任何消息。

And then these guys come out with like a crazy model that's like 10x cheaper in API pricing than GPT-40 and 15x cheaper than Sonnet, I believe.

然後，這些傢伙又推出了一個瘋狂的型號，其 API 價格比 GPT-40 便宜 10 倍，比 Sonnet 便宜 15 倍。

Really fast, 60 tokens per second.

速度非常快，每秒 60 塊令牌。

And pretty much equal or better in some benchmarks than worse than some others, but like roughly in that ballpark of 4OS quality.

在某些基準測試中，它的性能相當或更好，而在其他一些基準測試中，它的性能則較差，但大致與 4OS 的品質相當。

And they did it all with like approximately just 2048H800 GPUs, which is actually equivalent to like somewhere around 1500 or 1500H100 GPUs.

他們使用大約 2048H800 GPU 完成了這一切，這實際上相當於大約 1500 或 1500H100 GPU。

That's like 20 to 30x lower than the amount of GPUs that GPT-4 is usually trained on.

這比 GPT-4 通常使用的 GPU 數量要低 20 到 30 倍。

And they did roughly $5 million in total compute budget.

他們的計算預算總額約為 500 萬美元。

They did it with so little money and such an amazing model.

他們用如此少的錢和如此出色的模型做到了這一點。

Gave it away for free.

免費贈送。

Wrote a technical paper.

撰寫了一篇技術論文。

And definitely it makes us all question like, okay, like if we have the equivalent of DOGE for like model training, this is an example of that, right?

這肯定會讓我們產生這樣的疑問：好吧，如果我們有等同於 DOGE 的模型訓練方法，這不就是一個例子嗎？

Right.

對

Efficiency is what you're getting at.

效率就是你要表達的意思。

So fraction of the price, fraction of the time, dumbed down GPUs essentially.

是以，只需幾分之一的價格，幾分之一的時間，基本上就是傻瓜式 GPU。

What was your surprise when you understood what they had done?

當你明白他們的所作所為時，你有什麼驚訝？

So my surprise was that when I actually went through the technical paper, the amount of clever solutions they came up with.

是以，讓我感到驚訝的是，當我真正翻閱技術文件時，他們提出了大量巧妙的解決方案。

First of all, they trained a experts model.

首先，他們訓練了一個專家模型。

It's not that easy to train.

訓練沒那麼容易

There's a lot of like, the main reason people find it difficult to catch up with OpenAI, especially the MOE architecture, is that there's a lot of irregular loss spikes.

人們發現 OpenAI（尤其是 MOE 架構）難以追趕的主要原因是，有很多不規則的損失峰值。

The numerics are not stable.

數值不穩定。

So often like you've got to restart the training checkpoint again.

所以，你經常要重新開始訓練檢查點。

And a lot of infrastructure needs to be built for that.

為此需要建設大量的基礎設施。

And they came up with very clever solutions to balance that without adding additional hacks.

他們想出了非常巧妙的辦法，在不增加額外黑客的情況下實現了平衡。

And they also figured out 8-bit training, at least for some of the numerics.

他們還想出了 8 位訓練方法，至少對某些數字進行了訓練。

And they cleverly figured out which has to be in higher precision, which has to be in lower precision.

他們巧妙地算出了哪個精度要高，哪個精度要低。

And to my knowledge, I think floating point 8 training is not that well understood.

據我所知，我認為浮點 8 訓練還沒有得到很好的理解。

Most of the training in America is still running.

美國的大部分培訓仍在進行。

And maybe OpenAI and some people are trying to explore that, but it's pretty difficult to get it right.

也許 OpenAI 和一些人正在嘗試探索這一點，但要做到這一點非常困難。

So because necessities, I'm going to mention, because they don't have that much memory, that many GPUs, they figured out a lot of numerical stability stuff that makes their work.

是以，我想說的是，因為他們沒有那麼多內存和 GPU，所以他們想出了很多數值穩定性的方法，使他們的工作變得更加穩定。

And they claimed in the paper that the majority of the training was stable, which means they can always rerun those training runs again on more data or better data.

他們在論文中聲稱，大部分訓練是穩定的，這意味著他們隨時可以在更多數據或更好的數據上重新運行這些訓練運行。

And then it only trained for 60 days.

然後只培訓 60 天。

So it's pretty amazing.

所以，這非常了不起。

It's nice to say you were surprised.

很高興你能說你很驚訝。

So I was definitely surprised.

所以，我絕對是大吃一驚。

Usually the wisdom, or I would say the myth, is that Chinese are just good at copying.

通常，人們認為中國人只是善於模仿，或者說是神話。

So if we stop writing research papers in America, if we stop describing the details of our infrastructure, our architecture, and stop open sourcing, they're not going to be able to catch up.

是以，如果我們停止在美國撰寫研究論文，如果我們停止描述我們的基礎設施和架構的細節，停止開放源代碼，他們將無法迎頭趕上。

But the reality is, some of the details in DeepSea 3 are so good that I wouldn't be surprised if Meta took a look at it and incorporated some of that in Llamas 4.

但實際情況是，《深海 3》中的一些細節非常出色，如果 Meta 借鏡了《深海 3》，並在《傲骨賢妻 4》中加入一些細節，我也不會感到驚訝。

I wouldn't necessarily say copy.

我不一定說是抄襲。

It's all sharing science, engineering.

這都是科學和工程學的共享。

But the point is, it's changing.

但問題是，情況正在發生變化。

It's not like China is copycat.

中國又不是模仿者。

They're also innovating.

他們還在不斷創新。

We don't know exactly the data that it was trained on, even though it's open source.

儘管它是開源的，但我們並不清楚其訓練所依據的數據。

We know some of the ways and things it was trained on, but not everything.

我們知道它的一些訓練方式和訓練內容，但不是全部。

And there's this idea that it was trained on public chat GPT outputs, which would mean it just was copied.

還有一種說法是，它是在公開哈拉 GPT 輸出上訓練出來的，這意味著它只是被複制了。

But you're saying it goes beyond that.

但你說的不止於此。

There's real innovation.

有真正的創新。

Yeah, look, they've trained it on 14.8 trillion tokens.

是啊，你看，他們已經在 14.8 萬億個代幣上進行了訓練。

The internet has so much chat GPT.

互聯網上有很多哈拉 GPT。

If you actually go to any LinkedIn post or X post now, most of the comments are written by AI.

如果你現在真的去看 LinkedIn 上的帖子或 X 帖子，大部分評論都是人工智能寫的。

You can just see it.

你可以看到它。

People are just trying to write.

人們只是在努力寫作。

Even with an X, there's a Grok tweet enhancer.

即使是 X，也有一個 Grok 鳴叫增強器。

Or in LinkedIn, there's an AI enhancer.

或者在 LinkedIn 中，有一個人工智能增強器。

Or in Google Docs and Word, there are AI tools to rewrite your stuff.

或者在谷歌文檔和 Word 中，也有人工智能工具來改寫你的內容。

So if you do something there and copy-paste it somewhere on the internet, it's naturally going to have some elements of a chat GPT-like training.

是以，如果你在那裡做了一些事情，然後複製粘貼到互聯網上的某個地方，自然會有一些類似哈拉 GPT 培訓的元素。

And there's a lot of people who don't even bother to strip away that I'm a language model part.

有很多人甚至懶得剝去我是語言模型的部分。

So they just paste it somewhere.

所以他們就把它粘貼在某個地方。

It's very difficult to control for this.

這一點很難控制。

I think XAI has spoken about this too.

我想 XAI 也談到過這個問題。

So I wouldn't disregard their technical accomplishment just because for some prompts, like, who are you?

是以，我不會因為某些提示，比如你是誰？

Or which model are you in response to that?

或者說，你是哪種型號？

It doesn't even matter in my opinion.

在我看來，這根本不重要。

For a long time, we thought, I don't know if you agreed with us, China was behind in AI.

在很長一段時間裡，我們都認為--我不知道你是否同意--中國在人工智能方面落後了。

What does this do to that race?

這對比賽有什麼影響？

Can we say that China is catching up or hasn't caught up?

我們能說中國正在追趕還是沒有追趕嗎？

I mean, if we say the meta is catching up to open AI and entropic, if you make that claim, then the same claim can be made for China catching up to America.

我的意思是，如果我們說元語言正在追趕開放式人工智能和熵，如果你這麼說，那麼同樣的說法也可以適用於中國追趕美國。

A lot of papers from China that have tried to replicate O1.

中國有很多論文試圖複製 O1。

In fact, I saw more papers from China after O1 announcement that tried to replicate it than from America.

事實上，在 O1 公佈後，我看到中國試圖複製 O1 的論文比美國還多。

And the amount of compute DeepSeq has access to is roughly similar to what PhD students in the US have access to.

DeepSeq 所能使用的計算量與美國博士生所能使用的計算量大致相當。

This is not meant to criticize others, even for ourselves.

這並不是要責備別人，即使是責備我們自己。

For Perplexity, we decided not to train models because we thought it's a very expensive thing.

對於 Perplexity，我們決定不訓練模型，因為我們認為這是一件非常昂貴的事情。

And we thought there's no way to catch up with the rest.

而我們卻認為沒有辦法趕上其他人。

Will you incorporate DeepSeq into Perplexity?

你們會將 DeepSeq 納入 Perplexity 嗎？

We already are beginning to use it.

我們已經開始使用它。

I assume they have an API and they have open source of AI so we can host it ourselves too.

我想他們應該有一個應用程序接口，而且他們還開放了人工智能的源代碼，這樣我們也可以自己託管。

And it's good to try to start using that because it actually allows us to do a lot of the things at lower cost.

嘗試開始使用這種方法是件好事，因為它實際上可以讓我們以更低的成本完成很多事情。

But what I'm kind of thinking is beyond that, it's just like, okay, these guys actually could train such a great model with a good team.

但我的想法是，除了這一點之外，這些人其實還可以和一支優秀的團隊一起訓練出如此出色的模型。

And there's no excuse anymore for companies in the US, including ourselves, to not try to do something like that.

包括我們自己在內的美國公司再也沒有藉口不嘗試這樣做了。

You hear a lot in public from a lot of thought leaders and generative AI, on the research side, on the entrepreneurial side.

在公開場合，你會聽到很多思想領袖和生成式人工智能的觀點，無論是在研究方面，還是在創業方面。

Elon Musk and others say that China can't catch up.

埃隆-馬斯克等人說，中國趕不上。

The stakes are too big, the geopolitical stakes.

地緣政治的賭注太大了。

Whoever dominates AI is going to dominate the economy, dominate the world.

誰主宰了人工智能，誰就能主宰經濟，主宰世界。

It's been talked about in those massive terms.

我們一直在談論這些大型活動。

Are you worried about what China proved it was able to do?

你擔心中國證明了自己的能力嗎？

Firstly, I don't know if Elon ever said China can't catch up.

首先，我不知道埃隆是否說過中國趕不上。

I'm not aware of that.

我不知道。

Just the threat of China.

只是中國的威脅。

He's only identified the threat of letting China.

他只指出了中國的威脅。

Sam Altman has said similar things.

薩姆-奧特曼也說過類似的話。

We can't let China win.

我們不能讓中國贏。

I think you've got to decouple what someone like Sam says to what is in his self interest.

我認為，你必須把薩姆這樣的人所說的話與他自身的利益脫鉤。

My point is, whatever you did to not let them catch up didn't even matter.

我想說的是，不管你做了什麼，不讓他們追上來都沒有關係。

They ended up catching up anyway.

結果他們還是趕上了。

Necessity is the mother of invention, like you said.

正如你所說，需要是發明之母。

Exactly.

沒錯。

What's more dangerous than trying to do all the things to not let them catch up?

還有什麼比竭盡全力不讓他們趕上更危險的呢？

What's more dangerous is they have the best open source model, and all the American developers are building on that.

更危險的是，他們擁有最好的開源模式，所有的美國開發者都在此基礎上進行開發。

That's more dangerous, because then they get to own the mindshare, the ecosystem, the entire American AI ecosystem.

這就更危險了，因為這樣一來，他們就能掌握整個美國人工智能生態系統的思想份額和生態系統。

In general, it's known that once open source is caught up or improved over closed source software, all developers migrate to that.

一般來說，眾所周知，一旦開放源碼軟件趕上或改進了封閉源碼軟件，所有開發人員都會遷移到開放源碼軟件。

It's historically known.

這在歷史上是眾所周知的。

When LLAMA was being built and becoming more widely used, there was this question, should we trust Zuckerberg?

當 LLAMA 建立起來並被越來越廣泛地使用時，人們不禁要問：我們應該相信扎克伯格嗎？

But now the question is, should we trust China?

但現在的問題是，我們應該相信中國嗎？

Should we trust open source?

我們應該信任開放源代碼嗎？

It's not about who is in Zuckerberg.

這與扎克伯格的身份無關。

Does it matter then if it's Chinese, if it's open source?

那麼，它是否是中國的，是否是開源的，這重要嗎？

It doesn't matter in the sense that you still have full control.

這並不重要，因為你仍然可以完全控制。

You run it as your own weights on your own computer.

您可以在自己的電腦上將其作為自己的權重運行。

You are in charge of the model.

你負責模型。

It's not a great look for our own talent to rely on software built by others.

依賴他人開發的軟件對我們自己的人才來說並不光彩。

There's always a point where open source can stop being open source too.

開放源代碼也有停止開放源代碼的時候。

The licenses are very favorable today, but over time, they can always change the license.

今天的許可證非常有利，但隨著時間的推移，他們可以隨時更改許可證。

It's important that we actually have people here in America building, and that's why Meta is so important.

重要的是，我們要讓美國人民真正參與到建設中來，這也是 Meta 如此重要的原因。

I still think Meta will build a better model than DeepSecret 3 and open source, and what they call LLAMA 4 or 3.something.

我仍然認為，Meta 將建立一個比 DeepSecret 3 和開源，以及他們所謂的 LLAMA 4 或 3.something 更好的模型。

It doesn't matter.

沒關係。

What is more key is that we don't try to focus all our energy on banning them, stopping them, and just try to out-compete and win.

更關鍵的是，我們不要試圖把所有精力都放在禁止它們、阻止它們上，而要努力超越它們，贏得勝利。

That's just the American way of doing things.

這就是美國人的做事方式。

Just be better.

做得更好

It feels like we hear a lot more about these Chinese companies who are developing in a similar way, a lot more efficiently, a lot more cost-effectively, right?

我們似乎聽到了更多關於這些中國公司的報道，它們以類似的方式發展，效率更高，成本效益更好，對嗎？

Again, it's hard to fake scarcity, right?

還是那句話，稀缺是很難偽造的，不是嗎？

If you raise $10 billion and you are decided to spend 80% of it on a computer cluster, it's hard for you to come up with the exact same solution that $5 million would do, and there's no need to berate those who are putting more money.

如果你籌集了 100 億美元，卻決定把其中的 80% 用在計算機集群上，那麼你很難想出與 500 萬美元完全相同的解決方案，也就沒有必要斥責那些投入更多資金的人。

They're trying to do it as fast as they can.

他們想盡快完成任務。

When we say open source, there are so many different versions.

當我們說開放源代碼時，有許多不同的版本。

Some people criticize Meta for not publishing everything, and even DeepSpeak itself isn't totally transparent.

有人責備 Meta 沒有公佈所有資訊，甚至 DeepSpeak 本身也不是完全透明的。

Sure.

當然。

You can go to the limits of open source and say, I should exactly be able to replicate your training run, but first of all, how many people even have the resources to do that?

你可以挑戰開源的極限，說我應該可以完全複製你的訓練運行，但首先，有多少人有這樣的資源？

I think the amount of detail they've shared in the technical report, actually Meta did that too, by the way.

我認為他們在技術報告中分享了大量細節，事實上，Meta 也是這麼做的。

Meta's Lama 3.3 technical report is incredibly detailed and very great for science.

Meta 的喇嘛 3.3 技術報告非常詳細，對科學非常有幫助。

The amount of detailed data these people are sharing is already a lot more than what the other companies are doing right now.

這些人共享的詳細數據量已經遠遠超過了其他公司現在所做的。

When you think about how much it costs DeepSpeak to do this, less than $6 million, think about what OpenAI has spent to develop GPT models.

想想 DeepSpeak 做這件事的成本（不到 600 萬美元），再想想 OpenAI 開發 GPT 模型的成本。

What does that mean for the closed source model, ecosystem trajectory, momentum?

這對閉源模式、生態系統軌跡和發展勢頭意味著什麼？

What does it mean for OpenAI?

這對 OpenAI 意味著什麼？

It's very clear that we'll have an open source version of 4.0, or even better than that, and much cheaper than that, open source, completely in this year.

很明顯，我們將在今年完全推出 4.0 的開源版本，甚至比它更好，比它更便宜的開源版本。

Made by OpenAI?

由 OpenAI 製造？

Probably not.

可能不會。

Most likely not.

很可能不會。

I don't think they care if it's not made by them.

我認為，如果不是他們生產的，他們也不在乎。

I think they've already moved to a new paradigm called the O1 family of models.

我認為他們已經轉向了一種新的模式，即 O1 系列機型。

Ilya Sutskiy came and said, pre-training is a wall.

伊利亞-蘇茨基來了，他說，訓練前是一堵牆。

He didn't exactly use the word, but he clearly said the age of pre-training is a wall.

他沒有確切地使用這個詞，但他明確地說，訓練前的年齡是一堵牆。

Many people have said that.

很多人都這麼說過。

That doesn't mean scaling is a wall.

這並不意味著縮放就是一堵牆。

I think we're scaling on different dimensions now.

我想，我們現在是在不同的維度上進行縮放。

The amount of time the model spends thinking at this time, reinforcement learning, trying to make the model, if it doesn't know what to do for a new prompt, it'll go and reason and collect data and interact with the world, use a bunch of tools.

如果模型對新的提示不知道該怎麼做，它就會去推理、收集數據並與世界互動，使用一堆工具。

I think that's where things are headed.

我認為這就是事情的發展方向。

I feel like OpenAI is more focused on that right now.

我覺得 OpenAI 現在更專注於此。

Instead of just a bigger, better model of reasoning capacity.

而不僅僅是一個更大、更好的推理能力模型。

But didn't you say that DeepSeek is likely to turn their attention to reasoning?

但你不是說 DeepSeek 可能會把注意力轉向推理嗎？

A hundred percent.

百分之百。

I think they will.

我想他們會的。

That's why I'm pretty excited about what they'll produce next.

這就是為什麼我對他們的下一部作品相當期待。

I guess my question is, what's OpenAI's moat now?

我想我的問題是，OpenAI 現在的護城河是什麼？

I still think that no one else has produced a system similar to the O1 yet, exactly.

我仍然認為，確切地說，還沒有人生產出與 O1 類似的系統。

I know that there's debates about whether O1 is actually worth it.

我知道，關於 O1 是否真的值得的爭論一直存在。

Maybe a few prompts, it's really better.

也許再加上一些提示，效果會更好。

But most of the time, it's not producing any differentiated output from Sonnet.

但在大多數情況下，它與 Sonnet 沒有任何區別。

But at least the results they showed in O3 where they had competitive coding performance, almost like an AI software engineer level.

但至少，他們在 O3 中表現出的編碼性能很有競爭力，幾乎達到了人工智能軟件工程師的水準。

Isn't it just a matter of time before the internet is filled with reasoning data?