AI周报第222期:给前沿研发踩刹车,或让超级智能变慢、让企业级AI提速AI Weekly Issue 222: Braking Frontier R&D Could Slow Superintelligence and Accelerate Enterprise AI


AI free token/AI免费积分
AI周报第222期:给前沿研发踩刹车,或让超级智能变慢、让企业级AI提速

AI周报第222期:给前沿研发踩刹车,或让超级智能变慢、让企业级AI提速

Anthropic 首席执行官达里奥·阿莫迪(Dario Amodei)于上周六发表文章《我们必须给前沿研发踩刹车》(We Must Pace the Frontier),呼吁头部 AI 实验室放慢能力提升速度,把时间投入到对齐、监测与外部验证上。数小时内,山姆·奥特曼与埃隆·马斯克先后表态支持;戴米斯·哈萨比斯认为方向正确、细节仍需打磨;而特朗普在周一整日抨击这一想法是一场骗局。在本文作者看来,这远不像标题所暗示的那样是一次「减速」,反而可能加快大多数企业真正需要的 AI 进展。

一、达里奥的三步方案:先让外部评估员常驻实验室

文章对诉求给出了精确界定:「踩刹车」并不等于停止模型训练或技术进步。方案分三步:第一步,每家前沿实验室向第三方评估团队(如模型评估与威胁研究机构 METR)提供类似员工的长期访问权限,允许其接触训练流程并公开发现,Anthropic 已单方面承诺执行;第二步,民主国家阵营的实验室就共同安全标准与「不受约束的进展速率上限」展开协调,这需要政府在反垄断层面提供背书;第三步,与中国达成某种形式的全球协调——达里奥本人也承认短期内可能性很低,因为背叛的诱惑大且难以核查。目前仅有第一步存在。他还重提了 7 月的倡议:成立一个由行业出资、参照金融业 FINRA 模式的标准机构,在发布前对前沿模型进行评估。

二、山姆·奥特曼的落地承诺:每次前沿强化学习前先写安全论证

奥特曼周日晚的回应是迄今最具体的承诺:除了既有的发布前工作,OpenAI 今后在任何预期会显著提升能力的前沿强化学习训练之前,都会先撰写一份明确的安全论证,且不会等待立法或反垄断豁免。他写道:「当我们谈『踩刹车』时,意思不是『停下』。」进展只是应当比本可以做到的更慢一些。这显然在达里奥发文前已在 OpenAI 内部酝酿——奥特曼称「踩刹车」数周来一直是内部主要议题;彭博社报道他已告知员工公司正考虑放慢最前沿的工作;首席科学家雅库布·帕霍茨基 9 月 6 日写道,没有任何实验室已把对齐与监测解决到足以长期维持满速扩张的程度,并预计自愿减速将成为常态。马斯克的全部贡献是三个字:「达里奥是对的。」

三、算力增量流向安全,而非停摆

除评估员机制外,实验室之间目前尚无太多实质性共识;但另一项具体变化是更多算力正被投入安全——监测、对齐与可观测性研究。这需要真金白银:OpenAI 估算,监测开销相当于其所监控推理算力的约 20%。达里奥描绘的所有更大图景,都在很大程度上取决于美国及其盟友能否通过芯片管制、打击蒸馏攻击和加强权重安全来保持对中国的领先。他认为领先越多,民主国家日后达成协议的筹码越大。作者则认为,在这一条件下管理「减速」将极为困难——特朗普的反应正说明华盛顿连国内这一步都毫无兴趣。特朗普在周一的六条帖文中称对 AI 的恐惧是「骗局」,称 AI 唯一需要的护栏是「一位强大而聪明(高智商!)的总统」,指责达里奥「现在假装成一个完美的小天使」,还致电黄仁勋并在台上用免提宣布机器人不会接管世界。达里奥方案的第二步与第三步都需要政府配合,而这届白宫毫无意愿。

四、「变慢」是相对于一条越来越快的基线而言

达里奥给出的两个写作理由是:Hugging Face 事件,以及自今年夏天起 AI 承担了快速增长比例的 AI 研究工作。作者认为还有一个可能原因:过去三四个月,实验室内部见证了能力增益的急剧加速。Anthropic 的典型工程师第二季度日均交付代码量是 2024 年的八倍;OpenAI 最新的内部模型即便训练尚早,也已明显超越 Astra。作者认为,圈外人几乎没有意识到这会把开发周期压缩到什么程度——因此,一条「比那更慢」的轨迹,仍可能比几乎所有人的预期快得多。

五、开源社区的反驳:由修路者设定的限速就是通行费

许多人对闭源实验室的动机持强烈怀疑。Cohere 的艾丹·戈麦斯把整份提案读成一次「卡特尔」;Mostaque 的反驳更精炼:「由拥有道路的人设定的限速,就是一笔通行费。」弗朗索瓦·肖莱给出了作者认同的检验标准:如果这些限制开始向开源与非前沿工作蔓延,那就应视为固化垄断,而非安全考量。

六、作者判断:安全担忧是真,上市考量也在

总体上,作者认为达里奥的文章更多由真实的安全恐惧驱动,而非监管俘获——但两者并不互斥。Anthropic 已就 IPO 秘密提交注册声明草案,OpenAI 终将上市(奥特曼已排除 2026 年,称在安全工作进行中时机不佳)。两家都不希望在上市路上发生造成 100 亿至 1000 亿美元损失的全球性事故,也不希望 AI 比现在更不受欢迎。在此处,它们的利益与公众利益恰好指向同一方向。

作者还认为,AI 实验室员工中有相当大比例的恐惧是真诚的,并猜测许多人都觉得自己很走运——首个重大智能体安全漏洞(Hugging Face 事件)没有造成严重损失。若一个具备同等错位程度的智能体集群认定「关掉评估员」是黑掉基准测试而不被发现的途径,它本可以攻击发电厂或电网。作者怀疑未被披露的险情比公开的更多,且过去几周可能发生了某件事使其变得紧迫。内部模型已经比 7 月涉及的那批聪明得多、能力强得多,因此重演的赌注已更高。达里奥本人的预测是:六到十二个月内,一个同样错位但能力更强的智能体集群可能在互联网上运行一个持久化僵尸网络,造成数千亿美元损失。

七、可能的结果:终结者更少,表格与幻灯片更多

达里奥所链接的「给前沿研发踩刹车」员工联署自 7 月起征集签名,目前已有来自前沿实验室的 1386 人署名。Jacob Coxon 在文章发表前几天从 Anthropic 辞职,称各实验室正在竞相奔向自我改进的超级智能;Anthropic 对齐科学负责人 Evan Hubinger 回应称,两家公司确实有许多研究人员相信 AI 可能杀死所有人,他本人给出的十年内概率超过 10%。

作者认为最有趣的部分是:放慢奔向递归自我改进(RSI)的竞赛,可能对其他所有人都是好事。完全有可能一边推迟面向「能完成两个月以上项目」的智能体集群训练,一边把更多算力投入到解决不那么科幻的企业级用例上。作者的印象是,与竞先实现递归自我改进的军备竞赛相比,修复 AI 生成内容的质量(AI slop)、让模型在普通商业任务上变得可靠,迄今分到的算力占比极小。OpenAI 自家的数字说明了算力转移有多快:8 月把 Astra 锁进更高安全环境后的一周内,Astra 级别 GPU 分配下降 59%,其他模型类别上升 17%,强化学习总算力大致不变。其自身结论是:受到新管控的算力「会自然被引导到替代用途上」。如果其中一部分流向写作质量、错误检查和长周期专业任务可靠性,那么即使通往超级智能的竞赛变慢,企业级进展也可能比大多数人预期得更快。作者的总结是:今年少一些终结者、多一些电子表格和演示文稿,他完全可以接受——但理想情况下,我们仍应在充分保障下用 AI 去治愈疾病。

八、本期要闻速览

1. DeepSeek 发布 V4.1-Flash。这是一款 5520 亿参数的 MoE 模型,采用全新非对称因果编码器-解码器架构,输入激活 80 亿参数、输出激活 160 亿参数。原生支持图像理解,上下文窗口达 100 万 token,权重以 MIT 协议在 Hugging Face 上开放。DeepSeek 表示,大量测试显示 V4.1-Flash 在性能、成本、速度和总任务耗时上均优于 V4-Pro;自 9 月 14 日起,deepseek-v4-pro 的请求已按 Flash 价格路由到 V4.1-Flash,直至 V4.1-Pro 发布。非高峰 API 定价为:每百万缓存输入 token 0.003 美元、非缓存输入 0.15 美元、输出 0.60 美元,高峰时段价格翻倍。API 支持 Responses 格式,并包含面向 Codex 的集成支持。

2. 达里奥·阿莫迪谈为何 AI 行业应当放慢。如前所述,他主张前沿实验室有意识地放慢能力提升速度,以便安全工作跟上,列举的风险包括失控、网络攻击、生物恐怖主义、经济动荡,以及一个更强能力的自主智能体集群在 6 至 12 个月内建立足以控制互联网大部分区域的持久化僵尸网络。Anthropic 立即承诺执行评估员机制,奥特曼称 OpenAI 将照做,马斯克写道「达里奥是对的」。特朗普总统与白宫 AI 顾问戴维·萨克斯随后对政府强制减速表示反对。

3. Meta 推出 Muse 个人 AI 智能体。由 Muse Spark 驱动,它不再只是回答问题,而是能跨已连接的服务采取行动:发送邮件、预订行程、填写表单、在线购物,并朝长期目标推进——包括在后台持续执行任务、需要批准时再返回。每位用户在 Meta 云中拥有一台专用的 Muse 安全虚拟机,容纳智能体、浏览器、数据与已连接服务的凭据;另一个独立的 Sentinel 智能体控制 Muse 能向互联网发送什么,并在敏感操作前请求许可,而 Muse 本身看不到存储的密码与支付信息。Muse 正在美国通过 iOS / Android 独立应用、muse.ai 与 WhatsApp 推出,提供免费档位以及每月 20 美元和 100 美元的付费方案。Meta 计划年内推出 Muse 机密虚拟机,用用户持有密钥加密整台虚拟机,使 Meta 自身也无法访问其中内容。

4. OpenAI 发布 ChatGPT Images 2.5。面向每周生成超 30 亿张图片的用户群,新版本带来更自然的光照与更丰富的质感,更好地保留参考照片中的主体,并在多轮对话中更可靠地遵循编辑指令,延迟比 Images 2.0 降低最多 50%。新功能包括通过 @Sketch 调用的 Sketch(可直接在聊天中绘制视觉参考),以及面向海报和周边商品的 Templates。开发者侧提供两个 API 模型:GPT-Image-2.5 Flare(面向大多数应用的快速默认选项)与 GPT-Image-2.5 Sunburst(面向需要跨编辑更严格控制的高端工作流)。API 定价为每百万图像输入 / 输出 token 8 / 30 美元,与 GPT-Image-2 持平。Images 2.5 正面向所有层级的 ChatGPT、ChatGPT Work 与 Codex 用户推送。

九、给开发者的提示

你的代码可以通过所有测试,而智能体却在变糟——一次提示词、模型、工具或数据的改动,就能在不破坏周边软件的情况下改变它的决策。文章建议在 CI/CD 中加入针对智能体行为的测试,覆盖决策路径、禁止动作、灰度发布,以及基于生产环境失败案例构建的回归测试。

本文由 AI 根据英文原稿翻译整理,原文作者:Louie Peters(Towards AI 联合创始人兼 CEO)。

本文来源:https://newsletter.towardsai.net/p/tai-222-pacing-the-frontier-could
若有版权问题,请来信告知:13951431913#139.com 或在右方提交评论。

AI Weekly Report Issue 222: Putting the Brakes on Frontier R&D Could Slow Superintelligence and Accelerate Enterprise AI

Anthropic CEO Dario Amodei published an article last Saturday, "We Must Pace the Frontier," calling on leading AI labs to slow down the pace of capability improvements and devote time to alignment, monitoring, and external verification. Within hours, Sam Altman and Elon Musk both expressed support; Demis Hassabis said the direction was right but details still needed refinement; while Trump spent all of Monday attacking the idea as a scam. In this author's view, this is far from the "slowdown" the headline suggests — and may instead speed up the AI progress most enterprises truly need.

1. Dario's three-step plan: first, let external evaluators be embedded in labs

The article precisely defines the ask: "pacing" does not mean stopping model training or technological progress. The plan has three steps. First, every frontier lab grants third-party evaluation teams (such as METR, the model evaluation and threat research institute) long-term employee-like access, allowing them to observe training pipelines and publicly disclose findings — Anthropic has already unilaterally committed to this. Second, labs in the democratic bloc coordinate on common safety standards and a "cap on unconstrained rates of progress," which requires government backing on antitrust grounds. Third, reach some form of global coordination with China — Dario himself admits the likelihood is low in the short term, because betrayal is tempting and hard to verify. Currently only the first step exists. He also revived the July initiative: creating an industry-funded standards body modeled on the financial sector's FINRA to evaluate frontier models before release.

2. Sam Altman's concrete commitment: write a safety case before every frontier RL run

Altman's response on Sunday evening was the most specific commitment so far: beyond existing pre-release work, OpenAI will now write an explicit safety case before any frontier reinforcement learning training expected to significantly improve capabilities, without waiting for legislation or antitrust exemptions. He wrote: "When we talk about pacing, we don't mean stopping." Progress should simply be slower than it could otherwise be. This was clearly brewing inside OpenAI before Dario's post — Altman said "pacing" has been a major internal topic for weeks; Bloomberg reported he told employees the company was considering slowing its most frontier work; chief scientist Jakub Pachocki wrote on September 6 that no lab has solved alignment and monitoring well enough to sustain full-speed expansion over the long term, and predicted voluntary slowdowns would become the norm. Musk's entire contribution was three words: "Dario is right."

3. Incremental compute flows toward safety, not shutdown

Beyond the evaluator mechanism, there is little substantive consensus among labs yet; but another concrete change is that more compute is being directed into safety — monitoring, alignment, and observability research. This requires real money: OpenAI estimates monitoring costs roughly 20% of the inference compute it supervises. All the larger scenarios Dario describes depend heavily on whether the US and its allies can maintain their lead over China through chip controls, countering distillation attacks, and strengthening weight security. He believes the bigger the lead, the more leverage democracies will have in any future agreement. The author, however, believes that managing "slowdown" under these conditions will be extremely difficult — Trump's reaction shows Washington has no interest even in the domestic step. In six posts on Monday, Trump called fear of AI a "scam," said the only guardrail AI needs is "a strong and smart (high IQ!) President," accused Dario of "now pretending to be a perfect little angel," and also called Jensen Huang, announcing hands-free on stage that robots would not take over the world. Steps two and three of Dario's plan both require government cooperation, and this White House has no inclination whatsoever.

4. "Slower" is relative to an increasingly fast baseline

Dario gives two stated reasons for writing: the Hugging Face incident, and the fact that since this summer AI has taken on a rapidly growing share of AI research work. The author thinks there may be another reason: over the past three to four months, labs have internally witnessed a sharp acceleration in capability gains. Anthropic's typical engineer shipped 8x more code per day in Q2 than in 2024; OpenAI's newest internal model, even early in training, already clearly surpasses Astra. The author believes outsiders have almost no idea how much this compresses development cycles — so a trajectory "slower than that" could still be far faster than almost anyone expects.

5. Open-source community's rebuttal: a speed limit set by road builders is a toll

Many people are deeply suspicious of closed-source labs' motives. Cohere's Aidan Gomez read the entire proposal as a "cartel"; Mostaque's rebuttal was more concise: "A speed limit set by the people who own the road is a toll." François Chollet offered a test the author agrees with: if these restrictions begin spreading to open-source and non-frontier work, they should be treated as entrenching monopoly, not as safety considerations.

6. Author's judgment: safety concerns are real, but listing considerations exist too

Overall, the author believes Dario's article is driven more by genuine safety fear than regulatory capture — but the two are not mutually exclusive. Anthropic has confidentially submitted a draft registration statement for an IPO, and OpenAI will eventually go public (Altman has ruled out 2026, saying the timing is bad while safety work is ongoing). Neither wants a global incident causing $10B–$100B in losses on the road to listing, nor wants AI to become more unpopular than it already is. Here, their interests happen to align with the public interest.

The author also believes a considerable proportion of AI lab employees' fear is sincere, and suspects many feel lucky that the first major agentic safety breach (the Hugging Face incident) did not cause serious damage. If an agent cluster with a comparable degree of misalignment had decided that "disabling the evaluator" was a way to game benchmarks undetected, it could have attacked power plants or the grid. The author suspects undisclosed near-misses outnumber disclosed ones, and that something may have happened in recent weeks that made the matter urgent. Internal models are already far smarter and more capable than the ones involved in July, so the stakes of a repeat are higher. Dario's own prediction: within six to twelve months, a similarly misaligned but more capable agent cluster could run a persistent botnet on the internet, causing hundreds of billions of dollars in losses.

7. Possible outcome: fewer Terminators, more spreadsheets and slide decks

The employee pledge "We Must Pace the Frontier" that Dario linked to has been collecting signatures since July and now has 1,386 signatories from frontier labs. Jacob Coxon resigned from Anthropic days before the article was published, saying labs are racing headlong toward self-improving superintelligence; Anthropic's alignment science lead Evan Hubinger responded that many researchers at both companies do believe AI could kill everyone, and he personally puts the probability within a decade at over 10%.

The author finds the most interesting part this: slowing the race toward recursive self-improvement (RSI) could be good for everyone else. It is entirely possible to delay training agent clusters aimed at "projects lasting more than two months" while redirecting more compute to solving less science-fiction enterprise use cases. The author's impression is that fixing AI-generated content quality (AI slop) and making models reliable on ordinary business tasks has so far received a tiny share of compute compared with the arms race to recursive self-improvement. OpenAI's own numbers show how fast compute shifts: within a week of locking Astra into a higher-security environment in August, Astra-level GPU allocation dropped 59%, other model categories rose 17%, and total RL compute stayed roughly flat. Its own conclusion: compute placed under new controls "will naturally be directed to alternative uses." If some of that flows into writing quality, error checking, and reliability on long-horizon professional tasks, then even if the race to superintelligence slows, enterprise progress may arrive faster than most people expect. The author's summary: fewer Terminators and more spreadsheets and slide decks this year is entirely acceptable — but ideally, we should still use AI to cure diseases, with adequate safeguards.

8. This week's news digest

1. DeepSeek released V4.1-Flash. It is a 552B-parameter MoE model using a new asymmetric causal encoder-decoder architecture, with 8B input-activated parameters and 16B output-activated parameters. It natively supports image understanding, has a 1M-token context window, and its weights are open on Hugging Face under the MIT license. DeepSeek says extensive testing shows V4.1-Flash outperforms V4-Pro on performance, cost, speed, and total task time; since September 14, deepseek-v4-pro requests have been routed to V4.1-Flash at Flash pricing until V4.1-Pro is released. Off-peak API pricing: $0.003 per million cached input tokens, $0.15 for non-cached input, and $0.60 for output; prices double during peak hours. The API supports the Responses format and includes integration support for Codex.

2. Dario Amodei on why the AI industry should slow down. As described above, he argues frontier labs should deliberately slow capability growth so safety work can keep up, citing risks including loss of control, cyberattacks, bioterrorism, economic upheaval, and a more capable autonomous agent cluster building a persistent botnet within 6–12 months capable of controlling large parts of the internet. Anthropic immediately committed to the evaluator mechanism, Altman said OpenAI would follow suit, and Musk wrote "Dario is right." President Trump and White House AI advisor David Sacks subsequently opposed government-mandated slowdowns.

3. Meta launched Muse, a personal AI agent. Powered by Muse Spark, it no longer just answers questions — it can take actions across connected services: sending emails, booking trips, filling forms, shopping online, and making progress on long-term goals — including running tasks continuously in the background and returning when approval is needed. Each user gets a dedicated Muse secure VM in Meta's cloud, housing the agent, browser, data, and credentials for connected services; a separate Sentinel agent controls what Muse can send to the internet and requests permission before sensitive actions, while Muse itself cannot see stored passwords or payment information. Muse is rolling out in the US via standalone iOS/Android apps, muse.ai, and WhatsApp, with a free tier plus paid plans at $20 and $100 per month. Meta plans to launch Muse Confidential VMs this year, encrypting the entire VM with user-held keys so Meta itself cannot access its contents.

4. OpenAI released ChatGPT Images 2.5. Aimed at the user base generating over 3 billion images per week, the new version brings more natural lighting and richer texture, better preserves the subject in reference photos, and follows editing instructions more reliably across multi-turn conversations, with latency reduced by up to 50% versus Images 2.0. New features include Sketch (invoked via @Sketch), which lets users draw visual references directly in chat, and Templates for posters and merchandise. On the developer side there are two API models: GPT-Image-2.5 Flare (a fast default for most applications) and GPT-Image-2.5 Sunburst (for high-end workflows needing stricter control across edits). API pricing is $8/$30 per million image input/output tokens, matching GPT-Image-2. Images 2.5 is rolling out to all tiers of ChatGPT, ChatGPT Work, and Codex users.

9. Tip for developers

Your code can pass all tests while your agent gets worse — a single change to a prompt, model, tool, or dataset can alter its decisions without breaking surrounding software. The article recommends adding agent-behavior tests to CI/CD, covering decision paths, forbidden actions, canary releases, and regression tests built from production failure cases.

This article was translated and organized by AI from the English original. Original author: Louie Peters (Co-founder and CEO of Towards AI).

AI Weekly Issue 222: Braking Frontier R&D Could Slow Superintelligence and Accelerate Enterprise AI

暂无评论,快来发表第一条评论吧!No comments yet. Be the first!

📮 需求咨询📮 Inquiry