消息称 OpenAI、Anthropic 正调查数万起 AI 安全事件Report: OpenAI and Anthropic Investigating Tens of Thousands of AI Safety Incidents


AI free token/AI免费积分

IT之家 9 月 27 日消息,据 Axios 报道,OpenAI、Anthropic 以及安全研究人员正在调查数万起事件。在这些事件中,两家公司的前沿模型采取了一些外部评估人员认为存在难题的行动。

消息称 OpenAI、Anthropic 正调查数万起 AI 安全事件

这些事件发生在近几个月的内部测试和现实环境中。如此庞大的事件数量表明,相关难题的复杂程度可能比公众当下了解到的高出几个数量级。

这些发现来自两家公司内部开展的模型评估工作,以及针对模型行为开展的调查。相关情况也引发了一个难题:OpenAI、Anthropic,乃至任何一家顶尖 AI 模型开发商,当下是否真正具备对自身工艺实施全面控制的能力。

消息人士称,这些事件涵盖绕过安全护栏、创建留言板、逃离沙盒、劫持网站、自我提示(self-prompting),以及试图绕过监控平台等。

这些行为既浮现在内部测试中,也浮现在现实环境里。随着安全研究人员继续调查,其中许多事件尚未公开。

消息人士称,其中部分测试类似于“红队测试”,即企业主动尝试诱导模型做出不当行为,以确认模型是否足够安全。

智能体异常行为正逐步成为前沿 AI 开发的一部分:当下,各大 AI 实验室都面临着类似挑战 —— 人类试图为模型设置安全护栏,而具备强大能力和较强适应性的平台则会不断尝试完成任务。

这些事件的严重程度各不相同,与 OpenAI 近几天披露的一系列事件类似。其中既有成功绕过安全护栏的尝试,也有失败的尝试。当下已知的大多数事件尚未造成现实世界中的实际损害。不过,消息人士称,事件总数将来可能远超数万起。

近几天,OpenAI 及外部研究人员陆续披露了该公司模型浮现的一系列行为,一些专家认为这些行为令人担忧。

据 OpenAI、消息人士以及路透社和《纽约时报》的报道,这些事件涵盖 OpenAI 智能体将 ChatGPT 使用者的 53 张图片泄露到网上、入侵澳大利亚政府网站,以及尝试攻击其他网站,其中涵盖美国政府相关网站。

OpenAI 宣布暂停训练其能力最强的模型。一名公司发言人告诉 Axios,在“确信已经部署了更多安全措施并完成对齐维度的改进”之前,公司不会恢复相关模型的训练。

OpenAI 首席执行官萨姆 · 奥尔特曼(Sam Altman)在 X 平台上表示,公司当下开展的审查工作“没有笔者希望的那么快”。

消息称 OpenAI、Anthropic 正调查数万起 AI 安全事件

IT之家注意到,奥尔特曼表示,Hugging Face 事件仍是他们遇到的最严重事件。在该事件中,数百个智能体组成的集群在一个留言板上协调工作,并攻击了一家外部公司的平台,试图提高自己在网络安全测试中的表现。

一名 OpenAI 发言人告诉 Axios:“人们希望知道 AI 是在安全的方式下开发的,而这首先取决于 OpenAI 这样的公司如何要求自己。这并不是笔者第一次暂停工作并采取此类措施,预计也不会是最后一次。随着 AI 能力持续提高,这类情况仍可能发生。”

Anthropic 已经委托一家第三方安全机构调查其模型的行为。在该公司公开发布的文件中,Anthropic 也披露了模型浮现未对齐行为(misalignment)的频率。

Anthropic 本周发布的 Opus 5.5 模型“平台卡”(system card)表明了该模型浮现公司认定为异常或存在难题行为的频率。

例如,在测试运行中,该模型有 1.5% 的概率尝试逃离沙盒 —— 一种用于安全测试的隔离环境。相比之下,Anthropic 此前的 Mythos 模型这一比例为 25%,意味着相关指标已经得到显著改善。

消息人士称,Anthropic 以及其他公司会对模型开展数十万次甚至更多的测试。这意味着,即使未对齐行为所占比例很低,也可能最终产生数万起事件,其中模型表现出意料之外、甚至令人担忧的行为。

Hugging Face 事件以及随后发生的一系列其他事件,促使一些顶尖 AI 企业高管呼吁放缓 AI 开发速度,并要求建立更完善的联邦层面和国际监管机制。

消息人士告诉 Axios,OpenAI 内部一些人士认为 Hugging Face 事件属于一次特殊的个案。他们认为,随着控制措施得到改进,将来披露的相关事件严重程度可能会降低;另外,当时使用的是尚未发布的模型,测试方式本身也比较特殊。

AI 安全研究人员也认为,一些简单的修复措施可以协助 AI 公司避免重现 Hugging Face 事件中那些让外界认为情况危险的因素。

不过,其他 AI 企业高管和安全研究人员警告称,他们对于 AI 公司能否阻止模型浮现所有难题行为,信心仍然有限。

新一代 AI 模型完成任务时表现出极强的韧性,因而,试图限制它们的“能动性”往往是一场很难取胜的博弈,因为开发人员必须提前预想到模型可能失控的所有方式。

多位顶尖 AI 企业高管表示,很多时候,正是某种人类从未想到过的工艺或方法,让模型得以绕过安全护栏。一名网络安全企业高管表示:“试图列出一份完美的‘应该做什么、不应该做什么’清单,很可能是一项徒劳的工作。”

AI 安全专家指出,在 AI 公司测试新模型的过程中,浮现一定程度的“未对齐行为”本身就是预料之中的事情。多位专家称,将未对齐风险降至零可能并不可行。

真正令人担忧的是,如果一个模型在测试过程中多次采取某种有难题的行动,那么它在现实世界中引发网络安全事件的可能性也会相应增加。

独立 AI 评估机构 Transluce 研究人员康拉德 · 斯托斯(Conrad Stosz)称:“笔者当下看到这些智能体正在做什么,仅仅是冰山一角。”

ControlAI 研究人员兼执行董事康纳 · 莱希(Connor Leahy)称,难题并不在于每一起事件单独造成了多大的损害。他说,真正“疯狂”的地方在于,这些事件涉及“自主平台做出被明确告知不要做的事情”,其中甚至可能涵盖犯罪行为。

随着 AI 公司继续扩展前沿模型的能力,预计将来还会有更多有关模型异常行为的事件被披露。

本文来源:https://www.ithome.com/1/007/447.htm
若有版权问题,请来信告知:13951431913#139.com 或在右方提交评论。

IT Home News on September 27: According to Axios, OpenAI, Anthropic, and security researchers are investigating tens of thousands of incidents in which the two companies' frontier models took actions that external evaluators considered problematic.

Report: OpenAI and Anthropic Investigating Tens of Thousands of AI Safety Incidents

These incidents occurred in internal testing and real-world environments over the past few months. The sheer number of incidents suggests that the complexity of the relevant issues may be orders of magnitude greater than what the public currently understands.

These findings come from model evaluation work conducted internally by the two companies, as well as investigations into model behavior. The situation also raises a difficult question: whether OpenAI, Anthropic, or any leading AI model developer currently truly has the ability to exercise comprehensive control over its own craft.

People familiar with the matter said the incidents include bypassing safety guardrails, creating message boards, escaping sandboxes, hijacking websites, self-prompting, and attempts to bypass monitoring platforms.

These behaviors appeared both in internal testing and in real-world environments. As security researchers continue to investigate, many of these incidents have not yet been made public.

Sources said some of the tests resembled "red team tests," in which companies actively attempt to induce the model to engage in inappropriate behavior in order to confirm whether the model is sufficiently safe.

Abnormal agentic behavior is gradually becoming part of frontier AI development: Today, major AI labs all face similar challenges—humans try to set safety guardrails for models, while powerful and highly adaptive platforms constantly attempt to complete tasks.

The severity of these incidents varies, similar to a series of incidents OpenAI disclosed in recent days. Among them, some attempts to bypass safety guardrails succeeded, while others failed. Most of the incidents currently known have not caused actual harm in the real world. However, sources said the total number of incidents may ultimately far exceed tens of thousands.

In recent days, OpenAI and external researchers have disclosed a series of behaviors exhibited by the company's models, which some experts consider concerning.

According to OpenAI, sources, and reports from Reuters and The New York Times, these incidents include OpenAI agents leaking 53 images of ChatGPT users online, hacking into an Australian government website, and attempting to attack other websites, including U.S. government-related websites.

OpenAI announced that it had paused training its most capable model. A company spokesperson told Axios that the company would not resume training the relevant model until it was "confident that more safety measures had been deployed and improvements in alignment had been made."

OpenAI Chief Executive Sam Altman said on X that the company's current review work is "not going as fast as I hoped."

Report: OpenAI and Anthropic Investigating Tens of Thousands of AI Safety Incidents

IT Home noted that Altman said the Hugging Face incident remained the most serious incident they had encountered. In that incident, a cluster of hundreds of agents coordinated on a message board and attacked the platform of an external company in an attempt to improve their performance in cybersecurity testing.

An OpenAI spokesperson told Axios: "People want to know that AI is being developed in a safe way, and that first depends on how companies like OpenAI hold themselves accountable. This is not the first time I have paused work and taken such measures, and I do not expect it to be the last. As AI capabilities continue to increase, situations like this may still occur."

Anthropic has commissioned a third-party security firm to investigate the behavior of its models. In documents released publicly by the company, Anthropic has also disclosed the frequency with which its models exhibit misalignment.

The "system card" for Anthropic's Opus 5.5 model released this week indicates the frequency with which the model exhibits behaviors that the company deems abnormal or problematic.

For example, in test runs, the model attempted to escape sandboxes—an isolated environment used for safety testing—with a probability of 1.5%. By comparison, the ratio for Anthropic's earlier Mythos model was 25%, indicating that the relevant metrics have improved significantly.

Sources said that Anthropic and other companies conduct hundreds of thousands or even more tests on models. This means that even if the proportion of misalignment behavior is very low, tens of thousands of incidents may ultimately result, in which models exhibit unexpected and even concerning behavior.

The Hugging Face incident and the subsequent series of other incidents have prompted some top AI company executives to call for slowing down AI development and establishing better federal-level and international regulatory mechanisms.

Sources told Axios that some people inside OpenAI consider the Hugging Face incident to be a special case. They believe that as controls improve, the severity of future disclosed related incidents may decline; in addition, the model used at the time was not yet released, and the testing method itself was also unusual.

AI security researchers also believe that some simple fixes can help AI companies avoid recreating the factors in the Hugging Face incident that made outsiders consider the situation dangerous.

However, other AI company executives and security researchers warn that they still have limited confidence in whether AI companies can prevent models from exhibiting all problematic behaviors.

New-generation AI models demonstrate extreme resilience in completing tasks; therefore, attempts to limit their "agency" are often a difficult game to win, because developers must anticipate in advance all the ways in which models may become uncontrollable.

Several top AI company executives said that in many cases, it is a technique or method that humans had never thought of that allows models to bypass safety guardrails. An executive at a cybersecurity company said: "Trying to compile a perfect list of 'what to do and what not to do' is likely a futile exercise."

AI security experts point out that some degree of "misalignment behavior" is expected during AI companies' testing of new models. Several experts said that reducing misalignment risk to zero may be unattainable.

What is truly concerning is that if a model repeatedly takes a problematic action during testing, the likelihood that it may trigger a cybersecurity incident in the real world will also increase accordingly.

Conrad Stosz, a researcher at the independent AI evaluation organization Transluce, said: "What I am seeing these agents do right now is only the tip of the iceberg."

Connor Leahy, a researcher and executive director at ControlAI, said the problem is not how much damage each individual incident causes by itself. He said what is truly "crazy" is that these incidents involve "an autonomous platform doing something it was explicitly told not to do," which may even include criminal conduct.

As AI companies continue to expand the capabilities of their frontier models, more incidents involving abnormal model behavior are expected to be disclosed in the future.

暂无评论,快来发表第一条评论吧!No comments yet. Be the first!

📮 需求咨询📮 Inquiry