AI 巨头联合支持新的爬虫退出标准,OpenAI、谷歌、微软在列The biggest names in AI are backing a new way to opt out of scraping


AI free token/AI免费积分
AI 巨头联合支持新的爬虫退出标准,OpenAI、谷歌、微软在列

科技 关闭 科技 来自该主题的帖子将添加到您的每日电子邮件摘要和主页动态中。关注 关注 查看全部 科技

来自该主题的帖子将添加到您的每日电子邮件摘要和主页动态中。

AI 关闭 AI 来自该主题的帖子将添加到您的每日电子邮件摘要和主页动态中。关注 关注 查看全部 AI

来自该主题的帖子将添加到您的每日电子邮件摘要和主页动态中。

新闻 关闭 新闻 来自该主题的帖子将添加到您的每日电子邮件摘要和主页动态中。关注 关注 查看全部 新闻

来自该主题的帖子将添加到您的每日电子邮件摘要和主页动态中。

这项新的控制功能将允许发布者在多个爬虫中应用其 AI 训练偏好。

这项新的控制功能将允许发布者在多个爬虫中应用其 AI 训练偏好。

来自该作者的帖子将添加到您的每日电子邮件摘要和主页动态中。

来自该作者的帖子将添加到您的每日电子邮件摘要和主页动态中。

据 Search Engine Roundtable 早前报道,OpenAI、Google、Microsoft 和 Anthropic 表示,他们将采用一项新标准,发布者可借此选择不被 AI 抓取。借助这项控制功能,发布者可以跨多家 AI 公司统一应用其 AI 训练偏好,而无需逐一单独设置。

这个共享的“policy token”信号由 Anthropic 标准技术负责人 Kevin Kelley 提出,属于互联网工程任务组(IETF)的一部分。该信号将放置在网站的 robots.txt 文件中。长期以来,robots.txt 一直用于表明哪些网络爬虫可以或不可以抓取网站——无论是用于搜索索引、存档还是 AI 训练。目前,网站必须分别向每家 AI 公司发送其 AI 训练偏好。

这项公告发布之际,正值发布者应对持续涌入的 AI 爬虫——它们抓取网站内容并将其喂给 AI 模型。一些公司和组织已尝试对抗未经授权的抓取,例如 Really Simple Licensing(RSL)标准让发布者能够对访问其网站的爬虫收费,而 Cloudflare 则默认拦截 AI 爬虫。

OpenAI、Google、Microsoft 和 Anthropic 表示,一旦遇到该共享信号,他们将限制其 AI 爬虫访问相应网站。Meta 沟通负责人 Andy Stone 也表达了对这一新设置的支持,但并未说明公司是否会实际遵循。“我们认为,创建一个能一次向多个提供商传达此类偏好的通用信号是重要的一步,”Stone 说。“我们期待与其他 AI 公司就该标准及其在全行业的实施展开合作。”

Microsoft 表示,将提供细粒度的控制选项,使发布者能够选择退出 AI 训练或 AI grounding——后者是将 AI 模型指向经过验证的信息来源以减少幻觉的过程。Microsoft 指出,无论偏好如何,网站仍将出现在 Bing 的搜索结果中。Google 此前也实施了类似控制,允许网站决定其内容是否出现在 AI 搜索功能中或被用于 AI 训练,同时不影响其在搜索中的可见性。

尽管 Anthropic 已采用该信号,Google 和 Microsoft 计划明年支持。OpenAI 表示将在“未来数月”内实施。

更新,10 月 30 日:新增 Meta 的评论。

Emma Roth 关闭 Emma Roth 新闻撰稿人 来自该作者的帖子将添加到您的每日电子邮件摘要和主页动态中。关注 关注 查看全部 作者:Emma Roth

来自该作者的帖子将添加到您的每日电子邮件摘要和主页动态中。

AI 关闭 AI 来自该主题的帖子将添加到您的每日电子邮件摘要和主页动态中。关注 关注 查看全部 AI

来自该主题的帖子将添加到您的每日电子邮件摘要和主页动态中。

Anthropic 关闭 Anthropic 来自该主题的帖子将添加到您的每日电子邮件摘要和主页动态中。关注 关注 查看全部 Anthropic

来自该主题的帖子将添加到您的每日电子邮件摘要和主页动态中。

Google 关闭 Google 来自该主题的帖子将添加到您的每日电子邮件摘要和主页动态中。关注 关注 查看全部 Google

来自该主题的帖子将添加到您的每日电子邮件摘要和主页动态中。

Microsoft 关闭 Microsoft 来自该主题的帖子将添加到您的每日电子邮件摘要和主页动态中。关注 关注 查看全部 Microsoft

来自该主题的帖子将添加到您的每日电子邮件摘要和主页动态中。

新闻 关闭 新闻 来自该主题的帖子将添加到您的每日电子邮件摘要和主页动态中。关注 关注 查看全部 新闻

来自该主题的帖子将添加到您的每日电子邮件摘要和主页动态中。

科技 关闭 科技 来自该主题的帖子将添加到您的每日电子邮件摘要和主页动态中。关注 关注 查看全部 科技

来自该主题的帖子将添加到您的每日电子邮件摘要和主页动态中。

Sam Altman 表示,RAM 放在数据中心比放在你的桌面上更好

Apple Pencil 让 Duo 成了我新的最爱笔记本

Sam Altman 同情数学家,但表示 OpenAI 的数学攻势不会停止

HomeView 和 Echo Show 16 或许正是我一直想要的家用电脑

Reddit 将强制所有人登录

本文来源:https://www.theverge.com/tech/1023763/openai-google-microsoft-ai-scraping-standard-opt-out
若有版权问题,请来信告知:13951431913#139.com 或在右方提交评论。

Tech Close Tech Posts from this topic will be added to your daily email digest and your homepage feed. Follow Follow See All Tech

Posts from this topic will be added to your daily email digest and your homepage feed.

AI Close AI Posts from this topic will be added to your daily email digest and your homepage feed. Follow Follow See All AI

News Close News Posts from this topic will be added to your daily email digest and your homepage feed. Follow Follow See All News

The new control will allow publishers to apply their AI training preferences across multiple crawlers.

Posts from this author will be added to your daily email digest and your homepage feed.

OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable . With this control, publishers can apply their AI training preferences across multiple AI companies, instead of setting them individually.

The shared “policy token” signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It will live within a website’s robots.txt file , which has long signaled which web crawlers can and can’t scrape a site — whether for search indexing, archiving, or AI training. Currently, sites have to signal their AI training preferences separately to each AI company.

The announcement comes as publishers grapple with the continuing influx of AI crawlers scraping their sites and feeding their content into AI models. Some companies and organizations have attempted to combat unauthorized scraping, with the Really Simple Licensing (RSL) standard giving publishers the ability to charge crawlers that visit their sites, and Cloudflare blocking AI crawlers by default .

OpenAI , Google , Microsoft , and Anthropic say they will restrict their AI crawlers from accessing a site if they encounter the shared token. Meta communications head Andy Stone also signaled support for the new setting, but doesn’t say whether the company will honor it. “We believe creating a common signal that conveys such preferences to multiple providers at once is an important step,” Stone says. “And we look forward to working with other AI companies on the standards and their implementation across the industry.”

Microsoft says it will offer granular controls that will give publishers the ability to opt out of AI training or AI grounding , the process of pointing an AI model to a verified source of information to reduce hallucinations. Microsoft notes that websites will still appear in Bing’s search results regardless of their preferences. Google previously implemented similar controls that allow sites to control whether their content appears in AI search features or is used for AI training, without impacting visibility in search.

The biggest names in AI are backing a new way to opt out of scraping

While Anthropic has already adopted the signal, Google and Microsoft plan to support it next year. OpenAI says it will implement it in the “coming months.”

Update, October 30th: Added a comment from Meta.

Emma Roth Close Emma Roth News Writer Posts from this author will be added to your daily email digest and your homepage feed. Follow Follow See All by Emma Roth

Anthropic Close Anthropic Posts from this topic will be added to your daily email digest and your homepage feed. Follow Follow See All Anthropic

Google Close Google Posts from this topic will be added to your daily email digest and your homepage feed. Follow Follow See All Google

Microsoft Close Microsoft Posts from this topic will be added to your daily email digest and your homepage feed. Follow Follow See All Microsoft

Sam Altman says it’s better for RAM to be in a data center than on your desk

The Apple Pencil turned the Duo into my new favorite notebook

Sam Altman sympathizes with mathematicians, but says OpenAI’s math onslaught won’t stop

The HomeView and Echo Show 16 could be the family computers I’ve been asking for

暂无评论,快来发表第一条评论吧!No comments yet. Be the first!

📮 需求咨询📮 Inquiry