免费用到极致:在AMD Token Lab上运行Gemma 4 31B,搭配本地聊天页与ADK智能体Can't Beat Free: Running Gemma 4 31B on AMD Token Lab with a Local Chat Page and an ADK Agent


AI free token/AI免费积分

免费用到极致:在AMD Token Lab上运行Gemma 4 31B,搭配本地聊天页与ADK智能体

本文来源:https://dev.to/gde/cant-beat-free-gemma-4-31b-on-amd-token-lab-with-a-local-chat-page-and-an-adk-agent-c5b
若有版权问题,请来信告知:13951431913#139.com 或在右方提交评论。

本文提供了通过 AMD Token Lab 在 AMD GPU 上调用 Gemma 4 31B 的分步指南。AMD Token Lab 是一个免费的 API,随 AMD AI 开发者计划会员资格一同提供。在此基础上构建了一个本地聊天页面、一个终端客户端和一个小型 ADK 代理,均来自同一套设备。

https://github.com/xbill9/gemma4-dev/tree/main/tokenlab-sglang-amd-31b

问题很简单:您可以在一个免费的 API 上构建什么?Token Lab 为每位计划成员提供一个密钥、30 天的访问期和积分预算,无需信用卡,也无需信用申请。以下所有数据均来自 2026 年 11 月 13 日向其发送的请求。

Token Lab 将自己描述为“适用于领先开源模型的 OpenAI 兼容 API,在 AMD Instinct 和 Radeon 上提供服务”。基本 URL 为 https://api.tokenlab.amd.com/v1 ,它可用于 curl、OpenAI 和 Anthropic SDK、LangChain 以及 AMD AI Notebooks。

这是 AMD AI 开发者计划的会员福利之一。该计划的“云额度”页面提供三个选项:AMD Instinct Developer Cloud、Fireworks AI 和 Token Lab。Token Lab 是唯一一个标有“无需申请积分”的选项。

该目录包含10个模型,每个模型均标有“免费”字样:Gemma 4 31B、DeepSeek V4 Flash和V4.1 Flash、Qwen3.8 27B、三个MiniCPM模型和三个决策模型。Gemma-4-31B-IT 可以读取文本和图像,具有 262,144 个 token 的上下文。

Token Lab 表示,积分衡量的是每个模型的相对使用量,较大的模型每个 token 使用的积分更多。运行本文中的所有内容,23 个请求使用了 10 个积分中的 0.000335 个。

对于代理:google-adk 和 litellm ( pip install google-adk litellm )

gemma4-dev 的克隆,或仅是 tokenlab-sglang-amd-31b 目录

该计划的主页上明确写道:“该计划可免费加入,可帮助您在 AMD 硬件上构建、优化和扩展 AI。”

前往 https://developer.amd.com ,然后选择“加入计划”。对于 AMD 新用户,该页面列出的步骤如下:

填写 AMD AI 开发者计划注册表

如果您已有 AMD 账户,请登录并打开加入表单。

会员资格还包括一个月 DeepLearning.AI 会员资格、活动提前报名、与 AMD 工程师的专属 Discord 群组,以及每月一次的 Radeon 显卡或 Ryzen AI PC 抽奖活动。

打开 https://notebooks.amd.com/tokenlab ,然后使用 AMD AI 开发者计划(全球)登录信息登录。该页面包含三个步骤:获取您的密钥、复制基本 URL、从目录中调用模型。

仪表板显示剩余积分、过去 30 天的请求数、剩余天数、速率限制和密钥状态,并提供用于重新生成或撤销密钥的按钮。重新生成将保留您的积分和访问期。

将密钥放入rig的.env中,只有您才能读取,且绝不会提交:

基本 URL 和模型名称位于已提交的 tpu.env 中:

make query 发送一个聊天补全,并打印原始回复:

每个回复都包含一个以积分表示的费用字段,因此客户可以自行记录累计总额。

模特制作模型卡,Token Lab发布:

Gemma 4 31B 在 SGLang 上运行:卡片上显示“sglang-router”,回复中包含 SGLang 的 matched_stop 字段。同一目录中的三个 MiniCPM 模型都是“vllm-router”服务。Token Lab 仅将硬件命名为 AMD Instinct 和 Radeon。

工具调用和JSON输出均在支持的参数中,这是第7步中智能体所需要的。

chat_ui.py 是本仓库 TPU v5e vLLM 配置的聊天页面,指向 Token Lab。它是 Python 标准库:它在 127.0.0.1 上提供一个页面,并将 /v1/* 转发到 Token Lab,同时附上密钥,因此密钥会保留在您的计算机上,绝不会传到浏览器。

侧面板包含最大令牌数、温度、top-p、思考开关和系统提示;上一条回复的第一个令牌时间、解码速率、令牌数和积分;模型卡;以及会话总计。该回复用了 4.96 秒生成第一个 token,解码速度为每秒 9.5 个 token,消耗了 0.00000942 个积分。

🖼按钮用于附加图像,该图像以OpenAI图像格式的数据URL形式发送。“思考”通过 chat_template_kwargs.enable_thinking 启用 Gemma 的推理功能,推理内容将以流式传输的形式显示在答案上方的可折叠部分中。

ask.py 遵循此仓库中 GTX 1650 Ti 矿机的 ask 客户端:一个问题然后退出、一个管道或一段有记忆的对话。

管道输入成为材料,参数成为指令。-T 开启思考,-I photo.jpg 附加一张图片,-r 打印原始 JSON,不带参数则打开一个对话,其中 ! 用于清除历史记录。

tokenlab_agent/是Google Agent Development Kit的一个小型代理。ADK 通过 LiteLLM 的 OpenAI 提供商连接到 Token Lab,因此整个连接是一个模型对象:

该代理有两个工具。list_tokenlab_models返回Token Lab当前提供的服务。count_and_range在代码中进行计数、筛选并查找最小值和最大值,因此模型会引用一个确切的数字,并说明其使用的筛选条件:

run_agent.py 运行一个回合,并打印每个工具调用及其时间:

Gemma 将“10 或更多”解读为 at_least: 10 ,工具返回了 8,答案引用了工具的计数和筛选条件。第二个工具:

同一个代理在 ADK 的自有工具中运行:在终端中运行 make agent for adk,并在端口 8767 上运行 make agent-web for adk web。

一次一个流式传输请求,温度为 0,来自笔记本电脑:

对于一个请求,全天解码速度在每秒 2.0 至 9.5 个 token 之间,第一个 token 在 1.2 至 5.0 秒后到达。在计时运行中,响应头在半秒钟内到达,流在第一个令牌之前一直承载着“保持连接”ping。

该端点由所有Token Lab用户共享。Gemma 4 31B 对所有用户的并发请求数限制为 128 个,超过此限制的请求将收到 HTTP 429 响应,并附带 Retry-After: 1 。这些数字描述了 Token Lab 当天的负载情况,但并未说明其背后的 GPU 的速度。

当模型达到共享限制或您的密钥每分钟的请求超过 20 次时,Token Lab 会返回 429;当后端重启时,会返回 502、503 或 504。其常见问题解答要求客户端等待几秒钟,然后再次发送请求,遵守 Retry-After 。

在回复开始之前,tokenlab.py 会对聊天页面和终端客户端执行此操作,代理从 LiteLLM 的num_retries中获取该值。在开始后中断的流没有可用于重试的状态码:完整的流以data: [DONE]结尾,聊天页面会标记没有以该字符串结尾的回复。

流还包含严格解析器会报错的行: : ping while queued、id: N after each event以及event: done before [DONE] 。仅读取 data: 行。

该计划的“云额度”页面上有三个选项:

从Token Lab开始。它完全免费,除了密钥之外,无需申请,也无需进行任何设置。您可以在这里将Gemma 4 31B与其他目录中的模型进行比较,构建原型,并连接代理。

当原型需要您信赖的速度、目录之外的模型或专用的服务器时,请申请 AMD Instinct Developer Cloud 积分,并自行提供模型服务。此代码仓库的 gpu-vllm-mi300x-31b 配置在 vLLM 上提供同一模型的服务。

本文的目的是探索可以在 AMD Token Lab 上构建哪些内容,AMD Token Lab 是 AMD AI 开发者计划随附的免费 API。解决方案的关键在于一个小型客户端模块,该模块持有密钥,重试 Token Lab 要求重试的内容,并提供聊天页面、终端客户端和 ADK 代理。结果如下:

🟢 加入该计划是免费的,Token Lab 不需要信用额度请求:一个密钥附带 10 个积分、30 天和每分钟 20 个请求。

🟢 Gemma 4 31B 通过 OpenAI API 回答文本和图像,具备思考能力、工具和 262,144 个 token 的上下文。

🟢 本文中的所有内容在 23 次请求中使用了 10 分中的 0.000335 分。

🟢 本地聊天页面会将密钥保存在您的计算机上,并显示每条回复的时间和积分。

🟢 ADK 代理使用一个 LiteLlm 对象访问 Token Lab 并调用工具:它使用工具的精确结果回答了一个计数问题。

⚠️ 终端节点是共享的:一个请求以每秒 2.0 至 9.5 个 token 的速度解码,第一个 token 最多需要 5 秒。

⚠️ Gemma 4 31B 在 Token Lab 背后的 SGLang 上运行,GPU 型号未公开。

❌ 超过共享的 128 个请求限制的请求将收到 429 状态码,并且必须重新发送;在回复过程中中断的流无法自动重试。

范围:一台笔记本电脑于 2026 年 11 月 13 日调用 api.tokenlab.amd.com,一次一个请求,Gemma-4-31B-IT,计时表温度为 0。时间包括网络和其他用户对共享服务的负载,因此每小时都会发生变化。未测量并发性、长提示和答案准确性。代理使用了 google-adk 2.13.0 和 litellm 1.106.4。

在AMD代币实验室的基础上构建的策略通过循序渐进的方法得到了验证。

深入浅出的精彩分析,@xbill!AMD Token Lab的零卡、积分预算模型为开发人员提供了巨大的便利,使他们能够在不消耗云资源的情况下构建ADK代理原型。

在open-weights上运行类似代理循环的两个架构问题/注意事项:

ADK 代理回合延迟和前缀缓存:在运行带有工具声明的代理循环时,每次回合都要重新发送系统提示和工具定义,这会很快耗尽积分预算。Token Lab 的 SGLang 后端是否在多轮代理调用中保留 KV 前缀缓存?在我们的测试中,SGLang 基数注意力/前缀缓存将 310 亿参数模型的首个token生成时间 (TTFT) 缩短了约 8 倍。

代理工具目录的上下文压缩:对于本地代理平台,与将完整的 JSON 架构转储到 Gemma 的上下文窗口相比,延迟加载工具架构(或代码模式运行时)可大幅减少输入 token。

我们最近发布了一篇配套的硬件分析文章,内容涉及消费级GPU的分配以及如何避免静默式CPU降级:dev.to/thezerodollardev/your-8gb-g……

很高兴看到AMD和GDE重点介绍了真正具备生产能力的零美元开发者等级!

我喜欢低风险/零成本的方法。我选择了 Gemma,因为我喜欢使用这种模式,但还有很多其他选项可供您试用。

100% 认可 Gemma——它在指令准确性和轻量级令牌占用空间之间的架构平衡,使其在 ADK 状态循环中表现得非常迅捷。

您是否已经在AMD Token Lab上尝试运行过Qwen 2.5 Coder或DeepSeek?在我们对各种预算有限的开发环境进行的测试中,与标准指令模型相比,Qwen 2.5(尤其是7B和14B)在严格的JSON模式下表现出显著较低的幻觉率。

此外,AMD Token Lab 不强制用户预先提供信用卡信息,这一点非常重要。这种顺畅的设置让独立开发者能够测试多轮对话代理原型,而不会遇到意外的账单惊喜。

我通常为了保持一致性而坚持使用Gemma模型,但这种方法应该适用于他们提供的任何模型。我会把尝试几个模型列入我的待办事项清单。

这完全说得通——Gemma的提示格式和对指令的遵循在各种任务中都非常可靠。如果你在Token Lab上测试其他模型,Qwen 2.5 Coder(14B/32B)在代码/模式生成方面特别有意思,而DeepSeek-R1 Distill在多步推理方面表现出色。期待阅读您的后续基准测试结果!

完全同意Gemma的观点——它在遵循指令和代币经济之间的架构平衡使其反应格外灵敏,尤其是在ADK代理状态管理方面。

想知道您是否已经在 Token Lab 基础设施上试用过 Qwen 2.5 Coder 或 DeepSeek?在我们的资源受限环境基准测试中,与结构化 JSON 工具调用相比,Qwen 2.5 7B/14B 的幻觉率低得惊人。

Can't Beat Free: Running Gemma 4 31B on AMD Token Lab with a Local Chat Page and an ADK Agent

本文来源:https://dev.to/gde/cant-beat-free-gemma-4-31b-on-amd-token-lab-with-a-local-chat-page-and-an-adk-agent-c5b
若有版权问题,请来信告知:13951431913#139.com 或在右方提交评论。

This article provides a step by step guide to calling Gemma 4 31B on AMD GPUs through AMD Token Lab, a free API that comes with membership of the AMD AI Developer Program. A local chat page, a terminal client and a small ADK agent are built on top of it, all from the same rig.

https://github.com/xbill9/gemma4-dev/tree/main/tokenlab-sglang-amd-31b

The question is simple: what can you build on an API that costs nothing? Token Lab gives every program member a key, a 30-day access period and a points budget, with no credit card and no credit request. Every figure below comes from requests sent to it on 2026-11-13.

Token Lab describes itself as an OpenAI-compatible API "for leading open models, served on AMD Instinct and Radeon". The base URL is https://api.tokenlab.amd.com/v1 , and it works from curl, the OpenAI and Anthropic SDKs, LangChain, and AMD AI Notebooks.

It is one of the member perks of the AMD AI Developer Program. The program's Cloud Credits page offers three options: AMD Instinct Developer Cloud, Fireworks AI, and Token Lab. Token Lab is the only one marked "No credit request needed".

The catalog held 10 models, every one marked Free: Gemma 4 31B, DeepSeek V4 Flash and V4.1 Flash, Qwen3.8 27B, three MiniCPM models, and three decision models. Gemma-4-31B-IT reads text and images and has a 262,144-token context.

Token Lab says points measure relative usage per model, with larger models using more points per token. Running everything in this article, 23 requests used 0.000335 of the 10 points.

For the agent: google-adk and litellm ( pip install google-adk litellm )

A clone of gemma4-dev , or just the tokenlab-sglang-amd-31b directory

The program's home page says it plainly: "The program is free to join and helps you build, optimize, and scale AI on AMD hardware."

Go to https://developer.amd.com and choose Join the Program . New to AMD, the steps the page lists are:

Complete the AMD AI Developer Program sign-up form

With an existing AMD account, log in and open the join form instead.

Membership also lists a one-month DeepLearning.AI membership, early event registration, a private Discord with AMD engineers, and a monthly sweepstakes for a Radeon GPU or a Ryzen AI PC.

Open https://notebooks.amd.com/tokenlab and sign in with AMD AI Developer Program (Global) Login . The page has three steps: get your key, copy the base URL, call a model from the catalog.

The dashboard shows points remaining, requests in the last 30 days, days left, the rate limit and the key's status, with buttons to regenerate or revoke the key. Regenerating keeps your points and your access period.

Put the key in the rig's .env , readable only by you and never committed:

The base URL and the model name live in the committed tpu.env :

make query sends one chat completion and prints the raw reply:

Every reply carries a cost field in points, so a client can keep its own running total.

make models prints the model card Token Lab publishes:

Gemma 4 31B runs on SGLang: the card says "sglang-router", and replies carry SGLang's matched_stop field. Three MiniCPM models in the same catalog are "vllm-router" services. Token Lab names the hardware only as AMD Instinct and Radeon.

Tool calling and JSON output are both in the supported parameters, which is what the agent in Step 7 needs.

chat_ui.py is the chat page from this repo's TPU v5e vLLM rig, pointed at Token Lab. It is standard-library Python: it serves one page on 127.0.0.1 and forwards /v1/* to Token Lab with the key attached, so the key stays on your machine and never reaches the browser.

The side panel has max tokens, temperature, top-p, a thinking switch and a system prompt; the last reply's first-token time, decode rate, tokens and points; the model card; and the session's totals. That reply took 4.96 s to its first token, decoded at 9.5 tokens per second, and used 0.00000942 points.

The 🖼 button attaches an image, sent as a data URL in the OpenAI image format. Thinking turns on Gemma's reasoning with chat_template_kwargs.enable_thinking , and the reasoning streams into a collapsible section above the answer.

ask.py follows the ask clients of this repo's GTX 1650 Ti rigs: one question and exit, a pipe, or a conversation with memory.

Piped input becomes the material and the arguments become the instruction. -T turns thinking on, -I photo.jpg attaches an image, -r prints the raw JSON, and no arguments opens a conversation where ! clears the history.

tokenlab_agent/ is a small agent for Google's Agent Development Kit. ADK reaches Token Lab through LiteLLM's OpenAI provider, so the whole connection is one model object:

The agent has two tools. list_tokenlab_models returns what Token Lab serves right now. count_and_range counts, filters and finds the minimum and maximum in code, so the model quotes an exact figure and states the filter it used:

run_agent.py runs one turn and prints every tool call with its time:

Gemma read "10 or more" as at_least: 10 , the tool returned 8, and the answer quoted the tool's count and filter. The second tool:

The same agent runs in ADK's own tools: make agent for adk run in the terminal and make agent-web for adk web on port 8767.

One streamed request at a time, temperature 0, from a laptop:

Decode speed ranged from 2.0 to 9.5 tokens per second over the day for one request, and the first token arrived after 1.2 to 5.0 s. In the timed runs the reply headers arrived within half a second, and the stream carried keep-alive pings until the first token.

The endpoint is shared by every Token Lab user. Gemma 4 31B has a limit of 128 requests at once across all of them, and a request over it gets HTTP 429 with Retry-After: 1 . These numbers describe Token Lab's load on the day, and say nothing about the speed of the GPUs behind it.

Token Lab answers 429 when the model is at its shared limit or your key is over 20 requests a minute, and 502, 503 or 504 when a backend restarts. Its FAQ asks clients to wait a few seconds and send the request again, honouring Retry-After .

tokenlab.py does that for the chat page and the terminal client, before the reply starts, and the agent gets it from LiteLLM's num_retries . A stream that breaks after it started has no status code to retry on: a complete stream ends with data: [DONE] , and the chat page flags a reply that ends without it.

The stream also carries lines a strict parser trips on: : ping while queued, id: N after each event, and event: done before [DONE] . Read only the data: lines.

The three options on the program's Cloud Credits page:

Start with Token Lab. It costs nothing and needs no request and no setup beyond a key. It is the place to try Gemma 4 31B against the other models in the catalog, to build a prototype, and to wire up an agent.

When a prototype needs speed you can rely on, a model outside the catalog, or a server of its own, request the AMD Instinct Developer Cloud credit and serve the model yourself; this repo's gpu-vllm-mi300x-31b rig serves the same model on vLLM.

The goal of this article was to find what can be built on AMD Token Lab, the free API that comes with the AMD AI Developer Program. The key to the solution was one small client module that holds the key, retries what Token Lab asks to be retried, and serves a chat page, a terminal client and an ADK agent. The results were:

🟢 Joining the program is free, and Token Lab needs no credit request: a key comes with 10 points, 30 days and 20 requests per minute.

🟢 Gemma 4 31B answers text and images, with thinking, tools and a 262,144-token context, on the OpenAI API.

🟢 Everything in this article used 0.000335 of the 10 points over 23 requests.

🟢 The local chat page keeps the key on your machine and shows each reply's timings and points.

🟢 An ADK agent reaches Token Lab with one LiteLlm object and calls tools: it answered a counting question from a tool's exact result.

⚠️ The endpoint is shared: one request decoded at 2.0 to 9.5 tokens per second, and the first token took up to 5 s.

⚠️ Gemma 4 31B runs on SGLang behind Token Lab, and the GPU model is not published.

❌ A request over the shared 128-request limit gets 429 and must be sent again; a stream cut off mid-reply cannot be retried automatically.

Scope: one laptop calling api.tokenlab.amd.com on 2026-11-13, one request at a time, Gemma-4-31B-IT, temperature 0 for the timing table. Times include the network and other users' load on a shared service, so they change hour to hour. Concurrency, long prompts and answer accuracy were not measured. The agent used google-adk 2.13.0 and litellm 1.106.4.

The strategy for building on AMD Token Lab was validated with an incremental step by step approach.

Fascinating deep-dive, @xbill ! The zero-card, points-budget model of AMD Token Lab is a massive unlock for developers prototyping ADK agents without cloud burn.

Two architectural questions/notes from running similar agent loops on open-weights:

ADK Agent Turn Latency & Prefix Caching : When running an agent loop with tool declarations, re-sending the system prompt and tool definitions on every turn eats through points budgets fast. Does Token Lab's SGLang backend preserve KV prefix caching across multi-turn agent calls? In our testing, SGLang radix attention/prefix caching reduces time-to-first-token (TTFT) by ~8x on 31B parameter models.

Context Compaction for Agent Tool Catalogs : For local agent rigs, lazy tool schemas (or Code Mode runtimes) drastically reduce input tokens compared to dumping full JSON schemas into Gemma's context window.

We recently published a companion hardware analysis on consumer GPU allocation and avoiding silent CPU fallback: dev.to/thezerodollardev/your-8gb-g...

Great to see AMD and GDE highlighting zero-dollar developer tiers that are genuinely production-capable!

I like the low risk/no cost approach. I went with Gemma because I like working with that model- but there are tons of options you can test drive.

100% on Gemma—its architectural balance between instruction accuracy and lightweight token footprint makes it really snappy for ADK state loops.

Have you tried running Qwen 2.5 Coder or DeepSeek on AMD Token Lab yet? In our testing across budget developer environments, Qwen 2.5 (especially 7B and 14B) shows remarkably low hallucination rates on rigid JSON schemas compared to standard instruction models.

Also, AMD Token Lab not forcing a credit card upfront is huge. That friction-free setup lets solo developers test multi-turn agent prototypes without unexpected billing surprises.

I usually stick to gemma models for consistency - but the approach should work with any model they offer. putting it on my todo list to sample a couple of models

That makes total sense—Gemma's prompt format and instruction-following are super reliable across tasks. If you do test others on Token Lab, Qwen 2.5 Coder (14B/32B) is particularly interesting for code/schema generation, and DeepSeek-R1 Distill for multi-step reasoning steps. Looking forward to reading your follow-up benchmarks!

Totally agree on Gemma—its architectural balance between instruction-following and token economy makes it exceptionally responsive, especially for ADK agent state management.

Curious if you've experimented with Qwen 2.5 Coder or DeepSeek on the Token Lab infrastructure yet? In our benchmarks on resource-constrained setups, Qwen 2.5 7B/14B exhibits remarkably low hallucination rates on structured JSON tool calling compare

暂无评论,快来发表第一条评论吧!No comments yet. Be the first!

📮 需求咨询📮 Inquiry