本文来源:https://dev.to/jangwook_kim_e31e7291ad98/nvidia-nemotron-3-nano-omni-multimodal-agent-dev-guide-54d7
若有版权问题,请来信告知:13951431913#139.com 或在右方提交评论。
当大多数团队构建多模态 AI 代理时,他们会将不同的模型拼接在一起:用于处理屏幕截图的视觉模型、用于处理音频的语音转文本模型、用于处理 PDF 的文档解析器,以及用于跨所有内容进行推理的语言模型。这种管道的维护成本高昂,上下文交接会丢失信息,而且每个模型都会增加延迟。
NVIDIA 的 Nemotron 3 Nano Omni 于 2026 年 4 月 28 日发布,旨在将该堆栈整合为一个单一的开放模型。它以原生方式处理文本、图像、视频、音频和文档——不是通过一系列专门的子模型,而是通过一个统一的架构,其中所有模态都共享相同的推理路径。而且,它在执行此操作时,每个推理token仅激活其300亿个参数中的30亿个,因此可以在配备25GB VRAM的单个GPU上部署。
本指南介绍了架构、经过验证的基准数据、部署选项,以及Nemotron 3 Nano Omni在真实代理系统中适用(和不适用)的场景。
该模型在技术上的核心赌注是混合的 Mamba-Transformer 混合专家 (MoE) 主干。了解每个部分有助于解释英伟达为何做出这一设计选择。
标准密集模型会为每个token激活每个参数。一个 300 亿参数的稠密模型在每次前向传播中都需要计算 300 亿个参数。Nemotron 3 Nano Omni 使用稀疏 MoE:每个 MoE 层有 128 个专家,将每个 token 路由到前 6 个专家和一个共享专家。对于任何给定的token,只有一小部分模型的权重是激活的。
这就是为什么该模型在 25GB 的 RAM(FP8 量化)上运行,而不是在同等密集模型所需的 60+ GB 上运行。
23 个 MoE 层,配备 128 名专家、前 6 名路由和一名共享专家
Mamba 层可高效处理 25.6 万个 token 的上下文窗口。传统注意力机制的计算量随上下文长度的增加而呈平方级增长——使用纯注意力模型处理 256K 个 token 的文档,计算成本非常高。Mamba 的循环式处理方式使长上下文的成本保持在可控范围内。
6 个 GQA 层经过战略性布置,以便在最关键的地方保留全局上下文。这种混合意味着模型获得了 Mamba 的上下文效率,同时又没有失去注意力机制为复杂推理提供的全局表达能力。
Vision:C-RADIOv4-H — 处理图像、图表、屏幕截图和文档的动态分辨率
音频:Parakeet-TDT-0.6B-v2 — 处理语音和一般音频
视频:Conv3D 在视频帧进入上下文窗口之前对其进行高效的时间压缩采样
所有三个编码器都输入到同一个推理主干中,因此模型可以在一个推理流中将音频片段中说的话与同一时间戳出现在屏幕上的内容联系起来。
Nemotron 3 Nano Omni 接受在单个提示中对这些输入的任意组合:
图像:照片、图表、示意图、屏幕截图、文档页面
视频:MP4 和类似格式;Conv3D 时间压缩可处理长片段
音频:语音识别、音频内容分析
文档:PDF、财务报表、合同——视觉编码器同时处理文档布局和文本提取
256K 的上下文窗口意味着单个提示可以同时包含多份长篇文档、一份音频转录文本和多个视频帧序列。
这些数据来自 NVIDIA 的官方来源以及由 Coactive 运行的 MediaPerf 基准测试。
MediaPerf 以每小时计算时间内处理的视频小时数来衡量吞吐量,并衡量每次运行的成本。
关于标记任务(大规模对视频内容进行分类):
Nemotron 3 Nano Omni:每计算小时 9.91 小时视频
成本:每次运行 14.27 美元(所有基准模型中最低)
Nemotron 3 Nano Omni:每计算小时 10.79 小时视频
与具有同等交互性的其他开放式 Omni 模型相比,该模型在多文档工作负载上的系统效率高 7.4 倍,在视频工作负载上高 9.2 倍。
在多模态场景下,单流推理速度比替代方案快 2.9 倍。
这些是 NVIDIA 报告的针对特定工作负载的基准测试结果。针对您特定数据的实际性能会有所不同。吞吐量优势是结构性的(MoE 稀疏激活)且可重现的,但文档布局质量、音频清晰度和视频内容复杂性都会影响实际结果。
尝试 Nemotron 3 Nano Omni 的最快途径是免费的 OpenRouter 端点。截至撰写本文时,每百万个输入和输出token的费用为0美元,但适用速率限制(免费套餐下,每分钟约20个请求,每天200个请求)。
模型 ID:nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free
与OpenAI兼容的API意味着任何使用OpenAI SDK的现有代码都可以在此处通过更换URL来使用。
NVIDIA NIM 提供优化的容器化部署,并具备硬件感知的推理能力。对于希望获得 NVIDIA 支持的可靠性和 TensorRT-LLM 优化的生产工作负载,这是推荐的路径。
通过 build.nvidia.com 访问。NIM 将量化和批处理配置抽象化;您调用的是相同的 OpenAI 兼容 API,但后端是运行在 NVIDIA 优化基础设施上的 TensorRT-LLM。
如果您需要数据驻留或希望在自己的 GPU 基础设施上运行,vLLM 是社区支持的途径。官方 vLLM 博客文章(与模型发布同日发布)介绍了完整的设置过程。
如果您的 GPU 支持 FP8,请使用 --dtype fp8,以将内存使用量减半。对于多 GPU 配置,请增加 --tensor-parallel-size 。
提供服务后,端点将与 OpenAI 兼容,网址为 http://localhost:5000/v1。
NVIDIA 在 NVIDIA-NeMo/Nemotron GitHub 仓库的 usage-cookbook/Nemotron-3-Nano-Omni/vllm_cookbook.ipynb 中提供了 vLLM 食谱,其中涵盖了多模态输入格式化和批处理策略。
AWS 于 2026 年 4 月 29 日(发布后一天)将 Nemotron 3 Nano Omni 添加到 SageMaker JumpStart 中。这为您提供了具有 SageMaker 的扩展、日志记录和 IAM 集成的托管部署,如果您的基础设施已在 AWS 中,这将非常有用。
对于没有GPU优化开销的本地实验,LM Studio支持该模型。在LM Studio模型浏览器中搜索“nemotron-3-omni”。此路径比 vLLM 慢,但无需进行服务器设置。
Nemotron 3 Nano Omni 被定位为感知子代理:一种处理所有多模态输入并向编排层提供结构化理解的模型。
该模型针对计算机使用场景进行了训练——它可以解读 GUI 屏幕截图,跟踪一系列帧中的 UI 状态,并推断接下来应采取的操作。实际应用包括需要了解当前显示内容的浏览器自动化代理、监控仪表板的事件响应代理以及分析屏幕录像的客户支持工作流程。
对于处理合同、财务报表或研究论文的管道,Nemotron 3 Nano Omni 的文档理解功能可以处理布局感知的提取——它能识别表格结构及其中的文本,而不是将页面视为一串平面的字符。256K 的上下文长度意味着一份冗长的合同或多页报告可以一次性处理完毕。
客户支持通话分析、带视觉上下文的会议转录、大规模视频内容标记——这些工作负载的 MediaPerf 指标会转化为实际的成本差异。在标记任务中,每计算小时可处理 9.91 小时的视频,因此运行持续视频监控在成本上变得可行,而 GPT-5.1 或 Gemini 3.0 Pro 等效工作负载则无法做到这一点。
模型本身描述了其预期作用:将感知输出输入到单独的推理或编排代理中。一种常见模式是使用 Nemotron 3 Nano Omni 从多模态输入中提取结构化观察结果,然后将这些观察结果传递给推理模型(或使用 Nemotron 3 Nano Omni 自身的推理能力进行下一步)。
在所有情况下将其视为专业模型的替代品
与将多个专用模型组合用于混合模态工作负载相比,Nemotron 3 Nano Omni 更快、更便宜。对于纯文本任务或您已对专用模型进行过优化的专业高精度文档 OCR,专用模型可能仍然表现更佳。
BF16 检查点需要大约 60GB 的磁盘空间和 48GB 以上的 VRAM。大多数团队会希望使用 FP8(25GB VRAM)或 NVFP4(需要 Blackwell GPU)。在拉取之前,请检查 huggingface-cli 下载的磁盘空间要求。
忽略 Mamba 特有的注意力模式
混合Mamba-Transformer架构意味着一些注意力模式与纯Transformer模型有所不同。如果您要迁移的提示依赖于非常具体的跨文档注意力行为(例如,跨文档的细粒度引用链接),请对照您的具体数据进行测试——不要假定其行为与纯Transformer模型相同。
在没有足够VRAM的情况下,在vLLM中将max-model-len设置得过高
256K 的上下文窗口是理论上的最大值。在实践中,包含多模态内容的 256K 个 token 的上下文所需的 VRAM 远多于纯文本。从 --max-model-len 131072 (128K) 开始,然后根据您的实际工作负载要求和可用内存进行增加。
模型权重在 Hugging Face 上根据 NVIDIA 的开放模型许可证(而非 Apache 2.0)公开提供。在将模型用于商业用途之前,您可以在 HuggingFace 模型卡上查看许可证。NVIDIA-NeMo/Nemotron 中的训练方案和使用指南采用 MIT 许可证。
问:9 倍吞吐量的说法与我在实践中看到的相比如何?
9 倍的数字来自 MediaPerf,这是一个专注于大规模标记和摘要的视频理解基准。这反映了 MoE 架构在批量工作负载方面的结构优势。单请求交互式使用所表现出的优势较小(约2.9倍单流)。对于生产视频处理管道,批量吞吐量的提升是实实在在的;对于小规模的交互式应用,差距则会缩小。
NVIDIA 在 NVIDIA-NeMo/Nemotron GitHub 仓库中提供了训练方案。NVFP4 检查点专为 Blackwell GPU 微调而设计。对完整的 BF16 模型进行微调需要多 GPU 配置。Hugging Face GGUF 社区变体(来自 Unsloth)更适合在消费级硬件上进行 LoRA 微调。
问:Omni 和基础版 Nemotron 3 Nano 有什么区别?
基础版Nemotron 3 Nano(也在Hugging Face上,网址为nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 )仅支持文本,不具备视觉或音频编码器。它使用相同的 MoE 语言主干。Nemotron 3 Nano Omni 在基础上增加了 C-RADIOv4-H 视觉编码器和 Parakeet-TDT 音频编码器,以实现全面的多模态支持。
本文来源:https://dev.to/jangwook_kim_e31e7291ad98/nvidia-nemotron-3-nano-omni-multimodal-agent-dev-guide-54d7
若有版权问题,请来信告知:13951431913#139.com 或在右方提交评论。
When most teams build a multimodal AI agent, they stitch together separate models: a vision model for screenshots, a speech-to-text model for audio, a document parser for PDFs, and a language model to reason across all of it. The pipeline is expensive to maintain, the context handoffs lose information, and each model adds latency.
NVIDIA's Nemotron 3 Nano Omni, released April 28, 2026, is designed to collapse that stack into a single open model. It processes text, images, video, audio, and documents natively — not through a pipeline of specialized sub-models, but through a unified architecture where all modalities share the same reasoning path. And it does this while activating only 3 billion of its 30 billion parameters per inference token, making it deployable on a single GPU with 25GB of VRAM.
This guide covers the architecture, verified benchmark numbers, deployment options, and where Nemotron 3 Nano Omni fits (and doesn't fit) in a real agentic system.
The model's core technical bet is a hybrid Mamba-Transformer Mixture-of-Experts (MoE) backbone. Understanding each piece helps explain why NVIDIA made this design choice.
Standard dense models activate every parameter for every token. A 30B dense model costs the compute of 30B parameters on each forward pass. Nemotron 3 Nano Omni uses sparse MoE: 128 experts per MoE layer, routing each token to the top 6 experts plus a shared expert. For any given token, only a fraction of the model's weights are active.
This is why the model runs on 25GB of RAM (FP8 quantized) rather than the 60+ GB a comparable dense model would require.
23 MoE layers with 128 experts, top-6 routing, and a shared expert
The Mamba layers handle the 256K context window efficiently. Traditional attention scales quadratically with context length — processing a 256K-token document with a pure-attention model is computationally expensive. Mamba's recurrent-style processing keeps long-context costs manageable.
The 6 GQA layers are placed strategically to preserve global context where it matters most. This hybrid means the model gets Mamba's context efficiency without losing the global expressivity that attention provides for complex reasoning.
Vision : C-RADIOv4-H — handles dynamic resolution for images, charts, screenshots, and documents
Audio : Parakeet-TDT-0.6B-v2 — processes speech and general audio
Video : Conv3D temporal compression samples video frames efficiently before they enter the context window
All three encoders feed into the same reasoning backbone, so the model can tie what was said in an audio clip to what appeared on screen at the same timestamp — in a single reasoning stream.
Nemotron 3 Nano Omni accepts any combination of these inputs in a single prompt:
Images : photos, charts, diagrams, screenshots, document pages
Video : MP4 and similar formats; Conv3D temporal compression handles long clips
Audio : speech recognition, audio content analysis
Documents : PDFs, financial statements, contracts — the vision encoder handles document layout and text extraction together
The 256K context window means a single prompt can include multiple long documents, an audio transcript, and several video frame sequences simultaneously.
These figures come from NVIDIA's official sources and the MediaPerf benchmark run by Coactive.
MediaPerf measures throughput in hours-of-video processed per hour of compute time, and cost per run.
On the tagging task (classifying video content at scale):
Nemotron 3 Nano Omni: 9.91 hours of video per compute hour
Cost: $14.27 per run (lowest across all benchmarked models)
Nemotron 3 Nano Omni: 10.79 hours of video per compute hour
Compared to other open omni models with equivalent interactivity, the model delivers 7.4× higher system efficiency on multi-document workloads and 9.2× higher on video workloads.
2.9× the single-stream reasoning speed compared to alternatives in multimodal scenarios.
These are NVIDIA-reported benchmarks on specific workloads. Real-world performance on your specific data will vary. The throughput advantage is structural (MoE sparse activation) and reproducible — but document layout quality, audio clarity, and video content complexity all affect practical outcomes.
The fastest path to trying Nemotron 3 Nano Omni is the free OpenRouter endpoint. At time of writing, it costs $0 per million input and output tokens, though rate limits apply (approximately 20 requests per minute, 200 per day on the free tier).
Model ID: nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free
The OpenAI-compatible API means any existing code using the OpenAI SDK works here with a URL swap.
NVIDIA NIM provides an optimized containerized deployment with hardware-aware inference. This is the recommended path for production workloads where you want NVIDIA-supported reliability and TensorRT-LLM optimizations.
Access through build.nvidia.com. The NIM abstracts away quantization and batching configuration; you call the same OpenAI-compatible API but the backend is TensorRT-LLM running on NVIDIA-optimized infrastructure.
If you need data residency or want to run on your own GPU infrastructure, vLLM is the community-supported path. The official vLLM blog post (published the same day as the model release) covers the full setup.
Use --dtype fp8 if your GPU supports FP8 to halve memory usage. For multi-GPU setups, increase --tensor-parallel-size .
After serving, the endpoint is OpenAI-compatible at http://localhost:5000/v1 .
NVIDIA provides a vLLM cookbook in the NVIDIA-NeMo/Nemotron GitHub repository at usage-cookbook/Nemotron-3-Nano-Omni/vllm_cookbook.ipynb , which covers multimodal input formatting and batching strategies.
AWS added Nemotron 3 Nano Omni to SageMaker JumpStart on April 29, 2026, one day after release. This gives you a managed deployment with SageMaker's scaling, logging, and IAM integration — useful if your infrastructure is already in AWS.
For local experimentation without GPU optimization overhead, LM Studio supports the model. Search for "nemotron-3-omni" in the LM Studio model browser. This path is slower than vLLM but requires no server setup.
Nemotron 3 Nano Omni is positioned as a perception sub-agent : a model that handles all multimodal input processing and feeds structured understanding into an orchestration layer.
The model is trained for computer use scenarios — it can interpret a GUI screenshot, track UI state across a sequence of frames, and reason about what action to take next. Practical applications include browser automation agents that need to understand what's currently displayed, incident response agents monitoring dashboards, and customer support workflows that analyze screen recordings.
For pipelines processing contracts, financial statements, or research papers, Nemotron 3 Nano Omni's document understanding handles layout-aware extraction — it sees the table structure and the text within it rather than treating the page as a flat stream of characters. The 256K context means an entire lengthy contract or multi-page report can fit in a single call.
Customer support call analysis, meeting transcription with visual context, video content tagging at scale — these are the workloads where the MediaPerf numbers translate to real cost differences. At 9.91 hours of video processed per compute hour on tagging tasks, running continuous video monitoring becomes cost-viable in a way that GPT-5.1 or Gemini 3.0 Pro equivalent workloads are not.
The model itself describes its intended role: feed the perceptual output into a separate reasoning or orchestration agent. A common pattern is using Nemotron 3 Nano Omni to extract structured observations from multimodal inputs, then passing those observations to a reasoning model (or using Nemotron 3 Nano Omni's own reasoning capabilities for the next step).
Treating it as a replacement for specialized models in all cases
Nemotron 3 Nano Omni is faster and cheaper than combining multiple specialized models for mixed-modality workloads. For pure-text tasks or specialized high-accuracy document OCR where you've tuned a specialized model, a dedicated model may still outperform it.
The BF16 checkpoint requires roughly 60GB of disk space and 48GB+ VRAM. Most teams will want FP8 (25GB VRAM) or NVFP4 (requires Blackwell GPU). Check huggingface-cli download disk space requirements before pulling.
Ignoring the Mamba-specific attention pattern
The hybrid Mamba-Transformer architecture means some attention patterns differ from pure-transformer models. If you're migrating a prompt that relies on very specific cross-document attention behavior (e.g., fine-grained citation linking across documents), test against your specific data — don't assume the same behavior as a pure-transformer model.
Setting max-model-len too high in vLLM without enough VRAM
The 256K context window is the theoretical maximum. In practice, a 256K-token context with multimodal content requires substantially more VRAM than text-only. Start with --max-model-len 131072 (128K) and increase based on your actual workload requirements and available memory.
The model weights are publicly available on Hugging Face under NVIDIA's open model license — not Apache 2.0. You can inspect the license at the HuggingFace model card before using it commercially. The training recipes and usage cookbooks in NVIDIA-NeMo/Nemotron are MIT-licensed.
Q: How does the 9x throughput claim compare to what I'll see in practice?
The 9x figure comes from MediaPerf, a video understanding benchmark focused on tagging and summarization at scale. This reflects the MoE architecture's structural advantage on batch workloads. Single-request interactive use shows a smaller advantage (~2.9x single-stream). For production video processing pipelines, the batch throughput improvement is real — for low-volume interactive applications, the gap narrows.
NVIDIA provides training recipes in the NVIDIA-NeMo/Nemotron GitHub repository. The NVFP4 checkpoint is designed for Blackwell GPU fine-tuning. Fine-tuning the full BF16 model requires multi-GPU setup. The Hugging Face GGUF community variant (from Unsloth) is better suited for LoRA fine-tuning on consumer hardware.
Q: What's the difference between the Omni and the base Nemotron 3 Nano?
The base Nemotron 3 Nano (also on Hugging Face at nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 ) is text-only — no vision or audio encoders. It uses the same MoE language backbone. Nemotron 3 Nano Omni adds the C-RADIOv4-H vision encoder and Parakeet-TDT audio encoder to the base for full multimodal support.
