NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic

Model Overview

• Model Architecture: Hybrid Mamba-2 + Latent Mixture-of-Experts (LatentMoE) with Multi-Token Prediction (MTP)
  • Input: Text
  • Output: Text
  • Total Parameters: 550B
  • Active Parameters: 55B
• Model Optimizations:
  • Activation quantization: FP8
  • Weight quantization: FP8
• Intended Use Cases:
  • Reasoning and complex problem solving.
  • Mathematics and science.
  • Code generation.
  • Instruction following.
• Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws).
• Release Date: 06/04/2025
• Version: 1.0
• Model Developers: Red Hat

Quantized version of nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16.

Model Optimizations

This model was obtained by quantizing the weights and activations of nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 to FP8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%.

Only the weights and activations of the linear operators within transformer blocks are quantized. Weights are quantized with a symmetric static per-channel scheme, whereas activations are quantized with a symmetric dynamic per-token scheme. The llm-compressor library is used for quantization.

Deployment

Use with vLLM

This model can be deployed efficiently using the vLLM backend.

Install dependencies:

uv pip install git+https://github.com/vllm-project/vllm.git
uv pip install llmcompressor

Launch the vLLM server:

vllm serve RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic \
 --host 0.0.0.0 --port 8088 \
 --tensor-parallel-size 8 \
 --enable-expert-parallel \
 --max-model-len 262144 \
 --gpu-memory-utilization 0.90 \
 --max-num-seqs 32 \
 --max-num-batched-tokens 32768 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --reasoning-parser nemotron_v3 \
 --mamba-ssm-cache-dtype float16 \
 --mamba-backend flashinfer \
 --enable-mamba-cache-stochastic-rounding \
 --mamba-cache-philox-rounds 5 \
 --speculative-config '{"method": "nemotron_h_mtp", "num_speculative_tokens": 5}' \
 --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 96}' \
 --trust-remote-code

Send requests:

from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8088/v1"

client = OpenAI(
 api_key=openai_api_key,
 base_url=openai_api_base,
)

model = "RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic"

messages = [
 {"role": "user", "content": "Solve for x: 2x + 5 = 13"},
]

outputs = client.chat.completions.create(
 model=model,
 messages=messages,
)

generated_text = outputs.choices[0].message.content
print(generated_text)

Creation

This model was quantized using the llm-compressor library as shown below.

from llmcompressor import model_free_ptq

MODEL_ID = "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16"
SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-FP8-dynamic"

model_free_ptq(
 model_stub=MODEL_ID,
 save_directory=SAVE_DIR,
 scheme="FP8_DYNAMIC",
 ignore=[
 "re:.*gate$",
 "lm_head",
 "model.embed_tokens",
 "re:.*mixer.conv1d.*",
 "re:.*norm_f*",
 "re:.*bias$",
 "re:.*embed_tokens$",
 "backbone.embeddings"
 ],
 max_workers=15,
 device="cuda:0",
)

Evaluation

The model was evaluated on reasoning tasks using lighteval. vLLM was used as the serving backend for all evaluations.

Install dependencies:

uv pip install git+https://github.com/vllm-project/vllm.git
uv pip install lighteval==0.13.0
uv pip install "litellm[caching]>=1.66.0"

Launch the vLLM server:

vllm serve RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic \
 --host 0.0.0.0 --port 8088 \
 --tensor-parallel-size 8 \
 --enable-expert-parallel \
 --max-model-len 262144 \
 --gpu-memory-utilization 0.90 \
 --max-num-seqs 32 \
 --max-num-batched-tokens 32768 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --reasoning-parser nemotron_v3 \
 --mamba-ssm-cache-dtype float16 \
 --mamba-backend flashinfer \
 --enable-mamba-cache-stochastic-rounding \
 --mamba-cache-philox-rounds 5 \
 --speculative-config '{"method": "nemotron_h_mtp", "num_speculative_tokens": 5}' \
 --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 96}' \
 --trust-remote-code

AIME 2025:

lighteval endpoint litellm \
 "model_name=hosted_vllm/RedHatAI__NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic,provider=hosted_vllm,base_url=http://127.0.0.1:8088/v1,timeout=3600,concurrent_requests=32,generation_parameters={temperature:1.0,top_p:0.95,max_new_tokens:32768}" \
 "aime25|0" \
 --output-dir results --save-details

GPQA Diamond:

lighteval endpoint litellm \
 "model_name=hosted_vllm/RedHatAI__NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic,provider=hosted_vllm,base_url=http://127.0.0.1:8088/v1,timeout=3600,concurrent_requests=32,generation_parameters={temperature:1.0,top_p:0.95,max_new_tokens:32768}" \
 "gpqa:diamond|0" \
 --output-dir results --save-details

Accuracy

Benchmark	nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16	nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4	RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic (this model)	RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-block	RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16
AIME 2025 (pass@1)	90.00	90.00 (100.0%)	93.33 (103.7%)	86.67 (96.3%)	86.67 (96.3%)
GPQA Diamond (pass@1)	78.79	84.85 (107.7%)	82.32 (104.5%)	81.31 (103.2%)	81.82 (103.8%)
Average	84.39	87.42 (103.6%)	87.83 (104.1%)	83.99 (99.5%)	84.24 (99.8%)

Base model

nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16

(17)