NVIDIA AI Releases Nemotron-Labs-Sigma, a Continuous Diffusion Language Model in 14 Checkpoints at 3B and 8B

NVIDIA AI Releases Nemotron-Labs-Sigma, a Continuous Diffusion Language Model in 14 Checkpoints at 3B and 8B
2 min read

NVIDIA has released Nemotron-Labs-Sigma, a collection of 14 open checkpoints, available in 3B and 8B parameter sizes. This family of language models belongs to the continuous diffusion category: it denoises text in a smooth 16-dimensional latent space. According to NVIDIA’s team, this approach makes few-step distillation and the steering of generation trajectories more practical for text.

Most diffusion models, like LLaDA or Dream, operate on discrete tokens, meaning text units treated as distinct states. Sigma operates in a continuous space: each token is represented by a point in an embedding space, a numerical representation learned by the model. During training, Gaussian noise is added to these representations, and the model learns to remove it. The geometry of this space is learned simultaneously with the denoiser.

A Multi-Phase Architecture and Training

Generation is performed in blocks. Each denoised block is stored in a KV cache, a memory that retains already-performed computations, before the next one begins. By default, blocks consist of 64 tokens. The architecture is based on a Ministral-type Transformer decoder, with grouped-query attention, SwiGLU, and rotary embeddings.

Training begins with autoregressive pre-training, meaning word-by-word generation, on approximately 1T tokens. It continues with around 0.3T tokens of diffusion training. For inference, the model card recommends a classifier-free guidance of 3 and a score temperature of 0.3, with 32 DDIM steps. The base model’s context window is 4,096 tokens, and 8,192 tokens for Sigma-8B-instruct.

On NVIDIA’s six-task benchmark suite, the base Sigma-8B scores an average of 71.16, ahead of NLD-8B (70.22), Dream-7B (58.74), and LLaDA-8B (41.78). Qwen3-8B, in autoregressive mode, scores 68.91. The model excels in programming, with 65.12 on HumanEval versus 62.80 for NLD-8B, but remains behind on MBPP, at 65.74 versus 67.00.

After supervised fine-tuning, Sigma-8B-instruct scores 91.96 on GSM8K, 78.40 on MATH-500, and 82.50 on HumanEval. Hard reasoning remains limited, with 23.33 on AIME24 and 17.00 on AIME25. Its average across ten tasks reaches 59.14, slightly below that of NLD-8B at 59.92. These scores are evaluations conducted by NVIDIA.

Speed and Deployment

The distilled variant Sigma-distill-8B-nfe8 decodes each block of 32 tokens in eight calls to the denoiser, plus a final readout. Its card indicates an average of 70.34 on six tasks, with 88.44 on GSM8K and 65.18 on HumanEval. The distill-spec variants add speculative decoding: a diffusion head proposes a block of 32 tokens, which the model’s autoregressive head verifies. According to NVIDIA, the result is identical to greedy autoregressive decoding.

On a GB200 with SGLang, the speculative 8B model is on average 3.46 times faster than autoregressive decoding, compared to 3.04 times for NLD-8B. Peaks reach 4.20 times for the 8B and 3.18 times for the 3B. These measurements are self-reported by NVIDIA.

The 14 checkpoints are available on Hugging Face under the NVIDIA Nemotron Open Model license, whose cards indicate it permits commercial use. They require a CUDA GPU in bfloat16, among Ampere, Lovelace, Hopper, or Blackwell architectures, and do not support CPU inference. On an H100, the base model generates approximately 120 to 140 tokens per second. NVIDIA does not host any API.

S’abonner
Notification pour
guest

0 Commentaires
Le plus ancien
Le plus récent Le plus populaire
0
Nous aimerions avoir votre avis, veuillez laisser un commentaire.x