google.com, pub-2571979842820424, DIRECT, f08c47fec0942fa0
Artificial intelligence

NVIDIA Releases Audex (Nemotron-Labs-Audex-30B-A3B): An Integrated Audio-Text LLM That Preserves Its Core Text Intelligence

Photo of admin admin Send an email 28 minutes ago
0 0 4 minutes read
NVIDIA Releases Audex (Nemotron-Labs-Audex-30B-A3B): An Integrated Audio-Text LLM That Preserves Its Core Text Intelligence





NVIDIA has been released Audex (Nemotron-Labs-Audex-30B-A3B)a large-scale text-to-text language model. It understands and produces both sound and speech. It also retains the intellectual intelligence of its core. Checkpoints, and the smaller Audex-2B, are released under a non-commercial license.

Many multimodal models pay textual tax. When labs add audio or visual output, text benchmarks tend to drop. The NVIDIA research team reports this even with speech output models only. Audex is designed to avoid that downside.

The TL;DR

  • Audex is a single model 30B-A3B MoE that handles sound inside and out.
  • Audio input goes into text embedding; audio output is treated as text tokens.
  • Text points are similar to the backbone, with small gains and small losses per benchmark.
  • Multi-stage SFT plus text-only-Cascade RL avoids multimodal text regression.
  • A few open models produce normal sound beyond speech.

What is Audex?

Audex is a single Mixture-of-Experts (MoE) Transformer decoder. It has 30B total parameters and 3B open per token. The backbone is the Nemotron-Cascade-2-30B-A3B, MoE LLM text only. That core is a hybrid Mamba-Transformer with 52 layers. It uses 128 active experts and 6 activated experts.

The design is deliberately simple. Audio input is encoded and displayed in the text embedding space. Text tokens and limited audio tokens are treated the same during generation. There is no speaker isolation and no stacked models.

Because the design remains simple, Audex uses standard LLM stacks. These include Megatron-LM for training and vLLM for imaging. It supports both teaching mode and reasoning mode. Context length is up to 1M tokens.

How Integrated Design Works

Three components sit around the core of the LLM:

  • An audio encoder reads audio. Audex uses AF-Whisper from Audio Flamingo 3. It shares the architecture of Whisper Large-v3 and handles 16kHz input.
  • Two-layer MLP adapters map sound properties to model dimensions.
  • An extended vocabulary holds different audio output tokens. The initial 131,072 tokens grew to 205,312.

Audex uses two output codecs. Speech uses X-Codec2 at 50 tokens per second. It works with single-layer finite scalar quantization (FSQ) with a codebook of 65,536.

Non-speech audio uses the X-Codec at 200 tokens per second. It uses four residual vector quantization (RVQ) layers. Complex audio gets a larger token budget than speech. The interactive demo below calculates these token counts at any given time.