Meta has unveiled Muse Spark Open, a large open-weight model designed for coding and ‘agentic’ tasks, meaning where the model autonomously chains multiple actions. According to Hugging Face, which published a detailed guide on the model, it is the most capable American open-weight model to date. It accepts text, images, audio, and video as input and produces only text.
Muse Spark Open is the large version of the Muse series, following the compact Muse Glimmer model Meta released in August. It is based on a mixture-of-experts (MoE) architecture, where only a portion of the network is activated for each computation. The model has approximately 816 billion parameters, with about 42 billion active for each token, the unit of text processed by the model. Its context window, meaning the amount of information it can consider at once, reaches up to one million tokens.
Meta trained it to read a codebase, write and debug code, and call tools. According to Hugging Face, it is optimized for real-world developer use, such as multi-file refactoring or long debugging sessions. It can also plan, act, observe, and correct its own errors on long tasks, including navigating the web with images.
A Large-Scale Architecture
Each MoE layer has 256 experts, and a router selects 8 for each token. These experts account for about 97% of the model’s parameters. All remain loaded in memory, so deploying it requires the memory of an 816-billion-parameter model, while its computational speed is that of a 42-billion-parameter model. The BF16 checkpoint weighs about 1.5 TB in safetensors format. The NVFP4 version, stored in 4-bit precision, is intended for NVIDIA Blackwell GPUs, with a precision loss deemed negligible.
For images and video, a 1.8-billion-parameter perception encoder transforms images into tokens. The audio module produces one token every 80 milliseconds, or 12.5 tokens per second. By default, videos are sampled at 2 frames per second, up to a limit of 96 frames.
Deployment and Fine-Tuning
The model is accessible via Transformers, llama.cpp, vLLM, SGLang, and MLX. With llama.cpp, a Q2_K quantized version runs at 18 tokens per second on a 512 GB Mac M3 Ultra. Via oMLX, a 3-bit version occupies about 344 GB and reaches 21 tokens per second in generation, according to Hugging Face’s tests. For hosted usage, Hugging Face offers deployment on Inference Endpoints, with 8 NVIDIA RTX Pro 6000 98 GB GPUs for a 131,072-token window, or 8 NVIDIA H200 141 GB GPUs for the full one-million-token window.
Hugging Face also demonstrated fine-tuning with TRL, which trains LoRA adapters on the attention layers while the experts remain frozen. Training lasted 500 steps, about two hours on 64 H100 GPUs, at around 8,000 tokens per second. The average token accuracy increased from 72% to 84%, and the loss decreased from 1.26 to 0.57, a signal that training went well, according to the article.
The published demonstrations illustrate agentic uses. A coding agent generates and improves 3D objects in Three.js, feeding the model screenshots of its own renderings. Another agent controls a browser using only screenshots, without access to the page’s HTML code. A final one searches for products on Amazon based on a voice description.
