On July 30, 2026, Transformers enabled PyTorch’s scaled dot-product attention (SDPA) for T5. This side effect allowed the UMT5 decoder to look at future tokens, i.e., the response it was supposed to predict, during training. Nothing crashed, and the bug remained in the main branch for nearly two months. Hugging Face introduces Model Integration Contracts (MIC), a test format based on small models, which allowed it to be spotted during a replay of commits.
Transformers supports over 750 model types, and its test suite is extensive: over 88,000 CPU tests on the main branch, and over 200,000 GPU tests in a daily run. Integration tests load real checkpoints and use the model API on real hardware. Running them on every pull request would monopolize a large number of GPUs all day, which is not realistic. A regression can therefore reach the main branch before being detected by a scheduled run.
Shrink the model rather than test the large one
The approach involves creating a fixture, i.e., a small model built from the configuration of the real checkpoint, with fewer, narrower layers, but of all the types present in the original model. Its tokenizer and processor files remain unchanged. This fixture is published once and then reused, and fast tests run on it on a simple CPU. In the team’s CPU study, the experimental UMT5 fixture weighed 66 MB, compared to 1,227 MB for the source checkpoint. This method also applies to very large models, some of which don’t fit on standard GPUs but whose fixture runs on CPU.
The tests go beyond simple loading and an inference pass. The UMT5 bug would have passed this stage without difficulty. Fixtures do not replace full integration tests, but they offer a chance to detect errors before they reach the main branch.
A MIC consists of three elements: the fixture, a usage path (an executable example, such as text generation or a training step), and baselines, the outputs recorded when running that example and validated by a human. The contract is respected if execution on the fixture always produces these reference outputs.
Feedback on the UMT5 bug
The team replayed frozen MICs over five months of Transformers commits. The UMT5 fixture flagged the regression from its introduction on July 30, 2026, until its correction on September 28. The label log probabilities differed from the validated reference by 1.22, and the training loss by about 0.20. One detail mattered: the contract used SDPA, the default attention path. Running the “eager” attention variant alone did not reveal the bug. During a demonstration on a test branch of Transformers’ CI, with the fix reverted, the verification failed with the same discrepancies, and the T5 runner step took 13.4 seconds on CPU, excluding setup.
Generate contracts with agents
A command-line tool, tmic, creates the contract and builds the fixture. For a new model, an LLM writes the usage code and manifest, then CPU and GPU runs measure candidate baselines. According to the team, the LLM writes but does not approve: any change in expectations requires human validation. The model reduction report for Llama 4 Scout illustrates this method: 108.6 billion parameters become 44.7 million, the vision part goes from 34 to 2 layers and the language part from 48 to 4.
For test models, the team compared random weights to weights trained on a few examples. In the weight initialization study, random Whisper fixtures detected 18 injected faults out of 24, versus 24 out of 24 for trained fixtures. These faults were synthetic, and both versions missed an assisted decoding bug revealed by the real Whisper checkpoint.
As of November 1, 2026, the MIC job in Transformers’ CI reports failures without blocking pull requests. The question of making it blocking remains open, due to a lack of metrics on the number of real bugs detected and false alarms.
