Hugging Face proposes Model Integration Contracts for Transformers: tiny real-architecture fixtures on CPU that would have caught a UMT5 bug that sat in main for almost two months
A way to test hundreds of model types on every PR without GPUs, by pinning shrunk checkpoints and human-reviewed baselines.
// Key points
- Each contract has a fixture (a shrunk model keeping every layer type), a runnable usage path, and human-reviewed baselines
- Transformers supports over 750 model types and runs 88,000+ CPU tests on main plus 200,000+ GPU tests daily
- A July 2026 SDPA change let UMT5's decoder see future tokens in training; the 66 MB fixture (vs 1,227 MB) flagged it in replay, and a T5 contract ran in 13.4 seconds on CPU
- As of November 1 the job reports failures without blocking PRs; coverage is still partial
Builder's takeAnyone self-hosting open models through Transformers should note how long a silent training bug can live upstream. For the Smart Interview fine-tunes I'd pin library versions and keep a tiny fixture plus recorded outputs of our own, so an upgrade fails in CI instead of in a training run.
// Background · from #Open models
Full timeline →- Nov 5 IBM Research's Palette builds slide decks with a 20B open-weight model and one LoRA adapter, scoring 59.7 on PresentBench versus 60.8 for NotebookLM
- Nov 4 Amazon Bedrock adds in-country inference in India for Kimi K3, keeping processing within the Mumbai and Hyderabad Regions
- Nov 3 AWS distills a GLM-5.3 shopping agent into Qwen3.5-4B: held-out success rises from 0.297 to 0.541, level with the teacher