Assembly of Experts: Linear-time construction of ultra-large language model variants

Requiring 10^13-10^15 FLOPs to calculate one 8 bit weight in an LLM is inefficient and not beautiful. Therefore, we developed the new “Assembly-of-Experts” construction method. It allows to create meaningful child variants of existing Mixture-of-Experts parent models in linear time. For this, parts of the parent models are recombined, guided by structural and semantic features.

Using AoE, we constructed the DeepSeek R1T “Chimera”, a 671B open-weights hybrid model combining DeepSeek’s V3-0324 and R1 model variants. The child is achieving about R1-level intelligence, while using about 40% fewer output tokens, close to V3. Constructed without any fine-tuning or distillation, the Chimera exhibits surprisingly compact, orderly reasoning compared to its parent models. Moreover, it showed behaviour outside of the distribution, i.e. it is not just an interpolation of the parent models.

Community response has been encouraging. The model has 8,200 downloads on Hugging Face and, at peak at the end of May, reached 1.35 billion processed tokens per day on OpenRouter and almost 5 billion tokens per day on chutes.ai.

Since then, we have developed the second generation, DeepSeek-TNG R1T2 Chimera. It is a Tri-Mind that also leverages the new R1-0528, and uses a more refined construction method. It appears to be 20% faster than R1, while being almost always more intelligent. It is not quite as intelligent as R1-0528, but typically more than twices as fast, and also with a beautiful chain of thought like R1T. Since R1T2s release on OpenRouter on July 7th, it reached 1 billion tokens per day already on its first full day of operation. It broke 2 B/d on July 17th and grew to become the 5th most popular LLM on Chutes on July 19th. It currently runs on 30 Chutes instances of 8xH200 each, totalling 240 H200 GPUs.