Infrastructure
MoE (Mixture of Experts)
A model architecture that splits its parameters into specialized “expert” sub-networks and activates only a few of them per request, instead of the whole model.
A dense model uses all of its parameters for every single token it processes. A Mixture-of-Experts (MoE) model instead contains many smaller expert sub-networks and a “router” that picks a small subset — often just a handful out of dozens — to handle each token.
The appeal is efficiency: MoE lets a model have a very large total parameter count, and the capability that implies, while only paying the inference compute cost of a much smaller active model. Several frontier-scale models in 2025-2026 use MoE designs for exactly this reason.