(a) Standard MoE architecture. (b) LatentMoE architecture. Figure 1 : Standard MoE vs. LatentMoE architectures. In LatentMoE, tokens are projected from the model hidden dimension d d into a smaller latent dimension ℓ \ell for expert routing and computation, which reduces routed parameter loads and all - to - all traffic by a factor of d / ℓ d/\ell . We use this efficiency to increase the total number of experts and the top-k active experts per token by the same factor d / ℓ d/\ell , which improves the accuracy of the model while keeping overall inference cost approximately constant.
