Moe

Introduction TL;DR: Meta released Llama 4 Scout and Llama 4 Maverick on 2025-04-05. Scout targets ultra-long context (10M tokens) with 17B activated / 109B total params, while Maverick offers 1M tokens with 17B activated / 400B total params. Both are natively multimodal (text+image inputs) and use a Mixture-of-Experts (MoE) design. Benchmarks shared by Hugging Face show strong gains vs earlier Llama generations, but leaderboard integrity and “variant mismatch” issues mean you should validate on your own workloads. The “Llama 4 Community License” includes practical obligations and a major threshold clause (700M MAU) you must review before production use. In this post, we’ll focus on what’s verifiable from public artifacts (model cards, the license text, and release notes), then translate it into an engineer-friendly decision checklist. ...

Introduction TL;DR MoE activates only a small subset of expert FFNs per token (conditional computation), scaling total capacity without proportional per-token compute. In Transformers, the mainstream pattern is replacing the dense FFN/MLP with an MoE FFN (router + experts). Production bottlenecks often come from routing imbalance, capacity overflow (drops), all-to-all communication, and memory bandwidth; serving requires observability and cluster tuning. Why it matters: MoE is a combined model + distributed-systems problem, not just a modeling trick. ...

Introduction Alibaba Cloud has recently expanded its Qwen family of large language models (LLMs) with the release of the new Qwen3-VL series, which includes the highly efficient Qwen3-VL-30B-A3B. This model is a significant development in the open-source AI landscape, combining powerful multimodal capabilities—processing text, images, and video—with a resource-efficient architecture. The Qwen3-VL-30B-A3B leverages the Mixture-of-Experts (MoE) architecture, boasting approximately 30.5 billion total parameters while activating only about 3.3 billion during inference, a key feature for practical, cost-effective deployment. Released as part of the Qwen3-VL rollout in late 2025 (e.g., Qwen3-VL-30B-A3B-Instruct in October 2025), it offers developers a commercially viable, high-performance solution licensed under Apache 2.0. ...