In this article, we will explore the Mixture-of-Experts (MoE) and discuss the idea behind the gating mechanism used by the Sparse MoE.