https://nlp.unibo.it/tag/mixture-of-experts/Mixture of ExpertsHugo Blox Builder (https://hugoblox.com)en-usMon, 02 Mar 2026 00:00:00 +0000https://nlp.unibo.it/media/logo.svghttps://nlp.unibo.it/tag/mixture-of-experts/https://nlp.unibo.it/proposals/interpretability/mixture/Mon, 02 Mar 2026 00:00:00 +0000https://nlp.unibo.it/proposals/interpretability/mixture/<h2 id="context">Context</h2> <p>Selective rationalization trains a model to explain each prediction with highlights of its input text. These highlights, called rationales, are a part of the input that a user can read and that faithfully shows how the model classifies. In the Select-then-Predict Pipeline (SPP), a generator selects the rationale and a predictor classifies it. Training an SPP can get stuck in poor solutions, called local minima, because of interlocking: a suboptimal interplay between generator and predictor.</p> <h2 id="objective">Objective</h2> <p>A mixture of experts (MoE) trains several models, called experts, on the same data, and each expert specializes in a subset of it. This proposal studies whether an MoE model for selective rationalization can address interlocking.</p> <figure class="lt-viz lt-svg" style="--lt-svg-min: 640px"> <svg viewBox="0 0 800 336" role="img" aria-labelledby="mixture-title" xmlns="http://www.w3.org/2000/svg"> <defs><marker id="mixture-arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="lt-s-head" d="M0 0 L10 5 L0 10 z"/></marker></defs> <title id="mixture-title">Two ideas side by side. Left, the Select-then-Predict Pipeline: a generator selects the rationale and a predictor classifies it, and their interplay can cause interlocking. Right, a mixture of experts: three experts train on the same data, and each specializes in a subset. Both lead to the research question: can a mixture of experts model for selective rationalization address interlocking?</title> <text class="lt-s-sub" x="40" y="30">SELECT-THEN-PREDICT PIPELINE</text> <rect class="lt-s-box lt-s-sel" x="60" y="44" width="280" height="56" rx="12"/> <text class="lt-s-txt lt-s-strong" x="200" y="68">Generator</text> <text class="lt-s-txt lt-s-small" x="200" y="88">selects the rationale</text> <path class="lt-s-arrow" marker-start="url(#mixture-arrow)" marker-end="url(#mixture-arrow)" d="M200 104 V150"/> <text class="lt-s-sub" x="216" y="131">INTERLOCKING</text> <rect class="lt-s-box lt-s-pred" x="60" y="154" width="280" height="56" rx="12"/> <text class="lt-s-txt lt-s-strong" x="200" y="178">Predictor</text> <text class="lt-s-txt lt-s-small" x="200" y="198">classifies the rationale</text> <text class="lt-s-sub" x="420" y="30">MIXTURE OF EXPERTS</text> <rect class="lt-s-tok" x="440" y="44" width="300" height="44" rx="12"/> <text class="lt-s-txt lt-s-strong" x="590" y="71">Same data</text> <path class="lt-s-arrow" marker-end="url(#mixture-arrow)" d="M590 92 V126"/> <path class="lt-s-arrow" marker-end="url(#mixture-arrow)" d="M590 106 H486 V126"/> <path class="lt-s-arrow" marker-end="url(#mixture-arrow)" d="M590 106 H694 V126"/> <rect class="lt-s-box lt-s-sel" x="438" y="130" width="96" height="56" rx="12"/> <text class="lt-s-txt lt-s-strong" x="486" y="154">Expert 1</text> <text class="lt-s-txt lt-s-small" x="486" y="174">subset 1</text> <rect class="lt-s-box lt-s-sel" x="542" y="130" width="96" height="56" rx="12"/> <text class="lt-s-txt lt-s-strong" x="590" y="154">Expert 2</text> <text class="lt-s-txt lt-s-small" x="590" y="174">subset 2</text> <rect class="lt-s-box lt-s-sel" x="646" y="130" width="96" height="56" rx="12"/> <text class="lt-s-txt lt-s-strong" x="694" y="154">Expert 3</text> <text class="lt-s-txt lt-s-small" x="694" y="174">subset 3</text> <text class="lt-s-txt lt-s-small" x="590" y="210">each expert specializes in a subset of the data</text> <path class="lt-s-arrow" marker-end="url(#mixture-arrow)" d="M200 214 V252"/> <path class="lt-s-arrow" marker-end="url(#mixture-arrow)" d="M590 220 V252"/> <rect class="lt-s-tok" x="60" y="256" width="680" height="60" rx="12"/> <text class="lt-s-txt lt-s-strong" x="400" y="281">Research question</text> <text class="lt-s-txt lt-s-small" x="400" y="302">Can an MoE model for selective rationalization address interlocking?</text> </svg> <figcaption>The two ideas this proposal brings together. Left, the SPP, where the interplay between generator and predictor can cause interlocking. Right, an MoE, where three experts stand for several.</figcaption> </figure>