New MoDES Framework Skips 88% of Experts in Multimodal Models While Retaining 97% Performance

A new research framework, Multimodal Dynamic Expert Skipping (MoDES), significantly enhances the inference efficiency of multimodal large models. MoDES, developed by a research team from Hong Kong University of Science and Technology, Beihang University, and Peking University, intelligently bypasses 88% of redundant experts without additional training, yet maintains 97% of the original model performance. This approach accelerates inference speed by a factor of two.

Visual representation of a Mixture-of-Experts (MoE) neural network, showing active and inactive expert modules.

The Mixture-of-Experts (MoE) architecture has become a common choice for large multimodal models as they scale to handle higher-resolution images, longer video sequences, and more complex cross-modal tasks. MoE aims to reduce computational overhead by activating only a subset of expert networks. However, even with MoE, the inference cost remains high because each token interacts with multiple experts, leading to substantial computation on experts that are not critical. This redundancy is particularly amplified in video understanding and long-context scenarios, creating an inference bottleneck.

Previous attempts to dynamically skip redundant experts during inference in pure text large language models (LLMs) have shown some success. However, applying these methods directly to multimodal models often results in significant performance degradation, with higher skipping ratios leading to severe performance drops or even model collapse.

MoDES systematically analyzes the reasons for these failures in multimodal MoE skipping. The framework, detailed in a paper accepted by CVPR, provides a training-free dynamic expert skipping mechanism for multimodal MoE.

On the Qwen3-VL-MoE-30B model, MoDES retained 97.33% of its original performance while skipping 88% of experts, challenging the long-held assumption that high-ratio expert skipping inevitably leads to unacceptable performance loss.

Addressing Multimodal Skipping Challenges

MoDES does not introduce new skipping rules but instead identifies two key reasons why text-model skipping methods fail in multimodal MoE:

First, the global contributions of experts in different layers to the final output are highly imbalanced. Existing skipping methods typically assess expert importance based solely on the routing probability of the current layer. However, the impact of experts in different layers on the final prediction distribution varies significantly. Experiments showed that reducing experts in shallow layers leads to more substantial performance degradation, while deep-layer experts have a relatively minor impact. This suggests that errors in shallow layers can amplify in subsequent layers, causing performance collapse. Therefore, expert importance is not just about local routing probability but also about its impact on the final output. Adopting a layer-agnostic rule risks skipping critical shallow-layer experts.

Second, there are notable differences in the behavior of text and visual tokens. Analysis of token representations before and after the Feed-Forward Network (FFN) revealed that text tokens exhibit significantly larger update magnitudes in FFN, while visual tokens are more orthogonal to expert weights, and experts have a relatively smaller impact on visual tokens. This implies that experts are more critical for text inference, whereas there is higher redundancy for visual tokens. Skipping strategies that do not differentiate between modalities may inadvertently remove experts crucial for text understanding, leading to performance degradation.

Conceptual image showing distinct processing of text and visual tokens in an AI model, highlighting modality differences.

These observations led to a core conclusion: expert importance in multimodal MoE must be both output-aware and modality-aware.

Output-Aware and Modality-Aware Dynamic Skipping

Based on these insights, MoDES developed an output-aware and modality-aware dynamic expert skipping mechanism.

The framework introduces a hierarchical global importance factor, derived from the original routing probability, to characterize the overall impact of experts in a given layer on the final output distribution. This factor is determined through offline calibration by comparing the model's output distribution before and after removing experts in that layer. The new expert importance score combines local routing probability with this global factor. This approach conserves shallow-layer experts more cautiously while allowing for more aggressive skipping of deep-layer experts, achieving output-aware skipping.

MoDES also incorporates a dual-modality threshold mechanism, setting distinct skipping thresholds for text and visual tokens. This modality-specific differentiation refines expert skipping decisions, preventing the accidental removal of critical experts.

To efficiently identify the optimal threshold combination, MoDES employs a frontier search algorithm. This algorithm leverages the monotonicity between performance and skipping ratio, reducing search complexity from O(N^2) to O(N) and shortening search time by approximately 45 times while maintaining consistent results.

Abstract visualization of a frontier search algorithm identifying optimal thresholds for AI model performance.

Experimental Validation and Efficiency

In experiments on Kimi-VL-A3B-Instruct, MoDES retained 96.25% of its original performance when skipping 83% of experts, whereas most existing methods showed an average performance drop exceeding 11%. On the larger Qwen3-VL-MoE-30B-A3B-Instruct, MoDES retained 97.33% of its original performance while skipping 88% of experts, outperforming MC-MoE (86.66% retention) and DiEP (85.30% retention). MoDES achieved optimal or near-optimal performance across 13 image and video understanding benchmarks.

These results indicate that high-ratio skipping is viable if the global contribution of experts to the final output and the behavioral differences of tokens across modalities are accurately modeled.

Inference tests on an H200 GPU showed that MoDES achieved approximately 2x acceleration during the Prefill stage and about 1.2x improvement in the Decoding stage. As a training-free method, MoDES introduces no additional computational overhead during inference, ensuring stable acceleration. Furthermore, MoDES is compatible with mixed-precision quantization, maintaining high performance under low-bit quantization and demonstrating that skipping and quantization can jointly reduce computational costs in multimodal MoE.

High-tech server rack with glowing GPU, symbolizing accelerated AI inference and computational efficiency.

MoDES's core contribution lies in its output-aware and modality-aware expert skipping mechanism. By explicitly modeling the global contribution of experts across layers and the update characteristics of tokens in different modalities, MoDES demonstrates that significant expert skipping (over 80%) can be achieved without compromising model performance, provided the skipping is sufficiently intelligent. This approach offers a robust and practical path for optimizing the inference efficiency of large multimodal models as their scale continues to expand.