DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts

Yuchen Feng, Bowen Shen, Naibin Gu, Jiaxuan Zhao, Peng Fu, Zheng Lin, Weiping Wang

Pdf / Code

• Existing MoE reconstruction methods often overlook the diversity among experts, leading to potential redundancy. We present a Diversity-Enhanced reconstruction method named DIVE, based on the observation that a dense LLM exhibits notable diversity after being pruned on different calibration datasets. The recipe of DIVE includes domain affinity mining, pruning-based expert reconstruction, and efficient retraining, achieving high efficiency with minimal accuracy loss.

Blink: Dynamic Visual Token Resolution for Enhanced Multimodal Understanding

Yuchen Feng, Zhenyu Zhang, Naibin Gu, Yilong Chen, Peng Fu, Zheng Lin, Shuohuan Wang, Yu Sun, Hua Wu, Weiping Wang, Haifeng Wang

Pdf / Code

• Our pilot analysis shows that MLLMs shift their focus across layers and that allocating more computation to salient tokens improves visual perception. Building on this insight, we propose Blink, a dynamic visual token resolution framework that performs this human-inspired process within a single forward pass. It estimates token saliency from attention maps, expands important tokens via a plug-and-play token super-resolutions module, and drops them in the next layer once they lose focus.

CBP-Tuning: Efficient Local Customization for Black-box Large Language Models

Jiaxuan Zhao, Naibin Gu*, Yuchen Feng, Xiyu Liu, Peng Fu, Zheng Lin, Weiping Wang

Pdf / Code

Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts

Naibin Gu, Zhenyu Zhang*, Yuchen Feng, Yilong Chen, Peng Fu, Zheng Lin, Shuohuan Wang, Yu Sun, Hua Wu, Weiping Wang, Haifeng Wang

Pdf / Code