Knowledge Injection in MoE? Exploring Expert-Aware Contrast Decoding for Mitigating LLM Hallucinations
Abstract
Existing methods for mitigating hallucinations in large language models (LLMs)—including prompt engineering and model optimization—either struggle to modify the model's internal knowledge or exhibit poor cross-domain generalization. Contrastive decoding has emerged as a promising approach, leveraging layer-wise differences within LLMs to reduce hallucinations. However, prior research has focused primarily on transformer-based architectures such as GPT, leaving other effective frameworks, such as Mixture-of-Experts (MoE) models, underexplored.
Since MoE modifies the traditional transformer architecture, we conduct empirical studies to investigate whether analogous layer-wise differences exist in MoE models. Our findings reveal that such differences are absent in MoE models with shared experts. However, across different MoE architectures, higher layers display distinct expert activation patterns between factual and non-factual outputs.
Building on these insights, we propose EAACD (Expert-Aware Adaptive Contrast Decoding), a novel decoding strategy that leverages expert differences in the higher layers of MoE models to mitigate hallucinations on question-answering tasks. EAACD partitions high-layer experts into one high-reliability group and several lower-reliability groups, based on their confidence and consistency. It then contrasts the predictions of the high-reliability group against each lower-reliability group to calibrate the model's original outputs. To strengthen this contrast, EAACD amplifies hallucinations from lower-reliability experts via attention and masking mechanisms, providing stronger negative references. EAACD consistently outperforms all baseline methods across four benchmark datasets.
Keywords: Mixture-of-Experts, MoE, hallucination mitigation, contrastive decoding, expert-aware decoding, large language models, knowledge injection, expert activation patterns
Submitted on 8 May 2026. Accepted by ACL2.
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as: arXiv:2607.20426 [cs.CL]
via ArXiv CL