Margin-Regularized Structured Semantic Alignment for Brain-Language Correspondence
Jiaqi Wang, Huawen Hu, Shu Zhang
Submitted: 17 August 2026
Abstract
The rapid advancement of large language models (LLMs) has significantly accelerated progress in brain-language decoding. However, a fundamental question remains: does the decoded content genuinely reflect the underlying neural representations, or is it largely reconstructed by the language model itself? This ambiguity limits the interpretability of decoding results and hinders the investigation of intrinsic brain-language correspondence.
To address this challenge, we propose MD-SigLIP, a margin-regularized structured semantic alignment framework. Our approach directly aligns brain embeddings with text embeddings in a shared semantic space, enabling retrieval-based decoding. This formulation allows for explicit modeling of the correspondence between neural representations and language semantics.
Building upon duplicate-aware sigmoid contrastive learning, we introduce a listwise margin-regularized term that enforces structured ranking constraints between positive semantic clusters and negative samples. By simultaneously modeling multi-positive semantic structure and margin-based ordering, the method captures the manifold organization of language embeddings as reflected in neural signals.
Experiments demonstrate state-of-the-art retrieval performance under both full-vocabulary and subset evaluation settings, validating the effectiveness of the proposed framework for studying brain-language correspondence.
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as: arXiv:2608.16975 [cs.CL]
DOI: https://doi.org/10.48550/arXiv.2608.16975
via ArXiv CL+LG