CausalGate: Causal Importance Distillation for Transformer Module Pruning

Authors: Kiran Nair, Smriti Regmi, Rodrigue Rizk

Subject: Machine Learning (cs.LG); also appears in Computation and Language (cs.CL), Computer Vision and Pattern Recognition (cs.CV), and Machine Learning (stat.ML)

Submitted: 21 July 2026 (v1)

Abstract

Existing adaptive inference methods for Large Language Models (LLMs) rely on observational heuristics—such as hidden-state similarity or activation magnitudes—to drop redundant modules. However, these correlation-based metrics often fail to capture the subtle, non-linear structural computations that are critical for maintaining semantic accuracy.

In response, we introduce CausalGate, an intervention-guided framework for compute-efficient transformer inference. During a calibration phase, CausalGate isolates individual Attention and MLP sub-layers, zeros out their outputs, and measures the exact semantic damage by computing the Kullback–Leibler divergence of the final logit distribution. To eliminate runtime routing overhead, this structural importance hierarchy is distilled into a global set of static, lightweight scalar gates using an Exponential Moving Average (EMA) smoothing objective combined with a differentiable pairwise ranking loss.

Evaluated on TinyLlama-1.1B, Qwen2.5-3B, and Llama-3.1-8B across language modeling and commonsense reasoning benchmarks, CausalGate consistently outperforms prominent dynamic routing and layer-skipping baselines. It translates theoretical compute savings into concrete hardware latency reductions with zero operational overhead, offering a practical solution for efficient LLM deployment in the evolving 2026 AI landscape.

Keywords

Causal inference · Transformer pruning · Module gating · Knowledge distillation · LLM optimization · Adaptive inference · Layer skipping · Machine learning · 2026

Note: This paper is currently in-review at a conference.

via ArXiv LG