Overview
A new paper by Volodymyr Ovcharov (arXiv:2609.15990 [cs.CL]) investigates a well-known but poorly understood phenomenon: few-shot prompting sometimes degrades language model performance rather than improving it. Published on 23 May 2026, the study combines behavioral evaluation with representational analysis across 12 open-weight models, two Ukrainian-language tasks, and two architectures.
The Behavioral Picture: Task-Dependent Effects
Across news classification and legal case outcome prediction, the effect of few-shot prompting is strongly task-dependent. The same models that gain +24 percentage points on news classification show only +3.4 pp on legal text โ and two models actually degrade when demonstrations are added. This asymmetry motivates the paper's central question: what does few-shot prompting actually change inside a model?
The Measurement Problem: Prompt Length as a Confound
Prior work measures how much hidden states shift between zero-shot and few-shot modes. But few-shot prompts are substantially longer, and that length difference alone perturbs representations โ inflating any naive "shift" metric.
The paper proposes a clean fix: replace demonstrations with length-matched random text to estimate the shift attributable purely to prompt length, then subtract it. The resulting metric, content delta, isolates how much the model's representations change because of what the demonstrations say, not how long they are.
Key Finding: Content, Not Raw Shift, Predicts Benefit
Correcting for length flips the interpretation entirely:
- Raw representational shift does not predict whether few-shot prompting helps or hurts (r = 0.20).
- Content delta does (rho = +0.65, p = 0.043).
- Models that restructure representations more from demonstration content benefit more.
This is the opposite of the intuitive "distortion" explanation, in which greater representational change would signal harm. Instead, meaningful restructuring appears to be a signature of effective in-context learning.
Causal Confirmation
Masking demonstrations in Llama 3.3 70B confirms the finding causally: accuracy recovers above the zero-shot baseline, consistent with the content delta account.
Paper Details
- Title: Few-Shot Degradation Is Not What It Seems: Behavioral Evidence, Representation Analysis, and a Random-Text Control Across 12 Models, 2 Tasks, and 2 Architectures
- Author: Volodymyr Ovcharov
- Submitted: 23 May 2026
- Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
- ACM Class: I.2.7
- Length: 12 pages, 6 figures, 4 tables
- Data: Hugging Face dataset
- Cite as: arXiv:2609.15990 [cs.CL] โ https://doi.org/10.48550/arXiv.2609.15990
Why It Matters in 2026
As few-shot and in-context learning remain default tools in deployed LLM pipelines โ and as low-resource languages like Ukrainian gain more attention in multilingual evaluation โ distinguishing genuine in-context learning from prompt-length artifacts is increasingly important. The content delta metric offers a lightweight, model-agnostic diagnostic for predicting when demonstrations will help, and it reframes "degradation" as a signal worth examining rather than a nuisance to be avoided.
via ArXiv CL+LG