ICARE: Cross-Domain Text Classification with Incremental Class

ICARE: Cross-Domain Text Classification with Incremental Class-Aware Representation and Distillation Learning


Authors: Son Duong, Truong X. Tran, Hai-Anh Tran

Submitted: 11 November 2026

Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)

arXiv: 2611.24959 [cs.CL]

DOI: https://doi.org/10.48550/arXiv.2611.24959

Code: https://github.com/CongSon01/ICARE_NLP.git


Abstract


The rise of powerful, fine-tuned large language models (LLMs) has made robust natural language inference (NLI) systems increasingly important. Ideally, these NLI systems are trained across a variety of tasks spanning diverse concepts such as sentiment analysis, news classification, and coreference resolution. However, this continual, multi-task approach can lead to catastrophic forgetting, in which previously learned tasks are lost—particularly in domains that lack specific labels. Rehearsing past labeled examples can further exacerbate the problem by causing overfitting and hindering the system's ability to classify new categories.


To address these challenges, this research proposes ICARE, a novel framework built on two key components: an instance selection module and a knowledge distillation method. The first is based on representation learning and carefully selects the training examples presented to the NLI system. The second—knowledge distillation with feature- and logit-level transfer—facilitates knowledge transfer between tasks. It leverages pruning techniques and optimized objective functions to transfer both essential feature-level and decision-making (logit-level) knowledge, mitigating catastrophic forgetting while adapting to new patterns.


Extensive evaluations on benchmark text classification datasets demonstrate ICARE's effectiveness. The framework achieves a significant improvement in target accuracy and a reduction in the "forgetting metric." These results confirm ICARE's ability to preserve and leverage learned knowledge across diverse tasks, ultimately enhancing the adaptability and performance of NLI systems.


Key Contributions


  • Instance selection via representation learning: A module that meticulously selects which training examples to present to the NLI system, improving the quality of rehearsal data.
  • Feature- and logit-level knowledge distillation: A transfer mechanism that uses pruning techniques and optimized objective functions to preserve essential feature-level and decision-making knowledge across tasks.
  • Mitigated catastrophic forgetting: The framework adapts to new patterns while retaining prior knowledge, addressing both forgetting and overfitting in continual, multi-task settings.

Results


Benchmark evaluations on text classification datasets show that ICARE delivers a significant improvement in target accuracy alongside a reduction in the forgetting metric, underscoring its ability to preserve and leverage learned knowledge across diverse tasks.


Availability


Source code is available at: https://github.com/CongSon01/ICARE_NLP.git


Citation


arXiv:2611.24959 [cs.CL]
https://doi.org/10.48550/arXiv.2611.24959

via ArXiv CL+LG

Related