Transformer Models for Text Summarization: A Comparative Study of BART, BERT, and RoBERTa
Category: Computer Science > Computation and Language
Authors: Daisy Aptovska, Vinayak Elangovan
Submitted: June 2, 2026
Publication Reference: International Journal of Artificial Intelligence and Applications (IJAIA), Vol. 17, No. 3, May 2026
Abstract
Text summarization is the task of condensing a document into a shorter version while preserving its key information. Automatic text summarization (ATS), driven by advances in natural language processing (NLP), has evolved rapidly in recent years. ATS methods are typically categorized by input type (e.g., single-document or multi-document summarization) and by output type (extractive, abstractive, or hybrid). This article offers a focused review of modern summarization techniques, with an emphasis on transformer-based models and large language models (LLMs), specifically BERT, RoBERTa, and BART. We examine their architectures, pretraining strategies, and suitability for extractive and abstractive summarization tasks.
1. Introduction
Text summarization plays an increasingly vital role in managing information overload across domains such as news aggregation, scientific literature, and legal documents. With the advent of transformer architectures, the field has seen significant progress in both extractive and abstractive summarization capabilities. By 2026, transformer-based models and their large-scale variants have become the de facto standard for state-of-the-art summarization systems.
1.1 Motivation
Traditional summarization methods relied on statistical and graph-based techniques that often produced fragmented or less coherent summaries. The emergence of pretrained language models, particularly those based on the transformer architecture, has enabled more fluent and context-aware summarization. Understanding the strengths and limitations of prominent models like BERT, RoBERTa, and BART is crucial for researchers and practitioners selecting appropriate models for their summarization needs.
1.2 Scope
This paper provides a comparative analysis of three widely used transformer models in the context of text summarization. We focus on architectural differences, pretraining objectives, and empirical performance on extractive and abstractive summarization benchmarks. The discussion also highlights practical considerations for deploying these models in real-world applications.
2. Background
2.1 Text Summarization Types
Text summarization can be broadly classified into two main categories:
- Extractive summarization: Selects salient sentences or phrases directly from the source document and concatenates them to form a summary.
- Abstractive summarization: Generates new sentences that may paraphrase or rephrase the original content, allowing for more concise and coherent summaries.
Hybrid approaches combine both strategies to leverage their respective advantages.
2.2 Transformer Architecture
The transformer model, introduced by Vaswani et al. (2017), replaced recurrent and convolutional architectures with a fully attention-based mechanism. Its self-attention layers allow parallel processing and capture long-range dependencies effectively, making it well-suited for summarization tasks.
3. Model Architectures
3.1 BERT (Bidirectional Encoder Representations from Transformers)
BERT is an encoder-only transformer pretrained using masked language modeling and next-sentence prediction. It excels at understanding context but is not inherently designed for sequence generation. In summarization, BERT is typically used as a feature extractor for extractive systems, where sentence embeddings are scored to select the most informative sentences.
3.2 RoBERTa (Robustly Optimized BERT Approach)
RoBERTa builds on BERT with more extensive pretraining, dynamic masking, and larger training data. It removes the next-sentence prediction objective, resulting in improved performance on downstream tasks, including extractive summarization. RoBERTa's richer representations often yield better sentence selection quality compared to BERT.
3.3 BART (Bidirectional and Auto-Regressive Transformer)
BART is a denoising autoencoder that combines a bidirectional encoder with an autoregressive decoder. It is pretrained by corrupting text and learning to reconstruct the original, making it particularly effective for generation tasks. BART has become a dominant model for abstractive summarization, often achieving state-of-the-art results on benchmarks like CNN/DailyMail.
4. Comparative Analysis
4.1 Architectural Suitability
- BERT/RoBERTa: Encoder-only models are well-suited for extractive summarization, where understanding context and relevance is key.
- BART: With its encoder-decoder structure, BART is naturally adapted for abstractive summarization, enabling fluent and diverse phrasing.
4.2 Pretraining Strategies
The pretraining objective significantly influences downstream performance. BERT's masked language modeling and next-sentence prediction provide strong contextualized embeddings, while RoBERTa's refined training on more data yields even better representations. BART's denoising objective, which includes text infilling and sentence permutation, equips it to handle noisy inputs and generate coherent summaries.
4.3 Empirical Performance
In recent benchmark evaluations (e.g., CNN/DailyMail, XSum, and arXiv), BART consistently outperforms BERT and RoBERTa on abstractive summarization metrics like ROUGE and BERTScore. For extractive tasks, RoBERTa often surpasses BERT due to its robust pretraining. However, the choice of model also depends on computational resources and latency requirements.
5. Future Directions
As of 2026, large language models (LLMs) such as GPT-4 and Claude are increasingly used for summarization via prompting or fine-tuning. These models offer flexibility and high quality but come with significant computational costs. Emerging research focuses on efficient fine-tuning, distillation, and retrieval-augmented summarization to balance performance and resource constraints.
6. Conclusion
Transformer-based models have revolutionized text summarization, offering powerful tools for both extractive and abstractive approaches. BERT and RoBERTa remain strong choices for extractive summarization, while BART excels in abstractive settings. As the field evolves, hybrid and LLM-based methods are likely to dominate, but understanding these foundational models remains essential for informed model selection and system design.
For the full paper, see the arXiv preprint: arXiv:2608.19200 [cs.CL].
via ArXiv CL+LG