Auditing Preference Biases and Fine-Tuning Language Models with Direct Preference Optimization on Anthropic HH

Introduction

In this tutorial, we design an end-to-end preference-learning workflow using the Anthropic HH-RLHF dataset and Direct Preference Optimization (DPO). We begin by preparing a robust Colab environment, loading and parsing chosen–rejected response pairs, and auditing the dataset for structural and length-based preference biases. We then run lexical shortcut diagnostics to determine whether surface-level linguistic patterns can separate preferred from rejected responses, prepare conversational data with tokenizer-aware length filtering, and construct a version-robust DPO training pipeline with TRL and optional LoRA adaptation. Finally, we fine-tune a Qwen2.5-0.5B-Instruct model, evaluate reward accuracy and training behavior, analyze performance across individual HH-RLHF subsets, inspect potential length bias, generate sample responses, and save the resulting policy for further experimentation.

Setting Up the Environment

To ensure a smooth execution, we install the required dependencies and remove any conflicting packages that may be present in the default Colab environment.

import dataclasses
import importlib.util
import inspect
import os
import re
import subprocess
import sys
import warnings
warnings.filterwarnings("ignore", category=UserWarning)

REQUIRED = ["trl>=0.12", "transformers>=4.45", "accelerate", "datasets", "peft", "scikit-learn"]

def ensure_deps():
    """Install in ONE pip call so the resolver picks a mutually compatible set."""
    try:
        import trl
        import transformers
        return False
    except ImportError:
        print("Installing dependencies...")
        subprocess.check_call([sys.executable, "-m", "pip", "install", "-q", "-U", *REQUIRED])
        return True

def drop_broken_torchao():
    """
    Colab ships torchao 0.10.0; peft demands >0.16 and raises rather than skipping.
    Nothing here uses torchao, so removing it is safer than upgrading (an upgrade can
    drag in a torch build that does not match this runtime).
    """
    if importlib.util.find_spec("torchao") is None:
        return False
    try:
        from peft.import_utils import is_torchao_available
        is_torchao_available()
        return False
    except ImportError:
        print("Removing incompatible torchao (unused, but peft raises on it)...")
        subprocess.call([sys.executable, "-m", "pip", "uninstall", "-y", "-q", "torchao"])
        return True
    except Exception:
        return False

_installed = ensure_deps()
_removed = drop_broken_torchao() if not _installed else False
# Rest of the script continues here...

Loading and Auditing the Dataset

We load the Anthropic HH-RLHF dataset, which contains pairs of responses where one is chosen as preferred and the other is rejected. We parse these pairs into a structured format and conduct a preliminary audit to detect any biases.

# Assuming the dataset is loaded via datasets library
from datasets import load_dataset

dataset = load_dataset("Anthropic/hh-rlhf")
# Parse chosen-rejected pairs and inspect
# ...

Length-Based Bias Analysis

We examine whether there is a systematic difference in response lengths between chosen and rejected outputs, which could indicate a bias toward longer or shorter responses.

Lexical Shortcut Diagnostics

We run a diagnostic to check if simple surface-level features (e.g., word frequencies, punctuation) can predict the preferred response, which would suggest the model might rely on shortcuts rather than deeper understanding.

Preparing Training Data

We filter the dataset based on tokenizer-aware length constraints to ensure that all sequences fit within the model's context window during training.

Training with DPO and LoRA

We construct a DPO training pipeline using the TRL library, with optional LoRA adaptation for parameter-efficient fine-tuning. The model is Qwen2.5-0.5B-Instruct.

from trl import DPOTrainer
from peft import LoraConfig

# Define training arguments, model, and tokenizer
# ...

Evaluation and Analysis

After training, we evaluate the model's reward accuracy, analyze performance across different HH-RLHF subsets, and inspect for any introduced length bias. We also generate sample responses to qualitatively assess the fine-tuned model.

Saving the Final Policy

Finally, we save the trained policy for future use and experimentation.

Conclusion

This tutorial provides a comprehensive workflow for auditing and fine-tuning language models with DPO, offering insights into preference biases and practical steps for implementation. As of 2026, these techniques remain critical for aligning AI systems with human values, and the approach here can be adapted to newer models and datasets.

via MarkTechPost