Overview
This tutorial presents a complete post-training pipeline for a compact instruction-tuned language model, built on AllenAI’s Open Instruct framework. The workflow covers three key training stages: Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Learning with Verifiable Rewards (RLVR) using GRPO. The original multi-GPU Tulu 3 stack is adapted to fit within a 16 GB runtime environment, making it suitable for single-GPU setups like Google Colab.
The pipeline selectively loads native loss and utility functions from the Open Instruct repository, configures LoRA adapters for parameter-efficient training, prepares GSM8K data for each stage, and uses deterministic verifiers to evaluate model-generated mathematical answers. Distributed components such as vLLM, Ray actors, DeepSpeed, and asynchronous rollout queues are replaced with lightweight Hugging Face and PyTorch implementations to ensure compatibility with constrained hardware.
Setting Up The Environment
The first step initializes the runtime with required dependencies and clones the Open Instruct repository. This setup is designed to work across both local and cloud environments, automatically detecting the appropriate directory paths.
import os, sys, subprocess, textwrap, json, math, random, re, ast, types, dataclasses, gc, contextlib
REPO_URL = "https://github.com/allenai/open-instruct.git"
REPO_DIR = "/content/open-instruct" if os.path.isdir("/content") else "./open-instruct"
PIP_PKGS = [
"peft", "accelerate",
"ray", "wandb", "beaker-py",
"langdetect==1.0.9", "immutabledict==1.2.0", "nltk",
"absl-py", "sympy", "antlr4-python3-runtime==4.11",
"tiktoken",
]
def sh(*args):
print("$", " ".join(args))
subprocess.run(args, check=False)
def setup():
sh(sys.executable, "-m", "pip", "install", "-q", *PIP_PKGS)
if not os.path.isdir(REPO_DIR):
sh("git", "clone", "--depth", "1", REPO_URL, REPO_DIR)
if REPO_DIR not in sys.path:
sys.path.insert(0, REPO_DIR)
os.environ.setdefault("WANDB_MODE", "disabled")
os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
os.environ.setdefault("RAY_DISABLE_IMPORT_WARNING", "1")
setup()
import numpy as np
import torch
import torch.nn.functional as F
from torch.utils.data import DataLoader
from datasets import load_dataset, Dataset
from transformers import AutoModelForCausalLM, DataCollatorForSeq2Seq, get_cosine_schedule_with_warmup
from peft import LoraConfig, get_peft_model
Configuring Hardware And Precision
The pipeline detects the available compute device and configures mixed precision accordingly. It uses bfloat16 on newer NVIDIA GPUs (compute capability 8.0+) and falls back to float16 with gradient scaling for older hardware.
DEV = "cuda" if torch.cuda.is_available() else "cpu"
try:
_bf16 = DEV == "cuda" and torch.cuda.is_bf16_supported(including_emulation=False)
except TypeError:
_bf16 = DEV == "cuda" and torch.cuda.get_device_properties(0).major >= 8
AMP_DTYPE = torch.bfloat16 if _bf16 else torch.float16
USE_SCALER = AMP_DTYPE is torch.float16
print(f"device={DEV} autocast dtype={AMP_DTYPE} gpu={torch.cuda.get_device_name(0) if DEV=='cuda' else '-'}")
Loading Open Instruct Utilities
The next step selectively loads native functions from the Open Instruct source code, preserving the core optimization logic while avoiding heavy distributed dependencies. The oi_load helper function parses the Python source and extracts only the specified functions and classes.
def oi_load(relpath, names, ns=None):
src = open(os.path.join(REPO_DIR, relpath)).read()
tree = ast.parse(src)
found = {n.name: n for n in tree.body
if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)) and n.name in names}
missing = ...
Training Pipeline Stages
Stage 1: Supervised Fine-Tuning (SFT)
The first training stage fine-tunes the base model on instruction-response pairs. The GSM8K dataset provides high-quality mathematical reasoning examples, where each question has a definitive answer that can be verified programmatically. LoRA adapters reduce memory footprint while preserving model quality.
Stage 2: Direct Preference Optimization (DPO)
DPO aligns the model with human preferences without requiring a separate reward model. This stage uses pairs of chosen and rejected responses, optimizing the policy directly to favor preferred outputs. The approach reduces training complexity compared to traditional RLHF while maintaining alignment quality.
Stage 3: Reinforcement Learning with Verifiable Rewards (RLVR) via GRPO
The final stage uses GRPO to optimize for verifiable outcomes. Unlike traditional reward models, RLVR leverages deterministic verifiers—in this case, checking whether model-generated answers match the exact GSM8K ground truth. The GRPO variant computes group-relative advantages, enabling stable policy optimization with reduced variance compared to standard PPO.
Verifier-Based Evaluation
The pipeline employs deterministic verifiers as a reliable evaluation mechanism for mathematical tasks. The verifier extracts the final answer from model generation and compares it against the expected result with strict equality. This approach provides:
- Objective scoring: Removes subjectivity from reward assignment
- Efficient training: Eliminates the need for learned reward models
- Interpretability: Clear signal on whether the model produces correct outputs
2026 Context And Practical Considerations
The Tulu 3 post-training framework remains highly relevant in 2026, as efficient post-training methods continue to gain prominence. With rising interest in domain-specific specialized models, the ability to adapt compact models for reasoning tasks on consumer hardware addresses a critical need.
Key insights for practitioners:
- Memory optimization: LoRA adapters and gradient checkpointing are essential for fitting models in constrained VRAM
- Data quality: GSM8K's verifiable answers make it ideal for RLVR without external reward models
- Reproducibility: The deterministic verification approach enables consistent evaluation across runs
- Scalability: While this tutorial uses a single GPU, the core logic extends naturally to distributed settings when resources allow
Conclusion
This tutorial demonstrates a complete, resource-efficient post-training pipeline combining SFT, DPO, and RLVR with GRPO on a compact language model. By leveraging Open Instruct's proven components and adapting them for single-GPU environments, it offers a practical template for reproducible post-training experiments. The verifier-based evaluation strategy proves particularly effective for mathematical reasoning tasks, providing clear, objective feedback signals throughout the training process.
For researchers and practitioners working with limited hardware, this approach balances technical sophistication with practical accessibility, enabling meaningful experimentation with state-of-the-art post-training techniques.
via MarkTechPost