Columbia and CertiK Researchers Unveil TensorLean for Formal Verification of GPU Kernels
Researchers from Columbia University and CertiK have introduced TensorLean, a framework for formally verifying GPU kernels at the PTX level. TensorLean is a Lean-based verifier that proves compiled kernels match a tensor-level mathematical specification.
The team evaluated 149 kernel instances, including every kernel in FlashAttention-2, -3, and -4 as well as DeepGEMM. TensorLean proved 83 of them correct; the remaining 66 led to the discovery of 38 correctness bugs. The work arrives at a critical moment in 2026, as AI infrastructure increasingly depends on hand-tuned, hardware-specific kernels for Nvidia's A100, H100, B200, and newer accelerators, where subtle concurrency and memory-ordering bugs can silently corrupt results at scale.
TL;DR
- Size: A 1.8K-line PTX parser. Specifications total 10,278 lines against 151,285 lines of kernel source—a ratio of roughly 15:1.
- Runs on: A 16-core workstation. It checks PTX compiled for A100, H100, and B200 kernels.
- Performance: 83 of 149 kernel instances proved correct, with 38 bugs found across 6 projects.
- Best result: Caught a FlashAttention-4 data race that never appeared in 17,760 test runs.
What Is TensorLean?
TensorLean is a formal verification framework for GPU kernels at the PTX level. It uses the Lean proof assistant to establish that compiled kernels—rather than only their high-level source—match a tensor-level mathematical specification. This matters because GPU compilers and hand-written assembly can introduce bugs that conventional testing misses, especially in the massively parallel execution model of modern AI hardware.
Why PTX-Level Verification Matters in 2026
As large language models and other AI workloads push GPUs to their limits, kernel correctness has become a first-order infrastructure concern. FlashAttention and DeepGEMM-style kernels are routinely deployed in production training and inference stacks, and a single data race can produce nondeterministic, hard-to-reproduce failures. Formal verification at the PTX level offers a stronger guarantee than extensive test suites—as TensorLean's discovery of a FlashAttention-4 data race that survived 17,760 test runs illustrates.
Key Takeaways
- TensorLean verifies compiled GPU kernels against formal tensor-level specifications.
- It proved 83 of 149 kernel instances correct and uncovered 38 bugs across 6 projects.
- The framework runs on modest hardware—a 16-core workstation—and targets PTX for A100, H100, and B200 GPUs.
- Its most notable catch: a FlashAttention-4 data race invisible to conventional testing.
For researchers and engineers building the next generation of AI systems, TensorLean points toward a future in which formal methods become a standard part of the GPU kernel development pipeline.
via MarkTechPost
