via FreeCodeCamp
Product Experimentation with Doubly Robust Estimation: When Both Your Models Are Wrong in LLM Applications
Your AI product shipped an agent-mode opt-in feature six months ago. After running a propensity analysis—adjusting for engagement tier and query confidence—you reported a clean +8 percentage-point lift in task completion. The number made it into the quarterly business review, and everyone was pleased.
Inevitably, a rigorous data scientist asks an uncomfortable question: How confident are you that the propensity model captured every confounder? What if a key variable was missed, and your logistic regression is estimating the wrong selection probability? What if your outcome regression is also misspecified—because task completion has a nonlinear relationship with query confidence that a linear model can't capture?
You have two models, you're not sure which one is right, and both are load-bearing.
Opt-in AI products hit this wall by default. In causal inference for LLM-based experiments run without randomization, you observe outcomes for users who opted in and those who didn't. The complication is that the groups chose themselves, so every model you build to recover the causal effect is an approximation of an unknown truth.
Propensity weighting alone fails if the propensity model is wrong. Regression adjustment alone fails if the outcome model is wrong. Each method bets everything on a single model being correctly specified.
Doubly robust estimation—specifically the augmented inverse-probability weighting (AIPW) estimator—takes a different bet. It combines a propensity model and an outcome model into a single estimator that remains consistent if either is correctly specified. You need both to fail simultaneously for AIPW to break.
That guarantee comes from the semiparametric efficiency theory underlying the estimator, a mathematical property baked into its construction. Think of it as redundancy engineering for causal estimates—the same fault-tolerance logic that underpins reliable systems, applied to the fragile act of estimating causal effects from observational data.
In 2026, as LLM-based features become ubiquitous—agentic assistants, personalized recommendations, and auto-generated responses—product teams increasingly rely on non-randomized opt-in data. Doubly robust estimation offers a pragmatic path forward: even when your propensity and outcome models are both imperfect, AIPW provides a safety net that keeps your causal estimates credible, as long as at least one model is reasonably well-specified.
To use AIPW effectively, ensure you measure and document all potential confounders, validate your models with cross-fitting to reduce overfitting bias, and treat the doubly robust property as a design principle rather than a license to be careless. The estimator trusts you to build at least one model well—so invest in feature engineering, diagnostic checks, and transparent reporting of uncertainty.
The next time you present a lift number, you can answer the uncomfortable question with more than hope. With AIPW, you demonstrate that even when both models are wrong, your conclusion can still stand—provided you've built them with care.