Supabase has introduced Supabase Evals, an open-source benchmark and testing framework designed to evaluate how effectively AI agents build with Supabase. This tool runs coding agents—including Claude Code, Codex, and OpenCode—against real-world tasks such as creating a database schema, debugging a failed Edge Function, or fixing a broken Row-Level Security (RLS) policy, and then scores their performance. The framework currently powers a public leaderboard at supabase.com/evals and an internal regression suite that is monitored daily.
Is It Deployable?
Yes, it is available today. The repository supabase/evals is publicly hosted under the Apache-2.0 license and can be run locally using pnpm.
- Industries: This tool is relevant for developer tooling, cloud infrastructure, data platforms, and regulated backends in sectors like fintech or healthcare, where an agent-generated incorrect RLS policy could constitute a security incident.
- Applications: It is useful for regression-testing documentation updates and skill edits, gating SDK releases, and comparing agent harnesses head-to-head.
- Constraints: Running the local stack requires a Docker daemon, provider API keys, and available ports 54321–54329.
How Does the Harness Work?
Supabase has defined three evaluation dimensions: products (database, auth, storage, edge-functions, realtime, cron, queues, vectors, and data-api), topics (RLS, security, migrations, SQL, SDK, observability, self-hosting, tests, and declarative-schema), and stages (build, deploy, investigate, and resolve). The scenarios were selected to cover each dimension at least once, grounding them in real-world sources such as support tickets, bug reports, and GitHub issues.
The harness scores agents based on their ability to complete these tasks, providing a standardized measure of agent performance. This infrastructure not only supports the public leaderboard but also serves as a daily regression check for Supabase’s internal development, ensuring that changes to docs, skills, or SDKs do not degrade agent performance.
As AI coding agents become increasingly integrated into development workflows, standardized benchmarks like Supabase Evals are critical for ensuring reliability and security. By open-sourcing this framework, Supabase aims to foster broader adoption and improvement of agent-based development tools across the industry.
via MarkTechPost