Model cards often report quality under server-class, full-precision conditions, but those numbers rarely predict real-world behavior on a phone. This week, Liquid AI released Pipette, an open-source platform for benchmarking foundation models on edge devices. Developed in partnership with Artificial Analysis as an independent methodology validator, Pipette treats on-device behavior as a property of the deployed system rather than the model in isolation. Its unit of measurement is a full configuration: model + quantization + runtime + device. The launch dataset covers five on-device performance metrics across more than 1,000 model ร quantization ร runtime ร device ร context configurations, spanning 30+ models, llama.cpp builds for macOS, iOS, Windows, and Android, and context lengths from 256 to 8,192 tokens. Initial verified results come from a MacBook Pro with M5 Max, an iPhone 17 Pro, and a Galaxy S26 Ultra. The practical claim is testable: two 350M models at the same quantization on the same phone retain 78.4% and 33.8% of decode throughput at 4,096 tokens, respectively.
Is It Deployable?
Yes, Pipette ships as Apache 2.0 infrastructure (pipette-mgmt, pipette-clients, pipette-scores), a public results dataset, a hosted dashboard, and native iOS and Android apps.
via MarkTechPost