Perplexity has released Lily as an open-source inference engine, the technology powering Hybrid Compute in Perplexity Computer. Designed as a single-process runtime, Lily leverages a Rust layer to load checkpoints and manage the generation loop, streams tokens through an OpenAI-compatible chat-completions API, and executes the model with hand-written Metal kernels. Notably, neither PyTorch nor MLX is in the execution path. Lily is intentionally focused, supporting one model—Qwen3.6-35B-A3B—on one hardware family, and this narrowness is central to its performance benefits.
Is it deployable? Yes. A standalone demo is available in the pplx-garden repository, featuring a Rust and Metal inference server that delivers greedy text generation via a minimal OpenAI-compatible HTTP API. The 4-bit checkpoint requires approximately 19.4 GB of storage, making an Apple silicon Mac with 32 GB or more of unified memory the practical minimum. Perplexity’s Hybrid Compute product officially supports macOS 15+ with a 24 GB minimum and recommends 32 GB for optimal performance.
Why Specialize?
Lily’s design philosophy centers on specialization. By focusing exclusively on Qwen3.6-35B-A3B and Apple Silicon, the engine can optimize every layer of the stack—from memory management to kernel scheduling—without the overhead of compatibility layers. This approach allows for fine-grained tuning of Metal kernels and Rust-based execution logic, yielding lower latency and better throughput compared to generic frameworks.
Technical Highlights
- Rust Core: The generation loop and checkpoint loading are implemented in Rust, ensuring memory safety and high performance.
- Metal Kernels: Hand-written Metal compute kernels handle all model operations, bypassing higher-level frameworks for direct GPU control.
- OpenAI-Compatible API: A simple HTTP server exposes chat-completions endpoints, enabling drop-in integration with existing tools.
- Single-Process Runtime: No external inference dependencies; the entire pipeline runs in one process, reducing overhead and simplifying deployment.
System Requirements
To run Lily effectively, users should have:
- An Apple Silicon Mac (M-series chip)
- macOS 15 or later
- 32 GB unified memory (24 GB minimum for basic operation)
- At least 20 GB of free storage for the checkpoint
Community and Licensing
Lily is open source under a permissive license, allowing developers to inspect, modify, and integrate the engine into their projects. The codebase is hosted in the pplx-garden repository, which also contains other tools from Perplexity’s engineering team.
Looking Forward
With the 2026 release, Lily represents a growing trend toward specialized, hardware-native inference solutions. As Apple Silicon becomes more prevalent in edge computing and local AI, engines like Lily could bridge the gap between cloud-scale models and local privacy-preserving deployments. Perplexity’s open-source move invites community contributions to expand model support and optimize further for future Apple chips.
via MarkTechPost