Retrieval quality in an AI search product is constrained by two factors: the capability of the embedding model and the efficiency of running it across an index. This week, Perplexity Engineering published "Fast Embeddings on GPUs," offering an in-depth look at the serving infrastructure behind pplx-embed and the ranking models used across Perplexity Search, Computer, and the API Platform.
Perplexity notes that GPU-side embedding inference has largely converged across engines on mature Hopper and Blackwell hardware. The real performance gains lie in the runtime and harness surrounding the model: CUDA graph management, an asynchronous result-tracking abstraction, and a Rust-based request path.
Unified Engine for Two Traffic Patterns
Perplexity categorizes embedding serving into two workloads. Batch embedding occurs during vector database construction or re-indexing, where throughput drives down costs. Online embedding happens at query time, where a short query must be embedded with minimal latency. Scoring falls in between: after vector search, large document batches are ranked, balancing throughput and latency.
A key architectural decision was not to build a separate embedding engine. Since embedding models are small transformers, batch embedding resembles compute-bound prefill, while online embedding (often just a few tokens) mirrors memory-bound decode. Perplexity thus reuses the prefill and decode kernels from its LLM stack.
The Software Stack: Ivy, Tulip, and ROSE
Three services handle each request:
- Ivy serves as the request layer, managing incoming queries and routing them appropriately.
- Tulip orchestrates GPU resources, coordinating embedding and scoring tasks efficiently.
- ROSE (likely an acronym for the ranking or optimization service) handles the post-vector-search scoring, ensuring relevance across large document batches.
This modular design allows Perplexity to optimize each stage independently, from request intake to final ranking, while maintaining a unified engine core.
Performance and Future Outlook
By leveraging a common engine for both embedding and LLM workloads, Perplexity maximizes hardware utilization and simplifies maintenance. The focus on runtime optimizations—such as CUDA graphs to reduce kernel launch overhead and async tracking for better pipeline overlap—demonstrates a mature approach to AI infrastructure.
As of 2026, with Blackwell hardware becoming more prevalent, Perplexity's stack is well-positioned to support increasingly complex retrieval models and growing query volumes. The company emphasizes that while model architecture remains crucial, infrastructure efficiency is equally vital for delivering fast and accurate AI search at scale.
For developers and researchers, this breakdown offers valuable insights into production-grade embedding serving, highlighting that cutting-edge models require equally sophisticated serving systems to truly excel.
via MarkTechPost