Perplexity Details Its GPU Embedding Stack: How Ivy, Tulip and ROSE Serve pplx-embed
2026-09-06 11:20Models🔥 42.2 heat score
1sources
1days unfolding
42.2heat score
4mentions
SummaryAI generated
On September 6, 2026, Perplexity announced the launch of the Fast Embeddings feature, which relies on成熟的 GPU hardware such as Hopper and Blackwell to support its pplx-embed and ranking models. This solution does not build an independent engine; instead, it utilizes pre-filled and decoding cores in the LLM stack, with Ivy (Rust HTTP gateway), Tulip (gRPC inference server), and ROSE (inference engine) working together to process requests. For small-batch scenarios, the team modified upstream FlashInfer to support full model CUDA graph capture, and combined with a delayed capture mechanism and LazyTensor technology to optimize CPU-GPU interaction efficiency.
Perplexity released Fast Embeddings, utilizing GPU infrastructure to support its pplx-embed and ranking models. The team noted that on mature Hopper and Blackwell hardware, GPU-based embedding inference has reached a state of convergence. The key optimizations lie in the runtime and toolset, including CUDA graph management, asynchronous result tracking abstraction, and Rust request paths. Perplexity did not build an independent engine but reused pre-filled and decoding kernels from the LLM stack, with Ivy (Rust HTTP gateway), Tulip (gRPC inference server), and ROSE (inference engine) working together to process requests. For small batch scenarios, the team submitted changes to FlashInfer upstream to support full model CUDA graph capture and adopted a delayed capture mechanism and LazyTensor technology to optimize CPU-GPU interaction efficiency.