Distributed Systems · MLOps · 2025
Lumen: Low-Latency Enterprise Vector Search & RAG
Sub-30ms hybrid lexical-vector retrieval architecture over 40M enterprise documents with semantic caching and cross-encoder reranking.
- Role
- Machine Learning Architect
- Timeline
- 8 weeks
- Year
- 2025
- Disciplines
- Distributed Systems, MLOps
Hybrid Retrieval Architecture
Pure vector retrieval frequently fails on exact keyword matches, code symbols, and alphanumeric model codes. Lumen deploys a hybrid retrieval architecture combining BM25 inverted indexes with dense vector embeddings.
Reciprocal Rank Fusion (RRF) harmonizes sparse and dense results, passing top candidates to a lightweight cross-encoder reranker compiled with ONNX Runtime.
Performance & Semantic Caching
By implementing a semantic vector cache that matches incoming queries within a tight cosine similarity threshold, the architecture eliminates redundant retrieval queries for frequent topics.
Ray Serve manages request batching and model concurrency, maintaining stable sub-30ms latencies even during high peak query spikes.