Distributed Systems · Data Engineering · 2025

Pulse: High-Throughput Streaming Feature Store

A distributed streaming feature store ingesting 1.8B events/day with Apache Spark & Kafka, achieving p99 retrieval latency under 4ms.

Role
Principal Data Engineer
Timeline
10 weeks
Year
2025
Disciplines
Distributed Systems, Data Engineering
Pulse distributed feature store architecture diagram with Spark and Kafka pipelines

Unified Feature Ingestion

Data science teams frequently re-implement the same features in both SQL for offline training and Python microservices for online serving. Pulse provides a unified declarative feature repository where transforms are defined once and compiled into both streaming Kafka jobs and batch Spark pipelines.

The architecture ingests over 1.8 billion daily interactions, materializing real-time aggregates into an in-memory key-value tier while streaming raw Parquet files into a partitioned lakehouse.

Point-in-Time Correctness

Feature leakage is the most pervasive cause of failed ML deployments. Pulse implements an as-of join engine in PySpark that reconstructs the exact state of entity features at any historical timestamp.

Data scientists can run training dataset generation across terabytes of historical logs without risking future information contamination.

Next project

Verdant: Clinical NLP & Entity Extraction Engine

Transformer-based token classification and entity linking across 12M+ biomedical records, achieving 96.8% macro F1 score.

View project