Data Engineering · Distributed Systems · 2024

Atlas: Multi-Petabyte Lakehouse Migration & Query Optimizer

Architected a modern Delta Lake storage foundation replacing legacy Hadoop clusters, reducing compute expenditure by 52% and query runtimes by 4.2x.

Role
Staff Data Platform Engineer
Timeline
16 weeks
Year
2024
Disciplines
Data Engineering, Distributed Systems
Atlas Lakehouse storage architecture and distributed query benchmarks

The Lakehouse Modernization

Legacy Hive and Hadoop clusters struggled under growing analytical workloads, causing table lock contention, silent read corruption, and massive compute costs. Atlas migrated 3.4PB of historical data to an open Delta Lake architecture.

We introduced automated file compaction, metadata vacuuming, and bloom filters that drastically reduced small-file problems and enhanced storage efficiency.

Query Acceleration & Cost Governance

We deployed Trino as the unified query engine, enabling quantitative analysts and data scientists to query petabyte-scale datasets in seconds using standard SQL. DuckDB was integrated for fast in-memory prototyping on developer laptops.

Through compute rightsizing and intelligent partition pruning, monthly infrastructure costs dropped by 52% while user query satisfaction hit all-time highs.

Next project

Aurora: Real-Time Fraud & Anomaly Detection

A sub-15ms fraud detection pipeline processing 25,000 transactions/sec with a calibrated ensemble model, cutting chargebacks by 44%.

View project