Machine Learning · Data Engineering · 2024

Verdant: Clinical NLP & Entity Extraction Engine

Transformer-based token classification and entity linking across 12M+ biomedical records, achieving 96.8% macro F1 score.

Role
Staff Applied Scientist
Timeline
14 weeks
Year
2024
Disciplines
Machine Learning, Data Engineering
Verdant clinical entity extraction visualization with token confidence metrics

Clinical Document Parsing

Healthcare data is predominantly stored in messy, unstructured clinical notes laden with shorthand, inconsistent formatting, and complex medical syntax. Verdant fine-tunes domain-specific Transformers to extract symptoms, medication dosages, and anatomical sites.

Extracted mentions are linked to standardized SNOMED-CT concepts through a dense vector retrieval index, disambiguating complex clinical terminology.

Distributed Inference at Scale

To backfill 12 million historical clinical records, we scaled the model across an elastic Ray cluster. Dynamic batching and mixed-precision inference allowed the system to process over 8,000 notes per minute.

Prediction uncertainties were calibrated using temperature scaling, routing low-confidence decisions to clinical reviewers while auto-approving 91% of extractions.

Next project

Lumen: Low-Latency Enterprise Vector Search & RAG

Sub-30ms hybrid lexical-vector retrieval architecture over 40M enterprise documents with semantic caching and cross-encoder reranking.

View project