Machine Learning · Data Engineering · 2024
Verdant: Clinical NLP & Entity Extraction Engine
Transformer-based token classification and entity linking across 12M+ biomedical records, achieving 96.8% macro F1 score.
- Role
- Staff Applied Scientist
- Timeline
- 14 weeks
- Year
- 2024
- Disciplines
- Machine Learning, Data Engineering
Clinical Document Parsing
Healthcare data is predominantly stored in messy, unstructured clinical notes laden with shorthand, inconsistent formatting, and complex medical syntax. Verdant fine-tunes domain-specific Transformers to extract symptoms, medication dosages, and anatomical sites.
Extracted mentions are linked to standardized SNOMED-CT concepts through a dense vector retrieval index, disambiguating complex clinical terminology.
Distributed Inference at Scale
To backfill 12 million historical clinical records, we scaled the model across an elastic Ray cluster. Dynamic batching and mixed-precision inference allowed the system to process over 8,000 notes per minute.
Prediction uncertainties were calibrated using temperature scaling, routing low-confidence decisions to clinical reviewers while auto-approving 91% of extractions.