--- title: "N1 Healthcare — Biomarker Grouping Pipeline" --- N1 Healthcare — Biomarker Grouping Pipeline ## Overview Historical reference.** This document describes the pre-Rosetta biomarker grouping pipeline (alias lookup + embedding matching). For the current BiomarkerDefinition engine that supersedes it, see [Rosetta — Complete Architecture](/architecture/rosetta-complete/). The **enrichment-biomarkers** refactor replaces the current full ML pipeline (embeddings + agglomerative clustering + LLM fixing for every biomarker) with a two-tier approach: a global alias lookup table handles ~95% of biomarkers instantly, while embedding matching handles the remaining ~5% edge cases. Grouping runs inline per record, enrichment is decoupled entirely. ```text Pipeline Steps ~95% Alias Lookup ~5% Embedding Fallback LLM Calls at Runtime ``` **Core change:** The vast majority of biomarkers skip the ML pipeline completely. Cost no longer scales quadratically with user history. Concurrent processing is unblocked. Enrichment failures can't take down grouping. ### Prerequisite: Global Alias Table ~1,000–2,000 standardized biomarker names with ~5–6 aliases per name. Seeded by LLM, covering the vast majority of lab biomarkers. Simple direct non-LLM lookup with known aliases — used for name standardization. Grows automatically as new raw names are mapped. Includes expected unit families and default sample source per definition. ## Current Problems ##### 1. Full ML for every biomarker A routine "HGB" gets embeddings, agglomerative clustering, and multi-pass LLM fixing — the same as a rare specialized marker. ~95% are common lab tests that could be a simple name lookup. ##### 2. Non-grouping work bundled in Systems assignment, unit conversion, and data validation are inside the grouper. A unit conversion failure blocks grouping from completing. ##### 3. Single-user locking One grouping batch at a time per user. Can't group two records concurrently. Users with the most records wait the longest. ##### 4. O(n²) cost scaling Agglomerative clustering compares every biomarker against every other. 250 biomarkers across 5 records is drastically more expensive than 50 from 1 record. Most active users get worst performance. ## Pipeline Flowchart Per-record inline grouping. All records process concurrently. No user-level locks. The lookup table handles ~95% in sub-milliseconds. Embedding fallback only for the ~5% that miss. ```text S1 Alias Lookup Deterministic Direct match raw name against global alias table • No LLM, no fuzzy • Sub-millisecond per biomarker Match found in alias table? ~95% — Yes Proceed to assign or create ~5% — No S2 Embedding Match ML Compare against user's existing canonicals only (40–80 vectors, not all biomarkers) S3 Assign or Create Deterministic Check user's existing canonicals for name + sample source + unit compatibility User has canonical with this name + same source? **Yes, units convertible** Assign to existing canonical (conversion in enrichment) **Yes, units incompatible** Create separate canonical with unit in name, e.g. "Vitamin D (IU/L)" **Different source** Create separate canonical (clinically distinct, e.g. serum vs urine) **No match at all** Create new canonical for user S4 Dedup Deterministic Quick query: merge duplicate canonicals for this user (same name, same source) • Handles concurrency races User sees grouped data as each record finishes After all records complete E1 Enrichment LLM + Deterministic Health areas / systems (LLM) • Unit standardization (deterministic) • Runs in parallel, separate process ``` ## Stage Details #### Alias Lookup — Direct Name Match Each biomarker's raw name is matched directly against the global alias table. No fuzzy matching, no LLM. Exact string comparison against known aliases. ```text Input `biomarker.raw_name` Output `canonical_name | null` ``` ##### How it works - **Alias table:** ~1,000–2,000 canonical names, ~5–6 aliases each (e.g. "HGB", "Hemoglobin", "Hgb", "Haemoglobin" → `Hemoglobin`) - **Lookup:** Case-insensitive exact match against all aliases - **Performance:** Sub-millisecond per biomarker, handles ~95% of all biomarkers - **Growth:** Table grows automatically as new raw names are mapped via embedding fallback ##### Table metadata per canonical - **Expected unit families:** Groups of convertible units (e.g. mg/dL, g/dL, mmol/L for same family) - **Default sample source:** Most common source for this biomarker (e.g. "serum" for glucose) ```text alias-table ``` #### Embedding Match — Lookup Miss Only For the ~5% that miss the alias table. Generates an embedding for the raw biomarker name and compares against the user's existing canonical embeddings only (typically 40–80 vectors). No agglomerative clustering across all biomarkers. ```text Input `biomarker.raw_name (unmatched)` Output `closest_canonical | null` ``` ##### How it works - **Generate embedding** for the unmatched raw name - **Compare** against user's existing canonical embeddings (cosine distance) - **Match if:** distance < threshold AND compatible sample source AND compatible units - **No match:** Create new canonical for user ##### Key difference from current - **Current:** O(n²) — every biomarker compared against every other biomarker - **Proposed:** O(k) — one biomarker compared against ~40–80 existing canonicals ```text MedEmbed ``` #### Assign or Create — User Canonical Check Now we have a canonical name (from Step 1 or 2). Check whether the user already has a canonical with this name + matching sample source + compatible units. ```text Input `canonical_name + biomarker metadata` Output `assigned_definition_id | new_canonical` ``` ##### Decision tree - **Name + source match, units convertible:** Assign to existing canonical. Unit conversion happens downstream in enrichment. - **Name + source match, units incompatible:** Create separate canonical with unit appended to name (e.g. "Vitamin D (IU/L)"). Different assay methods produce fundamentally different measurements. - **Same name, different source:** Create separate canonical. Serum vs urine glucose are clinically distinct measurements. - **Name + source not in user DB:** Create new canonical for user. - **No match at all (S2 miss):** Create new canonical. ##### Unit convertibility check - Uses the alias table's expected unit families per canonical - Units in the same family (mg/dL ↔ mmol/L) = convertible - Units in different families (IU/L vs ng/mL) = incompatible = separate canonical ```text alias-table user-canonicals-db ``` #### Dedup — Concurrent Race Cleanup Quick query after each record completes. Finds duplicate canonicals for this user (same name, same source) created by concurrent record processing. Merges them by keeping the lowest ID, reassigning biomarkers, and deleting extras. ```text Input `user_id + newly created canonicals` Output `merged canonicals (no duplicates)` ``` ##### Why this is needed - Records process concurrently — two records can both create a "Hemoglobin" canonical at the same time - The dedup query runs after each record, not in a batch — duplicates are cleaned up immediately - This is a fast DB query + merge, not an expensive computation ##### Merge strategy - Keep canonical with lowest ID (first created) - Reassign all biomarkers from duplicate canonicals to the kept one - Delete the duplicate canonical records ```text user-canonicals-db E1 ``` #### Enrichment — Separate Process Runs after all records complete, as a separate parallel process. Completely decoupled from grouping — a failure here does not block biomarker visibility. ```text Input `new/updated canonicals` Output `enriched canonicals` ``` ##### What it does - **Health areas / systems assignment:** LLM-powered classification of canonicals into body systems (cardiovascular, metabolic, etc.) - **Unit standardization:** Convert biomarker values to the canonical standard unit where units differ but are convertible - **Data validation:** Quality checks moved out of grouper ##### What this unblocks - Graphing and trend visualization - Health report summarization - Cross-biomarker analysis ```text LLM (systems) unit-conversion ``` ## Sample Source & Unit Mismatch Handling ##### Different sample source - Always creates separate canonicals, even with the same biomarker name - Example: serum glucose vs urine glucose are clinically distinct measurements - The canonical name stays the same, but they're different entries for the user ##### Different units, same source - Check the alias table's expected unit families for that canonical - **Convertible** (e.g. mg/dL ↔ mmol/L): same canonical, unit conversion happens in enrichment step - **Incompatible** (e.g. IU/L vs ng/mL — different assay methods): separate canonical, append the incompatible unit to the canonical name to disambiguate (e.g. "Vitamin D (IU/L)") ##### Unknown or missing sample source - Default to the most common source for that canonical (from alias table metadata) - Flag for review if ambiguous ## Before vs After ```text Current Proposed **Common biomarkers (~95%)** Full ML pipeline (embeddings + clustering + LLM) Direct alias lookup (sub-ms, no LLM) **Rare biomarkers (~5%)** Full ML pipeline Embedding match against user's canonicals only **Cost scaling** O(n²) — every biomarker vs every biomarker O(k) — one biomarker vs ~40–80 canonicals **Systems / health areas** Inside grouper Separate enrichment process **Unit conversion** Inside grouper Separate enrichment process **Concurrency** Locked per user_id, sequential No locks, dedup handles races **User sees grouped data** After entire pipeline completes As each record finishes **LLM calls at runtime** Multi-pass cluster fixing + name standardization Zero (enrichment is a separate step) ``` ## What's Removed from Grouping LLM name standardization at runtime LLM cluster fixing (multi-pass) Agglomerative clustering on all biomarkers Systems / health area assignment Unit conversion Data validation within grouper Per-record user_id lock and batching Idempotency keys Systems assignment, unit conversion, and data validation are moved to the enrichment step — not deleted, just decoupled from the grouping critical path. ## Current Pipeline Documentation for the current enrichment-biomarkers pipeline. This page documents the proposed refactored pipeline. ### Current Flow ```text S1 Embedding Generation ML MedEmbed embeddings for all biomarkers in user's history S2 Agglomerative Clustering ML O(n²) — every biomarker compared against every other • Cosine distance < 0.05 S3 LLM Cluster Fixing LLM Multi-pass LLM review of clusters • Name standardization • Merge/split decisions S4 Systems & Enrichment LLM + Deterministic Health areas, unit conversion, data validation — all bundled inside the grouper User sees results after entire pipeline completes ``` ### Key Issues ##### Full ML for every biomarker ~95% of biomarkers are common lab tests that don't need embeddings or LLM. ##### O(n²) cost Agglomerative clustering scales quadratically. Most active users are most expensive. ##### Enrichment bundled in Unit conversion or system assignment failure blocks grouping entirely. ##### Sequential processing Per-user lock means records queue up. No concurrency.