N1 Healthcare — Biomarker Grouping Pipeline

Overview

Historical reference.** This document describes the pre-Rosetta biomarker grouping pipeline (alias lookup + embedding matching). For the current BiomarkerDefinition engine that supersedes it, see Rosetta — Complete Architecture.

The enrichment-biomarkers refactor replaces the current full ML pipeline (embeddings + agglomerative clustering + LLM fixing for every biomarker) with a two-tier approach: a global alias lookup table handles ~95% of biomarkers instantly, while embedding matching handles the remaining ~5% edge cases. Grouping runs inline per record, enrichment is decoupled entirely.

Pipeline Steps

~95%
Alias Lookup

~5%
Embedding Fallback

LLM Calls at Runtime

Core change: The vast majority of biomarkers skip the ML pipeline completely. Cost no longer scales quadratically with user history. Concurrent processing is unblocked. Enrichment failures can’t take down grouping.

Prerequisite: Global Alias Table

~1,000–2,000 standardized biomarker names with ~5–6 aliases per name. Seeded by LLM, covering the vast majority of lab biomarkers. Simple direct non-LLM lookup with known aliases — used for name standardization. Grows automatically as new raw names are mapped. Includes expected unit families and default sample source per definition.

Current Problems

1. Full ML for every biomarker

A routine “HGB” gets embeddings, agglomerative clustering, and multi-pass LLM fixing — the same as a rare specialized marker. ~95% are common lab tests that could be a simple name lookup.

2. Non-grouping work bundled in

Systems assignment, unit conversion, and data validation are inside the grouper. A unit conversion failure blocks grouping from completing.

3. Single-user locking

One grouping batch at a time per user. Can’t group two records concurrently. Users with the most records wait the longest.

4. O(n²) cost scaling

Agglomerative clustering compares every biomarker against every other. 250 biomarkers across 5 records is drastically more expensive than 50 from 1 record. Most active users get worst performance.

Pipeline Flowchart

Per-record inline grouping. All records process concurrently. No user-level locks. The lookup table handles ~95% in sub-milliseconds. Embedding fallback only for the ~5% that miss.

S1
Alias Lookup
Deterministic

Direct match raw name against global alias table &bull; No LLM, no fuzzy &bull; Sub-millisecond per biomarker

Match found in alias table?

~95% — Yes

Proceed to assign or create

~5% — No

S2
Embedding Match
ML

Compare against user's existing canonicals only (40–80 vectors, not all biomarkers)

S3
Assign or Create
Deterministic

Check user's existing canonicals for name + sample source + unit compatibility

User has canonical with this name + same source?

**Yes, units convertible**
Assign to existing canonical (conversion in enrichment)

**Yes, units incompatible**
Create separate canonical with unit in name, e.g. "Vitamin D (IU/L)"

**Different source**
Create separate canonical (clinically distinct, e.g. serum vs urine)

**No match at all**
Create new canonical for user

S4
Dedup
Deterministic

Quick query: merge duplicate canonicals for this user (same name, same source) &bull; Handles concurrency races

User sees grouped data as each record finishes

After all records complete

E1
Enrichment
LLM + Deterministic

Health areas / systems (LLM) &bull; Unit standardization (deterministic) &bull; Runs in parallel, separate process

Stage Details

Alias Lookup — Direct Name Match

Each biomarker’s raw name is matched directly against the global alias table. No fuzzy matching, no LLM. Exact string comparison against known aliases.

Input
`biomarker.raw_name`

Output
`canonical_name | null`

How it works

Alias table: ~1,000–2,000 canonical names, ~5–6 aliases each (e.g. “HGB”, “Hemoglobin”, “Hgb”, “Haemoglobin” → Hemoglobin)
Lookup: Case-insensitive exact match against all aliases
Performance: Sub-millisecond per biomarker, handles ~95% of all biomarkers
Growth: Table grows automatically as new raw names are mapped via embedding fallback

Table metadata per canonical

Expected unit families: Groups of convertible units (e.g. mg/dL, g/dL, mmol/L for same family)
Default sample source: Most common source for this biomarker (e.g. “serum” for glucose)

alias-table

Embedding Match — Lookup Miss Only

For the ~5% that miss the alias table. Generates an embedding for the raw biomarker name and compares against the user’s existing canonical embeddings only (typically 40–80 vectors). No agglomerative clustering across all biomarkers.

Input
`biomarker.raw_name (unmatched)`

Output
`closest_canonical | null`

How it works

Generate embedding for the unmatched raw name
Compare against user’s existing canonical embeddings (cosine distance)
Match if: distance < threshold AND compatible sample source AND compatible units
No match: Create new canonical for user

Key difference from current

Current: O(n²) — every biomarker compared against every other biomarker
Proposed: O(k) — one biomarker compared against ~40–80 existing canonicals

MedEmbed

Assign or Create — User Canonical Check

Now we have a canonical name (from Step 1 or 2). Check whether the user already has a canonical with this name + matching sample source + compatible units.

Input
`canonical_name + biomarker metadata`

Output
`assigned_definition_id | new_canonical`

Decision tree

Name + source match, units convertible: Assign to existing canonical. Unit conversion happens downstream in enrichment.
Name + source match, units incompatible: Create separate canonical with unit appended to name (e.g. “Vitamin D (IU/L)”). Different assay methods produce fundamentally different measurements.
Same name, different source: Create separate canonical. Serum vs urine glucose are clinically distinct measurements.
Name + source not in user DB: Create new canonical for user.
No match at all (S2 miss): Create new canonical.

Unit convertibility check

Uses the alias table’s expected unit families per canonical
Units in the same family (mg/dL ↔ mmol/L) = convertible
Units in different families (IU/L vs ng/mL) = incompatible = separate canonical

alias-table
user-canonicals-db

Dedup — Concurrent Race Cleanup

Quick query after each record completes. Finds duplicate canonicals for this user (same name, same source) created by concurrent record processing. Merges them by keeping the lowest ID, reassigning biomarkers, and deleting extras.

Input
`user_id + newly created canonicals`

Output
`merged canonicals (no duplicates)`

Why this is needed

Records process concurrently — two records can both create a “Hemoglobin” canonical at the same time
The dedup query runs after each record, not in a batch — duplicates are cleaned up immediately
This is a fast DB query + merge, not an expensive computation

Merge strategy

Keep canonical with lowest ID (first created)
Reassign all biomarkers from duplicate canonicals to the kept one
Delete the duplicate canonical records

user-canonicals-db

E1

Enrichment — Separate Process

Runs after all records complete, as a separate parallel process. Completely decoupled from grouping — a failure here does not block biomarker visibility.

Input
`new/updated canonicals`

Output
`enriched canonicals`

What it does

Health areas / systems assignment: LLM-powered classification of canonicals into body systems (cardiovascular, metabolic, etc.)
Unit standardization: Convert biomarker values to the canonical standard unit where units differ but are convertible
Data validation: Quality checks moved out of grouper

What this unblocks

Graphing and trend visualization
Health report summarization
Cross-biomarker analysis

LLM (systems)
unit-conversion

Sample Source & Unit Mismatch Handling

Different sample source

Always creates separate canonicals, even with the same biomarker name
Example: serum glucose vs urine glucose are clinically distinct measurements
The canonical name stays the same, but they’re different entries for the user

Different units, same source

Check the alias table’s expected unit families for that canonical
Convertible (e.g. mg/dL ↔ mmol/L): same canonical, unit conversion happens in enrichment step
Incompatible (e.g. IU/L vs ng/mL — different assay methods): separate canonical, append the incompatible unit to the canonical name to disambiguate (e.g. “Vitamin D (IU/L)”)

Unknown or missing sample source

Default to the most common source for that canonical (from alias table metadata)
Flag for review if ambiguous

Before vs After

Current
Proposed

**Common biomarkers (~95%)**
Full ML pipeline (embeddings + clustering + LLM)
Direct alias lookup (sub-ms, no LLM)

**Rare biomarkers (~5%)**
Full ML pipeline
Embedding match against user's canonicals only

**Cost scaling**
O(n²) — every biomarker vs every biomarker
O(k) — one biomarker vs ~40–80 canonicals

**Systems / health areas**
Inside grouper
Separate enrichment process

**Unit conversion**
Inside grouper
Separate enrichment process

**Concurrency**
Locked per user_id, sequential
No locks, dedup handles races

**User sees grouped data**
After entire pipeline completes
As each record finishes

**LLM calls at runtime**
Multi-pass cluster fixing + name standardization
Zero (enrichment is a separate step)

What’s Removed from Grouping

LLM name standardization at runtime LLM cluster fixing (multi-pass) Agglomerative clustering on all biomarkers Systems / health area assignment Unit conversion Data validation within grouper Per-record user_id lock and batching Idempotency keys

Systems assignment, unit conversion, and data validation are moved to the enrichment step — not deleted, just decoupled from the grouping critical path.

Current Pipeline

Documentation for the current enrichment-biomarkers pipeline. This page documents the proposed refactored pipeline.

Current Flow

S1
Embedding Generation
ML

MedEmbed embeddings for all biomarkers in user's history

S2
Agglomerative Clustering
ML

O(n²) — every biomarker compared against every other &bull; Cosine distance < 0.05

S3
LLM Cluster Fixing
LLM

Multi-pass LLM review of clusters &bull; Name standardization &bull; Merge/split decisions

S4
Systems & Enrichment
LLM + Deterministic

Health areas, unit conversion, data validation — all bundled inside the grouper

User sees results after entire pipeline completes

Key Issues

Full ML for every biomarker

~95% of biomarkers are common lab tests that don’t need embeddings or LLM.

O(n²) cost

Agglomerative clustering scales quadratically. Most active users are most expensive.

Enrichment bundled in

Unit conversion or system assignment failure blocks grouping entirely.

Sequential processing

Per-user lock means records queue up. No concurrency.