--- title: "Rosetta — Complete Architecture" description: Complete architecture specification for Rosetta, the BiomarkerDefinition engine for N1. --- Rosetta answers one question, millions of times: are these two clinical records measuring the same thing? Get it right and a patient's history forms coherent trends across labs, years, and languages. Get it wrong and the chart silently breaks. This document explains how Rosetta gets it right — and why the architecture looks the way it does. ### Contents 1. [The problem Rosetta solves](#problem) 1. [The core insight: identity is not a name](#insight) 1. [Where Rosetta sits in the platform](#placement) 1. [The original design & what we learned](#lessons) 1. [The multi-domain architecture (v1)](#multidomain) 1. [Component deep-dives](#components) 1. [The three guarantees](#guarantees) 1. [Caching](#caching) 1. [The corrections facility](#corrections) 1. [What is NOT in Rosetta](#boundaries) 1. [Rollout plan](#rollout) 1. [Glossary](#glossary) ## 01The problem Rosetta solves N1 shows each clinician a longitudinal view of a patient's clinical data over time. Two hemoglobin measurements taken a year apart, from two different labs, in two different languages, must appear on the same chart so the trend is visible. The same holds for every biomarker, diagnosis, medication, allergy panel, cognitive assessment, and genetic variant. For that to work, the platform has to answer one question, millions of times: **are these two clinical records measuring the same thing?** Trivial when both come from the same lab with the same name. Hard the moment a patient uploads records from many labs, across years, in many languages: - "TSH" / "Thyroid Stimulating Hormone" / "Thyrotropin" / "甲状腺刺激ホルモン" — same test, **must merge**. - "Hemoglobin" (g/dL) / "Haemoglobin" (g/L) / "HGB" / "血红蛋白" — same test, **must merge**. - "Free Cortisone 1st Morning" vs "2nd Morning" — **different** tests (timepoint-specific references), must NOT merge. - "MCH" (pg) vs "MCHC" (g/dL) — different quantities, same word stem, must NOT merge. - "Systolic BP (Recovery)" vs "(Exercise Stage 2)" vs plain — three distinct stress-test readings, must NOT merge. ```text Why this matters clinically If a patient's CA 19-9 surveillance trend is split across two "canonical" rows because an LLM emitted two display strings on two days, the clinician sees only half the trend. For a tumour marker whose entire value is trend detection, the chart is **silently broken**. ``` ## 02The core insight: identity is not a name The system before Rosetta deduplicated records **by display-name string equality**. It asked an LLM to produce a `canonical_name`, then merged records whose names matched as strings. This can never work, for one structural reason: ```text The load-bearing idea An LLM is a **generator, not a classifier**. Asked to name TSH, it emits "TSH", "Thyrotropin", "Thyroid Stimulating Hormone", "Thyrotropin (TSH)" across calls — all correct, all different. No prompt engineering removes this; stylistic variation is intrinsic. So if the dedup key is a non-deterministic string, every variation creates a duplicate. One staging patient carried **23 distinct clusters of duplicate canonicals**. ``` The fix: stop deduplicating on the display string. Identity must be a **stable, deterministic, structured key**; the display name becomes a *derived property* of that key. Rosetta decomposes each entity into axes borrowed from established ontologies (LOINC for labs): ```text test_name: "Thyrotropin" | unit: mIU/L | specimen: Serum component = "thyrotropin" ← what is measured property = "MCnc" ← kind of quantity system = "ser/plas" ← biological system method = "" ← assay method (if identity-defining) time_aspect = "" ← timing constraint (if any) qualifier = "" ← variant info (if any) group_key = "thyrotropin|mcnc|ser/plas|||" ← THE IDENTITY (stable) canonical_name = "Thyrotropin (TSH)" ← DERIVED display ``` Every record decomposing to the same `group_key` is the same concept and belongs on the same chart — regardless of the original string or the chosen display. **This is the whole idea.** Everything else is engineering to make that decomposition fast, deterministic, and correct across every kind of clinical data. ## 03Where Rosetta sits in the platform Rosetta is a **library** (`n1r-rosetta`), not a service. The dependency arrow always points *into* Rosetta; it never depends on N1 application code. #### n1r-rosetta Decomposes one `(name, unit, specimen)` tuple into a BiomarkerDefinition. Stateless except for the cache. Plain Python. #### rosetta-grouper-v1 Fetches a patient's records, resolves each via Rosetta, groups by `group_key`, persists identities + FKs. #### Platform DB Stores identities in dedicated tables; each record FK-links to its identity. Rosetta knows nothing about patients, auth, HTTP, or the database. It is a **pure function from clinical text to BiomarkerDefinition**, with a cache. ## 04The original design (biochem) & what we learned The first production Rosetta handled one domain: biochemistry. It works, and it taught three lessons that shaped the multi-domain rewrite. ### What worked - **Structured LOINC-axis decomposition** — identity-as-key eliminated the duplicate-canonical problem at the root. - **Schema-constrained output** gives shape guarantees. - A **validator stack** (dimensional analysis, plausibility, timing) catches a real class of LLM errors before the DB. ### What we learned the hard way ```text Lesson 1 — One giant prompt is fragile The biochem prompt grew to ~500 lines. We measured the cost: patching it to fix gamma-glutamyl transferase spelling **broke triglycerides** (13/21 → 1/21 correct). Adding a rule reshuffles the model's attention across the whole prompt. Large prompts don't compose. Lesson 2 — The eval was independent but not comprehensive gemini-3.5-flash vs production gemini-3.1-pro-preview agreed within 0.5% on 1,351 synthetic inputs — but only **55%** on a real patient's 8,387 biomarkers. The disagreements clustered in shapes the synthetic set never had (stress-test stages, allergen panels). Real patient data has shapes curated tests miss. Lesson 3 — "First-write-wins" stabilization is a smell To suppress display-name jitter, biochem added two extra cache layers that locked in whichever display string was written first. That made the name a patient saw depend on *who resolved it first* — an arbitrary race. The right fix: make the display name a pure function of the axes, so there is nothing to stabilize. ``` ## 05The multi-domain architecture (v1) The vision: Rosetta becomes the identity engine for **all** clinical data. A lightweight classifier routes each input to a domain-specific pipeline. Each domain has its own small prompt, identity shape, validators, and cache namespace. v1 implements **two domains end-to-end**: biochem (migrated) and allergy (greenfield). The other five are recognised by the classifier but route to biochem fallback until their decomposers ship — so the classifier returns its true verdict from day one, while consumers never hit an unimplemented-domain exception. ```text BiochemLOINC · 6 axesLive (migrated) Allergyallergen·reaction·severity·qualv1 (new) DiagnosisICD-10Future MedicationRxNormFuture ProcedureCPTFuture Cognitiveinstrument-keyedFuture GeneticsHGVS · ClinVarFuture Why route at all Routing is what makes each prompt small. The allergy prompt knows nothing about LOINC; the biochem prompt knows nothing about reaction types. **Lesson 1 dissolves** — a fix in the allergy prompt cannot break biochem because they are different prompts. The 77 catastrophic decomposition failures we saw on one patient (spice-IgG panels, drug allergies, wheal readings) were all biochem being asked to handle non-biochem inputs. Still a lean library Routing adds exactly two things: a classifier (one cheap cached LLM call) and a dispatch table (a dict lookup). No framework, no orchestration engine, no agent runtime. The resolver shrinks from ~1,300 lines to ~250 lines of router; each domain is a self-contained ~500-line module. **Plain Python, start to finish.** ``` ## 06Component deep-dives ### 6.1 · The classifier - **Model:** `gemini-3.1-flash-lite` — classification is tiny and well-bounded; it doesn't need a frontier model. Cheap and fast on the hot path. - **Output:** JSON-schema `{domain, confidence}` where both are **enums**. The model physically cannot emit a domain outside the set. - **Multilingual by design:** ≥6 non-English examples in the prompt (French allergy, Mandarin diagnosis, Japanese medication, Korean procedure, Spanish cognitive, German biochem). - **Soft-fail:** any error returns `{biochem, low}` and is *not* cached — a transient outage never burns a permanent miss. - **Disambiguation** is abstract, not enumerative: five short rules ("X IgE → allergy even when X looks like a vitamin"), never a list of substances. ### 6.2 · The resolver / dispatcher The `Resolver` is a **router, not a worker**. It owns the OpenAI client, the shared cache, the classifier, the corrections registry, and the `{domain → decomposer}` table. Every cache write, LLM call, validator, and correction lives *inside* the decomposers. 1. `classify(...)` → `{domain, confidence}` 1. If confidence is low *or* the domain has no decomposer yet → route to **biochem** (logged at INFO). 1. Dispatch to `domains[domain].resolve_groupable(...)`. 1. Stamp the chosen `domain` onto the result. The public API (`resolve`, `resolve_groupable`, `resolve_batch`, `resolve_batch_groupable`) is **unchanged** — `rosetta-grouper-v1` keeps working without modification. ### 6.3 · A domain decomposer Each domain is a self-contained module: `identity.py` (frozen dataclass, axes + derived properties), `prompt.py` (small prompt + enum schema), `decomposer.py` (LLM call + assembly + cache + corrections), `format.py` (the pure display-name function), `validators.py` (domain checks). Domains share only a duck-typed surface (`.group_key`, `.canonical_name`, `.domain`) — never a common base class, so divergent domains never pollute each other's shape. ## 07The three guarantees The rewrite makes three promises the original design could not. #### Closed-vocabulary fields cannot drift Every finite-value field is an `enum` in the JSON schema — the model physically cannot emit an out-of-set value. We removed the "propose a new label" affordance that let taxonomy fields drift run-to-run. #### Same input → same identity, forever For a given prompt version + correction set + model, the same tuple deterministically produces the same identity. Temperature 0; the cache makes the first resolution permanent until a version bump. #### Same identity → same display name, on every machine, forever `canonical_name` is a **pure function** of the axes — derived, never LLM-emitted, never cached as a stabilized string. The structural fix for "first-write-wins": there is no first writer to win. The 1,544 "same identity, different display" rows on one patient drop to **zero**. ```text On display-stylization tables A small `_DISPLAY_OVERRIDES` per domain handles stylization (e.g. "Hemoglobin A1c (HbA1c)") — but this is **stylization, not equivalence**: one identity → one prettier string, never many surface forms → one canonical. Every entry carries a one-line comment explaining why the mechanical builder falls short. ``` ## 08Caching The original three-layer cache collapses to **one layer per namespace**: ```text NamespaceKey shapeWriteBump when ``` `` is `{normalized_unit}||{normalized_specimen}||{test_name}`. Each domain owns its version integer (biochem starts at **v13**, bumped from v12 to signal the identity-shape change; allergy at v1). ```text Why the extra layers are gone The canonicalizer and group-key-index layers existed only to stabilize LLM stochasticity. With enum schemas (Guarantee 1) and pure-function display names (Guarantee 3), there is nothing stochastic left to stabilize. Plain `SET` is correct because every write for a key produces identical bytes. The cache also **fails open**: a Redis hiccup is a miss, never a crash; if Redis is down at startup, fall back to in-memory. ``` ## 09The corrections facility Even with small prompts, the LLM will have specific documented misfires. Corrections patch them **without touching prompts** — sidestepping Lesson 1's cross-coupling entirely. A correction is a tested YAML file under `rosetta/domains//corrections/`: ```text id: BIOCHEM-0002 ``` description: "GGT: LOINC canonical includes a space." domain: biochem match: component: "gamma glutamyltransferase" # the LLM's wrong output correct: component: "gamma glutamyl transferase" # the patch test_cases: - input: {test_name: "GGT", unit: "U/L", specimen: "Serum"} ```json expected_axes: {component: "gamma glutamyl transferase"} ``` provenance: source: "patient-9141cb0a, 11/11 Liver misses" affected_rows_observed: 11 retire_when: - "default GGT spacing lands in the biochem prompt" ```text Why this is NOT a hidden synonym table A correction maps **one documented wrong LLM output → one corrected output**, with evidence and a regression test. It encodes a *known model failure*, not general medical knowledge. It applies as a pure post-processing transform — it does not change the prompt, so the GGT correction literally cannot affect triglycerides. And it has a **retirement path** (`retire_when`): a quarterly audit disables each correction and checks whether anything regresses. ``` The facility is deliberately constrained: transforms come from a **hard whitelist** (literal replacement, axis-template, `extract_parens`) — no `eval`, no arbitrary code in data files. Two corrections that match the same identity and touch the same axis raise `CorrectionAmbiguityError` **at startup**. Every YAML's `test_cases` are **auto-discovered by pytest** and run offline. ## 10What is deliberately NOT in Rosetta Firm architectural boundaries, not omissions. #### No hardcoded medical-equivalence tables Code never claims "peanuts" == "groundnut". The LLM does medical reasoning; code does mechanical invariants. A correction patching one misfire is allowed; a synonym dictionary is not. #### No agent framework No agno, no orchestration engine, no agent runtime. Rosetta is a stateless library on a hot path of thousands of resolutions per patient; per-call object overhead would be a real regression. Resolution is one-shot structured extraction — not agent-shaped. #### No external vocabulary (yet) v1 accepts a cache-covered determinism gap on free-form fields. A future v2 may snap them to RxNorm/UMLS codes loaded as *data* (never code) — explicitly deferred. #### No LLM-emitted display string Ever. `canonical_name` is always derived. And no dependency on N1 application code — the arrow points into Rosetta, always. ## 11Rollout plan The multi-domain work ships as a sequence of self-contained PRs on branch `multi-domain-rosetta`. ```text PRScopeShips value ``` Each PR is independently shippable. Future domains (diagnosis, medication, procedure, cognitive, genetics) follow the same template — one PR each. Diagnosis (ICD-10) and genetics (ClinVar/HGVS) are hardest because their vocabularies are huge; those are the candidates for a data-backed lookup when their time comes. ## 12Glossary 🧬 Rosetta - Complete Architecture Target architecture for n1r-rosetta multi-domain routing v1 — biochem in production, routing & allergy in active development on branch `multi-domain-rosetta`. Supersedes the pre-Rosetta alias-lookup + embedding pipeline — see [Biomarker Grouping Pipeline](../biomarker-grouping/) for historical context. Confidential — N1 internal only.