--- title: "Forge-runner Safe Harbor De-identification Plan (2026-05-09)" --- Forge-runner Safe Harbor De-identification Plan (2026-05-09) # Forge-runner — HIPAA Safe Harbor De-identification Plan Branch `feat/data-field-allowlist-phase2` · HEAD `721aef3` · Plan version 2026-05-09 ### Goal: Safe Harbor compliance — no Expert Determination required Bring forge-runner's on-disk artifacts to HIPAA Safe Harbor** (45 CFR 164.514(b)(2)) so the data agents process is, by regulation, no longer PHI. This lifts BAA-only constraints from downstream agent infrastructure, simplifies LLM provider sourcing, and shrinks the incident blast radius of any workspace leak / log capture / debug dump. Safe Harbor is a structural standard: every one of the 18 listed identifier categories must be removed; no statistical risk assessment required. The alternative — Expert Determination — would let us preserve more precision (e.g., date-shifting) but requires a documented methodology and qualified-expert sign-off, which we are explicitly avoiding. **Hard consequence**: free-text fields drop, dates lose month/day (year only), full DOB becomes age-in-years (90+ aggregated), URLs and human-readable filenames go away. **Time-of-day is preserved** on biomarker rows — it's orthogonal to the date axis and not a "date element" under §C, and it's clinically essential for circadian markers (cortisol, glucose, melatonin). ## Contents - [Workflow impact](#impact) - [Summary table](#summary) - [1. profile → patient_info.json](#profile) - [2. biomarkers → biomarkers.json](#biomarkers) - [3. biomarkers_raw → biomarkers_raw.json](#biomarkers_raw) - [4. canonical_biomarkers → canonical_biomarkers.json](#canonical_biomarkers) - [5. diagnoses → diagnoses.json](#diagnoses) - [6. procedures → procedures.json](#procedures) - [7. medications → medications.json](#medications) - [8. records → records.json](#records) - [Phased rollout](#rollout) ### Reading this plan Each entity has up to three sections. The state badge tells you what's happening to a field: ```text KEPT stays on disk, unchanged — already Safe Harbor compliant TRANSFORMED reshaped at the processor (e.g., `date` → `year`); old field gone, new field replaces it DROPPED removed entirely — no replacement ``` Within **DROPPED**, fields are split by reason: ```text HIPAA dropped because it carries PHI risk under Safe Harbor UNUSED dropped because no agent / workflow consumer reads it ``` ### One filter layer — config declares, processor enforces There is exactly one place where fields land on disk and exactly one place that decides which: the processor reads a declarative allowlist from `config.py` and applies it while building the output dict. No second post-filter, no env override, no "what the CHR sees can drift from what config says." - **`config.py` — declarative spec.** One Python literal per entity stating the fields a CHR may see. Easy to audit, easy to diff in PR review. - **`processors/*.py` — single execution point.** Reshapes the DB row, runs imperative transforms (DOB → age, date → year, file_name → token), and emits exactly the fields config allows — nothing more. CHRs consume this output as-is; the data is safe by construction. **Removed in this rollout:** the `field_projector` post-step in `loader.py`, the `FIELD_ALLOWLISTS_ENV` override, and the `allowed_fields()` resolution helper as it lives today (it gets called by the processor instead of by the loader). One way of doing it. `record_id` is implicitly preserved on biomarkers / biomarkers_raw / diagnoses / procedures / medications regardless of allowlist contents (`filters.filter_patient_data` depends on it). Random UUIDs are permitted under §R via the re-identification code exception in 164.514(c) — they don't encode patient info and the mapping back is never disclosed alongside the data. ## Workflow impact What breaks for downstream consumers, and the mitigation. ```text ConsumerWhat changesMitigation `age-me`, `longevity-report`Biomarker dates collapse to year (month/day gone); `time_of_day` is preserved.Charts: x-axis becomes `year` (with optional jitter from `time_of_day` for same-year tests). Year-over-year trajectory shape preserved. If intra-year ordering ever becomes load-bearing for a feature, revisit with a derived `days_since_prior` field — not needed today. `precision-clinical-health-analysis`Same year-only intra-year ordering caveatSame mitigation Circadian biomarker analysis (cortisol, glucose, melatonin)`time_of_day` stays — pattern preserved.No change needed. Cortisol awakening response, fasting vs postprandial glucose, and melatonin diurnal patterns all rely on time-of-day, not calendar date. `workflow-abc` (`healthcare_agent`)`Medication.notes` is populated from `medications.notes` via `fetch_patient_data` — becomes emptyEither workflow-abc maps notes-derived info to structured fields (dosage / frequency already cover the common case), or accepts `None` workflow-abc / VHI citation rendering`records..file_name` changes from real filename to `record_.pdf` token; `records..url` goes away**Zero code change needed**. `display_name = record_name or file_name` chain works with the synthetic token. `if url:` branch in `references.py:402` already handles missing URL — citations render as `Source: record_3f8a9c21.pdf (page 3)`. Any UI showing "diagnosed on Mar 15, 2024"Year onlyUI shows "diagnosed in 2024" Derived markers (clinical MCP)Unaffected — needs age, height, weight, biomarker values; all still flowVerify: any derived marker depending on exact age in days? If yes, flag. Citation page numbersUnaffected — `page_number` stays— ``` ## Summary Field counts: current allowlist vs Safe Harbor target. ```text EntityDB columnsProcessor emits (current)On disk nowSafe Harbor targetNet change `profile`12 (per-row) + open name space444 (dob → age_years)1 transform `biomarkers` (slim)33191919 + 1 new (date → year + time_of_day)1 transform, 1 added `biomarkers_raw`33222222 + 1 new (test_date → test_year + time_of_day)1 transform, 1 added `canonical_biomarkers`20181616none `diagnoses`22121010 (2 dates → 2 years)2 transforms `procedures`201088 (1 date → 1 year)1 transform `medications`24131311 (2 dates → 2 years; drop reason + notes)2 transforms, 2 drops `records`263 + outer `user_id`42 (drop user_id + url; transform file_name)1 transform, 2 drops TOTAL~1901019692 emitted (and 2 new interval fields)9 transforms, 4 drops ``` ## 1. `profile` → patient_info.json (processors/demographics.py · DB: UserProfileAttribute rows) ### KEPT patient_info.json 3 fields stay as-is FieldTypeSource / reason ```text patient.genderstr← `attribute_name=gender`. Used by clinical reference ranges and recommendations. patient.height_in_cmfloat← `attribute_name=height_in_cm`. BMI, dosing, longevity calcs. patient.weight_in_kgfloat← `attribute_name=weight_in_kg`. BMI, dosing, longevity calcs. ``` ### TRANSFORMED 1 field reshaped at the processor Old fieldNew fieldReason ```text patient.dob (date)→patient.age_years (int | "90+")§C — full date forbidden; 90+ aggregated. Computed from `attribute_name=dob` at processor; original date never emitted. CHRs render as-is — type is `int | str`. ``` ### DROPPED 12 fields removed entirely #### HIPAA PHI risk — must not land on disk FieldTypeReason ```json user_idUUID§R direct patient identifier. Hard processor-layer drop. idUUIDPer-attribute row UUID — joinable to patient via DB; not emitted in reshape. attribute_timestampdatetime§C — date the attribute was set, tied to the individual. attribute_name (other values)strAny row whose `attribute_name` is not in {dob, gender, height_in_cm, weight_in_kg} is filtered out — covers ethnicity, smoking_status, occupation, ZIP (§B), contact info (§D/F), etc. ``` #### Unused no agent consumer; dropped to keep on-disk surface minimal FieldTypeReason ```text attribute_namestrFilter key, not emitted. Only rows with allowed names are honored. attribute_valuestrRemapped onto the corresponding `patient.*` key; never emitted as-is. attribute_typestrStorage typing metadata; reshape hardcodes types per attribute. attribute_groupstrInternal grouping metadata; not consumed downstream. attribute_orderintUI ordering hint. attribute_unitstrPer-attribute unit; reshape hardcodes `cm` / `kg`. validation_rulestrBackend validation metadata. related_attributeslist[str]Cross-reference metadata between attributes. ``` ## 2. `biomarkers` → biomarkers.json (processors/biomarkers.py:slim_biomarkers · DB: UserBiomarker) ### KEPT biomarkers.json 18 fields stay as-is FieldTypeSource / reason ```text idUUIDRandom UUID — §R re-id code exception (164.514(c)). record_idstrMandatory for filters; UUID, same exception. page_numberintDocument pagination — used by citation rendering. name, canonical_name, canonical_idstr / UUIDClinical concept / reference; not patient-linked. result, num, unit, ref, ref_min, ref_maxstr / floatClinical values, no PHI. abnormalbool← `out_of_range`. health_areas, group_name, group_key, medical_specialties, samplestr / list[str]Reference data, no PHI. ``` ### TRANSFORMED 1 field reshaped + 1 new time field Old fieldNew field(s)Reason ```text date (date / datetime)→year (int) + time_of_day (str, "HH:MM" or "HH:MM:SS")§C — month/day forbidden as date elements. Time-of-day is orthogonal to the date axis and is preserved at source precision: clinically essential for circadian biomarkers (cortisol diurnal pattern, fasting vs postprandial glucose, melatonin). If the source row has no time, `time_of_day` is `null`. ``` ### DROPPED 12+ fields removed entirely #### HIPAA PHI risk — must not land on disk FieldTypeReason ```text user_idUUID§R direct patient identifier. created_at, updated_atdatetime§C audit timestamps tied to the individual. record_namestr§A — source filename can carry the patient's name (e.g. "JaneDoe_LabCorp.pdf"). `record_id` covers the join. contextstrFree-text narrative — structural PHI catch-all (§A/B/C/D/F/H). additional_datadictCatch-all dict — can hold any narrative-PHI; structurally unsafe. ``` #### Unused no agent consumer; dropped to keep on-disk surface minimal FieldTypeReason ```text reviewed, reviewed_by, reviewed_onbool / UUID / datetimeInternal QA workflow fields; no consumer reads them. expanded_test_namestrLab's expanded form (e.g. "Glucose, fasting, plasma") — kept in raw shape only; slim uses `name`. clinical_identity_idUUIDID-system provenance; in slim it's collapsed into `canonical_id`. edited, excludedboolInternal workflow flags. methodstrLOINC method override — only `canonical_biomarkers` exposes this. result_numeric, reference_range_min, reference_range_maxfloatConsumed by reshape — emitted as `num` / `ref_min` / `ref_max` fallbacks. converted_*floatConsumed by reshape — emitted as primary `num` / `ref_*` values. biomarker_hashstrInternal dedup hash; backend-only signal. ``` ## 3. `biomarkers_raw` → biomarkers_raw.json (processors/biomarkers.py:raw_biomarkers · DB: UserBiomarker) ### KEPT biomarkers_raw.json 21 fields stay as-is FieldTypeSource / reason ```text id, record_idUUID / strUUIDs — §R re-id exception. page_numberintPagination — used by citation rendering. test_name, expanded_test_namestrClinical concept. result, result_numeric, converted_result_numeric, unit, converted_unitstr / floatClinical values. reference_range, reference_range_min/max, converted_reference_range_min/maxstr / floatReference ranges. out_of_rangeboolFlag. sample_sourcestrSample type. canonical_id, clinical_identity_id, legacy_canonical_idUUIDID-system provenance — not patient identifiers. canonical_recorddictNested clinical reference; no PHI inside. ``` ### TRANSFORMED 1 field reshaped + 1 new time field Old fieldNew field(s)Reason ```text test_date (date / datetime)→test_year (int) + time_of_day (str, "HH:MM" or "HH:MM:SS")§C — same approach as slim. Year-of(`test_date`) plus the time-of-day component (preserved at source precision; `null` if absent). ``` ### DROPPED 9+ fields removed entirely #### HIPAA PHI risk — must not land on disk FieldTypeReason ```text user_idUUID§R direct patient identifier. created_at, updated_atdatetime§C audit timestamps tied to the individual. record_namestr§A — source filename can carry the patient's name; `record_id` covers the join. contextstrFree-text narrative — structural PHI catch-all. additional_datadictCatch-all dict — can hold any narrative-PHI; structurally unsafe. ``` #### Unused no agent consumer; dropped to keep on-disk surface minimal FieldTypeReason ```text reviewed, reviewed_by, reviewed_onbool / UUID / datetimeInternal QA workflow fields. edited, excludedboolInternal workflow flags. methodstrLOINC method — only `canonical_biomarkers` exposes this. biomarker_hashstrInternal dedup hash; backend-only signal. ``` ## 4. `canonical_biomarkers` → canonical_biomarkers.json (processors/canonical_biomarkers.py · DB: BiomarkerClinicalIdentity) **No Safe Harbor change.** Reference data, not patient-specific. Allowlist already drops two unused QA fields. ### KEPT canonical_biomarkers.json 16 fields stay as-is FieldTypeReason ```text id, canonical_name, group_key, component, property, system, method, time_aspect, qualifier, standard_unit, reference_range_min/max, health_areas, medical_specialties, group_name, sample_source— mixed —Reference / clinical concept data; no PHI. ``` ### DROPPED 4 fields removed entirely #### Unused reference data only — nothing here is patient-derived FieldTypeReason ```text member_countintInternal stat — zero references. reviewedboolInternal QA flag — zero references. created_at, updated_atdatetimeReference-data audit timestamps; not patient-tied, no consumer. ``` ## 5. `diagnoses` → diagnoses.json (processors/diagnoses.py · DB: UserDiagnosis) ### KEPT diagnoses.json 8 fields stay as-is FieldTypeSource / reason ```text id, record_idUUID / strUUIDs — §R exception. page_numberintPagination — used by citation rendering. name, statusstrClinical label, status enum. severitystrClinical. icd_code, snomed_codestrStandardized clinical codes. Rare ICDs raise re-ID risk in combination — tolerable under Safe Harbor's structural standard. ``` ### TRANSFORMED 2 fields reshaped at the processor Old fieldNew fieldReason ```text date_diagnosed (date)→year_diagnosed (int)§C — year-only sufficient (intra-year ordering rarely matters clinically for diagnoses). date_resolved (date)→year_resolved (int | null)§C — null when unresolved. ``` ### DROPPED 8+ fields removed entirely #### HIPAA PHI risk — must not land on disk FieldTypeReason ```text presentation_patternstrFree-text narrative — structural PHI catch-all. explanationstrFree-text narrative — structural PHI catch-all. user_idUUID§R direct patient identifier. created_at, updated_atdatetime§C audit timestamps tied to the individual. record_namestr§A — source filename can carry the patient's name; `record_id` covers the join. additional_datadictCatch-all dict — can hold any narrative-PHI; structurally unsafe. ``` #### Unused no agent consumer; dropped to keep on-disk surface minimal FieldTypeReason ```text created_by, reviewed_byUUIDStaff identities — not patient identifiers, no agent consumer. reviewed, reviewed_onbool / datetimeInternal QA workflow fields. canonical_diagnosis_idUUIDReference to a canonical diagnosis entity; no current consumer reads it. ``` ## 6. `procedures` → procedures.json (processors/procedures.py · DB: UserProcedure) ### KEPT procedures.json 7 fields stay as-is FieldTypeSource / reason ```text id, record_idUUID / strUUIDs — §R exception. page_numberintPagination — used by citation rendering. name, cpt_code, outcome, impact_score— mixed —Clinical concepts and codes. ``` ### TRANSFORMED 1 field reshaped at the processor Old fieldNew fieldReason ```text date_performed (date)→year_performed (int)§C — year-only. ``` ### DROPPED 7+ fields removed entirely #### HIPAA PHI risk — must not land on disk FieldTypeReason ```text descriptionstrFree-text narrative — structural PHI catch-all. explanationstrFree-text narrative — structural PHI catch-all. user_idUUID§R direct patient identifier. created_at, updated_atdatetime§C audit timestamps tied to the individual. record_namestr§A — source filename can carry the patient's name; `record_id` covers the join. additional_datadictCatch-all dict — can hold any narrative-PHI; structurally unsafe. ``` #### Unused no agent consumer; dropped to keep on-disk surface minimal FieldTypeReason ```text created_by, reviewed_byUUIDStaff identities — not patient identifiers, no agent consumer. reviewed, reviewed_onbool / datetimeInternal QA workflow fields. ``` ## 7. `medications` → medications.json (processors/medications.py · DB: UserMedication) **Biggest change in the plan:** two free-text fields are dropped entirely (no replacement). Safe Harbor is structural, not statistical — free-text fields can hold names (§A), addresses (§B), full dates (§C), phone numbers (§D), email (§F), MRNs (§H). You cannot enumerate what's in them, so the only Safe Harbor-compliant move is removal. PHI scrubbers (Presidio, AWS Comprehend Medical) are statistical and would push us into Expert Determination territory. ### KEPT medications.json 9 fields stay as-is FieldTypeSource / reason ```text id, record_idUUID / strUUIDs — §R exception. page_numberintPagination — used by citation rendering. name, brand_name, dosage, unit, type, frequency— mixed —Structured clinical fields. Drug name is a clinical concept, not a patient identifier. ``` ### TRANSFORMED 2 fields reshaped at the processor Old fieldNew fieldReason ```text started_from (date)→year_started (int)§C — year-only. stopped_on (date)→year_stopped (int | null)§C — null while ongoing. ``` ### DROPPED 8+ fields removed entirely #### HIPAA PHI risk — must not land on disk FieldTypeReason ```text reasonstrFree-text narrative — structural PHI catch-all. Could come back as a controlled-vocabulary field if the parser emits ICD codes. notesstrFree-text narrative — highest-risk catch-all. workflow-abc consumer migration required (`Medication.notes` becomes empty). user_idUUID§R direct patient identifier. created_at, updated_atdatetime§C audit timestamps tied to the individual. record_namestr§A — source filename can carry the patient's name; `record_id` covers the join. additional_datadictCatch-all dict — can hold any narrative-PHI. Read selectively for `notes` only (and that read goes away in Phase 3d). ``` #### Unused no agent consumer; dropped to keep on-disk surface minimal FieldTypeReason ```text created_by, reviewed_byUUIDStaff identities — not patient identifiers, no agent consumer. reviewed, reviewed_onbool / datetimeInternal QA workflow fields. urlstrDrug-label URL (public reference like RxList, not patient-specific). Not consumed by agents. statusstrWorkflow status flag (active / discontinued); `started_from` + `stopped_on` already carry the clinical signal. ``` ## 8. `records` → records.json (fetchers.py:_fetch_records_cache · DB: N1RecordState) **Verified safe to drop:** - **`file_name` usage** — only as a display label in `workflows/workflow-abc/.../utils/references.py` (lines 393, 435, 1001) and `workflows/visual-health-intelligence/scripts/common.py` (lines 192, 407). Pattern: `display_name = record_name or file_name`. Synthetic token works as drop-in. - **`url` usage** — same files, only to make citations clickable (`[name](url#page=3)`). The `if url:` branches at `references.py:402` already gracefully handle absence. Plain-text citations result. - **`user_id` usage** — read by workflow-abc's `_load_json_file` setdefault but only as a value preserved through the citation pipeline; nothing requires the patient's actual UUID. Empty string works. ### KEPT records.json 1 field stays as-is FieldTypeSource / reason ```text records.str (key)UUID — §R exception. Per-record map key. ``` ### TRANSFORMED 1 field reshaped at the processor Old fieldNew fieldReason ```text records..file_name (real, e.g. "JaneDoe_labs.pdf")→records..file_name = "record_" + record_id[:8] + ".pdf"§A — real filename can carry the patient's name. Token derived deterministically from the UUID — already non-identifying. Citation UX preserved (`Source: record_3f8a9c21.pdf (page 3)`). ``` ### DROPPED 8+ fields removed entirely #### HIPAA PHI risk — must not land on disk FieldTypeReason ```text user_id (top-level)str§R direct patient identifier. records..urlstr§N URL — also dereferences to the source PDF, which is itself PHI. created_by, user_id (per-record)UUID§R — patient / uploader UUIDs. created_at, updated_at, test_datedatetime§C date metadata tied to the individual. original_file_name, original_content_typestr§A — original filename carries the same name risk as `file_name`. ``` #### Unused no agent consumer; dropped to keep on-disk surface minimal FieldTypeReason ```text progress, error, message, status, type_, version— mixed —Parser status fields. file_hash, task_id, batch_id, meta— mixed —Internal parser metadata. progress_stage, error_details, is_update_counter, status_history, was_converted, page_count— mixed —Internal parser state. ``` ## Phased rollout Each phase is a separate PR; all target `develop`. ```text PhaseScopeRiskBlocks on **3a**`profile.dob` → `age_years` (with 90+ aggregation). New `infrastructure/data/deidentify.py` helper module.Low — only age-me / longevity-report consume age, and they already do day→year math from DOB.— **3b**Date → year on `diagnoses`, `procedures`, `medications`. Year-only, no interval encoding.Medium — UI / report templates rendering full dates need updates.3a **3c**`biomarkers` + `biomarkers_raw`: `date` → `year` + `time_of_day`. Source `UserBiomarker.test_date` is already `datetime.datetime`; the time component is currently thrown away by `date_value()`'s `[:10]` truncation in `processors/common.py`. Replace that helper with `year_of()` + `time_of_day_of()` for biomarker rows. No parser/API change needed.Low — clinical signal is already upstream; we stop discarding it. Charts gain time-of-day structure that wasn't there before.3b **3d**Drop `medications.reason` + `medications.notes`. workflow-abc `Medication.notes` becomes empty.Medium — coordination with workflow-abc; pin a regression test that asserts notes is empty.workflow-abc PR accepting empty notes **3e**`records`: drop top-level `user_id`, drop `records..url`, token-replace `file_name`. Needs the deferred nested-map projector OR a processor-side rewrite of the records bundle.Low — citation rendering verified to gracefully degrade.— **3f**CI guards: regex test for `YYYY-MM-DD` patterns in any output JSON; fixture with deliberately PHI-shaped narrative content + assertion that none appears on disk.Low — pure test code.3a–3e **3g****Architecture cleanup.** Move allowlist enforcement into the processor (processor reads `config.DEFAULT_FIELD_ALLOWLISTS` and emits exactly that). Delete `field_projector` from `loader.py`, delete `FIELD_ALLOWLISTS_ENV` + `_parse_allowlist_env`, delete the field-projector warning path. One config, one execution point.Low — pure refactor; covered by existing loader tests + new test asserting on-disk JSON matches the config literal exactly.3a–3e **3h**Replace `FIELD_AUDIT.md` with a Safe Harbor compliance matrix. Add `infrastructure/data/SAFE_HARBOR.md` covering scope, "actual knowledge" attestation (§(b)(2)(ii)), residual risks, annual review process.Low — documentation only.3a–3g ``` ## Open decisions before implementing 1. **Biomarker time encoding — confirmed: `year + time_of_day`**. Time-of-day is preserved at source precision because it's orthogonal to the date axis under §C and clinically essential for circadian markers. Same-year intra-year ordering is currently unsupported; revisit with `days_since_prior` only if a feature actually requires it. 1. **Source-data check pending**: the current biomarkers `date` output type is `date` (no time) — verify whether the parser/DB actually carries time-of-day before Phase 3c. If not, time_of_day will simply always be `null` until the parser is updated; the schema is forward-compatible. 1. **`medications.notes` migration — sequence with workflow-abc**: drop after workflow-abc accepts empty `Medication.notes` (Phase 3d gate), or drop now and let workflow-abc adapt? Recommendation: stage workflow-abc PR first to avoid a broken release window. 1. **`file_name` token format — confirmed: `record_.pdf`**. Looks like a filename to citation rendering; deterministic from the UUID; non-identifying. 1. **Age 90+ representation — confirmed: string `"90+"`**, integer otherwise; CHRs render the field as-is. 1. **"Actual knowledge" attestation** — Safe Harbor §(b)(2)(ii) requires the covered entity to attest no actual knowledge that the data could be re-identified. Who signs? When? Annually? This is a process question, not a code question. Plan generated 2026-05-09 from `infrastructure/data/config.py` (`DEFAULT_FIELD_ALLOWLISTS`), `infrastructure/data/processors/*.py`, the `n1_api_client.models.*` attrs definitions, `workflows/workflow-abc/.../utils/references.py`, and `workflows/visual-health-intelligence/scripts/common.py`. Branch `feat/data-field-allowlist-phase2` at HEAD `721aef3`. Update this file alongside any change to those sources or the plan.