Forge-runner Safe Harbor De-identification Plan (2026-05-09)

Forge-runner Safe Harbor De-identification Plan (2026-05-09)

Forge-runner — HIPAA Safe Harbor De-identification Plan

Branch feat/data-field-allowlist-phase2 · HEAD 721aef3 · Plan version 2026-05-09

Goal: Safe Harbor compliance — no Expert Determination required

Bring forge-runner’s on-disk artifacts to HIPAA Safe Harbor** (45 CFR 164.514(b)(2)) so the data agents process is, by regulation, no longer PHI. This lifts BAA-only constraints from downstream agent infrastructure, simplifies LLM provider sourcing, and shrinks the incident blast radius of any workspace leak / log capture / debug dump.

Safe Harbor is a structural standard: every one of the 18 listed identifier categories must be removed; no statistical risk assessment required. The alternative — Expert Determination — would let us preserve more precision (e.g., date-shifting) but requires a documented methodology and qualified-expert sign-off, which we are explicitly avoiding.

Hard consequence: free-text fields drop, dates lose month/day (year only), full DOB becomes age-in-years (90+ aggregated), URLs and human-readable filenames go away. Time-of-day is preserved on biomarker rows — it’s orthogonal to the date axis and not a “date element” under §C, and it’s clinically essential for circadian markers (cortisol, glucose, melatonin).

Contents

• Workflow impact
• Summary table
• 1. profile → patient_info.json
• 2. biomarkers → biomarkers.json
• 3. biomarkers_raw → biomarkers_raw.json
• 4. canonical_biomarkers → canonical_biomarkers.json
• 5. diagnoses → diagnoses.json
• 6. procedures → procedures.json
• 7. medications → medications.json
• 8. records → records.json
• Phased rollout

Reading this plan

Each entity has up to three sections. The state badge tells you what’s happening to a field:

KEPT stays on disk, unchanged — already Safe Harbor compliant
TRANSFORMED reshaped at the processor (e.g., `date` → `year`); old field gone, new field replaces it
DROPPED removed entirely — no replacement

Within DROPPED, fields are split by reason:

HIPAA dropped because it carries PHI risk under Safe Harbor
UNUSED dropped because no agent / workflow consumer reads it

One filter layer — config declares, processor enforces

There is exactly one place where fields land on disk and exactly one place that decides which: the processor reads a declarative allowlist from config.py and applies it while building the output dict. No second post-filter, no env override, no “what the CHR sees can drift from what config says.”

• config.py — declarative spec. One Python literal per entity stating the fields a CHR may see. Easy to audit, easy to diff in PR review.
• processors/*.py — single execution point. Reshapes the DB row, runs imperative transforms (DOB → age, date → year, file_name → token), and emits exactly the fields config allows — nothing more. CHRs consume this output as-is; the data is safe by construction.

Removed in this rollout: the field_projector post-step in loader.py, the FIELD_ALLOWLISTS_ENV override, and the allowed_fields() resolution helper as it lives today (it gets called by the processor instead of by the loader). One way of doing it.

record_id is implicitly preserved on biomarkers / biomarkers_raw / diagnoses / procedures / medications regardless of allowlist contents (filters.filter_patient_data depends on it). Random UUIDs are permitted under §R via the re-identification code exception in 164.514(c) — they don’t encode patient info and the mapping back is never disclosed alongside the data.

Workflow impact

What breaks for downstream consumers, and the mitigation.

ConsumerWhat changesMitigation

`age-me`, `longevity-report`Biomarker dates collapse to year (month/day gone); `time_of_day` is preserved.Charts: x-axis becomes `year` (with optional jitter from `time_of_day` for same-year tests). Year-over-year trajectory shape preserved. If intra-year ordering ever becomes load-bearing for a feature, revisit with a derived `days_since_prior` field — not needed today.

`precision-clinical-health-analysis`Same year-only intra-year ordering caveatSame mitigation

Circadian biomarker analysis (cortisol, glucose, melatonin)`time_of_day` stays — pattern preserved.No change needed. Cortisol awakening response, fasting vs postprandial glucose, and melatonin diurnal patterns all rely on time-of-day, not calendar date.

`workflow-abc` (`healthcare_agent`)`Medication.notes` is populated from `medications.notes` via `fetch_patient_data` — becomes emptyEither workflow-abc maps notes-derived info to structured fields (dosage / frequency already cover the common case), or accepts `None`

workflow-abc / VHI citation rendering`records.<id>.file_name` changes from real filename to `record_<short-uuid>.pdf` token; `records.<id>.url` goes away**Zero code change needed**. `display_name = record_name or file_name` chain works with the synthetic token. `if url:` branch in `references.py:402` already handles missing URL — citations render as `Source: record_3f8a9c21.pdf (page 3)`.

Any UI showing "diagnosed on Mar 15, 2024"Year onlyUI shows "diagnosed in 2024"

Derived markers (clinical MCP)Unaffected — needs age, height, weight, biomarker values; all still flowVerify: any derived marker depending on exact age in days? If yes, flag.

Citation page numbersUnaffected — `page_number` stays—

Summary

Field counts: current allowlist vs Safe Harbor target.

EntityDB columnsProcessor emits (current)On disk nowSafe Harbor targetNet change

`profile`12 (per-row) + open name space444 (dob → age_years)1 transform
`biomarkers` (slim)33191919 + 1 new (date → year + time_of_day)1 transform, 1 added
`biomarkers_raw`33222222 + 1 new (test_date → test_year + time_of_day)1 transform, 1 added
`canonical_biomarkers`20181616none
`diagnoses`22121010 (2 dates → 2 years)2 transforms
`procedures`201088 (1 date → 1 year)1 transform
`medications`24131311 (2 dates → 2 years; drop reason + notes)2 transforms, 2 drops
`records`263 + outer `user_id`42 (drop user_id + url; transform file_name)1 transform, 2 drops
TOTAL~1901019692 emitted (and 2 new interval fields)9 transforms, 4 drops

1. profile → patient_info.json (processors/demographics.py · DB: UserProfileAttribute rows)

KEPT patient_info.json 3 fields stay as-is

FieldTypeSource / reason

patient.genderstr← `attribute_name=gender`. Used by clinical reference ranges and recommendations.
patient.height_in_cmfloat← `attribute_name=height_in_cm`. BMI, dosing, longevity calcs.
patient.weight_in_kgfloat← `attribute_name=weight_in_kg`. BMI, dosing, longevity calcs.

TRANSFORMED 1 field reshaped at the processor

Old fieldNew fieldReason

patient.dob (date)→patient.age_years (int | "90+")§C — full date forbidden; 90+ aggregated. Computed from `attribute_name=dob` at processor; original date never emitted. CHRs render as-is — type is `int | str`.

DROPPED 12 fields removed entirely

HIPAA PHI risk — must not land on disk

FieldTypeReason

user_idUUID§R direct patient identifier. Hard processor-layer drop.
idUUIDPer-attribute row UUID — joinable to patient via DB; not emitted in reshape.
attribute_timestampdatetime§C — date the attribute was set, tied to the individual.
attribute_name (other values)strAny row whose `attribute_name` is not in {dob, gender, height_in_cm, weight_in_kg} is filtered out — covers ethnicity, smoking_status, occupation, ZIP (§B), contact info (§D/F), etc.

Unused no agent consumer; dropped to keep on-disk surface minimal

FieldTypeReason

attribute_namestrFilter key, not emitted. Only rows with allowed names are honored.
attribute_valuestrRemapped onto the corresponding `patient.*` key; never emitted as-is.
attribute_typestrStorage typing metadata; reshape hardcodes types per attribute.
attribute_groupstrInternal grouping metadata; not consumed downstream.
attribute_orderintUI ordering hint.
attribute_unitstrPer-attribute unit; reshape hardcodes `cm` / `kg`.
validation_rulestrBackend validation metadata.
related_attributeslist[str]Cross-reference metadata between attributes.

2. biomarkers → biomarkers.json (processors/biomarkers.py:slim_biomarkers · DB: UserBiomarker)

KEPT biomarkers.json 18 fields stay as-is

FieldTypeSource / reason

idUUIDRandom UUID — §R re-id code exception (164.514(c)).
record_idstrMandatory for filters; UUID, same exception.
page_numberintDocument pagination — used by citation rendering.
name, canonical_name, canonical_idstr / UUIDClinical concept / reference; not patient-linked.
result, num, unit, ref, ref_min, ref_maxstr / floatClinical values, no PHI.
abnormalbool← `out_of_range`.
health_areas, group_name, group_key, medical_specialties, samplestr / list[str]Reference data, no PHI.

TRANSFORMED 1 field reshaped + 1 new time field

Old fieldNew field(s)Reason

date (date / datetime)→year (int) + time_of_day (str, "HH:MM" or "HH:MM:SS")§C — month/day forbidden as date elements. Time-of-day is orthogonal to the date axis and is preserved at source precision: clinically essential for circadian biomarkers (cortisol diurnal pattern, fasting vs postprandial glucose, melatonin). If the source row has no time, `time_of_day` is `null`.

DROPPED 12+ fields removed entirely

HIPAA PHI risk — must not land on disk

FieldTypeReason

user_idUUID§R direct patient identifier.
created_at, updated_atdatetime§C audit timestamps tied to the individual.
record_namestr§A — source filename can carry the patient's name (e.g. "JaneDoe_LabCorp.pdf"). `record_id` covers the join.
contextstrFree-text narrative — structural PHI catch-all (§A/B/C/D/F/H).
additional_datadictCatch-all dict — can hold any narrative-PHI; structurally unsafe.

Unused no agent consumer; dropped to keep on-disk surface minimal

FieldTypeReason

reviewed, reviewed_by, reviewed_onbool / UUID / datetimeInternal QA workflow fields; no consumer reads them.
expanded_test_namestrLab's expanded form (e.g. "Glucose, fasting, plasma") — kept in raw shape only; slim uses `name`.
clinical_identity_idUUIDID-system provenance; in slim it's collapsed into `canonical_id`.
edited, excludedboolInternal workflow flags.
methodstrLOINC method override — only `canonical_biomarkers` exposes this.
result_numeric, reference_range_min, reference_range_maxfloatConsumed by reshape — emitted as `num` / `ref_min` / `ref_max` fallbacks.
converted_*floatConsumed by reshape — emitted as primary `num` / `ref_*` values.
biomarker_hashstrInternal dedup hash; backend-only signal.

3. biomarkers_raw → biomarkers_raw.json (processors/biomarkers.py:raw_biomarkers · DB: UserBiomarker)

KEPT biomarkers_raw.json 21 fields stay as-is

FieldTypeSource / reason

id, record_idUUID / strUUIDs — §R re-id exception.
page_numberintPagination — used by citation rendering.
test_name, expanded_test_namestrClinical concept.
result, result_numeric, converted_result_numeric, unit, converted_unitstr / floatClinical values.
reference_range, reference_range_min/max, converted_reference_range_min/maxstr / floatReference ranges.
out_of_rangeboolFlag.
sample_sourcestrSample type.
canonical_id, clinical_identity_id, legacy_canonical_idUUIDID-system provenance — not patient identifiers.
canonical_recorddictNested clinical reference; no PHI inside.

TRANSFORMED 1 field reshaped + 1 new time field

Old fieldNew field(s)Reason

test_date (date / datetime)→test_year (int) + time_of_day (str, "HH:MM" or "HH:MM:SS")§C — same approach as slim. Year-of(`test_date`) plus the time-of-day component (preserved at source precision; `null` if absent).

DROPPED 9+ fields removed entirely

HIPAA PHI risk — must not land on disk

FieldTypeReason

user_idUUID§R direct patient identifier.
created_at, updated_atdatetime§C audit timestamps tied to the individual.
record_namestr§A — source filename can carry the patient's name; `record_id` covers the join.
contextstrFree-text narrative — structural PHI catch-all.
additional_datadictCatch-all dict — can hold any narrative-PHI; structurally unsafe.

Unused no agent consumer; dropped to keep on-disk surface minimal

FieldTypeReason

reviewed, reviewed_by, reviewed_onbool / UUID / datetimeInternal QA workflow fields.
edited, excludedboolInternal workflow flags.
methodstrLOINC method — only `canonical_biomarkers` exposes this.
biomarker_hashstrInternal dedup hash; backend-only signal.

4. canonical_biomarkers → canonical_biomarkers.json (processors/canonical_biomarkers.py · DB: BiomarkerClinicalIdentity)

No Safe Harbor change. Reference data, not patient-specific. Allowlist already drops two unused QA fields.

KEPT canonical_biomarkers.json 16 fields stay as-is

FieldTypeReason

id, canonical_name, group_key, component, property, system, method, time_aspect, qualifier, standard_unit, reference_range_min/max, health_areas, medical_specialties, group_name, sample_source— mixed —Reference / clinical concept data; no PHI.

DROPPED 4 fields removed entirely

Unused reference data only — nothing here is patient-derived

FieldTypeReason

member_countintInternal stat — zero references.
reviewedboolInternal QA flag — zero references.
created_at, updated_atdatetimeReference-data audit timestamps; not patient-tied, no consumer.

5. diagnoses → diagnoses.json (processors/diagnoses.py · DB: UserDiagnosis)

KEPT diagnoses.json 8 fields stay as-is

FieldTypeSource / reason

id, record_idUUID / strUUIDs — §R exception.
page_numberintPagination — used by citation rendering.
name, statusstrClinical label, status enum.
severitystrClinical.
icd_code, snomed_codestrStandardized clinical codes. Rare ICDs raise re-ID risk in combination — tolerable under Safe Harbor's structural standard.

TRANSFORMED 2 fields reshaped at the processor

Old fieldNew fieldReason

date_diagnosed (date)→year_diagnosed (int)§C — year-only sufficient (intra-year ordering rarely matters clinically for diagnoses).
date_resolved (date)→year_resolved (int | null)§C — null when unresolved.

DROPPED 8+ fields removed entirely

HIPAA PHI risk — must not land on disk

FieldTypeReason

presentation_patternstrFree-text narrative — structural PHI catch-all.
explanationstrFree-text narrative — structural PHI catch-all.
user_idUUID§R direct patient identifier.
created_at, updated_atdatetime§C audit timestamps tied to the individual.
record_namestr§A — source filename can carry the patient's name; `record_id` covers the join.
additional_datadictCatch-all dict — can hold any narrative-PHI; structurally unsafe.

Unused no agent consumer; dropped to keep on-disk surface minimal

FieldTypeReason

created_by, reviewed_byUUIDStaff identities — not patient identifiers, no agent consumer.
reviewed, reviewed_onbool / datetimeInternal QA workflow fields.
canonical_diagnosis_idUUIDReference to a canonical diagnosis entity; no current consumer reads it.

6. procedures → procedures.json (processors/procedures.py · DB: UserProcedure)

KEPT procedures.json 7 fields stay as-is

FieldTypeSource / reason

id, record_idUUID / strUUIDs — §R exception.
page_numberintPagination — used by citation rendering.
name, cpt_code, outcome, impact_score— mixed —Clinical concepts and codes.

TRANSFORMED 1 field reshaped at the processor

Old fieldNew fieldReason

date_performed (date)→year_performed (int)§C — year-only.

DROPPED 7+ fields removed entirely

HIPAA PHI risk — must not land on disk

FieldTypeReason

descriptionstrFree-text narrative — structural PHI catch-all.
explanationstrFree-text narrative — structural PHI catch-all.
user_idUUID§R direct patient identifier.
created_at, updated_atdatetime§C audit timestamps tied to the individual.
record_namestr§A — source filename can carry the patient's name; `record_id` covers the join.
additional_datadictCatch-all dict — can hold any narrative-PHI; structurally unsafe.

Unused no agent consumer; dropped to keep on-disk surface minimal

FieldTypeReason

created_by, reviewed_byUUIDStaff identities — not patient identifiers, no agent consumer.
reviewed, reviewed_onbool / datetimeInternal QA workflow fields.

7. medications → medications.json (processors/medications.py · DB: UserMedication)

Biggest change in the plan: two free-text fields are dropped entirely (no replacement). Safe Harbor is structural, not statistical — free-text fields can hold names (§A), addresses (§B), full dates (§C), phone numbers (§D), email (§F), MRNs (§H). You cannot enumerate what’s in them, so the only Safe Harbor-compliant move is removal. PHI scrubbers (Presidio, AWS Comprehend Medical) are statistical and would push us into Expert Determination territory.

KEPT medications.json 9 fields stay as-is

FieldTypeSource / reason

id, record_idUUID / strUUIDs — §R exception.
page_numberintPagination — used by citation rendering.
name, brand_name, dosage, unit, type, frequency— mixed —Structured clinical fields. Drug name is a clinical concept, not a patient identifier.

TRANSFORMED 2 fields reshaped at the processor

Old fieldNew fieldReason

started_from (date)→year_started (int)§C — year-only.
stopped_on (date)→year_stopped (int | null)§C — null while ongoing.

DROPPED 8+ fields removed entirely

HIPAA PHI risk — must not land on disk

FieldTypeReason

reasonstrFree-text narrative — structural PHI catch-all. Could come back as a controlled-vocabulary field if the parser emits ICD codes.
notesstrFree-text narrative — highest-risk catch-all. workflow-abc consumer migration required (`Medication.notes` becomes empty).
user_idUUID§R direct patient identifier.
created_at, updated_atdatetime§C audit timestamps tied to the individual.
record_namestr§A — source filename can carry the patient's name; `record_id` covers the join.
additional_datadictCatch-all dict — can hold any narrative-PHI. Read selectively for `notes` only (and that read goes away in Phase 3d).

Unused no agent consumer; dropped to keep on-disk surface minimal

FieldTypeReason

created_by, reviewed_byUUIDStaff identities — not patient identifiers, no agent consumer.
reviewed, reviewed_onbool / datetimeInternal QA workflow fields.
urlstrDrug-label URL (public reference like RxList, not patient-specific). Not consumed by agents.
statusstrWorkflow status flag (active / discontinued); `started_from` + `stopped_on` already carry the clinical signal.

8. records → records.json (fetchers.py:_fetch_records_cache · DB: N1RecordState)

Verified safe to drop:

• file_name usage — only as a display label in workflows/workflow-abc/.../utils/references.py (lines 393, 435, 1001) and workflows/visual-health-intelligence/scripts/common.py (lines 192, 407). Pattern: display_name = record_name or file_name. Synthetic token works as drop-in.
• url usage — same files, only to make citations clickable ([name](url#page=3)). The if url: branches at references.py:402 already gracefully handle absence. Plain-text citations result.
• user_id usage — read by workflow-abc’s _load_json_file setdefault but only as a value preserved through the citation pipeline; nothing requires the patient’s actual UUID. Empty string works.

KEPT records.json 1 field stays as-is

FieldTypeSource / reason

records.<record_id>str (key)UUID — §R exception. Per-record map key.

TRANSFORMED 1 field reshaped at the processor

Old fieldNew fieldReason

records.<id>.file_name (real, e.g. "JaneDoe_labs.pdf")→records.<id>.file_name = "record_" + record_id[:8] + ".pdf"§A — real filename can carry the patient's name. Token derived deterministically from the UUID — already non-identifying. Citation UX preserved (`Source: record_3f8a9c21.pdf (page 3)`).

DROPPED 8+ fields removed entirely

HIPAA PHI risk — must not land on disk

FieldTypeReason

user_id (top-level)str§R direct patient identifier.
records.<id>.urlstr§N URL — also dereferences to the source PDF, which is itself PHI.
created_by, user_id (per-record)UUID§R — patient / uploader UUIDs.
created_at, updated_at, test_datedatetime§C date metadata tied to the individual.
original_file_name, original_content_typestr§A — original filename carries the same name risk as `file_name`.

Unused no agent consumer; dropped to keep on-disk surface minimal

FieldTypeReason

progress, error, message, status, type_, version— mixed —Parser status fields.
file_hash, task_id, batch_id, meta— mixed —Internal parser metadata.
progress_stage, error_details, is_update_counter, status_history, was_converted, page_count— mixed —Internal parser state.

Phased rollout

Each phase is a separate PR; all target develop.

PhaseScopeRiskBlocks on

**3a**`profile.dob` → `age_years` (with 90+ aggregation). New `infrastructure/data/deidentify.py` helper module.Low — only age-me / longevity-report consume age, and they already do day→year math from DOB.—
**3b**Date → year on `diagnoses`, `procedures`, `medications`. Year-only, no interval encoding.Medium — UI / report templates rendering full dates need updates.3a
**3c**`biomarkers` + `biomarkers_raw`: `date` → `year` + `time_of_day`. Source `UserBiomarker.test_date` is already `datetime.datetime`; the time component is currently thrown away by `date_value()`'s `[:10]` truncation in `processors/common.py`. Replace that helper with `year_of()` + `time_of_day_of()` for biomarker rows. No parser/API change needed.Low — clinical signal is already upstream; we stop discarding it. Charts gain time-of-day structure that wasn't there before.3b
**3d**Drop `medications.reason` + `medications.notes`. workflow-abc `Medication.notes` becomes empty.Medium — coordination with workflow-abc; pin a regression test that asserts notes is empty.workflow-abc PR accepting empty notes
**3e**`records`: drop top-level `user_id`, drop `records.<id>.url`, token-replace `file_name`. Needs the deferred nested-map projector OR a processor-side rewrite of the records bundle.Low — citation rendering verified to gracefully degrade.—
**3f**CI guards: regex test for `YYYY-MM-DD` patterns in any output JSON; fixture with deliberately PHI-shaped narrative content + assertion that none appears on disk.Low — pure test code.3a–3e
**3g****Architecture cleanup.** Move allowlist enforcement into the processor (processor reads `config.DEFAULT_FIELD_ALLOWLISTS` and emits exactly that). Delete `field_projector` from `loader.py`, delete `FIELD_ALLOWLISTS_ENV` + `_parse_allowlist_env`, delete the field-projector warning path. One config, one execution point.Low — pure refactor; covered by existing loader tests + new test asserting on-disk JSON matches the config literal exactly.3a–3e
**3h**Replace `FIELD_AUDIT.md` with a Safe Harbor compliance matrix. Add `infrastructure/data/SAFE_HARBOR.md` covering scope, "actual knowledge" attestation (§(b)(2)(ii)), residual risks, annual review process.Low — documentation only.3a–3g

Open decisions before implementing

1. Biomarker time encoding — confirmed: year + time_of_day. Time-of-day is preserved at source precision because it’s orthogonal to the date axis under §C and clinically essential for circadian markers. Same-year intra-year ordering is currently unsupported; revisit with days_since_prior only if a feature actually requires it.
2. Source-data check pending: the current biomarkers date output type is date (no time) — verify whether the parser/DB actually carries time-of-day before Phase 3c. If not, time_of_day will simply always be null until the parser is updated; the schema is forward-compatible.
3. medications.notes migration — sequence with workflow-abc: drop after workflow-abc accepts empty Medication.notes (Phase 3d gate), or drop now and let workflow-abc adapt? Recommendation: stage workflow-abc PR first to avoid a broken release window.
4. file_name token format — confirmed: record_<first-8-of-uuid>.pdf. Looks like a filename to citation rendering; deterministic from the UUID; non-identifying.
5. Age 90+ representation — confirmed: string "90+", integer otherwise; CHRs render the field as-is.
6. “Actual knowledge” attestation — Safe Harbor §(b)(2)(ii) requires the covered entity to attest no actual knowledge that the data could be re-identified. Who signs? When? Annually? This is a process question, not a code question.

Plan generated 2026-05-09 from infrastructure/data/config.py (DEFAULT_FIELD_ALLOWLISTS), infrastructure/data/processors/*.py, the n1_api_client.models.* attrs definitions, workflows/workflow-abc/.../utils/references.py, and workflows/visual-health-intelligence/scripts/common.py. Branch feat/data-field-allowlist-phase2 at HEAD 721aef3. Update this file alongside any change to those sources or the plan.