✓
GO FOR DEPLOY
All systems nominal. 0/0 services healthy. 2 checks passed with 0 warnings across billing-only variant. No blockers identified.
Human approval required before deploy
2026-06-10
05:31 UTC
billing-only
Service Health
—
N/A
Not tested
Code Quality
—
N/A
Not tested
Data Quality
—
N/A
Not tested
UX Stability
—
N/A
Not tested
Observability
—
N/A
Not tested
Release
—
N/A
Not tested
2
Passed
0
Warnings
0
Failed
11
Skipped
13
Total
Pass 2 Warn 0 Fail 0 Skip 11

Layer Results

PASS
Content Evals
Biomarker F1 94.7%
P 92% / R 98% · 15 docs
View details →
PASS
Billing Tests
7/7 scenarios passed
View details →
PASS
Release Review
Skipped
View details →
Run: 2026-06-10 05:31:41 UTC Variant: billing-only Duration: 322s Decision: GO

Go / No-Go

Deployment decision based on aggregated layer results. This page is the single decision document for the on-call engineer.

GO
2026-06-10 · 05:31 UTC
billing-only
0
Must Fix
0
Monitor
2
Passed
11
Skipped
13
Total
✓ Safe to deploy
Health — Pytest — Data — AI QA — SigNoz — Release ✓
Layer-by-Layer Verdict
LayerStatusKey MetricReason
Service Health Skip — Not included in this variant.
Pytest (CI) Skip — Not included in this variant.
Data Quality Skip — Not included in this variant.
AI QA Skip — Not included in this variant.
SigNoz Skip — Not included in this variant.
Billing Tests Pass 7/7 scenarios All Stripe test-clock journeys green (282.9s).
Release Review Pass — No failing CI, no open PRs. 0 repos with undeployed changes.
Ready to Deploy
1 layer clear
✓
Release Review — 0 repos ready
No failing CI, no open PRs.
Release Review
Run: 2026-06-10 05:31:41 UTC Duration: 321.6s Decision: GO Must Fix: 0 Monitor: 0 Checks: 2 pass / 0 fail / 11 skip

Service Health

Endpoint health checks across staging and production environments.

Health checks were not included in this run variant.

Pytest (CI Tests)

GitHub Actions test workflows triggered across 0 repositories. Each repo's test.yml or ci.yml was dispatched and polled for completion.

Pytest CI triggers were not included in this run variant.

Forge

Forge-sentinel test suite and nightly runner workflows.

Forge-sentinel workflows were not included in this run variant.

Evals

Legacy pipeline evaluation suite.

Pipeline evals were not included in this run variant.

Forge Runner

MCP infrastructure and agent pipeline tests.

Not in variant

Parser V2

Document parsing smoke tests and E2E pipeline verification.

Not in variant

Content Evals

Parser output checked against ground truth on 15 documents (comparison-only).

What's happening
Precision dragged down by false positives, mainly one document
Overall F1 is 94.7% (precision 91.7%, recall 98.0%) across 395 ground-truth biomarkers, but 35 false positives — 31 of them from a single colonoscopy report — are the dominant error mode.
395
Ground truth
387
Caught correctly
35
Extra (shouldn't be there)
8
Missed
94.7%
F1
What's going wrong
Colonoscopy report drives most false positives
Affects 31 of 35 false positives across 1 of 15 documents
The '20111012 - hem - medical report - colonoscopy' document alone produced 31 extra extractions (plus 5 missed), yielding F1=0.79 — far below every other document. This single report accounts for ~89% of all false positives in the eval, suggesting the parser is over-extracting non-biomarker content (likely procedure findings, vitals, or symptoms) from narrative-style clinical reports rather than structured lab panels.
Parser team
Manually review the colonoscopy report's 31 extras to identify the over-extraction pattern (narrative vs. tabular parsing) and add a document-type or section-level filter.
Non-biomarker concepts extracted as biomarkers
Affects 16 of 34 sampled false positives (symptom=7, vital=6, imaging=3)
Of 34 sampled false positives, 16 are clearly non-laboratory concepts: symptoms (e.g. Jaundice, Muscle Power, BMI), vitals (e.g. Height, Pallor, Visual Acuity), and imaging (e.g. Chest X-Ray, Left Kidney Bipolar Length). The parser lacks a strong type gate distinguishing lab analytes from physical exam findings, anthropometrics, and imaging descriptors.
Parser team
Add an explicit deny-list / type classifier to reject symptom, vital-sign, and imaging entities before they enter the biomarker output.
Ambiguous true-lab terms flagged as FPs needs clinical adjudication
Affects 18 of 34 sampled false positives
The largest sampled FP category (18/34) is 'ambiguous' and includes legitimate-sounding lab analytes such as ALT (SGPT), RBC, GGT, eGFR, Lymphocytes#, and Neutrophils#. These may be duplicates, alternate-name mismatches against ground truth, or genuinely valid extractions miscounted as FPs — distinct from clear non-biomarker errors.
Clinical QA
Adjudicate the 18 ambiguous FPs to determine whether they are name-normalization/duplication failures vs. true over-extraction, and update ground truth or synonym maps accordingly.
Field-value accuracy below extraction accuracy
Affects 387 matched biomarkers (unit 82%, reference_range 84%, sample_source 84%, result
Even on correctly matched biomarkers, unit (82%), reference_range (84%), and sample_source (84%) lag well behind test_date (98%) and overall recall. This indicates the parser identifies the right analyte but struggles to associate adjacent structured fields, likely due to layout/column-binding errors in tabular lab reports.
Parser team
Run a targeted error analysis on unit and reference_range mismatches across matched biomarkers to identify whether the issue is OCR layout, column alignment, or post-processing normalization.
What happened in each document (15)
Click a document to see exactly which biomarkers were extracted that shouldn't have been, plus field mismatches on the ones that were correct.
20111012 - hem - medical report - colonoscopy
GT: 72 Predicted: 98 31 extra 5 missed
F1 78.8% ▾
Biomarker F1
79%
result
84%
reference range
82%
unit
81%
test date
100%
sample source
88%
imaging3 false positives
Chest X-Raypage 0
Result: Normal
Sample Source: Patient
PRpage 2
Result: 68.0 · Unit: beats/min
Sample Source: Patient
Left Kidney Bipolar Lengthpage 10
Result: 9.5 · Unit: cm
Sample Source: Kidney
symptom7 false positives
Muscle Powerpage 3
Result: 5/5
Sample Source: Patient
BMI Body Mass Indexpage 2
Result: 20
Sample Source: Patient
BPpage 2
Result: 125/88 · Unit: mmHg
Sample Source: Patient
Colour Visionpage 2
Result: Normal
Sample Source: Patient
Jaundicepage 2
Result: Nil
Sample Source: Patient
Weightpage 2
Result: 62.0 · Unit: kg
Sample Source: Patient
Temp.page 2
Result: 37.0 · Unit: °C
Sample Source: Patient
vital6 false positives
Ultrasound Abdomen and Pelvispage 0
Result: Normal
Sample Source: Patient
Oedemapage 2
Result: Nil
Sample Source: Patient
Heightpage 2
Result: 174.0 · Unit: cm
Sample Source: Patient
Visual Acuitypage 2
Result: 6/6
Sample Source: Patient
Lymphadenopathypage 2
Result: Nil
Sample Source: Patient
Pallorpage 2
Result: Nil
Sample Source: Patient
ambiguous15 false positives
Alanine Aminotransferase-ALT (SGPT)page 13
Result: 12.0 · Unit: u/l
Sample Source: Blood
RBCpage 13
Result: 4.98 · Unit: 10*12/l
Sample Source: Blood
Gamma-Glutamyl Transferase (GGT)page 13
Result: 17.0 · Unit: u/l
Sample Source: Blood
eGFRpage 13
Result: >60 · Unit: mL/min/1.73m^2
Sample Source: Blood
Lymphocytes#page 13
Result: 1.89 · Unit: 10*9/l
Sample Source: Blood
Neutrophils#page 13
Result: 2.06 · Unit: 10*9/l
Sample Source: Blood
Aspartate Transaminase-AST (SGOT)page 13
Result: 19.0 · Unit: u/l
Sample Source: Blood
WBCpage 13
Result: 4.3 · Unit: 10*9/l
Sample Source: Blood
Plateletspage 13
Result: 148.0 · Unit: 10*9/l
Sample Source: Blood
Lymphocytosispage 0
Result: Present
Sample Source: Blood
Thrombocytopeniapage 0
Result: Present
Sample Source: Blood
Cholesterolpage 0
Result: Elevated
Sample Source: Blood
Right Kidney Bipolar Lengthpage 10
Result: 10.8 · Unit: cm
Sample Source: Kidney
Stool Occult Bloodpage 14
Result: Negative
Sample Source: Stool
Basophils#page 14
Result: 0.00
Sample Source: Blood
Field mismatches on the 67 correctly-matched biomarkers
result (83.6%): 11 mismatches — e.g. 30.7 vs 0.0, 13.5 vs 0.7893.
reference range (82.1%): 12 mismatches — e.g. <4.3 vs <3.4, 27.0-32.0 vs 0.0 - 0.0.
unit (80.6%): 13 mismatches — e.g. pg vs g/dL, ∅ vs ratio.
sample source (88.1%): 8 mismatches — e.g. Serum vs Blood, Serum vs Blood.
20240530-jean-yves-sireau-diet
GT: 17 Predicted: 20 3 extra
F1 91.9% ▾
Biomarker F1
92%
result
100%
reference range
0%
unit
100%
test date
100%
sample source
100%
ambiguous3 false positives
IgA/IgG Class IIpage 4
Result: Eel, Red Snapper, Trout, Tuna, Whey
Sample Source: Serum
IgA Class Ipage 4

Sample Source: Serum
IgA/IgG Class III and IVpage 4

Sample Source: Serum
Field mismatches on the 17 correctly-matched biomarkers
reference range (0.0%): 17 mismatches — e.g. ∅ vs Class 0-I, ∅ vs Class 0-I.
+ 13 more documents with F1 ≥ 95% (clean lab panels)
How we measured this
Precision
0.917
Of the biomarkers the parser said it found, what fraction were real?
422 predicted, 387 real → 91.7%. Lower precision = over-extraction.
Recall
0.980
Of the biomarkers actually present, what fraction did the parser find?
395 real, 387 caught, 8 missed → 98.0%.
F1-Score
0.947
Harmonic mean of precision & recall — drops when either is weak.
Lands at 94.7%.
Accuracy
0.900
Fraction of all decisions (extract/skip) that were correct.
Every false positive counts against it → 90.0%.
Confusion Matrix
Predicted
NoYes
ActualNo0true neg35false pos
Yes8false neg387true pos
What each cell means
387 True positives — correctly extracted a real biomarker.
35 False positives — extracted something that shouldn't be there. The over-extraction problem.
8 False negatives — real biomarkers the parser missed.
0 True negatives — no concept of "correctly not extracting" here, so always zero.
Field-level accuracy across 387 correctly-matched biomarkers
FieldAccuracyMatchesMismatches
result 89.4% 34641
reference range 83.7% 32463
unit 82.4% 31968
test date 97.7% 3789
sample source 84.0% 32562

Report E2E

Layer 3.5 -- Report generation, validation, hallucination detection.

Report E2E was not included in this run variant.

Playwright

End-to-end UX tests against the React frontend.

Playwright UX tests were not included in this run variant.

Agent Exploration

Automated route exploration to detect console errors, network failures, and performance issues.

Agent exploration was not included in this run variant.

Data Quality

Parser output validation — 17 checks across biomarkers, diagnoses, procedures, genetics. Results from staging Redis (last 48h). 0 records across 0 users.

Data validation was not included in this run variant.

SigNoz

Error and fatal log counts from SigNoz over the past 24 hours across all monitored services.

SigNoz observability checks were not included in this run variant.

Performance Baselines

Pre-deploy staging validation — checks whether anything has slowed down or started erroring more in staging before promoting develop to main. Production performance is monitored separately.

Performance baselines were not included in this run variant.

Stripe Test-Clock Billing Tests

variant: billing-only

Six time-shifted billing journeys driven by Stripe test clocks against staging: trial → first paid period, successful renewal, failed-renewal dunning, recovery from past_due, cancel-at-period-end, and plan-switch-at-next-cycle.

282.9s wall, ~7 mo simulated
ran 2026-06-10 05:31 UTC
WHY
Regression coverage for billing-service#173. We had no automated way to test long-running billing flows before this. Real customer bugs only surfaced in production.

Assertion contract

billing-tests/CLAUDE.md
Asserted in v1
Stripe SDK state
Subscription status, invoice transitions, payment_intent outcomes, dunning attempt counters, scheduled cancellations and plan switches.
Asserted in v1
billing-service /quota/status
Subscription tier, charge_recorded flags, past_due transitions, seat-quota deltas, webhook idempotency, recorded plan changes.
Deferred to v2
UI banners and gates
Trial-ending banner, payment-failed banner, past_due lockout, plan switch confirmation modal.
Frontend gates /billing on email_verified being true and no programmatic verify-skip exists. A CF-admin helper in authentication-service unblocks this.
Scenarios
7 / 7
all passing
Simulated time
~7 mo
in 283s wall
Surfaces
2 / 3
UI deferred
Failed
0
none today

P0 scenarios 4 of 7, revenue-critical

4/4 pass · 181.7s total
P0-1
Free-trial to first paid period
scenarios/trial-to-paid.spec.ts
Key momentConversion at signup
17.8sPASS
T+0
Signupfree_trial attached
T+0
CheckoutStripe sub created
T+0
First chargestarter, payment succeeded
Key moment
Stripe SDK
billing-service /quota/status
App UI
Conversion at signup
invoice.payment_succeeded
subscription.status = active, starter price
plan.code = starter
free_trial → starter via customer.subscription.created webhook
deferred to v2
would assert: payment receipt visible
P0-2
Successful renewal after first paid period
scenarios/successful-renewal.spec.ts
Key momentT+44d renewal
32.9sPASS
T+0
Signup
T+14d
First charge
T+44d
Renewalmonthly cycle
Key moment
Stripe SDK
billing-service /quota/status
App UI
T+44d renewal
invoice.payment_succeeded (2nd)
subscription cycled, period advanced
charge_recorded = true
webhook fired exactly once (idempotency)
deferred to v2
would assert: monthly receipt email
P0-3
Failed renewal triggers dunning
scenarios/failed-renewal-dunning.spec.ts
Key momentT+51d past_due
64.9sPASS
T+0
Signupcard-fails-on-renewal
T+14d
First charge
T+44d
Renewal failscard declined
T+47d
Retry 1dunning email
T+51d
Retry 2past_due
Key moment
Stripe SDK
billing-service /quota/status
App UI
T+51d past_due
subscription.status = past_due
2 retry attempts logged, dunning emails sent
past_due = true
tier preserved, write-access lockout flag set
deferred to v2
would assert: payment-failed banner + lockout
P0-4
Recovery from past_due after card swap
scenarios/recovery-from-past-due.spec.ts
Key momentT+50d recovery
66.0sPASS
T+0
Signup
T+44d
Renewal fails
T+47d
past_due
T+50d
Card swapnew card added
T+50d
Auto-retrycharge succeeds
Key moment
Stripe SDK
billing-service /quota/status
App UI
T+50d recovery
subscription.status = active
past_due cleared, payment_method swapped
past_due = false
write-access restored, lockout flag cleared
deferred to v2
would assert: lockout banner clears

P1 scenarios 3 of 7, operational

3/3 pass · 101.3s total
P1-5
Cancel at period end
scenarios/cancel-at-period-end.spec.ts
Key momentT+44d effective
25.1sPASS
T+0
Signup
T+14d
First charge
T+30d
Cancel requestedscheduled
T+44d
Cancellation effective
Key moment
Stripe SDK
billing-service /quota/status
App UI
T+44d effective
subscription.status = canceled
no future invoices, period_end matches
tier downgraded to free
access window respected through T+44d
deferred to v2
would assert: downgrade confirmation modal
P1-6
Plan switch starter → professional at next cycle
scenarios/plan-switch-at-next-cycle.spec.ts
Key momentT+44d new cycle
33.7sPASS
T+0
Signupstarter plan
T+14d
First chargestarter price
T+20d
Switch requestedto professional
T+44d
Renewal at new price
Key moment
Stripe SDK
billing-service /quota/status
App UI
T+44d new cycle
subscription.items[0].price = professional
no proration, switch at cycle boundary
tier = professional
seat-quota updated, charge_recorded = true
deferred to v2
would assert: plan-active banner reflects new tier
P1-7
Plan switch starter → practice at next cycle
scenarios/plan-switch-starter-to-practice.spec.ts
Key momentT+44d new cycle
42.5sPASS
T+0
Signupstarter plan
T+14d
First chargestarter price
T+20d
Switch requestedto practice (top tier)
T+44d
Renewal at new pricepractice tier
Key moment
Stripe SDK
billing-service /quota/status
App UI
T+44d new cycle
subscription.items[0].price = practice
no proration, switch at cycle boundary
tier = practice
practice quotas applied (5000 pages, 50 reports)
deferred to v2
would assert: plan-active banner reflects practice tier

Run locally

$ cd nightly && uv run python -m nightly --variant billing-only --no-agent
Or run the Playwright suite directly: cd billing-tests && npm test. Requires billing-tests/.env.staging populated with BILLING_TESTS_STRIPE_TEST_KEY, BILLING_API_KEY, and the two plan price IDs. See billing-tests/CLAUDE.md.
Auto-generated by the N1 nightly production test system. Variant: billing-only · 05:31 UTC on 2026-06-10 · 282.9s · layers/billing_tests/report.html

AI QA Test Cases

Agent-driven test case execution via Claude Agent SDK + Playwright MCP.

Manual test cases were not included in this run variant.

CHR Reconciliation

Layer 5.9 -- Per-CHR lifecycle join across api-backend trigger logs, forge-sentinel terminal events, and the api-backend sweeper. Surfaces silent stalls, stuck reports, and clustered failures.

Not included in this variant.

Release Review

Comparison of main vs develop across all tracked repositories.

Release review was not included in this run variant.

Auto-generated by the N1 nightly production test system. Variant: billing-only · 05:31 UTC on 2026-06-10 · 322s · nightly-data-2026-06-10_05-31-41.json