Phase 1 — Usage Counters + Upload Review

Problem → Solution

Problem: we don’t know how many pages doctors upload per month, and today’s upload flow auto-fires parsing — doctors have no visibility into what they just queued, no chance to cancel a wrong file before it parses, and no page-count data to price a subscription against. Solution: count pages on every PDF upload; decouple parse from upload so the doctor sees a review screen listing each file + its page count and explicitly clicks Extract to start parsing. Cancel-before-extract is free. Records list also shows the per-record page count. Verification via 5-doctor hand-sample over 2 weeks.

What users can do after this phase

Upload records and see the page count per file on the upload confirmation.
Review the full batch on a confirmation screen before any parsing starts.
Cancel the entire batch (or individual records) while still in review — no parse runs.
Explicitly click Extract to commit the batch to parsing.
See a page count badge next to every row in the records list.

No quota numbers, no plan context, no billing — that’s Phase 2 + 3. The UX is honest: “here’s what you uploaded, parse it when you’re ready.”

Goal

record_requests.page_count populated on every upload.
/records/add stops auto-queueing parse; parse queues only on an explicit extract call.
UploadModal flows: upload → review → extract.
Records list renders page_count.
Orphaned SYNCED records (> 24h, never extracted) cleaned up by cron.
Ops can verify per-doctor totals via read-replica SQL; 5-doctor hand-sample shows ±1 page per record.

Repos touched

Repo	Changes
api-backend	Page counter utility, `page_count` column + migration, drop auto-queue in `/records/add`, new `/records/batch-extract` endpoint, orphan cleanup cron
react-frontend	`UploadModal` refactor (upload → review → extract), page-count badge in records list, upload-result toast

Step 1 — PDF page counter

Problem: api-backend has no function that returns a page count for a PDF file. Solution: thin wrapper around pypdf.PdfReader.get_num_pages that accepts a file-like object and raises UnsupportedForCounting on unreadable PDFs.

from typing import BinaryIO
from pypdf import PdfReader
from pypdf.errors import PdfReadError, EmptyFileError

class UnsupportedForCounting(Exception):
    """Raised when a PDF cannot be counted (corrupt, empty, or otherwise unreadable)."""

def count_pdf_pages(file: BinaryIO) -> int:
    try:
        return PdfReader(file).get_num_pages()
    except (PdfReadError, EmptyFileError) as e:
        raise UnsupportedForCounting(f"unreadable PDF: {e}") from e

No fallback to None, no “best effort” count. Only pypdf’s typed errors are caught; everything else propagates so it surfaces in SigNoz.

Step 2 — `page_count` column + Alembic migration

Problem: record_requests has no column for per-upload page count. Solution: nullable integer column, generated Alembic revision.

uv run alembic revision -m "add page_count to record_requests"

def upgrade():
    op.add_column("record_requests", sa.Column("page_count", sa.Integer(), nullable=True))

def downgrade():
    op.drop_column("record_requests", "page_count")

Nullable so the migration runs instantly over existing rows. Pre-migration rows stay null. Never hand-write revision IDs (collision risk per api-backend CLAUDE.md).

Step 3 — Count on upload, drop the auto-queue

Problem: the upload route counts nothing and auto-queues parse. We need the count written at insert time and the parse trigger moved downstream. Solution: count pages after convert_to_pdf returns, persist page_count on the ORM insert, remove the queue_parse_job call from /records/add and /records/add/images. The record lands in SYNCED and waits for an explicit extract call.

# routes/records.py  (inside /records/add and /records/add/images, after conversion)
try:
    page_count = count_pdf_pages(BytesIO(pdf_bytes))
except UnsupportedForCounting as e:
    raise HTTPException(status_code=415, detail=str(e))

return await service.update_user_records(
    file_object=BytesIO(pdf_bytes),
    content_type="application/pdf",
    page_count=page_count,
    ...
)

async def update_user_records(self, *, file_object, content_type, page_count, ...):
    request_orm = UserRecordRequestORM(
        id=record_id,
        user_id=user_id,
        file_name=file_name,
        file_hash=file_hash,
        page_count=page_count,   # new
        progress=0,
        type="RECORD",
        status="PENDING",        # transitions to SYNCED after GCS upload completes
        batch_id=batch_id,
        ...
    )
    db.add(request_orm); db.commit()
    # GCS upload unchanged. queue_parse_job(record.id) is REMOVED from this path.

Two BytesIO(pdf_bytes) wrappers are deliberate — PdfReader consumes the first stream to EOF, so the service needs a fresh cursor over the same underlying bytes.

Fail-secure on UnsupportedForCounting: 415 returned before the service is called — no row, no GCS blob, no downstream work.

Step 4 — `POST /records/batch-extract` (no billing yet)

Problem: with auto-queue gone, something has to queue parse. That something is the explicit extract click. Solution: new endpoint that accepts a list of record ids, validates ownership + SYNCED status, queues parse for each, returns the outcome. No billing call — Phase 3 inserts /quota/charge into this same code path.

@router.post("/records/batch-extract")
@handle_exceptions(error_message="Failed to extract records")
async def batch_extract(
    body: BatchExtractRequest,
    caller: CallerContext = Depends(get_caller),
):
    records = await records_service.fetch_owned_records(caller.caller_id, body.record_ids)
    results = []
    for record in records:
        if record.status != "SYNCED":
            results.append({"record_id": record.id, "status": "not_extractable"})
            continue
        await queue_parse_job(record.id)
        results.append({"record_id": record.id, "status": "queued"})
    return {"results": results}

Per-record result array so the frontend can render mixed outcomes. Phase 3 widens the result shape with event_id, overage_count, and quota_exceeded branches — the endpoint signature stays the same.

Step 5 — `UploadModal` refactor (upload → review → extract)

Problem: today’s UploadModal auto-fires processBatch in a useEffect the moment uploads complete. The doctor never sees page counts and has no confirmation step. With auto-queue gone server-side, the modal would silently leave records in SYNCED with no UX. Solution: two-phase modal — Upload & Review → Extract.

Flow:

File selection. User drops / picks files. checkPdfEncryption (existing in @/lib/pdf-utils) runs client-side; encrypted files prompt for password.
Upload. Each file POST’d to /records/add. Response carries id, page_count, status: "SYNCED". Modal accumulates a per-file list.
Review. Once the batch finishes uploading, the modal flips to a review card. Auto-fire of processBatch is removed.

┌─────────────────────────────────────────────┐
│  Ready to extract — 3 documents, 47 pages   │
│                                              │
│    lab-report.pdf              12 pages      │
│    imaging-notes.pdf           28 pages      │
│    referral-letter.pdf          7 pages      │
│                                              │
│  [Cancel all]           [Extract 47 pages]  │
└─────────────────────────────────────────────┘

Extract. User clicks “Extract N pages.” Frontend fires one POST /records/batch-extract with all record ids. Per-record results render in the modal; modal closes on all-queued.
Cancel all. Frontend fires DELETE /records/{id} for every SYNCED row in the batch.

No quota line, no plan context — Phase 3 adds a quota summary line above the Extract button (“412 of 2,000 left · after extract: 365”). Phase 1 ships the structural flow; Phase 3 layers numbers on top.

Step 6 — Page count badge in the records list

Problem: the records list already renders upload date, filename, status — but nothing about size. Doctors can’t tell a 2-page lab report from a 200-page imaging dump at a glance. Solution: include page_count on each record in the list response and render it in RecordRow.

Backend: page_count added to the existing records list serializer. No new endpoint. Frontend: RecordRow renders “12 pages” next to the filename (or nothing for pre-migration rows with page_count = null).

Upload-result toast also shows “Uploaded — 12 pages” for immediate feedback.

Step 7 — Orphan cleanup cron

Problem: decoupling parse from upload means records can land in SYNCED and never get extracted (user closes the tab mid-review, network drops, etc.). Without cleanup these accumulate indefinitely. Solution: hourly cron deletes records in SYNCED > 24h old with no downstream parse job queued.

DELETE FROM record_requests
WHERE status = 'SYNCED'
  AND created_at < now() - INTERVAL '24 hours'
  AND id NOT IN (SELECT record_id FROM parse_jobs);

GCS lifecycle policy already handles the orphaned blobs — no separate blob cleanup code. Log one line per deletion; no audit table (these are abandoned uploads, not user data).

Phase 3 keeps this cron unchanged — it only touches SYNCED records (never extracted, no quota_events row), so there’s no interaction with the charge or reprocess path.

Step 8 — Verification SQL (ops-only, no endpoint)

Problem: the exit criterion requires a 5-doctor hand-sample proving counts are within ±1 page per record over 2 weeks. Solution: documented SQL runnable against the staging / prod read replica. No endpoint, no auth.

-- pages committed, per doctor, UTC month
SELECT COALESCE(SUM(page_count), 0) AS pages_used
FROM record_requests
WHERE user_id = :user_id
  AND created_at >= :start AND created_at < :end;

-- reports, per doctor, same window
SELECT COUNT(*) AS reports_used
FROM report_gen_requests
WHERE user_id = :user_id
  AND created_at >= :start AND created_at < :end;

Count at start, not at completion. Every record_requests row carries its page_count from the upload instant; every report_gen_requests row exists from the Generate click. Parse failures or deletions later don’t rewrite the numbers — Phase 3 recovers failed parses via free reprocessing (Step 9), which replays the same row rather than writing a new one. Table name is report_gen_requests (ORM UserReportGenRequestORM). Month bounds are UTC [start, end) half-open.

Hand-sample protocol:

Ops picks 5 active staging doctors with upload activity.
Runs the query for the current month.
Opens the doctor’s records list, hand-counts pages per record (or totals from the PDFs).
Expects system total within ±1 page per record.
Any mismatches logged with the record id + file for triage.

After Phase 2. Billing-service’s Contract B doesn’t call this SQL — it reads its own quota_events ledger (master spec’s “no Contract A” rule). This Step 8 SQL stays as an ops-side cross-check: SUM(record_requests.page_count) is an upper bound on committed pages per doctor, and divergence against SUM(quota_events.count) is expected (uploaded-but-not-extracted records never reach /quota/charge) — documented in Phase 2’s Accuracy invariant.

Testing

Unit: count_pdf_pages happy path (3-page PDF → 3), UnsupportedForCounting on corrupt / empty PDFs.
Integration: upload 3-page PDF → page_count = 3, record lands in SYNCED, no parse queued; upload a corrupt PDF → 415, no row, no blob. batch-extract with valid SYNCED ids → all queued. batch-extract with a non-SYNCED id → not_extractable result, no queue.
Frontend: upload modal opens → upload completes → review screen shows file list with page counts → Extract click fires batch-extract → modal closes. Cancel all → DELETE for each → modal closes, records gone.
Manual staging: 5 doctors uploading a known mix of formats; Step 8 SQL matches hand counts.
Orphan cleanup: record in SYNCED > 24h with no parse job → deleted by next cron run.

Done when

5-doctor hand-sample matches system counts ±1 page per record (Step 8 SQL).
Upload → review → extract flow works end-to-end in staging.
Corrupt PDFs rejected with 415 before a row is written.
Orphan cleanup cron deletes abandoned SYNCED records.

Depends on

Nothing upstream.

Feeds

Phase 2 (plan display needs page_count and the batch-extract endpoint).
Phase 3 (charge call inserts into the same batch-extract path).

Phase 1 — Usage Counters + Upload Review

Phase 1 — Usage Counters + Upload Review

Problem → Solution

What users can do after this phase

Goal

Repos touched

Step 1 — PDF page counter

Step 2 — page_count column + Alembic migration

Step 3 — Count on upload, drop the auto-queue

Step 4 — POST /records/batch-extract (no billing yet)

Step 5 — UploadModal refactor (upload → review → extract)

Step 6 — Page count badge in the records list

Step 7 — Orphan cleanup cron

Step 8 — Verification SQL (ops-only, no endpoint)

Testing

Done when

Depends on

Feeds

Step 2 — `page_count` column + Alembic migration

Step 4 — `POST /records/batch-extract` (no billing yet)

Step 5 — `UploadModal` refactor (upload → review → extract)