---
title: Phase 1 — Usage Counters + Upload Review
---

# Phase 1 — Usage Counters + Upload Review

## Problem → Solution

**Problem:** we don't know how many pages doctors upload per month, and today's upload flow auto-fires parsing — doctors have no visibility into what they just queued, no chance to cancel a wrong file before it parses, and no page-count data to price a subscription against.
**Solution:** count pages on every PDF upload; decouple parse from upload so the doctor sees a review screen listing each file + its page count and explicitly clicks **Extract** to start parsing. Cancel-before-extract is free. Records list also shows the per-record page count. Verification via 5-doctor hand-sample over 2 weeks.

## What users can do after this phase

- Upload records and see the page count per file on the upload confirmation.
- Review the full batch on a confirmation screen before any parsing starts.
- Cancel the entire batch (or individual records) while still in review — no parse runs.
- Explicitly click **Extract** to commit the batch to parsing.
- See a page count badge next to every row in the records list.

No quota numbers, no plan context, no billing — that's Phase 2 + 3. The UX is honest: "here's what you uploaded, parse it when you're ready."

## Goal

- `record_requests.page_count` populated on every upload.
- `/records/add` stops auto-queueing parse; parse queues only on an explicit extract call.
- `UploadModal` flows: upload → review → extract.
- Records list renders `page_count`.
- Orphaned SYNCED records (> 24h, never extracted) cleaned up by cron.
- Ops can verify per-doctor totals via read-replica SQL; 5-doctor hand-sample shows ±1 page per record.

## Repos touched

| Repo | Changes |
|---|---|
| **api-backend** | Page counter utility, `page_count` column + migration, drop auto-queue in `/records/add`, new `/records/batch-extract` endpoint, orphan cleanup cron |
| **react-frontend** | `UploadModal` refactor (upload → review → extract), page-count badge in records list, upload-result toast |
---

## Step 1 — PDF page counter

**Problem:** api-backend has no function that returns a page count for a PDF file.
**Solution:** thin wrapper around `pypdf.PdfReader.get_num_pages` that accepts a file-like object and raises `UnsupportedForCounting` on unreadable PDFs.

```python
# services/page_counter.py
from typing import BinaryIO
from pypdf import PdfReader
from pypdf.errors import PdfReadError, EmptyFileError

class UnsupportedForCounting(Exception):
    """Raised when a PDF cannot be counted (corrupt, empty, or otherwise unreadable)."""

def count_pdf_pages(file: BinaryIO) -> int:
    try:
        return PdfReader(file).get_num_pages()
    except (PdfReadError, EmptyFileError) as e:
        raise UnsupportedForCounting(f"unreadable PDF: {e}") from e
```

No fallback to `None`, no "best effort" count. Only pypdf's typed errors are caught; everything else propagates so it surfaces in SigNoz.

---

## Step 2 — `page_count` column + Alembic migration

**Problem:** `record_requests` has no column for per-upload page count.
**Solution:** nullable integer column, generated Alembic revision.

```bash
uv run alembic revision -m "add page_count to record_requests"
```

```python
def upgrade():
    op.add_column("record_requests", sa.Column("page_count", sa.Integer(), nullable=True))

def downgrade():
    op.drop_column("record_requests", "page_count")
```

Nullable so the migration runs instantly over existing rows. Pre-migration rows stay null. Never hand-write revision IDs (collision risk per api-backend CLAUDE.md).

---

## Step 3 — Count on upload, drop the auto-queue

**Problem:** the upload route counts nothing and auto-queues parse. We need the count written at insert time and the parse trigger moved downstream.
**Solution:** count pages after `convert_to_pdf` returns, persist `page_count` on the ORM insert, **remove the `queue_parse_job` call** from `/records/add` and `/records/add/images`. The record lands in `SYNCED` and waits for an explicit extract call.

```python
# routes/records.py  (inside /records/add and /records/add/images, after conversion)
try:
    page_count = count_pdf_pages(BytesIO(pdf_bytes))
except UnsupportedForCounting as e:
    raise HTTPException(status_code=415, detail=str(e))

return await service.update_user_records(
    file_object=BytesIO(pdf_bytes),
    content_type="application/pdf",
    page_count=page_count,
    ...
)
```

```python
# services/records_service.py
async def update_user_records(self, *, file_object, content_type, page_count, ...):
    request_orm = UserRecordRequestORM(
        id=record_id,
        user_id=user_id,
        file_name=file_name,
        file_hash=file_hash,
        page_count=page_count,   # new
        progress=0,
        type="RECORD",
        status="PENDING",        # transitions to SYNCED after GCS upload completes
        batch_id=batch_id,
        ...
    )
    db.add(request_orm); db.commit()
    # GCS upload unchanged. queue_parse_job(record.id) is REMOVED from this path.
```

Two `BytesIO(pdf_bytes)` wrappers are deliberate — `PdfReader` consumes the first stream to EOF, so the service needs a fresh cursor over the same underlying bytes.

Fail-secure on `UnsupportedForCounting`: 415 returned before the service is called — no row, no GCS blob, no downstream work.

---

## Step 4 — `POST /records/batch-extract` (no billing yet)

**Problem:** with auto-queue gone, something has to queue parse. That something is the explicit extract click.
**Solution:** new endpoint that accepts a list of record ids, validates ownership + SYNCED status, queues parse for each, returns the outcome. No billing call — Phase 3 inserts `/quota/charge` into this same code path.

```python
# routes/records.py
@router.post("/records/batch-extract")
@handle_exceptions(error_message="Failed to extract records")
async def batch_extract(
    body: BatchExtractRequest,
    caller: CallerContext = Depends(get_caller),
):
    records = await records_service.fetch_owned_records(caller.caller_id, body.record_ids)
    results = []
    for record in records:
        if record.status != "SYNCED":
            results.append({"record_id": record.id, "status": "not_extractable"})
            continue
        await queue_parse_job(record.id)
        results.append({"record_id": record.id, "status": "queued"})
    return {"results": results}
```

Per-record result array so the frontend can render mixed outcomes. Phase 3 widens the result shape with `event_id`, `overage_count`, and `quota_exceeded` branches — the endpoint signature stays the same.

---

## Step 5 — `UploadModal` refactor (upload → review → extract)

**Problem:** today's `UploadModal` auto-fires `processBatch` in a `useEffect` the moment uploads complete. The doctor never sees page counts and has no confirmation step. With auto-queue gone server-side, the modal would silently leave records in SYNCED with no UX.
**Solution:** two-phase modal — **Upload & Review** → **Extract**.

**Flow:**

1. **File selection.** User drops / picks files. `checkPdfEncryption` (existing in `@/lib/pdf-utils`) runs client-side; encrypted files prompt for password.
2. **Upload.** Each file POST'd to `/records/add`. Response carries `id`, `page_count`, `status: "SYNCED"`. Modal accumulates a per-file list.
3. **Review.** Once the batch finishes uploading, the modal flips to a review card. **Auto-fire of `processBatch` is removed.**

```
┌─────────────────────────────────────────────┐
│  Ready to extract — 3 documents, 47 pages   │
│                                              │
│    lab-report.pdf              12 pages      │
│    imaging-notes.pdf           28 pages      │
│    referral-letter.pdf          7 pages      │
│                                              │
│  [Cancel all]           [Extract 47 pages]  │
└─────────────────────────────────────────────┘
```

4. **Extract.** User clicks "Extract N pages." Frontend fires one `POST /records/batch-extract` with all record ids. Per-record results render in the modal; modal closes on all-queued.
5. **Cancel all.** Frontend fires `DELETE /records/{id}` for every SYNCED row in the batch.

No quota line, no plan context — Phase 3 adds a quota summary line above the Extract button ("412 of 2,000 left · after extract: 365"). Phase 1 ships the structural flow; Phase 3 layers numbers on top.

---

## Step 6 — Page count badge in the records list

**Problem:** the records list already renders upload date, filename, status — but nothing about size. Doctors can't tell a 2-page lab report from a 200-page imaging dump at a glance.
**Solution:** include `page_count` on each record in the list response and render it in `RecordRow`.

Backend: `page_count` added to the existing records list serializer. No new endpoint.
Frontend: `RecordRow` renders "12 pages" next to the filename (or nothing for pre-migration rows with `page_count = null`).

Upload-result toast also shows "Uploaded — 12 pages" for immediate feedback.

---

## Step 7 — Orphan cleanup cron

**Problem:** decoupling parse from upload means records can land in SYNCED and never get extracted (user closes the tab mid-review, network drops, etc.). Without cleanup these accumulate indefinitely.
**Solution:** hourly cron deletes records in SYNCED > 24h old with no downstream parse job queued.

```sql
DELETE FROM record_requests
WHERE status = 'SYNCED'
  AND created_at < now() - INTERVAL '24 hours'
  AND id NOT IN (SELECT record_id FROM parse_jobs);
```

GCS lifecycle policy already handles the orphaned blobs — no separate blob cleanup code. Log one line per deletion; no audit table (these are abandoned uploads, not user data).

Phase 3 keeps this cron unchanged — it only touches SYNCED records (never extracted, no `quota_events` row), so there's no interaction with the charge or reprocess path.

---

## Step 8 — Verification SQL (ops-only, no endpoint)

**Problem:** the exit criterion requires a 5-doctor hand-sample proving counts are within ±1 page per record over 2 weeks.
**Solution:** documented SQL runnable against the staging / prod read replica. No endpoint, no auth.

```sql
-- pages committed, per doctor, UTC month
SELECT COALESCE(SUM(page_count), 0) AS pages_used
FROM record_requests
WHERE user_id = :user_id
  AND created_at >= :start AND created_at < :end;

-- reports, per doctor, same window
SELECT COUNT(*) AS reports_used
FROM report_gen_requests
WHERE user_id = :user_id
  AND created_at >= :start AND created_at < :end;
```

**Count at start, not at completion.** Every `record_requests` row carries its `page_count` from the upload instant; every `report_gen_requests` row exists from the Generate click. Parse failures or deletions later don't rewrite the numbers — Phase 3 recovers failed parses via free reprocessing (Step 9), which replays the same row rather than writing a new one. Table name is `report_gen_requests` (ORM `UserReportGenRequestORM`). Month bounds are UTC `[start, end)` half-open.

**Hand-sample protocol:**
1. Ops picks 5 active staging doctors with upload activity.
2. Runs the query for the current month.
3. Opens the doctor's records list, hand-counts pages per record (or totals from the PDFs).
4. Expects system total within ±1 page per record.
5. Any mismatches logged with the record id + file for triage.

**After Phase 2.** Billing-service's Contract B doesn't call this SQL — it reads its own `quota_events` ledger (master spec's "no Contract A" rule). This Step 8 SQL stays as an ops-side cross-check: `SUM(record_requests.page_count)` is an upper bound on committed pages per doctor, and divergence against `SUM(quota_events.count)` is expected (uploaded-but-not-extracted records never reach `/quota/charge`) — documented in Phase 2's Accuracy invariant.

---

## Testing

- Unit: `count_pdf_pages` happy path (3-page PDF → 3), `UnsupportedForCounting` on corrupt / empty PDFs.
- Integration: upload 3-page PDF → `page_count = 3`, record lands in SYNCED, no parse queued; upload a corrupt PDF → 415, no row, no blob. `batch-extract` with valid SYNCED ids → all queued. `batch-extract` with a non-SYNCED id → `not_extractable` result, no queue.
- Frontend: upload modal opens → upload completes → review screen shows file list with page counts → Extract click fires `batch-extract` → modal closes. Cancel all → `DELETE` for each → modal closes, records gone.
- Manual staging: 5 doctors uploading a known mix of formats; Step 8 SQL matches hand counts.
- Orphan cleanup: record in SYNCED > 24h with no parse job → deleted by next cron run.

## Done when

- 5-doctor hand-sample matches system counts ±1 page per record (Step 8 SQL).
- Upload → review → extract flow works end-to-end in staging.
- Corrupt PDFs rejected with 415 before a row is written.
- Orphan cleanup cron deletes abandoned SYNCED records.

## Depends on

- Nothing upstream.

## Feeds

- Phase 2 (plan display needs `page_count` and the `batch-extract` endpoint).
- Phase 3 (charge call inserts into the same `batch-extract` path).