--- title: Service Authentication — Zero-Trust Architecture --- # Service Authentication — Zero-Trust Architecture
A calling service obtains a short-lived, scoped OAuth token from Hydra and uses it for a receiver that validates audience and scope.
Service-to-service OAuth2 client-credentials flow. Download SVG source
> **Current state as of 2026-06-18.** Deployed on `feat/auth-zero-trust`. Covers service-to-service > (s2s) auth only. User-facing auth (browser login, JWT bearer, Oathkeeper) is described in > [N1 landscape](/landscape/). --- ## What changed and why Previously every backend service that called another N1 service passed a static pre-shared secret in a custom `N1-Api-Key` HTTP header. The receiving service compared the incoming value against a secret loaded from the pod environment at startup. That model has three hard problems: 1. **No per-service identity.** Every service shared the same secret; a leak anywhere compromised everything. There was no way to know which service made a given call. 2. **No scope enforcement.** Any holder of the static key could call any endpoint — a parser had the same access as the forge orchestrator. 3. **No short-lived credentials.** Rotating the key meant coordinating a simultaneous restart of every service that held it. The new model gives every backend service its own Hydra OAuth2 client identity, issues short-lived Bearer tokens via the `client_credentials` grant, and enforces named scopes at the receiving endpoint. The static `N1-Api-Key` header and `N1_API_KEY` env var no longer exist on the service-to-service path. --- ## How it works ### Token acquisition A service that needs to call another N1 service: 1. Reads `HYDRA_URL`, `HYDRA_CLIENT_ID`, `HYDRA_CLIENT_SECRET` from its pod environment (injected via External Secrets from AWS Secrets Manager). 2. Posts to `http://hydra-public:4444/oauth2/token` with HTTP Basic auth and `grant_type=client_credentials`. 3. Receives a short-lived opaque access token (default 1 hour, controlled by Hydra's `access_token.lifespan` setting). 4. Caches the token in memory until 60 seconds before expiry, then re-fetches. ``` Service A Hydra public │ │ ├─ POST /oauth2/token ──────────────────────▶ │ │ Basic n1-auth-svc: │ │ grant_type=client_credentials │ │ ◀────┤ │ {"access_token": "...", "expires_in": 3600} │ │ (cache token in memory until expiry - 60s) ``` ### Token presentation The calling service sets the token on outbound HTTP requests: ``` Authorization: Bearer ``` No custom headers. No `N1-Api-Key`. ### Token verification (acceptor side) The receiving service calls Hydra admin directly — not through Oathkeeper — to introspect the token: ``` Service B Hydra admin │ │ ├─ POST /admin/oauth2/introspect ──────────▶ │ │ token= │ │ ◀────┤ │ {"active": true, "client_id": "n1-forge-sentinel-svc", │ "scope": "reports:read records:read billing:write", ...} │ ├── check: active == true ├── check: required scope present └── proceed (or 403) ``` On HTTP 401 from the downstream service, the caller invalidates its token cache and fetches a fresh token before retrying once. This handles the case where a token was revoked or Hydra was restarted. --- ## Client registry Every service that calls another N1 service has a registered Hydra client. Clients are provisioned by the `hydra-client-provisioner` Helm chart, which runs as a one-shot Kubernetes Job on install/upgrade and upserts each client via the Hydra admin API. Secrets are stored in AWS Secrets Manager at `{env}-cluster/hydra-clients/{clientId}` and pulled into the pod by External Secrets Operator. | Client ID | Service | Scopes | Calls | |---|---|---|---| | `n1-auth-svc` | authentication-service | `profile:read profile:write records:read billing:read billing:write` | api-backend, billing-service | | `n1-api-backend-svc` | api-backend | `profile:read profile:write records:read records:write reports:read reports:write billing:read billing:write` | authentication-service, billing-service | | `n1-billing-svc` | billing-service | `profile:read` | authentication-service | | `n1-ws-proxy-svc` | api-websocket-proxy | `profile:read` | api-backend | | `n1-parser-router-svc` | parser-router | `records:read records:write` | api-backend | | `n1-parser-sequential-svc` | parser-sequential | `records:read records:write genetics:write` | api-backend | | `n1-rosetta-grouper-svc` | rosetta-grouper | `records:read records:write` | api-backend | | `n1-helix-parser-svc` | helix-parser | `records:read genetics:read genetics:write` | api-backend | | `n1-data-validation-svc` | data-validation-service | `records:read` | api-backend | | `n1-forge-sentinel-svc` | forge-sentinel | `reports:read reports:write records:read billing:read billing:write` | api-backend, billing-service | | `n1-forge-runner-svc` | forge-runner / workflow-* | `reports:read reports:write records:read genetics:read` | api-backend | | `n1-admin-dashboard-svc` | admin-dashboard | `profile:read profile:write records:read records:write reports:read reports:write billing:read` | api-backend, billing-service, api-websocket-proxy | --- ## Scope catalog Scopes are defined by the accepting service. The caller requests the scopes it needs; the acceptor enforces the minimum required for each endpoint. | Scope | Meaning | Accepting service(s) | |---|---|---| | `profile:read` | Read user profile and identity data | api-backend, authentication-service | | `profile:write` | Create or modify user profiles | api-backend, authentication-service | | `records:read` | Read medical records and extracted biomarkers | api-backend | | `records:write` | Write or update parsed clinical data | api-backend | | `genetics:read` | Read genetic variants | api-backend | | `genetics:write` | Write genetic variants | api-backend | | `reports:read` | Read CHR report state and status | api-backend | | `reports:write` | Create or update CHR reports | api-backend | | `billing:read` | Query quota, plans, and usage | billing-service | | `billing:write` | Onboard users, issue credits, create keys | billing-service | --- ## Accepting services — server-side enforcement ### api-backend All PHI endpoints require a valid s2s token when called without a user JWT. The scope required depends on the endpoint family: - `GET /records/*` — `records:read` - `POST /records/*`, `PATCH /records/*` — `records:write` - `GET /reports/*` — `reports:read` - `POST /reports/*` — `reports:write` - `GET /users/*` — `profile:read` ### billing-service Two scope tiers enforced via `require_service_token(scope)` FastAPI dependency: - `billing:read` — quota status, plan listing, usage queries - `billing:write` — user onboarding, credit issuance, API key management ### api-websocket-proxy The `/events/push` endpoint (server → client push) requires a token whose introspection returns `account_type=service`. Only `n1-api-backend-svc` holds a token with this claim. ### authentication-service Inbound calls from billing and api-backend are verified against their respective client IDs before any user data is modified. --- ## Token lifecycle ```text make call ──────▶ │ if now() > _expires - 60s: │ ```text │ fetch new token │ │ else: │ │ use cached token │ ``` ``` Key properties: - Cache is process-local (not Redis). A pod restart always fetches a fresh token. - The 60-second buffer prevents using a token that expires mid-flight. - A single 401-triggered retry covers Hydra restarts and secret rotations. - The `_ServiceTokenCache.invalidate()` method in `n1r-phoenix` and equivalent helpers in other services all follow the same pattern. --- ## Secret storage and rotation | What | Where | |---|---| | Client secrets | AWS Secrets Manager: `{env}-cluster/hydra-clients/{clientId}` | | Pod injection | External Secrets Operator → Kubernetes Secret → pod env | | Rotation | Update the value in Secrets Manager; ESO refresh interval is 1 hour; pod picks it up on the next token fetch after the cache expires | Rotation does not require a pod restart. Because tokens are short-lived (1 hour), and clients re-fetch on expiry or 401, a rotated secret is picked up within at most 1 hour with no downtime. --- ## Environment variables Every service that calls another N1 service expects these three env vars: | Variable | Example | Purpose | |---|---|---| | `HYDRA_URL` | `http://hydra-public:4444` | Token endpoint base URL | | `HYDRA_CLIENT_ID` | `n1-forge-sentinel-svc` | OAuth2 client identity | | `HYDRA_CLIENT_SECRET` | *(from Secrets Manager)* | Credential for token fetch | | `HYDRA_CLIENT_SCOPE` | `reports:read records:read` | Scopes to request (space-separated) | `HYDRA_CLIENT_SCOPE` is optional on callers that request their full registered scope by default. Acceptors do not use `HYDRA_CLIENT_SCOPE`; they read `HYDRA_ADMIN_URL` instead: | Variable | Example | Purpose | |---|---|---| | `HYDRA_ADMIN_URL` | `http://hydra-admin:4445` | Introspection endpoint | --- ## What was removed The following are no longer used on any service-to-service path and should not appear in new code or configuration: - `N1-Api-Key` HTTP header - `N1_API_KEY` environment variable (in the context of s2s calls — it may still appear in legacy tooling or developer scripts where Hydra credentials are not yet configured, but those are transitional and not a supported production path) - Static pre-shared secrets in Helm values or Kubernetes ConfigMaps - Any `validate_api_key()` function in the s2s authentication path PHI still flows between services. The authentication change removes static secrets; it does not add any guardrails on the content of PHI-carrying API calls, which is the correct behaviour for a HIPAA platform where the LLM is the intended consumer of PHI. --- ## Provisioning a new service To add a new backend service to the s2s trust fabric: 1. **Add a client entry** in `n1-helm-charts/environments/{env}/hydra-client-provisioner.yaml` with the narrowest scope the service actually needs. 2. **Create the secret** in AWS Secrets Manager at `{env}-cluster/hydra-clients/{clientId}` (generate a random 40+ character secret). 3. **Add the External Secret** entry in the `hydra-client-provisioner` chart values so ESO pulls the secret into the pod. 4. **Set the four env vars** (`HYDRA_URL`, `HYDRA_CLIENT_ID`, `HYDRA_CLIENT_SECRET`, `HYDRA_CLIENT_SCOPE`) on the calling service's Helm chart deployment. 5. **Verify on the acceptor side** that the scope the new client requests is enforced on the relevant endpoints. 6. **Run the provisioner** — it is a `post-install,post-upgrade` Helm hook, so a `helm upgrade` on the hydra-client-provisioner chart triggers the Job. --- ## Diagram — who calls whom ``` ```text token fetch ─────┘ introspect ─┘ ``` ┌───────────────────────────┼──────────────────────┼────────────────────┐ │ │ │ │ │ phoenix workers │ api-backend │ billing-service │ │ (router, parser, │ ◀── Bearer token ── │ ◀── Bearer token │ │ grouper, helix) │ + scope check │ + scope check │ │ │ │ │ │ │ └──── Bearer ────▶ api-backend │ │ │ │ │ │ │ forge-sentinel ─── Bearer ──────▶ api-backend │ │ │ forge-sentinel ─── Bearer ──────────────────────▶ billing-service │ │ │ │ authentication-service ─── Bearer ──────────────▶ billing-service │ │ authentication-service ─── Bearer ──▶ api-backend │ │ │ │ api-websocket-proxy ─── Bearer ──────▶ api-backend (/events/push) │ │ │ │ admin-dashboard ─── Bearer ──────────▶ api-backend │ │ admin-dashboard ─── Bearer ──────────────────────▶ billing-service │ └────────────────────────────────────────────────────────────────────────┘ ``` ---