Skip to content
Internal

Service Authentication — Zero-Trust Architecture

Service Authentication — Zero-Trust Architecture

Section titled “Service Authentication — Zero-Trust Architecture”
A calling service obtains a short-lived, scoped OAuth token from Hydra and uses it for a receiver that validates audience and scope.
Service-to-service OAuth2 client-credentials flow. Download SVG source

Current state as of 2026-06-18. Deployed on feat/auth-zero-trust. Covers service-to-service (s2s) auth only. User-facing auth (browser login, JWT bearer, Oathkeeper) is described in N1 landscape.

What changed and why

Section titled “What changed and why”

Previously every backend service that called another N1 service passed a static pre-shared secret in a custom N1-Api-Key HTTP header. The receiving service compared the incoming value against a secret loaded from the pod environment at startup. That model has three hard problems:

  1. No per-service identity. Every service shared the same secret; a leak anywhere compromised everything. There was no way to know which service made a given call.
  2. No scope enforcement. Any holder of the static key could call any endpoint — a parser had the same access as the forge orchestrator.
  3. No short-lived credentials. Rotating the key meant coordinating a simultaneous restart of every service that held it.

The new model gives every backend service its own Hydra OAuth2 client identity, issues short-lived Bearer tokens via the client_credentials grant, and enforces named scopes at the receiving endpoint. The static N1-Api-Key header and N1_API_KEY env var no longer exist on the service-to-service path.

How it works

Section titled “How it works”

Token acquisition

Section titled “Token acquisition”

A service that needs to call another N1 service:

  1. Reads HYDRA_URL, HYDRA_CLIENT_ID, HYDRA_CLIENT_SECRET from its pod environment (injected via External Secrets from AWS Secrets Manager).
  2. Posts to http://hydra-public:4444/oauth2/token with HTTP Basic auth and grant_type=client_credentials.
  3. Receives a short-lived opaque access token (default 1 hour, controlled by Hydra’s access_token.lifespan setting).
  4. Caches the token in memory until 60 seconds before expiry, then re-fetches.
Service A                                  Hydra public
  │                                             │
  ├─ POST /oauth2/token ──────────────────────▶ │
  │    Basic n1-auth-svc:<secret>               │
  │    grant_type=client_credentials            │
  │                                        ◀────┤
  │    {"access_token": "...", "expires_in": 3600}
  │
  │  (cache token in memory until expiry - 60s)

Token presentation

Section titled “Token presentation”

The calling service sets the token on outbound HTTP requests:

Authorization: Bearer <access_token>

No custom headers. No N1-Api-Key.

Token verification (acceptor side)

Section titled “Token verification (acceptor side)”

The receiving service calls Hydra admin directly — not through Oathkeeper — to introspect the token:

Service B                                  Hydra admin
  │                                             │
  ├─ POST /admin/oauth2/introspect ──────────▶ │
  │    token=<received Bearer token>            │
  │                                        ◀────┤
  │    {"active": true, "client_id": "n1-forge-sentinel-svc",
  │     "scope": "reports:read records:read billing:write", ...}
  │
  ├── check: active == true
  ├── check: required scope present
  └── proceed (or 403)

On HTTP 401 from the downstream service, the caller invalidates its token cache and fetches a fresh token before retrying once. This handles the case where a token was revoked or Hydra was restarted.

Client registry

Section titled “Client registry”

Every service that calls another N1 service has a registered Hydra client. Clients are provisioned by the hydra-client-provisioner Helm chart, which runs as a one-shot Kubernetes Job on install/upgrade and upserts each client via the Hydra admin API. Secrets are stored in AWS Secrets Manager at {env}-cluster/hydra-clients/{clientId} and pulled into the pod by External Secrets Operator.

Client ID Service Scopes Calls
n1-auth-svc authentication-service profile:read profile:write records:read billing:read billing:write api-backend, billing-service
n1-api-backend-svc api-backend profile:read profile:write records:read records:write reports:read reports:write billing:read billing:write authentication-service, billing-service
n1-billing-svc billing-service profile:read authentication-service
n1-ws-proxy-svc api-websocket-proxy profile:read api-backend
n1-parser-router-svc parser-router records:read records:write api-backend
n1-parser-sequential-svc parser-sequential records:read records:write genetics:write api-backend
n1-rosetta-grouper-svc rosetta-grouper records:read records:write api-backend
n1-helix-parser-svc helix-parser records:read genetics:read genetics:write api-backend
n1-data-validation-svc data-validation-service records:read api-backend
n1-forge-sentinel-svc forge-sentinel reports:read reports:write records:read billing:read billing:write api-backend, billing-service
n1-forge-runner-svc forge-runner / workflow-* reports:read reports:write records:read genetics:read api-backend
n1-admin-dashboard-svc admin-dashboard profile:read profile:write records:read records:write reports:read reports:write billing:read api-backend, billing-service, api-websocket-proxy

Scope catalog

Section titled “Scope catalog”

Scopes are defined by the accepting service. The caller requests the scopes it needs; the acceptor enforces the minimum required for each endpoint.

Scope Meaning Accepting service(s)
profile:read Read user profile and identity data api-backend, authentication-service
profile:write Create or modify user profiles api-backend, authentication-service
records:read Read medical records and extracted biomarkers api-backend
records:write Write or update parsed clinical data api-backend
genetics:read Read genetic variants api-backend
genetics:write Write genetic variants api-backend
reports:read Read CHR report state and status api-backend
reports:write Create or update CHR reports api-backend
billing:read Query quota, plans, and usage billing-service
billing:write Onboard users, issue credits, create keys billing-service

Accepting services — server-side enforcement

Section titled “Accepting services — server-side enforcement”

api-backend

Section titled “api-backend”

All PHI endpoints require a valid s2s token when called without a user JWT. The scope required depends on the endpoint family:

  • GET /records/* — records:read
  • POST /records/*, PATCH /records/* — records:write
  • GET /reports/* — reports:read
  • POST /reports/* — reports:write
  • GET /users/* — profile:read

billing-service

Section titled “billing-service”

Two scope tiers enforced via require_service_token(scope) FastAPI dependency:

  • billing:read — quota status, plan listing, usage queries
  • billing:write — user onboarding, credit issuance, API key management

api-websocket-proxy

Section titled “api-websocket-proxy”

The /events/push endpoint (server → client push) requires a token whose introspection returns account_type=service. Only n1-api-backend-svc holds a token with this claim.

authentication-service

Section titled “authentication-service”

Inbound calls from billing and api-backend are verified against their respective client IDs before any user data is modified.

Token lifecycle

Section titled “Token lifecycle”
   make call ──────▶ │  if now() > _expires - 60s:  │
```text
│     fetch new token          │
│  else:                       │
│     use cached token         │
Key properties:
- Cache is process-local (not Redis). A pod restart always fetches a fresh token.
- The 60-second buffer prevents using a token that expires mid-flight.
- A single 401-triggered retry covers Hydra restarts and secret rotations.
- The `_ServiceTokenCache.invalidate()` method in `n1r-phoenix` and equivalent helpers in other
  services all follow the same pattern.


---


## Secret storage and rotation


| What | Where |
|---|---|
| Client secrets | AWS Secrets Manager: `{env}-cluster/hydra-clients/{clientId}` |
| Pod injection | External Secrets Operator → Kubernetes Secret → pod env |
| Rotation | Update the value in Secrets Manager; ESO refresh interval is 1 hour; pod picks it up on the next token fetch after the cache expires |
Rotation does not require a pod restart. Because tokens are short-lived (1 hour), and clients
re-fetch on expiry or 401, a rotated secret is picked up within at most 1 hour with no downtime.


---


## Environment variables


Every service that calls another N1 service expects these three env vars:


| Variable | Example | Purpose |
|---|---|---|
| `HYDRA_URL` | `http://hydra-public:4444` | Token endpoint base URL |
| `HYDRA_CLIENT_ID` | `n1-forge-sentinel-svc` | OAuth2 client identity |
| `HYDRA_CLIENT_SECRET` | *(from Secrets Manager)* | Credential for token fetch |
| `HYDRA_CLIENT_SCOPE` | `reports:read records:read` | Scopes to request (space-separated) |
`HYDRA_CLIENT_SCOPE` is optional on callers that request their full registered scope by default.
Acceptors do not use `HYDRA_CLIENT_SCOPE`; they read `HYDRA_ADMIN_URL` instead:


| Variable | Example | Purpose |
|---|---|---|
| `HYDRA_ADMIN_URL` | `http://hydra-admin:4445` | Introspection endpoint |
---


## What was removed


The following are no longer used on any service-to-service path and should not appear in new
code or configuration:


- `N1-Api-Key` HTTP header
- `N1_API_KEY` environment variable (in the context of s2s calls — it may still appear in legacy
  tooling or developer scripts where Hydra credentials are not yet configured, but those are
  transitional and not a supported production path)
- Static pre-shared secrets in Helm values or Kubernetes ConfigMaps
- Any `validate_api_key()` function in the s2s authentication path


PHI still flows between services. The authentication change removes static secrets; it does not
add any guardrails on the content of PHI-carrying API calls, which is the correct behaviour for a
HIPAA platform where the LLM is the intended consumer of PHI.


---


## Provisioning a new service


To add a new backend service to the s2s trust fabric:


1. **Add a client entry** in `n1-helm-charts/environments/{env}/hydra-client-provisioner.yaml`
   with the narrowest scope the service actually needs.
2. **Create the secret** in AWS Secrets Manager at
   `{env}-cluster/hydra-clients/{clientId}` (generate a random 40+ character secret).
3. **Add the External Secret** entry in the `hydra-client-provisioner` chart values so ESO
   pulls the secret into the pod.
4. **Set the four env vars** (`HYDRA_URL`, `HYDRA_CLIENT_ID`, `HYDRA_CLIENT_SECRET`,
   `HYDRA_CLIENT_SCOPE`) on the calling service's Helm chart deployment.
5. **Verify on the acceptor side** that the scope the new client requests is enforced on the
   relevant endpoints.
6. **Run the provisioner** — it is a `post-install,post-upgrade` Helm hook, so a `helm upgrade`
   on the hydra-client-provisioner chart triggers the Job.


---


## Diagram — who calls whom
token fetch ─────┘          introspect ─┘

┌───────────────────────────┼──────────────────────┼────────────────────┐ │ │ │ │ │ phoenix workers │ api-backend │ billing-service │ │ (router, parser, │ ◀── Bearer token ── │ ◀── Bearer token │ │ grouper, helix) │ + scope check │ + scope check │ │ │ │ │ │ │ └──── Bearer ────▶ api-backend │ │ │ │ │ │ │ forge-sentinel ─── Bearer ──────▶ api-backend │ │ │ forge-sentinel ─── Bearer ──────────────────────▶ billing-service │ │ │ │ authentication-service ─── Bearer ──────────────▶ billing-service │ │ authentication-service ─── Bearer ──▶ api-backend │ │ │ │ api-websocket-proxy ─── Bearer ──────▶ api-backend (/events/push) │ │ │ │ admin-dashboard ─── Bearer ──────────▶ api-backend │ │ admin-dashboard ─── Bearer ──────────────────────▶ billing-service │ └────────────────────────────────────────────────────────────────────────┘

---