# Journey: Tenant integrations API keys and webhooks The platform integrations registry has a strong secret-safe API-key lifecycle and a real signed-webhook state machine. `/operator/tenant` can inspect the registry, create one key, create one sandbox subscription, and run the simulator. Rotation, revoke, dispatch, receipt, replay, deactivate, and connector controls remain API-only. Most importantly, “dispatch” and the replay worker produce signed request envelopes but never perform the partner HTTP request; delivery receipts must be posted by an external actor. ## Personas - **Platform integration operator** — holds `admin:*` or `admin:studio` and mutates the central registry. - **Partner developer** — receives a one-time API token or webhook signing secret and operates an external endpoint. - **Delivery worker** — is expected to send signed requests and report partner outcomes, but is not implemented in this BFF. - **Auditor** — reads structured `admin.integrations.*` lifecycle evidence. ## Pre-conditions - Platform-admin authentication is available. Tenant-console workspace scope intentionally receives `403` from every registry mutation/read. - The UI defaults to registered sandbox tenant `t-sandbox-alpha`; simulator and dispatch reject non-sandbox tenants. - Admin durable snapshots are configured if registry state must survive restart. API-key hashes, subscriptions, deliveries/events, connector state, and display names are persisted. - Webhook raw signing secrets are held only in a process-local map. They are not part of the durable snapshot, so a restored subscription cannot sign until it is recreated. ## Steps ### 1. Inspect the registry `GET /v1/admin/integrations/snapshot` returns redacted API-key summaries, redacted subscriptions, deliveries, connectors, docs, sandbox tenants, and a seed-data notice. The notice disappears after the first real registry record. The browser panel shows the latest three keys, webhooks, and deliveries. ### 2. Create and use an API key Create accepts tenant id, display name, scopes, optional IP allowlist, rate limit, and expiry. The raw `oshun_…` token is returned once; the durable record stores its SHA-256 hash and public prefix. Later snapshots never reveal token or hash. `GET /v1/integrations/auth-probe` is the only customer-like resource wired to this key authenticator. It checks token state/expiry, tenant, required scope, and IP, then updates `lastUsedAt`. This proves the auth primitive, not broad adoption across tenant APIs. ### 3. Rotate or revoke at the API boundary Rotation creates a new key linked by `rotatedFromKeyId`, returns its raw token once, and gives the prior key a grace expiry. Revoke timestamps the key; duplicate revoke returns `409`. These operations and their read-backs are covered over real HTTP but have no controls in `TenantIntegrationsPanel`. ### 4. Create a webhook and run the simulator The browser/API create an active subscription with tenant, display name, topic patterns, HTTPS target, retry parameters, and one-time signing secret. A matching sandbox simulation builds a canonical body and HMAC signature, then synthetically moves the delivery to success. It does not call the configured target URL. Inactive or nonmatching simulations record a failed delivery and return no signed request; a production tenant is rejected. ### 5. Dispatch and record partner outcomes API-only dispatch validates sandbox tenant, active state, topic, and available raw signing secret. It stores an event and an `in-flight` delivery and returns the signed request envelope. There is no `fetch`, queue adapter, or network client in the route/store. An external sender can post a receipt. Success records response metadata; transient failure schedules retry; exhausted failure becomes `dead-letter`. The receipt endpoint trusts the authenticated platform operator's supplied outcome—it is not a callback authenticated by the partner endpoint. ### 6. Replay due deliveries The replay route and optional interval scan due pending records, rebuild their signed request, and move them back to `in-flight`. The interval is explicitly single-instance/leader-election work. It also does not send the request; without an external consumer, a replay can remain in-flight indefinitely. After durable hydration, signing secrets are empty by design. Existing subscriptions/deliveries remain visible, but simulator, dispatch, and replay skip/fail with `signing_secret_unavailable` until the subscription is recreated. ### 7. Maintain connector and audit projections Connector upgrade enforces known/nondeprecated versions. Health-probe accepts a caller-supplied `healthy`/failure verdict and response metadata and updates the circuit state; it does not itself contact the connector. API-key, webhook, simulator, dispatch, receipt, replay, activation, upgrade, and probe actions emit structured `admin.integrations.*` events searchable through both admin search and the audit-log explorer. ## Post-conditions - API-key hashes and lifecycle metadata are durable and a valid key can pass the dedicated scoped auth probe. - Webhook subscriptions/delivery state and audit evidence are durable, while the raw signing material needed for future signatures is not. - The browser can create keys/subscriptions and prove a synthetic signed simulator result without exposing persisted secrets. - Retry/dead-letter state transitions are real when receipts are supplied. - No partner HTTP delivery, partner-authenticated receipt, automatic response observation, or post-restart signing continuity is provided by this path. ## Failure modes - **Wrong scope** — tenant-console workspace admins cannot read or mutate even a tenant-local subset. - **One-time secret loss** — leaving the creation result loses the raw API key or signing secret; neither can be recovered. - **Webhook restart break** — durable subscriptions restore without raw signing secrets, so dispatch/replay cannot sign them. - **Synthetic simulator** — a success proves canonical signing logic, not that the partner URL is reachable or accepts the payload. - **No sender** — dispatch/replay returns an envelope and state transition but performs no network request. - **Receipt trust boundary** — platform-authenticated callers provide status and response excerpts; no delivery-specific partner signature is verified. - **In-flight orphan** — absent an external sender/receipt, deliveries do not resolve. - **Auth-probe island** — key enforcement is proven on one probe route, not on general tenant resources. - **Reported health** — connector probes record the submitted verdict rather than measuring an endpoint themselves. - **Fire-and-forget persistence** — registry mutations return before async snapshot flush; a crash in that interval can lose the latest delta. ## E2E coverage - [`apps/oshun/web/e2e/tenant-integrations-api-keys-webhooks-bff.spec.ts`](../../apps/oshun/web/e2e/tenant-integrations-api-keys-webhooks-bff.spec.ts) covers platform gates, key create/rotate/revoke/auth-probe, redaction, simulator branches, dispatch/receipt/replay/dead-letter, activation, connector changes, and audit/search read-back over the real BFF. - [`apps/oshun/web/e2e/tenant-integrations-ui.spec.ts`](../../apps/oshun/web/e2e/tenant-integrations-ui.spec.ts) covers live snapshot, browser key/subscription creation, one-time secret rendering, redacted refresh, and signed simulator output. - **Coverage depth: deep for the state machine, partial for delivery.** Tests correctly prove envelope generation and caller-posted receipts; no test can prove external HTTP send/receipt or restored-secret continuity because those joins are absent. ## Per-view files touched - [`operator/operator-tenant.md`](../operator/operator-tenant.md) — registry, key create, webhook create, and simulator UI. - [`operator/operator-audit.md`](../operator/operator-audit.md) — lifecycle evidence. - [`operator/operator-tenants.md`](../operator/operator-tenants.md) — adjacent platform tenant context. - Rotation/revoke/dispatch/replay/connector operations have no dedicated browser view in this journey. ## Cross-references - [`tenant-bulk-member-invite-scim.md`](./tenant-bulk-member-invite-scim.md) — adjacent identity integration, not driven by these webhook envelopes. - [`tenant-oneroster-roster-sync.md`](./tenant-oneroster-roster-sync.md) — adjacent roster import, not automatically dispatched here. - [`tenant-audit-log-investigation.md`](./tenant-audit-log-investigation.md) — structured audit follow-up. - Sources: `admin-integrations-registry.ts`, `admin-integrations-registry-store.ts`, `durable-backed-integrations-registry-store.ts`, `server.ts`, and `TenantIntegrationsPanel.tsx` under `apps/oshun`. ## Open questions - Which queue/worker performs the actual HTTPS request, consumes replay envelopes, enforces timeouts, and posts authenticated receipts? - How should webhook signing secrets be encrypted and restored without making them visible in operator snapshots? - Should a delivery-specific capability authenticate receipt writes instead of a general platform-admin bearer? - Which real tenant APIs adopt the integration-key guard beyond the auth probe? - Which controls belong in the operator UI for rotate/revoke/deactivate, delivery inspection/replay, and connector health—and which belong in a tenant-admin-scoped subset?