# Demeter Troubleshooting Guide

Common failure modes across the Demeter API, tests, and local dev, with the
concrete cause and fix. Ordered roughly by how often they bite.

## API won't start

**Symptom:** `buildServer()` throws on boot; process exits immediately.

- **Env validation failed.** `src/config.ts` validates the environment with Zod
  and throws on the first missing/invalid variable (e.g. absent `JWT_SECRET`,
  malformed `DATABASE_URL`, or `DB_POOL_MIN > DB_POOL_MAX`). Read the thrown
  message — it names the offending variable. Fix the env, restart.
- **Fastify plugin version mismatch.** A plugin pinned to the wrong major (e.g.
  `fastify: '4.x'` on a 5.x app) makes `buildServer()` throw at registration.
  All Demeter plugins must declare `fastify: '5.x'`. (This was a real bug fixed
  during the persistence work.)

## Database

**Symptom:** `/health/ready` reports `checks.database.status !== "ok"`, or
queries hang/error.

- **Postgres unreachable.** Confirm the dev stack is up
  (`docker compose -f docker/docker-compose.dev.yml ps`) and `DATABASE_URL`
  points at it. The readiness probe surfaces DB latency; a connection error
  there is the fastest signal.
- **Pool exhaustion under load.** If requests queue and
  `connectionTimeoutMillis` (5s) trips, raise `DB_POOL_MAX`. The pool drains
  cleanly on shutdown via the `onClose` hook; a "Cannot use a pool after calling
  end" error means something is querying after shutdown began.
- **Missing tables / "relation does not exist".** Migrations weren't applied to
  the target DB. Run `pnpm --filter @demeter/api db:migrate`.

## Integration tests

**Symptom:** integration specs fail or skip.

- **Suite skipped entirely.** The harness `skipIf(!dbAvailable)`s when it can't
  reach `demeter_test`. Bring up Postgres and ensure `DEMETER_TEST_DATABASE_URL`
  (default `postgresql://oshun:oshun_dev@localhost:5432/demeter_test`) resolves.
- **"relation already exists" / stale schema after editing a migration.** The
  harness only _applies new_ migration files; it never re-applies a changed one.
  Reset the test schema so it replays from scratch:
  ```bash
  psql "$DEMETER_TEST_DATABASE_URL" -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;'
  ```
- **Table-count smoke test fails.** `server.integration.spec.ts` asserts the
  table count (29). A new migration that adds/removes a table must update that
  assertion in the same change.
- **Auth rate-limit flakiness across tests.** Integration specs use distinct
  per-test source IPs (`10.X.*`) to dodge the per-user/IP rate limiter; reusing
  an IP across many requests can trip a 429. Give each test its own IP, or set
  `RATE_LIMIT_ENABLED=false` for throughput-style tests.

## Auth at runtime

- **401 on a valid-looking request.** The access token expired — clients must
  refresh (rotation issues a new refresh token each time; store the newest).
- **Logged out of all devices unexpectedly.** A revoked refresh token was
  replayed, so its whole family was revoked (reuse-detection working as
  designed). The client should treat this as "re-authenticate".
- **OAuth login fails in tests.** Don't call live providers — inject
  deterministic verifiers via `buildServer({ verifiers })` (see ADR-0005).

## Rate limiting

**Symptom:** `429 Too Many Requests`.

- Expected under burst load per the per-user/IP tier. For load/perf testing set
  `RATE_LIMIT_ENABLED=false` (the perf harness does this). Don't disable it in
  production.

## Redis

- **Rate limiting/caching degraded.** If `REDIS_URL` is unreachable, the
  readiness probe reports it. Confirm Redis is up in the dev stack and the URL
  is correct.

## Mobile test suite (this CI box)

**Symptom:** `jest-expo` mobile tests fail to run with
`Could not locate module react-native/...` or an `expo` "winter" runtime
`require`-outside-scope error.

- This is a known environment limitation, not a code defect: `react-native` is
  hoisted to the workspace root (so the app-local `node_modules/react-native`
  path the jest config expects is absent), and the expo winter runtime is
  incompatible with the current `jest-runtime` sandbox. It blocks the **entire**
  mobile suite. Mobile tests are written and type-check (`tsc --noEmit`) but
  must be executed in a properly provisioned mobile CI environment.
