Common failure modes across the Demeter API, tests, and local dev, with the concrete cause and fix. Ordered roughly by how often they bite.
API won't start#
Symptom: buildServer() throws on boot; process exits immediately.
- Env validation failed.
src/config.tsvalidates the environment with Zod and throws on the first missing/invalid variable (e.g. absentJWT_SECRET, malformedDATABASE_URL, orDB_POOL_MIN > DB_POOL_MAX). Read the thrown message — it names the offending variable. Fix the env, restart. - Fastify plugin version mismatch. A plugin pinned to the wrong major (e.g.
fastify: '4.x'on a 5.x app) makesbuildServer()throw at registration. All Demeter plugins must declarefastify: '5.x'. (This was a real bug fixed during the persistence work.)
Database#
Symptom: /health/ready reports checks.database.status !== "ok", or
queries hang/error.
- Postgres unreachable. Confirm the dev stack is up
(
docker compose -f docker/docker-compose.dev.yml ps) andDATABASE_URLpoints at it. The readiness probe surfaces DB latency; a connection error there is the fastest signal. - Pool exhaustion under load. If requests queue and
connectionTimeoutMillis(5s) trips, raiseDB_POOL_MAX. The pool drains cleanly on shutdown via theonClosehook; a "Cannot use a pool after calling end" error means something is querying after shutdown began. - Missing tables / "relation does not exist". Migrations weren't applied to
the target DB. Run
pnpm --filter @demeter/api db:migrate.
Integration tests#
Symptom: integration specs fail or skip.
- Suite skipped entirely. The harness
skipIf(!dbAvailable)s when it can't reachdemeter_test. Bring up Postgres and ensureDEMETER_TEST_DATABASE_URL(defaultpostgresql://oshun:oshun_dev@localhost:5432/demeter_test) resolves. - "relation already exists" / stale schema after editing a migration. The
harness only applies new migration files; it never re-applies a changed one.
Reset the test schema so it replays from scratch:bash
psql "$DEMETER_TEST_DATABASE_URL" -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;' - Table-count smoke test fails.
server.integration.spec.tsasserts the table count (29). A new migration that adds/removes a table must update that assertion in the same change. - Auth rate-limit flakiness across tests. Integration specs use distinct
per-test source IPs (
10.X.*) to dodge the per-user/IP rate limiter; reusing an IP across many requests can trip a 429. Give each test its own IP, or setRATE_LIMIT_ENABLED=falsefor throughput-style tests.
Auth at runtime#
- 401 on a valid-looking request. The access token expired — clients must refresh (rotation issues a new refresh token each time; store the newest).
- Logged out of all devices unexpectedly. A revoked refresh token was replayed, so its whole family was revoked (reuse-detection working as designed). The client should treat this as "re-authenticate".
- OAuth login fails in tests. Don't call live providers — inject
deterministic verifiers via
buildServer({ verifiers })(see ADR-0005).
Rate limiting#
Symptom: 429 Too Many Requests.
- Expected under burst load per the per-user/IP tier. For load/perf testing set
RATE_LIMIT_ENABLED=false(the perf harness does this). Don't disable it in production.
Redis#
- Rate limiting/caching degraded. If
REDIS_URLis unreachable, the readiness probe reports it. Confirm Redis is up in the dev stack and the URL is correct.
Mobile test suite (this CI box)#
Symptom: jest-expo mobile tests fail to run with
Could not locate module react-native/... or an expo "winter" runtime
require-outside-scope error.
- This is a known environment limitation, not a code defect:
react-nativeis hoisted to the workspace root (so the app-localnode_modules/react-nativepath the jest config expects is absent), and the expo winter runtime is incompatible with the currentjest-runtimesandbox. It blocks the entire mobile suite. Mobile tests are written and type-check (tsc --noEmit) but must be executed in a properly provisioned mobile CI environment.