# Yemaya Domain - Developer Guide

Yemaya is the Creative Studio domain. It serves as the central orchestration
layer and user-facing platform, providing the API gateway, background workers,
web studio, desktop application, and CLI. Yemaya integrates with all other
domains (Isis for generation, Hathor for worldbuilding, Sophia for research,
Bellona for builds) and manages users, projects, organizations, assets,
collaboration, and subscriptions.

---

## Prerequisites

### Required Software

| Software   | Minimum Version | Purpose                                   |
| ---------- | --------------- | ----------------------------------------- |
| Node.js    | >= 20.0.0       | Runtime for API, workers, and CLI         |
| pnpm       | >= 9.0.0        | Package manager (monorepo workspace)      |
| Docker     | >= 24.0         | Infrastructure (PostgreSQL, Redis, MinIO) |
| TypeScript | ^6.0.3          | Type checking and compilation             |

### For Desktop Development

| Software | Purpose                                |
| -------- | -------------------------------------- |
| Electron | Desktop app framework (studio-desktop) |

---

## Project Structure

```
apps/yemaya/
  api/               # API Gateway - RESTful + gRPC (Hono, OpenAPI, Scalar docs)
  workers/           # Background task processors (BullMQ)
  studio-web/        # React SPA for the creative studio
  studio-desktop/    # Electron desktop application
  cli/               # Command-line interface

libs/yemaya/
  sdk/               # TypeScript SDK with React hooks
  sdk-cpp/           # C++ SDK (CMake)
  sdk-python/        # Python SDK
  database/          # Prisma schema and generated client
  core/              # Core domain logic
  auth/              # Authentication library
  agents/            # AI agent orchestration
  assets/            # Asset management
  asset-library/     # Asset library browsing
  collaboration/     # Real-time collaboration
  community/         # Community features
  enterprise/        # Enterprise features
  event-handlers/    # Cross-domain event handling
  event-publisher/   # Cross-domain event publishing
  marketplace/       # Plugin marketplace
  orchestration/     # Multi-domain orchestration
  organizations/     # Organization management
  projects/          # Project management
  rbac/              # Role-based access control
  safety/            # Content safety and moderation
  ui/                # Shared UI components
  users/             # User management
  autonomous-pipelines/ # Autonomous production pipelines
  activity/          # Activity tracking
```

### API Dependencies

The API integrates with every domain via their client libraries:

- `@isis/client` - Generation services
- `@sophia/client` - Search and knowledge services
- `@hathor/client` - Worldbuilding services
- `@oshun/bellona-client` - Build and engine services

---

## Local Development Setup

### 1. Start Infrastructure

Yemaya requires PostgreSQL, Redis, and MinIO (S3-compatible storage).

```bash
# Start all core infrastructure
docker compose -f docker/docker-compose.dev.yml up -d
```

This starts:

- PostgreSQL (port 5432) - with `yemaya` database
- Redis (port 6379) - for caching and job queues
- MinIO (ports 9000/9001) - for asset storage

### 2. Set Environment Variables

| Variable                | Default                                              | Used By      |
| ----------------------- | ---------------------------------------------------- | ------------ |
| `YEMAYA_DATABASE_URL`   | `postgresql://oshun:oshun_dev@localhost:5432/yemaya` | database     |
| `DATABASE_URL`          | Same as above                                        | workers      |
| `REDIS_HOST`            | `localhost`                                          | api, workers |
| `REDIS_PORT`            | `6379`                                               | api, workers |
| `REDIS_PASSWORD`        | (none for local)                                     | api, workers |
| `MINIO_ENDPOINT`        | `localhost`                                          | workers      |
| `MINIO_PORT`            | `9000`                                               | workers      |
| `MINIO_ACCESS_KEY`      | `minioadmin`                                         | workers      |
| `MINIO_SECRET_KEY`      | `minioadmin`                                         | workers      |
| `ANTHROPIC_API_KEY`     | (optional)                                           | AI agents    |
| `OPENAI_API_KEY`        | (optional)                                           | AI agents    |
| `GOOGLE_API_KEY`        | (optional)                                           | AI agents    |
| `LOG_LEVEL`             | `info`                                               | all services |
| `LOG_PRETTY`            | `true` (dev) / `false` (prod)                        | all services |
| `WORKER_CONCURRENCY`    | `5`                                                  | workers      |
| `WORKER_JOB_TIMEOUT_MS` | `300000` (5 min)                                     | workers      |

### 3. Database Setup (Prisma)

```bash
cd libs/yemaya/database

# Generate the Prisma client
npx prisma generate

# Apply schema to local database
npx prisma db push

# Or run migrations if available
npx prisma migrate dev
```

### 4. Install Dependencies

```bash
pnpm install
```

---

## Running Services

### API Gateway

The main API service built with Hono, providing REST and gRPC endpoints with
OpenAPI documentation (via Scalar).

```bash
# Development (hot reload)
pnpm nx serve @yemaya/api

# Or directly
cd apps/yemaya/api
pnpm dev   # tsx watch src/index.ts
```

**Features:**

- OpenAPI schema generation: `pnpm nx openapi:generate @yemaya/api`
- OpenAPI validation: `pnpm nx openapi:validate @yemaya/api`
- Code generation for SDKs: `pnpm nx codegen @yemaya/api` (TypeScript, Python,
  Go)
- Docker build: `pnpm nx docker-build @yemaya/api`
- Type checking: `pnpm nx typecheck @yemaya/api`

**API architecture:**

- Hono routes with `@hono/zod-openapi` for typed endpoints
- Swagger UI via `@hono/swagger-ui`
- Scalar API reference via `@scalar/hono-api-reference`
- JWT authentication via `jose`
- Password hashing via `bcrypt`
- Structured logging via `pino`
- Redis caching via `ioredis`

### Background Workers

Process background tasks using BullMQ queues. Workers can be started
individually or all at once.

```bash
# Start all workers
pnpm nx serve @yemaya/workers

# Start specific worker types
cd apps/yemaya/workers
pnpm start:orchestration   # --worker=orchestration
pnpm start:rendering       # --worker=rendering
pnpm start:export          # --worker=export
pnpm start:pipeline        # --worker=pipeline
pnpm start:notification    # --worker=notification
```

**Worker types:**

| Worker               | File                      | Purpose                          |
| -------------------- | ------------------------- | -------------------------------- |
| Agent Worker         | `agent-worker.ts`         | AI agent task execution          |
| Asset Worker         | `asset-worker.ts`         | Asset processing and thumbnails  |
| Media Worker         | `media-worker.ts`         | Media transcoding and processing |
| 3D Generation Worker | `3d-generation-worker.ts` | 3D model generation              |
| Notification Worker  | `notification-worker.ts`  | Push/email notifications         |

**Worker configuration** is loaded from environment variables via Zod schema
validation (`src/config.ts`). Invalid configuration causes the worker to exit
with a descriptive error.

### Studio Web (React SPA)

```bash
pnpm nx serve @yemaya/studio-web
```

**Features:**

- Vite dev server
- i18n support (English, Spanish, French)
- Storybook stories for UI components
- Zustand stores for state management (auth, projects, assets, AI,
  collaboration, themes)
- PWA support with service worker (`src/sw.ts`)
- Route-based code splitting with lazy loading (`src/lib/lazy.tsx`)
- Accessibility testing (`src/test/a11y.test.tsx`)
- Deep linking support (`src/lib/deepLink.ts`)
- Performance monitoring (`src/lib/performance.ts`)

### Studio Desktop (Electron)

```bash
pnpm nx serve @yemaya/studio-desktop
```

### CLI

```bash
pnpm nx serve @yemaya/cli
```

---

## Building

### Build All Yemaya Projects

```bash
pnpm nx run-many --target=build --projects=tag:scope:yemaya
```

### Build Individual Projects

```bash
# API (tsup bundler)
pnpm nx build @yemaya/api

# Workers (tsup bundler)
pnpm nx build @yemaya/workers

# Studio Web (Vite)
pnpm nx build @yemaya/studio-web

# Desktop (Electron)
pnpm nx build @yemaya/studio-desktop

# SDK (tsup)
pnpm nx build @yemaya/sdk

# C++ SDK (CMake)
cd libs/yemaya/sdk-cpp
mkdir build && cd build
cmake ..
make

# Python SDK
cd libs/yemaya/sdk-python
pip install -e .
```

### Docker Builds

```bash
# API Docker image
pnpm nx docker-build @yemaya/api

# Workers Docker image
pnpm nx docker-build @yemaya/workers
```

---

## Testing

### Test Runner Configuration

All Yemaya apps use Vitest via the `@nx/vite:test` executor.

### Running Tests

```bash
# All Yemaya projects
pnpm nx run-many --target=test --projects=tag:scope:yemaya

# Individual projects
pnpm nx test @yemaya/api
pnpm nx test @yemaya/workers
pnpm nx test @yemaya/studio-web

# With coverage
cd apps/yemaya/api
pnpm test:coverage   # vitest run --coverage

# Watch mode
cd apps/yemaya/api
pnpm test:watch      # vitest
```

### Linting & Formatting

```bash
# Lint all
pnpm nx run-many --target=lint --projects=tag:scope:yemaya

# Individual
pnpm nx lint @yemaya/api

# Format check
cd apps/yemaya/api
pnpm format:check    # prettier --check src/

# Auto-fix
cd apps/yemaya/api
pnpm lint:fix        # eslint src/ --fix
pnpm format          # prettier --write src/
```

### Type Checking

```bash
pnpm nx typecheck @yemaya/api
pnpm nx typecheck @yemaya/workers
```

---

## Database Schema

The Yemaya database (`libs/yemaya/database/prisma/schema.prisma`) uses
PostgreSQL and is the largest schema in the monorepo. It manages:

- **Users & Authentication** - Users, accounts (OAuth), sessions, API keys,
  preferences
- **Organizations & Teams** - Organizations, members, roles, invitations
- **Projects & Workspaces** - Projects, project members, permissions
- **Asset Library** - Assets with metadata, tags, thumbnails, versions
- **Activity & Notifications** - Activity feed, in-app and email notifications
- **Billing & Subscriptions** - Plans, subscriptions, invoices, payment methods
- **Plugin Marketplace** - Plugins, versions, reviews, installations

The database URL is `YEMAYA_DATABASE_URL`.

Domain-specific data (generation jobs, world state, research documents, build
pipelines) lives in separate domain databases accessed via domain client
libraries.

### Prisma Workflow

```bash
cd libs/yemaya/database

# Generate client after schema changes
npx prisma generate

# Create new migration
npx prisma migrate dev --name your_change_description

# Apply migrations in production
npx prisma migrate deploy

# View database in browser
npx prisma studio
```

---

## Common Development Tasks

### Adding a New Production Pipeline

Yemaya's autonomous pipelines (`libs/yemaya/autonomous-pipelines/`) orchestrate
multi-step creative workflows across domains.

1. Define the pipeline steps and their domain dependencies:

   ```typescript
   export const myPipeline = {
     name: 'character-sheet-pipeline',
     steps: [
       { domain: 'sophia', action: 'research-character-archetype' },
       { domain: 'isis', action: 'generate-character-art' },
       { domain: 'hathor', action: 'create-world-entity' },
       { domain: 'bellona', action: 'export-to-engine' },
     ],
   };
   ```

2. Implement the pipeline in `libs/yemaya/autonomous-pipelines/`.

3. Add a BullMQ queue for the pipeline in `apps/yemaya/workers/src/queues.ts`.

4. Create or extend a worker in `apps/yemaya/workers/src/workers/` to process
   the pipeline jobs.

5. Add an API endpoint in `apps/yemaya/api/` for triggering the pipeline.

6. The worker uses domain client libraries (`@isis/client`, `@sophia/client`,
   `@hathor/client`, `@oshun/bellona-client`) to interact with each domain.

### Adding a New API Endpoint

1. Create the route file in `apps/yemaya/api/src/routes/`:

   ```typescript
   import { createRoute, z } from '@hono/zod-openapi';
   import type { AppType } from '../app.js';

   const myRoute = createRoute({
     method: 'get',
     path: '/my-resource',
     responses: {
       200: {
         description: 'Success',
         content: {
           'application/json': {
             schema: z.object({
               /* ... */
             }),
           },
         },
       },
     },
   });

   export function registerMyRoutes(app: AppType): void {
     app.openapi(myRoute, async (c) => {
       // Handler implementation
       return c.json({
         /* ... */
       });
     });
   }
   ```

2. Register the routes in the app setup.

3. Run `pnpm nx openapi:generate @yemaya/api` to update the OpenAPI spec.

4. Run `pnpm nx openapi:validate @yemaya/api` to verify the spec.

### Adding a New Worker Type

1. Create the worker file in `apps/yemaya/workers/src/workers/`:

   ```typescript
   import { Worker, type Job } from 'bullmq';

   export function createMyWorker(connection: RedisConnection) {
     return new Worker(
       'my-queue',
       async (job: Job) => {
         // Process the job
       },
       { connection, concurrency: 5 }
     );
   }
   ```

2. Add the queue definition in `src/queues.ts`.

3. Register the worker in `src/workers/index.ts`.

4. Add the worker type to the config schema in `src/config.ts`:

   ```typescript
   type: z.enum(['all', 'agent', 'media', 'asset', 'notification',
                  'generation-3d', 'my-worker']).default('all'),
   ```

5. Add a start script in `package.json`:

   ```json
   "start:my-worker": "node dist/index.js --worker=my-worker"
   ```

### Adding SDK React Hooks

The TypeScript SDK (`libs/yemaya/sdk/`) includes React hooks:

1. Add a new hook in `libs/yemaya/sdk/src/hooks/`:

   ```typescript
   import { useQuery, useMutation } from '@tanstack/react-query';
   import { useYemayaClient } from './useYemaya.js';

   export function useMyResource(id: string) {
     const client = useYemayaClient();
     return useQuery({
       queryKey: ['my-resource', id],
       queryFn: () => client.myResource.get(id),
     });
   }
   ```

2. Export from `libs/yemaya/sdk/src/react.ts`.

---

## Debugging Tips

### API Not Starting

- Verify PostgreSQL is running and `YEMAYA_DATABASE_URL` is set
- Check Redis connectivity (required for caching)
- Ensure Prisma client is generated
- The API uses `jose` for JWT; check `JWT_SECRET` environment variable
- Review Hono middleware chain for error middleware

### Worker Not Processing Jobs

- Workers connect to Redis via `REDIS_HOST`/`REDIS_PORT`
- Check which worker type is running (see `--worker=` argument)
- Verify queue names match between API job submission and worker registration
- Worker config validation is strict (Zod schema); check console for validation
  errors on startup
- Stalled jobs are detected at `WORKER_STALLED_INTERVAL` (default 30s)
- Jobs timeout at `WORKER_JOB_TIMEOUT_MS` (default 5 minutes)

### Studio Web Issues

- The studio uses Zustand for state management with devtools
- Check store middleware (`src/stores/middleware.ts`) for persistence issues
- i18n locale files are in `src/i18n/locales/{lang}/`
- Route prefetching is in `src/lib/routePrefetch.ts`
- Service worker issues: check `src/sw.ts`

### Cross-Domain Integration Issues

- The API imports domain clients (`@isis/client`, etc.)
- Event handlers (`@yemaya/event-handlers`) process events from other domains
- Event publishing (`@yemaya/event-publisher`) sends events to other domains
- Check the event bus connection (Redis Streams)

### Database Issues

- The Yemaya schema is large; `prisma generate` may take a few seconds
- Use `npx prisma studio` to browse data visually
- For migration conflicts: `npx prisma migrate reset` (destructive)
- The generated Prisma client is at `libs/yemaya/database/src/generated/client/`

### Performance

- API uses Redis caching via `@oshun/cache`
- Workers have configurable concurrency (default 5 per type)
- The studio web uses lazy loading and route-based code splitting
- Performance monitoring is available via `src/lib/performance.ts`
- Optimistic updates are in `src/stores/optimistic.ts`
