# Yaa Inter-Package Communication Patterns

This document provides a comprehensive analysis of communication mechanisms used
across the Yaa codebase, including message queues, pub/sub patterns, REST APIs,
event systems, and database access patterns.

---

## Table of Contents

1. [Overview](#overview)
2. [BullMQ Job Queue Architecture](#bullmq-job-queue-architecture)
3. [Redis Pub/Sub Message Bus](#redis-pubsub-message-bus)
4. [REST API Communication](#rest-api-communication)
5. [Event Emitter Patterns](#event-emitter-patterns)
6. [Database Communication](#database-communication)
7. [Inter-Package Dependencies](#inter-package-dependencies)
8. [Communication Flow Diagrams](#communication-flow-diagrams)
9. [Integration Points Summary](#integration-points-summary)

---

## Overview

The Yaa codebase employs a distributed, event-driven architecture with multiple
communication mechanisms:

| Mechanism       | Technology         | Primary Use Case                |
| --------------- | ------------------ | ------------------------------- |
| Job Queues      | BullMQ + Redis     | Async task processing pipelines |
| Pub/Sub         | IORedis            | Inter-agent real-time messaging |
| REST APIs       | Fastify + fetch    | External service integration    |
| Event Emitters  | Node.js events     | Intra-process event handling    |
| Direct Database | pg (node-postgres) | Shared state management         |
| Package Imports | TypeScript         | Shared utilities and models     |

---

## BullMQ Job Queue Architecture

### Queue Registry

The system uses three primary BullMQ queues for content processing pipelines:

#### 1. Content Processing Queue

**Queue Name:** `yaa:content-processing`

**Location:** `packages/ingestion/processing-queue/`

| File                 | Purpose                            |
| -------------------- | ---------------------------------- |
| `src/queue.ts`       | Queue definition and configuration |
| `src/producer.ts`    | Job enqueueing logic               |
| `src/worker.ts`      | Job processing implementation      |
| `src/producer-db.ts` | Database queries for producer      |
| `src/worker-db.ts`   | Database queries for worker        |
| `src/pipeline.ts`    | Processing pipeline stages         |

**Job Type:** `process_feed_item`

**Job Data Structure:**

```typescript
interface ProcessFeedItemJob {
  feedItemId: number;
  normalizedFeedItemId: number | null;
  didNormalize: boolean;
  didDedupe: boolean;
  didLanguageDetect: boolean;
  didTimestampNormalize: boolean;
  didMediaExtract: boolean;
  didAttribution: boolean;
}
```

**Queue Configuration:**

```typescript
{
  defaultJobOptions: {
    attempts: 8,
    backoff: {
      type: 'exponential',
      delay: 1000  // 1 second base delay
    },
    removeOnComplete: 1000,
    removeOnFail: 1000
  }
}
```

**Pipeline Stages:**

1. Content normalization (HTML parsing, text extraction)
2. Semantic deduplication (Simhash64 fingerprinting)
3. Language detection (multi-language support)
4. Timestamp normalization (date parsing)
5. Media extraction (images, videos, links)
6. Source attribution

---

#### 2. Audio Generation Queue

**Queue Name:** `yaa:audio-generation`

**Location:** `packages/audio/audio-generation-queue/`

| File              | Purpose                    |
| ----------------- | -------------------------- |
| `src/queue.ts`    | Queue definition           |
| `src/producer.ts` | Article audio job creation |
| `src/worker.ts`   | TTS generation processing  |

**Job Type:** `generate_article_audio`

**Job Data Structure:**

```typescript
interface GenerateAudioJob {
  articleId: number;
  voiceProfileId: number;
  voiceProfileSlug: string;
  priority: number; // Computed from publish timestamp
}
```

**Queue Configuration:**

```typescript
{
  defaultJobOptions: {
    attempts: 6,
    backoff: {
      type: 'exponential',
      delay: 2000  // 2 second base delay
    },
    removeOnComplete: 1000,
    removeOnFail: 1000
  }
}
```

**TTS Providers:**

- **ElevenLabs** (`@yaa/audio-elevenlabs-service`) - English TTS
- **Ghana NLP** (`@yaa/ghana-nlp`) - Twi, Ewe, Ga, Hausa, Dagbani TTS

---

#### 3. Video Generation Queue

**Queue Name:** `yaa:video-generation`

**Location:** `packages/video/video-generation-queue/`

| File              | Purpose                                |
| ----------------- | -------------------------------------- |
| `src/queue.ts`    | Queue definition                       |
| `src/producer.ts` | Video job creation with anchor routing |
| `src/worker.ts`   | Video synthesis and packaging          |

**Job Types:**

| Job Type                    | Description                     |
| --------------------------- | ------------------------------- |
| `generate_article_video`    | Full video synthesis via HeyGen |
| `package_article_video_hls` | HLS packaging for streaming     |

**Job Data Structure:**

```typescript
interface GenerateVideoJob {
  articleId: number;
  anchorProfileId: number;
  anchorMode: 'fixed' | 'auto';
  priority: number;
}
```

**Anchor Routing Modes:**

- **Fixed Mode:** All articles use a single specified anchor profile
- **Auto Mode:** Load-balances articles across active anchor profiles
- **Fallback:** Uses fallback profile when no active anchors available

**Queue Configuration:**

```typescript
{
  defaultJobOptions: {
    attempts: 6,
    backoff: {
      type: 'exponential',
      delay: 2000
    },
    removeOnComplete: 1000,
    removeOnFail: 1000
  }
}
```

---

#### 4. Social Media Unified Scheduler

**Location:** `packages/social/unified-scheduler/`

**Job Type:** `publishUnified`

**Job Data Structure:**

```typescript
interface PublishUnifiedJob {
  postId: string;
  post: UnifiedScheduledPost;
}
```

**Features:**

- Multi-platform orchestration (Instagram, TikTok, YouTube, X/Twitter)
- Configurable concurrency per platform
- Sequential platform publishing with delays
- Retry handling per platform

---

### Producer-Consumer Flow

```
┌─────────────────────────────────────────────────────────────────────┐
│                     Content Processing Pipeline                      │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  RSS/Sitemap     ┌──────────────┐      ┌─────────────────┐          │
│  Collectors  ──▶ │  feed_items  │ ───▶ │ Processing Queue│          │
│                  │  (database)  │      │    Producer     │          │
│                  └──────────────┘      └────────┬────────┘          │
│                                                 │                    │
│                                                 ▼                    │
│                                        ┌──────────────┐             │
│                                        │ yaa:content- │             │
│                                        │  processing  │             │
│                                        │   (Redis)    │             │
│                                        └──────┬───────┘             │
│                                               │                      │
│                                               ▼                      │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │                    Processing Worker                         │   │
│  │  ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌──────────┐ │   │
│  │  │ Normalize  │▶│  Dedupe    │▶│ Lang Detect│▶│ Extract  │ │   │
│  │  │  Content   │ │ (Simhash)  │ │            │ │  Media   │ │   │
│  │  └────────────┘ └────────────┘ └────────────┘ └──────────┘ │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                               │                      │
│                                               ▼                      │
│                                   ┌───────────────────┐             │
│                                   │ normalized_feed_  │             │
│                                   │  items (database) │             │
│                                   └───────────────────┘             │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘
```

---

## Redis Pub/Sub Message Bus

### Agent Communication System

**Location:** `packages/ai/agents/communication/message-bus.ts`

The MessageBus provides real-time inter-agent communication using Redis Pub/Sub.

### Architecture

```typescript
class MessageBus {
  private redis: Redis; // Main Redis client
  private subscriber: Redis; // Dedicated subscriber client
  private queues: Map<string, Message[]>; // In-memory message queues
  private subscriptions: Map<string, Set<(msg: Message) => void>>;
}
```

### Channel Naming Convention

| Channel Pattern           | Purpose                               |
| ------------------------- | ------------------------------------- |
| `yaa:messages:${agentId}` | Direct agent-to-agent messages        |
| `yaa:messages:broadcast`  | System-wide broadcasts                |
| `yaa:queue:${agentId}`    | Persistent message queue (Redis List) |

### Message Structure

```typescript
interface Message<T = unknown> {
  type: string;
  from: string; // Sender agent ID
  to: string | null; // Recipient (null for broadcast)
  payload: T;
  priority: 'low' | 'normal' | 'high' | 'critical';
  ttlMs: number; // Time-to-live (default: 5 minutes)
  timestamp: number;
  correlationId?: string; // For request-response tracking
}
```

### Message Bus Operations

#### Sending Direct Messages

```typescript
// Send targeted message to specific agent
await messageBus.send({
  type: 'TASK_ASSIGNMENT',
  from: 'orchestrator',
  to: 'content-analyzer-1',
  payload: { articleId: 12345 },
  priority: 'high',
  ttlMs: 300000, // 5 minutes
});
```

#### Broadcasting

```typescript
// Broadcast to all subscribed agents
await messageBus.broadcast({
  type: 'SYSTEM_ALERT',
  from: 'coordinator',
  to: null,
  payload: { message: 'Scaling up workers' },
  priority: 'critical',
});
```

#### Subscribing

```typescript
// Subscribe to messages
messageBus.subscribe('my-agent-id', (message) => {
  console.log(`Received: ${message.type} from ${message.from}`);
});
```

### Message Persistence

Messages are persisted to Redis Lists for reliability:

```typescript
// Queue key pattern
const queueKey = `yaa:queue:${agentId}`;

// Operations
LPUSH queueKey message      // Enqueue
LRANGE queueKey 0 -1        // Read all
LTRIM queueKey 0 maxSize    // Limit queue size
```

### Retry Configuration

```typescript
const DEFAULT_OPTIONS = {
  maxQueueSize: 1000,
  maxRetries: 3,
  retryBackoffMs: 1000, // Exponential backoff base
  messageTtlMs: 300000, // 5 minutes default TTL
};
```

### Statistics API

```typescript
interface MessageBusStats {
  subscriptions: number; // Active subscriptions
  broadcasts: number; // Broadcasts sent
  queued: number; // Messages in queues
  pending: number; // Pending delivery
  historySize: number; // Messages in history
}
```

---

## REST API Communication

### API Server Architecture

**Location:** `packages/api/`

**Framework:** Fastify

### Route Structure

```
packages/api/src/
├── interfaces/http/
│   ├── server.ts           # Fastify server setup
│   └── routes/
│       ├── v1.ts           # Main v1 API routes
│       └── health.ts       # Health check endpoints
└── application/
    ├── articles/           # Article CRUD operations
    ├── bias/               # Political bias scoring
    ├── factCheck/          # Fact-checking API
    └── writingAssistant/   # AI writing assistance
```

### Core API Endpoints

#### Article Operations

| Endpoint                                | Handler                    | Description            |
| --------------------------------------- | -------------------------- | ---------------------- |
| `GET /v1/articles`                      | `listArticles.ts`          | Paginated article list |
| `GET /v1/articles/:id`                  | `getArticleById.ts`        | Single article fetch   |
| `GET /v1/articles/search`               | `searchArticles.ts`        | Full-text search       |
| `GET /v1/articles/:id/related`          | `getRelatedArticles.ts`    | Related content        |
| `POST /v1/articles/search/personalized` | `searchPersonalization.ts` | Personalized ranking   |

#### NLP Services

| Endpoint                   | Handler                   | NLP Package Used                        |
| -------------------------- | ------------------------- | --------------------------------------- |
| `POST /v1/bias/score`      | `politicalBiasScoring.ts` | `@yaa/nlp-topic-classification-service` |
| `POST /v1/factcheck/claim` | `factCheckClaim.ts`       | `@yaa/nlp-evidence-retrieval-service`   |
| `POST /v1/writing/assist`  | `assist.ts`               | `@yaa/ai-agents` (LLMClient)            |

### External Service Integration

The system integrates with multiple external APIs:

#### HeyGen Video API

**Location:** `packages/video/heygen-service/src/client.ts`

```typescript
class HeyGenClient {
  private baseUrl: string;
  private apiKey: string;

  async generateVideo(config: VideoConfig): Promise<VideoResult> {
    const response = await fetch(`${this.baseUrl}/v2/video/generate`, {
      method: 'POST',
      headers: {
        'X-Api-Key': this.apiKey,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify(config),
      signal: AbortSignal.timeout(30000),
    });
    return this.handleResponse(response);
  }
}
```

#### External Data Sources

| Service                | Location                                                      | Data Type          |
| ---------------------- | ------------------------------------------------------------- | ------------------ |
| Ghana Stock Exchange   | `packages/ai/market-summary-generator/src/gse.ts`             | Stock prices       |
| Forex Rates            | `packages/ai/market-summary-generator/src/forex.ts`           | Currency exchange  |
| Weather (GMET)         | `packages/ai/weather-report-generator/src/gmet.ts`            | Weather forecasts  |
| Event Calendar (Nager) | `packages/ai/event-calendar-updater/src/nager.ts`             | Public holidays    |
| Fuel Prices (NPA)      | `packages/ai/fuel-price-update-generator/src/npa.ts`          | Fuel pricing       |
| Google Fact-Check      | `packages/ai/fact-check-integrations/src/google-factcheck.ts` | Claim verification |
| ClaimBuster            | `packages/ai/fact-check-integrations/src/claimbuster.ts`      | Claim detection    |

### HTTP Client Patterns

All external calls use consistent patterns:

```typescript
// Standard fetch with timeout
async function fetchWithTimeout(
  url: string,
  options: RequestInit,
  timeoutMs: number
) {
  const controller = new AbortController();
  const timeoutId = setTimeout(() => controller.abort(), timeoutMs);

  try {
    const response = await fetch(url, {
      ...options,
      signal: controller.signal,
    });
    return response;
  } finally {
    clearTimeout(timeoutId);
  }
}
```

### Redis Connection Factory

**Location:** `packages/api/src/infrastructure/redis/createRedis.ts`

```typescript
function createRedis(options: RedisOptions): Redis {
  return new Redis({
    ...options,
    retryStrategy: (times) => {
      if (times > 3) return null; // Stop retrying
      return Math.min(times * 1000, 3000); // Max 3s delay
    },
    maxRetriesPerRequest: 3,
  });
}
```

---

## Event Emitter Patterns

### Agent Event System

**Location:** `packages/ai/agents/core/base-agent.ts`

All AI agents extend Node.js EventEmitter for lifecycle events:

```typescript
import { EventEmitter } from 'events';

abstract class BaseAgent extends EventEmitter {
  protected messageBus: MessageBus;
  protected stateManager: StateManager;
  protected metricsCollector: MetricsCollector;
  protected llmClient: LLMClient;

  constructor(config: AgentConfig) {
    super();
    this.setupEventHandlers();
  }

  private setupEventHandlers(): void {
    this.on('taskStarted', this.handleTaskStarted.bind(this));
    this.on('taskCompleted', this.handleTaskCompleted.bind(this));
    this.on('taskFailed', this.handleTaskFailed.bind(this));
    this.on('stateUpdated', this.handleStateUpdated.bind(this));
  }
}
```

### Agent Event Types

| Event              | Trigger                    | Payload                          |
| ------------------ | -------------------------- | -------------------------------- |
| `taskStarted`      | Task begins processing     | `{ taskId, agentId, timestamp }` |
| `taskCompleted`    | Task finishes successfully | `{ taskId, result, duration }`   |
| `taskFailed`       | Task encounters error      | `{ taskId, error, retryCount }`  |
| `stateUpdated`     | Agent state changes        | `{ previousState, newState }`    |
| `messageDelivered` | Message sent successfully  | `{ messageId, recipient }`       |
| `messageDiscarded` | Message TTL expired        | `{ messageId, reason }`          |

### Orchestrator Pattern

**Location:** `packages/ai/agents/orchestration/orchestrator.ts`

```typescript
class Orchestrator extends EventEmitter {
  private agents: Map<string, BaseAgent>;
  private taskQueue: TaskQueue;

  async routeTask(task: Task): Promise<void> {
    const agent = this.selectAgent(task);
    this.emit('taskRouted', { taskId: task.id, agentId: agent.id });
    await agent.process(task);
  }

  async broadcastToAgents(message: Message): Promise<void> {
    for (const agent of this.agents.values()) {
      await this.messageBus.send({ ...message, to: agent.id });
    }
    this.emit('broadcastSent', { messageType: message.type });
  }
}
```

---

## Database Communication

### PostgreSQL Access Pattern

All database access uses the `pg` (node-postgres) library with connection
pooling:

```typescript
import { Pool } from 'pg';

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
  application_name: 'yaa-processing-queue-worker',
  max: 20,
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 2000,
});
```

### Shared Database Schema

The following tables are accessed by multiple packages:

#### Content Tables

| Table        | Primary Writers          | Readers                  |
| ------------ | ------------------------ | ------------------------ |
| `sources`    | Ingestion workers        | API, NLP services        |
| `articles`   | AI generators, ingestion | API, video/audio workers |
| `authors`    | Ingestion workers        | API                      |
| `categories` | Ingestion workers        | API, classification      |
| `topics`     | Topic classifier         | API, personalization     |
| `tags`       | Keyword extractor        | API, search              |

#### Relationship Tables

| Table                  | Purpose                     |
| ---------------------- | --------------------------- |
| `article_authors`      | Article-to-author mapping   |
| `article_attributions` | Source attribution tracking |
| `article_categories`   | Article categorization      |
| `article_topics`       | Topic assignment            |
| `article_tags`         | Tag attachment              |

#### Ingestion Tables

| Table                     | Purpose                        |
| ------------------------- | ------------------------------ |
| `feed_items`              | Raw ingested content           |
| `normalized_feed_items`   | Processed content              |
| `content_fingerprints`    | Simhash fingerprints for dedup |
| `content_dedupe_mappings` | Duplicate item relationships   |

### Transaction Patterns

**Location:** `packages/ai/market-summary-generator/src/db.ts`

Complex operations use explicit transaction management:

```typescript
async function withTransaction<T>(
  pool: Pool,
  fn: (client: PoolClient) => Promise<T>
): Promise<T> {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    const result = await fn(client);
    await client.query('COMMIT');
    return result;
  } catch (error) {
    await client.query('ROLLBACK');
    throw error;
  } finally {
    client.release();
  }
}
```

### Article Publishing Transaction

The market summary generator demonstrates a complex multi-step transaction:

```typescript
async function publishMarketSummary(client: PoolClient, data: MarketSummary) {
  // Step 1: Upsert source
  const sourceId = await upsertSource(client, data.source);

  // Step 2: Upsert categories
  const categoryIds = await upsertCategories(client, data.categories);

  // Step 3: Upsert topics
  const topicIds = await upsertTopics(client, data.topics);

  // Step 4: Upsert author
  const authorId = await upsertAuthor(client, data.author);

  // Step 5: Upsert tags
  const tagIds = await upsertTags(client, data.tags);

  // Step 6: Upsert article (ON CONFLICT DO UPDATE)
  const { articleId, isNew } = await upsertArticle(client, {
    slug: data.slug,
    title: data.title,
    content: data.content,
    sourceId,
    publishedAt: data.publishedAt,
  });

  // Step 7: Attach author
  await attachAuthor(client, articleId, authorId);

  // Step 8: Upsert attributions
  await upsertAttributions(client, articleId, data.attributions);

  // Step 9-11: Attach categories, topics, tags
  await attachCategories(client, articleId, categoryIds);
  await attachTopics(client, articleId, topicIds);
  await attachTags(client, articleId, tagIds);

  return { articleId, isNew };
}
```

### Deduplication Pipeline

**Location:** `packages/ingestion/processing-queue/src/pipeline.ts`

```typescript
// Simhash-based content fingerprinting
async function deduplicateContent(
  client: PoolClient,
  content: string,
  itemId: number
) {
  // Generate Simhash64 fingerprint
  const fingerprint = simhash64(content);

  // Find similar fingerprints in same bucket
  const bucket = fingerprint >> 48; // Top 16 bits as bucket
  const candidates = await findCandidatesInBucket(client, bucket);

  // Check Hamming distance for near-duplicates
  for (const candidate of candidates) {
    const distance = hammingDistance(fingerprint, candidate.fingerprint);
    if (distance <= SIMILARITY_THRESHOLD) {
      await recordDuplicate(client, itemId, candidate.itemId, distance);
      return { isDuplicate: true, originalId: candidate.itemId };
    }
  }

  // Store new fingerprint
  await storeFingerprint(client, itemId, fingerprint, bucket);
  return { isDuplicate: false };
}
```

---

## Inter-Package Dependencies

### Dependency Graph Overview

The system has 133 unique `@yaa/*` package imports. Key dependency clusters:

#### Core Shared Packages

```
@yaa/models           ← Shared TypeScript types and enums
@yaa/ai-agents        ← Agent framework, LLM client, logging
@yaa/nlp-service-kit  ← Base NLP utilities
```

#### NLP Service Dependencies

```
                    ┌─────────────────────────────┐
                    │    @yaa/nlp-service-kit     │
                    └─────────────┬───────────────┘
                                  │
        ┌─────────────────────────┼─────────────────────────┐
        │                         │                         │
        ▼                         ▼                         ▼
┌───────────────┐    ┌────────────────────┐    ┌────────────────────┐
│ @yaa/nlp-     │    │ @yaa/nlp-topic-    │    │ @yaa/nlp-keyword-  │
│ embedding-    │    │ classification-    │    │ extraction-        │
│ service       │    │ service            │    │ service            │
└───────────────┘    └────────────────────┘    └────────────────────┘
```

#### Ingestion Dependencies

```
@yaa/ingestion-rss-collector
        │
        ▼
@yaa/ingestion-scraper
        │
        ▼
@yaa/ingestion-content-normalization
        │
        ├──▶ @yaa/ingestion-language-detection
        ├──▶ @yaa/ingestion-timestamp-normalization
        ├──▶ @yaa/ingestion-media-extraction
        └──▶ @yaa/ingestion-semantic-dedup
```

#### Video Pipeline Dependencies

```
@yaa/video-generation-queue
        │
        ├──▶ @yaa/video-heygen-service
        ├──▶ @yaa/video-avatar-profiles
        ├──▶ @yaa/video-captioning-service
        ├──▶ @yaa/video-thumbnail-generator
        └──▶ @yaa/video-vertical-converter
                    │
                    ├──▶ @yaa/video-tiktok-video-producer
                    ├──▶ @yaa/video-youtube-upload-service
                    └──▶ @yaa/video-youtube-shorts-upload-service
```

#### Social Media Dependencies

```
@yaa/social-unified-scheduler
        │
        ├──▶ @yaa/social-instagram-scheduler
        ├──▶ @yaa/social-tiktok-upload-service
        ├──▶ @yaa/social-youtube-channel-setup
        └──▶ @yaa/social-whatsapp-business-service
```

---

## Communication Flow Diagrams

### End-to-End Content Pipeline

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                        CONTENT INGESTION FLOW                               │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌──────────────┐     ┌──────────────┐     ┌──────────────┐               │
│  │ RSS Feeds    │     │ Sitemaps     │     │ Social APIs  │               │
│  │ (300+ feeds) │     │              │     │ (X/Twitter)  │               │
│  └──────┬───────┘     └──────┬───────┘     └──────┬───────┘               │
│         │                    │                    │                        │
│         └────────────────────┼────────────────────┘                        │
│                              ▼                                             │
│                    ┌──────────────────┐                                    │
│                    │   feed_items     │                                    │
│                    │   (PostgreSQL)   │                                    │
│                    └────────┬─────────┘                                    │
│                             │                                              │
│                             ▼                                              │
│                    ┌──────────────────┐                                    │
│                    │  Processing      │                                    │
│                    │  Queue Producer  │                                    │
│                    └────────┬─────────┘                                    │
│                             │                                              │
│                             ▼                                              │
│                    ┌──────────────────┐                                    │
│                    │ yaa:content-     │                                    │
│                    │ processing       │                                    │
│                    │ (BullMQ/Redis)   │                                    │
│                    └────────┬─────────┘                                    │
│                             │                                              │
│                             ▼                                              │
│  ┌──────────────────────────────────────────────────────────────────┐     │
│  │                    Processing Worker                              │     │
│  │                                                                   │     │
│  │  ┌─────────┐   ┌─────────┐   ┌─────────┐   ┌─────────┐          │     │
│  │  │Normalize│──▶│ Dedupe  │──▶│  Lang   │──▶│ Extract │          │     │
│  │  │ Content │   │(Simhash)│   │ Detect  │   │  Media  │          │     │
│  │  └─────────┘   └─────────┘   └─────────┘   └─────────┘          │     │
│  │                                                                   │     │
│  └──────────────────────────────────────────────────────────────────┘     │
│                             │                                              │
│                             ▼                                              │
│                    ┌──────────────────┐                                    │
│                    │normalized_feed_  │                                    │
│                    │items (PostgreSQL)│                                    │
│                    └──────────────────┘                                    │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘
```

### Media Generation Pipeline

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                      MEDIA GENERATION FLOW                                  │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│                    ┌──────────────────┐                                    │
│                    │    articles      │                                    │
│                    │   (PostgreSQL)   │                                    │
│                    └────────┬─────────┘                                    │
│                             │                                              │
│              ┌──────────────┴──────────────┐                               │
│              │                             │                               │
│              ▼                             ▼                               │
│    ┌──────────────────┐          ┌──────────────────┐                     │
│    │  Audio Queue     │          │  Video Queue     │                     │
│    │  Producer        │          │  Producer        │                     │
│    └────────┬─────────┘          └────────┬─────────┘                     │
│             │                             │                               │
│             ▼                             ▼                               │
│    ┌──────────────────┐          ┌──────────────────┐                     │
│    │ yaa:audio-       │          │ yaa:video-       │                     │
│    │ generation       │          │ generation       │                     │
│    │ (BullMQ/Redis)   │          │ (BullMQ/Redis)   │                     │
│    └────────┬─────────┘          └────────┬─────────┘                     │
│             │                             │                               │
│             ▼                             ▼                               │
│    ┌──────────────────┐          ┌──────────────────┐                     │
│    │  Audio Worker    │          │  Video Worker    │                     │
│    │                  │          │                  │                     │
│    │  ┌────────────┐  │          │  ┌────────────┐  │                     │
│    │  │ ElevenLabs │  │          │  │   HeyGen   │  │                     │
│    │  │    TTS     │  │          │  │  Avatar    │  │                     │
│    │  └────────────┘  │          │  └────────────┘  │                     │
│    │  ┌────────────┐  │          │  ┌────────────┐  │                     │
│    │  │ Ghana NLP  │  │          │  │  Caption   │  │                     │
│    │  │    TTS     │  │          │  │  Service   │  │                     │
│    │  └────────────┘  │          │  └────────────┘  │                     │
│    └────────┬─────────┘          └────────┬─────────┘                     │
│             │                             │                               │
│             └──────────────┬──────────────┘                               │
│                            ▼                                              │
│                   ┌──────────────────┐                                    │
│                   │  S3 / Database   │                                    │
│                   │  (Media Storage) │                                    │
│                   └──────────────────┘                                    │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘
```

### Agent Communication Flow

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                      AGENT COMMUNICATION FLOW                               │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│                          ┌─────────────────┐                               │
│                          │   Orchestrator  │                               │
│                          │     Agent       │                               │
│                          └────────┬────────┘                               │
│                                   │                                        │
│                                   │ broadcast()                            │
│                                   ▼                                        │
│                          ┌─────────────────┐                               │
│                          │   MessageBus    │                               │
│                          │ (Redis Pub/Sub) │                               │
│                          └────────┬────────┘                               │
│                                   │                                        │
│            ┌──────────────────────┼──────────────────────┐                 │
│            │                      │                      │                 │
│            ▼                      ▼                      ▼                 │
│   ┌─────────────────┐   ┌─────────────────┐   ┌─────────────────┐         │
│   │  Content        │   │   Fact-Check    │   │   Sentiment     │         │
│   │  Analyzer       │   │   Agent         │   │   Analyzer      │         │
│   │  Agent          │   │                 │   │   Agent         │         │
│   └────────┬────────┘   └────────┬────────┘   └────────┬────────┘         │
│            │                      │                      │                 │
│            │ send()               │ send()               │ send()          │
│            │                      │                      │                 │
│            └──────────────────────┼──────────────────────┘                 │
│                                   │                                        │
│                                   ▼                                        │
│                          ┌─────────────────┐                               │
│                          │   MessageBus    │                               │
│                          │  (direct msgs)  │                               │
│                          └────────┬────────┘                               │
│                                   │                                        │
│                                   ▼                                        │
│                          ┌─────────────────┐                               │
│                          │   Orchestrator  │                               │
│                          │   (aggregates)  │                               │
│                          └─────────────────┘                               │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘
```

---

## Integration Points Summary

### Cross-Cutting Communication Matrix

| From \ To             | Database           | BullMQ                       | Redis PubSub | REST API | External API          |
| --------------------- | ------------------ | ---------------------------- | ------------ | -------- | --------------------- |
| **Ingestion**         | Write (feed_items) | Enqueue (content-processing) | -            | -        | RSS/Sitemap           |
| **Processing Worker** | Read/Write         | Consume                      | -            | -        | -                     |
| **Audio Worker**      | Read/Write         | Consume                      | -            | -        | ElevenLabs, Ghana NLP |
| **Video Worker**      | Read/Write         | Consume                      | -            | -        | HeyGen                |
| **AI Agents**         | Read               | -                            | Pub/Sub      | -        | LLM APIs              |
| **API Server**        | Read/Write         | -                            | Cache        | Serve    | Various               |
| **Social Scheduler**  | Read               | Enqueue/Consume              | -            | -        | Social Platforms      |

### Critical Integration Points

1. **Database as Source of Truth**
   - All packages share PostgreSQL database
   - Consistent schema across all workers
   - Transaction isolation for complex operations

2. **BullMQ as Task Coordinator**
   - Reliable job delivery with retries
   - Priority-based scheduling
   - Automatic cleanup and monitoring

3. **Redis as Real-Time Bus**
   - Agent-to-agent messaging
   - Broadcast notifications
   - Message persistence for reliability

4. **REST APIs for External Integration**
   - Fastify server for client requests
   - Native fetch for external services
   - Consistent timeout and retry handling

---

## Recommendations for Oshun Integration

### Migration Considerations

1. **Queue Standardization**
   - Align BullMQ queue naming with Oshun conventions
   - Consolidate Redis connection management
   - Implement shared queue monitoring

2. **Database Integration**
   - Map Yaa tables to Oshun schema patterns
   - Consider Prisma migration for type safety
   - Plan for cross-domain query patterns

3. **API Gateway**
   - Route Yaa endpoints through Oshun API gateway
   - Implement consistent authentication
   - Add rate limiting and caching

4. **Agent Framework**
   - Evaluate integration with Oshun agent patterns
   - Standardize MessageBus interface
   - Consolidate LLM client configurations

---

_Last Updated: January 2026_

_Part of the Veritas domain integration documentation_
