Eve library

Yaa Inter-Package Communication Patterns

Guide

Revision 1

On this page

This document provides a comprehensive analysis of communication mechanisms used across the Yaa codebase, including message queues, pub/sub patterns, REST APIs, event systems, and database access patterns.


Table of Contents#

  1. Overview
  2. BullMQ Job Queue Architecture
  3. Redis Pub/Sub Message Bus
  4. REST API Communication
  5. Event Emitter Patterns
  6. Database Communication
  7. Inter-Package Dependencies
  8. Communication Flow Diagrams
  9. Integration Points Summary

Overview#

The Yaa codebase employs a distributed, event-driven architecture with multiple communication mechanisms:

Mechanism Technology Primary Use Case
Job Queues BullMQ + Redis Async task processing pipelines
Pub/Sub IORedis Inter-agent real-time messaging
REST APIs Fastify + fetch External service integration
Event Emitters Node.js events Intra-process event handling
Direct Database pg (node-postgres) Shared state management
Package Imports TypeScript Shared utilities and models

BullMQ Job Queue Architecture#

Queue Registry#

The system uses three primary BullMQ queues for content processing pipelines:

1. Content Processing Queue#

Queue Name: yaa:content-processing

Location: packages/ingestion/processing-queue/

File Purpose
src/queue.ts Queue definition and configuration
src/producer.ts Job enqueueing logic
src/worker.ts Job processing implementation
src/producer-db.ts Database queries for producer
src/worker-db.ts Database queries for worker
src/pipeline.ts Processing pipeline stages

Job Type: process_feed_item

Job Data Structure:

typescript
interface ProcessFeedItemJob {
  feedItemId: number;
  normalizedFeedItemId: number | null;
  didNormalize: boolean;
  didDedupe: boolean;
  didLanguageDetect: boolean;
  didTimestampNormalize: boolean;
  didMediaExtract: boolean;
  didAttribution: boolean;
}

Queue Configuration:

typescript
{
  defaultJobOptions: {
    attempts: 8,
    backoff: {
      type: 'exponential',
      delay: 1000  // 1 second base delay
    },
    removeOnComplete: 1000,
    removeOnFail: 1000
  }
}

Pipeline Stages:

  1. Content normalization (HTML parsing, text extraction)
  2. Semantic deduplication (Simhash64 fingerprinting)
  3. Language detection (multi-language support)
  4. Timestamp normalization (date parsing)
  5. Media extraction (images, videos, links)
  6. Source attribution

2. Audio Generation Queue#

Queue Name: yaa:audio-generation

Location: packages/audio/audio-generation-queue/

File Purpose
src/queue.ts Queue definition
src/producer.ts Article audio job creation
src/worker.ts TTS generation processing

Job Type: generate_article_audio

Job Data Structure:

typescript
interface GenerateAudioJob {
  articleId: number;
  voiceProfileId: number;
  voiceProfileSlug: string;
  priority: number; // Computed from publish timestamp
}

Queue Configuration:

typescript
{
  defaultJobOptions: {
    attempts: 6,
    backoff: {
      type: 'exponential',
      delay: 2000  // 2 second base delay
    },
    removeOnComplete: 1000,
    removeOnFail: 1000
  }
}

TTS Providers:

  • ElevenLabs (@yaa/audio-elevenlabs-service) - English TTS
  • Ghana NLP (@yaa/ghana-nlp) - Twi, Ewe, Ga, Hausa, Dagbani TTS

3. Video Generation Queue#

Queue Name: yaa:video-generation

Location: packages/video/video-generation-queue/

File Purpose
src/queue.ts Queue definition
src/producer.ts Video job creation with anchor routing
src/worker.ts Video synthesis and packaging

Job Types:

Job Type Description
generate_article_video Full video synthesis via HeyGen
package_article_video_hls HLS packaging for streaming

Job Data Structure:

typescript
interface GenerateVideoJob {
  articleId: number;
  anchorProfileId: number;
  anchorMode: 'fixed' | 'auto';
  priority: number;
}

Anchor Routing Modes:

  • Fixed Mode: All articles use a single specified anchor profile
  • Auto Mode: Load-balances articles across active anchor profiles
  • Fallback: Uses fallback profile when no active anchors available

Queue Configuration:

typescript
{
  defaultJobOptions: {
    attempts: 6,
    backoff: {
      type: 'exponential',
      delay: 2000
    },
    removeOnComplete: 1000,
    removeOnFail: 1000
  }
}

4. Social Media Unified Scheduler#

Location: packages/social/unified-scheduler/

Job Type: publishUnified

Job Data Structure:

typescript
interface PublishUnifiedJob {
  postId: string;
  post: UnifiedScheduledPost;
}

Features:

  • Multi-platform orchestration (Instagram, TikTok, YouTube, X/Twitter)
  • Configurable concurrency per platform
  • Sequential platform publishing with delays
  • Retry handling per platform

Producer-Consumer Flow#

text
┌─────────────────────────────────────────────────────────────────────┐
│                     Content Processing Pipeline                      │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  RSS/Sitemap     ┌──────────────┐      ┌─────────────────┐          │
│  Collectors  ──▶ │  feed_items  │ ───▶ │ Processing Queue│          │
│                  │  (database)  │      │    Producer     │          │
│                  └──────────────┘      └────────┬────────┘          │
│                                                 │                    │
│                                                 ▼                    │
│                                        ┌──────────────┐             │
│                                        │ yaa:content- │             │
│                                        │  processing  │             │
│                                        │   (Redis)    │             │
│                                        └──────┬───────┘             │
│                                               │                      │
│                                               ▼                      │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │                    Processing Worker                         │   │
│  │  ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌──────────┐ │   │
│  │  │ Normalize  │▶│  Dedupe    │▶│ Lang Detect│▶│ Extract  │ │   │
│  │  │  Content   │ │ (Simhash)  │ │            │ │  Media   │ │   │
│  │  └────────────┘ └────────────┘ └────────────┘ └──────────┘ │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                               │                      │
│                                               ▼                      │
│                                   ┌───────────────────┐             │
│                                   │ normalized_feed_  │             │
│                                   │  items (database) │             │
│                                   └───────────────────┘             │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

Redis Pub/Sub Message Bus#

Agent Communication System#

Location: packages/ai/agents/communication/message-bus.ts

The MessageBus provides real-time inter-agent communication using Redis Pub/Sub.

Architecture#

typescript
class MessageBus {
  private redis: Redis; // Main Redis client
  private subscriber: Redis; // Dedicated subscriber client
  private queues: Map<string, Message[]>; // In-memory message queues
  private subscriptions: Map<string, Set<(msg: Message) => void>>;
}

Channel Naming Convention#

Channel Pattern Purpose
yaa:messages:${agentId} Direct agent-to-agent messages
yaa:messages:broadcast System-wide broadcasts
yaa:queue:${agentId} Persistent message queue (Redis List)

Message Structure#

typescript
interface Message<T = unknown> {
  type: string;
  from: string; // Sender agent ID
  to: string | null; // Recipient (null for broadcast)
  payload: T;
  priority: 'low' | 'normal' | 'high' | 'critical';
  ttlMs: number; // Time-to-live (default: 5 minutes)
  timestamp: number;
  correlationId?: string; // For request-response tracking
}

Message Bus Operations#

Sending Direct Messages#

typescript
// Send targeted message to specific agent
await messageBus.send({
  type: 'TASK_ASSIGNMENT',
  from: 'orchestrator',
  to: 'content-analyzer-1',
  payload: { articleId: 12345 },
  priority: 'high',
  ttlMs: 300000, // 5 minutes
});

Broadcasting#

typescript
// Broadcast to all subscribed agents
await messageBus.broadcast({
  type: 'SYSTEM_ALERT',
  from: 'coordinator',
  to: null,
  payload: { message: 'Scaling up workers' },
  priority: 'critical',
});

Subscribing#

typescript
// Subscribe to messages
messageBus.subscribe('my-agent-id', (message) => {
  console.log(`Received: ${message.type} from ${message.from}`);
});

Message Persistence#

Messages are persisted to Redis Lists for reliability:

typescript
// Queue key pattern
const queueKey = `yaa:queue:${agentId}`;

// Operations
LPUSH queueKey message      // Enqueue
LRANGE queueKey 0 -1        // Read all
LTRIM queueKey 0 maxSize    // Limit queue size

Retry Configuration#

typescript
const DEFAULT_OPTIONS = {
  maxQueueSize: 1000,
  maxRetries: 3,
  retryBackoffMs: 1000, // Exponential backoff base
  messageTtlMs: 300000, // 5 minutes default TTL
};

Statistics API#

typescript
interface MessageBusStats {
  subscriptions: number; // Active subscriptions
  broadcasts: number; // Broadcasts sent
  queued: number; // Messages in queues
  pending: number; // Pending delivery
  historySize: number; // Messages in history
}

REST API Communication#

API Server Architecture#

Location: packages/api/

Framework: Fastify

Route Structure#

text
packages/api/src/
├── interfaces/http/
│   ├── server.ts           # Fastify server setup
│   └── routes/
│       ├── v1.ts           # Main v1 API routes
│       └── health.ts       # Health check endpoints
└── application/
    ├── articles/           # Article CRUD operations
    ├── bias/               # Political bias scoring
    ├── factCheck/          # Fact-checking API
    └── writingAssistant/   # AI writing assistance

Core API Endpoints#

Article Operations#

Endpoint Handler Description
GET /v1/articles listArticles.ts Paginated article list
GET /v1/articles/:id getArticleById.ts Single article fetch
GET /v1/articles/search searchArticles.ts Full-text search
GET /v1/articles/:id/related getRelatedArticles.ts Related content
POST /v1/articles/search/personalized searchPersonalization.ts Personalized ranking

NLP Services#

Endpoint Handler NLP Package Used
POST /v1/bias/score politicalBiasScoring.ts @yaa/nlp-topic-classification-service
POST /v1/factcheck/claim factCheckClaim.ts @yaa/nlp-evidence-retrieval-service
POST /v1/writing/assist assist.ts @yaa/ai-agents (LLMClient)

External Service Integration#

The system integrates with multiple external APIs:

HeyGen Video API#

Location: packages/video/heygen-service/src/client.ts

typescript
class HeyGenClient {
  private baseUrl: string;
  private apiKey: string;

  async generateVideo(config: VideoConfig): Promise<VideoResult> {
    const response = await fetch(`${this.baseUrl}/v2/video/generate`, {
      method: 'POST',
      headers: {
        'X-Api-Key': this.apiKey,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify(config),
      signal: AbortSignal.timeout(30000),
    });
    return this.handleResponse(response);
  }
}

External Data Sources#

Service Location Data Type
Ghana Stock Exchange packages/ai/market-summary-generator/src/gse.ts Stock prices
Forex Rates packages/ai/market-summary-generator/src/forex.ts Currency exchange
Weather (GMET) packages/ai/weather-report-generator/src/gmet.ts Weather forecasts
Event Calendar (Nager) packages/ai/event-calendar-updater/src/nager.ts Public holidays
Fuel Prices (NPA) packages/ai/fuel-price-update-generator/src/npa.ts Fuel pricing
Google Fact-Check packages/ai/fact-check-integrations/src/google-factcheck.ts Claim verification
ClaimBuster packages/ai/fact-check-integrations/src/claimbuster.ts Claim detection

HTTP Client Patterns#

All external calls use consistent patterns:

typescript
// Standard fetch with timeout
async function fetchWithTimeout(
  url: string,
  options: RequestInit,
  timeoutMs: number
) {
  const controller = new AbortController();
  const timeoutId = setTimeout(() => controller.abort(), timeoutMs);

  try {
    const response = await fetch(url, {
      ...options,
      signal: controller.signal,
    });
    return response;
  } finally {
    clearTimeout(timeoutId);
  }
}

Redis Connection Factory#

Location: packages/api/src/infrastructure/redis/createRedis.ts

typescript
function createRedis(options: RedisOptions): Redis {
  return new Redis({
    ...options,
    retryStrategy: (times) => {
      if (times > 3) return null; // Stop retrying
      return Math.min(times * 1000, 3000); // Max 3s delay
    },
    maxRetriesPerRequest: 3,
  });
}

Event Emitter Patterns#

Agent Event System#

Location: packages/ai/agents/core/base-agent.ts

All AI agents extend Node.js EventEmitter for lifecycle events:

typescript
import { EventEmitter } from 'events';

abstract class BaseAgent extends EventEmitter {
  protected messageBus: MessageBus;
  protected stateManager: StateManager;
  protected metricsCollector: MetricsCollector;
  protected llmClient: LLMClient;

  constructor(config: AgentConfig) {
    super();
    this.setupEventHandlers();
  }

  private setupEventHandlers(): void {
    this.on('taskStarted', this.handleTaskStarted.bind(this));
    this.on('taskCompleted', this.handleTaskCompleted.bind(this));
    this.on('taskFailed', this.handleTaskFailed.bind(this));
    this.on('stateUpdated', this.handleStateUpdated.bind(this));
  }
}

Agent Event Types#

Event Trigger Payload
taskStarted Task begins processing { taskId, agentId, timestamp }
taskCompleted Task finishes successfully { taskId, result, duration }
taskFailed Task encounters error { taskId, error, retryCount }
stateUpdated Agent state changes { previousState, newState }
messageDelivered Message sent successfully { messageId, recipient }
messageDiscarded Message TTL expired { messageId, reason }

Orchestrator Pattern#

Location: packages/ai/agents/orchestration/orchestrator.ts

typescript
class Orchestrator extends EventEmitter {
  private agents: Map<string, BaseAgent>;
  private taskQueue: TaskQueue;

  async routeTask(task: Task): Promise<void> {
    const agent = this.selectAgent(task);
    this.emit('taskRouted', { taskId: task.id, agentId: agent.id });
    await agent.process(task);
  }

  async broadcastToAgents(message: Message): Promise<void> {
    for (const agent of this.agents.values()) {
      await this.messageBus.send({ ...message, to: agent.id });
    }
    this.emit('broadcastSent', { messageType: message.type });
  }
}

Database Communication#

PostgreSQL Access Pattern#

All database access uses the pg (node-postgres) library with connection pooling:

typescript
import { Pool } from 'pg';

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
  application_name: 'yaa-processing-queue-worker',
  max: 20,
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 2000,
});

Shared Database Schema#

The following tables are accessed by multiple packages:

Content Tables#

Table Primary Writers Readers
sources Ingestion workers API, NLP services
articles AI generators, ingestion API, video/audio workers
authors Ingestion workers API
categories Ingestion workers API, classification
topics Topic classifier API, personalization
tags Keyword extractor API, search

Relationship Tables#

Table Purpose
article_authors Article-to-author mapping
article_attributions Source attribution tracking
article_categories Article categorization
article_topics Topic assignment
article_tags Tag attachment

Ingestion Tables#

Table Purpose
feed_items Raw ingested content
normalized_feed_items Processed content
content_fingerprints Simhash fingerprints for dedup
content_dedupe_mappings Duplicate item relationships

Transaction Patterns#

Location: packages/ai/market-summary-generator/src/db.ts

Complex operations use explicit transaction management:

typescript
async function withTransaction<T>(
  pool: Pool,
  fn: (client: PoolClient) => Promise<T>
): Promise<T> {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    const result = await fn(client);
    await client.query('COMMIT');
    return result;
  } catch (error) {
    await client.query('ROLLBACK');
    throw error;
  } finally {
    client.release();
  }
}

Article Publishing Transaction#

The market summary generator demonstrates a complex multi-step transaction:

typescript
async function publishMarketSummary(client: PoolClient, data: MarketSummary) {
  // Step 1: Upsert source
  const sourceId = await upsertSource(client, data.source);

  // Step 2: Upsert categories
  const categoryIds = await upsertCategories(client, data.categories);

  // Step 3: Upsert topics
  const topicIds = await upsertTopics(client, data.topics);

  // Step 4: Upsert author
  const authorId = await upsertAuthor(client, data.author);

  // Step 5: Upsert tags
  const tagIds = await upsertTags(client, data.tags);

  // Step 6: Upsert article (ON CONFLICT DO UPDATE)
  const { articleId, isNew } = await upsertArticle(client, {
    slug: data.slug,
    title: data.title,
    content: data.content,
    sourceId,
    publishedAt: data.publishedAt,
  });

  // Step 7: Attach author
  await attachAuthor(client, articleId, authorId);

  // Step 8: Upsert attributions
  await upsertAttributions(client, articleId, data.attributions);

  // Step 9-11: Attach categories, topics, tags
  await attachCategories(client, articleId, categoryIds);
  await attachTopics(client, articleId, topicIds);
  await attachTags(client, articleId, tagIds);

  return { articleId, isNew };
}

Deduplication Pipeline#

Location: packages/ingestion/processing-queue/src/pipeline.ts

typescript
// Simhash-based content fingerprinting
async function deduplicateContent(
  client: PoolClient,
  content: string,
  itemId: number
) {
  // Generate Simhash64 fingerprint
  const fingerprint = simhash64(content);

  // Find similar fingerprints in same bucket
  const bucket = fingerprint >> 48; // Top 16 bits as bucket
  const candidates = await findCandidatesInBucket(client, bucket);

  // Check Hamming distance for near-duplicates
  for (const candidate of candidates) {
    const distance = hammingDistance(fingerprint, candidate.fingerprint);
    if (distance <= SIMILARITY_THRESHOLD) {
      await recordDuplicate(client, itemId, candidate.itemId, distance);
      return { isDuplicate: true, originalId: candidate.itemId };
    }
  }

  // Store new fingerprint
  await storeFingerprint(client, itemId, fingerprint, bucket);
  return { isDuplicate: false };
}

Inter-Package Dependencies#

Dependency Graph Overview#

The system has 133 unique @yaa/* package imports. Key dependency clusters:

Core Shared Packages#

text
@yaa/models           ← Shared TypeScript types and enums
@yaa/ai-agents        ← Agent framework, LLM client, logging
@yaa/nlp-service-kit  ← Base NLP utilities

NLP Service Dependencies#

text
                    ┌─────────────────────────────┐
                    │    @yaa/nlp-service-kit     │
                    └─────────────┬───────────────┘
                                  │
        ┌─────────────────────────┼─────────────────────────┐
        │                         │                         │
        ▼                         ▼                         ▼
┌───────────────┐    ┌────────────────────┐    ┌────────────────────┐
│ @yaa/nlp-     │    │ @yaa/nlp-topic-    │    │ @yaa/nlp-keyword-  │
│ embedding-    │    │ classification-    │    │ extraction-        │
│ service       │    │ service            │    │ service            │
└───────────────┘    └────────────────────┘    └────────────────────┘

Ingestion Dependencies#

text
@yaa/ingestion-rss-collector
        │
        ▼
@yaa/ingestion-scraper
        │
        ▼
@yaa/ingestion-content-normalization
        │
        ├──▶ @yaa/ingestion-language-detection
        ├──▶ @yaa/ingestion-timestamp-normalization
        ├──▶ @yaa/ingestion-media-extraction
        └──▶ @yaa/ingestion-semantic-dedup

Video Pipeline Dependencies#

text
@yaa/video-generation-queue
        │
        ├──▶ @yaa/video-heygen-service
        ├──▶ @yaa/video-avatar-profiles
        ├──▶ @yaa/video-captioning-service
        ├──▶ @yaa/video-thumbnail-generator
        └──▶ @yaa/video-vertical-converter
                    │
                    ├──▶ @yaa/video-tiktok-video-producer
                    ├──▶ @yaa/video-youtube-upload-service
                    └──▶ @yaa/video-youtube-shorts-upload-service

Social Media Dependencies#

text
@yaa/social-unified-scheduler
        │
        ├──▶ @yaa/social-instagram-scheduler
        ├──▶ @yaa/social-tiktok-upload-service
        ├──▶ @yaa/social-youtube-channel-setup
        └──▶ @yaa/social-whatsapp-business-service

Communication Flow Diagrams#

End-to-End Content Pipeline#

text
┌─────────────────────────────────────────────────────────────────────────────┐
│                        CONTENT INGESTION FLOW                               │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌──────────────┐     ┌──────────────┐     ┌──────────────┐               │
│  │ RSS Feeds    │     │ Sitemaps     │     │ Social APIs  │               │
│  │ (300+ feeds) │     │              │     │ (X/Twitter)  │               │
│  └──────┬───────┘     └──────┬───────┘     └──────┬───────┘               │
│         │                    │                    │                        │
│         └────────────────────┼────────────────────┘                        │
│                              ▼                                             │
│                    ┌──────────────────┐                                    │
│                    │   feed_items     │                                    │
│                    │   (PostgreSQL)   │                                    │
│                    └────────┬─────────┘                                    │
│                             │                                              │
│                             ▼                                              │
│                    ┌──────────────────┐                                    │
│                    │  Processing      │                                    │
│                    │  Queue Producer  │                                    │
│                    └────────┬─────────┘                                    │
│                             │                                              │
│                             ▼                                              │
│                    ┌──────────────────┐                                    │
│                    │ yaa:content-     │                                    │
│                    │ processing       │                                    │
│                    │ (BullMQ/Redis)   │                                    │
│                    └────────┬─────────┘                                    │
│                             │                                              │
│                             ▼                                              │
│  ┌──────────────────────────────────────────────────────────────────┐     │
│  │                    Processing Worker                              │     │
│  │                                                                   │     │
│  │  ┌─────────┐   ┌─────────┐   ┌─────────┐   ┌─────────┐          │     │
│  │  │Normalize│──▶│ Dedupe  │──▶│  Lang   │──▶│ Extract │          │     │
│  │  │ Content │   │(Simhash)│   │ Detect  │   │  Media  │          │     │
│  │  └─────────┘   └─────────┘   └─────────┘   └─────────┘          │     │
│  │                                                                   │     │
│  └──────────────────────────────────────────────────────────────────┘     │
│                             │                                              │
│                             ▼                                              │
│                    ┌──────────────────┐                                    │
│                    │normalized_feed_  │                                    │
│                    │items (PostgreSQL)│                                    │
│                    └──────────────────┘                                    │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Media Generation Pipeline#

text
┌─────────────────────────────────────────────────────────────────────────────┐
│                      MEDIA GENERATION FLOW                                  │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│                    ┌──────────────────┐                                    │
│                    │    articles      │                                    │
│                    │   (PostgreSQL)   │                                    │
│                    └────────┬─────────┘                                    │
│                             │                                              │
│              ┌──────────────┴──────────────┐                               │
│              │                             │                               │
│              ▼                             ▼                               │
│    ┌──────────────────┐          ┌──────────────────┐                     │
│    │  Audio Queue     │          │  Video Queue     │                     │
│    │  Producer        │          │  Producer        │                     │
│    └────────┬─────────┘          └────────┬─────────┘                     │
│             │                             │                               │
│             ▼                             ▼                               │
│    ┌──────────────────┐          ┌──────────────────┐                     │
│    │ yaa:audio-       │          │ yaa:video-       │                     │
│    │ generation       │          │ generation       │                     │
│    │ (BullMQ/Redis)   │          │ (BullMQ/Redis)   │                     │
│    └────────┬─────────┘          └────────┬─────────┘                     │
│             │                             │                               │
│             ▼                             ▼                               │
│    ┌──────────────────┐          ┌──────────────────┐                     │
│    │  Audio Worker    │          │  Video Worker    │                     │
│    │                  │          │                  │                     │
│    │  ┌────────────┐  │          │  ┌────────────┐  │                     │
│    │  │ ElevenLabs │  │          │  │   HeyGen   │  │                     │
│    │  │    TTS     │  │          │  │  Avatar    │  │                     │
│    │  └────────────┘  │          │  └────────────┘  │                     │
│    │  ┌────────────┐  │          │  ┌────────────┐  │                     │
│    │  │ Ghana NLP  │  │          │  │  Caption   │  │                     │
│    │  │    TTS     │  │          │  │  Service   │  │                     │
│    │  └────────────┘  │          │  └────────────┘  │                     │
│    └────────┬─────────┘          └────────┬─────────┘                     │
│             │                             │                               │
│             └──────────────┬──────────────┘                               │
│                            ▼                                              │
│                   ┌──────────────────┐                                    │
│                   │  S3 / Database   │                                    │
│                   │  (Media Storage) │                                    │
│                   └──────────────────┘                                    │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Agent Communication Flow#

text
┌─────────────────────────────────────────────────────────────────────────────┐
│                      AGENT COMMUNICATION FLOW                               │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│                          ┌─────────────────┐                               │
│                          │   Orchestrator  │                               │
│                          │     Agent       │                               │
│                          └────────┬────────┘                               │
│                                   │                                        │
│                                   │ broadcast()                            │
│                                   ▼                                        │
│                          ┌─────────────────┐                               │
│                          │   MessageBus    │                               │
│                          │ (Redis Pub/Sub) │                               │
│                          └────────┬────────┘                               │
│                                   │                                        │
│            ┌──────────────────────┼──────────────────────┐                 │
│            │                      │                      │                 │
│            ▼                      ▼                      ▼                 │
│   ┌─────────────────┐   ┌─────────────────┐   ┌─────────────────┐         │
│   │  Content        │   │   Fact-Check    │   │   Sentiment     │         │
│   │  Analyzer       │   │   Agent         │   │   Analyzer      │         │
│   │  Agent          │   │                 │   │   Agent         │         │
│   └────────┬────────┘   └────────┬────────┘   └────────┬────────┘         │
│            │                      │                      │                 │
│            │ send()               │ send()               │ send()          │
│            │                      │                      │                 │
│            └──────────────────────┼──────────────────────┘                 │
│                                   │                                        │
│                                   ▼                                        │
│                          ┌─────────────────┐                               │
│                          │   MessageBus    │                               │
│                          │  (direct msgs)  │                               │
│                          └────────┬────────┘                               │
│                                   │                                        │
│                                   ▼                                        │
│                          ┌─────────────────┐                               │
│                          │   Orchestrator  │                               │
│                          │   (aggregates)  │                               │
│                          └─────────────────┘                               │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Integration Points Summary#

Cross-Cutting Communication Matrix#

From \ To Database BullMQ Redis PubSub REST API External API
Ingestion Write (feed_items) Enqueue (content-processing) - - RSS/Sitemap
Processing Worker Read/Write Consume - - -
Audio Worker Read/Write Consume - - ElevenLabs, Ghana NLP
Video Worker Read/Write Consume - - HeyGen
AI Agents Read - Pub/Sub - LLM APIs
API Server Read/Write - Cache Serve Various
Social Scheduler Read Enqueue/Consume - - Social Platforms

Critical Integration Points#

  1. Database as Source of Truth

    • All packages share PostgreSQL database
    • Consistent schema across all workers
    • Transaction isolation for complex operations
  2. BullMQ as Task Coordinator

    • Reliable job delivery with retries
    • Priority-based scheduling
    • Automatic cleanup and monitoring
  3. Redis as Real-Time Bus

    • Agent-to-agent messaging
    • Broadcast notifications
    • Message persistence for reliability
  4. REST APIs for External Integration

    • Fastify server for client requests
    • Native fetch for external services
    • Consistent timeout and retry handling

Recommendations for Oshun Integration#

Migration Considerations#

  1. Queue Standardization

    • Align BullMQ queue naming with Oshun conventions
    • Consolidate Redis connection management
    • Implement shared queue monitoring
  2. Database Integration

    • Map Yaa tables to Oshun schema patterns
    • Consider Prisma migration for type safety
    • Plan for cross-domain query patterns
  3. API Gateway

    • Route Yaa endpoints through Oshun API gateway
    • Implement consistent authentication
    • Add rate limiting and caching
  4. Agent Framework

    • Evaluate integration with Oshun agent patterns
    • Standardize MessageBus interface
    • Consolidate LLM client configurations

Last Updated: January 2026

Part of the Veritas domain integration documentation