This document provides a comprehensive analysis of communication mechanisms used across the Yaa codebase, including message queues, pub/sub patterns, REST APIs, event systems, and database access patterns.
Table of Contents#
- Overview
- BullMQ Job Queue Architecture
- Redis Pub/Sub Message Bus
- REST API Communication
- Event Emitter Patterns
- Database Communication
- Inter-Package Dependencies
- Communication Flow Diagrams
- Integration Points Summary
Overview#
The Yaa codebase employs a distributed, event-driven architecture with multiple communication mechanisms:
| Mechanism | Technology | Primary Use Case |
|---|---|---|
| Job Queues | BullMQ + Redis | Async task processing pipelines |
| Pub/Sub | IORedis | Inter-agent real-time messaging |
| REST APIs | Fastify + fetch | External service integration |
| Event Emitters | Node.js events | Intra-process event handling |
| Direct Database | pg (node-postgres) | Shared state management |
| Package Imports | TypeScript | Shared utilities and models |
BullMQ Job Queue Architecture#
Queue Registry#
The system uses three primary BullMQ queues for content processing pipelines:
1. Content Processing Queue#
Queue Name: yaa:content-processing
Location: packages/ingestion/processing-queue/
| File | Purpose |
|---|---|
src/queue.ts |
Queue definition and configuration |
src/producer.ts |
Job enqueueing logic |
src/worker.ts |
Job processing implementation |
src/producer-db.ts |
Database queries for producer |
src/worker-db.ts |
Database queries for worker |
src/pipeline.ts |
Processing pipeline stages |
Job Type: process_feed_item
Job Data Structure:
interface ProcessFeedItemJob {
feedItemId: number;
normalizedFeedItemId: number | null;
didNormalize: boolean;
didDedupe: boolean;
didLanguageDetect: boolean;
didTimestampNormalize: boolean;
didMediaExtract: boolean;
didAttribution: boolean;
}
Queue Configuration:
{
defaultJobOptions: {
attempts: 8,
backoff: {
type: 'exponential',
delay: 1000 // 1 second base delay
},
removeOnComplete: 1000,
removeOnFail: 1000
}
}
Pipeline Stages:
- Content normalization (HTML parsing, text extraction)
- Semantic deduplication (Simhash64 fingerprinting)
- Language detection (multi-language support)
- Timestamp normalization (date parsing)
- Media extraction (images, videos, links)
- Source attribution
2. Audio Generation Queue#
Queue Name: yaa:audio-generation
Location: packages/audio/audio-generation-queue/
| File | Purpose |
|---|---|
src/queue.ts |
Queue definition |
src/producer.ts |
Article audio job creation |
src/worker.ts |
TTS generation processing |
Job Type: generate_article_audio
Job Data Structure:
interface GenerateAudioJob {
articleId: number;
voiceProfileId: number;
voiceProfileSlug: string;
priority: number; // Computed from publish timestamp
}
Queue Configuration:
{
defaultJobOptions: {
attempts: 6,
backoff: {
type: 'exponential',
delay: 2000 // 2 second base delay
},
removeOnComplete: 1000,
removeOnFail: 1000
}
}
TTS Providers:
- ElevenLabs (
@yaa/audio-elevenlabs-service) - English TTS - Ghana NLP (
@yaa/ghana-nlp) - Twi, Ewe, Ga, Hausa, Dagbani TTS
3. Video Generation Queue#
Queue Name: yaa:video-generation
Location: packages/video/video-generation-queue/
| File | Purpose |
|---|---|
src/queue.ts |
Queue definition |
src/producer.ts |
Video job creation with anchor routing |
src/worker.ts |
Video synthesis and packaging |
Job Types:
| Job Type | Description |
|---|---|
generate_article_video |
Full video synthesis via HeyGen |
package_article_video_hls |
HLS packaging for streaming |
Job Data Structure:
interface GenerateVideoJob {
articleId: number;
anchorProfileId: number;
anchorMode: 'fixed' | 'auto';
priority: number;
}
Anchor Routing Modes:
- Fixed Mode: All articles use a single specified anchor profile
- Auto Mode: Load-balances articles across active anchor profiles
- Fallback: Uses fallback profile when no active anchors available
Queue Configuration:
{
defaultJobOptions: {
attempts: 6,
backoff: {
type: 'exponential',
delay: 2000
},
removeOnComplete: 1000,
removeOnFail: 1000
}
}
4. Social Media Unified Scheduler#
Location: packages/social/unified-scheduler/
Job Type: publishUnified
Job Data Structure:
interface PublishUnifiedJob {
postId: string;
post: UnifiedScheduledPost;
}
Features:
- Multi-platform orchestration (Instagram, TikTok, YouTube, X/Twitter)
- Configurable concurrency per platform
- Sequential platform publishing with delays
- Retry handling per platform
Producer-Consumer Flow#
┌─────────────────────────────────────────────────────────────────────┐
│ Content Processing Pipeline │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ RSS/Sitemap ┌──────────────┐ ┌─────────────────┐ │
│ Collectors ──▶ │ feed_items │ ───▶ │ Processing Queue│ │
│ │ (database) │ │ Producer │ │
│ └──────────────┘ └────────┬────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────┐ │
│ │ yaa:content- │ │
│ │ processing │ │
│ │ (Redis) │ │
│ └──────┬───────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Processing Worker │ │
│ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌──────────┐ │ │
│ │ │ Normalize │▶│ Dedupe │▶│ Lang Detect│▶│ Extract │ │ │
│ │ │ Content │ │ (Simhash) │ │ │ │ Media │ │ │
│ │ └────────────┘ └────────────┘ └────────────┘ └──────────┘ │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌───────────────────┐ │
│ │ normalized_feed_ │ │
│ │ items (database) │ │
│ └───────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
Redis Pub/Sub Message Bus#
Agent Communication System#
Location: packages/ai/agents/communication/message-bus.ts
The MessageBus provides real-time inter-agent communication using Redis Pub/Sub.
Architecture#
class MessageBus {
private redis: Redis; // Main Redis client
private subscriber: Redis; // Dedicated subscriber client
private queues: Map<string, Message[]>; // In-memory message queues
private subscriptions: Map<string, Set<(msg: Message) => void>>;
}
Channel Naming Convention#
| Channel Pattern | Purpose |
|---|---|
yaa:messages:${agentId} |
Direct agent-to-agent messages |
yaa:messages:broadcast |
System-wide broadcasts |
yaa:queue:${agentId} |
Persistent message queue (Redis List) |
Message Structure#
interface Message<T = unknown> {
type: string;
from: string; // Sender agent ID
to: string | null; // Recipient (null for broadcast)
payload: T;
priority: 'low' | 'normal' | 'high' | 'critical';
ttlMs: number; // Time-to-live (default: 5 minutes)
timestamp: number;
correlationId?: string; // For request-response tracking
}
Message Bus Operations#
Sending Direct Messages#
// Send targeted message to specific agent
await messageBus.send({
type: 'TASK_ASSIGNMENT',
from: 'orchestrator',
to: 'content-analyzer-1',
payload: { articleId: 12345 },
priority: 'high',
ttlMs: 300000, // 5 minutes
});
Broadcasting#
// Broadcast to all subscribed agents
await messageBus.broadcast({
type: 'SYSTEM_ALERT',
from: 'coordinator',
to: null,
payload: { message: 'Scaling up workers' },
priority: 'critical',
});
Subscribing#
// Subscribe to messages
messageBus.subscribe('my-agent-id', (message) => {
console.log(`Received: ${message.type} from ${message.from}`);
});
Message Persistence#
Messages are persisted to Redis Lists for reliability:
// Queue key pattern
const queueKey = `yaa:queue:${agentId}`;
// Operations
LPUSH queueKey message // Enqueue
LRANGE queueKey 0 -1 // Read all
LTRIM queueKey 0 maxSize // Limit queue size
Retry Configuration#
const DEFAULT_OPTIONS = {
maxQueueSize: 1000,
maxRetries: 3,
retryBackoffMs: 1000, // Exponential backoff base
messageTtlMs: 300000, // 5 minutes default TTL
};
Statistics API#
interface MessageBusStats {
subscriptions: number; // Active subscriptions
broadcasts: number; // Broadcasts sent
queued: number; // Messages in queues
pending: number; // Pending delivery
historySize: number; // Messages in history
}
REST API Communication#
API Server Architecture#
Location: packages/api/
Framework: Fastify
Route Structure#
packages/api/src/
├── interfaces/http/
│ ├── server.ts # Fastify server setup
│ └── routes/
│ ├── v1.ts # Main v1 API routes
│ └── health.ts # Health check endpoints
└── application/
├── articles/ # Article CRUD operations
├── bias/ # Political bias scoring
├── factCheck/ # Fact-checking API
└── writingAssistant/ # AI writing assistance
Core API Endpoints#
Article Operations#
| Endpoint | Handler | Description |
|---|---|---|
GET /v1/articles |
listArticles.ts |
Paginated article list |
GET /v1/articles/:id |
getArticleById.ts |
Single article fetch |
GET /v1/articles/search |
searchArticles.ts |
Full-text search |
GET /v1/articles/:id/related |
getRelatedArticles.ts |
Related content |
POST /v1/articles/search/personalized |
searchPersonalization.ts |
Personalized ranking |
NLP Services#
| Endpoint | Handler | NLP Package Used |
|---|---|---|
POST /v1/bias/score |
politicalBiasScoring.ts |
@yaa/nlp-topic-classification-service |
POST /v1/factcheck/claim |
factCheckClaim.ts |
@yaa/nlp-evidence-retrieval-service |
POST /v1/writing/assist |
assist.ts |
@yaa/ai-agents (LLMClient) |
External Service Integration#
The system integrates with multiple external APIs:
HeyGen Video API#
Location: packages/video/heygen-service/src/client.ts
class HeyGenClient {
private baseUrl: string;
private apiKey: string;
async generateVideo(config: VideoConfig): Promise<VideoResult> {
const response = await fetch(`${this.baseUrl}/v2/video/generate`, {
method: 'POST',
headers: {
'X-Api-Key': this.apiKey,
'Content-Type': 'application/json',
},
body: JSON.stringify(config),
signal: AbortSignal.timeout(30000),
});
return this.handleResponse(response);
}
}
External Data Sources#
| Service | Location | Data Type |
|---|---|---|
| Ghana Stock Exchange | packages/ai/market-summary-generator/src/gse.ts |
Stock prices |
| Forex Rates | packages/ai/market-summary-generator/src/forex.ts |
Currency exchange |
| Weather (GMET) | packages/ai/weather-report-generator/src/gmet.ts |
Weather forecasts |
| Event Calendar (Nager) | packages/ai/event-calendar-updater/src/nager.ts |
Public holidays |
| Fuel Prices (NPA) | packages/ai/fuel-price-update-generator/src/npa.ts |
Fuel pricing |
| Google Fact-Check | packages/ai/fact-check-integrations/src/google-factcheck.ts |
Claim verification |
| ClaimBuster | packages/ai/fact-check-integrations/src/claimbuster.ts |
Claim detection |
HTTP Client Patterns#
All external calls use consistent patterns:
// Standard fetch with timeout
async function fetchWithTimeout(
url: string,
options: RequestInit,
timeoutMs: number
) {
const controller = new AbortController();
const timeoutId = setTimeout(() => controller.abort(), timeoutMs);
try {
const response = await fetch(url, {
...options,
signal: controller.signal,
});
return response;
} finally {
clearTimeout(timeoutId);
}
}
Redis Connection Factory#
Location: packages/api/src/infrastructure/redis/createRedis.ts
function createRedis(options: RedisOptions): Redis {
return new Redis({
...options,
retryStrategy: (times) => {
if (times > 3) return null; // Stop retrying
return Math.min(times * 1000, 3000); // Max 3s delay
},
maxRetriesPerRequest: 3,
});
}
Event Emitter Patterns#
Agent Event System#
Location: packages/ai/agents/core/base-agent.ts
All AI agents extend Node.js EventEmitter for lifecycle events:
import { EventEmitter } from 'events';
abstract class BaseAgent extends EventEmitter {
protected messageBus: MessageBus;
protected stateManager: StateManager;
protected metricsCollector: MetricsCollector;
protected llmClient: LLMClient;
constructor(config: AgentConfig) {
super();
this.setupEventHandlers();
}
private setupEventHandlers(): void {
this.on('taskStarted', this.handleTaskStarted.bind(this));
this.on('taskCompleted', this.handleTaskCompleted.bind(this));
this.on('taskFailed', this.handleTaskFailed.bind(this));
this.on('stateUpdated', this.handleStateUpdated.bind(this));
}
}
Agent Event Types#
| Event | Trigger | Payload |
|---|---|---|
taskStarted |
Task begins processing | { taskId, agentId, timestamp } |
taskCompleted |
Task finishes successfully | { taskId, result, duration } |
taskFailed |
Task encounters error | { taskId, error, retryCount } |
stateUpdated |
Agent state changes | { previousState, newState } |
messageDelivered |
Message sent successfully | { messageId, recipient } |
messageDiscarded |
Message TTL expired | { messageId, reason } |
Orchestrator Pattern#
Location: packages/ai/agents/orchestration/orchestrator.ts
class Orchestrator extends EventEmitter {
private agents: Map<string, BaseAgent>;
private taskQueue: TaskQueue;
async routeTask(task: Task): Promise<void> {
const agent = this.selectAgent(task);
this.emit('taskRouted', { taskId: task.id, agentId: agent.id });
await agent.process(task);
}
async broadcastToAgents(message: Message): Promise<void> {
for (const agent of this.agents.values()) {
await this.messageBus.send({ ...message, to: agent.id });
}
this.emit('broadcastSent', { messageType: message.type });
}
}
Database Communication#
PostgreSQL Access Pattern#
All database access uses the pg (node-postgres) library with connection
pooling:
import { Pool } from 'pg';
const pool = new Pool({
connectionString: process.env.DATABASE_URL,
application_name: 'yaa-processing-queue-worker',
max: 20,
idleTimeoutMillis: 30000,
connectionTimeoutMillis: 2000,
});
Shared Database Schema#
The following tables are accessed by multiple packages:
Content Tables#
| Table | Primary Writers | Readers |
|---|---|---|
sources |
Ingestion workers | API, NLP services |
articles |
AI generators, ingestion | API, video/audio workers |
authors |
Ingestion workers | API |
categories |
Ingestion workers | API, classification |
topics |
Topic classifier | API, personalization |
tags |
Keyword extractor | API, search |
Relationship Tables#
| Table | Purpose |
|---|---|
article_authors |
Article-to-author mapping |
article_attributions |
Source attribution tracking |
article_categories |
Article categorization |
article_topics |
Topic assignment |
article_tags |
Tag attachment |
Ingestion Tables#
| Table | Purpose |
|---|---|
feed_items |
Raw ingested content |
normalized_feed_items |
Processed content |
content_fingerprints |
Simhash fingerprints for dedup |
content_dedupe_mappings |
Duplicate item relationships |
Transaction Patterns#
Location: packages/ai/market-summary-generator/src/db.ts
Complex operations use explicit transaction management:
async function withTransaction<T>(
pool: Pool,
fn: (client: PoolClient) => Promise<T>
): Promise<T> {
const client = await pool.connect();
try {
await client.query('BEGIN');
const result = await fn(client);
await client.query('COMMIT');
return result;
} catch (error) {
await client.query('ROLLBACK');
throw error;
} finally {
client.release();
}
}
Article Publishing Transaction#
The market summary generator demonstrates a complex multi-step transaction:
async function publishMarketSummary(client: PoolClient, data: MarketSummary) {
// Step 1: Upsert source
const sourceId = await upsertSource(client, data.source);
// Step 2: Upsert categories
const categoryIds = await upsertCategories(client, data.categories);
// Step 3: Upsert topics
const topicIds = await upsertTopics(client, data.topics);
// Step 4: Upsert author
const authorId = await upsertAuthor(client, data.author);
// Step 5: Upsert tags
const tagIds = await upsertTags(client, data.tags);
// Step 6: Upsert article (ON CONFLICT DO UPDATE)
const { articleId, isNew } = await upsertArticle(client, {
slug: data.slug,
title: data.title,
content: data.content,
sourceId,
publishedAt: data.publishedAt,
});
// Step 7: Attach author
await attachAuthor(client, articleId, authorId);
// Step 8: Upsert attributions
await upsertAttributions(client, articleId, data.attributions);
// Step 9-11: Attach categories, topics, tags
await attachCategories(client, articleId, categoryIds);
await attachTopics(client, articleId, topicIds);
await attachTags(client, articleId, tagIds);
return { articleId, isNew };
}
Deduplication Pipeline#
Location: packages/ingestion/processing-queue/src/pipeline.ts
// Simhash-based content fingerprinting
async function deduplicateContent(
client: PoolClient,
content: string,
itemId: number
) {
// Generate Simhash64 fingerprint
const fingerprint = simhash64(content);
// Find similar fingerprints in same bucket
const bucket = fingerprint >> 48; // Top 16 bits as bucket
const candidates = await findCandidatesInBucket(client, bucket);
// Check Hamming distance for near-duplicates
for (const candidate of candidates) {
const distance = hammingDistance(fingerprint, candidate.fingerprint);
if (distance <= SIMILARITY_THRESHOLD) {
await recordDuplicate(client, itemId, candidate.itemId, distance);
return { isDuplicate: true, originalId: candidate.itemId };
}
}
// Store new fingerprint
await storeFingerprint(client, itemId, fingerprint, bucket);
return { isDuplicate: false };
}
Inter-Package Dependencies#
Dependency Graph Overview#
The system has 133 unique @yaa/* package imports. Key dependency clusters:
Core Shared Packages#
@yaa/models ← Shared TypeScript types and enums
@yaa/ai-agents ← Agent framework, LLM client, logging
@yaa/nlp-service-kit ← Base NLP utilities
NLP Service Dependencies#
┌─────────────────────────────┐
│ @yaa/nlp-service-kit │
└─────────────┬───────────────┘
│
┌─────────────────────────┼─────────────────────────┐
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌────────────────────┐ ┌────────────────────┐
│ @yaa/nlp- │ │ @yaa/nlp-topic- │ │ @yaa/nlp-keyword- │
│ embedding- │ │ classification- │ │ extraction- │
│ service │ │ service │ │ service │
└───────────────┘ └────────────────────┘ └────────────────────┘
Ingestion Dependencies#
@yaa/ingestion-rss-collector
│
▼
@yaa/ingestion-scraper
│
▼
@yaa/ingestion-content-normalization
│
├──▶ @yaa/ingestion-language-detection
├──▶ @yaa/ingestion-timestamp-normalization
├──▶ @yaa/ingestion-media-extraction
└──▶ @yaa/ingestion-semantic-dedup
Video Pipeline Dependencies#
@yaa/video-generation-queue
│
├──▶ @yaa/video-heygen-service
├──▶ @yaa/video-avatar-profiles
├──▶ @yaa/video-captioning-service
├──▶ @yaa/video-thumbnail-generator
└──▶ @yaa/video-vertical-converter
│
├──▶ @yaa/video-tiktok-video-producer
├──▶ @yaa/video-youtube-upload-service
└──▶ @yaa/video-youtube-shorts-upload-service
Social Media Dependencies#
@yaa/social-unified-scheduler
│
├──▶ @yaa/social-instagram-scheduler
├──▶ @yaa/social-tiktok-upload-service
├──▶ @yaa/social-youtube-channel-setup
└──▶ @yaa/social-whatsapp-business-service
Communication Flow Diagrams#
End-to-End Content Pipeline#
┌─────────────────────────────────────────────────────────────────────────────┐
│ CONTENT INGESTION FLOW │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ RSS Feeds │ │ Sitemaps │ │ Social APIs │ │
│ │ (300+ feeds) │ │ │ │ (X/Twitter) │ │
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
│ │ │ │ │
│ └────────────────────┼────────────────────┘ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ feed_items │ │
│ │ (PostgreSQL) │ │
│ └────────┬─────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ Processing │ │
│ │ Queue Producer │ │
│ └────────┬─────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ yaa:content- │ │
│ │ processing │ │
│ │ (BullMQ/Redis) │ │
│ └────────┬─────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────────┐ │
│ │ Processing Worker │ │
│ │ │ │
│ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │
│ │ │Normalize│──▶│ Dedupe │──▶│ Lang │──▶│ Extract │ │ │
│ │ │ Content │ │(Simhash)│ │ Detect │ │ Media │ │ │
│ │ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │ │
│ │ │ │
│ └──────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────┐ │
│ │normalized_feed_ │ │
│ │items (PostgreSQL)│ │
│ └──────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Media Generation Pipeline#
┌─────────────────────────────────────────────────────────────────────────────┐
│ MEDIA GENERATION FLOW │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────┐ │
│ │ articles │ │
│ │ (PostgreSQL) │ │
│ └────────┬─────────┘ │
│ │ │
│ ┌──────────────┴──────────────┐ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Audio Queue │ │ Video Queue │ │
│ │ Producer │ │ Producer │ │
│ └────────┬─────────┘ └────────┬─────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ yaa:audio- │ │ yaa:video- │ │
│ │ generation │ │ generation │ │
│ │ (BullMQ/Redis) │ │ (BullMQ/Redis) │ │
│ └────────┬─────────┘ └────────┬─────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Audio Worker │ │ Video Worker │ │
│ │ │ │ │ │
│ │ ┌────────────┐ │ │ ┌────────────┐ │ │
│ │ │ ElevenLabs │ │ │ │ HeyGen │ │ │
│ │ │ TTS │ │ │ │ Avatar │ │ │
│ │ └────────────┘ │ │ └────────────┘ │ │
│ │ ┌────────────┐ │ │ ┌────────────┐ │ │
│ │ │ Ghana NLP │ │ │ │ Caption │ │ │
│ │ │ TTS │ │ │ │ Service │ │ │
│ │ └────────────┘ │ │ └────────────┘ │ │
│ └────────┬─────────┘ └────────┬─────────┘ │
│ │ │ │
│ └──────────────┬──────────────┘ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ S3 / Database │ │
│ │ (Media Storage) │ │
│ └──────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Agent Communication Flow#
┌─────────────────────────────────────────────────────────────────────────────┐
│ AGENT COMMUNICATION FLOW │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────┐ │
│ │ Orchestrator │ │
│ │ Agent │ │
│ └────────┬────────┘ │
│ │ │
│ │ broadcast() │
│ ▼ │
│ ┌─────────────────┐ │
│ │ MessageBus │ │
│ │ (Redis Pub/Sub) │ │
│ └────────┬────────┘ │
│ │ │
│ ┌──────────────────────┼──────────────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Content │ │ Fact-Check │ │ Sentiment │ │
│ │ Analyzer │ │ Agent │ │ Analyzer │ │
│ │ Agent │ │ │ │ Agent │ │
│ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ │
│ │ │ │ │
│ │ send() │ send() │ send() │
│ │ │ │ │
│ └──────────────────────┼──────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ MessageBus │ │
│ │ (direct msgs) │ │
│ └────────┬────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Orchestrator │ │
│ │ (aggregates) │ │
│ └─────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Integration Points Summary#
Cross-Cutting Communication Matrix#
| From \ To | Database | BullMQ | Redis PubSub | REST API | External API |
|---|---|---|---|---|---|
| Ingestion | Write (feed_items) | Enqueue (content-processing) | - | - | RSS/Sitemap |
| Processing Worker | Read/Write | Consume | - | - | - |
| Audio Worker | Read/Write | Consume | - | - | ElevenLabs, Ghana NLP |
| Video Worker | Read/Write | Consume | - | - | HeyGen |
| AI Agents | Read | - | Pub/Sub | - | LLM APIs |
| API Server | Read/Write | - | Cache | Serve | Various |
| Social Scheduler | Read | Enqueue/Consume | - | - | Social Platforms |
Critical Integration Points#
-
Database as Source of Truth
- All packages share PostgreSQL database
- Consistent schema across all workers
- Transaction isolation for complex operations
-
BullMQ as Task Coordinator
- Reliable job delivery with retries
- Priority-based scheduling
- Automatic cleanup and monitoring
-
Redis as Real-Time Bus
- Agent-to-agent messaging
- Broadcast notifications
- Message persistence for reliability
-
REST APIs for External Integration
- Fastify server for client requests
- Native fetch for external services
- Consistent timeout and retry handling
Recommendations for Oshun Integration#
Migration Considerations#
-
Queue Standardization
- Align BullMQ queue naming with Oshun conventions
- Consolidate Redis connection management
- Implement shared queue monitoring
-
Database Integration
- Map Yaa tables to Oshun schema patterns
- Consider Prisma migration for type safety
- Plan for cross-domain query patterns
-
API Gateway
- Route Yaa endpoints through Oshun API gateway
- Implement consistent authentication
- Add rate limiting and caching
-
Agent Framework
- Evaluate integration with Oshun agent patterns
- Standardize MessageBus interface
- Consolidate LLM client configurations
Last Updated: January 2026
Part of the Veritas domain integration documentation