Eve library

Veritas AI Agent System Documentation

Guide

Revision 1

On this page

Technical documentation for the Veritas multi-agent AI newsroom system, covering agent roles, interaction patterns, task management, and escalation procedures.


Table of Contents#

  1. System Overview
  2. Agent Roles and Responsibilities
  3. Agent Lifecycle
  4. Task Management
  5. Agent Interaction Patterns
  6. LLM Integration
  7. Memory and Context
  8. Health Monitoring
  9. Escalation Procedures
  10. Configuration Reference
  11. API Reference
  12. Metrics and Observability
  13. Operational Procedures

1. System Overview#

1.1 Architecture#

The Veritas AI Agent System is a multi-agent orchestration platform that powers an autonomous AI newsroom. It coordinates specialized AI agents to perform journalistic tasks including content creation, editing, fact-checking, and distribution.

text
┌─────────────────────────────────────────────────────────┐
│                     HTTP API (Hono)                      │
│              Port 3006 (AGENTS_PORT)                     │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  ┌───────────────────────────────────────────────────┐  │
│  │              Orchestrator                          │  │
│  │  - Task routing and distribution                   │  │
│  │  - Agent registry                                  │  │
│  │  - Backlog management                              │  │
│  │  - Load balancing                                  │  │
│  └───────────────────┬───────────────────────────────┘  │
│                      │                                   │
│  ┌───────────────────┴───────────────────────────────┐  │
│  │              Message Bus                           │  │
│  │  - Local (in-memory) or Redis (distributed)        │  │
│  │  - Pub/Sub with retry logic                        │  │
│  │  - Per-agent message queues                        │  │
│  └───────────────────┬───────────────────────────────┘  │
│                      │                                   │
│  ┌───────┬───────┬───┴───┬───────┬───────┬───────────┐  │
│  │ EdIC  │ MngEd │ Journ │ FactCk│ CtStr │ SocMedia  │  │
│  │       │       │ x2    │       │       │           │  │
│  └───────┴───────┴───────┴───────┴───────┴───────────┘  │
│                                                         │
│  ┌───────────────────────────────────────────────────┐  │
│  │  Health Monitor  │  State Manager  │  Metrics     │  │
│  └───────────────────────────────────────────────────┘  │
│                                                         │
└─────────────────────────────────────────────────────────┘

1.2 Service Configuration#

Setting Environment Variable Default
HTTP Port AGENTS_PORT 3006
HTTP Host AGENTS_HOST 0.0.0.0
Redis URL REDIS_URL —
Anthropic API Key ANTHROPIC_API_KEY —
OpenAI API Key OPENAI_API_KEY —
Health Check Interval — 30,000 ms
Metrics Collection — 60,000 ms
Log Level — info

1.3 Default Agent Fleet#

The system starts with 7 pre-registered agents:

Agent ID Role Specialization Priority Max Concurrent
editor-in-chief-1 Editor-in-Chief — 10 5
managing-editor-1 Managing Editor — 9 5
journalist-1 Journalist Investigative 7 3
journalist-2 Journalist Political 7 3
fact-checker-1 Fact Checker — 8 3
content-strategist-1 Content Strategist — 6 3
social-media-manager-1 Social Media Manager — 5 5

2. Agent Roles and Responsibilities#

2.1 Editor-in-Chief#

Role: editor_in_chief

The highest editorial authority in the agent system. Responsible for final editorial decisions and publication approval.

Capabilities:

  • Make editorial decisions
  • Publish content
  • Edit content
  • Review content
  • Assign stories

Task Types Handled:

  • reviewContent — Final editorial review before publication
  • assignStory — Story assignment to journalist agents
  • prioritizeNews — News prioritization decisions

LLM Configuration:

  • Model: claude-sonnet-4-20250514
  • Temperature: 0.3 (analytical, conservative)
  • Max Tokens: 4,096

Human Oversight: Required for critical topics (national security, election coverage, defamation risk).

2.2 Managing Editor#

Role: managing_editor

Day-to-day operations manager. Handles content editing, scheduling, and operational decisions.

Capabilities:

  • Edit content
  • Make editorial decisions

Task Types Handled:

  • editArticle — Content editing and refinement
  • Task scheduling and workflow coordination

LLM Configuration:

  • Model: claude-sonnet-4-20250514
  • Temperature: 0.4
  • Max Tokens: 4,096

Human Oversight: Weekly editorial review.

2.3 Journalist#

Role: journalist

Content creator with optional specialization. The primary content-producing agent.

Capabilities:

  • Generate content
  • Access external APIs

Specializations:

Specialization Focus Area Special Considerations
political Government, parliament, parties Balance monitoring required
business Economy, markets, corporate Financial data verification
sports Athletics, football, events Score/result verification
entertainment Arts, culture, celebrity Privacy considerations
regional Local government, development Local source verification
investigative In-depth, exposés, accountability Multiple source requirement
breaking_news Urgent, time-sensitive events Rapid verification protocol
general All other topics Standard editorial procedures

Task Types Handled:

  • writeArticle — Original article creation
  • translateContent — Content translation
  • summarizeContent — Content summarization
  • generateHeadline — Headline generation
  • generateVideoScript — Video script writing
  • researchTopic — Topic research and background

LLM Configuration:

  • Model: claude-sonnet-4-20250514
  • Temperature: 0.7 (creative, varied output)
  • Max Tokens: 8,192

Human Oversight: Pre-publication editorial review for all content.

2.4 Fact Checker#

Role: fact_checker

Verification and evidence evaluation specialist. Ensures accuracy of claims and statements.

Capabilities:

  • Access external APIs (search engines, fact-check databases)

Task Types Handled:

  • factCheck — Full fact-check of article or statement
  • verifyClaim — Verification of individual claims
  • analyzeSource — Source credibility analysis

LLM Configuration:

  • Model: claude-sonnet-4-20250514
  • Temperature: 0.2 (strict, deterministic)
  • Max Tokens: 4,096

Human Oversight: Critical verdict review for high-impact claims.

2.5 Content Strategist#

Role: content_strategist

Cross-platform content planning and audience analysis.

Capabilities:

  • Generate content

Task Types Handled:

  • Content planning and scheduling
  • Audience analysis
  • Cross-platform distribution strategy

LLM Configuration:

  • Model: claude-sonnet-4-20250514
  • Temperature: 0.5
  • Max Tokens: 4,096

Human Oversight: Strategy review on a periodic basis.

2.6 Social Media Manager#

Role: social_media_manager

Social media content creation and platform distribution.

Capabilities:

  • Generate content
  • Publish content

Task Types Handled:

  • createSocialPost — Social media post creation
  • Platform-specific content adaptation
  • Engagement monitoring

LLM Configuration:

  • Model: claude-3-5-haiku
  • Temperature: 0.8 (engaging, creative)
  • Max Tokens: 2,048

Human Oversight: Post review before distribution.

2.7 Orchestrator (System Agent)#

Role: orchestrator

Central task coordinator and workflow manager. Not a content-producing agent.

Responsibilities:

  • Task routing based on configurable rules
  • Agent registry management
  • Backlog monitoring and distribution
  • Load balancing across agents
  • Scaling signals (up/down)

LLM Configuration:

  • Model: claude-3-5-haiku
  • Temperature: 0.1 (deterministic decisions)
  • Max Tokens: 1,024

2.8 Monitor (System Agent)#

Role: monitor

Health and status monitoring agent.

Responsibilities:

  • Health check execution
  • Issue detection and alerting
  • Self-healing action coordination
  • Metrics collection

LLM Configuration:

  • Model: claude-3-5-haiku
  • Temperature: 0.1 (deterministic)
  • Max Tokens: 1,024

3. Agent Lifecycle#

3.1 Lifecycle States#

text
                ┌─────────────┐
                │ initializing │
                └──────┬──────┘
                       │ onInitialize, onReady
                       ▼
                ┌─────────┐
        ┌──────→│  idle   │◄──────────┐
        │       └────┬────┘           │
        │            │ task received  │ task complete
        │            ▼                │
        │       ┌─────────┐           │
        │       │  busy   │───────────┘
        │       └────┬────┘
        │            │ pause()
        │            ▼
        │       ┌─────────┐
        │       │ paused  │
        │       └────┬────┘
        │            │ resume()
        └────────────┘
                │ error
                ▼
          ┌──────────┐
          │  error   │
          └────┬─────┘
               │ stop()
               ▼
     ┌───────────────┐
     │ shutting_down │
     └───────┬───────┘
             │ cleanup complete
             ▼
        ┌─────────┐
        │ stopped │
        └─────────┘

3.2 Initialization Sequence#

When an agent starts (BaseAgent.start()):

  1. Load saved state from persistence (if available within TTL)
  2. Call onInitialize lifecycle hook
  3. Restore previous state: task queue, context, statistics
  4. Call agent-specific initialize() method
  5. Subscribe to the message bus
  6. Start task processing loop (100 ms interval)
  7. Start health check loop (30 s interval)
  8. Set status to idle and health to healthy
  9. Call onReady lifecycle hook
  10. Emit agentStarted event to the message bus

3.3 Task Processing Loop#

The processing loop runs every 100 ms:

text
loop:
  if paused or shutting_down → skip
  if active_tasks >= maxConcurrentTasks → skip
  if queue is empty → set status to idle, skip

  task = queue.dequeue()
  set status to busy
  set task.status to running
  record start timestamp

  try:
    result = await processTask(task)  // abstract method
    update statistics (completed, duration, tokens, cost)
    broadcast taskComplete message
  catch error:
    if retries remaining → re-queue with backoff
    else → mark task failed, broadcast taskFailed
  finally:
    remove from active tasks
    if queue empty → set status to idle

3.4 Graceful Shutdown#

When an agent stops (BaseAgent.stop()):

  1. Set isShuttingDown flag
  2. Set status to shutting_down
  3. Call onShutdown lifecycle hook
  4. Stop processing loops (clear intervals)
  5. Wait up to 30 seconds for active tasks to complete
  6. Persist full state to storage (queue, context, statistics)
  7. Unsubscribe from the message bus
  8. Call agent-specific cleanup() method
  9. Set status to stopped
  10. Emit agentStopped event with uptime duration

3.5 Lifecycle Hooks#

Agents can implement the following hooks for custom behavior:

Hook Trigger Use Case
onInitialize() Before startup Resource allocation
onReady() After agent is ready Announce availability
onTaskReceived() When a task is assigned Task-specific setup
onTaskCompleted() After task completes Cleanup, logging
onPause() When agent is paused Save intermediate state
onResume() When agent resumes Restore intermediate state
onShutdown() During shutdown Resource cleanup
onError() On error Error logging, recovery

4. Task Management#

4.1 Task Types#

The system supports 18 task types across four categories:

Content Creation#

Task Type Description Default Agent Role
writeArticle Write an original article Journalist
editArticle Edit and refine content Managing Editor
translateContent Translate to another language Journalist
summarizeContent Create a summary Journalist
generateHeadline Generate article headline Journalist
generateVideoScript Write a video script Journalist
createSocialPost Create social media content Social Media Manager

Verification#

Task Type Description Default Agent Role
factCheck Verify article claims Fact Checker
verifyClaim Verify a specific claim Fact Checker
analyzeSource Assess source credibility Fact Checker

Editorial#

Task Type Description Default Agent Role
reviewContent Final editorial review Editor-in-Chief
assignStory Assign story to journalist Editor-in-Chief
prioritizeNews Prioritize news items Editor-in-Chief
researchTopic Research a topic Journalist

System#

Task Type Description Default Agent Role
healthCheck System health check Monitor
syncState Synchronize agent state Orchestrator
cleanup Cleanup stale data Orchestrator
custom Custom task type Per routing rules

4.2 Task Priority Levels#

Priority Value Description Use Case
critical 0 Immediate processing Breaking news, corrections
high 1 Processed before normal tasks Important stories, fact-checks
normal 2 Standard processing order Regular news content
low 3 Processed when capacity allows Feature articles, analysis
background 4 Lowest priority Batch processing, cleanup

4.3 Task Status Flow#

text
pending ──→ queued ──→ running ──→ completed
                         │
                         ├──→ failed (after max retries)
                         ├──→ cancelled
                         └──→ timed_out

4.4 Task Structure#

typescript
interface Task {
  id: TaskId; // Unique branded identifier
  type: TaskType; // One of 18 task types
  title: string; // Human-readable title
  description: string; // Detailed description
  priority: TaskPriority; // 0-4 (critical to background)
  createdBy: AgentId; // Agent that created the task
  assignedTo?: AgentId; // Assigned agent
  status: TaskStatus; // Current status
  timestamps: {
    created: number;
    started?: number;
    completed?: number;
    updated: number;
  };
  input: {
    content: string; // Primary content to process
    context?: Record<string, unknown>;
    references?: string[]; // Source URLs/documents
    constraints?: {
      maxLength?: number;
      style?: string;
      tone?: string;
      language?: string;
      deadline?: number; // Unix timestamp
      requiredSourcesCount?: number;
    };
  };
  parentTaskId?: TaskId; // For task hierarchies
  childTaskIds: TaskId[]; // Sub-tasks
  dependencies: TaskId[]; // Must complete first
  deadline?: number;
  timeoutMs?: number;
  retryConfig: {
    maxAttempts: number; // Default: 3
    backoffMs: number; // Default: 1000ms
    currentAttempt: number;
  };
  metadata: Record<string, unknown>;
}

4.5 Task Routing Rules#

The orchestrator uses 14 default routing rules to assign tasks to appropriate agents:

Task Type Target Role Priority Constraints
writeArticle journalist normal generate_content capability
editArticle managing_editor normal edit_content capability
factCheck fact_checker high access_external_apis
verifyClaim fact_checker high access_external_apis
analyzeSource fact_checker normal access_external_apis
reviewContent editor_in_chief high make_editorial_decisions
assignStory editor_in_chief high make_editorial_decisions
prioritizeNews editor_in_chief normal make_editorial_decisions
translateContent journalist normal generate_content
summarizeContent journalist normal generate_content
generateHeadline journalist normal generate_content
generateVideoScript journalist normal generate_content
createSocialPost social_media_manager normal publish_content
researchTopic journalist low access_external_apis

4.6 Task Distribution Strategies#

The orchestrator supports three distribution strategies:

Round Robin#

Assigns tasks to the agent with the oldest last activity timestamp. Ensures equal distribution across agents of the same role.

Load Balanced (Default)#

Assigns tasks to the agent with the lowest current task count. Optimizes for throughput by avoiding overloaded agents.

Priority Based#

Assigns tasks to the agent with the highest priority level. Ensures the most capable agents handle tasks first.

4.7 Task Result Structure#

typescript
interface TaskResult<T = unknown> {
  taskId: TaskId;
  success: boolean;
  data?: T;
  error?: string;
  metrics: {
    executionTimeMs: number;
    llmLatencyMs: number;
    inputTokens: number;
    outputTokens: number;
    costUsd: number;
    llmCalls: number;
    retries: number;
  };
  completedAt: number;
}

5. Agent Interaction Patterns#

5.1 Message Bus Architecture#

Agents communicate through a publish-subscribe message bus with two implementations:

Implementation Backing Store Use Case Performance
Local In-memory Testing, single-node Sub-millisecond
Redis Redis Pub/Sub Production, distributed Low-latency

The Redis implementation uses the channel veritas:agents:messages.

5.2 Message Types#

Task Messages#

Type Description Sender Receiver
taskAssign Assign a task to an agent Orchestrator Target agent
taskSubmit Submit a task for processing Any agent Orchestrator
taskUpdate Update task progress Processing agent Orchestrator
taskComplete Report task completion Processing agent Orchestrator
taskFailed Report task failure Processing agent Orchestrator
taskCancel Cancel a task Any agent Processing agent

Agent Lifecycle Messages#

Type Description Sender Receiver
agentStart Agent has started Agent Broadcast
agentStop Agent is stopping Agent Broadcast
agentPause Agent is paused Agent Broadcast
agentResume Agent has resumed Agent Broadcast

Status Messages#

Type Description Sender Receiver
statusRequest Request agent status Monitor Target agent
statusResponse Respond with status Agent Monitor
capabilityRequest Query agent capabilities Orchestrator Target agent
capabilityResponse Respond with capabilities Agent Orchestrator

Event Messages#

Type Description Sender Receiver
eventBroadcast General event notification Any agent Broadcast
alert Alert notification Monitor Broadcast

5.3 Message Structure#

typescript
interface Message<T = unknown> {
  id: MessageId; // Unique branded identifier
  type: MessageType; // One of the types above
  source: AgentId; // Sending agent
  destination: AgentId | 'broadcast';
  payload: T; // Type-specific data
  correlationId?: string; // Request-response correlation
  replyTo?: AgentId; // Reply routing
  priority: number; // 0-4 (0 = highest)
  ttl?: number; // Time-to-live in milliseconds
  timestamp: number; // Creation timestamp
  headers: Record<string, string>;
}

5.4 Message Delivery Guarantees#

  • Per-agent queue: Each agent has a message queue (max 1,000 messages)
  • Retry logic: Up to 3 attempts with exponential backoff (100 ms base)
  • TTL enforcement: Expired messages are dropped before delivery
  • Broadcast: Delivered to all subscribed agents except the sender
  • Order: Messages within a queue are processed in order

5.5 Event System#

The agent system emits events for observability and cross-agent coordination:

Event Category Events
Agent agentStarted, agentStopped, agentError, agentHealthChanged
Task taskCreated, taskAssigned, taskStarted, taskProgress, taskCompleted, taskFailed
Content contentCreated, contentUpdated, contentPublished
News breakingNews, trendingTopic
Health alertTriggered, alertResolved

5.6 Common Interaction Patterns#

Content Creation Workflow#

text
1. Orchestrator creates writeArticle task
2. Orchestrator routes to journalist agent (load-balanced)
3. Journalist receives taskAssign message
4. Journalist generates article using LLM
5. Journalist submits taskComplete with article content
6. Orchestrator creates factCheck task → routes to fact-checker
7. Fact-checker verifies claims, submits result
8. Orchestrator creates reviewContent task → routes to editor-in-chief
9. Editor-in-chief reviews, approves or requests edits
10. If approved → content published via contentPublished event

Breaking News Escalation#

text
1. Monitor detects breaking news event
2. Monitor broadcasts breakingNews event (critical priority)
3. Orchestrator creates writeArticle task (critical priority)
4. Journalist-2 (breaking_news specialization) receives assignment
5. Rapid content creation with minimal verification
6. Parallel factCheck task created (high priority)
7. Managing editor review (bypassing full editorial chain)
8. Publish with "developing story" label
9. Follow-up verification and updates scheduled

Fact-Check Chain#

text
1. Article submitted for review
2. Orchestrator creates factCheck task
3. Fact-checker extracts individual claims
4. For each claim: creates verifyClaim sub-task
5. Sub-tasks processed with evidence retrieval
6. Results aggregated into fact-check report
7. Report attached to article via taskComplete
8. Editor-in-chief reviews fact-check results

6. LLM Integration#

6.1 Supported Providers#

Provider SDK Models Available
Anthropic @anthropic-ai/sdk Claude Opus 4, Claude Sonnet 4, Claude 3.5 Haiku
OpenAI openai GPT-4o, GPT-4o mini
Local Veritas adapter Supported through provider injection / fallback

6.2 LLM Client Interface#

typescript
interface LLMClient {
  complete(
    messages: ConversationMessage[],
    systemPrompt?: string
  ): Promise<LLMResponse>;

  completeStream(
    messages: ConversationMessage[],
    systemPrompt?: string,
    callbacks?: StreamCallbacks
  ): Promise<LLMResponse>;

  isAvailable(): Promise<boolean>;
  getConfig(): LLMConfig;
  estimateTokens(text: string): number;
  calculateCost(inputTokens: number, outputTokens: number): number;
}

6.3 Default LLM Configuration by Role#

Role Model Temperature Max Tokens Reasoning
Editor-in-Chief claude-sonnet-4-20250514 0.3 4,096 Analytical, conservative
Managing Editor claude-sonnet-4-20250514 0.4 4,096 Balanced editing
Journalist claude-sonnet-4-20250514 0.7 8,192 Creative, varied output
Fact Checker claude-sonnet-4-20250514 0.2 4,096 Strict, deterministic
Content Strategist claude-sonnet-4-20250514 0.5 4,096 Balanced analysis
Social Media Manager claude-3-5-haiku 0.8 2,048 Engaging, creative
Orchestrator claude-3-5-haiku 0.1 1,024 Deterministic routing
Monitor claude-3-5-haiku 0.1 1,024 Deterministic monitoring

6.4 Cost Model#

Token pricing per 1 million tokens:

Model Input Cost Output Cost
Claude Opus 4 $15.00 $75.00
Claude Sonnet 4 $3.00 $15.00
Claude 3.5 Haiku $0.80 $4.00
GPT-4o $2.50 $10.00
GPT-4o mini $0.15 $0.60

All API calls track token consumption and cost. Per-agent cost statistics are available via the metrics API.

6.5 Retry Configuration#

  • Max Attempts: 3
  • Backoff: Exponential, 1,000 ms base, 10,000 ms maximum
  • Non-Retryable Errors: 401 (Unauthorized), 403 (Forbidden), 400 (Bad Request)
  • Retryable Errors: All other errors (rate limits, server errors, timeouts)

6.6 Token Estimation#

Rough estimation formula: ~4 characters per token

Used for pre-flight checks to avoid exceeding model context limits.


7. Memory and Context#

7.1 Context Structure#

Each agent maintains an execution context with four components:

typescript
interface AgentContext {
  conversationHistory: ConversationMessage[];
  workingMemory: Map<string, unknown>;
  longTermMemoryRefs: string[];
  preferences: Record<string, unknown>;
}

7.2 Conversation History#

  • Purpose: Multi-turn context for LLM interactions
  • Structure: Ordered list of messages with role (user, assistant, system), content, and timestamp
  • Bounded: Limited size to prevent context overflow
  • Passed to LLM: Included in every LLM completion request
  • Persistence: Saved on shutdown, restored on startup

7.3 Working Memory#

  • Purpose: Transient, session-scoped intermediate data
  • Structure: Key-value map (Map<string, unknown>)
  • Lifetime: Cleared on agent restart
  • Use Cases: Intermediate research results, draft content, analysis state

7.4 Long-Term Memory References#

  • Purpose: References to external persistent storage
  • Structure: Array of string identifiers (database IDs, document references)
  • Lifetime: Persisted across sessions
  • Use Cases: Article drafts, research notes, source databases

7.5 Preferences#

  • Purpose: Learned agent preferences and style settings
  • Structure: Key-value record
  • Lifetime: Persisted across sessions
  • Use Cases: Writing tone, formatting preferences, topic expertise weights

7.6 State Persistence#

Agent state is persisted through a state manager with two implementations:

Local State Manager (In-Memory)#

  • Storage: In-process Map
  • Version tracking with incrementing counter
  • TTL enforcement (default: 24 hours)
  • No durability — lost on process restart

Redis State Manager (Distributed)#

  • Key format: veritas:agents:state:{agentId}
  • Version tracking via separate key
  • Optional gzip compression (threshold: 1 KB)
  • TTL via Redis EXPIRE (default: 24 hours)
  • Agent index via Redis SET for enumeration
  • Durable across process restarts

7.7 State Restoration#

On agent startup:

  1. Check for saved state in persistence layer
  2. If state exists and is within TTL:
    • Restore task queue (pending tasks only)
    • Restore conversation history
    • Restore preferences
    • Restore statistics
  3. If state is expired or absent:
    • Start with clean state
  4. Active tasks are not restored — they are re-queued from pending state

8. Health Monitoring#

8.1 Health Status Levels#

Status Meaning Action Required
healthy All systems normal None
degraded Functional but with issues Monitor, investigate
unhealthy Critical issues detected Immediate intervention
unknown Not yet checked Wait for first check

8.2 Health Check Process#

Health checks run every 30 seconds (configurable):

text
1. Send statusRequest to agent
2. Wait for response (5-second timeout)
3. If response received:
   - Reset consecutive failure count
   - Analyze metrics for issues
4. If timeout:
   - Increment consecutive failure count
   - If count >= 3 → mark unhealthy
5. Emit agentHealthChanged event if status changed

8.3 Health Metrics#

Each agent reports the following metrics:

Metric Type Threshold (Warning) Threshold (Issue)
CPU Usage Percent — —
Memory Usage Percent — > 90%
Task Queue Size Count — > 100
Average Response Time Ms — > 5,000 ms
Error Rate Percent — > 10%

8.4 Health Component Tracking#

Each agent tracks health for individual components:

Component Description
llm LLM provider connectivity
message_bus Message bus connectivity
state_manager State persistence connectivity
task_processor Task processing loop health

8.5 Issue Detection#

8 issue types are automatically detected:

Issue Type Trigger Severity Auto-Heal Action
agentUnresponsive 3+ consecutive check failures critical restartAgent
highLatency Response time > 5s high alertOperator
highErrorRate Error rate > 10% high alertOperator
queueBacklog Queue size > 100 medium reduceLoad
memoryPressure Memory usage > 90% high clearQueue
llmUnavailable LLM component unhealthy critical failover
stateCorruption State integrity check failed critical restartAgent
dependencyFailure Dependency agent failed high alertOperator

9. Escalation Procedures#

9.1 Automatic Self-Healing#

When self-healing is enabled (default: true), the system can automatically respond to detected issues:

Healing Action Description Rate Limit
restartAgent Stop and restart the affected agent 10 per hour
clearQueue Clear the agent's task queue 10 per hour
reduceLoad Reduce concurrency or pause task intake 10 per hour
failover Switch to backup LLM provider 10 per hour
alertOperator Send alert to human operator 10 per hour

9.2 Self-Healing Flow#

text
Issue Detected
  │
  ├─→ Is self-healing enabled?
  │     No → Emit alert event only
  │     Yes ↓
  │
  ├─→ Is rate limit exceeded? (10 actions/hour)
  │     Yes → Skip healing, log warning
  │     No ↓
  │
  ├─→ Map issue type to healing action
  │
  ├─→ Is handler registered for this action?
  │     No → Log warning, skip
  │     Yes ↓
  │
  ├─→ Execute healing handler
  │
  ├─→ Record result (success/failure, timestamp)
  │
  └─→ Emit healing action event

9.3 Escalation Matrix#

Issue Severity Self-Heal Human Escalation
Agent unresponsive Critical Restart agent If restart fails
LLM provider down Critical Failover provider If all providers down
State corruption Critical Restart agent If restart fails
High latency (> 5s) High Alert operator Investigate cause
High error rate (> 10%) High Alert operator Investigate cause
Queue backlog (> 100) Medium Reduce load Scale up agents
Memory pressure (> 90%) High Clear queue Increase resources
Dependency failure High Alert operator Fix dependency

9.4 Human-in-the-Loop Escalation#

Certain scenarios require human intervention:

Editorial Escalation#

Scenario Escalation Target Response
Content flagged for bias Managing Editor Editorial review
Fact-check verdict disputed Editor-in-Chief Final verdict
NMC compliance violation Editor-in-Chief Compliance remediation
Election coverage imbalance Editor-in-Chief Coverage adjustment
Defamation risk identified Legal team Legal review
Source credibility challenge Fact-Check team Source assessment

Technical Escalation#

Scenario Escalation Target Response
All LLM providers unavailable Operations team Provider recovery
Database connectivity lost Operations team Infrastructure fix
Agent fleet unhealthy (> 50%) Operations team Fleet recovery
Message bus failure Operations team Bus recovery
Cost anomaly detected Operations team Budget review

9.5 Healing Action History#

All healing actions are logged and queryable:

typescript
interface HealingAction {
  type: HealingActionType; // restart, clear, reduce, failover, alert
  targetAgentId: AgentId;
  triggeredBy: HealthIssueType;
  timestamp: number;
  success: boolean;
  error?: string;
}

History is bounded to 1,000 entries. Available via GET /v1/health/history endpoint.


10. Configuration Reference#

10.1 Message Bus Configuration#

Setting Default Description
maxQueueSize 1,000 Maximum messages per agent queue
maxRetries 3 Delivery retry attempts
retryDelayMs 100 Base retry delay (exponential backoff)
enableDebugHistory false Enable message history (debug mode)
maxDebugHistorySize 1,000 Maximum debug history entries

10.2 State Manager Configuration#

Setting Default Description
ttlSeconds 86,400 (24h) State persistence TTL
enableCompression true Enable gzip compression
compressionThreshold 1,024 bytes Minimum size for compression
keyPrefix veritas:agents:state: Redis key prefix

10.3 Health Monitor Configuration#

Setting Default Description
checkIntervalMs 30,000 Health check interval
checkTimeoutMs 5,000 Health check response timeout
failureThreshold 3 Consecutive failures before unhealthy
latencyThresholdMs 5,000 Latency threshold for issue
errorRateThreshold 0.1 Error rate threshold (10%)
queueBacklogThreshold 100 Queue size threshold for issue
enableSelfHealing true Enable automatic healing
maxHealingActionsPerHour 10 Rate limit for healing actions

10.4 Orchestrator Configuration#

Setting Default Description
distributionStrategy load_balanced Task distribution strategy
stateSyncIntervalMs 60,000 State synchronization interval
healthCheckIntervalMs 30,000 Health check interval
scaleUpThreshold 100 Backlog size to signal scale-up
scaleDownIdleMs 300,000 (5 min) Idle time to signal scale-down

10.5 Agent Configuration Template#

typescript
interface AgentConfig {
  id: AgentId; // Unique agent identifier
  name: string; // Human-readable name
  role: AgentRole; // Agent role
  specialization?: JournalistSpec; // Journalist specialization
  capabilities: AgentCapability[]; // Capability flags
  llmConfig: LLMConfig; // LLM provider configuration
  maxConcurrentTasks: number; // Concurrent task limit
  maxQueueSize: number; // Task queue limit
  taskTimeoutMs: number; // Per-task timeout
  priority: number; // Agent priority (0-10)
  dependencies: AgentId[]; // Dependency agents
  metadata: Record<string, unknown>; // Custom metadata
}

10.6 Agent Capabilities#

Capability Description Roles
generate_content Create original content Journalist, Strategist, Social
edit_content Edit and refine content Managing Editor, EdIC
publish_content Publish content to platforms EdIC, Social
access_external_apis Call external APIs Journalist, Fact Checker
spawn_agents Create new agent instances Orchestrator
make_editorial_decisions Make editorial judgments EdIC, Managing Editor
access_user_data Access user information —
modify_config Modify system configuration —

11. API Reference#

11.1 Health Endpoints#

GET /health#

Basic service health check.

Response (200):

json
{
  "status": "ok",
  "service": "veritas-agents",
  "timestamp": "2025-01-26T12:00:00.000Z"
}

GET /ready#

Readiness probe. Checks Redis connectivity if configured.

Response (200 or 503):

json
{
  "status": "ok",
  "checks": {
    "redis": "ok"
  }
}

11.2 Agent Endpoints#

GET /v1/agents#

List all registered agents with their current status.

Response (200):

json
{
  "agents": [
    {
      "id": "editor-in-chief-1",
      "name": "Editor-in-Chief",
      "role": "editor_in_chief",
      "status": "idle",
      "health": "healthy",
      "activeTaskCount": 0,
      "queueSize": 0
    }
  ]
}

GET /v1/agents/:id#

Get detailed agent information including configuration and statistics.

GET /v1/agents/role/:role#

Get all agents with a specific role.

GET /v1/agents/:id/health#

Get detailed health information for a specific agent.

GET /v1/agents/:id/issues#

Get active health issues for a specific agent.

11.3 Task Endpoints#

POST /v1/tasks#

Create a new task.

Request Body:

json
{
  "type": "writeArticle",
  "title": "Breaking: Parliament votes on new bill",
  "description": "Write a news article about the parliamentary vote",
  "priority": 1,
  "input": {
    "content": "Background information and context...",
    "constraints": {
      "maxLength": 2000,
      "style": "news",
      "language": "en"
    }
  },
  "assignTo": "journalist-2"
}

Response (201):

json
{
  "taskId": "task_abc123",
  "status": "queued",
  "assignedTo": "journalist-2"
}

GET /v1/tasks/:id#

Get task details including status, result, and metrics.

DELETE /v1/tasks/:id#

Cancel a pending or queued task.

11.4 Orchestrator Endpoints#

GET /v1/orchestrator/stats#

Get orchestration statistics.

Response (200):

json
{
  "activeAgents": 7,
  "totalTasks": 150,
  "pendingTasks": 5,
  "runningTasks": 3,
  "completedTasks": 140,
  "failedTasks": 2,
  "backlogSize": 5,
  "distributionStrategy": "load_balanced"
}

POST /v1/orchestrator/distribute#

Force immediate backlog distribution.

11.5 Health Monitoring Endpoints#

GET /v1/health/issues#

Get all active health issues across all agents.

GET /v1/health/history#

Get healing action history.

11.6 Metrics Endpoints#

GET /v1/metrics#

JSON metrics export.

GET /v1/metrics/prometheus#

Prometheus text format (0.0.4) metrics export.


12. Metrics and Observability#

12.1 Exported Metrics#

The agent system exports 20 metrics in Prometheus format:

Agent Metrics#

Metric Type Labels
agent_tasks_total Counter agent_id, status
agent_tasks_duration_seconds Histogram agent_id, task_type
agent_tokens_consumed Counter agent_id, direction
agent_cost_usd Counter agent_id
agent_queue_size Gauge agent_id
agent_active_tasks Gauge agent_id
agent_errors_total Counter agent_id, error_type
agent_health_status Gauge agent_id

Health status gauge values: 0 = unknown, 1 = healthy, 2 = degraded, 3 = unhealthy.

Orchestrator Metrics#

Metric Type Labels
orchestrator_tasks_created Counter —
orchestrator_tasks_completed Counter —
orchestrator_tasks_failed Counter —
orchestrator_backlog_size Gauge —
orchestrator_active_agents Gauge —

Message Bus Metrics#

Metric Type Labels
message_bus_messages_sent Counter —
message_bus_messages_delivered Counter —
message_bus_messages_failed Counter —
message_bus_retries Counter —

Health Monitor Metrics#

Metric Type Labels
health_check_duration_seconds Histogram agent_id
health_issues_total Counter issue_type
healing_actions_total Counter action_type, success

12.2 Histogram Buckets#

Default buckets for duration histograms:

text
0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10 (seconds)

12.3 Monitoring Dashboards#

Recommended Grafana dashboard panels:

Panel Metric Visualization
Agent Status Overview agent_health_status Status map
Task Throughput orchestrator_tasks_completed Time series
Task Duration by Type agent_tasks_duration_seconds Heatmap
Token Consumption agent_tokens_consumed Stacked bar
Cost by Agent agent_cost_usd Pie chart
Queue Depth agent_queue_size Time series
Error Rate agent_errors_total Time series
Message Bus Health message_bus_* Time series
Health Issues health_issues_total Counter
Healing Actions healing_actions_total Bar chart

13. Operational Procedures#

13.1 Starting the Agent Service#

bash
# Development
pnpm nx serve veritas-agents

# Production (built binary)
node dist/apps/veritas/agents/main.js

# Required environment variables
export ANTHROPIC_API_KEY=sk-ant-...
export REDIS_URL=redis://localhost:6379  # Optional but recommended
export AGENTS_PORT=3006
export AGENTS_HOST=0.0.0.0

13.2 Verifying Agent Health#

bash
# Service health
curl http://localhost:3006/health

# Readiness (includes Redis check)
curl http://localhost:3006/ready

# List all agents with status
curl http://localhost:3006/v1/agents

# Check specific agent health
curl http://localhost:3006/v1/agents/editor-in-chief-1/health

# Check active issues
curl http://localhost:3006/v1/health/issues

# View metrics
curl http://localhost:3006/v1/metrics

13.3 Creating and Monitoring Tasks#

bash
# Create a task
curl -X POST http://localhost:3006/v1/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "type": "writeArticle",
    "title": "Market Update",
    "priority": 2,
    "input": {
      "content": "Ghana Stock Exchange closes higher...",
      "constraints": {"language": "en", "maxLength": 1500}
    }
  }'

# Check task status
curl http://localhost:3006/v1/tasks/{taskId}

# Cancel a task
curl -X DELETE http://localhost:3006/v1/tasks/{taskId}

# Check orchestrator stats
curl http://localhost:3006/v1/orchestrator/stats

# Force task distribution
curl -X POST http://localhost:3006/v1/orchestrator/distribute

13.4 Troubleshooting Agent Issues#

Agent Stuck in Busy State#

bash
# Check agent status
curl http://localhost:3006/v1/agents/{agentId}

# Check active issues
curl http://localhost:3006/v1/agents/{agentId}/issues

# Check healing history
curl http://localhost:3006/v1/health/history

If the agent is stuck, the health monitor will detect it as unresponsive after 3 consecutive check failures and attempt a restart.

LLM Provider Issues#

bash
# Check if provider is responding
curl https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-3-5-haiku-20241022","max_tokens":10,"messages":[{"role":"user","content":"test"}]}'

The system will automatically attempt failover to backup providers if the primary provider becomes unavailable.

High Cost Alert#

Monitor the agent_cost_usd metric. If costs exceed expected thresholds:

  1. Check token consumption per agent via agent_tokens_consumed
  2. Review task types and frequency
  3. Consider switching high-volume agents to cheaper models (Haiku instead of Sonnet)
  4. Adjust temperature settings (lower temperature = fewer retries)

13.5 Scaling Agents#

Adding More Agents#

Additional agents can be registered programmatically by adding configurations to the agent registry. Each new agent needs:

  • Unique agent ID
  • Role and specialization
  • LLM configuration
  • Concurrency limits
  • Priority level

Scaling Signals#

The orchestrator emits scaling signals:

  • Scale Up: When backlog exceeds scaleUpThreshold (default: 100)
  • Scale Down: When agents are idle for scaleDownIdleMs (default: 5 minutes)

These signals can be consumed by external autoscaling systems (Kubernetes HPA, AWS Auto Scaling) to dynamically adjust the agent fleet.