Technical documentation for the Veritas multi-agent AI newsroom system, covering agent roles, interaction patterns, task management, and escalation procedures.
Table of Contents#
- System Overview
- Agent Roles and Responsibilities
- Agent Lifecycle
- Task Management
- Agent Interaction Patterns
- LLM Integration
- Memory and Context
- Health Monitoring
- Escalation Procedures
- Configuration Reference
- API Reference
- Metrics and Observability
- Operational Procedures
1. System Overview#
1.1 Architecture#
The Veritas AI Agent System is a multi-agent orchestration platform that powers an autonomous AI newsroom. It coordinates specialized AI agents to perform journalistic tasks including content creation, editing, fact-checking, and distribution.
┌─────────────────────────────────────────────────────────┐
│ HTTP API (Hono) │
│ Port 3006 (AGENTS_PORT) │
├─────────────────────────────────────────────────────────┤
│ │
│ ┌───────────────────────────────────────────────────┐ │
│ │ Orchestrator │ │
│ │ - Task routing and distribution │ │
│ │ - Agent registry │ │
│ │ - Backlog management │ │
│ │ - Load balancing │ │
│ └───────────────────┬───────────────────────────────┘ │
│ │ │
│ ┌───────────────────┴───────────────────────────────┐ │
│ │ Message Bus │ │
│ │ - Local (in-memory) or Redis (distributed) │ │
│ │ - Pub/Sub with retry logic │ │
│ │ - Per-agent message queues │ │
│ └───────────────────┬───────────────────────────────┘ │
│ │ │
│ ┌───────┬───────┬───┴───┬───────┬───────┬───────────┐ │
│ │ EdIC │ MngEd │ Journ │ FactCk│ CtStr │ SocMedia │ │
│ │ │ │ x2 │ │ │ │ │
│ └───────┴───────┴───────┴───────┴───────┴───────────┘ │
│ │
│ ┌───────────────────────────────────────────────────┐ │
│ │ Health Monitor │ State Manager │ Metrics │ │
│ └───────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────┘
1.2 Service Configuration#
| Setting | Environment Variable | Default |
|---|---|---|
| HTTP Port | AGENTS_PORT |
3006 |
| HTTP Host | AGENTS_HOST |
0.0.0.0 |
| Redis URL | REDIS_URL |
— |
| Anthropic API Key | ANTHROPIC_API_KEY |
— |
| OpenAI API Key | OPENAI_API_KEY |
— |
| Health Check Interval | — | 30,000 ms |
| Metrics Collection | — | 60,000 ms |
| Log Level | — | info |
1.3 Default Agent Fleet#
The system starts with 7 pre-registered agents:
| Agent ID | Role | Specialization | Priority | Max Concurrent |
|---|---|---|---|---|
editor-in-chief-1 |
Editor-in-Chief | — | 10 | 5 |
managing-editor-1 |
Managing Editor | — | 9 | 5 |
journalist-1 |
Journalist | Investigative | 7 | 3 |
journalist-2 |
Journalist | Political | 7 | 3 |
fact-checker-1 |
Fact Checker | — | 8 | 3 |
content-strategist-1 |
Content Strategist | — | 6 | 3 |
social-media-manager-1 |
Social Media Manager | — | 5 | 5 |
2. Agent Roles and Responsibilities#
2.1 Editor-in-Chief#
Role: editor_in_chief
The highest editorial authority in the agent system. Responsible for final editorial decisions and publication approval.
Capabilities:
- Make editorial decisions
- Publish content
- Edit content
- Review content
- Assign stories
Task Types Handled:
reviewContent— Final editorial review before publicationassignStory— Story assignment to journalist agentsprioritizeNews— News prioritization decisions
LLM Configuration:
- Model: claude-sonnet-4-20250514
- Temperature: 0.3 (analytical, conservative)
- Max Tokens: 4,096
Human Oversight: Required for critical topics (national security, election coverage, defamation risk).
2.2 Managing Editor#
Role: managing_editor
Day-to-day operations manager. Handles content editing, scheduling, and operational decisions.
Capabilities:
- Edit content
- Make editorial decisions
Task Types Handled:
editArticle— Content editing and refinement- Task scheduling and workflow coordination
LLM Configuration:
- Model: claude-sonnet-4-20250514
- Temperature: 0.4
- Max Tokens: 4,096
Human Oversight: Weekly editorial review.
2.3 Journalist#
Role: journalist
Content creator with optional specialization. The primary content-producing agent.
Capabilities:
- Generate content
- Access external APIs
Specializations:
| Specialization | Focus Area | Special Considerations |
|---|---|---|
political |
Government, parliament, parties | Balance monitoring required |
business |
Economy, markets, corporate | Financial data verification |
sports |
Athletics, football, events | Score/result verification |
entertainment |
Arts, culture, celebrity | Privacy considerations |
regional |
Local government, development | Local source verification |
investigative |
In-depth, exposés, accountability | Multiple source requirement |
breaking_news |
Urgent, time-sensitive events | Rapid verification protocol |
general |
All other topics | Standard editorial procedures |
Task Types Handled:
writeArticle— Original article creationtranslateContent— Content translationsummarizeContent— Content summarizationgenerateHeadline— Headline generationgenerateVideoScript— Video script writingresearchTopic— Topic research and background
LLM Configuration:
- Model: claude-sonnet-4-20250514
- Temperature: 0.7 (creative, varied output)
- Max Tokens: 8,192
Human Oversight: Pre-publication editorial review for all content.
2.4 Fact Checker#
Role: fact_checker
Verification and evidence evaluation specialist. Ensures accuracy of claims and statements.
Capabilities:
- Access external APIs (search engines, fact-check databases)
Task Types Handled:
factCheck— Full fact-check of article or statementverifyClaim— Verification of individual claimsanalyzeSource— Source credibility analysis
LLM Configuration:
- Model: claude-sonnet-4-20250514
- Temperature: 0.2 (strict, deterministic)
- Max Tokens: 4,096
Human Oversight: Critical verdict review for high-impact claims.
2.5 Content Strategist#
Role: content_strategist
Cross-platform content planning and audience analysis.
Capabilities:
- Generate content
Task Types Handled:
- Content planning and scheduling
- Audience analysis
- Cross-platform distribution strategy
LLM Configuration:
- Model: claude-sonnet-4-20250514
- Temperature: 0.5
- Max Tokens: 4,096
Human Oversight: Strategy review on a periodic basis.
2.6 Social Media Manager#
Role: social_media_manager
Social media content creation and platform distribution.
Capabilities:
- Generate content
- Publish content
Task Types Handled:
createSocialPost— Social media post creation- Platform-specific content adaptation
- Engagement monitoring
LLM Configuration:
- Model: claude-3-5-haiku
- Temperature: 0.8 (engaging, creative)
- Max Tokens: 2,048
Human Oversight: Post review before distribution.
2.7 Orchestrator (System Agent)#
Role: orchestrator
Central task coordinator and workflow manager. Not a content-producing agent.
Responsibilities:
- Task routing based on configurable rules
- Agent registry management
- Backlog monitoring and distribution
- Load balancing across agents
- Scaling signals (up/down)
LLM Configuration:
- Model: claude-3-5-haiku
- Temperature: 0.1 (deterministic decisions)
- Max Tokens: 1,024
2.8 Monitor (System Agent)#
Role: monitor
Health and status monitoring agent.
Responsibilities:
- Health check execution
- Issue detection and alerting
- Self-healing action coordination
- Metrics collection
LLM Configuration:
- Model: claude-3-5-haiku
- Temperature: 0.1 (deterministic)
- Max Tokens: 1,024
3. Agent Lifecycle#
3.1 Lifecycle States#
┌─────────────┐
│ initializing │
└──────┬──────┘
│ onInitialize, onReady
▼
┌─────────┐
┌──────→│ idle │◄──────────┐
│ └────┬────┘ │
│ │ task received │ task complete
│ ▼ │
│ ┌─────────┐ │
│ │ busy │───────────┘
│ └────┬────┘
│ │ pause()
│ ▼
│ ┌─────────┐
│ │ paused │
│ └────┬────┘
│ │ resume()
└────────────┘
│ error
▼
┌──────────┐
│ error │
└────┬─────┘
│ stop()
▼
┌───────────────┐
│ shutting_down │
└───────┬───────┘
│ cleanup complete
▼
┌─────────┐
│ stopped │
└─────────┘
3.2 Initialization Sequence#
When an agent starts (BaseAgent.start()):
- Load saved state from persistence (if available within TTL)
- Call
onInitializelifecycle hook - Restore previous state: task queue, context, statistics
- Call agent-specific
initialize()method - Subscribe to the message bus
- Start task processing loop (100 ms interval)
- Start health check loop (30 s interval)
- Set status to
idleand health tohealthy - Call
onReadylifecycle hook - Emit
agentStartedevent to the message bus
3.3 Task Processing Loop#
The processing loop runs every 100 ms:
loop:
if paused or shutting_down → skip
if active_tasks >= maxConcurrentTasks → skip
if queue is empty → set status to idle, skip
task = queue.dequeue()
set status to busy
set task.status to running
record start timestamp
try:
result = await processTask(task) // abstract method
update statistics (completed, duration, tokens, cost)
broadcast taskComplete message
catch error:
if retries remaining → re-queue with backoff
else → mark task failed, broadcast taskFailed
finally:
remove from active tasks
if queue empty → set status to idle
3.4 Graceful Shutdown#
When an agent stops (BaseAgent.stop()):
- Set
isShuttingDownflag - Set status to
shutting_down - Call
onShutdownlifecycle hook - Stop processing loops (clear intervals)
- Wait up to 30 seconds for active tasks to complete
- Persist full state to storage (queue, context, statistics)
- Unsubscribe from the message bus
- Call agent-specific
cleanup()method - Set status to
stopped - Emit
agentStoppedevent with uptime duration
3.5 Lifecycle Hooks#
Agents can implement the following hooks for custom behavior:
| Hook | Trigger | Use Case |
|---|---|---|
onInitialize() |
Before startup | Resource allocation |
onReady() |
After agent is ready | Announce availability |
onTaskReceived() |
When a task is assigned | Task-specific setup |
onTaskCompleted() |
After task completes | Cleanup, logging |
onPause() |
When agent is paused | Save intermediate state |
onResume() |
When agent resumes | Restore intermediate state |
onShutdown() |
During shutdown | Resource cleanup |
onError() |
On error | Error logging, recovery |
4. Task Management#
4.1 Task Types#
The system supports 18 task types across four categories:
Content Creation#
| Task Type | Description | Default Agent Role |
|---|---|---|
writeArticle |
Write an original article | Journalist |
editArticle |
Edit and refine content | Managing Editor |
translateContent |
Translate to another language | Journalist |
summarizeContent |
Create a summary | Journalist |
generateHeadline |
Generate article headline | Journalist |
generateVideoScript |
Write a video script | Journalist |
createSocialPost |
Create social media content | Social Media Manager |
Verification#
| Task Type | Description | Default Agent Role |
|---|---|---|
factCheck |
Verify article claims | Fact Checker |
verifyClaim |
Verify a specific claim | Fact Checker |
analyzeSource |
Assess source credibility | Fact Checker |
Editorial#
| Task Type | Description | Default Agent Role |
|---|---|---|
reviewContent |
Final editorial review | Editor-in-Chief |
assignStory |
Assign story to journalist | Editor-in-Chief |
prioritizeNews |
Prioritize news items | Editor-in-Chief |
researchTopic |
Research a topic | Journalist |
System#
| Task Type | Description | Default Agent Role |
|---|---|---|
healthCheck |
System health check | Monitor |
syncState |
Synchronize agent state | Orchestrator |
cleanup |
Cleanup stale data | Orchestrator |
custom |
Custom task type | Per routing rules |
4.2 Task Priority Levels#
| Priority | Value | Description | Use Case |
|---|---|---|---|
critical |
0 | Immediate processing | Breaking news, corrections |
high |
1 | Processed before normal tasks | Important stories, fact-checks |
normal |
2 | Standard processing order | Regular news content |
low |
3 | Processed when capacity allows | Feature articles, analysis |
background |
4 | Lowest priority | Batch processing, cleanup |
4.3 Task Status Flow#
pending ──→ queued ──→ running ──→ completed
│
├──→ failed (after max retries)
├──→ cancelled
└──→ timed_out
4.4 Task Structure#
interface Task {
id: TaskId; // Unique branded identifier
type: TaskType; // One of 18 task types
title: string; // Human-readable title
description: string; // Detailed description
priority: TaskPriority; // 0-4 (critical to background)
createdBy: AgentId; // Agent that created the task
assignedTo?: AgentId; // Assigned agent
status: TaskStatus; // Current status
timestamps: {
created: number;
started?: number;
completed?: number;
updated: number;
};
input: {
content: string; // Primary content to process
context?: Record<string, unknown>;
references?: string[]; // Source URLs/documents
constraints?: {
maxLength?: number;
style?: string;
tone?: string;
language?: string;
deadline?: number; // Unix timestamp
requiredSourcesCount?: number;
};
};
parentTaskId?: TaskId; // For task hierarchies
childTaskIds: TaskId[]; // Sub-tasks
dependencies: TaskId[]; // Must complete first
deadline?: number;
timeoutMs?: number;
retryConfig: {
maxAttempts: number; // Default: 3
backoffMs: number; // Default: 1000ms
currentAttempt: number;
};
metadata: Record<string, unknown>;
}
4.5 Task Routing Rules#
The orchestrator uses 14 default routing rules to assign tasks to appropriate agents:
| Task Type | Target Role | Priority | Constraints |
|---|---|---|---|
writeArticle |
journalist | normal | generate_content capability |
editArticle |
managing_editor | normal | edit_content capability |
factCheck |
fact_checker | high | access_external_apis |
verifyClaim |
fact_checker | high | access_external_apis |
analyzeSource |
fact_checker | normal | access_external_apis |
reviewContent |
editor_in_chief | high | make_editorial_decisions |
assignStory |
editor_in_chief | high | make_editorial_decisions |
prioritizeNews |
editor_in_chief | normal | make_editorial_decisions |
translateContent |
journalist | normal | generate_content |
summarizeContent |
journalist | normal | generate_content |
generateHeadline |
journalist | normal | generate_content |
generateVideoScript |
journalist | normal | generate_content |
createSocialPost |
social_media_manager | normal | publish_content |
researchTopic |
journalist | low | access_external_apis |
4.6 Task Distribution Strategies#
The orchestrator supports three distribution strategies:
Round Robin#
Assigns tasks to the agent with the oldest last activity timestamp. Ensures equal distribution across agents of the same role.
Load Balanced (Default)#
Assigns tasks to the agent with the lowest current task count. Optimizes for throughput by avoiding overloaded agents.
Priority Based#
Assigns tasks to the agent with the highest priority level. Ensures the most capable agents handle tasks first.
4.7 Task Result Structure#
interface TaskResult<T = unknown> {
taskId: TaskId;
success: boolean;
data?: T;
error?: string;
metrics: {
executionTimeMs: number;
llmLatencyMs: number;
inputTokens: number;
outputTokens: number;
costUsd: number;
llmCalls: number;
retries: number;
};
completedAt: number;
}
5. Agent Interaction Patterns#
5.1 Message Bus Architecture#
Agents communicate through a publish-subscribe message bus with two implementations:
| Implementation | Backing Store | Use Case | Performance |
|---|---|---|---|
| Local | In-memory | Testing, single-node | Sub-millisecond |
| Redis | Redis Pub/Sub | Production, distributed | Low-latency |
The Redis implementation uses the channel veritas:agents:messages.
5.2 Message Types#
Task Messages#
| Type | Description | Sender | Receiver |
|---|---|---|---|
taskAssign |
Assign a task to an agent | Orchestrator | Target agent |
taskSubmit |
Submit a task for processing | Any agent | Orchestrator |
taskUpdate |
Update task progress | Processing agent | Orchestrator |
taskComplete |
Report task completion | Processing agent | Orchestrator |
taskFailed |
Report task failure | Processing agent | Orchestrator |
taskCancel |
Cancel a task | Any agent | Processing agent |
Agent Lifecycle Messages#
| Type | Description | Sender | Receiver |
|---|---|---|---|
agentStart |
Agent has started | Agent | Broadcast |
agentStop |
Agent is stopping | Agent | Broadcast |
agentPause |
Agent is paused | Agent | Broadcast |
agentResume |
Agent has resumed | Agent | Broadcast |
Status Messages#
| Type | Description | Sender | Receiver |
|---|---|---|---|
statusRequest |
Request agent status | Monitor | Target agent |
statusResponse |
Respond with status | Agent | Monitor |
capabilityRequest |
Query agent capabilities | Orchestrator | Target agent |
capabilityResponse |
Respond with capabilities | Agent | Orchestrator |
Event Messages#
| Type | Description | Sender | Receiver |
|---|---|---|---|
eventBroadcast |
General event notification | Any agent | Broadcast |
alert |
Alert notification | Monitor | Broadcast |
5.3 Message Structure#
interface Message<T = unknown> {
id: MessageId; // Unique branded identifier
type: MessageType; // One of the types above
source: AgentId; // Sending agent
destination: AgentId | 'broadcast';
payload: T; // Type-specific data
correlationId?: string; // Request-response correlation
replyTo?: AgentId; // Reply routing
priority: number; // 0-4 (0 = highest)
ttl?: number; // Time-to-live in milliseconds
timestamp: number; // Creation timestamp
headers: Record<string, string>;
}
5.4 Message Delivery Guarantees#
- Per-agent queue: Each agent has a message queue (max 1,000 messages)
- Retry logic: Up to 3 attempts with exponential backoff (100 ms base)
- TTL enforcement: Expired messages are dropped before delivery
- Broadcast: Delivered to all subscribed agents except the sender
- Order: Messages within a queue are processed in order
5.5 Event System#
The agent system emits events for observability and cross-agent coordination:
| Event Category | Events |
|---|---|
| Agent | agentStarted, agentStopped, agentError, agentHealthChanged |
| Task | taskCreated, taskAssigned, taskStarted, taskProgress, taskCompleted, taskFailed |
| Content | contentCreated, contentUpdated, contentPublished |
| News | breakingNews, trendingTopic |
| Health | alertTriggered, alertResolved |
5.6 Common Interaction Patterns#
Content Creation Workflow#
1. Orchestrator creates writeArticle task
2. Orchestrator routes to journalist agent (load-balanced)
3. Journalist receives taskAssign message
4. Journalist generates article using LLM
5. Journalist submits taskComplete with article content
6. Orchestrator creates factCheck task → routes to fact-checker
7. Fact-checker verifies claims, submits result
8. Orchestrator creates reviewContent task → routes to editor-in-chief
9. Editor-in-chief reviews, approves or requests edits
10. If approved → content published via contentPublished event
Breaking News Escalation#
1. Monitor detects breaking news event
2. Monitor broadcasts breakingNews event (critical priority)
3. Orchestrator creates writeArticle task (critical priority)
4. Journalist-2 (breaking_news specialization) receives assignment
5. Rapid content creation with minimal verification
6. Parallel factCheck task created (high priority)
7. Managing editor review (bypassing full editorial chain)
8. Publish with "developing story" label
9. Follow-up verification and updates scheduled
Fact-Check Chain#
1. Article submitted for review
2. Orchestrator creates factCheck task
3. Fact-checker extracts individual claims
4. For each claim: creates verifyClaim sub-task
5. Sub-tasks processed with evidence retrieval
6. Results aggregated into fact-check report
7. Report attached to article via taskComplete
8. Editor-in-chief reviews fact-check results
6. LLM Integration#
6.1 Supported Providers#
| Provider | SDK | Models Available |
|---|---|---|
| Anthropic | @anthropic-ai/sdk |
Claude Opus 4, Claude Sonnet 4, Claude 3.5 Haiku |
| OpenAI | openai |
GPT-4o, GPT-4o mini |
| Local | Veritas adapter | Supported through provider injection / fallback |
6.2 LLM Client Interface#
interface LLMClient {
complete(
messages: ConversationMessage[],
systemPrompt?: string
): Promise<LLMResponse>;
completeStream(
messages: ConversationMessage[],
systemPrompt?: string,
callbacks?: StreamCallbacks
): Promise<LLMResponse>;
isAvailable(): Promise<boolean>;
getConfig(): LLMConfig;
estimateTokens(text: string): number;
calculateCost(inputTokens: number, outputTokens: number): number;
}
6.3 Default LLM Configuration by Role#
| Role | Model | Temperature | Max Tokens | Reasoning |
|---|---|---|---|---|
| Editor-in-Chief | claude-sonnet-4-20250514 | 0.3 | 4,096 | Analytical, conservative |
| Managing Editor | claude-sonnet-4-20250514 | 0.4 | 4,096 | Balanced editing |
| Journalist | claude-sonnet-4-20250514 | 0.7 | 8,192 | Creative, varied output |
| Fact Checker | claude-sonnet-4-20250514 | 0.2 | 4,096 | Strict, deterministic |
| Content Strategist | claude-sonnet-4-20250514 | 0.5 | 4,096 | Balanced analysis |
| Social Media Manager | claude-3-5-haiku | 0.8 | 2,048 | Engaging, creative |
| Orchestrator | claude-3-5-haiku | 0.1 | 1,024 | Deterministic routing |
| Monitor | claude-3-5-haiku | 0.1 | 1,024 | Deterministic monitoring |
6.4 Cost Model#
Token pricing per 1 million tokens:
| Model | Input Cost | Output Cost |
|---|---|---|
| Claude Opus 4 | $15.00 | $75.00 |
| Claude Sonnet 4 | $3.00 | $15.00 |
| Claude 3.5 Haiku | $0.80 | $4.00 |
| GPT-4o | $2.50 | $10.00 |
| GPT-4o mini | $0.15 | $0.60 |
All API calls track token consumption and cost. Per-agent cost statistics are available via the metrics API.
6.5 Retry Configuration#
- Max Attempts: 3
- Backoff: Exponential, 1,000 ms base, 10,000 ms maximum
- Non-Retryable Errors: 401 (Unauthorized), 403 (Forbidden), 400 (Bad Request)
- Retryable Errors: All other errors (rate limits, server errors, timeouts)
6.6 Token Estimation#
Rough estimation formula: ~4 characters per token
Used for pre-flight checks to avoid exceeding model context limits.
7. Memory and Context#
7.1 Context Structure#
Each agent maintains an execution context with four components:
interface AgentContext {
conversationHistory: ConversationMessage[];
workingMemory: Map<string, unknown>;
longTermMemoryRefs: string[];
preferences: Record<string, unknown>;
}
7.2 Conversation History#
- Purpose: Multi-turn context for LLM interactions
- Structure: Ordered list of messages with role (
user,assistant,system), content, and timestamp - Bounded: Limited size to prevent context overflow
- Passed to LLM: Included in every LLM completion request
- Persistence: Saved on shutdown, restored on startup
7.3 Working Memory#
- Purpose: Transient, session-scoped intermediate data
- Structure: Key-value map (
Map<string, unknown>) - Lifetime: Cleared on agent restart
- Use Cases: Intermediate research results, draft content, analysis state
7.4 Long-Term Memory References#
- Purpose: References to external persistent storage
- Structure: Array of string identifiers (database IDs, document references)
- Lifetime: Persisted across sessions
- Use Cases: Article drafts, research notes, source databases
7.5 Preferences#
- Purpose: Learned agent preferences and style settings
- Structure: Key-value record
- Lifetime: Persisted across sessions
- Use Cases: Writing tone, formatting preferences, topic expertise weights
7.6 State Persistence#
Agent state is persisted through a state manager with two implementations:
Local State Manager (In-Memory)#
- Storage: In-process
Map - Version tracking with incrementing counter
- TTL enforcement (default: 24 hours)
- No durability — lost on process restart
Redis State Manager (Distributed)#
- Key format:
veritas:agents:state:{agentId} - Version tracking via separate key
- Optional gzip compression (threshold: 1 KB)
- TTL via Redis EXPIRE (default: 24 hours)
- Agent index via Redis SET for enumeration
- Durable across process restarts
7.7 State Restoration#
On agent startup:
- Check for saved state in persistence layer
- If state exists and is within TTL:
- Restore task queue (pending tasks only)
- Restore conversation history
- Restore preferences
- Restore statistics
- If state is expired or absent:
- Start with clean state
- Active tasks are not restored — they are re-queued from pending state
8. Health Monitoring#
8.1 Health Status Levels#
| Status | Meaning | Action Required |
|---|---|---|
healthy |
All systems normal | None |
degraded |
Functional but with issues | Monitor, investigate |
unhealthy |
Critical issues detected | Immediate intervention |
unknown |
Not yet checked | Wait for first check |
8.2 Health Check Process#
Health checks run every 30 seconds (configurable):
1. Send statusRequest to agent
2. Wait for response (5-second timeout)
3. If response received:
- Reset consecutive failure count
- Analyze metrics for issues
4. If timeout:
- Increment consecutive failure count
- If count >= 3 → mark unhealthy
5. Emit agentHealthChanged event if status changed
8.3 Health Metrics#
Each agent reports the following metrics:
| Metric | Type | Threshold (Warning) | Threshold (Issue) |
|---|---|---|---|
| CPU Usage | Percent | — | — |
| Memory Usage | Percent | — | > 90% |
| Task Queue Size | Count | — | > 100 |
| Average Response Time | Ms | — | > 5,000 ms |
| Error Rate | Percent | — | > 10% |
8.4 Health Component Tracking#
Each agent tracks health for individual components:
| Component | Description |
|---|---|
llm |
LLM provider connectivity |
message_bus |
Message bus connectivity |
state_manager |
State persistence connectivity |
task_processor |
Task processing loop health |
8.5 Issue Detection#
8 issue types are automatically detected:
| Issue Type | Trigger | Severity | Auto-Heal Action |
|---|---|---|---|
agentUnresponsive |
3+ consecutive check failures | critical | restartAgent |
highLatency |
Response time > 5s | high | alertOperator |
highErrorRate |
Error rate > 10% | high | alertOperator |
queueBacklog |
Queue size > 100 | medium | reduceLoad |
memoryPressure |
Memory usage > 90% | high | clearQueue |
llmUnavailable |
LLM component unhealthy | critical | failover |
stateCorruption |
State integrity check failed | critical | restartAgent |
dependencyFailure |
Dependency agent failed | high | alertOperator |
9. Escalation Procedures#
9.1 Automatic Self-Healing#
When self-healing is enabled (default: true), the system can automatically respond to detected issues:
| Healing Action | Description | Rate Limit |
|---|---|---|
restartAgent |
Stop and restart the affected agent | 10 per hour |
clearQueue |
Clear the agent's task queue | 10 per hour |
reduceLoad |
Reduce concurrency or pause task intake | 10 per hour |
failover |
Switch to backup LLM provider | 10 per hour |
alertOperator |
Send alert to human operator | 10 per hour |
9.2 Self-Healing Flow#
Issue Detected
│
├─→ Is self-healing enabled?
│ No → Emit alert event only
│ Yes ↓
│
├─→ Is rate limit exceeded? (10 actions/hour)
│ Yes → Skip healing, log warning
│ No ↓
│
├─→ Map issue type to healing action
│
├─→ Is handler registered for this action?
│ No → Log warning, skip
│ Yes ↓
│
├─→ Execute healing handler
│
├─→ Record result (success/failure, timestamp)
│
└─→ Emit healing action event
9.3 Escalation Matrix#
| Issue | Severity | Self-Heal | Human Escalation |
|---|---|---|---|
| Agent unresponsive | Critical | Restart agent | If restart fails |
| LLM provider down | Critical | Failover provider | If all providers down |
| State corruption | Critical | Restart agent | If restart fails |
| High latency (> 5s) | High | Alert operator | Investigate cause |
| High error rate (> 10%) | High | Alert operator | Investigate cause |
| Queue backlog (> 100) | Medium | Reduce load | Scale up agents |
| Memory pressure (> 90%) | High | Clear queue | Increase resources |
| Dependency failure | High | Alert operator | Fix dependency |
9.4 Human-in-the-Loop Escalation#
Certain scenarios require human intervention:
Editorial Escalation#
| Scenario | Escalation Target | Response |
|---|---|---|
| Content flagged for bias | Managing Editor | Editorial review |
| Fact-check verdict disputed | Editor-in-Chief | Final verdict |
| NMC compliance violation | Editor-in-Chief | Compliance remediation |
| Election coverage imbalance | Editor-in-Chief | Coverage adjustment |
| Defamation risk identified | Legal team | Legal review |
| Source credibility challenge | Fact-Check team | Source assessment |
Technical Escalation#
| Scenario | Escalation Target | Response |
|---|---|---|
| All LLM providers unavailable | Operations team | Provider recovery |
| Database connectivity lost | Operations team | Infrastructure fix |
| Agent fleet unhealthy (> 50%) | Operations team | Fleet recovery |
| Message bus failure | Operations team | Bus recovery |
| Cost anomaly detected | Operations team | Budget review |
9.5 Healing Action History#
All healing actions are logged and queryable:
interface HealingAction {
type: HealingActionType; // restart, clear, reduce, failover, alert
targetAgentId: AgentId;
triggeredBy: HealthIssueType;
timestamp: number;
success: boolean;
error?: string;
}
History is bounded to 1,000 entries. Available via GET /v1/health/history
endpoint.
10. Configuration Reference#
10.1 Message Bus Configuration#
| Setting | Default | Description |
|---|---|---|
maxQueueSize |
1,000 | Maximum messages per agent queue |
maxRetries |
3 | Delivery retry attempts |
retryDelayMs |
100 | Base retry delay (exponential backoff) |
enableDebugHistory |
false | Enable message history (debug mode) |
maxDebugHistorySize |
1,000 | Maximum debug history entries |
10.2 State Manager Configuration#
| Setting | Default | Description |
|---|---|---|
ttlSeconds |
86,400 (24h) | State persistence TTL |
enableCompression |
true | Enable gzip compression |
compressionThreshold |
1,024 bytes | Minimum size for compression |
keyPrefix |
veritas:agents:state: |
Redis key prefix |
10.3 Health Monitor Configuration#
| Setting | Default | Description |
|---|---|---|
checkIntervalMs |
30,000 | Health check interval |
checkTimeoutMs |
5,000 | Health check response timeout |
failureThreshold |
3 | Consecutive failures before unhealthy |
latencyThresholdMs |
5,000 | Latency threshold for issue |
errorRateThreshold |
0.1 | Error rate threshold (10%) |
queueBacklogThreshold |
100 | Queue size threshold for issue |
enableSelfHealing |
true | Enable automatic healing |
maxHealingActionsPerHour |
10 | Rate limit for healing actions |
10.4 Orchestrator Configuration#
| Setting | Default | Description |
|---|---|---|
distributionStrategy |
load_balanced |
Task distribution strategy |
stateSyncIntervalMs |
60,000 | State synchronization interval |
healthCheckIntervalMs |
30,000 | Health check interval |
scaleUpThreshold |
100 | Backlog size to signal scale-up |
scaleDownIdleMs |
300,000 (5 min) | Idle time to signal scale-down |
10.5 Agent Configuration Template#
interface AgentConfig {
id: AgentId; // Unique agent identifier
name: string; // Human-readable name
role: AgentRole; // Agent role
specialization?: JournalistSpec; // Journalist specialization
capabilities: AgentCapability[]; // Capability flags
llmConfig: LLMConfig; // LLM provider configuration
maxConcurrentTasks: number; // Concurrent task limit
maxQueueSize: number; // Task queue limit
taskTimeoutMs: number; // Per-task timeout
priority: number; // Agent priority (0-10)
dependencies: AgentId[]; // Dependency agents
metadata: Record<string, unknown>; // Custom metadata
}
10.6 Agent Capabilities#
| Capability | Description | Roles |
|---|---|---|
generate_content |
Create original content | Journalist, Strategist, Social |
edit_content |
Edit and refine content | Managing Editor, EdIC |
publish_content |
Publish content to platforms | EdIC, Social |
access_external_apis |
Call external APIs | Journalist, Fact Checker |
spawn_agents |
Create new agent instances | Orchestrator |
make_editorial_decisions |
Make editorial judgments | EdIC, Managing Editor |
access_user_data |
Access user information | — |
modify_config |
Modify system configuration | — |
11. API Reference#
11.1 Health Endpoints#
GET /health#
Basic service health check.
Response (200):
{
"status": "ok",
"service": "veritas-agents",
"timestamp": "2025-01-26T12:00:00.000Z"
}
GET /ready#
Readiness probe. Checks Redis connectivity if configured.
Response (200 or 503):
{
"status": "ok",
"checks": {
"redis": "ok"
}
}
11.2 Agent Endpoints#
GET /v1/agents#
List all registered agents with their current status.
Response (200):
{
"agents": [
{
"id": "editor-in-chief-1",
"name": "Editor-in-Chief",
"role": "editor_in_chief",
"status": "idle",
"health": "healthy",
"activeTaskCount": 0,
"queueSize": 0
}
]
}
GET /v1/agents/:id#
Get detailed agent information including configuration and statistics.
GET /v1/agents/role/:role#
Get all agents with a specific role.
GET /v1/agents/:id/health#
Get detailed health information for a specific agent.
GET /v1/agents/:id/issues#
Get active health issues for a specific agent.
11.3 Task Endpoints#
POST /v1/tasks#
Create a new task.
Request Body:
{
"type": "writeArticle",
"title": "Breaking: Parliament votes on new bill",
"description": "Write a news article about the parliamentary vote",
"priority": 1,
"input": {
"content": "Background information and context...",
"constraints": {
"maxLength": 2000,
"style": "news",
"language": "en"
}
},
"assignTo": "journalist-2"
}
Response (201):
{
"taskId": "task_abc123",
"status": "queued",
"assignedTo": "journalist-2"
}
GET /v1/tasks/:id#
Get task details including status, result, and metrics.
DELETE /v1/tasks/:id#
Cancel a pending or queued task.
11.4 Orchestrator Endpoints#
GET /v1/orchestrator/stats#
Get orchestration statistics.
Response (200):
{
"activeAgents": 7,
"totalTasks": 150,
"pendingTasks": 5,
"runningTasks": 3,
"completedTasks": 140,
"failedTasks": 2,
"backlogSize": 5,
"distributionStrategy": "load_balanced"
}
POST /v1/orchestrator/distribute#
Force immediate backlog distribution.
11.5 Health Monitoring Endpoints#
GET /v1/health/issues#
Get all active health issues across all agents.
GET /v1/health/history#
Get healing action history.
11.6 Metrics Endpoints#
GET /v1/metrics#
JSON metrics export.
GET /v1/metrics/prometheus#
Prometheus text format (0.0.4) metrics export.
12. Metrics and Observability#
12.1 Exported Metrics#
The agent system exports 20 metrics in Prometheus format:
Agent Metrics#
| Metric | Type | Labels |
|---|---|---|
agent_tasks_total |
Counter | agent_id, status |
agent_tasks_duration_seconds |
Histogram | agent_id, task_type |
agent_tokens_consumed |
Counter | agent_id, direction |
agent_cost_usd |
Counter | agent_id |
agent_queue_size |
Gauge | agent_id |
agent_active_tasks |
Gauge | agent_id |
agent_errors_total |
Counter | agent_id, error_type |
agent_health_status |
Gauge | agent_id |
Health status gauge values: 0 = unknown, 1 = healthy, 2 = degraded, 3 = unhealthy.
Orchestrator Metrics#
| Metric | Type | Labels |
|---|---|---|
orchestrator_tasks_created |
Counter | — |
orchestrator_tasks_completed |
Counter | — |
orchestrator_tasks_failed |
Counter | — |
orchestrator_backlog_size |
Gauge | — |
orchestrator_active_agents |
Gauge | — |
Message Bus Metrics#
| Metric | Type | Labels |
|---|---|---|
message_bus_messages_sent |
Counter | — |
message_bus_messages_delivered |
Counter | — |
message_bus_messages_failed |
Counter | — |
message_bus_retries |
Counter | — |
Health Monitor Metrics#
| Metric | Type | Labels |
|---|---|---|
health_check_duration_seconds |
Histogram | agent_id |
health_issues_total |
Counter | issue_type |
healing_actions_total |
Counter | action_type, success |
12.2 Histogram Buckets#
Default buckets for duration histograms:
0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10 (seconds)
12.3 Monitoring Dashboards#
Recommended Grafana dashboard panels:
| Panel | Metric | Visualization |
|---|---|---|
| Agent Status Overview | agent_health_status |
Status map |
| Task Throughput | orchestrator_tasks_completed |
Time series |
| Task Duration by Type | agent_tasks_duration_seconds |
Heatmap |
| Token Consumption | agent_tokens_consumed |
Stacked bar |
| Cost by Agent | agent_cost_usd |
Pie chart |
| Queue Depth | agent_queue_size |
Time series |
| Error Rate | agent_errors_total |
Time series |
| Message Bus Health | message_bus_* |
Time series |
| Health Issues | health_issues_total |
Counter |
| Healing Actions | healing_actions_total |
Bar chart |
13. Operational Procedures#
13.1 Starting the Agent Service#
# Development
pnpm nx serve veritas-agents
# Production (built binary)
node dist/apps/veritas/agents/main.js
# Required environment variables
export ANTHROPIC_API_KEY=sk-ant-...
export REDIS_URL=redis://localhost:6379 # Optional but recommended
export AGENTS_PORT=3006
export AGENTS_HOST=0.0.0.0
13.2 Verifying Agent Health#
# Service health
curl http://localhost:3006/health
# Readiness (includes Redis check)
curl http://localhost:3006/ready
# List all agents with status
curl http://localhost:3006/v1/agents
# Check specific agent health
curl http://localhost:3006/v1/agents/editor-in-chief-1/health
# Check active issues
curl http://localhost:3006/v1/health/issues
# View metrics
curl http://localhost:3006/v1/metrics
13.3 Creating and Monitoring Tasks#
# Create a task
curl -X POST http://localhost:3006/v1/tasks \
-H "Content-Type: application/json" \
-d '{
"type": "writeArticle",
"title": "Market Update",
"priority": 2,
"input": {
"content": "Ghana Stock Exchange closes higher...",
"constraints": {"language": "en", "maxLength": 1500}
}
}'
# Check task status
curl http://localhost:3006/v1/tasks/{taskId}
# Cancel a task
curl -X DELETE http://localhost:3006/v1/tasks/{taskId}
# Check orchestrator stats
curl http://localhost:3006/v1/orchestrator/stats
# Force task distribution
curl -X POST http://localhost:3006/v1/orchestrator/distribute
13.4 Troubleshooting Agent Issues#
Agent Stuck in Busy State#
# Check agent status
curl http://localhost:3006/v1/agents/{agentId}
# Check active issues
curl http://localhost:3006/v1/agents/{agentId}/issues
# Check healing history
curl http://localhost:3006/v1/health/history
If the agent is stuck, the health monitor will detect it as unresponsive after 3 consecutive check failures and attempt a restart.
LLM Provider Issues#
# Check if provider is responding
curl https://api.anthropic.com/v1/messages \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{"model":"claude-3-5-haiku-20241022","max_tokens":10,"messages":[{"role":"user","content":"test"}]}'
The system will automatically attempt failover to backup providers if the primary provider becomes unavailable.
High Cost Alert#
Monitor the agent_cost_usd metric. If costs exceed expected thresholds:
- Check token consumption per agent via
agent_tokens_consumed - Review task types and frequency
- Consider switching high-volume agents to cheaper models (Haiku instead of Sonnet)
- Adjust temperature settings (lower temperature = fewer retries)
13.5 Scaling Agents#
Adding More Agents#
Additional agents can be registered programmatically by adding configurations to the agent registry. Each new agent needs:
- Unique agent ID
- Role and specialization
- LLM configuration
- Concurrency limits
- Priority level
Scaling Signals#
The orchestrator emits scaling signals:
- Scale Up: When backlog exceeds
scaleUpThreshold(default: 100) - Scale Down: When agents are idle for
scaleDownIdleMs(default: 5 minutes)
These signals can be consumed by external autoscaling systems (Kubernetes HPA, AWS Auto Scaling) to dynamically adjust the agent fleet.