This guide covers the flexible routing rules system for GPU job dispatch. The routing rules allow you to configure job routing behavior without code changes.
Table of Contents#
- Overview
- Configuration Structure
- Rule Definition
- Provider Configuration
- Loading Configuration
- Examples
- Best Practices
Overview#
The routing rules system provides:
- File-based configuration: JSON configuration files that can be modified without code changes
- Rule priority: Rules are evaluated in priority order, with higher priority rules taking precedence
- Match conditions: Filter rules based on job type, priority, time, cost, GPU requirements, and custom expressions
- Provider selection: Specify provider preferences, exclusions, and capacity thresholds
- Runtime updates: Configuration can be reloaded without restarting the service
Configuration Structure#
typescript
interface RoutingRulesConfig {
version: string; // Configuration version (e.g., "1.0.0")
name: string; // Configuration name
description?: string; // Optional description
updatedAt?: string; // Last updated timestamp
rules: RoutingRule[]; // Routing rules (evaluated in priority order)
providers: ProviderConfig[]; // Provider-specific settings
fallback: FallbackConfig; // Fallback configuration
globalSettings: GlobalSettings; // Global settings
featureFlags: Record<string, boolean>; // Feature flags
}
Rule Definition#
Each routing rule has match conditions and routing actions:
Basic Rule Structure#
json
{
"id": "unique-rule-id",
"name": "Human-readable name",
"description": "What this rule does",
"priority": 100,
"enabled": true,
// Match conditions (all must match)
"jobTypes": ["texture-upscale", "gaussian-splatting"],
"priorities": ["high", "urgent"],
// Routing actions
"providers": ["runpod-serverless", "modal"],
"excludeProviders": ["aws-ecs-spot"],
"preferSpeed": true
}
Match Conditions#
| Condition | Description | Example |
|---|---|---|
jobTypes |
Job types this rule applies to | ["texture-upscale", "blender-render"] |
priorities |
Job priorities to match | ["high", "urgent"] |
timeRanges |
Time-based activation | See time ranges below |
costConditions |
Cost thresholds | {"maxCost": 1.0} |
gpuRequirements |
GPU requirements | {"minVRAM": 24} |
metadataMatches |
Custom metadata matches | {"customerTier": "enterprise"} |
matchExpression |
Custom JavaScript expression | "job.priority === 'urgent'" |
Time Range Configuration#
json
{
"timeRanges": [
{
"startHour": 9,
"endHour": 18,
"daysOfWeek": [1, 2, 3, 4, 5],
"timezone": "America/New_York"
}
]
}
startHour/endHour: 0-23 (24-hour format)daysOfWeek: 0=Sunday, 6=Saturdaytimezone: IANA timezone string
Routing Actions#
| Action | Description | Example |
|---|---|---|
providers |
Preferred providers in order | ["runpod-serverless", "modal"] |
excludeProviders |
Providers to exclude | ["aws-ecs-spot"] |
preferSpot |
Prefer spot instances | true |
preferSpeed |
Prioritize speed over cost | true |
maxCostOverride |
Override max cost | 0.50 |
priorityBoost |
Adjust job priority (-100 to 100) | 10 |
addTags |
Tags to add to matched jobs | ["enterprise", "fast-track"] |
capacityThresholds |
Capacity requirements | See below |
Provider Configuration#
Configure provider-specific settings:
json
{
"providers": [
{
"provider": "runpod-serverless",
"enabled": true,
"weight": 100,
"capacityThresholds": {
"maxQueueDepth": 100,
"maxWaitTimeSeconds": 300,
"minAvailablePercent": 5
},
"costThresholds": {
"maxCostPerHour": 2.0
},
"allowedJobTypes": ["texture-upscale", "gaussian-splatting"],
"deniedJobTypes": ["video-encode"],
"geoConfig": {
"preferredRegions": ["us-east-1"]
}
}
]
}
Provider Settings#
| Setting | Description |
|---|---|
weight |
Load balancing weight (0-100) |
capacityThresholds |
When to skip provider |
costThresholds |
Cost limits |
allowedJobTypes |
Whitelist of job types |
deniedJobTypes |
Blacklist of job types |
availabilityWindows |
Time-based availability |
geoConfig |
Geographic preferences |
Loading Configuration#
From File#
typescript
import { RoutingRulesLoader, RoutingRulesEngine } from '@yemaya/gpu-jobs';
// Load from JSON file
const config = RoutingRulesLoader.loadFromFile('./routing-config.json');
const engine = new RoutingRulesEngine(config);
From Environment Variable#
bash
export GPU_ROUTING_CONFIG='{"version":"1.0.0",...}'
typescript
const config = RoutingRulesLoader.loadFromEnv('GPU_ROUTING_CONFIG');
With Fallback Chain#
typescript
const config = RoutingRulesLoader.load({
configPath: './routing-config.json',
envVar: 'GPU_ROUTING_CONFIG',
defaultConfig: DEFAULT_ROUTING_RULES_CONFIG,
});
Examples#
High Priority Fast Routing#
Route urgent jobs to fastest providers:
json
{
"id": "urgent-fast",
"name": "Urgent jobs fast routing",
"priority": 100,
"enabled": true,
"priorities": ["urgent"],
"providers": ["runpod-serverless", "modal"],
"preferSpeed": true
}
Cost Optimization for Low Priority#
Route low priority jobs to cheapest providers:
json
{
"id": "low-priority-cheap",
"name": "Low priority cost optimization",
"priority": 40,
"enabled": true,
"priorities": ["low"],
"providers": ["aws-ecs-spot"],
"preferSpot": true,
"maxCostOverride": 0.25
}
Time-Based Routing#
Use spot instances during off-peak hours:
json
{
"id": "off-peak-spot",
"name": "Off-peak spot instances",
"priority": 50,
"enabled": true,
"timeRanges": [
{
"startHour": 22,
"endHour": 6,
"daysOfWeek": [0, 1, 2, 3, 4, 5, 6]
}
],
"providers": ["aws-ecs-spot"],
"preferSpot": true
}
Large VRAM Requirements#
Route high-memory jobs to appropriate GPUs:
json
{
"id": "large-vram",
"name": "Large VRAM to A100",
"priority": 90,
"enabled": true,
"gpuRequirements": {
"minVRAM": 40
},
"providers": ["runpod-serverless"]
}
Enterprise Customer Routing#
Dedicated routing for enterprise customers:
json
{
"id": "enterprise",
"name": "Enterprise dedicated routing",
"priority": 110,
"enabled": true,
"metadataMatches": {
"customerTier": "enterprise"
},
"providers": ["aws-ecs-ondemand"],
"preferSpeed": true,
"addTags": ["enterprise", "sla-guaranteed"]
}
Best Practices#
1. Priority Planning#
- 100+: Critical rules (enterprise, urgent)
- 80-99: Job type specific rules
- 60-79: Time-based rules
- 40-59: Cost optimization rules
- <40: Catch-all rules
2. Rule Overlap#
Rules can overlap - all matching rules contribute to the decision. Use
excludeProviders to prevent unwanted matches.
3. Capacity Thresholds#
Set realistic capacity thresholds:
json
{
"capacityThresholds": {
"maxQueueDepth": 100,
"maxWaitTimeSeconds": 300,
"minAvailablePercent": 10
}
}
4. Testing Changes#
- Use
dryRunin API to test routing decisions - Enable
debugLogginginglobalSettingsfor detailed logs - Use A/B testing for significant changes
5. Version Control#
Keep routing configuration in version control:
text
configs/
routing/
production.json
staging.json
development.json
6. Monitoring#
Monitor these metrics:
- Rule match rates
- Provider selection distribution
- Fallback usage rate
- Cache hit rates