Eve library

GPU Jobs Routing Rules Configuration

Guide

Revision 1

On this page

This guide covers the flexible routing rules system for GPU job dispatch. The routing rules allow you to configure job routing behavior without code changes.

Table of Contents#

Overview#

The routing rules system provides:

  • File-based configuration: JSON configuration files that can be modified without code changes
  • Rule priority: Rules are evaluated in priority order, with higher priority rules taking precedence
  • Match conditions: Filter rules based on job type, priority, time, cost, GPU requirements, and custom expressions
  • Provider selection: Specify provider preferences, exclusions, and capacity thresholds
  • Runtime updates: Configuration can be reloaded without restarting the service

Configuration Structure#

typescript
interface RoutingRulesConfig {
  version: string; // Configuration version (e.g., "1.0.0")
  name: string; // Configuration name
  description?: string; // Optional description
  updatedAt?: string; // Last updated timestamp

  rules: RoutingRule[]; // Routing rules (evaluated in priority order)
  providers: ProviderConfig[]; // Provider-specific settings
  fallback: FallbackConfig; // Fallback configuration
  globalSettings: GlobalSettings; // Global settings
  featureFlags: Record<string, boolean>; // Feature flags
}

Rule Definition#

Each routing rule has match conditions and routing actions:

Basic Rule Structure#

json
{
  "id": "unique-rule-id",
  "name": "Human-readable name",
  "description": "What this rule does",
  "priority": 100,
  "enabled": true,

  // Match conditions (all must match)
  "jobTypes": ["texture-upscale", "gaussian-splatting"],
  "priorities": ["high", "urgent"],

  // Routing actions
  "providers": ["runpod-serverless", "modal"],
  "excludeProviders": ["aws-ecs-spot"],
  "preferSpeed": true
}

Match Conditions#

Condition Description Example
jobTypes Job types this rule applies to ["texture-upscale", "blender-render"]
priorities Job priorities to match ["high", "urgent"]
timeRanges Time-based activation See time ranges below
costConditions Cost thresholds {"maxCost": 1.0}
gpuRequirements GPU requirements {"minVRAM": 24}
metadataMatches Custom metadata matches {"customerTier": "enterprise"}
matchExpression Custom JavaScript expression "job.priority === 'urgent'"

Time Range Configuration#

json
{
  "timeRanges": [
    {
      "startHour": 9,
      "endHour": 18,
      "daysOfWeek": [1, 2, 3, 4, 5],
      "timezone": "America/New_York"
    }
  ]
}
  • startHour/endHour: 0-23 (24-hour format)
  • daysOfWeek: 0=Sunday, 6=Saturday
  • timezone: IANA timezone string

Routing Actions#

Action Description Example
providers Preferred providers in order ["runpod-serverless", "modal"]
excludeProviders Providers to exclude ["aws-ecs-spot"]
preferSpot Prefer spot instances true
preferSpeed Prioritize speed over cost true
maxCostOverride Override max cost 0.50
priorityBoost Adjust job priority (-100 to 100) 10
addTags Tags to add to matched jobs ["enterprise", "fast-track"]
capacityThresholds Capacity requirements See below

Provider Configuration#

Configure provider-specific settings:

json
{
  "providers": [
    {
      "provider": "runpod-serverless",
      "enabled": true,
      "weight": 100,
      "capacityThresholds": {
        "maxQueueDepth": 100,
        "maxWaitTimeSeconds": 300,
        "minAvailablePercent": 5
      },
      "costThresholds": {
        "maxCostPerHour": 2.0
      },
      "allowedJobTypes": ["texture-upscale", "gaussian-splatting"],
      "deniedJobTypes": ["video-encode"],
      "geoConfig": {
        "preferredRegions": ["us-east-1"]
      }
    }
  ]
}

Provider Settings#

Setting Description
weight Load balancing weight (0-100)
capacityThresholds When to skip provider
costThresholds Cost limits
allowedJobTypes Whitelist of job types
deniedJobTypes Blacklist of job types
availabilityWindows Time-based availability
geoConfig Geographic preferences

Loading Configuration#

From File#

typescript
import { RoutingRulesLoader, RoutingRulesEngine } from '@yemaya/gpu-jobs';

// Load from JSON file
const config = RoutingRulesLoader.loadFromFile('./routing-config.json');
const engine = new RoutingRulesEngine(config);

From Environment Variable#

bash
export GPU_ROUTING_CONFIG='{"version":"1.0.0",...}'
typescript
const config = RoutingRulesLoader.loadFromEnv('GPU_ROUTING_CONFIG');

With Fallback Chain#

typescript
const config = RoutingRulesLoader.load({
  configPath: './routing-config.json',
  envVar: 'GPU_ROUTING_CONFIG',
  defaultConfig: DEFAULT_ROUTING_RULES_CONFIG,
});

Examples#

High Priority Fast Routing#

Route urgent jobs to fastest providers:

json
{
  "id": "urgent-fast",
  "name": "Urgent jobs fast routing",
  "priority": 100,
  "enabled": true,
  "priorities": ["urgent"],
  "providers": ["runpod-serverless", "modal"],
  "preferSpeed": true
}

Cost Optimization for Low Priority#

Route low priority jobs to cheapest providers:

json
{
  "id": "low-priority-cheap",
  "name": "Low priority cost optimization",
  "priority": 40,
  "enabled": true,
  "priorities": ["low"],
  "providers": ["aws-ecs-spot"],
  "preferSpot": true,
  "maxCostOverride": 0.25
}

Time-Based Routing#

Use spot instances during off-peak hours:

json
{
  "id": "off-peak-spot",
  "name": "Off-peak spot instances",
  "priority": 50,
  "enabled": true,
  "timeRanges": [
    {
      "startHour": 22,
      "endHour": 6,
      "daysOfWeek": [0, 1, 2, 3, 4, 5, 6]
    }
  ],
  "providers": ["aws-ecs-spot"],
  "preferSpot": true
}

Large VRAM Requirements#

Route high-memory jobs to appropriate GPUs:

json
{
  "id": "large-vram",
  "name": "Large VRAM to A100",
  "priority": 90,
  "enabled": true,
  "gpuRequirements": {
    "minVRAM": 40
  },
  "providers": ["runpod-serverless"]
}

Enterprise Customer Routing#

Dedicated routing for enterprise customers:

json
{
  "id": "enterprise",
  "name": "Enterprise dedicated routing",
  "priority": 110,
  "enabled": true,
  "metadataMatches": {
    "customerTier": "enterprise"
  },
  "providers": ["aws-ecs-ondemand"],
  "preferSpeed": true,
  "addTags": ["enterprise", "sla-guaranteed"]
}

Best Practices#

1. Priority Planning#

  • 100+: Critical rules (enterprise, urgent)
  • 80-99: Job type specific rules
  • 60-79: Time-based rules
  • 40-59: Cost optimization rules
  • <40: Catch-all rules

2. Rule Overlap#

Rules can overlap - all matching rules contribute to the decision. Use excludeProviders to prevent unwanted matches.

3. Capacity Thresholds#

Set realistic capacity thresholds:

json
{
  "capacityThresholds": {
    "maxQueueDepth": 100,
    "maxWaitTimeSeconds": 300,
    "minAvailablePercent": 10
  }
}

4. Testing Changes#

  1. Use dryRun in API to test routing decisions
  2. Enable debugLogging in globalSettings for detailed logs
  3. Use A/B testing for significant changes

5. Version Control#

Keep routing configuration in version control:

text
configs/
  routing/
    production.json
    staging.json
    development.json

6. Monitoring#

Monitor these metrics:

  • Rule match rates
  • Provider selection distribution
  • Fallback usage rate
  • Cache hit rates