This guide covers the cost estimation system, pricing models, budget management, and cost optimization strategies.
Table of Contents#
- Overview
- Pricing Data
- Cost Estimation Algorithm
- Cost Breakdown
- Budget Management
- Cost Optimization
- Reporting & Analytics
- API Reference
Overview#
The GPU Jobs cost estimation system provides accurate cost predictions before job submission, enabling informed decisions about provider selection and budget management.
Key Features#
- Pre-dispatch estimation: Get cost estimates before submitting jobs
- Multi-provider comparison: Compare costs across all configured providers
- Budget enforcement: Automatic rejection of jobs exceeding budgets
- Real-time tracking: Track actual vs. estimated costs
- Cost optimization recommendations: Suggestions for reducing GPU spend
Pricing Data#
GPU Pricing Table (as of 2024)#
| GPU | RunPod Serverless | RunPod Pod | Modal | AWS Spot | AWS On-Demand |
|---|---|---|---|---|---|
| RTX 3090 | $1.58/hr | $0.69/hr | - | - | - |
| RTX 4090 | $2.48/hr | $1.14/hr | - | - | - |
| T4 | - | - | $0.59/hr | $0.16/hr | $0.53/hr |
| L4 | - | - | $1.10/hr | - | - |
| A10 | $2.09/hr | $1.00/hr | $1.10/hr | $0.30/hr | $1.01/hr |
| A100 40GB | $5.00/hr | $3.19/hr | $3.88/hr | $2.46/hr | $8.19/hr |
| A100 80GB | $7.99/hr | $5.19/hr | $5.63/hr | $3.69/hr | $12.29/hr |
| H100 | - | $14.00/hr | $8.12/hr | $9.83/hr | $32.77/hr |
Data Transfer Costs#
| Provider | Ingress | Egress |
|---|---|---|
| RunPod | Free | $0.05/GB |
| Modal | Free | $0.05/GB |
| AWS | Free | $0.09/GB |
| Vast.ai | Varies | Varies |
| Local | Free | Free |
Storage Costs#
| Provider | Persistent Storage | Temporary |
|---|---|---|
| RunPod Pod | $0.10/GB/month | Included |
| AWS EBS | $0.08/GB/month | Included |
| Modal | Free (ephemeral) | N/A |
Cost Estimation Algorithm#
Basic Formula#
text
Total Cost = Compute Cost + Data Transfer Cost + Platform Fee
Where:
Compute Cost = (Duration × GPU Price) × GPU Count
Data Transfer Cost = (Input Size × Ingress Rate) + (Output Size × Egress Rate)
Platform Fee = Compute Cost × Platform Fee Rate
Duration Estimation#
Duration is estimated based on job type and GPU:
typescript
interface DurationEstimate {
baseSeconds: number; // Base processing time
vramMultiplier: number; // Higher VRAM = faster
gpuGenerationMultiplier: number; // Newer GPU = faster
}
const durationFactors: Record<GPUJobType, DurationEstimate> = {
'texture-upscale': {
baseSeconds: 30,
vramMultiplier: 0.8, // 20% faster per 8GB VRAM
gpuGenerationMultiplier: 0.9, // 10% faster per generation
},
'gaussian-splatting': {
baseSeconds: 1800, // 30 minutes base
vramMultiplier: 0.7, // 30% faster with more VRAM
gpuGenerationMultiplier: 0.85,
},
'blender-render': {
baseSeconds: 120, // Per frame base
vramMultiplier: 0.75,
gpuGenerationMultiplier: 0.8,
},
// ... other job types
};
GPU Selection Algorithm#
typescript
function selectGPU(
requirements: GPURequirements,
preferCheapest: boolean = true
): GPUType {
// 1. Filter GPUs meeting VRAM requirement
const eligible = GPU_TYPES.filter((gpu) => gpu.vram >= requirements.minVRAM);
// 2. Filter by feature requirements (tensor cores, etc.)
const compatible = eligible.filter((gpu) =>
meetsFeatureRequirements(gpu, requirements)
);
// 3. Apply preference
if (
requirements.preferredGPU &&
compatible.includes(requirements.preferredGPU)
) {
return requirements.preferredGPU;
}
// 4. Select based on strategy
if (preferCheapest) {
return compatible.sort((a, b) => a.pricePerHour - b.pricePerHour)[0];
} else {
return compatible.sort((a, b) => b.performance - a.performance)[0];
}
}
Cold Start Estimation#
Cold start times are factored into duration estimates:
| Provider | GPU Type | Average Cold Start |
|---|---|---|
| RunPod Serverless | Any | 3-5 seconds |
| RunPod Pod | Any | 30-60 seconds |
| Modal | T4/L4 | 5-10 seconds |
| Modal | A10/A100 | 10-15 seconds |
| Modal | H100 | 15-30 seconds |
| AWS ECS | Any | 60-120 seconds |
| Vast.ai | Any | 30-120 seconds |
| Local | N/A | 0 seconds |
Cost Breakdown#
Detailed Breakdown Structure#
typescript
interface CostBreakdown {
// Compute costs
compute: {
gpuTime: number; // GPU compute time cost
cpuTime: number; // CPU compute time cost (if any)
coldStart: number; // Cold start overhead cost
};
// Data transfer
transfer: {
ingress: number; // Input data download cost
egress: number; // Output data upload cost
internal: number; // Internal transfer cost
};
// Storage
storage: {
persistent: number; // Persistent volume cost
temporary: number; // Temporary storage cost
};
// Platform fees
platform: {
baseFee: number; // Provider platform fee
premiumFee: number; // Priority/SLA fee (if applicable)
};
// Total
total: number;
currency: 'USD';
}
Example Breakdown#
typescript
// Example: 4K texture upscaling job
const breakdown: CostBreakdown = {
compute: {
gpuTime: 0.0413, // 60 seconds on RTX 4090 at $2.48/hr
cpuTime: 0,
coldStart: 0.0021, // 3 second cold start
},
transfer: {
ingress: 0, // 50MB input (free)
egress: 0.01, // 200MB output at $0.05/GB
internal: 0,
},
storage: {
persistent: 0,
temporary: 0,
},
platform: {
baseFee: 0, // No additional fee
premiumFee: 0,
},
total: 0.0534,
currency: 'USD',
};
Budget Management#
Budget Configuration#
typescript
interface BudgetConfig {
// Per-job limits
maxCostPerJob?: number; // Max cost for any single job
// Time-based limits
hourlyBudget?: number; // Max spend per hour
dailyBudget?: number; // Max spend per day
weeklyBudget?: number; // Max spend per week
monthlyBudget?: number; // Max spend per month
// Alerts
alertThresholds?: {
warning: number; // Alert at this percentage (e.g., 0.8 = 80%)
critical: number; // Critical alert (e.g., 0.95 = 95%)
};
// Enforcement
enforcement?: 'soft' | 'hard'; // Soft = warn, Hard = reject
// Rollover
rolloverUnused?: boolean; // Carry unused budget to next period
}
Budget Enforcement#
typescript
const dispatcher = createJobDispatcher({
costEstimator,
budgetConfig: {
maxCostPerJob: 10.0,
dailyBudget: 100.0,
monthlyBudget: 1000.0,
alertThresholds: {
warning: 0.8,
critical: 0.95,
},
enforcement: 'hard',
},
});
// Job submission with budget check
try {
await dispatcher.dispatch(job);
} catch (error) {
if (error.code === 'BUDGET_EXCEEDED') {
console.log(`Budget exceeded: ${error.message}`);
console.log(`Current spend: $${error.currentSpend}`);
console.log(`Budget limit: $${error.budgetLimit}`);
}
}
Budget Status API#
typescript
// Get current budget status
const status = await dispatcher.getBudgetStatus();
console.log('Budget Status:', {
daily: {
spent: status.daily.spent,
limit: status.daily.limit,
remaining: status.daily.remaining,
percentUsed: status.daily.percentUsed,
},
monthly: {
spent: status.monthly.spent,
limit: status.monthly.limit,
remaining: status.monthly.remaining,
percentUsed: status.monthly.percentUsed,
},
alerts: status.alerts, // Active alerts
forecast: status.forecast, // Projected spend
});
Alert Webhooks#
typescript
dispatcher.on('budget-warning', (event) => {
console.log(`Budget warning: ${event.budgetType} at ${event.percentUsed}%`);
// Send notification
});
dispatcher.on('budget-critical', (event) => {
console.log(`Budget critical: ${event.budgetType} at ${event.percentUsed}%`);
// Escalate alert
});
dispatcher.on('budget-exceeded', (event) => {
console.log(`Budget exceeded: ${event.budgetType}`);
// Take action (pause jobs, etc.)
});
Cost Optimization#
Provider Selection Strategies#
typescript
// 1. Cheapest provider (default)
const cheapest = createJobDispatcher({
costEstimator,
selectionStrategy: 'cheapest',
});
// 2. Fastest provider (minimize duration + cold start)
const fastest = createJobDispatcher({
costEstimator,
selectionStrategy: 'fastest',
});
// 3. Best value (cost per performance)
const bestValue = createJobDispatcher({
costEstimator,
selectionStrategy: 'best-value',
});
// 4. Custom strategy
const custom = createJobDispatcher({
costEstimator,
selectionStrategy: (estimates) => {
// Custom logic to select provider
return (
estimates.find(
(e) => e.provider === 'modal' && e.gpuType === 'a100-40gb'
) || estimates[0]
);
},
});
Spot Instance Usage#
typescript
const job: GPUJobRequestBase = {
type: 'gaussian-splatting',
priority: 'normal',
gpuRequirements: {
minVRAM: 40,
canUseSpot: true, // Allow spot/preemptible
estimatedDuration: 3600,
},
// ...
};
// Spot savings can be 60-80% vs. on-demand
const estimate = costEstimator.estimate(job);
console.log(`On-demand cost: $${estimate.onDemandCost}`);
console.log(`Spot cost: $${estimate.spotCost}`);
console.log(`Savings: ${estimate.spotSavingsPercent}%`);
Job Batching#
typescript
// Batch multiple small jobs to reduce cold start overhead
const batcher = createJobBatcher({
dispatcher,
maxBatchSize: 10,
maxWaitTime: 5000, // Max wait before dispatching
batchBy: 'job-type', // Batch by job type
});
// Submit jobs - they'll be batched automatically
await batcher.submit(job1);
await batcher.submit(job2);
await batcher.submit(job3);
// Flush immediately if needed
await batcher.flush();
Warm Pool Pre-provisioning#
typescript
const warmPool = createWarmPoolManager({
dispatcher,
pools: {
'texture-upscale': {
minWorkers: 2,
maxWorkers: 10,
gpuType: 'rtx-4090',
provider: 'runpod-serverless',
idleTimeout: 300, // Keep warm for 5 minutes
},
'blender-render': {
minWorkers: 1,
maxWorkers: 5,
gpuType: 'a10',
provider: 'modal',
idleTimeout: 600,
},
},
});
// Jobs use warm workers instantly (no cold start)
const result = await dispatcher.dispatch(job);
Cost Optimization Recommendations#
typescript
// Get optimization recommendations
const recommendations = await costEstimator.getOptimizationRecommendations({
jobHistory: await getRecentJobs(30), // Last 30 days
currentConfig: dispatcher.getConfig(),
});
recommendations.forEach((rec) => {
console.log(`${rec.type}: ${rec.description}`);
console.log(` Potential savings: $${rec.potentialSavings}/month`);
console.log(` Implementation: ${rec.implementation}`);
});
// Example recommendations:
// - "Switch to spot instances": Save $150/month by using spot for low-priority jobs
// - "Use warm pools": Save $50/month by reducing cold starts
// - "Batch similar jobs": Save $30/month by batching texture-upscale jobs
// - "Right-size GPU selection": Save $75/month by using A10 instead of A100 for jobs under 16GB VRAM
Reporting & Analytics#
Usage Reports#
typescript
import { getGPUJobsMonitoringService } from '@yemaya/gpu-jobs';
const monitoring = getGPUJobsMonitoringService();
// Daily report
const dailyReport = await monitoring.getDailyReport(new Date());
console.log('Daily Report:', {
totalJobs: dailyReport.totalJobs,
totalCost: dailyReport.totalCost,
averageCostPerJob: dailyReport.averageCostPerJob,
costByProvider: dailyReport.costByProvider,
costByJobType: dailyReport.costByJobType,
costByUser: dailyReport.costByUser,
estimateAccuracy: dailyReport.estimateAccuracy,
});
// Monthly report
const monthlyReport = await monitoring.getMonthlyReport(2024, 11);
console.log('Monthly Report:', {
totalCost: monthlyReport.totalCost,
trend: monthlyReport.trend, // vs. previous month
topUsers: monthlyReport.topUsers,
topJobTypes: monthlyReport.topJobTypes,
savingsAchieved: monthlyReport.savingsAchieved,
});
Cost Metrics#
typescript
// Real-time metrics
const metrics = monitoring.getCostMetrics();
console.log('Cost Metrics:', {
// Running costs
currentHourCost: metrics.currentHourCost,
currentDayCost: metrics.currentDayCost,
runningJobsCost: metrics.runningJobsCost,
// Historical
last24HoursCost: metrics.last24HoursCost,
last7DaysCost: metrics.last7DaysCost,
last30DaysCost: metrics.last30DaysCost,
// Estimates
projectedDailyCost: metrics.projectedDailyCost,
projectedMonthlyCost: metrics.projectedMonthlyCost,
// Accuracy
estimateAccuracy: metrics.estimateAccuracy, // Percentage
avgEstimateDeviation: metrics.avgEstimateDeviation,
});
Estimate Accuracy Tracking#
typescript
// Track estimate accuracy over time
const accuracyReport = await monitoring.getEstimateAccuracyReport({
startDate: new Date('2024-10-01'),
endDate: new Date('2024-10-31'),
groupBy: 'provider',
});
accuracyReport.forEach((entry) => {
console.log(`${entry.provider}:`);
console.log(` Average accuracy: ${entry.avgAccuracy}%`);
console.log(` Over-estimates: ${entry.overEstimatePercent}%`);
console.log(` Under-estimates: ${entry.underEstimatePercent}%`);
console.log(` Max deviation: $${entry.maxDeviation}`);
});
API Reference#
GPUCostEstimator#
typescript
class GPUCostEstimator {
// Estimate cost for a single job
estimate(job: GPUJobRequestBase): CostEstimate;
// Estimate for specific provider
estimateForProvider(
job: GPUJobRequestBase,
provider: GPUProviderType,
gpuType: GPUType
): CostEstimate;
// Compare all providers
estimateAllProviders(job: GPUJobRequestBase): CostEstimate[];
// Get cheapest option
getCheapestOption(job: GPUJobRequestBase): CostEstimate;
// Get fastest option
getFastestOption(job: GPUJobRequestBase): CostEstimate;
// Calculate batch cost
estimateBatch(jobs: GPUJobRequestBase[]): BatchCostEstimate;
// Get optimization recommendations
getOptimizationRecommendations(
config: OptimizationConfig
): Promise<Recommendation[]>;
// Update pricing data
updatePricing(provider: GPUProviderType, pricing: ProviderPricing): void;
}
CostEstimate Interface#
typescript
interface CostEstimate {
// Core estimates
estimatedCost: number; // Total estimated cost in USD
estimatedDuration: number; // Estimated duration in seconds
// Provider info
provider: GPUProviderType;
gpuType: GPUType;
gpuCount: number;
// Detailed breakdown
breakdown: CostBreakdown;
// Confidence
confidence: 'high' | 'medium' | 'low';
confidenceFactors: string[];
// Alternatives
alternatives?: {
provider: GPUProviderType;
gpuType: GPUType;
cost: number;
duration: number;
}[];
// Spot pricing (if available)
spotCost?: number;
spotSavingsPercent?: number;
}
Budget Status Interface#
typescript
interface BudgetStatus {
// Current period status
hourly: PeriodStatus;
daily: PeriodStatus;
weekly: PeriodStatus;
monthly: PeriodStatus;
// Active alerts
alerts: BudgetAlert[];
// Forecast
forecast: {
endOfDay: number;
endOfWeek: number;
endOfMonth: number;
};
// Running jobs
runningJobsCost: number;
pendingJobsCost: number;
}
interface PeriodStatus {
spent: number;
limit: number | null;
remaining: number | null;
percentUsed: number;
status: 'ok' | 'warning' | 'critical' | 'exceeded';
}
interface BudgetAlert {
type: 'warning' | 'critical' | 'exceeded';
budgetType: 'hourly' | 'daily' | 'weekly' | 'monthly' | 'per-job';
message: string;
timestamp: Date;
currentValue: number;
threshold: number;
}
Best Practices#
1. Always Estimate Before Dispatch#
typescript
// Good: Check cost before dispatch
const estimate = dispatcher.estimateCost(job);
if (estimate.estimatedCost > acceptableThreshold) {
console.log('Job too expensive, consider alternatives');
console.log('Alternatives:', estimate.alternatives);
} else {
await dispatcher.dispatch(job);
}
2. Set Appropriate Budgets#
typescript
// Per-project budget tracking
const projectBudget = createBudgetTracker({
projectId: 'project-123',
monthlyBudget: 500,
alertWebhook: 'https://api.example.com/budget-alerts',
});
3. Use Spot When Possible#
typescript
// Enable spot for non-critical jobs
const job: GPUJobRequestBase = {
// ...
gpuRequirements: {
canUseSpot: priority !== 'urgent',
},
};
4. Right-Size GPU Selection#
typescript
// Don't over-provision
const requirements = {
minVRAM: calculateMinVRAM(jobInput), // Calculate actual need
// NOT: minVRAM: 80 // Always using A100 80GB
};
5. Monitor Estimate Accuracy#
typescript
// Track and improve estimates
dispatcher.on('job-completed', async (event) => {
const accuracy = event.actualCost / event.estimatedCost;
if (accuracy < 0.8 || accuracy > 1.2) {
// Investigate significant deviations
await reportEstimateDeviation(event);
}
});