The Problem: Batch Inference Jobs Were GPU-Bound and Cost-Inefficient

Forge Labs runs large-scale data-labeling pipelines that depend on LLM-assisted pre-annotation before human review. Their original setup statically provisioned GPU pools per job type, which meant annotation pipelines for low-throughput label categories sat at 8-15% utilization while high-throughput categories queued for capacity. The underlying issue was static partitioning of compute by job type rather than dynamic allocation by actual load.

Forge Labs' infrastructure team estimated that roughly a third of their monthly GPU spend went to idle capacity reserved for burst scenarios that materialized in fewer than 15% of billing cycles.

The Implementation: Synapse OS Dynamic Batch Scheduler

Forge Labs adopted the Synapse OS Dynamic Batch Scheduler, which pools GPU capacity across job types and packs inference requests into continuous batches using a 50ms aggregation window. Requests are scored by estimated token length (via a lightweight tokenizer pre-pass) and bin-packed to maximize tensor-core utilization without exceeding per-batch memory ceilings.

The scheduler treats GPU memory as the actual constraint, not job category. Once we stopped pretending our annotation jobs needed dedicated pools, utilization just went up.

Synapse OS's autoscaler also integrates with Forge Labs' existing Kubernetes cluster via a custom metrics adapter, scaling the underlying node pool based on queue depth rather than CPU/GPU utilization alone, which previously caused scale-up to lag actual demand by 60-90 seconds.

Results: Measured Infrastructure Impact

Forge Labs measured a 340% increase in throughput per GPU-hour after migrating to dynamic batch scheduling, driven primarily by eliminating idle reserved capacity and improving average batch fill rate from 41% to 89%. Total idle GPU cost fell by 62% quarter-over-quarter, while end-to-end annotation latency (time from job submission to pre-annotated output) improved from a P90 of 14 minutes to 4.2 minutes.