# How The Forecasting Company got 22x faster batch inference
The Forecasting Company (TFC) builds time-series foundation models: one model that forecasts anything predictable, from retail demand and SKU sales to electricity production and demand, using only historical context. Joachim, TFC's CTO and co-founder, pointed Claude at a machine0 H100 and let it optimize the model's inference path for days. Batch inference came out 22x faster, using 6x less GPU memory.
## The problem
TFC's model is zero-shot. It learns in context, never trains on customer data, and is sold as an API. That makes speed a product problem, in two places.
The first is inference. Retailers forecast hundreds of thousands to millions of SKUs on tight daily SLAs. Faster inference means meeting those SLAs on fewer GPUs, which makes every forecast cheaper.
The second is backtesting, which is expensive. The goal is instant feedback when adding a variable.
Memory mattered too. Batching more series into one forward pass lets the model learn across them, which helps with cold-start products that have little history of their own. TFC was capped at 5,000 series per GPU.
"This GPU stuff is hard enough as it is."
— Joachim, CTO & Co-founder, The Forecasting Company
## How Joachim ran it
Joachim used Claude, running locally on his laptop, as the orchestrator. Before any optimization work, he built a benchmark harness: speed measurement plus multi-tier parity checks, so the agent could not claim wins that weren't real.
The first attempt showed why that mattered. The optimizations overfit to fixed input shapes and broke on realistic inputs, so he hardened the harness before going further.
He then stripped the project down to what the agent needed:
- Only the inference slice of the model
- Synthetic data
- The benchmark harness
- An operating contract for the agent
- Rules explaining which speedups and parity deltas matter
That bundle was synced to a machine0 H100 with no credentials, no customer data and no production access.
"I really wanted a sandbox where we could just play."
— Joachim
He told the agent to keep going for days and not to be conservative on spend. It still had an upper bound, and it never got near it. Joachim started the run in San Francisco, flew to Europe, and landed to finished experiments and a full experiment log.
## Results
- 22x faster batch inference than the unoptimized FP32 baseline used for parity checks.
- 6.3x faster than the BF16 production stack, while at parity with FP32. Faster and more precise.
- 6x less GPU memory, going from 5,000 to 1,000,000 series in a single forward pass.
- Cold-start compile dropped from about 1 minute to under 1 second for typical forecast shapes.
- About 200,000 forecasts per second on a typical retail shape, with the full model, no distillation and identical output.
Getting there took about 300 experiments, 87 of which failed parity tests, over 9 active GPU days, for a few thousand dollars in total.
## Why a machine, not a job
TFC's GCP cluster is job-shaped: launch, run, die, collect logs. Modal is function-shaped, which suits one-off benchmarks. This work needed custom kernels and CUDA graphs, developed and tested on real Hopper hardware. Raw EC2 or GCP VMs would have meant managing spot instances, availability, sizes and networking.
"What was really powerful with machine0 is having a machine on all the time, with a loop on itself."
— Joachim
Suspend and resume kept the machine's full state for weeks, without paying for idle compute.
"I haven't turned the machine on for a couple of weeks. I can still just turn it on and start exactly where I was."
— Joachim
## GPU access without a commitment
Hyperscalers asked TFC for three-year commitments, and neoclouds for six-month minimums. TFC needs to ramp up gradually. GPUs on machine0 are on demand, with no commitment.
## Status
The optimized inference is running on TFC's internal endpoint and on custom endpoints for pilot customers where speed matters, ahead of a full rollout.
"In the end, it's Claude and a good VM."
— Joachim
Get Started Now
NixOS or Ubuntu with tools pre-installed. Dedicated resources, static IPs, per-minute billing.
