Inference infrastructure
Single-digit millisecond LLM inference.
Halide routes, batches and caches requests behind one control plane. P99 down to single-digit milliseconds, billed per token, integrated in five minutes.
- Routing
- Batching
- Caching
- Per-token billing
The numbers
One number you can put in a slide.
Every request lands in the same control plane, so the tail of the distribution is the number you report, not the average of a warm cache.
P99
Time to first token, measured at the edge
ms
0.0002
Cost per input token, on the standard model
per token
5 min
From key to first token in production
to production
The pipeline
Three stages, one request.
Routing, batching and caching happen on the same control plane. Scroll to walk the three stages and read what each one actually does.
01 / Routing
Every request lands on one control plane
Model, region and capacity are decided in a single request. Failures are retried with backoff instead of a cold start.
02 / Batching
Requests are pooled before they touch a model
Short requests wait in a queue while the long ones finish. Nothing is scheduled alone, so the queue never becomes the queue.
03 / Caching
Identical prompts return in milliseconds
Replayed prompts are served from cache. When the answer is new, the same model is called, never a new one.
Billing
Per token, measured at the edge.
You pay for the tokens you send. Output tokens are counted separately, and the meter is the same one your own telemetry reads.
| Input tokens | 342 |
|---|---|
| Output tokens | 186 |
| Input cost | $0.0002 × 342 |
| Output cost | $0.0006 × 186 |
| Total for this request | $0.00030 |
Billing is reconciled per region and per model, so a rate change is a configuration change rather than a billing surprise.
Token budget
The same meter your telemetry reads
Integration
Five minutes from key to first token.
No migration, no new runtime. Halide sits in front of the model you already ship.
-
01
Create the key
One key per environment. Nothing to install and nothing to restart.
-
02
Point the client at the endpoint
One base URL, the same call shape your service already has.
-
03
Watch the first P99
Latency and tokens stream into your own dashboard as they happen.
// the same call you already have import { Halide } from "@halide/sdk"; const halide = new Halide({ key: process.env.HALIDE_KEY, model: "llm/v1", }); const response = await halide.chat({ model: "llm/v1", messages: [...], });
