Get more sellable tokens from every GPU
AI-native inference companies earn on tokens per GPU hour, and request routing decides how many tokens each GPU produces
Your margin is decided on every request
An inference company buys GPU capacity at a fixed cost and sells tokens at a falling price. The difference is margin, and much of it is decided in the request path: which GPU serves a request, whether its cache is warm, and whether it waits in a queue. Round robin treats every request alike. F5 routes each one on live serving signals, on the F5 Application Delivery and Security Platform.
More tokens from the same GPUs
Up to 40% more token throughput in DPU mode, in testing validated by The Tolly Group.
Faster first tokens
Up to 61% faster time to first token, in testing validated by The Tolly Group.
Keep the serving stack you run
Routes on signals from NVIDIA Dynamo and NVIDIA NIM; nothing in the stack is replaced.
Govern and secure every customer
Track and limit tokens per customer, and protect the APIs that expose your models.
F5 helps inference companies protect margin
Inference-aware routing
PRODUCT
BIG-IP Next for Kubernetes
Inference-aware traffic control in front of model server pools, on standard host CPUs.
PRODUCT
BIG-IP Next for Kubernetes deployed on NVIDIA BlueField DPUs
The same routing with the data path offloaded to the DPU.
PRODUCT
NGINX Gateway Fabric
Gateway API implementation with Inference Extension support for in-cluster routing.
Time to first token
PRODUCT
BIG-IP Next for Kubernetes deployed on NVIDIA BlueField DPUs
The configuration measured in testing validated by The Tolly Group.
PRODUCT
BIG-IP Next for Kubernetes
The same routing logic on standard host CPUs; size on your own test results.
Token governance per customer
PRODUCT
BIG-IP Next for Kubernetes
Per-customer token tracking, token-based limits, and tenant isolation.
PRODUCT
BIG-IP Next for Kubernetes deployed on NVIDIA BlueField DPUs
Tenant enforcement on the DPU, separate from the host operating system.
Secure model APIs and agents
PRODUCT
Distributed Cloud DDoS Mitigation
Cloud-delivered DDoS scrubbing ahead of inference endpoints.
Resources
Reports
Frequently asked questions
Inference-aware routing sends each AI request to the model server best placed to serve it, using live signals from the serving layer such as queue depth, KV cache utilization, and GPU load. Generic load balancing spreads requests evenly, but inference requests vary widely in cost, so an even spread produces uneven work. The Kubernetes Gateway API Inference Extension standardizes this pattern.
To reduce time to first token, start with queueing, because a request that waits behind long generations on a busy GPU returns its first token late regardless of model speed. Route requests to endpoints with short queues and warm caches, keep prompt prefixes on servers that already hold them, and separate very long requests from short interactive ones. Model-side techniques such as batching settings and quantization help further.
Cost per token is roughly the cost of a GPU hour divided by the useful tokens produced in that hour. The cost side is largely fixed once hardware is bought or leased, so the lever is output: utilization, cache hit rates, batching efficiency, and how evenly requests are spread across GPUs. Small gains in tokens per GPU hour move margin directly when token prices are falling.
No, inference-aware routing does not replace NVIDIA Dynamo or vLLM. Serving frameworks decide how a model runs on a GPU: batching, caching, and scheduling. Inference-aware routing decides which server a request reaches and under what policy. BIG-IP Next for Kubernetes routes on signals those systems expose, so the serving stack stays in place.
The published results were measured on NVIDIA BlueField-3 DPUs in DPU mode and do not carry over automatically to host deployments. BIG-IP Next for Kubernetes runs on standard host CPUs with the same routing logic, which is where most inference companies on leased capacity will start. Size a host deployment on its own test results.