Inference Economics Benchmarking

Overview

Inference cost is the number that decides whether a product is viable, and it is almost never the number in the vendor deck. Real serving cost depends on your sequence-length distribution, batch composition, cache hit rate, latency target and tail behaviour — interactions that aggregate benchmarks hide. This product measures inference economics on your actual workload and traffic shape, and attributes cost to the levers that move it.

What is covered

• Throughput and latency characterisation per accelerator on your model family and quantisation
• Cost per served token and per request, at target percentile latency rather than mean
• Batching and continuous-batching efficiency: how much of theoretical throughput you actually realise
• Prefix and KV-cache hit-rate impact on cost, with cache-strategy recommendation
• Autoscaling behaviour under diurnal and bursty traffic, including scale-down lag cost
• Model-size and quantisation trade studies against quality targets you define
• Hardware comparison on identical workloads: accelerator, interconnect, memory bandwidth and rack density effects

Deliverables

• Benchmark report: throughput, latency percentiles and cost per token by configuration
• Attribution analysis — which lever moves cost most for your traffic shape
• Configuration recommendation with modelled cost reduction and quality impact
• Hardware comparison matrix on identical workloads
• Capacity model translating traffic forecast into accelerator requirement and cost

Who this is for

Teams serving models at scale whose unit economics do not match their vendor benchmarks.

Specification

Measurement basis

Percentile latency at target SLO, not mean throughput

Workload fidelity

Your sequence-length and prompt distribution, replayed

Cost basis

Per served token, per request, per concurrent session

Levers analysed

Batching, caching, quantisation, autoscaling, hardware, topology

Output

Benchmark report, attribution, configuration and capacity model

Related services