9 Key LLM Inference Metrics Explained in 5 Minutes
KDnuggets, Wednesday, November 11th, 2026
A primer on nine LLM inference metrics, with formulas for latency, token generation speed, server load and goodput.
High token throughput does not guarantee a fast user experience, because the first token may arrive late or later tokens may stream slowly under load.
The article defines nine metrics that break inference performance into parts, each with its formula and guidance on finding bottlenecks.
Per-request measures are time to first token, end-to-end latency, token generation time, time per output token and inter-token latency, and across many requests it uses the mean, median and tail percentiles such as P99, computed from the combined raw samples of all servers.
Server load is measured in requests and tokens per second, and service quality by goodput.