LLM Inference Speed (Tokens/Sec) Calculator

Pick the hardware, the model and the format its weights are stored in, then the length of the prompt and of the reply. The calculator estimates how one request runs: how long the prompt takes before the first token appears, and how many tokens a second the reply streams at after that. Choose Custom hardware or Custom architecture to enter any device's specifications or any model's config.json. The math runs entirely in your browser; to size memory in more detail, use the LLM VRAM calculator.

How the estimate works

Inference has two phases that hit different limits. Prefill pushes every prompt token through the weights at once, so the multiplications set the pace. Decode produces one token at a time, and each token has to read every active weight and the whole KV cache from memory, so bandwidth sets the pace. The calculator takes the slower of the two limits in each phase:

prompt time = max(FLOPs ÷ effective compute, bytes read ÷ effective bandwidth), where FLOPs = 2 × N_active × T + layers × heads × (d_k + d_v) × T × (T + 1)

tokens per second = effective bandwidth ÷ (active weight bytes + KV cache bytes)

The first term counts a multiply and an add for every active parameter and prompt token; the second is causal attention, which grows with the square of the prompt length T. Effective means peak times the efficiency you set. N counts only the parameters a token is multiplied through: the embedding table is a lookup, not a multiplication, and the output layer runs only for tokens that need a prediction, which vLLM and llama.cpp limit to the last token of the prompt. The cache grows as the reply does, so generation speed is worked out at the reply's midpoint, which makes the total time exact while decoding is limited by memory.

The defaults run Llama 3.1 8B in BF16 on an RTX 4090, with a 2,000-token prompt and a 500-token reply. Of its 8,030,261,248 parameters, 1,050,673,152 are the embedding table and output layer, leaving 6,979,588,096 that every token is multiplied through. The prompt costs 27.92 trillion FLOPs for those, plus 1.05 trillion for attention and 1.05 billion for one pass through the output layer: 0.25 seconds at 70% of the card's 165.2 TFLOPS, or 7,983.84 tokens a second. Each reply token then reads 13.96 GB of weights, 1.05 GB of output layer and 0.29 GB of cache at the average context of 2,250 tokens, 15.30 GB in all, which takes 21.69 milliseconds at 70% of 1,008 GB/s: 46.1 tokens a second, and 11.07 seconds for the whole reply. Prompts shorter than 176 tokens are held back by reading the weights rather than by the arithmetic.

Why the two phases differ

Samuel Williams, Andrew Waterman and David Patterson's roofline model, published in Communications of the ACM in April 2009, bounds a program by the lesser of a chip's peak arithmetic and its memory bandwidth times the program's operational intensity, the operations it performs per byte it moves. Jared Kaplan and colleagues at OpenAI put a transformer's forward pass at about 2N FLOPs per token in January 2020, with N "excluding all vocabulary and positional embeddings"; their attention term has the same leading order as the one above. A prompt reuses each weight for every one of its tokens and lands on the flat, compute-bound part of the roof. A generated token uses each weight once and lands on the slope.

Reiner Pope and colleagues at Google called the phases prefill and decode in November 2022. They reached 76% of peak FLOPs on prompts in batches of 512 and found that "low-batch-size cost is dominated by weight loading time". In October 2023 Databricks defined model bandwidth utilization as achieved over peak memory bandwidth and measured 66% for MPT-7B on one A100. Microsoft's Splitwise, in November 2023, and DistServe, in January 2024, went as far as running the two phases on separate machines. Apple's November 2025 post on the Neural Accelerators in the M5 shows the split cleanly: time to first token up to four times faster than on the M4, but generation only 19% to 27% faster, in line with 28% more memory bandwidth, 153 GB/s against 120.

The hardware presets

NVIDIA's figures are dense tensor-core rates with 32-bit accumulation, the path PyTorch and vLLM take for BF16 and FP8; the RTX 3090 has no FP8 units. GeForce cards accumulate in 32-bit at half speed, so the 419 TFLOPS often quoted as the RTX 5090's 16-bit rate holds only with sparsity or 16-bit accumulation; without either, 419 is its FP8 rate. NVIDIA's DGX B200 datasheet gives its eight GPUs 64 TB/s, 8 TB/s each, though its HGX B200 summary says up to 62 TB/s. The FLOPs per byte column is the roofline's ridge point. For a 16-bit model it is close to the prompt length where compute takes over: 164 tokens on the RTX 4090, which the output layer moves to 176 in the worked example because it is read in full but multiplied only once.

Apple publishes no GPU throughput for these chips, so their figures are estimates. The M3 Max and M4 Max assume 256 FLOPs per GPU core per clock in 16- or 32-bit, the rate Philip Turner's benchmarks measured on the M1 and M2, at the GPU clocks in Wikipedia's Apple silicon article. Apple calls the M3 Ultra two M3 Max dies, says the M5 Max has over four times the peak GPU compute of the generation before, and builds the M5 Ultra from two M5 Max chips; its own 4.3 and 4.5 times the M3 Ultra would put the M5 Ultra at 121.5 to 127.2 instead. Memory is the largest configuration sold, including the 512 GB M3 Ultra that Apple's current spec page no longer lists. By default macOS lets the GPU use about three quarters of it, which llama.cpp reports as 96 GiB on a 128 GB M4 Max, and sudo sysctl iogpu.wired_limit_mb raises the limit.

Checked against measured speeds

Efficiency is the one figure no spec sheet gives. The default of 70% for both phases sits where measurements cluster: FlashAttention-2 and FlashAttention-3 reach 73% and 75% of peak, and llama.cpp's own benchmarks of Llama 2 7B, entered as a custom model, compare with the estimates like this:

Speeds are tokens a second for a 512-token prompt and a 128-token reply, the NVIDIA runs with flash attention and the Q4_0 estimates at 4.5 bits per weight. The largest misses have causes the estimate leaves out. On NVIDIA cards llama.cpp multiplies quantized weights in 8-bit integers, at up to four times the 16-bit rate, so its prompts beat the estimate. The Ultra chips fall furthest behind on small 4-bit models: llama.cpp's M3 Ultra run reaches 42.2% of bandwidth against the 70% assumed. On an H100 it generates at 31.4%, so a serving engine is the better guide there. To calibrate for your own setup, measure a speed and divide it by the estimate at 100%.

Mixture-of-experts models

A mixture-of-experts model routes each token through a few experts per layer, so it reads far fewer weights than it stores. From each model's configuration, the presets multiply through 3.35 billion parameters in Qwen3 30B-A3B, 22.19 billion in Qwen3 235B-A22B, 37.55 billion in DeepSeek-V3 and 32.86 billion in Kimi K2.6, counting embeddings, which matches the 3.3, 22, 37 and 32 billion their makers quote. A prompt touches more experts the longer it gets, a share of 1 − (1 − k/X)^T when each token picks k of X, so a 10-token prompt reads 27.2% of DeepSeek-V3's routed experts and a 100-token prompt 95.8%.

What the estimate leaves out

The page models one request on one device. Serving engines batch requests so that one read of the weights serves many of them, which raises total throughput far above these figures. Splitting a model across GPUs adds bandwidth at a cost: Databricks measured MPT-7B falling from 66% of bandwidth on one A100 to 47% on two and 30% on four, and llama.cpp's default split, which runs each card's layers in turn, adds memory but not speed to a single request. Speculative decoding, FP4 on Blackwell cards, sliding-window and hybrid attention, and spilling weights to system memory are not modeled. The memory check adds no allowance for the runtime, which the LLM VRAM calculator puts at 20%, and the LLM API cost calculator prices the same tokens from hosted providers.

Figures are planning estimates, not guarantees: real speed depends on the inference engine, its settings, drivers and cooling. This site is not affiliated with any hardware or model maker named above. See the site disclaimer.