LLM VRAM & GPU Sizing Calculator

Pick a model, the format its weights are stored in, how many tokens of context each request holds and how many requests run at once. The calculator adds up the weights, the KV cache those requests keep and an allowance for the runtime, then shows how many of ten common GPUs it takes to hold the total. Choose Custom architecture to type in any model's numbers from its config.json. The math runs entirely in your browser and nothing you type is sent anywhere.

How the estimate works

Inference memory has three parts: the weights, fixed by the model and its format; the KV cache, which grows with every token in flight because each layer keeps a key and a value for every token it has seen; and an allowance for everything else, from activations to the CUDA context.

total = parameters × bits ÷ 8 + 2 × layers × KV heads × head size × bytes × tokens × sequences + overhead

The overhead is a share of the weights, 20% unless you change it. That is the rule of thumb EleutherAI published in its Transformer Math 101 guide in April 2023: inference takes about 1.2 times the memory of the model itself, the extra being "≤ 20%" in the authors' experience. Their rule covers the forward pass, not the cache, so the KV cache is counted separately here.

The defaults run Llama 3.1 8B in BF16, two bytes a parameter, with an 8,192-token context and one sequence. Its 8,030,261,248 parameters take 14.96 GiB. Each token costs 2 × 32 layers × 8 KV heads × 128 × 2 bytes, which is 128 KiB, so 8,192 tokens add exactly 1 GiB. The allowance is 2.99 GiB and the total 18.95 GiB: one 24 GB card filled to 79%, and more than a 16 GB card holds. The formula assumes every layer caches every token, and the presets are limited to models where that is true.

Where the KV cache comes from

A transformer generates one token at a time, each attending to every token before it, and caching their keys and values trades recomputation for memory. NVIDIA's November 2023 guide to inference optimization gave the formula this page uses with a Llama 2 7B case, 1 × 4096 × 2 × 32 × 4096 × 2 bytes, "~2 GB" for one 4,096-token sequence; in binary units it is exactly 2 GiB. The PagedAttention paper from the vLLM team, presented at SOSP in October 2023, put OPT-13B at 800 KB per token and as much as 1.6 GB for one 2,048-token request, and found that serving it on an A100 left about 65% of memory to the weights and close to 30% to that per-request state.

Model designers have spent years shrinking it. Noam Shazeer's multi-query attention, in 2019, let every query head share a single key and value head. Grouped-query attention, from Joshua Ainslie and colleagues in 2023, settled in between with a few shared heads; Meta used it in Llama 2's 34B and 70B models that July and across both Llama 3 sizes, 8B and 70B, in April 2024. DeepSeek-V2 went further in May 2024 with multi-head latent attention, which caches one compressed vector and a small positional key per layer instead of full keys and values. DeepSeek's own table puts DeepSeek-V3 at 70.272 KB a token against 516.096 KB for Llama 3.1 405B; this page shows the same bytes as 68.63 KiB and 504 KiB.

Fewer bits per weight

Quantization stores weights in fewer bits. LLM.int8(), from Tim Dettmers and colleagues in August 2022, cut the memory needed for inference in half "while retaining full precision performance". GPTQ followed that October, reducing GPT models with 175 billion parameters to 3 or 4 bits per weight in about four GPU hours, and AWQ in June 2023, later the best paper at MLSys 2024. NVIDIA had announced FP8 tensor cores for the H100 in March 2022, and Paulius Micikevicius and colleagues set out the E4M3 and E5M2 encodings that September. For local use, Georgi Gerganov's first llama.cpp commit landed on 10 March 2023, its k-quants in June and the GGUF file format in August.

Bits per weight is not the whole story. The GPTQ authors kept "the embeddings and the output layer" in FP16, vLLM's FP8 path quantizes every linear layer "except for the final lm_head", and AWQ and GPTQ checkpoints store a 16-bit scale and a 4-bit zero point for every group of weights, 128 by default in Hugging Face's settings for both. The FP8 and INT4 formats below follow those rules. GGUF files quantize everything, so their figures are flat: Q8_0 is exactly 8.5 bits, and Q4_K_M mixes block types to about 4.89 bits by llama.cpp's measurement on Llama 3.1 8B.

The rule matters most for small models with large vocabularies. Llama 3.1 8B in INT4 comes to 5,727,460,432 bytes, or 5.33 GiB, within 0.01% of the 5,727,938,576-byte AWQ download on Hugging Face, where a flat 4 bits a weight would claim 3.74 GiB.

The model presets

Each preset takes its parameter count from Hugging Face and its attention settings from its config.json, checked in October 2026. The exception is Qwen3 1.7B, whose files store its tied output layer twice: Hugging Face counts 2.03 billion, and the preset counts the shared matrix once, as memory holds it. Mixture-of-experts models count every expert, because any of them can be routed a token. Three counts include a vision encoder, and DeepSeek's leaves out the extra prediction layer it ships for speculative decoding, matching the 671 billion DeepSeek quotes for its main model.

GPU memory as it is sold

Memory chips are made in powers of two, and the JEDEC standards that govern them keep a gigabyte at 2³⁰ bytes, as the data storage converter explains. NVIDIA's own MIG user guide shows an 80 GB H100 reporting 81,559 MiB, more than a decimal 80 GB could hold, so this page reads every card's GB as GiB and reports its results in GiB to match. Model downloads use decimal gigabytes, and the gap is about 7%: the GPTQ authors put GPT-3's 175 billion 16-bit parameters at "326GB (counting in multiples of 1024)", where decimal units give 350 GB.

The RTX 3090, Radeon RX 7900 XTX and L4 also carry 24 GB, the Radeon AI PRO R9700 32 GB and the A100 80GB 80 GB. Blackwell Ultra and Rubin are left out because NVIDIA's own pages disagree on their memory per GPU, and Apple's unified-memory Macs and the DGX Spark because the share their GPU may use is decided at run time.

The count in each row is a floor. Tensor parallelism splits every layer's attention heads across the GPUs, and vLLM refuses a GPU count that does not divide them: the total number of attention heads "must be divisible by tensor parallel size". Its documentation suggests pipeline parallelism when the count does not divide evenly. Once the GPU count passes a model's KV heads, vLLM copies those heads rather than splitting them, so the cache outgrows this estimate. And vLLM claims 92% of each GPU by default, so treat a row filled beyond that as a cue to raise its memory setting or add a card.

Hybrid models and what the estimate leaves out

Many of 2026's open models break the every-layer assumption. gpt-oss alternates full-attention layers with layers that see only the last 128 tokens, and Gemma 4's larger models limit five layers in six to a window of 1,024. Qwen3.5 and 3.6 and NVIDIA's Nemotron 3 replace most attention layers with linear or Mamba layers that keep a fixed-size state, and DeepSeek V4 compresses its cache. None is a preset, because the memory those layers take depends on the inference engine. For a close estimate, choose Custom architecture and enter only the full-attention layers: Qwen3.6 27B caches 16 of its 64 layers, each with 4 KV heads of 256, which comes to 64 KiB a token, and the remaining layers add a fixed amount per sequence that this leaves out.

The 20% allowance is generous for large models split across several GPUs. Llama 3.3 70B in BF16 comes to 160.20 GiB, three H100s; at 10% it is 147.06 GiB, which fits two at 92%. Lower the allowance once you have measured your own engine. It also leaves out vision-encoder activations and speculative-decoding draft models, and fine-tuning has its own calculator.

Figures are planning estimates, not guarantees: real memory use depends on the inference engine, its settings and the driver. This site is not affiliated with any model or hardware maker named above. See the site disclaimer.

Frequently asked questions

How much VRAM does Llama 3.3 70B need?

With an 8,192-token context and one sequence, 160.20 GiB in BF16, 83.70 GiB in FP8 and 46.94 GiB in INT4, each including the 20% overhead allowance. That is three H100s, two or one, and in INT4 it also fits one 48 GB L40S at 98%. Every extra token of context adds 320 KiB.

Can I run a 70B model on two RTX 4090s?

In INT4, only just. Llama 3.3 70B with an 8,192-token context needs 46.94 GiB, which fills two 24 GB cards to 98%. Doubling the context to 16,384 tokens raises it to 49.44 GiB and calls for a third card. GGUF Q4_K_M spends more bits per weight than INT4 on a model this size, so it needs more room still.

Why does context length use so much memory?

Every layer keeps a key and a value for every token of every sequence in flight. Llama 3.1 8B stores 128 KiB per token, so a 131,072-token context adds 16 GiB of cache to 14.96 GiB of BF16 weights, and the 33.95 GiB total needs two 24 GB cards. Switching the cache to 8-bit FP8 halves it to 8 GiB.

Do mixture-of-experts models need less VRAM?

No. Only a few experts run for each token, which saves compute, but any of them can be chosen, so all of them stay in memory. Qwen3 30B-A3B needs its full 30.5 billion parameters, 56.87 GiB in BF16, even though the A3B in its name refers to the roughly 3 billion that are active for each token.

How much memory does fine-tuning need?

Far more than inference, and this site's LLM fine-tuning VRAM calculator estimates it. The ZeRO paper counts 16 bytes per parameter for mixed-precision training with Adam: 2 for 16-bit weights, 2 for gradients and 12 for the optimizer's 32-bit copy, momentum and variance. For GPT-2's 1.5 billion parameters that is at least 24 GB, before activations, which grow with batch size and sequence length.

Why are the results in GiB rather than GB?

Because GPU memory is counted in binary units. NVIDIA's documentation shows an 80 GB H100 reporting 81,559 MiB, which is 79.65 GiB, so the label means 80 GiB less a small driver reserve. Model downloads use decimal gigabytes instead: Llama 3.1 8B's 16.06 GB of BF16 weights is the 14.96 GiB shown here.

Sources

1. Transformer Math 101, EleutherAI, 2023
2. Mastering LLM Techniques: Inference Optimization, NVIDIA Technical Blog, 2023
3. Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv, 2023
4. Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures, arXiv, 2025
5. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, arXiv, 2022
6. Getting Started with MIG, NVIDIA
7. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, arXiv, 2019