LLM Fine-Tuning & Training VRAM Calculator

Pick a model, a fine-tuning method and an optimizer, then the sequence length, micro-batch and GPUs you plan to train with. The calculator adds up what each GPU holds: weights, gradients, optimizer state and activations. It compares full fine-tuning with LoRA and QLoRA at three ranks, and finds the fewest of ten common GPUs that fit at your ZeRO stage. Choose Custom architecture to enter any dense model from its config.json. The math runs entirely in your browser and nothing you type is sent anywhere.

How the estimate works

Training memory has two halves: the model states, as the ZeRO paper calls the weights, gradients and optimizer states, and the activations each layer keeps for the backward pass.

memory per GPU = (weights + gradients + optimizer states) ÷ ZeRO split + activations

Full fine-tuning gives every parameter a 16-bit weight, a 16-bit gradient and optimizer state. LoRA freezes the base and trains small adapters, so only they carry gradients and optimizer state; QLoRA does the same over a 4-bit base. Results are in GiB, about 7% more than decimal GB as the data storage converter shows, and each card's GB is read as GiB, as the LLM VRAM calculator for inference explains.

The defaults run LoRA at rank 16 on Llama 3.1 8B, with adapters on every linear layer, AdamW, a 2,048-token sequence and gradient checkpointing on one GPU. The adapters hold 41,943,040 trainable parameters, 0.52% of the model. The frozen BF16 base and the adapters take 15.04 GiB, gradients 0.08 GiB, optimizer states 0.47 GiB and activations 1.76 GiB, for 17.34 GiB: one 24 GB card filled to 72%. Full fine-tuning the same model needs 121.42 GiB, and QLoRA at rank 16 needs 7.7 GiB.

Sixteen bytes a parameter

Mixed-precision training, from Paulius Micikevicius and colleagues in October 2017, runs the forward and backward passes in 16 bits but keeps "an FP32 master copy of weights" for the optimizer step. The ZeRO paper from Microsoft, first posted in October 2019 and presented at SC20, put a number on it. Each parameter has 2 bytes of 16-bit weight and 2 of gradient, plus a 32-bit copy, momentum and variance at 4 bytes each: "Mixed-precision Adam has K=12", for 16 bytes in all. GPT-2's 1.5 billion parameters fit in 3 GB as 16-bit weights but need at least 24 GB to train.

Tim Dettmers and colleagues cut the optimizer's share in October 2021 with 8-bit optimizers, presented at ICLR 2022. In their count, Adam's states take 8 GB for a model with a billion parameters in 32 bits, and 2 GB in 8 bits. The QLoRA paper added paged optimizers, which use NVIDIA's unified memory to move optimizer states to CPU RAM only when the GPU runs out, so paging survives a spike but does not shrink the count.

The page counts paged 8-bit AdamW at 2 bytes a parameter, because bitsandbytes updates each weight in its own precision and keeps no 32-bit copy. EleutherAI's Transformer Math 101 counts 6 bytes because it keeps that copy, so add 4 bytes a trainable parameter if your setup does. The page also leaves out the 32-bit scale bitsandbytes stores for every 256 values of each state, about 0.03 bytes a parameter.

LoRA and QLoRA

LoRA, from Edward Hu and colleagues at Microsoft in June 2021, freezes each weight matrix and learns its update as the product of two thin matrices: B, with d rows and r columns, and A, with r rows and k columns. Each adapted matrix adds r × (d + k) trainable parameters. Against GPT-3 175B fine-tuned with Adam, the authors reported 10,000 times fewer trainable parameters and a third of the GPU memory. Most of their experiments adapted only the query and value projections, and rank 4 on both across GPT-3's 96 layers comes to 18,874,368 parameters, the "18M" budget in the paper.

QLoRA, from Tim Dettmers and colleagues at the University of Washington in May 2023, stores the frozen base in 4-bit NormalFloat, a data type built for normally distributed weights, with one scale for every 64 weights. Double quantization stores those scales in 8 bits, cutting their cost from 0.5 to 0.127 bits a parameter, so each weight takes about 4.127 bits. The adapters stay in BF16. The authors fine-tuned a 65-billion-parameter model on a single 48 GB GPU. They found that adapters on all linear layers were needed to match full fine-tuning, and that the rank did not affect performance.

The QLoRA option quantizes every linear layer and keeps the embeddings and output layer at 16 bits, as Hugging Face Transformers does: it replaces only linear layers and skips the output layer. A script that calls PEFT's prepare_model_for_kbit_training also converts every 16-bit parameter that isn't 4-bit to 32 bits, which for Llama 3.1 8B adds 1.96 GiB.

Activations and gradient checkpointing

Vijay Korthikanti and colleagues at NVIDIA counted activations in May 2022 for 16-bit training. In every layer, each token costs 34 bytes per unit of hidden size, plus 5 bytes per attention head per token of context for the attention scores. FlashAttention, from Tri Dao and colleagues that same month, never stores the scores and needs memory linear in the sequence length, so the page drops that term. The loss also keeps the logits in 32 bits, 4 bytes per token for every entry in the vocabulary. That is a lot for Llama 3's 128,256 entries and Qwen3's 151,936.

activations = 34 × s × b × h × layers + 4 × s × b × (h + vocab)

Here s is the sequence length, b the micro-batch and h the hidden size. Gradient checkpointing keeps only each layer's input, 2 bytes per token per unit of hidden size, and recomputes the rest during the backward pass one layer at a time. Tianqi Chen and colleagues described the approach in April 2016 at the cost of one extra forward pass. At the defaults it cuts activations from 9.51 GiB to 1.76 GiB, of which the output layer and its logits take 1.01 GiB. The micro-batch sets the peak, because gradient accumulation runs micro-batches one after another without adding memory.

Adapters do not shrink activations, because gradients still flow back through every frozen layer to reach the adapters below it. QLoRA's authors found that most fine-tuning memory "comes from activation gradients and not from the learned LoRA parameters". Korthikanti's count assumes a GPT-style layer whose MLP is four times the hidden size. Llama, Qwen and Granite use gated MLPs of other widths, which the count does not model, so treat the activations as approximate.

ZeRO stages and the GPU table

DeepSpeed's ZeRO splits the model states across GPUs training in data parallel. Stage 1 partitions the optimizer states, which for Adam are "32-bit weights, and the first, and second moment estimates". Stage 2 also partitions the gradients, and stage 3 partitions the 16-bit weights as well, gathering them during the forward and backward passes. The paper's example, a 7.5-billion-parameter model on 64 GPUs, needs 120 GB per GPU without ZeRO, 31.4 GB with stage 1, 16.6 GB with stage 2 and 1.88 GB with stage 3, and this page gives the same bytes. Activations are not split, since each GPU runs its own micro-batch. With LoRA and QLoRA, stages 1 and 2 split only the adapters' state, and stage 3 splits the frozen base too.

The GPU table counts the fewest of each card that fit at the chosen stage: 14 H100s to fully fine-tune Llama 3.3 70B with stage 3. Stage 2 cannot fit that job on H100s at any count, because each GPU still holds all 131.42 GiB of 16-bit weights.

Each count is a floor. DeepSpeed's own estimator keeps a 32-bit copy of the gradients, 4 more bytes a parameter, and adds 4 bytes for every parameter of the largest layer under stage 3. NVIDIA's Megatron-LM counts 18 bytes a parameter for the same 32-bit gradients. Communication buffers, the CUDA context, fragmentation and temporary tensors come on top, so leave headroom on any row filled close to 100%. To weigh renting GPUs against owning them, see the cloud vs. self-hosted calculator.

The model presets

Each preset's layer sizes come from its config.json, checked in October 2026, and every parameter count can be rebuilt from them exactly. The four smallest models tie their input and output embeddings, so the shared matrix counts once, though Qwen3 1.7B's files store it twice. Mixture-of-experts models, models with a vision encoder and Phi-4, which fuses its projections, are not presets, because their adapter counts depend on which experts, encoders and fused layers a framework adapts.

Figures are planning estimates, not guarantees: real memory use depends on the training framework, its kernels and settings, and the driver. This site is not affiliated with any model, framework or hardware maker named above. See the site disclaimer.

Frequently asked questions

How much VRAM does it take to fine-tune Llama 3.1 8B?

With a 2,048-token sequence, a micro-batch of one and gradient checkpointing on one GPU: 121.42 GiB for full fine-tuning with AdamW, 17.34 GiB for LoRA at rank 16 and 7.7 GiB for QLoRA at rank 16, with adapters on every linear layer. LoRA fits one 24 GB card at 72%. Full fine-tuning needs a card of at least 141 GB, such as an H200, or ZeRO across several GPUs.

Can I fine-tune a 70B model on one GPU?

With QLoRA, yes. Llama 3.3 70B at rank 16 with a 2,048-token sequence needs 43.93 GiB, which fits one 48 GB L40S at 92%, and paged 8-bit AdamW brings it to 42 GiB. QLoRA's authors fine-tuned a 65-billion-parameter model on a single 48 GB GPU the same way. Plain LoRA keeps the frozen base at 16 bits, 131.42 GiB of it, so it needs two H100s even with ZeRO stage 3.

Why does full fine-tuning need 16 bytes per parameter?

Mixed-precision training with AdamW keeps 2 bytes of 16-bit weights and 2 bytes of gradients for every parameter, plus a 32-bit copy of the weights and 32-bit momentum and variance at 4 bytes each. The ZeRO paper writes this as 2 + 2 + 12 = 16 bytes. Llama 3.1 8B's 8,030,261,248 parameters therefore take 119.66 GiB before activations, against 14.96 GiB for the weights alone.

Does LoRA reduce activation memory?

No. The backward pass still runs through every frozen layer to reach the adapters below it, so activations are the same for full fine-tuning, LoRA and QLoRA. Gradient checkpointing is what shrinks them: for Llama 3.1 8B at 2,048 tokens it cuts them from 9.51 GiB to 1.76 GiB. At 32,768 tokens they reach 28.16 GiB even with checkpointing.

Which ZeRO stage should I use?

The lowest one that fits, because stage 3 has to gather every layer's weights during each forward and backward pass. Full fine-tuning Llama 3.1 8B with AdamW on eight GPUs takes 121.42 GiB per GPU without ZeRO, 42.89 GiB with stage 1, 29.81 GiB with stage 2 and 16.72 GiB with stage 3. Stage 1 fits 48 GB cards, stage 2 fits 32 GB cards and only stage 3 fits 24 GB cards.

How much does paged 8-bit AdamW save?

It stores momentum and variance in 1 byte each, 2 bytes a parameter where 32-bit AdamW with its 32-bit copy of the weights takes 12. For full fine-tuning of Llama 3.1 8B, the optimizer states fall from 89.75 GiB to 14.96 GiB and the total from 121.42 GiB to 46.63 GiB. With LoRA they start small: 0.47 GiB becomes 0.08 GiB. Paging moves states to CPU memory only when the GPU runs out, so it prevents crashes rather than shrinking the total.

Sources

1. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, Rajbhandari, Rasley, Ruwase and He, SC20, 2019
2. Mixed Precision Training, Micikevicius et al., ICLR 2018, 2017
3. 8-bit Optimizers via Block-wise Quantization, Dettmers, Lewis, Shleifer and Zettlemoyer, ICLR 2022, 2021
4. LoRA: Low-Rank Adaptation of Large Language Models, Hu et al., ICLR 2022, 2021
5. QLoRA: Efficient Finetuning of Quantized LLMs, Dettmers, Pagnoni, Holtzman and Zettlemoyer, NeurIPS 2023, 2023
6. Reducing Activation Recomputation in Large Transformer Models, Korthikanti et al., NVIDIA, 2022
7. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, Dao, Fu, Ermon, Rudra and Ré, 2022
8. Training Deep Nets with Sublinear Memory Cost, Chen, Xu, Zhang and Guestrin, 2016
9. Transformer Math 101, EleutherAI, 2023
10. Zero Redundancy Optimizer tutorial, DeepSpeed
11. 8-bit optimizers and paged optimizers, bitsandbytes documentation
12. LoRA developer guide, Hugging Face PEFT
13. bitsandbytes quantization, Hugging Face Transformers