3-2-1 Backup & GFS Retention Storage Calculator

Enter how much data you protect, how fast it grows and changes, how well your backups compress, and the grandfather-father-son retention you keep. The calculator works out the disk each backup copy needs and the total for every copy the 3-2-1 rule asks for. It steps through the plan one day at a time and reports the most a copy ever holds. The defaults describe 10 TB growing 10 percent a year and changing 5 percent a day, compressed 2 to 1, kept for 14 days with 4 weekly, 12 monthly and 3 yearly fulls over three years. Each copy needs 120.94 TB at its peak, and the two backup copies need 241.88 TB.

How the storage is worked out

One backup runs every day. A full copies the whole data set; an incremental copies only what changed since the day before, so it is useless without the full that starts its chain and every incremental in between. The model takes a full on the first day and on every seventh day after it, an incremental on the other days, and lets the data set grow a little each day:

data on day k = S × (1 + g)^{k ÷ 365}

A full stores the data set as it was on its day and an incremental stores the daily change rate c times it. Retention then works in four tiers. The daily tier keeps every backup from the last D days, the weekly tier the newest W fulls, the monthly tier the newest M fulls that were the first of their calendar month, and the yearly tier the newest Y first fulls of a year. A full that belongs to several tiers is stored once. Because an incremental needs its chain, the oldest daily backup keeps everything back to the full that started it, which is how Veeam's documentation describes removing a forward-incremental chain.

A copy needs the most room just after a backup finishes and before retention deletes what has aged out, because the new backup is written first. At that moment the daily tier can hold up to D + 7 backups, and each long-term tier can hold its newest full alongside the one it is about to replace. The calculator finds the busiest day of each year and reports the largest:

per copy = S × (F + c × I) ÷ R

F and I add up the fulls and incrementals on disk at that moment, each as a multiple of the starting data set S at the size it had on its own day, and R is the compression and deduplication ratio. Every backup copy holds the same restore points, so all copies together need the number of copies times the figure for one. In the table of tiers, each row counts only what that tier adds to the rows above it: a full the daily chain already holds costs nothing more when it is also a weekly or monthly full.

A worked example with the defaults

Ten TB growing 10 percent a year becomes 11 TB after one year, 12.1 TB after two and 13.31 TB after three. Each copy peaks on 6 October of year 3, the day the first full of October is written. September of that year had 5 fulls, so September's monthly full, taken on the 1st, is no longer one of the four newest weekly fulls and is kept as a monthly alone. The full from 8 September, the oldest weekly, and the one from 7 October of year 2, the oldest monthly, are still on disk, because retention runs only after the new backup is safe.

The copy then holds 31 restore points: the last 15 days of backups, 3 of them fulls, two more weekly fulls, 12 monthly fulls, and the yearly fulls of years 1 and 2, since the yearly full of year 3 is also January's monthly. Its 19 fulls run from 10 TB, the oldest, to 13.01 TB, the newest, and each incremental holds about 0.65 TB. Together they come to 241.89 TB of backups, which a ratio of 2 to 1 stores in 120.94 TB. Retention then deletes the two expiring fulls, leaving 17 fulls in 108.57 TB until the next peak. The busiest days of years 1 and 2 needed 87.96 TB and 105.39 TB, and two copies need 241.88 TB at the peak.

Why the monthly fulls cost the most

With active fulls, every long-term restore point is a complete copy of the data. At the defaults the 12 monthly fulls take 74.13 TB of the 120.94 TB, more than the daily chain, the weekly fulls and the yearly fulls together. Keeping 6 monthly fulls instead cuts each copy to 90.82 TB, while dropping the yearly fulls saves 10.5 TB. Weekly fulls can even be free: with 14 days of daily backups the two newest weekly fulls are already in the daily chain, so keeping none, one or two of them gives the same 112.27 TB.

Compression and deduplication divide everything by their ratio. Veeam's best-practice guide puts the typical daily change rate of a mid-size or enterprise environment at between 2% and 5%, with database servers much higher, and says data reduction from compression and deduplication is usually 2:1 or more but should be estimated conservatively.

Where incremental backups and the 3-2-1 rule came from

The Seventh Edition of the UNIX Programmer's Manual, dated January 1979, describes dump as an incremental file system dump. Level 0 dumps the entire file system, and a higher level dumps only the files modified since the last dump at a lesser level, with the dates recorded in /etc/ddate. Grandfather-father-son retention keeps such backups on several schedules at once. Veeam's software, for example, marks full backup files with weekly, monthly or yearly flags, keeps each flag for its own period, and lets a file be deleted only once it has no flag left.

The 3-2-1 rule settles how many copies to keep. A 2012 paper written for US-CERT, Data Backup Options by Paul Ruggiero and Matthew A. Heckathorn of Carnegie Mellon University, states it as keeping 3 copies of any important file, 1 primary and 2 backups, on 2 different media types, with 1 copy stored offsite. Its footnote credits Peter Krogh's The DAM Book: Digital Asset Management for Photographers, in its second edition of 2009. The data you work on counts as the first copy, so the rule needs two backup copies, which is the calculator's starting point; the third option adds a copy kept offline.

Tools differ in how they count a backup that several rules would keep. restic keeps the most recent snapshot in each day, week, month or year it is asked to keep, lets one snapshot satisfy several rules at once, and runs its weeks from Monday to Sunday. Borg applies its rules from the shortest period to the longest, and a backup kept by one rule does not count towards the next. The calculator counts a full once for every tier it belongs to, as Veeam's flags and restic's rules do. Both tools also deduplicate, storing a piece of data once however many backups contain it. Venti, which Sean Quinlan and Sean Dorward of Bell Labs presented in 2002, did the same for blocks: it identifies each block by a hash of its contents, coalesces duplicate copies, and never overwrites what it has written.

What the figures leave out

The model is for active fulls, where every full is a complete copy. Repositories that build synthetic fulls by block cloning, such as Veeam's Fast Clone on ReFS or XFS, and programs that keep only chunks they have not seen before, such as Borg, or only changed blobs, such as restic, store long-term restore points for a fraction of that. Enter the reduction ratio you see in practice, or read these figures as a ceiling. Sizes are in decimal units, as drives are sold: the data storage converter shows them in the binary units an operating system reports, and the RAID and ZFS capacity calculator works out how many drives hold a copy. Seeding the offsite copy takes time too, which the download time calculator works out from its size and your upload speed.

Retention here runs after every backup; a job that prunes once a week holds up to six more days of backups at its peak. Veeam's guide also advises room for one-off full backups and for transforming backup chains, at least the size of a full backup multiplied by 1.25. The calendar starts the plan with a full on 1 January; in the setups tested, starting it on another day of that week never needed fewer fulls at the peak and sometimes needed one more. Immutability periods, which keep backups past their retention, and the repository's own metadata are not included.

Frequently asked questions

How much backup storage do I need?

Enough for the most a copy ever holds, which comes just after a backup and before retention deletes what has aged out. For 10 TB growing 10% a year and changing 5% a day, compressed 2 to 1 and kept for 14 days with 4 weekly, 12 monthly and 3 yearly fulls, each copy peaks at 120.94 TB in year 3, and two copies need 241.88 TB.

What is the 3-2-1 backup rule?

Keep 3 copies of any important file, 1 primary and 2 backups, on 2 different media types, with 1 copy offsite. That is how a 2012 paper written for US-CERT states it, citing Peter Krogh's The DAM Book. The data you work on is the first copy, so the rule needs two backup copies, and each one needs the full storage this calculator works out.

What is GFS retention?

Grandfather-father-son retention keeps backups on several schedules at once: every backup from the last few days, plus full backups kept weekly, monthly and yearly. A full that belongs to several tiers is stored once. With the defaults, the 4 weekly fulls add only 2 to the daily chain, because the 2 newest are already in it.

Why do monthly fulls take most of the space?

Each one is a complete copy of the data. With the defaults, the 12 monthly fulls take 74.13 TB of the 120.94 TB each copy needs, more than the other tiers together, and keeping 6 instead cuts each copy to 90.82 TB. Repositories that clone blocks or store each chunk once keep long-term fulls far more cheaply, so enter the reduction ratio they actually achieve.

Why does 14 days of retention keep more than 14 backups?

An incremental is useless without the full and the incrementals before it, so the oldest daily backup keeps its whole chain, which adds up to six backups with a weekly full. The new backup is also written before old ones are deleted. With daily retention alone and the other defaults, 14 days peaks at 21 restore points: 3 fulls and 18 incrementals.

How much do compression and deduplication save?

A ratio of R divides the storage by R. With the defaults each copy needs 241.89 TB at 1 to 1, 120.94 TB at 2 to 1 and 48.38 TB at 5 to 1. Veeam's best-practice guide says data reduction from compression and deduplication is usually 2:1 or more, and advises staying conservative when estimating space.

Sources

1. Data Backup Options, US-CERT, 2012
2. UNIX Programmer's Manual, Seventh Edition: dump(1M), Bell Laboratories, in the Unix History Repository, 1979
3. Retention for Forward Incremental Backup, Veeam Backup & Replication 9.0 User Guide
4. Long-Term Retention Policy (GFS), Veeam Backup & Replication 11 User Guide
5. Repository storage, Veeam Best Practice Guide
6. Removing backup snapshots, restic documentation
7. References: Backups and Deduplication, restic documentation
8. borg prune, Borg documentation
9. Borg Documentation, BorgBackup
10. Venti: a new approach to archival storage, USENIX FAST '02, 2002