Another Incremental Back-up script?
Recently, we published a similar guide on creating an incremental back-up script, which was nice, but we could improve. The retention period of the previous script was determined by changing the interval and the amount of increments. E.g. a daily interval and 14 increments, translated to a retention period 14 days. But what if we want a retention period of a month? or ten years? This would mean 30 or 3650 increments, resp.
Even with incremental back-ups, many increments need lots of disk space. Moreover, it's simply not dynamic, thus not suitable as a proper back-up solution. We've created a new iteration, allowing you to set multiple intervals (.e.g hourly, daily, weekly, monthly, yearly) and set the amount of increments per interval.
This will hopefully be the last back-up script you'll need in a long time. It's the only one I'm using anyways.
TL;DR: The entire script
How it works
INCREMENTS variable stores the amount of increments to save per period. The
DURATIONS variable stores how long — in seconds — a period is, this variable only needs changing if you want to alter the duration or add new periods.
INCREMENTS, you can set the amount to "
0" to exclude the increment. For every increment you include, a folder on the back-up target location is created automatically.
Keep in mind both vars are associative arrays, make sure the formatting is right. If you're interested, Andy Balaam's blogpost is a great explanation. If you're not interested, just look at the current formatting and change as you wish.
Copy the contents of the entire script in a file you name "
backup", store it in "
Don't forget, if you've created an hourly or even shorter period, the script needs to be called more often. Save the file somewhere else and call it accordingly from
/etc/crontab (or any other method you like).
The following variables probably need changing:
BACKUP_LOCAL_DIR: Folder to keep required tracking files;
BACKUP_DIRS: The folders you want to have back when you computer or server dies, don't remove
RSYNC_TARGET: Where the actual back-up should be stored — the remote, possibly off-site, location;
RSYNC_SECRET: Optional, if Rsync profile on the remote server requires a secret;
MYSQL: Either change username/password parameters, or use
MYSQLDUMP: Same as the
The following variables only need checked:
INCREMENTS: Change the amount of increments you want to save in addition to the full back-up per period.
Full and Incremental Back-ups
The first time the script runs, it creates one full back-up and stores it in the
The next run — the first increment — it stores that increment in
. Files modified since the last run are moved to this folder and the latest copy is moved to the "
current" folder. For the amount of given increments for a period, new increments are created every run, e.g
Finally, when the amount of increments has reached, a new full back-up is stored in the "
current" folder. After that, the increment folders are updated one by one.
To make it easier to find files modified before certain date or time, each folder's timestamp is updated to match the time it ran.
Back-up to a local folder or USB disk instead of a remote Rsync server
RSYNC_TARGET variable dictates the remote — preferably off-site location — for the back-up. For example, if your back-up USB disk is mounted at
lsblk to find out where it's mounted) change it as follows:
Don't forget to empty the
RSYNC_SECRET variable, to ensure it all works as it should.
The back-up script keeps track of which increment it last completed, by storing a file per period in the
BACKUP_LOCAL_DIR folder, e.g. "
last_inc_daily", etc. The timestamp of these files is used to determine when that increment was created, to see if the period is elapsed. If you remove these files, or the dir entirely, the script will start with "current" as explained above.
This ensures that if a run was missed — because your laptop was powered off, or because your server was rebooting — the next increment is created as soon as it powers on again, though not sooner than given period duration.
Finally, this folder contains a local copy of all created database dumps.
Is this the final back-up scrip we'll ever create? Probably not, there's always room for improvement. This script allows to create proper back-ups you can rely on, that span big retention periods, without requiring an unhealthy amount of disk space. In our opinion, it's a healthy mix between incremental an full back-ups, allowing for proper disaster recovery.
The current script will only function on Linux systems, or at least systems running Bash. It's unknown if it will run using the Linux Subsystem for Windows 10. Therefore a proper Windows Shell replacement for this script would be a very welcome addition.
Finally, the script is probably not suitable for the less tech-savvy among us, but that — in my humble opinion — might be a pretty good user filter on itself: If you can't get it running, don't use it.
Perfacilis Back-up Service
The Perfacilis Back-up service alerts you when a back-up didn't finish in the scheduled time-frame, or if a lot of data changes at once (which might indicate an encrypter virus). You only pay for the amount of space your back-ups use.
Want to learn more?
- Bump version to 0.7.1
log/ logs/ *.log from exclusion list, it's good practice to back-up log files as well.
current dir per increment folder, only one
current dir inside the root directory will be created.
- Bump version to 0.8.1; Added check to enforce only one running instance, using pidof check.