Remember to audit your backups!
When I migrated my old Arch servers to Debian with ZFS last year I also changed the entire data layout on the arrays. I broke things out into smaller sized chunks to make ad hoc backups run swifter, modularize things so I stood a change of remembering what was what just by doing a directory listing, that kind of thing.
All week I've been working on finally getting a web site going for the root of this domain. Basically composed of trying out a bunch of hugo themes, loading some markdown files from another live site into the content directory to see how it renders, yadda, yadda.
In the process of doing this, at some point I must have fat fingered something and deleted/moved the source files for the active site. I only discovered this when trying to copy the posts from the active site to a new one I was testing with a new theme.
rsync -av ~oldsite/site/content/posts ~newsite/site/content/.
"... No such file or directory ..."
Wat...?
Ok, whatever, start looking at the incremental backups... hmm, not there... not there... not there... wat? List the backup crontabs again... no reference. Other host with backups that don't match this pattern... not there... WTAF?!
The location where I have my web dev stuff located is not backed up at all!
In the root of the array where this is located there are 4 other directories that are backed up but for some reason this one is not. Either I had a blind spot on it or I forgot or I didn't set it up because there were no web projects when the server was set up or...
I got lucky - had made a bunch of copies while experimenting during the week and was able to find "out of band" copies of the files. But, if during testing I had blown away all my testing data without knowing there were no backups I would have not been happy!
Anyway...
Audit your backups!
Test restoring from your backups!