Evilham.com
ZFS replication tools
Introduction
Back when I first started using FreeBSD and ZFS, I needed tools with an extremely low barrier of entry, that did their job well.
When it comes to creating and pruning snapshots, the job was first done by
sysutils/zfs-periodic.
Over time however, even with minor improvements on my part, it has come short for my current needs.
As inspired by Dan Langille’s wonderful blog (Dan always writes great blog posts, that end up being lovely complementary documentation), I document here my somewhat bumpy road when it comes to ZFS replication tools and why you might want to use something different at each step.
Table of contents
Quick reminder of ZFS
ZFS is nowadays (at least on FreeBSD and Linux) actually OpenZFS. Let’s see how they define it:
OpenZFS is an advanced file system and volume manager which was originally developed for Solaris and is now maintained by the OpenZFS community. This repository contains the code for running OpenZFS on Linux and FreeBSD.
Amongst the many beautiful things of ZFS, there is the fact that it works in a
Copy on Write fashion, which, amongst other things, enables it to create
snapshots in an instantaneous fashion.
That is, we can assign a name to a known state with, e.g.
zfs snapshot -r zroot@goodstate.
Keep on working and, if we get on to a bad state, discard any changes made
since then by issuing a rollback.
This means, we can have a very handy way of having real-life save points, and work in a more relaxed fashion.
Together with zfs-send(8) and zfs-receive(8), we
quickly have the basis for a remote backup system that is incremental.
Now that ZFS data set encryption (zfs-load-key(8)) is a thing, it too can
be encrypted.
Snapshot creation and prunning with zfs-periodic(8)
Now, back in ~2019, I was mostly a Linux user that was jumping to FreeBSD and the last thing I wanted was to fiddle too much with a new file system.
Luckily, there is a very simple way to manage ZFS snapshots in an automatic
fashion right in the ports system: zfs-periodic.
It profits from the periodic(8) system, to run at regular
intervals, and gets configured in a minute with:
$ pkg install -y zfs-periodic
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
[1/1] Fetching zfs-periodic-1.0.20130213.pkg: 100% 2 KiB 2.3kB/s 00:01
Checking integrity... done (0 conflicting)
[1/1] Installing zfs-periodic-1.0.20130213...
[1/1] Extracting zfs-periodic-1.0.20130213: 100%
=====
Message from zfs-periodic-1.0.20130213:
--
In order to enable periodic snapshots you need
to add these lines to your /etc/periodic.conf
hourly_output="root"
hourly_show_success="NO"
hourly_show_info="YES"
hourly_show_badconfig="NO"
hourly_zfs_snapshot_enable="YES"
hourly_zfs_snapshot_pools="tank"
hourly_zfs_snapshot_keep=6
daily_zfs_snapshot_enable="YES"
daily_zfs_snapshot_pools="tank"
daily_zfs_snapshot_keep=7
weekly_zfs_snapshot_enable="YES"
weekly_zfs_snapshot_pools="tank"
weekly_zfs_snapshot_keep=5
monthly_zfs_snapshot_enable="YES"
monthly_zfs_snapshot_pools="tank"
monthly_zfs_snapshot_keep=2
To get hourly snapshots you also need to add
something like this to /etc/crontab:
2 * * * * root periodic hourly
We literally do that (adding the crontab(5) entry and configuring
zfs-periodic(8) in /usr/local/etc/periodic.conf (*)), and we have a simple
system that takes hourly, daily, weekly and monthly snapshots.
Half way there with zero effort, this is what has kept me using it for years.
(*): I prefer to use
/usr/local/etc/periodic.conffor things outside the base system. This is a matter of taste and it helps easen administration.
Adding quarter_hourly
Given how cheap these snapshots are, I (and people at work) started wanting to have them more often, which in my case meant every 15 minutes.
Luckily, periodic(8) supports arbitrary directories:
directory An arbitrary directory containing a set of executables to be
run.
However, it wasn’t as simple as:
mkdir /usr/local/etc/periodic/quarter_hourly/
ln -s /usr/local/etc/periodic/daily/000.zfs-snapshot \
/usr/local/periodic/quarter_hourly/000.zfs-snapshot
Because the actual file in there depends on the name of the directory, so it
has to be customised.
And also because the /usr/local/bin/zfs-snapshot script that gets installed
needs to understand all the periodic.conf settings.
Luckily making these changes was fairly simple.
Since I won’t be using zfs-periodic(8) any longer, and the patch has been
more than tested over the years, I’m opening a
Merge Request, which you can use to benefit from these
changes.
Take into account that it’ll also need a corresponding crontab(5) entry:
*/15 * * * * root periodic quarter_hourly
ZFS Replication with zxfer(8)
Snapshots take care of local mess ups, but they don’t serve as a backup.
For that, I took to zxfer, which is simple to use and has worked great with some caveats.
A lovely thing of zxfer(8) is that it supports an rsync(1) mode, that
allows us to back up remote systems that do not support ZFS, to a ZFS-based
system that takes care of snapshots and so on.
The caveats
Mostly, even with a pull-based approach, zxfer(8) doesn’t protect much
against snapshots disappearing on the source, which means that there is a
somewhat plausible scenario where pulling data results in breaking certain
retention policies.
While there is a -g flag to protect old snapshots, it means that we have to
replicate snapshot pruning on the destination system, but cannot create
snapshots as they could collide with the source snapshots.
That means zfs-periodic(8) and zxfer(8) are better than nothing and work
decently, but have important limitations.
Encrypted datasets
When encrypted datasets got added to OpenZFS, I realised that
we need to use zfs send -w, or else the backup will happen unencrypted.
So I wrote a patch for zxfer to do just that.
You can help land that patch in zxfer(8), it is basically
only missing the manpage and USAGE texts, but my manpage syntax is rusty
and time is not an abundantly available commodity.
I’d love to help you land this change though!
Snapshot creation and pruning with sanoid
After having read Dan Langille’s blog post on ZFS tools several months ago, I had been wanting to give it a try, since it looked simple to setup and more akin to my current needs.
I based my configuration mostly off Dan’s, so checked that great post :-).
Migrating zfs-periodic(8) snapshots to sanoid
This ends up being somewhat simple with periodic2sanoid.sh:
# Check locally and remotely that actions make sense (and results match)
# output lists "DS" as the base dataset, it acts recursively
./periodic2sanoid.sh zroot/usr/home
[...]
DS@daily-2023-06-28 --> DS@autosnap_2023-06-28_12:24:44_daily
[...]
# Actually apply changes
env DRY_RUN=NO ./periodic2sanoid.sh zroot/usr/home
And here is the code for that script, you can (and should!) audit it:
#!/bin/sh -eu
# Copyright (c) 2023, Evilham (https://evilham.com)
# BSD 2-Clause copyright and disclaimer apply.
# See: http://www.opensource.org/licenses/bsd-license.php
DS="$1"
DRY_RUN="${DRY_RUN:-YES}"
epoch_to_time() {
epoch="$1"
date -j -f '%s' "${epoch}" '+%Y-%m-%d_%H:%M:%S'
}
target_name() {
snap_epoch="$1"
snap_name="$2"
snap_time="$(epoch_to_time "${snap_epoch}")"
case "${snap_name}" in
quarter_hourly-*)
# quarter_hourly --> frequently
printf "autosnap_%s_%s" "${snap_time}" "frequently"
;;
hourly-*)
printf "autosnap_%s_%s" "${snap_time}" "hourly"
;;
daily-*)
printf "autosnap_%s_%s" "${snap_time}" "daily"
;;
weekly-*)
printf "autosnap_%s_%s" "${snap_time}" "weekly"
;;
monthly-*)
printf "autosnap_%s_%s" "${snap_time}" "monthly"
;;
yearly-*)
printf "autosnap_%s_%s" "${snap_time}" "yearly"
;;
*)
# Empty --> no renaming
;;
esac
}
zfs list -rpHt snap -s creation -o creation,name "$DS" | while IFS='' read -r line; do
snap_epoch="$(echo "${line}" | cut -f 1)"
full_snap="$(echo "${line}" | cut -f 2)"
ds_name="$(echo "${full_snap}" | cut -d '@' -f 1)"
snap_name="$(echo "${full_snap}" | cut -d '@' -f 2-)"
snap_target_name="$(target_name "${snap_epoch}" "${snap_name}")"
if [ -n "${snap_target_name}" ]; then
# We use DS to facilitate comparing local and remote actions
printf "%s@%s\t-->\t%s@%s\n" "DS" "${snap_name}" "DS" "${snap_target_name}"
if [ "${DRY_RUN}" != "YES" ]; then
echo "zfs rename \"${full_snap}\" \"${ds_name}@${snap_target_name}\""
zfs rename "${full_snap}" "${ds_name}@${snap_target_name}"
fi
fi
done
snapshot creation and pruning with sanoid
This, as Dan documented it, works wonderfully after taking
into account the crontab(5) caveat and using lockf(1).
ZFS Replication with syncoid
When installing sanoid, we also get syncoid as a replication tool.
Issues with permissions
While trying to replicate the datasets, I realised that it expects either
to run as root or to have sudo available.
Of course, that’s not ideal, as I prefer to rely on zfs-allow(8) and do not
even have sudo as an available command.
There is a flag --no-privilege-elevation which helps circumvent these checks,
but the fact that it fails and requires the flag to work as non-root, is kind
of a red flag for me.
Issues with public key limitations
As with zxfer(8), I was setting up a pull-based approach over SSH.
Since this user and that SSH key should only be allowed to use zfs send
and little more, I use the authorized_keys file to force the command
to a particular script.
This resulted in issues with syncoid‘s shell escaping, which lead me to
this reported issue from 2022.
I believe the person that reported it was in a similar position to mine.
I ended up “fixing it” (the security implications are still to be determined),
by expanding the list of allowed commands and using sh -s with
$SSH_ORIGINAL_COMMAND:
#!/bin/sh
case "${SSH_ORIGINAL_COMMAND}" in
"uname")
;;
"zfs get "*)
;;
"/sbin/zfs get "*)
;;
"zfs list "*)
;;
"/sbin/zfs list "*)
;;
"zfs send -n "*)
;;
"zfs send -w "*)
;;
" zfs send -w "*)
;;
"/sbin/zfs send -w "*)
;;
"echo -n")
;;
"command -v lzop")
;;
"command -v mbuffer")
;;
"zpool get -o value -H feature@extensible_dataset "*)
;;
"exit")
;;
*)
echo "Command not allowed! See ya!"
exit 1
;;
esac
echo "${SSH_ORIGINAL_COMMAND}" >> /tmp/test
sh -s <<EOF
${SSH_ORIGINAL_COMMAND}
EOF
Encrypted datasets
This is also a bit of an issue, and we need to specify --sendoptions="w".
All together
syncoid -r --no-privilege-elevation --sendoptions="w" \
sucker@HOST:zroot/usr/home backups/pools/HOST/zroot/usr/home
This kind of works, but when looking to refine it, I realised that it does not
offer me any benefit regarding zxfer(8) when it comes to protecting the
snapshots.
Remembering zrepl
After getting frustrated again, I realised this wasn’t quite looking as I was expecting (I was so sure this was doable!).
Turns out, I had read about zrepl some months ago, and checked the documentation and determined it was something I needed to use.
So, this misremembering caused me to look deeply into sanoid and syncoid.
But hey, at least now I am very sure about what I want, and I can probably quickly adapt my renaming script for a migration to zrepl.
Conclusion
I really should blog more often, if anything to save me some time.
This will have to be a two-parter, with this one being what I’ve tried and why I’m not keeping it, and the second one being how I get to setup zrepl to my liking.
UPDATE: Bonus tools from the fediverse
I posted this on the fediverse, and some people shared what they use and why!
zfs_autobackup
@stefano@mdon.stefanomarinelli.it mentioned zfs_autobackup, and they have a very nice blog post series documenting this, and other steps they took on migrating from Linux to FreeBSD.
zrep
@dk3jf@radiosocial.de mentioned zrep: