Evilham

Evilham.com

Eines de replicació amb ZFS

Introduction

Back when I first started using FreeBSD and ZFS, I needed tools with an extremely low barrier of entry, that did their job well.

When it comes to creating and pruning snapshots, the job was first done by sysutils/zfs-periodic.

Over time however, even with minor improvements on my part, it has come short for my current needs.

As inspired by Dan Langille’s wonderful blog (Dan always writes great blog posts, that end up being lovely complementary documentation), I document here my somewhat bumpy road when it comes to ZFS replication tools and why you might want to use something different at each step.

Table of contents

Quick reminder of ZFS

ZFS is nowadays (at least on FreeBSD and Linux) actually OpenZFS. Let’s see how they define it:

OpenZFS is an advanced file system and volume manager which was originally developed for Solaris and is now maintained by the OpenZFS community. This repository contains the code for running OpenZFS on Linux and FreeBSD.

Amongst the many beautiful things of ZFS, there is the fact that it works in a Copy on Write fashion, which, amongst other things, enables it to create snapshots in an instantaneous fashion.

That is, we can assign a name to a known state with, e.g. zfs snapshot -r zroot@goodstate.

Keep on working and, if we get on to a bad state, discard any changes made since then by issuing a rollback.

This means, we can have a very handy way of having real-life save points, and work in a more relaxed fashion.

Together with zfs-send(8) and zfs-receive(8), we quickly have the basis for a remote backup system that is incremental.

Now that ZFS data set encryption (zfs-load-key(8)) is a thing, it too can be encrypted.

Snapshot creation and prunning with zfs-periodic(8)

Now, back in ~2019, I was mostly a Linux user that was jumping to FreeBSD and the last thing I wanted was to fiddle too much with a new file system.

Luckily, there is a very simple way to manage ZFS snapshots in an automatic fashion right in the ports system: zfs-periodic.

It profits from the periodic(8) system, to run at regular intervals, and gets configured in a minute with:

$ pkg install -y zfs-periodic
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.

[1/1] Fetching zfs-periodic-1.0.20130213.pkg: 100%    2 KiB   2.3kB/s    00:01
Checking integrity... done (0 conflicting)
[1/1] Installing zfs-periodic-1.0.20130213...
[1/1] Extracting zfs-periodic-1.0.20130213: 100%
=====
Message from zfs-periodic-1.0.20130213:

--
In order to enable periodic snapshots you need
to add these lines to your /etc/periodic.conf

hourly_output="root"
hourly_show_success="NO"
hourly_show_info="YES"
hourly_show_badconfig="NO"
hourly_zfs_snapshot_enable="YES"
hourly_zfs_snapshot_pools="tank"
hourly_zfs_snapshot_keep=6
daily_zfs_snapshot_enable="YES"
daily_zfs_snapshot_pools="tank"
daily_zfs_snapshot_keep=7
weekly_zfs_snapshot_enable="YES"
weekly_zfs_snapshot_pools="tank"
weekly_zfs_snapshot_keep=5
monthly_zfs_snapshot_enable="YES"
monthly_zfs_snapshot_pools="tank"
monthly_zfs_snapshot_keep=2

To get hourly snapshots you also need to add
something like this to /etc/crontab:

2       *       *       *       *       root    periodic hourly

We literally do that (adding the crontab(5) entry and configuring zfs-periodic(8) in /usr/local/etc/periodic.conf (*)), and we have a simple system that takes hourly, daily, weekly and monthly snapshots.

Half way there with zero effort, this is what has kept me using it for years.

(*): I prefer to use /usr/local/etc/periodic.conf for things outside the base system. This is a matter of taste and it helps easen administration.

Adding quarter_hourly

Given how cheap these snapshots are, I (and people at work) started wanting to have them more often, which in my case meant every 15 minutes.

Luckily, periodic(8) supports arbitrary directories:

directory  An arbitrary directory containing a set of executables to be
           run.

However, it wasn’t as simple as:

mkdir /usr/local/etc/periodic/quarter_hourly/
ln -s /usr/local/etc/periodic/daily/000.zfs-snapshot \
      /usr/local/periodic/quarter_hourly/000.zfs-snapshot

Because the actual file in there depends on the name of the directory, so it has to be customised. And also because the /usr/local/bin/zfs-snapshot script that gets installed needs to understand all the periodic.conf settings.

Luckily making these changes was fairly simple. Since I won’t be using zfs-periodic(8) any longer, and the patch has been more than tested over the years, I’m opening a Merge Request, which you can use to benefit from these changes.

Take into account that it’ll also need a corresponding crontab(5) entry:

*/15       *       *       *       *       root    periodic quarter_hourly

ZFS Replication with zxfer(8)

Snapshots take care of local mess ups, but they don’t serve as a backup.

For that, I took to zxfer, which is simple to use and has worked great with some caveats.

A lovely thing of zxfer(8) is that it supports an rsync(1) mode, that allows us to back up remote systems that do not support ZFS, to a ZFS-based system that takes care of snapshots and so on.

The caveats

Mostly, even with a pull-based approach, zxfer(8) doesn’t protect much against snapshots disappearing on the source, which means that there is a somewhat plausible scenario where pulling data results in breaking certain retention policies.

While there is a -g flag to protect old snapshots, it means that we have to replicate snapshot pruning on the destination system, but cannot create snapshots as they could collide with the source snapshots.

That means zfs-periodic(8) and zxfer(8) are better than nothing and work decently, but have important limitations.

Encrypted datasets

When encrypted datasets got added to OpenZFS, I realised that we need to use zfs send -w, or else the backup will happen unencrypted. So I wrote a patch for zxfer to do just that.

You can help land that patch in zxfer(8), it is basically only missing the manpage and USAGE texts, but my manpage syntax is rusty and time is not an abundantly available commodity.

I’d love to help you land this change though!

Snapshot creation and pruning with sanoid

After having read Dan Langille’s blog post on ZFS tools several months ago, I had been wanting to give it a try, since it looked simple to setup and more akin to my current needs.

I based my configuration mostly off Dan’s, so checked that great post :-).

Migrating zfs-periodic(8) snapshots to sanoid

This ends up being somewhat simple with periodic2sanoid.sh:

# Check locally and remotely that actions make sense (and results match)
#  output lists "DS" as the base dataset, it acts recursively
./periodic2sanoid.sh zroot/usr/home
[...]
DS@daily-2023-06-28 --> DS@autosnap_2023-06-28_12:24:44_daily
[...]
# Actually apply changes
env DRY_RUN=NO ./periodic2sanoid.sh zroot/usr/home

And here is the code for that script, you can (and should!) audit it:

#!/bin/sh -eu

# Copyright (c) 2023, Evilham (https://evilham.com)
# BSD 2-Clause copyright and disclaimer apply.
# See: http://www.opensource.org/licenses/bsd-license.php

DS="$1"

DRY_RUN="${DRY_RUN:-YES}"

epoch_to_time() {
    epoch="$1"
    date -j -f '%s' "${epoch}" '+%Y-%m-%d_%H:%M:%S'
}

target_name() {
    snap_epoch="$1"
    snap_name="$2"
    snap_time="$(epoch_to_time "${snap_epoch}")"
    case "${snap_name}" in
        quarter_hourly-*)
            # quarter_hourly --> frequently
            printf "autosnap_%s_%s" "${snap_time}" "frequently"
        ;;
        hourly-*)
            printf "autosnap_%s_%s" "${snap_time}" "hourly"
        ;;
        daily-*)
            printf "autosnap_%s_%s" "${snap_time}" "daily"
        ;;
        weekly-*)
            printf "autosnap_%s_%s" "${snap_time}" "weekly"
        ;;
        monthly-*)
            printf "autosnap_%s_%s" "${snap_time}" "monthly"
        ;;
        yearly-*)
            printf "autosnap_%s_%s" "${snap_time}" "yearly"
        ;;
        *)
            # Empty --> no renaming
        ;;
    esac
}


zfs list -rpHt snap -s creation -o creation,name "$DS" | while IFS='' read -r line; do
    snap_epoch="$(echo "${line}" | cut -f 1)"
    full_snap="$(echo "${line}" | cut -f 2)"
    ds_name="$(echo "${full_snap}" | cut -d '@' -f 1)"
    snap_name="$(echo "${full_snap}" | cut -d '@' -f 2-)"

    snap_target_name="$(target_name "${snap_epoch}" "${snap_name}")"

    if [ -n "${snap_target_name}" ]; then
        # We use DS to facilitate comparing local and remote actions
        printf "%s@%s\t-->\t%s@%s\n" "DS" "${snap_name}" "DS" "${snap_target_name}"
        if [ "${DRY_RUN}" != "YES" ]; then
            echo "zfs rename \"${full_snap}\" \"${ds_name}@${snap_target_name}\""
            zfs rename "${full_snap}" "${ds_name}@${snap_target_name}"
        fi
    fi
done

snapshot creation and pruning with sanoid

This, as Dan documented it, works wonderfully after taking into account the crontab(5) caveat and using lockf(1).

ZFS Replication with syncoid

When installing sanoid, we also get syncoid as a replication tool.

Issues with permissions

While trying to replicate the datasets, I realised that it expects either to run as root or to have sudo available.

Of course, that’s not ideal, as I prefer to rely on zfs-allow(8) and do not even have sudo as an available command.

There is a flag --no-privilege-elevation which helps circumvent these checks, but the fact that it fails and requires the flag to work as non-root, is kind of a red flag for me.

Issues with public key limitations

As with zxfer(8), I was setting up a pull-based approach over SSH. Since this user and that SSH key should only be allowed to use zfs send and little more, I use the authorized_keys file to force the command to a particular script.

This resulted in issues with syncoid‘s shell escaping, which lead me to this reported issue from 2022. I believe the person that reported it was in a similar position to mine.

I ended up “fixing it” (the security implications are still to be determined), by expanding the list of allowed commands and using sh -s with $SSH_ORIGINAL_COMMAND:

#!/bin/sh
case "${SSH_ORIGINAL_COMMAND}" in
    "uname")
    ;;
    "zfs get "*)
    ;;
    "/sbin/zfs get "*)
    ;;
    "zfs list "*)
    ;;
    "/sbin/zfs list "*)
    ;;
    "zfs send -n "*)
    ;;
    "zfs send -w "*)
    ;;
    " zfs send -w "*)
    ;;
    "/sbin/zfs send -w "*)
    ;;
    "echo -n")
    ;;
    "command -v lzop")
    ;;
    "command -v mbuffer")
    ;;
    "zpool get -o value -H feature@extensible_dataset "*)
    ;;
    "exit")
    ;;
    *)
        echo "Command not allowed! See ya!"
        exit 1
    ;;
esac
echo "${SSH_ORIGINAL_COMMAND}" >> /tmp/test
sh -s <<EOF
${SSH_ORIGINAL_COMMAND}
EOF

Encrypted datasets

This is also a bit of an issue, and we need to specify --sendoptions="w".

All together

syncoid -r --no-privilege-elevation --sendoptions="w" \
  sucker@HOST:zroot/usr/home backups/pools/HOST/zroot/usr/home

This kind of works, but when looking to refine it, I realised that it does not offer me any benefit regarding zxfer(8) when it comes to protecting the snapshots.

Remembering zrepl

After getting frustrated again, I realised this wasn’t quite looking as I was expecting (I was so sure this was doable!).

Turns out, I had read about zrepl some months ago, and checked the documentation and determined it was something I needed to use.

So, this misremembering caused me to look deeply into sanoid and syncoid.

But hey, at least now I am very sure about what I want, and I can probably quickly adapt my renaming script for a migration to zrepl.

Conclusion

I really should blog more often, if anything to save me some time.

This will have to be a two-parter, with this one being what I’ve tried and why I’m not keeping it, and the second one being how I get to setup zrepl to my liking.

UPDATE: Bonus tools from the fediverse

I posted this on the fediverse, and some people shared what they use and why!

zfs_autobackup

@stefano@mdon.stefanomarinelli.it mentioned zfs_autobackup, and they have a very nice blog post series documenting this, and other steps they took on migrating from Linux to FreeBSD.

zrep

@dk3jf@radiosocial.de mentioned zrep: