This page is a draft.

I think backups are important, and underrated; conceptually simple, practically awkward and generally over-complicated. Two influential things I read that informed me were The Tao of Backup and jwz's page on Backups. Here's my take.

  1. Nutshell
  2. Details
    1. Disk encryption
    2. Mount namespace
    3. Software

Nutshell

I use a NAS at my house as both the long-term, proper home for most of my data (on one drive), and as the place for backups to go (on another drive). I backup everything else to the NAS; the NAS's data drive to the backup drive; and the backup drive to a pair of external drives, which spend the rest of their time off-site.

Details

Disk encryption

LUKS; keyfiles. used

Mount namespace

If the backup data was always available to the rest of the system, there is a risk of accidental or malicious damage to the backup data -- a stray rm typed in error, for example. To mitigate this, my backup volume is only access from a different mount namespace from the rest of the system.

The mount namespace is created using this mkbackupns script. It must be run as root, and is designed to be safe to run repeatedly.

mkbackupns

#!/bin/bash
set -euo pipefail

nsdir=/var/namespaces
nsfile=$nsdir/backup
nsfilex="$(echo $nsfile | sed 's#/#\\/#'g)"

private_propagation() {
    findmnt -o+PROPAGATION "$nsdir" | grep -q private
}
nsfs_is_mounted() {
    test "nsfs" = "$(awk "/$nsfilex/ { print \$3 }" /proc/mounts)"
}

if ! nsfs_is_mounted; then

    if ! private_propagation; then
        mkdir -p "$nsdir"
        mount --bind --make-private "$nsdir" "$nsdir"
    fi

    touch "$nsfile"
    unshare --mount="$nsfile" true

    nsenter --mount=/var/namespaces/backup mount /dev/phobos_backup/backup /backup

I run a second SSHd within this namespace. This allows me to access it conveniently from any host (including localhost) at home. It requires a separate Systemd service file and a separate sshd_config (omitted here since it's not too interesting)

ssh-borg.service

[Unit]
Description=SSHd for Borg
After=network.target nss-user-lookup.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
ExecStartPre=/home/jon/bin/mkbackupns
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/bin/nsenter \
        --mount=/var/namespaces/backup \
        /usr/sbin/sshd -D -f /etc/ssh/sshd_borg_config
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify

[Install]
WantedBy=multi-user.target

Software

Borg v1.4 as the primary backup software. Rsync in a couple of places where running Borg is awkward. Job scheduling, inter-dependencies and triggering via Systemd. Disk encryption using LUKS. Some additional resilience via Linux mount namespaces.

Borgmatic (scheduling)

backup-exthdd.service

[Unit]
OnFailure=status-email-user@%n.service blinkstick-fail.service

Requires=systemd-cryptsetup@extbackup.service
After=systemd-cryptsetup@extbackup.service

[Service]
Type=exec
User=root
Group=root
ExecStartPre=/home/jon/bin/mkbackupns
ExecStart=/usr/bin/nsenter \
        --setuid=1000 \
        --setgid=1000 \
        --mount=/var/namespaces/backup \
        /home/jon/bin/phobos-backup-monthly 1, ncl

# extbackup drive 1
[Install]
WantedBy=dev-disk-by\x2duuid-1764d71d\x2d6379\x2d4001\x2d95d6\x2db61173f82d6d.device