A quick post, while I work on establishing some research and publishing momentum.

TL;DR Docker + disk-space issues meant I had to be a little bit creative, or at least careful.

This blog is hosted on a small DigitalOcean droplet, which I also use for a personal Nextcloud instance.¹

There’s some redundancy already built into this; my files are replicated across a couple of devices aside from the server, but it’s not ideal either. A theoretical attack that deleted my files would get propagated across those same devices, I suppose.

So, backups. DigitalOcean offer “Spaces”, their implementation of object storage with an S3-compatible API, which seems ideal. My threat model at this stage does not include their data centre being razed.

The S3 API means I can use something like s3cmd to upload snapshots. In theory, I could just tar up the data directory and upload it via s3cmd.

In practice, there are a couple of issues that motivated this post in case anyone else has similar constraints:

• Nextcloud is running via Docker, meaning the data directory is slightly abstracted
• I don’t have a large server, so Nextcloud occupies > 50% of disk space and there’s no room for that tar file! I’ll have to figure out streaming as well.

Solution

Preparation

Minor note, but we want to put Nextcloud into maintenance-mode for the duration. The following command will do that:

php occ maintenance:mode --on

I also install a trap-EXIT in the shell script to disable maintenance mode at the end.

Docker

Well… docker-compose, which is another minor layer of abstraction to thread through. The command above would actually be invoked by

docker-compose exec -T --user www-data app php occ maintenance:mode --on

(“app” is the name of the docker-compose container running nextcloud; there’s also one for the database, and the server)

Accessing the data was something I tried a few things around, and it’s possibly not the best solution. I feel like you should be able to back up the volume in a portable way, but for now I rely on invoking tar directly from within the container, streaming to stdout.

That final point — “stdout” — is also a bit tricky; the secret is the -T argument to exec, so we wind up with:

docker-compose exec -T --user www-data app tar czf - data

(and something similar for the database). The -T disables the pseudo-TTY that otherwise gets allocated.

Uploading, and gotchas

Now we have a stream of data to upload. From memory the documentation wasn’t great, but s3cmd can operate in this way, as long as you specify the object name, so you end up with something like (note the “-” in both commands):

tar cvzf - ... |
s3cmd put - s3://mystoragebucket/backup_file

And we’re almost done. But we don’t want to back up everything for ever more! So you need to configure a lifecyle for your backups, so they will expire (and also, so I don’t tip over the included storage quota).

As it turns out, I did tip over the edge (only by $4), because there’s a trick I wasn’t aware of. Your lifecycle can configure a prefix, which determines the files your expiration policy applies to. I have everything under /backups, so I tried:

<Prefix>/backups</Prefix>

This did not work! Sparing the colourful language that ensued, it turns out that you don’t want to include the first /. I needed

<Prefix>backups</Prefix>

(In fact, that bucket is only used for backups so now I just use <Prefix />, but note that <Prefix>/</Prefix> has the same problem). I’m assuming the API refers to object names, and I’ve been thinking in terms of directory structures.

Putting it all together

To wrap up, this is the shell script (as an Ansible template) that I install, and then gets invoked by cron:

#!/bin/bash

set -eufo pipefail

DB_BACKUP=nextcloud-db-$(date +%Y%m%d-%s).dmp
DATA_BACKUP=nextcloud-data-$(date +%Y%m%d-%s).tgz

# maintenance mode off again, even if we error out:
finish() {
    docker-compose exec -T --user www-data app php occ maintenance:mode --off
}

trap finish EXIT

cd /srv/www/nextcloud

# maintenance mode on
docker-compose exec -T --user www-data app php occ maintenance:mode --on

# docker-compose to tar to output

docker-compose exec -T --user www-data app tar czf - data |
s3cmd --access_key={{ storage_apikey }} \
--secret_key={{ storage_secret_key }} \
--region={{ storage_region }} \
--host=digitaloceanspaces.com \
--host-bucket="%(bucket)s.{{ storage_region }}.digitaloceanspaces.com" \
put - s3://{{ storage_bucket_name }}/backups/$DATA_BACKUP

# db backup
docker-compose exec -T --user postgres db pg_dump -Fc nextcloud |
s3cmd --access_key={{ storage_apikey }} \
--secret_key={{ storage_secret_key }} \
--region={{ storage_region }} \
--host=digitaloceanspaces.com \
--host-bucket="%(bucket)s.{{ storage_region }}.digitaloceanspaces.com" \
put - s3://{{ storage_bucket_name }}/backups/$DB_BACKUP