Velero

I tried Velero and it did not work reliably all the time. Sometimes the kubernetes cluster crashes during recovery or data is not fully recovered.

Feel free to test it out and update this documentation once you feel that it's working reliably. It is very likely that Digital Ocean had some bugs when I tried out the steps below.

We use velero for on premise backups, we tested on version v0.11.0, you can find their documentation here.

Our kubernets configurations adds some annotations to pods. The annotations define the important persistent volumes that need to be backed up. Velero will pick them up and store the volumes in the same cluster but in another namespace velero.

Prequisites

You have to install the binary velero on your computer and get a tarball of the latest release. We use v0.11.0 so visit the release page and download and extract e.g. velero-v0.11.0-linux-arm64.tar.gz.

Setup Velero Namespace

Follow their getting started instructions to setup the Velero namespace. We use Minio and restic, so check out Velero's instructions how to setup restic:

# run from the extracted folder of the tarball
$ kubectl apply -f config/common/00-prereqs.yaml
$ kubectl apply -f config/minio/

Once completed, you should see the namespace in your kubernetes dashboard.

Manually Create an On-Premise Backup

When you create your deployments for Human Connection the required annotations should already be in place. So when you create a backup of namespace human-connection:

$ velero backup create hc-backup --include-namespaces=human-connection

That should backup your persistent volumes, too. When you enter:

$ velero backup describe hc-backup --details

You should see the persistent volumes at the end of the log:

....

Restic Backups:
  Completed:
    human-connection/nitro-backend-5b6dd96d6b-q77n6: uploads
    human-connection/nitro-neo4j-686d768598-z2vhh: neo4j-data

Simulate a Disaster

Feel free to try out if you loose any data when you simulate a disaster and try to restore the namespace from the backup:

$ kubectl delete namespace human-connection

Wait until the wrongdoing has completed, then:

$ velero restore create --from-backup hc-backup

Now, I keep my fingers crossed that everything comes back again. If not, I feel very sorry for you.

Schedule a Regular Backup

Check out the docs. You can create a regular schedule e.g. with:

$ velero schedule create hc-weekly-backup --schedule="@weekly" --include-namespaces=human-connection

Inspect the created backups:

$ velero schedule get
NAME               STATUS    CREATED                          SCHEDULE   BACKUP TTL   LAST BACKUP   SELECTOR
hc-weekly-backup   Enabled   2019-05-08 17:51:31 +0200 CEST   @weekly    720h0m0s     6s ago        <none> 

$ velero backup get
NAME                              STATUS      CREATED                          EXPIRES   STORAGE LOCATION   SELECTOR
hc-weekly-backup-20190508155132   Completed   2019-05-08 17:51:32 +0200 CEST   29d       default            <none>

$ velero backup describe hc-weekly-backup-20190508155132 --details
# see if the persistent volumes are backed up