Velero

I tried Velero and it did not work reliably all the time. Sometimes the kubernetes cluster crashes during recovery or data is not fully recovered.
Feel free to test it out and update this documentation once you feel that it's working reliably. It is very likely that Digital Ocean had some bugs when I tried out the steps below.
We use velero for on premise backups, we tested on version v0.11.0, you can find their documentation here.
Our kubernets configurations adds some annotations to pods. The annotations define the important persistent volumes that need to be backed up. Velero will pick them up and store the volumes in the same cluster but in another namespace velero.
Prequisites
You have to install the binary velero on your computer and get a tarball of the latest release. We use v0.11.0 so visit the release page and download and extract e.g. velero-v0.11.0-linux-arm64.tar.gz.
Setup Velero Namespace
Follow their getting started instructions to setup the Velero namespace. We use Minio and restic, so check out Velero's instructions how to setup restic:
# run from the extracted folder of the tarball
$ kubectl apply -f config/common/00-prereqs.yaml
$ kubectl apply -f config/minio/
Once completed, you should see the namespace in your kubernetes dashboard.
Manually Create an On-Premise Backup
When you create your deployments for Human Connection the required annotations should already be in place. So when you create a backup of namespace human-connection:
$ velero backup create hc-backup --include-namespaces=human-connection
That should backup your persistent volumes, too. When you enter:
$ velero backup describe hc-backup --details
You should see the persistent volumes at the end of the log:
....
​
Restic Backups:
  Completed:
    human-connection/nitro-backend-5b6dd96d6b-q77n6: uploads
    human-connection/nitro-neo4j-686d768598-z2vhh: neo4j-data
Simulate a Disaster
Feel free to try out if you loose any data when you simulate a disaster and try to restore the namespace from the backup:
$ kubectl delete namespace human-connection
Wait until the wrongdoing has completed, then:
$ velero restore create --from-backup hc-backup
Now, I keep my fingers crossed that everything comes back again. If not, I feel very sorry for you.
Schedule a Regular Backup
Check out the docs. You can create a regular schedule e.g. with:
$ velero schedule create hc-weekly-backup --schedule="@weekly" --include-namespaces=human-connection
Inspect the created backups:
$ velero schedule get
NAME               STATUS    CREATED                          SCHEDULE   BACKUP TTL   LAST BACKUP   SELECTOR
hc-weekly-backup   Enabled   2019-05-08 17:51:31 +0200 CEST   @weekly    720h0m0s     6s ago        <none> 
​
$ velero backup get
NAME                              STATUS      CREATED                          EXPIRES   STORAGE LOCATION   SELECTOR
hc-weekly-backup-20190508155132   Completed   2019-05-08 17:51:32 +0200 CEST   29d       default            <none>
​
$ velero backup describe hc-weekly-backup-20190508155132 --details
# see if the persistent volumes are backed up
Reclaim Policy
Metrics
Last modified 4yr ago