Velero
I tried Velero and it did not work reliably all the time. Sometimes the kubernetes cluster crashes during recovery or data is not fully recovered.
Feel free to test it out and update this documentation once you feel that it's working reliably. It is very likely that Digital Ocean had some bugs when I tried out the steps below.
Our kubernets configurations adds some annotations to pods. The annotations define the important persistent volumes that need to be backed up. Velero will pick them up and store the volumes in the same cluster but in another namespace
velero
.You have to install the binary
velero
on your computer and get a tarball of the latest release. We use v0.11.0
so visit the release page and download and extract e.g. velero-v0.11.0-linux-arm64.tar.gz.Follow their getting started instructions to setup the Velero namespace. We use Minio and restic, so check out Velero's instructions how to setup restic:
# run from the extracted folder of the tarball
$ kubectl apply -f config/common/00-prereqs.yaml
$ kubectl apply -f config/minio/
Once completed, you should see the namespace in your kubernetes dashboard.
When you create your deployments for Human Connection the required annotations should already be in place. So when you create a backup of namespace
human-connection
:$ velero backup create hc-backup --include-namespaces=human-connection
That should backup your persistent volumes, too. When you enter:
$ velero backup describe hc-backup --details
You should see the persistent volumes at the end of the log:
....
Restic Backups:
Completed:
human-connection/nitro-backend-5b6dd96d6b-q77n6: uploads
human-connection/nitro-neo4j-686d768598-z2vhh: neo4j-data
Feel free to try out if you loose any data when you simulate a disaster and try to restore the namespace from the backup:
$ kubectl delete namespace human-connection
Wait until the wrongdoing has completed, then:
$ velero restore create --from-backup hc-backup
Now, I keep my fingers crossed that everything comes back again. If not, I feel very sorry for you.
$ velero schedule create hc-weekly-backup --schedule="@weekly" --include-namespaces=human-connection
Inspect the created backups:
$ velero schedule get
NAME STATUS CREATED SCHEDULE BACKUP TTL LAST BACKUP SELECTOR
hc-weekly-backup Enabled 2019-05-08 17:51:31 +0200 CEST @weekly 720h0m0s 6s ago <none>
$ velero backup get
NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR
hc-weekly-backup-20190508155132 Completed 2019-05-08 17:51:32 +0200 CEST 29d default <none>
$ velero backup describe hc-weekly-backup-20190508155132 --details
# see if the persistent volumes are backed up
Last modified 4yr ago