Add steps for restoring backups

README.md

@@ -1,6 +1,22 @@
 # Raspberry Pi IaC
 Contains the Infrastructure as Code files which are used for provisioning my in house Raspberry Pi.
 
+- [Raspberry Pi IaC](#raspberry-pi-iac)
+  - [Prerequisites](#prerequisites)
+    - [Ansible control node prerequisites](#ansible-control-node-prerequisites)
+    - [Environment prerequisites](#environment-prerequisites)
+    - [Raspberry Pi preparation](#raspberry-pi-preparation)
+  - [Provisioning](#provisioning)
+  - [Overview of installed software](#overview-of-installed-software)
+    - [SSH with ZSH](#ssh-with-zsh)
+  - [Other](#other)
+    - [Reinstalling the Pi](#reinstalling-the-pi)
+    - [The external Kingston drive](#the-external-kingston-drive)
+    - [Debugging users other than the main user](#debugging-users-other-than-the-main-user)
+    - [Manually restoring a backup](#manually-restoring-a-backup)
+      - [Restoring the database content](#restoring-the-database-content)
+      - [Restoring docker backups](#restoring-docker-backups)
+
 ## Prerequisites
 The files within this repository should be run on a computer with Ansible installed which is only supported on MacOS and Linux systems. For Windows it can be run within a WSL distro (see [this guide](https://code.visualstudio.com/docs/remote/wsl-tutorial) on how to set this up, make sure the repository is checked out on the WSL distribution itself).
 
@@ -29,17 +45,17 @@ The Raspberry Pi IaC code contained within this repository provisions the Raspbe
 1. [A prepared Raspberry Pi]((#raspberry-pi-preperation)).
 3. [Cloudflare](https://dash.cloudflare.com/login) should be setup for managing the domain records of *kleinendorst.info*.
 
-### Raspberry Pi preperation
+### Raspberry Pi preparation
 The Raspberry Pi should be installed and running with reachable SSH from the network.
 
 1. Setup its MicroSD card using the Raspberry Pi Imager ([download](https://www.raspberrypi.com/software/)). For **CHOOSE OS** select the : **Raspberry Pi OS (other)** > **Raspberry Pi OS Lite (64-bit)** option.
-2. When asked: **Would you like to apply OS customisation settings?** select **EDIT SETTINGS**. Select and fill in the following settings:
+2. When asked: **Would you like to apply OS customization settings?** select **EDIT SETTINGS**. Select and fill in the following settings:
     1. **Set username and password**
     2. **Set locale settings**
     3. **Enable SSH** > **Allow public-key authentication only** and enter your computer's public key.
     4. Disable **Eject media when finished** (probably not really important but I heard it could prevent problems on Windows).
 3. Start the Raspberry Pi with an ethernet cable attached.
-4. Find the assigned IP of the Raspberry Pi in the [router](http://asusrouter.com/) and configure DHCP to statically asign this address to the Raspberry Pi.
+4. Find the assigned IP of the Raspberry Pi in the [router](http://asusrouter.com/) and configure DHCP to statically assign this address to the Raspberry Pi.
 5. Add the new Raspberry Pi to the *hosts* file using the internal IP if it isn't there already.
 6. Test if the Raspberry Pi is correctly configured by opening an SSH session to it (using its IP address).
 
@@ -52,14 +68,14 @@ ansible-playbook playbook.yml
 
 ## Overview of installed software
 ### SSH with ZSH
-It's possible to connect to the Raspberry Pi from the internal network via either its FQDN or IP address **using a public key only** setup as part of [the Raspberry Pi preperation](#raspberry-pi-preperation).
+It's possible to connect to the Raspberry Pi from the internal network via either its FQDN or IP address **using a public key only** setup as part of [the Raspberry Pi preparation](#raspberry-pi-preparation).
 When logged in the user will be prompted with the **zsh** configured with **[Oh My Zsh](https://ohmyz.sh)** and **[Starhip](https://starship.rs) prompts**.
 
 ![zsh](./images/zsh.png)
 
 ## Other
 ### Reinstalling the Pi
-It can be handy to reinstall the Pi. First shutdown the pi by running `sudo shutdown` from SSH. Next take out the memory card and follow all steps in [Raspberry Pi preperation](#raspberry-pi-preperation).
+It can be handy to reinstall the Pi. First shutdown the pi by running `sudo shutdown` from SSH. Next take out the memory card and follow all steps in [Raspberry Pi preparation](#raspberry-pi-preparation).
 For the next step remove the current *known_hosts* entry with: `ssh-keygen -R '192.168.50.27'` for all PCs that had SSH access to the Pi.
 
 ### The external Kingston drive
@@ -80,3 +96,103 @@ sudo login
 This is verified to be working:
 
 ![new users](./images/login_success.png)
+
+### Manually restoring a backup
+The `backups` role will install scripting which automatically backs up specific files and uploads it to an external service called
+[BorgBase](https://www.borgbase.com). This section describes manually restoring such a backup after cleanly reinstalling the Pi from scratch.
+
+First what's needed is a reference to the Borg repo URL and passphrase. They can be found in the vault file located at: `roles/backups/vars/main/vault.yml`.
+Read out the values using the `ansible-vault` binary. Log into the Raspberry Pi, change to root and fill the `BORG_REPO` and `BORG_PASSPHRASE` variables:
+
+```bash
+sudo -i
+export BORG_REPO='...'
+export BORG_PASSPHRASE='...'
+```
+
+The `backups` role generates a new SSH key pair when run. Since we're assuming in this section a new install took place we'll have to upload the newly
+generated public key into BorgBase. The correct key is stored in the `root` user's home folder, view it with `cat ~/.ssh/id_ssh_rsa.pub` and upload it
+to [BorgBase](https://www.borgbase.com). Make sure to also remove any old keys and add it to the repository with "full access".
+
+It should now be possible to view backups contained within the borg repo by running: `borg list`. Now run the following commands (replacing `[ARCHIVE_NAME]`):
+
+```bash
+mkdir /bulk/backup_work_dir
+cd /bulk/backup_work_dir/
+
+borg extract '::[ARCHIVE_NAME]' ./docker_volumes ./postgres
+```
+
+The files are now unpacked in this directory, make sure to clean up this directory after use.
+
+#### Restoring the database content
+In order to restore the database backups: first log into the PostgreSQL database's REPL shell with the credentials
+of the `postgres` user found in the role's vault (if required). Now run the following commands (substituting `[DBNAME]`):
+
+First:
+
+```bash
+docker exec -ti postgres-postgres-1 psql --user postgres
+```
+
+Then:
+
+```sql
+CREATE DATABASE [DBNAME];
+CREATE USER [DBNAME] WITH PASSWORD '...';
+GRANT ALL PRIVILEGES ON DATABASE [DBNAME] TO [DBNAME] WITH GRANT OPTION;
+ALTER DATABASE [DBNAME] OWNER TO [DBNAME];
+```
+
+Now that the user exists, exit from the `psql` REPL shell and perform the following actions:
+
+```bash
+# Copy a database backup into the container
+docker cp ./postgres/[DBNAME].sql postgres-postgres-1:/dump.sql
+
+# Load the data into the database
+docker exec -ti postgres-postgres-1 psql --username '[DBNAME]' --dbname '[DBNAME]' --file '/dump.sql'
+
+# Remove the dump file from the container
+docker exec postgres-postgres-1 rm /dump.sql
+```
+
+#### Restoring docker backups
+The backups service stores a configuration file at: `/etc/borg_backup_script/backup_configuration.yaml` which
+contains an overview of backups. We can use it with a script to loop over the available backups and restore them
+(in a similar way to how the backup script itself creates the backups). This script assumes that we want to restore
+all backups, change where necessary.
+
+Copy this script to the local folder (still assuming this is the folder which contains the `docker_volumes` and `postgres` directories):
+
+> **Note**: This assumes that the containers are already running. They will be stopped by the script. The script also assumes that
+> these directories already exist.
+
+```bash
+#!/bin/bash
+# Copied and altered from backup_script.sh:
+configurationFileLocation='/etc/borg_backup_script/backup_configuration.yaml'
+nrOfConfigurations="$(yq '.docker_volume_backups | length' <"$configurationFileLocation")"
+
+for ((i = 0 ; i < "$nrOfConfigurations" ; i++)); do
+    dockerConfiguration="$(yq ".docker_volume_backups[$i]" <"$configurationFileLocation")"
+    containerName="$(echo "$dockerConfiguration" | jq -r '.container_name')"
+    volumeName="$(echo "$dockerConfiguration" | jq -r '.volume_name')"
+
+    specificVolumeBackupPath="./docker_volumes/$volumeName"
+
+    echo "Backup up Docker volume: $volumeName from running container: $containerName..."
+
+    echo "Stopping container: $containerName..."
+    docker stop "$containerName"
+
+    echo "Starting new container which copies over files..."
+    start=$SECONDS
+    docker run --rm -v "$specificVolumeBackupPath:/source" -v "$volumeName:/target" --entrypoint "ash" \
+        alpine -c "rm -rf /target/*; cp -rf /source/* /target/"
+
+    elapsedSeconds=$(( SECONDS - start ))
+    echo "Copying succeeded (in $elapsedSeconds seconds), restarting container..."
+    docker start "$containerName"
+done
+```