# Lab — Docker Compose and a Nextcloud NAS

**Time:** 120 minutes — 10 setup, 80 core, 20 evidence, 10 teardown. The Nextcloud image pull is the long pole; start it before the concept recap ends.

**Prerequisite:** the S1 account, the S3 volume experience and the S4 zone. Your root domain's Cloudflare zone is Active, and you have a `Zone:DNS:Edit` token. You need `doctl`, `ssh`, `scp`, `dig`, `curl`, `jq` and `rclone` on your laptop.

**You need to know:** an image is a read-only template; a container adds a thin writable layer that disappears when the container is recreated; a bind mount puts a host path into the container so the data outlives it. A health check tests whether a service is usable, not whether its process exists. RAID1 mirrors every write, so it is availability rather than backup. Cloudflare in **Full** mode encrypts to the origin without validating the origin certificate.

**Safe-data acknowledgement:** I have read the lab safe-data policy. I will use only the course-supplied synthetic media and will not upload real personal media to any cloud service.

**Reference run.** Every command below is the sequence the verified runner executes; `run.sh`, `remote.sh` and `COMMANDS.md` in this folder are that record. Type each command yourself and read its output — do not run the script as your lab.

## Task

Build a two-volume mirrored NAS at `/srv/nas`, run Nextcloud 35 with MariaDB, Redis and Caddy as a four-service Compose stack on top of it, publish it behind a proxied `nas.<root>` record, back it up to a private R2 bucket, prove a restore, start the stack with the NAS unmounted, diagnose and recover, then tear everything down.

## Step 0 — Pre-flight

Confirm the tools, the zone and the zone's SSL mode.

```bash
doctl account get
```
Shows the active DigitalOcean account and its limits. If this fails, `doctl auth init` and paste a token with Droplet, volume and firewall scopes.

*Panel:* DigitalOcean Control Panel → **Account**.

```bash
doctl compute image list --public --format Slug --no-header | grep -qx rockylinux-10-x64
```
Exits 0 only if the Rocky Linux 10 image slug exists in SGP1. No output means success.

```bash
dig NS <root> +short
```
Returns Cloudflare nameservers such as `phil.ns.cloudflare.com.` — proof the zone is Active and can hold the proxied record.

*Panel:* Cloudflare dashboard → your zone → **Overview**.

```bash
curl -s -H "Authorization: Bearer $CLOUDFLARE_TOKEN" \
  "https://api.cloudflare.com/client/v4/zones?name=<root>" | jq -r '.result[0].id'
```
Returns the zone id used by every DNS call. The token needs `Zone:Read` for this call and `Zone:DNS:Edit` for the next one.

## Step 1 — Firewall

Admit SSH from your address and 443 from everywhere; drop the rest.

```bash
MYIP=$(curl -s https://api.ipify.org)
```
Records your public address. SSH will be allowed only from it.

```bash
doctl compute firewall create --name "course-<id>-s5" \
  --inbound-rules "protocol:tcp,ports:22,address:$MYIP/32 protocol:tcp,ports:443,address:0.0.0.0/0" \
  --outbound-rules "protocol:tcp,ports:0,address:0.0.0.0/0 protocol:udp,ports:0,address:0.0.0.0/0 protocol:icmp,ports:0,address:0.0.0.0/0" \
  --tag-names "week:5"
```
Creates a cloud firewall attached to every Droplet tagged `week:5`. Only 22 and 443 are reachable from outside; outbound traffic is unrestricted.

*Panel:* **Networking → Firewalls → Create**, then add the same two inbound rules and apply the tag `week:5`.

```bash
doctl compute firewall list --format ID,Name,InboundRules
```
Confirms the rules and the tag. Note the firewall id for teardown.

## Step 2 — Two raw volumes

Create two 10 GiB volumes with no filesystem, so `mdadm` can own the whole disk.

```bash
doctl compute volume create "course-<id>-s5-nas1" --region sgp1 --size 10GiB --tag "week:5" --format ID --no-header
doctl compute volume create "course-<id>-s5-nas2" --region sgp1 --size 10GiB --tag "week:5" --format ID --no-header
```
Creates two block-storage volumes in SGP1. Omitting `--fs-type` is deliberate: a filesystem on the volume would hide the whole disk from `mdadm`.

*Panel:* **Volumes → Create**, 10 GiB, SGP1, tag `week:5`.

```bash
doctl compute volume list --format ID,Name,Size,Region,DropletIDs,Tags
```
Both volumes should show `10 GiB`, `sgp1` and no Droplet yet. A detached volume still bills.

## Step 3 — Droplet with both volumes attached

```bash
doctl compute droplet create "course-<id>-s5" --image rockylinux-10-x64 \
  --size s-2vcpu-4gb --region sgp1 --ssh-keys <fingerprint> \
  --volumes <vol1-id>,<vol2-id> --tag-names "course:42-05-0112,week:5" --wait \
  --format ID --no-header
```
Creates the 2 vCPU / 4 GiB Droplet with both volumes attached at boot. Attaching at creation is what makes `/dev/disk/by-id` visible in the first boot.

*Panel:* **Droplets → Create**, Rocky Linux 10, SGP1, `s-2vcpu-4gb`, add both volumes under **Add Block Storage**, add the tag `week:5`.

```bash
doctl compute droplet get <droplet-id> --format PublicIPv4 --no-header
```
Returns the public IPv4 used by DNS and SSH.

## Step 4 — DNS and SSL/TLS mode

Point a proxied record at the Droplet.

```bash
curl -s -H "Authorization: Bearer $CLOUDFLARE_TOKEN" \
  "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/settings/ssl" | jq -r '.result.value'
```
Prints the zone's current SSL/TLS mode, for example `strict`. Record it: teardown restores this value after the lab's **Full** mode.

```bash
ZONE_ID=$(curl -s -H "Authorization: Bearer $CLOUDFLARE_TOKEN" \
  "https://api.cloudflare.com/client/v4/zones?name=<root>" | jq -r '.result[0].id')
curl -s -X POST -H "Authorization: Bearer $CLOUDFLARE_TOKEN" -H "Content-Type: application/json" \
  "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
  --data "{\"type\":\"A\",\"name\":\"nas\",\"content\":\"<droplet-ip>\",\"proxied\":true,\"ttl\":1}"
```
Creates the `nas.<root>` A record with the orange cloud on. `proxied:true` is what makes Cloudflare terminate TLS and forward to the origin.

*Panel:* **DNS → Records → Add record**, type A, name `nas`, IPv4 the Droplet address, **Proxy status: Proxied**.

```bash
dig +short nas.<root>
```
Returns Cloudflare anycast addresses, not the Droplet IP. That is the check that the record is proxied.

```bash
curl -s -H "Authorization: Bearer $CLOUDFLARE_TOKEN" \
  "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records?name=nas.<root>" \
  | jq -r '.result[] | "\(.name) \(.type) proxied=\(.proxied)"'
```
Reads the record back from the API as evidence. The `proxied=true` field is the machine-checkable version of the orange cloud.

```bash
curl -s -X PATCH -H "Authorization: Bearer $CLOUDFLARE_TOKEN" -H "Content-Type: application/json" \
  "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/settings/ssl" --data '{"value":"full"}'
```
Sets SSL/TLS mode **Full**: Cloudflare encrypts to the origin but does not validate Caddy's internal certificate. **Full (strict)** would reject it.

*Panel only:* **SSL/TLS → Overview → Full**. The API call above is the runner-verified path; the panel is the learner's path.

## Step 5 — Reach the Droplet and install Docker

The verified runner imports a throwaway key for the run. If your S1 key is already registered, skip these three commands and pass its fingerprint to `--ssh-keys` in Step 3; the rest of the sequence is identical.

```bash
ssh-keygen -t ed25519 -N "" -C course-s5-nas -f .ssh/id_ed25519
```
Creates a disposable key pair for the run.

```bash
doctl compute ssh-key import course-s5-nas-key --public-key-file .ssh/id_ed25519.pub
```
Registers the public half with the account. The private half stays on the laptop.

```bash
doctl compute ssh-key list --format ID,FingerPrint --no-header | grep <fingerprint>
```
Confirms the key is registered and gives the fingerprint for `--ssh-keys`.

```bash
ssh -i <private-key> root@<droplet-ip> "mkdir -p /root/s5 /root/results"
```
Creates the project directory and the results directory on the Droplet before any file is copied.

```bash
scp -i <private-key> remote.sh compose.yaml Caddyfile .env.example \
  data/catalogue.json data/photo-01.jpg root@<droplet-ip>:/root/s5/
```
Stages the phased runner, the stack files and the synthetic upload on the Droplet. Your private key stays on the laptop.

```bash
ssh -i <private-key> root@<droplet-ip>
```
Connects as root. Keep this session open; the remaining commands run inside the Droplet.

```bash
dnf -y install dnf-plugins-core jq curl
```
Installs the repository tooling and `jq`, used to read JSON evidence.

```bash
dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo
```
Adds Docker's RHEL repository. Rocky 10 uses the RHEL 9/10 repository; the RPMs are compatible.

```bash
rpm --import https://download.docker.com/linux/rhel/gpg
```
Imports Docker's signing key before the first install. On a fresh Droplet the image may still be running its own package transaction, which holds the RPM lock; retry the import after a few seconds if it reports a lock error.

```bash
dnf -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
```
Installs Docker Engine and the Compose plugin. There is no separate `docker-compose` binary; the plugin is invoked as `docker compose`.

```bash
dnf -y install iptables
```
Installs the user-space `iptables`. The Docker bridge driver programs NAT rules through it.

```bash
KVER=$(uname -r)
KVER=${KVER%.x86_64}
dnf -y install "kernel-modules-extra-${KVER}"
depmod -a
modprobe xt_addrtype
```
Installs the extra kernel modules that match the **running** kernel and loads `xt_addrtype`, which Docker's NAT rules require. DigitalOcean's Rocky 10 image ships a kernel whose module tree lacks the extras; without this, `docker` fails with `Extension addrtype revision 0 not supported, missing kernel module?`. The provider's kernel is part of the platform contract you have to reconcile with the software you install.

```bash
systemctl reset-failed docker
systemctl enable --now docker
```
Clears any failed start from the package installation, then starts Docker now and at boot.

```bash
docker version
docker compose version
```
`docker version` must show both a Client and a Server section; a Client-only output means the daemon is not running.

```bash
systemctl is-active firewalld
firewall-cmd --state
```
Records the host firewall state. The Rocky image ships `firewalld` inactive; the DigitalOcean Cloud Firewall filters in front of the host, so the host firewall is not the boundary in this lab.

```bash
dnf -y install rclone
```
Tries the distribution package. Rocky Linux 10 does not ship `rclone` in its enabled repositories, so this normally reports `No match for argument: rclone`; that is expected, not a failure.

```bash
curl -fsSL https://downloads.rclone.org/rclone-current-linux-amd64.zip -o /tmp/rclone.zip
dnf -y install unzip
unzip -q -o /tmp/rclone.zip -d /tmp/rclone
install -m 755 /tmp/rclone/rclone-*-linux-amd64/rclone /usr/local/bin/rclone
```
Installs rclone from the official static binary: download, unpack and place it on `PATH`. A static binary is the portable installation path when a distribution has no package.

```bash
rclone version
```
Prints the version. `command not found` means the binary was not placed on `PATH`.

## Step 6 — The mirrored NAS

```bash
dnf -y install mdadm xfsprogs
```
Installs the array tool and the XFS formatter.

```bash
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT
```
Lists the two 10 GiB disks with no `FSTYPE`. If a filesystem is present, the volume was created with one and must be recreated.

```bash
ls -l /dev/disk/by-id/scsi-0DO_Volume_*
```
Shows the stable whole-disk names, for example `scsi-0DO_Volume_course-<id>-s5-nas1`. Use the whole disk, not a `-part1` suffix; the prototype verified this.

*Panel:* **Volumes → the volume → Attached to**, where the device name is also listed.

```bash
mdadm --create /dev/md1 --level=1 --raid-devices=2 --run \
  /dev/disk/by-id/scsi-0DO_Volume_course-<id>-s5-nas1 \
  /dev/disk/by-id/scsi-0DO_Volume_course-<id>-s5-nas2
```
Creates the RAID1 array over the two whole disks. `--run` keeps it non-interactive when the disks look unfamiliar.

```bash
mdadm --wait /dev/md1
cat /proc/mdstat
```
Waits for the initial mirror build. `mdstat` must show `[2/2]` and `[UU]`; the resync counter may still move briefly.

```bash
mdadm --detail /dev/md1
```
Shows `Raid Level : raid1`, `Array Size : 10476544` (about 10 GiB), `State : clean` and both devices as `active sync`.

```bash
mkfs.xfs -f /dev/md1
```
Formats the array. `-f` forces the format past any stale signature; XFS is the filesystem DigitalOcean documents for volumes, while the array itself is not a documented DigitalOcean configuration.

```bash
mkdir -p /srv/nas
mount /dev/md1 /srv/nas
```
Mounts the array at the state path. The directory exists before the mount so the mount point has somewhere to appear.

```bash
mdadm --detail --scan > /etc/mdadm.conf
cat /etc/mdadm.conf
```
Persists the array definition so it can be assembled at boot, then prints it to show the saved line.

```bash
blkid -s UUID -o value /dev/md1
```
Prints the filesystem UUID used in the next line.

```bash
echo "UUID=<uuid> /srv/nas xfs defaults,nofail 0 2" >> /etc/fstab
```
Adds the mount to `/etc/fstab`. `nofail` lets the host boot if the array is missing — the trade that makes the later empty-instance failure reachable instead of an unbootable machine.

```bash
findmnt /srv/nas
df -h /srv/nas
```
`findmnt` proves `/srv/nas` is a mount point and shows `/dev/md1`; `df` shows about 10 GiB of usable space from 20 GiB raw.

```bash
mkdir -p /srv/nas/nextcloud /srv/nas/config /srv/nas/db
ls -ld /srv/nas /srv/nas/nextcloud /srv/nas/config /srv/nas/db
```
Creates the three state directories. Everything the application must keep goes into one of them.

## Step 7 — Compose, Caddy and the environment

The lab folder ships `compose.yaml` and `Caddyfile`. Copy them to the Droplet, or create them from the versions shown in the week folder.

```bash
mkdir -p /srv/s5 && cd /srv/s5
```
Creates the deployment directory that holds the Compose file, the Caddyfile and the secrets file.

```bash
cp .env.example .env
```
Copies the template. The real `.env` is never committed.

```bash
sed -i 's|<root>|<root-domain>|g' .env
```
Replaces the domain placeholder with your zone.

```bash
sed -i 's|^MYSQL_PASSWORD=.*|MYSQL_PASSWORD=<random>|' .env
sed -i 's|^MARIADB_ROOT_PASSWORD=.*|MARIADB_ROOT_PASSWORD=<random>|' .env
sed -i 's|^NEXTCLOUD_ADMIN_PASSWORD=.*|NEXTCLOUD_ADMIN_PASSWORD=<random>|' .env
```
Sets three independent secrets. Generate them with `openssl rand -hex 16`; never reuse one password.

```bash
chmod 600 .env
```
Makes the secrets file readable only by root.

```bash
cat > .env.rclone <<'EOF'
RCLONE_CONFIG_R2_TYPE=s3
RCLONE_CONFIG_R2_PROVIDER=Cloudflare
RCLONE_CONFIG_R2_ACL=private
RCLONE_CONFIG_R2_NO_CHECK_BUCKET=true
EOF
```
Writes the non-secret half of the rclone configuration. rclone reads `RCLONE_CONFIG_R2_*` from the environment, so no credential file is needed on disk.

```bash
echo "RCLONE_CONFIG_R2_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com" >> .env.rclone
echo "RCLONE_CONFIG_R2_ACCESS_KEY_ID=<access-key>" >> .env.rclone
echo "RCLONE_CONFIG_R2_SECRET_ACCESS_KEY=<secret-key>" >> .env.rclone
echo "R2_BUCKET=course-s5-backup" >> .env.rclone
echo "R2_PREFIX=" >> .env.rclone
chmod 600 .env.rclone
```
Adds the endpoint, the bucket-scoped R2 keys and the bucket name. `R2_PREFIX` is empty for your own bucket; the instructor fallback gives each student a prefix.

*Panel:* **R2 → Manage R2 API Tokens → Create API token**, Object Read & Write, one bucket, then copy the endpoint, access key and secret. Create the bucket `course-s5-backup` under **R2 → Create bucket** first.

```bash
docker compose config --quiet
```
Validates the Compose file and the environment interpolation before anything starts. Silence means valid.

## Step 8 — Pull and start

```bash
docker compose pull
```
Pulls the four images: `caddy:2-alpine`, `nextcloud:35-apache`, `mariadb:lts` and `redis:7-alpine`. About 500 MiB compressed in total.

```bash
docker compose up -d
```
Creates the containers in dependency order: MariaDB and Redis first, then Nextcloud, then Caddy. `-d` returns the prompt.

```bash
docker compose ps
docker compose images
```
`ps` shows four services with their state and ports; only Caddy publishes `0.0.0.0:443->443/tcp`. `images` shows the pinned image tags.

```bash
docker compose logs --tail=50 nextcloud db
```
Shows the first-boot install progress. Nextcloud waits for the database health check, then installs itself using the `NEXTCLOUD_ADMIN_*` and `MYSQL_*` variables.

## Step 9 — First boot, trusted domain and upload

```bash
curl -kfsS --resolve nas.<root>:443:127.0.0.1 "https://nas.<root>/status.php"
```
Reads Nextcloud's status through Caddy, resolving the hostname to localhost so the request does not leave the Droplet. `-k` accepts Caddy's internal certificate. Wait until it reports `"installed":true`.

```bash
docker compose exec -u www-data nextcloud php occ status
```
Shows `installed: true` and the application version from inside the container.

```bash
docker compose exec -u www-data nextcloud php occ config:system:set overwriteprotocol --value=https
docker compose exec -u www-data nextcloud php occ config:system:set overwrite.cli.url --value="https://nas.<root>"
```
Tells Nextcloud that every generated link is HTTPS behind the proxy. Without this, WebDAV and share links point at `http://` and break at the edge.

```bash
docker compose exec -u www-data nextcloud php occ config:system:get overwriteprotocol
```
Confirms the value is `https`.

```bash
docker compose exec -u www-data nextcloud php occ config:system:get instanceid
```
Records the instance id. This is the identifier the failure step compares; copy it into your notes.

```bash
curl -kfsS --resolve nas.<root>:443:127.0.0.1 -u "<admin>:<admin-password>" \
  -T catalogue.json "https://nas.<root>/remote.php/dav/files/<admin>/catalogue.json"
curl -kfsS --resolve nas.<root>:443:127.0.0.1 -u "<admin>:<admin-password>" \
  -T photo-01.jpg "https://nas.<root>/remote.php/dav/files/<admin>/photo-01.jpg"
```
Uploads the synthetic catalogue and photo through WebDAV. The files land in the admin user's directory on the mounted NAS.

*Panel:* open `https://nas.<root>`, sign in, **Files → Upload**, and choose the two files. The UI path produces the same state.

```bash
docker compose exec -u www-data nextcloud php occ files:scan --all
```
Rebuilds the file cache after the uploads. The scan output lists the files it indexed.

```bash
docker compose exec nextcloud ls -l "/var/www/html/data/<admin>/files"
```
Shows the two uploaded files inside the container, on the bind mount.

```bash
curl -fsS "https://nas.<root>/status.php"
```
The same status through the public Cloudflare edge, with certificate validation on. A 200 with `"installed":true` proves the proxy, the edge certificate and the origin all agree.

## Step 10 — Prove the container is disposable

```bash
docker compose exec nextcloud sha256sum "/var/www/html/data/<admin>/files/catalogue.json" \
  | tee checksum-before.txt
```
Records the catalogue hash from inside the container and saves it to a file for the comparison.

```bash
docker compose up -d --force-recreate nextcloud
```
Destroys the Nextcloud container and creates a new one from the same image. The data directory and configuration are bind mounts, so the new container sees the same state.

```bash
docker compose exec nextcloud sha256sum "/var/www/html/data/<admin>/files/catalogue.json" \
  | tee checksum-after.txt
```
The hash must be identical to the first one. Equal hashes are the thesis in one line.

```bash
diff <(cut -d' ' -f1 checksum-before.txt) <(cut -d' ' -f1 checksum-after.txt)
```
Compares only the hash fields. No output means the two are identical; any output means the data moved with the container.

```bash
docker compose ps
docker compose ps --status running --services
docker inspect --format '{{.Name}} {{json .State.Health.Status}}' $(docker compose ps -q nextcloud)
docker inspect --format '{{.Name}} {{json .State.Health.Status}}' $(docker compose ps -q db)
docker inspect --format '{{.Name}} {{json .State.Health.Status}}' $(docker compose ps -q redis)
```
Shows every service running, the filtered running list, and the three health checks reporting `"healthy"`. Caddy has no health check; its `running` state plus the successful HTTPS request is its evidence.

## Step 11 — Back up to R2

```bash
docker compose exec -u www-data nextcloud php occ maintenance:mode --on
```
Freezes writes so the data copy and the dump describe the same moment.

```bash
rclone copy /srv/nas/nextcloud "r2:$R2_BUCKET/$R2_PREFIX/nextcloud" --progress
rclone copy /srv/nas/config "r2:$R2_BUCKET/$R2_PREFIX/config" --progress
```
Copies the user data and the instance configuration to the private bucket. The progress line reports transferred bytes and the final listing.

```bash
docker compose exec db mariadb-dump --single-transaction --default-character-set=utf8mb4 \
  -u"<db-user>" -p"<db-password>" nextcloud > /srv/nas/s5-db.sql
```
Dumps the database with a consistent snapshot. Copying the live database files is not a backup; this is.

```bash
ls -l /srv/nas/s5-db.sql
```
Shows the dump size. A dump of a few kilobytes is correct for a fresh instance.

```bash
rclone copy /srv/nas/s5-db.sql "r2:$R2_BUCKET/$R2_PREFIX/db" --progress
```
Copies the dump to the bucket.

```bash
docker compose exec -u www-data nextcloud php occ maintenance:mode --off
```
Leaves maintenance mode.

```bash
rclone ls "r2:$R2_BUCKET/$R2_PREFIX"
```
Lists the objects in the bucket. You should see the data trees, the configuration tree and the dump. This listing is evidence that the backup exists off the machine.

## Step 12 — Restore drill

```bash
docker compose exec -T nextcloud sha256sum "/var/www/html/data/<admin>/files/photo-01.jpg"
```
Records the original photo hash.

```bash
curl -kfsS -X DELETE --resolve nas.<root>:443:127.0.0.1 -u "<admin>:<admin-password>" \
  "https://nas.<root>/remote.php/dav/files/<admin>/photo-01.jpg"
docker compose exec nextcloud test ! -e "/var/www/html/data/<admin>/files/photo-01.jpg"
```
Deletes the file through WebDAV and asserts it is gone. This is the "disaster": a real deletion, mirrored instantly by RAID1.

```bash
rm -rf /tmp/restore && mkdir -p /tmp/restore
rclone copy "r2:$R2_BUCKET/$R2_PREFIX/nextcloud/<admin>/files/photo-01.jpg" /tmp/restore --progress
```
Clears the scratch directory, recreates it and fetches the backed-up object from R2.

```bash
curl -kfsS --resolve nas.<root>:443:127.0.0.1 -u "<admin>:<admin-password>" \
  -T /tmp/restore/photo-01.jpg "https://nas.<root>/remote.php/dav/files/<admin>/photo-01.jpg"
```
Uploads the restored file back into Nextcloud.

```bash
docker compose exec -T nextcloud sha256sum "/var/www/html/data/<admin>/files/photo-01.jpg"
```
The hash must equal the original. Equal hashes prove the backup is readable and complete, not merely present.

## Step 13 — Failure: start the stack with the NAS unmounted

```bash
docker compose down
umount /srv/nas
```
Stops the stack and detaches the array. `/srv/nas` is now an ordinary empty directory on the root disk.

```bash
findmnt /srv/nas
```
Prints nothing. That empty output is the first clue; `docker compose logs` would look normal.

```bash
docker compose up -d
```
Starts the stack again. Docker creates the bind-mount sources on the root disk because the array is not mounted, so MariaDB initialises a new database and Nextcloud installs a new instance. No error is printed.

```bash
curl -kfsS --resolve nas.<root>:443:127.0.0.1 "https://nas.<root>/status.php"
```
Reports `"installed":true` — but for the empty instance, not yours.

```bash
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT
```
Still shows `md1` and both volumes; the storage exists but is not mounted at `/srv/nas`.

```bash
docker inspect --format '{{json .Mounts}}' $(docker compose ps -q nextcloud)
```
Shows the bind-mount sources as `/srv/nas/...`, which looks correct. Docker did what the file said; the path itself was wrong.

```bash
docker compose exec -u www-data nextcloud php occ status
docker compose exec -u www-data nextcloud php occ config:system:get instanceid
```
The instance id differs from the one recorded at first boot. A different id means a different installation.

```bash
docker compose logs --tail=50 db nextcloud
```
Shows a fresh database initialisation and a fresh Nextcloud install, with no error. The absence of an error is the lesson: the failure is silent.

## Step 14 — Recovery

```bash
mount /dev/md1 /srv/nas
findmnt /srv/nas
```
Reattaches the array. `findmnt` must show `/dev/md1` again.

```bash
docker compose up -d --force-recreate
```
Recreates every container. A plain `restart` is not enough: the old container's mount namespace still points at the empty root-disk directory.

```bash
curl -kfsS --resolve nas.<root>:443:127.0.0.1 "https://nas.<root>/status.php"
docker compose exec -u www-data nextcloud php occ config:system:get instanceid
```
The status reports installed and the instance id equals the original. Your instance is back.

```bash
docker compose exec -T nextcloud sha256sum "/var/www/html/data/<admin>/files/catalogue.json"
```
The catalogue hash equals the value from Step 10. The data survived the failure and the recovery.

```bash
docker compose exec nextcloud ls -l "/var/www/html/data/<admin>/files"
```
Both uploaded files are present again.

## Step 15 — Record the evidence

```bash
uname -r
cat /etc/os-release | grep -E '^(NAME|VERSION)='
findmnt /srv/nas
df -h /srv/nas
mdadm --detail /dev/md1
cat /proc/mdstat
```
Captures the host and array facts the notebook parses: kernel, OS, mount, capacity, RAID level and state.

Copy the evidence to your laptop before teardown:

```bash
exit
scp -i <private-key> -r root@<droplet-ip>:/root/results ./results
verify.sh                      # OUT=results by default
```
`exit` leaves the SSH session; the `scp` collects the recorded outputs. `verify.sh` asserts the mirror state, the health checks, both checksum comparisons, the backup listing, the restore hash, the instance ids and the teardown lists; it must print `VERIFY: all S5 evidence checks pass` and exit 0 before you submit. The notebook then reads this folder.

## Evidence bundle

Ten items, scored 3/2/1/0:

1. `compose.yaml` with secrets redacted, plus `.env.example`.
2. `docker compose ps` and the health JSON for Nextcloud, MariaDB and Redis.
3. A screenshot of Nextcloud showing the uploaded catalogue and photo.
4. `findmnt /srv/nas`, `mdadm --detail /dev/md1` and `df -h /srv/nas`.
5. The catalogue checksum before and after `--force-recreate`.
6. The unmounted-NAS diagnosis (`findmnt`, `lsblk`, `.Mounts`, two different instance ids) and the recovered instance.
7. The dump size, the `rclone ls` listing and the restored photo checksum.
8. The sanitized command log (`commands-host.log` and the phase logs).
9. The teardown log, the empty Droplet/volume/firewall lists, the empty bucket prefix and `doctl balance get`.
10. The 150–250-word *what this proves* note.

## Failure and recovery

The deliberate failure is starting the stack with `/srv/nas` unmounted. The diagnosis is `findmnt` (not a mount point), `lsblk` (the array exists), `.Mounts` (Docker used the path it was given) and the instance id (a different installation). Recovery is `mount /dev/md1 /srv/nas` followed by `docker compose up -d --force-recreate`. The extension for fast students is the secret-mismatch crash loop: change `MYSQL_PASSWORD` in `.env`, recreate Nextcloud and read the database authentication error.

## Cost ceiling

Under **$0.20**: a `s-2vcpu-4gb` Droplet at $0.03571/hour for about two hours (about $0.07), two 10 GiB volumes at $0.10/GiB-month (about $0.006), R2 within the free tier. The volumes bill whether attached or detached.

## Teardown

Delete the Droplet first, then both volumes, the firewall, the `nas` record, the bucket prefix and the R2 token. Restore the zone's original SSL/TLS mode. Keep the project, the S1 key, the DO token, the zone and the spend alert.

```bash
docker compose down -v
umount /srv/nas
mdadm --stop /dev/md1
```
Stops the stack, removes the Caddy named volume, unmounts the array and stops it. Deleting the Droplet would discard all of this anyway; the explicit stop is what records a clean shutdown before the host disappears.

```bash
doctl compute droplet delete <droplet-id> --force
```
Deletes the Droplet. Wait until `doctl compute droplet get <droplet-id>` returns not found before deleting the volumes.

```bash
doctl compute volume delete <vol1-id> --force
doctl compute volume delete <vol2-id> --force
```
Deletes both volumes. A volume that is merely detached still bills.

```bash
doctl compute firewall delete <firewall-id> --force
```
Removes the cloud firewall.

```bash
curl -s -X DELETE -H "Authorization: Bearer $CLOUDFLARE_TOKEN" \
  "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/<record-id>"
```
Deletes the `nas` record. *Panel:* **DNS → Records → the record → Delete**.

```bash
curl -s -X PATCH -H "Authorization: Bearer $CLOUDFLARE_TOKEN" -H "Content-Type: application/json" \
  "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/settings/ssl" --data '{"value":"strict"}'
```
Restores the SSL/TLS mode you recorded in Step 4. Replace `strict` with that recorded value; the demo zone is `strict`, and leaving the whole zone on the looser mode outlives the lab.

```bash
rclone purge "r2:$R2_BUCKET/$R2_PREFIX"
```
Empties the backup prefix. If the bucket is yours, delete the bucket in the panel afterwards.

```bash
doctl compute droplet list --format ID,Name
doctl compute volume list --format ID,Name
doctl compute firewall list --format ID,Name
doctl balance get
```
The three lists must be empty of your resources, and the balance is the cost evidence. Delete the R2 API token in the panel; a token that is never used is still a credential.

## Fallback

- **Zone inactive:** use the instructor demo hostname `nas.scigrad.com` and record it as fallback evidence.
- **Docker Hub blocked:** use the instructor-pulled images from a registry mirror; the Compose file is unchanged.
- **R2 blocked:** use the instructor bucket with your student prefix; the commands differ only by `R2_BUCKET` and `R2_PREFIX`.
- **Volume limit reached:** create one 20 GiB volume and use it as a single-disk XFS filesystem; record that the mirror was not built.
- **Docker unavailable:** run the same Compose stack locally and submit the published NAS evidence; label each item as fallback.

## Notebook

Open [the lab notebook](notebook.ipynb). It reads the recorded `data/run/` evidence, asserts the mirror, the four-service health, the checksum survival, the backup objects and the restored hash, and writes `nas_results.json`. Run it after you copy your own `results/` into `data/run/` if you want your numbers instead of the recorded run.
