# Lab — Volumes, RAID 5/6/10 and ZFS

**Time:** 120 minutes — 10 setup, 10 packages and baseline, 50 for the five arrays (10 each), 10 buffer, 20 evidence, 10 teardown.

**Prerequisite:** the S1 account, project and SSH key; the S2 habit of deleting what you create. The S2 Droplet is already gone, so this lab creates its own.

**You need to know:** a block volume is a raw disk attached to one Droplet, and it bills until it is deleted. `mdadm` builds a software RAID array from several volumes; ZFS builds a pool from vdevs. RAID and raidz keep a service online when a disk fails, but they do not restore a deleted file. A rebuild or resilver is the promise under test: the array is not protected until the repair has finished and the state is clean.

**Cost ceiling:** **$0.20** — an `s-2vcpu-4gb` Droplet for about two hours at $0.036/hour, plus 50 GiB of volumes at $0.10/GiB-month for the same two hours. Detached volumes still bill, so the teardown deletes all five.

**Safe-data acknowledgement:** I have read the lab safe-data policy. I will use only the course-supplied synthetic media and will not upload real personal media to any cloud service.

## Task

Build five 10 GiB Block Storage volumes, benchmark a local-disk and a single-volume baseline, then create and measure mdadm RAID 5, RAID 6 and RAID 10 and ZFS raidz1 and raidz2. Fail or offline one disk in every array, watch the rebuild or resilver finish, compare capacity and throughput, and tear everything down.

**What you submit (ten items):**

1. `summary.csv` and the raw fio JSON files.
2. The `lsblk` output and the whole-disk by-id list.
3. Each mdadm array's `--detail` and `/proc/mdstat` in normal, degraded and rebuilt states.
4. The RAID10 `--fail` → `--remove` → `--zero-superblock` → `--add` sequence.
5. Each ZFS pool's `zpool status` in normal, degraded, resilvering and completed states.
6. The capacity table and `storage_results.json` from the notebook.
7. The sanitized command log.
8. The empty Droplet, volume and firewall lists plus `doctl balance get`.
9. The 150–250-word "what this proves" note.
10. Your terminal transcript, if you keep one.

Scored 3/2/1/0: 3 = all five arrays built, all seven configurations measured, every repair clean, teardown and cost respected; 2 = one check missing; 1 = partial or no teardown; 0 = nothing, unsafe data, or billed resources left behind.

**Path note.** This brief is the step-by-step path: type each command and read what it prints. The verified reference run is `run.sh` + `remote.sh` + `verify.sh` in this folder; `COMMANDS.md` is its sanitized command record. Do not run the script in place of the steps — the observation is the lab.

**Reference-run control only.** These commands are how `run.sh` drives the phases; the learner never types them.

```bash
scp -i <private-key> remote.sh root@<droplet-ip>:/root/remote.sh
ssh -i <private-key> root@<droplet-ip> "FIO_RUNTIME=10 bash /root/remote.sh base"
ssh -i <private-key> root@<droplet-ip> "FIO_RUNTIME=10 bash /root/remote.sh raid5"
ssh -i <private-key> root@<droplet-ip> "FIO_RUNTIME=10 bash /root/remote.sh raid6"
ssh -i <private-key> root@<droplet-ip> "FIO_RUNTIME=10 bash /root/remote.sh raid10"
ssh -i <private-key> root@<droplet-ip> "FIO_RUNTIME=10 bash /root/remote.sh zfs1"
ssh -i <private-key> root@<droplet-ip> "FIO_RUNTIME=10 bash /root/remote.sh zfs2"
ssh -i <private-key> root@<droplet-ip> "FIO_RUNTIME=10 bash /root/remote.sh summary"
```

## Before you start: credentials and the panel path

### 0.1 Create a scoped token (panel only)

Panel: **API → Tokens → Generate New Token**, name it `course-s3-<id>`, expiry **90 days**, and grant the custom scopes:

- `droplet:read`, `droplet:create`, `droplet:delete`
- `block_storage:read`, `block_storage:create`, `block_storage:delete`
- `block_storage_action:read`
- `tag:read`, `tag:create`
- `firewall:read`, `firewall:create`, `firewall:delete`
- `ssh_key:read` (`ssh_key:create` is only needed if you let the reference run import its throwaway key)
- `project:read`, `project:assign_resource`
- `image:read`, `region:read`, `size:read`, `action:read`
- `account:read` and `billing:read` (for `account get` and `balance get`)

Token creation has **no CLI path**; the panel is the only route. Copy the token once — the panel will not show it again. The scopes are the least privilege this lab needs; `block_storage:*` is the volume scope name, not `volume:*`.

### 0.2 Point `doctl` at the token

```bash
doctl auth init --context course-s3
```

*What it does:* starts an interactive prompt that stores the token in a named `doctl` context called `course-s3`, so you do not paste the token into every command. The token itself is never written into this repository.

```bash
export DIGITALOCEAN_CONTEXT=course-s3
```

*What it does:* selects that context for this shell. If you close the terminal, export it again; the token stays in the config file.

### 0.3 Confirm the account and that nothing is left over

```bash
doctl account get
```

*What it does:* proves the token works and shows the account's Droplet limit. Look at the status column: it must be `active`, and the limit must be at least 3.

```bash
doctl compute volume list --format ID,Name --no-header
```

*What it does:* lists existing volumes. **Expected: no output.** A volume from an earlier lab that is still present is billing; delete it before you add five more.

```bash
doctl compute droplet list --format ID,Name --no-header
```

*What it does:* confirms no Droplet is running. Expected: no output.

```bash
doctl compute image list --public --format Slug --no-header | grep -qx rockylinux-10-x64
```

*What it does:* confirms the Rocky Linux 10 image slug exists in the region catalogue. **Expected: no output and exit status 0**; a printed error means the slug has changed.

```bash
doctl projects list --format ID,Name,IsDefault --no-header
```

*What it does:* shows the project UUID that the Droplet will join. Copy the UUID; you will pass it as `<project-uuid>` below.

## Step 1 — SSH key, tags and firewall

### 1.1 Choose the SSH key

```bash
doctl compute ssh-key list --format ID,Name,FingerPrint --no-header
```

*What it does:* lists the keys already in the account. Use the S1 key's fingerprint; the private key stays on your laptop.

**Matching fingerprints.** `doctl` and the Control Panel show **MD5** fingerprints (`aa:bb:…`), while `ssh-keygen` prints SHA256 by default (`SHA256:…`). Match the key by its **name** in the list, or ask `ssh-keygen` for the same form if you are comparing by hand:

```bash
ssh-keygen -E md5 -lf ~/.ssh/id_ed25519.pub
```

*What it does:* prints the MD5 fingerprint of your public key in the form the DigitalOcean list uses, so the comparison is exact.

**Reference run only.** The unattended runner cannot use your private key, so it generates a throwaway key and imports it:

```bash
ssh-keygen -t ed25519 -N "" -C course-<id>-s3 -f .ssh/id_ed25519
```

*What it does:* creates a new ed25519 key pair under `.ssh/` in this folder with no passphrase, for the runner only. The `.ssh/` directory is gitignored and is deleted at teardown.

```bash
doctl compute ssh-key list --format FingerPrint --no-header
```

*What it does:* checks whether the throwaway key is already in the account, so the runner imports it only once and does not create duplicates on a retry.

```bash
doctl compute ssh-key import course-<id>-s3-key --public-key-file .ssh/id_ed25519.pub
```

*What it does:* uploads the public half so the Droplet will accept the matching private key. The runner deletes this key at teardown.

```bash
doctl compute ssh-key list --format ID,FingerPrint --no-header
```

*What it does:* shows the numeric key ID that `droplet create` needs. Put it in `<key-id>`.

### 1.2 Create the tags

Panel: **Networking → Tags → Create Tag**.

```bash
doctl compute tag create course:42-05-0112
```

*What it does:* creates the course tag. The firewall below targets Droplets by the `week:3` tag, and a firewall cannot reference a tag that does not exist yet.

```bash
doctl compute tag create student:<id>
```

*What it does:* tags this student's resources so they can be found and deleted together.

```bash
doctl compute tag create week:3
```

*What it does:* creates the tag the firewall attaches to. Every resource in this lab carries all three tags.

### 1.3 Create the firewall

Panel: **Networking → Firewalls → Create Firewall**, name `course-<id>-s3-fw`, inbound **SSH (22)** from **your IP**, outbound **all**, and apply it to the tag `week:3`.

```bash
MYIP=$(curl -s https://api.ipify.org)
```

*What it does:* stores your public IP in a shell variable. The firewall should admit SSH only from this address, not from the whole internet.

```bash
doctl compute firewall create --name "course-<id>-s3-fw" \
  --inbound-rules "protocol:tcp,ports:22,address:$MYIP/32" \
  --outbound-rules "protocol:tcp,ports:0,address:0.0.0.0/0 protocol:udp,ports:0,address:0.0.0.0/0 protocol:icmp,ports:0,address:0.0.0.0/0" \
  --tag-names "week:3" --format ID --no-header
```

*What it does:* creates a default-deny firewall that admits inbound TCP 22 from your IP only, allows all outbound traffic, and applies to Droplets carrying the `week:3` tag. Save the ID it prints for teardown. If you do this in the panel instead, the CLI form beside it produces the same rules.

## Step 2 — Five 10 GiB volumes

Panel: **Volumes → Create Volume**, region **SGP1**, size **10 GiB**, name `course-<id>-s3-vol1` … `vol5`. The panel cannot set tags on volumes; add them with the CLI in this step or from **Volumes → the volume → Tags**.

```bash
doctl compute volume create course-<id>-s3-vol1 --region sgp1 --size 10GiB \
  --desc "course 42-05-0112 S3" --tag "course:42-05-0112,student:<id>,week:3" \
  --format ID --no-header
```

*What it does:* creates the first 10 GiB volume in Singapore. It bills from this moment, whether or not it is attached.

```bash
doctl compute volume create course-<id>-s3-vol2 --region sgp1 --size 10GiB \
  --desc "course 42-05-0112 S3" --tag "course:42-05-0112,student:<id>,week:3" \
  --format ID --no-header
```

*What it does:* creates the second volume with the same region, size and tags.

```bash
doctl compute volume create course-<id>-s3-vol3 --region sgp1 --size 10GiB \
  --desc "course 42-05-0112 S3" --tag "course:42-05-0112,student:<id>,week:3" \
  --format ID --no-header
```

*What it does:* creates the third volume, the last member S3's raidz1 pool needs.

```bash
doctl compute volume create course-<id>-s3-vol4 --region sgp1 --size 10GiB \
  --desc "course 42-05-0112 S3" --tag "course:42-05-0112,student:<id>,week:3" \
  --format ID --no-header
```

*What it does:* creates the fourth volume, the spare that replaces a failed member in the RAID layouts.

```bash
doctl compute volume create course-<id>-s3-vol5 --region sgp1 --size 10GiB \
  --desc "course 42-05-0112 S3" --tag "course:42-05-0112,student:<id>,week:3" \
  --format ID --no-header
```

*What it does:* creates the fifth and final volume. Collect all five IDs; the Droplet step consumes them as one comma-separated list.

```bash
doctl compute volume list --format ID,Name,Size --no-header
```

*What it does:* confirms five volumes exist and shows their IDs. Check that each is `10 GiB` and in `sgp1`; a volume in another region cannot attach to this Droplet.

```bash
doctl compute volume list --format ID,Name --no-header | awk '/course-<id>-s3-vol/ {print $1}' | paste -sd, -
```

*What it does:* filters the list client-side and joins the IDs into one comma-separated string for the Droplet create command. `volume list` has no tag filter, so `awk` does the matching.

## Step 3 — The Droplet with its volumes

Panel: **Create Droplet** → **SGP1**, **Rocky Linux 10 ×64**, **Basic / Regular / `s-2vcpu-4gb`**, select your SSH key, add the three tags. Attach the volumes either at creation (**Add Block Storage**) or afterwards from **Volumes → Attach to Droplet**; the CLI form below attaches them at creation so they are present on first boot.

```bash
doctl compute droplet create course-<id>-s3 --region sgp1 --image rockylinux-10-x64 \
  --size s-2vcpu-4gb --ssh-keys <key-id> \
  --tag-names "course:42-05-0112,student:<id>,week:3" \
  --volumes <vol1-id>,<vol2-id>,<vol3-id>,<vol4-id>,<vol5-id> \
  --project-id <project-uuid> --wait --format ID --no-header
```

*What it does:* creates the Droplet and attaches all five volumes in one API call, then waits for it to boot. `--wait` polls the Droplet until it is active. Drop `--project-id` if you are not using a project.

```bash
doctl compute droplet get <droplet-id> --format PublicIPv4 --no-header
```

*What it does:* prints the Droplet's public IP directly. Use this or the list command below; the IP is what SSH and the firewall rule refer to.

```bash
doctl compute droplet list --format ID,Name,PublicIPv4 --no-header
```

*What it does:* shows the Droplet's public IP; that IP is what the firewall allows SSH from, and what you pass to `ssh`.

```bash
doctl compute droplet list -o json > results/droplet-before.json
```

*What it does:* saves the pre-lab resource state as evidence. This is the "before" half of the teardown proof.

```bash
doctl compute volume list -o json > results/volume-before.json
```

*What it does:* saves the five attached volumes and their Droplet links as evidence.

```bash
doctl compute firewall list -o json > results/firewall-before.json
```

*What it does:* saves the firewall list before the lab so the teardown proof shows your rule and then its removal.

## Step 4 — Connect, discover the devices, install the tools

```bash
ssh -i <private-key> root@<droplet-ip>
```

*What it does:* opens a root shell on the Droplet. Everything in steps 4–12 runs inside that shell; the `doctl` commands in steps 8 and 13 run on your laptop.

```bash
dnf -y install mdadm fio xfsprogs jq
```

*What it does:* installs the RAID tool, the benchmark, the XFS formatter and `jq` for parsing fio's JSON. Rocky's repositories carry all four.

```bash
uname -a
```

*What it does:* records the running kernel; the ZFS DKMS build depends on this exact version, so the evidence should show it.

```bash
df -h
```

*What it does:* records the filesystem layout before any volume is formatted, so the before/after difference is visible.

```bash
dnf repolist
```

*What it does:* lists the enabled repositories. If the ZFS repository is missing after the release package install, this is where you see it.

```bash
mokutil --sb-state
```

*What it does:* reports Secure Boot state. DigitalOcean Droplets normally have it off; if it is on, the unsigned ZFS module will not load and the install needs the fallback path.

```bash
ls -1 /dev/disk/by-id/scsi-0DO_Volume_* | sort
```

*What it does:* lists the devices the five volumes appear as. **Use only the names without `-part`**: the whole disk belongs to the array and must not carry a partition table.

```bash
lsblk -o NAME,SIZE,TYPE,SERIAL,MOUNTPOINT
```

*What it does:* shows size, type and mount point for every block device. Confirm five 10 GiB disks and no mount points on them; save the output as evidence.

### 4.1 Pre-lab ZFS install

This is the pre-lab step. It takes a few minutes; run it now, before the arrays, so it is not on the critical path. If ZFS is already installed (`zfs version` works), skip to step 5.

```bash
dnf -y install https://zfsonlinux.org/epel/zfs-release-3-0.el10.noarch.rpm
```

*What it does:* adds the OpenZFS repository for Enterprise Linux 10. The file name is the EL10 release package, not a kernel module.

```bash
dnf -y install epel-release
```

*What it does:* adds EPEL, which supplies dependencies the ZFS packages expect.

```bash
dnf -y install kernel-devel-$(uname -r)
```

*What it does:* installs the headers for the **running** kernel. DKMS needs them to compile the module; without them the install appears to succeed but `modprobe` fails.

```bash
dnf -y install zfs
```

*What it does:* installs OpenZFS and its DKMS module, building the kernel module against the installed headers.

```bash
modprobe zfs
```

*What it does:* loads the module into the running kernel. **Expected: no output.** An error here is the install failure to fix before the ZFS steps.

```bash
zfs version
```

*What it does:* prints the userland and kernel module versions. The verified run reports `zfs-2.2.11` and `zfs-kmod-2.2.11`; record the numbers in your evidence.

```bash
dnf -y install mdadm fio xfsprogs jq
```

*What it does:* re-checks the four tools after the ZFS install; on a fresh Droplet the earlier step already installed them, so this is fast and idempotent.

## Step 5 — Baseline: local disk and one volume

```bash
mkdir -p /root/fio-local
```

*What it does:* creates the directory the local-disk test writes into.

```bash
mkdir -p /root/results
```

*What it does:* creates the results directory every test writes its JSON to.

The four fio commands below are the template for every configuration. Run them once against `/root/fio-local`, then again against each mount point, changing only `--directory` and the output name.

```bash
fio --name=seqread --directory=/root/fio-local --rw=read --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-local-disk-seqread.json
```

*What it does:* reads 1 MiB blocks sequentially for 10 seconds and writes the result as JSON. 1 MiB blocks and queue depth 8 approximate streaming video playback.

```bash
fio --name=seqwrite --directory=/root/fio-local --rw=write --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-local-disk-seqwrite.json
```

*What it does:* writes 1 MiB blocks sequentially for 10 seconds — the ingest pattern. On a network volume this is the write rate an ingest actually gets.

```bash
fio --name=randread --directory=/root/fio-local --rw=randread --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-local-disk-randread.json
```

*What it does:* issues 4 KiB random reads from four jobs at queue depth 32 for 10 seconds, approximating a database index lookup.

```bash
fio --name=randwrite --directory=/root/fio-local --rw=randwrite --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-local-disk-randwrite.json
```

*What it does:* issues 4 KiB random writes — the pattern where parity costs the most. Compare this number between RAID 5 and RAID 10.

```bash
mkfs.xfs -f /dev/disk/by-id/scsi-0DO_Volume_<vol1>
```

*What it does:* puts an XFS filesystem on the first whole volume. `-f` allows a reformat of the blank device; there is no partition table.

```bash
mkdir -p /mnt/base
```

*What it does:* creates the mount point for the single-volume baseline.

```bash
mount /dev/disk/by-id/scsi-0DO_Volume_<vol1> /mnt/base
```

*What it does:* mounts the XFS filesystem so fio can write files through the filesystem, which is the same path the array tests use.

```bash
fio --name=seqread --directory=/mnt/base --rw=read --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-single-volume-seqread.json
```

*What it does:* repeats the sequential read against the single volume and records it under its own configuration name.

```bash
fio --name=seqwrite --directory=/mnt/base --rw=write --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-single-volume-seqwrite.json
```

*What it does:* repeats the sequential write against the single volume.

```bash
fio --name=randread --directory=/mnt/base --rw=randread --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-single-volume-randread.json
```

*What it does:* repeats the random read against the single volume.

```bash
fio --name=randwrite --directory=/mnt/base --rw=randwrite --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-single-volume-randwrite.json
```

*What it does:* repeats the random write against the single volume.

```bash
umount /mnt/base
```

*What it does:* unmounts the baseline. An array cannot claim a volume that is still mounted.

## Step 6 — mdadm RAID 5: three active disks plus a spare

```bash
mdadm --stop /dev/md5
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol1>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol2>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol3>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol4>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol5>
```

*What it does:* stops any array named `md5` left from an earlier attempt and clears the mdadm metadata from all five volumes. `--stop` on a non-existent array prints an error and continues; a stale superblock is what makes the next `--create` refuse a device.

```bash
mdadm --create /dev/md5 --level=5 --raid-devices=3 --spare-devices=1 --run \
  /dev/disk/by-id/scsi-0DO_Volume_<vol1> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol2> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol3> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol4>
```

*What it does:* creates a RAID 5 array from three active devices (two data plus one parity) with the fourth as a hot spare. Usable capacity is about 20 GiB. `--run` starts it without waiting for confirmation.

```bash
cat /proc/mdstat
```

*What it does:* shows the array's live state. During the initial sync you will see `[2/3]` or `[resync]`; wait until it reads `[3/3] [UUU]` before benchmarking.

```bash
mdadm --detail /dev/md5
```

*What it does:* prints the member disks, their roles, and `State : clean`. Save this as the normal-state evidence.

```bash
mkdir -p /mnt/raid5
```

*What it does:* creates the mount point.

```bash
mkfs.xfs -f /dev/md5
```

*What it does:* puts XFS on the array, not on a member disk. Formatting a member would destroy the array.

```bash
mount /dev/md5 /mnt/raid5
```

*What it does:* mounts the array so the four fio tests can write through a filesystem.

```bash
fio --name=seqread --directory=/mnt/raid5 --rw=read --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid5-seqread.json
```

*What it does:* measures streaming read on the array. Compare with the single-volume row: RAID 5 reads can use more than one disk.

```bash
fio --name=seqwrite --directory=/mnt/raid5 --rw=write --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid5-seqwrite.json
```

*What it does:* measures streaming write; parity is computed per stripe, so this is usually below the read rate.

```bash
fio --name=randread --directory=/mnt/raid5 --rw=randread --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid5-randread.json
```

*What it does:* measures small random reads, which can be served from multiple disks in parallel.

```bash
fio --name=randwrite --directory=/mnt/raid5 --rw=randwrite --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid5-randwrite.json
```

*What it does:* measures small random writes; watch for the read-modify-write penalty relative to RAID 10.

### Break a disk and watch the spare take over

```bash
echo 50000 > /proc/sys/dev/raid/speed_limit_min
```

*What it does:* raises the rebuild speed floor. The default is deliberately gentle; for a 20 GiB array in a lab, a faster rebuild finishes inside the session.

```bash
echo 200000 > /proc/sys/dev/raid/speed_limit_max
```

*What it does:* raises the ceiling above the floor so the kernel can use the available bandwidth.

```bash
mdadm --fail /dev/md5 /dev/disk/by-id/scsi-0DO_Volume_<vol1>
```

*What it does:* marks the first member as failed. The hot spare should begin rebuilding within seconds; this is the failure the array was built to absorb.

```bash
mdadm --detail /dev/md5
```

*What it does:* shows the degraded/recovering state: one device failed, the spare active. Save it as evidence.

```bash
cat /proc/mdstat
```

*What it does:* shows the rebuild progress in the `[=>...]` bar. Save this state before it disappears.

```bash
mdadm --remove /dev/md5 /dev/disk/by-id/scsi-0DO_Volume_<vol1>
```

*What it does:* removes the failed member from the array so it can be re-added as the new spare.

```bash
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol1>
```

*What it does:* clears the old mdadm superblock so the kernel treats the disk as new. Skipping this is the usual cause of a re-add that is refused.

```bash
mdadm --add /dev/md5 /dev/disk/by-id/scsi-0DO_Volume_<vol1>
```

*What it does:* adds the repaired disk back as the new spare. The array now has three active members and one spare again.

```bash
mdadm --detail /dev/md5
```

*What it does:* confirms the array returned to `[3/3] [UUU]` and `State : clean` — the rebuild promise, tested.

```bash
cat /proc/mdstat
```

*What it does:* confirms no resync or recovery is still running. Only now is the array protected again.

## Step 7 — mdadm RAID 6: four active disks plus a spare

```bash
umount /mnt/raid5
```

*What it does:* unmounts the RAID 5 array.

```bash
mdadm --stop /dev/md5
```

*What it does:* stops the array and releases the member devices for reuse.

```bash
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol1>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol2>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol3>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol4>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol5>
```

*What it does:* clears each member's superblock before the next array is created. A leftover superblock can make `--create` refuse a device or, worse, assemble the old array.

```bash
mdadm --create /dev/md6 --level=6 --raid-devices=4 --spare-devices=1 --run \
  /dev/disk/by-id/scsi-0DO_Volume_<vol1> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol2> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol3> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol4> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol5>
```

*What it does:* creates RAID 6 from four active devices (two data plus two parity) and the fifth as a spare. It survives two failures and still yields about 20 GiB usable.

```bash
cat /proc/mdstat
```

*What it does:* shows the initial sync. Wait for `[4/4] [UUUU]` before measuring.

```bash
mdadm --detail /dev/md6
```

*What it does:* confirms four active members, one spare and `State : clean`.

```bash
mkdir -p /mnt/raid6
```

*What it does:* creates the mount point.

```bash
mkfs.xfs -f /dev/md6
```

*What it does:* formats the array with XFS.

```bash
mount /dev/md6 /mnt/raid6
```

*What it does:* mounts it for the four tests.

```bash
fio --name=seqread --directory=/mnt/raid6 --rw=read --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid6-seqread.json
```

*What it does:* measures sequential read on RAID 6.

```bash
fio --name=seqwrite --directory=/mnt/raid6 --rw=write --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid6-seqwrite.json
```

*What it does:* measures sequential write; two parity blocks cost more than RAID 5's one.

```bash
fio --name=randread --directory=/mnt/raid6 --rw=randread --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid6-randread.json
```

*What it does:* measures random reads on RAID 6.

```bash
fio --name=randwrite --directory=/mnt/raid6 --rw=randwrite --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid6-randwrite.json
```

*What it does:* measures random writes, where the dual-parity read-modify-write shows.

```bash
echo 50000 > /proc/sys/dev/raid/speed_limit_min
```

*What it does:* raises the rebuild speed floor for this array.

```bash
echo 200000 > /proc/sys/dev/raid/speed_limit_max
```

*What it does:* raises the ceiling.

```bash
mdadm --fail /dev/md6 /dev/disk/by-id/scsi-0DO_Volume_<vol1>
```

*What it does:* fails the first member; the spare starts rebuilding.

```bash
mdadm --detail /dev/md6
```

*What it does:* records the degraded/recovering state.

```bash
cat /proc/mdstat
```

*What it does:* records rebuild progress.

```bash
mdadm --remove /dev/md6 /dev/disk/by-id/scsi-0DO_Volume_<vol1>
```

*What it does:* removes the failed member.

```bash
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol1>
```

*What it does:* clears its superblock.

```bash
mdadm --add /dev/md6 /dev/disk/by-id/scsi-0DO_Volume_<vol1>
```

*What it does:* re-adds it as the new spare.

```bash
mdadm --detail /dev/md6
```

*What it does:* confirms `[4/4] [UUUU]` and clean state after the rebuild.

```bash
cat /proc/mdstat
```

*What it does:* confirms the rebuild finished.

## Step 8 — mdadm RAID 10: four active disks, fifth added on failure

```bash
umount /mnt/raid6
```

*What it does:* unmounts RAID 6.

```bash
mdadm --stop /dev/md6
```

*What it does:* stops the array.

```bash
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol1>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol2>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol3>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol4>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol5>
```

*What it does:* clears the superblock on each of the five members before the RAID 10 array is created.

```bash
mdadm --create /dev/md10 --level=10 --raid-devices=4 --run \
  /dev/disk/by-id/scsi-0DO_Volume_<vol1> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol2> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol3> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol4>
```

*What it does:* creates a RAID 10 array from four devices — two mirrored pairs, striped. There is no spare: usable capacity is 20 GiB and one failure per pair is survivable.

```bash
cat /proc/mdstat
```

*What it does:* shows the initial sync; wait for `[4/4] [UUUU]`.

```bash
mdadm --detail /dev/md10
```

*What it does:* confirms the layout and clean state.

```bash
mkdir -p /mnt/raid10
```

*What it does:* creates the mount point.

```bash
mkfs.xfs -f /dev/md10
```

*What it does:* formats the array.

```bash
mount /dev/md10 /mnt/raid10
```

*What it does:* mounts it for the four tests.

```bash
fio --name=seqread --directory=/mnt/raid10 --rw=read --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid10-seqread.json
```

*What it does:* measures sequential read on the mirrored stripe.

```bash
fio --name=seqwrite --directory=/mnt/raid10 --rw=write --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid10-seqwrite.json
```

*What it does:* measures sequential write; every write goes to two disks, so the raw rate is half before striping.

```bash
fio --name=randread --directory=/mnt/raid10 --rw=randread --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid10-randread.json
```

*What it does:* measures random reads; either copy of a block can serve them.

```bash
fio --name=randwrite --directory=/mnt/raid10 --rw=randwrite --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid10-randwrite.json
```

*What it does:* measures random writes, the place RAID 10 is expected to beat RAID 5/6.

```bash
echo 50000 > /proc/sys/dev/raid/speed_limit_min
```

*What it does:* raises the rebuild speed floor.

```bash
echo 200000 > /proc/sys/dev/raid/speed_limit_max
```

*What it does:* raises the ceiling.

```bash
mdadm --fail /dev/md10 /dev/disk/by-id/scsi-0DO_Volume_<vol1>
```

*What it does:* fails one member of the first mirror pair. There is no spare, so the array stays degraded until you add the fifth volume by hand.

```bash
mdadm --detail /dev/md10
```

*What it does:* records the degraded state — this is the window a hot spare would have shortened.

```bash
cat /proc/mdstat
```

*What it does:* shows `[4/3] [UU_U]` or similar: still serving, one member short.

```bash
mdadm --remove /dev/md10 /dev/disk/by-id/scsi-0DO_Volume_<vol1>
```

*What it does:* removes the failed member.

```bash
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol1>
```

*What it does:* clears the failed member's superblock.

```bash
mdadm --add /dev/md10 /dev/disk/by-id/scsi-0DO_Volume_<vol5>
```

*What it does:* adds the fifth volume as the replacement and begins the rebuild. Note the disk changing: RAID 10 needs a new disk, not the failed one back.

```bash
mdadm --detail /dev/md10
```

*What it does:* confirms the array returned to `[4/4] [UUUU]` and clean state.

```bash
cat /proc/mdstat
```

*What it does:* confirms the rebuild finished.

## Step 9 — ZFS raidz1: three disks, offline and replace

```bash
umount /mnt/raid10
```

*What it does:* unmounts the RAID 10 array.

```bash
mdadm --stop /dev/md10
```

*What it does:* stops it and releases the members.

```bash
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol1>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol2>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol3>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol4>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol5>
```

*What it does:* clears the mdadm metadata on all five volumes; the ZFS pool must not see a foreign RAID signature.

```bash
wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol1>
wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol2>
wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol3>
wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol4>
wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol5>
```

*What it does:* wipes any remaining filesystem or RAID signature from each volume; ZFS refuses a device with a foreign signature.

```bash
zpool create -f tank1 raidz1 /dev/disk/by-id/scsi-0DO_Volume_<vol1> /dev/disk/by-id/scsi-0DO_Volume_<vol2> /dev/disk/by-id/scsi-0DO_Volume_<vol3>
```

*What it does:* creates pool `tank1` with one three-disk `raidz1` vdev. It survives one disk failure and yields about 20 GiB usable.

```bash
zfs create tank1/data
```

*What it does:* creates a dataset inside the pool; fio writes into the dataset, which is how ZFS is normally used.

```bash
fio --name=seqread --directory=/tank1/data --rw=read --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-zfs-raidz1-seqread.json
```

*What it does:* measures sequential read through ZFS. Checksums and the ARC are in this path, unlike the raw mdadm path.

```bash
fio --name=seqwrite --directory=/tank1/data --rw=write --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-zfs-raidz1-seqwrite.json
```

*What it does:* measures sequential write; copy-on-write allocates new blocks instead of overwriting.

```bash
fio --name=randread --directory=/tank1/data --rw=randread --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-zfs-raidz1-randread.json
```

*What it does:* measures random reads; the ARC and the checksummed block layout are visible here.

```bash
fio --name=randwrite --directory=/tank1/data --rw=randwrite --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-zfs-raidz1-randwrite.json
```

*What it does:* measures random writes; expect this to be ZFS's weakest number because every small write allocates and checksums a new block.

```bash
zpool status tank1
```

*What it does:* shows the pool `ONLINE` and the vdev with three devices. Save it as the normal-state evidence.

```bash
zpool offline tank1 /dev/disk/by-id/scsi-0DO_Volume_<vol1>
```

*What it does:* takes one disk out of the vdev without failing the pool. This is the controlled failure — the pool is now `DEGRADED` and still serves reads and writes.

```bash
zpool status tank1
```

*What it does:* shows the degraded state and the offline device. Save it.

```bash
zpool replace tank1 /dev/disk/by-id/scsi-0DO_Volume_<vol1> /dev/disk/by-id/scsi-0DO_Volume_<vol4>
```

*What it does:* replaces the offline disk with the fourth volume and starts the resilver. Note that ZFS rebuilds the allocated blocks, not the whole disk.

```bash
zpool status tank1
```

*What it does:* shows the resilver in progress and how much data is being scanned. Save it as the resilvering evidence.

```bash
zpool status tank1
```

*What it does:* wait until the same command shows `resilvered` and both vdev members `ONLINE`. This is the promise tested; save the completed state.

## Step 10 — ZFS raidz2: four disks, survive two

```bash
zpool destroy -f tank2
```

*What it does:* destroys any `tank2` left by an earlier attempt, so the next `create` starts from a clean name.

```bash
zpool destroy -f tank1
```

*What it does:* destroys the first pool. Double-check the pool name: there is no undo for a `destroy`.

```bash
wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol1>
wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol2>
wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol3>
wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol4>
wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol5>
```

*What it does:* wipes any leftover label from each volume before the next pool is created.

```bash
zpool create -f tank2 raidz2 /dev/disk/by-id/scsi-0DO_Volume_<vol1> /dev/disk/by-id/scsi-0DO_Volume_<vol2> /dev/disk/by-id/scsi-0DO_Volume_<vol3> /dev/disk/by-id/scsi-0DO_Volume_<vol4>
```

*What it does:* creates pool `tank2` with one four-disk `raidz2` vdev — two parity blocks, two failures survivable, about 20 GiB usable.

```bash
zfs create tank2/data
```

*What it does:* creates the dataset for the tests.

```bash
fio --name=seqread --directory=/tank2/data --rw=read --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-zfs-raidz2-seqread.json
```

*What it does:* measures sequential read on raidz2.

```bash
fio --name=seqwrite --directory=/tank2/data --rw=write --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-zfs-raidz2-seqwrite.json
```

*What it does:* measures sequential write; two parity blocks cost write bandwidth.

```bash
fio --name=randread --directory=/tank2/data --rw=randread --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-zfs-raidz2-randread.json
```

*What it does:* measures random reads on raidz2.

```bash
fio --name=randwrite --directory=/tank2/data --rw=randwrite --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-zfs-raidz2-randwrite.json
```

*What it does:* measures random writes; compare the cost with raidz1's single parity.

```bash
zpool status tank2
```

*What it does:* shows the pool online with four devices. Save it.

```bash
zpool offline tank2 /dev/disk/by-id/scsi-0DO_Volume_<vol1>
```

*What it does:* takes one disk offline; raidz2 remains `DEGRADED` but still has one parity block to spare.

```bash
zpool status tank2
```

*What it does:* records the degraded state.

```bash
zpool replace tank2 /dev/disk/by-id/scsi-0DO_Volume_<vol1> /dev/disk/by-id/scsi-0DO_Volume_<vol5>
```

*What it does:* replaces it with the fifth volume and starts the resilver.

```bash
zpool status tank2
```

*What it does:* records the resilver progress.

```bash
zpool status tank2
```

*What it does:* wait for `resilvered` and all four devices `ONLINE`; save the completed state.

## Step 11 — Collect the summary and leave the Droplet

```bash
cat /root/results/summary.csv
```

*What it does:* prints the 28 measurements — seven configurations × four tests — that the notebook pivots. If a row is missing, run the missing fio command again.

```bash
lsblk -o NAME,SIZE,TYPE,SERIAL,MOUNTPOINT
```

*What it does:* captures the final device state for the evidence bundle.

```bash
exit
```

*What it does:* closes the SSH session and returns you to the laptop.

```bash
mkdir -p results
```

*What it does:* creates a local folder for the raw evidence on the laptop.

```bash
scp -i <private-key> -r root@<droplet-ip>:/root/results/. results/
```

*What it does:* copies every fio JSON, mdadm/ZFS status file and the command logs from the Droplet. This must happen **before** teardown; deleting the Droplet deletes the evidence.

```bash
bash verify.sh
```

*What it does:* checks the collected evidence: 28 summary rows, 28 fio JSON files, the rebuild and resilver states, the empty teardown lists and the balance. It exits non-zero if anything is missing. (`OUT=results` is the default; set `OUT=results` if you copied elsewhere.)

## Step 12 — Teardown

```bash
ssh -i <private-key> root@<droplet-ip> "bash /root/remote.sh cleanup"
```

*What it does:* unmounts every array and dataset, exports the pools and stops mdadm so the volumes are released cleanly. `remote.sh cleanup` runs the three commands below; run them by hand if you did not copy the script.

```bash
umount /mnt/raid5 /mnt/raid6 /mnt/raid10 /mnt/base /tank1/data /tank2/data
```

*What it does:* releases every mount point in one call, including the two ZFS datasets. An unmount that reports "not mounted" is harmless when an earlier step already released it.

```bash
zpool export -a
```

*What it does:* exports every imported pool so ZFS releases the member volumes; a volume held by an imported pool cannot be deleted cleanly.

```bash
mdadm --stop --scan
```

*What it does:* stops every running md array so the kernel releases its member volumes.

```bash
doctl compute droplet delete <droplet-id> --force
```

*What it does:* deletes the Droplet. The volumes survive this and keep billing until step 12.4.

```bash
doctl compute droplet get <droplet-id>
```

*What it does:* confirms the Droplet is gone; **expected: a "not found" error.** A successful response means the delete has not completed, and deleting volumes now may fail while they are still attached.

```bash
doctl compute volume delete <vol1-id> --force
doctl compute volume delete <vol2-id> --force
doctl compute volume delete <vol3-id> --force
doctl compute volume delete <vol4-id> --force
doctl compute volume delete <vol5-id> --force
```

*What it does:* deletes all five volumes. A volume that is still attached may refuse once while the Droplet releases; wait a few seconds and retry. **Delete, do not detach** — a detached volume still bills.

```bash
doctl compute firewall delete <firewall-id> --force
```

*What it does:* deletes the `week:3` firewall.

```bash
doctl compute ssh-key delete <key-id> --force
```

*What it does:* deletes the throwaway key the reference run imported. Skip this if you used your own S1 key: that key stays for the rest of the course.

```bash
doctl compute droplet list -o json > results/droplet-after.json
```

*What it does:* records the empty Droplet list as teardown proof.

```bash
doctl compute volume list -o json > results/volume-after.json
```

*What it does:* records the empty volume list. **This is the proof that nothing billable is left.**

```bash
doctl compute firewall list -o json > results/firewall-after.json
```

*What it does:* records the firewall list without your rule.

```bash
doctl balance get
```

*What it does:* shows the account balance and month-to-date usage. Record it; the run must stay under $0.20.

## Evidence bundle

Submit the ten items listed under **Task**, plus the notebook's `storage_results.json`. The 150–250-word note must answer: which layout would you put the weekly working set on, which would you not, and what did the rebuild or resilver time tell you that the capacity table did not?

## Fallback chain

| Failure | Fallback (same evidence shape) |
|---|---|
| The token lacks volume scopes | Create the token again with `block_storage:*`; custom scopes cannot be edited after creation. Or use the panel for volumes and keep the CLI for the rest. |
| `doctl` blocked or not installed | Panel path for Droplets, volumes and firewalls; volume **tags** are CLI-only, so record that as a deviation. |
| Five volumes refused | Create three 10 GiB volumes and run raidz1 only; label the missing configurations in the evidence. |
| ZFS install fails | Reinstall `kernel-devel-$(uname -r)`, run `modprobe zfs`, and retry `dnf -y install zfs`. If it still fails, complete the three mdadm arrays and use the published ZFS numbers in `data/` for the ZFS rows, labelled as a fallback. |
| RAID 5/6 has no spare capacity on the account | Run the array without a spare and add a volume at failure; record that the degraded window was longer. |
| No account at all | Instructor demo account with a booked slot; the evidence stays individual. |

## Teardown checklist

- Droplet deleted.
- All five volumes **deleted** (not detached).
- `week:3` firewall deleted.
- The imported throwaway SSH key deleted (reference run only).
- `results/droplet-after.json`, `results/volume-after.json` and `results/firewall-after.json` empty.
- `doctl balance get` recorded.
- The project, your S1 key and the spend alert stay; the scoped token may stay until it expires or be deleted deliberately.
