Storage, RAID and ZFS

Lab: Volumes, RAID 5/6/10 and ZFS. PLO: PLO2 / PLO1. Points: 7.

Overview

The unit keeps about 1 TB of video and a 200 GB image/3D asset library, and the archive grows every month. The working set — the footage being cut this week — has to sit on fast storage attached to a machine. The long tail can sit in object storage until someone requests it.

The one big idea for this session: RAID buys availability, not backup, and a rebuild is a promise you test. Combining disks can keep a media service online when one disk fails, but it cannot bring back a file someone deleted, and the protection only counts if a rebuild actually completes. The lab builds five arrays, measures each one, breaks a disk in every array, and watches the repair finish.

The targets from the scenario spine: the last 90 days of footage restores in under four hours, the deep archive in under 24 hours, and the weekly working set survives the loss of one disk without a restore from backup.

Lesson

Three shapes of storage

Storage comes in three shapes, and a media pipeline uses all of them.

Shape What it is This course
Object storage Whole objects addressed by key in a bucket, reached over HTTP, with versioning and lifecycle rules Cloudflare R2 (session 4)
Block storage A fixed number of blocks on a device attached to one virtual machine, formatted with a filesystem DigitalOcean Volumes (this session)
File storage A shared filesystem with directories, mounted by several machines at once The Nextcloud NAS (session 5)

The access pattern decides. A video editor reads and writes large files at high speed and needs a filesystem with normal file semantics, so the working set lives on block storage. An archive is read rarely, by key, from anywhere, so it lives in object storage. The same 1 TB costs very different amounts in the two shapes, which is the calculation at the end of this page.

How a cloud block volume works

A DigitalOcean Volume is network-attached block storage in one region. You create it at a size (the lab uses five of 10 GiB), attach it to a Droplet, and the Droplet sees a raw disk device. The volume is not a filesystem yet: you choose the filesystem, or you put several volumes together into a RAID array and format the array.

Four facts shape everything in this lab.

  • A volume attaches to one Droplet at a time. It is not shared storage; session 5 builds shared file storage on top of volumes.
  • A volume bills while it exists, attached or not. Creating five volumes and forgetting them costs money every hour. Deleting them is part of the lab.
  • DigitalOcean has regions, not availability zones. A volume lives in sgp1 beside the Droplet. Resilience here means RAID plus a copy elsewhere, not a second zone.
  • DigitalOcean documents Ext4 and XFS on Volumes. It does not document mdadm or ZFS. What this lab does is measured, not supported. The page repeats that caveat because a support question later depends on it.

The by-id device names matter. A volume appears as /dev/disk/by-id/scsi-0DO_Volume_<name> plus partition variants such as -part1. The lab always uses the whole-disk name and never the -part* links, and it never puts a partition table on a member disk: the array owns the whole device, and a stale partition signature is a classic way to assemble the wrong thing.

RAID: what combining disks buys

Striping spreads consecutive blocks across disks so reads and writes use several disks at once. It raises throughput and capacity but keeps no copy: one failed disk loses part of every file.

Mirroring writes every block to two disks. One disk can fail and the data is still there, but half the raw capacity buys the copy.

Parity is the third idea. For each stripe, the array stores one extra block computed from the data blocks — usually bitwise XOR. If one data block is missing, XOR the surviving blocks and the parity block to reconstruct it. One parity block per stripe therefore survives one disk failure while costing one disk of capacity across the array. Two parity blocks survive two failures and cost two disks.

Mode Layout Disks needed Survives Usable capacity
RAID 0 stripe only 2+ no disk failure all disks
RAID 1 mirror 2 one disk half
RAID 5 stripe + one parity 3+ one disk (n − 1) disks
RAID 6 stripe + two parity 4+ two disks (n − 2) disks
RAID 10 mirrored stripes 4+ one per mirror pair half

RAID 10 is a stripe of mirrors: blocks are striped across pairs, and each pair is a mirror. It survives one failure per pair, rebuilds by copying a mirror rather than recomputing parity, and gives up half the capacity. In the lab, four active disks give 20 GiB usable, and the fifth volume is added by hand after a failure.

Two costs are invisible in the table. Parity writes are read-modify-write: to update one block, RAID 5 reads the old data and parity, computes the new parity, and writes both. Small random writes therefore cost more than they do on a mirror. A rebuild reads every surviving disk, so it competes with the workload that is still being served. A RAID 5 array that loses a second disk during the rebuild loses the array; RAID 6 exists precisely to survive that window.

mdadm: software RAID on Linux

mdadm assembles ordinary block devices into a Linux software RAID array. The superblock at the start of each member records the array identity and member role, which is how the kernel re-assembles the array after a reboot without a configuration file.

Useful states to read in /proc/mdstat and mdadm --detail:

  • [UUU] — every member is up. [UU_] — one member is missing or failed.
  • State : clean — no background work. State : clean, degraded — running with a failed member. State : clean, degraded, recovering — a spare is rebuilding.
  • [3/3] on a RAID 5 with three active devices means all three active devices are present. A hot spare appears as an extra device in --detail but not in the active count.

A hot spare is a disk the array does not use until a member fails; the kernel then starts rebuilding onto it automatically. The lab gives RAID 5 and RAID 6 one spare each, so a failure begins repairing itself immediately. RAID 10 gets no spare: the fifth volume is added by hand after the failure, which demonstrates the manual path and the time a spare would have saved.

ZFS: pools, checksums and recovery

ZFS is a filesystem and a volume manager in one. You do not format a ZFS device with mkfs; you create a pool from one or more vdevs, and vdevs are where the redundancy lives. A single-disk vdev has no redundancy; a mirror vdev survives all but one of its disks; a raidz1 vdev survives one disk and a raidz2 vdev survives two. Raidz is ZFS's parity layout: it distributes parity across the disks, and unlike RAID 5 it does not suffer the write hole, because every block is written copy-on-write and carries its own checksum.

ZFS checks every block it reads against its checksum. If a mirror or raidz can reconstruct the block from redundancy, ZFS repairs it automatically; a checksum failure that cannot be repaired is reported rather than silently returned. That is a different promise from RAID alone, which detects a failed disk but cannot tell whether the good-looking blocks are correct.

Recovery has its own vocabulary:

  • A resilver is ZFS rebuilding a vdev after a disk is replaced or comes back. It walks the pool's allocated blocks, not the raw disk, so a pool holding less data resilvers faster than a full one.
  • A rebuild is the mdadm equivalent: the kernel copies or reconstructs the whole member onto a replacement or spare.
  • zpool status shows the pool state (ONLINE, DEGRADED, FAULTED) and, during a resilver, a progress line with the amount of data scanned.

The lab's ZFS path is OpenZFS 2.2.11 installed from the zfs-release-3-0.el10 repository with DKMS. DKMS builds the kernel module against the running kernel, which is what makes it survive DigitalOcean's kernel point releases; a prebuilt kmod package lags. Installation is a pre-lab step: it takes a few minutes and the lab block does not have room for it twice.

Measuring disks with fio

fio issues a defined I/O pattern and reports bandwidth and IOPS. The lab runs four tests against each configuration for 10 seconds each:

Test Pattern What it shows
seqread 1 MiB sequential reads, queue depth 8 streaming read, the shape of video playback
seqwrite 1 MiB sequential writes, queue depth 8 streaming write, the shape of an ingest
randread 4 KiB random reads, four jobs, queue depth 32 small-block read, the shape of a database index
randwrite 4 KiB random writes, four jobs, queue depth 32 small-block write, where parity costs the most

Read the numbers as a burst, not a benchmark. DigitalOcean shared Droplets have a per-Droplet storage cap (around 7,500 IOPS and 300 MB/s sustained, with a burst to around 10,000 IOPS and 450 MB/s for up to about 60 seconds). The lab's 10-second tests are shorter than that window, so they measure the burst. The test file is also only 256 MiB, which fits in the Droplet's 4 GB of RAM; after the first pass the page cache can serve reads, which is why a sequential read can appear faster than the advertised cap. The honest comparisons on this page are the write numbers and the ratios between configurations, not an absolute claim about the volume.

The interesting trade-off the measurements show is architectural: mdadm RAID 5/6 is strong on sequential and small random writes, because it is a block layer under a conventional filesystem; ZFS pays for checksums and copy-on-write on small random writes (every write allocates new blocks and updates metadata) but reads large sequential and random blocks quickly. Neither wins everywhere, which is why "which is faster" is the wrong question and "faster at what" is the right one.

Capacity and the cost of keeping it

All five arrays in the lab start from the same 50 GiB of raw volumes and all give 20 GiB usable — the layouts spend the rest differently:

Array Disks used Layout Usable The other 30 GiB
RAID 5 3 + 1 spare 2 data + 1 parity 20 GiB parity and a hot spare
RAID 6 4 + 1 spare 2 data + 2 parity 20 GiB two parity blocks and a spare
RAID 10 4 two mirrored stripes 20 GiB the mirror copies
raidz1 3 parity distributed 20 GiB parity
raidz2 4 two parity blocks 20 GiB parity and the fifth volume unused

Capacity is therefore not the deciding variable. Choose by the redundancy you need, the write pattern you have, and the time a repair takes — then price the bytes.

The price comparison is the point of the session's research question. DigitalOcean block storage is $0.10 per GiB-month; the 1 TB archive would cost about $102 per month on volumes. Cloudflare R2 standard storage is $0.015 per GiB-month with no egress charge, so the same 1 TB is about $15 per month. Block storage is between five and seven times the price of object storage for bytes that are only occasionally read.

There is also an operational limit: five 10 GiB volumes worked in the lab, but 1 TB in 10 GiB pieces is about 103 volumes, and a Droplet accepts 15. Larger volumes exist (up to 16 TiB), but the price per GiB does not change. If the archive has to be online on block storage, it has to be fewer, larger volumes — and the monthly bill arrives either way. Detached volumes still bill, which is why teardown deletes rather than detaches.

That is the tiering decision the research question asks you to cost: keep the working set on block storage, tier the tail to object storage, and set a restore objective you have actually tested.

What the lab proves

The lab is built to make four claims visible:

  1. Five arrays can be created from the same five volumes, and each reports the capacity the table predicts.
  2. Every array survives one disk failure — and RAID 6 and raidz2 survive the degraded state with more room to spare.
  3. Every array completes a rebuild or resilver, and the array returns to a clean state; the elapsed time is the data the availability promise depends on.
  4. The cost and capacity arithmetic points the archive at object storage, not at a bigger RAID array.

The lab records what worked and what did not. It makes no support claim: DigitalOcean documents Ext4 and XFS on Volumes, not mdadm or ZFS.

Check yourself

  1. A RAID 5 array has six 4 TB disks. How much usable capacity does it have, and how many disk failures can it survive? What changes with RAID 6?
  2. Why can a sequential read from a 256 MiB test file appear faster than the Droplet's advertised storage cap?
  3. During a RAID 5 rebuild, why is a second failure fatal while the array is still serving reads?
  4. ZFS checks every block against a checksum. Why does that let a mirror or raidz repair a bad block, while a plain RAID 5 array cannot tell which block is corrupted?
  5. At $0.10/GiB-month for block storage and $0.015/GiB-month for R2, what does keeping the whole 1 TB archive on volumes cost per year, and what would you keep there instead?
สรุปภาษาไทย หน่วยสื่อเก็บวิดีโอประมาณ 1 TB และแอสเซ็ตภาพ/โมเดล 3D อีก 200 GB ชุดทำงานที่ตัดต่ออยู่รายสัปดาห์ควรอยู่บนบล็อกสตอเรจที่ต่อกับเครื่อง และส่วนที่อ่านนาน ๆ ครั้งควรอยู่ในออบเจกต์สตอเรจ แนวคิดหลักคือ RAID ให้ความพร้อมใช้งาน ไม่ใช่แบ็กอัป และการ rebuild เป็นคำสัญญาที่ต้องทดสอบจริง สื่อ-block ของ DigitalOcean มีราคา $0.10 ต่อ GiB-เดือน ขณะที่ R2 ราคา $0.015 ต่อ GiB-เดือน ดังนั้น 1 TB บนบล็อกสตอเรจประมาณ $102 ต่อเดือน เทียบกับประมาณ $15 ต่อเดือนบน R2 ห้าเลย์เอาต์ในแล็บ (RAID 5, RAID 6, RAID 10, raidz1, raidz2) ให้พื้นที่ใช้งาน 20 GiB เท่ากันจาก 50 GiB ดิบ จึงเลือกตามความซ้ำซ้อนและรูปแบบการเขียน ไม่ใช่ตามความจุ
中文总结 媒体单元存放约 1 TB 视频,以及另外 200 GB 的图像/3D 模型资产。每周正在剪辑的工作集应放在连接机器的块存储上,而很少读取的部分应放在对象存储中。核心概念是:RAID 提供可用性,而不是备份;重建(rebuild)是一项必须实际测试的承诺。DigitalOcean 的块存储价格为每 GiB-月 $0.10,而 R2 为每 GiB-月 $0.015,因此 1 TB 放在块存储上每月约 $102,而放在 R2 上每月约 $15。实验中的五种布局(RAID 5、RAID 6、RAID 10、raidz1、raidz2)都从 50 GiB 原始容量给出相同的 20 GiB 可用空间,因此要按冗余和写入模式来选择,而不是按容量。

Interactive: RAID modes

Switch between RAID 0, 1, 5, 6 and 10 and watch the blocks change. Data blocks are green; parity blocks are orange. Hover a block for its role.

Switch a RAID mode, then hover a block. Drag to orbit, scroll to zoom.

Reading the scene. RAID 0 stripes blocks across disks for speed but keeps no copy; RAID 1 mirrors every block; RAID 5 adds one parity block per stripe and survives one disk failure; RAID 6 adds two and survives two. RAID 10 stripes across mirrored pairs, so it survives one failure in each pair. Green blocks are data and orange blocks are parity. A rebuild reads every surviving disk, so RAID protects availability, not a deleted file — the backup question belongs to snapshots and versioning.

Mode Minimum disks Survives Usable capacity A deleted file survives?
RAID 0 2 no disk 100% no
RAID 1 2 one disk 50% no
RAID 5 3 one disk all but one disk no
RAID 6 4 two disks all but two disks no
RAID 10 4 one per mirror pair 50% no

Figure 3.2 is the static ZFS counterpart: a pool built from a raidz2 vdev, with each block carrying its own checksum and repair reading the remaining disks.

ZFS pool anatomy: a pool named tank2 built from a four-disk raidz2 vdev, with datasets on top, per-block checksums and a spare disk.

A static version of the RAID diagram is available if WebGL is unavailable.

ZFS term What it means in this lab
Pool The named container, tank1/tank2, holding vdevs and datasets
vdev The redundancy unit: a raidz1 of three disks or a raidz2 of four
Dataset A filesystem inside the pool, such as tank2/data, where fio writes
Resilver Rebuilding the vdev after zpool offline and zpool replace
Checksum A per-block hash verified on every read; repair uses the other disks

Prerequisite: The DigitalOcean account, project and token from Session 1; the lab creates its own Block Storage volumes and tears them down. Set it up in Session 0: Accounts.

Lab — Volumes, RAID 5/6/10 and ZFS

Time: 120 minutes — 10 setup, 10 packages and baseline, 50 for the five arrays (10 each), 10 buffer, 20 evidence, 10 teardown.

Prerequisite: the S1 account, project and SSH key; the S2 habit of deleting what you create. The S2 Droplet is already gone, so this lab creates its own.

You need to know: a block volume is a raw disk attached to one Droplet, and it bills until it is deleted. mdadm builds a software RAID array from several volumes; ZFS builds a pool from vdevs. RAID and raidz keep a service online when a disk fails, but they do not restore a deleted file. A rebuild or resilver is the promise under test: the array is not protected until the repair has finished and the state is clean.

Cost ceiling: $0.20 — an s-2vcpu-4gb Droplet for about two hours at $0.036/hour, plus 50 GiB of volumes at $0.10/GiB-month for the same two hours. Detached volumes still bill, so the teardown deletes all five.

Safe-data acknowledgement: I have read the lab safe-data policy. I will use only the course-supplied synthetic media and will not upload real personal media to any cloud service.

Task

Build five 10 GiB Block Storage volumes, benchmark a local-disk and a single-volume baseline, then create and measure mdadm RAID 5, RAID 6 and RAID 10 and ZFS raidz1 and raidz2. Fail or offline one disk in every array, watch the rebuild or resilver finish, compare capacity and throughput, and tear everything down.

What you submit (ten items):

  1. summary.csv and the raw fio JSON files.
  2. The lsblk output and the whole-disk by-id list.
  3. Each mdadm array's --detail and /proc/mdstat in normal, degraded and rebuilt states.
  4. The RAID10 --fail → --remove → --zero-superblock → --add sequence.
  5. Each ZFS pool's zpool status in normal, degraded, resilvering and completed states.
  6. The capacity table and storage_results.json from the notebook.
  7. The sanitized command log.
  8. The empty Droplet, volume and firewall lists plus doctl balance get.
  9. The 150–250-word "what this proves" note.
  10. Your terminal transcript, if you keep one.

Scored 3/2/1/0: 3 = all five arrays built, all seven configurations measured, every repair clean, teardown and cost respected; 2 = one check missing; 1 = partial or no teardown; 0 = nothing, unsafe data, or billed resources left behind.

Path note. This brief is the step-by-step path: type each command and read what it prints. The verified reference run is run.sh + remote.sh + verify.sh in this folder; COMMANDS.md is its sanitized command record. Do not run the script in place of the steps — the observation is the lab.

Reference-run control only. These commands are how run.sh drives the phases; the learner never types them.

scp -i <private-key> remote.sh root@<droplet-ip>:/root/remote.sh
ssh -i <private-key> root@<droplet-ip> "FIO_RUNTIME=10 bash /root/remote.sh base"
ssh -i <private-key> root@<droplet-ip> "FIO_RUNTIME=10 bash /root/remote.sh raid5"
ssh -i <private-key> root@<droplet-ip> "FIO_RUNTIME=10 bash /root/remote.sh raid6"
ssh -i <private-key> root@<droplet-ip> "FIO_RUNTIME=10 bash /root/remote.sh raid10"
ssh -i <private-key> root@<droplet-ip> "FIO_RUNTIME=10 bash /root/remote.sh zfs1"
ssh -i <private-key> root@<droplet-ip> "FIO_RUNTIME=10 bash /root/remote.sh zfs2"
ssh -i <private-key> root@<droplet-ip> "FIO_RUNTIME=10 bash /root/remote.sh summary"

Before you start: credentials and the panel path

0.1 Create a scoped token (panel only)

Panel: API → Tokens → Generate New Token, name it course-s3-<id>, expiry 90 days, and grant the custom scopes:

  • droplet:read, droplet:create, droplet:delete
  • block_storage:read, block_storage:create, block_storage:delete
  • block_storage_action:read
  • tag:read, tag:create
  • firewall:read, firewall:create, firewall:delete
  • ssh_key:read (ssh_key:create is only needed if you let the reference run import its throwaway key)
  • project:read, project:assign_resource
  • image:read, region:read, size:read, action:read
  • account:read and billing:read (for account get and balance get)

Token creation has no CLI path; the panel is the only route. Copy the token once — the panel will not show it again. The scopes are the least privilege this lab needs; block_storage:* is the volume scope name, not volume:*.

0.2 Point doctl at the token

doctl auth init --context course-s3

What it does: starts an interactive prompt that stores the token in a named doctl context called course-s3, so you do not paste the token into every command. The token itself is never written into this repository.

export DIGITALOCEAN_CONTEXT=course-s3

What it does: selects that context for this shell. If you close the terminal, export it again; the token stays in the config file.

0.3 Confirm the account and that nothing is left over

doctl account get

What it does: proves the token works and shows the account's Droplet limit. Look at the status column: it must be active, and the limit must be at least 3.

doctl compute volume list --format ID,Name --no-header

What it does: lists existing volumes. Expected: no output. A volume from an earlier lab that is still present is billing; delete it before you add five more.

doctl compute droplet list --format ID,Name --no-header

What it does: confirms no Droplet is running. Expected: no output.

doctl compute image list --public --format Slug --no-header | grep -qx rockylinux-10-x64

What it does: confirms the Rocky Linux 10 image slug exists in the region catalogue. Expected: no output and exit status 0; a printed error means the slug has changed.

doctl projects list --format ID,Name,IsDefault --no-header

What it does: shows the project UUID that the Droplet will join. Copy the UUID; you will pass it as <project-uuid> below.

Step 1 — SSH key, tags and firewall

1.1 Choose the SSH key

doctl compute ssh-key list --format ID,Name,FingerPrint --no-header

What it does: lists the keys already in the account. Use the S1 key's fingerprint; the private key stays on your laptop.

Matching fingerprints. doctl and the Control Panel show MD5 fingerprints (aa:bb:…), while ssh-keygen prints SHA256 by default (SHA256:…). Match the key by its name in the list, or ask ssh-keygen for the same form if you are comparing by hand:

ssh-keygen -E md5 -lf ~/.ssh/id_ed25519.pub

What it does: prints the MD5 fingerprint of your public key in the form the DigitalOcean list uses, so the comparison is exact.

Reference run only. The unattended runner cannot use your private key, so it generates a throwaway key and imports it:

ssh-keygen -t ed25519 -N "" -C course-<id>-s3 -f .ssh/id_ed25519

What it does: creates a new ed25519 key pair under .ssh/ in this folder with no passphrase, for the runner only. The .ssh/ directory is gitignored and is deleted at teardown.

doctl compute ssh-key list --format FingerPrint --no-header

What it does: checks whether the throwaway key is already in the account, so the runner imports it only once and does not create duplicates on a retry.

doctl compute ssh-key import course-<id>-s3-key --public-key-file .ssh/id_ed25519.pub

What it does: uploads the public half so the Droplet will accept the matching private key. The runner deletes this key at teardown.

doctl compute ssh-key list --format ID,FingerPrint --no-header

What it does: shows the numeric key ID that droplet create needs. Put it in <key-id>.

1.2 Create the tags

Panel: Networking → Tags → Create Tag.

doctl compute tag create course:42-05-0112

What it does: creates the course tag. The firewall below targets Droplets by the week:3 tag, and a firewall cannot reference a tag that does not exist yet.

doctl compute tag create student:<id>

What it does: tags this student's resources so they can be found and deleted together.

doctl compute tag create week:3

What it does: creates the tag the firewall attaches to. Every resource in this lab carries all three tags.

1.3 Create the firewall

Panel: Networking → Firewalls → Create Firewall, name course-<id>-s3-fw, inbound SSH (22) from your IP, outbound all, and apply it to the tag week:3.

MYIP=$(curl -s https://api.ipify.org)

What it does: stores your public IP in a shell variable. The firewall should admit SSH only from this address, not from the whole internet.

doctl compute firewall create --name "course-<id>-s3-fw" \
  --inbound-rules "protocol:tcp,ports:22,address:$MYIP/32" \
  --outbound-rules "protocol:tcp,ports:0,address:0.0.0.0/0 protocol:udp,ports:0,address:0.0.0.0/0 protocol:icmp,ports:0,address:0.0.0.0/0" \
  --tag-names "week:3" --format ID --no-header

What it does: creates a default-deny firewall that admits inbound TCP 22 from your IP only, allows all outbound traffic, and applies to Droplets carrying the week:3 tag. Save the ID it prints for teardown. If you do this in the panel instead, the CLI form beside it produces the same rules.

Step 2 — Five 10 GiB volumes

Panel: Volumes → Create Volume, region SGP1, size 10 GiB, name course-<id>-s3-vol1 … vol5. The panel cannot set tags on volumes; add them with the CLI in this step or from Volumes → the volume → Tags.

doctl compute volume create course-<id>-s3-vol1 --region sgp1 --size 10GiB \
  --desc "course 42-05-0112 S3" --tag "course:42-05-0112,student:<id>,week:3" \
  --format ID --no-header

What it does: creates the first 10 GiB volume in Singapore. It bills from this moment, whether or not it is attached.

doctl compute volume create course-<id>-s3-vol2 --region sgp1 --size 10GiB \
  --desc "course 42-05-0112 S3" --tag "course:42-05-0112,student:<id>,week:3" \
  --format ID --no-header

What it does: creates the second volume with the same region, size and tags.

doctl compute volume create course-<id>-s3-vol3 --region sgp1 --size 10GiB \
  --desc "course 42-05-0112 S3" --tag "course:42-05-0112,student:<id>,week:3" \
  --format ID --no-header

What it does: creates the third volume, the last member S3's raidz1 pool needs.

doctl compute volume create course-<id>-s3-vol4 --region sgp1 --size 10GiB \
  --desc "course 42-05-0112 S3" --tag "course:42-05-0112,student:<id>,week:3" \
  --format ID --no-header

What it does: creates the fourth volume, the spare that replaces a failed member in the RAID layouts.

doctl compute volume create course-<id>-s3-vol5 --region sgp1 --size 10GiB \
  --desc "course 42-05-0112 S3" --tag "course:42-05-0112,student:<id>,week:3" \
  --format ID --no-header

What it does: creates the fifth and final volume. Collect all five IDs; the Droplet step consumes them as one comma-separated list.

doctl compute volume list --format ID,Name,Size --no-header

What it does: confirms five volumes exist and shows their IDs. Check that each is 10 GiB and in sgp1; a volume in another region cannot attach to this Droplet.

doctl compute volume list --format ID,Name --no-header | awk '/course-<id>-s3-vol/ {print $1}' | paste -sd, -

What it does: filters the list client-side and joins the IDs into one comma-separated string for the Droplet create command. volume list has no tag filter, so awk does the matching.

Step 3 — The Droplet with its volumes

Panel: Create Droplet → SGP1, Rocky Linux 10 ×64, Basic / Regular / s-2vcpu-4gb, select your SSH key, add the three tags. Attach the volumes either at creation (Add Block Storage) or afterwards from Volumes → Attach to Droplet; the CLI form below attaches them at creation so they are present on first boot.

doctl compute droplet create course-<id>-s3 --region sgp1 --image rockylinux-10-x64 \
  --size s-2vcpu-4gb --ssh-keys <key-id> \
  --tag-names "course:42-05-0112,student:<id>,week:3" \
  --volumes <vol1-id>,<vol2-id>,<vol3-id>,<vol4-id>,<vol5-id> \
  --project-id <project-uuid> --wait --format ID --no-header

What it does: creates the Droplet and attaches all five volumes in one API call, then waits for it to boot. --wait polls the Droplet until it is active. Drop --project-id if you are not using a project.

doctl compute droplet get <droplet-id> --format PublicIPv4 --no-header

What it does: prints the Droplet's public IP directly. Use this or the list command below; the IP is what SSH and the firewall rule refer to.

doctl compute droplet list --format ID,Name,PublicIPv4 --no-header

What it does: shows the Droplet's public IP; that IP is what the firewall allows SSH from, and what you pass to ssh.

doctl compute droplet list -o json > results/droplet-before.json

What it does: saves the pre-lab resource state as evidence. This is the "before" half of the teardown proof.

doctl compute volume list -o json > results/volume-before.json

What it does: saves the five attached volumes and their Droplet links as evidence.

doctl compute firewall list -o json > results/firewall-before.json

What it does: saves the firewall list before the lab so the teardown proof shows your rule and then its removal.

Step 4 — Connect, discover the devices, install the tools

ssh -i <private-key> root@<droplet-ip>

What it does: opens a root shell on the Droplet. Everything in steps 4–12 runs inside that shell; the doctl commands in steps 8 and 13 run on your laptop.

dnf -y install mdadm fio xfsprogs jq

What it does: installs the RAID tool, the benchmark, the XFS formatter and jq for parsing fio's JSON. Rocky's repositories carry all four.

uname -a

What it does: records the running kernel; the ZFS DKMS build depends on this exact version, so the evidence should show it.

df -h

What it does: records the filesystem layout before any volume is formatted, so the before/after difference is visible.

dnf repolist

What it does: lists the enabled repositories. If the ZFS repository is missing after the release package install, this is where you see it.

mokutil --sb-state

What it does: reports Secure Boot state. DigitalOcean Droplets normally have it off; if it is on, the unsigned ZFS module will not load and the install needs the fallback path.

ls -1 /dev/disk/by-id/scsi-0DO_Volume_* | sort

What it does: lists the devices the five volumes appear as. Use only the names without -part: the whole disk belongs to the array and must not carry a partition table.

lsblk -o NAME,SIZE,TYPE,SERIAL,MOUNTPOINT

What it does: shows size, type and mount point for every block device. Confirm five 10 GiB disks and no mount points on them; save the output as evidence.

4.1 Pre-lab ZFS install

This is the pre-lab step. It takes a few minutes; run it now, before the arrays, so it is not on the critical path. If ZFS is already installed (zfs version works), skip to step 5.

dnf -y install https://zfsonlinux.org/epel/zfs-release-3-0.el10.noarch.rpm

What it does: adds the OpenZFS repository for Enterprise Linux 10. The file name is the EL10 release package, not a kernel module.

dnf -y install epel-release

What it does: adds EPEL, which supplies dependencies the ZFS packages expect.

dnf -y install kernel-devel-$(uname -r)

What it does: installs the headers for the running kernel. DKMS needs them to compile the module; without them the install appears to succeed but modprobe fails.

dnf -y install zfs

What it does: installs OpenZFS and its DKMS module, building the kernel module against the installed headers.

modprobe zfs

What it does: loads the module into the running kernel. Expected: no output. An error here is the install failure to fix before the ZFS steps.

zfs version

What it does: prints the userland and kernel module versions. The verified run reports zfs-2.2.11 and zfs-kmod-2.2.11; record the numbers in your evidence.

dnf -y install mdadm fio xfsprogs jq

What it does: re-checks the four tools after the ZFS install; on a fresh Droplet the earlier step already installed them, so this is fast and idempotent.

Step 5 — Baseline: local disk and one volume

mkdir -p /root/fio-local

What it does: creates the directory the local-disk test writes into.

mkdir -p /root/results

What it does: creates the results directory every test writes its JSON to.

The four fio commands below are the template for every configuration. Run them once against /root/fio-local, then again against each mount point, changing only --directory and the output name.

fio --name=seqread --directory=/root/fio-local --rw=read --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-local-disk-seqread.json

What it does: reads 1 MiB blocks sequentially for 10 seconds and writes the result as JSON. 1 MiB blocks and queue depth 8 approximate streaming video playback.

fio --name=seqwrite --directory=/root/fio-local --rw=write --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-local-disk-seqwrite.json

What it does: writes 1 MiB blocks sequentially for 10 seconds — the ingest pattern. On a network volume this is the write rate an ingest actually gets.

fio --name=randread --directory=/root/fio-local --rw=randread --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-local-disk-randread.json

What it does: issues 4 KiB random reads from four jobs at queue depth 32 for 10 seconds, approximating a database index lookup.

fio --name=randwrite --directory=/root/fio-local --rw=randwrite --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-local-disk-randwrite.json

What it does: issues 4 KiB random writes — the pattern where parity costs the most. Compare this number between RAID 5 and RAID 10.

mkfs.xfs -f /dev/disk/by-id/scsi-0DO_Volume_<vol1>

What it does: puts an XFS filesystem on the first whole volume. -f allows a reformat of the blank device; there is no partition table.

mkdir -p /mnt/base

What it does: creates the mount point for the single-volume baseline.

mount /dev/disk/by-id/scsi-0DO_Volume_<vol1> /mnt/base

What it does: mounts the XFS filesystem so fio can write files through the filesystem, which is the same path the array tests use.

fio --name=seqread --directory=/mnt/base --rw=read --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-single-volume-seqread.json

What it does: repeats the sequential read against the single volume and records it under its own configuration name.

fio --name=seqwrite --directory=/mnt/base --rw=write --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-single-volume-seqwrite.json

What it does: repeats the sequential write against the single volume.

fio --name=randread --directory=/mnt/base --rw=randread --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-single-volume-randread.json

What it does: repeats the random read against the single volume.

fio --name=randwrite --directory=/mnt/base --rw=randwrite --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-single-volume-randwrite.json

What it does: repeats the random write against the single volume.

umount /mnt/base

What it does: unmounts the baseline. An array cannot claim a volume that is still mounted.

Step 6 — mdadm RAID 5: three active disks plus a spare

mdadm --stop /dev/md5
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol1>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol2>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol3>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol4>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol5>

What it does: stops any array named md5 left from an earlier attempt and clears the mdadm metadata from all five volumes. --stop on a non-existent array prints an error and continues; a stale superblock is what makes the next --create refuse a device.

mdadm --create /dev/md5 --level=5 --raid-devices=3 --spare-devices=1 --run \
  /dev/disk/by-id/scsi-0DO_Volume_<vol1> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol2> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol3> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol4>

What it does: creates a RAID 5 array from three active devices (two data plus one parity) with the fourth as a hot spare. Usable capacity is about 20 GiB. --run starts it without waiting for confirmation.

cat /proc/mdstat

What it does: shows the array's live state. During the initial sync you will see [2/3] or [resync]; wait until it reads [3/3] [UUU] before benchmarking.

mdadm --detail /dev/md5

What it does: prints the member disks, their roles, and State : clean. Save this as the normal-state evidence.

mkdir -p /mnt/raid5

What it does: creates the mount point.

mkfs.xfs -f /dev/md5

What it does: puts XFS on the array, not on a member disk. Formatting a member would destroy the array.

mount /dev/md5 /mnt/raid5

What it does: mounts the array so the four fio tests can write through a filesystem.

fio --name=seqread --directory=/mnt/raid5 --rw=read --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid5-seqread.json

What it does: measures streaming read on the array. Compare with the single-volume row: RAID 5 reads can use more than one disk.

fio --name=seqwrite --directory=/mnt/raid5 --rw=write --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid5-seqwrite.json

What it does: measures streaming write; parity is computed per stripe, so this is usually below the read rate.

fio --name=randread --directory=/mnt/raid5 --rw=randread --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid5-randread.json

What it does: measures small random reads, which can be served from multiple disks in parallel.

fio --name=randwrite --directory=/mnt/raid5 --rw=randwrite --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid5-randwrite.json

What it does: measures small random writes; watch for the read-modify-write penalty relative to RAID 10.

Break a disk and watch the spare take over

echo 50000 > /proc/sys/dev/raid/speed_limit_min

What it does: raises the rebuild speed floor. The default is deliberately gentle; for a 20 GiB array in a lab, a faster rebuild finishes inside the session.

echo 200000 > /proc/sys/dev/raid/speed_limit_max

What it does: raises the ceiling above the floor so the kernel can use the available bandwidth.

mdadm --fail /dev/md5 /dev/disk/by-id/scsi-0DO_Volume_<vol1>

What it does: marks the first member as failed. The hot spare should begin rebuilding within seconds; this is the failure the array was built to absorb.

mdadm --detail /dev/md5

What it does: shows the degraded/recovering state: one device failed, the spare active. Save it as evidence.

cat /proc/mdstat

What it does: shows the rebuild progress in the [=>...] bar. Save this state before it disappears.

mdadm --remove /dev/md5 /dev/disk/by-id/scsi-0DO_Volume_<vol1>

What it does: removes the failed member from the array so it can be re-added as the new spare.

mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol1>

What it does: clears the old mdadm superblock so the kernel treats the disk as new. Skipping this is the usual cause of a re-add that is refused.

mdadm --add /dev/md5 /dev/disk/by-id/scsi-0DO_Volume_<vol1>

What it does: adds the repaired disk back as the new spare. The array now has three active members and one spare again.

mdadm --detail /dev/md5

What it does: confirms the array returned to [3/3] [UUU] and State : clean — the rebuild promise, tested.

cat /proc/mdstat

What it does: confirms no resync or recovery is still running. Only now is the array protected again.

Step 7 — mdadm RAID 6: four active disks plus a spare

umount /mnt/raid5

What it does: unmounts the RAID 5 array.

mdadm --stop /dev/md5

What it does: stops the array and releases the member devices for reuse.

mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol1>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol2>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol3>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol4>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol5>

What it does: clears each member's superblock before the next array is created. A leftover superblock can make --create refuse a device or, worse, assemble the old array.

mdadm --create /dev/md6 --level=6 --raid-devices=4 --spare-devices=1 --run \
  /dev/disk/by-id/scsi-0DO_Volume_<vol1> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol2> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol3> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol4> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol5>

What it does: creates RAID 6 from four active devices (two data plus two parity) and the fifth as a spare. It survives two failures and still yields about 20 GiB usable.

cat /proc/mdstat

What it does: shows the initial sync. Wait for [4/4] [UUUU] before measuring.

mdadm --detail /dev/md6

What it does: confirms four active members, one spare and State : clean.

mkdir -p /mnt/raid6

What it does: creates the mount point.

mkfs.xfs -f /dev/md6

What it does: formats the array with XFS.

mount /dev/md6 /mnt/raid6

What it does: mounts it for the four tests.

fio --name=seqread --directory=/mnt/raid6 --rw=read --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid6-seqread.json

What it does: measures sequential read on RAID 6.

fio --name=seqwrite --directory=/mnt/raid6 --rw=write --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid6-seqwrite.json

What it does: measures sequential write; two parity blocks cost more than RAID 5's one.

fio --name=randread --directory=/mnt/raid6 --rw=randread --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid6-randread.json

What it does: measures random reads on RAID 6.

fio --name=randwrite --directory=/mnt/raid6 --rw=randwrite --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid6-randwrite.json

What it does: measures random writes, where the dual-parity read-modify-write shows.

echo 50000 > /proc/sys/dev/raid/speed_limit_min

What it does: raises the rebuild speed floor for this array.

echo 200000 > /proc/sys/dev/raid/speed_limit_max

What it does: raises the ceiling.

mdadm --fail /dev/md6 /dev/disk/by-id/scsi-0DO_Volume_<vol1>

What it does: fails the first member; the spare starts rebuilding.

mdadm --detail /dev/md6

What it does: records the degraded/recovering state.

cat /proc/mdstat

What it does: records rebuild progress.

mdadm --remove /dev/md6 /dev/disk/by-id/scsi-0DO_Volume_<vol1>

What it does: removes the failed member.

mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol1>

What it does: clears its superblock.

mdadm --add /dev/md6 /dev/disk/by-id/scsi-0DO_Volume_<vol1>

What it does: re-adds it as the new spare.

mdadm --detail /dev/md6

What it does: confirms [4/4] [UUUU] and clean state after the rebuild.

cat /proc/mdstat

What it does: confirms the rebuild finished.

Step 8 — mdadm RAID 10: four active disks, fifth added on failure

umount /mnt/raid6

What it does: unmounts RAID 6.

mdadm --stop /dev/md6

What it does: stops the array.

mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol1>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol2>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol3>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol4>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol5>

What it does: clears the superblock on each of the five members before the RAID 10 array is created.

mdadm --create /dev/md10 --level=10 --raid-devices=4 --run \
  /dev/disk/by-id/scsi-0DO_Volume_<vol1> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol2> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol3> \
  /dev/disk/by-id/scsi-0DO_Volume_<vol4>

What it does: creates a RAID 10 array from four devices — two mirrored pairs, striped. There is no spare: usable capacity is 20 GiB and one failure per pair is survivable.

cat /proc/mdstat

What it does: shows the initial sync; wait for [4/4] [UUUU].

mdadm --detail /dev/md10

What it does: confirms the layout and clean state.

mkdir -p /mnt/raid10

What it does: creates the mount point.

mkfs.xfs -f /dev/md10

What it does: formats the array.

mount /dev/md10 /mnt/raid10

What it does: mounts it for the four tests.

fio --name=seqread --directory=/mnt/raid10 --rw=read --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid10-seqread.json

What it does: measures sequential read on the mirrored stripe.

fio --name=seqwrite --directory=/mnt/raid10 --rw=write --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid10-seqwrite.json

What it does: measures sequential write; every write goes to two disks, so the raw rate is half before striping.

fio --name=randread --directory=/mnt/raid10 --rw=randread --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid10-randread.json

What it does: measures random reads; either copy of a block can serve them.

fio --name=randwrite --directory=/mnt/raid10 --rw=randwrite --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-raid10-randwrite.json

What it does: measures random writes, the place RAID 10 is expected to beat RAID 5/6.

echo 50000 > /proc/sys/dev/raid/speed_limit_min

What it does: raises the rebuild speed floor.

echo 200000 > /proc/sys/dev/raid/speed_limit_max

What it does: raises the ceiling.

mdadm --fail /dev/md10 /dev/disk/by-id/scsi-0DO_Volume_<vol1>

What it does: fails one member of the first mirror pair. There is no spare, so the array stays degraded until you add the fifth volume by hand.

mdadm --detail /dev/md10

What it does: records the degraded state — this is the window a hot spare would have shortened.

cat /proc/mdstat

What it does: shows [4/3] [UU_U] or similar: still serving, one member short.

mdadm --remove /dev/md10 /dev/disk/by-id/scsi-0DO_Volume_<vol1>

What it does: removes the failed member.

mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol1>

What it does: clears the failed member's superblock.

mdadm --add /dev/md10 /dev/disk/by-id/scsi-0DO_Volume_<vol5>

What it does: adds the fifth volume as the replacement and begins the rebuild. Note the disk changing: RAID 10 needs a new disk, not the failed one back.

mdadm --detail /dev/md10

What it does: confirms the array returned to [4/4] [UUUU] and clean state.

cat /proc/mdstat

What it does: confirms the rebuild finished.

Step 9 — ZFS raidz1: three disks, offline and replace

umount /mnt/raid10

What it does: unmounts the RAID 10 array.

mdadm --stop /dev/md10

What it does: stops it and releases the members.

mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol1>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol2>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol3>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol4>
mdadm --zero-superblock /dev/disk/by-id/scsi-0DO_Volume_<vol5>

What it does: clears the mdadm metadata on all five volumes; the ZFS pool must not see a foreign RAID signature.

wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol1>
wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol2>
wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol3>
wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol4>
wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol5>

What it does: wipes any remaining filesystem or RAID signature from each volume; ZFS refuses a device with a foreign signature.

zpool create -f tank1 raidz1 /dev/disk/by-id/scsi-0DO_Volume_<vol1> /dev/disk/by-id/scsi-0DO_Volume_<vol2> /dev/disk/by-id/scsi-0DO_Volume_<vol3>

What it does: creates pool tank1 with one three-disk raidz1 vdev. It survives one disk failure and yields about 20 GiB usable.

zfs create tank1/data

What it does: creates a dataset inside the pool; fio writes into the dataset, which is how ZFS is normally used.

fio --name=seqread --directory=/tank1/data --rw=read --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-zfs-raidz1-seqread.json

What it does: measures sequential read through ZFS. Checksums and the ARC are in this path, unlike the raw mdadm path.

fio --name=seqwrite --directory=/tank1/data --rw=write --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-zfs-raidz1-seqwrite.json

What it does: measures sequential write; copy-on-write allocates new blocks instead of overwriting.

fio --name=randread --directory=/tank1/data --rw=randread --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-zfs-raidz1-randread.json

What it does: measures random reads; the ARC and the checksummed block layout are visible here.

fio --name=randwrite --directory=/tank1/data --rw=randwrite --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-zfs-raidz1-randwrite.json

What it does: measures random writes; expect this to be ZFS's weakest number because every small write allocates and checksums a new block.

zpool status tank1

What it does: shows the pool ONLINE and the vdev with three devices. Save it as the normal-state evidence.

zpool offline tank1 /dev/disk/by-id/scsi-0DO_Volume_<vol1>

What it does: takes one disk out of the vdev without failing the pool. This is the controlled failure — the pool is now DEGRADED and still serves reads and writes.

zpool status tank1

What it does: shows the degraded state and the offline device. Save it.

zpool replace tank1 /dev/disk/by-id/scsi-0DO_Volume_<vol1> /dev/disk/by-id/scsi-0DO_Volume_<vol4>

What it does: replaces the offline disk with the fourth volume and starts the resilver. Note that ZFS rebuilds the allocated blocks, not the whole disk.

zpool status tank1

What it does: shows the resilver in progress and how much data is being scanned. Save it as the resilvering evidence.

zpool status tank1

What it does: wait until the same command shows resilvered and both vdev members ONLINE. This is the promise tested; save the completed state.

Step 10 — ZFS raidz2: four disks, survive two

zpool destroy -f tank2

What it does: destroys any tank2 left by an earlier attempt, so the next create starts from a clean name.

zpool destroy -f tank1

What it does: destroys the first pool. Double-check the pool name: there is no undo for a destroy.

wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol1>
wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol2>
wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol3>
wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol4>
wipefs -a /dev/disk/by-id/scsi-0DO_Volume_<vol5>

What it does: wipes any leftover label from each volume before the next pool is created.

zpool create -f tank2 raidz2 /dev/disk/by-id/scsi-0DO_Volume_<vol1> /dev/disk/by-id/scsi-0DO_Volume_<vol2> /dev/disk/by-id/scsi-0DO_Volume_<vol3> /dev/disk/by-id/scsi-0DO_Volume_<vol4>

What it does: creates pool tank2 with one four-disk raidz2 vdev — two parity blocks, two failures survivable, about 20 GiB usable.

zfs create tank2/data

What it does: creates the dataset for the tests.

fio --name=seqread --directory=/tank2/data --rw=read --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-zfs-raidz2-seqread.json

What it does: measures sequential read on raidz2.

fio --name=seqwrite --directory=/tank2/data --rw=write --bs=1M --size=256M \
  --numjobs=1 --iodepth=8 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-zfs-raidz2-seqwrite.json

What it does: measures sequential write; two parity blocks cost write bandwidth.

fio --name=randread --directory=/tank2/data --rw=randread --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-zfs-raidz2-randread.json

What it does: measures random reads on raidz2.

fio --name=randwrite --directory=/tank2/data --rw=randwrite --bs=4k --size=256M \
  --numjobs=4 --iodepth=32 --runtime=10 --time_based --group_reporting \
  --output-format=json --output=/root/results/fio-zfs-raidz2-randwrite.json

What it does: measures random writes; compare the cost with raidz1's single parity.

zpool status tank2

What it does: shows the pool online with four devices. Save it.

zpool offline tank2 /dev/disk/by-id/scsi-0DO_Volume_<vol1>

What it does: takes one disk offline; raidz2 remains DEGRADED but still has one parity block to spare.

zpool status tank2

What it does: records the degraded state.

zpool replace tank2 /dev/disk/by-id/scsi-0DO_Volume_<vol1> /dev/disk/by-id/scsi-0DO_Volume_<vol5>

What it does: replaces it with the fifth volume and starts the resilver.

zpool status tank2

What it does: records the resilver progress.

zpool status tank2

What it does: wait for resilvered and all four devices ONLINE; save the completed state.

Step 11 — Collect the summary and leave the Droplet

cat /root/results/summary.csv

What it does: prints the 28 measurements — seven configurations × four tests — that the notebook pivots. If a row is missing, run the missing fio command again.

lsblk -o NAME,SIZE,TYPE,SERIAL,MOUNTPOINT

What it does: captures the final device state for the evidence bundle.

exit

What it does: closes the SSH session and returns you to the laptop.

mkdir -p results

What it does: creates a local folder for the raw evidence on the laptop.

scp -i <private-key> -r root@<droplet-ip>:/root/results/. results/

What it does: copies every fio JSON, mdadm/ZFS status file and the command logs from the Droplet. This must happen before teardown; deleting the Droplet deletes the evidence.

bash verify.sh

What it does: checks the collected evidence: 28 summary rows, 28 fio JSON files, the rebuild and resilver states, the empty teardown lists and the balance. It exits non-zero if anything is missing. (OUT=results is the default; set OUT=results if you copied elsewhere.)

Step 12 — Teardown

ssh -i <private-key> root@<droplet-ip> "bash /root/remote.sh cleanup"

What it does: unmounts every array and dataset, exports the pools and stops mdadm so the volumes are released cleanly. remote.sh cleanup runs the three commands below; run them by hand if you did not copy the script.

umount /mnt/raid5 /mnt/raid6 /mnt/raid10 /mnt/base /tank1/data /tank2/data

What it does: releases every mount point in one call, including the two ZFS datasets. An unmount that reports "not mounted" is harmless when an earlier step already released it.

zpool export -a

What it does: exports every imported pool so ZFS releases the member volumes; a volume held by an imported pool cannot be deleted cleanly.

mdadm --stop --scan

What it does: stops every running md array so the kernel releases its member volumes.

doctl compute droplet delete <droplet-id> --force

What it does: deletes the Droplet. The volumes survive this and keep billing until step 12.4.

doctl compute droplet get <droplet-id>

What it does: confirms the Droplet is gone; expected: a "not found" error. A successful response means the delete has not completed, and deleting volumes now may fail while they are still attached.

doctl compute volume delete <vol1-id> --force
doctl compute volume delete <vol2-id> --force
doctl compute volume delete <vol3-id> --force
doctl compute volume delete <vol4-id> --force
doctl compute volume delete <vol5-id> --force

What it does: deletes all five volumes. A volume that is still attached may refuse once while the Droplet releases; wait a few seconds and retry. Delete, do not detach — a detached volume still bills.

doctl compute firewall delete <firewall-id> --force

What it does: deletes the week:3 firewall.

doctl compute ssh-key delete <key-id> --force

What it does: deletes the throwaway key the reference run imported. Skip this if you used your own S1 key: that key stays for the rest of the course.

doctl compute droplet list -o json > results/droplet-after.json

What it does: records the empty Droplet list as teardown proof.

doctl compute volume list -o json > results/volume-after.json

What it does: records the empty volume list. This is the proof that nothing billable is left.

doctl compute firewall list -o json > results/firewall-after.json

What it does: records the firewall list without your rule.

doctl balance get

What it does: shows the account balance and month-to-date usage. Record it; the run must stay under $0.20.

Evidence bundle

Submit the ten items listed under Task, plus the notebook's storage_results.json. The 150–250-word note must answer: which layout would you put the weekly working set on, which would you not, and what did the rebuild or resilver time tell you that the capacity table did not?

Fallback chain

Failure Fallback (same evidence shape)
The token lacks volume scopes Create the token again with block_storage:*; custom scopes cannot be edited after creation. Or use the panel for volumes and keep the CLI for the rest.
doctl blocked or not installed Panel path for Droplets, volumes and firewalls; volume tags are CLI-only, so record that as a deviation.
Five volumes refused Create three 10 GiB volumes and run raidz1 only; label the missing configurations in the evidence.
ZFS install fails Reinstall kernel-devel-$(uname -r), run modprobe zfs, and retry dnf -y install zfs. If it still fails, complete the three mdadm arrays and use the published ZFS numbers in data/ for the ZFS rows, labelled as a fallback.
RAID 5/6 has no spare capacity on the account Run the array without a spare and add a volume at failure; record that the degraded window was longer.
No account at all Instructor demo account with a booked slot; the evidence stays individual.

Teardown checklist

  • Droplet deleted.
  • All five volumes deleted (not detached).
  • week:3 firewall deleted.
  • The imported throwaway SSH key deleted (reference run only).
  • results/droplet-after.json, results/volume-after.json and results/firewall-after.json empty.
  • doctl balance get recorded.
  • The project, your S1 key and the spend alert stay; the scoped token may stay until it expires or be deleted deliberately.

Downloads for this session

Each document name opens a rendered view you can read in the browser. The download link beside it gives you the raw Markdown working file: the copy you edit, run and submit. The page above and the lab steps carry the teaching.

lab.md download lab.md
The numbered lab brief: the steps, the commands and the evidence you submit.
refs.md download refs.md
The reference list behind this session's claims.
PLO.md download PLO.md
This session's scoring rows and descriptors.
download notebook.ipynb
The runnable notebook that produces the lab's numbers.

Research question

Which mix of block and object storage, and which tiering and snapshot policy, keeps the 1 TB video archive and 200 GB asset library both cheap and restorable?

Submit an 800–1,200 word answer with 5–8 sources at the next session. The rubric and the exemplar guidance show the expected shape.

References — Session 3: Storage, RAID and ZFS

  1. Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003). The Google file system. Proceedings of SOSP '03, 29–43. https://doi.org/10.1145/945445.945450
  2. Chunked, replicated storage designed for large sequential files — the shape the 1 TB video archive takes, and the reason object storage, not a RAID array, carries the long tail. The open problem is that GFS assumes few large files and append-heavy access; a media catalogue with 6,000 small records and collaborative review changes that assumption. Redraw the architecture diagram; no third-party figure is reused.
  3. Rodeh, O., Bacik, J., & Mason, C. (2013). BTRFS: The Linux B-tree filesystem. ACM Transactions on Storage, 9(3), 1–32. https://doi.org/10.1145/2501620.2501623
  4. Makes checksums, copy-on-write and snapshots concrete — the mechanisms the ZFS half of the lab demonstrates and the reason a resilver walks allocated blocks rather than raw capacity. Snapshot cost and restore time for very large video files are rarely quantified. The B-tree figures can be rebuilt as a stacked slab.
  5. DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., & Vogels, W. (2007). Dynamo: Amazon's highly available key-value store. Proceedings of SOSP '07, 205–220. https://doi.org/10.1145/1294261.1294281
  6. Frames availability as a design target rather than a property you buy — the frame the lab applies when it treats a finished rebuild, not a benchmark, as the availability promise. Durable, consistent review of the media catalogue during a repair remains open. The ring diagram is a redraw candidate.

Teaching note. Narrate Ghemawat et al. (2003) for 5–8 minutes and connect it to the capacity table: the archive is a sequence of large files and belongs in object storage, while the working set is a filesystem on block storage. Assign Rodeh et al. (2013)'s abstract before the ZFS steps, then restate the open problem as the weekly question: which mix of block and object storage, and which tiering and snapshot policy, keeps the archive both cheap and restorable? The references are third-party; redraw by default and publish no figure without a clear-rights sidecar.

PLO — Session 3: Storage, RAID and ZFS

CLO: By the end of this session, the student can choose a block layout for a media workload, measure its throughput and capacity, and prove its failure path.

PLO Role Research answer Lab evidence Points Evidence Rubric
PLO2 primary 4 3 7 An 800–1,200-word block-and-object tiering policy with 5–8 sources; the mdadm and ZFS layouts, the fio results, the capacity table and the rebuild or resilver log Research 4/3/2/1; Lab 3/2/1/0
PLO1 secondary 0 0 0 Lecture context only; not scored —
Total 4 3 7

Total: 7 points

Four-level descriptors

Research (4/3/2/1). 4 = a costed policy that compares block and object storage, sets a restore objective and tier thresholds, and weighs the counterargument about retrieval cost. 3 = a sensible policy with some costing and a restore test. 2 = describes the tiers but does not choose a policy or price it. 1 = a summary with fewer than five sources or no decision. 0 = nothing submitted or misconduct.

Lab (3/2/1/0). 3 = the mdadm RAID5/6/10 and ZFS raidz1/2 arrays are built, the fio results and capacity table are complete, a rebuild and a resilver finish clean, teardown and cost respected. 2 = mostly complete, one check missing. 1 = partial restore or incomplete teardown. 0 = nothing submitted, unsafe data, or no teardown.

The full course arithmetic is in production/baseline.md; this file is the weekly instance.