Rendered view of lab.md. Read it as a page; the download link gives you the raw Markdown working file.

Download lab.md

Lab — QLoRA fine-tune on the caption catalogue

Time: 120 minutes — 15 setup, 60 fine-tune, 25 evaluate, 10 evidence, 10 teardown. On the Mac the full run is about 15 minutes; on a free Kaggle T4 the measured full run is about 18 minutes end to end (14.3 min training + ~2 min load/eval) at 5.78 GB peak VRAM; on the faculty RTX Pro 6000 Blackwell it is about 4.5 minutes (3.7 min training + ~40 s load/eval) at 5.76 GB. Start the smoke run first so the full run is not the first time you meet the runtime.

Prerequisite: S8; Python 3.14; basic ML vocabulary. The data is bundled at data/caption_catalogue.jsonl — 1,000 synthetic rows, 800 train / 200 validation, course-owned, seed 42. The provenance, licence and fetch instructions are in data/README.md; there is no runtime download.

You need to know: at batch 1, inference is memory-bound; precision divides the weights' memory bill; LoRA freezes the base and trains a small adapter; QLoRA adds a 4-bit base and memory-saving optimizer tricks.

Safe-data acknowledgement: I have read the lab safe-data policy. I will use only the course-supplied synthetic media and will not upload real personal media to any cloud service.

Task

Fine-tune a 4-bit QLoRA adapter on the synthetic caption catalogue, compare the base and tuned captions with token-F1 on held-out rows, and explain when this beats prompting or retrieval.

Choose your runtime

Runtime Notebook Time Cost
This Apple Silicon Mac notebook.ipynb (mlx-vlm) about 15 min $0
Kaggle T4 (free) notebook-cuda.ipynb (transformers, peft, bitsandbytes) ~18 min (measured: 855.4 s train, 5.78 GB VRAM) $0 (30 GPU-hours/week)
Faculty RTX Pro 6000 notebook-cuda.ipynb ~4.5 min (measured: 219.8 s train, 5.76 GB VRAM) $0 (faculty card)
RunPod RTX Pro 6000 (fallback) notebook-cuda.ipynb ~4.5 min + setup about $1.69/hr, under $3.40

Steps

1. Set up and run the smoke cell

Mac. Open notebook.ipynb. The install cell (the single documented pip install exception) checks imports first: in the course environment the MLX stack is already there, so it prints already installed — nothing to do and installs nothing. If you built the environment by hand it installs mlx==0.32.4, mlx-metal==0.32.4 and mlx-vlm[train]==0.7.12 once; restart the kernel after that run. For the smoke run, start the kernel with the two documented overrides — export them in the terminal you launch JupyterLab from, or edit the S9_ITERS and S9_EVAL_ROWS defaults in the configuration cell:

export S9_ITERS=2 S9_EVAL_ROWS=2   # smoke run; unset both (or use 100 and 10) for the full run

Then run the cells from the import cell down.

Kaggle. Create a private notebook and attach the catalogue as a private dataset. The notebook searches /kaggle/input for caption_catalogue.jsonl, so the dataset name and Kaggle's mount layout do not matter; to be explicit, set COURSE_DATA to the folder that contains the file. Set Accelerator → GPU T4 ×2 (the notebook puts the model on one 16 GB GPU). Open notebook-cuda.ipynb, turn Internet on for the install cell, run it, restart the kernel, and set ITERS=10, EVAL_ROWS=5 for the smoke run.

Faculty RTX. Use the same CUDA notebook; the training stack may already be installed. Confirm the GPU with nvidia-smi before starting.

Console/UI checks:

# Kaggle: Settings → Accelerator shows GPU T4 x2; Internet is ON for the install cell.
# Faculty RTX / RunPod:
nvidia-smi
python3 -c "import torch; print(torch.cuda.get_device_name(0))"

Record the GPU name, the smoke time and the peak memory. If the smoke run's loss does not fall, stop and diagnose before the full run.

2. Run the full fine-tune

Restore the full settings (ITERS=100, EVAL_ROWS=20 on CUDA; 100 and 10 on MLX) and run from the data cell down. The notebook prints the loss every ten steps and writes the result JSON.

  • Mac: results/mlx_results.json.
  • CUDA: results/cuda_results.json.

Record the training time, evaluation time and peak memory. The expected MLX peak is about 4–5 GB; the measured CUDA peak is 5.8 GB on both the T4 and the RTX card, well under the T4's 15.6 GB. The measured 100-step T4 run takes 855 s; if your session is slower, set ITERS=50. If it runs out of memory, keep MAX_SEQ_LEN=512 and batch 1, and rely on gradient accumulation.

3. Evaluate and read the samples

The notebook computes token-F1 for the base and the tuned model on the same held-out rows. Build the base-vs-tuned table:

Model Token-F1 Seconds Peak memory
Base
Tuned

Read three sample captions from the JSON. Say what the base got wrong and what the tuned model changed. Token-F1 is a teaching metric; it does not judge style, and it can miss a factual error.

4. Explain the choice

Write 150–250 words: when would this fine-tune beat retrieval over a larger frozen model, and when would retrieval win? Use your numbers — training time, adapter size, inference speed, update cost — and name one thing the unit would need to redo when the product list changes. This is the lab half of the research question.

5. Save the evidence and tear down

Copy the result JSON and the three samples out of the notebook before the session ends. On Kaggle, stop the session; the output is not a durable store. On RunPod, terminate the pod and its network volume; the container disk clears on stop but a network volume persists.

Evidence bundle

  • The result JSON (mlx_results.json or cuda_results.json) with runtime, model, time, peak memory and base/tuned F1.
  • The base-vs-tuned table and three sample captions.
  • A three-sentence what this proves note: what the base got wrong, what the tuned model changed, and why LoRA is cheaper than full training.
  • A 150–250 word fine-tune-versus-retrieval argument with your numbers.
  • A teardown log: session stopped or pod and volume terminated, adapter and outputs copied or deleted, GPU confirmed idle.

The bundle is scored 3/2/1/0.

Failure and recovery

The deliberate failure is the memory wall: on a 16 GB T4, raise MAX_SEQ_LEN to 1024 (or batch_size above 1) and watch it run out of memory. Recover by returning to MAX_SEQ_LEN=512, batch 1, gradient accumulation and ITERS=100 (or 50 if time is short). On the Mac, the same experiment with a larger sequence or batch shows the unified-memory limit; keep grad_checkpoint=True and the 4-bit model. The lesson is that the memory budget, not the parameter count alone, sets the feasible configuration.

If the loss stalls or the model diverges: confirm the chat template strips thinking, check that train_on_completions is on, lower the learning rate, and compare a base sample to a tuned sample rather than trusting one scalar.

Cost ceiling

$0 on this Mac, the free Kaggle T4 and the faculty RTX Pro 6000. A RunPod fallback stays under $3.40 at $1.69/hr for two hours; set S9_HOURLY_COST in the CUDA notebook to record the line. There is no persistent storage in the lab beyond the result JSON.

Teardown

  • Mac: delete the adapter and the results/ folder after copying the JSON; confirm no process is using the GPU.
  • Kaggle: stop the session; download the result JSON and samples first.
  • Faculty RTX: stop the kernel; remove the adapter and outputs from the shared workspace.
  • RunPod: terminate the pod and the network volume; confirm the billing page shows no running instance.

Fallback

If the runtime cannot complete the fine-tune, use the precomputed prototype results in prototypes/ai-inference-finetuning/results/ (t4_smoke.json, t4_full.json, rtx_pro_6000_*.json) and run the evaluation on the base model only; the evidence shape is unchanged but the note must say which row is measured and which is cited. If the 4B model does not fit, use Qwen3.5-2B and record the change. The T4 smoke run is the gate and has been verified; if a student's session fails, use the Mac MLX path or the faculty RTX card.

Notebooks