References — Session 9
- Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2106.09685
- Makes fine-tuning a small model affordable by training low-rank adapters instead of all weights — the method the lab uses. Adapter capacity and rank selection for multimodal media tasks are still empirical. The LoRA adapter diagram needs a redraw or permission.
- Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient finetuning of quantized LLMs. Advances in Neural Information Processing Systems 36. https://arxiv.org/abs/2305.14314
- Shows how 4-bit quantisation plus LoRA fits a useful model into a free T4, which is the lab's constraint. The quality cost of 4-bit fine-tuning on small domain datasets is task-dependent and under-reported. The memory diagram is a redraw candidate.
- Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems 33. https://arxiv.org/abs/2005.11401
- The alternative to fine-tuning: keep the model frozen and supply documents, which changes cost, freshness and governance. RAG versus fine-tuning for media-domain knowledge has no settled cost model. The RAG pipeline can be rebuilt as a node-graph scene.
- Kaplan, J., McCandlish, S., Henighan, T., et al. (2020). Scaling laws for neural language models. arXiv:2001.08361. https://arxiv.org/abs/2001.08361
- Explains why model size, data and compute trade against each other — the numbers behind the week's memory-bandwidth discussion. Scaling laws for small, domain-specific models and inference-optimised architectures are less clear. The loss curves can be redrawn from the reported data.
Teaching note. Narrate Hu et al. (2022) for 5–8 minutes, assign the abstract of Dettmers et al. (2023) as reading, and close by restating the open problem as the weekly question. Figures are third-party; redraw by default.