LoRA fine-tuning Llama 3.1 8B on a single RTX 5090 in an afternoon

Fine-tuning used to need a node with eight cards. For most practical adaptations of an 8B model, a single RTX 5090 with 32 GB of VRAM is enough, and the whole job fits in an afternoon. This walkthrough takes Llama 3.1 8B Instruct, a 10,000-example instruction dataset, and a QLoRA configuration that trains in about 40 minutes per epoch, then shows how to evaluate the adapter before you merge it and serve it.

What fits in 32 GB

Full fine-tuning of 8B parameters in BF16 needs weights, gradients and optimiser states: roughly 16 + 16 + 64 GB. It does not fit. LoRA trains small adapter matrices instead of the weights, and QLoRA additionally keeps the frozen base in 4-bit. The memory picture on the 5090 then looks like this:

That leaves headroom to raise the batch size or context to 8k. Plain LoRA in BF16 (no quantisation) also fits, at around 26 GB, and trains about 30% faster; use it if your data is under 4k tokens per example.

The environment

Start from the LLaMA-Factory or Unsloth template, or build it yourself on the Ubuntu 24.04 CUDA 12.8 base:

uv venv /opt/ft && source /opt/ft/bin/activate
uv pip install torch==2.5.1 --index-url https://download.pytorch.org/whl/cu124
uv pip install "transformers==4.46.*" "peft==0.13.*" "trl==0.12.*" bitsandbytes datasets accelerate
huggingface-cli download meta-llama/Llama-3.1-8B-Instruct --local-dir /data/models/llama-8b

Data: the part that decides the outcome

Most failed fine-tunes are data failures. Format every example as the chat template the model already knows, with a system message if you use one in production, and keep the answer style consistent: the model learns the format as much as the content.

{"messages": [
  {"role": "system", "content": "You are a support assistant for Acme Cloud."},
  {"role": "user", "content": "How do I rotate my API key?"},
  {"role": "assistant", "content": "Open Account → API keys, click Rotate next to the key…"}
]}

Ten thousand examples of this shape is plenty for style and domain adaptation. Hold out 500 for evaluation before training and never look at them while tuning hyperparameters. Deduplicate near-identical prompts; duplicates make the loss curve look great and the model worse.

The training script

from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
import torch

base = "/data/models/llama-8b"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(
    base, torch_dtype=torch.bfloat16,
    quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                                           bnb_4bit_compute_dtype=torch.bfloat16))
ds = load_dataset("json", data_files={"train": "train.jsonl", "eval": "eval.jsonl"})
cfg = SFTConfig(output_dir="/data/runs/support-v1", num_train_epochs=2,
    per_device_train_batch_size=4, gradient_accumulation_steps=4, learning_rate=2e-4,
    lr_scheduler_type="cosine", warmup_ratio=0.03, bf16=True, gradient_checkpointing=True,
    max_seq_length=4096, logging_steps=10, eval_strategy="steps", eval_steps=100, save_steps=100)
lora = LoraConfig(r=32, lora_alpha=64, lora_dropout=0.05, task_type="CAUSAL_LM",
    target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"])
SFTTrainer(model=model, args=cfg, peft_config=lora, train_dataset=ds["train"],
           eval_dataset=ds["eval"], tokenizer=tok).train()

The choices worth explaining: rank 32 on all linear layers is the sweet spot for 8B in our tests, learning rate 2e-4 is standard for LoRA and would be far too high for full fine-tuning, and an effective batch of 16 with cosine decay over two epochs is where the eval loss usually bottoms out for datasets this size.

How long it takes

Watch nvidia-smi during the first minutes: SM utilisation should sit above 90%. If it does not, the dataloader is the bottleneck; pre-tokenise the dataset or raise the number of workers.

Evaluate before you merge

Eval loss going down proves the model memorised your format, not that it is better. Run the held-out prompts through the base model and the adapter, and compare the answers, ideally with a rubric and a second model as judge, or with a human on a sample of fifty. Three failure modes show up here and nowhere else:

• Over-fitting to phrasing: every answer starts the same way. Lower the epochs to one or reduce rank.
• Forgetting: the model is worse at general questions it used to handle. Mix 10 to 20% of general instruction data into the training set.
• Refusals gone: if your data never shows the model declining anything, it stops declining. Include examples of the behaviour you want kept.

Merge and serve

from peft import PeftModel
m = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16)
m = PeftModel.from_pretrained(m, "/data/runs/support-v1/checkpoint-1250").merge_and_unload()
m.save_pretrained("/data/models/support-v1"); tok.save_pretrained("/data/models/support-v1")

The merged model serves with vLLM exactly like the base model; the vLLM guide applies unchanged, and an 8B model in BF16 on the same 5090 serves around 90 tokens per second per request. Keep the adapter checkpoint too: it is 300 MB and lets you re-merge onto a newer base later.

What it costs

A full afternoon on an RTX 5090 plan, including downloads and three training runs while you tune, is a few dollars of the monthly price. The same job on hourly cloud GPUs is comparable for one run and much more for the iteration that real fine-tuning takes; a monthly card you can leave models on is what makes the second and third attempts cheap.