I Spent $47.23 Fine-Tuning Llama 3 (Here’s Every Mistake I Made)

I broke Llama 3 fine-tuning 17 times, so you don’t have to. Real costs, actual mistakes, and the settings that survived 12 hours on Colab’s free tier.

I spent exactly $47.23 fine-tuning Llama 3 last month. That’s including the failed runs, the accidental spot instance that kept running overnight, and the three energy drinks I demolished at 3 AM, debugging a tokenizer error. Most tutorials you’ll find online skip straight to the happy path, showing you pristine code that works on the first try with mysteriously perfect datasets. That’s not how any of this actually works.

My problem with the existing Llama 3 fine-tuning content? It’s either written by researchers with unlimited A100 access or rehashed documentation that glosses over the parts where things break. What you’re reading is different. I’m documenting the complete 48-hour journey I took to fine-tune a model for a legal document summarization task, including every GPU-poor workaround, budget constraint, and 2 AM panic attack.

By the end of this guide, you’ll understand how to train Llama 3 on private company data without burning through your savings, why the “minimum dataset size” numbers floating around are mostly nonsense, and exactly which settings survived 12 hours of training on Google Colab’s free tier. I broke this seventeen times, so you don’t have to.

Section 1: Dataset Preparation That Actually Works (The 500-Example Minimum Myth)

Let’s kill a misconception. You’ve probably read that you need 500 to 1,000 examples minimum for fine-tuning. Sound familiar? That number gets thrown around constantly, but OpenAI’s actual documentation says you can start with as few as 10 examples, recommending 50 to 100 well-crafted examples as a good starting point. People keep parroting inflated numbers without context.

For Llama 3 specifically, particularly for domain adaptation tasks, I’ve seen meaningful improvements with as few as 150 high-quality examples. High-quality is doing a lot of heavy lifting in that sentence. My dataset preparation actually looked like this in practice:

Quality beats quantity every time. My legal summarization dataset started with 2,000 scraped examples. After cleaning, I was down to 347 that were actually worth using. The others had formatting inconsistencies, ambiguous instructions, or outputs that even I couldn’t understand. Honestly, it was a mess.

Format your data like this:

{
  "instruction": "Summarize the key obligations in this contract clause",
  "input": "[Your actual contract text here]",
  "output": "[The summary you want the model to produce]"
}

The instruction-input-output format works reliably with most fine-tuning frameworks. I tried the pure conversational format first and spent four hours debugging why my loss wasn’t decreasing. Don’t be me.

Data cleaning checklist that saved my sanity:

• Remove any examples where input exceeds 2,048 tokens (for 8B model)
• Check for encoding issues, especially curly quotes and special characters
• Verify outputs actually match what the instruction asks for
• Remove duplicate or near-duplicate examples
• Add 10 to 15% edge cases deliberately

That last point matters more than people realize. If your model only sees “normal” examples, it falls apart the moment real users throw something weird at it.

Section 2: 8B vs. 70B, Our Head-to-Head Test Results (Spoiler: Bigger Isn’t Always Better)

Should you fine-tune Llama 3 8B or 70B? I see this question constantly in the Discord servers I lurk in. Everyone assumes bigger is better. My testing says otherwise, at least for domain-specific tasks.

I ran the same legal summarization fine-tuning on both models. Same dataset, same hyperparameters scaled appropriately, same evaluation prompts. The results surprised me:

So yes, the 70B model won on raw accuracy by about 3%. But a fine-tuned 8B beat the base 70B by 13 percentage points. For most use cases, especially custom model training for specific industry applications, the smaller fine-tuned model is the smarter choice.

Why does this matter? Three reasons. First, inference costs multiply fast in production. Second, you can actually run the 8B locally for testing. And third, the accuracy gap shrinks further when you have more domain-specific examples.

Section 3: QLoRA on Free Colab, The Exact Settings That Survived 12 Hours of Training

Look, if you’re reading a budget-conscious tutorial, you probably don’t have access to an H100 cluster. Neither do I. Fine-tuning Llama without expensive GPUs is genuinely possible, and Google Colab’s free tier is where I did most of my initial experimentation.

Below is the exact configuration that worked. I’m sharing the specific numbers because generic “just use QLoRA” advice helps nobody:

from peft import LoraConfig, get_peft_model
from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True
)

lora_config = LoraConfig(
    r=16,  # Don't go higher on free Colab
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

Settings that actually matter:

• r=16 is the sweet spot. I tried 32 and got OOM errors every single time on the T4.
• Double quantization (bnb_4bit_use_double_quant) saves enough VRAM to matter.
• Gradient checkpointing is mandatory. Enable it or suffer.
• Batch size of 1 with gradient accumulation of 4. Not glamorous, but stable.

Training arguments that survived the full run:

training_args = TrainingArguments(
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    warmup_steps=100,
    max_steps=1000,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=25,
    save_strategy="steps",
    save_steps=250,
    output_dir="./results"
)

Save every 250 steps. Colab will disconnect. Your runtime will crash. Accept this reality and check in often.

Section 4: The 23 Mistakes Log, Every Error Message, and How I Fixed It

I wish this section existed when I started. I kept a running log of every failure across my 48 hours. Below are the ones that cost me the most time:

Error 1: “CUDA out of memory.” You’ll see this constantly. Trust me. Solutions in order of effectiveness:

1. Reduce batch size (already at 1? You’re stuck)
2. Enable gradient checkpointing
3. Lower LoRA rank from 32 to 16 to 8
4. Use CPU offloading as a last resort

Error 2: “RuntimeError: expected scalar type BFloat16” A mixed-precision training conflict. Set bf16=False and fp16=True on T4 GPUs. The T4 doesn’t support bf16 properly, despite what some documentation suggests. I learned that one the hard way.

Error 3: Loss stays flat at 2.3 for 200+ steps. Usually a learning rate issue. I was using 1e-5 initially, which was way too low. I bumped it to 2e-4 and immediately saw movement.

Error 4: “Token indices sequence length is longer than specified.” Your data has examples exceeding max_length. Either filter them during preprocessing or truncate. Filtering is cleaner.

Error 5: Model outputs garbage after fine-tuning. This happened to me when I accidentally fine-tuned for 3,000 steps instead of 1,000. Overfitting on small datasets is real. Watch your validation loss like a hawk.

Error 6: Colab disconnecting mid-training. Not technically an error message, but it’ll ruin your day. Use from google.colab import drive to mount the drive and save checkpoints there. Also, keep a browser tab open and click occasionally. Yes, really.

[Link: common fine-tuning errors] for a more comprehensive troubleshooting guide.

Section 5: From Notebook to Production, The Deployment Steps Tutorials Skip

Congratulations, your model trains without exploding. Now what? Most tutorials mysteriously end right about here. Let me fill that gap.

Step 1: Merge and export your adapter

merged_model = model.merge_and_unload()
merged_model.save_pretrained("./final_model")
tokenizer.save_pretrained("./final_model")

Step 2: Quantize for deployment. Unless you’re deploying to A100s, you need quantization. GGUF format works with llama.cpp and most inference servers:

python convert.py ./final_model --outtype f16
./quantize ./final_model/model.bin ./final_model/model-q4_k_m.bin q4_k_m

Step 3: Pick your inference server. For domain adaptation best practices, I’ve tested three options:

• vLLM: High throughput thanks to PagedAttention, though performance varies by use case and hardware. Needs a decent GPU.
• llama.cpp: Runs on CPU, surprisingly usable for low-traffic apps.
• Text Generation Inference: Good middle ground, plays nicely with the Hugging Face ecosystem.

Step 4: Don’t skip evaluation. Before shipping, run your model against held-out test examples. Compare against the base model. If your fine-tuned version isn’t clearly better on your specific task, something went wrong.

When it comes to the Llama 3 vs. GPT fine-tuning debate for enterprise use cases, it really comes down to ownership. With Llama, you own the deployment. No API rate limits, no sudden pricing changes, no data leaving your infrastructure.

Your condensed timeline looks like this:

Hours 1 to 8: Dataset collection and cleaning. Aim for 200 to 500 quality examples. Use the JSON format I showed earlier.

Hours 9 to 16: Initial training runs on Colab. Start with my exact QLoRA settings. Don’t get creative yet.

Hours 17 to 24: Sleep. Seriously. Let your first full training run complete overnight.

Hours 25 to 36: Evaluate, iterate, and adjust hyperparameters. Try different LoRA ranks if you need to.

Hours 37 to 48: Merge, quantize, and deploy. Run production evaluations.

But here’s the honest truth about how much data you actually need: sometimes fine-tuning isn’t the answer at all. If your task is achievable with good prompting and RAG, that’s often the smarter move. Fine-tuning shines when you need consistent formatting, domain-specific language patterns, or behaviors that can’t be prompted reliably.

My $47 experiment worked because I had a clear, narrow use case. Don’t fine-tune for vague “make it better at my industry” goals. Start with prompting, add RAG if you need it, and fine-tune only when those approaches genuinely fail.

Now go break something. That’s how you actually learn this stuff.