Fine-Tuning Large Language Models on amplitUDE

This guide covers fine-tuning large language models (LLMs) on amplitUDE’s GPU infrastructure, from small models (100M parameters) to large models (70B+ parameters).

Overview

amplitUDE’s H200 and H100 GPUs are well-suited for LLM fine-tuning:

H200: 141 GB memory - can fine-tune 70B models with quantization
H100: 80 GB memory - can fine-tune up to 30B models with quantization
Multi-GPU: Combine GPUs for larger models

What You’ll Learn:

Different fine-tuning methods (Full, LoRA, QLoRA)
GPU memory requirements by model size
Complete examples from DistilGPT-2 to Llama-3.1-70B
Production deployment with vLLM

Prerequisites

Environment Setup

module load anaconda3/2023

# Create LLM fine-tuning environment
mamba create -n llm-finetune python=3.11 \
    pytorch pytorch-cuda=12.1 -c pytorch -c nvidia

conda activate llm-finetune

# Install required packages
pip install transformers datasets accelerate evaluate
pip install peft bitsandbytes  # For LoRA/QLoRA
pip install tensorboard wandb   # Monitoring

Verify GPU Access

# Request GPU node
salloc --partition=GPU-H200 --gres=gpu:1 --time=01:00:00
srun --pty bash

# Load environment
module load anaconda3/2023
conda activate llm-finetune

# Check GPU
python << EOF
import torch
from transformers import __version__ as transformers_version

print(f"PyTorch: {torch.__version__}")
print(f"Transformers: {transformers_version}")
print(f"CUDA available: {torch.cuda.is_available()}")
if torch.cuda.is_available():
    print(f"GPU: {torch.cuda.get_device_name(0)}")
    print(f"Memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
EOF

Fine-Tuning Methods Comparison

Method Overview

Method	Trainable Params	Memory	Speed	Use When
Full Fine-Tuning	100%	Highest	Slow	Best performance needed, have resources
LoRA	~0.1-1%	Medium	Fast	Good balance, limited GPU memory
QLoRA	~0.1-1%	Lowest	Fast	Very limited GPU memory
Prefix Tuning	<1%	Low	Fast	Quick experiments

GPU Memory Requirements

Model Size	Full FT	LoRA (FP16)	QLoRA (4-bit)
1B params	~4 GB	~2 GB	~1 GB
7B params	~28 GB	~14 GB	~6 GB
13B params	~52 GB	~26 GB	~10 GB
30B params	~120 GB	~60 GB	~20 GB
70B params	~280 GB	~140 GB	~35 GB

Rule of thumb: Model size × 4 bytes/param for FP32, × 2 for FP16, × 0.5 for 4-bit

Quick Start: Fine-Tune DistilGPT-2

Start with a small model to verify your setup.

Complete Example

from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    TrainingArguments,
    Trainer,
    DataCollatorForLanguageModeling
)
from datasets import load_dataset

# Load model and tokenizer
model_name = "distilgpt2"  # 82M parameters
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(model_name)

print(f"Model parameters: {model.num_parameters() / 1e6:.1f}M")

# Load dataset (example: WikiText)
dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="train[:1000]")

# Tokenize
def tokenize_function(examples):
    return tokenizer(examples["text"], truncation=True, max_length=128)

tokenized_dataset = dataset.map(tokenize_function, batched=True, remove_columns=["text"])

# Data collator for language modeling
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)

# Training arguments
training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=8,
    save_steps=500,
    save_total_limit=2,
    logging_steps=100,
    learning_rate=5e-5,
    fp16=True,  # Mixed precision
)

# Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
    data_collator=data_collator,
)

# Train
trainer.train()

# Save
model.save_pretrained("./distilgpt2-finetuned")
tokenizer.save_pretrained("./distilgpt2-finetuned")

Job Script

#!/bin/bash
#SBATCH --job-name=finetune-gpt2
#SBATCH --partition=GPU-H200
#SBATCH --gres=gpu:1
#SBATCH --time=02:00:00
#SBATCH --mem=32G
#SBATCH --cpus-per-task=4
#SBATCH --output=logs/gpt2-%j.out

mkdir -p logs

module load anaconda3/2023
conda activate llm-finetune

python train_gpt2.py

LoRA Fine-Tuning (Recommended)

LoRA (Low-Rank Adaptation) is the best method for most use cases.

Why LoRA?

✅ Fast training: 3-10x faster than full fine-tuning
✅ Low memory: Fits larger models in available memory
✅ Good results: Usually <5% performance difference vs full FT
✅ Easy deployment: Merge adapter or use separately

LoRA Example: Llama-3.1-8B

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer
from datasets import load_dataset

# Load model
model_name = "meta-llama/Llama-3.1-8B"

tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,  # Use BF16 on H200/H100
    device_map="auto"
)

# Configure LoRA
lora_config = LoraConfig(
    r=16,                             # Rank
    lora_alpha=32,                    # Scaling factor
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],  # Which layers
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM
)

# Apply LoRA to model
model = get_peft_model(model, lora_config)

# Print trainable parameters
model.print_trainable_parameters()
# Output: trainable params: 33.5M || all params: 8033.5M || trainable%: 0.42

# Load dataset (example: instruction-following)
dataset = load_dataset("timdettmers/openassistant-guanaco", split="train[:1000]")

# Training arguments
training_args = TrainingArguments(
    output_dir="./llama-lora",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,  # Effective batch size: 16
    learning_rate=2e-4,
    fp16=False,
    bf16=True,  # Use BF16 on H200/H100
    logging_steps=10,
    save_strategy="epoch",
    optim="paged_adamw_8bit",  # Memory-efficient optimizer
)

# Trainer (using SFTTrainer for instruction tuning)
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    tokenizer=tokenizer,
    args=training_args,
    dataset_text_field="text",
    max_seq_length=512,
)

# Train
trainer.train()

# Save LoRA adapter
model.save_pretrained("./llama-lora-adapter")
tokenizer.save_pretrained("./llama-lora-adapter")

Load and Use LoRA Model

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "./llama-lora-adapter")

# Inference
tokenizer = AutoTokenizer.from_pretrained("./llama-lora-adapter")
inputs = tokenizer("Explain quantum computing:", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_length=200)
print(tokenizer.decode(outputs[0]))

QLoRA Fine-Tuning (Maximum Efficiency)

QLoRA uses 4-bit quantization to dramatically reduce memory usage.

QLoRA Example: Llama-3.1-8B in ~6 GB

import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    TrainingArguments
)
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

# Load model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=bnb_config,
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")
tokenizer.pad_token = tokenizer.eos_token

# LoRA config (same as before)
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)

# Training (rest same as LoRA example)
# ...

Memory usage:

Full precision (FP32): ~32 GB
Half precision (FP16): ~16 GB
LoRA (FP16): ~14 GB
QLoRA (4-bit): ~6 GB ✨

Full Fine-Tuning (Maximum Performance)

Use full fine-tuning when you need the absolute best performance and have sufficient GPU memory.

When to Use Full Fine-Tuning

✅ Model size < 7B parameters
✅ Have 40+ GB GPU memory available
✅ Need maximum performance
✅ Domain shift is large

Example: Full Fine-Tuning GPT-2

from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    TrainingArguments,
    Trainer
)

model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

# All parameters are trainable (no PEFT)
print(f"Trainable parameters: {model.num_parameters() / 1e6:.1f}M")

training_args = TrainingArguments(
    output_dir="./gpt2-full",
    num_train_epochs=5,
    per_device_train_batch_size=8,
    gradient_accumulation_steps=2,
    learning_rate=5e-5,
    weight_decay=0.01,
    fp16=True,
    logging_steps=100,
    save_strategy="epoch"
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset
)

trainer.train()

Dataset Preparation

Dataset Formats

1. Text Completion

{"text": "This is a training example."}
{"text": "Another example for language modeling."}

2. Instruction-Following

{
    "instruction": "Translate to French:",
    "input": "Hello, world!",
    "output": "Bonjour, le monde!"
}

3. Chat Format

{
    "messages": [
        {"role": "user", "content": "What is Python?"},
        {"role": "assistant", "content": "Python is a programming language..."}
    ]
}

Preparing Your Dataset

from datasets import Dataset
import pandas as pd

# From CSV
df = pd.read_csv("my_data.csv")
dataset = Dataset.from_pandas(df)

# From JSON
from datasets import load_dataset
dataset = load_dataset("json", data_files="my_data.json")

# From text files
dataset = load_dataset("text", data_files="corpus.txt")

# Save to Hub (optional)
dataset.push_to_hub("username/my-dataset")

Data Quality Tips

Clean your data
- Remove duplicates
- Filter out low-quality examples
- Remove PII (personal identifiable information)
Format consistently
- Use same prompt template
- Consistent instruction format
Balanced dataset
- Mix of easy and hard examples
- Cover all target use cases
Size recommendations
- Small task: 1,000 examples
- Medium task: 10,000 examples
- Large task: 100,000+ examples

Training Strategies

Hyperparameters

Learning Rate:

# Rules of thumb:
# Full fine-tuning: 5e-5 to 3e-5
# LoRA: 2e-4 to 1e-4
# QLoRA: 2e-4 to 1e-4

learning_rate=2e-4  # LoRA/QLoRA

Batch Size:

# Effective batch size = batch_size × gradient_accumulation_steps × num_gpus
# Target effective batch size: 32-128

per_device_train_batch_size=4
gradient_accumulation_steps=8
# Effective: 4 × 8 = 32

Epochs:

# Too few: underfitting
# Too many: overfitting
# Sweet spot: 3-5 epochs for most tasks

num_train_epochs=3

Memory Optimization Techniques

1. Gradient Checkpointing

model.gradient_checkpointing_enable()
# Trades compute for memory
# ~30% slower, ~50% less memory

2. Gradient Accumulation

gradient_accumulation_steps=4
# Effective larger batch size without OOM

3. Mixed Precision

# FP16 (older GPUs)
fp16=True

# BF16 (H200/H100 - recommended)
bf16=True

4. DeepSpeed (Multi-GPU)

# Install
pip install deepspeed

# Use in training
deepspeed --num_gpus=4 train.py --deepspeed ds_config.json

DeepSpeed config (ds_config.json):

{
    "fp16": {
        "enabled": true
    },
    "zero_optimization": {
        "stage": 2
    },
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto"
}

Multi-GPU Training

Single-Node Multi-GPU

#!/bin/bash
#SBATCH --job-name=multi-gpu-llm
#SBATCH --partition=GPU-H200
#SBATCH --gres=gpu:4
#SBATCH --time=12:00:00
#SBATCH --mem=512G
#SBATCH --cpus-per-task=32

module load anaconda3/2023
conda activate llm-finetune

# PyTorch DDP
torchrun --nproc_per_node=4 train.py

Training script modifications:

import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP

# Initialize
dist.init_process_group(backend="nccl")
local_rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(local_rank)

# Wrap model
model = DDP(model, device_ids=[local_rank])

# Use DistributedSampler
from torch.utils.data.distributed import DistributedSampler
train_sampler = DistributedSampler(train_dataset)

Evaluation and Testing

Evaluation Metrics

from evaluate import load

# Perplexity
perplexity = load("perplexity", module_type="metric")

# BLEU (for translation)
bleu = load("bleu")

# ROUGE (for summarization)
rouge = load("rouge")

# Accuracy (for classification)
accuracy = load("accuracy")

Evaluation Script

import torch
from transformers import pipeline

# Load fine-tuned model
generator = pipeline(
    "text-generation",
    model="./llama-lora-adapter",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Test prompts
test_prompts = [
    "Explain machine learning:",
    "Write a Python function to reverse a string:",
    "What is the capital of France?"
]

for prompt in test_prompts:
    output = generator(prompt, max_length=200, num_return_sequences=1)
    print(f"\nPrompt: {prompt}")
    print(f"Output: {output[0]['generated_text']}")

Production Deployment with vLLM

After fine-tuning, deploy with vLLM for fast inference.

Install vLLM

conda activate llm-finetune
pip install vllm

Serve Model

#!/bin/bash
#SBATCH --job-name=vllm-serve
#SBATCH --partition=GPU-H200
#SBATCH --gres=gpu:1
#SBATCH --time=48:00:00
#SBATCH --mem=128G

module load anaconda3/2023
conda activate llm-finetune

# Serve fine-tuned model
python -m vllm.entrypoints.openai.api_server \
    --model ./llama-lora-adapter \
    --host 0.0.0.0 \
    --port 8000 \
    --gpu-memory-utilization 0.9

Client Usage

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy"
)

response = client.chat.completions.create(
    model="./llama-lora-adapter",
    messages=[
        {"role": "user", "content": "Explain quantum computing"}
    ]
)

print(response.choices[0].message.content)

Complete Examples

Example 1: Sentiment Classification

from datasets import load_dataset
from transformers import AutoModelForSequenceClassification, AutoTokenizer, Trainer, TrainingArguments

# Load dataset
dataset = load_dataset("imdb")

# Load model
model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=2)
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

# Tokenize
def tokenize(batch):
    return tokenizer(batch["text"], padding=True, truncation=True)

tokenized_dataset = dataset.map(tokenize, batched=True)

# Train
training_args = TrainingArguments(
    output_dir="./sentiment-model",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    evaluation_strategy="epoch"
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["test"]
)

trainer.train()

Example 2: Code Generation

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer, TrainingArguments

# Load CodeGen model
model_name = "Salesforce/codegen-350M-mono"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# LoRA config
lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["qkv_proj"],
    lora_dropout=0.05
)

model = get_peft_model(model, lora_config)

# Load code dataset
dataset = load_dataset("codeparrot/github-code", split="train[:1000]")

# Train
training_args = TrainingArguments(
    output_dir="./codegen-lora",
    num_train_epochs=3,
    per_device_train_batch_size=4
)

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    tokenizer=tokenizer,
    args=training_args,
    dataset_text_field="code"
)

trainer.train()

Troubleshooting

Out of Memory (OOM)

Solutions:

Reduce batch size

per_device_train_batch_size=2  # Instead of 8

Enable gradient checkpointing
```
model.gradient_checkpointing_enable()
```

Use QLoRA instead of LoRA

# Reduces memory by ~50%
quantization_config = BitsAndBytesConfig(load_in_4bit=True)

Reduce sequence length
```
max_seq_length=512  # Instead of 2048
```
Use gradient accumulation
```
gradient_accumulation_steps=8
```

Slow Training

Solutions:

Enable mixed precision
```
bf16=True  # On H200/H100
```
Increase num_workers
```
dataloader_num_workers=4
```
Use faster optimizer
```
optim="adamw_torch_fused"
```

Check data loading

# Profile with
with torch.profiler.profile() as prof:
    trainer.train()
print(prof.key_averages())

Poor Results

Solutions:

Increase training data
- More examples = better results
- Minimum: 1,000 examples

Adjust learning rate

# Try different values:
learning_rate=1e-4  # If loss not decreasing
learning_rate=1e-5  # If training unstable

Train longer
```
num_train_epochs=5  # Instead of 3
```
Check data quality
- Remove duplicates
- Fix formatting issues
- Balance classes

Further Resources

Hugging Face Transformers - Main library docs
PEFT Documentation - LoRA/QLoRA guide
TRL Documentation - Trainer library
vLLM Documentation - Inference deployment

Last updated: May 2026

Fine-Tuning Large Language Models on amplitUDE

Overview

Prerequisites

Environment Setup

Verify GPU Access

Fine-Tuning Methods Comparison

Method Overview

GPU Memory Requirements

Quick Start: Fine-Tune DistilGPT-2

Complete Example

Job Script

LoRA Fine-Tuning (Recommended)

Why LoRA?

LoRA Example: Llama-3.1-8B

Load and Use LoRA Model

QLoRA Fine-Tuning (Maximum Efficiency)

QLoRA Example: Llama-3.1-8B in ~6 GB

Full Fine-Tuning (Maximum Performance)

When to Use Full Fine-Tuning

Example: Full Fine-Tuning GPT-2

Dataset Preparation

Dataset Formats

Preparing Your Dataset

Data Quality Tips

Training Strategies

Hyperparameters

Memory Optimization Techniques

Multi-GPU Training

Single-Node Multi-GPU

Evaluation and Testing

Evaluation Metrics

Evaluation Script

Production Deployment with vLLM

Install vLLM

Serve Model

Client Usage

Complete Examples

Example 1: Sentiment Classification

Example 2: Code Generation

Troubleshooting

Out of Memory (OOM)

Slow Training

Poor Results

Best Practices Checklist

Further Resources