Fine-tuning teaches an open model your format, tone or domain from your own examples. With QLoRA — the base model loaded in 4-bit, plus small trainable "adapters" — a 7–8B model such as Qwen2.5 7B, Llama 3.1 8B or Mistral 7B fits on a single 24 GB card. Here is the whole recipe on a rented RTX 4090 or RTX 3090.

What you need

  • A 24 GB card. An RTX 4090 costs about $0.49 an hour here; an RTX 3090, at about $0.31, is slower but runs the same script.
  • The PyTorch 2.4 + JupyterLab environment, chosen at checkout on the marketplace.
  • Your examples as JSON Lines — one conversation per line, in the chat format most models are trained on:
{"messages": [{"role": "user", "content": "Summarise this ticket: ..."}, {"role": "assistant", "content": "..."}]}

A few hundred to a few thousand good examples go a long way; quality matters more than quantity.

1. Upload your data

Put train.jsonl in /workspace: drop it into the Files panel on the rental page, paste a link there to have the machine download it, or copy it with scp or rsync, which handle any size — see SSH into a rented GPU.

2. Install the libraries

Open JupyterLab from Web access on the rental page, start a terminal, and run:

pip install transformers peft trl bitsandbytes datasets accelerate

3. Train

Paste this into a notebook and run it:

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

base = "Qwen/Qwen2.5-7B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    base,
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True, bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16),
    device_map="auto",
)
data = load_dataset("json", data_files="/workspace/train.jsonl", split="train")

trainer = SFTTrainer(
    model=model,
    train_dataset=data,
    peft_config=LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                           target_modules="all-linear", task_type="CAUSAL_LM"),
    args=SFTConfig(output_dir="/workspace/run", num_train_epochs=2,
                   per_device_train_batch_size=2, gradient_accumulation_steps=8,
                   learning_rate=2e-4, bf16=True, logging_steps=10,
                   gradient_checkpointing=True),
)
trainer.train()
trainer.save_model("/workspace/my-adapter")

The first run downloads the base model, about 15 GB. Training then logs its loss every ten steps; a loss that falls and then levels off is what you want.

How long, and what it costs

For a few thousand examples of ordinary length, expect anything from under an hour to a few hours on an RTX 4090, depending on how long the examples are and how many epochs you run; an RTX 3090 takes longer. Two hours on an RTX 4090 cost about 2 × $0.49. If time runs short, extend the rental from its page.

4. Keep the adapter

/workspace/my-adapter holds only the adapter — tens to a few hundred megabytes, not a whole model. Download it from the Files panel, copy it with scp, or save it to My Drive (2 GB that stays with your account) before the rental ends, since /workspace is deleted then. Auto-save to My Drive, on the rental page, is a good safety net.

To use it, load the base model and the adapter with PEFT (PeftModel.from_pretrained), merge them with merge_and_unload(), or serve the adapter with vLLM, which can load LoRA adapters directly.

Bigger models

  • 13–14B: QLoRA needs about 18–24 GB, tight on a 24 GB card; a 48 GB RTX 6000 Ada or A6000, at about $0.56 an hour, is comfortable.
  • 70B: QLoRA needs roughly 48–80 GB; an A100 80 GB costs about $1.05 an hour.
  • Several GPUs: rent a machine with several cards, or a cluster — several machines rented together with a private network between them. Each machine knows its rank, so the same torchrun command starts the job on all of them. That network runs through RentGPUPower: fine for coordinating work and moderate amounts of data, not for datacenter-speed gradient exchange, so for heavy training one machine with several cards is faster.

The memory each method needs is in How much GPU memory do I need?

FAQ

Can I fine-tune Llama 3.1 8B instead?

Yes: change base. Llama models ask you to accept Meta's licence on Hugging Face and to sign in with a token on the machine; use a read-only token and delete it afterwards, as on any rented computer. Qwen2.5 downloads without an account.

What if the rental ends in the middle of training?

Extend it from its page before it does. For long runs, set save_steps in SFTConfig to save checkpoints in /workspace/run more often, and copy them to My Drive.

Is QLoRA as good as a full fine-tune?

For style and format it often comes close, at a fraction of the memory. A full fine-tune of a 7–8B model needs roughly 80–120 GB of GPU memory.

Is a 16 GB card enough?

For a 7–8B model, QLoRA needs about 10–16 GB: a 16 GB card can work with short examples and a batch size of 1, but 24 GB leaves room. Models of 1–3B fit in 12 GB.