A Three-GPU LoRA Run That Was Slower Than One GPU

Adding devices should reduce elapsed time only when the training job has enough useful work to distribute. A LoRA run can miss that condition even when the base model is large. Small microbatches, short sequences, slow tokenisation, frequent checkpoints, and synchronization after every step can make a three-GPU run slower than a carefully configured single-GPU run.

The workload in this example was instruction tuning on medium-length records. Each record was padded to a fixed maximum sequence length, the adapter targeted attention and feed-forward projections, and the run used gradient accumulation to reach its effective batch size. The first distributed attempt assumed that three GPUs would provide close to three times the throughput. That assumption was never measured component by component.

Break elapsed time into stages

A single wall-clock number hides the useful diagnosis. For each run, record at least these stages:

A short sequence and small per-device batch shrink the amount of compute between synchronization points. In that case, the workers spend proportionally more time waiting for each other. The problem becomes more visible when gradient accumulation is low: each tiny microbatch triggers another collective operation.

Start by capturing a simple time series during a bounded run. The command below samples GPU activity once per second and writes it to a local CSV file.

nvidia-smi   --query-gpu=timestamp,index,utilization.gpu,utilization.memory,memory.used,power.draw   --format=csv,noheader,nounits   --loop=1 > gpu-utilization.csv

Run the sampler in one terminal, then use a separate terminal for a short timed training run. The time utility records overall process duration while the application logs should report data, step, and checkpoint events.

time dock finetune \
  --base Qwen/Qwen2.5-7B-Instruct \
  --data ./support-train.jsonl \
  --validation-data ./support-validation.jsonl \
  --output qwen7b-ticket-triage-distributed \
  --rank 16 \
  --alpha 32 \
  --learning-rate 2e-4 \
  --max-seq-length 1024 \
  --epochs 1 \
  --seed 42 \
  --devices 0,1,2 \
  --per-device-batch-size 1 \
  --gradient-accumulation-steps 12 \
  --checkpoint-interval 500

The command is intentionally a one-epoch diagnostic run. A performance investigation should be shorter than a full training run until the bottleneck is known.

Effective batch size is not the whole story

Effective batch size is commonly expressed as:

effective_batch = devices * per_device_batch * gradient_accumulation_steps

That calculation is necessary for comparable optimisation behaviour, but it does not predict scaling. Consider two configurations with the same effective batch:

• One device, microbatch 6, accumulation 6.
• Three devices, microbatch 1, accumulation 12.

Both can process a similar number of examples per optimizer update. The second configuration performs more synchronization relative to useful compute because each device works on a much smaller microbatch. It may also leave memory headroom unused.

Sequence length matters in the same way. A task dominated by 200-token records gives each worker less computation than one dominated by 1,000-token records. Padding can make this worse: if a batch combines a few long records with many short ones, all workers may pay for the longest sequence.

Before changing the number of devices, inspect the distribution of tokenized lengths. A simple report can identify whether a fixed sequence limit is mostly padding:

import json
import sys
from statistics import mean

lengths = []

with open(sys.argv[1], encoding="utf-8") as handle:
    for line in handle:
        record = json.loads(line)
        text = record["prompt"] + "\n" + record["completion"]
        lengths.append(len(text.split()))

if not lengths:
    raise SystemExit("no records found")

lengths.sort()
for percentile in (50, 90, 95, 99):
    index = min(len(lengths) - 1, round((percentile / 100) * (len(lengths) - 1)))
    print(f"p{percentile}: {lengths[index]} whitespace-delimited tokens")
print(f"mean: {mean(lengths):.1f}")

This is not a tokenizer-accurate count, but it is enough to reveal whether the input shape is unexpectedly skewed. Replace it with the exact tokenizer report before making a final capacity decision.

Use an illustrative timing table

The table below is an illustrative format, not a benchmark result. Fill it with measurements from the same dataset slice, seed, checkpoint interval, and validation policy.

A useful result is often negative: the distributed run is slower, but the table shows why. If GPU utilisation repeatedly falls while the CPU is parsing records, increase data-loader workers, cache tokenised examples, or simplify expensive preprocessing. If all devices become idle after backward passes, increase per-device work where memory allows, group batches by length, or test fewer synchronization-heavy settings. If pauses line up with checkpoints, reduce checkpoint frequency during the diagnostic phase and confirm that storage throughput is not the limiting resource.

Decide whether to scale out

Keep the job on one device when it reaches acceptable utilisation, fits memory, and completes inside the required iteration window. The simpler configuration is easier to reproduce and usually has fewer failure modes.

Scale out only after a single-device profile shows that compute, rather than data preparation or file I/O, is the dominant cost. Then increase per-device batch size before assuming more accumulation steps solve the problem. Keep the dataset slice, seed, and checkpoint policy fixed while testing that change.

The practical checklist is short: sample device use, time the run, inspect sequence lengths, isolate checkpoint pauses, compare equal-quality configurations, and choose the smallest setup that meets the iteration target. More GPUs are a capacity option, not a default configuration.

Make the diagnostic repeatable

Run the comparison on a fixed subset before changing the production schedule. The subset should preserve the normal length distribution and include enough records to pass through several checkpoint boundaries. Record software versions, visible devices, precision mode, data-loader worker count, and storage location. A comparison made after one run has warmed filesystem caches and another has not is not useful evidence.

Also watch for a deceptively fast distributed run that changes training behaviour. If a larger effective batch changes the number of optimizer updates per epoch, the elapsed time may improve while the checkpoint quality is no longer comparable. Keep either optimizer updates or total examples processed aligned, then evaluate the same held-out set. Performance decisions belong beside quality decisions, not ahead of them.

When the profile points to CPU preparation, a modest pre-tokenisation cache can be more valuable than an additional GPU. When it points to communication, reducing the device count can be the correct optimisation. The result is not a failure of distributed training; it is evidence that the workload shape did not justify it.

Keep the profiling artifacts with the training run: the command, device sample, subset manifest, and final comparison table. A later model revision may change the balance entirely, and the old profile gives the next investigation a trustworthy starting point instead of another assumption about scaling.

Repeat the check after major tokenizer, driver, storage, or base-model changes; a profile is evidence for one workload, not a permanent capacity rule.