My Default LoRA Configuration for Small Instruction Datasets

A small instruction dataset benefits from a configuration that is easy to explain and easy to change. The aim is not to find a permanent recipe. It is to establish a baseline that exposes whether the next improvement should come from data, labels, prompt formatting, or adapter capacity.

For this note, “small” means a dataset with enough examples to define a narrow task but not enough coverage to hide inconsistent labels. That often means a few thousand to a few tens of thousands of examples, limited task diversity, and a clear output format. A dataset with several unrelated tasks is not small in the useful sense, even if its file size is modest.

Start with a stable baseline

My default starting point for a text instruction adapter is:

The configuration is useful because every setting has a visible failure signal. Rank that is too low can miss consistent task distinctions. Rank that is too high can increase file size without improving held-out decisions. A learning rate that is too high can make validation behaviour unstable; one that is too low can leave the output unchanged after the available epochs. Too many epochs can make summaries sound increasingly similar to training examples. Too few can leave schema or label formatting unreliable.

Sequence length deserves a separate audit. If most records are short, a large fixed limit creates padding and reduces useful batch work. If records regularly contain long quoted histories, truncating blindly can remove the actual request. Preserve the instruction and latest user content first, then measure how often the chosen limit cuts meaningful text.

Use one complete run as the reference point

The following command creates a reproducible baseline for an instruction dataset. It uses a current LoRADock base model and stores the resulting adapter under a distinct output name.

dock finetune \
  --base Qwen/Qwen2.5-7B-Instruct \
  --data ./train.jsonl \
  --validation-data ./validation.jsonl \
  --output qwen7b-ticket-triage-baseline \
  --rank 16 \
  --alpha 32 \
  --dropout 0.05 \
  --learning-rate 2e-4 \
  --max-seq-length 1024 \
  --epochs 3 \
  --seed 42

Before changing any setting, save the dataset digest, base-model revision, prompt template, split identifiers, and decoding settings used for validation. Those details are part of the experiment. A configuration without its data and evaluation context is only half a record.

Tune in a fixed order

I change one variable at a time in this order:

1. Correct dataset and label errors.
2. Fix split leakage and missing rare cases.
3. Adjust sequence length and prompt formatting.
4. Test learning rate or epochs.
5. Compare rank 8, 16, and 32.
6. Revisit dropout only after the data and capacity choices are clear.

This order avoids using rank as a remedy for a data problem. If the adapter emits invalid JSON, first inspect examples and output formatting. If it misses a rare category, verify that the category exists in both training and evaluation with stable labels. If it copies phrases from examples, review duplicates and epochs before adding capacity.

A good baseline should be documented in the registry with:

• adapter slug and semantic version;
• base model and revision;
• dataset digest and split policy;
• rank, alpha, dropout, learning rate, sequence length, epochs, and seed;
• intended task and unsupported cases;
• validation metrics, high-risk slices, and known failure modes.

That list makes later changes legible. When a newer release improves one decision but regresses another, the team can compare complete run records instead of guessing what changed.

The baseline is successful when it gives a reliable next decision. Keep it when validation is stable and the task boundaries are clear. Change it only when a specific failure has evidence behind it. Small instruction datasets reward that restraint: clean examples and disciplined evaluation usually move the result more than an immediate jump to larger rank or longer training.

Use validation signals to choose the next change

The most useful failure signal is one that has a narrow remedy. Repeated invalid JSON suggests an output-format or example problem before it suggests a larger adapter. Correct labels with poor performance on one product category point to coverage or split balance. Good validation accuracy but poor real prompt behaviour can point to an input-template mismatch. A steadily improving training loss paired with flat held-out results often means that another epoch is not the first change to make.

I keep a small error log with the prompt ID, expected structure, model output, failure category, checkpoint, and reviewer note. Ten reviewed failures can be more actionable than a single aggregate score. The log also prevents a change made for one error from being credited with an unrelated improvement in the next run.

Use a fixed decoding configuration for every checkpoint comparison. Temperature, max output tokens, stop sequences, and response parsing alter the visible result. When those settings change at the same time as rank or learning rate, the training record stops answering the question it was meant to answer.

Finally, preserve a baseline artifact even after a newer run is promoted. A stable rank-16 checkpoint is a useful rollback target and a control for future dataset revisions. Registry versions should make that relationship visible: the newest artifact is not automatically the best reference, but it should be comparable to a known configuration with documented evidence.

The configuration table is therefore a starting contract, not an optimisation claim. Any deviation should name the problem it intends to solve and the held-out evidence that will decide whether the change stays. That keeps short datasets from accumulating unexplained experiments.

The same record should list the reviewer responsible for the release decision and the date of the final validation pass.