Fine-Tuning Qwen2.5-7B on 18,000 Support Tickets

A support-routing adapter has a narrower job than a general support assistant. It should identify the owning queue, recognise escalation conditions, and return a compact handoff record without inventing a resolution. That scope made a 18,000-ticket training set manageable, but only after the records were reduced to fields that were stable across teams and time periods.

The target release was a private draft of qwen7b-ticket-triage on Qwen/Qwen2.5-7B-Instruct. The goal was not to answer customers directly. Each output had to contain a queue, a priority, an escalation flag, and a short internal summary. Keeping those fields separate made failures easier to classify than a single free-form response.

Define the record before cleaning the dataset

Each source ticket was converted into a single instruction example with four input fields:

• subject: the customer-provided title after identifier redaction
• body: the message content after privacy filtering
• channel: email, web, or chat
• product_area: a stable product family when known

The completion used a JSON object with this shape:

{
  "queue": "billing",
  "priority": "normal",
  "escalate": false,
  "summary": "Customer reports a duplicate card charge after a plan change."
}

The label taxonomy had six queues: account, billing, integrations, product, security, and technical. Priority was limited to low, normal, high, and urgent. An escalation did not mean that the ticket was severe; it meant that a human owner needed to review it before the usual queue workflow could continue. Payment disputes, account lockouts, suspected compromise, and data-loss reports were common escalation categories.

I treated privacy filtering as a data-preparation stage, not a last-minute string replacement. Direct identifiers such as email addresses, phone numbers, order IDs, access tokens, and IP addresses were removed or replaced with stable placeholders. Contextual identifiers still needed review. A message can reveal a person or organisation through a rare project name, an internal hostname, or a verbatim log line even when obvious fields are absent.

The split was 15,300 records for training, 1,350 for validation, and 1,350 for a held-out evaluation set. Records from the same conversation thread stayed in one split. The split was also stratified by queue and priority, so rare security and urgent examples did not disappear from validation.

Validate JSONL before uploading

The training file used prompt/completion JSONL because the desired output shape was compact and deterministic. A representative record looks like this:

{"prompt":"Route this support ticket. Subject: Duplicate charge after upgrade\nChannel: web\nProduct area: subscriptions\nBody: I changed plans yesterday and now see two charges. I need to know which one will be refunded.","completion":"{\"queue\":\"billing\",\"priority\":\"normal\",\"escalate\":false,\"summary\":\"Customer reports a duplicate charge after a plan change.\"}"}

The validator below checks line-level JSON, required top-level fields, non-empty strings, completion JSON, and allowed label values. It intentionally stops on the first invalid record. For a training upload, one clear error is more useful than a long list that hides the earliest data defect.

#!/usr/bin/env python3
import json
import sys
from pathlib import Path

QUEUES = {"account", "billing", "integrations", "product", "security", "technical"}
PRIORITIES = {"low", "normal", "high", "urgent"}
REQUIRED_OUTPUT = {"queue", "priority", "escalate", "summary"}


def fail(line_number: int, message: str) -> None:
    raise ValueError(f"line {line_number}: {message}")


def validate_record(line_number: int, record: dict) -> None:
    if set(record) != {"prompt", "completion"}:
        fail(line_number, "record must contain only prompt and completion")

    for field in ("prompt", "completion"):
        if not isinstance(record[field], str) or not record[field].strip():
            fail(line_number, f"{field} must be a non-empty string")

    try:
        output = json.loads(record["completion"])
    except json.JSONDecodeError as exc:
        fail(line_number, f"completion is not valid JSON: {exc.msg}")

    if set(output) != REQUIRED_OUTPUT:
        fail(line_number, "completion has unexpected or missing keys")
    if output["queue"] not in QUEUES:
        fail(line_number, "completion queue is not allowed")
    if output["priority"] not in PRIORITIES:
        fail(line_number, "completion priority is not allowed")
    if not isinstance(output["escalate"], bool):
        fail(line_number, "completion escalate must be a boolean")
    if not isinstance(output["summary"], str) or not output["summary"].strip():
        fail(line_number, "completion summary must be a non-empty string")


def main() -> int:
    if len(sys.argv) != 2:
        print("usage: validate_tickets.py train.jsonl", file=sys.stderr)
        return 2

    path = Path(sys.argv[1])
    valid_records = 0

    with path.open(encoding="utf-8") as handle:
        for line_number, line in enumerate(handle, start=1):
            if not line.strip():
                fail(line_number, "blank lines are not allowed")
            try:
                record = json.loads(line)
            except json.JSONDecodeError as exc:
                fail(line_number, f"record is not valid JSON: {exc.msg}")
            validate_record(line_number, record)
            valid_records += 1

    print(f"validated {valid_records} records in {path}")
    return 0


if __name__ == "__main__":
    try:
        raise SystemExit(main())
    except ValueError as exc:
        print(f"validation failed: {exc}", file=sys.stderr)
        raise SystemExit(1)

Run it against every split, not only the training file:

python3 validate_tickets.py train.jsonl
python3 validate_tickets.py validation.jsonl
python3 validate_tickets.py evaluation.jsonl

The validator does not prove that a queue label is correct. It prevents structural errors from reaching the run queue. Label quality needs a separate sample review, especially for tickets that could plausibly belong to both technical and integrations support.

Choose a conservative training configuration

The initial configuration used rank 16 and alpha 32. Rank 16 provided enough adapter capacity for queue distinctions and terse summaries without making the first run expensive to iterate on. Alpha 32 kept the scaling factor aligned with the rank choice. Dropout was set to 0.05 because the ticket corpus contained recurring phrasing, templated replies, and repeated issue descriptions.

A learning rate of 2e-4 was a starting point, not a claim that it is optimal for all Qwen tasks. The run used three epochs with a maximum sequence length of 1,024 tokens. Longer tickets were truncated from the oldest quoted material first, after preserving the subject and the latest customer message. That policy reduced accidental training on agent signatures and historical reply chains.

Checkpoint selection should not default to the last checkpoint. Keep checkpoints at regular intervals, then evaluate each candidate on the validation set with fixed decoding parameters. A lower training loss is useful evidence, but it does not confirm that output JSON remains valid or that escalation cases are retained.

The corresponding LoRADock command is:

dock finetune \
  --base Qwen/Qwen2.5-7B-Instruct \
  --data ./train.jsonl \
  --validation-data ./validation.jsonl \
  --output qwen7b-ticket-triage \
  --rank 16 \
  --alpha 32 \
  --dropout 0.05 \
  --learning-rate 2e-4 \
  --max-seq-length 1024 \
  --epochs 3 \
  --seed 42

The expected stages are upload validation, dataset intake, queued execution, checkpoint creation, validation, and private-registry publication. Record the dataset digest, configuration, base-model revision, and selected checkpoint with the resulting adapter. Without those details, a later regression cannot be traced to data, settings, or the base model.

Evaluate the routing decision separately from the prose

The held-out set was scored on four independent dimensions:

Escalation recall matters more than overall accuracy for the urgent and security slices. A model can achieve a good aggregate queue score while missing a small class that should never be silently downgraded. Review false negatives by subtype: account lockout, suspected compromise, payment dispute, or data-loss signal. Those categories often need additional examples or a rule outside the model rather than a larger rank.

Malformed output should be counted before any repair step. If an integration later adds a JSON repair layer, track repaired and unrepaired validity separately. Otherwise the training run can appear more reliable than the model actually is.

Release only when the operating rules are clear

The first private release should include a short model card with the supported queues, known ambiguous cases, dataset version, base-model revision, and evaluation procedure. It should also state that the adapter routes work for human review; it does not make irreversible account, billing, or security decisions.

A retrain is justified when the queue taxonomy changes, a new product area introduces a material volume of unmatched tickets, the malformed-output rate rises after a base-model or prompt-template change, or review finds a repeated escalation failure. A handful of unusual messages is not enough reason to append examples blindly. First decide whether the issue is a label-policy gap, a preprocessing defect, an evaluation blind spot, or a task that should remain rule-based.

The durable workflow is simple: validate the records, preserve split boundaries, keep the configuration reproducible, score the high-risk slices independently, and publish only the checkpoint whose limitations are documented.