A failed training run is not always a training failure. When a job stops before meaningful progress, the first question is whether the worker received the dataset that was intended. A truncated upload, invalid JSONL line, changed text encoding, or stale local file can produce symptoms that look like configuration trouble.

I start by separating intake failures from model failures. Upload corruption usually appears before the first stable training step: the system rejects the artifact, reports an unexpected size, cannot parse records, or stops while preparing batches. A configuration problem usually appears after the dataset has been accepted, such as an invalid model option, memory exhaustion, or loss becoming unstable.

Preserve the evidence first

Do not overwrite the original local file while investigating. Copy the command output, run identifier, dataset filename, expected digest, and configuration into the run record. If the file was uploaded successfully but the run failed later, those details make it possible to decide whether a resume is safe.

Check file size, line count, encoding, and digest from the source machine:

wc -l train.jsonl
wc -c train.jsonl
file -I train.jsonl
shasum -a 256 train.jsonl

The line count should be plausible for the expected dataset. A file that is much smaller than the source manifest, ends without a final newline, or reports a different encoding deserves inspection before another upload. The checksum identifies the exact local artifact; it does not prove that the labels are correct.

Find the first broken record

JSONL fails best when validation reports the first bad line and the reason. The script below checks that every non-empty line is JSON and that the prompt/completion fields are non-empty strings.

#!/usr/bin/env python3
import json
import sys
from pathlib import Path

def validate(path: Path) -> int:
    records = 0

    with path.open(encoding="utf-8") as handle:
        for line_number, line in enumerate(handle, start=1):
            if not line.strip():
                raise ValueError(f"line {line_number}: blank line")

            try:
                record = json.loads(line)
            except json.JSONDecodeError as exc:
                raise ValueError(
                    f"line {line_number}: invalid JSON ({exc.msg})"
                ) from exc

            if set(record) != {"prompt", "completion"}:
                raise ValueError(
                    f"line {line_number}: expected prompt and completion keys"
                )

            for key in ("prompt", "completion"):
                if not isinstance(record[key], str) or not record[key].strip():
                    raise ValueError(
                        f"line {line_number}: {key} must be a non-empty string"
                    )

            records += 1

    return records

if len(sys.argv) != 2:
    raise SystemExit("usage: validate_jsonl.py train.jsonl")

try:
    total = validate(Path(sys.argv[1]))
except ValueError as error:
    raise SystemExit(f"validation failed: {error}")

print(f"validated {total} records")

Run the validator before re-uploading:

python3 validate_jsonl.py train.jsonl

If the script identifies a malformed line, fix the source transformation rather than editing one record by hand. A hand edit can conceal a systematic exporter bug that will appear again in the next dataset.

Re-upload and decide whether to resume

After validation, create a fresh artifact record with the new digest. Upload the verified file and confirm that the server reports the same size and expected record count. Keep the failed artifact in the run log for traceability, but do not reuse it.

A run can usually resume only when the uploaded dataset digest is unchanged and the failure happened after a checkpoint that was fully written. If the dataset was truncated, relabelled, or re-split, start a new run. Continuing with changed input data makes loss curves and checkpoint comparisons misleading.

Add three preventative checks to the intake path:

  • Validate JSONL before the upload command is available.
  • Store the input digest and record count with each queued run.
  • Reject a resume request when the submitted dataset digest differs from the original.

That small discipline turns a vague “training failed” report into a specific recovery decision. Verify the artifact, identify the first invalid record, create a fresh run when data changed, and reserve resume for checkpoints built from the same verified input.

Add intake checks before the queue

The simplest prevention is to make the upload boundary explicit. A local preflight command should validate the JSONL file, write the digest into a small manifest, and retain the exact command that queued the run. The server can then compare the declared byte count and digest after transfer rather than treating an accepted HTTP request as proof that the source file was intact.

Encoding deserves a visible check as well. UTF-8 with a byte-order mark, copied spreadsheet characters, or a file saved in a local legacy encoding can pass a superficial line-count test while failing later in a tokenizer or parser. Keep source exports in UTF-8 and reject undecodable input during intake. Do not silently replace invalid bytes; replacement turns a transport problem into changed training content.

For longer uploads, retries should create a new temporary artifact and promote it only after validation completes. Retrying into the same partially written destination makes it difficult to distinguish a clean retry from an accidental append. The run record should retain the final accepted digest, not every failed transfer attempt.

These checks are cheap enough to run for every dataset. They prevent the recovery procedure from becoming a routine part of training and keep the distinction between input integrity and model behaviour clear.