Debugging NaN Loss After 40 Minutes of Training

Provide an ordered diagnostic workflow for unstable training. The useful result is a method that preserves evidence, exposes constraints, and makes the next decision reviewable. This note describes an engineering workflow rather than a claim about a particular organization or production event.

Establish the boundary

Capture the first invalid step and preserve configuration and logs. Keep the source version, configuration, and stable identifier together while this work is performed. That gives a reviewer a direct path from an observed outcome to the input and rule that produced it. Treat automatic checks and human judgment as separate controls, then record both in the incident note.

Use a fixed holdout or review sample for this step. Include normal records, rare boundary cases, and at least one input designed to fail safely. A useful change improves the intended condition without removing evidence about a known limitation.

Check input values, token lengths, empty records, and corrupt images

Check input values, token lengths, empty records, and corrupt images. Keep the source version, configuration, and stable identifier together while this work is performed. That gives a reviewer a direct path from an observed outcome to the input and rule that produced it. Treat automatic checks and human judgment as separate controls, then record both in the incident note.

Learning rate and inspect mixed-precision behaviour

Reduce learning rate and inspect mixed-precision behaviour. Keep the source version, configuration, and stable identifier together while this work is performed. That gives a reviewer a direct path from an observed outcome to the input and rule that produced it. Treat automatic checks and human judgment as separate controls, then record both in the incident note.

Reproduce on a tiny deterministic subset

Reproduce on a tiny deterministic subset. Keep the source version, configuration, and stable identifier together while this work is performed. That gives a reviewer a direct path from an observed outcome to the input and rule that produced it. Treat automatic checks and human judgment as separate controls, then record both in the incident note.

Change one factor at a time and record the result

Change one factor at a time and record the result. Keep the source version, configuration, and stable identifier together while this work is performed. That gives a reviewer a direct path from an observed outcome to the input and rule that produced it. Treat automatic checks and human judgment as separate controls, then record both in the incident note.

Add finite-value and dataset checks before long runs

Add finite-value and dataset checks before long runs. Keep the source version, configuration, and stable identifier together while this work is performed. That gives a reviewer a direct path from an observed outcome to the input and rule that produced it. Treat automatic checks and human judgment as separate controls, then record both in the incident note.

Practical check

Python finite-value checks and a small bash reproduction command. The example below is intentionally small enough to run before a larger recovery or release workflow.

from pathlib import Path

for path in Path("runs").glob("*.json"):
    if path.stat().st_size == 0:
        print(f"empty run record: {path}")

Evidence for the next decision

Keep the input digest, outcome, and reviewer note with the resulting artifact or postmortem. Attributing every NaN to mixed precision. When evidence is incomplete, preserve the failing example, define the missing validation, and defer promotion until results can be compared fairly.

Make the process idempotent. Repeating the same step on unchanged input should not append metadata, discard extra detail, or silently change ownership. Idempotence makes retries safe and exposes hidden state when outputs differ.

Review results by failure category rather than a single blended number. A small number of high-risk failures can outweigh an improvement on common cases, particularly when output feeds a release workflow.

Keep accepted and rejected examples together. A concise rejected sample with its expected outcome is a durable regression test and prevents the same edge condition from returning as an unexplained surprise.

Use a separate holdout set for release decisions. Training records can guide implementation, but they cannot demonstrate correct behaviour on new wording, new sources, or changed input order.