Diagnostics

A run either finishes or raises, and nothing in between is visible. When a number comes out wrong, diagnostics=True writes what each stage of the pipeline actually held into diagnostics.log, beside that run’s results.json:

task = Registry.from_key(
    TOY_TASK,
    diagnostics=True,
    trainer_args={"fast_dev_run": True},
)
task.run()

Seven stages report, in the order a batch meets them: the frames the loader parsed, each step of a Pipeline, the batch the collator assembled with its two axes side by side, the states a backbone produced, the selection before and after the empty-selection repair, the mask the predictor actually reads, the namespace every loss binds to with each term’s value, and, once per split since neither changes between batches, the namespace the metrics bind to with the fields each of them names. A tensor reports its shape, dtype, device, non-finite count and range, and a mask, which is a batch and one axis of nothing but zeros and ones, reports how many of them are on. That last count is where a mask which is neither zero nor one, a word dropped on the word axis but still read on the subtoken axis, or a nan inside a pooled state shows up, and where a metric shows nothing.

Only under a smoke test. The record is per batch, so a full run writes gigabytes of it and pays the formatting on every step. A task asked to diagnose a run whose training batches nobody bounded raises rather than writing it: pass fast_dev_run, or a limit_train_batches of your own, which is what trainer_args already forwards and what task_args passes to a whole grid at once. A fraction of 1.0 is Lightning’s own default and bounds nothing, so it is refused like an absent one.

GenSPPTask is checked differently, because its search runs outside Lightning and reads the whole split once per candidate: what bounds it is population_size and n_generations, and a search of more than a handful of candidates is refused however the trainer was bounded. A diagnosed search also marks every candidate and every generation, and scores one candidate at a time however many devices it was given: the stages report in the order they run, so two threads writing at once would record a model that never existed.

It is written through the standard library’s logging under the pyhighlights.diagnostics logger, so a caller who wants the record on a console rather than in a file adds a handler to that logger and sets its level. Nothing is formatted while nothing is listening.

API

Diagnostics: what the pipeline held, stage by stage, for one bounded run.

A run either finishes or raises, and nothing in between is visible. When a number comes out wrong there is no record of what the pipeline actually held: the frames a loader parsed, the splits a preprocessing step returned, the batch a collator assembled, the states a backbone produced, the mask a selector proposed, the input the predictor was handed, or the terms a loss summed. Every one of those is otherwise reconstructed by hand, against a model built a second time.

So the stages report themselves, through the standard library’s logging under one logger. Nothing is written unless something asks for it: active() is a level check, and a call site that finds it false costs the check and the call. SPPTask turns it on for a run it has bounded, writing into that run’s own directory beside results.json.

Reading the record is the point rather than keeping it. What it answers that nothing else does: whether the word axis and the subtoken axis agree on every batch, whether a dropped word is absent from the predictor’s input on the backbone in use, how often the empty-selection repair fires, and whether every loss and metric binding finds the fields it names on every split.

pyhighlights.utility.diagnostics.active()

Whether anything is listening, which is what a call site checks.

A level check rather than a flag, so the record follows the logger the same way every other message in this library does, and a caller that configured logging themselves is not overridden.

Return type:

bool

pyhighlights.utility.diagnostics.describe(value)

One line for one thing the pipeline held.

A tensor reports the shape, the dtype, the device, how many entries are not finite and the range they cover. A mask is a batch and one axis of nothing but zeros and ones. A mask also reports how many entries are on, because every mask covers zero to one and only the count separates two masks. These fields show a mask that is neither zero nor one, a selection the predictor’s axis did not keep, a selection rate of 1.0 at the first batch, or a nan inside a pooled state. No metric shows these. A frame reports its rows, its columns and how many rows carry an annotation. Anything else reports itself, shortened, since a stage is free to name a number or a string beside its tensors.

Return type:

str

Parameters:

value (Any)

pyhighlights.utility.diagnostics.logger = <Logger pyhighlights.diagnostics (WARNING)>

One logger for every stage. A caller that wants the record on a console rather than in a file adds a handler to this and sets its level.

pyhighlights.utility.diagnostics.record(stage, /, **values)

Report what one stage held, under the name that stage goes by.

Formatting happens only when something is listening: every value here is a tensor the forward pass is holding anyway. Describing one costs a reduction over it, so a diagnosed run has to be a bounded one.

stage is positional-only because the values are named by whatever the caller is reporting: a corpus whose splits include one called stage would otherwise crash the run inside the call meant to explain it.

Return type:

None

Parameters:

• stage (str)

• values (Any)

pyhighlights.utility.diagnostics.writing(directory)

Send the record to directory for the length of one run.

The handler and the level are put back afterwards, so a task that diagnosed one run leaves the logger as it found it and a second task in the same process is not still writing into the first one’s directory.

Return type:

Iterator[Path]

Parameters:

directory (Path)