Network Doctor

Hunts and Triage

A scenario tests a fault somebody thought of. A hunt looks for the ones nobody thought of.

netdoc-sim hunt takes a known-good control scenario and mutates it deterministically, from a seed, then runs the ordinary simulation lifecycle on each generated case and asks whether Network Doctor recognised what the simulator can prove it did.

Triage is the automation on top: hunt fixed baselines, reproduce every candidate finding, and only then treat it as real.

Running a hunt

netdoc-sim hunt healthy --seed 20260101 --cases 20
netdoc-sim hunt healthy --seed 20260101 --case 4 --json
netdoc-sim hunt healthy --seed 20260101 --case 4 --dry-run --json

The property that makes this usable: case N is derived from the hunt seed, the base scenario, and the case number alone. So --case N --seed S regenerates that exact case without first running cases 0 through N-1. A colleague can reproduce case 4 of your hunt in one command.

netdoc-sim help prints the flags and their defaults and limits. The accepted base scenarios and the mutation registry live in internal/simulation/hunt_generate.go.

Every case report carries the generator version, the root and case seeds, the materialised mutations, and a case fingerprint. Keep the seed, the case number, the generator version, and the reproduction command with any failure report; without them the finding is not reproducible and cannot be acted on.

Findings use semantic diagnosis fingerprints: they exclude prose and incidental timing, paths, process ids, and kernel names. That is what stops a reworded summary from looking like a new bug.

What a hunt false negative actually means

This is the single most misunderstood thing about hunts, so it is worth stating precisely.

A hunt false negative means the simulator independently established a network condition whose diagnostic meaning Network Doctor failed to recognise.

It does not mean “a mutation expected probe X to fail and probe X did not fail.”

The oracle is that contract in code, and it runs on a vocabulary of network conditions, domain facts such as “IPv4 internet reachability lost”, “the target served an expired TLS certificate”, “the proxy refused its CONNECT destination”, “QUIC datagrams dropped on UDP/443”.

It keeps two halves rigidly apart:

Half Reads Never reads
observed Simulator evidence and derived simulator truth The diagnosis
recognized One diagnosis Simulator evidence

On the observed side, a mutation that was merely generated or applied establishes nothing. Only real traces count: the certificate a client actually refused, the CONNECT a proxy actually declined, the kernel counter that actually matched a packet, or the client’s own dial of a controlled endpoint.

On the recognized side, recognition is expressed over netdoc’s stable cause vocabulary and its structured per-family verdicts, not over probe IDs. That is deliberate and load-bearing: a probe that is renamed, split, or merged without changing what the user is told leaves the oracle correct.

Recognition is deliberately strict

An expired certificate reported as a generic handshake failure, a refused destination reported as an unreachable proxy, and any unrelated failing row are all misses, because each one sends the user somewhere else. A cause on a passing row is context, not recognition.

That strictness is the point. A generic “the run failed somehow” comparison would score netdoc correct for naming a different fault with a different fix, which is exactly the bug class a hunt exists to find.

Where the oracle deliberately says nothing

Two observed faults imply no condition, because netdoc reports no failure for either by design:

  • An HTTP error status, which is a working service answering.
  • An invalid DoH response while DoT still resolves, which is encrypted DNS working.

A control scenario pins both down. Adding an expectation there would invent a contract the probes never made.

Reconciliation also runs only on stable paths, on the final client diagnosis. Unknown or unavailable address families, persistent netem, and actual timed path impairments are not treated as a final-state oracle. The opposite direction, where the simulator reached a family the diagnosis calls unreachable, is reported as a diagnostic contradiction rather than as a false negative, because it is a different kind of problem.

Generator versions

The mutation registry is versioned, and the reason is worth understanding before you add an operator.

Selection draws from a permutation of the applicable operators. So adding one operator, or reordering the list, changes which mutation every existing case number lands on, repointing cases that published artefacts already name.

The rules that follow from that:

  • Each operator carries the version it first appeared in.
  • An older generator is simply the registry truncated there.
  • New operators are therefore appended, never interleaved.
  • Case seeds are not versioned: two adjacent versions draw the same numbers and differ exactly where the operator list does.

A test pins an older generator against a fixed manifest, so this cannot silently break.

Telling absences apart

A recurring difficulty in this design, and a nice illustration of why evidence has to be structured rather than inferred.

Three condition families are about a route that is not there, and none of them can be established from reachability alone. So there are two different readings of the routing state:

  • “Where does this destination go?”: route evidence for a specific destination.
  • “What routes exist at all?”: the route table, read with ip route show from inside the node at the end of the run, recorded for every family the node has an address in.

An empty route list is therefore the positive statement “the table was read and held nothing”, which is what “no default route” needs. A missing record means nobody looked, which establishes nothing at all. Those are not the same, and collapsing them would let a measurement that never ran masquerade as a finding.

“Wrong default route” needs one thing more, because a default that goes nowhere and a network that is broken past the gateway look identical from the client: the control endpoint behind the original next hop, reached over its own specific route, has to still answer. That is what proves the old gateway still forwards and only the choice of default changed.

Two more families are about a port, and they are each other’s negative. The simulator records the outcome of its own dial rather than only whether it worked, because a reset and a timeout are different faults with different fixes, and the dialling end is the only place that difference is visible. “Connection refused” requires the dial to have been refused and no drop counter to have matched; “TCP port blocked” requires the opposite of both. Neither can be established by a dial that merely failed, and a single run cannot satisfy both.

Triage

netdoc-sim triage                               # observe; file nothing
netdoc-sim triage --scenarios healthy --cases 5
netdoc-sim triage --json
netdoc-sim triage --create                      # create issues through gh

Triage hunts the fixed baselines, re-runs each candidate’s exact case, and requires both the case fingerprint and the finding fingerprint to match before it will treat the candidate as real.

An unreproduced candidate is reported but never filed. That is the whole value of the command: a hunt on a busy machine will occasionally produce a case that does not reproduce, and filing those would poison the issue tracker.

--create is the only mode that writes to GitHub. It uses the configured gh client, suppresses duplicates by stable fingerprint, and treats a failed hunt, reproduction, parse, or gh call as an error rather than as a clean result, so a broken pipeline cannot report “nothing found.”

The fixed baselines and their seeds are authoritative in internal/simulation/triage.go.

Nightly automation

.github/workflows/hunt.yml is authoritative for the schedule, runner, permissions, case count, and issue-creation opt-in.

Two safety properties in that workflow are worth knowing about, because they are easy to break while editing it:

  • Scheduled issue creation requires a repository variable; manual dispatch requires an explicit input. It does not file issues by default.
  • Observation-only runs withhold the token entirely, even though the job declares the permission an opted-in run would need.

The workflow also needs a seeded-netem-compatible runner and its explicit Bash/pipefail behaviour preserved.

Relationship to Challenge Mode

Challenge Mode adds no fault model. A challenge id resolves, deterministically and with no state on disk, to a hunt base scenario and a hunt case number, and the case is materialised by the ordinary hunt generator with a maximum of one mutation.

Truth comes from the same collection path, so a mutation counts only when the executed run left independent evidence for it: the same rule the hunt uses. Recognition of a condition the hunt oracle already grades reuses that oracle’s recognized half rather than restating it, and the shared wire predicates are the one place either side reads evidence.

What is challenge-specific is only: the answer vocabulary, the eligibility contract, the difficulty metadata, and the match-up. See Challenge Mode.

Where next

This page has one editable copy: edit it at the source.