Method

Supported hardware

Each architecture's units, recipes, notes, and flags are in docs/units.json. The tool writes that file from its tables, and a test keeps it current (collect/units_json.rs, units_json).

NVIDIA chip counters

Diescope reads hardware counters with CUPTI PM sampling. PM sampling reads device-wide counters on a timer from Diescope's own process. It sees every process's work and the idle time between kernels, and it doesn't load code into the measured process (collect/nvidia/cupti.rs).

PM sampling reads only metric sets that fit in one counter pass. Every table's exact set fits one pass on every chip it lists, checked with CUPTI's host API (bench/results/metrics/nvidia/pm-passes.txt, and the test every_table_fits_one_pass_on_every_chip_it_lists).

Units on Ampere, Ada, and Blackwell

Each unit is a counter, or the mean of two, as a percent of that unit's peak. The recipe for each unit is recorded in every report (results.unit_recipes).

Every chip counter is read as .avg.pct_of_peak_sustained_elapsed (or .pct_of_peak_sustained_elapsed for the DRAM sum). "Peak" is the sustained peak CUPTI defines for the counter on the chip. Diescope doesn't compute it. Whether it follows the live clock is settled for FP32 only; see "Clocks and percent of peak".

FP32 is the FMA pipe. The FMA pipe runs FP32, fp16, and integer multiply-add, in two halves: the heavy half runs all three, and the lite half runs FP32 and fp16. On Blackwell, FP32 is the whole pipe's instruction counter. Code that runs only integer multiply-adds fills the heavy half while FP32 reads 50%, so both halves are kept as diagnostics.

Why not one half. Method 2 read FP32 as the busier half. On GB205, FFMA loops with added MOV, IADD, and LOP3 instructions showed that back-to-back integer and logic instructions also count on the heavy half, and some FFMAs move from one half to the other (bench/results/alu-sweep/). So the busier half read above the FFMA work. The whole-pipe counter matched the FFMA work in all 172 sweep rows with a work value. On llama.cpp generate it reads 15% where the busier half read 37%, over samples with SM active above 50% (bench/results/fma-umbrella/). On prompt processing the two are closer, 25.7% against 31.1%, in one run of 77 samples.

Ampere and Ada read the halves' mean. There the whole-pipe counter has no realtime form for PM sampling, and its plain form takes a second pass beside the table (bench/results/metrics/nvidia/pm-fma-whole-pipe.txt). So FP32 is (FMA heavy + FMA lite)/2. If Ampere and Ada count back-to-back integer and logic instructions on the heavy half as GB205 does, the mean counts them at half weight, and the whole-pipe counter doesn't count them. On a recorded llama.cpp generate run on GB205, the halves' mean over the whole run was 21.0% where the whole-pipe counter read 14.3% (bench/results/fp32-pipe-fixtures/). Compare FP32 across architectures with that in mind. The unit's note says so in every report. It hasn't been checked on Ampere or Ada hardware.

DRAM is bytes moved. Method 1 used dram__throughput, the higher of DRAM active cycles and bytes moved: a busy-ness measure. Since method 2, DRAM is bytes moved, so DRAM's percentage is a share of bandwidth. Even fully DRAM-bound work reads below 100%, because DRAM has overhead it can't avoid; stranded capacity at DRAM includes that overhead. dram__throughput is kept as the DRAM throughput diagnostic. On GB205 the byte counter's total was 0.98 to 1.00 of the bytes calibrate moved, and the two DRAM counters agreed to every printed digit (bench/results/calibrations/, quick-14).

Rates. Where a table counts DRAM bytes as a raw total (dram__bytes.sum), reports give the rate in bytes per second (results.rates_per_s). It's the count summed over the samples, with a sample the window cuts counted for its share, divided by the time those samples cover (metrics.rs, count_rates). It is counted, never a percent times a peak.

Units on Turing

Turing's tables use the closest one-pass counters:

• Tensor+FP16: the pipe tensor cores share with fp16 math (sm__pipe_shared_cycles_active_realtime), so it reads higher than tensor work alone whenever fp16 math runs. A different measure.
• FP32: the FMA-pipe instruction counter, as on Blackwell (sm__inst_executed_pipe_fma_realtime). Not yet checked on Turing hardware.
• DRAM: framebuffer partition throughput (fbpa__throughput). Neither DRAM bytes nor dram__throughput fits one pass on TU104. A different measure.

There's no DRAM throughput diagnostic and no counted rate on Turing.

Diagnostics

SM active and occupancy on every architecture; FMA heavy, FMA lite, and DRAM throughput on Ampere, Ada, and Blackwell. Diagnostics aren't part of DIE-Util.

Sampling and draining

• Interval. Samples are 10 ms by default (run --interval-ms; the agent opens counters at 10 ms, agent.rs, AGENT_INTERVAL_MS).
• First sample dropped. The first sample after each start spans the setup, so it's dropped (collect/nvidia/counters.rs).
• Broken samples dropped. A sample with a NaN unit or diagnostic, an empty span, or a missing end time is dropped. A missing raw count leaves the sample in (metrics.rs, span).
• Draining. Samples are decoded from the counter buffer when Diescope drains it. run drains every 1 s, apart from its 100 ms loop that watches the command, signals, GPU-Util, the video engines, clocks, and power (run.rs, DRAIN_MS, POLL_MS). The agent drains at each poll, every 1 s by default. Reports record the drain period (collection.drain_ms).
• What a drain costs. On GB205, each drain costs Load/store most of the one 10 ms sample spanning it: that sample reads near 0, about 8.6 ms lost per drain. At 100 ms drains Load/store read 7 to 9% low. At 1 s drains a fit across the drain periods gives 0.9% low. That's the fit's figure, not a measurement: all four 1 s points were unusable, because the work wasn't running for 2 to 6% of the window. FP32 (the FMA pipe's instruction counter) and FMA lite aren't affected: on the FFMA points the FMA pipe counter read 0.003% below FMA lite at every drain period, and on the usable drained points FMA lite read within 0.1% of the work. Image size had no effect. One 85 s run (bench/results/drain-study/). No correction factor is applied. Load/store's note in each report gives this figure.
• Video engines. NVML reports engine use over its own recent window. run reads the engines every 100 ms, and each sample gets the first reading at or after its end (run.rs, EngineReads). An engine NVML can't read is NaN in its slot, and is averaged over the time it was available. Unavailable is never treated as zero.

GPU-Util

GPU-Util is NVIDIA's own number, the one nvidia-smi shows.

• Source. NVML's timestamped samples (nvmlDeviceGetSamples), read by UtilReader::read (collect/nvidia/nvml.rs). On the test laptop a sample arrives every 200 ms.
• What a sample covers. Each sample is booked to the time since the previous sample, and to at most 1 s before it, NVML's longest documented averaging period (MAX_NVML_PERIOD_NS, spans_from_samples). One run on the test laptop checked this against a known workload: each sample matched the 200 ms ending at its timestamp, to within 1.5 points at the edges of busy periods (bench/results/nvml-timing-check.md).
• Same window as the counters. GPU-Util is weighted by its overlap with the chip samples, so both numbers cover the same time (metrics.rs, gpu_util_over_samples).
• End of a run. run waits up to 1 s after the command ends for an NVML sample at or after the end.
• Fallback. Diescope switches a GPU to polling nvmlDeviceGetUtilizationRates for good, and says why, when the GPU reports no samples, the driver lacks the call, reading samples fails 3 times over at least 7 s, or no new sample arrives for 10 s. A polled reading is booked to the time since the previous poll, up to 1 s, so polled spans are estimates. The polling path hasn't run on real hardware.

DIE-Util, gap, and stranded capacity

• Per-sample maximum. For each sample, DIE-Util is the throughput of the busiest available unit (dieutil.rs, of_sample). Those values are time-weighted over the window, idle gaps included, and the mean is clamped to 0 to 100% (metrics.rs, summarize). A single sample isn't clamped: a unit that reads around its peak would otherwise average below its own mean, and DIE-Util could read below its limiting unit. samples.csv shows each sample's value as read. Reports name this rule in die_util_algorithm, as max-unit, status experimental.
• Why the busiest unit. A chip can't go faster than its bottleneck, so the busiest unit shows how close the chip is to its limit. An average across units would mix a saturated unit with idle ones.
• Limiting unit. The unit with the highest time-weighted throughput among units available for at least 90% of the window (MIN_LIMITING_AVAILABILITY). There's none, with a reason, when no unit reaches 1% of its peak, or when the units that do weren't available for 90% of the window.
• Gap and stranded capacity. As defined above, from GPU-Util and DIE-Util over the same window (metrics.rs, Headline).
• Rounding. Screens and report tables show one decimal. The gap, stranded capacity, idle, and active are computed in whole tenths from the rounded inputs, so the shown parts add up exactly (Headline::rounded). Files keep full precision.
• Noise floor. NVML reports GPU-Util in whole percents, so a gap within half a percent of zero can't be told apart from zero. The floor is half of GPU-Util's resolution, which each report records (collection.gpu_util_resolution_pct; metrics.rs, gap_floor_points). A gap from the floor to 0 counts as 0, with idle taking all of stranded capacity. Below the floor the split is n/a, with the reason. A negative gap is real when the busiest unit is one GPU-Util doesn't count, such as the video encoder: a 4K HEVC encode read GPU-Util 8.4% and DIE-Util 93.8% on the test laptop (rust/src/testdata/golden/).

Coverage

• GPU-Util coverage. The share of chip-sampled time GPU-Util covers. Below 90%, the gap and the split aren't computed, and the reason is given instead (MIN_GPU_UTIL_COVERAGE).
• Chip coverage. The share of the run that usable chip samples cover (metrics.rs, ChipCoverage). Below 90%, the result is partial. The report title, run's first line, and DIE-Util's row say "partial (counters covered N% of the run)", and every headline number covers only that share. GPU-Util over the whole run is shown beside it when NVML covered at least 90% of the run. Reports carry results.chip_coverage_pct and results.chip_coverage_gate. bench leaves partial runs out of its statistics and names them.
• Duty cycles. With --duty, the counters run on part of each period. Chip coverage is then gated against the time they were scheduled to run, and the report says what share of the run was sampled. A duty run is partial only if the counters covered less than 90% of their scheduled time.

The steady window

run --window markers also measures the steady state between two lines the workload prints, diescope: measure start and diescope: measure stop, after its setup and before its teardown (run.rs, Marks). Markers are timed when the line reaches run, so a workload must flush them; the bundled ones do. A workload that learns its window afterwards prints the times instead. The llama.cpp workload does: its window ends when llama-bench reports its result and is as long as the timed repetitions, so loading the model and llama-bench's warm-up fall outside it.

The report keeps the whole run's numbers in results and adds the window's in window.results, with its own chip coverage and gate. It says where the window was and how it was found. The report's clocks then cover the window only, not the whole run. bench asks for the window on every workload and summarizes from it. A round whose window went missing is left out and named, so whole-run numbers never average in with steady states.

Clocks and percent of peak

Reports record the SM and memory clocks over the run, or over the steady window: lowest, highest, and time-weighted mean, from a reading every 100 ms (report.rs, ClockWatch). The agent records them over each chip window, over the time its samples cover.

Reports also record why the GPU held its clocks down, as NVML says, read with the clocks (hardware.throttle_reasons; calibration.rs, limits_over). The list holds every reason read over the run, or over the steady window. An empty list means nothing limited the clocks; null means NVML couldn't say. A window too short to hold a reading takes the nearest one.

For FP32, the percentage follows the live SM clock: it matched its work computed at the clock the kernels measured, from 1,757 to 2,627 MHz (bench/results/fp32-pipe-fixtures/calib-f12fe7c-quick.json). Other SM pipes haven't been checked.

For DRAM, whether the percentage follows the live memory clock is open. On the test laptop, DRAM's implied peak was 671 GB/s at a steady 14,001 MHz memory clock (bench/results/calibrations/, quick-5 and quick-14). In one llama.cpp generate recording, sample-implied peaks were 432 and 528 GB/s, 671 GB/s scaled by 9,001/14,001 and by 11,001/14,001 (bench/results/m2-1s-fixtures/). The samples don't record the memory clock, so it's inferred from those ratios. An earlier calibrate point averaged NVML's clock over each batch. It read 501 GB/s at a steady 9,001 MHz, against 432 GB/s if the peak followed the clock (bench/results/calibrations/, quick).

On the test laptop the power cap is active in nearly every run, so this applies to most results here. Until calibrate settles it, Diescope doesn't say which peak such a percentage is a share of. Reports give the clocks and the counted DRAM rate in bytes per second, so a reader can compute the share of either.

Measurement cost

Enabling the counters slows the measured work. Every report and agent record carries the cost: an estimate, or the reason there's none.

• Disabling, not pausing. Pausing sampling while the counters stay enabled saves nothing (bench/results/duty_experiment.txt). The agent disables them between duty windows, and run disables them as soon as the command ends.

• The estimates. Slowdowns were measured with diescope agent beside ResNet-50 AMP training on the test laptop, on mains power: 5 interleaved rounds of 120 s per setting after a discarded warm-up round (bench/results/cost_2026-10/; cost.rs, MEASURED). Each estimate is the 95% t-interval of the mean slowdown over the rounds. An interval that includes zero is reported as not distinguishable from run-to-run noise.

  Counters	Slowdown, 95% interval
  always on	1.5 to 4.9%
  2000/10000 ms	-1.6 to 2.3%
  2000/30000 ms	-1.0 to 1.3%
  1000/30000 ms	-0.8 to 1.5%
  2000/60000 ms, the agent's default	-0.5 to 1.5%
  off (agent running, counters off)	-2.1 to 0.7%

• When an estimate is given. The table covers one GPU model, one workload, counters every 10 ms, and mains power, with the counters run by the agent. An agent record on that GPU model, at a setting in the table, gets the estimate whatever its workload. run reports carry no estimate, with the reason. So do agent records on another GPU model, at another interval, on battery, when the power source changed, or when it couldn't be read.

• Measured on-time. Each agent record carries the seconds the counters were actually on in it (counters_on_s) beside the setting's long-run share. On-time errs toward more cost.

Calibration

diescope calibrate checks a unit's counter against work whose size Diescope knows (collect/nvidia/calibrate.rs). Each unit gets a workload that keeps it busy, run on all SMs and on fewer, with work queued so the GPU never waits on the host inside the counter window, and a step counts only if the work's own timestamps show it running for at least 99% of the counter window (MIN_COVERAGE).

calibrate runs on Blackwell only so far. On other architectures it stops and says so.

• FP32. Diescope knows the operations launched, the pipe's peak per cycle, the SMs, the clock the kernels measured, and the time. So it computes what the counter should read at every load and compares, within 1% of the expected reading and at least 0.1 point (RELATIVE_TOLERANCE, MIN_TOLERANCE_POINTS). Two more steps are reported, not judged, on Blackwell: integer multiply-add alone, which only the heavy half runs, so FP32 should read half the pipe, and multiply-add with FFMA, where FP32 should read the work at the whole pipe's rate. On GB205 the second left the halves equally busy (78.85% and 78.84%; bench/results/fp32-pipe-fixtures/calib-f12fe7c-quick.json).
• DRAM. The counter must be in proportion to the bytes moved: the peak it implies must agree within 3% across loads at one steady memory clock (IMPLIED_PEAK_SPREAD). The byte counter's total must be within 5% of the bytes launched (BYTES_RATIO_TOLERANCE).
• Ceilings. The reading at full load decides too: FP32 at least 90% (SM_PIPE_CEILING_PCT), DRAM at least 80% (MEMORY_CEILING_PCT).

On GB205, FP32 read its work at every load in one quick check on method 3: FFMA at 46, 23, 11, and 1 SMs, integer multiply-add at 49.93% against 49.93% implied, and the mixed step at 78.84% against 78.82% (bench/results/fp32-pipe-fixtures/calib-f12fe7c-quick.json). DRAM as bytes moved passed in both runs that read it that way, quick-14 and that check. Earlier runs read dram__throughput and passed in all but one, which was inconclusive (bench/results/calibrations/). None of these runs is a calibration. One is a full-length run from before method 2, and the rest are quick setup checks.

How a report finds its calibration

run --report looks for calibration files where calibrate writes them, $XDG_DATA_HOME/diescope/calibrations or ~/.local/share/diescope/calibrations, or at run --calibration PATH, a file or a folder (calibration.rs, lookup). A path that doesn't exist is an error.

• What fits. A calibration file, schema 3 or later, for the same GPU model, driver, and method, that lists how it read each unit it checked. Each of those must be a unit of the report, read the same way. A verdict counts only for a unit whose recipe the file lists.
• Which file. Only regular .json files up to 16 MB are read (MAX_FILE_BYTES). Among the files that fit, a full calibration wins over a quick setup check, then the newest by the date inside it, whatever the file is called.
• When none fits. In a folder, files that don't fit are passed over, and the report says there's no calibration file for this GPU model, driver, and method. A file named with --calibration says why it wasn't taken: that it couldn't be read, and why, or the first way it doesn't fit, such as "driver 610.57.04 vs 611.01", the file's value first.

The report's status is:

• validated: a full calibration in which every unit in the report passed;
• partial: a full calibration in which some units passed and the rest failed or have no check;
• unvalidated: anything else, including a quick setup check or no file.

The report names every unit's verdict (calibration.units) and, where the file gives one, the reading each unit reached at full load (calibration.ceiling_pct), and its calibration line names the units by verdict. It identifies the file by name and SHA-256, with the file's schema, method, commit, date, and whether it was a quick check. calibrate checks only FP32 and DRAM today, so a full calibration gives "partial" at best.

Not yet checked

calibrate has checked FP32 and DRAM on GB205. No other unit has a calibrate check yet, on any chip. Reports flag a unit that reads a stand-in counter or a different measure (results.unit_flags, and chip.unit_flags in agent records). No flag means the counter reads the unit directly, not that it's validated.

No flag on Blackwell means a counter reads a direct measure, not that it's validated. The checks behind Blackwell's FP32 and DRAM are quick runs on one GB205. Other Blackwell chips haven't been run. Blackwell's Load/store stays flagged. Its note gives the drain study's GB205 figures at 1 s and 100 ms drains, and says the agent's loss follows its poll interval.

Known limits

• One machine. Every recorded result is from one GPU model on one laptop.
• Shared GPUs. The counters cover the whole device. With MIG, MPS, or time-slicing, chip numbers can't be split per process or instance. Not tested. Reports count other processes holding the GPU in hygiene: compute, graphics, and holders that exited before they could be traced. Each count is the most seen in one read, about once a second, and null when NVIDIA's process list couldn't be read. A holder shorter than a second can be missed.
• Duty-cycle bias. With duty cycling, chip numbers come only from the sampled windows. How far that is from the whole period is not measured. Windows aligned to wall-clock time can alias with work that repeats on the same period.
• Power limit drift. On the test laptop, on mains power, the enforced power limit ranged from 90 to 115 W within and between the cost runs (bench/results/cost_2026-10/). A different limit changes clocks and so the numbers. Reports record the limit's range over each run.
• Counter decoder crashes. NVIDIA's counter decoder crashed twice on the test laptop before the counters moved to a child process, with no known trigger. Now a crash ends the counters, not Diescope. run writes the report from the samples sent before it, with a note saying why they stop. The chip coverage gate marks it partial when the samples cover under 90% of the run. The agent restarts the sampler with a backoff. Any other counter error during run ends with GPU-Util only and no report. The child path is tested with a sampler that crashes on purpose.
• Wall-clock jumps. When the wall clock steps by more than 0.1 s against the monotonic clock (clocks.rs, MAX_CLOCK_SKEW_NS) or the machine suspends, the agent restarts the record and says so in the next one, and run lists each jump in hygiene.clock_jumps with a note that the numbers across it may be wrong. How counter timestamps behave across a real clock step is untested.
• Chosen thresholds. The 90% coverage, availability, and chip coverage rules and the 1% busy rule were chosen, not measured.

Versioning

The method version changes when anything changes that can change a reported number:

• the counter behind a unit, how two counters combine, or the set of units;
• how samples are dropped, weighted, combined, or drained;
• how GPU-Util is read, booked to time, or clipped;
• a threshold: the coverage and availability rules, the 1% busy rule, the noise floor, the GPU-Util fallback timings, the duty minimums;
• the rounding rules;
• the cost table.

Changes to wording, layout, or file formats don't. Method 2 changed DRAM to bytes moved and FP32 to the busier half of the FMA pipe. Method 3 changed FP32 to the FMA pipe as a whole on Blackwell, and to the mean of its two halves on Ampere and Ada. Reports and agent records carry the method version as method, and the format version as schema. Schema 3 adds run_type (single, one round of a repeated bench run, or sustained), the workload block, die_util_algorithm, expected baselines per workload and architecture, and hygiene. Every expected value is null until a measured result backs it, and all are null today. Reports from earlier methods stay labeled with theirs. Every result is labeled with its run type: a single run, one round of a repeated set, or a sustained run. Calibration status is labeled separately.