Verification harness · v0.2.0 · MIT

Replace the claim with a measurement

Every grade elsewhere on this site is computed from what sellers state in writing. The harness is how a fleet leaves that tier: it measures DCB — delivered collective bandwidth, the all-reduce bus bandwidth across the largest allocation you sell, under production tenancy (ASSAY-1 §3.3, §8). DCB is the number that separates a training cluster from eight GPUs sharing a name, and it is the one number a rate card cannot fake.

Status, stated plainly. v0.2.0 ships one probe of the seven in §8.1 — nccl-allreduce. No fleet holds a Verified badge yet; the first will be announced in the weekly report. The remaining probes (nccl-alltoall, csd-probe, checkpoint-burst, serving-latency, soak-72h, power-telemetry) land in order of buyer demand.

1. Download

assay_harness.py Python 3, stdlib only · sha256 f596d59a275fa847d3ab7b8e531bf2cd9e43b9cfe0628e2b26442f95deb2610c

curl -O https://computeassay.com/harness/assay_harness.py
shasum -a 256 assay_harness.py     # must match the hash above
python3 assay_harness.py --self-test

You also need NVIDIA nccl-tests built for your stack (all_reduce_perf on $PATH, or --binary), and your normal launcher (mpirun/srun) for multi-node runs.

2. Get a run nonce

A nonce binds a result to a moment. It is a value from the public drand randomness beacon (League of Entropy): unpredictable before its round time, publicly verifiable after — so a result carrying it provably started after that moment, and anyone can check it without trusting us. Valid for 72 hours; a result that starts before its nonce round is rejected. Without one, a run ingests as self-reported, never Verified. The harness fetches one itself with --fetch-nonce, or issue one here:

3. Run it on the fleet you actually sell

# single node, 8 GPUs (measures the scale-up domain: NVLink/NVSwitch)
python3 assay_harness.py --gpus 8 --fetch-nonce

# multi-node — the run that matters for training-class grades (measures the scale-out fabric)
python3 assay_harness.py --launcher "mpirun -np 64 -hostfile hosts.txt" --gpus-per-node 1 \
    --fetch-nonce --tenants 3 --allocation "64 GPU / 8 node block" --max-sellable \
    --line-rate-gbps 400

# already ran all_reduce_perf yourself? parse the log
python3 assay_harness.py --from-output run.log --nonce <NONCE>

Flag Why it exists (§8.2)

--allocation, --max-sellable The allocation must equal the largest unit you sell as one block. A golden slice measures a different product.

--tenants Concurrent tenants on the fabric during the run. Production tenancy is the condition; a quiet-window run is recorded as such.

--line-rate-gbps Your published per-GPU scale-out line rate. The output reports delivered ÷ line rate: ≥ 0.9 excellent, 0.5–0.9 real but note the gap, < 0.5 a red flag. Single-node runs measure NVLink, so leave it unset there.

--sign GPG-sign the output. Operator-submitted results are signed by the operator; the registry verifies the signature, the nonce and the disclosures — not the metal — and says so on the certificate.

Flag	Why it exists (§8.2)
`--allocation`, `--max-sellable`	The allocation must equal the largest unit you sell as one block. A golden slice measures a different product.
`--tenants`	Concurrent tenants on the fabric during the run. Production tenancy is the condition; a quiet-window run is recorded as such.
`--line-rate-gbps`	Your published per-GPU scale-out line rate. The output reports delivered ÷ line rate: ≥ 0.9 excellent, 0.5–0.9 real but note the gap, < 0.5 a red flag. Single-node runs measure NVLink, so leave it unset there.
`--sign`	GPG-sign the output. Operator-submitted results are signed by the operator; the registry verifies the signature, the nonce and the disclosures — not the metal — and says so on the certificate.

Output: assay-probe-<timestamp>.json — busbw at 1 GiB and the 16 MiB–1 GiB curve, environment capture (GPU model, driver, topology matrix, IB devices), the §8.2 disclosures, the nonce, and a SHA-256 self-hash.

Already ran ClusterMAX or NVIDIA's readiness checks? Attach them

Acceptance tests already exist, and a buyer shouldn't have to pick one. Since v0.2.0 the harness takes third-party outputs as acceptance evidence: each file is SHA-256 hashed, summarised and bound to your run nonce, so a counterparty can check later that the report they're shown is the one you bundled.

# attach evidence to a probe run
python3 assay_harness.py --launcher "mpirun -np 64 -hostfile hosts.txt" --gpus-per-node 1 --fetch-nonce \
    --evidence nvidia-cre:cre-report.json --evidence clustermax-audit:audit-output.json

# or bundle evidence alone, with no probe
python3 assay_harness.py --evidence-only --fetch-nonce \
    --evidence clustermax-audit:audit-output.json --evidence nccl-log:alltoall.log

Type What it is What the harness reads from it

clustermax-audit Output of SemiAnalysis's free ClusterMAX audit tool Hash; counts of pass/fail status values if JSON, otherwise lines mentioning failure

nvidia-cre NVIDIA Cluster Readiness Engine certification report (burn-in, NCCL, failure detection) Hash; counts of pass/fail status values

nccl-log Raw nccl-tests output (all-reduce or all-to-all) Hash; bus bandwidth at the largest message size

other Burn-in logs, provider acceptance reports Hash only

Type	What it is	What the harness reads from it
`clustermax-audit`	Output of SemiAnalysis's free ClusterMAX audit tool	Hash; counts of pass/fail status values if JSON, otherwise lines mentioning failure
`nvidia-cre`	NVIDIA Cluster Readiness Engine certification report (burn-in, NCCL, failure detection)	Hash; counts of pass/fail status values
`nccl-log`	Raw nccl-tests output (all-reduce or all-to-all)	Hash; bus bandwidth at the largest message size
`other`	Burn-in logs, provider acceptance reports	Hash only

What this does and does not mean: the summary is a convenience, and review always reads the original file. The nonce proves when the evidence was bundled, not when it was generated, so under ASSAY-TT-1 it must be produced inside the acceptance window. An evidence bundle on its own never earns a Verified DCB grade; that still takes the nccl-allreduce probe. The MCP tool grade_contract counts a ClusterMAX audit or NVIDIA readiness report as independent acceptance evidence.

4. Submit

Reply to any issue of the Delivered Compute Report with the JSON (and .asc) attached and the word "harness". At ingestion the registry checks: the nonce against the beacon round and its 72-hour window (/api/nonce?verify= — anyone can run the same check), that the run started after the nonce was issued, the §8.2 disclosures against your published specs, and plausibility of the curve. A passing run becomes a Tier-3 measured grade with an assay certificate — source class operator-submitted (you ran it) or first-party-assay (we rented and ran it) — that expires 90 days after its window unless re-run. Staleness is a defined event, not a silent condition.

The founding cohort

The first ten fleets are verified free: we run the probe with your team, you keep the raw output, and the teardown publishes whether it flatters you or not. After that, verification is operator-funded — the credential has to mean something before it carries a price. Operator details.

What a measured grade does and does not attest

It attests what was measured, under which disclosed conditions, when, and signed by whom. It does not attest the fleet in general (n = 1: a certificate covers its window) and, for operator-submitted runs, it does not prove the run was not scheduled into a quiet window — repeated sampling is the defence, and seeded, non-recognisable workloads and hardware attestation are staged for when a grade moves contract money (§8.3, §10.2).

Source: assay_harness.py (this page's copy is canonical for v0.2.0) · spec repository github.com/ComputeAssay/spec · forecast-ledger verifier verify_ledger.py.