CLI Reference¶

The harness has six classic commands. One config, one record, one command each.

python -m harness <campaign-directory> [--dump-inputs DIR]
python -m harness sweep [--jobs N] [--match GLOB]
python -m harness regression [--environment PATH]
python -m harness coverage
python -m harness matrix [--environment PATH]
python -m harness benign --corpus <dir> [--sample N]

Under uv, prefix with uv run:

uv run python -m harness campaigns/known-bypasses-manual

Every MCP operation is also a subcommand¶

The CLI and the MCP server expose the same operations, and tests/test_surface_parity.py enforces it: a tool added on one side without the other fails the build. The canonical list is harness/surface.py, and the two surfaces call the same implementation, so a signature change moves both.

python -m harness <operation> [--<param> ...] [--out FILE]

Each parameter is a flag named after the operation's parameter; a list takes a repeatable flag, an object takes a JSON string. The result is printed as JSON, or written to --out.

sweep, coverage, matrix and benign keep their classic spellings above; the MCP server exposes them as the tools of the same name.

python -m harness <campaign-directory>¶

Runs one campaign and writes its record.

Arguments¶

What happens before the first attempt¶

1. The behaviour validator's calibration suite runs. A failure refuses the campaign with exit 1; no bypass number is publishable from a build that cannot tell a live chain from a dead one.
2. campaign.yml is loaded and strictly validated. An unknown key is a mistake worth stopping for, not a comment.
3. The declared TrustSight version is compared against the installed one.
4. TrustSight's data and config directories are bound to campaign-local paths.
5. The database is restored, the canary is analysed, and its score is recorded.
6. One API/CLI parity check runs.

Output¶

A JSON summary on stdout; campaign name, attempt count, non-zero outcomes, and the bypass rate with its interval:

{
  "campaign": "known-bypasses-manual",
  "attempts": 8,
  "outcomes": { "behavior_lost": 3, "detected": 4, "partial_evasion": 1 },
  "bypass_rate": { "estimate": 0.0, "ci_95_wilson": [0.0, 0.434482],
                   "denominator": "attempts reaching TrustSight",
                   "denominator_value": 5,
                   "note": "lower bound (validator is conservative)" }
}

The summary omits zero-valued outcomes; record.json keeps all of them.

Files written¶

python -m harness sweep¶

Runs every campaign under campaigns/, in parallel where asked. The campaigns are independent — each binds its own database and writes only inside its own directory — so a re-baseline is a worker pool, not a loop.

Each campaign still verifies its own environment and canary before its first attempt. A campaign that fails is reported in the summary and does not stop the others; the command exits 2 if any failed, 0 otherwise.

uv run python -m harness sweep --jobs auto
# {"campaigns": 74, "attempts": 1402, "bypasses": 0, "bypassing": []}

python -m harness regression¶

Replays every bypass committed by every campaign in campaigns/ against the current environment.

Arguments¶

Behaviour¶

For each committed bypass hash: locate its diff by re-hashing the embedded diff in evidence.jsonl (never by filename), restore the database, re-analyse, and classify.

• Still UNFLAGGED → open
• A positive finding → closed, with the closing version recorded
• Fails bash -n now → unreplayable, with the reason
• Analysis stage failed (stage_degraded, the tokenizer sandbox timing out under load) → degraded, retried once serially, never counted closed
• The tool declined to answer (any other coverage gap forbids UNFLAGGED) → inconclusive, never counted closed and never stamped with a closing version. A gap is evidence about the analysis, not about the payload, so retiring a bypass on one would be the gate being wrong in the only direction it must never be.

What is re-analysed. The gate replays the PKGBUILD the recorded verdict was reached on, which evidence.jsonl carries in analysed_text beside the diff. A unified diff is not the whole recipe: reconstructing a side from its hunks drops every line outside one, so the reconstruction has no url= and no source= — the two declarations a bundle of rules measures an undeclared fetch against. Replaying it asks TrustSight a different question from the one the record answers, and the two disagreed often enough to matter. A record written before the field existed still replays, on the reconstruction, and the report's analysed_from counts say how much of the run rests on which shape:

"analysed_from": {"recorded": 36, "reconstruction": 1282}

The gate also replays the canary and one API/CLI parity check per environment, so a "closed" bypass is never an artefact of a broken harness.

HARNESS_ENV_ROOT=/dev/shm/trustsight-harness uv run python -m harness regression --jobs auto

Output¶

Of 6299 known bypasses, 6299 closed, 0 open as of 0.19.0.

and regression/report.json with the per-bypass detail.

Exit codes¶

The regression gate uses 0 and 2 only. A caller scripting it should never have to distinguish "misconfigured" from "broken" to know whether a report exists. A missing environment file is therefore exit 2, not exit 1.

python -m harness coverage¶

Maps every campaign to the rules it sets out to test, against TrustSight's own rule taxonomy, and writes coverage/report.json.

The harness is adversarial, not exhaustive. This makes the untested surface a number: of 244 rules, how many a campaign names, grouped by category, with the rules a cold analyze_text campaign cannot reach named and explained. Two kinds of unreachability are separated, because they are different facts: the adoption and composition rules need a corpus cycle or a recorded observation, and twelve more need a committed companion file, which no campaign in this suite can produce — every generator diffs one PKGBUILD against one baseline.

uv run python -m harness coverage
# 38 of 272 rules targeted by 103 campaigns; 234 untargeted, 23 with a reason and all 23 unreachable by this harness's format.

python -m harness matrix¶

Renders the hardness matrix: the shape archive's grid of (rule, shape) cells for the TrustSight version in the environment file. Each cell is open (a bypass is confirmed), closed (a patch is verified), unshaped (attempts the catalog cannot name) or, per rule with no cells, unproven.

Writes regression/matrix.json and regression/matrix.md for the current version, plus the frozen regression/cellmatrix-<version>.json/.md snapshot the matrix is diffed against between releases. The diff between two versions' matrices is the regression artifact: cells a patch closed, cells a new bypass opened, and newly named shapes. A row for a rule the pinned TrustSight does not ship is a configuration error, so the matrix can never claim a rule the instrument does not have.

uv run python -m harness matrix
# 6 hardened, 11 open, 241 unproven of 258 rules; wrote regression/cellmatrix-0.19.0.json

python -m harness benign¶

Scans a directory of benign *.diff files and reports the false-positive rate: how many a healthy update causes TrustSight to flag. This is the other half of the measurement every campaign leaves out.

It is deliberately not a campaign. A benign update carries no fetch-to-execute chain, so the behaviour validator would discard it as behavior_lost before TrustSight saw it; and an unflagged benign diff is not a bypass, so the exporter must not file it as one. This path calls trustsight.analysis.scan_diff directly, with the host's pacman answer frozen to the cold-machine value so the number is reproducible.

Arguments¶

Output¶

benign/report.json, plus a one-line summary:

15 of 186 benign diffs flagged at threshold 20 (rate 0.0806).

Against TrustSight's own 3,739-diff corpus the whole-run figure is ~7.8%, the same order as the tool's published calibration.

Exit codes¶

Whether the numbers are good news is never encoded in an exit code. See Exit Codes.