---
title: Eval tools and probes
description: Pin the language servers and formatters every eval trial uses with make eval-tools, check which reasoning efforts each provider really accepts with make effort-probe, and list MCP Registry servers for the cloud catalog with make mcp-catalog-sync.
section: evals
order: 30
audience: [operator, builder, agent]
sources: [evals/tools.json, evals/src/install-tools.ts, scripts/effort-probe.mjs, scripts/mcp-catalog-sync.mjs, Makefile]
features: []
---

Three maintainer tools sit next to the eval suite. `make eval-tools` makes trials reproducible. `make effort-probe` and `make mcp-catalog-sync` check venom's tables against the outside world on demand; neither runs in tests.

## make eval-tools {#eval-tools}

Trials run with an empty temp home, so which language servers and formatters they find must not depend on what happens to be installed on the machine. The suite pins its own in `evals/tools.json`:

| Tool | Pin |
|---|---|
| typescript-language-server | 6.0.1, with `typescript` 5.9.3 next to it |
| prettier | 3.9.9 |
| gopls | v0.23.0 (`go install`) |
| gofmt | the Go toolchain's (required, not pinned) |

```bash
make eval-tools                    # install into evals/.tools/
make eval-tools ARGS=--force       # reinstall
```

It installs exact versions into `evals/.tools/` (gitignored), nothing global, and links the binaries into `evals/.tools/bin`. It does nothing when `.tools/installed.json` already matches the manifest. It ends by printing what a trial finds and exits 1 if that misses the manifest. It is the one eval step that needs the network; runs don't.

The runner puts `evals/.tools/bin` first on `PATH` and in `VENOM_TOOL_DIRS`, which the harness searches before `PATH` for language servers and formatters, so the pinned binaries win. Every result records each tool found with its path and version, and `make eval-compare` warns when two runs differ. Run with `--require-tools` to refuse a run whose tools don't match ([Running evals](/docs/evals/running.md#options)).

## make effort-probe {#effort-probe}

Providers accept different `reasoning_effort` levels, and the lists change. The probe sends one tiny chat completion (16 output tokens) per level of the ladder `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`, plus a bogus level, to each provider whose key is in the environment, and compares what worked with what venom declares: the cloud's effort adapter and the orchestrator's per-model quirks.

```bash
make effort-probe ARGS="--dry-run"                                         # the plan and estimate
make effort-probe ARGS="--provider openai,qwen --max-usd 0.25"
make effort-probe ARGS="--orchestrator http://127.0.0.1:8787 --paths go"   # through a running orchestrator
```

Each probed model gets up to three rows:

| Path | What is sent | Must |
|---|---|---|
| `raw` | The level as is, straight to the provider. | Shows which levels the provider rejects (400). |
| `adapter` | The body after the cloud's effort adapter rewrote it. | Never 400. |
| `go` | The level as is, through a running orchestrator (model `upstream/model`), so its quirks apply. | Never 400. |

A model that also accepts the bogus level ignores the field, so its raw row proves nothing.

| Option | What it does | Default |
|---|---|---|
| `--max-usd <usd>` | Refuses to start when the upper-bound estimate exceeds it, then stops at the measured spend. | `1` |
| `--dry-run` | Print the plan and estimate, send nothing. | off |
| `--provider <a,b>` | Only these providers. | every keyed one |
| `--models <p=m1,m2>` | Override a provider's model list. | built-in list |
| `--orchestrator <url>` | The orchestrator for the `go` path. | none (no `go` rows) |
| `--paths <list>` | `raw`, `adapter`, `go`. | all three |
| `--env-file <file>` | Where to read keys missing from the environment. Values are never printed. | the checkout's `.env` |
| `--out <file>` | Where the JSON report goes. | `dist/effort-probe/<time>.json` |

Exit 0 when everything matches, 1 on any mismatch, 2 on a usage or budget error. It is live: it needs keys and the network and costs a few cents. Report the cost of each run.

## make mcp-catalog-sync {#mcp-catalog-sync}

Venom Cloud keeps a curated catalog of MCP servers an org policy can offer. `make mcp-catalog-sync` lists candidates for it from the official MCP Registry: active servers a policy could carry without a secret (a streamable-HTTP remote with no required header and no URL variable; with `--packages`, also npm or PyPI stdio packages with no required environment variable or argument), minus the ones the catalog already has. It prints a table, or JSON. It never edits the catalog; each candidate is checked against the vendor's own docs and added by hand.

```bash
make mcp-catalog-sync
make mcp-catalog-sync ARGS="--search docs --packages --json"
```

| Option | What it does | Default |
|---|---|---|
| `--registry <url>` | The registry to page through. | `https://registry.modelcontextprotocol.io` |
| `--input <file>` | Read a saved registry response instead of the network. | none |
| `--catalog <file>` | The catalog to subtract. | the cloud's `catalog.json` |
| `--packages` | Include stdio packages, not only remotes. | off |
| `--search <text>` | Only servers matching the text. | all |
| `--max-pages <n>` | Pages of 100 to read. | `1000` |
| `--json` | JSON instead of a table. | off |

It is maintainer-side and on demand only: the cloud never calls a registry at runtime.

## Related

- [Running evals](/docs/evals/running.md).
- [Models and reasoning](/docs/configuration/models.md): the effort levels a session asks for.
- [Search and MCP](/docs/orchestrator/search-and-mcp.md): MCP through the orchestrator.
