---
title: Operations - probes, metrics, local models and mesh
description: Monitor the orchestrator with health probes, Prometheus metrics and the profiler, manage local models, connect it to Venom Cloud, and replay traffic from cassettes.
section: orchestrator
order: 130
audience: [operator, agent]
sources: [docs/architecture.md, services/go-orchestrator/AGENTS.md, services/go-orchestrator/internal/server/server.go, services/go-orchestrator/internal/server/debug.go, deploy/observability]
features: [orchestrator.ops, orchestrator.admin-debug, orchestrator.admin-local, orchestrator.admin-mesh, orchestrator.admin-daemon, orchestrator.admin-heal]
---

## Health probes {#probes}

| Endpoint | Auth | Answer |
|---|---|---|
| `GET /healthz` | none | Liveness: `200 {"status":"ok"}` while the process serves. |
| `GET /readyz` | none | `200` with at least one healthy upstream; `503 {"status":"no healthy upstream"}` otherwise, or `"checking upstreams"` with `checking_upstreams` > 0 right after a start or reload while upstreams await their first probe. |

An upstream counts as unhealthy until its first probe passes. A probe that only timed out keeps a healthy upstream healthy if live traffic got a 2xx from it since the last probe, otherwise it takes two in a row; an error status or a refused connection counts at once. In Kubernetes, point the liveness probe at `/healthz` and readiness at `/readyz`.

`GET /health` and `/metrics` on port 9797 belong to the separate public-mesh auditor, not the orchestrator ([Environment variables](/docs/orchestrator/environment.md#auditor)).

## Metrics {#metrics}

`GET /metrics` serves Prometheus text format 0.0.4, no auth. From a checkout, `make up SERVICES="orchestrator prometheus grafana"` runs Prometheus (`http://localhost:9090`) and Grafana (`http://localhost:3001`) with a provisioned dashboard.

| Metric | Labels |
|---|---|
| `venom_requests_total` | `model`, `upstream`, `code` (0 = no upstream reached, 499 = client closed) |
| `venom_request_duration_seconds`, `venom_ttft_seconds` | `model` |
| `venom_tokens_total` | `model`, `type` (`prompt`, `completion`, `cached`, `cache_write`) |
| `venom_cost_usd_total` | `model` |
| `venom_cache_hit_ratio`, `venom_cache_breaks_total` | `model`, `reason` |
| `venom_wasted_tokens_total`, `venom_wasted_usd_total` | `model`, `reason` (`abandoned`, `ttft_timeout`, `client_disconnect`, `retried`) |
| `venom_cost_anomaly`, `venom_cost_anomalies_total` | `scope`, `metric` |
| `venom_response_cache_hits_total`, `venom_response_cache_stores_total` | `kind`, `how` |
| `venom_dedup_total` | `how` (`joined`, `replayed`, `lagged`, `alone`, `mismatch`) |
| `venom_tool_call_repairs_total` | `upstream`, `kind` |
| `venom_upstream_healthy`, `venom_upstream_inflight`, `venom_upstream_circuit_state` (0 closed, 1 open, 2 half open), `venom_upstream_latency_ewma_seconds` | `upstream` |
| `venom_upstream_attempts_total`, `venom_upstream_retries_skipped_total`, `venom_upstream_retry_budget_available` | `upstream`, `attempt` or `reason` |
| `venom_upstream_quota_exhausted` | `upstream`, `model` |
| `venom_upstream_requests_waiting`, `venom_upstream_kv_cache_usage` | `upstream` (engine load) |
| `venom_upstream_shaping_queue_depth`, `venom_upstream_shaping_requests_total`, `venom_upstream_shaping_rejected_total`, `venom_upstream_shaping_wait_seconds` | `upstream`, `outcome` or `reason` |
| `venom_route_target_available` | `route`, `upstream`, `model` |
| `venom_key_spend_usd`, `venom_key_budget_usd`, `venom_budget_soft_limit_crossed_total`, `venom_budget_exceeded_total` | `key`, `key_id`, `period`, `limit` |
| `venom_request_log_rows`, `venom_request_log_dropped_total`, `venom_active_streams`, `venom_uptime_seconds` | none |
| `go_goroutines`, `go_heap_objects_bytes`, `go_memory_total_bytes`, `go_gc_cycles_total`, `process_open_fds`, `process_cpu_seconds_total` | none |

`model` is the requested alias or id; unknown names are recorded as `(unknown)`, so label cardinality is bounded by the config.

## Profiler {#pprof}

Go's profiler is served behind the admin token at `/admin/v1/debug/pprof/`: the standard `net/http/pprof` paths (the `/debug/pprof/` index, `/debug/pprof/profile`, `/debug/pprof/trace`, `/debug/pprof/block`, `/debug/pprof/mutex`, `/debug/pprof/cmdline`, `/debug/pprof/symbol`, plus `heap`, `allocs`, `goroutine`) under the `/admin/v1` prefix. Block and mutex sampling are switched on only for the duration of a `?seconds=N` profile.

```bash
curl -H "Authorization: Bearer $VENOM_ADMIN_TOKEN" -o cpu.pb.gz \
  'http://127.0.0.1:8787/admin/v1/debug/pprof/profile?seconds=10'
go tool pprof -top cpu.pb.gz
```

## Tracing and logs {#tracing}

Set `OTEL_EXPORTER_OTLP_ENDPOINT` to export a span per request over OTLP/HTTP; a client that sends `traceparent` (the venom CLI does) gets the orchestrator's span joined to its trace. Spans never carry prompt or reply content. Logs go to stderr; `LOG_LEVEL` and `LOG_JSON=1` set the level and format.

## Model catalog and guardrail pages {#catalog}

- `GET /api/models/providers/{id}` (no key, not under `/v1`) serves `<VENOM_MODEL_CATALOG>/<id>.json`, a vetted model-catalog shard, with a strong `ETag` (`If-None-Match` → `304`). An unknown id or no directory is `404`.
- `GET /g/{code}` (no key) is a guardrail code's page, where a `403 guardrail_blocked` and the `x-venom-guardrail` header link. HTML, or JSON with `Accept: application/json`. Without the admin token it shows only the public summary.

## Local models {#local}

`/admin/v1/local/*` is what the console's Local models page and `venom models` drive: this machine's hardware, which models fit, and pulling, selecting and serving them. Nothing here is on the `/v1` path.

| Endpoint | What it does |
|---|---|
| `GET /admin/v1/local` | Hardware, llmfit status, backends (Ollama, vLLM: reachable, models), the current selection per tier, strain and running pulls. |
| `GET /admin/v1/local/catalog[?refresh=1]` | Recommended models for this hardware (default and fast tier) and the catalog with fit estimates. |
| `GET /admin/v1/local/suggest[?days=7&gpu_usd_per_hour=X]` | Which cloud side-role traffic (classifier, summary, reflex, workers) a local model could take, and the savings. |
| `POST /admin/v1/local/pull` | `{model, select?: "default"\|"fast"}`: start (or join) an Ollama pull; `202`. |
| `POST /admin/v1/local/select` | `{model, tier: "default"\|"fast"}`: put that model behind `venom-default` or `venom-fast` (sets `VENOM_OLLAMA_MODEL` or `VENOM_OLLAMA_FAST_MODEL` and reloads; kept in `$VENOM_DATA_DIR/local-models.json`). |
| `POST /admin/v1/local/setup` | Pull and select the recommended models in one step; `202`. |
| `POST /admin/v1/local/serve` | `{model}`: one-click vLLM in Docker (`VENOM_VLLM_IMAGE`); `202`. |
| `DELETE /admin/v1/local/serve` | Stop it; `404 not_running`. |
| `DELETE /admin/v1/local/models/{model}` | Delete an Ollama model; `409 selected` while it is selected. |
| `POST /admin/v1/local/unload` | `{model}`: free its VRAM now. |

Hardware and fit come from `llmfit` (`VENOM_LLMFIT_URL`, else `VENOM_LLMFIT_BIN`); without it the catalog still works.

## Mesh mode {#mesh}

In mesh mode the orchestrator is a Venom Cloud daemon: it dials out to the cloud, serves the cloud's jobs with its own upstreams, and keeps provider keys local. Turn it on with `VENOM_CLOUD_URL` and a credential, or from the dashboard:

```bash
orchestrator mesh join https://cloud.usevenom.dev -      # reads the vjt_… join token from stdin
orchestrator mesh status
orchestrator mesh rotate
orchestrator mesh leave
```

| Endpoint | What it does |
|---|---|
| `GET /admin/v1/mesh` | `source` (`env`, `dashboard`, `none`), `state` (`off`, `connecting`, `connected`, `revoked`, `error`), cloud URL, name, ids, region, `paused`, running `jobs` and `tokens_per_s`. |
| `POST /admin/v1/mesh/connect` | `{join_token, cloud_url?, name?, region?}`: enroll and start mesh mode without a restart (`cloud_url` defaults to `https://cloud.usevenom.dev`). `409 mesh_managed_by_env` when `VENOM_CLOUD_URL` controls it, `409 mesh_already_connected`, `502 mesh_join_failed`. |
| `POST /admin/v1/mesh/disconnect` | Drain running jobs, deregister, remove the credential. |
| `POST /admin/v1/mesh/pause`, `POST /admin/v1/mesh/resume` | Stay connected but take no new jobs (kept across restarts); `409 mesh_off` with no connection. |

The mesh variables are listed in [Environment variables](/docs/orchestrator/environment.md#mesh).

## Desktop daemon {#daemon}

When `venom-daemon` runs the orchestrator, these drive its dashboard and menu bar app (`404 not_managed` otherwise):

| Endpoint | What it does |
|---|---|
| `GET /admin/v1/daemon` | `managed`, version, OS, start-at-login state, whether Ollama is installed and where to get it. |
| `PUT /admin/v1/daemon/autostart` | `{enabled}`: add or remove the login item. |
| `GET /admin/v1/daemon/projects` | The [project folders and workspaces](/docs/deploy/projects.md) and whether `venom serve` runs: `{folders: [{path, exists}], max, serve: {running, url?, error?}, managed, workspaces?}`. |
| `POST /admin/v1/daemon/projects` | `{path}`: add a folder and restart `venom serve`. `400 invalid_project`. |
| `DELETE /admin/v1/daemon/projects?path=` | Remove a folder and restart `venom serve`. `404 project_not_found`. |
| `DELETE /admin/v1/daemon/projects?workspace=` | Delete a [workspace](/docs/deploy/projects.md#workspaces) and its files. `404 workspace_not_found`. |
| `GET /admin/v1/daemon/update` | Auto-update status: `auto_update`, `state`, current and latest version, last and next check, progress. No network. |
| `PUT /admin/v1/daemon/update` | `{auto_update}`: off by default; on, checks run about every 6 hours. |
| `POST /admin/v1/daemon/update/check` | One fetch of the releases' `SHA256SUMS`; `502 check_failed`. |
| `POST /admin/v1/daemon/update/apply` | Download, verify, wait for an idle window, swap and restart (or roll back); `202`. `409 up_to_date`, `update_busy`, `not_supported`. |
| `GET /admin/v1/heal?limit=50` | The self-heal supervisor: `state` (`ok`, `healing`, `degraded`), failing components with attempts and next try, and recent repair events. `404 not_available` without one. |

## Cassettes {#cassettes}

`VENOM_CASSETTE_MODE=record` writes every complete `200` the orchestrator relays to `$VENOM_CASSETTE_DIR/<key>.json` (status, headers, exact body bytes, plus every attempt made for it with secrets redacted). `VENOM_CASSETTE_MODE=replay` answers from those files and never reaches an upstream: nothing is spent and the request log gets a zero-cost row; a replayed answer carries `x-venom-cassette: replay`. A request with no cassette is `409 cassette_miss`. The key is a hash of the endpoint, route profile, `Accept` and the normalised body (not the client key), so cassettes are portable and can be committed for deterministic tests of an app built on venom. Failures, partial streams and audio are not recorded.

## Related

- [Environment variables](/docs/orchestrator/environment.md), [Admin API](/docs/orchestrator/admin-api.md), [Configuration reference](/docs/orchestrator/configuration.md#history).
