Skip to content
Docs menu

The public mesh

Host a public node that serves strangers' requests, or route your requests to public nodes, and how signed receipts, sampled audits, canaries and witnessed QoS keep nodes honest.

View .md

On the public mesh anyone can host a daemon, and orgs that allow it can route requests to those daemons. The operator of a public node is a stranger who could serve a cheaper model, edit prompts, inflate token counts or fake latency, and the daemon's code is open, so no check can rest on a secret the daemon sees. The cloud therefore verifies instead of trusting: every public job carries a signed receipt, is witnessed by the cloud, and may be replayed by an auditor on reference hardware. The normative contract is docs/public-mesh.md; the decision record is ADR 0023.

Scope

  • A public daemon is one enrolled with a join token from a public-pool key group (the API calls it a team). There is no identity check: reputation is attached to the daemon's key, and a new key always starts in probation.
  • A public request is one routed to a public daemon. Only /v1/chat/completions and /v1/completions are public, streamed or not. Embeddings, Responses, decision models and remote venom serve sessions stay on private daemons (the org's own).
  • Public requests are relayed only: the cloud carries every byte, so it can time the job and check the receipt against what the client received. No direct tickets are issued for public daemons.
  • Private daemons (an org's own) are not affected by anything on this page.

Use public nodes

A request can go to a public daemon only when one of these allows it:

Setting Where Effect
min_trust_class: public on a client key console Keys, or the admin API (team is the default) the key may be routed to public daemons as well as its key group's own
An org routing policy with trusted console Routing; nodes allowlisted from the Nodes page only allowlisted public nodes, after own daemons
An org routing policy with public or managed console Routing the whole public pool (managed adds the cloud's providers)

Under an org routing policy the policy decides and the key's min_trust_class is not read. The console's Nodes page lists public nodes with their witnessed quality and trust state.

Public routing refuses a request that cannot be verified, with the reason public_params, and it then goes only to private daemons when there are any:

  • frequency_penalty, presence_penalty or repetition_penalty other than 0 or 1;
  • logit_bias, min_p, n above 1, best_of;
  • top_k above 64 or -1 (absent is fine; the cloud uses 64);
  • seed (the per-request nonce replaces it).

A prompt that looks like it contains a secret (an sk-…, ghp_…, github_pat_…, AKIA… or xox… token, a PEM private key, or a venom vck_/vak_/vjt_ key) is never sent to a public daemon (secret_in_prompt). Public daemons see the key, tenant and key group as pub, and a session id only as an HMAC, so they learn nothing about which org sent a request.

Host a public node

You need a vLLM or SGLang engine with the venom-sampler/1 logits processor, or llama.cpp behind the venom-llamacpp shim, serving the exact artifact of a registry entry. Ollama is not admitted.

  1. Run the engine with the sampler. For vLLM, install the vllm-venom-sampler package into vLLM's own Python environment (the source is services/vllm-venom-sampler) and start vLLM with --max-logprobs 64. Installing the package is enough: vLLM loads its vllm.logits_processors entry point. Do not turn on speculative decoding; custom logits processors and speculative decoding cannot run together.

    vllm serve Qwen/Qwen3-8B-AWQ --max-logprobs 64 --port 8000
  2. Declare the model's provenance in config/orchestrator.yaml. The model must use a vllm or sglang upstream (the llama.cpp shim is configured as kind: vllm):

    upstreams:
      - name: vllm-local
        kind: vllm
        base_url: http://127.0.0.1:8000/v1
    models:
      - id: qwen3-8b
        upstream: vllm-local
        provenance: {registry_id: qwen3-8b-awq@1, sampler: venom-sampler/1}

    models[].provenance.registry_id names the registry entry; models[].provenance.sampler must be venom-sampler/1. See Orchestrator configuration.

  3. Join with a public-pool token exactly as for a private daemon (Connect a daemon). The cloud operator mints it for its public-pool key group. A key that was barred is never readmitted (403 daemon_barred).

  4. Pass the preflight. Before the daemon advertises registry_id and sampler for a model, it fetches the registry entry (GET /api/mesh/v1/registry/{id}) and runs its golden cases (at most 8) against your engine, locally: the requests never pass through the cloud. It checks that the prompt token ids match (prompt_ids, catches a wrong chat template or tokenizer), that enough outputs match the reference on their first 16 tokens (output, catches other weights or a sampler that ignores the nonce), and, when the engine runs on loopback and names its model path, the weight file hashes (weights). It reruns every 6 hours, on restart and when the upstream changes. A model that fails is not advertised and gets no public jobs; GET /admin/v1/local on the daemon shows the last result under mesh.preflight, with the failed check and a hint (chat template and tokenizer files, quant file, KV cache dtype, speculative decoding, plugin version). An entry without a golden set skips the preflight.

The preflight protects honest operators from being quarantined for a misconfiguration. It is not a security boundary: the audits are.

llama.cpp

llama.cpp has no logits processor a request can drive, so venom-llamacpp runs the sampler outside the engine, over a stock llama-server, and serves the vLLM-shaped surface the daemon and the auditor need:

llama-server -m Qwen3-0.6B-Q8_0.gguf --host 127.0.0.1 --port 8080 -c 8192 -np 2
venom-llamacpp --listen 127.0.0.1:8110 --upstream http://127.0.0.1:8080 --model-name qwen3-0.6b

Give the daemon a kind: vllm upstream at http://127.0.0.1:8110/v1. The upstream must be on loopback unless you pass --allow-remote-upstream. The registry entry must list a llama.cpp engine (llamacpp>=b10925); do not mix a vLLM daemon and a llama.cpp reference in one entry. Expect roughly 55 to 70% of llama-server's own throughput.

How nodes are checked

Step What happens
Sampling Every public job gets a secret 16-byte nonce. The engine picks each token with venom-sampler/1: a Gumbel-max draw over the top-k/top-p candidates with noise from HMAC-SHA256(nonce, position, token). Given the model's log-probabilities, anyone can replay the choice.
Receipt Just before the end of the job the daemon sends a receipt signed with its enrolled key: hashes of the prompt token ids, the output token ids and the streamed text, the token counts and the sampler parameters. See Provenance and receipts.
Witness The cloud checks the signature and recomputes the text hash from the bytes it relayed. A missing or wrong receipt means the job is not billed or credited and counts as a failure. It stamps time to first token on the first line that carries a real output token.
Selection Each job is selected for audit by an HMAC of its receipt under an hourly epoch secret that is published only after the epoch ends (GET /api/mesh/v1/epochs). The daemon cannot steer selection, and anyone can recompute it afterwards. The base rate is 2%, 10% in probation, ×3 with tool calls, 100% for canaries, and up to 50% for a daemon under suspicion.
Audit venom-auditor replays the job on a reference engine running the registered artifact: the prompt hash, that the streamed text is in the committed tokens, each token against the sampler's expected choice, and the token counts.
Confirmation A failed audit is rerun on a second reference; only a second failure is confirmed.
Sequential evidence Small, steady deviations (a coarser quantisation, a guessed first token) that pass single audits add up in a per-daemon CUSUM statistic; crossing the threshold quarantines the daemon.
Canaries The auditor sends requests shaped like real traffic (multi-turn, varied prompts, Poisson arrivals), at least 20 a day and 0.5% of a daemon's public jobs. Nothing in the dispatch marks them.

Trust states

State Enters when Effect
probation a public daemon enrolls at most 1% of a model's public dispatches per daemon, 10% for all probation daemons together; audited at 10%
good at least 1,000 witnessed jobs with valid receipts and 10 passed audits, no confirmed failure normal routing
quarantined one confirmed failed audit, or the evidence threshold no public traffic; listed for review
barred an admin bars it, or a second confirmed failure within 30 days excluded; the key is never readmitted

An admin returns a quarantined daemon to probation with POST /api/admin/v1/daemons/{id}/trust and {"action": "clear"}, or bars it with {"action": "bar"}. Automatic transitions only narrow routing.

Witnessed QoS

For public daemons the router scores time to first token, tokens per second and SLO misses (first token over 10 s, or a mid-stream error) from what the cloud witnessed over the last 15 minutes, not from what the daemon claims in its heartbeats. The claims are kept for comparison: a daemon whose claimed median TTFT is under half the witnessed one is flagged. A daemon sees its own figures, trust state and audit verdicts (once their epoch has ended, never which jobs were canaries) with GET /api/mesh/v1/self/qos; an admin sees the full report at GET /api/admin/v1/daemons/{id}/qos and on the console's daemon page.

Fair share

A public node serves many orgs. The cloud holds a public node's queue and releases jobs only into open slots, in weighted fair order across orgs (plan weights free 1, team 2, enterprise 4; an operator can set settings.fair_share.weight). One org holds at most three quarters of a node's slots, so a flood leaves room for others; a job held past its dispatch deadline ends 504 dispatch_timeout for the org that sent it.

Run the auditor

The public mesh's operator runs venom-auditor, a long-lived service with two loops: claim sampled audits and replay them, and send canaries.

VENOM_CLOUD_URL=https://cloud.usevenom.dev \
VENOM_CLOUD_AUDITOR_TOKEN=… \
VENOM_AUDITOR_REFERENCES="ref1=http://127.0.0.1:8001#qwen3-8b,ref2=http://127.0.0.1:8002#qwen3-8b" \
venom-auditor serve
Command What it does
venom-auditor serve (default) both loops
venom-auditor run audits only
venom-auditor canary canaries only
venom-auditor golden --reference <url[#model]> --registry <id> [--n 8] [--max-tokens 16] [--corpus dir] [--model name] build a registry entry's golden set on a reference engine and print the body for PATCH /api/admin/v1/registry/{id}; needs no cloud
Variable Default What it does
VENOM_CLOUD_URL required the cloud
VENOM_CLOUD_AUDITOR_TOKEN required the auditor token; never logged
VENOM_AUDITOR_REFERENCES required for audits comma-separated [id=]url[#model] reference engines; the second runs confirmation reruns
VENOM_AUDITOR_LISTEN 127.0.0.1:9797 its /health and /metrics; off for none
VENOM_AUDITOR_CORPUS none a directory of task.json prompts and source files for canaries
VENOM_AUDITOR_CONCURRENCY 2 audits replayed at once
VENOM_AUDITOR_POLL 10s wait when no audit is pending

A reference is a vLLM with the sampler plugin, --max-logprobs 64 and batch-invariant kernels where available, or a venom-llamacpp shim over the same GGUF for a llama.cpp entry.

Test it end to end

make public-mesh-e2e (also make smoke STAGES=public-mesh) runs the whole public mesh on loopback with no paid model and nothing beyond localhost: a local cloud, three Go daemons on a toy engine (one honest, one serving another model, one inflating token counts) and venom-auditor. It sends 200 request chains plus a canary session and asserts that honest audits pass, the cheater is confirmed, quarantined and then barred, the over-reporter's receipts fail, and claimed and witnessed figures diverge only for the cheater. It takes one to two minutes; E2E_N changes the number of chains. It sets VENOM_PUBLIC_TEST_P_AUDIT_PPM, a test-only floor under the audit rate, which must never be set on a deployment.

Limits

Not in v1: verifying a node's region, an attested trust class, payouts and held balances (holds are reputation only), direct-path public jobs, and Ollama. A daemon learns that it failed an audit once the epoch ends, though not how close it is to the evidence threshold. The public mesh hardening page lists the measured weaknesses.