Skip to content
Docs menu

Observability with Prometheus and Grafana

Run Prometheus and a provisioned Grafana dashboard next to the orchestrator with the observability compose profile, or scrape the orchestrator's /metrics from your own monitoring.

View .md

The orchestrator serves Prometheus metrics at GET /metrics (no auth, the same port as /v1). A venom checkout ships a Prometheus scrape config and a Grafana dashboard for it under deploy/observability/, wired into compose behind the observability profile.

Start it

make up SERVICES="orchestrator prometheus grafana"
open http://127.0.0.1:3001        # Grafana, the "Venom orchestrator" dashboard
open http://127.0.0.1:9090        # Prometheus

Or with compose directly: docker compose --profile observability up -d. Both bind to 127.0.0.1 only. Grafana opens anonymously as a Viewer on the provisioned dashboard; sign in as admin with GRAFANA_ADMIN_PASSWORD to edit.

Settings

Variable Default What it does
PROMETHEUS_PORT 9090 Prometheus's host port.
PROMETHEUS_RETENTION 15d How long Prometheus keeps samples (--storage.tsdb.retention.time).
GRAFANA_PORT 3001 Grafana's host port.
GRAFANA_ADMIN_PASSWORD admin Grafana's admin password. Change it before you publish Grafana anywhere.

Data lives in the prometheus-data and grafana-data volumes; make down keeps them.

Grafana runs with usage reporting, update checks, plugin update checks, the news feed and plugin preinstalls turned off, so it makes no outbound calls.

What is scraped

deploy/observability/prometheus.yml scrapes every 5 seconds:

Job Target Notes
venom-orchestrator orchestrator:8787/metrics Every orchestrator metric.
vllm vllm:8000/metrics vLLM's prefix-cache counters for the "Engine prefix cache" panel. Shows as down while the gpu profile is off; that is expected.

SGLang has no compose service. For one on the host started with --enable-metrics, add a job:

  - job_name: sglang
    static_configs:
      - targets: ["host.docker.internal:30000"]

The dashboard

The provisioned "Venom orchestrator" dashboard (Grafana's home dashboard, read from deploy/observability/grafana/dashboards/venom.json) has: requests per second, error rate, p95 latency, p95 time to first token, cache hit ratio, active streams, request rate by model, error rate by status, latency and TTFT p50/p95, tokens per second, the upstream circuit breaker, upstream health and in-flight requests, and the vLLM engine prefix cache. The data source is provisioned as Prometheus (uid venom-prometheus) and is not editable from the UI.

Your own monitoring

Point any Prometheus-compatible scraper at the orchestrator:

scrape_configs:
  - job_name: venom-orchestrator
    metrics_path: /metrics
    static_configs:
      - targets: ["venom.example.com:8787"]

Every metric and its labels are listed in Orchestrator operations. Label cardinality is bounded by your config: unknown model names are recorded as (unknown). On Kubernetes the same endpoint is on the venom-orchestrator Service (Kubernetes). Traces go to your own OpenTelemetry collector with OTEL_EXPORTER_OTLP_ENDPOINT (Orchestrator operations).