---
title: Observability with Prometheus and Grafana
description: Run Prometheus and a provisioned Grafana dashboard next to the orchestrator with the observability compose profile, or scrape the orchestrator's /metrics from your own monitoring.
section: deploy
order: 30
audience: [operator]
sources: [deploy/observability/prometheus.yml, deploy/observability/grafana, docker-compose.yml]
features: []
---

The orchestrator serves Prometheus metrics at `GET /metrics` (no auth, the same port as `/v1`). A venom checkout ships a Prometheus scrape config and a Grafana dashboard for it under `deploy/observability/`, wired into compose behind the `observability` profile.

## Start it {#start}

```bash
make up SERVICES="orchestrator prometheus grafana"
open http://127.0.0.1:3001        # Grafana, the "Venom orchestrator" dashboard
open http://127.0.0.1:9090        # Prometheus
```

Or with compose directly: `docker compose --profile observability up -d`. Both bind to `127.0.0.1` only. Grafana opens anonymously as a Viewer on the provisioned dashboard; sign in as `admin` with `GRAFANA_ADMIN_PASSWORD` to edit.

## Settings {#settings}

| Variable | Default | What it does |
|---|---|---|
| `PROMETHEUS_PORT` | `9090` | Prometheus's host port. |
| `PROMETHEUS_RETENTION` | `15d` | How long Prometheus keeps samples (`--storage.tsdb.retention.time`). |
| `GRAFANA_PORT` | `3001` | Grafana's host port. |
| `GRAFANA_ADMIN_PASSWORD` | `admin` | Grafana's admin password. Change it before you publish Grafana anywhere. |

Data lives in the `prometheus-data` and `grafana-data` volumes; `make down` keeps them.

Grafana runs with usage reporting, update checks, plugin update checks, the news feed and plugin preinstalls turned off, so it makes no outbound calls.

## What is scraped {#scrape}

`deploy/observability/prometheus.yml` scrapes every 5 seconds:

| Job | Target | Notes |
|---|---|---|
| `venom-orchestrator` | `orchestrator:8787/metrics` | Every orchestrator metric. |
| `vllm` | `vllm:8000/metrics` | vLLM's prefix-cache counters for the "Engine prefix cache" panel. Shows as down while the `gpu` profile is off; that is expected. |

SGLang has no compose service. For one on the host started with `--enable-metrics`, add a job:

```yaml
  - job_name: sglang
    static_configs:
      - targets: ["host.docker.internal:30000"]
```

## The dashboard {#dashboard}

The provisioned "Venom orchestrator" dashboard (Grafana's home dashboard, read from `deploy/observability/grafana/dashboards/venom.json`) has: requests per second, error rate, p95 latency, p95 time to first token, cache hit ratio, active streams, request rate by model, error rate by status, latency and TTFT p50/p95, tokens per second, the upstream circuit breaker, upstream health and in-flight requests, and the vLLM engine prefix cache. The data source is provisioned as `Prometheus` (uid `venom-prometheus`) and is not editable from the UI.

## Your own monitoring {#own}

Point any Prometheus-compatible scraper at the orchestrator:

```yaml
scrape_configs:
  - job_name: venom-orchestrator
    metrics_path: /metrics
    static_configs:
      - targets: ["venom.example.com:8787"]
```

Every metric and its labels are listed in [Orchestrator operations](/docs/orchestrator/operations.md#metrics). Label cardinality is bounded by your config: unknown model names are recorded as `(unknown)`. On Kubernetes the same endpoint is on the `venom-orchestrator` Service ([Kubernetes](/docs/deploy/kubernetes.md)). Traces go to your own OpenTelemetry collector with `OTEL_EXPORTER_OTLP_ENDPOINT` ([Orchestrator operations](/docs/orchestrator/operations.md#tracing)).

## Related

- [Docker Compose and profiles](/docs/deploy/compose.md)
- [Orchestrator operations](/docs/orchestrator/operations.md): probes, metrics, the profiler and tracing.
