Skip to content
Docs menu

Kubernetes

Run the orchestrator on Kubernetes from venom's kustomize manifests, with keys in Secrets and the config as a generated ConfigMap, optionally as a Venom Cloud daemon.

View .md

deploy/kubernetes/ in a venom checkout is a kustomize base (the orchestrator) and one overlay (the same orchestrator as a Venom Cloud daemon). These are plain manifests, not an operator or a Helm chart (ADR 0037): the orchestrator is one SQLite-backed process with one config file and a few secrets, which is what a Deployment already describes. kustomize is built into kubectl, so there is nothing else to install, and nothing in the manifests calls out anywhere.

To run Venom Cloud, its scheduler and Postgres as well, use the Helm chart. This page covers the orchestrator alone.

Install

The orchestrator image is a release artifact (venom/orchestrator:<version>, built locally by make release, Releases and signatures). Push it to a registry your cluster can pull from, then:

kubectl create namespace venom
kubectl -n venom create secret generic venom-orchestrator-keys \
  --from-literal=VENOM_API_KEY="$(openssl rand -hex 24)" \
  --from-literal=VENOM_ADMIN_TOKEN="$(openssl rand -hex 24)"
kubectl -n venom create secret generic venom-provider-keys --from-env-file=provider-keys.env   # OPENAI_API_KEY=...

cd deploy/kubernetes/base
kustomize edit set image venom/orchestrator=registry.example.com/venom/orchestrator:<version>
cd ../../..
kubectl kustomize --load-restrictor=LoadRestrictionsNone deploy/kubernetes/base | kubectl -n venom apply -f -
kubectl -n venom rollout status deploy/venom-orchestrator

The load restrictor is off because the ConfigMap is generated from config/orchestrator.yaml and config/prices.yaml, outside the kustomization directory. Without kustomize edit, the base uses the tag latest.

Clients inside the cluster use http://venom-orchestrator.venom.svc:8787/v1 with the VENOM_API_KEY you created.

What gets created

Object What
Deployment venom-orchestrator One replica, strategy Recreate. Non-root (uid 100, gid 101, the image's venom user), read-only root filesystem, every capability dropped, RuntimeDefault seccomp, no service account token. Requests 100m CPU and 128 Mi, limit 512 Mi. An emptyDir at /tmp.
Service venom-orchestrator ClusterIP on 8787: /v1/*, /admin/v1/*, /metrics, /healthz, /readyz. Nothing is exposed outside the cluster.
ConfigMap venom-orchestrator-config-<hash> orchestrator.yaml and prices.yaml, mounted read-only at /app/config. A changed file changes the name hash and rolls the pod. ${VAR} in the file expands from the container environment, as in compose.
PersistentVolumeClaim venom-orchestrator-data 5 Gi, ReadWriteOnce, at /data: keys, usage, sessions and the mesh credential.
ServiceAccount venom-orchestrator No RBAC; the pod does not talk to the Kubernetes API.

Probes. Startup (every 2 s, up to 30 tries) and liveness (every 10 s) on /healthz. Readiness on /readyz, which answers 503 until at least one upstream is healthy, so a pod with no reachable inference backend gets no traffic.

Why one replica. The stores are SQLite on a ReadWriteOnce volume; two writers would corrupt them. More capacity means more orchestrator instances (each its own release of the manifests), not more replicas. The orchestrator migrates its own database at start, so an upgrade is a new image tag.

Keys and config

Keys come from Secrets only; the manifests hold none.

Secret Keys Required
venom-orchestrator-keys VENOM_API_KEY, VENOM_ADMIN_TOKEN yes. Without a client key the orchestrator refuses to start on a non-loopback address.
venom-provider-keys any: each key becomes an environment variable (OPENAI_API_KEY, ANTHROPIC_API_KEY, OLLAMA_BASE_URL, VLLM_BASE_URL, …) no. An upstream whose key is empty is disabled.
venom-mesh join-token only for the mesh overlay.

deploy/kubernetes/secrets.example.yaml is a template with placeholder values and is part of no kustomization; prefer the kubectl create secret commands above, which never write a key to a file.

To change upstreams, point orchestrator.yaml= in base/kustomization.yaml at your own copy. In-cluster model servers are plain environment variables: put OLLAMA_BASE_URL=http://ollama.models.svc:11434/v1 (for example) in venom-provider-keys or in a patch.

Exposing it

Add your own Ingress or Gateway route to the venom-orchestrator Service, with TLS, and only after the keys above are real. The admin API is on the same port, so consider a route for /v1 only and reach /admin/v1 with kubectl port-forward:

kubectl -n venom port-forward svc/venom-orchestrator 8787:8787
curl -s -H "Authorization: Bearer $VENOM_ADMIN_TOKEN" http://127.0.0.1:8787/admin/v1/upstreams

As a Venom Cloud daemon

kubectl -n venom create secret generic venom-mesh --from-file=join-token=/dev/stdin   # paste the vjt_ token, then Ctrl-D
# set VENOM_CLOUD_URL in deploy/kubernetes/overlays/mesh/mesh-env.yaml
kubectl kustomize --load-restrictor=LoadRestrictionsNone deploy/kubernetes/overlays/mesh | kubectl -n venom apply -f -

The overlay adds VENOM_CLOUD_URL (a placeholder, https://cloud.example.com, that you edit), VENOM_MESH_NAME (the pod name) and VENOM_MESH_JOIN_TOKEN (from the optional venom-mesh Secret). The daemon dials out to the cloud and needs no inbound route. It enrolls once and keeps its credential on the data volume, so after the first start you can delete the join-token key. Provider keys stay in the cluster. The base never sets VENOM_CLOUD_URL, so mesh mode exists only where the overlay is applied.

Troubleshooting

Symptom Cause and fix
Pod never becomes ready /readyz is 503 until one upstream is healthy. Check GET /admin/v1/upstreams through a port-forward, and that model server URLs resolve from the pod.
CreateContainerConfigError The venom-orchestrator-keys Secret is missing in the namespace.
kubectl kustomize fails with a load restriction error Add --load-restrictor=LoadRestrictionsNone.
Second pod stuck in ContainerCreating Something raised replicas; the volume is ReadWriteOnce. Keep one replica.