Desktop computers
A computer that is a full Linux desktop instead of a browser alone, with a terminal, a file manager and any app in the image. What it costs, how to run one, and what an agent can do on it.
A computer has a kind. The default, browser, is a Chromium with no screen server. A desktop computer is a small
Linux desktop: its own X display, a window manager, a visible Chromium, a terminal (xterm) and a file manager (pcmanfm).
Use it when the task is not only in a web page: a desktop app, a file dialog, a terminal beside the browser.
It lives in the same host container as browser computers, keeps the same persistent profile and work folder, and hibernates the same way. A hibernated desktop costs only disk.
Cost
A desktop computer counts 500 MiB against the daemon's memory budget, against 450 MiB for a browser computer, because it
adds the display, the window manager and a full browser window. The daemon starts from that figure and then follows what
it measures for each kind. Change the starting figures with VENOM_COMPUTERS_MEM_PER_SEAT_MIB and
VENOM_COMPUTERS_DESKTOP_MEM_PER_SEAT_MIB (environment); leave them alone unless you
have measured your own workload. Measured with ten active desktops in the image: 279 MiB each, so 500 keeps room for the apps an agent opens. Open a heavy app and the daemon follows what it sees.
Run one
Create a computer of kind desktop (the Computers app, or kind in the API). The daemon must run the
current venom-computer image; an older image refuses the kind with kind desktop is not supported.
What an agent can do
An agent on a desktop computer gets seven extra tools. They are declared once, when its session starts, and only for a desktop; a browser computer's agent does not see them, so the tool list never changes during a session.
| Tool | Does | Asks? |
|---|---|---|
computer_screenshot |
A JPEG of the whole screen. An unchanged screen is not sent twice; only the newest three stay in the conversation. | never |
computer_click x, y, button, count |
Click, right-click or double-click at a pixel of the last screenshot. | no, except in plan mode (denied) |
computer_type text |
Type into the focused window. {{secret:NAME}} types a saved secret without the agent seeing it; a secret limited to a website, or to other tools, is refused. |
no |
computer_key keys |
Press keys or chords: Return, ctrl+l, alt+Tab. |
no |
computer_scroll x, y, dx, dy |
Mouse wheel notches. | no |
computer_shell cmd |
A shell command as the computer's own user in its work folder, with the desktop as DISPLAY (so xterm & opens a window). |
Judged as bash: the same rules, hooks and approvals. |
computer_files list, read, write |
Files in the work folder, nowhere else. | Writes ask in the default mode. |
Rules for bash apply to computer_shell: a deny rule such as bash(rm -rf:*) stops it here too. Rules can also
name a tool directly, for example deny: ["computer_click"] or ask: ["computer_key"]. Prefer the browser tools for web
pages; the screen tools are for what a page cannot show.
Watching
The live view shows the whole desktop, not only the browser window, over the same connection as a browser computer (watch and control). Taking control sends your mouse and keyboard to the desktop (a click is one action, so very fast pointer movement is thinned out). There is no VNC port and no separate viewer.
How it works
Per computer, as that computer's own Linux user and in one process group: an Xvfb display (never listening on TCP, so only
that user can connect), openbox, and Chromium. The screen is read with ffmpeg's x11grab and sent as JPEG frames; input
goes through xdotool. The frame rate follows the viewer's fps (12 by default, 30 at most). Why frames and not VNC is in
ADR 0078.
For operators and app builders
The daemon tells a session that its computer is a desktop with VENOM_COMPUTER_KIND=desktop (beside the address and token
it already sets), and the harness declares the tools from that. venom-seatd has four desktop-only endpoints on the
host's internal network, each with the host token; they answer 409 for a browser computer or one that is asleep:
| Endpoint | Does |
|---|---|
POST /seats/{id}/x11/{op} |
click, move, down, up, type, key, scroll with the fields of the tool of that name. |
GET /seats/{id}/x11/{op} |
screenshot: a JPEG of the screen (quality, w to scale). |
POST /seats/{id}/exec |
{cmd, timeout_ms}: a shell as the computer's user in its work folder, with the display set. 64 KiB per stream. |
POST /seats/{id}/files |
{op: list, read or write, path}: files under the work folder; a symlink cannot lead out of it. |