Skip to content

Give each bot its own Cua computer - #24

Open
sousavf wants to merge 18 commits into
nikuscs:mainfrom
sousavf:feature/bot-computer
Open

sousavf wants to merge 18 commits into
nikuscs:mainfrom
sousavf:feature/bot-computer

Conversation

@sousavf

@sousavf sousavf commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

What

Each bot can start its own small Linux desktop with Chromium, on demand, and work on it visually or through the browser. The owner can watch it, take over, and shut it down. This covers phases 1-3 of .agents/plans/pi-cua.md.

How it runs

  • Cua inside, Docker outside. Each computer is a container from Cua's slim ghcr.io/trycua/linux image (arm64/amd64, pinned). Orbs downloads cua-spacesd 0.12.1, verifies its SHA-256 and mounts it in; it provides screenshots, input, the accessibility tree, viewer tickets and input leases.
  • Docker Engine API over the unix socket, no CLI. It uses Docker Desktop's socket, falls back to /var/run/docker.sock, or follows DOCKER_HOST=unix://.... OrbStack, Colima or Podman's compatible socket should work too.
  • Limits per computer: 1.5 GB memory, 1 CPU, 1 GB shm, loopback-only ports, a per-bot home volume (browser profile and logins survive), and a narrowed seccomp profile so Chromium keeps its sandbox without --no-sandbox.
  • Lifecycle in a durable Pi task: running -> sleeping (docker pause) after 5 minutes idle -> off (docker stop) after 30 minutes asleep, with a 2-hour hard limit. It survives daemon restarts; deleting a bot removes its container, volume and token.

Bot tools

  • bot_computer_start, _status, _sleep, _shutdown, _read, _windows, _act (screenshot, click, type, keys, scroll, drag) and _handoff (asks the owner to take over, through the question flow).
  • bot_browser_open, _tabs, _snapshot, _act: Chromium over CDP with playwright-core.

Owner controls (web and macOS desktop)

  • A Computer section in the bot's details pane with a live thumbnail and Start, Sleep and Shut down.
  • A full view at /rooms/direct/$handle/computer with the Cua viewer, and Take over, which holds an input lease so the bot waits.
  • The daemon proxies the viewer at /computers/<botId>/. Access is through a short-lived viewer ticket that only a paired device can mint; client credentials are stripped and the viewer is sandboxed in an opaque origin.

New dependencies

  • @trycua/cua 0.10.1
  • playwright-core 1.64.1

Not included

  • iOS (phase 4)
  • time-limit settings (phase 5)
  • controlling the owner's own Mac (phase 6)

Requirements

Docker, or a Docker-API-compatible engine, must be running to start a computer. Everything else in Orbs works without it.

Checks

Rebased onto current main, which needed two merge fixes: bun.lock was regenerated, and runtime.service.ts keeps both the notes and computer extensions. bun run check and bun run lint pass (17/17, Fallow clean); the repo has no project test suite. Used on a live dev daemon: a bot's computer ran healthy in Docker Desktop.

Add a Docker container per bot running Cua's Linux image as the first
target, ahead of the owner's Mac. It starts on demand, sleeps after 5
minutes idle, shuts down after 30 minutes asleep or 2 hours after start,
and both the bot and the owner can start, sleep and shut it down. Add
browser tools over CDP next to the visual tools, and a Computer section
with a live thumbnail in the bot details on every client.
Add the computers contract and a native host that runs one Docker
container per bot from Cua's Linux image, with the pinned cua-spacesd
0.12.1 binary (sha256-checked) mounted in and a seccomp profile that
keeps Chromium's sandbox on. It talks to Docker over its socket and to
spacesd through @trycua/cua: screenshots, pointer and keyboard input,
windows, the accessibility tree, viewer tickets and thumbnails, plus
Chromium driven over CDP with playwright-core. Each container gets its
own token in the credentials table, a home volume and loopback-only
ports. DOCKER_HOST overrides the Docker socket.
One bots.computer Pi task per bot keeps the computer's time limits in a
stored sleep: sleep after 5 minutes idle, shut down after 30 minutes
asleep, and shut down 2 hours after start. Host failures retry with
backoff, and startup shuts down containers whose bot has no live task.
Bots get bot_computer_* tools (start, sleep, shut down, status, windows,
read, act, handoff) and bot_browser_* tools over CDP. Acting tools wait
while the owner has taken control; trusted reads never boot a computer.
Clearing or deleting a bot shuts its computer down or removes it.
Add the server computer domain and computers.* RPCs (status, start,
sleep, shut down, touch, release, thumbnail, viewer) and wire the host
into the daemon. /computers/<botId>/ forwards Cua's viewer, its gRPC-Web
calls, media socket and files, authorized by a short-lived viewer
ticket only a paired device can mint. Client credentials are stripped
and the viewer runs sandboxed in an opaque origin. The web server and
Vite proxy the route, including WebSocket subprotocols.
A Computer section in the bot details shows a live thumbnail, the state
with time left, and Start, Sleep and Shut down. The thumbnail opens a
large viewer of the live screen with Take over and I'm done; watching
keeps the computer awake, and closing the view hands control back.
@sousavf
sousavf requested a review from nikuscs as a code owner November 7, 2026 10:19
@nikuscs

nikuscs commented Nov 8, 2026

Copy link
Copy Markdown
Owner

Hey Vasco, big one, so I pushed a set of fixes on top, one commit each (head bcb4630). All from a review pass, all checked against the code:

  • Orphan volume sweep (0819167): deleting a bot while Docker was down left the container and the browser-profile volume (saved logins) behind. Remove no longer swallows Docker errors, and the startup sweep now fully removes computers of bots that are gone from the bot table. It only touches computers this daemon has a token for, so a dev and prod daemon sharing Docker don't wipe each other.
  • Stop on failed start (bab57d6): if readiness failed after Docker started the container, it kept running with no lifecycle task. It now shuts down and rethrows (only when it was off or asleep before).
  • Release by device (d3a247a): any paired device could end another device's takeover. Release now carries the caller's device ID; the handoff tool still releases unconditionally.
  • Renew control (e0aa28c): control ended at ticket expiry while the UI still said "controlling". The touch interval now mints a fresh control ticket before expiry, falling back to watch mode if that fails.
  • Recheck before sleep (9f92e34): a sleep picked in check still ran after new activity or a takeover. apply now recomputes the schedule first.
  • "I'm done" answers the handoff (438bad7): it only released control, so the bot kept waiting on its question.
  • Keep awake during handoff (1a6e130): the schedule now pushes sleep out while a handoff question is pending.
  • Shut down while sleeping (7022f08): sleeping computers only offered Start.
  • Tab listing is read-only (76170b0): bot_browser_tabs now only lists and is in readOnlyTools; switch/close moved to bot_browser_tab. Plan table updated.
  • Reads may auto-start (bcb4630): read tools start or wake an off computer, as the plan says.

Two things worth knowing:

  • While controlling, renewing control swaps the viewer URL, so the viewer reloads roughly every 8 minutes. Better than the bot silently taking over again, but if the viewer can take a new ticket without a reload that'd be nicer.
  • No Docker on my machine, so none of this ran against real containers (only a fake socket for the sweep and handoff timing). Could you give it a run with Docker up?

Feel free to push back on any of these if I misread how something is meant to work.

The viewer's renewal took control unconditionally, so it could take it
back from another device or from the bot after a handoff was answered in
chat. A renewal now fails with CONFLICT unless this device still holds
control, and the viewer falls back to watching.
@sousavf

sousavf commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for these, I went through each one and they all hold up. I gave it a run with Docker up: a separate daemon from this branch on its own port and data, two paired devices, real Cua containers, and a Haiku bot (through the Claude Code provider from #26, merged locally only) for the handoff.

Scenario Result
Start a computer Running in about 9s first time, 1.5 to 2.6s after
Take control, renew while holding it Works
Device B takes over, A tries to release B keeps control (d3a247a)
Sleep, shut down while sleeping, start Works (7022f08)
Delete a bot with Docker up Container, volume and folder removed
Bot removed while the daemon was down, then restart Sweep removed its container, volume, token and folder (0819167); my dev daemon's computers on the same Docker were untouched
Bot calls bot_computer_handoff, owner takes control and presses I'm done Question answered, handoff cleared, control released (438bad7)
Same, but the owner answers from the chat card The tool releases control and clears the handoff

One problem, fixed in c770796: renewing control in the viewer took control unconditionally, so after B took over, A's next renewal took it back. Likewise, after a handoff answered from the chat card, the renewal grabbed control from the bot again. Renewal now sends isRenewal, and the server answers CONFLICT unless that device still holds control; the viewer then falls back to watching, as your code already did. Both cases above now refuse the renewal.

Not tested: shutdown after a failed start (bab57d6); I couldn't make a real start fail.

Smaller points, none blocking:

  • The reload roughly every 8 minutes while controlling: if the viewer only checks its ticket on connect, the server could just extend control.until and keep the open connection, avoiding the new URL. I haven't checked that.
  • bot_browser_tabs and bot_browser_tab differ by one letter; something like bot_browser_switch_tab would be harder for a model to mix up.
  • If the handoff tool replays after a restart, handoff.until is set again while the question keeps its original expiry, so the computer stays awake a bit longer than needed. Harmless today, since both timeouts are 5 minutes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants