hyprcage: a virtual screen for agents
When an agent tests a web page, it opens a browser. If that browser opens on my desktop, it steals my focus, moves my cursor and covers my work. I wanted the agent to have a screen of its own.
What hyprcage does
hyprcage gives an agent a virtual screen
on Hyprland. Each screen is a small cage compositor with its own output and its
own seat. Hyprland never learns that it exists. My monitors, workspaces, focus
and cursor stay untouched.
I can watch through a mirror window on a spare workspace. Clicking in the mirror does nothing. The agent works on its own.
I forked the project and added four things.
Read the screen as text
A screenshot costs about 1300 tokens. Text is cheaper. snapshot lists the
elements on a screen, one per line, with a ref, a role, a name and coordinates.
act clicks, types or scrolls on a ref. find waits for an element.
The text comes from the first source that answers: the DevTools protocol, then AT-SPI, then OCR. The agent takes a screenshot only when the text does not explain the screen.
Control of the real desktop
With screen: "desktop", the same tools work on my own desktop. Actions try
AT-SPI first, which needs no cursor and no focus change. There are also tools to
move a window, switch a workspace, and read my notifications. Every desktop tool
refuses while the session is locked.
Device emulation
devtools_emulate makes a page a phone or a tablet. It sets the viewport, the
pixel ratio, touch, the user agent, a network preset, CPU slowdown and the color
scheme.
Recording
record_start records a screen to a video. I added:
- a crop to the browser viewport, so tabs and the address bar do not show;
- an idle cut that drops pauses longer than 1 second. A 39 second demo became a 9.4 second video;
- a pointer glide and a click ring, so a viewer can follow the cursor.
Still open
- Thunar leaves the AT-SPI bus after its Create Folder dialog opens.
- One browser screen costs about 720 MB, so I cap a session at 8 screens.
- My next idea is OS-level tools, but a plain shell covers most of that. I would add them only when I hit a real gap.