feat(workspace): embed live GUI apps via Xvfb capture with input forwarding
authorSvjatoslav Agejenko <svjatoslav@svjatoslav.eu>
Thu, 10 Sep 2026 15:30:42 +0000 (18:30 +0300)
committerSvjatoslav Agejenko <svjatoslav@svjatoslav.eu>
Thu, 10 Sep 2026 15:30:42 +0000 (18:30 +0300)
commit7bf0044c670bafb7a46e86e6741a964ae2d1d165
tree225130ae08ceb6716409b2236fb7134c40c10b8d
parentb1914ec7f453e24bd564ba4d60566c64399d2d4b
feat(workspace): embed live GUI apps via Xvfb capture with input forwarding

Add a bridge/x11 package that runs GUI applications on a private,
per-app Xvfb display and captures the screen into a texture:
XvfbServer manages display lifecycle, X11Native reads the root window
into caller-owned ARGB pixels over JNA, and GuiAppSession owns the app
process (dbus-run-session isolation, generated Firefox profile that
disables system extensions) plus a 10 Hz capture thread that repaints
only on changed frames.

FirefoxPanel renders the captured screen on a textured rectangle and,
while focused, forwards input into the virtual display over XTEST:
clicks at the perspective-correct texture coordinate (with an 80 ms
button hold so Firefox synthesizes DOM clicks), keyboard events 1:1
through an AWT-to-X keysym table, scroll on both axes (X buttons
4-7), mouse back/forward (8/9, dispatched on press since AWT never
fires CLICKED for extra buttons), and pointer hover motion. Shift+ESC
or a middle click releases focus; focus loss releases server-side
modifiers so Shift never sticks.

Workspace now spawns a 4x4 grid of independent PTY terminals whose
mouse wheel maps to arrow keys honoring DECCKM. UnixPty pads the
posix_spawn structs to 512 bytes (the amd64 attribute block is 336)
and destroys them after spawning.

Main gains headless verification drivers: --guitest, --clicktest,
--keytest, --scrolltest, --hovertest. TODO.org collects ideas for
embodied agents in the world.
TODO.org [new file with mode: 0644]
src/main/java/eu/svjatoslav/sixth/bridge/pty/UnixPty.java
src/main/java/eu/svjatoslav/sixth/bridge/x11/AwtKeysyms.java [new file with mode: 0644]
src/main/java/eu/svjatoslav/sixth/bridge/x11/GuiAppSession.java [new file with mode: 0644]
src/main/java/eu/svjatoslav/sixth/bridge/x11/X11Native.java [new file with mode: 0644]
src/main/java/eu/svjatoslav/sixth/bridge/x11/XvfbServer.java [new file with mode: 0644]
src/main/java/eu/svjatoslav/sixth/core/Main.java
src/main/java/eu/svjatoslav/sixth/workspace/FirefoxPanel.java [new file with mode: 0644]
src/main/java/eu/svjatoslav/sixth/workspace/TerminalPanel.java
src/main/java/eu/svjatoslav/sixth/workspace/Workspace.java