From: Svjatoslav Agejenko Date: Sun, 20 Sep 2026 14:27:46 +0000 (+0300) Subject: feat: virtual 3D workspace with captured apps, terminals and editors X-Git-Url: http://www2.svjatoslav.eu/gitweb/?a=commitdiff_plain;h=0697f7918b02134d2947120ca3139545eed5b3ba;p=aukio.git feat: virtual 3D workspace with captured apps, terminals and editors The Aukio desktop: ordinary Linux GUI applications captured live over X11, PTY terminals and rich text editors placed as interactive objects in a flyable 3D world built on the aukio-3d software rasterizer. Keyboard, mouse and scroll input is routed into the captured apps (including texture-space click forwarding); panels are placed, focused and managed from the 3D view. Includes launcher tooling and user documentation. --- 0697f7918b02134d2947120ca3139545eed5b3ba diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..096db16 --- /dev/null +++ b/.gitignore @@ -0,0 +1,11 @@ +/.idea/ +/.classpath +/.project +/test.byar +/target/ +/bin +/src/main/resources/rebel.xml +/aukio.iml +/.settings/ +/*.iml +*.html \ No newline at end of file diff --git a/COPYING b/COPYING new file mode 100644 index 0000000..0e259d4 --- /dev/null +++ b/COPYING @@ -0,0 +1,121 @@ +Creative Commons Legal Code + +CC0 1.0 Universal + + CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE + LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN + ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS + INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES + REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS + PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM + THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED + HEREUNDER. + +Statement of Purpose + +The laws of most jurisdictions throughout the world automatically confer +exclusive Copyright and Related Rights (defined below) upon the creator +and subsequent owner(s) (each and all, an "owner") of an original work of +authorship and/or a database (each, a "Work"). + +Certain owners wish to permanently relinquish those rights to a Work for +the purpose of contributing to a commons of creative, cultural and +scientific works ("Commons") that the public can reliably and without fear +of later claims of infringement build upon, modify, incorporate in other +works, reuse and redistribute as freely as possible in any form whatsoever +and for any purposes, including without limitation commercial purposes. +These owners may contribute to the Commons to promote the ideal of a free +culture and the further production of creative, cultural and scientific +works, or to gain reputation or greater distribution for their Work in +part through the use and efforts of others. + +For these and/or other purposes and motivations, and without any +expectation of additional consideration or compensation, the person +associating CC0 with a Work (the "Affirmer"), to the extent that he or she +is an owner of Copyright and Related Rights in the Work, voluntarily +elects to apply CC0 to the Work and publicly distribute the Work under its +terms, with knowledge of his or her Copyright and Related Rights in the +Work and the meaning and intended legal effect of CC0 on those rights. + +1. Copyright and Related Rights. A Work made available under CC0 may be +protected by copyright and related or neighboring rights ("Copyright and +Related Rights"). Copyright and Related Rights include, but are not +limited to, the following: + + i. the right to reproduce, adapt, distribute, perform, display, + communicate, and translate a Work; + ii. moral rights retained by the original author(s) and/or performer(s); +iii. publicity and privacy rights pertaining to a person's image or + likeness depicted in a Work; + iv. rights protecting against unfair competition in regards to a Work, + subject to the limitations in paragraph 4(a), below; + v. rights protecting the extraction, dissemination, use and reuse of data + in a Work; + vi. database rights (such as those arising under Directive 96/9/EC of the + European Parliament and of the Council of 11 March 1996 on the legal + protection of databases, and under any national implementation + thereof, including any amended or successor version of such + directive); and +vii. other similar, equivalent or corresponding rights throughout the + world based on applicable law or treaty, and any national + implementations thereof. + +2. Waiver. To the greatest extent permitted by, but not in contravention +of, applicable law, Affirmer hereby overtly, fully, permanently, +irrevocably and unconditionally waives, abandons, and surrenders all of +Affirmer's Copyright and Related Rights and associated claims and causes +of action, whether now known or unknown (including existing as well as +future claims and causes of action), in the Work (i) in all territories +worldwide, (ii) for the maximum duration provided by applicable law or +treaty (including future time extensions), (iii) in any current or future +medium and for any number of copies, and (iv) for any purpose whatsoever, +including without limitation commercial, advertising or promotional +purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each +member of the public at large and to the detriment of Affirmer's heirs and +successors, fully intending that such Waiver shall not be subject to +revocation, rescission, cancellation, termination, or any other legal or +equitable action to disrupt the quiet enjoyment of the Work by the public +as contemplated by Affirmer's express Statement of Purpose. + +3. Public License Fallback. Should any part of the Waiver for any reason +be judged legally invalid or ineffective under applicable law, then the +Waiver shall be preserved to the maximum extent permitted taking into +account Affirmer's express Statement of Purpose. In addition, to the +extent the Waiver is so judged Affirmer hereby grants to each affected +person a royalty-free, non transferable, non sublicensable, non exclusive, +irrevocable and unconditional license to exercise Affirmer's Copyright and +Related Rights in the Work (i) in all territories worldwide, (ii) for the +maximum duration provided by applicable law or treaty (including future +time extensions), (iii) in any current or future medium and for any number +of copies, and (iv) for any purpose whatsoever, including without +limitation commercial, advertising or promotional purposes (the +"License"). The License shall be deemed effective as of the date CC0 was +applied by Affirmer to the Work. Should any part of the License for any +reason be judged legally invalid or ineffective under applicable law, such +partial invalidity or ineffectiveness shall not invalidate the remainder +of the License, and in such case Affirmer hereby affirms that he or she +will not (i) exercise any of his or her remaining Copyright and Related +Rights in the Work or (ii) assert any associated claims and causes of +action with respect to the Work, in either case contrary to Affirmer's +express Statement of Purpose. + +4. Limitations and Disclaimers. + + a. No trademark or patent rights held by Affirmer are waived, abandoned, + surrendered, licensed or otherwise affected by this document. + b. Affirmer offers the Work as-is and makes no representations or + warranties of any kind concerning the Work, express, implied, + statutory or otherwise, including without limitation warranties of + title, merchantability, fitness for a particular purpose, non + infringement, or the absence of latent or other defects, accuracy, or + the present or absence of errors, whether or not discoverable, all to + the greatest extent permissible under applicable law. + c. Affirmer disclaims responsibility for clearing rights of other persons + that may apply to the Work or any use thereof, including without + limitation any person's Copyright and Related Rights in the Work. + Further, Affirmer disclaims responsibility for obtaining any necessary + consents, permissions or other rights required for any use of the + Work. + d. Affirmer understands and acknowledges that Creative Commons is not a + party to this document and has no duty or obligation with respect to + this CC0 or use of the Work. diff --git a/Documentation/index.org b/Documentation/index.org new file mode 100644 index 0000000..5611539 --- /dev/null +++ b/Documentation/index.org @@ -0,0 +1,200 @@ +#+SETUPFILE: ~/.emacs.d/org-styles/html/darksun.theme +#+TITLE: Aukio — System for Data Storage, Computation, Exploration and Interaction +#+LANGUAGE: en +#+LATEX_HEADER: \usepackage[margin=1.0in]{geometry} +#+LATEX_HEADER: \usepackage{parskip} +#+LATEX_HEADER: \usepackage[none]{hyphenat} + +#+OPTIONS: H:20 num:20 +#+OPTIONS: author:nil + +* Vision + :PROPERTIES: + :CUSTOM_ID: vision + :ID: 1f4e1c17-d25f-4d92-aa9b-5785f1d86f4f + :END: + +: A tool to amplify human ability + +The goal is a /bicycle for the mind/ — a powerful, extensible, hackable, +general-purpose computing environment for working with knowledge. + +The system is built around the following priorities: + ++ *Knowledge-first.* Data and insights should be easy to discover, + understand, manipulate, transform and visualize. + ++ *Intuitive, visual, real-time, 3D-first interface.* + + #+BEGIN_QUOTE + "Virtual reality holds the key to the evolution of the human mind." + — Dr. Lawrence Angelo, /The Lawnmower Man/ (1992) + #+END_QUOTE + + + +** Extensible, Programmable Computing Environment — An Example + :PROPERTIES: + :CUSTOM_ID: extensible-programmable-computing-environment + :ID: c19c5a3b-dfb0-4f7f-961c-a387b925669f + :END: + +[[https://www.johndcook.com/blog/2008/04/27/one-program-to-rule-them-all/][GNU Emacs]] is perhaps the best existing example of this philosophy. At +its core, Emacs is a text editor built on top of a Lisp runtime. Data +storage and computation can be expressed in [[https://www.defmacro.org/ramblings/lisp.html][Lisp]], which is itself a +programmable programming language — new paradigms and domain-specific +languages can be added dynamically. Text buffers serve as building +blocks for arbitrary user interfaces. The result is an environment +that can be adapted and extended to fit virtually any problem domain. + +** Architecture and Components + :PROPERTIES: + :CUSTOM_ID: architecture-and-components + :ID: 52dbbf4c-2ef4-42a6-8331-ad006b6a52ae + :END: + ++ [[https://www3.svjatoslav.eu/projects/aukio/][Aukio]] — Parent project and the product: a spatial computing + environment (3D virtual workspace). + + [[https://www3.svjatoslav.eu/projects/aukio-data/][Aukio Data]] — Data storage and computation engine. (Very early + stage, nothing to see yet) + + [[https://www3.svjatoslav.eu/projects/aukio-3d/][Aukio 3D]] — Real-time 3D engine for user interface and data + visualization. (Working and usable, fast evolving) + + [[https://www3.svjatoslav.eu/projects/aukio-3d-demos/][Aukio 3D Demos]] — Demonstration scenes showcasing the + capabilities of the Aukio 3D engine. (Many demos, in good shape) + +The system is far from complete — the scope is large and available +time is limited. + +** Roles and boundaries + ++ *Aukio 3D* is a reusable library. Anyone can take it to build games + or visualizations. It keeps only generic mechanisms: the renderer, + scene graph, camera, input primitives, and the widget layer + (~GuiComponent~, ~TextCanvas~, ~TextEditComponent~) that games can + also use (in-game consoles, interactive screens). ++ *Aukio* is the concrete product built on that library: the virtual + workspace you actually work in. Application-integration bridges + (PTY/terminal, future VNC client, window capture) live here. + Exception to the "no native code in the engine" rule: XR glasses + head tracking (RayNeo IMU over hidraw, ~eu.svjatoslav.aukio.e3d.gui.headtrack~) + lives in the ENGINE, so every app and demo gets look-around head + tracking automatically when glasses are plugged in — including + hot-plug at runtime. ++ *Aukio 3D Demos* remains the capability gallery and regression + harness. Workspace features are prototyped as demos, but the real + thing lives in Aukio. + +** Virtual workspace + +Aukio is a spatial computing environment: a persistent 3D world where +every program you use — text editor, terminal, browser, virtual +machine, data view — is a virtual object you place, walk to, and work +at. It replaces window-switching with spatial memory: related tools +live near each other, and your body remembers where things are. + +Roadmap for external applications, from cheapest to most general: + ++ *Virtual machines* — talk VNC/SPICE to the guest display instead of + scraping a window; the protocol provides both framebuffer updates + and input injection. ++ *Arbitrary Linux applications* (browser, etc.) — run the app on a + dedicated ~Xvfb~ display, capture pixels via ~XShmGetImage~, inject + input via XTEST; the app lives only inside the 3D world. + +* Ideas +:PROPERTIES: +:CUSTOM_ID: ideas +:END: + +** Molecular dynamics integration +:PROPERTIES: +:CUSTOM_ID: molecular-dynamics-integration +:END: + +Integrate with [[https://www.gromacs.org/][GROMACS]] or a similar molecular dynamics simulator. + +Users could place atoms in molecular dynamics models and manipulate +them directly with hand gestures (in VR) or mouse drag-and-drop +(desktop), seeing immediate force-based reactions. in 3D, the +simulation becomes a tangible object you "hold" and adjust. + +Teams in VR could jointly manipulate a protein model while viewing +real-time simulation data, with each member's avatar contributing to +the workspace. + +** Emacs client integration +:PROPERTIES: +:CUSTOM_ID: emacs-client-integration +:END: + +Allow Aukio to act as an Emacs client, connecting to a running Emacs +instance. Text editors could then be spawned as objects in 3D space, +each backed by the live Emacs process. This would make all existing +Emacs plugins and functionality available inside the Aukio environment. + +** Document represented as geometry + +Text editors would be replaced by floating 3D "cubes" where code +structure is visualized spatially - functions as connected nodes, +dependencies as glowing wires. Moving a function node would +automatically rewire related components, making refactoring a spatial +act. In 2D, this requires mental mapping; in 3D, the structure is +visible and manipulable. + +Ideas in Obsidian-like systems could be arranged in 3D mind maps where +proximity indicates relevance - clicking a node pulls connected +documents into view, creating a navigable "knowledge landscape". + +** Electronics circuit simulation integration +:PROPERTIES: +:CUSTOM_ID: electronics-circuit-simulation-integration +:END: + +Integrate with [[https://ngspice.sourceforge.io/][ngspice]], [[http://www.bwsst.com/][SPICE]], or a similar electronics circuit +simulator, enabling interactive circuit design and simulation in 3D. + +A circuit designer using ngspice could place resistors/capacitors in +3D space, connect them with virtual wires, and simultaneously +visualize current flow as animated particles while overlaying +Pandas-generated performance metrics. This eliminates +context-switching between tools - data, simulation, and visualization +coexist in one spatial workflow. + +This would provide contextual continuity. Related tools (e.g., a +simulation and its data analysis) can be spatially adjacent in space, +avoiding the "window-switching" friction of 2D desktops. + +** 3D visualization of statistical data + +Dataframes from Pandas would render as 3D grids where rows/columns +occupy spatial axes. Dragging a region of the grid would instantly +filter related data. Unlike 2D tables where multidimensional +relationships are flattened, 3D preserves natural spatial +relationships- e.g., high-value data points "tower" above low-value +ones, creating intuitive topographical insights. + +A 3D scatter plot shows all variables simultaneously; rotating it +reveals patterns invisible in 2D projections. + +Students could "walk through" a dataset experiencing trends as spatial +terrain rather than abstract graphs. + +* Source code +:PROPERTIES: +:CUSTOM_ID: source-code +:END: + +*This program is free software: released under Creative Commons Zero +(CC0) license* + +*Program author:* +- Svjatoslav Agejenko +- Homepage: https://svjatoslav.eu +- Email: mailto://svjatoslav@svjatoslav.eu +- See also: [[https://www.svjatoslav.eu/projects/][Other software projects hosted at svjatoslav.eu]] + +*Getting the source code:* +- [[https://www2.svjatoslav.eu/gitweb/?p=aukio.git;a=snapshot;h=HEAD;sf=tgz][Download latest snapshot in TAR GZ format]] +- [[https://www2.svjatoslav.eu/gitweb/?p=aukio.git;a=summary][Browse Git repository online]] +- Clone Git repository using command: + : git clone https://www2.svjatoslav.eu/git/aukio.git diff --git a/TODO.org b/TODO.org new file mode 100644 index 0000000..590c157 --- /dev/null +++ b/TODO.org @@ -0,0 +1,239 @@ +* Features +** Put agents in 3D world + +- Agent has body that resembles human + - TODO: find good design for humanoid + +- Agent has ability to move around cyberspace + +- Agent has ability to see (project image from agent perspective) + +- Agent has ability to write and execute programs + +- Agent has ability to read and write documents in cyberspace + +- Agent can hear your voice (STT, whisper) + +- Agent can speak (TTS, Piper) + +- Local LLM as well as cloud based LLM can be used + - OpenRouter free LLMs + - Kimi K3 + + + +** World is built around intent + +- you start with the world and you name intent for it + +- AI is monitoring what you are doing and is always ready to help + - rename space to better capture intent + +- you create new (optionally linked) spaces with (sub intents) + - you can inherit copy of existing apps into new space or star with empty space + + +* 3D rendering + +Roadmap and measurement notes from the 2026-09-13 perf study +(Fallout 4 environment, downtown Boston spawn — camera +"7146.48,-7989.42,-5598.31,-2.49,-0.83,0.00", 3840x2091). Resume +work here. + +** Where we are — measured baseline + +Single-threaded frame at the Boston spawn (ProfProbe harness): +transform 120 ms (21%) + painter sort 40 ms (7%) + paint 411 ms +(72%) = 571 ms/frame. Paint decomposes (linear fit across +resolutions): per-triangle setup 51%, per-span 25%, pixel fill 24%. +SETUP dominates, not fill. + +Triangle funnel: 1.27M loaded -> 894k reach paint. Of those: +- 46% backfacing (painted, then covered) +- 26% die at the vertical clamp AFTER full per-tri setup +- 20% tinier than 2x2 px at 4K (44% at 960x540!) +- overdraw 3.28x (every pixel written 3.28 times) +- frustum culling: ZERO composites in the FO4 scene — the engine's + AABB frustum culling never engages (shapes added flat to root) + +Live app was at 0.7 fps (bugreport-20260913-030135) vs ~1.75 fps +single-thread probe: heap histogram shows 3.6 GB texture int[] + +1.4 GB G1 filler arrays in 12.4/16 GB heap -> GC pressure is a +separate live-session problem. + +Nanite viability floor (NanoTriBench, 24-core tiny): pixel-sized +triangles rasterize at 26 Mtri/s single-thread, 126 Mtri/s on 22 +threads (8 ns/tri); texture fetch free at that size. Java CAN feed +a micro-triangle pipeline. Modelled Nanite frame at 4K: ~110 ms +(~9 fps) without occlusion culling, ~60 ms (~16 fps) with HZB — +and that number is FLAT with world size. + +NOTE (2026-09-13 evening): fo4 default farRadius is now 2, not 4 +(bugreport-20260913-165044 — the terrain-only FAR doughnut rings +2-4 read as "empty near, detailed far"; BTO object LOD now starts +at ring 3). Baselines above were probed with farRadius=4 (81 +streamed cells at the Boston spawn); reruns now see 25 cells and a +different tri mix — re-baseline ProfProbe before comparing. + +** Road to Nanite — evolutionary stages + +Each stage ships independently and keeps its value; nothing is +thrown away later. The seam of the whole plan is Stage 1 (touches +every rasterizer); after that everything is additive. + +*** TODO Stage 0a: per-cell composite frustum culling +Wrap each streamed cell's shapes in a CompositeShape with an AABB +so the engine's existing frustum test engages. Est -20-30% frame. +Already the top item in aukio-environment-fo4/AGENTS.org. + +*** TODO Stage 0b: per-material backface culling +MEASURED -11% paint (-De3d.backface=true exists in the engine, +global, default off). Production version must key off the NIF +two-sided shader flag (cutout trees/fences are double-sided), so +wire a flag from NifFile shader properties into TexturedTriangle. + +*** TODO Stage 1: z-buffer (w-buffer) + two-pass rendering +Opaque pass: depth test+write. Transparent pass (glass, GUI, text): +back-to-front painter, test but NEVER write depth (avoids the +"insert between merged layers" problem — see Design decisions). +Cutout alpha (fences, foliage) joins the opaque pass with texel +test. Store 1/z (w-buffer): our perspective path already +interpolates 1/z per pixel, depth is a byproduct we throw away; +hyperbolic precision suits the 130k-unit horizon. +Seam: every span writer in the engine must learn depth (same class +of sweep as the stereo X-clipping change — grep all +drawHorizontalLine variants). Needs demo app + golden re-render +with VISION review (z intentionally changes pixels where painter +was wrong — road mottling, depthBias hack, dithered _lod statics, +patchy distant buildings all die in this commit). + +*** TODO Stage 2: opaque near-to-far (early reject) +Flip the comparator for the opaque pass. Est -12% frame from +killing the 3.28x overdraw. Hours once Stage 1 exists. + +*** TODO Stage 3: HZB occlusion culling +Downsample depth into a coarse mip pyramid per frame; test +cell/object AABBs against it before streaming/drawing; cull whole +cells — skips transform+sort+setup+fill. Nanite's two-phase trick +applies: reuse LAST frame's pyramid, draw last frame's visible set +first, then test the rest. View-dependent: moderate at the aerial +spawn, large at street level. + +*** TODO Stage 4: tiny-tri stamp fast path +At paint entry: screen bbox < 2x2 px -> stamp one average-colored +quad (nearest mip texel at centroid) instead of full setup. 20% of +triangles at 4K; est -7-10% paint. This is the SEED of the +micro-rasterizer grown inside the existing one. Object-level +variant: drop whole placements under ~2 px bbox at transform time +(with hysteresis against pop-in). + +*** TODO Stage 5: cluster DAG remeshing (Nanite proper) +The pipeline can't tell where triangles came from — a cluster DAG +is just another shape source, so build it mesh-by-mesh, no flag +day. Offline tool: cluster ~128-tri groups, simplification DAG +with boundary locking; emit clusters; render through the same +scene graph. Prototype on one kit (Red Rocket shell), measure, +then batch. Naive vertex-clustering decimation = days; good +quadric-error version = the multi-week meshoptimizer-class nut, +but isolated offline. FO4's own _lod.nif levels are a quality +reference. End state: ~15 fps at 4K FLAT with world size; all +hand/baked LOD (_lod.nif, BTR, BTO, near ring) replaced uniformly; +distant patchiness gone. + +*** Optional fork: visibility buffer +Write triangle-ID + depth, resolve materials once per visible +pixel in a second pass. Worth it when shading dominates; borderline +at 4K already. Cheap to try once Stage 1 exists (ID write rides +with the depth write). + +** Other optimization proposals (not on the Nanite road) + +- Heap/GC diet (live-session 0.7 fps cause): 3.6 GB texture int[] + (mip chains) + G1 humongous fragmentation. Options: bigger heap, + drop deepest mips for distant-only textures, 16-bit pixel + storage. Own work item, independent of rendering changes. +- Temporal coherence: cache per-cell visibility/culling decisions + while the camera stays in a cell; re-evaluate on cell change or + large rotation. +- Coverage mask (painter-era occlusion, NO z-buffer): rasterize + nearest ~10-20% opaque tris into a coarse coverage bitmap + front-first, skip far objects fully covered. Exact under painter + rules (only nearer+opaque sets bits). SUPERSEDED by Stage 3 if + the z-buffer lands — kept here in case we stay painter-only. +- Terrain horizon culling: heightfield horizon-angle test; cells + behind hills culled nearly free. Big in hilly country, minor in + flat Boston. +- BSP exact ordering: engine already has BSP-ranked shapes (skip + Z-sort). Correctness + sort time, not occlusion. Pairs with + everything above. +- Resolution honesty: ~900k tris at 4K on CPU won't hit 60 fps by + any of these. Internal-render-at-lower-res + upscale is the + blunt instrument (540p -> 4K saves ~40% paint, looks soft). + +** Exotic techniques survey (verdicts from the discussion) + +All complexity-independent techniques share one idea: query a +prebuilt spatial index per pixel instead of drawing per triangle; +cost = pixels x log(world). + +- BVH ray tracing: THE complexity-independent endpoint. Subsumes + occlusion/frustum/backface/overdraw/painter-sort by never + touching invisible geometry. Static world = ideal (build once). + Java CPU: ~10-50M rays/s -> a few fps at 1080p, ~1 at 4K, flat + as world grows; shadows/GI nearly free per extra ray. Hybrid + variant: rasterize near ring, ray-trace past it. Aligned with + user's taste (chose real RT over lightmap upscaling for GI). +- Surfel/point-cloud (QSplat/Potree): the 4px-blob idea + industrialized. Octree of representative points, screen-error + LOD walk, hard POINT BUDGET per frame -> world-size independent. + Most painter-friendly; cheapest exotic to try. +- Voxel octree raymarching (SVO, Teardown; Euclideon was the + marketing version): geometry forgotten after voxelization. + Niche for us: voxelize the coarse BTR/BTO horizon blocks. +- 3D Gaussian splatting: differentiable painter's algorithm; + beautiful but a CAPTURE representation (from photos), wrong + input format for game meshes. +- SDF ray marching: procedural content only; converting FO4's + detailed meshes to distance fields is lossy. Bad fit. + +** Design decisions (resolved discussions, don't re-litigate) + +- Z-buffer speedup mechanics: z alone gives CORRECTNESS, not + speed; speed comes from flipping opaque order near-to-far so + behind-fragments become cheap depth-test rejects. Per-pixel + reject ~1/10 of a shaded write. +- Transparent polygons under z: two passes (opaque test+write; + transparent back-to-front test-only). One depth slot per pixel + cannot represent a stack of see-through layers. +- REJECTED: single-pass alpha-accumulation (framebuffer color + + per-pixel alpha + depth, compose on arrival). Math is valid + (front-to-back "over" is associative, A' = A + a(1-A)) but + requires strict per-pixel near-to-far arrival; per-triangle + centroid sorting violates it routinely (existing road mottling + proves the error rate), and once layers merge to alpha=1 a later + fragment landing between them gets weight 0 -> OPAQUE geometry + can vanish. Two-pass fails cosmetically (wrong tint between + glasses, rare); one-pass fails structurally (missing geometry, + common). Real descendants if ever needed: depth peeling, + weighted blended OIT. +- User preference noted: evolutionary stages with each step + shippable, over big-bang rewrites. + +** Harnesses and flags (for resuming work) + +- ProfProbe.java — per-stage frame profiler at the bugreport pose + (world load like Fo4Shot, times transform/sort/paint, prints + counters). NanoTriBench.java — micro-triangle floor bench. + Both in /tmp/hermes-verify-redrocket/ — VOLATILE (tmpfs); + recreate or move into the repo before reboot. +- Engine instrumentation (UNCOMMITTED in aukio-3d): -De3d.prof=true + enables TexturedTriangle counters (tris/backface/offscreenY/ + tiny/spans/pixels, profReset()); -De3d.backface=true sets the + backface-culling default. Probe classpath: prepend + aukio-3d/target/classes so the instrumented engine wins over the + installed jar. +- FO4 goldens: Fo4Shot spawn| [w h]; spawn pose + lives in ~/.config/aukio/config.properties (fo4.spawnAt). +- Project docs: aukio-environment-fo4/AGENTS.org (streaming/LOD + details), skills n0-aukio-environment-fo4, n0-aukio-3d-engine. + diff --git a/Tools/Open with IntelliJ IDEA b/Tools/Open with IntelliJ IDEA new file mode 100755 index 0000000..de9bae5 --- /dev/null +++ b/Tools/Open with IntelliJ IDEA @@ -0,0 +1,18 @@ +#!/bin/bash + +# +# This is a helper bash script that starts IntelliJ with the current project. +# Script is written is such a way that you can simply click on it in file +# navigator to run it. +# +# +# Script assumes: +# +# + GNU operating system +# + IntelliJ is installed and commandline launcher "idea" is enabled. +# + +cd "${0%/*}" +cd .. + +setsid idea . &>/dev/null diff --git a/Tools/Update web site b/Tools/Update web site new file mode 100755 index 0000000..028c6e4 --- /dev/null +++ b/Tools/Update web site @@ -0,0 +1,76 @@ +#!/bin/bash +cd "${0%/*}"; if [ "$1" != "T" ]; then gnome-terminal -e "'$0' T"; exit; fi; + +cd .. + +export_org_to_html() { + local org_file=$1 + local dir=$(dirname "$org_file") + local base=$(basename "$org_file" .org) + ( + cd "$dir" || return 1 + local html_file="${base}.html" + if [ -f "$html_file" ]; then + rm -f "$html_file" + fi + echo "Exporting: $org_file → $dir/$html_file" + emacs --batch -l ~/.emacs --visit="${base}.org" --funcall=org-html-export-to-html --kill + if [ $? -eq 0 ]; then + echo "✓ Successfully exported $org_file" + else + echo "✗ Failed to export $org_file" + return 1 + fi + ) +} + +export_org_files_to_html() { + echo "🔍 Searching for .org files in Documentation/ ..." + echo "=======================================" + + mapfile -t ORG_FILES < <(find Documentation -type f -name "*.org" | sort) + + if [ ${#ORG_FILES[@]} -eq 0 ]; then + echo "❌ No .org files found!" + return 1 + fi + + echo "Found ${#ORG_FILES[@]} .org file(s):" + printf '%s\n' "${ORG_FILES[@]}" + echo "=======================================" + + SUCCESS_COUNT=0 + FAILED_COUNT=0 + + for org_file in "${ORG_FILES[@]}"; do + export_org_to_html "$org_file" + if [ $? -eq 0 ]; then + ((SUCCESS_COUNT++)) + else + ((FAILED_COUNT++)) + fi + done + + echo "=======================================" + echo "📊 SUMMARY:" + echo " ✓ Successful: $SUCCESS_COUNT" + echo " ✗ Failed: $FAILED_COUNT" + echo " Total: $((SUCCESS_COUNT + FAILED_COUNT))" + echo "" +} + +export_org_files_to_html + +echo "📤 Uploading to server..." +rsync -avz --delete -e 'ssh -p 10006' Documentation/ \ + n0@www3.svjatoslav.eu:/mnt/big/projects/aukio/ + +if [ $? -eq 0 ]; then + echo "✓ Upload completed successfully!" +else + echo "✗ Upload failed!" +fi + +echo "" +echo "Press ENTER to close this window." +read \ No newline at end of file diff --git a/pom.xml b/pom.xml new file mode 100644 index 0000000..94aba4d --- /dev/null +++ b/pom.xml @@ -0,0 +1,125 @@ + + 4.0.0 + eu.svjatoslav + aukio + 1.0.0-SNAPSHOT + Aukio + Spatial computing environment + + + UTF-8 + UTF-8 + 21 + 21 + + + + + svjatoslav.eu + http://svjatoslav.eu + + + + + + eu.svjatoslav + aukio-3d + 1.0.0-SNAPSHOT + + + + net.java.dev.jna + jna + 5.14.0 + + + + eu.svjatoslav + svjatoslavcommons + 1.8 + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.13.0 + + 21 + 21 + UTF-8 + + + + + org.apache.maven.plugins + maven-source-plugin + 2.2.1 + + + attach-sources + + jar + + + + + + + org.apache.maven.plugins + maven-javadoc-plugin + 2.9 + + + attach-javadocs + + jar + + + + + + + + + + org.apache.maven.wagon + wagon-ssh-external + 2.6 + + + + + + + + svjatoslav.eu + svjatoslav.eu + scpexe://svjatoslav.eu:10006/srv/maven + + + svjatoslav.eu + svjatoslav.eu + scpexe://svjatoslav.eu:10006/srv/maven + + + + + + svjatoslav.eu + Svjatoslav repository + https://www3.svjatoslav.eu/maven/ + + + + + scm:git:ssh://n0@svjatoslav.eu:10006/home/n0/git/aukio.git + scm:git:ssh://n0@svjatoslav.eu:10006/home/n0/git/aukio.git + HEAD + + + + diff --git a/src/main/java/eu/svjatoslav/aukio/bridge/pty/PtySession.java b/src/main/java/eu/svjatoslav/aukio/bridge/pty/PtySession.java new file mode 100644 index 0000000..a96d794 --- /dev/null +++ b/src/main/java/eu/svjatoslav/aukio/bridge/pty/PtySession.java @@ -0,0 +1,167 @@ +/* + * Aukio spatial computing environment. Author: Svjatoslav Agejenko. + * This project is released under Creative Commons Zero (CC0) license. + */ +package eu.svjatoslav.aukio.bridge.pty; + +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.nio.charset.StandardCharsets; +import java.util.HashMap; +import java.util.Map; + +/** + * A shell process running on a real pseudo-terminal, feeding a + * {@link ScreenBuffer} through a {@link Vt100Emulator}. + * + *

Engine-independent: this class knows nothing about the 3D world. + * Rendering and keyboard delivery are the caller's job (see + * {@code workspace.TerminalPanel}).

+ * + *

Output path: a daemon reader thread pumps decoded characters into the + * emulator and invokes the registered {@link ContentListener} after every + * batch, so the renderer can repaint. Input path: {@link #send(String)} + * writes bytes to the PTY master.

+ */ +public class PtySession { + + /** + * Called on the reader thread after terminal content changed. + */ + public interface ContentListener { + void contentChanged(); + } + + private final UnixPty pty; + private final ScreenBuffer screenBuffer; + private final Vt100Emulator emulator; + private final Thread readerThread; + + private volatile ContentListener contentListener; + private volatile boolean running = true; + + /** + * Spawns an interactive bash on a new PTY. + * + * @param columns terminal width in characters + * @param rows terminal height in characters + */ + public PtySession(final int columns, final int rows) { + screenBuffer = new ScreenBuffer(columns, rows); + emulator = new Vt100Emulator(screenBuffer); + + final Map environment = new HashMap<>( + System.getenv()); + environment.put("TERM", "xterm-256color"); + environment.put("COLORTERM", "truecolor"); + + // Do not inherit an "active conda environment" from the parent + // process: a fresh terminal session must re-activate through + // bashrc like any desktop terminal. Inheriting CONDA_DEFAULT_ENV + // without CONDA_PREFIX crashes conda's activator, which then + // blocks shell startup with an interactive error-report question. + environment.remove("CONDA_DEFAULT_ENV"); + environment.remove("CONDA_SHLVL"); + environment.remove("CONDA_PROMPT_MODIFIER"); + environment.remove("CONDA_PREFIX"); + // never block shell startup on conda's interactive error upload + environment.put("CONDA_REPORT_ERRORS", "false"); + + pty = new UnixPty(new String[]{"/bin/bash"}, + System.getProperty("user.dir"), environment, columns, rows); + + readerThread = new Thread(this::readLoop, "pty-reader"); + readerThread.setDaemon(true); + readerThread.start(); + } + + private InputStream ptyInputStream() { + return new InputStream() { + private final byte[] chunk = new byte[8192]; + + @Override + public int read() { + final byte[] one = new byte[1]; + return read(one, 0, 1) < 0 ? -1 : (one[0] & 0xFF); + } + + @Override + public int read(final byte[] buffer, final int offset, + final int length) { + final int count = pty.read(chunk); + if (count < 0) + return -1; + final int amount = Math.min(count, length); + System.arraycopy(chunk, 0, buffer, offset, amount); + return amount; + } + }; + } + + private void readLoop() { + final char[] chunk = new char[8192]; + try (final InputStreamReader reader = new InputStreamReader( + ptyInputStream(), StandardCharsets.UTF_8)) { + while (running) { + final int count = reader.read(chunk); + if (count < 0) + break; + emulator.accept(chunk, count); + final ContentListener listener = contentListener; + if (listener != null) + try { + listener.contentChanged(); + } catch (final Throwable t) { + // The renderer can briefly reallocate canvas + // internals during a window resize, racing this + // writer thread. A listener failure must never + // kill the reader loop — the next output batch + // repaints anyway. + t.printStackTrace(); + } + } + } catch (final IOException e) { + // PTY closed: shell exited or session stopped + } + running = false; + } + + /** + * Sends text to the shell as if typed (UTF-8 encoded). + */ + public void send(final String text) { + final byte[] bytes = text.getBytes(StandardCharsets.UTF_8); + pty.write(bytes, 0, bytes.length); + } + + public ScreenBuffer getScreenBuffer() { + return screenBuffer; + } + + public void setContentListener(final ContentListener listener) { + this.contentListener = listener; + } + + public boolean isRunning() { + return running && pty.isChildAlive(); + } + + /** + * Kills the shell and stops the reader thread. + * + *

Order matters: kill the child first so the reader's blocking + * read() returns, join the reader, and only then let the JVM move on + * to exit — a daemon thread mid-JNA-call during JVM teardown can + * crash the process with native heap errors.

+ */ + public void stop() { + running = false; + pty.destroy(); + try { + readerThread.join(2000); + } catch (final InterruptedException e) { + Thread.currentThread().interrupt(); + } + } +} diff --git a/src/main/java/eu/svjatoslav/aukio/bridge/pty/ScreenBuffer.java b/src/main/java/eu/svjatoslav/aukio/bridge/pty/ScreenBuffer.java new file mode 100644 index 0000000..34ace80 --- /dev/null +++ b/src/main/java/eu/svjatoslav/aukio/bridge/pty/ScreenBuffer.java @@ -0,0 +1,574 @@ +/* + * Aukio spatial computing environment. Author: Svjatoslav Agejenko. + * This project is released under Creative Commons Zero (CC0) license. + */ +package eu.svjatoslav.aukio.bridge.pty; + +/** + * Character cell grid backing a terminal screen. + * + *

Pure model: no rendering, no engine dependencies. Colors are stored as + * ANSI palette indices ({@code -1} = terminal default); mapping to actual + * RGB values is the renderer's job.

+ * + *

All public methods must be called while holding {@code synchronized} + * on this instance (the emulator and the renderer both access it).

+ */ +public class ScreenBuffer { + + /** + * Default color marker: use the terminal's default foreground/background. + */ + public static final int DEFAULT_COLOR = -1; + + /** + * One character cell: glyph plus attributes. + */ + public static final class Cell { + public char ch = ' '; + public int fg = DEFAULT_COLOR; + public int bg = DEFAULT_COLOR; + public boolean bold = false; + public boolean reverse = false; + + void set(final char c, final int fg, final int bg, + final boolean bold, final boolean reverse) { + this.ch = c; + this.fg = fg; + this.bg = bg; + this.bold = bold; + this.reverse = reverse; + } + + void clear(final int bg) { + set(' ', DEFAULT_COLOR, bg, false, false); + } + } + + private final int columns; + private final int rows; + + private Cell[][] screen; + private Cell[][] alternateScreen; + + public int cursorX = 0; + public int cursorY = 0; + public boolean cursorVisible = true; + + private int savedCursorX = 0; + private int savedCursorY = 0; + + private int scrollTop = 0; + private int scrollBottom; + + private int currentFg = DEFAULT_COLOR; + private int currentBg = DEFAULT_COLOR; + private boolean currentBold = false; + private boolean currentReverse = false; + + boolean autoWrap = true; + boolean insertMode = false; + boolean lineDrawing = false; + + /** + * DECCKM (?1h): when set, cursor keys must be reported as SS3 + * (ESC O A) instead of CSI (ESC [ A). Curses applications enable + * this; input translation reads it. + */ + public boolean applicationCursorKeys = false; + + private boolean pendingWrap = false; + + public ScreenBuffer(final int columns, final int rows) { + this.columns = columns; + this.rows = rows; + this.scrollBottom = rows - 1; + screen = newGrid(); + } + + private Cell[][] newGrid() { + final Cell[][] grid = new Cell[rows][columns]; + for (int row = 0; row < rows; row++) + for (int column = 0; column < columns; column++) { + grid[row][column] = new Cell(); + grid[row][column].clear(DEFAULT_COLOR); + } + return grid; + } + + public int getColumns() { + return columns; + } + + public int getRows() { + return rows; + } + + public Cell getCell(final int row, final int column) { + return screen[row][column]; + } + + // ------------------------------------------------------------------ + // printing + // ------------------------------------------------------------------ + + public void putChar(final char c) { + if (pendingWrap) { + pendingWrap = false; + carriageReturn(); + lineFeed(); + } + if (insertMode) + insertChars(1); + screen[cursorY][cursorX].set(mapLineDrawing(c), currentFg, currentBg, + currentBold, currentReverse); + if (cursorX == columns - 1) { + if (autoWrap) + pendingWrap = true; + } else + cursorX++; + } + + private char mapLineDrawing(final char c) { + if (!lineDrawing) + return c; + // DEC special graphics character set (approximate) + return switch (c) { + case 'j' -> '┘'; + case 'k' -> '┐'; + case 'l' -> '┌'; + case 'm' -> '└'; + case 'n' -> '┼'; + case 'q' -> '─'; + case 't' -> '├'; + case 'u' -> '┤'; + case 'v' -> '┴'; + case 'w' -> '┬'; + case 'x' -> '│'; + case 'a' -> '▒'; + default -> c; + }; + } + + // ------------------------------------------------------------------ + // control characters + // ------------------------------------------------------------------ + + public void carriageReturn() { + cursorX = 0; + pendingWrap = false; + } + + public void lineFeed() { + pendingWrap = false; + if (cursorY == scrollBottom) + scrollUp(1); + else if (cursorY < rows - 1) + cursorY++; + } + + /** + * Reverse index: cursor up, scrolling the region down at the top edge. + */ + public void reverseIndex() { + pendingWrap = false; + if (cursorY == scrollTop) + scrollDown(1); + else if (cursorY > 0) + cursorY--; + } + + public void backspace() { + pendingWrap = false; + if (cursorX > 0) + cursorX--; + } + + public void tab() { + pendingWrap = false; + cursorX = Math.min(columns - 1, (cursorX + 8) & ~7); + } + + // ------------------------------------------------------------------ + // cursor movement (CSI) + // ------------------------------------------------------------------ + + public void cursorUp(final int n) { + pendingWrap = false; + cursorY = Math.max(scrollTop, cursorY - Math.max(1, n)); + } + + public void cursorDown(final int n) { + pendingWrap = false; + cursorY = Math.min(scrollBottom, cursorY + Math.max(1, n)); + } + + public void cursorForward(final int n) { + pendingWrap = false; + cursorX = Math.min(columns - 1, cursorX + Math.max(1, n)); + } + + public void cursorBack(final int n) { + pendingWrap = false; + cursorX = Math.max(0, cursorX - Math.max(1, n)); + } + + /** + * 1-based absolute position, as sent by CSI H / CSI f. + */ + public void setCursorPosition(final int row1based, final int column1based) { + pendingWrap = false; + cursorY = clamp(row1based - 1, 0, rows - 1); + cursorX = clamp(column1based - 1, 0, columns - 1); + } + + public void setCursorColumn(final int column1based) { + pendingWrap = false; + cursorX = clamp(column1based - 1, 0, columns - 1); + } + + public void setCursorRow(final int row1based) { + pendingWrap = false; + cursorY = clamp(row1based - 1, 0, rows - 1); + } + + public void saveCursor() { + savedCursorX = cursorX; + savedCursorY = cursorY; + } + + public void restoreCursor() { + pendingWrap = false; + cursorX = savedCursorX; + cursorY = savedCursorY; + } + + private static int clamp(final int v, final int min, final int max) { + return Math.max(min, Math.min(max, v)); + } + + // ------------------------------------------------------------------ + // erasing + // ------------------------------------------------------------------ + + private void clearRange(final int row, final int fromColumn, + final int toColumn) { + for (int column = fromColumn; column <= toColumn; column++) + screen[row][column].clear(currentBg); + } + + /** + * CSI J: 0 = cursor to end, 1 = start to cursor, 2 = whole screen. + */ + public void eraseInDisplay(final int mode) { + pendingWrap = false; + switch (mode) { + case 0 -> { + clearRange(cursorY, cursorX, columns - 1); + for (int row = cursorY + 1; row < rows; row++) + clearRange(row, 0, columns - 1); + } + case 1 -> { + for (int row = 0; row < cursorY; row++) + clearRange(row, 0, columns - 1); + clearRange(cursorY, 0, cursorX); + } + case 2, 3 -> { + for (int row = 0; row < rows; row++) + clearRange(row, 0, columns - 1); + } + default -> { + } + } + } + + /** + * CSI K: 0 = cursor to end of line, 1 = start to cursor, 2 = whole line. + */ + public void eraseInLine(final int mode) { + pendingWrap = false; + switch (mode) { + case 0 -> clearRange(cursorY, cursorX, columns - 1); + case 1 -> clearRange(cursorY, 0, cursorX); + case 2 -> clearRange(cursorY, 0, columns - 1); + default -> { + } + } + } + + /** + * CSI X: erase n characters from cursor without moving it. + */ + public void eraseChars(final int n) { + pendingWrap = false; + final int count = Math.max(1, n); + clearRange(cursorY, cursorX, + Math.min(columns - 1, cursorX + count - 1)); + } + + // ------------------------------------------------------------------ + // inserting / deleting / scrolling + // ------------------------------------------------------------------ + + /** + * CSI P: delete n characters at cursor, shifting the rest of the line left. + */ + public void deleteChars(final int n) { + pendingWrap = false; + final int count = Math.max(1, n); + for (int column = cursorX; column < columns; column++) { + if (column + count < columns) + screen[cursorY][column].set( + screen[cursorY][column + count].ch, + screen[cursorY][column + count].fg, + screen[cursorY][column + count].bg, + screen[cursorY][column + count].bold, + screen[cursorY][column + count].reverse); + else + screen[cursorY][column].clear(currentBg); + } + } + + /** + * CSI @: insert n blank characters at cursor, shifting line right. + */ + public void insertChars(final int n) { + final int count = Math.max(1, n); + for (int column = columns - 1; column >= cursorX; column--) { + if (column - count >= cursorX) + screen[cursorY][column].set( + screen[cursorY][column - count].ch, + screen[cursorY][column - count].fg, + screen[cursorY][column - count].bg, + screen[cursorY][column - count].bold, + screen[cursorY][column - count].reverse); + else + screen[cursorY][column].clear(currentBg); + } + } + + /** + * CSI L: insert n blank lines at cursor row (within scroll region). + */ + public void insertLines(final int n) { + pendingWrap = false; + if (cursorY < scrollTop || cursorY > scrollBottom) + return; + final int count = Math.min(Math.max(1, n), scrollBottom - cursorY + 1); + for (int row = scrollBottom; row >= cursorY; row--) { + if (row - count >= cursorY) + copyRow(row - count, row); + else + clearRange(row, 0, columns - 1); + } + } + + /** + * CSI M: delete n lines at cursor row (within scroll region). + */ + public void deleteLines(final int n) { + pendingWrap = false; + if (cursorY < scrollTop || cursorY > scrollBottom) + return; + final int count = Math.min(Math.max(1, n), scrollBottom - cursorY + 1); + for (int row = cursorY; row <= scrollBottom; row++) { + if (row + count <= scrollBottom) + copyRow(row + count, row); + else + clearRange(row, 0, columns - 1); + } + } + + private void copyRow(final int fromRow, final int toRow) { + for (int column = 0; column < columns; column++) { + final Cell from = screen[fromRow][column]; + screen[toRow][column].set(from.ch, from.fg, from.bg, from.bold, + from.reverse); + } + } + + /** + * Scroll the scroll region up by n lines (content moves up, blanks at + * bottom). + */ + public void scrollUp(final int n) { + final int count = Math.min(Math.max(1, n), + scrollBottom - scrollTop + 1); + for (int row = scrollTop; row <= scrollBottom; row++) { + if (row + count <= scrollBottom) + copyRow(row + count, row); + else + clearRange(row, 0, columns - 1); + } + } + + /** + * Scroll the scroll region down by n lines (content moves down, blanks at + * top). + */ + public void scrollDown(final int n) { + final int count = Math.min(Math.max(1, n), + scrollBottom - scrollTop + 1); + for (int row = scrollBottom; row >= scrollTop; row--) { + if (row - count >= scrollTop) + copyRow(row - count, row); + else + clearRange(row, 0, columns - 1); + } + } + + /** + * CSI r: set scroll region, 1-based inclusive rows. Also homes the cursor + * per xterm behavior. + */ + public void setScrollRegion(final int top1based, final int bottom1based) { + final int top = clamp(top1based - 1, 0, rows - 1); + final int bottom = clamp(bottom1based - 1, 0, rows - 1); + if (top < bottom) { + scrollTop = top; + scrollBottom = bottom; + } + setCursorPosition(1, 1); + } + + // ------------------------------------------------------------------ + // attributes / modes + // ------------------------------------------------------------------ + + /** + * CSI m: select graphic rendition. {@code params[i] == -1} marks an + * omitted parameter (treated as 0). + */ + public void sgr(final int[] params) { + if (params.length == 0) { + resetAttributes(); + return; + } + for (int i = 0; i < params.length; i++) { + final int p = params[i] < 0 ? 0 : params[i]; + if (p == 0) + resetAttributes(); + else if (p == 1) + currentBold = true; + else if (p == 22) + currentBold = false; + else if (p == 7) + currentReverse = true; + else if (p == 27) + currentReverse = false; + else if (p >= 30 && p <= 37) + currentFg = p - 30; + else if (p == 39) + currentFg = DEFAULT_COLOR; + else if (p >= 40 && p <= 47) + currentBg = p - 40; + else if (p == 49) + currentBg = DEFAULT_COLOR; + else if (p >= 90 && p <= 97) + currentFg = p - 90 + 8; + else if (p >= 100 && p <= 107) + currentBg = p - 100 + 8; + else if ((p == 38 || p == 48) && i + 2 < params.length + && params[i + 1] == 5) { + // 256-color palette index; renderer maps it + if (p == 38) + currentFg = 16 + (params[i + 2] & 0xFF); + else + currentBg = 16 + (params[i + 2] & 0xFF); + i += 2; + } + // other attributes (underline, blink, ...) accepted but not + // visualized + } + } + + private void resetAttributes() { + currentFg = DEFAULT_COLOR; + currentBg = DEFAULT_COLOR; + currentBold = false; + currentReverse = false; + } + + /** + * DEC private mode set/reset (CSI ? ... h / l). + */ + public void setDecMode(final int mode, final boolean enabled) { + switch (mode) { + case 1 -> applicationCursorKeys = enabled; + case 7 -> autoWrap = enabled; + case 25 -> cursorVisible = enabled; + case 47, 1047 -> setAlternateScreen(enabled, false); + case 1048 -> { + if (enabled) + saveCursor(); + else + restoreCursor(); + } + case 1049 -> setAlternateScreen(enabled, true); + default -> { + // other private modes (bracketed paste, application + // keypad, mouse reporting, ...) accepted but ignored + } + } + } + + /** + * ANSI mode set/reset (CSI h / l without '?'). + */ + public void setAnsiMode(final int mode, final boolean enabled) { + if (mode == 4) + insertMode = enabled; + } + + private void setAlternateScreen(final boolean enabled, + final boolean saveCursor) { + if (enabled) { + if (saveCursor) + saveCursor(); + alternateScreen = screen; + screen = newGrid(); + setCursorPosition(1, 1); + } else { + if (alternateScreen != null) + screen = alternateScreen; + alternateScreen = null; + if (saveCursor) + restoreCursor(); + } + } + + public void fullReset() { + resetAttributes(); + autoWrap = true; + insertMode = false; + lineDrawing = false; + applicationCursorKeys = false; + cursorVisible = true; + pendingWrap = false; + scrollTop = 0; + scrollBottom = rows - 1; + alternateScreen = null; + screen = newGrid(); + cursorX = 0; + cursorY = 0; + } + + // ------------------------------------------------------------------ + // debugging / testing + // ------------------------------------------------------------------ + + /** + * Renders the visible screen as plain text (attributes stripped). + */ + public String dumpText() { + final StringBuilder result = new StringBuilder(); + for (int row = 0; row < rows; row++) { + final StringBuilder line = new StringBuilder(); + for (int column = 0; column < columns; column++) + line.append(screen[row][column].ch); + result.append(line.toString().replaceAll("\\s+$", "")) + .append('\n'); + } + return result.toString(); + } +} diff --git a/src/main/java/eu/svjatoslav/aukio/bridge/pty/UnixPty.java b/src/main/java/eu/svjatoslav/aukio/bridge/pty/UnixPty.java new file mode 100644 index 0000000..1c2b082 --- /dev/null +++ b/src/main/java/eu/svjatoslav/aukio/bridge/pty/UnixPty.java @@ -0,0 +1,243 @@ +/* + * Aukio spatial computing environment. Author: Svjatoslav Agejenko. + * This project is released under Creative Commons Zero (CC0) license. + */ +package eu.svjatoslav.aukio.bridge.pty; + +import com.sun.jna.Library; +import com.sun.jna.Memory; +import com.sun.jna.Native; +import com.sun.jna.Pointer; + +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +/** + * Minimal Linux PTY implementation via JNA direct libc calls. + * + *

Replaces the pty4j dependency: pty4j 0.12.x requires purejavacomm + * (jtermios classes), which is no longer published on any reachable Maven + * repository. All we need is {@code posix_openpt} + {@code fork} + + * {@code execve}, which is a handful of libc calls.

+ * + *

Linux-only by design; the workspace targets Linux.

+ */ +final class UnixPty { + + private static final int O_RDWR = 0x02; + private static final int O_NOCTTY = 0x100; + private static final int TIOCSWINSZ = 0x5414; + private static final int POSIX_SPAWN_SETSID = 0x80; // glibc >= 2.26 + private static final int SIGKILL = 9; + + private interface CLib extends Library { + CLib INSTANCE = Native.load("c", CLib.class); + + int posix_openpt(int flags); + + int grantpt(int fd); + + int unlockpt(int fd); + + String ptsname(int fd); + + int ioctl(int fd, int request, Pointer arg); + + int posix_spawn(Memory pid, String path, Pointer fileActions, + Pointer attributes, Pointer argv, Pointer envp); + + int posix_spawn_file_actions_init(Pointer fileActions); + + int posix_spawn_file_actions_addopen(Pointer fileActions, int fd, + String path, int oflag, + int mode); + + int posix_spawn_file_actions_adddup2(Pointer fileActions, int fd, + int newFd); + + int posix_spawn_file_actions_addclose(Pointer fileActions, int fd); + + int posix_spawnattr_init(Pointer attributes); + + int posix_spawnattr_setflags(Pointer attributes, short flags); + + int posix_spawn_file_actions_destroy(Pointer fileActions); + + int posix_spawnattr_destroy(Pointer attributes); + + int close(int fd); + + int read(int fd, Pointer buffer, int count); + + int write(int fd, Pointer buffer, int count); + + int waitpid(int pid, int[] status, int options); + + int kill(int pid, int signal); + } + + final int masterFd; + final int childPid; + + /** + * Single reused native buffer for reads: per-call Memory allocation + * risks the JNA cleaner freeing a buffer while a thread is blocked in + * libc read() on it during JVM shutdown. + */ + private final Memory readMemory = new Memory(65536); + + /** + * Creates a PTY and spawns the command on the slave side via + * {@code posix_spawn}: a new session whose first opened terminal + * becomes its controlling terminal. + * + *

posix_spawn is used instead of fork+exec because fork inside a + * live JVM child deadlocks on JVM internal locks (futex) before it can + * exec — observed empirically: the forked child hung as a second JVM + * and bash never started.

+ * + * @param command argv (absolute path first) + * @param workingDir child working directory (applied by the caller's + * command, e.g. bash is started with the dir as cwd + * via a chdir wrapper below) + * @param environment full environment for the child + * @param columns terminal width in characters + * @param rows terminal height in characters + */ + UnixPty(final String[] command, final String workingDir, + final Map environment, + final int columns, final int rows) { + + masterFd = CLib.INSTANCE.posix_openpt(O_RDWR | O_NOCTTY); + if (masterFd < 0) + throw new IllegalStateException("posix_openpt failed: " + + Native.getLastError()); + if (CLib.INSTANCE.grantpt(masterFd) != 0 + || CLib.INSTANCE.unlockpt(masterFd) != 0) + throw new IllegalStateException("grantpt/unlockpt failed: " + + Native.getLastError()); + + final String slaveName = CLib.INSTANCE.ptsname(masterFd); + + // window size must be set before spawn so the child inherits it + final Memory winsize = new Memory(8); + winsize.setShort(0, (short) rows); + winsize.setShort(2, (short) columns); + winsize.setShort(4, (short) 0); + winsize.setShort(6, (short) 0); + CLib.INSTANCE.ioctl(masterFd, TIOCSWINSZ, winsize); + + // child file actions: slave PTY onto stdin/stdout/stderr, drop + // the master. POSIX_SPAWN_SETSID makes the child a session leader; + // glibc applies it BEFORE file actions, so the addopen below (no + // O_NOCTTY) also acquires the slave as the controlling terminal. + // glibc's posix_spawnattr_t is 336 bytes on amd64; allocate a + // padded block for both structs rather than relying on ABI sizes. + final Memory fileActions = new Memory(512); + final Memory attributes = new Memory(512); + CLib.INSTANCE.posix_spawn_file_actions_init(fileActions); + CLib.INSTANCE.posix_spawnattr_init(attributes); + CLib.INSTANCE.posix_spawnattr_setflags(attributes, + (short) POSIX_SPAWN_SETSID); + CLib.INSTANCE.posix_spawn_file_actions_addopen(fileActions, 0, + slaveName, O_RDWR, 0); + CLib.INSTANCE.posix_spawn_file_actions_adddup2(fileActions, 0, 1); + CLib.INSTANCE.posix_spawn_file_actions_adddup2(fileActions, 0, 2); + CLib.INSTANCE.posix_spawn_file_actions_addclose(fileActions, + masterFd); + + final List envStrings = new ArrayList<>(); + environment.forEach((key, value) -> envStrings.add(key + "=" + value)); + final Pointer envp = pointerArray( + envStrings.toArray(new String[0])); + + // working directory: posix_spawn has no chdir file action (glibc + // added addchdir only in 2.29 as a GNU extension), so spawn a + // helper shell that chdirs and execs the real command with its + // arguments. + final StringBuilder execLine = new StringBuilder( + "cd \"$1\" && shift && exec"); + for (final String arg : command) + execLine.append(" \"").append(arg.replace("\"", "\\\"")) + .append('"'); + final String[] shellCommand = {"/bin/sh", "-c", + execLine.toString(), "sh", workingDir}; + final Pointer shellArgv = pointerArray(shellCommand); + + final Memory pidResult = new Memory(4); + final int rc = CLib.INSTANCE.posix_spawn(pidResult, "/bin/sh", + fileActions, attributes, shellArgv, envp); + CLib.INSTANCE.posix_spawn_file_actions_destroy(fileActions); + CLib.INSTANCE.posix_spawnattr_destroy(attributes); + if (rc != 0) + throw new IllegalStateException("posix_spawn failed: " + rc); + childPid = pidResult.getInt(0); + } + + /** + * Builds a NULL-terminated char** from Java strings. + * + *

The backing native memory is deliberately kept reachable for the + * JVM lifetime (a few dozen small strings per PTY): the child execs + * asynchronously after {@code fork}, so the memory must outlive this + * method call.

+ */ + private static final List NATIVE_STRING_STORAGE = + new ArrayList<>(); + + private static synchronized Pointer pointerArray( + final String[] strings) { + final Memory array = new Memory( + (long) (strings.length + 1) * Native.POINTER_SIZE); + NATIVE_STRING_STORAGE.add(array); + for (int i = 0; i < strings.length; i++) { + final byte[] bytes = strings[i].getBytes(StandardCharsets.UTF_8); + final Memory entry = new Memory(bytes.length + 1); + entry.write(0, bytes, 0, bytes.length); + entry.setByte(bytes.length, (byte) 0); + NATIVE_STRING_STORAGE.add(entry); + array.setPointer((long) i * Native.POINTER_SIZE, entry); + } + array.setPointer((long) strings.length * Native.POINTER_SIZE, null); + return array; + } + + /** + * Blocking read from the PTY master. + * + * @return number of bytes read, or -1 when the child side closed + */ + int read(final byte[] buffer) { + final int amount = Math.min(buffer.length, (int) readMemory.size()); + final int count = CLib.INSTANCE.read(masterFd, readMemory, amount); + if (count <= 0) + return -1; + readMemory.read(0, buffer, 0, count); + return count; + } + + void write(final byte[] buffer, final int offset, final int length) { + final Memory memory = new Memory(length); + memory.write(0, buffer, offset, length); + int written = 0; + while (written < length) { + final int count = CLib.INSTANCE.write(masterFd, + memory.share(written), length - written); + if (count <= 0) + return; + written += count; + } + } + + boolean isChildAlive() { + return CLib.INSTANCE.kill(childPid, 0) == 0; + } + + void destroy() { + CLib.INSTANCE.kill(childPid, SIGKILL); + CLib.INSTANCE.waitpid(childPid, new int[1], 0); + CLib.INSTANCE.close(masterFd); + } +} diff --git a/src/main/java/eu/svjatoslav/aukio/bridge/pty/Vt100Emulator.java b/src/main/java/eu/svjatoslav/aukio/bridge/pty/Vt100Emulator.java new file mode 100644 index 0000000..2e0f86d --- /dev/null +++ b/src/main/java/eu/svjatoslav/aukio/bridge/pty/Vt100Emulator.java @@ -0,0 +1,255 @@ +/* + * Aukio spatial computing environment. Author: Svjatoslav Agejenko. + * This project is released under Creative Commons Zero (CC0) license. + */ +package eu.svjatoslav.aukio.bridge.pty; + +/** + * Minimal VT100/xterm escape sequence interpreter feeding a + * {@link ScreenBuffer}. + * + *

Covers what an interactive shell and everyday tools actually emit: + * printable text (UTF-8 decoded by the caller), C0 controls, CSI cursor + * movement / erasing / insert-delete / scroll region / SGR colors, DEC + * private modes (autowrap, cursor visibility, alternate screen), OSC + * (ignored), and the DEC line-drawing charset.

+ * + *

Not a full xterm: no scrollback, no mouse reporting, no 8-bit C1. + * Unknown sequences are consumed and ignored rather than mishandled.

+ * + *

Threading: all calls arrive on the PTY reader thread; the emulator + * synchronizes on the buffer for every operation.

+ */ +public class Vt100Emulator { + + private enum State { + GROUND, ESCAPE, CSI, OSC, OSC_ESC, CHARSET, IGNORE_ONE + } + + private final ScreenBuffer buffer; + + private State state = State.GROUND; + + private final StringBuilder params = new StringBuilder(); + private boolean csiPrivate = false; + + public Vt100Emulator(final ScreenBuffer buffer) { + this.buffer = buffer; + } + + public void accept(final char[] chars, final int length) { + for (int i = 0; i < length; i++) + accept(chars[i]); + } + + public void accept(final char c) { + switch (state) { + case GROUND -> ground(c); + case ESCAPE -> escape(c); + case CSI -> csi(c); + case OSC -> osc(c); + case OSC_ESC -> { + state = State.GROUND; // consume ST's backslash + } + case CHARSET -> charset(c); + case IGNORE_ONE -> state = State.GROUND; + } + } + + // ------------------------------------------------------------------ + // GROUND + // ------------------------------------------------------------------ + + private void ground(final char c) { + if (c == 0x1B) { + state = State.ESCAPE; + return; + } + if (c < 0x20) { + control(c); + return; + } + if (c == 0x7F) + return; // DEL: ignored on input path + synchronized (buffer) { + buffer.putChar(c); + } + } + + private void control(final char c) { + synchronized (buffer) { + switch (c) { + case '\b' -> buffer.backspace(); + case '\t' -> buffer.tab(); + case '\n', 0x0B, 0x0C -> buffer.lineFeed(); + case '\r' -> buffer.carriageReturn(); + default -> { + // BEL and the rest: no visual effect + } + } + } + } + + // ------------------------------------------------------------------ + // ESC + // ------------------------------------------------------------------ + + private void escape(final char c) { + switch (c) { + case '[' -> { + params.setLength(0); + csiPrivate = false; + state = State.CSI; + } + case ']' -> state = State.OSC; + case '(' -> state = State.CHARSET; + case ')' -> state = State.IGNORE_ONE; // G1 charset: unused + case 'O' -> state = State.IGNORE_ONE; // SS3: single-char control + case '7' -> { + synchronized (buffer) { + buffer.saveCursor(); + } + state = State.GROUND; + } + case '8' -> { + synchronized (buffer) { + buffer.restoreCursor(); + } + state = State.GROUND; + } + case 'D' -> { + synchronized (buffer) { + buffer.lineFeed(); + } + state = State.GROUND; + } + case 'M' -> { + synchronized (buffer) { + buffer.reverseIndex(); + } + state = State.GROUND; + } + case 'E' -> { + synchronized (buffer) { + buffer.carriageReturn(); + buffer.lineFeed(); + } + state = State.GROUND; + } + case 'c' -> { + synchronized (buffer) { + buffer.fullReset(); + } + state = State.GROUND; + } + default -> state = State.GROUND; // =, > and unknowns: ignore + } + } + + private void charset(final char c) { + synchronized (buffer) { + buffer.lineDrawing = (c == '0'); + } + state = State.GROUND; + } + + private void osc(final char c) { + if (c == 0x07) + state = State.GROUND; // BEL terminates + else if (c == 0x1B) + state = State.OSC_ESC; // expect ST ("\") + } + + // ------------------------------------------------------------------ + // CSI + // ------------------------------------------------------------------ + + private void csi(final char c) { + if (c == '?') { + csiPrivate = true; + return; + } + if ((c >= '0' && c <= '9') || c == ';') { + params.append(c); + return; + } + if (c >= 0x20 && c <= 0x2F) + return; // intermediate bytes: ignored (e.g. space in "CSI 4 SP q") + if (c < 0x40 || c > 0x7E) { + state = State.GROUND; + return; + } + dispatchCsi(c); + state = State.GROUND; + } + + private int[] parsedParams() { + if (params.length() == 0) + return new int[0]; + final String[] parts = params.toString().split(";", -1); + final int[] result = new int[parts.length]; + for (int i = 0; i < parts.length; i++) + result[i] = parts[i].isEmpty() ? -1 : Integer.parseInt(parts[i]); + return result; + } + + private int param(final int[] p, final int index, final int defaultValue) { + if (index >= p.length || p[index] <= 0) + return defaultValue; + return p[index]; + } + + private void dispatchCsi(final char command) { + final int[] p = parsedParams(); + synchronized (buffer) { + switch (command) { + case 'A' -> buffer.cursorUp(param(p, 0, 1)); + case 'B' -> buffer.cursorDown(param(p, 0, 1)); + case 'C' -> buffer.cursorForward(param(p, 0, 1)); + case 'D' -> buffer.cursorBack(param(p, 0, 1)); + case 'E' -> { + buffer.cursorDown(param(p, 0, 1)); + buffer.carriageReturn(); + } + case 'F' -> { + buffer.cursorUp(param(p, 0, 1)); + buffer.carriageReturn(); + } + case 'G', '`' -> buffer.setCursorColumn(param(p, 0, 1)); + case 'd' -> buffer.setCursorRow(param(p, 0, 1)); + case 'H', 'f' -> buffer.setCursorPosition(param(p, 0, 1), + param(p, 1, 1)); + case 'J' -> buffer.eraseInDisplay(param(p, 0, 0)); + case 'K' -> buffer.eraseInLine(param(p, 0, 0)); + case 'L' -> buffer.insertLines(param(p, 0, 1)); + case 'M' -> buffer.deleteLines(param(p, 0, 1)); + case 'P' -> buffer.deleteChars(param(p, 0, 1)); + case '@' -> buffer.insertChars(param(p, 0, 1)); + case 'S' -> buffer.scrollUp(param(p, 0, 1)); + case 'T' -> buffer.scrollDown(param(p, 0, 1)); + case 'X' -> buffer.eraseChars(param(p, 0, 1)); + case 'm' -> buffer.sgr(p); + case 'r' -> buffer.setScrollRegion(param(p, 0, 1), + param(p, 1, buffer.getRows())); + case 's' -> buffer.saveCursor(); + case 'u' -> buffer.restoreCursor(); + case 'h' -> setModes(p, true); + case 'l' -> setModes(p, false); + default -> { + // unknown CSI: ignored + } + } + } + } + + private void setModes(final int[] p, final boolean enabled) { + for (final int mode : p) { + if (mode < 0) + continue; + if (csiPrivate) + buffer.setDecMode(mode, enabled); + else + buffer.setAnsiMode(mode, enabled); + } + } +} diff --git a/src/main/java/eu/svjatoslav/aukio/bridge/x11/AwtKeysyms.java b/src/main/java/eu/svjatoslav/aukio/bridge/x11/AwtKeysyms.java new file mode 100644 index 0000000..c1511a6 --- /dev/null +++ b/src/main/java/eu/svjatoslav/aukio/bridge/x11/AwtKeysyms.java @@ -0,0 +1,159 @@ +/* Aukio spatial computing environment. Author: Svjatoslav Agejenko. This project is released under Creative Commons Zero (CC0) license. */ +package eu.svjatoslav.aukio.bridge.x11; + +import java.awt.event.KeyEvent; + +/** + * Translates AWT key events to X11 keysyms for injection into the virtual + * display via the X TEST extension. + * + *

Mapping is by {@code keyCode}, not {@code keyChar}: letters map to + * their unshifted keysyms and uppercase is produced by the real Shift key + * events, which are forwarded like any other key. This mirrors how a + * physical keyboard works and assumes the virtual server's keymap is a + * US-style layout (Xvfb default).

+ * + *

Keys with no obvious mapping fall back to the Unicode keysym scheme + * ({@code 0x01000000 | codepoint}) when the event carries a character — + * this covers characters like ä and ö when they exist in the keymap.

+ */ +final class AwtKeysyms { + + // X11 keysym values (from keysymdef.h) + static final long XK_BACK_SPACE = 0xFF08; + static final long XK_TAB = 0xFF09; + static final long XK_RETURN = 0xFF0D; + static final long XK_ESCAPE = 0xFF1B; + static final long XK_HOME = 0xFF50; + static final long XK_LEFT = 0xFF51; + static final long XK_UP = 0xFF52; + static final long XK_RIGHT = 0xFF53; + static final long XK_DOWN = 0xFF54; + static final long XK_PAGE_UP = 0xFF55; + static final long XK_PAGE_DOWN = 0xFF56; + static final long XK_END = 0xFF57; + static final long XK_INSERT = 0xFF63; + static final long XK_F1 = 0xFFBE; + static final long XK_KP_MULTIPLY = 0xFFAA; + static final long XK_KP_ADD = 0xFFAB; + static final long XK_KP_SUBTRACT = 0xFFAD; + static final long XK_KP_DECIMAL = 0xFFAE; + static final long XK_KP_DIVIDE = 0xFFAF; + static final long XK_KP_0 = 0xFFB0; + static final long XK_SHIFT_L = 0xFFE1; + static final long XK_CONTROL_L = 0xFFE3; + static final long XK_CAPS_LOCK = 0xFFE5; + static final long XK_ALT_L = 0xFFE9; + static final long XK_DELETE = 0xFFFF; + + private AwtKeysyms() { + } + + /** + * Maps an AWT key event to an X keysym. + * + * @return the keysym, or {@code -1} when the key cannot be represented + */ + static long keysymFor(final KeyEvent event) { + final int code = event.getKeyCode(); + + // letters: unshifted keysyms; shift state comes from real Shift + // key events + if (code >= KeyEvent.VK_A && code <= KeyEvent.VK_Z) + return 'a' + (code - KeyEvent.VK_A); + if (code >= KeyEvent.VK_0 && code <= KeyEvent.VK_9) + return '0' + (code - KeyEvent.VK_0); + if (code >= KeyEvent.VK_F1 && code <= KeyEvent.VK_F12) + return XK_F1 + (code - KeyEvent.VK_F1); + if (code >= KeyEvent.VK_NUMPAD0 && code <= KeyEvent.VK_NUMPAD9) + return XK_KP_0 + (code - KeyEvent.VK_NUMPAD0); + + switch (code) { + case KeyEvent.VK_BACK_SPACE: + return XK_BACK_SPACE; + case KeyEvent.VK_TAB: + return XK_TAB; + case KeyEvent.VK_ENTER: + return XK_RETURN; + case KeyEvent.VK_ESCAPE: + return XK_ESCAPE; + case KeyEvent.VK_INSERT: + return XK_INSERT; + case KeyEvent.VK_DELETE: + return XK_DELETE; + case KeyEvent.VK_HOME: + return XK_HOME; + case KeyEvent.VK_END: + return XK_END; + case KeyEvent.VK_PAGE_UP: + return XK_PAGE_UP; + case KeyEvent.VK_PAGE_DOWN: + return XK_PAGE_DOWN; + case KeyEvent.VK_LEFT: + return XK_LEFT; + case KeyEvent.VK_UP: + return XK_UP; + case KeyEvent.VK_RIGHT: + return XK_RIGHT; + case KeyEvent.VK_DOWN: + return XK_DOWN; + case KeyEvent.VK_SHIFT: + return XK_SHIFT_L; + case KeyEvent.VK_CONTROL: + return XK_CONTROL_L; + case KeyEvent.VK_ALT: + return XK_ALT_L; + case KeyEvent.VK_CAPS_LOCK: + return XK_CAPS_LOCK; + case KeyEvent.VK_MULTIPLY: + return XK_KP_MULTIPLY; + case KeyEvent.VK_ADD: + return XK_KP_ADD; + case KeyEvent.VK_SUBTRACT: + return XK_KP_SUBTRACT; + case KeyEvent.VK_DECIMAL: + return XK_KP_DECIMAL; + case KeyEvent.VK_DIVIDE: + return XK_KP_DIVIDE; + case KeyEvent.VK_SPACE: + return ' '; + case KeyEvent.VK_COMMA: + return ','; + case KeyEvent.VK_PERIOD: + return '.'; + case KeyEvent.VK_SLASH: + return '/'; + case KeyEvent.VK_BACK_SLASH: + return '\\'; + case KeyEvent.VK_SEMICOLON: + return ';'; + case KeyEvent.VK_QUOTE: + return '\''; + case KeyEvent.VK_OPEN_BRACKET: + return '['; + case KeyEvent.VK_CLOSE_BRACKET: + return ']'; + case KeyEvent.VK_BACK_QUOTE: + return '`'; + case KeyEvent.VK_MINUS: + return '-'; + case KeyEvent.VK_EQUALS: + return '='; + default: + return unicodeFallback(event.getKeyChar()); + } + } + + /** + * Unicode keysym fallback ({@code 0x01000000 | codepoint}) for + * characters outside the keyCode-based map, e.g. ä, ö, é. + */ + private static long unicodeFallback(final char keyChar) { + if (keyChar == KeyEvent.CHAR_UNDEFINED || keyChar < 0x20) + return -1; + if (keyChar < 0x7F) + // printable ASCII not covered above + return keyChar; + return 0x01000000L | keyChar; + } +} diff --git a/src/main/java/eu/svjatoslav/aukio/bridge/x11/GuiAppSession.java b/src/main/java/eu/svjatoslav/aukio/bridge/x11/GuiAppSession.java new file mode 100644 index 0000000..d1d16d4 --- /dev/null +++ b/src/main/java/eu/svjatoslav/aukio/bridge/x11/GuiAppSession.java @@ -0,0 +1,418 @@ +/* + * Aukio spatial computing environment. Author: Svjatoslav Agejenko. + * This project is released under Creative Commons Zero (CC0) license. + */ +package eu.svjatoslav.aukio.bridge.x11; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.List; +import java.util.Map; + +/** + * A GUI application running on a private Xvfb display, with its screen + * continuously captured into a caller-owned ARGB pixel buffer. + * + *

The session owns three things: the Xvfb server, the application + * process, and a capture thread that grabs the root window at a fixed + * rate. The capture thread writes directly into the destination array + * (typically a texture's primary bitmap pixels) and invokes the frame + * listener only when the image actually changed, so a static screen costs + * no repaints.

+ */ +public final class GuiAppSession implements AutoCloseable { + + /** + * Called on the capture thread after the destination buffer was + * updated with a changed frame. + */ + public interface FrameListener { + void frameCaptured(); + } + + private static final long CAPTURE_INTERVAL_MS = 100; + + private final int width; + private final int height; + private final List command; + private final Map extraEnvironment; + private final boolean privateDbusSession; + private final String windowTitleFilter; + private final File logFile; + + private XvfbServer server; + private Process application; + private X11Native.Connection connection; + private Thread captureThread; + private Thread fitterThread; + private volatile boolean running; + + public GuiAppSession(final int width, final int height, + final List command, + final Map extraEnvironment) { + this(width, height, command, extraEnvironment, false, null); + } + + /** + * @param privateDbusSession wrap the app in {@code dbus-run-session} + * so it gets a private session bus. Needed + * because merely unsetting + * DBUS_SESSION_BUS_ADDRESS is not enough: + * GIO then falls back to the systemd user + * bus socket (/run/user/$UID/bus), where the + * app's desktop instance can still be found + * and command-line handling forwarded to it. + */ + public GuiAppSession(final int width, final int height, + final List command, + final Map extraEnvironment, + final boolean privateDbusSession) { + this(width, height, command, extraEnvironment, privateDbusSession, + null); + } + + /** + * @param windowTitleFilter when non-null, a background thread + * periodically moves/resizes every top-level + * window whose name contains this string to + * fill the whole virtual screen. For + * applications that cannot be sized on the + * command line (VMware Workstation). + */ + public GuiAppSession(final int width, final int height, + final List command, + final Map extraEnvironment, + final boolean privateDbusSession, + final String windowTitleFilter) { + this.width = width; + this.height = height; + this.command = command; + this.extraEnvironment = extraEnvironment; + this.privateDbusSession = privateDbusSession; + this.windowTitleFilter = windowTitleFilter; + // per-application log file: several sessions run side by side + // and a shared file would be truncated by each launch + final String logName = command.isEmpty() ? "app" + : command.get(0).replaceAll("[^A-Za-z0-9_.-]", "_"); + logFile = new File("/tmp/aukio-xapp-" + logName + ".log"); + } + + /** + * Starts Xvfb, launches the application onto it, and begins capturing. + * + * @param destination ARGB pixel buffer of exactly + * {@code width * height} ints; every captured frame + * is written into this array + * @param listener notified (on the capture thread) whenever a frame + * changed the buffer + */ + public void start(final int[] destination, final FrameListener listener) + throws IOException { + if (destination.length != width * height) + throw new IllegalArgumentException("destination buffer holds " + + destination.length + " pixels, expected " + + (width * height)); + + server = XvfbServer.start(width, height); + connection = connectWithRetry(server.getDisplayName()); + + final List effectiveCommand = privateDbusSession + ? java.util.stream.Stream.concat( + java.util.stream.Stream.of("dbus-run-session", "--"), + command.stream()).toList() + : command; + final ProcessBuilder builder = new ProcessBuilder(effectiveCommand); + final Map environment = builder.environment(); + environment.put("DISPLAY", server.getDisplayName()); + // Firefox/GTK apps locate an already-running instance through the + // SESSION D-Bus (GApplication), ignoring --no-remote and DISPLAY: + // the window then pops up on the user's desktop while our Xvfb + // screen stays black. Hide the session bus so the app is forced + // to be its own primary instance on the private display. + environment.remove("DBUS_SESSION_BUS_ADDRESS"); + environment.put("MOZ_NO_REMOTE", "1"); + // On Wayland desktops (GNOME/Mutter) Firefox ignores DISPLAY + // entirely and renders natively into the user's compositor — + // the window pops up on the real screen while the Xvfb display + // stays windowless. Force the X11 backend onto our display. + environment.remove("WAYLAND_DISPLAY"); + environment.put("MOZ_ENABLE_WAYLAND", "0"); + // GTK (Firefox, VMware UI) tries its Wayland backend FIRST and + // then passes the X11 DISPLAY string to wl_display_connect, + // which resolves it as $XDG_RUNTIME_DIR/ — that socket + // does not exist, and GTK gives up without ever trying X11 + // ("Error: cannot open display: :9x", zero X11-unix connects in + // strace). Pin the x11 backend so the attempt order cannot + // regress; the private display is X11-only by design. + environment.put("GDK_BACKEND", "x11"); + environment.putAll(extraEnvironment); + builder.redirectOutput(logFile); + builder.redirectError(logFile); + application = builder.start(); + + running = true; + captureThread = new Thread( + () -> captureLoop(destination, listener), "x11-capture"); + captureThread.setDaemon(true); + captureThread.start(); + + if (windowTitleFilter != null) { + fitterThread = new Thread(this::windowFitterLoop, + "x11-window-fitter"); + fitterThread.setDaemon(true); + fitterThread.start(); + } + } + + /** + * Periodically fits windows matching {@link #windowTitleFilter} to + * fill the virtual screen. Runs for the session's lifetime: the + * application may (re)position its windows long after startup, and + * one pass per second is cheap. + */ + private void windowFitterLoop() { + while (running) { + try { + X11Native.fitNamedWindows(connection, windowTitleFilter, + width, height); + } catch (final Throwable throwable) { + // window set churn (map/unmap races) must never kill the + // fitter + if (Boolean.getBoolean("aukio.xcapture.debug")) + throwable.printStackTrace(); + } + try { + Thread.sleep(1000); + } catch (final InterruptedException e) { + return; + } + } + } + + /** + * Opens the session's X connection, retrying briefly. There is a + * real race at Xvfb birth: the readiness probe inside + * {@link XvfbServer#start} connects and disconnects, and the server + * resets when its last client goes away — a connect issued in the + * next few milliseconds can be refused, which used to surface as a + * fatal {@code IllegalStateException: cannot open X display} right + * after a successful probe (observed twice in one day on the + * low-power-profile box, once on the user's desktop run). + */ + private static X11Native.Connection connectWithRetry( + final String displayName) throws IOException { + final long deadline = System.currentTimeMillis() + 10_000; + Throwable lastFailure = null; + while (System.currentTimeMillis() < deadline) { + try { + return new X11Native.Connection(displayName); + } catch (final Throwable failure) { + lastFailure = failure; + try { + Thread.sleep(100); + } catch (final InterruptedException e) { + Thread.currentThread().interrupt(); + break; + } + } + } + throw new IOException("cannot open X display " + displayName + + " after 10 s of retries", lastFailure); + } + + private void captureLoop(final int[] destination, + final FrameListener listener) { + final boolean debug = Boolean.getBoolean("aukio.xcapture.debug"); + int iterations = 0; + while (running) { + try { + final boolean changed = X11Native.captureRoot(connection, + width, height, destination); + if (debug && iterations++ % 10 == 0) { + final var distinct = new java.util.HashSet(); + for (int i = 0; i < destination.length; i += 97) + distinct.add(destination[i] & 0xFFFFFF); + System.out.println("XCAPTURE changed=" + changed + + " distinct=" + distinct.size() + + " appAlive=" + isApplicationRunning()); + } + if (changed) + listener.frameCaptured(); + } catch (final Throwable throwable) { + // capture failures (window closing, renderer racing the + // buffer) must never kill the capture thread + if (debug) + throwable.printStackTrace(); + } + try { + Thread.sleep(CAPTURE_INTERVAL_MS); + } catch (final InterruptedException e) { + return; + } + } + } + + /** + * Forwards a keyboard event (press or release) into the virtual + * display. Modifier keys are forwarded as ordinary keys, so the + * server-side modifier state tracks the user's real keyboard. + * + * @param event the AWT key event + * @param pressed {@code true} for key press, {@code false} for release + */ + public void sendKeyEvent(final java.awt.event.KeyEvent event, + final boolean pressed) { + if (connection == null || !isApplicationRunning()) + return; + final long keysym = AwtKeysyms.keysymFor(event); + if (keysym < 0) + return; + if (Boolean.getBoolean("aukio.xcapture.debug")) + System.out.println("XINPUT key keysym=0x" + + Long.toHexString(keysym) + (pressed ? " press" : " release") + + " on " + server.getDisplayName()); + X11Native.sendKeyEvent(connection, keysym, pressed); + } + + /** + * Releases Shift/Control/Alt server-side. Called when keyboard focus + * is lost mid-modifier (e.g. Shift+ESC pops focus while Shift is + * held), so the browser is not left with stuck modifiers. + */ + public void releaseModifiers() { + if (connection == null || !isApplicationRunning()) + return; + X11Native.sendKeyEvent(connection, AwtKeysyms.XK_SHIFT_L, false); + X11Native.sendKeyEvent(connection, AwtKeysyms.XK_CONTROL_L, false); + X11Native.sendKeyEvent(connection, AwtKeysyms.XK_ALT_L, false); + } + + /** + * Forwards scroll wheel input into the virtual display at the given + * screen coordinates (typically the current pointer hover position on + * the panel). + * + * @param verticalUnits positive = scroll down, negative = scroll up + * @param horizontalUnits positive = scroll right, negative = left + */ + public void sendScroll(final int verticalUnits, final int horizontalUnits, + final int x, final int y) { + if (connection == null || !isApplicationRunning()) + return; + final int clampedX = Math.max(0, Math.min(x, width - 1)); + final int clampedY = Math.max(0, Math.min(y, height - 1)); + X11Native.sendScroll(connection, clampedX, clampedY, verticalUnits, + horizontalUnits); + } + + /** + * Moves the pointer inside the virtual display without pressing + * buttons (hover forwarding). + */ + public void sendMouseMove(final int x, final int y) { + if (connection == null || !isApplicationRunning()) + return; + X11Native.sendMouseMove(connection, + Math.max(0, Math.min(x, width - 1)), + Math.max(0, Math.min(y, height - 1))); + } + + public boolean isApplicationRunning() { + return application != null && application.isAlive(); + } + + /** + * The private X display name (e.g. ":91"), or null before + * {@link #start} — useful for inspecting the session with standard + * tools (xwininfo, import). + */ + public String getDisplayName() { + return server == null ? null : server.getDisplayName(); + } + + /** + * Forwards a mouse click into the virtual display at screen + * coordinates (x, y). The captured texture IS the virtual screen, so + * texture pixels map 1:1 onto these coordinates. The click is + * synthesized with the X TEST extension, which applications treat as + * real device input. + * + * @param x screen X coordinate (clamped into the screen) + * @param y screen Y coordinate (clamped into the screen) + * @param button X button number: 1 = left, 3 = right + */ + public void sendMouseClick(final int x, final int y, final int button) { + if (connection == null || !isApplicationRunning()) + return; + final int clampedX = Math.max(0, Math.min(x, width - 1)); + final int clampedY = Math.max(0, Math.min(y, height - 1)); + if (Boolean.getBoolean("aukio.xcapture.debug")) + System.out.println("XINPUT click at " + clampedX + "," + clampedY + + " button " + button + " on " + + server.getDisplayName()); + X11Native.sendMouseClick(connection, clampedX, clampedY, button); + } + + @Override + public void close() { + running = false; + if (captureThread != null) + captureThread.interrupt(); + if (fitterThread != null) + fitterThread.interrupt(); + if (application != null) + application.destroyForcibly(); + if (connection != null) + connection.close(); + if (server != null) + server.close(); + } + + /** + * Builds a session that runs Firefox on a fresh throwaway profile, + * sized to fill the whole virtual screen. + */ + public static GuiAppSession firefox(final int width, final int height, + final String url) + throws IOException { + final File profile = Files.createTempDirectory( + "aukio-firefox-profile").toFile(); + + // keep first-run noise out of the captured screen + final String userJs = """ + user_pref("browser.shell.checkDefaultBrowser", false); + user_pref("browser.aboutwelcome.enabled", false); + user_pref("datareporting.policy.dataSubmissionEnabled", false); + user_pref("datareporting.policy.firstRunURL", ""); + user_pref("trailhead.firstrun.didSeeAboutWelcome", true); + // stop Debian's system-wide extensions (Web eID) from + // hijacking the first-run session with an active tab + user_pref("extensions.autoDisableScopes", 15); + user_pref("extensions.shownSelectionUI", true); + """; + Files.writeString(new File(profile, "user.js").toPath(), userJs); + + return new GuiAppSession(width, height, + List.of("firefox", "--no-remote", "--new-instance", + "--profile", profile.getAbsolutePath(), + "--width", String.valueOf(width), + "--height", String.valueOf(height), + url), + Map.of(), true); + } + + /** + * Builds a session that runs VMware Workstation sized to fill the + * whole virtual screen. VMware has no command-line size flags, so a + * background thread continuously moves/resizes windows titled + * "VMware" onto the full screen. + */ + public static GuiAppSession vmware(final int width, final int height) { + // Qt must use the X11 backend even when the desktop session is + // Wayland; the private display is X11-only (GuiAppSession also + // removes WAYLAND_DISPLAY for every app) + return new GuiAppSession(width, height, List.of("vmware"), + Map.of("QT_QPA_PLATFORM", "xcb"), false, "VMware"); + } +} diff --git a/src/main/java/eu/svjatoslav/aukio/bridge/x11/X11Native.java b/src/main/java/eu/svjatoslav/aukio/bridge/x11/X11Native.java new file mode 100644 index 0000000..3e61819 --- /dev/null +++ b/src/main/java/eu/svjatoslav/aukio/bridge/x11/X11Native.java @@ -0,0 +1,433 @@ +/* + * Aukio spatial computing environment. Author: Svjatoslav Agejenko. + * This project is released under Creative Commons Zero (CC0) license. + */ +package eu.svjatoslav.aukio.bridge.x11; + +import com.sun.jna.Function; +import com.sun.jna.Library; +import com.sun.jna.Native; +import com.sun.jna.NativeLong; +import com.sun.jna.Pointer; +import com.sun.jna.ptr.IntByReference; +import com.sun.jna.ptr.NativeLongByReference; +import com.sun.jna.ptr.PointerByReference; + +/** + * Minimal libX11 bindings for screen capture via JNA. + * + *

Only what {@link X11Capture} needs: open/close a display, resolve the + * root window, and {@code XGetImage} a rectangular area into ARGB pixels. + * The XImage struct is read through manual field offsets (amd64 layout) + * instead of a JNA {@code Structure} — the struct embeds function pointers + * and exact alignment is easier to get right explicitly.

+ * + *

Linux-only by design; the workspace targets Linux.

+ */ +final class X11Native { + + /** ZPixmap format constant for XGetImage. */ + static final int Z_PIXMAP = 2; + + /** XImage field offsets (amd64 / LP64 layout). */ + private static final int IMAGE_WIDTH = 0; + private static final int IMAGE_HEIGHT = 4; + private static final int IMAGE_DATA = 16; + private static final int IMAGE_BYTE_ORDER = 24; + private static final int IMAGE_DEPTH = 40; + private static final int IMAGE_BYTES_PER_LINE = 44; + private static final int IMAGE_BITS_PER_PIXEL = 48; + private static final int IMAGE_RED_MASK = 56; + private static final int IMAGE_GREEN_MASK = 64; + private static final int IMAGE_BLUE_MASK = 72; + private static final int IMAGE_DESTROY_FUNCTION = 96; + + private static final int LSB_FIRST = 0; + + private interface X11Lib extends Library { + X11Lib INSTANCE = Native.load("X11", X11Lib.class); + + Pointer XOpenDisplay(String displayName); + + int XCloseDisplay(Pointer display); + + int XDefaultScreen(Pointer display); + + NativeLong XRootWindow(Pointer display, int screenNumber); + + Pointer XGetImage(Pointer display, NativeLong drawable, int x, int y, + int width, int height, NativeLong planeMask, + int format); + + int XFlush(Pointer display); + + /** + * Converts a keysym to a keycode in the server's current keymap. + * Returns 0 when the keysym is not present in the keymap. + */ + int XKeysymToKeycode(Pointer display, NativeLong keysym); + + /** + * Lists the children of a window in stacking order (bottom first). + * + * @return 0 on failure; on success childrenReturn holds a malloc'd + * Window array that the caller must release with {@link #XFree} + */ + int XQueryTree(Pointer display, NativeLong window, + NativeLongByReference rootReturn, + NativeLongByReference parentReturn, + PointerByReference childrenReturn, + IntByReference nchildrenReturn); + + /** + * Reads a window's name (WM_NAME). + * + * @return 0 when the window has no name; on success nameReturn + * holds a malloc'd string that the caller must release with + * {@link #XFree} + */ + int XFetchName(Pointer display, NativeLong window, + PointerByReference nameReturn); + + int XFree(Pointer data); + + int XMoveResizeWindow(Pointer display, NativeLong window, int x, + int y, int width, int height); + + int XRaiseWindow(Pointer display, NativeLong window); + } + + /** + * libXtst (X TEST extension) bindings for synthesizing input. Fake + * input events are indistinguishable from real device events, unlike + * XSendEvent which many toolkits (GTK included) treat as untrusted. + */ + private interface XTestLib extends Library { + XTestLib INSTANCE = Native.load("Xtst", XTestLib.class); + + int XTestFakeMotionEvent(Pointer display, int screenNumber, + int x, int y, NativeLong delay); + + int XTestFakeButtonEvent(Pointer display, int button, + boolean isPress, NativeLong delay); + + /** + * Synthesizes a key press or release. + * + * @return nonzero on success + */ + int XTestFakeKeyEvent(Pointer display, int keycode, boolean isPress, + NativeLong delay); + } + + /** + * An open connection to an X server. + * + *

libX11 is not thread-safe unless XInitThreads was called; all + * requests on this connection are instead serialized through + * {@link #lock} (the capture loop and the input injector run on + * different threads).

+ */ + static final class Connection { + final Pointer display; + final NativeLong rootWindow; + final int screenNumber; + /** Serializes all X protocol traffic on this connection. */ + final Object lock = new Object(); + + Connection(final String displayName) { + display = X11Lib.INSTANCE.XOpenDisplay(displayName); + if (display == null) + throw new IllegalStateException( + "cannot open X display " + displayName); + screenNumber = X11Lib.INSTANCE.XDefaultScreen(display); + rootWindow = X11Lib.INSTANCE.XRootWindow(display, screenNumber); + } + + void close() { + synchronized (lock) { + if (display != null) + X11Lib.INSTANCE.XCloseDisplay(display); + } + } + } + + /** + * Grabs the top-left rectangle of the root window and converts it to + * ARGB, honoring the server's pixel masks, bits-per-pixel and byte + * order. Writes directly into {@code destination} and reports whether + * any pixel changed. + * + * @return true when at least one written pixel differs from the + * previous buffer content + */ + static boolean captureRoot(final Connection connection, + final int width, final int height, + final int[] destination) { + synchronized (connection.lock) { + final Pointer image = X11Lib.INSTANCE.XGetImage(connection.display, + connection.rootWindow, 0, 0, width, height, + new NativeLong(-1L), Z_PIXMAP); + if (image == null) + return false; + try { + return convert(image, width, height, destination); + } finally { + // XDestroyImage is a macro calling image->f.destroy_image + final Pointer destroyFunction = image.getPointer( + IMAGE_DESTROY_FUNCTION); + Function.getFunction(destroyFunction).invokeInt( + new Object[]{image}); + } + } + } + + /** + * Sends a single key press or release, identified by X keysym, to the + * virtual display. Modifier state is NOT synthesized here — real + * modifier key events (Shift/Ctrl/Alt) are forwarded like any other + * key, so the server-side modifier state tracks the user's keyboard. + * + * @param keysym the X keysym (see {@link AwtKeysyms}) + * @param press {@code true} for press, {@code false} for release + */ + static void sendKeyEvent(final Connection connection, final long keysym, + final boolean press) { + synchronized (connection.lock) { + final int keycode = X11Lib.INSTANCE.XKeysymToKeycode( + connection.display, new NativeLong(keysym)); + if (keycode == 0) + // keysym not present in the server keymap; nothing to send + return; + XTestLib.INSTANCE.XTestFakeKeyEvent(connection.display, keycode, + press, new NativeLong(0)); + X11Lib.INSTANCE.XFlush(connection.display); + } + } + + /** + * Sends scroll wheel input at the given root-window coordinates: moves + * the pointer there, then presses/releases the X scroll buttons + * (4 = up, 5 = down, 6 = left, 7 = right), one click per notch. + * + * @param verticalUnits positive = scroll down, negative = scroll up + * @param horizontalUnits positive = scroll right, negative = left + */ + static void sendScroll(final Connection connection, final int x, + final int y, final int verticalUnits, + final int horizontalUnits) { + synchronized (connection.lock) { + XTestLib.INSTANCE.XTestFakeMotionEvent(connection.display, + connection.screenNumber, x, y, new NativeLong(0)); + final int verticalButton = verticalUnits < 0 ? 4 : 5; + for (int i = 0; i < Math.abs(verticalUnits); i++) { + XTestLib.INSTANCE.XTestFakeButtonEvent(connection.display, + verticalButton, true, new NativeLong(0)); + XTestLib.INSTANCE.XTestFakeButtonEvent(connection.display, + verticalButton, false, new NativeLong(0)); + } + final int horizontalButton = horizontalUnits < 0 ? 6 : 7; + for (int i = 0; i < Math.abs(horizontalUnits); i++) { + XTestLib.INSTANCE.XTestFakeButtonEvent(connection.display, + horizontalButton, true, new NativeLong(0)); + XTestLib.INSTANCE.XTestFakeButtonEvent(connection.display, + horizontalButton, false, new NativeLong(0)); + } + X11Lib.INSTANCE.XFlush(connection.display); + } + } + + /** + * Moves the pointer to the given root-window coordinates without + * pressing any button (hover forwarding). + */ + static void sendMouseMove(final Connection conn, final int x, + final int y) { + synchronized (conn.lock) { + XTestLib.INSTANCE.XTestFakeMotionEvent(conn.display, + conn.screenNumber, x, y, new NativeLong(0)); + X11Lib.INSTANCE.XFlush(conn.display); + } + } + + /** + * Synthesizes a full mouse click (move pointer, press, release) at the + * given root-window coordinates. The pointer is moved first because a + * button press acts at the current pointer location. The button is + * held briefly because Firefox does not synthesize a DOM click from a + * zero-duration press+release. + * + * @param x root-window X coordinate + * @param y root-window Y coordinate + * @param button X button number (1 = left, 3 = right) + */ + static void sendMouseClick(final Connection connection, final int x, + final int y, final int button) { + synchronized (connection.lock) { + XTestLib.INSTANCE.XTestFakeMotionEvent(connection.display, + connection.screenNumber, x, y, new NativeLong(0)); + XTestLib.INSTANCE.XTestFakeButtonEvent(connection.display, button, + true, new NativeLong(0)); + X11Lib.INSTANCE.XFlush(connection.display); + // Firefox (EventStateManager) does not synthesize a DOM click + // from a zero-duration press+release; hold the button briefly + try { + Thread.sleep(80); + } catch (final InterruptedException e) { + Thread.currentThread().interrupt(); + } + XTestLib.INSTANCE.XTestFakeButtonEvent(connection.display, button, + false, new NativeLong(0)); + X11Lib.INSTANCE.XFlush(connection.display); + } + } + + /** + * Moves and resizes every top-level window whose name contains + * {@code titleFilter} to fill the whole virtual screen, raising it to + * the top of the stacking order. Used for applications that cannot be + * sized on the command line (e.g. VMware Workstation): a background + * thread calls this periodically so late-appearing windows get fitted + * too. + * + * @return the number of windows fitted in this pass + */ + static int fitNamedWindows(final Connection connection, + final String titleFilter, + final int width, final int height) { + synchronized (connection.lock) { + final NativeLongByReference rootReturn = new NativeLongByReference(); + final NativeLongByReference parentReturn = new NativeLongByReference(); + final PointerByReference childrenReturn = new PointerByReference(); + final IntByReference countReturn = new IntByReference(); + if (X11Lib.INSTANCE.XQueryTree(connection.display, + connection.rootWindow, rootReturn, parentReturn, + childrenReturn, countReturn) == 0) + return 0; + final Pointer children = childrenReturn.getValue(); + int fitted = 0; + try { + final int count = countReturn.getValue(); + if (children == null || count <= 0) + return 0; + for (int i = 0; i < count; i++) { + final NativeLong window = new NativeLong( + children.getNativeLong((long) i * NativeLong.SIZE) + .longValue()); + final PointerByReference nameReturn = + new PointerByReference(); + if (X11Lib.INSTANCE.XFetchName(connection.display, window, + nameReturn) == 0) + continue; // unnamed window + final Pointer name = nameReturn.getValue(); + if (name == null) + continue; + try { + final String title = name.getString(0); + if (title == null || !title.contains(titleFilter)) + continue; + X11Lib.INSTANCE.XMoveResizeWindow(connection.display, + window, 0, 0, width, height); + X11Lib.INSTANCE.XRaiseWindow(connection.display, + window); + fitted++; + } finally { + X11Lib.INSTANCE.XFree(name); + } + } + } finally { + X11Lib.INSTANCE.XFree(children); + } + if (fitted > 0) + X11Lib.INSTANCE.XFlush(connection.display); + return fitted; + } + } + + private static boolean convert(final Pointer image, final int width, + final int height, final int[] destination) { + final int imageWidth = image.getInt(IMAGE_WIDTH); + final int imageHeight = image.getInt(IMAGE_HEIGHT); + final Pointer data = image.getPointer(IMAGE_DATA); + final int byteOrder = image.getInt(IMAGE_BYTE_ORDER); + final int bytesPerLine = image.getInt(IMAGE_BYTES_PER_LINE); + final int bitsPerPixel = image.getInt(IMAGE_BITS_PER_PIXEL); + final long redMask = image.getLong(IMAGE_RED_MASK); + final long greenMask = image.getLong(IMAGE_GREEN_MASK); + final long blueMask = image.getLong(IMAGE_BLUE_MASK); + + final Channel red = new Channel(redMask); + final Channel green = new Channel(greenMask); + final Channel blue = new Channel(blueMask); + + boolean changed = false; + final int copyWidth = Math.min(width, imageWidth); + final int copyHeight = Math.min(height, imageHeight); + for (int y = 0; y < copyHeight; y++) { + final int rowOffset = y * bytesPerLine; + final int destinationRow = y * width; + for (int x = 0; x < copyWidth; x++) { + final long pixelValue = readPixel(data, rowOffset, x, + bitsPerPixel, byteOrder); + final int argb = 0xFF000000 + | (red.extract(pixelValue) << 16) + | (green.extract(pixelValue) << 8) + | blue.extract(pixelValue); + final int index = destinationRow + x; + if (destination[index] != argb) { + destination[index] = argb; + changed = true; + } + } + } + return changed; + } + + private static long readPixel(final Pointer data, final int rowOffset, + final int x, final int bitsPerPixel, + final int byteOrder) { + if (bitsPerPixel == 32) { + final int value = data.getInt(rowOffset + (long) x * 4); + return Integer.toUnsignedLong( + byteOrder == LSB_FIRST ? value : Integer.reverseBytes(value)); + } + if (bitsPerPixel == 24) { + final long offset = rowOffset + (long) x * 3; + final int b0 = data.getByte(offset) & 0xFF; + final int b1 = data.getByte(offset + 1) & 0xFF; + final int b2 = data.getByte(offset + 2) & 0xFF; + return byteOrder == LSB_FIRST + ? b0 | (b1 << 8) | (b2 << 16) + : (b0 << 16) | (b1 << 8) | b2; + } + if (bitsPerPixel == 16) { + final short value = data.getShort(rowOffset + (long) x * 2); + return Short.toUnsignedLong( + byteOrder == LSB_FIRST ? value : Short.reverseBytes(value)); + } + throw new IllegalStateException( + "unsupported bits_per_pixel " + bitsPerPixel); + } + + /** + * Extracts one 8-bit color channel from a raw pixel using the server + * pixel mask (any position and width, scaled to 0..255). + */ + private static final class Channel { + private final long mask; + private final int shift; + private final long maxValue; + + Channel(final long mask) { + this.mask = mask; + shift = mask == 0 ? 0 : Long.numberOfTrailingZeros(mask); + maxValue = mask == 0 ? 1 : (mask >>> shift); + } + + int extract(final long pixelValue) { + if (mask == 0) + return 0; + return (int) (((pixelValue & mask) >>> shift) * 255 / maxValue); + } + } +} diff --git a/src/main/java/eu/svjatoslav/aukio/bridge/x11/XvfbServer.java b/src/main/java/eu/svjatoslav/aukio/bridge/x11/XvfbServer.java new file mode 100644 index 0000000..73acb99 --- /dev/null +++ b/src/main/java/eu/svjatoslav/aukio/bridge/x11/XvfbServer.java @@ -0,0 +1,116 @@ +/* + * Aukio spatial computing environment. Author: Svjatoslav Agejenko. + * This project is released under Creative Commons Zero (CC0) license. + */ +package eu.svjatoslav.aukio.bridge.x11; + +import java.io.File; +import java.io.IOException; + +/** + * Lifecycle of a private {@code Xvfb} virtual X server that one captured + * GUI application renders into. + * + *

Each server gets its own display number (scanned upward from 90, + * skipping displays with an existing lock file or socket), so multiple + * captured apps never share a screen. Readiness is detected by polling for + * the unix socket and then proving the display with a real + * {@code XOpenDisplay} — just waiting for the socket is not enough.

+ */ +public final class XvfbServer implements AutoCloseable { + + private static final int FIRST_DISPLAY = 90; + private static final int LAST_DISPLAY = 200; + private static final long START_TIMEOUT_MS = 10_000; + + private final int displayNumber; + private final Process process; + + private XvfbServer(final int displayNumber, final Process process) { + this.displayNumber = displayNumber; + this.process = process; + } + + /** + * Starts an Xvfb server with a single screen of the given size and + * waits until the display actually accepts connections. + * + * @throws IOException if Xvfb fails to start within the timeout + */ + public static XvfbServer start(final int width, final int height) + throws IOException { + final int displayNumber = findFreeDisplay(); + final Process process = new ProcessBuilder("Xvfb", + ":" + displayNumber, + "-screen", "0", width + "x" + height + "x24", + "-nolisten", "tcp") + .redirectOutput(new File("/tmp/aukio-xvfb-" + + displayNumber + ".log")) + .redirectError(new File("/tmp/aukio-xvfb-" + + displayNumber + ".log")) + .start(); + + final XvfbServer server = new XvfbServer(displayNumber, process); + final long deadline = System.currentTimeMillis() + START_TIMEOUT_MS; + while (System.currentTimeMillis() < deadline) { + if (!process.isAlive()) + break; + if (new File(server.socketPath()).exists()) { + try { + final X11Native.Connection probe = + new X11Native.Connection(server.getDisplayName()); + probe.close(); + return server; + } catch (final Throwable ignored) { + // socket exists but server not accepting yet + } + } + try { + Thread.sleep(100); + } catch (final InterruptedException e) { + Thread.currentThread().interrupt(); + break; + } + } + process.destroyForcibly(); + throw new IOException("Xvfb :" + displayNumber + + " did not become ready within " + START_TIMEOUT_MS + " ms"); + } + + private static int findFreeDisplay() throws IOException { + for (int number = FIRST_DISPLAY; number <= LAST_DISPLAY; number++) + if (!new File("/tmp/.X" + number + "-lock").exists() + && !new File(socketPath(number)).exists()) + return number; + throw new IOException("no free X display number between " + + FIRST_DISPLAY + " and " + LAST_DISPLAY); + } + + private static String socketPath(final int displayNumber) { + return "/tmp/.X11-unix/X" + displayNumber; + } + + private String socketPath() { + return socketPath(displayNumber); + } + + /** + * Display name for the {@code DISPLAY} environment variable + * (e.g. {@code ":93"}). + */ + public String getDisplayName() { + return ":" + displayNumber; + } + + @Override + public void close() { + process.destroy(); + try { + if (!process.waitFor(2, java.util.concurrent.TimeUnit.SECONDS)) + process.destroyForcibly(); + } catch (final InterruptedException e) { + Thread.currentThread().interrupt(); + process.destroyForcibly(); + } + } +} diff --git a/src/main/java/eu/svjatoslav/aukio/core/Main.java b/src/main/java/eu/svjatoslav/aukio/core/Main.java new file mode 100644 index 0000000..04782c8 --- /dev/null +++ b/src/main/java/eu/svjatoslav/aukio/core/Main.java @@ -0,0 +1,955 @@ +/* + * Aukio spatial computing environment. Author: Svjatoslav Agejenko. + * This project is released under Creative Commons Zero (CC0) license. + */ +package eu.svjatoslav.aukio.core; + +import eu.svjatoslav.aukio.workspace.FirefoxPanel; +import eu.svjatoslav.aukio.workspace.TerminalPanel; +import eu.svjatoslav.aukio.workspace.Workspace; + +/** + * Entry point for the Aukio spatial computing environment. + * + *

Usage: {@code aukio [--selftest]}

+ * + *

{@code --selftest} opens the workspace, types {@code ls} and an + * arithmetic echo into the terminal programmatically, verifies the expected + * output appears on the terminal screen, prints the screen contents, and + * exits with status 0 (pass) or 1 (fail). Intended to run under + * {@code xvfb-run} in CI-style verification.

+ */ +public class Main { + + private static final long SELFTEST_TIMEOUT_MS = 15_000; + + public static void main(final String[] args) throws Exception { + // persistent log + telemetry FIRST, so all boot output is captured + eu.svjatoslav.aukio.e3d.diag.Diagnostics.install(); + + String environmentName = null; + for (final String arg : args) + if (arg.startsWith("--env=")) + environmentName = arg.substring("--env=".length()); + + if (hasArg(args, "--clicktest")) + prepareClickTestPage(); + + if (hasArg(args, "--keytest")) + prepareKeyTestPage(); + + if (hasArg(args, "--scrolltest")) + prepareScrollTestPage(); + + if (hasArg(args, "--hovertest")) + prepareHoverTestPage(); + + final Workspace workspace = new Workspace(); + try { + workspace.open(); + + if (environmentName != null) + openRequestedEnvironment(workspace, environmentName); + } catch (final Throwable throwable) { + // a failed startup (e.g. a GUI app's X display would not + // open) must not leave a half-open workspace rendering + // forever: main() dying alone does not stop the JVM — the + // view's render/present threads keep it alive as a zombie + // that holds its Xvfb displays and PTYs hostage + throwable.printStackTrace(); + System.err.println("startup failed, exiting"); + System.exit(1); + } + + if (hasArg(args, "--selftest")) + System.exit(runSelfTest(workspace)); + + if (hasArg(args, "--guitest")) + System.exit(runGuiTest(workspace)); + + if (hasArg(args, "--clicktest")) + System.exit(runClickTest(workspace)); + + if (hasArg(args, "--keytest")) + System.exit(runKeyTest(workspace)); + + if (hasArg(args, "--scrolltest")) + System.exit(runScrollTest(workspace)); + + if (hasArg(args, "--hovertest")) + System.exit(runHoverTest(workspace)); + + if (hasArg(args, "--headtest")) + System.exit(runHeadTest(workspace)); + + if (hasArg(args, "--headdebug")) + startHeadDebugLogger(workspace); + + Runtime.getRuntime().addShutdownHook( + new Thread(workspace::close)); + } + + /** + * True when the exact flag appears anywhere on the command line; + * test drivers must not be position-sensitive now that --env= can + * occupy args[0]. + */ + private static boolean hasArg(final String[] args, final String flag) { + for (final String arg : args) + if (flag.equals(arg)) + return true; + return false; + } + + /** + * Resolves the --env= provider, checks availability, and opens + * it into the running workspace. Exits with status 1 when the name + * is unknown or the provider's data files are missing. + */ + private static void openRequestedEnvironment(final Workspace workspace, + final String name) + throws Exception { + final var provider + = eu.svjatoslav.aukio.env.EnvironmentRegistry.findByName(name); + if (provider == null) { + System.out.println("no environment named '" + name + "'"); + eu.svjatoslav.aukio.env.EnvironmentRegistry + .printAvailableEnvironments(); + System.exit(1); + } + if (!provider.isAvailable()) { + System.out.println("environment '" + name + "' (" + + provider.getDisplayName() + ") is not available on" + + " this machine — its data files were not found"); + System.exit(1); + } + System.out.println("opening environment: " + + provider.getDisplayName()); + workspace.openEnvironment(provider); + } + + /** + * Drives the terminal without AWT events: injects bytes straight into + * the PTY, exactly what a focused keystroke would send. + */ + private static int runSelfTest(final Workspace workspace) + throws InterruptedException { + final TerminalPanel terminal = workspace.getTerminalPanel(); + final var terminals = workspace.getTerminalPanels(); + final int expectedTerminalCount = Workspace.TERMINAL_GRID_COLUMNS + * Workspace.TERMINAL_GRID_ROWS; + if (terminals.size() != expectedTerminalCount) + return fail(terminal, "expected " + expectedTerminalCount + + " terminal panels, got " + terminals.size()); + final long sessionCount = terminals.stream() + .map(TerminalPanel::getSession) + .distinct() + .count(); + if (sessionCount != terminals.size()) + return fail(terminal, "terminal panels share PTY sessions: " + + sessionCount + " sessions for " + terminals.size() + + " panels"); + final long runningSessionCount = terminals.stream() + .filter(panel -> panel.getSession().isRunning()) + .count(); + if (runningSessionCount != terminals.size()) + return fail(terminal, "only " + runningSessionCount + " of " + + terminals.size() + " PTY sessions are running"); + + // wait for the real shell prompt (bashrc noise may contain a bare + // "$", so match the user@host prompt text instead) + if (!waitFor(terminal, expectedShellPrompt())) + return fail(terminal, "no shell prompt appeared"); + + terminal.typeText("ls\r"); + if (!waitFor(terminal, "pom.xml")) + return fail(terminal, "'ls' output missing pom.xml"); + if (!waitFor(terminal, "src")) + return fail(terminal, "'ls' output missing src"); + + terminal.typeText("echo $((6*7))\r"); + if (!waitFor(terminal, "42")) + return fail(terminal, "arithmetic echo produced no 42"); + + // unique sentinel that exists ONLY on the main screen; mc/htop + // draw on the alternate screen, and mc shows both its own + // user@host command-line prompt AND file timestamps that can + // contain substrings like "42" — only a made-up string is a + // non-vacuous proof of being back at the shell + terminal.typeText("echo SENTINEL42XYZ\r"); + if (!waitFor(terminal, "SENTINEL42XYZ")) + return fail(terminal, "sentinel echo failed"); + + // fullscreen curses programs: mc must open (and enable + // application cursor keys), quit on F10; htop must quit on 'q'; + // mc must also quit on ESC 0 (proves ESC reaches programs) + terminal.typeText("mc\r"); + if (!waitFor(terminal, "Name")) + return fail(terminal, "mc did not open"); + if (!terminal.isApplicationCursorKeys()) + return fail(terminal, + "mc did not enable application cursor keys"); + terminal.typeText("\u001B[21~"); // F10 + if (!waitFor(terminal, "SENTINEL42XYZ")) + return fail(terminal, "mc did not quit on F10"); + + terminal.typeText("htop\r"); + if (!waitFor(terminal, "Tasks")) + return fail(terminal, "htop did not open"); + terminal.typeText("q"); + if (!waitFor(terminal, "SENTINEL42XYZ")) + return fail(terminal, "htop did not quit on 'q'"); + + terminal.typeText("mc\r"); + if (!waitFor(terminal, "Name")) + return fail(terminal, "mc did not reopen"); + terminal.typeText("\u001B"); + terminal.typeText("0"); // ESC 0 = F10 in mc + if (!waitFor(terminal, "SENTINEL42XYZ")) + return fail(terminal, "mc did not quit on ESC 0"); + + System.out.println("SELFTEST PASS — terminal screen:"); + System.out.println(terminal.getScreenText()); + workspace.close(); + return 0; + } + + /** + * Prints head tracker diagnostics every 2 seconds (frame flow, tick, + * dt, max gyro excursion, fused angles) so a live look-around test + * can be verified from the process log afterwards. + */ + private static void startHeadDebugLogger(final Workspace workspace) { + final Thread logger = new Thread(() -> { + while (true) { + final var tracker = workspace.getHeadTracker(); + System.out.println("HEADTRACK " + + (tracker == null ? "no device" + : tracker.getDebugString())); + try { + Thread.sleep(2000); + } catch (final InterruptedException e) { + return; + } + } + }, "head-debug-logger"); + logger.setDaemon(true); + logger.start(); + } + + /** + * Prints head tracker and camera angles for 30 seconds so axis/sign + * conventions can be verified against real head movement. Exit 0 if + * the tracker produced data at all. + */ + private static int runHeadTest(final Workspace workspace) + throws InterruptedException { + final var tracker = workspace.getHeadTracker(); + if (tracker == null) { + System.out.println("HEADTEST FAIL: no glasses detected"); + return 1; + } + // frame listeners do not run under xvfb (no render loop), so do + // the boot recenter here and compute what HeadLookController + // would apply, in-line + while (!tracker.isCalibrated()) + Thread.sleep(50); + Thread.sleep(1000); // let the filter settle + tracker.recenter(); + final var camera = workspace.getViewFrame().getViewPanel() + .getCamera(); + final double[] baseAngles = camera.getTransform().getRotation() + .toAngles(); + System.out.println("wait for calibration, then: turn head LEFT, " + + "watch yaw; nod DOWN, watch pitch (30s)"); + final long deadline = System.currentTimeMillis() + 30_000; + boolean sawMovement = false; + while (System.currentTimeMillis() < deadline) { + final double headYaw = tracker.getLookYaw(); + final double headPitch = tracker.getLookPitch(); + // frame listeners do not run under xvfb (no render loop), so + // compute what HeadLookController would apply, in-line + final double camYaw = baseAngles[0] - headYaw; + final double camPitch = baseAngles[1] + headPitch; + if (Math.abs(headYaw) > 0.05 || Math.abs(headPitch) > 0.05) + sawMovement = true; + System.out.printf("head yaw %+7.1f pitch %+7.1f deg | " + + "camera yaw %+7.3f pitch %+7.3f rad%n", + Math.toDegrees(headYaw), Math.toDegrees(headPitch), + camYaw, camPitch); + Thread.sleep(200); + } + workspace.close(); + System.out.println(sawMovement ? "HEADTEST PASS (movement seen)" + : "HEADTEST FAIL (no head movement detected)"); + return sawMovement ? 0 : 1; + } + + /** + * Verifies the GUI-app bridge end to end without AWT events: waits + * for Firefox to paint into the panel texture, then checks the + * captured frame is a real image (not a flat fill). Exit 0 = pass. + */ + private static int runGuiTest(final Workspace workspace) + throws InterruptedException { + final var panel = workspace.getFirefoxPanel(); + final long deadline = System.currentTimeMillis() + 60_000; + while (System.currentTimeMillis() < deadline) { + if (panel.hasLiveFrame()) + break; + Thread.sleep(500); + } + if (!panel.hasLiveFrame()) { + System.out.println("GUITEST FAIL: no live frame within 60s" + + " (firefox running: " + + panel.getSession().isApplicationRunning() + ")"); + workspace.close(); + return 1; + } + + // a real browser screen has many distinct colors; a dead/black + // capture collapses to one or two + final int[] pixels = panel.getTexture().primaryBitmap.pixels; + final var distinctColors = new java.util.HashSet(); + long luminanceSum = 0; + int samples = 0; + for (int i = 0; i < pixels.length; i += 997) { + final int pixel = pixels[i]; + distinctColors.add(pixel); + luminanceSum += ((pixel >> 16) & 0xFF) + ((pixel >> 8) & 0xFF) + + (pixel & 0xFF); + samples++; + } + final double meanLuminance = (double) luminanceSum / samples / 3.0; + System.out.println("GUITEST stats: distinct sampled colors=" + + distinctColors.size() + " mean luminance=" + + String.format("%.1f", meanLuminance)); + if (distinctColors.size() < 16 || meanLuminance < 5) { + System.out.println("GUITEST FAIL: captured frame looks flat"); + workspace.close(); + return 1; + } + + System.out.println("GUITEST PASS — firefox is painting into the" + + " workspace panel"); + workspace.close(); + return 0; + } + + /** + * Verifies mouse click forwarding into the virtual display: loads a + * test page that paints the whole screen red on click, focuses the + * Firefox panel, sends a left click through the panel's mouse + * interaction path, and requires a large pixel change. Also verifies + * that a middle click releases focus and is NOT forwarded. Exit 0 = + * pass. + */ + private static int runClickTest(final Workspace workspace) + throws Exception { + final var panel = workspace.getFirefoxPanel(); + final long deadline = System.currentTimeMillis() + 60_000; + while (System.currentTimeMillis() < deadline) { + if (panel.hasLiveFrame()) + break; + Thread.sleep(500); + } + if (!panel.hasLiveFrame()) { + System.out.println("CLICKTEST FAIL: no live frame within 60s"); + workspace.close(); + return 1; + } + // let the page finish rendering after the first live frame + Thread.sleep(3000); + + final int[] pixels = panel.getTexture().primaryBitmap.pixels; + final int[] before = pixels.clone(); + + // click while unfocused: grabs focus only, page must not change + panel.mouseClicked( + eu.svjatoslav.aukio.e3d.gui.humaninput.MouseEvent.BUTTON_LEFT, + 640, 480); + Thread.sleep(2000); + if (countChangedPixels(before, pixels) > 1000) { + System.out.println("CLICKTEST FAIL: focus-grab click leaked" + + " into the page"); + workspace.close(); + return 1; + } + if (!panel.hasKeyboardFocus()) { + System.out.println("CLICKTEST FAIL: first click did not focus" + + " the panel"); + workspace.close(); + return 1; + } + + // left click while focused: forwarded, page turns red + System.out.println("CLICKTEST focus=" + panel.hasKeyboardFocus() + + " — sending forwarded click"); + panel.mouseClicked( + eu.svjatoslav.aukio.e3d.gui.humaninput.MouseEvent.BUTTON_LEFT, + 640, 480); + final long clickDeadline = System.currentTimeMillis() + 10_000; + int changed = 0; + while (System.currentTimeMillis() < clickDeadline) { + Thread.sleep(500); + changed = countChangedPixels(before, pixels); + if (changed > 100_000) + break; + } + if (changed <= 100_000) { + saveTexturePng(panel, "/tmp/clicktest-texture.png"); + System.out.println("CLICKTEST FAIL: forwarded click changed only " + + changed + " pixels (texture dump: " + + "/tmp/clicktest-texture.png)"); + workspace.close(); + return 1; + } + + // mouse back button: navigates back to the gradient page + final int[] redPage = pixels.clone(); + panel.mouseClicked( + eu.svjatoslav.aukio.e3d.gui.humaninput.MouseEvent.BUTTON_BACK, + 640, 480); + int backChanged = awaitPixelChange(pixels, redPage, 10_000); + if (backChanged <= 100_000) { + System.out.println("CLICKTEST FAIL: back button did not" + + " navigate back (" + backChanged + " pixels)"); + workspace.close(); + return 1; + } + + // mouse forward button: navigates forward to the red page + final int[] gradientPage = pixels.clone(); + panel.mouseClicked( + eu.svjatoslav.aukio.e3d.gui.humaninput.MouseEvent.BUTTON_FORWARD, + 640, 480); + int forwardChanged = awaitPixelChange(pixels, gradientPage, 10_000); + if (forwardChanged <= 100_000) { + System.out.println("CLICKTEST FAIL: forward button did not" + + " navigate forward (" + forwardChanged + " pixels)"); + workspace.close(); + return 1; + } + + // middle click: releases focus, not forwarded + final int[] red = pixels.clone(); + panel.mouseClicked( + eu.svjatoslav.aukio.e3d.gui.humaninput.MouseEvent.BUTTON_MIDDLE, + 640, 480); + Thread.sleep(2000); + if (panel.hasKeyboardFocus()) { + System.out.println("CLICKTEST FAIL: middle click did not" + + " release focus"); + workspace.close(); + return 1; + } + if (countChangedPixels(red, pixels) > 1000) { + System.out.println("CLICKTEST FAIL: middle click leaked into" + + " the page"); + workspace.close(); + return 1; + } + + System.out.println("CLICKTEST PASS — focus grab, forwarded click (" + + changed + " pixels changed), back/forward navigation," + + " middle-click release"); + workspace.close(); + return 0; + } + + /** + * Writes the hover-target page and points the Firefox panel at it. + * The page turns orange on a mousemove event in its top-left quadrant + * (clientX < 300 && clientY < 300). Must run BEFORE the + * workspace is constructed. + */ + private static void prepareHoverTestPage() throws Exception { + final java.io.File page = java.io.File.createTempFile( + "aukio-hovertest", ".html"); + page.deleteOnExit(); + java.nio.file.Files.writeString(page.toPath(), """ + HOVERTEST + +
+ + """); + System.setProperty("aukio.firefox.url", page.toURI().toString()); + } + + /** + * Verifies hover forwarding: while unfocused, hovering the panel must + * NOT move the pointer inside Firefox; once focused, hovering the + * top-left area must deliver a mousemove there (page turns orange). + */ + private static int runHoverTest(final Workspace workspace) + throws Exception { + final FirefoxPanel panel = workspace.getFirefoxPanel(); + if (panel == null) { + System.out.println("HOVERTEST FAIL: no firefox panel"); + workspace.close(); + return 1; + } + final long deadline = System.currentTimeMillis() + 60_000; + while (System.currentTimeMillis() < deadline) { + if (panel.hasLiveFrame()) + break; + Thread.sleep(500); + } + if (!panel.hasLiveFrame()) { + System.out.println("HOVERTEST FAIL: no live firefox capture"); + workspace.close(); + return 1; + } + Thread.sleep(2_000); + final int[] pixels = panel.getTexture().primaryBitmap.pixels; + + // unfocused hover must NOT reach the page + final int[] beforeHover = pixels.clone(); + panel.mouseHover(150, 200); + Thread.sleep(2_000); + if (countChangedPixels(beforeHover, pixels) > 100_000) { + System.out.println("HOVERTEST FAIL: unfocused hover leaked" + + " into the page"); + workspace.close(); + return 1; + } + + // focus (click at 640,480 — forwarded pointer lands outside the + // top-left quadrant, page stays unchanged) + panel.mouseClicked( + eu.svjatoslav.aukio.e3d.gui.humaninput.MouseEvent.BUTTON_LEFT, + 640, 480); + Thread.sleep(1_500); + if (!panel.hasKeyboardFocus()) { + System.out.println("HOVERTEST FAIL: focus was not acquired"); + workspace.close(); + return 1; + } + + // focused hover into the top-left quadrant must turn it orange + final int[] focused = pixels.clone(); + panel.mouseHover(150, 200); + final int changed = awaitPixelChange(pixels, focused, 10_000); + if (changed <= 100_000) { + System.out.println("HOVERTEST FAIL: focused hover changed only " + + changed + " pixels"); + workspace.close(); + return 1; + } + + System.out.println("HOVERTEST PASS — unfocused hover not forwarded," + + " focused hover moved the pointer (" + changed + + " pixels changed)"); + workspace.close(); + return 0; + } + + private static int awaitPixelChange(final int[] pixels, + final int[] reference, + final long timeoutMs) + throws InterruptedException { + final long deadline = System.currentTimeMillis() + timeoutMs; + int changed = 0; + while (System.currentTimeMillis() < deadline) { + Thread.sleep(500); + changed = countChangedPixels(reference, pixels); + if (changed > 100_000) + break; + } + return changed; + } + + /** + * Verifies keyboard forwarding into the virtual display: the test page + * has an input field that turns the whole screen green once it + * receives the text "hello". The panel is focused, the input is + * clicked (focus it in the page), then "hello" is typed as AWT key + * events through the panel's keyboard path. Also verifies that + * Shift+ESC releases focus. Exit 0 = pass. + */ + private static int runKeyTest(final Workspace workspace) + throws Exception { + final var panel = workspace.getFirefoxPanel(); + final long deadline = System.currentTimeMillis() + 60_000; + while (System.currentTimeMillis() < deadline) { + if (panel.hasLiveFrame()) + break; + Thread.sleep(500); + } + if (!panel.hasLiveFrame()) { + System.out.println("KEYTEST FAIL: no live frame within 60s"); + workspace.close(); + return 1; + } + Thread.sleep(3000); + + final int[] pixels = panel.getTexture().primaryBitmap.pixels; + + // focus the panel (first click grabs focus only) + panel.mouseClicked( + eu.svjatoslav.aukio.e3d.gui.humaninput.MouseEvent.BUTTON_LEFT, + 640, 480); + if (!panel.hasKeyboardFocus()) { + System.out.println("KEYTEST FAIL: first click did not focus" + + " the panel"); + workspace.close(); + return 1; + } + + // click the input field (page coords 600,280 + chrome offset + // lands it near texture 700,390) so typed text goes into it + panel.mouseClicked( + eu.svjatoslav.aukio.e3d.gui.humaninput.MouseEvent.BUTTON_LEFT, + 700, 390); + Thread.sleep(1500); + + // type "hello" as raw key events through the panel keyboard path + final int[] before = pixels.clone(); + final var viewPanel = workspace.getViewFrame().getViewPanel(); + for (final char c : "hello".toCharArray()) { + final int keyCode = java.awt.event.KeyEvent.getExtendedKeyCodeForChar(c); + panel.keyPressed(new java.awt.event.KeyEvent(viewPanel, + java.awt.event.KeyEvent.KEY_PRESSED, + System.currentTimeMillis(), 0, keyCode, c), viewPanel); + panel.keyReleased(new java.awt.event.KeyEvent(viewPanel, + java.awt.event.KeyEvent.KEY_RELEASED, + System.currentTimeMillis(), 0, keyCode, c), viewPanel); + Thread.sleep(150); + } + + // the page turns green once the input contains "hello" + final long typeDeadline = System.currentTimeMillis() + 15_000; + int changed = 0; + while (System.currentTimeMillis() < typeDeadline) { + Thread.sleep(500); + changed = countChangedPixels(before, pixels); + if (changed > 100_000) + break; + } + if (changed <= 100_000) { + saveTexturePng(panel, "/tmp/keytest-texture.png"); + System.out.println("KEYTEST FAIL: typing changed only " + + changed + " pixels (texture dump: " + + "/tmp/keytest-texture.png)"); + workspace.close(); + return 1; + } + + // Shift+ESC releases focus + final int shiftMods = java.awt.event.InputEvent.SHIFT_DOWN_MASK; + panel.keyPressed(new java.awt.event.KeyEvent(viewPanel, + java.awt.event.KeyEvent.KEY_PRESSED, + System.currentTimeMillis(), shiftMods, + java.awt.event.KeyEvent.VK_SHIFT, + java.awt.event.KeyEvent.CHAR_UNDEFINED), viewPanel); + panel.keyPressed(new java.awt.event.KeyEvent(viewPanel, + java.awt.event.KeyEvent.KEY_PRESSED, + System.currentTimeMillis(), shiftMods, + java.awt.event.KeyEvent.VK_ESCAPE, ''), viewPanel); + Thread.sleep(500); + if (panel.hasKeyboardFocus()) { + System.out.println("KEYTEST FAIL: Shift+ESC did not release" + + " focus"); + workspace.close(); + return 1; + } + + System.out.println("KEYTEST PASS — typed text reached the page (" + + changed + " pixels changed), Shift+ESC released focus"); + workspace.close(); + return 0; + } + + /** + * Verifies scroll wheel forwarding on both axes: + *
    + *
  • Firefox: the test page counts wheel events (deltaY and + * deltaX) and turns blue once it has seen 2 notches down and + * 2 notches left.
    • + *
  • Terminal: the wheel becomes arrow keys — wheel-up at a prompt + * with a drafted line recalls history, wheel-left moves the + * cursor inside the drafted line.
    • + *
+ * Exit 0 = pass. + */ + private static int runScrollTest(final Workspace workspace) + throws Exception { + final var panel = workspace.getFirefoxPanel(); + final long deadline = System.currentTimeMillis() + 60_000; + while (System.currentTimeMillis() < deadline) { + if (panel.hasLiveFrame()) + break; + Thread.sleep(500); + } + if (!panel.hasLiveFrame()) { + System.out.println("SCROLLTEST FAIL: no live frame within 60s"); + workspace.close(); + return 1; + } + Thread.sleep(3000); + + // --- firefox: both wheel axes reach the page --- + panel.mouseClicked( + eu.svjatoslav.aukio.e3d.gui.humaninput.MouseEvent.BUTTON_LEFT, + 640, 480); + if (!panel.hasKeyboardFocus()) { + System.out.println("SCROLLTEST FAIL: first click did not focus" + + " the panel"); + workspace.close(); + return 1; + } + + final int[] pixels = panel.getTexture().primaryBitmap.pixels; + final int[] before = pixels.clone(); + panel.mouseWheelMoved(2, 0); // two notches down + Thread.sleep(300); + panel.mouseWheelMoved(0, -2); // two notches left + final long scrollDeadline = System.currentTimeMillis() + 15_000; + int changed = 0; + while (System.currentTimeMillis() < scrollDeadline) { + Thread.sleep(500); + changed = countChangedPixels(before, pixels); + if (changed > 100_000) + break; + } + if (changed <= 100_000) { + saveTexturePng(panel, "/tmp/scrolltest-texture.png"); + System.out.println("SCROLLTEST FAIL: wheel changed only " + + changed + " pixels (texture dump: " + + "/tmp/scrolltest-texture.png)"); + workspace.close(); + return 1; + } + + // --- terminal: wheel becomes arrow keys --- + final TerminalPanel terminal = workspace.getTerminalPanel(); + if (!waitFor(terminal, expectedShellPrompt())) { + System.out.println("SCROLLTEST FAIL: no shell prompt appeared"); + workspace.close(); + return 1; + } + // controlled history: wheel-up is 3 arrow-ups per notch, so after + // A, B, C the recalled line is deterministically A + terminal.typeText("echo WHEEL_A\r"); + if (!waitFor(terminal, "WHEEL_A")) { + System.out.println("SCROLLTEST FAIL: base command A failed"); + workspace.close(); + return 1; + } + terminal.typeText("echo WHEEL_B\r"); + Thread.sleep(700); + terminal.typeText("echo WHEEL_C\r"); + Thread.sleep(700); + + // draft a partial command, then wheel up: bash history recall + // (3 up-arrows per notch) replaces the draft with WHEEL_A's line + terminal.typeText("echo DRAFT"); + Thread.sleep(500); + terminal.mouseWheelMoved(-1, 0); + Thread.sleep(1000); + final String screenAfterUp = terminal.getScreenText().trim(); + if (!screenAfterUp.endsWith("echo WHEEL_A")) { + System.out.println("SCROLLTEST FAIL: wheel-up did not recall" + + " history; screen:\n" + screenAfterUp); + workspace.close(); + return 1; + } + + // wheel left: cursor moves into the recalled line, typed text + // lands in the middle of it: "echo WHEEL_A" (12 chars), 6 + // cursor-lefts -> insert at position 6 -> "echo WQHEEL_A" + terminal.mouseWheelMoved(0, -2); // 6 cursor-lefts + Thread.sleep(500); + terminal.typeText("Q"); + Thread.sleep(500); + if (!terminal.getScreenText().trim().endsWith("echo WQHEEL_A")) { + System.out.println("SCROLLTEST FAIL: horizontal wheel did not" + + " move the cursor left; screen:\n" + + terminal.getScreenText()); + workspace.close(); + return 1; + } + + System.out.println("SCROLLTEST PASS — firefox wheel both axes (" + + changed + " pixels changed), terminal wheel-as-arrows"); + workspace.close(); + return 0; + } + + /** + * Writes the scroll-target page and points the Firefox panel at it. + * Must run BEFORE the workspace is constructed, because the panel + * reads the URL system property in its constructor. + */ + private static void prepareScrollTestPage() throws Exception { + final java.io.File page = java.io.File.createTempFile( + "aukio-scrolltest", ".html"); + page.deleteOnExit(); + java.nio.file.Files.writeString(page.toPath(), """ + SCROLL none + +
+
+ Scroll down twice and left twice.
+ + """); + System.setProperty("aukio.firefox.url", page.toURI().toString()); + } + + /** + * Writes the keyboard-target page and points the Firefox panel at it. + * Must run BEFORE the workspace is constructed, because the panel + * reads the URL system property in its constructor. + */ + private static void prepareKeyTestPage() throws Exception { + final java.io.File page = java.io.File.createTempFile( + "aukio-keytest", ".html"); + page.deleteOnExit(); + java.nio.file.Files.writeString(page.toPath(), """ + KEYTEST none + +
+ + + """); + System.setProperty("aukio.firefox.url", page.toURI().toString()); + } + + /** + * Writes the click-target page and points the Firefox panel at it. + * Must run BEFORE the workspace is constructed, because the panel + * reads the URL system property in its constructor. + */ + private static void prepareClickTestPage() throws Exception { + // two pages linked by real navigation, so the mouse back/forward + // buttons have history to walk + final java.io.File page2 = java.io.File.createTempFile( + "aukio-clicktest-2", ".html"); + page2.deleteOnExit(); + java.nio.file.Files.writeString(page2.toPath(), """ + WITNESS page2 + +

CLICKED

+ + """); + + final java.io.File page1 = java.io.File.createTempFile( + "aukio-clicktest-1", ".html"); + page1.deleteOnExit(); + java.nio.file.Files.writeString(page1.toPath(), """ + WITNESS none + +
+
+ Click anywhere to navigate to the red page.
+ + """.replace("PAGE2URL", page2.toURI().toString())); + System.setProperty("aukio.firefox.url", page1.toURI().toString()); + } + + private static void saveTexturePng(final eu.svjatoslav.aukio.workspace.FirefoxPanel panel, + final String path) throws Exception { + final int w = eu.svjatoslav.aukio.workspace.FirefoxPanel.CAPTURE_WIDTH; + final int h = eu.svjatoslav.aukio.workspace.FirefoxPanel.CAPTURE_HEIGHT; + final var image = new java.awt.image.BufferedImage(w, h, + java.awt.image.BufferedImage.TYPE_INT_ARGB); + image.setRGB(0, 0, w, h, panel.getTexture().primaryBitmap.pixels, 0, w); + javax.imageio.ImageIO.write(image, "png", new java.io.File(path)); + } + + private static int countChangedPixels(final int[] a, final int[] b) { + int changed = 0; + for (int i = 0; i < a.length; i++) + if (a[i] != b[i]) + changed++; + return changed; + } + + /** + * Expected interactive shell prompt prefix ("user@host") for the + * account running the selftest. Derived at runtime so the test + * works on any machine, not just the author's. + */ + private static String expectedShellPrompt() { + String host; + try { + host = java.net.InetAddress.getLocalHost().getHostName(); + final int dot = host.indexOf('.'); + if (dot > 0) + host = host.substring(0, dot); // bash \h is the short name + } catch (final Exception e) { + host = "localhost"; + } + return System.getProperty("user.name") + "@" + host; + } + + private static boolean waitFor(final TerminalPanel terminal, + final String needle) + throws InterruptedException { + final long deadline = System.currentTimeMillis() + + SELFTEST_TIMEOUT_MS; + while (System.currentTimeMillis() < deadline) { + if (terminal.getScreenText().contains(needle)) + return true; + Thread.sleep(100); + } + return false; + } + + private static int fail(final TerminalPanel terminal, + final String reason) { + System.out.println("SELFTEST FAIL: " + reason); + System.out.println("terminal screen:"); + System.out.println(terminal.getScreenText()); + return 1; + } +} diff --git a/src/main/java/eu/svjatoslav/aukio/env/EnvironmentContext.java b/src/main/java/eu/svjatoslav/aukio/env/EnvironmentContext.java new file mode 100644 index 0000000..b847353 --- /dev/null +++ b/src/main/java/eu/svjatoslav/aukio/env/EnvironmentContext.java @@ -0,0 +1,71 @@ +/* + * Aukio spatial computing environment. Author: Svjatoslav Agejenko. + * This project is released under Creative Commons Zero (CC0) license. + */ +package eu.svjatoslav.aukio.env; + +import eu.svjatoslav.aukio.e3d.geometry.Camera; +import eu.svjatoslav.aukio.e3d.gui.FrameListener; +import eu.svjatoslav.aukio.e3d.gui.ViewPanel; +import eu.svjatoslav.aukio.e3d.renderer.raster.ShapeCollection; + +/** + * Host services handed to an {@link EnvironmentProvider} when it is + * opened. This is deliberately a narrow window into the workspace: the + * scene to add shapes to, the camera to place at the world's spawn + * point, and the per-frame tick for streaming worlds. + */ +public final class EnvironmentContext { + + private final ViewPanel viewPanel; + + public EnvironmentContext(final ViewPanel viewPanel) { + this.viewPanel = viewPanel; + } + + /** + * The root scene graph. The environment adds its shapes here and + * keeps references to whatever it must later remove in close(). + */ + public ShapeCollection getScene() { + return viewPanel.getRootShapeCollection(); + } + + /** + * The workspace camera. An environment typically moves it to its + * spawn point in open(); afterwards the user owns it (fly controls, + * head tracking, SpaceNavigator). + */ + public Camera getCamera() { + return viewPanel.getCamera(); + } + + /** + * Registers a per-frame callback — the streaming world's heartbeat: + * "recompute what should exist around the camera." Return true from + * the callback when the scene changed and needs a repaint. + */ + public void addFrameListener(final FrameListener listener) { + viewPanel.addFrameListener(listener); + } + + public void removeFrameListener(final FrameListener listener) { + viewPanel.removeFrameListener(listener); + } + + /** + * Requests a repaint after scene changes made outside a frame + * callback (e.g. from a background loader thread). + */ + public void requestRepaint() { + viewPanel.repaintDuringNextViewUpdate(); + } + + /** + * Escape hatch to the underlying view panel. Prefer the narrower + * methods above; this exists for things like reading viewport size. + */ + public ViewPanel getViewPanel() { + return viewPanel; + } +} diff --git a/src/main/java/eu/svjatoslav/aukio/env/EnvironmentProvider.java b/src/main/java/eu/svjatoslav/aukio/env/EnvironmentProvider.java new file mode 100644 index 0000000..a55880f --- /dev/null +++ b/src/main/java/eu/svjatoslav/aukio/env/EnvironmentProvider.java @@ -0,0 +1,77 @@ +/* + * Aukio spatial computing environment. Author: Svjatoslav Agejenko. + * This project is released under Creative Commons Zero (CC0) license. + */ +package eu.svjatoslav.aukio.env; + +/** + * A pluggable 3D environment ("world provider") for the Aukio workspace: + * a game world, a map, or a procedural scene that the workspace panels + * live inside of. Today the wallpaper is a Fallout 4 level, tomorrow a + * Quake map, the day after a 3D fractal — the workspace does not care, + * it only knows this interface. + * + *

Implementations are discovered in two ways:

+ *
    + *
  • built-in providers, registered directly in + * {@link EnvironmentRegistry};
    • + *
  • external providers on the classpath, discovered via + * {@link java.util.ServiceLoader} — a provider jar declares itself in + * {@code META-INF/services/eu.svjatoslav.aukio.env.EnvironmentProvider}. + * This is how separate environment projects (e.g. + * aukio-environment-fo4) plug in without Aukio depending on them.
    • + *
+ * + *

Selection: {@code aukio --env=} picks a provider by + * {@link #getName()}.

+ * + *

Lifecycle: {@link #open(EnvironmentContext)} is called once after the + * workspace window exists. The provider builds its initial scene into + * {@link EnvironmentContext#getScene()}, positions the camera via + * {@link EnvironmentContext#getCamera()}, and — for streaming worlds — + * registers a per-frame callback via + * {@link EnvironmentContext#addFrameListener} whose job is "recompute what + * should exist around the camera, add/remove shapes accordingly." + * {@link #close()} must unregister listeners and release resources.

+ * + *

What an environment is NOT: it never touches the engine internals, + * never creates windows, and never interferes with workspace panels — + * it only adds shapes to the shared scene and moves nothing it does not + * own.

+ */ +public interface EnvironmentProvider { + + /** + * Short unique selector used on the command line, e.g. "fallout4", + * "quake1", "fractal". Lower case, no spaces. + */ + String getName(); + + /** + * Human-readable name, e.g. "Fallout 4 — the Commonwealth". + */ + String getDisplayName(); + + /** + * True when this provider can actually run on this machine — for + * game worlds this means the game data files were found. Providers + * that need no external data (procedural worlds) always return true. + */ + boolean isAvailable(); + + /** + * Builds the initial scene and starts any background streaming. + * Called once, after the workspace window exists. + * + * @param context host services: the scene, the camera, frame ticks + * @throws Exception if the environment cannot be built (for game + * worlds: data files unreadable/corrupt) + */ + void open(EnvironmentContext context) throws Exception; + + /** + * Stops background work and removes the environment's shapes from + * the scene. Must tolerate being called after a failed open(). + */ + void close(); +} diff --git a/src/main/java/eu/svjatoslav/aukio/env/EnvironmentRegistry.java b/src/main/java/eu/svjatoslav/aukio/env/EnvironmentRegistry.java new file mode 100644 index 0000000..454cc7c --- /dev/null +++ b/src/main/java/eu/svjatoslav/aukio/env/EnvironmentRegistry.java @@ -0,0 +1,61 @@ +/* + * Aukio spatial computing environment. Author: Svjatoslav Agejenko. + * This project is released under Creative Commons Zero (CC0) license. + */ +package eu.svjatoslav.aukio.env; + +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.ServiceLoader; + +/** + * Finds the environments that can run on this machine: built-in + * providers plus any declared on the classpath via + * {@code META-INF/services/eu.svjatoslav.aukio.env.EnvironmentProvider} + * (the plug-in mechanism used by separate environment projects such as + * aukio-environment-fo4). + */ +public final class EnvironmentRegistry { + + private EnvironmentRegistry() { + } + + /** + * All discovered providers, available or not. Built-ins first, then + * classpath plug-ins; keyed by name, later duplicates lose. + */ + public static List discoverProviders() { + final Map providers + = new LinkedHashMap<>(); + providers.put("grid", new GridEnvironmentProvider()); + for (final EnvironmentProvider provider + : ServiceLoader.load(EnvironmentProvider.class)) + providers.putIfAbsent(provider.getName(), provider); + return new ArrayList<>(providers.values()); + } + + /** + * Looks up a provider by its command-line name; null when no + * provider with that name is on the classpath. + */ + public static EnvironmentProvider findByName(final String name) { + for (final EnvironmentProvider provider : discoverProviders()) + if (provider.getName().equals(name)) + return provider; + return null; + } + + /** + * Prints the known providers and their availability, for CLI help + * and for the error message when a requested provider is missing. + */ + public static void printAvailableEnvironments() { + System.out.println("known environments:"); + for (final EnvironmentProvider provider : discoverProviders()) + System.out.println(" " + provider.getName() + + (provider.isAvailable() ? "" : " (unavailable)") + + " — " + provider.getDisplayName()); + } +} diff --git a/src/main/java/eu/svjatoslav/aukio/env/GridEnvironmentProvider.java b/src/main/java/eu/svjatoslav/aukio/env/GridEnvironmentProvider.java new file mode 100644 index 0000000..4516d95 --- /dev/null +++ b/src/main/java/eu/svjatoslav/aukio/env/GridEnvironmentProvider.java @@ -0,0 +1,54 @@ +/* + * Aukio spatial computing environment. Author: Svjatoslav Agejenko. + * This project is released under Creative Commons Zero (CC0) license. + */ +package eu.svjatoslav.aukio.env; + +import eu.svjatoslav.aukio.e3d.geometry.Rectangle; +import eu.svjatoslav.aukio.e3d.math.Transform; +import eu.svjatoslav.aukio.e3d.renderer.raster.shapes.basic.line.LineAppearance; +import eu.svjatoslav.aukio.e3d.renderer.raster.shapes.composite.wireframe.Grid2D; + +import static eu.svjatoslav.aukio.e3d.renderer.raster.Color.hex; + +/** + * Built-in proof environment: a huge ground grid under the workspace. + * Exists to validate the provider contract (discovery, lifecycle, camera + * placement) without any external data — and as the reference + * implementation for how little a working provider can be. + */ +public class GridEnvironmentProvider implements EnvironmentProvider { + + @Override + public String getName() { + return "grid"; + } + + @Override + public String getDisplayName() { + return "Ground grid (built-in test environment)"; + } + + @Override + public boolean isAvailable() { + return true; + } + + @Override + public void open(final EnvironmentContext context) { + // ground plane at y=0, camera 150 units above it (negative Y is + // up in engine coordinates), looking slightly down + final Transform transform = Transform.fromAngles(0, 0, 0, 0, + Math.PI / 2, 0); + context.getScene().addShape(new Grid2D(transform, + new Rectangle(20000), 40, 40, + new LineAppearance(10, hex("5a5a3a")))); + context.getCamera().getTransform().set(0, -150, -300, 0, -0.4, 0); + context.requestRepaint(); + } + + @Override + public void close() { + // keeps no references and no background work — nothing to undo + } +} diff --git a/src/main/java/eu/svjatoslav/aukio/workspace/FirefoxPanel.java b/src/main/java/eu/svjatoslav/aukio/workspace/FirefoxPanel.java new file mode 100644 index 0000000..2e7692a --- /dev/null +++ b/src/main/java/eu/svjatoslav/aukio/workspace/FirefoxPanel.java @@ -0,0 +1,45 @@ +/* + * Aukio spatial computing environment. Author: Svjatoslav Agejenko. + * This project is released under Creative Commons Zero (CC0) license. + */ +package eu.svjatoslav.aukio.workspace; + +import eu.svjatoslav.aukio.bridge.x11.GuiAppSession; +import eu.svjatoslav.aukio.e3d.geometry.Point2D; +import eu.svjatoslav.aukio.e3d.gui.ViewPanel; +import eu.svjatoslav.aukio.e3d.math.Transform; + +import java.io.IOException; + +/** + * A live Firefox browser panel in 3D space. All capture and interaction + * machinery lives in {@link GuiAppPanel}; this class only defines the + * Firefox session. + */ +public class FirefoxPanel extends GuiAppPanel { + + /** + * Virtual screen (and texture) resolution: the browser window fills + * the whole Xvfb screen, so this is also the captured window size. + */ + public static final int CAPTURE_WIDTH = 1280; + public static final int CAPTURE_HEIGHT = 960; + + /** + * Creates the panel. Call {@link #start()} to launch the browser and + * begin streaming its screen into the texture. + * + * @param transform position in the world + * @param viewPanel the view panel this component belongs to + * @param sizeInWorldCoordinates panel size in world units; the browser + * capture is stretched to fill it + */ + public FirefoxPanel(final Transform transform, final ViewPanel viewPanel, + final Point2D sizeInWorldCoordinates) + throws IOException { + super(transform, viewPanel, sizeInWorldCoordinates, + CAPTURE_WIDTH, CAPTURE_HEIGHT, + GuiAppSession.firefox(CAPTURE_WIDTH, CAPTURE_HEIGHT, + System.getProperty("aukio.firefox.url", "about:home"))); + } +} diff --git a/src/main/java/eu/svjatoslav/aukio/workspace/GuiAppPanel.java b/src/main/java/eu/svjatoslav/aukio/workspace/GuiAppPanel.java new file mode 100644 index 0000000..04f9926 --- /dev/null +++ b/src/main/java/eu/svjatoslav/aukio/workspace/GuiAppPanel.java @@ -0,0 +1,251 @@ +/* + * Aukio spatial computing environment. Author: Svjatoslav Agejenko. + * This project is released under Creative Commons Zero (CC0) license. + */ +package eu.svjatoslav.aukio.workspace; + +import eu.svjatoslav.aukio.bridge.x11.GuiAppSession; +import eu.svjatoslav.aukio.e3d.geometry.Point2D; +import eu.svjatoslav.aukio.e3d.gui.GuiComponent; +import eu.svjatoslav.aukio.e3d.gui.ViewPanel; +import eu.svjatoslav.aukio.e3d.gui.humaninput.KeyboardFocusStack; +import eu.svjatoslav.aukio.e3d.math.Transform; +import eu.svjatoslav.aukio.e3d.renderer.raster.shapes.composite.TexturedRectangle; +import eu.svjatoslav.aukio.e3d.renderer.raster.texture.Texture; + +/** + * Base class for live GUI application panels in 3D space. + * + *

The application runs on a private Xvfb display (via + * {@link GuiAppSession}); a capture thread copies the virtual screen into + * the texture of a {@link TexturedRectangle}, so the application window + * appears as an object in the world — including in stereo/VR rendering, + * like every other shape.

+ * + *

Interaction: the first click focuses the panel; while focused, + * left/right clicks are forwarded into the application via the X TEST + * extension at the exact clicked position (the engine reports the + * perspective-correct texture coordinate of the hit), and all keyboard + * input is forwarded as X key events. Middle click or Shift+ESC releases + * focus and is never forwarded; plain ESC goes to the application. The + * capture thread writes the texture's primary bitmap directly and + * requests a repaint — the same benign writer/renderer race the terminal + * panel already accepts.

+ */ +public abstract class GuiAppPanel extends GuiComponent { + + protected final TexturedRectangle rectangle; + protected final GuiAppSession session; + protected final int captureWidth; + protected final int captureHeight; + private volatile boolean liveFrameSeen; + private double lastHoverU = Double.NaN; + private double lastHoverV = Double.NaN; + private int lastForwardedHoverX = -1; + private int lastForwardedHoverY = -1; + + /** + * Creates the panel. Call {@link #start()} to launch the application + * and begin streaming its screen into the texture. + * + * @param transform position in the world + * @param viewPanel the view panel this component belongs to + * @param sizeInWorldCoordinates panel size in world units; the capture + * is stretched to fill it + * @param captureWidth virtual screen (texture) width in pixels + * @param captureHeight virtual screen (texture) height in pixels + * @param session the application session to present + */ + protected GuiAppPanel(final Transform transform, final ViewPanel viewPanel, + final Point2D sizeInWorldCoordinates, + final int captureWidth, final int captureHeight, + final GuiAppSession session) { + super(transform, viewPanel, sizeInWorldCoordinates.to3D()); + this.captureWidth = captureWidth; + this.captureHeight = captureHeight; + + rectangle = new TexturedRectangle(new Transform(), + (int) sizeInWorldCoordinates.x, + (int) sizeInWorldCoordinates.y, + captureWidth, captureHeight, 1); + rectangle.setMouseInteractionController(this); + addShape(rectangle); + + this.session = session; + } + + /** + * Launches Xvfb + the application and starts the screen capture. + */ + public void start() throws java.io.IOException { + session.start(rectangle.getTexture().primaryBitmap.pixels, + this::onFrameCaptured); + } + + private void onFrameCaptured() { + try { + // captured pixels are already in the primary bitmap; drop the + // stale mipmaps so they regenerate lazily from the new frame + rectangle.getTexture().resetResampledBitmapCache(); + if (!liveFrameSeen) + liveFrameSeen = looksLikeRealScreen( + rectangle.getTexture().primaryBitmap.pixels); + viewPanel.repaintDuringNextViewUpdate(); + } catch (final Throwable throwable) { + // renderer may briefly reallocate internals (e.g. on window + // resize); never let that kill the capture thread + } + } + + /** + * A real application screen has many distinct colors; an empty Xvfb + * root (opaque black) collapses to a single one. The alpha channel + * is ignored: the very first capture of a black screen differs from + * the zero-initialized buffer in alpha only, and must not count. + */ + static boolean looksLikeRealScreen(final int[] pixels) { + final var distinctColors = new java.util.HashSet(); + // dense enough to hit antialiased text and icons: a very coarse + // stride sees only the few flat background colors of a mostly + // empty page and reports a live application screen as "flat" + for (int i = 0; i < pixels.length; i += 997) + distinctColors.add(pixels[i] & 0xFFFFFF); + return distinctColors.size() >= 16; + } + + /** + * Whether the application has painted a real screen into the texture + * (many distinct colors) — used by the automated GUI test. A black + * empty Xvfb root does not count. + */ + public boolean hasLiveFrame() { + return liveFrameSeen; + } + + /** + * The texture the captured application screen is written into. + */ + public Texture getTexture() { + return rectangle.getTexture(); + } + + public GuiAppSession getSession() { + return session; + } + + /** + * Focus and click-forwarding behavior: + *
    + *
  • click while unfocused: take keyboard focus (first click is a + * focus grab, not forwarded to the application)
    • + *
  • left/right click while focused: forwarded into the application + * at the clicked texture position (texture pixels map 1:1 onto + * the virtual screen)
    • + *
  • middle click: releases focus (workspace convention, like ESC) + * and is never forwarded to the application
    • + *
+ */ + @Override + public boolean mouseClicked(final int button, final double textureU, + final double textureV) { + // Direct-call path (click-injection tests synthesize clicks through + // this overload); the live engine dispatch uses the 4-arg overload. + return mouseClicked(button, textureU, textureV, + viewPanel.getKeyboardFocusStack()); + } + + @Override + public boolean mouseClicked(final int button, final double textureU, + final double textureV, + final KeyboardFocusStack focusStack) { + if (!hasKeyboardFocus() + || button == eu.svjatoslav.aukio.e3d.gui.humaninput.MouseEvent.BUTTON_MIDDLE + || Double.isNaN(textureU) || Double.isNaN(textureV)) + return super.mouseClicked(button, textureU, textureV, focusStack); + + // Mouse back/forward buttons are X buttons 8/9. AWT reports them + // as 6/7 on Linux and 4/5 on other platforms — accept both. + final int xButton = switch (button) { + case 4, 6 -> 8; // back + case 5, 7 -> 9; // forward + default -> button; + }; + session.sendMouseClick((int) textureU, (int) textureV, xButton); + return true; + } + + @Override + public boolean mouseHover(final double textureU, final double textureV) { + lastHoverU = textureU; + lastHoverV = textureV; + // while focused the app follows the mouse cursor without clicking + if (hasKeyboardFocus() && !Double.isNaN(textureU)) { + final int x = (int) textureU; + final int y = (int) textureV; + if (x != lastForwardedHoverX || y != lastForwardedHoverY) { + lastForwardedHoverX = x; + lastForwardedHoverY = y; + session.sendMouseMove(x, y); + } + } + return false; // no repaint needed + } + + @Override + public boolean mouseWheelMoved(final int verticalUnits, + final int horizontalUnits) { + // scroll at the current pointer position so the element under the + // cursor scrolls, like on a real desktop + final int x = Double.isNaN(lastHoverU) ? captureWidth / 2 + : (int) lastHoverU; + final int y = Double.isNaN(lastHoverV) ? captureHeight / 2 + : (int) lastHoverV; + session.sendScroll(verticalUnits, horizontalUnits, x, y); + return true; + } + + @Override + public boolean keyPressed(final java.awt.event.KeyEvent event, + final ViewPanel viewPanel) { + if (isShiftEscape(event)) + // focus-release shortcut, never forwarded to the application + return super.keyPressed(event, viewPanel); + session.sendKeyEvent(event, true); + return true; + } + + @Override + public boolean keyReleased(final java.awt.event.KeyEvent event, + final ViewPanel viewPanel) { + if (isShiftEscape(event)) + // press side already consumed this shortcut; swallow release + return true; + session.sendKeyEvent(event, false); + return true; + } + + /** + * Shift+ESC releases focus (same convention as the terminal panels). + * Plain ESC is forwarded to the application, which may need it itself. + */ + private static boolean isShiftEscape(final java.awt.event.KeyEvent event) { + return event.getKeyCode() == java.awt.event.KeyEvent.VK_ESCAPE + && event.isShiftDown(); + } + + @Override + public boolean focusLost(final ViewPanel viewPanel) { + // Shift+ESC pops focus while Shift is still held; the Shift press + // was forwarded but its release will go to the next focus owner, + // so release modifiers server-side to avoid a stuck Shift + session.releaseModifiers(); + return super.focusLost(viewPanel); + } + + /** + * Stops the capture thread, kills the application and its Xvfb server. + */ + public void stop() { + session.close(); + } +} diff --git a/src/main/java/eu/svjatoslav/aukio/workspace/TerminalPanel.java b/src/main/java/eu/svjatoslav/aukio/workspace/TerminalPanel.java new file mode 100644 index 0000000..95aacbe --- /dev/null +++ b/src/main/java/eu/svjatoslav/aukio/workspace/TerminalPanel.java @@ -0,0 +1,336 @@ +/* + * Aukio spatial computing environment. Author: Svjatoslav Agejenko. + * This project is released under Creative Commons Zero (CC0) license. + */ +package eu.svjatoslav.aukio.workspace; + +import eu.svjatoslav.aukio.bridge.pty.PtySession; +import eu.svjatoslav.aukio.bridge.pty.ScreenBuffer; +import eu.svjatoslav.aukio.e3d.geometry.Point2D; +import eu.svjatoslav.aukio.e3d.gui.GuiComponent; +import eu.svjatoslav.aukio.e3d.gui.TextPointer; +import eu.svjatoslav.aukio.e3d.gui.ViewPanel; +import eu.svjatoslav.aukio.e3d.math.Transform; +import eu.svjatoslav.aukio.e3d.renderer.raster.Color; +import eu.svjatoslav.aukio.e3d.renderer.raster.shapes.composite.textcanvas.TextCanvas; + +import java.awt.event.KeyEvent; + +/** + * A live terminal panel in 3D space: a real shell on a PTY (via + * {@link PtySession}) rendered onto a {@link TextCanvas}. + * + *

Click the panel to focus it; keystrokes are translated to terminal + * input bytes and written to the PTY. Shift+ESC releases focus (plain + * ESC goes to the shell — terminal programs need it).

+ * + *

Terminal output arrives on the PTY reader thread, which copies the + * screen buffer into the canvas and requests a repaint. This is the same + * benign writer/renderer race the text editor already accepts.

+ */ +public class TerminalPanel extends GuiComponent { + + /** + * Standard xterm palette, indices 0-15 (normal + bright). + */ + private static final Color[] PALETTE = { + new Color(0, 0, 0), // 0 black + new Color(205, 49, 49), // 1 red + new Color(13, 188, 121), // 2 green + new Color(229, 229, 16), // 3 yellow + new Color(36, 114, 200), // 4 blue + new Color(188, 63, 188), // 5 magenta + new Color(17, 168, 205), // 6 cyan + new Color(229, 229, 229), // 7 white + new Color(102, 102, 102), // 8 bright black + new Color(241, 76, 76), // 9 bright red + new Color(35, 209, 139), // 10 bright green + new Color(245, 245, 67), // 11 bright yellow + new Color(59, 142, 234), // 12 bright blue + new Color(214, 112, 214), // 13 bright magenta + new Color(41, 184, 219), // 14 bright cyan + new Color(255, 255, 255), // 15 bright white + }; + + private static final Color DEFAULT_FOREGROUND = new Color(220, 220, 220); + private static final Color DEFAULT_BACKGROUND = new Color(16, 16, 24); + private static final Color CURSOR_COLOR = new Color(200, 255, 200); + + private final PtySession session; + private final TextCanvas textCanvas; + + /** + * Creates a terminal panel running the given PTY session. + * + * @param transform position in the world + * @param viewPanel the view panel this component belongs to + * @param sizeInWorldCoordinates panel size; determines terminal columns + * and rows through the engine font metrics + * @param session a running PTY session whose buffer + * dimensions match the panel grid + */ + public TerminalPanel(final Transform transform, final ViewPanel viewPanel, + final Point2D sizeInWorldCoordinates, + final PtySession session) { + super(transform, viewPanel, sizeInWorldCoordinates.to3D()); + this.session = session; + + final int columns = (int) (sizeInWorldCoordinates.x + / TextCanvas.FONT_CHAR_WIDTH); + final int rows = (int) (sizeInWorldCoordinates.y + / TextCanvas.FONT_CHAR_HEIGHT); + + textCanvas = new TextCanvas(new Transform(), + new TextPointer(rows, columns), + DEFAULT_FOREGROUND, DEFAULT_BACKGROUND); + textCanvas.setMouseInteractionController(this); + addShape(textCanvas); + + session.setContentListener(this::onTerminalContent); + } + + private void onTerminalContent() { + syncBufferToCanvas(); + viewPanel.repaintDuringNextViewUpdate(); + } + + /** + * Copies the terminal screen buffer into the text canvas, resolving + * palette colors and painting the cursor as an inverted cell. + */ + private void syncBufferToCanvas() { + final ScreenBuffer buffer = session.getScreenBuffer(); + synchronized (buffer) { + for (int row = 0; row < buffer.getRows(); row++) + for (int column = 0; column < buffer.getColumns(); column++) { + final ScreenBuffer.Cell cell = buffer.getCell(row, column); + textCanvas.setForegroundColor(resolveForeground(cell)); + textCanvas.setBackgroundColor(resolveBackground(cell)); + textCanvas.putChar(row, column, cell.ch); + } + + if (buffer.cursorVisible) { + final ScreenBuffer.Cell cell = buffer.getCell(buffer.cursorY, + buffer.cursorX); + textCanvas.setForegroundColor(DEFAULT_BACKGROUND); + textCanvas.setBackgroundColor(CURSOR_COLOR); + textCanvas.putChar(buffer.cursorY, buffer.cursorX, + cell.ch == ' ' ? ' ' : cell.ch); + } + } + } + + private Color resolveForeground(final ScreenBuffer.Cell cell) { + if (cell.reverse) + return resolveColor(cell.bg, DEFAULT_BACKGROUND, false); + return resolveColor(cell.fg, DEFAULT_FOREGROUND, cell.bold); + } + + private Color resolveBackground(final ScreenBuffer.Cell cell) { + if (cell.reverse) + return resolveColor(cell.fg, DEFAULT_FOREGROUND, cell.bold); + return resolveColor(cell.bg, DEFAULT_BACKGROUND, false); + } + + private Color resolveColor(final int index, final Color defaultColor, + final boolean bold) { + if (index == ScreenBuffer.DEFAULT_COLOR) + return defaultColor; + if (index < 16) { + final int adjusted = (bold && index < 8) ? index + 8 : index; + return PALETTE[adjusted]; + } + // 256-color palette: 16-231 color cube, 232-255 grayscale + final int i = index - 16; + if (i < 216) { + final int r = i / 36; + final int g = (i / 6) % 6; + final int b = i % 6; + return new Color(cubeChannel(r), cubeChannel(g), cubeChannel(b)); + } + final int gray = 8 + (i - 216) * 10; + return new Color(gray, gray, gray); + } + + private static int cubeChannel(final int level) { + return level == 0 ? 0 : 55 + level * 40; + } + + @Override + public boolean keyPressed(final KeyEvent event, + final ViewPanel viewPanel) { + // Focus model: Shift+ESC releases focus (GuiComponent convention + // uses plain ESC, but terminal programs need ESC themselves — + // mc uses ESC for dialogs and ESC 0 for F10. Terminal apps cannot + // distinguish Shift+ESC from plain ESC on the wire, so no program + // loses a binding; Ctrl+ESC was avoided because desktops commonly + // intercept it). + if (event.getKeyChar() == '\u001B') { + if (event.isShiftDown()) + return super.keyPressed(event, viewPanel); + session.send("\u001B"); + return true; + } + + final String sequence = translate(event); + if (sequence != null) + session.send(sequence); + return true; + } + + @Override + public boolean mouseWheelMoved(final int verticalUnits, + final int horizontalUnits) { + // The emulator has no scrollback and no mouse reporting, so the + // wheel is translated to arrow keys — scrolls less/htop/mc, and + // walks the command history at a shell prompt. Three lines per + // notch, matching xterm. + for (int i = 0; i < Math.abs(verticalUnits) * 3; i++) + session.send(verticalUnits < 0 ? cursorKey('A', 'A') + : cursorKey('B', 'B')); + for (int i = 0; i < Math.abs(horizontalUnits) * 3; i++) + session.send(horizontalUnits < 0 ? cursorKey('D', 'D') + : cursorKey('C', 'C')); + return true; + } + + /** + * Whether the terminal is in application cursor keys mode (DECCKM). + * Curses programs (mc, htop, vim) enable it; cursor keys must then be + * reported as SS3 instead of CSI. + */ + public boolean isApplicationCursorKeys() { + synchronized (session.getScreenBuffer()) { + return session.getScreenBuffer().applicationCursorKeys; + } + } + + /** + * Cursor key sequence honoring application cursor mode. + */ + private String cursorKey(final char csiFinal, final char ss3Final) { + return isApplicationCursorKeys() + ? "\u001BO" + ss3Final + : "\u001B[" + csiFinal; + } + + /** + * Translates an AWT key event into the byte sequence a terminal would + * send. Returns null for keys with no terminal representation. + */ + private String translate(final KeyEvent event) { + switch (event.getKeyCode()) { + case KeyEvent.VK_ENTER -> { + return "\r"; + } + case KeyEvent.VK_BACK_SPACE -> { + return "\u007F"; + } + case KeyEvent.VK_TAB -> { + // xterm: Shift+Tab is backtab (kcbt) + return event.isShiftDown() ? "\u001B[Z" : "\t"; + } + case KeyEvent.VK_UP -> { + return cursorKey('A', 'A'); + } + case KeyEvent.VK_DOWN -> { + return cursorKey('B', 'B'); + } + case KeyEvent.VK_RIGHT -> { + return cursorKey('C', 'C'); + } + case KeyEvent.VK_LEFT -> { + return cursorKey('D', 'D'); + } + case KeyEvent.VK_HOME -> { + return cursorKey('H', 'H'); + } + case KeyEvent.VK_END -> { + return cursorKey('F', 'F'); + } + case KeyEvent.VK_DELETE -> { + return "\u001B[3~"; + } + case KeyEvent.VK_PAGE_UP -> { + return "\u001B[5~"; + } + case KeyEvent.VK_PAGE_DOWN -> { + return "\u001B[6~"; + } + // xterm function keys (mc is unusable without F10 = quit) + case KeyEvent.VK_F1 -> { + return "\u001BOP"; + } + case KeyEvent.VK_F2 -> { + return "\u001BOQ"; + } + case KeyEvent.VK_F3 -> { + return "\u001BOR"; + } + case KeyEvent.VK_F4 -> { + return "\u001BOS"; + } + case KeyEvent.VK_F5 -> { + return "\u001B[15~"; + } + case KeyEvent.VK_F6 -> { + return "\u001B[17~"; + } + case KeyEvent.VK_F7 -> { + return "\u001B[18~"; + } + case KeyEvent.VK_F8 -> { + return "\u001B[19~"; + } + case KeyEvent.VK_F9 -> { + return "\u001B[20~"; + } + case KeyEvent.VK_F10 -> { + return "\u001B[21~"; + } + case KeyEvent.VK_F11 -> { + return "\u001B[23~"; + } + case KeyEvent.VK_F12 -> { + return "\u001B[24~"; + } + default -> { + } + } + + final char keyChar = event.getKeyChar(); + if (keyChar == KeyEvent.CHAR_UNDEFINED) + return null; + // control combinations arrive as C0 control characters already + if (event.isControlDown() && keyChar < 0x20) + return String.valueOf(keyChar); + if (keyChar >= 0x20 && keyChar != 0x7F) { + // Alt+key is reported as ESC prefix (Meta), like xterm + if (event.isAltDown()) + return "\u001B" + keyChar; + return String.valueOf(keyChar); + } + return null; + } + + /** + * Programmatic text injection, used by the self-test and future + * automation. + */ + public void typeText(final String text) { + session.send(text); + } + + public PtySession getSession() { + return session; + } + + /** + * Current terminal screen as plain text (attributes stripped). + */ + public String getScreenText() { + synchronized (session.getScreenBuffer()) { + return session.getScreenBuffer().dumpText(); + } + } +} diff --git a/src/main/java/eu/svjatoslav/aukio/workspace/VmwarePanel.java b/src/main/java/eu/svjatoslav/aukio/workspace/VmwarePanel.java new file mode 100644 index 0000000..ec42e76 --- /dev/null +++ b/src/main/java/eu/svjatoslav/aukio/workspace/VmwarePanel.java @@ -0,0 +1,49 @@ +/* + * Aukio spatial computing environment. Author: Svjatoslav Agejenko. + * This project is released under Creative Commons Zero (CC0) license. + */ +package eu.svjatoslav.aukio.workspace; + +import eu.svjatoslav.aukio.bridge.x11.GuiAppSession; +import eu.svjatoslav.aukio.e3d.geometry.Point2D; +import eu.svjatoslav.aukio.e3d.gui.ViewPanel; +import eu.svjatoslav.aukio.e3d.math.Transform; + +import java.io.IOException; + +/** + * A live VMware Workstation panel in 3D space. All capture and + * interaction machinery lives in {@link GuiAppPanel}; this class only + * defines the VMware session. + * + *

Unlike Firefox, the VMware window cannot be sized on the command + * line, so the session continuously moves/resizes windows titled + * "VMware" to fill the virtual screen (see + * {@link GuiAppSession#vmware}).

+ */ +public class VmwarePanel extends GuiAppPanel { + + /** + * Virtual screen (and texture) resolution; the session fits the + * VMware window to fill it. + */ + public static final int CAPTURE_WIDTH = 1280; + public static final int CAPTURE_HEIGHT = 960; + + /** + * Creates the panel. Call {@link #start()} to launch VMware + * Workstation and begin streaming its screen into the texture. + * + * @param transform position in the world + * @param viewPanel the view panel this component belongs to + * @param sizeInWorldCoordinates panel size in world units; the capture + * is stretched to fill it + */ + public VmwarePanel(final Transform transform, final ViewPanel viewPanel, + final Point2D sizeInWorldCoordinates) + throws IOException { + super(transform, viewPanel, sizeInWorldCoordinates, + CAPTURE_WIDTH, CAPTURE_HEIGHT, + GuiAppSession.vmware(CAPTURE_WIDTH, CAPTURE_HEIGHT)); + } +} diff --git a/src/main/java/eu/svjatoslav/aukio/workspace/Workspace.java b/src/main/java/eu/svjatoslav/aukio/workspace/Workspace.java new file mode 100644 index 0000000..37a772c --- /dev/null +++ b/src/main/java/eu/svjatoslav/aukio/workspace/Workspace.java @@ -0,0 +1,208 @@ +/* + * Aukio spatial computing environment. Author: Svjatoslav Agejenko. + * This project is released under Creative Commons Zero (CC0) license. + */ +package eu.svjatoslav.aukio.workspace; + +import eu.svjatoslav.aukio.bridge.pty.PtySession; +import eu.svjatoslav.aukio.e3d.geometry.Point2D; +import eu.svjatoslav.aukio.e3d.geometry.Rectangle; +import eu.svjatoslav.aukio.e3d.gui.ViewFrame; +import eu.svjatoslav.aukio.e3d.gui.ViewPanel; +import eu.svjatoslav.aukio.e3d.gui.textEditorComponent.LookAndFeel; +import eu.svjatoslav.aukio.e3d.gui.textEditorComponent.TextEditComponent; +import eu.svjatoslav.aukio.e3d.math.Transform; +import eu.svjatoslav.aukio.e3d.renderer.raster.ShapeCollection; +import eu.svjatoslav.aukio.e3d.renderer.raster.shapes.basic.line.LineAppearance; +import eu.svjatoslav.aukio.e3d.renderer.raster.shapes.composite.textcanvas.TextCanvas; +import eu.svjatoslav.aukio.e3d.renderer.raster.shapes.composite.wireframe.Grid2D; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +import static eu.svjatoslav.aukio.e3d.geometry.Point3D.point; +import static eu.svjatoslav.aukio.e3d.renderer.raster.Color.hex; + +/** + * The Aukio workspace: a persistent 3D world where programs are placeable + * objects. + * + *

Current population: one text editor (engine component), a 4x4 + * grid of live terminals, each running its own bash on a real PTY, a + * live Firefox browser and a live VMware Workstation window (private + * Xvfb displays captured into textures). + * Click a panel to focus it and type; ESC or middle click releases focus; fly with the + * engine's standard camera controls.

+ */ +public class Workspace { + + /** + * Terminal geometry: 100 x 30 characters. + */ + public static final int TERMINAL_COLUMNS = 100; + public static final int TERMINAL_ROWS = 30; + public static final int TERMINAL_GRID_COLUMNS = 4; + public static final int TERMINAL_GRID_ROWS = 4; + + private final ViewFrame viewFrame; + private final List terminalPanels = new ArrayList<>(); + private FirefoxPanel firefoxPanel; + private VmwarePanel vmwarePanel; + private eu.svjatoslav.aukio.env.EnvironmentProvider environment; + + public Workspace() { + viewFrame = new ViewFrame("Aukio"); + } + + /** + * Attaches a pluggable environment (game world, procedural scene) + * after open(); the workspace then owns its lifecycle. + */ + public void openEnvironment( + final eu.svjatoslav.aukio.env.EnvironmentProvider provider) + throws Exception { + environment = provider; + provider.open(new eu.svjatoslav.aukio.env.EnvironmentContext( + viewFrame.getViewPanel())); + } + + /** + * Builds the scene and shows the window. + * + * @throws IOException if the terminal PTY cannot be created + */ + public void open() throws IOException { + final ViewPanel viewPanel = viewFrame.getViewPanel(); + final ShapeCollection scene = viewPanel.getRootShapeCollection(); + + viewPanel.getCamera().getTransform().set(150, -120, -350, 0, + -0.12, 0); + + addGrid(scene); + addTextEditor(viewPanel, scene); + addTerminals(viewPanel, scene); + addFirefox(viewPanel, scene); + addVmware(viewPanel, scene); + + viewPanel.repaintDuringNextViewUpdate(); + } + + private void addGrid(final ShapeCollection scene) { + final Transform transform = Transform.fromAngles(0, 100, 0, 0, + Math.PI / 2, 0); + final Rectangle rectangle = new Rectangle(2000); + final LineAppearance appearance = new LineAppearance(10, + hex("00b3ad")); + scene.addShape(new Grid2D(transform, rectangle, 10, 10, appearance)); + } + + private void addTextEditor(final ViewPanel viewPanel, + final ShapeCollection scene) { + final TextEditComponent editor = new TextEditComponent( + new Transform(point(-700, 0, 300)), viewPanel, + new Point2D(400, 240), new LookAndFeel()); + editor.setText("Aukio workspace\n\n" + + "Click a panel to focus it.\n" + + "Type into it. Middle click or ESC releases\n" + + "focus (terminal, browser, VMware: Shift+ESC).\n\n" + + "The 4x4 grid on the right contains\n" + + "independent bash shells.\n" + + "Try: ls, mc, htop\n\n" + + "Below this editor: Firefox and\n" + + "VMware Workstation, captured live."); + scene.addShape(editor); + } + + private void addTerminals(final ViewPanel viewPanel, + final ShapeCollection scene) throws IOException { + for (int x = 0; x < TERMINAL_GRID_COLUMNS; x++) { + for (int y = 0; y < TERMINAL_GRID_ROWS; y++) { + final PtySession session = new PtySession(TERMINAL_COLUMNS, + TERMINAL_ROWS); + final TerminalPanel terminalPanel = new TerminalPanel( + new Transform(point(100 + (x * 900), + -100 - (y * 600), 300)), viewPanel, + new Point2D( + TERMINAL_COLUMNS * TextCanvas.FONT_CHAR_WIDTH, + TERMINAL_ROWS * TextCanvas.FONT_CHAR_HEIGHT), + session); + terminalPanels.add(terminalPanel); + scene.addShape(terminalPanel); + } + } + } + + /** + * Adds a live Firefox browser below the text editor: a private Xvfb + * display whose screen is captured into a textured rectangle. + */ + private void addFirefox(final ViewPanel viewPanel, + final ShapeCollection scene) throws IOException { + firefoxPanel = new FirefoxPanel(new Transform(point(-700, -450, 300)), + viewPanel, new Point2D(640, 480)); + firefoxPanel.start(); + scene.addShape(firefoxPanel); + } + + /** + * Adds a live VMware Workstation window below the Firefox panel: a + * private Xvfb display whose screen is captured into a textured + * rectangle. + */ + private void addVmware(final ViewPanel viewPanel, + final ShapeCollection scene) throws IOException { + vmwarePanel = new VmwarePanel(new Transform(point(-700, -1050, 300)), + viewPanel, new Point2D(640, 480)); + vmwarePanel.start(); + scene.addShape(vmwarePanel); + } + + /** + * Returns the Firefox panel, used by the automated GUI test. + */ + public FirefoxPanel getFirefoxPanel() { + return firefoxPanel; + } + + /** + * Returns the VMware Workstation panel. + */ + public VmwarePanel getVmwarePanel() { + return vmwarePanel; + } + + /** + * Returns the first terminal, used by the automated terminal selftest. + */ + public TerminalPanel getTerminalPanel() { + return terminalPanels.get(0); + } + + public List getTerminalPanels() { + return Collections.unmodifiableList(terminalPanels); + } + + public eu.svjatoslav.aukio.e3d.gui.headtrack.HeadTracker getHeadTracker() { + return viewFrame.getViewPanel().getHeadTracker(); + } + + public ViewFrame getViewFrame() { + return viewFrame; + } + + /** + * Shuts down all background shell processes. + */ + public void close() { + if (environment != null) + environment.close(); + for (final TerminalPanel terminalPanel : terminalPanels) + terminalPanel.getSession().stop(); + if (firefoxPanel != null) + firefoxPanel.stop(); + if (vmwarePanel != null) + vmwarePanel.stop(); + } +} diff --git a/start.sh b/start.sh new file mode 100755 index 0000000..4750596 --- /dev/null +++ b/start.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# Start the Aukio workspace with the Fallout 4 environment. +# +# Rebuilds every module in dependency order first, so the classpath is +# always at the latest source state (the FO4 provider resolves `aukio` +# and `aukio-3d` from the local Maven repository — stale installs there +# are the classic reason the world silently doesn't show up). +# +# Extra args are forwarded to the JVM launcher, e.g.: +# ./start.sh -Daukio.fo4.cell=ConcordExt +# Actually: args go to Main. To pass -D properties, edit JAVA_OPTS below +# or export them before calling. + +set -euo pipefail + +# Sibling checkouts are resolved relative to this script (Aukio/* +# layout), so nothing here is tied to a particular machine. +ROOT="$(cd "${0%/*}/.." && pwd)" +FO4="$ROOT/aukio-environment-fo4" + +echo "==> building aukio-3d" +mvn -f "$ROOT/aukio-3d/pom.xml" install -q -DskipTests -Dmaven.javadoc.skip=true + +echo "==> building aukio" +mvn -f "$ROOT/aukio/pom.xml" install -q -DskipTests -Dmaven.javadoc.skip=true + +echo "==> building aukio-environment-fo4" +mvn -f "$FO4/pom.xml" package -q -DskipTests -Dmaven.javadoc.skip=true +mvn -f "$FO4/pom.xml" dependency:build-classpath -q -Dmdep.outputFile=target/cp.txt + +echo "==> starting Aukio with --env=fallout4" +cd "$FO4" +mkdir -p "$HOME/.cache/aukio/heapdumps" +exec java -Xmx20g --enable-native-access=ALL-UNNAMED \ + -XX:+HeapDumpOnOutOfMemoryError \ + -XX:HeapDumpPath="$HOME/.cache/aukio/heapdumps" \ + -Daukio.cull.subpixel=0.4 \ + ${AUKIO_FO4_PATH:+-Daukio.fo4.path=$AUKIO_FO4_PATH} \ + ${JAVA_OPTS:-} \ + -cp "target/classes:$(cat target/cp.txt)" \ + eu.svjatoslav.aukio.core.Main --env=fallout4 "$@"