From: Svjatoslav Agejenko Date: Sat, 18 Jul 2026 12:27:17 +0000 (+0300) Subject: fix(install): work on bare servers without PATH hermes or python3-venv X-Git-Url: http://www2.svjatoslav.eu/gitweb/?a=commitdiff_plain;h=b18f3d80431a38a95783be54aedaea86c1599f90;p=retinue.git fix(install): work on bare servers without PATH hermes or python3-venv Found during first install on a fresh Debian 13 server: - Installer now probes ~/.local/bin/hermes, ~/bin, /usr/local/bin and /usr/bin for the hermes CLI instead of relying on PATH alone. Non-login SSH shells never read ~/.profile, so "hermes CLI not found" fired and memory.provider was left unset. - New shared Tools/lib/common.sh picks a working Python for all Tools scripts: project .venv (only if sqlite-vec + model2vec actually import — a failed python3 -m venv on hosts without the python3-venv apt package leaves a half-created venv that passes -x checks), then uv (bundled with Hermes at ~/.hermes/bin/uv), then the Hermes runtime venv. Broken half-venvs are recreated instead of trusted. - Docs: prerequisites updated (no PATH / python3-venv needed), importer fallback and idempotency documented, systemd restart hint including the "Gateway already running (PID ...)" stale-lock case. - AGENTS.org: Tools/lib layout entry and three new pitfalls. --- diff --git a/AGENTS.org b/AGENTS.org index b508afb..2307863 100644 --- a/AGENTS.org +++ b/AGENTS.org @@ -52,6 +52,7 @@ retinue/ │ ├── index.org # Public user documentation (exports to HTML) │ └── Development/index.org # Architecture & integration documentation ├── Tools/ # Development helpers (filenames contain spaces) +│ ├── lib/common.sh # Shared bash helpers: RETINUE_PY resolution (venv → uv → Hermes venv) │ ├── Run tests # Standalone smoke test runner (no Hermes needed) │ ├── Show Retinue memory contents # Debug CLI: dump/search all memories │ ├── Import Holographic memories # Migrate facts from Holographic memory_store.db @@ -119,7 +120,10 @@ cd /path/to/retinue #+end_src Uses =.venv/= and =.hf-cache/= inside the project. If the venv is -missing: +missing (or broken — e.g. a half-created venv from a failed +=python3 -m venv= on a host without =python3-venv=), the script repairs +it automatically; it can also fall back to =uv= or to the Hermes runtime +venv. Manual creation is still possible: #+begin_src sh python3 -m venv .venv @@ -248,3 +252,23 @@ Never commit the generated =*.html= — they are gitignored. find-file + save has truncated org files on this host. Write with write_file, validate with a separate read-only batch export, delete the generated .html afterwards. +- *Pitfall: non-login SSH shells.* On servers, =hermes= typically lives + in =~/.local/bin/= which is only on PATH after =~/.profile= runs. Plain + =ssh host 'command'= is a non-login shell without it — =command -v + hermes= fails even though Hermes is installed. The installer probes + =~/.local/bin/hermes=, =~/bin/hermes=, =/usr/local/bin=, =/usr/bin= + explicitly; keep that probing intact. +- *Pitfall: bare Debian 13 lacks python3-venv.* =python3 -m venv .venv= + fails without the =python3.X-venv= apt package (no ensurepip) and — + worse — leaves a half-created =.venv= (python binary present, pip and + deps missing) that naively passes an =-x .venv/bin/python= check. + =Tools/lib/common.sh= handles this: it verifies deps import before + trusting a venv, recreates broken venvs, tries =uv= (bundled with + Hermes at =~/.hermes/bin/uv=), then falls back to the Hermes runtime + venv. All =Tools/= scripts source it — never open-code venv handling + in a Tools script. +- *Pitfall: gateway restart on the remote.* =systemctl --user restart + hermes-gateway.service= can leave the unit in =activating= when an old + gateway process (started with =gateway run --replace= outside systemd) + still holds the lock ("Gateway already running (PID ...)"). Kill the + stale PID once; =Restart=always= then brings the new instance up. diff --git a/Documentation/index.org b/Documentation/index.org index 8c84af2..38dba61 100644 --- a/Documentation/index.org +++ b/Documentation/index.org @@ -46,8 +46,16 @@ For technical details, read: :ID: bc83f42b-b126-4b48-b365-762d27618651 :END: -Prerequisites: Hermes Agent must be installed and the =hermes= CLI must be on -your =PATH=. +Prerequisites: + +- Hermes Agent must be installed. The =hermes= CLI does not need to be on + =PATH= — the installer also probes the standard locations + (=~/.local/bin/hermes=, =~/bin/hermes=, =/usr/local/bin=, =/usr/bin=), + which matters when installing over a non-login SSH shell where + =~/.profile= has not run. +- Nothing else: the installer uses the Python interpreter that Hermes + itself runs (=~/.hermes/hermes-agent/venv=), so it works even on a bare + Debian/Ubuntu server without the =python3-venv= package. Run the bundled installer from the Retinue source tree: @@ -80,7 +88,15 @@ hermes memory status *Important:* If Hermes is already running, restart it. Hermes loads plugins at startup, so an active session will not see the newly installed Retinue plugin -until it is restarted. +until it is restarted. On a typical systemd-user setup: + +#+begin_src sh +systemctl --user restart hermes-gateway.service +#+end_src + +If the service then stays in =activating= and the journal says "Gateway +already running (PID ...)", an older gateway process still holds the lock — +stop it once (=kill =) and =Restart=always= brings the new instance up. For technical details, read: [[file:Development/index.org::#what-the-installer-does][What the installer does.]] @@ -108,12 +124,23 @@ The importer performs three steps: 3. Writes new memories into the active Retinue database, skipping exact duplicates. +The importer needs a Python with =sqlite-vec= + =model2vec=. It uses the +project-local =.venv= when available; on a machine without the +=python3-venv= package it automatically falls back to =uv= (Hermes bundles +one at =~/.hermes/bin/uv=), and finally to the Hermes runtime venv that +=Install Retinue= already provisioned — so the importer works out of the +box right after installation, with no extra packages. + To preview what would be imported without writing anything: #+begin_src sh "Tools/Import Holographic memories" --source /path/to/memory_store.db --dry-run #+end_src +The importer is idempotent: re-running it skips every fact that is already +present in the Retinue database, so it is safe to run again after an +interrupted or partial import. + * Updating :PROPERTIES: :CUSTOM_ID: updating diff --git a/Install Retinue b/Install Retinue index 395cf51..fb17d83 100755 --- a/Install Retinue +++ b/Install Retinue @@ -33,9 +33,22 @@ resolve_hermes_python() { return fi - # Derive from the hermes launcher shebang. + # The hermes launcher often lives in ~/.local/bin (or ~/bin), which is + # only on PATH in login shells. Installers are frequently run over plain + # `ssh host 'cmd'` (non-login shell), where command -v hermes finds + # nothing — so probe the standard locations explicitly. local hermes_bin hermes_bin="$(command -v hermes 2>/dev/null || true)" + if [[ -z "$hermes_bin" ]]; then + local candidate + for candidate in "$HOME/.local/bin/hermes" "$HOME/bin/hermes" \ + /usr/local/bin/hermes /usr/bin/hermes; do + if [[ -x "$candidate" ]]; then + hermes_bin="$candidate" + break + fi + done + fi if [[ -n "$hermes_bin" && -r "$hermes_bin" ]]; then local shebang shebang="$(head -n1 "$hermes_bin" 2>/dev/null || true)" @@ -180,10 +193,22 @@ print(' Embedding model ready.') preload_embedding_model # Set Hermes memory provider if the CLI is available. -# This may fail in read-only sandbox configs; the user can set it manually. -if command -v hermes &>/dev/null; then - echo "Setting memory.provider to retinue..." - if hermes config set memory.provider retinue 2>/dev/null; then +# Same non-login-shell problem as above: hermes usually lives in ~/.local/bin, +# so probe the standard locations instead of relying on PATH alone. +HERMES_CLI="$(command -v hermes 2>/dev/null || true)" +if [[ -z "$HERMES_CLI" ]]; then + for candidate in "$HOME/.local/bin/hermes" "$HOME/bin/hermes" \ + /usr/local/bin/hermes /usr/bin/hermes; do + if [[ -x "$candidate" ]]; then + HERMES_CLI="$candidate" + break + fi + done +fi + +if [[ -n "$HERMES_CLI" ]]; then + echo "Setting memory.provider to retinue (via $HERMES_CLI)..." + if "$HERMES_CLI" config set memory.provider retinue 2>/dev/null; then echo " memory.provider set to retinue." else echo " Warning: could not set memory.provider automatically." @@ -222,12 +247,12 @@ echo "" echo "The embedding model is pre-downloaded and ready — no first-use delay." echo "" echo "Next steps:" -if command -v hermes &>/dev/null; then - echo " hermes memory setup" - echo " hermes memory status" - echo " hermes chat -q 'Remember something...'" +if [[ -n "$HERMES_CLI" ]]; then + echo " $HERMES_CLI memory status" + echo " $HERMES_CLI chat -q 'Remember something...'" else echo " Install hermes CLI, then:" - echo " hermes memory setup" echo " hermes memory status" fi +echo "" +echo "Restart any running Hermes sessions — plugins are loaded at startup." diff --git a/Tools/Import Holographic memories b/Tools/Import Holographic memories index eba9e61..0b223cc 100755 --- a/Tools/Import Holographic memories +++ b/Tools/Import Holographic memories @@ -21,17 +21,12 @@ set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" -VENV="$PROJECT_ROOT/.venv" -if [[ ! -d "$VENV" ]]; then - echo "Virtual environment not found at $VENV" - echo "Create it with:" - echo " python3 -m venv $VENV" - echo " $VENV/bin/python -m pip install sqlite-vec model2vec" - exit 1 -fi +# shellcheck source=lib/common.sh +source "$SCRIPT_DIR/lib/common.sh" +retinue_pick_python export HF_HOME="${HF_HOME:-$PROJECT_ROOT/.hf-cache}" export PYTHONPATH="$PROJECT_ROOT/src${PYTHONPATH:+:$PYTHONPATH}" -exec "$VENV/bin/python" -m retinue.import_holographic "$@" +exec "$RETINUE_PY" -m retinue.import_holographic "$@" diff --git a/Tools/Run tests b/Tools/Run tests index deae42c..9c968dc 100755 --- a/Tools/Run tests +++ b/Tools/Run tests @@ -7,17 +7,12 @@ set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" -VENV="$PROJECT_ROOT/.venv" TEST="$PROJECT_ROOT/test/test_standalone.py" -if [[ ! -d "$VENV" ]]; then - echo "Virtual environment not found at $VENV" - echo "Create it with:" - echo " python3 -m venv $VENV" - echo " $VENV/bin/python -m pip install sqlite-vec model2vec" - exit 1 -fi +# shellcheck source=lib/common.sh +source "$SCRIPT_DIR/lib/common.sh" +retinue_pick_python export HF_HOME="${HF_HOME:-$PROJECT_ROOT/.hf-cache}" -exec "$VENV/bin/python" "$TEST" +exec "$RETINUE_PY" "$TEST" diff --git a/Tools/Show Retinue memory contents b/Tools/Show Retinue memory contents index 9a1a1bc..63a24e7 100755 --- a/Tools/Show Retinue memory contents +++ b/Tools/Show Retinue memory contents @@ -19,17 +19,12 @@ set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" -VENV="$PROJECT_ROOT/.venv" -if [[ ! -d "$VENV" ]]; then - echo "Virtual environment not found at $VENV" - echo "Create it with:" - echo " python3 -m venv $VENV" - echo " $VENV/bin/python -m pip install sqlite-vec model2vec" - exit 1 -fi +# shellcheck source=lib/common.sh +source "$SCRIPT_DIR/lib/common.sh" +retinue_pick_python export HF_HOME="${HF_HOME:-$PROJECT_ROOT/.hf-cache}" export PYTHONPATH="$PROJECT_ROOT/src${PYTHONPATH:+:$PYTHONPATH}" -exec "$VENV/bin/python" -m retinue.cli "$@" +exec "$RETINUE_PY" -m retinue.cli "$@" diff --git a/Tools/lib/common.sh b/Tools/lib/common.sh new file mode 100644 index 0000000..bf09a8f --- /dev/null +++ b/Tools/lib/common.sh @@ -0,0 +1,82 @@ +#!/usr/bin/env bash +# Shared helpers for the Retinue "Tools/" scripts. +# +# This file is SOURCED, not executed. It expects the calling script to have +# already defined: +# PROJECT_ROOT absolute path of the Retinue source tree +# SCRIPT_DIR absolute path of the Tools/ directory +# +# After calling retinue_pick_python, the caller uses: +# RETINUE_PY Python interpreter to exec (project .venv or Hermes venv) + +# Resolve a Python interpreter that has sqlite-vec + model2vec importable. +# +# Preference order: +# 1. The project-local .venv (create it if possible). +# 2. The Hermes runtime venv ($HERMES_HOME/hermes-agent/venv) — after +# "Install Retinue" has run, the deps are guaranteed to be there. +# +# Returns 0 and sets RETINUE_PY on success, 1 otherwise. +retinue_pick_python() { + local VENV="$PROJECT_ROOT/.venv" + local HERMES_HOME="${HERMES_HOME:-$HOME/.hermes}" + local HERMES_VENV_PY="$HERMES_HOME/hermes-agent/venv/bin/python3" + + if [[ -x "$VENV/bin/python" ]] \ + && "$VENV/bin/python" -c "import sqlite_vec, model2vec" 2>/dev/null; then + RETINUE_PY="$VENV/bin/python" + return 0 + fi + + # A .venv may exist but be unusable: on hosts without the python3.X-venv + # package, `python3 -m venv` leaves a half-created venv (python present, + # pip/deps missing). Treat that as "no venv" and rebuild below. + if [[ -d "$VENV" ]]; then + echo "Existing $VENV is missing sqlite-vec/model2vec — recreating it." + fi + + # Try to create the project venv. Bare Debian/Ubuntu servers are missing + # the python3.X-venv package (no ensurepip), so plain venv creation fails + # there — try, and fall through to the next option on failure. + echo "Project venv not usable at $VENV — attempting to create it..." + rm -rf "$VENV" + if python3 -m venv "$VENV" 2>/dev/null \ + && "$VENV/bin/python" -m pip install -q sqlite-vec model2vec 2>/dev/null; then + RETINUE_PY="$VENV/bin/python" + return 0 + fi + rm -rf "$VENV" + + # uv (bundled with Hermes at ~/.hermes/bin/uv) can create a venv without + # ensurepip and is much faster than pip. + local UV + UV="$(command -v uv 2>/dev/null || true)" + [[ -z "$UV" && -x "$HERMES_HOME/bin/uv" ]] && UV="$HERMES_HOME/bin/uv" + if [[ -n "$UV" ]]; then + echo "Plain venv unavailable (python3-venv missing?) — trying uv..." + if "$UV" venv --quiet "$VENV" 2>/dev/null \ + && "$UV" pip install --quiet --python "$VENV/bin/python" \ + sqlite-vec model2vec 2>/dev/null; then + RETINUE_PY="$VENV/bin/python" + return 0 + fi + rm -rf "$VENV" + fi + + # Fall back to the Hermes runtime venv: "Install Retinue" installs + # sqlite-vec + model2vec into exactly that interpreter. + if [[ -x "$HERMES_VENV_PY" ]] \ + && "$HERMES_VENV_PY" -c "import sqlite_vec, model2vec" 2>/dev/null; then + echo "Note: using the Hermes runtime venv ($HERMES_VENV_PY)" + echo " (no project .venv; install python3-venv or uv to get one)." + RETINUE_PY="$HERMES_VENV_PY" + return 0 + fi + + echo "ERROR: no usable Python environment found." + echo "Fix it in one of three ways:" + echo " 1. sudo apt install python3-venv # then re-run this script" + echo " 2. install uv: https://docs.astral.sh/uv/ (Hermes bundles one at ~/.hermes/bin/uv)" + echo " 3. run: bash \"Install Retinue\" # deps land in the Hermes venv" + return 1 +}