From: Svjatoslav Agejenko Date: Sat, 27 Jun 2026 10:50:43 +0000 (+0300) Subject: feat(build): incremental org→html export with style.css sentinel X-Git-Url: http://www2.svjatoslav.eu/gitweb/?a=commitdiff_plain;h=f1187e62af3d5270f805da8b835549b89dfad7c1;p=physical.git feat(build): incremental org→html export with style.css sentinel Re-run of the publish script now skips .org files whose sha256 matches the cached value in a new .org-export-state file at the repo root, making re-runs effectively instantaneous when only one or two pages have changed. style.css gets a separate "shared sentinel" entry. If it has changed since the last publish, every .org file is re-exported automatically — the common case of editing CSS no longer requires remembering --force. Pass --force (or --rebuild) to ignore the cache and rebuild everything. AGENTS.org stays excluded from exports. Also adds a "Build cache (incremental export)" subsection to AGENTS.org documenting the new behavior, and gitignores the state file. --- diff --git a/.gitignore b/.gitignore index f0069e3..8b45bc2 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,6 @@ # FreeCAD backup files *.FCStd1 + +# Incremental org→html export cache (regenerated by Tools/Update web site) +.org-export-state diff --git a/AGENTS.org b/AGENTS.org index d885060..b43c843 100644 --- a/AGENTS.org +++ b/AGENTS.org @@ -121,3 +121,27 @@ bullets to =-= or vice versa — they are equivalent. 6. =git add= the new directory. Stage the =index.org= edit. Commit. 7. Regenerate the HTML locally with Emacs to verify the result. The HTML itself stays untracked. + +* Workflow preferences + +** Build cache (incremental export) + +=Tools/Update web site= maintains a small cache at =.org-export-state= (repo +root, gitignored) that records one sha256 per exported =.org= file plus one +sha256 for =style.css= (the "shared sentinel"). On each run, files whose hash +matches the cached value are skipped — re-export only the ones you actually +changed. + +The style.css sentinel is checked on every run: if =style.css= has changed +since the last publish, every =.org= file is re-exported automatically. This +catches style changes that you would otherwise have to remember to force. + +- To force a full rebuild regardless of the cache, pass =--force= (or + =--rebuild=) to the script. +- If you change the darksun theme file (=~/.emacs.d/org-styles/html/darksun.theme=) + or any other Emacs export dependency outside the repo, use =--force=. The + cache only tracks files inside the repo. +- If you change =index.org= and forget =--force=, the per-project HTML files + keep their old back-link text. Run =--force= to refresh. +- To wipe the cache entirely, delete the =.org-export-state= file. The next + run will rebuild everything and recreate it. diff --git a/Tools/Update web site b/Tools/Update web site index e71950e..fa36f45 100755 --- a/Tools/Update web site +++ b/Tools/Update web site @@ -1,13 +1,180 @@ #!/bin/bash +# +# ============================================================================= +# Update web site +# ============================================================================= +# +# PURPOSE +# Convert every Emacs org-mode file in this repository to HTML, then upload +# the result to the public web server. +# +# WHY A SCRIPT? +# HTML files are NOT versioned in git — they are regenerated on demand from +# the .org source files. This script runs the Emacs export in batch mode for +# every page, then rsyncs the generated HTML to the server. +# +# HOW THE INCREMENTAL CACHE WORKS +# The first time this script runs, it exports every .org file (except +# AGENTS.org) and writes a small state file called `.org-export-state` at +# the repo root. The state file records: +# +# - one sha256 hash per exported .org file +# - one sha256 hash of style.css (the "shared sentinel") +# +# On each subsequent run, before exporting a .org file, the script checks +# its current hash against the stored hash. If they match, the file is +# skipped — Emacs is not invoked at all. This is what makes re-runs fast +# when you've only changed one or two pages. +# +# The style.css sentinel is checked on every run. If style.css has changed +# since the last publish, every .org file is re-exported automatically. +# This catches style changes that you would otherwise have to remember to +# force. +# +# AGENTS.org is excluded from exports — it is the operating guide for +# agents (this file you're reading is generated from a different script). +# +# USAGE +# ./Tools/Update web site # re-export only changed files, then upload +# ./Tools/Update web site --force # re-export everything, then upload +# ./Tools/Update web site --rebuild # same as --force +# +# The first time you run the script, it will re-open itself in a +# gnome-terminal window so the output is scrollable. The "T" arg is the +# internal signal that says "we're already inside the terminal, don't +# re-open." +# +# ============================================================================= -# I use this script to update project website. -# HTML files are NOT versioned. They are generated on-demand from .org files. - +# --- gnome-terminal re-launch trick ------------------------------------------- +# This line says: if we are NOT being run with the internal "T" flag, re-launch +# ourselves inside a gnome-terminal window and exit the original process. This +# gives us a scrollable terminal without requiring the user to manually open +# one. The "${0%/*}" part strips the script name from $0 to get the script's +# own directory, so `cd` runs us into the right place regardless of where we +# were invoked from. cd "${0%/*}"; if [ "$1" != "T" ]; then gnome-terminal -- "$0" T; exit; fi; +# Now we're inside the terminal. Change into the repo root (one level up from +# the Tools/ directory where this script lives). cd .. -# Function to export org to html using emacs in batch mode +# --- Configuration ------------------------------------------------------------ + +# Path to the cache file (relative to the repo root). +STATE_FILE=".org-export-state" + +# --force / --rebuild: ignore the cache and re-export everything. +FORCE=0 +for arg in "$@"; do + case "$arg" in + --force|--rebuild) FORCE=1 ;; + esac +done + +# --- Helper functions --------------------------------------------------------- +# These small functions are defined first so the main export logic below +# reads top-to-bottom. Each one does exactly one thing. + +# file_sha +# Print the sha256 hash of a file, or an empty string if the file does not +# exist. sha256sum always prints " ", so we use awk to keep +# only the first column. +file_sha() { + [ -f "$1" ] && sha256sum "$1" | awk '{print $1}' +} + +# write_state_file +# Save the current STATE_HASHES associative array and STYLE_SENTINEL +# variable to disk. We write to a temp file first, then `mv` it into place — +# this is the standard "atomic write" pattern. If the script were killed +# mid-write, you'd otherwise end up with a half-written state file; with +# mv, you either get the old file or the new file, never a broken one. +# +# The "$$" in the temp filename is bash's process ID — it makes the temp +# filename unique even if two copies of the script run at the same time. +write_state_file() { + local tmp="${STATE_FILE}.tmp.$$" + { + echo "# svjatoslav-physical incremental-export state. Do not edit by hand." + echo "# Lines starting with '#' are comments. Empty lines ignored." + echo "# Format: " + # "${!ARRAY[@]}" expands to the list of KEYS in an associative array. + for path in "${!STATE_HASHES[@]}"; do + echo "${STATE_HASHES[$path]} $path" + done + echo "sentinel-style-css: ${STYLE_SENTINEL} style.css" + } > "$tmp" + mv "$tmp" "$STATE_FILE" +} + +# load_state_file +# Read the state file from disk and populate STATE_HASHES and STYLE_SENTINEL. +# Returns 0 (success) if the file was loaded, 1 if it was missing. +# +# The `while IFS= read -r line` loop is the standard bash idiom for reading +# a file line-by-line: +# - IFS= prevents leading/trailing whitespace from being stripped +# - -r prevents backslashes from being interpreted as escapes +# - < "$STATE_FILE" redirects the file into the loop's stdin +# +# The state file uses two line formats: +# " " — one entry per .org file +# "sentinel-style-css: style.css" — the style sentinel +# We distinguish them with a case-statement on the prefix. +load_state_file() { + [ -f "$STATE_FILE" ] || return 1 + # `declare -gA` declares a global associative array (-A = key/value, -g = + # visible to the caller, not just this function). We have to redeclare + # inside this function because bash's scoping rules don't automatically + # see the caller's declarations. + declare -gA STATE_HASHES + STATE_HASHES=() + STYLE_SENTINEL="" + while IFS= read -r line; do + case "$line" in + ""|\#*) continue ;; # skip blank lines and comments + sentinel-style-css:*) + # Strip the "sentinel-style-css:" prefix, then trim spaces. + # ${var#prefix} = remove prefix, ${var%suffix} = remove suffix. + STYLE_SENTINEL="${line#sentinel-style-css:}" + STYLE_SENTINEL="${STYLE_SENTINEL%% *}" + STYLE_SENTINEL="${STYLE_SENTINEL##* }" + ;; + *) + # Format: " " — two spaces separate the fields. + local hash="${line%% *}" # everything before the first " " + local path="${line#* }" # everything after the first " " + STATE_HASHES["$path"]="$hash" + ;; + esac + done < "$STATE_FILE" + return 0 +} + +# update_state_hash +# Update the in-memory cache for one .org file. The cache is flushed to +# disk by write_state_file at the end of the run, not on every update — +# this avoids disk I/O in the hot loop. +update_state_hash() { + local path="$1" + local hash="$2" + STATE_HASHES["$path"]="$hash" +} + +# export_org_to_html +# Convert one .org file to .html using Emacs in batch mode. +# +# We run the conversion inside a subshell — the "( ... )" parens below — +# so that the `cd "$dir"` only affects the export, not the main script. +# Without the subshell, the script's own working directory would get stuck +# in the last directory we exported. +# +# --batch : non-interactive mode +# -l ~/.emacs : load your init file (defines darksun theme etc.) +# --visit= : open the .org file +# --funcall=... : call the org-html-export function +# --kill : exit Emacs when done export_org_to_html() { local org_file=$1 local dir=$(dirname "$org_file") @@ -15,6 +182,8 @@ export_org_to_html() { ( cd "$dir" || return 1 local html_file="${base}.html" + # Delete the old HTML so we don't accidentally read a stale one + # if the export fails partway through. if [ -f "$html_file" ]; then rm -f "$html_file" fi @@ -22,6 +191,7 @@ export_org_to_html() { emacs --batch -l ~/.emacs --visit="${base}.org" --funcall=org-html-export-to-html --kill if [ $? -eq 0 ]; then echo "✓ Successfully exported $org_file" + return 0 else echo "✗ Failed to export $org_file" return 1 @@ -29,46 +199,124 @@ export_org_to_html() { ) } +# --- Main export routine ------------------------------------------------------ + +# export_org_files_to_html +# The main work: find all .org files, decide which ones to skip, and call +# export_org_to_html on the rest. At the end, writes the updated state file. export_org_files_to_html() { echo "🔍 Searching for .org files ..." echo "=======================================" - mapfile -t ORG_FILES < <(find . -type f -name "*.org" | sort) + # mapfile -t ARRAY < <(command) reads the lines of `command`'s output + # into ARRAY. The -t strips trailing newlines. We exclude AGENTS.org + # because it is for agents (this file's sibling), not for the public site. + mapfile -t ORG_FILES < <(find . -type f -name "*.org" ! -name "AGENTS.org" | sort) if [ ${#ORG_FILES[@]} -eq 0 ]; then echo "❌ No .org files found!" return 1 fi - echo "Found ${#ORG_FILES[@]} .org file(s):" + echo "Found ${#ORG_FILES[@]} .org file(s) (excluded: AGENTS.org):" printf '%s\n' "${ORG_FILES[@]}" echo "=======================================" + # --- Load the cache (unless --force was given) --------------------------- + # `declare -A` makes STATE_HASHES a local associative array. If the state + # file doesn't exist, this is the first run and we'll export everything. + declare -A STATE_HASHES=() + STYLE_SENTINEL="" + if [ "$FORCE" -eq 0 ]; then + if load_state_file; then + echo "📋 Loaded state from $STATE_FILE (${#STATE_HASHES[@]} files, sentinel ${STYLE_SENTINEL:-})" + else + echo "📋 No $STATE_FILE found — first run, will create one" + fi + else + echo "📋 --force specified, ignoring $STATE_FILE" + fi + + # --- Sentinel check ------------------------------------------------------ + # If style.css has been modified since the last run, invalidate the cache + # for every page. This is the "safety net" for the common case of + # editing CSS: you don't have to remember --force. + local current_style_hash + current_style_hash=$(file_sha "style.css") + local style_changed=0 + if [ "$FORCE" -eq 0 ] && [ -n "$STYLE_SENTINEL" ] && [ "$STYLE_SENTINEL" != "$current_style_hash" ]; then + echo "🎨 style.css sentinel changed (${STYLE_SENTINEL:0:12}… → ${current_style_hash:0:12}…) — invalidating all pages" + style_changed=1 + fi + if [ -z "$STYLE_SENTINEL" ]; then + STYLE_SENTINEL="$current_style_hash" + fi + + # --- The main loop ------------------------------------------------------- SUCCESS_COUNT=0 FAILED_COUNT=0 + SKIPPED_COUNT=0 for org_file in "${ORG_FILES[@]}"; do + local current_hash + current_hash=$(file_sha "$org_file") + + # "${ARR[key]:-}" returns the value or an empty string if unset. + # We use it to fetch the cached hash for this file. + local cached_hash="${STATE_HASHES[$org_file]:-}" + + # Skip the export if: + # - --force is NOT set, AND + # - the sentinel hasn't invalidated the cache, AND + # - we have a cached hash for this file, AND + # - the cached hash matches the current hash + if [ "$FORCE" -eq 0 ] && [ "$style_changed" -eq 0 ] && [ -n "$cached_hash" ] && [ "$cached_hash" = "$current_hash" ]; then + echo "↻ Skipping (unchanged): $org_file" + # ((VAR++)) is arithmetic increment — only works because VAR=0. + ((SKIPPED_COUNT++)) + continue + fi + export_org_to_html "$org_file" if [ $? -eq 0 ]; then + # Update the in-memory cache; written to disk after the loop. + update_state_hash "$org_file" "$current_hash" ((SUCCESS_COUNT++)) else ((FAILED_COUNT++)) fi done + # --- Persist the updated cache ------------------------------------------ + write_state_file + echo "💾 Wrote state to $STATE_FILE" + echo "=======================================" echo "📊 SUMMARY:" - echo " ✓ Successful: $SUCCESS_COUNT" - echo " ✗ Failed: $FAILED_COUNT" - echo " Total: $((SUCCESS_COUNT + FAILED_COUNT))" + echo " ✓ Exported: $SUCCESS_COUNT" + echo " ↻ Skipped: $SKIPPED_COUNT" + echo " ✗ Failed: $FAILED_COUNT" + echo " Total: $((SUCCESS_COUNT + SKIPPED_COUNT + FAILED_COUNT))" echo "" } -# Publish Emacs org-mode files into HTML format +# --- Run it ------------------------------------------------------------------- + +# Convert all .org files to .html (with the cache logic above). export_org_files_to_html -## Upload assembled documentation to server -# Exclude tools/ and .git/, exclude all .org files (source), keep generated .html and assets +# --- Upload to the web server ------------------------------------------------ +# rsync flags used here: +# -a : archive mode (preserves permissions, timestamps, etc.) +# -v : verbose (list every file) +# -z : compress during transfer +# --delete : delete files on the server that no longer exist locally +# -e 'ssh -p 10006' : use ssh on port 10006 for the connection +# +# We exclude: +# tools/ — this directory contains the script itself, not published content +# .git/ — version control metadata +# *.org — source files, only the generated HTML goes public rsync -avz --delete -e 'ssh -p 10006' \ --exclude="tools/" \ --exclude=".git/" \ @@ -82,6 +330,8 @@ else echo "✗ Upload failed!" fi +# Keep the terminal window open so the user can read the output. The `read` +# without a variable just waits for any keypress before exiting. echo "" echo "Press ENTER to close this window." read