← Dashboard · Docs ·Admin Guide

Grove Admin Guide

<!-- grove:last-verified v1.98 -->

> Build identity is per-release — check grove version or /api/version.

Fleet Overview

Cell Host Class Notes
mac localhost dev M4 16 GB; cutting edge, direct deploy.sh (local cp)
cell2 100.121.6.45 dev Pi 4 GB; real Pi/systemd — catches systemd-only bugs; Tailscale SSH
nookman 100.96.243.69 prod Pi 8 GB; jump host for cell2/palooza; signed releases
familynook (FN) 100.126.143.83 prod x86 WAN gateway; grove.nook.li; nginx TLS; canary-first on --promote
funbook 100.81.15.45 prod · catch-up tier Linux laptop on a cellular hotspot; signed releases + counted for durability, but it cannot block a release — unreachable during a --promote it is skipped loudly and caught up later
nook 100.96.150.107 prod · first-line Tucker's dogfood cell — updates via self-update / dashboard UI ONLY, never deploy.sh
palooza 100.86.29.8 external A friend's cell (cell1's old 8 GB chassis); self-update path; stays peered/friended

cell1 (Pi 8 GB, 100.104.249.123) is shelved — SD pulled, chassis repurposed as palooza.

Port 5678 everywhere. Port 5679 = watchdog panic page. Port 5680 = relay WebSocket.

See CLAUDE.md "Fleet Nodes & config" for the canonical class/topology table.

Deployment

Quick Start (new cell via invite)


curl -s https://grove.nook.li/invite/<token> | bash

Manual Install


mkdir -p ~/.grove
# Deps are pinned + hash-verified (#184) — install from the constraints file, not loose:
pip3 install --require-hashes -r constraints.txt
#   flask, flask-socketio, cryptography, requests, websockets, pynacl, gevent,
#   gevent-websocket, markdown (+ transitive) — every wheel sha256-pinned; a hash
#   mismatch aborts (tamper signal). install.sh embeds the same constraints inline.
# Copy grove.py, web.py, acme.py, acme_jws.py, watchdog.py, relay.py, grove-mountd.py + assets/ to ~/.grove/
cd ~/.grove && nohup python3 web.py </dev/null >/tmp/grove-web.log 2>&1 &
# Dashboard at http://localhost:5678

Use </dev/null on nohup invocations (especially over SSH) to prevent SIGHUP.

deploy.sh

The canonical fleet deploy script. Ships the build-hash set (grove.py, web.py, acme.py, acme_jws.py, watchdog.py) + the signed companions relay.py and grove-mountd.py + release.sig + the assets/ dir (deploy.sh SRC_FILES).


# Deploy to mac (dev, default)
bash deploy.sh mac

# Deploy to all nodes
bash deploy.sh

# Skip test suite
bash deploy.sh --skip-tests

# Roll back the previous build on a node
bash deploy.sh --rollback mac

# Promote the soaked build to production (required canary chain: FN → nookman)
bash deploy.sh --promote

--promote requires a clean working tree, a full gauntlet pass on mac, and refuses --skip-tests. It deploys the required canary chain — familynook first (x86 canary), waits (default 30 s, override with CANARY_SOAK_SEC=N), then nookman — and a failure there aborts the promote and offers a rollback.

funbook is a catch-up tier (2026-08-19): probed before the gauntlet, promoted after the required chain when reachable, and when it is not, the gauntlet is told not to gate on it, it is reported as NOT promoted, and a marker makes the next plain deploy keep saying so until bash deploy.sh funbook catches it up. The skip is never silent. nook + palooza are never deploy-pushed — they pick up the signed release via self-update.

Disk-headroom precheck (#167). Before writing source to any node deploy.sh checks free space in ~/.grove and refuses to deploy below DEPLOY_MIN_FREE_MB — a scp onto a full disk can leave null-byte (0-length) files that brick the cell (the cell2 incident). Free space and retry, or raise the floor via env if you know the write is safe.

Build hash = SHA-256 of grove.py + web.py + acme.py + acme_jws.py + watchdog.py + 12 frontend assets (assets/), first 12 hex chars. Both deploy.sh and web.py compute the same hash; mismatches surface in /api/version.

familynook special case

FN runs under a dedicated grove system account. deploy.sh scp's files to ~/ (nook's home), then calls bash deploy-grove.sh which uses sudo to copy them into /home/grove/.grove/ and restart the service. Because that path runs sudo systemctl restart grove plus sudo mkdir/cp/chown/tee into grove's store, the nook deploy user is granted broad passwordless sudo on FN (not a narrow per-binary rule).

(See fn-setup-assets.sh in the repo root for the FN provisioning helper.)

CI — hermetic gate

There is no GitHub Actions — origin is a bare repo on familynook. CI is a single

script, ci.sh, that runs the full hermetic suite (no live fleet needed):


bash ci.sh   # py_compile · ruff crash-class · render-smoke · pytest · render-diff (info)

HARD gates (fail the build): py_compile, ruff crash-class (F821/E9), render-smoke,

pytest tests/. render-diff runs informationally — template renders change

legitimately; re-baseline with python3 render-diff.py --update.

It's wired as an automatic pre-push gate (hooks/pre-push, active once you run

git config core.hooksPath hooks): a red suite blocks the push. Bypass a WIP push with

git push --no-verify. The fast per-commit hook only does crash-class checks; the pytest

suite runs at the push boundary.

Optional independent safety net — a cron runner on the dev box:


*/30 * * * * cd ~/grove && bash ci.sh >> /tmp/grove-ci.log 2>&1 || \
  osascript -e 'display notification "Grove CI is RED" with title "Grove CI"'

Truly non-bypassable upgrade (future): a server-side pre-receive hook on familynook's

grove.git that runs ci.sh and rejects a broken push — needs pytest/ruff installed

on FN, and a buggy hook can block all pushes, so it's deferred.

systemd Services

These unit files live in systemd/ and are installed via sudo bash systemd/install.sh.


sudo bash systemd/install.sh            # grove + watchdog
sudo bash systemd/install.sh --with-relay  # also installs grove-relay

Note the scope split: the small systemd/install.sh here installs only grove + watchdog (+ relay with the flag). The one-liner install.sh (repo root) is a superset — it also installs and enables grove-mountd.service when grove-mountd.py is present. On a cell that wants UI-driven USB mounting, install via the root install.sh (or drop the mountd unit in manually).

grove.service — main daemon

Abbreviated below — see systemd/grove.service for the full unit (it also carries a

security-hardening block: NoNewPrivileges, ProtectSystem=strict, ProtectHome=no,

PrivateTmp, ProtectKernel*/ControlGroups, RestrictSUIDSGID).


[Service]
Type=simple
User=grove
Group=grove
WorkingDirectory=/home/grove/.grove
ExecStart=/usr/bin/python3 /home/grove/.grove/web.py
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=5
StartLimitIntervalSec=300      # restart-storm guard: at most
StartLimitBurst=5              #   5 restarts / 5 min, else the unit gives up
MemoryMax=512M
TasksMax=64
ReadWritePaths=/home/grove/.grove /home/grove/GroveHome /tmp

grove-watchdog.service — crash recovery + panic page


[Service]
Type=simple
ExecStart=/usr/bin/python3 /home/grove/.grove/watchdog.py --no-redirect
Restart=always
RestartSec=3

Watchdog panic page at :5679. When Grove (5678) is down this page lets you restart it from a browser.

grove-ai.service — llama-server (AI hosts only)

Managed via the web UI or:


python3 ~/.grove/web.py ai-service install
python3 ~/.grove/web.py ai-service status
python3 ~/.grove/web.py ai-service uninstall

Type=forking; spawns the llama-server and exits. The watchdog defers to this unit when it is active. Do NOT enable on a Metal Mac — a full-offload server wires unevictable memory.

grove-relay.service — WebSocket relay (optional)

Only needed on a dedicated relay host. Most cells auto-connect to the fleet relay; this unit is not required for normal operation.

grove-mountd.service — privileged USB mount helper

The one Grove component that runs as root — it performs the actual mount/umount/fstab for UI-driven USB drives (§9). The unprivileged web app is its only client, over /run/grove-mountd.sock (owned root:<grove-user>, mode 0660); ops are typed + allowlisted, no shell. Installed and enable --now'd by the root install.sh (not the small systemd/install.sh), only when grove-mountd.py is present. Backs the Storage / USB drive actions below.

Pi keepalive cron (no grove.service)

Pis run grove-watchdog.service instead of grove.service. A */5 cron provides the reboot bootstrap and proactive memory management:


# crontab -e (as the grove user)
*/5 * * * * ~/.grove/grove-health-local.sh >> ~/.grove/health.log 2>&1

deploy.sh ships grove-health-local.sh to each Pi automatically.


# Install the watchdog service on a Pi
python3 ~/.grove/web.py watchdog-service install
sudo systemctl enable --now grove-watchdog

Reverse Proxy / Gateway / TLS

Any cell can act as a WAN gateway. Two options:

Option A — Grove's built-in ACME (Let's Encrypt HTTP-01)

Set public_url in ~/.grove/config.json and ensure port 80 is reachable. acme.py handles cert issuance and renewal automatically.

Option B — External nginx (familynook's setup)

nginx-grove.conf in the repo root is the reference config for grove.nook.li.

SECURITY-CRITICAL: on a public gateway, nginx must inject X-Grove-Public-Edge: 1 on every request proxied to :5678, overwriting any client-supplied value (since :5678 isn't directly reachable from the internet, this header is a trustworthy public-vs-tailnet signal). Grove's check_dashboard_auth uses this header to block the owner dashboard session and admin /api/* routes over the public edge — they remain tailnet-only. Without this tag, a valid owner session cookie obtained over Tailscale could be replayed over the public internet to unlock the full control panel.


# Required addition to every proxy_pass block that reaches :5678 from the public edge:
proxy_set_header X-Grove-Public-Edge "1";

The public edge serves: /portal, /site/, /invite/, /relay, /.well-known/, /login, /api/version, /api/health, and peer-sync APIs. The owner dashboard (/dashboard, /files, admin /api/*) is tailnet-only.

Relay WebSocket passes through to :5680 (relay.py) — no X-Grove-Public-Edge needed there.


# Relay — no grove auth, separate port
location /relay {
    proxy_pass http://127.0.0.1:5680;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 86400;
}

SSL with Certbot (nginx path)


sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d grove.nook.li
# Renewal is automatic via certbot.timer

Configuration

~/.grove/config.json key fields:


{
    "self_name": "my-cell",
    "web_port": 5678,
    "web_host": "0.0.0.0",
    "public_url": "https://grove.nook.li",
    "desired_factor": 2,
    "relay_url": "ws://100.96.243.69:5680",
    "relay_host": true,
    "storage_cap_gb": 50,
    "peers": [],
    "ai_enabled": true,
    "peer_model_policy": "friends",
    "disable_auto_update": false,
    "sync_window": {"enabled": false, "start": "01:00", "end": "06:00"}
}

peer_model_policy controls which peers may use this cell's local AI: "friends" (default), "all", "public", "active", or "none".

Sync Window

Restrict heavy background work (chunk replication, integrity sweep, durability accounting) to a daily window in the cell's local time — useful on a metered link or a box you don't want churning all day. Light work (serving, sharing, chat, presence, discovery, hygiene) always runs, so the cell stays reachable and responsive regardless.


grove sync-window 01:00 06:00   # heavy sync only 01:00–06:00 (wraps midnight if start > end)
grove sync-window off           # always-on (default)

Or via the API (owner-tier): POST /api/sync-window {enabled, start:"HH:MM", end:"HH:MM"} — it returns the updated sync_window and whether heavy sync is allowed right now (in_sync_window, also reported by the opp-sync status API). A critical-durability override still forces replication outside the window if any chunk this cell owns would drop to a single copy (critical_durability_overrides, default on).

Dev / Prod Tracks + Rollback

Production = familynook + nookman (the required canary chain) + funbook (same class and durability, but a catch-up tier that never blocks a release), all on signed releases via --release/--promote. Dev = mac + cell2 (cutting edge, direct deploy.sh). nook is the first-line dogfood cell and palooza a friend's external cell — both update via the self-update / dashboard path, never deploy.sh. cell1 is shelved.

Normal workflow:

1. Deploy to mac: bash deploy.sh mac

2. Let it soak; run gauntlet: python3 gauntlet.py

3. Promote to prod: bash deploy.sh --promote

If a build passes health checks but misbehaves:


bash deploy.sh --rollback familynook nookman

This restores the .bak snapshot that deploy.sh captured before the deploy.

Self-update health gate + quarantine

nook + palooza (and eventually family cells) pull the latest signed release themselves — ordered strictly by RELEASE_VERSION, never build hash. A freshly self-applied build must prove itself before it's committed:

AI Hosting

Grove uses llama.cpp (not ollama). Models live in ~/.grove/.

Supply-chain pinning (#184): the llama.cpp source build is pinned to a specific release tag (reproducible build vs floating HEAD), and the starter GGUF model is verified against a pinned sha256 on download — a checksum mismatch aborts. This is the AI-side counterpart to the hash-pinned pip deps in Install.

Manage via CLI:


python3 ~/.grove/web.py ai-service install    # install grove-ai.service
python3 ~/.grove/web.py ai-service status

Storage / USB Drives

USB drives can be added and mounted from the dashboard Storage card — no shelling in. The privileged mounting is done by grove-mountd (see systemd section): the unprivileged web app talks to it over /run/grove-mountd.sock, and mountd performs the actual mount/umount/fstab as root, with a typed + allowlisted op set (no shell). Endpoints (all owner-tier):

If grove-mountd.py / the unit isn't installed, the mount actions degrade gracefully (the UI shows status but can't add/eject).

Monitoring

Peers are reached over the best available transport, best route first. Yggdrasil (a Tailscale-independent mesh, 0200::/7 addresses) is a live fallback transport: a cell self-reports its ygg address in /api/version, peers harvest it into their routes (#182), and best_route probes it only when the primary (Tailscale/LAN) path is unavailable.

Logs:

Backup & Recovery

Back up identity keys (critical)


# Via dashboard: Settings → Backup Keys (downloads ZIP)
# Via API:
curl -sf http://localhost:5678/api/backup-keys > grove-keys.zip

If you lose ~/.grove/.key, your encrypted files are unrecoverable.

Restore keys


# Via dashboard: Settings → Restore Keys (upload ZIP)
# Via API:
curl -sf -X POST http://localhost:5678/api/restore-keys -F "file=@grove-keys.zip"

Forgot dashboard password

Three paths, in order of friction:

1. Logged in elsewhere? Settings → Recovery → Generate recovery link. Single-use, valid 24 h, resets only this cell.

2. Locked out everywhere? From a terminal on the device:


   grove auth set-password

3. No grove CLI? Visit /setup from localhost to set a new password.

Leaving Grove

Type APOPTOSIS to confirm. Signed tombstone propagates to peers. Irreversible.

Operational Safety

Never run network-severing commands over SSH on a remote node. If you sever your own connection, the Pi nodes may be unreachable for days.

Dangerous over SSH (requires a recovery timer or on-box scheduled job):

pkill web.py over SSH kills your own shell (the SSH session process matches the grep). Restart by PID instead:


# Find the PID
cat ~/.grove/grove-web.pid
# Kill just that PID
kill <pid>
# Restart
cd ~/.grove && nohup python3 web.py </dev/null >/tmp/grove-web.log 2>&1 &

Before restarting Grove, check for queued image-gen jobs:


python3 -c "import json; jobs=json.load(open('$HOME/.grove/ai_image_jobs.json')); print([j for j in jobs if j.get('status') in ('queued','running')])"

A blind restart kills the in-process worker; queued/running jobs become errors and must be re-submitted.

Security Hardening Checklist

Local owner token (#189 Stage 1). Each cell keeps a per-cell ~/.grove/owner_token (mode 0600, generated on first use, never handed to peers). Local owner-auth paths (CLI, self-update, all localhost) accept it so they can stop depending on the fleet-shared peer secret — additive today (web accepts both the owner token and the peer secret on localhost owner paths), so a later stage can retire and rotate the shared secret. Treat it like a key: it's owner-equivalent on localhost.