Grove Admin Guide

deploy.sh

The canonical fleet deploy script. Ships 5 Python files + relay.py + assets/ dir.

# Deploy to mac (dev, default)
bash deploy.sh mac

# Deploy to all nodes
bash deploy.sh

# Skip test suite
bash deploy.sh --skip-tests

# Roll back the previous build on a node
bash deploy.sh --rollback mac

# Promote the soaked build to production (FN → nookman → funbook, canary order)
bash deploy.sh --promote

--promote requires a clean working tree, a full gauntlet pass on mac, and refuses --skip-tests. It deploys familynook first (x86 canary), waits (default 30 s, override with CANARY_SOAK_SEC=N), then nookman, then funbook. nook + palooza are never deploy-pushed — they pick up the signed release via self-update.

Build hash = SHA-256 of grove.py + web.py + acme.py + acme_jws.py + watchdog.py + 12 frontend assets (assets/), first 12 hex chars. Both deploy.sh and web.py compute the same hash; mismatches surface in /api/version.

familynook special case

FN runs under a dedicated grove system account. deploy.sh scp's files to ~/ (nook's home), then calls bash deploy-grove.sh which uses sudo to copy them into /home/grove/.grove/. This requires a sudoers line on FN:

nook ALL=(grove) NOPASSWD: /bin/cp, /bin/bash

(See fn-grant-nopasswd.sh in the repo root for the exact grant.)

CI — hermetic gate

There is no GitHub Actions — origin is a bare repo on familynook. CI is a single

script, ci.sh, that runs the full hermetic suite (no live fleet needed):

bash ci.sh   # py_compile · ruff crash-class · render-smoke · pytest · render-diff (info)

HARD gates (fail the build): py_compile, ruff crash-class (F821/E9), render-smoke,

pytest tests/. render-diff runs informationally — template renders change

legitimately; re-baseline with python3 render-diff.py --update.

It's wired as an automatic pre-push gate (hooks/pre-push, active once you run

git config core.hooksPath hooks): a red suite blocks the push. Bypass a WIP push with

git push --no-verify. The fast per-commit hook only does crash-class checks; the pytest

suite runs at the push boundary.

Optional independent safety net — a cron runner on the dev box:

*/30 * * * * cd ~/grove && bash ci.sh >> /tmp/grove-ci.log 2>&1 || \
  osascript -e 'display notification "Grove CI is RED" with title "Grove CI"'

Truly non-bypassable upgrade (future): a server-side pre-receive hook on familynook's

grove.git that runs ci.sh and rejects a broken push — needs pytest/ruff installed

on FN, and a buggy hook can block all pushes, so it's deferred.

systemd Services

These unit files live in systemd/ and are installed via sudo bash systemd/install.sh.

sudo bash systemd/install.sh            # grove + watchdog
sudo bash systemd/install.sh --with-relay  # also installs grove-relay

grove.service — main daemon

[Unit]
Description=Grove Distributed Storage
After=network-online.target tailscaled.service

[Service]
Type=simple
User=grove
WorkingDirectory=/home/grove/.grove
ExecStart=/usr/bin/python3 /home/grove/.grove/web.py
Restart=always
RestartSec=5
MemoryMax=512M
ReadWritePaths=/home/grove/.grove /home/grove/GroveHome /tmp

grove-watchdog.service — crash recovery + panic page

[Service]
Type=simple
ExecStart=/usr/bin/python3 /home/grove/.grove/watchdog.py --no-redirect
Restart=always
RestartSec=3

Watchdog panic page at :5679. When Grove (5678) is down this page lets you restart it from a browser.

grove-ai.service — llama-server (AI hosts only)

Managed via the web UI or:

python3 ~/.grove/web.py ai-service install
python3 ~/.grove/web.py ai-service status
python3 ~/.grove/web.py ai-service uninstall

Type=forking; spawns the llama-server and exits. The watchdog defers to this unit when it is active. Do NOT enable on a Metal Mac — a full-offload server wires unevictable memory.

grove-relay.service — WebSocket relay (optional)

Only needed on a dedicated relay host. Most cells auto-connect to the fleet relay; this unit is not required for normal operation.

Pi keepalive cron (no grove.service)

Pis run grove-watchdog.service instead of grove.service. A */5 cron provides the reboot bootstrap and proactive memory management:

# crontab -e (as the grove user)
*/5 * * * * ~/.grove/grove-health-local.sh >> ~/.grove/health.log 2>&1

deploy.sh ships grove-health-local.sh to each Pi automatically.

# Install the watchdog service on a Pi
python3 ~/.grove/web.py watchdog-service install
sudo systemctl enable --now grove-watchdog

Reverse Proxy / Gateway / TLS

Any cell can act as a WAN gateway. Two options:

Option A — Grove's built-in ACME (Let's Encrypt HTTP-01)

Set public_url in ~/.grove/config.json and ensure port 80 is reachable. acme.py handles cert issuance and renewal automatically.

Option B — External nginx (familynook's setup)

nginx-grove.conf in the repo root is the reference config for grove.nook.li.

SECURITY-CRITICAL: on a public gateway, nginx must inject X-Grove-Public-Edge: 1 on every request proxied to :5678, overwriting any client-supplied value (since :5678 isn't directly reachable from the internet, this header is a trustworthy public-vs-tailnet signal). Grove's check_dashboard_auth uses this header to block the owner dashboard session and admin /api/* routes over the public edge — they remain tailnet-only. Without this tag, a valid owner session cookie obtained over Tailscale could be replayed over the public internet to unlock the full control panel.

# Required addition to every proxy_pass block that reaches :5678 from the public edge:
proxy_set_header X-Grove-Public-Edge "1";

The public edge serves: /portal, /site/, /invite/, /relay, /.well-known/, /login, /api/version, /api/health, and peer-sync APIs. The owner dashboard (/dashboard, /files, admin /api/*) is tailnet-only.

Relay WebSocket passes through to :5680 (relay.py) — no X-Grove-Public-Edge needed there.

# Relay — no grove auth, separate port
location /relay {
    proxy_pass http://127.0.0.1:5680;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 86400;
}

SSL with Certbot (nginx path)

sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d grove.nook.li
# Renewal is automatic via certbot.timer

Configuration

~/.grove/config.json key fields:

{
    "self_name": "my-cell",
    "web_port": 5678,
    "web_host": "0.0.0.0",
    "public_url": "https://grove.nook.li",
    "desired_factor": 2,
    "relay_url": "ws://100.96.243.69:5680",
    "relay_host": true,
    "storage_cap_gb": 50,
    "peers": [],
    "ai_enabled": true,
    "peer_model_policy": "friends",
    "disable_auto_update": false
}

peer_model_policy controls which peers may use this cell's local AI: "friends" (default), "all", "public", "active-only", or "none".

Dev / Prod Tracks + Rollback

Production = nookman + familynook + funbook (signed releases via --release/--promote). Dev = mac + cell2 (cutting edge, direct deploy.sh). nook is the first-line dogfood cell and palooza a friend's external cell — both update via the self-update / dashboard path, never deploy.sh. cell1 is shelved.

Normal workflow:

1. Deploy to mac: bash deploy.sh mac

2. Let it soak; run gauntlet: python3 gauntlet.py

3. Promote to prod: bash deploy.sh --promote

If a build passes health checks but misbehaves:

bash deploy.sh --rollback familynook nookman

This restores the .bak snapshot that deploy.sh captured before the deploy.

AI Hosting

Grove uses llama.cpp (not ollama). Models live in ~/.grove/.

• Set up via the AI tab in the dashboard (UI-driven download, build, enable)
• grove-ai.service supervises the llama-server on Linux/systemd hosts
• The */5 keepalive cron also starts the AI server if it's configured but not running (ensure_ai in grove-health-local.sh)
• Do NOT auto-start the llama-server at boot on a Mac with Metal — use the AI tab to start on demand

Manage via CLI:

python3 ~/.grove/web.py ai-service install    # install grove-ai.service
python3 ~/.grove/web.py ai-service status

Monitoring

• GET /api/version — build hash, version, uptime
• GET /api/health — disk, chunk count, peer count
• GET /api/replication-status — per-peer replication health
• GET /api/route-speeds — route performance data
• Watchdog panic page at :5679 — visible when Grove is down

Logs:

• /tmp/grove-web.log — main daemon stdout/stderr
• /tmp/grove-watchdog.log — watchdog log
• journalctl -u grove -f — on systemd boxes
• ~/.grove/health.log — keepalive cron log (Pis)

Backup & Recovery

Back up identity keys (critical)

# Via dashboard: Settings → Backup Keys (downloads ZIP)
# Via API:
curl -sf http://localhost:5678/api/backup-keys > grove-keys.zip

If you lose ~/.grove/.key, your encrypted files are unrecoverable.

Restore keys

# Via dashboard: Settings → Restore Keys (upload ZIP)
# Via API:
curl -sf -X POST http://localhost:5678/api/restore-keys -F "file=@grove-keys.zip"

Forgot dashboard password

Three paths, in order of friction:

1. Logged in elsewhere? Settings → Recovery → Generate recovery link. Single-use, valid 24 h, resets only this cell.

2. Locked out everywhere? From a terminal on the device:

   grove auth set-password

3. No grove CLI? Visit /setup from localhost to set a new password.

Leaving Grove

• This device only (data stays on the network): Settings → Leave Grove → "Leave from this device."
• Permanent network departure (peers delete your chunks):

  grove apoptosis

Type APOPTOSIS to confirm. Signed tombstone propagates to peers. Irreversible.

Operational Safety

Never run network-severing commands over SSH on a remote node. If you sever your own connection, the Pi nodes may be unreachable for days.

Dangerous over SSH (requires a recovery timer or on-box scheduled job):

• nmcli con down/modify, systemctl restart NetworkManager
• iptables -F or any firewall flush
• tailscale down, tailscale logout
• ip link set … down, reboots without a recovery at job

pkill web.py over SSH kills your own shell (the SSH session process matches the grep). Restart by PID instead:

# Find the PID
cat ~/.grove/grove-web.pid
# Kill just that PID
kill <pid>
# Restart
cd ~/.grove && nohup python3 web.py </dev/null >/tmp/grove-web.log 2>&1 &

Before restarting Grove, check for queued image-gen jobs:

python3 -c "import json; jobs=json.load(open('$HOME/.grove/ai_image_jobs.json')); print([j for j in jobs if j.get('status') in ('queued','running')])"

A blind restart kills the in-process worker; queued/running jobs become errors and must be re-submitted.

Security Hardening Checklist

• [ ] Private key files are chmod 600
• [ ] ~/.grove directory is chmod 700
• [ ] Dashboard password is set and strong (grove auth set-password)
• [ ] Public-facing cells behind nginx with HTTPS
• [ ] nginx injects X-Grove-Public-Edge: 1 on every request proxied to :5678
• [ ] Session cookies have HttpOnly + SameSite flags (automatic)
• [ ] Peer secret comparison is timing-safe (automatic)
• [ ] Owner dashboard is NOT reachable from the public internet (verify: curl -I https://grove.nook.li/dashboard should redirect to login, not render the dashboard)