← Dashboard · Docs ·Architecture

Grove Architecture

<!-- grove:last-verified v1.98 -->

> Build identity is per-release — check grove version or /api/version.

> The source of truth is always the code; this doc is a developer orientation map.

System Overview

Grove is a five-file core plus a twelve-file runtime asset set:

File Role
grove.py Core engine: crypto, chunking, manifests, placement, routing, sync, CLI (~765 KB)
web.py Flask web server: dashboard + portal UI, all REST/SocketIO API (~1.5 MB)
acme.py TLS cert provisioning (Let's Encrypt ACME)
acme_jws.py ACME JWS signing helpers
watchdog.py Process watchdog + panic page
assets/*.html/.css/.js Runtime-loaded templates; hashed into the build

web.py imports from grove.py. All six components run on every cell.

Build Hash & Versioning


BUILD_HASH = SHA-256(
    grove.py + web.py + acme.py + acme_jws.py + watchdog.py
    + sorted(RELEASE_ASSETS):
        assets/alphaTab.min.js + assets/Bravura.woff2
      + assets/dashboard.html + assets/error.html + assets/file-row.html
      + assets/invite-landing.html + assets/portal-dashboard.html
      + assets/portal-invite.html + assets/portal-login.html
      + assets/setup.html + assets/shared.css + assets/shared.js
)[:12 hex chars]

All 17 files are hashed in the order shown (5 py files in name order, then the 12 RELEASE_ASSETS in sorted order). web.py:_compute_build_hash() is canonical (== /api/version == deploy.sh == grove version). Note: relay.py and grove-mountd.py are signed companions (RELEASE_SIGNED_FILES, shipped by self-update) but deliberately not in the build hash — they don't change version identity. Build hash is a fingerprint only, never an ordering signal: two peers with the same hash never trigger an update, and hash/build_time differences do not decide "newer" (see Auto-Update).

Encryption

Identity Keys (~/.grove/)

File Key type Purpose
node.key Ed25519 private key (600) Cell identity, signing
node_x25519.key X25519 private key (600) Envelope encryption for manifest metadata
node_x25519.pub X25519 public key Shared with peers; embedded in manifest envelopes
.key 256-bit random master key (600) Per-file chunk key derivation

File Encryption Pipeline


master_key  ──HKDF-SHA256(salt="grove-v0.1", info="grove-chunk-encryption")──▶  per-file key
plaintext chunk  ──ChaCha20-Poly1305(per-file key, random 96-bit nonce)──▶  encrypted chunk
hash_chunk(plaintext)  ──BLAKE2b-256──▶  chunk_hash  (content address, stored as filename)

Key points:

Manifest Metadata Encryption (v1, §7)

v1 manifests encrypt the sensitive metadata (filename, source_path, version_note) inside a ChaCha20-Poly1305 blob:


metadata_key (32B random)
  ──encrypted for each recipient via X25519 KDF──▶  metadata_envelope {x25519_pubkey: ciphertext}
  ──ChaCha20-Poly1305(metadata_key)──▶  encrypted_metadata (base64 nonce ‖ ciphertext)

On v1 manifests the top-level filename field is null. The on-disk filename is <content_hash>.json (opaque). See INVARIANTS.md §4 for the rule + signable-byte spec (design note archived at archive/research/MANIFEST-METADATA-LEAK.md).

Chat / X25519 Key Exchange

Peer chat uses per-message X25519 Diffie-Hellman + ChaCha20-Poly1305. Each message is encrypted with a random nonce; there is no persistent session key.

Signing

Manifest Schema

Two schema generations coexist on disk and on the wire:

Field v0 v1
filename plaintext null
source_path plaintext null
version_note plaintext null
encrypted_metadata absent / null base64 blob
metadata_envelope absent / null {x25519_pub: wrapped_key, ...}
creator_pubkey optional required
signature optional required (covers "v1" tag + envelope canonical JSON)

Schema is discriminated by encrypted_metadata is not None. v0 manifests remain valid; they are decrypted/verified by the same FileManifest.from_json path.

On-disk filename: ~/.grove/manifests/<content_hash>.json (stable, opaque).

FileManifest dataclass (grove.py)

Core fields: total_size, chunk_size, chunks: list[ChunkInfo], creator_pubkey, signature, encrypted_file_key, creator_encryption_pubkey, shared_with, replication_policy, pending_delete_at, synced_at, version, previous_version.

get_content_hash() — BLAKE2b-256 of get_signable_data(), used as the on-disk filename and as the canonical grant key.

Other Core Dataclasses (grove.py)

Class Role
ChunkInfo hash, size, index — one chunk within a manifest
ShareGrant Sharing permission: manifest_hash, encrypted_file_key for recipient, Ed25519 signature
TombstoneAction Signed retire/delete propagation to peers
DeleteRequest (Currently disabled) signed delete request

Plaintext-Primary Storage Model

GroveHome (~/GroveHome/ by default) plaintext is one replica. The design avoids keeping redundant encrypted chunks on the origin cell when the plaintext source still exists.

load_chunk (grove.py)

Resolves an encrypted chunk from three sources in order:

1. Chunk drive — ~/.grove/chunks/{hash[:2]}/{hash} (or any registered chunk drive)

2. Regenerate from plaintext — _regenerate_chunk_from_plaintext(): reads the plaintext file from GroveHome (or a mirror drive's GroveHome clone), slices the correct window, encrypts with the per-file key

3. Returns None if neither source is available

The regeneration cache is ~100 MB LRU keyed by chunk_hash:file_key_fingerprint (so the same chunk under different keys caches separately).

Storage Drive Modes (grove.py get_storage_drives())

Mode Behaviour
fill Default chunk storage — fills up to capacity
mirror All chunks replicated here; includes a GroveHome clone
backup Distributed backup pool (capacity-sharing across backup drives, not per-drive redundant)
ingest Source of content only; files auto-ingested into Grove. Requires ingest_enabled=true to start

USB mount-from-UI (grove-mountd.py). Mounting a USB drive needs root, which the unprivileged Grove process lacks. A small signed root companion (grove-mountd.py, in RELEASE_SIGNED_FILES) listens on a localhost Unix socket (MOUNTD_SOCK, /run/grove-mountd.sock); web.py's _mountd_call() drives it from /api/drives/mount-add (list-block-devices → mount → persist to fstab). This lets the owner add and persist a drive from the dashboard without a shell.

vacuum / vacuum-versions (grove.py)

vacuum reclaims encrypted chunks whose plaintext still exists in GroveHome (regenerable on demand); it never deletes a chunk without a recoverable plaintext source, so it is safe when GroveHome is intact. vacuum_versions_command (vacuum-versions) additionally reclaims the chunks of superseded old file versions — see Hygiene & Reclamation below.

Data Flow: Ingest → Sync


ingest_file(filepath)              # wrapped in _ingest_path_lock (flock on the
  0. flock the canonical source path — a second ingest of the same path blocks,
     preventing forked duplicate v1 manifests of one file
  1. Read file, split into 4MB plaintext chunks
  2. hash_chunk(plaintext) → chunk_hash (BLAKE2b-256)
  3. encrypt_chunk(plaintext, file_key) → encrypted bytes
  4. Write encrypted chunk to ~/.grove/chunks/{hash[:2]}/{hash}
  5. Build FileManifest (v1): encrypt metadata, wrap keys, sign
  6. Write manifest to ~/.grove/manifests/<content_hash>.json
  7. Update placement.db (local holder = self)

Opp sync (every ~5 min, heavy work gated to the sync window — see below):
  8. For each reachable peer via best_route():
     a. Exchange manifest inventories (manifest hashes)
     b. Pull manifests + chunks peer has that we need
        (_cap_trim_pull trims the pull set to remaining offered-storage
         headroom, using held_foreign accounting — the offered cap binds on
         PULLs, not just pushes, #179)
     c. Push manifests + chunks they need (placement-driven)
     d. Sync grants (grove.db) and tombstones
  9. Update discovery (_check_peer_updates) → queue signed self-update if a
     peer advertises a strictly-higher SIGNED release_version (see Auto-Update)
 10. Self-heal: identify chunks below desired replication factor, push to fill

Sync window. sync_window_command lets the owner restrict heavy background work (replication, self-heal) to a daily window so a cell isn't churning during use. Light peer work (updates, tombstones, chat, presence) always runs; and critical_durability_overrides (default on) lets the durability-critical path punch through the gate when a chunk is at risk (_has_critical_durability_risk).

Placement & Replication

placement.db (SQLite, ~/.grove/placement.db)

Tracks which chunks exist on which peers (by pubkey), local chunk inventory, and replication factor. Migrated from placement.json in v0.4.

Replication logic:

Honest durability signal (audit #170, #172)

Durability is measured as "is every owned chunk held off this device by a PROD peer?", not pairwise parity. The real risk metric is zero_off_device — owned chunks with no copy off the local disk (the SPOF count, grove.py:~2697). _durability_exclude_pks() (web.py) drops dev cells (mac, cell2 — may be wiped mid-test) and external cells (palooza — a friended bonus peer) from the count, so durability reflects only Tucker's own PROD cells. /api/durability-facts surfaces the honest count to the dashboard.

Hygiene & Reclamation (v1.48 → v1.61)

Autonomous cleanup runs on a daily cadence gate (_version_reclaim_due, 86400 s) inside opp-sync (_maybe_reclaim_old_versions, web.py):

grove.db (SQLite, ~/.grove/grove.db)

Stores: share grants, bounty/bilateral ledger, growth events, milestones. The grants table carries a rel_path column (GroveHome-relative, plaintext, per-recipient) so a portal recipient can rebuild the folder tree for §7-opaque manifests.

Routing & Mesh

Route Types

Transport When used
lan Direct LAN IP (mDNS/Zeroconf discovery)
tailscale Tailscale VPN IP (100.x.x.x)
yggdrasil Yggdrasil overlay (202:...)
relay WebSocket relay (DERP-style, via ws(s)://…/relay)
multihop Multi-hop forwarding through an intermediate peer
wan / https Direct public IP

best_route() (grove.py)

best_route(peer) → Optional[dict] — wraps _best_route_impl. Probes all candidate routes for the peer in order: direct (LAN/Tailscale/WAN), UDP hole-punch (if a prior punch is cached), relay, multi-hop fallback. Returns the winning route dict or None.

Scoring: effective_score = latency - (throughput_MB/s × 0.1). Route speeds are benchmarked every 6 hours with real transfers and stored in ~/.grove/peer_speeds.json.

Probing is tiered — fast transports (LAN/Tailscale/WAN) are tried first and Yggdrasil last (it's a slower-to-warm but Tailscale-independent fallback, now live: warm ~0.4 s). Peers advertise their ygg address and record_peer_ygg_addr populates the route so ygg is actually reachable when the faster transports are down.

Topology Detect / Adapt

Phase-1/2/3 live in grove.py (shipped 2026-05-10):

Multi-Hop Forwarding

When no direct route exists, grove.py's routing falls back to the routing table (forward_to + intermediate peer). Forwarding is not an endpoint: the caller sets the X-Grove-Forward-To: <target-pubkey> header on the ORIGINAL request, and the intermediate peer's handle_forward_to before_request (web.py) proxies it on. That placement matters — it runs ahead of the auth gate and returns its own response, which is why it needs its own public-edge refusal (AUDIT-2026-07-25 P0-8). Chunk multipart pushes are re-batched to ≤8 MB at the forwarding hop, via /api/receive-chunks-b64.

UDP Hole Punch

After a successful TCP/HTTP probe establishes session context, grove can attempt a UDP punch to both sides simultaneously. If the punch succeeds, the UDP path is cached and preferred for subsequent chunk transfers.

WebSocket Relay


Cell A ──ws──▶ Relay server ──ws──▶ Cell B

Authentication

Peer API Auth

A direct (LAN/Tailscale) peer API call carries:

Per-peer derived secret (v0.4.1+):


shared = peer["shared_secret"]          # Bilaterally exchanged during handshake
pair   = sorted([my_pubkey, their_pubkey])
secret = HMAC-SHA256(shared, "|".join(pair)).hexdigest()

Symmetric (same value on both sides). Falls back to global peer_secret if no shared secret is stored.

Signed requests — secret-less peer auth (X-Grove-Sig, #124)

A request may instead authenticate with an Ed25519 signature in X-Grove-Sig (_verify_peer_request_sig, web.py). The signature covers method ‖ path ‖ sha256(body) ‖ timestamp ‖ sender_pk ‖ recipient_pk, so it is bound to this exact request and this recipient. It is checked first in verify_peer_request and, on success, short-circuits — no bearer secret needed, and it satisfies strict identity-binding. Because a relay holds no peer's private key, it cannot forge this signature; replay is blocked by a 300 s freshness window (_SIG_FRESH_WINDOW) plus a single-use replay cache (_sig_replay_seen). Dropping the shared secret on relay-forwarded calls has now shipped (#183): _relay_secret_droppable (grove.py) omits X-Grove-Secret unconditionally on the peer/public relay surface — the replication set + chunk-inventory, reachability, revoke-grant, version, ai/status, and the serve-chunk-peer/, serve-manifest/, grants-for/ prefixes — so a relay operator sees no replayable credential there. Only the owner-tier endpoints (friend-remove, peer-remove, delete-chunks, ai/proxy) still carry it, staged behind config relay_secret_drop_owner_tier (default OFF) until the whole fleet accepts the signed, secret-less call.

Local owner token (owner_token, #189)

Each cell writes a per-cell ~/.grove/owner_token (mode 600, never handed to peers). On localhost owner paths it is accepted alongside peer_secret; when the fleet-wide global secret is used instead, _note_shared_secret_use() records it (a soak-gauge) so the global secret can eventually be retired once nothing depends on it. It is log-only: counts live in the in-memory _SHARED_SECRET_USE dict and surface as 🔑 #189: log lines on first use per context — there is no HTTP route to query them, so read the cell's log.

Dashboard Auth (check_dashboard_auth, web.py)

Portal Auth (_portal_auth_required, web.py)

Portal has three identity tiers:

Onboarding & egress hardening

Web Layer

Template Architecture

All HTML is loaded at startup from assets/ via _load_asset():

Asset Used by
assets/dashboard.html HTML_TEMPLATE — main owner dashboard
assets/portal-dashboard.html PORTAL_DASHBOARD — portal for friends/visitors
assets/portal-login.html PORTAL_LOGIN_PAGE
assets/portal-invite.html PORTAL_INVITE_PAGE
assets/invite-landing.html INVITE_LANDING_PAGE
assets/setup.html SETUP_PAGE — first-run setup wizard
assets/error.html _ERROR_PAGE
assets/shared.css SHARED_CSS — injected via `{{ shared_css\ safe }}`
assets/shared.js SHARED_JS — injected via `{{ shared_js\ safe }}`

_portal_render(**kwargs) auto-injects shared_css and shared_js into every portal render. The cell dashboard does the same via render_template_string(HTML_TEMPLATE, ..., shared_css=SHARED_CSS, shared_js=SHARED_JS).

Dashboard Tabs

Home, Files, Feed, Chat, AI, Settings. Real-time updates via SocketIO. (Invites live inside Settings, not as a top-level tab.)

Home tab: simple_home=True (default) — health panel + activity feed + quick links. simple_home=False opts into the creature/XP/milestone-rich home (code on disk, not default-enabled).

Portal vs Dashboard

Dashboard Portal
Path /dashboard, /files, /api/* /portal/*
Auth Owner session or peer secret Owner / friend / visit-peer token
Access over public WAN Blocked (public-edge gate) Allowed (nginx exposes /portal/*)
Purpose Owner control panel Remote access + friend-facing view

Media Renderer (audio + video)

Folders tagged (or auto-detected) as the audio/video roles render as a streaming media library. The renderer is built on three additions:

1. Streaming spine — _manifest_range_response(m, file_key, mime_type, range_header, download_name=None) (web.py). The shared, chunk-wise, Range-aware streaming primitive. Grove chunks are fixed 4 MB plaintext, and each ChunkInfo carries its plaintext .size, so a requested byte range maps deterministically to the overlapping chunks. The function decrypts only those chunks (via load_chunk + decrypt_chunk) and streams them through a Flask generator Response — no whole-file buffering. It parses a single byte range (incl. suffix ranges), returns 206 for a range / 200 for the full body / 416 when unsatisfiable, and sets Accept-Ranges/Content-Range/Content-Length. The caller is responsible for access control and for resolving file_key. This one spine makes instant seek/scrub on large media (multi-GB FLAC/movies) work; both /view/<manifest> (owner) and /portal/stream/<manifest_file> (portal) call it when a Range header is present. Non-range requests (images, thumbnails, downloads) still assemble the whole file.

2. Media generalization. What began as an audio renderer was generalized to audio + video on the same spine. _detect_folder_role auto-tags a folder audio or video when ≥3 files of that kind AND ≥50% of the folder are that kind; audio/video join photos/notes/site/mixed as valid roles (persisted in folder_roles.json). /api/media-library (owner) and /portal/media-library (portal) build a folder-derived nested tree of dir/track nodes; each track carries media (audio/video), format, lossless, and web_playable. /api/audio-library and /portal/audio-library remain as back-compat aliases. The player UI is a single <video> element (which also plays audio) — gv-music / setupGvMusic / wireGvPlayer in assets/shared.js, styled in assets/shared.css, with the renderer blocks in assets/dashboard.html and assets/portal-dashboard.html.

3. Portal grant-path enrichment. Under §7 a portal recipient can't decrypt a manifest's own filename/source_path, so the folder tree would collapse to a flat list. To fix this the grants table gained a rel_path column (GroveHome-relative, plaintext, per-recipient). The sender stamps it (_share_to_portal_user → /api/share-grant, stored in api_share_grant), and _get_portal_manifests falls back to the grant's plaintext filename + rel_path when the manifest's own metadata is opaque — letting /portal/media-library (and the portal file tree generally) rebuild real names and folders.

4. Shared-with-me library parity (owner side). The owner's own Files view gets the same treatment as the portal: _build_shared_with_me_items + _prepend_shared_with_me_row render files others shared to Tucker under a synthetic SHARED_WITH_ME_PATH ("📥 Shared with me") sentinel, reconstructing a real Movies/Music library tree (dir/track nodes) from grant rel_paths — not a flat list. So a shared media folder renders as a browsable library on the cell dashboard, matching the portal.

Guitar-tab renderer (.gp*, §8)

Guitar Pro files render in a viewer-only tab renderer built on vendored alphaTab (assets/alphaTab.min.js + the assets/Bravura.woff2 music font — both in RELEASE_ASSETS/the build hash). It's lazy-loaded on first open (assets/shared.js, served at /lib/alphatab/…); /api/media-library?kinds=guitartab lists a folder's .gp* files; portal parity mirrors the cell. Playback is deferred to v2.

AI Subsystem

Local Inference

find_best_ai(peers) (web.py)

Selects the best inference target using a tiered routing strategy:

1. Local — local llama.cpp if running

2. Friends — mutual-friend peers that have AI enabled, gated by peer_model_policy

3. Escalation by difficulty tier (cheapest-capable-first)

Route-aware self-healing: stale peer AI hosts are re-probed via best_route() before use.

peer_model_policy

Config key controlling inbound AI proxy access:

E2E AI Proxy

When routing a query to a friend's cell, the prompt is E2E encrypted using the per-peer derived secret so intermediate relays cannot read the content. The receiving /api/ai/proxy decrypts using the same shared secret. status["e2e"] = True advertises this capability.

Growth Engine (grove.db)

Stored in grove.db tables growth_events and milestones:

The creature/XP home is hidden by default (simple_home=True).

Watchdog

watchdog.py runs as a separate process:

Auto-Update (P2P)

Discovery runs inside opp sync via _check_peer_updates (web.py, not grove.py). The sole "newer" signal is a strictly-higher signed release_version — build_hash/build_time/timestamps are never ordering inputs (they aren't monotonic and caused phantom "newer build" false-positives). A peer qualifies as an update source only when all hold:

1. peer_hash == BUILD_HASH → continue (identical build never updates, regardless of deploy time)

2. Peer advertises release_signed = true (a dev/unsigned/sideways build can never be a source)

3. Peer's release_version strictly out-ranks the running/pending floor

4. That release_version is not locally release-quarantined (a version that failed health/crash verification here is never re-pulled — else fail→rollback→reapply loops)

The peer's SIGNED manifest declares which files to stage; each staged name is traversal-guarded (_is_safe_stage_name) then the whole bundle is verified with verify_release — a valid Ed25519 release signature by a DEV_PUBKEYS signer over the bundle hash — and the staged bundle's own release_version is re-checked to be newer (defends a peer that lies in /api/version but serves an older signed bundle). Only then is it applied and the cell restarts. Auto-apply always requires a valid signature.

The transferred set is RELEASE_SIGNED_FILES = seven .py files (the 5 core + relay.py + grove-mountd.py) plus the 12 RELEASE_ASSETS. (That's a superset of the build hash, which fingerprints only the 5 core .py + 12 assets.) A slow-booting Pi no longer self-reverts a healthy build — the health gate rolls back only a reachable-but-wrong build, not one that's merely slow to answer (#151).

On-Disk Layout


~/.grove/
  node.key             Ed25519 private key (600)
  node_x25519.key      X25519 private key (600)
  node_x25519.pub      X25519 public key
  .key                 Master encryption key (600)
  peer_secret          Global API auth secret (600) — being retired (#189)
  owner_token          Per-cell local owner token (600; never shared, #189)
  dashboard_auth       PBKDF2 password hash (600)
  config.json          Cell config (peers, drives, relay, AI, sync window, etc.)
  placement.db         SQLite: chunk-to-peer placement
  grove.db             SQLite: grants, bounty/bilateral ledger,
                               growth events, milestones
  peer_speeds.json     Route benchmark history
  peer_health.json     Per-peer uptime records
  source_missing.json  Review-bin index: manifests whose plaintext source vanished
  sync_queue.json      Pending outbound sync work
  prune_tombstones.json  Deferred prune/tombstone bookkeeping
  update-stage/        Staged (traversal-guarded, signature-verified) self-update bundle
  update-quarantine.json  release_versions that failed health-check here (never re-pulled)
  manifests/
    <content_hash>.json   One per file (opaque name = BLAKE2b of signable data)
  chunks/
    <hash[:2]>/
      <hash>            Encrypted chunk (nonce ‖ ciphertext)
  grants/              (Legacy; migrated to grove.db. Kept for rsync compat.)
  invites/             Pending invite tokens
  image-gen-inits/     Stable-diffusion init images

/run/grove-mountd.sock   Localhost socket of the root USB-mount companion (grove-mountd.py)
~/GroveHome/           User files (plaintext; counts as one replica)

Classes Quick Reference (grove.py)

Class Key fields
FileManifest total_size, chunk_size, chunks, creator_pubkey, signature, encrypted_metadata, metadata_envelope, encrypted_file_key, shared_with, replication_policy, pending_delete_at
ChunkInfo hash (BLAKE2b-256 of plaintext), size, index
ShareGrant manifest_hash, encrypted_file_key, creator_pubkey, recipient_pubkey, signature, granted_at
TombstoneAction Signed retire/delete propagation
DeleteRequest (Disabled) signed delete
FileWatcher Monitors watched dirs, auto-ingest on change
GroveServiceListener mDNS/Zeroconf peer discovery