cleanup: remove spike/ directory

spike/vault_mind/DESIGN.md

@@ -1,170 +0,0 @@
-# Vault-mind spike — design & handoff
-
-This is the full write-up so the spike can be continued in local dev. It records
-(1) where the Chief Engineer's knowledge lives today, (2) what the pi-vault-mind
-method is and how this spike implements it, (3) how CE knowledge maps onto it,
-(4) how to read the eval, (5) **open questions / references needed from you**, and
-(6) the **continue-locally checklist**.
-
----
-
-## 1. Starting point — where CE knowledge lives today (grounded in the code)
-
-The Chief Engineer already has knowledge scattered across **four** surfaces, all
-keyed and retrieved by hand-rolled exact match. The spike's premise is that these
-are really *one* knowledge graph wearing four coats.
-
-| Surface | Code | Shape | How it's retrieved today |
-|---|---|---|---|
-| **Lessons** (seed/earned/ingested/sim) | `ledger.py`, `models.py:LessonEntry` | append-only `data/lessons.jsonl` rows: `material, geometry_type, env_temp, env_humidity, outcome, lesson, source, timestamp` | `LedgerManager.retrieve`: exact `material` AND `geometry_type`, then nearest by **normalized** `[temp,humidity]` Euclidean, top-k. **No vector DB** (deliberate — `docs/plan/00-MASTER.md`, `02-ARCHITECTURE.md`). |
-| **References** (hard params) | `ingest/distill.py:ReferenceFact` | `data/references.jsonl`: `material, param, value, source` | Injected into the prompt as a "Material Reference" block (`chief_engineer.advise(references=...)`). Not env-keyed. |
-| **Policy cells** (learned offsets) | `learn/policy.py` | `data/policy.json`: keyed `material/geometry/temp-bucket/hum-bucket` → offsets + trials/successes | Exact cell-key lookup; injected as `policy_note`. **Generalizes** by bucketing (the "not just a lookup" layer). |
-| **Sim observations** | `learn/loop.py`, `sim/outcome.py` | written back as `source="sim"` LessonEntry rows | same as lessons |
-
-**The locked retrieval thesis** (do not casually break it): exact `material` +
-`geometry_type`, normalized `[temp,humidity]` Euclidean, top 2–3; an empty result
-is a *valid, strong* outcome (model reasons from material properties). The spike
-must **respect** this as the baseline, not silently replace it.
-
-The base the ledger was modeled on (`pi-qmd-ledger`) is itself a configurable
-multi-ledger store — see `../pi-qmd-ledger.config.json`: named ledgers, per-ledger
-`schema`, `dedupField`, regex `injectors`, and a `qmd` binary with `defaultLimit`.
-pi-vault-mind sits on top of that family, adding the vector/FTS/graph index.
-
----
-
-## 2. The pi-vault-mind method (as implemented here)
-
-pi-vault-mind = **LanceDB store with three retrieval channels over the same rows**:
-
-1. **Vector** — embed each knowledge atom's text; ANN search for semantic nearest.
-2. **FTS** — LanceDB native full-text index over the same text; keyword recall.
-3. **Graph** — typed edges between atoms; walk them to pull connected context.
-
-This spike fuses (1)+(2) with **Reciprocal Rank Fusion** (reranker-free, transparent)
-and then does a one-hop **graph expansion** over the top hits. Every returned hit
-carries **provenance** — which channel(s) surfaced it (`vector|fts|graph`).
-
-> **The Obsidian half is intentionally absent.** pi-vault-mind (the product) also
-> watches an Obsidian vault of markdown notes, parses `[[wikilinks]]`/frontmatter
-> into the graph, and re-indexes on file changes. None of that is here: the CE has
-> no human-authored markdown vault, and the ask was explicitly to skip it. The
-> graph edges here are **synthesized from structured fields**, not parsed from notes.
-
-### Why RRF + manual hybrid instead of LanceDB's native hybrid
-LanceDB's built-in `query_type="hybrid"` wants a registered embedding function and
-a reranker to materialize (verified in-sandbox: it *accepts* the call but needs
-that plumbing). For a spike, manual vector+FTS+RRF is more transparent and has zero
-hidden network/model dependencies. **Reconcile to whatever pi-vault-mind actually
-does** once the repo is in hand (it may use native hybrid + a specific reranker).
-
----
-
-## 3. The mapping — CE knowledge → vault nodes + edges
-
-Two tables (`schema.py`):
-
-**`nodes`** — one row per knowledge atom. `text` is BOTH embedded and FTS-indexed;
-structured fields (`material, geometry_type, env_temp, env_humidity, outcome,
-source`) stay queryable for hard prefilters. Kinds:
-
-- `lesson` — one per `LessonEntry` (covers seed/earned/ingested/sim uniformly).
-- `reference` — one per `ReferenceFact`.
-- `cell` — **synthesized** policy-cell node per `material/geometry/temp-bucket/
-  hum-bucket` (mirrors `learn.policy.cell_key`); the aggregation hub the learned
-  policy keys on.
-- `concept` — one per material and per geometry; a graph hub for "all PETG knowledge".
-
-**`edges`** — typed, directed:
-
-- `informed_by`: `cell → lesson` (which lessons shaped a policy bucket).
-- `same_cell`: `lesson ↔ lesson` (siblings in one env bucket → **transfer neighbors**;
-  this is what lets a humid-PETG-bridge lesson reach the next humid-PETG-bridge job).
-- `about`: `lesson|reference → concept` (material/geometry hubs).
-
-This is the spike's claim: the policy bucketing that already exists in
-`learn/policy.py` *is* a graph relation; pi-vault-mind just makes it first-class and
-traversable alongside semantic recall, instead of locked behind an exact cell-key.
-
----
-
-## 4. How to read the eval (`eval.py`)
-
-It is **diagnostic, not a leaderboard**. We have no human relevance labels for
-"which past lesson should this job have surfaced", so claiming a win would be
-dishonest. Two lenses:
-
-- **LENS 1 — agreement.** With `hard_filter=True` (same exact material+geometry
-  constraint as the ledger), does the vault recover the ledger's top-k? In-sandbox:
-  **7/7 (100%)**. Meaning: constrained, the method reproduces today's behavior — so
-  adopting it is not a regression risk; it's a superset.
-- **LENS 2 — reach.** With the soft path, on probes where `ledger.retrieve` returns
-  **`[]`** (no exact precedent), what does the vault surface? In-sandbox: 5 probes
-  pulled same-material cross-geometry / same-geometry cross-material neighbors via
-  vector+FTS+graph. **These candidate rows are the entire reason to consider the
-  added complexity** — and with the offline lexical embedder they only prove the
-  wiring, not semantic quality.
-
-**The honest bottom line of the spike:** the method cleanly subsumes the current
-retrieval and adds a transfer-recall channel. Whether that channel is *good* is
-unproven until a real embedder + a bigger, labelled-ish corpus is in play. That's a
-local-dev call, and it's now a measurable one.
-
----
-
-## 5. Open questions / references needed from you
-
-These are the things that would turn this independent reading into a faithful mirror
-of *your* method. Drop them in local dev:
-
-1. **The `pi-vault-mind` repo itself** (top priority). Specifically:
-   - Table layout: one `nodes` table with a `kind` column (as here) vs. table-per-kind?
-   - Exact field names / schema, and how the vector + FTS indexes are declared.
-   - **Hybrid mechanism**: native LanceDB `hybrid` + which reranker, or a custom
-     fusion? (We used RRF — cheap to change.)
-   - **Graph model**: separate `edges` table (as here) vs. edges embedded on nodes;
-     what relation types it defines; how many hops it traverses on a query.
-   - The **embedder** it standardizes on (model + dim) so vectors are comparable.
-   - How it ingests from `pi-qmd-ledger` ledgers (the `qmd` binary? a direct read?).
-2. **`pi-qmd-ledger` repo** (helpful): the real `append_ledger`/injector/dedup API
-   and schema config, so the ingest side matches rather than infers from the one
-   `pi-qmd-ledger.config.json` we can see.
-3. **A bigger / real lesson corpus** (or the MCP-hackathon Modal ingestion output
-   referenced in `TODO.md`): 12 seeds is enough to prove plumbing, not to judge
-   recall quality.
-4. **Any relevance signal** — even a handful of "for job X, lesson Y was the useful
-   precedent" pairs would let LENS 2 become a real precision/recall number.
-5. **Decision input:** is the goal to *replace* `ledger.retrieve`, to *augment* it
-   (vault as an optional second retriever the LLM also sees), or purely research?
-   The current docs lock the exact-match path as the thesis — augment is the
-   lower-risk graduation path.
-
----
-
-## 6. Continue locally — checklist
-
-1. `uv pip install -r spike/vault_mind/requirements-spike.txt`
-2. **Swap the embedder.** Implement the `Embedder` protocol in `embed.py` with a
-   real model (uncomment the `OllamaEmbedder` sketch + `ollama pull nomic-embed-text`,
-   or `sentence-transformers`). Re-run `uv run python -m spike.vault_mind.eval` and compare
-   LENS 2 candidates against the lexical baseline — semantic neighbors should differ.
-3. **Reconcile to the real method.** With `pi-vault-mind` in hand, adjust `schema.py`
-   + `store.py` to match its table layout, hybrid mechanism, and graph model. Keep
-   the `Hit`/provenance shape — the cockpit will want "why did this surface".
-4. **Scale the corpus.** Point `eval.py` at the real/ingested lessons; watch whether
-   reach stays plausible as N grows (does it surface noise?).
-5. **Decide & graduate (or delete).** If reach earns its keep: graduate as an
-   *augmenting* retriever behind a flag, feeding extra candidates into
-   `chief_engineer.advise(retrieved=...)` alongside — not instead of — the locked
-   exact-match precedent, so the thesis stays intact. If not: `rm -rf spike/vault_mind/`.
-
----
-
-## Appendix — provenance & isolation invariants (don't regress)
-
-- The demo path never imports `spike/`. Keep it that way (grep before graduating).
-- `lancedb` stays out of the Space `requirements.txt` until/unless step 5 graduates it.
-- The offline `HashingEmbedder` must remain the zero-dependency default so the spike
-  runs in CI / the web sandbox without network.
-- `schema.env_bucket` is a deliberate copy of `learn.policy.env_bucket` to keep the
-  spike import-isolated. If the policy buckets change, mirror them here.

spike/vault_mind/README.md

@@ -1,83 +0,0 @@
-# Spike: the pi-vault-mind storage method for the Chief Engineer
-
-**Status:** runnable skeleton + design, validated offline in the web sandbox.
-**Scope:** spike only — isolated, removable, NOT wired into the judged demo.
-**Owner decision pending:** whether any of this graduates into the real ledger.
-
----
-
-## What this is
-
-A spike of the **pi-vault-mind storage method — LanceDB with vector + full-text
-search + a graph layer** — applied to the Chief Engineer's knowledge, with the
-**Obsidian vault-watching side deliberately left out** (per the ask: "spike test
-the pi vault method without worrying about obsidian aspects").
-
-It exists to answer one question with behavior instead of opinion: *what does the
-Chief Engineer's retrieval gain (and cost) if we revisit the locked
-"no embeddings / no vector DB" decision* (`core/ledger.py`, `docs/plan/00-MASTER.md`)
-*and store lessons the pi-vault-mind way instead?*
-
-It is a **contrast object** for the shipped `ledger.retrieve` (exact match on
-`material` AND `geometry_type`, ranked by normalized `[temp, humidity]` Euclidean
-distance, top-k). Nothing here replaces that path yet.
-
-## Why it's safe to have in the repo
-
-- Nothing in the demo path imports `spike/`. Grep-clean.
-- `lancedb` is a **spike-only** dependency (`requirements-spike.txt`), kept out of
-  the Space `requirements.txt`.
-- To remove the entire experiment: `rm -rf spike/vault_mind/`. Nothing else refers to it.
-
-## Run it
-
-```bash
-cd chief-engineer
-uv pip install -r spike/vault_mind/requirements-spike.txt
-uv run python -m spike.vault_mind.eval
-```
-
-No network, no model download, no GPU. The vector channel uses an offline
-deterministic **lexical** embedder (`embed.py`) so the *pipeline* runs anywhere;
-swapping in a real embedder is one class (see `embed.py` + `requirements-spike.txt`).
-
-## What the run showed (web sandbox, 12 seed lessons)
-
-```
-vault built: {'lesson': 12, 'cell': 11, 'concept': 6, 'edges': 38}
-
-LENS 1 — agreement (hard_filter, apples-to-apples):
-  vault recovered a ledger top-3 precedent in 7/7 probes (100%).
-LENS 2 — reach (soft path finds precedent where exact match is EMPTY):
-  5 probes with no exact-key precedent surfaced same-material / same-geometry
-  neighbors (e.g. PETG/adhesion -> PETG/stringing via graph; PETG/bridge ->
-  PETG/overhang sag). Candidates — judge relevance by eye.
-```
-
-**Read this honestly:**
-- LENS 1 (100%) shows the method **subsumes** the current behavior when you
-  constrain it to the same exact filter. Good — it's not a regression risk.
-- LENS 2 shows the *only* thing worth the added complexity: surfacing **transfer
-  precedent** that exact match structurally cannot reach (different material or
-  geometry, or the `retrieve() -> []` "no precedent" case). With the offline
-  **lexical** embedder these are token-overlap neighbors, not true semantic ones —
-  so this proves the **plumbing**, not the payoff. The payoff needs a real
-  embedder, which is the first local-dev step (below).
-
-## Files
-
-| File | Role |
-|------|------|
-| `embed.py` | `Embedder` protocol + offline `HashingEmbedder` default; the single seam to swap in a real model. |
-| `schema.py` | How CE knowledge maps to vault `nodes` + `edges` (kinds, relations, cell/concept ids). |
-| `store.py` | `VaultStore`: ingest JSONL → nodes+edges, hybrid (vector⊕FTS via RRF), graph expand, provenance. |
-| `eval.py` | Head-to-head vs `ledger.retrieve` (agreement + reach lenses). Diagnostic, not a scored win. |
-| `requirements-spike.txt` | `lancedb` + real-embedder options. Spike-only. |
-| `DESIGN.md` | Full write-up: mapping rationale, the method, eval interpretation, **open questions / references needed**, and the **local-dev continuation checklist**. |
-
-## Next (local dev)
-
-See **`DESIGN.md` → "Continue locally"**. Short version: (1) drop in a real
-embedder, (2) provide the actual **pi-vault-mind** repo so the schema/query are
-reconciled to the real method rather than this independent reading, (3) re-run the
-eval on a bigger corpus and decide if reach justifies the dependency.

spike/vault_mind/__init__.py

@@ -1,9 +0,0 @@
-"""vault_mind spike — an ISOLATED, REMOVABLE experiment.
-
-Spikes the pi-vault-mind storage method (LanceDB: vector + FTS + graph) over the
-Chief Engineer's knowledge, minus the Obsidian vault-watching side. Nothing in
-the judged demo path imports this package; `lancedb` is a spike-only optional
-dependency (see requirements-spike.txt), NOT in the Space requirements.txt.
-
-To remove the spike entirely: delete this directory. Nothing else references it.
-"""

spike/vault_mind/embed.py

@@ -1,68 +0,0 @@
-"""Embedding layer for the vault spike — pluggable, offline by default.
-
-The pi-vault-mind method is "vector + FTS + graph". The vector half needs an
-embedder. For a self-contained spike that runs with NO network and NO model
-download, the default here is a deterministic *lexical* embedder: bag-of-hashed
--tokens, L2-normalized. It is reproducible and dependency-free, which is exactly
-what a plumbing spike wants.
-
-HONESTY: this default carries NO real semantics — its "vector" similarity is
-essentially lexical overlap, so on its own it does not beat the existing
-keyword/FTS path. It exists to prove the *pipeline* (ingest → vector + FTS →
-fusion → graph expand → provenance) runs end to end. Swap in a real embedder
-(sentence-transformers, or an Ollama `embeddings` call against the same gemma
-stack) when moving to local dev to get genuine semantic recall. The Embedder
-protocol below is the single seam to do that — nothing else changes.
-"""
-
-from __future__ import annotations
-
-import hashlib
-import math
-import re
-from typing import Protocol, Sequence
-
-_TOKEN = re.compile(r"[a-z0-9]+")
-
-
-class Embedder(Protocol):
-    """The one seam. Implement this to swap the offline default for a real model."""
-
-    dim: int
-
-    def encode(self, texts: Sequence[str]) -> list[list[float]]: ...
-
-
-class HashingEmbedder:
-    """Deterministic, offline, lexical. Placeholder vector channel for the spike."""
-
-    def __init__(self, dim: int = 256) -> None:
-        self.dim = dim
-
-    def _one(self, text: str) -> list[float]:
-        vec = [0.0] * self.dim
-        for tok in _TOKEN.findall(text.lower()):
-            h = int(hashlib.blake2b(tok.encode(), digest_size=8).hexdigest(), 16)
-            idx = h % self.dim
-            sign = 1.0 if (h >> 1) & 1 else -1.0
-            vec[idx] += sign
-        norm = math.sqrt(sum(v * v for v in vec)) or 1.0
-        return [v / norm for v in vec]
-
-    def encode(self, texts: Sequence[str]) -> list[list[float]]:
-        return [self._one(t) for t in texts]
-
-
-# --- a sketch of the local-dev swap (left commented; needs network/model) ----
-# class OllamaEmbedder:
-#     """Real semantic vectors via the same Ollama stack the cockpit already uses.
-#     `ollama pull nomic-embed-text` (or mxbai-embed-large), then point at it."""
-#     def __init__(self, model: str = "nomic-embed-text", dim: int = 768) -> None:
-#         import ollama  # noqa
-#         self.model, self.dim, self._c = model, dim, ollama.Client()
-#     def encode(self, texts):
-#         return [self._c.embeddings(model=self.model, prompt=t)["embedding"] for t in texts]
-
-
-def default_embedder() -> Embedder:
-    return HashingEmbedder()

spike/vault_mind/eval.py

@@ -1,107 +0,0 @@
-"""Head-to-head: vault hybrid+graph retrieval vs. the locked `ledger.retrieve`.
-
-This is DIAGNOSTIC, not a scored win. We have no human relevance labels, so we
-do not claim one method is "better" — we surface WHAT CHANGES when you move from
-exact match + env-Euclidean to vector+FTS+graph, so the local-dev decision is
-informed by behavior, not vibes.
-
-Two lenses:
-  1. AGREEMENT  — with hard_filter on (apples-to-apples), does the vault recover
-     the same exact-match precedent the ledger returns? (Sanity: the method
-     subsumes the current behavior when constrained.)
-  2. REACH      — with the soft path, what cross-key precedent does the vault
-     surface that exact match structurally CANNOT (different material/geometry,
-     or the cases where ledger.retrieve returns [])? These are the candidates
-     the thesis is betting on — inspect them by eye for plausibility.
-
-Run:  uv run python -m spike.vault_mind.eval     (from chief-engineer/)
-"""
-
-from __future__ import annotations
-
-import sys
-import tempfile
-from pathlib import Path
-
-CE_ROOT = Path(__file__).resolve().parents[2]   # chief-engineer/
-if str(CE_ROOT) not in sys.path:
-    sys.path.insert(0, str(CE_ROOT))
-sys.path.insert(0, str(Path(__file__).resolve().parent))  # spike pkg local imports
-
-from core.ledger import LedgerManager           # noqa: E402  (the locked baseline)
-from store import VaultStore               # noqa: E402
-
-DATA = CE_ROOT / "data"
-LESSONS = DATA / "lessons.jsonl"
-SEEDS = DATA / "seed_lessons.jsonl"
-REFS = DATA / "references.jsonl"
-
-
-def _ledger_ids(lm: LedgerManager, material, geometry, temp, hum, k):
-    return [f"lesson::{e.job_id}" for e, _ in lm.retrieve(material, geometry, temp, hum, k=k)]
-
-
-def main() -> None:
-    sources = [p for p in (LESSONS, SEEDS) if p.exists() and p.stat().st_size > 0] or [SEEDS]
-    vault = VaultStore(tempfile.mkdtemp(prefix="vaultspike_"))
-    counts = vault.build(sources, REFS if REFS.exists() else None)
-    print("vault built:", counts)
-    print(f"baseline ledger source(s): {[s.name for s in sources]}\n")
-
-    # Point the baseline LedgerManager at the SAME data the vault saw, so LENS 1
-    # is apples-to-apples (default data/lessons.jsonl is empty until a run seeds it).
-    combined = Path(tempfile.mkdtemp(prefix="ledger_")) / "lessons.jsonl"
-    combined.write_text("\n".join(p.read_text().strip() for p in sources) + "\n")
-    lm = LedgerManager(path=combined)
-
-    # query set = every seed lesson's (material, geometry, env) as a probe
-    import json
-    probes = []
-    for s in sources:
-        for line in s.read_text().splitlines():
-            if line.strip():
-                d = json.loads(line)
-                probes.append((d["job_id"], d["material"], d["geometry_type"],
-                               d["env_temp"], d["env_humidity"]))
-
-    agree_hits = agree_tot = 0
-    reach_examples = []
-    for job_id, mat, geo, t, h in probes:
-        led = set(_ledger_ids(lm, mat, geo, t, h, k=3))
-        led.discard(f"lesson::{job_id}")  # leave-one-out
-
-        hard = {hit.id for hit in vault.query(material=mat, geometry_type=geo, env_temp=t,
-                                              env_humidity=h, k=3, hard_filter=True, expand=False)}
-        hard.discard(f"lesson::{job_id}")
-        if led:
-            agree_tot += 1
-            if led & hard:
-                agree_hits += 1
-
-        soft = vault.query(material=mat, geometry_type=geo, env_temp=t, env_humidity=h,
-                           k=4, hard_filter=False, expand=True)
-        cross = [hit for hit in soft
-                 if hit.id != f"lesson::{job_id}"
-                 and (hit.meta.get("material") != mat or hit.meta.get("geometry_type") != geo)]
-        if cross and not led:   # exactly the gap the thesis targets: no exact precedent
-            reach_examples.append((job_id, mat, geo, t, h, cross[:2]))
-
-    print("== LENS 1: agreement (hard_filter, apples-to-apples) ==")
-    rate = (agree_hits / agree_tot * 100) if agree_tot else 0.0
-    print(f"  vault recovered a ledger top-3 precedent in {agree_hits}/{agree_tot} probes "
-          f"that had any exact precedent ({rate:.0f}%).\n")
-
-    print("== LENS 2: reach (soft path finds precedent where exact match is EMPTY) ==")
-    if not reach_examples:
-        print("  (none in this dataset — expected when every probe has an exact-key sibling)")
-    for job_id, mat, geo, t, h, cross in reach_examples[:6]:
-        print(f"  {job_id}: {mat}/{geo} @ {t:.0f}C/{h:.0f}% had NO exact precedent; vault surfaced:")
-        for c in cross:
-            print(f"      - [{','.join(c.channels)}] {c.meta.get('material')}/"
-                  f"{c.meta.get('geometry_type')}: {c.text[:90]}...")
-    print("\nReminder: LENS 2 rows are CANDIDATES — judge relevance by eye. With the offline")
-    print("lexical embedder they reflect token overlap; a real embedder is needed for semantics.")
-
-
-if __name__ == "__main__":
-    main()

spike/vault_mind/requirements-spike.txt

@@ -1,14 +0,0 @@
-# Spike-only dependencies. NOT part of the Space/demo requirements.txt.
-# Install into your local dev venv to run the vault_mind spike:
-#   pip install -r spike/vault_mind/requirements-spike.txt
-#
-# lancedb 0.33.0 is verified working in the web sandbox (vector + native FTS +
-# hybrid all import and run). Pin loosely; bump when you move to local dev.
-lancedb==0.33.0
-
-# The vector channel uses an offline deterministic LEXICAL embedder by default
-# (embed.py:HashingEmbedder) so the spike runs with no network/model download.
-# For REAL semantic recall, add one of these locally and wire it via the
-# Embedder protocol (embed.py) — see the commented OllamaEmbedder sketch:
-#   ollama            # + `ollama pull nomic-embed-text`  (reuses the cockpit stack)
-#   sentence-transformers   # e.g. all-MiniLM-L6-v2, fully offline after first pull

spike/vault_mind/schema.py

@@ -1,62 +0,0 @@
-"""Vault schema — how the Chief Engineer's scattered knowledge maps onto the
-pi-vault-mind node+edge model.
-
-Two tables (this is the spike's reading of "vector + FTS + graph"):
-
-  nodes  — every knowledge atom as one row: a `text` field that is BOTH the
-           vector-embedded content and the FTS-indexed content, plus structured
-           metadata for hard filters, plus the embedding `vector`.
-  edges  — typed, directed relationships between node ids. The graph layer.
-
-Node kinds folded in from the existing CE surfaces:
-  lesson    — a LessonEntry row (seed | earned | ingested | sim) from lessons.jsonl
-  reference — a ReferenceFact (hard material parameter) from references.jsonl
-  cell      — a synthesized policy-cell node (material/geometry/temp-bucket/hum-bucket),
-              the aggregation point the learned policy keys on
-  concept   — a material or geometry concept node (graph hub for "all PETG knowledge")
-
-Edge relations:
-  informed_by  — cell  -> lesson   (this policy cell was shaped by this lesson)
-  same_cell    — lesson<-> lesson  (siblings in one policy bucket; transfer neighbors)
-  about        — lesson -> concept  / reference -> concept
-
-NOTE (reconcile locally): these names/kinds are the spike's independent reading.
-The real pi-vault-mind table layout, field names, and whether it uses one table
-with a `kind` column vs. a table-per-kind should be confirmed against the repo
-and this file adjusted to match. See OPEN-QUESTIONS in DESIGN.md.
-"""
-
-from __future__ import annotations
-
-NODES_TABLE = "nodes"
-EDGES_TABLE = "edges"
-
-# structured metadata carried on every node (None where not applicable)
-NODE_FIELDS = (
-    "id", "kind", "text", "vector",
-    "material", "geometry_type", "env_temp", "env_humidity",
-    "outcome", "source", "timestamp",
-)
-
-EDGE_FIELDS = ("src", "dst", "rel", "weight")
-
-REL_INFORMED_BY = "informed_by"
-REL_SAME_CELL = "same_cell"
-REL_ABOUT = "about"
-
-
-def env_bucket(temp: float, humidity: float) -> tuple[str, str]:
-    """Mirror of learn.policy.env_bucket — duplicated so the spike stays
-    import-isolated from the demo package (removable without touching it)."""
-    tb = "cool" if temp < 20 else ("warm" if temp > 26 else "mid")
-    hb = "dry" if humidity < 35 else ("humid" if humidity > 55 else "mid")
-    return tb, hb
-
-
-def cell_id(material: str, geometry: str, temp: float, humidity: float) -> str:
-    tb, hb = env_bucket(temp, humidity)
-    return f"cell::{material}/{geometry}/{tb}/{hb}"
-
-
-def concept_id(kind: str, value: str) -> str:
-    return f"concept::{kind}::{value}"

spike/vault_mind/store.py

@@ -1,241 +0,0 @@
-"""VaultStore — the spike's implementation of the pi-vault-mind storage method
-(LanceDB: vector + FTS + graph), applied to the Chief Engineer's knowledge,
-with the Obsidian vault-watching side deliberately left out.
-
-What it demonstrates end to end, offline:
-  1. INGEST   lessons.jsonl (+ optional references.jsonl) into a `nodes` table,
-              synthesizing policy-cell and concept nodes and the `edges` between
-              them (the graph).
-  2. HYBRID   retrieval = vector search ⊕ full-text search, fused with Reciprocal
-              Rank Fusion (transparent, reranker-free), with optional hard
-              metadata prefilters.
-  3. GRAPH    expansion = walk edges from the hybrid hits to pull in the
-              connected policy cell and its sibling lessons (cross-env transfer),
-              and concept hubs.
-  4. PROVENANCE  every returned hit says WHY it surfaced (vector | fts | graph).
-
-This is the contrast object for the locked exact-match `ledger.retrieve`. See
-eval.py for the head-to-head, and DESIGN.md for what it would buy / cost.
-"""
-
-from __future__ import annotations
-
-import json
-from collections import defaultdict
-from dataclasses import dataclass, field
-from pathlib import Path
-from typing import Iterable
-
-import lancedb
-
-from embed import Embedder, default_embedder
-from schema import (
-    EDGES_TABLE, NODES_TABLE, REL_ABOUT, REL_INFORMED_BY, REL_SAME_CELL,
-    cell_id, concept_id, env_bucket,
-)
-
-_RRF_K = 60  # standard Reciprocal Rank Fusion constant
-
-
-@dataclass
-class Hit:
-    id: str
-    kind: str
-    text: str
-    score: float
-    channels: list[str] = field(default_factory=list)  # provenance: vector|fts|graph
-    meta: dict = field(default_factory=dict)
-
-
-def _node_row(emb_text: str, vector: list[float], *, id: str, kind: str,
-              material=None, geometry_type=None, env_temp=None, env_humidity=None,
-              outcome=None, source=None, timestamp=None) -> dict:
-    return {
-        "id": id, "kind": kind, "text": emb_text, "vector": vector,
-        "material": material or "", "geometry_type": geometry_type or "",
-        "env_temp": float(env_temp) if env_temp is not None else 0.0,
-        "env_humidity": float(env_humidity) if env_humidity is not None else 0.0,
-        "outcome": outcome or "", "source": source or "", "timestamp": timestamp or "",
-    }
-
-
-class VaultStore:
-    def __init__(self, db_path: str | Path, embedder: Embedder | None = None) -> None:
-        self.db = lancedb.connect(str(db_path))
-        self.embedder = embedder or default_embedder()
-        self.dim = self.embedder.dim
-
-    # --- ingest ------------------------------------------------------------
-    @staticmethod
-    def _read_jsonl(path: Path) -> list[dict]:
-        if not path or not Path(path).exists():
-            return []
-        out = []
-        for line in Path(path).read_text(encoding="utf-8").splitlines():
-            line = line.strip()
-            if line:
-                try:
-                    out.append(json.loads(line))
-                except Exception:
-                    continue
-        return out
-
-    def build(self, lessons_paths: Iterable[str | Path],
-              references_path: str | Path | None = None) -> dict[str, int]:
-        """(Re)build the vault from JSONL. Returns counts per node kind."""
-        lessons: list[dict] = []
-        for p in lessons_paths:
-            lessons.extend(self._read_jsonl(Path(p)))
-        refs = self._read_jsonl(Path(references_path)) if references_path else []
-
-        nodes: list[dict] = []
-        edges: list[dict] = []
-        concepts: set[tuple[str, str]] = set()
-        cell_members: dict[str, list[str]] = defaultdict(list)
-
-        # lessons -> lesson nodes (+ cell membership, + concept edges)
-        for e in lessons:
-            lid = f"lesson::{e.get('job_id','?')}"
-            mat, geo = e.get("material", ""), e.get("geometry_type", "")
-            text = (f"{mat} {geo} at {e.get('env_temp','?')}C/"
-                    f"{e.get('env_humidity','?')}% RH -> {e.get('outcome','')}. "
-                    f"{e.get('lesson','')}")
-            nodes.append(_node_row(text, [], id=lid, kind="lesson", material=mat,
-                                   geometry_type=geo, env_temp=e.get("env_temp"),
-                                   env_humidity=e.get("env_humidity"),
-                                   outcome=e.get("outcome"), source=e.get("source", "earned"),
-                                   timestamp=e.get("timestamp", "")))
-            cid = cell_id(mat, geo, float(e.get("env_temp", 0)), float(e.get("env_humidity", 0)))
-            cell_members[cid].append(lid)
-            if mat:
-                concepts.add(("material", mat)); edges.append({"src": lid, "dst": concept_id("material", mat), "rel": REL_ABOUT, "weight": 1.0})
-            if geo:
-                concepts.add(("geometry", geo)); edges.append({"src": lid, "dst": concept_id("geometry", geo), "rel": REL_ABOUT, "weight": 1.0})
-
-        # references -> reference nodes (+ concept edge to material)
-        for i, r in enumerate(refs):
-            rid = f"reference::{i}"
-            mat = r.get("material", "")
-            text = (f"{mat} reference: {r.get('param','')} = {r.get('value','')} "
-                    f"(source: {r.get('source','')})")
-            nodes.append(_node_row(text, [], id=rid, kind="reference", material=mat,
-                                   source=r.get("source", "reference")))
-            if mat:
-                concepts.add(("material", mat)); edges.append({"src": rid, "dst": concept_id("material", mat), "rel": REL_ABOUT, "weight": 1.0})
-
-        # synthesized policy-cell nodes + informed_by / same_cell edges (the graph)
-        for cid, members in cell_members.items():
-            _, body = cid.split("::", 1)
-            nodes.append(_node_row(f"policy cell {body}", [], id=cid, kind="cell"))
-            for m in members:
-                edges.append({"src": cid, "dst": m, "rel": REL_INFORMED_BY, "weight": 1.0})
-            for a in members:  # sibling transfer edges within a bucket
-                for b in members:
-                    if a != b:
-                        edges.append({"src": a, "dst": b, "rel": REL_SAME_CELL, "weight": 0.5})
-
-        # concept hub nodes
-        for kind, val in concepts:
-            nodes.append(_node_row(f"{kind}: {val}", [], id=concept_id(kind, val), kind="concept",
-                                   material=val if kind == "material" else None,
-                                   geometry_type=val if kind == "geometry" else None))
-
-        # embed all node texts in one batch, then write
-        vectors = self.embedder.encode([n["text"] for n in nodes])
-        for n, v in zip(nodes, vectors):
-            n["vector"] = v
-
-        self.db.drop_table(NODES_TABLE, ignore_missing=True)
-        self.db.drop_table(EDGES_TABLE, ignore_missing=True)
-        ntbl = self.db.create_table(NODES_TABLE, data=nodes)
-        ntbl.create_fts_index("text", use_tantivy=False, replace=True)
-        self.db.create_table(EDGES_TABLE, data=edges or [{"src": "", "dst": "", "rel": "", "weight": 0.0}])
-
-        counts: dict[str, int] = defaultdict(int)
-        for n in nodes:
-            counts[n["kind"]] += 1
-        counts["edges"] = len(edges)
-        return dict(counts)
-
-    # --- retrieval ---------------------------------------------------------
-    def _nodes(self):
-        return self.db.open_table(NODES_TABLE)
-
-    def _edges(self):
-        return self.db.open_table(EDGES_TABLE)
-
-    @staticmethod
-    def _all_rows(tbl) -> list[dict]:
-        return tbl.to_arrow().to_pylist()
-
-    def _hybrid(self, query_text: str, limit: int, where: str | None) -> dict[str, Hit]:
-        """Vector ⊕ FTS, fused with RRF. Returns {id: Hit} with channel provenance."""
-        tbl = self._nodes()
-        qv = self.embedder.encode([query_text])[0]
-
-        vsearch = tbl.search(qv, vector_column_name="vector")
-        if where:
-            vsearch = vsearch.where(where, prefilter=True)
-        vec_ids = [r["id"] for r in vsearch.limit(limit * 4).to_list()]
-
-        try:
-            fsearch = tbl.search(query_text, query_type="fts")
-            if where:
-                fsearch = fsearch.where(where, prefilter=True)
-            fts_ids = [r["id"] for r in fsearch.limit(limit * 4).to_list()]
-        except Exception:
-            fts_ids = []
-
-        rows = {r["id"]: r for r in self._all_rows(tbl)}
-        fused: dict[str, Hit] = {}
-        for channel, ids in (("vector", vec_ids), ("fts", fts_ids)):
-            for rank, nid in enumerate(ids):
-                r = rows.get(nid)
-                if not r:
-                    continue
-                h = fused.get(nid) or Hit(id=nid, kind=r["kind"], text=r["text"], score=0.0,
-                                          meta={k: r[k] for k in ("material", "geometry_type",
-                                                "env_temp", "env_humidity", "outcome", "source")})
-                h.score += 1.0 / (_RRF_K + rank + 1)
-                if channel not in h.channels:
-                    h.channels.append(channel)
-                fused[nid] = h
-        return fused
-
-    def query(self, *, material: str = "", geometry_type: str = "", env_temp: float = 0.0,
-              env_humidity: float = 0.0, text: str | None = None, k: int = 3,
-              expand: bool = True, hard_filter: bool = False) -> list[Hit]:
-        """Hybrid + graph retrieval. `hard_filter=True` restricts to exact
-        material+geometry (apples-to-apples with ledger.retrieve); default is the
-        soft path that lets cross-key precedent surface — the spike's whole point."""
-        q = text or (f"{material} {geometry_type} at {env_temp:.0f}C/{env_humidity:.0f}% RH")
-        where = None
-        if hard_filter and material and geometry_type:
-            where = f"material = '{material}' AND geometry_type = '{geometry_type}'"
-
-        fused = self._hybrid(q, k, where)
-        # keep lessons/references as primary hits; cells/concepts arrive via graph
-        primary = {i: h for i, h in fused.items() if h.kind in ("lesson", "reference")}
-
-        if expand and primary:
-            etbl = self._edges()
-            top_ids = sorted(primary, key=lambda i: primary[i].score, reverse=True)[:k]
-            edge_rows = self._all_rows(etbl)
-            ntbl_rows = {r["id"]: r for r in self._all_rows(self._nodes())}
-            for nid in top_ids:
-                tb, hb = env_bucket(primary[nid].meta.get("env_temp", 0),
-                                    primary[nid].meta.get("env_humidity", 0))
-                for er in edge_rows:
-                    if er["src"] == nid and er["rel"] == REL_SAME_CELL:
-                        sib = ntbl_rows.get(er["dst"])
-                        if sib and er["dst"] not in primary:
-                            h = Hit(id=er["dst"], kind=sib["kind"], text=sib["text"],
-                                    score=primary[nid].score * 0.4, channels=["graph"],
-                                    meta={kk: sib[kk] for kk in ("material", "geometry_type",
-                                          "env_temp", "env_humidity", "outcome", "source")})
-                            primary[er["dst"]] = h
-                        elif sib and "graph" not in primary[er["dst"]].channels:
-                            primary[er["dst"]].channels.append("graph")
-                            primary[er["dst"]].score += primary[nid].score * 0.2
-
-        return sorted(primary.values(), key=lambda h: h.score, reverse=True)[:k]