teleo-infrastructure/docs/kb-rebuild-and-recompile.md

# Rebuilding Leo's Knowledge Database

## Outcome

Leo should improve by compiling durable source material and reviewed changes
into Postgres. Repeatedly changing prompts or retraining the chat behavior is
not the knowledge system.

There are two different rebuilds:

1. **Exact recovery** restores the current canonical database from a verified
   snapshot. This is working now.
2. **Semantic recompilation** starts from the retained source corpus and the
   reviewed change ledger, then reproduces the canonical rows. This is partly
   recoverable but is not yet complete.

## Exact Recovery: Working

Run:

```bash
.venv/bin/python ops/run_local_canonical_postgres_rebuild.py \
  --dump /private/path/teleo-canonical.dump \
  --source-manifest /private/path/source-manifest.jsonl \
  --output /tmp/teleo-canonical-rebuild-receipt.json
```

The 2026-07-13 live canary restored a fresh, network-isolated Postgres in 3.81
seconds and then removed it. The restored target matched the source across all
39 manifest tables and all 52,164 rows, with no schema, data, constraint, role,
or performance mismatch. Key rows included:

- 1,837 claims;
- 4,145 sources;
- 4,670 claim-evidence links;
- 4,916 claim edges;
- 17 reasoning tools;
- 26 proposals.

This is the fastest disaster-recovery path. It does not require Leo to
re-extract or relearn the corpus.

## Source Recompilation: Current Evidence

Read-only VPS inspection found two retained June import runs. Both point to the
Forgejo-era workspace at `/opt/teleo-eval/workspaces/main` and retained
inventory JSONL files under `/opt/teleo-eval/kb-import/`.

The database still retains:

- `kb_stage.import_runs` for the two inventories;
- staged claims, sources, claim-source links, and claim edges;
- `kb_stage.canonical_mappings` from legacy keys to canonical UUIDs;
- mappings for 1,807 of 1,837 canonical claims;
- mappings for all 4,145 canonical sources.

For the 1,807 mapped claims, current canonical type, text, status, confidence,
tags, and creator match the retained staged rows exactly. Creation timestamps
also follow a recoverable rule: use the legacy timestamp when present and the
mapping timestamp for the eight rows that had no legacy timestamp.

The simple retained-row joins currently reproduce:

- 4,254 of 4,670 canonical evidence links;
- 4,878 of 4,916 canonical edge rows can be accounted for by a staged relation;
  historical duplicate multiplicity still needs an explicit replay rule.

Re-run this aggregate, read-only coverage audit against any restored local
clone:

```bash
.venv/bin/python ops/audit_kb_rebuild_coverage.py \
  --container <restored-local-postgres-container> \
  --database teleo
```

The auditor emits no claim bodies or source excerpts and separates snapshot
recovery from source-derived recompilation readiness.

The remaining gaps are concrete rather than mysterious:

- 30 claims were created after or outside the mapped import;
- 284 mapped source rows do not have a matching retained `staged_sources` row
  and need the original source-synthesis rule or an explicit genesis record;
- 416 evidence links need source-synthesis or later-change provenance;
- 38 edge rows need later-change receipts or explicit replay records;
- old applied proposal rows do not describe every historical canonical write.

This proves that most of the initial database came from the retained file-KB
import path. It does not yet prove a clean blank-database recompile.

## Target Compiler

The durable rebuild model is:

```text
immutable source corpus + file hashes
-> deterministic inventory and classification
-> staged claims, sources, evidence links, and edges
-> stable canonical ID mapping
-> review decisions
-> append-only accepted apply payloads and receipts
-> canonical Postgres
-> render/sync/restart
-> answer benchmark
```

Use the current verified snapshot as **genesis epoch 1**. Preserve its dump,
manifest, source commit, inventory files, mappings, and aggregate rebuild
receipt. Every accepted change after that epoch must carry a replayable strict
apply payload and row-level postflight receipt. This prevents the historical
gap from growing while the old import rules are reconstructed.

The source compiler now turns one raw artifact, its strict UTF-8 extraction,
and a reviewed extraction manifest into a deterministic, hash-bound
`pending_review` proposal bundle:

```bash
.venv/bin/python scripts/compile_kb_source_packet.py \
  --artifact fixtures/working-leo/document-ingestion-v1.json \
  --text fixtures/working-leo/document-ingestion-v1.json \
  --manifest fixtures/working-leo/source-compiler-manifest-v1.json \
  --output /tmp/working-leo-source-packet-v1.json
```

The compiler verifies artifact and extraction hashes, stable source identity,
current schema taxonomies, unique logical keys, and exact claim/evidence quotes.
It reuses the existing proposal normalizer and staging preflight, but it has no
database connection and executes neither staging nor apply. Its output is the
review packet, not canonical knowledge.

The existing full-data clone canary separately proves that a reviewed packet
can create source, claim, evidence, and edge rows and affect later reasoning.
The next capability is to expose this compiler through Leo's bounded source
intake command, stage its output, and join accepted packets into a corpus runner
and replay ledger.

## Definition Of Working

Semantic recompilation is complete only when all of these pass:

1. A command creates a blank database from the retained source corpus plus the
   reviewed ledger.
2. Schema, constraints, roles, table counts, row hashes, and key query results
   match the canonical manifest.
3. Every canonical row traces to a genesis import record or a reviewed apply
   receipt.
4. A new document can be hash-captured, extracted into grounded candidates,
   deduplicated, staged, reviewed, applied in a disposable clone, and read back.
5. After render/sync and restart, Leo answers the related broad question using
   the new rows and cites the source chain.

Until then, exact snapshot recovery is production recovery; source
recompilation is an active build capability, not a finished claim.