teleo-infrastructure/docs/kb-rebuild-and-recompile.md

# Rebuilding Leo's Knowledge Database

## Outcome

Leo should improve by compiling durable source material and reviewed changes
into Postgres. Repeatedly changing prompts or retraining the chat behavior is
not the knowledge system.

There are two different rebuilds:

1. **Exact recovery** restores the current canonical database from a verified
   snapshot. This is working now.
2. **Semantic recompilation** starts from the retained source corpus and the
   reviewed change ledger, then reproduces the canonical rows. This is partly
   recoverable but is not yet complete.

## Exact Recovery: Working

Run:

```bash
.venv/bin/python ops/run_local_canonical_postgres_rebuild.py \
  --dump /private/path/teleo-canonical.dump \
  --source-manifest /private/path/source-manifest.jsonl \
  --output /tmp/teleo-canonical-rebuild-receipt.json
```

The retained 2026-07-14 post-V3 canary restored a fresh, network-isolated
Postgres and then removed it. The restored target matched the source across all
39 manifest tables and all 52,167 rows, with no schema, data, constraint, role,
or performance mismatch. The same source snapshot was subsequently restored to
a disposable private-TLS GCP Cloud SQL clone with exact parity, a bounded
no-send reasoning turn, and verified cleanup. Key rows included:

- 1,837 claims;
- 4,145 sources;
- 4,670 claim-evidence links;
- 4,916 claim edges;
- 17 reasoning tools;
- 29 proposals.

This snapshot includes the completed V3 source canary. Disposable GCP restore
parity is proven at that retained point. Persistent GCP `teleo_canonical`
remains the older staging copy, so ongoing parity, promotion, and production
cutover remain unproven.

This is the fastest disaster-recovery path. It does not require Leo to
re-extract or relearn the corpus.

## Source Recompilation: Current Evidence

Read-only VPS inspection found two retained June import runs. Both point to the
Forgejo-era workspace at `/opt/teleo-eval/workspaces/main` and retained
inventory JSONL files under `/opt/teleo-eval/kb-import/`.

The database still retains:

- `kb_stage.import_runs` for the two inventories;
- staged claims, sources, claim-source links, and claim edges;
- `kb_stage.canonical_mappings` from legacy keys to canonical UUIDs;
- mappings for 1,807 of 1,837 canonical claims;
- mappings for all 4,145 canonical sources.

For the 1,807 mapped claims, current canonical type, text, status, confidence,
tags, and creator match the retained staged rows exactly. Creation timestamps
also follow a recoverable rule: use the legacy timestamp when present and the
mapping timestamp for the eight rows that had no legacy timestamp.

The simple retained-row joins currently reproduce:

- 4,254 of 4,670 canonical evidence links;
- 4,878 of 4,916 canonical edge rows can be accounted for by a staged relation;
  historical duplicate multiplicity still needs an explicit replay rule.

Re-run this aggregate, read-only coverage audit against any restored local
clone:

```bash
.venv/bin/python ops/audit_kb_rebuild_coverage.py \
  --container <restored-local-postgres-container> \
  --database teleo
```

The auditor emits no claim bodies or source excerpts and separates snapshot
recovery from source-derived recompilation readiness.

The remaining gaps are concrete rather than mysterious:

- 30 claims were created after or outside the mapped import;
- 284 mapped source rows do not have a matching retained `staged_sources` row
  and need the original source-synthesis rule or an explicit genesis record;
- 416 evidence links need source-synthesis or later-change provenance;
- 38 edge rows need later-change receipts or explicit replay records;
- old applied proposal rows do not describe every historical canonical write.

This proves that most of the initial database came from the retained file-KB
import path. It does not yet prove a clean blank-database recompile.

## Genesis-Backed Source Lifecycle: Working T2 Canary

`scripts/run_leo_integrated_learning_lifecycle.py` can now start from one
retained real genesis dump rather than its legacy synthetic fixture schema. The
genesis mode is all-or-nothing: the dump, parity manifest, repository-pinned
trusted exporter receipt, and a digest-pinned PostgreSQL image are mandatory.

Run:

```bash
python3 scripts/run_leo_integrated_learning_lifecycle.py \
  --source-packet /private/path/source-packet \
  --genesis-dump /private/path/genesis.dump \
  --genesis-manifest /private/path/genesis-manifest.jsonl \
  --genesis-exporter-receipt /private/path/exporter-receipt.json \
  --image docker.io/library/postgres:16-alpine@sha256:57c72fd2a128e416c7fcc499958864df5301e940bca0a56f58fddf30ffc07777 \
  --review-action approve \
  --review-note "Approved exact retained payload for disposable T2 verification only." \
  --output /private/path/leo-genesis-lifecycle-receipt.json
```

Before the first Docker command, the runner copies the dump, manifest, exporter
receipt, source packet, controller, subprocess engines, and their transitive
repository imports into one mode `0500` private staging directory with mode
`0400` files. It validates the custom-dump signature; the exact
repository-pinned exporter-receipt hash, source identity, capture authorization,
and provenance binding; exporter-bound dump and manifest bytes; manifest
contract; image digest; executing Git commit; complete engine dependency hash
set; and a fresh deterministic compile against the retained
`proposal-packet.json`. Every enumerated engine file must be tracked at that
commit, its working-tree bytes must equal the commit blob, and its staged hash
must equal the same commit-bound hash. The compiler performs no model calls. Any
untrusted or incomplete exporter contract, moving image tag, engine drift,
staged-byte drift, invalid manifest, retained-packet drift, or output/input
collision fails before a container can start.

The child tools consume staged paths, and both runs bind their consumed file
set, engine set, and commit back to the same preflight manifest. The already
running Python controller imports repository modules before it creates staging;
the proof therefore establishes commit-matched imported dependencies and
byte-identical staged copies, not a claim that the controller process itself was
re-executed from staging. Successful public receipts retain logical names,
hashes, counts, and cleanup evidence without host-specific absolute paths.

Each of two independent runs then:

1. starts a uniquely labeled, networkless PostgreSQL container with tmpfs data;
2. creates only manifest-declared global roles, restores the supplied custom
   dump directly, replays captured database ACLs, and proves the full parity
   manifest before installing the current guarded-apply prerequisites;
3. skips the synthetic schema bootstrap and performs no canonical preseed;
4. proves the strict proposal, immutable approval, and every payload-owned
   claim/source/evidence/edge/tool target are absent before staging;
5. stages `pending_review`, explicitly approves as the independent review role,
   and drives the real apply worker through an exact-candidate report-only tick,
   an enabled pre-commit failure with persisted retry state, a successful
   password-authenticated retry with a mode `0600` replay receipt, and a
   post-apply no-op tick;
6. recompiles the retained packet identically and fingerprints only fields
   controlled by its immutable apply payload; and
7. removes the container and private temporary root, then proves its exact
   instance and shared two-run label have no remaining containers.

The two-run comparison excludes generated evidence/edge IDs, timestamps,
container identity, and the PostgreSQL system identifier. It requires the
source hashes, proposal and payload identities, phase states, restored key
counts, row counts, recompile hash, and payload-controlled row projections to
match exactly.

Older retained parity manifests may predate explicit
`application_roles[].settings` and `database_settings` arrays. Their raw bytes
and trusted exporter binding remain mandatory. For role creation and current
parity SQL, the runner narrowly projects missing arrays from the captured
`default_transaction_read_only` field (or an empty list), without changing the
retained bytes. Such a run may prove data/schema parity and captured role
attributes, but its receipt reports full role/global parity as `not_proven`.
Only a source manifest that actually captured both settings arrays can earn the
full role/global parity claim.

The focused Docker regression constructs a real custom-format genesis with
nondefault role and per-database settings, restores it twice, executes the real
worker lifecycle, compares post-apply fingerprints, and verifies container and
staging cleanup:

```bash
pytest -q \
  tests/test_run_leo_integrated_learning_lifecycle.py \
  -k test_real_docker_genesis_restore_worker_parity_and_repeatability
```

The detailed receipt can contain proposal payloads and canonical row excerpts.
It is written atomically as mode `0600` and must remain private. Standard output
contains only the artifact/status, receipt basename and SHA-256, aggregate
problem/check counts, and error type. A passing receipt proves T2 disposable
runtime reconstruction only. It does not prove or permit VPS/GCP mutation,
production access, live Telegram behavior, or a blank-schema rebuild of all
historical knowledge.

## Target Compiler

The durable rebuild model is:

```text
immutable source corpus + file hashes
-> deterministic inventory and classification
-> staged claims, sources, evidence links, and edges
-> stable canonical ID mapping
-> review decisions
-> append-only accepted apply payloads and receipts
-> canonical Postgres
-> render/sync/restart
-> answer benchmark
```

Use the current verified snapshot as **genesis epoch 1**. Preserve its dump,
manifest, source commit, inventory files, mappings, and aggregate rebuild
receipt. Every accepted change after that epoch must carry a replayable strict
apply payload and row-level postflight receipt. This prevents the historical
gap from growing while the old import rules are reconstructed.

The guarded apply CLI now enforces the receipt half of that policy. After a
successful apply it reloads the immutable applied proposal, selects the exact
canonical rows described by the strict payload, binds generated row IDs and
timestamps, hashes the exact executed apply SQL, payload, and rows, and
atomically writes a private mode `0600` receipt. It also supports read-only
recovery for a committed apply whose receipt file was lost, provided every
payload-controlled row still matches the immutable reviewed payload:

```bash
python3 scripts/apply_proposal.py <proposal-uuid> --receipt-only
```

The live VPS recovery canary used strict applied proposal
`00957f6c-9883-4015-95a4-6b09367efb0e`. It recovered exactly one canonical edge,
kept all database counts and the proposal payload hash unchanged, left the Leo
gateway on the same PID with zero restarts, and removed the temporary private
receipt. The full receipt is deliberately not committed because it can contain
claim bodies or source excerpts; the sanitized proof is retained as
`docs/reports/leo-working-state-20260709/kb-apply-replay-receipt-current.json`.

Normal applies mark the SQL hash as `exact_executed_sql`. A later
`--receipt-only` recovery marks it as `reconstructed_current_engine`; it never
pretends the current engine hash is historical proof of the originally executed
program.

This closes replay-receipt loss for new strict applies. The receipt alone does
not retain every column of the proposal ledger, so exact reconstruction also
needs the full approved proposal row, immutable approval snapshot, and final
applied proposal row.

New approvals retain the exact post-approval `kb_proposals` row inside
`kb_proposal_approvals`, including `updated_at`. After the worker applies and
writes its replay receipt, export the seven-field private material without any
pre-apply filesystem capture:

```bash
python3 scripts/export_kb_transition_replay_material.py <proposal-uuid> \
  --replay-receipt /private/path/<proposal-uuid>.json \
  --sequence 1 \
  --output /private/path/entry-0001.json
```

The exporter uses the restricted `kb_apply` read function in a read-only
transaction, validates exact approved/approval/applied identities and the
worker receipt, and publishes a mode-`0600` file once. An identical rerun is a
no-op; drift or output aliasing is refused. Already-applied legacy approvals
whose exact approved row was never retained remain explicitly unavailable.

## Genesis Plus Strict Ledger: Working Deterministic Slice

`ops/run_local_genesis_ledger_rebuild.py` now executes the first exact
genesis-plus-ledger slice in one command:

```bash
.venv/bin/python ops/run_local_genesis_ledger_rebuild.py \
  --genesis-dump /private/path/genesis.dump \
  --genesis-manifest /private/path/genesis-manifest.jsonl \
  --ledger /private/path/ledger.json \
  --ledger-sha256 "$LEDGER_SHA256" \
  --output /tmp/genesis-ledger-rebuild-receipt.json
```

The v1 ledger pins the genesis dump and manifest, final parity manifest,
reconstruction/restore/guard/apply/replay/parity engines, and every ordered
private material file. Each material file contains one existing `kb_apply_replay_receipt`, the
exact full proposal row immediately before apply, its immutable
`kb_proposal_approvals` row, and the exact full proposal row after apply. These
files can contain claim text or source excerpts and must remain private.

The ledger shape is:

```json
{
  "artifact": "teleo_genesis_plus_ledger",
  "contract_version": 1,
  "engine": {
    "reconstruction_command_sha256": "<sha256>",
    "base_rebuild_engine_sha256": "<sha256>",
    "apply_engine_sha256": "<sha256>",
    "replay_receipt_engine_sha256": "<sha256>",
    "guard_prerequisites_sha256": "<sha256>",
    "parity_sql_sha256": "<sha256>"
  },
  "genesis": {
    "dump_sha256": "<sha256>",
    "parity_manifest_sha256": "<sha256>"
  },
  "entries": [{
    "sequence": 1,
    "material": "private/0001.json",
    "sha256": "<material-file-sha256>",
    "replay_material_sha256": "<receipt-replay-material-sha256>"
  }],
  "final_parity": {
    "manifest": "final-manifest.jsonl",
    "sha256": "<sha256>"
  }
}
```

Each referenced material object has exact top-level keys
`artifact`, `contract_version`, `sequence`, `approved_proposal`,
`approval_snapshot`, `applied_proposal`, and `replay_receipt`. Proposal objects
must contain every current `kb_stage.kb_proposals` column; partial envelopes
are rejected.

The command verifies every hash before starting Docker, requires a
SHA-256-pinned Postgres image (and defaults to a pinned PostgreSQL 16 Alpine
multi-platform digest), restores it on tmpfs with `--network none`, reapplies
the current guarded prerequisites, and proves genesis parity. Insert-only
entries seed the receipt's exact canonical row IDs and timestamps before the
existing `kb_apply` payload-bound guard executes. For `revise_strategy`, the
runner seeds only the proposal and approval, captures the target agent's
prestate, executes the real guarded transition, validates the generated delta,
and then replaces only those generated rows with the receipt-pinned IDs and
timestamps. Every path checks exact proposal and canonical row readbacks before
verifying the complete final parity manifest. Its public
mode-`0600` receipt contains hashes, IDs, types, counts, parity summaries, and
cleanup proof, but no payloads, rows, SQL, source paths, or command errors.
The legacy `seed_exact` summary is the insert-only aggregate of
`proposal_seed_exact` and `canonical_seed_exact`; it is intentionally false for
successful mutating entries, which instead report
`mutating_delta_validated` and `mutating_poststate_normalized`.
Fresh guard bootstrap rows use a fixed baseline timestamp rather than wall
clock time, so repeated clean restores have identical row hashes.

The exact v1 claim ceiling is intentionally bounded:

- `add_edge`, `attach_evidence`, `approve_claim`, and `revise_strategy` strict
  receipts execute;
- sequence gaps, hash drift, engine drift, duplicate proposals, and legacy or
  freeform payloads fail before container start;
- `revise_strategy` is accepted only when the receipt pins the exact SQL that
  matches the current guarded apply engine. Immediately before each entry, the
  runner captures the target agent's strategy/node IDs, active strategy,
  non-retired nodes, and maximum version. It then validates the generated
  post-minus-pre delta, requires `version = previous maximum + 1`, and replaces
  only the generated strategy/node rows with the exact receipt rows;
- the original transaction timestamp is derived from the fresh strategy
  `created_at` and must equal every fresh node's `created_at` and `updated_at`.
  It must also fall within the immutable, timezone-aware interval
  `reviewed_at <= transaction timestamp <= applied_at`.
  Only node IDs observed as non-retired before apply receive that timestamp;
  already-retired, unrelated-agent, and shared NULL-agent rows stay untouched;
- generated nodes must have no anchors before normalization, preventing a
  delete-and-reinsert step from silently cascading future trigger-created rows;
- full proposal before/after rows are mandatory because the current receipt
  envelope does not retain proposal origin fields or exact `updated_at`;
- this proves only an isolated local reconstruction. It does not touch or prove
  VPS, GCP, Telegram, a live database, or blank-schema source recompilation.

V3 `approve_claim` receipts extend the same exact-ledger boundary to admitted
claim versions, content-addressed source versions, located evidence, typed
edges, and evidence assessments. Ledger order is also the schema-cutover order:
all V2 entries must replay before the first V3 entry, the hash-pinned V3 contract
is installed once at that boundary, guarded prerequisites are rerun, and any V2
entry after cutover fails before replay. Historical V2 proposals already present
in genesis remain visible; the reconstruction does not silently apply, delete,
or reinterpret them.

Receipt-pinned V3 rows are seeded before the idempotent guarded apply, then read
back exactly. PostgreSQL generates the proposal's apply timestamp, so the
disposable reconstruction takes an exclusive proposal-table lock, verifies the
exact V3 decision trigger is enabled, transactionally disables only that trigger,
normalizes the receipt-pinned proposal timestamps as `postgres`, re-enables the
trigger, and verifies it is enabled before commit. Any failure rolls the trigger
state and timestamp update back together. This is a disposable reconstruction
mechanism, not a production write path or a relaxation of the live V3 contract.

The transition contract for `revise_strategy` is final-state deterministic, not
a claim that the v1 receipt independently contains a historical before-image:

```text
exact genesis/pre-entry state
  + exact current/original guarded SQL
  + receipt-pinned fresh strategy and nodes
  -> prior active strategy inactive
  -> exactly the prior non-retired nodes retired at the original transaction time
  -> one receipt-exact active strategy and receipt-exact node set
```

The genesis and final manifests remain mandatory oracles. Any incorrect
prestate, unrelated-row mutation, missing/extra generated row, semantic drift,
version drift, hash drift, or final rowset difference fails the reconstruction.

The source compiler now turns one raw artifact, its strict UTF-8 extraction,
and a reviewed extraction manifest into a deterministic, hash-bound
`pending_review` proposal bundle:

```bash
.venv/bin/python scripts/compile_kb_source_packet.py \
  --artifact fixtures/working-leo/document-ingestion-v1.json \
  --text fixtures/working-leo/document-ingestion-v1.json \
  --manifest fixtures/working-leo/source-compiler-manifest-v1.json \
  --output /tmp/working-leo-source-packet-v1.json
```

The compiler verifies artifact and extraction hashes, stable source identity,
current schema taxonomies, unique logical keys, and exact claim/evidence quotes.
It reuses the existing proposal normalizer and staging preflight, but it has no
database connection and executes neither staging nor apply. Its output is the
review packet, not canonical knowledge.

The VPS also exposes a bounded preparation command for one text-like filesystem
document (or a binary artifact with a separately supplied strict UTF-8
extraction):

```bash
teleo-kb prepare-source \
  --artifact /home/teleo/.hermes/profiles/leoclean/state/kb-source-inbox/source.md \
  --identity document:stable-source-id \
  --source-key stable_source_key \
  --source-type article \
  --title "Stable source title" \
  --locator artifact://stable/source-id \
  --output-dir /home/teleo/.hermes/profiles/leoclean/state/kb-source-preparation/source-id
```

It queries canonical claims before extraction, caps a single document at three
new candidates with confidence at or below `0.75`, has the model select
densely numbered non-empty source fragments, resolves those IDs to exact source substrings, records
duplicate judgments, and validates a v2 manifest through the source compiler.
Unknown line IDs are rejected rather than fuzzily matched. It writes private
files only. The extracted atomic proposition is the proposed claim text; exact
source wording remains separately hash-bound as quote and evidence. A separate
`teleo-kb propose-source` call is required to create a `pending_review` row;
neither command applies canonical rows.

The existing full-data clone canary separately proves that a reviewed packet
can create source, claim, evidence, and edge rows and affect later reasoning.
The remaining reconstruction work is to backfill or explicitly reject legacy
freeform applies and extend beyond genesis recovery to a blank-schema source
compiler. The strict ledger runner does not prove that every historical
canonical row can be rebuilt semantically from retained sources.

## Definition Of Working

Semantic recompilation is complete only when all of these pass:

1. A command creates a blank database from the retained source corpus plus the
   reviewed ledger.
2. Schema, constraints, roles, table counts, row hashes, and key query results
   match the canonical manifest.
3. Every canonical row traces to a genesis import record or a reviewed apply
   receipt.
4. A new document can be hash-captured, extracted into grounded candidates,
   deduplicated, staged, reviewed, applied in a disposable clone, and read back.
5. After render/sync and restart, Leo answers the related broad question using
   the new rows and cites the source chain.

Until then, exact snapshot recovery is production recovery; source
recompilation is an active build capability, not a finished claim.