# KB Restore / Replication Runbook This runbook is for proving Living IP KB/database redundancy on GCP. ## Two Different Database Surfaces Do not call the SQLite shadow restore a canonical Leo database copy. Canonical Leo knowledge is currently: - host: VPS `77.42.65.182`; - container: `teleo-pg`; - engine/database: PostgreSQL 16, database `teleo`; - canonical schemas: `public` and `kb_stage`; - high-signal rows: claims, sources, claim evidence, claim edges, reasoning tools, and review-gated proposals. The older pipeline/evaluation database is a separate surface: - pipeline runtime DB: `/opt/teleo-eval/pipeline/pipeline.db` - engine: SQLite WAL - related Leo files: - `/opt/teleo-eval/workspaces/main/agents/leo` - `/opt/teleo-eval/workspaces/research-leo/agents/leo` - `/opt/teleo-eval/agent-state` `ops/run_gcp_cloudsql_restore_drill.sh` remains a legacy SQLite-to-Postgres shadow-schema drill. It reconstructs `teleo_restore` inside `teleo_kb`; it does not preserve the canonical Postgres schema, constraints, indexes, functions, roles, or row hashes. The last authenticated control-plane readback on 2026-07-10 reported this candidate GCP target; refresh it before any mutation: - project: `teleo-501523` - instance: `teleo-pgvector-standby` - database: `teleo_kb` - region: `europe-west6` - network: `teleo-staging-net` - private IP: `10.61.0.3` - admin password secret: `gcp-teleo-pgvector-standby-postgres-password` Do not call this redundancy complete until source data has been restored or replicated and queried from GCP. ## Canonical Postgres Snapshot And Parity Capture a custom-format dump and a full JSONL manifest from the same exported, read-only PostgreSQL snapshot: ```bash python3 ops/capture_vps_canonical_postgres_snapshot.py \ --execute \ --ssh-target root@77.42.65.182 \ --ssh-key ~/.ssh/livingip_hetzner_20260604_ed25519 \ --run-id canonical- \ --authorization-ref codex-delegation:THREAD_ID \ --output-dir ``` Replace `THREAD_ID` with the exact retained authorization reference for the capture. The v2 receipt binds that reference, exported snapshot ID, TXID snapshot, WAL LSN, source system identifier, dump hash, manifest hash, reviewed manifest-SQL hash, and source-context hash. A dump and manifest without this passing receipt are not eligible for GCP restore. The capture fails closed unless the structured source-service states before and after the snapshot are identical and healthy: `ActiveState=active`, `SubState=running`, a positive `MainPID`, and a nonnegative `NRestarts`. It retains a private custom dump, dump SHA-256, object TOC, catalog/data manifest, and both service-state objects. It never restarts Leo or writes to the source database. Prove that this exact snapshot can rebuild a blank Postgres target before using it for GCP: ```bash .venv/bin/python ops/run_local_canonical_postgres_rebuild.py \ --dump /teleo-canonical.dump \ --source-manifest /source-manifest.jsonl \ --output /tmp/teleo-canonical-rebuild-receipt.json ``` The local runner starts a uniquely named `postgres:16-alpine` container with network mode `none` and tmpfs-only database storage. It waits for an actual `psql` connection to the named database, restores with `pg_restore --no-owner --no-privileges --exit-on-error`, compares the full parity manifest, then removes the container and proves it is absent. A passing local receipt is the exact-recovery preflight; it is not semantic recompilation from raw source documents. Choose one bounded suffix and use it consistently on the GCP replay VM. The restore helper retains its mode-0700 evidence directory at the fixed private root: ```bash RUN_SUFFIX=20260715t010000z TARGET_DB="teleo_clone_${RUN_SUFFIX}" RESTORE_RUN_ID="gcp-restore-${RUN_SUFFIX}" PRIVATE_RUN_DIR="/var/lib/teleo-gcp-restore-runs/${RESTORE_RUN_ID}" ``` Use a generated target database such as `teleo_clone_`. Never import a drill into `teleo`, `teleo_kb`, or `teleo_canonical`. Database isolation does not isolate cluster-global roles or extensions, so verify those separately and do not run the Docker-only gate bootstrap against the shared Cloud SQL instance. The canonical restore command must include the v2 capture receipt. It also runs a live Cloud SQL control-plane preflight before resolving the database secret or creating the clone, and fails closed unless the exact instance is runnable, PostgreSQL 16, private-networked, bound to the expected RFC1918 host, and has public IP disabled. A GCE metadata preflight independently binds the operator runtime to `teleo-prod-1` in `europe-west6-a` on `teleo-staging-net`; a hard-coded compute label is not accepted: ```bash sudo python3 ops/restore_gcp_generated_postgres_snapshot.py restore \ --execute \ --target-db "$TARGET_DB" \ --run-id "$RESTORE_RUN_ID" \ --dump /teleo-canonical.dump \ --expected-dump-sha256 \ --source-manifest /source-manifest.jsonl \ --source-receipt /receipt.json \ --expected-source-receipt-sha256 \ --expected-capture-authorization-ref codex-delegation:THREAD_ID \ --project teleo-501523 \ --cloudsql-instance teleo-pgvector-standby \ --expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \ --expected-compute-instance teleo-prod-1 \ --expected-compute-zone europe-west6-a \ --expected-source-ssh-target root@77.42.65.182 \ --expected-source-container teleo-pg \ --expected-source-database teleo \ --expected-source-service leoclean-gateway.service ``` The restore writes both `target-manifest.jsonl` and `gcp-private-connectivity.json` as mode-0600 files in its private run directory. The latter is composed from live GCE metadata, Cloud SQL control-plane checks, and the private TLS database identity; do not hand-compose the connectivity receipt. The restore also records identical healthy before/after state for `leoclean-gcp-prod-parallel.service`, including its PID and restart count. Compare the captured target manifest and generated connectivity receipt to the authorized source: ```bash sudo python3 ops/verify_postgres_parity_manifest.py \ --source /source-manifest.jsonl \ --target "$PRIVATE_RUN_DIR/target-manifest.jsonl" \ --scope gcp_staging \ --connectivity-proof "$PRIVATE_RUN_DIR/gcp-private-connectivity.json" \ --output "$PRIVATE_RUN_DIR/gcp-parity.json" ``` The verifier checks all table row counts and collation-independent row hashes, plus schemas, columns/defaults, constraints, indexes, sequences, views, functions, triggers, enum/domain types, policies, required extensions, password-free application-role attributes, and bounded query timings. In GCP scope it also requires a receipt proving a staging compute source, a private server address, TLS, and public-IP-disabled instance metadata. After the parity verifier passes, run the no-send m3taversal blind reasoning replay from staging compute against that generated database, attest the replay host from live GCE metadata, and only then delete the generated database. This canonical path streams the custom dump through `pg_restore`; it does not create a GCS import object. Run the bounded ID-free reasoning canary with the generated parity receipt. The script re-executes itself in the Hermes virtualenv when that runtime is present: ```bash sudo python3 scripts/run_gcp_generated_db_blind_claim_canary.py \ --execute \ --target-db "$TARGET_DB" \ --cloudsql-tool hermes-agent/leoclean-bin/cloudsql_memory_tool.py \ --manifest-sql ops/postgres_parity_manifest.sql \ --parity-receipt "$PRIVATE_RUN_DIR/gcp-parity.json" \ --output "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \ --run-id "gcp-blind-claim-${RUN_SUFFIX}" ``` Before cleanup, bind that reasoning receipt to live metadata from the same GCE replay host: ```bash sudo python3 ops/attest_gcp_reasoning_compute.py \ --reasoning-receipt "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \ --restore-run-id "$RESTORE_RUN_ID" \ --target-db "$TARGET_DB" \ --project teleo-501523 \ --expected-compute-instance teleo-prod-1 \ --expected-compute-zone europe-west6-a \ --expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \ --output "$PRIVATE_RUN_DIR/reasoning-compute-attestation.json" ``` After the compute attestation passes, run the receipt-bound cleanup command retained by the restore. Then capture the VPS source again under a distinct postflight authorization reference. Run this capture from the same trusted source-access lane used for the baseline; it need not run on the GCP replay VM. The postflight capture must complete after cleanup, so the final verdict can state whether the VPS changed after the restored snapshot boundary. Compare the two source captures: ```bash python3 ops/capture_vps_canonical_postgres_snapshot.py \ --execute \ --ssh-target root@77.42.65.182 \ --ssh-key ~/.ssh/livingip_hetzner_20260604_ed25519 \ --run-id canonical-${RUN_SUFFIX}-postflight \ --authorization-ref codex-delegation:THREAD_ID:postflight \ --output-dir ``` ```bash sudo python3 ops/verify_vps_canonical_snapshot_delta.py \ --baseline-receipt /receipt.json \ --baseline-manifest /source-manifest.jsonl \ --baseline-authorization-ref codex-delegation:THREAD_ID \ --current-receipt /receipt.json \ --current-manifest /source-manifest.jsonl \ --current-authorization-ref codex-delegation:THREAD_ID:postflight \ --output "$PRIVATE_RUN_DIR/source-delta-receipt.json" ``` Finally, verify that all eight receipts belong to one bound lifecycle. This verifier rejects local-scope parity, stale or mismatched source hashes, a postflight source capture that predates reasoning or cleanup, hand-authored reasoning booleans without real retrieval/tool receipts, an unattested replay host, compute or Cloud SQL topology drift, and any cleanup receipt that leaves the clone present. The postflight capture may be at most 900 seconds old at verification; the GCP lifecycle age and restore-to-postflight span must each be at most 3600 seconds: ```bash sudo python3 ops/verify_gcp_canonical_lifecycle.py \ --source-receipt /receipt.json \ --current-source-receipt /receipt.json \ --source-delta-receipt "$PRIVATE_RUN_DIR/source-delta-receipt.json" \ --restore-receipt "$PRIVATE_RUN_DIR/restore-receipt.json" \ --parity-receipt "$PRIVATE_RUN_DIR/gcp-parity.json" \ --reasoning-receipt "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \ --reasoning-compute-receipt "$PRIVATE_RUN_DIR/reasoning-compute-attestation.json" \ --cleanup-receipt "$PRIVATE_RUN_DIR/cleanup-receipt.json" \ --max-postflight-age-seconds 900 \ --max-lifecycle-age-seconds 3600 \ --output "$PRIVATE_RUN_DIR/lifecycle-verification.json" ``` Retain the private receipt run directory and every useful lifecycle input, supporting receipt, and verifier output. Remove only transient upload/bundle directories, temporary Hermes profiles, helper clients, and child processes after their absence is verified. There is no canonical-path GCS object to delete. Do not remove the evidence directory until its useful receipts are integrated or explicitly rejected. ## Legacy SQLite Source Backup Canary Create a consistent source backup without stopping the VPS service: ```bash ops/backup_vps_sqlite_kb.sh ``` The script: - uses SQLite `.backup` against `/opt/teleo-eval/pipeline/pipeline.db`; - compresses and hashes the backup on the VPS; - archives Leo/KB files while excluding `secrets` and logs; - copies both artifacts locally; - verifies SHA-256 matches; - runs `PRAGMA integrity_check` on a local restored SQLite copy; - records proof under `outputs/gcp-infra-hardening-20260707/proofs/`. This proves source exportability and local restore integrity. It does not prove GCP DB redundancy until a GCP restore/import/query canary also passes. ## Legacy SQLite-To-Postgres Restore Canary Before importing into Cloud SQL, prove that the current SQLite backup can be converted and restored into PostgreSQL without row loss: ```bash SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-.db.gz \ ops/run_sqlite_postgres_restore_canary.sh ``` The canary: - generates a PostgreSQL import script with `ops/sqlite_to_postgres_dump.py`; - recreates a shadow schema in a disposable `postgres:16-alpine` container; - imports all user tables from the SQLite backup; - compares source and target row counts for every table; - writes a proof JSON under `outputs/gcp-infra-hardening-20260707/proofs/`; - removes only its temporary canary container. This is a local restore/parity proof, not GCP redundancy by itself. It is the preflight that should pass before the same generated import is applied through the approved Cloud SQL connector/VPC path. To pass this local preflight into a clean GitHub readiness run without uploading private backup paths, generated SQL, or target-count CSVs, create a redacted capsule from the proof: ```bash python3 ops/redact_sqlite_postgres_restore_canary.py \ --proof outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-.json \ --output outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-capsule-.json ``` The capsule keeps only non-secret evidence: proof hash, backup hash, source and target table/row counts, conversion notes/stats, and the redacted-field list. It does not prove that Cloud SQL imported the data; it only proves the local SQLite-to-Postgres parity preflight. To include the capsule in GitHub readiness: ```bash CAPSULE_B64="$(base64 < outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-capsule-.json | tr -d '\n')" gh workflow run gcp-readiness.yml \ --repo living-ip/teleo-infrastructure \ --ref main \ -f restore_canary_capsule_b64="${CAPSULE_B64}" ``` ## Legacy SQLite Cloud SQL Restore Drill Runner Prepare the exact GCS import and Cloud SQL import operation without mutating GCP: ```bash SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-.db.gz \ ops/run_gcp_cloudsql_restore_drill.sh ``` Execute it only from an authenticated operator environment that can write the versioned backup bucket and administer the standby Cloud SQL instance: ```bash EXECUTE=1 \ SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-.db.gz \ ops/run_gcp_cloudsql_restore_drill.sh ``` The runner: - regenerates the explicit PostgreSQL import script; - targets the shadow schema `teleo_restore` inside `teleo_kb`; - uploads the import script to `gs://teleo-501523-prod-backups/kb-dumps/cloudsql-restore-drills/...` when `EXECUTE=1`; - starts and waits for `gcloud sql import sql`; - writes `target-counts.sql` for the required trusted VPC/Cloud SQL connector query readback. The import operation alone is still not the final proof. The final proof needs `target-counts.sql` run against `teleo-pgvector-standby` and compared to the source counts in the drill proof. After the import operation is `DONE`, run the generated count query from a trusted VPC runtime or Cloud SQL connector path and retain CSV output: ```bash psql "$TELEO_CLOUDSQL_DATABASE_URL" \ --csv \ -f outputs/gcp-infra-hardening-20260707/private-cloudsql-restore-drills/gcp-cloudsql-restore-drill-/target-counts.sql \ > outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-target-counts-.csv ``` Then compare the Cloud SQL readback to the source proof: ```bash python3 ops/verify_gcp_cloudsql_restore_readback.py \ --drill-proof outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-restore-drill-.json \ --target-counts-csv outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-target-counts-.csv \ --output outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-restore-readback-verification-.json ``` Only a `status = pass` verifier output is enough for row-count parity. It still does not prove application cutover or continuous replication. ## Required Proof A successful restore or replication canary must retain: - source dataset identity: - source host or dump artifact; - dump timestamp or replication slot timestamp; - source schema/database name. - transfer proof: - for the canonical path, the v2 capture receipt, dump SHA-256, and exact private staging path used by streaming `pg_restore`; - for a legacy import or replication path, a versioned dump-object generation or logical replication subscription details; - row/table counts before import where available. - target proof: - `teleo-pgvector-standby` control-plane readback; - generated `teleo_clone_*` identity for canonical parity, or `teleo_kb` only for the explicitly legacy SQLite drill; - extension readback for `vector` if the restored schema needs pgvector; - representative query readback for core KB tables. - failure boundary: - exact missing secret, source access, schema incompatibility, extension issue, or import error. ## One-Shot SQLite Export / GCP Restore Path Use this while the canonical DB remains SQLite on the VPS and we need a GCP restore drill. 1. Run `ops/backup_vps_sqlite_kb.sh`. 2. Upload the resulting SQLite backup and Leo/KB tarball to a versioned GCS bucket such as `gs://teleo-501523-prod-backups/kb-dumps/`. 3. Run the local SQLite-to-Postgres restore canary above and retain its proof. 4. Run `ops/run_gcp_cloudsql_restore_drill.sh` in dry-run mode to generate the GCS import plan. 5. Run `EXECUTE=1 ops/run_gcp_cloudsql_restore_drill.sh` from an authenticated operator environment to upload and import the generated SQL. Do not run blind string rewrites against the SQLite dump. 6. Install required extensions on Cloud SQL: ```sql create extension if not exists vector; ``` 7. From a trusted VPC runtime or Cloud SQL connector path, run readbacks: ```sql select current_database(); select extname, extversion from pg_extension where extname = 'vector'; select schemaname, tablename from pg_tables where schemaname not in ('pg_catalog', 'information_schema') order by 1, 2 limit 50; ``` 8. Retain the SQLite backup hash, GCS object generation, import/conversion operation, query output, and row-count sample. 9. Run `ops/verify_gcp_cloudsql_restore_readback.py` and retain a passing parity proof. ## Logical Replication Path Use this only if the canonical source becomes Postgres or a Postgres mirror exists. SQLite cannot be logically replicated into Cloud SQL Postgres without an intermediate conversion/sync layer. Required source privileges: - replication-capable source user; - publication over the intended schemas/tables; - network path from GCP to source, or source-to-GCP path through an approved proxy/tunnel. Required target steps: ```sql create extension if not exists vector; create subscription connection '' publication ; ``` Retain only redacted connection metadata. Do not commit or paste credentials. ## Current Blocker As of 2026-07-15, a newest authorized canonical Postgres capture and isolated local restore can pass, but the required T3 live private GCP lifecycle cannot be started from the current operator route: - direct VM SSH and the private TCP route timed out; - local `gcloud` for `billy@livingip.xyz` requires reauthentication; no password was entered or inspected; - the intended operator WIF/IAP authority is unavailable because provider `teleo-iap-operator` is absent, disabled, or deleted; - the working artifact-builder WIF identity is intentionally limited and must not be expanded or mistaken for Cloud SQL/Compute operator access. The authorized recovery is a local `gcloud auth login billy@livingip.xyz --force` without sharing credentials, or administrator convergence of the reviewed operator WIF/IAP bootstrap. Until a route passes live readback, keep the claim at T2/source-local proof; do not promote staging or expose Cloud SQL publicly. Once access is restored, run: newest authorized `teleo` snapshot -> disposable private `teleo_clone_*` restore -> generated connectivity proof -> `gcp-parity.json` -> no-send m3taversal blind reasoning -> live GCE reasoning attestation -> receipt-bound clone cleanup -> postflight VPS capture and delta -> eight-receipt lifecycle verdict within the 900/3600-second freshness bounds.