teleo/teleo-infrastructure
8
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions 1
teleo-infrastructure/docs/gcp-kb-cloudsql-restore-runbook.md

20 KiB
Raw Blame History

KB Restore / Replication Runbook

This runbook is for proving Living IP KB/database redundancy on GCP.

Two Different Database Surfaces

Do not call the SQLite shadow restore a canonical Leo database copy.

Canonical Leo knowledge is currently:

  • host: VPS 77.42.65.182;
  • container: teleo-pg;
  • engine/database: PostgreSQL 16, database teleo;
  • canonical schemas: public and kb_stage;
  • high-signal rows: claims, sources, claim evidence, claim edges, reasoning tools, and review-gated proposals.

The older pipeline/evaluation database is a separate surface:

  • pipeline runtime DB: /opt/teleo-eval/pipeline/pipeline.db
  • engine: SQLite WAL
  • related Leo files:
    • /opt/teleo-eval/workspaces/main/agents/leo
    • /opt/teleo-eval/workspaces/research-leo/agents/leo
    • /opt/teleo-eval/agent-state

ops/run_gcp_cloudsql_restore_drill.sh remains a legacy SQLite-to-Postgres shadow-schema drill. It reconstructs teleo_restore inside teleo_kb; it does not preserve the canonical Postgres schema, constraints, indexes, functions, roles, or row hashes.

The last authenticated control-plane readback on 2026-07-10 reported this candidate GCP target; refresh it before any mutation:

  • project: teleo-501523
  • instance: teleo-pgvector-standby
  • database: teleo_kb
  • region: europe-west6
  • network: teleo-staging-net
  • private IP: 10.61.0.3
  • admin password secret: gcp-teleo-pgvector-standby-postgres-password

Do not call this redundancy complete until source data has been restored or replicated and queried from GCP.

Canonical Postgres Snapshot And Parity

Capture a custom-format dump and a full JSONL manifest from the same exported, read-only PostgreSQL snapshot:

python3 ops/capture_vps_canonical_postgres_snapshot.py \
  --execute \
  --ssh-target root@77.42.65.182 \
  --ssh-key ~/.ssh/livingip_hetzner_20260604_ed25519 \
  --run-id canonical-<timestamp> \
  --authorization-ref codex-delegation:THREAD_ID \
  --output-dir <private-output-dir>

Replace THREAD_ID with the exact retained authorization reference for the capture. The v2 receipt binds that reference, exported snapshot ID, TXID snapshot, WAL LSN, source system identifier, dump hash, manifest hash, reviewed manifest-SQL hash, and source-context hash. A dump and manifest without this passing receipt are not eligible for GCP restore.

The capture fails closed unless the structured source-service states before and after the snapshot are identical and healthy: ActiveState=active, SubState=running, a positive MainPID, and a nonnegative NRestarts. It retains a private custom dump, dump SHA-256, object TOC, catalog/data manifest, and both service-state objects. It never restarts Leo or writes to the source database.

Prove that this exact snapshot can rebuild a blank Postgres target before using it for GCP:

.venv/bin/python ops/run_local_canonical_postgres_rebuild.py \
  --dump <private-output-dir>/teleo-canonical.dump \
  --source-manifest <private-output-dir>/source-manifest.jsonl \
  --output /tmp/teleo-canonical-rebuild-receipt.json

The local runner starts a uniquely named postgres:16-alpine container with network mode none and tmpfs-only database storage. It waits for an actual psql connection to the named database, restores with pg_restore --no-owner --no-privileges --exit-on-error, compares the full parity manifest, then removes the container and proves it is absent. A passing local receipt is the exact-recovery preflight; it is not semantic recompilation from raw source documents.

Choose one bounded suffix and use it consistently on the GCP replay VM. The restore helper retains its mode-0700 evidence directory at the fixed private root:

RUN_SUFFIX=20260715t010000z
TARGET_DB="teleo_clone_${RUN_SUFFIX}"
RESTORE_RUN_ID="gcp-restore-${RUN_SUFFIX}"
PRIVATE_RUN_DIR="/var/lib/teleo-gcp-restore-runs/${RESTORE_RUN_ID}"

Use a generated target database such as teleo_clone_<run_id>. Never import a drill into teleo, teleo_kb, or teleo_canonical. Database isolation does not isolate cluster-global roles or extensions, so verify those separately and do not run the Docker-only gate bootstrap against the shared Cloud SQL instance.

The canonical restore command must include the v2 capture receipt. It also runs a live Cloud SQL control-plane preflight before resolving the database secret or creating the clone, and fails closed unless the exact instance is runnable, PostgreSQL 16, private-networked, bound to the expected RFC1918 host, and has public IP disabled. A GCE metadata preflight independently binds the operator runtime to teleo-prod-1 in europe-west6-a on teleo-staging-net; a hard-coded compute label is not accepted:

sudo python3 ops/restore_gcp_generated_postgres_snapshot.py restore \
  --execute \
  --target-db "$TARGET_DB" \
  --run-id "$RESTORE_RUN_ID" \
  --dump <private-output-dir>/teleo-canonical.dump \
  --expected-dump-sha256 <dump-sha256> \
  --source-manifest <private-output-dir>/source-manifest.jsonl \
  --source-receipt <private-output-dir>/receipt.json \
  --expected-source-receipt-sha256 <source-receipt-sha256> \
  --expected-capture-authorization-ref codex-delegation:THREAD_ID \
  --project teleo-501523 \
  --cloudsql-instance teleo-pgvector-standby \
  --expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \
  --expected-compute-instance teleo-prod-1 \
  --expected-compute-zone europe-west6-a \
  --expected-source-ssh-target root@77.42.65.182 \
  --expected-source-container teleo-pg \
  --expected-source-database teleo \
  --expected-source-service leoclean-gateway.service

The restore writes both target-manifest.jsonl and gcp-private-connectivity.json as mode-0600 files in its private run directory. The latter is composed from live GCE metadata, Cloud SQL control-plane checks, and the private TLS database identity; do not hand-compose the connectivity receipt. The restore also records identical healthy before/after state for leoclean-gcp-prod-parallel.service, including its PID and restart count.

Compare the captured target manifest and generated connectivity receipt to the authorized source:

sudo python3 ops/verify_postgres_parity_manifest.py \
  --source <private-output-dir>/source-manifest.jsonl \
  --target "$PRIVATE_RUN_DIR/target-manifest.jsonl" \
  --scope gcp_staging \
  --connectivity-proof "$PRIVATE_RUN_DIR/gcp-private-connectivity.json" \
  --output "$PRIVATE_RUN_DIR/gcp-parity.json"

The verifier checks all table row counts and collation-independent row hashes, plus schemas, columns/defaults, constraints, indexes, sequences, views, functions, triggers, enum/domain types, policies, required extensions, password-free application-role attributes, and bounded query timings. In GCP scope it also requires a receipt proving a staging compute source, a private server address, TLS, and public-IP-disabled instance metadata.

After the parity verifier passes, run the no-send m3taversal blind reasoning replay from staging compute against that generated database, attest the replay host from live GCE metadata, and only then delete the generated database. This canonical path streams the custom dump through pg_restore; it does not create a GCS import object.

Run the bounded ID-free reasoning canary with the generated parity receipt. The script re-executes itself in the Hermes virtualenv when that runtime is present:

sudo python3 scripts/run_gcp_generated_db_blind_claim_canary.py \
  --execute \
  --target-db "$TARGET_DB" \
  --cloudsql-tool hermes-agent/leoclean-bin/cloudsql_memory_tool.py \
  --manifest-sql ops/postgres_parity_manifest.sql \
  --parity-receipt "$PRIVATE_RUN_DIR/gcp-parity.json" \
  --output "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \
  --run-id "gcp-blind-claim-${RUN_SUFFIX}"

Before cleanup, bind that reasoning receipt to live metadata from the same GCE replay host:

sudo python3 ops/attest_gcp_reasoning_compute.py \
  --reasoning-receipt "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \
  --restore-run-id "$RESTORE_RUN_ID" \
  --target-db "$TARGET_DB" \
  --project teleo-501523 \
  --expected-compute-instance teleo-prod-1 \
  --expected-compute-zone europe-west6-a \
  --expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \
  --output "$PRIVATE_RUN_DIR/reasoning-compute-attestation.json"

After the compute attestation passes, run the receipt-bound cleanup command retained by the restore. Then capture the VPS source again under a distinct postflight authorization reference. Run this capture from the same trusted source-access lane used for the baseline; it need not run on the GCP replay VM. The postflight capture must complete after cleanup, so the final verdict can state whether the VPS changed after the restored snapshot boundary. Compare the two source captures:

python3 ops/capture_vps_canonical_postgres_snapshot.py \
  --execute \
  --ssh-target root@77.42.65.182 \
  --ssh-key ~/.ssh/livingip_hetzner_20260604_ed25519 \
  --run-id canonical-${RUN_SUFFIX}-postflight \
  --authorization-ref codex-delegation:THREAD_ID:postflight \
  --output-dir <private-postflight-dir>

sudo python3 ops/verify_vps_canonical_snapshot_delta.py \
  --baseline-receipt <private-output-dir>/receipt.json \
  --baseline-manifest <private-output-dir>/source-manifest.jsonl \
  --baseline-authorization-ref codex-delegation:THREAD_ID \
  --current-receipt <private-postflight-dir>/receipt.json \
  --current-manifest <private-postflight-dir>/source-manifest.jsonl \
  --current-authorization-ref codex-delegation:THREAD_ID:postflight \
  --output "$PRIVATE_RUN_DIR/source-delta-receipt.json"

Finally, verify that all eight receipts belong to one bound lifecycle. This verifier rejects local-scope parity, stale or mismatched source hashes, a postflight source capture that predates reasoning or cleanup, hand-authored reasoning booleans without real retrieval/tool receipts, an unattested replay host, compute or Cloud SQL topology drift, and any cleanup receipt that leaves the clone present. The postflight capture may be at most 900 seconds old at verification; the GCP lifecycle age and restore-to-postflight span must each be at most 3600 seconds:

sudo python3 ops/verify_gcp_canonical_lifecycle.py \
  --source-receipt <private-output-dir>/receipt.json \
  --current-source-receipt <private-postflight-dir>/receipt.json \
  --source-delta-receipt "$PRIVATE_RUN_DIR/source-delta-receipt.json" \
  --restore-receipt "$PRIVATE_RUN_DIR/restore-receipt.json" \
  --parity-receipt "$PRIVATE_RUN_DIR/gcp-parity.json" \
  --reasoning-receipt "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \
  --reasoning-compute-receipt "$PRIVATE_RUN_DIR/reasoning-compute-attestation.json" \
  --cleanup-receipt "$PRIVATE_RUN_DIR/cleanup-receipt.json" \
  --max-postflight-age-seconds 900 \
  --max-lifecycle-age-seconds 3600 \
  --output "$PRIVATE_RUN_DIR/lifecycle-verification.json"

Retain the private receipt run directory and every useful lifecycle input, supporting receipt, and verifier output. Remove only transient upload/bundle directories, temporary Hermes profiles, helper clients, and child processes after their absence is verified. There is no canonical-path GCS object to delete. Do not remove the evidence directory until its useful receipts are integrated or explicitly rejected.

Legacy SQLite Source Backup Canary

Create a consistent source backup without stopping the VPS service:

ops/backup_vps_sqlite_kb.sh

The script:

  • uses SQLite .backup against /opt/teleo-eval/pipeline/pipeline.db;
  • compresses and hashes the backup on the VPS;
  • archives Leo/KB files while excluding secrets and logs;
  • copies both artifacts locally;
  • verifies SHA-256 matches;
  • runs PRAGMA integrity_check on a local restored SQLite copy;
  • records proof under outputs/gcp-infra-hardening-20260707/proofs/.

This proves source exportability and local restore integrity. It does not prove GCP DB redundancy until a GCP restore/import/query canary also passes.

Legacy SQLite-To-Postgres Restore Canary

Before importing into Cloud SQL, prove that the current SQLite backup can be converted and restored into PostgreSQL without row loss:

SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-<timestamp>.db.gz \
  ops/run_sqlite_postgres_restore_canary.sh

The canary:

  • generates a PostgreSQL import script with ops/sqlite_to_postgres_dump.py;
  • recreates a shadow schema in a disposable postgres:16-alpine container;
  • imports all user tables from the SQLite backup;
  • compares source and target row counts for every table;
  • writes a proof JSON under outputs/gcp-infra-hardening-20260707/proofs/;
  • removes only its temporary canary container.

This is a local restore/parity proof, not GCP redundancy by itself. It is the preflight that should pass before the same generated import is applied through the approved Cloud SQL connector/VPC path.

To pass this local preflight into a clean GitHub readiness run without uploading private backup paths, generated SQL, or target-count CSVs, create a redacted capsule from the proof:

python3 ops/redact_sqlite_postgres_restore_canary.py \
  --proof outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-<timestamp>.json \
  --output outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-capsule-<timestamp>.json

The capsule keeps only non-secret evidence: proof hash, backup hash, source and target table/row counts, conversion notes/stats, and the redacted-field list. It does not prove that Cloud SQL imported the data; it only proves the local SQLite-to-Postgres parity preflight.

To include the capsule in GitHub readiness:

CAPSULE_B64="$(base64 < outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-capsule-<timestamp>.json | tr -d '\n')"
gh workflow run gcp-readiness.yml \
  --repo living-ip/teleo-infrastructure \
  --ref main \
  -f restore_canary_capsule_b64="${CAPSULE_B64}"

Legacy SQLite Cloud SQL Restore Drill Runner

Prepare the exact GCS import and Cloud SQL import operation without mutating GCP:

SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-<timestamp>.db.gz \
  ops/run_gcp_cloudsql_restore_drill.sh

Execute it only from an authenticated operator environment that can write the versioned backup bucket and administer the standby Cloud SQL instance:

EXECUTE=1 \
SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-<timestamp>.db.gz \
  ops/run_gcp_cloudsql_restore_drill.sh

The runner:

  • regenerates the explicit PostgreSQL import script;
  • targets the shadow schema teleo_restore inside teleo_kb;
  • uploads the import script to gs://teleo-501523-prod-backups/kb-dumps/cloudsql-restore-drills/... when EXECUTE=1;
  • starts and waits for gcloud sql import sql;
  • writes target-counts.sql for the required trusted VPC/Cloud SQL connector query readback.

The import operation alone is still not the final proof. The final proof needs target-counts.sql run against teleo-pgvector-standby and compared to the source counts in the drill proof.

After the import operation is DONE, run the generated count query from a trusted VPC runtime or Cloud SQL connector path and retain CSV output:

psql "$TELEO_CLOUDSQL_DATABASE_URL" \
  --csv \
  -f outputs/gcp-infra-hardening-20260707/private-cloudsql-restore-drills/gcp-cloudsql-restore-drill-<timestamp>/target-counts.sql \
  > outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-target-counts-<timestamp>.csv

Then compare the Cloud SQL readback to the source proof:

python3 ops/verify_gcp_cloudsql_restore_readback.py \
  --drill-proof outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-restore-drill-<timestamp>.json \
  --target-counts-csv outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-target-counts-<timestamp>.csv \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-restore-readback-verification-<timestamp>.json

Only a status = pass verifier output is enough for row-count parity. It still does not prove application cutover or continuous replication.

Required Proof

A successful restore or replication canary must retain:

  • source dataset identity:
    • source host or dump artifact;
    • dump timestamp or replication slot timestamp;
    • source schema/database name.
  • transfer proof:
    • for the canonical path, the v2 capture receipt, dump SHA-256, and exact private staging path used by streaming pg_restore;
    • for a legacy import or replication path, a versioned dump-object generation or logical replication subscription details;
    • row/table counts before import where available.
  • target proof:
    • teleo-pgvector-standby control-plane readback;
    • generated teleo_clone_* identity for canonical parity, or teleo_kb only for the explicitly legacy SQLite drill;
    • extension readback for vector if the restored schema needs pgvector;
    • representative query readback for core KB tables.
  • failure boundary:
    • exact missing secret, source access, schema incompatibility, extension issue, or import error.

One-Shot SQLite Export / GCP Restore Path

Use this while the canonical DB remains SQLite on the VPS and we need a GCP restore drill.

  1. Run ops/backup_vps_sqlite_kb.sh.
  2. Upload the resulting SQLite backup and Leo/KB tarball to a versioned GCS bucket such as gs://teleo-501523-prod-backups/kb-dumps/.
  3. Run the local SQLite-to-Postgres restore canary above and retain its proof.
  4. Run ops/run_gcp_cloudsql_restore_drill.sh in dry-run mode to generate the GCS import plan.
  5. Run EXECUTE=1 ops/run_gcp_cloudsql_restore_drill.sh from an authenticated operator environment to upload and import the generated SQL. Do not run blind string rewrites against the SQLite dump.
  6. Install required extensions on Cloud SQL:
create extension if not exists vector;
  1. From a trusted VPC runtime or Cloud SQL connector path, run readbacks:
select current_database();
select extname, extversion from pg_extension where extname = 'vector';
select schemaname, tablename from pg_tables where schemaname not in ('pg_catalog', 'information_schema') order by 1, 2 limit 50;
  1. Retain the SQLite backup hash, GCS object generation, import/conversion operation, query output, and row-count sample.
  2. Run ops/verify_gcp_cloudsql_restore_readback.py and retain a passing parity proof.

Logical Replication Path

Use this only if the canonical source becomes Postgres or a Postgres mirror exists. SQLite cannot be logically replicated into Cloud SQL Postgres without an intermediate conversion/sync layer.

Required source privileges:

  • replication-capable source user;
  • publication over the intended schemas/tables;
  • network path from GCP to source, or source-to-GCP path through an approved proxy/tunnel.

Required target steps:

create extension if not exists vector;
create subscription <subscription_name>
connection '<redacted source connection string>'
publication <publication_name>;

Retain only redacted connection metadata. Do not commit or paste credentials.

Current Blocker

As of 2026-07-15, a newest authorized canonical Postgres capture and isolated local restore can pass, but the required T3 live private GCP lifecycle cannot be started from the current operator route:

  • direct VM SSH and the private TCP route timed out;
  • local gcloud for billy@livingip.xyz requires reauthentication; no password was entered or inspected;
  • the intended operator WIF/IAP authority is unavailable because provider teleo-iap-operator is absent, disabled, or deleted;
  • the working artifact-builder WIF identity is intentionally limited and must not be expanded or mistaken for Cloud SQL/Compute operator access.

The authorized recovery is a local gcloud auth login billy@livingip.xyz --force without sharing credentials, or administrator convergence of the reviewed operator WIF/IAP bootstrap. Until a route passes live readback, keep the claim at T2/source-local proof; do not promote staging or expose Cloud SQL publicly.

Once access is restored, run: newest authorized teleo snapshot -> disposable private teleo_clone_* restore -> generated connectivity proof -> gcp-parity.json -> no-send m3taversal blind reasoning -> live GCE reasoning attestation -> receipt-bound clone cleanup -> postflight VPS capture and delta -> eight-receipt lifecycle verdict within the 900/3600-second freshness bounds.