teleo-infrastructure/docs/gcp-kb-cloudsql-restore-runbook.md

KB Restore / Replication Runbook

This runbook is for proving Living IP KB/database redundancy on GCP.

Two Different Database Surfaces

Do not call the SQLite shadow restore a canonical Leo database copy.

Canonical Leo knowledge is currently:

• host: VPS 77.42.65.182;
• container: teleo-pg;
• engine/database: PostgreSQL 16, database teleo;
• canonical schemas: public and kb_stage;
• high-signal rows: claims, sources, claim evidence, claim edges, reasoning tools, and review-gated proposals.

The older pipeline/evaluation database is a separate surface:

• pipeline runtime DB: /opt/teleo-eval/pipeline/pipeline.db
• engine: SQLite WAL
• related Leo files:
  • /opt/teleo-eval/workspaces/main/agents/leo
  • /opt/teleo-eval/workspaces/research-leo/agents/leo
  • /opt/teleo-eval/agent-state

ops/run_gcp_cloudsql_restore_drill.sh remains a legacy SQLite-to-Postgres shadow-schema drill. It reconstructs teleo_restore inside teleo_kb; it does not preserve the canonical Postgres schema, constraints, indexes, functions, roles, or row hashes.

The last authenticated control-plane readback on 2026-07-10 reported this candidate GCP target; refresh it before any mutation:

• project: teleo-501523
• instance: teleo-pgvector-standby
• database: teleo_kb
• region: europe-west6
• network: teleo-staging-net
• private IP: 10.61.0.3
• admin password secret: gcp-teleo-pgvector-standby-postgres-password

Do not call this redundancy complete until source data has been restored or replicated and queried from GCP.

Canonical Postgres Snapshot And Parity

Capture a custom-format dump and a full JSONL manifest from the same exported, read-only PostgreSQL snapshot:

python3 ops/capture_vps_canonical_postgres_snapshot.py \
  --execute \
  --ssh-target root@77.42.65.182 \
  --ssh-key ~/.ssh/livingip_hetzner_20260604_ed25519 \
  --run-id canonical-<timestamp> \
  --output-dir <private-output-dir>

The capture fails closed if the source service changes while it runs. It retains a private custom dump, dump SHA-256, object TOC, catalog/data manifest, and before/after service state. It never restarts Leo or writes to the source database.

Run ops/postgres_parity_manifest.sql against the isolated restored target, then compare source and target:

python3 ops/verify_postgres_parity_manifest.py \
  --source <private-output-dir>/source-manifest.jsonl \
  --target <private-output-dir>/target-manifest.jsonl \
  --scope gcp_staging \
  --connectivity-proof <private-output-dir>/gcp-private-connectivity.json \
  --output <private-output-dir>/gcp-parity.json

The verifier checks all table row counts and collation-independent row hashes, plus schemas, columns/defaults, constraints, indexes, sequences, views, functions, triggers, enum/domain types, policies, required extensions, password-free application-role attributes, and bounded query timings. In GCP scope it also requires a receipt proving a staging compute source, a private server address, TLS, and public-IP-disabled instance metadata.

Use a generated target database such as teleo_clone_<run_id>. Never import a drill into teleo, teleo_kb, or teleo_canonical. Database isolation does not isolate cluster-global roles or extensions, so verify those separately and do not run the Docker-only gate bootstrap against the shared Cloud SQL instance.

After the parity verifier passes, run the no-send Cory composition replay from staging compute against that generated database. Only then delete the generated database and uploaded import object and retain cleanup readback.

Legacy SQLite Source Backup Canary

Create a consistent source backup without stopping the VPS service:

ops/backup_vps_sqlite_kb.sh

The script:

• uses SQLite .backup against /opt/teleo-eval/pipeline/pipeline.db;
• compresses and hashes the backup on the VPS;
• archives Leo/KB files while excluding secrets and logs;
• copies both artifacts locally;
• verifies SHA-256 matches;
• runs PRAGMA integrity_check on a local restored SQLite copy;
• records proof under outputs/gcp-infra-hardening-20260707/proofs/.

This proves source exportability and local restore integrity. It does not prove GCP DB redundancy until a GCP restore/import/query canary also passes.

Legacy SQLite-To-Postgres Restore Canary

Before importing into Cloud SQL, prove that the current SQLite backup can be converted and restored into PostgreSQL without row loss:

SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-<timestamp>.db.gz \
  ops/run_sqlite_postgres_restore_canary.sh

The canary:

• generates a PostgreSQL import script with ops/sqlite_to_postgres_dump.py;
• recreates a shadow schema in a disposable postgres:16-alpine container;
• imports all user tables from the SQLite backup;
• compares source and target row counts for every table;
• writes a proof JSON under outputs/gcp-infra-hardening-20260707/proofs/;
• removes only its temporary canary container.

This is a local restore/parity proof, not GCP redundancy by itself. It is the preflight that should pass before the same generated import is applied through the approved Cloud SQL connector/VPC path.

To pass this local preflight into a clean GitHub readiness run without uploading private backup paths, generated SQL, or target-count CSVs, create a redacted capsule from the proof:

python3 ops/redact_sqlite_postgres_restore_canary.py \
  --proof outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-<timestamp>.json \
  --output outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-capsule-<timestamp>.json

The capsule keeps only non-secret evidence: proof hash, backup hash, source and target table/row counts, conversion notes/stats, and the redacted-field list. It does not prove that Cloud SQL imported the data; it only proves the local SQLite-to-Postgres parity preflight.

To include the capsule in GitHub readiness:

CAPSULE_B64="$(base64 < outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-capsule-<timestamp>.json | tr -d '\n')"
gh workflow run gcp-readiness.yml \
  --repo living-ip/teleo-infrastructure \
  --ref main \
  -f restore_canary_capsule_b64="${CAPSULE_B64}"

Legacy SQLite Cloud SQL Restore Drill Runner

Prepare the exact GCS import and Cloud SQL import operation without mutating GCP:

SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-<timestamp>.db.gz \
  ops/run_gcp_cloudsql_restore_drill.sh

Execute it only from an authenticated operator environment that can write the versioned backup bucket and administer the standby Cloud SQL instance:

EXECUTE=1 \
SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-<timestamp>.db.gz \
  ops/run_gcp_cloudsql_restore_drill.sh

The runner:

• regenerates the explicit PostgreSQL import script;
• targets the shadow schema teleo_restore inside teleo_kb;
• uploads the import script to gs://teleo-501523-prod-backups/kb-dumps/cloudsql-restore-drills/... when EXECUTE=1;
• starts and waits for gcloud sql import sql;
• writes target-counts.sql for the required trusted VPC/Cloud SQL connector query readback.

The import operation alone is still not the final proof. The final proof needs target-counts.sql run against teleo-pgvector-standby and compared to the source counts in the drill proof.

After the import operation is DONE, run the generated count query from a trusted VPC runtime or Cloud SQL connector path and retain CSV output:

psql "$TELEO_CLOUDSQL_DATABASE_URL" \
  --csv \
  -f outputs/gcp-infra-hardening-20260707/private-cloudsql-restore-drills/gcp-cloudsql-restore-drill-<timestamp>/target-counts.sql \
  > outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-target-counts-<timestamp>.csv

Then compare the Cloud SQL readback to the source proof:

python3 ops/verify_gcp_cloudsql_restore_readback.py \
  --drill-proof outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-restore-drill-<timestamp>.json \
  --target-counts-csv outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-target-counts-<timestamp>.csv \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-restore-readback-verification-<timestamp>.json

Only a status = pass verifier output is enough for row-count parity. It still does not prove application cutover or continuous replication.

Required Proof

A successful restore or replication canary must retain:

• source dataset identity:
  • source host or dump artifact;
  • dump timestamp or replication slot timestamp;
  • source schema/database name.
• transfer proof:
  • dump object path in a versioned bucket, or logical replication subscription details;
  • row/table counts before import where available.
• target proof:
  • teleo-pgvector-standby readback;
  • teleo_kb database readback;
  • extension readback for vector if the restored schema needs pgvector;
  • representative query readback for core KB tables.
• failure boundary:
  • exact missing secret, source access, schema incompatibility, extension issue, or import error.

One-Shot SQLite Export / GCP Restore Path

Use this while the canonical DB remains SQLite on the VPS and we need a GCP restore drill.

1. Run ops/backup_vps_sqlite_kb.sh.
2. Upload the resulting SQLite backup and Leo/KB tarball to a versioned GCS bucket such as gs://teleo-501523-prod-backups/kb-dumps/.
3. Run the local SQLite-to-Postgres restore canary above and retain its proof.
4. Run ops/run_gcp_cloudsql_restore_drill.sh in dry-run mode to generate the GCS import plan.
5. Run EXECUTE=1 ops/run_gcp_cloudsql_restore_drill.sh from an authenticated operator environment to upload and import the generated SQL. Do not run blind string rewrites against the SQLite dump.
6. Install required extensions on Cloud SQL:

create extension if not exists vector;

7. From a trusted VPC runtime or Cloud SQL connector path, run readbacks:

select current_database();
select extname, extversion from pg_extension where extname = 'vector';
select schemaname, tablename from pg_tables where schemaname not in ('pg_catalog', 'information_schema') order by 1, 2 limit 50;

8. Retain the SQLite backup hash, GCS object generation, import/conversion operation, query output, and row-count sample.
9. Run ops/verify_gcp_cloudsql_restore_readback.py and retain a passing parity proof.

Logical Replication Path

Use this only if the canonical source becomes Postgres or a Postgres mirror exists. SQLite cannot be logically replicated into Cloud SQL Postgres without an intermediate conversion/sync layer.

Required source privileges:

• replication-capable source user;
• publication over the intended schemas/tables;
• network path from GCP to source, or source-to-GCP path through an approved proxy/tunnel.

Required target steps:

create extension if not exists vector;
create subscription <subscription_name>
connection '<redacted source connection string>'
publication <publication_name>;

Retain only redacted connection metadata. Do not commit or paste credentials.

Current Blocker

As of 2026-07-11, the canonical Postgres exported-snapshot capture and isolated local restore parity pass. Live GCP restore and staging replay do not.

• GitHub WIF works for sa-artifact-builder, but that identity is intentionally limited to Artifact Registry and cannot inspect or mutate Cloud SQL/Compute.
• The configured sa-teleo-readiness and sa-teleo-restore-drill identities return IAM 404 and do not exist.
• The local privileged billy@livingip.xyz gcloud session requires password reauthentication. No password was entered or inspected.
• Direct VM SSH is closed to the current egress /32; IAP requires the same privileged GCP authentication.

That is why the readiness checker still reports:

• kb_source_restore_access = blocked
• kb_restore_or_replication = blocked

The immediate operator CTA is to complete gcloud auth login billy@livingip.xyz --force locally without sharing the password, or apply the reviewed IAM split with an authorized GCP administrator. The next non-user action is:

canonical teleo snapshot -> generated Cloud SQL database -> full parity and private-connectivity verifier -> no-send Cory composition replay from staging compute -> delete the generated database/object -> retain cleanup proof.