teleo-infrastructure/docs/gcp-kb-cloudsql-restore-runbook.md

KB Restore / Replication Runbook

This runbook is for proving Living IP KB/database redundancy on GCP.

Current source reality:

• canonical runtime DB: /opt/teleo-eval/pipeline/pipeline.db
• engine: SQLite WAL
• canonical Leo files:
  • /opt/teleo-eval/workspaces/main/agents/leo
  • /opt/teleo-eval/workspaces/research-leo/agents/leo
  • /opt/teleo-eval/agent-state

The standby target already exists:

• project: teleo-501523
• instance: teleo-pgvector-standby
• database: teleo_kb
• region: europe-west6
• network: teleo-staging-net
• private IP: 10.61.0.3
• admin password secret: gcp-teleo-pgvector-standby-postgres-password

Do not call this redundancy complete until source data has been restored or replicated and queried from GCP.

Source Backup Canary

Create a consistent source backup without stopping the VPS service:

ops/backup_vps_sqlite_kb.sh

The script:

• uses SQLite .backup against /opt/teleo-eval/pipeline/pipeline.db;
• compresses and hashes the backup on the VPS;
• archives Leo/KB files while excluding secrets and logs;
• copies both artifacts locally;
• verifies SHA-256 matches;
• runs PRAGMA integrity_check on a local restored SQLite copy;
• records proof under outputs/gcp-infra-hardening-20260707/proofs/.

This proves source exportability and local restore integrity. It does not prove GCP DB redundancy until a GCP restore/import/query canary also passes.

Local SQLite-To-Postgres Restore Canary

Before importing into Cloud SQL, prove that the current SQLite backup can be converted and restored into PostgreSQL without row loss:

SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-<timestamp>.db.gz \
  ops/run_sqlite_postgres_restore_canary.sh

The canary:

• generates a PostgreSQL import script with ops/sqlite_to_postgres_dump.py;
• recreates a shadow schema in a disposable postgres:16-alpine container;
• imports all user tables from the SQLite backup;
• compares source and target row counts for every table;
• writes a proof JSON under outputs/gcp-infra-hardening-20260707/proofs/;
• removes only its temporary canary container.

This is a local restore/parity proof, not GCP redundancy by itself. It is the preflight that should pass before the same generated import is applied through the approved Cloud SQL connector/VPC path.

To pass this local preflight into a clean GitHub readiness run without uploading private backup paths, generated SQL, or target-count CSVs, create a redacted capsule from the proof:

python3 ops/redact_sqlite_postgres_restore_canary.py \
  --proof outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-<timestamp>.json \
  --output outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-capsule-<timestamp>.json

The capsule keeps only non-secret evidence: proof hash, backup hash, source and target table/row counts, conversion notes/stats, and the redacted-field list. It does not prove that Cloud SQL imported the data; it only proves the local SQLite-to-Postgres parity preflight.

To include the capsule in GitHub readiness:

CAPSULE_B64="$(base64 < outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-capsule-<timestamp>.json | tr -d '\n')"
gh workflow run gcp-readiness.yml \
  --repo living-ip/teleo-infrastructure \
  --ref main \
  -f restore_canary_capsule_b64="${CAPSULE_B64}"

Cloud SQL Restore Drill Runner

Prepare the exact GCS import and Cloud SQL import operation without mutating GCP:

SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-<timestamp>.db.gz \
  ops/run_gcp_cloudsql_restore_drill.sh

Execute it only from an authenticated operator environment that can write the versioned backup bucket and administer the standby Cloud SQL instance:

EXECUTE=1 \
SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-<timestamp>.db.gz \
  ops/run_gcp_cloudsql_restore_drill.sh

The runner:

• regenerates the explicit PostgreSQL import script;
• targets the shadow schema teleo_restore inside teleo_kb;
• uploads the import script to gs://teleo-501523-prod-backups/kb-dumps/cloudsql-restore-drills/... when EXECUTE=1;
• starts and waits for gcloud sql import sql;
• writes target-counts.sql for the required trusted VPC/Cloud SQL connector query readback.

The import operation alone is still not the final proof. The final proof needs target-counts.sql run against teleo-pgvector-standby and compared to the source counts in the drill proof.

After the import operation is DONE, run the generated count query from a trusted VPC runtime or Cloud SQL connector path and retain CSV output:

psql "$TELEO_CLOUDSQL_DATABASE_URL" \
  --csv \
  -f outputs/gcp-infra-hardening-20260707/private-cloudsql-restore-drills/gcp-cloudsql-restore-drill-<timestamp>/target-counts.sql \
  > outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-target-counts-<timestamp>.csv

Then compare the Cloud SQL readback to the source proof:

python3 ops/verify_gcp_cloudsql_restore_readback.py \
  --drill-proof outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-restore-drill-<timestamp>.json \
  --target-counts-csv outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-target-counts-<timestamp>.csv \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-restore-readback-verification-<timestamp>.json

Only a status = pass verifier output is enough for row-count parity. It still does not prove application cutover or continuous replication.

Required Proof

A successful restore or replication canary must retain:

• source dataset identity:
  • source host or dump artifact;
  • dump timestamp or replication slot timestamp;
  • source schema/database name.
• transfer proof:
  • dump object path in a versioned bucket, or logical replication subscription details;
  • row/table counts before import where available.
• target proof:
  • teleo-pgvector-standby readback;
  • teleo_kb database readback;
  • extension readback for vector if the restored schema needs pgvector;
  • representative query readback for core KB tables.
• failure boundary:
  • exact missing secret, source access, schema incompatibility, extension issue, or import error.

One-Shot SQLite Export / GCP Restore Path

Use this while the canonical DB remains SQLite on the VPS and we need a GCP restore drill.

1. Run ops/backup_vps_sqlite_kb.sh.
2. Upload the resulting SQLite backup and Leo/KB tarball to a versioned GCS bucket such as gs://teleo-501523-prod-backups/kb-dumps/.
3. Run the local SQLite-to-Postgres restore canary above and retain its proof.
4. Run ops/run_gcp_cloudsql_restore_drill.sh in dry-run mode to generate the GCS import plan.
5. Run EXECUTE=1 ops/run_gcp_cloudsql_restore_drill.sh from an authenticated operator environment to upload and import the generated SQL. Do not run blind string rewrites against the SQLite dump.
6. Install required extensions on Cloud SQL:

create extension if not exists vector;

7. From a trusted VPC runtime or Cloud SQL connector path, run readbacks:

select current_database();
select extname, extversion from pg_extension where extname = 'vector';
select schemaname, tablename from pg_tables where schemaname not in ('pg_catalog', 'information_schema') order by 1, 2 limit 50;

8. Retain the SQLite backup hash, GCS object generation, import/conversion operation, query output, and row-count sample.
9. Run ops/verify_gcp_cloudsql_restore_readback.py and retain a passing parity proof.

Logical Replication Path

Use this only if the canonical source becomes Postgres or a Postgres mirror exists. SQLite cannot be logically replicated into Cloud SQL Postgres without an intermediate conversion/sync layer.

Required source privileges:

• replication-capable source user;
• publication over the intended schemas/tables;
• network path from GCP to source, or source-to-GCP path through an approved proxy/tunnel.

Required target steps:

create extension if not exists vector;
create subscription <subscription_name>
connection '<redacted source connection string>'
publication <publication_name>;

Retain only redacted connection metadata. Do not commit or paste credentials.

Current Blocker

As of this run, GCP has the standby target, a repeatable SQLite/KB export script, and a local SQLite-to-Postgres restore canary. It does not have an approved GCP upload/restore credential in the current local session, nor a retained Cloud SQL import/query proof. That is why the readiness checker still reports:

• kb_source_restore_access = blocked
• kb_restore_or_replication = blocked

The next real canary is:

source SQLite/KB backup -> upload to versioned GCS bucket -> convert/import into teleo_kb -> install/verify vector if needed -> run representative KB queries on GCP -> retain proof.