teleo-infrastructure/docs/gcp-kb-cloudsql-restore-runbook.md

# KB Restore / Replication Runbook

This runbook is for proving Living IP KB/database redundancy on GCP.

## Two Different Database Surfaces

Do not call the SQLite shadow restore a canonical Leo database copy.

Canonical Leo knowledge is currently:

- host: VPS `77.42.65.182`;
- container: `teleo-pg`;
- engine/database: PostgreSQL 16, database `teleo`;
- canonical schemas: `public` and `kb_stage`;
- high-signal rows: claims, sources, claim evidence, claim edges, reasoning
  tools, and review-gated proposals.

The older pipeline/evaluation database is a separate surface:

- pipeline runtime DB: `/opt/teleo-eval/pipeline/pipeline.db`
- engine: SQLite WAL
- related Leo files:
  - `/opt/teleo-eval/workspaces/main/agents/leo`
  - `/opt/teleo-eval/workspaces/research-leo/agents/leo`
  - `/opt/teleo-eval/agent-state`

`ops/run_gcp_cloudsql_restore_drill.sh` remains a legacy SQLite-to-Postgres
shadow-schema drill. It reconstructs `teleo_restore` inside `teleo_kb`; it does
not preserve the canonical Postgres schema, constraints, indexes, functions,
roles, or row hashes.

The last authenticated control-plane readback on 2026-07-10 reported this
candidate GCP target; refresh it before any mutation:

- project: `teleo-501523`
- instance: `teleo-pgvector-standby`
- database: `teleo_kb`
- region: `europe-west6`
- network: `teleo-staging-net`
- private IP: `10.61.0.3`
- admin password secret: `gcp-teleo-pgvector-standby-postgres-password`

Do not call this redundancy complete until source data has been restored or replicated and queried from GCP.

## Canonical Postgres Snapshot And Parity

Capture a custom-format dump and a full JSONL manifest from the same exported,
read-only PostgreSQL snapshot:

```bash
python3 ops/capture_vps_canonical_postgres_snapshot.py \
  --execute \
  --ssh-target root@77.42.65.182 \
  --ssh-key ~/.ssh/livingip_hetzner_20260604_ed25519 \
  --run-id canonical-<timestamp> \
  --output-dir <private-output-dir>
```

The capture fails closed if the source service changes while it runs. It
retains a private custom dump, dump SHA-256, object TOC, catalog/data manifest,
and before/after service state. It never restarts Leo or writes to the source
database.

Run `ops/postgres_parity_manifest.sql` against the isolated restored target,
then compare source and target:

```bash
python3 ops/verify_postgres_parity_manifest.py \
  --source <private-output-dir>/source-manifest.jsonl \
  --target <private-output-dir>/target-manifest.jsonl \
  --scope gcp_staging \
  --connectivity-proof <private-output-dir>/gcp-private-connectivity.json \
  --output <private-output-dir>/gcp-parity.json
```

The verifier checks all table row counts and collation-independent row hashes,
plus schemas, columns/defaults, constraints, indexes, sequences, views,
functions, triggers, enum/domain types, policies, required extensions,
password-free application-role attributes, and bounded query timings. In GCP
scope it also requires a receipt proving a staging compute source, a private
server address, TLS, and public-IP-disabled instance metadata.

Use a generated target database such as `teleo_clone_<run_id>`. Never import a
drill into `teleo`, `teleo_kb`, or `teleo_canonical`. Database isolation does
not isolate cluster-global roles or extensions, so verify those separately and
do not run the Docker-only gate bootstrap against the shared Cloud SQL instance.

After the parity verifier passes, run the no-send Cory composition replay from
staging compute against that generated database. Only then delete the generated
database and uploaded import object and retain cleanup readback.

## Legacy SQLite Source Backup Canary

Create a consistent source backup without stopping the VPS service:

```bash
ops/backup_vps_sqlite_kb.sh
```

The script:

- uses SQLite `.backup` against `/opt/teleo-eval/pipeline/pipeline.db`;
- compresses and hashes the backup on the VPS;
- archives Leo/KB files while excluding `secrets` and logs;
- copies both artifacts locally;
- verifies SHA-256 matches;
- runs `PRAGMA integrity_check` on a local restored SQLite copy;
- records proof under `outputs/gcp-infra-hardening-20260707/proofs/`.

This proves source exportability and local restore integrity. It does not prove GCP DB redundancy until a GCP restore/import/query canary also passes.

## Legacy SQLite-To-Postgres Restore Canary

Before importing into Cloud SQL, prove that the current SQLite backup can be
converted and restored into PostgreSQL without row loss:

```bash
SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-<timestamp>.db.gz \
  ops/run_sqlite_postgres_restore_canary.sh
```

The canary:

- generates a PostgreSQL import script with `ops/sqlite_to_postgres_dump.py`;
- recreates a shadow schema in a disposable `postgres:16-alpine` container;
- imports all user tables from the SQLite backup;
- compares source and target row counts for every table;
- writes a proof JSON under `outputs/gcp-infra-hardening-20260707/proofs/`;
- removes only its temporary canary container.

This is a local restore/parity proof, not GCP redundancy by itself. It is the
preflight that should pass before the same generated import is applied through
the approved Cloud SQL connector/VPC path.

To pass this local preflight into a clean GitHub readiness run without uploading
private backup paths, generated SQL, or target-count CSVs, create a redacted
capsule from the proof:

```bash
python3 ops/redact_sqlite_postgres_restore_canary.py \
  --proof outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-<timestamp>.json \
  --output outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-capsule-<timestamp>.json
```

The capsule keeps only non-secret evidence: proof hash, backup hash, source and
target table/row counts, conversion notes/stats, and the redacted-field list.
It does not prove that Cloud SQL imported the data; it only proves the local
SQLite-to-Postgres parity preflight.

To include the capsule in GitHub readiness:

```bash
CAPSULE_B64="$(base64 < outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-capsule-<timestamp>.json | tr -d '\n')"
gh workflow run gcp-readiness.yml \
  --repo living-ip/teleo-infrastructure \
  --ref main \
  -f restore_canary_capsule_b64="${CAPSULE_B64}"
```

## Legacy SQLite Cloud SQL Restore Drill Runner

Prepare the exact GCS import and Cloud SQL import operation without mutating GCP:

```bash
SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-<timestamp>.db.gz \
  ops/run_gcp_cloudsql_restore_drill.sh
```

Execute it only from an authenticated operator environment that can write the
versioned backup bucket and administer the standby Cloud SQL instance:

```bash
EXECUTE=1 \
SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-<timestamp>.db.gz \
  ops/run_gcp_cloudsql_restore_drill.sh
```

The runner:

- regenerates the explicit PostgreSQL import script;
- targets the shadow schema `teleo_restore` inside `teleo_kb`;
- uploads the import script to `gs://teleo-501523-prod-backups/kb-dumps/cloudsql-restore-drills/...` when `EXECUTE=1`;
- starts and waits for `gcloud sql import sql`;
- writes `target-counts.sql` for the required trusted VPC/Cloud SQL connector query readback.

The import operation alone is still not the final proof. The final proof needs
`target-counts.sql` run against `teleo-pgvector-standby` and compared to the
source counts in the drill proof.

After the import operation is `DONE`, run the generated count query from a
trusted VPC runtime or Cloud SQL connector path and retain CSV output:

```bash
psql "$TELEO_CLOUDSQL_DATABASE_URL" \
  --csv \
  -f outputs/gcp-infra-hardening-20260707/private-cloudsql-restore-drills/gcp-cloudsql-restore-drill-<timestamp>/target-counts.sql \
  > outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-target-counts-<timestamp>.csv
```

Then compare the Cloud SQL readback to the source proof:

```bash
python3 ops/verify_gcp_cloudsql_restore_readback.py \
  --drill-proof outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-restore-drill-<timestamp>.json \
  --target-counts-csv outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-target-counts-<timestamp>.csv \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-restore-readback-verification-<timestamp>.json
```

Only a `status = pass` verifier output is enough for row-count parity. It still
does not prove application cutover or continuous replication.

## Required Proof

A successful restore or replication canary must retain:

- source dataset identity:
  - source host or dump artifact;
  - dump timestamp or replication slot timestamp;
  - source schema/database name.
- transfer proof:
  - dump object path in a versioned bucket, or logical replication subscription details;
  - row/table counts before import where available.
- target proof:
  - `teleo-pgvector-standby` readback;
  - `teleo_kb` database readback;
  - extension readback for `vector` if the restored schema needs pgvector;
  - representative query readback for core KB tables.
- failure boundary:
  - exact missing secret, source access, schema incompatibility, extension issue, or import error.

## One-Shot SQLite Export / GCP Restore Path

Use this while the canonical DB remains SQLite on the VPS and we need a GCP restore drill.

1. Run `ops/backup_vps_sqlite_kb.sh`.
2. Upload the resulting SQLite backup and Leo/KB tarball to a versioned GCS bucket such as `gs://teleo-501523-prod-backups/kb-dumps/`.
3. Run the local SQLite-to-Postgres restore canary above and retain its proof.
4. Run `ops/run_gcp_cloudsql_restore_drill.sh` in dry-run mode to generate the GCS import plan.
5. Run `EXECUTE=1 ops/run_gcp_cloudsql_restore_drill.sh` from an authenticated operator environment to upload and import the generated SQL. Do not run blind string rewrites against the SQLite dump.
6. Install required extensions on Cloud SQL:

```sql
create extension if not exists vector;
```

7. From a trusted VPC runtime or Cloud SQL connector path, run readbacks:

```sql
select current_database();
select extname, extversion from pg_extension where extname = 'vector';
select schemaname, tablename from pg_tables where schemaname not in ('pg_catalog', 'information_schema') order by 1, 2 limit 50;
```

8. Retain the SQLite backup hash, GCS object generation, import/conversion operation, query output, and row-count sample.
9. Run `ops/verify_gcp_cloudsql_restore_readback.py` and retain a passing parity proof.

## Logical Replication Path

Use this only if the canonical source becomes Postgres or a Postgres mirror exists. SQLite cannot be logically replicated into Cloud SQL Postgres without an intermediate conversion/sync layer.

Required source privileges:

- replication-capable source user;
- publication over the intended schemas/tables;
- network path from GCP to source, or source-to-GCP path through an approved proxy/tunnel.

Required target steps:

```sql
create extension if not exists vector;
create subscription <subscription_name>
connection '<redacted source connection string>'
publication <publication_name>;
```

Retain only redacted connection metadata. Do not commit or paste credentials.

## Current Blocker

As of 2026-07-11, the canonical Postgres exported-snapshot capture and isolated
local restore parity pass. Live GCP restore and staging replay do not.

- GitHub WIF works for `sa-artifact-builder`, but that identity is intentionally
  limited to Artifact Registry and cannot inspect or mutate Cloud SQL/Compute.
- The configured `sa-teleo-readiness` and `sa-teleo-restore-drill` identities
  return IAM 404 and do not exist.
- The local privileged `billy@livingip.xyz` gcloud session requires password
  reauthentication. No password was entered or inspected.
- Direct VM SSH is closed to the current egress `/32`; IAP requires the same
  privileged GCP authentication.

That is why the readiness checker still reports:

- `kb_source_restore_access = blocked`
- `kb_restore_or_replication = blocked`

The immediate operator CTA is to complete
`gcloud auth login billy@livingip.xyz --force` locally without sharing the
password, or apply the reviewed IAM split with an authorized GCP administrator.
The next non-user action is:

canonical `teleo` snapshot -> generated Cloud SQL database -> full parity and
private-connectivity verifier -> no-send Cory composition replay from staging
compute -> delete the generated database/object -> retain cleanup proof.