teleo-infrastructure/docs/gcp-kb-cloudsql-restore-runbook.md

# KB Restore / Replication Runbook

This runbook is for proving Living IP KB/database redundancy on GCP.

## Two Different Database Surfaces

Do not call the SQLite shadow restore a canonical Leo database copy.

Canonical Leo knowledge is currently:

- host: VPS `77.42.65.182`;
- container: `teleo-pg`;
- engine/database: PostgreSQL 16, database `teleo`;
- canonical schemas: `public` and `kb_stage`;
- high-signal rows: claims, sources, claim evidence, claim edges, reasoning
  tools, and review-gated proposals.

The older pipeline/evaluation database is a separate surface:

- pipeline runtime DB: `/opt/teleo-eval/pipeline/pipeline.db`
- engine: SQLite WAL
- related Leo files:
  - `/opt/teleo-eval/workspaces/main/agents/leo`
  - `/opt/teleo-eval/workspaces/research-leo/agents/leo`
  - `/opt/teleo-eval/agent-state`

`ops/run_gcp_cloudsql_restore_drill.sh` remains a legacy SQLite-to-Postgres
shadow-schema drill. It reconstructs `teleo_restore` inside `teleo_kb`; it does
not preserve the canonical Postgres schema, constraints, indexes, functions,
roles, or row hashes.

The last authenticated control-plane readback on 2026-07-10 reported this
candidate GCP target; refresh it before any mutation:

- project: `teleo-501523`
- instance: `teleo-pgvector-standby`
- database: `teleo_kb`
- region: `europe-west6`
- network: `teleo-staging-net`
- private IP: `10.61.0.3`
- admin password secret: `gcp-teleo-pgvector-standby-postgres-password`

Do not call this redundancy complete until source data has been restored or replicated and queried from GCP.

## Canonical Postgres Snapshot And Parity

Capture a custom-format dump and a full JSONL manifest from the same exported,
read-only PostgreSQL snapshot:

```bash
python3 ops/capture_vps_canonical_postgres_snapshot.py \
  --execute \
  --ssh-target root@77.42.65.182 \
  --ssh-key ~/.ssh/livingip_hetzner_20260604_ed25519 \
  --run-id canonical-<timestamp> \
  --authorization-ref codex-delegation:THREAD_ID \
  --output-dir <private-output-dir>
```

Replace `THREAD_ID` with the exact retained authorization reference for the
capture. The v2 receipt binds that reference, exported snapshot ID, TXID
snapshot, WAL LSN, source system identifier, dump hash, manifest hash, reviewed
manifest-SQL hash, and source-context hash. A dump and manifest without this
passing receipt are not eligible for GCP restore.

The capture fails closed unless the structured source-service states before
and after the snapshot are identical and healthy: `ActiveState=active`,
`SubState=running`, a positive `MainPID`, and a nonnegative `NRestarts`. It
retains a private custom dump, dump SHA-256, object TOC, catalog/data manifest,
and both service-state objects. It never restarts Leo or writes to the source
database.

Prove that this exact snapshot can rebuild a blank Postgres target before using
it for GCP:

```bash
.venv/bin/python ops/run_local_canonical_postgres_rebuild.py \
  --dump <private-output-dir>/teleo-canonical.dump \
  --source-manifest <private-output-dir>/source-manifest.jsonl \
  --output /tmp/teleo-canonical-rebuild-receipt.json
```

The local runner starts a uniquely named `postgres:16-alpine` container with
network mode `none` and tmpfs-only database storage. It waits for an actual
`psql` connection to the named database, restores with `pg_restore
--no-owner --no-privileges --exit-on-error`, compares the full parity manifest,
then removes the container and proves it is absent. A passing local receipt is
the exact-recovery preflight; it is not semantic recompilation from raw source
documents. Full parity now includes `kb_gate_owner`, `kb_apply`, and
`kb_review`, their memberships, object ownership, and normalized database,
schema, relation, column, function, and type ACLs. A restore that recovers rows
but not those authorization semantics fails closed.

Choose one bounded suffix and use it consistently on the GCP replay VM. The
restore helper retains its mode-0700 evidence directory at the fixed private
root:

```bash
RUN_SUFFIX=20260715t010000z
TARGET_DB="teleo_clone_${RUN_SUFFIX}"
RESTORE_RUN_ID="gcp-restore-${RUN_SUFFIX}"
PRIVATE_RUN_DIR="/var/lib/teleo-gcp-restore-runs/${RESTORE_RUN_ID}"
```

Use a generated target database such as `teleo_clone_<run_id>`. Never import a
drill into `teleo`, `teleo_kb`, or `teleo_canonical`. Database isolation does
not isolate cluster-global roles or extensions, so verify those separately and
do not run the Docker-only gate bootstrap against the shared Cloud SQL instance.

The canonical restore command must include the v2 capture receipt. It also runs
a live Cloud SQL control-plane preflight before resolving the database secret or
creating the clone, and fails closed unless the exact instance is runnable,
PostgreSQL 16, private-networked, bound to the expected RFC1918 host, and has
public IP disabled. A GCE metadata preflight independently binds the operator
runtime to `teleo-prod-1` in `europe-west6-a` on `teleo-staging-net`; a
hard-coded compute label is not accepted. It also requires the exact
least-privilege restore identity and libpq `verify-full` using a reviewed server
name and private CA file. Determine the certificate DNS SAN from the live Cloud
SQL certificate before filling `--ssl-server-name`; do not substitute the
private IP or downgrade TLS verification:

```bash
sudo python3 ops/restore_gcp_generated_postgres_snapshot.py restore \
  --execute \
  --target-db "$TARGET_DB" \
  --run-id "$RESTORE_RUN_ID" \
  --dump <private-output-dir>/teleo-canonical.dump \
  --expected-dump-sha256 <dump-sha256> \
  --source-manifest <private-output-dir>/source-manifest.jsonl \
  --source-receipt <private-output-dir>/receipt.json \
  --expected-source-receipt-sha256 <source-receipt-sha256> \
  --expected-capture-authorization-ref codex-delegation:THREAD_ID \
  --project teleo-501523 \
  --cloudsql-instance teleo-pgvector-standby \
  --expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \
  --expected-compute-instance teleo-prod-1 \
  --expected-compute-zone europe-west6-a \
  --expected-restore-service-account sa-teleo-restore-drill@teleo-501523.iam.gserviceaccount.com \
  --ssl-server-name <dns-name-covered-by-cloudsql-server-certificate> \
  --ssl-root-cert /etc/teleo/cloudsql/server-ca.pem \
  --expected-source-ssh-target root@77.42.65.182 \
  --expected-source-container teleo-pg \
  --expected-source-database teleo \
  --expected-source-service leoclean-gateway.service
```

The restore writes both `target-manifest.jsonl` and
`gcp-private-connectivity.json` as mode-0600 files in its private run
directory. The latter is composed from live GCE metadata, Cloud SQL
control-plane checks, and the private TLS database identity; do not hand-compose
the connectivity receipt. The restore also records identical healthy
before/after state for `leoclean-gcp-prod-parallel.service`, including its PID
and restart count. Receipt payloads remain in mode-0600 files; command stdout
contains only a status, output path, and receipt hash.

Compare the captured target manifest and generated connectivity receipt to the
authorized source:

```bash
sudo python3 ops/verify_postgres_parity_manifest.py \
  --source <private-output-dir>/source-manifest.jsonl \
  --target "$PRIVATE_RUN_DIR/target-manifest.jsonl" \
  --scope gcp_staging \
  --connectivity-proof "$PRIVATE_RUN_DIR/gcp-private-connectivity.json" \
  --output "$PRIVATE_RUN_DIR/gcp-parity.json"
```

The verifier checks all table row counts and collation-independent row hashes,
plus schemas, columns/defaults, constraints, indexes, sequences, views,
functions, triggers, enum/domain types, policies, required extensions,
password-free application-role attributes, exact role memberships, object
owners, normalized ACLs, and bounded query timings. Role and extension sets are
symmetric: an unexpected extra role, superuser, or extension is a mismatch.
Every table must have a nonempty row hash. In GCP scope it also requires a
receipt proving a staging compute source, a private server address,
certificate-authenticated `verify-full` TLS, and public-IP-disabled instance
metadata.

The snapshot dump intentionally strips owners and ACLs. Therefore the GCP
restore cannot pass the expanded parity gate until a separately reviewed,
Cloud-SQL-compatible authorization replay reconstructs the captured ownership
and grants. Do not run the Docker/superuser bootstrap blindly on shared Cloud
SQL. Until that replay exists, this lane remains T2 preparation and must fail
before a T3 claim.

After the parity verifier passes, run the no-send m3taversal blind reasoning
replay from staging compute against that generated database, attest the replay
host from live GCE metadata, and only then delete the generated database. This
canonical path streams the custom dump through `pg_restore`; it does not create
a GCS import object.

Run the bounded ID-free reasoning canary with the generated parity receipt. The
script re-executes itself in the Hermes virtualenv when that runtime is present:

```bash
sudo python3 scripts/run_gcp_generated_db_blind_claim_canary.py \
  --execute \
  --target-db "$TARGET_DB" \
  --cloudsql-tool hermes-agent/leoclean-bin/cloudsql_memory_tool.py \
  --manifest-sql ops/postgres_parity_manifest.sql \
  --parity-receipt "$PRIVATE_RUN_DIR/gcp-parity.json" \
  --output "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \
  --run-id "gcp-blind-claim-${RUN_SUFFIX}"
```

Before cleanup, bind that reasoning receipt to live metadata from the same GCE
replay host:

```bash
sudo python3 ops/attest_gcp_reasoning_compute.py \
  --reasoning-receipt "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \
  --restore-run-id "$RESTORE_RUN_ID" \
  --target-db "$TARGET_DB" \
  --project teleo-501523 \
  --expected-compute-instance teleo-prod-1 \
  --expected-compute-zone europe-west6-a \
  --expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \
  --expected-restore-service-account sa-teleo-restore-drill@teleo-501523.iam.gserviceaccount.com \
  --output "$PRIVATE_RUN_DIR/reasoning-compute-attestation.json"
```

After the compute attestation passes, run the receipt-bound cleanup command
retained by the restore. Then capture the VPS source again under a distinct
postflight authorization reference. Run this capture from the same trusted
source-access lane used for the baseline; it need not run on the GCP replay VM.
The postflight capture must complete after cleanup, so the final verdict can
state whether the VPS changed after the restored snapshot boundary. Compare the
two source captures:

```bash
python3 ops/capture_vps_canonical_postgres_snapshot.py \
  --execute \
  --ssh-target root@77.42.65.182 \
  --ssh-key ~/.ssh/livingip_hetzner_20260604_ed25519 \
  --run-id canonical-${RUN_SUFFIX}-postflight \
  --authorization-ref codex-delegation:THREAD_ID:postflight \
  --output-dir <private-postflight-dir>
```

```bash
sudo python3 ops/verify_vps_canonical_snapshot_delta.py \
  --baseline-receipt <private-output-dir>/receipt.json \
  --baseline-manifest <private-output-dir>/source-manifest.jsonl \
  --baseline-authorization-ref codex-delegation:THREAD_ID \
  --current-receipt <private-postflight-dir>/receipt.json \
  --current-manifest <private-postflight-dir>/source-manifest.jsonl \
  --current-authorization-ref codex-delegation:THREAD_ID:postflight \
  --output "$PRIVATE_RUN_DIR/source-delta-receipt.json"
```

Finally, verify that all eight receipts belong to one bound lifecycle. This
verifier rejects local-scope parity, stale or mismatched source hashes, a
postflight source capture that predates reasoning or cleanup, hand-authored
reasoning booleans without real retrieval/tool receipts, an unattested replay
host, compute or Cloud SQL topology drift, and any cleanup receipt that leaves
the clone present or claims cleanup when the named clone never existed. Both
source captures must use the same reviewed manifest SQL. The postflight capture may be at most 900 seconds old at
verification; the GCP lifecycle age and restore-to-postflight span must each be
at most 3600 seconds:

```bash
sudo python3 ops/verify_gcp_canonical_lifecycle.py \
  --source-receipt <private-output-dir>/receipt.json \
  --current-source-receipt <private-postflight-dir>/receipt.json \
  --source-delta-receipt "$PRIVATE_RUN_DIR/source-delta-receipt.json" \
  --restore-receipt "$PRIVATE_RUN_DIR/restore-receipt.json" \
  --parity-receipt "$PRIVATE_RUN_DIR/gcp-parity.json" \
  --reasoning-receipt "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \
  --reasoning-compute-receipt "$PRIVATE_RUN_DIR/reasoning-compute-attestation.json" \
  --cleanup-receipt "$PRIVATE_RUN_DIR/cleanup-receipt.json" \
  --max-postflight-age-seconds 900 \
  --max-lifecycle-age-seconds 3600 \
  --output "$PRIVATE_RUN_DIR/lifecycle-verification.json"
```

This command validates only the internal consistency of caller-supplied
receipts. A clean result is emitted as
`prepared_external_attestation_required`, with
`current_tier=T2_structural_receipt_bundle`, `t3_receipt_ready=false`, and
`accepted_by_this_api=false`. It cannot emit a live-GCP/T3 claim. T3 requires a
separate independently authenticated provider or platform receipt that the
bundle author cannot mint.

Retain the private receipt run directory and every useful lifecycle input,
supporting receipt, and verifier output. Remove only transient upload/bundle
directories, temporary Hermes profiles, helper clients, and child processes
after their absence is verified. There is no canonical-path GCS object to
delete. Do not remove the evidence directory until its useful receipts are
integrated or explicitly rejected.

## Legacy SQLite Source Backup Canary

Create a consistent source backup without stopping the VPS service:

```bash
ops/backup_vps_sqlite_kb.sh
```

The script:

- uses SQLite `.backup` against `/opt/teleo-eval/pipeline/pipeline.db`;
- compresses and hashes the backup on the VPS;
- archives Leo/KB files while excluding `secrets` and logs;
- copies both artifacts locally;
- verifies SHA-256 matches;
- runs `PRAGMA integrity_check` on a local restored SQLite copy;
- records proof under `outputs/gcp-infra-hardening-20260707/proofs/`.

This proves source exportability and local restore integrity. It does not prove GCP DB redundancy until a GCP restore/import/query canary also passes.

## Legacy SQLite-To-Postgres Restore Canary

Before importing into Cloud SQL, prove that the current SQLite backup can be
converted and restored into PostgreSQL without row loss:

```bash
SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-<timestamp>.db.gz \
  ops/run_sqlite_postgres_restore_canary.sh
```

The canary:

- generates a PostgreSQL import script with `ops/sqlite_to_postgres_dump.py`;
- recreates a shadow schema in a disposable `postgres:16-alpine` container;
- imports all user tables from the SQLite backup;
- compares source and target row counts for every table;
- writes a proof JSON under `outputs/gcp-infra-hardening-20260707/proofs/`;
- removes only its temporary canary container.

This is a local restore/parity proof, not GCP redundancy by itself. It is the
preflight that should pass before the same generated import is applied through
the approved Cloud SQL connector/VPC path.

To pass this local preflight into a clean GitHub readiness run without uploading
private backup paths, generated SQL, or target-count CSVs, create a redacted
capsule from the proof:

```bash
python3 ops/redact_sqlite_postgres_restore_canary.py \
  --proof outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-<timestamp>.json \
  --output outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-capsule-<timestamp>.json
```

The capsule keeps only non-secret evidence: proof hash, backup hash, source and
target table/row counts, conversion notes/stats, and the redacted-field list.
It does not prove that Cloud SQL imported the data; it only proves the local
SQLite-to-Postgres parity preflight.

To include the capsule in GitHub readiness:

```bash
CAPSULE_B64="$(base64 < outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-capsule-<timestamp>.json | tr -d '\n')"
gh workflow run gcp-readiness.yml \
  --repo living-ip/teleo-infrastructure \
  --ref main \
  -f restore_canary_capsule_b64="${CAPSULE_B64}"
```

## Legacy SQLite Cloud SQL Restore Drill Runner

Prepare the exact GCS import and Cloud SQL import operation without mutating GCP:

```bash
SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-<timestamp>.db.gz \
  ops/run_gcp_cloudsql_restore_drill.sh
```

Execute it only from an authenticated operator environment that can write the
versioned backup bucket and administer the standby Cloud SQL instance:

```bash
EXECUTE=1 \
SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-<timestamp>.db.gz \
  ops/run_gcp_cloudsql_restore_drill.sh
```

The runner:

- regenerates the explicit PostgreSQL import script;
- targets the shadow schema `teleo_restore` inside `teleo_kb`;
- uploads the import script to `gs://teleo-501523-prod-backups/kb-dumps/cloudsql-restore-drills/...` when `EXECUTE=1`;
- starts and waits for `gcloud sql import sql`;
- writes `target-counts.sql` for the required trusted VPC/Cloud SQL connector query readback.

The import operation alone is still not the final proof. The final proof needs
`target-counts.sql` run against `teleo-pgvector-standby` and compared to the
source counts in the drill proof.

After the import operation is `DONE`, run the generated count query from a
trusted VPC runtime or Cloud SQL connector path and retain CSV output:

```bash
psql "$TELEO_CLOUDSQL_DATABASE_URL" \
  --csv \
  -f outputs/gcp-infra-hardening-20260707/private-cloudsql-restore-drills/gcp-cloudsql-restore-drill-<timestamp>/target-counts.sql \
  > outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-target-counts-<timestamp>.csv
```

Then compare the Cloud SQL readback to the source proof:

```bash
python3 ops/verify_gcp_cloudsql_restore_readback.py \
  --drill-proof outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-restore-drill-<timestamp>.json \
  --target-counts-csv outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-target-counts-<timestamp>.csv \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-restore-readback-verification-<timestamp>.json
```

Only a `status = pass` verifier output is enough for row-count parity. It still
does not prove application cutover or continuous replication.

## Required Proof

A successful restore or replication canary must retain:

- source dataset identity:
  - source host or dump artifact;
  - dump timestamp or replication slot timestamp;
  - source schema/database name.
- transfer proof:
  - for the canonical path, the v2 capture receipt, dump SHA-256, and exact
    private staging path used by streaming `pg_restore`;
  - for a legacy import or replication path, a versioned dump-object generation
    or logical replication subscription details;
  - row/table counts before import where available.
- target proof:
  - `teleo-pgvector-standby` control-plane readback;
  - generated `teleo_clone_*` identity for canonical parity, or `teleo_kb` only
    for the explicitly legacy SQLite drill;
  - extension readback for `vector` if the restored schema needs pgvector;
  - representative query readback for core KB tables.
- failure boundary:
  - exact missing secret, source access, schema incompatibility, extension issue, or import error.

## One-Shot SQLite Export / GCP Restore Path

Use this while the canonical DB remains SQLite on the VPS and we need a GCP restore drill.

1. Run `ops/backup_vps_sqlite_kb.sh`.
2. Upload the resulting SQLite backup and Leo/KB tarball to a versioned GCS bucket such as `gs://teleo-501523-prod-backups/kb-dumps/`.
3. Run the local SQLite-to-Postgres restore canary above and retain its proof.
4. Run `ops/run_gcp_cloudsql_restore_drill.sh` in dry-run mode to generate the GCS import plan.
5. Run `EXECUTE=1 ops/run_gcp_cloudsql_restore_drill.sh` from an authenticated operator environment to upload and import the generated SQL. Do not run blind string rewrites against the SQLite dump.
6. Install required extensions on Cloud SQL:

```sql
create extension if not exists vector;
```

7. From a trusted VPC runtime or Cloud SQL connector path, run readbacks:

```sql
select current_database();
select extname, extversion from pg_extension where extname = 'vector';
select schemaname, tablename from pg_tables where schemaname not in ('pg_catalog', 'information_schema') order by 1, 2 limit 50;
```

8. Retain the SQLite backup hash, GCS object generation, import/conversion operation, query output, and row-count sample.
9. Run `ops/verify_gcp_cloudsql_restore_readback.py` and retain a passing parity proof.

## Logical Replication Path

Use this only if the canonical source becomes Postgres or a Postgres mirror exists. SQLite cannot be logically replicated into Cloud SQL Postgres without an intermediate conversion/sync layer.

Required source privileges:

- replication-capable source user;
- publication over the intended schemas/tables;
- network path from GCP to source, or source-to-GCP path through an approved proxy/tunnel.

Required target steps:

```sql
create extension if not exists vector;
create subscription <subscription_name>
connection '<redacted source connection string>'
publication <publication_name>;
```

Retain only redacted connection metadata. Do not commit or paste credentials.

## Current Blocker

As of 2026-07-15, a newest authorized canonical Postgres capture and isolated
local restore can pass, but the required T3 live private GCP lifecycle cannot be
started from the current operator route:

- direct VM SSH and the private TCP route timed out;
- local `gcloud` for `billy@livingip.xyz` requires reauthentication; no
  password was entered or inspected;
- the intended operator WIF/IAP authority is unavailable because provider
  `teleo-iap-operator` is absent, disabled, or deleted;
- the working artifact-builder WIF identity is intentionally limited and must
  not be expanded or mistaken for Cloud SQL/Compute operator access.

The authorized recovery is a local
`gcloud auth login billy@livingip.xyz --force` without sharing credentials, or
administrator convergence of the reviewed operator WIF/IAP bootstrap. Until a
route passes live readback, keep the claim at T2/source-local proof; do not
promote staging or expose Cloud SQL publicly.

Once access is restored, run: newest authorized `teleo` snapshot -> disposable
private `teleo_clone_*` restore -> generated connectivity proof ->
`gcp-parity.json` -> no-send m3taversal blind reasoning -> live GCE reasoning
attestation -> receipt-bound clone cleanup -> postflight VPS capture and delta
-> eight-receipt lifecycle verdict within the 900/3600-second freshness bounds.