Add VPS SQLite KB backup canary (#42)

docs/gcp-infra-hardening-20260707.md

@@ -34,6 +34,11 @@ This records the current GCP hardening contract for `teleo-501523`.
   - encrypted-only SQL connections
   - automated backups and point-in-time recovery enabled
   - deletion protection enabled
+- Source-side Teleo DB/KB export canary exists:
+  - script: `ops/backup_vps_sqlite_kb.sh`
+  - source DB: `/opt/teleo-eval/pipeline/pipeline.db`
+  - source Leo/KB files: `workspaces/*/agents/leo` plus `agent-state`
+  - excludes secrets and logs
 
 ## How To Build
 
@@ -85,6 +90,7 @@ The check is read-only and prints no secret values. It verifies:
 - Compute disk snapshot policy attachment.
 - Backup bucket versioning and uniform access.
 - Cloud SQL standby target posture.
+- Source SQLite/KB backup/export repeatability.
 - Whether an approved source KB/Postgres dump or replication credential exists.
 - Whether source data has actually been restored or replicated into GCP and queried.
 
@@ -92,7 +98,7 @@ The check is read-only and prints no secret values. It verifies:
 
 The GCP Docker build/publish path is live through manual Cloud Build and GitHub Actions Workload Identity Federation. Native Cloud Build GitHub triggers are not configured because this project currently has no Cloud Build repository connection.
 
-Database redundancy is not complete. The current project now has a GCP Cloud SQL/Postgres standby target, backup buckets, and VM disk snapshots. It does not yet have source-data restore or replication from the canonical KB. Do not claim DB parity until one of these is true:
+Database redundancy is not complete. The current project now has a GCP Cloud SQL/Postgres standby target, backup buckets, VM disk snapshots, and a repeatable source SQLite/KB export script. It does not yet have source-data restore or replication into GCP. Do not claim DB parity until one of these is true:
 
 - the existing canonical KB database is replicated into GCP and read back; or
 - GCP Cloud SQL/Postgres becomes the canonical database and production services read/write it; or

docs/gcp-kb-cloudsql-restore-runbook.md

@@ -2,6 +2,15 @@
 
 This runbook is for proving Living IP KB/database redundancy on GCP.
 
+Current source reality:
+
+- canonical runtime DB: `/opt/teleo-eval/pipeline/pipeline.db`
+- engine: SQLite WAL
+- canonical Leo files:
+  - `/opt/teleo-eval/workspaces/main/agents/leo`
+  - `/opt/teleo-eval/workspaces/research-leo/agents/leo`
+  - `/opt/teleo-eval/agent-state`
+
 The standby target already exists:
 
 - project: `teleo-501523`
@@ -14,6 +23,26 @@ The standby target already exists:
 
 Do not call this redundancy complete until source data has been restored or replicated and queried from GCP.
 
+## Source Backup Canary
+
+Create a consistent source backup without stopping the VPS service:
+
+```bash
+ops/backup_vps_sqlite_kb.sh
+```
+
+The script:
+
+- uses SQLite `.backup` against `/opt/teleo-eval/pipeline/pipeline.db`;
+- compresses and hashes the backup on the VPS;
+- archives Leo/KB files while excluding `secrets` and logs;
+- copies both artifacts locally;
+- verifies SHA-256 matches;
+- runs `PRAGMA integrity_check` on a local restored SQLite copy;
+- records proof under `outputs/gcp-infra-hardening-20260707/proofs/`.
+
+This proves source exportability and local restore integrity. It does not prove GCP DB redundancy until a GCP restore/import/query canary also passes.
+
 ## Required Proof
 
 A successful restore or replication canary must retain:
@@ -33,22 +62,20 @@ A successful restore or replication canary must retain:
 - failure boundary:
   - exact missing secret, source access, schema incompatibility, extension issue, or import error.
 
-## One-Shot Dump / Restore Path
+## One-Shot SQLite Export / GCP Restore Path
 
-Use this when the canonical DB remains outside GCP and we need a restore drill.
+Use this while the canonical DB remains SQLite on the VPS and we need a GCP restore drill.
 
-1. Export from source with a least-privilege read-only source credential.
+1. Run `ops/backup_vps_sqlite_kb.sh`.
-2. Upload the dump to a versioned GCS bucket such as `gs://teleo-501523-prod-backups/kb-dumps/`.
+2. Upload the resulting SQLite backup and Leo/KB tarball to a versioned GCS bucket such as `gs://teleo-501523-prod-backups/kb-dumps/`.
-3. Import into Cloud SQL:
+3. Convert or replay the SQLite data into Cloud SQL with an explicit migration script. Do not run blind string rewrites against the SQLite dump.
+4. Install required extensions on Cloud SQL:
 
-```bash
+```sql
-gcloud sql import sql teleo-pgvector-standby \
+create extension if not exists vector;
-  gs://teleo-501523-prod-backups/kb-dumps/<dump-file>.sql.gz \
-  --project=teleo-501523 \
-  --database=teleo_kb
 ```
 
-4. From a trusted VPC runtime or Cloud SQL connector path, run readbacks:
+5. From a trusted VPC runtime or Cloud SQL connector path, run readbacks:
 
 ```sql
 select current_database();
@@ -56,11 +83,11 @@ select extname, extversion from pg_extension where extname = 'vector';
 select schemaname, tablename from pg_tables where schemaname not in ('pg_catalog', 'information_schema') order by 1, 2 limit 50;
 ```
 
-5. Retain the dump object generation, import operation, query output, and row-count sample.
+6. Retain the SQLite backup hash, GCS object generation, import/conversion operation, query output, and row-count sample.
 
 ## Logical Replication Path
 
-Use this only after the source database owner approves replication permissions.
+Use this only if the canonical source becomes Postgres or a Postgres mirror exists. SQLite cannot be logically replicated into Cloud SQL Postgres without an intermediate conversion/sync layer.
 
 Required source privileges:
 
@@ -81,11 +108,11 @@ Retain only redacted connection metadata. Do not commit or paste credentials.
 
 ## Current Blocker
 
-As of this run, GCP has the standby target but does not have an approved source KB/Postgres dump, source database credential, SSH tunnel credential, or logical replication credential. That is why the readiness checker still reports:
+As of this run, GCP has the standby target and a repeatable SQLite/KB export script, but it does not have an approved GCP upload/restore credential in the current local session, nor a retained Cloud SQL import/conversion/query proof. That is why the readiness checker still reports:
 
 - `kb_source_restore_access = blocked`
 - `kb_restore_or_replication = blocked`
 
 The next real canary is:
 
-source KB dump or replication credential -> import/replicate into `teleo_kb` -> run representative KB queries on GCP -> retain proof.
+source SQLite/KB backup -> upload to versioned GCS bucket -> convert/import into `teleo_kb` -> install/verify `vector` if needed -> run representative KB queries on GCP -> retain proof.

ops/backup_vps_sqlite_kb.sh

@@ -0,0 +1,99 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Creates a consistent SQLite backup and a secrets-excluded Leo/KB file archive
+# from the Teleo VPS. It writes local proof files but does not upload to GCP.
+
+TELEO_VPS_HOST="${TELEO_VPS_HOST:-root@77.42.65.182}"
+TELEO_SSH_KEY="${TELEO_SSH_KEY:-$HOME/.ssh/livingip_hetzner_20260604_ed25519}"
+TELEO_REMOTE_ROOT="${TELEO_REMOTE_ROOT:-/opt/teleo-eval}"
+REMOTE_BACKUP_ROOT="${REMOTE_BACKUP_ROOT:-/opt/teleo-eval/backups}"
+LOCAL_BACKUP_DIR="${LOCAL_BACKUP_DIR:-./outputs/gcp-infra-hardening-20260707/private-backups}"
+PROOF_DIR="${PROOF_DIR:-./outputs/gcp-infra-hardening-20260707/proofs}"
+TS="${TS:-$(date -u +%Y%m%dT%H%M%SZ)}"
+
+SQLITE_REMOTE_DIR="${REMOTE_BACKUP_ROOT}/pipeline"
+SQLITE_BASE="teleo-pipeline-sqlite-${TS}"
+KB_REMOTE_DIR="${REMOTE_BACKUP_ROOT}/kb"
+KB_BASE="teleo-leo-kb-files-${TS}"
+
+ssh_base=(
+  ssh
+  -i "$TELEO_SSH_KEY"
+  -o BatchMode=yes
+  -o StrictHostKeyChecking=accept-new
+)
+
+scp_base=(
+  scp
+  -i "$TELEO_SSH_KEY"
+  -o BatchMode=yes
+  -o StrictHostKeyChecking=accept-new
+)
+
+sha256_file() {
+  if command -v shasum >/dev/null 2>&1; then
+    shasum -a 256 "$1" | awk '{print $1}'
+  else
+    sha256sum "$1" | awk '{print $1}'
+  fi
+}
+
+mkdir -p "$LOCAL_BACKUP_DIR" "$PROOF_DIR"
+chmod 700 "$LOCAL_BACKUP_DIR"
+
+"${ssh_base[@]}" "$TELEO_VPS_HOST" "set -euo pipefail
+mkdir -p '$SQLITE_REMOTE_DIR' '$KB_REMOTE_DIR'
+chmod 700 '$SQLITE_REMOTE_DIR' '$KB_REMOTE_DIR'
+sqlite3 '$TELEO_REMOTE_ROOT/pipeline/pipeline.db' \".backup '$SQLITE_REMOTE_DIR/${SQLITE_BASE}.db'\"
+gzip -f -9 '$SQLITE_REMOTE_DIR/${SQLITE_BASE}.db'
+sha256sum '$SQLITE_REMOTE_DIR/${SQLITE_BASE}.db.gz' > '$SQLITE_REMOTE_DIR/${SQLITE_BASE}.db.gz.sha256'
+cd '$TELEO_REMOTE_ROOT'
+tar --warning=no-file-changed --exclude='secrets' --exclude='*.log' --exclude='logs' -czf '$KB_REMOTE_DIR/${KB_BASE}.tar.gz' \\
+  workspaces/main/agents/leo \\
+  workspaces/research-leo/agents/leo \\
+  agent-state
+sha256sum '$KB_REMOTE_DIR/${KB_BASE}.tar.gz' > '$KB_REMOTE_DIR/${KB_BASE}.tar.gz.sha256'
+ls -lh '$SQLITE_REMOTE_DIR/${SQLITE_BASE}.db.gz' '$KB_REMOTE_DIR/${KB_BASE}.tar.gz'
+" > "$PROOF_DIR/vps-backup-create-${TS}.txt"
+
+"${scp_base[@]}" "$TELEO_VPS_HOST:${SQLITE_REMOTE_DIR}/${SQLITE_BASE}.db.gz" "$LOCAL_BACKUP_DIR/"
+"${scp_base[@]}" "$TELEO_VPS_HOST:${SQLITE_REMOTE_DIR}/${SQLITE_BASE}.db.gz.sha256" "$LOCAL_BACKUP_DIR/"
+"${scp_base[@]}" "$TELEO_VPS_HOST:${KB_REMOTE_DIR}/${KB_BASE}.tar.gz" "$LOCAL_BACKUP_DIR/"
+"${scp_base[@]}" "$TELEO_VPS_HOST:${KB_REMOTE_DIR}/${KB_BASE}.tar.gz.sha256" "$LOCAL_BACKUP_DIR/"
+chmod 600 \
+  "$LOCAL_BACKUP_DIR/${SQLITE_BASE}.db.gz" \
+  "$LOCAL_BACKUP_DIR/${SQLITE_BASE}.db.gz.sha256" \
+  "$LOCAL_BACKUP_DIR/${KB_BASE}.tar.gz" \
+  "$LOCAL_BACKUP_DIR/${KB_BASE}.tar.gz.sha256"
+
+sqlite_expected="$(awk '{print $1}' "$LOCAL_BACKUP_DIR/${SQLITE_BASE}.db.gz.sha256")"
+sqlite_actual="$(sha256_file "$LOCAL_BACKUP_DIR/${SQLITE_BASE}.db.gz")"
+kb_expected="$(awk '{print $1}' "$LOCAL_BACKUP_DIR/${KB_BASE}.tar.gz.sha256")"
+kb_actual="$(sha256_file "$LOCAL_BACKUP_DIR/${KB_BASE}.tar.gz")"
+
+test "$sqlite_expected" = "$sqlite_actual"
+test "$kb_expected" = "$kb_actual"
+
+tmpdir="$(mktemp -d)"
+trap 'rm -rf "$tmpdir"' EXIT
+gunzip -c "$LOCAL_BACKUP_DIR/${SQLITE_BASE}.db.gz" > "$tmpdir/pipeline.db"
+
+{
+  echo "sqlite_backup=$LOCAL_BACKUP_DIR/${SQLITE_BASE}.db.gz"
+  echo "sqlite_remote=$SQLITE_REMOTE_DIR/${SQLITE_BASE}.db.gz"
+  echo "sqlite_backup_size_bytes=$(wc -c < "$LOCAL_BACKUP_DIR/${SQLITE_BASE}.db.gz" | tr -d ' ')"
+  echo "sqlite_db_size_bytes=$(wc -c < "$tmpdir/pipeline.db" | tr -d ' ')"
+  echo "sqlite_sha256=$sqlite_actual"
+  echo "sqlite_sha256_matches_vps=true"
+  echo "sqlite_integrity_check=$(sqlite3 "$tmpdir/pipeline.db" 'PRAGMA integrity_check;')"
+  echo "sqlite_user_table_count=$(sqlite3 "$tmpdir/pipeline.db" "select count(*) from sqlite_master where type='table' and name not like 'sqlite_%';")"
+  echo "kb_backup=$LOCAL_BACKUP_DIR/${KB_BASE}.tar.gz"
+  echo "kb_remote=$KB_REMOTE_DIR/${KB_BASE}.tar.gz"
+  echo "kb_backup_size_bytes=$(wc -c < "$LOCAL_BACKUP_DIR/${KB_BASE}.tar.gz" | tr -d ' ')"
+  echo "kb_sha256=$kb_actual"
+  echo "kb_sha256_matches_vps=true"
+  echo "kb_file_count=$(tar -tzf "$LOCAL_BACKUP_DIR/${KB_BASE}.tar.gz" | wc -l | tr -d ' ')"
+} > "$PROOF_DIR/vps-sqlite-kb-backup-proof-${TS}.txt"
+
+cat "$PROOF_DIR/vps-sqlite-kb-backup-proof-${TS}.txt"