Add GCP Cloud SQL standby readiness (#41)

docs/gcp-infra-hardening-20260707.md

@@ -26,6 +26,14 @@ This records the current GCP hardening contract for `teleo-501523`.
 - VM boot disks have a daily 7-day snapshot policy:
   - `teleo-prod-1`
   - `teleo-staging-1`
+- Cloud SQL standby target exists for KB restore/replication drills:
+  - instance: `teleo-pgvector-standby`
+  - database: `teleo_kb`
+  - version: `POSTGRES_16`
+  - private IP only on `teleo-staging-net`
+  - encrypted-only SQL connections
+  - automated backups and point-in-time recovery enabled
+  - deletion protection enabled
 
 ## How To Build
 
@@ -76,20 +84,21 @@ The check is read-only and prints no secret values. It verifies:
 - Runtime service-account posture for the prod/staging VMs.
 - Compute disk snapshot policy attachment.
 - Backup bucket versioning and uniform access.
-- Whether Cloud SQL/Postgres exists.
-- Whether a GCP-approved KB/Postgres access secret slot exists.
+- Cloud SQL standby target posture.
+- Whether an approved source KB/Postgres dump or replication credential exists.
+- Whether source data has actually been restored or replicated into GCP and queried.
 
 ## Current Boundaries
 
 The GCP Docker build/publish path is live through manual Cloud Build and GitHub Actions Workload Identity Federation. Native Cloud Build GitHub triggers are not configured because this project currently has no Cloud Build repository connection.
 
-Database redundancy is not complete. The current project has backup buckets and VM disk snapshots, but it does not yet have a GCP Cloud SQL/Postgres pgvector standby or primary. Do not claim DB parity until one of these is true:
+Database redundancy is not complete. The current project now has a GCP Cloud SQL/Postgres standby target, backup buckets, and VM disk snapshots. It does not yet have source-data restore or replication from the canonical KB. Do not claim DB parity until one of these is true:
 
 - the existing canonical KB database is replicated into GCP and read back; or
 - GCP Cloud SQL/Postgres becomes the canonical database and production services read/write it; or
 - an explicitly approved standby restore drill proves that a GCP database can be restored and queried from the retained backups.
 
-Do not create an empty Cloud SQL instance and call it redundancy. It only counts after source data, restore/replication, access controls, and readback are proven.
+Do not call the empty `teleo-pgvector-standby` instance redundancy by itself. It only counts after source data, restore/replication, access controls, and query readback are proven.
 
 ## Communication Posture
 

docs/gcp-kb-cloudsql-restore-runbook.md

@@ -0,0 +1,91 @@
+# KB Restore / Replication Runbook
+
+This runbook is for proving Living IP KB/database redundancy on GCP.
+
+The standby target already exists:
+
+- project: `teleo-501523`
+- instance: `teleo-pgvector-standby`
+- database: `teleo_kb`
+- region: `europe-west6`
+- network: `teleo-staging-net`
+- private IP: `10.61.0.3`
+- admin password secret: `gcp-teleo-pgvector-standby-postgres-password`
+
+Do not call this redundancy complete until source data has been restored or replicated and queried from GCP.
+
+## Required Proof
+
+A successful restore or replication canary must retain:
+
+- source dataset identity:
+  - source host or dump artifact;
+  - dump timestamp or replication slot timestamp;
+  - source schema/database name.
+- transfer proof:
+  - dump object path in a versioned bucket, or logical replication subscription details;
+  - row/table counts before import where available.
+- target proof:
+  - `teleo-pgvector-standby` readback;
+  - `teleo_kb` database readback;
+  - extension readback for `vector` if the restored schema needs pgvector;
+  - representative query readback for core KB tables.
+- failure boundary:
+  - exact missing secret, source access, schema incompatibility, extension issue, or import error.
+
+## One-Shot Dump / Restore Path
+
+Use this when the canonical DB remains outside GCP and we need a restore drill.
+
+1. Export from source with a least-privilege read-only source credential.
+2. Upload the dump to a versioned GCS bucket such as `gs://teleo-501523-prod-backups/kb-dumps/`.
+3. Import into Cloud SQL:
+
+```bash
+gcloud sql import sql teleo-pgvector-standby \
+  gs://teleo-501523-prod-backups/kb-dumps/<dump-file>.sql.gz \
+  --project=teleo-501523 \
+  --database=teleo_kb
+```
+
+4. From a trusted VPC runtime or Cloud SQL connector path, run readbacks:
+
+```sql
+select current_database();
+select extname, extversion from pg_extension where extname = 'vector';
+select schemaname, tablename from pg_tables where schemaname not in ('pg_catalog', 'information_schema') order by 1, 2 limit 50;
+```
+
+5. Retain the dump object generation, import operation, query output, and row-count sample.
+
+## Logical Replication Path
+
+Use this only after the source database owner approves replication permissions.
+
+Required source privileges:
+
+- replication-capable source user;
+- publication over the intended schemas/tables;
+- network path from GCP to source, or source-to-GCP path through an approved proxy/tunnel.
+
+Required target steps:
+
+```sql
+create extension if not exists vector;
+create subscription <subscription_name>
+connection '<redacted source connection string>'
+publication <publication_name>;
+```
+
+Retain only redacted connection metadata. Do not commit or paste credentials.
+
+## Current Blocker
+
+As of this run, GCP has the standby target but does not have an approved source KB/Postgres dump, source database credential, SSH tunnel credential, or logical replication credential. That is why the readiness checker still reports:
+
+- `kb_source_restore_access = blocked`
+- `kb_restore_or_replication = blocked`
+
+The next real canary is:
+
+source KB dump or replication credential -> import/replicate into `teleo_kb` -> run representative KB queries on GCP -> retain proof.

ops/check_gcp_infra_readiness.py

@@ -40,6 +40,9 @@ GITHUB_ARTIFACT_SERVICE_ACCOUNT = f"sa-artifact-builder@{PROJECT}.iam.gserviceac
 GITHUB_WIF_POOL = "github-actions"
 GITHUB_WIF_PROVIDER = "living-ip-github"
 GITHUB_REPOSITORY = "living-ip/teleo-infrastructure"
+CLOUDSQL_STANDBY_INSTANCE = "teleo-pgvector-standby"
+CLOUDSQL_STANDBY_DATABASE = "teleo_kb"
+CLOUDSQL_STANDBY_PASSWORD_SECRET = "gcp-teleo-pgvector-standby-postgres-password"
 
 
 @dataclass
@@ -379,43 +382,148 @@ def check_backup_buckets() -> Check:
     return Check("backup_buckets", "pass", " ".join(details))
 
 
-def check_cloud_sql() -> Check:
-    instances = run_json(
+def check_cloud_sql_standby() -> Check:
+    instance = run_json(
         [
             "gcloud",
             "sql",
             "instances",
-            "list",
+            "describe",
+            CLOUDSQL_STANDBY_INSTANCE,
             "--project",
             PROJECT,
             "--format=json",
         ]
     )
-    if not instances:
-        return Check(
-            "cloud_sql_pgvector",
-            "blocked",
-            "no Cloud SQL/Postgres instance exists in GCP; canonical KB still lives outside this GCP project",
-            required_tier="T3_live_readonly",
-            current_tier="T0_spec",
+    problems = []
+    if instance.get("databaseVersion") != "POSTGRES_16":
+        problems.append(f"databaseVersion={instance.get('databaseVersion')}")
+    if instance.get("state") != "RUNNABLE":
+        problems.append(f"state={instance.get('state')}")
+
+    settings = instance.get("settings", {})
+    if settings.get("deletionProtectionEnabled") is not True:
+        problems.append("deletionProtectionEnabled=false")
+
+    backup = settings.get("backupConfiguration", {})
+    if backup.get("enabled") is not True:
+        problems.append("backup_disabled")
+    if backup.get("pointInTimeRecoveryEnabled") is not True:
+        problems.append("pitr_disabled")
+    retained = int(backup.get("backupRetentionSettings", {}).get("retainedBackups", 0) or 0)
+    if retained < 7:
+        problems.append(f"retainedBackups={retained}")
+    log_days = int(backup.get("transactionLogRetentionDays", 0) or 0)
+    if log_days < 7:
+        problems.append(f"transactionLogRetentionDays={log_days}")
+
+    ip_config = settings.get("ipConfiguration", {})
+    if ip_config.get("ipv4Enabled") is not False:
+        problems.append("public_ipv4_enabled")
+    if ip_config.get("sslMode") != "ENCRYPTED_ONLY":
+        problems.append(f"sslMode={ip_config.get('sslMode')}")
+    if ip_config.get("privateNetwork") != f"projects/{PROJECT}/global/networks/teleo-staging-net":
+        problems.append(f"privateNetwork={ip_config.get('privateNetwork')}")
+
+    private_addresses = [
+        address.get("ipAddress")
+        for address in instance.get("ipAddresses", [])
+        if address.get("type") == "PRIVATE"
+    ]
+    public_addresses = [
+        address.get("ipAddress")
+        for address in instance.get("ipAddresses", [])
+        if address.get("type") in {"PRIMARY", "OUTGOING"}
+    ]
+    if not private_addresses:
+        problems.append("missing_private_ip")
+    if public_addresses:
+        problems.append(f"public_addresses={public_addresses}")
+
+    flags = {flag.get("name"): flag.get("value") for flag in settings.get("databaseFlags", [])}
+    if flags.get("cloudsql.iam_authentication") != "on":
+        problems.append("cloudsql.iam_authentication_not_on")
+
+    databases = run_json(
+        [
+            "gcloud",
+            "sql",
+            "databases",
+            "list",
+            "--instance",
+            CLOUDSQL_STANDBY_INSTANCE,
+            "--project",
+            PROJECT,
+            "--format=json",
+        ]
+    )
+    database_names = {database.get("name") for database in databases}
+    if CLOUDSQL_STANDBY_DATABASE not in database_names:
+        problems.append(f"missing_database={CLOUDSQL_STANDBY_DATABASE}")
+
+    secret = run_json(
+        [
+            "gcloud",
+            "secrets",
+            "describe",
+            CLOUDSQL_STANDBY_PASSWORD_SECRET,
+            "--project",
+            PROJECT,
+            "--format=json",
+        ]
+    )
+    versions = run_text(
+        [
+            "gcloud",
+            "secrets",
+            "versions",
+            "list",
+            CLOUDSQL_STANDBY_PASSWORD_SECRET,
+            "--project",
+            PROJECT,
+            "--filter=state:ENABLED",
+            "--format=value(name)",
+        ]
+    )
+    if not versions:
+        problems.append(f"secret_has_no_enabled_versions={secret.get('name')}")
+
+    if problems:
+        return Check("cloud_sql_standby_target", "fail", " ".join(problems))
+    return Check(
+        "cloud_sql_standby_target",
+        "pass",
+        f"instance={CLOUDSQL_STANDBY_INSTANCE} database={CLOUDSQL_STANDBY_DATABASE} private_ip={','.join(private_addresses)} backups=7 pitr=7d ssl=encrypted_only",
     )
-    names = ",".join(instance["name"] for instance in instances)
-    return Check("cloud_sql_pgvector", "pass", f"instances={names}")
 
 
-def check_kb_credential_slot() -> Check:
+def check_kb_source_restore_access() -> Check:
     secrets = run_json(["gcloud", "secrets", "list", "--project", PROJECT, "--format=json"])
     names = {secret["name"].split("/")[-1] for secret in secrets}
-    candidates = sorted(name for name in names if any(token in name.lower() for token in ("kb", "postgres", "pg", "ssh")))
-    if not candidates:
-        return Check(
-            "kb_gcp_access_secret",
-            "blocked",
-            f"no KB/Postgres/SSH secret slot found for approved GCP-to-KB access; current blocker remains {KB_HOST} auth",
-            required_tier="T3_live_readonly",
-            current_tier="T0_spec",
+    source_candidates = sorted(
+        name
+        for name in names
+        if any(token in name.lower() for token in ("source-kb", "kb-source", "source-postgres", "hetzner", "vps-db", "restore-source"))
+    )
+    if not source_candidates:
+        return Check(
+            "kb_source_restore_access",
+            "blocked",
+            f"no approved source KB/Postgres dump or replication credential found for restoring {KB_HOST} into GCP",
+            required_tier="T4_restore_or_replication_readback",
+            current_tier="T1_foundation",
+        )
+    return Check("kb_source_restore_access", "pass", "candidate_secret_names=" + ",".join(source_candidates))
+
+
+def check_kb_restore_or_replication() -> Check:
+    return Check(
+        "kb_restore_or_replication",
+        "blocked",
+        "GCP standby target exists, but no source-data restore, pg_dump import, logical replication, or query readback has been retained",
+        required_tier="T4_restore_or_replication_readback",
+        current_tier="T1_foundation",
     )
-    return Check("kb_gcp_access_secret", "pass", "candidate_secret_names=" + ",".join(candidates))
 
 
 def main() -> int:
@@ -428,8 +536,9 @@ def main() -> int:
         safe_check("compute_runtime_service_accounts", check_compute_runtime_service_accounts),
         safe_check("compute_disk_snapshots", check_snapshot_policy),
         safe_check("backup_buckets", check_backup_buckets),
-        safe_check("cloud_sql_pgvector", check_cloud_sql),
-        safe_check("kb_gcp_access_secret", check_kb_credential_slot),
+        safe_check("cloud_sql_standby_target", check_cloud_sql_standby),
+        safe_check("kb_source_restore_access", check_kb_source_restore_access),
+        safe_check("kb_restore_or_replication", check_kb_restore_or_replication),
     ]
     payload = {
         "project": PROJECT,