teleo/teleo-infrastructure
8
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions 1
teleo-infrastructure/docs/gcp-infra-hardening-20260707.md
twentyOne2x 8a935b4625
Some checks are pending
CI / lint-and-test (push) Waiting to run
Details
Add portable restore canary capsule (#55)
2026-07-07 14:22:03 +02:00

12 KiB
Raw Blame History

GCP CI/CD and Redundancy Hardening

This records the current GCP hardening contract for teleo-501523.

Current-State Rule

The latest retained gcp-readiness artifact is authoritative for what is live right now. The resource lists below define the target runtime contract and the checks this lane enforces; do not treat them as current production proof unless the current readiness run passes.

Target Runtime Contract

  • Artifact Registry Docker repositories exist in europe-west6:
    • teleo
    • livingip-web
  • Both repositories use immutable tags and active vulnerability scanning.
  • cloudbuild.gcp-staging.yaml builds the staging Teleo image, runs the image smoke test, then pushes to Artifact Registry.
  • Cloud Build runs as the dedicated service account:
    • sa-teleo-cloudbuild@teleo-501523.iam.gserviceaccount.com
  • The dedicated Cloud Build account has only the roles required for the current build/publish path:
    • roles/artifactregistry.writer
    • roles/logging.logWriter
    • roles/storage.objectViewer
  • GitHub Actions can publish Artifact Registry images through Workload Identity Federation:
    • workflow: .github/workflows/gcp-artifact.yml
    • provider: projects/785938879453/locations/global/workloadIdentityPools/github-actions/providers/living-ip-github
    • service account: sa-artifact-builder@teleo-501523.iam.gserviceaccount.com
    • repository scope: living-ip/teleo-infrastructure
  • Backup buckets are versioned and use uniform bucket-level access:
    • gs://teleo-501523-prod-backups
    • gs://teleo-501523-leoclean-backups
  • VM boot disks have a daily 7-day snapshot policy:
    • teleo-prod-1
    • teleo-staging-1
  • Cloud SQL standby target exists for KB restore/replication drills:
    • instance: teleo-pgvector-standby
    • database: teleo_kb
    • version: POSTGRES_16
    • private IP only on teleo-staging-net
    • encrypted-only SQL connections
    • automated backups and point-in-time recovery enabled
    • deletion protection enabled
  • Source-side Teleo DB/KB export canary exists:
    • script: ops/backup_vps_sqlite_kb.sh
    • source DB: /opt/teleo-eval/pipeline/pipeline.db
    • source Leo/KB files: workspaces/*/agents/leo plus agent-state
    • excludes secrets and logs
  • Local SQLite-to-Postgres restore canary exists:
    • scripts: ops/sqlite_to_postgres_dump.py and ops/run_sqlite_postgres_restore_canary.sh
    • target: disposable local postgres:16-alpine shadow schema
    • verifies source SQLite integrity and per-table source/target row-count parity

How To Build

Automatic Artifact Registry publishing runs on pushes to main through GitHub Actions. To run the same lane manually from GitHub:

gh workflow run gcp-artifact.yml --repo living-ip/teleo-infrastructure --ref main

The workflow authenticates to GCP with Workload Identity Federation, builds Dockerfile.gcp-staging, runs the image smoke test, pushes the image, and uploads gcp-artifact-image.txt as a run artifact.

For a read-only GCP posture probe through the same Workload Identity path:

gh workflow run gcp-readiness.yml --repo living-ip/teleo-infrastructure --ref main

This workflow runs ops/check_gcp_infra_readiness.py from GitHub Actions and uploads stdout, stderr, exit code, and a summary as the gcp-readiness artifact. It is intentionally non-mutating and defaults to sa-teleo-readiness@teleo-501523.iam.gserviceaccount.com.

If you need to test a specific service account during IAM repair, pass it explicitly:

gh workflow run gcp-readiness.yml \
  --repo living-ip/teleo-infrastructure \
  --ref main \
  -f service_account=sa-teleo-readiness@teleo-501523.iam.gserviceaccount.com

If you also want GitHub readiness to include a local SQLite-to-Postgres restore canary proof without uploading private backup paths or generated SQL, pass a redacted capsule:

python3 ops/redact_sqlite_postgres_restore_canary.py \
  --proof outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-<timestamp>.json \
  --output outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-capsule-<timestamp>.json

CAPSULE_B64="$(base64 < outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-capsule-<timestamp>.json | tr -d '\n')"
gh workflow run gcp-readiness.yml \
  --repo living-ip/teleo-infrastructure \
  --ref main \
  -f restore_canary_capsule_b64="${CAPSULE_B64}"

This only upgrades the local restore-preflight row. It is not GCP DB redundancy until the Cloud SQL import and target-count readback also pass.

For a local/manual Cloud Build proof:

REVISION="$(git rev-parse HEAD)"
TAG="$(git rev-parse --short=7 HEAD)-manual-$(date -u +%Y%m%d-%H%M%S)"

gcloud builds submit \
  --project=teleo-501523 \
  --config=cloudbuild.gcp-staging.yaml \
  --substitutions="_TAG=${TAG},_REVISION=${REVISION}"

Expected result:

  • build-staging-image succeeds.
  • smoke-test-image-before-push succeeds.
  • A Docker image is pushed to: europe-west6-docker.pkg.dev/teleo-501523/teleo/teleo-pipeline-gcp-staging:${TAG}

How To Check Readiness

Run from the repository root:

python3 ops/check_gcp_infra_readiness.py

The check is read-only and prints no secret values. It verifies:

  • Artifact Registry immutability and vulnerability scanning.
  • Cloud Build config contract.
  • Dedicated Cloud Build service account and roles.
  • GitHub Actions WIF Artifact Registry publishing contract.
  • Network ingress posture:
    • no enabled broad SSH/RDP ingress;
    • Teleo SSH rules are scoped to /32 source ranges and target tags.
  • Runtime service-account posture for the prod/staging VMs.
  • Compute disk snapshot policy attachment.
  • Backup bucket versioning and uniform access.
  • Cloud SQL standby target posture.
  • Source SQLite/KB backup/export repeatability.
  • Whether an approved source KB/Postgres dump or replication credential exists.
  • Whether source data has actually been restored or replicated into GCP and queried.
  • Whether the GitHub WIF readiness workflow exists for non-local readback.

Current Boundaries

The GCP Docker build/publish path is live through manual Cloud Build and GitHub Actions Workload Identity Federation. Native Cloud Build GitHub triggers are not configured because this project currently has no Cloud Build repository connection.

Database redundancy is not complete. The current project now has a GCP Cloud SQL/Postgres standby target, backup buckets, VM disk snapshots, and a repeatable source SQLite/KB export script. It does not yet have source-data restore or replication into GCP. Do not claim DB parity until one of these is true:

  • the existing canonical KB database is replicated into GCP and read back; or
  • GCP Cloud SQL/Postgres becomes the canonical database and production services read/write it; or
  • an explicitly approved standby restore drill proves that a GCP database can be restored and queried from the retained backups.

Do not call the empty teleo-pgvector-standby instance redundancy by itself. It only counts after source data, restore/replication, access controls, and query readback are proven.

The local restore canary narrows the remaining gap: the source SQLite backup can be converted and restored into PostgreSQL with table/row-count parity, but the same import still needs to run against the GCP Cloud SQL standby through an approved GCP auth and network path.

After Cloud SQL import, use the generated target-counts.sql and verify it with:

python3 ops/verify_gcp_cloudsql_restore_readback.py \
  --drill-proof outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-restore-drill-<timestamp>.json \
  --target-counts-csv outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-target-counts-<timestamp>.csv \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-restore-readback-verification-<timestamp>.json

The verifier must return status = pass before claiming row-count parity in GCP.

IAM Split Plan

Do not make sa-artifact-builder@teleo-501523.iam.gserviceaccount.com the broad infra account. Keep it scoped to Docker image publishing.

Use the generated plan for the next privilege boundary:

python3 ops/plan_gcp_iam_split.py --format json
python3 ops/plan_gcp_iam_split.py --format shell

For an idempotent retained apply attempt, use:

python3 ops/apply_gcp_iam_split.py \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-iam-split-apply-dry-run.json

python3 ops/apply_gcp_iam_split.py \
  --execute \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-iam-split-apply-execute.json

The first command is dry-run only. The second command mutates IAM and must run from an authenticated GCP admin shell. It is safe to re-run: existing service accounts are skipped, and IAM binding commands are additive/idempotent.

The plan creates two separate accounts:

  • sa-teleo-readiness@teleo-501523.iam.gserviceaccount.com
    • GitHub WIF account for read-only readiness checks.
    • Needs read-only roles for Artifact Registry, IAM/service-account/WIF metadata, Compute/network posture, Cloud SQL metadata, backup buckets, and Secret Manager metadata.
  • sa-teleo-restore-drill@teleo-501523.iam.gserviceaccount.com
    • Operator account for explicit Cloud SQL restore drills.
    • Needs Cloud SQL edit rights for the import operation and object-admin access to the restore bucket path.

Cloud SQL imports also require the Cloud SQL instance service account to read the GCS object. The plan includes a command that discovers that instance service account and grants it roles/storage.objectAdmin on gs://teleo-501523-prod-backups.

This plan is not itself a completed redundancy proof. DB redundancy is complete only after the restore drill imports source data into Cloud SQL and the retained target-counts.sql readback matches the source/local restore proof.

Runtime Baseline Runner

Do not create or repair the GCP runtime baseline manually in the console if an audited runner can do it. The runtime baseline runner is dry-run by default:

python3 ops/apply_gcp_runtime_baseline.py \
  --admin-ssh-cidr <operator-ip>/32 \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-runtime-baseline-dry-run.json

The dry-run proof records the exact service accounts, network, firewall, VM, snapshot, backup bucket, secret, and Cloud SQL operations needed for the readiness checker. It does not prove that those resources exist.

To apply after an authenticated GCP admin session is available:

export TELEO_CLOUDSQL_POSTGRES_PASSWORD='<store locally; do not commit or print>'
python3 ops/apply_gcp_runtime_baseline.py \
  --execute \
  --admin-ssh-cidr <operator-ip>/32 \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-runtime-baseline-execute.json

--execute refuses to run without a single trusted IPv4 /32 SSH CIDR. The Cloud SQL password is passed through the environment and redacted from retained operation commands. After execute mode succeeds, rerun gcp-readiness.yml with the dedicated readiness service account and then run the Cloud SQL restore drill/readback verifier before claiming database redundancy.

Communication Posture

The service-to-service communication contract is declared in:

  • config/gcp-service-communications.json

Validate it locally or in CI with:

python3 ops/check_gcp_service_communications.py \
  --contract config/gcp-service-communications.json \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-service-communications-check.json

The contract currently requires:

  • GitHub Actions artifact publishing only through sa-artifact-builder.
  • GitHub Actions readiness only through sa-teleo-readiness.
  • emergency SSH only from one operator IPv4 /32 to Teleo target tags.
  • VM image pulls over Private Google Access.
  • Cloud SQL only on private VPC paths with encrypted-only PostgreSQL.
  • Cloud SQL imports only from the versioned backup bucket through approved service accounts.
  • no broad SSH/RDP, no public database IP, no default Compute Engine service accounts, and no raw secret values in the contract.

This is still a contract until GCP readiness passes. Live proof requires the current gcp-readiness artifact to show the matching firewall, VM service accounts, bucket, and Cloud SQL checks passing.