teleo-infrastructure/docs/gcp-infra-hardening-20260707.md

# GCP CI/CD and Redundancy Hardening

This records the current GCP hardening contract for `teleo-501523`.

## Current-State Rule

The latest retained `gcp-readiness` artifact is authoritative for what is live
right now. The resource lists below define the target runtime contract and the
checks this lane enforces; do not treat them as current production proof unless
the current readiness run passes.

## Target Runtime Contract

- Artifact Registry Docker repositories exist in `europe-west6`:
  - `teleo`
  - `livingip-web`
- Both repositories use immutable tags and active vulnerability scanning.
- `cloudbuild.gcp-staging.yaml` builds the staging Teleo image, runs the image smoke test, then pushes to Artifact Registry.
- Cloud Build runs as the dedicated service account:
  - `sa-teleo-cloudbuild@teleo-501523.iam.gserviceaccount.com`
- The dedicated Cloud Build account has only the roles required for the current build/publish path:
  - `roles/artifactregistry.writer`
  - `roles/logging.logWriter`
  - `roles/storage.objectViewer`
- GitHub Actions can publish Artifact Registry images through Workload Identity Federation:
  - workflow: `.github/workflows/gcp-artifact.yml`
  - provider: `projects/785938879453/locations/global/workloadIdentityPools/github-actions/providers/living-ip-github`
  - service account: `sa-artifact-builder@teleo-501523.iam.gserviceaccount.com`
  - repository scope: `living-ip/teleo-infrastructure`
- Backup buckets are versioned and use uniform bucket-level access:
  - `gs://teleo-501523-prod-backups`
  - `gs://teleo-501523-leoclean-backups`
- VM boot disks have a daily 7-day snapshot policy:
  - `teleo-prod-1`
  - `teleo-staging-1`
- Cloud SQL standby target exists for KB restore/replication drills:
  - instance: `teleo-pgvector-standby`
  - database: `teleo_kb`
  - version: `POSTGRES_16`
  - private IP only on `teleo-staging-net`
  - encrypted-only SQL connections
  - automated backups and point-in-time recovery enabled
  - deletion protection enabled
- Source-side Teleo DB/KB export canary exists:
  - script: `ops/backup_vps_sqlite_kb.sh`
  - source DB: `/opt/teleo-eval/pipeline/pipeline.db`
  - source Leo/KB files: `workspaces/*/agents/leo` plus `agent-state`
  - excludes secrets and logs
- Local SQLite-to-Postgres restore canary exists:
  - scripts: `ops/sqlite_to_postgres_dump.py` and `ops/run_sqlite_postgres_restore_canary.sh`
  - target: disposable local `postgres:16-alpine` shadow schema
  - verifies source SQLite integrity and per-table source/target row-count parity

## How To Build

Automatic Artifact Registry publishing runs on pushes to `main` through GitHub Actions. To run the same lane manually from GitHub:

```bash
gh workflow run gcp-artifact.yml --repo living-ip/teleo-infrastructure --ref main
```

The workflow authenticates to GCP with Workload Identity Federation, builds `Dockerfile.gcp-staging`, runs the image smoke test, pushes the image, and uploads `gcp-artifact-image.txt` as a run artifact.

For a read-only GCP posture probe through the same Workload Identity path:

```bash
gh workflow run gcp-readiness.yml --repo living-ip/teleo-infrastructure --ref main
```

This workflow runs `ops/check_gcp_infra_readiness.py` from GitHub Actions and
uploads stdout, stderr, exit code, and a summary as the `gcp-readiness` artifact.
It is intentionally non-mutating and defaults to
`sa-teleo-readiness@teleo-501523.iam.gserviceaccount.com`.

If you need to test a specific service account during IAM repair, pass it
explicitly:

```bash
gh workflow run gcp-readiness.yml \
  --repo living-ip/teleo-infrastructure \
  --ref main \
  -f service_account=sa-teleo-readiness@teleo-501523.iam.gserviceaccount.com
```

If you also want GitHub readiness to include a local SQLite-to-Postgres restore
canary proof without uploading private backup paths or generated SQL, pass a
redacted capsule:

```bash
python3 ops/redact_sqlite_postgres_restore_canary.py \
  --proof outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-<timestamp>.json \
  --output outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-capsule-<timestamp>.json

CAPSULE_B64="$(base64 < outputs/gcp-infra-hardening-20260707/proofs/sqlite-postgres-restore-canary-capsule-<timestamp>.json | tr -d '\n')"
gh workflow run gcp-readiness.yml \
  --repo living-ip/teleo-infrastructure \
  --ref main \
  -f restore_canary_capsule_b64="${CAPSULE_B64}"
```

This only upgrades the local restore-preflight row. It is not GCP DB redundancy
until the Cloud SQL import and target-count readback also pass.

For a local/manual Cloud Build proof:

```bash
REVISION="$(git rev-parse HEAD)"
TAG="$(git rev-parse --short=7 HEAD)-manual-$(date -u +%Y%m%d-%H%M%S)"

gcloud builds submit \
  --project=teleo-501523 \
  --config=cloudbuild.gcp-staging.yaml \
  --substitutions="_TAG=${TAG},_REVISION=${REVISION}"
```

Expected result:

- `build-staging-image` succeeds.
- `smoke-test-image-before-push` succeeds.
- A Docker image is pushed to:
  `europe-west6-docker.pkg.dev/teleo-501523/teleo/teleo-pipeline-gcp-staging:${TAG}`

## How To Check Readiness

Run from the repository root:

```bash
python3 ops/check_gcp_infra_readiness.py
```

The check is read-only and prints no secret values. It verifies:

- Artifact Registry immutability and vulnerability scanning.
- Cloud Build config contract.
- Dedicated Cloud Build service account and roles.
- GitHub Actions WIF Artifact Registry publishing contract.
- Network ingress posture:
  - no enabled broad SSH/RDP ingress;
  - Teleo SSH rules are scoped to `/32` source ranges and target tags.
- Runtime service-account posture for the prod/staging VMs.
- Compute disk snapshot policy attachment.
- Backup bucket versioning and uniform access.
- Cloud SQL standby target posture.
- Source SQLite/KB backup/export repeatability.
- Whether an approved source KB/Postgres dump or replication credential exists.
- Whether source data has actually been restored or replicated into GCP and queried.
- Whether the GitHub WIF readiness workflow exists for non-local readback.

## Current Boundaries

The GCP Docker build/publish path is live through manual Cloud Build and GitHub Actions Workload Identity Federation. Native Cloud Build GitHub triggers are not configured because this project currently has no Cloud Build repository connection.

Database redundancy is not complete. The current project now has a GCP Cloud SQL/Postgres standby target, backup buckets, VM disk snapshots, and a repeatable source SQLite/KB export script. It does not yet have source-data restore or replication into GCP. Do not claim DB parity until one of these is true:

- the existing canonical KB database is replicated into GCP and read back; or
- GCP Cloud SQL/Postgres becomes the canonical database and production services read/write it; or
- an explicitly approved standby restore drill proves that a GCP database can be restored and queried from the retained backups.

Do not call the empty `teleo-pgvector-standby` instance redundancy by itself. It only counts after source data, restore/replication, access controls, and query readback are proven.

The local restore canary narrows the remaining gap: the source SQLite backup can
be converted and restored into PostgreSQL with table/row-count parity, but the
same import still needs to run against the GCP Cloud SQL standby through an
approved GCP auth and network path.

After Cloud SQL import, use the generated `target-counts.sql` and verify it with:

```bash
python3 ops/verify_gcp_cloudsql_restore_readback.py \
  --drill-proof outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-restore-drill-<timestamp>.json \
  --target-counts-csv outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-target-counts-<timestamp>.csv \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-cloudsql-restore-readback-verification-<timestamp>.json
```

The verifier must return `status = pass` before claiming row-count parity in GCP.

## IAM Split Plan

Do not make `sa-artifact-builder@teleo-501523.iam.gserviceaccount.com` the broad
infra account. Keep it scoped to Docker image publishing.

Use the generated plan for the next privilege boundary:

```bash
python3 ops/plan_gcp_iam_split.py --format json
python3 ops/plan_gcp_iam_split.py --format shell
```

For an idempotent retained apply attempt, use:

```bash
python3 ops/apply_gcp_iam_split.py \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-iam-split-apply-dry-run.json

python3 ops/apply_gcp_iam_split.py \
  --execute \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-iam-split-apply-execute.json
```

The first command is dry-run only. The second command mutates IAM and must run
from an authenticated GCP admin shell. It is safe to re-run: existing service
accounts are skipped, and IAM binding commands are additive/idempotent.

The plan creates two separate accounts:

- `sa-teleo-readiness@teleo-501523.iam.gserviceaccount.com`
  - GitHub WIF account for read-only readiness checks.
  - Needs read-only roles for Artifact Registry, IAM/service-account/WIF
    metadata, Compute/network posture, Cloud SQL metadata, backup buckets, and
    Secret Manager metadata.
- `sa-teleo-restore-drill@teleo-501523.iam.gserviceaccount.com`
  - Operator account for explicit Cloud SQL restore drills.
  - Needs Cloud SQL edit rights for the import operation and object-admin access
    to the restore bucket path.

Cloud SQL imports also require the Cloud SQL instance service account to read the
GCS object. The plan includes a command that discovers that instance service
account and grants it `roles/storage.objectAdmin` on
`gs://teleo-501523-prod-backups`.

This plan is not itself a completed redundancy proof. DB redundancy is complete
only after the restore drill imports source data into Cloud SQL and the retained
`target-counts.sql` readback matches the source/local restore proof.

## Runtime Baseline Runner

Do not create or repair the GCP runtime baseline manually in the console if an
audited runner can do it. The runtime baseline runner is dry-run by default:

```bash
python3 ops/apply_gcp_runtime_baseline.py \
  --admin-ssh-cidr <operator-ip>/32 \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-runtime-baseline-dry-run.json
```

The dry-run proof records the exact service accounts, network, firewall, VM,
snapshot, backup bucket, secret, and Cloud SQL operations needed for the
readiness checker. It does not prove that those resources exist.

To apply after an authenticated GCP admin session is available:

```bash
export TELEO_CLOUDSQL_POSTGRES_PASSWORD='<store locally; do not commit or print>'
python3 ops/apply_gcp_runtime_baseline.py \
  --execute \
  --admin-ssh-cidr <operator-ip>/32 \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-runtime-baseline-execute.json
```

`--execute` refuses to run without a single trusted IPv4 `/32` SSH CIDR. The
Cloud SQL password is passed through the environment and redacted from retained
operation commands. After execute mode succeeds, rerun `gcp-readiness.yml` with
the dedicated readiness service account and then run the Cloud SQL restore
drill/readback verifier before claiming database redundancy.

## Communication Posture

The service-to-service communication contract is declared in:

- `config/gcp-service-communications.json`

Validate it locally or in CI with:

```bash
python3 ops/check_gcp_service_communications.py \
  --contract config/gcp-service-communications.json \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-service-communications-check.json
```

The contract currently requires:

- GitHub Actions artifact publishing only through `sa-artifact-builder`.
- GitHub Actions readiness only through `sa-teleo-readiness`.
- emergency SSH only from one operator IPv4 `/32` to Teleo target tags.
- VM image pulls over Private Google Access.
- Cloud SQL only on private VPC paths with encrypted-only PostgreSQL.
- Cloud SQL imports only from the versioned backup bucket through approved
  service accounts.
- no broad SSH/RDP, no public database IP, no default Compute Engine service
  accounts, and no raw secret values in the contract.

This is still a contract until GCP readiness passes. Live proof requires the
current `gcp-readiness` artifact to show the matching firewall, VM service
accounts, bucket, and Cloud SQL checks passing.