teleo-infrastructure/docs/gcp-infra-hardening-20260707.md

# GCP CI/CD and Redundancy Hardening

This records the current GCP hardening contract for `teleo-501523`.

## What Is Live

- Artifact Registry Docker repositories exist in `europe-west6`:
  - `teleo`
  - `livingip-web`
- Both repositories use immutable tags and active vulnerability scanning.
- `cloudbuild.gcp-staging.yaml` builds the staging Teleo image, runs the image smoke test, then pushes to Artifact Registry.
- Cloud Build runs as the dedicated service account:
  - `sa-teleo-cloudbuild@teleo-501523.iam.gserviceaccount.com`
- The dedicated Cloud Build account has only the roles required for the current build/publish path:
  - `roles/artifactregistry.writer`
  - `roles/logging.logWriter`
  - `roles/storage.objectViewer`
- GitHub Actions can publish Artifact Registry images through Workload Identity Federation:
  - workflow: `.github/workflows/gcp-artifact.yml`
  - provider: `projects/785938879453/locations/global/workloadIdentityPools/github-actions/providers/living-ip-github`
  - service account: `sa-artifact-builder@teleo-501523.iam.gserviceaccount.com`
  - repository scope: `living-ip/teleo-infrastructure`
- Backup buckets are versioned and use uniform bucket-level access:
  - `gs://teleo-501523-prod-backups`
  - `gs://teleo-501523-leoclean-backups`
- VM boot disks have a daily 7-day snapshot policy:
  - `teleo-prod-1`
  - `teleo-staging-1`
- Cloud SQL standby target exists for KB restore/replication drills:
  - instance: `teleo-pgvector-standby`
  - database: `teleo_kb`
  - version: `POSTGRES_16`
  - private IP only on `teleo-staging-net`
  - encrypted-only SQL connections
  - automated backups and point-in-time recovery enabled
  - deletion protection enabled
- Source-side Teleo DB/KB export canary exists:
  - script: `ops/backup_vps_sqlite_kb.sh`
  - source DB: `/opt/teleo-eval/pipeline/pipeline.db`
  - source Leo/KB files: `workspaces/*/agents/leo` plus `agent-state`
  - excludes secrets and logs
- Local SQLite-to-Postgres restore canary exists:
  - scripts: `ops/sqlite_to_postgres_dump.py` and `ops/run_sqlite_postgres_restore_canary.sh`
  - target: disposable local `postgres:16-alpine` shadow schema
  - verifies source SQLite integrity and per-table source/target row-count parity

## How To Build

Automatic Artifact Registry publishing runs on pushes to `main` through GitHub Actions. To run the same lane manually from GitHub:

```bash
gh workflow run gcp-artifact.yml --repo living-ip/teleo-infrastructure --ref main
```

The workflow authenticates to GCP with Workload Identity Federation, builds `Dockerfile.gcp-staging`, runs the image smoke test, pushes the image, and uploads `gcp-artifact-image.txt` as a run artifact.

For a read-only GCP posture probe through the same Workload Identity path:

```bash
gh workflow run gcp-readiness.yml --repo living-ip/teleo-infrastructure --ref main
```

This workflow runs `ops/check_gcp_infra_readiness.py` from GitHub Actions and
uploads stdout, stderr, exit code, and a summary as the `gcp-readiness` artifact.
It is intentionally non-mutating. If the Artifact Registry service account lacks
broader read permissions, the artifact is still the proof of the exact IAM gap.

After applying the dedicated readiness service account from the IAM split plan,
run the same workflow against that account:

```bash
gh workflow run gcp-readiness.yml \
  --repo living-ip/teleo-infrastructure \
  --ref main \
  -f service_account=sa-teleo-readiness@teleo-501523.iam.gserviceaccount.com
```

For a local/manual Cloud Build proof:

```bash
REVISION="$(git rev-parse HEAD)"
TAG="$(git rev-parse --short=7 HEAD)-manual-$(date -u +%Y%m%d-%H%M%S)"

gcloud builds submit \
  --project=teleo-501523 \
  --config=cloudbuild.gcp-staging.yaml \
  --substitutions="_TAG=${TAG},_REVISION=${REVISION}"
```

Expected result:

- `build-staging-image` succeeds.
- `smoke-test-image-before-push` succeeds.
- A Docker image is pushed to:
  `europe-west6-docker.pkg.dev/teleo-501523/teleo/teleo-pipeline-gcp-staging:${TAG}`

## How To Check Readiness

Run from the repository root:

```bash
python3 ops/check_gcp_infra_readiness.py
```

The check is read-only and prints no secret values. It verifies:

- Artifact Registry immutability and vulnerability scanning.
- Cloud Build config contract.
- Dedicated Cloud Build service account and roles.
- GitHub Actions WIF Artifact Registry publishing contract.
- Network ingress posture:
  - no enabled broad SSH/RDP ingress;
  - Teleo SSH rules are scoped to `/32` source ranges and target tags.
- Runtime service-account posture for the prod/staging VMs.
- Compute disk snapshot policy attachment.
- Backup bucket versioning and uniform access.
- Cloud SQL standby target posture.
- Source SQLite/KB backup/export repeatability.
- Whether an approved source KB/Postgres dump or replication credential exists.
- Whether source data has actually been restored or replicated into GCP and queried.
- Whether the GitHub WIF readiness workflow exists for non-local readback.

## Current Boundaries

The GCP Docker build/publish path is live through manual Cloud Build and GitHub Actions Workload Identity Federation. Native Cloud Build GitHub triggers are not configured because this project currently has no Cloud Build repository connection.

Database redundancy is not complete. The current project now has a GCP Cloud SQL/Postgres standby target, backup buckets, VM disk snapshots, and a repeatable source SQLite/KB export script. It does not yet have source-data restore or replication into GCP. Do not claim DB parity until one of these is true:

- the existing canonical KB database is replicated into GCP and read back; or
- GCP Cloud SQL/Postgres becomes the canonical database and production services read/write it; or
- an explicitly approved standby restore drill proves that a GCP database can be restored and queried from the retained backups.

Do not call the empty `teleo-pgvector-standby` instance redundancy by itself. It only counts after source data, restore/replication, access controls, and query readback are proven.

The local restore canary narrows the remaining gap: the source SQLite backup can
be converted and restored into PostgreSQL with table/row-count parity, but the
same import still needs to run against the GCP Cloud SQL standby through an
approved GCP auth and network path.

## IAM Split Plan

Do not make `sa-artifact-builder@teleo-501523.iam.gserviceaccount.com` the broad
infra account. Keep it scoped to Docker image publishing.

Use the generated plan for the next privilege boundary:

```bash
python3 ops/plan_gcp_iam_split.py --format json
python3 ops/plan_gcp_iam_split.py --format shell
```

For an idempotent retained apply attempt, use:

```bash
python3 ops/apply_gcp_iam_split.py \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-iam-split-apply-dry-run.json

python3 ops/apply_gcp_iam_split.py \
  --execute \
  --output outputs/gcp-infra-hardening-20260707/proofs/gcp-iam-split-apply-execute.json
```

The first command is dry-run only. The second command mutates IAM and must run
from an authenticated GCP admin shell. It is safe to re-run: existing service
accounts are skipped, and IAM binding commands are additive/idempotent.

The plan creates two separate accounts:

- `sa-teleo-readiness@teleo-501523.iam.gserviceaccount.com`
  - GitHub WIF account for read-only readiness checks.
  - Needs read-only roles for Artifact Registry, IAM/service-account/WIF
    metadata, Compute/network posture, Cloud SQL metadata, backup buckets, and
    Secret Manager metadata.
- `sa-teleo-restore-drill@teleo-501523.iam.gserviceaccount.com`
  - Operator account for explicit Cloud SQL restore drills.
  - Needs Cloud SQL edit rights for the import operation and object-admin access
    to the restore bucket path.

Cloud SQL imports also require the Cloud SQL instance service account to read the
GCS object. The plan includes a command that discovers that instance service
account and grants it `roles/storage.objectAdmin` on
`gs://teleo-501523-prod-backups`.

This plan is not itself a completed redundancy proof. DB redundancy is complete
only after the restore drill imports source data into Cloud SQL and the retained
`target-counts.sql` readback matches the source/local restore proof.

## Communication Posture

Current GCP communication hardening is limited to the surfaces already read back:

- default SSH/RDP firewall rules are disabled;
- Teleo SSH access is scoped by instance tags and current source IP rules;
- the custom Teleo subnet has Private Google Access enabled.
- prod/staging VMs run as dedicated Teleo service accounts, not the default Compute Engine service account.

Before production cutover, add a dedicated runtime communication contract for service-to-service paths, including exact source tags/service accounts, allowed ports, health checks, and rollback.