# GCP Leo Runtime Reconciliation

## Scope

This runbook reconciles PR `#145` with the non-production GCP parallel Leo
service and removes its dependency on the PostgreSQL administrator credential.
It does not promote GCP, change the Telegram destination, copy newer VPS data,
or make `teleo_canonical` production-authoritative.

Target surfaces:

- project: `teleo-501523`;
- VM: `teleo-prod-1` in `europe-west6-a`;
- service: `leoclean-gcp-prod-parallel.service`;
- Cloud SQL database: `teleo_canonical` on private address `10.61.0.3`;
- runtime database role: `leoclean_kb_runtime`;
- staging-function owner: `leoclean_kb_stage_owner` (`NOLOGIN`);
- runtime secret: `gcp-teleo-pgvector-standby-leoclean-kb-runtime-password`.

Observed before reconciliation on 2026-07-14: the service was
`active/running`, but `/usr/local/bin/teleo-kb` matched unmerged PR `#145`, the
effective systemd environment selected database user `postgres` and the
administrator password secret, and a live `teleo-kb status` call exceeded a
20-second bound. Secret Manager access and private PostgreSQL reachability both
passed independently, isolating the timeout to the old helper path rather than
IAM or networking.

## Required End State

1. The live wrapper and Cloud SQL helper hashes match a merged repository
   commit.
2. `TELEO_KB_MODE=cloudsql`; missing tools or credentials fail closed.
3. Canonical zero-hit searches do not consult `teleo_restore` unless an
   operator explicitly opts in.
4. Leo can read the named canonical tables and stage only a
   `pending_review` proposal through `kb_stage.stage_leoclean_proposal(...)`.
5. Leo cannot directly insert, update, or delete `public.*` or
   `kb_stage.kb_proposals`, forge the proposer identity, `SET ROLE` into a
   broader principal, or execute reviewer/apply functions.
6. The VM runtime identity can access only the scoped password secret needed by
   this path, not the PostgreSQL administrator password secret.
7. Deployment proves the attached VM service account from the metadata server,
   uses an empty phase-local Cloud SDK configuration, and rolls the complete
   prior runtime set back if installation or post-restart verification fails.

## Safe Order Of Operations

### 1. Read-only access and IAM audit

Confirm the operator can read the project, use IAP/OS Login for the VM, inspect
Cloud SQL metadata, and inspect Secret Manager IAM. Identify the VM service
account and determine whether its secret access is project-wide or
secret-specific. Do not remove a project-wide binding until every legitimately
required runtime secret has an equivalent secret-level binding.

### 2. Merge reviewed repository code

Run CI on the completed PR `#145` repair before deployment. The deployment
source must be the resulting merged commit, not a working tree or unmerged
branch.

### 3. Create the scoped secret and grant only the VM runtime identity

Create the scoped secret without printing its value. Add one randomly generated
version, then grant `roles/secretmanager.secretAccessor` on that secret to the
VM runtime service account. Do not put `PGPASSWORD` in systemd, a repository,
an artifact, or a command transcript.

### 4. Provision the scoped PostgreSQL role once

Run `ops/gcp_leoclean_runtime_role.sql` as the Cloud SQL administrator from the
private VM path. Supply the new runtime password through
`TELEO_LEOCLEAN_DB_PASSWORD`; the SQL file does not contain it. The migration:

- creates or rotates `leoclean_kb_runtime`;
- creates a dedicated `NOLOGIN` function owner with no role memberships;
- removes stale table, column, sequence, function, and role-membership grants;
- grants exact canonical reads;
- creates a locked security-definer staging function that hard-codes both
  `pending_review` and canonical proposer `leo`;
- grants no direct table writes;
- removes all access to the legacy `teleo_restore` schema;
- aborts if either scoped role owns or can reach anything outside the explicit
  allowlist.

PostgreSQL grants `TEMP` to `PUBLIC` by default. The migration removes any
direct scoped `TEMP` grant and reports the remaining effective privilege, but
does not revoke `TEMP` from `PUBLIC`: PostgreSQL has no per-role deny, so that
would be a database-wide behavior change requiring a separate inventory of
`kb_apply`, reviewer, and operator use. The staging function remains protected
by a `pg_catalog, pg_temp` search path, schema-qualified relations, a fixed
`session_user`, and a tested temporary-object shadowing denial.

The administrator password is used only for this bounded bootstrap. Retain no
password output.

### 5. Preflight and deploy the merged runtime

From the exact merged checkout, run:

```bash
deploy/sync-gcp-leoclean-runtime.sh --dry-run
deploy/sync-gcp-leoclean-runtime.sh --restart
```

The deploy script runs a scoped, redacted database status preflight before it
installs files or restarts the service. It refuses dirty or unmerged runtime
files and refuses a live install without `--restart`. After preflight it stops
the parallel service, atomically replaces the existing Cloud SQL systemd
drop-in, installs the reviewed wrapper and helper, records the Git revision,
starts the service, and verifies all three file hashes. A failure after mutation
restores the prior wrapper, helper, drop-in, and revision as one set before the
old service is restarted.

### 6. Verify positive and negative behavior

Retain a redacted receipt containing:

- merged Git commit and deployed file hashes;
- service `ActiveState`, `SubState`, `MainPID`, and `NRestarts`;
- metadata-server identity and an access token obtained with an empty Cloud SDK
  configuration;
- `current_database()` and `current_user` showing
  `teleo_canonical|leoclean_kb_runtime`;
- a real canonical status/search receipt;
- a transaction-rolled-back call to `stage_leoclean_proposal(...)` that returns
  `pending_review`;
- denied direct insert into `kb_stage.kb_proposals`;
- denied canonical `public.*` write;
- denied reviewer/apply security-definer function;
- denied forged proposer identity and denied `SET ROLE` escalation;
- zero Telegram messages and zero committed canary rows.

### 7. Remove administrator-secret access

Only after the scoped service passes post-restart verification, remove the VM
runtime identity's access path to
`gcp-teleo-pgvector-standby-postgres-password`. Verify from the VM runtime
identity that the scoped secret is readable and the administrator secret is
denied. Never delete the administrator secret merely to enforce runtime least
privilege; backup/restore operators may still require it under a separate
identity.

## Stop Conditions

Stop without cutover if any of these are true:

- the deployed revision is not merged;
- the scoped status preflight fails;
- the role has direct proposal-table insert or canonical write permission;
- any approval/apply function is executable by the runtime role;
- removing broad Secret Manager access would break another required secret;
- GCP canonical rows are still stale relative to the chosen authority.

The last condition does not invalidate this runtime security repair. It means
GCP remains staging and data reconciliation stays a separate, explicitly
authorized slice.