157 lines
7.1 KiB
Markdown
157 lines
7.1 KiB
Markdown
# GCP Leo Runtime Reconciliation
|
|
|
|
## Scope
|
|
|
|
This runbook reconciles PR `#145` with the non-production GCP parallel Leo
|
|
service and removes its dependency on the PostgreSQL administrator credential.
|
|
It does not promote GCP, change the Telegram destination, copy newer VPS data,
|
|
or make `teleo_canonical` production-authoritative.
|
|
|
|
Target surfaces:
|
|
|
|
- project: `teleo-501523`;
|
|
- VM: `teleo-prod-1` in `europe-west6-a`;
|
|
- service: `leoclean-gcp-prod-parallel.service`;
|
|
- Cloud SQL database: `teleo_canonical` on private address `10.61.0.3`;
|
|
- runtime database role: `leoclean_kb_runtime`;
|
|
- staging-function owner: `leoclean_kb_stage_owner` (`NOLOGIN`);
|
|
- runtime secret: `gcp-teleo-pgvector-standby-leoclean-kb-runtime-password`.
|
|
|
|
Observed before reconciliation on 2026-07-14: the service was
|
|
`active/running`, but `/usr/local/bin/teleo-kb` matched unmerged PR `#145`, the
|
|
effective systemd environment selected database user `postgres` and the
|
|
administrator password secret, and a live `teleo-kb status` call exceeded a
|
|
20-second bound. Secret Manager access and private PostgreSQL reachability both
|
|
passed independently, isolating the timeout to the old helper path rather than
|
|
IAM or networking.
|
|
|
|
## Required End State
|
|
|
|
1. The live wrapper and Cloud SQL helper hashes match a merged repository
|
|
commit.
|
|
2. `TELEO_KB_MODE=cloudsql`; missing tools or credentials fail closed.
|
|
3. Canonical zero-hit searches do not consult `teleo_restore` unless an
|
|
operator explicitly opts in.
|
|
4. Leo can read the named canonical tables and stage only a
|
|
`pending_review` proposal through `kb_stage.stage_leoclean_proposal(...)`.
|
|
5. Leo cannot directly insert, update, or delete `public.*` or
|
|
`kb_stage.kb_proposals`, forge the proposer identity, `SET ROLE` into a
|
|
broader principal, or execute reviewer/apply functions.
|
|
6. The VM runtime identity can access only the scoped password secret needed by
|
|
this path, not the PostgreSQL administrator password secret.
|
|
7. Deployment proves the attached VM service account from the metadata server,
|
|
uses an empty phase-local Cloud SDK configuration, and rolls the complete
|
|
prior runtime set back if installation or post-restart verification fails.
|
|
|
|
## Safe Order Of Operations
|
|
|
|
### 1. Read-only access and IAM audit
|
|
|
|
Confirm the operator can read the project, use IAP/OS Login for the VM, inspect
|
|
Cloud SQL metadata, and inspect Secret Manager IAM. Identify the VM service
|
|
account and determine whether its secret access is project-wide or
|
|
secret-specific. Do not remove a project-wide binding until every legitimately
|
|
required runtime secret has an equivalent secret-level binding.
|
|
|
|
### 2. Merge reviewed repository code
|
|
|
|
Run CI on the completed PR `#145` repair before deployment. The deployment
|
|
source must be the resulting merged commit, not a working tree or unmerged
|
|
branch.
|
|
|
|
### 3. Create the scoped secret and grant only the VM runtime identity
|
|
|
|
Create the scoped secret without printing its value. Add one randomly generated
|
|
version, then grant `roles/secretmanager.secretAccessor` on that secret to the
|
|
VM runtime service account. Do not put `PGPASSWORD` in systemd, a repository,
|
|
an artifact, or a command transcript.
|
|
|
|
### 4. Provision the scoped PostgreSQL role once
|
|
|
|
Run `ops/gcp_leoclean_runtime_role.sql` as the Cloud SQL administrator from the
|
|
private VM path. Supply the new runtime password through
|
|
`TELEO_LEOCLEAN_DB_PASSWORD`; the SQL file does not contain it. The migration:
|
|
|
|
- creates or rotates `leoclean_kb_runtime`;
|
|
- creates a dedicated `NOLOGIN` function owner with no role memberships;
|
|
- removes stale table, column, sequence, function, and role-membership grants;
|
|
- grants exact canonical reads;
|
|
- creates a locked security-definer staging function that hard-codes both
|
|
`pending_review` and canonical proposer `leo`;
|
|
- grants no direct table writes;
|
|
- removes all access to the legacy `teleo_restore` schema;
|
|
- aborts if either scoped role owns or can reach anything outside the explicit
|
|
allowlist.
|
|
|
|
PostgreSQL grants `TEMP` to `PUBLIC` by default. The migration removes any
|
|
direct scoped `TEMP` grant and reports the remaining effective privilege, but
|
|
does not revoke `TEMP` from `PUBLIC`: PostgreSQL has no per-role deny, so that
|
|
would be a database-wide behavior change requiring a separate inventory of
|
|
`kb_apply`, reviewer, and operator use. The staging function remains protected
|
|
by a `pg_catalog, pg_temp` search path, schema-qualified relations, a fixed
|
|
`session_user`, and a tested temporary-object shadowing denial.
|
|
|
|
The administrator password is used only for this bounded bootstrap. Retain no
|
|
password output.
|
|
|
|
### 5. Preflight and deploy the merged runtime
|
|
|
|
From the exact merged checkout, run:
|
|
|
|
```bash
|
|
deploy/sync-gcp-leoclean-runtime.sh --dry-run
|
|
deploy/sync-gcp-leoclean-runtime.sh --restart
|
|
```
|
|
|
|
The deploy script runs a scoped, redacted database status preflight before it
|
|
installs files or restarts the service. It refuses dirty or unmerged runtime
|
|
files and refuses a live install without `--restart`. After preflight it stops
|
|
the parallel service, atomically replaces the existing Cloud SQL systemd
|
|
drop-in, installs the reviewed wrapper and helper, records the Git revision,
|
|
starts the service, and verifies all three file hashes. A failure after mutation
|
|
restores the prior wrapper, helper, drop-in, and revision as one set before the
|
|
old service is restarted.
|
|
|
|
### 6. Verify positive and negative behavior
|
|
|
|
Retain a redacted receipt containing:
|
|
|
|
- merged Git commit and deployed file hashes;
|
|
- service `ActiveState`, `SubState`, `MainPID`, and `NRestarts`;
|
|
- metadata-server identity and an access token obtained with an empty Cloud SDK
|
|
configuration;
|
|
- `current_database()` and `current_user` showing
|
|
`teleo_canonical|leoclean_kb_runtime`;
|
|
- a real canonical status/search receipt;
|
|
- a transaction-rolled-back call to `stage_leoclean_proposal(...)` that returns
|
|
`pending_review`;
|
|
- denied direct insert into `kb_stage.kb_proposals`;
|
|
- denied canonical `public.*` write;
|
|
- denied reviewer/apply security-definer function;
|
|
- denied forged proposer identity and denied `SET ROLE` escalation;
|
|
- zero Telegram messages and zero committed canary rows.
|
|
|
|
### 7. Remove administrator-secret access
|
|
|
|
Only after the scoped service passes post-restart verification, remove the VM
|
|
runtime identity's access path to
|
|
`gcp-teleo-pgvector-standby-postgres-password`. Verify from the VM runtime
|
|
identity that the scoped secret is readable and the administrator secret is
|
|
denied. Never delete the administrator secret merely to enforce runtime least
|
|
privilege; backup/restore operators may still require it under a separate
|
|
identity.
|
|
|
|
## Stop Conditions
|
|
|
|
Stop without cutover if any of these are true:
|
|
|
|
- the deployed revision is not merged;
|
|
- the scoped status preflight fails;
|
|
- the role has direct proposal-table insert or canonical write permission;
|
|
- any approval/apply function is executable by the runtime role;
|
|
- removing broad Secret Manager access would break another required secret;
|
|
- GCP canonical rows are still stale relative to the chosen authority.
|
|
|
|
The last condition does not invalidate this runtime security repair. It means
|
|
GCP remains staging and data reconciliation stays a separate, explicitly
|
|
authorized slice.
|