teleo-infrastructure/docs/gcp-leoclean-runtime-reconciliation.md

# GCP Leo Runtime Reconciliation

## Scope

This runbook reconciles PR `#148`, the least-privilege follow-up to PR `#145`,
with the non-production GCP parallel Leo service and removes its dependency on
the PostgreSQL administrator credential.
It does not promote GCP, change the Telegram destination, copy newer VPS data,
or make `teleo_canonical` production-authoritative.

Target surfaces:

- project: `teleo-501523`;
- VM: `teleo-prod-1` in `europe-west6-a`;
- service: `leoclean-gcp-prod-parallel.service`;
- Cloud SQL database: `teleo_canonical` on private address `10.61.0.3`;
- runtime database role: `leoclean_kb_runtime`;
- staging-function owner: `leoclean_kb_stage_owner` (`NOLOGIN`);
- runtime secret: `gcp-teleo-pgvector-standby-leoclean-kb-runtime-password`.

Observed before reconciliation on 2026-07-14: the service was
`active/running`, but `/usr/local/bin/teleo-kb` matched unmerged PR `#145`, the
effective systemd environment selected database user `postgres` and the
administrator password secret, and a live `teleo-kb status` call exceeded a
20-second bound. Secret Manager access and private PostgreSQL reachability both
passed independently, isolating the timeout to the old helper path rather than
IAM or networking.

## Required End State

1. The live wrapper and Cloud SQL helper hashes match a merged repository
   commit.
2. `TELEO_KB_MODE=cloudsql`; missing tools or credentials fail closed.
3. Canonical zero-hit searches do not consult `teleo_restore` unless an
   operator explicitly opts in. The service is pinned to database
   `teleo_canonical` with `TELEO_CLOUDSQL_ENABLE_AUDIT_FALLBACK=0`.
4. Leo can read the named canonical tables and stage only a
   `pending_review` proposal through `kb_stage.stage_leoclean_proposal(...)`.
5. Leo cannot directly insert, update, or delete `public.*` or
   `kb_stage.kb_proposals`, forge the proposer identity, `SET ROLE` into a
   broader principal, or execute reviewer/apply functions.
6. The VM runtime identity can access only the scoped password secret needed by
   this path, not the PostgreSQL administrator password secret.
7. Deployment proves the attached VM service account from the metadata server,
   uses an empty phase-local Cloud SDK configuration, and rolls the complete
   prior runtime set back if installation or post-restart verification fails.
8. Post-restart status, permission, and administrator-secret denial probes join
   both the actual service mount and network namespaces. Effective systemd
   network, address-family, socket-bind, syscall-filter, and LSM properties must
   match the reviewed unrestricted transport contract so a host-context helper
   cannot produce a false runtime receipt.

## Safe Order Of Operations

### 1. Read-only access and IAM audit

Confirm the operator can read the project, use IAP/OS Login for the VM, inspect
Cloud SQL metadata, and inspect Secret Manager IAM. Identify the VM service
account and determine whether its secret access is project-wide or
secret-specific. Do not remove a project-wide binding until every legitimately
required runtime secret has an equivalent secret-level binding.

### 2. Merge reviewed repository code

Run CI on the completed PR `#148` repair before deployment. The deployment
source must be the resulting merged commit, not a working tree or unmerged
branch.

### 3. Create the scoped secret and grant only the VM runtime identity

Create the scoped secret without printing its value. Add one randomly generated
version, then grant `roles/secretmanager.secretAccessor` on that secret to the
VM runtime service account. Do not put `PGPASSWORD` in systemd, a repository,
an artifact, or a command transcript.

### 4. Provision the scoped PostgreSQL role once

Run `ops/provision_gcp_leoclean_runtime_role.sh` as the Cloud SQL administrator
from the private VM path. Supply the new runtime password through
`TELEO_LEOCLEAN_DB_PASSWORD` and the administrator password through
`PGPASSWORD`. The wrapper captures and unsets both fields before its first
external child, pins the private endpoint and reviewed server CA, detaches
`psql` from the controlling terminal, and feeds the runtime value only to
psql's client-side `\password` prompt. The SQL file and argv never contain the
cleartext runtime password. The migration:

- creates or rotates `leoclean_kb_runtime`;
- changes any pre-existing `leoclean_kb_runtime` to `NOLOGIN`, removes its
  membership edges, and terminates its existing sessions before password
  handling; an interruption therefore leaves the old login disabled;
- enables `LOGIN` only as the last statement in the same transaction as the
  database ACL, canonical grant, routine, large-object, and topology checks;
- creates a dedicated `NOLOGIN` function owner with no role memberships;
- removes stale table, column, sequence, function, and role-membership grants;
- inventories existing principals, replaces `PUBLIC CONNECT` with explicit
  preserved grants, gives the runtime `CONNECT` only to `teleo_canonical`, and
  leaves the staging owner with no database connection target;
- inventories and preserves existing non-scoped application-routine and
  large-object access while removing `PUBLIC` and both scoped roles from all
  other application routines and every persistent large-object mutator;
- grants exact canonical reads;
- creates a locked security-definer staging function that hard-codes both
  `pending_review` and canonical proposer `leo`;
- grants no direct table writes;
- aborts if either scoped role owns or can reach anything outside the explicit
  allowlist.

PostgreSQL grants `TEMP` to `PUBLIC` by default. The migration removes any
direct scoped `TEMP` grant and reports the remaining effective privilege, but
does not revoke `TEMP` from `PUBLIC`: PostgreSQL has no per-role deny, so that
would be a database-wide behavior change requiring a separate inventory of
`kb_apply`, reviewer, and operator use. The staging function remains protected
by a `pg_catalog, pg_temp` search path, schema-qualified relations, a fixed
`session_user`, and a tested temporary-object shadowing denial.

The administrator password is used only for this bounded bootstrap. Retain no
password output.

`teleo_kb`, `template1`, and every other noncanonical database are actual
negative connection checks. PostgreSQL's startup protocol does not expose a
SQLSTATE through `psql`, so the verifier requires the exact C-locale
`permission denied for database` and `does not have CONNECT privilege`
diagnostics in addition to the catalog proof that the runtime's sole effective
target is `teleo_canonical`. Audit fallback remains disabled with
`TELEO_CLOUDSQL_ENABLE_AUDIT_FALLBACK=0`.

### 5. Preflight and deploy the merged runtime

From the exact merged checkout, run:

```bash
deploy/sync-gcp-leoclean-runtime.sh --dry-run
deploy/sync-gcp-leoclean-runtime.sh --preflight-only
deploy/sync-gcp-leoclean-runtime.sh --restart
```

`--preflight-only` extracts the candidate helper and permission verifier into a
root-created, root-owned, non-writable VM payload directory, verifies the actual
service `MainPID` environment, runs the candidate private Cloud SQL status path,
and runs the sanitized positive-and-negative permission verifier. A separate
`teleo`-owned directory holds only sanitized receipts; it never feeds an install
or rollback. The preflight does not stop or restart the service and does not
install or replace runtime files. It
captures `ActiveState`, `SubState`, `MainPID`, and `NRestarts` before and after
the checks and fails if any value changes. The preflight must pass before
`--restart` is attempted.

The deploy script refuses dirty, symlinked, or unmerged runtime files and
refuses a live install without `--restart`. Its payload and rollback backup are
root-owned and non-writable by `teleo`; existing runtime intermediates and
targets must be real root-owned, non-writable paths. After preflight it stops
the parallel service, atomically replaces the existing Cloud SQL systemd drop-in, installs the
reviewed wrapper, helper, and permission verifier into the root-controlled
`/usr/local/libexec/livingip/leoclean-kb` directory, records the Git revision,
and verifies every installed hash, owner, mode, path type, and revision before
systemd reloads the unit or starts the service. The wrapper uses fixed
`/bin/bash` and `/usr/bin/python3` interpreters, and every directory on the
service `PATH` must be root-owned and non-writable by `teleo`. Every ancestor of that
directory, the directory itself, the wrapper, helper, verifier, and revision
marker must be non-writable by `teleo`. The systemd service receives that
directory as a read-only bind at its existing
`/home/teleo/.hermes/profiles/leoclean/bin` path; verification compares the
source and service-namespace inode and hashes and proves the mount is read-only.
To prevent the `teleo` service user from renaming a writable ancestor and
recreating that absolute path, the service namespace mounts the complete
`/home/teleo` tree read-only, allows writes back only to the profile's `state`
and `workspace` subdirectories, clears every bounding and ambient capability,
and enforces `NoNewPrivileges`. Live verification requires the service home and
bound `bin` to be read-only, both allowed subdirectories to be writable, the
exact `teleo` group set, and every process capability field to be zero.

Preflight, post-restart, and `--verify-only` checks inspect the environment of the actual
systemd `MainPID` through `/proc/<MainPID>/environ`, rather than trusting the
configured unit environment alone. It also rejects effective credential
properties, auxiliary `Exec*` hooks, any `EnvironmentFile`, and any drop-in
ordered after the reviewed configuration. A standalone fail-closed verifier requires
the exact reviewed private target and runtime role, requires
`TELEO_GCP_METADATA_ONLY=1`, and rejects all PostgreSQL, Cloud SDK, proxy, TLS
trust, broader Google credential, shell-startup, dynamic-loader, Python-loader,
administrator-secret, excerpt, credential-mode, claim-base, and retry-count
overrides without printing their values. The reviewed root-owned empty
`CLOUDSDK_CONFIG`, pinned `PGSSLMODE=verify-ca`,
server CA, and Python isolation fields must match exactly. This blocks
pre-wrapper injection through fields such as `BASH_ENV`, any `LD_*`,
`PYTHONPATH`, `PYTHONHTTPSVERIFY`, or `SSLKEYLOGFILE`. The checks also require
the reviewed drop-in to appear in `DropInPaths`. The wrapper status probe then
imports the validated environment from the actual `MainPID` with `nsenter
--env`, enters its mount and network namespaces, drops to `teleo`, and invokes the bound
wrapper without reinjecting any database identity setting.

The effective unit must use the host network namespace (`PrivateNetwork=no`),
have no `NetworkNamespacePath`, `IPAddressDeny`, `IPAddressAllow`,
`RestrictAddressFamilies`, `SocketBindAllow`, `SocketBindDeny`,
`RestrictNetworkInterfaces`, `SystemCallFilter`, `AppArmorProfile`, or
`SELinuxContext` override, and load no
later drop-in. The executable regression harness injects each conflicting
property and requires the gate to fail. These checks cover restrictions that a
new `nsenter` process would not inherit merely by sharing the service network
namespace.

For cgroup/eBPF restrictions, post-restart probes do not rely on enumeration.
A root-only bounded runner verifies the service's single cgroup-v2 membership
against systemd's effective `ControlGroup`, forks a stopped child, moves only
that child into the service cgroup, confirms the move, and then resumes it to
join the service namespaces and drop to `teleo`. It propagates exit or signal
status and reaps the child on every path. The child installs a Linux
parent-death `SIGKILL` before stopping, with a parent-PID race check; the parent
blocks and handles `INT`, `TERM`, and `HUP` through cleanup before re-raising
the signal. Before resume, every supported Linux `RLIMIT_*` soft/hard pair is
copied from the live `MainPID` with a stable source and child readback. Live
`MainPID` verification also requires seccomp mode and filter count zero, the
exact `unconfined` LSM context, `PrivateUsers=no`, and the same user-namespace
inode as PID 1, preventing a per-process filter, profile, resource ceiling, or
user namespace from being mistaken for the reviewed service context.

The service runtime bypasses Cloud SDK configuration and credential caches
entirely. The helper obtains the attached service-account email and a bounded
access token from the fixed GCE metadata endpoint, requires the exact expected
service account, and accesses only the named scoped Secret Manager version over
the fixed HTTPS API. Its URL opener disables proxies. The scoped status read
and administrator-secret IAM denial are rerun after restart. The denial probe
imports the actual systemd `MainPID` environment and mount and network
namespaces, drops to
the service's `teleo` identity, and must observe the exact
`secretmanager.versions.access` permission denial. No token or
secret value is retained. The status read and permission verifier execute from
inside the service mount and network namespaces; the status path invokes the
bound `teleo-kb` wrapper and its adjacent helper rather than bypassing them with
a direct helper invocation. A failure
after mutation restores the prior runtime/drop-in directory existence and
metadata, wrapper, helper, both verifiers, drop-in, and revision as one set,
validates it against the root-only backup, and only then reloads systemd and
restarts the old service.

### 6. Verify positive and negative behavior

`--preflight-only`, `--restart`, and `--verify-only` run the verifier as the
`teleo` Unix user. To rerun the installed verifier directly on the VM and
retain its sanitized JSON receipt, use:

```bash
receipt_dir="$(mktemp -d /tmp/leoclean-permission-receipt.XXXXXX)"
sudo chown teleo:teleo "$receipt_dir"
sudo chmod 0700 "$receipt_dir"
run_id="$(date -u +%Y%m%d%H%M%S)-manual"
sudo -n -u teleo env -i HOME=/home/teleo \
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  /usr/bin/python3 -I /usr/local/libexec/livingip/leoclean-kb/verify_gcp_leoclean_runtime_permissions.py \
  --run-id "$run_id" \
  --output "$receipt_dir/permission-receipt.json"
sudo stat -c '%U:%G:%a' "$receipt_dir/permission-receipt.json"
```

The command must exit zero, print the same canonical JSON that it writes, and
leave the receipt as `teleo:teleo:600`. A passing receipt must report
`status=pass`, `mode=live_private_gcp_staging`, and both `required_tier` and
`current_tier` as `T3_live_readonly`. Those values are a receipt contract, not
a claim that this runbook has already been exercised successfully on the live
staging VM.

Retain a redacted receipt containing:

- merged Git commit and deployed file hashes;
- service `ActiveState`, `SubState`, `MainPID`, and `NRestarts`;
- metadata-server identity and an access token obtained with an empty Cloud SDK
  configuration;
- `current_database()` and `current_user` showing
  `teleo_canonical|leoclean_kb_runtime`;
- private server address `10.61.0.3`, port `5432`, and an active SSL session;
- exact safe runtime-role attributes and zero membership edges in
  `pg_auth_members`;
- zero effective database/schema `CREATE`, persistent ownership, table/column
  DML, sequence privileges, SELECT outside the exact table allowlist, or
  executable unexpected `SECURITY DEFINER` functions, plus the exact staging
  function owner/search-path/ACL contract;
- a real canonical status/search receipt;
- a transaction-rolled-back call to `stage_leoclean_proposal(...)` that returns
  `pending_review` for canonical proposer `leo`, with zero matching canary rows
  both before and after the transaction;
- denied direct insert, update, and delete on `kb_stage.kb_proposals`;
- denied insert, update, and delete on canonical `public.claims`;
- denied `lo_creat`, `lo_create`, and `lo_from_bytea`, zero effective
  large-object mutator execution, and zero scoped large-object ownership/ACLs;
- denied reviewer/apply security-definer functions;
- exact function-catalog posture: only the five-argument staging function is
  executable, the forged six-argument overload is absent, and each expected
  reviewer/apply function exists but is not executable by the runtime role;
- unavailable forged-proposer overload and denied `SET ROLE` escalation into
  the staging owner, reviewer, apply, or administrator roles;
- denied startup connections to both `teleo_kb` and `template1`, with the
  catalog showing `teleo_canonical` as the sole runtime connection target and
  zero remaining `PUBLIC CONNECT` grants;
- readable scoped runtime secret and an IAM permission denial for the
  administrator secret, with neither secret value retained;
- zero Telegram messages and zero committed canary rows.

### 7. Remove administrator-secret access

Only after the scoped service passes post-restart verification, remove the VM
runtime identity's access path to
`gcp-teleo-pgvector-standby-postgres-password`. Verify from the VM runtime
identity that the scoped secret is readable and the administrator secret is
denied. Never delete the administrator secret merely to enforce runtime least
privilege; backup/restore operators may still require it under a separate
identity.

## Stop Conditions

Stop without cutover if any of these are true:

- the deployed revision is not merged;
- the scoped status preflight fails;
- the role has direct proposal-table insert or canonical write permission;
- any approval/apply function is executable by the runtime role;
- removing broad Secret Manager access would break another required secret;
- GCP canonical rows are still stale relative to the chosen authority.

The last condition does not invalidate this runtime security repair. It means
GCP remains staging and data reconciliation stays a separate, explicitly
authorized slice.