teleo/teleo-infrastructure
8
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions 1
teleo-infrastructure/docs/gcp-leoclean-runtime-reconciliation.md

18 KiB
Raw Blame History

GCP Leo Runtime Reconciliation

Scope

This runbook reconciles PR #148, the least-privilege follow-up to PR #145, with the non-production GCP parallel Leo service and removes its dependency on the PostgreSQL administrator credential. It does not promote GCP, change the Telegram destination, copy newer VPS data, or make teleo_canonical production-authoritative.

Target surfaces:

  • project: teleo-501523;
  • VM: teleo-prod-1 in europe-west6-a;
  • service: leoclean-gcp-prod-parallel.service;
  • Cloud SQL database: teleo_canonical on private address 10.61.0.3;
  • runtime database role: leoclean_kb_runtime;
  • staging-function owner: leoclean_kb_stage_owner (NOLOGIN);
  • runtime secret: gcp-teleo-pgvector-standby-leoclean-kb-runtime-password.

Observed before reconciliation on 2026-07-14: the service was active/running, but /usr/local/bin/teleo-kb matched unmerged PR #145, the effective systemd environment selected database user postgres and the administrator password secret, and a live teleo-kb status call exceeded a 20-second bound. Secret Manager access and private PostgreSQL reachability both passed independently, isolating the timeout to the old helper path rather than IAM or networking.

Required End State

  1. The live wrapper and Cloud SQL helper hashes match a merged repository commit.
  2. TELEO_KB_MODE=cloudsql; missing tools or credentials fail closed.
  3. Canonical zero-hit searches do not consult teleo_restore unless an operator explicitly opts in. The service is pinned to database teleo_canonical with TELEO_CLOUDSQL_ENABLE_AUDIT_FALLBACK=0.
  4. Leo can read the named canonical tables and stage only a pending_review proposal through kb_stage.stage_leoclean_proposal(...).
  5. Leo cannot directly insert, update, or delete public.* or kb_stage.kb_proposals, forge the proposer identity, SET ROLE into a broader principal, or execute reviewer/apply functions.
  6. The VM runtime identity can access only the scoped password secret needed by this path, not the PostgreSQL administrator password secret.
  7. Deployment proves the attached VM service account from the metadata server, uses an empty phase-local Cloud SDK configuration, and rolls the complete prior runtime set back if installation or post-restart verification fails.
  8. Post-restart status, permission, and administrator-secret denial probes join both the actual service mount and network namespaces. Effective systemd network, address-family, socket-bind, syscall-filter, and LSM properties must match the reviewed unrestricted transport contract so a host-context helper cannot produce a false runtime receipt.

Safe Order Of Operations

1. Read-only access and IAM audit

Confirm the operator can read the project, use IAP/OS Login for the VM, inspect Cloud SQL metadata, and inspect Secret Manager IAM. Identify the VM service account and determine whether its secret access is project-wide or secret-specific. Do not remove a project-wide binding until every legitimately required runtime secret has an equivalent secret-level binding.

2. Merge reviewed repository code

Run CI on the completed PR #148 repair before deployment. The deployment source must be the resulting merged commit, not a working tree or unmerged branch.

3. Create the scoped secret and grant only the VM runtime identity

Create the scoped secret without printing its value. Add one randomly generated version, then grant roles/secretmanager.secretAccessor on that secret to the VM runtime service account. Do not put PGPASSWORD in systemd, a repository, an artifact, or a command transcript.

4. Provision the scoped PostgreSQL role once

Run ops/provision_gcp_leoclean_runtime_role.sh as the Cloud SQL administrator from the private VM path. Supply the new runtime password through TELEO_LEOCLEAN_DB_PASSWORD and the administrator password through PGPASSWORD. The wrapper captures and unsets both fields before its first external child, pins the private endpoint and reviewed server CA, detaches psql from the controlling terminal, and feeds the runtime value only to psql's client-side \password prompt. The SQL file and argv never contain the cleartext runtime password. The migration:

  • creates or rotates leoclean_kb_runtime;
  • changes any pre-existing leoclean_kb_runtime to NOLOGIN, removes its membership edges, and terminates its existing sessions before password handling; an interruption therefore leaves the old login disabled;
  • enables LOGIN only as the last statement in the same transaction as the database ACL, canonical grant, routine, large-object, and topology checks;
  • creates a dedicated NOLOGIN function owner with no role memberships;
  • removes stale table, column, sequence, function, and role-membership grants;
  • inventories existing principals, replaces PUBLIC CONNECT with explicit preserved grants, gives the runtime CONNECT only to teleo_canonical, and leaves the staging owner with no database connection target;
  • inventories and preserves existing non-scoped application-routine and large-object access while removing PUBLIC and both scoped roles from all other application routines and every persistent large-object mutator;
  • grants exact canonical reads;
  • creates a locked security-definer staging function that hard-codes both pending_review and canonical proposer leo;
  • grants no direct table writes;
  • aborts if either scoped role owns or can reach anything outside the explicit allowlist.

PostgreSQL grants TEMP to PUBLIC by default. The migration removes any direct scoped TEMP grant and reports the remaining effective privilege, but does not revoke TEMP from PUBLIC: PostgreSQL has no per-role deny, so that would be a database-wide behavior change requiring a separate inventory of kb_apply, reviewer, and operator use. The staging function remains protected by a pg_catalog, pg_temp search path, schema-qualified relations, a fixed session_user, and a tested temporary-object shadowing denial.

The administrator password is used only for this bounded bootstrap. Retain no password output.

teleo_kb, template1, and every other noncanonical database are actual negative connection checks. PostgreSQL's startup protocol does not expose a SQLSTATE through psql, so the verifier requires the exact C-locale permission denied for database and does not have CONNECT privilege diagnostics in addition to the catalog proof that the runtime's sole effective target is teleo_canonical. Audit fallback remains disabled with TELEO_CLOUDSQL_ENABLE_AUDIT_FALLBACK=0.

5. Preflight and deploy the merged runtime

From the exact merged checkout, run:

deploy/sync-gcp-leoclean-runtime.sh --dry-run
deploy/sync-gcp-leoclean-runtime.sh --preflight-only
deploy/sync-gcp-leoclean-runtime.sh --restart

--preflight-only extracts the candidate helper and permission verifier into a root-created, root-owned, non-writable VM payload directory, verifies the actual service MainPID environment, runs the candidate private Cloud SQL status path, and runs the sanitized positive-and-negative permission verifier. A separate teleo-owned directory holds only sanitized receipts; it never feeds an install or rollback. The preflight does not stop or restart the service and does not install or replace runtime files. It captures ActiveState, SubState, MainPID, and NRestarts before and after the checks and fails if any value changes. The preflight must pass before --restart is attempted.

The deploy script refuses dirty, symlinked, or unmerged runtime files and refuses a live install without --restart. Its payload and rollback backup are root-owned and non-writable by teleo; existing runtime intermediates and targets must be real root-owned, non-writable paths. After preflight it stops the parallel service, atomically replaces the existing Cloud SQL systemd drop-in, installs the reviewed wrapper, helper, and permission verifier into the root-controlled /usr/local/libexec/livingip/leoclean-kb directory, records the Git revision, and verifies every installed hash, owner, mode, path type, and revision before systemd reloads the unit or starts the service. The wrapper uses fixed /bin/bash and /usr/bin/python3 interpreters, and every directory on the service PATH must be root-owned and non-writable by teleo. Every ancestor of that directory, the directory itself, the wrapper, helper, verifier, and revision marker must be non-writable by teleo. The systemd service receives that directory as a read-only bind at its existing /home/teleo/.hermes/profiles/leoclean/bin path; verification compares the source and service-namespace inode and hashes and proves the mount is read-only. To prevent the teleo service user from renaming a writable ancestor and recreating that absolute path, the service namespace mounts the complete /home/teleo tree read-only, allows writes back only to the profile's state and workspace subdirectories, clears every bounding and ambient capability, and enforces NoNewPrivileges. Live verification requires the service home and bound bin to be read-only, both allowed subdirectories to be writable, the exact teleo group set, and every process capability field to be zero.

Preflight, post-restart, and --verify-only checks inspect the environment of the actual systemd MainPID through /proc/<MainPID>/environ, rather than trusting the configured unit environment alone. It also rejects effective credential properties, auxiliary Exec* hooks, any EnvironmentFile, and any drop-in ordered after the reviewed configuration. A standalone fail-closed verifier requires the exact reviewed private target and runtime role, requires TELEO_GCP_METADATA_ONLY=1, and rejects all PostgreSQL, Cloud SDK, proxy, TLS trust, broader Google credential, shell-startup, dynamic-loader, Python-loader, administrator-secret, excerpt, credential-mode, claim-base, and retry-count overrides without printing their values. The reviewed root-owned empty CLOUDSDK_CONFIG, pinned PGSSLMODE=verify-ca, server CA, and Python isolation fields must match exactly. This blocks pre-wrapper injection through fields such as BASH_ENV, any LD_*, PYTHONPATH, PYTHONHTTPSVERIFY, or SSLKEYLOGFILE. The checks also require the reviewed drop-in to appear in DropInPaths. The wrapper status probe then imports the validated environment from the actual MainPID with nsenter --env, enters its mount and network namespaces, drops to teleo, and invokes the bound wrapper without reinjecting any database identity setting.

The effective unit must use the host network namespace (PrivateNetwork=no), have no NetworkNamespacePath, IPAddressDeny, IPAddressAllow, RestrictAddressFamilies, SocketBindAllow, SocketBindDeny, RestrictNetworkInterfaces, SystemCallFilter, AppArmorProfile, or SELinuxContext override, and load no later drop-in. The executable regression harness injects each conflicting property and requires the gate to fail. These checks cover restrictions that a new nsenter process would not inherit merely by sharing the service network namespace.

For cgroup/eBPF restrictions, post-restart probes do not rely on enumeration. A root-only bounded runner verifies the service's single cgroup-v2 membership against systemd's effective ControlGroup, forks a stopped child, moves only that child into the service cgroup, confirms the move, and then resumes it to join the service namespaces and drop to teleo. It propagates exit or signal status and reaps the child on every path. The child installs a Linux parent-death SIGKILL before stopping, with a parent-PID race check; the parent blocks and handles INT, TERM, and HUP through cleanup before re-raising the signal. Before resume, every supported Linux RLIMIT_* soft/hard pair is copied from the live MainPID with a stable source and child readback. Live MainPID verification also requires seccomp mode and filter count zero, the exact unconfined LSM context, PrivateUsers=no, and the same user-namespace inode as PID 1, preventing a per-process filter, profile, resource ceiling, or user namespace from being mistaken for the reviewed service context.

The service runtime bypasses Cloud SDK configuration and credential caches entirely. The helper obtains the attached service-account email and a bounded access token from the fixed GCE metadata endpoint, requires the exact expected service account, and accesses only the named scoped Secret Manager version over the fixed HTTPS API. Its URL opener disables proxies. The scoped status read and administrator-secret IAM denial are rerun after restart. The denial probe imports the actual systemd MainPID environment and mount and network namespaces, drops to the service's teleo identity, and must observe the exact secretmanager.versions.access permission denial. No token or secret value is retained. The status read and permission verifier execute from inside the service mount and network namespaces; the status path invokes the bound teleo-kb wrapper and its adjacent helper rather than bypassing them with a direct helper invocation. A failure after mutation restores the prior runtime/drop-in directory existence and metadata, wrapper, helper, both verifiers, drop-in, and revision as one set, validates it against the root-only backup, and only then reloads systemd and restarts the old service.

6. Verify positive and negative behavior

--preflight-only, --restart, and --verify-only run the verifier as the teleo Unix user. To rerun the installed verifier directly on the VM and retain its sanitized JSON receipt, use:

receipt_dir="$(mktemp -d /tmp/leoclean-permission-receipt.XXXXXX)"
sudo chown teleo:teleo "$receipt_dir"
sudo chmod 0700 "$receipt_dir"
run_id="$(date -u +%Y%m%d%H%M%S)-manual"
sudo -n -u teleo env -i HOME=/home/teleo \
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  /usr/bin/python3 -I /usr/local/libexec/livingip/leoclean-kb/verify_gcp_leoclean_runtime_permissions.py \
  --run-id "$run_id" \
  --output "$receipt_dir/permission-receipt.json"
sudo stat -c '%U:%G:%a' "$receipt_dir/permission-receipt.json"

The command must exit zero, print the same canonical JSON that it writes, and leave the receipt as teleo:teleo:600. A passing receipt must report status=pass, mode=live_private_gcp_staging, and both required_tier and current_tier as T3_live_readonly. Those values are a receipt contract, not a claim that this runbook has already been exercised successfully on the live staging VM.

Retain a redacted receipt containing:

  • merged Git commit and deployed file hashes;
  • service ActiveState, SubState, MainPID, and NRestarts;
  • metadata-server identity and an access token obtained with an empty Cloud SDK configuration;
  • current_database() and current_user showing teleo_canonical|leoclean_kb_runtime;
  • private server address 10.61.0.3, port 5432, and an active SSL session;
  • exact safe runtime-role attributes and zero membership edges in pg_auth_members;
  • zero effective database/schema CREATE, persistent ownership, table/column DML, sequence privileges, SELECT outside the exact table allowlist, or executable unexpected SECURITY DEFINER functions, plus the exact staging function owner/search-path/ACL contract;
  • a real canonical status/search receipt;
  • a transaction-rolled-back call to stage_leoclean_proposal(...) that returns pending_review for canonical proposer leo, with zero matching canary rows both before and after the transaction;
  • denied direct insert, update, and delete on kb_stage.kb_proposals;
  • denied insert, update, and delete on canonical public.claims;
  • denied lo_creat, lo_create, and lo_from_bytea, zero effective large-object mutator execution, and zero scoped large-object ownership/ACLs;
  • denied reviewer/apply security-definer functions;
  • exact function-catalog posture: only the five-argument staging function is executable, the forged six-argument overload is absent, and each expected reviewer/apply function exists but is not executable by the runtime role;
  • unavailable forged-proposer overload and denied SET ROLE escalation into the staging owner, reviewer, apply, or administrator roles;
  • denied startup connections to both teleo_kb and template1, with the catalog showing teleo_canonical as the sole runtime connection target and zero remaining PUBLIC CONNECT grants;
  • readable scoped runtime secret and an IAM permission denial for the administrator secret, with neither secret value retained;
  • zero Telegram messages and zero committed canary rows.

7. Remove administrator-secret access

Only after the scoped service passes post-restart verification, remove the VM runtime identity's access path to gcp-teleo-pgvector-standby-postgres-password. Verify from the VM runtime identity that the scoped secret is readable and the administrator secret is denied. Never delete the administrator secret merely to enforce runtime least privilege; backup/restore operators may still require it under a separate identity.

Stop Conditions

Stop without cutover if any of these are true:

  • the deployed revision is not merged;
  • the scoped status preflight fails;
  • the role has direct proposal-table insert or canonical write permission;
  • any approval/apply function is executable by the runtime role;
  • removing broad Secret Manager access would break another required secret;
  • GCP canonical rows are still stale relative to the chosen authority.

The last condition does not invalidate this runtime security repair. It means GCP remains staging and data reconciliation stays a separate, explicitly authorized slice.