teleo-infrastructure/.agents/skills/teleo-gcp-parity-ops/SKILL.md
twentyOne2x 1a3305ac5d Merge remote-tracking branch 'origin/main' into codex/leo-gcp-data-parity-20260715
# Conflicts:
#	.agents/skills/teleo-gcp-parity-ops/SKILL.md
2026-07-15 10:26:26 +02:00

262 lines
14 KiB
Markdown

---
name: teleo-gcp-parity-ops
description: Use for passwordless Teleo GCP VM access, private Cloud SQL canonical parity, GCP Leo runtime readback, m3taversal no-send replay, rollback, and cleanup without collapsing VPS proof into GCP proof.
---
# Teleo GCP Parity Ops
## Working Target
Restore a copy of the VPS canonical Leo database to GCP, prove exact schema,
row, role membership, ownership, ACL, extension, performance, and
private-connectivity parity, then run the
real GCP Leo read/reasoning path without Telegram sends or DB mutation.
The required tier is T3: a live private GCP staging restore, readback,
reasoning, compute attestation, and cleanup receipt. Local restore proof or an
access-gate report does not satisfy this tier.
## Operator Paths
The direct alias was passwordless and live-verified on 2026-07-14:
```bash
ssh teleo-gcp-staging
```
It uses `~/.ssh/google_compute_engine` for the configured operator,
disables password and keyboard-interactive authentication, and does not store a
Google password. That is historical proof, not current reachability. The route
remains firewall-source-dependent. On
2026-07-15, both direct SSH and the private TCP route timed out; verify
`ssh -o BatchMode=yes teleo-gcp-staging true` before a long run and fail closed
if it does not return successfully.
The intended secondary path is `.github/workflows/gcp-iap-operator.yml`, using
short-lived GitHub OIDC, IAP, OS Login, and fixed reviewed operations. It is
merged but not bootstrapped: the current 2026-07-15 check still lacks the
reviewed operator WIF/IAP authority. Provider `teleo-iap-operator` is absent,
disabled, or deleted (`invalid_target`), and the available artifact identity is intentionally not
a Cloud SQL/Compute operator. Do not call this path working until a live `status` run passes
with the intended operator identity.
Historically verified and intended target; reverify before any new lifecycle:
- project `teleo-501523`;
- VM `teleo-prod-1` in `europe-west6-a`;
- Cloud SQL `teleo-pgvector-standby`, PostgreSQL 16.14;
- private endpoint `10.61.0.3:5432`, public IP disabled, TLS required;
- restore identity `sa-teleo-restore-drill@teleo-501523.iam.gserviceaccount.com`;
- service `leoclean-gcp-prod-parallel.service`.
Direct libpq connections must use `sslmode=verify-full`, a reviewed DNS name
covered by the server certificate, and a private regular CA file. The private
IP is `hostaddr`, not the certificate identity. Never downgrade to
`sslmode=require` or accept an arbitrary `*.iam.gserviceaccount.com` identity.
Never print the Cloud SQL or Google password. On the VM, use the attached service account
and Secret Manager through reviewed wrappers or a short-lived environment
variable, then unset it.
## Two Different Databases
- Canonical collective knowledge remains VPS PostgreSQL database `teleo`, with
`public.*` and `kb_stage.*` tables. Persistent Cloud SQL database
`teleo_canonical` is a staging copy and has not been promoted.
- Hermes conversation continuity is `state.db` plus session JSONL files. The
`leoclean-cloudsql-memory-sync.service` snapshot path copies this runtime
memory; it does not populate or prove canonical claims, sources, evidence,
edges, or proposals.
Do not describe a passing Hermes memory sync as canonical KB parity.
## Retained Historical Proof - 2026-07-14
- The newest captured VPS database has `39` tables and `52,167` rows, including
claims `1837`, sources `4145`, claim evidence `4670`, claim edges `4916`, and
proposals `29`.
- A disposable private-TLS GCP clone restored that snapshot with exact
`39/39`-table and `52,167/52,167`-row parity. Rowsets, schema objects, roles,
extensions, constraints, and performance checks had zero mismatches.
- A real no-send GCP Hermes turn received an ID-free claim challenge, performed
`search`, `show`, `evidence`, and `edges`, retrieved the expected claim and
both source rows, and passed `18/18` runtime checks plus `6/6` reasoning
outcomes. It did not send Telegram or write the DB.
- `status` and zero-hit search work on a canonical-only clone without the
optional `teleo_restore` audit schema. All read commands emit deterministic
retrieval receipts.
- The generated clone was deleted. The retained rollback database remains
connection-disabled with zero sessions. The GCP gateway remained PID
`148735`, `NRestarts=0`, active/running throughout that bounded experiment.
- PR `#144` merged the reviewed helper/skill. After deployment, the service
survived a controlled restart and is active/running at PID `304036`, start
time `2026-07-14 10:40:17 UTC`. Live post-restart `status` and `search`
returned Cloud SQL retrieval receipts with unchanged canonical counts.
- A live regression showed the old wrapper could time out its stronger status
probe and fall through to a different local tool. Supported GCP KB commands
now route directly to Cloud SQL and fail closed on errors; two behavioral
tests enforce that invariant.
- Persistent GCP `teleo_canonical` remains the older staging copy measured at
`52,164` rows and `26` proposals. It has not been promoted or cut over.
That historical clone predates the expanded ownership/ACL and independently
authenticated receipt contract. It remains useful historical row/schema proof,
but it is not current T3 proof under this skill.
## Open Least-Privilege Candidates
PR #148, `Scope GCP Leo runtime to least-privilege Cloud SQL access`, is open as
of the 2026-07-15 skill-pack reconciliation. Draft PR #162, `Harden GCP Leo
runtime least-privilege proof`, is its current hardening successor. They propose
scoped runtime roles, secret access, fail-closed Cloud SQL behavior, deployment
rollback, and positive plus negative permission checks. Those branch files and
proposed live outcomes are candidate evidence only. Before using them, run:
```bash
gh pr view 148 --json state,mergedAt,mergeCommit,headRefName,url
gh pr view 162 --json state,mergedAt,mergeCommit,headRefName,url
```
Until a successor is merged and its runtime receipt passes, use only paths present on
canonical `main`, keep persistent GCP classified as staging, do not promote it,
and do not infer least-privilege cutover from a branch or dry run.
Primary retained proof:
- `docs/reports/leo-working-state-20260709/gcp-db-first-working-leo-20260714.md`
- `docs/reports/leo-working-state-20260709/gcp-db-first-restore-current.json`
- `docs/reports/leo-working-state-20260709/gcp-db-first-parity-current.json`
- `docs/reports/leo-working-state-20260709/gcp-db-first-blind-claim-current.json`
- `docs/reports/leo-working-state-20260709/gcp-db-first-cleanup-current.json`
- `docs/reports/leo-working-state-20260709/gcp-db-first-live-deploy-restart-current.json`
## Required Parity Rows
Track these independently:
1. control-plane project, VM, Cloud SQL, private-IP, and TLS identity;
2. canonical database schema, counts, row hashes, constraints, indexes,
functions, extensions, exact roles and memberships, owners, normalized
schema/relation/column/function/type ACLs, and performance;
3. GCP service PID/restarts plus live profile and tool hashes;
4. `DC-01` through `DC-06` DB-read readiness;
5. real no-send model replies, strict score, and exact count consistency;
6. no-mutation fingerprints, child/profile cleanup, clone deletion, and rollback
disposition.
## Canonical Restore And Replay
Use:
- `ops/capture_vps_canonical_postgres_snapshot.py` for a single source dump and
manifest, with a required retained `--authorization-ref` and v2 provenance
receipt binding the exported snapshot identity plus dump/manifest hashes; it
requires identical healthy structured source-service states before/after
(`active/running`, positive PID, nonnegative restart count);
- `ops/restore_gcp_generated_postgres_snapshot.py restore --execute` for a
receipt-bound private-TLS restore into a bounded `teleo_clone_*` database;
pass the exact capture `--source-receipt`, and require its live Cloud SQL
control-plane check to prove public IP disabled before clone creation; it
requires the exact restore service account plus `--ssl-server-name` and
`--ssl-root-cert` for certificate-authenticated `verify-full` TLS; it
writes `target-manifest.jsonl`, `restore-receipt.json`, and the generated
`gcp-private-connectivity.json` under
`/var/lib/teleo-gcp-restore-runs/<restore-run-id>/`;
- `ops/restore_gcp_generated_postgres_snapshot.py cleanup --execute` for exact
clone cleanup with live-service and rollback readback;
- `ops/postgres_parity_manifest.sql` for row/catalog/role/performance readback;
- `ops/verify_postgres_parity_manifest.py --scope gcp_staging` for exact parity,
always retaining the canonical output name `gcp-parity.json`;
- `scripts/run_gcp_generated_db_direct_claim_suite.py` for the adapter-free,
read-only, no-send six-response replay against a generated `teleo_clone_*`.
- `scripts/run_gcp_generated_db_blind_claim_canary.py` for the bounded ID-free
claim challenge, source receipt, reasoning, no-write, and cleanup proof.
- `ops/attest_gcp_reasoning_compute.py` on the replay VM after blind reasoning
and before cleanup, binding the reasoning receipt SHA-256 to live GCE metadata
in `reasoning-compute-attestation.json`;
- a second authorized source capture after cleanup, followed by
`ops/verify_vps_canonical_snapshot_delta.py`, to retain the explicit
post-snapshot VPS delta in `source-delta-receipt.json`;
- `ops/verify_gcp_canonical_lifecycle.py` after cleanup to bind the exact source,
current source, source delta, restore, `gcp_staging` parity, ID-free reasoning,
reasoning-compute attestation, and cleanup receipts into one fail-closed
preparation verdict. Caller-writable JSON can never emit T3 from this API;
even an internally valid bundle returns
`prepared_external_attestation_required`, `current_tier=T2`, and
`accepted_by_this_api=false`. Pass `--max-postflight-age-seconds 900` and
`--max-lifecycle-age-seconds 3600`; the latter bounds both cleanup-to-verdict
age and restore-to-postflight span.
With `TARGET_DB`, `RESTORE_RUN_ID`, and `PRIVATE_RUN_DIR` set as shown in the
runbook, run the replay-host attestation after the blind receipt passes and
before the receipt-bound cleanup:
```bash
sudo python3 ops/attest_gcp_reasoning_compute.py \
--reasoning-receipt "${PRIVATE_RUN_DIR}/blind-reasoning-receipt.json" \
--restore-run-id "$RESTORE_RUN_ID" \
--target-db "$TARGET_DB" \
--project teleo-501523 \
--expected-compute-instance teleo-prod-1 \
--expected-compute-zone europe-west6-a \
--expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \
--expected-restore-service-account sa-teleo-restore-drill@teleo-501523.iam.gserviceaccount.com \
--output "${PRIVATE_RUN_DIR}/reasoning-compute-attestation.json"
```
The replay must use the Hermes virtualenv, not system Python:
```bash
/home/teleo/.hermes/hermes-agent/venv/bin/python \
scripts/run_gcp_generated_db_direct_claim_suite.py ...
```
The lifecycle is not complete until the generated clone, temporary profile,
children, transient upload/bundle directories, and any temporary client are
absent. Preserve the private receipt run directory and every useful lifecycle
input, supporting receipt, and verifier output; do not treat that evidence as
transient cleanup. The canonical restore streams through `pg_restore` and
creates no GCS import object.
The current snapshot and restore commands intentionally use `--no-owner` and
`--no-acl`/`--no-privileges`. The expanded manifest therefore fails GCP parity
until a separately reviewed Cloud-SQL-compatible authorization replay restores
the captured owner/grant semantics. Do not bypass this by dropping the new
manifest rows or by running the Docker/superuser gate bootstrap blindly on
shared Cloud SQL.
Static inspection or a dry run is insufficient for these helpers: verify a new
capture against the live VPS and run the focused tests. A current T3 claim also
requires a separate independently authenticated provider/platform receipt that
the bundle author cannot mint.
## Safety And Claim Ceiling
- Do not send Telegram messages from the GCP parity lane.
- Do not apply, approve, or stage KB changes during read/reasoning replay.
- Do not restart the live GCP gateway for tool-file synchronization unless a
separate restart window is explicitly requested.
- Do not call control-plane inventory, memory sync, route readiness, or a
nominal scorer pass full m3taversal parity.
The strongest claim emitted by the repository lifecycle verifier is T2
structural consistency. Exact DB parity plus real no-send model replies with
truthful counts and cleanup still requires independent T3 attestation. Neither proves Telegram
delivery, GCP canonical mutation, ongoing replication, or production cutover.
## Current Access And Next Action
As of 2026-07-15, direct SSH and private TCP timed out, local `gcloud` for
`billy@livingip.xyz` requires reauthentication, and the intended operator
WIF/IAP authority is unavailable. The artifact-builder WIF identity must not be
expanded or mistaken for database-operator access. Current retained T2/source
and local restore proof does not satisfy the required T3 lifecycle.
The next action after an authorized operator route is restored is: newest
authorized VPS capture -> disposable private `teleo_clone_*` restore ->
generated private-connectivity receipt -> `gcp-parity.json` -> no-send
m3taversal blind reasoning -> live GCE reasoning attestation -> receipt-bound
clone cleanup -> postflight VPS capture/delta -> eight-receipt lifecycle verdict
within the 900/3600-second freshness bounds. Do not promote
`teleo_canonical`, expose Cloud SQL publicly, send Telegram, or mutate canonical
knowledge.