teleo/teleo-infrastructure
8
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions 1

Merge main and repair GCP runtime integration

Browse source
This commit is contained in:
Codex Integration 2026-07-15 12:51:17 +02:00
commit 822438bd2f
42 changed files with 7165 additions and 347 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

156
.agents/skills/teleo-gcp-parity-ops/SKILL.md
View file

@ -8,45 +8,61 @@ description: Use for passwordless Teleo GCP VM access, private Cloud SQL canonic
## Working Target
Restore a copy of the VPS canonical Leo database to GCP, prove exact schema,
row, role, extension, performance, and private-connectivity parity, then run the
row, role membership, ownership, ACL, extension, performance, and
private-connectivity parity, then run the
real GCP Leo read/reasoning path without Telegram sends or DB mutation.
The required tier is T3: a live private GCP staging restore, readback,
reasoning, compute attestation, and cleanup receipt. Local restore proof or an
access-gate report does not satisfy this tier.
## Operator Paths
The direct alias is passwordless and was live-verified on 2026-07-14:
The direct alias was passwordless and live-verified on 2026-07-14:
```bash
ssh teleo-gcp-staging
```
It uses `/Users/user/.ssh/google_compute_engine` for the configured operator,
It uses `~/.ssh/google_compute_engine` for the configured operator,
disables password and keyboard-interactive authentication, and does not store a
Google password. `sudo -n` also works. The route remains
firewall-source-dependent, so verify `ssh -o BatchMode=yes teleo-gcp-staging true` before a
long run instead of assuming retained access is current.
Google password. That is historical proof, not current reachability. The route
remains firewall-source-dependent. On
2026-07-15, both direct SSH and the private TCP route timed out; verify
`ssh -o BatchMode=yes teleo-gcp-staging true` before a long run and fail closed
if it does not return successfully.
The intended secondary path is `.github/workflows/gcp-iap-operator.yml`, using
short-lived GitHub OIDC, IAP, OS Login, and fixed reviewed operations. It is
merged but not bootstrapped: workflow run `29208215340` failed at auth with
`invalid_target` because provider `teleo-iap-operator` is absent, disabled, or
deleted. Do not call this path working until a live `status` run passes.
merged but not bootstrapped: the current 2026-07-15 check still lacks the
reviewed operator WIF/IAP authority. Provider `teleo-iap-operator` is absent,
disabled, or deleted (`invalid_target`), and the available artifact identity is intentionally not
a Cloud SQL/Compute operator. Do not call this path working until a live `status` run passes
with the intended operator identity.
Verified target:
Historically verified and intended target; reverify before any new lifecycle:
- project `teleo-501523`;
- VM `teleo-prod-1` in `europe-west6-a`;
- Cloud SQL `teleo-pgvector-standby`, PostgreSQL 16.14;
- private endpoint `10.61.0.3:5432`, public IP disabled, TLS required;
- restore identity `sa-teleo-restore-drill@teleo-501523.iam.gserviceaccount.com`;
- service `leoclean-gcp-prod-parallel.service`.
Direct libpq connections must use `sslmode=verify-full`, a reviewed DNS name
covered by the server certificate, and a private regular CA file. The private
IP is `hostaddr`, not the certificate identity. Never downgrade to
`sslmode=require` or accept an arbitrary `*.iam.gserviceaccount.com` identity.
Never print the Cloud SQL or Google password. On the VM, use the attached service account
and Secret Manager through reviewed wrappers or a short-lived environment
variable, then unset it.
## Two Different Databases
- Canonical collective knowledge is Cloud SQL database `teleo_canonical`, with
`public.*` and `kb_stage.*` tables.
- Canonical collective knowledge remains VPS PostgreSQL database `teleo`, with
`public.*` and `kb_stage.*` tables. Persistent Cloud SQL database
`teleo_canonical` is a staging copy and has not been promoted.
- Hermes conversation continuity is `state.db` plus session JSONL files. The
`leoclean-cloudsql-memory-sync.service` snapshot path copies this runtime
memory; it does not populate or prove canonical claims, sources, evidence,
@ -54,7 +70,7 @@ variable, then unset it.
Do not describe a passing Hermes memory sync as canonical KB parity.
## Current Verified State - 2026-07-14
## Retained Historical Proof - 2026-07-14
- The newest captured VPS database has `39` tables and `52,167` rows, including
claims `1837`, sources `4145`, claim evidence `4670`, claim edges `4916`, and
@ -83,19 +99,25 @@ Do not describe a passing Hermes memory sync as canonical KB parity.
- Persistent GCP `teleo_canonical` remains the older staging copy measured at
`52,164` rows and `26` proposals. It has not been promoted or cut over.
## Open Least-Privilege Candidate
That historical clone predates the expanded ownership/ACL and independently
authenticated receipt contract. It remains useful historical row/schema proof,
but it is not current T3 proof under this skill.
## Open Least-Privilege Candidates
PR #148, `Scope GCP Leo runtime to least-privilege Cloud SQL access`, is open as
of the 2026-07-15 skill-pack reconciliation. It proposes scoped runtime roles,
secret access, fail-closed Cloud SQL behavior, deployment rollback, and positive
plus negative permission checks. Those branch files and proposed live outcomes
are candidate evidence only. Before using them, run:
of the 2026-07-15 skill-pack reconciliation. Draft PR #162, `Harden GCP Leo
runtime least-privilege proof`, is its current hardening successor. They propose
scoped runtime roles, secret access, fail-closed Cloud SQL behavior, deployment
rollback, and positive plus negative permission checks. Those branch files and
proposed live outcomes are candidate evidence only. Before using them, run:
```bash
gh pr view 148 --json state,mergedAt,mergeCommit,headRefName,url
gh pr view 162 --json state,mergedAt,mergeCommit,headRefName,url
```
Until the PR is merged and its runtime receipt passes, use only paths present on
Until a successor is merged and its runtime receipt passes, use only paths present on
canonical `main`, keep persistent GCP classified as staging, do not promote it,
and do not infer least-privilege cutover from a branch or dry run.
@ -114,7 +136,8 @@ Track these independently:
1. control-plane project, VM, Cloud SQL, private-IP, and TLS identity;
2. canonical database schema, counts, row hashes, constraints, indexes,
functions, extensions, roles, and performance;
functions, extensions, exact roles and memberships, owners, normalized
schema/relation/column/function/type ACLs, and performance;
3. GCP service PID/restarts plus live profile and tool hashes;
4. `DC-01` through `DC-06` DB-read readiness;
5. real no-send model replies, strict score, and exact count consistency;
@ -126,17 +149,60 @@ Track these independently:
Use:
- `ops/capture_vps_canonical_postgres_snapshot.py` for a single source dump and
manifest;
manifest, with a required retained `--authorization-ref` and v2 provenance
receipt binding the exported snapshot identity plus dump/manifest hashes; it
requires identical healthy structured source-service states before/after
(`active/running`, positive PID, nonnegative restart count);
- `ops/restore_gcp_generated_postgres_snapshot.py restore --execute` for a
receipt-bound private-TLS restore into a bounded `teleo_clone_*` database;
pass the exact capture `--source-receipt`, and require its live Cloud SQL
control-plane check to prove public IP disabled before clone creation; it
requires the exact restore service account plus `--ssl-server-name` and
`--ssl-root-cert` for certificate-authenticated `verify-full` TLS; it
writes `target-manifest.jsonl`, `restore-receipt.json`, and the generated
`gcp-private-connectivity.json` under
`/var/lib/teleo-gcp-restore-runs/<restore-run-id>/`;
- `ops/restore_gcp_generated_postgres_snapshot.py cleanup --execute` for exact
clone cleanup with live-service and rollback readback;
- `ops/postgres_parity_manifest.sql` for row/catalog/role/performance readback;
- `ops/verify_postgres_parity_manifest.py --scope gcp_staging` for exact parity;
- `ops/verify_postgres_parity_manifest.py --scope gcp_staging` for exact parity,
always retaining the canonical output name `gcp-parity.json`;
- `scripts/run_gcp_generated_db_direct_claim_suite.py` for the adapter-free,
read-only, no-send six-response replay against a generated `teleo_clone_*`.
- `scripts/run_gcp_generated_db_blind_claim_canary.py` for the bounded ID-free
claim challenge, source receipt, reasoning, no-write, and cleanup proof.
- `ops/attest_gcp_reasoning_compute.py` on the replay VM after blind reasoning
and before cleanup, binding the reasoning receipt SHA-256 to live GCE metadata
in `reasoning-compute-attestation.json`;
- a second authorized source capture after cleanup, followed by
`ops/verify_vps_canonical_snapshot_delta.py`, to retain the explicit
post-snapshot VPS delta in `source-delta-receipt.json`;
- `ops/verify_gcp_canonical_lifecycle.py` after cleanup to bind the exact source,
current source, source delta, restore, `gcp_staging` parity, ID-free reasoning,
reasoning-compute attestation, and cleanup receipts into one fail-closed
preparation verdict. Caller-writable JSON can never emit T3 from this API;
even an internally valid bundle returns
`prepared_external_attestation_required`, `current_tier=T2`, and
`accepted_by_this_api=false`. Pass `--max-postflight-age-seconds 900` and
`--max-lifecycle-age-seconds 3600`; the latter bounds both cleanup-to-verdict
age and restore-to-postflight span.
With `TARGET_DB`, `RESTORE_RUN_ID`, and `PRIVATE_RUN_DIR` set as shown in the
runbook, run the replay-host attestation after the blind receipt passes and
before the receipt-bound cleanup:
```bash
sudo python3 ops/attest_gcp_reasoning_compute.py \
--reasoning-receipt "${PRIVATE_RUN_DIR}/blind-reasoning-receipt.json" \
--restore-run-id "$RESTORE_RUN_ID" \
--target-db "$TARGET_DB" \
--project teleo-501523 \
--expected-compute-instance teleo-prod-1 \
--expected-compute-zone europe-west6-a \
--expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \
--expected-restore-service-account sa-teleo-restore-drill@teleo-501523.iam.gserviceaccount.com \
--output "${PRIVATE_RUN_DIR}/reasoning-compute-attestation.json"
```
The replay must use the Hermes virtualenv, not system Python:
@ -145,8 +211,24 @@ The replay must use the Hermes virtualenv, not system Python:
scripts/run_gcp_generated_db_direct_claim_suite.py ...
```
The replay is not complete until the generated clone, temporary profile,
children, upload/run directories, and any temporary client are absent.
The lifecycle is not complete until the generated clone, temporary profile,
children, transient upload/bundle directories, and any temporary client are
absent. Preserve the private receipt run directory and every useful lifecycle
input, supporting receipt, and verifier output; do not treat that evidence as
transient cleanup. The canonical restore streams through `pg_restore` and
creates no GCS import object.
The current snapshot and restore commands intentionally use `--no-owner` and
`--no-acl`/`--no-privileges`. The expanded manifest therefore fails GCP parity
until a separately reviewed Cloud-SQL-compatible authorization replay restores
the captured owner/grant semantics. Do not bypass this by dropping the new
manifest rows or by running the Docker/superuser gate bootstrap blindly on
shared Cloud SQL.
Static inspection or a dry run is insufficient for these helpers: verify a new
capture against the live VPS and run the focused tests. A current T3 claim also
requires a separate independently authenticated provider/platform receipt that
the bundle author cannot mint.
## Safety And Claim Ceiling
@ -157,18 +239,24 @@ children, upload/run directories, and any temporary client are absent.
- Do not call control-plane inventory, memory sync, route readiness, or a
nominal scorer pass full m3taversal parity.
The strongest accepted claim requires exact DB parity plus real no-send model
replies with truthful counts and cleanup. It still does not prove Telegram
The strongest claim emitted by the repository lifecycle verifier is T2
structural consistency. Exact DB parity plus real no-send model replies with
truthful counts and cleanup still requires independent T3 attestation. Neither proves Telegram
delivery, GCP canonical mutation, ongoing replication, or production cutover.
## Current Access And Next Action
Direct SSH and passwordless sudo are currently working. Local `gcloud` still
requires account reauthentication for control-plane metadata, so the current
private-IP/TLS proof comes from a live database connection while the public-IP-
disabled control-plane receipt remains dated 2026-07-12.
As of 2026-07-15, direct SSH and private TCP timed out, local `gcloud` for
`billy@livingip.xyz` requires reauthentication, and the intended operator
WIF/IAP authority is unavailable. The artifact-builder WIF identity must not be
expanded or mistaken for database-operator access. Current retained T2/source
and local restore proof does not satisfy the required T3 lifecycle.
The next production decision is not another restore drill. It is whether to
promote a newly verified snapshot into persistent GCP `teleo_canonical` and
repoint the production read adapter. Until that decision is explicit, keep GCP
classified as staging and do not expose Cloud SQL publicly.
The next action after an authorized operator route is restored is: newest
authorized VPS capture -> disposable private `teleo_clone_*` restore ->
generated private-connectivity receipt -> `gcp-parity.json` -> no-send
m3taversal blind reasoning -> live GCE reasoning attestation -> receipt-bound
clone cleanup -> postflight VPS capture/delta -> eight-receipt lifecycle verdict
within the 900/3600-second freshness bounds. Do not promote
`teleo_canonical`, expose Cloud SQL publicly, send Telegram, or mutate canonical
knowledge.

11
.github/workflows/ci.yml vendored
View file

@ -42,19 +42,24 @@ jobs:
lib/llm.py \
lib/post_extract.py \
ops/apply_gcp_iam_split.py \
ops/attest_gcp_reasoning_compute.py \
ops/capture_vps_canonical_postgres_snapshot.py \
ops/check_gcp_infra_readiness.py \
ops/run_gcp_infra_execute_canary.py \
ops/apply_gcp_runtime_baseline.py \
ops/check_gcp_service_communications.py \
ops/plan_gcp_iam_split.py \
ops/private_receipt_io.py \
ops/run_local_canonical_postgres_rebuild.py \
ops/redact_sqlite_postgres_restore_canary.py \
ops/restore_gcp_generated_postgres_snapshot.py \
ops/sqlite_to_postgres_dump.py \
ops/verify_gcp_cloudsql_restore_readback.py \
ops/verify_gcp_leoclean_runtime_permissions.py \
ops/verify_gcp_leoclean_service_environment.py \
ops/verify_gcp_canonical_lifecycle.py \
ops/verify_postgres_parity_manifest.py \
ops/verify_vps_canonical_snapshot_delta.py \
telegram/approvals.py \
hermes-agent/leoclean-bin/kb_tool.py \
hermes-agent/leoclean-bin/cloudsql_memory_tool.py \
@ -75,6 +80,7 @@ jobs:
scripts/working_leo_m3taversal_oos_benchmark.py \
scripts/working_leo_open_ended_benchmark.py \
tests/test_agent_routing.py \
tests/test_attest_gcp_reasoning_compute.py \
tests/test_assemble_telegram_visible_direct_claim_capture_receipt.py \
tests/test_build_working_leo_m3taversal_outcome_sandbox.py \
tests/test_decision_engine_replay.py \
@ -102,10 +108,15 @@ jobs:
tests/test_verify_leo_db_first_oos_canary.py \
tests/test_compile_kb_source_packet.py \
tests/test_verify_postgres_parity_manifest.py \
tests/test_verify_gcp_canonical_lifecycle.py \
tests/test_verify_vps_canonical_snapshot_delta.py \
tests/test_working_leo_m3taversal_oos_benchmark.py \
tests/test_working_leo_open_ended_benchmark.py \
tests/test_phase1b_end_to_end.py \
tests/test_private_receipt_io.py \
tests/test_restore_gcp_generated_postgres_snapshot.py \
tests/test_run_local_canonical_postgres_rebuild.py \
tests/test_run_local_genesis_ledger_rebuild.py \
tests/test_sqlite_to_postgres_dump.py \
tests/test_sqlite_postgres_restore_canary_capsule.py \
tests/test_teleo_agent_systemd.py \

176
.github/workflows/gcp-observatory-read-adapter.yml vendored
View file

@ -4,12 +4,13 @@ on:
workflow_dispatch:
inputs:
action:
description: Build only, or deploy the protected staging service and run live receipts
description: Build, preflight, or deploy the protected staging service and run live receipts
required: true
default: build_only
type: choice
options:
- build_only
- preflight_staging
- deploy_staging
permissions:
@ -28,10 +29,13 @@ env:
IMAGE_NAME: observatory-read-adapter
SERVICE_NAME: observatory-read-adapter-staging
RUNTIME_SERVICE_ACCOUNT: sa-observatory-read-adapter@teleo-501523.iam.gserviceaccount.com
BUILD_SERVICE_ACCOUNT: sa-artifact-builder@teleo-501523.iam.gserviceaccount.com
DEPLOY_SERVICE_ACCOUNT: sa-observatory-deployer@teleo-501523.iam.gserviceaccount.com
DB_IAM_USER: sa-observatory-read-adapter@teleo-501523.iam
API_KEY_SECRET: observatory-read-api-key-staging
WORKLOAD_IDENTITY_PROVIDER: projects/785938879453/locations/global/workloadIdentityPools/github-actions/providers/living-ip-github
DEPLOY_SERVICE_ACCOUNT: sa-artifact-builder@teleo-501523.iam.gserviceaccount.com
BUILD_WORKLOAD_IDENTITY_PROVIDER: projects/785938879453/locations/global/workloadIdentityPools/github-actions/providers/living-ip-github
DEPLOY_WORKLOAD_IDENTITY_PROVIDER: projects/785938879453/locations/global/workloadIdentityPools/github-actions/providers/observatory-read-adapter-main
EXPECTED_WORKFLOW_REF: living-ip/teleo-infrastructure/.github/workflows/gcp-observatory-read-adapter.yml@refs/heads/main
jobs:
build:
@ -57,8 +61,8 @@ jobs:
- id: auth
uses: google-github-actions/auth@v3
with:
workload_identity_provider: ${{ env.WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ env.DEPLOY_SERVICE_ACCOUNT }}
workload_identity_provider: ${{ env.BUILD_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ env.BUILD_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v3
with:
@ -68,28 +72,21 @@ jobs:
run: gcloud auth configure-docker "${REGION}-docker.pkg.dev" --quiet
- id: image
name: Build, smoke, and push immutable image
name: Build, smoke, and push run-unique immutable image
shell: bash
run: |
set -euo pipefail
tag="${GITHUB_SHA::12}"
tag="${GITHUB_SHA}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
image="${REGION}-docker.pkg.dev/${PROJECT_ID}/${REPOSITORY}/${IMAGE_NAME}:${tag}"
digest="$(gcloud artifacts docker images describe "${image}" \
--project="${PROJECT_ID}" \
--format='value(image_summary.digest)' 2>/dev/null || true)"
if [[ -n "${digest}" ]]; then
docker pull "${image}"
else
docker build -f Dockerfile.observatory-read-adapter -t "${image}" .
fi
docker build -f Dockerfile.observatory-read-adapter -t "${image}" .
docker run --rm --env PYTHONPYCACHEPREFIX=/tmp/pycache \
"${image}" python -m compileall -q /app/observatory_read_adapter
docker run --rm "${image}" python -c 'import observatory_read_adapter; print("adapter-import-ok")'
if [[ -z "${digest}" ]]; then
docker push "${image}" | tee docker-push.log
digest="$(awk '/digest: sha256:/ {print $3}' docker-push.log | tail -1)"
fi
test -n "${digest}"
docker push "${image}"
digest="$(gcloud artifacts docker images describe "${image}" \
--project="${PROJECT_ID}" \
--format='value(image_summary.digest)')"
[[ "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]]
image_ref="${image}@${digest}"
printf 'image_ref=%s\n' "${image_ref}" >> "${GITHUB_OUTPUT}"
printf 'revision=%s\nimage_ref=%s\n' "${GITHUB_SHA}" "${image_ref}" > observatory-adapter-image.txt
@ -100,31 +97,94 @@ jobs:
path: observatory-adapter-image.txt
if-no-files-found: error
deploy-staging:
name: Deploy and prove staging read path
if: ${{ inputs.action == 'deploy_staging' }}
preflight-staging:
name: Verify private staging prerequisites
if: ${{ inputs.action != 'build_only' }}
needs: build
runs-on: ubuntu-latest
timeout-minutes: 20
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- name: Enforce reviewed main-only staging path
shell: bash
run: |
set -euo pipefail
[[ "${GITHUB_REF}" == "refs/heads/main" ]]
[[ "${GITHUB_WORKFLOW_REF}" == "${EXPECTED_WORKFLOW_REF}" ]]
[[ "${GITHUB_EVENT_NAME}" == "workflow_dispatch" ]]
- id: auth
uses: google-github-actions/auth@v3
with:
workload_identity_provider: ${{ env.WORKLOAD_IDENTITY_PROVIDER }}
workload_identity_provider: ${{ env.DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ env.DEPLOY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v3
with:
project_id: ${{ env.PROJECT_ID }}
- name: Deploy bounded GCP staging service
- name: Run adapter-specific read-only GCP preflight
shell: bash
env:
IMAGE_REF: ${{ needs.build.outputs.image_ref }}
run: |
set -euo pipefail
python3 ops/check_observatory_read_adapter_gcp_preflight.py \
--image-ref "${IMAGE_REF}" \
--output observatory-read-adapter-preflight.json
- uses: actions/upload-artifact@v4
if: always()
with:
name: observatory-read-adapter-preflight
path: observatory-read-adapter-preflight.json
if-no-files-found: error
deploy-staging:
name: Deploy and prove staging read path
if: ${{ inputs.action == 'deploy_staging' }}
needs:
- build
- preflight-staging
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v4
- name: Enforce reviewed main-only staging path
shell: bash
run: |
set -euo pipefail
[[ "${GITHUB_REF}" == "refs/heads/main" ]]
[[ "${GITHUB_WORKFLOW_REF}" == "${EXPECTED_WORKFLOW_REF}" ]]
[[ "${GITHUB_EVENT_NAME}" == "workflow_dispatch" ]]
- id: auth
uses: google-github-actions/auth@v3
with:
workload_identity_provider: ${{ env.DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ env.DEPLOY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v3
with:
project_id: ${{ env.PROJECT_ID }}
- id: deploy
name: Deploy bounded GCP staging service
shell: bash
env:
IMAGE_REF: ${{ needs.build.outputs.image_ref }}
run: |
set -euo pipefail
api_key_version="$(gcloud secrets versions list "${API_KEY_SECRET}" \
--project="${PROJECT_ID}" \
--filter='state=ENABLED' \
--sort-by='~createTime' \
--limit=1 \
--format='value(name)')"
api_key_version="${api_key_version##*/}"
[[ "${api_key_version}" =~ ^[0-9]+$ ]]
gcloud run deploy "${SERVICE_NAME}" \
--project="${PROJECT_ID}" \
--region="${REGION}" \
@ -135,39 +195,69 @@ jobs:
--subnet=teleo-staging-europe-west6 \
--vpc-egress=private-ranges-only \
--ingress=all \
--allow-unauthenticated \
--no-invoker-iam-check \
--port=8080 \
--cpu=1 \
--memory=512Mi \
--concurrency=8 \
--max-instances=2 \
--timeout=15 \
--set-secrets="OBSERVATORY_API_KEY=${API_KEY_SECRET}:latest" \
--set-secrets="OBSERVATORY_API_KEY=${API_KEY_SECRET}:${api_key_version}" \
--set-env-vars="GCP_PROJECT_ID=${PROJECT_ID},CLOUD_SQL_INSTANCE=${PROJECT_ID}:${REGION}:teleo-pgvector-standby,DB_NAME=teleo_canonical,DB_IAM_USER=${DB_IAM_USER},DB_AUTHORIZATION_ROLE=kb_observatory_read,SERVICE_REVISION=${GITHUB_SHA}" \
--labels="livingip-tier=gcp-staging,livingip-surface=observatory-read-adapter" \
--quiet
printf 'api_key_version=%s\n' "${api_key_version}" >> "${GITHUB_OUTPUT}"
- name: Capture positive and negative live receipts
shell: bash
env:
API_KEY_VERSION: ${{ steps.deploy.outputs.api_key_version }}
EXPECTED_IMAGE_REF: ${{ needs.build.outputs.image_ref }}
run: |
set -euo pipefail
service_url="$(gcloud run services describe "${SERVICE_NAME}" --project="${PROJECT_ID}" --region="${REGION}" --format='value(status.url)')"
revision="$(gcloud run services describe "${SERVICE_NAME}" --project="${PROJECT_ID}" --region="${REGION}" --format='value(status.latestReadyRevisionName)')"
api_key="$(gcloud secrets versions access latest --secret="${API_KEY_SECRET}" --project="${PROJECT_ID}")"
service_json="$(gcloud run services describe "${SERVICE_NAME}" --project="${PROJECT_ID}" --region="${REGION}" --format=json)"
service_url="$(jq -r '.status.url // empty' <<<"${service_json}")"
revision="$(jq -r '.status.latestReadyRevisionName // empty' <<<"${service_json}")"
runtime_service_account="$(jq -r '.spec.template.spec.serviceAccountName // .template.serviceAccount // empty' <<<"${service_json}")"
deployed_image="$(jq -r '.spec.template.spec.containers[0].image // .template.containers[0].image // empty' <<<"${service_json}")"
deployed_secret_version="$(jq -r '
[
.spec.template.spec.containers[]?.env[]?
| select(.name == "OBSERVATORY_API_KEY")
| .valueFrom.secretKeyRef.key
][0] // [
.template.containers[]?.env[]?
| select(.name == "OBSERVATORY_API_KEY")
| .valueSource.secretKeyRef.version
][0] // empty
' <<<"${service_json}")"
expected_digest="${EXPECTED_IMAGE_REF##*@}"
api_key="$(gcloud secrets versions access "${API_KEY_VERSION}" --secret="${API_KEY_SECRET}" --project="${PROJECT_ID}")"
test -n "${service_url}"
test -n "${revision}"
test "${runtime_service_account}" = "${RUNTIME_SERVICE_ACCOUNT}"
test "${deployed_secret_version}" = "${API_KEY_VERSION}"
[[ "${deployed_image}" == "${REGION}-docker.pkg.dev/${PROJECT_ID}/${REPOSITORY}/${IMAGE_NAME}@${expected_digest}" ]]
test -n "${api_key}"
curl --fail --silent --show-error \
--header "X-Api-Key: ${api_key}" \
"${service_url}/v1/claims/sample" > positive.json
jq -e '
jq -e \
--arg project_id "${PROJECT_ID}" \
--arg cloud_sql_instance "${PROJECT_ID}:${REGION}:teleo-pgvector-standby" \
--arg db_iam_user "${DB_IAM_USER}" \
--arg git_sha "${GITHUB_SHA}" '
.schema == "livingip.observatory-canonical-claim.v1"
and .read_only == true
and .provenance.gcp_project == $project_id
and .provenance.cloud_sql_instance == $cloud_sql_instance
and .provenance.database == "teleo_canonical"
and .provenance.database_principal == $db_iam_user
and .provenance.authorization_role == "kb_observatory_read"
and .provenance.transaction_read_only == true
and .provenance.write_privileges_denied == true
and .provenance.service_revision == $git_sha
and (.canonical.evidence | length) > 0
and .proposal_ledger.distinct_from_canonical == true
' positive.json >/dev/null
@ -178,12 +268,23 @@ jobs:
--data '{"status":"applied"}' "${service_url}/v1/claims/sample")"
test "${anonymous_status}" = 401
test "${write_status}" = 405
curl --fail --silent --show-error \
--header "X-Api-Key: ${api_key}" \
"${service_url}/v1/claims/sample" > after-write-attempt.json
cmp <(jq -S . positive.json) <(jq -S . after-write-attempt.json)
jq -f ops/redact_observatory_read_adapter_receipt.jq \
positive.json > positive-redacted.json
jq -n \
--arg service_url "${service_url}" \
--arg revision "${revision}" \
--arg git_sha "${GITHUB_SHA}" \
--argjson positive "$(cat positive.json)" \
--arg runtime_service_account "${runtime_service_account}" \
--arg database_principal "${DB_IAM_USER}" \
--arg deployed_image "${deployed_image}" \
--arg api_key_secret_version "${API_KEY_VERSION}" \
--argjson positive "$(cat positive-redacted.json)" \
--arg anonymous_status "${anonymous_status}" \
--arg write_status "${write_status}" \
'{
@ -192,14 +293,21 @@ jobs:
service_url: $service_url,
revision: $revision,
git_sha: $git_sha,
runtime_service_account: $runtime_service_account,
database_principal: $database_principal,
deployed_image: $deployed_image,
api_key_secret_version: $api_key_secret_version,
positive: $positive,
negative: {
anonymous_http_status: ($anonymous_status | tonumber),
authenticated_post_http_status: ($write_status | tonumber)
authenticated_post_http_status: ($write_status | tonumber),
canonical_state_unchanged_after_post: true,
database_write_privileges_denied: $positive.provenance.write_privileges_denied
},
production_repoint_executed: false
}' > observatory-read-adapter-live-receipt.json
unset api_key
rm -f positive.json positive-redacted.json after-write-attempt.json anonymous.json write.json
- uses: actions/upload-artifact@v4
with:

275
docs/gcp-kb-cloudsql-restore-runbook.md
View file

@ -53,12 +53,21 @@ python3 ops/capture_vps_canonical_postgres_snapshot.py \
--ssh-target root@77.42.65.182 \
--ssh-key ~/.ssh/livingip_hetzner_20260604_ed25519 \
--run-id canonical-<timestamp> \
--authorization-ref codex-delegation:THREAD_ID \
--output-dir <private-output-dir>
```
The capture fails closed if the source service changes while it runs. It
Replace `THREAD_ID` with the exact retained authorization reference for the
capture. The v2 receipt binds that reference, exported snapshot ID, TXID
snapshot, WAL LSN, source system identifier, dump hash, manifest hash, reviewed
manifest-SQL hash, and source-context hash. A dump and manifest without this
passing receipt are not eligible for GCP restore.
The capture fails closed unless the structured source-service states before
and after the snapshot are identical and healthy: `ActiveState=active`,
`SubState=running`, a positive `MainPID`, and a nonnegative `NRestarts`. It
retains a private custom dump, dump SHA-256, object TOC, catalog/data manifest,
and before/after service state. It never restarts Leo or writes to the source
and both service-state objects. It never restarts Leo or writes to the source
database.
Prove that this exact snapshot can rebuild a blank Postgres target before using
@ -77,35 +86,207 @@ network mode `none` and tmpfs-only database storage. It waits for an actual
--no-owner --no-privileges --exit-on-error`, compares the full parity manifest,
then removes the container and proves it is absent. A passing local receipt is
the exact-recovery preflight; it is not semantic recompilation from raw source
documents.
documents. Full parity now includes `kb_gate_owner`, `kb_apply`, and
`kb_review`, their memberships, object ownership, and normalized database,
schema, relation, column, function, and type ACLs. A restore that recovers rows
but not those authorization semantics fails closed.
Run `ops/postgres_parity_manifest.sql` against the isolated restored target,
then compare source and target:
Choose one bounded suffix and use it consistently on the GCP replay VM. The
restore helper retains its mode-0700 evidence directory at the fixed private
root:
```bash
python3 ops/verify_postgres_parity_manifest.py \
--source <private-output-dir>/source-manifest.jsonl \
--target <private-output-dir>/target-manifest.jsonl \
--scope gcp_staging \
--connectivity-proof <private-output-dir>/gcp-private-connectivity.json \
--output <private-output-dir>/gcp-parity.json
RUN_SUFFIX=20260715t010000z
TARGET_DB="teleo_clone_${RUN_SUFFIX}"
RESTORE_RUN_ID="gcp-restore-${RUN_SUFFIX}"
PRIVATE_RUN_DIR="/var/lib/teleo-gcp-restore-runs/${RESTORE_RUN_ID}"
```
The verifier checks all table row counts and collation-independent row hashes,
plus schemas, columns/defaults, constraints, indexes, sequences, views,
functions, triggers, enum/domain types, policies, required extensions,
password-free application-role attributes, and bounded query timings. In GCP
scope it also requires a receipt proving a staging compute source, a private
server address, TLS, and public-IP-disabled instance metadata.
Use a generated target database such as `teleo_clone_<run_id>`. Never import a
drill into `teleo`, `teleo_kb`, or `teleo_canonical`. Database isolation does
not isolate cluster-global roles or extensions, so verify those separately and
do not run the Docker-only gate bootstrap against the shared Cloud SQL instance.
After the parity verifier passes, run the no-send operator composition replay from
staging compute against that generated database. Only then delete the generated
database and uploaded import object and retain cleanup readback.
The canonical restore command must include the v2 capture receipt. It also runs
a live Cloud SQL control-plane preflight before resolving the database secret or
creating the clone, and fails closed unless the exact instance is runnable,
PostgreSQL 16, private-networked, bound to the expected RFC1918 host, and has
public IP disabled. A GCE metadata preflight independently binds the operator
runtime to `teleo-prod-1` in `europe-west6-a` on `teleo-staging-net`; a
hard-coded compute label is not accepted. It also requires the exact
least-privilege restore identity and libpq `verify-full` using a reviewed server
name and private CA file. Determine the certificate DNS SAN from the live Cloud
SQL certificate before filling `--ssl-server-name`; do not substitute the
private IP or downgrade TLS verification:
```bash
sudo python3 ops/restore_gcp_generated_postgres_snapshot.py restore \
--execute \
--target-db "$TARGET_DB" \
--run-id "$RESTORE_RUN_ID" \
--dump <private-output-dir>/teleo-canonical.dump \
--expected-dump-sha256 <dump-sha256> \
--source-manifest <private-output-dir>/source-manifest.jsonl \
--source-receipt <private-output-dir>/receipt.json \
--expected-source-receipt-sha256 <source-receipt-sha256> \
--expected-capture-authorization-ref codex-delegation:THREAD_ID \
--project teleo-501523 \
--cloudsql-instance teleo-pgvector-standby \
--expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \
--expected-compute-instance teleo-prod-1 \
--expected-compute-zone europe-west6-a \
--expected-restore-service-account sa-teleo-restore-drill@teleo-501523.iam.gserviceaccount.com \
--ssl-server-name <dns-name-covered-by-cloudsql-server-certificate> \
--ssl-root-cert /etc/teleo/cloudsql/server-ca.pem \
--expected-source-ssh-target root@77.42.65.182 \
--expected-source-container teleo-pg \
--expected-source-database teleo \
--expected-source-service leoclean-gateway.service
```
The restore writes both `target-manifest.jsonl` and
`gcp-private-connectivity.json` as mode-0600 files in its private run
directory. The latter is composed from live GCE metadata, Cloud SQL
control-plane checks, and the private TLS database identity; do not hand-compose
the connectivity receipt. The restore also records identical healthy
before/after state for `leoclean-gcp-prod-parallel.service`, including its PID
and restart count. Receipt payloads remain in mode-0600 files; command stdout
contains only a status, output path, and receipt hash.
Compare the captured target manifest and generated connectivity receipt to the
authorized source:
```bash
sudo python3 ops/verify_postgres_parity_manifest.py \
--source <private-output-dir>/source-manifest.jsonl \
--target "$PRIVATE_RUN_DIR/target-manifest.jsonl" \
--scope gcp_staging \
--connectivity-proof "$PRIVATE_RUN_DIR/gcp-private-connectivity.json" \
--output "$PRIVATE_RUN_DIR/gcp-parity.json"
```
The verifier checks all table row counts and collation-independent row hashes,
plus schemas, columns/defaults, constraints, indexes, sequences, views,
functions, triggers, enum/domain types, policies, required extensions,
password-free application-role attributes, exact role memberships, object
owners, normalized ACLs, and bounded query timings. Role and extension sets are
symmetric: an unexpected extra role, superuser, or extension is a mismatch.
Every table must have a nonempty row hash. In GCP scope it also requires a
receipt proving a staging compute source, a private server address,
certificate-authenticated `verify-full` TLS, and public-IP-disabled instance
metadata.
The snapshot dump intentionally strips owners and ACLs. Therefore the GCP
restore cannot pass the expanded parity gate until a separately reviewed,
Cloud-SQL-compatible authorization replay reconstructs the captured ownership
and grants. Do not run the Docker/superuser bootstrap blindly on shared Cloud
SQL. Until that replay exists, this lane remains T2 preparation and must fail
before a T3 claim.
After the parity verifier passes, run the no-send m3taversal blind reasoning
replay from staging compute against that generated database, attest the replay
host from live GCE metadata, and only then delete the generated database. This
canonical path streams the custom dump through `pg_restore`; it does not create
a GCS import object.
Run the bounded ID-free reasoning canary with the generated parity receipt. The
script re-executes itself in the Hermes virtualenv when that runtime is present:
```bash
sudo python3 scripts/run_gcp_generated_db_blind_claim_canary.py \
--execute \
--target-db "$TARGET_DB" \
--cloudsql-tool hermes-agent/leoclean-bin/cloudsql_memory_tool.py \
--manifest-sql ops/postgres_parity_manifest.sql \
--parity-receipt "$PRIVATE_RUN_DIR/gcp-parity.json" \
--output "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \
--run-id "gcp-blind-claim-${RUN_SUFFIX}"
```
Before cleanup, bind that reasoning receipt to live metadata from the same GCE
replay host:
```bash
sudo python3 ops/attest_gcp_reasoning_compute.py \
--reasoning-receipt "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \
--restore-run-id "$RESTORE_RUN_ID" \
--target-db "$TARGET_DB" \
--project teleo-501523 \
--expected-compute-instance teleo-prod-1 \
--expected-compute-zone europe-west6-a \
--expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \
--expected-restore-service-account sa-teleo-restore-drill@teleo-501523.iam.gserviceaccount.com \
--output "$PRIVATE_RUN_DIR/reasoning-compute-attestation.json"
```
After the compute attestation passes, run the receipt-bound cleanup command
retained by the restore. Then capture the VPS source again under a distinct
postflight authorization reference. Run this capture from the same trusted
source-access lane used for the baseline; it need not run on the GCP replay VM.
The postflight capture must complete after cleanup, so the final verdict can
state whether the VPS changed after the restored snapshot boundary. Compare the
two source captures:
```bash
python3 ops/capture_vps_canonical_postgres_snapshot.py \
--execute \
--ssh-target root@77.42.65.182 \
--ssh-key ~/.ssh/livingip_hetzner_20260604_ed25519 \
--run-id canonical-${RUN_SUFFIX}-postflight \
--authorization-ref codex-delegation:THREAD_ID:postflight \
--output-dir <private-postflight-dir>
```
```bash
sudo python3 ops/verify_vps_canonical_snapshot_delta.py \
--baseline-receipt <private-output-dir>/receipt.json \
--baseline-manifest <private-output-dir>/source-manifest.jsonl \
--baseline-authorization-ref codex-delegation:THREAD_ID \
--current-receipt <private-postflight-dir>/receipt.json \
--current-manifest <private-postflight-dir>/source-manifest.jsonl \
--current-authorization-ref codex-delegation:THREAD_ID:postflight \
--output "$PRIVATE_RUN_DIR/source-delta-receipt.json"
```
Finally, verify that all eight receipts belong to one bound lifecycle. This
verifier rejects local-scope parity, stale or mismatched source hashes, a
postflight source capture that predates reasoning or cleanup, hand-authored
reasoning booleans without real retrieval/tool receipts, an unattested replay
host, compute or Cloud SQL topology drift, and any cleanup receipt that leaves
the clone present or claims cleanup when the named clone never existed. Both
source captures must use the same reviewed manifest SQL. The postflight capture may be at most 900 seconds old at
verification; the GCP lifecycle age and restore-to-postflight span must each be
at most 3600 seconds:
```bash
sudo python3 ops/verify_gcp_canonical_lifecycle.py \
--source-receipt <private-output-dir>/receipt.json \
--current-source-receipt <private-postflight-dir>/receipt.json \
--source-delta-receipt "$PRIVATE_RUN_DIR/source-delta-receipt.json" \
--restore-receipt "$PRIVATE_RUN_DIR/restore-receipt.json" \
--parity-receipt "$PRIVATE_RUN_DIR/gcp-parity.json" \
--reasoning-receipt "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \
--reasoning-compute-receipt "$PRIVATE_RUN_DIR/reasoning-compute-attestation.json" \
--cleanup-receipt "$PRIVATE_RUN_DIR/cleanup-receipt.json" \
--max-postflight-age-seconds 900 \
--max-lifecycle-age-seconds 3600 \
--output "$PRIVATE_RUN_DIR/lifecycle-verification.json"
```
This command validates only the internal consistency of caller-supplied
receipts. A clean result is emitted as
`prepared_external_attestation_required`, with
`current_tier=T2_structural_receipt_bundle`, `t3_receipt_ready=false`, and
`accepted_by_this_api=false`. It cannot emit a live-GCP/T3 claim. T3 requires a
separate independently authenticated provider or platform receipt that the
bundle author cannot mint.
Retain the private receipt run directory and every useful lifecycle input,
supporting receipt, and verifier output. Remove only transient upload/bundle
directories, temporary Hermes profiles, helper clients, and child processes
after their absence is verified. There is no canonical-path GCS object to
delete. Do not remove the evidence directory until its useful receipts are
integrated or explicitly rejected.
## Legacy SQLite Source Backup Canary
@ -236,11 +417,15 @@ A successful restore or replication canary must retain:
- dump timestamp or replication slot timestamp;
- source schema/database name.
- transfer proof:
- dump object path in a versioned bucket, or logical replication subscription details;
- for the canonical path, the v2 capture receipt, dump SHA-256, and exact
private staging path used by streaming `pg_restore`;
- for a legacy import or replication path, a versioned dump-object generation
or logical replication subscription details;
- row/table counts before import where available.
- target proof:
- `teleo-pgvector-standby` readback;
- `teleo_kb` database readback;
- `teleo-pgvector-standby` control-plane readback;
- generated `teleo_clone_*` identity for canonical parity, or `teleo_kb` only
for the explicitly legacy SQLite drill;
- extension readback for `vector` if the restored schema needs pgvector;
- representative query readback for core KB tables.
- failure boundary:
@ -295,28 +480,26 @@ Retain only redacted connection metadata. Do not commit or paste credentials.
## Current Blocker
As of 2026-07-11, the canonical Postgres exported-snapshot capture and isolated
local restore parity pass. Live GCP restore and staging replay do not.
As of 2026-07-15, a newest authorized canonical Postgres capture and isolated
local restore can pass, but the required T3 live private GCP lifecycle cannot be
started from the current operator route:
- GitHub WIF works for `sa-artifact-builder`, but that identity is intentionally
limited to Artifact Registry and cannot inspect or mutate Cloud SQL/Compute.
- The configured `sa-teleo-readiness` and `sa-teleo-restore-drill` identities
return IAM 404 and do not exist.
- The local privileged `billy@livingip.xyz` gcloud session requires password
reauthentication. No password was entered or inspected.
- Direct VM SSH is closed to the current egress `/32`; IAP requires the same
privileged GCP authentication.
- direct VM SSH and the private TCP route timed out;
- local `gcloud` for `billy@livingip.xyz` requires reauthentication; no
password was entered or inspected;
- the intended operator WIF/IAP authority is unavailable because provider
`teleo-iap-operator` is absent, disabled, or deleted;
- the working artifact-builder WIF identity is intentionally limited and must
not be expanded or mistaken for Cloud SQL/Compute operator access.
That is why the readiness checker still reports:
The authorized recovery is a local
`gcloud auth login billy@livingip.xyz --force` without sharing credentials, or
administrator convergence of the reviewed operator WIF/IAP bootstrap. Until a
route passes live readback, keep the claim at T2/source-local proof; do not
promote staging or expose Cloud SQL publicly.
- `kb_source_restore_access = blocked`
- `kb_restore_or_replication = blocked`
The immediate operator CTA is to complete
`gcloud auth login billy@livingip.xyz --force` locally without sharing the
password, or apply the reviewed IAM split with an authorized GCP administrator.
The next non-user action is:
canonical `teleo` snapshot -> generated Cloud SQL database -> full parity and
private-connectivity verifier -> no-send Cory composition replay from staging
compute -> delete the generated database/object -> retain cleanup proof.
Once access is restored, run: newest authorized `teleo` snapshot -> disposable
private `teleo_clone_*` restore -> generated connectivity proof ->
`gcp-parity.json` -> no-send m3taversal blind reasoning -> live GCE reasoning
attestation -> receipt-bound clone cleanup -> postflight VPS capture and delta
-> eight-receipt lifecycle verdict within the 900/3600-second freshness bounds.

112
docs/observatory-read-adapter-gcp-staging.md
View file

@ -34,6 +34,13 @@ it does not change Argus or PR #148's Leo runtime files.
- Direct VPC egress: `teleo-staging-net` / `teleo-staging-europe-west6`.
- Cloud SQL: `teleo-501523:europe-west6:teleo-pgvector-standby`, private IP.
- Database: `teleo_canonical` only.
- Build identity: `sa-artifact-builder@teleo-501523.iam.gserviceaccount.com`;
image build/push only.
- Staging deploy/canary identity:
`sa-observatory-deployer@teleo-501523.iam.gserviceaccount.com`; an exact
workflow/main-only WIF provider, a five-permission Cloud Run custom role,
repository read, exact runtime `actAs`, and exact secret access. Never grant
deploy permissions to the artifact builder.
- Runtime identity: `sa-observatory-read-adapter@teleo-501523.iam.gserviceaccount.com`.
- IAM database user: `sa-observatory-read-adapter@teleo-501523.iam`.
- Database authorization role: `kb_observatory_read` (`NOLOGIN`).
@ -52,24 +59,103 @@ effective write-denial checks drift.
These are staging mutations. Run them only with an authenticated GCP operator;
they do not route production traffic.
1. Create the dedicated runtime service account and grant only
`roles/cloudsql.client` and `roles/cloudsql.instanceUser` in project
`teleo-501523`.
2. Add the service account as a Cloud SQL IAM service-account user on
`teleo-pgvector-standby`.
3. From the existing private GCP VM database-admin path, run
1. Reauthenticate the project operator, then run the first full direct check:
```bash
gcloud projects describe teleo-501523 --format=value(projectId)
python3 ops/check_gcp_infra_readiness.py
```
2. Review the exact dry-run packet. It enables only required APIs, creates
separate staging deploy/runtime identities, creates the dedicated
`observatory-read-adapter-main` WIF provider, keeps the artifact builder
narrow, scopes Cloud SQL connect/login to `teleo-pgvector-standby`, and
scopes secret access to `observatory-read-api-key-staging`:
```bash
python3 ops/plan_observatory_read_adapter_gcp.py
python3 ops/plan_observatory_read_adapter_gcp.py --format shell
```
The shell output is unexecuted and uses `set -euo pipefail`. Apply it only
after review with an authenticated `teleo-501523` administrator. The
deployer custom role contains only `run.services.create`,
`run.services.update`, `run.services.get`, `run.services.setIamPolicy`, and
`run.operations.get`; it contains no delete permission. Do not add
`run.admin`, `run.developer`, `cloudsql.admin`, `secretmanager.admin`,
`compute.admin`, Editor, or Owner to either lane identity.
3. Add the dedicated runtime service account as a Cloud SQL IAM
service-account user on `teleo-pgvector-standby`.
4. From the existing private GCP VM database-admin path, run
`ops/observatory_read_role.sql` against `teleo_canonical` with
`OBSERVATORY_DB_IAM_USER=sa-observatory-read-adapter@teleo-501523.iam`.
Do not read, copy, or retain the administrator password.
4. Create `observatory-read-api-key-staging`, add a generated 32-byte-or-longer
value through stdin, and grant that secret's accessor role only to the
runtime service account and the bounded staging canary identity.
5. Dispatch `.github/workflows/gcp-observatory-read-adapter.yml` with
`action=deploy_staging` from the exact reviewed branch revision.
5. Create `observatory-read-api-key-staging`, add a generated whitespace-free
32-byte-or-longer value through stdin only when no enabled version exists,
and grant that secret's accessor role only to the runtime service account
and the dedicated staging deploy/canary identity. Deployment pins the exact
enabled version; it does not inject `latest` into Cloud Run.
6. Dispatch the read-only preflight from merged `main`:
The workflow builds an immutable image, deploys at most two Cloud Run
```bash
gh workflow run gcp-observatory-read-adapter.yml \
--repo living-ip/teleo-infrastructure \
--ref main \
-f action=preflight_staging
```
Require a passing `observatory-read-adapter-preflight` artifact before a
bounded `action=deploy_staging` dispatch. The preflight verifies the exact
WIF condition, custom-role permission sets, project/resource IAM bindings,
Cloud Run service agent, private VPC/subnet relationship, private-only Cloud
SQL network and IAM authentication, trimmed IAM database user, exact secret
policy/value shape, and the run-unique immutable image. It also rejects any
artifact-builder deploy, runtime `actAs`, secret, or service binding.
The workflow always builds a run-unique immutable image, deploys at most two Cloud Run
instances, and retains the positive response plus anonymous `401` and
authenticated POST `405` receipts. It does not modify Vercel.
authenticated POST `405` receipts. The retained positive receipt contains
identity/provenance, counts, and proposal/live state only; claim text, source
URLs, excerpts, unknown future provenance fields, and secret values are
removed by an explicit tested allowlist. The live gate binds the response to
the exact project, private instance, database principal, Git revision, Cloud
Run image digest, runtime identity, and pinned secret version. It does not
modify Vercel. The app-level `X-Api-Key` remains the staging authentication
boundary, so anonymous Cloud Run invocation reaches the adapter and returns
the required `401` instead of a platform-generated status.
## Current T2 Receipt And Exact T3 Gate
GitHub Actions run `29389090997` proves the existing `living-ip-github` WIF
provider can federate as the artifact builder, run the checker, upload its
artifact, and clean up credentials. The checker reported `8` pass, `10`
blocked, `0` fail. Eight blocks are missing inventory permissions on the
artifact-builder identity; two are missing restore/replication proof. This is
partial OIDC/T2 evidence, not T3 and not authorization to deploy with that
identity.
The exact external action is for `billy@livingip.xyz` to reauthenticate gcloud
and ADC, prove `gcloud projects describe teleo-501523`, run
`python3 ops/check_gcp_infra_readiness.py`, review and execute the planner's
shell output as a project IAM/API administrator, then apply
`ops/observatory_read_role.sql` from the existing private database-admin path.
The general readiness identity and artifact-builder identity are not deployer
substitutes. After those actions, dispatch `preflight_staging`; only a passing
artifact permits `deploy_staging`. A passing preflight remains T2 setup proof:
the fail-closed live claim and negative receipts are what establish T3.
The IAM basis is the Google Cloud deployment contract for Cloud Run (Cloud Run
access, Artifact Registry repository read, and exact runtime `actAs`), Secret
Manager resource-level accessor grants, Direct VPC access through the default
Cloud Run service agent, and conditional Cloud SQL client access to one named
instance:
- https://docs.cloud.google.com/run/docs/deploying
- https://docs.cloud.google.com/run/docs/authenticating/public
- https://docs.cloud.google.com/run/docs/configuring/vpc-direct-vpc
- https://docs.cloud.google.com/iam/docs/workload-identity-federation-with-deployment-pipelines
- https://docs.cloud.google.com/secret-manager/docs/manage-access-to-secrets
- https://docs.cloud.google.com/sql/docs/postgres/iam-conditions
## Unexecuted Cutover Packet

154
docs/reports/leo-working-state-20260709/gcp-canonical-parity-live-20260715.json Normal file
View file

@ -0,0 +1,154 @@
{
"artifact": "gcp_canonical_parity_live_attempt",
"schema": "livingip.gcpCanonicalParityLiveAttempt.v1",
"generated_at_utc": "2026-07-15T01:50:20Z",
"status": "gated_external",
"required_tier": "T3_live_private_gcp_staging",
"current_tier": "T2_historical_source_and_isolated_restore_under_superseded_parity_contract",
"current_canary": "Restore the newest authorized canonical snapshot into one private generated Cloud SQL database, prove exact parity and ID-free Leo reasoning from staging compute, then drop the generated database and verify absence.",
"current_source": {
"run_id": "canonical-20260715t014930z",
"captured_at_utc": "2026-07-15T01:49:39.104116+00:00",
"capture_authorization_ref": "codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621",
"receipt_sha256": "7d96b052591d36df3b67dc6291b0f74a7bbeb9602c9cc424779e7c21194d59a5",
"dump_sha256": "6c56a841c1159a17f9404c2d63e3e93ae8ff55e842bf311d6d1bff21e9b95c9a",
"manifest_sha256": "39f06c6b9fc806a89ecd19294be29c58c499c65234611c7a483ddefa3c78d7d3",
"provenance_binding_sha256": "fb587093a76b61af670b7d968d32b78d6a87572faa78ca1ad93ef6cba1c9b776",
"source_system_identifier": "7649789040005668902",
"table_count": 39,
"total_rows": 52167,
"service": {
"active_state": "active",
"sub_state": "running",
"main_pid": 347406,
"restart_count": 0,
"unchanged": true
},
"production_database_mutated": false,
"telegram_send_attempted": false
},
"source_stability": {
"baseline_run_id": "canonical-20260715t014814z",
"current_run_id": "canonical-20260715t014930z",
"receipt_sha256": "d761a2b8b0313ee03c9d01e50fcca709af779578281cf10de5aed4709e914d72",
"status": "pass",
"delta_detected": false,
"table_count": 39,
"total_rows": 52167,
"total_row_delta": 0,
"changed_tables": [],
"changed_structures": [],
"table_fingerprint_sha256": "f50d86fc3ce699f602bdc0029485bd901fd98bd8114eecd5dcc4346a0f52112a",
"structure_fingerprint_sha256": "61bf482330a652ebdb419de3fb78196f2c7d25d1a482b100f385e8102c8e8d52"
},
"isolated_restore": {
"receipt_sha256": "181901afc91bebee7c343500c601ed1e5b7a5e42a5b81583d445109bfc680d86",
"status": "pass",
"source_table_count": 39,
"target_table_count": 39,
"source_total_rows": 52167,
"target_total_rows": 52167,
"table_mismatches": [],
"structural_hashes_equal": true,
"constraint_hash": "8b26d26fda56f09325e0603aae48051da08dfc6c35322fa88880acdabb8ad859",
"application_role_mismatches": {},
"target_extra_application_roles": [],
"required_extension_mismatches": {},
"performance_problems": [],
"network_mode": "none",
"persistent_postgres_storage_used": false,
"container_absent_after_test": true,
"authorization_state_verified": false,
"prior_manifest_contract_only": true
},
"contract_reassessment": {
"status": "changes_required",
"reviewed_manifest_sql_sha256": "2bd3e353627d44d4a292949952c06b1897d1a9a9e98243905c7a834d007db188",
"prior_receipts_recomputed_under_current_contract": false,
"newly_required_parity": [
"kb_gate_owner plus exact application-role attributes",
"role memberships",
"database, schema, relation, function, and type ownership",
"normalized database, schema, relation, column, function, and type ACLs",
"symmetric role and extension sets",
"nonempty row hash for every table"
],
"t3_receipt_accepted_by_repository_api": false,
"independent_platform_attestation_required": true
},
"permission_profile": {
"approval_policy": "never",
"sandbox_mode": "danger-full-access",
"network": "enabled"
},
"config_readback": {
"gcloud_sdk_version": "551.0.0",
"configured_account": "billy@livingip.xyz",
"configured_project": "teleo-501523",
"configured_account_status": "failed_noninteractive_reauthentication",
"application_default_credentials_status": "failed_noninteractive_reauthentication",
"expected_cloudsql_instance_not_currently_verified": "teleo-pgvector-standby",
"expected_private_network_not_currently_verified": "teleo-staging-net",
"expected_private_address_not_currently_verified": "10.61.0.3",
"expected_staging_compute_not_currently_verified": "teleo-prod-1/europe-west6-a",
"public_ip_disabled_currently_verified": false,
"generated_database_inventory_currently_verified": false
},
"attempted_no_approval_routes": [
"configured privileged gcloud identity",
"three other cached gcloud identities",
"application-default credentials",
"direct SSH to the retained staging address",
"local gcloud IAP SSH dry run",
"local and VPS TCP probes to the expected private Cloud SQL address",
"fixed GitHub IAP status workflow receipt",
"five GitHub readiness workflow receipts across candidate service accounts",
"current successful Artifact Registry-only WIF workflow"
],
"exact_gate": "The privileged local Google OAuth session requires human reauthentication, the fixed IAP workflow names an unavailable teleo-iap-operator WIF provider, and no existing noninteractive service-account route can obtain a Cloud SQL/Compute-capable token. Current Cloud SQL topology and generated-database inventory cannot be read, so the private restore cannot start.",
"why_autonomous_repair_stops": "Google reauthentication may require a password, MFA, passkey, or consent that Codex must not retrieve or enter. Recreating or rebinding the WIF/IAP identities requires an already-authorized GCP administrator, and no safe noninteractive identity currently has that authority.",
"next_non_user_action": "After the reviewed operator route and exact restore service account are available, capture a new source snapshot with the expanded manifest, implement and review a Cloud-SQL-compatible authorization replay, use verify-full TLS with the reviewed server name and CA, run the disposable clone lifecycle, then obtain an independently authenticated platform receipt. The repository eight-receipt verifier remains preparation-only.",
"gcp_effects_from_this_attempt": {
"cloudsql_database_created": false,
"gcp_resource_created": false,
"gcp_resource_modified": false,
"public_ip_exposure_changed": false,
"staging_promoted": false,
"telegram_sent": false
},
"proof_receipts": {
"retained_private_runtime_artifacts": true,
"source_receipt": {
"filename": "receipt.json",
"sha256": "7d96b052591d36df3b67dc6291b0f74a7bbeb9602c9cc424779e7c21194d59a5"
},
"source_manifest": {
"filename": "source-manifest.jsonl",
"sha256": "39f06c6b9fc806a89ecd19294be29c58c499c65234611c7a483ddefa3c78d7d3"
},
"source_delta": {
"filename": "source-stability-delta.json",
"sha256": "d761a2b8b0313ee03c9d01e50fcca709af779578281cf10de5aed4709e914d72"
},
"isolated_restore": {
"filename": "local-rebuild-receipt.json",
"sha256": "181901afc91bebee7c343500c601ed1e5b7a5e42a5b81583d445109bfc680d86"
},
"route_recheck": {
"filename": "route-recheck.json",
"sha256": "0fda5e6d58d72dcb084dc277bc59312bf62c002c728e249a9f16e79ae096037b"
}
},
"strongest_claim_allowed": "The retained snapshot was stable and reconstructed data and schema under the prior parity contract. It does not prove current authorization parity or live GCP operation under the expanded contract.",
"not_proven": [
"current Cloud SQL topology or public-IP-disabled state",
"current generated-database inventory",
"private staging-to-Cloud-SQL TLS connectivity",
"current Cloud SQL restore parity",
"current role membership, ownership, and ACL parity",
"certificate-authenticated verify-full database access",
"independently authenticated T3 lifecycle attestation",
"blind Leo reasoning against the generated Cloud SQL database",
"generated Cloud SQL database teardown"
]
}

36
docs/reports/leo-working-state-20260709/gcp-canonical-parity-teardown-20260715.json Normal file
View file

@ -0,0 +1,36 @@
{
"artifact": "gcp_canonical_parity_attempt_teardown",
"schema": "livingip.gcpCanonicalParityAttemptTeardown.v1",
"generated_at_utc": "2026-07-15T01:50:20Z",
"status": "no_cloud_resource_created",
"required_tier": "T3_live_private_gcp_staging",
"required_tier_met": false,
"cloudsql_clone_created": false,
"cloudsql_clone_dropped": false,
"cloudsql_clone_absence_readback": "not_applicable_no_create",
"gcs_object_created": false,
"compute_resource_created": false,
"cleanup_required": false,
"cleanup_performed": false,
"orphan_risk_from_this_attempt": false,
"public_ip_exposure_changed": false,
"staging_promoted": false,
"local_restore_container": {
"name": "teleo-canonical-rebuild-20260715015004-4ec8d1daac32",
"force_remove_exit_code": 0,
"container_absent": true,
"persistent_storage_used": false,
"network_access_available": false
},
"current_gcp_orphan_inventory_verified": false,
"exact_gate": "No authenticated Cloud SQL/Compute-capable route was available, so execution stopped before any GCP resource or generated database was created.",
"proof_paths": {
"live_attempt": "docs/reports/leo-working-state-20260709/gcp-canonical-parity-live-20260715.json",
"route_recheck_filename": "route-recheck.json",
"route_recheck_sha256": "0fda5e6d58d72dcb084dc277bc59312bf62c002c728e249a9f16e79ae096037b",
"local_cleanup_receipt_filename": "local-rebuild-receipt.json",
"local_cleanup_receipt_sha256": "181901afc91bebee7c343500c601ed1e5b7a5e42a5b81583d445109bfc680d86",
"runtime_receipts_retained_privately": true
},
"strongest_claim_allowed": "This attempt introduced no GCP cleanup debt and its isolated local restore container was removed; it is not a teardown receipt for a live disposable Cloud SQL database."
}

91
docs/reports/observatory-read-adapter/gcp-oidc-readiness-29389090997-redacted.json Normal file
View file

@ -0,0 +1,91 @@
{
"schema": "livingip.observatoryReadAdapterOidcReadinessReceipt.v1",
"status": "partial_oidc_t2",
"required_runtime_tier": "T3_live_gcp_staging_api",
"current_runtime_tier": "T2_authenticated_github_oidc_partial_inventory",
"workflow": {
"run_id": 29389090997,
"url": "https://github.com/living-ip/teleo-infrastructure/actions/runs/29389090997",
"revision": "33399bd9acfa09e4e5409873e86c8c46ac694a8d",
"ref": "refs/heads/main",
"identity": "sa-artifact-builder@teleo-501523.iam.gserviceaccount.com",
"wif_authentication": "pass",
"gcloud_setup": "pass",
"checker_execution": "pass",
"artifact_upload": "pass",
"credential_cleanup": "pass",
"job_conclusion": "failure_due_to_blocked_readiness_enforcement"
},
"summary": {
"pass_count": 8,
"blocked_count": 10,
"fail_count": 0,
"passed_checks": [
"artifact_repositories",
"cloud_build_contract",
"github_actions_readiness_ci",
"gcp_cloudsql_restore_drill_contract",
"canonical_postgres_restore_contract",
"local_canonical_postgres_restore_preflight",
"gcp_runtime_baseline_contract",
"gcp_service_communications_contract"
],
"permission_or_resource_read_blocks": [
{
"check": "cloud_build_service_account",
"missing_permission": "iam.serviceAccounts.get"
},
{
"check": "github_actions_artifact_ci",
"missing_permission": "iam.workloadIdentityPoolProviders.get"
},
{
"check": "network_ingress",
"missing_permission": "compute.firewalls.list"
},
{
"check": "compute_runtime_service_accounts",
"missing_permission": "compute.instances.list"
},
{
"check": "compute_disk_snapshots",
"missing_permission": "compute.resourcePolicies.get"
},
{
"check": "backup_buckets",
"missing_permission": "bucket metadata read denied; exact permission was not retained by the checker"
},
{
"check": "cloud_sql_standby_target",
"missing_permission": "Cloud SQL instance read denied or target invisible to this identity"
},
{
"check": "kb_source_restore_access",
"missing_permission": "secretmanager.secrets.list"
}
],
"evidence_blocks": [
"local_sqlite_postgres_restore_canary",
"kb_restore_or_replication"
]
},
"security_readback": {
"artifact_builder_deploy_authorized": false,
"cloud_run_service_created": false,
"cloud_sql_queried": false,
"secret_values_logged": false,
"production_or_vercel_routing_changed": false
},
"source_artifact": {
"github_run_url": "https://github.com/living-ip/teleo-infrastructure/actions/runs/29389090997",
"artifact_name": "gcp-readiness",
"summary_sha256": "6a1be356c069b8affacc492ac3b1e1ca63b917f92a32d7d886511a226c52925b",
"checker_sha256": "56ce4e8ba655a47e6e620a2c7039ee54fa10b1c53431d1dc6ec4db5189bb17f5",
"local_download_path": "/private/tmp/observatory-read-adapter-oidc-29389090997/gcp-readiness"
},
"proof_paths": [
"docs/reports/observatory-read-adapter/gcp-oidc-readiness-29389090997-redacted.json",
"https://github.com/living-ip/teleo-infrastructure/actions/runs/29389090997"
],
"claim_ceiling": "GitHub OIDC and the artifact-only identity work. The receipt does not prove broad GCP inventory, adapter prerequisites, a live Cloud Run service, or a teleo_canonical query."
}

46
docs/reports/observatory-read-adapter/gcp-staging-current-gate.json
View file

@ -1,19 +1,20 @@
{
"schema": "livingip.observatoryReadAdapterCurrentGate.v1",
"current_canary": "Authenticated GET /v1/claims/sample in GCP staging returns a provenance-bearing teleo_canonical claim, anonymous GET returns 401, and authenticated POST returns 405.",
"permission_profile": "approval_policy_never_with_unrestricted_filesystem_and_network",
"required_runtime_tier": "T3_live_gcp_staging_api",
"current_runtime_tier": "T2_authenticated_github_oidc_partial_inventory",
"attempted_routes": [
{
"route": "merged GitHub Actions deployment with artifact-builder WIF",
"result": "image build, smoke, and push passed; deploy stopped because run.googleapis.com is disabled"
"result": "OIDC, image build, smoke, and push passed; the 29382683335 deploy stopped before resource creation because run.googleapis.com was disabled"
},
{
"route": "local billy@livingip.xyz gcloud user credential",
"result": "token refresh and service enable both stop at noninteractive Google reauthentication"
"route": "GitHub Actions GCP readiness with artifact-builder WIF, run 29389090997",
"result": "federation, gcloud setup, checker, artifact upload, and credential cleanup passed; checker result was 8 pass, 10 blocked, 0 fail"
},
{
"route": "local application-default credential",
"result": "token refresh stops at noninteractive Google reauthentication"
"route": "local billy@livingip.xyz gcloud user and application-default credentials",
"result": "both require interactive Google reauthentication before project-admin API or IAM changes"
},
{
"route": "other locally authenticated Google accounts",
@ -22,19 +23,30 @@
{
"route": "fixed read-only GCP IAP status workflow",
"result": "WIF token exchange failed because the teleo-iap-operator provider target is missing or disabled"
},
{
"route": "local browser or computer-use session",
"result": "no browser or computer-use tool is exposed to this task; no foreground takeover was attempted"
}
],
"exact_gate": {
"first_infrastructure_gate": "Cloud Run Admin API run.googleapis.com is disabled in teleo-501523.",
"current_authorization_gate": "The only local account authorized for teleo-501523, billy@livingip.xyz, requires Google password or MFA reauthentication before project APIs and staging IAM resources can be changed.",
"alternate_private_operator_gate": "The read-only teleo-iap-operator WIF provider returns invalid_target and cannot reach its status operation."
"non_gui_federation": "Working. sa-artifact-builder can federate through living-ip-github and read/push Artifact Registry resources.",
"artifact_builder_scope": "Intentionally insufficient for deployment or full inventory. Do not grant it Cloud Run, Cloud SQL, Compute, IAM, or Secret Manager deployment roles.",
"last_api_readback": "The last direct deployment reported run.googleapis.com disabled. The current artifact-builder identity cannot verify or enable the required project APIs.",
"t3_bootstrap": "An authenticated teleo-501523 administrator must review and apply ops/plan_observatory_read_adapter_gcp.py, including required APIs, dedicated deploy/runtime identities, the exact workflow/main-only observatory-read-adapter-main WIF provider, five-permission Cloud Run custom role, exact Cloud SQL instance grants, exact secret grants, trimmed IAM database user, and private database role SQL.",
"local_operator": "billy@livingip.xyz requires Google password or MFA reauthentication before that administrator action can run.",
"alternate_private_operator": "The fixed teleo-iap-operator provider is absent or disabled and is not an available substitute."
},
"why_autonomous_repair_stops": "Google password or MFA reauthentication is a human-only credential boundary, and no authenticated project-admin browser or computer-use surface is available in this task.",
"clear_CTA": "Run `gcloud auth login billy@livingip.xyz --update-adc`, complete Google password or MFA, and then report `GCP reauth complete`.",
"next_non_user_action": "Enable the required staging APIs; create or verify the runtime service account, API-key secret, Cloud SQL IAM database user, and exact read-only role grants; deploy the retained immutable image; capture authenticated GET, anonymous 401, authenticated POST 405, SQL write-denial, exact service/database identity, and cleanup receipts.",
"production_boundary": "Do not change Vercel, production routing, Cloud SQL network exposure, or any write endpoint."
"latest_direct_non_gui_check": {
"active_account": "billy@livingip.xyz",
"gcloud_projects_describe": "blocked: reauthentication failed and non-interactive execution cannot prompt",
"application_default_credentials": "blocked: reauthentication failed and non-interactive execution cannot prompt",
"full_readiness_checker_executed": false,
"why_not": "The required first command, gcloud projects describe teleo-501523, did not pass."
},
"why_t3_stops": "A live staging service requires authorized API, IAM, Secret Manager, Cloud SQL IAM-user, and private database-role mutations. Working artifact-builder federation does not authorize those actions.",
"clear_CTA": "Run `gcloud auth login billy@livingip.xyz --update-adc`, then `gcloud projects describe teleo-501523 --format=value(projectId)` and `python3 ops/check_gcp_infra_readiness.py`; review and apply the exact adapter packet emitted by `python3 ops/plan_observatory_read_adapter_gcp.py --format shell`.",
"next_non_user_action": "Dispatch action=preflight_staging from merged main using sa-observatory-deployer. Only after its redacted preflight artifact passes, dispatch the bounded staging deploy and capture authenticated GET, anonymous 401, authenticated POST 405, database write-denial, exact service/database identity, and cleanup receipts.",
"production_boundary": "Do not change Vercel, production routing, Cloud SQL network exposure, Telegram, x402, or any write endpoint.",
"proof_paths": [
"docs/reports/observatory-read-adapter/gcp-oidc-readiness-29389090997-redacted.json",
"docs/reports/observatory-read-adapter/gcp-staging-live-attempt-redacted.json",
"https://github.com/living-ip/teleo-infrastructure/actions/runs/29389090997"
]
}

39
docs/reports/observatory-read-adapter/gcp-staging-live-attempt-redacted.json
View file

@ -2,7 +2,7 @@
"schema": "livingip.observatoryReadAdapterStagingReceipt.v1",
"status": "blocked_before_runtime_creation",
"required_runtime_tier": "T3_live_gcp_staging_api",
"current_runtime_tier": "below_T3_build_artifact_only",
"current_runtime_tier": "T2_authenticated_github_oidc_partial_inventory",
"project_id": "teleo-501523",
"region": "europe-west6",
"service_name": "observatory-read-adapter-staging",
@ -26,6 +26,18 @@
"failure_detail": "run.googleapis.com is disabled in teleo-501523",
"authenticated_as": "sa-artifact-builder@teleo-501523.iam.gserviceaccount.com"
},
"readiness_reconciliation": {
"run_id": 29389090997,
"url": "https://github.com/living-ip/teleo-infrastructure/actions/runs/29389090997",
"identity": "sa-artifact-builder@teleo-501523.iam.gserviceaccount.com",
"wif_authentication": "pass",
"checker_result": {
"pass_count": 8,
"blocked_count": 10,
"fail_count": 0
},
"claim": "working federation with an intentionally narrow artifact identity; not a staging deploy or full inventory"
},
"live_receipts": {
"service_url": null,
"authenticated_get_sample": null,
@ -47,9 +59,26 @@
"orphan_local_build_processes": 0,
"artifact_registry_image_retained_for_next_deploy": true
},
"proof_paths": [
"/private/tmp/observatory-read-adapter-run-29382683335/observatory-adapter-image/observatory-adapter-image.txt",
"/private/tmp/observatory-iap-status-29382844567/gcp-iap-operator-29382844567/result.json"
"source_artifacts": [
{
"run_url": "https://github.com/living-ip/teleo-infrastructure/actions/runs/29382683335",
"artifact_name": "observatory-adapter-image",
"sha256": "6d958b8d345ec878db2f70aaab5fad4318b106441983bd7f8e9170a429f4e852"
},
{
"run_url": "https://github.com/living-ip/teleo-infrastructure/actions/runs/29382844567",
"artifact_name": "gcp-iap-operator-29382844567",
"sha256": "988f77123f7624469c15e3c38bc1e8864a1e9f61e571feb5f6c838e67fe039e9"
},
{
"run_url": "https://github.com/living-ip/teleo-infrastructure/actions/runs/29389090997",
"artifact_name": "gcp-readiness",
"sha256": "6a1be356c069b8affacc492ac3b1e1ca63b917f92a32d7d886511a226c52925b"
}
],
"claim_ceiling": "The immutable staging image exists, but no live API or database query receipt exists. T3 is not complete."
"proof_paths": [
"docs/reports/observatory-read-adapter/gcp-staging-live-attempt-redacted.json",
"docs/reports/observatory-read-adapter/gcp-oidc-readiness-29389090997-redacted.json"
],
"claim_ceiling": "The immutable staging image and working artifact-only OIDC path exist, but no live API or database query receipt exists. T3 is not complete."
}

10
hermes-agent/leoclean-bin/cloudsql_memory_tool.py
View file

@ -167,6 +167,8 @@ def runtime_cloudsdk_config() -> str:
except OSError:
raise SystemExit("Reviewed empty Cloud SDK configuration is unavailable or untrusted.") from None
return str(resolved)
CLONE_READ_ONLY_PGOPTIONS = "-c default_transaction_read_only=on"
RECEIPTED_READ_COMMANDS = frozenset(
{
@ -376,10 +378,7 @@ def validate_runtime_connection(args: argparse.Namespace) -> None:
)
or (
normalized_key.startswith("CLOUDSDK_")
and not (
normalized_key == "CLOUDSDK_CONFIG"
and value == cloudsdk_config
)
and not (normalized_key == "CLOUDSDK_CONFIG" and value == cloudsdk_config)
)
or normalized_key == "GOOGLE_APPLICATION_CREDENTIALS"
or normalized_key.startswith("LD_")
@ -392,7 +391,8 @@ def validate_runtime_connection(args: argparse.Namespace) -> None:
or normalized_key == "PYTHONUNBUFFERED"
)
)
or normalized_key in {
or normalized_key
in {
"BASH_ENV",
"BASHOPTS",
"CDPATH",

178
ops/attest_gcp_reasoning_compute.py Normal file
View file

@ -0,0 +1,178 @@
#!/usr/bin/env python3
"""Bind a no-send reasoning receipt to live GCE metadata on the replay host."""
from __future__ import annotations
import argparse
import json
import stat
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
try:
from .private_receipt_io import print_private_receipt_summary, write_private_json
from .restore_gcp_generated_postgres_snapshot import (
DEFAULT_COMPUTE_INSTANCE,
DEFAULT_COMPUTE_ZONE,
DEFAULT_PRIVATE_NETWORK,
DEFAULT_PROJECT,
DEFAULT_RESTORE_SERVICE_ACCOUNT,
RUN_ID_RE,
TARGET_DATABASE_RE,
gce_compute_identity,
sha256_file,
)
except ImportError: # pragma: no cover - direct script execution on the GCP VM
from private_receipt_io import print_private_receipt_summary, write_private_json
from restore_gcp_generated_postgres_snapshot import (
DEFAULT_COMPUTE_INSTANCE,
DEFAULT_COMPUTE_ZONE,
DEFAULT_PRIVATE_NETWORK,
DEFAULT_PROJECT,
DEFAULT_RESTORE_SERVICE_ACCOUNT,
RUN_ID_RE,
TARGET_DATABASE_RE,
gce_compute_identity,
sha256_file,
)
REASONING_SCHEMA = "livingip.gcpGeneratedDbBlindClaimCanary.v1"
ATTESTATION_SCHEMA = "livingip.gcpReasoningComputeAttestation.v1"
def utc_now() -> str:
return datetime.now(UTC).isoformat()
def parse_timestamp(value: Any, label: str) -> datetime:
text = str(value or "")
if text.endswith("Z"):
text = text[:-1] + "+00:00"
try:
parsed = datetime.fromisoformat(text)
except ValueError as exc:
raise ValueError(f"{label} is not an ISO-8601 timestamp") from exc
if parsed.tzinfo is None:
raise ValueError(f"{label} must include a timezone")
return parsed.astimezone(UTC)
def load_reasoning_receipt(path: Path) -> dict[str, Any]:
try:
mode = path.lstat().st_mode
except OSError as exc:
raise ValueError(f"reasoning receipt is unavailable: {exc}") from exc
if not stat.S_ISREG(mode) or path.is_symlink():
raise ValueError("reasoning receipt must be a regular non-symlink file")
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except json.JSONDecodeError as exc:
raise ValueError("reasoning receipt is invalid JSON") from exc
if not isinstance(payload, dict):
raise ValueError("reasoning receipt must be a JSON object")
return payload
def attest_reasoning_compute(
*,
reasoning_receipt_path: Path,
restore_run_id: str,
target_database: str,
project: str = DEFAULT_PROJECT,
expected_compute_instance: str = DEFAULT_COMPUTE_INSTANCE,
expected_compute_zone: str = DEFAULT_COMPUTE_ZONE,
expected_private_network: str = DEFAULT_PRIVATE_NETWORK,
expected_restore_service_account: str = DEFAULT_RESTORE_SERVICE_ACCOUNT,
metadata_timeout: float = 10.0,
generated_at_utc: str | None = None,
) -> dict[str, Any]:
if not RUN_ID_RE.fullmatch(restore_run_id):
raise ValueError("restore run id is invalid")
if not TARGET_DATABASE_RE.fullmatch(target_database):
raise ValueError("target database must be a disposable teleo_clone_* database")
reasoning_receipt_path = reasoning_receipt_path.resolve()
reasoning = load_reasoning_receipt(reasoning_receipt_path)
started = parse_timestamp(reasoning.get("generated_at_utc"), "reasoning start")
completed = parse_timestamp(reasoning.get("completed_at_utc"), "reasoning completion")
checks = {
"schema": reasoning.get("schema") == REASONING_SCHEMA,
"status": reasoning.get("status") == "pass" and not reasoning.get("errors"),
"target_database": reasoning.get("target_database") == target_database,
"source_compute": reasoning.get("source_compute") == expected_compute_instance,
"chronology": started <= completed,
"no_telegram_send": reasoning.get("posted_to_telegram") is False,
"no_database_write": reasoning.get("database_write_attempted") is False,
}
failed = sorted(name for name, passed in checks.items() if not passed)
if failed:
raise ValueError("reasoning receipt failed attestation preflight: " + ", ".join(failed))
compute = gce_compute_identity(
project,
expected_compute_instance,
expected_compute_zone,
expected_private_network,
expected_restore_service_account,
timeout=metadata_timeout,
)
return {
"artifact": "gcp_reasoning_compute_attestation",
"schema": ATTESTATION_SCHEMA,
"status": "pass",
"generated_at_utc": generated_at_utc or utc_now(),
"restore_run_id": restore_run_id,
"target_database": target_database,
"reasoning_receipt": {
"sha256": sha256_file(reasoning_receipt_path),
"schema": reasoning["schema"],
"generated_at_utc": reasoning["generated_at_utc"],
"completed_at_utc": reasoning["completed_at_utc"],
"claimed_source_compute": reasoning["source_compute"],
},
"compute": compute,
"checks": {**checks, "gce_metadata_identity": all(compute.get("checks", {}).values())},
"method": "gce_metadata_server_v1",
"mutation": {
"cloudsql_changed": False,
"compute_changed": False,
"database_changed": False,
"telegram_message_sent": False,
},
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--reasoning-receipt", required=True, type=Path)
parser.add_argument("--restore-run-id", required=True)
parser.add_argument("--target-db", required=True)
parser.add_argument("--project", default=DEFAULT_PROJECT)
parser.add_argument("--expected-compute-instance", default=DEFAULT_COMPUTE_INSTANCE)
parser.add_argument("--expected-compute-zone", default=DEFAULT_COMPUTE_ZONE)
parser.add_argument("--expected-private-network", default=DEFAULT_PRIVATE_NETWORK)
parser.add_argument("--expected-restore-service-account", default=DEFAULT_RESTORE_SERVICE_ACCOUNT)
parser.add_argument("--metadata-timeout", default=10.0, type=float)
parser.add_argument("--output", required=True, type=Path)
args = parser.parse_args()
try:
receipt = attest_reasoning_compute(
reasoning_receipt_path=args.reasoning_receipt,
restore_run_id=args.restore_run_id,
target_database=args.target_db,
project=args.project,
expected_compute_instance=args.expected_compute_instance,
expected_compute_zone=args.expected_compute_zone,
expected_private_network=args.expected_private_network,
expected_restore_service_account=args.expected_restore_service_account,
metadata_timeout=args.metadata_timeout,
)
write_private_json(args.output, receipt)
except (OSError, ValueError, RuntimeError) as exc:
parser.error(str(exc))
print_private_receipt_summary(args.output, receipt)
return 0
if __name__ == "__main__":
raise SystemExit(main())

205
ops/capture_vps_canonical_postgres_snapshot.py
View file

@ -18,22 +18,29 @@ from pathlib import Path
from typing import Any
try:
from .private_receipt_io import print_private_receipt_summary, write_private_json
from .verify_postgres_parity_manifest import load_manifest
except ImportError: # pragma: no cover - direct script execution
from private_receipt_io import print_private_receipt_summary, write_private_json
from verify_postgres_parity_manifest import load_manifest
SAFE_NAME_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}\Z")
SAFE_RUN_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_-]{7,63}\Z")
SAFE_SSH_TARGET_RE = re.compile(
r"(?:[A-Za-z0-9][A-Za-z0-9._-]{0,63}@)?[A-Za-z0-9][A-Za-z0-9.-]{0,252}\Z"
)
SAFE_AUTHORIZATION_REF_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.:/@+-]{7,255}\Z")
SAFE_SSH_TARGET_RE = re.compile(r"(?:[A-Za-z0-9][A-Za-z0-9._-]{0,63}@)?[A-Za-z0-9][A-Za-z0-9.-]{0,252}\Z")
SNAPSHOT_ID_RE = re.compile(r"[0-9A-F]+-[0-9A-F]+-[0-9]+\Z")
TXID_SNAPSHOT_RE = re.compile(r"[0-9]+:[0-9]+:(?:[0-9]+(?:,[0-9]+)*)?\Z")
WAL_LSN_RE = re.compile(r"[0-9A-F]+/[0-9A-F]+\Z")
SYSTEM_IDENTIFIER_RE = re.compile(r"[0-9]+\Z")
SOURCE_SNAPSHOT_RECEIPT_SCHEMA = "livingip.sourceSnapshotReceipt.v2"
EXPECTED_FILES = {
"teleo-canonical.dump",
"teleo-canonical.dump.sha256",
"teleo-canonical.toc",
"source-manifest.jsonl",
"source-context.txt",
"source-snapshot.json",
"service-before.txt",
"service-after.txt",
}
@ -47,6 +54,81 @@ def sha256(path: Path) -> str:
return digest.hexdigest()
def sha256_bytes(payload: bytes) -> str:
return hashlib.sha256(payload).hexdigest()
def canonical_sha256(value: Any) -> str:
payload = json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True)
return sha256_bytes(payload.encode())
def load_snapshot_metadata(path: Path) -> dict[str, str]:
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
raise RuntimeError("source snapshot metadata is unreadable") from exc
if not isinstance(payload, dict):
raise RuntimeError("source snapshot metadata must be an object")
fields = {
"exported_snapshot_id": SNAPSHOT_ID_RE,
"txid_snapshot": TXID_SNAPSHOT_RE,
"wal_lsn": WAL_LSN_RE,
"system_identifier": SYSTEM_IDENTIFIER_RE,
}
normalized: dict[str, str] = {}
for field, pattern in fields.items():
value = str(payload.get(field) or "")
if not pattern.fullmatch(value):
raise RuntimeError(f"source snapshot metadata has invalid {field}")
normalized[field] = value
captured_at_utc = str(payload.get("captured_at_utc") or "")
if not captured_at_utc or any(character in captured_at_utc for character in "\r\n\x00"):
raise RuntimeError("source snapshot metadata has invalid captured_at_utc")
normalized["captured_at_utc"] = captured_at_utc
return normalized
def load_service_state(path: Path) -> dict[str, str]:
try:
state = dict(line.split("=", 1) for line in path.read_text(encoding="utf-8").splitlines() if "=" in line)
main_pid = int(state.get("MainPID") or 0)
restarts = int(state.get("NRestarts") or 0)
except (OSError, TypeError, ValueError) as exc:
raise RuntimeError("source service state is unreadable") from exc
if state.get("ActiveState") != "active" or state.get("SubState") != "running" or main_pid <= 0 or restarts < 0:
raise RuntimeError("source service is not active/running with a positive MainPID")
return state
def build_provenance_binding(
*,
run_id: str,
authorization_ref: str,
source: dict[str, str],
snapshot: dict[str, str],
dump_sha256: str,
manifest_sql_sha256: str,
source_manifest_sha256: str,
source_context_sha256: str,
) -> dict[str, Any]:
payload = {
"receipt_schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"run_id": run_id,
"capture_authorization_ref": authorization_ref,
"source": source,
"exported_snapshot_id": snapshot["exported_snapshot_id"],
"txid_snapshot": snapshot["txid_snapshot"],
"wal_lsn": snapshot["wal_lsn"],
"source_system_identifier": snapshot["system_identifier"],
"dump_sha256": dump_sha256,
"manifest_sql_sha256": manifest_sql_sha256,
"source_manifest_sha256": source_manifest_sha256,
"source_context_sha256": source_context_sha256,
}
return {"algorithm": "sha256", "payload": payload, "sha256": canonical_sha256(payload)}
def ssh_base(identity_file: Path) -> list[str]:
return [
"ssh",
@ -73,12 +155,12 @@ def remote_capture_script(*, run_id: str, container: str, database: str, service
"container_dump": shlex.quote(container_dump),
}
return f"""set -euo pipefail
remote_root={quoted['remote_root']}
container={quoted['container']}
database={quoted['database']}
service={quoted['service']}
container_manifest={quoted['container_manifest']}
container_dump={quoted['container_dump']}
remote_root={quoted["remote_root"]}
container={quoted["container"]}
database={quoted["database"]}
service={quoted["service"]}
container_manifest={quoted["container_manifest"]}
container_dump={quoted["container_dump"]}
cleanup() {{
docker exec "$container" rm -f "$container_manifest" "$container_dump" >/dev/null 2>&1 || true
@ -86,11 +168,23 @@ cleanup() {{
}}
trap cleanup EXIT
assert_service_healthy() {{
local state_file="$1"
grep -qx 'ActiveState=active' "$state_file"
grep -qx 'SubState=running' "$state_file"
local main_pid restarts
main_pid="$(sed -n 's/^MainPID=//p' "$state_file")"
restarts="$(sed -n 's/^NRestarts=//p' "$state_file")"
[[ "$main_pid" =~ ^[1-9][0-9]*$ ]]
[[ "$restarts" =~ ^[0-9]+$ ]]
}}
test -d "$remote_root"
test -f "$remote_root/postgres-parity-manifest.sql"
docker inspect "$container" >/dev/null
docker exec "$container" pg_isready -U postgres -d "$database" >/dev/null
systemctl show "$service" -p ActiveState -p SubState -p MainPID -p NRestarts -p ActiveEnterTimestamp --no-pager > "$remote_root/service-before.txt"
assert_service_healthy "$remote_root/service-before.txt"
docker inspect "$container" --format 'container_id={{{{.Id}}}}\nrunning={{{{.State.Running}}}}\nnetwork_mode={{{{.HostConfig.NetworkMode}}}}' > "$remote_root/source-context.txt"
printf 'database=%s\ncaptured_at_utc=%s\n' "$database" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$remote_root/source-context.txt"
docker cp "$remote_root/postgres-parity-manifest.sql" "$container:$container_manifest" >/dev/null
@ -107,6 +201,19 @@ if [[ ! "$snapshot_id" =~ ^[0-9A-F]+-[0-9A-F]+-[0-9]+$ ]]; then
echo "invalid exported snapshot id" >&2
exit 1
fi
printf '%s\n' "select jsonb_build_object(
'exported_snapshot_id', '$snapshot_id',
'txid_snapshot', txid_current_snapshot()::text,
'wal_lsn', pg_current_wal_lsn()::text,
'system_identifier', (select system_identifier::text from pg_control_system()),
'captured_at_utc', clock_timestamp()
)::text;" >&3
IFS= read -r snapshot_metadata <&4
if [[ -z "$snapshot_metadata" ]]; then
echo "invalid source snapshot metadata" >&2
exit 1
fi
printf '%s\n' "$snapshot_metadata" > "$remote_root/source-snapshot.json"
docker exec "$container" pg_dump \
-U postgres -d "$database" \
@ -130,6 +237,7 @@ docker cp "$container:$container_dump" "$remote_root/teleo-canonical.dump" >/dev
sha256sum "$remote_root/teleo-canonical.dump" > "$remote_root/teleo-canonical.dump.sha256"
printf 'snapshot_exported=true\n' >> "$remote_root/source-context.txt"
systemctl show "$service" -p ActiveState -p SubState -p MainPID -p NRestarts -p ActiveEnterTimestamp --no-pager > "$remote_root/service-after.txt"
assert_service_healthy "$remote_root/service-after.txt"
cmp -s "$remote_root/service-before.txt" "$remote_root/service-after.txt"
tar -C "$remote_root" -cf - \
@ -138,12 +246,13 @@ tar -C "$remote_root" -cf - \
teleo-canonical.toc \
source-manifest.jsonl \
source-context.txt \
source-snapshot.json \
service-before.txt \
service-after.txt
"""
def upload_manifest(ssh: list[str], target: str, remote_root: str, manifest: Path) -> None:
def upload_manifest(ssh: list[str], target: str, remote_root: str, manifest: bytes) -> None:
command = (
f"umask 077; rm -rf {shlex.quote(remote_root)}; "
f"mkdir -m 700 {shlex.quote(remote_root)}; "
@ -151,7 +260,7 @@ def upload_manifest(ssh: list[str], target: str, remote_root: str, manifest: Pat
)
completed = subprocess.run(
[*ssh, target, "--", command],
input=manifest.read_bytes(),
input=manifest,
capture_output=True,
check=False,
)
@ -206,8 +315,10 @@ def capture(args: argparse.Namespace) -> dict[str, Any]:
output_dir.mkdir(mode=0o700)
remote_root = f"/tmp/teleo-canonical-snapshot-{args.run_id}"
ssh = ssh_base(args.ssh_key.resolve())
manifest_sql = args.manifest.resolve().read_bytes()
manifest_sql_sha256 = sha256_bytes(manifest_sql)
try:
upload_manifest(ssh, args.ssh_target, remote_root, args.manifest.resolve())
upload_manifest(ssh, args.ssh_target, remote_root, manifest_sql)
script = remote_capture_script(
run_id=args.run_id,
container=args.container,
@ -233,19 +344,38 @@ def capture(args: argparse.Namespace) -> dict[str, Any]:
if expected_sha != actual_sha:
raise RuntimeError("captured dump SHA-256 mismatch")
manifest = load_manifest(output_dir / "source-manifest.jsonl")
before = (output_dir / "service-before.txt").read_text()
after = (output_dir / "service-after.txt").read_text()
snapshot = load_snapshot_metadata(output_dir / "source-snapshot.json")
before = load_service_state(output_dir / "service-before.txt")
after = load_service_state(output_dir / "service-after.txt")
source = {
"ssh_target": args.ssh_target,
"container": args.container,
"database": args.database,
"service": args.service,
}
source_manifest_sha256 = sha256(output_dir / "source-manifest.jsonl")
source_context_sha256 = sha256(output_dir / "source-context.txt")
provenance_binding = build_provenance_binding(
run_id=args.run_id,
authorization_ref=args.authorization_ref,
source=source,
snapshot=snapshot,
dump_sha256=actual_sha,
manifest_sql_sha256=manifest_sql_sha256,
source_manifest_sha256=source_manifest_sha256,
source_context_sha256=source_context_sha256,
)
return {
"artifact": "vps_canonical_postgres_exported_snapshot",
"schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"receipt_version": 2,
"generated_at_utc": datetime.now(UTC).isoformat(),
"status": "pass",
"source": {
"ssh_target": args.ssh_target,
"container": args.container,
"database": args.database,
"service": args.service,
},
"run_id": args.run_id,
"capture_authorization_ref": args.authorization_ref,
"source": source,
"snapshot_exported": True,
"snapshot": snapshot,
"dump": {
"path": str(dump_path),
"bytes": dump_path.stat().st_size,
@ -254,10 +384,18 @@ def capture(args: argparse.Namespace) -> dict[str, Any]:
},
"manifest": {
"path": str(output_dir / "source-manifest.jsonl"),
"sha256": source_manifest_sha256,
"manifest_sql_sha256": manifest_sql_sha256,
"table_count": len(manifest["tables"]),
"total_rows": sum(int(row["row_count"]) for row in manifest["tables"].values()),
"application_roles": manifest["singleton"]["application_roles"]["items"],
},
"source_context": {
"path": str(output_dir / "source-context.txt"),
"sha256": source_context_sha256,
},
"source_service": {"before": before, "after": after, "unchanged": before == after},
"provenance_binding": provenance_binding,
"service_unchanged": before == after,
"production_db_mutated": False,
"telegram_send_attempted": False,
@ -269,7 +407,7 @@ def cleanup_failed_output(output_dir: Path, *, preexisting: bool) -> None:
shutil.rmtree(output_dir)
def parse_args() -> argparse.Namespace:
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--execute", action="store_true")
parser.add_argument("--ssh-target", required=True)
@ -279,9 +417,10 @@ def parse_args() -> argparse.Namespace:
parser.add_argument("--service", default="leoclean-gateway.service")
parser.add_argument("--manifest", default=Path("ops/postgres_parity_manifest.sql"), type=Path)
parser.add_argument("--run-id", required=True)
parser.add_argument("--authorization-ref", required=True)
parser.add_argument("--output-dir", required=True, type=Path)
parser.add_argument("--timeout", default=180, type=int)
args = parser.parse_args()
args = parser.parse_args(argv)
for value, flag in ((args.container, "--container"), (args.database, "--database"), (args.service, "--service")):
if not SAFE_NAME_RE.fullmatch(value):
parser.error(f"{flag} contains unsafe characters")
@ -289,6 +428,8 @@ def parse_args() -> argparse.Namespace:
parser.error("--ssh-target must be a host or user@host without options")
if not SAFE_RUN_RE.fullmatch(args.run_id):
parser.error("--run-id must be 8-64 safe characters")
if not SAFE_AUTHORIZATION_REF_RE.fullmatch(args.authorization_ref):
parser.error("--authorization-ref must be 8-256 safe metadata characters")
if not args.ssh_key.is_file():
parser.error("--ssh-key must be an existing file")
if not args.manifest.is_file():
@ -298,8 +439,12 @@ def parse_args() -> argparse.Namespace:
json.dumps(
{
"artifact": "vps_canonical_postgres_exported_snapshot_plan",
"schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"status": "dry_run",
"run_id": args.run_id,
"capture_authorization_ref": args.authorization_ref,
"source": f"{args.ssh_target}:{args.container}/{args.database}",
"manifest_sql_sha256": sha256(args.manifest.resolve()),
"output_dir": str(args.output_dir),
"mutates_production_db": False,
},
@ -321,16 +466,24 @@ def main() -> int:
cleanup_failed_output(output_dir, preexisting=output_preexisted)
payload = {
"artifact": "vps_canonical_postgres_exported_snapshot",
"schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"generated_at_utc": datetime.now(UTC).isoformat(),
"status": "fail",
"run_id": args.run_id,
"capture_authorization_ref": args.authorization_ref,
"error": str(exc),
}
print(json.dumps(payload, indent=2, sort_keys=True))
receipt = (
output_dir.parent / f".{output_dir.name}-{args.run_id}-failure.json"
if output_preexisted
else output_dir / "receipt.json"
)
write_private_json(receipt, payload)
print_private_receipt_summary(receipt, payload)
return 1
receipt = args.output_dir.resolve() / "receipt.json"
receipt.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
os.chmod(receipt, 0o600)
print(json.dumps(payload, indent=2, sort_keys=True))
write_private_json(receipt, payload)
print_private_receipt_summary(receipt, payload)
return 0

669
ops/check_observatory_read_adapter_gcp_preflight.py Normal file
View file

@ -0,0 +1,669 @@
#!/usr/bin/env python3
"""Run the adapter-specific, non-mutating GCP staging preflight."""
from __future__ import annotations
import argparse
import json
import re
import subprocess
from collections.abc import Callable
from dataclasses import asdict, dataclass
from pathlib import Path
PROJECT = "teleo-501523"
PROJECT_NUMBER = "785938879453"
REGION = "europe-west6"
NETWORK = "teleo-staging-net"
SUBNET = "teleo-staging-europe-west6"
INSTANCE = "teleo-pgvector-standby"
SERVICE = "observatory-read-adapter-staging"
DEPLOY_SERVICE_ACCOUNT = f"sa-observatory-deployer@{PROJECT}.iam.gserviceaccount.com"
RUNTIME_SERVICE_ACCOUNT = f"sa-observatory-read-adapter@{PROJECT}.iam.gserviceaccount.com"
BUILD_SERVICE_ACCOUNT = f"sa-artifact-builder@{PROJECT}.iam.gserviceaccount.com"
DB_IAM_USER = f"sa-observatory-read-adapter@{PROJECT}.iam"
SECRET = "observatory-read-api-key-staging"
REPOSITORY = "teleo"
WIF_POOL = "github-actions"
DEPLOY_WIF_PROVIDER = "observatory-read-adapter-main"
DEPLOY_WORKFLOW_REF = (
"living-ip/teleo-infrastructure/.github/workflows/"
"gcp-observatory-read-adapter.yml@refs/heads/main"
)
DEPLOY_WIF_ATTRIBUTE_CONDITION = (
"assertion.repository=='living-ip/teleo-infrastructure' && "
"assertion.ref=='refs/heads/main' && "
f"assertion.workflow_ref=='{DEPLOY_WORKFLOW_REF}'"
)
DEPLOY_WIF_MEMBER = (
f"principal://iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/"
f"workloadIdentityPools/{WIF_POOL}/subject/"
"repo:living-ip/teleo-infrastructure:ref:refs/heads/main"
)
PREFLIGHT_ROLE = f"projects/{PROJECT}/roles/observatoryStagingPreflight"
DEPLOY_ROLE = f"projects/{PROJECT}/roles/observatoryStagingDeployer"
PREFLIGHT_PERMISSIONS = {
"artifactregistry.repositories.getIamPolicy",
"cloudsql.instances.get",
"cloudsql.users.list",
"compute.networks.get",
"compute.subnetworks.get",
"iam.serviceAccounts.get",
"iam.serviceAccounts.getIamPolicy",
"iam.roles.get",
"iam.workloadIdentityPoolProviders.get",
"resourcemanager.projects.get",
"resourcemanager.projects.getIamPolicy",
"run.operations.get",
"run.services.get",
"run.services.getIamPolicy",
"secretmanager.secrets.getIamPolicy",
"serviceusage.services.get",
"serviceusage.services.use",
}
DEPLOY_PERMISSIONS = {
"run.operations.get",
"run.services.create",
"run.services.get",
"run.services.setIamPolicy",
"run.services.update",
}
REQUIRED_APIS = (
"artifactregistry.googleapis.com",
"compute.googleapis.com",
"iamcredentials.googleapis.com",
"run.googleapis.com",
"secretmanager.googleapis.com",
"sqladmin.googleapis.com",
"sts.googleapis.com",
)
IMAGE_RE = re.compile(
rf"^europe-west6-docker[.]pkg[.]dev/{PROJECT}/teleo/observatory-read-adapter:"
r"[0-9a-f]{40}-[0-9]+-[1-9][0-9]*@sha256:[0-9a-f]{64}$"
)
@dataclass(frozen=True)
class Check:
name: str
status: str
detail: str
def sanitize_error(value: str) -> str:
value = re.sub(r"/[^\s]*/gha-creds-[^\s]+[.]json", "<ephemeral-wif-credential>", value)
return " ".join(value.strip().split())[-600:] or "command_failed_without_stderr"
def run(command: list[str]) -> subprocess.CompletedProcess[str]:
try:
return subprocess.run(command, text=True, capture_output=True, timeout=30, check=False)
except subprocess.TimeoutExpired:
return subprocess.CompletedProcess(command, 124, stdout="", stderr="command_timed_out_after_30_seconds")
except OSError as exc:
return subprocess.CompletedProcess(command, 127, stdout="", stderr=f"command_unavailable:{exc}")
def command_check(
name: str,
command: list[str],
validate: Callable[[str], bool],
success_detail: str,
) -> Check:
result = run(command)
if result.returncode != 0:
return Check(name, "blocked", sanitize_error(result.stderr))
try:
valid = validate(result.stdout)
except (AttributeError, IndexError, KeyError, TypeError, ValueError, json.JSONDecodeError) as exc:
return Check(name, "blocked", f"invalid_readback:{type(exc).__name__}")
return Check(name, "pass" if valid else "blocked", success_detail if valid else "unexpected_readback")
def cloud_sql_is_private_and_iam_enabled(value: str) -> bool:
payload = json.loads(value)
flags = {
item.get("name"): str(item.get("value", "")).lower()
for item in payload.get("settings", {}).get("databaseFlags", [])
}
ip_configuration = payload.get("settings", {}).get("ipConfiguration", {})
address_types = {item.get("type") for item in payload.get("ipAddresses", [])}
return (
payload.get("name") == INSTANCE
and payload.get("region") == REGION
and str(payload.get("databaseVersion", "")).startswith("POSTGRES_")
and ip_configuration.get("ipv4Enabled") is False
and str(ip_configuration.get("privateNetwork", "")).endswith(f"/networks/{NETWORK}")
and "PRIVATE" in address_types
and "PRIMARY" not in address_types
and flags.get("cloudsql.iam_authentication") == "on"
)
def cloud_sql_iam_user_exists(value: str) -> bool:
users = json.loads(value)
return any(
user.get("name") == DB_IAM_USER and user.get("type") == "CLOUD_IAM_SERVICE_ACCOUNT" for user in users
)
def subnet_is_exact(value: str) -> bool:
payload = json.loads(value)
return (
payload.get("name") == SUBNET
and str(payload.get("region", "")).endswith(f"/regions/{REGION}")
and str(payload.get("network", "")).endswith(f"/networks/{NETWORK}")
)
def secret_is_valid(value: str) -> bool:
try:
encoded = value.encode("ascii")
except UnicodeEncodeError:
return False
return value == value.strip() and 32 <= len(encoded) <= 512
def normalized(value: str) -> str:
return " ".join(value.split())
def policy_has_binding(
policy: dict[str, object],
role: str,
member: str,
*,
condition_title: str | None = None,
condition_expression: str | None = None,
) -> bool:
bindings = policy.get("bindings", [])
if not isinstance(bindings, list):
return False
for binding in bindings:
if not isinstance(binding, dict) or binding.get("role") != role:
continue
members = binding.get("members", [])
if not isinstance(members, list) or member not in members:
continue
condition = binding.get("condition")
if condition_title is None and condition_expression is None:
return condition in (None, {})
if not isinstance(condition, dict):
continue
if condition_title is not None and condition.get("title") != condition_title:
continue
if condition_expression is not None and normalized(str(condition.get("expression", ""))) != normalized(
condition_expression
):
continue
return True
return False
def member_roles(policy: dict[str, object], member: str) -> set[str]:
roles: set[str] = set()
bindings = policy.get("bindings", [])
if not isinstance(bindings, list):
return roles
for binding in bindings:
if not isinstance(binding, dict):
continue
members = binding.get("members", [])
role = binding.get("role")
if isinstance(members, list) and member in members and isinstance(role, str):
roles.add(role)
return roles
def project_iam_is_exact(value: str) -> bool:
policy = json.loads(value)
deployer = f"serviceAccount:{DEPLOY_SERVICE_ACCOUNT}"
runtime = f"serviceAccount:{RUNTIME_SERVICE_ACCOUNT}"
builder = f"serviceAccount:{BUILD_SERVICE_ACCOUNT}"
service_agent = (
f"serviceAccount:service-{PROJECT_NUMBER}@serverless-robot-prod.iam.gserviceaccount.com"
)
sql_expression = (
f"resource.name == 'projects/{PROJECT}/instances/{INSTANCE}' && "
"resource.service == 'sqladmin.googleapis.com'"
)
forbidden_deployer = {
"roles/artifactregistry.writer",
"roles/cloudsql.admin",
"roles/compute.admin",
"roles/editor",
"roles/owner",
"roles/run.admin",
"roles/run.developer",
"roles/secretmanager.admin",
}
forbidden_builder = forbidden_deployer | {
DEPLOY_ROLE,
PREFLIGHT_ROLE,
"roles/iam.serviceAccountUser",
"roles/secretmanager.secretAccessor",
}
return (
policy_has_binding(policy, PREFLIGHT_ROLE, deployer)
and policy_has_binding(policy, DEPLOY_ROLE, deployer)
and policy_has_binding(
policy,
"roles/cloudsql.client",
runtime,
condition_title="observatory-exact-cloudsql-instance",
condition_expression=sql_expression,
)
and policy_has_binding(
policy,
"roles/cloudsql.instanceUser",
runtime,
condition_title="observatory-exact-cloudsql-instance",
condition_expression=sql_expression,
)
and policy_has_binding(policy, "roles/run.serviceAgent", service_agent)
and not (member_roles(policy, deployer) & forbidden_deployer)
and not (member_roles(policy, builder) & forbidden_builder)
)
def role_has_exact_permissions(value: str, expected: set[str]) -> bool:
payload = json.loads(value)
permissions = payload.get("includedPermissions", [])
return isinstance(permissions, list) and set(permissions) == expected and payload.get("deleted") is not True
def wif_provider_is_exact(value: str) -> bool:
payload = json.loads(value)
mapping = payload.get("attributeMapping", {})
expected_mapping = {
"google.subject": "assertion.sub",
"attribute.repository": "assertion.repository",
"attribute.ref": "assertion.ref",
"attribute.workflow_ref": "assertion.workflow_ref",
}
return (
payload.get("state") == "ACTIVE"
and mapping == expected_mapping
and normalized(str(payload.get("attributeCondition", ""))) == normalized(DEPLOY_WIF_ATTRIBUTE_CONDITION)
)
def deployer_service_account_policy_is_exact(value: str) -> bool:
policy = json.loads(value)
return policy_has_binding(policy, "roles/iam.workloadIdentityUser", DEPLOY_WIF_MEMBER)
def runtime_service_account_policy_is_exact(value: str) -> bool:
policy = json.loads(value)
deployer = f"serviceAccount:{DEPLOY_SERVICE_ACCOUNT}"
builder = f"serviceAccount:{BUILD_SERVICE_ACCOUNT}"
return policy_has_binding(policy, "roles/iam.serviceAccountUser", deployer) and not member_roles(policy, builder)
def artifact_policy_is_exact(value: str) -> bool:
policy = json.loads(value)
deployer = f"serviceAccount:{DEPLOY_SERVICE_ACCOUNT}"
return policy_has_binding(policy, "roles/artifactregistry.reader", deployer)
def secret_policy_is_exact(value: str) -> bool:
policy = json.loads(value)
deployer = f"serviceAccount:{DEPLOY_SERVICE_ACCOUNT}"
runtime = f"serviceAccount:{RUNTIME_SERVICE_ACCOUNT}"
builder = f"serviceAccount:{BUILD_SERVICE_ACCOUNT}"
return (
policy_has_binding(policy, "roles/secretmanager.secretAccessor", deployer)
and policy_has_binding(policy, "roles/secretmanager.secretAccessor", runtime)
and policy_has_binding(policy, "roles/secretmanager.viewer", deployer)
and not member_roles(policy, builder)
)
def api_key_secret_check() -> Check:
version_result = run(
[
"gcloud",
"secrets",
"versions",
"list",
SECRET,
"--project",
PROJECT,
"--filter=state=ENABLED",
"--sort-by=~createTime",
"--limit=1",
"--format=value(name)",
]
)
if version_result.returncode != 0:
return Check("api_key_secret", "blocked", sanitize_error(version_result.stderr))
version = version_result.stdout.strip().rsplit("/", 1)[-1]
if not version.isdigit():
return Check("api_key_secret", "blocked", "enabled_secret_version_missing")
secret_result = run(
[
"gcloud",
"secrets",
"versions",
"access",
version,
"--secret",
SECRET,
"--project",
PROJECT,
]
)
if secret_result.returncode != 0:
return Check("api_key_secret", "blocked", sanitize_error(secret_result.stderr))
valid = secret_is_valid(secret_result.stdout)
detail = f"enabled_version_{version}_valid_length_value_redacted" if valid else "invalid_secret_value"
return Check("api_key_secret", "pass" if valid else "blocked", detail)
def existing_service_policy_check() -> Check:
describe = run(
[
"gcloud",
"run",
"services",
"describe",
SERVICE,
"--project",
PROJECT,
"--region",
REGION,
"--format=value(metadata.name)",
]
)
if describe.returncode != 0:
error = sanitize_error(describe.stderr)
if "not found" in error.lower() or "not_found" in error.lower():
return Check("existing_service_policy", "pass", "service_not_created_yet")
return Check("existing_service_policy", "blocked", error)
if describe.stdout.strip() != SERVICE:
return Check("existing_service_policy", "blocked", "unexpected_service_readback")
policy_result = run(
[
"gcloud",
"run",
"services",
"get-iam-policy",
SERVICE,
"--project",
PROJECT,
"--region",
REGION,
"--format=json",
]
)
if policy_result.returncode != 0:
return Check("existing_service_policy", "blocked", sanitize_error(policy_result.stderr))
try:
policy = json.loads(policy_result.stdout)
builder = f"serviceAccount:{BUILD_SERVICE_ACCOUNT}"
valid = not member_roles(policy, builder)
except (AttributeError, TypeError, ValueError, json.JSONDecodeError) as exc:
return Check("existing_service_policy", "blocked", f"invalid_readback:{type(exc).__name__}")
return Check(
"existing_service_policy",
"pass" if valid else "blocked",
"artifact_builder_absent" if valid else "artifact_builder_has_service_binding",
)
def build_checks(image_ref: str) -> list[Check]:
image_name, image_digest = image_ref.rsplit("@", 1) if "@" in image_ref else (image_ref, "")
digest_ref = f"{image_name.rsplit(':', 1)[0]}@{image_digest}" if image_digest else image_ref
checks = [
command_check(
"active_identity",
["gcloud", "auth", "list", "--filter=status:ACTIVE", "--format=value(account)"],
lambda value: value.strip() == DEPLOY_SERVICE_ACCOUNT,
"dedicated_observatory_deployer",
),
command_check(
"project",
["gcloud", "projects", "describe", PROJECT, "--format=value(projectId)"],
lambda value: value.strip() == PROJECT,
PROJECT,
),
]
checks.extend(
command_check(
f"api_{api.removesuffix('.googleapis.com')}",
["gcloud", "services", "describe", api, "--project", PROJECT, "--format=value(state)"],
lambda value: value.strip() == "ENABLED",
"enabled",
)
for api in REQUIRED_APIS
)
checks.extend(
[
command_check(
"deployer_wif_provider",
[
"gcloud",
"iam",
"workload-identity-pools",
"providers",
"describe",
DEPLOY_WIF_PROVIDER,
"--project",
PROJECT,
"--location",
"global",
"--workload-identity-pool",
WIF_POOL,
"--format=json",
],
wif_provider_is_exact,
"main_and_exact_workflow_only",
),
command_check(
"preflight_custom_role",
[
"gcloud",
"iam",
"roles",
"describe",
"observatoryStagingPreflight",
"--project",
PROJECT,
"--format=json",
],
lambda value: role_has_exact_permissions(value, PREFLIGHT_PERMISSIONS),
"exact_read_only_permissions",
),
command_check(
"deploy_custom_role",
[
"gcloud",
"iam",
"roles",
"describe",
"observatoryStagingDeployer",
"--project",
PROJECT,
"--format=json",
],
lambda value: role_has_exact_permissions(value, DEPLOY_PERMISSIONS),
"five_permissions_no_delete",
),
command_check(
"project_iam",
["gcloud", "projects", "get-iam-policy", PROJECT, "--format=json"],
project_iam_is_exact,
"required_bindings_present_broad_roles_absent",
),
command_check(
"deployer_wif_binding",
[
"gcloud",
"iam",
"service-accounts",
"get-iam-policy",
DEPLOY_SERVICE_ACCOUNT,
"--project",
PROJECT,
"--format=json",
],
deployer_service_account_policy_is_exact,
"exact_main_subject_workload_identity_user",
),
command_check(
"runtime_act_as_policy",
[
"gcloud",
"iam",
"service-accounts",
"get-iam-policy",
RUNTIME_SERVICE_ACCOUNT,
"--project",
PROJECT,
"--format=json",
],
runtime_service_account_policy_is_exact,
"deployer_only_artifact_builder_absent",
),
command_check(
"artifact_repository_policy",
[
"gcloud",
"artifacts",
"repositories",
"get-iam-policy",
REPOSITORY,
"--project",
PROJECT,
"--location",
REGION,
"--format=json",
],
artifact_policy_is_exact,
"deployer_reader",
),
command_check(
"secret_policy",
[
"gcloud",
"secrets",
"get-iam-policy",
SECRET,
"--project",
PROJECT,
"--format=json",
],
secret_policy_is_exact,
"runtime_and_deployer_only_artifact_builder_absent",
),
existing_service_policy_check(),
]
)
checks.extend(
[
command_check(
"runtime_service_account",
[
"gcloud",
"iam",
"service-accounts",
"describe",
RUNTIME_SERVICE_ACCOUNT,
"--project",
PROJECT,
"--format=value(email)",
],
lambda value: value.strip() == RUNTIME_SERVICE_ACCOUNT,
RUNTIME_SERVICE_ACCOUNT,
),
command_check(
"vpc_network",
[
"gcloud",
"compute",
"networks",
"describe",
NETWORK,
"--project",
PROJECT,
"--format=value(name)",
],
lambda value: value.strip() == NETWORK,
NETWORK,
),
command_check(
"vpc_subnet",
[
"gcloud",
"compute",
"networks",
"subnets",
"describe",
SUBNET,
"--project",
PROJECT,
"--region",
REGION,
"--format=json",
],
subnet_is_exact,
f"{NETWORK}/{REGION}/{SUBNET}",
),
command_check(
"cloud_sql_private_iam",
["gcloud", "sql", "instances", "describe", INSTANCE, "--project", PROJECT, "--format=json"],
cloud_sql_is_private_and_iam_enabled,
f"{PROJECT}:{REGION}:{INSTANCE}:private_ip_only:iam_authentication_on",
),
command_check(
"cloud_sql_iam_user",
["gcloud", "sql", "users", "list", "--instance", INSTANCE, "--project", PROJECT, "--format=json"],
cloud_sql_iam_user_exists,
DB_IAM_USER,
),
api_key_secret_check(),
command_check(
"immutable_image",
["gcloud", "artifacts", "docker", "images", "describe", digest_ref, "--project", PROJECT, "--format=json"],
lambda _value: bool(IMAGE_RE.fullmatch(image_ref)),
image_digest or "missing_digest",
),
]
)
return checks
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--image-ref", required=True)
parser.add_argument("--output", type=Path)
args = parser.parse_args()
checks = build_checks(args.image_ref)
pass_count = sum(check.status == "pass" for check in checks)
blocked_count = len(checks) - pass_count
receipt = {
"schema": "livingip.observatory-read-adapter-gcp-preflight.v1",
"project": PROJECT,
"required_tier": "T3_live_readonly",
"current_tier": "T2_authenticated_gcp_preflight",
"ready_for_staging_deploy": blocked_count == 0,
"claim_ceiling": "Preflight only; T3 requires the live private Cloud SQL claim and negative receipts.",
"identity": DEPLOY_SERVICE_ACCOUNT,
"checks": [asdict(check) for check in checks],
"pass_count": pass_count,
"blocked_count": blocked_count,
"secret_values_retained": False,
"database_role_live_verification_deferred_to_fail_closed_canary": True,
"production_repoint_executed": False,
}
rendered = json.dumps(receipt, indent=2, sort_keys=True) + "\n"
if args.output:
args.output.write_text(rendered, encoding="utf-8")
else:
print(rendered, end="")
return 0 if blocked_count == 0 else 1
if __name__ == "__main__":
raise SystemExit(main())

475
ops/plan_observatory_read_adapter_gcp.py Normal file
View file

@ -0,0 +1,475 @@
#!/usr/bin/env python3
"""Emit the unexecuted least-privilege GCP staging packet for Observatory."""
from __future__ import annotations
import argparse
import json
import shlex
PROJECT = "teleo-501523"
PROJECT_NUMBER = "785938879453"
REGION = "europe-west6"
GITHUB_REPOSITORY = "living-ip/teleo-infrastructure"
WIF_POOL = "github-actions"
DEPLOY_WIF_PROVIDER = "observatory-read-adapter-main"
DEPLOY_WORKFLOW_REF = (
f"{GITHUB_REPOSITORY}/.github/workflows/gcp-observatory-read-adapter.yml@refs/heads/main"
)
DEPLOY_WIF_ATTRIBUTE_MAPPING = (
"google.subject=assertion.sub,attribute.repository=assertion.repository,"
"attribute.ref=assertion.ref,attribute.workflow_ref=assertion.workflow_ref"
)
DEPLOY_WIF_ATTRIBUTE_CONDITION = (
"assertion.repository=='living-ip/teleo-infrastructure' && "
"assertion.ref=='refs/heads/main' && "
f"assertion.workflow_ref=='{DEPLOY_WORKFLOW_REF}'"
)
BUILD_SERVICE_ACCOUNT = f"sa-artifact-builder@{PROJECT}.iam.gserviceaccount.com"
DEPLOY_SERVICE_ACCOUNT_ID = "sa-observatory-deployer"
DEPLOY_SERVICE_ACCOUNT = f"{DEPLOY_SERVICE_ACCOUNT_ID}@{PROJECT}.iam.gserviceaccount.com"
RUNTIME_SERVICE_ACCOUNT_ID = "sa-observatory-read-adapter"
RUNTIME_SERVICE_ACCOUNT = f"{RUNTIME_SERVICE_ACCOUNT_ID}@{PROJECT}.iam.gserviceaccount.com"
DB_IAM_USER = f"{RUNTIME_SERVICE_ACCOUNT_ID}@{PROJECT}.iam"
SERVICE = "observatory-read-adapter-staging"
INSTANCE = "teleo-pgvector-standby"
NETWORK = "teleo-staging-net"
SUBNET = "teleo-staging-europe-west6"
SECRET = "observatory-read-api-key-staging"
REPOSITORY = "teleo"
PREFLIGHT_ROLE_ID = "observatoryStagingPreflight"
PREFLIGHT_ROLE = f"projects/{PROJECT}/roles/{PREFLIGHT_ROLE_ID}"
DEPLOY_ROLE_ID = "observatoryStagingDeployer"
DEPLOY_ROLE = f"projects/{PROJECT}/roles/{DEPLOY_ROLE_ID}"
PREFLIGHT_PERMISSIONS = (
"artifactregistry.repositories.getIamPolicy",
"cloudsql.instances.get",
"cloudsql.users.list",
"compute.networks.get",
"compute.subnetworks.get",
"iam.serviceAccounts.get",
"iam.serviceAccounts.getIamPolicy",
"iam.roles.get",
"iam.workloadIdentityPoolProviders.get",
"resourcemanager.projects.get",
"resourcemanager.projects.getIamPolicy",
"run.operations.get",
"run.services.get",
"run.services.getIamPolicy",
"secretmanager.secrets.getIamPolicy",
"serviceusage.services.get",
"serviceusage.services.use",
)
DEPLOY_PERMISSIONS = (
"run.operations.get",
"run.services.create",
"run.services.get",
"run.services.setIamPolicy",
"run.services.update",
)
REQUIRED_APIS = (
"artifactregistry.googleapis.com",
"compute.googleapis.com",
"iamcredentials.googleapis.com",
"run.googleapis.com",
"secretmanager.googleapis.com",
"sqladmin.googleapis.com",
"sts.googleapis.com",
)
FORBIDDEN_DEPLOYER_ROLES = (
"roles/artifactregistry.writer",
"roles/cloudsql.admin",
"roles/compute.admin",
"roles/editor",
"roles/owner",
"roles/run.admin",
"roles/run.developer",
"roles/secretmanager.admin",
)
def shell_join(parts: list[str]) -> str:
return " ".join(shlex.quote(part) for part in parts)
def github_wif_member() -> str:
return (
f"principal://iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/"
f"workloadIdentityPools/{WIF_POOL}/subject/repo:{GITHUB_REPOSITORY}:ref:refs/heads/main"
)
def service_account_member(account: str) -> str:
return f"serviceAccount:{account}"
def project_binding(member: str, role: str, condition: str | None = None) -> str:
command = [
"gcloud",
"projects",
"add-iam-policy-binding",
PROJECT,
"--member",
member,
"--role",
role,
]
if condition:
command.extend(["--condition", condition])
return shell_join(command)
def custom_role_commands(
role_id: str,
title: str,
description: str,
permissions: tuple[str, ...],
) -> list[str]:
common = [
"--project",
PROJECT,
"--title",
title,
"--description",
description,
"--permissions",
",".join(permissions),
"--stage",
"GA",
]
return [
(
f"gcloud iam roles describe {shlex.quote(role_id)} --project {shlex.quote(PROJECT)} "
"|| "
+ shell_join(["gcloud", "iam", "roles", "create", role_id, *common])
),
shell_join(["gcloud", "iam", "roles", "update", role_id, *common]),
]
def build_plan() -> dict[str, object]:
deployer_member = service_account_member(DEPLOY_SERVICE_ACCOUNT)
runtime_member = service_account_member(RUNTIME_SERVICE_ACCOUNT)
cloud_sql_condition = (
"expression=resource.name == "
f"'projects/{PROJECT}/instances/{INSTANCE}' && resource.service == 'sqladmin.googleapis.com',"
"title=observatory-exact-cloudsql-instance"
)
create_accounts = [
(
DEPLOY_SERVICE_ACCOUNT,
DEPLOY_SERVICE_ACCOUNT_ID,
"Observatory staging deploy and canary",
),
(
RUNTIME_SERVICE_ACCOUNT,
RUNTIME_SERVICE_ACCOUNT_ID,
"Observatory private Cloud SQL reader",
),
]
commands = [
shell_join(["gcloud", "projects", "describe", PROJECT, "--format=value(projectId)"]),
shell_join(["gcloud", "services", "enable", *REQUIRED_APIS, "--project", PROJECT]),
project_binding(
f"serviceAccount:service-{PROJECT_NUMBER}@serverless-robot-prod.iam.gserviceaccount.com",
"roles/run.serviceAgent",
),
]
for email, account_id, display_name in create_accounts:
commands.append(
f"gcloud iam service-accounts describe {shlex.quote(email)} --project {shlex.quote(PROJECT)} "
f"|| {shell_join(['gcloud', 'iam', 'service-accounts', 'create', account_id, '--project', PROJECT, '--display-name', display_name])}"
)
commands.extend(
[
(
"gcloud iam workload-identity-pools providers describe "
+ shlex.quote(DEPLOY_WIF_PROVIDER)
+ " --project "
+ shlex.quote(PROJECT)
+ " --location global --workload-identity-pool "
+ shlex.quote(WIF_POOL)
+ " || "
+ shell_join(
[
"gcloud",
"iam",
"workload-identity-pools",
"providers",
"create-oidc",
DEPLOY_WIF_PROVIDER,
"--project",
PROJECT,
"--location",
"global",
"--workload-identity-pool",
WIF_POOL,
"--issuer-uri",
"https://token.actions.githubusercontent.com/",
"--attribute-mapping",
DEPLOY_WIF_ATTRIBUTE_MAPPING,
"--attribute-condition",
DEPLOY_WIF_ATTRIBUTE_CONDITION,
]
)
),
shell_join(
[
"gcloud",
"iam",
"workload-identity-pools",
"providers",
"update-oidc",
DEPLOY_WIF_PROVIDER,
"--project",
PROJECT,
"--location",
"global",
"--workload-identity-pool",
WIF_POOL,
"--attribute-mapping",
DEPLOY_WIF_ATTRIBUTE_MAPPING,
"--attribute-condition",
DEPLOY_WIF_ATTRIBUTE_CONDITION,
]
),
shell_join(
[
"gcloud",
"iam",
"service-accounts",
"add-iam-policy-binding",
DEPLOY_SERVICE_ACCOUNT,
"--project",
PROJECT,
"--member",
github_wif_member(),
"--role",
"roles/iam.workloadIdentityUser",
]
),
]
)
commands.extend(
custom_role_commands(
PREFLIGHT_ROLE_ID,
"Observatory staging preflight",
"Read only the exact resource classes required before the private staging deploy",
PREFLIGHT_PERMISSIONS,
)
)
commands.extend(
custom_role_commands(
DEPLOY_ROLE_ID,
"Observatory staging deployer",
"Create or update Cloud Run without delete or broad infrastructure permissions",
DEPLOY_PERMISSIONS,
)
)
commands.extend(
[
project_binding(deployer_member, PREFLIGHT_ROLE),
project_binding(deployer_member, DEPLOY_ROLE),
shell_join(
[
"gcloud",
"artifacts",
"repositories",
"add-iam-policy-binding",
REPOSITORY,
"--project",
PROJECT,
"--location",
REGION,
"--member",
deployer_member,
"--role",
"roles/artifactregistry.reader",
]
),
shell_join(
[
"gcloud",
"iam",
"service-accounts",
"add-iam-policy-binding",
RUNTIME_SERVICE_ACCOUNT,
"--project",
PROJECT,
"--member",
deployer_member,
"--role",
"roles/iam.serviceAccountUser",
]
),
project_binding(runtime_member, "roles/cloudsql.client", cloud_sql_condition),
project_binding(runtime_member, "roles/cloudsql.instanceUser", cloud_sql_condition),
f"gcloud secrets describe {shlex.quote(SECRET)} --project {shlex.quote(PROJECT)} "
f"|| {shell_join(['gcloud', 'secrets', 'create', SECRET, '--project', PROJECT, '--replication-policy', 'automatic'])}",
]
)
for member in (runtime_member, deployer_member):
commands.append(
shell_join(
[
"gcloud",
"secrets",
"add-iam-policy-binding",
SECRET,
"--project",
PROJECT,
"--member",
member,
"--role",
"roles/secretmanager.secretAccessor",
]
)
)
commands.extend(
[
shell_join(
[
"gcloud",
"secrets",
"add-iam-policy-binding",
SECRET,
"--project",
PROJECT,
"--member",
deployer_member,
"--role",
"roles/secretmanager.viewer",
]
),
(
"if ! gcloud secrets versions list "
+ shlex.quote(SECRET)
+ " --project "
+ shlex.quote(PROJECT)
+ " --filter='state=ENABLED' --limit=1 --format='value(name)' | grep -q .; then "
+ "openssl rand -hex 32 | tr -d '\\n' | gcloud secrets versions add "
+ shlex.quote(SECRET)
+ " --project "
+ shlex.quote(PROJECT)
+ " --data-file=-; fi"
),
(
"if ! gcloud sql users list --instance "
+ shlex.quote(INSTANCE)
+ " --project "
+ shlex.quote(PROJECT)
+ " --filter='type=CLOUD_IAM_SERVICE_ACCOUNT' --format='value(name)' | grep -Fxq "
+ shlex.quote(DB_IAM_USER)
+ "; then "
+ shell_join(
[
"gcloud",
"sql",
"users",
"create",
DB_IAM_USER,
"--instance",
INSTANCE,
"--project",
PROJECT,
"--type",
"CLOUD_IAM_SERVICE_ACCOUNT",
]
)
+ "; fi"
),
]
)
return {
"schema": "livingip.observatory-read-adapter-gcp-plan.v1",
"project": PROJECT,
"required_tier": "T3_live_readonly",
"current_tier": "T2_unexecuted_least_privilege_packet",
"claim_ceiling": "Plan only; no IAM, API, secret, Cloud SQL, Cloud Run, or routing mutation has run.",
"identities": {
"build": BUILD_SERVICE_ACCOUNT,
"deploy_and_canary": DEPLOY_SERVICE_ACCOUNT,
"runtime": RUNTIME_SERVICE_ACCOUNT,
"database_principal": DB_IAM_USER,
},
"kept_narrow": {
"artifact_builder": "image build and push only; never deploy or receive broad infra roles",
"deployer": "workflow-specific main-only WIF plus a five-permission Cloud Run role, repository read, exact runtime actAs, exact secret access",
"runtime": "exact Cloud SQL instance connect/login plus exact secret access",
},
"github_wif_provider": (
f"projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{WIF_POOL}/providers/{DEPLOY_WIF_PROVIDER}"
),
"github_wif_member": github_wif_member(),
"required_apis": list(REQUIRED_APIS),
"preflight_custom_role": {
"name": PREFLIGHT_ROLE,
"permissions": list(PREFLIGHT_PERMISSIONS),
},
"deploy_custom_role": {
"name": DEPLOY_ROLE,
"permissions": list(DEPLOY_PERMISSIONS),
},
"conditions": {
"cloud_sql": cloud_sql_condition,
"deployer_wif": DEPLOY_WIF_ATTRIBUTE_CONDITION,
},
"commands": commands,
"private_database_action": {
"location": "existing authenticated private GCP VM database-admin path",
"command": (
f"OBSERVATORY_DB_IAM_USER={DB_IAM_USER} psql --dbname=teleo_canonical "
"--set=ON_ERROR_STOP=1 --file=ops/observatory_read_role.sql"
),
"required_receipt": (
"database=teleo_canonical, authorization_role=kb_observatory_read, exact database principal, "
"required_reads=true, effective_table_writes_denied=true"
),
},
"cloud_run_service_agent_check": {
"principal": f"service-{PROJECT_NUMBER}@serverless-robot-prod.iam.gserviceaccount.com",
"required_role": "roles/run.serviceAgent",
"reason": "Default Cloud Run service-agent role carries Direct VPC permissions; do not grant them to the runtime identity.",
"empty_readback_action": "Stop. Repair the Google-managed service-agent grant as an administrator before preflight.",
},
"post_apply_checks": [
f"gcloud projects describe {PROJECT} --format=value(projectId)",
"python3 ops/check_gcp_infra_readiness.py",
"Require the adapter preflight to verify the dedicated WIF provider, custom roles, IAM bindings, private Cloud SQL, secret, and immutable image.",
(
"gh workflow run gcp-observatory-read-adapter.yml --repo "
f"{GITHUB_REPOSITORY} --ref main -f action=preflight_staging"
),
"Require a passing observatory-read-adapter-preflight artifact before action=deploy_staging.",
],
"forbidden_deployer_roles": list(FORBIDDEN_DEPLOYER_ROLES),
"forbidden_actions": [
"grant deploy or infra roles to sa-artifact-builder",
"reuse a pre-existing image tag instead of building the current workflow revision",
"add a public Cloud SQL address",
"expose a secret value in Git, logs, or a receipt",
"add a write route",
"change Vercel or production routing",
],
"production_repoint_executed": False,
}
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--format", choices=("json", "shell"), default="json")
args = parser.parse_args()
plan = build_plan()
if args.format == "shell":
print("# Unexecuted staging packet. Review and run only as an authenticated teleo-501523 administrator.")
print("set -euo pipefail")
for command in plan["commands"]:
print(command)
else:
print(json.dumps(plan, indent=2, sort_keys=True))
return 0
if __name__ == "__main__":
raise SystemExit(main())

287
ops/postgres_parity_manifest.sql
View file

@ -55,11 +55,296 @@ select jsonb_build_object(
'bypass_rls', rolbypassrls
) order by rolname)
from pg_roles
where rolname in ('kb_apply', 'kb_review')),
where rolname in ('kb_gate_owner', 'kb_apply', 'kb_review')),
'[]'::jsonb
)
)::text;
select jsonb_build_object(
'kind', 'role_memberships',
'items', coalesce(
(select jsonb_agg(
jsonb_build_object(
'role', role_row.rolname,
'member', member_row.rolname,
'grantor', grantor_row.rolname,
'admin_option', membership.admin_option,
'inherit_option', membership.inherit_option,
'set_option', membership.set_option
) order by role_row.rolname, member_row.rolname, grantor_row.rolname)
from pg_auth_members membership
join pg_roles role_row on role_row.oid = membership.roleid
join pg_roles member_row on member_row.oid = membership.member
join pg_roles grantor_row on grantor_row.oid = membership.grantor
where role_row.rolname in ('kb_gate_owner', 'kb_apply', 'kb_review')
or member_row.rolname in ('kb_gate_owner', 'kb_apply', 'kb_review')),
'[]'::jsonb
)
)::text;
with ownership as (
select
'database:canonical_database'::text as sort_key,
'database'::text as object_type,
null::text as schema,
'canonical_database'::text as name,
null::text as identity_arguments,
pg_get_userbyid(database_row.datdba) as owner
from pg_database database_row
where database_row.datname = current_database()
union all
select
format('schema:%s', namespace.nspname),
'schema',
namespace.nspname,
namespace.nspname,
null::text,
pg_get_userbyid(namespace.nspowner)
from pg_namespace namespace
where namespace.nspname in ('public', 'kb_stage')
union all
select
format('relation:%s.%s', namespace.nspname, relation.relname),
case relation.relkind
when 'r' then 'table'
when 'p' then 'partitioned_table'
when 'v' then 'view'
when 'm' then 'materialized_view'
when 'S' then 'sequence'
when 'f' then 'foreign_table'
end,
namespace.nspname,
relation.relname,
null::text,
pg_get_userbyid(relation.relowner)
from pg_class relation
join pg_namespace namespace on namespace.oid = relation.relnamespace
where namespace.nspname in ('public', 'kb_stage')
and relation.relkind in ('r', 'p', 'v', 'm', 'S', 'f')
union all
select
format(
'function:%s.%s(%s)',
namespace.nspname,
procedure.proname,
pg_get_function_identity_arguments(procedure.oid)
),
'function',
namespace.nspname,
procedure.proname,
pg_get_function_identity_arguments(procedure.oid),
pg_get_userbyid(procedure.proowner)
from pg_proc procedure
join pg_namespace namespace on namespace.oid = procedure.pronamespace
where namespace.nspname in ('public', 'kb_stage')
union all
select
format('type:%s.%s', namespace.nspname, type_row.typname),
case type_row.typtype when 'd' then 'domain' else 'enum' end,
namespace.nspname,
type_row.typname,
null::text,
pg_get_userbyid(type_row.typowner)
from pg_type type_row
join pg_namespace namespace on namespace.oid = type_row.typnamespace
where namespace.nspname in ('public', 'kb_stage')
and type_row.typtype in ('d', 'e')
)
select jsonb_build_object(
'kind', 'object_ownership',
'items', coalesce(
jsonb_agg(to_jsonb(ownership) - 'sort_key' order by sort_key),
'[]'::jsonb
)
)::text
from ownership;
with acl_rows as (
select
format(
'database:canonical_database:%s:%s:%s',
acl.grantee,
acl.privilege_type,
acl.is_grantable
) as sort_key,
'database'::text as object_type,
null::text as schema,
'canonical_database'::text as name,
null::text as column_name,
null::text as identity_arguments,
pg_get_userbyid(acl.grantor) as grantor,
case when acl.grantee = 0 then 'PUBLIC' else pg_get_userbyid(acl.grantee) end as grantee,
acl.privilege_type,
acl.is_grantable
from pg_database database_row
cross join lateral aclexplode(coalesce(database_row.datacl, acldefault('d', database_row.datdba))) acl
where database_row.datname = current_database()
union all
select
format(
'schema:%s:%s:%s:%s',
namespace.nspname,
acl.grantee,
acl.privilege_type,
acl.is_grantable
),
'schema',
namespace.nspname,
namespace.nspname,
null::text,
null::text,
pg_get_userbyid(acl.grantor),
case when acl.grantee = 0 then 'PUBLIC' else pg_get_userbyid(acl.grantee) end,
acl.privilege_type,
acl.is_grantable
from pg_namespace namespace
cross join lateral aclexplode(coalesce(namespace.nspacl, acldefault('n', namespace.nspowner))) acl
where namespace.nspname in ('public', 'kb_stage')
union all
select
format(
'relation:%s.%s:%s:%s:%s',
namespace.nspname,
relation.relname,
acl.grantee,
acl.privilege_type,
acl.is_grantable
),
case when relation.relkind = 'S' then 'sequence' else 'relation' end,
namespace.nspname,
relation.relname,
null::text,
null::text,
pg_get_userbyid(acl.grantor),
case when acl.grantee = 0 then 'PUBLIC' else pg_get_userbyid(acl.grantee) end,
acl.privilege_type,
acl.is_grantable
from pg_class relation
join pg_namespace namespace on namespace.oid = relation.relnamespace
cross join lateral aclexplode(
coalesce(
relation.relacl,
acldefault(case when relation.relkind = 'S' then 'S'::"char" else 'r'::"char" end, relation.relowner)
)
) acl
where namespace.nspname in ('public', 'kb_stage')
and relation.relkind in ('r', 'p', 'v', 'm', 'S', 'f')
union all
select
format(
'column:%s.%s.%s:%s:%s:%s',
namespace.nspname,
relation.relname,
attribute.attname,
acl.grantee,
acl.privilege_type,
acl.is_grantable
),
'column',
namespace.nspname,
relation.relname,
attribute.attname,
null::text,
pg_get_userbyid(acl.grantor),
case when acl.grantee = 0 then 'PUBLIC' else pg_get_userbyid(acl.grantee) end,
acl.privilege_type,
acl.is_grantable
from pg_attribute attribute
join pg_class relation on relation.oid = attribute.attrelid
join pg_namespace namespace on namespace.oid = relation.relnamespace
cross join lateral aclexplode(attribute.attacl) acl
where namespace.nspname in ('public', 'kb_stage')
and relation.relkind in ('r', 'p', 'v', 'm', 'f')
and attribute.attnum > 0
and not attribute.attisdropped
union all
select
format(
'function:%s.%s(%s):%s:%s:%s',
namespace.nspname,
procedure.proname,
pg_get_function_identity_arguments(procedure.oid),
acl.grantee,
acl.privilege_type,
acl.is_grantable
),
'function',
namespace.nspname,
procedure.proname,
null::text,
pg_get_function_identity_arguments(procedure.oid),
pg_get_userbyid(acl.grantor),
case when acl.grantee = 0 then 'PUBLIC' else pg_get_userbyid(acl.grantee) end,
acl.privilege_type,
acl.is_grantable
from pg_proc procedure
join pg_namespace namespace on namespace.oid = procedure.pronamespace
cross join lateral aclexplode(coalesce(procedure.proacl, acldefault('f', procedure.proowner))) acl
where namespace.nspname in ('public', 'kb_stage')
union all
select
format(
'type:%s.%s:%s:%s:%s',
namespace.nspname,
type_row.typname,
acl.grantee,
acl.privilege_type,
acl.is_grantable
),
'type',
namespace.nspname,
type_row.typname,
null::text,
null::text,
pg_get_userbyid(acl.grantor),
case when acl.grantee = 0 then 'PUBLIC' else pg_get_userbyid(acl.grantee) end,
acl.privilege_type,
acl.is_grantable
from pg_type type_row
join pg_namespace namespace on namespace.oid = type_row.typnamespace
cross join lateral aclexplode(coalesce(type_row.typacl, acldefault('T', type_row.typowner))) acl
where namespace.nspname in ('public', 'kb_stage')
and type_row.typtype in ('d', 'e')
)
select jsonb_build_object(
'kind', 'object_acl',
'items', coalesce(
jsonb_agg(
to_jsonb(acl_rows) - 'sort_key'
order by
object_type,
schema nulls first,
name,
column_name nulls first,
identity_arguments nulls first,
grantor,
grantee,
privilege_type,
is_grantable
),
'[]'::jsonb
)
)::text
from acl_rows;
select jsonb_build_object(
'kind', 'columns',
'items', coalesce(

63
ops/private_receipt_io.py Normal file
View file

@ -0,0 +1,63 @@
"""Atomic private JSON receipt output shared by GCP parity helpers."""
from __future__ import annotations
import hashlib
import json
import os
import uuid
from pathlib import Path
from typing import Any
def file_sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def write_private_json(path: Path, payload: dict[str, Any]) -> None:
"""Write one JSON receipt as mode 0600 without chmodding existing parents."""
parent_preexisting = path.parent.exists()
path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
if not path.parent.is_dir() or path.parent.is_symlink():
raise ValueError("output parent must be a real directory")
if not parent_preexisting:
os.chmod(path.parent, 0o700)
temporary = path.parent / f".{path.name}.{uuid.uuid4().hex}.tmp"
descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
try:
with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
json.dump(payload, handle, indent=2, sort_keys=True)
handle.write("\n")
handle.flush()
os.fsync(handle.fileno())
os.replace(temporary, path)
os.chmod(path, 0o600)
finally:
try:
temporary.unlink()
except FileNotFoundError:
pass
def private_receipt_summary(path: Path, payload: dict[str, Any]) -> dict[str, Any]:
"""Return a stdout-safe pointer without replaying private receipt contents."""
failed_checks = payload.get("failed_checks")
problems = payload.get("problems")
error = payload.get("error")
return {
"artifact": payload.get("artifact"),
"status": payload.get("status"),
"output_name": path.name,
"output_sha256": file_sha256(path),
"failed_check_count": len(failed_checks) if isinstance(failed_checks, list) else None,
"problem_count": len(problems) if isinstance(problems, list) else None,
"error_type": error.get("type") if isinstance(error, dict) else (type(error).__name__ if error else None),
}
def print_private_receipt_summary(path: Path, payload: dict[str, Any]) -> None:
print(json.dumps(private_receipt_summary(path, payload), indent=2, sort_keys=True))

29
ops/redact_observatory_read_adapter_receipt.jq Normal file
View file

@ -0,0 +1,29 @@
{
schema,
read_only,
provenance: {
gcp_project: .provenance.gcp_project,
cloud_sql_instance: .provenance.cloud_sql_instance,
database: .provenance.database,
database_principal: .provenance.database_principal,
authorization_role: .provenance.authorization_role,
transaction_read_only: .provenance.transaction_read_only,
write_privileges_denied: .provenance.write_privileges_denied,
service_revision: .provenance.service_revision
},
canonical: {
claim: {
id: .canonical.claim.id,
type: .canonical.claim.type,
status: .canonical.claim.status
},
evidence_count: (.canonical.evidence | length),
incoming_edge_count: ([.canonical.edges[] | select(.direction == "incoming")] | length),
outgoing_edge_count: ([.canonical.edges[] | select(.direction == "outgoing")] | length)
},
proposal_ledger: {
distinct_from_canonical: .proposal_ledger.distinct_from_canonical,
status_counts: .proposal_ledger.status_counts,
related_count: (.proposal_ledger.related | length)
}
}

688
ops/restore_gcp_generated_postgres_snapshot.py
View file

@ -16,34 +16,58 @@ import ipaddress
import json
import os
import re
import shlex
import signal
import stat
import subprocess
import time
import urllib.error
import urllib.request
import uuid
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
try:
from .verify_postgres_parity_manifest import compare_manifests, load_manifest
from .private_receipt_io import print_private_receipt_summary
from .verify_postgres_parity_manifest import REVIEWED_MANIFEST_SQL_SHA256, compare_manifests, load_manifest
except ImportError: # pragma: no cover - direct script execution on the GCP VM
from verify_postgres_parity_manifest import compare_manifests, load_manifest
from private_receipt_io import print_private_receipt_summary
from verify_postgres_parity_manifest import REVIEWED_MANIFEST_SQL_SHA256, compare_manifests, load_manifest
DEFAULT_HOST = "10.61.0.3"
DEFAULT_PROJECT = "teleo-501523"
DEFAULT_INSTANCE = "teleo-pgvector-standby"
DEFAULT_PRIVATE_NETWORK = "projects/teleo-501523/global/networks/teleo-staging-net"
DEFAULT_COMPUTE_INSTANCE = "teleo-prod-1"
DEFAULT_COMPUTE_ZONE = "europe-west6-a"
DEFAULT_RESTORE_SERVICE_ACCOUNT = "sa-teleo-restore-drill@teleo-501523.iam.gserviceaccount.com"
DEFAULT_SOURCE_SSH_TARGET = "root@77.42.65.182"
DEFAULT_SOURCE_CONTAINER = "teleo-pg"
DEFAULT_SOURCE_DATABASE = "teleo"
DEFAULT_SOURCE_SERVICE = "leoclean-gateway.service"
DEFAULT_SECRET = "gcp-teleo-pgvector-standby-postgres-password"
DEFAULT_SERVICE = "leoclean-gcp-prod-parallel.service"
DEFAULT_RUN_ROOT = Path("/var/lib/teleo-gcp-restore-runs")
DEFAULT_MANIFEST_SQL = Path(__file__).with_name("postgres_parity_manifest.sql")
REVIEWED_MANIFEST_SQL_SHA256 = "8b8cdc25d54fdd8de05eb38c6e4423d2836953eb6012d4545f5c9c71b5f0150a"
SOURCE_SNAPSHOT_RECEIPT_SCHEMA = "livingip.sourceSnapshotReceipt.v2"
TARGET_DATABASE_RE = re.compile(r"teleo_clone_[a-z0-9][a-z0-9_]{0,50}\Z")
RUN_ID_RE = re.compile(r"gcp-restore-[a-z0-9][a-z0-9-]{7,47}\Z")
SHA256_RE = re.compile(r"[0-9a-f]{64}\Z")
LOCALE_RE = re.compile(r"[A-Za-z0-9_.@-]{1,64}\Z")
CLOUDSQL_INSTANCE_RE = re.compile(r"[a-z][a-z0-9-]{0,97}[a-z0-9]\Z")
AUTHORIZATION_REF_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.:/@+-]{7,255}\Z")
PRIVATE_NETWORK_RE = re.compile(r"projects/[a-z][a-z0-9-]{4,28}[a-z0-9]/global/networks/[a-z][a-z0-9-]{0,61}[a-z0-9]\Z")
GCE_INSTANCE_RE = re.compile(r"[a-z](?:[a-z0-9-]{0,61}[a-z0-9])?\Z")
GCE_ZONE_RE = re.compile(r"[a-z]+-[a-z]+[0-9]-[a-z]\Z")
SAFE_NAME_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}\Z")
SAFE_SSH_TARGET_RE = re.compile(r"(?:[A-Za-z0-9][A-Za-z0-9._-]{0,63}@)?[A-Za-z0-9][A-Za-z0-9.-]{0,252}\Z")
SAFE_DNS_NAME_RE = re.compile(r"[A-Za-z0-9](?:[A-Za-z0-9.-]{0,251}[A-Za-z0-9])?\Z")
SERVICE_ACCOUNT_RE = re.compile(r"[a-z][a-z0-9-]{0,62}@[a-z][a-z0-9-]{4,28}[a-z0-9]\.iam\.gserviceaccount\.com\Z")
PG_RESTORE_VERSION_RE = re.compile(r"pg_restore \(PostgreSQL\) (?P<major>[0-9]+)(?:\.[0-9]+)*")
ROLLBACK_DATABASE = "teleo_canonical_pre_20260712t1905z"
GCE_METADATA_ROOT = "http://metadata.google.internal/computeMetadata/v1"
class RestoreError(RuntimeError):
@ -66,6 +90,11 @@ def sha256_file(path: Path) -> str:
return digest.hexdigest()
def canonical_sha256(value: Any) -> str:
payload = json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True)
return hashlib.sha256(payload.encode()).hexdigest()
def validate_regular_file(path: Path, label: str) -> Path:
try:
mode = path.lstat().st_mode
@ -89,6 +118,104 @@ def validate_custom_dump(path: Path, expected_sha256: str) -> dict[str, Any]:
return {"path": str(path), "bytes": path.stat().st_size, "sha256": actual_sha256}
def validate_source_snapshot_receipt(
path: Path,
*,
expected_receipt_sha256: str,
expected_authorization_ref: str,
expected_source: dict[str, str],
dump: dict[str, Any],
source_manifest_path: Path,
manifest_sql_path: Path,
source_manifest: dict[str, Any],
) -> dict[str, Any]:
path = validate_regular_file(path, "source snapshot receipt")
receipt_sha256 = sha256_file(path)
if receipt_sha256 != expected_receipt_sha256:
raise RestoreError("source snapshot receipt SHA-256 does not match --expected-source-receipt-sha256")
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except json.JSONDecodeError as exc:
raise RestoreError("source snapshot receipt is invalid JSON") from exc
if not isinstance(payload, dict):
raise RestoreError("source snapshot receipt must be a JSON object")
snapshot = payload.get("snapshot") or {}
manifest = payload.get("manifest") or {}
source = payload.get("source") or {}
source_context = payload.get("source_context") or {}
source_service = payload.get("source_service") or {}
binding = payload.get("provenance_binding") or {}
binding_payload = binding.get("payload") or {}
source_manifest_sha256 = sha256_file(source_manifest_path)
manifest_sql_sha256 = sha256_file(manifest_sql_path)
expected_table_count = len(source_manifest["tables"])
expected_total_rows = sum(int(row["row_count"]) for row in source_manifest["tables"].values())
expected_binding_payload = {
"receipt_schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"run_id": payload.get("run_id"),
"capture_authorization_ref": payload.get("capture_authorization_ref"),
"source": source,
"exported_snapshot_id": snapshot.get("exported_snapshot_id"),
"txid_snapshot": snapshot.get("txid_snapshot"),
"wal_lsn": snapshot.get("wal_lsn"),
"source_system_identifier": snapshot.get("system_identifier"),
"dump_sha256": dump["sha256"],
"manifest_sql_sha256": manifest_sql_sha256,
"source_manifest_sha256": source_manifest_sha256,
"source_context_sha256": source_context.get("sha256"),
}
checks = {
"artifact": payload.get("artifact") == "vps_canonical_postgres_exported_snapshot",
"schema": payload.get("schema") == SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"receipt_version": payload.get("receipt_version") == 2,
"status": payload.get("status") == "pass",
"snapshot_exported": payload.get("snapshot_exported") is True,
"service_unchanged": payload.get("service_unchanged") is True,
"source_service_healthy": source_service.get("unchanged") is True
and service_state_is_healthy(source_service.get("before"))
and source_service.get("before") == source_service.get("after"),
"source_not_mutated": payload.get("production_db_mutated") is False,
"telegram_not_sent": payload.get("telegram_send_attempted") is False,
"run_id": bool(payload.get("run_id")),
"capture_authorization_ref": payload.get("capture_authorization_ref") == expected_authorization_ref,
"source_identity": source == expected_source,
"source_database": source.get("database") == source_manifest["singleton"]["identity"].get("database"),
"snapshot_identity": all(
bool(snapshot.get(field))
for field in ("exported_snapshot_id", "txid_snapshot", "wal_lsn", "system_identifier", "captured_at_utc")
),
"dump_sha256": (payload.get("dump") or {}).get("sha256") == dump["sha256"],
"dump_bytes": (payload.get("dump") or {}).get("bytes") == dump["bytes"],
"source_manifest_sha256": manifest.get("sha256") == source_manifest_sha256,
"manifest_sql_sha256": manifest.get("manifest_sql_sha256") == manifest_sql_sha256,
"source_context_sha256": bool(SHA256_RE.fullmatch(str(source_context.get("sha256") or ""))),
"table_count": manifest.get("table_count") == expected_table_count,
"total_rows": manifest.get("total_rows") == expected_total_rows,
"binding_algorithm": binding.get("algorithm") == "sha256",
"binding_payload": binding_payload == expected_binding_payload,
"binding_sha256": binding.get("sha256") == canonical_sha256(expected_binding_payload),
}
failed = sorted(name for name, passed in checks.items() if not passed)
if failed:
raise RestoreError("source snapshot receipt failed validation: " + ", ".join(failed))
return {
"path": str(path),
"sha256": receipt_sha256,
"schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"run_id": payload["run_id"],
"capture_authorization_ref": payload["capture_authorization_ref"],
"snapshot": snapshot,
"provenance_binding_sha256": binding["sha256"],
"manifest_sha256": source_manifest_sha256,
"manifest_sql_sha256": manifest_sql_sha256,
"dump_sha256": dump["sha256"],
"table_count": expected_table_count,
"total_rows": expected_total_rows,
"checks": checks,
}
def _run(
command: list[str],
*,
@ -158,6 +285,165 @@ def service_state(service: str, *, timeout: float) -> dict[str, Any]:
return result
def validate_service_healthy(state: dict[str, Any], label: str) -> None:
if not service_state_is_healthy(state):
raise RestoreError(f"{label} service is not active/running with a positive MainPID")
def service_state_is_healthy(state: Any) -> bool:
if not isinstance(state, dict):
return False
try:
main_pid = int(state.get("MainPID") or 0)
restarts = int(state.get("NRestarts") or 0)
except (TypeError, ValueError):
return False
return (
state.get("ActiveState") == "active" and state.get("SubState") == "running" and main_pid > 0 and restarts >= 0
)
def cloudsql_control_plane_state(
project: str,
instance: str,
expected_host: str,
expected_private_network: str,
*,
timeout: float,
) -> dict[str, Any]:
completed = _run(
[
"gcloud",
"sql",
"instances",
"describe",
instance,
f"--project={project}",
"--format=json",
"--quiet",
],
timeout=timeout,
)
raw = _require_success(completed, "Cloud SQL control-plane readback")
try:
payload = json.loads(raw)
except json.JSONDecodeError as exc:
raise RestoreError("Cloud SQL control-plane readback returned invalid JSON") from exc
settings = payload.get("settings") or {}
ip_configuration = settings.get("ipConfiguration") or {}
addresses = payload.get("ipAddresses") or []
private_addresses = sorted(
str(row.get("ipAddress")) for row in addresses if row.get("type") == "PRIVATE" and row.get("ipAddress")
)
private_network = str(ip_configuration.get("privateNetwork") or "")
non_private_addresses = sorted(
str(row.get("ipAddress")) for row in addresses if row.get("type") != "PRIVATE" and row.get("ipAddress")
)
checks = {
"instance": payload.get("name") == instance,
"state": payload.get("state") == "RUNNABLE",
"postgres_16": str(payload.get("databaseVersion") or "").startswith("POSTGRES_16"),
"public_ip_disabled": ip_configuration.get("ipv4Enabled") is False,
"private_network": private_network == expected_private_network,
"expected_private_host": expected_host in private_addresses,
"no_non_private_address": not non_private_addresses,
}
failed = sorted(name for name, passed in checks.items() if not passed)
if failed:
raise RestoreError("Cloud SQL control-plane preflight failed: " + ", ".join(failed))
return {
"project": project,
"instance": instance,
"database_version": payload.get("databaseVersion"),
"region": payload.get("region"),
"state": payload.get("state"),
"tier": settings.get("tier"),
"private_network": private_network,
"expected_private_network": expected_private_network,
"private_addresses": private_addresses,
"expected_private_host": expected_host,
"public_ip_disabled": True,
"checks": checks,
}
def gce_metadata_value(path: str, *, timeout: float) -> str:
request = urllib.request.Request(
f"{GCE_METADATA_ROOT}/{path}",
headers={"Metadata-Flavor": "Google"},
method="GET",
)
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
try:
with opener.open(request, timeout=timeout) as response:
if response.headers.get("Metadata-Flavor") != "Google":
raise RestoreError("GCE metadata response is missing Metadata-Flavor: Google")
value = response.read(4096).decode("utf-8", errors="strict").strip()
except (OSError, UnicodeError, urllib.error.URLError) as exc:
raise RestoreError(f"GCE metadata readback failed for {path}") from exc
if not value or any(character in value for character in "\r\n\x00"):
raise RestoreError(f"GCE metadata readback is invalid for {path}")
return value
def gce_compute_identity(
project: str,
expected_instance: str,
expected_zone: str,
expected_private_network: str,
expected_service_account: str,
*,
timeout: float,
) -> dict[str, Any]:
metadata = {
"project": gce_metadata_value("project/project-id", timeout=timeout),
"project_number": gce_metadata_value("project/numeric-project-id", timeout=timeout),
"instance": gce_metadata_value("instance/name", timeout=timeout),
"instance_id": gce_metadata_value("instance/id", timeout=timeout),
"zone_resource": gce_metadata_value("instance/zone", timeout=timeout),
"network_resource": gce_metadata_value("instance/network-interfaces/0/network", timeout=timeout),
"private_ip": gce_metadata_value("instance/network-interfaces/0/ip", timeout=timeout),
"service_account": gce_metadata_value("instance/service-accounts/default/email", timeout=timeout),
}
zone = metadata["zone_resource"].rsplit("/", 1)[-1]
network_name = expected_private_network.rsplit("/", 1)[-1]
expected_network_resource = f"projects/{metadata['project_number']}/networks/{network_name}"
try:
private_ip = ipaddress.ip_address(metadata["private_ip"])
except ValueError as exc:
raise RestoreError("GCE metadata private IP is invalid") from exc
checks = {
"project": metadata["project"] == project,
"project_number": metadata["project_number"].isdigit(),
"expected_network_project": expected_private_network.startswith(f"projects/{project}/global/networks/"),
"instance": metadata["instance"] == expected_instance,
"instance_id": metadata["instance_id"].isdigit(),
"zone": zone == expected_zone,
"network": metadata["network_resource"] == expected_network_resource,
"private_ip": any(
private_ip in network
for network in (
ipaddress.ip_network("10.0.0.0/8"),
ipaddress.ip_network("172.16.0.0/12"),
ipaddress.ip_network("192.168.0.0/16"),
)
),
"service_account": metadata["service_account"] == expected_service_account,
}
failed = sorted(name for name, passed in checks.items() if not passed)
if failed:
raise RestoreError("GCE compute identity preflight failed: " + ", ".join(failed))
return {
**metadata,
"zone": zone,
"expected_zone": expected_zone,
"expected_private_network": expected_private_network,
"expected_network_resource": expected_network_resource,
"expected_service_account": expected_service_account,
"checks": checks,
}
def resolve_password(project: str, secret: str, *, timeout: float) -> str:
completed = _run(
[
@ -212,10 +498,29 @@ def postgres_env(password: str, *, read_only: bool) -> dict[str, str]:
return env
def conninfo_value(value: Any) -> str:
return "'" + str(value).replace("\\", "\\\\").replace("'", "\\'") + "'"
def postgres_conninfo(args: argparse.Namespace, database: str) -> str:
return " ".join(
(
f"host={conninfo_value(args.ssl_server_name)}",
f"hostaddr={conninfo_value(args.host)}",
"port=5432",
f"dbname={conninfo_value(database)}",
"user=postgres",
"sslmode=verify-full",
f"sslrootcert={conninfo_value(args.ssl_root_cert)}",
"connect_timeout=8",
)
)
def psql_command(args: argparse.Namespace, database: str) -> list[str]:
return [
args.psql_bin,
f"host={args.host} port=5432 dbname={database} user=postgres sslmode=require connect_timeout=8",
postgres_conninfo(args, database),
"-X",
"-Atq",
"-v",
@ -286,7 +591,7 @@ def restore_dump(args: argparse.Namespace, password: str) -> None:
command = [
args.pg_restore_bin,
"-d",
f"host={args.host} port=5432 dbname={args.target_db} user=postgres sslmode=require connect_timeout=8",
postgres_conninfo(args, args.target_db),
"--no-owner",
"--no-acl",
"--exit-on-error",
@ -310,7 +615,13 @@ def capture_target_manifest(args: argparse.Namespace, password: str) -> str:
return _require_success(completed, "target parity manifest capture", secrets=(password,))
def validate_private_identity(target_manifest: dict[str, Any], target_db: str) -> dict[str, Any]:
def validate_private_identity(
target_manifest: dict[str, Any],
target_db: str,
source_compute: str,
ssl_server_name: str,
ssl_root_cert_sha256: str,
) -> dict[str, Any]:
identity = target_manifest["singleton"]["identity"]
if identity.get("database") != target_db:
raise RestoreError("target manifest database does not match the generated clone")
@ -338,9 +649,13 @@ def validate_private_identity(target_manifest: dict[str, Any], target_db: str) -
"server_port": identity.get("server_port"),
"ssl": True,
"ssl_version": identity.get("ssl_version"),
"sslmode": "verify-full",
"ssl_server_name": ssl_server_name,
"ssl_root_cert_sha256": ssl_root_cert_sha256,
"server_identity_verified": True,
"private_connectivity": True,
"transaction_read_only": "on",
"source_compute": "teleo-prod-1",
"source_compute": source_compute,
}
@ -434,12 +749,33 @@ def write_private(path: Path, content: str | bytes) -> None:
def run_restore(args: argparse.Namespace) -> dict[str, Any]:
started = time.monotonic()
args.ssl_root_cert = validate_regular_file(args.ssl_root_cert, "Cloud SQL server CA")
ssl_root_cert_sha256 = sha256_file(args.ssl_root_cert)
run_dir = secure_run_dir(args.run_root, args.run_id, create=True)
receipt: dict[str, Any] = {
"artifact": "gcp_generated_postgres_snapshot_restore",
"generated_at_utc": utc_now(),
"run_id": args.run_id,
"target_database": args.target_db,
"execution_contract": {
"project": args.project,
"cloudsql_instance": args.cloudsql_instance,
"host": args.host,
"private_network": args.expected_private_network,
"compute_instance": args.expected_compute_instance,
"compute_zone": args.expected_compute_zone,
"restore_service_account": args.expected_restore_service_account,
"ssl_server_name": args.ssl_server_name,
"ssl_root_cert_sha256": ssl_root_cert_sha256,
"source": {
"ssh_target": args.expected_source_ssh_target,
"container": args.expected_source_container,
"database": args.expected_source_database,
"service": args.expected_source_service,
},
"password_secret": args.password_secret,
"service": args.service,
},
"status": "running",
"phase": "validation",
"source": {},
@ -475,12 +811,40 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]:
"sha256": sha256_file(args.source_manifest),
}
source_manifest = load_manifest(args.source_manifest)
receipt["source"]["capture_receipt"] = validate_source_snapshot_receipt(
args.source_receipt,
expected_receipt_sha256=args.expected_source_receipt_sha256,
expected_authorization_ref=args.expected_capture_authorization_ref,
expected_source=receipt["execution_contract"]["source"],
dump=receipt["source"]["dump"],
source_manifest_path=args.source_manifest,
manifest_sql_path=args.manifest_sql,
source_manifest=source_manifest,
)
receipt["toolchain"] = validate_pg_restore_version(
args.pg_restore_bin,
source_manifest,
timeout=args.command_timeout,
)
receipt["target"]["compute"] = gce_compute_identity(
args.project,
args.expected_compute_instance,
args.expected_compute_zone,
args.expected_private_network,
args.expected_restore_service_account,
timeout=args.command_timeout,
)
receipt["live_service"]["before"] = service_state(args.service, timeout=args.command_timeout)
validate_service_healthy(receipt["live_service"]["before"], "pre-restore")
receipt["phase"] = "cloudsql_control_plane_preflight"
receipt["target"]["cloudsql"] = cloudsql_control_plane_state(
args.project,
args.cloudsql_instance,
args.host,
args.expected_private_network,
timeout=args.command_timeout,
)
receipt["phase"] = "credential_resolution"
password = resolve_password(args.project, args.password_secret, timeout=args.command_timeout)
@ -506,9 +870,52 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]:
"manifest_path": str(target_manifest_path),
"manifest_bytes": target_manifest_path.stat().st_size,
"manifest_sha256": sha256_file(target_manifest_path),
"connectivity": validate_private_identity(target_manifest, args.target_db),
"connectivity": validate_private_identity(
target_manifest,
args.target_db,
receipt["target"]["compute"]["instance"],
args.ssl_server_name,
ssl_root_cert_sha256,
),
}
)
connectivity = receipt["target"]["connectivity"]
cloudsql = receipt["target"]["cloudsql"]
compute = receipt["target"]["compute"]
connectivity_proof = {
"artifact": "gcp_private_postgres_connectivity",
"schema": "livingip.gcpPrivatePostgresConnectivity.v2",
"generated_at_utc": utc_now(),
"status": "pass",
"restore_run_id": args.run_id,
"target_database": args.target_db,
"project": cloudsql["project"],
"cloudsql_instance": cloudsql["instance"],
"private_network": cloudsql["private_network"],
"source_compute": compute["instance"],
"source_compute_instance_id": compute["instance_id"],
"source_compute_zone": compute["zone"],
"source_compute_private_ip": compute["private_ip"],
"server_address": connectivity["server_address"],
"server_port": connectivity["server_port"],
"ssl": connectivity["ssl"],
"ssl_version": connectivity["ssl_version"],
"sslmode": connectivity["sslmode"],
"ssl_server_name": connectivity["ssl_server_name"],
"ssl_root_cert_sha256": connectivity["ssl_root_cert_sha256"],
"server_identity_verified": connectivity["server_identity_verified"],
"private_connectivity": connectivity["private_connectivity"],
"public_ip_disabled": cloudsql["public_ip_disabled"],
"cloudsql_control_plane_checks": cloudsql["checks"],
"compute_identity_checks": compute["checks"],
}
connectivity_proof_path = run_dir / "gcp-private-connectivity.json"
write_private(connectivity_proof_path, json.dumps(connectivity_proof, indent=2, sort_keys=True) + "\n")
receipt["target"]["connectivity_proof"] = {
"path": str(connectivity_proof_path),
"sha256": sha256_file(connectivity_proof_path),
"schema": connectivity_proof["schema"],
}
receipt["phase"] = "parity_compare"
problems, details = compare_manifests(
@ -529,6 +936,7 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]:
receipt["phase"] = "service_and_rollback_readback"
receipt["rollback_database"] = rollback_database_state(args, password)
receipt["live_service"]["after"] = service_state(args.service, timeout=args.command_timeout)
validate_service_healthy(receipt["live_service"]["after"], "post-restore")
receipt["live_service"]["unchanged"] = receipt["live_service"]["before"] == receipt["live_service"]["after"]
if not receipt["live_service"]["unchanged"]:
raise RestoreError("live GCP service state changed during disposable restore")
@ -539,9 +947,48 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]:
"required": True,
"attempted": False,
"clone_database_remaining": 1,
"exact_command": (
f"sudo python3 ops/restore_gcp_generated_postgres_snapshot.py cleanup --execute "
f"--target-db {args.target_db} --run-id {args.run_id}"
"exact_command": shlex.join(
[
"sudo",
"python3",
"ops/restore_gcp_generated_postgres_snapshot.py",
"cleanup",
"--execute",
"--target-db",
args.target_db,
"--run-id",
args.run_id,
"--host",
args.host,
"--project",
args.project,
"--cloudsql-instance",
args.cloudsql_instance,
"--expected-private-network",
args.expected_private_network,
"--expected-compute-instance",
args.expected_compute_instance,
"--expected-compute-zone",
args.expected_compute_zone,
"--expected-restore-service-account",
args.expected_restore_service_account,
"--ssl-server-name",
args.ssl_server_name,
"--ssl-root-cert",
str(args.ssl_root_cert),
"--expected-source-ssh-target",
args.expected_source_ssh_target,
"--expected-source-container",
args.expected_source_container,
"--expected-source-database",
args.expected_source_database,
"--expected-source-service",
args.expected_source_service,
"--password-secret",
args.password_secret,
"--service",
args.service,
]
),
}
except (
@ -583,15 +1030,19 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]:
except Exception as service_exc:
receipt["live_service"]["after_error"] = str(service_exc)[-2000:]
receipt["duration_seconds"] = round(time.monotonic() - started, 6)
receipt["completed_at_utc"] = utc_now()
write_private(run_dir / "restore-receipt.json", json.dumps(receipt, indent=2, sort_keys=True) + "\n")
password = ""
return receipt
def run_cleanup(args: argparse.Namespace) -> dict[str, Any]:
args.ssl_root_cert = validate_regular_file(args.ssl_root_cert, "Cloud SQL server CA")
ssl_root_cert_sha256 = sha256_file(args.ssl_root_cert)
run_dir = secure_run_dir(args.run_root, args.run_id, create=False)
restore_receipt_path = run_dir / "restore-receipt.json"
restore_receipt_path = validate_regular_file(restore_receipt_path, "restore receipt")
restore_receipt_sha256 = sha256_file(restore_receipt_path)
restore_receipt = json.loads(restore_receipt_path.read_text(encoding="utf-8"))
if restore_receipt.get("status") != "pass":
raise RestoreError("cleanup requires a passing restore receipt")
@ -600,23 +1051,70 @@ def run_cleanup(args: argparse.Namespace) -> dict[str, Any]:
if restore_receipt.get("run_id") != args.run_id:
raise RestoreError("cleanup run id does not match the retained restore receipt")
before = service_state(args.service, timeout=args.command_timeout)
password = resolve_password(args.project, args.password_secret, timeout=args.command_timeout)
try:
dropped = drop_clone(args, password)
rollback = rollback_database_state(args, password)
finally:
password = ""
after = service_state(args.service, timeout=args.command_timeout)
payload = {
expected_contract = {
"project": args.project,
"cloudsql_instance": args.cloudsql_instance,
"host": args.host,
"private_network": args.expected_private_network,
"compute_instance": args.expected_compute_instance,
"compute_zone": args.expected_compute_zone,
"restore_service_account": args.expected_restore_service_account,
"ssl_server_name": args.ssl_server_name,
"ssl_root_cert_sha256": ssl_root_cert_sha256,
"source": {
"ssh_target": args.expected_source_ssh_target,
"container": args.expected_source_container,
"database": args.expected_source_database,
"service": args.expected_source_service,
},
"password_secret": args.password_secret,
"service": args.service,
}
if restore_receipt.get("execution_contract") != expected_contract:
raise RestoreError("cleanup execution contract does not match the retained restore receipt")
restore_cloudsql = (restore_receipt.get("target") or {}).get("cloudsql") or {}
restore_topology_checks = {
"project": restore_cloudsql.get("project") == args.project,
"instance": restore_cloudsql.get("instance") == args.cloudsql_instance,
"host": restore_cloudsql.get("expected_private_host") == args.host,
"private_network": restore_cloudsql.get("expected_private_network") == args.expected_private_network,
"public_ip_disabled": restore_cloudsql.get("public_ip_disabled") is True,
}
failed_topology_checks = sorted(name for name, passed in restore_topology_checks.items() if not passed)
if failed_topology_checks:
raise RestoreError(
"cleanup target topology does not match the retained restore receipt: " + ", ".join(failed_topology_checks)
)
restore_compute = (restore_receipt.get("target") or {}).get("compute") or {}
restore_compute_checks = {
"project": restore_compute.get("project") == args.project,
"instance": restore_compute.get("instance") == args.expected_compute_instance,
"zone": restore_compute.get("zone") == args.expected_compute_zone,
"private_network": restore_compute.get("expected_private_network") == args.expected_private_network,
"service_account": restore_compute.get("service_account") == args.expected_restore_service_account,
"checks": bool(restore_compute.get("checks"))
and all(value is True for value in restore_compute["checks"].values()),
}
failed_compute_checks = sorted(name for name, passed in restore_compute_checks.items() if not passed)
if failed_compute_checks:
raise RestoreError(
"cleanup compute identity does not match the retained restore receipt: " + ", ".join(failed_compute_checks)
)
started = time.monotonic()
payload: dict[str, Any] = {
"artifact": "gcp_generated_postgres_snapshot_cleanup",
"generated_at_utc": utc_now(),
"run_id": args.run_id,
"target_database": args.target_db,
"status": "pass" if before == after and dropped["clone_database_remaining"] == 0 else "fail",
"database": dropped,
"rollback_database": rollback,
"live_service": {"before": before, "after": after, "unchanged": before == after},
"restore_receipt": {"path": str(restore_receipt_path), "sha256": restore_receipt_sha256},
"execution_contract": expected_contract,
"target": {},
"status": "running",
"phase": "cloudsql_control_plane_preflight",
"database": {"clone_database_remaining": None},
"rollback_database": {},
"live_service": {},
"safety": {
"live_database_named": False,
"live_profile_modified": False,
@ -624,8 +1122,96 @@ def run_cleanup(args: argparse.Namespace) -> dict[str, Any]:
"telegram_message_sent": False,
"secret_persisted": False,
},
"error": None,
}
write_private(run_dir / "cleanup-receipt.json", json.dumps(payload, indent=2, sort_keys=True) + "\n")
password = ""
try:
payload["target"]["cloudsql"] = cloudsql_control_plane_state(
args.project,
args.cloudsql_instance,
args.host,
args.expected_private_network,
timeout=args.command_timeout,
)
payload["target"]["compute"] = gce_compute_identity(
args.project,
args.expected_compute_instance,
args.expected_compute_zone,
args.expected_private_network,
args.expected_restore_service_account,
timeout=args.command_timeout,
)
cleanup_compute_mismatches = sorted(
field
for field in (
"project",
"project_number",
"instance",
"instance_id",
"zone",
"network_resource",
"private_ip",
"service_account",
)
if payload["target"]["compute"].get(field) != restore_compute.get(field)
)
if cleanup_compute_mismatches:
raise RestoreError("cleanup GCE identity changed since restore: " + ", ".join(cleanup_compute_mismatches))
payload["phase"] = "service_readback_before"
payload["live_service"]["before"] = service_state(args.service, timeout=args.command_timeout)
validate_service_healthy(payload["live_service"]["before"], "pre-cleanup")
payload["phase"] = "credential_resolution"
password = resolve_password(args.project, args.password_secret, timeout=args.command_timeout)
payload["phase"] = "database_drop"
payload["database"] = drop_clone(args, password)
payload["phase"] = "rollback_readback"
payload["rollback_database"] = rollback_database_state(args, password)
payload["phase"] = "service_readback_after"
payload["live_service"]["after"] = service_state(args.service, timeout=args.command_timeout)
validate_service_healthy(payload["live_service"]["after"], "post-cleanup")
payload["live_service"]["unchanged"] = payload["live_service"]["before"] == payload["live_service"]["after"]
payload["status"] = (
"pass"
if payload["live_service"]["unchanged"] and payload["database"]["clone_database_remaining"] == 0
else "fail"
)
payload["phase"] = "complete"
except (
OSError,
ValueError,
KeyError,
TypeError,
json.JSONDecodeError,
subprocess.TimeoutExpired,
RestoreError,
KeyboardInterrupt,
) as exc:
payload["status"] = "fail"
payload["error"] = {
"phase": payload.get("phase"),
"type": type(exc).__name__,
"message": str(exc)[-2000:],
}
if password and payload["database"].get("clone_database_remaining") is None:
try:
payload["database"]["clone_database_remaining"] = (
1 if database_exists(args, password, args.target_db) else 0
)
except Exception as readback_exc:
payload["database"]["readback_error"] = str(readback_exc)[-2000:]
if "before" in payload["live_service"] and "after" not in payload["live_service"]:
try:
payload["live_service"]["after"] = service_state(args.service, timeout=args.command_timeout)
payload["live_service"]["unchanged"] = (
payload["live_service"]["before"] == payload["live_service"]["after"]
)
except Exception as service_exc:
payload["live_service"]["after_error"] = str(service_exc)[-2000:]
finally:
password = ""
payload["duration_seconds"] = round(time.monotonic() - started, 6)
payload["completed_at_utc"] = utc_now()
write_private(run_dir / "cleanup-receipt.json", json.dumps(payload, indent=2, sort_keys=True) + "\n")
return payload
@ -635,6 +1221,17 @@ def add_common_args(parser: argparse.ArgumentParser) -> None:
parser.add_argument("--run-id", required=True)
parser.add_argument("--host", default=DEFAULT_HOST)
parser.add_argument("--project", default=DEFAULT_PROJECT)
parser.add_argument("--cloudsql-instance", default=DEFAULT_INSTANCE)
parser.add_argument("--expected-private-network", default=DEFAULT_PRIVATE_NETWORK)
parser.add_argument("--expected-compute-instance", default=DEFAULT_COMPUTE_INSTANCE)
parser.add_argument("--expected-compute-zone", default=DEFAULT_COMPUTE_ZONE)
parser.add_argument("--expected-restore-service-account", default=DEFAULT_RESTORE_SERVICE_ACCOUNT)
parser.add_argument("--ssl-server-name", required=True)
parser.add_argument("--ssl-root-cert", required=True, type=Path)
parser.add_argument("--expected-source-ssh-target", default=DEFAULT_SOURCE_SSH_TARGET)
parser.add_argument("--expected-source-container", default=DEFAULT_SOURCE_CONTAINER)
parser.add_argument("--expected-source-database", default=DEFAULT_SOURCE_DATABASE)
parser.add_argument("--expected-source-service", default=DEFAULT_SOURCE_SERVICE)
parser.add_argument("--password-secret", default=DEFAULT_SECRET)
parser.add_argument("--service", default=DEFAULT_SERVICE)
parser.add_argument("--command-timeout", type=float, default=120.0)
@ -650,6 +1247,9 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
restore.add_argument("--dump", required=True, type=Path)
restore.add_argument("--expected-dump-sha256", required=True)
restore.add_argument("--source-manifest", required=True, type=Path)
restore.add_argument("--source-receipt", required=True, type=Path)
restore.add_argument("--expected-source-receipt-sha256", required=True)
restore.add_argument("--expected-capture-authorization-ref", required=True)
restore.add_argument("--manifest-sql", default=DEFAULT_MANIFEST_SQL, type=Path)
restore.add_argument("--pg-restore-bin", default="pg_restore")
restore.add_argument("--restore-timeout", type=float, default=900.0)
@ -667,6 +1267,36 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser.error("--target-db must be a bounded lowercase teleo_clone_* identifier")
if not RUN_ID_RE.fullmatch(args.run_id):
parser.error("--run-id must match gcp-restore-<8-48 lowercase letters, digits, or hyphens>")
if not CLOUDSQL_INSTANCE_RE.fullmatch(args.cloudsql_instance):
parser.error("--cloudsql-instance must be a safe lowercase Cloud SQL instance name")
if not PRIVATE_NETWORK_RE.fullmatch(args.expected_private_network):
parser.error("--expected-private-network must be a full projects/.../global/networks/... resource")
if not args.expected_private_network.startswith(f"projects/{args.project}/global/networks/"):
parser.error("--expected-private-network project must match --project")
if not GCE_INSTANCE_RE.fullmatch(args.expected_compute_instance):
parser.error("--expected-compute-instance must be a safe lowercase GCE instance name")
if not GCE_ZONE_RE.fullmatch(args.expected_compute_zone):
parser.error("--expected-compute-zone must be a safe GCE zone")
if not SERVICE_ACCOUNT_RE.fullmatch(args.expected_restore_service_account):
parser.error("--expected-restore-service-account must be an exact GCP service-account email")
if not SAFE_DNS_NAME_RE.fullmatch(args.ssl_server_name):
parser.error("--ssl-server-name must be a safe DNS name covered by the Cloud SQL server certificate")
try:
ssl_root_cert_mode = args.ssl_root_cert.lstat().st_mode
except OSError:
parser.error("--ssl-root-cert must be an existing regular file")
if not stat.S_ISREG(ssl_root_cert_mode) or args.ssl_root_cert.is_symlink():
parser.error("--ssl-root-cert must be a regular non-symlink file")
args.ssl_root_cert = args.ssl_root_cert.resolve()
if not SAFE_SSH_TARGET_RE.fullmatch(args.expected_source_ssh_target):
parser.error("--expected-source-ssh-target must be a host or user@host without options")
for value, flag in (
(args.expected_source_container, "--expected-source-container"),
(args.expected_source_database, "--expected-source-database"),
(args.expected_source_service, "--expected-source-service"),
):
if not SAFE_NAME_RE.fullmatch(value):
parser.error(f"{flag} contains unsafe characters")
try:
host = ipaddress.ip_address(args.host)
except ValueError:
@ -683,6 +1313,10 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
if args.operation == "restore":
if not SHA256_RE.fullmatch(args.expected_dump_sha256):
parser.error("--expected-dump-sha256 must be 64 lowercase hexadecimal characters")
if not SHA256_RE.fullmatch(args.expected_source_receipt_sha256):
parser.error("--expected-source-receipt-sha256 must be 64 lowercase hexadecimal characters")
if not AUTHORIZATION_REF_RE.fullmatch(args.expected_capture_authorization_ref):
parser.error("--expected-capture-authorization-ref must be 8-256 safe metadata characters")
for value, flag in (
(args.restore_timeout, "--restore-timeout"),
(args.manifest_timeout, "--manifest-timeout"),
@ -709,7 +1343,9 @@ def main(argv: list[str] | None = None) -> int:
finally:
for watched_signal, previous in previous_handlers.items():
signal.signal(watched_signal, previous)
print(json.dumps(payload, indent=2, sort_keys=True))
receipt_name = "restore-receipt.json" if args.operation == "restore" else "cleanup-receipt.json"
receipt_path = args.run_root.resolve() / args.run_id / receipt_name
print_private_receipt_summary(receipt_path, payload)
return 0 if payload.get("status") == "pass" else 1

2
ops/run_local_canonical_postgres_rebuild.py
View file

@ -31,7 +31,7 @@ MANIFEST_DESTINATION = "/tmp/postgres-parity-manifest.sql"
CANARY_LABEL = "com.livingip.teleo.canonical-rebuild-canary"
DATABASE_NAME_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]{0,62}\Z")
CONTAINER_PREFIX_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,39}\Z")
ALLOWED_APPLICATION_ROLES = frozenset({"kb_apply", "kb_review"})
ALLOWED_APPLICATION_ROLES = frozenset({"kb_gate_owner", "kb_apply", "kb_review"})
ROLE_BOOLEAN_FIELDS = (
"can_login",
"superuser",

792
ops/verify_gcp_canonical_lifecycle.py Normal file
View file

@ -0,0 +1,792 @@
#!/usr/bin/env python3
"""Verify one bound source-to-GCP restore, reasoning, and cleanup lifecycle."""
from __future__ import annotations
import argparse
import hashlib
import ipaddress
import json
import re
from datetime import UTC, datetime
from itertools import pairwise
from pathlib import Path
from typing import Any
try:
from .private_receipt_io import print_private_receipt_summary
from .restore_gcp_generated_postgres_snapshot import DEFAULT_RESTORE_SERVICE_ACCOUNT
from .verify_postgres_parity_manifest import REVIEWED_MANIFEST_SQL_SHA256
from .verify_vps_canonical_snapshot_delta import DELTA_SCHEMA, parse_timestamp, write_private_json
except ImportError: # pragma: no cover - direct script execution
from private_receipt_io import print_private_receipt_summary
from restore_gcp_generated_postgres_snapshot import DEFAULT_RESTORE_SERVICE_ACCOUNT
from verify_postgres_parity_manifest import REVIEWED_MANIFEST_SQL_SHA256
from verify_vps_canonical_snapshot_delta import DELTA_SCHEMA, parse_timestamp, write_private_json
SOURCE_SCHEMA = "livingip.sourceSnapshotReceipt.v2"
BLIND_REASONING_SCHEMA = "livingip.gcpGeneratedDbBlindClaimCanary.v1"
REASONING_COMPUTE_SCHEMA = "livingip.gcpReasoningComputeAttestation.v1"
RETRIEVAL_RECEIPT_SCHEMA = "livingip.teleoKbRetrievalReceipt.v1"
UUID_RE = re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}\b", re.I)
SHA256_RE = re.compile(r"[0-9a-f]{64}\Z")
FORBIDDEN_HANDLE_RE = re.compile(r"\b(?:cory|m3ta)\b", re.I)
EXPECTED_BLIND_ROW_IDS = {
"2a7ae257-d01d-46f4-b813-63f81bb9c7c7",
"261c3532-fa32-47d8-a5b5-6cc45035c267",
"15740795-ecc6-40fa-9a01-3d6bc7c54f79",
}
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def load_receipt(path: Path, label: str) -> dict[str, Any]:
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
raise ValueError(f"{label} is unreadable or invalid JSON") from exc
if not isinstance(payload, dict):
raise ValueError(f"{label} must be a JSON object")
return payload
def is_rfc1918(value: Any) -> bool:
try:
address = ipaddress.ip_address(str(value))
except ValueError:
return False
return any(
address in network
for network in (
ipaddress.ip_network("10.0.0.0/8"),
ipaddress.ip_network("172.16.0.0/12"),
ipaddress.ip_network("192.168.0.0/16"),
)
)
def nonempty_all_true(value: Any) -> bool:
return isinstance(value, dict) and bool(value) and all(item is True for item in value.values())
def required_true_checks(value: Any, required: set[str]) -> bool:
return nonempty_all_true(value) and required <= set(value)
def service_healthy(value: Any) -> bool:
if not isinstance(value, dict):
return False
try:
main_pid = int(value.get("MainPID") or 0)
restarts = int(value.get("NRestarts") or 0)
except (TypeError, ValueError):
return False
return (
value.get("ActiveState") == "active" and value.get("SubState") == "running" and main_pid > 0 and restarts >= 0
)
def parity_evidence(details: dict[str, Any]) -> dict[str, bool]:
structural = details.get("structural_hashes") or {}
required_structures = {
"schemas",
"columns",
"constraints",
"indexes",
"sequences",
"views",
"functions",
"triggers",
"types",
"policies",
"role_memberships",
"object_ownership",
"object_acl",
}
structural_valid = required_structures <= set(structural) and all(
bool(SHA256_RE.fullmatch(str((structural.get(kind) or {}).get("source") or "")))
and (structural.get(kind) or {}).get("source") == (structural.get(kind) or {}).get("target")
for kind in required_structures
)
performance = details.get("performance") or []
performance_valid = bool(performance) and all(
bool(row.get("query"))
and isinstance(row.get("source_ms"), (int, float))
and isinstance(row.get("target_ms"), (int, float))
and isinstance(row.get("source_ratio"), (int, float))
for row in performance
)
source_table_count = details.get("source_table_count")
target_table_count = details.get("target_table_count")
source_total_rows = details.get("source_total_rows")
target_total_rows = details.get("target_total_rows")
return {
"table_counts": isinstance(source_table_count, int)
and source_table_count > 0
and source_table_count == target_table_count,
"row_counts": isinstance(source_total_rows, int)
and source_total_rows > 0
and source_total_rows == target_total_rows,
"table_hashes": details.get("table_mismatches") == [] and details.get("table_hash_problems") == [],
"structural_hashes": structural_valid,
"roles": details.get("application_role_mismatches") == {}
and details.get("target_extra_application_roles") == [],
"extensions": details.get("required_extension_mismatches") == {}
and details.get("target_extra_extensions") == [],
"performance": performance_valid and details.get("performance_problems") == [],
}
def blind_reasoning_evidence(reasoning: dict[str, Any], target_db: Any) -> dict[str, bool]:
result = reasoning.get("result") or {}
gateway = result.get("gateway") or {}
child = gateway.get("child_process") or {}
tool_surface = gateway.get("tool_surface") or {}
tool_trace = result.get("database_tool_trace") or {}
calls = tool_trace.get("calls") or []
wrapper = reasoning.get("wrapper_tool_proof") or {}
invocations = wrapper.get("invocations") or []
prompt = str(reasoning.get("prompt") or "")
reply = str(result.get("reply") or "")
outcomes = reasoning.get("outcomes") or {}
required_outcomes = (
"asks_for_user_iteration_before_live",
"challenges_weak_support",
"decomposes_the_bundled_legal_claim",
"does_not_claim_a_database_change",
"proposes_candidate_claims",
"uses_only_m3taversal_handle",
)
required_subcommands = {"show", "evidence"}
discovery_subcommands = {"search", "context"}
traced_subcommands = {
str(invocation.get("subcommand")) for call in calls for invocation in (call.get("database_invocations") or [])
}
wrapper_subcommands = {str(invocation.get("subcommand")) for invocation in invocations}
retrieved_row_ids = {str(row_id) for call in calls for row_id in ((call.get("result") or {}).get("row_ids") or [])}
call_row_ids_valid = bool(calls) and all(
isinstance((call.get("result") or {}).get("row_ids"), list)
and isinstance((call.get("result") or {}).get("row_id_count"), int)
and (call.get("result") or {}).get("row_id_count") >= 0
and (call.get("result") or {}).get("row_id_count") >= len((call.get("result") or {}).get("row_ids") or [])
for call in calls
)
call_receipts_valid = bool(calls) and all(
(call.get("result") or {}).get("nonempty") is True
and (call.get("result") or {}).get("error_detected") is False
and ((call.get("result") or {}).get("retrieval_receipt") or {}).get("schema") == RETRIEVAL_RECEIPT_SCHEMA
and bool(
SHA256_RE.fullmatch(
str(((call.get("result") or {}).get("retrieval_receipt") or {}).get("semantic_context_sha256") or "")
)
)
and bool(
SHA256_RE.fullmatch(
str(((call.get("result") or {}).get("retrieval_receipt") or {}).get("artifact_state_sha256") or "")
)
)
and str(
((call.get("result") or {}).get("retrieval_receipt") or {}).get("read_consistency_status") or ""
).startswith("stable")
for call in calls
)
wrapper_invocations_valid = bool(invocations) and all(
invocation.get("database") == target_db
and invocation.get("returncode") == 0
and ((invocation.get("database_identity") or {}).get("transaction_read_only") == "on")
and ((invocation.get("database_identity") or {}).get("default_transaction_read_only") == "on")
and ((invocation.get("database_identity") or {}).get("ssl") is True)
and ((invocation.get("database_identity") or {}).get("sslmode") == "verify-full")
and ((invocation.get("database_identity") or {}).get("server_identity_verified") is True)
and bool((invocation.get("database_identity") or {}).get("ssl_server_name"))
and bool(
SHA256_RE.fullmatch(str((invocation.get("database_identity") or {}).get("ssl_root_cert_sha256") or ""))
)
and is_rfc1918((invocation.get("database_identity") or {}).get("server_address"))
for invocation in invocations
)
return {
"mode": reasoning.get("mode") == "gcp_generated_db_gatewayrunner_blind_claim_no_send",
"timestamps": bool(reasoning.get("generated_at_utc")) and bool(reasoning.get("completed_at_utc")),
"source_compute": bool(reasoning.get("source_compute")),
"blind_prompt": bool(prompt) and UUID_RE.search(prompt) is None,
"reply_nonempty": bool(reply.strip()),
"reply_uses_only_m3taversal_handle": FORBIDDEN_HANDLE_RE.search(reply) is None,
"outcomes": all(outcomes.get(name) is True for name in required_outcomes),
"gateway": gateway.get("authorized") is True
and gateway.get("handler_invoked") is True
and gateway.get("model_free_fallback_used") is False
and gateway.get("posted_to_telegram") is False,
"gateway_cleanup": child.get("exitcode") == 0
and child.get("alive_after_readback") is False
and child.get("process_group_alive_after_readback") is False
and child.get("timed_out") is False,
"send_tool_absent": tool_surface.get("send_message_tool_enabled") is False,
"tool_trace": tool_trace.get("schema") == "livingip.leoKbToolTrace.v1"
and tool_trace.get("database_tool_call_proven") is True
and tool_trace.get("database_retrieval_receipt_proven") is True
and tool_trace.get("database_tool_calls_read_only") is True
and tool_trace.get("database_tool_call_count") == len(calls)
and tool_trace.get("database_tool_completed_count") == len(calls),
"tool_receipts": call_receipts_valid,
"expected_claim_and_sources_retrieved": call_row_ids_valid and retrieved_row_ids >= EXPECTED_BLIND_ROW_IDS,
"required_tool_sequence": required_subcommands <= traced_subcommands
and bool(discovery_subcommands & traced_subcommands),
"wrapper": wrapper.get("all_bound_to_supplied_target") is True
and wrapper.get("all_completed_successfully") is True
and wrapper.get("complete_start_end_pairing") is True
and wrapper.get("database_read_only_required") is True
and wrapper.get("default_read_only_required") is True,
"wrapper_invocations": wrapper_invocations_valid
and required_subcommands <= wrapper_subcommands
and bool(discovery_subcommands & wrapper_subcommands),
"canonical_status_unchanged": bool(reasoning.get("canonical_status_before"))
and reasoning.get("canonical_status_before") == reasoning.get("canonical_status_after"),
"database_identity_unchanged": bool(reasoning.get("database_identity_before"))
and reasoning.get("database_identity_before") == reasoning.get("database_identity_after"),
"service_unchanged": service_healthy(reasoning.get("service_before"))
and reasoning.get("service_before") == reasoning.get("service_after"),
"temporary_profile_removed": reasoning.get("temp_profile_absent") is True
and reasoning.get("temp_profile_removed") is True,
}
def verify_lifecycle(
*,
source_path: Path,
current_source_path: Path,
source_delta_path: Path,
restore_path: Path,
parity_path: Path,
reasoning_path: Path,
reasoning_compute_path: Path,
cleanup_path: Path,
max_postflight_age_seconds: float = 900.0,
max_lifecycle_age_seconds: float = 3600.0,
now: datetime | None = None,
) -> dict[str, Any]:
if max_postflight_age_seconds <= 0:
raise ValueError("max postflight age must be positive")
if max_lifecycle_age_seconds <= 0:
raise ValueError("max lifecycle age must be positive")
verification_time = (now or datetime.now(UTC)).astimezone(UTC)
source = load_receipt(source_path, "source receipt")
current_source = load_receipt(current_source_path, "current source receipt")
source_delta = load_receipt(source_delta_path, "source delta receipt")
restore = load_receipt(restore_path, "restore receipt")
parity = load_receipt(parity_path, "parity receipt")
reasoning = load_receipt(reasoning_path, "reasoning receipt")
reasoning_compute_attestation = load_receipt(reasoning_compute_path, "reasoning compute attestation")
cleanup = load_receipt(cleanup_path, "cleanup receipt")
target_db = restore.get("target_database")
run_id = restore.get("run_id")
restore_source = restore.get("source") or {}
capture_binding = restore_source.get("capture_receipt") or {}
restore_target = restore.get("target") or {}
restore_connectivity = restore_target.get("connectivity") or {}
restore_connectivity_proof = restore_target.get("connectivity_proof") or {}
restore_cloudsql = restore_target.get("cloudsql") or {}
restore_compute = restore_target.get("compute") or {}
restore_parity = restore.get("parity") or {}
restore_service = restore.get("live_service") or {}
parity_details = parity.get("details") or {}
parity_evidence_checks = parity_evidence(parity_details)
restore_inline_parity_checks = parity_evidence(restore_parity.get("details") or {})
parity_connectivity = parity.get("private_connectivity") or {}
parity_connectivity_proof = parity_connectivity.get("proof") or {}
reasoning_checks = reasoning.get("checks") or {}
reasoning_parity = reasoning.get("parity_receipt") or {}
before_fingerprint = reasoning.get("database_fingerprint_before") or {}
after_fingerprint = reasoning.get("database_fingerprint_after") or {}
cleanup_database = cleanup.get("database") or {}
cleanup_service = cleanup.get("live_service") or {}
cleanup_cloudsql = (cleanup.get("target") or {}).get("cloudsql") or {}
cleanup_compute = (cleanup.get("target") or {}).get("compute") or {}
restore_contract = restore.get("execution_contract") or {}
cleanup_contract = cleanup.get("execution_contract") or {}
producer_reasoning_checks = blind_reasoning_evidence(reasoning, target_db)
reasoning_compute = reasoning_compute_attestation.get("compute") or {}
reasoning_compute_checks = reasoning_compute_attestation.get("checks") or {}
reasoning_compute_mutation = reasoning_compute_attestation.get("mutation") or {}
delta_baseline = source_delta.get("baseline") or {}
delta_current = source_delta.get("current") or {}
delta_details = source_delta.get("delta") or {}
source_snapshot = source.get("snapshot") or {}
current_source_snapshot = current_source.get("snapshot") or {}
timeline_values = {
"source_snapshot": source_snapshot.get("captured_at_utc"),
"restore_started": restore.get("generated_at_utc"),
"restore_completed": restore.get("completed_at_utc"),
"parity_completed": parity.get("completed_at_utc"),
"reasoning_started": reasoning.get("generated_at_utc"),
"reasoning_completed": reasoning.get("completed_at_utc"),
"reasoning_compute_attested": reasoning_compute_attestation.get("generated_at_utc"),
"cleanup_started": cleanup.get("generated_at_utc"),
"cleanup_completed": cleanup.get("completed_at_utc"),
"current_source_snapshot": current_source_snapshot.get("captured_at_utc"),
}
timeline = [parse_timestamp(value, name) for name, value in timeline_values.items()]
lifecycle_chronology = all(earlier <= later for earlier, later in pairwise(timeline))
delta_detected = delta_details.get("delta_detected")
delta_change_present = bool(
delta_details.get("added_tables")
or delta_details.get("removed_tables")
or delta_details.get("changed_tables")
or delta_details.get("changed_structures")
or delta_details.get("total_row_delta")
)
current_source_time = parse_timestamp(
current_source_snapshot.get("captured_at_utc"), "current source snapshot time"
)
restore_started_time = parse_timestamp(restore.get("generated_at_utc"), "restore start time")
cleanup_completed_time = parse_timestamp(cleanup.get("completed_at_utc"), "cleanup completion time")
postflight_age_seconds = (verification_time - current_source_time).total_seconds()
lifecycle_age_seconds = (verification_time - cleanup_completed_time).total_seconds()
lifecycle_span_seconds = (current_source_time - restore_started_time).total_seconds()
postflight_after_lifecycle = current_source_time >= max(
parse_timestamp(reasoning.get("completed_at_utc"), "reasoning completion time"),
parse_timestamp(cleanup.get("completed_at_utc"), "cleanup completion time"),
)
required_reasoning_checks = (
"exact_blind_prompt_without_ids",
"claim_and_sources_retrieved",
"discovery_show_evidence_completed",
"retrieval_receipts_proven",
"private_tls_read_only_target",
"generated_database_unchanged",
"canonical_counts_unchanged",
"database_calls_read_only",
"no_database_write",
"no_telegram_send",
"live_service_unchanged",
"live_profile_unchanged",
"temporary_profile_removed",
)
checks = {
"source_schema_v2": source.get("schema") == SOURCE_SCHEMA and source.get("receipt_version") == 2,
"source_pass": source.get("status") == "pass"
and source.get("snapshot_exported") is True
and (source.get("source_service") or {}).get("unchanged") is True
and service_healthy((source.get("source_service") or {}).get("before"))
and (source.get("source_service") or {}).get("before") == (source.get("source_service") or {}).get("after"),
"source_authorized": bool(source.get("capture_authorization_ref")),
"source_authorization_bound": capture_binding.get("capture_authorization_ref")
== source.get("capture_authorization_ref"),
"source_service_unchanged": source.get("service_unchanged") is True,
"source_not_mutated": source.get("production_db_mutated") is False,
"current_source_schema_v2": current_source.get("schema") == SOURCE_SCHEMA
and current_source.get("receipt_version") == 2,
"current_source_pass": current_source.get("status") == "pass"
and current_source.get("snapshot_exported") is True
and current_source.get("service_unchanged") is True
and (current_source.get("source_service") or {}).get("unchanged") is True
and service_healthy((current_source.get("source_service") or {}).get("before"))
and (current_source.get("source_service") or {}).get("before")
== (current_source.get("source_service") or {}).get("after")
and current_source.get("production_db_mutated") is False,
"current_source_identity": current_source.get("source") == source.get("source")
and current_source_snapshot.get("system_identifier") == source_snapshot.get("system_identifier"),
"source_delta_pass": source_delta.get("artifact") == "vps_canonical_snapshot_postflight_delta"
and source_delta.get("schema") == DELTA_SCHEMA
and source_delta.get("status") == "pass",
"source_delta_receipts_bound": delta_baseline.get("capture_receipt_sha256") == sha256_file(source_path)
and delta_current.get("capture_receipt_sha256") == sha256_file(current_source_path),
"source_delta_manifests_bound": delta_baseline.get("manifest_sha256")
== (source.get("manifest") or {}).get("sha256")
and delta_current.get("manifest_sha256") == (current_source.get("manifest") or {}).get("sha256"),
"source_delta_manifest_sql_bound": delta_baseline.get("manifest_sql_sha256")
== (source.get("manifest") or {}).get("manifest_sql_sha256")
and delta_current.get("manifest_sql_sha256")
== (current_source.get("manifest") or {}).get("manifest_sql_sha256")
and delta_baseline.get("manifest_sql_sha256") == delta_current.get("manifest_sql_sha256"),
"source_delta_manifest_sql_reviewed": delta_baseline.get("manifest_sql_sha256") == REVIEWED_MANIFEST_SQL_SHA256
and delta_current.get("manifest_sql_sha256") == REVIEWED_MANIFEST_SQL_SHA256
and (source.get("manifest") or {}).get("manifest_sql_sha256") == REVIEWED_MANIFEST_SQL_SHA256
and (current_source.get("manifest") or {}).get("manifest_sql_sha256") == REVIEWED_MANIFEST_SQL_SHA256
and capture_binding.get("manifest_sql_sha256") == REVIEWED_MANIFEST_SQL_SHA256,
"source_delta_authorizations_bound": delta_baseline.get("capture_authorization_ref")
== source.get("capture_authorization_ref")
and delta_current.get("capture_authorization_ref") == current_source.get("capture_authorization_ref"),
"source_delta_explicit": isinstance(delta_details.get("delta_detected"), bool)
and isinstance(delta_details.get("added_tables"), list)
and isinstance(delta_details.get("removed_tables"), list)
and isinstance(delta_details.get("changed_tables"), list)
and isinstance(delta_details.get("changed_structures"), list)
and isinstance(delta_details.get("total_row_delta"), int),
"source_delta_baseline_shape": delta_baseline.get("run_id") == source.get("run_id")
and delta_baseline.get("captured_at_utc") == source_snapshot.get("captured_at_utc")
and delta_baseline.get("table_count") == (source.get("manifest") or {}).get("table_count")
and delta_baseline.get("total_rows") == (source.get("manifest") or {}).get("total_rows")
and bool(SHA256_RE.fullmatch(str(delta_baseline.get("table_fingerprint_sha256") or "")))
and bool(SHA256_RE.fullmatch(str(delta_baseline.get("structure_fingerprint_sha256") or ""))),
"source_delta_current_shape": delta_current.get("run_id") == current_source.get("run_id")
and delta_current.get("captured_at_utc") == current_source_snapshot.get("captured_at_utc")
and delta_current.get("table_count") == (current_source.get("manifest") or {}).get("table_count")
and delta_current.get("total_rows") == (current_source.get("manifest") or {}).get("total_rows")
and bool(SHA256_RE.fullmatch(str(delta_current.get("table_fingerprint_sha256") or "")))
and bool(SHA256_RE.fullmatch(str(delta_current.get("structure_fingerprint_sha256") or ""))),
"source_delta_consistent": delta_detected is delta_change_present
and delta_details.get("total_row_delta")
== (delta_current.get("total_rows") or 0) - (delta_baseline.get("total_rows") or 0)
and (
delta_detected is True
or (
delta_baseline.get("table_count") == delta_current.get("table_count")
and delta_baseline.get("total_rows") == delta_current.get("total_rows")
and delta_baseline.get("table_fingerprint_sha256") == delta_current.get("table_fingerprint_sha256")
and delta_baseline.get("structure_fingerprint_sha256")
== delta_current.get("structure_fingerprint_sha256")
)
),
"source_postflight_after_lifecycle": postflight_after_lifecycle,
"source_postflight_fresh": 0 <= postflight_age_seconds <= max_postflight_age_seconds,
"gcp_lifecycle_fresh": 0 <= lifecycle_age_seconds <= max_lifecycle_age_seconds,
"lifecycle_span_bounded": 0 <= lifecycle_span_seconds <= max_lifecycle_age_seconds,
"lifecycle_chronology": lifecycle_chronology,
"restore_pass": restore.get("artifact") == "gcp_generated_postgres_snapshot_restore"
and restore.get("status") == "pass",
"restore_target_is_disposable": isinstance(target_db, str) and target_db.startswith("teleo_clone_"),
"restore_run_id": isinstance(run_id, str) and run_id.startswith("gcp-restore-"),
"source_receipt_hash_bound": capture_binding.get("sha256") == sha256_file(source_path),
"source_provenance_bound": capture_binding.get("provenance_binding_sha256")
== (source.get("provenance_binding") or {}).get("sha256"),
"source_capture_checks": required_true_checks(
capture_binding.get("checks"),
{
"artifact",
"schema",
"receipt_version",
"status",
"snapshot_exported",
"service_unchanged",
"source_service_healthy",
"source_not_mutated",
"telegram_not_sent",
"run_id",
"capture_authorization_ref",
"source_identity",
"source_database",
"snapshot_identity",
"dump_sha256",
"dump_bytes",
"source_manifest_sha256",
"manifest_sql_sha256",
"source_context_sha256",
"table_count",
"total_rows",
"binding_algorithm",
"binding_payload",
"binding_sha256",
},
),
"source_dump_bound": capture_binding.get("dump_sha256") == (source.get("dump") or {}).get("sha256"),
"source_execution_contract_bound": restore_contract.get("source") == source.get("source"),
"source_manifest_bound": capture_binding.get("manifest_sha256") == (source.get("manifest") or {}).get("sha256"),
"restore_manifest_matches_parity_source": capture_binding.get("manifest_sha256")
== parity.get("source_manifest_sha256"),
"restore_target_manifest_matches_parity": restore_target.get("manifest_sha256")
== parity.get("target_manifest_sha256"),
"restore_private_tls": restore_connectivity.get("private_connectivity") is True
and restore_connectivity.get("ssl") is True
and restore_connectivity.get("sslmode") == "verify-full"
and restore_connectivity.get("server_identity_verified") is True
and restore_connectivity.get("ssl_server_name") == restore_contract.get("ssl_server_name")
and restore_connectivity.get("ssl_root_cert_sha256") == restore_contract.get("ssl_root_cert_sha256")
and is_rfc1918(restore_connectivity.get("server_address")),
"restore_public_ip_disabled": restore_cloudsql.get("public_ip_disabled") is True,
"restore_cloudsql_checks": required_true_checks(
restore_cloudsql.get("checks"),
{
"instance",
"state",
"postgres_16",
"public_ip_disabled",
"private_network",
"expected_private_host",
"no_non_private_address",
},
),
"restore_compute_checks": required_true_checks(
restore_compute.get("checks"),
{
"project",
"project_number",
"expected_network_project",
"instance",
"instance_id",
"zone",
"network",
"private_ip",
"service_account",
},
),
"restore_compute_contract": restore_compute.get("project") == restore_contract.get("project")
and restore_compute.get("instance") == restore_contract.get("compute_instance")
and restore_compute.get("zone") == restore_contract.get("compute_zone")
and restore_compute.get("expected_private_network") == restore_contract.get("private_network")
and bool(restore_compute.get("network_resource"))
and is_rfc1918(restore_compute.get("private_ip"))
and restore_contract.get("restore_service_account") == DEFAULT_RESTORE_SERVICE_ACCOUNT
and restore_compute.get("service_account") == DEFAULT_RESTORE_SERVICE_ACCOUNT,
"restore_tls_identity_contract": bool(restore_contract.get("ssl_server_name"))
and bool(SHA256_RE.fullmatch(str(restore_contract.get("ssl_root_cert_sha256") or ""))),
"restore_connectivity_compute_bound": restore_connectivity.get("source_compute")
== restore_compute.get("instance"),
"restore_inline_parity": restore_parity.get("status") == "pass" and restore_parity.get("problems") == [],
"restore_inline_parity_evidence": all(restore_inline_parity_checks.values()),
"restore_service_healthy_unchanged": restore_service.get("unchanged") is True
and service_healthy(restore_service.get("before"))
and restore_service.get("before") == restore_service.get("after"),
"parity_pass": parity.get("artifact") == "canonical_postgres_parity_verification"
and parity.get("status") == "pass"
and parity.get("scope") == "gcp_staging"
and parity.get("problems") == [],
"parity_target_bound": parity_details.get("target_database") == target_db,
"parity_exact_evidence": all(parity_evidence_checks.values()),
"parity_connectivity_hash_bound": restore_connectivity_proof.get("sha256")
== parity.get("connectivity_proof_sha256"),
"parity_private_proof": parity_connectivity.get("required") is True
and parity_connectivity_proof.get("artifact") == "gcp_private_postgres_connectivity"
and parity_connectivity_proof.get("status") == "pass"
and parity_connectivity_proof.get("schema") == "livingip.gcpPrivatePostgresConnectivity.v2"
and parity_connectivity_proof.get("restore_run_id") == run_id
and parity_connectivity_proof.get("private_connectivity") is True
and parity_connectivity_proof.get("public_ip_disabled") is True
and parity_connectivity_proof.get("target_database") == target_db
and parity_connectivity_proof.get("project") == restore_cloudsql.get("project")
and parity_connectivity_proof.get("cloudsql_instance") == restore_cloudsql.get("instance")
and parity_connectivity_proof.get("private_network") == restore_cloudsql.get("private_network")
and parity_connectivity_proof.get("source_compute") == reasoning.get("source_compute")
and parity_connectivity_proof.get("source_compute_instance_id") == restore_compute.get("instance_id")
and parity_connectivity_proof.get("source_compute_zone") == restore_compute.get("zone")
and is_rfc1918(parity_connectivity_proof.get("source_compute_private_ip"))
and parity_connectivity_proof.get("server_address") == restore_connectivity.get("server_address")
and parity_connectivity_proof.get("sslmode") == "verify-full"
and parity_connectivity_proof.get("server_identity_verified") is True
and parity_connectivity_proof.get("ssl_server_name") == restore_contract.get("ssl_server_name")
and parity_connectivity_proof.get("ssl_root_cert_sha256") == restore_contract.get("ssl_root_cert_sha256")
and required_true_checks(
parity_connectivity_proof.get("cloudsql_control_plane_checks"),
{
"instance",
"state",
"postgres_16",
"public_ip_disabled",
"private_network",
"expected_private_host",
"no_non_private_address",
},
)
and required_true_checks(
parity_connectivity_proof.get("compute_identity_checks"),
{
"project",
"project_number",
"expected_network_project",
"instance",
"instance_id",
"zone",
"network",
"private_ip",
"service_account",
},
),
"reasoning_schema": reasoning.get("schema") == BLIND_REASONING_SCHEMA,
"reasoning_pass": reasoning.get("status") == "pass" and not reasoning.get("errors"),
"reasoning_target_bound": reasoning.get("target_database") == target_db,
"reasoning_compute_bound": reasoning.get("source_compute") == restore_compute.get("instance"),
"reasoning_parity_hash_bound": reasoning_parity.get("sha256") == sha256_file(parity_path),
"reasoning_required_checks": all(reasoning_checks.get(name) is True for name in required_reasoning_checks),
"reasoning_producer_evidence": all(producer_reasoning_checks.values()),
"reasoning_compute_attestation": reasoning_compute_attestation.get("artifact")
== "gcp_reasoning_compute_attestation"
and reasoning_compute_attestation.get("schema") == REASONING_COMPUTE_SCHEMA
and reasoning_compute_attestation.get("status") == "pass"
and reasoning_compute_attestation.get("method") == "gce_metadata_server_v1"
and nonempty_all_true(reasoning_compute_checks)
and reasoning_compute_attestation.get("restore_run_id") == run_id
and reasoning_compute_attestation.get("target_database") == target_db
and all(value is False for value in reasoning_compute_mutation.values())
and {
"cloudsql_changed",
"compute_changed",
"database_changed",
"telegram_message_sent",
}
<= set(reasoning_compute_mutation),
"reasoning_compute_receipt_bound": (reasoning_compute_attestation.get("reasoning_receipt") or {}).get("sha256")
== sha256_file(reasoning_path),
"reasoning_compute_identity_bound": reasoning_compute.get("project") == restore_compute.get("project")
and reasoning_compute.get("project_number") == restore_compute.get("project_number")
and reasoning_compute.get("instance") == restore_compute.get("instance")
and reasoning_compute.get("instance_id") == restore_compute.get("instance_id")
and reasoning_compute.get("zone") == restore_compute.get("zone")
and reasoning_compute.get("expected_private_network") == restore_compute.get("expected_private_network")
and reasoning_compute.get("network_resource") == restore_compute.get("network_resource")
and reasoning_compute.get("private_ip") == restore_compute.get("private_ip")
and reasoning_compute.get("service_account") == restore_compute.get("service_account")
and required_true_checks(
reasoning_compute.get("checks"),
{
"project",
"project_number",
"expected_network_project",
"instance",
"instance_id",
"zone",
"network",
"private_ip",
"service_account",
},
),
"reasoning_no_send_no_write": reasoning.get("posted_to_telegram") is False
and reasoning.get("database_write_attempted") is False,
"reasoning_fingerprint_unchanged": bool(before_fingerprint.get("sha256"))
and before_fingerprint.get("sha256") == after_fingerprint.get("sha256"),
"cleanup_pass": cleanup.get("artifact") == "gcp_generated_postgres_snapshot_cleanup"
and cleanup.get("status") == "pass",
"cleanup_run_bound": cleanup.get("run_id") == run_id and cleanup.get("target_database") == target_db,
"cleanup_restore_receipt_bound": (cleanup.get("restore_receipt") or {}).get("sha256")
== sha256_file(restore_path),
"cleanup_execution_contract_bound": bool(restore_contract) and cleanup_contract == restore_contract,
"cleanup_cloudsql_bound": cleanup_cloudsql.get("project") == restore_cloudsql.get("project")
and cleanup_cloudsql.get("instance") == restore_cloudsql.get("instance")
and cleanup_cloudsql.get("expected_private_host") == restore_cloudsql.get("expected_private_host")
and cleanup_cloudsql.get("expected_private_network") == restore_cloudsql.get("expected_private_network")
and cleanup_cloudsql.get("public_ip_disabled") is True
and required_true_checks(
cleanup_cloudsql.get("checks"),
{
"instance",
"state",
"postgres_16",
"public_ip_disabled",
"private_network",
"expected_private_host",
"no_non_private_address",
},
),
"cleanup_compute_bound": cleanup_compute.get("instance_id") == restore_compute.get("instance_id")
and cleanup_compute.get("project") == restore_compute.get("project")
and cleanup_compute.get("project_number") == restore_compute.get("project_number")
and cleanup_compute.get("instance") == restore_compute.get("instance")
and cleanup_compute.get("zone") == restore_compute.get("zone")
and cleanup_compute.get("expected_private_network") == restore_compute.get("expected_private_network")
and cleanup_compute.get("network_resource") == restore_compute.get("network_resource")
and cleanup_compute.get("private_ip") == restore_compute.get("private_ip")
and cleanup_compute.get("service_account") == restore_compute.get("service_account")
and required_true_checks(
cleanup_compute.get("checks"),
{
"project",
"project_number",
"expected_network_project",
"instance",
"instance_id",
"zone",
"network",
"private_ip",
"service_account",
},
),
"clone_absent": cleanup_database.get("existed_before") is True
and cleanup_database.get("clone_database_remaining") == 0,
"cleanup_service_unchanged": cleanup_service.get("unchanged") is True
and service_healthy(cleanup_service.get("before"))
and cleanup_service.get("before") == cleanup_service.get("after"),
}
failed = sorted(name for name, passed in checks.items() if not passed)
structurally_valid = not failed
return {
"artifact": "gcp_canonical_snapshot_lifecycle_verification",
"generated_at_utc": verification_time.isoformat(),
"status": "prepared_external_attestation_required" if structurally_valid else "fail",
"current_tier": "T2_structural_receipt_bundle",
"required_tier": "T3_live_private_gcp_staging",
"t3_receipt_ready": False,
"accepted_by_this_api": False,
"target_database": target_db,
"run_id": run_id,
"checks": checks,
"reasoning_producer_checks": producer_reasoning_checks,
"restore_inline_parity_checks": restore_inline_parity_checks,
"parity_evidence_checks": parity_evidence_checks,
"post_snapshot_source_delta": delta_details,
"timeline": timeline_values,
"postflight_age_seconds": postflight_age_seconds,
"max_postflight_age_seconds": max_postflight_age_seconds,
"lifecycle_age_seconds": lifecycle_age_seconds,
"lifecycle_span_seconds": lifecycle_span_seconds,
"max_lifecycle_age_seconds": max_lifecycle_age_seconds,
"failed_checks": failed,
"receipt_sha256": {
"source": sha256_file(source_path),
"current_source": sha256_file(current_source_path),
"source_delta": sha256_file(source_delta_path),
"restore": sha256_file(restore_path),
"parity": sha256_file(parity_path),
"reasoning": sha256_file(reasoning_path),
"reasoning_compute": sha256_file(reasoning_compute_path),
"cleanup": sha256_file(cleanup_path),
},
"strongest_claim_allowed": (
"caller-supplied lifecycle receipts are internally consistent; an independently authenticated platform receipt is still required"
if structurally_valid
else "lifecycle preparation is incomplete; failed checks must be repaired"
),
"not_proven": ["production cutover", "ongoing replication", "Telegram delivery"],
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--source-receipt", required=True, type=Path)
parser.add_argument("--current-source-receipt", required=True, type=Path)
parser.add_argument("--source-delta-receipt", required=True, type=Path)
parser.add_argument("--restore-receipt", required=True, type=Path)
parser.add_argument("--parity-receipt", required=True, type=Path)
parser.add_argument("--reasoning-receipt", required=True, type=Path)
parser.add_argument("--reasoning-compute-receipt", required=True, type=Path)
parser.add_argument("--cleanup-receipt", required=True, type=Path)
parser.add_argument("--output", required=True, type=Path)
parser.add_argument("--max-postflight-age-seconds", default=900.0, type=float)
parser.add_argument("--max-lifecycle-age-seconds", default=3600.0, type=float)
args = parser.parse_args()
try:
payload = verify_lifecycle(
source_path=args.source_receipt,
current_source_path=args.current_source_receipt,
source_delta_path=args.source_delta_receipt,
restore_path=args.restore_receipt,
parity_path=args.parity_receipt,
reasoning_path=args.reasoning_receipt,
reasoning_compute_path=args.reasoning_compute_receipt,
cleanup_path=args.cleanup_receipt,
max_postflight_age_seconds=args.max_postflight_age_seconds,
max_lifecycle_age_seconds=args.max_lifecycle_age_seconds,
)
except (OSError, ValueError, KeyError, TypeError) as exc:
payload = {
"artifact": "gcp_canonical_snapshot_lifecycle_verification",
"generated_at_utc": datetime.now(UTC).isoformat(),
"status": "fail",
"required_tier": "T3_live_private_gcp_staging",
"failed_checks": ["receipt_load_or_validation_error"],
"error": str(exc),
"strongest_claim_allowed": "lifecycle incomplete; receipts could not be validated",
}
write_private_json(args.output, payload)
print_private_receipt_summary(args.output, payload)
return 0 if payload.get("status") == "prepared_external_attestation_required" else 1
if __name__ == "__main__":
raise SystemExit(main())

8
ops/verify_gcp_leoclean_runtime_permissions.py
View file

@ -259,6 +259,7 @@ CATALOG_TRUE_FIELDS: tuple[str, ...] = (
"effective_setting_contract",
)
@dataclass(frozen=True)
class CommandResult:
returncode: int
@ -384,8 +385,7 @@ def _assert_trusted_executable(raw_path: str) -> None:
except OSError:
raise VerificationError("runtime_dependency_untrusted", "runtime_dependencies") from None
if requested_metadata.st_uid != 0 or (
not stat.S_ISLNK(requested_metadata.st_mode)
and requested_metadata.st_mode & (stat.S_IWGRP | stat.S_IWOTH)
not stat.S_ISLNK(requested_metadata.st_mode) and requested_metadata.st_mode & (stat.S_IWGRP | stat.S_IWOTH)
):
raise VerificationError("runtime_dependency_untrusted", "runtime_dependencies")
for candidate in (*_path_and_parents(requested.parent), *_path_and_parents(resolved)):
@ -790,9 +790,7 @@ def _catalog_privilege_posture_sql() -> str:
allowlist_values = ",\n ".join(
f"({_sql_literal(schema)}, {_sql_literal(relation)})" for schema, relation in READ_TABLE_ALLOWLIST
)
owner_insert_values = ",\n ".join(
f"({_sql_literal(column_name)})" for column_name in STAGE_OWNER_INSERT_COLUMNS
)
owner_insert_values = ",\n ".join(f"({_sql_literal(column_name)})" for column_name in STAGE_OWNER_INSERT_COLUMNS)
return f"""
with runtime_role as (
select oid from pg_catalog.pg_roles where rolname = {_sql_literal(RUNTIME_DATABASE_ROLE)}

85
ops/verify_postgres_parity_manifest.py
View file

@ -7,15 +7,27 @@ import argparse
import hashlib
import ipaddress
import json
import re
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
try:
from .private_receipt_io import print_private_receipt_summary, write_private_json
except ImportError: # pragma: no cover - direct script execution
from private_receipt_io import print_private_receipt_summary, write_private_json
REVIEWED_MANIFEST_SQL_SHA256 = "2bd3e353627d44d4a292949952c06b1897d1a9a9e98243905c7a834d007db188"
ROWSET_MD5_RE = re.compile(r"[0-9a-f]{32}\Z")
SINGLETON_KINDS = {
"identity",
"schemas",
"extensions",
"application_roles",
"role_memberships",
"object_ownership",
"object_acl",
"columns",
"constraints",
"indexes",
@ -33,6 +45,14 @@ def canonical_hash(value: Any) -> str:
return hashlib.sha256(payload.encode()).hexdigest()
def file_sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def load_manifest(path: Path) -> dict[str, Any]:
singleton: dict[str, dict[str, Any]] = {}
tables: dict[str, dict[str, Any]] = {}
@ -72,7 +92,18 @@ def load_manifest(path: Path) -> dict[str, Any]:
def item_map(row: dict[str, Any], key: str) -> dict[str, dict[str, Any]]:
return {str(item[key]): item for item in row.get("items", [])}
items = row.get("items", [])
if not isinstance(items, list):
raise ValueError(f"manifest {row.get('kind')} items must be a list")
mapped: dict[str, dict[str, Any]] = {}
for item in items:
if not isinstance(item, dict) or key not in item:
raise ValueError(f"manifest {row.get('kind')} item is missing {key}")
item_key = str(item[key])
if item_key in mapped:
raise ValueError(f"manifest {row.get('kind')} contains duplicate {key}={item_key}")
mapped[item_key] = item
return mapped
def compare_manifests(
@ -107,9 +138,14 @@ def compare_manifests(
+ json.dumps(sorted(set(target_tables) - set(source_tables)))
)
table_mismatches = []
table_hash_problems = []
for table in sorted(set(source_tables) & set(target_tables)):
source_row = source_tables[table]
target_row = target_tables[table]
for side, row in (("source", source_row), ("target", target_row)):
rowset_md5 = row.get("rowset_md5")
if not isinstance(rowset_md5, str) or not ROWSET_MD5_RE.fullmatch(rowset_md5):
table_hash_problems.append(f"{table}:{side}:rowset_md5_invalid")
mismatch = {
field: {"source": source_row.get(field), "target": target_row.get(field)}
for field in ("row_count", "rowset_md5")
@ -119,6 +155,8 @@ def compare_manifests(
table_mismatches.append({"table": table, "mismatch": mismatch})
if table_mismatches:
problems.append(f"table_content_mismatches={len(table_mismatches)}")
if table_hash_problems:
problems.append(f"table_hash_problems={len(table_hash_problems)}")
structural_hashes: dict[str, dict[str, str]] = {}
for kind in (
@ -132,6 +170,9 @@ def compare_manifests(
"triggers",
"types",
"policies",
"role_memberships",
"object_ownership",
"object_acl",
):
source_hash = canonical_hash(source_singleton[kind].get("items", []))
target_hash = canonical_hash(target_singleton[kind].get("items", []))
@ -142,19 +183,19 @@ def compare_manifests(
source_extensions = item_map(source_singleton["extensions"], "name")
target_extensions = item_map(target_singleton["extensions"], "name")
extension_mismatches = {
name: {"source": item, "target": target_extensions.get(name)}
for name, item in source_extensions.items()
if target_extensions.get(name) != item
name: {"source": source_extensions.get(name), "target": target_extensions.get(name)}
for name in sorted(set(source_extensions) | set(target_extensions))
if source_extensions.get(name) != target_extensions.get(name)
}
if extension_mismatches:
problems.append(f"required_extension_mismatches={len(extension_mismatches)}")
problems.append(f"extension_mismatches={len(extension_mismatches)}")
source_roles = item_map(source_singleton["application_roles"], "name")
target_roles = item_map(target_singleton["application_roles"], "name")
role_mismatches = {
name: {"source": item, "target": target_roles.get(name)}
for name, item in source_roles.items()
if target_roles.get(name) != item
name: {"source": source_roles.get(name), "target": target_roles.get(name)}
for name in sorted(set(source_roles) | set(target_roles))
if source_roles.get(name) != target_roles.get(name)
}
if role_mismatches:
problems.append(f"application_role_mismatches={len(role_mismatches)}")
@ -204,8 +245,10 @@ def compare_manifests(
"source_total_rows": sum(int(row["row_count"]) for row in source_tables.values()),
"target_total_rows": sum(int(row["row_count"]) for row in target_tables.values()),
"table_mismatches": table_mismatches,
"table_hash_problems": table_hash_problems,
"structural_hashes": structural_hashes,
"required_extension_mismatches": extension_mismatches,
"target_extra_extensions": sorted(set(target_extensions) - set(source_extensions)),
"application_role_mismatches": role_mismatches,
"target_extra_application_roles": sorted(set(target_roles) - set(source_roles)),
"performance": performance_rows,
@ -280,6 +323,14 @@ def load_connectivity(
problems.append("gcp_target_manifest_tls_not_proven")
if proof.get("ssl") is not True:
problems.append("gcp_connectivity_tls_not_proven")
if proof.get("sslmode") != "verify-full":
problems.append("gcp_connectivity_server_identity_mode_not_proven")
if proof.get("server_identity_verified") is not True:
problems.append("gcp_connectivity_server_identity_not_proven")
if not proof.get("ssl_server_name"):
problems.append("gcp_connectivity_ssl_server_name_missing")
if not re.fullmatch(r"[0-9a-f]{64}", str(proof.get("ssl_root_cert_sha256") or "")):
problems.append("gcp_connectivity_ssl_root_cert_hash_missing")
return problems, {"required": True, "path": str(path), "proof": proof}
@ -295,6 +346,9 @@ def main() -> int:
args = parser.parse_args()
try:
source_manifest_sha256 = file_sha256(args.source)
target_manifest_sha256 = file_sha256(args.target)
connectivity_proof_sha256 = file_sha256(args.connectivity_proof) if args.connectivity_proof else None
source = load_manifest(args.source)
target = load_manifest(args.target)
problems, details = compare_manifests(
@ -316,18 +370,26 @@ def main() -> int:
problems = [f"manifest_error={exc}"]
details = {}
connectivity = {"required": args.scope == "gcp_staging", "status": "not_checked"}
source_manifest_sha256 = None
target_manifest_sha256 = None
connectivity_proof_sha256 = None
error = str(exc)
completed_at_utc = datetime.now(UTC).isoformat()
payload = {
"artifact": "canonical_postgres_parity_verification",
"generated_at_utc": datetime.now(UTC).isoformat(),
"generated_at_utc": completed_at_utc,
"completed_at_utc": completed_at_utc,
"scope": args.scope,
"source_manifest": str(args.source),
"source_manifest_sha256": source_manifest_sha256,
"target_manifest": str(args.target),
"target_manifest_sha256": target_manifest_sha256,
"status": status,
"problems": problems,
"details": details,
"private_connectivity": connectivity,
"connectivity_proof_sha256": connectivity_proof_sha256,
"error": error,
"not_proven_by_this_artifact": [
"GCP staging Leo composition replay",
@ -335,9 +397,8 @@ def main() -> int:
"production cutover",
],
}
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
print(json.dumps(payload, indent=2, sort_keys=True))
write_private_json(args.output, payload)
print_private_receipt_summary(args.output, payload)
return 0 if status == "pass" else 1

303
ops/verify_vps_canonical_snapshot_delta.py Normal file
View file

@ -0,0 +1,303 @@
#!/usr/bin/env python3
"""Bind a postflight VPS manifest to a captured snapshot and report exact deltas."""
from __future__ import annotations
import argparse
import json
import re
import stat
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
try:
from .capture_vps_canonical_postgres_snapshot import (
SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
build_provenance_binding,
)
from .private_receipt_io import print_private_receipt_summary, write_private_json
from .verify_postgres_parity_manifest import (
REVIEWED_MANIFEST_SQL_SHA256,
SINGLETON_KINDS,
canonical_hash,
file_sha256,
load_manifest,
)
except ImportError: # pragma: no cover - direct script execution
from capture_vps_canonical_postgres_snapshot import SOURCE_SNAPSHOT_RECEIPT_SCHEMA, build_provenance_binding
from private_receipt_io import print_private_receipt_summary, write_private_json
from verify_postgres_parity_manifest import (
REVIEWED_MANIFEST_SQL_SHA256,
SINGLETON_KINDS,
canonical_hash,
file_sha256,
load_manifest,
)
DELTA_SCHEMA = "livingip.vpsCanonicalSnapshotDelta.v1"
SHA256_RE = re.compile(r"[0-9a-f]{64}\Z")
def service_healthy(value: Any) -> bool:
if not isinstance(value, dict):
return False
try:
main_pid = int(value.get("MainPID") or 0)
restarts = int(value.get("NRestarts") or 0)
except (TypeError, ValueError):
return False
return (
value.get("ActiveState") == "active" and value.get("SubState") == "running" and main_pid > 0 and restarts >= 0
)
def regular_file(path: Path, label: str) -> Path:
try:
mode = path.lstat().st_mode
except OSError as exc:
raise ValueError(f"{label} is unavailable: {exc}") from exc
if not stat.S_ISREG(mode) or path.is_symlink():
raise ValueError(f"{label} must be a regular non-symlink file")
return path.resolve()
def load_json(path: Path, label: str) -> dict[str, Any]:
path = regular_file(path, label)
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except json.JSONDecodeError as exc:
raise ValueError(f"{label} is invalid JSON") from exc
if not isinstance(payload, dict):
raise ValueError(f"{label} must be a JSON object")
return payload
def parse_timestamp(value: Any, label: str) -> datetime:
text = str(value or "")
if text.endswith("Z"):
text = text[:-1] + "+00:00"
try:
parsed = datetime.fromisoformat(text)
except ValueError as exc:
raise ValueError(f"{label} is not an ISO-8601 timestamp") from exc
if parsed.tzinfo is None:
raise ValueError(f"{label} must include a timezone")
return parsed.astimezone(UTC)
def validate_capture(
receipt_path: Path,
manifest_path: Path,
*,
expected_authorization_ref: str,
label: str,
) -> tuple[dict[str, Any], dict[str, Any]]:
receipt = load_json(receipt_path, f"{label} receipt")
manifest_path = regular_file(manifest_path, f"{label} manifest")
manifest = load_manifest(manifest_path)
snapshot = receipt.get("snapshot") or {}
source = receipt.get("source") or {}
dump = receipt.get("dump") or {}
manifest_receipt = receipt.get("manifest") or {}
source_context = receipt.get("source_context") or {}
source_service = receipt.get("source_service") or {}
binding = receipt.get("provenance_binding") or {}
expected_binding = build_provenance_binding(
run_id=str(receipt.get("run_id") or ""),
authorization_ref=str(receipt.get("capture_authorization_ref") or ""),
source=source,
snapshot=snapshot,
dump_sha256=str(dump.get("sha256") or ""),
manifest_sql_sha256=str(manifest_receipt.get("manifest_sql_sha256") or ""),
source_manifest_sha256=file_sha256(manifest_path),
source_context_sha256=str(source_context.get("sha256") or ""),
)
total_rows = sum(int(row["row_count"]) for row in manifest["tables"].values())
checks = {
"schema": receipt.get("schema") == SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"receipt_version": receipt.get("receipt_version") == 2,
"status": receipt.get("status") == "pass",
"authorization": receipt.get("capture_authorization_ref") == expected_authorization_ref,
"snapshot_exported": receipt.get("snapshot_exported") is True,
"service_unchanged": receipt.get("service_unchanged") is True,
"source_service_healthy": source_service.get("unchanged") is True
and service_healthy(source_service.get("before"))
and source_service.get("before") == source_service.get("after"),
"source_not_mutated": receipt.get("production_db_mutated") is False,
"telegram_not_sent": receipt.get("telegram_send_attempted") is False,
"manifest_hash": manifest_receipt.get("sha256") == file_sha256(manifest_path),
"manifest_table_count": manifest_receipt.get("table_count") == len(manifest["tables"]),
"manifest_total_rows": manifest_receipt.get("total_rows") == total_rows,
"manifest_read_only": manifest["singleton"]["identity"].get("transaction_read_only") == "on",
"manifest_database": manifest["singleton"]["identity"].get("database") == source.get("database"),
"binding_payload": binding.get("payload") == expected_binding["payload"],
"binding_sha256": binding.get("sha256") == expected_binding["sha256"],
"binding_algorithm": binding.get("algorithm") == "sha256",
"dump_sha256": bool(SHA256_RE.fullmatch(str(dump.get("sha256") or ""))),
"manifest_sql_sha256": bool(SHA256_RE.fullmatch(str(manifest_receipt.get("manifest_sql_sha256") or ""))),
"manifest_sql_reviewed": manifest_receipt.get("manifest_sql_sha256") == REVIEWED_MANIFEST_SQL_SHA256,
"source_context_sha256": bool(SHA256_RE.fullmatch(str(source_context.get("sha256") or ""))),
"snapshot_identity": all(
bool(snapshot.get(field))
for field in ("captured_at_utc", "exported_snapshot_id", "txid_snapshot", "wal_lsn", "system_identifier")
),
}
failed = sorted(name for name, passed in checks.items() if not passed)
if failed:
raise ValueError(f"{label} capture failed validation: " + ", ".join(failed))
return receipt, manifest
def manifest_fingerprint(manifest: dict[str, Any]) -> dict[str, Any]:
tables = {
name: {"row_count": int(row["row_count"]), "rowset_md5": row.get("rowset_md5")}
for name, row in sorted(manifest["tables"].items())
}
structures = {
kind: canonical_hash(manifest["singleton"][kind].get("items", []))
for kind in sorted(SINGLETON_KINDS - {"identity"})
}
return {
"table_count": len(tables),
"total_rows": sum(row["row_count"] for row in tables.values()),
"table_fingerprint_sha256": canonical_hash(tables),
"structure_fingerprint_sha256": canonical_hash(structures),
"tables": tables,
"structures": structures,
}
def verify_delta(
*,
baseline_receipt_path: Path,
baseline_manifest_path: Path,
baseline_authorization_ref: str,
current_receipt_path: Path,
current_manifest_path: Path,
current_authorization_ref: str,
) -> dict[str, Any]:
baseline_receipt, baseline_manifest = validate_capture(
baseline_receipt_path,
baseline_manifest_path,
expected_authorization_ref=baseline_authorization_ref,
label="baseline",
)
current_receipt, current_manifest = validate_capture(
current_receipt_path,
current_manifest_path,
expected_authorization_ref=current_authorization_ref,
label="current",
)
baseline_snapshot = baseline_receipt["snapshot"]
current_snapshot = current_receipt["snapshot"]
baseline_time = parse_timestamp(baseline_snapshot["captured_at_utc"], "baseline snapshot time")
current_time = parse_timestamp(current_snapshot["captured_at_utc"], "current snapshot time")
if current_time <= baseline_time:
raise ValueError("current source capture must be newer than the restored baseline capture")
if current_receipt.get("source") != baseline_receipt.get("source"):
raise ValueError("current source identity does not match the restored baseline source")
if current_snapshot.get("system_identifier") != baseline_snapshot.get("system_identifier"):
raise ValueError("current source PostgreSQL system identifier changed")
baseline_manifest_sql_sha256 = str((baseline_receipt.get("manifest") or {}).get("manifest_sql_sha256") or "")
current_manifest_sql_sha256 = str((current_receipt.get("manifest") or {}).get("manifest_sql_sha256") or "")
if baseline_manifest_sql_sha256 != current_manifest_sql_sha256:
raise ValueError("baseline and current captures used different manifest SQL algorithms")
baseline = manifest_fingerprint(baseline_manifest)
current = manifest_fingerprint(current_manifest)
baseline_tables = baseline.pop("tables")
current_tables = current.pop("tables")
baseline_structures = baseline.pop("structures")
current_structures = current.pop("structures")
added_tables = sorted(set(current_tables) - set(baseline_tables))
removed_tables = sorted(set(baseline_tables) - set(current_tables))
changed_tables = [
{"table": table, "baseline": baseline_tables[table], "current": current_tables[table]}
for table in sorted(set(baseline_tables) & set(current_tables))
if baseline_tables[table] != current_tables[table]
]
changed_structures = [
{
"kind": kind,
"baseline_sha256": baseline_structures.get(kind),
"current_sha256": current_structures.get(kind),
}
for kind in sorted(set(baseline_structures) | set(current_structures))
if baseline_structures.get(kind) != current_structures.get(kind)
]
delta_detected = bool(added_tables or removed_tables or changed_tables or changed_structures)
return {
"artifact": "vps_canonical_snapshot_postflight_delta",
"schema": DELTA_SCHEMA,
"generated_at_utc": datetime.now(UTC).isoformat(),
"status": "pass",
"baseline": {
"run_id": baseline_receipt["run_id"],
"capture_authorization_ref": baseline_authorization_ref,
"captured_at_utc": baseline_snapshot["captured_at_utc"],
"capture_receipt_sha256": file_sha256(baseline_receipt_path),
"manifest_sha256": file_sha256(baseline_manifest_path),
"manifest_sql_sha256": baseline_manifest_sql_sha256,
**baseline,
},
"current": {
"run_id": current_receipt["run_id"],
"capture_authorization_ref": current_authorization_ref,
"captured_at_utc": current_snapshot["captured_at_utc"],
"capture_receipt_sha256": file_sha256(current_receipt_path),
"manifest_sha256": file_sha256(current_manifest_path),
"manifest_sql_sha256": current_manifest_sql_sha256,
**current,
},
"delta": {
"delta_detected": delta_detected,
"added_tables": added_tables,
"removed_tables": removed_tables,
"changed_tables": changed_tables,
"changed_structures": changed_structures,
"total_row_delta": current["total_rows"] - baseline["total_rows"],
},
"strongest_claim_allowed": (
"restored snapshot still matches the newest postflight VPS capture"
if not delta_detected
else "restored snapshot parity is bounded to its capture; post-snapshot VPS deltas are explicit"
),
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--baseline-receipt", required=True, type=Path)
parser.add_argument("--baseline-manifest", required=True, type=Path)
parser.add_argument("--baseline-authorization-ref", required=True)
parser.add_argument("--current-receipt", required=True, type=Path)
parser.add_argument("--current-manifest", required=True, type=Path)
parser.add_argument("--current-authorization-ref", required=True)
parser.add_argument("--output", required=True, type=Path)
args = parser.parse_args()
try:
payload = verify_delta(
baseline_receipt_path=args.baseline_receipt,
baseline_manifest_path=args.baseline_manifest,
baseline_authorization_ref=args.baseline_authorization_ref,
current_receipt_path=args.current_receipt,
current_manifest_path=args.current_manifest,
current_authorization_ref=args.current_authorization_ref,
)
except (OSError, ValueError, KeyError, TypeError, json.JSONDecodeError) as exc:
payload = {
"artifact": "vps_canonical_snapshot_postflight_delta",
"schema": DELTA_SCHEMA,
"generated_at_utc": datetime.now(UTC).isoformat(),
"status": "fail",
"error": str(exc),
"strongest_claim_allowed": "post-snapshot VPS delta is not proven",
}
write_private_json(args.output, payload)
print_private_receipt_summary(args.output, payload)
return 0 if payload.get("status") == "pass" else 1
if __name__ == "__main__":
raise SystemExit(main())

7
scripts/run_gcp_generated_db_direct_claim_suite.py
View file

@ -35,11 +35,15 @@ if __name__ == "__main__" and should_reexec_in_hermes(Path(sys.executable)):
)
HERE = Path(__file__).resolve().parent
REPO_ROOT = HERE.parent
sys.path.insert(0, str(REPO_ROOT))
sys.path.insert(0, str(HERE))
import run_leo_clone_bound_handler_checkpoint as bound # noqa: E402
import working_leo_open_ended_benchmark as benchmark # noqa: E402
from ops.verify_postgres_parity_manifest import REVIEWED_MANIFEST_SQL_SHA256 # noqa: E402
SCHEMA = "livingip.gcpGeneratedDbDirectClaimSuite.v1"
SOURCE_COMPUTE = "teleo-prod-1"
DEFAULT_SERVICE = "leoclean-gcp-prod-parallel.service"
@ -49,8 +53,7 @@ DEFAULT_SECRET = "gcp-teleo-pgvector-standby-postgres-password"
DEFAULT_SSL_ROOT_CERT = Path("/usr/local/libexec/livingip/leoclean-kb/cloudsql-server-ca.pem")
DEFAULT_SYSTEM_IDENTIFIER = "7659718422914359312"
DEFAULT_MANIFEST_SQL = HERE.parent / "ops" / "postgres_parity_manifest.sql"
REVIEWED_CLOUDSQL_TOOL_SHA256 = "17d5213756ea6492e4232784a04b7f584afee5dce5184b33a2227e58847472c1"
REVIEWED_MANIFEST_SQL_SHA256 = "8b8cdc25d54fdd8de05eb38c6e4423d2836953eb6012d4545f5c9c71b5f0150a"
REVIEWED_CLOUDSQL_TOOL_SHA256 = "f71d9dd5cfe5dd10ba0b4439a7b9631a0cb824ce6e05b1faa69dc25690f257f5"
COUNT_READBACK_RE = re.compile(
r"DB readback:\s*claims:\s*`?(?P<claims>\d+)`?;\s*"
r"sources:\s*`?(?P<sources>\d+)`?;\s*"

82
scripts/run_gcp_generated_db_working_leo_suite.py
View file

@ -98,9 +98,7 @@ def load_pinned_receipt(path: Path, expected_sha256: str, artifact: str) -> tupl
return payload, actual_sha256
def validate_target_identity_receipt(
args: argparse.Namespace, path: Path, expected_sha256: str
) -> dict[str, Any]:
def validate_target_identity_receipt(args: argparse.Namespace, path: Path, expected_sha256: str) -> dict[str, Any]:
payload, actual_sha256 = load_pinned_receipt(path, expected_sha256, TARGET_IDENTITY_ARTIFACT)
required = {
"target_database": args.target_db,
@ -130,9 +128,7 @@ def validate_target_identity_receipt(
}
def validate_source_baseline_receipt(
args: argparse.Namespace, path: Path, expected_sha256: str
) -> dict[str, Any]:
def validate_source_baseline_receipt(args: argparse.Namespace, path: Path, expected_sha256: str) -> dict[str, Any]:
payload, actual_sha256 = load_pinned_receipt(path, expected_sha256, SOURCE_BASELINE_ARTIFACT)
if payload.get("project") != args.project or payload.get("instance") != args.instance:
raise bound.CheckpointError("source baseline receipt project/instance context does not match")
@ -192,8 +188,7 @@ def composition_check_contract() -> list[str]:
if "source_packet_validated" in keys and "isolated_restart_and_memory_survived" in keys:
if len(keys) != EXPECTED_COMPOSITION_CHECK_COUNT:
raise bound.CheckpointError(
"composition scorer drifted: "
f"expected {EXPECTED_COMPOSITION_CHECK_COUNT} checks, found {len(keys)}"
f"composition scorer drifted: expected {EXPECTED_COMPOSITION_CHECK_COUNT} checks, found {len(keys)}"
)
return keys
raise bound.CheckpointError("existing composition scorer contract could not be located")
@ -334,8 +329,7 @@ def validate_capability_receipt(
"session_user": receipt.get("session_user") == role,
"current_user": receipt.get("current_user") == role,
"current_database": receipt.get("current_database") == args.target_db,
"default_transaction_read_only": receipt.get("default_transaction_read_only")
== ("off" if writable else "on"),
"default_transaction_read_only": receipt.get("default_transaction_read_only") == ("off" if writable else "on"),
"not_superuser": receipt.get("is_superuser") is False,
"can_select_proposals": receipt.get("can_select_proposals") is True,
"cannot_update_proposals": receipt.get("can_update_proposals") is False,
@ -370,12 +364,7 @@ def validate_capability_receipt(
"can_insert_canonical": True,
},
}
checks.update(
{
name: receipt.get(name) is expected
for name, expected in expected_capabilities[purpose].items()
}
)
checks.update({name: receipt.get(name) is expected for name, expected in expected_capabilities[purpose].items()})
failed = sorted(name for name, passed in checks.items() if not passed)
if failed:
raise bound.CheckpointError(f"{purpose} phase capability receipt failed: " + ", ".join(failed))
@ -581,7 +570,9 @@ def verify_distinct_credential_values(executor: CloudSqlExecutor) -> dict[str, A
values.clear()
def _completed(command: list[str], returncode: int, stdout: str = "", stderr: str = "") -> subprocess.CompletedProcess[str]:
def _completed(
command: list[str], returncode: int, stdout: str = "", stderr: str = ""
) -> subprocess.CompletedProcess[str]:
return subprocess.CompletedProcess(command, returncode, stdout, stderr)
@ -652,13 +643,10 @@ def _target_identity(args: argparse.Namespace, db_identity: dict[str, Any]) -> d
}
def assert_live_target_identity(
args: argparse.Namespace, retained: dict[str, Any], live: dict[str, Any]
) -> None:
def assert_live_target_identity(args: argparse.Namespace, retained: dict[str, Any], live: dict[str, Any]) -> None:
checks = {
"target_database": live.get("current_database") == retained.get("target_database") == args.target_db,
"system_identifier": str(live.get("system_identifier") or "")
== str(retained.get("system_identifier") or ""),
"system_identifier": str(live.get("system_identifier") or "") == str(retained.get("system_identifier") or ""),
"server_address": str(live.get("server_address") or "")
== str(retained.get("server_address") or "")
== args.host,
@ -751,7 +739,7 @@ def _cloudsql_lifecycle_wrapper(
tool_log=tool_log,
run_nonce=run_nonce,
)
marker = "if ! PGPASSWORD=\"$(gcloud secrets versions access latest"
marker = 'if ! PGPASSWORD="$(gcloud secrets versions access latest'
if marker not in base:
raise bound.CheckpointError("reviewed Cloud SQL wrapper shape changed; lifecycle injection refused")
internal = shlex.join(
@ -882,12 +870,16 @@ class RuntimeAdapter:
self._assert_target(container, database)
return self.executor.read(sql)
def _approve(self, _guard_args: argparse.Namespace, proposal_id: str, database: str, *, dry_run: bool = False) -> subprocess.CompletedProcess[str]:
def _approve(
self, _guard_args: argparse.Namespace, proposal_id: str, database: str, *, dry_run: bool = False
) -> subprocess.CompletedProcess[str]:
if database != self.args.target_db:
raise bound.CheckpointError("review attempted database fallback")
return _review_command(self.executor, proposal_id, dry_run=dry_run)
def _apply(self, _guard_args: argparse.Namespace, proposal_id: str, database: str, *, dry_run: bool = False) -> subprocess.CompletedProcess[str]:
def _apply(
self, _guard_args: argparse.Namespace, proposal_id: str, database: str, *, dry_run: bool = False
) -> subprocess.CompletedProcess[str]:
if database != self.args.target_db:
raise bound.CheckpointError("apply attempted database fallback")
return _apply_command(self.executor, proposal_id, dry_run=dry_run)
@ -935,9 +927,7 @@ class RuntimeAdapter:
self._restore()
raise
def _assert_disposable_target(
self, container: str, database: str
) -> tuple[dict[str, Any], dict[str, Any]]:
def _assert_disposable_target(self, container: str, database: str) -> tuple[dict[str, Any], dict[str, Any]]:
self._assert_target(container, database)
target = self._container_identity(container)
source = self._container_identity(bound.PRODUCTION_CONTAINER)
@ -1201,9 +1191,7 @@ def catalog_session_isolation(results: list[dict[str, Any]]) -> dict[str, bool]:
dc = [result for result in results if str(result.get("prompt_id") or "").startswith("DC-")]
session_ids = [result.get("persisted_session_id") for result in dc]
return {
"all_profiles_unique": bool(profile_ids)
and all(profile_ids)
and len(profile_ids) == len(set(profile_ids)),
"all_profiles_unique": bool(profile_ids) and all(profile_ids) and len(profile_ids) == len(set(profile_ids)),
"dc_isolated": len(dc) == len(benchmark.M3TAVERSAL_DIRECT_CLAIM_FOLLOWUP_SCENARIOS)
and all(result.get("fresh_private_profile") is True and result.get("prior_prompt_ids") == [] for result in dc)
and all(session_ids)
@ -1448,9 +1436,7 @@ async def run_live_suite(args: argparse.Namespace, validated: dict[str, Any]) ->
report[name] = capture()
except Exception as exc:
report[name] = None
report["errors"].append(
{"phase": name, "type": type(exc).__name__, "message": safe_message(exc)}
)
report["errors"].append({"phase": name, "type": type(exc).__name__, "message": safe_message(exc)})
report["database_fingerprint_after"] = (
database_fingerprint_from_manifest(report["database_manifest_after"])
if report.get("database_manifest_after")
@ -1509,12 +1495,10 @@ async def run_live_suite(args: argparse.Namespace, validated: dict[str, Any]) ->
"source_baseline_is_independently_retained_and_physically_distinct": bool(source_baseline.get("sha256"))
and source_identity.get("system_identifier")
and str(source_identity.get("system_identifier")) != str(before_identity.get("system_identifier")),
"proposal_lifecycle_exact_under_gcp_supported_checks": report["lifecycle_gcp_evaluation"]["passed"]
is True,
"proposal_lifecycle_exact_under_gcp_supported_checks": report["lifecycle_gcp_evaluation"]["passed"] is True,
"process_memory_survived": (composition_checks.get("isolated_restart_and_memory_survived") is True)
and (lifecycle_checks.get("isolated_handler_reopened_with_same_session") is True),
"database_identity_stable": bool(before_identity)
and before_identity == report.get("database_identity_after"),
"database_identity_stable": bool(before_identity) and before_identity == report.get("database_identity_after"),
"physical_target_matches_sha_pinned_receipt": bool(target_identity.get("sha256"))
and str(before_identity.get("system_identifier")) == target_identity.get("system_identifier"),
"service_unchanged": before_service == report.get("service_after"),
@ -1525,12 +1509,8 @@ async def run_live_suite(args: argparse.Namespace, validated: dict[str, Any]) ->
)
is True
and report["object_level_changes"].get("lifecycle_unrelated_rows_unchanged") is True
and all(
item.get("changed") is True
for item in (report["object_level_changes"].get("objects") or {}).values()
),
"phase_roles_and_capabilities_verified": {"read", "stage", "review", "apply"}
<= operation_purposes
and all(item.get("changed") is True for item in (report["object_level_changes"].get("objects") or {}).values()),
"phase_roles_and_capabilities_verified": {"read", "stage", "review", "apply"} <= operation_purposes
and bool(executor.receipts)
and all(
(
@ -1538,18 +1518,12 @@ async def run_live_suite(args: argparse.Namespace, validated: dict[str, Any]) ->
and receipt["purpose"] in ALLOWED_WRITE_PURPOSES
and receipt["default_transaction_read_only"] == "off"
)
or (
receipt["writable"] is False
and receipt["default_transaction_read_only"] == "on"
)
or (receipt["writable"] is False and receipt["default_transaction_read_only"] == "on")
for receipt in executor.receipts
),
"credential_values_verified_distinct_without_retention": report["credential_value_separation"]
== {"checked": True, "distinct": True, "credential_count": 4},
"no_send_proven_by_component_tool_surfaces": catalog_checks.get(
"send_and_delivery_absent_per_prompt"
)
is True
"no_send_proven_by_component_tool_surfaces": catalog_checks.get("send_and_delivery_absent_per_prompt") is True
and composition_checks.get("no_telegram_send") is True
and lifecycle_checks.get("no_telegram_send") is True,
"no_systemd_restart": report["systemd_restart_attempted"] is False,
@ -1743,9 +1717,7 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
args.instance,
)
if not all(receipt_inputs):
parser.error(
"execution requires --instance plus SHA-pinned target identity and source baseline receipts"
)
parser.error("execution requires --instance plus SHA-pinned target identity and source baseline receipts")
missing_flags = [name for name in ("stage", "review", "apply") if not getattr(args, f"allow_{name}")]
if missing_flags:
parser.error("execution requires explicit controller flags: " + ", ".join(missing_flags))

130
tests/test_attest_gcp_reasoning_compute.py Normal file
View file

@ -0,0 +1,130 @@
import json
from pathlib import Path
import pytest
from ops import attest_gcp_reasoning_compute as attestation
def reasoning_receipt(path: Path) -> Path:
path.write_text(
json.dumps(
{
"schema": attestation.REASONING_SCHEMA,
"status": "pass",
"generated_at_utc": "2026-07-15T01:00:00+00:00",
"completed_at_utc": "2026-07-15T01:01:00+00:00",
"target_database": "teleo_clone_current_test",
"source_compute": "teleo-prod-1",
"errors": [],
"posted_to_telegram": False,
"database_write_attempted": False,
},
sort_keys=True,
)
+ "\n",
encoding="utf-8",
)
return path
def compute_identity() -> dict:
return {
"project": "teleo-501523",
"project_number": "785938879453",
"instance": "teleo-prod-1",
"instance_id": "123456789",
"zone": "europe-west6-a",
"zone_resource": "projects/123456789/zones/europe-west6-a",
"network_resource": "projects/123456789/networks/teleo-staging-net",
"private_ip": "10.61.0.2",
"service_account": attestation.DEFAULT_RESTORE_SERVICE_ACCOUNT,
"expected_private_network": "projects/teleo-501523/global/networks/teleo-staging-net",
"checks": {
"project": True,
"project_number": True,
"expected_network_project": True,
"instance": True,
"instance_id": True,
"zone": True,
"network": True,
"private_ip": True,
"service_account": True,
},
}
def test_attestation_binds_reasoning_hash_to_live_metadata(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
receipt_path = reasoning_receipt(tmp_path / "reasoning.json")
observed = {}
def fake_identity(project, instance, zone, network, service_account, *, timeout):
observed.update(
{
"project": project,
"instance": instance,
"zone": zone,
"network": network,
"service_account": service_account,
"timeout": timeout,
}
)
return compute_identity()
monkeypatch.setattr(attestation, "gce_compute_identity", fake_identity)
result = attestation.attest_reasoning_compute(
reasoning_receipt_path=receipt_path,
restore_run_id="gcp-restore-current-test123",
target_database="teleo_clone_current_test",
generated_at_utc="2026-07-15T01:01:05+00:00",
)
assert result["status"] == "pass"
assert result["checks"]["gce_metadata_identity"] is True
assert result["reasoning_receipt"]["sha256"] == attestation.sha256_file(receipt_path)
assert result["compute"]["instance_id"] == "123456789"
assert observed == {
"project": "teleo-501523",
"instance": "teleo-prod-1",
"zone": "europe-west6-a",
"network": "projects/teleo-501523/global/networks/teleo-staging-net",
"service_account": attestation.DEFAULT_RESTORE_SERVICE_ACCOUNT,
"timeout": 10.0,
}
def test_attestation_rejects_claimed_wrong_compute_before_metadata(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
receipt_path = reasoning_receipt(tmp_path / "reasoning.json")
payload = json.loads(receipt_path.read_text())
payload["source_compute"] = "untrusted-vm"
receipt_path.write_text(json.dumps(payload) + "\n", encoding="utf-8")
monkeypatch.setattr(
attestation,
"gce_compute_identity",
lambda *args, **kwargs: pytest.fail("metadata must not be queried"),
)
with pytest.raises(ValueError, match="source_compute"):
attestation.attest_reasoning_compute(
reasoning_receipt_path=receipt_path,
restore_run_id="gcp-restore-current-test123",
target_database="teleo_clone_current_test",
)
def test_attestation_rejects_send_or_write_receipt(tmp_path: Path) -> None:
receipt_path = reasoning_receipt(tmp_path / "reasoning.json")
payload = json.loads(receipt_path.read_text())
payload["posted_to_telegram"] = True
payload["database_write_attempted"] = True
receipt_path.write_text(json.dumps(payload) + "\n", encoding="utf-8")
with pytest.raises(ValueError, match=r"no_database_write.*no_telegram_send"):
attestation.attest_reasoning_compute(
reasoning_receipt_path=receipt_path,
restore_run_id="gcp-restore-current-test123",
target_database="teleo_clone_current_test",
)

142
tests/test_capture_vps_canonical_postgres_snapshot.py
View file

@ -1,8 +1,18 @@
import json
from pathlib import Path
import pytest
from ops.capture_vps_canonical_postgres_snapshot import (
SAFE_AUTHORIZATION_REF_RE,
SAFE_SSH_TARGET_RE,
SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
build_provenance_binding,
canonical_sha256,
cleanup_failed_output,
load_service_state,
load_snapshot_metadata,
parse_args,
remote_capture_script,
toc_counts,
)
@ -21,8 +31,14 @@ def test_remote_capture_uses_one_exported_read_only_snapshot() -> None:
assert "[0-9A-F]+-[0-9A-F]+-[0-9]+" in script
assert '--snapshot="$snapshot_id"' in script
assert '-v "snapshot_id=$snapshot_id"' in script
assert "txid_current_snapshot()::text" in script
assert "pg_current_wal_lsn()::text" in script
assert "system_identifier::text from pg_control_system()" in script
assert "source-snapshot.json" in script
assert "--no-owner --no-acl" in script
assert "cmp -s" in script
assert 'assert_service_healthy "$remote_root/service-before.txt"' in script
assert 'assert_service_healthy "$remote_root/service-after.txt"' in script
assert "systemctl restart" not in script
assert "docker restart" not in script
@ -32,6 +48,11 @@ def test_manifest_hash_order_is_collation_independent() -> None:
assert 'collate "C"' in manifest
assert "'server_address', host(inet_server_addr())" in manifest
assert "'kb_gate_owner', 'kb_apply', 'kb_review'" in manifest
assert "'kind', 'role_memberships'" in manifest
assert "'kind', 'object_ownership'" in manifest
assert "'kind', 'object_acl'" in manifest
assert "aclexplode(attribute.attacl)" in manifest
def test_toc_counts_database_objects(tmp_path: Path) -> None:
@ -62,8 +83,129 @@ def test_failure_cleanup_never_removes_preexisting_output(tmp_path: Path) -> Non
assert not created_by_run.exists()
def test_source_service_state_requires_healthy_unchanged_capable_fields(tmp_path: Path) -> None:
state_path = tmp_path / "service.txt"
state_path.write_text("ActiveState=active\nSubState=running\nMainPID=347406\nNRestarts=0\n")
assert load_service_state(state_path)["MainPID"] == "347406"
state_path.write_text("ActiveState=failed\nSubState=dead\nMainPID=0\nNRestarts=0\n")
with pytest.raises(RuntimeError, match="not active/running"):
load_service_state(state_path)
def test_ssh_target_rejects_open_ssh_option_injection() -> None:
assert SAFE_SSH_TARGET_RE.fullmatch("root@77.42.65.182")
assert SAFE_SSH_TARGET_RE.fullmatch("teleo-staging-1.internal")
assert not SAFE_SSH_TARGET_RE.fullmatch("-oProxyCommand=malicious")
assert not SAFE_SSH_TARGET_RE.fullmatch("root@host command")
def test_source_snapshot_metadata_and_provenance_binding(tmp_path: Path) -> None:
snapshot_path = tmp_path / "source-snapshot.json"
snapshot_path.write_text(
json.dumps(
{
"exported_snapshot_id": "00000003-000001A2-1",
"txid_snapshot": "100:120:105,111",
"wal_lsn": "0/16B6C50",
"system_identifier": "7649789040005668902",
"captured_at_utc": "2026-07-15T01:02:03.456789+00:00",
}
)
)
snapshot = load_snapshot_metadata(snapshot_path)
binding = build_provenance_binding(
run_id="canonical-20260715",
authorization_ref="codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621",
source={
"ssh_target": "root@77.42.65.182",
"container": "teleo-pg",
"database": "teleo",
"service": "leoclean-gateway.service",
},
snapshot=snapshot,
dump_sha256="1" * 64,
manifest_sql_sha256="2" * 64,
source_manifest_sha256="3" * 64,
source_context_sha256="4" * 64,
)
assert binding["algorithm"] == "sha256"
assert binding["payload"] == {
"receipt_schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"run_id": "canonical-20260715",
"capture_authorization_ref": "codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621",
"source": {
"ssh_target": "root@77.42.65.182",
"container": "teleo-pg",
"database": "teleo",
"service": "leoclean-gateway.service",
},
"exported_snapshot_id": "00000003-000001A2-1",
"txid_snapshot": "100:120:105,111",
"wal_lsn": "0/16B6C50",
"source_system_identifier": "7649789040005668902",
"dump_sha256": "1" * 64,
"manifest_sql_sha256": "2" * 64,
"source_manifest_sha256": "3" * 64,
"source_context_sha256": "4" * 64,
}
assert binding["sha256"] == canonical_sha256(binding["payload"])
@pytest.mark.parametrize(
("field", "value"),
[
("exported_snapshot_id", "unsafe snapshot"),
("txid_snapshot", "100:120:not-a-txid"),
("wal_lsn", "not-an-lsn"),
("system_identifier", "system-1"),
],
)
def test_source_snapshot_metadata_rejects_invalid_identity(tmp_path: Path, field: str, value: str) -> None:
payload = {
"exported_snapshot_id": "00000003-000001A2-1",
"txid_snapshot": "100:120:",
"wal_lsn": "0/16B6C50",
"system_identifier": "7649789040005668902",
"captured_at_utc": "2026-07-15T01:02:03+00:00",
}
payload[field] = value
path = tmp_path / "source-snapshot.json"
path.write_text(json.dumps(payload))
with pytest.raises(RuntimeError, match=field):
load_snapshot_metadata(path)
def test_authorization_ref_is_required_and_safe_metadata(tmp_path: Path) -> None:
ssh_key = tmp_path / "id_ed25519"
ssh_key.write_text("test-only")
manifest = tmp_path / "manifest.sql"
manifest.write_text("select 1;\n")
base = [
"--execute",
"--ssh-target",
"root@77.42.65.182",
"--ssh-key",
str(ssh_key),
"--manifest",
str(manifest),
"--run-id",
"canonical-20260715",
"--output-dir",
str(tmp_path / "capture"),
]
with pytest.raises(SystemExit):
parse_args(base)
with pytest.raises(SystemExit):
parse_args([*base, "--authorization-ref", "unsafe authorization ref"])
authorization_ref = "codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621"
args = parse_args([*base, "--authorization-ref", authorization_ref])
assert args.authorization_ref == authorization_ref
assert SAFE_AUTHORIZATION_REF_RE.fullmatch(authorization_ref)
assert not SAFE_AUTHORIZATION_REF_RE.fullmatch("authorization\nforged")

4
tests/test_gcp_generated_db_direct_claim_suite.py
View file

@ -166,7 +166,7 @@ def test_manifest_normalization_excludes_nondeterministic_timing() -> None:
"BEGIN",
'{"kind":"identity","database":"teleo_clone_test","captured_at":"2026-07-12T00:00:00Z"}',
'{"kind":"schemas","items":["kb_stage","public"]}',
'{"kind":"table","schema":"public","table":"claims","row_count":2,"rowset_md5":"abc"}',
'{"kind":"table","schema":"public","table":"claims","row_count":2,"rowset_md5":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}',
'{"kind":"performance","query":"count_claims","elapsed_ms":1.2,"observed_rows":2}',
"ROLLBACK",
]
@ -196,7 +196,7 @@ def test_database_fingerprint_uses_full_normalized_manifest(monkeypatch: pytest.
'{"kind":"identity","database":"teleo_clone_test","server_address":"10.61.0.3",'
'"ssl":true,"transaction_read_only":"on","captured_at":"one"}',
'{"kind":"schemas","items":["kb_stage","public"]}',
'{"kind":"table","schema":"public","table":"claims","row_count":2,"rowset_md5":"abc"}',
'{"kind":"table","schema":"public","table":"claims","row_count":2,"rowset_md5":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}',
]
)
monkeypatch.setattr("scripts.run_gcp_generated_db_direct_claim_suite.database_password", lambda _args: "secret")

26
tests/test_gcp_generated_db_working_leo_suite.py
View file

@ -95,9 +95,7 @@ def source_baseline_payload() -> dict[str, object]:
}
def capability_receipt(
role: str, target_db: str = "teleo_clone_working_leo", *, writable: bool | None = None
) -> str:
def capability_receipt(role: str, target_db: str = "teleo_clone_working_leo", *, writable: bool | None = None) -> str:
capabilities = {
"kb_read": (False, False, False, False, False),
"kb_stage_writer": (True, False, False, False, False),
@ -337,14 +335,18 @@ def test_source_snapshots_are_receipt_backed_and_inherited_source_checks_are_not
}
adapter = suite.RuntimeAdapter(args, suite.CloudSqlExecutor(args), source, {"system_identifier": "target"})
assert adapter._guard_snapshot(
suite.bound.PRODUCTION_CONTAINER,
suite.bound.PRODUCTION_DB,
tables=suite.composition.PRODUCTION_TABLES,
) == source_payload["guard_snapshot"]
assert adapter._gate_schema(suite.bound.PRODUCTION_CONTAINER, suite.bound.PRODUCTION_DB) == source_payload[
"gate_schema"
]
assert (
adapter._guard_snapshot(
suite.bound.PRODUCTION_CONTAINER,
suite.bound.PRODUCTION_DB,
tables=suite.composition.PRODUCTION_TABLES,
)
== source_payload["guard_snapshot"]
)
assert (
adapter._gate_schema(suite.bound.PRODUCTION_CONTAINER, suite.bound.PRODUCTION_DB)
== source_payload["gate_schema"]
)
contract = suite.composition_check_contract()
required = [name for name in contract if name not in suite.UNSUPPORTED_INHERITED_COMPOSITION_CHECKS]
assert len(contract) == 34
@ -524,7 +526,7 @@ def test_lifecycle_wrapper_exposes_only_read_or_exact_stage_and_never_delivery(t
assert "accepts no model-supplied arguments" in wrapper
assert "--stage-password-secret" in wrapper
assert "--read-role" in wrapper
assert 'user=$READ_ROLE' in wrapper
assert "user=$READ_ROLE" in wrapper
assert "default_transaction_read_only=on" in wrapper
assert "systemctl restart" not in wrapper
assert "send_message" not in wrapper

19
tests/test_gcp_leoclean_runtime_permissions.py
View file

@ -167,9 +167,7 @@ class FakeRunner:
stage_definition.update(self.stage_definition_override)
return verifier.CommandResult(0, json.dumps(stage_definition), f"ignored {SECRET_VALUE}")
if sql == verifier._catalog_privilege_posture_sql():
catalog_posture: dict[str, object] = {
field: 0 for field in verifier.CATALOG_ZERO_COUNT_FIELDS
}
catalog_posture: dict[str, object] = {field: 0 for field in verifier.CATALOG_ZERO_COUNT_FIELDS}
catalog_posture.update(
{
"leak": SECRET_VALUE,
@ -476,10 +474,7 @@ def _drifted_value(expected: object) -> object:
@pytest.mark.parametrize(
("field", "unsafe_value"),
[
(field, _drifted_value(expected))
for field, expected in verifier.STAGE_FUNCTION_METADATA_EXPECTATIONS
],
[(field, _drifted_value(expected)) for field, expected in verifier.STAGE_FUNCTION_METADATA_EXPECTATIONS],
)
def test_any_stage_function_metadata_drift_fails_closed(field: str, unsafe_value: object) -> None:
runner = FakeRunner(stage_definition_override={field: unsafe_value})
@ -550,9 +545,9 @@ def test_stage_body_normalization_accepts_only_line_ending_and_framing_newline_c
def test_embedded_stage_body_attestation_is_tied_to_the_provisioning_sql_source() -> None:
provisioning_sql = (
Path(__file__).resolve().parents[1] / "ops" / "gcp_leoclean_runtime_role.sql"
).read_text(encoding="utf-8")
provisioning_sql = (Path(__file__).resolve().parents[1] / "ops" / "gcp_leoclean_runtime_role.sql").read_text(
encoding="utf-8"
)
function_anchor = "create or replace function kb_stage.stage_leoclean_proposal("
assert provisioning_sql.count(function_anchor) == 1
function_section = provisioning_sql.split(function_anchor, 1)[1]
@ -644,9 +639,7 @@ def test_legacy_and_template_database_connect_are_both_behaviorally_denied() ->
"result": "denied",
}
database_calls = [
" ".join(command)
for command, _env, _timeout, _discard in runner.calls
if command[0] == verifier.PSQL_BIN
" ".join(command) for command, _env, _timeout, _discard in runner.calls if command[0] == verifier.PSQL_BIN
]
assert any(f"dbname={verifier.LEGACY_DATABASE}" in command for command in database_calls)
assert any(f"dbname={verifier.TEMPLATE_DATABASE}" in command for command in database_calls)

7
tests/test_hermes_leoclean_kb_bridge_source.py
View file

@ -324,7 +324,8 @@ def _configure_runtime_environment(monkeypatch: pytest.MonkeyPatch) -> None:
}
or normalized_key == "PROXY"
or normalized_key.endswith("_PROXY")
or normalized_key in {
or normalized_key
in {
"CURL_CA_BUNDLE",
"PYTHONHTTPSVERIFY",
"REQUESTS_CA_BUNDLE",
@ -679,7 +680,9 @@ def test_cloudsql_runtime_metadata_and_secret_requests_are_exact_and_silent(
def test_cloudsql_runtime_http_opener_disables_proxies_and_redirects() -> None:
module = _load_module(BRIDGE_DIR / "cloudsql_memory_tool.py")
proxy_handlers = [
handler for handler in module.RUNTIME_HTTP_OPENER.handlers if isinstance(handler, module.urllib.request.ProxyHandler)
handler
for handler in module.RUNTIME_HTTP_OPENER.handlers
if isinstance(handler, module.urllib.request.ProxyHandler)
]
redirect_handlers = [
handler

383
tests/test_observatory_read_adapter_gcp_plan.py Normal file
View file

@ -0,0 +1,383 @@
from __future__ import annotations
import json
import shutil
import subprocess
from pathlib import Path
import pytest
import yaml
from ops import check_observatory_read_adapter_gcp_preflight as preflight
ROOT = Path(__file__).resolve().parents[1]
def run_plan(*args: str) -> str:
return subprocess.check_output(
["python3", "ops/plan_observatory_read_adapter_gcp.py", *args],
cwd=ROOT,
text=True,
)
def test_plan_splits_build_deploy_and_runtime_identities() -> None:
plan = json.loads(run_plan())
assert plan["current_tier"] == "T2_unexecuted_least_privilege_packet"
assert plan["identities"] == {
"build": "sa-artifact-builder@teleo-501523.iam.gserviceaccount.com",
"deploy_and_canary": "sa-observatory-deployer@teleo-501523.iam.gserviceaccount.com",
"runtime": "sa-observatory-read-adapter@teleo-501523.iam.gserviceaccount.com",
"database_principal": "sa-observatory-read-adapter@teleo-501523.iam",
}
commands = "\n".join(plan["commands"])
assert "roles/iam.workloadIdentityUser" in commands
assert "roles/iam.serviceAccountUser" in commands
assert "roles/cloudsql.client" in commands
assert "roles/cloudsql.instanceUser" in commands
assert "roles/artifactregistry.reader" in commands
assert "roles/secretmanager.secretAccessor" in commands
assert "subject/repo:living-ip/teleo-infrastructure:ref:refs/heads/main" in plan["github_wif_member"]
assert plan["conditions"]["deployer_wif"].endswith(
"assertion.workflow_ref=='living-ip/teleo-infrastructure/.github/workflows/"
"gcp-observatory-read-adapter.yml@refs/heads/main'"
)
assert plan["github_wif_provider"].endswith("/providers/observatory-read-adapter-main")
assert "roles/run.serviceAgent" in commands
assert "teleo-pgvector-standby" in plan["conditions"]["cloud_sql"]
assert set(plan["deploy_custom_role"]["permissions"]) == {
"run.operations.get",
"run.services.create",
"run.services.get",
"run.services.setIamPolicy",
"run.services.update",
}
assert "roles/run.admin" not in commands
assert "roles/run.developer" not in commands
assert "sa-artifact-builder" not in commands
assert not set(plan["forbidden_deployer_roles"]) & set(commands.split())
assert plan["production_repoint_executed"] is False
def test_shell_packet_is_unexecuted_and_preserves_private_boundaries() -> None:
shell = run_plan("--format", "shell")
assert shell.startswith("# Unexecuted staging packet")
assert "set -euo pipefail" in shell
assert "run.googleapis.com" in shell
assert "--type CLOUD_IAM_SERVICE_ACCOUNT" in shell
assert "openssl rand -hex 32 | tr -d '\\n'" in shell
assert "--data-file=-" in shell
assert "grep -Fxq sa-observatory-read-adapter@teleo-501523.iam" in shell
assert "users create sa-observatory-read-adapter@teleo-501523.iam" in shell
assert "add-iam-policy-binding teleo" in shell
assert "roles/artifactregistry.writer" not in shell
assert "roles/cloudsql.admin" not in shell
assert "Vercel" not in shell
def test_workflow_requires_dedicated_preflight_before_staging_deploy() -> None:
workflow_path = ROOT / ".github" / "workflows" / "gcp-observatory-read-adapter.yml"
workflow = yaml.safe_load(workflow_path.read_text(encoding="utf-8"))
text = workflow_path.read_text(encoding="utf-8")
assert workflow["env"]["BUILD_SERVICE_ACCOUNT"].startswith("sa-artifact-builder@")
assert workflow["env"]["DEPLOY_SERVICE_ACCOUNT"].startswith("sa-observatory-deployer@")
assert workflow["env"]["DEPLOY_WORKLOAD_IDENTITY_PROVIDER"].endswith(
"/providers/observatory-read-adapter-main"
)
assert workflow["jobs"]["deploy-staging"]["needs"] == ["build", "preflight-staging"]
assert "action=preflight_staging" not in text
assert "inputs.action != 'build_only'" in text
assert "EXPECTED_WORKFLOW_REF" in text
assert "${GITHUB_SHA}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" in text
assert "docker images describe" in text
assert "2>/dev/null || true" not in text
assert "--no-invoker-iam-check" in text
assert "--allow-unauthenticated" not in text
assert "API_KEY_VERSION" in text
assert "EXPECTED_IMAGE_REF" in text
assert "check_observatory_read_adapter_gcp_preflight.py" in text
assert 'service_account: ${{ env.BUILD_SERVICE_ACCOUNT }}' in text
assert text.count('service_account: ${{ env.DEPLOY_SERVICE_ACCOUNT }}') == 2
assert "positive-redacted.json" in text
assert "redact_observatory_read_adapter_receipt.jq" in text
assert "database_principal == $db_iam_user" in text
assert "canonical_state_unchanged_after_post: true" in text
assert "cmp <(jq -S . positive.json) <(jq -S . after-write-attempt.json)" in text
assert "rm -f positive.json" in text
def test_preflight_private_cloud_sql_and_secret_validators() -> None:
instance = {
"name": "teleo-pgvector-standby",
"region": "europe-west6",
"databaseVersion": "POSTGRES_16",
"settings": {
"databaseFlags": [{"name": "cloudsql.iam_authentication", "value": "on"}],
"ipConfiguration": {
"ipv4Enabled": False,
"privateNetwork": "projects/teleo-501523/global/networks/teleo-staging-net",
},
},
"ipAddresses": [{"type": "PRIVATE", "ipAddress": "10.61.0.3"}],
}
assert preflight.cloud_sql_is_private_and_iam_enabled(json.dumps(instance)) is True
instance["settings"]["ipConfiguration"]["ipv4Enabled"] = True
instance["ipAddresses"].append({"type": "PRIMARY", "ipAddress": "203.0.113.1"})
assert preflight.cloud_sql_is_private_and_iam_enabled(json.dumps(instance)) is False
assert preflight.cloud_sql_iam_user_exists(
json.dumps(
[
{
"name": "sa-observatory-read-adapter@teleo-501523.iam",
"type": "CLOUD_IAM_SERVICE_ACCOUNT",
}
]
)
)
assert preflight.subnet_is_exact(
json.dumps(
{
"name": "teleo-staging-europe-west6",
"region": "https://www.googleapis.com/compute/v1/projects/teleo-501523/regions/europe-west6",
"network": "https://www.googleapis.com/compute/v1/projects/teleo-501523/global/networks/teleo-staging-net",
}
)
)
assert preflight.secret_is_valid("a" * 32) is True
assert preflight.secret_is_valid("too-short") is False
assert preflight.secret_is_valid("a" * 32 + "\n") is False
assert preflight.secret_is_valid(" " + "a" * 32) is False
assert "gha-creds" not in preflight.sanitize_error("/tmp/work/gha-creds-deadbeef.json denied")
def test_preflight_end_to_end_snapshot_retains_no_secret(monkeypatch) -> None:
instance = {
"name": "teleo-pgvector-standby",
"region": "europe-west6",
"databaseVersion": "POSTGRES_16",
"settings": {
"databaseFlags": [{"name": "cloudsql.iam_authentication", "value": "on"}],
"ipConfiguration": {
"ipv4Enabled": False,
"privateNetwork": "projects/teleo-501523/global/networks/teleo-staging-net",
},
},
"ipAddresses": [{"type": "PRIVATE", "ipAddress": "10.61.0.3"}],
}
secret_value = "never-retain-this-secret-value-000000000000"
deployer = "serviceAccount:sa-observatory-deployer@teleo-501523.iam.gserviceaccount.com"
runtime = "serviceAccount:sa-observatory-read-adapter@teleo-501523.iam.gserviceaccount.com"
project_policy = {
"bindings": [
{"role": preflight.PREFLIGHT_ROLE, "members": [deployer]},
{"role": preflight.DEPLOY_ROLE, "members": [deployer]},
{
"role": "roles/cloudsql.client",
"members": [runtime],
"condition": {
"title": "observatory-exact-cloudsql-instance",
"expression": (
"resource.name == 'projects/teleo-501523/instances/teleo-pgvector-standby' "
"&& resource.service == 'sqladmin.googleapis.com'"
),
},
},
{
"role": "roles/cloudsql.instanceUser",
"members": [runtime],
"condition": {
"title": "observatory-exact-cloudsql-instance",
"expression": (
"resource.name == 'projects/teleo-501523/instances/teleo-pgvector-standby' "
"&& resource.service == 'sqladmin.googleapis.com'"
),
},
},
{
"role": "roles/run.serviceAgent",
"members": [
"serviceAccount:service-785938879453@serverless-robot-prod.iam.gserviceaccount.com"
],
},
]
}
deployer_policy = {
"bindings": [
{"role": "roles/iam.workloadIdentityUser", "members": [preflight.DEPLOY_WIF_MEMBER]}
]
}
runtime_policy = {
"bindings": [{"role": "roles/iam.serviceAccountUser", "members": [deployer]}]
}
artifact_policy = {
"bindings": [{"role": "roles/artifactregistry.reader", "members": [deployer]}]
}
secret_policy = {
"bindings": [
{"role": "roles/secretmanager.secretAccessor", "members": [deployer, runtime]},
{"role": "roles/secretmanager.viewer", "members": [deployer]},
]
}
provider = {
"state": "ACTIVE",
"attributeCondition": preflight.DEPLOY_WIF_ATTRIBUTE_CONDITION,
"attributeMapping": {
"google.subject": "assertion.sub",
"attribute.repository": "assertion.repository",
"attribute.ref": "assertion.ref",
"attribute.workflow_ref": "assertion.workflow_ref",
},
}
def fake_run(command: list[str]) -> subprocess.CompletedProcess[str]:
joined = " ".join(command)
stdout = ""
stderr = ""
returncode = 0
if "auth list" in joined:
stdout = "sa-observatory-deployer@teleo-501523.iam.gserviceaccount.com\n"
elif "projects describe" in joined:
stdout = "teleo-501523\n"
elif command[:3] == ["gcloud", "services", "describe"]:
stdout = "ENABLED\n"
elif "workload-identity-pools providers describe" in joined:
stdout = json.dumps(provider)
elif "iam roles describe observatoryStagingPreflight" in joined:
stdout = json.dumps({"includedPermissions": sorted(preflight.PREFLIGHT_PERMISSIONS)})
elif "iam roles describe observatoryStagingDeployer" in joined:
stdout = json.dumps({"includedPermissions": sorted(preflight.DEPLOY_PERMISSIONS)})
elif "projects get-iam-policy" in joined:
stdout = json.dumps(project_policy)
elif f"service-accounts get-iam-policy {preflight.DEPLOY_SERVICE_ACCOUNT}" in joined:
stdout = json.dumps(deployer_policy)
elif f"service-accounts get-iam-policy {preflight.RUNTIME_SERVICE_ACCOUNT}" in joined:
stdout = json.dumps(runtime_policy)
elif "artifacts repositories get-iam-policy" in joined:
stdout = json.dumps(artifact_policy)
elif "secrets get-iam-policy" in joined:
stdout = json.dumps(secret_policy)
elif "run services describe" in joined:
returncode = 1
stderr = "NOT_FOUND: service not found"
elif "service-accounts describe" in joined:
stdout = "sa-observatory-read-adapter@teleo-501523.iam.gserviceaccount.com\n"
elif "networks subnets describe" in joined:
stdout = json.dumps(
{
"name": "teleo-staging-europe-west6",
"region": "https://www.googleapis.com/compute/v1/projects/teleo-501523/regions/europe-west6",
"network": "https://www.googleapis.com/compute/v1/projects/teleo-501523/global/networks/teleo-staging-net",
}
)
elif "networks describe" in joined:
stdout = "teleo-staging-net\n"
elif "sql instances describe" in joined:
stdout = json.dumps(instance)
elif "sql users list" in joined:
stdout = json.dumps(
[
{
"name": "sa-observatory-read-adapter@teleo-501523.iam",
"type": "CLOUD_IAM_SERVICE_ACCOUNT",
}
]
)
elif "secrets versions list" in joined:
stdout = "7\n"
elif "secrets versions access" in joined:
stdout = secret_value
elif "artifacts docker images describe" in joined:
stdout = "{}"
return subprocess.CompletedProcess(command, returncode, stdout=stdout, stderr=stderr)
monkeypatch.setattr(preflight, "run", fake_run)
image_ref = (
"europe-west6-docker.pkg.dev/teleo-501523/teleo/observatory-read-adapter:"
"33399bd9acfa09e4e5409873e86c8c46ac694a8d-29389090997-1@sha256:" + "a" * 64
)
checks = preflight.build_checks(image_ref)
assert checks
assert {check.status for check in checks} == {"pass"}
assert secret_value not in json.dumps([check.__dict__ for check in checks])
def test_malformed_successful_readback_becomes_blocked_check(monkeypatch) -> None:
monkeypatch.setattr(
preflight,
"run",
lambda command: subprocess.CompletedProcess(command, 0, stdout="[]", stderr=""),
)
check = preflight.command_check(
"malformed",
["gcloud", "sql", "instances", "describe"],
preflight.cloud_sql_is_private_and_iam_enabled,
"unexpected",
)
assert check.status == "blocked"
assert check.detail == "invalid_readback:AttributeError"
@pytest.mark.skipif(shutil.which("jq") is None, reason="jq is required")
def test_receipt_redaction_is_an_explicit_allowlist() -> None:
payload = {
"schema": "livingip.observatory-canonical-claim.v1",
"read_only": True,
"provenance": {
"gcp_project": "teleo-501523",
"cloud_sql_instance": "teleo-501523:europe-west6:teleo-pgvector-standby",
"database": "teleo_canonical",
"database_principal": "sa-observatory-read-adapter@teleo-501523.iam",
"authorization_role": "kb_observatory_read",
"transaction_read_only": True,
"write_privileges_denied": True,
"service_revision": "a" * 40,
"future_sensitive_field": "must-not-survive",
},
"canonical": {
"claim": {"id": "claim-1", "type": "fact", "status": "live", "text": "secret claim text"},
"evidence": [{"excerpt": "secret excerpt", "source_url": "https://private.invalid"}],
"edges": [{"direction": "incoming"}, {"direction": "outgoing"}],
},
"proposal_ledger": {
"distinct_from_canonical": True,
"status_counts": {"approved": 1},
"related": [{"proposal_text": "secret proposal"}],
},
}
result = subprocess.run(
["jq", "-f", str(ROOT / "ops" / "redact_observatory_read_adapter_receipt.jq")],
input=json.dumps(payload),
text=True,
capture_output=True,
check=True,
)
redacted = json.loads(result.stdout)
rendered = json.dumps(redacted)
assert set(redacted["provenance"]) == {
"gcp_project",
"cloud_sql_instance",
"database",
"database_principal",
"authorization_role",
"transaction_read_only",
"write_privileges_denied",
"service_revision",
}
assert redacted["canonical"]["evidence_count"] == 1
assert redacted["proposal_ledger"]["related_count"] == 1
for sensitive in (
"must-not-survive",
"secret claim text",
"secret excerpt",
"https://private.invalid",
"secret proposal",
):
assert sensitive not in rendered

30
tests/test_observatory_read_role_postgres.py
View file

@ -44,22 +44,22 @@ def run(command: list[str], *, input_text: str | None = None, check: bool = True
@pytest.mark.skipif(shutil.which("docker") is None, reason="Docker is required")
def test_observatory_role_is_read_only_and_exactly_allowlisted() -> None:
container = f"teleo-observatory-role-{uuid.uuid4().hex[:10]}"
run(
[
"docker",
"run",
"--detach",
"--rm",
"--name",
container,
"--env",
f"POSTGRES_PASSWORD={TEST_PASSWORD}",
"--env",
f"POSTGRES_DB={DATABASE}",
POSTGRES_IMAGE,
]
)
try:
run(
[
"docker",
"run",
"--detach",
"--rm",
"--name",
container,
"--env",
f"POSTGRES_PASSWORD={TEST_PASSWORD}",
"--env",
f"POSTGRES_DB={DATABASE}",
POSTGRES_IMAGE,
]
)
deadline = time.monotonic() + 25
while time.monotonic() < deadline:
ready = run(

29
tests/test_private_receipt_io.py Normal file
View file

@ -0,0 +1,29 @@
from __future__ import annotations
import json
from pathlib import Path
from ops.private_receipt_io import print_private_receipt_summary, write_private_json
def test_stdout_summary_never_replays_private_receipt_payload(tmp_path: Path, capsys) -> None:
output = tmp_path / "private" / "receipt.json"
payload = {
"artifact": "private_test_receipt",
"status": "pass",
"private_claim_body": "must-not-appear-on-stdout",
"database_password": "must-not-appear-on-stdout-either",
}
write_private_json(output, payload)
print_private_receipt_summary(output, payload)
stdout = capsys.readouterr().out
summary = json.loads(stdout)
assert "must-not-appear" not in stdout
assert summary["artifact"] == "private_test_receipt"
assert summary["status"] == "pass"
assert summary["output_name"] == output.name
assert str(output.parent) not in stdout
assert len(summary["output_sha256"]) == 64
assert output.stat().st_mode & 0o777 == 0o600

483
tests/test_restore_gcp_generated_postgres_snapshot.py
View file

@ -7,6 +7,7 @@ from pathlib import Path
import pytest
from ops import restore_gcp_generated_postgres_snapshot as restore
from ops.capture_vps_canonical_postgres_snapshot import build_provenance_binding
SERVICE_STATE = {
"ActiveState": "active",
@ -15,6 +16,7 @@ SERVICE_STATE = {
"NRestarts": "0",
"ExecMainStartTimestamp": "Thu 2026-07-09 07:00:04 UTC",
}
AUTHORIZATION_REF = "codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621"
def manifest_rows(database: str, *, target: bool, row_count: int = 1) -> list[dict]:
@ -47,7 +49,27 @@ def manifest_rows(database: str, *, target: bool, row_count: int = 1) -> list[di
"inherit": True,
"replication": False,
"bypass_rls": False,
}
},
{
"name": "kb_gate_owner",
"can_login": False,
"superuser": False,
"create_db": False,
"create_role": False,
"inherit": True,
"replication": False,
"bypass_rls": False,
},
{
"name": "kb_review",
"can_login": True,
"superuser": False,
"create_db": False,
"create_role": False,
"inherit": True,
"replication": False,
"bypass_rls": False,
},
],
},
]
@ -63,6 +85,9 @@ def manifest_rows(database: str, *, target: bool, row_count: int = 1) -> list[di
"triggers",
"types",
"policies",
"role_memberships",
"object_ownership",
"object_acl",
)
)
rows.extend(
@ -72,7 +97,7 @@ def manifest_rows(database: str, *, target: bool, row_count: int = 1) -> list[di
"schema": "public",
"table": "claims",
"row_count": row_count,
"rowset_md5": "same" if row_count == 1 else "different",
"rowset_md5": ("a" if row_count == 1 else "b") * 32,
},
{
"kind": "performance",
@ -94,6 +119,67 @@ def args_for(tmp_path: Path, **overrides) -> argparse.Namespace:
dump.write_bytes(b"PGDMPfixture")
source_manifest = tmp_path / "source-manifest.jsonl"
source_manifest.write_text(manifest_text("teleo", target=False), encoding="utf-8")
manifest_sql = Path("ops/postgres_parity_manifest.sql").resolve()
ssl_root_cert = tmp_path / "cloudsql-server-ca.pem"
ssl_root_cert.write_text("test-only-ca\n", encoding="utf-8")
source_context_sha256 = "4" * 64
snapshot = {
"exported_snapshot_id": "00000003-000001A2-1",
"txid_snapshot": "100:120:105,111",
"wal_lsn": "0/16B6C50",
"system_identifier": "7649789040005668902",
"captured_at_utc": "2026-07-15T01:02:03.456789+00:00",
}
source = {
"ssh_target": "root@77.42.65.182",
"container": "teleo-pg",
"database": "teleo",
"service": "leoclean-gateway.service",
}
dump_sha256 = hashlib.sha256(dump.read_bytes()).hexdigest()
source_manifest_sha256 = hashlib.sha256(source_manifest.read_bytes()).hexdigest()
manifest_sql_sha256 = hashlib.sha256(manifest_sql.read_bytes()).hexdigest()
source_receipt = tmp_path / "source-receipt.json"
source_receipt.write_text(
json.dumps(
{
"artifact": "vps_canonical_postgres_exported_snapshot",
"schema": restore.SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"receipt_version": 2,
"generated_at_utc": "2026-07-15T01:02:04+00:00",
"status": "pass",
"run_id": "canonical-20260715",
"capture_authorization_ref": AUTHORIZATION_REF,
"source": source,
"snapshot_exported": True,
"snapshot": snapshot,
"dump": {"path": "/private/source.dump", "bytes": dump.stat().st_size, "sha256": dump_sha256},
"manifest": {
"path": "/private/source-manifest.jsonl",
"sha256": source_manifest_sha256,
"manifest_sql_sha256": manifest_sql_sha256,
"table_count": 1,
"total_rows": 1,
},
"source_context": {"path": "/private/source-context.txt", "sha256": source_context_sha256},
"source_service": {"before": SERVICE_STATE, "after": SERVICE_STATE, "unchanged": True},
"provenance_binding": build_provenance_binding(
run_id="canonical-20260715",
authorization_ref=AUTHORIZATION_REF,
source=source,
snapshot=snapshot,
dump_sha256=dump_sha256,
manifest_sql_sha256=manifest_sql_sha256,
source_manifest_sha256=source_manifest_sha256,
source_context_sha256=source_context_sha256,
),
"service_unchanged": True,
"production_db_mutated": False,
"telegram_send_attempted": False,
}
),
encoding="utf-8",
)
values = {
"operation": "restore",
"execute": True,
@ -101,15 +187,29 @@ def args_for(tmp_path: Path, **overrides) -> argparse.Namespace:
"run_id": "gcp-restore-test12345",
"host": restore.DEFAULT_HOST,
"project": restore.DEFAULT_PROJECT,
"cloudsql_instance": restore.DEFAULT_INSTANCE,
"expected_private_network": restore.DEFAULT_PRIVATE_NETWORK,
"expected_compute_instance": restore.DEFAULT_COMPUTE_INSTANCE,
"expected_compute_zone": restore.DEFAULT_COMPUTE_ZONE,
"expected_restore_service_account": restore.DEFAULT_RESTORE_SERVICE_ACCOUNT,
"ssl_server_name": "teleo-pgvector-standby.private.teleo.internal",
"ssl_root_cert": ssl_root_cert,
"expected_source_ssh_target": restore.DEFAULT_SOURCE_SSH_TARGET,
"expected_source_container": restore.DEFAULT_SOURCE_CONTAINER,
"expected_source_database": restore.DEFAULT_SOURCE_DATABASE,
"expected_source_service": restore.DEFAULT_SOURCE_SERVICE,
"password_secret": restore.DEFAULT_SECRET,
"service": restore.DEFAULT_SERVICE,
"command_timeout": 2.0,
"psql_bin": "psql",
"run_root": tmp_path / "runs",
"dump": dump,
"expected_dump_sha256": hashlib.sha256(dump.read_bytes()).hexdigest(),
"expected_dump_sha256": dump_sha256,
"source_manifest": source_manifest,
"manifest_sql": Path("ops/postgres_parity_manifest.sql").resolve(),
"source_receipt": source_receipt,
"expected_source_receipt_sha256": hashlib.sha256(source_receipt.read_bytes()).hexdigest(),
"expected_capture_authorization_ref": AUTHORIZATION_REF,
"manifest_sql": manifest_sql,
"pg_restore_bin": "pg_restore",
"restore_timeout": 5.0,
"manifest_timeout": 5.0,
@ -128,6 +228,54 @@ def install_restore_fakes(
state = {"exists": False, "dropped": False}
monkeypatch.setattr(restore.os, "geteuid", lambda: 0)
monkeypatch.setattr(restore, "service_state", lambda *_args, **_kwargs: dict(SERVICE_STATE))
monkeypatch.setattr(
restore,
"gce_compute_identity",
lambda project, instance, zone, expected_private_network, expected_service_account, **_kwargs: {
"project": project,
"project_number": "785938879453",
"instance": instance,
"instance_id": "123456789",
"zone": zone,
"expected_private_network": expected_private_network,
"private_ip": "10.61.0.2",
"network_resource": "projects/123456789/networks/teleo-staging-net",
"service_account": expected_service_account,
"checks": {
"project": True,
"project_number": True,
"expected_network_project": True,
"instance": True,
"instance_id": True,
"zone": True,
"network": True,
"private_ip": True,
"service_account": True,
},
},
)
monkeypatch.setattr(
restore,
"cloudsql_control_plane_state",
lambda project, instance, host, expected_private_network, **_kwargs: {
"project": project,
"instance": instance,
"private_addresses": [host],
"expected_private_host": host,
"private_network": expected_private_network,
"expected_private_network": expected_private_network,
"public_ip_disabled": True,
"checks": {
"instance": True,
"state": True,
"postgres_16": True,
"public_ip_disabled": True,
"private_network": True,
"expected_private_host": True,
"no_non_private_address": True,
},
},
)
monkeypatch.setattr(restore, "resolve_password", lambda *_args, **_kwargs: "private-password")
monkeypatch.setattr(
restore,
@ -179,7 +327,10 @@ def test_restore_success_retains_only_exact_parity_clone(
assert result["phase"] == "retained_for_no_send_replay"
assert result["parity"]["problems"] == []
assert result["parity"]["details"]["source_total_rows"] == 1
assert result["source"]["capture_receipt"]["schema"] == restore.SOURCE_SNAPSHOT_RECEIPT_SCHEMA
assert result["source"]["capture_receipt"]["provenance_binding_sha256"]
assert result["target"]["connectivity"]["private_connectivity"] is True
assert result["target"]["cloudsql"]["public_ip_disabled"] is True
assert result["target"]["connectivity"]["ssl"] is True
assert result["live_service"]["unchanged"] is True
assert result["cleanup"]["clone_database_remaining"] == 1
@ -187,6 +338,15 @@ def test_restore_success_retains_only_exact_parity_clone(
run_dir = args.run_root / args.run_id
assert (run_dir / "target-manifest.jsonl").stat().st_mode & 0o777 == 0o600
assert (run_dir / "restore-receipt.json").stat().st_mode & 0o777 == 0o600
connectivity_path = run_dir / "gcp-private-connectivity.json"
assert connectivity_path.stat().st_mode & 0o777 == 0o600
connectivity = json.loads(connectivity_path.read_text())
assert connectivity["target_database"] == args.target_db
assert connectivity["source_compute"] == restore.DEFAULT_COMPUTE_INSTANCE
assert connectivity["public_ip_disabled"] is True
assert (
result["target"]["connectivity_proof"]["sha256"] == hashlib.sha256(connectivity_path.read_bytes()).hexdigest()
)
def test_parity_failure_drops_generated_clone_and_receipts_both_failures(
@ -234,6 +394,79 @@ def test_create_command_failure_still_drops_a_committed_clone(
assert state == {"exists": False, "dropped": True}
def test_restore_rejects_manifest_not_bound_to_source_receipt_before_credentials(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
args = args_for(tmp_path)
install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True))
receipt = json.loads(args.source_receipt.read_text())
receipt["manifest"]["sha256"] = "0" * 64
args.source_receipt.write_text(json.dumps(receipt))
args.expected_source_receipt_sha256 = hashlib.sha256(args.source_receipt.read_bytes()).hexdigest()
credential_called = False
def unexpected_credential(*_args, **_kwargs):
nonlocal credential_called
credential_called = True
return "private-password"
monkeypatch.setattr(restore, "resolve_password", unexpected_credential)
result = restore.run_restore(args)
assert result["status"] == "fail"
assert result["error"]["phase"] == "validation"
assert "source_manifest_sha256" in result["error"]["message"]
assert credential_called is False
def test_restore_rejects_unexpected_capture_authorization_before_credentials(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
args = args_for(tmp_path, expected_capture_authorization_ref="codex-delegation:another-authorized-run")
install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True))
credential_called = False
def unexpected_credential(*_args, **_kwargs):
nonlocal credential_called
credential_called = True
return "private-password"
monkeypatch.setattr(restore, "resolve_password", unexpected_credential)
result = restore.run_restore(args)
assert result["status"] == "fail"
assert result["error"]["phase"] == "validation"
assert "capture_authorization_ref" in result["error"]["message"]
assert credential_called is False
def test_restore_rejects_noncanonical_source_identity_before_credentials(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
args = args_for(tmp_path, expected_source_container="different-postgres")
install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True))
credential_called = False
def unexpected_credential(*_args, **_kwargs):
nonlocal credential_called
credential_called = True
return "private-password"
monkeypatch.setattr(restore, "resolve_password", unexpected_credential)
result = restore.run_restore(args)
assert result["status"] == "fail"
assert result["error"]["phase"] == "validation"
assert "source_identity" in result["error"]["message"]
assert credential_called is False
def test_changed_live_service_turns_successful_restore_into_cleanup(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
@ -283,6 +516,17 @@ def test_cleanup_is_bound_to_passing_restore_receipt(
"run_id",
"host",
"project",
"cloudsql_instance",
"expected_private_network",
"expected_compute_instance",
"expected_compute_zone",
"expected_restore_service_account",
"ssl_server_name",
"ssl_root_cert",
"expected_source_ssh_target",
"expected_source_container",
"expected_source_database",
"expected_source_service",
"password_secret",
"service",
"command_timeout",
@ -302,6 +546,112 @@ def test_cleanup_is_bound_to_passing_restore_receipt(
assert state == {"exists": False, "dropped": True}
def test_cleanup_rejects_target_topology_drift_before_credentials(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
args = args_for(tmp_path)
state = install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True))
assert restore.run_restore(args)["status"] == "pass"
cleanup_args = argparse.Namespace(
**{
key: getattr(args, key)
for key in (
"target_db",
"run_id",
"host",
"project",
"cloudsql_instance",
"expected_private_network",
"expected_compute_instance",
"expected_compute_zone",
"expected_restore_service_account",
"ssl_server_name",
"ssl_root_cert",
"expected_source_ssh_target",
"expected_source_container",
"expected_source_database",
"expected_source_service",
"password_secret",
"service",
"command_timeout",
"psql_bin",
"run_root",
)
},
operation="cleanup",
execute=True,
)
cleanup_args.host = "10.61.0.4"
credential_called = False
def unexpected_credential(*_args, **_kwargs):
nonlocal credential_called
credential_called = True
return "private-password"
monkeypatch.setattr(restore, "resolve_password", unexpected_credential)
with pytest.raises(restore.RestoreError, match="execution contract"):
restore.run_cleanup(cleanup_args)
assert credential_called is False
assert state == {"exists": True, "dropped": False}
def test_cleanup_retains_failure_receipt_when_post_drop_readback_fails(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
args = args_for(tmp_path)
state = install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True))
assert restore.run_restore(args)["status"] == "pass"
cleanup_args = argparse.Namespace(
**{
key: getattr(args, key)
for key in (
"target_db",
"run_id",
"host",
"project",
"cloudsql_instance",
"expected_private_network",
"expected_compute_instance",
"expected_compute_zone",
"expected_restore_service_account",
"ssl_server_name",
"ssl_root_cert",
"expected_source_ssh_target",
"expected_source_container",
"expected_source_database",
"expected_source_service",
"password_secret",
"service",
"command_timeout",
"psql_bin",
"run_root",
)
},
operation="cleanup",
execute=True,
)
monkeypatch.setattr(
restore,
"rollback_database_state",
lambda *_args, **_kwargs: (_ for _ in ()).throw(restore.RestoreError("rollback readback unavailable")),
)
result = restore.run_cleanup(cleanup_args)
assert result["status"] == "fail"
assert result["error"]["phase"] == "rollback_readback"
assert result["database"]["clone_database_remaining"] == 0
assert state == {"exists": False, "dropped": True}
receipt_path = args.run_root / args.run_id / "cleanup-receipt.json"
assert receipt_path.stat().st_mode & 0o777 == 0o600
assert json.loads(receipt_path.read_text())["error"]["phase"] == "rollback_readback"
def test_restore_dump_uses_tls_no_owner_no_acl_and_keeps_secret_out_of_argv(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
@ -318,7 +668,10 @@ def test_restore_dump_uses_tls_no_owner_no_acl_and_keeps_secret_out_of_argv(
assert captured["command"][0] == "pg_restore"
assert captured["command"][1] == "-d"
assert "sslmode=require" in captured["command"][2]
assert "sslmode=verify-full" in captured["command"][2]
assert "hostaddr='10.61.0.3'" in captured["command"][2]
assert "host='teleo-pgvector-standby.private.teleo.internal'" in captured["command"][2]
assert f"sslrootcert='{args.ssl_root_cert}'" in captured["command"][2]
assert "--no-owner" in captured["command"]
assert "--no-acl" in captured["command"]
assert "--exit-on-error" in captured["command"]
@ -326,6 +679,126 @@ def test_restore_dump_uses_tls_no_owner_no_acl_and_keeps_secret_out_of_argv(
assert captured["env"]["PGPASSWORD"] == "private-password"
def test_cloudsql_control_plane_preflight_requires_private_only_expected_host(
monkeypatch: pytest.MonkeyPatch,
) -> None:
payload = {
"name": "teleo-pgvector-standby",
"state": "RUNNABLE",
"databaseVersion": "POSTGRES_16",
"region": "europe-west6",
"settings": {
"tier": "db-custom-1-3840",
"ipConfiguration": {
"ipv4Enabled": False,
"privateNetwork": "projects/teleo-501523/global/networks/teleo-staging-net",
},
},
"ipAddresses": [{"type": "PRIVATE", "ipAddress": "10.61.0.3"}],
}
monkeypatch.setattr(
restore,
"_run",
lambda *_args, **_kwargs: subprocess.CompletedProcess([], 0, json.dumps(payload), ""),
)
result = restore.cloudsql_control_plane_state(
"teleo-501523",
"teleo-pgvector-standby",
"10.61.0.3",
restore.DEFAULT_PRIVATE_NETWORK,
timeout=2.0,
)
assert result["public_ip_disabled"] is True
assert result["private_addresses"] == ["10.61.0.3"]
payload["settings"]["ipConfiguration"]["ipv4Enabled"] = True
payload["ipAddresses"].append({"type": "PRIMARY", "ipAddress": "34.1.2.3"})
with pytest.raises(restore.RestoreError, match="no_non_private_address, public_ip_disabled"):
restore.cloudsql_control_plane_state(
"teleo-501523",
"teleo-pgvector-standby",
"10.61.0.3",
restore.DEFAULT_PRIVATE_NETWORK,
timeout=2.0,
)
def test_gce_compute_identity_is_bound_to_metadata_instance_zone_and_network(
monkeypatch: pytest.MonkeyPatch,
) -> None:
metadata = {
"project/project-id": "teleo-501523",
"project/numeric-project-id": "785938879453",
"instance/name": "teleo-prod-1",
"instance/id": "123456789",
"instance/zone": "projects/123456789/zones/europe-west6-a",
"instance/network-interfaces/0/network": "projects/785938879453/networks/teleo-staging-net",
"instance/network-interfaces/0/ip": "10.61.0.2",
"instance/service-accounts/default/email": restore.DEFAULT_RESTORE_SERVICE_ACCOUNT,
}
monkeypatch.setattr(restore, "gce_metadata_value", lambda path, **_kwargs: metadata[path])
result = restore.gce_compute_identity(
"teleo-501523",
"teleo-prod-1",
"europe-west6-a",
restore.DEFAULT_PRIVATE_NETWORK,
restore.DEFAULT_RESTORE_SERVICE_ACCOUNT,
timeout=2.0,
)
assert result["instance_id"] == "123456789"
assert result["project_number"] == "785938879453"
assert result["zone"] == "europe-west6-a"
assert all(result["checks"].values())
with pytest.raises(restore.RestoreError, match="instance"):
restore.gce_compute_identity(
"teleo-501523",
"different-instance",
"europe-west6-a",
restore.DEFAULT_PRIVATE_NETWORK,
restore.DEFAULT_RESTORE_SERVICE_ACCOUNT,
timeout=2.0,
)
metadata["instance/name"] = "teleo-prod-1"
metadata["instance/network-interfaces/0/network"] = "projects/999999999999/networks/teleo-staging-net"
with pytest.raises(restore.RestoreError, match="network"):
restore.gce_compute_identity(
"teleo-501523",
"teleo-prod-1",
"europe-west6-a",
restore.DEFAULT_PRIVATE_NETWORK,
restore.DEFAULT_RESTORE_SERVICE_ACCOUNT,
timeout=2.0,
)
metadata["instance/network-interfaces/0/network"] = "projects/785938879453/networks/teleo-staging-net"
with pytest.raises(restore.RestoreError, match="expected_network_project"):
restore.gce_compute_identity(
"teleo-501523",
"teleo-prod-1",
"europe-west6-a",
"projects/other-project/global/networks/teleo-staging-net",
restore.DEFAULT_RESTORE_SERVICE_ACCOUNT,
timeout=2.0,
)
metadata["instance/network-interfaces/0/network"] = "projects/785938879453/networks/teleo-staging-net"
metadata["instance/service-accounts/default/email"] = "project-owner@teleo-501523.iam.gserviceaccount.com"
with pytest.raises(restore.RestoreError, match="service_account"):
restore.gce_compute_identity(
"teleo-501523",
"teleo-prod-1",
"europe-west6-a",
restore.DEFAULT_PRIVATE_NETWORK,
restore.DEFAULT_RESTORE_SERVICE_ACCOUNT,
timeout=2.0,
)
def test_pg_restore_preflight_rejects_client_older_than_snapshot(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,

5
tests/test_run_local_canonical_postgres_rebuild.py
View file

@ -137,6 +137,9 @@ def manifest_rows() -> list[dict]:
{"kind": "schemas", "items": ["kb_stage", "public"]},
{"kind": "extensions", "items": []},
{"kind": "application_roles", "items": [role]},
{"kind": "role_memberships", "items": []},
{"kind": "object_ownership", "items": []},
{"kind": "object_acl", "items": []},
]
rows.extend(
{"kind": kind, "items": []}
@ -159,7 +162,7 @@ def manifest_rows() -> list[dict]:
"schema": "public",
"table": "claims",
"row_count": 1837,
"rowset_md5": "same-row-hash",
"rowset_md5": "c" * 32,
},
{
"kind": "performance",

51
tests/test_run_local_genesis_ledger_rebuild.py
View file

@ -16,6 +16,7 @@ from pathlib import Path
import pytest
from ops import run_local_genesis_ledger_rebuild as rebuild
from ops.verify_postgres_parity_manifest import compare_manifests, load_manifest
CLAIM_A = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"
CLAIM_B = "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb"
@ -76,6 +77,9 @@ def manifest_rows(database: str = "teleo_fixture") -> list[dict]:
{"kind": "schemas", "items": ["kb_stage", "public"]},
{"kind": "extensions", "items": []},
{"kind": "application_roles", "items": [role]},
{"kind": "role_memberships", "items": []},
{"kind": "object_ownership", "items": []},
{"kind": "object_acl", "items": []},
]
rows.extend(
{"kind": kind, "items": []}
@ -98,7 +102,7 @@ def manifest_rows(database: str = "teleo_fixture") -> list[dict]:
"schema": "public",
"table": "claims",
"row_count": 2,
"rowset_md5": "fixture",
"rowset_md5": "f" * 32,
},
{
"kind": "performance",
@ -1083,6 +1087,51 @@ def source_postgres() -> Iterator[tuple[str, str]]:
pytest.fail(f"fixture PostgreSQL container was not removed: {container}")
def test_live_manifest_normalizes_physical_database_authorization_identity(
source_postgres: tuple[str, str],
tmp_path: Path,
) -> None:
container, database = source_postgres
clone_database = f"teleo_clone_{uuid.uuid4().hex[:12]}"
docker_psql(
container,
"postgres",
f'create database "{clone_database}" template "{database}";',
)
try:
source_manifest_path = tmp_path / "source-database-manifest.jsonl"
clone_manifest_path = tmp_path / "clone-database-manifest.jsonl"
capture_manifest(container, database, source_manifest_path)
capture_manifest(container, clone_database, clone_manifest_path)
source_manifest = load_manifest(source_manifest_path)
clone_manifest = load_manifest(clone_manifest_path)
problems, _details = compare_manifests(
source_manifest,
clone_manifest,
max_target_query_ms=1_000_000,
max_target_source_ratio=1_000_000,
)
assert problems == []
for manifest in (source_manifest, clone_manifest):
ownership = manifest["singleton"]["object_ownership"]["items"]
database_owner = next(row for row in ownership if row["object_type"] == "database")
assert database_owner["name"] == "canonical_database"
database_acls = [
row for row in manifest["singleton"]["object_acl"]["items"] if row["object_type"] == "database"
]
assert database_acls
assert {row["name"] for row in database_acls} == {"canonical_database"}
finally:
docker_psql(
container,
"postgres",
f'drop database if exists "{clone_database}" with (force);',
check=False,
)
def capture_source_snapshot(
source_container: str,
database: str,

19
tests/test_teleo_agent_systemd.py
View file

@ -339,24 +339,24 @@ def test_gcp_runtime_sync_is_source_bound_and_scoped():
'sudo install -o root -g root -m 0755 "$payload_dir/$SERVICE_ENV_VERIFIER_REL" '
'"$service_env_verifier_stage"' in script
)
assert 'assert_installed_files\nsudo systemctl daemon-reload' in script
assert "assert_installed_files\nsudo systemctl daemon-reload" in script
assert 'tar -C "$remote_tmp" -xf -' not in script
assert 'mkdir -p "$backup_dir"' not in script
assert 'for protected_file in \\\n "$wrapper" \\' in script
assert '"$permission_verifier" \\' in script
assert '"$service_env_verifier" \\' in script
assert 'sudo -n -u teleo test ! -w "$protected_file"' in script
assert 'for protected_dir in \\\n /usr \\' in script
assert "for protected_dir in \\\n /usr \\" in script
assert "/usr/local/libexec/livingip" in script
assert 'sudo -n -u teleo test ! -w "$protected_dir"' in script
assert 'for executable_dir in \\\n /usr/local/sbin \\' in script
assert 'sudo stat -Lc \'%U:%G\' "$executable_dir"' in script
assert "for executable_dir in \\\n /usr/local/sbin \\" in script
assert "sudo stat -Lc '%U:%G' \"$executable_dir\"" in script
assert 'sudo -n -u teleo test ! -w "$executable_dir"' in script
assert (
'for protected_file in /bin/bash /usr/bin/curl /usr/bin/env /usr/bin/gcloud '
'/usr/bin/psql /usr/bin/python3 /usr/bin/setpriv' in script
"for protected_file in /bin/bash /usr/bin/curl /usr/bin/env /usr/bin/gcloud "
"/usr/bin/psql /usr/bin/python3 /usr/bin/setpriv" in script
)
assert '/usr/bin/curl -q --noproxy' in script
assert "/usr/bin/curl -q --noproxy" in script
assert 'test "$(sudo cat "$revision")" = "$SOURCE_REVISION"' in script
assert 'assert_existing_root_directory "$existing_directory"' in script
assert 'assert_existing_root_file "$existing_file"' in script
@ -389,10 +389,7 @@ def test_gcp_runtime_sync_is_source_bound_and_scoped():
assert '"current_tier": "T3_live_readonly"' in script
assert '"status": "pass"' in script
assert "UnsetEnvironment=PGPASSWORD" in dropin
assert (
"UnsetEnvironment=GOOGLE_APPLICATION_CREDENTIALS CLOUDSDK_CORE_PROJECT "
"CLOUDSDK_AUTH_ACCESS_TOKEN" in dropin
)
assert "UnsetEnvironment=GOOGLE_APPLICATION_CREDENTIALS CLOUDSDK_CORE_PROJECT CLOUDSDK_AUTH_ACCESS_TOKEN" in dropin
assert "UnsetEnvironment=BASH_ENV ENV BASHOPTS SHELLOPTS" in dropin
assert "UnsetEnvironment=LD_PRELOAD LD_LIBRARY_PATH LD_AUDIT" in dropin
assert "UnsetEnvironment=PYTHONHOME PYTHONPATH PYTHONSTARTUP PYTHONINSPECT" in dropin

820
tests/test_verify_gcp_canonical_lifecycle.py Normal file
View file

@ -0,0 +1,820 @@
import hashlib
import json
from datetime import UTC, datetime
from pathlib import Path
from ops.restore_gcp_generated_postgres_snapshot import DEFAULT_RESTORE_SERVICE_ACCOUNT
from ops.verify_gcp_canonical_lifecycle import EXPECTED_BLIND_ROW_IDS
from ops.verify_gcp_canonical_lifecycle import verify_lifecycle as _verify_lifecycle
from ops.verify_postgres_parity_manifest import REVIEWED_MANIFEST_SQL_SHA256
VERIFY_NOW = datetime(2026, 7, 15, 1, 3, tzinfo=UTC)
def verify_lifecycle(**kwargs):
return _verify_lifecycle(**kwargs, now=VERIFY_NOW)
def write_json(path: Path, payload: dict) -> Path:
path.write_text(json.dumps(payload, sort_keys=True) + "\n", encoding="utf-8")
return path
def sha256(path: Path) -> str:
return hashlib.sha256(path.read_bytes()).hexdigest()
def lifecycle_receipts(tmp_path: Path) -> dict[str, Path]:
target_db = "teleo_clone_current_test"
run_id = "gcp-restore-current-test123"
healthy_service = {
"ActiveState": "active",
"SubState": "running",
"MainPID": "148735",
"NRestarts": "0",
}
structural_hashes = {
kind: {"source": format(index, "x") * 64, "target": format(index, "x") * 64}
for index, kind in enumerate(
(
"schemas",
"columns",
"constraints",
"indexes",
"sequences",
"views",
"functions",
"triggers",
"types",
"policies",
"role_memberships",
"object_ownership",
"object_acl",
),
1,
)
}
exact_parity_details = {
"source_database": "teleo",
"target_database": target_db,
"source_table_count": 39,
"target_table_count": 39,
"source_total_rows": 52167,
"target_total_rows": 52167,
"table_mismatches": [],
"table_hash_problems": [],
"structural_hashes": structural_hashes,
"required_extension_mismatches": {},
"target_extra_extensions": [],
"application_role_mismatches": {},
"target_extra_application_roles": [],
"performance": [{"query": "count_claims", "source_ms": 1.0, "target_ms": 2.0, "source_ratio": 0.4}],
"performance_problems": [],
}
source = {
"artifact": "vps_canonical_postgres_exported_snapshot",
"schema": "livingip.sourceSnapshotReceipt.v2",
"receipt_version": 2,
"status": "pass",
"run_id": "canonical-current-test",
"capture_authorization_ref": "codex-delegation:test1234",
"source": {
"ssh_target": "root@77.42.65.182",
"container": "teleo-pg",
"database": "teleo",
"service": "leoclean-gateway.service",
},
"snapshot_exported": True,
"snapshot": {
"captured_at_utc": "2026-07-15T00:59:00+00:00",
"exported_snapshot_id": "00000003-000001A2-1",
"txid_snapshot": "100:120:",
"wal_lsn": "0/16B6C50",
"system_identifier": "7649789040005668902",
},
"service_unchanged": True,
"source_service": {"before": healthy_service, "after": healthy_service, "unchanged": True},
"production_db_mutated": False,
"telegram_send_attempted": False,
"dump": {"sha256": "a" * 64},
"manifest": {
"sha256": "b" * 64,
"manifest_sql_sha256": REVIEWED_MANIFEST_SQL_SHA256,
"table_count": 39,
"total_rows": 52167,
},
"provenance_binding": {"algorithm": "sha256", "payload": {"bound": True}, "sha256": "c" * 64},
}
source_path = write_json(tmp_path / "source.json", source)
restore = {
"artifact": "gcp_generated_postgres_snapshot_restore",
"status": "pass",
"generated_at_utc": "2026-07-15T00:59:10+00:00",
"completed_at_utc": "2026-07-15T00:59:30+00:00",
"run_id": run_id,
"target_database": target_db,
"source": {
"capture_receipt": {
"sha256": sha256(source_path),
"capture_authorization_ref": "codex-delegation:test1234",
"provenance_binding_sha256": "c" * 64,
"dump_sha256": "a" * 64,
"manifest_sha256": "b" * 64,
"manifest_sql_sha256": REVIEWED_MANIFEST_SQL_SHA256,
"checks": {
name: True
for name in (
"artifact",
"schema",
"receipt_version",
"status",
"snapshot_exported",
"service_unchanged",
"source_service_healthy",
"source_not_mutated",
"telegram_not_sent",
"run_id",
"capture_authorization_ref",
"source_identity",
"source_database",
"snapshot_identity",
"dump_sha256",
"dump_bytes",
"source_manifest_sha256",
"manifest_sql_sha256",
"source_context_sha256",
"table_count",
"total_rows",
"binding_algorithm",
"binding_payload",
"binding_sha256",
)
},
}
},
"execution_contract": {
"project": "teleo-501523",
"cloudsql_instance": "teleo-pgvector-standby",
"host": "10.61.0.3",
"private_network": "projects/teleo-501523/global/networks/teleo-staging-net",
"compute_instance": "teleo-prod-1",
"compute_zone": "europe-west6-a",
"restore_service_account": DEFAULT_RESTORE_SERVICE_ACCOUNT,
"ssl_server_name": "teleo-pgvector-standby.private.teleo.internal",
"ssl_root_cert_sha256": "9" * 64,
"source": source["source"],
"password_secret": "gcp-teleo-pgvector-standby-postgres-password",
"service": "leoclean-gcp-prod-parallel.service",
},
"target": {
"manifest_sha256": "d" * 64,
"connectivity": {
"private_connectivity": True,
"ssl": True,
"sslmode": "verify-full",
"ssl_server_name": "teleo-pgvector-standby.private.teleo.internal",
"ssl_root_cert_sha256": "9" * 64,
"server_identity_verified": True,
"server_address": "10.61.0.3",
"source_compute": "teleo-prod-1",
},
"cloudsql": {
"project": "teleo-501523",
"instance": "teleo-pgvector-standby",
"expected_private_host": "10.61.0.3",
"private_network": "projects/teleo-501523/global/networks/teleo-staging-net",
"expected_private_network": "projects/teleo-501523/global/networks/teleo-staging-net",
"public_ip_disabled": True,
"checks": {
"instance": True,
"state": True,
"postgres_16": True,
"public_ip_disabled": True,
"private_network": True,
"expected_private_host": True,
"no_non_private_address": True,
},
},
"compute": {
"project": "teleo-501523",
"project_number": "785938879453",
"instance": "teleo-prod-1",
"instance_id": "123456789",
"zone": "europe-west6-a",
"expected_private_network": "projects/teleo-501523/global/networks/teleo-staging-net",
"network_resource": "projects/123456789/networks/teleo-staging-net",
"private_ip": "10.61.0.2",
"service_account": DEFAULT_RESTORE_SERVICE_ACCOUNT,
"checks": {
"project": True,
"project_number": True,
"expected_network_project": True,
"instance": True,
"instance_id": True,
"zone": True,
"network": True,
"private_ip": True,
"service_account": True,
},
},
"connectivity_proof": {
"sha256": "f" * 64,
"schema": "livingip.gcpPrivatePostgresConnectivity.v2",
},
},
"parity": {"status": "pass", "problems": [], "details": exact_parity_details},
"live_service": {"before": healthy_service, "after": healthy_service, "unchanged": True},
}
restore_path = write_json(tmp_path / "restore.json", restore)
parity = {
"artifact": "canonical_postgres_parity_verification",
"status": "pass",
"completed_at_utc": "2026-07-15T00:59:45+00:00",
"scope": "gcp_staging",
"problems": [],
"source_manifest_sha256": "b" * 64,
"target_manifest_sha256": "d" * 64,
"connectivity_proof_sha256": "f" * 64,
"details": exact_parity_details,
"private_connectivity": {
"required": True,
"proof": {
"artifact": "gcp_private_postgres_connectivity",
"schema": "livingip.gcpPrivatePostgresConnectivity.v2",
"status": "pass",
"restore_run_id": run_id,
"private_connectivity": True,
"public_ip_disabled": True,
"target_database": target_db,
"source_compute": "teleo-prod-1",
"source_compute_instance_id": "123456789",
"source_compute_zone": "europe-west6-a",
"source_compute_private_ip": "10.61.0.2",
"server_address": "10.61.0.3",
"sslmode": "verify-full",
"ssl_server_name": "teleo-pgvector-standby.private.teleo.internal",
"ssl_root_cert_sha256": "9" * 64,
"server_identity_verified": True,
"project": "teleo-501523",
"cloudsql_instance": "teleo-pgvector-standby",
"private_network": "projects/teleo-501523/global/networks/teleo-staging-net",
"cloudsql_control_plane_checks": restore["target"]["cloudsql"]["checks"],
"compute_identity_checks": restore["target"]["compute"]["checks"],
},
},
}
parity_path = write_json(tmp_path / "parity.json", parity)
required_checks = {
name: True
for name in (
"exact_blind_prompt_without_ids",
"claim_and_sources_retrieved",
"discovery_show_evidence_completed",
"retrieval_receipts_proven",
"private_tls_read_only_target",
"generated_database_unchanged",
"canonical_counts_unchanged",
"database_calls_read_only",
"no_database_write",
"no_telegram_send",
"live_service_unchanged",
"live_profile_unchanged",
"temporary_profile_removed",
)
}
reasoning = {
"schema": "livingip.gcpGeneratedDbBlindClaimCanary.v1",
"status": "pass",
"generated_at_utc": "2026-07-15T01:00:00+00:00",
"completed_at_utc": "2026-07-15T01:01:00+00:00",
"mode": "gcp_generated_db_gatewayrunner_blind_claim_no_send",
"source_compute": "teleo-prod-1",
"prompt": "Inspect the live claim without me giving you an ID. Do not change the database.",
"errors": [],
"target_database": target_db,
"posted_to_telegram": False,
"database_write_attempted": False,
"parity_receipt": {"sha256": sha256(parity_path)},
"checks": required_checks,
"database_fingerprint_before": {"sha256": "e" * 64},
"database_fingerprint_after": {"sha256": "e" * 64},
"canonical_status_before": {"claims": 1},
"canonical_status_after": {"claims": 1},
"database_identity_before": {"database": target_db},
"database_identity_after": {"database": target_db},
"service_before": healthy_service,
"service_after": healthy_service,
"temp_profile_absent": True,
"temp_profile_removed": True,
"outcomes": {
"asks_for_user_iteration_before_live": True,
"challenges_weak_support": True,
"decomposes_the_bundled_legal_claim": True,
"does_not_claim_a_database_change": True,
"proposes_candidate_claims": True,
"uses_only_m3taversal_handle": True,
},
"result": {
"reply": "The claim is weak; here are grounded, review-first alternatives.",
"database_tool_trace": {
"schema": "livingip.leoKbToolTrace.v1",
"database_tool_call_proven": True,
"database_retrieval_receipt_proven": True,
"database_tool_calls_read_only": True,
"database_tool_call_count": 3,
"database_tool_completed_count": 3,
"calls": [
{
"database_invocations": [{"access_mode": "read_only", "subcommand": subcommand}],
"result": {
"nonempty": True,
"error_detected": False,
"row_ids": sorted(EXPECTED_BLIND_ROW_IDS),
"row_id_count": len(EXPECTED_BLIND_ROW_IDS),
"retrieval_receipt": {
"schema": "livingip.teleoKbRetrievalReceipt.v1",
"semantic_context_sha256": "1" * 64,
"artifact_state_sha256": "2" * 64,
"read_consistency_status": "stable_wal_marker",
},
},
}
for subcommand in ("search", "show", "evidence")
],
},
"gateway": {
"authorized": True,
"handler_invoked": True,
"model_free_fallback_used": False,
"posted_to_telegram": False,
"child_process": {
"exitcode": 0,
"alive_after_readback": False,
"process_group_alive_after_readback": False,
"timed_out": False,
},
"tool_surface": {"send_message_tool_enabled": False},
},
},
"wrapper_tool_proof": {
"all_bound_to_supplied_target": True,
"all_completed_successfully": True,
"complete_start_end_pairing": True,
"database_read_only_required": True,
"default_read_only_required": True,
"invocations": [
{
"database": target_db,
"returncode": 0,
"subcommand": subcommand,
"database_identity": {
"transaction_read_only": "on",
"default_transaction_read_only": "on",
"ssl": True,
"sslmode": "verify-full",
"ssl_server_name": "teleo-pgvector-standby.private.teleo.internal",
"ssl_root_cert_sha256": "9" * 64,
"server_identity_verified": True,
"server_address": "10.61.0.3",
},
}
for subcommand in ("search", "show", "evidence")
],
},
}
reasoning_path = write_json(tmp_path / "reasoning.json", reasoning)
reasoning_compute = {
"artifact": "gcp_reasoning_compute_attestation",
"schema": "livingip.gcpReasoningComputeAttestation.v1",
"status": "pass",
"generated_at_utc": "2026-07-15T01:01:05+00:00",
"restore_run_id": run_id,
"target_database": target_db,
"reasoning_receipt": {
"sha256": sha256(reasoning_path),
"schema": reasoning["schema"],
"generated_at_utc": reasoning["generated_at_utc"],
"completed_at_utc": reasoning["completed_at_utc"],
"claimed_source_compute": reasoning["source_compute"],
},
"compute": restore["target"]["compute"],
"checks": {
"schema": True,
"status": True,
"target_database": True,
"source_compute": True,
"chronology": True,
"no_telegram_send": True,
"no_database_write": True,
"gce_metadata_identity": True,
},
"method": "gce_metadata_server_v1",
"mutation": {
"cloudsql_changed": False,
"compute_changed": False,
"database_changed": False,
"telegram_message_sent": False,
},
}
reasoning_compute_path = write_json(tmp_path / "reasoning-compute.json", reasoning_compute)
current_source = json.loads(json.dumps(source))
current_source["run_id"] = "canonical-current-postflight"
current_source["capture_authorization_ref"] = "codex-delegation:postflight1234"
current_source["snapshot"]["captured_at_utc"] = "2026-07-15T01:02:00+00:00"
current_source["snapshot"]["exported_snapshot_id"] = "00000003-000001B2-1"
current_source_path = write_json(tmp_path / "current-source.json", current_source)
source_delta = {
"artifact": "vps_canonical_snapshot_postflight_delta",
"schema": "livingip.vpsCanonicalSnapshotDelta.v1",
"status": "pass",
"baseline": {
"run_id": "canonical-current-test",
"capture_receipt_sha256": sha256(source_path),
"manifest_sha256": "b" * 64,
"manifest_sql_sha256": REVIEWED_MANIFEST_SQL_SHA256,
"capture_authorization_ref": "codex-delegation:test1234",
"captured_at_utc": "2026-07-15T00:59:00+00:00",
"table_count": 39,
"total_rows": 52167,
"table_fingerprint_sha256": "d" * 64,
"structure_fingerprint_sha256": "e" * 64,
},
"current": {
"run_id": "canonical-current-postflight",
"capture_receipt_sha256": sha256(current_source_path),
"manifest_sha256": "b" * 64,
"manifest_sql_sha256": REVIEWED_MANIFEST_SQL_SHA256,
"capture_authorization_ref": "codex-delegation:postflight1234",
"captured_at_utc": "2026-07-15T01:02:00+00:00",
"table_count": 39,
"total_rows": 52167,
"table_fingerprint_sha256": "d" * 64,
"structure_fingerprint_sha256": "e" * 64,
},
"delta": {
"delta_detected": False,
"added_tables": [],
"removed_tables": [],
"changed_tables": [],
"changed_structures": [],
"total_row_delta": 0,
},
}
source_delta_path = write_json(tmp_path / "source-delta.json", source_delta)
cleanup = {
"artifact": "gcp_generated_postgres_snapshot_cleanup",
"status": "pass",
"generated_at_utc": "2026-07-15T01:01:10+00:00",
"completed_at_utc": "2026-07-15T01:01:30+00:00",
"run_id": run_id,
"target_database": target_db,
"database": {"existed_before": True, "clone_database_remaining": 0},
"live_service": {"before": healthy_service, "after": healthy_service, "unchanged": True},
"restore_receipt": {"sha256": sha256(restore_path)},
"execution_contract": restore["execution_contract"],
"target": {
"cloudsql": restore["target"]["cloudsql"],
"compute": restore["target"]["compute"],
},
}
cleanup_path = write_json(tmp_path / "cleanup.json", cleanup)
return {
"source_path": source_path,
"current_source_path": current_source_path,
"source_delta_path": source_delta_path,
"restore_path": restore_path,
"parity_path": parity_path,
"reasoning_path": reasoning_path,
"reasoning_compute_path": reasoning_compute_path,
"cleanup_path": cleanup_path,
}
def test_caller_authored_bundle_is_preparation_only_even_when_all_checks_pass(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
result = verify_lifecycle(**paths)
assert result["status"] == "prepared_external_attestation_required"
assert result["failed_checks"] == []
assert result["checks"]["clone_absent"] is True
assert result["current_tier"] == "T2_structural_receipt_bundle"
assert result["required_tier"] == "T3_live_private_gcp_staging"
assert result["t3_receipt_ready"] is False
assert result["accepted_by_this_api"] is False
assert "current private GCP staging" not in result["strongest_claim_allowed"]
def test_rebound_unreviewed_manifest_sql_digest_fails_closed(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
unreviewed_digest = "f" * 64
source = json.loads(paths["source_path"].read_text())
source["manifest"]["manifest_sql_sha256"] = unreviewed_digest
write_json(paths["source_path"], source)
current_source = json.loads(paths["current_source_path"].read_text())
current_source["manifest"]["manifest_sql_sha256"] = unreviewed_digest
write_json(paths["current_source_path"], current_source)
restore = json.loads(paths["restore_path"].read_text())
restore["source"]["capture_receipt"]["sha256"] = sha256(paths["source_path"])
restore["source"]["capture_receipt"]["manifest_sql_sha256"] = unreviewed_digest
write_json(paths["restore_path"], restore)
delta = json.loads(paths["source_delta_path"].read_text())
delta["baseline"]["capture_receipt_sha256"] = sha256(paths["source_path"])
delta["baseline"]["manifest_sql_sha256"] = unreviewed_digest
delta["current"]["capture_receipt_sha256"] = sha256(paths["current_source_path"])
delta["current"]["manifest_sql_sha256"] = unreviewed_digest
write_json(paths["source_delta_path"], delta)
cleanup = json.loads(paths["cleanup_path"].read_text())
cleanup["restore_receipt"]["sha256"] = sha256(paths["restore_path"])
write_json(paths["cleanup_path"], cleanup)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "source_delta_manifest_sql_reviewed" in result["failed_checks"]
def test_local_parity_or_missing_cleanup_fails_closed(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
parity = json.loads(paths["parity_path"].read_text())
parity["scope"] = "local_restore"
write_json(paths["parity_path"], parity)
reasoning = json.loads(paths["reasoning_path"].read_text())
reasoning["parity_receipt"]["sha256"] = sha256(paths["parity_path"])
write_json(paths["reasoning_path"], reasoning)
cleanup = json.loads(paths["cleanup_path"].read_text())
cleanup["database"]["clone_database_remaining"] = 1
write_json(paths["cleanup_path"], cleanup)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert {"parity_pass", "clone_absent"} <= set(result["failed_checks"])
def test_cleanup_cannot_pass_when_the_named_clone_never_existed(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
cleanup = json.loads(paths["cleanup_path"].read_text())
cleanup["database"]["existed_before"] = False
write_json(paths["cleanup_path"], cleanup)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "clone_absent" in result["failed_checks"]
def test_mismatched_source_or_reasoning_receipt_fails_closed(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
restore = json.loads(paths["restore_path"].read_text())
restore["source"]["capture_receipt"]["sha256"] = "0" * 64
write_json(paths["restore_path"], restore)
reasoning = json.loads(paths["reasoning_path"].read_text())
reasoning["checks"]["claim_and_sources_retrieved"] = False
write_json(paths["reasoning_path"], reasoning)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert {"source_receipt_hash_bound", "reasoning_required_checks"} <= set(result["failed_checks"])
def test_hand_authored_reasoning_boole_without_producer_evidence_fail(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
reasoning = json.loads(paths["reasoning_path"].read_text())
reasoning.pop("result")
reasoning.pop("wrapper_tool_proof")
write_json(paths["reasoning_path"], reasoning)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "reasoning_producer_evidence" in result["failed_checks"]
assert result["reasoning_producer_checks"]["tool_trace"] is False
def test_reasoning_compute_attestation_must_bind_live_metadata(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
attestation = json.loads(paths["reasoning_compute_path"].read_text())
attestation["compute"]["instance_id"] = "999999999"
write_json(paths["reasoning_compute_path"], attestation)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "reasoning_compute_identity_bound" in result["failed_checks"]
def test_project_owner_shaped_service_account_cannot_pass_preparation(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
project_owner = "project-owner@teleo-501523.iam.gserviceaccount.com"
restore = json.loads(paths["restore_path"].read_text())
restore["target"]["compute"]["service_account"] = project_owner
restore["execution_contract"]["restore_service_account"] = project_owner
write_json(paths["restore_path"], restore)
attestation = json.loads(paths["reasoning_compute_path"].read_text())
attestation["compute"]["service_account"] = project_owner
write_json(paths["reasoning_compute_path"], attestation)
cleanup = json.loads(paths["cleanup_path"].read_text())
cleanup["target"]["compute"]["service_account"] = project_owner
cleanup["execution_contract"]["restore_service_account"] = project_owner
cleanup["restore_receipt"]["sha256"] = sha256(paths["restore_path"])
write_json(paths["cleanup_path"], cleanup)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "restore_compute_contract" in result["failed_checks"]
def test_source_postflight_must_be_newer_than_cleanup_completion(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
cleanup = json.loads(paths["cleanup_path"].read_text())
cleanup["completed_at_utc"] = "2026-07-15T02:00:00+00:00"
write_json(paths["cleanup_path"], cleanup)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "source_postflight_after_lifecycle" in result["failed_checks"]
def test_parity_must_retain_exact_structure_roles_rows_and_performance(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
parity = json.loads(paths["parity_path"].read_text())
parity["details"]["structural_hashes"].pop("constraints")
parity["details"]["application_role_mismatches"] = {"kb_apply": {"missing": True}}
parity["details"]["performance"] = []
write_json(paths["parity_path"], parity)
reasoning = json.loads(paths["reasoning_path"].read_text())
reasoning["parity_receipt"]["sha256"] = sha256(paths["parity_path"])
write_json(paths["reasoning_path"], reasoning)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "parity_exact_evidence" in result["failed_checks"]
assert result["parity_evidence_checks"]["structural_hashes"] is False
assert result["parity_evidence_checks"]["roles"] is False
assert result["parity_evidence_checks"]["performance"] is False
def test_reasoning_must_retrieve_expected_claim_sources_and_preserve_handle(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
reasoning = json.loads(paths["reasoning_path"].read_text())
for call in reasoning["result"]["database_tool_trace"]["calls"]:
call["result"]["row_ids"] = []
call["result"]["row_id_count"] = 0
reasoning["result"]["reply"] = "Cory found a weak claim and two sources."
write_json(paths["reasoning_path"], reasoning)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "reasoning_producer_evidence" in result["failed_checks"]
assert result["reasoning_producer_checks"]["expected_claim_and_sources_retrieved"] is False
assert result["reasoning_producer_checks"]["reply_uses_only_m3taversal_handle"] is False
def test_reasoning_wrapper_must_authenticate_the_cloudsql_server(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
reasoning = json.loads(paths["reasoning_path"].read_text())
for invocation in reasoning["wrapper_tool_proof"]["invocations"]:
invocation["database_identity"]["sslmode"] = "require"
invocation["database_identity"]["server_identity_verified"] = False
write_json(paths["reasoning_path"], reasoning)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "reasoning_producer_evidence" in result["failed_checks"]
assert result["reasoning_producer_checks"]["wrapper_invocations"] is False
def test_lifecycle_requires_restore_parity_reasoning_cleanup_chronology(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
cleanup = json.loads(paths["cleanup_path"].read_text())
cleanup["generated_at_utc"] = "2026-07-15T00:58:00+00:00"
write_json(paths["cleanup_path"], cleanup)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "lifecycle_chronology" in result["failed_checks"]
def test_source_delta_requires_real_fingerprints_and_consistent_no_delta(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
delta = json.loads(paths["source_delta_path"].read_text())
delta["baseline"].pop("table_fingerprint_sha256")
delta["current"]["total_rows"] += 1
write_json(paths["source_delta_path"], delta)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert {"source_delta_baseline_shape", "source_delta_current_shape", "source_delta_consistent"} <= set(
result["failed_checks"]
)
def test_detected_source_delta_requires_exact_row_arithmetic(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
delta = json.loads(paths["source_delta_path"].read_text())
delta["delta"]["delta_detected"] = True
delta["delta"]["changed_tables"] = [{"table": "public.claims"}]
delta["delta"]["total_row_delta"] = 99
write_json(paths["source_delta_path"], delta)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "source_delta_consistent" in result["failed_checks"]
def test_postflight_source_requires_structured_healthy_service(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
current = json.loads(paths["current_source_path"].read_text())
current["source_service"]["before"]["ActiveState"] = "failed"
current["source_service"]["after"]["ActiveState"] = "failed"
write_json(paths["current_source_path"], current)
delta = json.loads(paths["source_delta_path"].read_text())
delta["current"]["capture_receipt_sha256"] = sha256(paths["current_source_path"])
write_json(paths["source_delta_path"], delta)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "current_source_pass" in result["failed_checks"]
def test_unchanged_but_inactive_service_cannot_pass_t3(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
restore = json.loads(paths["restore_path"].read_text())
restore["live_service"]["before"]["ActiveState"] = "failed"
restore["live_service"]["after"]["ActiveState"] = "failed"
write_json(paths["restore_path"], restore)
cleanup = json.loads(paths["cleanup_path"].read_text())
cleanup["restore_receipt"]["sha256"] = sha256(paths["restore_path"])
write_json(paths["cleanup_path"], cleanup)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "restore_service_healthy_unchanged" in result["failed_checks"]
def test_historical_but_monotonic_receipts_cannot_claim_current_t3(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
result = _verify_lifecycle(**paths, now=datetime(2026, 7, 15, 3, 0, tzinfo=UTC))
assert result["status"] == "fail"
assert "source_postflight_fresh" in result["failed_checks"]
assert result["postflight_age_seconds"] > result["max_postflight_age_seconds"]
def test_fresh_postflight_cannot_mask_historical_gcp_lifecycle(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
source = json.loads(paths["source_path"].read_text())
source["snapshot"]["captured_at_utc"] = "2026-07-12T00:59:00+00:00"
write_json(paths["source_path"], source)
restore = json.loads(paths["restore_path"].read_text())
restore["generated_at_utc"] = "2026-07-12T00:59:10+00:00"
restore["completed_at_utc"] = "2026-07-12T00:59:30+00:00"
restore["source"]["capture_receipt"]["sha256"] = sha256(paths["source_path"])
write_json(paths["restore_path"], restore)
parity = json.loads(paths["parity_path"].read_text())
parity["completed_at_utc"] = "2026-07-12T00:59:45+00:00"
write_json(paths["parity_path"], parity)
reasoning = json.loads(paths["reasoning_path"].read_text())
reasoning["generated_at_utc"] = "2026-07-12T01:00:00+00:00"
reasoning["completed_at_utc"] = "2026-07-12T01:01:00+00:00"
reasoning["parity_receipt"]["sha256"] = sha256(paths["parity_path"])
write_json(paths["reasoning_path"], reasoning)
attestation = json.loads(paths["reasoning_compute_path"].read_text())
attestation["generated_at_utc"] = "2026-07-12T01:01:05+00:00"
attestation["reasoning_receipt"]["sha256"] = sha256(paths["reasoning_path"])
attestation["reasoning_receipt"]["generated_at_utc"] = reasoning["generated_at_utc"]
attestation["reasoning_receipt"]["completed_at_utc"] = reasoning["completed_at_utc"]
write_json(paths["reasoning_compute_path"], attestation)
cleanup = json.loads(paths["cleanup_path"].read_text())
cleanup["generated_at_utc"] = "2026-07-12T01:01:10+00:00"
cleanup["completed_at_utc"] = "2026-07-12T01:01:30+00:00"
cleanup["restore_receipt"]["sha256"] = sha256(paths["restore_path"])
write_json(paths["cleanup_path"], cleanup)
delta = json.loads(paths["source_delta_path"].read_text())
delta["baseline"]["captured_at_utc"] = "2026-07-12T00:59:00+00:00"
delta["baseline"]["capture_receipt_sha256"] = sha256(paths["source_path"])
write_json(paths["source_delta_path"], delta)
result = verify_lifecycle(**paths)
assert result["checks"]["source_postflight_fresh"] is True
assert result["checks"]["lifecycle_chronology"] is True
assert result["status"] == "fail"
assert {"gcp_lifecycle_fresh", "lifecycle_span_bounded"} <= set(result["failed_checks"])

77
tests/test_verify_postgres_parity_manifest.py
View file

@ -9,7 +9,7 @@ def manifest_rows(
*,
database: str,
row_count: int = 2,
rowset_md5: str = "abc",
rowset_md5: str = "a" * 32,
elapsed_ms: float = 4.0,
server_address: str | None = None,
ssl: bool = False,
@ -28,7 +28,23 @@ def manifest_rows(
{"kind": "extensions", "items": [{"name": "plpgsql", "version": "1.0"}]},
{
"kind": "application_roles",
"items": [{"name": "kb_apply", "can_login": True, "superuser": False}],
"items": [
{"name": "kb_apply", "can_login": True, "superuser": False},
{"name": "kb_gate_owner", "can_login": False, "superuser": False},
{"name": "kb_review", "can_login": True, "superuser": False},
],
},
{
"kind": "role_memberships",
"items": [{"role": "kb_gate_owner", "member": "kb_apply", "admin_option": False}],
},
{
"kind": "object_ownership",
"items": [{"object_type": "schema", "schema": "public", "name": "public", "owner": "kb_gate_owner"}],
},
{
"kind": "object_acl",
"items": [{"object_type": "schema", "name": "public", "grantee": "kb_apply", "privilege_type": "USAGE"}],
},
{"kind": "columns", "items": [{"schema": "public", "table": "claims", "name": "id"}]},
{"kind": "constraints", "items": [{"schema": "public", "table": "claims", "name": "claims_pkey"}]},
@ -90,17 +106,59 @@ def test_matching_local_manifests_pass(tmp_path: Path) -> None:
assert completed.returncode == 0, completed.stderr
assert payload["status"] == "pass"
assert payload["details"]["source_total_rows"] == 2
assert len(payload["source_manifest_sha256"]) == 64
assert len(payload["target_manifest_sha256"]) == 64
assert payload["private_connectivity"]["status"] == "not_applicable_local_restore"
assert (tmp_path / "result.json").stat().st_mode & 0o777 == 0o600
assert '"source_database"' not in completed.stdout
assert '"output_sha256"' in completed.stdout
def test_row_hash_and_role_mismatches_fail(tmp_path: Path) -> None:
target = manifest_rows(database="teleo_copy", row_count=1, rowset_md5="different")
target = manifest_rows(database="teleo_copy", row_count=1, rowset_md5="b" * 32)
target[3]["items"] = []
completed, payload = run_verifier(tmp_path, manifest_rows(database="teleo"), target)
assert completed.returncode == 1
assert "table_content_mismatches=1" in payload["problems"]
assert "application_role_mismatches=3" in payload["problems"]
def test_missing_row_hash_and_extra_roles_or_extensions_fail(tmp_path: Path) -> None:
target = manifest_rows(database="teleo_copy", rowset_md5="")
target[2]["items"].append({"name": "unreviewed", "version": "1.0"})
target[3]["items"].append({"name": "project_owner", "can_login": True, "superuser": True})
completed, payload = run_verifier(tmp_path, manifest_rows(database="teleo"), target)
assert completed.returncode == 1
assert "table_hash_problems=1" in payload["problems"]
assert "extension_mismatches=1" in payload["problems"]
assert "application_role_mismatches=1" in payload["problems"]
assert payload["details"]["target_extra_extensions"] == ["unreviewed"]
assert payload["details"]["target_extra_application_roles"] == ["project_owner"]
def test_matching_malformed_row_hashes_fail(tmp_path: Path) -> None:
completed, payload = run_verifier(
tmp_path,
manifest_rows(database="teleo", rowset_md5="not-a-hash"),
manifest_rows(database="teleo_copy", rowset_md5="not-a-hash"),
)
assert completed.returncode == 1
assert "table_hash_problems=2" in payload["problems"]
assert payload["details"]["table_mismatches"] == []
def test_authorization_ownership_or_acl_drift_fails(tmp_path: Path) -> None:
target = manifest_rows(database="teleo_copy")
target[5]["items"][0]["owner"] = "postgres"
completed, payload = run_verifier(tmp_path, manifest_rows(database="teleo"), target)
assert completed.returncode == 1
assert "object_ownership_hash_mismatch" in payload["problems"]
def test_gcp_scope_requires_private_connectivity_receipt(tmp_path: Path) -> None:
@ -129,12 +187,17 @@ def test_gcp_scope_accepts_matching_private_connectivity_receipt(tmp_path: Path)
"target_database": "teleo_gcp_copy",
"server_address": "10.61.0.3",
"ssl": True,
"sslmode": "verify-full",
"ssl_server_name": "teleo-pgvector-standby.private.teleo.internal",
"ssl_root_cert_sha256": "a" * 64,
"server_identity_verified": True,
},
)
assert completed.returncode == 0, completed.stderr
assert payload["status"] == "pass"
assert payload["private_connectivity"]["proof"]["source_compute"] == "teleo-staging-1"
assert len(payload["connectivity_proof_sha256"]) == 64
def test_gcp_scope_rejects_receipt_for_a_different_server(tmp_path: Path) -> None:
@ -151,6 +214,10 @@ def test_gcp_scope_rejects_receipt_for_a_different_server(tmp_path: Path) -> Non
"target_database": "teleo_gcp_copy",
"server_address": "10.61.0.4",
"ssl": True,
"sslmode": "verify-full",
"ssl_server_name": "teleo-pgvector-standby.private.teleo.internal",
"ssl_root_cert_sha256": "a" * 64,
"server_identity_verified": True,
},
)
@ -172,6 +239,10 @@ def test_gcp_scope_rejects_public_or_non_tls_target_manifest(tmp_path: Path) ->
"target_database": "teleo_gcp_copy",
"server_address": "34.1.2.3",
"ssl": False,
"sslmode": "verify-full",
"ssl_server_name": "teleo-pgvector-standby.private.teleo.internal",
"ssl_root_cert_sha256": "a" * 64,
"server_identity_verified": True,
},
)

308
tests/test_verify_vps_canonical_snapshot_delta.py Normal file
View file

@ -0,0 +1,308 @@
import hashlib
import json
from pathlib import Path
import pytest
from ops.capture_vps_canonical_postgres_snapshot import (
SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
build_provenance_binding,
)
from ops.verify_postgres_parity_manifest import REVIEWED_MANIFEST_SQL_SHA256
from ops.verify_vps_canonical_snapshot_delta import verify_delta, write_private_json
def manifest_text(*, claims: int = 1, constraint_name: str = "claims_pkey") -> str:
rows = [
{
"kind": "identity",
"database": "teleo",
"server_version_num": 160014,
"transaction_read_only": "on",
},
{"kind": "schemas", "items": ["kb_stage", "public"]},
{"kind": "extensions", "items": [{"name": "pgcrypto", "version": "1.3"}]},
{
"kind": "application_roles",
"items": [
{"name": "kb_apply", "can_login": True},
{"name": "kb_gate_owner", "can_login": False},
{"name": "kb_review", "can_login": True},
],
},
{"kind": "role_memberships", "items": []},
{"kind": "object_ownership", "items": []},
{"kind": "object_acl", "items": []},
{"kind": "columns", "items": []},
{"kind": "constraints", "items": [{"name": constraint_name}]},
{"kind": "indexes", "items": []},
{"kind": "sequences", "items": []},
{"kind": "views", "items": []},
{"kind": "functions", "items": []},
{"kind": "triggers", "items": []},
{"kind": "types", "items": []},
{"kind": "policies", "items": []},
{
"kind": "table",
"schema": "public",
"table": "claims",
"row_count": claims,
"rowset_md5": hashlib.md5(str(claims).encode(), usedforsecurity=False).hexdigest(),
},
{"kind": "performance", "query": "count_claims", "elapsed_ms": 1.0, "observed_rows": claims},
]
return "".join(json.dumps(row, sort_keys=True) + "\n" for row in rows)
def capture_fixture(
root: Path,
*,
name: str,
captured_at: str,
authorization_ref: str,
claims: int = 1,
constraint_name: str = "claims_pkey",
) -> tuple[Path, Path]:
manifest = root / f"{name}-manifest.jsonl"
manifest.write_text(manifest_text(claims=claims, constraint_name=constraint_name), encoding="utf-8")
dump_sha256 = hashlib.sha256(f"{name}-dump".encode()).hexdigest()
manifest_sha256 = hashlib.sha256(manifest.read_bytes()).hexdigest()
manifest_sql_sha256 = REVIEWED_MANIFEST_SQL_SHA256
source_context_sha256 = "3" * 64
source = {
"ssh_target": "root@77.42.65.182",
"container": "teleo-pg",
"database": "teleo",
"service": "leoclean-gateway.service",
}
healthy_service = {
"ActiveState": "active",
"SubState": "running",
"MainPID": "148735",
"NRestarts": "0",
}
snapshot = {
"exported_snapshot_id": "00000003-000001A2-1",
"txid_snapshot": "100:120:105,111",
"wal_lsn": "0/16B6C50",
"system_identifier": "7649789040005668902",
"captured_at_utc": captured_at,
}
receipt = {
"artifact": "vps_canonical_postgres_exported_snapshot",
"schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"receipt_version": 2,
"status": "pass",
"run_id": name,
"capture_authorization_ref": authorization_ref,
"source": source,
"snapshot_exported": True,
"snapshot": snapshot,
"dump": {"sha256": dump_sha256},
"manifest": {
"sha256": manifest_sha256,
"manifest_sql_sha256": manifest_sql_sha256,
"table_count": 1,
"total_rows": claims,
},
"source_context": {"sha256": source_context_sha256},
"provenance_binding": build_provenance_binding(
run_id=name,
authorization_ref=authorization_ref,
source=source,
snapshot=snapshot,
dump_sha256=dump_sha256,
manifest_sql_sha256=manifest_sql_sha256,
source_manifest_sha256=manifest_sha256,
source_context_sha256=source_context_sha256,
),
"service_unchanged": True,
"source_service": {"before": healthy_service, "after": healthy_service, "unchanged": True},
"production_db_mutated": False,
"telegram_send_attempted": False,
}
receipt_path = root / f"{name}-receipt.json"
receipt_path.write_text(json.dumps(receipt, sort_keys=True) + "\n", encoding="utf-8")
return receipt_path, manifest
def test_postflight_no_delta_is_bound_and_explicit(tmp_path: Path) -> None:
baseline_receipt, baseline_manifest = capture_fixture(
tmp_path,
name="canonical-baseline",
captured_at="2026-07-15T01:00:00+00:00",
authorization_ref="codex-delegation:baseline123",
)
current_receipt, current_manifest = capture_fixture(
tmp_path,
name="canonical-postflight",
captured_at="2026-07-15T02:00:00+00:00",
authorization_ref="codex-delegation:postflight123",
)
result = verify_delta(
baseline_receipt_path=baseline_receipt,
baseline_manifest_path=baseline_manifest,
baseline_authorization_ref="codex-delegation:baseline123",
current_receipt_path=current_receipt,
current_manifest_path=current_manifest,
current_authorization_ref="codex-delegation:postflight123",
)
assert result["status"] == "pass"
assert result["delta"]["delta_detected"] is False
assert result["delta"]["total_row_delta"] == 0
def test_postflight_data_and_structure_delta_is_reported_not_hidden(tmp_path: Path) -> None:
baseline_receipt, baseline_manifest = capture_fixture(
tmp_path,
name="canonical-baseline",
captured_at="2026-07-15T01:00:00+00:00",
authorization_ref="codex-delegation:baseline123",
)
current_receipt, current_manifest = capture_fixture(
tmp_path,
name="canonical-postflight",
captured_at="2026-07-15T02:00:00+00:00",
authorization_ref="codex-delegation:postflight123",
claims=2,
constraint_name="claims_pkey_v2",
)
result = verify_delta(
baseline_receipt_path=baseline_receipt,
baseline_manifest_path=baseline_manifest,
baseline_authorization_ref="codex-delegation:baseline123",
current_receipt_path=current_receipt,
current_manifest_path=current_manifest,
current_authorization_ref="codex-delegation:postflight123",
)
assert result["status"] == "pass"
assert result["delta"]["delta_detected"] is True
assert result["delta"]["total_row_delta"] == 1
assert result["delta"]["changed_tables"][0]["table"] == "public.claims"
assert result["delta"]["changed_structures"][0]["kind"] == "constraints"
def test_postflight_rejects_wrong_authorization_or_stale_capture(tmp_path: Path) -> None:
baseline_receipt, baseline_manifest = capture_fixture(
tmp_path,
name="canonical-baseline",
captured_at="2026-07-15T02:00:00+00:00",
authorization_ref="codex-delegation:baseline123",
)
current_receipt, current_manifest = capture_fixture(
tmp_path,
name="canonical-postflight",
captured_at="2026-07-15T01:00:00+00:00",
authorization_ref="codex-delegation:postflight123",
)
with pytest.raises(ValueError, match="newer than"):
verify_delta(
baseline_receipt_path=baseline_receipt,
baseline_manifest_path=baseline_manifest,
baseline_authorization_ref="codex-delegation:baseline123",
current_receipt_path=current_receipt,
current_manifest_path=current_manifest,
current_authorization_ref="codex-delegation:postflight123",
)
with pytest.raises(ValueError, match="authorization"):
verify_delta(
baseline_receipt_path=baseline_receipt,
baseline_manifest_path=baseline_manifest,
baseline_authorization_ref="codex-delegation:wrong123",
current_receipt_path=current_receipt,
current_manifest_path=current_manifest,
current_authorization_ref="codex-delegation:postflight123",
)
def test_postflight_rejects_unchanged_but_unhealthy_service(tmp_path: Path) -> None:
baseline_receipt, baseline_manifest = capture_fixture(
tmp_path,
name="canonical-baseline",
captured_at="2026-07-15T01:00:00+00:00",
authorization_ref="codex-delegation:baseline123",
)
current_receipt, current_manifest = capture_fixture(
tmp_path,
name="canonical-postflight",
captured_at="2026-07-15T02:00:00+00:00",
authorization_ref="codex-delegation:postflight123",
)
current = json.loads(current_receipt.read_text())
current["source_service"]["before"]["ActiveState"] = "failed"
current["source_service"]["after"]["ActiveState"] = "failed"
current_receipt.write_text(json.dumps(current, sort_keys=True) + "\n", encoding="utf-8")
with pytest.raises(ValueError, match="source_service_healthy"):
verify_delta(
baseline_receipt_path=baseline_receipt,
baseline_manifest_path=baseline_manifest,
baseline_authorization_ref="codex-delegation:baseline123",
current_receipt_path=current_receipt,
current_manifest_path=current_manifest,
current_authorization_ref="codex-delegation:postflight123",
)
def test_postflight_rejects_a_different_manifest_algorithm_even_when_rebound(tmp_path: Path) -> None:
baseline_receipt, baseline_manifest = capture_fixture(
tmp_path,
name="canonical-baseline",
captured_at="2026-07-15T01:00:00+00:00",
authorization_ref="codex-delegation:baseline123",
)
current_receipt, current_manifest = capture_fixture(
tmp_path,
name="canonical-postflight",
captured_at="2026-07-15T02:00:00+00:00",
authorization_ref="codex-delegation:postflight123",
)
current = json.loads(current_receipt.read_text())
current["manifest"]["manifest_sql_sha256"] = "f" * 64
current["provenance_binding"] = build_provenance_binding(
run_id=current["run_id"],
authorization_ref=current["capture_authorization_ref"],
source=current["source"],
snapshot=current["snapshot"],
dump_sha256=current["dump"]["sha256"],
manifest_sql_sha256="f" * 64,
source_manifest_sha256=current["manifest"]["sha256"],
source_context_sha256=current["source_context"]["sha256"],
)
current_receipt.write_text(json.dumps(current, sort_keys=True) + "\n", encoding="utf-8")
with pytest.raises(ValueError, match="manifest_sql_reviewed"):
verify_delta(
baseline_receipt_path=baseline_receipt,
baseline_manifest_path=baseline_manifest,
baseline_authorization_ref="codex-delegation:baseline123",
current_receipt_path=current_receipt,
current_manifest_path=current_manifest,
current_authorization_ref="codex-delegation:postflight123",
)
def test_delta_receipt_writer_keeps_private_mode(tmp_path: Path) -> None:
output = tmp_path / "private" / "delta.json"
write_private_json(output, {"status": "pass"})
assert output.stat().st_mode & 0o777 == 0o600
assert output.parent.stat().st_mode & 0o777 == 0o700
def test_delta_receipt_writer_does_not_chmod_an_existing_parent(tmp_path: Path) -> None:
parent = tmp_path / "existing"
parent.mkdir(mode=0o755)
output = parent / "delta.json"
write_private_json(output, {"status": "pass"})
assert parent.stat().st_mode & 0o777 == 0o755
assert output.stat().st_mode & 0o777 == 0o600