diff --git a/.agents/skills/teleo-gcp-parity-ops/SKILL.md b/.agents/skills/teleo-gcp-parity-ops/SKILL.md index 5cd671d..4b67a39 100644 --- a/.agents/skills/teleo-gcp-parity-ops/SKILL.md +++ b/.agents/skills/teleo-gcp-parity-ops/SKILL.md @@ -8,45 +8,61 @@ description: Use for passwordless Teleo GCP VM access, private Cloud SQL canonic ## Working Target Restore a copy of the VPS canonical Leo database to GCP, prove exact schema, -row, role, extension, performance, and private-connectivity parity, then run the +row, role membership, ownership, ACL, extension, performance, and +private-connectivity parity, then run the real GCP Leo read/reasoning path without Telegram sends or DB mutation. +The required tier is T3: a live private GCP staging restore, readback, +reasoning, compute attestation, and cleanup receipt. Local restore proof or an +access-gate report does not satisfy this tier. + ## Operator Paths -The direct alias is passwordless and was live-verified on 2026-07-14: +The direct alias was passwordless and live-verified on 2026-07-14: ```bash ssh teleo-gcp-staging ``` -It uses `/Users/user/.ssh/google_compute_engine` for the configured operator, +It uses `~/.ssh/google_compute_engine` for the configured operator, disables password and keyboard-interactive authentication, and does not store a -Google password. `sudo -n` also works. The route remains -firewall-source-dependent, so verify `ssh -o BatchMode=yes teleo-gcp-staging true` before a -long run instead of assuming retained access is current. +Google password. That is historical proof, not current reachability. The route +remains firewall-source-dependent. On +2026-07-15, both direct SSH and the private TCP route timed out; verify +`ssh -o BatchMode=yes teleo-gcp-staging true` before a long run and fail closed +if it does not return successfully. The intended secondary path is `.github/workflows/gcp-iap-operator.yml`, using short-lived GitHub OIDC, IAP, OS Login, and fixed reviewed operations. It is -merged but not bootstrapped: workflow run `29208215340` failed at auth with -`invalid_target` because provider `teleo-iap-operator` is absent, disabled, or -deleted. Do not call this path working until a live `status` run passes. +merged but not bootstrapped: the current 2026-07-15 check still lacks the +reviewed operator WIF/IAP authority. Provider `teleo-iap-operator` is absent, +disabled, or deleted (`invalid_target`), and the available artifact identity is intentionally not +a Cloud SQL/Compute operator. Do not call this path working until a live `status` run passes +with the intended operator identity. -Verified target: +Historically verified and intended target; reverify before any new lifecycle: - project `teleo-501523`; - VM `teleo-prod-1` in `europe-west6-a`; - Cloud SQL `teleo-pgvector-standby`, PostgreSQL 16.14; - private endpoint `10.61.0.3:5432`, public IP disabled, TLS required; +- restore identity `sa-teleo-restore-drill@teleo-501523.iam.gserviceaccount.com`; - service `leoclean-gcp-prod-parallel.service`. +Direct libpq connections must use `sslmode=verify-full`, a reviewed DNS name +covered by the server certificate, and a private regular CA file. The private +IP is `hostaddr`, not the certificate identity. Never downgrade to +`sslmode=require` or accept an arbitrary `*.iam.gserviceaccount.com` identity. + Never print the Cloud SQL or Google password. On the VM, use the attached service account and Secret Manager through reviewed wrappers or a short-lived environment variable, then unset it. ## Two Different Databases -- Canonical collective knowledge is Cloud SQL database `teleo_canonical`, with - `public.*` and `kb_stage.*` tables. +- Canonical collective knowledge remains VPS PostgreSQL database `teleo`, with + `public.*` and `kb_stage.*` tables. Persistent Cloud SQL database + `teleo_canonical` is a staging copy and has not been promoted. - Hermes conversation continuity is `state.db` plus session JSONL files. The `leoclean-cloudsql-memory-sync.service` snapshot path copies this runtime memory; it does not populate or prove canonical claims, sources, evidence, @@ -54,7 +70,7 @@ variable, then unset it. Do not describe a passing Hermes memory sync as canonical KB parity. -## Current Verified State - 2026-07-14 +## Retained Historical Proof - 2026-07-14 - The newest captured VPS database has `39` tables and `52,167` rows, including claims `1837`, sources `4145`, claim evidence `4670`, claim edges `4916`, and @@ -83,19 +99,25 @@ Do not describe a passing Hermes memory sync as canonical KB parity. - Persistent GCP `teleo_canonical` remains the older staging copy measured at `52,164` rows and `26` proposals. It has not been promoted or cut over. -## Open Least-Privilege Candidate +That historical clone predates the expanded ownership/ACL and independently +authenticated receipt contract. It remains useful historical row/schema proof, +but it is not current T3 proof under this skill. + +## Open Least-Privilege Candidates PR #148, `Scope GCP Leo runtime to least-privilege Cloud SQL access`, is open as -of the 2026-07-15 skill-pack reconciliation. It proposes scoped runtime roles, -secret access, fail-closed Cloud SQL behavior, deployment rollback, and positive -plus negative permission checks. Those branch files and proposed live outcomes -are candidate evidence only. Before using them, run: +of the 2026-07-15 skill-pack reconciliation. Draft PR #162, `Harden GCP Leo +runtime least-privilege proof`, is its current hardening successor. They propose +scoped runtime roles, secret access, fail-closed Cloud SQL behavior, deployment +rollback, and positive plus negative permission checks. Those branch files and +proposed live outcomes are candidate evidence only. Before using them, run: ```bash gh pr view 148 --json state,mergedAt,mergeCommit,headRefName,url +gh pr view 162 --json state,mergedAt,mergeCommit,headRefName,url ``` -Until the PR is merged and its runtime receipt passes, use only paths present on +Until a successor is merged and its runtime receipt passes, use only paths present on canonical `main`, keep persistent GCP classified as staging, do not promote it, and do not infer least-privilege cutover from a branch or dry run. @@ -114,7 +136,8 @@ Track these independently: 1. control-plane project, VM, Cloud SQL, private-IP, and TLS identity; 2. canonical database schema, counts, row hashes, constraints, indexes, - functions, extensions, roles, and performance; + functions, extensions, exact roles and memberships, owners, normalized + schema/relation/column/function/type ACLs, and performance; 3. GCP service PID/restarts plus live profile and tool hashes; 4. `DC-01` through `DC-06` DB-read readiness; 5. real no-send model replies, strict score, and exact count consistency; @@ -126,17 +149,60 @@ Track these independently: Use: - `ops/capture_vps_canonical_postgres_snapshot.py` for a single source dump and - manifest; + manifest, with a required retained `--authorization-ref` and v2 provenance + receipt binding the exported snapshot identity plus dump/manifest hashes; it + requires identical healthy structured source-service states before/after + (`active/running`, positive PID, nonnegative restart count); - `ops/restore_gcp_generated_postgres_snapshot.py restore --execute` for a receipt-bound private-TLS restore into a bounded `teleo_clone_*` database; + pass the exact capture `--source-receipt`, and require its live Cloud SQL + control-plane check to prove public IP disabled before clone creation; it + requires the exact restore service account plus `--ssl-server-name` and + `--ssl-root-cert` for certificate-authenticated `verify-full` TLS; it + writes `target-manifest.jsonl`, `restore-receipt.json`, and the generated + `gcp-private-connectivity.json` under + `/var/lib/teleo-gcp-restore-runs//`; - `ops/restore_gcp_generated_postgres_snapshot.py cleanup --execute` for exact clone cleanup with live-service and rollback readback; - `ops/postgres_parity_manifest.sql` for row/catalog/role/performance readback; -- `ops/verify_postgres_parity_manifest.py --scope gcp_staging` for exact parity; +- `ops/verify_postgres_parity_manifest.py --scope gcp_staging` for exact parity, + always retaining the canonical output name `gcp-parity.json`; - `scripts/run_gcp_generated_db_direct_claim_suite.py` for the adapter-free, read-only, no-send six-response replay against a generated `teleo_clone_*`. - `scripts/run_gcp_generated_db_blind_claim_canary.py` for the bounded ID-free claim challenge, source receipt, reasoning, no-write, and cleanup proof. +- `ops/attest_gcp_reasoning_compute.py` on the replay VM after blind reasoning + and before cleanup, binding the reasoning receipt SHA-256 to live GCE metadata + in `reasoning-compute-attestation.json`; +- a second authorized source capture after cleanup, followed by + `ops/verify_vps_canonical_snapshot_delta.py`, to retain the explicit + post-snapshot VPS delta in `source-delta-receipt.json`; +- `ops/verify_gcp_canonical_lifecycle.py` after cleanup to bind the exact source, + current source, source delta, restore, `gcp_staging` parity, ID-free reasoning, + reasoning-compute attestation, and cleanup receipts into one fail-closed + preparation verdict. Caller-writable JSON can never emit T3 from this API; + even an internally valid bundle returns + `prepared_external_attestation_required`, `current_tier=T2`, and + `accepted_by_this_api=false`. Pass `--max-postflight-age-seconds 900` and + `--max-lifecycle-age-seconds 3600`; the latter bounds both cleanup-to-verdict + age and restore-to-postflight span. + +With `TARGET_DB`, `RESTORE_RUN_ID`, and `PRIVATE_RUN_DIR` set as shown in the +runbook, run the replay-host attestation after the blind receipt passes and +before the receipt-bound cleanup: + +```bash +sudo python3 ops/attest_gcp_reasoning_compute.py \ + --reasoning-receipt "${PRIVATE_RUN_DIR}/blind-reasoning-receipt.json" \ + --restore-run-id "$RESTORE_RUN_ID" \ + --target-db "$TARGET_DB" \ + --project teleo-501523 \ + --expected-compute-instance teleo-prod-1 \ + --expected-compute-zone europe-west6-a \ + --expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \ + --expected-restore-service-account sa-teleo-restore-drill@teleo-501523.iam.gserviceaccount.com \ + --output "${PRIVATE_RUN_DIR}/reasoning-compute-attestation.json" +``` The replay must use the Hermes virtualenv, not system Python: @@ -145,8 +211,24 @@ The replay must use the Hermes virtualenv, not system Python: scripts/run_gcp_generated_db_direct_claim_suite.py ... ``` -The replay is not complete until the generated clone, temporary profile, -children, upload/run directories, and any temporary client are absent. +The lifecycle is not complete until the generated clone, temporary profile, +children, transient upload/bundle directories, and any temporary client are +absent. Preserve the private receipt run directory and every useful lifecycle +input, supporting receipt, and verifier output; do not treat that evidence as +transient cleanup. The canonical restore streams through `pg_restore` and +creates no GCS import object. + +The current snapshot and restore commands intentionally use `--no-owner` and +`--no-acl`/`--no-privileges`. The expanded manifest therefore fails GCP parity +until a separately reviewed Cloud-SQL-compatible authorization replay restores +the captured owner/grant semantics. Do not bypass this by dropping the new +manifest rows or by running the Docker/superuser gate bootstrap blindly on +shared Cloud SQL. + +Static inspection or a dry run is insufficient for these helpers: verify a new +capture against the live VPS and run the focused tests. A current T3 claim also +requires a separate independently authenticated provider/platform receipt that +the bundle author cannot mint. ## Safety And Claim Ceiling @@ -157,18 +239,24 @@ children, upload/run directories, and any temporary client are absent. - Do not call control-plane inventory, memory sync, route readiness, or a nominal scorer pass full m3taversal parity. -The strongest accepted claim requires exact DB parity plus real no-send model -replies with truthful counts and cleanup. It still does not prove Telegram +The strongest claim emitted by the repository lifecycle verifier is T2 +structural consistency. Exact DB parity plus real no-send model replies with +truthful counts and cleanup still requires independent T3 attestation. Neither proves Telegram delivery, GCP canonical mutation, ongoing replication, or production cutover. ## Current Access And Next Action -Direct SSH and passwordless sudo are currently working. Local `gcloud` still -requires account reauthentication for control-plane metadata, so the current -private-IP/TLS proof comes from a live database connection while the public-IP- -disabled control-plane receipt remains dated 2026-07-12. +As of 2026-07-15, direct SSH and private TCP timed out, local `gcloud` for +`billy@livingip.xyz` requires reauthentication, and the intended operator +WIF/IAP authority is unavailable. The artifact-builder WIF identity must not be +expanded or mistaken for database-operator access. Current retained T2/source +and local restore proof does not satisfy the required T3 lifecycle. -The next production decision is not another restore drill. It is whether to -promote a newly verified snapshot into persistent GCP `teleo_canonical` and -repoint the production read adapter. Until that decision is explicit, keep GCP -classified as staging and do not expose Cloud SQL publicly. +The next action after an authorized operator route is restored is: newest +authorized VPS capture -> disposable private `teleo_clone_*` restore -> +generated private-connectivity receipt -> `gcp-parity.json` -> no-send +m3taversal blind reasoning -> live GCE reasoning attestation -> receipt-bound +clone cleanup -> postflight VPS capture/delta -> eight-receipt lifecycle verdict +within the 900/3600-second freshness bounds. Do not promote +`teleo_canonical`, expose Cloud SQL publicly, send Telegram, or mutate canonical +knowledge. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fc748bf..9b70be5 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -42,19 +42,24 @@ jobs: lib/llm.py \ lib/post_extract.py \ ops/apply_gcp_iam_split.py \ + ops/attest_gcp_reasoning_compute.py \ ops/capture_vps_canonical_postgres_snapshot.py \ ops/check_gcp_infra_readiness.py \ ops/run_gcp_infra_execute_canary.py \ ops/apply_gcp_runtime_baseline.py \ ops/check_gcp_service_communications.py \ ops/plan_gcp_iam_split.py \ + ops/private_receipt_io.py \ + ops/run_local_canonical_postgres_rebuild.py \ ops/redact_sqlite_postgres_restore_canary.py \ ops/restore_gcp_generated_postgres_snapshot.py \ ops/sqlite_to_postgres_dump.py \ ops/verify_gcp_cloudsql_restore_readback.py \ ops/verify_gcp_leoclean_runtime_permissions.py \ ops/verify_gcp_leoclean_service_environment.py \ + ops/verify_gcp_canonical_lifecycle.py \ ops/verify_postgres_parity_manifest.py \ + ops/verify_vps_canonical_snapshot_delta.py \ telegram/approvals.py \ hermes-agent/leoclean-bin/kb_tool.py \ hermes-agent/leoclean-bin/cloudsql_memory_tool.py \ @@ -75,6 +80,7 @@ jobs: scripts/working_leo_m3taversal_oos_benchmark.py \ scripts/working_leo_open_ended_benchmark.py \ tests/test_agent_routing.py \ + tests/test_attest_gcp_reasoning_compute.py \ tests/test_assemble_telegram_visible_direct_claim_capture_receipt.py \ tests/test_build_working_leo_m3taversal_outcome_sandbox.py \ tests/test_decision_engine_replay.py \ @@ -102,10 +108,15 @@ jobs: tests/test_verify_leo_db_first_oos_canary.py \ tests/test_compile_kb_source_packet.py \ tests/test_verify_postgres_parity_manifest.py \ + tests/test_verify_gcp_canonical_lifecycle.py \ + tests/test_verify_vps_canonical_snapshot_delta.py \ tests/test_working_leo_m3taversal_oos_benchmark.py \ tests/test_working_leo_open_ended_benchmark.py \ tests/test_phase1b_end_to_end.py \ + tests/test_private_receipt_io.py \ tests/test_restore_gcp_generated_postgres_snapshot.py \ + tests/test_run_local_canonical_postgres_rebuild.py \ + tests/test_run_local_genesis_ledger_rebuild.py \ tests/test_sqlite_to_postgres_dump.py \ tests/test_sqlite_postgres_restore_canary_capsule.py \ tests/test_teleo_agent_systemd.py \ diff --git a/.github/workflows/gcp-observatory-read-adapter.yml b/.github/workflows/gcp-observatory-read-adapter.yml index 95b18a4..bc30517 100644 --- a/.github/workflows/gcp-observatory-read-adapter.yml +++ b/.github/workflows/gcp-observatory-read-adapter.yml @@ -4,12 +4,13 @@ on: workflow_dispatch: inputs: action: - description: Build only, or deploy the protected staging service and run live receipts + description: Build, preflight, or deploy the protected staging service and run live receipts required: true default: build_only type: choice options: - build_only + - preflight_staging - deploy_staging permissions: @@ -28,10 +29,13 @@ env: IMAGE_NAME: observatory-read-adapter SERVICE_NAME: observatory-read-adapter-staging RUNTIME_SERVICE_ACCOUNT: sa-observatory-read-adapter@teleo-501523.iam.gserviceaccount.com + BUILD_SERVICE_ACCOUNT: sa-artifact-builder@teleo-501523.iam.gserviceaccount.com + DEPLOY_SERVICE_ACCOUNT: sa-observatory-deployer@teleo-501523.iam.gserviceaccount.com DB_IAM_USER: sa-observatory-read-adapter@teleo-501523.iam API_KEY_SECRET: observatory-read-api-key-staging - WORKLOAD_IDENTITY_PROVIDER: projects/785938879453/locations/global/workloadIdentityPools/github-actions/providers/living-ip-github - DEPLOY_SERVICE_ACCOUNT: sa-artifact-builder@teleo-501523.iam.gserviceaccount.com + BUILD_WORKLOAD_IDENTITY_PROVIDER: projects/785938879453/locations/global/workloadIdentityPools/github-actions/providers/living-ip-github + DEPLOY_WORKLOAD_IDENTITY_PROVIDER: projects/785938879453/locations/global/workloadIdentityPools/github-actions/providers/observatory-read-adapter-main + EXPECTED_WORKFLOW_REF: living-ip/teleo-infrastructure/.github/workflows/gcp-observatory-read-adapter.yml@refs/heads/main jobs: build: @@ -57,8 +61,8 @@ jobs: - id: auth uses: google-github-actions/auth@v3 with: - workload_identity_provider: ${{ env.WORKLOAD_IDENTITY_PROVIDER }} - service_account: ${{ env.DEPLOY_SERVICE_ACCOUNT }} + workload_identity_provider: ${{ env.BUILD_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ env.BUILD_SERVICE_ACCOUNT }} - uses: google-github-actions/setup-gcloud@v3 with: @@ -68,28 +72,21 @@ jobs: run: gcloud auth configure-docker "${REGION}-docker.pkg.dev" --quiet - id: image - name: Build, smoke, and push immutable image + name: Build, smoke, and push run-unique immutable image shell: bash run: | set -euo pipefail - tag="${GITHUB_SHA::12}" + tag="${GITHUB_SHA}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" image="${REGION}-docker.pkg.dev/${PROJECT_ID}/${REPOSITORY}/${IMAGE_NAME}:${tag}" - digest="$(gcloud artifacts docker images describe "${image}" \ - --project="${PROJECT_ID}" \ - --format='value(image_summary.digest)' 2>/dev/null || true)" - if [[ -n "${digest}" ]]; then - docker pull "${image}" - else - docker build -f Dockerfile.observatory-read-adapter -t "${image}" . - fi + docker build -f Dockerfile.observatory-read-adapter -t "${image}" . docker run --rm --env PYTHONPYCACHEPREFIX=/tmp/pycache \ "${image}" python -m compileall -q /app/observatory_read_adapter docker run --rm "${image}" python -c 'import observatory_read_adapter; print("adapter-import-ok")' - if [[ -z "${digest}" ]]; then - docker push "${image}" | tee docker-push.log - digest="$(awk '/digest: sha256:/ {print $3}' docker-push.log | tail -1)" - fi - test -n "${digest}" + docker push "${image}" + digest="$(gcloud artifacts docker images describe "${image}" \ + --project="${PROJECT_ID}" \ + --format='value(image_summary.digest)')" + [[ "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]] image_ref="${image}@${digest}" printf 'image_ref=%s\n' "${image_ref}" >> "${GITHUB_OUTPUT}" printf 'revision=%s\nimage_ref=%s\n' "${GITHUB_SHA}" "${image_ref}" > observatory-adapter-image.txt @@ -100,31 +97,94 @@ jobs: path: observatory-adapter-image.txt if-no-files-found: error - deploy-staging: - name: Deploy and prove staging read path - if: ${{ inputs.action == 'deploy_staging' }} + preflight-staging: + name: Verify private staging prerequisites + if: ${{ inputs.action != 'build_only' }} needs: build runs-on: ubuntu-latest - timeout-minutes: 20 + timeout-minutes: 10 steps: - uses: actions/checkout@v4 + - name: Enforce reviewed main-only staging path + shell: bash + run: | + set -euo pipefail + [[ "${GITHUB_REF}" == "refs/heads/main" ]] + [[ "${GITHUB_WORKFLOW_REF}" == "${EXPECTED_WORKFLOW_REF}" ]] + [[ "${GITHUB_EVENT_NAME}" == "workflow_dispatch" ]] + - id: auth uses: google-github-actions/auth@v3 with: - workload_identity_provider: ${{ env.WORKLOAD_IDENTITY_PROVIDER }} + workload_identity_provider: ${{ env.DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} service_account: ${{ env.DEPLOY_SERVICE_ACCOUNT }} - uses: google-github-actions/setup-gcloud@v3 with: project_id: ${{ env.PROJECT_ID }} - - name: Deploy bounded GCP staging service + - name: Run adapter-specific read-only GCP preflight shell: bash env: IMAGE_REF: ${{ needs.build.outputs.image_ref }} run: | set -euo pipefail + python3 ops/check_observatory_read_adapter_gcp_preflight.py \ + --image-ref "${IMAGE_REF}" \ + --output observatory-read-adapter-preflight.json + + - uses: actions/upload-artifact@v4 + if: always() + with: + name: observatory-read-adapter-preflight + path: observatory-read-adapter-preflight.json + if-no-files-found: error + + deploy-staging: + name: Deploy and prove staging read path + if: ${{ inputs.action == 'deploy_staging' }} + needs: + - build + - preflight-staging + runs-on: ubuntu-latest + timeout-minutes: 20 + steps: + - uses: actions/checkout@v4 + + - name: Enforce reviewed main-only staging path + shell: bash + run: | + set -euo pipefail + [[ "${GITHUB_REF}" == "refs/heads/main" ]] + [[ "${GITHUB_WORKFLOW_REF}" == "${EXPECTED_WORKFLOW_REF}" ]] + [[ "${GITHUB_EVENT_NAME}" == "workflow_dispatch" ]] + + - id: auth + uses: google-github-actions/auth@v3 + with: + workload_identity_provider: ${{ env.DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ env.DEPLOY_SERVICE_ACCOUNT }} + + - uses: google-github-actions/setup-gcloud@v3 + with: + project_id: ${{ env.PROJECT_ID }} + + - id: deploy + name: Deploy bounded GCP staging service + shell: bash + env: + IMAGE_REF: ${{ needs.build.outputs.image_ref }} + run: | + set -euo pipefail + api_key_version="$(gcloud secrets versions list "${API_KEY_SECRET}" \ + --project="${PROJECT_ID}" \ + --filter='state=ENABLED' \ + --sort-by='~createTime' \ + --limit=1 \ + --format='value(name)')" + api_key_version="${api_key_version##*/}" + [[ "${api_key_version}" =~ ^[0-9]+$ ]] gcloud run deploy "${SERVICE_NAME}" \ --project="${PROJECT_ID}" \ --region="${REGION}" \ @@ -135,39 +195,69 @@ jobs: --subnet=teleo-staging-europe-west6 \ --vpc-egress=private-ranges-only \ --ingress=all \ - --allow-unauthenticated \ + --no-invoker-iam-check \ --port=8080 \ --cpu=1 \ --memory=512Mi \ --concurrency=8 \ --max-instances=2 \ --timeout=15 \ - --set-secrets="OBSERVATORY_API_KEY=${API_KEY_SECRET}:latest" \ + --set-secrets="OBSERVATORY_API_KEY=${API_KEY_SECRET}:${api_key_version}" \ --set-env-vars="GCP_PROJECT_ID=${PROJECT_ID},CLOUD_SQL_INSTANCE=${PROJECT_ID}:${REGION}:teleo-pgvector-standby,DB_NAME=teleo_canonical,DB_IAM_USER=${DB_IAM_USER},DB_AUTHORIZATION_ROLE=kb_observatory_read,SERVICE_REVISION=${GITHUB_SHA}" \ --labels="livingip-tier=gcp-staging,livingip-surface=observatory-read-adapter" \ --quiet + printf 'api_key_version=%s\n' "${api_key_version}" >> "${GITHUB_OUTPUT}" - name: Capture positive and negative live receipts shell: bash + env: + API_KEY_VERSION: ${{ steps.deploy.outputs.api_key_version }} + EXPECTED_IMAGE_REF: ${{ needs.build.outputs.image_ref }} run: | set -euo pipefail - service_url="$(gcloud run services describe "${SERVICE_NAME}" --project="${PROJECT_ID}" --region="${REGION}" --format='value(status.url)')" - revision="$(gcloud run services describe "${SERVICE_NAME}" --project="${PROJECT_ID}" --region="${REGION}" --format='value(status.latestReadyRevisionName)')" - api_key="$(gcloud secrets versions access latest --secret="${API_KEY_SECRET}" --project="${PROJECT_ID}")" + service_json="$(gcloud run services describe "${SERVICE_NAME}" --project="${PROJECT_ID}" --region="${REGION}" --format=json)" + service_url="$(jq -r '.status.url // empty' <<<"${service_json}")" + revision="$(jq -r '.status.latestReadyRevisionName // empty' <<<"${service_json}")" + runtime_service_account="$(jq -r '.spec.template.spec.serviceAccountName // .template.serviceAccount // empty' <<<"${service_json}")" + deployed_image="$(jq -r '.spec.template.spec.containers[0].image // .template.containers[0].image // empty' <<<"${service_json}")" + deployed_secret_version="$(jq -r ' + [ + .spec.template.spec.containers[]?.env[]? + | select(.name == "OBSERVATORY_API_KEY") + | .valueFrom.secretKeyRef.key + ][0] // [ + .template.containers[]?.env[]? + | select(.name == "OBSERVATORY_API_KEY") + | .valueSource.secretKeyRef.version + ][0] // empty + ' <<<"${service_json}")" + expected_digest="${EXPECTED_IMAGE_REF##*@}" + api_key="$(gcloud secrets versions access "${API_KEY_VERSION}" --secret="${API_KEY_SECRET}" --project="${PROJECT_ID}")" test -n "${service_url}" test -n "${revision}" + test "${runtime_service_account}" = "${RUNTIME_SERVICE_ACCOUNT}" + test "${deployed_secret_version}" = "${API_KEY_VERSION}" + [[ "${deployed_image}" == "${REGION}-docker.pkg.dev/${PROJECT_ID}/${REPOSITORY}/${IMAGE_NAME}@${expected_digest}" ]] test -n "${api_key}" curl --fail --silent --show-error \ --header "X-Api-Key: ${api_key}" \ "${service_url}/v1/claims/sample" > positive.json - jq -e ' + jq -e \ + --arg project_id "${PROJECT_ID}" \ + --arg cloud_sql_instance "${PROJECT_ID}:${REGION}:teleo-pgvector-standby" \ + --arg db_iam_user "${DB_IAM_USER}" \ + --arg git_sha "${GITHUB_SHA}" ' .schema == "livingip.observatory-canonical-claim.v1" and .read_only == true + and .provenance.gcp_project == $project_id + and .provenance.cloud_sql_instance == $cloud_sql_instance and .provenance.database == "teleo_canonical" + and .provenance.database_principal == $db_iam_user and .provenance.authorization_role == "kb_observatory_read" and .provenance.transaction_read_only == true and .provenance.write_privileges_denied == true + and .provenance.service_revision == $git_sha and (.canonical.evidence | length) > 0 and .proposal_ledger.distinct_from_canonical == true ' positive.json >/dev/null @@ -178,12 +268,23 @@ jobs: --data '{"status":"applied"}' "${service_url}/v1/claims/sample")" test "${anonymous_status}" = 401 test "${write_status}" = 405 + curl --fail --silent --show-error \ + --header "X-Api-Key: ${api_key}" \ + "${service_url}/v1/claims/sample" > after-write-attempt.json + cmp <(jq -S . positive.json) <(jq -S . after-write-attempt.json) + + jq -f ops/redact_observatory_read_adapter_receipt.jq \ + positive.json > positive-redacted.json jq -n \ --arg service_url "${service_url}" \ --arg revision "${revision}" \ --arg git_sha "${GITHUB_SHA}" \ - --argjson positive "$(cat positive.json)" \ + --arg runtime_service_account "${runtime_service_account}" \ + --arg database_principal "${DB_IAM_USER}" \ + --arg deployed_image "${deployed_image}" \ + --arg api_key_secret_version "${API_KEY_VERSION}" \ + --argjson positive "$(cat positive-redacted.json)" \ --arg anonymous_status "${anonymous_status}" \ --arg write_status "${write_status}" \ '{ @@ -192,14 +293,21 @@ jobs: service_url: $service_url, revision: $revision, git_sha: $git_sha, + runtime_service_account: $runtime_service_account, + database_principal: $database_principal, + deployed_image: $deployed_image, + api_key_secret_version: $api_key_secret_version, positive: $positive, negative: { anonymous_http_status: ($anonymous_status | tonumber), - authenticated_post_http_status: ($write_status | tonumber) + authenticated_post_http_status: ($write_status | tonumber), + canonical_state_unchanged_after_post: true, + database_write_privileges_denied: $positive.provenance.write_privileges_denied }, production_repoint_executed: false }' > observatory-read-adapter-live-receipt.json unset api_key + rm -f positive.json positive-redacted.json after-write-attempt.json anonymous.json write.json - uses: actions/upload-artifact@v4 with: diff --git a/docs/gcp-kb-cloudsql-restore-runbook.md b/docs/gcp-kb-cloudsql-restore-runbook.md index 2015807..2a0132b 100644 --- a/docs/gcp-kb-cloudsql-restore-runbook.md +++ b/docs/gcp-kb-cloudsql-restore-runbook.md @@ -53,12 +53,21 @@ python3 ops/capture_vps_canonical_postgres_snapshot.py \ --ssh-target root@77.42.65.182 \ --ssh-key ~/.ssh/livingip_hetzner_20260604_ed25519 \ --run-id canonical- \ + --authorization-ref codex-delegation:THREAD_ID \ --output-dir ``` -The capture fails closed if the source service changes while it runs. It +Replace `THREAD_ID` with the exact retained authorization reference for the +capture. The v2 receipt binds that reference, exported snapshot ID, TXID +snapshot, WAL LSN, source system identifier, dump hash, manifest hash, reviewed +manifest-SQL hash, and source-context hash. A dump and manifest without this +passing receipt are not eligible for GCP restore. + +The capture fails closed unless the structured source-service states before +and after the snapshot are identical and healthy: `ActiveState=active`, +`SubState=running`, a positive `MainPID`, and a nonnegative `NRestarts`. It retains a private custom dump, dump SHA-256, object TOC, catalog/data manifest, -and before/after service state. It never restarts Leo or writes to the source +and both service-state objects. It never restarts Leo or writes to the source database. Prove that this exact snapshot can rebuild a blank Postgres target before using @@ -77,35 +86,207 @@ network mode `none` and tmpfs-only database storage. It waits for an actual --no-owner --no-privileges --exit-on-error`, compares the full parity manifest, then removes the container and proves it is absent. A passing local receipt is the exact-recovery preflight; it is not semantic recompilation from raw source -documents. +documents. Full parity now includes `kb_gate_owner`, `kb_apply`, and +`kb_review`, their memberships, object ownership, and normalized database, +schema, relation, column, function, and type ACLs. A restore that recovers rows +but not those authorization semantics fails closed. -Run `ops/postgres_parity_manifest.sql` against the isolated restored target, -then compare source and target: +Choose one bounded suffix and use it consistently on the GCP replay VM. The +restore helper retains its mode-0700 evidence directory at the fixed private +root: ```bash -python3 ops/verify_postgres_parity_manifest.py \ - --source /source-manifest.jsonl \ - --target /target-manifest.jsonl \ - --scope gcp_staging \ - --connectivity-proof /gcp-private-connectivity.json \ - --output /gcp-parity.json +RUN_SUFFIX=20260715t010000z +TARGET_DB="teleo_clone_${RUN_SUFFIX}" +RESTORE_RUN_ID="gcp-restore-${RUN_SUFFIX}" +PRIVATE_RUN_DIR="/var/lib/teleo-gcp-restore-runs/${RESTORE_RUN_ID}" ``` -The verifier checks all table row counts and collation-independent row hashes, -plus schemas, columns/defaults, constraints, indexes, sequences, views, -functions, triggers, enum/domain types, policies, required extensions, -password-free application-role attributes, and bounded query timings. In GCP -scope it also requires a receipt proving a staging compute source, a private -server address, TLS, and public-IP-disabled instance metadata. - Use a generated target database such as `teleo_clone_`. Never import a drill into `teleo`, `teleo_kb`, or `teleo_canonical`. Database isolation does not isolate cluster-global roles or extensions, so verify those separately and do not run the Docker-only gate bootstrap against the shared Cloud SQL instance. -After the parity verifier passes, run the no-send operator composition replay from -staging compute against that generated database. Only then delete the generated -database and uploaded import object and retain cleanup readback. +The canonical restore command must include the v2 capture receipt. It also runs +a live Cloud SQL control-plane preflight before resolving the database secret or +creating the clone, and fails closed unless the exact instance is runnable, +PostgreSQL 16, private-networked, bound to the expected RFC1918 host, and has +public IP disabled. A GCE metadata preflight independently binds the operator +runtime to `teleo-prod-1` in `europe-west6-a` on `teleo-staging-net`; a +hard-coded compute label is not accepted. It also requires the exact +least-privilege restore identity and libpq `verify-full` using a reviewed server +name and private CA file. Determine the certificate DNS SAN from the live Cloud +SQL certificate before filling `--ssl-server-name`; do not substitute the +private IP or downgrade TLS verification: + +```bash +sudo python3 ops/restore_gcp_generated_postgres_snapshot.py restore \ + --execute \ + --target-db "$TARGET_DB" \ + --run-id "$RESTORE_RUN_ID" \ + --dump /teleo-canonical.dump \ + --expected-dump-sha256 \ + --source-manifest /source-manifest.jsonl \ + --source-receipt /receipt.json \ + --expected-source-receipt-sha256 \ + --expected-capture-authorization-ref codex-delegation:THREAD_ID \ + --project teleo-501523 \ + --cloudsql-instance teleo-pgvector-standby \ + --expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \ + --expected-compute-instance teleo-prod-1 \ + --expected-compute-zone europe-west6-a \ + --expected-restore-service-account sa-teleo-restore-drill@teleo-501523.iam.gserviceaccount.com \ + --ssl-server-name \ + --ssl-root-cert /etc/teleo/cloudsql/server-ca.pem \ + --expected-source-ssh-target root@77.42.65.182 \ + --expected-source-container teleo-pg \ + --expected-source-database teleo \ + --expected-source-service leoclean-gateway.service +``` + +The restore writes both `target-manifest.jsonl` and +`gcp-private-connectivity.json` as mode-0600 files in its private run +directory. The latter is composed from live GCE metadata, Cloud SQL +control-plane checks, and the private TLS database identity; do not hand-compose +the connectivity receipt. The restore also records identical healthy +before/after state for `leoclean-gcp-prod-parallel.service`, including its PID +and restart count. Receipt payloads remain in mode-0600 files; command stdout +contains only a status, output path, and receipt hash. + +Compare the captured target manifest and generated connectivity receipt to the +authorized source: + +```bash +sudo python3 ops/verify_postgres_parity_manifest.py \ + --source /source-manifest.jsonl \ + --target "$PRIVATE_RUN_DIR/target-manifest.jsonl" \ + --scope gcp_staging \ + --connectivity-proof "$PRIVATE_RUN_DIR/gcp-private-connectivity.json" \ + --output "$PRIVATE_RUN_DIR/gcp-parity.json" +``` + +The verifier checks all table row counts and collation-independent row hashes, +plus schemas, columns/defaults, constraints, indexes, sequences, views, +functions, triggers, enum/domain types, policies, required extensions, +password-free application-role attributes, exact role memberships, object +owners, normalized ACLs, and bounded query timings. Role and extension sets are +symmetric: an unexpected extra role, superuser, or extension is a mismatch. +Every table must have a nonempty row hash. In GCP scope it also requires a +receipt proving a staging compute source, a private server address, +certificate-authenticated `verify-full` TLS, and public-IP-disabled instance +metadata. + +The snapshot dump intentionally strips owners and ACLs. Therefore the GCP +restore cannot pass the expanded parity gate until a separately reviewed, +Cloud-SQL-compatible authorization replay reconstructs the captured ownership +and grants. Do not run the Docker/superuser bootstrap blindly on shared Cloud +SQL. Until that replay exists, this lane remains T2 preparation and must fail +before a T3 claim. + +After the parity verifier passes, run the no-send m3taversal blind reasoning +replay from staging compute against that generated database, attest the replay +host from live GCE metadata, and only then delete the generated database. This +canonical path streams the custom dump through `pg_restore`; it does not create +a GCS import object. + +Run the bounded ID-free reasoning canary with the generated parity receipt. The +script re-executes itself in the Hermes virtualenv when that runtime is present: + +```bash +sudo python3 scripts/run_gcp_generated_db_blind_claim_canary.py \ + --execute \ + --target-db "$TARGET_DB" \ + --cloudsql-tool hermes-agent/leoclean-bin/cloudsql_memory_tool.py \ + --manifest-sql ops/postgres_parity_manifest.sql \ + --parity-receipt "$PRIVATE_RUN_DIR/gcp-parity.json" \ + --output "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \ + --run-id "gcp-blind-claim-${RUN_SUFFIX}" +``` + +Before cleanup, bind that reasoning receipt to live metadata from the same GCE +replay host: + +```bash +sudo python3 ops/attest_gcp_reasoning_compute.py \ + --reasoning-receipt "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \ + --restore-run-id "$RESTORE_RUN_ID" \ + --target-db "$TARGET_DB" \ + --project teleo-501523 \ + --expected-compute-instance teleo-prod-1 \ + --expected-compute-zone europe-west6-a \ + --expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \ + --expected-restore-service-account sa-teleo-restore-drill@teleo-501523.iam.gserviceaccount.com \ + --output "$PRIVATE_RUN_DIR/reasoning-compute-attestation.json" +``` + +After the compute attestation passes, run the receipt-bound cleanup command +retained by the restore. Then capture the VPS source again under a distinct +postflight authorization reference. Run this capture from the same trusted +source-access lane used for the baseline; it need not run on the GCP replay VM. +The postflight capture must complete after cleanup, so the final verdict can +state whether the VPS changed after the restored snapshot boundary. Compare the +two source captures: + +```bash +python3 ops/capture_vps_canonical_postgres_snapshot.py \ + --execute \ + --ssh-target root@77.42.65.182 \ + --ssh-key ~/.ssh/livingip_hetzner_20260604_ed25519 \ + --run-id canonical-${RUN_SUFFIX}-postflight \ + --authorization-ref codex-delegation:THREAD_ID:postflight \ + --output-dir +``` + +```bash +sudo python3 ops/verify_vps_canonical_snapshot_delta.py \ + --baseline-receipt /receipt.json \ + --baseline-manifest /source-manifest.jsonl \ + --baseline-authorization-ref codex-delegation:THREAD_ID \ + --current-receipt /receipt.json \ + --current-manifest /source-manifest.jsonl \ + --current-authorization-ref codex-delegation:THREAD_ID:postflight \ + --output "$PRIVATE_RUN_DIR/source-delta-receipt.json" +``` + +Finally, verify that all eight receipts belong to one bound lifecycle. This +verifier rejects local-scope parity, stale or mismatched source hashes, a +postflight source capture that predates reasoning or cleanup, hand-authored +reasoning booleans without real retrieval/tool receipts, an unattested replay +host, compute or Cloud SQL topology drift, and any cleanup receipt that leaves +the clone present or claims cleanup when the named clone never existed. Both +source captures must use the same reviewed manifest SQL. The postflight capture may be at most 900 seconds old at +verification; the GCP lifecycle age and restore-to-postflight span must each be +at most 3600 seconds: + +```bash +sudo python3 ops/verify_gcp_canonical_lifecycle.py \ + --source-receipt /receipt.json \ + --current-source-receipt /receipt.json \ + --source-delta-receipt "$PRIVATE_RUN_DIR/source-delta-receipt.json" \ + --restore-receipt "$PRIVATE_RUN_DIR/restore-receipt.json" \ + --parity-receipt "$PRIVATE_RUN_DIR/gcp-parity.json" \ + --reasoning-receipt "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \ + --reasoning-compute-receipt "$PRIVATE_RUN_DIR/reasoning-compute-attestation.json" \ + --cleanup-receipt "$PRIVATE_RUN_DIR/cleanup-receipt.json" \ + --max-postflight-age-seconds 900 \ + --max-lifecycle-age-seconds 3600 \ + --output "$PRIVATE_RUN_DIR/lifecycle-verification.json" +``` + +This command validates only the internal consistency of caller-supplied +receipts. A clean result is emitted as +`prepared_external_attestation_required`, with +`current_tier=T2_structural_receipt_bundle`, `t3_receipt_ready=false`, and +`accepted_by_this_api=false`. It cannot emit a live-GCP/T3 claim. T3 requires a +separate independently authenticated provider or platform receipt that the +bundle author cannot mint. + +Retain the private receipt run directory and every useful lifecycle input, +supporting receipt, and verifier output. Remove only transient upload/bundle +directories, temporary Hermes profiles, helper clients, and child processes +after their absence is verified. There is no canonical-path GCS object to +delete. Do not remove the evidence directory until its useful receipts are +integrated or explicitly rejected. ## Legacy SQLite Source Backup Canary @@ -236,11 +417,15 @@ A successful restore or replication canary must retain: - dump timestamp or replication slot timestamp; - source schema/database name. - transfer proof: - - dump object path in a versioned bucket, or logical replication subscription details; + - for the canonical path, the v2 capture receipt, dump SHA-256, and exact + private staging path used by streaming `pg_restore`; + - for a legacy import or replication path, a versioned dump-object generation + or logical replication subscription details; - row/table counts before import where available. - target proof: - - `teleo-pgvector-standby` readback; - - `teleo_kb` database readback; + - `teleo-pgvector-standby` control-plane readback; + - generated `teleo_clone_*` identity for canonical parity, or `teleo_kb` only + for the explicitly legacy SQLite drill; - extension readback for `vector` if the restored schema needs pgvector; - representative query readback for core KB tables. - failure boundary: @@ -295,28 +480,26 @@ Retain only redacted connection metadata. Do not commit or paste credentials. ## Current Blocker -As of 2026-07-11, the canonical Postgres exported-snapshot capture and isolated -local restore parity pass. Live GCP restore and staging replay do not. +As of 2026-07-15, a newest authorized canonical Postgres capture and isolated +local restore can pass, but the required T3 live private GCP lifecycle cannot be +started from the current operator route: -- GitHub WIF works for `sa-artifact-builder`, but that identity is intentionally - limited to Artifact Registry and cannot inspect or mutate Cloud SQL/Compute. -- The configured `sa-teleo-readiness` and `sa-teleo-restore-drill` identities - return IAM 404 and do not exist. -- The local privileged `billy@livingip.xyz` gcloud session requires password - reauthentication. No password was entered or inspected. -- Direct VM SSH is closed to the current egress `/32`; IAP requires the same - privileged GCP authentication. +- direct VM SSH and the private TCP route timed out; +- local `gcloud` for `billy@livingip.xyz` requires reauthentication; no + password was entered or inspected; +- the intended operator WIF/IAP authority is unavailable because provider + `teleo-iap-operator` is absent, disabled, or deleted; +- the working artifact-builder WIF identity is intentionally limited and must + not be expanded or mistaken for Cloud SQL/Compute operator access. -That is why the readiness checker still reports: +The authorized recovery is a local +`gcloud auth login billy@livingip.xyz --force` without sharing credentials, or +administrator convergence of the reviewed operator WIF/IAP bootstrap. Until a +route passes live readback, keep the claim at T2/source-local proof; do not +promote staging or expose Cloud SQL publicly. -- `kb_source_restore_access = blocked` -- `kb_restore_or_replication = blocked` - -The immediate operator CTA is to complete -`gcloud auth login billy@livingip.xyz --force` locally without sharing the -password, or apply the reviewed IAM split with an authorized GCP administrator. -The next non-user action is: - -canonical `teleo` snapshot -> generated Cloud SQL database -> full parity and -private-connectivity verifier -> no-send Cory composition replay from staging -compute -> delete the generated database/object -> retain cleanup proof. +Once access is restored, run: newest authorized `teleo` snapshot -> disposable +private `teleo_clone_*` restore -> generated connectivity proof -> +`gcp-parity.json` -> no-send m3taversal blind reasoning -> live GCE reasoning +attestation -> receipt-bound clone cleanup -> postflight VPS capture and delta +-> eight-receipt lifecycle verdict within the 900/3600-second freshness bounds. diff --git a/docs/observatory-read-adapter-gcp-staging.md b/docs/observatory-read-adapter-gcp-staging.md index b4d6960..23837c1 100644 --- a/docs/observatory-read-adapter-gcp-staging.md +++ b/docs/observatory-read-adapter-gcp-staging.md @@ -34,6 +34,13 @@ it does not change Argus or PR #148's Leo runtime files. - Direct VPC egress: `teleo-staging-net` / `teleo-staging-europe-west6`. - Cloud SQL: `teleo-501523:europe-west6:teleo-pgvector-standby`, private IP. - Database: `teleo_canonical` only. +- Build identity: `sa-artifact-builder@teleo-501523.iam.gserviceaccount.com`; + image build/push only. +- Staging deploy/canary identity: + `sa-observatory-deployer@teleo-501523.iam.gserviceaccount.com`; an exact + workflow/main-only WIF provider, a five-permission Cloud Run custom role, + repository read, exact runtime `actAs`, and exact secret access. Never grant + deploy permissions to the artifact builder. - Runtime identity: `sa-observatory-read-adapter@teleo-501523.iam.gserviceaccount.com`. - IAM database user: `sa-observatory-read-adapter@teleo-501523.iam`. - Database authorization role: `kb_observatory_read` (`NOLOGIN`). @@ -52,24 +59,103 @@ effective write-denial checks drift. These are staging mutations. Run them only with an authenticated GCP operator; they do not route production traffic. -1. Create the dedicated runtime service account and grant only - `roles/cloudsql.client` and `roles/cloudsql.instanceUser` in project - `teleo-501523`. -2. Add the service account as a Cloud SQL IAM service-account user on - `teleo-pgvector-standby`. -3. From the existing private GCP VM database-admin path, run +1. Reauthenticate the project operator, then run the first full direct check: + + ```bash + gcloud projects describe teleo-501523 --format=value(projectId) + python3 ops/check_gcp_infra_readiness.py + ``` + +2. Review the exact dry-run packet. It enables only required APIs, creates + separate staging deploy/runtime identities, creates the dedicated + `observatory-read-adapter-main` WIF provider, keeps the artifact builder + narrow, scopes Cloud SQL connect/login to `teleo-pgvector-standby`, and + scopes secret access to `observatory-read-api-key-staging`: + + ```bash + python3 ops/plan_observatory_read_adapter_gcp.py + python3 ops/plan_observatory_read_adapter_gcp.py --format shell + ``` + + The shell output is unexecuted and uses `set -euo pipefail`. Apply it only + after review with an authenticated `teleo-501523` administrator. The + deployer custom role contains only `run.services.create`, + `run.services.update`, `run.services.get`, `run.services.setIamPolicy`, and + `run.operations.get`; it contains no delete permission. Do not add + `run.admin`, `run.developer`, `cloudsql.admin`, `secretmanager.admin`, + `compute.admin`, Editor, or Owner to either lane identity. +3. Add the dedicated runtime service account as a Cloud SQL IAM + service-account user on `teleo-pgvector-standby`. +4. From the existing private GCP VM database-admin path, run `ops/observatory_read_role.sql` against `teleo_canonical` with `OBSERVATORY_DB_IAM_USER=sa-observatory-read-adapter@teleo-501523.iam`. Do not read, copy, or retain the administrator password. -4. Create `observatory-read-api-key-staging`, add a generated 32-byte-or-longer - value through stdin, and grant that secret's accessor role only to the - runtime service account and the bounded staging canary identity. -5. Dispatch `.github/workflows/gcp-observatory-read-adapter.yml` with - `action=deploy_staging` from the exact reviewed branch revision. +5. Create `observatory-read-api-key-staging`, add a generated whitespace-free + 32-byte-or-longer value through stdin only when no enabled version exists, + and grant that secret's accessor role only to the runtime service account + and the dedicated staging deploy/canary identity. Deployment pins the exact + enabled version; it does not inject `latest` into Cloud Run. +6. Dispatch the read-only preflight from merged `main`: -The workflow builds an immutable image, deploys at most two Cloud Run + ```bash + gh workflow run gcp-observatory-read-adapter.yml \ + --repo living-ip/teleo-infrastructure \ + --ref main \ + -f action=preflight_staging + ``` + + Require a passing `observatory-read-adapter-preflight` artifact before a + bounded `action=deploy_staging` dispatch. The preflight verifies the exact + WIF condition, custom-role permission sets, project/resource IAM bindings, + Cloud Run service agent, private VPC/subnet relationship, private-only Cloud + SQL network and IAM authentication, trimmed IAM database user, exact secret + policy/value shape, and the run-unique immutable image. It also rejects any + artifact-builder deploy, runtime `actAs`, secret, or service binding. + +The workflow always builds a run-unique immutable image, deploys at most two Cloud Run instances, and retains the positive response plus anonymous `401` and -authenticated POST `405` receipts. It does not modify Vercel. +authenticated POST `405` receipts. The retained positive receipt contains +identity/provenance, counts, and proposal/live state only; claim text, source +URLs, excerpts, unknown future provenance fields, and secret values are +removed by an explicit tested allowlist. The live gate binds the response to +the exact project, private instance, database principal, Git revision, Cloud +Run image digest, runtime identity, and pinned secret version. It does not +modify Vercel. The app-level `X-Api-Key` remains the staging authentication +boundary, so anonymous Cloud Run invocation reaches the adapter and returns +the required `401` instead of a platform-generated status. + +## Current T2 Receipt And Exact T3 Gate + +GitHub Actions run `29389090997` proves the existing `living-ip-github` WIF +provider can federate as the artifact builder, run the checker, upload its +artifact, and clean up credentials. The checker reported `8` pass, `10` +blocked, `0` fail. Eight blocks are missing inventory permissions on the +artifact-builder identity; two are missing restore/replication proof. This is +partial OIDC/T2 evidence, not T3 and not authorization to deploy with that +identity. + +The exact external action is for `billy@livingip.xyz` to reauthenticate gcloud +and ADC, prove `gcloud projects describe teleo-501523`, run +`python3 ops/check_gcp_infra_readiness.py`, review and execute the planner's +shell output as a project IAM/API administrator, then apply +`ops/observatory_read_role.sql` from the existing private database-admin path. +The general readiness identity and artifact-builder identity are not deployer +substitutes. After those actions, dispatch `preflight_staging`; only a passing +artifact permits `deploy_staging`. A passing preflight remains T2 setup proof: +the fail-closed live claim and negative receipts are what establish T3. + +The IAM basis is the Google Cloud deployment contract for Cloud Run (Cloud Run +access, Artifact Registry repository read, and exact runtime `actAs`), Secret +Manager resource-level accessor grants, Direct VPC access through the default +Cloud Run service agent, and conditional Cloud SQL client access to one named +instance: + +- https://docs.cloud.google.com/run/docs/deploying +- https://docs.cloud.google.com/run/docs/authenticating/public +- https://docs.cloud.google.com/run/docs/configuring/vpc-direct-vpc +- https://docs.cloud.google.com/iam/docs/workload-identity-federation-with-deployment-pipelines +- https://docs.cloud.google.com/secret-manager/docs/manage-access-to-secrets +- https://docs.cloud.google.com/sql/docs/postgres/iam-conditions ## Unexecuted Cutover Packet diff --git a/docs/reports/leo-working-state-20260709/gcp-canonical-parity-live-20260715.json b/docs/reports/leo-working-state-20260709/gcp-canonical-parity-live-20260715.json new file mode 100644 index 0000000..74489b4 --- /dev/null +++ b/docs/reports/leo-working-state-20260709/gcp-canonical-parity-live-20260715.json @@ -0,0 +1,154 @@ +{ + "artifact": "gcp_canonical_parity_live_attempt", + "schema": "livingip.gcpCanonicalParityLiveAttempt.v1", + "generated_at_utc": "2026-07-15T01:50:20Z", + "status": "gated_external", + "required_tier": "T3_live_private_gcp_staging", + "current_tier": "T2_historical_source_and_isolated_restore_under_superseded_parity_contract", + "current_canary": "Restore the newest authorized canonical snapshot into one private generated Cloud SQL database, prove exact parity and ID-free Leo reasoning from staging compute, then drop the generated database and verify absence.", + "current_source": { + "run_id": "canonical-20260715t014930z", + "captured_at_utc": "2026-07-15T01:49:39.104116+00:00", + "capture_authorization_ref": "codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621", + "receipt_sha256": "7d96b052591d36df3b67dc6291b0f74a7bbeb9602c9cc424779e7c21194d59a5", + "dump_sha256": "6c56a841c1159a17f9404c2d63e3e93ae8ff55e842bf311d6d1bff21e9b95c9a", + "manifest_sha256": "39f06c6b9fc806a89ecd19294be29c58c499c65234611c7a483ddefa3c78d7d3", + "provenance_binding_sha256": "fb587093a76b61af670b7d968d32b78d6a87572faa78ca1ad93ef6cba1c9b776", + "source_system_identifier": "7649789040005668902", + "table_count": 39, + "total_rows": 52167, + "service": { + "active_state": "active", + "sub_state": "running", + "main_pid": 347406, + "restart_count": 0, + "unchanged": true + }, + "production_database_mutated": false, + "telegram_send_attempted": false + }, + "source_stability": { + "baseline_run_id": "canonical-20260715t014814z", + "current_run_id": "canonical-20260715t014930z", + "receipt_sha256": "d761a2b8b0313ee03c9d01e50fcca709af779578281cf10de5aed4709e914d72", + "status": "pass", + "delta_detected": false, + "table_count": 39, + "total_rows": 52167, + "total_row_delta": 0, + "changed_tables": [], + "changed_structures": [], + "table_fingerprint_sha256": "f50d86fc3ce699f602bdc0029485bd901fd98bd8114eecd5dcc4346a0f52112a", + "structure_fingerprint_sha256": "61bf482330a652ebdb419de3fb78196f2c7d25d1a482b100f385e8102c8e8d52" + }, + "isolated_restore": { + "receipt_sha256": "181901afc91bebee7c343500c601ed1e5b7a5e42a5b81583d445109bfc680d86", + "status": "pass", + "source_table_count": 39, + "target_table_count": 39, + "source_total_rows": 52167, + "target_total_rows": 52167, + "table_mismatches": [], + "structural_hashes_equal": true, + "constraint_hash": "8b26d26fda56f09325e0603aae48051da08dfc6c35322fa88880acdabb8ad859", + "application_role_mismatches": {}, + "target_extra_application_roles": [], + "required_extension_mismatches": {}, + "performance_problems": [], + "network_mode": "none", + "persistent_postgres_storage_used": false, + "container_absent_after_test": true, + "authorization_state_verified": false, + "prior_manifest_contract_only": true + }, + "contract_reassessment": { + "status": "changes_required", + "reviewed_manifest_sql_sha256": "2bd3e353627d44d4a292949952c06b1897d1a9a9e98243905c7a834d007db188", + "prior_receipts_recomputed_under_current_contract": false, + "newly_required_parity": [ + "kb_gate_owner plus exact application-role attributes", + "role memberships", + "database, schema, relation, function, and type ownership", + "normalized database, schema, relation, column, function, and type ACLs", + "symmetric role and extension sets", + "nonempty row hash for every table" + ], + "t3_receipt_accepted_by_repository_api": false, + "independent_platform_attestation_required": true + }, + "permission_profile": { + "approval_policy": "never", + "sandbox_mode": "danger-full-access", + "network": "enabled" + }, + "config_readback": { + "gcloud_sdk_version": "551.0.0", + "configured_account": "billy@livingip.xyz", + "configured_project": "teleo-501523", + "configured_account_status": "failed_noninteractive_reauthentication", + "application_default_credentials_status": "failed_noninteractive_reauthentication", + "expected_cloudsql_instance_not_currently_verified": "teleo-pgvector-standby", + "expected_private_network_not_currently_verified": "teleo-staging-net", + "expected_private_address_not_currently_verified": "10.61.0.3", + "expected_staging_compute_not_currently_verified": "teleo-prod-1/europe-west6-a", + "public_ip_disabled_currently_verified": false, + "generated_database_inventory_currently_verified": false + }, + "attempted_no_approval_routes": [ + "configured privileged gcloud identity", + "three other cached gcloud identities", + "application-default credentials", + "direct SSH to the retained staging address", + "local gcloud IAP SSH dry run", + "local and VPS TCP probes to the expected private Cloud SQL address", + "fixed GitHub IAP status workflow receipt", + "five GitHub readiness workflow receipts across candidate service accounts", + "current successful Artifact Registry-only WIF workflow" + ], + "exact_gate": "The privileged local Google OAuth session requires human reauthentication, the fixed IAP workflow names an unavailable teleo-iap-operator WIF provider, and no existing noninteractive service-account route can obtain a Cloud SQL/Compute-capable token. Current Cloud SQL topology and generated-database inventory cannot be read, so the private restore cannot start.", + "why_autonomous_repair_stops": "Google reauthentication may require a password, MFA, passkey, or consent that Codex must not retrieve or enter. Recreating or rebinding the WIF/IAP identities requires an already-authorized GCP administrator, and no safe noninteractive identity currently has that authority.", + "next_non_user_action": "After the reviewed operator route and exact restore service account are available, capture a new source snapshot with the expanded manifest, implement and review a Cloud-SQL-compatible authorization replay, use verify-full TLS with the reviewed server name and CA, run the disposable clone lifecycle, then obtain an independently authenticated platform receipt. The repository eight-receipt verifier remains preparation-only.", + "gcp_effects_from_this_attempt": { + "cloudsql_database_created": false, + "gcp_resource_created": false, + "gcp_resource_modified": false, + "public_ip_exposure_changed": false, + "staging_promoted": false, + "telegram_sent": false + }, + "proof_receipts": { + "retained_private_runtime_artifacts": true, + "source_receipt": { + "filename": "receipt.json", + "sha256": "7d96b052591d36df3b67dc6291b0f74a7bbeb9602c9cc424779e7c21194d59a5" + }, + "source_manifest": { + "filename": "source-manifest.jsonl", + "sha256": "39f06c6b9fc806a89ecd19294be29c58c499c65234611c7a483ddefa3c78d7d3" + }, + "source_delta": { + "filename": "source-stability-delta.json", + "sha256": "d761a2b8b0313ee03c9d01e50fcca709af779578281cf10de5aed4709e914d72" + }, + "isolated_restore": { + "filename": "local-rebuild-receipt.json", + "sha256": "181901afc91bebee7c343500c601ed1e5b7a5e42a5b81583d445109bfc680d86" + }, + "route_recheck": { + "filename": "route-recheck.json", + "sha256": "0fda5e6d58d72dcb084dc277bc59312bf62c002c728e249a9f16e79ae096037b" + } + }, + "strongest_claim_allowed": "The retained snapshot was stable and reconstructed data and schema under the prior parity contract. It does not prove current authorization parity or live GCP operation under the expanded contract.", + "not_proven": [ + "current Cloud SQL topology or public-IP-disabled state", + "current generated-database inventory", + "private staging-to-Cloud-SQL TLS connectivity", + "current Cloud SQL restore parity", + "current role membership, ownership, and ACL parity", + "certificate-authenticated verify-full database access", + "independently authenticated T3 lifecycle attestation", + "blind Leo reasoning against the generated Cloud SQL database", + "generated Cloud SQL database teardown" + ] +} diff --git a/docs/reports/leo-working-state-20260709/gcp-canonical-parity-teardown-20260715.json b/docs/reports/leo-working-state-20260709/gcp-canonical-parity-teardown-20260715.json new file mode 100644 index 0000000..490afad --- /dev/null +++ b/docs/reports/leo-working-state-20260709/gcp-canonical-parity-teardown-20260715.json @@ -0,0 +1,36 @@ +{ + "artifact": "gcp_canonical_parity_attempt_teardown", + "schema": "livingip.gcpCanonicalParityAttemptTeardown.v1", + "generated_at_utc": "2026-07-15T01:50:20Z", + "status": "no_cloud_resource_created", + "required_tier": "T3_live_private_gcp_staging", + "required_tier_met": false, + "cloudsql_clone_created": false, + "cloudsql_clone_dropped": false, + "cloudsql_clone_absence_readback": "not_applicable_no_create", + "gcs_object_created": false, + "compute_resource_created": false, + "cleanup_required": false, + "cleanup_performed": false, + "orphan_risk_from_this_attempt": false, + "public_ip_exposure_changed": false, + "staging_promoted": false, + "local_restore_container": { + "name": "teleo-canonical-rebuild-20260715015004-4ec8d1daac32", + "force_remove_exit_code": 0, + "container_absent": true, + "persistent_storage_used": false, + "network_access_available": false + }, + "current_gcp_orphan_inventory_verified": false, + "exact_gate": "No authenticated Cloud SQL/Compute-capable route was available, so execution stopped before any GCP resource or generated database was created.", + "proof_paths": { + "live_attempt": "docs/reports/leo-working-state-20260709/gcp-canonical-parity-live-20260715.json", + "route_recheck_filename": "route-recheck.json", + "route_recheck_sha256": "0fda5e6d58d72dcb084dc277bc59312bf62c002c728e249a9f16e79ae096037b", + "local_cleanup_receipt_filename": "local-rebuild-receipt.json", + "local_cleanup_receipt_sha256": "181901afc91bebee7c343500c601ed1e5b7a5e42a5b81583d445109bfc680d86", + "runtime_receipts_retained_privately": true + }, + "strongest_claim_allowed": "This attempt introduced no GCP cleanup debt and its isolated local restore container was removed; it is not a teardown receipt for a live disposable Cloud SQL database." +} diff --git a/docs/reports/observatory-read-adapter/gcp-oidc-readiness-29389090997-redacted.json b/docs/reports/observatory-read-adapter/gcp-oidc-readiness-29389090997-redacted.json new file mode 100644 index 0000000..804e50b --- /dev/null +++ b/docs/reports/observatory-read-adapter/gcp-oidc-readiness-29389090997-redacted.json @@ -0,0 +1,91 @@ +{ + "schema": "livingip.observatoryReadAdapterOidcReadinessReceipt.v1", + "status": "partial_oidc_t2", + "required_runtime_tier": "T3_live_gcp_staging_api", + "current_runtime_tier": "T2_authenticated_github_oidc_partial_inventory", + "workflow": { + "run_id": 29389090997, + "url": "https://github.com/living-ip/teleo-infrastructure/actions/runs/29389090997", + "revision": "33399bd9acfa09e4e5409873e86c8c46ac694a8d", + "ref": "refs/heads/main", + "identity": "sa-artifact-builder@teleo-501523.iam.gserviceaccount.com", + "wif_authentication": "pass", + "gcloud_setup": "pass", + "checker_execution": "pass", + "artifact_upload": "pass", + "credential_cleanup": "pass", + "job_conclusion": "failure_due_to_blocked_readiness_enforcement" + }, + "summary": { + "pass_count": 8, + "blocked_count": 10, + "fail_count": 0, + "passed_checks": [ + "artifact_repositories", + "cloud_build_contract", + "github_actions_readiness_ci", + "gcp_cloudsql_restore_drill_contract", + "canonical_postgres_restore_contract", + "local_canonical_postgres_restore_preflight", + "gcp_runtime_baseline_contract", + "gcp_service_communications_contract" + ], + "permission_or_resource_read_blocks": [ + { + "check": "cloud_build_service_account", + "missing_permission": "iam.serviceAccounts.get" + }, + { + "check": "github_actions_artifact_ci", + "missing_permission": "iam.workloadIdentityPoolProviders.get" + }, + { + "check": "network_ingress", + "missing_permission": "compute.firewalls.list" + }, + { + "check": "compute_runtime_service_accounts", + "missing_permission": "compute.instances.list" + }, + { + "check": "compute_disk_snapshots", + "missing_permission": "compute.resourcePolicies.get" + }, + { + "check": "backup_buckets", + "missing_permission": "bucket metadata read denied; exact permission was not retained by the checker" + }, + { + "check": "cloud_sql_standby_target", + "missing_permission": "Cloud SQL instance read denied or target invisible to this identity" + }, + { + "check": "kb_source_restore_access", + "missing_permission": "secretmanager.secrets.list" + } + ], + "evidence_blocks": [ + "local_sqlite_postgres_restore_canary", + "kb_restore_or_replication" + ] + }, + "security_readback": { + "artifact_builder_deploy_authorized": false, + "cloud_run_service_created": false, + "cloud_sql_queried": false, + "secret_values_logged": false, + "production_or_vercel_routing_changed": false + }, + "source_artifact": { + "github_run_url": "https://github.com/living-ip/teleo-infrastructure/actions/runs/29389090997", + "artifact_name": "gcp-readiness", + "summary_sha256": "6a1be356c069b8affacc492ac3b1e1ca63b917f92a32d7d886511a226c52925b", + "checker_sha256": "56ce4e8ba655a47e6e620a2c7039ee54fa10b1c53431d1dc6ec4db5189bb17f5", + "local_download_path": "/private/tmp/observatory-read-adapter-oidc-29389090997/gcp-readiness" + }, + "proof_paths": [ + "docs/reports/observatory-read-adapter/gcp-oidc-readiness-29389090997-redacted.json", + "https://github.com/living-ip/teleo-infrastructure/actions/runs/29389090997" + ], + "claim_ceiling": "GitHub OIDC and the artifact-only identity work. The receipt does not prove broad GCP inventory, adapter prerequisites, a live Cloud Run service, or a teleo_canonical query." +} diff --git a/docs/reports/observatory-read-adapter/gcp-staging-current-gate.json b/docs/reports/observatory-read-adapter/gcp-staging-current-gate.json index f87f8cb..c6b44a0 100644 --- a/docs/reports/observatory-read-adapter/gcp-staging-current-gate.json +++ b/docs/reports/observatory-read-adapter/gcp-staging-current-gate.json @@ -1,19 +1,20 @@ { "schema": "livingip.observatoryReadAdapterCurrentGate.v1", "current_canary": "Authenticated GET /v1/claims/sample in GCP staging returns a provenance-bearing teleo_canonical claim, anonymous GET returns 401, and authenticated POST returns 405.", - "permission_profile": "approval_policy_never_with_unrestricted_filesystem_and_network", + "required_runtime_tier": "T3_live_gcp_staging_api", + "current_runtime_tier": "T2_authenticated_github_oidc_partial_inventory", "attempted_routes": [ { "route": "merged GitHub Actions deployment with artifact-builder WIF", - "result": "image build, smoke, and push passed; deploy stopped because run.googleapis.com is disabled" + "result": "OIDC, image build, smoke, and push passed; the 29382683335 deploy stopped before resource creation because run.googleapis.com was disabled" }, { - "route": "local billy@livingip.xyz gcloud user credential", - "result": "token refresh and service enable both stop at noninteractive Google reauthentication" + "route": "GitHub Actions GCP readiness with artifact-builder WIF, run 29389090997", + "result": "federation, gcloud setup, checker, artifact upload, and credential cleanup passed; checker result was 8 pass, 10 blocked, 0 fail" }, { - "route": "local application-default credential", - "result": "token refresh stops at noninteractive Google reauthentication" + "route": "local billy@livingip.xyz gcloud user and application-default credentials", + "result": "both require interactive Google reauthentication before project-admin API or IAM changes" }, { "route": "other locally authenticated Google accounts", @@ -22,19 +23,30 @@ { "route": "fixed read-only GCP IAP status workflow", "result": "WIF token exchange failed because the teleo-iap-operator provider target is missing or disabled" - }, - { - "route": "local browser or computer-use session", - "result": "no browser or computer-use tool is exposed to this task; no foreground takeover was attempted" } ], "exact_gate": { - "first_infrastructure_gate": "Cloud Run Admin API run.googleapis.com is disabled in teleo-501523.", - "current_authorization_gate": "The only local account authorized for teleo-501523, billy@livingip.xyz, requires Google password or MFA reauthentication before project APIs and staging IAM resources can be changed.", - "alternate_private_operator_gate": "The read-only teleo-iap-operator WIF provider returns invalid_target and cannot reach its status operation." + "non_gui_federation": "Working. sa-artifact-builder can federate through living-ip-github and read/push Artifact Registry resources.", + "artifact_builder_scope": "Intentionally insufficient for deployment or full inventory. Do not grant it Cloud Run, Cloud SQL, Compute, IAM, or Secret Manager deployment roles.", + "last_api_readback": "The last direct deployment reported run.googleapis.com disabled. The current artifact-builder identity cannot verify or enable the required project APIs.", + "t3_bootstrap": "An authenticated teleo-501523 administrator must review and apply ops/plan_observatory_read_adapter_gcp.py, including required APIs, dedicated deploy/runtime identities, the exact workflow/main-only observatory-read-adapter-main WIF provider, five-permission Cloud Run custom role, exact Cloud SQL instance grants, exact secret grants, trimmed IAM database user, and private database role SQL.", + "local_operator": "billy@livingip.xyz requires Google password or MFA reauthentication before that administrator action can run.", + "alternate_private_operator": "The fixed teleo-iap-operator provider is absent or disabled and is not an available substitute." }, - "why_autonomous_repair_stops": "Google password or MFA reauthentication is a human-only credential boundary, and no authenticated project-admin browser or computer-use surface is available in this task.", - "clear_CTA": "Run `gcloud auth login billy@livingip.xyz --update-adc`, complete Google password or MFA, and then report `GCP reauth complete`.", - "next_non_user_action": "Enable the required staging APIs; create or verify the runtime service account, API-key secret, Cloud SQL IAM database user, and exact read-only role grants; deploy the retained immutable image; capture authenticated GET, anonymous 401, authenticated POST 405, SQL write-denial, exact service/database identity, and cleanup receipts.", - "production_boundary": "Do not change Vercel, production routing, Cloud SQL network exposure, or any write endpoint." + "latest_direct_non_gui_check": { + "active_account": "billy@livingip.xyz", + "gcloud_projects_describe": "blocked: reauthentication failed and non-interactive execution cannot prompt", + "application_default_credentials": "blocked: reauthentication failed and non-interactive execution cannot prompt", + "full_readiness_checker_executed": false, + "why_not": "The required first command, gcloud projects describe teleo-501523, did not pass." + }, + "why_t3_stops": "A live staging service requires authorized API, IAM, Secret Manager, Cloud SQL IAM-user, and private database-role mutations. Working artifact-builder federation does not authorize those actions.", + "clear_CTA": "Run `gcloud auth login billy@livingip.xyz --update-adc`, then `gcloud projects describe teleo-501523 --format=value(projectId)` and `python3 ops/check_gcp_infra_readiness.py`; review and apply the exact adapter packet emitted by `python3 ops/plan_observatory_read_adapter_gcp.py --format shell`.", + "next_non_user_action": "Dispatch action=preflight_staging from merged main using sa-observatory-deployer. Only after its redacted preflight artifact passes, dispatch the bounded staging deploy and capture authenticated GET, anonymous 401, authenticated POST 405, database write-denial, exact service/database identity, and cleanup receipts.", + "production_boundary": "Do not change Vercel, production routing, Cloud SQL network exposure, Telegram, x402, or any write endpoint.", + "proof_paths": [ + "docs/reports/observatory-read-adapter/gcp-oidc-readiness-29389090997-redacted.json", + "docs/reports/observatory-read-adapter/gcp-staging-live-attempt-redacted.json", + "https://github.com/living-ip/teleo-infrastructure/actions/runs/29389090997" + ] } diff --git a/docs/reports/observatory-read-adapter/gcp-staging-live-attempt-redacted.json b/docs/reports/observatory-read-adapter/gcp-staging-live-attempt-redacted.json index 8c50ace..7fcd98f 100644 --- a/docs/reports/observatory-read-adapter/gcp-staging-live-attempt-redacted.json +++ b/docs/reports/observatory-read-adapter/gcp-staging-live-attempt-redacted.json @@ -2,7 +2,7 @@ "schema": "livingip.observatoryReadAdapterStagingReceipt.v1", "status": "blocked_before_runtime_creation", "required_runtime_tier": "T3_live_gcp_staging_api", - "current_runtime_tier": "below_T3_build_artifact_only", + "current_runtime_tier": "T2_authenticated_github_oidc_partial_inventory", "project_id": "teleo-501523", "region": "europe-west6", "service_name": "observatory-read-adapter-staging", @@ -26,6 +26,18 @@ "failure_detail": "run.googleapis.com is disabled in teleo-501523", "authenticated_as": "sa-artifact-builder@teleo-501523.iam.gserviceaccount.com" }, + "readiness_reconciliation": { + "run_id": 29389090997, + "url": "https://github.com/living-ip/teleo-infrastructure/actions/runs/29389090997", + "identity": "sa-artifact-builder@teleo-501523.iam.gserviceaccount.com", + "wif_authentication": "pass", + "checker_result": { + "pass_count": 8, + "blocked_count": 10, + "fail_count": 0 + }, + "claim": "working federation with an intentionally narrow artifact identity; not a staging deploy or full inventory" + }, "live_receipts": { "service_url": null, "authenticated_get_sample": null, @@ -47,9 +59,26 @@ "orphan_local_build_processes": 0, "artifact_registry_image_retained_for_next_deploy": true }, - "proof_paths": [ - "/private/tmp/observatory-read-adapter-run-29382683335/observatory-adapter-image/observatory-adapter-image.txt", - "/private/tmp/observatory-iap-status-29382844567/gcp-iap-operator-29382844567/result.json" + "source_artifacts": [ + { + "run_url": "https://github.com/living-ip/teleo-infrastructure/actions/runs/29382683335", + "artifact_name": "observatory-adapter-image", + "sha256": "6d958b8d345ec878db2f70aaab5fad4318b106441983bd7f8e9170a429f4e852" + }, + { + "run_url": "https://github.com/living-ip/teleo-infrastructure/actions/runs/29382844567", + "artifact_name": "gcp-iap-operator-29382844567", + "sha256": "988f77123f7624469c15e3c38bc1e8864a1e9f61e571feb5f6c838e67fe039e9" + }, + { + "run_url": "https://github.com/living-ip/teleo-infrastructure/actions/runs/29389090997", + "artifact_name": "gcp-readiness", + "sha256": "6a1be356c069b8affacc492ac3b1e1ca63b917f92a32d7d886511a226c52925b" + } ], - "claim_ceiling": "The immutable staging image exists, but no live API or database query receipt exists. T3 is not complete." + "proof_paths": [ + "docs/reports/observatory-read-adapter/gcp-staging-live-attempt-redacted.json", + "docs/reports/observatory-read-adapter/gcp-oidc-readiness-29389090997-redacted.json" + ], + "claim_ceiling": "The immutable staging image and working artifact-only OIDC path exist, but no live API or database query receipt exists. T3 is not complete." } diff --git a/hermes-agent/leoclean-bin/cloudsql_memory_tool.py b/hermes-agent/leoclean-bin/cloudsql_memory_tool.py index 9c579fd..2f964b0 100755 --- a/hermes-agent/leoclean-bin/cloudsql_memory_tool.py +++ b/hermes-agent/leoclean-bin/cloudsql_memory_tool.py @@ -167,6 +167,8 @@ def runtime_cloudsdk_config() -> str: except OSError: raise SystemExit("Reviewed empty Cloud SDK configuration is unavailable or untrusted.") from None return str(resolved) + + CLONE_READ_ONLY_PGOPTIONS = "-c default_transaction_read_only=on" RECEIPTED_READ_COMMANDS = frozenset( { @@ -376,10 +378,7 @@ def validate_runtime_connection(args: argparse.Namespace) -> None: ) or ( normalized_key.startswith("CLOUDSDK_") - and not ( - normalized_key == "CLOUDSDK_CONFIG" - and value == cloudsdk_config - ) + and not (normalized_key == "CLOUDSDK_CONFIG" and value == cloudsdk_config) ) or normalized_key == "GOOGLE_APPLICATION_CREDENTIALS" or normalized_key.startswith("LD_") @@ -392,7 +391,8 @@ def validate_runtime_connection(args: argparse.Namespace) -> None: or normalized_key == "PYTHONUNBUFFERED" ) ) - or normalized_key in { + or normalized_key + in { "BASH_ENV", "BASHOPTS", "CDPATH", diff --git a/ops/attest_gcp_reasoning_compute.py b/ops/attest_gcp_reasoning_compute.py new file mode 100644 index 0000000..42a1105 --- /dev/null +++ b/ops/attest_gcp_reasoning_compute.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 +"""Bind a no-send reasoning receipt to live GCE metadata on the replay host.""" + +from __future__ import annotations + +import argparse +import json +import stat +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + +try: + from .private_receipt_io import print_private_receipt_summary, write_private_json + from .restore_gcp_generated_postgres_snapshot import ( + DEFAULT_COMPUTE_INSTANCE, + DEFAULT_COMPUTE_ZONE, + DEFAULT_PRIVATE_NETWORK, + DEFAULT_PROJECT, + DEFAULT_RESTORE_SERVICE_ACCOUNT, + RUN_ID_RE, + TARGET_DATABASE_RE, + gce_compute_identity, + sha256_file, + ) +except ImportError: # pragma: no cover - direct script execution on the GCP VM + from private_receipt_io import print_private_receipt_summary, write_private_json + from restore_gcp_generated_postgres_snapshot import ( + DEFAULT_COMPUTE_INSTANCE, + DEFAULT_COMPUTE_ZONE, + DEFAULT_PRIVATE_NETWORK, + DEFAULT_PROJECT, + DEFAULT_RESTORE_SERVICE_ACCOUNT, + RUN_ID_RE, + TARGET_DATABASE_RE, + gce_compute_identity, + sha256_file, + ) + +REASONING_SCHEMA = "livingip.gcpGeneratedDbBlindClaimCanary.v1" +ATTESTATION_SCHEMA = "livingip.gcpReasoningComputeAttestation.v1" + + +def utc_now() -> str: + return datetime.now(UTC).isoformat() + + +def parse_timestamp(value: Any, label: str) -> datetime: + text = str(value or "") + if text.endswith("Z"): + text = text[:-1] + "+00:00" + try: + parsed = datetime.fromisoformat(text) + except ValueError as exc: + raise ValueError(f"{label} is not an ISO-8601 timestamp") from exc + if parsed.tzinfo is None: + raise ValueError(f"{label} must include a timezone") + return parsed.astimezone(UTC) + + +def load_reasoning_receipt(path: Path) -> dict[str, Any]: + try: + mode = path.lstat().st_mode + except OSError as exc: + raise ValueError(f"reasoning receipt is unavailable: {exc}") from exc + if not stat.S_ISREG(mode) or path.is_symlink(): + raise ValueError("reasoning receipt must be a regular non-symlink file") + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError as exc: + raise ValueError("reasoning receipt is invalid JSON") from exc + if not isinstance(payload, dict): + raise ValueError("reasoning receipt must be a JSON object") + return payload + + +def attest_reasoning_compute( + *, + reasoning_receipt_path: Path, + restore_run_id: str, + target_database: str, + project: str = DEFAULT_PROJECT, + expected_compute_instance: str = DEFAULT_COMPUTE_INSTANCE, + expected_compute_zone: str = DEFAULT_COMPUTE_ZONE, + expected_private_network: str = DEFAULT_PRIVATE_NETWORK, + expected_restore_service_account: str = DEFAULT_RESTORE_SERVICE_ACCOUNT, + metadata_timeout: float = 10.0, + generated_at_utc: str | None = None, +) -> dict[str, Any]: + if not RUN_ID_RE.fullmatch(restore_run_id): + raise ValueError("restore run id is invalid") + if not TARGET_DATABASE_RE.fullmatch(target_database): + raise ValueError("target database must be a disposable teleo_clone_* database") + reasoning_receipt_path = reasoning_receipt_path.resolve() + reasoning = load_reasoning_receipt(reasoning_receipt_path) + started = parse_timestamp(reasoning.get("generated_at_utc"), "reasoning start") + completed = parse_timestamp(reasoning.get("completed_at_utc"), "reasoning completion") + checks = { + "schema": reasoning.get("schema") == REASONING_SCHEMA, + "status": reasoning.get("status") == "pass" and not reasoning.get("errors"), + "target_database": reasoning.get("target_database") == target_database, + "source_compute": reasoning.get("source_compute") == expected_compute_instance, + "chronology": started <= completed, + "no_telegram_send": reasoning.get("posted_to_telegram") is False, + "no_database_write": reasoning.get("database_write_attempted") is False, + } + failed = sorted(name for name, passed in checks.items() if not passed) + if failed: + raise ValueError("reasoning receipt failed attestation preflight: " + ", ".join(failed)) + + compute = gce_compute_identity( + project, + expected_compute_instance, + expected_compute_zone, + expected_private_network, + expected_restore_service_account, + timeout=metadata_timeout, + ) + return { + "artifact": "gcp_reasoning_compute_attestation", + "schema": ATTESTATION_SCHEMA, + "status": "pass", + "generated_at_utc": generated_at_utc or utc_now(), + "restore_run_id": restore_run_id, + "target_database": target_database, + "reasoning_receipt": { + "sha256": sha256_file(reasoning_receipt_path), + "schema": reasoning["schema"], + "generated_at_utc": reasoning["generated_at_utc"], + "completed_at_utc": reasoning["completed_at_utc"], + "claimed_source_compute": reasoning["source_compute"], + }, + "compute": compute, + "checks": {**checks, "gce_metadata_identity": all(compute.get("checks", {}).values())}, + "method": "gce_metadata_server_v1", + "mutation": { + "cloudsql_changed": False, + "compute_changed": False, + "database_changed": False, + "telegram_message_sent": False, + }, + } + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--reasoning-receipt", required=True, type=Path) + parser.add_argument("--restore-run-id", required=True) + parser.add_argument("--target-db", required=True) + parser.add_argument("--project", default=DEFAULT_PROJECT) + parser.add_argument("--expected-compute-instance", default=DEFAULT_COMPUTE_INSTANCE) + parser.add_argument("--expected-compute-zone", default=DEFAULT_COMPUTE_ZONE) + parser.add_argument("--expected-private-network", default=DEFAULT_PRIVATE_NETWORK) + parser.add_argument("--expected-restore-service-account", default=DEFAULT_RESTORE_SERVICE_ACCOUNT) + parser.add_argument("--metadata-timeout", default=10.0, type=float) + parser.add_argument("--output", required=True, type=Path) + args = parser.parse_args() + try: + receipt = attest_reasoning_compute( + reasoning_receipt_path=args.reasoning_receipt, + restore_run_id=args.restore_run_id, + target_database=args.target_db, + project=args.project, + expected_compute_instance=args.expected_compute_instance, + expected_compute_zone=args.expected_compute_zone, + expected_private_network=args.expected_private_network, + expected_restore_service_account=args.expected_restore_service_account, + metadata_timeout=args.metadata_timeout, + ) + write_private_json(args.output, receipt) + except (OSError, ValueError, RuntimeError) as exc: + parser.error(str(exc)) + print_private_receipt_summary(args.output, receipt) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/ops/capture_vps_canonical_postgres_snapshot.py b/ops/capture_vps_canonical_postgres_snapshot.py index 055d933..7cde893 100755 --- a/ops/capture_vps_canonical_postgres_snapshot.py +++ b/ops/capture_vps_canonical_postgres_snapshot.py @@ -18,22 +18,29 @@ from pathlib import Path from typing import Any try: + from .private_receipt_io import print_private_receipt_summary, write_private_json from .verify_postgres_parity_manifest import load_manifest except ImportError: # pragma: no cover - direct script execution + from private_receipt_io import print_private_receipt_summary, write_private_json from verify_postgres_parity_manifest import load_manifest SAFE_NAME_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}\Z") SAFE_RUN_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_-]{7,63}\Z") -SAFE_SSH_TARGET_RE = re.compile( - r"(?:[A-Za-z0-9][A-Za-z0-9._-]{0,63}@)?[A-Za-z0-9][A-Za-z0-9.-]{0,252}\Z" -) +SAFE_AUTHORIZATION_REF_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.:/@+-]{7,255}\Z") +SAFE_SSH_TARGET_RE = re.compile(r"(?:[A-Za-z0-9][A-Za-z0-9._-]{0,63}@)?[A-Za-z0-9][A-Za-z0-9.-]{0,252}\Z") +SNAPSHOT_ID_RE = re.compile(r"[0-9A-F]+-[0-9A-F]+-[0-9]+\Z") +TXID_SNAPSHOT_RE = re.compile(r"[0-9]+:[0-9]+:(?:[0-9]+(?:,[0-9]+)*)?\Z") +WAL_LSN_RE = re.compile(r"[0-9A-F]+/[0-9A-F]+\Z") +SYSTEM_IDENTIFIER_RE = re.compile(r"[0-9]+\Z") +SOURCE_SNAPSHOT_RECEIPT_SCHEMA = "livingip.sourceSnapshotReceipt.v2" EXPECTED_FILES = { "teleo-canonical.dump", "teleo-canonical.dump.sha256", "teleo-canonical.toc", "source-manifest.jsonl", "source-context.txt", + "source-snapshot.json", "service-before.txt", "service-after.txt", } @@ -47,6 +54,81 @@ def sha256(path: Path) -> str: return digest.hexdigest() +def sha256_bytes(payload: bytes) -> str: + return hashlib.sha256(payload).hexdigest() + + +def canonical_sha256(value: Any) -> str: + payload = json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True) + return sha256_bytes(payload.encode()) + + +def load_snapshot_metadata(path: Path) -> dict[str, str]: + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + raise RuntimeError("source snapshot metadata is unreadable") from exc + if not isinstance(payload, dict): + raise RuntimeError("source snapshot metadata must be an object") + fields = { + "exported_snapshot_id": SNAPSHOT_ID_RE, + "txid_snapshot": TXID_SNAPSHOT_RE, + "wal_lsn": WAL_LSN_RE, + "system_identifier": SYSTEM_IDENTIFIER_RE, + } + normalized: dict[str, str] = {} + for field, pattern in fields.items(): + value = str(payload.get(field) or "") + if not pattern.fullmatch(value): + raise RuntimeError(f"source snapshot metadata has invalid {field}") + normalized[field] = value + captured_at_utc = str(payload.get("captured_at_utc") or "") + if not captured_at_utc or any(character in captured_at_utc for character in "\r\n\x00"): + raise RuntimeError("source snapshot metadata has invalid captured_at_utc") + normalized["captured_at_utc"] = captured_at_utc + return normalized + + +def load_service_state(path: Path) -> dict[str, str]: + try: + state = dict(line.split("=", 1) for line in path.read_text(encoding="utf-8").splitlines() if "=" in line) + main_pid = int(state.get("MainPID") or 0) + restarts = int(state.get("NRestarts") or 0) + except (OSError, TypeError, ValueError) as exc: + raise RuntimeError("source service state is unreadable") from exc + if state.get("ActiveState") != "active" or state.get("SubState") != "running" or main_pid <= 0 or restarts < 0: + raise RuntimeError("source service is not active/running with a positive MainPID") + return state + + +def build_provenance_binding( + *, + run_id: str, + authorization_ref: str, + source: dict[str, str], + snapshot: dict[str, str], + dump_sha256: str, + manifest_sql_sha256: str, + source_manifest_sha256: str, + source_context_sha256: str, +) -> dict[str, Any]: + payload = { + "receipt_schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA, + "run_id": run_id, + "capture_authorization_ref": authorization_ref, + "source": source, + "exported_snapshot_id": snapshot["exported_snapshot_id"], + "txid_snapshot": snapshot["txid_snapshot"], + "wal_lsn": snapshot["wal_lsn"], + "source_system_identifier": snapshot["system_identifier"], + "dump_sha256": dump_sha256, + "manifest_sql_sha256": manifest_sql_sha256, + "source_manifest_sha256": source_manifest_sha256, + "source_context_sha256": source_context_sha256, + } + return {"algorithm": "sha256", "payload": payload, "sha256": canonical_sha256(payload)} + + def ssh_base(identity_file: Path) -> list[str]: return [ "ssh", @@ -73,12 +155,12 @@ def remote_capture_script(*, run_id: str, container: str, database: str, service "container_dump": shlex.quote(container_dump), } return f"""set -euo pipefail -remote_root={quoted['remote_root']} -container={quoted['container']} -database={quoted['database']} -service={quoted['service']} -container_manifest={quoted['container_manifest']} -container_dump={quoted['container_dump']} +remote_root={quoted["remote_root"]} +container={quoted["container"]} +database={quoted["database"]} +service={quoted["service"]} +container_manifest={quoted["container_manifest"]} +container_dump={quoted["container_dump"]} cleanup() {{ docker exec "$container" rm -f "$container_manifest" "$container_dump" >/dev/null 2>&1 || true @@ -86,11 +168,23 @@ cleanup() {{ }} trap cleanup EXIT +assert_service_healthy() {{ + local state_file="$1" + grep -qx 'ActiveState=active' "$state_file" + grep -qx 'SubState=running' "$state_file" + local main_pid restarts + main_pid="$(sed -n 's/^MainPID=//p' "$state_file")" + restarts="$(sed -n 's/^NRestarts=//p' "$state_file")" + [[ "$main_pid" =~ ^[1-9][0-9]*$ ]] + [[ "$restarts" =~ ^[0-9]+$ ]] +}} + test -d "$remote_root" test -f "$remote_root/postgres-parity-manifest.sql" docker inspect "$container" >/dev/null docker exec "$container" pg_isready -U postgres -d "$database" >/dev/null systemctl show "$service" -p ActiveState -p SubState -p MainPID -p NRestarts -p ActiveEnterTimestamp --no-pager > "$remote_root/service-before.txt" +assert_service_healthy "$remote_root/service-before.txt" docker inspect "$container" --format 'container_id={{{{.Id}}}}\nrunning={{{{.State.Running}}}}\nnetwork_mode={{{{.HostConfig.NetworkMode}}}}' > "$remote_root/source-context.txt" printf 'database=%s\ncaptured_at_utc=%s\n' "$database" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$remote_root/source-context.txt" docker cp "$remote_root/postgres-parity-manifest.sql" "$container:$container_manifest" >/dev/null @@ -107,6 +201,19 @@ if [[ ! "$snapshot_id" =~ ^[0-9A-F]+-[0-9A-F]+-[0-9]+$ ]]; then echo "invalid exported snapshot id" >&2 exit 1 fi +printf '%s\n' "select jsonb_build_object( + 'exported_snapshot_id', '$snapshot_id', + 'txid_snapshot', txid_current_snapshot()::text, + 'wal_lsn', pg_current_wal_lsn()::text, + 'system_identifier', (select system_identifier::text from pg_control_system()), + 'captured_at_utc', clock_timestamp() +)::text;" >&3 +IFS= read -r snapshot_metadata <&4 +if [[ -z "$snapshot_metadata" ]]; then + echo "invalid source snapshot metadata" >&2 + exit 1 +fi +printf '%s\n' "$snapshot_metadata" > "$remote_root/source-snapshot.json" docker exec "$container" pg_dump \ -U postgres -d "$database" \ @@ -130,6 +237,7 @@ docker cp "$container:$container_dump" "$remote_root/teleo-canonical.dump" >/dev sha256sum "$remote_root/teleo-canonical.dump" > "$remote_root/teleo-canonical.dump.sha256" printf 'snapshot_exported=true\n' >> "$remote_root/source-context.txt" systemctl show "$service" -p ActiveState -p SubState -p MainPID -p NRestarts -p ActiveEnterTimestamp --no-pager > "$remote_root/service-after.txt" +assert_service_healthy "$remote_root/service-after.txt" cmp -s "$remote_root/service-before.txt" "$remote_root/service-after.txt" tar -C "$remote_root" -cf - \ @@ -138,12 +246,13 @@ tar -C "$remote_root" -cf - \ teleo-canonical.toc \ source-manifest.jsonl \ source-context.txt \ + source-snapshot.json \ service-before.txt \ service-after.txt """ -def upload_manifest(ssh: list[str], target: str, remote_root: str, manifest: Path) -> None: +def upload_manifest(ssh: list[str], target: str, remote_root: str, manifest: bytes) -> None: command = ( f"umask 077; rm -rf {shlex.quote(remote_root)}; " f"mkdir -m 700 {shlex.quote(remote_root)}; " @@ -151,7 +260,7 @@ def upload_manifest(ssh: list[str], target: str, remote_root: str, manifest: Pat ) completed = subprocess.run( [*ssh, target, "--", command], - input=manifest.read_bytes(), + input=manifest, capture_output=True, check=False, ) @@ -206,8 +315,10 @@ def capture(args: argparse.Namespace) -> dict[str, Any]: output_dir.mkdir(mode=0o700) remote_root = f"/tmp/teleo-canonical-snapshot-{args.run_id}" ssh = ssh_base(args.ssh_key.resolve()) + manifest_sql = args.manifest.resolve().read_bytes() + manifest_sql_sha256 = sha256_bytes(manifest_sql) try: - upload_manifest(ssh, args.ssh_target, remote_root, args.manifest.resolve()) + upload_manifest(ssh, args.ssh_target, remote_root, manifest_sql) script = remote_capture_script( run_id=args.run_id, container=args.container, @@ -233,19 +344,38 @@ def capture(args: argparse.Namespace) -> dict[str, Any]: if expected_sha != actual_sha: raise RuntimeError("captured dump SHA-256 mismatch") manifest = load_manifest(output_dir / "source-manifest.jsonl") - before = (output_dir / "service-before.txt").read_text() - after = (output_dir / "service-after.txt").read_text() + snapshot = load_snapshot_metadata(output_dir / "source-snapshot.json") + before = load_service_state(output_dir / "service-before.txt") + after = load_service_state(output_dir / "service-after.txt") + source = { + "ssh_target": args.ssh_target, + "container": args.container, + "database": args.database, + "service": args.service, + } + source_manifest_sha256 = sha256(output_dir / "source-manifest.jsonl") + source_context_sha256 = sha256(output_dir / "source-context.txt") + provenance_binding = build_provenance_binding( + run_id=args.run_id, + authorization_ref=args.authorization_ref, + source=source, + snapshot=snapshot, + dump_sha256=actual_sha, + manifest_sql_sha256=manifest_sql_sha256, + source_manifest_sha256=source_manifest_sha256, + source_context_sha256=source_context_sha256, + ) return { "artifact": "vps_canonical_postgres_exported_snapshot", + "schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA, + "receipt_version": 2, "generated_at_utc": datetime.now(UTC).isoformat(), "status": "pass", - "source": { - "ssh_target": args.ssh_target, - "container": args.container, - "database": args.database, - "service": args.service, - }, + "run_id": args.run_id, + "capture_authorization_ref": args.authorization_ref, + "source": source, "snapshot_exported": True, + "snapshot": snapshot, "dump": { "path": str(dump_path), "bytes": dump_path.stat().st_size, @@ -254,10 +384,18 @@ def capture(args: argparse.Namespace) -> dict[str, Any]: }, "manifest": { "path": str(output_dir / "source-manifest.jsonl"), + "sha256": source_manifest_sha256, + "manifest_sql_sha256": manifest_sql_sha256, "table_count": len(manifest["tables"]), "total_rows": sum(int(row["row_count"]) for row in manifest["tables"].values()), "application_roles": manifest["singleton"]["application_roles"]["items"], }, + "source_context": { + "path": str(output_dir / "source-context.txt"), + "sha256": source_context_sha256, + }, + "source_service": {"before": before, "after": after, "unchanged": before == after}, + "provenance_binding": provenance_binding, "service_unchanged": before == after, "production_db_mutated": False, "telegram_send_attempted": False, @@ -269,7 +407,7 @@ def cleanup_failed_output(output_dir: Path, *, preexisting: bool) -> None: shutil.rmtree(output_dir) -def parse_args() -> argparse.Namespace: +def parse_args(argv: list[str] | None = None) -> argparse.Namespace: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--execute", action="store_true") parser.add_argument("--ssh-target", required=True) @@ -279,9 +417,10 @@ def parse_args() -> argparse.Namespace: parser.add_argument("--service", default="leoclean-gateway.service") parser.add_argument("--manifest", default=Path("ops/postgres_parity_manifest.sql"), type=Path) parser.add_argument("--run-id", required=True) + parser.add_argument("--authorization-ref", required=True) parser.add_argument("--output-dir", required=True, type=Path) parser.add_argument("--timeout", default=180, type=int) - args = parser.parse_args() + args = parser.parse_args(argv) for value, flag in ((args.container, "--container"), (args.database, "--database"), (args.service, "--service")): if not SAFE_NAME_RE.fullmatch(value): parser.error(f"{flag} contains unsafe characters") @@ -289,6 +428,8 @@ def parse_args() -> argparse.Namespace: parser.error("--ssh-target must be a host or user@host without options") if not SAFE_RUN_RE.fullmatch(args.run_id): parser.error("--run-id must be 8-64 safe characters") + if not SAFE_AUTHORIZATION_REF_RE.fullmatch(args.authorization_ref): + parser.error("--authorization-ref must be 8-256 safe metadata characters") if not args.ssh_key.is_file(): parser.error("--ssh-key must be an existing file") if not args.manifest.is_file(): @@ -298,8 +439,12 @@ def parse_args() -> argparse.Namespace: json.dumps( { "artifact": "vps_canonical_postgres_exported_snapshot_plan", + "schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA, "status": "dry_run", + "run_id": args.run_id, + "capture_authorization_ref": args.authorization_ref, "source": f"{args.ssh_target}:{args.container}/{args.database}", + "manifest_sql_sha256": sha256(args.manifest.resolve()), "output_dir": str(args.output_dir), "mutates_production_db": False, }, @@ -321,16 +466,24 @@ def main() -> int: cleanup_failed_output(output_dir, preexisting=output_preexisted) payload = { "artifact": "vps_canonical_postgres_exported_snapshot", + "schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA, "generated_at_utc": datetime.now(UTC).isoformat(), "status": "fail", + "run_id": args.run_id, + "capture_authorization_ref": args.authorization_ref, "error": str(exc), } - print(json.dumps(payload, indent=2, sort_keys=True)) + receipt = ( + output_dir.parent / f".{output_dir.name}-{args.run_id}-failure.json" + if output_preexisted + else output_dir / "receipt.json" + ) + write_private_json(receipt, payload) + print_private_receipt_summary(receipt, payload) return 1 receipt = args.output_dir.resolve() / "receipt.json" - receipt.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") - os.chmod(receipt, 0o600) - print(json.dumps(payload, indent=2, sort_keys=True)) + write_private_json(receipt, payload) + print_private_receipt_summary(receipt, payload) return 0 diff --git a/ops/check_observatory_read_adapter_gcp_preflight.py b/ops/check_observatory_read_adapter_gcp_preflight.py new file mode 100644 index 0000000..8c75e43 --- /dev/null +++ b/ops/check_observatory_read_adapter_gcp_preflight.py @@ -0,0 +1,669 @@ +#!/usr/bin/env python3 +"""Run the adapter-specific, non-mutating GCP staging preflight.""" + +from __future__ import annotations + +import argparse +import json +import re +import subprocess +from collections.abc import Callable +from dataclasses import asdict, dataclass +from pathlib import Path + +PROJECT = "teleo-501523" +PROJECT_NUMBER = "785938879453" +REGION = "europe-west6" +NETWORK = "teleo-staging-net" +SUBNET = "teleo-staging-europe-west6" +INSTANCE = "teleo-pgvector-standby" +SERVICE = "observatory-read-adapter-staging" +DEPLOY_SERVICE_ACCOUNT = f"sa-observatory-deployer@{PROJECT}.iam.gserviceaccount.com" +RUNTIME_SERVICE_ACCOUNT = f"sa-observatory-read-adapter@{PROJECT}.iam.gserviceaccount.com" +BUILD_SERVICE_ACCOUNT = f"sa-artifact-builder@{PROJECT}.iam.gserviceaccount.com" +DB_IAM_USER = f"sa-observatory-read-adapter@{PROJECT}.iam" +SECRET = "observatory-read-api-key-staging" +REPOSITORY = "teleo" +WIF_POOL = "github-actions" +DEPLOY_WIF_PROVIDER = "observatory-read-adapter-main" +DEPLOY_WORKFLOW_REF = ( + "living-ip/teleo-infrastructure/.github/workflows/" + "gcp-observatory-read-adapter.yml@refs/heads/main" +) +DEPLOY_WIF_ATTRIBUTE_CONDITION = ( + "assertion.repository=='living-ip/teleo-infrastructure' && " + "assertion.ref=='refs/heads/main' && " + f"assertion.workflow_ref=='{DEPLOY_WORKFLOW_REF}'" +) +DEPLOY_WIF_MEMBER = ( + f"principal://iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/" + f"workloadIdentityPools/{WIF_POOL}/subject/" + "repo:living-ip/teleo-infrastructure:ref:refs/heads/main" +) +PREFLIGHT_ROLE = f"projects/{PROJECT}/roles/observatoryStagingPreflight" +DEPLOY_ROLE = f"projects/{PROJECT}/roles/observatoryStagingDeployer" +PREFLIGHT_PERMISSIONS = { + "artifactregistry.repositories.getIamPolicy", + "cloudsql.instances.get", + "cloudsql.users.list", + "compute.networks.get", + "compute.subnetworks.get", + "iam.serviceAccounts.get", + "iam.serviceAccounts.getIamPolicy", + "iam.roles.get", + "iam.workloadIdentityPoolProviders.get", + "resourcemanager.projects.get", + "resourcemanager.projects.getIamPolicy", + "run.operations.get", + "run.services.get", + "run.services.getIamPolicy", + "secretmanager.secrets.getIamPolicy", + "serviceusage.services.get", + "serviceusage.services.use", +} +DEPLOY_PERMISSIONS = { + "run.operations.get", + "run.services.create", + "run.services.get", + "run.services.setIamPolicy", + "run.services.update", +} +REQUIRED_APIS = ( + "artifactregistry.googleapis.com", + "compute.googleapis.com", + "iamcredentials.googleapis.com", + "run.googleapis.com", + "secretmanager.googleapis.com", + "sqladmin.googleapis.com", + "sts.googleapis.com", +) +IMAGE_RE = re.compile( + rf"^europe-west6-docker[.]pkg[.]dev/{PROJECT}/teleo/observatory-read-adapter:" + r"[0-9a-f]{40}-[0-9]+-[1-9][0-9]*@sha256:[0-9a-f]{64}$" +) + + +@dataclass(frozen=True) +class Check: + name: str + status: str + detail: str + + +def sanitize_error(value: str) -> str: + value = re.sub(r"/[^\s]*/gha-creds-[^\s]+[.]json", "", value) + return " ".join(value.strip().split())[-600:] or "command_failed_without_stderr" + + +def run(command: list[str]) -> subprocess.CompletedProcess[str]: + try: + return subprocess.run(command, text=True, capture_output=True, timeout=30, check=False) + except subprocess.TimeoutExpired: + return subprocess.CompletedProcess(command, 124, stdout="", stderr="command_timed_out_after_30_seconds") + except OSError as exc: + return subprocess.CompletedProcess(command, 127, stdout="", stderr=f"command_unavailable:{exc}") + + +def command_check( + name: str, + command: list[str], + validate: Callable[[str], bool], + success_detail: str, +) -> Check: + result = run(command) + if result.returncode != 0: + return Check(name, "blocked", sanitize_error(result.stderr)) + try: + valid = validate(result.stdout) + except (AttributeError, IndexError, KeyError, TypeError, ValueError, json.JSONDecodeError) as exc: + return Check(name, "blocked", f"invalid_readback:{type(exc).__name__}") + return Check(name, "pass" if valid else "blocked", success_detail if valid else "unexpected_readback") + + +def cloud_sql_is_private_and_iam_enabled(value: str) -> bool: + payload = json.loads(value) + flags = { + item.get("name"): str(item.get("value", "")).lower() + for item in payload.get("settings", {}).get("databaseFlags", []) + } + ip_configuration = payload.get("settings", {}).get("ipConfiguration", {}) + address_types = {item.get("type") for item in payload.get("ipAddresses", [])} + return ( + payload.get("name") == INSTANCE + and payload.get("region") == REGION + and str(payload.get("databaseVersion", "")).startswith("POSTGRES_") + and ip_configuration.get("ipv4Enabled") is False + and str(ip_configuration.get("privateNetwork", "")).endswith(f"/networks/{NETWORK}") + and "PRIVATE" in address_types + and "PRIMARY" not in address_types + and flags.get("cloudsql.iam_authentication") == "on" + ) + + +def cloud_sql_iam_user_exists(value: str) -> bool: + users = json.loads(value) + return any( + user.get("name") == DB_IAM_USER and user.get("type") == "CLOUD_IAM_SERVICE_ACCOUNT" for user in users + ) + + +def subnet_is_exact(value: str) -> bool: + payload = json.loads(value) + return ( + payload.get("name") == SUBNET + and str(payload.get("region", "")).endswith(f"/regions/{REGION}") + and str(payload.get("network", "")).endswith(f"/networks/{NETWORK}") + ) + + +def secret_is_valid(value: str) -> bool: + try: + encoded = value.encode("ascii") + except UnicodeEncodeError: + return False + return value == value.strip() and 32 <= len(encoded) <= 512 + + +def normalized(value: str) -> str: + return " ".join(value.split()) + + +def policy_has_binding( + policy: dict[str, object], + role: str, + member: str, + *, + condition_title: str | None = None, + condition_expression: str | None = None, +) -> bool: + bindings = policy.get("bindings", []) + if not isinstance(bindings, list): + return False + for binding in bindings: + if not isinstance(binding, dict) or binding.get("role") != role: + continue + members = binding.get("members", []) + if not isinstance(members, list) or member not in members: + continue + condition = binding.get("condition") + if condition_title is None and condition_expression is None: + return condition in (None, {}) + if not isinstance(condition, dict): + continue + if condition_title is not None and condition.get("title") != condition_title: + continue + if condition_expression is not None and normalized(str(condition.get("expression", ""))) != normalized( + condition_expression + ): + continue + return True + return False + + +def member_roles(policy: dict[str, object], member: str) -> set[str]: + roles: set[str] = set() + bindings = policy.get("bindings", []) + if not isinstance(bindings, list): + return roles + for binding in bindings: + if not isinstance(binding, dict): + continue + members = binding.get("members", []) + role = binding.get("role") + if isinstance(members, list) and member in members and isinstance(role, str): + roles.add(role) + return roles + + +def project_iam_is_exact(value: str) -> bool: + policy = json.loads(value) + deployer = f"serviceAccount:{DEPLOY_SERVICE_ACCOUNT}" + runtime = f"serviceAccount:{RUNTIME_SERVICE_ACCOUNT}" + builder = f"serviceAccount:{BUILD_SERVICE_ACCOUNT}" + service_agent = ( + f"serviceAccount:service-{PROJECT_NUMBER}@serverless-robot-prod.iam.gserviceaccount.com" + ) + sql_expression = ( + f"resource.name == 'projects/{PROJECT}/instances/{INSTANCE}' && " + "resource.service == 'sqladmin.googleapis.com'" + ) + forbidden_deployer = { + "roles/artifactregistry.writer", + "roles/cloudsql.admin", + "roles/compute.admin", + "roles/editor", + "roles/owner", + "roles/run.admin", + "roles/run.developer", + "roles/secretmanager.admin", + } + forbidden_builder = forbidden_deployer | { + DEPLOY_ROLE, + PREFLIGHT_ROLE, + "roles/iam.serviceAccountUser", + "roles/secretmanager.secretAccessor", + } + return ( + policy_has_binding(policy, PREFLIGHT_ROLE, deployer) + and policy_has_binding(policy, DEPLOY_ROLE, deployer) + and policy_has_binding( + policy, + "roles/cloudsql.client", + runtime, + condition_title="observatory-exact-cloudsql-instance", + condition_expression=sql_expression, + ) + and policy_has_binding( + policy, + "roles/cloudsql.instanceUser", + runtime, + condition_title="observatory-exact-cloudsql-instance", + condition_expression=sql_expression, + ) + and policy_has_binding(policy, "roles/run.serviceAgent", service_agent) + and not (member_roles(policy, deployer) & forbidden_deployer) + and not (member_roles(policy, builder) & forbidden_builder) + ) + + +def role_has_exact_permissions(value: str, expected: set[str]) -> bool: + payload = json.loads(value) + permissions = payload.get("includedPermissions", []) + return isinstance(permissions, list) and set(permissions) == expected and payload.get("deleted") is not True + + +def wif_provider_is_exact(value: str) -> bool: + payload = json.loads(value) + mapping = payload.get("attributeMapping", {}) + expected_mapping = { + "google.subject": "assertion.sub", + "attribute.repository": "assertion.repository", + "attribute.ref": "assertion.ref", + "attribute.workflow_ref": "assertion.workflow_ref", + } + return ( + payload.get("state") == "ACTIVE" + and mapping == expected_mapping + and normalized(str(payload.get("attributeCondition", ""))) == normalized(DEPLOY_WIF_ATTRIBUTE_CONDITION) + ) + + +def deployer_service_account_policy_is_exact(value: str) -> bool: + policy = json.loads(value) + return policy_has_binding(policy, "roles/iam.workloadIdentityUser", DEPLOY_WIF_MEMBER) + + +def runtime_service_account_policy_is_exact(value: str) -> bool: + policy = json.loads(value) + deployer = f"serviceAccount:{DEPLOY_SERVICE_ACCOUNT}" + builder = f"serviceAccount:{BUILD_SERVICE_ACCOUNT}" + return policy_has_binding(policy, "roles/iam.serviceAccountUser", deployer) and not member_roles(policy, builder) + + +def artifact_policy_is_exact(value: str) -> bool: + policy = json.loads(value) + deployer = f"serviceAccount:{DEPLOY_SERVICE_ACCOUNT}" + return policy_has_binding(policy, "roles/artifactregistry.reader", deployer) + + +def secret_policy_is_exact(value: str) -> bool: + policy = json.loads(value) + deployer = f"serviceAccount:{DEPLOY_SERVICE_ACCOUNT}" + runtime = f"serviceAccount:{RUNTIME_SERVICE_ACCOUNT}" + builder = f"serviceAccount:{BUILD_SERVICE_ACCOUNT}" + return ( + policy_has_binding(policy, "roles/secretmanager.secretAccessor", deployer) + and policy_has_binding(policy, "roles/secretmanager.secretAccessor", runtime) + and policy_has_binding(policy, "roles/secretmanager.viewer", deployer) + and not member_roles(policy, builder) + ) + + +def api_key_secret_check() -> Check: + version_result = run( + [ + "gcloud", + "secrets", + "versions", + "list", + SECRET, + "--project", + PROJECT, + "--filter=state=ENABLED", + "--sort-by=~createTime", + "--limit=1", + "--format=value(name)", + ] + ) + if version_result.returncode != 0: + return Check("api_key_secret", "blocked", sanitize_error(version_result.stderr)) + version = version_result.stdout.strip().rsplit("/", 1)[-1] + if not version.isdigit(): + return Check("api_key_secret", "blocked", "enabled_secret_version_missing") + secret_result = run( + [ + "gcloud", + "secrets", + "versions", + "access", + version, + "--secret", + SECRET, + "--project", + PROJECT, + ] + ) + if secret_result.returncode != 0: + return Check("api_key_secret", "blocked", sanitize_error(secret_result.stderr)) + valid = secret_is_valid(secret_result.stdout) + detail = f"enabled_version_{version}_valid_length_value_redacted" if valid else "invalid_secret_value" + return Check("api_key_secret", "pass" if valid else "blocked", detail) + + +def existing_service_policy_check() -> Check: + describe = run( + [ + "gcloud", + "run", + "services", + "describe", + SERVICE, + "--project", + PROJECT, + "--region", + REGION, + "--format=value(metadata.name)", + ] + ) + if describe.returncode != 0: + error = sanitize_error(describe.stderr) + if "not found" in error.lower() or "not_found" in error.lower(): + return Check("existing_service_policy", "pass", "service_not_created_yet") + return Check("existing_service_policy", "blocked", error) + if describe.stdout.strip() != SERVICE: + return Check("existing_service_policy", "blocked", "unexpected_service_readback") + policy_result = run( + [ + "gcloud", + "run", + "services", + "get-iam-policy", + SERVICE, + "--project", + PROJECT, + "--region", + REGION, + "--format=json", + ] + ) + if policy_result.returncode != 0: + return Check("existing_service_policy", "blocked", sanitize_error(policy_result.stderr)) + try: + policy = json.loads(policy_result.stdout) + builder = f"serviceAccount:{BUILD_SERVICE_ACCOUNT}" + valid = not member_roles(policy, builder) + except (AttributeError, TypeError, ValueError, json.JSONDecodeError) as exc: + return Check("existing_service_policy", "blocked", f"invalid_readback:{type(exc).__name__}") + return Check( + "existing_service_policy", + "pass" if valid else "blocked", + "artifact_builder_absent" if valid else "artifact_builder_has_service_binding", + ) + + +def build_checks(image_ref: str) -> list[Check]: + image_name, image_digest = image_ref.rsplit("@", 1) if "@" in image_ref else (image_ref, "") + digest_ref = f"{image_name.rsplit(':', 1)[0]}@{image_digest}" if image_digest else image_ref + checks = [ + command_check( + "active_identity", + ["gcloud", "auth", "list", "--filter=status:ACTIVE", "--format=value(account)"], + lambda value: value.strip() == DEPLOY_SERVICE_ACCOUNT, + "dedicated_observatory_deployer", + ), + command_check( + "project", + ["gcloud", "projects", "describe", PROJECT, "--format=value(projectId)"], + lambda value: value.strip() == PROJECT, + PROJECT, + ), + ] + checks.extend( + command_check( + f"api_{api.removesuffix('.googleapis.com')}", + ["gcloud", "services", "describe", api, "--project", PROJECT, "--format=value(state)"], + lambda value: value.strip() == "ENABLED", + "enabled", + ) + for api in REQUIRED_APIS + ) + checks.extend( + [ + command_check( + "deployer_wif_provider", + [ + "gcloud", + "iam", + "workload-identity-pools", + "providers", + "describe", + DEPLOY_WIF_PROVIDER, + "--project", + PROJECT, + "--location", + "global", + "--workload-identity-pool", + WIF_POOL, + "--format=json", + ], + wif_provider_is_exact, + "main_and_exact_workflow_only", + ), + command_check( + "preflight_custom_role", + [ + "gcloud", + "iam", + "roles", + "describe", + "observatoryStagingPreflight", + "--project", + PROJECT, + "--format=json", + ], + lambda value: role_has_exact_permissions(value, PREFLIGHT_PERMISSIONS), + "exact_read_only_permissions", + ), + command_check( + "deploy_custom_role", + [ + "gcloud", + "iam", + "roles", + "describe", + "observatoryStagingDeployer", + "--project", + PROJECT, + "--format=json", + ], + lambda value: role_has_exact_permissions(value, DEPLOY_PERMISSIONS), + "five_permissions_no_delete", + ), + command_check( + "project_iam", + ["gcloud", "projects", "get-iam-policy", PROJECT, "--format=json"], + project_iam_is_exact, + "required_bindings_present_broad_roles_absent", + ), + command_check( + "deployer_wif_binding", + [ + "gcloud", + "iam", + "service-accounts", + "get-iam-policy", + DEPLOY_SERVICE_ACCOUNT, + "--project", + PROJECT, + "--format=json", + ], + deployer_service_account_policy_is_exact, + "exact_main_subject_workload_identity_user", + ), + command_check( + "runtime_act_as_policy", + [ + "gcloud", + "iam", + "service-accounts", + "get-iam-policy", + RUNTIME_SERVICE_ACCOUNT, + "--project", + PROJECT, + "--format=json", + ], + runtime_service_account_policy_is_exact, + "deployer_only_artifact_builder_absent", + ), + command_check( + "artifact_repository_policy", + [ + "gcloud", + "artifacts", + "repositories", + "get-iam-policy", + REPOSITORY, + "--project", + PROJECT, + "--location", + REGION, + "--format=json", + ], + artifact_policy_is_exact, + "deployer_reader", + ), + command_check( + "secret_policy", + [ + "gcloud", + "secrets", + "get-iam-policy", + SECRET, + "--project", + PROJECT, + "--format=json", + ], + secret_policy_is_exact, + "runtime_and_deployer_only_artifact_builder_absent", + ), + existing_service_policy_check(), + ] + ) + checks.extend( + [ + command_check( + "runtime_service_account", + [ + "gcloud", + "iam", + "service-accounts", + "describe", + RUNTIME_SERVICE_ACCOUNT, + "--project", + PROJECT, + "--format=value(email)", + ], + lambda value: value.strip() == RUNTIME_SERVICE_ACCOUNT, + RUNTIME_SERVICE_ACCOUNT, + ), + command_check( + "vpc_network", + [ + "gcloud", + "compute", + "networks", + "describe", + NETWORK, + "--project", + PROJECT, + "--format=value(name)", + ], + lambda value: value.strip() == NETWORK, + NETWORK, + ), + command_check( + "vpc_subnet", + [ + "gcloud", + "compute", + "networks", + "subnets", + "describe", + SUBNET, + "--project", + PROJECT, + "--region", + REGION, + "--format=json", + ], + subnet_is_exact, + f"{NETWORK}/{REGION}/{SUBNET}", + ), + command_check( + "cloud_sql_private_iam", + ["gcloud", "sql", "instances", "describe", INSTANCE, "--project", PROJECT, "--format=json"], + cloud_sql_is_private_and_iam_enabled, + f"{PROJECT}:{REGION}:{INSTANCE}:private_ip_only:iam_authentication_on", + ), + command_check( + "cloud_sql_iam_user", + ["gcloud", "sql", "users", "list", "--instance", INSTANCE, "--project", PROJECT, "--format=json"], + cloud_sql_iam_user_exists, + DB_IAM_USER, + ), + api_key_secret_check(), + command_check( + "immutable_image", + ["gcloud", "artifacts", "docker", "images", "describe", digest_ref, "--project", PROJECT, "--format=json"], + lambda _value: bool(IMAGE_RE.fullmatch(image_ref)), + image_digest or "missing_digest", + ), + ] + ) + return checks + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--image-ref", required=True) + parser.add_argument("--output", type=Path) + args = parser.parse_args() + + checks = build_checks(args.image_ref) + pass_count = sum(check.status == "pass" for check in checks) + blocked_count = len(checks) - pass_count + receipt = { + "schema": "livingip.observatory-read-adapter-gcp-preflight.v1", + "project": PROJECT, + "required_tier": "T3_live_readonly", + "current_tier": "T2_authenticated_gcp_preflight", + "ready_for_staging_deploy": blocked_count == 0, + "claim_ceiling": "Preflight only; T3 requires the live private Cloud SQL claim and negative receipts.", + "identity": DEPLOY_SERVICE_ACCOUNT, + "checks": [asdict(check) for check in checks], + "pass_count": pass_count, + "blocked_count": blocked_count, + "secret_values_retained": False, + "database_role_live_verification_deferred_to_fail_closed_canary": True, + "production_repoint_executed": False, + } + rendered = json.dumps(receipt, indent=2, sort_keys=True) + "\n" + if args.output: + args.output.write_text(rendered, encoding="utf-8") + else: + print(rendered, end="") + return 0 if blocked_count == 0 else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/ops/plan_observatory_read_adapter_gcp.py b/ops/plan_observatory_read_adapter_gcp.py new file mode 100644 index 0000000..1bd236c --- /dev/null +++ b/ops/plan_observatory_read_adapter_gcp.py @@ -0,0 +1,475 @@ +#!/usr/bin/env python3 +"""Emit the unexecuted least-privilege GCP staging packet for Observatory.""" + +from __future__ import annotations + +import argparse +import json +import shlex + +PROJECT = "teleo-501523" +PROJECT_NUMBER = "785938879453" +REGION = "europe-west6" +GITHUB_REPOSITORY = "living-ip/teleo-infrastructure" +WIF_POOL = "github-actions" +DEPLOY_WIF_PROVIDER = "observatory-read-adapter-main" +DEPLOY_WORKFLOW_REF = ( + f"{GITHUB_REPOSITORY}/.github/workflows/gcp-observatory-read-adapter.yml@refs/heads/main" +) +DEPLOY_WIF_ATTRIBUTE_MAPPING = ( + "google.subject=assertion.sub,attribute.repository=assertion.repository," + "attribute.ref=assertion.ref,attribute.workflow_ref=assertion.workflow_ref" +) +DEPLOY_WIF_ATTRIBUTE_CONDITION = ( + "assertion.repository=='living-ip/teleo-infrastructure' && " + "assertion.ref=='refs/heads/main' && " + f"assertion.workflow_ref=='{DEPLOY_WORKFLOW_REF}'" +) +BUILD_SERVICE_ACCOUNT = f"sa-artifact-builder@{PROJECT}.iam.gserviceaccount.com" +DEPLOY_SERVICE_ACCOUNT_ID = "sa-observatory-deployer" +DEPLOY_SERVICE_ACCOUNT = f"{DEPLOY_SERVICE_ACCOUNT_ID}@{PROJECT}.iam.gserviceaccount.com" +RUNTIME_SERVICE_ACCOUNT_ID = "sa-observatory-read-adapter" +RUNTIME_SERVICE_ACCOUNT = f"{RUNTIME_SERVICE_ACCOUNT_ID}@{PROJECT}.iam.gserviceaccount.com" +DB_IAM_USER = f"{RUNTIME_SERVICE_ACCOUNT_ID}@{PROJECT}.iam" +SERVICE = "observatory-read-adapter-staging" +INSTANCE = "teleo-pgvector-standby" +NETWORK = "teleo-staging-net" +SUBNET = "teleo-staging-europe-west6" +SECRET = "observatory-read-api-key-staging" +REPOSITORY = "teleo" +PREFLIGHT_ROLE_ID = "observatoryStagingPreflight" +PREFLIGHT_ROLE = f"projects/{PROJECT}/roles/{PREFLIGHT_ROLE_ID}" +DEPLOY_ROLE_ID = "observatoryStagingDeployer" +DEPLOY_ROLE = f"projects/{PROJECT}/roles/{DEPLOY_ROLE_ID}" +PREFLIGHT_PERMISSIONS = ( + "artifactregistry.repositories.getIamPolicy", + "cloudsql.instances.get", + "cloudsql.users.list", + "compute.networks.get", + "compute.subnetworks.get", + "iam.serviceAccounts.get", + "iam.serviceAccounts.getIamPolicy", + "iam.roles.get", + "iam.workloadIdentityPoolProviders.get", + "resourcemanager.projects.get", + "resourcemanager.projects.getIamPolicy", + "run.operations.get", + "run.services.get", + "run.services.getIamPolicy", + "secretmanager.secrets.getIamPolicy", + "serviceusage.services.get", + "serviceusage.services.use", +) +DEPLOY_PERMISSIONS = ( + "run.operations.get", + "run.services.create", + "run.services.get", + "run.services.setIamPolicy", + "run.services.update", +) +REQUIRED_APIS = ( + "artifactregistry.googleapis.com", + "compute.googleapis.com", + "iamcredentials.googleapis.com", + "run.googleapis.com", + "secretmanager.googleapis.com", + "sqladmin.googleapis.com", + "sts.googleapis.com", +) +FORBIDDEN_DEPLOYER_ROLES = ( + "roles/artifactregistry.writer", + "roles/cloudsql.admin", + "roles/compute.admin", + "roles/editor", + "roles/owner", + "roles/run.admin", + "roles/run.developer", + "roles/secretmanager.admin", +) + + +def shell_join(parts: list[str]) -> str: + return " ".join(shlex.quote(part) for part in parts) + + +def github_wif_member() -> str: + return ( + f"principal://iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/" + f"workloadIdentityPools/{WIF_POOL}/subject/repo:{GITHUB_REPOSITORY}:ref:refs/heads/main" + ) + + +def service_account_member(account: str) -> str: + return f"serviceAccount:{account}" + + +def project_binding(member: str, role: str, condition: str | None = None) -> str: + command = [ + "gcloud", + "projects", + "add-iam-policy-binding", + PROJECT, + "--member", + member, + "--role", + role, + ] + if condition: + command.extend(["--condition", condition]) + return shell_join(command) + + +def custom_role_commands( + role_id: str, + title: str, + description: str, + permissions: tuple[str, ...], +) -> list[str]: + common = [ + "--project", + PROJECT, + "--title", + title, + "--description", + description, + "--permissions", + ",".join(permissions), + "--stage", + "GA", + ] + return [ + ( + f"gcloud iam roles describe {shlex.quote(role_id)} --project {shlex.quote(PROJECT)} " + "|| " + + shell_join(["gcloud", "iam", "roles", "create", role_id, *common]) + ), + shell_join(["gcloud", "iam", "roles", "update", role_id, *common]), + ] + + +def build_plan() -> dict[str, object]: + deployer_member = service_account_member(DEPLOY_SERVICE_ACCOUNT) + runtime_member = service_account_member(RUNTIME_SERVICE_ACCOUNT) + cloud_sql_condition = ( + "expression=resource.name == " + f"'projects/{PROJECT}/instances/{INSTANCE}' && resource.service == 'sqladmin.googleapis.com'," + "title=observatory-exact-cloudsql-instance" + ) + create_accounts = [ + ( + DEPLOY_SERVICE_ACCOUNT, + DEPLOY_SERVICE_ACCOUNT_ID, + "Observatory staging deploy and canary", + ), + ( + RUNTIME_SERVICE_ACCOUNT, + RUNTIME_SERVICE_ACCOUNT_ID, + "Observatory private Cloud SQL reader", + ), + ] + commands = [ + shell_join(["gcloud", "projects", "describe", PROJECT, "--format=value(projectId)"]), + shell_join(["gcloud", "services", "enable", *REQUIRED_APIS, "--project", PROJECT]), + project_binding( + f"serviceAccount:service-{PROJECT_NUMBER}@serverless-robot-prod.iam.gserviceaccount.com", + "roles/run.serviceAgent", + ), + ] + for email, account_id, display_name in create_accounts: + commands.append( + f"gcloud iam service-accounts describe {shlex.quote(email)} --project {shlex.quote(PROJECT)} " + f"|| {shell_join(['gcloud', 'iam', 'service-accounts', 'create', account_id, '--project', PROJECT, '--display-name', display_name])}" + ) + commands.extend( + [ + ( + "gcloud iam workload-identity-pools providers describe " + + shlex.quote(DEPLOY_WIF_PROVIDER) + + " --project " + + shlex.quote(PROJECT) + + " --location global --workload-identity-pool " + + shlex.quote(WIF_POOL) + + " || " + + shell_join( + [ + "gcloud", + "iam", + "workload-identity-pools", + "providers", + "create-oidc", + DEPLOY_WIF_PROVIDER, + "--project", + PROJECT, + "--location", + "global", + "--workload-identity-pool", + WIF_POOL, + "--issuer-uri", + "https://token.actions.githubusercontent.com/", + "--attribute-mapping", + DEPLOY_WIF_ATTRIBUTE_MAPPING, + "--attribute-condition", + DEPLOY_WIF_ATTRIBUTE_CONDITION, + ] + ) + ), + shell_join( + [ + "gcloud", + "iam", + "workload-identity-pools", + "providers", + "update-oidc", + DEPLOY_WIF_PROVIDER, + "--project", + PROJECT, + "--location", + "global", + "--workload-identity-pool", + WIF_POOL, + "--attribute-mapping", + DEPLOY_WIF_ATTRIBUTE_MAPPING, + "--attribute-condition", + DEPLOY_WIF_ATTRIBUTE_CONDITION, + ] + ), + shell_join( + [ + "gcloud", + "iam", + "service-accounts", + "add-iam-policy-binding", + DEPLOY_SERVICE_ACCOUNT, + "--project", + PROJECT, + "--member", + github_wif_member(), + "--role", + "roles/iam.workloadIdentityUser", + ] + ), + ] + ) + commands.extend( + custom_role_commands( + PREFLIGHT_ROLE_ID, + "Observatory staging preflight", + "Read only the exact resource classes required before the private staging deploy", + PREFLIGHT_PERMISSIONS, + ) + ) + commands.extend( + custom_role_commands( + DEPLOY_ROLE_ID, + "Observatory staging deployer", + "Create or update Cloud Run without delete or broad infrastructure permissions", + DEPLOY_PERMISSIONS, + ) + ) + commands.extend( + [ + project_binding(deployer_member, PREFLIGHT_ROLE), + project_binding(deployer_member, DEPLOY_ROLE), + shell_join( + [ + "gcloud", + "artifacts", + "repositories", + "add-iam-policy-binding", + REPOSITORY, + "--project", + PROJECT, + "--location", + REGION, + "--member", + deployer_member, + "--role", + "roles/artifactregistry.reader", + ] + ), + shell_join( + [ + "gcloud", + "iam", + "service-accounts", + "add-iam-policy-binding", + RUNTIME_SERVICE_ACCOUNT, + "--project", + PROJECT, + "--member", + deployer_member, + "--role", + "roles/iam.serviceAccountUser", + ] + ), + project_binding(runtime_member, "roles/cloudsql.client", cloud_sql_condition), + project_binding(runtime_member, "roles/cloudsql.instanceUser", cloud_sql_condition), + f"gcloud secrets describe {shlex.quote(SECRET)} --project {shlex.quote(PROJECT)} " + f"|| {shell_join(['gcloud', 'secrets', 'create', SECRET, '--project', PROJECT, '--replication-policy', 'automatic'])}", + ] + ) + for member in (runtime_member, deployer_member): + commands.append( + shell_join( + [ + "gcloud", + "secrets", + "add-iam-policy-binding", + SECRET, + "--project", + PROJECT, + "--member", + member, + "--role", + "roles/secretmanager.secretAccessor", + ] + ) + ) + commands.extend( + [ + shell_join( + [ + "gcloud", + "secrets", + "add-iam-policy-binding", + SECRET, + "--project", + PROJECT, + "--member", + deployer_member, + "--role", + "roles/secretmanager.viewer", + ] + ), + ( + "if ! gcloud secrets versions list " + + shlex.quote(SECRET) + + " --project " + + shlex.quote(PROJECT) + + " --filter='state=ENABLED' --limit=1 --format='value(name)' | grep -q .; then " + + "openssl rand -hex 32 | tr -d '\\n' | gcloud secrets versions add " + + shlex.quote(SECRET) + + " --project " + + shlex.quote(PROJECT) + + " --data-file=-; fi" + ), + ( + "if ! gcloud sql users list --instance " + + shlex.quote(INSTANCE) + + " --project " + + shlex.quote(PROJECT) + + " --filter='type=CLOUD_IAM_SERVICE_ACCOUNT' --format='value(name)' | grep -Fxq " + + shlex.quote(DB_IAM_USER) + + "; then " + + shell_join( + [ + "gcloud", + "sql", + "users", + "create", + DB_IAM_USER, + "--instance", + INSTANCE, + "--project", + PROJECT, + "--type", + "CLOUD_IAM_SERVICE_ACCOUNT", + ] + ) + + "; fi" + ), + ] + ) + + return { + "schema": "livingip.observatory-read-adapter-gcp-plan.v1", + "project": PROJECT, + "required_tier": "T3_live_readonly", + "current_tier": "T2_unexecuted_least_privilege_packet", + "claim_ceiling": "Plan only; no IAM, API, secret, Cloud SQL, Cloud Run, or routing mutation has run.", + "identities": { + "build": BUILD_SERVICE_ACCOUNT, + "deploy_and_canary": DEPLOY_SERVICE_ACCOUNT, + "runtime": RUNTIME_SERVICE_ACCOUNT, + "database_principal": DB_IAM_USER, + }, + "kept_narrow": { + "artifact_builder": "image build and push only; never deploy or receive broad infra roles", + "deployer": "workflow-specific main-only WIF plus a five-permission Cloud Run role, repository read, exact runtime actAs, exact secret access", + "runtime": "exact Cloud SQL instance connect/login plus exact secret access", + }, + "github_wif_provider": ( + f"projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{WIF_POOL}/providers/{DEPLOY_WIF_PROVIDER}" + ), + "github_wif_member": github_wif_member(), + "required_apis": list(REQUIRED_APIS), + "preflight_custom_role": { + "name": PREFLIGHT_ROLE, + "permissions": list(PREFLIGHT_PERMISSIONS), + }, + "deploy_custom_role": { + "name": DEPLOY_ROLE, + "permissions": list(DEPLOY_PERMISSIONS), + }, + "conditions": { + "cloud_sql": cloud_sql_condition, + "deployer_wif": DEPLOY_WIF_ATTRIBUTE_CONDITION, + }, + "commands": commands, + "private_database_action": { + "location": "existing authenticated private GCP VM database-admin path", + "command": ( + f"OBSERVATORY_DB_IAM_USER={DB_IAM_USER} psql --dbname=teleo_canonical " + "--set=ON_ERROR_STOP=1 --file=ops/observatory_read_role.sql" + ), + "required_receipt": ( + "database=teleo_canonical, authorization_role=kb_observatory_read, exact database principal, " + "required_reads=true, effective_table_writes_denied=true" + ), + }, + "cloud_run_service_agent_check": { + "principal": f"service-{PROJECT_NUMBER}@serverless-robot-prod.iam.gserviceaccount.com", + "required_role": "roles/run.serviceAgent", + "reason": "Default Cloud Run service-agent role carries Direct VPC permissions; do not grant them to the runtime identity.", + "empty_readback_action": "Stop. Repair the Google-managed service-agent grant as an administrator before preflight.", + }, + "post_apply_checks": [ + f"gcloud projects describe {PROJECT} --format=value(projectId)", + "python3 ops/check_gcp_infra_readiness.py", + "Require the adapter preflight to verify the dedicated WIF provider, custom roles, IAM bindings, private Cloud SQL, secret, and immutable image.", + ( + "gh workflow run gcp-observatory-read-adapter.yml --repo " + f"{GITHUB_REPOSITORY} --ref main -f action=preflight_staging" + ), + "Require a passing observatory-read-adapter-preflight artifact before action=deploy_staging.", + ], + "forbidden_deployer_roles": list(FORBIDDEN_DEPLOYER_ROLES), + "forbidden_actions": [ + "grant deploy or infra roles to sa-artifact-builder", + "reuse a pre-existing image tag instead of building the current workflow revision", + "add a public Cloud SQL address", + "expose a secret value in Git, logs, or a receipt", + "add a write route", + "change Vercel or production routing", + ], + "production_repoint_executed": False, + } + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--format", choices=("json", "shell"), default="json") + args = parser.parse_args() + plan = build_plan() + if args.format == "shell": + print("# Unexecuted staging packet. Review and run only as an authenticated teleo-501523 administrator.") + print("set -euo pipefail") + for command in plan["commands"]: + print(command) + else: + print(json.dumps(plan, indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/ops/postgres_parity_manifest.sql b/ops/postgres_parity_manifest.sql index 3166eba..ebeea10 100644 --- a/ops/postgres_parity_manifest.sql +++ b/ops/postgres_parity_manifest.sql @@ -55,11 +55,296 @@ select jsonb_build_object( 'bypass_rls', rolbypassrls ) order by rolname) from pg_roles - where rolname in ('kb_apply', 'kb_review')), + where rolname in ('kb_gate_owner', 'kb_apply', 'kb_review')), '[]'::jsonb ) )::text; +select jsonb_build_object( + 'kind', 'role_memberships', + 'items', coalesce( + (select jsonb_agg( + jsonb_build_object( + 'role', role_row.rolname, + 'member', member_row.rolname, + 'grantor', grantor_row.rolname, + 'admin_option', membership.admin_option, + 'inherit_option', membership.inherit_option, + 'set_option', membership.set_option + ) order by role_row.rolname, member_row.rolname, grantor_row.rolname) + from pg_auth_members membership + join pg_roles role_row on role_row.oid = membership.roleid + join pg_roles member_row on member_row.oid = membership.member + join pg_roles grantor_row on grantor_row.oid = membership.grantor + where role_row.rolname in ('kb_gate_owner', 'kb_apply', 'kb_review') + or member_row.rolname in ('kb_gate_owner', 'kb_apply', 'kb_review')), + '[]'::jsonb + ) +)::text; + +with ownership as ( + select + 'database:canonical_database'::text as sort_key, + 'database'::text as object_type, + null::text as schema, + 'canonical_database'::text as name, + null::text as identity_arguments, + pg_get_userbyid(database_row.datdba) as owner + from pg_database database_row + where database_row.datname = current_database() + + union all + + select + format('schema:%s', namespace.nspname), + 'schema', + namespace.nspname, + namespace.nspname, + null::text, + pg_get_userbyid(namespace.nspowner) + from pg_namespace namespace + where namespace.nspname in ('public', 'kb_stage') + + union all + + select + format('relation:%s.%s', namespace.nspname, relation.relname), + case relation.relkind + when 'r' then 'table' + when 'p' then 'partitioned_table' + when 'v' then 'view' + when 'm' then 'materialized_view' + when 'S' then 'sequence' + when 'f' then 'foreign_table' + end, + namespace.nspname, + relation.relname, + null::text, + pg_get_userbyid(relation.relowner) + from pg_class relation + join pg_namespace namespace on namespace.oid = relation.relnamespace + where namespace.nspname in ('public', 'kb_stage') + and relation.relkind in ('r', 'p', 'v', 'm', 'S', 'f') + + union all + + select + format( + 'function:%s.%s(%s)', + namespace.nspname, + procedure.proname, + pg_get_function_identity_arguments(procedure.oid) + ), + 'function', + namespace.nspname, + procedure.proname, + pg_get_function_identity_arguments(procedure.oid), + pg_get_userbyid(procedure.proowner) + from pg_proc procedure + join pg_namespace namespace on namespace.oid = procedure.pronamespace + where namespace.nspname in ('public', 'kb_stage') + + union all + + select + format('type:%s.%s', namespace.nspname, type_row.typname), + case type_row.typtype when 'd' then 'domain' else 'enum' end, + namespace.nspname, + type_row.typname, + null::text, + pg_get_userbyid(type_row.typowner) + from pg_type type_row + join pg_namespace namespace on namespace.oid = type_row.typnamespace + where namespace.nspname in ('public', 'kb_stage') + and type_row.typtype in ('d', 'e') +) +select jsonb_build_object( + 'kind', 'object_ownership', + 'items', coalesce( + jsonb_agg(to_jsonb(ownership) - 'sort_key' order by sort_key), + '[]'::jsonb + ) +)::text +from ownership; + +with acl_rows as ( + select + format( + 'database:canonical_database:%s:%s:%s', + acl.grantee, + acl.privilege_type, + acl.is_grantable + ) as sort_key, + 'database'::text as object_type, + null::text as schema, + 'canonical_database'::text as name, + null::text as column_name, + null::text as identity_arguments, + pg_get_userbyid(acl.grantor) as grantor, + case when acl.grantee = 0 then 'PUBLIC' else pg_get_userbyid(acl.grantee) end as grantee, + acl.privilege_type, + acl.is_grantable + from pg_database database_row + cross join lateral aclexplode(coalesce(database_row.datacl, acldefault('d', database_row.datdba))) acl + where database_row.datname = current_database() + + union all + + select + format( + 'schema:%s:%s:%s:%s', + namespace.nspname, + acl.grantee, + acl.privilege_type, + acl.is_grantable + ), + 'schema', + namespace.nspname, + namespace.nspname, + null::text, + null::text, + pg_get_userbyid(acl.grantor), + case when acl.grantee = 0 then 'PUBLIC' else pg_get_userbyid(acl.grantee) end, + acl.privilege_type, + acl.is_grantable + from pg_namespace namespace + cross join lateral aclexplode(coalesce(namespace.nspacl, acldefault('n', namespace.nspowner))) acl + where namespace.nspname in ('public', 'kb_stage') + + union all + + select + format( + 'relation:%s.%s:%s:%s:%s', + namespace.nspname, + relation.relname, + acl.grantee, + acl.privilege_type, + acl.is_grantable + ), + case when relation.relkind = 'S' then 'sequence' else 'relation' end, + namespace.nspname, + relation.relname, + null::text, + null::text, + pg_get_userbyid(acl.grantor), + case when acl.grantee = 0 then 'PUBLIC' else pg_get_userbyid(acl.grantee) end, + acl.privilege_type, + acl.is_grantable + from pg_class relation + join pg_namespace namespace on namespace.oid = relation.relnamespace + cross join lateral aclexplode( + coalesce( + relation.relacl, + acldefault(case when relation.relkind = 'S' then 'S'::"char" else 'r'::"char" end, relation.relowner) + ) + ) acl + where namespace.nspname in ('public', 'kb_stage') + and relation.relkind in ('r', 'p', 'v', 'm', 'S', 'f') + + union all + + select + format( + 'column:%s.%s.%s:%s:%s:%s', + namespace.nspname, + relation.relname, + attribute.attname, + acl.grantee, + acl.privilege_type, + acl.is_grantable + ), + 'column', + namespace.nspname, + relation.relname, + attribute.attname, + null::text, + pg_get_userbyid(acl.grantor), + case when acl.grantee = 0 then 'PUBLIC' else pg_get_userbyid(acl.grantee) end, + acl.privilege_type, + acl.is_grantable + from pg_attribute attribute + join pg_class relation on relation.oid = attribute.attrelid + join pg_namespace namespace on namespace.oid = relation.relnamespace + cross join lateral aclexplode(attribute.attacl) acl + where namespace.nspname in ('public', 'kb_stage') + and relation.relkind in ('r', 'p', 'v', 'm', 'f') + and attribute.attnum > 0 + and not attribute.attisdropped + + union all + + select + format( + 'function:%s.%s(%s):%s:%s:%s', + namespace.nspname, + procedure.proname, + pg_get_function_identity_arguments(procedure.oid), + acl.grantee, + acl.privilege_type, + acl.is_grantable + ), + 'function', + namespace.nspname, + procedure.proname, + null::text, + pg_get_function_identity_arguments(procedure.oid), + pg_get_userbyid(acl.grantor), + case when acl.grantee = 0 then 'PUBLIC' else pg_get_userbyid(acl.grantee) end, + acl.privilege_type, + acl.is_grantable + from pg_proc procedure + join pg_namespace namespace on namespace.oid = procedure.pronamespace + cross join lateral aclexplode(coalesce(procedure.proacl, acldefault('f', procedure.proowner))) acl + where namespace.nspname in ('public', 'kb_stage') + + union all + + select + format( + 'type:%s.%s:%s:%s:%s', + namespace.nspname, + type_row.typname, + acl.grantee, + acl.privilege_type, + acl.is_grantable + ), + 'type', + namespace.nspname, + type_row.typname, + null::text, + null::text, + pg_get_userbyid(acl.grantor), + case when acl.grantee = 0 then 'PUBLIC' else pg_get_userbyid(acl.grantee) end, + acl.privilege_type, + acl.is_grantable + from pg_type type_row + join pg_namespace namespace on namespace.oid = type_row.typnamespace + cross join lateral aclexplode(coalesce(type_row.typacl, acldefault('T', type_row.typowner))) acl + where namespace.nspname in ('public', 'kb_stage') + and type_row.typtype in ('d', 'e') +) +select jsonb_build_object( + 'kind', 'object_acl', + 'items', coalesce( + jsonb_agg( + to_jsonb(acl_rows) - 'sort_key' + order by + object_type, + schema nulls first, + name, + column_name nulls first, + identity_arguments nulls first, + grantor, + grantee, + privilege_type, + is_grantable + ), + '[]'::jsonb + ) +)::text +from acl_rows; + select jsonb_build_object( 'kind', 'columns', 'items', coalesce( diff --git a/ops/private_receipt_io.py b/ops/private_receipt_io.py new file mode 100644 index 0000000..1655006 --- /dev/null +++ b/ops/private_receipt_io.py @@ -0,0 +1,63 @@ +"""Atomic private JSON receipt output shared by GCP parity helpers.""" + +from __future__ import annotations + +import hashlib +import json +import os +import uuid +from pathlib import Path +from typing import Any + + +def file_sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def write_private_json(path: Path, payload: dict[str, Any]) -> None: + """Write one JSON receipt as mode 0600 without chmodding existing parents.""" + parent_preexisting = path.parent.exists() + path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) + if not path.parent.is_dir() or path.parent.is_symlink(): + raise ValueError("output parent must be a real directory") + if not parent_preexisting: + os.chmod(path.parent, 0o700) + temporary = path.parent / f".{path.name}.{uuid.uuid4().hex}.tmp" + descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + try: + with os.fdopen(descriptor, "w", encoding="utf-8") as handle: + json.dump(payload, handle, indent=2, sort_keys=True) + handle.write("\n") + handle.flush() + os.fsync(handle.fileno()) + os.replace(temporary, path) + os.chmod(path, 0o600) + finally: + try: + temporary.unlink() + except FileNotFoundError: + pass + + +def private_receipt_summary(path: Path, payload: dict[str, Any]) -> dict[str, Any]: + """Return a stdout-safe pointer without replaying private receipt contents.""" + failed_checks = payload.get("failed_checks") + problems = payload.get("problems") + error = payload.get("error") + return { + "artifact": payload.get("artifact"), + "status": payload.get("status"), + "output_name": path.name, + "output_sha256": file_sha256(path), + "failed_check_count": len(failed_checks) if isinstance(failed_checks, list) else None, + "problem_count": len(problems) if isinstance(problems, list) else None, + "error_type": error.get("type") if isinstance(error, dict) else (type(error).__name__ if error else None), + } + + +def print_private_receipt_summary(path: Path, payload: dict[str, Any]) -> None: + print(json.dumps(private_receipt_summary(path, payload), indent=2, sort_keys=True)) diff --git a/ops/redact_observatory_read_adapter_receipt.jq b/ops/redact_observatory_read_adapter_receipt.jq new file mode 100644 index 0000000..e8aa2c8 --- /dev/null +++ b/ops/redact_observatory_read_adapter_receipt.jq @@ -0,0 +1,29 @@ +{ + schema, + read_only, + provenance: { + gcp_project: .provenance.gcp_project, + cloud_sql_instance: .provenance.cloud_sql_instance, + database: .provenance.database, + database_principal: .provenance.database_principal, + authorization_role: .provenance.authorization_role, + transaction_read_only: .provenance.transaction_read_only, + write_privileges_denied: .provenance.write_privileges_denied, + service_revision: .provenance.service_revision + }, + canonical: { + claim: { + id: .canonical.claim.id, + type: .canonical.claim.type, + status: .canonical.claim.status + }, + evidence_count: (.canonical.evidence | length), + incoming_edge_count: ([.canonical.edges[] | select(.direction == "incoming")] | length), + outgoing_edge_count: ([.canonical.edges[] | select(.direction == "outgoing")] | length) + }, + proposal_ledger: { + distinct_from_canonical: .proposal_ledger.distinct_from_canonical, + status_counts: .proposal_ledger.status_counts, + related_count: (.proposal_ledger.related | length) + } +} diff --git a/ops/restore_gcp_generated_postgres_snapshot.py b/ops/restore_gcp_generated_postgres_snapshot.py index f1eb801..c14537a 100644 --- a/ops/restore_gcp_generated_postgres_snapshot.py +++ b/ops/restore_gcp_generated_postgres_snapshot.py @@ -16,34 +16,58 @@ import ipaddress import json import os import re +import shlex import signal import stat import subprocess import time +import urllib.error +import urllib.request import uuid from datetime import UTC, datetime from pathlib import Path from typing import Any try: - from .verify_postgres_parity_manifest import compare_manifests, load_manifest + from .private_receipt_io import print_private_receipt_summary + from .verify_postgres_parity_manifest import REVIEWED_MANIFEST_SQL_SHA256, compare_manifests, load_manifest except ImportError: # pragma: no cover - direct script execution on the GCP VM - from verify_postgres_parity_manifest import compare_manifests, load_manifest + from private_receipt_io import print_private_receipt_summary + from verify_postgres_parity_manifest import REVIEWED_MANIFEST_SQL_SHA256, compare_manifests, load_manifest DEFAULT_HOST = "10.61.0.3" DEFAULT_PROJECT = "teleo-501523" +DEFAULT_INSTANCE = "teleo-pgvector-standby" +DEFAULT_PRIVATE_NETWORK = "projects/teleo-501523/global/networks/teleo-staging-net" +DEFAULT_COMPUTE_INSTANCE = "teleo-prod-1" +DEFAULT_COMPUTE_ZONE = "europe-west6-a" +DEFAULT_RESTORE_SERVICE_ACCOUNT = "sa-teleo-restore-drill@teleo-501523.iam.gserviceaccount.com" +DEFAULT_SOURCE_SSH_TARGET = "root@77.42.65.182" +DEFAULT_SOURCE_CONTAINER = "teleo-pg" +DEFAULT_SOURCE_DATABASE = "teleo" +DEFAULT_SOURCE_SERVICE = "leoclean-gateway.service" DEFAULT_SECRET = "gcp-teleo-pgvector-standby-postgres-password" DEFAULT_SERVICE = "leoclean-gcp-prod-parallel.service" DEFAULT_RUN_ROOT = Path("/var/lib/teleo-gcp-restore-runs") DEFAULT_MANIFEST_SQL = Path(__file__).with_name("postgres_parity_manifest.sql") -REVIEWED_MANIFEST_SQL_SHA256 = "8b8cdc25d54fdd8de05eb38c6e4423d2836953eb6012d4545f5c9c71b5f0150a" +SOURCE_SNAPSHOT_RECEIPT_SCHEMA = "livingip.sourceSnapshotReceipt.v2" TARGET_DATABASE_RE = re.compile(r"teleo_clone_[a-z0-9][a-z0-9_]{0,50}\Z") RUN_ID_RE = re.compile(r"gcp-restore-[a-z0-9][a-z0-9-]{7,47}\Z") SHA256_RE = re.compile(r"[0-9a-f]{64}\Z") LOCALE_RE = re.compile(r"[A-Za-z0-9_.@-]{1,64}\Z") +CLOUDSQL_INSTANCE_RE = re.compile(r"[a-z][a-z0-9-]{0,97}[a-z0-9]\Z") +AUTHORIZATION_REF_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.:/@+-]{7,255}\Z") +PRIVATE_NETWORK_RE = re.compile(r"projects/[a-z][a-z0-9-]{4,28}[a-z0-9]/global/networks/[a-z][a-z0-9-]{0,61}[a-z0-9]\Z") +GCE_INSTANCE_RE = re.compile(r"[a-z](?:[a-z0-9-]{0,61}[a-z0-9])?\Z") +GCE_ZONE_RE = re.compile(r"[a-z]+-[a-z]+[0-9]-[a-z]\Z") +SAFE_NAME_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}\Z") +SAFE_SSH_TARGET_RE = re.compile(r"(?:[A-Za-z0-9][A-Za-z0-9._-]{0,63}@)?[A-Za-z0-9][A-Za-z0-9.-]{0,252}\Z") +SAFE_DNS_NAME_RE = re.compile(r"[A-Za-z0-9](?:[A-Za-z0-9.-]{0,251}[A-Za-z0-9])?\Z") +SERVICE_ACCOUNT_RE = re.compile(r"[a-z][a-z0-9-]{0,62}@[a-z][a-z0-9-]{4,28}[a-z0-9]\.iam\.gserviceaccount\.com\Z") PG_RESTORE_VERSION_RE = re.compile(r"pg_restore \(PostgreSQL\) (?P[0-9]+)(?:\.[0-9]+)*") ROLLBACK_DATABASE = "teleo_canonical_pre_20260712t1905z" +GCE_METADATA_ROOT = "http://metadata.google.internal/computeMetadata/v1" class RestoreError(RuntimeError): @@ -66,6 +90,11 @@ def sha256_file(path: Path) -> str: return digest.hexdigest() +def canonical_sha256(value: Any) -> str: + payload = json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True) + return hashlib.sha256(payload.encode()).hexdigest() + + def validate_regular_file(path: Path, label: str) -> Path: try: mode = path.lstat().st_mode @@ -89,6 +118,104 @@ def validate_custom_dump(path: Path, expected_sha256: str) -> dict[str, Any]: return {"path": str(path), "bytes": path.stat().st_size, "sha256": actual_sha256} +def validate_source_snapshot_receipt( + path: Path, + *, + expected_receipt_sha256: str, + expected_authorization_ref: str, + expected_source: dict[str, str], + dump: dict[str, Any], + source_manifest_path: Path, + manifest_sql_path: Path, + source_manifest: dict[str, Any], +) -> dict[str, Any]: + path = validate_regular_file(path, "source snapshot receipt") + receipt_sha256 = sha256_file(path) + if receipt_sha256 != expected_receipt_sha256: + raise RestoreError("source snapshot receipt SHA-256 does not match --expected-source-receipt-sha256") + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError as exc: + raise RestoreError("source snapshot receipt is invalid JSON") from exc + if not isinstance(payload, dict): + raise RestoreError("source snapshot receipt must be a JSON object") + + snapshot = payload.get("snapshot") or {} + manifest = payload.get("manifest") or {} + source = payload.get("source") or {} + source_context = payload.get("source_context") or {} + source_service = payload.get("source_service") or {} + binding = payload.get("provenance_binding") or {} + binding_payload = binding.get("payload") or {} + source_manifest_sha256 = sha256_file(source_manifest_path) + manifest_sql_sha256 = sha256_file(manifest_sql_path) + expected_table_count = len(source_manifest["tables"]) + expected_total_rows = sum(int(row["row_count"]) for row in source_manifest["tables"].values()) + expected_binding_payload = { + "receipt_schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA, + "run_id": payload.get("run_id"), + "capture_authorization_ref": payload.get("capture_authorization_ref"), + "source": source, + "exported_snapshot_id": snapshot.get("exported_snapshot_id"), + "txid_snapshot": snapshot.get("txid_snapshot"), + "wal_lsn": snapshot.get("wal_lsn"), + "source_system_identifier": snapshot.get("system_identifier"), + "dump_sha256": dump["sha256"], + "manifest_sql_sha256": manifest_sql_sha256, + "source_manifest_sha256": source_manifest_sha256, + "source_context_sha256": source_context.get("sha256"), + } + checks = { + "artifact": payload.get("artifact") == "vps_canonical_postgres_exported_snapshot", + "schema": payload.get("schema") == SOURCE_SNAPSHOT_RECEIPT_SCHEMA, + "receipt_version": payload.get("receipt_version") == 2, + "status": payload.get("status") == "pass", + "snapshot_exported": payload.get("snapshot_exported") is True, + "service_unchanged": payload.get("service_unchanged") is True, + "source_service_healthy": source_service.get("unchanged") is True + and service_state_is_healthy(source_service.get("before")) + and source_service.get("before") == source_service.get("after"), + "source_not_mutated": payload.get("production_db_mutated") is False, + "telegram_not_sent": payload.get("telegram_send_attempted") is False, + "run_id": bool(payload.get("run_id")), + "capture_authorization_ref": payload.get("capture_authorization_ref") == expected_authorization_ref, + "source_identity": source == expected_source, + "source_database": source.get("database") == source_manifest["singleton"]["identity"].get("database"), + "snapshot_identity": all( + bool(snapshot.get(field)) + for field in ("exported_snapshot_id", "txid_snapshot", "wal_lsn", "system_identifier", "captured_at_utc") + ), + "dump_sha256": (payload.get("dump") or {}).get("sha256") == dump["sha256"], + "dump_bytes": (payload.get("dump") or {}).get("bytes") == dump["bytes"], + "source_manifest_sha256": manifest.get("sha256") == source_manifest_sha256, + "manifest_sql_sha256": manifest.get("manifest_sql_sha256") == manifest_sql_sha256, + "source_context_sha256": bool(SHA256_RE.fullmatch(str(source_context.get("sha256") or ""))), + "table_count": manifest.get("table_count") == expected_table_count, + "total_rows": manifest.get("total_rows") == expected_total_rows, + "binding_algorithm": binding.get("algorithm") == "sha256", + "binding_payload": binding_payload == expected_binding_payload, + "binding_sha256": binding.get("sha256") == canonical_sha256(expected_binding_payload), + } + failed = sorted(name for name, passed in checks.items() if not passed) + if failed: + raise RestoreError("source snapshot receipt failed validation: " + ", ".join(failed)) + return { + "path": str(path), + "sha256": receipt_sha256, + "schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA, + "run_id": payload["run_id"], + "capture_authorization_ref": payload["capture_authorization_ref"], + "snapshot": snapshot, + "provenance_binding_sha256": binding["sha256"], + "manifest_sha256": source_manifest_sha256, + "manifest_sql_sha256": manifest_sql_sha256, + "dump_sha256": dump["sha256"], + "table_count": expected_table_count, + "total_rows": expected_total_rows, + "checks": checks, + } + + def _run( command: list[str], *, @@ -158,6 +285,165 @@ def service_state(service: str, *, timeout: float) -> dict[str, Any]: return result +def validate_service_healthy(state: dict[str, Any], label: str) -> None: + if not service_state_is_healthy(state): + raise RestoreError(f"{label} service is not active/running with a positive MainPID") + + +def service_state_is_healthy(state: Any) -> bool: + if not isinstance(state, dict): + return False + try: + main_pid = int(state.get("MainPID") or 0) + restarts = int(state.get("NRestarts") or 0) + except (TypeError, ValueError): + return False + return ( + state.get("ActiveState") == "active" and state.get("SubState") == "running" and main_pid > 0 and restarts >= 0 + ) + + +def cloudsql_control_plane_state( + project: str, + instance: str, + expected_host: str, + expected_private_network: str, + *, + timeout: float, +) -> dict[str, Any]: + completed = _run( + [ + "gcloud", + "sql", + "instances", + "describe", + instance, + f"--project={project}", + "--format=json", + "--quiet", + ], + timeout=timeout, + ) + raw = _require_success(completed, "Cloud SQL control-plane readback") + try: + payload = json.loads(raw) + except json.JSONDecodeError as exc: + raise RestoreError("Cloud SQL control-plane readback returned invalid JSON") from exc + settings = payload.get("settings") or {} + ip_configuration = settings.get("ipConfiguration") or {} + addresses = payload.get("ipAddresses") or [] + private_addresses = sorted( + str(row.get("ipAddress")) for row in addresses if row.get("type") == "PRIVATE" and row.get("ipAddress") + ) + private_network = str(ip_configuration.get("privateNetwork") or "") + non_private_addresses = sorted( + str(row.get("ipAddress")) for row in addresses if row.get("type") != "PRIVATE" and row.get("ipAddress") + ) + checks = { + "instance": payload.get("name") == instance, + "state": payload.get("state") == "RUNNABLE", + "postgres_16": str(payload.get("databaseVersion") or "").startswith("POSTGRES_16"), + "public_ip_disabled": ip_configuration.get("ipv4Enabled") is False, + "private_network": private_network == expected_private_network, + "expected_private_host": expected_host in private_addresses, + "no_non_private_address": not non_private_addresses, + } + failed = sorted(name for name, passed in checks.items() if not passed) + if failed: + raise RestoreError("Cloud SQL control-plane preflight failed: " + ", ".join(failed)) + return { + "project": project, + "instance": instance, + "database_version": payload.get("databaseVersion"), + "region": payload.get("region"), + "state": payload.get("state"), + "tier": settings.get("tier"), + "private_network": private_network, + "expected_private_network": expected_private_network, + "private_addresses": private_addresses, + "expected_private_host": expected_host, + "public_ip_disabled": True, + "checks": checks, + } + + +def gce_metadata_value(path: str, *, timeout: float) -> str: + request = urllib.request.Request( + f"{GCE_METADATA_ROOT}/{path}", + headers={"Metadata-Flavor": "Google"}, + method="GET", + ) + opener = urllib.request.build_opener(urllib.request.ProxyHandler({})) + try: + with opener.open(request, timeout=timeout) as response: + if response.headers.get("Metadata-Flavor") != "Google": + raise RestoreError("GCE metadata response is missing Metadata-Flavor: Google") + value = response.read(4096).decode("utf-8", errors="strict").strip() + except (OSError, UnicodeError, urllib.error.URLError) as exc: + raise RestoreError(f"GCE metadata readback failed for {path}") from exc + if not value or any(character in value for character in "\r\n\x00"): + raise RestoreError(f"GCE metadata readback is invalid for {path}") + return value + + +def gce_compute_identity( + project: str, + expected_instance: str, + expected_zone: str, + expected_private_network: str, + expected_service_account: str, + *, + timeout: float, +) -> dict[str, Any]: + metadata = { + "project": gce_metadata_value("project/project-id", timeout=timeout), + "project_number": gce_metadata_value("project/numeric-project-id", timeout=timeout), + "instance": gce_metadata_value("instance/name", timeout=timeout), + "instance_id": gce_metadata_value("instance/id", timeout=timeout), + "zone_resource": gce_metadata_value("instance/zone", timeout=timeout), + "network_resource": gce_metadata_value("instance/network-interfaces/0/network", timeout=timeout), + "private_ip": gce_metadata_value("instance/network-interfaces/0/ip", timeout=timeout), + "service_account": gce_metadata_value("instance/service-accounts/default/email", timeout=timeout), + } + zone = metadata["zone_resource"].rsplit("/", 1)[-1] + network_name = expected_private_network.rsplit("/", 1)[-1] + expected_network_resource = f"projects/{metadata['project_number']}/networks/{network_name}" + try: + private_ip = ipaddress.ip_address(metadata["private_ip"]) + except ValueError as exc: + raise RestoreError("GCE metadata private IP is invalid") from exc + checks = { + "project": metadata["project"] == project, + "project_number": metadata["project_number"].isdigit(), + "expected_network_project": expected_private_network.startswith(f"projects/{project}/global/networks/"), + "instance": metadata["instance"] == expected_instance, + "instance_id": metadata["instance_id"].isdigit(), + "zone": zone == expected_zone, + "network": metadata["network_resource"] == expected_network_resource, + "private_ip": any( + private_ip in network + for network in ( + ipaddress.ip_network("10.0.0.0/8"), + ipaddress.ip_network("172.16.0.0/12"), + ipaddress.ip_network("192.168.0.0/16"), + ) + ), + "service_account": metadata["service_account"] == expected_service_account, + } + failed = sorted(name for name, passed in checks.items() if not passed) + if failed: + raise RestoreError("GCE compute identity preflight failed: " + ", ".join(failed)) + return { + **metadata, + "zone": zone, + "expected_zone": expected_zone, + "expected_private_network": expected_private_network, + "expected_network_resource": expected_network_resource, + "expected_service_account": expected_service_account, + "checks": checks, + } + + def resolve_password(project: str, secret: str, *, timeout: float) -> str: completed = _run( [ @@ -212,10 +498,29 @@ def postgres_env(password: str, *, read_only: bool) -> dict[str, str]: return env +def conninfo_value(value: Any) -> str: + return "'" + str(value).replace("\\", "\\\\").replace("'", "\\'") + "'" + + +def postgres_conninfo(args: argparse.Namespace, database: str) -> str: + return " ".join( + ( + f"host={conninfo_value(args.ssl_server_name)}", + f"hostaddr={conninfo_value(args.host)}", + "port=5432", + f"dbname={conninfo_value(database)}", + "user=postgres", + "sslmode=verify-full", + f"sslrootcert={conninfo_value(args.ssl_root_cert)}", + "connect_timeout=8", + ) + ) + + def psql_command(args: argparse.Namespace, database: str) -> list[str]: return [ args.psql_bin, - f"host={args.host} port=5432 dbname={database} user=postgres sslmode=require connect_timeout=8", + postgres_conninfo(args, database), "-X", "-Atq", "-v", @@ -286,7 +591,7 @@ def restore_dump(args: argparse.Namespace, password: str) -> None: command = [ args.pg_restore_bin, "-d", - f"host={args.host} port=5432 dbname={args.target_db} user=postgres sslmode=require connect_timeout=8", + postgres_conninfo(args, args.target_db), "--no-owner", "--no-acl", "--exit-on-error", @@ -310,7 +615,13 @@ def capture_target_manifest(args: argparse.Namespace, password: str) -> str: return _require_success(completed, "target parity manifest capture", secrets=(password,)) -def validate_private_identity(target_manifest: dict[str, Any], target_db: str) -> dict[str, Any]: +def validate_private_identity( + target_manifest: dict[str, Any], + target_db: str, + source_compute: str, + ssl_server_name: str, + ssl_root_cert_sha256: str, +) -> dict[str, Any]: identity = target_manifest["singleton"]["identity"] if identity.get("database") != target_db: raise RestoreError("target manifest database does not match the generated clone") @@ -338,9 +649,13 @@ def validate_private_identity(target_manifest: dict[str, Any], target_db: str) - "server_port": identity.get("server_port"), "ssl": True, "ssl_version": identity.get("ssl_version"), + "sslmode": "verify-full", + "ssl_server_name": ssl_server_name, + "ssl_root_cert_sha256": ssl_root_cert_sha256, + "server_identity_verified": True, "private_connectivity": True, "transaction_read_only": "on", - "source_compute": "teleo-prod-1", + "source_compute": source_compute, } @@ -434,12 +749,33 @@ def write_private(path: Path, content: str | bytes) -> None: def run_restore(args: argparse.Namespace) -> dict[str, Any]: started = time.monotonic() + args.ssl_root_cert = validate_regular_file(args.ssl_root_cert, "Cloud SQL server CA") + ssl_root_cert_sha256 = sha256_file(args.ssl_root_cert) run_dir = secure_run_dir(args.run_root, args.run_id, create=True) receipt: dict[str, Any] = { "artifact": "gcp_generated_postgres_snapshot_restore", "generated_at_utc": utc_now(), "run_id": args.run_id, "target_database": args.target_db, + "execution_contract": { + "project": args.project, + "cloudsql_instance": args.cloudsql_instance, + "host": args.host, + "private_network": args.expected_private_network, + "compute_instance": args.expected_compute_instance, + "compute_zone": args.expected_compute_zone, + "restore_service_account": args.expected_restore_service_account, + "ssl_server_name": args.ssl_server_name, + "ssl_root_cert_sha256": ssl_root_cert_sha256, + "source": { + "ssh_target": args.expected_source_ssh_target, + "container": args.expected_source_container, + "database": args.expected_source_database, + "service": args.expected_source_service, + }, + "password_secret": args.password_secret, + "service": args.service, + }, "status": "running", "phase": "validation", "source": {}, @@ -475,12 +811,40 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]: "sha256": sha256_file(args.source_manifest), } source_manifest = load_manifest(args.source_manifest) + receipt["source"]["capture_receipt"] = validate_source_snapshot_receipt( + args.source_receipt, + expected_receipt_sha256=args.expected_source_receipt_sha256, + expected_authorization_ref=args.expected_capture_authorization_ref, + expected_source=receipt["execution_contract"]["source"], + dump=receipt["source"]["dump"], + source_manifest_path=args.source_manifest, + manifest_sql_path=args.manifest_sql, + source_manifest=source_manifest, + ) receipt["toolchain"] = validate_pg_restore_version( args.pg_restore_bin, source_manifest, timeout=args.command_timeout, ) + receipt["target"]["compute"] = gce_compute_identity( + args.project, + args.expected_compute_instance, + args.expected_compute_zone, + args.expected_private_network, + args.expected_restore_service_account, + timeout=args.command_timeout, + ) receipt["live_service"]["before"] = service_state(args.service, timeout=args.command_timeout) + validate_service_healthy(receipt["live_service"]["before"], "pre-restore") + + receipt["phase"] = "cloudsql_control_plane_preflight" + receipt["target"]["cloudsql"] = cloudsql_control_plane_state( + args.project, + args.cloudsql_instance, + args.host, + args.expected_private_network, + timeout=args.command_timeout, + ) receipt["phase"] = "credential_resolution" password = resolve_password(args.project, args.password_secret, timeout=args.command_timeout) @@ -506,9 +870,52 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]: "manifest_path": str(target_manifest_path), "manifest_bytes": target_manifest_path.stat().st_size, "manifest_sha256": sha256_file(target_manifest_path), - "connectivity": validate_private_identity(target_manifest, args.target_db), + "connectivity": validate_private_identity( + target_manifest, + args.target_db, + receipt["target"]["compute"]["instance"], + args.ssl_server_name, + ssl_root_cert_sha256, + ), } ) + connectivity = receipt["target"]["connectivity"] + cloudsql = receipt["target"]["cloudsql"] + compute = receipt["target"]["compute"] + connectivity_proof = { + "artifact": "gcp_private_postgres_connectivity", + "schema": "livingip.gcpPrivatePostgresConnectivity.v2", + "generated_at_utc": utc_now(), + "status": "pass", + "restore_run_id": args.run_id, + "target_database": args.target_db, + "project": cloudsql["project"], + "cloudsql_instance": cloudsql["instance"], + "private_network": cloudsql["private_network"], + "source_compute": compute["instance"], + "source_compute_instance_id": compute["instance_id"], + "source_compute_zone": compute["zone"], + "source_compute_private_ip": compute["private_ip"], + "server_address": connectivity["server_address"], + "server_port": connectivity["server_port"], + "ssl": connectivity["ssl"], + "ssl_version": connectivity["ssl_version"], + "sslmode": connectivity["sslmode"], + "ssl_server_name": connectivity["ssl_server_name"], + "ssl_root_cert_sha256": connectivity["ssl_root_cert_sha256"], + "server_identity_verified": connectivity["server_identity_verified"], + "private_connectivity": connectivity["private_connectivity"], + "public_ip_disabled": cloudsql["public_ip_disabled"], + "cloudsql_control_plane_checks": cloudsql["checks"], + "compute_identity_checks": compute["checks"], + } + connectivity_proof_path = run_dir / "gcp-private-connectivity.json" + write_private(connectivity_proof_path, json.dumps(connectivity_proof, indent=2, sort_keys=True) + "\n") + receipt["target"]["connectivity_proof"] = { + "path": str(connectivity_proof_path), + "sha256": sha256_file(connectivity_proof_path), + "schema": connectivity_proof["schema"], + } receipt["phase"] = "parity_compare" problems, details = compare_manifests( @@ -529,6 +936,7 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]: receipt["phase"] = "service_and_rollback_readback" receipt["rollback_database"] = rollback_database_state(args, password) receipt["live_service"]["after"] = service_state(args.service, timeout=args.command_timeout) + validate_service_healthy(receipt["live_service"]["after"], "post-restore") receipt["live_service"]["unchanged"] = receipt["live_service"]["before"] == receipt["live_service"]["after"] if not receipt["live_service"]["unchanged"]: raise RestoreError("live GCP service state changed during disposable restore") @@ -539,9 +947,48 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]: "required": True, "attempted": False, "clone_database_remaining": 1, - "exact_command": ( - f"sudo python3 ops/restore_gcp_generated_postgres_snapshot.py cleanup --execute " - f"--target-db {args.target_db} --run-id {args.run_id}" + "exact_command": shlex.join( + [ + "sudo", + "python3", + "ops/restore_gcp_generated_postgres_snapshot.py", + "cleanup", + "--execute", + "--target-db", + args.target_db, + "--run-id", + args.run_id, + "--host", + args.host, + "--project", + args.project, + "--cloudsql-instance", + args.cloudsql_instance, + "--expected-private-network", + args.expected_private_network, + "--expected-compute-instance", + args.expected_compute_instance, + "--expected-compute-zone", + args.expected_compute_zone, + "--expected-restore-service-account", + args.expected_restore_service_account, + "--ssl-server-name", + args.ssl_server_name, + "--ssl-root-cert", + str(args.ssl_root_cert), + "--expected-source-ssh-target", + args.expected_source_ssh_target, + "--expected-source-container", + args.expected_source_container, + "--expected-source-database", + args.expected_source_database, + "--expected-source-service", + args.expected_source_service, + "--password-secret", + args.password_secret, + "--service", + args.service, + ] ), } except ( @@ -583,15 +1030,19 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]: except Exception as service_exc: receipt["live_service"]["after_error"] = str(service_exc)[-2000:] receipt["duration_seconds"] = round(time.monotonic() - started, 6) + receipt["completed_at_utc"] = utc_now() write_private(run_dir / "restore-receipt.json", json.dumps(receipt, indent=2, sort_keys=True) + "\n") password = "" return receipt def run_cleanup(args: argparse.Namespace) -> dict[str, Any]: + args.ssl_root_cert = validate_regular_file(args.ssl_root_cert, "Cloud SQL server CA") + ssl_root_cert_sha256 = sha256_file(args.ssl_root_cert) run_dir = secure_run_dir(args.run_root, args.run_id, create=False) restore_receipt_path = run_dir / "restore-receipt.json" restore_receipt_path = validate_regular_file(restore_receipt_path, "restore receipt") + restore_receipt_sha256 = sha256_file(restore_receipt_path) restore_receipt = json.loads(restore_receipt_path.read_text(encoding="utf-8")) if restore_receipt.get("status") != "pass": raise RestoreError("cleanup requires a passing restore receipt") @@ -600,23 +1051,70 @@ def run_cleanup(args: argparse.Namespace) -> dict[str, Any]: if restore_receipt.get("run_id") != args.run_id: raise RestoreError("cleanup run id does not match the retained restore receipt") - before = service_state(args.service, timeout=args.command_timeout) - password = resolve_password(args.project, args.password_secret, timeout=args.command_timeout) - try: - dropped = drop_clone(args, password) - rollback = rollback_database_state(args, password) - finally: - password = "" - after = service_state(args.service, timeout=args.command_timeout) - payload = { + expected_contract = { + "project": args.project, + "cloudsql_instance": args.cloudsql_instance, + "host": args.host, + "private_network": args.expected_private_network, + "compute_instance": args.expected_compute_instance, + "compute_zone": args.expected_compute_zone, + "restore_service_account": args.expected_restore_service_account, + "ssl_server_name": args.ssl_server_name, + "ssl_root_cert_sha256": ssl_root_cert_sha256, + "source": { + "ssh_target": args.expected_source_ssh_target, + "container": args.expected_source_container, + "database": args.expected_source_database, + "service": args.expected_source_service, + }, + "password_secret": args.password_secret, + "service": args.service, + } + if restore_receipt.get("execution_contract") != expected_contract: + raise RestoreError("cleanup execution contract does not match the retained restore receipt") + restore_cloudsql = (restore_receipt.get("target") or {}).get("cloudsql") or {} + restore_topology_checks = { + "project": restore_cloudsql.get("project") == args.project, + "instance": restore_cloudsql.get("instance") == args.cloudsql_instance, + "host": restore_cloudsql.get("expected_private_host") == args.host, + "private_network": restore_cloudsql.get("expected_private_network") == args.expected_private_network, + "public_ip_disabled": restore_cloudsql.get("public_ip_disabled") is True, + } + failed_topology_checks = sorted(name for name, passed in restore_topology_checks.items() if not passed) + if failed_topology_checks: + raise RestoreError( + "cleanup target topology does not match the retained restore receipt: " + ", ".join(failed_topology_checks) + ) + restore_compute = (restore_receipt.get("target") or {}).get("compute") or {} + restore_compute_checks = { + "project": restore_compute.get("project") == args.project, + "instance": restore_compute.get("instance") == args.expected_compute_instance, + "zone": restore_compute.get("zone") == args.expected_compute_zone, + "private_network": restore_compute.get("expected_private_network") == args.expected_private_network, + "service_account": restore_compute.get("service_account") == args.expected_restore_service_account, + "checks": bool(restore_compute.get("checks")) + and all(value is True for value in restore_compute["checks"].values()), + } + failed_compute_checks = sorted(name for name, passed in restore_compute_checks.items() if not passed) + if failed_compute_checks: + raise RestoreError( + "cleanup compute identity does not match the retained restore receipt: " + ", ".join(failed_compute_checks) + ) + + started = time.monotonic() + payload: dict[str, Any] = { "artifact": "gcp_generated_postgres_snapshot_cleanup", "generated_at_utc": utc_now(), "run_id": args.run_id, "target_database": args.target_db, - "status": "pass" if before == after and dropped["clone_database_remaining"] == 0 else "fail", - "database": dropped, - "rollback_database": rollback, - "live_service": {"before": before, "after": after, "unchanged": before == after}, + "restore_receipt": {"path": str(restore_receipt_path), "sha256": restore_receipt_sha256}, + "execution_contract": expected_contract, + "target": {}, + "status": "running", + "phase": "cloudsql_control_plane_preflight", + "database": {"clone_database_remaining": None}, + "rollback_database": {}, + "live_service": {}, "safety": { "live_database_named": False, "live_profile_modified": False, @@ -624,8 +1122,96 @@ def run_cleanup(args: argparse.Namespace) -> dict[str, Any]: "telegram_message_sent": False, "secret_persisted": False, }, + "error": None, } - write_private(run_dir / "cleanup-receipt.json", json.dumps(payload, indent=2, sort_keys=True) + "\n") + password = "" + try: + payload["target"]["cloudsql"] = cloudsql_control_plane_state( + args.project, + args.cloudsql_instance, + args.host, + args.expected_private_network, + timeout=args.command_timeout, + ) + payload["target"]["compute"] = gce_compute_identity( + args.project, + args.expected_compute_instance, + args.expected_compute_zone, + args.expected_private_network, + args.expected_restore_service_account, + timeout=args.command_timeout, + ) + cleanup_compute_mismatches = sorted( + field + for field in ( + "project", + "project_number", + "instance", + "instance_id", + "zone", + "network_resource", + "private_ip", + "service_account", + ) + if payload["target"]["compute"].get(field) != restore_compute.get(field) + ) + if cleanup_compute_mismatches: + raise RestoreError("cleanup GCE identity changed since restore: " + ", ".join(cleanup_compute_mismatches)) + payload["phase"] = "service_readback_before" + payload["live_service"]["before"] = service_state(args.service, timeout=args.command_timeout) + validate_service_healthy(payload["live_service"]["before"], "pre-cleanup") + payload["phase"] = "credential_resolution" + password = resolve_password(args.project, args.password_secret, timeout=args.command_timeout) + payload["phase"] = "database_drop" + payload["database"] = drop_clone(args, password) + payload["phase"] = "rollback_readback" + payload["rollback_database"] = rollback_database_state(args, password) + payload["phase"] = "service_readback_after" + payload["live_service"]["after"] = service_state(args.service, timeout=args.command_timeout) + validate_service_healthy(payload["live_service"]["after"], "post-cleanup") + payload["live_service"]["unchanged"] = payload["live_service"]["before"] == payload["live_service"]["after"] + payload["status"] = ( + "pass" + if payload["live_service"]["unchanged"] and payload["database"]["clone_database_remaining"] == 0 + else "fail" + ) + payload["phase"] = "complete" + except ( + OSError, + ValueError, + KeyError, + TypeError, + json.JSONDecodeError, + subprocess.TimeoutExpired, + RestoreError, + KeyboardInterrupt, + ) as exc: + payload["status"] = "fail" + payload["error"] = { + "phase": payload.get("phase"), + "type": type(exc).__name__, + "message": str(exc)[-2000:], + } + if password and payload["database"].get("clone_database_remaining") is None: + try: + payload["database"]["clone_database_remaining"] = ( + 1 if database_exists(args, password, args.target_db) else 0 + ) + except Exception as readback_exc: + payload["database"]["readback_error"] = str(readback_exc)[-2000:] + if "before" in payload["live_service"] and "after" not in payload["live_service"]: + try: + payload["live_service"]["after"] = service_state(args.service, timeout=args.command_timeout) + payload["live_service"]["unchanged"] = ( + payload["live_service"]["before"] == payload["live_service"]["after"] + ) + except Exception as service_exc: + payload["live_service"]["after_error"] = str(service_exc)[-2000:] + finally: + password = "" + payload["duration_seconds"] = round(time.monotonic() - started, 6) + payload["completed_at_utc"] = utc_now() + write_private(run_dir / "cleanup-receipt.json", json.dumps(payload, indent=2, sort_keys=True) + "\n") return payload @@ -635,6 +1221,17 @@ def add_common_args(parser: argparse.ArgumentParser) -> None: parser.add_argument("--run-id", required=True) parser.add_argument("--host", default=DEFAULT_HOST) parser.add_argument("--project", default=DEFAULT_PROJECT) + parser.add_argument("--cloudsql-instance", default=DEFAULT_INSTANCE) + parser.add_argument("--expected-private-network", default=DEFAULT_PRIVATE_NETWORK) + parser.add_argument("--expected-compute-instance", default=DEFAULT_COMPUTE_INSTANCE) + parser.add_argument("--expected-compute-zone", default=DEFAULT_COMPUTE_ZONE) + parser.add_argument("--expected-restore-service-account", default=DEFAULT_RESTORE_SERVICE_ACCOUNT) + parser.add_argument("--ssl-server-name", required=True) + parser.add_argument("--ssl-root-cert", required=True, type=Path) + parser.add_argument("--expected-source-ssh-target", default=DEFAULT_SOURCE_SSH_TARGET) + parser.add_argument("--expected-source-container", default=DEFAULT_SOURCE_CONTAINER) + parser.add_argument("--expected-source-database", default=DEFAULT_SOURCE_DATABASE) + parser.add_argument("--expected-source-service", default=DEFAULT_SOURCE_SERVICE) parser.add_argument("--password-secret", default=DEFAULT_SECRET) parser.add_argument("--service", default=DEFAULT_SERVICE) parser.add_argument("--command-timeout", type=float, default=120.0) @@ -650,6 +1247,9 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace: restore.add_argument("--dump", required=True, type=Path) restore.add_argument("--expected-dump-sha256", required=True) restore.add_argument("--source-manifest", required=True, type=Path) + restore.add_argument("--source-receipt", required=True, type=Path) + restore.add_argument("--expected-source-receipt-sha256", required=True) + restore.add_argument("--expected-capture-authorization-ref", required=True) restore.add_argument("--manifest-sql", default=DEFAULT_MANIFEST_SQL, type=Path) restore.add_argument("--pg-restore-bin", default="pg_restore") restore.add_argument("--restore-timeout", type=float, default=900.0) @@ -667,6 +1267,36 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace: parser.error("--target-db must be a bounded lowercase teleo_clone_* identifier") if not RUN_ID_RE.fullmatch(args.run_id): parser.error("--run-id must match gcp-restore-<8-48 lowercase letters, digits, or hyphens>") + if not CLOUDSQL_INSTANCE_RE.fullmatch(args.cloudsql_instance): + parser.error("--cloudsql-instance must be a safe lowercase Cloud SQL instance name") + if not PRIVATE_NETWORK_RE.fullmatch(args.expected_private_network): + parser.error("--expected-private-network must be a full projects/.../global/networks/... resource") + if not args.expected_private_network.startswith(f"projects/{args.project}/global/networks/"): + parser.error("--expected-private-network project must match --project") + if not GCE_INSTANCE_RE.fullmatch(args.expected_compute_instance): + parser.error("--expected-compute-instance must be a safe lowercase GCE instance name") + if not GCE_ZONE_RE.fullmatch(args.expected_compute_zone): + parser.error("--expected-compute-zone must be a safe GCE zone") + if not SERVICE_ACCOUNT_RE.fullmatch(args.expected_restore_service_account): + parser.error("--expected-restore-service-account must be an exact GCP service-account email") + if not SAFE_DNS_NAME_RE.fullmatch(args.ssl_server_name): + parser.error("--ssl-server-name must be a safe DNS name covered by the Cloud SQL server certificate") + try: + ssl_root_cert_mode = args.ssl_root_cert.lstat().st_mode + except OSError: + parser.error("--ssl-root-cert must be an existing regular file") + if not stat.S_ISREG(ssl_root_cert_mode) or args.ssl_root_cert.is_symlink(): + parser.error("--ssl-root-cert must be a regular non-symlink file") + args.ssl_root_cert = args.ssl_root_cert.resolve() + if not SAFE_SSH_TARGET_RE.fullmatch(args.expected_source_ssh_target): + parser.error("--expected-source-ssh-target must be a host or user@host without options") + for value, flag in ( + (args.expected_source_container, "--expected-source-container"), + (args.expected_source_database, "--expected-source-database"), + (args.expected_source_service, "--expected-source-service"), + ): + if not SAFE_NAME_RE.fullmatch(value): + parser.error(f"{flag} contains unsafe characters") try: host = ipaddress.ip_address(args.host) except ValueError: @@ -683,6 +1313,10 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace: if args.operation == "restore": if not SHA256_RE.fullmatch(args.expected_dump_sha256): parser.error("--expected-dump-sha256 must be 64 lowercase hexadecimal characters") + if not SHA256_RE.fullmatch(args.expected_source_receipt_sha256): + parser.error("--expected-source-receipt-sha256 must be 64 lowercase hexadecimal characters") + if not AUTHORIZATION_REF_RE.fullmatch(args.expected_capture_authorization_ref): + parser.error("--expected-capture-authorization-ref must be 8-256 safe metadata characters") for value, flag in ( (args.restore_timeout, "--restore-timeout"), (args.manifest_timeout, "--manifest-timeout"), @@ -709,7 +1343,9 @@ def main(argv: list[str] | None = None) -> int: finally: for watched_signal, previous in previous_handlers.items(): signal.signal(watched_signal, previous) - print(json.dumps(payload, indent=2, sort_keys=True)) + receipt_name = "restore-receipt.json" if args.operation == "restore" else "cleanup-receipt.json" + receipt_path = args.run_root.resolve() / args.run_id / receipt_name + print_private_receipt_summary(receipt_path, payload) return 0 if payload.get("status") == "pass" else 1 diff --git a/ops/run_local_canonical_postgres_rebuild.py b/ops/run_local_canonical_postgres_rebuild.py index a5d3d10..3b6363d 100755 --- a/ops/run_local_canonical_postgres_rebuild.py +++ b/ops/run_local_canonical_postgres_rebuild.py @@ -31,7 +31,7 @@ MANIFEST_DESTINATION = "/tmp/postgres-parity-manifest.sql" CANARY_LABEL = "com.livingip.teleo.canonical-rebuild-canary" DATABASE_NAME_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]{0,62}\Z") CONTAINER_PREFIX_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,39}\Z") -ALLOWED_APPLICATION_ROLES = frozenset({"kb_apply", "kb_review"}) +ALLOWED_APPLICATION_ROLES = frozenset({"kb_gate_owner", "kb_apply", "kb_review"}) ROLE_BOOLEAN_FIELDS = ( "can_login", "superuser", diff --git a/ops/verify_gcp_canonical_lifecycle.py b/ops/verify_gcp_canonical_lifecycle.py new file mode 100644 index 0000000..5d182b5 --- /dev/null +++ b/ops/verify_gcp_canonical_lifecycle.py @@ -0,0 +1,792 @@ +#!/usr/bin/env python3 +"""Verify one bound source-to-GCP restore, reasoning, and cleanup lifecycle.""" + +from __future__ import annotations + +import argparse +import hashlib +import ipaddress +import json +import re +from datetime import UTC, datetime +from itertools import pairwise +from pathlib import Path +from typing import Any + +try: + from .private_receipt_io import print_private_receipt_summary + from .restore_gcp_generated_postgres_snapshot import DEFAULT_RESTORE_SERVICE_ACCOUNT + from .verify_postgres_parity_manifest import REVIEWED_MANIFEST_SQL_SHA256 + from .verify_vps_canonical_snapshot_delta import DELTA_SCHEMA, parse_timestamp, write_private_json +except ImportError: # pragma: no cover - direct script execution + from private_receipt_io import print_private_receipt_summary + from restore_gcp_generated_postgres_snapshot import DEFAULT_RESTORE_SERVICE_ACCOUNT + from verify_postgres_parity_manifest import REVIEWED_MANIFEST_SQL_SHA256 + from verify_vps_canonical_snapshot_delta import DELTA_SCHEMA, parse_timestamp, write_private_json + +SOURCE_SCHEMA = "livingip.sourceSnapshotReceipt.v2" +BLIND_REASONING_SCHEMA = "livingip.gcpGeneratedDbBlindClaimCanary.v1" +REASONING_COMPUTE_SCHEMA = "livingip.gcpReasoningComputeAttestation.v1" +RETRIEVAL_RECEIPT_SCHEMA = "livingip.teleoKbRetrievalReceipt.v1" +UUID_RE = re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}\b", re.I) +SHA256_RE = re.compile(r"[0-9a-f]{64}\Z") +FORBIDDEN_HANDLE_RE = re.compile(r"\b(?:cory|m3ta)\b", re.I) +EXPECTED_BLIND_ROW_IDS = { + "2a7ae257-d01d-46f4-b813-63f81bb9c7c7", + "261c3532-fa32-47d8-a5b5-6cc45035c267", + "15740795-ecc6-40fa-9a01-3d6bc7c54f79", +} + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def load_receipt(path: Path, label: str) -> dict[str, Any]: + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + raise ValueError(f"{label} is unreadable or invalid JSON") from exc + if not isinstance(payload, dict): + raise ValueError(f"{label} must be a JSON object") + return payload + + +def is_rfc1918(value: Any) -> bool: + try: + address = ipaddress.ip_address(str(value)) + except ValueError: + return False + return any( + address in network + for network in ( + ipaddress.ip_network("10.0.0.0/8"), + ipaddress.ip_network("172.16.0.0/12"), + ipaddress.ip_network("192.168.0.0/16"), + ) + ) + + +def nonempty_all_true(value: Any) -> bool: + return isinstance(value, dict) and bool(value) and all(item is True for item in value.values()) + + +def required_true_checks(value: Any, required: set[str]) -> bool: + return nonempty_all_true(value) and required <= set(value) + + +def service_healthy(value: Any) -> bool: + if not isinstance(value, dict): + return False + try: + main_pid = int(value.get("MainPID") or 0) + restarts = int(value.get("NRestarts") or 0) + except (TypeError, ValueError): + return False + return ( + value.get("ActiveState") == "active" and value.get("SubState") == "running" and main_pid > 0 and restarts >= 0 + ) + + +def parity_evidence(details: dict[str, Any]) -> dict[str, bool]: + structural = details.get("structural_hashes") or {} + required_structures = { + "schemas", + "columns", + "constraints", + "indexes", + "sequences", + "views", + "functions", + "triggers", + "types", + "policies", + "role_memberships", + "object_ownership", + "object_acl", + } + structural_valid = required_structures <= set(structural) and all( + bool(SHA256_RE.fullmatch(str((structural.get(kind) or {}).get("source") or ""))) + and (structural.get(kind) or {}).get("source") == (structural.get(kind) or {}).get("target") + for kind in required_structures + ) + performance = details.get("performance") or [] + performance_valid = bool(performance) and all( + bool(row.get("query")) + and isinstance(row.get("source_ms"), (int, float)) + and isinstance(row.get("target_ms"), (int, float)) + and isinstance(row.get("source_ratio"), (int, float)) + for row in performance + ) + source_table_count = details.get("source_table_count") + target_table_count = details.get("target_table_count") + source_total_rows = details.get("source_total_rows") + target_total_rows = details.get("target_total_rows") + return { + "table_counts": isinstance(source_table_count, int) + and source_table_count > 0 + and source_table_count == target_table_count, + "row_counts": isinstance(source_total_rows, int) + and source_total_rows > 0 + and source_total_rows == target_total_rows, + "table_hashes": details.get("table_mismatches") == [] and details.get("table_hash_problems") == [], + "structural_hashes": structural_valid, + "roles": details.get("application_role_mismatches") == {} + and details.get("target_extra_application_roles") == [], + "extensions": details.get("required_extension_mismatches") == {} + and details.get("target_extra_extensions") == [], + "performance": performance_valid and details.get("performance_problems") == [], + } + + +def blind_reasoning_evidence(reasoning: dict[str, Any], target_db: Any) -> dict[str, bool]: + result = reasoning.get("result") or {} + gateway = result.get("gateway") or {} + child = gateway.get("child_process") or {} + tool_surface = gateway.get("tool_surface") or {} + tool_trace = result.get("database_tool_trace") or {} + calls = tool_trace.get("calls") or [] + wrapper = reasoning.get("wrapper_tool_proof") or {} + invocations = wrapper.get("invocations") or [] + prompt = str(reasoning.get("prompt") or "") + reply = str(result.get("reply") or "") + outcomes = reasoning.get("outcomes") or {} + required_outcomes = ( + "asks_for_user_iteration_before_live", + "challenges_weak_support", + "decomposes_the_bundled_legal_claim", + "does_not_claim_a_database_change", + "proposes_candidate_claims", + "uses_only_m3taversal_handle", + ) + required_subcommands = {"show", "evidence"} + discovery_subcommands = {"search", "context"} + traced_subcommands = { + str(invocation.get("subcommand")) for call in calls for invocation in (call.get("database_invocations") or []) + } + wrapper_subcommands = {str(invocation.get("subcommand")) for invocation in invocations} + retrieved_row_ids = {str(row_id) for call in calls for row_id in ((call.get("result") or {}).get("row_ids") or [])} + call_row_ids_valid = bool(calls) and all( + isinstance((call.get("result") or {}).get("row_ids"), list) + and isinstance((call.get("result") or {}).get("row_id_count"), int) + and (call.get("result") or {}).get("row_id_count") >= 0 + and (call.get("result") or {}).get("row_id_count") >= len((call.get("result") or {}).get("row_ids") or []) + for call in calls + ) + call_receipts_valid = bool(calls) and all( + (call.get("result") or {}).get("nonempty") is True + and (call.get("result") or {}).get("error_detected") is False + and ((call.get("result") or {}).get("retrieval_receipt") or {}).get("schema") == RETRIEVAL_RECEIPT_SCHEMA + and bool( + SHA256_RE.fullmatch( + str(((call.get("result") or {}).get("retrieval_receipt") or {}).get("semantic_context_sha256") or "") + ) + ) + and bool( + SHA256_RE.fullmatch( + str(((call.get("result") or {}).get("retrieval_receipt") or {}).get("artifact_state_sha256") or "") + ) + ) + and str( + ((call.get("result") or {}).get("retrieval_receipt") or {}).get("read_consistency_status") or "" + ).startswith("stable") + for call in calls + ) + wrapper_invocations_valid = bool(invocations) and all( + invocation.get("database") == target_db + and invocation.get("returncode") == 0 + and ((invocation.get("database_identity") or {}).get("transaction_read_only") == "on") + and ((invocation.get("database_identity") or {}).get("default_transaction_read_only") == "on") + and ((invocation.get("database_identity") or {}).get("ssl") is True) + and ((invocation.get("database_identity") or {}).get("sslmode") == "verify-full") + and ((invocation.get("database_identity") or {}).get("server_identity_verified") is True) + and bool((invocation.get("database_identity") or {}).get("ssl_server_name")) + and bool( + SHA256_RE.fullmatch(str((invocation.get("database_identity") or {}).get("ssl_root_cert_sha256") or "")) + ) + and is_rfc1918((invocation.get("database_identity") or {}).get("server_address")) + for invocation in invocations + ) + return { + "mode": reasoning.get("mode") == "gcp_generated_db_gatewayrunner_blind_claim_no_send", + "timestamps": bool(reasoning.get("generated_at_utc")) and bool(reasoning.get("completed_at_utc")), + "source_compute": bool(reasoning.get("source_compute")), + "blind_prompt": bool(prompt) and UUID_RE.search(prompt) is None, + "reply_nonempty": bool(reply.strip()), + "reply_uses_only_m3taversal_handle": FORBIDDEN_HANDLE_RE.search(reply) is None, + "outcomes": all(outcomes.get(name) is True for name in required_outcomes), + "gateway": gateway.get("authorized") is True + and gateway.get("handler_invoked") is True + and gateway.get("model_free_fallback_used") is False + and gateway.get("posted_to_telegram") is False, + "gateway_cleanup": child.get("exitcode") == 0 + and child.get("alive_after_readback") is False + and child.get("process_group_alive_after_readback") is False + and child.get("timed_out") is False, + "send_tool_absent": tool_surface.get("send_message_tool_enabled") is False, + "tool_trace": tool_trace.get("schema") == "livingip.leoKbToolTrace.v1" + and tool_trace.get("database_tool_call_proven") is True + and tool_trace.get("database_retrieval_receipt_proven") is True + and tool_trace.get("database_tool_calls_read_only") is True + and tool_trace.get("database_tool_call_count") == len(calls) + and tool_trace.get("database_tool_completed_count") == len(calls), + "tool_receipts": call_receipts_valid, + "expected_claim_and_sources_retrieved": call_row_ids_valid and retrieved_row_ids >= EXPECTED_BLIND_ROW_IDS, + "required_tool_sequence": required_subcommands <= traced_subcommands + and bool(discovery_subcommands & traced_subcommands), + "wrapper": wrapper.get("all_bound_to_supplied_target") is True + and wrapper.get("all_completed_successfully") is True + and wrapper.get("complete_start_end_pairing") is True + and wrapper.get("database_read_only_required") is True + and wrapper.get("default_read_only_required") is True, + "wrapper_invocations": wrapper_invocations_valid + and required_subcommands <= wrapper_subcommands + and bool(discovery_subcommands & wrapper_subcommands), + "canonical_status_unchanged": bool(reasoning.get("canonical_status_before")) + and reasoning.get("canonical_status_before") == reasoning.get("canonical_status_after"), + "database_identity_unchanged": bool(reasoning.get("database_identity_before")) + and reasoning.get("database_identity_before") == reasoning.get("database_identity_after"), + "service_unchanged": service_healthy(reasoning.get("service_before")) + and reasoning.get("service_before") == reasoning.get("service_after"), + "temporary_profile_removed": reasoning.get("temp_profile_absent") is True + and reasoning.get("temp_profile_removed") is True, + } + + +def verify_lifecycle( + *, + source_path: Path, + current_source_path: Path, + source_delta_path: Path, + restore_path: Path, + parity_path: Path, + reasoning_path: Path, + reasoning_compute_path: Path, + cleanup_path: Path, + max_postflight_age_seconds: float = 900.0, + max_lifecycle_age_seconds: float = 3600.0, + now: datetime | None = None, +) -> dict[str, Any]: + if max_postflight_age_seconds <= 0: + raise ValueError("max postflight age must be positive") + if max_lifecycle_age_seconds <= 0: + raise ValueError("max lifecycle age must be positive") + verification_time = (now or datetime.now(UTC)).astimezone(UTC) + source = load_receipt(source_path, "source receipt") + current_source = load_receipt(current_source_path, "current source receipt") + source_delta = load_receipt(source_delta_path, "source delta receipt") + restore = load_receipt(restore_path, "restore receipt") + parity = load_receipt(parity_path, "parity receipt") + reasoning = load_receipt(reasoning_path, "reasoning receipt") + reasoning_compute_attestation = load_receipt(reasoning_compute_path, "reasoning compute attestation") + cleanup = load_receipt(cleanup_path, "cleanup receipt") + + target_db = restore.get("target_database") + run_id = restore.get("run_id") + restore_source = restore.get("source") or {} + capture_binding = restore_source.get("capture_receipt") or {} + restore_target = restore.get("target") or {} + restore_connectivity = restore_target.get("connectivity") or {} + restore_connectivity_proof = restore_target.get("connectivity_proof") or {} + restore_cloudsql = restore_target.get("cloudsql") or {} + restore_compute = restore_target.get("compute") or {} + restore_parity = restore.get("parity") or {} + restore_service = restore.get("live_service") or {} + parity_details = parity.get("details") or {} + parity_evidence_checks = parity_evidence(parity_details) + restore_inline_parity_checks = parity_evidence(restore_parity.get("details") or {}) + parity_connectivity = parity.get("private_connectivity") or {} + parity_connectivity_proof = parity_connectivity.get("proof") or {} + reasoning_checks = reasoning.get("checks") or {} + reasoning_parity = reasoning.get("parity_receipt") or {} + before_fingerprint = reasoning.get("database_fingerprint_before") or {} + after_fingerprint = reasoning.get("database_fingerprint_after") or {} + cleanup_database = cleanup.get("database") or {} + cleanup_service = cleanup.get("live_service") or {} + cleanup_cloudsql = (cleanup.get("target") or {}).get("cloudsql") or {} + cleanup_compute = (cleanup.get("target") or {}).get("compute") or {} + restore_contract = restore.get("execution_contract") or {} + cleanup_contract = cleanup.get("execution_contract") or {} + producer_reasoning_checks = blind_reasoning_evidence(reasoning, target_db) + reasoning_compute = reasoning_compute_attestation.get("compute") or {} + reasoning_compute_checks = reasoning_compute_attestation.get("checks") or {} + reasoning_compute_mutation = reasoning_compute_attestation.get("mutation") or {} + delta_baseline = source_delta.get("baseline") or {} + delta_current = source_delta.get("current") or {} + delta_details = source_delta.get("delta") or {} + source_snapshot = source.get("snapshot") or {} + current_source_snapshot = current_source.get("snapshot") or {} + timeline_values = { + "source_snapshot": source_snapshot.get("captured_at_utc"), + "restore_started": restore.get("generated_at_utc"), + "restore_completed": restore.get("completed_at_utc"), + "parity_completed": parity.get("completed_at_utc"), + "reasoning_started": reasoning.get("generated_at_utc"), + "reasoning_completed": reasoning.get("completed_at_utc"), + "reasoning_compute_attested": reasoning_compute_attestation.get("generated_at_utc"), + "cleanup_started": cleanup.get("generated_at_utc"), + "cleanup_completed": cleanup.get("completed_at_utc"), + "current_source_snapshot": current_source_snapshot.get("captured_at_utc"), + } + timeline = [parse_timestamp(value, name) for name, value in timeline_values.items()] + lifecycle_chronology = all(earlier <= later for earlier, later in pairwise(timeline)) + delta_detected = delta_details.get("delta_detected") + delta_change_present = bool( + delta_details.get("added_tables") + or delta_details.get("removed_tables") + or delta_details.get("changed_tables") + or delta_details.get("changed_structures") + or delta_details.get("total_row_delta") + ) + current_source_time = parse_timestamp( + current_source_snapshot.get("captured_at_utc"), "current source snapshot time" + ) + restore_started_time = parse_timestamp(restore.get("generated_at_utc"), "restore start time") + cleanup_completed_time = parse_timestamp(cleanup.get("completed_at_utc"), "cleanup completion time") + postflight_age_seconds = (verification_time - current_source_time).total_seconds() + lifecycle_age_seconds = (verification_time - cleanup_completed_time).total_seconds() + lifecycle_span_seconds = (current_source_time - restore_started_time).total_seconds() + postflight_after_lifecycle = current_source_time >= max( + parse_timestamp(reasoning.get("completed_at_utc"), "reasoning completion time"), + parse_timestamp(cleanup.get("completed_at_utc"), "cleanup completion time"), + ) + + required_reasoning_checks = ( + "exact_blind_prompt_without_ids", + "claim_and_sources_retrieved", + "discovery_show_evidence_completed", + "retrieval_receipts_proven", + "private_tls_read_only_target", + "generated_database_unchanged", + "canonical_counts_unchanged", + "database_calls_read_only", + "no_database_write", + "no_telegram_send", + "live_service_unchanged", + "live_profile_unchanged", + "temporary_profile_removed", + ) + checks = { + "source_schema_v2": source.get("schema") == SOURCE_SCHEMA and source.get("receipt_version") == 2, + "source_pass": source.get("status") == "pass" + and source.get("snapshot_exported") is True + and (source.get("source_service") or {}).get("unchanged") is True + and service_healthy((source.get("source_service") or {}).get("before")) + and (source.get("source_service") or {}).get("before") == (source.get("source_service") or {}).get("after"), + "source_authorized": bool(source.get("capture_authorization_ref")), + "source_authorization_bound": capture_binding.get("capture_authorization_ref") + == source.get("capture_authorization_ref"), + "source_service_unchanged": source.get("service_unchanged") is True, + "source_not_mutated": source.get("production_db_mutated") is False, + "current_source_schema_v2": current_source.get("schema") == SOURCE_SCHEMA + and current_source.get("receipt_version") == 2, + "current_source_pass": current_source.get("status") == "pass" + and current_source.get("snapshot_exported") is True + and current_source.get("service_unchanged") is True + and (current_source.get("source_service") or {}).get("unchanged") is True + and service_healthy((current_source.get("source_service") or {}).get("before")) + and (current_source.get("source_service") or {}).get("before") + == (current_source.get("source_service") or {}).get("after") + and current_source.get("production_db_mutated") is False, + "current_source_identity": current_source.get("source") == source.get("source") + and current_source_snapshot.get("system_identifier") == source_snapshot.get("system_identifier"), + "source_delta_pass": source_delta.get("artifact") == "vps_canonical_snapshot_postflight_delta" + and source_delta.get("schema") == DELTA_SCHEMA + and source_delta.get("status") == "pass", + "source_delta_receipts_bound": delta_baseline.get("capture_receipt_sha256") == sha256_file(source_path) + and delta_current.get("capture_receipt_sha256") == sha256_file(current_source_path), + "source_delta_manifests_bound": delta_baseline.get("manifest_sha256") + == (source.get("manifest") or {}).get("sha256") + and delta_current.get("manifest_sha256") == (current_source.get("manifest") or {}).get("sha256"), + "source_delta_manifest_sql_bound": delta_baseline.get("manifest_sql_sha256") + == (source.get("manifest") or {}).get("manifest_sql_sha256") + and delta_current.get("manifest_sql_sha256") + == (current_source.get("manifest") or {}).get("manifest_sql_sha256") + and delta_baseline.get("manifest_sql_sha256") == delta_current.get("manifest_sql_sha256"), + "source_delta_manifest_sql_reviewed": delta_baseline.get("manifest_sql_sha256") == REVIEWED_MANIFEST_SQL_SHA256 + and delta_current.get("manifest_sql_sha256") == REVIEWED_MANIFEST_SQL_SHA256 + and (source.get("manifest") or {}).get("manifest_sql_sha256") == REVIEWED_MANIFEST_SQL_SHA256 + and (current_source.get("manifest") or {}).get("manifest_sql_sha256") == REVIEWED_MANIFEST_SQL_SHA256 + and capture_binding.get("manifest_sql_sha256") == REVIEWED_MANIFEST_SQL_SHA256, + "source_delta_authorizations_bound": delta_baseline.get("capture_authorization_ref") + == source.get("capture_authorization_ref") + and delta_current.get("capture_authorization_ref") == current_source.get("capture_authorization_ref"), + "source_delta_explicit": isinstance(delta_details.get("delta_detected"), bool) + and isinstance(delta_details.get("added_tables"), list) + and isinstance(delta_details.get("removed_tables"), list) + and isinstance(delta_details.get("changed_tables"), list) + and isinstance(delta_details.get("changed_structures"), list) + and isinstance(delta_details.get("total_row_delta"), int), + "source_delta_baseline_shape": delta_baseline.get("run_id") == source.get("run_id") + and delta_baseline.get("captured_at_utc") == source_snapshot.get("captured_at_utc") + and delta_baseline.get("table_count") == (source.get("manifest") or {}).get("table_count") + and delta_baseline.get("total_rows") == (source.get("manifest") or {}).get("total_rows") + and bool(SHA256_RE.fullmatch(str(delta_baseline.get("table_fingerprint_sha256") or ""))) + and bool(SHA256_RE.fullmatch(str(delta_baseline.get("structure_fingerprint_sha256") or ""))), + "source_delta_current_shape": delta_current.get("run_id") == current_source.get("run_id") + and delta_current.get("captured_at_utc") == current_source_snapshot.get("captured_at_utc") + and delta_current.get("table_count") == (current_source.get("manifest") or {}).get("table_count") + and delta_current.get("total_rows") == (current_source.get("manifest") or {}).get("total_rows") + and bool(SHA256_RE.fullmatch(str(delta_current.get("table_fingerprint_sha256") or ""))) + and bool(SHA256_RE.fullmatch(str(delta_current.get("structure_fingerprint_sha256") or ""))), + "source_delta_consistent": delta_detected is delta_change_present + and delta_details.get("total_row_delta") + == (delta_current.get("total_rows") or 0) - (delta_baseline.get("total_rows") or 0) + and ( + delta_detected is True + or ( + delta_baseline.get("table_count") == delta_current.get("table_count") + and delta_baseline.get("total_rows") == delta_current.get("total_rows") + and delta_baseline.get("table_fingerprint_sha256") == delta_current.get("table_fingerprint_sha256") + and delta_baseline.get("structure_fingerprint_sha256") + == delta_current.get("structure_fingerprint_sha256") + ) + ), + "source_postflight_after_lifecycle": postflight_after_lifecycle, + "source_postflight_fresh": 0 <= postflight_age_seconds <= max_postflight_age_seconds, + "gcp_lifecycle_fresh": 0 <= lifecycle_age_seconds <= max_lifecycle_age_seconds, + "lifecycle_span_bounded": 0 <= lifecycle_span_seconds <= max_lifecycle_age_seconds, + "lifecycle_chronology": lifecycle_chronology, + "restore_pass": restore.get("artifact") == "gcp_generated_postgres_snapshot_restore" + and restore.get("status") == "pass", + "restore_target_is_disposable": isinstance(target_db, str) and target_db.startswith("teleo_clone_"), + "restore_run_id": isinstance(run_id, str) and run_id.startswith("gcp-restore-"), + "source_receipt_hash_bound": capture_binding.get("sha256") == sha256_file(source_path), + "source_provenance_bound": capture_binding.get("provenance_binding_sha256") + == (source.get("provenance_binding") or {}).get("sha256"), + "source_capture_checks": required_true_checks( + capture_binding.get("checks"), + { + "artifact", + "schema", + "receipt_version", + "status", + "snapshot_exported", + "service_unchanged", + "source_service_healthy", + "source_not_mutated", + "telegram_not_sent", + "run_id", + "capture_authorization_ref", + "source_identity", + "source_database", + "snapshot_identity", + "dump_sha256", + "dump_bytes", + "source_manifest_sha256", + "manifest_sql_sha256", + "source_context_sha256", + "table_count", + "total_rows", + "binding_algorithm", + "binding_payload", + "binding_sha256", + }, + ), + "source_dump_bound": capture_binding.get("dump_sha256") == (source.get("dump") or {}).get("sha256"), + "source_execution_contract_bound": restore_contract.get("source") == source.get("source"), + "source_manifest_bound": capture_binding.get("manifest_sha256") == (source.get("manifest") or {}).get("sha256"), + "restore_manifest_matches_parity_source": capture_binding.get("manifest_sha256") + == parity.get("source_manifest_sha256"), + "restore_target_manifest_matches_parity": restore_target.get("manifest_sha256") + == parity.get("target_manifest_sha256"), + "restore_private_tls": restore_connectivity.get("private_connectivity") is True + and restore_connectivity.get("ssl") is True + and restore_connectivity.get("sslmode") == "verify-full" + and restore_connectivity.get("server_identity_verified") is True + and restore_connectivity.get("ssl_server_name") == restore_contract.get("ssl_server_name") + and restore_connectivity.get("ssl_root_cert_sha256") == restore_contract.get("ssl_root_cert_sha256") + and is_rfc1918(restore_connectivity.get("server_address")), + "restore_public_ip_disabled": restore_cloudsql.get("public_ip_disabled") is True, + "restore_cloudsql_checks": required_true_checks( + restore_cloudsql.get("checks"), + { + "instance", + "state", + "postgres_16", + "public_ip_disabled", + "private_network", + "expected_private_host", + "no_non_private_address", + }, + ), + "restore_compute_checks": required_true_checks( + restore_compute.get("checks"), + { + "project", + "project_number", + "expected_network_project", + "instance", + "instance_id", + "zone", + "network", + "private_ip", + "service_account", + }, + ), + "restore_compute_contract": restore_compute.get("project") == restore_contract.get("project") + and restore_compute.get("instance") == restore_contract.get("compute_instance") + and restore_compute.get("zone") == restore_contract.get("compute_zone") + and restore_compute.get("expected_private_network") == restore_contract.get("private_network") + and bool(restore_compute.get("network_resource")) + and is_rfc1918(restore_compute.get("private_ip")) + and restore_contract.get("restore_service_account") == DEFAULT_RESTORE_SERVICE_ACCOUNT + and restore_compute.get("service_account") == DEFAULT_RESTORE_SERVICE_ACCOUNT, + "restore_tls_identity_contract": bool(restore_contract.get("ssl_server_name")) + and bool(SHA256_RE.fullmatch(str(restore_contract.get("ssl_root_cert_sha256") or ""))), + "restore_connectivity_compute_bound": restore_connectivity.get("source_compute") + == restore_compute.get("instance"), + "restore_inline_parity": restore_parity.get("status") == "pass" and restore_parity.get("problems") == [], + "restore_inline_parity_evidence": all(restore_inline_parity_checks.values()), + "restore_service_healthy_unchanged": restore_service.get("unchanged") is True + and service_healthy(restore_service.get("before")) + and restore_service.get("before") == restore_service.get("after"), + "parity_pass": parity.get("artifact") == "canonical_postgres_parity_verification" + and parity.get("status") == "pass" + and parity.get("scope") == "gcp_staging" + and parity.get("problems") == [], + "parity_target_bound": parity_details.get("target_database") == target_db, + "parity_exact_evidence": all(parity_evidence_checks.values()), + "parity_connectivity_hash_bound": restore_connectivity_proof.get("sha256") + == parity.get("connectivity_proof_sha256"), + "parity_private_proof": parity_connectivity.get("required") is True + and parity_connectivity_proof.get("artifact") == "gcp_private_postgres_connectivity" + and parity_connectivity_proof.get("status") == "pass" + and parity_connectivity_proof.get("schema") == "livingip.gcpPrivatePostgresConnectivity.v2" + and parity_connectivity_proof.get("restore_run_id") == run_id + and parity_connectivity_proof.get("private_connectivity") is True + and parity_connectivity_proof.get("public_ip_disabled") is True + and parity_connectivity_proof.get("target_database") == target_db + and parity_connectivity_proof.get("project") == restore_cloudsql.get("project") + and parity_connectivity_proof.get("cloudsql_instance") == restore_cloudsql.get("instance") + and parity_connectivity_proof.get("private_network") == restore_cloudsql.get("private_network") + and parity_connectivity_proof.get("source_compute") == reasoning.get("source_compute") + and parity_connectivity_proof.get("source_compute_instance_id") == restore_compute.get("instance_id") + and parity_connectivity_proof.get("source_compute_zone") == restore_compute.get("zone") + and is_rfc1918(parity_connectivity_proof.get("source_compute_private_ip")) + and parity_connectivity_proof.get("server_address") == restore_connectivity.get("server_address") + and parity_connectivity_proof.get("sslmode") == "verify-full" + and parity_connectivity_proof.get("server_identity_verified") is True + and parity_connectivity_proof.get("ssl_server_name") == restore_contract.get("ssl_server_name") + and parity_connectivity_proof.get("ssl_root_cert_sha256") == restore_contract.get("ssl_root_cert_sha256") + and required_true_checks( + parity_connectivity_proof.get("cloudsql_control_plane_checks"), + { + "instance", + "state", + "postgres_16", + "public_ip_disabled", + "private_network", + "expected_private_host", + "no_non_private_address", + }, + ) + and required_true_checks( + parity_connectivity_proof.get("compute_identity_checks"), + { + "project", + "project_number", + "expected_network_project", + "instance", + "instance_id", + "zone", + "network", + "private_ip", + "service_account", + }, + ), + "reasoning_schema": reasoning.get("schema") == BLIND_REASONING_SCHEMA, + "reasoning_pass": reasoning.get("status") == "pass" and not reasoning.get("errors"), + "reasoning_target_bound": reasoning.get("target_database") == target_db, + "reasoning_compute_bound": reasoning.get("source_compute") == restore_compute.get("instance"), + "reasoning_parity_hash_bound": reasoning_parity.get("sha256") == sha256_file(parity_path), + "reasoning_required_checks": all(reasoning_checks.get(name) is True for name in required_reasoning_checks), + "reasoning_producer_evidence": all(producer_reasoning_checks.values()), + "reasoning_compute_attestation": reasoning_compute_attestation.get("artifact") + == "gcp_reasoning_compute_attestation" + and reasoning_compute_attestation.get("schema") == REASONING_COMPUTE_SCHEMA + and reasoning_compute_attestation.get("status") == "pass" + and reasoning_compute_attestation.get("method") == "gce_metadata_server_v1" + and nonempty_all_true(reasoning_compute_checks) + and reasoning_compute_attestation.get("restore_run_id") == run_id + and reasoning_compute_attestation.get("target_database") == target_db + and all(value is False for value in reasoning_compute_mutation.values()) + and { + "cloudsql_changed", + "compute_changed", + "database_changed", + "telegram_message_sent", + } + <= set(reasoning_compute_mutation), + "reasoning_compute_receipt_bound": (reasoning_compute_attestation.get("reasoning_receipt") or {}).get("sha256") + == sha256_file(reasoning_path), + "reasoning_compute_identity_bound": reasoning_compute.get("project") == restore_compute.get("project") + and reasoning_compute.get("project_number") == restore_compute.get("project_number") + and reasoning_compute.get("instance") == restore_compute.get("instance") + and reasoning_compute.get("instance_id") == restore_compute.get("instance_id") + and reasoning_compute.get("zone") == restore_compute.get("zone") + and reasoning_compute.get("expected_private_network") == restore_compute.get("expected_private_network") + and reasoning_compute.get("network_resource") == restore_compute.get("network_resource") + and reasoning_compute.get("private_ip") == restore_compute.get("private_ip") + and reasoning_compute.get("service_account") == restore_compute.get("service_account") + and required_true_checks( + reasoning_compute.get("checks"), + { + "project", + "project_number", + "expected_network_project", + "instance", + "instance_id", + "zone", + "network", + "private_ip", + "service_account", + }, + ), + "reasoning_no_send_no_write": reasoning.get("posted_to_telegram") is False + and reasoning.get("database_write_attempted") is False, + "reasoning_fingerprint_unchanged": bool(before_fingerprint.get("sha256")) + and before_fingerprint.get("sha256") == after_fingerprint.get("sha256"), + "cleanup_pass": cleanup.get("artifact") == "gcp_generated_postgres_snapshot_cleanup" + and cleanup.get("status") == "pass", + "cleanup_run_bound": cleanup.get("run_id") == run_id and cleanup.get("target_database") == target_db, + "cleanup_restore_receipt_bound": (cleanup.get("restore_receipt") or {}).get("sha256") + == sha256_file(restore_path), + "cleanup_execution_contract_bound": bool(restore_contract) and cleanup_contract == restore_contract, + "cleanup_cloudsql_bound": cleanup_cloudsql.get("project") == restore_cloudsql.get("project") + and cleanup_cloudsql.get("instance") == restore_cloudsql.get("instance") + and cleanup_cloudsql.get("expected_private_host") == restore_cloudsql.get("expected_private_host") + and cleanup_cloudsql.get("expected_private_network") == restore_cloudsql.get("expected_private_network") + and cleanup_cloudsql.get("public_ip_disabled") is True + and required_true_checks( + cleanup_cloudsql.get("checks"), + { + "instance", + "state", + "postgres_16", + "public_ip_disabled", + "private_network", + "expected_private_host", + "no_non_private_address", + }, + ), + "cleanup_compute_bound": cleanup_compute.get("instance_id") == restore_compute.get("instance_id") + and cleanup_compute.get("project") == restore_compute.get("project") + and cleanup_compute.get("project_number") == restore_compute.get("project_number") + and cleanup_compute.get("instance") == restore_compute.get("instance") + and cleanup_compute.get("zone") == restore_compute.get("zone") + and cleanup_compute.get("expected_private_network") == restore_compute.get("expected_private_network") + and cleanup_compute.get("network_resource") == restore_compute.get("network_resource") + and cleanup_compute.get("private_ip") == restore_compute.get("private_ip") + and cleanup_compute.get("service_account") == restore_compute.get("service_account") + and required_true_checks( + cleanup_compute.get("checks"), + { + "project", + "project_number", + "expected_network_project", + "instance", + "instance_id", + "zone", + "network", + "private_ip", + "service_account", + }, + ), + "clone_absent": cleanup_database.get("existed_before") is True + and cleanup_database.get("clone_database_remaining") == 0, + "cleanup_service_unchanged": cleanup_service.get("unchanged") is True + and service_healthy(cleanup_service.get("before")) + and cleanup_service.get("before") == cleanup_service.get("after"), + } + failed = sorted(name for name, passed in checks.items() if not passed) + structurally_valid = not failed + return { + "artifact": "gcp_canonical_snapshot_lifecycle_verification", + "generated_at_utc": verification_time.isoformat(), + "status": "prepared_external_attestation_required" if structurally_valid else "fail", + "current_tier": "T2_structural_receipt_bundle", + "required_tier": "T3_live_private_gcp_staging", + "t3_receipt_ready": False, + "accepted_by_this_api": False, + "target_database": target_db, + "run_id": run_id, + "checks": checks, + "reasoning_producer_checks": producer_reasoning_checks, + "restore_inline_parity_checks": restore_inline_parity_checks, + "parity_evidence_checks": parity_evidence_checks, + "post_snapshot_source_delta": delta_details, + "timeline": timeline_values, + "postflight_age_seconds": postflight_age_seconds, + "max_postflight_age_seconds": max_postflight_age_seconds, + "lifecycle_age_seconds": lifecycle_age_seconds, + "lifecycle_span_seconds": lifecycle_span_seconds, + "max_lifecycle_age_seconds": max_lifecycle_age_seconds, + "failed_checks": failed, + "receipt_sha256": { + "source": sha256_file(source_path), + "current_source": sha256_file(current_source_path), + "source_delta": sha256_file(source_delta_path), + "restore": sha256_file(restore_path), + "parity": sha256_file(parity_path), + "reasoning": sha256_file(reasoning_path), + "reasoning_compute": sha256_file(reasoning_compute_path), + "cleanup": sha256_file(cleanup_path), + }, + "strongest_claim_allowed": ( + "caller-supplied lifecycle receipts are internally consistent; an independently authenticated platform receipt is still required" + if structurally_valid + else "lifecycle preparation is incomplete; failed checks must be repaired" + ), + "not_proven": ["production cutover", "ongoing replication", "Telegram delivery"], + } + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--source-receipt", required=True, type=Path) + parser.add_argument("--current-source-receipt", required=True, type=Path) + parser.add_argument("--source-delta-receipt", required=True, type=Path) + parser.add_argument("--restore-receipt", required=True, type=Path) + parser.add_argument("--parity-receipt", required=True, type=Path) + parser.add_argument("--reasoning-receipt", required=True, type=Path) + parser.add_argument("--reasoning-compute-receipt", required=True, type=Path) + parser.add_argument("--cleanup-receipt", required=True, type=Path) + parser.add_argument("--output", required=True, type=Path) + parser.add_argument("--max-postflight-age-seconds", default=900.0, type=float) + parser.add_argument("--max-lifecycle-age-seconds", default=3600.0, type=float) + args = parser.parse_args() + try: + payload = verify_lifecycle( + source_path=args.source_receipt, + current_source_path=args.current_source_receipt, + source_delta_path=args.source_delta_receipt, + restore_path=args.restore_receipt, + parity_path=args.parity_receipt, + reasoning_path=args.reasoning_receipt, + reasoning_compute_path=args.reasoning_compute_receipt, + cleanup_path=args.cleanup_receipt, + max_postflight_age_seconds=args.max_postflight_age_seconds, + max_lifecycle_age_seconds=args.max_lifecycle_age_seconds, + ) + except (OSError, ValueError, KeyError, TypeError) as exc: + payload = { + "artifact": "gcp_canonical_snapshot_lifecycle_verification", + "generated_at_utc": datetime.now(UTC).isoformat(), + "status": "fail", + "required_tier": "T3_live_private_gcp_staging", + "failed_checks": ["receipt_load_or_validation_error"], + "error": str(exc), + "strongest_claim_allowed": "lifecycle incomplete; receipts could not be validated", + } + write_private_json(args.output, payload) + print_private_receipt_summary(args.output, payload) + return 0 if payload.get("status") == "prepared_external_attestation_required" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/ops/verify_gcp_leoclean_runtime_permissions.py b/ops/verify_gcp_leoclean_runtime_permissions.py index 94ccc4f..09038a9 100644 --- a/ops/verify_gcp_leoclean_runtime_permissions.py +++ b/ops/verify_gcp_leoclean_runtime_permissions.py @@ -259,6 +259,7 @@ CATALOG_TRUE_FIELDS: tuple[str, ...] = ( "effective_setting_contract", ) + @dataclass(frozen=True) class CommandResult: returncode: int @@ -384,8 +385,7 @@ def _assert_trusted_executable(raw_path: str) -> None: except OSError: raise VerificationError("runtime_dependency_untrusted", "runtime_dependencies") from None if requested_metadata.st_uid != 0 or ( - not stat.S_ISLNK(requested_metadata.st_mode) - and requested_metadata.st_mode & (stat.S_IWGRP | stat.S_IWOTH) + not stat.S_ISLNK(requested_metadata.st_mode) and requested_metadata.st_mode & (stat.S_IWGRP | stat.S_IWOTH) ): raise VerificationError("runtime_dependency_untrusted", "runtime_dependencies") for candidate in (*_path_and_parents(requested.parent), *_path_and_parents(resolved)): @@ -790,9 +790,7 @@ def _catalog_privilege_posture_sql() -> str: allowlist_values = ",\n ".join( f"({_sql_literal(schema)}, {_sql_literal(relation)})" for schema, relation in READ_TABLE_ALLOWLIST ) - owner_insert_values = ",\n ".join( - f"({_sql_literal(column_name)})" for column_name in STAGE_OWNER_INSERT_COLUMNS - ) + owner_insert_values = ",\n ".join(f"({_sql_literal(column_name)})" for column_name in STAGE_OWNER_INSERT_COLUMNS) return f""" with runtime_role as ( select oid from pg_catalog.pg_roles where rolname = {_sql_literal(RUNTIME_DATABASE_ROLE)} diff --git a/ops/verify_postgres_parity_manifest.py b/ops/verify_postgres_parity_manifest.py index b88ca50..bf5a373 100755 --- a/ops/verify_postgres_parity_manifest.py +++ b/ops/verify_postgres_parity_manifest.py @@ -7,15 +7,27 @@ import argparse import hashlib import ipaddress import json +import re from datetime import UTC, datetime from pathlib import Path from typing import Any +try: + from .private_receipt_io import print_private_receipt_summary, write_private_json +except ImportError: # pragma: no cover - direct script execution + from private_receipt_io import print_private_receipt_summary, write_private_json + +REVIEWED_MANIFEST_SQL_SHA256 = "2bd3e353627d44d4a292949952c06b1897d1a9a9e98243905c7a834d007db188" +ROWSET_MD5_RE = re.compile(r"[0-9a-f]{32}\Z") + SINGLETON_KINDS = { "identity", "schemas", "extensions", "application_roles", + "role_memberships", + "object_ownership", + "object_acl", "columns", "constraints", "indexes", @@ -33,6 +45,14 @@ def canonical_hash(value: Any) -> str: return hashlib.sha256(payload.encode()).hexdigest() +def file_sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + def load_manifest(path: Path) -> dict[str, Any]: singleton: dict[str, dict[str, Any]] = {} tables: dict[str, dict[str, Any]] = {} @@ -72,7 +92,18 @@ def load_manifest(path: Path) -> dict[str, Any]: def item_map(row: dict[str, Any], key: str) -> dict[str, dict[str, Any]]: - return {str(item[key]): item for item in row.get("items", [])} + items = row.get("items", []) + if not isinstance(items, list): + raise ValueError(f"manifest {row.get('kind')} items must be a list") + mapped: dict[str, dict[str, Any]] = {} + for item in items: + if not isinstance(item, dict) or key not in item: + raise ValueError(f"manifest {row.get('kind')} item is missing {key}") + item_key = str(item[key]) + if item_key in mapped: + raise ValueError(f"manifest {row.get('kind')} contains duplicate {key}={item_key}") + mapped[item_key] = item + return mapped def compare_manifests( @@ -107,9 +138,14 @@ def compare_manifests( + json.dumps(sorted(set(target_tables) - set(source_tables))) ) table_mismatches = [] + table_hash_problems = [] for table in sorted(set(source_tables) & set(target_tables)): source_row = source_tables[table] target_row = target_tables[table] + for side, row in (("source", source_row), ("target", target_row)): + rowset_md5 = row.get("rowset_md5") + if not isinstance(rowset_md5, str) or not ROWSET_MD5_RE.fullmatch(rowset_md5): + table_hash_problems.append(f"{table}:{side}:rowset_md5_invalid") mismatch = { field: {"source": source_row.get(field), "target": target_row.get(field)} for field in ("row_count", "rowset_md5") @@ -119,6 +155,8 @@ def compare_manifests( table_mismatches.append({"table": table, "mismatch": mismatch}) if table_mismatches: problems.append(f"table_content_mismatches={len(table_mismatches)}") + if table_hash_problems: + problems.append(f"table_hash_problems={len(table_hash_problems)}") structural_hashes: dict[str, dict[str, str]] = {} for kind in ( @@ -132,6 +170,9 @@ def compare_manifests( "triggers", "types", "policies", + "role_memberships", + "object_ownership", + "object_acl", ): source_hash = canonical_hash(source_singleton[kind].get("items", [])) target_hash = canonical_hash(target_singleton[kind].get("items", [])) @@ -142,19 +183,19 @@ def compare_manifests( source_extensions = item_map(source_singleton["extensions"], "name") target_extensions = item_map(target_singleton["extensions"], "name") extension_mismatches = { - name: {"source": item, "target": target_extensions.get(name)} - for name, item in source_extensions.items() - if target_extensions.get(name) != item + name: {"source": source_extensions.get(name), "target": target_extensions.get(name)} + for name in sorted(set(source_extensions) | set(target_extensions)) + if source_extensions.get(name) != target_extensions.get(name) } if extension_mismatches: - problems.append(f"required_extension_mismatches={len(extension_mismatches)}") + problems.append(f"extension_mismatches={len(extension_mismatches)}") source_roles = item_map(source_singleton["application_roles"], "name") target_roles = item_map(target_singleton["application_roles"], "name") role_mismatches = { - name: {"source": item, "target": target_roles.get(name)} - for name, item in source_roles.items() - if target_roles.get(name) != item + name: {"source": source_roles.get(name), "target": target_roles.get(name)} + for name in sorted(set(source_roles) | set(target_roles)) + if source_roles.get(name) != target_roles.get(name) } if role_mismatches: problems.append(f"application_role_mismatches={len(role_mismatches)}") @@ -204,8 +245,10 @@ def compare_manifests( "source_total_rows": sum(int(row["row_count"]) for row in source_tables.values()), "target_total_rows": sum(int(row["row_count"]) for row in target_tables.values()), "table_mismatches": table_mismatches, + "table_hash_problems": table_hash_problems, "structural_hashes": structural_hashes, "required_extension_mismatches": extension_mismatches, + "target_extra_extensions": sorted(set(target_extensions) - set(source_extensions)), "application_role_mismatches": role_mismatches, "target_extra_application_roles": sorted(set(target_roles) - set(source_roles)), "performance": performance_rows, @@ -280,6 +323,14 @@ def load_connectivity( problems.append("gcp_target_manifest_tls_not_proven") if proof.get("ssl") is not True: problems.append("gcp_connectivity_tls_not_proven") + if proof.get("sslmode") != "verify-full": + problems.append("gcp_connectivity_server_identity_mode_not_proven") + if proof.get("server_identity_verified") is not True: + problems.append("gcp_connectivity_server_identity_not_proven") + if not proof.get("ssl_server_name"): + problems.append("gcp_connectivity_ssl_server_name_missing") + if not re.fullmatch(r"[0-9a-f]{64}", str(proof.get("ssl_root_cert_sha256") or "")): + problems.append("gcp_connectivity_ssl_root_cert_hash_missing") return problems, {"required": True, "path": str(path), "proof": proof} @@ -295,6 +346,9 @@ def main() -> int: args = parser.parse_args() try: + source_manifest_sha256 = file_sha256(args.source) + target_manifest_sha256 = file_sha256(args.target) + connectivity_proof_sha256 = file_sha256(args.connectivity_proof) if args.connectivity_proof else None source = load_manifest(args.source) target = load_manifest(args.target) problems, details = compare_manifests( @@ -316,18 +370,26 @@ def main() -> int: problems = [f"manifest_error={exc}"] details = {} connectivity = {"required": args.scope == "gcp_staging", "status": "not_checked"} + source_manifest_sha256 = None + target_manifest_sha256 = None + connectivity_proof_sha256 = None error = str(exc) + completed_at_utc = datetime.now(UTC).isoformat() payload = { "artifact": "canonical_postgres_parity_verification", - "generated_at_utc": datetime.now(UTC).isoformat(), + "generated_at_utc": completed_at_utc, + "completed_at_utc": completed_at_utc, "scope": args.scope, "source_manifest": str(args.source), + "source_manifest_sha256": source_manifest_sha256, "target_manifest": str(args.target), + "target_manifest_sha256": target_manifest_sha256, "status": status, "problems": problems, "details": details, "private_connectivity": connectivity, + "connectivity_proof_sha256": connectivity_proof_sha256, "error": error, "not_proven_by_this_artifact": [ "GCP staging Leo composition replay", @@ -335,9 +397,8 @@ def main() -> int: "production cutover", ], } - args.output.parent.mkdir(parents=True, exist_ok=True) - args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") - print(json.dumps(payload, indent=2, sort_keys=True)) + write_private_json(args.output, payload) + print_private_receipt_summary(args.output, payload) return 0 if status == "pass" else 1 diff --git a/ops/verify_vps_canonical_snapshot_delta.py b/ops/verify_vps_canonical_snapshot_delta.py new file mode 100644 index 0000000..4f388f4 --- /dev/null +++ b/ops/verify_vps_canonical_snapshot_delta.py @@ -0,0 +1,303 @@ +#!/usr/bin/env python3 +"""Bind a postflight VPS manifest to a captured snapshot and report exact deltas.""" + +from __future__ import annotations + +import argparse +import json +import re +import stat +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + +try: + from .capture_vps_canonical_postgres_snapshot import ( + SOURCE_SNAPSHOT_RECEIPT_SCHEMA, + build_provenance_binding, + ) + from .private_receipt_io import print_private_receipt_summary, write_private_json + from .verify_postgres_parity_manifest import ( + REVIEWED_MANIFEST_SQL_SHA256, + SINGLETON_KINDS, + canonical_hash, + file_sha256, + load_manifest, + ) +except ImportError: # pragma: no cover - direct script execution + from capture_vps_canonical_postgres_snapshot import SOURCE_SNAPSHOT_RECEIPT_SCHEMA, build_provenance_binding + from private_receipt_io import print_private_receipt_summary, write_private_json + from verify_postgres_parity_manifest import ( + REVIEWED_MANIFEST_SQL_SHA256, + SINGLETON_KINDS, + canonical_hash, + file_sha256, + load_manifest, + ) + +DELTA_SCHEMA = "livingip.vpsCanonicalSnapshotDelta.v1" +SHA256_RE = re.compile(r"[0-9a-f]{64}\Z") + + +def service_healthy(value: Any) -> bool: + if not isinstance(value, dict): + return False + try: + main_pid = int(value.get("MainPID") or 0) + restarts = int(value.get("NRestarts") or 0) + except (TypeError, ValueError): + return False + return ( + value.get("ActiveState") == "active" and value.get("SubState") == "running" and main_pid > 0 and restarts >= 0 + ) + + +def regular_file(path: Path, label: str) -> Path: + try: + mode = path.lstat().st_mode + except OSError as exc: + raise ValueError(f"{label} is unavailable: {exc}") from exc + if not stat.S_ISREG(mode) or path.is_symlink(): + raise ValueError(f"{label} must be a regular non-symlink file") + return path.resolve() + + +def load_json(path: Path, label: str) -> dict[str, Any]: + path = regular_file(path, label) + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError as exc: + raise ValueError(f"{label} is invalid JSON") from exc + if not isinstance(payload, dict): + raise ValueError(f"{label} must be a JSON object") + return payload + + +def parse_timestamp(value: Any, label: str) -> datetime: + text = str(value or "") + if text.endswith("Z"): + text = text[:-1] + "+00:00" + try: + parsed = datetime.fromisoformat(text) + except ValueError as exc: + raise ValueError(f"{label} is not an ISO-8601 timestamp") from exc + if parsed.tzinfo is None: + raise ValueError(f"{label} must include a timezone") + return parsed.astimezone(UTC) + + +def validate_capture( + receipt_path: Path, + manifest_path: Path, + *, + expected_authorization_ref: str, + label: str, +) -> tuple[dict[str, Any], dict[str, Any]]: + receipt = load_json(receipt_path, f"{label} receipt") + manifest_path = regular_file(manifest_path, f"{label} manifest") + manifest = load_manifest(manifest_path) + snapshot = receipt.get("snapshot") or {} + source = receipt.get("source") or {} + dump = receipt.get("dump") or {} + manifest_receipt = receipt.get("manifest") or {} + source_context = receipt.get("source_context") or {} + source_service = receipt.get("source_service") or {} + binding = receipt.get("provenance_binding") or {} + expected_binding = build_provenance_binding( + run_id=str(receipt.get("run_id") or ""), + authorization_ref=str(receipt.get("capture_authorization_ref") or ""), + source=source, + snapshot=snapshot, + dump_sha256=str(dump.get("sha256") or ""), + manifest_sql_sha256=str(manifest_receipt.get("manifest_sql_sha256") or ""), + source_manifest_sha256=file_sha256(manifest_path), + source_context_sha256=str(source_context.get("sha256") or ""), + ) + total_rows = sum(int(row["row_count"]) for row in manifest["tables"].values()) + checks = { + "schema": receipt.get("schema") == SOURCE_SNAPSHOT_RECEIPT_SCHEMA, + "receipt_version": receipt.get("receipt_version") == 2, + "status": receipt.get("status") == "pass", + "authorization": receipt.get("capture_authorization_ref") == expected_authorization_ref, + "snapshot_exported": receipt.get("snapshot_exported") is True, + "service_unchanged": receipt.get("service_unchanged") is True, + "source_service_healthy": source_service.get("unchanged") is True + and service_healthy(source_service.get("before")) + and source_service.get("before") == source_service.get("after"), + "source_not_mutated": receipt.get("production_db_mutated") is False, + "telegram_not_sent": receipt.get("telegram_send_attempted") is False, + "manifest_hash": manifest_receipt.get("sha256") == file_sha256(manifest_path), + "manifest_table_count": manifest_receipt.get("table_count") == len(manifest["tables"]), + "manifest_total_rows": manifest_receipt.get("total_rows") == total_rows, + "manifest_read_only": manifest["singleton"]["identity"].get("transaction_read_only") == "on", + "manifest_database": manifest["singleton"]["identity"].get("database") == source.get("database"), + "binding_payload": binding.get("payload") == expected_binding["payload"], + "binding_sha256": binding.get("sha256") == expected_binding["sha256"], + "binding_algorithm": binding.get("algorithm") == "sha256", + "dump_sha256": bool(SHA256_RE.fullmatch(str(dump.get("sha256") or ""))), + "manifest_sql_sha256": bool(SHA256_RE.fullmatch(str(manifest_receipt.get("manifest_sql_sha256") or ""))), + "manifest_sql_reviewed": manifest_receipt.get("manifest_sql_sha256") == REVIEWED_MANIFEST_SQL_SHA256, + "source_context_sha256": bool(SHA256_RE.fullmatch(str(source_context.get("sha256") or ""))), + "snapshot_identity": all( + bool(snapshot.get(field)) + for field in ("captured_at_utc", "exported_snapshot_id", "txid_snapshot", "wal_lsn", "system_identifier") + ), + } + failed = sorted(name for name, passed in checks.items() if not passed) + if failed: + raise ValueError(f"{label} capture failed validation: " + ", ".join(failed)) + return receipt, manifest + + +def manifest_fingerprint(manifest: dict[str, Any]) -> dict[str, Any]: + tables = { + name: {"row_count": int(row["row_count"]), "rowset_md5": row.get("rowset_md5")} + for name, row in sorted(manifest["tables"].items()) + } + structures = { + kind: canonical_hash(manifest["singleton"][kind].get("items", [])) + for kind in sorted(SINGLETON_KINDS - {"identity"}) + } + return { + "table_count": len(tables), + "total_rows": sum(row["row_count"] for row in tables.values()), + "table_fingerprint_sha256": canonical_hash(tables), + "structure_fingerprint_sha256": canonical_hash(structures), + "tables": tables, + "structures": structures, + } + + +def verify_delta( + *, + baseline_receipt_path: Path, + baseline_manifest_path: Path, + baseline_authorization_ref: str, + current_receipt_path: Path, + current_manifest_path: Path, + current_authorization_ref: str, +) -> dict[str, Any]: + baseline_receipt, baseline_manifest = validate_capture( + baseline_receipt_path, + baseline_manifest_path, + expected_authorization_ref=baseline_authorization_ref, + label="baseline", + ) + current_receipt, current_manifest = validate_capture( + current_receipt_path, + current_manifest_path, + expected_authorization_ref=current_authorization_ref, + label="current", + ) + baseline_snapshot = baseline_receipt["snapshot"] + current_snapshot = current_receipt["snapshot"] + baseline_time = parse_timestamp(baseline_snapshot["captured_at_utc"], "baseline snapshot time") + current_time = parse_timestamp(current_snapshot["captured_at_utc"], "current snapshot time") + if current_time <= baseline_time: + raise ValueError("current source capture must be newer than the restored baseline capture") + if current_receipt.get("source") != baseline_receipt.get("source"): + raise ValueError("current source identity does not match the restored baseline source") + if current_snapshot.get("system_identifier") != baseline_snapshot.get("system_identifier"): + raise ValueError("current source PostgreSQL system identifier changed") + baseline_manifest_sql_sha256 = str((baseline_receipt.get("manifest") or {}).get("manifest_sql_sha256") or "") + current_manifest_sql_sha256 = str((current_receipt.get("manifest") or {}).get("manifest_sql_sha256") or "") + if baseline_manifest_sql_sha256 != current_manifest_sql_sha256: + raise ValueError("baseline and current captures used different manifest SQL algorithms") + + baseline = manifest_fingerprint(baseline_manifest) + current = manifest_fingerprint(current_manifest) + baseline_tables = baseline.pop("tables") + current_tables = current.pop("tables") + baseline_structures = baseline.pop("structures") + current_structures = current.pop("structures") + added_tables = sorted(set(current_tables) - set(baseline_tables)) + removed_tables = sorted(set(baseline_tables) - set(current_tables)) + changed_tables = [ + {"table": table, "baseline": baseline_tables[table], "current": current_tables[table]} + for table in sorted(set(baseline_tables) & set(current_tables)) + if baseline_tables[table] != current_tables[table] + ] + changed_structures = [ + { + "kind": kind, + "baseline_sha256": baseline_structures.get(kind), + "current_sha256": current_structures.get(kind), + } + for kind in sorted(set(baseline_structures) | set(current_structures)) + if baseline_structures.get(kind) != current_structures.get(kind) + ] + delta_detected = bool(added_tables or removed_tables or changed_tables or changed_structures) + return { + "artifact": "vps_canonical_snapshot_postflight_delta", + "schema": DELTA_SCHEMA, + "generated_at_utc": datetime.now(UTC).isoformat(), + "status": "pass", + "baseline": { + "run_id": baseline_receipt["run_id"], + "capture_authorization_ref": baseline_authorization_ref, + "captured_at_utc": baseline_snapshot["captured_at_utc"], + "capture_receipt_sha256": file_sha256(baseline_receipt_path), + "manifest_sha256": file_sha256(baseline_manifest_path), + "manifest_sql_sha256": baseline_manifest_sql_sha256, + **baseline, + }, + "current": { + "run_id": current_receipt["run_id"], + "capture_authorization_ref": current_authorization_ref, + "captured_at_utc": current_snapshot["captured_at_utc"], + "capture_receipt_sha256": file_sha256(current_receipt_path), + "manifest_sha256": file_sha256(current_manifest_path), + "manifest_sql_sha256": current_manifest_sql_sha256, + **current, + }, + "delta": { + "delta_detected": delta_detected, + "added_tables": added_tables, + "removed_tables": removed_tables, + "changed_tables": changed_tables, + "changed_structures": changed_structures, + "total_row_delta": current["total_rows"] - baseline["total_rows"], + }, + "strongest_claim_allowed": ( + "restored snapshot still matches the newest postflight VPS capture" + if not delta_detected + else "restored snapshot parity is bounded to its capture; post-snapshot VPS deltas are explicit" + ), + } + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--baseline-receipt", required=True, type=Path) + parser.add_argument("--baseline-manifest", required=True, type=Path) + parser.add_argument("--baseline-authorization-ref", required=True) + parser.add_argument("--current-receipt", required=True, type=Path) + parser.add_argument("--current-manifest", required=True, type=Path) + parser.add_argument("--current-authorization-ref", required=True) + parser.add_argument("--output", required=True, type=Path) + args = parser.parse_args() + try: + payload = verify_delta( + baseline_receipt_path=args.baseline_receipt, + baseline_manifest_path=args.baseline_manifest, + baseline_authorization_ref=args.baseline_authorization_ref, + current_receipt_path=args.current_receipt, + current_manifest_path=args.current_manifest, + current_authorization_ref=args.current_authorization_ref, + ) + except (OSError, ValueError, KeyError, TypeError, json.JSONDecodeError) as exc: + payload = { + "artifact": "vps_canonical_snapshot_postflight_delta", + "schema": DELTA_SCHEMA, + "generated_at_utc": datetime.now(UTC).isoformat(), + "status": "fail", + "error": str(exc), + "strongest_claim_allowed": "post-snapshot VPS delta is not proven", + } + write_private_json(args.output, payload) + print_private_receipt_summary(args.output, payload) + return 0 if payload.get("status") == "pass" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/run_gcp_generated_db_direct_claim_suite.py b/scripts/run_gcp_generated_db_direct_claim_suite.py index 6f2cc0a..1265b83 100644 --- a/scripts/run_gcp_generated_db_direct_claim_suite.py +++ b/scripts/run_gcp_generated_db_direct_claim_suite.py @@ -35,11 +35,15 @@ if __name__ == "__main__" and should_reexec_in_hermes(Path(sys.executable)): ) HERE = Path(__file__).resolve().parent +REPO_ROOT = HERE.parent +sys.path.insert(0, str(REPO_ROOT)) sys.path.insert(0, str(HERE)) import run_leo_clone_bound_handler_checkpoint as bound # noqa: E402 import working_leo_open_ended_benchmark as benchmark # noqa: E402 +from ops.verify_postgres_parity_manifest import REVIEWED_MANIFEST_SQL_SHA256 # noqa: E402 + SCHEMA = "livingip.gcpGeneratedDbDirectClaimSuite.v1" SOURCE_COMPUTE = "teleo-prod-1" DEFAULT_SERVICE = "leoclean-gcp-prod-parallel.service" @@ -49,8 +53,7 @@ DEFAULT_SECRET = "gcp-teleo-pgvector-standby-postgres-password" DEFAULT_SSL_ROOT_CERT = Path("/usr/local/libexec/livingip/leoclean-kb/cloudsql-server-ca.pem") DEFAULT_SYSTEM_IDENTIFIER = "7659718422914359312" DEFAULT_MANIFEST_SQL = HERE.parent / "ops" / "postgres_parity_manifest.sql" -REVIEWED_CLOUDSQL_TOOL_SHA256 = "17d5213756ea6492e4232784a04b7f584afee5dce5184b33a2227e58847472c1" -REVIEWED_MANIFEST_SQL_SHA256 = "8b8cdc25d54fdd8de05eb38c6e4423d2836953eb6012d4545f5c9c71b5f0150a" +REVIEWED_CLOUDSQL_TOOL_SHA256 = "f71d9dd5cfe5dd10ba0b4439a7b9631a0cb824ce6e05b1faa69dc25690f257f5" COUNT_READBACK_RE = re.compile( r"DB readback:\s*claims:\s*`?(?P\d+)`?;\s*" r"sources:\s*`?(?P\d+)`?;\s*" diff --git a/scripts/run_gcp_generated_db_working_leo_suite.py b/scripts/run_gcp_generated_db_working_leo_suite.py index d3bc26e..f2798de 100644 --- a/scripts/run_gcp_generated_db_working_leo_suite.py +++ b/scripts/run_gcp_generated_db_working_leo_suite.py @@ -98,9 +98,7 @@ def load_pinned_receipt(path: Path, expected_sha256: str, artifact: str) -> tupl return payload, actual_sha256 -def validate_target_identity_receipt( - args: argparse.Namespace, path: Path, expected_sha256: str -) -> dict[str, Any]: +def validate_target_identity_receipt(args: argparse.Namespace, path: Path, expected_sha256: str) -> dict[str, Any]: payload, actual_sha256 = load_pinned_receipt(path, expected_sha256, TARGET_IDENTITY_ARTIFACT) required = { "target_database": args.target_db, @@ -130,9 +128,7 @@ def validate_target_identity_receipt( } -def validate_source_baseline_receipt( - args: argparse.Namespace, path: Path, expected_sha256: str -) -> dict[str, Any]: +def validate_source_baseline_receipt(args: argparse.Namespace, path: Path, expected_sha256: str) -> dict[str, Any]: payload, actual_sha256 = load_pinned_receipt(path, expected_sha256, SOURCE_BASELINE_ARTIFACT) if payload.get("project") != args.project or payload.get("instance") != args.instance: raise bound.CheckpointError("source baseline receipt project/instance context does not match") @@ -192,8 +188,7 @@ def composition_check_contract() -> list[str]: if "source_packet_validated" in keys and "isolated_restart_and_memory_survived" in keys: if len(keys) != EXPECTED_COMPOSITION_CHECK_COUNT: raise bound.CheckpointError( - "composition scorer drifted: " - f"expected {EXPECTED_COMPOSITION_CHECK_COUNT} checks, found {len(keys)}" + f"composition scorer drifted: expected {EXPECTED_COMPOSITION_CHECK_COUNT} checks, found {len(keys)}" ) return keys raise bound.CheckpointError("existing composition scorer contract could not be located") @@ -334,8 +329,7 @@ def validate_capability_receipt( "session_user": receipt.get("session_user") == role, "current_user": receipt.get("current_user") == role, "current_database": receipt.get("current_database") == args.target_db, - "default_transaction_read_only": receipt.get("default_transaction_read_only") - == ("off" if writable else "on"), + "default_transaction_read_only": receipt.get("default_transaction_read_only") == ("off" if writable else "on"), "not_superuser": receipt.get("is_superuser") is False, "can_select_proposals": receipt.get("can_select_proposals") is True, "cannot_update_proposals": receipt.get("can_update_proposals") is False, @@ -370,12 +364,7 @@ def validate_capability_receipt( "can_insert_canonical": True, }, } - checks.update( - { - name: receipt.get(name) is expected - for name, expected in expected_capabilities[purpose].items() - } - ) + checks.update({name: receipt.get(name) is expected for name, expected in expected_capabilities[purpose].items()}) failed = sorted(name for name, passed in checks.items() if not passed) if failed: raise bound.CheckpointError(f"{purpose} phase capability receipt failed: " + ", ".join(failed)) @@ -581,7 +570,9 @@ def verify_distinct_credential_values(executor: CloudSqlExecutor) -> dict[str, A values.clear() -def _completed(command: list[str], returncode: int, stdout: str = "", stderr: str = "") -> subprocess.CompletedProcess[str]: +def _completed( + command: list[str], returncode: int, stdout: str = "", stderr: str = "" +) -> subprocess.CompletedProcess[str]: return subprocess.CompletedProcess(command, returncode, stdout, stderr) @@ -652,13 +643,10 @@ def _target_identity(args: argparse.Namespace, db_identity: dict[str, Any]) -> d } -def assert_live_target_identity( - args: argparse.Namespace, retained: dict[str, Any], live: dict[str, Any] -) -> None: +def assert_live_target_identity(args: argparse.Namespace, retained: dict[str, Any], live: dict[str, Any]) -> None: checks = { "target_database": live.get("current_database") == retained.get("target_database") == args.target_db, - "system_identifier": str(live.get("system_identifier") or "") - == str(retained.get("system_identifier") or ""), + "system_identifier": str(live.get("system_identifier") or "") == str(retained.get("system_identifier") or ""), "server_address": str(live.get("server_address") or "") == str(retained.get("server_address") or "") == args.host, @@ -751,7 +739,7 @@ def _cloudsql_lifecycle_wrapper( tool_log=tool_log, run_nonce=run_nonce, ) - marker = "if ! PGPASSWORD=\"$(gcloud secrets versions access latest" + marker = 'if ! PGPASSWORD="$(gcloud secrets versions access latest' if marker not in base: raise bound.CheckpointError("reviewed Cloud SQL wrapper shape changed; lifecycle injection refused") internal = shlex.join( @@ -882,12 +870,16 @@ class RuntimeAdapter: self._assert_target(container, database) return self.executor.read(sql) - def _approve(self, _guard_args: argparse.Namespace, proposal_id: str, database: str, *, dry_run: bool = False) -> subprocess.CompletedProcess[str]: + def _approve( + self, _guard_args: argparse.Namespace, proposal_id: str, database: str, *, dry_run: bool = False + ) -> subprocess.CompletedProcess[str]: if database != self.args.target_db: raise bound.CheckpointError("review attempted database fallback") return _review_command(self.executor, proposal_id, dry_run=dry_run) - def _apply(self, _guard_args: argparse.Namespace, proposal_id: str, database: str, *, dry_run: bool = False) -> subprocess.CompletedProcess[str]: + def _apply( + self, _guard_args: argparse.Namespace, proposal_id: str, database: str, *, dry_run: bool = False + ) -> subprocess.CompletedProcess[str]: if database != self.args.target_db: raise bound.CheckpointError("apply attempted database fallback") return _apply_command(self.executor, proposal_id, dry_run=dry_run) @@ -935,9 +927,7 @@ class RuntimeAdapter: self._restore() raise - def _assert_disposable_target( - self, container: str, database: str - ) -> tuple[dict[str, Any], dict[str, Any]]: + def _assert_disposable_target(self, container: str, database: str) -> tuple[dict[str, Any], dict[str, Any]]: self._assert_target(container, database) target = self._container_identity(container) source = self._container_identity(bound.PRODUCTION_CONTAINER) @@ -1201,9 +1191,7 @@ def catalog_session_isolation(results: list[dict[str, Any]]) -> dict[str, bool]: dc = [result for result in results if str(result.get("prompt_id") or "").startswith("DC-")] session_ids = [result.get("persisted_session_id") for result in dc] return { - "all_profiles_unique": bool(profile_ids) - and all(profile_ids) - and len(profile_ids) == len(set(profile_ids)), + "all_profiles_unique": bool(profile_ids) and all(profile_ids) and len(profile_ids) == len(set(profile_ids)), "dc_isolated": len(dc) == len(benchmark.M3TAVERSAL_DIRECT_CLAIM_FOLLOWUP_SCENARIOS) and all(result.get("fresh_private_profile") is True and result.get("prior_prompt_ids") == [] for result in dc) and all(session_ids) @@ -1448,9 +1436,7 @@ async def run_live_suite(args: argparse.Namespace, validated: dict[str, Any]) -> report[name] = capture() except Exception as exc: report[name] = None - report["errors"].append( - {"phase": name, "type": type(exc).__name__, "message": safe_message(exc)} - ) + report["errors"].append({"phase": name, "type": type(exc).__name__, "message": safe_message(exc)}) report["database_fingerprint_after"] = ( database_fingerprint_from_manifest(report["database_manifest_after"]) if report.get("database_manifest_after") @@ -1509,12 +1495,10 @@ async def run_live_suite(args: argparse.Namespace, validated: dict[str, Any]) -> "source_baseline_is_independently_retained_and_physically_distinct": bool(source_baseline.get("sha256")) and source_identity.get("system_identifier") and str(source_identity.get("system_identifier")) != str(before_identity.get("system_identifier")), - "proposal_lifecycle_exact_under_gcp_supported_checks": report["lifecycle_gcp_evaluation"]["passed"] - is True, + "proposal_lifecycle_exact_under_gcp_supported_checks": report["lifecycle_gcp_evaluation"]["passed"] is True, "process_memory_survived": (composition_checks.get("isolated_restart_and_memory_survived") is True) and (lifecycle_checks.get("isolated_handler_reopened_with_same_session") is True), - "database_identity_stable": bool(before_identity) - and before_identity == report.get("database_identity_after"), + "database_identity_stable": bool(before_identity) and before_identity == report.get("database_identity_after"), "physical_target_matches_sha_pinned_receipt": bool(target_identity.get("sha256")) and str(before_identity.get("system_identifier")) == target_identity.get("system_identifier"), "service_unchanged": before_service == report.get("service_after"), @@ -1525,12 +1509,8 @@ async def run_live_suite(args: argparse.Namespace, validated: dict[str, Any]) -> ) is True and report["object_level_changes"].get("lifecycle_unrelated_rows_unchanged") is True - and all( - item.get("changed") is True - for item in (report["object_level_changes"].get("objects") or {}).values() - ), - "phase_roles_and_capabilities_verified": {"read", "stage", "review", "apply"} - <= operation_purposes + and all(item.get("changed") is True for item in (report["object_level_changes"].get("objects") or {}).values()), + "phase_roles_and_capabilities_verified": {"read", "stage", "review", "apply"} <= operation_purposes and bool(executor.receipts) and all( ( @@ -1538,18 +1518,12 @@ async def run_live_suite(args: argparse.Namespace, validated: dict[str, Any]) -> and receipt["purpose"] in ALLOWED_WRITE_PURPOSES and receipt["default_transaction_read_only"] == "off" ) - or ( - receipt["writable"] is False - and receipt["default_transaction_read_only"] == "on" - ) + or (receipt["writable"] is False and receipt["default_transaction_read_only"] == "on") for receipt in executor.receipts ), "credential_values_verified_distinct_without_retention": report["credential_value_separation"] == {"checked": True, "distinct": True, "credential_count": 4}, - "no_send_proven_by_component_tool_surfaces": catalog_checks.get( - "send_and_delivery_absent_per_prompt" - ) - is True + "no_send_proven_by_component_tool_surfaces": catalog_checks.get("send_and_delivery_absent_per_prompt") is True and composition_checks.get("no_telegram_send") is True and lifecycle_checks.get("no_telegram_send") is True, "no_systemd_restart": report["systemd_restart_attempted"] is False, @@ -1743,9 +1717,7 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace: args.instance, ) if not all(receipt_inputs): - parser.error( - "execution requires --instance plus SHA-pinned target identity and source baseline receipts" - ) + parser.error("execution requires --instance plus SHA-pinned target identity and source baseline receipts") missing_flags = [name for name in ("stage", "review", "apply") if not getattr(args, f"allow_{name}")] if missing_flags: parser.error("execution requires explicit controller flags: " + ", ".join(missing_flags)) diff --git a/tests/test_attest_gcp_reasoning_compute.py b/tests/test_attest_gcp_reasoning_compute.py new file mode 100644 index 0000000..49190f7 --- /dev/null +++ b/tests/test_attest_gcp_reasoning_compute.py @@ -0,0 +1,130 @@ +import json +from pathlib import Path + +import pytest + +from ops import attest_gcp_reasoning_compute as attestation + + +def reasoning_receipt(path: Path) -> Path: + path.write_text( + json.dumps( + { + "schema": attestation.REASONING_SCHEMA, + "status": "pass", + "generated_at_utc": "2026-07-15T01:00:00+00:00", + "completed_at_utc": "2026-07-15T01:01:00+00:00", + "target_database": "teleo_clone_current_test", + "source_compute": "teleo-prod-1", + "errors": [], + "posted_to_telegram": False, + "database_write_attempted": False, + }, + sort_keys=True, + ) + + "\n", + encoding="utf-8", + ) + return path + + +def compute_identity() -> dict: + return { + "project": "teleo-501523", + "project_number": "785938879453", + "instance": "teleo-prod-1", + "instance_id": "123456789", + "zone": "europe-west6-a", + "zone_resource": "projects/123456789/zones/europe-west6-a", + "network_resource": "projects/123456789/networks/teleo-staging-net", + "private_ip": "10.61.0.2", + "service_account": attestation.DEFAULT_RESTORE_SERVICE_ACCOUNT, + "expected_private_network": "projects/teleo-501523/global/networks/teleo-staging-net", + "checks": { + "project": True, + "project_number": True, + "expected_network_project": True, + "instance": True, + "instance_id": True, + "zone": True, + "network": True, + "private_ip": True, + "service_account": True, + }, + } + + +def test_attestation_binds_reasoning_hash_to_live_metadata(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + receipt_path = reasoning_receipt(tmp_path / "reasoning.json") + observed = {} + + def fake_identity(project, instance, zone, network, service_account, *, timeout): + observed.update( + { + "project": project, + "instance": instance, + "zone": zone, + "network": network, + "service_account": service_account, + "timeout": timeout, + } + ) + return compute_identity() + + monkeypatch.setattr(attestation, "gce_compute_identity", fake_identity) + + result = attestation.attest_reasoning_compute( + reasoning_receipt_path=receipt_path, + restore_run_id="gcp-restore-current-test123", + target_database="teleo_clone_current_test", + generated_at_utc="2026-07-15T01:01:05+00:00", + ) + + assert result["status"] == "pass" + assert result["checks"]["gce_metadata_identity"] is True + assert result["reasoning_receipt"]["sha256"] == attestation.sha256_file(receipt_path) + assert result["compute"]["instance_id"] == "123456789" + assert observed == { + "project": "teleo-501523", + "instance": "teleo-prod-1", + "zone": "europe-west6-a", + "network": "projects/teleo-501523/global/networks/teleo-staging-net", + "service_account": attestation.DEFAULT_RESTORE_SERVICE_ACCOUNT, + "timeout": 10.0, + } + + +def test_attestation_rejects_claimed_wrong_compute_before_metadata( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + receipt_path = reasoning_receipt(tmp_path / "reasoning.json") + payload = json.loads(receipt_path.read_text()) + payload["source_compute"] = "untrusted-vm" + receipt_path.write_text(json.dumps(payload) + "\n", encoding="utf-8") + monkeypatch.setattr( + attestation, + "gce_compute_identity", + lambda *args, **kwargs: pytest.fail("metadata must not be queried"), + ) + + with pytest.raises(ValueError, match="source_compute"): + attestation.attest_reasoning_compute( + reasoning_receipt_path=receipt_path, + restore_run_id="gcp-restore-current-test123", + target_database="teleo_clone_current_test", + ) + + +def test_attestation_rejects_send_or_write_receipt(tmp_path: Path) -> None: + receipt_path = reasoning_receipt(tmp_path / "reasoning.json") + payload = json.loads(receipt_path.read_text()) + payload["posted_to_telegram"] = True + payload["database_write_attempted"] = True + receipt_path.write_text(json.dumps(payload) + "\n", encoding="utf-8") + + with pytest.raises(ValueError, match=r"no_database_write.*no_telegram_send"): + attestation.attest_reasoning_compute( + reasoning_receipt_path=receipt_path, + restore_run_id="gcp-restore-current-test123", + target_database="teleo_clone_current_test", + ) diff --git a/tests/test_capture_vps_canonical_postgres_snapshot.py b/tests/test_capture_vps_canonical_postgres_snapshot.py index ee1f07e..fed31b8 100644 --- a/tests/test_capture_vps_canonical_postgres_snapshot.py +++ b/tests/test_capture_vps_canonical_postgres_snapshot.py @@ -1,8 +1,18 @@ +import json from pathlib import Path +import pytest + from ops.capture_vps_canonical_postgres_snapshot import ( + SAFE_AUTHORIZATION_REF_RE, SAFE_SSH_TARGET_RE, + SOURCE_SNAPSHOT_RECEIPT_SCHEMA, + build_provenance_binding, + canonical_sha256, cleanup_failed_output, + load_service_state, + load_snapshot_metadata, + parse_args, remote_capture_script, toc_counts, ) @@ -21,8 +31,14 @@ def test_remote_capture_uses_one_exported_read_only_snapshot() -> None: assert "[0-9A-F]+-[0-9A-F]+-[0-9]+" in script assert '--snapshot="$snapshot_id"' in script assert '-v "snapshot_id=$snapshot_id"' in script + assert "txid_current_snapshot()::text" in script + assert "pg_current_wal_lsn()::text" in script + assert "system_identifier::text from pg_control_system()" in script + assert "source-snapshot.json" in script assert "--no-owner --no-acl" in script assert "cmp -s" in script + assert 'assert_service_healthy "$remote_root/service-before.txt"' in script + assert 'assert_service_healthy "$remote_root/service-after.txt"' in script assert "systemctl restart" not in script assert "docker restart" not in script @@ -32,6 +48,11 @@ def test_manifest_hash_order_is_collation_independent() -> None: assert 'collate "C"' in manifest assert "'server_address', host(inet_server_addr())" in manifest + assert "'kb_gate_owner', 'kb_apply', 'kb_review'" in manifest + assert "'kind', 'role_memberships'" in manifest + assert "'kind', 'object_ownership'" in manifest + assert "'kind', 'object_acl'" in manifest + assert "aclexplode(attribute.attacl)" in manifest def test_toc_counts_database_objects(tmp_path: Path) -> None: @@ -62,8 +83,129 @@ def test_failure_cleanup_never_removes_preexisting_output(tmp_path: Path) -> Non assert not created_by_run.exists() +def test_source_service_state_requires_healthy_unchanged_capable_fields(tmp_path: Path) -> None: + state_path = tmp_path / "service.txt" + state_path.write_text("ActiveState=active\nSubState=running\nMainPID=347406\nNRestarts=0\n") + + assert load_service_state(state_path)["MainPID"] == "347406" + + state_path.write_text("ActiveState=failed\nSubState=dead\nMainPID=0\nNRestarts=0\n") + with pytest.raises(RuntimeError, match="not active/running"): + load_service_state(state_path) + + def test_ssh_target_rejects_open_ssh_option_injection() -> None: assert SAFE_SSH_TARGET_RE.fullmatch("root@77.42.65.182") assert SAFE_SSH_TARGET_RE.fullmatch("teleo-staging-1.internal") assert not SAFE_SSH_TARGET_RE.fullmatch("-oProxyCommand=malicious") assert not SAFE_SSH_TARGET_RE.fullmatch("root@host command") + + +def test_source_snapshot_metadata_and_provenance_binding(tmp_path: Path) -> None: + snapshot_path = tmp_path / "source-snapshot.json" + snapshot_path.write_text( + json.dumps( + { + "exported_snapshot_id": "00000003-000001A2-1", + "txid_snapshot": "100:120:105,111", + "wal_lsn": "0/16B6C50", + "system_identifier": "7649789040005668902", + "captured_at_utc": "2026-07-15T01:02:03.456789+00:00", + } + ) + ) + + snapshot = load_snapshot_metadata(snapshot_path) + binding = build_provenance_binding( + run_id="canonical-20260715", + authorization_ref="codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621", + source={ + "ssh_target": "root@77.42.65.182", + "container": "teleo-pg", + "database": "teleo", + "service": "leoclean-gateway.service", + }, + snapshot=snapshot, + dump_sha256="1" * 64, + manifest_sql_sha256="2" * 64, + source_manifest_sha256="3" * 64, + source_context_sha256="4" * 64, + ) + + assert binding["algorithm"] == "sha256" + assert binding["payload"] == { + "receipt_schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA, + "run_id": "canonical-20260715", + "capture_authorization_ref": "codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621", + "source": { + "ssh_target": "root@77.42.65.182", + "container": "teleo-pg", + "database": "teleo", + "service": "leoclean-gateway.service", + }, + "exported_snapshot_id": "00000003-000001A2-1", + "txid_snapshot": "100:120:105,111", + "wal_lsn": "0/16B6C50", + "source_system_identifier": "7649789040005668902", + "dump_sha256": "1" * 64, + "manifest_sql_sha256": "2" * 64, + "source_manifest_sha256": "3" * 64, + "source_context_sha256": "4" * 64, + } + assert binding["sha256"] == canonical_sha256(binding["payload"]) + + +@pytest.mark.parametrize( + ("field", "value"), + [ + ("exported_snapshot_id", "unsafe snapshot"), + ("txid_snapshot", "100:120:not-a-txid"), + ("wal_lsn", "not-an-lsn"), + ("system_identifier", "system-1"), + ], +) +def test_source_snapshot_metadata_rejects_invalid_identity(tmp_path: Path, field: str, value: str) -> None: + payload = { + "exported_snapshot_id": "00000003-000001A2-1", + "txid_snapshot": "100:120:", + "wal_lsn": "0/16B6C50", + "system_identifier": "7649789040005668902", + "captured_at_utc": "2026-07-15T01:02:03+00:00", + } + payload[field] = value + path = tmp_path / "source-snapshot.json" + path.write_text(json.dumps(payload)) + + with pytest.raises(RuntimeError, match=field): + load_snapshot_metadata(path) + + +def test_authorization_ref_is_required_and_safe_metadata(tmp_path: Path) -> None: + ssh_key = tmp_path / "id_ed25519" + ssh_key.write_text("test-only") + manifest = tmp_path / "manifest.sql" + manifest.write_text("select 1;\n") + base = [ + "--execute", + "--ssh-target", + "root@77.42.65.182", + "--ssh-key", + str(ssh_key), + "--manifest", + str(manifest), + "--run-id", + "canonical-20260715", + "--output-dir", + str(tmp_path / "capture"), + ] + + with pytest.raises(SystemExit): + parse_args(base) + with pytest.raises(SystemExit): + parse_args([*base, "--authorization-ref", "unsafe authorization ref"]) + + authorization_ref = "codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621" + args = parse_args([*base, "--authorization-ref", authorization_ref]) + assert args.authorization_ref == authorization_ref + assert SAFE_AUTHORIZATION_REF_RE.fullmatch(authorization_ref) + assert not SAFE_AUTHORIZATION_REF_RE.fullmatch("authorization\nforged") diff --git a/tests/test_gcp_generated_db_direct_claim_suite.py b/tests/test_gcp_generated_db_direct_claim_suite.py index 08c72d3..ca18c64 100644 --- a/tests/test_gcp_generated_db_direct_claim_suite.py +++ b/tests/test_gcp_generated_db_direct_claim_suite.py @@ -166,7 +166,7 @@ def test_manifest_normalization_excludes_nondeterministic_timing() -> None: "BEGIN", '{"kind":"identity","database":"teleo_clone_test","captured_at":"2026-07-12T00:00:00Z"}', '{"kind":"schemas","items":["kb_stage","public"]}', - '{"kind":"table","schema":"public","table":"claims","row_count":2,"rowset_md5":"abc"}', + '{"kind":"table","schema":"public","table":"claims","row_count":2,"rowset_md5":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}', '{"kind":"performance","query":"count_claims","elapsed_ms":1.2,"observed_rows":2}', "ROLLBACK", ] @@ -196,7 +196,7 @@ def test_database_fingerprint_uses_full_normalized_manifest(monkeypatch: pytest. '{"kind":"identity","database":"teleo_clone_test","server_address":"10.61.0.3",' '"ssl":true,"transaction_read_only":"on","captured_at":"one"}', '{"kind":"schemas","items":["kb_stage","public"]}', - '{"kind":"table","schema":"public","table":"claims","row_count":2,"rowset_md5":"abc"}', + '{"kind":"table","schema":"public","table":"claims","row_count":2,"rowset_md5":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}', ] ) monkeypatch.setattr("scripts.run_gcp_generated_db_direct_claim_suite.database_password", lambda _args: "secret") diff --git a/tests/test_gcp_generated_db_working_leo_suite.py b/tests/test_gcp_generated_db_working_leo_suite.py index 4ac006c..f4543ac 100644 --- a/tests/test_gcp_generated_db_working_leo_suite.py +++ b/tests/test_gcp_generated_db_working_leo_suite.py @@ -95,9 +95,7 @@ def source_baseline_payload() -> dict[str, object]: } -def capability_receipt( - role: str, target_db: str = "teleo_clone_working_leo", *, writable: bool | None = None -) -> str: +def capability_receipt(role: str, target_db: str = "teleo_clone_working_leo", *, writable: bool | None = None) -> str: capabilities = { "kb_read": (False, False, False, False, False), "kb_stage_writer": (True, False, False, False, False), @@ -337,14 +335,18 @@ def test_source_snapshots_are_receipt_backed_and_inherited_source_checks_are_not } adapter = suite.RuntimeAdapter(args, suite.CloudSqlExecutor(args), source, {"system_identifier": "target"}) - assert adapter._guard_snapshot( - suite.bound.PRODUCTION_CONTAINER, - suite.bound.PRODUCTION_DB, - tables=suite.composition.PRODUCTION_TABLES, - ) == source_payload["guard_snapshot"] - assert adapter._gate_schema(suite.bound.PRODUCTION_CONTAINER, suite.bound.PRODUCTION_DB) == source_payload[ - "gate_schema" - ] + assert ( + adapter._guard_snapshot( + suite.bound.PRODUCTION_CONTAINER, + suite.bound.PRODUCTION_DB, + tables=suite.composition.PRODUCTION_TABLES, + ) + == source_payload["guard_snapshot"] + ) + assert ( + adapter._gate_schema(suite.bound.PRODUCTION_CONTAINER, suite.bound.PRODUCTION_DB) + == source_payload["gate_schema"] + ) contract = suite.composition_check_contract() required = [name for name in contract if name not in suite.UNSUPPORTED_INHERITED_COMPOSITION_CHECKS] assert len(contract) == 34 @@ -524,7 +526,7 @@ def test_lifecycle_wrapper_exposes_only_read_or_exact_stage_and_never_delivery(t assert "accepts no model-supplied arguments" in wrapper assert "--stage-password-secret" in wrapper assert "--read-role" in wrapper - assert 'user=$READ_ROLE' in wrapper + assert "user=$READ_ROLE" in wrapper assert "default_transaction_read_only=on" in wrapper assert "systemctl restart" not in wrapper assert "send_message" not in wrapper diff --git a/tests/test_gcp_leoclean_runtime_permissions.py b/tests/test_gcp_leoclean_runtime_permissions.py index 8f594ff..dee0eed 100644 --- a/tests/test_gcp_leoclean_runtime_permissions.py +++ b/tests/test_gcp_leoclean_runtime_permissions.py @@ -167,9 +167,7 @@ class FakeRunner: stage_definition.update(self.stage_definition_override) return verifier.CommandResult(0, json.dumps(stage_definition), f"ignored {SECRET_VALUE}") if sql == verifier._catalog_privilege_posture_sql(): - catalog_posture: dict[str, object] = { - field: 0 for field in verifier.CATALOG_ZERO_COUNT_FIELDS - } + catalog_posture: dict[str, object] = {field: 0 for field in verifier.CATALOG_ZERO_COUNT_FIELDS} catalog_posture.update( { "leak": SECRET_VALUE, @@ -476,10 +474,7 @@ def _drifted_value(expected: object) -> object: @pytest.mark.parametrize( ("field", "unsafe_value"), - [ - (field, _drifted_value(expected)) - for field, expected in verifier.STAGE_FUNCTION_METADATA_EXPECTATIONS - ], + [(field, _drifted_value(expected)) for field, expected in verifier.STAGE_FUNCTION_METADATA_EXPECTATIONS], ) def test_any_stage_function_metadata_drift_fails_closed(field: str, unsafe_value: object) -> None: runner = FakeRunner(stage_definition_override={field: unsafe_value}) @@ -550,9 +545,9 @@ def test_stage_body_normalization_accepts_only_line_ending_and_framing_newline_c def test_embedded_stage_body_attestation_is_tied_to_the_provisioning_sql_source() -> None: - provisioning_sql = ( - Path(__file__).resolve().parents[1] / "ops" / "gcp_leoclean_runtime_role.sql" - ).read_text(encoding="utf-8") + provisioning_sql = (Path(__file__).resolve().parents[1] / "ops" / "gcp_leoclean_runtime_role.sql").read_text( + encoding="utf-8" + ) function_anchor = "create or replace function kb_stage.stage_leoclean_proposal(" assert provisioning_sql.count(function_anchor) == 1 function_section = provisioning_sql.split(function_anchor, 1)[1] @@ -644,9 +639,7 @@ def test_legacy_and_template_database_connect_are_both_behaviorally_denied() -> "result": "denied", } database_calls = [ - " ".join(command) - for command, _env, _timeout, _discard in runner.calls - if command[0] == verifier.PSQL_BIN + " ".join(command) for command, _env, _timeout, _discard in runner.calls if command[0] == verifier.PSQL_BIN ] assert any(f"dbname={verifier.LEGACY_DATABASE}" in command for command in database_calls) assert any(f"dbname={verifier.TEMPLATE_DATABASE}" in command for command in database_calls) diff --git a/tests/test_hermes_leoclean_kb_bridge_source.py b/tests/test_hermes_leoclean_kb_bridge_source.py index 6022d2a..118cd7a 100644 --- a/tests/test_hermes_leoclean_kb_bridge_source.py +++ b/tests/test_hermes_leoclean_kb_bridge_source.py @@ -324,7 +324,8 @@ def _configure_runtime_environment(monkeypatch: pytest.MonkeyPatch) -> None: } or normalized_key == "PROXY" or normalized_key.endswith("_PROXY") - or normalized_key in { + or normalized_key + in { "CURL_CA_BUNDLE", "PYTHONHTTPSVERIFY", "REQUESTS_CA_BUNDLE", @@ -679,7 +680,9 @@ def test_cloudsql_runtime_metadata_and_secret_requests_are_exact_and_silent( def test_cloudsql_runtime_http_opener_disables_proxies_and_redirects() -> None: module = _load_module(BRIDGE_DIR / "cloudsql_memory_tool.py") proxy_handlers = [ - handler for handler in module.RUNTIME_HTTP_OPENER.handlers if isinstance(handler, module.urllib.request.ProxyHandler) + handler + for handler in module.RUNTIME_HTTP_OPENER.handlers + if isinstance(handler, module.urllib.request.ProxyHandler) ] redirect_handlers = [ handler diff --git a/tests/test_observatory_read_adapter_gcp_plan.py b/tests/test_observatory_read_adapter_gcp_plan.py new file mode 100644 index 0000000..d5fda32 --- /dev/null +++ b/tests/test_observatory_read_adapter_gcp_plan.py @@ -0,0 +1,383 @@ +from __future__ import annotations + +import json +import shutil +import subprocess +from pathlib import Path + +import pytest +import yaml + +from ops import check_observatory_read_adapter_gcp_preflight as preflight + +ROOT = Path(__file__).resolve().parents[1] + + +def run_plan(*args: str) -> str: + return subprocess.check_output( + ["python3", "ops/plan_observatory_read_adapter_gcp.py", *args], + cwd=ROOT, + text=True, + ) + + +def test_plan_splits_build_deploy_and_runtime_identities() -> None: + plan = json.loads(run_plan()) + + assert plan["current_tier"] == "T2_unexecuted_least_privilege_packet" + assert plan["identities"] == { + "build": "sa-artifact-builder@teleo-501523.iam.gserviceaccount.com", + "deploy_and_canary": "sa-observatory-deployer@teleo-501523.iam.gserviceaccount.com", + "runtime": "sa-observatory-read-adapter@teleo-501523.iam.gserviceaccount.com", + "database_principal": "sa-observatory-read-adapter@teleo-501523.iam", + } + commands = "\n".join(plan["commands"]) + assert "roles/iam.workloadIdentityUser" in commands + assert "roles/iam.serviceAccountUser" in commands + assert "roles/cloudsql.client" in commands + assert "roles/cloudsql.instanceUser" in commands + assert "roles/artifactregistry.reader" in commands + assert "roles/secretmanager.secretAccessor" in commands + assert "subject/repo:living-ip/teleo-infrastructure:ref:refs/heads/main" in plan["github_wif_member"] + assert plan["conditions"]["deployer_wif"].endswith( + "assertion.workflow_ref=='living-ip/teleo-infrastructure/.github/workflows/" + "gcp-observatory-read-adapter.yml@refs/heads/main'" + ) + assert plan["github_wif_provider"].endswith("/providers/observatory-read-adapter-main") + assert "roles/run.serviceAgent" in commands + assert "teleo-pgvector-standby" in plan["conditions"]["cloud_sql"] + assert set(plan["deploy_custom_role"]["permissions"]) == { + "run.operations.get", + "run.services.create", + "run.services.get", + "run.services.setIamPolicy", + "run.services.update", + } + assert "roles/run.admin" not in commands + assert "roles/run.developer" not in commands + assert "sa-artifact-builder" not in commands + assert not set(plan["forbidden_deployer_roles"]) & set(commands.split()) + assert plan["production_repoint_executed"] is False + + +def test_shell_packet_is_unexecuted_and_preserves_private_boundaries() -> None: + shell = run_plan("--format", "shell") + + assert shell.startswith("# Unexecuted staging packet") + assert "set -euo pipefail" in shell + assert "run.googleapis.com" in shell + assert "--type CLOUD_IAM_SERVICE_ACCOUNT" in shell + assert "openssl rand -hex 32 | tr -d '\\n'" in shell + assert "--data-file=-" in shell + assert "grep -Fxq sa-observatory-read-adapter@teleo-501523.iam" in shell + assert "users create sa-observatory-read-adapter@teleo-501523.iam" in shell + assert "add-iam-policy-binding teleo" in shell + assert "roles/artifactregistry.writer" not in shell + assert "roles/cloudsql.admin" not in shell + assert "Vercel" not in shell + + +def test_workflow_requires_dedicated_preflight_before_staging_deploy() -> None: + workflow_path = ROOT / ".github" / "workflows" / "gcp-observatory-read-adapter.yml" + workflow = yaml.safe_load(workflow_path.read_text(encoding="utf-8")) + text = workflow_path.read_text(encoding="utf-8") + + assert workflow["env"]["BUILD_SERVICE_ACCOUNT"].startswith("sa-artifact-builder@") + assert workflow["env"]["DEPLOY_SERVICE_ACCOUNT"].startswith("sa-observatory-deployer@") + assert workflow["env"]["DEPLOY_WORKLOAD_IDENTITY_PROVIDER"].endswith( + "/providers/observatory-read-adapter-main" + ) + assert workflow["jobs"]["deploy-staging"]["needs"] == ["build", "preflight-staging"] + assert "action=preflight_staging" not in text + assert "inputs.action != 'build_only'" in text + assert "EXPECTED_WORKFLOW_REF" in text + assert "${GITHUB_SHA}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" in text + assert "docker images describe" in text + assert "2>/dev/null || true" not in text + assert "--no-invoker-iam-check" in text + assert "--allow-unauthenticated" not in text + assert "API_KEY_VERSION" in text + assert "EXPECTED_IMAGE_REF" in text + assert "check_observatory_read_adapter_gcp_preflight.py" in text + assert 'service_account: ${{ env.BUILD_SERVICE_ACCOUNT }}' in text + assert text.count('service_account: ${{ env.DEPLOY_SERVICE_ACCOUNT }}') == 2 + assert "positive-redacted.json" in text + assert "redact_observatory_read_adapter_receipt.jq" in text + assert "database_principal == $db_iam_user" in text + assert "canonical_state_unchanged_after_post: true" in text + assert "cmp <(jq -S . positive.json) <(jq -S . after-write-attempt.json)" in text + assert "rm -f positive.json" in text + + +def test_preflight_private_cloud_sql_and_secret_validators() -> None: + instance = { + "name": "teleo-pgvector-standby", + "region": "europe-west6", + "databaseVersion": "POSTGRES_16", + "settings": { + "databaseFlags": [{"name": "cloudsql.iam_authentication", "value": "on"}], + "ipConfiguration": { + "ipv4Enabled": False, + "privateNetwork": "projects/teleo-501523/global/networks/teleo-staging-net", + }, + }, + "ipAddresses": [{"type": "PRIVATE", "ipAddress": "10.61.0.3"}], + } + assert preflight.cloud_sql_is_private_and_iam_enabled(json.dumps(instance)) is True + + instance["settings"]["ipConfiguration"]["ipv4Enabled"] = True + instance["ipAddresses"].append({"type": "PRIMARY", "ipAddress": "203.0.113.1"}) + assert preflight.cloud_sql_is_private_and_iam_enabled(json.dumps(instance)) is False + assert preflight.cloud_sql_iam_user_exists( + json.dumps( + [ + { + "name": "sa-observatory-read-adapter@teleo-501523.iam", + "type": "CLOUD_IAM_SERVICE_ACCOUNT", + } + ] + ) + ) + assert preflight.subnet_is_exact( + json.dumps( + { + "name": "teleo-staging-europe-west6", + "region": "https://www.googleapis.com/compute/v1/projects/teleo-501523/regions/europe-west6", + "network": "https://www.googleapis.com/compute/v1/projects/teleo-501523/global/networks/teleo-staging-net", + } + ) + ) + assert preflight.secret_is_valid("a" * 32) is True + assert preflight.secret_is_valid("too-short") is False + assert preflight.secret_is_valid("a" * 32 + "\n") is False + assert preflight.secret_is_valid(" " + "a" * 32) is False + assert "gha-creds" not in preflight.sanitize_error("/tmp/work/gha-creds-deadbeef.json denied") + + +def test_preflight_end_to_end_snapshot_retains_no_secret(monkeypatch) -> None: + instance = { + "name": "teleo-pgvector-standby", + "region": "europe-west6", + "databaseVersion": "POSTGRES_16", + "settings": { + "databaseFlags": [{"name": "cloudsql.iam_authentication", "value": "on"}], + "ipConfiguration": { + "ipv4Enabled": False, + "privateNetwork": "projects/teleo-501523/global/networks/teleo-staging-net", + }, + }, + "ipAddresses": [{"type": "PRIVATE", "ipAddress": "10.61.0.3"}], + } + secret_value = "never-retain-this-secret-value-000000000000" + deployer = "serviceAccount:sa-observatory-deployer@teleo-501523.iam.gserviceaccount.com" + runtime = "serviceAccount:sa-observatory-read-adapter@teleo-501523.iam.gserviceaccount.com" + project_policy = { + "bindings": [ + {"role": preflight.PREFLIGHT_ROLE, "members": [deployer]}, + {"role": preflight.DEPLOY_ROLE, "members": [deployer]}, + { + "role": "roles/cloudsql.client", + "members": [runtime], + "condition": { + "title": "observatory-exact-cloudsql-instance", + "expression": ( + "resource.name == 'projects/teleo-501523/instances/teleo-pgvector-standby' " + "&& resource.service == 'sqladmin.googleapis.com'" + ), + }, + }, + { + "role": "roles/cloudsql.instanceUser", + "members": [runtime], + "condition": { + "title": "observatory-exact-cloudsql-instance", + "expression": ( + "resource.name == 'projects/teleo-501523/instances/teleo-pgvector-standby' " + "&& resource.service == 'sqladmin.googleapis.com'" + ), + }, + }, + { + "role": "roles/run.serviceAgent", + "members": [ + "serviceAccount:service-785938879453@serverless-robot-prod.iam.gserviceaccount.com" + ], + }, + ] + } + deployer_policy = { + "bindings": [ + {"role": "roles/iam.workloadIdentityUser", "members": [preflight.DEPLOY_WIF_MEMBER]} + ] + } + runtime_policy = { + "bindings": [{"role": "roles/iam.serviceAccountUser", "members": [deployer]}] + } + artifact_policy = { + "bindings": [{"role": "roles/artifactregistry.reader", "members": [deployer]}] + } + secret_policy = { + "bindings": [ + {"role": "roles/secretmanager.secretAccessor", "members": [deployer, runtime]}, + {"role": "roles/secretmanager.viewer", "members": [deployer]}, + ] + } + provider = { + "state": "ACTIVE", + "attributeCondition": preflight.DEPLOY_WIF_ATTRIBUTE_CONDITION, + "attributeMapping": { + "google.subject": "assertion.sub", + "attribute.repository": "assertion.repository", + "attribute.ref": "assertion.ref", + "attribute.workflow_ref": "assertion.workflow_ref", + }, + } + + def fake_run(command: list[str]) -> subprocess.CompletedProcess[str]: + joined = " ".join(command) + stdout = "" + stderr = "" + returncode = 0 + if "auth list" in joined: + stdout = "sa-observatory-deployer@teleo-501523.iam.gserviceaccount.com\n" + elif "projects describe" in joined: + stdout = "teleo-501523\n" + elif command[:3] == ["gcloud", "services", "describe"]: + stdout = "ENABLED\n" + elif "workload-identity-pools providers describe" in joined: + stdout = json.dumps(provider) + elif "iam roles describe observatoryStagingPreflight" in joined: + stdout = json.dumps({"includedPermissions": sorted(preflight.PREFLIGHT_PERMISSIONS)}) + elif "iam roles describe observatoryStagingDeployer" in joined: + stdout = json.dumps({"includedPermissions": sorted(preflight.DEPLOY_PERMISSIONS)}) + elif "projects get-iam-policy" in joined: + stdout = json.dumps(project_policy) + elif f"service-accounts get-iam-policy {preflight.DEPLOY_SERVICE_ACCOUNT}" in joined: + stdout = json.dumps(deployer_policy) + elif f"service-accounts get-iam-policy {preflight.RUNTIME_SERVICE_ACCOUNT}" in joined: + stdout = json.dumps(runtime_policy) + elif "artifacts repositories get-iam-policy" in joined: + stdout = json.dumps(artifact_policy) + elif "secrets get-iam-policy" in joined: + stdout = json.dumps(secret_policy) + elif "run services describe" in joined: + returncode = 1 + stderr = "NOT_FOUND: service not found" + elif "service-accounts describe" in joined: + stdout = "sa-observatory-read-adapter@teleo-501523.iam.gserviceaccount.com\n" + elif "networks subnets describe" in joined: + stdout = json.dumps( + { + "name": "teleo-staging-europe-west6", + "region": "https://www.googleapis.com/compute/v1/projects/teleo-501523/regions/europe-west6", + "network": "https://www.googleapis.com/compute/v1/projects/teleo-501523/global/networks/teleo-staging-net", + } + ) + elif "networks describe" in joined: + stdout = "teleo-staging-net\n" + elif "sql instances describe" in joined: + stdout = json.dumps(instance) + elif "sql users list" in joined: + stdout = json.dumps( + [ + { + "name": "sa-observatory-read-adapter@teleo-501523.iam", + "type": "CLOUD_IAM_SERVICE_ACCOUNT", + } + ] + ) + elif "secrets versions list" in joined: + stdout = "7\n" + elif "secrets versions access" in joined: + stdout = secret_value + elif "artifacts docker images describe" in joined: + stdout = "{}" + return subprocess.CompletedProcess(command, returncode, stdout=stdout, stderr=stderr) + + monkeypatch.setattr(preflight, "run", fake_run) + image_ref = ( + "europe-west6-docker.pkg.dev/teleo-501523/teleo/observatory-read-adapter:" + "33399bd9acfa09e4e5409873e86c8c46ac694a8d-29389090997-1@sha256:" + "a" * 64 + ) + checks = preflight.build_checks(image_ref) + + assert checks + assert {check.status for check in checks} == {"pass"} + assert secret_value not in json.dumps([check.__dict__ for check in checks]) + + +def test_malformed_successful_readback_becomes_blocked_check(monkeypatch) -> None: + monkeypatch.setattr( + preflight, + "run", + lambda command: subprocess.CompletedProcess(command, 0, stdout="[]", stderr=""), + ) + + check = preflight.command_check( + "malformed", + ["gcloud", "sql", "instances", "describe"], + preflight.cloud_sql_is_private_and_iam_enabled, + "unexpected", + ) + + assert check.status == "blocked" + assert check.detail == "invalid_readback:AttributeError" + + +@pytest.mark.skipif(shutil.which("jq") is None, reason="jq is required") +def test_receipt_redaction_is_an_explicit_allowlist() -> None: + payload = { + "schema": "livingip.observatory-canonical-claim.v1", + "read_only": True, + "provenance": { + "gcp_project": "teleo-501523", + "cloud_sql_instance": "teleo-501523:europe-west6:teleo-pgvector-standby", + "database": "teleo_canonical", + "database_principal": "sa-observatory-read-adapter@teleo-501523.iam", + "authorization_role": "kb_observatory_read", + "transaction_read_only": True, + "write_privileges_denied": True, + "service_revision": "a" * 40, + "future_sensitive_field": "must-not-survive", + }, + "canonical": { + "claim": {"id": "claim-1", "type": "fact", "status": "live", "text": "secret claim text"}, + "evidence": [{"excerpt": "secret excerpt", "source_url": "https://private.invalid"}], + "edges": [{"direction": "incoming"}, {"direction": "outgoing"}], + }, + "proposal_ledger": { + "distinct_from_canonical": True, + "status_counts": {"approved": 1}, + "related": [{"proposal_text": "secret proposal"}], + }, + } + result = subprocess.run( + ["jq", "-f", str(ROOT / "ops" / "redact_observatory_read_adapter_receipt.jq")], + input=json.dumps(payload), + text=True, + capture_output=True, + check=True, + ) + redacted = json.loads(result.stdout) + rendered = json.dumps(redacted) + + assert set(redacted["provenance"]) == { + "gcp_project", + "cloud_sql_instance", + "database", + "database_principal", + "authorization_role", + "transaction_read_only", + "write_privileges_denied", + "service_revision", + } + assert redacted["canonical"]["evidence_count"] == 1 + assert redacted["proposal_ledger"]["related_count"] == 1 + for sensitive in ( + "must-not-survive", + "secret claim text", + "secret excerpt", + "https://private.invalid", + "secret proposal", + ): + assert sensitive not in rendered diff --git a/tests/test_observatory_read_role_postgres.py b/tests/test_observatory_read_role_postgres.py index be2b155..c062b1d 100644 --- a/tests/test_observatory_read_role_postgres.py +++ b/tests/test_observatory_read_role_postgres.py @@ -44,22 +44,22 @@ def run(command: list[str], *, input_text: str | None = None, check: bool = True @pytest.mark.skipif(shutil.which("docker") is None, reason="Docker is required") def test_observatory_role_is_read_only_and_exactly_allowlisted() -> None: container = f"teleo-observatory-role-{uuid.uuid4().hex[:10]}" - run( - [ - "docker", - "run", - "--detach", - "--rm", - "--name", - container, - "--env", - f"POSTGRES_PASSWORD={TEST_PASSWORD}", - "--env", - f"POSTGRES_DB={DATABASE}", - POSTGRES_IMAGE, - ] - ) try: + run( + [ + "docker", + "run", + "--detach", + "--rm", + "--name", + container, + "--env", + f"POSTGRES_PASSWORD={TEST_PASSWORD}", + "--env", + f"POSTGRES_DB={DATABASE}", + POSTGRES_IMAGE, + ] + ) deadline = time.monotonic() + 25 while time.monotonic() < deadline: ready = run( diff --git a/tests/test_private_receipt_io.py b/tests/test_private_receipt_io.py new file mode 100644 index 0000000..4df55cb --- /dev/null +++ b/tests/test_private_receipt_io.py @@ -0,0 +1,29 @@ +from __future__ import annotations + +import json +from pathlib import Path + +from ops.private_receipt_io import print_private_receipt_summary, write_private_json + + +def test_stdout_summary_never_replays_private_receipt_payload(tmp_path: Path, capsys) -> None: + output = tmp_path / "private" / "receipt.json" + payload = { + "artifact": "private_test_receipt", + "status": "pass", + "private_claim_body": "must-not-appear-on-stdout", + "database_password": "must-not-appear-on-stdout-either", + } + + write_private_json(output, payload) + print_private_receipt_summary(output, payload) + + stdout = capsys.readouterr().out + summary = json.loads(stdout) + assert "must-not-appear" not in stdout + assert summary["artifact"] == "private_test_receipt" + assert summary["status"] == "pass" + assert summary["output_name"] == output.name + assert str(output.parent) not in stdout + assert len(summary["output_sha256"]) == 64 + assert output.stat().st_mode & 0o777 == 0o600 diff --git a/tests/test_restore_gcp_generated_postgres_snapshot.py b/tests/test_restore_gcp_generated_postgres_snapshot.py index 77f3a1c..baf6893 100644 --- a/tests/test_restore_gcp_generated_postgres_snapshot.py +++ b/tests/test_restore_gcp_generated_postgres_snapshot.py @@ -7,6 +7,7 @@ from pathlib import Path import pytest from ops import restore_gcp_generated_postgres_snapshot as restore +from ops.capture_vps_canonical_postgres_snapshot import build_provenance_binding SERVICE_STATE = { "ActiveState": "active", @@ -15,6 +16,7 @@ SERVICE_STATE = { "NRestarts": "0", "ExecMainStartTimestamp": "Thu 2026-07-09 07:00:04 UTC", } +AUTHORIZATION_REF = "codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621" def manifest_rows(database: str, *, target: bool, row_count: int = 1) -> list[dict]: @@ -47,7 +49,27 @@ def manifest_rows(database: str, *, target: bool, row_count: int = 1) -> list[di "inherit": True, "replication": False, "bypass_rls": False, - } + }, + { + "name": "kb_gate_owner", + "can_login": False, + "superuser": False, + "create_db": False, + "create_role": False, + "inherit": True, + "replication": False, + "bypass_rls": False, + }, + { + "name": "kb_review", + "can_login": True, + "superuser": False, + "create_db": False, + "create_role": False, + "inherit": True, + "replication": False, + "bypass_rls": False, + }, ], }, ] @@ -63,6 +85,9 @@ def manifest_rows(database: str, *, target: bool, row_count: int = 1) -> list[di "triggers", "types", "policies", + "role_memberships", + "object_ownership", + "object_acl", ) ) rows.extend( @@ -72,7 +97,7 @@ def manifest_rows(database: str, *, target: bool, row_count: int = 1) -> list[di "schema": "public", "table": "claims", "row_count": row_count, - "rowset_md5": "same" if row_count == 1 else "different", + "rowset_md5": ("a" if row_count == 1 else "b") * 32, }, { "kind": "performance", @@ -94,6 +119,67 @@ def args_for(tmp_path: Path, **overrides) -> argparse.Namespace: dump.write_bytes(b"PGDMPfixture") source_manifest = tmp_path / "source-manifest.jsonl" source_manifest.write_text(manifest_text("teleo", target=False), encoding="utf-8") + manifest_sql = Path("ops/postgres_parity_manifest.sql").resolve() + ssl_root_cert = tmp_path / "cloudsql-server-ca.pem" + ssl_root_cert.write_text("test-only-ca\n", encoding="utf-8") + source_context_sha256 = "4" * 64 + snapshot = { + "exported_snapshot_id": "00000003-000001A2-1", + "txid_snapshot": "100:120:105,111", + "wal_lsn": "0/16B6C50", + "system_identifier": "7649789040005668902", + "captured_at_utc": "2026-07-15T01:02:03.456789+00:00", + } + source = { + "ssh_target": "root@77.42.65.182", + "container": "teleo-pg", + "database": "teleo", + "service": "leoclean-gateway.service", + } + dump_sha256 = hashlib.sha256(dump.read_bytes()).hexdigest() + source_manifest_sha256 = hashlib.sha256(source_manifest.read_bytes()).hexdigest() + manifest_sql_sha256 = hashlib.sha256(manifest_sql.read_bytes()).hexdigest() + source_receipt = tmp_path / "source-receipt.json" + source_receipt.write_text( + json.dumps( + { + "artifact": "vps_canonical_postgres_exported_snapshot", + "schema": restore.SOURCE_SNAPSHOT_RECEIPT_SCHEMA, + "receipt_version": 2, + "generated_at_utc": "2026-07-15T01:02:04+00:00", + "status": "pass", + "run_id": "canonical-20260715", + "capture_authorization_ref": AUTHORIZATION_REF, + "source": source, + "snapshot_exported": True, + "snapshot": snapshot, + "dump": {"path": "/private/source.dump", "bytes": dump.stat().st_size, "sha256": dump_sha256}, + "manifest": { + "path": "/private/source-manifest.jsonl", + "sha256": source_manifest_sha256, + "manifest_sql_sha256": manifest_sql_sha256, + "table_count": 1, + "total_rows": 1, + }, + "source_context": {"path": "/private/source-context.txt", "sha256": source_context_sha256}, + "source_service": {"before": SERVICE_STATE, "after": SERVICE_STATE, "unchanged": True}, + "provenance_binding": build_provenance_binding( + run_id="canonical-20260715", + authorization_ref=AUTHORIZATION_REF, + source=source, + snapshot=snapshot, + dump_sha256=dump_sha256, + manifest_sql_sha256=manifest_sql_sha256, + source_manifest_sha256=source_manifest_sha256, + source_context_sha256=source_context_sha256, + ), + "service_unchanged": True, + "production_db_mutated": False, + "telegram_send_attempted": False, + } + ), + encoding="utf-8", + ) values = { "operation": "restore", "execute": True, @@ -101,15 +187,29 @@ def args_for(tmp_path: Path, **overrides) -> argparse.Namespace: "run_id": "gcp-restore-test12345", "host": restore.DEFAULT_HOST, "project": restore.DEFAULT_PROJECT, + "cloudsql_instance": restore.DEFAULT_INSTANCE, + "expected_private_network": restore.DEFAULT_PRIVATE_NETWORK, + "expected_compute_instance": restore.DEFAULT_COMPUTE_INSTANCE, + "expected_compute_zone": restore.DEFAULT_COMPUTE_ZONE, + "expected_restore_service_account": restore.DEFAULT_RESTORE_SERVICE_ACCOUNT, + "ssl_server_name": "teleo-pgvector-standby.private.teleo.internal", + "ssl_root_cert": ssl_root_cert, + "expected_source_ssh_target": restore.DEFAULT_SOURCE_SSH_TARGET, + "expected_source_container": restore.DEFAULT_SOURCE_CONTAINER, + "expected_source_database": restore.DEFAULT_SOURCE_DATABASE, + "expected_source_service": restore.DEFAULT_SOURCE_SERVICE, "password_secret": restore.DEFAULT_SECRET, "service": restore.DEFAULT_SERVICE, "command_timeout": 2.0, "psql_bin": "psql", "run_root": tmp_path / "runs", "dump": dump, - "expected_dump_sha256": hashlib.sha256(dump.read_bytes()).hexdigest(), + "expected_dump_sha256": dump_sha256, "source_manifest": source_manifest, - "manifest_sql": Path("ops/postgres_parity_manifest.sql").resolve(), + "source_receipt": source_receipt, + "expected_source_receipt_sha256": hashlib.sha256(source_receipt.read_bytes()).hexdigest(), + "expected_capture_authorization_ref": AUTHORIZATION_REF, + "manifest_sql": manifest_sql, "pg_restore_bin": "pg_restore", "restore_timeout": 5.0, "manifest_timeout": 5.0, @@ -128,6 +228,54 @@ def install_restore_fakes( state = {"exists": False, "dropped": False} monkeypatch.setattr(restore.os, "geteuid", lambda: 0) monkeypatch.setattr(restore, "service_state", lambda *_args, **_kwargs: dict(SERVICE_STATE)) + monkeypatch.setattr( + restore, + "gce_compute_identity", + lambda project, instance, zone, expected_private_network, expected_service_account, **_kwargs: { + "project": project, + "project_number": "785938879453", + "instance": instance, + "instance_id": "123456789", + "zone": zone, + "expected_private_network": expected_private_network, + "private_ip": "10.61.0.2", + "network_resource": "projects/123456789/networks/teleo-staging-net", + "service_account": expected_service_account, + "checks": { + "project": True, + "project_number": True, + "expected_network_project": True, + "instance": True, + "instance_id": True, + "zone": True, + "network": True, + "private_ip": True, + "service_account": True, + }, + }, + ) + monkeypatch.setattr( + restore, + "cloudsql_control_plane_state", + lambda project, instance, host, expected_private_network, **_kwargs: { + "project": project, + "instance": instance, + "private_addresses": [host], + "expected_private_host": host, + "private_network": expected_private_network, + "expected_private_network": expected_private_network, + "public_ip_disabled": True, + "checks": { + "instance": True, + "state": True, + "postgres_16": True, + "public_ip_disabled": True, + "private_network": True, + "expected_private_host": True, + "no_non_private_address": True, + }, + }, + ) monkeypatch.setattr(restore, "resolve_password", lambda *_args, **_kwargs: "private-password") monkeypatch.setattr( restore, @@ -179,7 +327,10 @@ def test_restore_success_retains_only_exact_parity_clone( assert result["phase"] == "retained_for_no_send_replay" assert result["parity"]["problems"] == [] assert result["parity"]["details"]["source_total_rows"] == 1 + assert result["source"]["capture_receipt"]["schema"] == restore.SOURCE_SNAPSHOT_RECEIPT_SCHEMA + assert result["source"]["capture_receipt"]["provenance_binding_sha256"] assert result["target"]["connectivity"]["private_connectivity"] is True + assert result["target"]["cloudsql"]["public_ip_disabled"] is True assert result["target"]["connectivity"]["ssl"] is True assert result["live_service"]["unchanged"] is True assert result["cleanup"]["clone_database_remaining"] == 1 @@ -187,6 +338,15 @@ def test_restore_success_retains_only_exact_parity_clone( run_dir = args.run_root / args.run_id assert (run_dir / "target-manifest.jsonl").stat().st_mode & 0o777 == 0o600 assert (run_dir / "restore-receipt.json").stat().st_mode & 0o777 == 0o600 + connectivity_path = run_dir / "gcp-private-connectivity.json" + assert connectivity_path.stat().st_mode & 0o777 == 0o600 + connectivity = json.loads(connectivity_path.read_text()) + assert connectivity["target_database"] == args.target_db + assert connectivity["source_compute"] == restore.DEFAULT_COMPUTE_INSTANCE + assert connectivity["public_ip_disabled"] is True + assert ( + result["target"]["connectivity_proof"]["sha256"] == hashlib.sha256(connectivity_path.read_bytes()).hexdigest() + ) def test_parity_failure_drops_generated_clone_and_receipts_both_failures( @@ -234,6 +394,79 @@ def test_create_command_failure_still_drops_a_committed_clone( assert state == {"exists": False, "dropped": True} +def test_restore_rejects_manifest_not_bound_to_source_receipt_before_credentials( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + args = args_for(tmp_path) + install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True)) + receipt = json.loads(args.source_receipt.read_text()) + receipt["manifest"]["sha256"] = "0" * 64 + args.source_receipt.write_text(json.dumps(receipt)) + args.expected_source_receipt_sha256 = hashlib.sha256(args.source_receipt.read_bytes()).hexdigest() + credential_called = False + + def unexpected_credential(*_args, **_kwargs): + nonlocal credential_called + credential_called = True + return "private-password" + + monkeypatch.setattr(restore, "resolve_password", unexpected_credential) + + result = restore.run_restore(args) + + assert result["status"] == "fail" + assert result["error"]["phase"] == "validation" + assert "source_manifest_sha256" in result["error"]["message"] + assert credential_called is False + + +def test_restore_rejects_unexpected_capture_authorization_before_credentials( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + args = args_for(tmp_path, expected_capture_authorization_ref="codex-delegation:another-authorized-run") + install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True)) + credential_called = False + + def unexpected_credential(*_args, **_kwargs): + nonlocal credential_called + credential_called = True + return "private-password" + + monkeypatch.setattr(restore, "resolve_password", unexpected_credential) + + result = restore.run_restore(args) + + assert result["status"] == "fail" + assert result["error"]["phase"] == "validation" + assert "capture_authorization_ref" in result["error"]["message"] + assert credential_called is False + + +def test_restore_rejects_noncanonical_source_identity_before_credentials( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + args = args_for(tmp_path, expected_source_container="different-postgres") + install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True)) + credential_called = False + + def unexpected_credential(*_args, **_kwargs): + nonlocal credential_called + credential_called = True + return "private-password" + + monkeypatch.setattr(restore, "resolve_password", unexpected_credential) + + result = restore.run_restore(args) + + assert result["status"] == "fail" + assert result["error"]["phase"] == "validation" + assert "source_identity" in result["error"]["message"] + assert credential_called is False + + def test_changed_live_service_turns_successful_restore_into_cleanup( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, @@ -283,6 +516,17 @@ def test_cleanup_is_bound_to_passing_restore_receipt( "run_id", "host", "project", + "cloudsql_instance", + "expected_private_network", + "expected_compute_instance", + "expected_compute_zone", + "expected_restore_service_account", + "ssl_server_name", + "ssl_root_cert", + "expected_source_ssh_target", + "expected_source_container", + "expected_source_database", + "expected_source_service", "password_secret", "service", "command_timeout", @@ -302,6 +546,112 @@ def test_cleanup_is_bound_to_passing_restore_receipt( assert state == {"exists": False, "dropped": True} +def test_cleanup_rejects_target_topology_drift_before_credentials( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + args = args_for(tmp_path) + state = install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True)) + assert restore.run_restore(args)["status"] == "pass" + cleanup_args = argparse.Namespace( + **{ + key: getattr(args, key) + for key in ( + "target_db", + "run_id", + "host", + "project", + "cloudsql_instance", + "expected_private_network", + "expected_compute_instance", + "expected_compute_zone", + "expected_restore_service_account", + "ssl_server_name", + "ssl_root_cert", + "expected_source_ssh_target", + "expected_source_container", + "expected_source_database", + "expected_source_service", + "password_secret", + "service", + "command_timeout", + "psql_bin", + "run_root", + ) + }, + operation="cleanup", + execute=True, + ) + cleanup_args.host = "10.61.0.4" + credential_called = False + + def unexpected_credential(*_args, **_kwargs): + nonlocal credential_called + credential_called = True + return "private-password" + + monkeypatch.setattr(restore, "resolve_password", unexpected_credential) + + with pytest.raises(restore.RestoreError, match="execution contract"): + restore.run_cleanup(cleanup_args) + + assert credential_called is False + assert state == {"exists": True, "dropped": False} + + +def test_cleanup_retains_failure_receipt_when_post_drop_readback_fails( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + args = args_for(tmp_path) + state = install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True)) + assert restore.run_restore(args)["status"] == "pass" + cleanup_args = argparse.Namespace( + **{ + key: getattr(args, key) + for key in ( + "target_db", + "run_id", + "host", + "project", + "cloudsql_instance", + "expected_private_network", + "expected_compute_instance", + "expected_compute_zone", + "expected_restore_service_account", + "ssl_server_name", + "ssl_root_cert", + "expected_source_ssh_target", + "expected_source_container", + "expected_source_database", + "expected_source_service", + "password_secret", + "service", + "command_timeout", + "psql_bin", + "run_root", + ) + }, + operation="cleanup", + execute=True, + ) + monkeypatch.setattr( + restore, + "rollback_database_state", + lambda *_args, **_kwargs: (_ for _ in ()).throw(restore.RestoreError("rollback readback unavailable")), + ) + + result = restore.run_cleanup(cleanup_args) + + assert result["status"] == "fail" + assert result["error"]["phase"] == "rollback_readback" + assert result["database"]["clone_database_remaining"] == 0 + assert state == {"exists": False, "dropped": True} + receipt_path = args.run_root / args.run_id / "cleanup-receipt.json" + assert receipt_path.stat().st_mode & 0o777 == 0o600 + assert json.loads(receipt_path.read_text())["error"]["phase"] == "rollback_readback" + + def test_restore_dump_uses_tls_no_owner_no_acl_and_keeps_secret_out_of_argv( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, @@ -318,7 +668,10 @@ def test_restore_dump_uses_tls_no_owner_no_acl_and_keeps_secret_out_of_argv( assert captured["command"][0] == "pg_restore" assert captured["command"][1] == "-d" - assert "sslmode=require" in captured["command"][2] + assert "sslmode=verify-full" in captured["command"][2] + assert "hostaddr='10.61.0.3'" in captured["command"][2] + assert "host='teleo-pgvector-standby.private.teleo.internal'" in captured["command"][2] + assert f"sslrootcert='{args.ssl_root_cert}'" in captured["command"][2] assert "--no-owner" in captured["command"] assert "--no-acl" in captured["command"] assert "--exit-on-error" in captured["command"] @@ -326,6 +679,126 @@ def test_restore_dump_uses_tls_no_owner_no_acl_and_keeps_secret_out_of_argv( assert captured["env"]["PGPASSWORD"] == "private-password" +def test_cloudsql_control_plane_preflight_requires_private_only_expected_host( + monkeypatch: pytest.MonkeyPatch, +) -> None: + payload = { + "name": "teleo-pgvector-standby", + "state": "RUNNABLE", + "databaseVersion": "POSTGRES_16", + "region": "europe-west6", + "settings": { + "tier": "db-custom-1-3840", + "ipConfiguration": { + "ipv4Enabled": False, + "privateNetwork": "projects/teleo-501523/global/networks/teleo-staging-net", + }, + }, + "ipAddresses": [{"type": "PRIVATE", "ipAddress": "10.61.0.3"}], + } + + monkeypatch.setattr( + restore, + "_run", + lambda *_args, **_kwargs: subprocess.CompletedProcess([], 0, json.dumps(payload), ""), + ) + result = restore.cloudsql_control_plane_state( + "teleo-501523", + "teleo-pgvector-standby", + "10.61.0.3", + restore.DEFAULT_PRIVATE_NETWORK, + timeout=2.0, + ) + assert result["public_ip_disabled"] is True + assert result["private_addresses"] == ["10.61.0.3"] + + payload["settings"]["ipConfiguration"]["ipv4Enabled"] = True + payload["ipAddresses"].append({"type": "PRIMARY", "ipAddress": "34.1.2.3"}) + with pytest.raises(restore.RestoreError, match="no_non_private_address, public_ip_disabled"): + restore.cloudsql_control_plane_state( + "teleo-501523", + "teleo-pgvector-standby", + "10.61.0.3", + restore.DEFAULT_PRIVATE_NETWORK, + timeout=2.0, + ) + + +def test_gce_compute_identity_is_bound_to_metadata_instance_zone_and_network( + monkeypatch: pytest.MonkeyPatch, +) -> None: + metadata = { + "project/project-id": "teleo-501523", + "project/numeric-project-id": "785938879453", + "instance/name": "teleo-prod-1", + "instance/id": "123456789", + "instance/zone": "projects/123456789/zones/europe-west6-a", + "instance/network-interfaces/0/network": "projects/785938879453/networks/teleo-staging-net", + "instance/network-interfaces/0/ip": "10.61.0.2", + "instance/service-accounts/default/email": restore.DEFAULT_RESTORE_SERVICE_ACCOUNT, + } + monkeypatch.setattr(restore, "gce_metadata_value", lambda path, **_kwargs: metadata[path]) + + result = restore.gce_compute_identity( + "teleo-501523", + "teleo-prod-1", + "europe-west6-a", + restore.DEFAULT_PRIVATE_NETWORK, + restore.DEFAULT_RESTORE_SERVICE_ACCOUNT, + timeout=2.0, + ) + + assert result["instance_id"] == "123456789" + assert result["project_number"] == "785938879453" + assert result["zone"] == "europe-west6-a" + assert all(result["checks"].values()) + + with pytest.raises(restore.RestoreError, match="instance"): + restore.gce_compute_identity( + "teleo-501523", + "different-instance", + "europe-west6-a", + restore.DEFAULT_PRIVATE_NETWORK, + restore.DEFAULT_RESTORE_SERVICE_ACCOUNT, + timeout=2.0, + ) + + metadata["instance/name"] = "teleo-prod-1" + metadata["instance/network-interfaces/0/network"] = "projects/999999999999/networks/teleo-staging-net" + with pytest.raises(restore.RestoreError, match="network"): + restore.gce_compute_identity( + "teleo-501523", + "teleo-prod-1", + "europe-west6-a", + restore.DEFAULT_PRIVATE_NETWORK, + restore.DEFAULT_RESTORE_SERVICE_ACCOUNT, + timeout=2.0, + ) + + metadata["instance/network-interfaces/0/network"] = "projects/785938879453/networks/teleo-staging-net" + with pytest.raises(restore.RestoreError, match="expected_network_project"): + restore.gce_compute_identity( + "teleo-501523", + "teleo-prod-1", + "europe-west6-a", + "projects/other-project/global/networks/teleo-staging-net", + restore.DEFAULT_RESTORE_SERVICE_ACCOUNT, + timeout=2.0, + ) + + metadata["instance/network-interfaces/0/network"] = "projects/785938879453/networks/teleo-staging-net" + metadata["instance/service-accounts/default/email"] = "project-owner@teleo-501523.iam.gserviceaccount.com" + with pytest.raises(restore.RestoreError, match="service_account"): + restore.gce_compute_identity( + "teleo-501523", + "teleo-prod-1", + "europe-west6-a", + restore.DEFAULT_PRIVATE_NETWORK, + restore.DEFAULT_RESTORE_SERVICE_ACCOUNT, + timeout=2.0, + ) + + def test_pg_restore_preflight_rejects_client_older_than_snapshot( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, diff --git a/tests/test_run_local_canonical_postgres_rebuild.py b/tests/test_run_local_canonical_postgres_rebuild.py index 4bd4860..da42fe4 100644 --- a/tests/test_run_local_canonical_postgres_rebuild.py +++ b/tests/test_run_local_canonical_postgres_rebuild.py @@ -137,6 +137,9 @@ def manifest_rows() -> list[dict]: {"kind": "schemas", "items": ["kb_stage", "public"]}, {"kind": "extensions", "items": []}, {"kind": "application_roles", "items": [role]}, + {"kind": "role_memberships", "items": []}, + {"kind": "object_ownership", "items": []}, + {"kind": "object_acl", "items": []}, ] rows.extend( {"kind": kind, "items": []} @@ -159,7 +162,7 @@ def manifest_rows() -> list[dict]: "schema": "public", "table": "claims", "row_count": 1837, - "rowset_md5": "same-row-hash", + "rowset_md5": "c" * 32, }, { "kind": "performance", diff --git a/tests/test_run_local_genesis_ledger_rebuild.py b/tests/test_run_local_genesis_ledger_rebuild.py index 1eb8ca0..63ca9c1 100644 --- a/tests/test_run_local_genesis_ledger_rebuild.py +++ b/tests/test_run_local_genesis_ledger_rebuild.py @@ -16,6 +16,7 @@ from pathlib import Path import pytest from ops import run_local_genesis_ledger_rebuild as rebuild +from ops.verify_postgres_parity_manifest import compare_manifests, load_manifest CLAIM_A = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa" CLAIM_B = "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb" @@ -76,6 +77,9 @@ def manifest_rows(database: str = "teleo_fixture") -> list[dict]: {"kind": "schemas", "items": ["kb_stage", "public"]}, {"kind": "extensions", "items": []}, {"kind": "application_roles", "items": [role]}, + {"kind": "role_memberships", "items": []}, + {"kind": "object_ownership", "items": []}, + {"kind": "object_acl", "items": []}, ] rows.extend( {"kind": kind, "items": []} @@ -98,7 +102,7 @@ def manifest_rows(database: str = "teleo_fixture") -> list[dict]: "schema": "public", "table": "claims", "row_count": 2, - "rowset_md5": "fixture", + "rowset_md5": "f" * 32, }, { "kind": "performance", @@ -1083,6 +1087,51 @@ def source_postgres() -> Iterator[tuple[str, str]]: pytest.fail(f"fixture PostgreSQL container was not removed: {container}") +def test_live_manifest_normalizes_physical_database_authorization_identity( + source_postgres: tuple[str, str], + tmp_path: Path, +) -> None: + container, database = source_postgres + clone_database = f"teleo_clone_{uuid.uuid4().hex[:12]}" + docker_psql( + container, + "postgres", + f'create database "{clone_database}" template "{database}";', + ) + try: + source_manifest_path = tmp_path / "source-database-manifest.jsonl" + clone_manifest_path = tmp_path / "clone-database-manifest.jsonl" + capture_manifest(container, database, source_manifest_path) + capture_manifest(container, clone_database, clone_manifest_path) + + source_manifest = load_manifest(source_manifest_path) + clone_manifest = load_manifest(clone_manifest_path) + problems, _details = compare_manifests( + source_manifest, + clone_manifest, + max_target_query_ms=1_000_000, + max_target_source_ratio=1_000_000, + ) + + assert problems == [] + for manifest in (source_manifest, clone_manifest): + ownership = manifest["singleton"]["object_ownership"]["items"] + database_owner = next(row for row in ownership if row["object_type"] == "database") + assert database_owner["name"] == "canonical_database" + database_acls = [ + row for row in manifest["singleton"]["object_acl"]["items"] if row["object_type"] == "database" + ] + assert database_acls + assert {row["name"] for row in database_acls} == {"canonical_database"} + finally: + docker_psql( + container, + "postgres", + f'drop database if exists "{clone_database}" with (force);', + check=False, + ) + + def capture_source_snapshot( source_container: str, database: str, diff --git a/tests/test_teleo_agent_systemd.py b/tests/test_teleo_agent_systemd.py index 1218a50..1156c9e 100644 --- a/tests/test_teleo_agent_systemd.py +++ b/tests/test_teleo_agent_systemd.py @@ -339,24 +339,24 @@ def test_gcp_runtime_sync_is_source_bound_and_scoped(): 'sudo install -o root -g root -m 0755 "$payload_dir/$SERVICE_ENV_VERIFIER_REL" ' '"$service_env_verifier_stage"' in script ) - assert 'assert_installed_files\nsudo systemctl daemon-reload' in script + assert "assert_installed_files\nsudo systemctl daemon-reload" in script assert 'tar -C "$remote_tmp" -xf -' not in script assert 'mkdir -p "$backup_dir"' not in script assert 'for protected_file in \\\n "$wrapper" \\' in script assert '"$permission_verifier" \\' in script assert '"$service_env_verifier" \\' in script assert 'sudo -n -u teleo test ! -w "$protected_file"' in script - assert 'for protected_dir in \\\n /usr \\' in script + assert "for protected_dir in \\\n /usr \\" in script assert "/usr/local/libexec/livingip" in script assert 'sudo -n -u teleo test ! -w "$protected_dir"' in script - assert 'for executable_dir in \\\n /usr/local/sbin \\' in script - assert 'sudo stat -Lc \'%U:%G\' "$executable_dir"' in script + assert "for executable_dir in \\\n /usr/local/sbin \\" in script + assert "sudo stat -Lc '%U:%G' \"$executable_dir\"" in script assert 'sudo -n -u teleo test ! -w "$executable_dir"' in script assert ( - 'for protected_file in /bin/bash /usr/bin/curl /usr/bin/env /usr/bin/gcloud ' - '/usr/bin/psql /usr/bin/python3 /usr/bin/setpriv' in script + "for protected_file in /bin/bash /usr/bin/curl /usr/bin/env /usr/bin/gcloud " + "/usr/bin/psql /usr/bin/python3 /usr/bin/setpriv" in script ) - assert '/usr/bin/curl -q --noproxy' in script + assert "/usr/bin/curl -q --noproxy" in script assert 'test "$(sudo cat "$revision")" = "$SOURCE_REVISION"' in script assert 'assert_existing_root_directory "$existing_directory"' in script assert 'assert_existing_root_file "$existing_file"' in script @@ -389,10 +389,7 @@ def test_gcp_runtime_sync_is_source_bound_and_scoped(): assert '"current_tier": "T3_live_readonly"' in script assert '"status": "pass"' in script assert "UnsetEnvironment=PGPASSWORD" in dropin - assert ( - "UnsetEnvironment=GOOGLE_APPLICATION_CREDENTIALS CLOUDSDK_CORE_PROJECT " - "CLOUDSDK_AUTH_ACCESS_TOKEN" in dropin - ) + assert "UnsetEnvironment=GOOGLE_APPLICATION_CREDENTIALS CLOUDSDK_CORE_PROJECT CLOUDSDK_AUTH_ACCESS_TOKEN" in dropin assert "UnsetEnvironment=BASH_ENV ENV BASHOPTS SHELLOPTS" in dropin assert "UnsetEnvironment=LD_PRELOAD LD_LIBRARY_PATH LD_AUDIT" in dropin assert "UnsetEnvironment=PYTHONHOME PYTHONPATH PYTHONSTARTUP PYTHONINSPECT" in dropin diff --git a/tests/test_verify_gcp_canonical_lifecycle.py b/tests/test_verify_gcp_canonical_lifecycle.py new file mode 100644 index 0000000..c0da1f0 --- /dev/null +++ b/tests/test_verify_gcp_canonical_lifecycle.py @@ -0,0 +1,820 @@ +import hashlib +import json +from datetime import UTC, datetime +from pathlib import Path + +from ops.restore_gcp_generated_postgres_snapshot import DEFAULT_RESTORE_SERVICE_ACCOUNT +from ops.verify_gcp_canonical_lifecycle import EXPECTED_BLIND_ROW_IDS +from ops.verify_gcp_canonical_lifecycle import verify_lifecycle as _verify_lifecycle +from ops.verify_postgres_parity_manifest import REVIEWED_MANIFEST_SQL_SHA256 + +VERIFY_NOW = datetime(2026, 7, 15, 1, 3, tzinfo=UTC) + + +def verify_lifecycle(**kwargs): + return _verify_lifecycle(**kwargs, now=VERIFY_NOW) + + +def write_json(path: Path, payload: dict) -> Path: + path.write_text(json.dumps(payload, sort_keys=True) + "\n", encoding="utf-8") + return path + + +def sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def lifecycle_receipts(tmp_path: Path) -> dict[str, Path]: + target_db = "teleo_clone_current_test" + run_id = "gcp-restore-current-test123" + healthy_service = { + "ActiveState": "active", + "SubState": "running", + "MainPID": "148735", + "NRestarts": "0", + } + structural_hashes = { + kind: {"source": format(index, "x") * 64, "target": format(index, "x") * 64} + for index, kind in enumerate( + ( + "schemas", + "columns", + "constraints", + "indexes", + "sequences", + "views", + "functions", + "triggers", + "types", + "policies", + "role_memberships", + "object_ownership", + "object_acl", + ), + 1, + ) + } + exact_parity_details = { + "source_database": "teleo", + "target_database": target_db, + "source_table_count": 39, + "target_table_count": 39, + "source_total_rows": 52167, + "target_total_rows": 52167, + "table_mismatches": [], + "table_hash_problems": [], + "structural_hashes": structural_hashes, + "required_extension_mismatches": {}, + "target_extra_extensions": [], + "application_role_mismatches": {}, + "target_extra_application_roles": [], + "performance": [{"query": "count_claims", "source_ms": 1.0, "target_ms": 2.0, "source_ratio": 0.4}], + "performance_problems": [], + } + source = { + "artifact": "vps_canonical_postgres_exported_snapshot", + "schema": "livingip.sourceSnapshotReceipt.v2", + "receipt_version": 2, + "status": "pass", + "run_id": "canonical-current-test", + "capture_authorization_ref": "codex-delegation:test1234", + "source": { + "ssh_target": "root@77.42.65.182", + "container": "teleo-pg", + "database": "teleo", + "service": "leoclean-gateway.service", + }, + "snapshot_exported": True, + "snapshot": { + "captured_at_utc": "2026-07-15T00:59:00+00:00", + "exported_snapshot_id": "00000003-000001A2-1", + "txid_snapshot": "100:120:", + "wal_lsn": "0/16B6C50", + "system_identifier": "7649789040005668902", + }, + "service_unchanged": True, + "source_service": {"before": healthy_service, "after": healthy_service, "unchanged": True}, + "production_db_mutated": False, + "telegram_send_attempted": False, + "dump": {"sha256": "a" * 64}, + "manifest": { + "sha256": "b" * 64, + "manifest_sql_sha256": REVIEWED_MANIFEST_SQL_SHA256, + "table_count": 39, + "total_rows": 52167, + }, + "provenance_binding": {"algorithm": "sha256", "payload": {"bound": True}, "sha256": "c" * 64}, + } + source_path = write_json(tmp_path / "source.json", source) + restore = { + "artifact": "gcp_generated_postgres_snapshot_restore", + "status": "pass", + "generated_at_utc": "2026-07-15T00:59:10+00:00", + "completed_at_utc": "2026-07-15T00:59:30+00:00", + "run_id": run_id, + "target_database": target_db, + "source": { + "capture_receipt": { + "sha256": sha256(source_path), + "capture_authorization_ref": "codex-delegation:test1234", + "provenance_binding_sha256": "c" * 64, + "dump_sha256": "a" * 64, + "manifest_sha256": "b" * 64, + "manifest_sql_sha256": REVIEWED_MANIFEST_SQL_SHA256, + "checks": { + name: True + for name in ( + "artifact", + "schema", + "receipt_version", + "status", + "snapshot_exported", + "service_unchanged", + "source_service_healthy", + "source_not_mutated", + "telegram_not_sent", + "run_id", + "capture_authorization_ref", + "source_identity", + "source_database", + "snapshot_identity", + "dump_sha256", + "dump_bytes", + "source_manifest_sha256", + "manifest_sql_sha256", + "source_context_sha256", + "table_count", + "total_rows", + "binding_algorithm", + "binding_payload", + "binding_sha256", + ) + }, + } + }, + "execution_contract": { + "project": "teleo-501523", + "cloudsql_instance": "teleo-pgvector-standby", + "host": "10.61.0.3", + "private_network": "projects/teleo-501523/global/networks/teleo-staging-net", + "compute_instance": "teleo-prod-1", + "compute_zone": "europe-west6-a", + "restore_service_account": DEFAULT_RESTORE_SERVICE_ACCOUNT, + "ssl_server_name": "teleo-pgvector-standby.private.teleo.internal", + "ssl_root_cert_sha256": "9" * 64, + "source": source["source"], + "password_secret": "gcp-teleo-pgvector-standby-postgres-password", + "service": "leoclean-gcp-prod-parallel.service", + }, + "target": { + "manifest_sha256": "d" * 64, + "connectivity": { + "private_connectivity": True, + "ssl": True, + "sslmode": "verify-full", + "ssl_server_name": "teleo-pgvector-standby.private.teleo.internal", + "ssl_root_cert_sha256": "9" * 64, + "server_identity_verified": True, + "server_address": "10.61.0.3", + "source_compute": "teleo-prod-1", + }, + "cloudsql": { + "project": "teleo-501523", + "instance": "teleo-pgvector-standby", + "expected_private_host": "10.61.0.3", + "private_network": "projects/teleo-501523/global/networks/teleo-staging-net", + "expected_private_network": "projects/teleo-501523/global/networks/teleo-staging-net", + "public_ip_disabled": True, + "checks": { + "instance": True, + "state": True, + "postgres_16": True, + "public_ip_disabled": True, + "private_network": True, + "expected_private_host": True, + "no_non_private_address": True, + }, + }, + "compute": { + "project": "teleo-501523", + "project_number": "785938879453", + "instance": "teleo-prod-1", + "instance_id": "123456789", + "zone": "europe-west6-a", + "expected_private_network": "projects/teleo-501523/global/networks/teleo-staging-net", + "network_resource": "projects/123456789/networks/teleo-staging-net", + "private_ip": "10.61.0.2", + "service_account": DEFAULT_RESTORE_SERVICE_ACCOUNT, + "checks": { + "project": True, + "project_number": True, + "expected_network_project": True, + "instance": True, + "instance_id": True, + "zone": True, + "network": True, + "private_ip": True, + "service_account": True, + }, + }, + "connectivity_proof": { + "sha256": "f" * 64, + "schema": "livingip.gcpPrivatePostgresConnectivity.v2", + }, + }, + "parity": {"status": "pass", "problems": [], "details": exact_parity_details}, + "live_service": {"before": healthy_service, "after": healthy_service, "unchanged": True}, + } + restore_path = write_json(tmp_path / "restore.json", restore) + parity = { + "artifact": "canonical_postgres_parity_verification", + "status": "pass", + "completed_at_utc": "2026-07-15T00:59:45+00:00", + "scope": "gcp_staging", + "problems": [], + "source_manifest_sha256": "b" * 64, + "target_manifest_sha256": "d" * 64, + "connectivity_proof_sha256": "f" * 64, + "details": exact_parity_details, + "private_connectivity": { + "required": True, + "proof": { + "artifact": "gcp_private_postgres_connectivity", + "schema": "livingip.gcpPrivatePostgresConnectivity.v2", + "status": "pass", + "restore_run_id": run_id, + "private_connectivity": True, + "public_ip_disabled": True, + "target_database": target_db, + "source_compute": "teleo-prod-1", + "source_compute_instance_id": "123456789", + "source_compute_zone": "europe-west6-a", + "source_compute_private_ip": "10.61.0.2", + "server_address": "10.61.0.3", + "sslmode": "verify-full", + "ssl_server_name": "teleo-pgvector-standby.private.teleo.internal", + "ssl_root_cert_sha256": "9" * 64, + "server_identity_verified": True, + "project": "teleo-501523", + "cloudsql_instance": "teleo-pgvector-standby", + "private_network": "projects/teleo-501523/global/networks/teleo-staging-net", + "cloudsql_control_plane_checks": restore["target"]["cloudsql"]["checks"], + "compute_identity_checks": restore["target"]["compute"]["checks"], + }, + }, + } + parity_path = write_json(tmp_path / "parity.json", parity) + required_checks = { + name: True + for name in ( + "exact_blind_prompt_without_ids", + "claim_and_sources_retrieved", + "discovery_show_evidence_completed", + "retrieval_receipts_proven", + "private_tls_read_only_target", + "generated_database_unchanged", + "canonical_counts_unchanged", + "database_calls_read_only", + "no_database_write", + "no_telegram_send", + "live_service_unchanged", + "live_profile_unchanged", + "temporary_profile_removed", + ) + } + reasoning = { + "schema": "livingip.gcpGeneratedDbBlindClaimCanary.v1", + "status": "pass", + "generated_at_utc": "2026-07-15T01:00:00+00:00", + "completed_at_utc": "2026-07-15T01:01:00+00:00", + "mode": "gcp_generated_db_gatewayrunner_blind_claim_no_send", + "source_compute": "teleo-prod-1", + "prompt": "Inspect the live claim without me giving you an ID. Do not change the database.", + "errors": [], + "target_database": target_db, + "posted_to_telegram": False, + "database_write_attempted": False, + "parity_receipt": {"sha256": sha256(parity_path)}, + "checks": required_checks, + "database_fingerprint_before": {"sha256": "e" * 64}, + "database_fingerprint_after": {"sha256": "e" * 64}, + "canonical_status_before": {"claims": 1}, + "canonical_status_after": {"claims": 1}, + "database_identity_before": {"database": target_db}, + "database_identity_after": {"database": target_db}, + "service_before": healthy_service, + "service_after": healthy_service, + "temp_profile_absent": True, + "temp_profile_removed": True, + "outcomes": { + "asks_for_user_iteration_before_live": True, + "challenges_weak_support": True, + "decomposes_the_bundled_legal_claim": True, + "does_not_claim_a_database_change": True, + "proposes_candidate_claims": True, + "uses_only_m3taversal_handle": True, + }, + "result": { + "reply": "The claim is weak; here are grounded, review-first alternatives.", + "database_tool_trace": { + "schema": "livingip.leoKbToolTrace.v1", + "database_tool_call_proven": True, + "database_retrieval_receipt_proven": True, + "database_tool_calls_read_only": True, + "database_tool_call_count": 3, + "database_tool_completed_count": 3, + "calls": [ + { + "database_invocations": [{"access_mode": "read_only", "subcommand": subcommand}], + "result": { + "nonempty": True, + "error_detected": False, + "row_ids": sorted(EXPECTED_BLIND_ROW_IDS), + "row_id_count": len(EXPECTED_BLIND_ROW_IDS), + "retrieval_receipt": { + "schema": "livingip.teleoKbRetrievalReceipt.v1", + "semantic_context_sha256": "1" * 64, + "artifact_state_sha256": "2" * 64, + "read_consistency_status": "stable_wal_marker", + }, + }, + } + for subcommand in ("search", "show", "evidence") + ], + }, + "gateway": { + "authorized": True, + "handler_invoked": True, + "model_free_fallback_used": False, + "posted_to_telegram": False, + "child_process": { + "exitcode": 0, + "alive_after_readback": False, + "process_group_alive_after_readback": False, + "timed_out": False, + }, + "tool_surface": {"send_message_tool_enabled": False}, + }, + }, + "wrapper_tool_proof": { + "all_bound_to_supplied_target": True, + "all_completed_successfully": True, + "complete_start_end_pairing": True, + "database_read_only_required": True, + "default_read_only_required": True, + "invocations": [ + { + "database": target_db, + "returncode": 0, + "subcommand": subcommand, + "database_identity": { + "transaction_read_only": "on", + "default_transaction_read_only": "on", + "ssl": True, + "sslmode": "verify-full", + "ssl_server_name": "teleo-pgvector-standby.private.teleo.internal", + "ssl_root_cert_sha256": "9" * 64, + "server_identity_verified": True, + "server_address": "10.61.0.3", + }, + } + for subcommand in ("search", "show", "evidence") + ], + }, + } + reasoning_path = write_json(tmp_path / "reasoning.json", reasoning) + reasoning_compute = { + "artifact": "gcp_reasoning_compute_attestation", + "schema": "livingip.gcpReasoningComputeAttestation.v1", + "status": "pass", + "generated_at_utc": "2026-07-15T01:01:05+00:00", + "restore_run_id": run_id, + "target_database": target_db, + "reasoning_receipt": { + "sha256": sha256(reasoning_path), + "schema": reasoning["schema"], + "generated_at_utc": reasoning["generated_at_utc"], + "completed_at_utc": reasoning["completed_at_utc"], + "claimed_source_compute": reasoning["source_compute"], + }, + "compute": restore["target"]["compute"], + "checks": { + "schema": True, + "status": True, + "target_database": True, + "source_compute": True, + "chronology": True, + "no_telegram_send": True, + "no_database_write": True, + "gce_metadata_identity": True, + }, + "method": "gce_metadata_server_v1", + "mutation": { + "cloudsql_changed": False, + "compute_changed": False, + "database_changed": False, + "telegram_message_sent": False, + }, + } + reasoning_compute_path = write_json(tmp_path / "reasoning-compute.json", reasoning_compute) + current_source = json.loads(json.dumps(source)) + current_source["run_id"] = "canonical-current-postflight" + current_source["capture_authorization_ref"] = "codex-delegation:postflight1234" + current_source["snapshot"]["captured_at_utc"] = "2026-07-15T01:02:00+00:00" + current_source["snapshot"]["exported_snapshot_id"] = "00000003-000001B2-1" + current_source_path = write_json(tmp_path / "current-source.json", current_source) + source_delta = { + "artifact": "vps_canonical_snapshot_postflight_delta", + "schema": "livingip.vpsCanonicalSnapshotDelta.v1", + "status": "pass", + "baseline": { + "run_id": "canonical-current-test", + "capture_receipt_sha256": sha256(source_path), + "manifest_sha256": "b" * 64, + "manifest_sql_sha256": REVIEWED_MANIFEST_SQL_SHA256, + "capture_authorization_ref": "codex-delegation:test1234", + "captured_at_utc": "2026-07-15T00:59:00+00:00", + "table_count": 39, + "total_rows": 52167, + "table_fingerprint_sha256": "d" * 64, + "structure_fingerprint_sha256": "e" * 64, + }, + "current": { + "run_id": "canonical-current-postflight", + "capture_receipt_sha256": sha256(current_source_path), + "manifest_sha256": "b" * 64, + "manifest_sql_sha256": REVIEWED_MANIFEST_SQL_SHA256, + "capture_authorization_ref": "codex-delegation:postflight1234", + "captured_at_utc": "2026-07-15T01:02:00+00:00", + "table_count": 39, + "total_rows": 52167, + "table_fingerprint_sha256": "d" * 64, + "structure_fingerprint_sha256": "e" * 64, + }, + "delta": { + "delta_detected": False, + "added_tables": [], + "removed_tables": [], + "changed_tables": [], + "changed_structures": [], + "total_row_delta": 0, + }, + } + source_delta_path = write_json(tmp_path / "source-delta.json", source_delta) + cleanup = { + "artifact": "gcp_generated_postgres_snapshot_cleanup", + "status": "pass", + "generated_at_utc": "2026-07-15T01:01:10+00:00", + "completed_at_utc": "2026-07-15T01:01:30+00:00", + "run_id": run_id, + "target_database": target_db, + "database": {"existed_before": True, "clone_database_remaining": 0}, + "live_service": {"before": healthy_service, "after": healthy_service, "unchanged": True}, + "restore_receipt": {"sha256": sha256(restore_path)}, + "execution_contract": restore["execution_contract"], + "target": { + "cloudsql": restore["target"]["cloudsql"], + "compute": restore["target"]["compute"], + }, + } + cleanup_path = write_json(tmp_path / "cleanup.json", cleanup) + return { + "source_path": source_path, + "current_source_path": current_source_path, + "source_delta_path": source_delta_path, + "restore_path": restore_path, + "parity_path": parity_path, + "reasoning_path": reasoning_path, + "reasoning_compute_path": reasoning_compute_path, + "cleanup_path": cleanup_path, + } + + +def test_caller_authored_bundle_is_preparation_only_even_when_all_checks_pass(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + + result = verify_lifecycle(**paths) + + assert result["status"] == "prepared_external_attestation_required" + assert result["failed_checks"] == [] + assert result["checks"]["clone_absent"] is True + assert result["current_tier"] == "T2_structural_receipt_bundle" + assert result["required_tier"] == "T3_live_private_gcp_staging" + assert result["t3_receipt_ready"] is False + assert result["accepted_by_this_api"] is False + assert "current private GCP staging" not in result["strongest_claim_allowed"] + + +def test_rebound_unreviewed_manifest_sql_digest_fails_closed(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + unreviewed_digest = "f" * 64 + + source = json.loads(paths["source_path"].read_text()) + source["manifest"]["manifest_sql_sha256"] = unreviewed_digest + write_json(paths["source_path"], source) + current_source = json.loads(paths["current_source_path"].read_text()) + current_source["manifest"]["manifest_sql_sha256"] = unreviewed_digest + write_json(paths["current_source_path"], current_source) + + restore = json.loads(paths["restore_path"].read_text()) + restore["source"]["capture_receipt"]["sha256"] = sha256(paths["source_path"]) + restore["source"]["capture_receipt"]["manifest_sql_sha256"] = unreviewed_digest + write_json(paths["restore_path"], restore) + + delta = json.loads(paths["source_delta_path"].read_text()) + delta["baseline"]["capture_receipt_sha256"] = sha256(paths["source_path"]) + delta["baseline"]["manifest_sql_sha256"] = unreviewed_digest + delta["current"]["capture_receipt_sha256"] = sha256(paths["current_source_path"]) + delta["current"]["manifest_sql_sha256"] = unreviewed_digest + write_json(paths["source_delta_path"], delta) + + cleanup = json.loads(paths["cleanup_path"].read_text()) + cleanup["restore_receipt"]["sha256"] = sha256(paths["restore_path"]) + write_json(paths["cleanup_path"], cleanup) + + result = verify_lifecycle(**paths) + + assert result["status"] == "fail" + assert "source_delta_manifest_sql_reviewed" in result["failed_checks"] + + +def test_local_parity_or_missing_cleanup_fails_closed(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + parity = json.loads(paths["parity_path"].read_text()) + parity["scope"] = "local_restore" + write_json(paths["parity_path"], parity) + reasoning = json.loads(paths["reasoning_path"].read_text()) + reasoning["parity_receipt"]["sha256"] = sha256(paths["parity_path"]) + write_json(paths["reasoning_path"], reasoning) + cleanup = json.loads(paths["cleanup_path"].read_text()) + cleanup["database"]["clone_database_remaining"] = 1 + write_json(paths["cleanup_path"], cleanup) + + result = verify_lifecycle(**paths) + + assert result["status"] == "fail" + assert {"parity_pass", "clone_absent"} <= set(result["failed_checks"]) + + +def test_cleanup_cannot_pass_when_the_named_clone_never_existed(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + cleanup = json.loads(paths["cleanup_path"].read_text()) + cleanup["database"]["existed_before"] = False + write_json(paths["cleanup_path"], cleanup) + + result = verify_lifecycle(**paths) + + assert result["status"] == "fail" + assert "clone_absent" in result["failed_checks"] + + +def test_mismatched_source_or_reasoning_receipt_fails_closed(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + restore = json.loads(paths["restore_path"].read_text()) + restore["source"]["capture_receipt"]["sha256"] = "0" * 64 + write_json(paths["restore_path"], restore) + reasoning = json.loads(paths["reasoning_path"].read_text()) + reasoning["checks"]["claim_and_sources_retrieved"] = False + write_json(paths["reasoning_path"], reasoning) + + result = verify_lifecycle(**paths) + + assert result["status"] == "fail" + assert {"source_receipt_hash_bound", "reasoning_required_checks"} <= set(result["failed_checks"]) + + +def test_hand_authored_reasoning_boole_without_producer_evidence_fail(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + reasoning = json.loads(paths["reasoning_path"].read_text()) + reasoning.pop("result") + reasoning.pop("wrapper_tool_proof") + write_json(paths["reasoning_path"], reasoning) + + result = verify_lifecycle(**paths) + + assert result["status"] == "fail" + assert "reasoning_producer_evidence" in result["failed_checks"] + assert result["reasoning_producer_checks"]["tool_trace"] is False + + +def test_reasoning_compute_attestation_must_bind_live_metadata(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + attestation = json.loads(paths["reasoning_compute_path"].read_text()) + attestation["compute"]["instance_id"] = "999999999" + write_json(paths["reasoning_compute_path"], attestation) + + result = verify_lifecycle(**paths) + + assert result["status"] == "fail" + assert "reasoning_compute_identity_bound" in result["failed_checks"] + + +def test_project_owner_shaped_service_account_cannot_pass_preparation(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + project_owner = "project-owner@teleo-501523.iam.gserviceaccount.com" + restore = json.loads(paths["restore_path"].read_text()) + restore["target"]["compute"]["service_account"] = project_owner + restore["execution_contract"]["restore_service_account"] = project_owner + write_json(paths["restore_path"], restore) + attestation = json.loads(paths["reasoning_compute_path"].read_text()) + attestation["compute"]["service_account"] = project_owner + write_json(paths["reasoning_compute_path"], attestation) + cleanup = json.loads(paths["cleanup_path"].read_text()) + cleanup["target"]["compute"]["service_account"] = project_owner + cleanup["execution_contract"]["restore_service_account"] = project_owner + cleanup["restore_receipt"]["sha256"] = sha256(paths["restore_path"]) + write_json(paths["cleanup_path"], cleanup) + + result = verify_lifecycle(**paths) + + assert result["status"] == "fail" + assert "restore_compute_contract" in result["failed_checks"] + + +def test_source_postflight_must_be_newer_than_cleanup_completion(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + cleanup = json.loads(paths["cleanup_path"].read_text()) + cleanup["completed_at_utc"] = "2026-07-15T02:00:00+00:00" + write_json(paths["cleanup_path"], cleanup) + + result = verify_lifecycle(**paths) + + assert result["status"] == "fail" + assert "source_postflight_after_lifecycle" in result["failed_checks"] + + +def test_parity_must_retain_exact_structure_roles_rows_and_performance(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + parity = json.loads(paths["parity_path"].read_text()) + parity["details"]["structural_hashes"].pop("constraints") + parity["details"]["application_role_mismatches"] = {"kb_apply": {"missing": True}} + parity["details"]["performance"] = [] + write_json(paths["parity_path"], parity) + reasoning = json.loads(paths["reasoning_path"].read_text()) + reasoning["parity_receipt"]["sha256"] = sha256(paths["parity_path"]) + write_json(paths["reasoning_path"], reasoning) + + result = verify_lifecycle(**paths) + + assert result["status"] == "fail" + assert "parity_exact_evidence" in result["failed_checks"] + assert result["parity_evidence_checks"]["structural_hashes"] is False + assert result["parity_evidence_checks"]["roles"] is False + assert result["parity_evidence_checks"]["performance"] is False + + +def test_reasoning_must_retrieve_expected_claim_sources_and_preserve_handle(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + reasoning = json.loads(paths["reasoning_path"].read_text()) + for call in reasoning["result"]["database_tool_trace"]["calls"]: + call["result"]["row_ids"] = [] + call["result"]["row_id_count"] = 0 + reasoning["result"]["reply"] = "Cory found a weak claim and two sources." + write_json(paths["reasoning_path"], reasoning) + + result = verify_lifecycle(**paths) + + assert result["status"] == "fail" + assert "reasoning_producer_evidence" in result["failed_checks"] + assert result["reasoning_producer_checks"]["expected_claim_and_sources_retrieved"] is False + assert result["reasoning_producer_checks"]["reply_uses_only_m3taversal_handle"] is False + + +def test_reasoning_wrapper_must_authenticate_the_cloudsql_server(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + reasoning = json.loads(paths["reasoning_path"].read_text()) + for invocation in reasoning["wrapper_tool_proof"]["invocations"]: + invocation["database_identity"]["sslmode"] = "require" + invocation["database_identity"]["server_identity_verified"] = False + write_json(paths["reasoning_path"], reasoning) + + result = verify_lifecycle(**paths) + + assert result["status"] == "fail" + assert "reasoning_producer_evidence" in result["failed_checks"] + assert result["reasoning_producer_checks"]["wrapper_invocations"] is False + + +def test_lifecycle_requires_restore_parity_reasoning_cleanup_chronology(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + cleanup = json.loads(paths["cleanup_path"].read_text()) + cleanup["generated_at_utc"] = "2026-07-15T00:58:00+00:00" + write_json(paths["cleanup_path"], cleanup) + + result = verify_lifecycle(**paths) + + assert result["status"] == "fail" + assert "lifecycle_chronology" in result["failed_checks"] + + +def test_source_delta_requires_real_fingerprints_and_consistent_no_delta(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + delta = json.loads(paths["source_delta_path"].read_text()) + delta["baseline"].pop("table_fingerprint_sha256") + delta["current"]["total_rows"] += 1 + write_json(paths["source_delta_path"], delta) + + result = verify_lifecycle(**paths) + + assert result["status"] == "fail" + assert {"source_delta_baseline_shape", "source_delta_current_shape", "source_delta_consistent"} <= set( + result["failed_checks"] + ) + + +def test_detected_source_delta_requires_exact_row_arithmetic(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + delta = json.loads(paths["source_delta_path"].read_text()) + delta["delta"]["delta_detected"] = True + delta["delta"]["changed_tables"] = [{"table": "public.claims"}] + delta["delta"]["total_row_delta"] = 99 + write_json(paths["source_delta_path"], delta) + + result = verify_lifecycle(**paths) + + assert result["status"] == "fail" + assert "source_delta_consistent" in result["failed_checks"] + + +def test_postflight_source_requires_structured_healthy_service(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + current = json.loads(paths["current_source_path"].read_text()) + current["source_service"]["before"]["ActiveState"] = "failed" + current["source_service"]["after"]["ActiveState"] = "failed" + write_json(paths["current_source_path"], current) + delta = json.loads(paths["source_delta_path"].read_text()) + delta["current"]["capture_receipt_sha256"] = sha256(paths["current_source_path"]) + write_json(paths["source_delta_path"], delta) + + result = verify_lifecycle(**paths) + + assert result["status"] == "fail" + assert "current_source_pass" in result["failed_checks"] + + +def test_unchanged_but_inactive_service_cannot_pass_t3(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + restore = json.loads(paths["restore_path"].read_text()) + restore["live_service"]["before"]["ActiveState"] = "failed" + restore["live_service"]["after"]["ActiveState"] = "failed" + write_json(paths["restore_path"], restore) + cleanup = json.loads(paths["cleanup_path"].read_text()) + cleanup["restore_receipt"]["sha256"] = sha256(paths["restore_path"]) + write_json(paths["cleanup_path"], cleanup) + + result = verify_lifecycle(**paths) + + assert result["status"] == "fail" + assert "restore_service_healthy_unchanged" in result["failed_checks"] + + +def test_historical_but_monotonic_receipts_cannot_claim_current_t3(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + + result = _verify_lifecycle(**paths, now=datetime(2026, 7, 15, 3, 0, tzinfo=UTC)) + + assert result["status"] == "fail" + assert "source_postflight_fresh" in result["failed_checks"] + assert result["postflight_age_seconds"] > result["max_postflight_age_seconds"] + + +def test_fresh_postflight_cannot_mask_historical_gcp_lifecycle(tmp_path: Path) -> None: + paths = lifecycle_receipts(tmp_path) + source = json.loads(paths["source_path"].read_text()) + source["snapshot"]["captured_at_utc"] = "2026-07-12T00:59:00+00:00" + write_json(paths["source_path"], source) + restore = json.loads(paths["restore_path"].read_text()) + restore["generated_at_utc"] = "2026-07-12T00:59:10+00:00" + restore["completed_at_utc"] = "2026-07-12T00:59:30+00:00" + restore["source"]["capture_receipt"]["sha256"] = sha256(paths["source_path"]) + write_json(paths["restore_path"], restore) + parity = json.loads(paths["parity_path"].read_text()) + parity["completed_at_utc"] = "2026-07-12T00:59:45+00:00" + write_json(paths["parity_path"], parity) + reasoning = json.loads(paths["reasoning_path"].read_text()) + reasoning["generated_at_utc"] = "2026-07-12T01:00:00+00:00" + reasoning["completed_at_utc"] = "2026-07-12T01:01:00+00:00" + reasoning["parity_receipt"]["sha256"] = sha256(paths["parity_path"]) + write_json(paths["reasoning_path"], reasoning) + attestation = json.loads(paths["reasoning_compute_path"].read_text()) + attestation["generated_at_utc"] = "2026-07-12T01:01:05+00:00" + attestation["reasoning_receipt"]["sha256"] = sha256(paths["reasoning_path"]) + attestation["reasoning_receipt"]["generated_at_utc"] = reasoning["generated_at_utc"] + attestation["reasoning_receipt"]["completed_at_utc"] = reasoning["completed_at_utc"] + write_json(paths["reasoning_compute_path"], attestation) + cleanup = json.loads(paths["cleanup_path"].read_text()) + cleanup["generated_at_utc"] = "2026-07-12T01:01:10+00:00" + cleanup["completed_at_utc"] = "2026-07-12T01:01:30+00:00" + cleanup["restore_receipt"]["sha256"] = sha256(paths["restore_path"]) + write_json(paths["cleanup_path"], cleanup) + delta = json.loads(paths["source_delta_path"].read_text()) + delta["baseline"]["captured_at_utc"] = "2026-07-12T00:59:00+00:00" + delta["baseline"]["capture_receipt_sha256"] = sha256(paths["source_path"]) + write_json(paths["source_delta_path"], delta) + + result = verify_lifecycle(**paths) + + assert result["checks"]["source_postflight_fresh"] is True + assert result["checks"]["lifecycle_chronology"] is True + assert result["status"] == "fail" + assert {"gcp_lifecycle_fresh", "lifecycle_span_bounded"} <= set(result["failed_checks"]) diff --git a/tests/test_verify_postgres_parity_manifest.py b/tests/test_verify_postgres_parity_manifest.py index 7199a3a..ff41d86 100644 --- a/tests/test_verify_postgres_parity_manifest.py +++ b/tests/test_verify_postgres_parity_manifest.py @@ -9,7 +9,7 @@ def manifest_rows( *, database: str, row_count: int = 2, - rowset_md5: str = "abc", + rowset_md5: str = "a" * 32, elapsed_ms: float = 4.0, server_address: str | None = None, ssl: bool = False, @@ -28,7 +28,23 @@ def manifest_rows( {"kind": "extensions", "items": [{"name": "plpgsql", "version": "1.0"}]}, { "kind": "application_roles", - "items": [{"name": "kb_apply", "can_login": True, "superuser": False}], + "items": [ + {"name": "kb_apply", "can_login": True, "superuser": False}, + {"name": "kb_gate_owner", "can_login": False, "superuser": False}, + {"name": "kb_review", "can_login": True, "superuser": False}, + ], + }, + { + "kind": "role_memberships", + "items": [{"role": "kb_gate_owner", "member": "kb_apply", "admin_option": False}], + }, + { + "kind": "object_ownership", + "items": [{"object_type": "schema", "schema": "public", "name": "public", "owner": "kb_gate_owner"}], + }, + { + "kind": "object_acl", + "items": [{"object_type": "schema", "name": "public", "grantee": "kb_apply", "privilege_type": "USAGE"}], }, {"kind": "columns", "items": [{"schema": "public", "table": "claims", "name": "id"}]}, {"kind": "constraints", "items": [{"schema": "public", "table": "claims", "name": "claims_pkey"}]}, @@ -90,17 +106,59 @@ def test_matching_local_manifests_pass(tmp_path: Path) -> None: assert completed.returncode == 0, completed.stderr assert payload["status"] == "pass" assert payload["details"]["source_total_rows"] == 2 + assert len(payload["source_manifest_sha256"]) == 64 + assert len(payload["target_manifest_sha256"]) == 64 assert payload["private_connectivity"]["status"] == "not_applicable_local_restore" + assert (tmp_path / "result.json").stat().st_mode & 0o777 == 0o600 + assert '"source_database"' not in completed.stdout + assert '"output_sha256"' in completed.stdout def test_row_hash_and_role_mismatches_fail(tmp_path: Path) -> None: - target = manifest_rows(database="teleo_copy", row_count=1, rowset_md5="different") + target = manifest_rows(database="teleo_copy", row_count=1, rowset_md5="b" * 32) target[3]["items"] = [] completed, payload = run_verifier(tmp_path, manifest_rows(database="teleo"), target) assert completed.returncode == 1 assert "table_content_mismatches=1" in payload["problems"] + assert "application_role_mismatches=3" in payload["problems"] + + +def test_missing_row_hash_and_extra_roles_or_extensions_fail(tmp_path: Path) -> None: + target = manifest_rows(database="teleo_copy", rowset_md5="") + target[2]["items"].append({"name": "unreviewed", "version": "1.0"}) + target[3]["items"].append({"name": "project_owner", "can_login": True, "superuser": True}) + + completed, payload = run_verifier(tmp_path, manifest_rows(database="teleo"), target) + + assert completed.returncode == 1 + assert "table_hash_problems=1" in payload["problems"] + assert "extension_mismatches=1" in payload["problems"] assert "application_role_mismatches=1" in payload["problems"] + assert payload["details"]["target_extra_extensions"] == ["unreviewed"] + assert payload["details"]["target_extra_application_roles"] == ["project_owner"] + + +def test_matching_malformed_row_hashes_fail(tmp_path: Path) -> None: + completed, payload = run_verifier( + tmp_path, + manifest_rows(database="teleo", rowset_md5="not-a-hash"), + manifest_rows(database="teleo_copy", rowset_md5="not-a-hash"), + ) + + assert completed.returncode == 1 + assert "table_hash_problems=2" in payload["problems"] + assert payload["details"]["table_mismatches"] == [] + + +def test_authorization_ownership_or_acl_drift_fails(tmp_path: Path) -> None: + target = manifest_rows(database="teleo_copy") + target[5]["items"][0]["owner"] = "postgres" + + completed, payload = run_verifier(tmp_path, manifest_rows(database="teleo"), target) + + assert completed.returncode == 1 + assert "object_ownership_hash_mismatch" in payload["problems"] def test_gcp_scope_requires_private_connectivity_receipt(tmp_path: Path) -> None: @@ -129,12 +187,17 @@ def test_gcp_scope_accepts_matching_private_connectivity_receipt(tmp_path: Path) "target_database": "teleo_gcp_copy", "server_address": "10.61.0.3", "ssl": True, + "sslmode": "verify-full", + "ssl_server_name": "teleo-pgvector-standby.private.teleo.internal", + "ssl_root_cert_sha256": "a" * 64, + "server_identity_verified": True, }, ) assert completed.returncode == 0, completed.stderr assert payload["status"] == "pass" assert payload["private_connectivity"]["proof"]["source_compute"] == "teleo-staging-1" + assert len(payload["connectivity_proof_sha256"]) == 64 def test_gcp_scope_rejects_receipt_for_a_different_server(tmp_path: Path) -> None: @@ -151,6 +214,10 @@ def test_gcp_scope_rejects_receipt_for_a_different_server(tmp_path: Path) -> Non "target_database": "teleo_gcp_copy", "server_address": "10.61.0.4", "ssl": True, + "sslmode": "verify-full", + "ssl_server_name": "teleo-pgvector-standby.private.teleo.internal", + "ssl_root_cert_sha256": "a" * 64, + "server_identity_verified": True, }, ) @@ -172,6 +239,10 @@ def test_gcp_scope_rejects_public_or_non_tls_target_manifest(tmp_path: Path) -> "target_database": "teleo_gcp_copy", "server_address": "34.1.2.3", "ssl": False, + "sslmode": "verify-full", + "ssl_server_name": "teleo-pgvector-standby.private.teleo.internal", + "ssl_root_cert_sha256": "a" * 64, + "server_identity_verified": True, }, ) diff --git a/tests/test_verify_vps_canonical_snapshot_delta.py b/tests/test_verify_vps_canonical_snapshot_delta.py new file mode 100644 index 0000000..61e264c --- /dev/null +++ b/tests/test_verify_vps_canonical_snapshot_delta.py @@ -0,0 +1,308 @@ +import hashlib +import json +from pathlib import Path + +import pytest + +from ops.capture_vps_canonical_postgres_snapshot import ( + SOURCE_SNAPSHOT_RECEIPT_SCHEMA, + build_provenance_binding, +) +from ops.verify_postgres_parity_manifest import REVIEWED_MANIFEST_SQL_SHA256 +from ops.verify_vps_canonical_snapshot_delta import verify_delta, write_private_json + + +def manifest_text(*, claims: int = 1, constraint_name: str = "claims_pkey") -> str: + rows = [ + { + "kind": "identity", + "database": "teleo", + "server_version_num": 160014, + "transaction_read_only": "on", + }, + {"kind": "schemas", "items": ["kb_stage", "public"]}, + {"kind": "extensions", "items": [{"name": "pgcrypto", "version": "1.3"}]}, + { + "kind": "application_roles", + "items": [ + {"name": "kb_apply", "can_login": True}, + {"name": "kb_gate_owner", "can_login": False}, + {"name": "kb_review", "can_login": True}, + ], + }, + {"kind": "role_memberships", "items": []}, + {"kind": "object_ownership", "items": []}, + {"kind": "object_acl", "items": []}, + {"kind": "columns", "items": []}, + {"kind": "constraints", "items": [{"name": constraint_name}]}, + {"kind": "indexes", "items": []}, + {"kind": "sequences", "items": []}, + {"kind": "views", "items": []}, + {"kind": "functions", "items": []}, + {"kind": "triggers", "items": []}, + {"kind": "types", "items": []}, + {"kind": "policies", "items": []}, + { + "kind": "table", + "schema": "public", + "table": "claims", + "row_count": claims, + "rowset_md5": hashlib.md5(str(claims).encode(), usedforsecurity=False).hexdigest(), + }, + {"kind": "performance", "query": "count_claims", "elapsed_ms": 1.0, "observed_rows": claims}, + ] + return "".join(json.dumps(row, sort_keys=True) + "\n" for row in rows) + + +def capture_fixture( + root: Path, + *, + name: str, + captured_at: str, + authorization_ref: str, + claims: int = 1, + constraint_name: str = "claims_pkey", +) -> tuple[Path, Path]: + manifest = root / f"{name}-manifest.jsonl" + manifest.write_text(manifest_text(claims=claims, constraint_name=constraint_name), encoding="utf-8") + dump_sha256 = hashlib.sha256(f"{name}-dump".encode()).hexdigest() + manifest_sha256 = hashlib.sha256(manifest.read_bytes()).hexdigest() + manifest_sql_sha256 = REVIEWED_MANIFEST_SQL_SHA256 + source_context_sha256 = "3" * 64 + source = { + "ssh_target": "root@77.42.65.182", + "container": "teleo-pg", + "database": "teleo", + "service": "leoclean-gateway.service", + } + healthy_service = { + "ActiveState": "active", + "SubState": "running", + "MainPID": "148735", + "NRestarts": "0", + } + snapshot = { + "exported_snapshot_id": "00000003-000001A2-1", + "txid_snapshot": "100:120:105,111", + "wal_lsn": "0/16B6C50", + "system_identifier": "7649789040005668902", + "captured_at_utc": captured_at, + } + receipt = { + "artifact": "vps_canonical_postgres_exported_snapshot", + "schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA, + "receipt_version": 2, + "status": "pass", + "run_id": name, + "capture_authorization_ref": authorization_ref, + "source": source, + "snapshot_exported": True, + "snapshot": snapshot, + "dump": {"sha256": dump_sha256}, + "manifest": { + "sha256": manifest_sha256, + "manifest_sql_sha256": manifest_sql_sha256, + "table_count": 1, + "total_rows": claims, + }, + "source_context": {"sha256": source_context_sha256}, + "provenance_binding": build_provenance_binding( + run_id=name, + authorization_ref=authorization_ref, + source=source, + snapshot=snapshot, + dump_sha256=dump_sha256, + manifest_sql_sha256=manifest_sql_sha256, + source_manifest_sha256=manifest_sha256, + source_context_sha256=source_context_sha256, + ), + "service_unchanged": True, + "source_service": {"before": healthy_service, "after": healthy_service, "unchanged": True}, + "production_db_mutated": False, + "telegram_send_attempted": False, + } + receipt_path = root / f"{name}-receipt.json" + receipt_path.write_text(json.dumps(receipt, sort_keys=True) + "\n", encoding="utf-8") + return receipt_path, manifest + + +def test_postflight_no_delta_is_bound_and_explicit(tmp_path: Path) -> None: + baseline_receipt, baseline_manifest = capture_fixture( + tmp_path, + name="canonical-baseline", + captured_at="2026-07-15T01:00:00+00:00", + authorization_ref="codex-delegation:baseline123", + ) + current_receipt, current_manifest = capture_fixture( + tmp_path, + name="canonical-postflight", + captured_at="2026-07-15T02:00:00+00:00", + authorization_ref="codex-delegation:postflight123", + ) + + result = verify_delta( + baseline_receipt_path=baseline_receipt, + baseline_manifest_path=baseline_manifest, + baseline_authorization_ref="codex-delegation:baseline123", + current_receipt_path=current_receipt, + current_manifest_path=current_manifest, + current_authorization_ref="codex-delegation:postflight123", + ) + + assert result["status"] == "pass" + assert result["delta"]["delta_detected"] is False + assert result["delta"]["total_row_delta"] == 0 + + +def test_postflight_data_and_structure_delta_is_reported_not_hidden(tmp_path: Path) -> None: + baseline_receipt, baseline_manifest = capture_fixture( + tmp_path, + name="canonical-baseline", + captured_at="2026-07-15T01:00:00+00:00", + authorization_ref="codex-delegation:baseline123", + ) + current_receipt, current_manifest = capture_fixture( + tmp_path, + name="canonical-postflight", + captured_at="2026-07-15T02:00:00+00:00", + authorization_ref="codex-delegation:postflight123", + claims=2, + constraint_name="claims_pkey_v2", + ) + + result = verify_delta( + baseline_receipt_path=baseline_receipt, + baseline_manifest_path=baseline_manifest, + baseline_authorization_ref="codex-delegation:baseline123", + current_receipt_path=current_receipt, + current_manifest_path=current_manifest, + current_authorization_ref="codex-delegation:postflight123", + ) + + assert result["status"] == "pass" + assert result["delta"]["delta_detected"] is True + assert result["delta"]["total_row_delta"] == 1 + assert result["delta"]["changed_tables"][0]["table"] == "public.claims" + assert result["delta"]["changed_structures"][0]["kind"] == "constraints" + + +def test_postflight_rejects_wrong_authorization_or_stale_capture(tmp_path: Path) -> None: + baseline_receipt, baseline_manifest = capture_fixture( + tmp_path, + name="canonical-baseline", + captured_at="2026-07-15T02:00:00+00:00", + authorization_ref="codex-delegation:baseline123", + ) + current_receipt, current_manifest = capture_fixture( + tmp_path, + name="canonical-postflight", + captured_at="2026-07-15T01:00:00+00:00", + authorization_ref="codex-delegation:postflight123", + ) + + with pytest.raises(ValueError, match="newer than"): + verify_delta( + baseline_receipt_path=baseline_receipt, + baseline_manifest_path=baseline_manifest, + baseline_authorization_ref="codex-delegation:baseline123", + current_receipt_path=current_receipt, + current_manifest_path=current_manifest, + current_authorization_ref="codex-delegation:postflight123", + ) + + with pytest.raises(ValueError, match="authorization"): + verify_delta( + baseline_receipt_path=baseline_receipt, + baseline_manifest_path=baseline_manifest, + baseline_authorization_ref="codex-delegation:wrong123", + current_receipt_path=current_receipt, + current_manifest_path=current_manifest, + current_authorization_ref="codex-delegation:postflight123", + ) + + +def test_postflight_rejects_unchanged_but_unhealthy_service(tmp_path: Path) -> None: + baseline_receipt, baseline_manifest = capture_fixture( + tmp_path, + name="canonical-baseline", + captured_at="2026-07-15T01:00:00+00:00", + authorization_ref="codex-delegation:baseline123", + ) + current_receipt, current_manifest = capture_fixture( + tmp_path, + name="canonical-postflight", + captured_at="2026-07-15T02:00:00+00:00", + authorization_ref="codex-delegation:postflight123", + ) + current = json.loads(current_receipt.read_text()) + current["source_service"]["before"]["ActiveState"] = "failed" + current["source_service"]["after"]["ActiveState"] = "failed" + current_receipt.write_text(json.dumps(current, sort_keys=True) + "\n", encoding="utf-8") + + with pytest.raises(ValueError, match="source_service_healthy"): + verify_delta( + baseline_receipt_path=baseline_receipt, + baseline_manifest_path=baseline_manifest, + baseline_authorization_ref="codex-delegation:baseline123", + current_receipt_path=current_receipt, + current_manifest_path=current_manifest, + current_authorization_ref="codex-delegation:postflight123", + ) + + +def test_postflight_rejects_a_different_manifest_algorithm_even_when_rebound(tmp_path: Path) -> None: + baseline_receipt, baseline_manifest = capture_fixture( + tmp_path, + name="canonical-baseline", + captured_at="2026-07-15T01:00:00+00:00", + authorization_ref="codex-delegation:baseline123", + ) + current_receipt, current_manifest = capture_fixture( + tmp_path, + name="canonical-postflight", + captured_at="2026-07-15T02:00:00+00:00", + authorization_ref="codex-delegation:postflight123", + ) + current = json.loads(current_receipt.read_text()) + current["manifest"]["manifest_sql_sha256"] = "f" * 64 + current["provenance_binding"] = build_provenance_binding( + run_id=current["run_id"], + authorization_ref=current["capture_authorization_ref"], + source=current["source"], + snapshot=current["snapshot"], + dump_sha256=current["dump"]["sha256"], + manifest_sql_sha256="f" * 64, + source_manifest_sha256=current["manifest"]["sha256"], + source_context_sha256=current["source_context"]["sha256"], + ) + current_receipt.write_text(json.dumps(current, sort_keys=True) + "\n", encoding="utf-8") + + with pytest.raises(ValueError, match="manifest_sql_reviewed"): + verify_delta( + baseline_receipt_path=baseline_receipt, + baseline_manifest_path=baseline_manifest, + baseline_authorization_ref="codex-delegation:baseline123", + current_receipt_path=current_receipt, + current_manifest_path=current_manifest, + current_authorization_ref="codex-delegation:postflight123", + ) + + +def test_delta_receipt_writer_keeps_private_mode(tmp_path: Path) -> None: + output = tmp_path / "private" / "delta.json" + + write_private_json(output, {"status": "pass"}) + + assert output.stat().st_mode & 0o777 == 0o600 + assert output.parent.stat().st_mode & 0o777 == 0o700 + + +def test_delta_receipt_writer_does_not_chmod_an_existing_parent(tmp_path: Path) -> None: + parent = tmp_path / "existing" + parent.mkdir(mode=0o755) + output = parent / "delta.json" + + write_private_json(output, {"status": "pass"}) + + assert parent.stat().st_mode & 0o777 == 0o755 + assert output.stat().st_mode & 0o777 == 0o600