feat(ops): bind snapshots to private GCP parity lifecycle

This commit is contained in:
twentyOne2x 2026-07-15 04:11:29 +02:00
parent a47415b60e
commit 37c0ed091d
18 changed files with 4123 additions and 111 deletions

View file

@ -11,27 +11,35 @@ Restore a copy of the VPS canonical Leo database to GCP, prove exact schema,
row, role, extension, performance, and private-connectivity parity, then run the row, role, extension, performance, and private-connectivity parity, then run the
real GCP Leo read/reasoning path without Telegram sends or DB mutation. real GCP Leo read/reasoning path without Telegram sends or DB mutation.
The required tier is T3: a live private GCP staging restore, readback,
reasoning, compute attestation, and cleanup receipt. Local restore proof or an
access-gate report does not satisfy this tier.
## Operator Paths ## Operator Paths
The direct alias is passwordless and was live-verified on 2026-07-14: The direct alias was passwordless and live-verified on 2026-07-14:
```bash ```bash
ssh teleo-gcp-staging ssh teleo-gcp-staging
``` ```
It uses `/Users/user/.ssh/google_compute_engine` for the configured operator, It uses `~/.ssh/google_compute_engine` for the configured operator,
disables password and keyboard-interactive authentication, and does not store a disables password and keyboard-interactive authentication, and does not store a
Google password. `sudo -n` also works. The route remains Google password. That is historical proof, not current reachability. The route
firewall-source-dependent, so verify `ssh -o BatchMode=yes teleo-gcp-staging true` before a remains firewall-source-dependent. On
long run instead of assuming retained access is current. 2026-07-15, both direct SSH and the private TCP route timed out; verify
`ssh -o BatchMode=yes teleo-gcp-staging true` before a long run and fail closed
if it does not return successfully.
The intended secondary path is `.github/workflows/gcp-iap-operator.yml`, using The intended secondary path is `.github/workflows/gcp-iap-operator.yml`, using
short-lived GitHub OIDC, IAP, OS Login, and fixed reviewed operations. It is short-lived GitHub OIDC, IAP, OS Login, and fixed reviewed operations. It is
merged but not bootstrapped: workflow run `29208215340` failed at auth with merged but not bootstrapped: the current 2026-07-15 check still lacks the
`invalid_target` because provider `teleo-iap-operator` is absent, disabled, or reviewed operator WIF/IAP authority. Provider `teleo-iap-operator` is absent,
deleted. Do not call this path working until a live `status` run passes. disabled, or deleted (`invalid_target`), and the available artifact identity is intentionally not
a Cloud SQL/Compute operator. Do not call this path working until a live `status` run passes
with the intended operator identity.
Verified target: Historically verified and intended target; reverify before any new lifecycle:
- project `teleo-501523`; - project `teleo-501523`;
- VM `teleo-prod-1` in `europe-west6-a`; - VM `teleo-prod-1` in `europe-west6-a`;
@ -45,8 +53,9 @@ variable, then unset it.
## Two Different Databases ## Two Different Databases
- Canonical collective knowledge is Cloud SQL database `teleo_canonical`, with - Canonical collective knowledge remains VPS PostgreSQL database `teleo`, with
`public.*` and `kb_stage.*` tables. `public.*` and `kb_stage.*` tables. Persistent Cloud SQL database
`teleo_canonical` is a staging copy and has not been promoted.
- Hermes conversation continuity is `state.db` plus session JSONL files. The - Hermes conversation continuity is `state.db` plus session JSONL files. The
`leoclean-cloudsql-memory-sync.service` snapshot path copies this runtime `leoclean-cloudsql-memory-sync.service` snapshot path copies this runtime
memory; it does not populate or prove canonical claims, sources, evidence, memory; it does not populate or prove canonical claims, sources, evidence,
@ -54,7 +63,7 @@ variable, then unset it.
Do not describe a passing Hermes memory sync as canonical KB parity. Do not describe a passing Hermes memory sync as canonical KB parity.
## Current Verified State - 2026-07-14 ## Retained Historical Proof - 2026-07-14
- The newest captured VPS database has `39` tables and `52,167` rows, including - The newest captured VPS database has `39` tables and `52,167` rows, including
claims `1837`, sources `4145`, claim evidence `4670`, claim edges `4916`, and claims `1837`, sources `4145`, claim evidence `4670`, claim edges `4916`, and
@ -110,17 +119,54 @@ Track these independently:
Use: Use:
- `ops/capture_vps_canonical_postgres_snapshot.py` for a single source dump and - `ops/capture_vps_canonical_postgres_snapshot.py` for a single source dump and
manifest; manifest, with a required retained `--authorization-ref` and v2 provenance
receipt binding the exported snapshot identity plus dump/manifest hashes; it
requires identical healthy structured source-service states before/after
(`active/running`, positive PID, nonnegative restart count);
- `ops/restore_gcp_generated_postgres_snapshot.py restore --execute` for a - `ops/restore_gcp_generated_postgres_snapshot.py restore --execute` for a
receipt-bound private-TLS restore into a bounded `teleo_clone_*` database; receipt-bound private-TLS restore into a bounded `teleo_clone_*` database;
pass the exact capture `--source-receipt`, and require its live Cloud SQL
control-plane check to prove public IP disabled before clone creation; it
writes `target-manifest.jsonl`, `restore-receipt.json`, and the generated
`gcp-private-connectivity.json` under
`/var/lib/teleo-gcp-restore-runs/<restore-run-id>/`;
- `ops/restore_gcp_generated_postgres_snapshot.py cleanup --execute` for exact - `ops/restore_gcp_generated_postgres_snapshot.py cleanup --execute` for exact
clone cleanup with live-service and rollback readback; clone cleanup with live-service and rollback readback;
- `ops/postgres_parity_manifest.sql` for row/catalog/role/performance readback; - `ops/postgres_parity_manifest.sql` for row/catalog/role/performance readback;
- `ops/verify_postgres_parity_manifest.py --scope gcp_staging` for exact parity; - `ops/verify_postgres_parity_manifest.py --scope gcp_staging` for exact parity,
always retaining the canonical output name `gcp-parity.json`;
- `scripts/run_gcp_generated_db_direct_claim_suite.py` for the adapter-free, - `scripts/run_gcp_generated_db_direct_claim_suite.py` for the adapter-free,
read-only, no-send six-response replay against a generated `teleo_clone_*`. read-only, no-send six-response replay against a generated `teleo_clone_*`.
- `scripts/run_gcp_generated_db_blind_claim_canary.py` for the bounded ID-free - `scripts/run_gcp_generated_db_blind_claim_canary.py` for the bounded ID-free
claim challenge, source receipt, reasoning, no-write, and cleanup proof. claim challenge, source receipt, reasoning, no-write, and cleanup proof.
- `ops/attest_gcp_reasoning_compute.py` on the replay VM after blind reasoning
and before cleanup, binding the reasoning receipt SHA-256 to live GCE metadata
in `reasoning-compute-attestation.json`;
- a second authorized source capture after cleanup, followed by
`ops/verify_vps_canonical_snapshot_delta.py`, to retain the explicit
post-snapshot VPS delta in `source-delta-receipt.json`;
- `ops/verify_gcp_canonical_lifecycle.py` after cleanup to bind the exact source,
current source, source delta, restore, `gcp_staging` parity, ID-free reasoning,
reasoning-compute attestation, and cleanup receipts into one fail-closed T3
lifecycle verdict. Pass `--max-postflight-age-seconds 900` and
`--max-lifecycle-age-seconds 3600`; the latter bounds both cleanup-to-verdict
age and restore-to-postflight span.
With `TARGET_DB`, `RESTORE_RUN_ID`, and `PRIVATE_RUN_DIR` set as shown in the
runbook, run the replay-host attestation after the blind receipt passes and
before the receipt-bound cleanup:
```bash
sudo python3 ops/attest_gcp_reasoning_compute.py \
--reasoning-receipt "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \
--restore-run-id "$RESTORE_RUN_ID" \
--target-db "$TARGET_DB" \
--project teleo-501523 \
--expected-compute-instance teleo-prod-1 \
--expected-compute-zone europe-west6-a \
--expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \
--output "$PRIVATE_RUN_DIR/reasoning-compute-attestation.json"
```
The replay must use the Hermes virtualenv, not system Python: The replay must use the Hermes virtualenv, not system Python:
@ -129,8 +175,16 @@ The replay must use the Hermes virtualenv, not system Python:
scripts/run_gcp_generated_db_direct_claim_suite.py ... scripts/run_gcp_generated_db_direct_claim_suite.py ...
``` ```
The replay is not complete until the generated clone, temporary profile, The lifecycle is not complete until the generated clone, temporary profile,
children, upload/run directories, and any temporary client are absent. children, transient upload/bundle directories, and any temporary client are
absent. Preserve the private receipt run directory and every useful lifecycle
input, supporting receipt, and verifier output; do not treat that evidence as
transient cleanup. The canonical restore streams through `pg_restore` and
creates no GCS import object.
Static inspection or a dry run is insufficient for these helpers: verify a new
capture against the live VPS, run the focused tests, and require a passing live
lifecycle receipt before describing the workflow as current parity.
## Safety And Claim Ceiling ## Safety And Claim Ceiling
@ -147,12 +201,17 @@ delivery, GCP canonical mutation, ongoing replication, or production cutover.
## Current Access And Next Action ## Current Access And Next Action
Direct SSH and passwordless sudo are currently working. Local `gcloud` still As of 2026-07-15, direct SSH and private TCP timed out, local `gcloud` for
requires account reauthentication for control-plane metadata, so the current `billy@livingip.xyz` requires reauthentication, and the intended operator
private-IP/TLS proof comes from a live database connection while the public-IP- WIF/IAP authority is unavailable. The artifact-builder WIF identity must not be
disabled control-plane receipt remains dated 2026-07-12. expanded or mistaken for database-operator access. Current retained T2/source
and local restore proof does not satisfy the required T3 lifecycle.
The next production decision is not another restore drill. It is whether to The next action after an authorized operator route is restored is: newest
promote a newly verified snapshot into persistent GCP `teleo_canonical` and authorized VPS capture -> disposable private `teleo_clone_*` restore ->
repoint the production read adapter. Until that decision is explicit, keep GCP generated private-connectivity receipt -> `gcp-parity.json` -> no-send
classified as staging and do not expose Cloud SQL publicly. m3taversal blind reasoning -> live GCE reasoning attestation -> receipt-bound
clone cleanup -> postflight VPS capture/delta -> eight-receipt lifecycle verdict
within the 900/3600-second freshness bounds. Do not promote
`teleo_canonical`, expose Cloud SQL publicly, send Telegram, or mutate canonical
knowledge.

View file

@ -42,17 +42,21 @@ jobs:
lib/llm.py \ lib/llm.py \
lib/post_extract.py \ lib/post_extract.py \
ops/apply_gcp_iam_split.py \ ops/apply_gcp_iam_split.py \
ops/attest_gcp_reasoning_compute.py \
ops/capture_vps_canonical_postgres_snapshot.py \ ops/capture_vps_canonical_postgres_snapshot.py \
ops/check_gcp_infra_readiness.py \ ops/check_gcp_infra_readiness.py \
ops/run_gcp_infra_execute_canary.py \ ops/run_gcp_infra_execute_canary.py \
ops/apply_gcp_runtime_baseline.py \ ops/apply_gcp_runtime_baseline.py \
ops/check_gcp_service_communications.py \ ops/check_gcp_service_communications.py \
ops/plan_gcp_iam_split.py \ ops/plan_gcp_iam_split.py \
ops/private_receipt_io.py \
ops/redact_sqlite_postgres_restore_canary.py \ ops/redact_sqlite_postgres_restore_canary.py \
ops/restore_gcp_generated_postgres_snapshot.py \ ops/restore_gcp_generated_postgres_snapshot.py \
ops/sqlite_to_postgres_dump.py \ ops/sqlite_to_postgres_dump.py \
ops/verify_gcp_cloudsql_restore_readback.py \ ops/verify_gcp_cloudsql_restore_readback.py \
ops/verify_gcp_canonical_lifecycle.py \
ops/verify_postgres_parity_manifest.py \ ops/verify_postgres_parity_manifest.py \
ops/verify_vps_canonical_snapshot_delta.py \
telegram/approvals.py \ telegram/approvals.py \
hermes-agent/leoclean-bin/kb_tool.py \ hermes-agent/leoclean-bin/kb_tool.py \
hermes-agent/leoclean-bin/cloudsql_memory_tool.py \ hermes-agent/leoclean-bin/cloudsql_memory_tool.py \
@ -72,6 +76,7 @@ jobs:
scripts/working_leo_m3taversal_oos_benchmark.py \ scripts/working_leo_m3taversal_oos_benchmark.py \
scripts/working_leo_open_ended_benchmark.py \ scripts/working_leo_open_ended_benchmark.py \
tests/test_agent_routing.py \ tests/test_agent_routing.py \
tests/test_attest_gcp_reasoning_compute.py \
tests/test_assemble_telegram_visible_direct_claim_capture_receipt.py \ tests/test_assemble_telegram_visible_direct_claim_capture_receipt.py \
tests/test_build_working_leo_m3taversal_outcome_sandbox.py \ tests/test_build_working_leo_m3taversal_outcome_sandbox.py \
tests/test_decision_engine_replay.py \ tests/test_decision_engine_replay.py \
@ -96,6 +101,8 @@ jobs:
tests/test_verify_leo_db_first_oos_canary.py \ tests/test_verify_leo_db_first_oos_canary.py \
tests/test_compile_kb_source_packet.py \ tests/test_compile_kb_source_packet.py \
tests/test_verify_postgres_parity_manifest.py \ tests/test_verify_postgres_parity_manifest.py \
tests/test_verify_gcp_canonical_lifecycle.py \
tests/test_verify_vps_canonical_snapshot_delta.py \
tests/test_working_leo_m3taversal_oos_benchmark.py \ tests/test_working_leo_m3taversal_oos_benchmark.py \
tests/test_working_leo_open_ended_benchmark.py \ tests/test_working_leo_open_ended_benchmark.py \
tests/test_phase1b_end_to_end.py \ tests/test_phase1b_end_to_end.py \

View file

@ -53,12 +53,21 @@ python3 ops/capture_vps_canonical_postgres_snapshot.py \
--ssh-target root@77.42.65.182 \ --ssh-target root@77.42.65.182 \
--ssh-key ~/.ssh/livingip_hetzner_20260604_ed25519 \ --ssh-key ~/.ssh/livingip_hetzner_20260604_ed25519 \
--run-id canonical-<timestamp> \ --run-id canonical-<timestamp> \
--authorization-ref codex-delegation:THREAD_ID \
--output-dir <private-output-dir> --output-dir <private-output-dir>
``` ```
The capture fails closed if the source service changes while it runs. It Replace `THREAD_ID` with the exact retained authorization reference for the
capture. The v2 receipt binds that reference, exported snapshot ID, TXID
snapshot, WAL LSN, source system identifier, dump hash, manifest hash, reviewed
manifest-SQL hash, and source-context hash. A dump and manifest without this
passing receipt are not eligible for GCP restore.
The capture fails closed unless the structured source-service states before
and after the snapshot are identical and healthy: `ActiveState=active`,
`SubState=running`, a positive `MainPID`, and a nonnegative `NRestarts`. It
retains a private custom dump, dump SHA-256, object TOC, catalog/data manifest, retains a private custom dump, dump SHA-256, object TOC, catalog/data manifest,
and before/after service state. It never restarts Leo or writes to the source and both service-state objects. It never restarts Leo or writes to the source
database. database.
Prove that this exact snapshot can rebuild a blank Postgres target before using Prove that this exact snapshot can rebuild a blank Postgres target before using
@ -79,16 +88,70 @@ then removes the container and proves it is absent. A passing local receipt is
the exact-recovery preflight; it is not semantic recompilation from raw source the exact-recovery preflight; it is not semantic recompilation from raw source
documents. documents.
Run `ops/postgres_parity_manifest.sql` against the isolated restored target, Choose one bounded suffix and use it consistently on the GCP replay VM. The
then compare source and target: restore helper retains its mode-0700 evidence directory at the fixed private
root:
```bash ```bash
python3 ops/verify_postgres_parity_manifest.py \ RUN_SUFFIX=20260715t010000z
TARGET_DB="teleo_clone_${RUN_SUFFIX}"
RESTORE_RUN_ID="gcp-restore-${RUN_SUFFIX}"
PRIVATE_RUN_DIR="/var/lib/teleo-gcp-restore-runs/${RESTORE_RUN_ID}"
```
Use a generated target database such as `teleo_clone_<run_id>`. Never import a
drill into `teleo`, `teleo_kb`, or `teleo_canonical`. Database isolation does
not isolate cluster-global roles or extensions, so verify those separately and
do not run the Docker-only gate bootstrap against the shared Cloud SQL instance.
The canonical restore command must include the v2 capture receipt. It also runs
a live Cloud SQL control-plane preflight before resolving the database secret or
creating the clone, and fails closed unless the exact instance is runnable,
PostgreSQL 16, private-networked, bound to the expected RFC1918 host, and has
public IP disabled. A GCE metadata preflight independently binds the operator
runtime to `teleo-prod-1` in `europe-west6-a` on `teleo-staging-net`; a
hard-coded compute label is not accepted:
```bash
sudo python3 ops/restore_gcp_generated_postgres_snapshot.py restore \
--execute \
--target-db "$TARGET_DB" \
--run-id "$RESTORE_RUN_ID" \
--dump <private-output-dir>/teleo-canonical.dump \
--expected-dump-sha256 <dump-sha256> \
--source-manifest <private-output-dir>/source-manifest.jsonl \
--source-receipt <private-output-dir>/receipt.json \
--expected-source-receipt-sha256 <source-receipt-sha256> \
--expected-capture-authorization-ref codex-delegation:THREAD_ID \
--project teleo-501523 \
--cloudsql-instance teleo-pgvector-standby \
--expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \
--expected-compute-instance teleo-prod-1 \
--expected-compute-zone europe-west6-a \
--expected-source-ssh-target root@77.42.65.182 \
--expected-source-container teleo-pg \
--expected-source-database teleo \
--expected-source-service leoclean-gateway.service
```
The restore writes both `target-manifest.jsonl` and
`gcp-private-connectivity.json` as mode-0600 files in its private run
directory. The latter is composed from live GCE metadata, Cloud SQL
control-plane checks, and the private TLS database identity; do not hand-compose
the connectivity receipt. The restore also records identical healthy
before/after state for `leoclean-gcp-prod-parallel.service`, including its PID
and restart count.
Compare the captured target manifest and generated connectivity receipt to the
authorized source:
```bash
sudo python3 ops/verify_postgres_parity_manifest.py \
--source <private-output-dir>/source-manifest.jsonl \ --source <private-output-dir>/source-manifest.jsonl \
--target <private-output-dir>/target-manifest.jsonl \ --target "$PRIVATE_RUN_DIR/target-manifest.jsonl" \
--scope gcp_staging \ --scope gcp_staging \
--connectivity-proof <private-output-dir>/gcp-private-connectivity.json \ --connectivity-proof "$PRIVATE_RUN_DIR/gcp-private-connectivity.json" \
--output <private-output-dir>/gcp-parity.json --output "$PRIVATE_RUN_DIR/gcp-parity.json"
``` ```
The verifier checks all table row counts and collation-independent row hashes, The verifier checks all table row counts and collation-independent row hashes,
@ -98,14 +161,100 @@ password-free application-role attributes, and bounded query timings. In GCP
scope it also requires a receipt proving a staging compute source, a private scope it also requires a receipt proving a staging compute source, a private
server address, TLS, and public-IP-disabled instance metadata. server address, TLS, and public-IP-disabled instance metadata.
Use a generated target database such as `teleo_clone_<run_id>`. Never import a After the parity verifier passes, run the no-send m3taversal blind reasoning
drill into `teleo`, `teleo_kb`, or `teleo_canonical`. Database isolation does replay from staging compute against that generated database, attest the replay
not isolate cluster-global roles or extensions, so verify those separately and host from live GCE metadata, and only then delete the generated database. This
do not run the Docker-only gate bootstrap against the shared Cloud SQL instance. canonical path streams the custom dump through `pg_restore`; it does not create
a GCS import object.
After the parity verifier passes, run the no-send operator composition replay from Run the bounded ID-free reasoning canary with the generated parity receipt. The
staging compute against that generated database. Only then delete the generated script re-executes itself in the Hermes virtualenv when that runtime is present:
database and uploaded import object and retain cleanup readback.
```bash
sudo python3 scripts/run_gcp_generated_db_blind_claim_canary.py \
--execute \
--target-db "$TARGET_DB" \
--cloudsql-tool hermes-agent/leoclean-bin/cloudsql_memory_tool.py \
--manifest-sql ops/postgres_parity_manifest.sql \
--parity-receipt "$PRIVATE_RUN_DIR/gcp-parity.json" \
--output "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \
--run-id "gcp-blind-claim-${RUN_SUFFIX}"
```
Before cleanup, bind that reasoning receipt to live metadata from the same GCE
replay host:
```bash
sudo python3 ops/attest_gcp_reasoning_compute.py \
--reasoning-receipt "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \
--restore-run-id "$RESTORE_RUN_ID" \
--target-db "$TARGET_DB" \
--project teleo-501523 \
--expected-compute-instance teleo-prod-1 \
--expected-compute-zone europe-west6-a \
--expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \
--output "$PRIVATE_RUN_DIR/reasoning-compute-attestation.json"
```
After the compute attestation passes, run the receipt-bound cleanup command
retained by the restore. Then capture the VPS source again under a distinct
postflight authorization reference. Run this capture from the same trusted
source-access lane used for the baseline; it need not run on the GCP replay VM.
The postflight capture must complete after cleanup, so the final verdict can
state whether the VPS changed after the restored snapshot boundary. Compare the
two source captures:
```bash
python3 ops/capture_vps_canonical_postgres_snapshot.py \
--execute \
--ssh-target root@77.42.65.182 \
--ssh-key ~/.ssh/livingip_hetzner_20260604_ed25519 \
--run-id canonical-${RUN_SUFFIX}-postflight \
--authorization-ref codex-delegation:THREAD_ID:postflight \
--output-dir <private-postflight-dir>
```
```bash
sudo python3 ops/verify_vps_canonical_snapshot_delta.py \
--baseline-receipt <private-output-dir>/receipt.json \
--baseline-manifest <private-output-dir>/source-manifest.jsonl \
--baseline-authorization-ref codex-delegation:THREAD_ID \
--current-receipt <private-postflight-dir>/receipt.json \
--current-manifest <private-postflight-dir>/source-manifest.jsonl \
--current-authorization-ref codex-delegation:THREAD_ID:postflight \
--output "$PRIVATE_RUN_DIR/source-delta-receipt.json"
```
Finally, verify that all eight receipts belong to one bound lifecycle. This
verifier rejects local-scope parity, stale or mismatched source hashes, a
postflight source capture that predates reasoning or cleanup, hand-authored
reasoning booleans without real retrieval/tool receipts, an unattested replay
host, compute or Cloud SQL topology drift, and any cleanup receipt that leaves
the clone present. The postflight capture may be at most 900 seconds old at
verification; the GCP lifecycle age and restore-to-postflight span must each be
at most 3600 seconds:
```bash
sudo python3 ops/verify_gcp_canonical_lifecycle.py \
--source-receipt <private-output-dir>/receipt.json \
--current-source-receipt <private-postflight-dir>/receipt.json \
--source-delta-receipt "$PRIVATE_RUN_DIR/source-delta-receipt.json" \
--restore-receipt "$PRIVATE_RUN_DIR/restore-receipt.json" \
--parity-receipt "$PRIVATE_RUN_DIR/gcp-parity.json" \
--reasoning-receipt "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \
--reasoning-compute-receipt "$PRIVATE_RUN_DIR/reasoning-compute-attestation.json" \
--cleanup-receipt "$PRIVATE_RUN_DIR/cleanup-receipt.json" \
--max-postflight-age-seconds 900 \
--max-lifecycle-age-seconds 3600 \
--output "$PRIVATE_RUN_DIR/lifecycle-verification.json"
```
Retain the private receipt run directory and every useful lifecycle input,
supporting receipt, and verifier output. Remove only transient upload/bundle
directories, temporary Hermes profiles, helper clients, and child processes
after their absence is verified. There is no canonical-path GCS object to
delete. Do not remove the evidence directory until its useful receipts are
integrated or explicitly rejected.
## Legacy SQLite Source Backup Canary ## Legacy SQLite Source Backup Canary
@ -236,11 +385,15 @@ A successful restore or replication canary must retain:
- dump timestamp or replication slot timestamp; - dump timestamp or replication slot timestamp;
- source schema/database name. - source schema/database name.
- transfer proof: - transfer proof:
- dump object path in a versioned bucket, or logical replication subscription details; - for the canonical path, the v2 capture receipt, dump SHA-256, and exact
private staging path used by streaming `pg_restore`;
- for a legacy import or replication path, a versioned dump-object generation
or logical replication subscription details;
- row/table counts before import where available. - row/table counts before import where available.
- target proof: - target proof:
- `teleo-pgvector-standby` readback; - `teleo-pgvector-standby` control-plane readback;
- `teleo_kb` database readback; - generated `teleo_clone_*` identity for canonical parity, or `teleo_kb` only
for the explicitly legacy SQLite drill;
- extension readback for `vector` if the restored schema needs pgvector; - extension readback for `vector` if the restored schema needs pgvector;
- representative query readback for core KB tables. - representative query readback for core KB tables.
- failure boundary: - failure boundary:
@ -295,28 +448,26 @@ Retain only redacted connection metadata. Do not commit or paste credentials.
## Current Blocker ## Current Blocker
As of 2026-07-11, the canonical Postgres exported-snapshot capture and isolated As of 2026-07-15, a newest authorized canonical Postgres capture and isolated
local restore parity pass. Live GCP restore and staging replay do not. local restore can pass, but the required T3 live private GCP lifecycle cannot be
started from the current operator route:
- GitHub WIF works for `sa-artifact-builder`, but that identity is intentionally - direct VM SSH and the private TCP route timed out;
limited to Artifact Registry and cannot inspect or mutate Cloud SQL/Compute. - local `gcloud` for `billy@livingip.xyz` requires reauthentication; no
- The configured `sa-teleo-readiness` and `sa-teleo-restore-drill` identities password was entered or inspected;
return IAM 404 and do not exist. - the intended operator WIF/IAP authority is unavailable because provider
- The local privileged `billy@livingip.xyz` gcloud session requires password `teleo-iap-operator` is absent, disabled, or deleted;
reauthentication. No password was entered or inspected. - the working artifact-builder WIF identity is intentionally limited and must
- Direct VM SSH is closed to the current egress `/32`; IAP requires the same not be expanded or mistaken for Cloud SQL/Compute operator access.
privileged GCP authentication.
That is why the readiness checker still reports: The authorized recovery is a local
`gcloud auth login billy@livingip.xyz --force` without sharing credentials, or
administrator convergence of the reviewed operator WIF/IAP bootstrap. Until a
route passes live readback, keep the claim at T2/source-local proof; do not
promote staging or expose Cloud SQL publicly.
- `kb_source_restore_access = blocked` Once access is restored, run: newest authorized `teleo` snapshot -> disposable
- `kb_restore_or_replication = blocked` private `teleo_clone_*` restore -> generated connectivity proof ->
`gcp-parity.json` -> no-send m3taversal blind reasoning -> live GCE reasoning
The immediate operator CTA is to complete attestation -> receipt-bound clone cleanup -> postflight VPS capture and delta
`gcloud auth login billy@livingip.xyz --force` locally without sharing the -> eight-receipt lifecycle verdict within the 900/3600-second freshness bounds.
password, or apply the reviewed IAM split with an authorized GCP administrator.
The next non-user action is:
canonical `teleo` snapshot -> generated Cloud SQL database -> full parity and
private-connectivity verifier -> no-send Cory composition replay from staging
compute -> delete the generated database/object -> retain cleanup proof.

View file

@ -0,0 +1,134 @@
{
"artifact": "gcp_canonical_parity_live_attempt",
"schema": "livingip.gcpCanonicalParityLiveAttempt.v1",
"generated_at_utc": "2026-07-15T01:50:20Z",
"status": "gated_external",
"required_tier": "T3_live_private_gcp_staging",
"current_tier": "T2_current_source_and_isolated_restore_plus_live_access_gate",
"current_canary": "Restore the newest authorized canonical snapshot into one private generated Cloud SQL database, prove exact parity and ID-free Leo reasoning from staging compute, then drop the generated database and verify absence.",
"current_source": {
"run_id": "canonical-20260715t014930z",
"captured_at_utc": "2026-07-15T01:49:39.104116+00:00",
"capture_authorization_ref": "codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621",
"receipt_sha256": "7d96b052591d36df3b67dc6291b0f74a7bbeb9602c9cc424779e7c21194d59a5",
"dump_sha256": "6c56a841c1159a17f9404c2d63e3e93ae8ff55e842bf311d6d1bff21e9b95c9a",
"manifest_sha256": "39f06c6b9fc806a89ecd19294be29c58c499c65234611c7a483ddefa3c78d7d3",
"provenance_binding_sha256": "fb587093a76b61af670b7d968d32b78d6a87572faa78ca1ad93ef6cba1c9b776",
"source_system_identifier": "7649789040005668902",
"table_count": 39,
"total_rows": 52167,
"service": {
"active_state": "active",
"sub_state": "running",
"main_pid": 347406,
"restart_count": 0,
"unchanged": true
},
"production_database_mutated": false,
"telegram_send_attempted": false
},
"source_stability": {
"baseline_run_id": "canonical-20260715t014814z",
"current_run_id": "canonical-20260715t014930z",
"receipt_sha256": "d761a2b8b0313ee03c9d01e50fcca709af779578281cf10de5aed4709e914d72",
"status": "pass",
"delta_detected": false,
"table_count": 39,
"total_rows": 52167,
"total_row_delta": 0,
"changed_tables": [],
"changed_structures": [],
"table_fingerprint_sha256": "f50d86fc3ce699f602bdc0029485bd901fd98bd8114eecd5dcc4346a0f52112a",
"structure_fingerprint_sha256": "61bf482330a652ebdb419de3fb78196f2c7d25d1a482b100f385e8102c8e8d52"
},
"isolated_restore": {
"receipt_sha256": "181901afc91bebee7c343500c601ed1e5b7a5e42a5b81583d445109bfc680d86",
"status": "pass",
"source_table_count": 39,
"target_table_count": 39,
"source_total_rows": 52167,
"target_total_rows": 52167,
"table_mismatches": [],
"structural_hashes_equal": true,
"constraint_hash": "8b26d26fda56f09325e0603aae48051da08dfc6c35322fa88880acdabb8ad859",
"application_role_mismatches": {},
"target_extra_application_roles": [],
"required_extension_mismatches": {},
"performance_problems": [],
"network_mode": "none",
"persistent_postgres_storage_used": false,
"container_absent_after_test": true
},
"permission_profile": {
"approval_policy": "never",
"sandbox_mode": "danger-full-access",
"network": "enabled"
},
"config_readback": {
"gcloud_sdk_version": "551.0.0",
"configured_account": "billy@livingip.xyz",
"configured_project": "teleo-501523",
"configured_account_status": "failed_noninteractive_reauthentication",
"application_default_credentials_status": "failed_noninteractive_reauthentication",
"expected_cloudsql_instance_not_currently_verified": "teleo-pgvector-standby",
"expected_private_network_not_currently_verified": "teleo-staging-net",
"expected_private_address_not_currently_verified": "10.61.0.3",
"expected_staging_compute_not_currently_verified": "teleo-prod-1/europe-west6-a",
"public_ip_disabled_currently_verified": false,
"generated_database_inventory_currently_verified": false
},
"attempted_no_approval_routes": [
"configured privileged gcloud identity",
"three other cached gcloud identities",
"application-default credentials",
"direct SSH to the retained staging address",
"local gcloud IAP SSH dry run",
"local and VPS TCP probes to the expected private Cloud SQL address",
"fixed GitHub IAP status workflow receipt",
"five GitHub readiness workflow receipts across candidate service accounts",
"current successful Artifact Registry-only WIF workflow"
],
"exact_gate": "The privileged local Google OAuth session requires human reauthentication, the fixed IAP workflow names an unavailable teleo-iap-operator WIF provider, and no existing noninteractive service-account route can obtain a Cloud SQL/Compute-capable token. Current Cloud SQL topology and generated-database inventory cannot be read, so the private restore cannot start.",
"why_autonomous_repair_stops": "Google reauthentication may require a password, MFA, passkey, or consent that Codex must not retrieve or enter. Recreating or rebinding the WIF/IAP identities requires an already-authorized GCP administrator, and no safe noninteractive identity currently has that authority.",
"next_non_user_action": "After operator-controlled Google reauthentication or administrator convergence of the reviewed least-privilege IAP operator, read the current Cloud SQL and GCE topology, stream the receipt-bound dump to a teleo_clone_* database from teleo-prod-1, run exact parity, blind reasoning, live GCE metadata attestation, cleanup, postflight source capture, and the eight-receipt lifecycle verifier.",
"gcp_effects_from_this_attempt": {
"cloudsql_database_created": false,
"gcp_resource_created": false,
"gcp_resource_modified": false,
"public_ip_exposure_changed": false,
"staging_promoted": false,
"telegram_sent": false
},
"proof_receipts": {
"retained_private_runtime_artifacts": true,
"source_receipt": {
"filename": "receipt.json",
"sha256": "7d96b052591d36df3b67dc6291b0f74a7bbeb9602c9cc424779e7c21194d59a5"
},
"source_manifest": {
"filename": "source-manifest.jsonl",
"sha256": "39f06c6b9fc806a89ecd19294be29c58c499c65234611c7a483ddefa3c78d7d3"
},
"source_delta": {
"filename": "source-stability-delta.json",
"sha256": "d761a2b8b0313ee03c9d01e50fcca709af779578281cf10de5aed4709e914d72"
},
"isolated_restore": {
"filename": "local-rebuild-receipt.json",
"sha256": "181901afc91bebee7c343500c601ed1e5b7a5e42a5b81583d445109bfc680d86"
},
"route_recheck": {
"filename": "route-recheck.json",
"sha256": "0fda5e6d58d72dcb084dc277bc59312bf62c002c728e249a9f16e79ae096037b"
}
},
"strongest_claim_allowed": "The newest authorized VPS snapshot is stable and exactly reconstructs in an isolated PostgreSQL 16 canary; current private GCP restore, reasoning, and cleanup are not proven because the live operator route is externally gated.",
"not_proven": [
"current Cloud SQL topology or public-IP-disabled state",
"current generated-database inventory",
"private staging-to-Cloud-SQL TLS connectivity",
"current Cloud SQL restore parity",
"blind Leo reasoning against the generated Cloud SQL database",
"generated Cloud SQL database teardown"
]
}

View file

@ -0,0 +1,36 @@
{
"artifact": "gcp_canonical_parity_attempt_teardown",
"schema": "livingip.gcpCanonicalParityAttemptTeardown.v1",
"generated_at_utc": "2026-07-15T01:50:20Z",
"status": "no_cloud_resource_created",
"required_tier": "T3_live_private_gcp_staging",
"required_tier_met": false,
"cloudsql_clone_created": false,
"cloudsql_clone_dropped": false,
"cloudsql_clone_absence_readback": "not_applicable_no_create",
"gcs_object_created": false,
"compute_resource_created": false,
"cleanup_required": false,
"cleanup_performed": false,
"orphan_risk_from_this_attempt": false,
"public_ip_exposure_changed": false,
"staging_promoted": false,
"local_restore_container": {
"name": "teleo-canonical-rebuild-20260715015004-4ec8d1daac32",
"force_remove_exit_code": 0,
"container_absent": true,
"persistent_storage_used": false,
"network_access_available": false
},
"current_gcp_orphan_inventory_verified": false,
"exact_gate": "No authenticated Cloud SQL/Compute-capable route was available, so execution stopped before any GCP resource or generated database was created.",
"proof_paths": {
"live_attempt": "docs/reports/leo-working-state-20260709/gcp-canonical-parity-live-20260715.json",
"route_recheck_filename": "route-recheck.json",
"route_recheck_sha256": "0fda5e6d58d72dcb084dc277bc59312bf62c002c728e249a9f16e79ae096037b",
"local_cleanup_receipt_filename": "local-rebuild-receipt.json",
"local_cleanup_receipt_sha256": "181901afc91bebee7c343500c601ed1e5b7a5e42a5b81583d445109bfc680d86",
"runtime_receipts_retained_privately": true
},
"strongest_claim_allowed": "This attempt introduced no GCP cleanup debt and its isolated local restore container was removed; it is not a teardown receipt for a live disposable Cloud SQL database."
}

View file

@ -0,0 +1,172 @@
#!/usr/bin/env python3
"""Bind a no-send reasoning receipt to live GCE metadata on the replay host."""
from __future__ import annotations
import argparse
import json
import stat
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
try:
from .private_receipt_io import write_private_json
from .restore_gcp_generated_postgres_snapshot import (
DEFAULT_COMPUTE_INSTANCE,
DEFAULT_COMPUTE_ZONE,
DEFAULT_PRIVATE_NETWORK,
DEFAULT_PROJECT,
RUN_ID_RE,
TARGET_DATABASE_RE,
gce_compute_identity,
sha256_file,
)
except ImportError: # pragma: no cover - direct script execution on the GCP VM
from private_receipt_io import write_private_json
from restore_gcp_generated_postgres_snapshot import (
DEFAULT_COMPUTE_INSTANCE,
DEFAULT_COMPUTE_ZONE,
DEFAULT_PRIVATE_NETWORK,
DEFAULT_PROJECT,
RUN_ID_RE,
TARGET_DATABASE_RE,
gce_compute_identity,
sha256_file,
)
REASONING_SCHEMA = "livingip.gcpGeneratedDbBlindClaimCanary.v1"
ATTESTATION_SCHEMA = "livingip.gcpReasoningComputeAttestation.v1"
def utc_now() -> str:
return datetime.now(UTC).isoformat()
def parse_timestamp(value: Any, label: str) -> datetime:
text = str(value or "")
if text.endswith("Z"):
text = text[:-1] + "+00:00"
try:
parsed = datetime.fromisoformat(text)
except ValueError as exc:
raise ValueError(f"{label} is not an ISO-8601 timestamp") from exc
if parsed.tzinfo is None:
raise ValueError(f"{label} must include a timezone")
return parsed.astimezone(UTC)
def load_reasoning_receipt(path: Path) -> dict[str, Any]:
try:
mode = path.lstat().st_mode
except OSError as exc:
raise ValueError(f"reasoning receipt is unavailable: {exc}") from exc
if not stat.S_ISREG(mode) or path.is_symlink():
raise ValueError("reasoning receipt must be a regular non-symlink file")
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except json.JSONDecodeError as exc:
raise ValueError("reasoning receipt is invalid JSON") from exc
if not isinstance(payload, dict):
raise ValueError("reasoning receipt must be a JSON object")
return payload
def attest_reasoning_compute(
*,
reasoning_receipt_path: Path,
restore_run_id: str,
target_database: str,
project: str = DEFAULT_PROJECT,
expected_compute_instance: str = DEFAULT_COMPUTE_INSTANCE,
expected_compute_zone: str = DEFAULT_COMPUTE_ZONE,
expected_private_network: str = DEFAULT_PRIVATE_NETWORK,
metadata_timeout: float = 10.0,
generated_at_utc: str | None = None,
) -> dict[str, Any]:
if not RUN_ID_RE.fullmatch(restore_run_id):
raise ValueError("restore run id is invalid")
if not TARGET_DATABASE_RE.fullmatch(target_database):
raise ValueError("target database must be a disposable teleo_clone_* database")
reasoning_receipt_path = reasoning_receipt_path.resolve()
reasoning = load_reasoning_receipt(reasoning_receipt_path)
started = parse_timestamp(reasoning.get("generated_at_utc"), "reasoning start")
completed = parse_timestamp(reasoning.get("completed_at_utc"), "reasoning completion")
checks = {
"schema": reasoning.get("schema") == REASONING_SCHEMA,
"status": reasoning.get("status") == "pass" and not reasoning.get("errors"),
"target_database": reasoning.get("target_database") == target_database,
"source_compute": reasoning.get("source_compute") == expected_compute_instance,
"chronology": started <= completed,
"no_telegram_send": reasoning.get("posted_to_telegram") is False,
"no_database_write": reasoning.get("database_write_attempted") is False,
}
failed = sorted(name for name, passed in checks.items() if not passed)
if failed:
raise ValueError("reasoning receipt failed attestation preflight: " + ", ".join(failed))
compute = gce_compute_identity(
project,
expected_compute_instance,
expected_compute_zone,
expected_private_network,
timeout=metadata_timeout,
)
return {
"artifact": "gcp_reasoning_compute_attestation",
"schema": ATTESTATION_SCHEMA,
"status": "pass",
"generated_at_utc": generated_at_utc or utc_now(),
"restore_run_id": restore_run_id,
"target_database": target_database,
"reasoning_receipt": {
"sha256": sha256_file(reasoning_receipt_path),
"schema": reasoning["schema"],
"generated_at_utc": reasoning["generated_at_utc"],
"completed_at_utc": reasoning["completed_at_utc"],
"claimed_source_compute": reasoning["source_compute"],
},
"compute": compute,
"checks": {**checks, "gce_metadata_identity": all(compute.get("checks", {}).values())},
"method": "gce_metadata_server_v1",
"mutation": {
"cloudsql_changed": False,
"compute_changed": False,
"database_changed": False,
"telegram_message_sent": False,
},
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--reasoning-receipt", required=True, type=Path)
parser.add_argument("--restore-run-id", required=True)
parser.add_argument("--target-db", required=True)
parser.add_argument("--project", default=DEFAULT_PROJECT)
parser.add_argument("--expected-compute-instance", default=DEFAULT_COMPUTE_INSTANCE)
parser.add_argument("--expected-compute-zone", default=DEFAULT_COMPUTE_ZONE)
parser.add_argument("--expected-private-network", default=DEFAULT_PRIVATE_NETWORK)
parser.add_argument("--metadata-timeout", default=10.0, type=float)
parser.add_argument("--output", required=True, type=Path)
args = parser.parse_args()
try:
receipt = attest_reasoning_compute(
reasoning_receipt_path=args.reasoning_receipt,
restore_run_id=args.restore_run_id,
target_database=args.target_db,
project=args.project,
expected_compute_instance=args.expected_compute_instance,
expected_compute_zone=args.expected_compute_zone,
expected_private_network=args.expected_private_network,
metadata_timeout=args.metadata_timeout,
)
write_private_json(args.output, receipt)
except (OSError, ValueError, RuntimeError) as exc:
parser.error(str(exc))
print(json.dumps(receipt, indent=2, sort_keys=True))
return 0
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -25,15 +25,20 @@ except ImportError: # pragma: no cover - direct script execution
SAFE_NAME_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}\Z") SAFE_NAME_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}\Z")
SAFE_RUN_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_-]{7,63}\Z") SAFE_RUN_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_-]{7,63}\Z")
SAFE_SSH_TARGET_RE = re.compile( SAFE_AUTHORIZATION_REF_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.:/@+-]{7,255}\Z")
r"(?:[A-Za-z0-9][A-Za-z0-9._-]{0,63}@)?[A-Za-z0-9][A-Za-z0-9.-]{0,252}\Z" SAFE_SSH_TARGET_RE = re.compile(r"(?:[A-Za-z0-9][A-Za-z0-9._-]{0,63}@)?[A-Za-z0-9][A-Za-z0-9.-]{0,252}\Z")
) SNAPSHOT_ID_RE = re.compile(r"[0-9A-F]+-[0-9A-F]+-[0-9]+\Z")
TXID_SNAPSHOT_RE = re.compile(r"[0-9]+:[0-9]+:(?:[0-9]+(?:,[0-9]+)*)?\Z")
WAL_LSN_RE = re.compile(r"[0-9A-F]+/[0-9A-F]+\Z")
SYSTEM_IDENTIFIER_RE = re.compile(r"[0-9]+\Z")
SOURCE_SNAPSHOT_RECEIPT_SCHEMA = "livingip.sourceSnapshotReceipt.v2"
EXPECTED_FILES = { EXPECTED_FILES = {
"teleo-canonical.dump", "teleo-canonical.dump",
"teleo-canonical.dump.sha256", "teleo-canonical.dump.sha256",
"teleo-canonical.toc", "teleo-canonical.toc",
"source-manifest.jsonl", "source-manifest.jsonl",
"source-context.txt", "source-context.txt",
"source-snapshot.json",
"service-before.txt", "service-before.txt",
"service-after.txt", "service-after.txt",
} }
@ -47,6 +52,81 @@ def sha256(path: Path) -> str:
return digest.hexdigest() return digest.hexdigest()
def sha256_bytes(payload: bytes) -> str:
return hashlib.sha256(payload).hexdigest()
def canonical_sha256(value: Any) -> str:
payload = json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True)
return sha256_bytes(payload.encode())
def load_snapshot_metadata(path: Path) -> dict[str, str]:
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
raise RuntimeError("source snapshot metadata is unreadable") from exc
if not isinstance(payload, dict):
raise RuntimeError("source snapshot metadata must be an object")
fields = {
"exported_snapshot_id": SNAPSHOT_ID_RE,
"txid_snapshot": TXID_SNAPSHOT_RE,
"wal_lsn": WAL_LSN_RE,
"system_identifier": SYSTEM_IDENTIFIER_RE,
}
normalized: dict[str, str] = {}
for field, pattern in fields.items():
value = str(payload.get(field) or "")
if not pattern.fullmatch(value):
raise RuntimeError(f"source snapshot metadata has invalid {field}")
normalized[field] = value
captured_at_utc = str(payload.get("captured_at_utc") or "")
if not captured_at_utc or any(character in captured_at_utc for character in "\r\n\x00"):
raise RuntimeError("source snapshot metadata has invalid captured_at_utc")
normalized["captured_at_utc"] = captured_at_utc
return normalized
def load_service_state(path: Path) -> dict[str, str]:
try:
state = dict(line.split("=", 1) for line in path.read_text(encoding="utf-8").splitlines() if "=" in line)
main_pid = int(state.get("MainPID") or 0)
restarts = int(state.get("NRestarts") or 0)
except (OSError, TypeError, ValueError) as exc:
raise RuntimeError("source service state is unreadable") from exc
if state.get("ActiveState") != "active" or state.get("SubState") != "running" or main_pid <= 0 or restarts < 0:
raise RuntimeError("source service is not active/running with a positive MainPID")
return state
def build_provenance_binding(
*,
run_id: str,
authorization_ref: str,
source: dict[str, str],
snapshot: dict[str, str],
dump_sha256: str,
manifest_sql_sha256: str,
source_manifest_sha256: str,
source_context_sha256: str,
) -> dict[str, Any]:
payload = {
"receipt_schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"run_id": run_id,
"capture_authorization_ref": authorization_ref,
"source": source,
"exported_snapshot_id": snapshot["exported_snapshot_id"],
"txid_snapshot": snapshot["txid_snapshot"],
"wal_lsn": snapshot["wal_lsn"],
"source_system_identifier": snapshot["system_identifier"],
"dump_sha256": dump_sha256,
"manifest_sql_sha256": manifest_sql_sha256,
"source_manifest_sha256": source_manifest_sha256,
"source_context_sha256": source_context_sha256,
}
return {"algorithm": "sha256", "payload": payload, "sha256": canonical_sha256(payload)}
def ssh_base(identity_file: Path) -> list[str]: def ssh_base(identity_file: Path) -> list[str]:
return [ return [
"ssh", "ssh",
@ -73,12 +153,12 @@ def remote_capture_script(*, run_id: str, container: str, database: str, service
"container_dump": shlex.quote(container_dump), "container_dump": shlex.quote(container_dump),
} }
return f"""set -euo pipefail return f"""set -euo pipefail
remote_root={quoted['remote_root']} remote_root={quoted["remote_root"]}
container={quoted['container']} container={quoted["container"]}
database={quoted['database']} database={quoted["database"]}
service={quoted['service']} service={quoted["service"]}
container_manifest={quoted['container_manifest']} container_manifest={quoted["container_manifest"]}
container_dump={quoted['container_dump']} container_dump={quoted["container_dump"]}
cleanup() {{ cleanup() {{
docker exec "$container" rm -f "$container_manifest" "$container_dump" >/dev/null 2>&1 || true docker exec "$container" rm -f "$container_manifest" "$container_dump" >/dev/null 2>&1 || true
@ -86,11 +166,23 @@ cleanup() {{
}} }}
trap cleanup EXIT trap cleanup EXIT
assert_service_healthy() {{
local state_file="$1"
grep -qx 'ActiveState=active' "$state_file"
grep -qx 'SubState=running' "$state_file"
local main_pid restarts
main_pid="$(sed -n 's/^MainPID=//p' "$state_file")"
restarts="$(sed -n 's/^NRestarts=//p' "$state_file")"
[[ "$main_pid" =~ ^[1-9][0-9]*$ ]]
[[ "$restarts" =~ ^[0-9]+$ ]]
}}
test -d "$remote_root" test -d "$remote_root"
test -f "$remote_root/postgres-parity-manifest.sql" test -f "$remote_root/postgres-parity-manifest.sql"
docker inspect "$container" >/dev/null docker inspect "$container" >/dev/null
docker exec "$container" pg_isready -U postgres -d "$database" >/dev/null docker exec "$container" pg_isready -U postgres -d "$database" >/dev/null
systemctl show "$service" -p ActiveState -p SubState -p MainPID -p NRestarts -p ActiveEnterTimestamp --no-pager > "$remote_root/service-before.txt" systemctl show "$service" -p ActiveState -p SubState -p MainPID -p NRestarts -p ActiveEnterTimestamp --no-pager > "$remote_root/service-before.txt"
assert_service_healthy "$remote_root/service-before.txt"
docker inspect "$container" --format 'container_id={{{{.Id}}}}\nrunning={{{{.State.Running}}}}\nnetwork_mode={{{{.HostConfig.NetworkMode}}}}' > "$remote_root/source-context.txt" docker inspect "$container" --format 'container_id={{{{.Id}}}}\nrunning={{{{.State.Running}}}}\nnetwork_mode={{{{.HostConfig.NetworkMode}}}}' > "$remote_root/source-context.txt"
printf 'database=%s\ncaptured_at_utc=%s\n' "$database" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$remote_root/source-context.txt" printf 'database=%s\ncaptured_at_utc=%s\n' "$database" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$remote_root/source-context.txt"
docker cp "$remote_root/postgres-parity-manifest.sql" "$container:$container_manifest" >/dev/null docker cp "$remote_root/postgres-parity-manifest.sql" "$container:$container_manifest" >/dev/null
@ -107,6 +199,19 @@ if [[ ! "$snapshot_id" =~ ^[0-9A-F]+-[0-9A-F]+-[0-9]+$ ]]; then
echo "invalid exported snapshot id" >&2 echo "invalid exported snapshot id" >&2
exit 1 exit 1
fi fi
printf '%s\n' "select jsonb_build_object(
'exported_snapshot_id', '$snapshot_id',
'txid_snapshot', txid_current_snapshot()::text,
'wal_lsn', pg_current_wal_lsn()::text,
'system_identifier', (select system_identifier::text from pg_control_system()),
'captured_at_utc', clock_timestamp()
)::text;" >&3
IFS= read -r snapshot_metadata <&4
if [[ -z "$snapshot_metadata" ]]; then
echo "invalid source snapshot metadata" >&2
exit 1
fi
printf '%s\n' "$snapshot_metadata" > "$remote_root/source-snapshot.json"
docker exec "$container" pg_dump \ docker exec "$container" pg_dump \
-U postgres -d "$database" \ -U postgres -d "$database" \
@ -130,6 +235,7 @@ docker cp "$container:$container_dump" "$remote_root/teleo-canonical.dump" >/dev
sha256sum "$remote_root/teleo-canonical.dump" > "$remote_root/teleo-canonical.dump.sha256" sha256sum "$remote_root/teleo-canonical.dump" > "$remote_root/teleo-canonical.dump.sha256"
printf 'snapshot_exported=true\n' >> "$remote_root/source-context.txt" printf 'snapshot_exported=true\n' >> "$remote_root/source-context.txt"
systemctl show "$service" -p ActiveState -p SubState -p MainPID -p NRestarts -p ActiveEnterTimestamp --no-pager > "$remote_root/service-after.txt" systemctl show "$service" -p ActiveState -p SubState -p MainPID -p NRestarts -p ActiveEnterTimestamp --no-pager > "$remote_root/service-after.txt"
assert_service_healthy "$remote_root/service-after.txt"
cmp -s "$remote_root/service-before.txt" "$remote_root/service-after.txt" cmp -s "$remote_root/service-before.txt" "$remote_root/service-after.txt"
tar -C "$remote_root" -cf - \ tar -C "$remote_root" -cf - \
@ -138,12 +244,13 @@ tar -C "$remote_root" -cf - \
teleo-canonical.toc \ teleo-canonical.toc \
source-manifest.jsonl \ source-manifest.jsonl \
source-context.txt \ source-context.txt \
source-snapshot.json \
service-before.txt \ service-before.txt \
service-after.txt service-after.txt
""" """
def upload_manifest(ssh: list[str], target: str, remote_root: str, manifest: Path) -> None: def upload_manifest(ssh: list[str], target: str, remote_root: str, manifest: bytes) -> None:
command = ( command = (
f"umask 077; rm -rf {shlex.quote(remote_root)}; " f"umask 077; rm -rf {shlex.quote(remote_root)}; "
f"mkdir -m 700 {shlex.quote(remote_root)}; " f"mkdir -m 700 {shlex.quote(remote_root)}; "
@ -151,7 +258,7 @@ def upload_manifest(ssh: list[str], target: str, remote_root: str, manifest: Pat
) )
completed = subprocess.run( completed = subprocess.run(
[*ssh, target, "--", command], [*ssh, target, "--", command],
input=manifest.read_bytes(), input=manifest,
capture_output=True, capture_output=True,
check=False, check=False,
) )
@ -206,8 +313,10 @@ def capture(args: argparse.Namespace) -> dict[str, Any]:
output_dir.mkdir(mode=0o700) output_dir.mkdir(mode=0o700)
remote_root = f"/tmp/teleo-canonical-snapshot-{args.run_id}" remote_root = f"/tmp/teleo-canonical-snapshot-{args.run_id}"
ssh = ssh_base(args.ssh_key.resolve()) ssh = ssh_base(args.ssh_key.resolve())
manifest_sql = args.manifest.resolve().read_bytes()
manifest_sql_sha256 = sha256_bytes(manifest_sql)
try: try:
upload_manifest(ssh, args.ssh_target, remote_root, args.manifest.resolve()) upload_manifest(ssh, args.ssh_target, remote_root, manifest_sql)
script = remote_capture_script( script = remote_capture_script(
run_id=args.run_id, run_id=args.run_id,
container=args.container, container=args.container,
@ -233,19 +342,38 @@ def capture(args: argparse.Namespace) -> dict[str, Any]:
if expected_sha != actual_sha: if expected_sha != actual_sha:
raise RuntimeError("captured dump SHA-256 mismatch") raise RuntimeError("captured dump SHA-256 mismatch")
manifest = load_manifest(output_dir / "source-manifest.jsonl") manifest = load_manifest(output_dir / "source-manifest.jsonl")
before = (output_dir / "service-before.txt").read_text() snapshot = load_snapshot_metadata(output_dir / "source-snapshot.json")
after = (output_dir / "service-after.txt").read_text() before = load_service_state(output_dir / "service-before.txt")
return { after = load_service_state(output_dir / "service-after.txt")
"artifact": "vps_canonical_postgres_exported_snapshot", source = {
"generated_at_utc": datetime.now(UTC).isoformat(),
"status": "pass",
"source": {
"ssh_target": args.ssh_target, "ssh_target": args.ssh_target,
"container": args.container, "container": args.container,
"database": args.database, "database": args.database,
"service": args.service, "service": args.service,
}, }
source_manifest_sha256 = sha256(output_dir / "source-manifest.jsonl")
source_context_sha256 = sha256(output_dir / "source-context.txt")
provenance_binding = build_provenance_binding(
run_id=args.run_id,
authorization_ref=args.authorization_ref,
source=source,
snapshot=snapshot,
dump_sha256=actual_sha,
manifest_sql_sha256=manifest_sql_sha256,
source_manifest_sha256=source_manifest_sha256,
source_context_sha256=source_context_sha256,
)
return {
"artifact": "vps_canonical_postgres_exported_snapshot",
"schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"receipt_version": 2,
"generated_at_utc": datetime.now(UTC).isoformat(),
"status": "pass",
"run_id": args.run_id,
"capture_authorization_ref": args.authorization_ref,
"source": source,
"snapshot_exported": True, "snapshot_exported": True,
"snapshot": snapshot,
"dump": { "dump": {
"path": str(dump_path), "path": str(dump_path),
"bytes": dump_path.stat().st_size, "bytes": dump_path.stat().st_size,
@ -254,10 +382,18 @@ def capture(args: argparse.Namespace) -> dict[str, Any]:
}, },
"manifest": { "manifest": {
"path": str(output_dir / "source-manifest.jsonl"), "path": str(output_dir / "source-manifest.jsonl"),
"sha256": source_manifest_sha256,
"manifest_sql_sha256": manifest_sql_sha256,
"table_count": len(manifest["tables"]), "table_count": len(manifest["tables"]),
"total_rows": sum(int(row["row_count"]) for row in manifest["tables"].values()), "total_rows": sum(int(row["row_count"]) for row in manifest["tables"].values()),
"application_roles": manifest["singleton"]["application_roles"]["items"], "application_roles": manifest["singleton"]["application_roles"]["items"],
}, },
"source_context": {
"path": str(output_dir / "source-context.txt"),
"sha256": source_context_sha256,
},
"source_service": {"before": before, "after": after, "unchanged": before == after},
"provenance_binding": provenance_binding,
"service_unchanged": before == after, "service_unchanged": before == after,
"production_db_mutated": False, "production_db_mutated": False,
"telegram_send_attempted": False, "telegram_send_attempted": False,
@ -269,7 +405,7 @@ def cleanup_failed_output(output_dir: Path, *, preexisting: bool) -> None:
shutil.rmtree(output_dir) shutil.rmtree(output_dir)
def parse_args() -> argparse.Namespace: def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__) parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--execute", action="store_true") parser.add_argument("--execute", action="store_true")
parser.add_argument("--ssh-target", required=True) parser.add_argument("--ssh-target", required=True)
@ -279,9 +415,10 @@ def parse_args() -> argparse.Namespace:
parser.add_argument("--service", default="leoclean-gateway.service") parser.add_argument("--service", default="leoclean-gateway.service")
parser.add_argument("--manifest", default=Path("ops/postgres_parity_manifest.sql"), type=Path) parser.add_argument("--manifest", default=Path("ops/postgres_parity_manifest.sql"), type=Path)
parser.add_argument("--run-id", required=True) parser.add_argument("--run-id", required=True)
parser.add_argument("--authorization-ref", required=True)
parser.add_argument("--output-dir", required=True, type=Path) parser.add_argument("--output-dir", required=True, type=Path)
parser.add_argument("--timeout", default=180, type=int) parser.add_argument("--timeout", default=180, type=int)
args = parser.parse_args() args = parser.parse_args(argv)
for value, flag in ((args.container, "--container"), (args.database, "--database"), (args.service, "--service")): for value, flag in ((args.container, "--container"), (args.database, "--database"), (args.service, "--service")):
if not SAFE_NAME_RE.fullmatch(value): if not SAFE_NAME_RE.fullmatch(value):
parser.error(f"{flag} contains unsafe characters") parser.error(f"{flag} contains unsafe characters")
@ -289,6 +426,8 @@ def parse_args() -> argparse.Namespace:
parser.error("--ssh-target must be a host or user@host without options") parser.error("--ssh-target must be a host or user@host without options")
if not SAFE_RUN_RE.fullmatch(args.run_id): if not SAFE_RUN_RE.fullmatch(args.run_id):
parser.error("--run-id must be 8-64 safe characters") parser.error("--run-id must be 8-64 safe characters")
if not SAFE_AUTHORIZATION_REF_RE.fullmatch(args.authorization_ref):
parser.error("--authorization-ref must be 8-256 safe metadata characters")
if not args.ssh_key.is_file(): if not args.ssh_key.is_file():
parser.error("--ssh-key must be an existing file") parser.error("--ssh-key must be an existing file")
if not args.manifest.is_file(): if not args.manifest.is_file():
@ -298,8 +437,12 @@ def parse_args() -> argparse.Namespace:
json.dumps( json.dumps(
{ {
"artifact": "vps_canonical_postgres_exported_snapshot_plan", "artifact": "vps_canonical_postgres_exported_snapshot_plan",
"schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"status": "dry_run", "status": "dry_run",
"run_id": args.run_id,
"capture_authorization_ref": args.authorization_ref,
"source": f"{args.ssh_target}:{args.container}/{args.database}", "source": f"{args.ssh_target}:{args.container}/{args.database}",
"manifest_sql_sha256": sha256(args.manifest.resolve()),
"output_dir": str(args.output_dir), "output_dir": str(args.output_dir),
"mutates_production_db": False, "mutates_production_db": False,
}, },
@ -321,8 +464,11 @@ def main() -> int:
cleanup_failed_output(output_dir, preexisting=output_preexisted) cleanup_failed_output(output_dir, preexisting=output_preexisted)
payload = { payload = {
"artifact": "vps_canonical_postgres_exported_snapshot", "artifact": "vps_canonical_postgres_exported_snapshot",
"schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"generated_at_utc": datetime.now(UTC).isoformat(), "generated_at_utc": datetime.now(UTC).isoformat(),
"status": "fail", "status": "fail",
"run_id": args.run_id,
"capture_authorization_ref": args.authorization_ref,
"error": str(exc), "error": str(exc),
} }
print(json.dumps(payload, indent=2, sort_keys=True)) print(json.dumps(payload, indent=2, sort_keys=True))

34
ops/private_receipt_io.py Normal file
View file

@ -0,0 +1,34 @@
"""Atomic private JSON receipt output shared by GCP parity helpers."""
from __future__ import annotations
import json
import os
import uuid
from pathlib import Path
from typing import Any
def write_private_json(path: Path, payload: dict[str, Any]) -> None:
"""Write one JSON receipt as mode 0600 without chmodding existing parents."""
parent_preexisting = path.parent.exists()
path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
if not path.parent.is_dir() or path.parent.is_symlink():
raise ValueError("output parent must be a real directory")
if not parent_preexisting:
os.chmod(path.parent, 0o700)
temporary = path.parent / f".{path.name}.{uuid.uuid4().hex}.tmp"
descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
try:
with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
json.dump(payload, handle, indent=2, sort_keys=True)
handle.write("\n")
handle.flush()
os.fsync(handle.fileno())
os.replace(temporary, path)
os.chmod(path, 0o600)
finally:
try:
temporary.unlink()
except FileNotFoundError:
pass

View file

@ -16,10 +16,13 @@ import ipaddress
import json import json
import os import os
import re import re
import shlex
import signal import signal
import stat import stat
import subprocess import subprocess
import time import time
import urllib.error
import urllib.request
import uuid import uuid
from datetime import UTC, datetime from datetime import UTC, datetime
from pathlib import Path from pathlib import Path
@ -33,17 +36,34 @@ except ImportError: # pragma: no cover - direct script execution on the GCP VM
DEFAULT_HOST = "10.61.0.3" DEFAULT_HOST = "10.61.0.3"
DEFAULT_PROJECT = "teleo-501523" DEFAULT_PROJECT = "teleo-501523"
DEFAULT_INSTANCE = "teleo-pgvector-standby"
DEFAULT_PRIVATE_NETWORK = "projects/teleo-501523/global/networks/teleo-staging-net"
DEFAULT_COMPUTE_INSTANCE = "teleo-prod-1"
DEFAULT_COMPUTE_ZONE = "europe-west6-a"
DEFAULT_SOURCE_SSH_TARGET = "root@77.42.65.182"
DEFAULT_SOURCE_CONTAINER = "teleo-pg"
DEFAULT_SOURCE_DATABASE = "teleo"
DEFAULT_SOURCE_SERVICE = "leoclean-gateway.service"
DEFAULT_SECRET = "gcp-teleo-pgvector-standby-postgres-password" DEFAULT_SECRET = "gcp-teleo-pgvector-standby-postgres-password"
DEFAULT_SERVICE = "leoclean-gcp-prod-parallel.service" DEFAULT_SERVICE = "leoclean-gcp-prod-parallel.service"
DEFAULT_RUN_ROOT = Path("/var/lib/teleo-gcp-restore-runs") DEFAULT_RUN_ROOT = Path("/var/lib/teleo-gcp-restore-runs")
DEFAULT_MANIFEST_SQL = Path(__file__).with_name("postgres_parity_manifest.sql") DEFAULT_MANIFEST_SQL = Path(__file__).with_name("postgres_parity_manifest.sql")
REVIEWED_MANIFEST_SQL_SHA256 = "8b8cdc25d54fdd8de05eb38c6e4423d2836953eb6012d4545f5c9c71b5f0150a" REVIEWED_MANIFEST_SQL_SHA256 = "8b8cdc25d54fdd8de05eb38c6e4423d2836953eb6012d4545f5c9c71b5f0150a"
SOURCE_SNAPSHOT_RECEIPT_SCHEMA = "livingip.sourceSnapshotReceipt.v2"
TARGET_DATABASE_RE = re.compile(r"teleo_clone_[a-z0-9][a-z0-9_]{0,50}\Z") TARGET_DATABASE_RE = re.compile(r"teleo_clone_[a-z0-9][a-z0-9_]{0,50}\Z")
RUN_ID_RE = re.compile(r"gcp-restore-[a-z0-9][a-z0-9-]{7,47}\Z") RUN_ID_RE = re.compile(r"gcp-restore-[a-z0-9][a-z0-9-]{7,47}\Z")
SHA256_RE = re.compile(r"[0-9a-f]{64}\Z") SHA256_RE = re.compile(r"[0-9a-f]{64}\Z")
LOCALE_RE = re.compile(r"[A-Za-z0-9_.@-]{1,64}\Z") LOCALE_RE = re.compile(r"[A-Za-z0-9_.@-]{1,64}\Z")
CLOUDSQL_INSTANCE_RE = re.compile(r"[a-z][a-z0-9-]{0,97}[a-z0-9]\Z")
AUTHORIZATION_REF_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.:/@+-]{7,255}\Z")
PRIVATE_NETWORK_RE = re.compile(r"projects/[a-z][a-z0-9-]{4,28}[a-z0-9]/global/networks/[a-z][a-z0-9-]{0,61}[a-z0-9]\Z")
GCE_INSTANCE_RE = re.compile(r"[a-z](?:[a-z0-9-]{0,61}[a-z0-9])?\Z")
GCE_ZONE_RE = re.compile(r"[a-z]+-[a-z]+[0-9]-[a-z]\Z")
SAFE_NAME_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}\Z")
SAFE_SSH_TARGET_RE = re.compile(r"(?:[A-Za-z0-9][A-Za-z0-9._-]{0,63}@)?[A-Za-z0-9][A-Za-z0-9.-]{0,252}\Z")
PG_RESTORE_VERSION_RE = re.compile(r"pg_restore \(PostgreSQL\) (?P<major>[0-9]+)(?:\.[0-9]+)*") PG_RESTORE_VERSION_RE = re.compile(r"pg_restore \(PostgreSQL\) (?P<major>[0-9]+)(?:\.[0-9]+)*")
ROLLBACK_DATABASE = "teleo_canonical_pre_20260712t1905z" ROLLBACK_DATABASE = "teleo_canonical_pre_20260712t1905z"
GCE_METADATA_ROOT = "http://metadata.google.internal/computeMetadata/v1"
class RestoreError(RuntimeError): class RestoreError(RuntimeError):
@ -66,6 +86,11 @@ def sha256_file(path: Path) -> str:
return digest.hexdigest() return digest.hexdigest()
def canonical_sha256(value: Any) -> str:
payload = json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True)
return hashlib.sha256(payload.encode()).hexdigest()
def validate_regular_file(path: Path, label: str) -> Path: def validate_regular_file(path: Path, label: str) -> Path:
try: try:
mode = path.lstat().st_mode mode = path.lstat().st_mode
@ -89,6 +114,104 @@ def validate_custom_dump(path: Path, expected_sha256: str) -> dict[str, Any]:
return {"path": str(path), "bytes": path.stat().st_size, "sha256": actual_sha256} return {"path": str(path), "bytes": path.stat().st_size, "sha256": actual_sha256}
def validate_source_snapshot_receipt(
path: Path,
*,
expected_receipt_sha256: str,
expected_authorization_ref: str,
expected_source: dict[str, str],
dump: dict[str, Any],
source_manifest_path: Path,
manifest_sql_path: Path,
source_manifest: dict[str, Any],
) -> dict[str, Any]:
path = validate_regular_file(path, "source snapshot receipt")
receipt_sha256 = sha256_file(path)
if receipt_sha256 != expected_receipt_sha256:
raise RestoreError("source snapshot receipt SHA-256 does not match --expected-source-receipt-sha256")
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except json.JSONDecodeError as exc:
raise RestoreError("source snapshot receipt is invalid JSON") from exc
if not isinstance(payload, dict):
raise RestoreError("source snapshot receipt must be a JSON object")
snapshot = payload.get("snapshot") or {}
manifest = payload.get("manifest") or {}
source = payload.get("source") or {}
source_context = payload.get("source_context") or {}
source_service = payload.get("source_service") or {}
binding = payload.get("provenance_binding") or {}
binding_payload = binding.get("payload") or {}
source_manifest_sha256 = sha256_file(source_manifest_path)
manifest_sql_sha256 = sha256_file(manifest_sql_path)
expected_table_count = len(source_manifest["tables"])
expected_total_rows = sum(int(row["row_count"]) for row in source_manifest["tables"].values())
expected_binding_payload = {
"receipt_schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"run_id": payload.get("run_id"),
"capture_authorization_ref": payload.get("capture_authorization_ref"),
"source": source,
"exported_snapshot_id": snapshot.get("exported_snapshot_id"),
"txid_snapshot": snapshot.get("txid_snapshot"),
"wal_lsn": snapshot.get("wal_lsn"),
"source_system_identifier": snapshot.get("system_identifier"),
"dump_sha256": dump["sha256"],
"manifest_sql_sha256": manifest_sql_sha256,
"source_manifest_sha256": source_manifest_sha256,
"source_context_sha256": source_context.get("sha256"),
}
checks = {
"artifact": payload.get("artifact") == "vps_canonical_postgres_exported_snapshot",
"schema": payload.get("schema") == SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"receipt_version": payload.get("receipt_version") == 2,
"status": payload.get("status") == "pass",
"snapshot_exported": payload.get("snapshot_exported") is True,
"service_unchanged": payload.get("service_unchanged") is True,
"source_service_healthy": source_service.get("unchanged") is True
and service_state_is_healthy(source_service.get("before"))
and source_service.get("before") == source_service.get("after"),
"source_not_mutated": payload.get("production_db_mutated") is False,
"telegram_not_sent": payload.get("telegram_send_attempted") is False,
"run_id": bool(payload.get("run_id")),
"capture_authorization_ref": payload.get("capture_authorization_ref") == expected_authorization_ref,
"source_identity": source == expected_source,
"source_database": source.get("database") == source_manifest["singleton"]["identity"].get("database"),
"snapshot_identity": all(
bool(snapshot.get(field))
for field in ("exported_snapshot_id", "txid_snapshot", "wal_lsn", "system_identifier", "captured_at_utc")
),
"dump_sha256": (payload.get("dump") or {}).get("sha256") == dump["sha256"],
"dump_bytes": (payload.get("dump") or {}).get("bytes") == dump["bytes"],
"source_manifest_sha256": manifest.get("sha256") == source_manifest_sha256,
"manifest_sql_sha256": manifest.get("manifest_sql_sha256") == manifest_sql_sha256,
"source_context_sha256": bool(SHA256_RE.fullmatch(str(source_context.get("sha256") or ""))),
"table_count": manifest.get("table_count") == expected_table_count,
"total_rows": manifest.get("total_rows") == expected_total_rows,
"binding_algorithm": binding.get("algorithm") == "sha256",
"binding_payload": binding_payload == expected_binding_payload,
"binding_sha256": binding.get("sha256") == canonical_sha256(expected_binding_payload),
}
failed = sorted(name for name, passed in checks.items() if not passed)
if failed:
raise RestoreError("source snapshot receipt failed validation: " + ", ".join(failed))
return {
"path": str(path),
"sha256": receipt_sha256,
"schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"run_id": payload["run_id"],
"capture_authorization_ref": payload["capture_authorization_ref"],
"snapshot": snapshot,
"provenance_binding_sha256": binding["sha256"],
"manifest_sha256": source_manifest_sha256,
"manifest_sql_sha256": manifest_sql_sha256,
"dump_sha256": dump["sha256"],
"table_count": expected_table_count,
"total_rows": expected_total_rows,
"checks": checks,
}
def _run( def _run(
command: list[str], command: list[str],
*, *,
@ -158,6 +281,163 @@ def service_state(service: str, *, timeout: float) -> dict[str, Any]:
return result return result
def validate_service_healthy(state: dict[str, Any], label: str) -> None:
if not service_state_is_healthy(state):
raise RestoreError(f"{label} service is not active/running with a positive MainPID")
def service_state_is_healthy(state: Any) -> bool:
if not isinstance(state, dict):
return False
try:
main_pid = int(state.get("MainPID") or 0)
restarts = int(state.get("NRestarts") or 0)
except (TypeError, ValueError):
return False
return (
state.get("ActiveState") == "active" and state.get("SubState") == "running" and main_pid > 0 and restarts >= 0
)
def cloudsql_control_plane_state(
project: str,
instance: str,
expected_host: str,
expected_private_network: str,
*,
timeout: float,
) -> dict[str, Any]:
completed = _run(
[
"gcloud",
"sql",
"instances",
"describe",
instance,
f"--project={project}",
"--format=json",
"--quiet",
],
timeout=timeout,
)
raw = _require_success(completed, "Cloud SQL control-plane readback")
try:
payload = json.loads(raw)
except json.JSONDecodeError as exc:
raise RestoreError("Cloud SQL control-plane readback returned invalid JSON") from exc
settings = payload.get("settings") or {}
ip_configuration = settings.get("ipConfiguration") or {}
addresses = payload.get("ipAddresses") or []
private_addresses = sorted(
str(row.get("ipAddress")) for row in addresses if row.get("type") == "PRIVATE" and row.get("ipAddress")
)
private_network = str(ip_configuration.get("privateNetwork") or "")
non_private_addresses = sorted(
str(row.get("ipAddress")) for row in addresses if row.get("type") != "PRIVATE" and row.get("ipAddress")
)
checks = {
"instance": payload.get("name") == instance,
"state": payload.get("state") == "RUNNABLE",
"postgres_16": str(payload.get("databaseVersion") or "").startswith("POSTGRES_16"),
"public_ip_disabled": ip_configuration.get("ipv4Enabled") is False,
"private_network": private_network == expected_private_network,
"expected_private_host": expected_host in private_addresses,
"no_non_private_address": not non_private_addresses,
}
failed = sorted(name for name, passed in checks.items() if not passed)
if failed:
raise RestoreError("Cloud SQL control-plane preflight failed: " + ", ".join(failed))
return {
"project": project,
"instance": instance,
"database_version": payload.get("databaseVersion"),
"region": payload.get("region"),
"state": payload.get("state"),
"tier": settings.get("tier"),
"private_network": private_network,
"expected_private_network": expected_private_network,
"private_addresses": private_addresses,
"expected_private_host": expected_host,
"public_ip_disabled": True,
"checks": checks,
}
def gce_metadata_value(path: str, *, timeout: float) -> str:
request = urllib.request.Request(
f"{GCE_METADATA_ROOT}/{path}",
headers={"Metadata-Flavor": "Google"},
method="GET",
)
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
try:
with opener.open(request, timeout=timeout) as response:
if response.headers.get("Metadata-Flavor") != "Google":
raise RestoreError("GCE metadata response is missing Metadata-Flavor: Google")
value = response.read(4096).decode("utf-8", errors="strict").strip()
except (OSError, UnicodeError, urllib.error.URLError) as exc:
raise RestoreError(f"GCE metadata readback failed for {path}") from exc
if not value or any(character in value for character in "\r\n\x00"):
raise RestoreError(f"GCE metadata readback is invalid for {path}")
return value
def gce_compute_identity(
project: str,
expected_instance: str,
expected_zone: str,
expected_private_network: str,
*,
timeout: float,
) -> dict[str, Any]:
metadata = {
"project": gce_metadata_value("project/project-id", timeout=timeout),
"project_number": gce_metadata_value("project/numeric-project-id", timeout=timeout),
"instance": gce_metadata_value("instance/name", timeout=timeout),
"instance_id": gce_metadata_value("instance/id", timeout=timeout),
"zone_resource": gce_metadata_value("instance/zone", timeout=timeout),
"network_resource": gce_metadata_value("instance/network-interfaces/0/network", timeout=timeout),
"private_ip": gce_metadata_value("instance/network-interfaces/0/ip", timeout=timeout),
"service_account": gce_metadata_value("instance/service-accounts/default/email", timeout=timeout),
}
zone = metadata["zone_resource"].rsplit("/", 1)[-1]
network_name = expected_private_network.rsplit("/", 1)[-1]
expected_network_resource = f"projects/{metadata['project_number']}/networks/{network_name}"
try:
private_ip = ipaddress.ip_address(metadata["private_ip"])
except ValueError as exc:
raise RestoreError("GCE metadata private IP is invalid") from exc
checks = {
"project": metadata["project"] == project,
"project_number": metadata["project_number"].isdigit(),
"expected_network_project": expected_private_network.startswith(f"projects/{project}/global/networks/"),
"instance": metadata["instance"] == expected_instance,
"instance_id": metadata["instance_id"].isdigit(),
"zone": zone == expected_zone,
"network": metadata["network_resource"] == expected_network_resource,
"private_ip": any(
private_ip in network
for network in (
ipaddress.ip_network("10.0.0.0/8"),
ipaddress.ip_network("172.16.0.0/12"),
ipaddress.ip_network("192.168.0.0/16"),
)
),
"service_account": metadata["service_account"].endswith(".iam.gserviceaccount.com"),
}
failed = sorted(name for name, passed in checks.items() if not passed)
if failed:
raise RestoreError("GCE compute identity preflight failed: " + ", ".join(failed))
return {
**metadata,
"zone": zone,
"expected_zone": expected_zone,
"expected_private_network": expected_private_network,
"expected_network_resource": expected_network_resource,
"checks": checks,
}
def resolve_password(project: str, secret: str, *, timeout: float) -> str: def resolve_password(project: str, secret: str, *, timeout: float) -> str:
completed = _run( completed = _run(
[ [
@ -310,7 +590,11 @@ def capture_target_manifest(args: argparse.Namespace, password: str) -> str:
return _require_success(completed, "target parity manifest capture", secrets=(password,)) return _require_success(completed, "target parity manifest capture", secrets=(password,))
def validate_private_identity(target_manifest: dict[str, Any], target_db: str) -> dict[str, Any]: def validate_private_identity(
target_manifest: dict[str, Any],
target_db: str,
source_compute: str,
) -> dict[str, Any]:
identity = target_manifest["singleton"]["identity"] identity = target_manifest["singleton"]["identity"]
if identity.get("database") != target_db: if identity.get("database") != target_db:
raise RestoreError("target manifest database does not match the generated clone") raise RestoreError("target manifest database does not match the generated clone")
@ -340,7 +624,7 @@ def validate_private_identity(target_manifest: dict[str, Any], target_db: str) -
"ssl_version": identity.get("ssl_version"), "ssl_version": identity.get("ssl_version"),
"private_connectivity": True, "private_connectivity": True,
"transaction_read_only": "on", "transaction_read_only": "on",
"source_compute": "teleo-prod-1", "source_compute": source_compute,
} }
@ -440,6 +724,22 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]:
"generated_at_utc": utc_now(), "generated_at_utc": utc_now(),
"run_id": args.run_id, "run_id": args.run_id,
"target_database": args.target_db, "target_database": args.target_db,
"execution_contract": {
"project": args.project,
"cloudsql_instance": args.cloudsql_instance,
"host": args.host,
"private_network": args.expected_private_network,
"compute_instance": args.expected_compute_instance,
"compute_zone": args.expected_compute_zone,
"source": {
"ssh_target": args.expected_source_ssh_target,
"container": args.expected_source_container,
"database": args.expected_source_database,
"service": args.expected_source_service,
},
"password_secret": args.password_secret,
"service": args.service,
},
"status": "running", "status": "running",
"phase": "validation", "phase": "validation",
"source": {}, "source": {},
@ -475,12 +775,39 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]:
"sha256": sha256_file(args.source_manifest), "sha256": sha256_file(args.source_manifest),
} }
source_manifest = load_manifest(args.source_manifest) source_manifest = load_manifest(args.source_manifest)
receipt["source"]["capture_receipt"] = validate_source_snapshot_receipt(
args.source_receipt,
expected_receipt_sha256=args.expected_source_receipt_sha256,
expected_authorization_ref=args.expected_capture_authorization_ref,
expected_source=receipt["execution_contract"]["source"],
dump=receipt["source"]["dump"],
source_manifest_path=args.source_manifest,
manifest_sql_path=args.manifest_sql,
source_manifest=source_manifest,
)
receipt["toolchain"] = validate_pg_restore_version( receipt["toolchain"] = validate_pg_restore_version(
args.pg_restore_bin, args.pg_restore_bin,
source_manifest, source_manifest,
timeout=args.command_timeout, timeout=args.command_timeout,
) )
receipt["target"]["compute"] = gce_compute_identity(
args.project,
args.expected_compute_instance,
args.expected_compute_zone,
args.expected_private_network,
timeout=args.command_timeout,
)
receipt["live_service"]["before"] = service_state(args.service, timeout=args.command_timeout) receipt["live_service"]["before"] = service_state(args.service, timeout=args.command_timeout)
validate_service_healthy(receipt["live_service"]["before"], "pre-restore")
receipt["phase"] = "cloudsql_control_plane_preflight"
receipt["target"]["cloudsql"] = cloudsql_control_plane_state(
args.project,
args.cloudsql_instance,
args.host,
args.expected_private_network,
timeout=args.command_timeout,
)
receipt["phase"] = "credential_resolution" receipt["phase"] = "credential_resolution"
password = resolve_password(args.project, args.password_secret, timeout=args.command_timeout) password = resolve_password(args.project, args.password_secret, timeout=args.command_timeout)
@ -506,9 +833,46 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]:
"manifest_path": str(target_manifest_path), "manifest_path": str(target_manifest_path),
"manifest_bytes": target_manifest_path.stat().st_size, "manifest_bytes": target_manifest_path.stat().st_size,
"manifest_sha256": sha256_file(target_manifest_path), "manifest_sha256": sha256_file(target_manifest_path),
"connectivity": validate_private_identity(target_manifest, args.target_db), "connectivity": validate_private_identity(
target_manifest,
args.target_db,
receipt["target"]["compute"]["instance"],
),
} }
) )
connectivity = receipt["target"]["connectivity"]
cloudsql = receipt["target"]["cloudsql"]
compute = receipt["target"]["compute"]
connectivity_proof = {
"artifact": "gcp_private_postgres_connectivity",
"schema": "livingip.gcpPrivatePostgresConnectivity.v2",
"generated_at_utc": utc_now(),
"status": "pass",
"restore_run_id": args.run_id,
"target_database": args.target_db,
"project": cloudsql["project"],
"cloudsql_instance": cloudsql["instance"],
"private_network": cloudsql["private_network"],
"source_compute": compute["instance"],
"source_compute_instance_id": compute["instance_id"],
"source_compute_zone": compute["zone"],
"source_compute_private_ip": compute["private_ip"],
"server_address": connectivity["server_address"],
"server_port": connectivity["server_port"],
"ssl": connectivity["ssl"],
"ssl_version": connectivity["ssl_version"],
"private_connectivity": connectivity["private_connectivity"],
"public_ip_disabled": cloudsql["public_ip_disabled"],
"cloudsql_control_plane_checks": cloudsql["checks"],
"compute_identity_checks": compute["checks"],
}
connectivity_proof_path = run_dir / "gcp-private-connectivity.json"
write_private(connectivity_proof_path, json.dumps(connectivity_proof, indent=2, sort_keys=True) + "\n")
receipt["target"]["connectivity_proof"] = {
"path": str(connectivity_proof_path),
"sha256": sha256_file(connectivity_proof_path),
"schema": connectivity_proof["schema"],
}
receipt["phase"] = "parity_compare" receipt["phase"] = "parity_compare"
problems, details = compare_manifests( problems, details = compare_manifests(
@ -529,6 +893,7 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]:
receipt["phase"] = "service_and_rollback_readback" receipt["phase"] = "service_and_rollback_readback"
receipt["rollback_database"] = rollback_database_state(args, password) receipt["rollback_database"] = rollback_database_state(args, password)
receipt["live_service"]["after"] = service_state(args.service, timeout=args.command_timeout) receipt["live_service"]["after"] = service_state(args.service, timeout=args.command_timeout)
validate_service_healthy(receipt["live_service"]["after"], "post-restore")
receipt["live_service"]["unchanged"] = receipt["live_service"]["before"] == receipt["live_service"]["after"] receipt["live_service"]["unchanged"] = receipt["live_service"]["before"] == receipt["live_service"]["after"]
if not receipt["live_service"]["unchanged"]: if not receipt["live_service"]["unchanged"]:
raise RestoreError("live GCP service state changed during disposable restore") raise RestoreError("live GCP service state changed during disposable restore")
@ -539,9 +904,42 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]:
"required": True, "required": True,
"attempted": False, "attempted": False,
"clone_database_remaining": 1, "clone_database_remaining": 1,
"exact_command": ( "exact_command": shlex.join(
f"sudo python3 ops/restore_gcp_generated_postgres_snapshot.py cleanup --execute " [
f"--target-db {args.target_db} --run-id {args.run_id}" "sudo",
"python3",
"ops/restore_gcp_generated_postgres_snapshot.py",
"cleanup",
"--execute",
"--target-db",
args.target_db,
"--run-id",
args.run_id,
"--host",
args.host,
"--project",
args.project,
"--cloudsql-instance",
args.cloudsql_instance,
"--expected-private-network",
args.expected_private_network,
"--expected-compute-instance",
args.expected_compute_instance,
"--expected-compute-zone",
args.expected_compute_zone,
"--expected-source-ssh-target",
args.expected_source_ssh_target,
"--expected-source-container",
args.expected_source_container,
"--expected-source-database",
args.expected_source_database,
"--expected-source-service",
args.expected_source_service,
"--password-secret",
args.password_secret,
"--service",
args.service,
]
), ),
} }
except ( except (
@ -583,6 +981,7 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]:
except Exception as service_exc: except Exception as service_exc:
receipt["live_service"]["after_error"] = str(service_exc)[-2000:] receipt["live_service"]["after_error"] = str(service_exc)[-2000:]
receipt["duration_seconds"] = round(time.monotonic() - started, 6) receipt["duration_seconds"] = round(time.monotonic() - started, 6)
receipt["completed_at_utc"] = utc_now()
write_private(run_dir / "restore-receipt.json", json.dumps(receipt, indent=2, sort_keys=True) + "\n") write_private(run_dir / "restore-receipt.json", json.dumps(receipt, indent=2, sort_keys=True) + "\n")
password = "" password = ""
return receipt return receipt
@ -592,6 +991,7 @@ def run_cleanup(args: argparse.Namespace) -> dict[str, Any]:
run_dir = secure_run_dir(args.run_root, args.run_id, create=False) run_dir = secure_run_dir(args.run_root, args.run_id, create=False)
restore_receipt_path = run_dir / "restore-receipt.json" restore_receipt_path = run_dir / "restore-receipt.json"
restore_receipt_path = validate_regular_file(restore_receipt_path, "restore receipt") restore_receipt_path = validate_regular_file(restore_receipt_path, "restore receipt")
restore_receipt_sha256 = sha256_file(restore_receipt_path)
restore_receipt = json.loads(restore_receipt_path.read_text(encoding="utf-8")) restore_receipt = json.loads(restore_receipt_path.read_text(encoding="utf-8"))
if restore_receipt.get("status") != "pass": if restore_receipt.get("status") != "pass":
raise RestoreError("cleanup requires a passing restore receipt") raise RestoreError("cleanup requires a passing restore receipt")
@ -600,23 +1000,66 @@ def run_cleanup(args: argparse.Namespace) -> dict[str, Any]:
if restore_receipt.get("run_id") != args.run_id: if restore_receipt.get("run_id") != args.run_id:
raise RestoreError("cleanup run id does not match the retained restore receipt") raise RestoreError("cleanup run id does not match the retained restore receipt")
before = service_state(args.service, timeout=args.command_timeout) expected_contract = {
password = resolve_password(args.project, args.password_secret, timeout=args.command_timeout) "project": args.project,
try: "cloudsql_instance": args.cloudsql_instance,
dropped = drop_clone(args, password) "host": args.host,
rollback = rollback_database_state(args, password) "private_network": args.expected_private_network,
finally: "compute_instance": args.expected_compute_instance,
password = "" "compute_zone": args.expected_compute_zone,
after = service_state(args.service, timeout=args.command_timeout) "source": {
payload = { "ssh_target": args.expected_source_ssh_target,
"container": args.expected_source_container,
"database": args.expected_source_database,
"service": args.expected_source_service,
},
"password_secret": args.password_secret,
"service": args.service,
}
if restore_receipt.get("execution_contract") != expected_contract:
raise RestoreError("cleanup execution contract does not match the retained restore receipt")
restore_cloudsql = (restore_receipt.get("target") or {}).get("cloudsql") or {}
restore_topology_checks = {
"project": restore_cloudsql.get("project") == args.project,
"instance": restore_cloudsql.get("instance") == args.cloudsql_instance,
"host": restore_cloudsql.get("expected_private_host") == args.host,
"private_network": restore_cloudsql.get("expected_private_network") == args.expected_private_network,
"public_ip_disabled": restore_cloudsql.get("public_ip_disabled") is True,
}
failed_topology_checks = sorted(name for name, passed in restore_topology_checks.items() if not passed)
if failed_topology_checks:
raise RestoreError(
"cleanup target topology does not match the retained restore receipt: " + ", ".join(failed_topology_checks)
)
restore_compute = (restore_receipt.get("target") or {}).get("compute") or {}
restore_compute_checks = {
"project": restore_compute.get("project") == args.project,
"instance": restore_compute.get("instance") == args.expected_compute_instance,
"zone": restore_compute.get("zone") == args.expected_compute_zone,
"private_network": restore_compute.get("expected_private_network") == args.expected_private_network,
"checks": bool(restore_compute.get("checks"))
and all(value is True for value in restore_compute["checks"].values()),
}
failed_compute_checks = sorted(name for name, passed in restore_compute_checks.items() if not passed)
if failed_compute_checks:
raise RestoreError(
"cleanup compute identity does not match the retained restore receipt: " + ", ".join(failed_compute_checks)
)
started = time.monotonic()
payload: dict[str, Any] = {
"artifact": "gcp_generated_postgres_snapshot_cleanup", "artifact": "gcp_generated_postgres_snapshot_cleanup",
"generated_at_utc": utc_now(), "generated_at_utc": utc_now(),
"run_id": args.run_id, "run_id": args.run_id,
"target_database": args.target_db, "target_database": args.target_db,
"status": "pass" if before == after and dropped["clone_database_remaining"] == 0 else "fail", "restore_receipt": {"path": str(restore_receipt_path), "sha256": restore_receipt_sha256},
"database": dropped, "execution_contract": expected_contract,
"rollback_database": rollback, "target": {},
"live_service": {"before": before, "after": after, "unchanged": before == after}, "status": "running",
"phase": "cloudsql_control_plane_preflight",
"database": {"clone_database_remaining": None},
"rollback_database": {},
"live_service": {},
"safety": { "safety": {
"live_database_named": False, "live_database_named": False,
"live_profile_modified": False, "live_profile_modified": False,
@ -624,7 +1067,94 @@ def run_cleanup(args: argparse.Namespace) -> dict[str, Any]:
"telegram_message_sent": False, "telegram_message_sent": False,
"secret_persisted": False, "secret_persisted": False,
}, },
"error": None,
} }
password = ""
try:
payload["target"]["cloudsql"] = cloudsql_control_plane_state(
args.project,
args.cloudsql_instance,
args.host,
args.expected_private_network,
timeout=args.command_timeout,
)
payload["target"]["compute"] = gce_compute_identity(
args.project,
args.expected_compute_instance,
args.expected_compute_zone,
args.expected_private_network,
timeout=args.command_timeout,
)
cleanup_compute_mismatches = sorted(
field
for field in (
"project",
"project_number",
"instance",
"instance_id",
"zone",
"network_resource",
"private_ip",
"service_account",
)
if payload["target"]["compute"].get(field) != restore_compute.get(field)
)
if cleanup_compute_mismatches:
raise RestoreError("cleanup GCE identity changed since restore: " + ", ".join(cleanup_compute_mismatches))
payload["phase"] = "service_readback_before"
payload["live_service"]["before"] = service_state(args.service, timeout=args.command_timeout)
validate_service_healthy(payload["live_service"]["before"], "pre-cleanup")
payload["phase"] = "credential_resolution"
password = resolve_password(args.project, args.password_secret, timeout=args.command_timeout)
payload["phase"] = "database_drop"
payload["database"] = drop_clone(args, password)
payload["phase"] = "rollback_readback"
payload["rollback_database"] = rollback_database_state(args, password)
payload["phase"] = "service_readback_after"
payload["live_service"]["after"] = service_state(args.service, timeout=args.command_timeout)
validate_service_healthy(payload["live_service"]["after"], "post-cleanup")
payload["live_service"]["unchanged"] = payload["live_service"]["before"] == payload["live_service"]["after"]
payload["status"] = (
"pass"
if payload["live_service"]["unchanged"] and payload["database"]["clone_database_remaining"] == 0
else "fail"
)
payload["phase"] = "complete"
except (
OSError,
ValueError,
KeyError,
TypeError,
json.JSONDecodeError,
subprocess.TimeoutExpired,
RestoreError,
KeyboardInterrupt,
) as exc:
payload["status"] = "fail"
payload["error"] = {
"phase": payload.get("phase"),
"type": type(exc).__name__,
"message": str(exc)[-2000:],
}
if password and payload["database"].get("clone_database_remaining") is None:
try:
payload["database"]["clone_database_remaining"] = (
1 if database_exists(args, password, args.target_db) else 0
)
except Exception as readback_exc:
payload["database"]["readback_error"] = str(readback_exc)[-2000:]
if "before" in payload["live_service"] and "after" not in payload["live_service"]:
try:
payload["live_service"]["after"] = service_state(args.service, timeout=args.command_timeout)
payload["live_service"]["unchanged"] = (
payload["live_service"]["before"] == payload["live_service"]["after"]
)
except Exception as service_exc:
payload["live_service"]["after_error"] = str(service_exc)[-2000:]
finally:
password = ""
payload["duration_seconds"] = round(time.monotonic() - started, 6)
payload["completed_at_utc"] = utc_now()
write_private(run_dir / "cleanup-receipt.json", json.dumps(payload, indent=2, sort_keys=True) + "\n") write_private(run_dir / "cleanup-receipt.json", json.dumps(payload, indent=2, sort_keys=True) + "\n")
return payload return payload
@ -635,6 +1165,14 @@ def add_common_args(parser: argparse.ArgumentParser) -> None:
parser.add_argument("--run-id", required=True) parser.add_argument("--run-id", required=True)
parser.add_argument("--host", default=DEFAULT_HOST) parser.add_argument("--host", default=DEFAULT_HOST)
parser.add_argument("--project", default=DEFAULT_PROJECT) parser.add_argument("--project", default=DEFAULT_PROJECT)
parser.add_argument("--cloudsql-instance", default=DEFAULT_INSTANCE)
parser.add_argument("--expected-private-network", default=DEFAULT_PRIVATE_NETWORK)
parser.add_argument("--expected-compute-instance", default=DEFAULT_COMPUTE_INSTANCE)
parser.add_argument("--expected-compute-zone", default=DEFAULT_COMPUTE_ZONE)
parser.add_argument("--expected-source-ssh-target", default=DEFAULT_SOURCE_SSH_TARGET)
parser.add_argument("--expected-source-container", default=DEFAULT_SOURCE_CONTAINER)
parser.add_argument("--expected-source-database", default=DEFAULT_SOURCE_DATABASE)
parser.add_argument("--expected-source-service", default=DEFAULT_SOURCE_SERVICE)
parser.add_argument("--password-secret", default=DEFAULT_SECRET) parser.add_argument("--password-secret", default=DEFAULT_SECRET)
parser.add_argument("--service", default=DEFAULT_SERVICE) parser.add_argument("--service", default=DEFAULT_SERVICE)
parser.add_argument("--command-timeout", type=float, default=120.0) parser.add_argument("--command-timeout", type=float, default=120.0)
@ -650,6 +1188,9 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
restore.add_argument("--dump", required=True, type=Path) restore.add_argument("--dump", required=True, type=Path)
restore.add_argument("--expected-dump-sha256", required=True) restore.add_argument("--expected-dump-sha256", required=True)
restore.add_argument("--source-manifest", required=True, type=Path) restore.add_argument("--source-manifest", required=True, type=Path)
restore.add_argument("--source-receipt", required=True, type=Path)
restore.add_argument("--expected-source-receipt-sha256", required=True)
restore.add_argument("--expected-capture-authorization-ref", required=True)
restore.add_argument("--manifest-sql", default=DEFAULT_MANIFEST_SQL, type=Path) restore.add_argument("--manifest-sql", default=DEFAULT_MANIFEST_SQL, type=Path)
restore.add_argument("--pg-restore-bin", default="pg_restore") restore.add_argument("--pg-restore-bin", default="pg_restore")
restore.add_argument("--restore-timeout", type=float, default=900.0) restore.add_argument("--restore-timeout", type=float, default=900.0)
@ -667,6 +1208,25 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser.error("--target-db must be a bounded lowercase teleo_clone_* identifier") parser.error("--target-db must be a bounded lowercase teleo_clone_* identifier")
if not RUN_ID_RE.fullmatch(args.run_id): if not RUN_ID_RE.fullmatch(args.run_id):
parser.error("--run-id must match gcp-restore-<8-48 lowercase letters, digits, or hyphens>") parser.error("--run-id must match gcp-restore-<8-48 lowercase letters, digits, or hyphens>")
if not CLOUDSQL_INSTANCE_RE.fullmatch(args.cloudsql_instance):
parser.error("--cloudsql-instance must be a safe lowercase Cloud SQL instance name")
if not PRIVATE_NETWORK_RE.fullmatch(args.expected_private_network):
parser.error("--expected-private-network must be a full projects/.../global/networks/... resource")
if not args.expected_private_network.startswith(f"projects/{args.project}/global/networks/"):
parser.error("--expected-private-network project must match --project")
if not GCE_INSTANCE_RE.fullmatch(args.expected_compute_instance):
parser.error("--expected-compute-instance must be a safe lowercase GCE instance name")
if not GCE_ZONE_RE.fullmatch(args.expected_compute_zone):
parser.error("--expected-compute-zone must be a safe GCE zone")
if not SAFE_SSH_TARGET_RE.fullmatch(args.expected_source_ssh_target):
parser.error("--expected-source-ssh-target must be a host or user@host without options")
for value, flag in (
(args.expected_source_container, "--expected-source-container"),
(args.expected_source_database, "--expected-source-database"),
(args.expected_source_service, "--expected-source-service"),
):
if not SAFE_NAME_RE.fullmatch(value):
parser.error(f"{flag} contains unsafe characters")
try: try:
host = ipaddress.ip_address(args.host) host = ipaddress.ip_address(args.host)
except ValueError: except ValueError:
@ -683,6 +1243,10 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
if args.operation == "restore": if args.operation == "restore":
if not SHA256_RE.fullmatch(args.expected_dump_sha256): if not SHA256_RE.fullmatch(args.expected_dump_sha256):
parser.error("--expected-dump-sha256 must be 64 lowercase hexadecimal characters") parser.error("--expected-dump-sha256 must be 64 lowercase hexadecimal characters")
if not SHA256_RE.fullmatch(args.expected_source_receipt_sha256):
parser.error("--expected-source-receipt-sha256 must be 64 lowercase hexadecimal characters")
if not AUTHORIZATION_REF_RE.fullmatch(args.expected_capture_authorization_ref):
parser.error("--expected-capture-authorization-ref must be 8-256 safe metadata characters")
for value, flag in ( for value, flag in (
(args.restore_timeout, "--restore-timeout"), (args.restore_timeout, "--restore-timeout"),
(args.manifest_timeout, "--manifest-timeout"), (args.manifest_timeout, "--manifest-timeout"),

View file

@ -0,0 +1,754 @@
#!/usr/bin/env python3
"""Verify one bound source-to-GCP restore, reasoning, and cleanup lifecycle."""
from __future__ import annotations
import argparse
import hashlib
import ipaddress
import json
import re
from datetime import UTC, datetime
from itertools import pairwise
from pathlib import Path
from typing import Any
try:
from .verify_vps_canonical_snapshot_delta import DELTA_SCHEMA, parse_timestamp, write_private_json
except ImportError: # pragma: no cover - direct script execution
from verify_vps_canonical_snapshot_delta import DELTA_SCHEMA, parse_timestamp, write_private_json
SOURCE_SCHEMA = "livingip.sourceSnapshotReceipt.v2"
BLIND_REASONING_SCHEMA = "livingip.gcpGeneratedDbBlindClaimCanary.v1"
REASONING_COMPUTE_SCHEMA = "livingip.gcpReasoningComputeAttestation.v1"
RETRIEVAL_RECEIPT_SCHEMA = "livingip.teleoKbRetrievalReceipt.v1"
UUID_RE = re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}\b", re.I)
SHA256_RE = re.compile(r"[0-9a-f]{64}\Z")
FORBIDDEN_HANDLE_RE = re.compile(r"\b(?:cory|m3ta)\b", re.I)
EXPECTED_BLIND_ROW_IDS = {
"2a7ae257-d01d-46f4-b813-63f81bb9c7c7",
"261c3532-fa32-47d8-a5b5-6cc45035c267",
"15740795-ecc6-40fa-9a01-3d6bc7c54f79",
}
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def load_receipt(path: Path, label: str) -> dict[str, Any]:
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
raise ValueError(f"{label} is unreadable or invalid JSON") from exc
if not isinstance(payload, dict):
raise ValueError(f"{label} must be a JSON object")
return payload
def is_rfc1918(value: Any) -> bool:
try:
address = ipaddress.ip_address(str(value))
except ValueError:
return False
return any(
address in network
for network in (
ipaddress.ip_network("10.0.0.0/8"),
ipaddress.ip_network("172.16.0.0/12"),
ipaddress.ip_network("192.168.0.0/16"),
)
)
def nonempty_all_true(value: Any) -> bool:
return isinstance(value, dict) and bool(value) and all(item is True for item in value.values())
def required_true_checks(value: Any, required: set[str]) -> bool:
return nonempty_all_true(value) and required <= set(value)
def service_healthy(value: Any) -> bool:
if not isinstance(value, dict):
return False
try:
main_pid = int(value.get("MainPID") or 0)
restarts = int(value.get("NRestarts") or 0)
except (TypeError, ValueError):
return False
return (
value.get("ActiveState") == "active" and value.get("SubState") == "running" and main_pid > 0 and restarts >= 0
)
def parity_evidence(details: dict[str, Any]) -> dict[str, bool]:
structural = details.get("structural_hashes") or {}
required_structures = {
"schemas",
"columns",
"constraints",
"indexes",
"sequences",
"views",
"functions",
"triggers",
"types",
"policies",
}
structural_valid = required_structures <= set(structural) and all(
bool(SHA256_RE.fullmatch(str((structural.get(kind) or {}).get("source") or "")))
and (structural.get(kind) or {}).get("source") == (structural.get(kind) or {}).get("target")
for kind in required_structures
)
performance = details.get("performance") or []
performance_valid = bool(performance) and all(
bool(row.get("query"))
and isinstance(row.get("source_ms"), (int, float))
and isinstance(row.get("target_ms"), (int, float))
and isinstance(row.get("source_ratio"), (int, float))
for row in performance
)
source_table_count = details.get("source_table_count")
target_table_count = details.get("target_table_count")
source_total_rows = details.get("source_total_rows")
target_total_rows = details.get("target_total_rows")
return {
"table_counts": isinstance(source_table_count, int)
and source_table_count > 0
and source_table_count == target_table_count,
"row_counts": isinstance(source_total_rows, int)
and source_total_rows > 0
and source_total_rows == target_total_rows,
"table_hashes": details.get("table_mismatches") == [],
"structural_hashes": structural_valid,
"roles": details.get("application_role_mismatches") == {}
and details.get("target_extra_application_roles") == [],
"extensions": details.get("required_extension_mismatches") == {},
"performance": performance_valid and details.get("performance_problems") == [],
}
def blind_reasoning_evidence(reasoning: dict[str, Any], target_db: Any) -> dict[str, bool]:
result = reasoning.get("result") or {}
gateway = result.get("gateway") or {}
child = gateway.get("child_process") or {}
tool_surface = gateway.get("tool_surface") or {}
tool_trace = result.get("database_tool_trace") or {}
calls = tool_trace.get("calls") or []
wrapper = reasoning.get("wrapper_tool_proof") or {}
invocations = wrapper.get("invocations") or []
prompt = str(reasoning.get("prompt") or "")
reply = str(result.get("reply") or "")
outcomes = reasoning.get("outcomes") or {}
required_outcomes = (
"asks_for_user_iteration_before_live",
"challenges_weak_support",
"decomposes_the_bundled_legal_claim",
"does_not_claim_a_database_change",
"proposes_candidate_claims",
"uses_only_m3taversal_handle",
)
required_subcommands = {"show", "evidence"}
discovery_subcommands = {"search", "context"}
traced_subcommands = {
str(invocation.get("subcommand")) for call in calls for invocation in (call.get("database_invocations") or [])
}
wrapper_subcommands = {str(invocation.get("subcommand")) for invocation in invocations}
retrieved_row_ids = {str(row_id) for call in calls for row_id in ((call.get("result") or {}).get("row_ids") or [])}
call_row_ids_valid = bool(calls) and all(
isinstance((call.get("result") or {}).get("row_ids"), list)
and isinstance((call.get("result") or {}).get("row_id_count"), int)
and (call.get("result") or {}).get("row_id_count") >= 0
and (call.get("result") or {}).get("row_id_count") >= len((call.get("result") or {}).get("row_ids") or [])
for call in calls
)
call_receipts_valid = bool(calls) and all(
(call.get("result") or {}).get("nonempty") is True
and (call.get("result") or {}).get("error_detected") is False
and ((call.get("result") or {}).get("retrieval_receipt") or {}).get("schema") == RETRIEVAL_RECEIPT_SCHEMA
and bool(
SHA256_RE.fullmatch(
str(((call.get("result") or {}).get("retrieval_receipt") or {}).get("semantic_context_sha256") or "")
)
)
and bool(
SHA256_RE.fullmatch(
str(((call.get("result") or {}).get("retrieval_receipt") or {}).get("artifact_state_sha256") or "")
)
)
and str(
((call.get("result") or {}).get("retrieval_receipt") or {}).get("read_consistency_status") or ""
).startswith("stable")
for call in calls
)
wrapper_invocations_valid = bool(invocations) and all(
invocation.get("database") == target_db
and invocation.get("returncode") == 0
and ((invocation.get("database_identity") or {}).get("transaction_read_only") == "on")
and ((invocation.get("database_identity") or {}).get("default_transaction_read_only") == "on")
and ((invocation.get("database_identity") or {}).get("ssl") is True)
and is_rfc1918((invocation.get("database_identity") or {}).get("server_address"))
for invocation in invocations
)
return {
"mode": reasoning.get("mode") == "gcp_generated_db_gatewayrunner_blind_claim_no_send",
"timestamps": bool(reasoning.get("generated_at_utc")) and bool(reasoning.get("completed_at_utc")),
"source_compute": bool(reasoning.get("source_compute")),
"blind_prompt": bool(prompt) and UUID_RE.search(prompt) is None,
"reply_nonempty": bool(reply.strip()),
"reply_uses_only_m3taversal_handle": FORBIDDEN_HANDLE_RE.search(reply) is None,
"outcomes": all(outcomes.get(name) is True for name in required_outcomes),
"gateway": gateway.get("authorized") is True
and gateway.get("handler_invoked") is True
and gateway.get("model_free_fallback_used") is False
and gateway.get("posted_to_telegram") is False,
"gateway_cleanup": child.get("exitcode") == 0
and child.get("alive_after_readback") is False
and child.get("process_group_alive_after_readback") is False
and child.get("timed_out") is False,
"send_tool_absent": tool_surface.get("send_message_tool_enabled") is False,
"tool_trace": tool_trace.get("schema") == "livingip.leoKbToolTrace.v1"
and tool_trace.get("database_tool_call_proven") is True
and tool_trace.get("database_retrieval_receipt_proven") is True
and tool_trace.get("database_tool_calls_read_only") is True
and tool_trace.get("database_tool_call_count") == len(calls)
and tool_trace.get("database_tool_completed_count") == len(calls),
"tool_receipts": call_receipts_valid,
"expected_claim_and_sources_retrieved": call_row_ids_valid and retrieved_row_ids >= EXPECTED_BLIND_ROW_IDS,
"required_tool_sequence": required_subcommands <= traced_subcommands
and bool(discovery_subcommands & traced_subcommands),
"wrapper": wrapper.get("all_bound_to_supplied_target") is True
and wrapper.get("all_completed_successfully") is True
and wrapper.get("complete_start_end_pairing") is True
and wrapper.get("database_read_only_required") is True
and wrapper.get("default_read_only_required") is True,
"wrapper_invocations": wrapper_invocations_valid
and required_subcommands <= wrapper_subcommands
and bool(discovery_subcommands & wrapper_subcommands),
"canonical_status_unchanged": bool(reasoning.get("canonical_status_before"))
and reasoning.get("canonical_status_before") == reasoning.get("canonical_status_after"),
"database_identity_unchanged": bool(reasoning.get("database_identity_before"))
and reasoning.get("database_identity_before") == reasoning.get("database_identity_after"),
"service_unchanged": service_healthy(reasoning.get("service_before"))
and reasoning.get("service_before") == reasoning.get("service_after"),
"temporary_profile_removed": reasoning.get("temp_profile_absent") is True
and reasoning.get("temp_profile_removed") is True,
}
def verify_lifecycle(
*,
source_path: Path,
current_source_path: Path,
source_delta_path: Path,
restore_path: Path,
parity_path: Path,
reasoning_path: Path,
reasoning_compute_path: Path,
cleanup_path: Path,
max_postflight_age_seconds: float = 900.0,
max_lifecycle_age_seconds: float = 3600.0,
now: datetime | None = None,
) -> dict[str, Any]:
if max_postflight_age_seconds <= 0:
raise ValueError("max postflight age must be positive")
if max_lifecycle_age_seconds <= 0:
raise ValueError("max lifecycle age must be positive")
verification_time = (now or datetime.now(UTC)).astimezone(UTC)
source = load_receipt(source_path, "source receipt")
current_source = load_receipt(current_source_path, "current source receipt")
source_delta = load_receipt(source_delta_path, "source delta receipt")
restore = load_receipt(restore_path, "restore receipt")
parity = load_receipt(parity_path, "parity receipt")
reasoning = load_receipt(reasoning_path, "reasoning receipt")
reasoning_compute_attestation = load_receipt(reasoning_compute_path, "reasoning compute attestation")
cleanup = load_receipt(cleanup_path, "cleanup receipt")
target_db = restore.get("target_database")
run_id = restore.get("run_id")
restore_source = restore.get("source") or {}
capture_binding = restore_source.get("capture_receipt") or {}
restore_target = restore.get("target") or {}
restore_connectivity = restore_target.get("connectivity") or {}
restore_connectivity_proof = restore_target.get("connectivity_proof") or {}
restore_cloudsql = restore_target.get("cloudsql") or {}
restore_compute = restore_target.get("compute") or {}
restore_parity = restore.get("parity") or {}
restore_service = restore.get("live_service") or {}
parity_details = parity.get("details") or {}
parity_evidence_checks = parity_evidence(parity_details)
restore_inline_parity_checks = parity_evidence(restore_parity.get("details") or {})
parity_connectivity = parity.get("private_connectivity") or {}
parity_connectivity_proof = parity_connectivity.get("proof") or {}
reasoning_checks = reasoning.get("checks") or {}
reasoning_parity = reasoning.get("parity_receipt") or {}
before_fingerprint = reasoning.get("database_fingerprint_before") or {}
after_fingerprint = reasoning.get("database_fingerprint_after") or {}
cleanup_database = cleanup.get("database") or {}
cleanup_service = cleanup.get("live_service") or {}
cleanup_cloudsql = (cleanup.get("target") or {}).get("cloudsql") or {}
cleanup_compute = (cleanup.get("target") or {}).get("compute") or {}
restore_contract = restore.get("execution_contract") or {}
cleanup_contract = cleanup.get("execution_contract") or {}
producer_reasoning_checks = blind_reasoning_evidence(reasoning, target_db)
reasoning_compute = reasoning_compute_attestation.get("compute") or {}
reasoning_compute_checks = reasoning_compute_attestation.get("checks") or {}
reasoning_compute_mutation = reasoning_compute_attestation.get("mutation") or {}
delta_baseline = source_delta.get("baseline") or {}
delta_current = source_delta.get("current") or {}
delta_details = source_delta.get("delta") or {}
source_snapshot = source.get("snapshot") or {}
current_source_snapshot = current_source.get("snapshot") or {}
timeline_values = {
"source_snapshot": source_snapshot.get("captured_at_utc"),
"restore_started": restore.get("generated_at_utc"),
"restore_completed": restore.get("completed_at_utc"),
"parity_completed": parity.get("completed_at_utc"),
"reasoning_started": reasoning.get("generated_at_utc"),
"reasoning_completed": reasoning.get("completed_at_utc"),
"reasoning_compute_attested": reasoning_compute_attestation.get("generated_at_utc"),
"cleanup_started": cleanup.get("generated_at_utc"),
"cleanup_completed": cleanup.get("completed_at_utc"),
"current_source_snapshot": current_source_snapshot.get("captured_at_utc"),
}
timeline = [parse_timestamp(value, name) for name, value in timeline_values.items()]
lifecycle_chronology = all(earlier <= later for earlier, later in pairwise(timeline))
delta_detected = delta_details.get("delta_detected")
delta_change_present = bool(
delta_details.get("added_tables")
or delta_details.get("removed_tables")
or delta_details.get("changed_tables")
or delta_details.get("changed_structures")
or delta_details.get("total_row_delta")
)
current_source_time = parse_timestamp(
current_source_snapshot.get("captured_at_utc"), "current source snapshot time"
)
restore_started_time = parse_timestamp(restore.get("generated_at_utc"), "restore start time")
cleanup_completed_time = parse_timestamp(cleanup.get("completed_at_utc"), "cleanup completion time")
postflight_age_seconds = (verification_time - current_source_time).total_seconds()
lifecycle_age_seconds = (verification_time - cleanup_completed_time).total_seconds()
lifecycle_span_seconds = (current_source_time - restore_started_time).total_seconds()
postflight_after_lifecycle = current_source_time >= max(
parse_timestamp(reasoning.get("completed_at_utc"), "reasoning completion time"),
parse_timestamp(cleanup.get("completed_at_utc"), "cleanup completion time"),
)
required_reasoning_checks = (
"exact_blind_prompt_without_ids",
"claim_and_sources_retrieved",
"discovery_show_evidence_completed",
"retrieval_receipts_proven",
"private_tls_read_only_target",
"generated_database_unchanged",
"canonical_counts_unchanged",
"database_calls_read_only",
"no_database_write",
"no_telegram_send",
"live_service_unchanged",
"live_profile_unchanged",
"temporary_profile_removed",
)
checks = {
"source_schema_v2": source.get("schema") == SOURCE_SCHEMA and source.get("receipt_version") == 2,
"source_pass": source.get("status") == "pass"
and source.get("snapshot_exported") is True
and (source.get("source_service") or {}).get("unchanged") is True
and service_healthy((source.get("source_service") or {}).get("before"))
and (source.get("source_service") or {}).get("before") == (source.get("source_service") or {}).get("after"),
"source_authorized": bool(source.get("capture_authorization_ref")),
"source_authorization_bound": capture_binding.get("capture_authorization_ref")
== source.get("capture_authorization_ref"),
"source_service_unchanged": source.get("service_unchanged") is True,
"source_not_mutated": source.get("production_db_mutated") is False,
"current_source_schema_v2": current_source.get("schema") == SOURCE_SCHEMA
and current_source.get("receipt_version") == 2,
"current_source_pass": current_source.get("status") == "pass"
and current_source.get("snapshot_exported") is True
and current_source.get("service_unchanged") is True
and (current_source.get("source_service") or {}).get("unchanged") is True
and service_healthy((current_source.get("source_service") or {}).get("before"))
and (current_source.get("source_service") or {}).get("before")
== (current_source.get("source_service") or {}).get("after")
and current_source.get("production_db_mutated") is False,
"current_source_identity": current_source.get("source") == source.get("source")
and current_source_snapshot.get("system_identifier") == source_snapshot.get("system_identifier"),
"source_delta_pass": source_delta.get("artifact") == "vps_canonical_snapshot_postflight_delta"
and source_delta.get("schema") == DELTA_SCHEMA
and source_delta.get("status") == "pass",
"source_delta_receipts_bound": delta_baseline.get("capture_receipt_sha256") == sha256_file(source_path)
and delta_current.get("capture_receipt_sha256") == sha256_file(current_source_path),
"source_delta_manifests_bound": delta_baseline.get("manifest_sha256")
== (source.get("manifest") or {}).get("sha256")
and delta_current.get("manifest_sha256") == (current_source.get("manifest") or {}).get("sha256"),
"source_delta_authorizations_bound": delta_baseline.get("capture_authorization_ref")
== source.get("capture_authorization_ref")
and delta_current.get("capture_authorization_ref") == current_source.get("capture_authorization_ref"),
"source_delta_explicit": isinstance(delta_details.get("delta_detected"), bool)
and isinstance(delta_details.get("added_tables"), list)
and isinstance(delta_details.get("removed_tables"), list)
and isinstance(delta_details.get("changed_tables"), list)
and isinstance(delta_details.get("changed_structures"), list)
and isinstance(delta_details.get("total_row_delta"), int),
"source_delta_baseline_shape": delta_baseline.get("run_id") == source.get("run_id")
and delta_baseline.get("captured_at_utc") == source_snapshot.get("captured_at_utc")
and delta_baseline.get("table_count") == (source.get("manifest") or {}).get("table_count")
and delta_baseline.get("total_rows") == (source.get("manifest") or {}).get("total_rows")
and bool(SHA256_RE.fullmatch(str(delta_baseline.get("table_fingerprint_sha256") or "")))
and bool(SHA256_RE.fullmatch(str(delta_baseline.get("structure_fingerprint_sha256") or ""))),
"source_delta_current_shape": delta_current.get("run_id") == current_source.get("run_id")
and delta_current.get("captured_at_utc") == current_source_snapshot.get("captured_at_utc")
and delta_current.get("table_count") == (current_source.get("manifest") or {}).get("table_count")
and delta_current.get("total_rows") == (current_source.get("manifest") or {}).get("total_rows")
and bool(SHA256_RE.fullmatch(str(delta_current.get("table_fingerprint_sha256") or "")))
and bool(SHA256_RE.fullmatch(str(delta_current.get("structure_fingerprint_sha256") or ""))),
"source_delta_consistent": delta_detected is delta_change_present
and delta_details.get("total_row_delta")
== (delta_current.get("total_rows") or 0) - (delta_baseline.get("total_rows") or 0)
and (
delta_detected is True
or (
delta_baseline.get("table_count") == delta_current.get("table_count")
and delta_baseline.get("total_rows") == delta_current.get("total_rows")
and delta_baseline.get("table_fingerprint_sha256") == delta_current.get("table_fingerprint_sha256")
and delta_baseline.get("structure_fingerprint_sha256")
== delta_current.get("structure_fingerprint_sha256")
)
),
"source_postflight_after_lifecycle": postflight_after_lifecycle,
"source_postflight_fresh": 0 <= postflight_age_seconds <= max_postflight_age_seconds,
"gcp_lifecycle_fresh": 0 <= lifecycle_age_seconds <= max_lifecycle_age_seconds,
"lifecycle_span_bounded": 0 <= lifecycle_span_seconds <= max_lifecycle_age_seconds,
"lifecycle_chronology": lifecycle_chronology,
"restore_pass": restore.get("artifact") == "gcp_generated_postgres_snapshot_restore"
and restore.get("status") == "pass",
"restore_target_is_disposable": isinstance(target_db, str) and target_db.startswith("teleo_clone_"),
"restore_run_id": isinstance(run_id, str) and run_id.startswith("gcp-restore-"),
"source_receipt_hash_bound": capture_binding.get("sha256") == sha256_file(source_path),
"source_provenance_bound": capture_binding.get("provenance_binding_sha256")
== (source.get("provenance_binding") or {}).get("sha256"),
"source_capture_checks": required_true_checks(
capture_binding.get("checks"),
{
"artifact",
"schema",
"receipt_version",
"status",
"snapshot_exported",
"service_unchanged",
"source_service_healthy",
"source_not_mutated",
"telegram_not_sent",
"run_id",
"capture_authorization_ref",
"source_identity",
"source_database",
"snapshot_identity",
"dump_sha256",
"dump_bytes",
"source_manifest_sha256",
"manifest_sql_sha256",
"source_context_sha256",
"table_count",
"total_rows",
"binding_algorithm",
"binding_payload",
"binding_sha256",
},
),
"source_dump_bound": capture_binding.get("dump_sha256") == (source.get("dump") or {}).get("sha256"),
"source_execution_contract_bound": restore_contract.get("source") == source.get("source"),
"source_manifest_bound": capture_binding.get("manifest_sha256") == (source.get("manifest") or {}).get("sha256"),
"restore_manifest_matches_parity_source": capture_binding.get("manifest_sha256")
== parity.get("source_manifest_sha256"),
"restore_target_manifest_matches_parity": restore_target.get("manifest_sha256")
== parity.get("target_manifest_sha256"),
"restore_private_tls": restore_connectivity.get("private_connectivity") is True
and restore_connectivity.get("ssl") is True
and is_rfc1918(restore_connectivity.get("server_address")),
"restore_public_ip_disabled": restore_cloudsql.get("public_ip_disabled") is True,
"restore_cloudsql_checks": required_true_checks(
restore_cloudsql.get("checks"),
{
"instance",
"state",
"postgres_16",
"public_ip_disabled",
"private_network",
"expected_private_host",
"no_non_private_address",
},
),
"restore_compute_checks": required_true_checks(
restore_compute.get("checks"),
{
"project",
"project_number",
"expected_network_project",
"instance",
"instance_id",
"zone",
"network",
"private_ip",
"service_account",
},
),
"restore_compute_contract": restore_compute.get("project") == restore_contract.get("project")
and restore_compute.get("instance") == restore_contract.get("compute_instance")
and restore_compute.get("zone") == restore_contract.get("compute_zone")
and restore_compute.get("expected_private_network") == restore_contract.get("private_network")
and bool(restore_compute.get("network_resource"))
and is_rfc1918(restore_compute.get("private_ip"))
and str(restore_compute.get("service_account") or "").endswith(".iam.gserviceaccount.com"),
"restore_connectivity_compute_bound": restore_connectivity.get("source_compute")
== restore_compute.get("instance"),
"restore_inline_parity": restore_parity.get("status") == "pass" and restore_parity.get("problems") == [],
"restore_inline_parity_evidence": all(restore_inline_parity_checks.values()),
"restore_service_healthy_unchanged": restore_service.get("unchanged") is True
and service_healthy(restore_service.get("before"))
and restore_service.get("before") == restore_service.get("after"),
"parity_pass": parity.get("artifact") == "canonical_postgres_parity_verification"
and parity.get("status") == "pass"
and parity.get("scope") == "gcp_staging"
and parity.get("problems") == [],
"parity_target_bound": parity_details.get("target_database") == target_db,
"parity_exact_evidence": all(parity_evidence_checks.values()),
"parity_connectivity_hash_bound": restore_connectivity_proof.get("sha256")
== parity.get("connectivity_proof_sha256"),
"parity_private_proof": parity_connectivity.get("required") is True
and parity_connectivity_proof.get("artifact") == "gcp_private_postgres_connectivity"
and parity_connectivity_proof.get("status") == "pass"
and parity_connectivity_proof.get("schema") == "livingip.gcpPrivatePostgresConnectivity.v2"
and parity_connectivity_proof.get("restore_run_id") == run_id
and parity_connectivity_proof.get("private_connectivity") is True
and parity_connectivity_proof.get("public_ip_disabled") is True
and parity_connectivity_proof.get("target_database") == target_db
and parity_connectivity_proof.get("project") == restore_cloudsql.get("project")
and parity_connectivity_proof.get("cloudsql_instance") == restore_cloudsql.get("instance")
and parity_connectivity_proof.get("private_network") == restore_cloudsql.get("private_network")
and parity_connectivity_proof.get("source_compute") == reasoning.get("source_compute")
and parity_connectivity_proof.get("source_compute_instance_id") == restore_compute.get("instance_id")
and parity_connectivity_proof.get("source_compute_zone") == restore_compute.get("zone")
and is_rfc1918(parity_connectivity_proof.get("source_compute_private_ip"))
and parity_connectivity_proof.get("server_address") == restore_connectivity.get("server_address")
and required_true_checks(
parity_connectivity_proof.get("cloudsql_control_plane_checks"),
{
"instance",
"state",
"postgres_16",
"public_ip_disabled",
"private_network",
"expected_private_host",
"no_non_private_address",
},
)
and required_true_checks(
parity_connectivity_proof.get("compute_identity_checks"),
{
"project",
"project_number",
"expected_network_project",
"instance",
"instance_id",
"zone",
"network",
"private_ip",
"service_account",
},
),
"reasoning_schema": reasoning.get("schema") == BLIND_REASONING_SCHEMA,
"reasoning_pass": reasoning.get("status") == "pass" and not reasoning.get("errors"),
"reasoning_target_bound": reasoning.get("target_database") == target_db,
"reasoning_compute_bound": reasoning.get("source_compute") == restore_compute.get("instance"),
"reasoning_parity_hash_bound": reasoning_parity.get("sha256") == sha256_file(parity_path),
"reasoning_required_checks": all(reasoning_checks.get(name) is True for name in required_reasoning_checks),
"reasoning_producer_evidence": all(producer_reasoning_checks.values()),
"reasoning_compute_attestation": reasoning_compute_attestation.get("artifact")
== "gcp_reasoning_compute_attestation"
and reasoning_compute_attestation.get("schema") == REASONING_COMPUTE_SCHEMA
and reasoning_compute_attestation.get("status") == "pass"
and reasoning_compute_attestation.get("method") == "gce_metadata_server_v1"
and nonempty_all_true(reasoning_compute_checks)
and reasoning_compute_attestation.get("restore_run_id") == run_id
and reasoning_compute_attestation.get("target_database") == target_db
and all(value is False for value in reasoning_compute_mutation.values())
and {
"cloudsql_changed",
"compute_changed",
"database_changed",
"telegram_message_sent",
}
<= set(reasoning_compute_mutation),
"reasoning_compute_receipt_bound": (reasoning_compute_attestation.get("reasoning_receipt") or {}).get("sha256")
== sha256_file(reasoning_path),
"reasoning_compute_identity_bound": reasoning_compute.get("project") == restore_compute.get("project")
and reasoning_compute.get("project_number") == restore_compute.get("project_number")
and reasoning_compute.get("instance") == restore_compute.get("instance")
and reasoning_compute.get("instance_id") == restore_compute.get("instance_id")
and reasoning_compute.get("zone") == restore_compute.get("zone")
and reasoning_compute.get("expected_private_network") == restore_compute.get("expected_private_network")
and reasoning_compute.get("network_resource") == restore_compute.get("network_resource")
and reasoning_compute.get("private_ip") == restore_compute.get("private_ip")
and reasoning_compute.get("service_account") == restore_compute.get("service_account")
and required_true_checks(
reasoning_compute.get("checks"),
{
"project",
"project_number",
"expected_network_project",
"instance",
"instance_id",
"zone",
"network",
"private_ip",
"service_account",
},
),
"reasoning_no_send_no_write": reasoning.get("posted_to_telegram") is False
and reasoning.get("database_write_attempted") is False,
"reasoning_fingerprint_unchanged": bool(before_fingerprint.get("sha256"))
and before_fingerprint.get("sha256") == after_fingerprint.get("sha256"),
"cleanup_pass": cleanup.get("artifact") == "gcp_generated_postgres_snapshot_cleanup"
and cleanup.get("status") == "pass",
"cleanup_run_bound": cleanup.get("run_id") == run_id and cleanup.get("target_database") == target_db,
"cleanup_restore_receipt_bound": (cleanup.get("restore_receipt") or {}).get("sha256")
== sha256_file(restore_path),
"cleanup_execution_contract_bound": bool(restore_contract) and cleanup_contract == restore_contract,
"cleanup_cloudsql_bound": cleanup_cloudsql.get("project") == restore_cloudsql.get("project")
and cleanup_cloudsql.get("instance") == restore_cloudsql.get("instance")
and cleanup_cloudsql.get("expected_private_host") == restore_cloudsql.get("expected_private_host")
and cleanup_cloudsql.get("expected_private_network") == restore_cloudsql.get("expected_private_network")
and cleanup_cloudsql.get("public_ip_disabled") is True
and required_true_checks(
cleanup_cloudsql.get("checks"),
{
"instance",
"state",
"postgres_16",
"public_ip_disabled",
"private_network",
"expected_private_host",
"no_non_private_address",
},
),
"cleanup_compute_bound": cleanup_compute.get("instance_id") == restore_compute.get("instance_id")
and cleanup_compute.get("project") == restore_compute.get("project")
and cleanup_compute.get("project_number") == restore_compute.get("project_number")
and cleanup_compute.get("instance") == restore_compute.get("instance")
and cleanup_compute.get("zone") == restore_compute.get("zone")
and cleanup_compute.get("expected_private_network") == restore_compute.get("expected_private_network")
and cleanup_compute.get("network_resource") == restore_compute.get("network_resource")
and cleanup_compute.get("private_ip") == restore_compute.get("private_ip")
and cleanup_compute.get("service_account") == restore_compute.get("service_account")
and required_true_checks(
cleanup_compute.get("checks"),
{
"project",
"project_number",
"expected_network_project",
"instance",
"instance_id",
"zone",
"network",
"private_ip",
"service_account",
},
),
"clone_absent": cleanup_database.get("clone_database_remaining") == 0,
"cleanup_service_unchanged": cleanup_service.get("unchanged") is True
and service_healthy(cleanup_service.get("before"))
and cleanup_service.get("before") == cleanup_service.get("after"),
}
failed = sorted(name for name, passed in checks.items() if not passed)
return {
"artifact": "gcp_canonical_snapshot_lifecycle_verification",
"generated_at_utc": verification_time.isoformat(),
"status": "pass" if not failed else "fail",
"required_tier": "T3_live_private_gcp_staging",
"target_database": target_db,
"run_id": run_id,
"checks": checks,
"reasoning_producer_checks": producer_reasoning_checks,
"restore_inline_parity_checks": restore_inline_parity_checks,
"parity_evidence_checks": parity_evidence_checks,
"post_snapshot_source_delta": delta_details,
"timeline": timeline_values,
"postflight_age_seconds": postflight_age_seconds,
"max_postflight_age_seconds": max_postflight_age_seconds,
"lifecycle_age_seconds": lifecycle_age_seconds,
"lifecycle_span_seconds": lifecycle_span_seconds,
"max_lifecycle_age_seconds": max_lifecycle_age_seconds,
"failed_checks": failed,
"receipt_sha256": {
"source": sha256_file(source_path),
"current_source": sha256_file(current_source_path),
"source_delta": sha256_file(source_delta_path),
"restore": sha256_file(restore_path),
"parity": sha256_file(parity_path),
"reasoning": sha256_file(reasoning_path),
"reasoning_compute": sha256_file(reasoning_compute_path),
"cleanup": sha256_file(cleanup_path),
},
"strongest_claim_allowed": (
(
"current private GCP staging parity, ID-free Leo reasoning, cleanup, and zero post-snapshot VPS delta verified"
if delta_details.get("delta_detected") is False
else "private GCP staging parity at the declared snapshot boundary, ID-free Leo reasoning, cleanup, and explicit post-snapshot VPS delta verified"
)
if not failed
else "lifecycle incomplete; failed checks must be repaired"
),
"not_proven": ["production cutover", "ongoing replication", "Telegram delivery"],
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--source-receipt", required=True, type=Path)
parser.add_argument("--current-source-receipt", required=True, type=Path)
parser.add_argument("--source-delta-receipt", required=True, type=Path)
parser.add_argument("--restore-receipt", required=True, type=Path)
parser.add_argument("--parity-receipt", required=True, type=Path)
parser.add_argument("--reasoning-receipt", required=True, type=Path)
parser.add_argument("--reasoning-compute-receipt", required=True, type=Path)
parser.add_argument("--cleanup-receipt", required=True, type=Path)
parser.add_argument("--output", required=True, type=Path)
parser.add_argument("--max-postflight-age-seconds", default=900.0, type=float)
parser.add_argument("--max-lifecycle-age-seconds", default=3600.0, type=float)
args = parser.parse_args()
try:
payload = verify_lifecycle(
source_path=args.source_receipt,
current_source_path=args.current_source_receipt,
source_delta_path=args.source_delta_receipt,
restore_path=args.restore_receipt,
parity_path=args.parity_receipt,
reasoning_path=args.reasoning_receipt,
reasoning_compute_path=args.reasoning_compute_receipt,
cleanup_path=args.cleanup_receipt,
max_postflight_age_seconds=args.max_postflight_age_seconds,
max_lifecycle_age_seconds=args.max_lifecycle_age_seconds,
)
except (OSError, ValueError, KeyError, TypeError) as exc:
payload = {
"artifact": "gcp_canonical_snapshot_lifecycle_verification",
"generated_at_utc": datetime.now(UTC).isoformat(),
"status": "fail",
"required_tier": "T3_live_private_gcp_staging",
"failed_checks": ["receipt_load_or_validation_error"],
"error": str(exc),
"strongest_claim_allowed": "lifecycle incomplete; receipts could not be validated",
}
write_private_json(args.output, payload)
print(json.dumps(payload, indent=2, sort_keys=True))
return 0 if payload.get("status") == "pass" else 1
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -11,6 +11,11 @@ from datetime import UTC, datetime
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
try:
from .private_receipt_io import write_private_json
except ImportError: # pragma: no cover - direct script execution
from private_receipt_io import write_private_json
SINGLETON_KINDS = { SINGLETON_KINDS = {
"identity", "identity",
"schemas", "schemas",
@ -33,6 +38,14 @@ def canonical_hash(value: Any) -> str:
return hashlib.sha256(payload.encode()).hexdigest() return hashlib.sha256(payload.encode()).hexdigest()
def file_sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def load_manifest(path: Path) -> dict[str, Any]: def load_manifest(path: Path) -> dict[str, Any]:
singleton: dict[str, dict[str, Any]] = {} singleton: dict[str, dict[str, Any]] = {}
tables: dict[str, dict[str, Any]] = {} tables: dict[str, dict[str, Any]] = {}
@ -295,6 +308,9 @@ def main() -> int:
args = parser.parse_args() args = parser.parse_args()
try: try:
source_manifest_sha256 = file_sha256(args.source)
target_manifest_sha256 = file_sha256(args.target)
connectivity_proof_sha256 = file_sha256(args.connectivity_proof) if args.connectivity_proof else None
source = load_manifest(args.source) source = load_manifest(args.source)
target = load_manifest(args.target) target = load_manifest(args.target)
problems, details = compare_manifests( problems, details = compare_manifests(
@ -316,18 +332,26 @@ def main() -> int:
problems = [f"manifest_error={exc}"] problems = [f"manifest_error={exc}"]
details = {} details = {}
connectivity = {"required": args.scope == "gcp_staging", "status": "not_checked"} connectivity = {"required": args.scope == "gcp_staging", "status": "not_checked"}
source_manifest_sha256 = None
target_manifest_sha256 = None
connectivity_proof_sha256 = None
error = str(exc) error = str(exc)
completed_at_utc = datetime.now(UTC).isoformat()
payload = { payload = {
"artifact": "canonical_postgres_parity_verification", "artifact": "canonical_postgres_parity_verification",
"generated_at_utc": datetime.now(UTC).isoformat(), "generated_at_utc": completed_at_utc,
"completed_at_utc": completed_at_utc,
"scope": args.scope, "scope": args.scope,
"source_manifest": str(args.source), "source_manifest": str(args.source),
"source_manifest_sha256": source_manifest_sha256,
"target_manifest": str(args.target), "target_manifest": str(args.target),
"target_manifest_sha256": target_manifest_sha256,
"status": status, "status": status,
"problems": problems, "problems": problems,
"details": details, "details": details,
"private_connectivity": connectivity, "private_connectivity": connectivity,
"connectivity_proof_sha256": connectivity_proof_sha256,
"error": error, "error": error,
"not_proven_by_this_artifact": [ "not_proven_by_this_artifact": [
"GCP staging Leo composition replay", "GCP staging Leo composition replay",
@ -335,8 +359,7 @@ def main() -> int:
"production cutover", "production cutover",
], ],
} }
args.output.parent.mkdir(parents=True, exist_ok=True) write_private_json(args.output, payload)
args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
print(json.dumps(payload, indent=2, sort_keys=True)) print(json.dumps(payload, indent=2, sort_keys=True))
return 0 if status == "pass" else 1 return 0 if status == "pass" else 1

View file

@ -0,0 +1,284 @@
#!/usr/bin/env python3
"""Bind a postflight VPS manifest to a captured snapshot and report exact deltas."""
from __future__ import annotations
import argparse
import json
import re
import stat
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
try:
from .capture_vps_canonical_postgres_snapshot import (
SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
build_provenance_binding,
)
from .private_receipt_io import write_private_json
from .verify_postgres_parity_manifest import SINGLETON_KINDS, canonical_hash, file_sha256, load_manifest
except ImportError: # pragma: no cover - direct script execution
from capture_vps_canonical_postgres_snapshot import SOURCE_SNAPSHOT_RECEIPT_SCHEMA, build_provenance_binding
from private_receipt_io import write_private_json
from verify_postgres_parity_manifest import SINGLETON_KINDS, canonical_hash, file_sha256, load_manifest
DELTA_SCHEMA = "livingip.vpsCanonicalSnapshotDelta.v1"
SHA256_RE = re.compile(r"[0-9a-f]{64}\Z")
def service_healthy(value: Any) -> bool:
if not isinstance(value, dict):
return False
try:
main_pid = int(value.get("MainPID") or 0)
restarts = int(value.get("NRestarts") or 0)
except (TypeError, ValueError):
return False
return (
value.get("ActiveState") == "active" and value.get("SubState") == "running" and main_pid > 0 and restarts >= 0
)
def regular_file(path: Path, label: str) -> Path:
try:
mode = path.lstat().st_mode
except OSError as exc:
raise ValueError(f"{label} is unavailable: {exc}") from exc
if not stat.S_ISREG(mode) or path.is_symlink():
raise ValueError(f"{label} must be a regular non-symlink file")
return path.resolve()
def load_json(path: Path, label: str) -> dict[str, Any]:
path = regular_file(path, label)
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except json.JSONDecodeError as exc:
raise ValueError(f"{label} is invalid JSON") from exc
if not isinstance(payload, dict):
raise ValueError(f"{label} must be a JSON object")
return payload
def parse_timestamp(value: Any, label: str) -> datetime:
text = str(value or "")
if text.endswith("Z"):
text = text[:-1] + "+00:00"
try:
parsed = datetime.fromisoformat(text)
except ValueError as exc:
raise ValueError(f"{label} is not an ISO-8601 timestamp") from exc
if parsed.tzinfo is None:
raise ValueError(f"{label} must include a timezone")
return parsed.astimezone(UTC)
def validate_capture(
receipt_path: Path,
manifest_path: Path,
*,
expected_authorization_ref: str,
label: str,
) -> tuple[dict[str, Any], dict[str, Any]]:
receipt = load_json(receipt_path, f"{label} receipt")
manifest_path = regular_file(manifest_path, f"{label} manifest")
manifest = load_manifest(manifest_path)
snapshot = receipt.get("snapshot") or {}
source = receipt.get("source") or {}
dump = receipt.get("dump") or {}
manifest_receipt = receipt.get("manifest") or {}
source_context = receipt.get("source_context") or {}
source_service = receipt.get("source_service") or {}
binding = receipt.get("provenance_binding") or {}
expected_binding = build_provenance_binding(
run_id=str(receipt.get("run_id") or ""),
authorization_ref=str(receipt.get("capture_authorization_ref") or ""),
source=source,
snapshot=snapshot,
dump_sha256=str(dump.get("sha256") or ""),
manifest_sql_sha256=str(manifest_receipt.get("manifest_sql_sha256") or ""),
source_manifest_sha256=file_sha256(manifest_path),
source_context_sha256=str(source_context.get("sha256") or ""),
)
total_rows = sum(int(row["row_count"]) for row in manifest["tables"].values())
checks = {
"schema": receipt.get("schema") == SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"receipt_version": receipt.get("receipt_version") == 2,
"status": receipt.get("status") == "pass",
"authorization": receipt.get("capture_authorization_ref") == expected_authorization_ref,
"snapshot_exported": receipt.get("snapshot_exported") is True,
"service_unchanged": receipt.get("service_unchanged") is True,
"source_service_healthy": source_service.get("unchanged") is True
and service_healthy(source_service.get("before"))
and source_service.get("before") == source_service.get("after"),
"source_not_mutated": receipt.get("production_db_mutated") is False,
"telegram_not_sent": receipt.get("telegram_send_attempted") is False,
"manifest_hash": manifest_receipt.get("sha256") == file_sha256(manifest_path),
"manifest_table_count": manifest_receipt.get("table_count") == len(manifest["tables"]),
"manifest_total_rows": manifest_receipt.get("total_rows") == total_rows,
"manifest_read_only": manifest["singleton"]["identity"].get("transaction_read_only") == "on",
"manifest_database": manifest["singleton"]["identity"].get("database") == source.get("database"),
"binding_payload": binding.get("payload") == expected_binding["payload"],
"binding_sha256": binding.get("sha256") == expected_binding["sha256"],
"binding_algorithm": binding.get("algorithm") == "sha256",
"dump_sha256": bool(SHA256_RE.fullmatch(str(dump.get("sha256") or ""))),
"manifest_sql_sha256": bool(SHA256_RE.fullmatch(str(manifest_receipt.get("manifest_sql_sha256") or ""))),
"source_context_sha256": bool(SHA256_RE.fullmatch(str(source_context.get("sha256") or ""))),
"snapshot_identity": all(
bool(snapshot.get(field))
for field in ("captured_at_utc", "exported_snapshot_id", "txid_snapshot", "wal_lsn", "system_identifier")
),
}
failed = sorted(name for name, passed in checks.items() if not passed)
if failed:
raise ValueError(f"{label} capture failed validation: " + ", ".join(failed))
return receipt, manifest
def manifest_fingerprint(manifest: dict[str, Any]) -> dict[str, Any]:
tables = {
name: {"row_count": int(row["row_count"]), "rowset_md5": row.get("rowset_md5")}
for name, row in sorted(manifest["tables"].items())
}
structures = {
kind: canonical_hash(manifest["singleton"][kind].get("items", []))
for kind in sorted(SINGLETON_KINDS - {"identity"})
}
return {
"table_count": len(tables),
"total_rows": sum(row["row_count"] for row in tables.values()),
"table_fingerprint_sha256": canonical_hash(tables),
"structure_fingerprint_sha256": canonical_hash(structures),
"tables": tables,
"structures": structures,
}
def verify_delta(
*,
baseline_receipt_path: Path,
baseline_manifest_path: Path,
baseline_authorization_ref: str,
current_receipt_path: Path,
current_manifest_path: Path,
current_authorization_ref: str,
) -> dict[str, Any]:
baseline_receipt, baseline_manifest = validate_capture(
baseline_receipt_path,
baseline_manifest_path,
expected_authorization_ref=baseline_authorization_ref,
label="baseline",
)
current_receipt, current_manifest = validate_capture(
current_receipt_path,
current_manifest_path,
expected_authorization_ref=current_authorization_ref,
label="current",
)
baseline_snapshot = baseline_receipt["snapshot"]
current_snapshot = current_receipt["snapshot"]
baseline_time = parse_timestamp(baseline_snapshot["captured_at_utc"], "baseline snapshot time")
current_time = parse_timestamp(current_snapshot["captured_at_utc"], "current snapshot time")
if current_time <= baseline_time:
raise ValueError("current source capture must be newer than the restored baseline capture")
if current_receipt.get("source") != baseline_receipt.get("source"):
raise ValueError("current source identity does not match the restored baseline source")
if current_snapshot.get("system_identifier") != baseline_snapshot.get("system_identifier"):
raise ValueError("current source PostgreSQL system identifier changed")
baseline = manifest_fingerprint(baseline_manifest)
current = manifest_fingerprint(current_manifest)
baseline_tables = baseline.pop("tables")
current_tables = current.pop("tables")
baseline_structures = baseline.pop("structures")
current_structures = current.pop("structures")
added_tables = sorted(set(current_tables) - set(baseline_tables))
removed_tables = sorted(set(baseline_tables) - set(current_tables))
changed_tables = [
{"table": table, "baseline": baseline_tables[table], "current": current_tables[table]}
for table in sorted(set(baseline_tables) & set(current_tables))
if baseline_tables[table] != current_tables[table]
]
changed_structures = [
{
"kind": kind,
"baseline_sha256": baseline_structures.get(kind),
"current_sha256": current_structures.get(kind),
}
for kind in sorted(set(baseline_structures) | set(current_structures))
if baseline_structures.get(kind) != current_structures.get(kind)
]
delta_detected = bool(added_tables or removed_tables or changed_tables or changed_structures)
return {
"artifact": "vps_canonical_snapshot_postflight_delta",
"schema": DELTA_SCHEMA,
"generated_at_utc": datetime.now(UTC).isoformat(),
"status": "pass",
"baseline": {
"run_id": baseline_receipt["run_id"],
"capture_authorization_ref": baseline_authorization_ref,
"captured_at_utc": baseline_snapshot["captured_at_utc"],
"capture_receipt_sha256": file_sha256(baseline_receipt_path),
"manifest_sha256": file_sha256(baseline_manifest_path),
**baseline,
},
"current": {
"run_id": current_receipt["run_id"],
"capture_authorization_ref": current_authorization_ref,
"captured_at_utc": current_snapshot["captured_at_utc"],
"capture_receipt_sha256": file_sha256(current_receipt_path),
"manifest_sha256": file_sha256(current_manifest_path),
**current,
},
"delta": {
"delta_detected": delta_detected,
"added_tables": added_tables,
"removed_tables": removed_tables,
"changed_tables": changed_tables,
"changed_structures": changed_structures,
"total_row_delta": current["total_rows"] - baseline["total_rows"],
},
"strongest_claim_allowed": (
"restored snapshot still matches the newest postflight VPS capture"
if not delta_detected
else "restored snapshot parity is bounded to its capture; post-snapshot VPS deltas are explicit"
),
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--baseline-receipt", required=True, type=Path)
parser.add_argument("--baseline-manifest", required=True, type=Path)
parser.add_argument("--baseline-authorization-ref", required=True)
parser.add_argument("--current-receipt", required=True, type=Path)
parser.add_argument("--current-manifest", required=True, type=Path)
parser.add_argument("--current-authorization-ref", required=True)
parser.add_argument("--output", required=True, type=Path)
args = parser.parse_args()
try:
payload = verify_delta(
baseline_receipt_path=args.baseline_receipt,
baseline_manifest_path=args.baseline_manifest,
baseline_authorization_ref=args.baseline_authorization_ref,
current_receipt_path=args.current_receipt,
current_manifest_path=args.current_manifest,
current_authorization_ref=args.current_authorization_ref,
)
except (OSError, ValueError, KeyError, TypeError, json.JSONDecodeError) as exc:
payload = {
"artifact": "vps_canonical_snapshot_postflight_delta",
"schema": DELTA_SCHEMA,
"generated_at_utc": datetime.now(UTC).isoformat(),
"status": "fail",
"error": str(exc),
"strongest_claim_allowed": "post-snapshot VPS delta is not proven",
}
write_private_json(args.output, payload)
print(json.dumps(payload, indent=2, sort_keys=True))
return 0 if payload.get("status") == "pass" else 1
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -0,0 +1,128 @@
import json
from pathlib import Path
import pytest
from ops import attest_gcp_reasoning_compute as attestation
def reasoning_receipt(path: Path) -> Path:
path.write_text(
json.dumps(
{
"schema": attestation.REASONING_SCHEMA,
"status": "pass",
"generated_at_utc": "2026-07-15T01:00:00+00:00",
"completed_at_utc": "2026-07-15T01:01:00+00:00",
"target_database": "teleo_clone_current_test",
"source_compute": "teleo-prod-1",
"errors": [],
"posted_to_telegram": False,
"database_write_attempted": False,
},
sort_keys=True,
)
+ "\n",
encoding="utf-8",
)
return path
def compute_identity() -> dict:
return {
"project": "teleo-501523",
"project_number": "785938879453",
"instance": "teleo-prod-1",
"instance_id": "123456789",
"zone": "europe-west6-a",
"zone_resource": "projects/123456789/zones/europe-west6-a",
"network_resource": "projects/123456789/networks/teleo-staging-net",
"private_ip": "10.61.0.2",
"service_account": "teleo-runtime@teleo-501523.iam.gserviceaccount.com",
"expected_private_network": "projects/teleo-501523/global/networks/teleo-staging-net",
"checks": {
"project": True,
"project_number": True,
"expected_network_project": True,
"instance": True,
"instance_id": True,
"zone": True,
"network": True,
"private_ip": True,
"service_account": True,
},
}
def test_attestation_binds_reasoning_hash_to_live_metadata(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
receipt_path = reasoning_receipt(tmp_path / "reasoning.json")
observed = {}
def fake_identity(project, instance, zone, network, *, timeout):
observed.update(
{
"project": project,
"instance": instance,
"zone": zone,
"network": network,
"timeout": timeout,
}
)
return compute_identity()
monkeypatch.setattr(attestation, "gce_compute_identity", fake_identity)
result = attestation.attest_reasoning_compute(
reasoning_receipt_path=receipt_path,
restore_run_id="gcp-restore-current-test123",
target_database="teleo_clone_current_test",
generated_at_utc="2026-07-15T01:01:05+00:00",
)
assert result["status"] == "pass"
assert result["checks"]["gce_metadata_identity"] is True
assert result["reasoning_receipt"]["sha256"] == attestation.sha256_file(receipt_path)
assert result["compute"]["instance_id"] == "123456789"
assert observed == {
"project": "teleo-501523",
"instance": "teleo-prod-1",
"zone": "europe-west6-a",
"network": "projects/teleo-501523/global/networks/teleo-staging-net",
"timeout": 10.0,
}
def test_attestation_rejects_claimed_wrong_compute_before_metadata(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
receipt_path = reasoning_receipt(tmp_path / "reasoning.json")
payload = json.loads(receipt_path.read_text())
payload["source_compute"] = "untrusted-vm"
receipt_path.write_text(json.dumps(payload) + "\n", encoding="utf-8")
monkeypatch.setattr(
attestation,
"gce_compute_identity",
lambda *args, **kwargs: pytest.fail("metadata must not be queried"),
)
with pytest.raises(ValueError, match="source_compute"):
attestation.attest_reasoning_compute(
reasoning_receipt_path=receipt_path,
restore_run_id="gcp-restore-current-test123",
target_database="teleo_clone_current_test",
)
def test_attestation_rejects_send_or_write_receipt(tmp_path: Path) -> None:
receipt_path = reasoning_receipt(tmp_path / "reasoning.json")
payload = json.loads(receipt_path.read_text())
payload["posted_to_telegram"] = True
payload["database_write_attempted"] = True
receipt_path.write_text(json.dumps(payload) + "\n", encoding="utf-8")
with pytest.raises(ValueError, match=r"no_database_write.*no_telegram_send"):
attestation.attest_reasoning_compute(
reasoning_receipt_path=receipt_path,
restore_run_id="gcp-restore-current-test123",
target_database="teleo_clone_current_test",
)

View file

@ -1,8 +1,18 @@
import json
from pathlib import Path from pathlib import Path
import pytest
from ops.capture_vps_canonical_postgres_snapshot import ( from ops.capture_vps_canonical_postgres_snapshot import (
SAFE_AUTHORIZATION_REF_RE,
SAFE_SSH_TARGET_RE, SAFE_SSH_TARGET_RE,
SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
build_provenance_binding,
canonical_sha256,
cleanup_failed_output, cleanup_failed_output,
load_service_state,
load_snapshot_metadata,
parse_args,
remote_capture_script, remote_capture_script,
toc_counts, toc_counts,
) )
@ -21,8 +31,14 @@ def test_remote_capture_uses_one_exported_read_only_snapshot() -> None:
assert "[0-9A-F]+-[0-9A-F]+-[0-9]+" in script assert "[0-9A-F]+-[0-9A-F]+-[0-9]+" in script
assert '--snapshot="$snapshot_id"' in script assert '--snapshot="$snapshot_id"' in script
assert '-v "snapshot_id=$snapshot_id"' in script assert '-v "snapshot_id=$snapshot_id"' in script
assert "txid_current_snapshot()::text" in script
assert "pg_current_wal_lsn()::text" in script
assert "system_identifier::text from pg_control_system()" in script
assert "source-snapshot.json" in script
assert "--no-owner --no-acl" in script assert "--no-owner --no-acl" in script
assert "cmp -s" in script assert "cmp -s" in script
assert 'assert_service_healthy "$remote_root/service-before.txt"' in script
assert 'assert_service_healthy "$remote_root/service-after.txt"' in script
assert "systemctl restart" not in script assert "systemctl restart" not in script
assert "docker restart" not in script assert "docker restart" not in script
@ -62,8 +78,129 @@ def test_failure_cleanup_never_removes_preexisting_output(tmp_path: Path) -> Non
assert not created_by_run.exists() assert not created_by_run.exists()
def test_source_service_state_requires_healthy_unchanged_capable_fields(tmp_path: Path) -> None:
state_path = tmp_path / "service.txt"
state_path.write_text("ActiveState=active\nSubState=running\nMainPID=347406\nNRestarts=0\n")
assert load_service_state(state_path)["MainPID"] == "347406"
state_path.write_text("ActiveState=failed\nSubState=dead\nMainPID=0\nNRestarts=0\n")
with pytest.raises(RuntimeError, match="not active/running"):
load_service_state(state_path)
def test_ssh_target_rejects_open_ssh_option_injection() -> None: def test_ssh_target_rejects_open_ssh_option_injection() -> None:
assert SAFE_SSH_TARGET_RE.fullmatch("root@77.42.65.182") assert SAFE_SSH_TARGET_RE.fullmatch("root@77.42.65.182")
assert SAFE_SSH_TARGET_RE.fullmatch("teleo-staging-1.internal") assert SAFE_SSH_TARGET_RE.fullmatch("teleo-staging-1.internal")
assert not SAFE_SSH_TARGET_RE.fullmatch("-oProxyCommand=malicious") assert not SAFE_SSH_TARGET_RE.fullmatch("-oProxyCommand=malicious")
assert not SAFE_SSH_TARGET_RE.fullmatch("root@host command") assert not SAFE_SSH_TARGET_RE.fullmatch("root@host command")
def test_source_snapshot_metadata_and_provenance_binding(tmp_path: Path) -> None:
snapshot_path = tmp_path / "source-snapshot.json"
snapshot_path.write_text(
json.dumps(
{
"exported_snapshot_id": "00000003-000001A2-1",
"txid_snapshot": "100:120:105,111",
"wal_lsn": "0/16B6C50",
"system_identifier": "7649789040005668902",
"captured_at_utc": "2026-07-15T01:02:03.456789+00:00",
}
)
)
snapshot = load_snapshot_metadata(snapshot_path)
binding = build_provenance_binding(
run_id="canonical-20260715",
authorization_ref="codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621",
source={
"ssh_target": "root@77.42.65.182",
"container": "teleo-pg",
"database": "teleo",
"service": "leoclean-gateway.service",
},
snapshot=snapshot,
dump_sha256="1" * 64,
manifest_sql_sha256="2" * 64,
source_manifest_sha256="3" * 64,
source_context_sha256="4" * 64,
)
assert binding["algorithm"] == "sha256"
assert binding["payload"] == {
"receipt_schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"run_id": "canonical-20260715",
"capture_authorization_ref": "codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621",
"source": {
"ssh_target": "root@77.42.65.182",
"container": "teleo-pg",
"database": "teleo",
"service": "leoclean-gateway.service",
},
"exported_snapshot_id": "00000003-000001A2-1",
"txid_snapshot": "100:120:105,111",
"wal_lsn": "0/16B6C50",
"source_system_identifier": "7649789040005668902",
"dump_sha256": "1" * 64,
"manifest_sql_sha256": "2" * 64,
"source_manifest_sha256": "3" * 64,
"source_context_sha256": "4" * 64,
}
assert binding["sha256"] == canonical_sha256(binding["payload"])
@pytest.mark.parametrize(
("field", "value"),
[
("exported_snapshot_id", "unsafe snapshot"),
("txid_snapshot", "100:120:not-a-txid"),
("wal_lsn", "not-an-lsn"),
("system_identifier", "system-1"),
],
)
def test_source_snapshot_metadata_rejects_invalid_identity(tmp_path: Path, field: str, value: str) -> None:
payload = {
"exported_snapshot_id": "00000003-000001A2-1",
"txid_snapshot": "100:120:",
"wal_lsn": "0/16B6C50",
"system_identifier": "7649789040005668902",
"captured_at_utc": "2026-07-15T01:02:03+00:00",
}
payload[field] = value
path = tmp_path / "source-snapshot.json"
path.write_text(json.dumps(payload))
with pytest.raises(RuntimeError, match=field):
load_snapshot_metadata(path)
def test_authorization_ref_is_required_and_safe_metadata(tmp_path: Path) -> None:
ssh_key = tmp_path / "id_ed25519"
ssh_key.write_text("test-only")
manifest = tmp_path / "manifest.sql"
manifest.write_text("select 1;\n")
base = [
"--execute",
"--ssh-target",
"root@77.42.65.182",
"--ssh-key",
str(ssh_key),
"--manifest",
str(manifest),
"--run-id",
"canonical-20260715",
"--output-dir",
str(tmp_path / "capture"),
]
with pytest.raises(SystemExit):
parse_args(base)
with pytest.raises(SystemExit):
parse_args([*base, "--authorization-ref", "unsafe authorization ref"])
authorization_ref = "codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621"
args = parse_args([*base, "--authorization-ref", authorization_ref])
assert args.authorization_ref == authorization_ref
assert SAFE_AUTHORIZATION_REF_RE.fullmatch(authorization_ref)
assert not SAFE_AUTHORIZATION_REF_RE.fullmatch("authorization\nforged")

View file

@ -7,6 +7,7 @@ from pathlib import Path
import pytest import pytest
from ops import restore_gcp_generated_postgres_snapshot as restore from ops import restore_gcp_generated_postgres_snapshot as restore
from ops.capture_vps_canonical_postgres_snapshot import build_provenance_binding
SERVICE_STATE = { SERVICE_STATE = {
"ActiveState": "active", "ActiveState": "active",
@ -15,6 +16,7 @@ SERVICE_STATE = {
"NRestarts": "0", "NRestarts": "0",
"ExecMainStartTimestamp": "Thu 2026-07-09 07:00:04 UTC", "ExecMainStartTimestamp": "Thu 2026-07-09 07:00:04 UTC",
} }
AUTHORIZATION_REF = "codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621"
def manifest_rows(database: str, *, target: bool, row_count: int = 1) -> list[dict]: def manifest_rows(database: str, *, target: bool, row_count: int = 1) -> list[dict]:
@ -94,6 +96,65 @@ def args_for(tmp_path: Path, **overrides) -> argparse.Namespace:
dump.write_bytes(b"PGDMPfixture") dump.write_bytes(b"PGDMPfixture")
source_manifest = tmp_path / "source-manifest.jsonl" source_manifest = tmp_path / "source-manifest.jsonl"
source_manifest.write_text(manifest_text("teleo", target=False), encoding="utf-8") source_manifest.write_text(manifest_text("teleo", target=False), encoding="utf-8")
manifest_sql = Path("ops/postgres_parity_manifest.sql").resolve()
source_context_sha256 = "4" * 64
snapshot = {
"exported_snapshot_id": "00000003-000001A2-1",
"txid_snapshot": "100:120:105,111",
"wal_lsn": "0/16B6C50",
"system_identifier": "7649789040005668902",
"captured_at_utc": "2026-07-15T01:02:03.456789+00:00",
}
source = {
"ssh_target": "root@77.42.65.182",
"container": "teleo-pg",
"database": "teleo",
"service": "leoclean-gateway.service",
}
dump_sha256 = hashlib.sha256(dump.read_bytes()).hexdigest()
source_manifest_sha256 = hashlib.sha256(source_manifest.read_bytes()).hexdigest()
manifest_sql_sha256 = hashlib.sha256(manifest_sql.read_bytes()).hexdigest()
source_receipt = tmp_path / "source-receipt.json"
source_receipt.write_text(
json.dumps(
{
"artifact": "vps_canonical_postgres_exported_snapshot",
"schema": restore.SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"receipt_version": 2,
"generated_at_utc": "2026-07-15T01:02:04+00:00",
"status": "pass",
"run_id": "canonical-20260715",
"capture_authorization_ref": AUTHORIZATION_REF,
"source": source,
"snapshot_exported": True,
"snapshot": snapshot,
"dump": {"path": "/private/source.dump", "bytes": dump.stat().st_size, "sha256": dump_sha256},
"manifest": {
"path": "/private/source-manifest.jsonl",
"sha256": source_manifest_sha256,
"manifest_sql_sha256": manifest_sql_sha256,
"table_count": 1,
"total_rows": 1,
},
"source_context": {"path": "/private/source-context.txt", "sha256": source_context_sha256},
"source_service": {"before": SERVICE_STATE, "after": SERVICE_STATE, "unchanged": True},
"provenance_binding": build_provenance_binding(
run_id="canonical-20260715",
authorization_ref=AUTHORIZATION_REF,
source=source,
snapshot=snapshot,
dump_sha256=dump_sha256,
manifest_sql_sha256=manifest_sql_sha256,
source_manifest_sha256=source_manifest_sha256,
source_context_sha256=source_context_sha256,
),
"service_unchanged": True,
"production_db_mutated": False,
"telegram_send_attempted": False,
}
),
encoding="utf-8",
)
values = { values = {
"operation": "restore", "operation": "restore",
"execute": True, "execute": True,
@ -101,15 +162,26 @@ def args_for(tmp_path: Path, **overrides) -> argparse.Namespace:
"run_id": "gcp-restore-test12345", "run_id": "gcp-restore-test12345",
"host": restore.DEFAULT_HOST, "host": restore.DEFAULT_HOST,
"project": restore.DEFAULT_PROJECT, "project": restore.DEFAULT_PROJECT,
"cloudsql_instance": restore.DEFAULT_INSTANCE,
"expected_private_network": restore.DEFAULT_PRIVATE_NETWORK,
"expected_compute_instance": restore.DEFAULT_COMPUTE_INSTANCE,
"expected_compute_zone": restore.DEFAULT_COMPUTE_ZONE,
"expected_source_ssh_target": restore.DEFAULT_SOURCE_SSH_TARGET,
"expected_source_container": restore.DEFAULT_SOURCE_CONTAINER,
"expected_source_database": restore.DEFAULT_SOURCE_DATABASE,
"expected_source_service": restore.DEFAULT_SOURCE_SERVICE,
"password_secret": restore.DEFAULT_SECRET, "password_secret": restore.DEFAULT_SECRET,
"service": restore.DEFAULT_SERVICE, "service": restore.DEFAULT_SERVICE,
"command_timeout": 2.0, "command_timeout": 2.0,
"psql_bin": "psql", "psql_bin": "psql",
"run_root": tmp_path / "runs", "run_root": tmp_path / "runs",
"dump": dump, "dump": dump,
"expected_dump_sha256": hashlib.sha256(dump.read_bytes()).hexdigest(), "expected_dump_sha256": dump_sha256,
"source_manifest": source_manifest, "source_manifest": source_manifest,
"manifest_sql": Path("ops/postgres_parity_manifest.sql").resolve(), "source_receipt": source_receipt,
"expected_source_receipt_sha256": hashlib.sha256(source_receipt.read_bytes()).hexdigest(),
"expected_capture_authorization_ref": AUTHORIZATION_REF,
"manifest_sql": manifest_sql,
"pg_restore_bin": "pg_restore", "pg_restore_bin": "pg_restore",
"restore_timeout": 5.0, "restore_timeout": 5.0,
"manifest_timeout": 5.0, "manifest_timeout": 5.0,
@ -128,6 +200,54 @@ def install_restore_fakes(
state = {"exists": False, "dropped": False} state = {"exists": False, "dropped": False}
monkeypatch.setattr(restore.os, "geteuid", lambda: 0) monkeypatch.setattr(restore.os, "geteuid", lambda: 0)
monkeypatch.setattr(restore, "service_state", lambda *_args, **_kwargs: dict(SERVICE_STATE)) monkeypatch.setattr(restore, "service_state", lambda *_args, **_kwargs: dict(SERVICE_STATE))
monkeypatch.setattr(
restore,
"gce_compute_identity",
lambda project, instance, zone, expected_private_network, **_kwargs: {
"project": project,
"project_number": "785938879453",
"instance": instance,
"instance_id": "123456789",
"zone": zone,
"expected_private_network": expected_private_network,
"private_ip": "10.61.0.2",
"network_resource": "projects/123456789/networks/teleo-staging-net",
"service_account": "teleo-runtime@teleo-501523.iam.gserviceaccount.com",
"checks": {
"project": True,
"project_number": True,
"expected_network_project": True,
"instance": True,
"instance_id": True,
"zone": True,
"network": True,
"private_ip": True,
"service_account": True,
},
},
)
monkeypatch.setattr(
restore,
"cloudsql_control_plane_state",
lambda project, instance, host, expected_private_network, **_kwargs: {
"project": project,
"instance": instance,
"private_addresses": [host],
"expected_private_host": host,
"private_network": expected_private_network,
"expected_private_network": expected_private_network,
"public_ip_disabled": True,
"checks": {
"instance": True,
"state": True,
"postgres_16": True,
"public_ip_disabled": True,
"private_network": True,
"expected_private_host": True,
"no_non_private_address": True,
},
},
)
monkeypatch.setattr(restore, "resolve_password", lambda *_args, **_kwargs: "private-password") monkeypatch.setattr(restore, "resolve_password", lambda *_args, **_kwargs: "private-password")
monkeypatch.setattr( monkeypatch.setattr(
restore, restore,
@ -179,7 +299,10 @@ def test_restore_success_retains_only_exact_parity_clone(
assert result["phase"] == "retained_for_no_send_replay" assert result["phase"] == "retained_for_no_send_replay"
assert result["parity"]["problems"] == [] assert result["parity"]["problems"] == []
assert result["parity"]["details"]["source_total_rows"] == 1 assert result["parity"]["details"]["source_total_rows"] == 1
assert result["source"]["capture_receipt"]["schema"] == restore.SOURCE_SNAPSHOT_RECEIPT_SCHEMA
assert result["source"]["capture_receipt"]["provenance_binding_sha256"]
assert result["target"]["connectivity"]["private_connectivity"] is True assert result["target"]["connectivity"]["private_connectivity"] is True
assert result["target"]["cloudsql"]["public_ip_disabled"] is True
assert result["target"]["connectivity"]["ssl"] is True assert result["target"]["connectivity"]["ssl"] is True
assert result["live_service"]["unchanged"] is True assert result["live_service"]["unchanged"] is True
assert result["cleanup"]["clone_database_remaining"] == 1 assert result["cleanup"]["clone_database_remaining"] == 1
@ -187,6 +310,15 @@ def test_restore_success_retains_only_exact_parity_clone(
run_dir = args.run_root / args.run_id run_dir = args.run_root / args.run_id
assert (run_dir / "target-manifest.jsonl").stat().st_mode & 0o777 == 0o600 assert (run_dir / "target-manifest.jsonl").stat().st_mode & 0o777 == 0o600
assert (run_dir / "restore-receipt.json").stat().st_mode & 0o777 == 0o600 assert (run_dir / "restore-receipt.json").stat().st_mode & 0o777 == 0o600
connectivity_path = run_dir / "gcp-private-connectivity.json"
assert connectivity_path.stat().st_mode & 0o777 == 0o600
connectivity = json.loads(connectivity_path.read_text())
assert connectivity["target_database"] == args.target_db
assert connectivity["source_compute"] == restore.DEFAULT_COMPUTE_INSTANCE
assert connectivity["public_ip_disabled"] is True
assert (
result["target"]["connectivity_proof"]["sha256"] == hashlib.sha256(connectivity_path.read_bytes()).hexdigest()
)
def test_parity_failure_drops_generated_clone_and_receipts_both_failures( def test_parity_failure_drops_generated_clone_and_receipts_both_failures(
@ -234,6 +366,79 @@ def test_create_command_failure_still_drops_a_committed_clone(
assert state == {"exists": False, "dropped": True} assert state == {"exists": False, "dropped": True}
def test_restore_rejects_manifest_not_bound_to_source_receipt_before_credentials(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
args = args_for(tmp_path)
install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True))
receipt = json.loads(args.source_receipt.read_text())
receipt["manifest"]["sha256"] = "0" * 64
args.source_receipt.write_text(json.dumps(receipt))
args.expected_source_receipt_sha256 = hashlib.sha256(args.source_receipt.read_bytes()).hexdigest()
credential_called = False
def unexpected_credential(*_args, **_kwargs):
nonlocal credential_called
credential_called = True
return "private-password"
monkeypatch.setattr(restore, "resolve_password", unexpected_credential)
result = restore.run_restore(args)
assert result["status"] == "fail"
assert result["error"]["phase"] == "validation"
assert "source_manifest_sha256" in result["error"]["message"]
assert credential_called is False
def test_restore_rejects_unexpected_capture_authorization_before_credentials(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
args = args_for(tmp_path, expected_capture_authorization_ref="codex-delegation:another-authorized-run")
install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True))
credential_called = False
def unexpected_credential(*_args, **_kwargs):
nonlocal credential_called
credential_called = True
return "private-password"
monkeypatch.setattr(restore, "resolve_password", unexpected_credential)
result = restore.run_restore(args)
assert result["status"] == "fail"
assert result["error"]["phase"] == "validation"
assert "capture_authorization_ref" in result["error"]["message"]
assert credential_called is False
def test_restore_rejects_noncanonical_source_identity_before_credentials(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
args = args_for(tmp_path, expected_source_container="different-postgres")
install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True))
credential_called = False
def unexpected_credential(*_args, **_kwargs):
nonlocal credential_called
credential_called = True
return "private-password"
monkeypatch.setattr(restore, "resolve_password", unexpected_credential)
result = restore.run_restore(args)
assert result["status"] == "fail"
assert result["error"]["phase"] == "validation"
assert "source_identity" in result["error"]["message"]
assert credential_called is False
def test_changed_live_service_turns_successful_restore_into_cleanup( def test_changed_live_service_turns_successful_restore_into_cleanup(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
@ -283,6 +488,14 @@ def test_cleanup_is_bound_to_passing_restore_receipt(
"run_id", "run_id",
"host", "host",
"project", "project",
"cloudsql_instance",
"expected_private_network",
"expected_compute_instance",
"expected_compute_zone",
"expected_source_ssh_target",
"expected_source_container",
"expected_source_database",
"expected_source_service",
"password_secret", "password_secret",
"service", "service",
"command_timeout", "command_timeout",
@ -302,6 +515,106 @@ def test_cleanup_is_bound_to_passing_restore_receipt(
assert state == {"exists": False, "dropped": True} assert state == {"exists": False, "dropped": True}
def test_cleanup_rejects_target_topology_drift_before_credentials(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
args = args_for(tmp_path)
state = install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True))
assert restore.run_restore(args)["status"] == "pass"
cleanup_args = argparse.Namespace(
**{
key: getattr(args, key)
for key in (
"target_db",
"run_id",
"host",
"project",
"cloudsql_instance",
"expected_private_network",
"expected_compute_instance",
"expected_compute_zone",
"expected_source_ssh_target",
"expected_source_container",
"expected_source_database",
"expected_source_service",
"password_secret",
"service",
"command_timeout",
"psql_bin",
"run_root",
)
},
operation="cleanup",
execute=True,
)
cleanup_args.host = "10.61.0.4"
credential_called = False
def unexpected_credential(*_args, **_kwargs):
nonlocal credential_called
credential_called = True
return "private-password"
monkeypatch.setattr(restore, "resolve_password", unexpected_credential)
with pytest.raises(restore.RestoreError, match="execution contract"):
restore.run_cleanup(cleanup_args)
assert credential_called is False
assert state == {"exists": True, "dropped": False}
def test_cleanup_retains_failure_receipt_when_post_drop_readback_fails(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
args = args_for(tmp_path)
state = install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True))
assert restore.run_restore(args)["status"] == "pass"
cleanup_args = argparse.Namespace(
**{
key: getattr(args, key)
for key in (
"target_db",
"run_id",
"host",
"project",
"cloudsql_instance",
"expected_private_network",
"expected_compute_instance",
"expected_compute_zone",
"expected_source_ssh_target",
"expected_source_container",
"expected_source_database",
"expected_source_service",
"password_secret",
"service",
"command_timeout",
"psql_bin",
"run_root",
)
},
operation="cleanup",
execute=True,
)
monkeypatch.setattr(
restore,
"rollback_database_state",
lambda *_args, **_kwargs: (_ for _ in ()).throw(restore.RestoreError("rollback readback unavailable")),
)
result = restore.run_cleanup(cleanup_args)
assert result["status"] == "fail"
assert result["error"]["phase"] == "rollback_readback"
assert result["database"]["clone_database_remaining"] == 0
assert state == {"exists": False, "dropped": True}
receipt_path = args.run_root / args.run_id / "cleanup-receipt.json"
assert receipt_path.stat().st_mode & 0o777 == 0o600
assert json.loads(receipt_path.read_text())["error"]["phase"] == "rollback_readback"
def test_restore_dump_uses_tls_no_owner_no_acl_and_keeps_secret_out_of_argv( def test_restore_dump_uses_tls_no_owner_no_acl_and_keeps_secret_out_of_argv(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
@ -326,6 +639,110 @@ def test_restore_dump_uses_tls_no_owner_no_acl_and_keeps_secret_out_of_argv(
assert captured["env"]["PGPASSWORD"] == "private-password" assert captured["env"]["PGPASSWORD"] == "private-password"
def test_cloudsql_control_plane_preflight_requires_private_only_expected_host(
monkeypatch: pytest.MonkeyPatch,
) -> None:
payload = {
"name": "teleo-pgvector-standby",
"state": "RUNNABLE",
"databaseVersion": "POSTGRES_16",
"region": "europe-west6",
"settings": {
"tier": "db-custom-1-3840",
"ipConfiguration": {
"ipv4Enabled": False,
"privateNetwork": "projects/teleo-501523/global/networks/teleo-staging-net",
},
},
"ipAddresses": [{"type": "PRIVATE", "ipAddress": "10.61.0.3"}],
}
monkeypatch.setattr(
restore,
"_run",
lambda *_args, **_kwargs: subprocess.CompletedProcess([], 0, json.dumps(payload), ""),
)
result = restore.cloudsql_control_plane_state(
"teleo-501523",
"teleo-pgvector-standby",
"10.61.0.3",
restore.DEFAULT_PRIVATE_NETWORK,
timeout=2.0,
)
assert result["public_ip_disabled"] is True
assert result["private_addresses"] == ["10.61.0.3"]
payload["settings"]["ipConfiguration"]["ipv4Enabled"] = True
payload["ipAddresses"].append({"type": "PRIMARY", "ipAddress": "34.1.2.3"})
with pytest.raises(restore.RestoreError, match="no_non_private_address, public_ip_disabled"):
restore.cloudsql_control_plane_state(
"teleo-501523",
"teleo-pgvector-standby",
"10.61.0.3",
restore.DEFAULT_PRIVATE_NETWORK,
timeout=2.0,
)
def test_gce_compute_identity_is_bound_to_metadata_instance_zone_and_network(
monkeypatch: pytest.MonkeyPatch,
) -> None:
metadata = {
"project/project-id": "teleo-501523",
"project/numeric-project-id": "785938879453",
"instance/name": "teleo-prod-1",
"instance/id": "123456789",
"instance/zone": "projects/123456789/zones/europe-west6-a",
"instance/network-interfaces/0/network": "projects/785938879453/networks/teleo-staging-net",
"instance/network-interfaces/0/ip": "10.61.0.2",
"instance/service-accounts/default/email": "teleo-runtime@teleo-501523.iam.gserviceaccount.com",
}
monkeypatch.setattr(restore, "gce_metadata_value", lambda path, **_kwargs: metadata[path])
result = restore.gce_compute_identity(
"teleo-501523",
"teleo-prod-1",
"europe-west6-a",
restore.DEFAULT_PRIVATE_NETWORK,
timeout=2.0,
)
assert result["instance_id"] == "123456789"
assert result["project_number"] == "785938879453"
assert result["zone"] == "europe-west6-a"
assert all(result["checks"].values())
with pytest.raises(restore.RestoreError, match="instance"):
restore.gce_compute_identity(
"teleo-501523",
"different-instance",
"europe-west6-a",
restore.DEFAULT_PRIVATE_NETWORK,
timeout=2.0,
)
metadata["instance/name"] = "teleo-prod-1"
metadata["instance/network-interfaces/0/network"] = "projects/999999999999/networks/teleo-staging-net"
with pytest.raises(restore.RestoreError, match="network"):
restore.gce_compute_identity(
"teleo-501523",
"teleo-prod-1",
"europe-west6-a",
restore.DEFAULT_PRIVATE_NETWORK,
timeout=2.0,
)
metadata["instance/network-interfaces/0/network"] = "projects/785938879453/networks/teleo-staging-net"
with pytest.raises(restore.RestoreError, match="expected_network_project"):
restore.gce_compute_identity(
"teleo-501523",
"teleo-prod-1",
"europe-west6-a",
"projects/other-project/global/networks/teleo-staging-net",
timeout=2.0,
)
def test_pg_restore_preflight_rejects_client_older_than_snapshot( def test_pg_restore_preflight_rejects_client_older_than_snapshot(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,

View file

@ -0,0 +1,703 @@
import hashlib
import json
from datetime import UTC, datetime
from pathlib import Path
from ops.verify_gcp_canonical_lifecycle import EXPECTED_BLIND_ROW_IDS
from ops.verify_gcp_canonical_lifecycle import verify_lifecycle as _verify_lifecycle
VERIFY_NOW = datetime(2026, 7, 15, 1, 3, tzinfo=UTC)
def verify_lifecycle(**kwargs):
return _verify_lifecycle(**kwargs, now=VERIFY_NOW)
def write_json(path: Path, payload: dict) -> Path:
path.write_text(json.dumps(payload, sort_keys=True) + "\n", encoding="utf-8")
return path
def sha256(path: Path) -> str:
return hashlib.sha256(path.read_bytes()).hexdigest()
def lifecycle_receipts(tmp_path: Path) -> dict[str, Path]:
target_db = "teleo_clone_current_test"
run_id = "gcp-restore-current-test123"
healthy_service = {
"ActiveState": "active",
"SubState": "running",
"MainPID": "148735",
"NRestarts": "0",
}
structural_hashes = {
kind: {"source": format(index, "x") * 64, "target": format(index, "x") * 64}
for index, kind in enumerate(
(
"schemas",
"columns",
"constraints",
"indexes",
"sequences",
"views",
"functions",
"triggers",
"types",
"policies",
),
1,
)
}
exact_parity_details = {
"source_database": "teleo",
"target_database": target_db,
"source_table_count": 39,
"target_table_count": 39,
"source_total_rows": 52167,
"target_total_rows": 52167,
"table_mismatches": [],
"structural_hashes": structural_hashes,
"required_extension_mismatches": {},
"application_role_mismatches": {},
"target_extra_application_roles": [],
"performance": [{"query": "count_claims", "source_ms": 1.0, "target_ms": 2.0, "source_ratio": 0.4}],
"performance_problems": [],
}
source = {
"artifact": "vps_canonical_postgres_exported_snapshot",
"schema": "livingip.sourceSnapshotReceipt.v2",
"receipt_version": 2,
"status": "pass",
"run_id": "canonical-current-test",
"capture_authorization_ref": "codex-delegation:test1234",
"source": {
"ssh_target": "root@77.42.65.182",
"container": "teleo-pg",
"database": "teleo",
"service": "leoclean-gateway.service",
},
"snapshot_exported": True,
"snapshot": {
"captured_at_utc": "2026-07-15T00:59:00+00:00",
"exported_snapshot_id": "00000003-000001A2-1",
"txid_snapshot": "100:120:",
"wal_lsn": "0/16B6C50",
"system_identifier": "7649789040005668902",
},
"service_unchanged": True,
"source_service": {"before": healthy_service, "after": healthy_service, "unchanged": True},
"production_db_mutated": False,
"telegram_send_attempted": False,
"dump": {"sha256": "a" * 64},
"manifest": {"sha256": "b" * 64, "table_count": 39, "total_rows": 52167},
"provenance_binding": {"algorithm": "sha256", "payload": {"bound": True}, "sha256": "c" * 64},
}
source_path = write_json(tmp_path / "source.json", source)
restore = {
"artifact": "gcp_generated_postgres_snapshot_restore",
"status": "pass",
"generated_at_utc": "2026-07-15T00:59:10+00:00",
"completed_at_utc": "2026-07-15T00:59:30+00:00",
"run_id": run_id,
"target_database": target_db,
"source": {
"capture_receipt": {
"sha256": sha256(source_path),
"capture_authorization_ref": "codex-delegation:test1234",
"provenance_binding_sha256": "c" * 64,
"dump_sha256": "a" * 64,
"manifest_sha256": "b" * 64,
"checks": {
name: True
for name in (
"artifact",
"schema",
"receipt_version",
"status",
"snapshot_exported",
"service_unchanged",
"source_service_healthy",
"source_not_mutated",
"telegram_not_sent",
"run_id",
"capture_authorization_ref",
"source_identity",
"source_database",
"snapshot_identity",
"dump_sha256",
"dump_bytes",
"source_manifest_sha256",
"manifest_sql_sha256",
"source_context_sha256",
"table_count",
"total_rows",
"binding_algorithm",
"binding_payload",
"binding_sha256",
)
},
}
},
"execution_contract": {
"project": "teleo-501523",
"cloudsql_instance": "teleo-pgvector-standby",
"host": "10.61.0.3",
"private_network": "projects/teleo-501523/global/networks/teleo-staging-net",
"compute_instance": "teleo-prod-1",
"compute_zone": "europe-west6-a",
"source": source["source"],
"password_secret": "gcp-teleo-pgvector-standby-postgres-password",
"service": "leoclean-gcp-prod-parallel.service",
},
"target": {
"manifest_sha256": "d" * 64,
"connectivity": {
"private_connectivity": True,
"ssl": True,
"server_address": "10.61.0.3",
"source_compute": "teleo-prod-1",
},
"cloudsql": {
"project": "teleo-501523",
"instance": "teleo-pgvector-standby",
"expected_private_host": "10.61.0.3",
"private_network": "projects/teleo-501523/global/networks/teleo-staging-net",
"expected_private_network": "projects/teleo-501523/global/networks/teleo-staging-net",
"public_ip_disabled": True,
"checks": {
"instance": True,
"state": True,
"postgres_16": True,
"public_ip_disabled": True,
"private_network": True,
"expected_private_host": True,
"no_non_private_address": True,
},
},
"compute": {
"project": "teleo-501523",
"project_number": "785938879453",
"instance": "teleo-prod-1",
"instance_id": "123456789",
"zone": "europe-west6-a",
"expected_private_network": "projects/teleo-501523/global/networks/teleo-staging-net",
"network_resource": "projects/123456789/networks/teleo-staging-net",
"private_ip": "10.61.0.2",
"service_account": "teleo-runtime@teleo-501523.iam.gserviceaccount.com",
"checks": {
"project": True,
"project_number": True,
"expected_network_project": True,
"instance": True,
"instance_id": True,
"zone": True,
"network": True,
"private_ip": True,
"service_account": True,
},
},
"connectivity_proof": {
"sha256": "f" * 64,
"schema": "livingip.gcpPrivatePostgresConnectivity.v2",
},
},
"parity": {"status": "pass", "problems": [], "details": exact_parity_details},
"live_service": {"before": healthy_service, "after": healthy_service, "unchanged": True},
}
restore_path = write_json(tmp_path / "restore.json", restore)
parity = {
"artifact": "canonical_postgres_parity_verification",
"status": "pass",
"completed_at_utc": "2026-07-15T00:59:45+00:00",
"scope": "gcp_staging",
"problems": [],
"source_manifest_sha256": "b" * 64,
"target_manifest_sha256": "d" * 64,
"connectivity_proof_sha256": "f" * 64,
"details": exact_parity_details,
"private_connectivity": {
"required": True,
"proof": {
"artifact": "gcp_private_postgres_connectivity",
"schema": "livingip.gcpPrivatePostgresConnectivity.v2",
"status": "pass",
"restore_run_id": run_id,
"private_connectivity": True,
"public_ip_disabled": True,
"target_database": target_db,
"source_compute": "teleo-prod-1",
"source_compute_instance_id": "123456789",
"source_compute_zone": "europe-west6-a",
"source_compute_private_ip": "10.61.0.2",
"server_address": "10.61.0.3",
"project": "teleo-501523",
"cloudsql_instance": "teleo-pgvector-standby",
"private_network": "projects/teleo-501523/global/networks/teleo-staging-net",
"cloudsql_control_plane_checks": restore["target"]["cloudsql"]["checks"],
"compute_identity_checks": restore["target"]["compute"]["checks"],
},
},
}
parity_path = write_json(tmp_path / "parity.json", parity)
required_checks = {
name: True
for name in (
"exact_blind_prompt_without_ids",
"claim_and_sources_retrieved",
"discovery_show_evidence_completed",
"retrieval_receipts_proven",
"private_tls_read_only_target",
"generated_database_unchanged",
"canonical_counts_unchanged",
"database_calls_read_only",
"no_database_write",
"no_telegram_send",
"live_service_unchanged",
"live_profile_unchanged",
"temporary_profile_removed",
)
}
reasoning = {
"schema": "livingip.gcpGeneratedDbBlindClaimCanary.v1",
"status": "pass",
"generated_at_utc": "2026-07-15T01:00:00+00:00",
"completed_at_utc": "2026-07-15T01:01:00+00:00",
"mode": "gcp_generated_db_gatewayrunner_blind_claim_no_send",
"source_compute": "teleo-prod-1",
"prompt": "Inspect the live claim without me giving you an ID. Do not change the database.",
"errors": [],
"target_database": target_db,
"posted_to_telegram": False,
"database_write_attempted": False,
"parity_receipt": {"sha256": sha256(parity_path)},
"checks": required_checks,
"database_fingerprint_before": {"sha256": "e" * 64},
"database_fingerprint_after": {"sha256": "e" * 64},
"canonical_status_before": {"claims": 1},
"canonical_status_after": {"claims": 1},
"database_identity_before": {"database": target_db},
"database_identity_after": {"database": target_db},
"service_before": healthy_service,
"service_after": healthy_service,
"temp_profile_absent": True,
"temp_profile_removed": True,
"outcomes": {
"asks_for_user_iteration_before_live": True,
"challenges_weak_support": True,
"decomposes_the_bundled_legal_claim": True,
"does_not_claim_a_database_change": True,
"proposes_candidate_claims": True,
"uses_only_m3taversal_handle": True,
},
"result": {
"reply": "The claim is weak; here are grounded, review-first alternatives.",
"database_tool_trace": {
"schema": "livingip.leoKbToolTrace.v1",
"database_tool_call_proven": True,
"database_retrieval_receipt_proven": True,
"database_tool_calls_read_only": True,
"database_tool_call_count": 3,
"database_tool_completed_count": 3,
"calls": [
{
"database_invocations": [{"access_mode": "read_only", "subcommand": subcommand}],
"result": {
"nonempty": True,
"error_detected": False,
"row_ids": sorted(EXPECTED_BLIND_ROW_IDS),
"row_id_count": len(EXPECTED_BLIND_ROW_IDS),
"retrieval_receipt": {
"schema": "livingip.teleoKbRetrievalReceipt.v1",
"semantic_context_sha256": "1" * 64,
"artifact_state_sha256": "2" * 64,
"read_consistency_status": "stable_wal_marker",
},
},
}
for subcommand in ("search", "show", "evidence")
],
},
"gateway": {
"authorized": True,
"handler_invoked": True,
"model_free_fallback_used": False,
"posted_to_telegram": False,
"child_process": {
"exitcode": 0,
"alive_after_readback": False,
"process_group_alive_after_readback": False,
"timed_out": False,
},
"tool_surface": {"send_message_tool_enabled": False},
},
},
"wrapper_tool_proof": {
"all_bound_to_supplied_target": True,
"all_completed_successfully": True,
"complete_start_end_pairing": True,
"database_read_only_required": True,
"default_read_only_required": True,
"invocations": [
{
"database": target_db,
"returncode": 0,
"subcommand": subcommand,
"database_identity": {
"transaction_read_only": "on",
"default_transaction_read_only": "on",
"ssl": True,
"server_address": "10.61.0.3",
},
}
for subcommand in ("search", "show", "evidence")
],
},
}
reasoning_path = write_json(tmp_path / "reasoning.json", reasoning)
reasoning_compute = {
"artifact": "gcp_reasoning_compute_attestation",
"schema": "livingip.gcpReasoningComputeAttestation.v1",
"status": "pass",
"generated_at_utc": "2026-07-15T01:01:05+00:00",
"restore_run_id": run_id,
"target_database": target_db,
"reasoning_receipt": {
"sha256": sha256(reasoning_path),
"schema": reasoning["schema"],
"generated_at_utc": reasoning["generated_at_utc"],
"completed_at_utc": reasoning["completed_at_utc"],
"claimed_source_compute": reasoning["source_compute"],
},
"compute": restore["target"]["compute"],
"checks": {
"schema": True,
"status": True,
"target_database": True,
"source_compute": True,
"chronology": True,
"no_telegram_send": True,
"no_database_write": True,
"gce_metadata_identity": True,
},
"method": "gce_metadata_server_v1",
"mutation": {
"cloudsql_changed": False,
"compute_changed": False,
"database_changed": False,
"telegram_message_sent": False,
},
}
reasoning_compute_path = write_json(tmp_path / "reasoning-compute.json", reasoning_compute)
current_source = json.loads(json.dumps(source))
current_source["run_id"] = "canonical-current-postflight"
current_source["capture_authorization_ref"] = "codex-delegation:postflight1234"
current_source["snapshot"]["captured_at_utc"] = "2026-07-15T01:02:00+00:00"
current_source["snapshot"]["exported_snapshot_id"] = "00000003-000001B2-1"
current_source_path = write_json(tmp_path / "current-source.json", current_source)
source_delta = {
"artifact": "vps_canonical_snapshot_postflight_delta",
"schema": "livingip.vpsCanonicalSnapshotDelta.v1",
"status": "pass",
"baseline": {
"run_id": "canonical-current-test",
"capture_receipt_sha256": sha256(source_path),
"manifest_sha256": "b" * 64,
"capture_authorization_ref": "codex-delegation:test1234",
"captured_at_utc": "2026-07-15T00:59:00+00:00",
"table_count": 39,
"total_rows": 52167,
"table_fingerprint_sha256": "d" * 64,
"structure_fingerprint_sha256": "e" * 64,
},
"current": {
"run_id": "canonical-current-postflight",
"capture_receipt_sha256": sha256(current_source_path),
"manifest_sha256": "b" * 64,
"capture_authorization_ref": "codex-delegation:postflight1234",
"captured_at_utc": "2026-07-15T01:02:00+00:00",
"table_count": 39,
"total_rows": 52167,
"table_fingerprint_sha256": "d" * 64,
"structure_fingerprint_sha256": "e" * 64,
},
"delta": {
"delta_detected": False,
"added_tables": [],
"removed_tables": [],
"changed_tables": [],
"changed_structures": [],
"total_row_delta": 0,
},
}
source_delta_path = write_json(tmp_path / "source-delta.json", source_delta)
cleanup = {
"artifact": "gcp_generated_postgres_snapshot_cleanup",
"status": "pass",
"generated_at_utc": "2026-07-15T01:01:10+00:00",
"completed_at_utc": "2026-07-15T01:01:30+00:00",
"run_id": run_id,
"target_database": target_db,
"database": {"clone_database_remaining": 0},
"live_service": {"before": healthy_service, "after": healthy_service, "unchanged": True},
"restore_receipt": {"sha256": sha256(restore_path)},
"execution_contract": restore["execution_contract"],
"target": {
"cloudsql": restore["target"]["cloudsql"],
"compute": restore["target"]["compute"],
},
}
cleanup_path = write_json(tmp_path / "cleanup.json", cleanup)
return {
"source_path": source_path,
"current_source_path": current_source_path,
"source_delta_path": source_delta_path,
"restore_path": restore_path,
"parity_path": parity_path,
"reasoning_path": reasoning_path,
"reasoning_compute_path": reasoning_compute_path,
"cleanup_path": cleanup_path,
}
def test_bound_live_lifecycle_passes(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
result = verify_lifecycle(**paths)
assert result["status"] == "pass"
assert result["failed_checks"] == []
assert result["checks"]["clone_absent"] is True
def test_local_parity_or_missing_cleanup_fails_closed(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
parity = json.loads(paths["parity_path"].read_text())
parity["scope"] = "local_restore"
write_json(paths["parity_path"], parity)
reasoning = json.loads(paths["reasoning_path"].read_text())
reasoning["parity_receipt"]["sha256"] = sha256(paths["parity_path"])
write_json(paths["reasoning_path"], reasoning)
cleanup = json.loads(paths["cleanup_path"].read_text())
cleanup["database"]["clone_database_remaining"] = 1
write_json(paths["cleanup_path"], cleanup)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert {"parity_pass", "clone_absent"} <= set(result["failed_checks"])
def test_mismatched_source_or_reasoning_receipt_fails_closed(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
restore = json.loads(paths["restore_path"].read_text())
restore["source"]["capture_receipt"]["sha256"] = "0" * 64
write_json(paths["restore_path"], restore)
reasoning = json.loads(paths["reasoning_path"].read_text())
reasoning["checks"]["claim_and_sources_retrieved"] = False
write_json(paths["reasoning_path"], reasoning)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert {"source_receipt_hash_bound", "reasoning_required_checks"} <= set(result["failed_checks"])
def test_hand_authored_reasoning_boole_without_producer_evidence_fail(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
reasoning = json.loads(paths["reasoning_path"].read_text())
reasoning.pop("result")
reasoning.pop("wrapper_tool_proof")
write_json(paths["reasoning_path"], reasoning)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "reasoning_producer_evidence" in result["failed_checks"]
assert result["reasoning_producer_checks"]["tool_trace"] is False
def test_reasoning_compute_attestation_must_bind_live_metadata(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
attestation = json.loads(paths["reasoning_compute_path"].read_text())
attestation["compute"]["instance_id"] = "999999999"
write_json(paths["reasoning_compute_path"], attestation)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "reasoning_compute_identity_bound" in result["failed_checks"]
def test_source_postflight_must_be_newer_than_cleanup_completion(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
cleanup = json.loads(paths["cleanup_path"].read_text())
cleanup["completed_at_utc"] = "2026-07-15T02:00:00+00:00"
write_json(paths["cleanup_path"], cleanup)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "source_postflight_after_lifecycle" in result["failed_checks"]
def test_parity_must_retain_exact_structure_roles_rows_and_performance(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
parity = json.loads(paths["parity_path"].read_text())
parity["details"]["structural_hashes"].pop("constraints")
parity["details"]["application_role_mismatches"] = {"kb_apply": {"missing": True}}
parity["details"]["performance"] = []
write_json(paths["parity_path"], parity)
reasoning = json.loads(paths["reasoning_path"].read_text())
reasoning["parity_receipt"]["sha256"] = sha256(paths["parity_path"])
write_json(paths["reasoning_path"], reasoning)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "parity_exact_evidence" in result["failed_checks"]
assert result["parity_evidence_checks"]["structural_hashes"] is False
assert result["parity_evidence_checks"]["roles"] is False
assert result["parity_evidence_checks"]["performance"] is False
def test_reasoning_must_retrieve_expected_claim_sources_and_preserve_handle(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
reasoning = json.loads(paths["reasoning_path"].read_text())
for call in reasoning["result"]["database_tool_trace"]["calls"]:
call["result"]["row_ids"] = []
call["result"]["row_id_count"] = 0
reasoning["result"]["reply"] = "Cory found a weak claim and two sources."
write_json(paths["reasoning_path"], reasoning)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "reasoning_producer_evidence" in result["failed_checks"]
assert result["reasoning_producer_checks"]["expected_claim_and_sources_retrieved"] is False
assert result["reasoning_producer_checks"]["reply_uses_only_m3taversal_handle"] is False
def test_lifecycle_requires_restore_parity_reasoning_cleanup_chronology(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
cleanup = json.loads(paths["cleanup_path"].read_text())
cleanup["generated_at_utc"] = "2026-07-15T00:58:00+00:00"
write_json(paths["cleanup_path"], cleanup)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "lifecycle_chronology" in result["failed_checks"]
def test_source_delta_requires_real_fingerprints_and_consistent_no_delta(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
delta = json.loads(paths["source_delta_path"].read_text())
delta["baseline"].pop("table_fingerprint_sha256")
delta["current"]["total_rows"] += 1
write_json(paths["source_delta_path"], delta)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert {"source_delta_baseline_shape", "source_delta_current_shape", "source_delta_consistent"} <= set(
result["failed_checks"]
)
def test_detected_source_delta_requires_exact_row_arithmetic(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
delta = json.loads(paths["source_delta_path"].read_text())
delta["delta"]["delta_detected"] = True
delta["delta"]["changed_tables"] = [{"table": "public.claims"}]
delta["delta"]["total_row_delta"] = 99
write_json(paths["source_delta_path"], delta)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "source_delta_consistent" in result["failed_checks"]
def test_postflight_source_requires_structured_healthy_service(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
current = json.loads(paths["current_source_path"].read_text())
current["source_service"]["before"]["ActiveState"] = "failed"
current["source_service"]["after"]["ActiveState"] = "failed"
write_json(paths["current_source_path"], current)
delta = json.loads(paths["source_delta_path"].read_text())
delta["current"]["capture_receipt_sha256"] = sha256(paths["current_source_path"])
write_json(paths["source_delta_path"], delta)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "current_source_pass" in result["failed_checks"]
def test_unchanged_but_inactive_service_cannot_pass_t3(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
restore = json.loads(paths["restore_path"].read_text())
restore["live_service"]["before"]["ActiveState"] = "failed"
restore["live_service"]["after"]["ActiveState"] = "failed"
write_json(paths["restore_path"], restore)
cleanup = json.loads(paths["cleanup_path"].read_text())
cleanup["restore_receipt"]["sha256"] = sha256(paths["restore_path"])
write_json(paths["cleanup_path"], cleanup)
result = verify_lifecycle(**paths)
assert result["status"] == "fail"
assert "restore_service_healthy_unchanged" in result["failed_checks"]
def test_historical_but_monotonic_receipts_cannot_claim_current_t3(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
result = _verify_lifecycle(**paths, now=datetime(2026, 7, 15, 3, 0, tzinfo=UTC))
assert result["status"] == "fail"
assert "source_postflight_fresh" in result["failed_checks"]
assert result["postflight_age_seconds"] > result["max_postflight_age_seconds"]
def test_fresh_postflight_cannot_mask_historical_gcp_lifecycle(tmp_path: Path) -> None:
paths = lifecycle_receipts(tmp_path)
source = json.loads(paths["source_path"].read_text())
source["snapshot"]["captured_at_utc"] = "2026-07-12T00:59:00+00:00"
write_json(paths["source_path"], source)
restore = json.loads(paths["restore_path"].read_text())
restore["generated_at_utc"] = "2026-07-12T00:59:10+00:00"
restore["completed_at_utc"] = "2026-07-12T00:59:30+00:00"
restore["source"]["capture_receipt"]["sha256"] = sha256(paths["source_path"])
write_json(paths["restore_path"], restore)
parity = json.loads(paths["parity_path"].read_text())
parity["completed_at_utc"] = "2026-07-12T00:59:45+00:00"
write_json(paths["parity_path"], parity)
reasoning = json.loads(paths["reasoning_path"].read_text())
reasoning["generated_at_utc"] = "2026-07-12T01:00:00+00:00"
reasoning["completed_at_utc"] = "2026-07-12T01:01:00+00:00"
reasoning["parity_receipt"]["sha256"] = sha256(paths["parity_path"])
write_json(paths["reasoning_path"], reasoning)
attestation = json.loads(paths["reasoning_compute_path"].read_text())
attestation["generated_at_utc"] = "2026-07-12T01:01:05+00:00"
attestation["reasoning_receipt"]["sha256"] = sha256(paths["reasoning_path"])
attestation["reasoning_receipt"]["generated_at_utc"] = reasoning["generated_at_utc"]
attestation["reasoning_receipt"]["completed_at_utc"] = reasoning["completed_at_utc"]
write_json(paths["reasoning_compute_path"], attestation)
cleanup = json.loads(paths["cleanup_path"].read_text())
cleanup["generated_at_utc"] = "2026-07-12T01:01:10+00:00"
cleanup["completed_at_utc"] = "2026-07-12T01:01:30+00:00"
cleanup["restore_receipt"]["sha256"] = sha256(paths["restore_path"])
write_json(paths["cleanup_path"], cleanup)
delta = json.loads(paths["source_delta_path"].read_text())
delta["baseline"]["captured_at_utc"] = "2026-07-12T00:59:00+00:00"
delta["baseline"]["capture_receipt_sha256"] = sha256(paths["source_path"])
write_json(paths["source_delta_path"], delta)
result = verify_lifecycle(**paths)
assert result["checks"]["source_postflight_fresh"] is True
assert result["checks"]["lifecycle_chronology"] is True
assert result["status"] == "fail"
assert {"gcp_lifecycle_fresh", "lifecycle_span_bounded"} <= set(result["failed_checks"])

View file

@ -90,7 +90,10 @@ def test_matching_local_manifests_pass(tmp_path: Path) -> None:
assert completed.returncode == 0, completed.stderr assert completed.returncode == 0, completed.stderr
assert payload["status"] == "pass" assert payload["status"] == "pass"
assert payload["details"]["source_total_rows"] == 2 assert payload["details"]["source_total_rows"] == 2
assert len(payload["source_manifest_sha256"]) == 64
assert len(payload["target_manifest_sha256"]) == 64
assert payload["private_connectivity"]["status"] == "not_applicable_local_restore" assert payload["private_connectivity"]["status"] == "not_applicable_local_restore"
assert (tmp_path / "result.json").stat().st_mode & 0o777 == 0o600
def test_row_hash_and_role_mismatches_fail(tmp_path: Path) -> None: def test_row_hash_and_role_mismatches_fail(tmp_path: Path) -> None:
@ -135,6 +138,7 @@ def test_gcp_scope_accepts_matching_private_connectivity_receipt(tmp_path: Path)
assert completed.returncode == 0, completed.stderr assert completed.returncode == 0, completed.stderr
assert payload["status"] == "pass" assert payload["status"] == "pass"
assert payload["private_connectivity"]["proof"]["source_compute"] == "teleo-staging-1" assert payload["private_connectivity"]["proof"]["source_compute"] == "teleo-staging-1"
assert len(payload["connectivity_proof_sha256"]) == 64
def test_gcp_scope_rejects_receipt_for_a_different_server(tmp_path: Path) -> None: def test_gcp_scope_rejects_receipt_for_a_different_server(tmp_path: Path) -> None:

View file

@ -0,0 +1,259 @@
import hashlib
import json
from pathlib import Path
import pytest
from ops.capture_vps_canonical_postgres_snapshot import (
SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
build_provenance_binding,
)
from ops.verify_vps_canonical_snapshot_delta import verify_delta, write_private_json
def manifest_text(*, claims: int = 1, constraint_name: str = "claims_pkey") -> str:
rows = [
{
"kind": "identity",
"database": "teleo",
"server_version_num": 160014,
"transaction_read_only": "on",
},
{"kind": "schemas", "items": ["kb_stage", "public"]},
{"kind": "extensions", "items": [{"name": "pgcrypto", "version": "1.3"}]},
{"kind": "application_roles", "items": [{"name": "kb_apply", "can_login": True}]},
{"kind": "columns", "items": []},
{"kind": "constraints", "items": [{"name": constraint_name}]},
{"kind": "indexes", "items": []},
{"kind": "sequences", "items": []},
{"kind": "views", "items": []},
{"kind": "functions", "items": []},
{"kind": "triggers", "items": []},
{"kind": "types", "items": []},
{"kind": "policies", "items": []},
{
"kind": "table",
"schema": "public",
"table": "claims",
"row_count": claims,
"rowset_md5": hashlib.md5(str(claims).encode(), usedforsecurity=False).hexdigest(),
},
{"kind": "performance", "query": "count_claims", "elapsed_ms": 1.0, "observed_rows": claims},
]
return "".join(json.dumps(row, sort_keys=True) + "\n" for row in rows)
def capture_fixture(
root: Path,
*,
name: str,
captured_at: str,
authorization_ref: str,
claims: int = 1,
constraint_name: str = "claims_pkey",
) -> tuple[Path, Path]:
manifest = root / f"{name}-manifest.jsonl"
manifest.write_text(manifest_text(claims=claims, constraint_name=constraint_name), encoding="utf-8")
dump_sha256 = hashlib.sha256(f"{name}-dump".encode()).hexdigest()
manifest_sha256 = hashlib.sha256(manifest.read_bytes()).hexdigest()
manifest_sql_sha256 = "2" * 64
source_context_sha256 = "3" * 64
source = {
"ssh_target": "root@77.42.65.182",
"container": "teleo-pg",
"database": "teleo",
"service": "leoclean-gateway.service",
}
healthy_service = {
"ActiveState": "active",
"SubState": "running",
"MainPID": "148735",
"NRestarts": "0",
}
snapshot = {
"exported_snapshot_id": "00000003-000001A2-1",
"txid_snapshot": "100:120:105,111",
"wal_lsn": "0/16B6C50",
"system_identifier": "7649789040005668902",
"captured_at_utc": captured_at,
}
receipt = {
"artifact": "vps_canonical_postgres_exported_snapshot",
"schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"receipt_version": 2,
"status": "pass",
"run_id": name,
"capture_authorization_ref": authorization_ref,
"source": source,
"snapshot_exported": True,
"snapshot": snapshot,
"dump": {"sha256": dump_sha256},
"manifest": {
"sha256": manifest_sha256,
"manifest_sql_sha256": manifest_sql_sha256,
"table_count": 1,
"total_rows": claims,
},
"source_context": {"sha256": source_context_sha256},
"provenance_binding": build_provenance_binding(
run_id=name,
authorization_ref=authorization_ref,
source=source,
snapshot=snapshot,
dump_sha256=dump_sha256,
manifest_sql_sha256=manifest_sql_sha256,
source_manifest_sha256=manifest_sha256,
source_context_sha256=source_context_sha256,
),
"service_unchanged": True,
"source_service": {"before": healthy_service, "after": healthy_service, "unchanged": True},
"production_db_mutated": False,
"telegram_send_attempted": False,
}
receipt_path = root / f"{name}-receipt.json"
receipt_path.write_text(json.dumps(receipt, sort_keys=True) + "\n", encoding="utf-8")
return receipt_path, manifest
def test_postflight_no_delta_is_bound_and_explicit(tmp_path: Path) -> None:
baseline_receipt, baseline_manifest = capture_fixture(
tmp_path,
name="canonical-baseline",
captured_at="2026-07-15T01:00:00+00:00",
authorization_ref="codex-delegation:baseline123",
)
current_receipt, current_manifest = capture_fixture(
tmp_path,
name="canonical-postflight",
captured_at="2026-07-15T02:00:00+00:00",
authorization_ref="codex-delegation:postflight123",
)
result = verify_delta(
baseline_receipt_path=baseline_receipt,
baseline_manifest_path=baseline_manifest,
baseline_authorization_ref="codex-delegation:baseline123",
current_receipt_path=current_receipt,
current_manifest_path=current_manifest,
current_authorization_ref="codex-delegation:postflight123",
)
assert result["status"] == "pass"
assert result["delta"]["delta_detected"] is False
assert result["delta"]["total_row_delta"] == 0
def test_postflight_data_and_structure_delta_is_reported_not_hidden(tmp_path: Path) -> None:
baseline_receipt, baseline_manifest = capture_fixture(
tmp_path,
name="canonical-baseline",
captured_at="2026-07-15T01:00:00+00:00",
authorization_ref="codex-delegation:baseline123",
)
current_receipt, current_manifest = capture_fixture(
tmp_path,
name="canonical-postflight",
captured_at="2026-07-15T02:00:00+00:00",
authorization_ref="codex-delegation:postflight123",
claims=2,
constraint_name="claims_pkey_v2",
)
result = verify_delta(
baseline_receipt_path=baseline_receipt,
baseline_manifest_path=baseline_manifest,
baseline_authorization_ref="codex-delegation:baseline123",
current_receipt_path=current_receipt,
current_manifest_path=current_manifest,
current_authorization_ref="codex-delegation:postflight123",
)
assert result["status"] == "pass"
assert result["delta"]["delta_detected"] is True
assert result["delta"]["total_row_delta"] == 1
assert result["delta"]["changed_tables"][0]["table"] == "public.claims"
assert result["delta"]["changed_structures"][0]["kind"] == "constraints"
def test_postflight_rejects_wrong_authorization_or_stale_capture(tmp_path: Path) -> None:
baseline_receipt, baseline_manifest = capture_fixture(
tmp_path,
name="canonical-baseline",
captured_at="2026-07-15T02:00:00+00:00",
authorization_ref="codex-delegation:baseline123",
)
current_receipt, current_manifest = capture_fixture(
tmp_path,
name="canonical-postflight",
captured_at="2026-07-15T01:00:00+00:00",
authorization_ref="codex-delegation:postflight123",
)
with pytest.raises(ValueError, match="newer than"):
verify_delta(
baseline_receipt_path=baseline_receipt,
baseline_manifest_path=baseline_manifest,
baseline_authorization_ref="codex-delegation:baseline123",
current_receipt_path=current_receipt,
current_manifest_path=current_manifest,
current_authorization_ref="codex-delegation:postflight123",
)
with pytest.raises(ValueError, match="authorization"):
verify_delta(
baseline_receipt_path=baseline_receipt,
baseline_manifest_path=baseline_manifest,
baseline_authorization_ref="codex-delegation:wrong123",
current_receipt_path=current_receipt,
current_manifest_path=current_manifest,
current_authorization_ref="codex-delegation:postflight123",
)
def test_postflight_rejects_unchanged_but_unhealthy_service(tmp_path: Path) -> None:
baseline_receipt, baseline_manifest = capture_fixture(
tmp_path,
name="canonical-baseline",
captured_at="2026-07-15T01:00:00+00:00",
authorization_ref="codex-delegation:baseline123",
)
current_receipt, current_manifest = capture_fixture(
tmp_path,
name="canonical-postflight",
captured_at="2026-07-15T02:00:00+00:00",
authorization_ref="codex-delegation:postflight123",
)
current = json.loads(current_receipt.read_text())
current["source_service"]["before"]["ActiveState"] = "failed"
current["source_service"]["after"]["ActiveState"] = "failed"
current_receipt.write_text(json.dumps(current, sort_keys=True) + "\n", encoding="utf-8")
with pytest.raises(ValueError, match="source_service_healthy"):
verify_delta(
baseline_receipt_path=baseline_receipt,
baseline_manifest_path=baseline_manifest,
baseline_authorization_ref="codex-delegation:baseline123",
current_receipt_path=current_receipt,
current_manifest_path=current_manifest,
current_authorization_ref="codex-delegation:postflight123",
)
def test_delta_receipt_writer_keeps_private_mode(tmp_path: Path) -> None:
output = tmp_path / "private" / "delta.json"
write_private_json(output, {"status": "pass"})
assert output.stat().st_mode & 0o777 == 0o600
assert output.parent.stat().st_mode & 0o777 == 0o700
def test_delta_receipt_writer_does_not_chmod_an_existing_parent(tmp_path: Path) -> None:
parent = tmp_path / "existing"
parent.mkdir(mode=0o755)
output = parent / "delta.json"
write_private_json(output, {"status": "pass"})
assert parent.stat().st_mode & 0o777 == 0o755
assert output.stat().st_mode & 0o777 == 0o600