feat(ops): bind snapshots to private GCP parity lifecycle
This commit is contained in:
parent
a47415b60e
commit
37c0ed091d
18 changed files with 4123 additions and 111 deletions
|
|
@ -11,27 +11,35 @@ Restore a copy of the VPS canonical Leo database to GCP, prove exact schema,
|
||||||
row, role, extension, performance, and private-connectivity parity, then run the
|
row, role, extension, performance, and private-connectivity parity, then run the
|
||||||
real GCP Leo read/reasoning path without Telegram sends or DB mutation.
|
real GCP Leo read/reasoning path without Telegram sends or DB mutation.
|
||||||
|
|
||||||
|
The required tier is T3: a live private GCP staging restore, readback,
|
||||||
|
reasoning, compute attestation, and cleanup receipt. Local restore proof or an
|
||||||
|
access-gate report does not satisfy this tier.
|
||||||
|
|
||||||
## Operator Paths
|
## Operator Paths
|
||||||
|
|
||||||
The direct alias is passwordless and was live-verified on 2026-07-14:
|
The direct alias was passwordless and live-verified on 2026-07-14:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh teleo-gcp-staging
|
ssh teleo-gcp-staging
|
||||||
```
|
```
|
||||||
|
|
||||||
It uses `/Users/user/.ssh/google_compute_engine` for the configured operator,
|
It uses `~/.ssh/google_compute_engine` for the configured operator,
|
||||||
disables password and keyboard-interactive authentication, and does not store a
|
disables password and keyboard-interactive authentication, and does not store a
|
||||||
Google password. `sudo -n` also works. The route remains
|
Google password. That is historical proof, not current reachability. The route
|
||||||
firewall-source-dependent, so verify `ssh -o BatchMode=yes teleo-gcp-staging true` before a
|
remains firewall-source-dependent. On
|
||||||
long run instead of assuming retained access is current.
|
2026-07-15, both direct SSH and the private TCP route timed out; verify
|
||||||
|
`ssh -o BatchMode=yes teleo-gcp-staging true` before a long run and fail closed
|
||||||
|
if it does not return successfully.
|
||||||
|
|
||||||
The intended secondary path is `.github/workflows/gcp-iap-operator.yml`, using
|
The intended secondary path is `.github/workflows/gcp-iap-operator.yml`, using
|
||||||
short-lived GitHub OIDC, IAP, OS Login, and fixed reviewed operations. It is
|
short-lived GitHub OIDC, IAP, OS Login, and fixed reviewed operations. It is
|
||||||
merged but not bootstrapped: workflow run `29208215340` failed at auth with
|
merged but not bootstrapped: the current 2026-07-15 check still lacks the
|
||||||
`invalid_target` because provider `teleo-iap-operator` is absent, disabled, or
|
reviewed operator WIF/IAP authority. Provider `teleo-iap-operator` is absent,
|
||||||
deleted. Do not call this path working until a live `status` run passes.
|
disabled, or deleted (`invalid_target`), and the available artifact identity is intentionally not
|
||||||
|
a Cloud SQL/Compute operator. Do not call this path working until a live `status` run passes
|
||||||
|
with the intended operator identity.
|
||||||
|
|
||||||
Verified target:
|
Historically verified and intended target; reverify before any new lifecycle:
|
||||||
|
|
||||||
- project `teleo-501523`;
|
- project `teleo-501523`;
|
||||||
- VM `teleo-prod-1` in `europe-west6-a`;
|
- VM `teleo-prod-1` in `europe-west6-a`;
|
||||||
|
|
@ -45,8 +53,9 @@ variable, then unset it.
|
||||||
|
|
||||||
## Two Different Databases
|
## Two Different Databases
|
||||||
|
|
||||||
- Canonical collective knowledge is Cloud SQL database `teleo_canonical`, with
|
- Canonical collective knowledge remains VPS PostgreSQL database `teleo`, with
|
||||||
`public.*` and `kb_stage.*` tables.
|
`public.*` and `kb_stage.*` tables. Persistent Cloud SQL database
|
||||||
|
`teleo_canonical` is a staging copy and has not been promoted.
|
||||||
- Hermes conversation continuity is `state.db` plus session JSONL files. The
|
- Hermes conversation continuity is `state.db` plus session JSONL files. The
|
||||||
`leoclean-cloudsql-memory-sync.service` snapshot path copies this runtime
|
`leoclean-cloudsql-memory-sync.service` snapshot path copies this runtime
|
||||||
memory; it does not populate or prove canonical claims, sources, evidence,
|
memory; it does not populate or prove canonical claims, sources, evidence,
|
||||||
|
|
@ -54,7 +63,7 @@ variable, then unset it.
|
||||||
|
|
||||||
Do not describe a passing Hermes memory sync as canonical KB parity.
|
Do not describe a passing Hermes memory sync as canonical KB parity.
|
||||||
|
|
||||||
## Current Verified State - 2026-07-14
|
## Retained Historical Proof - 2026-07-14
|
||||||
|
|
||||||
- The newest captured VPS database has `39` tables and `52,167` rows, including
|
- The newest captured VPS database has `39` tables and `52,167` rows, including
|
||||||
claims `1837`, sources `4145`, claim evidence `4670`, claim edges `4916`, and
|
claims `1837`, sources `4145`, claim evidence `4670`, claim edges `4916`, and
|
||||||
|
|
@ -110,17 +119,54 @@ Track these independently:
|
||||||
Use:
|
Use:
|
||||||
|
|
||||||
- `ops/capture_vps_canonical_postgres_snapshot.py` for a single source dump and
|
- `ops/capture_vps_canonical_postgres_snapshot.py` for a single source dump and
|
||||||
manifest;
|
manifest, with a required retained `--authorization-ref` and v2 provenance
|
||||||
|
receipt binding the exported snapshot identity plus dump/manifest hashes; it
|
||||||
|
requires identical healthy structured source-service states before/after
|
||||||
|
(`active/running`, positive PID, nonnegative restart count);
|
||||||
- `ops/restore_gcp_generated_postgres_snapshot.py restore --execute` for a
|
- `ops/restore_gcp_generated_postgres_snapshot.py restore --execute` for a
|
||||||
receipt-bound private-TLS restore into a bounded `teleo_clone_*` database;
|
receipt-bound private-TLS restore into a bounded `teleo_clone_*` database;
|
||||||
|
pass the exact capture `--source-receipt`, and require its live Cloud SQL
|
||||||
|
control-plane check to prove public IP disabled before clone creation; it
|
||||||
|
writes `target-manifest.jsonl`, `restore-receipt.json`, and the generated
|
||||||
|
`gcp-private-connectivity.json` under
|
||||||
|
`/var/lib/teleo-gcp-restore-runs/<restore-run-id>/`;
|
||||||
- `ops/restore_gcp_generated_postgres_snapshot.py cleanup --execute` for exact
|
- `ops/restore_gcp_generated_postgres_snapshot.py cleanup --execute` for exact
|
||||||
clone cleanup with live-service and rollback readback;
|
clone cleanup with live-service and rollback readback;
|
||||||
- `ops/postgres_parity_manifest.sql` for row/catalog/role/performance readback;
|
- `ops/postgres_parity_manifest.sql` for row/catalog/role/performance readback;
|
||||||
- `ops/verify_postgres_parity_manifest.py --scope gcp_staging` for exact parity;
|
- `ops/verify_postgres_parity_manifest.py --scope gcp_staging` for exact parity,
|
||||||
|
always retaining the canonical output name `gcp-parity.json`;
|
||||||
- `scripts/run_gcp_generated_db_direct_claim_suite.py` for the adapter-free,
|
- `scripts/run_gcp_generated_db_direct_claim_suite.py` for the adapter-free,
|
||||||
read-only, no-send six-response replay against a generated `teleo_clone_*`.
|
read-only, no-send six-response replay against a generated `teleo_clone_*`.
|
||||||
- `scripts/run_gcp_generated_db_blind_claim_canary.py` for the bounded ID-free
|
- `scripts/run_gcp_generated_db_blind_claim_canary.py` for the bounded ID-free
|
||||||
claim challenge, source receipt, reasoning, no-write, and cleanup proof.
|
claim challenge, source receipt, reasoning, no-write, and cleanup proof.
|
||||||
|
- `ops/attest_gcp_reasoning_compute.py` on the replay VM after blind reasoning
|
||||||
|
and before cleanup, binding the reasoning receipt SHA-256 to live GCE metadata
|
||||||
|
in `reasoning-compute-attestation.json`;
|
||||||
|
- a second authorized source capture after cleanup, followed by
|
||||||
|
`ops/verify_vps_canonical_snapshot_delta.py`, to retain the explicit
|
||||||
|
post-snapshot VPS delta in `source-delta-receipt.json`;
|
||||||
|
- `ops/verify_gcp_canonical_lifecycle.py` after cleanup to bind the exact source,
|
||||||
|
current source, source delta, restore, `gcp_staging` parity, ID-free reasoning,
|
||||||
|
reasoning-compute attestation, and cleanup receipts into one fail-closed T3
|
||||||
|
lifecycle verdict. Pass `--max-postflight-age-seconds 900` and
|
||||||
|
`--max-lifecycle-age-seconds 3600`; the latter bounds both cleanup-to-verdict
|
||||||
|
age and restore-to-postflight span.
|
||||||
|
|
||||||
|
With `TARGET_DB`, `RESTORE_RUN_ID`, and `PRIVATE_RUN_DIR` set as shown in the
|
||||||
|
runbook, run the replay-host attestation after the blind receipt passes and
|
||||||
|
before the receipt-bound cleanup:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo python3 ops/attest_gcp_reasoning_compute.py \
|
||||||
|
--reasoning-receipt "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \
|
||||||
|
--restore-run-id "$RESTORE_RUN_ID" \
|
||||||
|
--target-db "$TARGET_DB" \
|
||||||
|
--project teleo-501523 \
|
||||||
|
--expected-compute-instance teleo-prod-1 \
|
||||||
|
--expected-compute-zone europe-west6-a \
|
||||||
|
--expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \
|
||||||
|
--output "$PRIVATE_RUN_DIR/reasoning-compute-attestation.json"
|
||||||
|
```
|
||||||
|
|
||||||
The replay must use the Hermes virtualenv, not system Python:
|
The replay must use the Hermes virtualenv, not system Python:
|
||||||
|
|
||||||
|
|
@ -129,8 +175,16 @@ The replay must use the Hermes virtualenv, not system Python:
|
||||||
scripts/run_gcp_generated_db_direct_claim_suite.py ...
|
scripts/run_gcp_generated_db_direct_claim_suite.py ...
|
||||||
```
|
```
|
||||||
|
|
||||||
The replay is not complete until the generated clone, temporary profile,
|
The lifecycle is not complete until the generated clone, temporary profile,
|
||||||
children, upload/run directories, and any temporary client are absent.
|
children, transient upload/bundle directories, and any temporary client are
|
||||||
|
absent. Preserve the private receipt run directory and every useful lifecycle
|
||||||
|
input, supporting receipt, and verifier output; do not treat that evidence as
|
||||||
|
transient cleanup. The canonical restore streams through `pg_restore` and
|
||||||
|
creates no GCS import object.
|
||||||
|
|
||||||
|
Static inspection or a dry run is insufficient for these helpers: verify a new
|
||||||
|
capture against the live VPS, run the focused tests, and require a passing live
|
||||||
|
lifecycle receipt before describing the workflow as current parity.
|
||||||
|
|
||||||
## Safety And Claim Ceiling
|
## Safety And Claim Ceiling
|
||||||
|
|
||||||
|
|
@ -147,12 +201,17 @@ delivery, GCP canonical mutation, ongoing replication, or production cutover.
|
||||||
|
|
||||||
## Current Access And Next Action
|
## Current Access And Next Action
|
||||||
|
|
||||||
Direct SSH and passwordless sudo are currently working. Local `gcloud` still
|
As of 2026-07-15, direct SSH and private TCP timed out, local `gcloud` for
|
||||||
requires account reauthentication for control-plane metadata, so the current
|
`billy@livingip.xyz` requires reauthentication, and the intended operator
|
||||||
private-IP/TLS proof comes from a live database connection while the public-IP-
|
WIF/IAP authority is unavailable. The artifact-builder WIF identity must not be
|
||||||
disabled control-plane receipt remains dated 2026-07-12.
|
expanded or mistaken for database-operator access. Current retained T2/source
|
||||||
|
and local restore proof does not satisfy the required T3 lifecycle.
|
||||||
|
|
||||||
The next production decision is not another restore drill. It is whether to
|
The next action after an authorized operator route is restored is: newest
|
||||||
promote a newly verified snapshot into persistent GCP `teleo_canonical` and
|
authorized VPS capture -> disposable private `teleo_clone_*` restore ->
|
||||||
repoint the production read adapter. Until that decision is explicit, keep GCP
|
generated private-connectivity receipt -> `gcp-parity.json` -> no-send
|
||||||
classified as staging and do not expose Cloud SQL publicly.
|
m3taversal blind reasoning -> live GCE reasoning attestation -> receipt-bound
|
||||||
|
clone cleanup -> postflight VPS capture/delta -> eight-receipt lifecycle verdict
|
||||||
|
within the 900/3600-second freshness bounds. Do not promote
|
||||||
|
`teleo_canonical`, expose Cloud SQL publicly, send Telegram, or mutate canonical
|
||||||
|
knowledge.
|
||||||
|
|
|
||||||
7
.github/workflows/ci.yml
vendored
7
.github/workflows/ci.yml
vendored
|
|
@ -42,17 +42,21 @@ jobs:
|
||||||
lib/llm.py \
|
lib/llm.py \
|
||||||
lib/post_extract.py \
|
lib/post_extract.py \
|
||||||
ops/apply_gcp_iam_split.py \
|
ops/apply_gcp_iam_split.py \
|
||||||
|
ops/attest_gcp_reasoning_compute.py \
|
||||||
ops/capture_vps_canonical_postgres_snapshot.py \
|
ops/capture_vps_canonical_postgres_snapshot.py \
|
||||||
ops/check_gcp_infra_readiness.py \
|
ops/check_gcp_infra_readiness.py \
|
||||||
ops/run_gcp_infra_execute_canary.py \
|
ops/run_gcp_infra_execute_canary.py \
|
||||||
ops/apply_gcp_runtime_baseline.py \
|
ops/apply_gcp_runtime_baseline.py \
|
||||||
ops/check_gcp_service_communications.py \
|
ops/check_gcp_service_communications.py \
|
||||||
ops/plan_gcp_iam_split.py \
|
ops/plan_gcp_iam_split.py \
|
||||||
|
ops/private_receipt_io.py \
|
||||||
ops/redact_sqlite_postgres_restore_canary.py \
|
ops/redact_sqlite_postgres_restore_canary.py \
|
||||||
ops/restore_gcp_generated_postgres_snapshot.py \
|
ops/restore_gcp_generated_postgres_snapshot.py \
|
||||||
ops/sqlite_to_postgres_dump.py \
|
ops/sqlite_to_postgres_dump.py \
|
||||||
ops/verify_gcp_cloudsql_restore_readback.py \
|
ops/verify_gcp_cloudsql_restore_readback.py \
|
||||||
|
ops/verify_gcp_canonical_lifecycle.py \
|
||||||
ops/verify_postgres_parity_manifest.py \
|
ops/verify_postgres_parity_manifest.py \
|
||||||
|
ops/verify_vps_canonical_snapshot_delta.py \
|
||||||
telegram/approvals.py \
|
telegram/approvals.py \
|
||||||
hermes-agent/leoclean-bin/kb_tool.py \
|
hermes-agent/leoclean-bin/kb_tool.py \
|
||||||
hermes-agent/leoclean-bin/cloudsql_memory_tool.py \
|
hermes-agent/leoclean-bin/cloudsql_memory_tool.py \
|
||||||
|
|
@ -72,6 +76,7 @@ jobs:
|
||||||
scripts/working_leo_m3taversal_oos_benchmark.py \
|
scripts/working_leo_m3taversal_oos_benchmark.py \
|
||||||
scripts/working_leo_open_ended_benchmark.py \
|
scripts/working_leo_open_ended_benchmark.py \
|
||||||
tests/test_agent_routing.py \
|
tests/test_agent_routing.py \
|
||||||
|
tests/test_attest_gcp_reasoning_compute.py \
|
||||||
tests/test_assemble_telegram_visible_direct_claim_capture_receipt.py \
|
tests/test_assemble_telegram_visible_direct_claim_capture_receipt.py \
|
||||||
tests/test_build_working_leo_m3taversal_outcome_sandbox.py \
|
tests/test_build_working_leo_m3taversal_outcome_sandbox.py \
|
||||||
tests/test_decision_engine_replay.py \
|
tests/test_decision_engine_replay.py \
|
||||||
|
|
@ -96,6 +101,8 @@ jobs:
|
||||||
tests/test_verify_leo_db_first_oos_canary.py \
|
tests/test_verify_leo_db_first_oos_canary.py \
|
||||||
tests/test_compile_kb_source_packet.py \
|
tests/test_compile_kb_source_packet.py \
|
||||||
tests/test_verify_postgres_parity_manifest.py \
|
tests/test_verify_postgres_parity_manifest.py \
|
||||||
|
tests/test_verify_gcp_canonical_lifecycle.py \
|
||||||
|
tests/test_verify_vps_canonical_snapshot_delta.py \
|
||||||
tests/test_working_leo_m3taversal_oos_benchmark.py \
|
tests/test_working_leo_m3taversal_oos_benchmark.py \
|
||||||
tests/test_working_leo_open_ended_benchmark.py \
|
tests/test_working_leo_open_ended_benchmark.py \
|
||||||
tests/test_phase1b_end_to_end.py \
|
tests/test_phase1b_end_to_end.py \
|
||||||
|
|
|
||||||
|
|
@ -53,12 +53,21 @@ python3 ops/capture_vps_canonical_postgres_snapshot.py \
|
||||||
--ssh-target root@77.42.65.182 \
|
--ssh-target root@77.42.65.182 \
|
||||||
--ssh-key ~/.ssh/livingip_hetzner_20260604_ed25519 \
|
--ssh-key ~/.ssh/livingip_hetzner_20260604_ed25519 \
|
||||||
--run-id canonical-<timestamp> \
|
--run-id canonical-<timestamp> \
|
||||||
|
--authorization-ref codex-delegation:THREAD_ID \
|
||||||
--output-dir <private-output-dir>
|
--output-dir <private-output-dir>
|
||||||
```
|
```
|
||||||
|
|
||||||
The capture fails closed if the source service changes while it runs. It
|
Replace `THREAD_ID` with the exact retained authorization reference for the
|
||||||
|
capture. The v2 receipt binds that reference, exported snapshot ID, TXID
|
||||||
|
snapshot, WAL LSN, source system identifier, dump hash, manifest hash, reviewed
|
||||||
|
manifest-SQL hash, and source-context hash. A dump and manifest without this
|
||||||
|
passing receipt are not eligible for GCP restore.
|
||||||
|
|
||||||
|
The capture fails closed unless the structured source-service states before
|
||||||
|
and after the snapshot are identical and healthy: `ActiveState=active`,
|
||||||
|
`SubState=running`, a positive `MainPID`, and a nonnegative `NRestarts`. It
|
||||||
retains a private custom dump, dump SHA-256, object TOC, catalog/data manifest,
|
retains a private custom dump, dump SHA-256, object TOC, catalog/data manifest,
|
||||||
and before/after service state. It never restarts Leo or writes to the source
|
and both service-state objects. It never restarts Leo or writes to the source
|
||||||
database.
|
database.
|
||||||
|
|
||||||
Prove that this exact snapshot can rebuild a blank Postgres target before using
|
Prove that this exact snapshot can rebuild a blank Postgres target before using
|
||||||
|
|
@ -79,16 +88,70 @@ then removes the container and proves it is absent. A passing local receipt is
|
||||||
the exact-recovery preflight; it is not semantic recompilation from raw source
|
the exact-recovery preflight; it is not semantic recompilation from raw source
|
||||||
documents.
|
documents.
|
||||||
|
|
||||||
Run `ops/postgres_parity_manifest.sql` against the isolated restored target,
|
Choose one bounded suffix and use it consistently on the GCP replay VM. The
|
||||||
then compare source and target:
|
restore helper retains its mode-0700 evidence directory at the fixed private
|
||||||
|
root:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python3 ops/verify_postgres_parity_manifest.py \
|
RUN_SUFFIX=20260715t010000z
|
||||||
|
TARGET_DB="teleo_clone_${RUN_SUFFIX}"
|
||||||
|
RESTORE_RUN_ID="gcp-restore-${RUN_SUFFIX}"
|
||||||
|
PRIVATE_RUN_DIR="/var/lib/teleo-gcp-restore-runs/${RESTORE_RUN_ID}"
|
||||||
|
```
|
||||||
|
|
||||||
|
Use a generated target database such as `teleo_clone_<run_id>`. Never import a
|
||||||
|
drill into `teleo`, `teleo_kb`, or `teleo_canonical`. Database isolation does
|
||||||
|
not isolate cluster-global roles or extensions, so verify those separately and
|
||||||
|
do not run the Docker-only gate bootstrap against the shared Cloud SQL instance.
|
||||||
|
|
||||||
|
The canonical restore command must include the v2 capture receipt. It also runs
|
||||||
|
a live Cloud SQL control-plane preflight before resolving the database secret or
|
||||||
|
creating the clone, and fails closed unless the exact instance is runnable,
|
||||||
|
PostgreSQL 16, private-networked, bound to the expected RFC1918 host, and has
|
||||||
|
public IP disabled. A GCE metadata preflight independently binds the operator
|
||||||
|
runtime to `teleo-prod-1` in `europe-west6-a` on `teleo-staging-net`; a
|
||||||
|
hard-coded compute label is not accepted:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo python3 ops/restore_gcp_generated_postgres_snapshot.py restore \
|
||||||
|
--execute \
|
||||||
|
--target-db "$TARGET_DB" \
|
||||||
|
--run-id "$RESTORE_RUN_ID" \
|
||||||
|
--dump <private-output-dir>/teleo-canonical.dump \
|
||||||
|
--expected-dump-sha256 <dump-sha256> \
|
||||||
|
--source-manifest <private-output-dir>/source-manifest.jsonl \
|
||||||
|
--source-receipt <private-output-dir>/receipt.json \
|
||||||
|
--expected-source-receipt-sha256 <source-receipt-sha256> \
|
||||||
|
--expected-capture-authorization-ref codex-delegation:THREAD_ID \
|
||||||
|
--project teleo-501523 \
|
||||||
|
--cloudsql-instance teleo-pgvector-standby \
|
||||||
|
--expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \
|
||||||
|
--expected-compute-instance teleo-prod-1 \
|
||||||
|
--expected-compute-zone europe-west6-a \
|
||||||
|
--expected-source-ssh-target root@77.42.65.182 \
|
||||||
|
--expected-source-container teleo-pg \
|
||||||
|
--expected-source-database teleo \
|
||||||
|
--expected-source-service leoclean-gateway.service
|
||||||
|
```
|
||||||
|
|
||||||
|
The restore writes both `target-manifest.jsonl` and
|
||||||
|
`gcp-private-connectivity.json` as mode-0600 files in its private run
|
||||||
|
directory. The latter is composed from live GCE metadata, Cloud SQL
|
||||||
|
control-plane checks, and the private TLS database identity; do not hand-compose
|
||||||
|
the connectivity receipt. The restore also records identical healthy
|
||||||
|
before/after state for `leoclean-gcp-prod-parallel.service`, including its PID
|
||||||
|
and restart count.
|
||||||
|
|
||||||
|
Compare the captured target manifest and generated connectivity receipt to the
|
||||||
|
authorized source:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo python3 ops/verify_postgres_parity_manifest.py \
|
||||||
--source <private-output-dir>/source-manifest.jsonl \
|
--source <private-output-dir>/source-manifest.jsonl \
|
||||||
--target <private-output-dir>/target-manifest.jsonl \
|
--target "$PRIVATE_RUN_DIR/target-manifest.jsonl" \
|
||||||
--scope gcp_staging \
|
--scope gcp_staging \
|
||||||
--connectivity-proof <private-output-dir>/gcp-private-connectivity.json \
|
--connectivity-proof "$PRIVATE_RUN_DIR/gcp-private-connectivity.json" \
|
||||||
--output <private-output-dir>/gcp-parity.json
|
--output "$PRIVATE_RUN_DIR/gcp-parity.json"
|
||||||
```
|
```
|
||||||
|
|
||||||
The verifier checks all table row counts and collation-independent row hashes,
|
The verifier checks all table row counts and collation-independent row hashes,
|
||||||
|
|
@ -98,14 +161,100 @@ password-free application-role attributes, and bounded query timings. In GCP
|
||||||
scope it also requires a receipt proving a staging compute source, a private
|
scope it also requires a receipt proving a staging compute source, a private
|
||||||
server address, TLS, and public-IP-disabled instance metadata.
|
server address, TLS, and public-IP-disabled instance metadata.
|
||||||
|
|
||||||
Use a generated target database such as `teleo_clone_<run_id>`. Never import a
|
After the parity verifier passes, run the no-send m3taversal blind reasoning
|
||||||
drill into `teleo`, `teleo_kb`, or `teleo_canonical`. Database isolation does
|
replay from staging compute against that generated database, attest the replay
|
||||||
not isolate cluster-global roles or extensions, so verify those separately and
|
host from live GCE metadata, and only then delete the generated database. This
|
||||||
do not run the Docker-only gate bootstrap against the shared Cloud SQL instance.
|
canonical path streams the custom dump through `pg_restore`; it does not create
|
||||||
|
a GCS import object.
|
||||||
|
|
||||||
After the parity verifier passes, run the no-send operator composition replay from
|
Run the bounded ID-free reasoning canary with the generated parity receipt. The
|
||||||
staging compute against that generated database. Only then delete the generated
|
script re-executes itself in the Hermes virtualenv when that runtime is present:
|
||||||
database and uploaded import object and retain cleanup readback.
|
|
||||||
|
```bash
|
||||||
|
sudo python3 scripts/run_gcp_generated_db_blind_claim_canary.py \
|
||||||
|
--execute \
|
||||||
|
--target-db "$TARGET_DB" \
|
||||||
|
--cloudsql-tool hermes-agent/leoclean-bin/cloudsql_memory_tool.py \
|
||||||
|
--manifest-sql ops/postgres_parity_manifest.sql \
|
||||||
|
--parity-receipt "$PRIVATE_RUN_DIR/gcp-parity.json" \
|
||||||
|
--output "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \
|
||||||
|
--run-id "gcp-blind-claim-${RUN_SUFFIX}"
|
||||||
|
```
|
||||||
|
|
||||||
|
Before cleanup, bind that reasoning receipt to live metadata from the same GCE
|
||||||
|
replay host:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo python3 ops/attest_gcp_reasoning_compute.py \
|
||||||
|
--reasoning-receipt "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \
|
||||||
|
--restore-run-id "$RESTORE_RUN_ID" \
|
||||||
|
--target-db "$TARGET_DB" \
|
||||||
|
--project teleo-501523 \
|
||||||
|
--expected-compute-instance teleo-prod-1 \
|
||||||
|
--expected-compute-zone europe-west6-a \
|
||||||
|
--expected-private-network projects/teleo-501523/global/networks/teleo-staging-net \
|
||||||
|
--output "$PRIVATE_RUN_DIR/reasoning-compute-attestation.json"
|
||||||
|
```
|
||||||
|
|
||||||
|
After the compute attestation passes, run the receipt-bound cleanup command
|
||||||
|
retained by the restore. Then capture the VPS source again under a distinct
|
||||||
|
postflight authorization reference. Run this capture from the same trusted
|
||||||
|
source-access lane used for the baseline; it need not run on the GCP replay VM.
|
||||||
|
The postflight capture must complete after cleanup, so the final verdict can
|
||||||
|
state whether the VPS changed after the restored snapshot boundary. Compare the
|
||||||
|
two source captures:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 ops/capture_vps_canonical_postgres_snapshot.py \
|
||||||
|
--execute \
|
||||||
|
--ssh-target root@77.42.65.182 \
|
||||||
|
--ssh-key ~/.ssh/livingip_hetzner_20260604_ed25519 \
|
||||||
|
--run-id canonical-${RUN_SUFFIX}-postflight \
|
||||||
|
--authorization-ref codex-delegation:THREAD_ID:postflight \
|
||||||
|
--output-dir <private-postflight-dir>
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo python3 ops/verify_vps_canonical_snapshot_delta.py \
|
||||||
|
--baseline-receipt <private-output-dir>/receipt.json \
|
||||||
|
--baseline-manifest <private-output-dir>/source-manifest.jsonl \
|
||||||
|
--baseline-authorization-ref codex-delegation:THREAD_ID \
|
||||||
|
--current-receipt <private-postflight-dir>/receipt.json \
|
||||||
|
--current-manifest <private-postflight-dir>/source-manifest.jsonl \
|
||||||
|
--current-authorization-ref codex-delegation:THREAD_ID:postflight \
|
||||||
|
--output "$PRIVATE_RUN_DIR/source-delta-receipt.json"
|
||||||
|
```
|
||||||
|
|
||||||
|
Finally, verify that all eight receipts belong to one bound lifecycle. This
|
||||||
|
verifier rejects local-scope parity, stale or mismatched source hashes, a
|
||||||
|
postflight source capture that predates reasoning or cleanup, hand-authored
|
||||||
|
reasoning booleans without real retrieval/tool receipts, an unattested replay
|
||||||
|
host, compute or Cloud SQL topology drift, and any cleanup receipt that leaves
|
||||||
|
the clone present. The postflight capture may be at most 900 seconds old at
|
||||||
|
verification; the GCP lifecycle age and restore-to-postflight span must each be
|
||||||
|
at most 3600 seconds:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo python3 ops/verify_gcp_canonical_lifecycle.py \
|
||||||
|
--source-receipt <private-output-dir>/receipt.json \
|
||||||
|
--current-source-receipt <private-postflight-dir>/receipt.json \
|
||||||
|
--source-delta-receipt "$PRIVATE_RUN_DIR/source-delta-receipt.json" \
|
||||||
|
--restore-receipt "$PRIVATE_RUN_DIR/restore-receipt.json" \
|
||||||
|
--parity-receipt "$PRIVATE_RUN_DIR/gcp-parity.json" \
|
||||||
|
--reasoning-receipt "$PRIVATE_RUN_DIR/blind-reasoning-receipt.json" \
|
||||||
|
--reasoning-compute-receipt "$PRIVATE_RUN_DIR/reasoning-compute-attestation.json" \
|
||||||
|
--cleanup-receipt "$PRIVATE_RUN_DIR/cleanup-receipt.json" \
|
||||||
|
--max-postflight-age-seconds 900 \
|
||||||
|
--max-lifecycle-age-seconds 3600 \
|
||||||
|
--output "$PRIVATE_RUN_DIR/lifecycle-verification.json"
|
||||||
|
```
|
||||||
|
|
||||||
|
Retain the private receipt run directory and every useful lifecycle input,
|
||||||
|
supporting receipt, and verifier output. Remove only transient upload/bundle
|
||||||
|
directories, temporary Hermes profiles, helper clients, and child processes
|
||||||
|
after their absence is verified. There is no canonical-path GCS object to
|
||||||
|
delete. Do not remove the evidence directory until its useful receipts are
|
||||||
|
integrated or explicitly rejected.
|
||||||
|
|
||||||
## Legacy SQLite Source Backup Canary
|
## Legacy SQLite Source Backup Canary
|
||||||
|
|
||||||
|
|
@ -236,11 +385,15 @@ A successful restore or replication canary must retain:
|
||||||
- dump timestamp or replication slot timestamp;
|
- dump timestamp or replication slot timestamp;
|
||||||
- source schema/database name.
|
- source schema/database name.
|
||||||
- transfer proof:
|
- transfer proof:
|
||||||
- dump object path in a versioned bucket, or logical replication subscription details;
|
- for the canonical path, the v2 capture receipt, dump SHA-256, and exact
|
||||||
|
private staging path used by streaming `pg_restore`;
|
||||||
|
- for a legacy import or replication path, a versioned dump-object generation
|
||||||
|
or logical replication subscription details;
|
||||||
- row/table counts before import where available.
|
- row/table counts before import where available.
|
||||||
- target proof:
|
- target proof:
|
||||||
- `teleo-pgvector-standby` readback;
|
- `teleo-pgvector-standby` control-plane readback;
|
||||||
- `teleo_kb` database readback;
|
- generated `teleo_clone_*` identity for canonical parity, or `teleo_kb` only
|
||||||
|
for the explicitly legacy SQLite drill;
|
||||||
- extension readback for `vector` if the restored schema needs pgvector;
|
- extension readback for `vector` if the restored schema needs pgvector;
|
||||||
- representative query readback for core KB tables.
|
- representative query readback for core KB tables.
|
||||||
- failure boundary:
|
- failure boundary:
|
||||||
|
|
@ -295,28 +448,26 @@ Retain only redacted connection metadata. Do not commit or paste credentials.
|
||||||
|
|
||||||
## Current Blocker
|
## Current Blocker
|
||||||
|
|
||||||
As of 2026-07-11, the canonical Postgres exported-snapshot capture and isolated
|
As of 2026-07-15, a newest authorized canonical Postgres capture and isolated
|
||||||
local restore parity pass. Live GCP restore and staging replay do not.
|
local restore can pass, but the required T3 live private GCP lifecycle cannot be
|
||||||
|
started from the current operator route:
|
||||||
|
|
||||||
- GitHub WIF works for `sa-artifact-builder`, but that identity is intentionally
|
- direct VM SSH and the private TCP route timed out;
|
||||||
limited to Artifact Registry and cannot inspect or mutate Cloud SQL/Compute.
|
- local `gcloud` for `billy@livingip.xyz` requires reauthentication; no
|
||||||
- The configured `sa-teleo-readiness` and `sa-teleo-restore-drill` identities
|
password was entered or inspected;
|
||||||
return IAM 404 and do not exist.
|
- the intended operator WIF/IAP authority is unavailable because provider
|
||||||
- The local privileged `billy@livingip.xyz` gcloud session requires password
|
`teleo-iap-operator` is absent, disabled, or deleted;
|
||||||
reauthentication. No password was entered or inspected.
|
- the working artifact-builder WIF identity is intentionally limited and must
|
||||||
- Direct VM SSH is closed to the current egress `/32`; IAP requires the same
|
not be expanded or mistaken for Cloud SQL/Compute operator access.
|
||||||
privileged GCP authentication.
|
|
||||||
|
|
||||||
That is why the readiness checker still reports:
|
The authorized recovery is a local
|
||||||
|
`gcloud auth login billy@livingip.xyz --force` without sharing credentials, or
|
||||||
|
administrator convergence of the reviewed operator WIF/IAP bootstrap. Until a
|
||||||
|
route passes live readback, keep the claim at T2/source-local proof; do not
|
||||||
|
promote staging or expose Cloud SQL publicly.
|
||||||
|
|
||||||
- `kb_source_restore_access = blocked`
|
Once access is restored, run: newest authorized `teleo` snapshot -> disposable
|
||||||
- `kb_restore_or_replication = blocked`
|
private `teleo_clone_*` restore -> generated connectivity proof ->
|
||||||
|
`gcp-parity.json` -> no-send m3taversal blind reasoning -> live GCE reasoning
|
||||||
The immediate operator CTA is to complete
|
attestation -> receipt-bound clone cleanup -> postflight VPS capture and delta
|
||||||
`gcloud auth login billy@livingip.xyz --force` locally without sharing the
|
-> eight-receipt lifecycle verdict within the 900/3600-second freshness bounds.
|
||||||
password, or apply the reviewed IAM split with an authorized GCP administrator.
|
|
||||||
The next non-user action is:
|
|
||||||
|
|
||||||
canonical `teleo` snapshot -> generated Cloud SQL database -> full parity and
|
|
||||||
private-connectivity verifier -> no-send Cory composition replay from staging
|
|
||||||
compute -> delete the generated database/object -> retain cleanup proof.
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,134 @@
|
||||||
|
{
|
||||||
|
"artifact": "gcp_canonical_parity_live_attempt",
|
||||||
|
"schema": "livingip.gcpCanonicalParityLiveAttempt.v1",
|
||||||
|
"generated_at_utc": "2026-07-15T01:50:20Z",
|
||||||
|
"status": "gated_external",
|
||||||
|
"required_tier": "T3_live_private_gcp_staging",
|
||||||
|
"current_tier": "T2_current_source_and_isolated_restore_plus_live_access_gate",
|
||||||
|
"current_canary": "Restore the newest authorized canonical snapshot into one private generated Cloud SQL database, prove exact parity and ID-free Leo reasoning from staging compute, then drop the generated database and verify absence.",
|
||||||
|
"current_source": {
|
||||||
|
"run_id": "canonical-20260715t014930z",
|
||||||
|
"captured_at_utc": "2026-07-15T01:49:39.104116+00:00",
|
||||||
|
"capture_authorization_ref": "codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621",
|
||||||
|
"receipt_sha256": "7d96b052591d36df3b67dc6291b0f74a7bbeb9602c9cc424779e7c21194d59a5",
|
||||||
|
"dump_sha256": "6c56a841c1159a17f9404c2d63e3e93ae8ff55e842bf311d6d1bff21e9b95c9a",
|
||||||
|
"manifest_sha256": "39f06c6b9fc806a89ecd19294be29c58c499c65234611c7a483ddefa3c78d7d3",
|
||||||
|
"provenance_binding_sha256": "fb587093a76b61af670b7d968d32b78d6a87572faa78ca1ad93ef6cba1c9b776",
|
||||||
|
"source_system_identifier": "7649789040005668902",
|
||||||
|
"table_count": 39,
|
||||||
|
"total_rows": 52167,
|
||||||
|
"service": {
|
||||||
|
"active_state": "active",
|
||||||
|
"sub_state": "running",
|
||||||
|
"main_pid": 347406,
|
||||||
|
"restart_count": 0,
|
||||||
|
"unchanged": true
|
||||||
|
},
|
||||||
|
"production_database_mutated": false,
|
||||||
|
"telegram_send_attempted": false
|
||||||
|
},
|
||||||
|
"source_stability": {
|
||||||
|
"baseline_run_id": "canonical-20260715t014814z",
|
||||||
|
"current_run_id": "canonical-20260715t014930z",
|
||||||
|
"receipt_sha256": "d761a2b8b0313ee03c9d01e50fcca709af779578281cf10de5aed4709e914d72",
|
||||||
|
"status": "pass",
|
||||||
|
"delta_detected": false,
|
||||||
|
"table_count": 39,
|
||||||
|
"total_rows": 52167,
|
||||||
|
"total_row_delta": 0,
|
||||||
|
"changed_tables": [],
|
||||||
|
"changed_structures": [],
|
||||||
|
"table_fingerprint_sha256": "f50d86fc3ce699f602bdc0029485bd901fd98bd8114eecd5dcc4346a0f52112a",
|
||||||
|
"structure_fingerprint_sha256": "61bf482330a652ebdb419de3fb78196f2c7d25d1a482b100f385e8102c8e8d52"
|
||||||
|
},
|
||||||
|
"isolated_restore": {
|
||||||
|
"receipt_sha256": "181901afc91bebee7c343500c601ed1e5b7a5e42a5b81583d445109bfc680d86",
|
||||||
|
"status": "pass",
|
||||||
|
"source_table_count": 39,
|
||||||
|
"target_table_count": 39,
|
||||||
|
"source_total_rows": 52167,
|
||||||
|
"target_total_rows": 52167,
|
||||||
|
"table_mismatches": [],
|
||||||
|
"structural_hashes_equal": true,
|
||||||
|
"constraint_hash": "8b26d26fda56f09325e0603aae48051da08dfc6c35322fa88880acdabb8ad859",
|
||||||
|
"application_role_mismatches": {},
|
||||||
|
"target_extra_application_roles": [],
|
||||||
|
"required_extension_mismatches": {},
|
||||||
|
"performance_problems": [],
|
||||||
|
"network_mode": "none",
|
||||||
|
"persistent_postgres_storage_used": false,
|
||||||
|
"container_absent_after_test": true
|
||||||
|
},
|
||||||
|
"permission_profile": {
|
||||||
|
"approval_policy": "never",
|
||||||
|
"sandbox_mode": "danger-full-access",
|
||||||
|
"network": "enabled"
|
||||||
|
},
|
||||||
|
"config_readback": {
|
||||||
|
"gcloud_sdk_version": "551.0.0",
|
||||||
|
"configured_account": "billy@livingip.xyz",
|
||||||
|
"configured_project": "teleo-501523",
|
||||||
|
"configured_account_status": "failed_noninteractive_reauthentication",
|
||||||
|
"application_default_credentials_status": "failed_noninteractive_reauthentication",
|
||||||
|
"expected_cloudsql_instance_not_currently_verified": "teleo-pgvector-standby",
|
||||||
|
"expected_private_network_not_currently_verified": "teleo-staging-net",
|
||||||
|
"expected_private_address_not_currently_verified": "10.61.0.3",
|
||||||
|
"expected_staging_compute_not_currently_verified": "teleo-prod-1/europe-west6-a",
|
||||||
|
"public_ip_disabled_currently_verified": false,
|
||||||
|
"generated_database_inventory_currently_verified": false
|
||||||
|
},
|
||||||
|
"attempted_no_approval_routes": [
|
||||||
|
"configured privileged gcloud identity",
|
||||||
|
"three other cached gcloud identities",
|
||||||
|
"application-default credentials",
|
||||||
|
"direct SSH to the retained staging address",
|
||||||
|
"local gcloud IAP SSH dry run",
|
||||||
|
"local and VPS TCP probes to the expected private Cloud SQL address",
|
||||||
|
"fixed GitHub IAP status workflow receipt",
|
||||||
|
"five GitHub readiness workflow receipts across candidate service accounts",
|
||||||
|
"current successful Artifact Registry-only WIF workflow"
|
||||||
|
],
|
||||||
|
"exact_gate": "The privileged local Google OAuth session requires human reauthentication, the fixed IAP workflow names an unavailable teleo-iap-operator WIF provider, and no existing noninteractive service-account route can obtain a Cloud SQL/Compute-capable token. Current Cloud SQL topology and generated-database inventory cannot be read, so the private restore cannot start.",
|
||||||
|
"why_autonomous_repair_stops": "Google reauthentication may require a password, MFA, passkey, or consent that Codex must not retrieve or enter. Recreating or rebinding the WIF/IAP identities requires an already-authorized GCP administrator, and no safe noninteractive identity currently has that authority.",
|
||||||
|
"next_non_user_action": "After operator-controlled Google reauthentication or administrator convergence of the reviewed least-privilege IAP operator, read the current Cloud SQL and GCE topology, stream the receipt-bound dump to a teleo_clone_* database from teleo-prod-1, run exact parity, blind reasoning, live GCE metadata attestation, cleanup, postflight source capture, and the eight-receipt lifecycle verifier.",
|
||||||
|
"gcp_effects_from_this_attempt": {
|
||||||
|
"cloudsql_database_created": false,
|
||||||
|
"gcp_resource_created": false,
|
||||||
|
"gcp_resource_modified": false,
|
||||||
|
"public_ip_exposure_changed": false,
|
||||||
|
"staging_promoted": false,
|
||||||
|
"telegram_sent": false
|
||||||
|
},
|
||||||
|
"proof_receipts": {
|
||||||
|
"retained_private_runtime_artifacts": true,
|
||||||
|
"source_receipt": {
|
||||||
|
"filename": "receipt.json",
|
||||||
|
"sha256": "7d96b052591d36df3b67dc6291b0f74a7bbeb9602c9cc424779e7c21194d59a5"
|
||||||
|
},
|
||||||
|
"source_manifest": {
|
||||||
|
"filename": "source-manifest.jsonl",
|
||||||
|
"sha256": "39f06c6b9fc806a89ecd19294be29c58c499c65234611c7a483ddefa3c78d7d3"
|
||||||
|
},
|
||||||
|
"source_delta": {
|
||||||
|
"filename": "source-stability-delta.json",
|
||||||
|
"sha256": "d761a2b8b0313ee03c9d01e50fcca709af779578281cf10de5aed4709e914d72"
|
||||||
|
},
|
||||||
|
"isolated_restore": {
|
||||||
|
"filename": "local-rebuild-receipt.json",
|
||||||
|
"sha256": "181901afc91bebee7c343500c601ed1e5b7a5e42a5b81583d445109bfc680d86"
|
||||||
|
},
|
||||||
|
"route_recheck": {
|
||||||
|
"filename": "route-recheck.json",
|
||||||
|
"sha256": "0fda5e6d58d72dcb084dc277bc59312bf62c002c728e249a9f16e79ae096037b"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"strongest_claim_allowed": "The newest authorized VPS snapshot is stable and exactly reconstructs in an isolated PostgreSQL 16 canary; current private GCP restore, reasoning, and cleanup are not proven because the live operator route is externally gated.",
|
||||||
|
"not_proven": [
|
||||||
|
"current Cloud SQL topology or public-IP-disabled state",
|
||||||
|
"current generated-database inventory",
|
||||||
|
"private staging-to-Cloud-SQL TLS connectivity",
|
||||||
|
"current Cloud SQL restore parity",
|
||||||
|
"blind Leo reasoning against the generated Cloud SQL database",
|
||||||
|
"generated Cloud SQL database teardown"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,36 @@
|
||||||
|
{
|
||||||
|
"artifact": "gcp_canonical_parity_attempt_teardown",
|
||||||
|
"schema": "livingip.gcpCanonicalParityAttemptTeardown.v1",
|
||||||
|
"generated_at_utc": "2026-07-15T01:50:20Z",
|
||||||
|
"status": "no_cloud_resource_created",
|
||||||
|
"required_tier": "T3_live_private_gcp_staging",
|
||||||
|
"required_tier_met": false,
|
||||||
|
"cloudsql_clone_created": false,
|
||||||
|
"cloudsql_clone_dropped": false,
|
||||||
|
"cloudsql_clone_absence_readback": "not_applicable_no_create",
|
||||||
|
"gcs_object_created": false,
|
||||||
|
"compute_resource_created": false,
|
||||||
|
"cleanup_required": false,
|
||||||
|
"cleanup_performed": false,
|
||||||
|
"orphan_risk_from_this_attempt": false,
|
||||||
|
"public_ip_exposure_changed": false,
|
||||||
|
"staging_promoted": false,
|
||||||
|
"local_restore_container": {
|
||||||
|
"name": "teleo-canonical-rebuild-20260715015004-4ec8d1daac32",
|
||||||
|
"force_remove_exit_code": 0,
|
||||||
|
"container_absent": true,
|
||||||
|
"persistent_storage_used": false,
|
||||||
|
"network_access_available": false
|
||||||
|
},
|
||||||
|
"current_gcp_orphan_inventory_verified": false,
|
||||||
|
"exact_gate": "No authenticated Cloud SQL/Compute-capable route was available, so execution stopped before any GCP resource or generated database was created.",
|
||||||
|
"proof_paths": {
|
||||||
|
"live_attempt": "docs/reports/leo-working-state-20260709/gcp-canonical-parity-live-20260715.json",
|
||||||
|
"route_recheck_filename": "route-recheck.json",
|
||||||
|
"route_recheck_sha256": "0fda5e6d58d72dcb084dc277bc59312bf62c002c728e249a9f16e79ae096037b",
|
||||||
|
"local_cleanup_receipt_filename": "local-rebuild-receipt.json",
|
||||||
|
"local_cleanup_receipt_sha256": "181901afc91bebee7c343500c601ed1e5b7a5e42a5b81583d445109bfc680d86",
|
||||||
|
"runtime_receipts_retained_privately": true
|
||||||
|
},
|
||||||
|
"strongest_claim_allowed": "This attempt introduced no GCP cleanup debt and its isolated local restore container was removed; it is not a teardown receipt for a live disposable Cloud SQL database."
|
||||||
|
}
|
||||||
172
ops/attest_gcp_reasoning_compute.py
Normal file
172
ops/attest_gcp_reasoning_compute.py
Normal file
|
|
@ -0,0 +1,172 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Bind a no-send reasoning receipt to live GCE metadata on the replay host."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import stat
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
try:
|
||||||
|
from .private_receipt_io import write_private_json
|
||||||
|
from .restore_gcp_generated_postgres_snapshot import (
|
||||||
|
DEFAULT_COMPUTE_INSTANCE,
|
||||||
|
DEFAULT_COMPUTE_ZONE,
|
||||||
|
DEFAULT_PRIVATE_NETWORK,
|
||||||
|
DEFAULT_PROJECT,
|
||||||
|
RUN_ID_RE,
|
||||||
|
TARGET_DATABASE_RE,
|
||||||
|
gce_compute_identity,
|
||||||
|
sha256_file,
|
||||||
|
)
|
||||||
|
except ImportError: # pragma: no cover - direct script execution on the GCP VM
|
||||||
|
from private_receipt_io import write_private_json
|
||||||
|
from restore_gcp_generated_postgres_snapshot import (
|
||||||
|
DEFAULT_COMPUTE_INSTANCE,
|
||||||
|
DEFAULT_COMPUTE_ZONE,
|
||||||
|
DEFAULT_PRIVATE_NETWORK,
|
||||||
|
DEFAULT_PROJECT,
|
||||||
|
RUN_ID_RE,
|
||||||
|
TARGET_DATABASE_RE,
|
||||||
|
gce_compute_identity,
|
||||||
|
sha256_file,
|
||||||
|
)
|
||||||
|
|
||||||
|
REASONING_SCHEMA = "livingip.gcpGeneratedDbBlindClaimCanary.v1"
|
||||||
|
ATTESTATION_SCHEMA = "livingip.gcpReasoningComputeAttestation.v1"
|
||||||
|
|
||||||
|
|
||||||
|
def utc_now() -> str:
|
||||||
|
return datetime.now(UTC).isoformat()
|
||||||
|
|
||||||
|
|
||||||
|
def parse_timestamp(value: Any, label: str) -> datetime:
|
||||||
|
text = str(value or "")
|
||||||
|
if text.endswith("Z"):
|
||||||
|
text = text[:-1] + "+00:00"
|
||||||
|
try:
|
||||||
|
parsed = datetime.fromisoformat(text)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise ValueError(f"{label} is not an ISO-8601 timestamp") from exc
|
||||||
|
if parsed.tzinfo is None:
|
||||||
|
raise ValueError(f"{label} must include a timezone")
|
||||||
|
return parsed.astimezone(UTC)
|
||||||
|
|
||||||
|
|
||||||
|
def load_reasoning_receipt(path: Path) -> dict[str, Any]:
|
||||||
|
try:
|
||||||
|
mode = path.lstat().st_mode
|
||||||
|
except OSError as exc:
|
||||||
|
raise ValueError(f"reasoning receipt is unavailable: {exc}") from exc
|
||||||
|
if not stat.S_ISREG(mode) or path.is_symlink():
|
||||||
|
raise ValueError("reasoning receipt must be a regular non-symlink file")
|
||||||
|
try:
|
||||||
|
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise ValueError("reasoning receipt is invalid JSON") from exc
|
||||||
|
if not isinstance(payload, dict):
|
||||||
|
raise ValueError("reasoning receipt must be a JSON object")
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
def attest_reasoning_compute(
|
||||||
|
*,
|
||||||
|
reasoning_receipt_path: Path,
|
||||||
|
restore_run_id: str,
|
||||||
|
target_database: str,
|
||||||
|
project: str = DEFAULT_PROJECT,
|
||||||
|
expected_compute_instance: str = DEFAULT_COMPUTE_INSTANCE,
|
||||||
|
expected_compute_zone: str = DEFAULT_COMPUTE_ZONE,
|
||||||
|
expected_private_network: str = DEFAULT_PRIVATE_NETWORK,
|
||||||
|
metadata_timeout: float = 10.0,
|
||||||
|
generated_at_utc: str | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
if not RUN_ID_RE.fullmatch(restore_run_id):
|
||||||
|
raise ValueError("restore run id is invalid")
|
||||||
|
if not TARGET_DATABASE_RE.fullmatch(target_database):
|
||||||
|
raise ValueError("target database must be a disposable teleo_clone_* database")
|
||||||
|
reasoning_receipt_path = reasoning_receipt_path.resolve()
|
||||||
|
reasoning = load_reasoning_receipt(reasoning_receipt_path)
|
||||||
|
started = parse_timestamp(reasoning.get("generated_at_utc"), "reasoning start")
|
||||||
|
completed = parse_timestamp(reasoning.get("completed_at_utc"), "reasoning completion")
|
||||||
|
checks = {
|
||||||
|
"schema": reasoning.get("schema") == REASONING_SCHEMA,
|
||||||
|
"status": reasoning.get("status") == "pass" and not reasoning.get("errors"),
|
||||||
|
"target_database": reasoning.get("target_database") == target_database,
|
||||||
|
"source_compute": reasoning.get("source_compute") == expected_compute_instance,
|
||||||
|
"chronology": started <= completed,
|
||||||
|
"no_telegram_send": reasoning.get("posted_to_telegram") is False,
|
||||||
|
"no_database_write": reasoning.get("database_write_attempted") is False,
|
||||||
|
}
|
||||||
|
failed = sorted(name for name, passed in checks.items() if not passed)
|
||||||
|
if failed:
|
||||||
|
raise ValueError("reasoning receipt failed attestation preflight: " + ", ".join(failed))
|
||||||
|
|
||||||
|
compute = gce_compute_identity(
|
||||||
|
project,
|
||||||
|
expected_compute_instance,
|
||||||
|
expected_compute_zone,
|
||||||
|
expected_private_network,
|
||||||
|
timeout=metadata_timeout,
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"artifact": "gcp_reasoning_compute_attestation",
|
||||||
|
"schema": ATTESTATION_SCHEMA,
|
||||||
|
"status": "pass",
|
||||||
|
"generated_at_utc": generated_at_utc or utc_now(),
|
||||||
|
"restore_run_id": restore_run_id,
|
||||||
|
"target_database": target_database,
|
||||||
|
"reasoning_receipt": {
|
||||||
|
"sha256": sha256_file(reasoning_receipt_path),
|
||||||
|
"schema": reasoning["schema"],
|
||||||
|
"generated_at_utc": reasoning["generated_at_utc"],
|
||||||
|
"completed_at_utc": reasoning["completed_at_utc"],
|
||||||
|
"claimed_source_compute": reasoning["source_compute"],
|
||||||
|
},
|
||||||
|
"compute": compute,
|
||||||
|
"checks": {**checks, "gce_metadata_identity": all(compute.get("checks", {}).values())},
|
||||||
|
"method": "gce_metadata_server_v1",
|
||||||
|
"mutation": {
|
||||||
|
"cloudsql_changed": False,
|
||||||
|
"compute_changed": False,
|
||||||
|
"database_changed": False,
|
||||||
|
"telegram_message_sent": False,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--reasoning-receipt", required=True, type=Path)
|
||||||
|
parser.add_argument("--restore-run-id", required=True)
|
||||||
|
parser.add_argument("--target-db", required=True)
|
||||||
|
parser.add_argument("--project", default=DEFAULT_PROJECT)
|
||||||
|
parser.add_argument("--expected-compute-instance", default=DEFAULT_COMPUTE_INSTANCE)
|
||||||
|
parser.add_argument("--expected-compute-zone", default=DEFAULT_COMPUTE_ZONE)
|
||||||
|
parser.add_argument("--expected-private-network", default=DEFAULT_PRIVATE_NETWORK)
|
||||||
|
parser.add_argument("--metadata-timeout", default=10.0, type=float)
|
||||||
|
parser.add_argument("--output", required=True, type=Path)
|
||||||
|
args = parser.parse_args()
|
||||||
|
try:
|
||||||
|
receipt = attest_reasoning_compute(
|
||||||
|
reasoning_receipt_path=args.reasoning_receipt,
|
||||||
|
restore_run_id=args.restore_run_id,
|
||||||
|
target_database=args.target_db,
|
||||||
|
project=args.project,
|
||||||
|
expected_compute_instance=args.expected_compute_instance,
|
||||||
|
expected_compute_zone=args.expected_compute_zone,
|
||||||
|
expected_private_network=args.expected_private_network,
|
||||||
|
metadata_timeout=args.metadata_timeout,
|
||||||
|
)
|
||||||
|
write_private_json(args.output, receipt)
|
||||||
|
except (OSError, ValueError, RuntimeError) as exc:
|
||||||
|
parser.error(str(exc))
|
||||||
|
print(json.dumps(receipt, indent=2, sort_keys=True))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
|
|
@ -25,15 +25,20 @@ except ImportError: # pragma: no cover - direct script execution
|
||||||
|
|
||||||
SAFE_NAME_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}\Z")
|
SAFE_NAME_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}\Z")
|
||||||
SAFE_RUN_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_-]{7,63}\Z")
|
SAFE_RUN_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_-]{7,63}\Z")
|
||||||
SAFE_SSH_TARGET_RE = re.compile(
|
SAFE_AUTHORIZATION_REF_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.:/@+-]{7,255}\Z")
|
||||||
r"(?:[A-Za-z0-9][A-Za-z0-9._-]{0,63}@)?[A-Za-z0-9][A-Za-z0-9.-]{0,252}\Z"
|
SAFE_SSH_TARGET_RE = re.compile(r"(?:[A-Za-z0-9][A-Za-z0-9._-]{0,63}@)?[A-Za-z0-9][A-Za-z0-9.-]{0,252}\Z")
|
||||||
)
|
SNAPSHOT_ID_RE = re.compile(r"[0-9A-F]+-[0-9A-F]+-[0-9]+\Z")
|
||||||
|
TXID_SNAPSHOT_RE = re.compile(r"[0-9]+:[0-9]+:(?:[0-9]+(?:,[0-9]+)*)?\Z")
|
||||||
|
WAL_LSN_RE = re.compile(r"[0-9A-F]+/[0-9A-F]+\Z")
|
||||||
|
SYSTEM_IDENTIFIER_RE = re.compile(r"[0-9]+\Z")
|
||||||
|
SOURCE_SNAPSHOT_RECEIPT_SCHEMA = "livingip.sourceSnapshotReceipt.v2"
|
||||||
EXPECTED_FILES = {
|
EXPECTED_FILES = {
|
||||||
"teleo-canonical.dump",
|
"teleo-canonical.dump",
|
||||||
"teleo-canonical.dump.sha256",
|
"teleo-canonical.dump.sha256",
|
||||||
"teleo-canonical.toc",
|
"teleo-canonical.toc",
|
||||||
"source-manifest.jsonl",
|
"source-manifest.jsonl",
|
||||||
"source-context.txt",
|
"source-context.txt",
|
||||||
|
"source-snapshot.json",
|
||||||
"service-before.txt",
|
"service-before.txt",
|
||||||
"service-after.txt",
|
"service-after.txt",
|
||||||
}
|
}
|
||||||
|
|
@ -47,6 +52,81 @@ def sha256(path: Path) -> str:
|
||||||
return digest.hexdigest()
|
return digest.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def sha256_bytes(payload: bytes) -> str:
|
||||||
|
return hashlib.sha256(payload).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def canonical_sha256(value: Any) -> str:
|
||||||
|
payload = json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True)
|
||||||
|
return sha256_bytes(payload.encode())
|
||||||
|
|
||||||
|
|
||||||
|
def load_snapshot_metadata(path: Path) -> dict[str, str]:
|
||||||
|
try:
|
||||||
|
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
except (OSError, json.JSONDecodeError) as exc:
|
||||||
|
raise RuntimeError("source snapshot metadata is unreadable") from exc
|
||||||
|
if not isinstance(payload, dict):
|
||||||
|
raise RuntimeError("source snapshot metadata must be an object")
|
||||||
|
fields = {
|
||||||
|
"exported_snapshot_id": SNAPSHOT_ID_RE,
|
||||||
|
"txid_snapshot": TXID_SNAPSHOT_RE,
|
||||||
|
"wal_lsn": WAL_LSN_RE,
|
||||||
|
"system_identifier": SYSTEM_IDENTIFIER_RE,
|
||||||
|
}
|
||||||
|
normalized: dict[str, str] = {}
|
||||||
|
for field, pattern in fields.items():
|
||||||
|
value = str(payload.get(field) or "")
|
||||||
|
if not pattern.fullmatch(value):
|
||||||
|
raise RuntimeError(f"source snapshot metadata has invalid {field}")
|
||||||
|
normalized[field] = value
|
||||||
|
captured_at_utc = str(payload.get("captured_at_utc") or "")
|
||||||
|
if not captured_at_utc or any(character in captured_at_utc for character in "\r\n\x00"):
|
||||||
|
raise RuntimeError("source snapshot metadata has invalid captured_at_utc")
|
||||||
|
normalized["captured_at_utc"] = captured_at_utc
|
||||||
|
return normalized
|
||||||
|
|
||||||
|
|
||||||
|
def load_service_state(path: Path) -> dict[str, str]:
|
||||||
|
try:
|
||||||
|
state = dict(line.split("=", 1) for line in path.read_text(encoding="utf-8").splitlines() if "=" in line)
|
||||||
|
main_pid = int(state.get("MainPID") or 0)
|
||||||
|
restarts = int(state.get("NRestarts") or 0)
|
||||||
|
except (OSError, TypeError, ValueError) as exc:
|
||||||
|
raise RuntimeError("source service state is unreadable") from exc
|
||||||
|
if state.get("ActiveState") != "active" or state.get("SubState") != "running" or main_pid <= 0 or restarts < 0:
|
||||||
|
raise RuntimeError("source service is not active/running with a positive MainPID")
|
||||||
|
return state
|
||||||
|
|
||||||
|
|
||||||
|
def build_provenance_binding(
|
||||||
|
*,
|
||||||
|
run_id: str,
|
||||||
|
authorization_ref: str,
|
||||||
|
source: dict[str, str],
|
||||||
|
snapshot: dict[str, str],
|
||||||
|
dump_sha256: str,
|
||||||
|
manifest_sql_sha256: str,
|
||||||
|
source_manifest_sha256: str,
|
||||||
|
source_context_sha256: str,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
payload = {
|
||||||
|
"receipt_schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
|
||||||
|
"run_id": run_id,
|
||||||
|
"capture_authorization_ref": authorization_ref,
|
||||||
|
"source": source,
|
||||||
|
"exported_snapshot_id": snapshot["exported_snapshot_id"],
|
||||||
|
"txid_snapshot": snapshot["txid_snapshot"],
|
||||||
|
"wal_lsn": snapshot["wal_lsn"],
|
||||||
|
"source_system_identifier": snapshot["system_identifier"],
|
||||||
|
"dump_sha256": dump_sha256,
|
||||||
|
"manifest_sql_sha256": manifest_sql_sha256,
|
||||||
|
"source_manifest_sha256": source_manifest_sha256,
|
||||||
|
"source_context_sha256": source_context_sha256,
|
||||||
|
}
|
||||||
|
return {"algorithm": "sha256", "payload": payload, "sha256": canonical_sha256(payload)}
|
||||||
|
|
||||||
|
|
||||||
def ssh_base(identity_file: Path) -> list[str]:
|
def ssh_base(identity_file: Path) -> list[str]:
|
||||||
return [
|
return [
|
||||||
"ssh",
|
"ssh",
|
||||||
|
|
@ -73,12 +153,12 @@ def remote_capture_script(*, run_id: str, container: str, database: str, service
|
||||||
"container_dump": shlex.quote(container_dump),
|
"container_dump": shlex.quote(container_dump),
|
||||||
}
|
}
|
||||||
return f"""set -euo pipefail
|
return f"""set -euo pipefail
|
||||||
remote_root={quoted['remote_root']}
|
remote_root={quoted["remote_root"]}
|
||||||
container={quoted['container']}
|
container={quoted["container"]}
|
||||||
database={quoted['database']}
|
database={quoted["database"]}
|
||||||
service={quoted['service']}
|
service={quoted["service"]}
|
||||||
container_manifest={quoted['container_manifest']}
|
container_manifest={quoted["container_manifest"]}
|
||||||
container_dump={quoted['container_dump']}
|
container_dump={quoted["container_dump"]}
|
||||||
|
|
||||||
cleanup() {{
|
cleanup() {{
|
||||||
docker exec "$container" rm -f "$container_manifest" "$container_dump" >/dev/null 2>&1 || true
|
docker exec "$container" rm -f "$container_manifest" "$container_dump" >/dev/null 2>&1 || true
|
||||||
|
|
@ -86,11 +166,23 @@ cleanup() {{
|
||||||
}}
|
}}
|
||||||
trap cleanup EXIT
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
assert_service_healthy() {{
|
||||||
|
local state_file="$1"
|
||||||
|
grep -qx 'ActiveState=active' "$state_file"
|
||||||
|
grep -qx 'SubState=running' "$state_file"
|
||||||
|
local main_pid restarts
|
||||||
|
main_pid="$(sed -n 's/^MainPID=//p' "$state_file")"
|
||||||
|
restarts="$(sed -n 's/^NRestarts=//p' "$state_file")"
|
||||||
|
[[ "$main_pid" =~ ^[1-9][0-9]*$ ]]
|
||||||
|
[[ "$restarts" =~ ^[0-9]+$ ]]
|
||||||
|
}}
|
||||||
|
|
||||||
test -d "$remote_root"
|
test -d "$remote_root"
|
||||||
test -f "$remote_root/postgres-parity-manifest.sql"
|
test -f "$remote_root/postgres-parity-manifest.sql"
|
||||||
docker inspect "$container" >/dev/null
|
docker inspect "$container" >/dev/null
|
||||||
docker exec "$container" pg_isready -U postgres -d "$database" >/dev/null
|
docker exec "$container" pg_isready -U postgres -d "$database" >/dev/null
|
||||||
systemctl show "$service" -p ActiveState -p SubState -p MainPID -p NRestarts -p ActiveEnterTimestamp --no-pager > "$remote_root/service-before.txt"
|
systemctl show "$service" -p ActiveState -p SubState -p MainPID -p NRestarts -p ActiveEnterTimestamp --no-pager > "$remote_root/service-before.txt"
|
||||||
|
assert_service_healthy "$remote_root/service-before.txt"
|
||||||
docker inspect "$container" --format 'container_id={{{{.Id}}}}\nrunning={{{{.State.Running}}}}\nnetwork_mode={{{{.HostConfig.NetworkMode}}}}' > "$remote_root/source-context.txt"
|
docker inspect "$container" --format 'container_id={{{{.Id}}}}\nrunning={{{{.State.Running}}}}\nnetwork_mode={{{{.HostConfig.NetworkMode}}}}' > "$remote_root/source-context.txt"
|
||||||
printf 'database=%s\ncaptured_at_utc=%s\n' "$database" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$remote_root/source-context.txt"
|
printf 'database=%s\ncaptured_at_utc=%s\n' "$database" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$remote_root/source-context.txt"
|
||||||
docker cp "$remote_root/postgres-parity-manifest.sql" "$container:$container_manifest" >/dev/null
|
docker cp "$remote_root/postgres-parity-manifest.sql" "$container:$container_manifest" >/dev/null
|
||||||
|
|
@ -107,6 +199,19 @@ if [[ ! "$snapshot_id" =~ ^[0-9A-F]+-[0-9A-F]+-[0-9]+$ ]]; then
|
||||||
echo "invalid exported snapshot id" >&2
|
echo "invalid exported snapshot id" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
printf '%s\n' "select jsonb_build_object(
|
||||||
|
'exported_snapshot_id', '$snapshot_id',
|
||||||
|
'txid_snapshot', txid_current_snapshot()::text,
|
||||||
|
'wal_lsn', pg_current_wal_lsn()::text,
|
||||||
|
'system_identifier', (select system_identifier::text from pg_control_system()),
|
||||||
|
'captured_at_utc', clock_timestamp()
|
||||||
|
)::text;" >&3
|
||||||
|
IFS= read -r snapshot_metadata <&4
|
||||||
|
if [[ -z "$snapshot_metadata" ]]; then
|
||||||
|
echo "invalid source snapshot metadata" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
printf '%s\n' "$snapshot_metadata" > "$remote_root/source-snapshot.json"
|
||||||
|
|
||||||
docker exec "$container" pg_dump \
|
docker exec "$container" pg_dump \
|
||||||
-U postgres -d "$database" \
|
-U postgres -d "$database" \
|
||||||
|
|
@ -130,6 +235,7 @@ docker cp "$container:$container_dump" "$remote_root/teleo-canonical.dump" >/dev
|
||||||
sha256sum "$remote_root/teleo-canonical.dump" > "$remote_root/teleo-canonical.dump.sha256"
|
sha256sum "$remote_root/teleo-canonical.dump" > "$remote_root/teleo-canonical.dump.sha256"
|
||||||
printf 'snapshot_exported=true\n' >> "$remote_root/source-context.txt"
|
printf 'snapshot_exported=true\n' >> "$remote_root/source-context.txt"
|
||||||
systemctl show "$service" -p ActiveState -p SubState -p MainPID -p NRestarts -p ActiveEnterTimestamp --no-pager > "$remote_root/service-after.txt"
|
systemctl show "$service" -p ActiveState -p SubState -p MainPID -p NRestarts -p ActiveEnterTimestamp --no-pager > "$remote_root/service-after.txt"
|
||||||
|
assert_service_healthy "$remote_root/service-after.txt"
|
||||||
cmp -s "$remote_root/service-before.txt" "$remote_root/service-after.txt"
|
cmp -s "$remote_root/service-before.txt" "$remote_root/service-after.txt"
|
||||||
|
|
||||||
tar -C "$remote_root" -cf - \
|
tar -C "$remote_root" -cf - \
|
||||||
|
|
@ -138,12 +244,13 @@ tar -C "$remote_root" -cf - \
|
||||||
teleo-canonical.toc \
|
teleo-canonical.toc \
|
||||||
source-manifest.jsonl \
|
source-manifest.jsonl \
|
||||||
source-context.txt \
|
source-context.txt \
|
||||||
|
source-snapshot.json \
|
||||||
service-before.txt \
|
service-before.txt \
|
||||||
service-after.txt
|
service-after.txt
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
||||||
def upload_manifest(ssh: list[str], target: str, remote_root: str, manifest: Path) -> None:
|
def upload_manifest(ssh: list[str], target: str, remote_root: str, manifest: bytes) -> None:
|
||||||
command = (
|
command = (
|
||||||
f"umask 077; rm -rf {shlex.quote(remote_root)}; "
|
f"umask 077; rm -rf {shlex.quote(remote_root)}; "
|
||||||
f"mkdir -m 700 {shlex.quote(remote_root)}; "
|
f"mkdir -m 700 {shlex.quote(remote_root)}; "
|
||||||
|
|
@ -151,7 +258,7 @@ def upload_manifest(ssh: list[str], target: str, remote_root: str, manifest: Pat
|
||||||
)
|
)
|
||||||
completed = subprocess.run(
|
completed = subprocess.run(
|
||||||
[*ssh, target, "--", command],
|
[*ssh, target, "--", command],
|
||||||
input=manifest.read_bytes(),
|
input=manifest,
|
||||||
capture_output=True,
|
capture_output=True,
|
||||||
check=False,
|
check=False,
|
||||||
)
|
)
|
||||||
|
|
@ -206,8 +313,10 @@ def capture(args: argparse.Namespace) -> dict[str, Any]:
|
||||||
output_dir.mkdir(mode=0o700)
|
output_dir.mkdir(mode=0o700)
|
||||||
remote_root = f"/tmp/teleo-canonical-snapshot-{args.run_id}"
|
remote_root = f"/tmp/teleo-canonical-snapshot-{args.run_id}"
|
||||||
ssh = ssh_base(args.ssh_key.resolve())
|
ssh = ssh_base(args.ssh_key.resolve())
|
||||||
|
manifest_sql = args.manifest.resolve().read_bytes()
|
||||||
|
manifest_sql_sha256 = sha256_bytes(manifest_sql)
|
||||||
try:
|
try:
|
||||||
upload_manifest(ssh, args.ssh_target, remote_root, args.manifest.resolve())
|
upload_manifest(ssh, args.ssh_target, remote_root, manifest_sql)
|
||||||
script = remote_capture_script(
|
script = remote_capture_script(
|
||||||
run_id=args.run_id,
|
run_id=args.run_id,
|
||||||
container=args.container,
|
container=args.container,
|
||||||
|
|
@ -233,19 +342,38 @@ def capture(args: argparse.Namespace) -> dict[str, Any]:
|
||||||
if expected_sha != actual_sha:
|
if expected_sha != actual_sha:
|
||||||
raise RuntimeError("captured dump SHA-256 mismatch")
|
raise RuntimeError("captured dump SHA-256 mismatch")
|
||||||
manifest = load_manifest(output_dir / "source-manifest.jsonl")
|
manifest = load_manifest(output_dir / "source-manifest.jsonl")
|
||||||
before = (output_dir / "service-before.txt").read_text()
|
snapshot = load_snapshot_metadata(output_dir / "source-snapshot.json")
|
||||||
after = (output_dir / "service-after.txt").read_text()
|
before = load_service_state(output_dir / "service-before.txt")
|
||||||
|
after = load_service_state(output_dir / "service-after.txt")
|
||||||
|
source = {
|
||||||
|
"ssh_target": args.ssh_target,
|
||||||
|
"container": args.container,
|
||||||
|
"database": args.database,
|
||||||
|
"service": args.service,
|
||||||
|
}
|
||||||
|
source_manifest_sha256 = sha256(output_dir / "source-manifest.jsonl")
|
||||||
|
source_context_sha256 = sha256(output_dir / "source-context.txt")
|
||||||
|
provenance_binding = build_provenance_binding(
|
||||||
|
run_id=args.run_id,
|
||||||
|
authorization_ref=args.authorization_ref,
|
||||||
|
source=source,
|
||||||
|
snapshot=snapshot,
|
||||||
|
dump_sha256=actual_sha,
|
||||||
|
manifest_sql_sha256=manifest_sql_sha256,
|
||||||
|
source_manifest_sha256=source_manifest_sha256,
|
||||||
|
source_context_sha256=source_context_sha256,
|
||||||
|
)
|
||||||
return {
|
return {
|
||||||
"artifact": "vps_canonical_postgres_exported_snapshot",
|
"artifact": "vps_canonical_postgres_exported_snapshot",
|
||||||
|
"schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
|
||||||
|
"receipt_version": 2,
|
||||||
"generated_at_utc": datetime.now(UTC).isoformat(),
|
"generated_at_utc": datetime.now(UTC).isoformat(),
|
||||||
"status": "pass",
|
"status": "pass",
|
||||||
"source": {
|
"run_id": args.run_id,
|
||||||
"ssh_target": args.ssh_target,
|
"capture_authorization_ref": args.authorization_ref,
|
||||||
"container": args.container,
|
"source": source,
|
||||||
"database": args.database,
|
|
||||||
"service": args.service,
|
|
||||||
},
|
|
||||||
"snapshot_exported": True,
|
"snapshot_exported": True,
|
||||||
|
"snapshot": snapshot,
|
||||||
"dump": {
|
"dump": {
|
||||||
"path": str(dump_path),
|
"path": str(dump_path),
|
||||||
"bytes": dump_path.stat().st_size,
|
"bytes": dump_path.stat().st_size,
|
||||||
|
|
@ -254,10 +382,18 @@ def capture(args: argparse.Namespace) -> dict[str, Any]:
|
||||||
},
|
},
|
||||||
"manifest": {
|
"manifest": {
|
||||||
"path": str(output_dir / "source-manifest.jsonl"),
|
"path": str(output_dir / "source-manifest.jsonl"),
|
||||||
|
"sha256": source_manifest_sha256,
|
||||||
|
"manifest_sql_sha256": manifest_sql_sha256,
|
||||||
"table_count": len(manifest["tables"]),
|
"table_count": len(manifest["tables"]),
|
||||||
"total_rows": sum(int(row["row_count"]) for row in manifest["tables"].values()),
|
"total_rows": sum(int(row["row_count"]) for row in manifest["tables"].values()),
|
||||||
"application_roles": manifest["singleton"]["application_roles"]["items"],
|
"application_roles": manifest["singleton"]["application_roles"]["items"],
|
||||||
},
|
},
|
||||||
|
"source_context": {
|
||||||
|
"path": str(output_dir / "source-context.txt"),
|
||||||
|
"sha256": source_context_sha256,
|
||||||
|
},
|
||||||
|
"source_service": {"before": before, "after": after, "unchanged": before == after},
|
||||||
|
"provenance_binding": provenance_binding,
|
||||||
"service_unchanged": before == after,
|
"service_unchanged": before == after,
|
||||||
"production_db_mutated": False,
|
"production_db_mutated": False,
|
||||||
"telegram_send_attempted": False,
|
"telegram_send_attempted": False,
|
||||||
|
|
@ -269,7 +405,7 @@ def cleanup_failed_output(output_dir: Path, *, preexisting: bool) -> None:
|
||||||
shutil.rmtree(output_dir)
|
shutil.rmtree(output_dir)
|
||||||
|
|
||||||
|
|
||||||
def parse_args() -> argparse.Namespace:
|
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||||
parser = argparse.ArgumentParser(description=__doc__)
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
parser.add_argument("--execute", action="store_true")
|
parser.add_argument("--execute", action="store_true")
|
||||||
parser.add_argument("--ssh-target", required=True)
|
parser.add_argument("--ssh-target", required=True)
|
||||||
|
|
@ -279,9 +415,10 @@ def parse_args() -> argparse.Namespace:
|
||||||
parser.add_argument("--service", default="leoclean-gateway.service")
|
parser.add_argument("--service", default="leoclean-gateway.service")
|
||||||
parser.add_argument("--manifest", default=Path("ops/postgres_parity_manifest.sql"), type=Path)
|
parser.add_argument("--manifest", default=Path("ops/postgres_parity_manifest.sql"), type=Path)
|
||||||
parser.add_argument("--run-id", required=True)
|
parser.add_argument("--run-id", required=True)
|
||||||
|
parser.add_argument("--authorization-ref", required=True)
|
||||||
parser.add_argument("--output-dir", required=True, type=Path)
|
parser.add_argument("--output-dir", required=True, type=Path)
|
||||||
parser.add_argument("--timeout", default=180, type=int)
|
parser.add_argument("--timeout", default=180, type=int)
|
||||||
args = parser.parse_args()
|
args = parser.parse_args(argv)
|
||||||
for value, flag in ((args.container, "--container"), (args.database, "--database"), (args.service, "--service")):
|
for value, flag in ((args.container, "--container"), (args.database, "--database"), (args.service, "--service")):
|
||||||
if not SAFE_NAME_RE.fullmatch(value):
|
if not SAFE_NAME_RE.fullmatch(value):
|
||||||
parser.error(f"{flag} contains unsafe characters")
|
parser.error(f"{flag} contains unsafe characters")
|
||||||
|
|
@ -289,6 +426,8 @@ def parse_args() -> argparse.Namespace:
|
||||||
parser.error("--ssh-target must be a host or user@host without options")
|
parser.error("--ssh-target must be a host or user@host without options")
|
||||||
if not SAFE_RUN_RE.fullmatch(args.run_id):
|
if not SAFE_RUN_RE.fullmatch(args.run_id):
|
||||||
parser.error("--run-id must be 8-64 safe characters")
|
parser.error("--run-id must be 8-64 safe characters")
|
||||||
|
if not SAFE_AUTHORIZATION_REF_RE.fullmatch(args.authorization_ref):
|
||||||
|
parser.error("--authorization-ref must be 8-256 safe metadata characters")
|
||||||
if not args.ssh_key.is_file():
|
if not args.ssh_key.is_file():
|
||||||
parser.error("--ssh-key must be an existing file")
|
parser.error("--ssh-key must be an existing file")
|
||||||
if not args.manifest.is_file():
|
if not args.manifest.is_file():
|
||||||
|
|
@ -298,8 +437,12 @@ def parse_args() -> argparse.Namespace:
|
||||||
json.dumps(
|
json.dumps(
|
||||||
{
|
{
|
||||||
"artifact": "vps_canonical_postgres_exported_snapshot_plan",
|
"artifact": "vps_canonical_postgres_exported_snapshot_plan",
|
||||||
|
"schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
|
||||||
"status": "dry_run",
|
"status": "dry_run",
|
||||||
|
"run_id": args.run_id,
|
||||||
|
"capture_authorization_ref": args.authorization_ref,
|
||||||
"source": f"{args.ssh_target}:{args.container}/{args.database}",
|
"source": f"{args.ssh_target}:{args.container}/{args.database}",
|
||||||
|
"manifest_sql_sha256": sha256(args.manifest.resolve()),
|
||||||
"output_dir": str(args.output_dir),
|
"output_dir": str(args.output_dir),
|
||||||
"mutates_production_db": False,
|
"mutates_production_db": False,
|
||||||
},
|
},
|
||||||
|
|
@ -321,8 +464,11 @@ def main() -> int:
|
||||||
cleanup_failed_output(output_dir, preexisting=output_preexisted)
|
cleanup_failed_output(output_dir, preexisting=output_preexisted)
|
||||||
payload = {
|
payload = {
|
||||||
"artifact": "vps_canonical_postgres_exported_snapshot",
|
"artifact": "vps_canonical_postgres_exported_snapshot",
|
||||||
|
"schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
|
||||||
"generated_at_utc": datetime.now(UTC).isoformat(),
|
"generated_at_utc": datetime.now(UTC).isoformat(),
|
||||||
"status": "fail",
|
"status": "fail",
|
||||||
|
"run_id": args.run_id,
|
||||||
|
"capture_authorization_ref": args.authorization_ref,
|
||||||
"error": str(exc),
|
"error": str(exc),
|
||||||
}
|
}
|
||||||
print(json.dumps(payload, indent=2, sort_keys=True))
|
print(json.dumps(payload, indent=2, sort_keys=True))
|
||||||
|
|
|
||||||
34
ops/private_receipt_io.py
Normal file
34
ops/private_receipt_io.py
Normal file
|
|
@ -0,0 +1,34 @@
|
||||||
|
"""Atomic private JSON receipt output shared by GCP parity helpers."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import uuid
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
|
def write_private_json(path: Path, payload: dict[str, Any]) -> None:
|
||||||
|
"""Write one JSON receipt as mode 0600 without chmodding existing parents."""
|
||||||
|
parent_preexisting = path.parent.exists()
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||||
|
if not path.parent.is_dir() or path.parent.is_symlink():
|
||||||
|
raise ValueError("output parent must be a real directory")
|
||||||
|
if not parent_preexisting:
|
||||||
|
os.chmod(path.parent, 0o700)
|
||||||
|
temporary = path.parent / f".{path.name}.{uuid.uuid4().hex}.tmp"
|
||||||
|
descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||||
|
try:
|
||||||
|
with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
|
||||||
|
json.dump(payload, handle, indent=2, sort_keys=True)
|
||||||
|
handle.write("\n")
|
||||||
|
handle.flush()
|
||||||
|
os.fsync(handle.fileno())
|
||||||
|
os.replace(temporary, path)
|
||||||
|
os.chmod(path, 0o600)
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
temporary.unlink()
|
||||||
|
except FileNotFoundError:
|
||||||
|
pass
|
||||||
|
|
@ -16,10 +16,13 @@ import ipaddress
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
|
import shlex
|
||||||
import signal
|
import signal
|
||||||
import stat
|
import stat
|
||||||
import subprocess
|
import subprocess
|
||||||
import time
|
import time
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
import uuid
|
import uuid
|
||||||
from datetime import UTC, datetime
|
from datetime import UTC, datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
@ -33,17 +36,34 @@ except ImportError: # pragma: no cover - direct script execution on the GCP VM
|
||||||
|
|
||||||
DEFAULT_HOST = "10.61.0.3"
|
DEFAULT_HOST = "10.61.0.3"
|
||||||
DEFAULT_PROJECT = "teleo-501523"
|
DEFAULT_PROJECT = "teleo-501523"
|
||||||
|
DEFAULT_INSTANCE = "teleo-pgvector-standby"
|
||||||
|
DEFAULT_PRIVATE_NETWORK = "projects/teleo-501523/global/networks/teleo-staging-net"
|
||||||
|
DEFAULT_COMPUTE_INSTANCE = "teleo-prod-1"
|
||||||
|
DEFAULT_COMPUTE_ZONE = "europe-west6-a"
|
||||||
|
DEFAULT_SOURCE_SSH_TARGET = "root@77.42.65.182"
|
||||||
|
DEFAULT_SOURCE_CONTAINER = "teleo-pg"
|
||||||
|
DEFAULT_SOURCE_DATABASE = "teleo"
|
||||||
|
DEFAULT_SOURCE_SERVICE = "leoclean-gateway.service"
|
||||||
DEFAULT_SECRET = "gcp-teleo-pgvector-standby-postgres-password"
|
DEFAULT_SECRET = "gcp-teleo-pgvector-standby-postgres-password"
|
||||||
DEFAULT_SERVICE = "leoclean-gcp-prod-parallel.service"
|
DEFAULT_SERVICE = "leoclean-gcp-prod-parallel.service"
|
||||||
DEFAULT_RUN_ROOT = Path("/var/lib/teleo-gcp-restore-runs")
|
DEFAULT_RUN_ROOT = Path("/var/lib/teleo-gcp-restore-runs")
|
||||||
DEFAULT_MANIFEST_SQL = Path(__file__).with_name("postgres_parity_manifest.sql")
|
DEFAULT_MANIFEST_SQL = Path(__file__).with_name("postgres_parity_manifest.sql")
|
||||||
REVIEWED_MANIFEST_SQL_SHA256 = "8b8cdc25d54fdd8de05eb38c6e4423d2836953eb6012d4545f5c9c71b5f0150a"
|
REVIEWED_MANIFEST_SQL_SHA256 = "8b8cdc25d54fdd8de05eb38c6e4423d2836953eb6012d4545f5c9c71b5f0150a"
|
||||||
|
SOURCE_SNAPSHOT_RECEIPT_SCHEMA = "livingip.sourceSnapshotReceipt.v2"
|
||||||
TARGET_DATABASE_RE = re.compile(r"teleo_clone_[a-z0-9][a-z0-9_]{0,50}\Z")
|
TARGET_DATABASE_RE = re.compile(r"teleo_clone_[a-z0-9][a-z0-9_]{0,50}\Z")
|
||||||
RUN_ID_RE = re.compile(r"gcp-restore-[a-z0-9][a-z0-9-]{7,47}\Z")
|
RUN_ID_RE = re.compile(r"gcp-restore-[a-z0-9][a-z0-9-]{7,47}\Z")
|
||||||
SHA256_RE = re.compile(r"[0-9a-f]{64}\Z")
|
SHA256_RE = re.compile(r"[0-9a-f]{64}\Z")
|
||||||
LOCALE_RE = re.compile(r"[A-Za-z0-9_.@-]{1,64}\Z")
|
LOCALE_RE = re.compile(r"[A-Za-z0-9_.@-]{1,64}\Z")
|
||||||
|
CLOUDSQL_INSTANCE_RE = re.compile(r"[a-z][a-z0-9-]{0,97}[a-z0-9]\Z")
|
||||||
|
AUTHORIZATION_REF_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.:/@+-]{7,255}\Z")
|
||||||
|
PRIVATE_NETWORK_RE = re.compile(r"projects/[a-z][a-z0-9-]{4,28}[a-z0-9]/global/networks/[a-z][a-z0-9-]{0,61}[a-z0-9]\Z")
|
||||||
|
GCE_INSTANCE_RE = re.compile(r"[a-z](?:[a-z0-9-]{0,61}[a-z0-9])?\Z")
|
||||||
|
GCE_ZONE_RE = re.compile(r"[a-z]+-[a-z]+[0-9]-[a-z]\Z")
|
||||||
|
SAFE_NAME_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}\Z")
|
||||||
|
SAFE_SSH_TARGET_RE = re.compile(r"(?:[A-Za-z0-9][A-Za-z0-9._-]{0,63}@)?[A-Za-z0-9][A-Za-z0-9.-]{0,252}\Z")
|
||||||
PG_RESTORE_VERSION_RE = re.compile(r"pg_restore \(PostgreSQL\) (?P<major>[0-9]+)(?:\.[0-9]+)*")
|
PG_RESTORE_VERSION_RE = re.compile(r"pg_restore \(PostgreSQL\) (?P<major>[0-9]+)(?:\.[0-9]+)*")
|
||||||
ROLLBACK_DATABASE = "teleo_canonical_pre_20260712t1905z"
|
ROLLBACK_DATABASE = "teleo_canonical_pre_20260712t1905z"
|
||||||
|
GCE_METADATA_ROOT = "http://metadata.google.internal/computeMetadata/v1"
|
||||||
|
|
||||||
|
|
||||||
class RestoreError(RuntimeError):
|
class RestoreError(RuntimeError):
|
||||||
|
|
@ -66,6 +86,11 @@ def sha256_file(path: Path) -> str:
|
||||||
return digest.hexdigest()
|
return digest.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def canonical_sha256(value: Any) -> str:
|
||||||
|
payload = json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True)
|
||||||
|
return hashlib.sha256(payload.encode()).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
def validate_regular_file(path: Path, label: str) -> Path:
|
def validate_regular_file(path: Path, label: str) -> Path:
|
||||||
try:
|
try:
|
||||||
mode = path.lstat().st_mode
|
mode = path.lstat().st_mode
|
||||||
|
|
@ -89,6 +114,104 @@ def validate_custom_dump(path: Path, expected_sha256: str) -> dict[str, Any]:
|
||||||
return {"path": str(path), "bytes": path.stat().st_size, "sha256": actual_sha256}
|
return {"path": str(path), "bytes": path.stat().st_size, "sha256": actual_sha256}
|
||||||
|
|
||||||
|
|
||||||
|
def validate_source_snapshot_receipt(
|
||||||
|
path: Path,
|
||||||
|
*,
|
||||||
|
expected_receipt_sha256: str,
|
||||||
|
expected_authorization_ref: str,
|
||||||
|
expected_source: dict[str, str],
|
||||||
|
dump: dict[str, Any],
|
||||||
|
source_manifest_path: Path,
|
||||||
|
manifest_sql_path: Path,
|
||||||
|
source_manifest: dict[str, Any],
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
path = validate_regular_file(path, "source snapshot receipt")
|
||||||
|
receipt_sha256 = sha256_file(path)
|
||||||
|
if receipt_sha256 != expected_receipt_sha256:
|
||||||
|
raise RestoreError("source snapshot receipt SHA-256 does not match --expected-source-receipt-sha256")
|
||||||
|
try:
|
||||||
|
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise RestoreError("source snapshot receipt is invalid JSON") from exc
|
||||||
|
if not isinstance(payload, dict):
|
||||||
|
raise RestoreError("source snapshot receipt must be a JSON object")
|
||||||
|
|
||||||
|
snapshot = payload.get("snapshot") or {}
|
||||||
|
manifest = payload.get("manifest") or {}
|
||||||
|
source = payload.get("source") or {}
|
||||||
|
source_context = payload.get("source_context") or {}
|
||||||
|
source_service = payload.get("source_service") or {}
|
||||||
|
binding = payload.get("provenance_binding") or {}
|
||||||
|
binding_payload = binding.get("payload") or {}
|
||||||
|
source_manifest_sha256 = sha256_file(source_manifest_path)
|
||||||
|
manifest_sql_sha256 = sha256_file(manifest_sql_path)
|
||||||
|
expected_table_count = len(source_manifest["tables"])
|
||||||
|
expected_total_rows = sum(int(row["row_count"]) for row in source_manifest["tables"].values())
|
||||||
|
expected_binding_payload = {
|
||||||
|
"receipt_schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
|
||||||
|
"run_id": payload.get("run_id"),
|
||||||
|
"capture_authorization_ref": payload.get("capture_authorization_ref"),
|
||||||
|
"source": source,
|
||||||
|
"exported_snapshot_id": snapshot.get("exported_snapshot_id"),
|
||||||
|
"txid_snapshot": snapshot.get("txid_snapshot"),
|
||||||
|
"wal_lsn": snapshot.get("wal_lsn"),
|
||||||
|
"source_system_identifier": snapshot.get("system_identifier"),
|
||||||
|
"dump_sha256": dump["sha256"],
|
||||||
|
"manifest_sql_sha256": manifest_sql_sha256,
|
||||||
|
"source_manifest_sha256": source_manifest_sha256,
|
||||||
|
"source_context_sha256": source_context.get("sha256"),
|
||||||
|
}
|
||||||
|
checks = {
|
||||||
|
"artifact": payload.get("artifact") == "vps_canonical_postgres_exported_snapshot",
|
||||||
|
"schema": payload.get("schema") == SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
|
||||||
|
"receipt_version": payload.get("receipt_version") == 2,
|
||||||
|
"status": payload.get("status") == "pass",
|
||||||
|
"snapshot_exported": payload.get("snapshot_exported") is True,
|
||||||
|
"service_unchanged": payload.get("service_unchanged") is True,
|
||||||
|
"source_service_healthy": source_service.get("unchanged") is True
|
||||||
|
and service_state_is_healthy(source_service.get("before"))
|
||||||
|
and source_service.get("before") == source_service.get("after"),
|
||||||
|
"source_not_mutated": payload.get("production_db_mutated") is False,
|
||||||
|
"telegram_not_sent": payload.get("telegram_send_attempted") is False,
|
||||||
|
"run_id": bool(payload.get("run_id")),
|
||||||
|
"capture_authorization_ref": payload.get("capture_authorization_ref") == expected_authorization_ref,
|
||||||
|
"source_identity": source == expected_source,
|
||||||
|
"source_database": source.get("database") == source_manifest["singleton"]["identity"].get("database"),
|
||||||
|
"snapshot_identity": all(
|
||||||
|
bool(snapshot.get(field))
|
||||||
|
for field in ("exported_snapshot_id", "txid_snapshot", "wal_lsn", "system_identifier", "captured_at_utc")
|
||||||
|
),
|
||||||
|
"dump_sha256": (payload.get("dump") or {}).get("sha256") == dump["sha256"],
|
||||||
|
"dump_bytes": (payload.get("dump") or {}).get("bytes") == dump["bytes"],
|
||||||
|
"source_manifest_sha256": manifest.get("sha256") == source_manifest_sha256,
|
||||||
|
"manifest_sql_sha256": manifest.get("manifest_sql_sha256") == manifest_sql_sha256,
|
||||||
|
"source_context_sha256": bool(SHA256_RE.fullmatch(str(source_context.get("sha256") or ""))),
|
||||||
|
"table_count": manifest.get("table_count") == expected_table_count,
|
||||||
|
"total_rows": manifest.get("total_rows") == expected_total_rows,
|
||||||
|
"binding_algorithm": binding.get("algorithm") == "sha256",
|
||||||
|
"binding_payload": binding_payload == expected_binding_payload,
|
||||||
|
"binding_sha256": binding.get("sha256") == canonical_sha256(expected_binding_payload),
|
||||||
|
}
|
||||||
|
failed = sorted(name for name, passed in checks.items() if not passed)
|
||||||
|
if failed:
|
||||||
|
raise RestoreError("source snapshot receipt failed validation: " + ", ".join(failed))
|
||||||
|
return {
|
||||||
|
"path": str(path),
|
||||||
|
"sha256": receipt_sha256,
|
||||||
|
"schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
|
||||||
|
"run_id": payload["run_id"],
|
||||||
|
"capture_authorization_ref": payload["capture_authorization_ref"],
|
||||||
|
"snapshot": snapshot,
|
||||||
|
"provenance_binding_sha256": binding["sha256"],
|
||||||
|
"manifest_sha256": source_manifest_sha256,
|
||||||
|
"manifest_sql_sha256": manifest_sql_sha256,
|
||||||
|
"dump_sha256": dump["sha256"],
|
||||||
|
"table_count": expected_table_count,
|
||||||
|
"total_rows": expected_total_rows,
|
||||||
|
"checks": checks,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def _run(
|
def _run(
|
||||||
command: list[str],
|
command: list[str],
|
||||||
*,
|
*,
|
||||||
|
|
@ -158,6 +281,163 @@ def service_state(service: str, *, timeout: float) -> dict[str, Any]:
|
||||||
return result
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def validate_service_healthy(state: dict[str, Any], label: str) -> None:
|
||||||
|
if not service_state_is_healthy(state):
|
||||||
|
raise RestoreError(f"{label} service is not active/running with a positive MainPID")
|
||||||
|
|
||||||
|
|
||||||
|
def service_state_is_healthy(state: Any) -> bool:
|
||||||
|
if not isinstance(state, dict):
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
main_pid = int(state.get("MainPID") or 0)
|
||||||
|
restarts = int(state.get("NRestarts") or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return False
|
||||||
|
return (
|
||||||
|
state.get("ActiveState") == "active" and state.get("SubState") == "running" and main_pid > 0 and restarts >= 0
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def cloudsql_control_plane_state(
|
||||||
|
project: str,
|
||||||
|
instance: str,
|
||||||
|
expected_host: str,
|
||||||
|
expected_private_network: str,
|
||||||
|
*,
|
||||||
|
timeout: float,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
completed = _run(
|
||||||
|
[
|
||||||
|
"gcloud",
|
||||||
|
"sql",
|
||||||
|
"instances",
|
||||||
|
"describe",
|
||||||
|
instance,
|
||||||
|
f"--project={project}",
|
||||||
|
"--format=json",
|
||||||
|
"--quiet",
|
||||||
|
],
|
||||||
|
timeout=timeout,
|
||||||
|
)
|
||||||
|
raw = _require_success(completed, "Cloud SQL control-plane readback")
|
||||||
|
try:
|
||||||
|
payload = json.loads(raw)
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise RestoreError("Cloud SQL control-plane readback returned invalid JSON") from exc
|
||||||
|
settings = payload.get("settings") or {}
|
||||||
|
ip_configuration = settings.get("ipConfiguration") or {}
|
||||||
|
addresses = payload.get("ipAddresses") or []
|
||||||
|
private_addresses = sorted(
|
||||||
|
str(row.get("ipAddress")) for row in addresses if row.get("type") == "PRIVATE" and row.get("ipAddress")
|
||||||
|
)
|
||||||
|
private_network = str(ip_configuration.get("privateNetwork") or "")
|
||||||
|
non_private_addresses = sorted(
|
||||||
|
str(row.get("ipAddress")) for row in addresses if row.get("type") != "PRIVATE" and row.get("ipAddress")
|
||||||
|
)
|
||||||
|
checks = {
|
||||||
|
"instance": payload.get("name") == instance,
|
||||||
|
"state": payload.get("state") == "RUNNABLE",
|
||||||
|
"postgres_16": str(payload.get("databaseVersion") or "").startswith("POSTGRES_16"),
|
||||||
|
"public_ip_disabled": ip_configuration.get("ipv4Enabled") is False,
|
||||||
|
"private_network": private_network == expected_private_network,
|
||||||
|
"expected_private_host": expected_host in private_addresses,
|
||||||
|
"no_non_private_address": not non_private_addresses,
|
||||||
|
}
|
||||||
|
failed = sorted(name for name, passed in checks.items() if not passed)
|
||||||
|
if failed:
|
||||||
|
raise RestoreError("Cloud SQL control-plane preflight failed: " + ", ".join(failed))
|
||||||
|
return {
|
||||||
|
"project": project,
|
||||||
|
"instance": instance,
|
||||||
|
"database_version": payload.get("databaseVersion"),
|
||||||
|
"region": payload.get("region"),
|
||||||
|
"state": payload.get("state"),
|
||||||
|
"tier": settings.get("tier"),
|
||||||
|
"private_network": private_network,
|
||||||
|
"expected_private_network": expected_private_network,
|
||||||
|
"private_addresses": private_addresses,
|
||||||
|
"expected_private_host": expected_host,
|
||||||
|
"public_ip_disabled": True,
|
||||||
|
"checks": checks,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def gce_metadata_value(path: str, *, timeout: float) -> str:
|
||||||
|
request = urllib.request.Request(
|
||||||
|
f"{GCE_METADATA_ROOT}/{path}",
|
||||||
|
headers={"Metadata-Flavor": "Google"},
|
||||||
|
method="GET",
|
||||||
|
)
|
||||||
|
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
|
||||||
|
try:
|
||||||
|
with opener.open(request, timeout=timeout) as response:
|
||||||
|
if response.headers.get("Metadata-Flavor") != "Google":
|
||||||
|
raise RestoreError("GCE metadata response is missing Metadata-Flavor: Google")
|
||||||
|
value = response.read(4096).decode("utf-8", errors="strict").strip()
|
||||||
|
except (OSError, UnicodeError, urllib.error.URLError) as exc:
|
||||||
|
raise RestoreError(f"GCE metadata readback failed for {path}") from exc
|
||||||
|
if not value or any(character in value for character in "\r\n\x00"):
|
||||||
|
raise RestoreError(f"GCE metadata readback is invalid for {path}")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def gce_compute_identity(
|
||||||
|
project: str,
|
||||||
|
expected_instance: str,
|
||||||
|
expected_zone: str,
|
||||||
|
expected_private_network: str,
|
||||||
|
*,
|
||||||
|
timeout: float,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
metadata = {
|
||||||
|
"project": gce_metadata_value("project/project-id", timeout=timeout),
|
||||||
|
"project_number": gce_metadata_value("project/numeric-project-id", timeout=timeout),
|
||||||
|
"instance": gce_metadata_value("instance/name", timeout=timeout),
|
||||||
|
"instance_id": gce_metadata_value("instance/id", timeout=timeout),
|
||||||
|
"zone_resource": gce_metadata_value("instance/zone", timeout=timeout),
|
||||||
|
"network_resource": gce_metadata_value("instance/network-interfaces/0/network", timeout=timeout),
|
||||||
|
"private_ip": gce_metadata_value("instance/network-interfaces/0/ip", timeout=timeout),
|
||||||
|
"service_account": gce_metadata_value("instance/service-accounts/default/email", timeout=timeout),
|
||||||
|
}
|
||||||
|
zone = metadata["zone_resource"].rsplit("/", 1)[-1]
|
||||||
|
network_name = expected_private_network.rsplit("/", 1)[-1]
|
||||||
|
expected_network_resource = f"projects/{metadata['project_number']}/networks/{network_name}"
|
||||||
|
try:
|
||||||
|
private_ip = ipaddress.ip_address(metadata["private_ip"])
|
||||||
|
except ValueError as exc:
|
||||||
|
raise RestoreError("GCE metadata private IP is invalid") from exc
|
||||||
|
checks = {
|
||||||
|
"project": metadata["project"] == project,
|
||||||
|
"project_number": metadata["project_number"].isdigit(),
|
||||||
|
"expected_network_project": expected_private_network.startswith(f"projects/{project}/global/networks/"),
|
||||||
|
"instance": metadata["instance"] == expected_instance,
|
||||||
|
"instance_id": metadata["instance_id"].isdigit(),
|
||||||
|
"zone": zone == expected_zone,
|
||||||
|
"network": metadata["network_resource"] == expected_network_resource,
|
||||||
|
"private_ip": any(
|
||||||
|
private_ip in network
|
||||||
|
for network in (
|
||||||
|
ipaddress.ip_network("10.0.0.0/8"),
|
||||||
|
ipaddress.ip_network("172.16.0.0/12"),
|
||||||
|
ipaddress.ip_network("192.168.0.0/16"),
|
||||||
|
)
|
||||||
|
),
|
||||||
|
"service_account": metadata["service_account"].endswith(".iam.gserviceaccount.com"),
|
||||||
|
}
|
||||||
|
failed = sorted(name for name, passed in checks.items() if not passed)
|
||||||
|
if failed:
|
||||||
|
raise RestoreError("GCE compute identity preflight failed: " + ", ".join(failed))
|
||||||
|
return {
|
||||||
|
**metadata,
|
||||||
|
"zone": zone,
|
||||||
|
"expected_zone": expected_zone,
|
||||||
|
"expected_private_network": expected_private_network,
|
||||||
|
"expected_network_resource": expected_network_resource,
|
||||||
|
"checks": checks,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def resolve_password(project: str, secret: str, *, timeout: float) -> str:
|
def resolve_password(project: str, secret: str, *, timeout: float) -> str:
|
||||||
completed = _run(
|
completed = _run(
|
||||||
[
|
[
|
||||||
|
|
@ -310,7 +590,11 @@ def capture_target_manifest(args: argparse.Namespace, password: str) -> str:
|
||||||
return _require_success(completed, "target parity manifest capture", secrets=(password,))
|
return _require_success(completed, "target parity manifest capture", secrets=(password,))
|
||||||
|
|
||||||
|
|
||||||
def validate_private_identity(target_manifest: dict[str, Any], target_db: str) -> dict[str, Any]:
|
def validate_private_identity(
|
||||||
|
target_manifest: dict[str, Any],
|
||||||
|
target_db: str,
|
||||||
|
source_compute: str,
|
||||||
|
) -> dict[str, Any]:
|
||||||
identity = target_manifest["singleton"]["identity"]
|
identity = target_manifest["singleton"]["identity"]
|
||||||
if identity.get("database") != target_db:
|
if identity.get("database") != target_db:
|
||||||
raise RestoreError("target manifest database does not match the generated clone")
|
raise RestoreError("target manifest database does not match the generated clone")
|
||||||
|
|
@ -340,7 +624,7 @@ def validate_private_identity(target_manifest: dict[str, Any], target_db: str) -
|
||||||
"ssl_version": identity.get("ssl_version"),
|
"ssl_version": identity.get("ssl_version"),
|
||||||
"private_connectivity": True,
|
"private_connectivity": True,
|
||||||
"transaction_read_only": "on",
|
"transaction_read_only": "on",
|
||||||
"source_compute": "teleo-prod-1",
|
"source_compute": source_compute,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -440,6 +724,22 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]:
|
||||||
"generated_at_utc": utc_now(),
|
"generated_at_utc": utc_now(),
|
||||||
"run_id": args.run_id,
|
"run_id": args.run_id,
|
||||||
"target_database": args.target_db,
|
"target_database": args.target_db,
|
||||||
|
"execution_contract": {
|
||||||
|
"project": args.project,
|
||||||
|
"cloudsql_instance": args.cloudsql_instance,
|
||||||
|
"host": args.host,
|
||||||
|
"private_network": args.expected_private_network,
|
||||||
|
"compute_instance": args.expected_compute_instance,
|
||||||
|
"compute_zone": args.expected_compute_zone,
|
||||||
|
"source": {
|
||||||
|
"ssh_target": args.expected_source_ssh_target,
|
||||||
|
"container": args.expected_source_container,
|
||||||
|
"database": args.expected_source_database,
|
||||||
|
"service": args.expected_source_service,
|
||||||
|
},
|
||||||
|
"password_secret": args.password_secret,
|
||||||
|
"service": args.service,
|
||||||
|
},
|
||||||
"status": "running",
|
"status": "running",
|
||||||
"phase": "validation",
|
"phase": "validation",
|
||||||
"source": {},
|
"source": {},
|
||||||
|
|
@ -475,12 +775,39 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]:
|
||||||
"sha256": sha256_file(args.source_manifest),
|
"sha256": sha256_file(args.source_manifest),
|
||||||
}
|
}
|
||||||
source_manifest = load_manifest(args.source_manifest)
|
source_manifest = load_manifest(args.source_manifest)
|
||||||
|
receipt["source"]["capture_receipt"] = validate_source_snapshot_receipt(
|
||||||
|
args.source_receipt,
|
||||||
|
expected_receipt_sha256=args.expected_source_receipt_sha256,
|
||||||
|
expected_authorization_ref=args.expected_capture_authorization_ref,
|
||||||
|
expected_source=receipt["execution_contract"]["source"],
|
||||||
|
dump=receipt["source"]["dump"],
|
||||||
|
source_manifest_path=args.source_manifest,
|
||||||
|
manifest_sql_path=args.manifest_sql,
|
||||||
|
source_manifest=source_manifest,
|
||||||
|
)
|
||||||
receipt["toolchain"] = validate_pg_restore_version(
|
receipt["toolchain"] = validate_pg_restore_version(
|
||||||
args.pg_restore_bin,
|
args.pg_restore_bin,
|
||||||
source_manifest,
|
source_manifest,
|
||||||
timeout=args.command_timeout,
|
timeout=args.command_timeout,
|
||||||
)
|
)
|
||||||
|
receipt["target"]["compute"] = gce_compute_identity(
|
||||||
|
args.project,
|
||||||
|
args.expected_compute_instance,
|
||||||
|
args.expected_compute_zone,
|
||||||
|
args.expected_private_network,
|
||||||
|
timeout=args.command_timeout,
|
||||||
|
)
|
||||||
receipt["live_service"]["before"] = service_state(args.service, timeout=args.command_timeout)
|
receipt["live_service"]["before"] = service_state(args.service, timeout=args.command_timeout)
|
||||||
|
validate_service_healthy(receipt["live_service"]["before"], "pre-restore")
|
||||||
|
|
||||||
|
receipt["phase"] = "cloudsql_control_plane_preflight"
|
||||||
|
receipt["target"]["cloudsql"] = cloudsql_control_plane_state(
|
||||||
|
args.project,
|
||||||
|
args.cloudsql_instance,
|
||||||
|
args.host,
|
||||||
|
args.expected_private_network,
|
||||||
|
timeout=args.command_timeout,
|
||||||
|
)
|
||||||
|
|
||||||
receipt["phase"] = "credential_resolution"
|
receipt["phase"] = "credential_resolution"
|
||||||
password = resolve_password(args.project, args.password_secret, timeout=args.command_timeout)
|
password = resolve_password(args.project, args.password_secret, timeout=args.command_timeout)
|
||||||
|
|
@ -506,9 +833,46 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]:
|
||||||
"manifest_path": str(target_manifest_path),
|
"manifest_path": str(target_manifest_path),
|
||||||
"manifest_bytes": target_manifest_path.stat().st_size,
|
"manifest_bytes": target_manifest_path.stat().st_size,
|
||||||
"manifest_sha256": sha256_file(target_manifest_path),
|
"manifest_sha256": sha256_file(target_manifest_path),
|
||||||
"connectivity": validate_private_identity(target_manifest, args.target_db),
|
"connectivity": validate_private_identity(
|
||||||
|
target_manifest,
|
||||||
|
args.target_db,
|
||||||
|
receipt["target"]["compute"]["instance"],
|
||||||
|
),
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
|
connectivity = receipt["target"]["connectivity"]
|
||||||
|
cloudsql = receipt["target"]["cloudsql"]
|
||||||
|
compute = receipt["target"]["compute"]
|
||||||
|
connectivity_proof = {
|
||||||
|
"artifact": "gcp_private_postgres_connectivity",
|
||||||
|
"schema": "livingip.gcpPrivatePostgresConnectivity.v2",
|
||||||
|
"generated_at_utc": utc_now(),
|
||||||
|
"status": "pass",
|
||||||
|
"restore_run_id": args.run_id,
|
||||||
|
"target_database": args.target_db,
|
||||||
|
"project": cloudsql["project"],
|
||||||
|
"cloudsql_instance": cloudsql["instance"],
|
||||||
|
"private_network": cloudsql["private_network"],
|
||||||
|
"source_compute": compute["instance"],
|
||||||
|
"source_compute_instance_id": compute["instance_id"],
|
||||||
|
"source_compute_zone": compute["zone"],
|
||||||
|
"source_compute_private_ip": compute["private_ip"],
|
||||||
|
"server_address": connectivity["server_address"],
|
||||||
|
"server_port": connectivity["server_port"],
|
||||||
|
"ssl": connectivity["ssl"],
|
||||||
|
"ssl_version": connectivity["ssl_version"],
|
||||||
|
"private_connectivity": connectivity["private_connectivity"],
|
||||||
|
"public_ip_disabled": cloudsql["public_ip_disabled"],
|
||||||
|
"cloudsql_control_plane_checks": cloudsql["checks"],
|
||||||
|
"compute_identity_checks": compute["checks"],
|
||||||
|
}
|
||||||
|
connectivity_proof_path = run_dir / "gcp-private-connectivity.json"
|
||||||
|
write_private(connectivity_proof_path, json.dumps(connectivity_proof, indent=2, sort_keys=True) + "\n")
|
||||||
|
receipt["target"]["connectivity_proof"] = {
|
||||||
|
"path": str(connectivity_proof_path),
|
||||||
|
"sha256": sha256_file(connectivity_proof_path),
|
||||||
|
"schema": connectivity_proof["schema"],
|
||||||
|
}
|
||||||
|
|
||||||
receipt["phase"] = "parity_compare"
|
receipt["phase"] = "parity_compare"
|
||||||
problems, details = compare_manifests(
|
problems, details = compare_manifests(
|
||||||
|
|
@ -529,6 +893,7 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]:
|
||||||
receipt["phase"] = "service_and_rollback_readback"
|
receipt["phase"] = "service_and_rollback_readback"
|
||||||
receipt["rollback_database"] = rollback_database_state(args, password)
|
receipt["rollback_database"] = rollback_database_state(args, password)
|
||||||
receipt["live_service"]["after"] = service_state(args.service, timeout=args.command_timeout)
|
receipt["live_service"]["after"] = service_state(args.service, timeout=args.command_timeout)
|
||||||
|
validate_service_healthy(receipt["live_service"]["after"], "post-restore")
|
||||||
receipt["live_service"]["unchanged"] = receipt["live_service"]["before"] == receipt["live_service"]["after"]
|
receipt["live_service"]["unchanged"] = receipt["live_service"]["before"] == receipt["live_service"]["after"]
|
||||||
if not receipt["live_service"]["unchanged"]:
|
if not receipt["live_service"]["unchanged"]:
|
||||||
raise RestoreError("live GCP service state changed during disposable restore")
|
raise RestoreError("live GCP service state changed during disposable restore")
|
||||||
|
|
@ -539,9 +904,42 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]:
|
||||||
"required": True,
|
"required": True,
|
||||||
"attempted": False,
|
"attempted": False,
|
||||||
"clone_database_remaining": 1,
|
"clone_database_remaining": 1,
|
||||||
"exact_command": (
|
"exact_command": shlex.join(
|
||||||
f"sudo python3 ops/restore_gcp_generated_postgres_snapshot.py cleanup --execute "
|
[
|
||||||
f"--target-db {args.target_db} --run-id {args.run_id}"
|
"sudo",
|
||||||
|
"python3",
|
||||||
|
"ops/restore_gcp_generated_postgres_snapshot.py",
|
||||||
|
"cleanup",
|
||||||
|
"--execute",
|
||||||
|
"--target-db",
|
||||||
|
args.target_db,
|
||||||
|
"--run-id",
|
||||||
|
args.run_id,
|
||||||
|
"--host",
|
||||||
|
args.host,
|
||||||
|
"--project",
|
||||||
|
args.project,
|
||||||
|
"--cloudsql-instance",
|
||||||
|
args.cloudsql_instance,
|
||||||
|
"--expected-private-network",
|
||||||
|
args.expected_private_network,
|
||||||
|
"--expected-compute-instance",
|
||||||
|
args.expected_compute_instance,
|
||||||
|
"--expected-compute-zone",
|
||||||
|
args.expected_compute_zone,
|
||||||
|
"--expected-source-ssh-target",
|
||||||
|
args.expected_source_ssh_target,
|
||||||
|
"--expected-source-container",
|
||||||
|
args.expected_source_container,
|
||||||
|
"--expected-source-database",
|
||||||
|
args.expected_source_database,
|
||||||
|
"--expected-source-service",
|
||||||
|
args.expected_source_service,
|
||||||
|
"--password-secret",
|
||||||
|
args.password_secret,
|
||||||
|
"--service",
|
||||||
|
args.service,
|
||||||
|
]
|
||||||
),
|
),
|
||||||
}
|
}
|
||||||
except (
|
except (
|
||||||
|
|
@ -583,6 +981,7 @@ def run_restore(args: argparse.Namespace) -> dict[str, Any]:
|
||||||
except Exception as service_exc:
|
except Exception as service_exc:
|
||||||
receipt["live_service"]["after_error"] = str(service_exc)[-2000:]
|
receipt["live_service"]["after_error"] = str(service_exc)[-2000:]
|
||||||
receipt["duration_seconds"] = round(time.monotonic() - started, 6)
|
receipt["duration_seconds"] = round(time.monotonic() - started, 6)
|
||||||
|
receipt["completed_at_utc"] = utc_now()
|
||||||
write_private(run_dir / "restore-receipt.json", json.dumps(receipt, indent=2, sort_keys=True) + "\n")
|
write_private(run_dir / "restore-receipt.json", json.dumps(receipt, indent=2, sort_keys=True) + "\n")
|
||||||
password = ""
|
password = ""
|
||||||
return receipt
|
return receipt
|
||||||
|
|
@ -592,6 +991,7 @@ def run_cleanup(args: argparse.Namespace) -> dict[str, Any]:
|
||||||
run_dir = secure_run_dir(args.run_root, args.run_id, create=False)
|
run_dir = secure_run_dir(args.run_root, args.run_id, create=False)
|
||||||
restore_receipt_path = run_dir / "restore-receipt.json"
|
restore_receipt_path = run_dir / "restore-receipt.json"
|
||||||
restore_receipt_path = validate_regular_file(restore_receipt_path, "restore receipt")
|
restore_receipt_path = validate_regular_file(restore_receipt_path, "restore receipt")
|
||||||
|
restore_receipt_sha256 = sha256_file(restore_receipt_path)
|
||||||
restore_receipt = json.loads(restore_receipt_path.read_text(encoding="utf-8"))
|
restore_receipt = json.loads(restore_receipt_path.read_text(encoding="utf-8"))
|
||||||
if restore_receipt.get("status") != "pass":
|
if restore_receipt.get("status") != "pass":
|
||||||
raise RestoreError("cleanup requires a passing restore receipt")
|
raise RestoreError("cleanup requires a passing restore receipt")
|
||||||
|
|
@ -600,23 +1000,66 @@ def run_cleanup(args: argparse.Namespace) -> dict[str, Any]:
|
||||||
if restore_receipt.get("run_id") != args.run_id:
|
if restore_receipt.get("run_id") != args.run_id:
|
||||||
raise RestoreError("cleanup run id does not match the retained restore receipt")
|
raise RestoreError("cleanup run id does not match the retained restore receipt")
|
||||||
|
|
||||||
before = service_state(args.service, timeout=args.command_timeout)
|
expected_contract = {
|
||||||
password = resolve_password(args.project, args.password_secret, timeout=args.command_timeout)
|
"project": args.project,
|
||||||
try:
|
"cloudsql_instance": args.cloudsql_instance,
|
||||||
dropped = drop_clone(args, password)
|
"host": args.host,
|
||||||
rollback = rollback_database_state(args, password)
|
"private_network": args.expected_private_network,
|
||||||
finally:
|
"compute_instance": args.expected_compute_instance,
|
||||||
password = ""
|
"compute_zone": args.expected_compute_zone,
|
||||||
after = service_state(args.service, timeout=args.command_timeout)
|
"source": {
|
||||||
payload = {
|
"ssh_target": args.expected_source_ssh_target,
|
||||||
|
"container": args.expected_source_container,
|
||||||
|
"database": args.expected_source_database,
|
||||||
|
"service": args.expected_source_service,
|
||||||
|
},
|
||||||
|
"password_secret": args.password_secret,
|
||||||
|
"service": args.service,
|
||||||
|
}
|
||||||
|
if restore_receipt.get("execution_contract") != expected_contract:
|
||||||
|
raise RestoreError("cleanup execution contract does not match the retained restore receipt")
|
||||||
|
restore_cloudsql = (restore_receipt.get("target") or {}).get("cloudsql") or {}
|
||||||
|
restore_topology_checks = {
|
||||||
|
"project": restore_cloudsql.get("project") == args.project,
|
||||||
|
"instance": restore_cloudsql.get("instance") == args.cloudsql_instance,
|
||||||
|
"host": restore_cloudsql.get("expected_private_host") == args.host,
|
||||||
|
"private_network": restore_cloudsql.get("expected_private_network") == args.expected_private_network,
|
||||||
|
"public_ip_disabled": restore_cloudsql.get("public_ip_disabled") is True,
|
||||||
|
}
|
||||||
|
failed_topology_checks = sorted(name for name, passed in restore_topology_checks.items() if not passed)
|
||||||
|
if failed_topology_checks:
|
||||||
|
raise RestoreError(
|
||||||
|
"cleanup target topology does not match the retained restore receipt: " + ", ".join(failed_topology_checks)
|
||||||
|
)
|
||||||
|
restore_compute = (restore_receipt.get("target") or {}).get("compute") or {}
|
||||||
|
restore_compute_checks = {
|
||||||
|
"project": restore_compute.get("project") == args.project,
|
||||||
|
"instance": restore_compute.get("instance") == args.expected_compute_instance,
|
||||||
|
"zone": restore_compute.get("zone") == args.expected_compute_zone,
|
||||||
|
"private_network": restore_compute.get("expected_private_network") == args.expected_private_network,
|
||||||
|
"checks": bool(restore_compute.get("checks"))
|
||||||
|
and all(value is True for value in restore_compute["checks"].values()),
|
||||||
|
}
|
||||||
|
failed_compute_checks = sorted(name for name, passed in restore_compute_checks.items() if not passed)
|
||||||
|
if failed_compute_checks:
|
||||||
|
raise RestoreError(
|
||||||
|
"cleanup compute identity does not match the retained restore receipt: " + ", ".join(failed_compute_checks)
|
||||||
|
)
|
||||||
|
|
||||||
|
started = time.monotonic()
|
||||||
|
payload: dict[str, Any] = {
|
||||||
"artifact": "gcp_generated_postgres_snapshot_cleanup",
|
"artifact": "gcp_generated_postgres_snapshot_cleanup",
|
||||||
"generated_at_utc": utc_now(),
|
"generated_at_utc": utc_now(),
|
||||||
"run_id": args.run_id,
|
"run_id": args.run_id,
|
||||||
"target_database": args.target_db,
|
"target_database": args.target_db,
|
||||||
"status": "pass" if before == after and dropped["clone_database_remaining"] == 0 else "fail",
|
"restore_receipt": {"path": str(restore_receipt_path), "sha256": restore_receipt_sha256},
|
||||||
"database": dropped,
|
"execution_contract": expected_contract,
|
||||||
"rollback_database": rollback,
|
"target": {},
|
||||||
"live_service": {"before": before, "after": after, "unchanged": before == after},
|
"status": "running",
|
||||||
|
"phase": "cloudsql_control_plane_preflight",
|
||||||
|
"database": {"clone_database_remaining": None},
|
||||||
|
"rollback_database": {},
|
||||||
|
"live_service": {},
|
||||||
"safety": {
|
"safety": {
|
||||||
"live_database_named": False,
|
"live_database_named": False,
|
||||||
"live_profile_modified": False,
|
"live_profile_modified": False,
|
||||||
|
|
@ -624,8 +1067,95 @@ def run_cleanup(args: argparse.Namespace) -> dict[str, Any]:
|
||||||
"telegram_message_sent": False,
|
"telegram_message_sent": False,
|
||||||
"secret_persisted": False,
|
"secret_persisted": False,
|
||||||
},
|
},
|
||||||
|
"error": None,
|
||||||
}
|
}
|
||||||
write_private(run_dir / "cleanup-receipt.json", json.dumps(payload, indent=2, sort_keys=True) + "\n")
|
password = ""
|
||||||
|
try:
|
||||||
|
payload["target"]["cloudsql"] = cloudsql_control_plane_state(
|
||||||
|
args.project,
|
||||||
|
args.cloudsql_instance,
|
||||||
|
args.host,
|
||||||
|
args.expected_private_network,
|
||||||
|
timeout=args.command_timeout,
|
||||||
|
)
|
||||||
|
payload["target"]["compute"] = gce_compute_identity(
|
||||||
|
args.project,
|
||||||
|
args.expected_compute_instance,
|
||||||
|
args.expected_compute_zone,
|
||||||
|
args.expected_private_network,
|
||||||
|
timeout=args.command_timeout,
|
||||||
|
)
|
||||||
|
cleanup_compute_mismatches = sorted(
|
||||||
|
field
|
||||||
|
for field in (
|
||||||
|
"project",
|
||||||
|
"project_number",
|
||||||
|
"instance",
|
||||||
|
"instance_id",
|
||||||
|
"zone",
|
||||||
|
"network_resource",
|
||||||
|
"private_ip",
|
||||||
|
"service_account",
|
||||||
|
)
|
||||||
|
if payload["target"]["compute"].get(field) != restore_compute.get(field)
|
||||||
|
)
|
||||||
|
if cleanup_compute_mismatches:
|
||||||
|
raise RestoreError("cleanup GCE identity changed since restore: " + ", ".join(cleanup_compute_mismatches))
|
||||||
|
payload["phase"] = "service_readback_before"
|
||||||
|
payload["live_service"]["before"] = service_state(args.service, timeout=args.command_timeout)
|
||||||
|
validate_service_healthy(payload["live_service"]["before"], "pre-cleanup")
|
||||||
|
payload["phase"] = "credential_resolution"
|
||||||
|
password = resolve_password(args.project, args.password_secret, timeout=args.command_timeout)
|
||||||
|
payload["phase"] = "database_drop"
|
||||||
|
payload["database"] = drop_clone(args, password)
|
||||||
|
payload["phase"] = "rollback_readback"
|
||||||
|
payload["rollback_database"] = rollback_database_state(args, password)
|
||||||
|
payload["phase"] = "service_readback_after"
|
||||||
|
payload["live_service"]["after"] = service_state(args.service, timeout=args.command_timeout)
|
||||||
|
validate_service_healthy(payload["live_service"]["after"], "post-cleanup")
|
||||||
|
payload["live_service"]["unchanged"] = payload["live_service"]["before"] == payload["live_service"]["after"]
|
||||||
|
payload["status"] = (
|
||||||
|
"pass"
|
||||||
|
if payload["live_service"]["unchanged"] and payload["database"]["clone_database_remaining"] == 0
|
||||||
|
else "fail"
|
||||||
|
)
|
||||||
|
payload["phase"] = "complete"
|
||||||
|
except (
|
||||||
|
OSError,
|
||||||
|
ValueError,
|
||||||
|
KeyError,
|
||||||
|
TypeError,
|
||||||
|
json.JSONDecodeError,
|
||||||
|
subprocess.TimeoutExpired,
|
||||||
|
RestoreError,
|
||||||
|
KeyboardInterrupt,
|
||||||
|
) as exc:
|
||||||
|
payload["status"] = "fail"
|
||||||
|
payload["error"] = {
|
||||||
|
"phase": payload.get("phase"),
|
||||||
|
"type": type(exc).__name__,
|
||||||
|
"message": str(exc)[-2000:],
|
||||||
|
}
|
||||||
|
if password and payload["database"].get("clone_database_remaining") is None:
|
||||||
|
try:
|
||||||
|
payload["database"]["clone_database_remaining"] = (
|
||||||
|
1 if database_exists(args, password, args.target_db) else 0
|
||||||
|
)
|
||||||
|
except Exception as readback_exc:
|
||||||
|
payload["database"]["readback_error"] = str(readback_exc)[-2000:]
|
||||||
|
if "before" in payload["live_service"] and "after" not in payload["live_service"]:
|
||||||
|
try:
|
||||||
|
payload["live_service"]["after"] = service_state(args.service, timeout=args.command_timeout)
|
||||||
|
payload["live_service"]["unchanged"] = (
|
||||||
|
payload["live_service"]["before"] == payload["live_service"]["after"]
|
||||||
|
)
|
||||||
|
except Exception as service_exc:
|
||||||
|
payload["live_service"]["after_error"] = str(service_exc)[-2000:]
|
||||||
|
finally:
|
||||||
|
password = ""
|
||||||
|
payload["duration_seconds"] = round(time.monotonic() - started, 6)
|
||||||
|
payload["completed_at_utc"] = utc_now()
|
||||||
|
write_private(run_dir / "cleanup-receipt.json", json.dumps(payload, indent=2, sort_keys=True) + "\n")
|
||||||
return payload
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -635,6 +1165,14 @@ def add_common_args(parser: argparse.ArgumentParser) -> None:
|
||||||
parser.add_argument("--run-id", required=True)
|
parser.add_argument("--run-id", required=True)
|
||||||
parser.add_argument("--host", default=DEFAULT_HOST)
|
parser.add_argument("--host", default=DEFAULT_HOST)
|
||||||
parser.add_argument("--project", default=DEFAULT_PROJECT)
|
parser.add_argument("--project", default=DEFAULT_PROJECT)
|
||||||
|
parser.add_argument("--cloudsql-instance", default=DEFAULT_INSTANCE)
|
||||||
|
parser.add_argument("--expected-private-network", default=DEFAULT_PRIVATE_NETWORK)
|
||||||
|
parser.add_argument("--expected-compute-instance", default=DEFAULT_COMPUTE_INSTANCE)
|
||||||
|
parser.add_argument("--expected-compute-zone", default=DEFAULT_COMPUTE_ZONE)
|
||||||
|
parser.add_argument("--expected-source-ssh-target", default=DEFAULT_SOURCE_SSH_TARGET)
|
||||||
|
parser.add_argument("--expected-source-container", default=DEFAULT_SOURCE_CONTAINER)
|
||||||
|
parser.add_argument("--expected-source-database", default=DEFAULT_SOURCE_DATABASE)
|
||||||
|
parser.add_argument("--expected-source-service", default=DEFAULT_SOURCE_SERVICE)
|
||||||
parser.add_argument("--password-secret", default=DEFAULT_SECRET)
|
parser.add_argument("--password-secret", default=DEFAULT_SECRET)
|
||||||
parser.add_argument("--service", default=DEFAULT_SERVICE)
|
parser.add_argument("--service", default=DEFAULT_SERVICE)
|
||||||
parser.add_argument("--command-timeout", type=float, default=120.0)
|
parser.add_argument("--command-timeout", type=float, default=120.0)
|
||||||
|
|
@ -650,6 +1188,9 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||||
restore.add_argument("--dump", required=True, type=Path)
|
restore.add_argument("--dump", required=True, type=Path)
|
||||||
restore.add_argument("--expected-dump-sha256", required=True)
|
restore.add_argument("--expected-dump-sha256", required=True)
|
||||||
restore.add_argument("--source-manifest", required=True, type=Path)
|
restore.add_argument("--source-manifest", required=True, type=Path)
|
||||||
|
restore.add_argument("--source-receipt", required=True, type=Path)
|
||||||
|
restore.add_argument("--expected-source-receipt-sha256", required=True)
|
||||||
|
restore.add_argument("--expected-capture-authorization-ref", required=True)
|
||||||
restore.add_argument("--manifest-sql", default=DEFAULT_MANIFEST_SQL, type=Path)
|
restore.add_argument("--manifest-sql", default=DEFAULT_MANIFEST_SQL, type=Path)
|
||||||
restore.add_argument("--pg-restore-bin", default="pg_restore")
|
restore.add_argument("--pg-restore-bin", default="pg_restore")
|
||||||
restore.add_argument("--restore-timeout", type=float, default=900.0)
|
restore.add_argument("--restore-timeout", type=float, default=900.0)
|
||||||
|
|
@ -667,6 +1208,25 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||||
parser.error("--target-db must be a bounded lowercase teleo_clone_* identifier")
|
parser.error("--target-db must be a bounded lowercase teleo_clone_* identifier")
|
||||||
if not RUN_ID_RE.fullmatch(args.run_id):
|
if not RUN_ID_RE.fullmatch(args.run_id):
|
||||||
parser.error("--run-id must match gcp-restore-<8-48 lowercase letters, digits, or hyphens>")
|
parser.error("--run-id must match gcp-restore-<8-48 lowercase letters, digits, or hyphens>")
|
||||||
|
if not CLOUDSQL_INSTANCE_RE.fullmatch(args.cloudsql_instance):
|
||||||
|
parser.error("--cloudsql-instance must be a safe lowercase Cloud SQL instance name")
|
||||||
|
if not PRIVATE_NETWORK_RE.fullmatch(args.expected_private_network):
|
||||||
|
parser.error("--expected-private-network must be a full projects/.../global/networks/... resource")
|
||||||
|
if not args.expected_private_network.startswith(f"projects/{args.project}/global/networks/"):
|
||||||
|
parser.error("--expected-private-network project must match --project")
|
||||||
|
if not GCE_INSTANCE_RE.fullmatch(args.expected_compute_instance):
|
||||||
|
parser.error("--expected-compute-instance must be a safe lowercase GCE instance name")
|
||||||
|
if not GCE_ZONE_RE.fullmatch(args.expected_compute_zone):
|
||||||
|
parser.error("--expected-compute-zone must be a safe GCE zone")
|
||||||
|
if not SAFE_SSH_TARGET_RE.fullmatch(args.expected_source_ssh_target):
|
||||||
|
parser.error("--expected-source-ssh-target must be a host or user@host without options")
|
||||||
|
for value, flag in (
|
||||||
|
(args.expected_source_container, "--expected-source-container"),
|
||||||
|
(args.expected_source_database, "--expected-source-database"),
|
||||||
|
(args.expected_source_service, "--expected-source-service"),
|
||||||
|
):
|
||||||
|
if not SAFE_NAME_RE.fullmatch(value):
|
||||||
|
parser.error(f"{flag} contains unsafe characters")
|
||||||
try:
|
try:
|
||||||
host = ipaddress.ip_address(args.host)
|
host = ipaddress.ip_address(args.host)
|
||||||
except ValueError:
|
except ValueError:
|
||||||
|
|
@ -683,6 +1243,10 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||||
if args.operation == "restore":
|
if args.operation == "restore":
|
||||||
if not SHA256_RE.fullmatch(args.expected_dump_sha256):
|
if not SHA256_RE.fullmatch(args.expected_dump_sha256):
|
||||||
parser.error("--expected-dump-sha256 must be 64 lowercase hexadecimal characters")
|
parser.error("--expected-dump-sha256 must be 64 lowercase hexadecimal characters")
|
||||||
|
if not SHA256_RE.fullmatch(args.expected_source_receipt_sha256):
|
||||||
|
parser.error("--expected-source-receipt-sha256 must be 64 lowercase hexadecimal characters")
|
||||||
|
if not AUTHORIZATION_REF_RE.fullmatch(args.expected_capture_authorization_ref):
|
||||||
|
parser.error("--expected-capture-authorization-ref must be 8-256 safe metadata characters")
|
||||||
for value, flag in (
|
for value, flag in (
|
||||||
(args.restore_timeout, "--restore-timeout"),
|
(args.restore_timeout, "--restore-timeout"),
|
||||||
(args.manifest_timeout, "--manifest-timeout"),
|
(args.manifest_timeout, "--manifest-timeout"),
|
||||||
|
|
|
||||||
754
ops/verify_gcp_canonical_lifecycle.py
Normal file
754
ops/verify_gcp_canonical_lifecycle.py
Normal file
|
|
@ -0,0 +1,754 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Verify one bound source-to-GCP restore, reasoning, and cleanup lifecycle."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import hashlib
|
||||||
|
import ipaddress
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from itertools import pairwise
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
try:
|
||||||
|
from .verify_vps_canonical_snapshot_delta import DELTA_SCHEMA, parse_timestamp, write_private_json
|
||||||
|
except ImportError: # pragma: no cover - direct script execution
|
||||||
|
from verify_vps_canonical_snapshot_delta import DELTA_SCHEMA, parse_timestamp, write_private_json
|
||||||
|
|
||||||
|
SOURCE_SCHEMA = "livingip.sourceSnapshotReceipt.v2"
|
||||||
|
BLIND_REASONING_SCHEMA = "livingip.gcpGeneratedDbBlindClaimCanary.v1"
|
||||||
|
REASONING_COMPUTE_SCHEMA = "livingip.gcpReasoningComputeAttestation.v1"
|
||||||
|
RETRIEVAL_RECEIPT_SCHEMA = "livingip.teleoKbRetrievalReceipt.v1"
|
||||||
|
UUID_RE = re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}\b", re.I)
|
||||||
|
SHA256_RE = re.compile(r"[0-9a-f]{64}\Z")
|
||||||
|
FORBIDDEN_HANDLE_RE = re.compile(r"\b(?:cory|m3ta)\b", re.I)
|
||||||
|
EXPECTED_BLIND_ROW_IDS = {
|
||||||
|
"2a7ae257-d01d-46f4-b813-63f81bb9c7c7",
|
||||||
|
"261c3532-fa32-47d8-a5b5-6cc45035c267",
|
||||||
|
"15740795-ecc6-40fa-9a01-3d6bc7c54f79",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def sha256_file(path: Path) -> str:
|
||||||
|
digest = hashlib.sha256()
|
||||||
|
with path.open("rb") as handle:
|
||||||
|
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||||
|
digest.update(chunk)
|
||||||
|
return digest.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def load_receipt(path: Path, label: str) -> dict[str, Any]:
|
||||||
|
try:
|
||||||
|
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
except (OSError, json.JSONDecodeError) as exc:
|
||||||
|
raise ValueError(f"{label} is unreadable or invalid JSON") from exc
|
||||||
|
if not isinstance(payload, dict):
|
||||||
|
raise ValueError(f"{label} must be a JSON object")
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
def is_rfc1918(value: Any) -> bool:
|
||||||
|
try:
|
||||||
|
address = ipaddress.ip_address(str(value))
|
||||||
|
except ValueError:
|
||||||
|
return False
|
||||||
|
return any(
|
||||||
|
address in network
|
||||||
|
for network in (
|
||||||
|
ipaddress.ip_network("10.0.0.0/8"),
|
||||||
|
ipaddress.ip_network("172.16.0.0/12"),
|
||||||
|
ipaddress.ip_network("192.168.0.0/16"),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def nonempty_all_true(value: Any) -> bool:
|
||||||
|
return isinstance(value, dict) and bool(value) and all(item is True for item in value.values())
|
||||||
|
|
||||||
|
|
||||||
|
def required_true_checks(value: Any, required: set[str]) -> bool:
|
||||||
|
return nonempty_all_true(value) and required <= set(value)
|
||||||
|
|
||||||
|
|
||||||
|
def service_healthy(value: Any) -> bool:
|
||||||
|
if not isinstance(value, dict):
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
main_pid = int(value.get("MainPID") or 0)
|
||||||
|
restarts = int(value.get("NRestarts") or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return False
|
||||||
|
return (
|
||||||
|
value.get("ActiveState") == "active" and value.get("SubState") == "running" and main_pid > 0 and restarts >= 0
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def parity_evidence(details: dict[str, Any]) -> dict[str, bool]:
|
||||||
|
structural = details.get("structural_hashes") or {}
|
||||||
|
required_structures = {
|
||||||
|
"schemas",
|
||||||
|
"columns",
|
||||||
|
"constraints",
|
||||||
|
"indexes",
|
||||||
|
"sequences",
|
||||||
|
"views",
|
||||||
|
"functions",
|
||||||
|
"triggers",
|
||||||
|
"types",
|
||||||
|
"policies",
|
||||||
|
}
|
||||||
|
structural_valid = required_structures <= set(structural) and all(
|
||||||
|
bool(SHA256_RE.fullmatch(str((structural.get(kind) or {}).get("source") or "")))
|
||||||
|
and (structural.get(kind) or {}).get("source") == (structural.get(kind) or {}).get("target")
|
||||||
|
for kind in required_structures
|
||||||
|
)
|
||||||
|
performance = details.get("performance") or []
|
||||||
|
performance_valid = bool(performance) and all(
|
||||||
|
bool(row.get("query"))
|
||||||
|
and isinstance(row.get("source_ms"), (int, float))
|
||||||
|
and isinstance(row.get("target_ms"), (int, float))
|
||||||
|
and isinstance(row.get("source_ratio"), (int, float))
|
||||||
|
for row in performance
|
||||||
|
)
|
||||||
|
source_table_count = details.get("source_table_count")
|
||||||
|
target_table_count = details.get("target_table_count")
|
||||||
|
source_total_rows = details.get("source_total_rows")
|
||||||
|
target_total_rows = details.get("target_total_rows")
|
||||||
|
return {
|
||||||
|
"table_counts": isinstance(source_table_count, int)
|
||||||
|
and source_table_count > 0
|
||||||
|
and source_table_count == target_table_count,
|
||||||
|
"row_counts": isinstance(source_total_rows, int)
|
||||||
|
and source_total_rows > 0
|
||||||
|
and source_total_rows == target_total_rows,
|
||||||
|
"table_hashes": details.get("table_mismatches") == [],
|
||||||
|
"structural_hashes": structural_valid,
|
||||||
|
"roles": details.get("application_role_mismatches") == {}
|
||||||
|
and details.get("target_extra_application_roles") == [],
|
||||||
|
"extensions": details.get("required_extension_mismatches") == {},
|
||||||
|
"performance": performance_valid and details.get("performance_problems") == [],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def blind_reasoning_evidence(reasoning: dict[str, Any], target_db: Any) -> dict[str, bool]:
|
||||||
|
result = reasoning.get("result") or {}
|
||||||
|
gateway = result.get("gateway") or {}
|
||||||
|
child = gateway.get("child_process") or {}
|
||||||
|
tool_surface = gateway.get("tool_surface") or {}
|
||||||
|
tool_trace = result.get("database_tool_trace") or {}
|
||||||
|
calls = tool_trace.get("calls") or []
|
||||||
|
wrapper = reasoning.get("wrapper_tool_proof") or {}
|
||||||
|
invocations = wrapper.get("invocations") or []
|
||||||
|
prompt = str(reasoning.get("prompt") or "")
|
||||||
|
reply = str(result.get("reply") or "")
|
||||||
|
outcomes = reasoning.get("outcomes") or {}
|
||||||
|
required_outcomes = (
|
||||||
|
"asks_for_user_iteration_before_live",
|
||||||
|
"challenges_weak_support",
|
||||||
|
"decomposes_the_bundled_legal_claim",
|
||||||
|
"does_not_claim_a_database_change",
|
||||||
|
"proposes_candidate_claims",
|
||||||
|
"uses_only_m3taversal_handle",
|
||||||
|
)
|
||||||
|
required_subcommands = {"show", "evidence"}
|
||||||
|
discovery_subcommands = {"search", "context"}
|
||||||
|
traced_subcommands = {
|
||||||
|
str(invocation.get("subcommand")) for call in calls for invocation in (call.get("database_invocations") or [])
|
||||||
|
}
|
||||||
|
wrapper_subcommands = {str(invocation.get("subcommand")) for invocation in invocations}
|
||||||
|
retrieved_row_ids = {str(row_id) for call in calls for row_id in ((call.get("result") or {}).get("row_ids") or [])}
|
||||||
|
call_row_ids_valid = bool(calls) and all(
|
||||||
|
isinstance((call.get("result") or {}).get("row_ids"), list)
|
||||||
|
and isinstance((call.get("result") or {}).get("row_id_count"), int)
|
||||||
|
and (call.get("result") or {}).get("row_id_count") >= 0
|
||||||
|
and (call.get("result") or {}).get("row_id_count") >= len((call.get("result") or {}).get("row_ids") or [])
|
||||||
|
for call in calls
|
||||||
|
)
|
||||||
|
call_receipts_valid = bool(calls) and all(
|
||||||
|
(call.get("result") or {}).get("nonempty") is True
|
||||||
|
and (call.get("result") or {}).get("error_detected") is False
|
||||||
|
and ((call.get("result") or {}).get("retrieval_receipt") or {}).get("schema") == RETRIEVAL_RECEIPT_SCHEMA
|
||||||
|
and bool(
|
||||||
|
SHA256_RE.fullmatch(
|
||||||
|
str(((call.get("result") or {}).get("retrieval_receipt") or {}).get("semantic_context_sha256") or "")
|
||||||
|
)
|
||||||
|
)
|
||||||
|
and bool(
|
||||||
|
SHA256_RE.fullmatch(
|
||||||
|
str(((call.get("result") or {}).get("retrieval_receipt") or {}).get("artifact_state_sha256") or "")
|
||||||
|
)
|
||||||
|
)
|
||||||
|
and str(
|
||||||
|
((call.get("result") or {}).get("retrieval_receipt") or {}).get("read_consistency_status") or ""
|
||||||
|
).startswith("stable")
|
||||||
|
for call in calls
|
||||||
|
)
|
||||||
|
wrapper_invocations_valid = bool(invocations) and all(
|
||||||
|
invocation.get("database") == target_db
|
||||||
|
and invocation.get("returncode") == 0
|
||||||
|
and ((invocation.get("database_identity") or {}).get("transaction_read_only") == "on")
|
||||||
|
and ((invocation.get("database_identity") or {}).get("default_transaction_read_only") == "on")
|
||||||
|
and ((invocation.get("database_identity") or {}).get("ssl") is True)
|
||||||
|
and is_rfc1918((invocation.get("database_identity") or {}).get("server_address"))
|
||||||
|
for invocation in invocations
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"mode": reasoning.get("mode") == "gcp_generated_db_gatewayrunner_blind_claim_no_send",
|
||||||
|
"timestamps": bool(reasoning.get("generated_at_utc")) and bool(reasoning.get("completed_at_utc")),
|
||||||
|
"source_compute": bool(reasoning.get("source_compute")),
|
||||||
|
"blind_prompt": bool(prompt) and UUID_RE.search(prompt) is None,
|
||||||
|
"reply_nonempty": bool(reply.strip()),
|
||||||
|
"reply_uses_only_m3taversal_handle": FORBIDDEN_HANDLE_RE.search(reply) is None,
|
||||||
|
"outcomes": all(outcomes.get(name) is True for name in required_outcomes),
|
||||||
|
"gateway": gateway.get("authorized") is True
|
||||||
|
and gateway.get("handler_invoked") is True
|
||||||
|
and gateway.get("model_free_fallback_used") is False
|
||||||
|
and gateway.get("posted_to_telegram") is False,
|
||||||
|
"gateway_cleanup": child.get("exitcode") == 0
|
||||||
|
and child.get("alive_after_readback") is False
|
||||||
|
and child.get("process_group_alive_after_readback") is False
|
||||||
|
and child.get("timed_out") is False,
|
||||||
|
"send_tool_absent": tool_surface.get("send_message_tool_enabled") is False,
|
||||||
|
"tool_trace": tool_trace.get("schema") == "livingip.leoKbToolTrace.v1"
|
||||||
|
and tool_trace.get("database_tool_call_proven") is True
|
||||||
|
and tool_trace.get("database_retrieval_receipt_proven") is True
|
||||||
|
and tool_trace.get("database_tool_calls_read_only") is True
|
||||||
|
and tool_trace.get("database_tool_call_count") == len(calls)
|
||||||
|
and tool_trace.get("database_tool_completed_count") == len(calls),
|
||||||
|
"tool_receipts": call_receipts_valid,
|
||||||
|
"expected_claim_and_sources_retrieved": call_row_ids_valid and retrieved_row_ids >= EXPECTED_BLIND_ROW_IDS,
|
||||||
|
"required_tool_sequence": required_subcommands <= traced_subcommands
|
||||||
|
and bool(discovery_subcommands & traced_subcommands),
|
||||||
|
"wrapper": wrapper.get("all_bound_to_supplied_target") is True
|
||||||
|
and wrapper.get("all_completed_successfully") is True
|
||||||
|
and wrapper.get("complete_start_end_pairing") is True
|
||||||
|
and wrapper.get("database_read_only_required") is True
|
||||||
|
and wrapper.get("default_read_only_required") is True,
|
||||||
|
"wrapper_invocations": wrapper_invocations_valid
|
||||||
|
and required_subcommands <= wrapper_subcommands
|
||||||
|
and bool(discovery_subcommands & wrapper_subcommands),
|
||||||
|
"canonical_status_unchanged": bool(reasoning.get("canonical_status_before"))
|
||||||
|
and reasoning.get("canonical_status_before") == reasoning.get("canonical_status_after"),
|
||||||
|
"database_identity_unchanged": bool(reasoning.get("database_identity_before"))
|
||||||
|
and reasoning.get("database_identity_before") == reasoning.get("database_identity_after"),
|
||||||
|
"service_unchanged": service_healthy(reasoning.get("service_before"))
|
||||||
|
and reasoning.get("service_before") == reasoning.get("service_after"),
|
||||||
|
"temporary_profile_removed": reasoning.get("temp_profile_absent") is True
|
||||||
|
and reasoning.get("temp_profile_removed") is True,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def verify_lifecycle(
|
||||||
|
*,
|
||||||
|
source_path: Path,
|
||||||
|
current_source_path: Path,
|
||||||
|
source_delta_path: Path,
|
||||||
|
restore_path: Path,
|
||||||
|
parity_path: Path,
|
||||||
|
reasoning_path: Path,
|
||||||
|
reasoning_compute_path: Path,
|
||||||
|
cleanup_path: Path,
|
||||||
|
max_postflight_age_seconds: float = 900.0,
|
||||||
|
max_lifecycle_age_seconds: float = 3600.0,
|
||||||
|
now: datetime | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
if max_postflight_age_seconds <= 0:
|
||||||
|
raise ValueError("max postflight age must be positive")
|
||||||
|
if max_lifecycle_age_seconds <= 0:
|
||||||
|
raise ValueError("max lifecycle age must be positive")
|
||||||
|
verification_time = (now or datetime.now(UTC)).astimezone(UTC)
|
||||||
|
source = load_receipt(source_path, "source receipt")
|
||||||
|
current_source = load_receipt(current_source_path, "current source receipt")
|
||||||
|
source_delta = load_receipt(source_delta_path, "source delta receipt")
|
||||||
|
restore = load_receipt(restore_path, "restore receipt")
|
||||||
|
parity = load_receipt(parity_path, "parity receipt")
|
||||||
|
reasoning = load_receipt(reasoning_path, "reasoning receipt")
|
||||||
|
reasoning_compute_attestation = load_receipt(reasoning_compute_path, "reasoning compute attestation")
|
||||||
|
cleanup = load_receipt(cleanup_path, "cleanup receipt")
|
||||||
|
|
||||||
|
target_db = restore.get("target_database")
|
||||||
|
run_id = restore.get("run_id")
|
||||||
|
restore_source = restore.get("source") or {}
|
||||||
|
capture_binding = restore_source.get("capture_receipt") or {}
|
||||||
|
restore_target = restore.get("target") or {}
|
||||||
|
restore_connectivity = restore_target.get("connectivity") or {}
|
||||||
|
restore_connectivity_proof = restore_target.get("connectivity_proof") or {}
|
||||||
|
restore_cloudsql = restore_target.get("cloudsql") or {}
|
||||||
|
restore_compute = restore_target.get("compute") or {}
|
||||||
|
restore_parity = restore.get("parity") or {}
|
||||||
|
restore_service = restore.get("live_service") or {}
|
||||||
|
parity_details = parity.get("details") or {}
|
||||||
|
parity_evidence_checks = parity_evidence(parity_details)
|
||||||
|
restore_inline_parity_checks = parity_evidence(restore_parity.get("details") or {})
|
||||||
|
parity_connectivity = parity.get("private_connectivity") or {}
|
||||||
|
parity_connectivity_proof = parity_connectivity.get("proof") or {}
|
||||||
|
reasoning_checks = reasoning.get("checks") or {}
|
||||||
|
reasoning_parity = reasoning.get("parity_receipt") or {}
|
||||||
|
before_fingerprint = reasoning.get("database_fingerprint_before") or {}
|
||||||
|
after_fingerprint = reasoning.get("database_fingerprint_after") or {}
|
||||||
|
cleanup_database = cleanup.get("database") or {}
|
||||||
|
cleanup_service = cleanup.get("live_service") or {}
|
||||||
|
cleanup_cloudsql = (cleanup.get("target") or {}).get("cloudsql") or {}
|
||||||
|
cleanup_compute = (cleanup.get("target") or {}).get("compute") or {}
|
||||||
|
restore_contract = restore.get("execution_contract") or {}
|
||||||
|
cleanup_contract = cleanup.get("execution_contract") or {}
|
||||||
|
producer_reasoning_checks = blind_reasoning_evidence(reasoning, target_db)
|
||||||
|
reasoning_compute = reasoning_compute_attestation.get("compute") or {}
|
||||||
|
reasoning_compute_checks = reasoning_compute_attestation.get("checks") or {}
|
||||||
|
reasoning_compute_mutation = reasoning_compute_attestation.get("mutation") or {}
|
||||||
|
delta_baseline = source_delta.get("baseline") or {}
|
||||||
|
delta_current = source_delta.get("current") or {}
|
||||||
|
delta_details = source_delta.get("delta") or {}
|
||||||
|
source_snapshot = source.get("snapshot") or {}
|
||||||
|
current_source_snapshot = current_source.get("snapshot") or {}
|
||||||
|
timeline_values = {
|
||||||
|
"source_snapshot": source_snapshot.get("captured_at_utc"),
|
||||||
|
"restore_started": restore.get("generated_at_utc"),
|
||||||
|
"restore_completed": restore.get("completed_at_utc"),
|
||||||
|
"parity_completed": parity.get("completed_at_utc"),
|
||||||
|
"reasoning_started": reasoning.get("generated_at_utc"),
|
||||||
|
"reasoning_completed": reasoning.get("completed_at_utc"),
|
||||||
|
"reasoning_compute_attested": reasoning_compute_attestation.get("generated_at_utc"),
|
||||||
|
"cleanup_started": cleanup.get("generated_at_utc"),
|
||||||
|
"cleanup_completed": cleanup.get("completed_at_utc"),
|
||||||
|
"current_source_snapshot": current_source_snapshot.get("captured_at_utc"),
|
||||||
|
}
|
||||||
|
timeline = [parse_timestamp(value, name) for name, value in timeline_values.items()]
|
||||||
|
lifecycle_chronology = all(earlier <= later for earlier, later in pairwise(timeline))
|
||||||
|
delta_detected = delta_details.get("delta_detected")
|
||||||
|
delta_change_present = bool(
|
||||||
|
delta_details.get("added_tables")
|
||||||
|
or delta_details.get("removed_tables")
|
||||||
|
or delta_details.get("changed_tables")
|
||||||
|
or delta_details.get("changed_structures")
|
||||||
|
or delta_details.get("total_row_delta")
|
||||||
|
)
|
||||||
|
current_source_time = parse_timestamp(
|
||||||
|
current_source_snapshot.get("captured_at_utc"), "current source snapshot time"
|
||||||
|
)
|
||||||
|
restore_started_time = parse_timestamp(restore.get("generated_at_utc"), "restore start time")
|
||||||
|
cleanup_completed_time = parse_timestamp(cleanup.get("completed_at_utc"), "cleanup completion time")
|
||||||
|
postflight_age_seconds = (verification_time - current_source_time).total_seconds()
|
||||||
|
lifecycle_age_seconds = (verification_time - cleanup_completed_time).total_seconds()
|
||||||
|
lifecycle_span_seconds = (current_source_time - restore_started_time).total_seconds()
|
||||||
|
postflight_after_lifecycle = current_source_time >= max(
|
||||||
|
parse_timestamp(reasoning.get("completed_at_utc"), "reasoning completion time"),
|
||||||
|
parse_timestamp(cleanup.get("completed_at_utc"), "cleanup completion time"),
|
||||||
|
)
|
||||||
|
|
||||||
|
required_reasoning_checks = (
|
||||||
|
"exact_blind_prompt_without_ids",
|
||||||
|
"claim_and_sources_retrieved",
|
||||||
|
"discovery_show_evidence_completed",
|
||||||
|
"retrieval_receipts_proven",
|
||||||
|
"private_tls_read_only_target",
|
||||||
|
"generated_database_unchanged",
|
||||||
|
"canonical_counts_unchanged",
|
||||||
|
"database_calls_read_only",
|
||||||
|
"no_database_write",
|
||||||
|
"no_telegram_send",
|
||||||
|
"live_service_unchanged",
|
||||||
|
"live_profile_unchanged",
|
||||||
|
"temporary_profile_removed",
|
||||||
|
)
|
||||||
|
checks = {
|
||||||
|
"source_schema_v2": source.get("schema") == SOURCE_SCHEMA and source.get("receipt_version") == 2,
|
||||||
|
"source_pass": source.get("status") == "pass"
|
||||||
|
and source.get("snapshot_exported") is True
|
||||||
|
and (source.get("source_service") or {}).get("unchanged") is True
|
||||||
|
and service_healthy((source.get("source_service") or {}).get("before"))
|
||||||
|
and (source.get("source_service") or {}).get("before") == (source.get("source_service") or {}).get("after"),
|
||||||
|
"source_authorized": bool(source.get("capture_authorization_ref")),
|
||||||
|
"source_authorization_bound": capture_binding.get("capture_authorization_ref")
|
||||||
|
== source.get("capture_authorization_ref"),
|
||||||
|
"source_service_unchanged": source.get("service_unchanged") is True,
|
||||||
|
"source_not_mutated": source.get("production_db_mutated") is False,
|
||||||
|
"current_source_schema_v2": current_source.get("schema") == SOURCE_SCHEMA
|
||||||
|
and current_source.get("receipt_version") == 2,
|
||||||
|
"current_source_pass": current_source.get("status") == "pass"
|
||||||
|
and current_source.get("snapshot_exported") is True
|
||||||
|
and current_source.get("service_unchanged") is True
|
||||||
|
and (current_source.get("source_service") or {}).get("unchanged") is True
|
||||||
|
and service_healthy((current_source.get("source_service") or {}).get("before"))
|
||||||
|
and (current_source.get("source_service") or {}).get("before")
|
||||||
|
== (current_source.get("source_service") or {}).get("after")
|
||||||
|
and current_source.get("production_db_mutated") is False,
|
||||||
|
"current_source_identity": current_source.get("source") == source.get("source")
|
||||||
|
and current_source_snapshot.get("system_identifier") == source_snapshot.get("system_identifier"),
|
||||||
|
"source_delta_pass": source_delta.get("artifact") == "vps_canonical_snapshot_postflight_delta"
|
||||||
|
and source_delta.get("schema") == DELTA_SCHEMA
|
||||||
|
and source_delta.get("status") == "pass",
|
||||||
|
"source_delta_receipts_bound": delta_baseline.get("capture_receipt_sha256") == sha256_file(source_path)
|
||||||
|
and delta_current.get("capture_receipt_sha256") == sha256_file(current_source_path),
|
||||||
|
"source_delta_manifests_bound": delta_baseline.get("manifest_sha256")
|
||||||
|
== (source.get("manifest") or {}).get("sha256")
|
||||||
|
and delta_current.get("manifest_sha256") == (current_source.get("manifest") or {}).get("sha256"),
|
||||||
|
"source_delta_authorizations_bound": delta_baseline.get("capture_authorization_ref")
|
||||||
|
== source.get("capture_authorization_ref")
|
||||||
|
and delta_current.get("capture_authorization_ref") == current_source.get("capture_authorization_ref"),
|
||||||
|
"source_delta_explicit": isinstance(delta_details.get("delta_detected"), bool)
|
||||||
|
and isinstance(delta_details.get("added_tables"), list)
|
||||||
|
and isinstance(delta_details.get("removed_tables"), list)
|
||||||
|
and isinstance(delta_details.get("changed_tables"), list)
|
||||||
|
and isinstance(delta_details.get("changed_structures"), list)
|
||||||
|
and isinstance(delta_details.get("total_row_delta"), int),
|
||||||
|
"source_delta_baseline_shape": delta_baseline.get("run_id") == source.get("run_id")
|
||||||
|
and delta_baseline.get("captured_at_utc") == source_snapshot.get("captured_at_utc")
|
||||||
|
and delta_baseline.get("table_count") == (source.get("manifest") or {}).get("table_count")
|
||||||
|
and delta_baseline.get("total_rows") == (source.get("manifest") or {}).get("total_rows")
|
||||||
|
and bool(SHA256_RE.fullmatch(str(delta_baseline.get("table_fingerprint_sha256") or "")))
|
||||||
|
and bool(SHA256_RE.fullmatch(str(delta_baseline.get("structure_fingerprint_sha256") or ""))),
|
||||||
|
"source_delta_current_shape": delta_current.get("run_id") == current_source.get("run_id")
|
||||||
|
and delta_current.get("captured_at_utc") == current_source_snapshot.get("captured_at_utc")
|
||||||
|
and delta_current.get("table_count") == (current_source.get("manifest") or {}).get("table_count")
|
||||||
|
and delta_current.get("total_rows") == (current_source.get("manifest") or {}).get("total_rows")
|
||||||
|
and bool(SHA256_RE.fullmatch(str(delta_current.get("table_fingerprint_sha256") or "")))
|
||||||
|
and bool(SHA256_RE.fullmatch(str(delta_current.get("structure_fingerprint_sha256") or ""))),
|
||||||
|
"source_delta_consistent": delta_detected is delta_change_present
|
||||||
|
and delta_details.get("total_row_delta")
|
||||||
|
== (delta_current.get("total_rows") or 0) - (delta_baseline.get("total_rows") or 0)
|
||||||
|
and (
|
||||||
|
delta_detected is True
|
||||||
|
or (
|
||||||
|
delta_baseline.get("table_count") == delta_current.get("table_count")
|
||||||
|
and delta_baseline.get("total_rows") == delta_current.get("total_rows")
|
||||||
|
and delta_baseline.get("table_fingerprint_sha256") == delta_current.get("table_fingerprint_sha256")
|
||||||
|
and delta_baseline.get("structure_fingerprint_sha256")
|
||||||
|
== delta_current.get("structure_fingerprint_sha256")
|
||||||
|
)
|
||||||
|
),
|
||||||
|
"source_postflight_after_lifecycle": postflight_after_lifecycle,
|
||||||
|
"source_postflight_fresh": 0 <= postflight_age_seconds <= max_postflight_age_seconds,
|
||||||
|
"gcp_lifecycle_fresh": 0 <= lifecycle_age_seconds <= max_lifecycle_age_seconds,
|
||||||
|
"lifecycle_span_bounded": 0 <= lifecycle_span_seconds <= max_lifecycle_age_seconds,
|
||||||
|
"lifecycle_chronology": lifecycle_chronology,
|
||||||
|
"restore_pass": restore.get("artifact") == "gcp_generated_postgres_snapshot_restore"
|
||||||
|
and restore.get("status") == "pass",
|
||||||
|
"restore_target_is_disposable": isinstance(target_db, str) and target_db.startswith("teleo_clone_"),
|
||||||
|
"restore_run_id": isinstance(run_id, str) and run_id.startswith("gcp-restore-"),
|
||||||
|
"source_receipt_hash_bound": capture_binding.get("sha256") == sha256_file(source_path),
|
||||||
|
"source_provenance_bound": capture_binding.get("provenance_binding_sha256")
|
||||||
|
== (source.get("provenance_binding") or {}).get("sha256"),
|
||||||
|
"source_capture_checks": required_true_checks(
|
||||||
|
capture_binding.get("checks"),
|
||||||
|
{
|
||||||
|
"artifact",
|
||||||
|
"schema",
|
||||||
|
"receipt_version",
|
||||||
|
"status",
|
||||||
|
"snapshot_exported",
|
||||||
|
"service_unchanged",
|
||||||
|
"source_service_healthy",
|
||||||
|
"source_not_mutated",
|
||||||
|
"telegram_not_sent",
|
||||||
|
"run_id",
|
||||||
|
"capture_authorization_ref",
|
||||||
|
"source_identity",
|
||||||
|
"source_database",
|
||||||
|
"snapshot_identity",
|
||||||
|
"dump_sha256",
|
||||||
|
"dump_bytes",
|
||||||
|
"source_manifest_sha256",
|
||||||
|
"manifest_sql_sha256",
|
||||||
|
"source_context_sha256",
|
||||||
|
"table_count",
|
||||||
|
"total_rows",
|
||||||
|
"binding_algorithm",
|
||||||
|
"binding_payload",
|
||||||
|
"binding_sha256",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
"source_dump_bound": capture_binding.get("dump_sha256") == (source.get("dump") or {}).get("sha256"),
|
||||||
|
"source_execution_contract_bound": restore_contract.get("source") == source.get("source"),
|
||||||
|
"source_manifest_bound": capture_binding.get("manifest_sha256") == (source.get("manifest") or {}).get("sha256"),
|
||||||
|
"restore_manifest_matches_parity_source": capture_binding.get("manifest_sha256")
|
||||||
|
== parity.get("source_manifest_sha256"),
|
||||||
|
"restore_target_manifest_matches_parity": restore_target.get("manifest_sha256")
|
||||||
|
== parity.get("target_manifest_sha256"),
|
||||||
|
"restore_private_tls": restore_connectivity.get("private_connectivity") is True
|
||||||
|
and restore_connectivity.get("ssl") is True
|
||||||
|
and is_rfc1918(restore_connectivity.get("server_address")),
|
||||||
|
"restore_public_ip_disabled": restore_cloudsql.get("public_ip_disabled") is True,
|
||||||
|
"restore_cloudsql_checks": required_true_checks(
|
||||||
|
restore_cloudsql.get("checks"),
|
||||||
|
{
|
||||||
|
"instance",
|
||||||
|
"state",
|
||||||
|
"postgres_16",
|
||||||
|
"public_ip_disabled",
|
||||||
|
"private_network",
|
||||||
|
"expected_private_host",
|
||||||
|
"no_non_private_address",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
"restore_compute_checks": required_true_checks(
|
||||||
|
restore_compute.get("checks"),
|
||||||
|
{
|
||||||
|
"project",
|
||||||
|
"project_number",
|
||||||
|
"expected_network_project",
|
||||||
|
"instance",
|
||||||
|
"instance_id",
|
||||||
|
"zone",
|
||||||
|
"network",
|
||||||
|
"private_ip",
|
||||||
|
"service_account",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
"restore_compute_contract": restore_compute.get("project") == restore_contract.get("project")
|
||||||
|
and restore_compute.get("instance") == restore_contract.get("compute_instance")
|
||||||
|
and restore_compute.get("zone") == restore_contract.get("compute_zone")
|
||||||
|
and restore_compute.get("expected_private_network") == restore_contract.get("private_network")
|
||||||
|
and bool(restore_compute.get("network_resource"))
|
||||||
|
and is_rfc1918(restore_compute.get("private_ip"))
|
||||||
|
and str(restore_compute.get("service_account") or "").endswith(".iam.gserviceaccount.com"),
|
||||||
|
"restore_connectivity_compute_bound": restore_connectivity.get("source_compute")
|
||||||
|
== restore_compute.get("instance"),
|
||||||
|
"restore_inline_parity": restore_parity.get("status") == "pass" and restore_parity.get("problems") == [],
|
||||||
|
"restore_inline_parity_evidence": all(restore_inline_parity_checks.values()),
|
||||||
|
"restore_service_healthy_unchanged": restore_service.get("unchanged") is True
|
||||||
|
and service_healthy(restore_service.get("before"))
|
||||||
|
and restore_service.get("before") == restore_service.get("after"),
|
||||||
|
"parity_pass": parity.get("artifact") == "canonical_postgres_parity_verification"
|
||||||
|
and parity.get("status") == "pass"
|
||||||
|
and parity.get("scope") == "gcp_staging"
|
||||||
|
and parity.get("problems") == [],
|
||||||
|
"parity_target_bound": parity_details.get("target_database") == target_db,
|
||||||
|
"parity_exact_evidence": all(parity_evidence_checks.values()),
|
||||||
|
"parity_connectivity_hash_bound": restore_connectivity_proof.get("sha256")
|
||||||
|
== parity.get("connectivity_proof_sha256"),
|
||||||
|
"parity_private_proof": parity_connectivity.get("required") is True
|
||||||
|
and parity_connectivity_proof.get("artifact") == "gcp_private_postgres_connectivity"
|
||||||
|
and parity_connectivity_proof.get("status") == "pass"
|
||||||
|
and parity_connectivity_proof.get("schema") == "livingip.gcpPrivatePostgresConnectivity.v2"
|
||||||
|
and parity_connectivity_proof.get("restore_run_id") == run_id
|
||||||
|
and parity_connectivity_proof.get("private_connectivity") is True
|
||||||
|
and parity_connectivity_proof.get("public_ip_disabled") is True
|
||||||
|
and parity_connectivity_proof.get("target_database") == target_db
|
||||||
|
and parity_connectivity_proof.get("project") == restore_cloudsql.get("project")
|
||||||
|
and parity_connectivity_proof.get("cloudsql_instance") == restore_cloudsql.get("instance")
|
||||||
|
and parity_connectivity_proof.get("private_network") == restore_cloudsql.get("private_network")
|
||||||
|
and parity_connectivity_proof.get("source_compute") == reasoning.get("source_compute")
|
||||||
|
and parity_connectivity_proof.get("source_compute_instance_id") == restore_compute.get("instance_id")
|
||||||
|
and parity_connectivity_proof.get("source_compute_zone") == restore_compute.get("zone")
|
||||||
|
and is_rfc1918(parity_connectivity_proof.get("source_compute_private_ip"))
|
||||||
|
and parity_connectivity_proof.get("server_address") == restore_connectivity.get("server_address")
|
||||||
|
and required_true_checks(
|
||||||
|
parity_connectivity_proof.get("cloudsql_control_plane_checks"),
|
||||||
|
{
|
||||||
|
"instance",
|
||||||
|
"state",
|
||||||
|
"postgres_16",
|
||||||
|
"public_ip_disabled",
|
||||||
|
"private_network",
|
||||||
|
"expected_private_host",
|
||||||
|
"no_non_private_address",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
and required_true_checks(
|
||||||
|
parity_connectivity_proof.get("compute_identity_checks"),
|
||||||
|
{
|
||||||
|
"project",
|
||||||
|
"project_number",
|
||||||
|
"expected_network_project",
|
||||||
|
"instance",
|
||||||
|
"instance_id",
|
||||||
|
"zone",
|
||||||
|
"network",
|
||||||
|
"private_ip",
|
||||||
|
"service_account",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
"reasoning_schema": reasoning.get("schema") == BLIND_REASONING_SCHEMA,
|
||||||
|
"reasoning_pass": reasoning.get("status") == "pass" and not reasoning.get("errors"),
|
||||||
|
"reasoning_target_bound": reasoning.get("target_database") == target_db,
|
||||||
|
"reasoning_compute_bound": reasoning.get("source_compute") == restore_compute.get("instance"),
|
||||||
|
"reasoning_parity_hash_bound": reasoning_parity.get("sha256") == sha256_file(parity_path),
|
||||||
|
"reasoning_required_checks": all(reasoning_checks.get(name) is True for name in required_reasoning_checks),
|
||||||
|
"reasoning_producer_evidence": all(producer_reasoning_checks.values()),
|
||||||
|
"reasoning_compute_attestation": reasoning_compute_attestation.get("artifact")
|
||||||
|
== "gcp_reasoning_compute_attestation"
|
||||||
|
and reasoning_compute_attestation.get("schema") == REASONING_COMPUTE_SCHEMA
|
||||||
|
and reasoning_compute_attestation.get("status") == "pass"
|
||||||
|
and reasoning_compute_attestation.get("method") == "gce_metadata_server_v1"
|
||||||
|
and nonempty_all_true(reasoning_compute_checks)
|
||||||
|
and reasoning_compute_attestation.get("restore_run_id") == run_id
|
||||||
|
and reasoning_compute_attestation.get("target_database") == target_db
|
||||||
|
and all(value is False for value in reasoning_compute_mutation.values())
|
||||||
|
and {
|
||||||
|
"cloudsql_changed",
|
||||||
|
"compute_changed",
|
||||||
|
"database_changed",
|
||||||
|
"telegram_message_sent",
|
||||||
|
}
|
||||||
|
<= set(reasoning_compute_mutation),
|
||||||
|
"reasoning_compute_receipt_bound": (reasoning_compute_attestation.get("reasoning_receipt") or {}).get("sha256")
|
||||||
|
== sha256_file(reasoning_path),
|
||||||
|
"reasoning_compute_identity_bound": reasoning_compute.get("project") == restore_compute.get("project")
|
||||||
|
and reasoning_compute.get("project_number") == restore_compute.get("project_number")
|
||||||
|
and reasoning_compute.get("instance") == restore_compute.get("instance")
|
||||||
|
and reasoning_compute.get("instance_id") == restore_compute.get("instance_id")
|
||||||
|
and reasoning_compute.get("zone") == restore_compute.get("zone")
|
||||||
|
and reasoning_compute.get("expected_private_network") == restore_compute.get("expected_private_network")
|
||||||
|
and reasoning_compute.get("network_resource") == restore_compute.get("network_resource")
|
||||||
|
and reasoning_compute.get("private_ip") == restore_compute.get("private_ip")
|
||||||
|
and reasoning_compute.get("service_account") == restore_compute.get("service_account")
|
||||||
|
and required_true_checks(
|
||||||
|
reasoning_compute.get("checks"),
|
||||||
|
{
|
||||||
|
"project",
|
||||||
|
"project_number",
|
||||||
|
"expected_network_project",
|
||||||
|
"instance",
|
||||||
|
"instance_id",
|
||||||
|
"zone",
|
||||||
|
"network",
|
||||||
|
"private_ip",
|
||||||
|
"service_account",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
"reasoning_no_send_no_write": reasoning.get("posted_to_telegram") is False
|
||||||
|
and reasoning.get("database_write_attempted") is False,
|
||||||
|
"reasoning_fingerprint_unchanged": bool(before_fingerprint.get("sha256"))
|
||||||
|
and before_fingerprint.get("sha256") == after_fingerprint.get("sha256"),
|
||||||
|
"cleanup_pass": cleanup.get("artifact") == "gcp_generated_postgres_snapshot_cleanup"
|
||||||
|
and cleanup.get("status") == "pass",
|
||||||
|
"cleanup_run_bound": cleanup.get("run_id") == run_id and cleanup.get("target_database") == target_db,
|
||||||
|
"cleanup_restore_receipt_bound": (cleanup.get("restore_receipt") or {}).get("sha256")
|
||||||
|
== sha256_file(restore_path),
|
||||||
|
"cleanup_execution_contract_bound": bool(restore_contract) and cleanup_contract == restore_contract,
|
||||||
|
"cleanup_cloudsql_bound": cleanup_cloudsql.get("project") == restore_cloudsql.get("project")
|
||||||
|
and cleanup_cloudsql.get("instance") == restore_cloudsql.get("instance")
|
||||||
|
and cleanup_cloudsql.get("expected_private_host") == restore_cloudsql.get("expected_private_host")
|
||||||
|
and cleanup_cloudsql.get("expected_private_network") == restore_cloudsql.get("expected_private_network")
|
||||||
|
and cleanup_cloudsql.get("public_ip_disabled") is True
|
||||||
|
and required_true_checks(
|
||||||
|
cleanup_cloudsql.get("checks"),
|
||||||
|
{
|
||||||
|
"instance",
|
||||||
|
"state",
|
||||||
|
"postgres_16",
|
||||||
|
"public_ip_disabled",
|
||||||
|
"private_network",
|
||||||
|
"expected_private_host",
|
||||||
|
"no_non_private_address",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
"cleanup_compute_bound": cleanup_compute.get("instance_id") == restore_compute.get("instance_id")
|
||||||
|
and cleanup_compute.get("project") == restore_compute.get("project")
|
||||||
|
and cleanup_compute.get("project_number") == restore_compute.get("project_number")
|
||||||
|
and cleanup_compute.get("instance") == restore_compute.get("instance")
|
||||||
|
and cleanup_compute.get("zone") == restore_compute.get("zone")
|
||||||
|
and cleanup_compute.get("expected_private_network") == restore_compute.get("expected_private_network")
|
||||||
|
and cleanup_compute.get("network_resource") == restore_compute.get("network_resource")
|
||||||
|
and cleanup_compute.get("private_ip") == restore_compute.get("private_ip")
|
||||||
|
and cleanup_compute.get("service_account") == restore_compute.get("service_account")
|
||||||
|
and required_true_checks(
|
||||||
|
cleanup_compute.get("checks"),
|
||||||
|
{
|
||||||
|
"project",
|
||||||
|
"project_number",
|
||||||
|
"expected_network_project",
|
||||||
|
"instance",
|
||||||
|
"instance_id",
|
||||||
|
"zone",
|
||||||
|
"network",
|
||||||
|
"private_ip",
|
||||||
|
"service_account",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
"clone_absent": cleanup_database.get("clone_database_remaining") == 0,
|
||||||
|
"cleanup_service_unchanged": cleanup_service.get("unchanged") is True
|
||||||
|
and service_healthy(cleanup_service.get("before"))
|
||||||
|
and cleanup_service.get("before") == cleanup_service.get("after"),
|
||||||
|
}
|
||||||
|
failed = sorted(name for name, passed in checks.items() if not passed)
|
||||||
|
return {
|
||||||
|
"artifact": "gcp_canonical_snapshot_lifecycle_verification",
|
||||||
|
"generated_at_utc": verification_time.isoformat(),
|
||||||
|
"status": "pass" if not failed else "fail",
|
||||||
|
"required_tier": "T3_live_private_gcp_staging",
|
||||||
|
"target_database": target_db,
|
||||||
|
"run_id": run_id,
|
||||||
|
"checks": checks,
|
||||||
|
"reasoning_producer_checks": producer_reasoning_checks,
|
||||||
|
"restore_inline_parity_checks": restore_inline_parity_checks,
|
||||||
|
"parity_evidence_checks": parity_evidence_checks,
|
||||||
|
"post_snapshot_source_delta": delta_details,
|
||||||
|
"timeline": timeline_values,
|
||||||
|
"postflight_age_seconds": postflight_age_seconds,
|
||||||
|
"max_postflight_age_seconds": max_postflight_age_seconds,
|
||||||
|
"lifecycle_age_seconds": lifecycle_age_seconds,
|
||||||
|
"lifecycle_span_seconds": lifecycle_span_seconds,
|
||||||
|
"max_lifecycle_age_seconds": max_lifecycle_age_seconds,
|
||||||
|
"failed_checks": failed,
|
||||||
|
"receipt_sha256": {
|
||||||
|
"source": sha256_file(source_path),
|
||||||
|
"current_source": sha256_file(current_source_path),
|
||||||
|
"source_delta": sha256_file(source_delta_path),
|
||||||
|
"restore": sha256_file(restore_path),
|
||||||
|
"parity": sha256_file(parity_path),
|
||||||
|
"reasoning": sha256_file(reasoning_path),
|
||||||
|
"reasoning_compute": sha256_file(reasoning_compute_path),
|
||||||
|
"cleanup": sha256_file(cleanup_path),
|
||||||
|
},
|
||||||
|
"strongest_claim_allowed": (
|
||||||
|
(
|
||||||
|
"current private GCP staging parity, ID-free Leo reasoning, cleanup, and zero post-snapshot VPS delta verified"
|
||||||
|
if delta_details.get("delta_detected") is False
|
||||||
|
else "private GCP staging parity at the declared snapshot boundary, ID-free Leo reasoning, cleanup, and explicit post-snapshot VPS delta verified"
|
||||||
|
)
|
||||||
|
if not failed
|
||||||
|
else "lifecycle incomplete; failed checks must be repaired"
|
||||||
|
),
|
||||||
|
"not_proven": ["production cutover", "ongoing replication", "Telegram delivery"],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--source-receipt", required=True, type=Path)
|
||||||
|
parser.add_argument("--current-source-receipt", required=True, type=Path)
|
||||||
|
parser.add_argument("--source-delta-receipt", required=True, type=Path)
|
||||||
|
parser.add_argument("--restore-receipt", required=True, type=Path)
|
||||||
|
parser.add_argument("--parity-receipt", required=True, type=Path)
|
||||||
|
parser.add_argument("--reasoning-receipt", required=True, type=Path)
|
||||||
|
parser.add_argument("--reasoning-compute-receipt", required=True, type=Path)
|
||||||
|
parser.add_argument("--cleanup-receipt", required=True, type=Path)
|
||||||
|
parser.add_argument("--output", required=True, type=Path)
|
||||||
|
parser.add_argument("--max-postflight-age-seconds", default=900.0, type=float)
|
||||||
|
parser.add_argument("--max-lifecycle-age-seconds", default=3600.0, type=float)
|
||||||
|
args = parser.parse_args()
|
||||||
|
try:
|
||||||
|
payload = verify_lifecycle(
|
||||||
|
source_path=args.source_receipt,
|
||||||
|
current_source_path=args.current_source_receipt,
|
||||||
|
source_delta_path=args.source_delta_receipt,
|
||||||
|
restore_path=args.restore_receipt,
|
||||||
|
parity_path=args.parity_receipt,
|
||||||
|
reasoning_path=args.reasoning_receipt,
|
||||||
|
reasoning_compute_path=args.reasoning_compute_receipt,
|
||||||
|
cleanup_path=args.cleanup_receipt,
|
||||||
|
max_postflight_age_seconds=args.max_postflight_age_seconds,
|
||||||
|
max_lifecycle_age_seconds=args.max_lifecycle_age_seconds,
|
||||||
|
)
|
||||||
|
except (OSError, ValueError, KeyError, TypeError) as exc:
|
||||||
|
payload = {
|
||||||
|
"artifact": "gcp_canonical_snapshot_lifecycle_verification",
|
||||||
|
"generated_at_utc": datetime.now(UTC).isoformat(),
|
||||||
|
"status": "fail",
|
||||||
|
"required_tier": "T3_live_private_gcp_staging",
|
||||||
|
"failed_checks": ["receipt_load_or_validation_error"],
|
||||||
|
"error": str(exc),
|
||||||
|
"strongest_claim_allowed": "lifecycle incomplete; receipts could not be validated",
|
||||||
|
}
|
||||||
|
write_private_json(args.output, payload)
|
||||||
|
print(json.dumps(payload, indent=2, sort_keys=True))
|
||||||
|
return 0 if payload.get("status") == "pass" else 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
|
|
@ -11,6 +11,11 @@ from datetime import UTC, datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
|
try:
|
||||||
|
from .private_receipt_io import write_private_json
|
||||||
|
except ImportError: # pragma: no cover - direct script execution
|
||||||
|
from private_receipt_io import write_private_json
|
||||||
|
|
||||||
SINGLETON_KINDS = {
|
SINGLETON_KINDS = {
|
||||||
"identity",
|
"identity",
|
||||||
"schemas",
|
"schemas",
|
||||||
|
|
@ -33,6 +38,14 @@ def canonical_hash(value: Any) -> str:
|
||||||
return hashlib.sha256(payload.encode()).hexdigest()
|
return hashlib.sha256(payload.encode()).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def file_sha256(path: Path) -> str:
|
||||||
|
digest = hashlib.sha256()
|
||||||
|
with path.open("rb") as handle:
|
||||||
|
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||||
|
digest.update(chunk)
|
||||||
|
return digest.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
def load_manifest(path: Path) -> dict[str, Any]:
|
def load_manifest(path: Path) -> dict[str, Any]:
|
||||||
singleton: dict[str, dict[str, Any]] = {}
|
singleton: dict[str, dict[str, Any]] = {}
|
||||||
tables: dict[str, dict[str, Any]] = {}
|
tables: dict[str, dict[str, Any]] = {}
|
||||||
|
|
@ -295,6 +308,9 @@ def main() -> int:
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
try:
|
try:
|
||||||
|
source_manifest_sha256 = file_sha256(args.source)
|
||||||
|
target_manifest_sha256 = file_sha256(args.target)
|
||||||
|
connectivity_proof_sha256 = file_sha256(args.connectivity_proof) if args.connectivity_proof else None
|
||||||
source = load_manifest(args.source)
|
source = load_manifest(args.source)
|
||||||
target = load_manifest(args.target)
|
target = load_manifest(args.target)
|
||||||
problems, details = compare_manifests(
|
problems, details = compare_manifests(
|
||||||
|
|
@ -316,18 +332,26 @@ def main() -> int:
|
||||||
problems = [f"manifest_error={exc}"]
|
problems = [f"manifest_error={exc}"]
|
||||||
details = {}
|
details = {}
|
||||||
connectivity = {"required": args.scope == "gcp_staging", "status": "not_checked"}
|
connectivity = {"required": args.scope == "gcp_staging", "status": "not_checked"}
|
||||||
|
source_manifest_sha256 = None
|
||||||
|
target_manifest_sha256 = None
|
||||||
|
connectivity_proof_sha256 = None
|
||||||
error = str(exc)
|
error = str(exc)
|
||||||
|
|
||||||
|
completed_at_utc = datetime.now(UTC).isoformat()
|
||||||
payload = {
|
payload = {
|
||||||
"artifact": "canonical_postgres_parity_verification",
|
"artifact": "canonical_postgres_parity_verification",
|
||||||
"generated_at_utc": datetime.now(UTC).isoformat(),
|
"generated_at_utc": completed_at_utc,
|
||||||
|
"completed_at_utc": completed_at_utc,
|
||||||
"scope": args.scope,
|
"scope": args.scope,
|
||||||
"source_manifest": str(args.source),
|
"source_manifest": str(args.source),
|
||||||
|
"source_manifest_sha256": source_manifest_sha256,
|
||||||
"target_manifest": str(args.target),
|
"target_manifest": str(args.target),
|
||||||
|
"target_manifest_sha256": target_manifest_sha256,
|
||||||
"status": status,
|
"status": status,
|
||||||
"problems": problems,
|
"problems": problems,
|
||||||
"details": details,
|
"details": details,
|
||||||
"private_connectivity": connectivity,
|
"private_connectivity": connectivity,
|
||||||
|
"connectivity_proof_sha256": connectivity_proof_sha256,
|
||||||
"error": error,
|
"error": error,
|
||||||
"not_proven_by_this_artifact": [
|
"not_proven_by_this_artifact": [
|
||||||
"GCP staging Leo composition replay",
|
"GCP staging Leo composition replay",
|
||||||
|
|
@ -335,8 +359,7 @@ def main() -> int:
|
||||||
"production cutover",
|
"production cutover",
|
||||||
],
|
],
|
||||||
}
|
}
|
||||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
write_private_json(args.output, payload)
|
||||||
args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
|
|
||||||
print(json.dumps(payload, indent=2, sort_keys=True))
|
print(json.dumps(payload, indent=2, sort_keys=True))
|
||||||
return 0 if status == "pass" else 1
|
return 0 if status == "pass" else 1
|
||||||
|
|
||||||
|
|
|
||||||
284
ops/verify_vps_canonical_snapshot_delta.py
Normal file
284
ops/verify_vps_canonical_snapshot_delta.py
Normal file
|
|
@ -0,0 +1,284 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Bind a postflight VPS manifest to a captured snapshot and report exact deltas."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import stat
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
try:
|
||||||
|
from .capture_vps_canonical_postgres_snapshot import (
|
||||||
|
SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
|
||||||
|
build_provenance_binding,
|
||||||
|
)
|
||||||
|
from .private_receipt_io import write_private_json
|
||||||
|
from .verify_postgres_parity_manifest import SINGLETON_KINDS, canonical_hash, file_sha256, load_manifest
|
||||||
|
except ImportError: # pragma: no cover - direct script execution
|
||||||
|
from capture_vps_canonical_postgres_snapshot import SOURCE_SNAPSHOT_RECEIPT_SCHEMA, build_provenance_binding
|
||||||
|
from private_receipt_io import write_private_json
|
||||||
|
from verify_postgres_parity_manifest import SINGLETON_KINDS, canonical_hash, file_sha256, load_manifest
|
||||||
|
|
||||||
|
DELTA_SCHEMA = "livingip.vpsCanonicalSnapshotDelta.v1"
|
||||||
|
SHA256_RE = re.compile(r"[0-9a-f]{64}\Z")
|
||||||
|
|
||||||
|
|
||||||
|
def service_healthy(value: Any) -> bool:
|
||||||
|
if not isinstance(value, dict):
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
main_pid = int(value.get("MainPID") or 0)
|
||||||
|
restarts = int(value.get("NRestarts") or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return False
|
||||||
|
return (
|
||||||
|
value.get("ActiveState") == "active" and value.get("SubState") == "running" and main_pid > 0 and restarts >= 0
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def regular_file(path: Path, label: str) -> Path:
|
||||||
|
try:
|
||||||
|
mode = path.lstat().st_mode
|
||||||
|
except OSError as exc:
|
||||||
|
raise ValueError(f"{label} is unavailable: {exc}") from exc
|
||||||
|
if not stat.S_ISREG(mode) or path.is_symlink():
|
||||||
|
raise ValueError(f"{label} must be a regular non-symlink file")
|
||||||
|
return path.resolve()
|
||||||
|
|
||||||
|
|
||||||
|
def load_json(path: Path, label: str) -> dict[str, Any]:
|
||||||
|
path = regular_file(path, label)
|
||||||
|
try:
|
||||||
|
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise ValueError(f"{label} is invalid JSON") from exc
|
||||||
|
if not isinstance(payload, dict):
|
||||||
|
raise ValueError(f"{label} must be a JSON object")
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
def parse_timestamp(value: Any, label: str) -> datetime:
|
||||||
|
text = str(value or "")
|
||||||
|
if text.endswith("Z"):
|
||||||
|
text = text[:-1] + "+00:00"
|
||||||
|
try:
|
||||||
|
parsed = datetime.fromisoformat(text)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise ValueError(f"{label} is not an ISO-8601 timestamp") from exc
|
||||||
|
if parsed.tzinfo is None:
|
||||||
|
raise ValueError(f"{label} must include a timezone")
|
||||||
|
return parsed.astimezone(UTC)
|
||||||
|
|
||||||
|
|
||||||
|
def validate_capture(
|
||||||
|
receipt_path: Path,
|
||||||
|
manifest_path: Path,
|
||||||
|
*,
|
||||||
|
expected_authorization_ref: str,
|
||||||
|
label: str,
|
||||||
|
) -> tuple[dict[str, Any], dict[str, Any]]:
|
||||||
|
receipt = load_json(receipt_path, f"{label} receipt")
|
||||||
|
manifest_path = regular_file(manifest_path, f"{label} manifest")
|
||||||
|
manifest = load_manifest(manifest_path)
|
||||||
|
snapshot = receipt.get("snapshot") or {}
|
||||||
|
source = receipt.get("source") or {}
|
||||||
|
dump = receipt.get("dump") or {}
|
||||||
|
manifest_receipt = receipt.get("manifest") or {}
|
||||||
|
source_context = receipt.get("source_context") or {}
|
||||||
|
source_service = receipt.get("source_service") or {}
|
||||||
|
binding = receipt.get("provenance_binding") or {}
|
||||||
|
expected_binding = build_provenance_binding(
|
||||||
|
run_id=str(receipt.get("run_id") or ""),
|
||||||
|
authorization_ref=str(receipt.get("capture_authorization_ref") or ""),
|
||||||
|
source=source,
|
||||||
|
snapshot=snapshot,
|
||||||
|
dump_sha256=str(dump.get("sha256") or ""),
|
||||||
|
manifest_sql_sha256=str(manifest_receipt.get("manifest_sql_sha256") or ""),
|
||||||
|
source_manifest_sha256=file_sha256(manifest_path),
|
||||||
|
source_context_sha256=str(source_context.get("sha256") or ""),
|
||||||
|
)
|
||||||
|
total_rows = sum(int(row["row_count"]) for row in manifest["tables"].values())
|
||||||
|
checks = {
|
||||||
|
"schema": receipt.get("schema") == SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
|
||||||
|
"receipt_version": receipt.get("receipt_version") == 2,
|
||||||
|
"status": receipt.get("status") == "pass",
|
||||||
|
"authorization": receipt.get("capture_authorization_ref") == expected_authorization_ref,
|
||||||
|
"snapshot_exported": receipt.get("snapshot_exported") is True,
|
||||||
|
"service_unchanged": receipt.get("service_unchanged") is True,
|
||||||
|
"source_service_healthy": source_service.get("unchanged") is True
|
||||||
|
and service_healthy(source_service.get("before"))
|
||||||
|
and source_service.get("before") == source_service.get("after"),
|
||||||
|
"source_not_mutated": receipt.get("production_db_mutated") is False,
|
||||||
|
"telegram_not_sent": receipt.get("telegram_send_attempted") is False,
|
||||||
|
"manifest_hash": manifest_receipt.get("sha256") == file_sha256(manifest_path),
|
||||||
|
"manifest_table_count": manifest_receipt.get("table_count") == len(manifest["tables"]),
|
||||||
|
"manifest_total_rows": manifest_receipt.get("total_rows") == total_rows,
|
||||||
|
"manifest_read_only": manifest["singleton"]["identity"].get("transaction_read_only") == "on",
|
||||||
|
"manifest_database": manifest["singleton"]["identity"].get("database") == source.get("database"),
|
||||||
|
"binding_payload": binding.get("payload") == expected_binding["payload"],
|
||||||
|
"binding_sha256": binding.get("sha256") == expected_binding["sha256"],
|
||||||
|
"binding_algorithm": binding.get("algorithm") == "sha256",
|
||||||
|
"dump_sha256": bool(SHA256_RE.fullmatch(str(dump.get("sha256") or ""))),
|
||||||
|
"manifest_sql_sha256": bool(SHA256_RE.fullmatch(str(manifest_receipt.get("manifest_sql_sha256") or ""))),
|
||||||
|
"source_context_sha256": bool(SHA256_RE.fullmatch(str(source_context.get("sha256") or ""))),
|
||||||
|
"snapshot_identity": all(
|
||||||
|
bool(snapshot.get(field))
|
||||||
|
for field in ("captured_at_utc", "exported_snapshot_id", "txid_snapshot", "wal_lsn", "system_identifier")
|
||||||
|
),
|
||||||
|
}
|
||||||
|
failed = sorted(name for name, passed in checks.items() if not passed)
|
||||||
|
if failed:
|
||||||
|
raise ValueError(f"{label} capture failed validation: " + ", ".join(failed))
|
||||||
|
return receipt, manifest
|
||||||
|
|
||||||
|
|
||||||
|
def manifest_fingerprint(manifest: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
tables = {
|
||||||
|
name: {"row_count": int(row["row_count"]), "rowset_md5": row.get("rowset_md5")}
|
||||||
|
for name, row in sorted(manifest["tables"].items())
|
||||||
|
}
|
||||||
|
structures = {
|
||||||
|
kind: canonical_hash(manifest["singleton"][kind].get("items", []))
|
||||||
|
for kind in sorted(SINGLETON_KINDS - {"identity"})
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
"table_count": len(tables),
|
||||||
|
"total_rows": sum(row["row_count"] for row in tables.values()),
|
||||||
|
"table_fingerprint_sha256": canonical_hash(tables),
|
||||||
|
"structure_fingerprint_sha256": canonical_hash(structures),
|
||||||
|
"tables": tables,
|
||||||
|
"structures": structures,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def verify_delta(
|
||||||
|
*,
|
||||||
|
baseline_receipt_path: Path,
|
||||||
|
baseline_manifest_path: Path,
|
||||||
|
baseline_authorization_ref: str,
|
||||||
|
current_receipt_path: Path,
|
||||||
|
current_manifest_path: Path,
|
||||||
|
current_authorization_ref: str,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
baseline_receipt, baseline_manifest = validate_capture(
|
||||||
|
baseline_receipt_path,
|
||||||
|
baseline_manifest_path,
|
||||||
|
expected_authorization_ref=baseline_authorization_ref,
|
||||||
|
label="baseline",
|
||||||
|
)
|
||||||
|
current_receipt, current_manifest = validate_capture(
|
||||||
|
current_receipt_path,
|
||||||
|
current_manifest_path,
|
||||||
|
expected_authorization_ref=current_authorization_ref,
|
||||||
|
label="current",
|
||||||
|
)
|
||||||
|
baseline_snapshot = baseline_receipt["snapshot"]
|
||||||
|
current_snapshot = current_receipt["snapshot"]
|
||||||
|
baseline_time = parse_timestamp(baseline_snapshot["captured_at_utc"], "baseline snapshot time")
|
||||||
|
current_time = parse_timestamp(current_snapshot["captured_at_utc"], "current snapshot time")
|
||||||
|
if current_time <= baseline_time:
|
||||||
|
raise ValueError("current source capture must be newer than the restored baseline capture")
|
||||||
|
if current_receipt.get("source") != baseline_receipt.get("source"):
|
||||||
|
raise ValueError("current source identity does not match the restored baseline source")
|
||||||
|
if current_snapshot.get("system_identifier") != baseline_snapshot.get("system_identifier"):
|
||||||
|
raise ValueError("current source PostgreSQL system identifier changed")
|
||||||
|
|
||||||
|
baseline = manifest_fingerprint(baseline_manifest)
|
||||||
|
current = manifest_fingerprint(current_manifest)
|
||||||
|
baseline_tables = baseline.pop("tables")
|
||||||
|
current_tables = current.pop("tables")
|
||||||
|
baseline_structures = baseline.pop("structures")
|
||||||
|
current_structures = current.pop("structures")
|
||||||
|
added_tables = sorted(set(current_tables) - set(baseline_tables))
|
||||||
|
removed_tables = sorted(set(baseline_tables) - set(current_tables))
|
||||||
|
changed_tables = [
|
||||||
|
{"table": table, "baseline": baseline_tables[table], "current": current_tables[table]}
|
||||||
|
for table in sorted(set(baseline_tables) & set(current_tables))
|
||||||
|
if baseline_tables[table] != current_tables[table]
|
||||||
|
]
|
||||||
|
changed_structures = [
|
||||||
|
{
|
||||||
|
"kind": kind,
|
||||||
|
"baseline_sha256": baseline_structures.get(kind),
|
||||||
|
"current_sha256": current_structures.get(kind),
|
||||||
|
}
|
||||||
|
for kind in sorted(set(baseline_structures) | set(current_structures))
|
||||||
|
if baseline_structures.get(kind) != current_structures.get(kind)
|
||||||
|
]
|
||||||
|
delta_detected = bool(added_tables or removed_tables or changed_tables or changed_structures)
|
||||||
|
return {
|
||||||
|
"artifact": "vps_canonical_snapshot_postflight_delta",
|
||||||
|
"schema": DELTA_SCHEMA,
|
||||||
|
"generated_at_utc": datetime.now(UTC).isoformat(),
|
||||||
|
"status": "pass",
|
||||||
|
"baseline": {
|
||||||
|
"run_id": baseline_receipt["run_id"],
|
||||||
|
"capture_authorization_ref": baseline_authorization_ref,
|
||||||
|
"captured_at_utc": baseline_snapshot["captured_at_utc"],
|
||||||
|
"capture_receipt_sha256": file_sha256(baseline_receipt_path),
|
||||||
|
"manifest_sha256": file_sha256(baseline_manifest_path),
|
||||||
|
**baseline,
|
||||||
|
},
|
||||||
|
"current": {
|
||||||
|
"run_id": current_receipt["run_id"],
|
||||||
|
"capture_authorization_ref": current_authorization_ref,
|
||||||
|
"captured_at_utc": current_snapshot["captured_at_utc"],
|
||||||
|
"capture_receipt_sha256": file_sha256(current_receipt_path),
|
||||||
|
"manifest_sha256": file_sha256(current_manifest_path),
|
||||||
|
**current,
|
||||||
|
},
|
||||||
|
"delta": {
|
||||||
|
"delta_detected": delta_detected,
|
||||||
|
"added_tables": added_tables,
|
||||||
|
"removed_tables": removed_tables,
|
||||||
|
"changed_tables": changed_tables,
|
||||||
|
"changed_structures": changed_structures,
|
||||||
|
"total_row_delta": current["total_rows"] - baseline["total_rows"],
|
||||||
|
},
|
||||||
|
"strongest_claim_allowed": (
|
||||||
|
"restored snapshot still matches the newest postflight VPS capture"
|
||||||
|
if not delta_detected
|
||||||
|
else "restored snapshot parity is bounded to its capture; post-snapshot VPS deltas are explicit"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--baseline-receipt", required=True, type=Path)
|
||||||
|
parser.add_argument("--baseline-manifest", required=True, type=Path)
|
||||||
|
parser.add_argument("--baseline-authorization-ref", required=True)
|
||||||
|
parser.add_argument("--current-receipt", required=True, type=Path)
|
||||||
|
parser.add_argument("--current-manifest", required=True, type=Path)
|
||||||
|
parser.add_argument("--current-authorization-ref", required=True)
|
||||||
|
parser.add_argument("--output", required=True, type=Path)
|
||||||
|
args = parser.parse_args()
|
||||||
|
try:
|
||||||
|
payload = verify_delta(
|
||||||
|
baseline_receipt_path=args.baseline_receipt,
|
||||||
|
baseline_manifest_path=args.baseline_manifest,
|
||||||
|
baseline_authorization_ref=args.baseline_authorization_ref,
|
||||||
|
current_receipt_path=args.current_receipt,
|
||||||
|
current_manifest_path=args.current_manifest,
|
||||||
|
current_authorization_ref=args.current_authorization_ref,
|
||||||
|
)
|
||||||
|
except (OSError, ValueError, KeyError, TypeError, json.JSONDecodeError) as exc:
|
||||||
|
payload = {
|
||||||
|
"artifact": "vps_canonical_snapshot_postflight_delta",
|
||||||
|
"schema": DELTA_SCHEMA,
|
||||||
|
"generated_at_utc": datetime.now(UTC).isoformat(),
|
||||||
|
"status": "fail",
|
||||||
|
"error": str(exc),
|
||||||
|
"strongest_claim_allowed": "post-snapshot VPS delta is not proven",
|
||||||
|
}
|
||||||
|
write_private_json(args.output, payload)
|
||||||
|
print(json.dumps(payload, indent=2, sort_keys=True))
|
||||||
|
return 0 if payload.get("status") == "pass" else 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
128
tests/test_attest_gcp_reasoning_compute.py
Normal file
128
tests/test_attest_gcp_reasoning_compute.py
Normal file
|
|
@ -0,0 +1,128 @@
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from ops import attest_gcp_reasoning_compute as attestation
|
||||||
|
|
||||||
|
|
||||||
|
def reasoning_receipt(path: Path) -> Path:
|
||||||
|
path.write_text(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"schema": attestation.REASONING_SCHEMA,
|
||||||
|
"status": "pass",
|
||||||
|
"generated_at_utc": "2026-07-15T01:00:00+00:00",
|
||||||
|
"completed_at_utc": "2026-07-15T01:01:00+00:00",
|
||||||
|
"target_database": "teleo_clone_current_test",
|
||||||
|
"source_compute": "teleo-prod-1",
|
||||||
|
"errors": [],
|
||||||
|
"posted_to_telegram": False,
|
||||||
|
"database_write_attempted": False,
|
||||||
|
},
|
||||||
|
sort_keys=True,
|
||||||
|
)
|
||||||
|
+ "\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def compute_identity() -> dict:
|
||||||
|
return {
|
||||||
|
"project": "teleo-501523",
|
||||||
|
"project_number": "785938879453",
|
||||||
|
"instance": "teleo-prod-1",
|
||||||
|
"instance_id": "123456789",
|
||||||
|
"zone": "europe-west6-a",
|
||||||
|
"zone_resource": "projects/123456789/zones/europe-west6-a",
|
||||||
|
"network_resource": "projects/123456789/networks/teleo-staging-net",
|
||||||
|
"private_ip": "10.61.0.2",
|
||||||
|
"service_account": "teleo-runtime@teleo-501523.iam.gserviceaccount.com",
|
||||||
|
"expected_private_network": "projects/teleo-501523/global/networks/teleo-staging-net",
|
||||||
|
"checks": {
|
||||||
|
"project": True,
|
||||||
|
"project_number": True,
|
||||||
|
"expected_network_project": True,
|
||||||
|
"instance": True,
|
||||||
|
"instance_id": True,
|
||||||
|
"zone": True,
|
||||||
|
"network": True,
|
||||||
|
"private_ip": True,
|
||||||
|
"service_account": True,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_attestation_binds_reasoning_hash_to_live_metadata(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
receipt_path = reasoning_receipt(tmp_path / "reasoning.json")
|
||||||
|
observed = {}
|
||||||
|
|
||||||
|
def fake_identity(project, instance, zone, network, *, timeout):
|
||||||
|
observed.update(
|
||||||
|
{
|
||||||
|
"project": project,
|
||||||
|
"instance": instance,
|
||||||
|
"zone": zone,
|
||||||
|
"network": network,
|
||||||
|
"timeout": timeout,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return compute_identity()
|
||||||
|
|
||||||
|
monkeypatch.setattr(attestation, "gce_compute_identity", fake_identity)
|
||||||
|
|
||||||
|
result = attestation.attest_reasoning_compute(
|
||||||
|
reasoning_receipt_path=receipt_path,
|
||||||
|
restore_run_id="gcp-restore-current-test123",
|
||||||
|
target_database="teleo_clone_current_test",
|
||||||
|
generated_at_utc="2026-07-15T01:01:05+00:00",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result["status"] == "pass"
|
||||||
|
assert result["checks"]["gce_metadata_identity"] is True
|
||||||
|
assert result["reasoning_receipt"]["sha256"] == attestation.sha256_file(receipt_path)
|
||||||
|
assert result["compute"]["instance_id"] == "123456789"
|
||||||
|
assert observed == {
|
||||||
|
"project": "teleo-501523",
|
||||||
|
"instance": "teleo-prod-1",
|
||||||
|
"zone": "europe-west6-a",
|
||||||
|
"network": "projects/teleo-501523/global/networks/teleo-staging-net",
|
||||||
|
"timeout": 10.0,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_attestation_rejects_claimed_wrong_compute_before_metadata(
|
||||||
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
receipt_path = reasoning_receipt(tmp_path / "reasoning.json")
|
||||||
|
payload = json.loads(receipt_path.read_text())
|
||||||
|
payload["source_compute"] = "untrusted-vm"
|
||||||
|
receipt_path.write_text(json.dumps(payload) + "\n", encoding="utf-8")
|
||||||
|
monkeypatch.setattr(
|
||||||
|
attestation,
|
||||||
|
"gce_compute_identity",
|
||||||
|
lambda *args, **kwargs: pytest.fail("metadata must not be queried"),
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="source_compute"):
|
||||||
|
attestation.attest_reasoning_compute(
|
||||||
|
reasoning_receipt_path=receipt_path,
|
||||||
|
restore_run_id="gcp-restore-current-test123",
|
||||||
|
target_database="teleo_clone_current_test",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_attestation_rejects_send_or_write_receipt(tmp_path: Path) -> None:
|
||||||
|
receipt_path = reasoning_receipt(tmp_path / "reasoning.json")
|
||||||
|
payload = json.loads(receipt_path.read_text())
|
||||||
|
payload["posted_to_telegram"] = True
|
||||||
|
payload["database_write_attempted"] = True
|
||||||
|
receipt_path.write_text(json.dumps(payload) + "\n", encoding="utf-8")
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match=r"no_database_write.*no_telegram_send"):
|
||||||
|
attestation.attest_reasoning_compute(
|
||||||
|
reasoning_receipt_path=receipt_path,
|
||||||
|
restore_run_id="gcp-restore-current-test123",
|
||||||
|
target_database="teleo_clone_current_test",
|
||||||
|
)
|
||||||
|
|
@ -1,8 +1,18 @@
|
||||||
|
import json
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
from ops.capture_vps_canonical_postgres_snapshot import (
|
from ops.capture_vps_canonical_postgres_snapshot import (
|
||||||
|
SAFE_AUTHORIZATION_REF_RE,
|
||||||
SAFE_SSH_TARGET_RE,
|
SAFE_SSH_TARGET_RE,
|
||||||
|
SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
|
||||||
|
build_provenance_binding,
|
||||||
|
canonical_sha256,
|
||||||
cleanup_failed_output,
|
cleanup_failed_output,
|
||||||
|
load_service_state,
|
||||||
|
load_snapshot_metadata,
|
||||||
|
parse_args,
|
||||||
remote_capture_script,
|
remote_capture_script,
|
||||||
toc_counts,
|
toc_counts,
|
||||||
)
|
)
|
||||||
|
|
@ -21,8 +31,14 @@ def test_remote_capture_uses_one_exported_read_only_snapshot() -> None:
|
||||||
assert "[0-9A-F]+-[0-9A-F]+-[0-9]+" in script
|
assert "[0-9A-F]+-[0-9A-F]+-[0-9]+" in script
|
||||||
assert '--snapshot="$snapshot_id"' in script
|
assert '--snapshot="$snapshot_id"' in script
|
||||||
assert '-v "snapshot_id=$snapshot_id"' in script
|
assert '-v "snapshot_id=$snapshot_id"' in script
|
||||||
|
assert "txid_current_snapshot()::text" in script
|
||||||
|
assert "pg_current_wal_lsn()::text" in script
|
||||||
|
assert "system_identifier::text from pg_control_system()" in script
|
||||||
|
assert "source-snapshot.json" in script
|
||||||
assert "--no-owner --no-acl" in script
|
assert "--no-owner --no-acl" in script
|
||||||
assert "cmp -s" in script
|
assert "cmp -s" in script
|
||||||
|
assert 'assert_service_healthy "$remote_root/service-before.txt"' in script
|
||||||
|
assert 'assert_service_healthy "$remote_root/service-after.txt"' in script
|
||||||
assert "systemctl restart" not in script
|
assert "systemctl restart" not in script
|
||||||
assert "docker restart" not in script
|
assert "docker restart" not in script
|
||||||
|
|
||||||
|
|
@ -62,8 +78,129 @@ def test_failure_cleanup_never_removes_preexisting_output(tmp_path: Path) -> Non
|
||||||
assert not created_by_run.exists()
|
assert not created_by_run.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_source_service_state_requires_healthy_unchanged_capable_fields(tmp_path: Path) -> None:
|
||||||
|
state_path = tmp_path / "service.txt"
|
||||||
|
state_path.write_text("ActiveState=active\nSubState=running\nMainPID=347406\nNRestarts=0\n")
|
||||||
|
|
||||||
|
assert load_service_state(state_path)["MainPID"] == "347406"
|
||||||
|
|
||||||
|
state_path.write_text("ActiveState=failed\nSubState=dead\nMainPID=0\nNRestarts=0\n")
|
||||||
|
with pytest.raises(RuntimeError, match="not active/running"):
|
||||||
|
load_service_state(state_path)
|
||||||
|
|
||||||
|
|
||||||
def test_ssh_target_rejects_open_ssh_option_injection() -> None:
|
def test_ssh_target_rejects_open_ssh_option_injection() -> None:
|
||||||
assert SAFE_SSH_TARGET_RE.fullmatch("root@77.42.65.182")
|
assert SAFE_SSH_TARGET_RE.fullmatch("root@77.42.65.182")
|
||||||
assert SAFE_SSH_TARGET_RE.fullmatch("teleo-staging-1.internal")
|
assert SAFE_SSH_TARGET_RE.fullmatch("teleo-staging-1.internal")
|
||||||
assert not SAFE_SSH_TARGET_RE.fullmatch("-oProxyCommand=malicious")
|
assert not SAFE_SSH_TARGET_RE.fullmatch("-oProxyCommand=malicious")
|
||||||
assert not SAFE_SSH_TARGET_RE.fullmatch("root@host command")
|
assert not SAFE_SSH_TARGET_RE.fullmatch("root@host command")
|
||||||
|
|
||||||
|
|
||||||
|
def test_source_snapshot_metadata_and_provenance_binding(tmp_path: Path) -> None:
|
||||||
|
snapshot_path = tmp_path / "source-snapshot.json"
|
||||||
|
snapshot_path.write_text(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"exported_snapshot_id": "00000003-000001A2-1",
|
||||||
|
"txid_snapshot": "100:120:105,111",
|
||||||
|
"wal_lsn": "0/16B6C50",
|
||||||
|
"system_identifier": "7649789040005668902",
|
||||||
|
"captured_at_utc": "2026-07-15T01:02:03.456789+00:00",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
snapshot = load_snapshot_metadata(snapshot_path)
|
||||||
|
binding = build_provenance_binding(
|
||||||
|
run_id="canonical-20260715",
|
||||||
|
authorization_ref="codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621",
|
||||||
|
source={
|
||||||
|
"ssh_target": "root@77.42.65.182",
|
||||||
|
"container": "teleo-pg",
|
||||||
|
"database": "teleo",
|
||||||
|
"service": "leoclean-gateway.service",
|
||||||
|
},
|
||||||
|
snapshot=snapshot,
|
||||||
|
dump_sha256="1" * 64,
|
||||||
|
manifest_sql_sha256="2" * 64,
|
||||||
|
source_manifest_sha256="3" * 64,
|
||||||
|
source_context_sha256="4" * 64,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert binding["algorithm"] == "sha256"
|
||||||
|
assert binding["payload"] == {
|
||||||
|
"receipt_schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
|
||||||
|
"run_id": "canonical-20260715",
|
||||||
|
"capture_authorization_ref": "codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621",
|
||||||
|
"source": {
|
||||||
|
"ssh_target": "root@77.42.65.182",
|
||||||
|
"container": "teleo-pg",
|
||||||
|
"database": "teleo",
|
||||||
|
"service": "leoclean-gateway.service",
|
||||||
|
},
|
||||||
|
"exported_snapshot_id": "00000003-000001A2-1",
|
||||||
|
"txid_snapshot": "100:120:105,111",
|
||||||
|
"wal_lsn": "0/16B6C50",
|
||||||
|
"source_system_identifier": "7649789040005668902",
|
||||||
|
"dump_sha256": "1" * 64,
|
||||||
|
"manifest_sql_sha256": "2" * 64,
|
||||||
|
"source_manifest_sha256": "3" * 64,
|
||||||
|
"source_context_sha256": "4" * 64,
|
||||||
|
}
|
||||||
|
assert binding["sha256"] == canonical_sha256(binding["payload"])
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("field", "value"),
|
||||||
|
[
|
||||||
|
("exported_snapshot_id", "unsafe snapshot"),
|
||||||
|
("txid_snapshot", "100:120:not-a-txid"),
|
||||||
|
("wal_lsn", "not-an-lsn"),
|
||||||
|
("system_identifier", "system-1"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_source_snapshot_metadata_rejects_invalid_identity(tmp_path: Path, field: str, value: str) -> None:
|
||||||
|
payload = {
|
||||||
|
"exported_snapshot_id": "00000003-000001A2-1",
|
||||||
|
"txid_snapshot": "100:120:",
|
||||||
|
"wal_lsn": "0/16B6C50",
|
||||||
|
"system_identifier": "7649789040005668902",
|
||||||
|
"captured_at_utc": "2026-07-15T01:02:03+00:00",
|
||||||
|
}
|
||||||
|
payload[field] = value
|
||||||
|
path = tmp_path / "source-snapshot.json"
|
||||||
|
path.write_text(json.dumps(payload))
|
||||||
|
|
||||||
|
with pytest.raises(RuntimeError, match=field):
|
||||||
|
load_snapshot_metadata(path)
|
||||||
|
|
||||||
|
|
||||||
|
def test_authorization_ref_is_required_and_safe_metadata(tmp_path: Path) -> None:
|
||||||
|
ssh_key = tmp_path / "id_ed25519"
|
||||||
|
ssh_key.write_text("test-only")
|
||||||
|
manifest = tmp_path / "manifest.sql"
|
||||||
|
manifest.write_text("select 1;\n")
|
||||||
|
base = [
|
||||||
|
"--execute",
|
||||||
|
"--ssh-target",
|
||||||
|
"root@77.42.65.182",
|
||||||
|
"--ssh-key",
|
||||||
|
str(ssh_key),
|
||||||
|
"--manifest",
|
||||||
|
str(manifest),
|
||||||
|
"--run-id",
|
||||||
|
"canonical-20260715",
|
||||||
|
"--output-dir",
|
||||||
|
str(tmp_path / "capture"),
|
||||||
|
]
|
||||||
|
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
parse_args(base)
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
parse_args([*base, "--authorization-ref", "unsafe authorization ref"])
|
||||||
|
|
||||||
|
authorization_ref = "codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621"
|
||||||
|
args = parse_args([*base, "--authorization-ref", authorization_ref])
|
||||||
|
assert args.authorization_ref == authorization_ref
|
||||||
|
assert SAFE_AUTHORIZATION_REF_RE.fullmatch(authorization_ref)
|
||||||
|
assert not SAFE_AUTHORIZATION_REF_RE.fullmatch("authorization\nforged")
|
||||||
|
|
|
||||||
|
|
@ -7,6 +7,7 @@ from pathlib import Path
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from ops import restore_gcp_generated_postgres_snapshot as restore
|
from ops import restore_gcp_generated_postgres_snapshot as restore
|
||||||
|
from ops.capture_vps_canonical_postgres_snapshot import build_provenance_binding
|
||||||
|
|
||||||
SERVICE_STATE = {
|
SERVICE_STATE = {
|
||||||
"ActiveState": "active",
|
"ActiveState": "active",
|
||||||
|
|
@ -15,6 +16,7 @@ SERVICE_STATE = {
|
||||||
"NRestarts": "0",
|
"NRestarts": "0",
|
||||||
"ExecMainStartTimestamp": "Thu 2026-07-09 07:00:04 UTC",
|
"ExecMainStartTimestamp": "Thu 2026-07-09 07:00:04 UTC",
|
||||||
}
|
}
|
||||||
|
AUTHORIZATION_REF = "codex-delegation:019f47af-f2cc-7c10-b928-a9283afa2621"
|
||||||
|
|
||||||
|
|
||||||
def manifest_rows(database: str, *, target: bool, row_count: int = 1) -> list[dict]:
|
def manifest_rows(database: str, *, target: bool, row_count: int = 1) -> list[dict]:
|
||||||
|
|
@ -94,6 +96,65 @@ def args_for(tmp_path: Path, **overrides) -> argparse.Namespace:
|
||||||
dump.write_bytes(b"PGDMPfixture")
|
dump.write_bytes(b"PGDMPfixture")
|
||||||
source_manifest = tmp_path / "source-manifest.jsonl"
|
source_manifest = tmp_path / "source-manifest.jsonl"
|
||||||
source_manifest.write_text(manifest_text("teleo", target=False), encoding="utf-8")
|
source_manifest.write_text(manifest_text("teleo", target=False), encoding="utf-8")
|
||||||
|
manifest_sql = Path("ops/postgres_parity_manifest.sql").resolve()
|
||||||
|
source_context_sha256 = "4" * 64
|
||||||
|
snapshot = {
|
||||||
|
"exported_snapshot_id": "00000003-000001A2-1",
|
||||||
|
"txid_snapshot": "100:120:105,111",
|
||||||
|
"wal_lsn": "0/16B6C50",
|
||||||
|
"system_identifier": "7649789040005668902",
|
||||||
|
"captured_at_utc": "2026-07-15T01:02:03.456789+00:00",
|
||||||
|
}
|
||||||
|
source = {
|
||||||
|
"ssh_target": "root@77.42.65.182",
|
||||||
|
"container": "teleo-pg",
|
||||||
|
"database": "teleo",
|
||||||
|
"service": "leoclean-gateway.service",
|
||||||
|
}
|
||||||
|
dump_sha256 = hashlib.sha256(dump.read_bytes()).hexdigest()
|
||||||
|
source_manifest_sha256 = hashlib.sha256(source_manifest.read_bytes()).hexdigest()
|
||||||
|
manifest_sql_sha256 = hashlib.sha256(manifest_sql.read_bytes()).hexdigest()
|
||||||
|
source_receipt = tmp_path / "source-receipt.json"
|
||||||
|
source_receipt.write_text(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"artifact": "vps_canonical_postgres_exported_snapshot",
|
||||||
|
"schema": restore.SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
|
||||||
|
"receipt_version": 2,
|
||||||
|
"generated_at_utc": "2026-07-15T01:02:04+00:00",
|
||||||
|
"status": "pass",
|
||||||
|
"run_id": "canonical-20260715",
|
||||||
|
"capture_authorization_ref": AUTHORIZATION_REF,
|
||||||
|
"source": source,
|
||||||
|
"snapshot_exported": True,
|
||||||
|
"snapshot": snapshot,
|
||||||
|
"dump": {"path": "/private/source.dump", "bytes": dump.stat().st_size, "sha256": dump_sha256},
|
||||||
|
"manifest": {
|
||||||
|
"path": "/private/source-manifest.jsonl",
|
||||||
|
"sha256": source_manifest_sha256,
|
||||||
|
"manifest_sql_sha256": manifest_sql_sha256,
|
||||||
|
"table_count": 1,
|
||||||
|
"total_rows": 1,
|
||||||
|
},
|
||||||
|
"source_context": {"path": "/private/source-context.txt", "sha256": source_context_sha256},
|
||||||
|
"source_service": {"before": SERVICE_STATE, "after": SERVICE_STATE, "unchanged": True},
|
||||||
|
"provenance_binding": build_provenance_binding(
|
||||||
|
run_id="canonical-20260715",
|
||||||
|
authorization_ref=AUTHORIZATION_REF,
|
||||||
|
source=source,
|
||||||
|
snapshot=snapshot,
|
||||||
|
dump_sha256=dump_sha256,
|
||||||
|
manifest_sql_sha256=manifest_sql_sha256,
|
||||||
|
source_manifest_sha256=source_manifest_sha256,
|
||||||
|
source_context_sha256=source_context_sha256,
|
||||||
|
),
|
||||||
|
"service_unchanged": True,
|
||||||
|
"production_db_mutated": False,
|
||||||
|
"telegram_send_attempted": False,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
values = {
|
values = {
|
||||||
"operation": "restore",
|
"operation": "restore",
|
||||||
"execute": True,
|
"execute": True,
|
||||||
|
|
@ -101,15 +162,26 @@ def args_for(tmp_path: Path, **overrides) -> argparse.Namespace:
|
||||||
"run_id": "gcp-restore-test12345",
|
"run_id": "gcp-restore-test12345",
|
||||||
"host": restore.DEFAULT_HOST,
|
"host": restore.DEFAULT_HOST,
|
||||||
"project": restore.DEFAULT_PROJECT,
|
"project": restore.DEFAULT_PROJECT,
|
||||||
|
"cloudsql_instance": restore.DEFAULT_INSTANCE,
|
||||||
|
"expected_private_network": restore.DEFAULT_PRIVATE_NETWORK,
|
||||||
|
"expected_compute_instance": restore.DEFAULT_COMPUTE_INSTANCE,
|
||||||
|
"expected_compute_zone": restore.DEFAULT_COMPUTE_ZONE,
|
||||||
|
"expected_source_ssh_target": restore.DEFAULT_SOURCE_SSH_TARGET,
|
||||||
|
"expected_source_container": restore.DEFAULT_SOURCE_CONTAINER,
|
||||||
|
"expected_source_database": restore.DEFAULT_SOURCE_DATABASE,
|
||||||
|
"expected_source_service": restore.DEFAULT_SOURCE_SERVICE,
|
||||||
"password_secret": restore.DEFAULT_SECRET,
|
"password_secret": restore.DEFAULT_SECRET,
|
||||||
"service": restore.DEFAULT_SERVICE,
|
"service": restore.DEFAULT_SERVICE,
|
||||||
"command_timeout": 2.0,
|
"command_timeout": 2.0,
|
||||||
"psql_bin": "psql",
|
"psql_bin": "psql",
|
||||||
"run_root": tmp_path / "runs",
|
"run_root": tmp_path / "runs",
|
||||||
"dump": dump,
|
"dump": dump,
|
||||||
"expected_dump_sha256": hashlib.sha256(dump.read_bytes()).hexdigest(),
|
"expected_dump_sha256": dump_sha256,
|
||||||
"source_manifest": source_manifest,
|
"source_manifest": source_manifest,
|
||||||
"manifest_sql": Path("ops/postgres_parity_manifest.sql").resolve(),
|
"source_receipt": source_receipt,
|
||||||
|
"expected_source_receipt_sha256": hashlib.sha256(source_receipt.read_bytes()).hexdigest(),
|
||||||
|
"expected_capture_authorization_ref": AUTHORIZATION_REF,
|
||||||
|
"manifest_sql": manifest_sql,
|
||||||
"pg_restore_bin": "pg_restore",
|
"pg_restore_bin": "pg_restore",
|
||||||
"restore_timeout": 5.0,
|
"restore_timeout": 5.0,
|
||||||
"manifest_timeout": 5.0,
|
"manifest_timeout": 5.0,
|
||||||
|
|
@ -128,6 +200,54 @@ def install_restore_fakes(
|
||||||
state = {"exists": False, "dropped": False}
|
state = {"exists": False, "dropped": False}
|
||||||
monkeypatch.setattr(restore.os, "geteuid", lambda: 0)
|
monkeypatch.setattr(restore.os, "geteuid", lambda: 0)
|
||||||
monkeypatch.setattr(restore, "service_state", lambda *_args, **_kwargs: dict(SERVICE_STATE))
|
monkeypatch.setattr(restore, "service_state", lambda *_args, **_kwargs: dict(SERVICE_STATE))
|
||||||
|
monkeypatch.setattr(
|
||||||
|
restore,
|
||||||
|
"gce_compute_identity",
|
||||||
|
lambda project, instance, zone, expected_private_network, **_kwargs: {
|
||||||
|
"project": project,
|
||||||
|
"project_number": "785938879453",
|
||||||
|
"instance": instance,
|
||||||
|
"instance_id": "123456789",
|
||||||
|
"zone": zone,
|
||||||
|
"expected_private_network": expected_private_network,
|
||||||
|
"private_ip": "10.61.0.2",
|
||||||
|
"network_resource": "projects/123456789/networks/teleo-staging-net",
|
||||||
|
"service_account": "teleo-runtime@teleo-501523.iam.gserviceaccount.com",
|
||||||
|
"checks": {
|
||||||
|
"project": True,
|
||||||
|
"project_number": True,
|
||||||
|
"expected_network_project": True,
|
||||||
|
"instance": True,
|
||||||
|
"instance_id": True,
|
||||||
|
"zone": True,
|
||||||
|
"network": True,
|
||||||
|
"private_ip": True,
|
||||||
|
"service_account": True,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
restore,
|
||||||
|
"cloudsql_control_plane_state",
|
||||||
|
lambda project, instance, host, expected_private_network, **_kwargs: {
|
||||||
|
"project": project,
|
||||||
|
"instance": instance,
|
||||||
|
"private_addresses": [host],
|
||||||
|
"expected_private_host": host,
|
||||||
|
"private_network": expected_private_network,
|
||||||
|
"expected_private_network": expected_private_network,
|
||||||
|
"public_ip_disabled": True,
|
||||||
|
"checks": {
|
||||||
|
"instance": True,
|
||||||
|
"state": True,
|
||||||
|
"postgres_16": True,
|
||||||
|
"public_ip_disabled": True,
|
||||||
|
"private_network": True,
|
||||||
|
"expected_private_host": True,
|
||||||
|
"no_non_private_address": True,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
)
|
||||||
monkeypatch.setattr(restore, "resolve_password", lambda *_args, **_kwargs: "private-password")
|
monkeypatch.setattr(restore, "resolve_password", lambda *_args, **_kwargs: "private-password")
|
||||||
monkeypatch.setattr(
|
monkeypatch.setattr(
|
||||||
restore,
|
restore,
|
||||||
|
|
@ -179,7 +299,10 @@ def test_restore_success_retains_only_exact_parity_clone(
|
||||||
assert result["phase"] == "retained_for_no_send_replay"
|
assert result["phase"] == "retained_for_no_send_replay"
|
||||||
assert result["parity"]["problems"] == []
|
assert result["parity"]["problems"] == []
|
||||||
assert result["parity"]["details"]["source_total_rows"] == 1
|
assert result["parity"]["details"]["source_total_rows"] == 1
|
||||||
|
assert result["source"]["capture_receipt"]["schema"] == restore.SOURCE_SNAPSHOT_RECEIPT_SCHEMA
|
||||||
|
assert result["source"]["capture_receipt"]["provenance_binding_sha256"]
|
||||||
assert result["target"]["connectivity"]["private_connectivity"] is True
|
assert result["target"]["connectivity"]["private_connectivity"] is True
|
||||||
|
assert result["target"]["cloudsql"]["public_ip_disabled"] is True
|
||||||
assert result["target"]["connectivity"]["ssl"] is True
|
assert result["target"]["connectivity"]["ssl"] is True
|
||||||
assert result["live_service"]["unchanged"] is True
|
assert result["live_service"]["unchanged"] is True
|
||||||
assert result["cleanup"]["clone_database_remaining"] == 1
|
assert result["cleanup"]["clone_database_remaining"] == 1
|
||||||
|
|
@ -187,6 +310,15 @@ def test_restore_success_retains_only_exact_parity_clone(
|
||||||
run_dir = args.run_root / args.run_id
|
run_dir = args.run_root / args.run_id
|
||||||
assert (run_dir / "target-manifest.jsonl").stat().st_mode & 0o777 == 0o600
|
assert (run_dir / "target-manifest.jsonl").stat().st_mode & 0o777 == 0o600
|
||||||
assert (run_dir / "restore-receipt.json").stat().st_mode & 0o777 == 0o600
|
assert (run_dir / "restore-receipt.json").stat().st_mode & 0o777 == 0o600
|
||||||
|
connectivity_path = run_dir / "gcp-private-connectivity.json"
|
||||||
|
assert connectivity_path.stat().st_mode & 0o777 == 0o600
|
||||||
|
connectivity = json.loads(connectivity_path.read_text())
|
||||||
|
assert connectivity["target_database"] == args.target_db
|
||||||
|
assert connectivity["source_compute"] == restore.DEFAULT_COMPUTE_INSTANCE
|
||||||
|
assert connectivity["public_ip_disabled"] is True
|
||||||
|
assert (
|
||||||
|
result["target"]["connectivity_proof"]["sha256"] == hashlib.sha256(connectivity_path.read_bytes()).hexdigest()
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_parity_failure_drops_generated_clone_and_receipts_both_failures(
|
def test_parity_failure_drops_generated_clone_and_receipts_both_failures(
|
||||||
|
|
@ -234,6 +366,79 @@ def test_create_command_failure_still_drops_a_committed_clone(
|
||||||
assert state == {"exists": False, "dropped": True}
|
assert state == {"exists": False, "dropped": True}
|
||||||
|
|
||||||
|
|
||||||
|
def test_restore_rejects_manifest_not_bound_to_source_receipt_before_credentials(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
args = args_for(tmp_path)
|
||||||
|
install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True))
|
||||||
|
receipt = json.loads(args.source_receipt.read_text())
|
||||||
|
receipt["manifest"]["sha256"] = "0" * 64
|
||||||
|
args.source_receipt.write_text(json.dumps(receipt))
|
||||||
|
args.expected_source_receipt_sha256 = hashlib.sha256(args.source_receipt.read_bytes()).hexdigest()
|
||||||
|
credential_called = False
|
||||||
|
|
||||||
|
def unexpected_credential(*_args, **_kwargs):
|
||||||
|
nonlocal credential_called
|
||||||
|
credential_called = True
|
||||||
|
return "private-password"
|
||||||
|
|
||||||
|
monkeypatch.setattr(restore, "resolve_password", unexpected_credential)
|
||||||
|
|
||||||
|
result = restore.run_restore(args)
|
||||||
|
|
||||||
|
assert result["status"] == "fail"
|
||||||
|
assert result["error"]["phase"] == "validation"
|
||||||
|
assert "source_manifest_sha256" in result["error"]["message"]
|
||||||
|
assert credential_called is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_restore_rejects_unexpected_capture_authorization_before_credentials(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
args = args_for(tmp_path, expected_capture_authorization_ref="codex-delegation:another-authorized-run")
|
||||||
|
install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True))
|
||||||
|
credential_called = False
|
||||||
|
|
||||||
|
def unexpected_credential(*_args, **_kwargs):
|
||||||
|
nonlocal credential_called
|
||||||
|
credential_called = True
|
||||||
|
return "private-password"
|
||||||
|
|
||||||
|
monkeypatch.setattr(restore, "resolve_password", unexpected_credential)
|
||||||
|
|
||||||
|
result = restore.run_restore(args)
|
||||||
|
|
||||||
|
assert result["status"] == "fail"
|
||||||
|
assert result["error"]["phase"] == "validation"
|
||||||
|
assert "capture_authorization_ref" in result["error"]["message"]
|
||||||
|
assert credential_called is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_restore_rejects_noncanonical_source_identity_before_credentials(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
args = args_for(tmp_path, expected_source_container="different-postgres")
|
||||||
|
install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True))
|
||||||
|
credential_called = False
|
||||||
|
|
||||||
|
def unexpected_credential(*_args, **_kwargs):
|
||||||
|
nonlocal credential_called
|
||||||
|
credential_called = True
|
||||||
|
return "private-password"
|
||||||
|
|
||||||
|
monkeypatch.setattr(restore, "resolve_password", unexpected_credential)
|
||||||
|
|
||||||
|
result = restore.run_restore(args)
|
||||||
|
|
||||||
|
assert result["status"] == "fail"
|
||||||
|
assert result["error"]["phase"] == "validation"
|
||||||
|
assert "source_identity" in result["error"]["message"]
|
||||||
|
assert credential_called is False
|
||||||
|
|
||||||
|
|
||||||
def test_changed_live_service_turns_successful_restore_into_cleanup(
|
def test_changed_live_service_turns_successful_restore_into_cleanup(
|
||||||
tmp_path: Path,
|
tmp_path: Path,
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
|
@ -283,6 +488,14 @@ def test_cleanup_is_bound_to_passing_restore_receipt(
|
||||||
"run_id",
|
"run_id",
|
||||||
"host",
|
"host",
|
||||||
"project",
|
"project",
|
||||||
|
"cloudsql_instance",
|
||||||
|
"expected_private_network",
|
||||||
|
"expected_compute_instance",
|
||||||
|
"expected_compute_zone",
|
||||||
|
"expected_source_ssh_target",
|
||||||
|
"expected_source_container",
|
||||||
|
"expected_source_database",
|
||||||
|
"expected_source_service",
|
||||||
"password_secret",
|
"password_secret",
|
||||||
"service",
|
"service",
|
||||||
"command_timeout",
|
"command_timeout",
|
||||||
|
|
@ -302,6 +515,106 @@ def test_cleanup_is_bound_to_passing_restore_receipt(
|
||||||
assert state == {"exists": False, "dropped": True}
|
assert state == {"exists": False, "dropped": True}
|
||||||
|
|
||||||
|
|
||||||
|
def test_cleanup_rejects_target_topology_drift_before_credentials(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
args = args_for(tmp_path)
|
||||||
|
state = install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True))
|
||||||
|
assert restore.run_restore(args)["status"] == "pass"
|
||||||
|
cleanup_args = argparse.Namespace(
|
||||||
|
**{
|
||||||
|
key: getattr(args, key)
|
||||||
|
for key in (
|
||||||
|
"target_db",
|
||||||
|
"run_id",
|
||||||
|
"host",
|
||||||
|
"project",
|
||||||
|
"cloudsql_instance",
|
||||||
|
"expected_private_network",
|
||||||
|
"expected_compute_instance",
|
||||||
|
"expected_compute_zone",
|
||||||
|
"expected_source_ssh_target",
|
||||||
|
"expected_source_container",
|
||||||
|
"expected_source_database",
|
||||||
|
"expected_source_service",
|
||||||
|
"password_secret",
|
||||||
|
"service",
|
||||||
|
"command_timeout",
|
||||||
|
"psql_bin",
|
||||||
|
"run_root",
|
||||||
|
)
|
||||||
|
},
|
||||||
|
operation="cleanup",
|
||||||
|
execute=True,
|
||||||
|
)
|
||||||
|
cleanup_args.host = "10.61.0.4"
|
||||||
|
credential_called = False
|
||||||
|
|
||||||
|
def unexpected_credential(*_args, **_kwargs):
|
||||||
|
nonlocal credential_called
|
||||||
|
credential_called = True
|
||||||
|
return "private-password"
|
||||||
|
|
||||||
|
monkeypatch.setattr(restore, "resolve_password", unexpected_credential)
|
||||||
|
|
||||||
|
with pytest.raises(restore.RestoreError, match="execution contract"):
|
||||||
|
restore.run_cleanup(cleanup_args)
|
||||||
|
|
||||||
|
assert credential_called is False
|
||||||
|
assert state == {"exists": True, "dropped": False}
|
||||||
|
|
||||||
|
|
||||||
|
def test_cleanup_retains_failure_receipt_when_post_drop_readback_fails(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
args = args_for(tmp_path)
|
||||||
|
state = install_restore_fakes(monkeypatch, target_manifest=manifest_text(args.target_db, target=True))
|
||||||
|
assert restore.run_restore(args)["status"] == "pass"
|
||||||
|
cleanup_args = argparse.Namespace(
|
||||||
|
**{
|
||||||
|
key: getattr(args, key)
|
||||||
|
for key in (
|
||||||
|
"target_db",
|
||||||
|
"run_id",
|
||||||
|
"host",
|
||||||
|
"project",
|
||||||
|
"cloudsql_instance",
|
||||||
|
"expected_private_network",
|
||||||
|
"expected_compute_instance",
|
||||||
|
"expected_compute_zone",
|
||||||
|
"expected_source_ssh_target",
|
||||||
|
"expected_source_container",
|
||||||
|
"expected_source_database",
|
||||||
|
"expected_source_service",
|
||||||
|
"password_secret",
|
||||||
|
"service",
|
||||||
|
"command_timeout",
|
||||||
|
"psql_bin",
|
||||||
|
"run_root",
|
||||||
|
)
|
||||||
|
},
|
||||||
|
operation="cleanup",
|
||||||
|
execute=True,
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
restore,
|
||||||
|
"rollback_database_state",
|
||||||
|
lambda *_args, **_kwargs: (_ for _ in ()).throw(restore.RestoreError("rollback readback unavailable")),
|
||||||
|
)
|
||||||
|
|
||||||
|
result = restore.run_cleanup(cleanup_args)
|
||||||
|
|
||||||
|
assert result["status"] == "fail"
|
||||||
|
assert result["error"]["phase"] == "rollback_readback"
|
||||||
|
assert result["database"]["clone_database_remaining"] == 0
|
||||||
|
assert state == {"exists": False, "dropped": True}
|
||||||
|
receipt_path = args.run_root / args.run_id / "cleanup-receipt.json"
|
||||||
|
assert receipt_path.stat().st_mode & 0o777 == 0o600
|
||||||
|
assert json.loads(receipt_path.read_text())["error"]["phase"] == "rollback_readback"
|
||||||
|
|
||||||
|
|
||||||
def test_restore_dump_uses_tls_no_owner_no_acl_and_keeps_secret_out_of_argv(
|
def test_restore_dump_uses_tls_no_owner_no_acl_and_keeps_secret_out_of_argv(
|
||||||
tmp_path: Path,
|
tmp_path: Path,
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
|
@ -326,6 +639,110 @@ def test_restore_dump_uses_tls_no_owner_no_acl_and_keeps_secret_out_of_argv(
|
||||||
assert captured["env"]["PGPASSWORD"] == "private-password"
|
assert captured["env"]["PGPASSWORD"] == "private-password"
|
||||||
|
|
||||||
|
|
||||||
|
def test_cloudsql_control_plane_preflight_requires_private_only_expected_host(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
payload = {
|
||||||
|
"name": "teleo-pgvector-standby",
|
||||||
|
"state": "RUNNABLE",
|
||||||
|
"databaseVersion": "POSTGRES_16",
|
||||||
|
"region": "europe-west6",
|
||||||
|
"settings": {
|
||||||
|
"tier": "db-custom-1-3840",
|
||||||
|
"ipConfiguration": {
|
||||||
|
"ipv4Enabled": False,
|
||||||
|
"privateNetwork": "projects/teleo-501523/global/networks/teleo-staging-net",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"ipAddresses": [{"type": "PRIVATE", "ipAddress": "10.61.0.3"}],
|
||||||
|
}
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
restore,
|
||||||
|
"_run",
|
||||||
|
lambda *_args, **_kwargs: subprocess.CompletedProcess([], 0, json.dumps(payload), ""),
|
||||||
|
)
|
||||||
|
result = restore.cloudsql_control_plane_state(
|
||||||
|
"teleo-501523",
|
||||||
|
"teleo-pgvector-standby",
|
||||||
|
"10.61.0.3",
|
||||||
|
restore.DEFAULT_PRIVATE_NETWORK,
|
||||||
|
timeout=2.0,
|
||||||
|
)
|
||||||
|
assert result["public_ip_disabled"] is True
|
||||||
|
assert result["private_addresses"] == ["10.61.0.3"]
|
||||||
|
|
||||||
|
payload["settings"]["ipConfiguration"]["ipv4Enabled"] = True
|
||||||
|
payload["ipAddresses"].append({"type": "PRIMARY", "ipAddress": "34.1.2.3"})
|
||||||
|
with pytest.raises(restore.RestoreError, match="no_non_private_address, public_ip_disabled"):
|
||||||
|
restore.cloudsql_control_plane_state(
|
||||||
|
"teleo-501523",
|
||||||
|
"teleo-pgvector-standby",
|
||||||
|
"10.61.0.3",
|
||||||
|
restore.DEFAULT_PRIVATE_NETWORK,
|
||||||
|
timeout=2.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_gce_compute_identity_is_bound_to_metadata_instance_zone_and_network(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
metadata = {
|
||||||
|
"project/project-id": "teleo-501523",
|
||||||
|
"project/numeric-project-id": "785938879453",
|
||||||
|
"instance/name": "teleo-prod-1",
|
||||||
|
"instance/id": "123456789",
|
||||||
|
"instance/zone": "projects/123456789/zones/europe-west6-a",
|
||||||
|
"instance/network-interfaces/0/network": "projects/785938879453/networks/teleo-staging-net",
|
||||||
|
"instance/network-interfaces/0/ip": "10.61.0.2",
|
||||||
|
"instance/service-accounts/default/email": "teleo-runtime@teleo-501523.iam.gserviceaccount.com",
|
||||||
|
}
|
||||||
|
monkeypatch.setattr(restore, "gce_metadata_value", lambda path, **_kwargs: metadata[path])
|
||||||
|
|
||||||
|
result = restore.gce_compute_identity(
|
||||||
|
"teleo-501523",
|
||||||
|
"teleo-prod-1",
|
||||||
|
"europe-west6-a",
|
||||||
|
restore.DEFAULT_PRIVATE_NETWORK,
|
||||||
|
timeout=2.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result["instance_id"] == "123456789"
|
||||||
|
assert result["project_number"] == "785938879453"
|
||||||
|
assert result["zone"] == "europe-west6-a"
|
||||||
|
assert all(result["checks"].values())
|
||||||
|
|
||||||
|
with pytest.raises(restore.RestoreError, match="instance"):
|
||||||
|
restore.gce_compute_identity(
|
||||||
|
"teleo-501523",
|
||||||
|
"different-instance",
|
||||||
|
"europe-west6-a",
|
||||||
|
restore.DEFAULT_PRIVATE_NETWORK,
|
||||||
|
timeout=2.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
metadata["instance/name"] = "teleo-prod-1"
|
||||||
|
metadata["instance/network-interfaces/0/network"] = "projects/999999999999/networks/teleo-staging-net"
|
||||||
|
with pytest.raises(restore.RestoreError, match="network"):
|
||||||
|
restore.gce_compute_identity(
|
||||||
|
"teleo-501523",
|
||||||
|
"teleo-prod-1",
|
||||||
|
"europe-west6-a",
|
||||||
|
restore.DEFAULT_PRIVATE_NETWORK,
|
||||||
|
timeout=2.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
metadata["instance/network-interfaces/0/network"] = "projects/785938879453/networks/teleo-staging-net"
|
||||||
|
with pytest.raises(restore.RestoreError, match="expected_network_project"):
|
||||||
|
restore.gce_compute_identity(
|
||||||
|
"teleo-501523",
|
||||||
|
"teleo-prod-1",
|
||||||
|
"europe-west6-a",
|
||||||
|
"projects/other-project/global/networks/teleo-staging-net",
|
||||||
|
timeout=2.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_pg_restore_preflight_rejects_client_older_than_snapshot(
|
def test_pg_restore_preflight_rejects_client_older_than_snapshot(
|
||||||
tmp_path: Path,
|
tmp_path: Path,
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
|
|
||||||
703
tests/test_verify_gcp_canonical_lifecycle.py
Normal file
703
tests/test_verify_gcp_canonical_lifecycle.py
Normal file
|
|
@ -0,0 +1,703 @@
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from ops.verify_gcp_canonical_lifecycle import EXPECTED_BLIND_ROW_IDS
|
||||||
|
from ops.verify_gcp_canonical_lifecycle import verify_lifecycle as _verify_lifecycle
|
||||||
|
|
||||||
|
VERIFY_NOW = datetime(2026, 7, 15, 1, 3, tzinfo=UTC)
|
||||||
|
|
||||||
|
|
||||||
|
def verify_lifecycle(**kwargs):
|
||||||
|
return _verify_lifecycle(**kwargs, now=VERIFY_NOW)
|
||||||
|
|
||||||
|
|
||||||
|
def write_json(path: Path, payload: dict) -> Path:
|
||||||
|
path.write_text(json.dumps(payload, sort_keys=True) + "\n", encoding="utf-8")
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def sha256(path: Path) -> str:
|
||||||
|
return hashlib.sha256(path.read_bytes()).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def lifecycle_receipts(tmp_path: Path) -> dict[str, Path]:
|
||||||
|
target_db = "teleo_clone_current_test"
|
||||||
|
run_id = "gcp-restore-current-test123"
|
||||||
|
healthy_service = {
|
||||||
|
"ActiveState": "active",
|
||||||
|
"SubState": "running",
|
||||||
|
"MainPID": "148735",
|
||||||
|
"NRestarts": "0",
|
||||||
|
}
|
||||||
|
structural_hashes = {
|
||||||
|
kind: {"source": format(index, "x") * 64, "target": format(index, "x") * 64}
|
||||||
|
for index, kind in enumerate(
|
||||||
|
(
|
||||||
|
"schemas",
|
||||||
|
"columns",
|
||||||
|
"constraints",
|
||||||
|
"indexes",
|
||||||
|
"sequences",
|
||||||
|
"views",
|
||||||
|
"functions",
|
||||||
|
"triggers",
|
||||||
|
"types",
|
||||||
|
"policies",
|
||||||
|
),
|
||||||
|
1,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
exact_parity_details = {
|
||||||
|
"source_database": "teleo",
|
||||||
|
"target_database": target_db,
|
||||||
|
"source_table_count": 39,
|
||||||
|
"target_table_count": 39,
|
||||||
|
"source_total_rows": 52167,
|
||||||
|
"target_total_rows": 52167,
|
||||||
|
"table_mismatches": [],
|
||||||
|
"structural_hashes": structural_hashes,
|
||||||
|
"required_extension_mismatches": {},
|
||||||
|
"application_role_mismatches": {},
|
||||||
|
"target_extra_application_roles": [],
|
||||||
|
"performance": [{"query": "count_claims", "source_ms": 1.0, "target_ms": 2.0, "source_ratio": 0.4}],
|
||||||
|
"performance_problems": [],
|
||||||
|
}
|
||||||
|
source = {
|
||||||
|
"artifact": "vps_canonical_postgres_exported_snapshot",
|
||||||
|
"schema": "livingip.sourceSnapshotReceipt.v2",
|
||||||
|
"receipt_version": 2,
|
||||||
|
"status": "pass",
|
||||||
|
"run_id": "canonical-current-test",
|
||||||
|
"capture_authorization_ref": "codex-delegation:test1234",
|
||||||
|
"source": {
|
||||||
|
"ssh_target": "root@77.42.65.182",
|
||||||
|
"container": "teleo-pg",
|
||||||
|
"database": "teleo",
|
||||||
|
"service": "leoclean-gateway.service",
|
||||||
|
},
|
||||||
|
"snapshot_exported": True,
|
||||||
|
"snapshot": {
|
||||||
|
"captured_at_utc": "2026-07-15T00:59:00+00:00",
|
||||||
|
"exported_snapshot_id": "00000003-000001A2-1",
|
||||||
|
"txid_snapshot": "100:120:",
|
||||||
|
"wal_lsn": "0/16B6C50",
|
||||||
|
"system_identifier": "7649789040005668902",
|
||||||
|
},
|
||||||
|
"service_unchanged": True,
|
||||||
|
"source_service": {"before": healthy_service, "after": healthy_service, "unchanged": True},
|
||||||
|
"production_db_mutated": False,
|
||||||
|
"telegram_send_attempted": False,
|
||||||
|
"dump": {"sha256": "a" * 64},
|
||||||
|
"manifest": {"sha256": "b" * 64, "table_count": 39, "total_rows": 52167},
|
||||||
|
"provenance_binding": {"algorithm": "sha256", "payload": {"bound": True}, "sha256": "c" * 64},
|
||||||
|
}
|
||||||
|
source_path = write_json(tmp_path / "source.json", source)
|
||||||
|
restore = {
|
||||||
|
"artifact": "gcp_generated_postgres_snapshot_restore",
|
||||||
|
"status": "pass",
|
||||||
|
"generated_at_utc": "2026-07-15T00:59:10+00:00",
|
||||||
|
"completed_at_utc": "2026-07-15T00:59:30+00:00",
|
||||||
|
"run_id": run_id,
|
||||||
|
"target_database": target_db,
|
||||||
|
"source": {
|
||||||
|
"capture_receipt": {
|
||||||
|
"sha256": sha256(source_path),
|
||||||
|
"capture_authorization_ref": "codex-delegation:test1234",
|
||||||
|
"provenance_binding_sha256": "c" * 64,
|
||||||
|
"dump_sha256": "a" * 64,
|
||||||
|
"manifest_sha256": "b" * 64,
|
||||||
|
"checks": {
|
||||||
|
name: True
|
||||||
|
for name in (
|
||||||
|
"artifact",
|
||||||
|
"schema",
|
||||||
|
"receipt_version",
|
||||||
|
"status",
|
||||||
|
"snapshot_exported",
|
||||||
|
"service_unchanged",
|
||||||
|
"source_service_healthy",
|
||||||
|
"source_not_mutated",
|
||||||
|
"telegram_not_sent",
|
||||||
|
"run_id",
|
||||||
|
"capture_authorization_ref",
|
||||||
|
"source_identity",
|
||||||
|
"source_database",
|
||||||
|
"snapshot_identity",
|
||||||
|
"dump_sha256",
|
||||||
|
"dump_bytes",
|
||||||
|
"source_manifest_sha256",
|
||||||
|
"manifest_sql_sha256",
|
||||||
|
"source_context_sha256",
|
||||||
|
"table_count",
|
||||||
|
"total_rows",
|
||||||
|
"binding_algorithm",
|
||||||
|
"binding_payload",
|
||||||
|
"binding_sha256",
|
||||||
|
)
|
||||||
|
},
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"execution_contract": {
|
||||||
|
"project": "teleo-501523",
|
||||||
|
"cloudsql_instance": "teleo-pgvector-standby",
|
||||||
|
"host": "10.61.0.3",
|
||||||
|
"private_network": "projects/teleo-501523/global/networks/teleo-staging-net",
|
||||||
|
"compute_instance": "teleo-prod-1",
|
||||||
|
"compute_zone": "europe-west6-a",
|
||||||
|
"source": source["source"],
|
||||||
|
"password_secret": "gcp-teleo-pgvector-standby-postgres-password",
|
||||||
|
"service": "leoclean-gcp-prod-parallel.service",
|
||||||
|
},
|
||||||
|
"target": {
|
||||||
|
"manifest_sha256": "d" * 64,
|
||||||
|
"connectivity": {
|
||||||
|
"private_connectivity": True,
|
||||||
|
"ssl": True,
|
||||||
|
"server_address": "10.61.0.3",
|
||||||
|
"source_compute": "teleo-prod-1",
|
||||||
|
},
|
||||||
|
"cloudsql": {
|
||||||
|
"project": "teleo-501523",
|
||||||
|
"instance": "teleo-pgvector-standby",
|
||||||
|
"expected_private_host": "10.61.0.3",
|
||||||
|
"private_network": "projects/teleo-501523/global/networks/teleo-staging-net",
|
||||||
|
"expected_private_network": "projects/teleo-501523/global/networks/teleo-staging-net",
|
||||||
|
"public_ip_disabled": True,
|
||||||
|
"checks": {
|
||||||
|
"instance": True,
|
||||||
|
"state": True,
|
||||||
|
"postgres_16": True,
|
||||||
|
"public_ip_disabled": True,
|
||||||
|
"private_network": True,
|
||||||
|
"expected_private_host": True,
|
||||||
|
"no_non_private_address": True,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"compute": {
|
||||||
|
"project": "teleo-501523",
|
||||||
|
"project_number": "785938879453",
|
||||||
|
"instance": "teleo-prod-1",
|
||||||
|
"instance_id": "123456789",
|
||||||
|
"zone": "europe-west6-a",
|
||||||
|
"expected_private_network": "projects/teleo-501523/global/networks/teleo-staging-net",
|
||||||
|
"network_resource": "projects/123456789/networks/teleo-staging-net",
|
||||||
|
"private_ip": "10.61.0.2",
|
||||||
|
"service_account": "teleo-runtime@teleo-501523.iam.gserviceaccount.com",
|
||||||
|
"checks": {
|
||||||
|
"project": True,
|
||||||
|
"project_number": True,
|
||||||
|
"expected_network_project": True,
|
||||||
|
"instance": True,
|
||||||
|
"instance_id": True,
|
||||||
|
"zone": True,
|
||||||
|
"network": True,
|
||||||
|
"private_ip": True,
|
||||||
|
"service_account": True,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"connectivity_proof": {
|
||||||
|
"sha256": "f" * 64,
|
||||||
|
"schema": "livingip.gcpPrivatePostgresConnectivity.v2",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"parity": {"status": "pass", "problems": [], "details": exact_parity_details},
|
||||||
|
"live_service": {"before": healthy_service, "after": healthy_service, "unchanged": True},
|
||||||
|
}
|
||||||
|
restore_path = write_json(tmp_path / "restore.json", restore)
|
||||||
|
parity = {
|
||||||
|
"artifact": "canonical_postgres_parity_verification",
|
||||||
|
"status": "pass",
|
||||||
|
"completed_at_utc": "2026-07-15T00:59:45+00:00",
|
||||||
|
"scope": "gcp_staging",
|
||||||
|
"problems": [],
|
||||||
|
"source_manifest_sha256": "b" * 64,
|
||||||
|
"target_manifest_sha256": "d" * 64,
|
||||||
|
"connectivity_proof_sha256": "f" * 64,
|
||||||
|
"details": exact_parity_details,
|
||||||
|
"private_connectivity": {
|
||||||
|
"required": True,
|
||||||
|
"proof": {
|
||||||
|
"artifact": "gcp_private_postgres_connectivity",
|
||||||
|
"schema": "livingip.gcpPrivatePostgresConnectivity.v2",
|
||||||
|
"status": "pass",
|
||||||
|
"restore_run_id": run_id,
|
||||||
|
"private_connectivity": True,
|
||||||
|
"public_ip_disabled": True,
|
||||||
|
"target_database": target_db,
|
||||||
|
"source_compute": "teleo-prod-1",
|
||||||
|
"source_compute_instance_id": "123456789",
|
||||||
|
"source_compute_zone": "europe-west6-a",
|
||||||
|
"source_compute_private_ip": "10.61.0.2",
|
||||||
|
"server_address": "10.61.0.3",
|
||||||
|
"project": "teleo-501523",
|
||||||
|
"cloudsql_instance": "teleo-pgvector-standby",
|
||||||
|
"private_network": "projects/teleo-501523/global/networks/teleo-staging-net",
|
||||||
|
"cloudsql_control_plane_checks": restore["target"]["cloudsql"]["checks"],
|
||||||
|
"compute_identity_checks": restore["target"]["compute"]["checks"],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
parity_path = write_json(tmp_path / "parity.json", parity)
|
||||||
|
required_checks = {
|
||||||
|
name: True
|
||||||
|
for name in (
|
||||||
|
"exact_blind_prompt_without_ids",
|
||||||
|
"claim_and_sources_retrieved",
|
||||||
|
"discovery_show_evidence_completed",
|
||||||
|
"retrieval_receipts_proven",
|
||||||
|
"private_tls_read_only_target",
|
||||||
|
"generated_database_unchanged",
|
||||||
|
"canonical_counts_unchanged",
|
||||||
|
"database_calls_read_only",
|
||||||
|
"no_database_write",
|
||||||
|
"no_telegram_send",
|
||||||
|
"live_service_unchanged",
|
||||||
|
"live_profile_unchanged",
|
||||||
|
"temporary_profile_removed",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
reasoning = {
|
||||||
|
"schema": "livingip.gcpGeneratedDbBlindClaimCanary.v1",
|
||||||
|
"status": "pass",
|
||||||
|
"generated_at_utc": "2026-07-15T01:00:00+00:00",
|
||||||
|
"completed_at_utc": "2026-07-15T01:01:00+00:00",
|
||||||
|
"mode": "gcp_generated_db_gatewayrunner_blind_claim_no_send",
|
||||||
|
"source_compute": "teleo-prod-1",
|
||||||
|
"prompt": "Inspect the live claim without me giving you an ID. Do not change the database.",
|
||||||
|
"errors": [],
|
||||||
|
"target_database": target_db,
|
||||||
|
"posted_to_telegram": False,
|
||||||
|
"database_write_attempted": False,
|
||||||
|
"parity_receipt": {"sha256": sha256(parity_path)},
|
||||||
|
"checks": required_checks,
|
||||||
|
"database_fingerprint_before": {"sha256": "e" * 64},
|
||||||
|
"database_fingerprint_after": {"sha256": "e" * 64},
|
||||||
|
"canonical_status_before": {"claims": 1},
|
||||||
|
"canonical_status_after": {"claims": 1},
|
||||||
|
"database_identity_before": {"database": target_db},
|
||||||
|
"database_identity_after": {"database": target_db},
|
||||||
|
"service_before": healthy_service,
|
||||||
|
"service_after": healthy_service,
|
||||||
|
"temp_profile_absent": True,
|
||||||
|
"temp_profile_removed": True,
|
||||||
|
"outcomes": {
|
||||||
|
"asks_for_user_iteration_before_live": True,
|
||||||
|
"challenges_weak_support": True,
|
||||||
|
"decomposes_the_bundled_legal_claim": True,
|
||||||
|
"does_not_claim_a_database_change": True,
|
||||||
|
"proposes_candidate_claims": True,
|
||||||
|
"uses_only_m3taversal_handle": True,
|
||||||
|
},
|
||||||
|
"result": {
|
||||||
|
"reply": "The claim is weak; here are grounded, review-first alternatives.",
|
||||||
|
"database_tool_trace": {
|
||||||
|
"schema": "livingip.leoKbToolTrace.v1",
|
||||||
|
"database_tool_call_proven": True,
|
||||||
|
"database_retrieval_receipt_proven": True,
|
||||||
|
"database_tool_calls_read_only": True,
|
||||||
|
"database_tool_call_count": 3,
|
||||||
|
"database_tool_completed_count": 3,
|
||||||
|
"calls": [
|
||||||
|
{
|
||||||
|
"database_invocations": [{"access_mode": "read_only", "subcommand": subcommand}],
|
||||||
|
"result": {
|
||||||
|
"nonempty": True,
|
||||||
|
"error_detected": False,
|
||||||
|
"row_ids": sorted(EXPECTED_BLIND_ROW_IDS),
|
||||||
|
"row_id_count": len(EXPECTED_BLIND_ROW_IDS),
|
||||||
|
"retrieval_receipt": {
|
||||||
|
"schema": "livingip.teleoKbRetrievalReceipt.v1",
|
||||||
|
"semantic_context_sha256": "1" * 64,
|
||||||
|
"artifact_state_sha256": "2" * 64,
|
||||||
|
"read_consistency_status": "stable_wal_marker",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for subcommand in ("search", "show", "evidence")
|
||||||
|
],
|
||||||
|
},
|
||||||
|
"gateway": {
|
||||||
|
"authorized": True,
|
||||||
|
"handler_invoked": True,
|
||||||
|
"model_free_fallback_used": False,
|
||||||
|
"posted_to_telegram": False,
|
||||||
|
"child_process": {
|
||||||
|
"exitcode": 0,
|
||||||
|
"alive_after_readback": False,
|
||||||
|
"process_group_alive_after_readback": False,
|
||||||
|
"timed_out": False,
|
||||||
|
},
|
||||||
|
"tool_surface": {"send_message_tool_enabled": False},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"wrapper_tool_proof": {
|
||||||
|
"all_bound_to_supplied_target": True,
|
||||||
|
"all_completed_successfully": True,
|
||||||
|
"complete_start_end_pairing": True,
|
||||||
|
"database_read_only_required": True,
|
||||||
|
"default_read_only_required": True,
|
||||||
|
"invocations": [
|
||||||
|
{
|
||||||
|
"database": target_db,
|
||||||
|
"returncode": 0,
|
||||||
|
"subcommand": subcommand,
|
||||||
|
"database_identity": {
|
||||||
|
"transaction_read_only": "on",
|
||||||
|
"default_transaction_read_only": "on",
|
||||||
|
"ssl": True,
|
||||||
|
"server_address": "10.61.0.3",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for subcommand in ("search", "show", "evidence")
|
||||||
|
],
|
||||||
|
},
|
||||||
|
}
|
||||||
|
reasoning_path = write_json(tmp_path / "reasoning.json", reasoning)
|
||||||
|
reasoning_compute = {
|
||||||
|
"artifact": "gcp_reasoning_compute_attestation",
|
||||||
|
"schema": "livingip.gcpReasoningComputeAttestation.v1",
|
||||||
|
"status": "pass",
|
||||||
|
"generated_at_utc": "2026-07-15T01:01:05+00:00",
|
||||||
|
"restore_run_id": run_id,
|
||||||
|
"target_database": target_db,
|
||||||
|
"reasoning_receipt": {
|
||||||
|
"sha256": sha256(reasoning_path),
|
||||||
|
"schema": reasoning["schema"],
|
||||||
|
"generated_at_utc": reasoning["generated_at_utc"],
|
||||||
|
"completed_at_utc": reasoning["completed_at_utc"],
|
||||||
|
"claimed_source_compute": reasoning["source_compute"],
|
||||||
|
},
|
||||||
|
"compute": restore["target"]["compute"],
|
||||||
|
"checks": {
|
||||||
|
"schema": True,
|
||||||
|
"status": True,
|
||||||
|
"target_database": True,
|
||||||
|
"source_compute": True,
|
||||||
|
"chronology": True,
|
||||||
|
"no_telegram_send": True,
|
||||||
|
"no_database_write": True,
|
||||||
|
"gce_metadata_identity": True,
|
||||||
|
},
|
||||||
|
"method": "gce_metadata_server_v1",
|
||||||
|
"mutation": {
|
||||||
|
"cloudsql_changed": False,
|
||||||
|
"compute_changed": False,
|
||||||
|
"database_changed": False,
|
||||||
|
"telegram_message_sent": False,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
reasoning_compute_path = write_json(tmp_path / "reasoning-compute.json", reasoning_compute)
|
||||||
|
current_source = json.loads(json.dumps(source))
|
||||||
|
current_source["run_id"] = "canonical-current-postflight"
|
||||||
|
current_source["capture_authorization_ref"] = "codex-delegation:postflight1234"
|
||||||
|
current_source["snapshot"]["captured_at_utc"] = "2026-07-15T01:02:00+00:00"
|
||||||
|
current_source["snapshot"]["exported_snapshot_id"] = "00000003-000001B2-1"
|
||||||
|
current_source_path = write_json(tmp_path / "current-source.json", current_source)
|
||||||
|
source_delta = {
|
||||||
|
"artifact": "vps_canonical_snapshot_postflight_delta",
|
||||||
|
"schema": "livingip.vpsCanonicalSnapshotDelta.v1",
|
||||||
|
"status": "pass",
|
||||||
|
"baseline": {
|
||||||
|
"run_id": "canonical-current-test",
|
||||||
|
"capture_receipt_sha256": sha256(source_path),
|
||||||
|
"manifest_sha256": "b" * 64,
|
||||||
|
"capture_authorization_ref": "codex-delegation:test1234",
|
||||||
|
"captured_at_utc": "2026-07-15T00:59:00+00:00",
|
||||||
|
"table_count": 39,
|
||||||
|
"total_rows": 52167,
|
||||||
|
"table_fingerprint_sha256": "d" * 64,
|
||||||
|
"structure_fingerprint_sha256": "e" * 64,
|
||||||
|
},
|
||||||
|
"current": {
|
||||||
|
"run_id": "canonical-current-postflight",
|
||||||
|
"capture_receipt_sha256": sha256(current_source_path),
|
||||||
|
"manifest_sha256": "b" * 64,
|
||||||
|
"capture_authorization_ref": "codex-delegation:postflight1234",
|
||||||
|
"captured_at_utc": "2026-07-15T01:02:00+00:00",
|
||||||
|
"table_count": 39,
|
||||||
|
"total_rows": 52167,
|
||||||
|
"table_fingerprint_sha256": "d" * 64,
|
||||||
|
"structure_fingerprint_sha256": "e" * 64,
|
||||||
|
},
|
||||||
|
"delta": {
|
||||||
|
"delta_detected": False,
|
||||||
|
"added_tables": [],
|
||||||
|
"removed_tables": [],
|
||||||
|
"changed_tables": [],
|
||||||
|
"changed_structures": [],
|
||||||
|
"total_row_delta": 0,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
source_delta_path = write_json(tmp_path / "source-delta.json", source_delta)
|
||||||
|
cleanup = {
|
||||||
|
"artifact": "gcp_generated_postgres_snapshot_cleanup",
|
||||||
|
"status": "pass",
|
||||||
|
"generated_at_utc": "2026-07-15T01:01:10+00:00",
|
||||||
|
"completed_at_utc": "2026-07-15T01:01:30+00:00",
|
||||||
|
"run_id": run_id,
|
||||||
|
"target_database": target_db,
|
||||||
|
"database": {"clone_database_remaining": 0},
|
||||||
|
"live_service": {"before": healthy_service, "after": healthy_service, "unchanged": True},
|
||||||
|
"restore_receipt": {"sha256": sha256(restore_path)},
|
||||||
|
"execution_contract": restore["execution_contract"],
|
||||||
|
"target": {
|
||||||
|
"cloudsql": restore["target"]["cloudsql"],
|
||||||
|
"compute": restore["target"]["compute"],
|
||||||
|
},
|
||||||
|
}
|
||||||
|
cleanup_path = write_json(tmp_path / "cleanup.json", cleanup)
|
||||||
|
return {
|
||||||
|
"source_path": source_path,
|
||||||
|
"current_source_path": current_source_path,
|
||||||
|
"source_delta_path": source_delta_path,
|
||||||
|
"restore_path": restore_path,
|
||||||
|
"parity_path": parity_path,
|
||||||
|
"reasoning_path": reasoning_path,
|
||||||
|
"reasoning_compute_path": reasoning_compute_path,
|
||||||
|
"cleanup_path": cleanup_path,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_bound_live_lifecycle_passes(tmp_path: Path) -> None:
|
||||||
|
paths = lifecycle_receipts(tmp_path)
|
||||||
|
|
||||||
|
result = verify_lifecycle(**paths)
|
||||||
|
|
||||||
|
assert result["status"] == "pass"
|
||||||
|
assert result["failed_checks"] == []
|
||||||
|
assert result["checks"]["clone_absent"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_local_parity_or_missing_cleanup_fails_closed(tmp_path: Path) -> None:
|
||||||
|
paths = lifecycle_receipts(tmp_path)
|
||||||
|
parity = json.loads(paths["parity_path"].read_text())
|
||||||
|
parity["scope"] = "local_restore"
|
||||||
|
write_json(paths["parity_path"], parity)
|
||||||
|
reasoning = json.loads(paths["reasoning_path"].read_text())
|
||||||
|
reasoning["parity_receipt"]["sha256"] = sha256(paths["parity_path"])
|
||||||
|
write_json(paths["reasoning_path"], reasoning)
|
||||||
|
cleanup = json.loads(paths["cleanup_path"].read_text())
|
||||||
|
cleanup["database"]["clone_database_remaining"] = 1
|
||||||
|
write_json(paths["cleanup_path"], cleanup)
|
||||||
|
|
||||||
|
result = verify_lifecycle(**paths)
|
||||||
|
|
||||||
|
assert result["status"] == "fail"
|
||||||
|
assert {"parity_pass", "clone_absent"} <= set(result["failed_checks"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_mismatched_source_or_reasoning_receipt_fails_closed(tmp_path: Path) -> None:
|
||||||
|
paths = lifecycle_receipts(tmp_path)
|
||||||
|
restore = json.loads(paths["restore_path"].read_text())
|
||||||
|
restore["source"]["capture_receipt"]["sha256"] = "0" * 64
|
||||||
|
write_json(paths["restore_path"], restore)
|
||||||
|
reasoning = json.loads(paths["reasoning_path"].read_text())
|
||||||
|
reasoning["checks"]["claim_and_sources_retrieved"] = False
|
||||||
|
write_json(paths["reasoning_path"], reasoning)
|
||||||
|
|
||||||
|
result = verify_lifecycle(**paths)
|
||||||
|
|
||||||
|
assert result["status"] == "fail"
|
||||||
|
assert {"source_receipt_hash_bound", "reasoning_required_checks"} <= set(result["failed_checks"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_hand_authored_reasoning_boole_without_producer_evidence_fail(tmp_path: Path) -> None:
|
||||||
|
paths = lifecycle_receipts(tmp_path)
|
||||||
|
reasoning = json.loads(paths["reasoning_path"].read_text())
|
||||||
|
reasoning.pop("result")
|
||||||
|
reasoning.pop("wrapper_tool_proof")
|
||||||
|
write_json(paths["reasoning_path"], reasoning)
|
||||||
|
|
||||||
|
result = verify_lifecycle(**paths)
|
||||||
|
|
||||||
|
assert result["status"] == "fail"
|
||||||
|
assert "reasoning_producer_evidence" in result["failed_checks"]
|
||||||
|
assert result["reasoning_producer_checks"]["tool_trace"] is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_reasoning_compute_attestation_must_bind_live_metadata(tmp_path: Path) -> None:
|
||||||
|
paths = lifecycle_receipts(tmp_path)
|
||||||
|
attestation = json.loads(paths["reasoning_compute_path"].read_text())
|
||||||
|
attestation["compute"]["instance_id"] = "999999999"
|
||||||
|
write_json(paths["reasoning_compute_path"], attestation)
|
||||||
|
|
||||||
|
result = verify_lifecycle(**paths)
|
||||||
|
|
||||||
|
assert result["status"] == "fail"
|
||||||
|
assert "reasoning_compute_identity_bound" in result["failed_checks"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_source_postflight_must_be_newer_than_cleanup_completion(tmp_path: Path) -> None:
|
||||||
|
paths = lifecycle_receipts(tmp_path)
|
||||||
|
cleanup = json.loads(paths["cleanup_path"].read_text())
|
||||||
|
cleanup["completed_at_utc"] = "2026-07-15T02:00:00+00:00"
|
||||||
|
write_json(paths["cleanup_path"], cleanup)
|
||||||
|
|
||||||
|
result = verify_lifecycle(**paths)
|
||||||
|
|
||||||
|
assert result["status"] == "fail"
|
||||||
|
assert "source_postflight_after_lifecycle" in result["failed_checks"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_parity_must_retain_exact_structure_roles_rows_and_performance(tmp_path: Path) -> None:
|
||||||
|
paths = lifecycle_receipts(tmp_path)
|
||||||
|
parity = json.loads(paths["parity_path"].read_text())
|
||||||
|
parity["details"]["structural_hashes"].pop("constraints")
|
||||||
|
parity["details"]["application_role_mismatches"] = {"kb_apply": {"missing": True}}
|
||||||
|
parity["details"]["performance"] = []
|
||||||
|
write_json(paths["parity_path"], parity)
|
||||||
|
reasoning = json.loads(paths["reasoning_path"].read_text())
|
||||||
|
reasoning["parity_receipt"]["sha256"] = sha256(paths["parity_path"])
|
||||||
|
write_json(paths["reasoning_path"], reasoning)
|
||||||
|
|
||||||
|
result = verify_lifecycle(**paths)
|
||||||
|
|
||||||
|
assert result["status"] == "fail"
|
||||||
|
assert "parity_exact_evidence" in result["failed_checks"]
|
||||||
|
assert result["parity_evidence_checks"]["structural_hashes"] is False
|
||||||
|
assert result["parity_evidence_checks"]["roles"] is False
|
||||||
|
assert result["parity_evidence_checks"]["performance"] is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_reasoning_must_retrieve_expected_claim_sources_and_preserve_handle(tmp_path: Path) -> None:
|
||||||
|
paths = lifecycle_receipts(tmp_path)
|
||||||
|
reasoning = json.loads(paths["reasoning_path"].read_text())
|
||||||
|
for call in reasoning["result"]["database_tool_trace"]["calls"]:
|
||||||
|
call["result"]["row_ids"] = []
|
||||||
|
call["result"]["row_id_count"] = 0
|
||||||
|
reasoning["result"]["reply"] = "Cory found a weak claim and two sources."
|
||||||
|
write_json(paths["reasoning_path"], reasoning)
|
||||||
|
|
||||||
|
result = verify_lifecycle(**paths)
|
||||||
|
|
||||||
|
assert result["status"] == "fail"
|
||||||
|
assert "reasoning_producer_evidence" in result["failed_checks"]
|
||||||
|
assert result["reasoning_producer_checks"]["expected_claim_and_sources_retrieved"] is False
|
||||||
|
assert result["reasoning_producer_checks"]["reply_uses_only_m3taversal_handle"] is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_lifecycle_requires_restore_parity_reasoning_cleanup_chronology(tmp_path: Path) -> None:
|
||||||
|
paths = lifecycle_receipts(tmp_path)
|
||||||
|
cleanup = json.loads(paths["cleanup_path"].read_text())
|
||||||
|
cleanup["generated_at_utc"] = "2026-07-15T00:58:00+00:00"
|
||||||
|
write_json(paths["cleanup_path"], cleanup)
|
||||||
|
|
||||||
|
result = verify_lifecycle(**paths)
|
||||||
|
|
||||||
|
assert result["status"] == "fail"
|
||||||
|
assert "lifecycle_chronology" in result["failed_checks"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_source_delta_requires_real_fingerprints_and_consistent_no_delta(tmp_path: Path) -> None:
|
||||||
|
paths = lifecycle_receipts(tmp_path)
|
||||||
|
delta = json.loads(paths["source_delta_path"].read_text())
|
||||||
|
delta["baseline"].pop("table_fingerprint_sha256")
|
||||||
|
delta["current"]["total_rows"] += 1
|
||||||
|
write_json(paths["source_delta_path"], delta)
|
||||||
|
|
||||||
|
result = verify_lifecycle(**paths)
|
||||||
|
|
||||||
|
assert result["status"] == "fail"
|
||||||
|
assert {"source_delta_baseline_shape", "source_delta_current_shape", "source_delta_consistent"} <= set(
|
||||||
|
result["failed_checks"]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_detected_source_delta_requires_exact_row_arithmetic(tmp_path: Path) -> None:
|
||||||
|
paths = lifecycle_receipts(tmp_path)
|
||||||
|
delta = json.loads(paths["source_delta_path"].read_text())
|
||||||
|
delta["delta"]["delta_detected"] = True
|
||||||
|
delta["delta"]["changed_tables"] = [{"table": "public.claims"}]
|
||||||
|
delta["delta"]["total_row_delta"] = 99
|
||||||
|
write_json(paths["source_delta_path"], delta)
|
||||||
|
|
||||||
|
result = verify_lifecycle(**paths)
|
||||||
|
|
||||||
|
assert result["status"] == "fail"
|
||||||
|
assert "source_delta_consistent" in result["failed_checks"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_postflight_source_requires_structured_healthy_service(tmp_path: Path) -> None:
|
||||||
|
paths = lifecycle_receipts(tmp_path)
|
||||||
|
current = json.loads(paths["current_source_path"].read_text())
|
||||||
|
current["source_service"]["before"]["ActiveState"] = "failed"
|
||||||
|
current["source_service"]["after"]["ActiveState"] = "failed"
|
||||||
|
write_json(paths["current_source_path"], current)
|
||||||
|
delta = json.loads(paths["source_delta_path"].read_text())
|
||||||
|
delta["current"]["capture_receipt_sha256"] = sha256(paths["current_source_path"])
|
||||||
|
write_json(paths["source_delta_path"], delta)
|
||||||
|
|
||||||
|
result = verify_lifecycle(**paths)
|
||||||
|
|
||||||
|
assert result["status"] == "fail"
|
||||||
|
assert "current_source_pass" in result["failed_checks"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_unchanged_but_inactive_service_cannot_pass_t3(tmp_path: Path) -> None:
|
||||||
|
paths = lifecycle_receipts(tmp_path)
|
||||||
|
restore = json.loads(paths["restore_path"].read_text())
|
||||||
|
restore["live_service"]["before"]["ActiveState"] = "failed"
|
||||||
|
restore["live_service"]["after"]["ActiveState"] = "failed"
|
||||||
|
write_json(paths["restore_path"], restore)
|
||||||
|
cleanup = json.loads(paths["cleanup_path"].read_text())
|
||||||
|
cleanup["restore_receipt"]["sha256"] = sha256(paths["restore_path"])
|
||||||
|
write_json(paths["cleanup_path"], cleanup)
|
||||||
|
|
||||||
|
result = verify_lifecycle(**paths)
|
||||||
|
|
||||||
|
assert result["status"] == "fail"
|
||||||
|
assert "restore_service_healthy_unchanged" in result["failed_checks"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_historical_but_monotonic_receipts_cannot_claim_current_t3(tmp_path: Path) -> None:
|
||||||
|
paths = lifecycle_receipts(tmp_path)
|
||||||
|
|
||||||
|
result = _verify_lifecycle(**paths, now=datetime(2026, 7, 15, 3, 0, tzinfo=UTC))
|
||||||
|
|
||||||
|
assert result["status"] == "fail"
|
||||||
|
assert "source_postflight_fresh" in result["failed_checks"]
|
||||||
|
assert result["postflight_age_seconds"] > result["max_postflight_age_seconds"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_fresh_postflight_cannot_mask_historical_gcp_lifecycle(tmp_path: Path) -> None:
|
||||||
|
paths = lifecycle_receipts(tmp_path)
|
||||||
|
source = json.loads(paths["source_path"].read_text())
|
||||||
|
source["snapshot"]["captured_at_utc"] = "2026-07-12T00:59:00+00:00"
|
||||||
|
write_json(paths["source_path"], source)
|
||||||
|
restore = json.loads(paths["restore_path"].read_text())
|
||||||
|
restore["generated_at_utc"] = "2026-07-12T00:59:10+00:00"
|
||||||
|
restore["completed_at_utc"] = "2026-07-12T00:59:30+00:00"
|
||||||
|
restore["source"]["capture_receipt"]["sha256"] = sha256(paths["source_path"])
|
||||||
|
write_json(paths["restore_path"], restore)
|
||||||
|
parity = json.loads(paths["parity_path"].read_text())
|
||||||
|
parity["completed_at_utc"] = "2026-07-12T00:59:45+00:00"
|
||||||
|
write_json(paths["parity_path"], parity)
|
||||||
|
reasoning = json.loads(paths["reasoning_path"].read_text())
|
||||||
|
reasoning["generated_at_utc"] = "2026-07-12T01:00:00+00:00"
|
||||||
|
reasoning["completed_at_utc"] = "2026-07-12T01:01:00+00:00"
|
||||||
|
reasoning["parity_receipt"]["sha256"] = sha256(paths["parity_path"])
|
||||||
|
write_json(paths["reasoning_path"], reasoning)
|
||||||
|
attestation = json.loads(paths["reasoning_compute_path"].read_text())
|
||||||
|
attestation["generated_at_utc"] = "2026-07-12T01:01:05+00:00"
|
||||||
|
attestation["reasoning_receipt"]["sha256"] = sha256(paths["reasoning_path"])
|
||||||
|
attestation["reasoning_receipt"]["generated_at_utc"] = reasoning["generated_at_utc"]
|
||||||
|
attestation["reasoning_receipt"]["completed_at_utc"] = reasoning["completed_at_utc"]
|
||||||
|
write_json(paths["reasoning_compute_path"], attestation)
|
||||||
|
cleanup = json.loads(paths["cleanup_path"].read_text())
|
||||||
|
cleanup["generated_at_utc"] = "2026-07-12T01:01:10+00:00"
|
||||||
|
cleanup["completed_at_utc"] = "2026-07-12T01:01:30+00:00"
|
||||||
|
cleanup["restore_receipt"]["sha256"] = sha256(paths["restore_path"])
|
||||||
|
write_json(paths["cleanup_path"], cleanup)
|
||||||
|
delta = json.loads(paths["source_delta_path"].read_text())
|
||||||
|
delta["baseline"]["captured_at_utc"] = "2026-07-12T00:59:00+00:00"
|
||||||
|
delta["baseline"]["capture_receipt_sha256"] = sha256(paths["source_path"])
|
||||||
|
write_json(paths["source_delta_path"], delta)
|
||||||
|
|
||||||
|
result = verify_lifecycle(**paths)
|
||||||
|
|
||||||
|
assert result["checks"]["source_postflight_fresh"] is True
|
||||||
|
assert result["checks"]["lifecycle_chronology"] is True
|
||||||
|
assert result["status"] == "fail"
|
||||||
|
assert {"gcp_lifecycle_fresh", "lifecycle_span_bounded"} <= set(result["failed_checks"])
|
||||||
|
|
@ -90,7 +90,10 @@ def test_matching_local_manifests_pass(tmp_path: Path) -> None:
|
||||||
assert completed.returncode == 0, completed.stderr
|
assert completed.returncode == 0, completed.stderr
|
||||||
assert payload["status"] == "pass"
|
assert payload["status"] == "pass"
|
||||||
assert payload["details"]["source_total_rows"] == 2
|
assert payload["details"]["source_total_rows"] == 2
|
||||||
|
assert len(payload["source_manifest_sha256"]) == 64
|
||||||
|
assert len(payload["target_manifest_sha256"]) == 64
|
||||||
assert payload["private_connectivity"]["status"] == "not_applicable_local_restore"
|
assert payload["private_connectivity"]["status"] == "not_applicable_local_restore"
|
||||||
|
assert (tmp_path / "result.json").stat().st_mode & 0o777 == 0o600
|
||||||
|
|
||||||
|
|
||||||
def test_row_hash_and_role_mismatches_fail(tmp_path: Path) -> None:
|
def test_row_hash_and_role_mismatches_fail(tmp_path: Path) -> None:
|
||||||
|
|
@ -135,6 +138,7 @@ def test_gcp_scope_accepts_matching_private_connectivity_receipt(tmp_path: Path)
|
||||||
assert completed.returncode == 0, completed.stderr
|
assert completed.returncode == 0, completed.stderr
|
||||||
assert payload["status"] == "pass"
|
assert payload["status"] == "pass"
|
||||||
assert payload["private_connectivity"]["proof"]["source_compute"] == "teleo-staging-1"
|
assert payload["private_connectivity"]["proof"]["source_compute"] == "teleo-staging-1"
|
||||||
|
assert len(payload["connectivity_proof_sha256"]) == 64
|
||||||
|
|
||||||
|
|
||||||
def test_gcp_scope_rejects_receipt_for_a_different_server(tmp_path: Path) -> None:
|
def test_gcp_scope_rejects_receipt_for_a_different_server(tmp_path: Path) -> None:
|
||||||
|
|
|
||||||
259
tests/test_verify_vps_canonical_snapshot_delta.py
Normal file
259
tests/test_verify_vps_canonical_snapshot_delta.py
Normal file
|
|
@ -0,0 +1,259 @@
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from ops.capture_vps_canonical_postgres_snapshot import (
|
||||||
|
SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
|
||||||
|
build_provenance_binding,
|
||||||
|
)
|
||||||
|
from ops.verify_vps_canonical_snapshot_delta import verify_delta, write_private_json
|
||||||
|
|
||||||
|
|
||||||
|
def manifest_text(*, claims: int = 1, constraint_name: str = "claims_pkey") -> str:
|
||||||
|
rows = [
|
||||||
|
{
|
||||||
|
"kind": "identity",
|
||||||
|
"database": "teleo",
|
||||||
|
"server_version_num": 160014,
|
||||||
|
"transaction_read_only": "on",
|
||||||
|
},
|
||||||
|
{"kind": "schemas", "items": ["kb_stage", "public"]},
|
||||||
|
{"kind": "extensions", "items": [{"name": "pgcrypto", "version": "1.3"}]},
|
||||||
|
{"kind": "application_roles", "items": [{"name": "kb_apply", "can_login": True}]},
|
||||||
|
{"kind": "columns", "items": []},
|
||||||
|
{"kind": "constraints", "items": [{"name": constraint_name}]},
|
||||||
|
{"kind": "indexes", "items": []},
|
||||||
|
{"kind": "sequences", "items": []},
|
||||||
|
{"kind": "views", "items": []},
|
||||||
|
{"kind": "functions", "items": []},
|
||||||
|
{"kind": "triggers", "items": []},
|
||||||
|
{"kind": "types", "items": []},
|
||||||
|
{"kind": "policies", "items": []},
|
||||||
|
{
|
||||||
|
"kind": "table",
|
||||||
|
"schema": "public",
|
||||||
|
"table": "claims",
|
||||||
|
"row_count": claims,
|
||||||
|
"rowset_md5": hashlib.md5(str(claims).encode(), usedforsecurity=False).hexdigest(),
|
||||||
|
},
|
||||||
|
{"kind": "performance", "query": "count_claims", "elapsed_ms": 1.0, "observed_rows": claims},
|
||||||
|
]
|
||||||
|
return "".join(json.dumps(row, sort_keys=True) + "\n" for row in rows)
|
||||||
|
|
||||||
|
|
||||||
|
def capture_fixture(
|
||||||
|
root: Path,
|
||||||
|
*,
|
||||||
|
name: str,
|
||||||
|
captured_at: str,
|
||||||
|
authorization_ref: str,
|
||||||
|
claims: int = 1,
|
||||||
|
constraint_name: str = "claims_pkey",
|
||||||
|
) -> tuple[Path, Path]:
|
||||||
|
manifest = root / f"{name}-manifest.jsonl"
|
||||||
|
manifest.write_text(manifest_text(claims=claims, constraint_name=constraint_name), encoding="utf-8")
|
||||||
|
dump_sha256 = hashlib.sha256(f"{name}-dump".encode()).hexdigest()
|
||||||
|
manifest_sha256 = hashlib.sha256(manifest.read_bytes()).hexdigest()
|
||||||
|
manifest_sql_sha256 = "2" * 64
|
||||||
|
source_context_sha256 = "3" * 64
|
||||||
|
source = {
|
||||||
|
"ssh_target": "root@77.42.65.182",
|
||||||
|
"container": "teleo-pg",
|
||||||
|
"database": "teleo",
|
||||||
|
"service": "leoclean-gateway.service",
|
||||||
|
}
|
||||||
|
healthy_service = {
|
||||||
|
"ActiveState": "active",
|
||||||
|
"SubState": "running",
|
||||||
|
"MainPID": "148735",
|
||||||
|
"NRestarts": "0",
|
||||||
|
}
|
||||||
|
snapshot = {
|
||||||
|
"exported_snapshot_id": "00000003-000001A2-1",
|
||||||
|
"txid_snapshot": "100:120:105,111",
|
||||||
|
"wal_lsn": "0/16B6C50",
|
||||||
|
"system_identifier": "7649789040005668902",
|
||||||
|
"captured_at_utc": captured_at,
|
||||||
|
}
|
||||||
|
receipt = {
|
||||||
|
"artifact": "vps_canonical_postgres_exported_snapshot",
|
||||||
|
"schema": SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
|
||||||
|
"receipt_version": 2,
|
||||||
|
"status": "pass",
|
||||||
|
"run_id": name,
|
||||||
|
"capture_authorization_ref": authorization_ref,
|
||||||
|
"source": source,
|
||||||
|
"snapshot_exported": True,
|
||||||
|
"snapshot": snapshot,
|
||||||
|
"dump": {"sha256": dump_sha256},
|
||||||
|
"manifest": {
|
||||||
|
"sha256": manifest_sha256,
|
||||||
|
"manifest_sql_sha256": manifest_sql_sha256,
|
||||||
|
"table_count": 1,
|
||||||
|
"total_rows": claims,
|
||||||
|
},
|
||||||
|
"source_context": {"sha256": source_context_sha256},
|
||||||
|
"provenance_binding": build_provenance_binding(
|
||||||
|
run_id=name,
|
||||||
|
authorization_ref=authorization_ref,
|
||||||
|
source=source,
|
||||||
|
snapshot=snapshot,
|
||||||
|
dump_sha256=dump_sha256,
|
||||||
|
manifest_sql_sha256=manifest_sql_sha256,
|
||||||
|
source_manifest_sha256=manifest_sha256,
|
||||||
|
source_context_sha256=source_context_sha256,
|
||||||
|
),
|
||||||
|
"service_unchanged": True,
|
||||||
|
"source_service": {"before": healthy_service, "after": healthy_service, "unchanged": True},
|
||||||
|
"production_db_mutated": False,
|
||||||
|
"telegram_send_attempted": False,
|
||||||
|
}
|
||||||
|
receipt_path = root / f"{name}-receipt.json"
|
||||||
|
receipt_path.write_text(json.dumps(receipt, sort_keys=True) + "\n", encoding="utf-8")
|
||||||
|
return receipt_path, manifest
|
||||||
|
|
||||||
|
|
||||||
|
def test_postflight_no_delta_is_bound_and_explicit(tmp_path: Path) -> None:
|
||||||
|
baseline_receipt, baseline_manifest = capture_fixture(
|
||||||
|
tmp_path,
|
||||||
|
name="canonical-baseline",
|
||||||
|
captured_at="2026-07-15T01:00:00+00:00",
|
||||||
|
authorization_ref="codex-delegation:baseline123",
|
||||||
|
)
|
||||||
|
current_receipt, current_manifest = capture_fixture(
|
||||||
|
tmp_path,
|
||||||
|
name="canonical-postflight",
|
||||||
|
captured_at="2026-07-15T02:00:00+00:00",
|
||||||
|
authorization_ref="codex-delegation:postflight123",
|
||||||
|
)
|
||||||
|
|
||||||
|
result = verify_delta(
|
||||||
|
baseline_receipt_path=baseline_receipt,
|
||||||
|
baseline_manifest_path=baseline_manifest,
|
||||||
|
baseline_authorization_ref="codex-delegation:baseline123",
|
||||||
|
current_receipt_path=current_receipt,
|
||||||
|
current_manifest_path=current_manifest,
|
||||||
|
current_authorization_ref="codex-delegation:postflight123",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result["status"] == "pass"
|
||||||
|
assert result["delta"]["delta_detected"] is False
|
||||||
|
assert result["delta"]["total_row_delta"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_postflight_data_and_structure_delta_is_reported_not_hidden(tmp_path: Path) -> None:
|
||||||
|
baseline_receipt, baseline_manifest = capture_fixture(
|
||||||
|
tmp_path,
|
||||||
|
name="canonical-baseline",
|
||||||
|
captured_at="2026-07-15T01:00:00+00:00",
|
||||||
|
authorization_ref="codex-delegation:baseline123",
|
||||||
|
)
|
||||||
|
current_receipt, current_manifest = capture_fixture(
|
||||||
|
tmp_path,
|
||||||
|
name="canonical-postflight",
|
||||||
|
captured_at="2026-07-15T02:00:00+00:00",
|
||||||
|
authorization_ref="codex-delegation:postflight123",
|
||||||
|
claims=2,
|
||||||
|
constraint_name="claims_pkey_v2",
|
||||||
|
)
|
||||||
|
|
||||||
|
result = verify_delta(
|
||||||
|
baseline_receipt_path=baseline_receipt,
|
||||||
|
baseline_manifest_path=baseline_manifest,
|
||||||
|
baseline_authorization_ref="codex-delegation:baseline123",
|
||||||
|
current_receipt_path=current_receipt,
|
||||||
|
current_manifest_path=current_manifest,
|
||||||
|
current_authorization_ref="codex-delegation:postflight123",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result["status"] == "pass"
|
||||||
|
assert result["delta"]["delta_detected"] is True
|
||||||
|
assert result["delta"]["total_row_delta"] == 1
|
||||||
|
assert result["delta"]["changed_tables"][0]["table"] == "public.claims"
|
||||||
|
assert result["delta"]["changed_structures"][0]["kind"] == "constraints"
|
||||||
|
|
||||||
|
|
||||||
|
def test_postflight_rejects_wrong_authorization_or_stale_capture(tmp_path: Path) -> None:
|
||||||
|
baseline_receipt, baseline_manifest = capture_fixture(
|
||||||
|
tmp_path,
|
||||||
|
name="canonical-baseline",
|
||||||
|
captured_at="2026-07-15T02:00:00+00:00",
|
||||||
|
authorization_ref="codex-delegation:baseline123",
|
||||||
|
)
|
||||||
|
current_receipt, current_manifest = capture_fixture(
|
||||||
|
tmp_path,
|
||||||
|
name="canonical-postflight",
|
||||||
|
captured_at="2026-07-15T01:00:00+00:00",
|
||||||
|
authorization_ref="codex-delegation:postflight123",
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="newer than"):
|
||||||
|
verify_delta(
|
||||||
|
baseline_receipt_path=baseline_receipt,
|
||||||
|
baseline_manifest_path=baseline_manifest,
|
||||||
|
baseline_authorization_ref="codex-delegation:baseline123",
|
||||||
|
current_receipt_path=current_receipt,
|
||||||
|
current_manifest_path=current_manifest,
|
||||||
|
current_authorization_ref="codex-delegation:postflight123",
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="authorization"):
|
||||||
|
verify_delta(
|
||||||
|
baseline_receipt_path=baseline_receipt,
|
||||||
|
baseline_manifest_path=baseline_manifest,
|
||||||
|
baseline_authorization_ref="codex-delegation:wrong123",
|
||||||
|
current_receipt_path=current_receipt,
|
||||||
|
current_manifest_path=current_manifest,
|
||||||
|
current_authorization_ref="codex-delegation:postflight123",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_postflight_rejects_unchanged_but_unhealthy_service(tmp_path: Path) -> None:
|
||||||
|
baseline_receipt, baseline_manifest = capture_fixture(
|
||||||
|
tmp_path,
|
||||||
|
name="canonical-baseline",
|
||||||
|
captured_at="2026-07-15T01:00:00+00:00",
|
||||||
|
authorization_ref="codex-delegation:baseline123",
|
||||||
|
)
|
||||||
|
current_receipt, current_manifest = capture_fixture(
|
||||||
|
tmp_path,
|
||||||
|
name="canonical-postflight",
|
||||||
|
captured_at="2026-07-15T02:00:00+00:00",
|
||||||
|
authorization_ref="codex-delegation:postflight123",
|
||||||
|
)
|
||||||
|
current = json.loads(current_receipt.read_text())
|
||||||
|
current["source_service"]["before"]["ActiveState"] = "failed"
|
||||||
|
current["source_service"]["after"]["ActiveState"] = "failed"
|
||||||
|
current_receipt.write_text(json.dumps(current, sort_keys=True) + "\n", encoding="utf-8")
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="source_service_healthy"):
|
||||||
|
verify_delta(
|
||||||
|
baseline_receipt_path=baseline_receipt,
|
||||||
|
baseline_manifest_path=baseline_manifest,
|
||||||
|
baseline_authorization_ref="codex-delegation:baseline123",
|
||||||
|
current_receipt_path=current_receipt,
|
||||||
|
current_manifest_path=current_manifest,
|
||||||
|
current_authorization_ref="codex-delegation:postflight123",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_delta_receipt_writer_keeps_private_mode(tmp_path: Path) -> None:
|
||||||
|
output = tmp_path / "private" / "delta.json"
|
||||||
|
|
||||||
|
write_private_json(output, {"status": "pass"})
|
||||||
|
|
||||||
|
assert output.stat().st_mode & 0o777 == 0o600
|
||||||
|
assert output.parent.stat().st_mode & 0o777 == 0o700
|
||||||
|
|
||||||
|
|
||||||
|
def test_delta_receipt_writer_does_not_chmod_an_existing_parent(tmp_path: Path) -> None:
|
||||||
|
parent = tmp_path / "existing"
|
||||||
|
parent.mkdir(mode=0o755)
|
||||||
|
output = parent / "delta.json"
|
||||||
|
|
||||||
|
write_private_json(output, {"status": "pass"})
|
||||||
|
|
||||||
|
assert parent.stat().st_mode & 0o777 == 0o755
|
||||||
|
assert output.stat().st_mode & 0o777 == 0o600
|
||||||
Loading…
Reference in a new issue