Integrate hardened source-first V3 rebuild foundation
This commit is contained in:
parent
658083e330
commit
0b72b3cbce
50 changed files with 23391 additions and 227 deletions
153
.agents/skills/teleo-corpus-knowledge-rebuild/SKILL.md
Normal file
153
.agents/skills/teleo-corpus-knowledge-rebuild/SKILL.md
Normal file
|
|
@ -0,0 +1,153 @@
|
||||||
|
---
|
||||||
|
name: teleo-corpus-knowledge-rebuild
|
||||||
|
description: Use when locally snapshotting an extracted UTF-8 Forgejo or text corpus into a deterministic Teleo source library, or sharding its verified extraction-work index for parallel processing before any claim generation, review, admission, or database write.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Teleo Corpus Knowledge Rebuild
|
||||||
|
|
||||||
|
## Job
|
||||||
|
|
||||||
|
Turn one local UTF-8 source directory into a deterministic, self-contained
|
||||||
|
source snapshot bundle. This is the source-library phase only: it preserves
|
||||||
|
bytes, paths, hashes, and syntactic section locators without interpreting any
|
||||||
|
section as a claim. It does not perform semantic claim extraction.
|
||||||
|
|
||||||
|
After that inventory verifies, optionally divide its exact work rows into
|
||||||
|
deterministic local shards so independent extractors can process a large corpus
|
||||||
|
in parallel. Sharding schedules work; it does not interpret source content.
|
||||||
|
|
||||||
|
## Operator Path
|
||||||
|
|
||||||
|
From the repository root, build into a path that does not exist:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 ops/run_local_corpus_knowledge_rebuild.py \
|
||||||
|
--source-root tests/fixtures/teleo_corpus_rebuild \
|
||||||
|
--output /private/tmp/teleo-corpus-rebuild-bundle \
|
||||||
|
--mode source-inventory-only
|
||||||
|
```
|
||||||
|
|
||||||
|
Then run the read-only verifier:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 ops/run_local_corpus_knowledge_rebuild.py \
|
||||||
|
--verify-bundle /private/tmp/teleo-corpus-rebuild-bundle
|
||||||
|
```
|
||||||
|
|
||||||
|
For parallel processing, choose a shard count from 1 through 1024 that does not
|
||||||
|
exceed the verified work-item count. Build into another path that does not
|
||||||
|
exist:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 ops/shard_corpus_extraction_work.py \
|
||||||
|
--inventory-bundle /private/tmp/teleo-corpus-rebuild-bundle \
|
||||||
|
--output-dir /private/tmp/teleo-corpus-extraction-shards \
|
||||||
|
--shard-count 64
|
||||||
|
```
|
||||||
|
|
||||||
|
Then verify the shards against the same inventory:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 ops/shard_corpus_extraction_work.py \
|
||||||
|
--inventory-bundle /private/tmp/teleo-corpus-rebuild-bundle \
|
||||||
|
--verify-bundle /private/tmp/teleo-corpus-extraction-shards
|
||||||
|
```
|
||||||
|
|
||||||
|
Inspect these stable entry points:
|
||||||
|
|
||||||
|
```text
|
||||||
|
source-manifest.json
|
||||||
|
extraction-work-index.json
|
||||||
|
receipt.json
|
||||||
|
snapshots/sha256/<prefix>/<sha256>.utf8
|
||||||
|
shard-manifest.json
|
||||||
|
shards/shard-<index>-of-<count>.json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Required Readback
|
||||||
|
|
||||||
|
Require all of these:
|
||||||
|
|
||||||
|
- every source row binds a normalized source-root-relative path to one exact,
|
||||||
|
content-addressed byte snapshot;
|
||||||
|
- every section has exact one-based line and UTF-8 byte ranges plus a quote
|
||||||
|
hash;
|
||||||
|
- every work row is `source_section_for_semantic_extraction`;
|
||||||
|
- `semantic_claim_extraction_performed=false`,
|
||||||
|
`database_write_performed=false`, and `admission_performed=false` throughout;
|
||||||
|
- the receipt says the directory snapshot was not atomic and only guarantees
|
||||||
|
the exact files copied from stable descriptors;
|
||||||
|
- the verifier recomputes snapshots, locators, cardinality, artifact hashes,
|
||||||
|
bundle-tree hash, and receipt hash;
|
||||||
|
- a second run over unchanged inputs produces byte-identical output.
|
||||||
|
- every inventory work item appears in exactly one shard selected by
|
||||||
|
`sha256(utf8(work_item_id)) mod shard_count`;
|
||||||
|
- each shard binds the exact inventory tree, source-manifest, work-index, and
|
||||||
|
inventory-receipt hashes without embedding source-body text or absolute paths;
|
||||||
|
- the shard verifier rejects wrong inventories, missing, duplicate, unknown, or
|
||||||
|
misplaced work IDs, symlinks, extras, incomplete output, and rehashed
|
||||||
|
side-effect overclaims;
|
||||||
|
- shard directories are `0700`, files are `0600`, and repeated runs with the
|
||||||
|
same inventory and shard count are byte-identical.
|
||||||
|
|
||||||
|
Run focused checks:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m pytest -q tests/test_run_local_corpus_knowledge_rebuild.py
|
||||||
|
python3 -m pytest -q tests/test_shard_corpus_extraction_work.py
|
||||||
|
python3 -m ruff check \
|
||||||
|
ops/run_local_corpus_knowledge_rebuild.py \
|
||||||
|
ops/shard_corpus_extraction_work.py \
|
||||||
|
tests/test_run_local_corpus_knowledge_rebuild.py \
|
||||||
|
tests/test_shard_corpus_extraction_work.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Safety Boundary
|
||||||
|
|
||||||
|
The build reserves a new output directory atomically and never replaces an
|
||||||
|
existing destination. It publishes `receipt.json` while the incomplete marker
|
||||||
|
is still present, then removes that marker as the final completion step. It
|
||||||
|
opens source files descriptor-first without following
|
||||||
|
symlinks, streams exact bytes into private content-addressed snapshots, rejects
|
||||||
|
invalid UTF-8, binary controls, lone CR newlines, unsupported entries, path
|
||||||
|
collisions, and configured resource-limit overruns, and deduplicates identical
|
||||||
|
content.
|
||||||
|
|
||||||
|
The defaults are bounded. Raise them only with the explicit `--max-*` options;
|
||||||
|
the command enforces hard upper bounds on every override.
|
||||||
|
|
||||||
|
The sharder independently verifies the inventory before reading work metadata,
|
||||||
|
reserves a new output directory without replacement, writes through held
|
||||||
|
directory descriptors, keeps an incomplete marker through receipt publication,
|
||||||
|
and revalidates the inventory before publishing. It reads no snapshot bodies.
|
||||||
|
Hash shards can be empty; membership is stable only for the exact inventory and
|
||||||
|
the same shard count.
|
||||||
|
|
||||||
|
## Semantic Handoff
|
||||||
|
|
||||||
|
The inventory plus one verified shard is input to a separate semantic
|
||||||
|
extractor, not to a database writer. The extractor must verify both bundles,
|
||||||
|
then resolve each relative snapshot path and exact byte range from the supplied
|
||||||
|
inventory. Only after that extractor has produced explicit, source-backed V3
|
||||||
|
manifests may the operator run:
|
||||||
|
|
||||||
|
```text
|
||||||
|
scripts/compile_kb_source_packet.py
|
||||||
|
```
|
||||||
|
|
||||||
|
That compiler's output still requires separate review and guarded admission.
|
||||||
|
Do not fabricate V3 manifests from raw sections in this inventory helper.
|
||||||
|
|
||||||
|
Use `teleo-reconstruction-recovery` for restore/replay selection and
|
||||||
|
`teleo-kb-db-change-workflow` only for an explicitly authorized later review or
|
||||||
|
apply task.
|
||||||
|
|
||||||
|
## Claim Ceiling
|
||||||
|
|
||||||
|
The output proves deterministic local inventory of the exact copied UTF-8
|
||||||
|
files, bounded syntactic sectioning, bundle integrity, and an exact once-only
|
||||||
|
hash partition of extraction work for a fixed shard count. It does not prove an
|
||||||
|
atomic snapshot of concurrent directory changes, non-empty shards,
|
||||||
|
format-specific syntax, semantic extraction or extractor quality, claim
|
||||||
|
atomicity or correctness, database deduplication, review, admission,
|
||||||
|
application, binary-document extraction, or any live environment.
|
||||||
|
|
@ -39,6 +39,40 @@ A useful composed change must retain:
|
||||||
Do not flatten strategy, governance, concept maps, identity, and reasoning tools
|
Do not flatten strategy, governance, concept maps, identity, and reasoning tools
|
||||||
into generic claim rows merely because the input document discusses them.
|
into generic claim rows merely because the input document discusses them.
|
||||||
|
|
||||||
|
## V3 Source-To-Rebuild Route
|
||||||
|
|
||||||
|
For new reconstruction work, use this order:
|
||||||
|
|
||||||
|
1. Run `ops/run_local_corpus_knowledge_rebuild.py` to retain exact UTF-8
|
||||||
|
snapshots and section work items. Its output contains no semantic claims.
|
||||||
|
2. For a large corpus, run `ops/shard_corpus_extraction_work.py` to create
|
||||||
|
deterministic exact-once work partitions. Sharding schedules extraction; it
|
||||||
|
does not read source bodies or produce claims.
|
||||||
|
3. Have a separately pinned extractor produce an explicit
|
||||||
|
`livingip.kbSourceExtractionManifest.v3`. Every evidence quote must bind to
|
||||||
|
exact UTF-8 byte offsets in the retained extraction.
|
||||||
|
4. Run `scripts/compile_kb_source_packet.py` with the original artifact, exact
|
||||||
|
extracted text, manifest, and extractor specification. The result is
|
||||||
|
build-only and remains `pending_review`.
|
||||||
|
5. On a disposable PostgreSQL clone, stage with
|
||||||
|
`scripts/stage_normalized_proposal.py --compiled-v3-bundle ...`, then render
|
||||||
|
the exact rows with `scripts/kb_proposal_review_packet.py`.
|
||||||
|
6. Record a real review separately with `scripts/approve_proposal.py` and exact
|
||||||
|
`--reviewed-by m3taversal`. Never use test-entered review metadata as
|
||||||
|
evidence that `m3taversal` actually reviewed a production packet.
|
||||||
|
7. Apply only the exact UUID with `scripts/apply_worker.py`, `--enable`,
|
||||||
|
`--contract-version 3`, and `--proposal-id ...`; refuse rather than draining
|
||||||
|
another approved proposal from the queue.
|
||||||
|
8. Export the applied transition with
|
||||||
|
`scripts/export_kb_transition_replay_material.py`, assemble ordered V3
|
||||||
|
materials with `ops/build_v3_genesis_ledger_bundle.py`, and independently
|
||||||
|
replay with `ops/run_local_genesis_ledger_rebuild.py`.
|
||||||
|
|
||||||
|
This repository does not yet provide a general PDF/office extractor or an
|
||||||
|
automatic corpus-wide semantic extractor. For binary originals, retain their
|
||||||
|
exact bytes and pin the separately produced UTF-8 extraction plus extractor
|
||||||
|
specification before step 3.
|
||||||
|
|
||||||
Run the deterministic source-to-proposal canary:
|
Run the deterministic source-to-proposal canary:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|
@ -47,6 +81,17 @@ Run the deterministic source-to-proposal canary:
|
||||||
--output /tmp/leo-local-ingestion-proposal-canary-current.json
|
--output /tmp/leo-local-ingestion-proposal-canary-current.json
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Run the complete disposable V3 source-to-clean-rebuild lifecycle:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python -m pytest -q tests/test_v3_source_rebuild_lifecycle_postgres.py
|
||||||
|
```
|
||||||
|
|
||||||
|
This lifecycle uses a synthetic pinned extractor manifest and synthetic
|
||||||
|
operator-supplied review metadata. It proves the local machinery and exact
|
||||||
|
replay boundary, not semantic extraction quality, real human approval, or a
|
||||||
|
live database mutation.
|
||||||
|
|
||||||
Require the source, claim, evidence, and proposal UUIDs to link exactly; claim
|
Require the source, claim, evidence, and proposal UUIDs to link exactly; claim
|
||||||
and evidence text must be exact substrings of the source; status must remain
|
and evidence text must be exact substrings of the source; status must remain
|
||||||
`pending_review`; the container must use network `none` and leave no volume or
|
`pending_review`; the container must use network `none` and leave no volume or
|
||||||
|
|
@ -54,7 +99,12 @@ container behind.
|
||||||
|
|
||||||
## Canonical Apply Contract
|
## Canonical Apply Contract
|
||||||
|
|
||||||
The strict `approve_claim` v2 payload may create:
|
The strict `approve_claim` v2 contract remains supported for its retained
|
||||||
|
proofs. V2/V3 payloads may create the supported canonical rows for their exact
|
||||||
|
contract version; V3 additionally carries explicit provenance, access,
|
||||||
|
assessment, and admission fields.
|
||||||
|
|
||||||
|
The shared row families include:
|
||||||
|
|
||||||
- `public.claims`
|
- `public.claims`
|
||||||
- `public.sources`
|
- `public.sources`
|
||||||
|
|
@ -64,9 +114,19 @@ The strict `approve_claim` v2 payload may create:
|
||||||
|
|
||||||
The lifecycle is split across:
|
The lifecycle is split across:
|
||||||
|
|
||||||
|
- `scripts/compile_kb_source_packet.py`: hash-bound source plus V3 semantic
|
||||||
|
manifest to a deterministic build-only bundle;
|
||||||
|
- `scripts/stage_normalized_proposal.py`: revalidate and stage an exact bundle
|
||||||
|
only on the named disposable database;
|
||||||
|
- `scripts/kb_proposal_review_packet.py`: read-only review of exact proposed
|
||||||
|
canonical rows;
|
||||||
- `scripts/kb_proposal_normalize.py`: fail-closed rich intent to strict payload;
|
- `scripts/kb_proposal_normalize.py`: fail-closed rich intent to strict payload;
|
||||||
- `scripts/approve_proposal.py`: `kb_review` approves the exact type/payload;
|
- `scripts/approve_proposal.py`: `kb_review` approves the exact type/payload;
|
||||||
- `scripts/apply_proposal.py`: operator-only `kb_apply` writes and verifies;
|
- `scripts/apply_proposal.py`: operator-only `kb_apply` writes and verifies;
|
||||||
|
- `scripts/apply_worker.py`: exact-ID, one-at-a-time V2/V3 worker wrapper;
|
||||||
|
- `scripts/export_kb_transition_replay_material.py`: read-only applied-transition
|
||||||
|
exporter;
|
||||||
|
- `ops/build_v3_genesis_ledger_bundle.py`: V3-only ordered ledger assembler;
|
||||||
- `scripts/kb_apply_prereqs.sql`: roles, immutable approval row, gate functions,
|
- `scripts/kb_apply_prereqs.sql`: roles, immutable approval row, gate functions,
|
||||||
ownership, indexes, and ACLs;
|
ownership, indexes, and ACLs;
|
||||||
- `scripts/run_approve_claim_isolated_container_canary.sh`: disposable
|
- `scripts/run_approve_claim_isolated_container_canary.sh`: disposable
|
||||||
|
|
|
||||||
|
|
@ -90,6 +90,7 @@ Run the deterministic isolated acceptance canary with:
|
||||||
| GCP and Cloud SQL | `teleo-gcp-parity-ops` | Current main evidence; keep open PR #148 candidate-only |
|
| GCP and Cloud SQL | `teleo-gcp-parity-ops` | Current main evidence; keep open PR #148 candidate-only |
|
||||||
| Hermes packaging/runtime | `nousresearch-hermes-agent` | `hermes-agent/` plus the live service-specific ops skill |
|
| Hermes packaging/runtime | `nousresearch-hermes-agent` | `hermes-agent/` plus the live service-specific ops skill |
|
||||||
| Database provenance | `teleo-infra-provenance` and `teleo-db-operator` | Separate Git, runtime, SQLite, Postgres, and retained proof |
|
| Database provenance | `teleo-infra-provenance` and `teleo-db-operator` | Separate Git, runtime, SQLite, Postgres, and retained proof |
|
||||||
|
| Local corpus inventory and extraction preparation | `teleo-corpus-knowledge-rebuild` | Exact UTF-8 snapshots and section work items; semantic claims are a separate reviewed step |
|
||||||
| Source ingestion and proposal/apply | `teleo-kb-db-change-workflow` | Hash-bound source, review, guarded apply, receipt |
|
| Source ingestion and proposal/apply | `teleo-kb-db-change-workflow` | Hash-bound source, review, guarded apply, receipt |
|
||||||
| Reconstruction and recovery | `teleo-reconstruction-recovery` | `docs/kb-rebuild-and-recompile.md` |
|
| Reconstruction and recovery | `teleo-reconstruction-recovery` | `docs/kb-rebuild-and-recompile.md` |
|
||||||
| Identity and rendered soul | `working-leo-m3taversal-outcomes` | Canonical identity rows before `SOUL.md` |
|
| Identity and rendered soul | `working-leo-m3taversal-outcomes` | Canonical identity rows before `SOUL.md` |
|
||||||
|
|
|
||||||
|
|
@ -48,6 +48,11 @@ SEMANTIC_CONTRACT = {
|
||||||
"PR #148 is a separate, open GCP least-privilege candidate",
|
"PR #148 is a separate, open GCP least-privilege candidate",
|
||||||
"PR #148 open candidate only; not canonical main",
|
"PR #148 open candidate only; not canonical main",
|
||||||
),
|
),
|
||||||
|
"local_corpus_source_inventory": (
|
||||||
|
"teleo-corpus-knowledge-rebuild",
|
||||||
|
"It does not perform semantic claim extraction.",
|
||||||
|
"deterministic source snapshots and extraction work items; no claims, proposals, or DB writes",
|
||||||
|
),
|
||||||
"next_safe_action": (
|
"next_safe_action": (
|
||||||
"teleo-leo-onboarding",
|
"teleo-leo-onboarding",
|
||||||
".agents/skills/teleo-leo-onboarding/scripts/validate_skill_pack.py --root .",
|
".agents/skills/teleo-leo-onboarding/scripts/validate_skill_pack.py --root .",
|
||||||
|
|
|
||||||
|
|
@ -30,7 +30,10 @@ as canonical `main` recovery instructions until it merges.
|
||||||
| Need | Path | Honest result |
|
| Need | Path | Honest result |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| Restore the latest verified state quickly | `ops/run_local_canonical_postgres_rebuild.py` | Exact snapshot recovery |
|
| Restore the latest verified state quickly | `ops/run_local_canonical_postgres_rebuild.py` | Exact snapshot recovery |
|
||||||
|
| Snapshot a Forgejo/text corpus for extraction | `ops/run_local_corpus_knowledge_rebuild.py` | Content-addressed UTF-8 sources and work items; no claims |
|
||||||
|
| Partition verified extraction work | `ops/shard_corpus_extraction_work.py` | Deterministic exact-once work shards; no source interpretation |
|
||||||
| Prove strict post-genesis applies replay exactly | `ops/run_local_genesis_ledger_rebuild.py` | Isolated insert-only ledger replay |
|
| Prove strict post-genesis applies replay exactly | `ops/run_local_genesis_ledger_rebuild.py` | Isolated insert-only ledger replay |
|
||||||
|
| Assemble reviewed V3 replay materials | `ops/build_v3_genesis_ledger_bundle.py` | Hash-pinned V3 ledger; optional operator-supplied admission metadata |
|
||||||
| Turn one retained source into a reviewed packet | `scripts/compile_kb_source_packet.py` | Deterministic proposal packet, not canonical knowledge |
|
| Turn one retained source into a reviewed packet | `scripts/compile_kb_source_packet.py` | Deterministic proposal packet, not canonical knowledge |
|
||||||
| Rebuild every row from original sources | Follow the semantic recompilation contract in `docs/kb-rebuild-and-recompile.md` | Partial until every row and receipt is accounted for |
|
| Rebuild every row from original sources | Follow the semantic recompilation contract in `docs/kb-rebuild-and-recompile.md` | Partial until every row and receipt is accounted for |
|
||||||
| Diagnose a live incident before recovery | Use `.agents/skills/teleo-vps-runtime-ops/SKILL.md` and `.agents/skills/teleo-infra-provenance/SKILL.md` | Fresh read-only incident map |
|
| Diagnose a live incident before recovery | Use `.agents/skills/teleo-vps-runtime-ops/SKILL.md` and `.agents/skills/teleo-infra-provenance/SKILL.md` | Fresh read-only incident map |
|
||||||
|
|
@ -40,12 +43,18 @@ as canonical `main` recovery instructions until it merges.
|
||||||
1. Identify the intended source database, snapshot, manifest, commit, and hashes.
|
1. Identify the intended source database, snapshot, manifest, commit, and hashes.
|
||||||
2. Keep private dumps, row payloads, source excerpts, and replay material outside
|
2. Keep private dumps, row payloads, source excerpts, and replay material outside
|
||||||
the repository and mode `0600`.
|
the repository and mode `0600`.
|
||||||
3. Run the selected path in an isolated, network-disabled local container.
|
3. Inventory text sources before semantic extraction. For a binary original,
|
||||||
4. Verify schema, constraints, roles, counts, row hashes, key queries, and exact
|
retain its exact bytes and provide text from a separately pinned extractor;
|
||||||
|
this repository does not yet supply a general PDF/office extractor.
|
||||||
|
4. Build and independently verify the V3 ledger before running genesis replay.
|
||||||
|
Admission metadata is operator-supplied and is not a cryptographic signature
|
||||||
|
from `m3taversal`.
|
||||||
|
5. Run the selected path in an isolated, network-disabled local container.
|
||||||
|
6. Verify schema, constraints, roles, counts, row hashes, key queries, and exact
|
||||||
cleanup.
|
cleanup.
|
||||||
5. Run the relevant focused tests, then the unseen-reasoning verifier when the
|
7. Run the relevant focused tests, then the unseen-reasoning verifier when the
|
||||||
restored state is meant to support Leo answers.
|
restored state is meant to support Leo answers.
|
||||||
6. Separate local reconstruction from VPS restore, GCP restore, service restart,
|
8. Separate local reconstruction from VPS restore, GCP restore, service restart,
|
||||||
Telegram delivery, promotion, and production apply. Each is its own proof row.
|
Telegram delivery, promotion, and production apply. Each is its own proof row.
|
||||||
|
|
||||||
## Commands
|
## Commands
|
||||||
|
|
@ -54,9 +63,16 @@ Use the repository virtual environment documented in `README.md`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
.venv/bin/python ops/run_local_canonical_postgres_rebuild.py --help
|
.venv/bin/python ops/run_local_canonical_postgres_rebuild.py --help
|
||||||
|
.venv/bin/python ops/run_local_corpus_knowledge_rebuild.py --help
|
||||||
|
.venv/bin/python ops/shard_corpus_extraction_work.py --help
|
||||||
|
.venv/bin/python ops/build_v3_genesis_ledger_bundle.py --help
|
||||||
.venv/bin/python ops/run_local_genesis_ledger_rebuild.py --help
|
.venv/bin/python ops/run_local_genesis_ledger_rebuild.py --help
|
||||||
.venv/bin/python scripts/compile_kb_source_packet.py --help
|
.venv/bin/python scripts/compile_kb_source_packet.py --help
|
||||||
.venv/bin/python -m pytest -q \
|
.venv/bin/python -m pytest -q \
|
||||||
|
tests/test_run_local_corpus_knowledge_rebuild.py \
|
||||||
|
tests/test_shard_corpus_extraction_work.py \
|
||||||
|
tests/test_v3_source_rebuild_lifecycle_postgres.py \
|
||||||
|
tests/test_build_v3_genesis_ledger_bundle.py \
|
||||||
tests/test_run_local_canonical_postgres_rebuild.py \
|
tests/test_run_local_canonical_postgres_rebuild.py \
|
||||||
tests/test_run_local_genesis_ledger_rebuild.py \
|
tests/test_run_local_genesis_ledger_rebuild.py \
|
||||||
tests/test_verify_leo_unseen_reasoning_chain.py
|
tests/test_verify_leo_unseen_reasoning_chain.py
|
||||||
|
|
@ -76,6 +92,10 @@ would promote GCP or mutate production without exact authorization.
|
||||||
## Claim Ceiling
|
## Claim Ceiling
|
||||||
|
|
||||||
- Exact snapshot recovery is working when its parity receipt passes.
|
- Exact snapshot recovery is working when its parity receipt passes.
|
||||||
|
- Corpus inventory proves exact copied text bytes and extraction work locations,
|
||||||
|
not semantic claims or binary extraction.
|
||||||
|
- The V3 ledger builder accepts only exact source-normalized applied materials,
|
||||||
|
selects every ordered entry, and never infers admission from proposal status.
|
||||||
- The merged genesis-plus-ledger path proves an isolated insert-only slice.
|
- The merged genesis-plus-ledger path proves an isolated insert-only slice.
|
||||||
- Semantic source-to-blank-database recompilation remains incomplete until every
|
- Semantic source-to-blank-database recompilation remains incomplete until every
|
||||||
canonical row traces to a genesis record or reviewed replay receipt.
|
canonical row traces to a genesis record or reviewed replay receipt.
|
||||||
|
|
|
||||||
2
.github/workflows/ci.yml
vendored
2
.github/workflows/ci.yml
vendored
|
|
@ -160,6 +160,8 @@ jobs:
|
||||||
timeout-minutes: 20
|
timeout-minutes: 20
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@v4
|
||||||
|
with:
|
||||||
|
fetch-depth: 0
|
||||||
- uses: actions/setup-python@v5
|
- uses: actions/setup-python@v5
|
||||||
with:
|
with:
|
||||||
python-version: ${{ env.PYTHON_VERSION }}
|
python-version: ${{ env.PYTHON_VERSION }}
|
||||||
|
|
|
||||||
|
|
@ -96,6 +96,104 @@ The remaining gaps are concrete rather than mysterious:
|
||||||
This proves that most of the initial database came from the retained file-KB
|
This proves that most of the initial database came from the retained file-KB
|
||||||
import path. It does not yet prove a clean blank-database recompile.
|
import path. It does not yet prove a clean blank-database recompile.
|
||||||
|
|
||||||
|
## Genesis-Backed Source Lifecycle: Working T2 Canary
|
||||||
|
|
||||||
|
`scripts/run_leo_integrated_learning_lifecycle.py` can now start from one
|
||||||
|
retained real genesis dump rather than its legacy synthetic fixture schema. The
|
||||||
|
genesis mode is all-or-nothing: the dump, parity manifest, repository-pinned
|
||||||
|
trusted exporter receipt, and a digest-pinned PostgreSQL image are mandatory.
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/run_leo_integrated_learning_lifecycle.py \
|
||||||
|
--source-packet /private/path/source-packet \
|
||||||
|
--genesis-dump /private/path/genesis.dump \
|
||||||
|
--genesis-manifest /private/path/genesis-manifest.jsonl \
|
||||||
|
--genesis-exporter-receipt /private/path/exporter-receipt.json \
|
||||||
|
--image docker.io/library/postgres:16-alpine@sha256:57c72fd2a128e416c7fcc499958864df5301e940bca0a56f58fddf30ffc07777 \
|
||||||
|
--review-action approve \
|
||||||
|
--review-note "Approved exact retained payload for disposable T2 verification only." \
|
||||||
|
--output /private/path/leo-genesis-lifecycle-receipt.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Before the first Docker command, the runner copies the dump, manifest, exporter
|
||||||
|
receipt, source packet, controller, subprocess engines, and their transitive
|
||||||
|
repository imports into one mode `0500` private staging directory with mode
|
||||||
|
`0400` files. It validates the custom-dump signature; the exact
|
||||||
|
repository-pinned exporter-receipt hash, source identity, capture authorization,
|
||||||
|
and provenance binding; exporter-bound dump and manifest bytes; manifest
|
||||||
|
contract; image digest; executing Git commit; complete engine dependency hash
|
||||||
|
set; and a fresh deterministic compile against the retained
|
||||||
|
`proposal-packet.json`. Every enumerated engine file must be tracked at that
|
||||||
|
commit, its working-tree bytes must equal the commit blob, and its staged hash
|
||||||
|
must equal the same commit-bound hash. The compiler performs no model calls. Any
|
||||||
|
untrusted or incomplete exporter contract, moving image tag, engine drift,
|
||||||
|
staged-byte drift, invalid manifest, retained-packet drift, or output/input
|
||||||
|
collision fails before a container can start.
|
||||||
|
|
||||||
|
The child tools consume staged paths, and both runs bind their consumed file
|
||||||
|
set, engine set, and commit back to the same preflight manifest. The already
|
||||||
|
running Python controller imports repository modules before it creates staging;
|
||||||
|
the proof therefore establishes commit-matched imported dependencies and
|
||||||
|
byte-identical staged copies, not a claim that the controller process itself was
|
||||||
|
re-executed from staging. Successful public receipts retain logical names,
|
||||||
|
hashes, counts, and cleanup evidence without host-specific absolute paths.
|
||||||
|
|
||||||
|
Each of two independent runs then:
|
||||||
|
|
||||||
|
1. starts a uniquely labeled, networkless PostgreSQL container with tmpfs data;
|
||||||
|
2. creates only manifest-declared global roles, restores the supplied custom
|
||||||
|
dump directly, replays captured database ACLs, and proves the full parity
|
||||||
|
manifest before installing the current guarded-apply prerequisites;
|
||||||
|
3. skips the synthetic schema bootstrap and performs no canonical preseed;
|
||||||
|
4. proves the strict proposal, immutable approval, and every payload-owned
|
||||||
|
claim/source/evidence/edge/tool target are absent before staging;
|
||||||
|
5. stages `pending_review`, explicitly approves as the independent review role,
|
||||||
|
and drives the real apply worker through an exact-candidate report-only tick,
|
||||||
|
an enabled pre-commit failure with persisted retry state, a successful
|
||||||
|
password-authenticated retry with a mode `0600` replay receipt, and a
|
||||||
|
post-apply no-op tick;
|
||||||
|
6. recompiles the retained packet identically and fingerprints only fields
|
||||||
|
controlled by its immutable apply payload; and
|
||||||
|
7. removes the container and private temporary root, then proves its exact
|
||||||
|
instance and shared two-run label have no remaining containers.
|
||||||
|
|
||||||
|
The two-run comparison excludes generated evidence/edge IDs, timestamps,
|
||||||
|
container identity, and the PostgreSQL system identifier. It requires the
|
||||||
|
source hashes, proposal and payload identities, phase states, restored key
|
||||||
|
counts, row counts, recompile hash, and payload-controlled row projections to
|
||||||
|
match exactly.
|
||||||
|
|
||||||
|
Older retained parity manifests may predate explicit
|
||||||
|
`application_roles[].settings` and `database_settings` arrays. Their raw bytes
|
||||||
|
and trusted exporter binding remain mandatory. For role creation and current
|
||||||
|
parity SQL, the runner narrowly projects missing arrays from the captured
|
||||||
|
`default_transaction_read_only` field (or an empty list), without changing the
|
||||||
|
retained bytes. Such a run may prove data/schema parity and captured role
|
||||||
|
attributes, but its receipt reports full role/global parity as `not_proven`.
|
||||||
|
Only a source manifest that actually captured both settings arrays can earn the
|
||||||
|
full role/global parity claim.
|
||||||
|
|
||||||
|
The focused Docker regression constructs a real custom-format genesis with
|
||||||
|
nondefault role and per-database settings, restores it twice, executes the real
|
||||||
|
worker lifecycle, compares post-apply fingerprints, and verifies container and
|
||||||
|
staging cleanup:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest -q \
|
||||||
|
tests/test_run_leo_integrated_learning_lifecycle.py \
|
||||||
|
-k test_real_docker_genesis_restore_worker_parity_and_repeatability
|
||||||
|
```
|
||||||
|
|
||||||
|
The detailed receipt can contain proposal payloads and canonical row excerpts.
|
||||||
|
It is written atomically as mode `0600` and must remain private. Standard output
|
||||||
|
contains only the artifact/status, receipt basename and SHA-256, aggregate
|
||||||
|
problem/check counts, and error type. A passing receipt proves T2 disposable
|
||||||
|
runtime reconstruction only. It does not prove or permit VPS/GCP mutation,
|
||||||
|
production access, live Telegram behavior, or a blank-schema rebuild of all
|
||||||
|
historical knowledge.
|
||||||
|
|
||||||
## Target Compiler
|
## Target Compiler
|
||||||
|
|
||||||
The durable rebuild model is:
|
The durable rebuild model is:
|
||||||
|
|
@ -148,6 +246,24 @@ not retain every column of the proposal ledger, so exact reconstruction also
|
||||||
needs the full approved proposal row, immutable approval snapshot, and final
|
needs the full approved proposal row, immutable approval snapshot, and final
|
||||||
applied proposal row.
|
applied proposal row.
|
||||||
|
|
||||||
|
New approvals retain the exact post-approval `kb_proposals` row inside
|
||||||
|
`kb_proposal_approvals`, including `updated_at`. After the worker applies and
|
||||||
|
writes its replay receipt, export the seven-field private material without any
|
||||||
|
pre-apply filesystem capture:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/export_kb_transition_replay_material.py <proposal-uuid> \
|
||||||
|
--replay-receipt /private/path/<proposal-uuid>.json \
|
||||||
|
--sequence 1 \
|
||||||
|
--output /private/path/entry-0001.json
|
||||||
|
```
|
||||||
|
|
||||||
|
The exporter uses the restricted `kb_apply` read function in a read-only
|
||||||
|
transaction, validates exact approved/approval/applied identities and the
|
||||||
|
worker receipt, and publishes a mode-`0600` file once. An identical rerun is a
|
||||||
|
no-op; drift or output aliasing is refused. Already-applied legacy approvals
|
||||||
|
whose exact approved row was never retained remain explicitly unavailable.
|
||||||
|
|
||||||
## Genesis Plus Strict Ledger: Working Deterministic Slice
|
## Genesis Plus Strict Ledger: Working Deterministic Slice
|
||||||
|
|
||||||
`ops/run_local_genesis_ledger_rebuild.py` now executes the first exact
|
`ops/run_local_genesis_ledger_rebuild.py` now executes the first exact
|
||||||
|
|
@ -251,6 +367,24 @@ The exact v1 claim ceiling is intentionally bounded:
|
||||||
- this proves only an isolated local reconstruction. It does not touch or prove
|
- this proves only an isolated local reconstruction. It does not touch or prove
|
||||||
VPS, GCP, Telegram, a live database, or blank-schema source recompilation.
|
VPS, GCP, Telegram, a live database, or blank-schema source recompilation.
|
||||||
|
|
||||||
|
V3 `approve_claim` receipts extend the same exact-ledger boundary to admitted
|
||||||
|
claim versions, content-addressed source versions, located evidence, typed
|
||||||
|
edges, and evidence assessments. Ledger order is also the schema-cutover order:
|
||||||
|
all V2 entries must replay before the first V3 entry, the hash-pinned V3 contract
|
||||||
|
is installed once at that boundary, guarded prerequisites are rerun, and any V2
|
||||||
|
entry after cutover fails before replay. Historical V2 proposals already present
|
||||||
|
in genesis remain visible; the reconstruction does not silently apply, delete,
|
||||||
|
or reinterpret them.
|
||||||
|
|
||||||
|
Receipt-pinned V3 rows are seeded before the idempotent guarded apply, then read
|
||||||
|
back exactly. PostgreSQL generates the proposal's apply timestamp, so the
|
||||||
|
disposable reconstruction takes an exclusive proposal-table lock, verifies the
|
||||||
|
exact V3 decision trigger is enabled, transactionally disables only that trigger,
|
||||||
|
normalizes the receipt-pinned proposal timestamps as `postgres`, re-enables the
|
||||||
|
trigger, and verifies it is enabled before commit. Any failure rolls the trigger
|
||||||
|
state and timestamp update back together. This is a disposable reconstruction
|
||||||
|
mechanism, not a production write path or a relaxation of the live V3 contract.
|
||||||
|
|
||||||
The transition contract for `revise_strategy` is final-state deterministic, not
|
The transition contract for `revise_strategy` is final-state deterministic, not
|
||||||
a claim that the v1 receipt independently contains a historical before-image:
|
a claim that the v1 receipt independently contains a historical before-image:
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
{
|
{
|
||||||
"pack": "leo-teleo-skill-pack",
|
"pack": "leo-teleo-skill-pack",
|
||||||
"created_date": "2026-07-09",
|
"created_date": "2026-07-09",
|
||||||
"last_updated_utc": "2026-07-15T00:16:53Z",
|
"last_updated_utc": "2026-07-18T22:02:36Z",
|
||||||
"status": "repo_native_path_validated",
|
"status": "repo_native_path_validated",
|
||||||
"contains_secrets": false,
|
"contains_secrets": false,
|
||||||
"production_mutation_authorized": false,
|
"production_mutation_authorized": false,
|
||||||
|
|
@ -46,6 +46,11 @@
|
||||||
"role": "Snapshot recovery, genesis-ledger replay, source recompilation, and incident recovery",
|
"role": "Snapshot recovery, genesis-ledger replay, source recompilation, and incident recovery",
|
||||||
"path": ".agents/skills/teleo-reconstruction-recovery/SKILL.md"
|
"path": ".agents/skills/teleo-reconstruction-recovery/SKILL.md"
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"name": "teleo-corpus-knowledge-rebuild",
|
||||||
|
"role": "Deterministic UTF-8 source snapshots and semantic-extraction work indexes; no claim generation",
|
||||||
|
"path": ".agents/skills/teleo-corpus-knowledge-rebuild/SKILL.md"
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"name": "leo-telegram-canary-ops",
|
"name": "leo-telegram-canary-ops",
|
||||||
"role": "Live Telegram bot testing through authenticated Chrome and Computer Use",
|
"role": "Live Telegram bot testing through authenticated Chrome and Computer Use",
|
||||||
|
|
@ -103,9 +108,9 @@
|
||||||
"gcp": ["teleo-gcp-parity-ops"],
|
"gcp": ["teleo-gcp-parity-ops"],
|
||||||
"hermes_runtime": ["nousresearch-hermes-agent", "teleo-vps-runtime-ops"],
|
"hermes_runtime": ["nousresearch-hermes-agent", "teleo-vps-runtime-ops"],
|
||||||
"database_provenance": ["teleo-infra-provenance", "teleo-db-operator"],
|
"database_provenance": ["teleo-infra-provenance", "teleo-db-operator"],
|
||||||
"ingestion": ["teleo-kb-db-change-workflow", "living-ip-kb-interop"],
|
"ingestion": ["teleo-corpus-knowledge-rebuild", "teleo-kb-db-change-workflow", "living-ip-kb-interop"],
|
||||||
"proposal_apply": ["teleo-kb-db-change-workflow"],
|
"proposal_apply": ["teleo-kb-db-change-workflow"],
|
||||||
"reconstruction": ["teleo-reconstruction-recovery"],
|
"reconstruction": ["teleo-corpus-knowledge-rebuild", "teleo-reconstruction-recovery"],
|
||||||
"identity": ["teleo-leo-onboarding", "working-leo-m3taversal-outcomes"],
|
"identity": ["teleo-leo-onboarding", "working-leo-m3taversal-outcomes"],
|
||||||
"testing": ["teleo-leo-onboarding", "teleo-proof-handoff", "crabbox", "decision-engine-refinement"],
|
"testing": ["teleo-leo-onboarding", "teleo-proof-handoff", "crabbox", "decision-engine-refinement"],
|
||||||
"security": ["teleo-vps-runtime-ops", "teleo-gcp-parity-ops", "private-password-storage"],
|
"security": ["teleo-vps-runtime-ops", "teleo-gcp-parity-ops", "private-password-storage"],
|
||||||
|
|
@ -134,6 +139,9 @@
|
||||||
".agents/skills/teleo-leo-onboarding/scripts/run_clean_context_canary.py",
|
".agents/skills/teleo-leo-onboarding/scripts/run_clean_context_canary.py",
|
||||||
".agents/skills/teleo-leo-onboarding/scripts/validate_skill_pack.py",
|
".agents/skills/teleo-leo-onboarding/scripts/validate_skill_pack.py",
|
||||||
"ops/run_local_canonical_postgres_rebuild.py",
|
"ops/run_local_canonical_postgres_rebuild.py",
|
||||||
|
"ops/run_local_corpus_knowledge_rebuild.py",
|
||||||
|
"ops/shard_corpus_extraction_work.py",
|
||||||
|
"ops/build_v3_genesis_ledger_bundle.py",
|
||||||
"ops/run_local_genesis_ledger_rebuild.py",
|
"ops/run_local_genesis_ledger_rebuild.py",
|
||||||
"ops/capture_vps_canonical_postgres_snapshot.py",
|
"ops/capture_vps_canonical_postgres_snapshot.py",
|
||||||
"ops/restore_gcp_generated_postgres_snapshot.py",
|
"ops/restore_gcp_generated_postgres_snapshot.py",
|
||||||
|
|
|
||||||
|
|
@ -374,9 +374,23 @@ def sql_array(values: list[str], cast: str = "text") -> str:
|
||||||
|
|
||||||
def psql_json(args: argparse.Namespace, sql: str) -> list[dict[str, Any]]:
|
def psql_json(args: argparse.Namespace, sql: str) -> list[dict[str, Any]]:
|
||||||
if args.local:
|
if args.local:
|
||||||
command = ["docker", "exec", "-i", args.container, "psql", "-U", "postgres", "-d", args.db, "-At", "-q"]
|
command = [
|
||||||
|
"docker",
|
||||||
|
"exec",
|
||||||
|
"-i",
|
||||||
|
args.container,
|
||||||
|
"psql",
|
||||||
|
"-U",
|
||||||
|
"postgres",
|
||||||
|
"-d",
|
||||||
|
args.db,
|
||||||
|
"-At",
|
||||||
|
"-q",
|
||||||
|
"-v",
|
||||||
|
"ON_ERROR_STOP=1",
|
||||||
|
]
|
||||||
else:
|
else:
|
||||||
remote = f"docker exec -i {args.container} psql -U postgres -d {args.db} -At -q"
|
remote = f"docker exec -i {args.container} psql -U postgres -d {args.db} -At -q -v ON_ERROR_STOP=1"
|
||||||
command = ["ssh", "-o", "ConnectTimeout=15", "-o", "BatchMode=yes", args.ssh, remote]
|
command = ["ssh", "-o", "ConnectTimeout=15", "-o", "BatchMode=yes", args.ssh, remote]
|
||||||
result = subprocess.run(command, text=True, capture_output=True, input=sql, check=False)
|
result = subprocess.run(command, text=True, capture_output=True, input=sql, check=False)
|
||||||
if result.returncode != 0:
|
if result.returncode != 0:
|
||||||
|
|
|
||||||
1101
ops/build_v3_genesis_ledger_bundle.py
Normal file
1101
ops/build_v3_genesis_ledger_bundle.py
Normal file
File diff suppressed because it is too large
Load diff
357
ops/rollback_teleo_v3_epistemic_contract.sql
Normal file
357
ops/rollback_teleo_v3_epistemic_contract.sql
Normal file
|
|
@ -0,0 +1,357 @@
|
||||||
|
-- Schema-only rollback for ops/teleo_v3_epistemic_contract.sql.
|
||||||
|
--
|
||||||
|
-- The rollback refuses to discard proposal-linked V3 data. A disposable
|
||||||
|
-- verifier may delete its own canary rows first; a real migration needs an
|
||||||
|
-- explicit export/port decision before this file can run.
|
||||||
|
|
||||||
|
\set ON_ERROR_STOP on
|
||||||
|
set timezone = 'UTC';
|
||||||
|
|
||||||
|
begin;
|
||||||
|
|
||||||
|
do $ownership_guard$
|
||||||
|
declare
|
||||||
|
v_artifacts_present boolean;
|
||||||
|
v_ledger_owner_valid boolean;
|
||||||
|
v_ledger_row_valid boolean;
|
||||||
|
v_missing_objects text;
|
||||||
|
begin
|
||||||
|
select
|
||||||
|
pg_catalog.to_regclass('kb_stage.teleo_v3_installation_ownership') is not null
|
||||||
|
or pg_catalog.to_regclass('public.claim_evidence_assessments') is not null
|
||||||
|
or exists (
|
||||||
|
select 1
|
||||||
|
from information_schema.columns columns
|
||||||
|
where columns.table_schema = 'public'
|
||||||
|
and (columns.table_name, columns.column_name) in (
|
||||||
|
('claims', 'owner_agent_id'),
|
||||||
|
('claims', 'proposition'),
|
||||||
|
('claims', 'body_markdown'),
|
||||||
|
('claims', 'scope'),
|
||||||
|
('claims', 'access_scope'),
|
||||||
|
('claims', 'admission_state'),
|
||||||
|
('claims', 'revision_criteria'),
|
||||||
|
('claims', 'revision_kind'),
|
||||||
|
('claims', 'supersedes_id'),
|
||||||
|
('claims', 'accepted_by_proposal_id'),
|
||||||
|
('sources', 'canonical_url'),
|
||||||
|
('sources', 'storage_uri'),
|
||||||
|
('sources', 'content_hash'),
|
||||||
|
('sources', 'access_scope'),
|
||||||
|
('sources', 'ingestion_origin'),
|
||||||
|
('sources', 'provenance_status'),
|
||||||
|
('sources', 'source_class'),
|
||||||
|
('sources', 'auto_promotion_policy'),
|
||||||
|
('sources', 'accepted_by_proposal_id'),
|
||||||
|
('claim_evidence', 'excerpt'),
|
||||||
|
('claim_evidence', 'locator_json'),
|
||||||
|
('claim_evidence', 'source_content_hash'),
|
||||||
|
('claim_evidence', 'rationale'),
|
||||||
|
('claim_evidence', 'polarity'),
|
||||||
|
('claim_evidence', 'accepted_from_proposal_id'),
|
||||||
|
('claim_edges', 'relation_type'),
|
||||||
|
('claim_edges', 'rationale'),
|
||||||
|
('claim_edges', 'scope_or_conditions'),
|
||||||
|
('claim_edges', 'accepted_by_proposal_id')
|
||||||
|
)
|
||||||
|
)
|
||||||
|
or exists (
|
||||||
|
select 1
|
||||||
|
from pg_catalog.pg_proc procedure
|
||||||
|
join pg_catalog.pg_namespace namespace on namespace.oid = procedure.pronamespace
|
||||||
|
where namespace.nspname = 'kb_stage'
|
||||||
|
and procedure.proname like 'teleo\_v3\_%' escape '\'
|
||||||
|
)
|
||||||
|
or exists (
|
||||||
|
select 1
|
||||||
|
from pg_catalog.pg_trigger trigger_row
|
||||||
|
where not trigger_row.tgisinternal
|
||||||
|
and trigger_row.tgname like 'teleo\_v3\_%' escape '\'
|
||||||
|
)
|
||||||
|
or exists (
|
||||||
|
select 1
|
||||||
|
from pg_catalog.pg_constraint constraint_row
|
||||||
|
where constraint_row.conname like 'teleo\_v3\_%' escape '\'
|
||||||
|
)
|
||||||
|
or exists (
|
||||||
|
select 1
|
||||||
|
from pg_catalog.pg_class index_row
|
||||||
|
where index_row.relkind = 'i'
|
||||||
|
and index_row.relname like 'teleo\_v3\_%' escape '\'
|
||||||
|
)
|
||||||
|
into v_artifacts_present;
|
||||||
|
|
||||||
|
if pg_catalog.to_regclass('kb_stage.teleo_v3_installation_ownership') is null then
|
||||||
|
if v_artifacts_present then
|
||||||
|
raise exception using
|
||||||
|
errcode = '55000',
|
||||||
|
message = 'teleo_v3 rollback refused: V3-named objects lack exact installation ownership';
|
||||||
|
end if;
|
||||||
|
return;
|
||||||
|
end if;
|
||||||
|
|
||||||
|
select owner_role.rolname = 'kb_gate_owner'
|
||||||
|
into v_ledger_owner_valid
|
||||||
|
from pg_catalog.pg_class relation
|
||||||
|
join pg_catalog.pg_namespace namespace on namespace.oid = relation.relnamespace
|
||||||
|
join pg_catalog.pg_roles owner_role on owner_role.oid = relation.relowner
|
||||||
|
where namespace.nspname = 'kb_stage'
|
||||||
|
and relation.relname = 'teleo_v3_installation_ownership'
|
||||||
|
and relation.relkind in ('r', 'p');
|
||||||
|
if not coalesce(v_ledger_owner_valid, false) then
|
||||||
|
raise exception using
|
||||||
|
errcode = '55000',
|
||||||
|
message = 'teleo_v3 rollback refused: installation ownership ledger owner is invalid';
|
||||||
|
end if;
|
||||||
|
|
||||||
|
execute $ownership_query$
|
||||||
|
select count(*) = 1
|
||||||
|
and bool_and(
|
||||||
|
contract_name = 'livingip.teleo-v3-epistemic-contract'
|
||||||
|
and contract_version = 1
|
||||||
|
and installation_token = 'collision-free-install-v1'
|
||||||
|
)
|
||||||
|
from kb_stage.teleo_v3_installation_ownership
|
||||||
|
$ownership_query$ into v_ledger_row_valid;
|
||||||
|
if not coalesce(v_ledger_row_valid, false) then
|
||||||
|
raise exception using
|
||||||
|
errcode = '55000',
|
||||||
|
message = 'teleo_v3 rollback refused: installation ownership ledger is not exact';
|
||||||
|
end if;
|
||||||
|
|
||||||
|
with expected_columns(table_name, column_name) as (
|
||||||
|
values
|
||||||
|
('claims', 'owner_agent_id'),
|
||||||
|
('claims', 'proposition'),
|
||||||
|
('claims', 'body_markdown'),
|
||||||
|
('claims', 'scope'),
|
||||||
|
('claims', 'access_scope'),
|
||||||
|
('claims', 'admission_state'),
|
||||||
|
('claims', 'revision_criteria'),
|
||||||
|
('claims', 'revision_kind'),
|
||||||
|
('claims', 'supersedes_id'),
|
||||||
|
('claims', 'accepted_by_proposal_id'),
|
||||||
|
('sources', 'canonical_url'),
|
||||||
|
('sources', 'storage_uri'),
|
||||||
|
('sources', 'content_hash'),
|
||||||
|
('sources', 'access_scope'),
|
||||||
|
('sources', 'ingestion_origin'),
|
||||||
|
('sources', 'provenance_status'),
|
||||||
|
('sources', 'source_class'),
|
||||||
|
('sources', 'auto_promotion_policy'),
|
||||||
|
('sources', 'accepted_by_proposal_id'),
|
||||||
|
('claim_evidence', 'excerpt'),
|
||||||
|
('claim_evidence', 'locator_json'),
|
||||||
|
('claim_evidence', 'source_content_hash'),
|
||||||
|
('claim_evidence', 'rationale'),
|
||||||
|
('claim_evidence', 'polarity'),
|
||||||
|
('claim_evidence', 'accepted_from_proposal_id'),
|
||||||
|
('claim_edges', 'relation_type'),
|
||||||
|
('claim_edges', 'rationale'),
|
||||||
|
('claim_edges', 'scope_or_conditions'),
|
||||||
|
('claim_edges', 'accepted_by_proposal_id')
|
||||||
|
), missing(object_name) as (
|
||||||
|
select pg_catalog.format('%I.%I', expected.table_name, expected.column_name)
|
||||||
|
from expected_columns expected
|
||||||
|
where not exists (
|
||||||
|
select 1
|
||||||
|
from information_schema.columns columns
|
||||||
|
where columns.table_schema = 'public'
|
||||||
|
and columns.table_name = expected.table_name
|
||||||
|
and columns.column_name = expected.column_name
|
||||||
|
)
|
||||||
|
union all
|
||||||
|
select 'public.claim_evidence_assessments'
|
||||||
|
where pg_catalog.to_regclass('public.claim_evidence_assessments') is null
|
||||||
|
)
|
||||||
|
select pg_catalog.string_agg(object_name, ', ' order by object_name)
|
||||||
|
into v_missing_objects
|
||||||
|
from missing;
|
||||||
|
if v_missing_objects is not null then
|
||||||
|
raise exception using
|
||||||
|
errcode = '55000',
|
||||||
|
message = pg_catalog.format(
|
||||||
|
'teleo_v3 rollback refused: installation-owned object set is incomplete: %s',
|
||||||
|
v_missing_objects
|
||||||
|
);
|
||||||
|
end if;
|
||||||
|
end
|
||||||
|
$ownership_guard$;
|
||||||
|
|
||||||
|
do $data_guard$
|
||||||
|
declare
|
||||||
|
contract_rows_exist boolean := false;
|
||||||
|
begin
|
||||||
|
if to_regclass('public.claim_evidence_assessments') is not null then
|
||||||
|
execute 'select exists (select 1 from public.claim_evidence_assessments)'
|
||||||
|
into contract_rows_exist;
|
||||||
|
if contract_rows_exist then
|
||||||
|
raise exception using
|
||||||
|
errcode = '55000',
|
||||||
|
message = 'teleo_v3 rollback refused: claim_evidence_assessments contains V3 data';
|
||||||
|
end if;
|
||||||
|
end if;
|
||||||
|
|
||||||
|
if exists (
|
||||||
|
select 1 from information_schema.columns
|
||||||
|
where table_schema = 'public' and table_name = 'claim_edges'
|
||||||
|
and column_name = 'accepted_by_proposal_id'
|
||||||
|
) then
|
||||||
|
execute 'select exists (select 1 from public.claim_edges where accepted_by_proposal_id is not null)'
|
||||||
|
into contract_rows_exist;
|
||||||
|
if contract_rows_exist then
|
||||||
|
raise exception using
|
||||||
|
errcode = '55000',
|
||||||
|
message = 'teleo_v3 rollback refused: claim_edges contains proposal-linked V3 data';
|
||||||
|
end if;
|
||||||
|
end if;
|
||||||
|
|
||||||
|
if exists (
|
||||||
|
select 1 from information_schema.columns
|
||||||
|
where table_schema = 'public' and table_name = 'claim_evidence'
|
||||||
|
and column_name = 'accepted_from_proposal_id'
|
||||||
|
) then
|
||||||
|
execute 'select exists (select 1 from public.claim_evidence where accepted_from_proposal_id is not null)'
|
||||||
|
into contract_rows_exist;
|
||||||
|
if contract_rows_exist then
|
||||||
|
raise exception using
|
||||||
|
errcode = '55000',
|
||||||
|
message = 'teleo_v3 rollback refused: claim_evidence contains proposal-linked V3 data';
|
||||||
|
end if;
|
||||||
|
end if;
|
||||||
|
|
||||||
|
if exists (
|
||||||
|
select 1 from information_schema.columns
|
||||||
|
where table_schema = 'public' and table_name = 'sources'
|
||||||
|
and column_name = 'accepted_by_proposal_id'
|
||||||
|
) then
|
||||||
|
execute 'select exists (select 1 from public.sources where accepted_by_proposal_id is not null)'
|
||||||
|
into contract_rows_exist;
|
||||||
|
if contract_rows_exist then
|
||||||
|
raise exception using
|
||||||
|
errcode = '55000',
|
||||||
|
message = 'teleo_v3 rollback refused: sources contains proposal-linked V3 data';
|
||||||
|
end if;
|
||||||
|
end if;
|
||||||
|
|
||||||
|
if exists (
|
||||||
|
select 1 from information_schema.columns
|
||||||
|
where table_schema = 'public' and table_name = 'claims'
|
||||||
|
and column_name = 'admission_state'
|
||||||
|
) then
|
||||||
|
execute $query$
|
||||||
|
select exists (
|
||||||
|
select 1 from public.claims
|
||||||
|
where admission_state = 'admitted' or accepted_by_proposal_id is not null
|
||||||
|
)
|
||||||
|
$query$ into contract_rows_exist;
|
||||||
|
if contract_rows_exist then
|
||||||
|
raise exception using
|
||||||
|
errcode = '55000',
|
||||||
|
message = 'teleo_v3 rollback refused: claims contains admitted or proposal-linked V3 data';
|
||||||
|
end if;
|
||||||
|
end if;
|
||||||
|
end
|
||||||
|
$data_guard$;
|
||||||
|
|
||||||
|
drop trigger if exists teleo_v3_guard_linked_proposal_decision on kb_stage.kb_proposals;
|
||||||
|
drop trigger if exists teleo_v3_00_enforce_apply_contract_cutover on kb_stage.kb_proposals;
|
||||||
|
alter table kb_stage.kb_proposals
|
||||||
|
drop constraint if exists teleo_v3_proposal_apply_receipt_check;
|
||||||
|
|
||||||
|
drop trigger if exists teleo_v3_edge_immutable on public.claim_edges;
|
||||||
|
drop trigger if exists teleo_v3_edge_reject_delete on public.claim_edges;
|
||||||
|
drop trigger if exists teleo_v3_edge_validate_endpoints on public.claim_edges;
|
||||||
|
drop trigger if exists teleo_v3_edge_require_accepted_proposal on public.claim_edges;
|
||||||
|
drop trigger if exists teleo_v3_00_edge_require_linked_insert on public.claim_edges;
|
||||||
|
|
||||||
|
drop trigger if exists teleo_v3_evidence_immutable on public.claim_evidence;
|
||||||
|
drop trigger if exists teleo_v3_evidence_reject_delete on public.claim_evidence;
|
||||||
|
drop trigger if exists teleo_v3_evidence_validate_provenance on public.claim_evidence;
|
||||||
|
drop trigger if exists teleo_v3_evidence_require_accepted_proposal on public.claim_evidence;
|
||||||
|
drop trigger if exists teleo_v3_00_evidence_require_linked_insert on public.claim_evidence;
|
||||||
|
|
||||||
|
drop trigger if exists teleo_v3_source_immutable on public.sources;
|
||||||
|
drop trigger if exists teleo_v3_source_reject_delete on public.sources;
|
||||||
|
drop trigger if exists teleo_v3_source_require_accepted_proposal on public.sources;
|
||||||
|
drop trigger if exists teleo_v3_00_source_require_linked_insert on public.sources;
|
||||||
|
|
||||||
|
drop trigger if exists teleo_v3_claim_immutable on public.claims;
|
||||||
|
drop trigger if exists teleo_v3_claim_reject_delete on public.claims;
|
||||||
|
drop trigger if exists teleo_v3_claim_require_accepted_proposal on public.claims;
|
||||||
|
drop trigger if exists teleo_v3_00_claim_require_linked_insert on public.claims;
|
||||||
|
|
||||||
|
drop index if exists public.teleo_v3_edge_typed_adjacency_idx;
|
||||||
|
drop index if exists public.teleo_v3_evidence_claim_polarity_idx;
|
||||||
|
drop index if exists public.teleo_v3_assessment_one_successor_idx;
|
||||||
|
drop index if exists public.teleo_v3_claim_one_admitted_successor_idx;
|
||||||
|
|
||||||
|
drop table if exists public.claim_evidence_assessments;
|
||||||
|
|
||||||
|
alter table public.claim_edges
|
||||||
|
drop constraint if exists teleo_v3_edge_contract_check,
|
||||||
|
drop constraint if exists teleo_v3_edge_accepted_proposal_fkey,
|
||||||
|
drop column if exists relation_type,
|
||||||
|
drop column if exists rationale,
|
||||||
|
drop column if exists scope_or_conditions,
|
||||||
|
drop column if exists accepted_by_proposal_id;
|
||||||
|
|
||||||
|
alter table public.claim_evidence
|
||||||
|
drop constraint if exists teleo_v3_evidence_contract_check,
|
||||||
|
drop constraint if exists teleo_v3_evidence_accepted_proposal_fkey,
|
||||||
|
drop column if exists excerpt,
|
||||||
|
drop column if exists locator_json,
|
||||||
|
drop column if exists source_content_hash,
|
||||||
|
drop column if exists rationale,
|
||||||
|
drop column if exists polarity,
|
||||||
|
drop column if exists accepted_from_proposal_id;
|
||||||
|
|
||||||
|
alter table public.sources
|
||||||
|
drop constraint if exists teleo_v3_source_contract_check,
|
||||||
|
drop constraint if exists teleo_v3_source_accepted_proposal_fkey,
|
||||||
|
drop column if exists canonical_url,
|
||||||
|
drop column if exists storage_uri,
|
||||||
|
drop column if exists content_hash,
|
||||||
|
drop column if exists access_scope,
|
||||||
|
drop column if exists ingestion_origin,
|
||||||
|
drop column if exists provenance_status,
|
||||||
|
drop column if exists source_class,
|
||||||
|
drop column if exists auto_promotion_policy,
|
||||||
|
drop column if exists accepted_by_proposal_id;
|
||||||
|
|
||||||
|
alter table public.claims
|
||||||
|
drop constraint if exists teleo_v3_claim_admission_check,
|
||||||
|
drop constraint if exists teleo_v3_claim_accepted_proposal_fkey,
|
||||||
|
drop constraint if exists teleo_v3_claim_supersedes_fkey,
|
||||||
|
drop constraint if exists teleo_v3_claim_owner_agent_fkey,
|
||||||
|
drop column if exists owner_agent_id,
|
||||||
|
drop column if exists proposition,
|
||||||
|
drop column if exists body_markdown,
|
||||||
|
drop column if exists scope,
|
||||||
|
drop column if exists access_scope,
|
||||||
|
drop column if exists admission_state,
|
||||||
|
drop column if exists revision_criteria,
|
||||||
|
drop column if exists revision_kind,
|
||||||
|
drop column if exists supersedes_id,
|
||||||
|
drop column if exists accepted_by_proposal_id;
|
||||||
|
|
||||||
|
drop function if exists kb_stage.teleo_v3_guard_linked_proposal_decision();
|
||||||
|
drop function if exists kb_stage.teleo_v3_acquire_protected_relation_locks();
|
||||||
|
drop function if exists kb_stage.teleo_v3_enforce_apply_contract_cutover();
|
||||||
|
drop function if exists kb_stage.teleo_v3_validate_evidence_assessment();
|
||||||
|
drop function if exists kb_stage.teleo_v3_validate_claim_edge();
|
||||||
|
drop function if exists kb_stage.teleo_v3_validate_claim_evidence();
|
||||||
|
drop function if exists kb_stage.teleo_v3_guard_immutable_fields();
|
||||||
|
drop function if exists kb_stage.teleo_v3_reject_delete();
|
||||||
|
drop function if exists kb_stage.teleo_v3_require_linked_insert();
|
||||||
|
drop function if exists kb_stage.teleo_v3_require_accepted_proposal();
|
||||||
|
|
||||||
|
drop table if exists kb_stage.teleo_v3_installation_ownership;
|
||||||
|
|
||||||
|
commit;
|
||||||
|
|
||||||
|
select jsonb_build_object(
|
||||||
|
'contract', 'livingip.teleo-v3-epistemic-contract',
|
||||||
|
'version', 1,
|
||||||
|
'migration', 'rolled_back'
|
||||||
|
)::text;
|
||||||
1363
ops/run_local_corpus_knowledge_rebuild.py
Executable file
1363
ops/run_local_corpus_knowledge_rebuild.py
Executable file
File diff suppressed because it is too large
Load diff
|
|
@ -2,7 +2,7 @@
|
||||||
"""Deterministically rebuild a disposable Postgres from genesis plus a strict ledger.
|
"""Deterministically rebuild a disposable Postgres from genesis plus a strict ledger.
|
||||||
|
|
||||||
The v1 replay boundary supports strict ``add_edge``, ``attach_evidence``,
|
The v1 replay boundary supports strict ``add_edge``, ``attach_evidence``,
|
||||||
``approve_claim``, and ``revise_strategy`` receipts. Insert-only actions seed
|
V2/V3 ``approve_claim``, and ``revise_strategy`` receipts. Insert-only actions seed
|
||||||
their receipt-pinned rows before the guarded transition. ``revise_strategy``
|
their receipt-pinned rows before the guarded transition. ``revise_strategy``
|
||||||
instead captures the exact prestate identities, executes the exact hash-matched
|
instead captures the exact prestate identities, executes the exact hash-matched
|
||||||
guarded SQL, validates the generated delta, and normalizes only that delta to the
|
guarded SQL, validates the generated delta, and normalizes only that delta to the
|
||||||
|
|
@ -41,20 +41,34 @@ if str(SCRIPTS_DIR) not in sys.path:
|
||||||
sys.path.insert(0, str(SCRIPTS_DIR))
|
sys.path.insert(0, str(SCRIPTS_DIR))
|
||||||
|
|
||||||
import apply_proposal as apply_engine # noqa: E402, I001
|
import apply_proposal as apply_engine # noqa: E402, I001
|
||||||
|
import export_kb_transition_replay_material as replay_exporter # noqa: E402
|
||||||
import kb_apply_replay_receipt as replay_receipt # noqa: E402
|
import kb_apply_replay_receipt as replay_receipt # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
LEDGER_ARTIFACT = "teleo_genesis_plus_ledger"
|
LEDGER_ARTIFACT = "teleo_genesis_plus_ledger"
|
||||||
MATERIAL_ARTIFACT = "teleo_strict_proposal_replay_material"
|
MATERIAL_ARTIFACT = "teleo_strict_proposal_replay_material"
|
||||||
|
ADMISSION_ARTIFACT = "teleo_ledger_admission_manifest"
|
||||||
LEDGER_CONTRACT_VERSION = 1
|
LEDGER_CONTRACT_VERSION = 1
|
||||||
MATERIAL_CONTRACT_VERSION = 1
|
MATERIAL_CONTRACT_VERSION = 1
|
||||||
|
ADMISSION_CONTRACT_VERSION = 1
|
||||||
|
CANONICAL_REVIEWER_HANDLE = "m3taversal"
|
||||||
|
REVIEW_METADATA_PROVENANCE = "operator_supplied_explicit_review_metadata"
|
||||||
|
ADMISSION_BASES = frozenset(
|
||||||
|
{
|
||||||
|
"forgejo_primary_source",
|
||||||
|
"reviewed_telegram_steering",
|
||||||
|
"reviewed_primary_source",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
ADMISSION_SCOPES = frozenset({"collective_core", "agent_identity", "knowledge_commons"})
|
||||||
DEFAULT_REBUILD_IMAGE = (
|
DEFAULT_REBUILD_IMAGE = (
|
||||||
"docker.io/library/postgres:16-alpine@sha256:57c72fd2a128e416c7fcc499958864df5301e940bca0a56f58fddf30ffc07777"
|
"docker.io/library/postgres:16-alpine@sha256:57c72fd2a128e416c7fcc499958864df5301e940bca0a56f58fddf30ffc07777"
|
||||||
)
|
)
|
||||||
SUPPORTED_PROPOSAL_TYPES = frozenset({"add_edge", "attach_evidence", "approve_claim", "revise_strategy"})
|
SUPPORTED_PROPOSAL_TYPES = frozenset({"add_edge", "attach_evidence", "approve_claim", "revise_strategy"})
|
||||||
CLAIM_CEILING = (
|
CLAIM_CEILING = (
|
||||||
"exact genesis plus ordered strict proposal replay; supported types are add_edge, "
|
"exact genesis plus ordered strict proposal replay; supported types are add_edge, "
|
||||||
"attach_evidence, approve_claim, and revise_strategy; mutating strategy replay "
|
"attach_evidence, V2/V3 approve_claim, and revise_strategy; V3 cutover is ordered and exact; "
|
||||||
|
"mutating strategy replay "
|
||||||
"captures prestate identities and normalizes the exact receipt-pinned poststate; "
|
"captures prestate identities and normalizes the exact receipt-pinned poststate; "
|
||||||
"full approved/applied proposal rows and immutable approval snapshots are required"
|
"full approved/applied proposal rows and immutable approval snapshots are required"
|
||||||
)
|
)
|
||||||
|
|
@ -67,14 +81,17 @@ NOT_PROVEN = (
|
||||||
APPLY_ENGINE_PATH = SCRIPTS_DIR / "apply_proposal.py"
|
APPLY_ENGINE_PATH = SCRIPTS_DIR / "apply_proposal.py"
|
||||||
REPLAY_RECEIPT_ENGINE_PATH = SCRIPTS_DIR / "kb_apply_replay_receipt.py"
|
REPLAY_RECEIPT_ENGINE_PATH = SCRIPTS_DIR / "kb_apply_replay_receipt.py"
|
||||||
GUARD_PREREQUISITES_PATH = SCRIPTS_DIR / "kb_apply_prereqs.sql"
|
GUARD_PREREQUISITES_PATH = SCRIPTS_DIR / "kb_apply_prereqs.sql"
|
||||||
|
V3_CONTRACT_PATH = REPO_ROOT / "ops" / "teleo_v3_epistemic_contract.sql"
|
||||||
PARITY_SQL_PATH = REPO_ROOT / "ops" / "postgres_parity_manifest.sql"
|
PARITY_SQL_PATH = REPO_ROOT / "ops" / "postgres_parity_manifest.sql"
|
||||||
BASE_REBUILD_PATH = REPO_ROOT / "ops" / "run_local_canonical_postgres_rebuild.py"
|
BASE_REBUILD_PATH = REPO_ROOT / "ops" / "run_local_canonical_postgres_rebuild.py"
|
||||||
GUARD_PREREQUISITES_DESTINATION = "/tmp/kb-apply-prereqs.sql"
|
GUARD_PREREQUISITES_DESTINATION = "/tmp/kb-apply-prereqs.sql"
|
||||||
|
V3_CONTRACT_DESTINATION = "/tmp/teleo-v3-epistemic-contract.sql"
|
||||||
FIXED_ENGINE_PATHS = frozenset(
|
FIXED_ENGINE_PATHS = frozenset(
|
||||||
{
|
{
|
||||||
APPLY_ENGINE_PATH.resolve(),
|
APPLY_ENGINE_PATH.resolve(),
|
||||||
REPLAY_RECEIPT_ENGINE_PATH.resolve(),
|
REPLAY_RECEIPT_ENGINE_PATH.resolve(),
|
||||||
GUARD_PREREQUISITES_PATH.resolve(),
|
GUARD_PREREQUISITES_PATH.resolve(),
|
||||||
|
V3_CONTRACT_PATH.resolve(),
|
||||||
PARITY_SQL_PATH.resolve(),
|
PARITY_SQL_PATH.resolve(),
|
||||||
BASE_REBUILD_PATH.resolve(),
|
BASE_REBUILD_PATH.resolve(),
|
||||||
Path(__file__).resolve(),
|
Path(__file__).resolve(),
|
||||||
|
|
@ -83,6 +100,7 @@ FIXED_ENGINE_PATHS = frozenset(
|
||||||
|
|
||||||
SHA256_RE = re.compile(r"[0-9a-f]{64}\Z")
|
SHA256_RE = re.compile(r"[0-9a-f]{64}\Z")
|
||||||
IMAGE_DIGEST_RE = re.compile(r"\S+@sha256:[0-9a-f]{64}\Z")
|
IMAGE_DIGEST_RE = re.compile(r"\S+@sha256:[0-9a-f]{64}\Z")
|
||||||
|
TYPED_TIMESTAMP_FIELDS = frozenset({"captured_at", "created_at", "updated_at", "reviewed_at", "applied_at"})
|
||||||
PROPOSAL_TRANSITION_FIELDS = frozenset(
|
PROPOSAL_TRANSITION_FIELDS = frozenset(
|
||||||
{"status", "applied_by_handle", "applied_by_agent_id", "applied_at", "updated_at"}
|
{"status", "applied_by_handle", "applied_by_agent_id", "applied_at", "updated_at"}
|
||||||
)
|
)
|
||||||
|
|
@ -125,6 +143,7 @@ CANONICAL_TABLE_ORDER = (
|
||||||
"sources",
|
"sources",
|
||||||
"reasoning_tools",
|
"reasoning_tools",
|
||||||
"claim_evidence",
|
"claim_evidence",
|
||||||
|
"claim_evidence_assessments",
|
||||||
"claim_edges",
|
"claim_edges",
|
||||||
)
|
)
|
||||||
STRATEGY_REQUIRED_FIELDS = frozenset(
|
STRATEGY_REQUIRED_FIELDS = frozenset(
|
||||||
|
|
@ -173,6 +192,7 @@ class LedgerEntry:
|
||||||
material_path: Path
|
material_path: Path
|
||||||
material_sha256: str
|
material_sha256: str
|
||||||
replay_material_sha256: str
|
replay_material_sha256: str
|
||||||
|
contract_version: int
|
||||||
approved_proposal: dict[str, Any]
|
approved_proposal: dict[str, Any]
|
||||||
approval_snapshot: dict[str, Any]
|
approval_snapshot: dict[str, Any]
|
||||||
applied_proposal: dict[str, Any]
|
applied_proposal: dict[str, Any]
|
||||||
|
|
@ -196,6 +216,10 @@ class ReconstructionBundle:
|
||||||
final_manifest_bytes: int
|
final_manifest_bytes: int
|
||||||
engine_hashes: dict[str, str]
|
engine_hashes: dict[str, str]
|
||||||
entries: tuple[LedgerEntry, ...]
|
entries: tuple[LedgerEntry, ...]
|
||||||
|
admission: dict[str, Any]
|
||||||
|
admission_manifest_path: Path | None
|
||||||
|
admission_manifest_sha256: str | None
|
||||||
|
admission_manifest_bytes: int | None
|
||||||
protected_paths: frozenset[Path]
|
protected_paths: frozenset[Path]
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -228,12 +252,18 @@ def _require_exact_keys(value: Any, expected: frozenset[str], field: str) -> dic
|
||||||
|
|
||||||
def _load_json_object(path: Path, field: str) -> dict[str, Any]:
|
def _load_json_object(path: Path, field: str) -> dict[str, Any]:
|
||||||
try:
|
try:
|
||||||
value = json.loads(path.read_text(encoding="utf-8"))
|
raw = path.read_text(encoding="utf-8")
|
||||||
except (OSError, UnicodeError, json.JSONDecodeError) as exc:
|
return replay_exporter._parse_json_object(raw, label=field)
|
||||||
raise ReconstructionError("invalid_json_artifact", f"{field} is not valid UTF-8 JSON") from exc
|
except replay_exporter.ExportError as exc:
|
||||||
if not isinstance(value, dict):
|
raise ReconstructionError(
|
||||||
raise ReconstructionError("invalid_json_artifact", f"{field} must contain one JSON object")
|
"invalid_json_artifact",
|
||||||
return value
|
f"{field} is not exact duplicate-free UTF-8 JSON",
|
||||||
|
) from exc
|
||||||
|
except (OSError, UnicodeError) as exc:
|
||||||
|
raise ReconstructionError(
|
||||||
|
"invalid_json_artifact",
|
||||||
|
f"{field} is not exact duplicate-free UTF-8 JSON",
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
|
||||||
def _resolve_artifact(ledger_path: Path, declared: Any, field: str) -> Path:
|
def _resolve_artifact(ledger_path: Path, declared: Any, field: str) -> Path:
|
||||||
|
|
@ -263,11 +293,183 @@ def _engine_hashes() -> dict[str, str]:
|
||||||
"apply_engine_sha256": APPLY_ENGINE_PATH,
|
"apply_engine_sha256": APPLY_ENGINE_PATH,
|
||||||
"replay_receipt_engine_sha256": REPLAY_RECEIPT_ENGINE_PATH,
|
"replay_receipt_engine_sha256": REPLAY_RECEIPT_ENGINE_PATH,
|
||||||
"guard_prerequisites_sha256": GUARD_PREREQUISITES_PATH,
|
"guard_prerequisites_sha256": GUARD_PREREQUISITES_PATH,
|
||||||
|
"v3_contract_sha256": V3_CONTRACT_PATH,
|
||||||
"parity_sql_sha256": PARITY_SQL_PATH,
|
"parity_sql_sha256": PARITY_SQL_PATH,
|
||||||
}
|
}
|
||||||
return {name: canonical_rebuild.sha256_file(path) for name, path in paths.items()}
|
return {name: canonical_rebuild.sha256_file(path) for name, path in paths.items()}
|
||||||
|
|
||||||
|
|
||||||
|
def _proposal_contract_version(proposal: dict[str, Any]) -> int:
|
||||||
|
payload = proposal.get("payload")
|
||||||
|
apply_payload = payload.get("apply_payload") if isinstance(payload, dict) else None
|
||||||
|
if not isinstance(apply_payload, dict):
|
||||||
|
raise ReconstructionError("invalid_material", "proposal is missing a strict apply payload")
|
||||||
|
raw_version = apply_payload.get("contract_version", 2)
|
||||||
|
if type(raw_version) is not int or raw_version not in {2, 3}:
|
||||||
|
raise ReconstructionError("unsupported_contract_version", "proposal contract version is unsupported")
|
||||||
|
if raw_version == 3 and proposal.get("proposal_type") != "approve_claim":
|
||||||
|
raise ReconstructionError("unsupported_contract_version", "V3 replay currently supports approve_claim only")
|
||||||
|
return raw_version
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_contract_order(entries: list[LedgerEntry]) -> None:
|
||||||
|
v3_started = False
|
||||||
|
for entry in entries:
|
||||||
|
if entry.contract_version == 3:
|
||||||
|
v3_started = True
|
||||||
|
elif v3_started:
|
||||||
|
raise ReconstructionError(
|
||||||
|
"contract_order_mismatch",
|
||||||
|
"V2 ledger entries cannot appear after the ordered V3 cutover",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _load_admission_manifest(
|
||||||
|
args: argparse.Namespace,
|
||||||
|
*,
|
||||||
|
ledger_sha256: str,
|
||||||
|
entries: list[LedgerEntry],
|
||||||
|
) -> tuple[dict[str, Any], Path | None, str | None, int | None]:
|
||||||
|
manifest_path = getattr(args, "admission_manifest", None)
|
||||||
|
manifest_pin = getattr(args, "admission_manifest_sha256", None)
|
||||||
|
if manifest_path is None and manifest_pin is None:
|
||||||
|
return (
|
||||||
|
{
|
||||||
|
"status": "forensic_only",
|
||||||
|
"canonical_retrieval_eligible": False,
|
||||||
|
"reason": "no explicit admission manifest was supplied",
|
||||||
|
"entry_count": len(entries),
|
||||||
|
},
|
||||||
|
None,
|
||||||
|
None,
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if manifest_path is None or manifest_pin is None:
|
||||||
|
raise ReconstructionError(
|
||||||
|
"incomplete_admission_pin",
|
||||||
|
"admission requires both a manifest and its SHA-256 pin",
|
||||||
|
)
|
||||||
|
if not isinstance(manifest_path, Path) or not manifest_path.is_file():
|
||||||
|
raise ReconstructionError("missing_admission_manifest", "admission manifest is not an existing regular file")
|
||||||
|
|
||||||
|
manifest_path = manifest_path.resolve()
|
||||||
|
manifest_sha256 = _assert_file_hash(manifest_path, manifest_pin, "--admission-manifest-sha256")
|
||||||
|
manifest = _require_exact_keys(
|
||||||
|
_load_json_object(manifest_path, "admission manifest"),
|
||||||
|
frozenset(
|
||||||
|
{
|
||||||
|
"artifact",
|
||||||
|
"contract_version",
|
||||||
|
"ledger_sha256",
|
||||||
|
"reviewed_by_handle",
|
||||||
|
"reviewed_by_agent_id",
|
||||||
|
"reviewed_at",
|
||||||
|
"review_note",
|
||||||
|
"entries",
|
||||||
|
}
|
||||||
|
),
|
||||||
|
"admission manifest",
|
||||||
|
)
|
||||||
|
if manifest["artifact"] != ADMISSION_ARTIFACT or manifest["contract_version"] != ADMISSION_CONTRACT_VERSION:
|
||||||
|
raise ReconstructionError(
|
||||||
|
"unsupported_admission_contract",
|
||||||
|
"admission manifest is not the supported v1 contract",
|
||||||
|
)
|
||||||
|
if _validate_sha256(manifest["ledger_sha256"], "admission manifest ledger_sha256") != ledger_sha256:
|
||||||
|
raise ReconstructionError("admission_ledger_mismatch", "admission manifest does not select this exact ledger")
|
||||||
|
|
||||||
|
reviewer_handle = manifest["reviewed_by_handle"]
|
||||||
|
review_note = manifest["review_note"]
|
||||||
|
if reviewer_handle != CANONICAL_REVIEWER_HANDLE:
|
||||||
|
raise ReconstructionError(
|
||||||
|
"invalid_admission_review",
|
||||||
|
"admission reviewer handle must equal m3taversal exactly",
|
||||||
|
)
|
||||||
|
if not isinstance(review_note, str) or not review_note.strip():
|
||||||
|
raise ReconstructionError("invalid_admission_review", "admission review note must be non-empty")
|
||||||
|
try:
|
||||||
|
reviewer_agent_id = str(uuid.UUID(str(manifest["reviewed_by_agent_id"])))
|
||||||
|
reviewed_at = replay_receipt.canonical_utc_timestamp(
|
||||||
|
manifest["reviewed_at"],
|
||||||
|
path="admission manifest reviewed_at",
|
||||||
|
)
|
||||||
|
except (AttributeError, TypeError, ValueError) as exc:
|
||||||
|
raise ReconstructionError(
|
||||||
|
"invalid_admission_review",
|
||||||
|
"admission review requires one reviewer UUID and timezone-aware timestamp",
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
raw_entries = manifest["entries"]
|
||||||
|
if not isinstance(raw_entries, list) or len(raw_entries) != len(entries):
|
||||||
|
raise ReconstructionError(
|
||||||
|
"admission_selection_mismatch",
|
||||||
|
"admission manifest must select every entry in the supplied clean-genesis ledger exactly once",
|
||||||
|
)
|
||||||
|
selected: list[dict[str, Any]] = []
|
||||||
|
for index, (raw_entry, entry) in enumerate(zip(raw_entries, entries, strict=True), 1):
|
||||||
|
selected_entry = _require_exact_keys(
|
||||||
|
raw_entry,
|
||||||
|
frozenset(
|
||||||
|
{
|
||||||
|
"sequence",
|
||||||
|
"proposal_id",
|
||||||
|
"material_sha256",
|
||||||
|
"replay_material_sha256",
|
||||||
|
"admission_scope",
|
||||||
|
"basis",
|
||||||
|
}
|
||||||
|
),
|
||||||
|
f"admission manifest entries[{index - 1}]",
|
||||||
|
)
|
||||||
|
if entry.contract_version != 3:
|
||||||
|
raise ReconstructionError(
|
||||||
|
"legacy_admission_forbidden",
|
||||||
|
"only source-backed V3 entries may enter an admitted clean genesis",
|
||||||
|
)
|
||||||
|
expected = {
|
||||||
|
"sequence": entry.sequence,
|
||||||
|
"proposal_id": entry.applied_proposal["id"],
|
||||||
|
"material_sha256": entry.material_sha256,
|
||||||
|
"replay_material_sha256": entry.replay_material_sha256,
|
||||||
|
}
|
||||||
|
if any(selected_entry[field] != value for field, value in expected.items()):
|
||||||
|
raise ReconstructionError(
|
||||||
|
"admission_selection_mismatch",
|
||||||
|
f"admission manifest entry {index} does not pin the exact ledger material",
|
||||||
|
)
|
||||||
|
scope = selected_entry["admission_scope"]
|
||||||
|
basis = selected_entry["basis"]
|
||||||
|
if scope not in ADMISSION_SCOPES or basis not in ADMISSION_BASES:
|
||||||
|
raise ReconstructionError(
|
||||||
|
"invalid_admission_classification",
|
||||||
|
f"admission manifest entry {index} has an unsupported scope or basis",
|
||||||
|
)
|
||||||
|
selected.append({**expected, "admission_scope": scope, "basis": basis})
|
||||||
|
|
||||||
|
return (
|
||||||
|
{
|
||||||
|
"status": "review_metadata_supplied",
|
||||||
|
"canonical_retrieval_eligible": False,
|
||||||
|
"reason": "operator-supplied review metadata is hash-pinned but not authenticated",
|
||||||
|
"artifact": ADMISSION_ARTIFACT,
|
||||||
|
"contract_version": ADMISSION_CONTRACT_VERSION,
|
||||||
|
"review_metadata_provenance": REVIEW_METADATA_PROVENANCE,
|
||||||
|
"cryptographically_authenticated": False,
|
||||||
|
"manifest_sha256": manifest_sha256,
|
||||||
|
"ledger_sha256": ledger_sha256,
|
||||||
|
"reviewed_by_handle": reviewer_handle.strip(),
|
||||||
|
"reviewed_by_agent_id": reviewer_agent_id,
|
||||||
|
"reviewed_at": reviewed_at,
|
||||||
|
"review_note_sha256": _sha256_text(review_note),
|
||||||
|
"entry_count": len(selected),
|
||||||
|
"entries": selected,
|
||||||
|
},
|
||||||
|
manifest_path,
|
||||||
|
manifest_sha256,
|
||||||
|
manifest_path.stat().st_size,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _validate_proposal_rows(
|
def _validate_proposal_rows(
|
||||||
approved: dict[str, Any],
|
approved: dict[str, Any],
|
||||||
applied: dict[str, Any],
|
applied: dict[str, Any],
|
||||||
|
|
@ -337,7 +539,25 @@ def _validate_proposal_rows(
|
||||||
"applied_by_agent_id",
|
"applied_by_agent_id",
|
||||||
"applied_at",
|
"applied_at",
|
||||||
)
|
)
|
||||||
if any(receipt_proposal.get(field) != applied[field] for field in envelope_fields):
|
try:
|
||||||
|
envelope_mismatch = any(
|
||||||
|
replay_receipt.normalize_typed_value(
|
||||||
|
field,
|
||||||
|
receipt_proposal.get(field),
|
||||||
|
path=f"replay_receipt.proposal.{field}",
|
||||||
|
)
|
||||||
|
!= replay_receipt.normalize_typed_value(
|
||||||
|
field,
|
||||||
|
applied[field],
|
||||||
|
path=f"material.applied_proposal.{field}",
|
||||||
|
)
|
||||||
|
for field in envelope_fields
|
||||||
|
)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise ReconstructionError(
|
||||||
|
"invalid_material", "replay receipt proposal envelope contains an invalid typed value"
|
||||||
|
) from exc
|
||||||
|
if envelope_mismatch:
|
||||||
raise ReconstructionError(
|
raise ReconstructionError(
|
||||||
"material_mismatch", "replay receipt proposal envelope does not match applied proposal row"
|
"material_mismatch", "replay receipt proposal envelope does not match applied proposal row"
|
||||||
)
|
)
|
||||||
|
|
@ -552,6 +772,7 @@ def _validate_entry_material(
|
||||||
material_path=material_path,
|
material_path=material_path,
|
||||||
material_sha256=material_sha256,
|
material_sha256=material_sha256,
|
||||||
replay_material_sha256=actual_replay_hash,
|
replay_material_sha256=actual_replay_hash,
|
||||||
|
contract_version=_proposal_contract_version(proposal),
|
||||||
approved_proposal=approved,
|
approved_proposal=approved,
|
||||||
approval_snapshot=approval,
|
approval_snapshot=approval,
|
||||||
applied_proposal=applied,
|
applied_proposal=applied,
|
||||||
|
|
@ -644,6 +865,15 @@ def load_bundle(args: argparse.Namespace) -> ReconstructionBundle:
|
||||||
entries.append(entry)
|
entries.append(entry)
|
||||||
protected_paths.add(material_path)
|
protected_paths.add(material_path)
|
||||||
|
|
||||||
|
_validate_contract_order(entries)
|
||||||
|
admission, admission_path, admission_sha256, admission_bytes = _load_admission_manifest(
|
||||||
|
args,
|
||||||
|
ledger_sha256=ledger_sha256,
|
||||||
|
entries=entries,
|
||||||
|
)
|
||||||
|
if admission_path is not None:
|
||||||
|
protected_paths.add(admission_path)
|
||||||
|
|
||||||
if args.output.resolve() in protected_paths:
|
if args.output.resolve() in protected_paths:
|
||||||
raise ReconstructionError("unsafe_output_path", "--output must not overwrite any reconstruction input")
|
raise ReconstructionError("unsafe_output_path", "--output must not overwrite any reconstruction input")
|
||||||
|
|
||||||
|
|
@ -661,6 +891,10 @@ def load_bundle(args: argparse.Namespace) -> ReconstructionBundle:
|
||||||
final_manifest_bytes=final_manifest_path.stat().st_size,
|
final_manifest_bytes=final_manifest_path.stat().st_size,
|
||||||
engine_hashes=actual_engine_hashes,
|
engine_hashes=actual_engine_hashes,
|
||||||
entries=tuple(entries),
|
entries=tuple(entries),
|
||||||
|
admission=admission,
|
||||||
|
admission_manifest_path=admission_path,
|
||||||
|
admission_manifest_sha256=admission_sha256,
|
||||||
|
admission_manifest_bytes=admission_bytes,
|
||||||
protected_paths=frozenset(protected_paths),
|
protected_paths=frozenset(protected_paths),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
@ -669,7 +903,15 @@ def _jsonb_literal(value: Any) -> str:
|
||||||
return apply_engine.sql_literal(_canonical_json(value)) + "::jsonb"
|
return apply_engine.sql_literal(_canonical_json(value)) + "::jsonb"
|
||||||
|
|
||||||
|
|
||||||
|
def _approval_seed_row(approved_proposal: dict[str, Any], approval_snapshot: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
**approval_snapshot,
|
||||||
|
"approved_proposal_snapshot": approved_proposal,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def build_seed_sql(entry: LedgerEntry) -> str:
|
def build_seed_sql(entry: LedgerEntry) -> str:
|
||||||
|
approval_seed = _approval_seed_row(entry.approved_proposal, entry.approval_snapshot)
|
||||||
statements = [
|
statements = [
|
||||||
"begin;",
|
"begin;",
|
||||||
"set local standard_conforming_strings = on;",
|
"set local standard_conforming_strings = on;",
|
||||||
|
|
@ -679,25 +921,30 @@ def build_seed_sql(entry: LedgerEntry) -> str:
|
||||||
") seeded",
|
") seeded",
|
||||||
"on conflict (id) do nothing;",
|
"on conflict (id) do nothing;",
|
||||||
"insert into kb_stage.kb_proposal_approvals",
|
"insert into kb_stage.kb_proposal_approvals",
|
||||||
"select seeded.* from jsonb_populate_record(",
|
"select seeded.*",
|
||||||
f" null::kb_stage.kb_proposal_approvals, {_jsonb_literal(entry.approval_snapshot)}",
|
"from kb_stage.kb_proposals proposal",
|
||||||
|
"cross join lateral jsonb_populate_record(",
|
||||||
|
" null::kb_stage.kb_proposal_approvals,",
|
||||||
|
f" jsonb_set({_jsonb_literal(approval_seed)}, '{{approved_proposal_snapshot}}', to_jsonb(proposal), true)",
|
||||||
") seeded",
|
") seeded",
|
||||||
|
f"where proposal.id = {apply_engine.sql_literal(entry.approved_proposal['id'])}::uuid",
|
||||||
"on conflict (proposal_id) do nothing;",
|
"on conflict (proposal_id) do nothing;",
|
||||||
]
|
]
|
||||||
rows = entry.receipt["canonical_rows"]
|
if entry.contract_version != 3:
|
||||||
for table in CANONICAL_TABLE_ORDER:
|
rows = entry.receipt["canonical_rows"]
|
||||||
table_rows = rows.get(table)
|
for table in CANONICAL_TABLE_ORDER:
|
||||||
if not table_rows:
|
table_rows = rows.get(table)
|
||||||
continue
|
if not table_rows:
|
||||||
statements.extend(
|
continue
|
||||||
[
|
statements.extend(
|
||||||
f"insert into public.{table}",
|
[
|
||||||
"select seeded.* from jsonb_populate_recordset(",
|
f"insert into public.{table}",
|
||||||
f" null::public.{table}, {_jsonb_literal(table_rows)}",
|
"select seeded.* from jsonb_populate_recordset(",
|
||||||
") seeded",
|
f" null::public.{table}, {_jsonb_literal(table_rows)}",
|
||||||
"on conflict do nothing;",
|
") seeded",
|
||||||
]
|
"on conflict do nothing;",
|
||||||
)
|
]
|
||||||
|
)
|
||||||
statements.append("commit;")
|
statements.append("commit;")
|
||||||
return "\n".join(statements) + "\n"
|
return "\n".join(statements) + "\n"
|
||||||
|
|
||||||
|
|
@ -967,14 +1214,163 @@ def build_proposal_readback_sql(proposal_id: str) -> str:
|
||||||
return f"""select jsonb_build_object(
|
return f"""select jsonb_build_object(
|
||||||
'proposal', (select to_jsonb(p) from kb_stage.kb_proposals p
|
'proposal', (select to_jsonb(p) from kb_stage.kb_proposals p
|
||||||
where p.id = {literal}::uuid),
|
where p.id = {literal}::uuid),
|
||||||
'approval_snapshot', (select to_jsonb(a) from kb_stage.kb_proposal_approvals a
|
'approval_snapshot', (select to_jsonb(a) - 'approved_proposal_snapshot'
|
||||||
where a.proposal_id = {literal}::uuid)
|
from kb_stage.kb_proposal_approvals a
|
||||||
|
where a.proposal_id = {literal}::uuid),
|
||||||
|
'approved_proposal_snapshot', (select a.approved_proposal_snapshot
|
||||||
|
from kb_stage.kb_proposal_approvals a
|
||||||
|
where a.proposal_id = {literal}::uuid)
|
||||||
)::text;
|
)::text;
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def build_v3_canonical_timestamp_normalization_sql(entry: LedgerEntry) -> str:
|
||||||
|
if entry.contract_version != 3:
|
||||||
|
raise ReconstructionError(
|
||||||
|
"invalid_v3_normalization",
|
||||||
|
"canonical V3 timestamp normalization requires a V3 ledger entry",
|
||||||
|
)
|
||||||
|
proposal_id = entry.applied_proposal["id"]
|
||||||
|
table_contracts = {
|
||||||
|
"claims": ("accepted_by_proposal_id", "teleo_v3_claim_immutable"),
|
||||||
|
"sources": ("accepted_by_proposal_id", "teleo_v3_source_immutable"),
|
||||||
|
"claim_evidence": ("accepted_from_proposal_id", "teleo_v3_evidence_immutable"),
|
||||||
|
"claim_evidence_assessments": (
|
||||||
|
"accepted_by_proposal_id",
|
||||||
|
"teleo_v3_assessment_immutable",
|
||||||
|
),
|
||||||
|
"claim_edges": ("accepted_by_proposal_id", "teleo_v3_edge_immutable"),
|
||||||
|
}
|
||||||
|
statements = ["begin;", "set local standard_conforming_strings = on;"]
|
||||||
|
for table in CANONICAL_TABLE_ORDER:
|
||||||
|
rows = entry.receipt["canonical_rows"].get(table) or []
|
||||||
|
if not rows:
|
||||||
|
continue
|
||||||
|
if table not in table_contracts or any("id" not in row for row in rows):
|
||||||
|
raise ReconstructionError(
|
||||||
|
"invalid_v3_normalization",
|
||||||
|
"V3 canonical timestamp rows lack an exact proposal-owned identity",
|
||||||
|
)
|
||||||
|
timestamp_fields = sorted(TYPED_TIMESTAMP_FIELDS.intersection(*(set(row) for row in rows)))
|
||||||
|
if not timestamp_fields:
|
||||||
|
continue
|
||||||
|
proposal_column, immutable_trigger = table_contracts[table]
|
||||||
|
assignments = ",\n ".join(f"{field} = pinned.{field}" for field in timestamp_fields)
|
||||||
|
statements.append(
|
||||||
|
f"""lock table public.{table} in access exclusive mode;
|
||||||
|
do $preflight_{table}$
|
||||||
|
begin
|
||||||
|
if not exists (
|
||||||
|
select 1
|
||||||
|
from pg_catalog.pg_trigger trigger
|
||||||
|
where trigger.tgrelid = 'public.{table}'::regclass
|
||||||
|
and trigger.tgname = '{immutable_trigger}'
|
||||||
|
and trigger.tgfoid = 'kb_stage.teleo_v3_guard_immutable_fields()'::regprocedure
|
||||||
|
and not trigger.tgisinternal
|
||||||
|
and trigger.tgenabled = 'O'
|
||||||
|
) then
|
||||||
|
raise exception 'V3 {table} immutable guard is not enabled before deterministic normalization';
|
||||||
|
end if;
|
||||||
|
end
|
||||||
|
$preflight_{table}$;
|
||||||
|
alter table public.{table} disable trigger {immutable_trigger};
|
||||||
|
do $normalize_{table}$
|
||||||
|
declare
|
||||||
|
changed_rows integer;
|
||||||
|
begin
|
||||||
|
update public.{table} target
|
||||||
|
set {assignments}
|
||||||
|
from jsonb_populate_recordset(
|
||||||
|
null::public.{table}, {_jsonb_literal(rows)}
|
||||||
|
) pinned
|
||||||
|
where target.id = pinned.id
|
||||||
|
and target.{proposal_column} = {apply_engine.sql_literal(proposal_id)}::uuid;
|
||||||
|
get diagnostics changed_rows = row_count;
|
||||||
|
if changed_rows <> {len(rows)} then
|
||||||
|
raise exception 'deterministic V3 {table} timestamp normalization failed';
|
||||||
|
end if;
|
||||||
|
end
|
||||||
|
$normalize_{table}$;
|
||||||
|
alter table public.{table} enable trigger {immutable_trigger};
|
||||||
|
do $postflight_{table}$
|
||||||
|
begin
|
||||||
|
if not exists (
|
||||||
|
select 1
|
||||||
|
from pg_catalog.pg_trigger trigger
|
||||||
|
where trigger.tgrelid = 'public.{table}'::regclass
|
||||||
|
and trigger.tgname = '{immutable_trigger}'
|
||||||
|
and trigger.tgfoid = 'kb_stage.teleo_v3_guard_immutable_fields()'::regprocedure
|
||||||
|
and not trigger.tgisinternal
|
||||||
|
and trigger.tgenabled = 'O'
|
||||||
|
) then
|
||||||
|
raise exception 'V3 {table} immutable guard was not restored after deterministic normalization';
|
||||||
|
end if;
|
||||||
|
end
|
||||||
|
$postflight_{table}$;"""
|
||||||
|
)
|
||||||
|
statements.append("commit;")
|
||||||
|
return "\n".join(statements) + "\n"
|
||||||
|
|
||||||
|
|
||||||
def build_applied_timestamp_normalization_sql(entry: LedgerEntry) -> str:
|
def build_applied_timestamp_normalization_sql(entry: LedgerEntry) -> str:
|
||||||
applied = entry.applied_proposal
|
applied = entry.applied_proposal
|
||||||
|
if entry.contract_version == 3:
|
||||||
|
return f"""begin;
|
||||||
|
lock table kb_stage.kb_proposals in access exclusive mode;
|
||||||
|
do $preflight$
|
||||||
|
begin
|
||||||
|
if not exists (
|
||||||
|
select 1
|
||||||
|
from pg_catalog.pg_trigger trigger
|
||||||
|
where trigger.tgrelid = 'kb_stage.kb_proposals'::regclass
|
||||||
|
and trigger.tgname = 'teleo_v3_guard_linked_proposal_decision'
|
||||||
|
and trigger.tgfoid = 'kb_stage.teleo_v3_guard_linked_proposal_decision()'::regprocedure
|
||||||
|
and not trigger.tgisinternal
|
||||||
|
and trigger.tgenabled = 'O'
|
||||||
|
) then
|
||||||
|
raise exception 'V3 proposal decision guard is not enabled before deterministic normalization';
|
||||||
|
end if;
|
||||||
|
end
|
||||||
|
$preflight$;
|
||||||
|
alter table kb_stage.kb_proposals
|
||||||
|
disable trigger teleo_v3_guard_linked_proposal_decision;
|
||||||
|
do $normalization$
|
||||||
|
declare
|
||||||
|
changed_rows integer;
|
||||||
|
begin
|
||||||
|
update kb_stage.kb_proposals
|
||||||
|
set applied_at = {apply_engine.sql_literal(applied["applied_at"])}::timestamptz,
|
||||||
|
updated_at = {apply_engine.sql_literal(applied["updated_at"])}::timestamptz
|
||||||
|
where id = {apply_engine.sql_literal(applied["id"])}::uuid
|
||||||
|
and status = 'applied'
|
||||||
|
and applied_by_handle = {apply_engine.sql_literal(applied["applied_by_handle"])}
|
||||||
|
and applied_by_agent_id is not distinct from
|
||||||
|
{apply_engine.sql_literal(applied["applied_by_agent_id"])}::uuid;
|
||||||
|
get diagnostics changed_rows = row_count;
|
||||||
|
if changed_rows <> 1 then
|
||||||
|
raise exception 'deterministic V3 proposal timestamp normalization failed';
|
||||||
|
end if;
|
||||||
|
end
|
||||||
|
$normalization$;
|
||||||
|
alter table kb_stage.kb_proposals
|
||||||
|
enable trigger teleo_v3_guard_linked_proposal_decision;
|
||||||
|
do $postflight$
|
||||||
|
begin
|
||||||
|
if not exists (
|
||||||
|
select 1
|
||||||
|
from pg_catalog.pg_trigger trigger
|
||||||
|
where trigger.tgrelid = 'kb_stage.kb_proposals'::regclass
|
||||||
|
and trigger.tgname = 'teleo_v3_guard_linked_proposal_decision'
|
||||||
|
and trigger.tgfoid = 'kb_stage.teleo_v3_guard_linked_proposal_decision()'::regprocedure
|
||||||
|
and not trigger.tgisinternal
|
||||||
|
and trigger.tgenabled = 'O'
|
||||||
|
) then
|
||||||
|
raise exception 'V3 proposal decision guard was not restored after deterministic normalization';
|
||||||
|
end if;
|
||||||
|
end
|
||||||
|
$postflight$;
|
||||||
|
commit;
|
||||||
|
"""
|
||||||
return f"""do $reconstruction$
|
return f"""do $reconstruction$
|
||||||
declare
|
declare
|
||||||
changed_rows integer;
|
changed_rows integer;
|
||||||
|
|
@ -1089,17 +1485,191 @@ def _read_json(
|
||||||
raise ReconstructionError(code, f"{action} returned an invalid private JSON readback") from exc
|
raise ReconstructionError(code, f"{action} returned an invalid private JSON readback") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def install_v3_contract(args: argparse.Namespace, container: str) -> dict[str, Any]:
|
||||||
|
completed = canonical_rebuild._run(
|
||||||
|
[
|
||||||
|
args.docker_bin,
|
||||||
|
"cp",
|
||||||
|
str(V3_CONTRACT_PATH),
|
||||||
|
f"{container}:{V3_CONTRACT_DESTINATION}",
|
||||||
|
],
|
||||||
|
timeout=args.command_timeout,
|
||||||
|
)
|
||||||
|
_require_command_success(
|
||||||
|
completed,
|
||||||
|
code="v3_contract_copy_failed",
|
||||||
|
action="V3 epistemic contract copy",
|
||||||
|
)
|
||||||
|
|
||||||
|
cutover = _read_json(
|
||||||
|
args,
|
||||||
|
container,
|
||||||
|
"""begin;
|
||||||
|
alter role kb_apply nologin;
|
||||||
|
do $cutover$
|
||||||
|
begin
|
||||||
|
if exists (
|
||||||
|
select 1
|
||||||
|
from pg_catalog.pg_stat_activity activity
|
||||||
|
where activity.datid = (select oid from pg_catalog.pg_database where datname = current_database())
|
||||||
|
and activity.usename = 'kb_apply'
|
||||||
|
and activity.pid <> pg_catalog.pg_backend_pid()
|
||||||
|
) then
|
||||||
|
raise exception 'V3 reconstruction cutover found an active kb_apply session';
|
||||||
|
end if;
|
||||||
|
end
|
||||||
|
$cutover$;
|
||||||
|
commit;
|
||||||
|
select jsonb_build_object(
|
||||||
|
'apply_role_login_disabled', not (select role.rolcanlogin from pg_catalog.pg_roles role where role.rolname = 'kb_apply'),
|
||||||
|
'active_apply_sessions', (
|
||||||
|
select count(*)
|
||||||
|
from pg_catalog.pg_stat_activity activity
|
||||||
|
where activity.datid = (select oid from pg_catalog.pg_database where datname = current_database())
|
||||||
|
and activity.usename = 'kb_apply'
|
||||||
|
and activity.pid <> pg_catalog.pg_backend_pid()
|
||||||
|
)
|
||||||
|
)::text;
|
||||||
|
""",
|
||||||
|
code="v3_cutover_preflight_failed",
|
||||||
|
action="V3 disposable writer cutover preflight",
|
||||||
|
)
|
||||||
|
if cutover != {"apply_role_login_disabled": True, "active_apply_sessions": 0}:
|
||||||
|
raise ReconstructionError(
|
||||||
|
"v3_cutover_preflight_failed",
|
||||||
|
"V3 disposable writer cutover did not disable login with zero active apply sessions",
|
||||||
|
)
|
||||||
|
|
||||||
|
for destination, label in (
|
||||||
|
(V3_CONTRACT_DESTINATION, "V3 epistemic contract"),
|
||||||
|
(GUARD_PREREQUISITES_DESTINATION, "guarded apply prerequisites after V3"),
|
||||||
|
):
|
||||||
|
completed = canonical_rebuild._run(
|
||||||
|
[
|
||||||
|
args.docker_bin,
|
||||||
|
"exec",
|
||||||
|
container,
|
||||||
|
"psql",
|
||||||
|
"-X",
|
||||||
|
"-U",
|
||||||
|
"postgres",
|
||||||
|
"-d",
|
||||||
|
args.database,
|
||||||
|
"-Atq",
|
||||||
|
"-v",
|
||||||
|
"ON_ERROR_STOP=1",
|
||||||
|
"-f",
|
||||||
|
destination,
|
||||||
|
],
|
||||||
|
timeout=args.apply_timeout,
|
||||||
|
)
|
||||||
|
_require_command_success(
|
||||||
|
completed,
|
||||||
|
code="v3_contract_install_failed",
|
||||||
|
action=label,
|
||||||
|
)
|
||||||
|
|
||||||
|
catalog = _read_json(
|
||||||
|
args,
|
||||||
|
container,
|
||||||
|
"""select jsonb_build_object(
|
||||||
|
'assessment_table_present', to_regclass('public.claim_evidence_assessments') is not null,
|
||||||
|
'apply_role_login_restored', (select role.rolcanlogin from pg_catalog.pg_roles role where role.rolname = 'kb_apply'),
|
||||||
|
'cutover_guard_enabled', exists (
|
||||||
|
select 1 from pg_catalog.pg_trigger trigger
|
||||||
|
where trigger.tgrelid = 'kb_stage.kb_proposals'::regclass
|
||||||
|
and trigger.tgname = 'teleo_v3_00_enforce_apply_contract_cutover'
|
||||||
|
and trigger.tgfoid = 'kb_stage.teleo_v3_enforce_apply_contract_cutover()'::regprocedure
|
||||||
|
and not trigger.tgisinternal
|
||||||
|
and trigger.tgenabled = 'O'
|
||||||
|
),
|
||||||
|
'decision_guard_enabled', exists (
|
||||||
|
select 1 from pg_catalog.pg_trigger trigger
|
||||||
|
where trigger.tgrelid = 'kb_stage.kb_proposals'::regclass
|
||||||
|
and trigger.tgname = 'teleo_v3_guard_linked_proposal_decision'
|
||||||
|
and trigger.tgfoid = 'kb_stage.teleo_v3_guard_linked_proposal_decision()'::regprocedure
|
||||||
|
and not trigger.tgisinternal
|
||||||
|
and trigger.tgenabled = 'O'
|
||||||
|
)
|
||||||
|
)::text;
|
||||||
|
""",
|
||||||
|
code="v3_contract_readback_failed",
|
||||||
|
action="V3 epistemic contract catalog readback",
|
||||||
|
)
|
||||||
|
if catalog != {
|
||||||
|
"assessment_table_present": True,
|
||||||
|
"apply_role_login_restored": True,
|
||||||
|
"cutover_guard_enabled": True,
|
||||||
|
"decision_guard_enabled": True,
|
||||||
|
}:
|
||||||
|
raise ReconstructionError("v3_contract_readback_failed", "V3 contract catalog readback was incomplete")
|
||||||
|
return {
|
||||||
|
"status": "pass",
|
||||||
|
"sha256": canonical_rebuild.sha256_file(V3_CONTRACT_PATH),
|
||||||
|
"cutover": cutover,
|
||||||
|
"guard_prerequisites_rerun": True,
|
||||||
|
"catalog": catalog,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def _assert_exact_json(actual: Any, expected: Any, *, code: str, action: str) -> None:
|
def _assert_exact_json(actual: Any, expected: Any, *, code: str, action: str) -> None:
|
||||||
if _canonical_json(actual) != _canonical_json(expected):
|
if _canonical_json(actual) != _canonical_json(expected):
|
||||||
raise ReconstructionError(code, f"{action} did not match the hash-pinned material")
|
raise ReconstructionError(code, f"{action} did not match the hash-pinned material")
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_exact_canonical_rows(actual: Any, expected: Any, *, code: str, action: str) -> None:
|
||||||
|
def normalize(value: Any, path: str) -> dict[str, list[dict[str, Any]]]:
|
||||||
|
if not isinstance(value, dict):
|
||||||
|
raise ValueError(f"{path} must be an object")
|
||||||
|
normalized: dict[str, list[dict[str, Any]]] = {}
|
||||||
|
for table, rows in value.items():
|
||||||
|
if (
|
||||||
|
not isinstance(table, str)
|
||||||
|
or not isinstance(rows, list)
|
||||||
|
or any(not isinstance(row, dict) for row in rows)
|
||||||
|
):
|
||||||
|
raise ValueError(f"{path} must map table names to row arrays")
|
||||||
|
normalized[table] = [
|
||||||
|
replay_receipt.normalize_typed_row(row, path=f"{path}.{table}[{index}]")
|
||||||
|
for index, row in enumerate(rows)
|
||||||
|
]
|
||||||
|
return normalized
|
||||||
|
|
||||||
|
try:
|
||||||
|
normalized_actual = normalize(actual, "actual canonical rows")
|
||||||
|
normalized_expected = normalize(expected, "expected canonical rows")
|
||||||
|
except ValueError as exc:
|
||||||
|
raise ReconstructionError(code, f"{action} returned invalid typed canonical rows") from exc
|
||||||
|
_assert_exact_json(normalized_actual, normalized_expected, code=code, action=action)
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_exact_proposal_readback(actual: Any, expected: Any, *, code: str, action: str) -> None:
|
||||||
|
def normalize(value: Any, path: str) -> dict[str, Any]:
|
||||||
|
if not isinstance(value, dict):
|
||||||
|
raise ValueError(f"{path} must be an object")
|
||||||
|
normalized = dict(value)
|
||||||
|
for field in ("proposal", "approval_snapshot", "approved_proposal_snapshot"):
|
||||||
|
row = normalized.get(field)
|
||||||
|
if not isinstance(row, dict):
|
||||||
|
raise ValueError(f"{path}.{field} must be an object")
|
||||||
|
normalized[field] = replay_receipt.normalize_typed_row(row, path=f"{path}.{field}")
|
||||||
|
return normalized
|
||||||
|
|
||||||
|
try:
|
||||||
|
normalized_actual = normalize(actual, "actual proposal readback")
|
||||||
|
normalized_expected = normalize(expected, "expected proposal readback")
|
||||||
|
except ValueError as exc:
|
||||||
|
raise ReconstructionError(code, f"{action} returned invalid typed proposal rows") from exc
|
||||||
|
_assert_exact_json(normalized_actual, normalized_expected, code=code, action=action)
|
||||||
|
|
||||||
|
|
||||||
def _entry_summary(entry: LedgerEntry) -> dict[str, Any]:
|
def _entry_summary(entry: LedgerEntry) -> dict[str, Any]:
|
||||||
proposal = entry.applied_proposal
|
proposal = entry.applied_proposal
|
||||||
return {
|
return {
|
||||||
"sequence": entry.sequence,
|
"sequence": entry.sequence,
|
||||||
"proposal_id": proposal["id"],
|
"proposal_id": proposal["id"],
|
||||||
"proposal_type": proposal["proposal_type"],
|
"proposal_type": proposal["proposal_type"],
|
||||||
|
"contract_version": entry.contract_version,
|
||||||
"material_sha256": entry.material_sha256,
|
"material_sha256": entry.material_sha256,
|
||||||
"replay_material_sha256": entry.replay_material_sha256,
|
"replay_material_sha256": entry.replay_material_sha256,
|
||||||
"original_apply_sql_sha256": entry.receipt["apply_engine"]["apply_sql_sha256"],
|
"original_apply_sql_sha256": entry.receipt["apply_engine"]["apply_sql_sha256"],
|
||||||
|
|
@ -1142,11 +1712,12 @@ def apply_entry(
|
||||||
code="entry_seed_readback_failed",
|
code="entry_seed_readback_failed",
|
||||||
action=f"ledger entry {entry.sequence} proposal seed readback",
|
action=f"ledger entry {entry.sequence} proposal seed readback",
|
||||||
)
|
)
|
||||||
_assert_exact_json(
|
_assert_exact_proposal_readback(
|
||||||
proposal_readback,
|
proposal_readback,
|
||||||
{
|
{
|
||||||
"proposal": entry.approved_proposal,
|
"proposal": entry.approved_proposal,
|
||||||
"approval_snapshot": entry.approval_snapshot,
|
"approval_snapshot": entry.approval_snapshot,
|
||||||
|
"approved_proposal_snapshot": entry.approved_proposal,
|
||||||
},
|
},
|
||||||
code="entry_seed_mismatch",
|
code="entry_seed_mismatch",
|
||||||
action=f"ledger entry {entry.sequence} proposal seed",
|
action=f"ledger entry {entry.sequence} proposal seed",
|
||||||
|
|
@ -1163,6 +1734,21 @@ def apply_entry(
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
summary["mutating_prestate_captured"] = True
|
summary["mutating_prestate_captured"] = True
|
||||||
|
elif entry.contract_version == 3:
|
||||||
|
seeded_rows = _read_json(
|
||||||
|
args,
|
||||||
|
container,
|
||||||
|
replay_receipt.build_postflight_sql(entry.receipt["proposal"]),
|
||||||
|
code="entry_seed_readback_failed",
|
||||||
|
action=f"ledger entry {entry.sequence} V3 empty canonical prestate readback",
|
||||||
|
)
|
||||||
|
_assert_exact_canonical_rows(
|
||||||
|
seeded_rows,
|
||||||
|
{table: [] for table in entry.receipt["canonical_rows"]},
|
||||||
|
code="entry_seed_mismatch",
|
||||||
|
action=f"ledger entry {entry.sequence} V3 empty canonical prestate",
|
||||||
|
)
|
||||||
|
summary["canonical_seed_exact"] = True
|
||||||
else:
|
else:
|
||||||
seeded_rows = _read_json(
|
seeded_rows = _read_json(
|
||||||
args,
|
args,
|
||||||
|
|
@ -1171,7 +1757,7 @@ def apply_entry(
|
||||||
code="entry_seed_readback_failed",
|
code="entry_seed_readback_failed",
|
||||||
action=f"ledger entry {entry.sequence} canonical seed readback",
|
action=f"ledger entry {entry.sequence} canonical seed readback",
|
||||||
)
|
)
|
||||||
_assert_exact_json(
|
_assert_exact_canonical_rows(
|
||||||
seeded_rows,
|
seeded_rows,
|
||||||
entry.receipt["canonical_rows"],
|
entry.receipt["canonical_rows"],
|
||||||
code="entry_seed_mismatch",
|
code="entry_seed_mismatch",
|
||||||
|
|
@ -1211,6 +1797,17 @@ def apply_entry(
|
||||||
action=f"ledger entry {entry.sequence} revise_strategy exact poststate normalization",
|
action=f"ledger entry {entry.sequence} revise_strategy exact poststate normalization",
|
||||||
)
|
)
|
||||||
summary["mutating_poststate_normalized"] = True
|
summary["mutating_poststate_normalized"] = True
|
||||||
|
elif entry.contract_version == 3:
|
||||||
|
_psql(
|
||||||
|
args,
|
||||||
|
container,
|
||||||
|
build_v3_canonical_timestamp_normalization_sql(entry),
|
||||||
|
role="postgres",
|
||||||
|
timeout=args.command_timeout,
|
||||||
|
code="v3_canonical_timestamp_normalization_failed",
|
||||||
|
action=f"ledger entry {entry.sequence} V3 canonical timestamp normalization",
|
||||||
|
)
|
||||||
|
summary["mutating_poststate_normalized"] = True
|
||||||
|
|
||||||
_psql(
|
_psql(
|
||||||
args,
|
args,
|
||||||
|
|
@ -1228,11 +1825,12 @@ def apply_entry(
|
||||||
code="entry_postflight_failed",
|
code="entry_postflight_failed",
|
||||||
action=f"ledger entry {entry.sequence} applied proposal readback",
|
action=f"ledger entry {entry.sequence} applied proposal readback",
|
||||||
)
|
)
|
||||||
_assert_exact_json(
|
_assert_exact_proposal_readback(
|
||||||
proposal_readback,
|
proposal_readback,
|
||||||
{
|
{
|
||||||
"proposal": entry.applied_proposal,
|
"proposal": entry.applied_proposal,
|
||||||
"approval_snapshot": entry.approval_snapshot,
|
"approval_snapshot": entry.approval_snapshot,
|
||||||
|
"approved_proposal_snapshot": entry.approved_proposal,
|
||||||
},
|
},
|
||||||
code="entry_postflight_mismatch",
|
code="entry_postflight_mismatch",
|
||||||
action=f"ledger entry {entry.sequence} applied proposal",
|
action=f"ledger entry {entry.sequence} applied proposal",
|
||||||
|
|
@ -1246,7 +1844,7 @@ def apply_entry(
|
||||||
code="entry_postflight_failed",
|
code="entry_postflight_failed",
|
||||||
action=f"ledger entry {entry.sequence} exact row readback",
|
action=f"ledger entry {entry.sequence} exact row readback",
|
||||||
)
|
)
|
||||||
_assert_exact_json(
|
_assert_exact_canonical_rows(
|
||||||
final_rows,
|
final_rows,
|
||||||
entry.receipt["canonical_rows"],
|
entry.receipt["canonical_rows"],
|
||||||
code="entry_postflight_mismatch",
|
code="entry_postflight_mismatch",
|
||||||
|
|
@ -1314,6 +1912,7 @@ def _initial_receipt(args: argparse.Namespace, container: str) -> dict[str, Any]
|
||||||
"genesis_dump": {"bytes": None, "sha256": None, "pin_verified": False},
|
"genesis_dump": {"bytes": None, "sha256": None, "pin_verified": False},
|
||||||
"genesis_manifest": {"bytes": None, "sha256": None, "pin_verified": False},
|
"genesis_manifest": {"bytes": None, "sha256": None, "pin_verified": False},
|
||||||
"ledger": {"bytes": None, "sha256": None, "pin_verified": False},
|
"ledger": {"bytes": None, "sha256": None, "pin_verified": False},
|
||||||
|
"admission_manifest": {"bytes": None, "sha256": None, "pin_verified": False},
|
||||||
"final_manifest": {"bytes": None, "sha256": None, "pin_verified": False},
|
"final_manifest": {"bytes": None, "sha256": None, "pin_verified": False},
|
||||||
"engine_hashes": {},
|
"engine_hashes": {},
|
||||||
},
|
},
|
||||||
|
|
@ -1327,9 +1926,14 @@ def _initial_receipt(args: argparse.Namespace, container: str) -> dict[str, Any]
|
||||||
"isolation": None,
|
"isolation": None,
|
||||||
},
|
},
|
||||||
"guard_prerequisites": {"status": "pending", "sha256": None},
|
"guard_prerequisites": {"status": "pending", "sha256": None},
|
||||||
|
"v3_contract": {"status": "not_required", "sha256": None},
|
||||||
"genesis_parity": {"status": "pending"},
|
"genesis_parity": {"status": "pending"},
|
||||||
"genesis_key_counts": {},
|
"genesis_key_counts": {},
|
||||||
"ledger": {"entry_count": None, "entries": []},
|
"ledger": {"entry_count": None, "entries": []},
|
||||||
|
"admission": {
|
||||||
|
"status": "pending",
|
||||||
|
"canonical_retrieval_eligible": False,
|
||||||
|
},
|
||||||
"final_parity": {"status": "pending"},
|
"final_parity": {"status": "pending"},
|
||||||
"final_key_counts": {},
|
"final_key_counts": {},
|
||||||
"cleanup": {
|
"cleanup": {
|
||||||
|
|
@ -1375,6 +1979,11 @@ def run_reconstruction(args: argparse.Namespace) -> dict[str, Any]:
|
||||||
"sha256": bundle.ledger_sha256,
|
"sha256": bundle.ledger_sha256,
|
||||||
"pin_verified": True,
|
"pin_verified": True,
|
||||||
},
|
},
|
||||||
|
"admission_manifest": {
|
||||||
|
"bytes": bundle.admission_manifest_bytes,
|
||||||
|
"sha256": bundle.admission_manifest_sha256,
|
||||||
|
"pin_verified": bundle.admission_manifest_sha256 is not None,
|
||||||
|
},
|
||||||
"final_manifest": {
|
"final_manifest": {
|
||||||
"bytes": bundle.final_manifest_bytes,
|
"bytes": bundle.final_manifest_bytes,
|
||||||
"sha256": bundle.final_manifest_sha256,
|
"sha256": bundle.final_manifest_sha256,
|
||||||
|
|
@ -1384,6 +1993,14 @@ def run_reconstruction(args: argparse.Namespace) -> dict[str, Any]:
|
||||||
}
|
}
|
||||||
receipt["ledger"]["entry_count"] = len(bundle.entries)
|
receipt["ledger"]["entry_count"] = len(bundle.entries)
|
||||||
receipt["ledger"]["entries"] = [_entry_summary(entry) for entry in bundle.entries]
|
receipt["ledger"]["entries"] = [_entry_summary(entry) for entry in bundle.entries]
|
||||||
|
receipt["admission"] = bundle.admission
|
||||||
|
if bundle.admission["canonical_retrieval_eligible"] is not True:
|
||||||
|
receipt["claim_ceiling"] = f"forensic-only reconstruction; {CLAIM_CEILING}"
|
||||||
|
if any(entry.contract_version == 3 for entry in bundle.entries):
|
||||||
|
receipt["v3_contract"] = {
|
||||||
|
"status": "pending",
|
||||||
|
"sha256": bundle.engine_hashes["v3_contract_sha256"],
|
||||||
|
}
|
||||||
|
|
||||||
phase = "container_name_preflight"
|
phase = "container_name_preflight"
|
||||||
absent, absence_error = canonical_rebuild.inspect_container_absence(
|
absent, absence_error = canonical_rebuild.inspect_container_absence(
|
||||||
|
|
@ -1548,7 +2165,13 @@ def run_reconstruction(args: argparse.Namespace) -> dict[str, Any]:
|
||||||
)
|
)
|
||||||
|
|
||||||
phase = "ordered_ledger_replay"
|
phase = "ordered_ledger_replay"
|
||||||
|
v3_installed = False
|
||||||
for entry, summary in zip(bundle.entries, receipt["ledger"]["entries"], strict=True):
|
for entry, summary in zip(bundle.entries, receipt["ledger"]["entries"], strict=True):
|
||||||
|
if entry.contract_version == 3 and not v3_installed:
|
||||||
|
phase = "v3_contract_install"
|
||||||
|
receipt["v3_contract"] = install_v3_contract(args, container)
|
||||||
|
v3_installed = True
|
||||||
|
phase = "ordered_ledger_replay"
|
||||||
apply_entry(args, container, entry, summary)
|
apply_entry(args, container, entry, summary)
|
||||||
|
|
||||||
phase = "final_parity"
|
phase = "final_parity"
|
||||||
|
|
@ -1651,6 +2274,8 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||||
parser.add_argument("--genesis-manifest", required=True, type=Path)
|
parser.add_argument("--genesis-manifest", required=True, type=Path)
|
||||||
parser.add_argument("--ledger", required=True, type=Path)
|
parser.add_argument("--ledger", required=True, type=Path)
|
||||||
parser.add_argument("--ledger-sha256", required=True)
|
parser.add_argument("--ledger-sha256", required=True)
|
||||||
|
parser.add_argument("--admission-manifest", type=Path)
|
||||||
|
parser.add_argument("--admission-manifest-sha256")
|
||||||
parser.add_argument("--database", default="teleo")
|
parser.add_argument("--database", default="teleo")
|
||||||
parser.add_argument("--image", default=DEFAULT_REBUILD_IMAGE)
|
parser.add_argument("--image", default=DEFAULT_REBUILD_IMAGE)
|
||||||
parser.add_argument("--container-prefix", default="teleo-genesis-ledger-rebuild")
|
parser.add_argument("--container-prefix", default="teleo-genesis-ledger-rebuild")
|
||||||
|
|
@ -1698,6 +2323,13 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||||
parser.error(f"{flag} must be an existing regular file")
|
parser.error(f"{flag} must be an existing regular file")
|
||||||
if not SHA256_RE.fullmatch(args.ledger_sha256):
|
if not SHA256_RE.fullmatch(args.ledger_sha256):
|
||||||
parser.error("--ledger-sha256 must be a lowercase SHA-256 digest")
|
parser.error("--ledger-sha256 must be a lowercase SHA-256 digest")
|
||||||
|
if (args.admission_manifest is None) != (args.admission_manifest_sha256 is None):
|
||||||
|
parser.error("--admission-manifest and --admission-manifest-sha256 must be supplied together")
|
||||||
|
if args.admission_manifest is not None:
|
||||||
|
if not args.admission_manifest.is_file():
|
||||||
|
parser.error("--admission-manifest must be an existing regular file")
|
||||||
|
if not SHA256_RE.fullmatch(args.admission_manifest_sha256):
|
||||||
|
parser.error("--admission-manifest-sha256 must be a lowercase SHA-256 digest")
|
||||||
try:
|
try:
|
||||||
protected = {
|
protected = {
|
||||||
args.genesis_dump.resolve(),
|
args.genesis_dump.resolve(),
|
||||||
|
|
@ -1705,6 +2337,8 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||||
*_declared_artifact_paths(args.ledger),
|
*_declared_artifact_paths(args.ledger),
|
||||||
*FIXED_ENGINE_PATHS,
|
*FIXED_ENGINE_PATHS,
|
||||||
}
|
}
|
||||||
|
if args.admission_manifest is not None:
|
||||||
|
protected.add(args.admission_manifest.resolve())
|
||||||
except ReconstructionError as exc:
|
except ReconstructionError as exc:
|
||||||
parser.error(exc.safe_message)
|
parser.error(exc.safe_message)
|
||||||
if args.output.resolve() in protected:
|
if args.output.resolve() in protected:
|
||||||
|
|
|
||||||
1023
ops/shard_corpus_extraction_work.py
Normal file
1023
ops/shard_corpus_extraction_work.py
Normal file
File diff suppressed because it is too large
Load diff
2293
ops/teleo_v3_epistemic_contract.sql
Normal file
2293
ops/teleo_v3_epistemic_contract.sql
Normal file
File diff suppressed because it is too large
Load diff
1926
ops/verify_teleo_v3_epistemic_contract.py
Normal file
1926
ops/verify_teleo_v3_epistemic_contract.py
Normal file
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
|
|
@ -24,6 +24,9 @@ Safety posture
|
||||||
* Gated OFF by default. Without ``--enable`` (or ``KB_APPLY_WORKER_ENABLED=1``) the
|
* Gated OFF by default. Without ``--enable`` (or ``KB_APPLY_WORKER_ENABLED=1``) the
|
||||||
worker only REPORTS what it would apply -- it performs no writes. Flip it on only
|
worker only REPORTS what it would apply -- it performs no writes. Flip it on only
|
||||||
after the first apply has been proven by hand.
|
after the first apply has been proven by hand.
|
||||||
|
* Pinned to one proposal contract version. V2 remains the default; a V3 worker
|
||||||
|
must be selected explicitly with ``--contract-version 3`` or
|
||||||
|
``KB_APPLY_WORKER_CONTRACT_VERSION=3`` after the V3 schema is installed.
|
||||||
* The optional render step is a configurable hook (``--render-cmd`` /
|
* The optional render step is a configurable hook (``--render-cmd`` /
|
||||||
``KB_APPLY_RENDER_CMD``). Only ``revise_strategy`` currently has a proven
|
``KB_APPLY_RENDER_CMD``). Only ``revise_strategy`` currently has a proven
|
||||||
agent-owned render contract. Other apply types update canonical state without
|
agent-owned render contract. Other apply types update canonical state without
|
||||||
|
|
@ -38,6 +41,7 @@ import os
|
||||||
import stat
|
import stat
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
|
import uuid
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
|
|
@ -56,6 +60,9 @@ RENDER_TYPES = ("revise_strategy",)
|
||||||
|
|
||||||
POSTCOMMIT_RECEIPT_FAILURE_MARKER = "committed, but private replay receipt capture failed"
|
POSTCOMMIT_RECEIPT_FAILURE_MARKER = "committed, but private replay receipt capture failed"
|
||||||
POSTCOMMIT_RECEIPT_RECOVERY_MARKER = ". Recover without reapplying via: "
|
POSTCOMMIT_RECEIPT_RECOVERY_MARKER = ". Recover without reapplying via: "
|
||||||
|
CONTRACT_STATE_V2 = "V2"
|
||||||
|
CONTRACT_STATE_V3 = "V3"
|
||||||
|
CONTRACT_STATE_INVALID = "INVALID"
|
||||||
|
|
||||||
|
|
||||||
class PostCommitReceiptError(RuntimeError):
|
class PostCommitReceiptError(RuntimeError):
|
||||||
|
|
@ -69,28 +76,71 @@ def build_candidate_query(
|
||||||
types: tuple = WORKER_TYPES,
|
types: tuple = WORKER_TYPES,
|
||||||
limit: int = 20,
|
limit: int = 20,
|
||||||
excluded_ids: tuple = (),
|
excluded_ids: tuple = (),
|
||||||
|
contract_version: int = 2,
|
||||||
|
proposal_id: str | None = None,
|
||||||
) -> str:
|
) -> str:
|
||||||
"""SELECT approved, applyable proposals that carry an apply_payload.
|
"""SELECT approved proposals for one exact apply contract version.
|
||||||
|
|
||||||
Filters on status='approved' at the DB level -- this is the structural
|
Filters on status='approved' at the DB level -- this is the structural
|
||||||
guarantee that the worker cannot act on anything a human has not approved.
|
guarantee that the worker cannot act on anything a human has not approved.
|
||||||
IDs at the failure ceiling are excluded before LIMIT so exhausted rows cannot
|
IDs at the failure ceiling are excluded before LIMIT so exhausted rows cannot
|
||||||
occupy every fetched slot and starve later eligible proposals.
|
occupy every fetched slot and starve later eligible proposals.
|
||||||
"""
|
"""
|
||||||
type_list = ", ".join(ap.sql_literal(t) for t in types)
|
if contract_version not in {2, 3}:
|
||||||
|
raise ValueError("worker contract_version must be 2 or 3")
|
||||||
|
requested_types = tuple(dict.fromkeys(types))
|
||||||
|
if contract_version == 3:
|
||||||
|
eligible_types = tuple(proposal_type for proposal_type in requested_types if proposal_type == "approve_claim")
|
||||||
|
if not eligible_types:
|
||||||
|
raise ValueError("V3 worker requires proposal_type 'approve_claim'")
|
||||||
|
type_contract_clause = (
|
||||||
|
"proposal_type = 'approve_claim'\n"
|
||||||
|
" and pg_catalog.jsonb_typeof(payload->'apply_payload'->'contract_version') = 'number'\n"
|
||||||
|
" and payload->'apply_payload'->>'contract_version' = '3'"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
eligible_types = tuple(proposal_type for proposal_type in requested_types if proposal_type in WORKER_TYPES)
|
||||||
|
if not eligible_types:
|
||||||
|
raise ValueError("V2 worker has no supported proposal types")
|
||||||
|
legacy_types = tuple(proposal_type for proposal_type in eligible_types if proposal_type != "approve_claim")
|
||||||
|
clauses = []
|
||||||
|
if legacy_types:
|
||||||
|
legacy_type_list = ", ".join(ap.sql_literal(proposal_type) for proposal_type in legacy_types)
|
||||||
|
clauses.append(
|
||||||
|
f"(proposal_type in ({legacy_type_list}) and not (payload->'apply_payload' ? 'contract_version'))"
|
||||||
|
)
|
||||||
|
if "approve_claim" in eligible_types:
|
||||||
|
clauses.append(
|
||||||
|
"(proposal_type = 'approve_claim' "
|
||||||
|
"and pg_catalog.jsonb_typeof(payload->'apply_payload'->'contract_version') = 'number' "
|
||||||
|
"and payload->'apply_payload'->>'contract_version' = '2')"
|
||||||
|
)
|
||||||
|
type_contract_clause = "(\n " + "\n or ".join(clauses) + "\n )"
|
||||||
excluded = tuple(str(proposal_id) for proposal_id in excluded_ids)
|
excluded = tuple(str(proposal_id) for proposal_id in excluded_ids)
|
||||||
exclusion_clause = ""
|
exclusion_clause = ""
|
||||||
if excluded:
|
if excluded:
|
||||||
excluded_list = ", ".join(ap.sql_literal(proposal_id) for proposal_id in excluded)
|
excluded_list = ", ".join(ap.sql_literal(proposal_id) for proposal_id in excluded)
|
||||||
exclusion_clause = f"\n and id::text not in ({excluded_list})"
|
exclusion_clause = f"\n and id::text not in ({excluded_list})"
|
||||||
|
exact_clause = ""
|
||||||
|
if proposal_id is not None:
|
||||||
|
try:
|
||||||
|
canonical_proposal_id = str(uuid.UUID(str(proposal_id)))
|
||||||
|
except ValueError as exc:
|
||||||
|
raise ValueError("proposal_id must be a full canonical UUID") from exc
|
||||||
|
if str(proposal_id) != canonical_proposal_id:
|
||||||
|
raise ValueError("proposal_id must be a canonical lowercase UUID")
|
||||||
|
exact_clause = f"\n and id = {ap.sql_literal(canonical_proposal_id)}::uuid"
|
||||||
return f"""select jsonb_build_object(
|
return f"""select jsonb_build_object(
|
||||||
'id', id::text,
|
'id', id::text,
|
||||||
'proposal_type', proposal_type,
|
'proposal_type', proposal_type,
|
||||||
'agent_id', payload->'apply_payload'->>'agent_id')::text
|
'agent_id', payload->'apply_payload'->>'agent_id',
|
||||||
|
'contract_version', coalesce(payload->'apply_payload'->>'contract_version', '2'))::text
|
||||||
from kb_stage.kb_proposals
|
from kb_stage.kb_proposals
|
||||||
where status = 'approved'
|
where status = 'approved'
|
||||||
and proposal_type in ({type_list})
|
and pg_catalog.jsonb_typeof(payload) = 'object'
|
||||||
and payload ? 'apply_payload'{exclusion_clause}
|
and payload ? 'apply_payload'
|
||||||
|
and pg_catalog.jsonb_typeof(payload->'apply_payload') = 'object'
|
||||||
|
and {type_contract_clause}{exact_clause}{exclusion_clause}
|
||||||
order by created_at asc
|
order by created_at asc
|
||||||
limit {int(limit)};"""
|
limit {int(limit)};"""
|
||||||
|
|
||||||
|
|
@ -238,9 +288,29 @@ def fetch_candidates(
|
||||||
password: str,
|
password: str,
|
||||||
excluded_ids: tuple,
|
excluded_ids: tuple,
|
||||||
) -> list[dict[str, Any]]:
|
) -> list[dict[str, Any]]:
|
||||||
sql = build_candidate_query(limit=args.limit, excluded_ids=excluded_ids)
|
sql = build_candidate_query(
|
||||||
|
limit=args.limit,
|
||||||
|
excluded_ids=excluded_ids,
|
||||||
|
contract_version=args.contract_version,
|
||||||
|
proposal_id=getattr(args, "proposal_id", None),
|
||||||
|
)
|
||||||
out = ap.run_psql(_psql_args(args), sql, password)
|
out = ap.run_psql(_psql_args(args), sql, password)
|
||||||
return parse_candidates(out)
|
candidates = parse_candidates(out)
|
||||||
|
expected_version = str(args.contract_version)
|
||||||
|
allowed_types = {"approve_claim"} if args.contract_version == 3 else set(WORKER_TYPES)
|
||||||
|
for candidate in candidates:
|
||||||
|
if candidate.get("proposal_type") not in allowed_types or candidate.get("contract_version") != expected_version:
|
||||||
|
raise RuntimeError("candidate query returned a proposal outside the exact worker contract")
|
||||||
|
return candidates
|
||||||
|
|
||||||
|
|
||||||
|
def detect_installed_contract(args: argparse.Namespace, password: str) -> str:
|
||||||
|
"""Classify the exact database apply contract before candidate selection."""
|
||||||
|
sql = ap.build_v3_contract_state_query() + ";"
|
||||||
|
raw = ap.run_psql(_psql_args(args), sql, password).strip()
|
||||||
|
if raw not in {CONTRACT_STATE_V2, CONTRACT_STATE_V3, CONTRACT_STATE_INVALID}:
|
||||||
|
raise RuntimeError("database apply contract readback was not V2, V3, or INVALID")
|
||||||
|
return raw
|
||||||
|
|
||||||
|
|
||||||
def apply_one(args: argparse.Namespace, proposal_id: str) -> None:
|
def apply_one(args: argparse.Namespace, proposal_id: str) -> None:
|
||||||
|
|
@ -250,6 +320,8 @@ def apply_one(args: argparse.Namespace, proposal_id: str) -> None:
|
||||||
sys.executable,
|
sys.executable,
|
||||||
str(args.apply_script),
|
str(args.apply_script),
|
||||||
proposal_id,
|
proposal_id,
|
||||||
|
"--expected-contract-version",
|
||||||
|
str(args.contract_version),
|
||||||
"--applied-by",
|
"--applied-by",
|
||||||
args.applied_by,
|
args.applied_by,
|
||||||
"--secrets-file",
|
"--secrets-file",
|
||||||
|
|
@ -267,6 +339,9 @@ def apply_one(args: argparse.Namespace, proposal_id: str) -> None:
|
||||||
"--receipt-out",
|
"--receipt-out",
|
||||||
str(receipt_path),
|
str(receipt_path),
|
||||||
]
|
]
|
||||||
|
fresh_preflight = getattr(args, "fresh_preflight", None)
|
||||||
|
if fresh_preflight is not None:
|
||||||
|
cmd.extend(("--fresh-preflight", str(fresh_preflight)))
|
||||||
result = subprocess.run(cmd, text=True, capture_output=True, check=False)
|
result = subprocess.run(cmd, text=True, capture_output=True, check=False)
|
||||||
committed = result.returncode == 0 or has_exact_postcommit_receipt_failure(
|
committed = result.returncode == 0 or has_exact_postcommit_receipt_failure(
|
||||||
result.stderr,
|
result.stderr,
|
||||||
|
|
@ -327,10 +402,56 @@ def render_one(args: argparse.Namespace, agent_id: str | None) -> str:
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
def run(args: argparse.Namespace) -> int:
|
def run(args: argparse.Namespace) -> int:
|
||||||
password = ap.load_password(args.secrets_file)
|
password = ap.load_password(args.secrets_file)
|
||||||
|
installed_contract = detect_installed_contract(args, password)
|
||||||
|
if installed_contract == CONTRACT_STATE_INVALID:
|
||||||
|
print(
|
||||||
|
"REFUSE execution: database apply contract integrity is INVALID; "
|
||||||
|
"no candidate selection or writes attempted.",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
return 1
|
||||||
|
if args.contract_version == 3 and installed_contract == CONTRACT_STATE_V2:
|
||||||
|
print(
|
||||||
|
"REFUSE contract v3 execution: the database V3 cutover is not installed; "
|
||||||
|
"no candidate selection or writes attempted.",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
return 1
|
||||||
|
|
||||||
failure_counts = load_failure_state(args.failure_state_file)
|
failure_counts = load_failure_state(args.failure_state_file)
|
||||||
exhausted_ids = tuple(proposal_id for proposal_id, count in failure_counts.items() if count >= args.max_attempts)
|
exhausted_ids = tuple(proposal_id for proposal_id, count in failure_counts.items() if count >= args.max_attempts)
|
||||||
candidates = fetch_candidates(args, password, exhausted_ids)
|
fetch_exclusions = () if args.contract_version == 2 and installed_contract == CONTRACT_STATE_V3 else exhausted_ids
|
||||||
|
candidates = fetch_candidates(args, password, fetch_exclusions)
|
||||||
|
|
||||||
|
exact_proposal_id = getattr(args, "proposal_id", None)
|
||||||
|
if exact_proposal_id is not None and (len(candidates) != 1 or candidates[0].get("id") != exact_proposal_id):
|
||||||
|
if not candidates and getattr(args, "require_candidate", False):
|
||||||
|
print(
|
||||||
|
f"required approved+applyable contract v{args.contract_version} proposal "
|
||||||
|
f"{exact_proposal_id} was not selected; no writes attempted.",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
return 2
|
||||||
|
print(
|
||||||
|
f"REFUSE exact proposal execution: approved+applyable contract v{args.contract_version} "
|
||||||
|
f"proposal {exact_proposal_id} was not selected exactly once; no writes attempted.",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
if args.contract_version == 2 and installed_contract == CONTRACT_STATE_V3:
|
||||||
|
print(
|
||||||
|
"SKIP contract v2 execution: the database V3 cutover is installed; "
|
||||||
|
f"{len(candidates)} approved V2 proposal(s) in this fetch remain review-visible. "
|
||||||
|
"Migrate each packet to a newly reviewed V3 proposal or explicitly reject/cancel it; no writes attempted.",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
for candidate in candidates:
|
||||||
|
print(
|
||||||
|
f" retained V2 proposal {candidate['id']} ({candidate['proposal_type']})",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
return 0
|
||||||
for proposal_id in exhausted_ids:
|
for proposal_id in exhausted_ids:
|
||||||
print(
|
print(
|
||||||
f"EXCLUDE failure-ceiling ID {proposal_id}: failed "
|
f"EXCLUDE failure-ceiling ID {proposal_id}: failed "
|
||||||
|
|
@ -340,7 +461,7 @@ def run(args: argparse.Namespace) -> int:
|
||||||
)
|
)
|
||||||
|
|
||||||
if not candidates:
|
if not candidates:
|
||||||
print("no approved+applyable proposals; nothing to do")
|
print(f"no approved+applyable contract v{args.contract_version} proposals; nothing to do")
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
split = partition_candidates(candidates, failure_counts, args.max_attempts, args.max_per_tick)
|
split = partition_candidates(candidates, failure_counts, args.max_attempts, args.max_per_tick)
|
||||||
|
|
@ -356,7 +477,8 @@ def run(args: argparse.Namespace) -> int:
|
||||||
enabled = args.enable or os.environ.get("KB_APPLY_WORKER_ENABLED") == "1"
|
enabled = args.enable or os.environ.get("KB_APPLY_WORKER_ENABLED") == "1"
|
||||||
if not enabled:
|
if not enabled:
|
||||||
print(
|
print(
|
||||||
f"[report-only] worker disabled; {len(split['to_apply'])} proposal(s) would apply "
|
f"[report-only] contract v{args.contract_version} worker disabled; "
|
||||||
|
f"{len(split['to_apply'])} proposal(s) would apply "
|
||||||
f"this tick (cap {args.max_per_tick}):"
|
f"this tick (cap {args.max_per_tick}):"
|
||||||
)
|
)
|
||||||
for c in split["to_apply"]:
|
for c in split["to_apply"]:
|
||||||
|
|
@ -405,7 +527,23 @@ def parse_args(argv: list[str]) -> argparse.Namespace:
|
||||||
action="store_true",
|
action="store_true",
|
||||||
help="actually apply (default: report-only). Also honored via KB_APPLY_WORKER_ENABLED=1.",
|
help="actually apply (default: report-only). Also honored via KB_APPLY_WORKER_ENABLED=1.",
|
||||||
)
|
)
|
||||||
|
p.add_argument(
|
||||||
|
"--contract-version",
|
||||||
|
type=int,
|
||||||
|
choices=(2, 3),
|
||||||
|
default=os.environ.get("KB_APPLY_WORKER_CONTRACT_VERSION", "2"),
|
||||||
|
help="apply only proposals for one exact contract version (default 2)",
|
||||||
|
)
|
||||||
p.add_argument("--limit", type=int, default=20, help="max candidates fetched per tick")
|
p.add_argument("--limit", type=int, default=20, help="max candidates fetched per tick")
|
||||||
|
p.add_argument(
|
||||||
|
"--proposal-id",
|
||||||
|
help="select exactly one canonical UUID; refuse instead of falling through to another approved proposal",
|
||||||
|
)
|
||||||
|
p.add_argument(
|
||||||
|
"--require-candidate",
|
||||||
|
action="store_true",
|
||||||
|
help="fail when --proposal-id is not selected instead of treating an empty tick as success",
|
||||||
|
)
|
||||||
p.add_argument(
|
p.add_argument(
|
||||||
"--max-per-tick",
|
"--max-per-tick",
|
||||||
type=int,
|
type=int,
|
||||||
|
|
@ -435,6 +573,11 @@ def parse_args(argv: list[str]) -> argparse.Namespace:
|
||||||
default=str(HERE / "apply_proposal.py"),
|
default=str(HERE / "apply_proposal.py"),
|
||||||
help="path to the apply_proposal.py engine (the sole apply path)",
|
help="path to the apply_proposal.py engine (the sole apply path)",
|
||||||
)
|
)
|
||||||
|
p.add_argument(
|
||||||
|
"--fresh-preflight",
|
||||||
|
type=Path,
|
||||||
|
help="exact fresh-owned preflight forwarded to the apply engine; requires --proposal-id",
|
||||||
|
)
|
||||||
p.add_argument(
|
p.add_argument(
|
||||||
"--receipt-dir",
|
"--receipt-dir",
|
||||||
default=(
|
default=(
|
||||||
|
|
@ -456,7 +599,19 @@ def parse_args(argv: list[str]) -> argparse.Namespace:
|
||||||
p.add_argument("--db", default=ap.DEFAULT_DB)
|
p.add_argument("--db", default=ap.DEFAULT_DB)
|
||||||
p.add_argument("--host", default=ap.DEFAULT_HOST)
|
p.add_argument("--host", default=ap.DEFAULT_HOST)
|
||||||
p.add_argument("--role", default=ap.DEFAULT_ROLE)
|
p.add_argument("--role", default=ap.DEFAULT_ROLE)
|
||||||
return p.parse_args(argv)
|
args = p.parse_args(argv)
|
||||||
|
if args.proposal_id is not None:
|
||||||
|
try:
|
||||||
|
canonical_proposal_id = str(uuid.UUID(args.proposal_id))
|
||||||
|
except (AttributeError, TypeError, ValueError) as exc:
|
||||||
|
p.error(f"--proposal-id must be a UUID: {exc}")
|
||||||
|
if args.proposal_id != canonical_proposal_id:
|
||||||
|
p.error("--proposal-id must be a canonical lowercase UUID")
|
||||||
|
if args.require_candidate and args.proposal_id is None:
|
||||||
|
p.error("--require-candidate requires --proposal-id")
|
||||||
|
if args.fresh_preflight is not None and args.proposal_id is None:
|
||||||
|
p.error("--fresh-preflight requires --proposal-id")
|
||||||
|
return args
|
||||||
|
|
||||||
|
|
||||||
def main(argv: list[str] | None = None) -> int:
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
|
|
||||||
|
|
@ -2,10 +2,10 @@
|
||||||
"""Compile one hash-bound source document into a pending-review KB proposal.
|
"""Compile one hash-bound source document into a pending-review KB proposal.
|
||||||
|
|
||||||
The command is deliberately build-only. It validates a raw artifact, its UTF-8
|
The command is deliberately build-only. It validates a raw artifact, its UTF-8
|
||||||
text extraction, and an extraction manifest; then it reuses Leo's rich proposal
|
text extraction, and an extraction manifest. V1/V2 reuse Leo's rich proposal
|
||||||
normalizer and strict staging preflight to produce a schema-valid
|
normalizer; V3 additionally requires the exact extractor specification and
|
||||||
``approve_claim`` child. It never connects to Postgres and never executes the
|
builds the explicit epistemic payload. The command never connects to Postgres
|
||||||
generated staging SQL.
|
or executes the generated staging SQL.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
@ -18,6 +18,7 @@ import re
|
||||||
import sys
|
import sys
|
||||||
import tempfile
|
import tempfile
|
||||||
import uuid
|
import uuid
|
||||||
|
from datetime import datetime, timezone
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
from urllib.parse import urlsplit
|
from urllib.parse import urlsplit
|
||||||
|
|
@ -31,11 +32,14 @@ import stage_normalized_proposal as stage # noqa: E402
|
||||||
MANIFEST_SCHEMA = "livingip.kbSourceExtractionManifest.v1"
|
MANIFEST_SCHEMA = "livingip.kbSourceExtractionManifest.v1"
|
||||||
MANIFEST_SCHEMA_V2 = "livingip.kbSourceExtractionManifest.v2"
|
MANIFEST_SCHEMA_V2 = "livingip.kbSourceExtractionManifest.v2"
|
||||||
MANIFEST_SCHEMAS = {MANIFEST_SCHEMA, MANIFEST_SCHEMA_V2}
|
MANIFEST_SCHEMAS = {MANIFEST_SCHEMA, MANIFEST_SCHEMA_V2}
|
||||||
|
MANIFEST_SCHEMA_V3 = "livingip.kbSourceExtractionManifest.v3"
|
||||||
BUNDLE_SCHEMA = "livingip.kbSourceProposalBundle.v1"
|
BUNDLE_SCHEMA = "livingip.kbSourceProposalBundle.v1"
|
||||||
|
BUNDLE_SCHEMA_V3 = "livingip.kbSourceProposalBundle.v3"
|
||||||
STATUS_SCHEMA = "livingip.kbSourceCompilerStatus.v1"
|
STATUS_SCHEMA = "livingip.kbSourceCompilerStatus.v1"
|
||||||
SHA256_RE = re.compile(r"^[0-9a-f]{64}$")
|
SHA256_RE = re.compile(r"^[0-9a-f]{64}$")
|
||||||
KEY_RE = re.compile(r"^[a-z0-9][a-z0-9._:-]{0,127}$")
|
KEY_RE = re.compile(r"^[a-z0-9][a-z0-9._:-]{0,127}$")
|
||||||
URI_SCHEME_RE = re.compile(r"^[a-z][a-z0-9+.-]*:", re.IGNORECASE)
|
URI_SCHEME_RE = re.compile(r"^[a-z][a-z0-9+.-]*:", re.IGNORECASE)
|
||||||
|
ISO_TIMESTAMP_RE = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{1,6})?(?:Z|[+-]\d{2}:\d{2})$")
|
||||||
UNSAFE_URI_SCHEMES = {"data", "javascript", "vbscript"}
|
UNSAFE_URI_SCHEMES = {"data", "javascript", "vbscript"}
|
||||||
|
|
||||||
TOP_LEVEL_KEYS = {
|
TOP_LEVEL_KEYS = {
|
||||||
|
|
@ -55,6 +59,57 @@ DEDUPE_KEYS = {"database_search_query", "candidate_claims", "duplicates"}
|
||||||
DEDUPE_CANDIDATE_KEYS = {"id", "text", "score", "evidence_count", "edge_count"}
|
DEDUPE_CANDIDATE_KEYS = {"id", "text", "score", "evidence_count", "edge_count"}
|
||||||
DEDUPE_DUPLICATE_KEYS = {"claim_id", "source_quote", "reason"}
|
DEDUPE_DUPLICATE_KEYS = {"claim_id", "source_quote", "reason"}
|
||||||
|
|
||||||
|
TOP_LEVEL_KEYS_V3 = {
|
||||||
|
"schema",
|
||||||
|
"artifact_sha256",
|
||||||
|
"extracted_text_sha256",
|
||||||
|
"extractor",
|
||||||
|
"agent_id",
|
||||||
|
"source",
|
||||||
|
"claims",
|
||||||
|
}
|
||||||
|
EXTRACTOR_KEYS_V3 = {"name", "version", "spec_sha256"}
|
||||||
|
SOURCE_KEYS_V3 = {
|
||||||
|
"identity",
|
||||||
|
"source_key",
|
||||||
|
"source_type",
|
||||||
|
"title",
|
||||||
|
"locator",
|
||||||
|
"access_scope",
|
||||||
|
"ingestion_origin",
|
||||||
|
"provenance_status",
|
||||||
|
"source_class",
|
||||||
|
"auto_promotion_policy",
|
||||||
|
"captured_at",
|
||||||
|
}
|
||||||
|
CLAIM_KEYS_V3 = {
|
||||||
|
"claim_key",
|
||||||
|
"type",
|
||||||
|
"proposition",
|
||||||
|
"body_markdown",
|
||||||
|
"scope",
|
||||||
|
"access_scope",
|
||||||
|
"owner_agent_id",
|
||||||
|
"status",
|
||||||
|
"revision_criteria",
|
||||||
|
"revision_kind",
|
||||||
|
"supersedes_id",
|
||||||
|
"tags",
|
||||||
|
"evidence",
|
||||||
|
"assessment",
|
||||||
|
}
|
||||||
|
EVIDENCE_KEYS_V3 = {"evidence_key", "quote", "polarity", "locator", "rationale"}
|
||||||
|
LOCATOR_KEYS_V3 = {"unit", "start", "end"}
|
||||||
|
ASSESSMENT_KEYS_V3 = {
|
||||||
|
"support_tier",
|
||||||
|
"challenge_tier",
|
||||||
|
"overall_state",
|
||||||
|
"rationale",
|
||||||
|
"supersedes_assessment_id",
|
||||||
|
}
|
||||||
|
TRUTH_APT_V3_CLAIM_TYPES = {"empirical", "predictive", "causal"}
|
||||||
|
ASSESSMENT_RELEVANT_POLARITIES = {"supports", "challenges"}
|
||||||
|
|
||||||
|
|
||||||
class CompilerError(RuntimeError):
|
class CompilerError(RuntimeError):
|
||||||
"""Raised when an input cannot safely become a reviewable proposal."""
|
"""Raised when an input cannot safely become a reviewable proposal."""
|
||||||
|
|
@ -72,10 +127,37 @@ def canonical_sha256(value: Any) -> str:
|
||||||
return sha256_bytes(canonical_json_bytes(value))
|
return sha256_bytes(canonical_json_bytes(value))
|
||||||
|
|
||||||
|
|
||||||
|
def _load_duplicate_free_json(value: bytes, label: str) -> Any:
|
||||||
|
try:
|
||||||
|
text = value.decode("utf-8", errors="strict")
|
||||||
|
except UnicodeDecodeError as exc:
|
||||||
|
raise CompilerError(f"{label} must be valid UTF-8 JSON: {exc}") from exc
|
||||||
|
|
||||||
|
def reject_duplicate_pairs(pairs: list[tuple[str, Any]]) -> dict[str, Any]:
|
||||||
|
result: dict[str, Any] = {}
|
||||||
|
for key, item in pairs:
|
||||||
|
if key in result:
|
||||||
|
raise CompilerError(f"{label} contains duplicate JSON object key: {key!r}")
|
||||||
|
result[key] = item
|
||||||
|
return result
|
||||||
|
|
||||||
|
try:
|
||||||
|
return json.loads(text, object_pairs_hook=reject_duplicate_pairs)
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise CompilerError(f"{label} must be valid UTF-8 JSON: {exc}") from exc
|
||||||
|
|
||||||
|
|
||||||
def stable_uuid(label: str) -> str:
|
def stable_uuid(label: str) -> str:
|
||||||
return str(uuid.uuid5(uuid.NAMESPACE_URL, f"livingip:kb-source-compiler:v1:{label}"))
|
return str(uuid.uuid5(uuid.NAMESPACE_URL, f"livingip:kb-source-compiler:v1:{label}"))
|
||||||
|
|
||||||
|
|
||||||
|
def stable_uuid_v3(row_kind: str, packet_identity_sha256: str, semantic_key: Any) -> str:
|
||||||
|
material = canonical_json_bytes(
|
||||||
|
["livingip", "kb-source-compiler", 3, packet_identity_sha256, row_kind, semantic_key]
|
||||||
|
).decode("ascii")
|
||||||
|
return str(uuid.uuid5(uuid.NAMESPACE_URL, material))
|
||||||
|
|
||||||
|
|
||||||
def _require_object(value: Any, label: str) -> dict[str, Any]:
|
def _require_object(value: Any, label: str) -> dict[str, Any]:
|
||||||
if not isinstance(value, dict):
|
if not isinstance(value, dict):
|
||||||
raise CompilerError(f"{label} must be an object")
|
raise CompilerError(f"{label} must be an object")
|
||||||
|
|
@ -88,6 +170,13 @@ def _require_exact_keys(value: dict[str, Any], allowed: set[str], label: str) ->
|
||||||
raise CompilerError(f"{label} contains unsupported keys: {', '.join(unknown)}")
|
raise CompilerError(f"{label} contains unsupported keys: {', '.join(unknown)}")
|
||||||
|
|
||||||
|
|
||||||
|
def _require_exact_shape(value: dict[str, Any], required: set[str], label: str) -> None:
|
||||||
|
_require_exact_keys(value, required, label)
|
||||||
|
missing = sorted(required - set(value))
|
||||||
|
if missing:
|
||||||
|
raise CompilerError(f"{label} is missing required keys: {', '.join(missing)}")
|
||||||
|
|
||||||
|
|
||||||
def _require_nonempty_string(value: Any, label: str) -> str:
|
def _require_nonempty_string(value: Any, label: str) -> str:
|
||||||
if not isinstance(value, str) or not value.strip():
|
if not isinstance(value, str) or not value.strip():
|
||||||
raise CompilerError(f"{label} must be a non-empty string")
|
raise CompilerError(f"{label} must be a non-empty string")
|
||||||
|
|
@ -110,6 +199,51 @@ def _require_key(value: Any, label: str) -> str:
|
||||||
return text
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
def _require_canonical_uuid(value: Any, label: str, *, nullable: bool = False) -> str | None:
|
||||||
|
if value is None and nullable:
|
||||||
|
return None
|
||||||
|
if not isinstance(value, str):
|
||||||
|
raise CompilerError(f"{label} must be a canonical lowercase hyphenated UUID")
|
||||||
|
try:
|
||||||
|
canonical = str(uuid.UUID(value))
|
||||||
|
except ValueError as exc:
|
||||||
|
raise CompilerError(f"{label} must be a canonical lowercase hyphenated UUID") from exc
|
||||||
|
if value != canonical:
|
||||||
|
raise CompilerError(f"{label} must be a canonical lowercase hyphenated UUID")
|
||||||
|
return canonical
|
||||||
|
|
||||||
|
|
||||||
|
def _require_canonical_utc_timestamp(value: Any, label: str) -> str:
|
||||||
|
text = _require_nonempty_string(value, label)
|
||||||
|
if not ISO_TIMESTAMP_RE.fullmatch(text):
|
||||||
|
raise CompilerError(f"{label} must be a timezone-aware ISO-8601 timestamp with at most 6 fractional digits")
|
||||||
|
try:
|
||||||
|
parsed = datetime.fromisoformat(text.replace("Z", "+00:00"))
|
||||||
|
except ValueError as exc:
|
||||||
|
raise CompilerError(f"{label} must be a valid ISO-8601 timestamp") from exc
|
||||||
|
if parsed.tzinfo is None:
|
||||||
|
raise CompilerError(f"{label} must include a timezone")
|
||||||
|
return parsed.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ")
|
||||||
|
|
||||||
|
|
||||||
|
def _require_nullable_enum(value: Any, allowed: set[str], label: str) -> str | None:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
if not isinstance(value, str) or value not in allowed:
|
||||||
|
raise CompilerError(f"{label} must be null or one of {sorted(allowed)}")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _require_string_list(value: Any, label: str) -> list[str]:
|
||||||
|
if not isinstance(value, list) or any(
|
||||||
|
not isinstance(item, str) or not item.strip() or item != item.strip() for item in value
|
||||||
|
):
|
||||||
|
raise CompilerError(f"{label} must be a list of non-empty strings without surrounding whitespace")
|
||||||
|
if len(value) != len(set(value)):
|
||||||
|
raise CompilerError(f"{label} contains duplicates")
|
||||||
|
return sorted(value)
|
||||||
|
|
||||||
|
|
||||||
def _require_stable_reference(value: Any, label: str) -> str:
|
def _require_stable_reference(value: Any, label: str) -> str:
|
||||||
text = _require_nonempty_string(value, label)
|
text = _require_nonempty_string(value, label)
|
||||||
if any(character.isspace() for character in text) or not URI_SCHEME_RE.match(text):
|
if any(character.isspace() for character in text) or not URI_SCHEME_RE.match(text):
|
||||||
|
|
@ -162,6 +296,258 @@ def _require_nonnegative_integer(value: Any, label: str) -> int:
|
||||||
return value
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _v3_quote_binding(text_bytes: bytes, quote: Any, locator_value: Any, label: str) -> dict[str, Any]:
|
||||||
|
exact = _require_nonempty_string(quote, f"{label}.quote")
|
||||||
|
locator = _require_object(locator_value, f"{label}.locator")
|
||||||
|
_require_exact_shape(locator, LOCATOR_KEYS_V3, f"{label}.locator")
|
||||||
|
if locator.get("unit") != "utf8_bytes":
|
||||||
|
raise CompilerError(f"{label}.locator.unit must equal 'utf8_bytes'")
|
||||||
|
start = _require_nonnegative_integer(locator.get("start"), f"{label}.locator.start")
|
||||||
|
end = _require_nonnegative_integer(locator.get("end"), f"{label}.locator.end")
|
||||||
|
quote_bytes = exact.encode("utf-8")
|
||||||
|
if end <= start or end > len(text_bytes) or text_bytes[start:end] != quote_bytes:
|
||||||
|
raise CompilerError(f"{label}.locator does not bind the exact quote bytes in the extracted text")
|
||||||
|
return {
|
||||||
|
"quote": exact,
|
||||||
|
"quote_sha256": sha256_bytes(quote_bytes),
|
||||||
|
"unit": "utf8_bytes",
|
||||||
|
"start": start,
|
||||||
|
"end": end,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_v3_assessment(value: Any, claim_type: str, evidence: list[dict[str, Any]], label: str) -> dict[str, Any]:
|
||||||
|
assessment = _require_object(value, label)
|
||||||
|
_require_exact_shape(assessment, ASSESSMENT_KEYS_V3, label)
|
||||||
|
support_tier = _require_nullable_enum(assessment.get("support_tier"), ap.V3_SUPPORT_TIERS, f"{label}.support_tier")
|
||||||
|
challenge_tier = _require_nullable_enum(
|
||||||
|
assessment.get("challenge_tier"), ap.V3_SUPPORT_TIERS, f"{label}.challenge_tier"
|
||||||
|
)
|
||||||
|
if claim_type in TRUTH_APT_V3_CLAIM_TYPES and (support_tier is None or challenge_tier is None):
|
||||||
|
raise CompilerError(f"{label} requires support_tier and challenge_tier for a truth-apt claim")
|
||||||
|
overall_state = assessment.get("overall_state")
|
||||||
|
if overall_state not in ap.V3_ASSESSMENT_STATES:
|
||||||
|
raise CompilerError(f"{label}.overall_state must be one of {sorted(ap.V3_ASSESSMENT_STATES)}")
|
||||||
|
|
||||||
|
polarities = {row["polarity"] for row in evidence}
|
||||||
|
if not polarities & ASSESSMENT_RELEVANT_POLARITIES:
|
||||||
|
raise CompilerError(f"{label} requires at least one supports or challenges evidence row")
|
||||||
|
if support_tier not in {None, "none"} and "supports" not in polarities:
|
||||||
|
raise CompilerError(f"{label}.support_tier requires supports evidence")
|
||||||
|
if challenge_tier not in {None, "none"} and "challenges" not in polarities:
|
||||||
|
raise CompilerError(f"{label}.challenge_tier requires challenges evidence")
|
||||||
|
if overall_state == "support_dominant" and "supports" not in polarities:
|
||||||
|
raise CompilerError(f"{label}.overall_state=support_dominant requires supports evidence")
|
||||||
|
if overall_state == "challenge_dominant" and "challenges" not in polarities:
|
||||||
|
raise CompilerError(f"{label}.overall_state=challenge_dominant requires challenges evidence")
|
||||||
|
if overall_state == "mixed" and not {"supports", "challenges"} <= polarities:
|
||||||
|
raise CompilerError(f"{label}.overall_state=mixed requires supports and challenges evidence")
|
||||||
|
if assessment.get("supersedes_assessment_id") is not None:
|
||||||
|
raise CompilerError(f"{label}.supersedes_assessment_id must be null for a first assessment")
|
||||||
|
return {
|
||||||
|
"support_tier": support_tier,
|
||||||
|
"challenge_tier": challenge_tier,
|
||||||
|
"overall_state": overall_state,
|
||||||
|
"rationale": _require_nonempty_string(assessment.get("rationale"), f"{label}.rationale"),
|
||||||
|
"supersedes_assessment_id": None,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_manifest_v3(
|
||||||
|
manifest: dict[str, Any],
|
||||||
|
artifact_sha256: str,
|
||||||
|
text_sha256: str,
|
||||||
|
extractor_spec_sha256: str,
|
||||||
|
text_bytes: bytes,
|
||||||
|
) -> tuple[dict[str, Any], list[dict[str, Any]]]:
|
||||||
|
_require_exact_shape(manifest, TOP_LEVEL_KEYS_V3, "manifest")
|
||||||
|
if manifest.get("schema") != MANIFEST_SCHEMA_V3:
|
||||||
|
raise CompilerError(f"manifest.schema must equal {MANIFEST_SCHEMA_V3!r}")
|
||||||
|
|
||||||
|
declared_artifact_sha256 = _require_sha256(manifest.get("artifact_sha256"), "manifest.artifact_sha256")
|
||||||
|
declared_text_sha256 = _require_sha256(manifest.get("extracted_text_sha256"), "manifest.extracted_text_sha256")
|
||||||
|
if declared_artifact_sha256 != artifact_sha256:
|
||||||
|
raise CompilerError("artifact SHA-256 does not match manifest.artifact_sha256")
|
||||||
|
if declared_text_sha256 != text_sha256:
|
||||||
|
raise CompilerError("extracted text SHA-256 does not match manifest.extracted_text_sha256")
|
||||||
|
|
||||||
|
extractor = _require_object(manifest.get("extractor"), "manifest.extractor")
|
||||||
|
_require_exact_shape(extractor, EXTRACTOR_KEYS_V3, "manifest.extractor")
|
||||||
|
declared_spec_sha256 = _require_sha256(extractor.get("spec_sha256"), "manifest.extractor.spec_sha256")
|
||||||
|
if declared_spec_sha256 != extractor_spec_sha256:
|
||||||
|
raise CompilerError("extractor specification SHA-256 does not match manifest.extractor.spec_sha256")
|
||||||
|
validated_extractor = {
|
||||||
|
"name": _require_nonempty_string(extractor.get("name"), "manifest.extractor.name"),
|
||||||
|
"version": _require_nonempty_string(extractor.get("version"), "manifest.extractor.version"),
|
||||||
|
"spec_sha256": extractor_spec_sha256,
|
||||||
|
}
|
||||||
|
agent_id = _require_canonical_uuid(manifest.get("agent_id"), "manifest.agent_id")
|
||||||
|
|
||||||
|
source = _require_object(manifest.get("source"), "manifest.source")
|
||||||
|
_require_exact_shape(source, SOURCE_KEYS_V3, "manifest.source")
|
||||||
|
source_type = source.get("source_type")
|
||||||
|
if source_type not in ap.SOURCE_TYPES:
|
||||||
|
raise CompilerError(f"manifest.source.source_type must be one of {sorted(ap.SOURCE_TYPES)}")
|
||||||
|
source_access_scope = source.get("access_scope")
|
||||||
|
if source_access_scope not in ap.V3_ACCESS_SCOPES:
|
||||||
|
raise CompilerError(f"manifest.source.access_scope must be one of {sorted(ap.V3_ACCESS_SCOPES)}")
|
||||||
|
ingestion_origin = source.get("ingestion_origin")
|
||||||
|
if ingestion_origin not in ap.V3_SOURCE_ORIGINS:
|
||||||
|
raise CompilerError(f"manifest.source.ingestion_origin must be one of {sorted(ap.V3_SOURCE_ORIGINS)}")
|
||||||
|
provenance_status = source.get("provenance_status")
|
||||||
|
if provenance_status not in ap.V3_PROVENANCE_STATES:
|
||||||
|
raise CompilerError(f"manifest.source.provenance_status must be one of {sorted(ap.V3_PROVENANCE_STATES)}")
|
||||||
|
source_class = source.get("source_class")
|
||||||
|
if source_class not in ap.V3_SOURCE_CLASSES:
|
||||||
|
raise CompilerError(f"manifest.source.source_class must be one of {sorted(ap.V3_SOURCE_CLASSES)}")
|
||||||
|
auto_promotion_policy = source.get("auto_promotion_policy")
|
||||||
|
if auto_promotion_policy not in ap.V3_AUTO_PROMOTION_POLICIES:
|
||||||
|
raise CompilerError(
|
||||||
|
f"manifest.source.auto_promotion_policy must be one of {sorted(ap.V3_AUTO_PROMOTION_POLICIES)}"
|
||||||
|
)
|
||||||
|
source_identity = _require_stable_reference(source.get("identity"), "manifest.source.identity")
|
||||||
|
source_locator = _require_stable_reference(source.get("locator"), "manifest.source.locator")
|
||||||
|
expected_source_identity = f"document:sha256:{artifact_sha256}"
|
||||||
|
expected_source_locator = f"artifact://sha256/{artifact_sha256}"
|
||||||
|
if source_identity != expected_source_identity:
|
||||||
|
raise CompilerError(
|
||||||
|
"manifest.source.identity must equal the content-addressed identity for the actual artifact bytes"
|
||||||
|
)
|
||||||
|
if source_locator != expected_source_locator:
|
||||||
|
raise CompilerError("manifest.source.locator must equal artifact://sha256/<actual artifact SHA-256>")
|
||||||
|
validated_source = {
|
||||||
|
"identity": source_identity,
|
||||||
|
"source_key": _require_key(source.get("source_key"), "manifest.source.source_key"),
|
||||||
|
"source_type": source_type,
|
||||||
|
"title": _require_nonempty_string(source.get("title"), "manifest.source.title"),
|
||||||
|
"locator": source_locator,
|
||||||
|
"access_scope": source_access_scope,
|
||||||
|
"ingestion_origin": ingestion_origin,
|
||||||
|
"provenance_status": provenance_status,
|
||||||
|
"source_class": source_class,
|
||||||
|
"auto_promotion_policy": auto_promotion_policy,
|
||||||
|
"captured_at": _require_canonical_utc_timestamp(source.get("captured_at"), "manifest.source.captured_at"),
|
||||||
|
}
|
||||||
|
|
||||||
|
raw_claims = manifest.get("claims")
|
||||||
|
if not isinstance(raw_claims, list) or not raw_claims:
|
||||||
|
raise CompilerError("manifest.claims must be a non-empty list of objects")
|
||||||
|
if len(raw_claims) > ap.MAX_BUNDLE_ROWS["claims"]:
|
||||||
|
raise CompilerError(f"manifest.claims exceeds {ap.MAX_BUNDLE_ROWS['claims']} rows")
|
||||||
|
|
||||||
|
validated_claims: list[dict[str, Any]] = []
|
||||||
|
quote_bindings: list[dict[str, Any]] = []
|
||||||
|
seen_claim_keys: set[str] = set()
|
||||||
|
evidence_count = 0
|
||||||
|
for claim_index, raw_claim in enumerate(raw_claims):
|
||||||
|
label = f"manifest.claims[{claim_index}]"
|
||||||
|
claim = _require_object(raw_claim, label)
|
||||||
|
_require_exact_shape(claim, CLAIM_KEYS_V3, label)
|
||||||
|
claim_key = _require_key(claim.get("claim_key"), f"{label}.claim_key")
|
||||||
|
if claim_key in seen_claim_keys:
|
||||||
|
raise CompilerError(f"duplicate claim_key: {claim_key}")
|
||||||
|
seen_claim_keys.add(claim_key)
|
||||||
|
claim_type = claim.get("type")
|
||||||
|
if claim_type not in ap.V3_CLAIM_TYPES:
|
||||||
|
raise CompilerError(f"{label}.type must be one of {sorted(ap.V3_CLAIM_TYPES)}")
|
||||||
|
access_scope = claim.get("access_scope")
|
||||||
|
if access_scope not in ap.V3_ACCESS_SCOPES:
|
||||||
|
raise CompilerError(f"{label}.access_scope must be one of {sorted(ap.V3_ACCESS_SCOPES)}")
|
||||||
|
owner_agent_id = _require_canonical_uuid(claim.get("owner_agent_id"), f"{label}.owner_agent_id", nullable=True)
|
||||||
|
if access_scope == "agent_private" and owner_agent_id is None:
|
||||||
|
raise CompilerError(f"{label}.owner_agent_id is required for agent_private claims")
|
||||||
|
status = claim.get("status")
|
||||||
|
if status not in {"active", "contested"}:
|
||||||
|
raise CompilerError(f"{label}.status must be active or contested")
|
||||||
|
if claim.get("revision_kind") != "original" or claim.get("supersedes_id") is not None:
|
||||||
|
raise CompilerError(f"{label} must describe an original claim with supersedes_id=null")
|
||||||
|
|
||||||
|
raw_evidence = claim.get("evidence")
|
||||||
|
if not isinstance(raw_evidence, list) or not raw_evidence:
|
||||||
|
raise CompilerError(f"{label}.evidence must be a non-empty list of exact locator-bound rows")
|
||||||
|
evidence_count += len(raw_evidence)
|
||||||
|
if evidence_count > ap.MAX_BUNDLE_ROWS["evidence"]:
|
||||||
|
raise CompilerError(f"manifest evidence exceeds {ap.MAX_BUNDLE_ROWS['evidence']} rows")
|
||||||
|
validated_evidence: list[dict[str, Any]] = []
|
||||||
|
seen_evidence_keys: set[str] = set()
|
||||||
|
seen_polarities: set[str] = set()
|
||||||
|
for evidence_index, raw_row in enumerate(raw_evidence):
|
||||||
|
evidence_label = f"{label}.evidence[{evidence_index}]"
|
||||||
|
row = _require_object(raw_row, evidence_label)
|
||||||
|
_require_exact_shape(row, EVIDENCE_KEYS_V3, evidence_label)
|
||||||
|
evidence_key = _require_key(row.get("evidence_key"), f"{evidence_label}.evidence_key")
|
||||||
|
if evidence_key in seen_evidence_keys:
|
||||||
|
raise CompilerError(f"{evidence_label}.evidence_key duplicates an earlier evidence row")
|
||||||
|
seen_evidence_keys.add(evidence_key)
|
||||||
|
polarity = row.get("polarity")
|
||||||
|
if polarity not in ap.V3_EVIDENCE_POLARITIES:
|
||||||
|
raise CompilerError(f"{evidence_label}.polarity must be one of {sorted(ap.V3_EVIDENCE_POLARITIES)}")
|
||||||
|
if polarity in seen_polarities:
|
||||||
|
raise CompilerError(f"{evidence_label}.polarity duplicates a claim-local source/polarity key")
|
||||||
|
seen_polarities.add(polarity)
|
||||||
|
binding = _v3_quote_binding(text_bytes, row.get("quote"), row.get("locator"), evidence_label)
|
||||||
|
validated_evidence.append(
|
||||||
|
{
|
||||||
|
"evidence_key": evidence_key,
|
||||||
|
"quote": binding["quote"],
|
||||||
|
"polarity": polarity,
|
||||||
|
"locator": {
|
||||||
|
"unit": binding["unit"],
|
||||||
|
"start": binding["start"],
|
||||||
|
"end": binding["end"],
|
||||||
|
},
|
||||||
|
"rationale": _require_nonempty_string(row.get("rationale"), f"{evidence_label}.rationale"),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
quote_bindings.append(
|
||||||
|
{
|
||||||
|
"claim_key": claim_key,
|
||||||
|
"evidence_key": evidence_key,
|
||||||
|
"kind": "evidence",
|
||||||
|
**binding,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
validated_evidence.sort(key=lambda row: row["evidence_key"])
|
||||||
|
|
||||||
|
validated_claims.append(
|
||||||
|
{
|
||||||
|
"claim_key": claim_key,
|
||||||
|
"type": claim_type,
|
||||||
|
"proposition": _require_nonempty_string(claim.get("proposition"), f"{label}.proposition"),
|
||||||
|
"body_markdown": _require_nonempty_string(claim.get("body_markdown"), f"{label}.body_markdown"),
|
||||||
|
"scope": _require_nonempty_string(claim.get("scope"), f"{label}.scope"),
|
||||||
|
"access_scope": access_scope,
|
||||||
|
"owner_agent_id": owner_agent_id,
|
||||||
|
"status": status,
|
||||||
|
"revision_criteria": _require_nonempty_string(
|
||||||
|
claim.get("revision_criteria"), f"{label}.revision_criteria"
|
||||||
|
),
|
||||||
|
"revision_kind": "original",
|
||||||
|
"supersedes_id": None,
|
||||||
|
"tags": _require_string_list(claim.get("tags"), f"{label}.tags"),
|
||||||
|
"evidence": validated_evidence,
|
||||||
|
"assessment": _validate_v3_assessment(
|
||||||
|
claim.get("assessment"), claim_type, validated_evidence, f"{label}.assessment"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
validated_claims.sort(key=lambda row: row["claim_key"])
|
||||||
|
quote_bindings.sort(key=lambda row: (row["claim_key"], row["evidence_key"]))
|
||||||
|
return (
|
||||||
|
{
|
||||||
|
"schema": MANIFEST_SCHEMA_V3,
|
||||||
|
"artifact_sha256": artifact_sha256,
|
||||||
|
"extracted_text_sha256": text_sha256,
|
||||||
|
"extractor": validated_extractor,
|
||||||
|
"agent_id": agent_id,
|
||||||
|
"source": validated_source,
|
||||||
|
"claims": validated_claims,
|
||||||
|
},
|
||||||
|
quote_bindings,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _validate_dedupe(value: Any, extracted_text: str) -> dict[str, Any]:
|
def _validate_dedupe(value: Any, extracted_text: str) -> dict[str, Any]:
|
||||||
dedupe = _require_object(value, "manifest.dedupe")
|
dedupe = _require_object(value, "manifest.dedupe")
|
||||||
_require_exact_keys(dedupe, DEDUPE_KEYS, "manifest.dedupe")
|
_require_exact_keys(dedupe, DEDUPE_KEYS, "manifest.dedupe")
|
||||||
|
|
@ -460,6 +846,408 @@ def build_parent_proposal(
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _v3_identity_material(manifest: dict[str, Any], manifest_file_sha256: str) -> tuple[dict[str, Any], str, str]:
|
||||||
|
candidate_content = {
|
||||||
|
"agent_id": manifest["agent_id"],
|
||||||
|
"source": manifest["source"],
|
||||||
|
"claims": manifest["claims"],
|
||||||
|
}
|
||||||
|
candidate_content_sha256 = canonical_sha256(candidate_content)
|
||||||
|
identity_material = {
|
||||||
|
"artifact_sha256": manifest["artifact_sha256"],
|
||||||
|
"extracted_text_sha256": manifest["extracted_text_sha256"],
|
||||||
|
"extractor": manifest["extractor"],
|
||||||
|
"manifest_file_sha256": manifest_file_sha256,
|
||||||
|
"source_identity": manifest["source"]["identity"],
|
||||||
|
"source_locator": manifest["source"]["locator"],
|
||||||
|
"normalized_candidate_content_sha256": candidate_content_sha256,
|
||||||
|
}
|
||||||
|
return identity_material, candidate_content_sha256, canonical_sha256(identity_material)
|
||||||
|
|
||||||
|
|
||||||
|
def _build_v3_parent_proposal(
|
||||||
|
manifest: dict[str, Any],
|
||||||
|
manifest_canonical_sha256: str,
|
||||||
|
identity_material: dict[str, Any],
|
||||||
|
candidate_content_sha256: str,
|
||||||
|
packet_identity_sha256: str,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
source = manifest["source"]
|
||||||
|
parent_id = stable_uuid_v3("source_proposal", packet_identity_sha256, candidate_content_sha256)
|
||||||
|
return {
|
||||||
|
"id": parent_id,
|
||||||
|
"proposal_type": "approve_claim",
|
||||||
|
"status": "pending_review",
|
||||||
|
"proposed_by_handle": "leo",
|
||||||
|
"proposed_by_agent_id": manifest["agent_id"],
|
||||||
|
"channel": "source_document_compiler_v3",
|
||||||
|
"source_ref": f"source-compiler-v3:{packet_identity_sha256[:32]}",
|
||||||
|
"rationale": "Compile one retained semantic extraction into a hash-bound V3 claim proposal for review.",
|
||||||
|
"payload": {
|
||||||
|
"source_document": {
|
||||||
|
"manifest_schema": MANIFEST_SCHEMA_V3,
|
||||||
|
"manifest_file_sha256": identity_material["manifest_file_sha256"],
|
||||||
|
"manifest_canonical_sha256": manifest_canonical_sha256,
|
||||||
|
"artifact_sha256": manifest["artifact_sha256"],
|
||||||
|
"extracted_text_sha256": manifest["extracted_text_sha256"],
|
||||||
|
"source_identity": source["identity"],
|
||||||
|
"source_locator": source["locator"],
|
||||||
|
"extractor": manifest["extractor"],
|
||||||
|
},
|
||||||
|
"identity_material": identity_material,
|
||||||
|
"normalized_candidate_content_sha256": candidate_content_sha256,
|
||||||
|
"packet_identity_sha256": packet_identity_sha256,
|
||||||
|
"semantic_extraction_is_retained_input": True,
|
||||||
|
"model_training_claim": False,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _v3_source_excerpt(manifest: dict[str, Any]) -> str:
|
||||||
|
excerpts: list[str] = []
|
||||||
|
seen: set[str] = set()
|
||||||
|
for claim in manifest["claims"]:
|
||||||
|
for evidence in claim["evidence"]:
|
||||||
|
quote = evidence["quote"]
|
||||||
|
if quote not in seen:
|
||||||
|
seen.add(quote)
|
||||||
|
excerpts.append(quote)
|
||||||
|
return "\n\n".join(excerpts)
|
||||||
|
|
||||||
|
|
||||||
|
def _v3_locator_json(manifest: dict[str, Any], binding: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"schema": "livingip.extractedTextQuoteLocator.v1",
|
||||||
|
"source_identity": manifest["source"]["identity"],
|
||||||
|
"source_locator": manifest["source"]["locator"],
|
||||||
|
"artifact_sha256": manifest["artifact_sha256"],
|
||||||
|
"extracted_text_sha256": manifest["extracted_text_sha256"],
|
||||||
|
"extractor": manifest["extractor"],
|
||||||
|
"unit": binding["unit"],
|
||||||
|
"start": binding["start"],
|
||||||
|
"end": binding["end"],
|
||||||
|
"quote_sha256": binding["quote_sha256"],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_distinct_v3_ids(values: list[str]) -> None:
|
||||||
|
if len(values) != len(set(values)):
|
||||||
|
raise CompilerError("compiled V3 packet contains duplicate deterministic ids")
|
||||||
|
|
||||||
|
|
||||||
|
def _build_v3_strict_child(
|
||||||
|
manifest: dict[str, Any],
|
||||||
|
manifest_canonical_sha256: str,
|
||||||
|
quote_bindings: list[dict[str, Any]],
|
||||||
|
parent: dict[str, Any],
|
||||||
|
identity_material: dict[str, Any],
|
||||||
|
candidate_content_sha256: str,
|
||||||
|
packet_identity_sha256: str,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
source = manifest["source"]
|
||||||
|
source_id = stable_uuid_v3("source", packet_identity_sha256, source["source_key"])
|
||||||
|
claim_ids = {
|
||||||
|
claim["claim_key"]: stable_uuid_v3("claim", packet_identity_sha256, claim["claim_key"])
|
||||||
|
for claim in manifest["claims"]
|
||||||
|
}
|
||||||
|
binding_by_key = {(binding["claim_key"], binding["evidence_key"]): binding for binding in quote_bindings}
|
||||||
|
canonical_url, storage_uri = _locator_fields(source["locator"])
|
||||||
|
source_row = {
|
||||||
|
"id": source_id,
|
||||||
|
"source_type": source["source_type"],
|
||||||
|
"canonical_url": canonical_url,
|
||||||
|
"storage_uri": storage_uri,
|
||||||
|
"excerpt": _v3_source_excerpt(manifest),
|
||||||
|
"content_hash": manifest["artifact_sha256"],
|
||||||
|
"access_scope": source["access_scope"],
|
||||||
|
"ingestion_origin": source["ingestion_origin"],
|
||||||
|
"provenance_status": source["provenance_status"],
|
||||||
|
"source_class": source["source_class"],
|
||||||
|
"auto_promotion_policy": source["auto_promotion_policy"],
|
||||||
|
"captured_at": source["captured_at"],
|
||||||
|
"created_by": manifest["agent_id"],
|
||||||
|
}
|
||||||
|
|
||||||
|
claim_rows: list[dict[str, Any]] = []
|
||||||
|
evidence_rows: list[dict[str, Any]] = []
|
||||||
|
assessment_rows: list[dict[str, Any]] = []
|
||||||
|
for claim in manifest["claims"]:
|
||||||
|
claim_id = claim_ids[claim["claim_key"]]
|
||||||
|
claim_rows.append(
|
||||||
|
{
|
||||||
|
"id": claim_id,
|
||||||
|
"owner_agent_id": claim["owner_agent_id"],
|
||||||
|
"type": claim["type"],
|
||||||
|
"proposition": claim["proposition"],
|
||||||
|
"body_markdown": claim["body_markdown"],
|
||||||
|
"scope": claim["scope"],
|
||||||
|
"access_scope": claim["access_scope"],
|
||||||
|
"status": claim["status"],
|
||||||
|
"revision_criteria": claim["revision_criteria"],
|
||||||
|
"revision_kind": "original",
|
||||||
|
"supersedes_id": None,
|
||||||
|
"tags": claim["tags"],
|
||||||
|
"created_by": manifest["agent_id"],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
for evidence in claim["evidence"]:
|
||||||
|
binding = binding_by_key[(claim["claim_key"], evidence["evidence_key"])]
|
||||||
|
evidence_rows.append(
|
||||||
|
{
|
||||||
|
"id": stable_uuid_v3(
|
||||||
|
"evidence",
|
||||||
|
packet_identity_sha256,
|
||||||
|
[claim["claim_key"], evidence["evidence_key"]],
|
||||||
|
),
|
||||||
|
"claim_id": claim_id,
|
||||||
|
"source_id": source_id,
|
||||||
|
"polarity": evidence["polarity"],
|
||||||
|
"excerpt": evidence["quote"],
|
||||||
|
"locator_json": _v3_locator_json(manifest, binding),
|
||||||
|
"source_content_hash": manifest["artifact_sha256"],
|
||||||
|
"rationale": evidence["rationale"],
|
||||||
|
"created_by": manifest["agent_id"],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
for claim in manifest["claims"]:
|
||||||
|
claim_id = claim_ids[claim["claim_key"]]
|
||||||
|
assessment = claim["assessment"]
|
||||||
|
evidence_set_hash = ap._v3_evidence_set_hash(claim_id, evidence_rows)
|
||||||
|
assessment_rows.append(
|
||||||
|
{
|
||||||
|
"id": stable_uuid_v3("assessment", packet_identity_sha256, claim["claim_key"]),
|
||||||
|
"claim_id": claim_id,
|
||||||
|
"support_tier": assessment["support_tier"],
|
||||||
|
"challenge_tier": assessment["challenge_tier"],
|
||||||
|
"overall_state": assessment["overall_state"],
|
||||||
|
"rationale": assessment["rationale"],
|
||||||
|
"evidence_set_hash": evidence_set_hash,
|
||||||
|
"supersedes_assessment_id": None,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
parent_packet_sha256 = canonical_sha256(parent)
|
||||||
|
apply_payload = {
|
||||||
|
"contract_version": 3,
|
||||||
|
"source_proposal_id": parent["id"],
|
||||||
|
"agent_id": manifest["agent_id"],
|
||||||
|
"claims": claim_rows,
|
||||||
|
"sources": [source_row],
|
||||||
|
"evidence": evidence_rows,
|
||||||
|
"edges": [],
|
||||||
|
"assessments": assessment_rows,
|
||||||
|
"reasoning_tools": [],
|
||||||
|
"normalization_manifest": {
|
||||||
|
"schema": "livingip.kbSourceProposalNormalization.v3",
|
||||||
|
"semantic_extraction_statement": (
|
||||||
|
"The semantic extraction is a retained input; this is a reproducible learning input, "
|
||||||
|
"not a model-training claim."
|
||||||
|
),
|
||||||
|
"semantic_extraction_is_retained_input": True,
|
||||||
|
"model_training_claim": False,
|
||||||
|
"manifest_schema": MANIFEST_SCHEMA_V3,
|
||||||
|
"manifest_file_sha256": identity_material["manifest_file_sha256"],
|
||||||
|
"manifest_identity_binding": "sha256(exact duplicate-free UTF-8 JSON file bytes)",
|
||||||
|
"manifest_canonical_sha256": manifest_canonical_sha256,
|
||||||
|
"artifact_sha256": manifest["artifact_sha256"],
|
||||||
|
"extracted_text_sha256": manifest["extracted_text_sha256"],
|
||||||
|
"extractor": manifest["extractor"],
|
||||||
|
"source_identity": source["identity"],
|
||||||
|
"source_locator": source["locator"],
|
||||||
|
"normalized_candidate_content_sha256": candidate_content_sha256,
|
||||||
|
"identity_material": identity_material,
|
||||||
|
"packet_identity_sha256": packet_identity_sha256,
|
||||||
|
"parent_packet_sha256": parent_packet_sha256,
|
||||||
|
"quote_bindings": quote_bindings,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
payload = {"apply_payload": apply_payload}
|
||||||
|
payload_sha256 = canonical_sha256(payload)
|
||||||
|
child_id = stable_uuid_v3("pending_review_proposal", packet_identity_sha256, payload_sha256)
|
||||||
|
_ensure_distinct_v3_ids(
|
||||||
|
[
|
||||||
|
parent["id"],
|
||||||
|
child_id,
|
||||||
|
source_id,
|
||||||
|
*[row["id"] for row in claim_rows],
|
||||||
|
*[row["id"] for row in evidence_rows],
|
||||||
|
*[row["id"] for row in assessment_rows],
|
||||||
|
]
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"id": child_id,
|
||||||
|
"proposal_type": "approve_claim",
|
||||||
|
"status": "pending_review",
|
||||||
|
"proposed_by_handle": parent["proposed_by_handle"],
|
||||||
|
"proposed_by_agent_id": manifest["agent_id"],
|
||||||
|
"channel": parent["channel"],
|
||||||
|
"source_ref": f"source-compiler-v3:{packet_identity_sha256[:32]}:{payload_sha256[:16]}",
|
||||||
|
"rationale": f"Strict V3 source proposal compiled from retained packet {parent['id']}.",
|
||||||
|
"payload": payload,
|
||||||
|
"parent_packet_sha256": parent_packet_sha256,
|
||||||
|
"payload_sha256": payload_sha256,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_v3_compiled_child(
|
||||||
|
child: dict[str, Any], manifest: dict[str, Any], quote_bindings: list[dict[str, Any]]
|
||||||
|
) -> None:
|
||||||
|
if child.get("status") != "pending_review" or child.get("proposal_type") != "approve_claim":
|
||||||
|
raise CompilerError("V3 compilation did not produce one pending_review approve_claim proposal")
|
||||||
|
payload = child.get("payload")
|
||||||
|
apply_payload = payload.get("apply_payload") if isinstance(payload, dict) else None
|
||||||
|
if not isinstance(apply_payload, dict) or apply_payload.get("contract_version") != 3:
|
||||||
|
raise CompilerError("strict child has no schema-valid approve_claim V3 apply_payload")
|
||||||
|
if apply_payload.get("reasoning_tools") != [] or apply_payload.get("edges") != []:
|
||||||
|
raise CompilerError("V3 source compilation admits neither reasoning_tools nor inferred edges")
|
||||||
|
if child.get("payload_sha256") != canonical_sha256(payload):
|
||||||
|
raise CompilerError("strict child payload_sha256 does not bind the exact V3 payload")
|
||||||
|
|
||||||
|
claims = apply_payload.get("claims") or []
|
||||||
|
sources = apply_payload.get("sources") or []
|
||||||
|
evidence = apply_payload.get("evidence") or []
|
||||||
|
assessments = apply_payload.get("assessments") or []
|
||||||
|
if len(claims) != len(manifest["claims"]) or len(sources) != 1:
|
||||||
|
raise CompilerError("V3 strict child row counts differ from the extraction manifest")
|
||||||
|
if len(evidence) != sum(len(claim["evidence"]) for claim in manifest["claims"]):
|
||||||
|
raise CompilerError("V3 strict child evidence count differs from the extraction manifest")
|
||||||
|
if len(assessments) != len(claims):
|
||||||
|
raise CompilerError("V3 strict child requires exactly one assessment per claim")
|
||||||
|
_ensure_distinct_v3_ids(
|
||||||
|
[
|
||||||
|
child["id"],
|
||||||
|
*[row.get("id") for row in sources],
|
||||||
|
*[row.get("id") for row in claims],
|
||||||
|
*[row.get("id") for row in evidence],
|
||||||
|
*[row.get("id") for row in assessments],
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
source_row = sources[0]
|
||||||
|
if source_row.get("content_hash") != manifest["artifact_sha256"]:
|
||||||
|
raise CompilerError("V3 source content_hash does not bind the original artifact")
|
||||||
|
expected_url, expected_storage_uri = _locator_fields(manifest["source"]["locator"])
|
||||||
|
if source_row.get("canonical_url") != expected_url or source_row.get("storage_uri") != expected_storage_uri:
|
||||||
|
raise CompilerError("V3 source row does not retain the validated content-addressed locator")
|
||||||
|
if source_row.get("captured_at") != _require_canonical_utc_timestamp(
|
||||||
|
source_row.get("captured_at"), "V3 source captured_at"
|
||||||
|
):
|
||||||
|
raise CompilerError("V3 source captured_at is not in writer-canonical fixed-microsecond UTC Z form")
|
||||||
|
binding_by_locator = {canonical_sha256(_v3_locator_json(manifest, binding)): binding for binding in quote_bindings}
|
||||||
|
for row in evidence:
|
||||||
|
binding = binding_by_locator.get(canonical_sha256(row.get("locator_json")))
|
||||||
|
if binding is None or row.get("excerpt") != binding["quote"]:
|
||||||
|
raise CompilerError("V3 evidence does not retain an exact locator-bound quote")
|
||||||
|
if row.get("source_content_hash") != manifest["artifact_sha256"]:
|
||||||
|
raise CompilerError("V3 evidence source_content_hash does not bind the original artifact")
|
||||||
|
assessment_by_claim = {row.get("claim_id"): row for row in assessments}
|
||||||
|
if len(assessment_by_claim) != len(claims):
|
||||||
|
raise CompilerError("V3 strict child contains duplicate assessment claim ids")
|
||||||
|
for claim in claims:
|
||||||
|
assessment = assessment_by_claim.get(claim.get("id"))
|
||||||
|
if assessment is None:
|
||||||
|
raise CompilerError("V3 strict child is missing a claim-local evidence assessment")
|
||||||
|
if assessment.get("evidence_set_hash") != ap._v3_evidence_set_hash(claim["id"], evidence):
|
||||||
|
raise CompilerError("V3 assessment evidence_set_hash does not match claim-local evidence")
|
||||||
|
|
||||||
|
|
||||||
|
def _compile_v3_source_packet(
|
||||||
|
artifact_bytes: bytes,
|
||||||
|
text_bytes: bytes,
|
||||||
|
manifest_bytes: bytes,
|
||||||
|
extractor_spec_bytes: bytes,
|
||||||
|
manifest_object: dict[str, Any],
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
artifact_sha256 = sha256_bytes(artifact_bytes)
|
||||||
|
text_sha256 = sha256_bytes(text_bytes)
|
||||||
|
extractor_spec_sha256 = sha256_bytes(extractor_spec_bytes)
|
||||||
|
manifest_file_sha256 = sha256_bytes(manifest_bytes)
|
||||||
|
manifest, quote_bindings = _validate_manifest_v3(
|
||||||
|
manifest_object,
|
||||||
|
artifact_sha256,
|
||||||
|
text_sha256,
|
||||||
|
extractor_spec_sha256,
|
||||||
|
text_bytes,
|
||||||
|
)
|
||||||
|
manifest_canonical_sha256 = canonical_sha256(manifest)
|
||||||
|
identity_material, candidate_content_sha256, packet_identity_sha256 = _v3_identity_material(
|
||||||
|
manifest, manifest_file_sha256
|
||||||
|
)
|
||||||
|
parent = _build_v3_parent_proposal(
|
||||||
|
manifest,
|
||||||
|
manifest_canonical_sha256,
|
||||||
|
identity_material,
|
||||||
|
candidate_content_sha256,
|
||||||
|
packet_identity_sha256,
|
||||||
|
)
|
||||||
|
child = _build_v3_strict_child(
|
||||||
|
manifest,
|
||||||
|
manifest_canonical_sha256,
|
||||||
|
quote_bindings,
|
||||||
|
parent,
|
||||||
|
identity_material,
|
||||||
|
candidate_content_sha256,
|
||||||
|
packet_identity_sha256,
|
||||||
|
)
|
||||||
|
_validate_v3_compiled_child(child, manifest, quote_bindings)
|
||||||
|
stage_sql = stage.build_stage_sql(child)
|
||||||
|
stable_bundle = {
|
||||||
|
"schema": BUNDLE_SCHEMA_V3,
|
||||||
|
"status": "pending_review",
|
||||||
|
"build_only": True,
|
||||||
|
"database_write_performed": False,
|
||||||
|
"canonical_apply_performed": False,
|
||||||
|
"inputs": {
|
||||||
|
"artifact_bytes": len(artifact_bytes),
|
||||||
|
"extracted_text_bytes": len(text_bytes),
|
||||||
|
"manifest_bytes": len(manifest_bytes),
|
||||||
|
"extractor_spec_bytes": len(extractor_spec_bytes),
|
||||||
|
},
|
||||||
|
"validated_manifest": manifest,
|
||||||
|
"quote_bindings": quote_bindings,
|
||||||
|
"parent_proposal": parent,
|
||||||
|
"strict_child_proposal": child,
|
||||||
|
"stage_preview": {
|
||||||
|
"target_table": "kb_stage.kb_proposals",
|
||||||
|
"target_status": "pending_review",
|
||||||
|
"executed": False,
|
||||||
|
"stage_sql_sha256": sha256_bytes(stage_sql.encode("utf-8")),
|
||||||
|
},
|
||||||
|
"compiler_contract": {
|
||||||
|
"manifest_contract": MANIFEST_SCHEMA_V3,
|
||||||
|
"output_contract": BUNDLE_SCHEMA_V3,
|
||||||
|
"approve_claim_contract_version": 3,
|
||||||
|
"strict_preflight": "scripts/compile_kb_source_packet.py::_validate_v3_compiled_child",
|
||||||
|
"writer_apply_invoked": False,
|
||||||
|
"writer_timestamp_contract": "fixed-microsecond UTC Z",
|
||||||
|
"manifest_identity_binding": "sha256(exact duplicate-free UTF-8 JSON file bytes)",
|
||||||
|
"extractor_spec_binding": "sha256(exact independently supplied artifact bytes)",
|
||||||
|
"semantic_extraction_is_retained_input": True,
|
||||||
|
"semantic_extraction_statement": (
|
||||||
|
"The semantic extraction is a retained input; this is the smallest reproducible learning input, "
|
||||||
|
"not a model-training claim."
|
||||||
|
),
|
||||||
|
"model_training_claim": False,
|
||||||
|
"source_from_chat_labels": False,
|
||||||
|
"database_stage_executed": False,
|
||||||
|
},
|
||||||
|
"hashes": {
|
||||||
|
"artifact_sha256": artifact_sha256,
|
||||||
|
"extracted_text_sha256": text_sha256,
|
||||||
|
"extractor_spec_sha256": extractor_spec_sha256,
|
||||||
|
"manifest_file_sha256": manifest_file_sha256,
|
||||||
|
"manifest_canonical_sha256": manifest_canonical_sha256,
|
||||||
|
"normalized_candidate_content_sha256": candidate_content_sha256,
|
||||||
|
"packet_identity_sha256": packet_identity_sha256,
|
||||||
|
"parent_packet_sha256": child["parent_packet_sha256"],
|
||||||
|
"strict_child_payload_sha256": child["payload_sha256"],
|
||||||
|
},
|
||||||
|
}
|
||||||
|
stable_bundle["hashes"]["bundle_content_sha256"] = canonical_sha256(stable_bundle)
|
||||||
|
return stable_bundle
|
||||||
|
|
||||||
|
|
||||||
def _validate_compiled_child(
|
def _validate_compiled_child(
|
||||||
child: dict[str, Any], manifest: dict[str, Any], quote_bindings: list[dict[str, Any]]
|
child: dict[str, Any], manifest: dict[str, Any], quote_bindings: list[dict[str, Any]]
|
||||||
) -> None:
|
) -> None:
|
||||||
|
|
@ -503,7 +1291,12 @@ def _validate_compiled_child(
|
||||||
raise CompilerError("one or more exact quotes exceed the current canonical source excerpt capacity")
|
raise CompilerError("one or more exact quotes exceed the current canonical source excerpt capacity")
|
||||||
|
|
||||||
|
|
||||||
def compile_source_packet(artifact_path: Path, text_path: Path, manifest_path: Path) -> dict[str, Any]:
|
def compile_source_packet(
|
||||||
|
artifact_path: Path,
|
||||||
|
text_path: Path,
|
||||||
|
manifest_path: Path,
|
||||||
|
extractor_spec_path: Path | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
artifact_bytes = _read_required_file(artifact_path, "artifact")
|
artifact_bytes = _read_required_file(artifact_path, "artifact")
|
||||||
text_bytes = _read_required_file(text_path, "extracted text")
|
text_bytes = _read_required_file(text_path, "extracted text")
|
||||||
manifest_bytes = _read_required_file(manifest_path, "extraction manifest")
|
manifest_bytes = _read_required_file(manifest_path, "extraction manifest")
|
||||||
|
|
@ -511,14 +1304,24 @@ def compile_source_packet(artifact_path: Path, text_path: Path, manifest_path: P
|
||||||
extracted_text = text_bytes.decode("utf-8", errors="strict")
|
extracted_text = text_bytes.decode("utf-8", errors="strict")
|
||||||
except UnicodeDecodeError as exc:
|
except UnicodeDecodeError as exc:
|
||||||
raise CompilerError(f"extracted text must be valid UTF-8: {exc}") from exc
|
raise CompilerError(f"extracted text must be valid UTF-8: {exc}") from exc
|
||||||
try:
|
manifest_raw = _load_duplicate_free_json(manifest_bytes, "extraction manifest")
|
||||||
manifest_raw = json.loads(manifest_bytes.decode("utf-8", errors="strict"))
|
|
||||||
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
|
||||||
raise CompilerError(f"extraction manifest must be valid UTF-8 JSON: {exc}") from exc
|
|
||||||
manifest_object = _require_object(manifest_raw, "manifest")
|
manifest_object = _require_object(manifest_raw, "manifest")
|
||||||
|
|
||||||
artifact_sha256 = sha256_bytes(artifact_bytes)
|
artifact_sha256 = sha256_bytes(artifact_bytes)
|
||||||
text_sha256 = sha256_bytes(text_bytes)
|
text_sha256 = sha256_bytes(text_bytes)
|
||||||
|
if manifest_object.get("schema") == MANIFEST_SCHEMA_V3:
|
||||||
|
if extractor_spec_path is None:
|
||||||
|
raise CompilerError("V3 requires an independently supplied extractor specification artifact")
|
||||||
|
extractor_spec_bytes = _read_required_file(extractor_spec_path, "extractor specification")
|
||||||
|
return _compile_v3_source_packet(
|
||||||
|
artifact_bytes,
|
||||||
|
text_bytes,
|
||||||
|
manifest_bytes,
|
||||||
|
extractor_spec_bytes,
|
||||||
|
manifest_object,
|
||||||
|
)
|
||||||
|
if extractor_spec_path is not None:
|
||||||
|
raise CompilerError("extractor specification input is only valid for a V3 manifest")
|
||||||
manifest, quote_bindings = _validate_manifest(manifest_object, artifact_sha256, text_sha256, extracted_text)
|
manifest, quote_bindings = _validate_manifest(manifest_object, artifact_sha256, text_sha256, extracted_text)
|
||||||
manifest_canonical_sha256 = canonical_sha256(manifest)
|
manifest_canonical_sha256 = canonical_sha256(manifest)
|
||||||
parent = build_parent_proposal(manifest, manifest_canonical_sha256, quote_bindings)
|
parent = build_parent_proposal(manifest, manifest_canonical_sha256, quote_bindings)
|
||||||
|
|
@ -573,6 +1376,11 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||||
parser.add_argument("--artifact", required=True, type=Path, help="raw source artifact; any binary format")
|
parser.add_argument("--artifact", required=True, type=Path, help="raw source artifact; any binary format")
|
||||||
parser.add_argument("--text", required=True, type=Path, help="strict UTF-8 extraction of the artifact")
|
parser.add_argument("--text", required=True, type=Path, help="strict UTF-8 extraction of the artifact")
|
||||||
parser.add_argument("--manifest", required=True, type=Path, help="extraction manifest JSON")
|
parser.add_argument("--manifest", required=True, type=Path, help="extraction manifest JSON")
|
||||||
|
parser.add_argument(
|
||||||
|
"--extractor-spec",
|
||||||
|
type=Path,
|
||||||
|
help="exact extractor specification artifact; required for V3 and rejected for V1/V2",
|
||||||
|
)
|
||||||
parser.add_argument("--output", required=True, type=Path, help="private path for the compiled proposal bundle")
|
parser.add_argument("--output", required=True, type=Path, help="private path for the compiled proposal bundle")
|
||||||
return parser.parse_args(argv)
|
return parser.parse_args(argv)
|
||||||
|
|
||||||
|
|
@ -612,9 +1420,11 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
try:
|
try:
|
||||||
output_path = args.output.resolve()
|
output_path = args.output.resolve()
|
||||||
input_paths = {args.artifact.resolve(), args.text.resolve(), args.manifest.resolve()}
|
input_paths = {args.artifact.resolve(), args.text.resolve(), args.manifest.resolve()}
|
||||||
|
if args.extractor_spec is not None:
|
||||||
|
input_paths.add(args.extractor_spec.resolve())
|
||||||
if output_path in input_paths:
|
if output_path in input_paths:
|
||||||
raise CompilerError("--output must not overwrite an input file")
|
raise CompilerError("--output must not overwrite an input file")
|
||||||
bundle = compile_source_packet(args.artifact, args.text, args.manifest)
|
bundle = compile_source_packet(args.artifact, args.text, args.manifest, args.extractor_spec)
|
||||||
except (CompilerError, OSError, ValueError):
|
except (CompilerError, OSError, ValueError):
|
||||||
_print_status("rejected", reason="input_validation_failed")
|
_print_status("rejected", reason="input_validation_failed")
|
||||||
return 2
|
return 2
|
||||||
|
|
|
||||||
507
scripts/export_kb_transition_replay_material.py
Normal file
507
scripts/export_kb_transition_replay_material.py
Normal file
|
|
@ -0,0 +1,507 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Export exact private replay material for one guarded applied proposal.
|
||||||
|
|
||||||
|
The exporter is intentionally read-only. It validates an existing worker replay
|
||||||
|
receipt, reads the approved and applied proposal rows through one gate-owned
|
||||||
|
function, and atomically publishes deterministic genesis-ledger input.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import stat
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import uuid
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
HERE = Path(__file__).resolve().parent
|
||||||
|
sys.path.insert(0, str(HERE))
|
||||||
|
|
||||||
|
import apply_proposal as apply_engine # noqa: E402
|
||||||
|
import kb_apply_replay_receipt as replay_receipt # noqa: E402
|
||||||
|
|
||||||
|
MATERIAL_ARTIFACT = "teleo_strict_proposal_replay_material"
|
||||||
|
MATERIAL_CONTRACT_VERSION = 1
|
||||||
|
MATERIAL_FIELDS = frozenset(
|
||||||
|
{
|
||||||
|
"artifact",
|
||||||
|
"contract_version",
|
||||||
|
"sequence",
|
||||||
|
"approved_proposal",
|
||||||
|
"approval_snapshot",
|
||||||
|
"applied_proposal",
|
||||||
|
"replay_receipt",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
PROPOSAL_FIELDS = frozenset(
|
||||||
|
{
|
||||||
|
"id",
|
||||||
|
"proposal_type",
|
||||||
|
"status",
|
||||||
|
"proposed_by_handle",
|
||||||
|
"proposed_by_agent_id",
|
||||||
|
"channel",
|
||||||
|
"source_ref",
|
||||||
|
"rationale",
|
||||||
|
"payload",
|
||||||
|
"reviewed_by_handle",
|
||||||
|
"reviewed_by_agent_id",
|
||||||
|
"reviewed_at",
|
||||||
|
"review_note",
|
||||||
|
"applied_by_handle",
|
||||||
|
"applied_by_agent_id",
|
||||||
|
"applied_at",
|
||||||
|
"created_at",
|
||||||
|
"updated_at",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
APPROVAL_FIELDS = frozenset(
|
||||||
|
{
|
||||||
|
"proposal_id",
|
||||||
|
"proposal_type",
|
||||||
|
"payload",
|
||||||
|
"reviewed_by_handle",
|
||||||
|
"reviewed_by_agent_id",
|
||||||
|
"reviewed_by_db_role",
|
||||||
|
"reviewed_at",
|
||||||
|
"review_note",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
PROPOSAL_TRANSITION_FIELDS = frozenset(
|
||||||
|
{"status", "applied_by_handle", "applied_by_agent_id", "applied_at", "updated_at"}
|
||||||
|
)
|
||||||
|
PROPOSAL_UUID_FIELDS = frozenset({"id", "proposed_by_agent_id", "reviewed_by_agent_id", "applied_by_agent_id"})
|
||||||
|
PROPOSAL_TIMESTAMP_FIELDS = frozenset({"reviewed_at", "applied_at", "created_at", "updated_at"})
|
||||||
|
APPROVAL_UUID_FIELDS = frozenset({"proposal_id", "reviewed_by_agent_id"})
|
||||||
|
APPROVAL_TIMESTAMP_FIELDS = frozenset({"reviewed_at"})
|
||||||
|
RECEIPT_PROPOSAL_FIELDS = (
|
||||||
|
"id",
|
||||||
|
"proposal_type",
|
||||||
|
"status",
|
||||||
|
"payload",
|
||||||
|
"reviewed_by_handle",
|
||||||
|
"reviewed_by_agent_id",
|
||||||
|
"reviewed_at",
|
||||||
|
"review_note",
|
||||||
|
"applied_by_handle",
|
||||||
|
"applied_by_agent_id",
|
||||||
|
"applied_at",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ExportError(RuntimeError):
|
||||||
|
"""A fail-closed error that contains no proposal payload material."""
|
||||||
|
|
||||||
|
|
||||||
|
def _canonical_json(value: Any) -> str:
|
||||||
|
return json.dumps(value, ensure_ascii=False, separators=(",", ":"), sort_keys=True)
|
||||||
|
|
||||||
|
|
||||||
|
def _same_json(left: Any, right: Any) -> bool:
|
||||||
|
return _canonical_json(left) == _canonical_json(right)
|
||||||
|
|
||||||
|
|
||||||
|
def _reject_duplicate_keys(pairs: list[tuple[str, Any]]) -> dict[str, Any]:
|
||||||
|
result: dict[str, Any] = {}
|
||||||
|
for key, value in pairs:
|
||||||
|
if key in result:
|
||||||
|
raise ExportError("JSON input contains duplicate object fields")
|
||||||
|
result[key] = value
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _reject_nonfinite_number(value: str) -> None:
|
||||||
|
raise ExportError(f"JSON input contains invalid numeric constant {value}")
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_json_object(raw: str, *, label: str) -> dict[str, Any]:
|
||||||
|
try:
|
||||||
|
value = json.loads(
|
||||||
|
raw,
|
||||||
|
object_pairs_hook=_reject_duplicate_keys,
|
||||||
|
parse_constant=_reject_nonfinite_number,
|
||||||
|
)
|
||||||
|
except (json.JSONDecodeError, UnicodeError) as exc:
|
||||||
|
raise ExportError(f"{label} is not valid UTF-8 JSON") from exc
|
||||||
|
if not isinstance(value, dict):
|
||||||
|
raise ExportError(f"{label} must contain one JSON object")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _absolute_path(value: str | Path) -> Path:
|
||||||
|
path = Path(value).expanduser()
|
||||||
|
if not path.is_absolute():
|
||||||
|
path = Path.cwd() / path
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def _load_private_receipt(path: Path) -> dict[str, Any]:
|
||||||
|
path = _absolute_path(path)
|
||||||
|
try:
|
||||||
|
metadata = path.lstat()
|
||||||
|
except OSError as exc:
|
||||||
|
raise ExportError("worker replay receipt is unavailable") from exc
|
||||||
|
if path.is_symlink() or not stat.S_ISREG(metadata.st_mode):
|
||||||
|
raise ExportError("worker replay receipt must be a regular non-symlink file")
|
||||||
|
if stat.S_IMODE(metadata.st_mode) != 0o600:
|
||||||
|
raise ExportError("worker replay receipt permissions must be exactly 0600")
|
||||||
|
try:
|
||||||
|
raw = path.read_text(encoding="utf-8")
|
||||||
|
except (OSError, UnicodeError) as exc:
|
||||||
|
raise ExportError("worker replay receipt is unreadable") from exc
|
||||||
|
return _parse_json_object(raw, label="worker replay receipt")
|
||||||
|
|
||||||
|
|
||||||
|
def _require_exact_fields(value: Any, expected: frozenset[str], *, label: str) -> dict[str, Any]:
|
||||||
|
if not isinstance(value, dict):
|
||||||
|
raise ExportError(f"{label} must be an object")
|
||||||
|
observed = frozenset(value)
|
||||||
|
if observed != expected:
|
||||||
|
raise ExportError(
|
||||||
|
f"{label} has invalid fields; missing_count={len(expected - observed)}, "
|
||||||
|
f"extra_count={len(observed - expected)}"
|
||||||
|
)
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _canonical_uuid(value: Any, *, label: str, nullable: bool = False) -> str | None:
|
||||||
|
if value is None and nullable:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return str(uuid.UUID(str(value)))
|
||||||
|
except (AttributeError, TypeError, ValueError) as exc:
|
||||||
|
raise ExportError(f"{label} must be a UUID") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _canonical_timestamp(value: Any, *, label: str, nullable: bool = False) -> str | None:
|
||||||
|
if value is None and nullable:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return replay_receipt.canonical_utc_timestamp(value, path=label)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise ExportError(f"{label} must be a timezone-aware timestamp") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_proposal(value: Any, *, label: str) -> dict[str, Any]:
|
||||||
|
row = dict(_require_exact_fields(value, PROPOSAL_FIELDS, label=label))
|
||||||
|
for field in PROPOSAL_UUID_FIELDS:
|
||||||
|
row[field] = _canonical_uuid(
|
||||||
|
row[field],
|
||||||
|
label=f"{label}.{field}",
|
||||||
|
nullable=field != "id",
|
||||||
|
)
|
||||||
|
for field in PROPOSAL_TIMESTAMP_FIELDS:
|
||||||
|
row[field] = _canonical_timestamp(
|
||||||
|
row[field],
|
||||||
|
label=f"{label}.{field}",
|
||||||
|
nullable=field in {"reviewed_at", "applied_at"},
|
||||||
|
)
|
||||||
|
if not isinstance(row["proposal_type"], str) or not row["proposal_type"]:
|
||||||
|
raise ExportError(f"{label}.proposal_type must be a non-empty string")
|
||||||
|
if not isinstance(row["status"], str) or not row["status"]:
|
||||||
|
raise ExportError(f"{label}.status must be a non-empty string")
|
||||||
|
if not isinstance(row["payload"], dict):
|
||||||
|
raise ExportError(f"{label}.payload must be an object")
|
||||||
|
for field in (
|
||||||
|
PROPOSAL_FIELDS
|
||||||
|
- PROPOSAL_UUID_FIELDS
|
||||||
|
- PROPOSAL_TIMESTAMP_FIELDS
|
||||||
|
- {
|
||||||
|
"payload",
|
||||||
|
}
|
||||||
|
):
|
||||||
|
if field in {"proposal_type", "status"}:
|
||||||
|
continue
|
||||||
|
if row[field] is not None and not isinstance(row[field], str):
|
||||||
|
raise ExportError(f"{label}.{field} must be text or null")
|
||||||
|
return row
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_approval(value: Any, *, label: str) -> dict[str, Any]:
|
||||||
|
row = dict(_require_exact_fields(value, APPROVAL_FIELDS, label=label))
|
||||||
|
for field in APPROVAL_UUID_FIELDS:
|
||||||
|
row[field] = _canonical_uuid(row[field], label=f"{label}.{field}")
|
||||||
|
for field in APPROVAL_TIMESTAMP_FIELDS:
|
||||||
|
row[field] = _canonical_timestamp(row[field], label=f"{label}.{field}")
|
||||||
|
if not isinstance(row["payload"], dict):
|
||||||
|
raise ExportError(f"{label}.payload must be an object")
|
||||||
|
for field in APPROVAL_FIELDS - APPROVAL_UUID_FIELDS - APPROVAL_TIMESTAMP_FIELDS - {"payload"}:
|
||||||
|
if not isinstance(row[field], str) or not row[field].strip():
|
||||||
|
raise ExportError(f"{label}.{field} must be non-empty text")
|
||||||
|
return row
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_timestamp(value: str) -> datetime:
|
||||||
|
return datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||||
|
|
||||||
|
|
||||||
|
def validate_material(
|
||||||
|
material: dict[str, Any],
|
||||||
|
*,
|
||||||
|
expected_proposal_id: str | None = None,
|
||||||
|
expected_sequence: int | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Validate and canonicalize the exact seven-field replay material."""
|
||||||
|
_require_exact_fields(material, MATERIAL_FIELDS, label="replay material")
|
||||||
|
if material["artifact"] != MATERIAL_ARTIFACT:
|
||||||
|
raise ExportError("replay material artifact is unsupported")
|
||||||
|
if type(material["contract_version"]) is not int or material["contract_version"] != MATERIAL_CONTRACT_VERSION:
|
||||||
|
raise ExportError("replay material contract version is unsupported")
|
||||||
|
sequence = material["sequence"]
|
||||||
|
if type(sequence) is not int or sequence <= 0:
|
||||||
|
raise ExportError("replay material sequence must be a positive integer")
|
||||||
|
if expected_sequence is not None and sequence != expected_sequence:
|
||||||
|
raise ExportError("replay material sequence does not match the requested sequence")
|
||||||
|
|
||||||
|
approved = _normalize_proposal(material["approved_proposal"], label="approved_proposal")
|
||||||
|
approval = _normalize_approval(material["approval_snapshot"], label="approval_snapshot")
|
||||||
|
applied = _normalize_proposal(material["applied_proposal"], label="applied_proposal")
|
||||||
|
proposal_id = _canonical_uuid(applied["id"], label="applied_proposal.id")
|
||||||
|
if expected_proposal_id is not None and proposal_id != _canonical_uuid(
|
||||||
|
expected_proposal_id, label="expected_proposal_id"
|
||||||
|
):
|
||||||
|
raise ExportError("database proposal id does not match the requested proposal")
|
||||||
|
if approved["id"] != proposal_id or approval["proposal_id"] != proposal_id:
|
||||||
|
raise ExportError("approved, approval, and applied proposal ids do not match")
|
||||||
|
if approved["status"] != "approved" or applied["status"] != "applied":
|
||||||
|
raise ExportError("replay material requires exact approved and applied proposal states")
|
||||||
|
if any(approved[field] is not None for field in ("applied_by_handle", "applied_by_agent_id", "applied_at")):
|
||||||
|
raise ExportError("approved proposal snapshot already contains apply metadata")
|
||||||
|
if not applied["applied_by_handle"] or not applied["applied_by_agent_id"] or not applied["applied_at"]:
|
||||||
|
raise ExportError("applied proposal row lacks exact apply metadata")
|
||||||
|
if not approved["reviewed_by_handle"] or not approved["reviewed_by_agent_id"] or not approved["reviewed_at"]:
|
||||||
|
raise ExportError("approved proposal row lacks exact review metadata")
|
||||||
|
|
||||||
|
changed_fields = {field for field in PROPOSAL_FIELDS if not _same_json(approved[field], applied[field])}
|
||||||
|
if changed_fields != PROPOSAL_TRANSITION_FIELDS:
|
||||||
|
raise ExportError("proposal rows differ outside or omit the exact guarded apply transition")
|
||||||
|
|
||||||
|
approval_projection = {
|
||||||
|
"proposal_id": approved["id"],
|
||||||
|
"proposal_type": approved["proposal_type"],
|
||||||
|
"payload": approved["payload"],
|
||||||
|
"reviewed_by_handle": approved["reviewed_by_handle"],
|
||||||
|
"reviewed_by_agent_id": approved["reviewed_by_agent_id"],
|
||||||
|
"reviewed_at": approved["reviewed_at"],
|
||||||
|
"review_note": approved["review_note"],
|
||||||
|
}
|
||||||
|
for field, expected in approval_projection.items():
|
||||||
|
if not _same_json(approval[field], expected):
|
||||||
|
raise ExportError(f"approval snapshot does not match approved proposal field {field}")
|
||||||
|
|
||||||
|
receipt = material["replay_receipt"]
|
||||||
|
if not isinstance(receipt, dict):
|
||||||
|
raise ExportError("replay_receipt must be an object")
|
||||||
|
try:
|
||||||
|
replay_receipt.validate_replay_receipt(receipt, expected_proposal_id=proposal_id)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise ExportError("worker replay receipt failed its exact row-level contract") from exc
|
||||||
|
receipt_proposal = receipt.get("proposal")
|
||||||
|
if not isinstance(receipt_proposal, dict):
|
||||||
|
raise ExportError("worker replay receipt has no proposal envelope")
|
||||||
|
applied_envelope = {field: applied[field] for field in RECEIPT_PROPOSAL_FIELDS}
|
||||||
|
if not _same_json(receipt_proposal, applied_envelope):
|
||||||
|
raise ExportError("worker replay receipt does not match the exact applied proposal row")
|
||||||
|
|
||||||
|
created_at = _parse_timestamp(approved["created_at"])
|
||||||
|
reviewed_at = _parse_timestamp(approved["reviewed_at"])
|
||||||
|
approved_updated_at = _parse_timestamp(approved["updated_at"])
|
||||||
|
applied_at = _parse_timestamp(applied["applied_at"])
|
||||||
|
applied_updated_at = _parse_timestamp(applied["updated_at"])
|
||||||
|
if not created_at <= reviewed_at <= approved_updated_at < applied_at <= applied_updated_at:
|
||||||
|
raise ExportError("proposal transition timestamps are not in exact lifecycle order")
|
||||||
|
|
||||||
|
return {
|
||||||
|
"artifact": MATERIAL_ARTIFACT,
|
||||||
|
"contract_version": MATERIAL_CONTRACT_VERSION,
|
||||||
|
"sequence": sequence,
|
||||||
|
"approved_proposal": approved,
|
||||||
|
"approval_snapshot": approval,
|
||||||
|
"applied_proposal": applied,
|
||||||
|
"replay_receipt": receipt,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_transition_rows(args: argparse.Namespace, password: str, proposal_id: str) -> dict[str, Any]:
|
||||||
|
sql = f"""begin transaction isolation level repeatable read read only;
|
||||||
|
select kb_stage.export_applied_proposal_replay_rows(
|
||||||
|
{apply_engine.sql_literal(proposal_id)}::uuid
|
||||||
|
)::text;
|
||||||
|
rollback;
|
||||||
|
"""
|
||||||
|
raw = apply_engine.run_psql(args, sql, password, redact_output_on_error=True).strip()
|
||||||
|
if not raw or "\n" in raw:
|
||||||
|
raise ExportError("applied transition export returned an invalid row count")
|
||||||
|
return _parse_json_object(raw, label="applied transition export")
|
||||||
|
|
||||||
|
|
||||||
|
def _resolved_output(path: Path) -> Path:
|
||||||
|
absolute = _absolute_path(path)
|
||||||
|
if absolute.name in {"", ".", ".."}:
|
||||||
|
raise ExportError("output path must name a file")
|
||||||
|
return absolute.parent.resolve(strict=False) / absolute.name
|
||||||
|
|
||||||
|
|
||||||
|
def _paths_alias(left: Path, right: Path) -> bool:
|
||||||
|
if left.resolve(strict=False) == right.resolve(strict=False):
|
||||||
|
return True
|
||||||
|
try:
|
||||||
|
return left.exists() and right.exists() and os.path.samefile(left, right)
|
||||||
|
except OSError:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_output_safe(path: Path, protected_paths: tuple[Path, ...]) -> None:
|
||||||
|
for protected in protected_paths:
|
||||||
|
if _paths_alias(path, protected):
|
||||||
|
raise ExportError("output path aliases a protected input or engine file")
|
||||||
|
try:
|
||||||
|
metadata = path.lstat()
|
||||||
|
except FileNotFoundError:
|
||||||
|
return
|
||||||
|
if path.is_symlink() or not stat.S_ISREG(metadata.st_mode):
|
||||||
|
raise ExportError("output path must be a regular non-symlink file")
|
||||||
|
if metadata.st_nlink != 1:
|
||||||
|
raise ExportError("output path has unsafe hard-link aliases")
|
||||||
|
if stat.S_IMODE(metadata.st_mode) != 0o600:
|
||||||
|
raise ExportError("existing output permissions must be exactly 0600")
|
||||||
|
|
||||||
|
|
||||||
|
def _sync_directory(path: Path) -> None:
|
||||||
|
directory_fd = os.open(path, os.O_RDONLY)
|
||||||
|
try:
|
||||||
|
os.fsync(directory_fd)
|
||||||
|
finally:
|
||||||
|
os.close(directory_fd)
|
||||||
|
|
||||||
|
|
||||||
|
def write_private_material(
|
||||||
|
path: Path,
|
||||||
|
material: dict[str, Any],
|
||||||
|
*,
|
||||||
|
protected_paths: tuple[Path, ...],
|
||||||
|
) -> tuple[Path, str]:
|
||||||
|
"""Publish once, or accept an exact 0600 rerun without overwriting drift."""
|
||||||
|
output = _resolved_output(path)
|
||||||
|
output.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||||
|
_assert_output_safe(output, protected_paths)
|
||||||
|
payload = (json.dumps(material, ensure_ascii=False, indent=2, sort_keys=True) + "\n").encode("utf-8")
|
||||||
|
if output.exists():
|
||||||
|
if output.read_bytes() != payload:
|
||||||
|
raise ExportError("existing output contains different replay material")
|
||||||
|
return output, "unchanged"
|
||||||
|
|
||||||
|
fd, temporary_name = tempfile.mkstemp(prefix=f".{output.name}.", dir=str(output.parent))
|
||||||
|
temporary = Path(temporary_name)
|
||||||
|
published = False
|
||||||
|
try:
|
||||||
|
os.fchmod(fd, 0o600)
|
||||||
|
with os.fdopen(fd, "wb") as handle:
|
||||||
|
handle.write(payload)
|
||||||
|
handle.flush()
|
||||||
|
os.fsync(handle.fileno())
|
||||||
|
try:
|
||||||
|
os.link(temporary, output)
|
||||||
|
published = True
|
||||||
|
except FileExistsError:
|
||||||
|
_assert_output_safe(output, protected_paths)
|
||||||
|
if output.read_bytes() != payload:
|
||||||
|
raise ExportError("output changed concurrently during replay material export") from None
|
||||||
|
_sync_directory(output.parent)
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
temporary.unlink()
|
||||||
|
except FileNotFoundError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
_assert_output_safe(output, protected_paths)
|
||||||
|
if output.read_bytes() != payload:
|
||||||
|
raise ExportError("published replay material failed exact readback")
|
||||||
|
return output, "created" if published else "unchanged"
|
||||||
|
|
||||||
|
|
||||||
|
def _positive_sequence(value: str) -> int:
|
||||||
|
try:
|
||||||
|
sequence = int(value)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise argparse.ArgumentTypeError("sequence must be a positive integer") from exc
|
||||||
|
if sequence <= 0:
|
||||||
|
raise argparse.ArgumentTypeError("sequence must be a positive integer")
|
||||||
|
return sequence
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args(argv: list[str]) -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("proposal_id", help="UUID of the applied proposal")
|
||||||
|
parser.add_argument("--replay-receipt", required=True, help="private 0600 worker replay receipt")
|
||||||
|
parser.add_argument("--sequence", required=True, type=_positive_sequence, help="positive ledger sequence")
|
||||||
|
parser.add_argument("--output", required=True, help="private replay material output path")
|
||||||
|
parser.add_argument("--secrets-file", default=apply_engine.DEFAULT_SECRETS_FILE)
|
||||||
|
parser.add_argument("--container", default=apply_engine.DEFAULT_CONTAINER)
|
||||||
|
parser.add_argument("--db", default=apply_engine.DEFAULT_DB)
|
||||||
|
parser.add_argument("--host", default=apply_engine.DEFAULT_HOST)
|
||||||
|
parser.add_argument("--role", default=apply_engine.DEFAULT_ROLE)
|
||||||
|
return parser.parse_args(argv)
|
||||||
|
|
||||||
|
|
||||||
|
def run(args: argparse.Namespace) -> int:
|
||||||
|
proposal_id = _canonical_uuid(args.proposal_id, label="proposal_id")
|
||||||
|
receipt_path = _absolute_path(args.replay_receipt)
|
||||||
|
receipt = _load_private_receipt(receipt_path)
|
||||||
|
try:
|
||||||
|
replay_receipt.validate_replay_receipt(receipt, expected_proposal_id=proposal_id)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise ExportError("worker replay receipt failed its exact row-level contract") from exc
|
||||||
|
|
||||||
|
password = apply_engine.load_password(args.secrets_file)
|
||||||
|
rows = fetch_transition_rows(args, password, proposal_id)
|
||||||
|
material = validate_material(
|
||||||
|
{
|
||||||
|
"artifact": MATERIAL_ARTIFACT,
|
||||||
|
"contract_version": MATERIAL_CONTRACT_VERSION,
|
||||||
|
"sequence": args.sequence,
|
||||||
|
"approved_proposal": rows.get("approved_proposal"),
|
||||||
|
"approval_snapshot": rows.get("approval_snapshot"),
|
||||||
|
"applied_proposal": rows.get("applied_proposal"),
|
||||||
|
"replay_receipt": receipt,
|
||||||
|
},
|
||||||
|
expected_proposal_id=proposal_id,
|
||||||
|
expected_sequence=args.sequence,
|
||||||
|
)
|
||||||
|
protected_paths = tuple(
|
||||||
|
_absolute_path(path)
|
||||||
|
for path in (
|
||||||
|
receipt_path,
|
||||||
|
args.secrets_file,
|
||||||
|
Path(__file__),
|
||||||
|
Path(apply_engine.__file__),
|
||||||
|
Path(replay_receipt.__file__),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
_, disposition = write_private_material(
|
||||||
|
Path(args.output),
|
||||||
|
material,
|
||||||
|
protected_paths=protected_paths,
|
||||||
|
)
|
||||||
|
print(f"replay material {disposition}: proposal={proposal_id} sequence={args.sequence}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
try:
|
||||||
|
return run(parse_args(sys.argv[1:] if argv is None else argv))
|
||||||
|
except ExportError as exc:
|
||||||
|
print(f"replay material export refused: {exc}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
except OSError:
|
||||||
|
print("replay material export refused: private file operation failed", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
|
|
@ -10,6 +10,12 @@
|
||||||
--
|
--
|
||||||
-- The deployer provisions passwords separately in kb-review.env/kb-apply.env.
|
-- The deployer provisions passwords separately in kb-review.env/kb-apply.env.
|
||||||
-- This file contains no credentials and is safe to reuse in disposable clones.
|
-- This file contains no credentials and is safe to reuse in disposable clones.
|
||||||
|
-- Before the first V3 contract migration, run this file, stop the V2 worker,
|
||||||
|
-- then ALTER ROLE kb_apply NOLOGIN and end its sessions. After V3 is installed,
|
||||||
|
-- this file may re-enable the login: a V2-configured worker detects the cutover
|
||||||
|
-- and leaves historical V2 approvals review-visible until migrate/reject.
|
||||||
|
|
||||||
|
set timezone = 'UTC';
|
||||||
|
|
||||||
begin;
|
begin;
|
||||||
|
|
||||||
|
|
@ -74,10 +80,54 @@ insert into public.agents (id, handle, kind)
|
||||||
values ('44444444-4444-4444-4444-444444444444', 'kb-apply', 'service')
|
values ('44444444-4444-4444-4444-444444444444', 'kb-apply', 'service')
|
||||||
on conflict (handle) do nothing;
|
on conflict (handle) do nothing;
|
||||||
|
|
||||||
|
-- Canonicalize the existing human reviewer identity without changing its UUID.
|
||||||
|
-- Historical proposal/approval reviewer text is deliberately left untouched.
|
||||||
|
do $reviewer_identity$
|
||||||
|
declare
|
||||||
|
v_legacy_id uuid;
|
||||||
|
v_canonical_id uuid;
|
||||||
|
begin
|
||||||
|
select id into v_legacy_id
|
||||||
|
from public.agents
|
||||||
|
where handle = 'm3ta'
|
||||||
|
for update;
|
||||||
|
|
||||||
|
select id into v_canonical_id
|
||||||
|
from public.agents
|
||||||
|
where handle = 'm3taversal'
|
||||||
|
for update;
|
||||||
|
|
||||||
|
if v_legacy_id is not null
|
||||||
|
and v_canonical_id is not null
|
||||||
|
and v_legacy_id <> v_canonical_id then
|
||||||
|
raise exception
|
||||||
|
'kb_apply_prereqs: both reviewer handles m3ta (%) and m3taversal (%) exist with different agent UUIDs; resolve the identity conflict before rerunning',
|
||||||
|
v_legacy_id, v_canonical_id;
|
||||||
|
end if;
|
||||||
|
|
||||||
|
if v_canonical_id is null and v_legacy_id is not null then
|
||||||
|
update public.agents
|
||||||
|
set handle = 'm3taversal'
|
||||||
|
where id = v_legacy_id
|
||||||
|
and handle = 'm3ta';
|
||||||
|
if not found then
|
||||||
|
raise exception
|
||||||
|
'kb_apply_prereqs: legacy reviewer m3ta changed during canonical identity migration';
|
||||||
|
end if;
|
||||||
|
v_canonical_id := v_legacy_id;
|
||||||
|
end if;
|
||||||
|
|
||||||
|
if v_canonical_id is null then
|
||||||
|
raise exception
|
||||||
|
'kb_apply_prereqs: required human reviewer agent is missing; restore exactly one public.agents row with handle m3ta or m3taversal before rerunning';
|
||||||
|
end if;
|
||||||
|
end
|
||||||
|
$reviewer_identity$;
|
||||||
|
|
||||||
-- DB-login to human-reviewer mapping. Only the gate owner/admin can change it;
|
-- DB-login to human-reviewer mapping. Only the gate owner/admin can change it;
|
||||||
-- the approval function verifies --reviewed-by against this mapping rather than
|
-- the approval function verifies --reviewed-by against this mapping rather than
|
||||||
-- trusting caller-supplied identity. The current dedicated reviewer credential
|
-- trusting caller-supplied identity. The dedicated reviewer credential always
|
||||||
-- maps to the existing canonical m3ta human agent when that row is present.
|
-- maps to the canonical m3taversal human agent.
|
||||||
create table if not exists kb_stage.kb_review_principals (
|
create table if not exists kb_stage.kb_review_principals (
|
||||||
db_role name primary key,
|
db_role name primary key,
|
||||||
reviewed_by_handle text not null,
|
reviewed_by_handle text not null,
|
||||||
|
|
@ -96,7 +146,8 @@ create table if not exists kb_stage.kb_proposal_approvals (
|
||||||
reviewed_by_agent_id uuid not null references public.agents(id),
|
reviewed_by_agent_id uuid not null references public.agents(id),
|
||||||
reviewed_by_db_role name not null,
|
reviewed_by_db_role name not null,
|
||||||
reviewed_at timestamptz not null,
|
reviewed_at timestamptz not null,
|
||||||
review_note text not null
|
review_note text not null,
|
||||||
|
approved_proposal_snapshot jsonb
|
||||||
);
|
);
|
||||||
|
|
||||||
-- Existing canonical/proposal tables must remain owned by the deployer. Gate
|
-- Existing canonical/proposal tables must remain owned by the deployer. Gate
|
||||||
|
|
@ -119,22 +170,28 @@ begin
|
||||||
('public', 'strategies'),
|
('public', 'strategies'),
|
||||||
('public', 'strategy_nodes'),
|
('public', 'strategy_nodes'),
|
||||||
('public', 'claim_evidence'),
|
('public', 'claim_evidence'),
|
||||||
|
('public', 'claim_evidence_assessments'),
|
||||||
('public', 'claim_edges'),
|
('public', 'claim_edges'),
|
||||||
('public', 'claims'),
|
('public', 'claims'),
|
||||||
('public', 'sources'),
|
('public', 'sources'),
|
||||||
('public', 'reasoning_tools'),
|
('public', 'reasoning_tools'),
|
||||||
('kb_stage', 'kb_proposals'),
|
('kb_stage', 'kb_proposals'),
|
||||||
('kb_stage', 'kb_review_principals'),
|
('kb_stage', 'kb_review_principals'),
|
||||||
('kb_stage', 'kb_proposal_approvals')
|
('kb_stage', 'kb_proposal_approvals'),
|
||||||
|
('kb_stage', 'teleo_v3_installation_ownership')
|
||||||
)
|
)
|
||||||
and relation.relkind in ('r', 'p')
|
and relation.relkind in ('r', 'p')
|
||||||
and not (
|
and not (
|
||||||
(namespace.nspname = 'kb_stage'
|
(namespace.nspname = 'kb_stage'
|
||||||
and relation.relname in ('kb_review_principals', 'kb_proposal_approvals')
|
and relation.relname in (
|
||||||
|
'kb_review_principals', 'kb_proposal_approvals', 'teleo_v3_installation_ownership'
|
||||||
|
)
|
||||||
and owner_role.rolname in (current_user, 'kb_gate_owner'))
|
and owner_role.rolname in (current_user, 'kb_gate_owner'))
|
||||||
or (not (
|
or (not (
|
||||||
namespace.nspname = 'kb_stage'
|
namespace.nspname = 'kb_stage'
|
||||||
and relation.relname in ('kb_review_principals', 'kb_proposal_approvals')
|
and relation.relname in (
|
||||||
|
'kb_review_principals', 'kb_proposal_approvals', 'teleo_v3_installation_ownership'
|
||||||
|
)
|
||||||
)
|
)
|
||||||
and owner_role.rolname = current_user)
|
and owner_role.rolname = current_user)
|
||||||
);
|
);
|
||||||
|
|
@ -161,7 +218,8 @@ begin
|
||||||
and procedure.proname in (
|
and procedure.proname in (
|
||||||
'approve_strict_proposal',
|
'approve_strict_proposal',
|
||||||
'assert_approved_proposal',
|
'assert_approved_proposal',
|
||||||
'finish_approved_proposal'
|
'finish_approved_proposal',
|
||||||
|
'export_applied_proposal_replay_rows'
|
||||||
)
|
)
|
||||||
and owner_role.rolname not in (current_user, 'kb_gate_owner');
|
and owner_role.rolname not in (current_user, 'kb_gate_owner');
|
||||||
|
|
||||||
|
|
@ -185,7 +243,8 @@ begin
|
||||||
and procedure.proname in (
|
and procedure.proname in (
|
||||||
'approve_strict_proposal',
|
'approve_strict_proposal',
|
||||||
'assert_approved_proposal',
|
'assert_approved_proposal',
|
||||||
'finish_approved_proposal'
|
'finish_approved_proposal',
|
||||||
|
'export_applied_proposal_replay_rows'
|
||||||
)
|
)
|
||||||
and not (
|
and not (
|
||||||
(procedure.proname = 'approve_strict_proposal'
|
(procedure.proname = 'approve_strict_proposal'
|
||||||
|
|
@ -199,6 +258,8 @@ begin
|
||||||
or (procedure.proname = 'finish_approved_proposal'
|
or (procedure.proname = 'finish_approved_proposal'
|
||||||
and pg_catalog.oidvectortypes(procedure.proargtypes) =
|
and pg_catalog.oidvectortypes(procedure.proargtypes) =
|
||||||
'uuid, text, jsonb, text, uuid, timestamp with time zone, text, text')
|
'uuid, text, jsonb, text, uuid, timestamp with time zone, text, text')
|
||||||
|
or (procedure.proname = 'export_applied_proposal_replay_rows'
|
||||||
|
and pg_catalog.oidvectortypes(procedure.proargtypes) = 'uuid')
|
||||||
);
|
);
|
||||||
|
|
||||||
if v_drift is not null then
|
if v_drift is not null then
|
||||||
|
|
@ -224,8 +285,19 @@ begin
|
||||||
coalesce(relation.relacl, pg_catalog.acldefault('r', relation.relowner))
|
coalesce(relation.relacl, pg_catalog.acldefault('r', relation.relowner))
|
||||||
) acl
|
) acl
|
||||||
where (namespace.nspname, relation.relname) in (
|
where (namespace.nspname, relation.relname) in (
|
||||||
|
('public', 'agents'),
|
||||||
|
('public', 'strategies'),
|
||||||
|
('public', 'strategy_nodes'),
|
||||||
|
('public', 'claim_evidence'),
|
||||||
|
('public', 'claim_evidence_assessments'),
|
||||||
|
('public', 'claim_edges'),
|
||||||
|
('public', 'claims'),
|
||||||
|
('public', 'sources'),
|
||||||
|
('public', 'reasoning_tools'),
|
||||||
|
('kb_stage', 'kb_proposals'),
|
||||||
('kb_stage', 'kb_review_principals'),
|
('kb_stage', 'kb_review_principals'),
|
||||||
('kb_stage', 'kb_proposal_approvals')
|
('kb_stage', 'kb_proposal_approvals'),
|
||||||
|
('kb_stage', 'teleo_v3_installation_ownership')
|
||||||
)
|
)
|
||||||
and acl.grantee <> 0
|
and acl.grantee <> 0
|
||||||
and acl.grantee <> relation.relowner
|
and acl.grantee <> relation.relowner
|
||||||
|
|
@ -256,8 +328,19 @@ begin
|
||||||
join pg_catalog.pg_namespace namespace on namespace.oid = relation.relnamespace
|
join pg_catalog.pg_namespace namespace on namespace.oid = relation.relnamespace
|
||||||
cross join lateral pg_catalog.aclexplode(attribute.attacl) acl
|
cross join lateral pg_catalog.aclexplode(attribute.attacl) acl
|
||||||
where (namespace.nspname, relation.relname) in (
|
where (namespace.nspname, relation.relname) in (
|
||||||
|
('public', 'agents'),
|
||||||
|
('public', 'strategies'),
|
||||||
|
('public', 'strategy_nodes'),
|
||||||
|
('public', 'claim_evidence'),
|
||||||
|
('public', 'claim_evidence_assessments'),
|
||||||
|
('public', 'claim_edges'),
|
||||||
|
('public', 'claims'),
|
||||||
|
('public', 'sources'),
|
||||||
|
('public', 'reasoning_tools'),
|
||||||
|
('kb_stage', 'kb_proposals'),
|
||||||
('kb_stage', 'kb_review_principals'),
|
('kb_stage', 'kb_review_principals'),
|
||||||
('kb_stage', 'kb_proposal_approvals')
|
('kb_stage', 'kb_proposal_approvals'),
|
||||||
|
('kb_stage', 'teleo_v3_installation_ownership')
|
||||||
)
|
)
|
||||||
and not attribute.attisdropped
|
and not attribute.attisdropped
|
||||||
and acl.grantee <> 0
|
and acl.grantee <> 0
|
||||||
|
|
@ -293,7 +376,8 @@ begin
|
||||||
and procedure.proname in (
|
and procedure.proname in (
|
||||||
'approve_strict_proposal',
|
'approve_strict_proposal',
|
||||||
'assert_approved_proposal',
|
'assert_approved_proposal',
|
||||||
'finish_approved_proposal'
|
'finish_approved_proposal',
|
||||||
|
'export_applied_proposal_replay_rows'
|
||||||
)
|
)
|
||||||
and acl.grantee <> 0
|
and acl.grantee <> 0
|
||||||
and acl.grantee <> procedure.proowner
|
and acl.grantee <> procedure.proowner
|
||||||
|
|
@ -314,13 +398,41 @@ drop function if exists kb_stage.approve_strict_proposal(uuid, text, text);
|
||||||
alter table kb_stage.kb_review_principals owner to kb_gate_owner;
|
alter table kb_stage.kb_review_principals owner to kb_gate_owner;
|
||||||
alter table kb_stage.kb_proposal_approvals owner to kb_gate_owner;
|
alter table kb_stage.kb_proposal_approvals owner to kb_gate_owner;
|
||||||
|
|
||||||
|
alter table kb_stage.kb_proposal_approvals
|
||||||
|
add column if not exists approved_proposal_snapshot jsonb;
|
||||||
|
|
||||||
|
comment on column kb_stage.kb_proposal_approvals.approved_proposal_snapshot is
|
||||||
|
'Exact post-approval kb_proposals row. NULL means exact replay material is unavailable.';
|
||||||
|
|
||||||
|
-- A still-approved legacy row is the only state whose complete approved shape
|
||||||
|
-- remains recoverable without guessing. Already-applied rows stay NULL because
|
||||||
|
-- their pre-apply fields, especially updated_at, cannot be reconstructed.
|
||||||
|
update kb_stage.kb_proposal_approvals approval
|
||||||
|
set approved_proposal_snapshot = pg_catalog.to_jsonb(proposal)
|
||||||
|
from kb_stage.kb_proposals proposal
|
||||||
|
where approval.proposal_id = proposal.id
|
||||||
|
and approval.approved_proposal_snapshot is null
|
||||||
|
and proposal.status = 'approved'
|
||||||
|
and proposal.applied_by_handle is null
|
||||||
|
and proposal.applied_by_agent_id is null
|
||||||
|
and proposal.applied_at is null
|
||||||
|
and approval.proposal_type = proposal.proposal_type
|
||||||
|
and approval.payload = proposal.payload
|
||||||
|
and approval.reviewed_by_handle = proposal.reviewed_by_handle
|
||||||
|
and approval.reviewed_by_agent_id is not distinct from proposal.reviewed_by_agent_id
|
||||||
|
and approval.reviewed_at = proposal.reviewed_at
|
||||||
|
and approval.review_note = proposal.review_note;
|
||||||
|
|
||||||
insert into kb_stage.kb_review_principals
|
insert into kb_stage.kb_review_principals
|
||||||
(db_role, reviewed_by_handle, reviewed_by_agent_id, active, created_at)
|
(db_role, reviewed_by_handle, reviewed_by_agent_id, active, created_at)
|
||||||
select 'kb_review'::name, a.handle, a.id, true,
|
select 'kb_review'::name, 'm3taversal', a.id, true,
|
||||||
'1970-01-01 00:00:00+00'::timestamptz
|
'1970-01-01 00:00:00+00'::timestamptz
|
||||||
from public.agents a
|
from public.agents a
|
||||||
where a.handle = 'm3ta'
|
where a.handle = 'm3taversal'
|
||||||
on conflict (db_role) do nothing;
|
on conflict (db_role) do update
|
||||||
|
set reviewed_by_handle = excluded.reviewed_by_handle,
|
||||||
|
reviewed_by_agent_id = excluded.reviewed_by_agent_id,
|
||||||
|
active = excluded.active;
|
||||||
|
|
||||||
-- Reset every explicit privilege held by the protected roles before granting
|
-- Reset every explicit privilege held by the protected roles before granting
|
||||||
-- the documented matrix. This removes stale DELETE/TRUNCATE/REFERENCES/TRIGGER
|
-- the documented matrix. This removes stale DELETE/TRUNCATE/REFERENCES/TRIGGER
|
||||||
|
|
@ -372,13 +484,15 @@ begin
|
||||||
('public', 'strategies'),
|
('public', 'strategies'),
|
||||||
('public', 'strategy_nodes'),
|
('public', 'strategy_nodes'),
|
||||||
('public', 'claim_evidence'),
|
('public', 'claim_evidence'),
|
||||||
|
('public', 'claim_evidence_assessments'),
|
||||||
('public', 'claim_edges'),
|
('public', 'claim_edges'),
|
||||||
('public', 'claims'),
|
('public', 'claims'),
|
||||||
('public', 'sources'),
|
('public', 'sources'),
|
||||||
('public', 'reasoning_tools'),
|
('public', 'reasoning_tools'),
|
||||||
('kb_stage', 'kb_proposals'),
|
('kb_stage', 'kb_proposals'),
|
||||||
('kb_stage', 'kb_review_principals'),
|
('kb_stage', 'kb_review_principals'),
|
||||||
('kb_stage', 'kb_proposal_approvals')
|
('kb_stage', 'kb_proposal_approvals'),
|
||||||
|
('kb_stage', 'teleo_v3_installation_ownership')
|
||||||
)
|
)
|
||||||
and attribute.attnum > 0
|
and attribute.attnum > 0
|
||||||
and not attribute.attisdropped
|
and not attribute.attisdropped
|
||||||
|
|
@ -392,7 +506,8 @@ begin
|
||||||
);
|
);
|
||||||
if protected_table.schema_name = 'kb_stage'
|
if protected_table.schema_name = 'kb_stage'
|
||||||
and protected_table.table_name in (
|
and protected_table.table_name in (
|
||||||
'kb_proposals', 'kb_review_principals', 'kb_proposal_approvals'
|
'kb_proposals', 'kb_review_principals', 'kb_proposal_approvals',
|
||||||
|
'teleo_v3_installation_ownership'
|
||||||
) then
|
) then
|
||||||
execute pg_catalog.format(
|
execute pg_catalog.format(
|
||||||
'revoke all privileges (%s) on table %I.%I from public',
|
'revoke all privileges (%s) on table %I.%I from public',
|
||||||
|
|
@ -417,6 +532,45 @@ grant select, insert on public.claims to kb_apply;
|
||||||
grant select, insert on public.sources to kb_apply;
|
grant select, insert on public.sources to kb_apply;
|
||||||
grant select, insert on public.reasoning_tools to kb_apply;
|
grant select, insert on public.reasoning_tools to kb_apply;
|
||||||
|
|
||||||
|
-- The V3 assessment table is installed by a separate additive migration. Keep
|
||||||
|
-- this prerequisite runnable before that migration, then normalize its ACLs on
|
||||||
|
-- the first post-migration rerun.
|
||||||
|
do $v3_assessment_acl$
|
||||||
|
declare
|
||||||
|
v_columns text;
|
||||||
|
begin
|
||||||
|
if pg_catalog.to_regclass('public.claim_evidence_assessments') is not null then
|
||||||
|
execute 'revoke all privileges on table public.claim_evidence_assessments '
|
||||||
|
'from public, kb_gate_owner, kb_apply, kb_review';
|
||||||
|
select pg_catalog.string_agg(pg_catalog.format('%I', attribute.attname), ', ' order by attribute.attnum)
|
||||||
|
into v_columns
|
||||||
|
from pg_catalog.pg_attribute attribute
|
||||||
|
where attribute.attrelid = 'public.claim_evidence_assessments'::regclass
|
||||||
|
and attribute.attnum > 0
|
||||||
|
and not attribute.attisdropped;
|
||||||
|
if v_columns is not null then
|
||||||
|
execute pg_catalog.format(
|
||||||
|
'revoke all privileges (%s) on table public.claim_evidence_assessments '
|
||||||
|
'from public, kb_gate_owner, kb_apply, kb_review',
|
||||||
|
v_columns
|
||||||
|
);
|
||||||
|
end if;
|
||||||
|
execute 'grant select, insert on public.claim_evidence_assessments to kb_apply';
|
||||||
|
execute 'grant select on public.claims, public.sources, public.claim_evidence, '
|
||||||
|
'public.claim_edges, public.claim_evidence_assessments to kb_gate_owner';
|
||||||
|
end if;
|
||||||
|
end
|
||||||
|
$v3_assessment_acl$;
|
||||||
|
|
||||||
|
do $v3_installation_ownership_acl$
|
||||||
|
begin
|
||||||
|
if pg_catalog.to_regclass('kb_stage.teleo_v3_installation_ownership') is not null then
|
||||||
|
execute 'revoke all privileges on table kb_stage.teleo_v3_installation_ownership '
|
||||||
|
'from public, kb_gate_owner, kb_apply, kb_review';
|
||||||
|
end if;
|
||||||
|
end
|
||||||
|
$v3_installation_ownership_acl$;
|
||||||
|
|
||||||
-- Keep the primary ledger denial explicit in addition to the full reset above.
|
-- Keep the primary ledger denial explicit in addition to the full reset above.
|
||||||
revoke update on kb_stage.kb_proposals from kb_apply;
|
revoke update on kb_stage.kb_proposals from kb_apply;
|
||||||
grant select on kb_stage.kb_proposals to kb_apply;
|
grant select on kb_stage.kb_proposals to kb_apply;
|
||||||
|
|
@ -527,14 +681,6 @@ begin
|
||||||
end if;
|
end if;
|
||||||
|
|
||||||
v_reviewed_at := pg_catalog.clock_timestamp();
|
v_reviewed_at := pg_catalog.clock_timestamp();
|
||||||
insert into kb_stage.kb_proposal_approvals
|
|
||||||
(proposal_id, proposal_type, payload, reviewed_by_handle,
|
|
||||||
reviewed_by_agent_id, reviewed_by_db_role, reviewed_at, review_note)
|
|
||||||
values
|
|
||||||
(v_row.id, v_row.proposal_type, v_row.payload, v_principal.reviewed_by_handle,
|
|
||||||
v_principal.reviewed_by_agent_id, session_user::name, v_reviewed_at,
|
|
||||||
pg_catalog.btrim(p_review_note));
|
|
||||||
|
|
||||||
update kb_stage.kb_proposals
|
update kb_stage.kb_proposals
|
||||||
set status = 'approved',
|
set status = 'approved',
|
||||||
reviewed_by_handle = v_principal.reviewed_by_handle,
|
reviewed_by_handle = v_principal.reviewed_by_handle,
|
||||||
|
|
@ -552,6 +698,15 @@ begin
|
||||||
p_proposal_id;
|
p_proposal_id;
|
||||||
end if;
|
end if;
|
||||||
|
|
||||||
|
insert into kb_stage.kb_proposal_approvals
|
||||||
|
(proposal_id, proposal_type, payload, reviewed_by_handle,
|
||||||
|
reviewed_by_agent_id, reviewed_by_db_role, reviewed_at, review_note,
|
||||||
|
approved_proposal_snapshot)
|
||||||
|
values
|
||||||
|
(v_row.id, v_row.proposal_type, v_row.payload, v_principal.reviewed_by_handle,
|
||||||
|
v_principal.reviewed_by_agent_id, session_user::name, v_reviewed_at,
|
||||||
|
pg_catalog.btrim(p_review_note), pg_catalog.to_jsonb(v_row));
|
||||||
|
|
||||||
return pg_catalog.jsonb_build_object(
|
return pg_catalog.jsonb_build_object(
|
||||||
'id', v_row.id::text,
|
'id', v_row.id::text,
|
||||||
'proposal_type', v_row.proposal_type,
|
'proposal_type', v_row.proposal_type,
|
||||||
|
|
@ -598,6 +753,7 @@ begin
|
||||||
and a.reviewed_by_agent_id is not distinct from p_reviewed_by_agent_id
|
and a.reviewed_by_agent_id is not distinct from p_reviewed_by_agent_id
|
||||||
and a.reviewed_at = p_reviewed_at
|
and a.reviewed_at = p_reviewed_at
|
||||||
and a.review_note = p_review_note
|
and a.review_note = p_review_note
|
||||||
|
and a.approved_proposal_snapshot = pg_catalog.to_jsonb(p)
|
||||||
for update of p;
|
for update of p;
|
||||||
if not found then
|
if not found then
|
||||||
raise exception 'assert_approved_proposal: proposal % is not the exact immutable approved snapshot',
|
raise exception 'assert_approved_proposal: proposal % is not the exact immutable approved snapshot',
|
||||||
|
|
@ -649,6 +805,7 @@ begin
|
||||||
and a.reviewed_by_agent_id is not distinct from p_reviewed_by_agent_id
|
and a.reviewed_by_agent_id is not distinct from p_reviewed_by_agent_id
|
||||||
and a.reviewed_at = p_reviewed_at
|
and a.reviewed_at = p_reviewed_at
|
||||||
and a.review_note = p_review_note
|
and a.review_note = p_review_note
|
||||||
|
and a.approved_proposal_snapshot = pg_catalog.to_jsonb(p)
|
||||||
for update of p;
|
for update of p;
|
||||||
if not found then
|
if not found then
|
||||||
raise exception 'finish_approved_proposal: proposal % is not the locked immutable approved snapshot',
|
raise exception 'finish_approved_proposal: proposal % is not the locked immutable approved snapshot',
|
||||||
|
|
@ -694,12 +851,88 @@ begin
|
||||||
end
|
end
|
||||||
$function$;
|
$function$;
|
||||||
|
|
||||||
|
-- Read-only transition export. kb_apply receives only EXECUTE on this function,
|
||||||
|
-- never SELECT on the approval table. The stored approved row must be a complete
|
||||||
|
-- kb_proposals record and differ from the applied row only by the guarded apply
|
||||||
|
-- transition before any private material leaves PostgreSQL.
|
||||||
|
create or replace function kb_stage.export_applied_proposal_replay_rows(
|
||||||
|
p_proposal_id uuid
|
||||||
|
) returns jsonb
|
||||||
|
language plpgsql
|
||||||
|
stable
|
||||||
|
security definer
|
||||||
|
set search_path = pg_catalog, pg_temp
|
||||||
|
as $function$
|
||||||
|
declare
|
||||||
|
v_rows jsonb;
|
||||||
|
begin
|
||||||
|
select pg_catalog.jsonb_build_object(
|
||||||
|
'approved_proposal', approval.approved_proposal_snapshot,
|
||||||
|
'approval_snapshot',
|
||||||
|
pg_catalog.to_jsonb(approval) - 'approved_proposal_snapshot',
|
||||||
|
'applied_proposal', pg_catalog.to_jsonb(proposal)
|
||||||
|
)
|
||||||
|
into v_rows
|
||||||
|
from kb_stage.kb_proposals proposal
|
||||||
|
join kb_stage.kb_proposal_approvals approval
|
||||||
|
on approval.proposal_id = proposal.id
|
||||||
|
where proposal.id = p_proposal_id
|
||||||
|
and proposal.status = 'applied'
|
||||||
|
and proposal.applied_by_handle is not null
|
||||||
|
and proposal.applied_by_agent_id is not null
|
||||||
|
and proposal.applied_at is not null
|
||||||
|
and proposal.reviewed_by_handle is not null
|
||||||
|
and proposal.reviewed_by_agent_id is not null
|
||||||
|
and proposal.reviewed_at is not null
|
||||||
|
and proposal.review_note is not null
|
||||||
|
and nullif(pg_catalog.btrim(approval.reviewed_by_db_role::text), '') is not null
|
||||||
|
and pg_catalog.jsonb_typeof(approval.approved_proposal_snapshot) = 'object'
|
||||||
|
and approval.approved_proposal_snapshot = pg_catalog.to_jsonb(
|
||||||
|
pg_catalog.jsonb_populate_record(
|
||||||
|
null::kb_stage.kb_proposals,
|
||||||
|
approval.approved_proposal_snapshot
|
||||||
|
)
|
||||||
|
)
|
||||||
|
and approval.approved_proposal_snapshot->>'status' = 'approved'
|
||||||
|
and approval.approved_proposal_snapshot->'applied_by_handle' = 'null'::jsonb
|
||||||
|
and approval.approved_proposal_snapshot->'applied_by_agent_id' = 'null'::jsonb
|
||||||
|
and approval.approved_proposal_snapshot->'applied_at' = 'null'::jsonb
|
||||||
|
and approval.proposal_id = (approval.approved_proposal_snapshot->>'id')::uuid
|
||||||
|
and approval.proposal_type = approval.approved_proposal_snapshot->>'proposal_type'
|
||||||
|
and approval.payload = approval.approved_proposal_snapshot->'payload'
|
||||||
|
and approval.reviewed_by_handle =
|
||||||
|
approval.approved_proposal_snapshot->>'reviewed_by_handle'
|
||||||
|
and approval.reviewed_by_agent_id =
|
||||||
|
(approval.approved_proposal_snapshot->>'reviewed_by_agent_id')::uuid
|
||||||
|
and approval.reviewed_at =
|
||||||
|
(approval.approved_proposal_snapshot->>'reviewed_at')::timestamptz
|
||||||
|
and approval.review_note = approval.approved_proposal_snapshot->>'review_note'
|
||||||
|
and (
|
||||||
|
approval.approved_proposal_snapshot || pg_catalog.jsonb_build_object(
|
||||||
|
'status', proposal.status,
|
||||||
|
'applied_by_handle', proposal.applied_by_handle,
|
||||||
|
'applied_by_agent_id', proposal.applied_by_agent_id,
|
||||||
|
'applied_at', proposal.applied_at,
|
||||||
|
'updated_at', proposal.updated_at
|
||||||
|
)
|
||||||
|
) = pg_catalog.to_jsonb(proposal);
|
||||||
|
|
||||||
|
if v_rows is null then
|
||||||
|
raise exception 'export_applied_proposal_replay_rows: exact applied transition unavailable for proposal %',
|
||||||
|
p_proposal_id;
|
||||||
|
end if;
|
||||||
|
return v_rows;
|
||||||
|
end
|
||||||
|
$function$;
|
||||||
|
|
||||||
alter function kb_stage.approve_strict_proposal(uuid, text, jsonb, text, text)
|
alter function kb_stage.approve_strict_proposal(uuid, text, jsonb, text, text)
|
||||||
owner to kb_gate_owner;
|
owner to kb_gate_owner;
|
||||||
alter function kb_stage.assert_approved_proposal(uuid, text, jsonb, text, uuid, timestamptz, text)
|
alter function kb_stage.assert_approved_proposal(uuid, text, jsonb, text, uuid, timestamptz, text)
|
||||||
owner to kb_gate_owner;
|
owner to kb_gate_owner;
|
||||||
alter function kb_stage.finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamptz, text, text)
|
alter function kb_stage.finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamptz, text, text)
|
||||||
owner to kb_gate_owner;
|
owner to kb_gate_owner;
|
||||||
|
alter function kb_stage.export_applied_proposal_replay_rows(uuid)
|
||||||
|
owner to kb_gate_owner;
|
||||||
|
|
||||||
revoke all on function kb_stage.approve_strict_proposal(uuid, text, jsonb, text, text)
|
revoke all on function kb_stage.approve_strict_proposal(uuid, text, jsonb, text, text)
|
||||||
from public, kb_gate_owner, kb_apply, kb_review;
|
from public, kb_gate_owner, kb_apply, kb_review;
|
||||||
|
|
@ -707,6 +940,8 @@ revoke all on function kb_stage.assert_approved_proposal(uuid, text, jsonb, text
|
||||||
from public, kb_gate_owner, kb_apply, kb_review;
|
from public, kb_gate_owner, kb_apply, kb_review;
|
||||||
revoke all on function kb_stage.finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamptz, text, text)
|
revoke all on function kb_stage.finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamptz, text, text)
|
||||||
from public, kb_gate_owner, kb_apply, kb_review;
|
from public, kb_gate_owner, kb_apply, kb_review;
|
||||||
|
revoke all on function kb_stage.export_applied_proposal_replay_rows(uuid)
|
||||||
|
from public, kb_gate_owner, kb_apply, kb_review;
|
||||||
|
|
||||||
grant execute on function kb_stage.approve_strict_proposal(uuid, text, jsonb, text, text)
|
grant execute on function kb_stage.approve_strict_proposal(uuid, text, jsonb, text, text)
|
||||||
to kb_review;
|
to kb_review;
|
||||||
|
|
@ -714,6 +949,8 @@ grant execute on function kb_stage.assert_approved_proposal(uuid, text, jsonb, t
|
||||||
to kb_apply;
|
to kb_apply;
|
||||||
grant execute on function kb_stage.finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamptz, text, text)
|
grant execute on function kb_stage.finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamptz, text, text)
|
||||||
to kb_apply;
|
to kb_apply;
|
||||||
|
grant execute on function kb_stage.export_applied_proposal_replay_rows(uuid)
|
||||||
|
to kb_apply;
|
||||||
|
|
||||||
-- Assert the complete protected catalog contract before commit. These checks
|
-- Assert the complete protected catalog contract before commit. These checks
|
||||||
-- make reruns an upgrade proof rather than a best-effort sequence of grants.
|
-- make reruns an upgrade proof rather than a best-effort sequence of grants.
|
||||||
|
|
@ -755,6 +992,20 @@ begin
|
||||||
raise exception 'kb_apply_prereqs: protected role membership appeared during migration: %', v_drift;
|
raise exception 'kb_apply_prereqs: protected role membership appeared during migration: %', v_drift;
|
||||||
end if;
|
end if;
|
||||||
|
|
||||||
|
if exists (select 1 from public.agents where handle = 'm3ta')
|
||||||
|
or not exists (
|
||||||
|
select 1
|
||||||
|
from kb_stage.kb_review_principals principal
|
||||||
|
join public.agents reviewer
|
||||||
|
on reviewer.id = principal.reviewed_by_agent_id
|
||||||
|
where principal.db_role = 'kb_review'::name
|
||||||
|
and principal.reviewed_by_handle = 'm3taversal'
|
||||||
|
and principal.active
|
||||||
|
and reviewer.handle = 'm3taversal'
|
||||||
|
) then
|
||||||
|
raise exception 'kb_apply_prereqs: kb_review principal did not normalize to active m3taversal';
|
||||||
|
end if;
|
||||||
|
|
||||||
select pg_catalog.string_agg(
|
select pg_catalog.string_agg(
|
||||||
pg_catalog.format(
|
pg_catalog.format(
|
||||||
'%I on %I usage=%s create=%s',
|
'%I on %I usage=%s create=%s',
|
||||||
|
|
@ -792,21 +1043,27 @@ begin
|
||||||
('public', 'strategies'),
|
('public', 'strategies'),
|
||||||
('public', 'strategy_nodes'),
|
('public', 'strategy_nodes'),
|
||||||
('public', 'claim_evidence'),
|
('public', 'claim_evidence'),
|
||||||
|
('public', 'claim_evidence_assessments'),
|
||||||
('public', 'claim_edges'),
|
('public', 'claim_edges'),
|
||||||
('public', 'claims'),
|
('public', 'claims'),
|
||||||
('public', 'sources'),
|
('public', 'sources'),
|
||||||
('public', 'reasoning_tools'),
|
('public', 'reasoning_tools'),
|
||||||
('kb_stage', 'kb_proposals'),
|
('kb_stage', 'kb_proposals'),
|
||||||
('kb_stage', 'kb_review_principals'),
|
('kb_stage', 'kb_review_principals'),
|
||||||
('kb_stage', 'kb_proposal_approvals')
|
('kb_stage', 'kb_proposal_approvals'),
|
||||||
|
('kb_stage', 'teleo_v3_installation_ownership')
|
||||||
)
|
)
|
||||||
and not (
|
and not (
|
||||||
(namespace.nspname = 'kb_stage'
|
(namespace.nspname = 'kb_stage'
|
||||||
and relation.relname in ('kb_review_principals', 'kb_proposal_approvals')
|
and relation.relname in (
|
||||||
|
'kb_review_principals', 'kb_proposal_approvals', 'teleo_v3_installation_ownership'
|
||||||
|
)
|
||||||
and owner_role.rolname = 'kb_gate_owner')
|
and owner_role.rolname = 'kb_gate_owner')
|
||||||
or (not (
|
or (not (
|
||||||
namespace.nspname = 'kb_stage'
|
namespace.nspname = 'kb_stage'
|
||||||
and relation.relname in ('kb_review_principals', 'kb_proposal_approvals')
|
and relation.relname in (
|
||||||
|
'kb_review_principals', 'kb_proposal_approvals', 'teleo_v3_installation_ownership'
|
||||||
|
)
|
||||||
)
|
)
|
||||||
and owner_role.rolname = current_user)
|
and owner_role.rolname = current_user)
|
||||||
);
|
);
|
||||||
|
|
@ -815,11 +1072,24 @@ begin
|
||||||
raise exception 'kb_apply_prereqs: protected table ownership mismatch: %', v_drift;
|
raise exception 'kb_apply_prereqs: protected table ownership mismatch: %', v_drift;
|
||||||
end if;
|
end if;
|
||||||
|
|
||||||
|
if not exists (
|
||||||
|
select 1
|
||||||
|
from pg_catalog.pg_attribute attribute
|
||||||
|
where attribute.attrelid = 'kb_stage.kb_proposal_approvals'::regclass
|
||||||
|
and attribute.attname = 'approved_proposal_snapshot'
|
||||||
|
and attribute.atttypid = 'pg_catalog.jsonb'::regtype
|
||||||
|
and not attribute.attnotnull
|
||||||
|
and not attribute.attisdropped
|
||||||
|
) then
|
||||||
|
raise exception 'kb_apply_prereqs: approved proposal snapshot column is not nullable JSONB';
|
||||||
|
end if;
|
||||||
|
|
||||||
with expected(signature) as (
|
with expected(signature) as (
|
||||||
values
|
values
|
||||||
('approve_strict_proposal(uuid, text, jsonb, text, text)'),
|
('approve_strict_proposal(uuid, text, jsonb, text, text)'),
|
||||||
('assert_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text)'),
|
('assert_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text)'),
|
||||||
('finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text, text)')
|
('finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text, text)'),
|
||||||
|
('export_applied_proposal_replay_rows(uuid)')
|
||||||
), actual(signature) as (
|
), actual(signature) as (
|
||||||
select procedure.proname || '(' || pg_catalog.oidvectortypes(procedure.proargtypes) || ')'
|
select procedure.proname || '(' || pg_catalog.oidvectortypes(procedure.proargtypes) || ')'
|
||||||
from pg_catalog.pg_proc procedure
|
from pg_catalog.pg_proc procedure
|
||||||
|
|
@ -828,7 +1098,8 @@ begin
|
||||||
and procedure.proname in (
|
and procedure.proname in (
|
||||||
'approve_strict_proposal',
|
'approve_strict_proposal',
|
||||||
'assert_approved_proposal',
|
'assert_approved_proposal',
|
||||||
'finish_approved_proposal'
|
'finish_approved_proposal',
|
||||||
|
'export_applied_proposal_replay_rows'
|
||||||
)
|
)
|
||||||
), delta(kind, signature) as (
|
), delta(kind, signature) as (
|
||||||
select 'unexpected', actual.signature
|
select 'unexpected', actual.signature
|
||||||
|
|
@ -867,13 +1138,22 @@ begin
|
||||||
and procedure.proname in (
|
and procedure.proname in (
|
||||||
'approve_strict_proposal',
|
'approve_strict_proposal',
|
||||||
'assert_approved_proposal',
|
'assert_approved_proposal',
|
||||||
'finish_approved_proposal'
|
'finish_approved_proposal',
|
||||||
|
'export_applied_proposal_replay_rows'
|
||||||
)
|
)
|
||||||
and (
|
and (
|
||||||
owner_role.rolname <> 'kb_gate_owner'
|
owner_role.rolname <> 'kb_gate_owner'
|
||||||
or not procedure.prosecdef
|
or not procedure.prosecdef
|
||||||
or pg_catalog.array_to_string(procedure.proconfig, ',') is distinct from
|
or pg_catalog.array_to_string(procedure.proconfig, ',') is distinct from
|
||||||
'search_path=pg_catalog, pg_temp'
|
'search_path=pg_catalog, pg_temp'
|
||||||
|
or (
|
||||||
|
procedure.proname = 'export_applied_proposal_replay_rows'
|
||||||
|
and (
|
||||||
|
procedure.provolatile <> 's'
|
||||||
|
or procedure.prokind <> 'f'
|
||||||
|
or procedure.prorettype <> 'pg_catalog.jsonb'::regtype
|
||||||
|
)
|
||||||
|
)
|
||||||
);
|
);
|
||||||
|
|
||||||
if v_drift is not null then
|
if v_drift is not null then
|
||||||
|
|
@ -886,13 +1166,15 @@ begin
|
||||||
('public', 'strategies'),
|
('public', 'strategies'),
|
||||||
('public', 'strategy_nodes'),
|
('public', 'strategy_nodes'),
|
||||||
('public', 'claim_evidence'),
|
('public', 'claim_evidence'),
|
||||||
|
('public', 'claim_evidence_assessments'),
|
||||||
('public', 'claim_edges'),
|
('public', 'claim_edges'),
|
||||||
('public', 'claims'),
|
('public', 'claims'),
|
||||||
('public', 'sources'),
|
('public', 'sources'),
|
||||||
('public', 'reasoning_tools'),
|
('public', 'reasoning_tools'),
|
||||||
('kb_stage', 'kb_proposals'),
|
('kb_stage', 'kb_proposals'),
|
||||||
('kb_stage', 'kb_review_principals'),
|
('kb_stage', 'kb_review_principals'),
|
||||||
('kb_stage', 'kb_proposal_approvals')
|
('kb_stage', 'kb_proposal_approvals'),
|
||||||
|
('kb_stage', 'teleo_v3_installation_ownership')
|
||||||
)
|
)
|
||||||
select pg_catalog.string_agg(
|
select pg_catalog.string_agg(
|
||||||
pg_catalog.format(
|
pg_catalog.format(
|
||||||
|
|
@ -920,8 +1202,7 @@ begin
|
||||||
left join pg_catalog.pg_roles grantee_role on grantee_role.oid = acl.grantee
|
left join pg_catalog.pg_roles grantee_role on grantee_role.oid = acl.grantee
|
||||||
where attribute.attnum > 0
|
where attribute.attnum > 0
|
||||||
and not attribute.attisdropped
|
and not attribute.attisdropped
|
||||||
and (acl.grantee = 0
|
and acl.grantee <> relation.relowner;
|
||||||
or grantee_role.rolname in ('kb_gate_owner', 'kb_review', 'kb_apply'));
|
|
||||||
|
|
||||||
if v_drift is not null then
|
if v_drift is not null then
|
||||||
raise exception 'kb_apply_prereqs: protected column ACL mismatch: %', v_drift;
|
raise exception 'kb_apply_prereqs: protected column ACL mismatch: %', v_drift;
|
||||||
|
|
@ -952,19 +1233,42 @@ begin
|
||||||
('kb_stage', 'kb_proposals', 'kb_gate_owner', 'UPDATE', false),
|
('kb_stage', 'kb_proposals', 'kb_gate_owner', 'UPDATE', false),
|
||||||
('kb_stage', 'kb_proposals', 'kb_apply', 'SELECT', false),
|
('kb_stage', 'kb_proposals', 'kb_apply', 'SELECT', false),
|
||||||
('kb_stage', 'kb_proposals', 'kb_review', 'SELECT', false)
|
('kb_stage', 'kb_proposals', 'kb_review', 'SELECT', false)
|
||||||
|
union all
|
||||||
|
select 'public', 'claim_evidence_assessments', 'kb_apply', 'SELECT', false
|
||||||
|
where pg_catalog.to_regclass('public.claim_evidence_assessments') is not null
|
||||||
|
union all
|
||||||
|
select 'public', 'claim_evidence_assessments', 'kb_apply', 'INSERT', false
|
||||||
|
where pg_catalog.to_regclass('public.claim_evidence_assessments') is not null
|
||||||
|
union all
|
||||||
|
select 'public', 'claims', 'kb_gate_owner', 'SELECT', false
|
||||||
|
where pg_catalog.to_regclass('public.claim_evidence_assessments') is not null
|
||||||
|
union all
|
||||||
|
select 'public', 'sources', 'kb_gate_owner', 'SELECT', false
|
||||||
|
where pg_catalog.to_regclass('public.claim_evidence_assessments') is not null
|
||||||
|
union all
|
||||||
|
select 'public', 'claim_evidence', 'kb_gate_owner', 'SELECT', false
|
||||||
|
where pg_catalog.to_regclass('public.claim_evidence_assessments') is not null
|
||||||
|
union all
|
||||||
|
select 'public', 'claim_edges', 'kb_gate_owner', 'SELECT', false
|
||||||
|
where pg_catalog.to_regclass('public.claim_evidence_assessments') is not null
|
||||||
|
union all
|
||||||
|
select 'public', 'claim_evidence_assessments', 'kb_gate_owner', 'SELECT', false
|
||||||
|
where pg_catalog.to_regclass('public.claim_evidence_assessments') is not null
|
||||||
), tracked(schema_name, object_name) as (
|
), tracked(schema_name, object_name) as (
|
||||||
values
|
values
|
||||||
('public', 'agents'),
|
('public', 'agents'),
|
||||||
('public', 'strategies'),
|
('public', 'strategies'),
|
||||||
('public', 'strategy_nodes'),
|
('public', 'strategy_nodes'),
|
||||||
('public', 'claim_evidence'),
|
('public', 'claim_evidence'),
|
||||||
|
('public', 'claim_evidence_assessments'),
|
||||||
('public', 'claim_edges'),
|
('public', 'claim_edges'),
|
||||||
('public', 'claims'),
|
('public', 'claims'),
|
||||||
('public', 'sources'),
|
('public', 'sources'),
|
||||||
('public', 'reasoning_tools'),
|
('public', 'reasoning_tools'),
|
||||||
('kb_stage', 'kb_proposals'),
|
('kb_stage', 'kb_proposals'),
|
||||||
('kb_stage', 'kb_review_principals'),
|
('kb_stage', 'kb_review_principals'),
|
||||||
('kb_stage', 'kb_proposal_approvals')
|
('kb_stage', 'kb_proposal_approvals'),
|
||||||
|
('kb_stage', 'teleo_v3_installation_ownership')
|
||||||
), actual(schema_name, object_name, grantee, privilege_type, is_grantable) as (
|
), actual(schema_name, object_name, grantee, privilege_type, is_grantable) as (
|
||||||
select distinct
|
select distinct
|
||||||
namespace.nspname::text,
|
namespace.nspname::text,
|
||||||
|
|
@ -985,8 +1289,6 @@ begin
|
||||||
) acl
|
) acl
|
||||||
left join pg_catalog.pg_roles grantee_role on grantee_role.oid = acl.grantee
|
left join pg_catalog.pg_roles grantee_role on grantee_role.oid = acl.grantee
|
||||||
where acl.grantee <> relation.relowner
|
where acl.grantee <> relation.relowner
|
||||||
and (acl.grantee = 0
|
|
||||||
or grantee_role.rolname in ('kb_gate_owner', 'kb_review', 'kb_apply'))
|
|
||||||
), delta(kind, schema_name, object_name, grantee, privilege_type, is_grantable) as (
|
), delta(kind, schema_name, object_name, grantee, privilege_type, is_grantable) as (
|
||||||
select 'unexpected', actual.*
|
select 'unexpected', actual.*
|
||||||
from actual
|
from actual
|
||||||
|
|
@ -1020,6 +1322,8 @@ begin
|
||||||
('assert_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text)',
|
('assert_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text)',
|
||||||
'kb_apply', 'EXECUTE', false),
|
'kb_apply', 'EXECUTE', false),
|
||||||
('finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text, text)',
|
('finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text, text)',
|
||||||
|
'kb_apply', 'EXECUTE', false),
|
||||||
|
('export_applied_proposal_replay_rows(uuid)',
|
||||||
'kb_apply', 'EXECUTE', false)
|
'kb_apply', 'EXECUTE', false)
|
||||||
), actual(signature, grantee, privilege_type, is_grantable) as (
|
), actual(signature, grantee, privilege_type, is_grantable) as (
|
||||||
select distinct
|
select distinct
|
||||||
|
|
@ -1039,7 +1343,8 @@ begin
|
||||||
and procedure.proname in (
|
and procedure.proname in (
|
||||||
'approve_strict_proposal',
|
'approve_strict_proposal',
|
||||||
'assert_approved_proposal',
|
'assert_approved_proposal',
|
||||||
'finish_approved_proposal'
|
'finish_approved_proposal',
|
||||||
|
'export_applied_proposal_replay_rows'
|
||||||
)
|
)
|
||||||
and acl.grantee <> procedure.proowner
|
and acl.grantee <> procedure.proowner
|
||||||
), delta(kind, signature, grantee, privilege_type, is_grantable) as (
|
), delta(kind, signature, grantee, privilege_type, is_grantable) as (
|
||||||
|
|
|
||||||
|
|
@ -13,13 +13,79 @@ from __future__ import annotations
|
||||||
import hashlib
|
import hashlib
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
import re
|
||||||
import tempfile
|
import tempfile
|
||||||
|
import uuid
|
||||||
from datetime import datetime, timezone
|
from datetime import datetime, timezone
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
RECEIPT_CONTRACT_VERSION = 1
|
RECEIPT_CONTRACT_VERSION = 1
|
||||||
DEFAULT_RECEIPT_DIR = "/opt/teleo-eval/logs/kb-apply-receipts"
|
DEFAULT_RECEIPT_DIR = "/opt/teleo-eval/logs/kb-apply-receipts"
|
||||||
|
UTC_SESSION_SQL = "set timezone = 'UTC';"
|
||||||
|
_ISO_TIMESTAMP = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{1,6})?(?:Z|[+-]\d{2}:\d{2})$")
|
||||||
|
_UUID_FIELDS = {
|
||||||
|
"id",
|
||||||
|
"agent_id",
|
||||||
|
"owner_agent_id",
|
||||||
|
"created_by",
|
||||||
|
"superseded_by",
|
||||||
|
"supersedes_id",
|
||||||
|
"accepted_by_proposal_id",
|
||||||
|
"claim_id",
|
||||||
|
"source_id",
|
||||||
|
"accepted_from_proposal_id",
|
||||||
|
"from_claim",
|
||||||
|
"to_claim",
|
||||||
|
"supersedes_assessment_id",
|
||||||
|
"reviewed_by_agent_id",
|
||||||
|
"applied_by_agent_id",
|
||||||
|
}
|
||||||
|
_TIMESTAMP_FIELDS = {"captured_at", "created_at", "updated_at", "reviewed_at", "applied_at"}
|
||||||
|
|
||||||
|
|
||||||
|
def canonical_utc_timestamp(value: Any, *, path: str = "timestamp") -> str:
|
||||||
|
"""Normalize a typed timestamp to fixed-microsecond UTC receipt text."""
|
||||||
|
if not isinstance(value, str) or not _ISO_TIMESTAMP.fullmatch(value):
|
||||||
|
raise ValueError(f"{path} must be a timezone-aware ISO-8601 timestamp with at most 6 fractional digits")
|
||||||
|
try:
|
||||||
|
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||||
|
except ValueError as exc:
|
||||||
|
raise ValueError(f"{path} must be a valid ISO-8601 timestamp") from exc
|
||||||
|
if parsed.tzinfo is None:
|
||||||
|
raise ValueError(f"{path} must include a timezone")
|
||||||
|
return parsed.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ")
|
||||||
|
|
||||||
|
|
||||||
|
def _canonical_uuid(value: Any, *, path: str) -> str:
|
||||||
|
try:
|
||||||
|
return str(uuid.UUID(str(value)))
|
||||||
|
except (AttributeError, TypeError, ValueError) as exc:
|
||||||
|
raise ValueError(f"{path} must be a UUID") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_typed_value(field: str, value: Any, *, path: str) -> Any:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
if field in _UUID_FIELDS:
|
||||||
|
return _canonical_uuid(value, path=path)
|
||||||
|
if field in _TIMESTAMP_FIELDS:
|
||||||
|
return canonical_utc_timestamp(value, path=path)
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_typed_row(row: dict[str, Any], *, path: str) -> dict[str, Any]:
|
||||||
|
return {field: _normalize_typed_value(field, value, path=f"{path}.{field}") for field, value in row.items()}
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_typed_value(field: str, value: Any, *, path: str) -> Any:
|
||||||
|
"""Canonicalize a value according to its PostgreSQL column type."""
|
||||||
|
return _normalize_typed_value(field, value, path=path)
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_typed_row(row: dict[str, Any], *, path: str) -> dict[str, Any]:
|
||||||
|
"""Canonicalize UUID and timestamp columns without weakening text equality."""
|
||||||
|
return _normalize_typed_row(row, path=path)
|
||||||
|
|
||||||
|
|
||||||
def _canonical_json(value: Any) -> str:
|
def _canonical_json(value: Any) -> str:
|
||||||
|
|
@ -56,6 +122,12 @@ def _require_applied_strict_proposal(proposal: dict[str, Any]) -> dict[str, Any]
|
||||||
apply_payload = payload.get("apply_payload") if isinstance(payload, dict) else None
|
apply_payload = payload.get("apply_payload") if isinstance(payload, dict) else None
|
||||||
if not isinstance(apply_payload, dict):
|
if not isinstance(apply_payload, dict):
|
||||||
raise ValueError("replay receipt requires a strict payload.apply_payload object")
|
raise ValueError("replay receipt requires a strict payload.apply_payload object")
|
||||||
|
raw_version = apply_payload.get("contract_version")
|
||||||
|
if proposal.get("proposal_type") == "approve_claim":
|
||||||
|
if type(raw_version) is not int or raw_version not in {2, 3}:
|
||||||
|
raise ValueError("approve_claim replay receipt requires integer contract_version 2 or 3")
|
||||||
|
elif "contract_version" in apply_payload:
|
||||||
|
raise ValueError("legacy replay receipt proposal types must omit contract_version")
|
||||||
return apply_payload
|
return apply_payload
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -108,6 +180,8 @@ def _build_attach_evidence_postflight_sql(apply_payload: dict[str, Any]) -> str:
|
||||||
|
|
||||||
|
|
||||||
def _build_approve_claim_postflight_sql(apply_payload: dict[str, Any]) -> str:
|
def _build_approve_claim_postflight_sql(apply_payload: dict[str, Any]) -> str:
|
||||||
|
if apply_payload.get("contract_version") == 3:
|
||||||
|
return _build_approve_claim_v3_postflight_sql(apply_payload)
|
||||||
claims = apply_payload.get("claims") or []
|
claims = apply_payload.get("claims") or []
|
||||||
sources = apply_payload.get("sources") or []
|
sources = apply_payload.get("sources") or []
|
||||||
evidence = apply_payload.get("evidence") or []
|
evidence = apply_payload.get("evidence") or []
|
||||||
|
|
@ -154,6 +228,44 @@ def _build_approve_claim_postflight_sql(apply_payload: dict[str, Any]) -> str:
|
||||||
return f"select jsonb_build_object({arguments})::text;"
|
return f"select jsonb_build_object({arguments})::text;"
|
||||||
|
|
||||||
|
|
||||||
|
def _build_approve_claim_v3_postflight_sql(apply_payload: dict[str, Any]) -> str:
|
||||||
|
fields = {
|
||||||
|
"claims": _coalesced_json_agg(
|
||||||
|
"public.claims",
|
||||||
|
"c",
|
||||||
|
_id_filter("c", [row.get("id") for row in apply_payload.get("claims") or []]),
|
||||||
|
"c.id::text",
|
||||||
|
),
|
||||||
|
"sources": _coalesced_json_agg(
|
||||||
|
"public.sources",
|
||||||
|
"s",
|
||||||
|
_id_filter("s", [row.get("id") for row in apply_payload.get("sources") or []]),
|
||||||
|
"s.id::text",
|
||||||
|
),
|
||||||
|
"claim_evidence": _coalesced_json_agg(
|
||||||
|
"public.claim_evidence",
|
||||||
|
"ce",
|
||||||
|
_id_filter("ce", [row.get("id") for row in apply_payload.get("evidence") or []]),
|
||||||
|
"ce.id::text",
|
||||||
|
),
|
||||||
|
"claim_edges": _coalesced_json_agg(
|
||||||
|
"public.claim_edges",
|
||||||
|
"e",
|
||||||
|
_id_filter("e", [row.get("id") for row in apply_payload.get("edges") or []]),
|
||||||
|
"e.id::text",
|
||||||
|
),
|
||||||
|
"claim_evidence_assessments": _coalesced_json_agg(
|
||||||
|
"public.claim_evidence_assessments",
|
||||||
|
"a",
|
||||||
|
_id_filter("a", [row.get("id") for row in apply_payload.get("assessments") or []]),
|
||||||
|
"a.id::text",
|
||||||
|
),
|
||||||
|
"reasoning_tools": "'[]'::jsonb",
|
||||||
|
}
|
||||||
|
arguments = ", ".join(f"{_sql_literal(name)}, {expression}" for name, expression in fields.items())
|
||||||
|
return f"select jsonb_build_object({arguments})::text;"
|
||||||
|
|
||||||
|
|
||||||
def _build_revise_strategy_postflight_sql(proposal: dict[str, Any], apply_payload: dict[str, Any]) -> str:
|
def _build_revise_strategy_postflight_sql(proposal: dict[str, Any], apply_payload: dict[str, Any]) -> str:
|
||||||
strategy = apply_payload.get("strategy") or {}
|
strategy = apply_payload.get("strategy") or {}
|
||||||
agent_id = apply_payload.get("agent_id")
|
agent_id = apply_payload.get("agent_id")
|
||||||
|
|
@ -193,14 +305,16 @@ def build_postflight_sql(proposal: dict[str, Any]) -> str:
|
||||||
apply_payload = _require_applied_strict_proposal(proposal)
|
apply_payload = _require_applied_strict_proposal(proposal)
|
||||||
proposal_type = proposal.get("proposal_type")
|
proposal_type = proposal.get("proposal_type")
|
||||||
if proposal_type == "add_edge":
|
if proposal_type == "add_edge":
|
||||||
return _build_add_edge_postflight_sql(apply_payload)
|
query = _build_add_edge_postflight_sql(apply_payload)
|
||||||
if proposal_type == "attach_evidence":
|
elif proposal_type == "attach_evidence":
|
||||||
return _build_attach_evidence_postflight_sql(apply_payload)
|
query = _build_attach_evidence_postflight_sql(apply_payload)
|
||||||
if proposal_type == "approve_claim":
|
elif proposal_type == "approve_claim":
|
||||||
return _build_approve_claim_postflight_sql(apply_payload)
|
query = _build_approve_claim_postflight_sql(apply_payload)
|
||||||
if proposal_type == "revise_strategy":
|
elif proposal_type == "revise_strategy":
|
||||||
return _build_revise_strategy_postflight_sql(proposal, apply_payload)
|
query = _build_revise_strategy_postflight_sql(proposal, apply_payload)
|
||||||
raise ValueError(f"replay receipt does not support proposal_type {proposal_type!r}")
|
else:
|
||||||
|
raise ValueError(f"replay receipt does not support proposal_type {proposal_type!r}")
|
||||||
|
return UTC_SESSION_SQL + "\n" + query
|
||||||
|
|
||||||
|
|
||||||
def expected_row_counts(proposal: dict[str, Any]) -> dict[str, int]:
|
def expected_row_counts(proposal: dict[str, Any]) -> dict[str, int]:
|
||||||
|
|
@ -211,6 +325,15 @@ def expected_row_counts(proposal: dict[str, Any]) -> dict[str, int]:
|
||||||
if proposal_type == "attach_evidence":
|
if proposal_type == "attach_evidence":
|
||||||
return {"claim_evidence": len(apply_payload.get("evidence") or [])}
|
return {"claim_evidence": len(apply_payload.get("evidence") or [])}
|
||||||
if proposal_type == "approve_claim":
|
if proposal_type == "approve_claim":
|
||||||
|
if apply_payload.get("contract_version") == 3:
|
||||||
|
return {
|
||||||
|
"claims": len(apply_payload.get("claims") or []),
|
||||||
|
"sources": len(apply_payload.get("sources") or []),
|
||||||
|
"claim_evidence": len(apply_payload.get("evidence") or []),
|
||||||
|
"claim_edges": len(apply_payload.get("edges") or []),
|
||||||
|
"claim_evidence_assessments": len(apply_payload.get("assessments") or []),
|
||||||
|
"reasoning_tools": 0,
|
||||||
|
}
|
||||||
return {
|
return {
|
||||||
"claims": len(apply_payload.get("claims") or []),
|
"claims": len(apply_payload.get("claims") or []),
|
||||||
"sources": len(apply_payload.get("sources") or []),
|
"sources": len(apply_payload.get("sources") or []),
|
||||||
|
|
@ -232,12 +355,13 @@ def _normalize_rows(canonical_rows: dict[str, Any], expected_counts: dict[str, i
|
||||||
rows = canonical_rows.get(table)
|
rows = canonical_rows.get(table)
|
||||||
if not isinstance(rows, list) or any(not isinstance(row, dict) for row in rows):
|
if not isinstance(rows, list) or any(not isinstance(row, dict) for row in rows):
|
||||||
raise ValueError(f"postflight {table} must be a list of row objects")
|
raise ValueError(f"postflight {table} must be a list of row objects")
|
||||||
normalized[table] = sorted(rows, key=_canonical_json)
|
typed_rows = [_normalize_typed_row(row, path=f"postflight.{table}") for row in rows]
|
||||||
|
normalized[table] = sorted(typed_rows, key=_canonical_json)
|
||||||
return normalized
|
return normalized
|
||||||
|
|
||||||
|
|
||||||
def _semantic_projection(row: dict[str, Any], fields: tuple[str, ...]) -> dict[str, Any]:
|
def _semantic_projection(row: dict[str, Any], fields: tuple[str, ...]) -> dict[str, Any]:
|
||||||
return {field: row.get(field) for field in fields}
|
return {field: _normalize_typed_value(field, row.get(field), path=f"semantic.{field}") for field in fields}
|
||||||
|
|
||||||
|
|
||||||
def _expected_semantic_rows(proposal: dict[str, Any]) -> dict[str, list[dict[str, Any]]]:
|
def _expected_semantic_rows(proposal: dict[str, Any]) -> dict[str, list[dict[str, Any]]]:
|
||||||
|
|
@ -269,6 +393,8 @@ def _expected_semantic_rows(proposal: dict[str, Any]) -> dict[str, list[dict[str
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
if proposal_type == "approve_claim":
|
if proposal_type == "approve_claim":
|
||||||
|
if apply_payload.get("contract_version") == 3:
|
||||||
|
return _expected_v3_semantic_rows(proposal, apply_payload)
|
||||||
agent_id = apply_payload.get("agent_id")
|
agent_id = apply_payload.get("agent_id")
|
||||||
return {
|
return {
|
||||||
"claims": [
|
"claims": [
|
||||||
|
|
@ -359,12 +485,115 @@ def _expected_semantic_rows(proposal: dict[str, Any]) -> dict[str, list[dict[str
|
||||||
raise ValueError(f"replay receipt does not support proposal_type {proposal_type!r}")
|
raise ValueError(f"replay receipt does not support proposal_type {proposal_type!r}")
|
||||||
|
|
||||||
|
|
||||||
|
def _expected_v3_semantic_rows(
|
||||||
|
proposal: dict[str, Any], apply_payload: dict[str, Any]
|
||||||
|
) -> dict[str, list[dict[str, Any]]]:
|
||||||
|
proposal_id = _canonical_uuid(proposal.get("id"), path="proposal.id")
|
||||||
|
agent_id = apply_payload.get("agent_id")
|
||||||
|
polarity_to_role = {"supports": "grounds", "challenges": "contradicts", "illustrates": "illustrates"}
|
||||||
|
return {
|
||||||
|
"claims": [
|
||||||
|
{
|
||||||
|
"id": row.get("id"),
|
||||||
|
"owner_agent_id": row.get("owner_agent_id"),
|
||||||
|
"type": row.get("type"),
|
||||||
|
"text": row.get("proposition"),
|
||||||
|
"proposition": row.get("proposition"),
|
||||||
|
"body_markdown": row.get("body_markdown"),
|
||||||
|
"scope": row.get("scope"),
|
||||||
|
"access_scope": row.get("access_scope"),
|
||||||
|
"status": row.get("status", "active"),
|
||||||
|
"admission_state": "admitted",
|
||||||
|
"revision_criteria": row.get("revision_criteria"),
|
||||||
|
"revision_kind": "original",
|
||||||
|
"supersedes_id": None,
|
||||||
|
"confidence": None,
|
||||||
|
"tags": row.get("tags") or [],
|
||||||
|
"created_by": row.get("created_by", agent_id),
|
||||||
|
"superseded_by": None,
|
||||||
|
"accepted_by_proposal_id": proposal_id,
|
||||||
|
}
|
||||||
|
for row in apply_payload.get("claims") or []
|
||||||
|
],
|
||||||
|
"sources": [
|
||||||
|
{
|
||||||
|
"id": row.get("id"),
|
||||||
|
"source_type": row.get("source_type"),
|
||||||
|
"url": row.get("canonical_url"),
|
||||||
|
"storage_path": row.get("storage_uri"),
|
||||||
|
"excerpt": row.get("excerpt"),
|
||||||
|
"hash": row.get("content_hash"),
|
||||||
|
"created_by": row.get("created_by", agent_id),
|
||||||
|
"captured_at": row.get("captured_at"),
|
||||||
|
"canonical_url": row.get("canonical_url"),
|
||||||
|
"storage_uri": row.get("storage_uri"),
|
||||||
|
"content_hash": row.get("content_hash"),
|
||||||
|
"access_scope": row.get("access_scope"),
|
||||||
|
"ingestion_origin": row.get("ingestion_origin"),
|
||||||
|
"provenance_status": row.get("provenance_status"),
|
||||||
|
"source_class": row.get("source_class"),
|
||||||
|
"auto_promotion_policy": row.get("auto_promotion_policy"),
|
||||||
|
"accepted_by_proposal_id": proposal_id,
|
||||||
|
}
|
||||||
|
for row in apply_payload.get("sources") or []
|
||||||
|
],
|
||||||
|
"claim_evidence": [
|
||||||
|
{
|
||||||
|
"id": row.get("id"),
|
||||||
|
"claim_id": row.get("claim_id"),
|
||||||
|
"source_id": row.get("source_id"),
|
||||||
|
"role": polarity_to_role.get(row.get("polarity")),
|
||||||
|
"weight": None,
|
||||||
|
"created_by": row.get("created_by", agent_id),
|
||||||
|
"excerpt": row.get("excerpt"),
|
||||||
|
"locator_json": row.get("locator_json"),
|
||||||
|
"source_content_hash": row.get("source_content_hash"),
|
||||||
|
"rationale": row.get("rationale"),
|
||||||
|
"polarity": row.get("polarity"),
|
||||||
|
"accepted_from_proposal_id": proposal_id,
|
||||||
|
}
|
||||||
|
for row in apply_payload.get("evidence") or []
|
||||||
|
],
|
||||||
|
"claim_edges": [
|
||||||
|
{
|
||||||
|
"id": row.get("id"),
|
||||||
|
"from_claim": row.get("from_claim"),
|
||||||
|
"to_claim": row.get("to_claim"),
|
||||||
|
"edge_type": "requires" if row.get("relation_type") == "depends_on" else row.get("relation_type"),
|
||||||
|
"weight": None,
|
||||||
|
"created_by": row.get("created_by", agent_id),
|
||||||
|
"relation_type": row.get("relation_type"),
|
||||||
|
"rationale": row.get("rationale"),
|
||||||
|
"scope_or_conditions": row.get("scope_or_conditions"),
|
||||||
|
"accepted_by_proposal_id": proposal_id,
|
||||||
|
}
|
||||||
|
for row in apply_payload.get("edges") or []
|
||||||
|
],
|
||||||
|
"claim_evidence_assessments": [
|
||||||
|
{
|
||||||
|
"id": row.get("id"),
|
||||||
|
"claim_id": row.get("claim_id"),
|
||||||
|
"support_tier": row.get("support_tier"),
|
||||||
|
"challenge_tier": row.get("challenge_tier"),
|
||||||
|
"overall_state": row.get("overall_state"),
|
||||||
|
"rationale": row.get("rationale"),
|
||||||
|
"evidence_set_hash": row.get("evidence_set_hash"),
|
||||||
|
"accepted_by_proposal_id": proposal_id,
|
||||||
|
"supersedes_assessment_id": row.get("supersedes_assessment_id"),
|
||||||
|
}
|
||||||
|
for row in apply_payload.get("assessments") or []
|
||||||
|
],
|
||||||
|
"reasoning_tools": [],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def _assert_semantic_rows_match(proposal: dict[str, Any], rows: dict[str, list[dict[str, Any]]]) -> None:
|
def _assert_semantic_rows_match(proposal: dict[str, Any], rows: dict[str, list[dict[str, Any]]]) -> None:
|
||||||
expected = _expected_semantic_rows(proposal)
|
expected = _expected_semantic_rows(proposal)
|
||||||
for table, expected_rows in expected.items():
|
for table, expected_rows in expected.items():
|
||||||
fields = tuple(expected_rows[0]) if expected_rows else ()
|
fields = tuple(expected_rows[0]) if expected_rows else ()
|
||||||
|
expected_projected = [_semantic_projection(row, fields) for row in expected_rows]
|
||||||
actual_projected = [_semantic_projection(row, fields) for row in rows.get(table, [])]
|
actual_projected = [_semantic_projection(row, fields) for row in rows.get(table, [])]
|
||||||
if sorted(expected_rows, key=_canonical_json) != sorted(actual_projected, key=_canonical_json):
|
if sorted(expected_projected, key=_canonical_json) != sorted(actual_projected, key=_canonical_json):
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
f"row-level postflight semantic mismatch for {table}; "
|
f"row-level postflight semantic mismatch for {table}; "
|
||||||
"canonical values do not match the immutable strict payload"
|
"canonical values do not match the immutable strict payload"
|
||||||
|
|
@ -397,17 +626,21 @@ def build_replay_receipt(
|
||||||
_assert_semantic_rows_match(proposal, rows)
|
_assert_semantic_rows_match(proposal, rows)
|
||||||
|
|
||||||
proposal_envelope = {
|
proposal_envelope = {
|
||||||
"id": str(proposal.get("id")),
|
"id": _canonical_uuid(proposal.get("id"), path="proposal.id"),
|
||||||
"proposal_type": proposal.get("proposal_type"),
|
"proposal_type": proposal.get("proposal_type"),
|
||||||
"status": proposal.get("status"),
|
"status": proposal.get("status"),
|
||||||
"payload": proposal.get("payload"),
|
"payload": proposal.get("payload"),
|
||||||
"reviewed_by_handle": proposal.get("reviewed_by_handle"),
|
"reviewed_by_handle": proposal.get("reviewed_by_handle"),
|
||||||
"reviewed_by_agent_id": proposal.get("reviewed_by_agent_id"),
|
"reviewed_by_agent_id": _normalize_typed_value(
|
||||||
"reviewed_at": proposal.get("reviewed_at"),
|
"reviewed_by_agent_id", proposal.get("reviewed_by_agent_id"), path="proposal.reviewed_by_agent_id"
|
||||||
|
),
|
||||||
|
"reviewed_at": _normalize_typed_value("reviewed_at", proposal.get("reviewed_at"), path="proposal.reviewed_at"),
|
||||||
"review_note": proposal.get("review_note"),
|
"review_note": proposal.get("review_note"),
|
||||||
"applied_by_handle": proposal.get("applied_by_handle"),
|
"applied_by_handle": proposal.get("applied_by_handle"),
|
||||||
"applied_by_agent_id": proposal.get("applied_by_agent_id"),
|
"applied_by_agent_id": _normalize_typed_value(
|
||||||
"applied_at": proposal.get("applied_at"),
|
"applied_by_agent_id", proposal.get("applied_by_agent_id"), path="proposal.applied_by_agent_id"
|
||||||
|
),
|
||||||
|
"applied_at": _normalize_typed_value("applied_at", proposal.get("applied_at"), path="proposal.applied_at"),
|
||||||
}
|
}
|
||||||
row_hashes = {table: [_sha256_json(row) for row in table_rows] for table, table_rows in rows.items()}
|
row_hashes = {table: [_sha256_json(row) for row in table_rows] for table, table_rows in rows.items()}
|
||||||
table_hashes = {table: _sha256_json(table_rows) for table, table_rows in rows.items()}
|
table_hashes = {table: _sha256_json(table_rows) for table, table_rows in rows.items()}
|
||||||
|
|
@ -423,7 +656,9 @@ def build_replay_receipt(
|
||||||
return {
|
return {
|
||||||
**replay_material,
|
**replay_material,
|
||||||
"artifact": "kb_apply_replay_receipt",
|
"artifact": "kb_apply_replay_receipt",
|
||||||
"exported_at_utc": exported_at_utc or datetime.now(timezone.utc).isoformat(),
|
"exported_at_utc": canonical_utc_timestamp(
|
||||||
|
exported_at_utc or datetime.now(timezone.utc).isoformat(), path="exported_at_utc"
|
||||||
|
),
|
||||||
"expected_row_counts": expected,
|
"expected_row_counts": expected,
|
||||||
"actual_row_counts": actual,
|
"actual_row_counts": actual,
|
||||||
"checks": {
|
"checks": {
|
||||||
|
|
@ -464,7 +699,9 @@ def validate_replay_receipt(
|
||||||
apply_engine = receipt.get("apply_engine")
|
apply_engine = receipt.get("apply_engine")
|
||||||
if not isinstance(proposal, dict) or not isinstance(rows, dict) or not isinstance(apply_engine, dict):
|
if not isinstance(proposal, dict) or not isinstance(rows, dict) or not isinstance(apply_engine, dict):
|
||||||
raise ValueError("replay receipt is missing proposal, canonical rows, or apply-engine metadata")
|
raise ValueError("replay receipt is missing proposal, canonical rows, or apply-engine metadata")
|
||||||
if expected_proposal_id is not None and proposal.get("id") != expected_proposal_id:
|
if expected_proposal_id is not None and proposal.get("id") != _canonical_uuid(
|
||||||
|
expected_proposal_id, path="expected_proposal_id"
|
||||||
|
):
|
||||||
raise ValueError("replay receipt proposal id does not match")
|
raise ValueError("replay receipt proposal id does not match")
|
||||||
try:
|
try:
|
||||||
rebuilt = build_replay_receipt(
|
rebuilt = build_replay_receipt(
|
||||||
|
|
|
||||||
|
|
@ -18,6 +18,7 @@ a file for tests and UI development.
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
import hashlib
|
||||||
import json
|
import json
|
||||||
import sys
|
import sys
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
@ -30,6 +31,11 @@ import apply_proposal as ap # noqa: E402
|
||||||
import apply_worker as aw # noqa: E402
|
import apply_worker as aw # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
def canonical_sha256(value: Any) -> str:
|
||||||
|
encoded = json.dumps(value, ensure_ascii=True, separators=(",", ":"), sort_keys=True).encode("utf-8")
|
||||||
|
return hashlib.sha256(encoded).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
def _payload_list(payload: dict[str, Any], key: str) -> list[Any]:
|
def _payload_list(payload: dict[str, Any], key: str) -> list[Any]:
|
||||||
value = payload.get(key)
|
value = payload.get(key)
|
||||||
return value if isinstance(value, list) else []
|
return value if isinstance(value, list) else []
|
||||||
|
|
@ -47,7 +53,9 @@ def classify_proposal(proposal: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
|
||||||
status = proposal.get("status")
|
status = proposal.get("status")
|
||||||
proposal_type = proposal.get("proposal_type")
|
proposal_type = proposal.get("proposal_type")
|
||||||
has_apply_payload = "apply_payload" in payload
|
apply_payload = _payload_dict(payload, "apply_payload")
|
||||||
|
has_apply_payload = bool(apply_payload)
|
||||||
|
contract_version = apply_payload.get("contract_version", 2) if has_apply_payload else None
|
||||||
worker_supported_type = proposal_type in aw.WORKER_TYPES
|
worker_supported_type = proposal_type in aw.WORKER_TYPES
|
||||||
worker_applyable = (
|
worker_applyable = (
|
||||||
status == "approved" and worker_supported_type and has_apply_payload
|
status == "approved" and worker_supported_type and has_apply_payload
|
||||||
|
|
@ -73,6 +81,11 @@ def classify_proposal(proposal: dict[str, Any]) -> dict[str, Any]:
|
||||||
governance_standard = _payload_dict(payload, "proposed_governance_standard")
|
governance_standard = _payload_dict(payload, "proposed_governance_standard")
|
||||||
approval_request = _payload_dict(payload, "approval_request")
|
approval_request = _payload_dict(payload, "approval_request")
|
||||||
review_request = _payload_dict(payload, "review_request")
|
review_request = _payload_dict(payload, "review_request")
|
||||||
|
strict_rows = {
|
||||||
|
key: _payload_list(apply_payload, key)
|
||||||
|
for key in ("claims", "sources", "evidence", "edges", "assessments", "reasoning_tools")
|
||||||
|
}
|
||||||
|
strict_normalization_manifest = _payload_dict(apply_payload, "normalization_manifest")
|
||||||
|
|
||||||
missing_contract = []
|
missing_contract = []
|
||||||
if not has_apply_payload:
|
if not has_apply_payload:
|
||||||
|
|
@ -125,6 +138,7 @@ def classify_proposal(proposal: dict[str, Any]) -> dict[str, Any]:
|
||||||
},
|
},
|
||||||
"payload_summary": {
|
"payload_summary": {
|
||||||
"proposal_kind": payload.get("proposal_kind"),
|
"proposal_kind": payload.get("proposal_kind"),
|
||||||
|
"contract_version": contract_version,
|
||||||
"old_claim_id": _payload_dict(payload, "old_claim").get("id"),
|
"old_claim_id": _payload_dict(payload, "old_claim").get("id"),
|
||||||
"claim_candidate_count": len(claim_candidates),
|
"claim_candidate_count": len(claim_candidates),
|
||||||
"source_candidate_count": len(source_candidates),
|
"source_candidate_count": len(source_candidates),
|
||||||
|
|
@ -136,6 +150,13 @@ def classify_proposal(proposal: dict[str, Any]) -> dict[str, Any]:
|
||||||
"do_not_approve_yet": approval_request.get("do_not_approve_yet", []),
|
"do_not_approve_yet": approval_request.get("do_not_approve_yet", []),
|
||||||
"requested_decision": review_request.get("requested_decision"),
|
"requested_decision": review_request.get("requested_decision"),
|
||||||
"not_requested": review_request.get("not_requested"),
|
"not_requested": review_request.get("not_requested"),
|
||||||
|
"strict_row_counts": {key: len(rows) for key, rows in strict_rows.items()},
|
||||||
|
},
|
||||||
|
"strict_review_material": {
|
||||||
|
"apply_payload_sha256": canonical_sha256(apply_payload) if has_apply_payload else None,
|
||||||
|
"proposal_payload_sha256": canonical_sha256(payload),
|
||||||
|
"normalization_manifest": strict_normalization_manifest,
|
||||||
|
"rows": strict_rows,
|
||||||
},
|
},
|
||||||
"applyability": {
|
"applyability": {
|
||||||
"missing_contract": missing_contract,
|
"missing_contract": missing_contract,
|
||||||
|
|
@ -225,8 +246,10 @@ def render_markdown(packets: list[dict[str, Any]]) -> str:
|
||||||
f"- Worker applyable: {p['worker_applyable']}",
|
f"- Worker applyable: {p['worker_applyable']}",
|
||||||
f"- Reviewed by: {auth.get('reviewed_by_handle') or 'unreviewed'}",
|
f"- Reviewed by: {auth.get('reviewed_by_handle') or 'unreviewed'}",
|
||||||
f"- Proposal kind: {summary.get('proposal_kind') or 'unknown'}",
|
f"- Proposal kind: {summary.get('proposal_kind') or 'unknown'}",
|
||||||
|
f"- Contract version: {summary.get('contract_version') or 'none'}",
|
||||||
f"- Claim candidates: {summary['claim_candidate_count']}",
|
f"- Claim candidates: {summary['claim_candidate_count']}",
|
||||||
f"- Source candidates: {summary['source_candidate_count']}",
|
f"- Source candidates: {summary['source_candidate_count']}",
|
||||||
|
f"- Strict rows: {json.dumps(summary['strict_row_counts'], sort_keys=True)}",
|
||||||
f"- Missing contract: {', '.join(applyability['missing_contract']) or 'none'}",
|
f"- Missing contract: {', '.join(applyability['missing_contract']) or 'none'}",
|
||||||
f"- Next admin action: {p['next_admin_action']}",
|
f"- Next admin action: {p['next_admin_action']}",
|
||||||
]
|
]
|
||||||
|
|
|
||||||
|
|
@ -119,6 +119,11 @@ def _require_approved_bundle(proposal: dict[str, Any]) -> dict[str, Any]:
|
||||||
apply_payload = payload.get("apply_payload") if isinstance(payload, dict) else None
|
apply_payload = payload.get("apply_payload") if isinstance(payload, dict) else None
|
||||||
if not isinstance(apply_payload, dict):
|
if not isinstance(apply_payload, dict):
|
||||||
raise ValueError("approved proposal requires payload.apply_payload")
|
raise ValueError("approved proposal requires payload.apply_payload")
|
||||||
|
if apply_payload.get("contract_version") != 2:
|
||||||
|
raise ValueError(
|
||||||
|
"compensating rollback supports approve_claim contract_version 2 only; "
|
||||||
|
"V3 admitted rows are immutable and require an append-only correction"
|
||||||
|
)
|
||||||
# Reuse the audited builder as the strict contract validator.
|
# Reuse the audited builder as the strict contract validator.
|
||||||
ap.build_apply_sql(proposal, ap.SERVICE_AGENT_HANDLE)
|
ap.build_apply_sql(proposal, ap.SERVICE_AGENT_HANDLE)
|
||||||
return apply_payload
|
return apply_payload
|
||||||
|
|
|
||||||
|
|
@ -32,7 +32,7 @@ import apply_proposal as ap # noqa: E402
|
||||||
import proposal_apply_lifecycle as lifecycle # noqa: E402
|
import proposal_apply_lifecycle as lifecycle # noqa: E402
|
||||||
|
|
||||||
SAFE_DB_RE = re.compile(r"^[a-z][a-z0-9_]{1,62}$")
|
SAFE_DB_RE = re.compile(r"^[a-z][a-z0-9_]{1,62}$")
|
||||||
REVIEWER_HANDLE = "m3ta"
|
REVIEWER_HANDLE = "m3taversal"
|
||||||
REVIEW_NOTE = "Reviewed exact strict payload inside the disposable clone."
|
REVIEW_NOTE = "Reviewed exact strict payload inside the disposable clone."
|
||||||
DEFAULT_REVIEW_SECRETS_FILE = "/home/teleo/.hermes/profiles/leoclean/secrets/kb-review.env"
|
DEFAULT_REVIEW_SECRETS_FILE = "/home/teleo/.hermes/profiles/leoclean/secrets/kb-review.env"
|
||||||
SECURITY_DEFINER_DEPENDENCIES = {
|
SECURITY_DEFINER_DEPENDENCIES = {
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -30,7 +30,7 @@ READ_ROLE_SQL = ROOT / "ops" / "observatory_read_role.sql"
|
||||||
DATABASE = "teleo_canonical"
|
DATABASE = "teleo_canonical"
|
||||||
IAM_USER = "sa-observatory-read-adapter@teleo-501523.iam"
|
IAM_USER = "sa-observatory-read-adapter@teleo-501523.iam"
|
||||||
PARTICIPANT_HANDLE = "m3taversal"
|
PARTICIPANT_HANDLE = "m3taversal"
|
||||||
DEFAULT_IMAGE = "postgres:16-alpine"
|
DEFAULT_IMAGE = os.environ.get("LEOCLEAN_RUNTIME_POSTGRES_IMAGE", "postgres:16-alpine")
|
||||||
CANARY_LABEL = "leo-negative-permissions"
|
CANARY_LABEL = "leo-negative-permissions"
|
||||||
|
|
||||||
BOOTSTRAP_SQL = r"""
|
BOOTSTRAP_SQL = r"""
|
||||||
|
|
|
||||||
|
|
@ -4,6 +4,7 @@
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
import copy
|
||||||
import hashlib
|
import hashlib
|
||||||
import json
|
import json
|
||||||
import sys
|
import sys
|
||||||
|
|
@ -18,6 +19,9 @@ import apply_proposal as ap # noqa: E402
|
||||||
import kb_proposal_normalize as normalize # noqa: E402
|
import kb_proposal_normalize as normalize # noqa: E402
|
||||||
import run_leo_clone_bound_handler_checkpoint as bound # noqa: E402
|
import run_leo_clone_bound_handler_checkpoint as bound # noqa: E402
|
||||||
|
|
||||||
|
V3_BUNDLE_SCHEMA = "livingip.kbSourceProposalBundle.v3"
|
||||||
|
V3_MANIFEST_SCHEMA = "livingip.kbSourceExtractionManifest.v3"
|
||||||
|
|
||||||
|
|
||||||
def canonical_sha256(value: Any) -> str:
|
def canonical_sha256(value: Any) -> str:
|
||||||
encoded = json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode()
|
encoded = json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode()
|
||||||
|
|
@ -175,24 +179,116 @@ commit;
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
||||||
def stage_normalized_child(container: str, database: str, parent: dict[str, Any]) -> dict[str, Any]:
|
def prepare_compiled_v3_child(bundle: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
"""Revalidate one build-only V3 compiler bundle before disposable staging."""
|
||||||
|
|
||||||
|
if bundle.get("schema") != V3_BUNDLE_SCHEMA:
|
||||||
|
raise bound.CheckpointError(f"compiled bundle schema must equal {V3_BUNDLE_SCHEMA!r}")
|
||||||
|
if (
|
||||||
|
bundle.get("status") != "pending_review"
|
||||||
|
or bundle.get("build_only") is not True
|
||||||
|
or bundle.get("database_write_performed") is not False
|
||||||
|
or bundle.get("canonical_apply_performed") is not False
|
||||||
|
):
|
||||||
|
raise bound.CheckpointError("compiled V3 bundle is not build-only pending-review material")
|
||||||
|
|
||||||
|
hashes = bundle.get("hashes")
|
||||||
|
if not isinstance(hashes, dict):
|
||||||
|
raise bound.CheckpointError("compiled V3 bundle hashes must be an object")
|
||||||
|
expected_bundle_hash = hashes.get("bundle_content_sha256")
|
||||||
|
preimage = copy.deepcopy(bundle)
|
||||||
|
preimage_hashes = preimage.get("hashes")
|
||||||
|
if not isinstance(expected_bundle_hash, str) or not isinstance(preimage_hashes, dict):
|
||||||
|
raise bound.CheckpointError("compiled V3 bundle content hash is missing")
|
||||||
|
preimage_hashes.pop("bundle_content_sha256", None)
|
||||||
|
if canonical_sha256(preimage) != expected_bundle_hash:
|
||||||
|
raise bound.CheckpointError("compiled V3 bundle content hash does not match")
|
||||||
|
|
||||||
|
manifest = bundle.get("validated_manifest")
|
||||||
|
child = bundle.get("strict_child_proposal")
|
||||||
|
if not isinstance(manifest, dict) or manifest.get("schema") != V3_MANIFEST_SCHEMA:
|
||||||
|
raise bound.CheckpointError("compiled V3 bundle has no validated V3 manifest")
|
||||||
|
if not isinstance(child, dict):
|
||||||
|
raise bound.CheckpointError("compiled V3 bundle has no strict child proposal")
|
||||||
|
payload = child.get("payload")
|
||||||
|
apply_payload = payload.get("apply_payload") if isinstance(payload, dict) else None
|
||||||
|
if (
|
||||||
|
child.get("status") != "pending_review"
|
||||||
|
or child.get("proposal_type") != "approve_claim"
|
||||||
|
or not isinstance(apply_payload, dict)
|
||||||
|
or apply_payload.get("contract_version") != 3
|
||||||
|
):
|
||||||
|
raise bound.CheckpointError("compiled V3 child is not one pending-review approve_claim v3 proposal")
|
||||||
|
|
||||||
|
payload_hash = canonical_sha256(payload)
|
||||||
|
if child.get("payload_sha256") != payload_hash or hashes.get("strict_child_payload_sha256") != payload_hash:
|
||||||
|
raise bound.CheckpointError("compiled V3 child payload hash does not match")
|
||||||
|
if child.get("parent_packet_sha256") != hashes.get("parent_packet_sha256"):
|
||||||
|
raise bound.CheckpointError("compiled V3 parent packet hash does not match")
|
||||||
|
|
||||||
|
stage_preview = bundle.get("stage_preview")
|
||||||
|
stage_sql_hash = hashlib.sha256(build_stage_sql(child).encode("utf-8")).hexdigest()
|
||||||
|
if (
|
||||||
|
not isinstance(stage_preview, dict)
|
||||||
|
or stage_preview.get("target_table") != "kb_stage.kb_proposals"
|
||||||
|
or stage_preview.get("target_status") != "pending_review"
|
||||||
|
or stage_preview.get("executed") is not False
|
||||||
|
or stage_preview.get("stage_sql_sha256") != stage_sql_hash
|
||||||
|
):
|
||||||
|
raise bound.CheckpointError("compiled V3 stage preview does not match the exact child")
|
||||||
|
|
||||||
|
validation_snapshot = copy.deepcopy(child)
|
||||||
|
validation_snapshot.update(
|
||||||
|
{
|
||||||
|
"status": "approved",
|
||||||
|
"reviewed_by_handle": "v3-compiler-preflight",
|
||||||
|
"reviewed_by_agent_id": None,
|
||||||
|
"reviewed_at": "2000-01-01T00:00:00+00:00",
|
||||||
|
"review_note": "Build-only validation of exact compiled V3 material.",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
ap.build_apply_sql(validation_snapshot, ap.SERVICE_AGENT_HANDLE)
|
||||||
|
except (TypeError, ValueError) as exc:
|
||||||
|
raise bound.CheckpointError(f"compiled V3 child failed strict writer preflight: {exc}") from exc
|
||||||
|
return copy.deepcopy(child)
|
||||||
|
|
||||||
|
|
||||||
|
def _stage_child(container: str, database: str, child: dict[str, Any]) -> dict[str, Any]:
|
||||||
target = bound.container_identity(container)
|
target = bound.container_identity(container)
|
||||||
production = bound.container_identity(bound.PRODUCTION_CONTAINER)
|
try:
|
||||||
|
production = bound.container_identity(bound.PRODUCTION_CONTAINER)
|
||||||
|
except bound.CheckpointError as exc:
|
||||||
|
detail = str(exc).lower()
|
||||||
|
if "no such object" not in detail and "no such container" not in detail:
|
||||||
|
raise
|
||||||
|
production = {"id": None, "mount_sources": []}
|
||||||
bound.validate_disposable_target_identity(target, production)
|
bound.validate_disposable_target_identity(target, production)
|
||||||
if target["id"] == production["id"]:
|
if target["id"] == production["id"]:
|
||||||
raise bound.CheckpointError("normalized staging target resolves to production")
|
raise bound.CheckpointError("normalized staging target resolves to production")
|
||||||
child = prepare_normalized_child(parent)
|
|
||||||
result = bound._psql_json(str(target["id"]), database, build_stage_sql(child))
|
result = bound._psql_json(str(target["id"]), database, build_stage_sql(child))
|
||||||
if result.get("proposal_id") != child["id"] or result.get("status") != "pending_review":
|
if result.get("proposal_id") != child["id"] or result.get("status") != "pending_review":
|
||||||
raise bound.CheckpointError("normalized staging did not return the exact pending child")
|
raise bound.CheckpointError("normalized staging did not return the exact pending child")
|
||||||
return {"child": child, "database_receipt": result, "bound_container_id": str(target["id"])}
|
return {"child": child, "database_receipt": result, "bound_container_id": str(target["id"])}
|
||||||
|
|
||||||
|
|
||||||
|
def stage_normalized_child(container: str, database: str, parent: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
return _stage_child(container, database, prepare_normalized_child(parent))
|
||||||
|
|
||||||
|
|
||||||
|
def stage_compiled_v3_bundle(container: str, database: str, bundle: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
return _stage_child(container, database, prepare_compiled_v3_child(bundle))
|
||||||
|
|
||||||
|
|
||||||
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||||
parser = argparse.ArgumentParser(description=__doc__)
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
parser.add_argument("--container", required=True)
|
parser.add_argument("--container", required=True)
|
||||||
parser.add_argument("--db", required=True)
|
parser.add_argument("--db", required=True)
|
||||||
parser.add_argument("--proposal-file", required=True, type=Path)
|
source = parser.add_mutually_exclusive_group(required=True)
|
||||||
|
source.add_argument("--proposal-file", type=Path, help="rich V1/V2 proposal to normalize then stage")
|
||||||
|
source.add_argument(
|
||||||
|
"--compiled-v3-bundle", type=Path, help="build-only V3 compiler bundle to revalidate then stage"
|
||||||
|
)
|
||||||
return parser.parse_args(argv)
|
return parser.parse_args(argv)
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -201,10 +297,15 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
try:
|
try:
|
||||||
if args.container == bound.PRODUCTION_CONTAINER:
|
if args.container == bound.PRODUCTION_CONTAINER:
|
||||||
raise bound.CheckpointError("--container must be a disposable non-production target")
|
raise bound.CheckpointError("--container must be a disposable non-production target")
|
||||||
parent = json.loads(args.proposal_file.read_text(encoding="utf-8"))
|
input_path = args.compiled_v3_bundle or args.proposal_file
|
||||||
if not isinstance(parent, dict):
|
packet = json.loads(input_path.read_text(encoding="utf-8"))
|
||||||
raise bound.CheckpointError("--proposal-file must contain one JSON object")
|
if not isinstance(packet, dict):
|
||||||
result = stage_normalized_child(args.container, args.db, parent)
|
raise bound.CheckpointError("staging input must contain one JSON object")
|
||||||
|
result = (
|
||||||
|
stage_compiled_v3_bundle(args.container, args.db, packet)
|
||||||
|
if args.compiled_v3_bundle
|
||||||
|
else stage_normalized_child(args.container, args.db, packet)
|
||||||
|
)
|
||||||
except (OSError, json.JSONDecodeError, bound.CheckpointError, SystemExit, ValueError) as exc:
|
except (OSError, json.JSONDecodeError, bound.CheckpointError, SystemExit, ValueError) as exc:
|
||||||
print(json.dumps({"status": "rejected", "error": str(exc)}, sort_keys=True))
|
print(json.dumps({"status": "rejected", "error": str(exc)}, sort_keys=True))
|
||||||
return 2
|
return 2
|
||||||
|
|
@ -216,6 +317,9 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
"payload_sha256": result["child"]["payload_sha256"],
|
"payload_sha256": result["child"]["payload_sha256"],
|
||||||
"created": result["database_receipt"].get("created"),
|
"created": result["database_receipt"].get("created"),
|
||||||
"bound_container_id": result["bound_container_id"],
|
"bound_container_id": result["bound_container_id"],
|
||||||
|
"contract_version": (result["child"].get("payload") or {})
|
||||||
|
.get("apply_payload", {})
|
||||||
|
.get("contract_version", 2),
|
||||||
},
|
},
|
||||||
sort_keys=True,
|
sort_keys=True,
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -14,7 +14,11 @@ ExecStart=/opt/teleo-eval/pipeline/.venv/bin/python3 \
|
||||||
/opt/teleo-eval/workspaces/deploy-infra/scripts/apply_worker.py
|
/opt/teleo-eval/workspaces/deploy-infra/scripts/apply_worker.py
|
||||||
# Ships INERT: no --enable here, so each tick is report-only until an operator opts
|
# Ships INERT: no --enable here, so each tick is report-only until an operator opts
|
||||||
# in. Flip on (after the first manual apply is proven) by dropping an EnvironmentFile
|
# in. Flip on (after the first manual apply is proven) by dropping an EnvironmentFile
|
||||||
# with KB_APPLY_WORKER_ENABLED=1 and KB_APPLY_RENDER_CMD=... — no unit edit needed.
|
# with KB_APPLY_WORKER_ENABLED=1, KB_APPLY_WORKER_CONTRACT_VERSION=2|3,
|
||||||
|
# and KB_APPLY_RENDER_CMD=... -- no unit edit needed. Once the V3 database cutover
|
||||||
|
# is installed, a V2-configured tick skips retained historical V2 approvals without
|
||||||
|
# applying them; set version 3 before enabling new execution. Migrate or reject each
|
||||||
|
# retained V2 packet explicitly through review instead of deleting it from the queue.
|
||||||
EnvironmentFile=-/opt/teleo-eval/secrets/kb-apply-worker.env
|
EnvironmentFile=-/opt/teleo-eval/secrets/kb-apply-worker.env
|
||||||
StandardOutput=journal
|
StandardOutput=journal
|
||||||
StandardError=journal
|
StandardError=journal
|
||||||
|
|
|
||||||
11
tests/fixtures/teleo_corpus_rebuild/forgejo/project-record.md
vendored
Normal file
11
tests/fixtures/teleo_corpus_rebuild/forgejo/project-record.md
vendored
Normal file
|
|
@ -0,0 +1,11 @@
|
||||||
|
# Forgejo Project Record
|
||||||
|
|
||||||
|
The retained project record names the repository and keeps source provenance reviewable.
|
||||||
|
|
||||||
|
Repository: living-ip/teleo-infrastructure
|
||||||
|
|
||||||
|
## Review Boundary
|
||||||
|
|
||||||
|
Every extracted section remains a pending candidate until a separate reviewer approves it.
|
||||||
|
|
||||||
|
No local corpus rebuild may stage or apply canonical database rows.
|
||||||
2
tests/fixtures/teleo_corpus_rebuild/operator-notes.txt
vendored
Normal file
2
tests/fixtures/teleo_corpus_rebuild/operator-notes.txt
vendored
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
The local rebuild inventories exact UTF-8 bytes and source-root-relative paths.
|
||||||
|
Its receipt binds the source manifest, candidate index, extractor specification, and V3 proposal files.
|
||||||
|
|
@ -4,6 +4,8 @@ Pure tests: exercise the SQL builders, payload validation, and dispatch without
|
||||||
a live database. Runnable under pytest or standalone (`python3 tests/test_apply_proposal.py`).
|
a live database. Runnable under pytest or standalone (`python3 tests/test_apply_proposal.py`).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import re
|
||||||
import sys
|
import sys
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
@ -100,6 +102,26 @@ def test_revise_strategy_sql_shape():
|
||||||
assert "::jsonb" in sql
|
assert "::jsonb" in sql
|
||||||
|
|
||||||
|
|
||||||
|
def test_guarded_sql_canonicalizes_reviewed_timestamp_to_stable_utc_text():
|
||||||
|
payload = _revise_payload()["apply_payload"]
|
||||||
|
approval = _approval_meta("revise_strategy", payload)
|
||||||
|
approval["reviewed_at"] = "2026-07-10T02:00:00.1+02:00"
|
||||||
|
|
||||||
|
sql = ap.build_revise_strategy_sql(payload, "pid-1", "kb-apply", approval)
|
||||||
|
|
||||||
|
assert sql.count("2026-07-10T00:00:00.100000Z") == 2
|
||||||
|
assert "2026-07-10T02:00:00.1+02:00" not in sql
|
||||||
|
|
||||||
|
|
||||||
|
def test_guarded_sql_rejects_reviewed_timestamp_without_timezone():
|
||||||
|
payload = _revise_payload()["apply_payload"]
|
||||||
|
approval = _approval_meta("revise_strategy", payload)
|
||||||
|
approval["reviewed_at"] = "2026-07-10T00:00:00"
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="timezone-aware ISO-8601"):
|
||||||
|
ap.build_revise_strategy_sql(payload, "pid-1", "kb-apply", approval)
|
||||||
|
|
||||||
|
|
||||||
def test_revise_strategy_requires_agent_id():
|
def test_revise_strategy_requires_agent_id():
|
||||||
payload = _revise_payload()["apply_payload"]
|
payload = _revise_payload()["apply_payload"]
|
||||||
del payload["agent_id"]
|
del payload["agent_id"]
|
||||||
|
|
@ -240,6 +262,135 @@ def _approve_claim_bundle():
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _approve_claim_v3_bundle():
|
||||||
|
agent_id = "11111111-1111-4111-8111-111111111111"
|
||||||
|
source_b = "dddddddd-dddd-4ddd-8ddd-dddddddddddd"
|
||||||
|
evidence_a = {
|
||||||
|
"id": "eeeeeeee-eeee-4eee-8eee-eeeeeeeeeeee",
|
||||||
|
"claim_id": CLAIM_A,
|
||||||
|
"source_id": SOURCE_A,
|
||||||
|
"polarity": "supports",
|
||||||
|
"excerpt": "The primary record supports claim A.",
|
||||||
|
"locator_json": {"section": "A", "paragraph": 1},
|
||||||
|
"source_content_hash": "a" * 64,
|
||||||
|
"rationale": "The cited paragraph directly supports the bounded proposition.",
|
||||||
|
"created_by": agent_id,
|
||||||
|
}
|
||||||
|
evidence_b = {
|
||||||
|
"id": "ffffffff-ffff-4fff-8fff-ffffffffffff",
|
||||||
|
"claim_id": CLAIM_B,
|
||||||
|
"source_id": source_b,
|
||||||
|
"polarity": "challenges",
|
||||||
|
"excerpt": "The primary record challenges claim B.",
|
||||||
|
"locator_json": {"section": "B", "paragraph": 2},
|
||||||
|
"source_content_hash": "b" * 64,
|
||||||
|
"rationale": "The cited paragraph directly challenges the bounded proposition.",
|
||||||
|
"created_by": agent_id,
|
||||||
|
}
|
||||||
|
evidence = [evidence_a, evidence_b]
|
||||||
|
return {
|
||||||
|
"contract_version": 3,
|
||||||
|
"agent_id": agent_id,
|
||||||
|
"claims": [
|
||||||
|
{
|
||||||
|
"id": CLAIM_A,
|
||||||
|
"type": "empirical",
|
||||||
|
"proposition": "Claim A has a reproducible evidence chain.",
|
||||||
|
"body_markdown": "Claim A is bounded to the cited primary record and exact locator.",
|
||||||
|
"scope": "V3 apply-writer disposable verification.",
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"status": "active",
|
||||||
|
"revision_criteria": "Revise if the cited source or locator cannot be reproduced.",
|
||||||
|
"revision_kind": "original",
|
||||||
|
"supersedes_id": None,
|
||||||
|
"tags": ["v3", "apply-writer"],
|
||||||
|
"created_by": agent_id,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": CLAIM_B,
|
||||||
|
"type": "conceptual",
|
||||||
|
"proposition": "Claim B remains explicitly challengeable.",
|
||||||
|
"body_markdown": "Claim B records a conceptual object together with its challenge evidence.",
|
||||||
|
"scope": "V3 apply-writer disposable verification.",
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"status": "contested",
|
||||||
|
"revision_criteria": "Revise when the challenge is resolved or materially changes.",
|
||||||
|
"revision_kind": "original",
|
||||||
|
"supersedes_id": None,
|
||||||
|
"tags": ["v3", "challenge"],
|
||||||
|
"created_by": agent_id,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
"sources": [
|
||||||
|
{
|
||||||
|
"id": SOURCE_A,
|
||||||
|
"source_type": "paper",
|
||||||
|
"canonical_url": "https://example.test/v3/a",
|
||||||
|
"storage_uri": None,
|
||||||
|
"excerpt": "Primary source A.",
|
||||||
|
"content_hash": "a" * 64,
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"ingestion_origin": "operator_upload",
|
||||||
|
"provenance_status": "verified",
|
||||||
|
"source_class": "primary_record",
|
||||||
|
"auto_promotion_policy": "human_required",
|
||||||
|
"captured_at": "2026-07-18T10:00:00+00:00",
|
||||||
|
"created_by": agent_id,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": source_b,
|
||||||
|
"source_type": "observation",
|
||||||
|
"canonical_url": None,
|
||||||
|
"storage_uri": "gs://livingip-test/v3/source-b",
|
||||||
|
"excerpt": "Primary source B.",
|
||||||
|
"content_hash": "b" * 64,
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"ingestion_origin": "system_import",
|
||||||
|
"provenance_status": "verified",
|
||||||
|
"source_class": "primary_record",
|
||||||
|
"auto_promotion_policy": "human_required",
|
||||||
|
"captured_at": "2026-07-18T10:01:00+00:00",
|
||||||
|
"created_by": agent_id,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
"evidence": evidence,
|
||||||
|
"edges": [
|
||||||
|
{
|
||||||
|
"id": "12121212-1212-4121-8121-121212121212",
|
||||||
|
"from_claim": CLAIM_A,
|
||||||
|
"to_claim": CLAIM_B,
|
||||||
|
"relation_type": "challenges",
|
||||||
|
"rationale": "Claim A supplies a bounded reason to challenge claim B.",
|
||||||
|
"scope_or_conditions": "Only inside the disposable V3 verification scope.",
|
||||||
|
"created_by": agent_id,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"assessments": [
|
||||||
|
{
|
||||||
|
"id": "13131313-1313-4131-8131-131313131313",
|
||||||
|
"claim_id": CLAIM_A,
|
||||||
|
"support_tier": "strong",
|
||||||
|
"challenge_tier": "none",
|
||||||
|
"overall_state": "support_dominant",
|
||||||
|
"rationale": "One exact primary record supports the bounded empirical claim.",
|
||||||
|
"evidence_set_hash": ap._v3_evidence_set_hash(CLAIM_A, evidence),
|
||||||
|
"supersedes_assessment_id": None,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "14141414-1414-4141-8141-141414141414",
|
||||||
|
"claim_id": CLAIM_B,
|
||||||
|
"support_tier": None,
|
||||||
|
"challenge_tier": None,
|
||||||
|
"overall_state": "challenge_dominant",
|
||||||
|
"rationale": "The conceptual claim has an explicit challenge without numeric evidence tiers.",
|
||||||
|
"evidence_set_hash": ap._v3_evidence_set_hash(CLAIM_B, evidence),
|
||||||
|
"supersedes_assessment_id": None,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
"reasoning_tools": [],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def test_approve_claim_bundle_sql_is_atomic_and_row_verified():
|
def test_approve_claim_bundle_sql_is_atomic_and_row_verified():
|
||||||
payload = _approve_claim_bundle()
|
payload = _approve_claim_bundle()
|
||||||
sql = ap.build_approve_claim_sql(
|
sql = ap.build_approve_claim_sql(
|
||||||
|
|
@ -264,6 +415,233 @@ def test_approve_claim_bundle_sql_is_atomic_and_row_verified():
|
||||||
assert "kb_stage.finish_approved_proposal" in sql
|
assert "kb_stage.finish_approved_proposal" in sql
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_approve_claim_sql_writes_complete_epistemic_bundle() -> None:
|
||||||
|
payload = _approve_claim_v3_bundle()
|
||||||
|
proposal_id = "99999999-9999-4999-8999-999999999999"
|
||||||
|
sql = ap.build_approve_claim_sql(
|
||||||
|
payload,
|
||||||
|
proposal_id,
|
||||||
|
None,
|
||||||
|
_approval_meta("approve_claim", payload),
|
||||||
|
)
|
||||||
|
|
||||||
|
shared_lock = f"select pg_catalog.pg_advisory_lock_shared({ap.V3_CATALOG_DDL_ADVISORY_LOCK_KEY});"
|
||||||
|
shared_unlock = f"select pg_catalog.pg_advisory_unlock_shared({ap.V3_CATALOG_DDL_ADVISORY_LOCK_KEY});"
|
||||||
|
assert sql.startswith(f"{shared_lock}\nbegin;")
|
||||||
|
assert "set transaction isolation level serializable" in sql
|
||||||
|
assert sql.rstrip().endswith(shared_unlock)
|
||||||
|
assert sql.count(shared_lock) == 1
|
||||||
|
assert sql.count(shared_unlock) == 1
|
||||||
|
assert "insert into public.claims" in sql
|
||||||
|
assert "body_markdown" in sql
|
||||||
|
assert "accepted_by_proposal_id" in sql
|
||||||
|
assert "insert into public.sources" in sql
|
||||||
|
assert "content_hash" in sql
|
||||||
|
assert "insert into public.claim_evidence (" in sql
|
||||||
|
assert "locator_json" in sql
|
||||||
|
assert "insert into public.claim_edges (" in sql
|
||||||
|
assert "relation_type" in sql
|
||||||
|
assert "insert into public.claim_evidence_assessments" in sql
|
||||||
|
assert "evidence_set_hash" in sql
|
||||||
|
assert "insert into public.reasoning_tools" not in sql
|
||||||
|
expected_inserts = sum(len(payload[key]) for key in ("sources", "claims", "evidence", "edges", "assessments"))
|
||||||
|
assert sql.lower().count("on conflict do nothing") == expected_inserts
|
||||||
|
assert sql.index("do $v3_contract_guard$") < sql.index("kb_stage.assert_approved_proposal")
|
||||||
|
relation_locks = [
|
||||||
|
f"select kb_stage.{ap.V3_CATALOG_LOCK_FUNCTION}();",
|
||||||
|
"lock table public.claim_edges in row exclusive mode;",
|
||||||
|
"lock table public.claim_evidence in row exclusive mode;",
|
||||||
|
"lock table public.claim_evidence_assessments in row exclusive mode;",
|
||||||
|
"lock table public.claims in row exclusive mode;",
|
||||||
|
"lock table public.sources in row exclusive mode;",
|
||||||
|
]
|
||||||
|
ordered_catalog_guard = [
|
||||||
|
shared_lock,
|
||||||
|
"begin;",
|
||||||
|
"set transaction isolation level serializable;",
|
||||||
|
*relation_locks,
|
||||||
|
"do $v3_contract_guard$",
|
||||||
|
"commit;",
|
||||||
|
shared_unlock,
|
||||||
|
]
|
||||||
|
assert [sql.index(statement) for statement in ordered_catalog_guard] == sorted(
|
||||||
|
sql.index(statement) for statement in ordered_catalog_guard
|
||||||
|
)
|
||||||
|
assert sql.index("kb_stage.assert_approved_proposal") < sql.index("insert into public.sources")
|
||||||
|
assert sql.index("insert into public.claim_evidence_assessments") < sql.index("kb_stage.finish_approved_proposal")
|
||||||
|
assert sql.index("on conflict do nothing") < sql.index("do $verify$")
|
||||||
|
assert "set local timezone = 'UTC'" in sql
|
||||||
|
assert "apply_proposal: V3 catalog contract is %s; refusing mutating transaction" in sql
|
||||||
|
assert "and superseded_by is null" in sql
|
||||||
|
expected_creator_checks = sum(len(payload[key]) for key in ("sources", "claims", "evidence", "edges"))
|
||||||
|
assert sql.count("created_by is not distinct from") == expected_creator_checks
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_catalog_guard_requires_the_full_hardened_contract() -> None:
|
||||||
|
sql = ap.build_v3_contract_state_query()
|
||||||
|
|
||||||
|
assert ap.V3_CATALOG_DDL_ADVISORY_LOCK_KEY == 6072343533146485505
|
||||||
|
assert ap.V3_PROTECTED_RELATIONS == (
|
||||||
|
("kb_stage", "kb_proposals"),
|
||||||
|
("public", "claim_edges"),
|
||||||
|
("public", "claim_evidence"),
|
||||||
|
("public", "claim_evidence_assessments"),
|
||||||
|
("public", "claims"),
|
||||||
|
("public", "sources"),
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
tuple(sorted({(row["schema_name"], row["table_name"]) for row in ap.V3_EXPECTED_TRIGGER_CONTRACT}))
|
||||||
|
== ap.V3_PROTECTED_RELATIONS
|
||||||
|
)
|
||||||
|
assert len(ap.V3_EXPECTED_TRIGGER_CONTRACT) == 24
|
||||||
|
assert len({row["trigger_name"] for row in ap.V3_EXPECTED_TRIGGER_CONTRACT}) == 24
|
||||||
|
assert all(row["when_expression"] is None for row in ap.V3_EXPECTED_TRIGGER_CONTRACT)
|
||||||
|
assert "when not artifacts_present then 'V2'" in sql
|
||||||
|
assert "when functions_valid and catalog_lock_function_valid and triggers_valid" in sql
|
||||||
|
assert "and owner_role_valid and memberships_valid then 'V3'" in sql
|
||||||
|
assert "else 'INVALID'" in sql
|
||||||
|
assert "actual.tgenabled = 'O'" in sql
|
||||||
|
assert "actual.tgtype = expected.trigger_type" in sql
|
||||||
|
assert "actual.update_columns = expected.update_columns" in sql
|
||||||
|
assert "actual.when_expression is not distinct from expected.when_expression" in sql
|
||||||
|
assert "actual.argument_hex = expected.argument_hex" in sql
|
||||||
|
assert "actual.tgconstraint = 0::pg_catalog.oid" in sql
|
||||||
|
assert "actual.tgoldtable is null" in sql
|
||||||
|
assert "actual.tgnewtable is null" in sql
|
||||||
|
assert "actual.tgfoid" in sql
|
||||||
|
assert "owner_role.rolname = E'kb_gate_owner'" in sql
|
||||||
|
assert "actual.function_security_definer = expected.function_security_definer" in sql
|
||||||
|
assert "actual.function_source_sha256 = expected.function_source_sha256" in sql
|
||||||
|
assert f"procedure.proname = E'{ap.V3_CATALOG_LOCK_FUNCTION}'" in sql
|
||||||
|
assert ap.V3_CATALOG_LOCK_FUNCTION_SOURCE_SHA256 in sql
|
||||||
|
assert "and prorettype = 'pg_catalog.void'::pg_catalog.regtype" in sql
|
||||||
|
assert "and grantee.rolname = 'kb_apply'" in sql
|
||||||
|
assert "and execute_acl_exact" in sql
|
||||||
|
assert "acl.grantee <> procedure.proowner" in sql
|
||||||
|
assert "from pg_catalog.pg_auth_members membership" in sql
|
||||||
|
|
||||||
|
|
||||||
|
def test_migration_trigger_contract_and_catalog_queries_match_worker_builder() -> None:
|
||||||
|
migration = (REPO_ROOT / "ops" / "teleo_v3_epistemic_contract.sql").read_text(encoding="utf-8")
|
||||||
|
contract_match = re.search(r"\$trigger_contract\$\s*(\[.*?\])\s*\$trigger_contract\$", migration, re.DOTALL)
|
||||||
|
|
||||||
|
assert contract_match is not None
|
||||||
|
assert json.loads(contract_match.group(1)) == list(ap.V3_EXPECTED_TRIGGER_CONTRACT)
|
||||||
|
exclusive_lock = f"select pg_catalog.pg_advisory_xact_lock({ap.V3_CATALOG_DDL_ADVISORY_LOCK_KEY});"
|
||||||
|
assert migration.count(exclusive_lock) == 1
|
||||||
|
assert migration.index(exclusive_lock) < migration.index(
|
||||||
|
f"create or replace function kb_stage.{ap.V3_CATALOG_LOCK_FUNCTION}()"
|
||||||
|
)
|
||||||
|
assert migration.index(exclusive_lock) < migration.index(
|
||||||
|
"create or replace function kb_stage.teleo_v3_enforce_apply_contract_cutover()"
|
||||||
|
)
|
||||||
|
assert migration.index(exclusive_lock) < migration.index("create or replace trigger")
|
||||||
|
|
||||||
|
expected_query = ap.build_v3_contract_state_query(
|
||||||
|
into="v_contract_state",
|
||||||
|
expected_contract_expression="pg_catalog.current_setting('teleo.v3_expected_trigger_contract')",
|
||||||
|
)
|
||||||
|
for marker in ("STATE", "POSTFLIGHT"):
|
||||||
|
query_match = re.search(
|
||||||
|
rf"TELEO_V3_CONTRACT_{marker}_QUERY_BEGIN\n(.*?)\n\s*-- TELEO_V3_CONTRACT_{marker}_QUERY_END",
|
||||||
|
migration,
|
||||||
|
re.DOTALL,
|
||||||
|
)
|
||||||
|
assert query_match is not None
|
||||||
|
assert query_match.group(1).strip() == expected_query + ";"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("mutation", "error"),
|
||||||
|
[
|
||||||
|
(lambda payload: payload["claims"][0].update(body_markdown=""), "body_markdown"),
|
||||||
|
(lambda payload: payload["claims"][0].update(revision_kind="semantic"), "original claims only"),
|
||||||
|
(lambda payload: payload["sources"][0].update(content_hash="not-a-hash"), "SHA-256"),
|
||||||
|
(lambda payload: payload["evidence"][0].update(source_content_hash="b" * 64), "does not match"),
|
||||||
|
(lambda payload: payload["assessments"][0].update(evidence_set_hash="c" * 64), "evidence_set_hash"),
|
||||||
|
(
|
||||||
|
lambda payload: payload["assessments"][0].update(
|
||||||
|
supersedes_assessment_id="15151515-1515-4151-8151-151515151515"
|
||||||
|
),
|
||||||
|
"assessment supersession",
|
||||||
|
),
|
||||||
|
(lambda payload: payload.update(reasoning_tools=[{"id": CLAIM_A}]), "reasoning_tools"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_v3_approve_claim_rejects_incomplete_or_unbound_payloads(mutation, error) -> None:
|
||||||
|
payload = _approve_claim_v3_bundle()
|
||||||
|
mutation(payload)
|
||||||
|
with pytest.raises(ValueError, match=error):
|
||||||
|
ap.build_approve_claim_sql(
|
||||||
|
payload,
|
||||||
|
"99999999-9999-4999-8999-999999999999",
|
||||||
|
None,
|
||||||
|
_approval_meta("approve_claim", payload),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_approve_claim_preserves_exact_reviewed_text_for_replay_receipts() -> None:
|
||||||
|
payload = _approve_claim_v3_bundle()
|
||||||
|
payload["claims"][0]["body_markdown"] = " Exact reviewed body with intentional spacing. "
|
||||||
|
|
||||||
|
normalized = ap._normalize_v3_approve_claim(payload)
|
||||||
|
|
||||||
|
assert normalized["claims"][0]["body_markdown"] == payload["claims"][0]["body_markdown"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("captured_at", "expected"),
|
||||||
|
[
|
||||||
|
("2026-07-18T10:00:00Z", "2026-07-18T10:00:00.000000Z"),
|
||||||
|
("2026-07-18T12:00:00+02:00", "2026-07-18T10:00:00.000000Z"),
|
||||||
|
("2026-07-18T12:00:00.123400+02:00", "2026-07-18T10:00:00.123400Z"),
|
||||||
|
("2026-07-18T10:00:00.1Z", "2026-07-18T10:00:00.100000Z"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_v3_approve_claim_normalizes_capture_timestamp_to_fixed_utc(
|
||||||
|
captured_at: str,
|
||||||
|
expected: str,
|
||||||
|
) -> None:
|
||||||
|
payload = _approve_claim_v3_bundle()
|
||||||
|
payload["sources"][0]["captured_at"] = captured_at
|
||||||
|
|
||||||
|
normalized = ap._normalize_v3_approve_claim(payload)
|
||||||
|
|
||||||
|
assert normalized["sources"][0]["captured_at"] == expected
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"captured_at",
|
||||||
|
[
|
||||||
|
"2026-07-18 10:00:00+00:00",
|
||||||
|
"2026-07-18T10:00:00",
|
||||||
|
"2026-07-18T10:00:00.1234567Z",
|
||||||
|
"not-a-timestamp",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_v3_approve_claim_rejects_untyped_capture_timestamp(captured_at: str) -> None:
|
||||||
|
payload = _approve_claim_v3_bundle()
|
||||||
|
payload["sources"][0]["captured_at"] = captured_at
|
||||||
|
with pytest.raises(ValueError, match="timezone-aware ISO-8601"):
|
||||||
|
ap._normalize_v3_approve_claim(payload)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"spelling",
|
||||||
|
[
|
||||||
|
CLAIM_A.upper(),
|
||||||
|
"{" + CLAIM_A + "}",
|
||||||
|
CLAIM_A.replace("-", ""),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_v3_approve_claim_rejects_noncanonical_uuid_spellings(spelling: str) -> None:
|
||||||
|
payload = _approve_claim_v3_bundle()
|
||||||
|
payload["claims"][0]["id"] = spelling
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="canonical lowercase hyphenated UUID"):
|
||||||
|
ap._normalize_v3_approve_claim(payload)
|
||||||
|
|
||||||
|
|
||||||
def test_approve_claim_semantic_rows_accept_matches_and_reject_payload_mismatches():
|
def test_approve_claim_semantic_rows_accept_matches_and_reject_payload_mismatches():
|
||||||
payload = _approve_claim_bundle()
|
payload = _approve_claim_bundle()
|
||||||
sql = ap.build_approve_claim_sql(
|
sql = ap.build_approve_claim_sql(
|
||||||
|
|
@ -445,6 +823,72 @@ def test_build_apply_sql_requires_apply_payload():
|
||||||
ap.build_apply_sql(proposal, None)
|
ap.build_apply_sql(proposal, None)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("proposal", "error"),
|
||||||
|
[
|
||||||
|
({"proposal_type": "approve_claim", "payload": []}, "payload must be a JSON object"),
|
||||||
|
(
|
||||||
|
{"proposal_type": "approve_claim", "payload": {"apply_payload": []}},
|
||||||
|
"payload.apply_payload must be a JSON object",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
{
|
||||||
|
"proposal_type": "approve_claim",
|
||||||
|
"payload": {"apply_payload": {"contract_version": "3"}},
|
||||||
|
},
|
||||||
|
"must be integer 2 or 3",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
{
|
||||||
|
"proposal_type": "approve_claim",
|
||||||
|
"payload": {"apply_payload": {"contract_version": 3.0}},
|
||||||
|
},
|
||||||
|
"must be integer 2 or 3",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
{
|
||||||
|
"proposal_type": "approve_claim",
|
||||||
|
"payload": {"apply_payload": {"contract_version": True}},
|
||||||
|
},
|
||||||
|
"must be integer 2 or 3",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
{
|
||||||
|
"proposal_type": "add_edge",
|
||||||
|
"payload": {"apply_payload": {"contract_version": 3}},
|
||||||
|
},
|
||||||
|
"must omit contract_version",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
{
|
||||||
|
"proposal_type": "add_edge",
|
||||||
|
"payload": {"apply_payload": {"contract_version": None}},
|
||||||
|
},
|
||||||
|
"must omit contract_version",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_exact_contract_gate_rejects_malformed_and_hybrid_proposals(
|
||||||
|
proposal: dict,
|
||||||
|
error: str,
|
||||||
|
) -> None:
|
||||||
|
with pytest.raises(ValueError, match=error):
|
||||||
|
ap.proposal_contract_version(proposal)
|
||||||
|
|
||||||
|
|
||||||
|
def test_mutating_subprocess_contract_expectation_is_reasserted() -> None:
|
||||||
|
payload = _approve_claim_v3_bundle()
|
||||||
|
proposal = {
|
||||||
|
"id": "99999999-9999-4999-8999-999999999999",
|
||||||
|
"proposal_type": "approve_claim",
|
||||||
|
"payload": {"apply_payload": payload},
|
||||||
|
}
|
||||||
|
|
||||||
|
assert ap.assert_expected_contract_version(proposal, 3) == 3
|
||||||
|
with pytest.raises(ValueError, match="does not match expected version 2"):
|
||||||
|
ap.assert_expected_contract_version(proposal, 2)
|
||||||
|
|
||||||
|
|
||||||
def test_build_apply_sql_rejects_unsupported_type():
|
def test_build_apply_sql_rejects_unsupported_type():
|
||||||
proposal = {"id": "pid", "proposal_type": "reject_claim", "payload": {"apply_payload": {}}}
|
proposal = {"id": "pid", "proposal_type": "reject_claim", "payload": {"apply_payload": {}}}
|
||||||
with pytest.raises(ValueError):
|
with pytest.raises(ValueError):
|
||||||
|
|
|
||||||
|
|
@ -42,18 +42,27 @@ sys.path.insert(0, str(REPO_ROOT / "scripts"))
|
||||||
|
|
||||||
import apply_worker as w # noqa: E402
|
import apply_worker as w # noqa: E402
|
||||||
|
|
||||||
|
REAL_DETECT_INSTALLED_CONTRACT = w.detect_installed_contract
|
||||||
|
|
||||||
|
|
||||||
def _dependency_restore():
|
def _dependency_restore():
|
||||||
original_load_password = w.ap.load_password
|
original_load_password = w.ap.load_password
|
||||||
|
original_run_psql = w.ap.run_psql
|
||||||
original_load_failure_state = w.load_failure_state
|
original_load_failure_state = w.load_failure_state
|
||||||
original_fetch_candidates = w.fetch_candidates
|
original_fetch_candidates = w.fetch_candidates
|
||||||
|
original_detect_installed_contract = w.detect_installed_contract
|
||||||
original_apply_one = w.apply_one
|
original_apply_one = w.apply_one
|
||||||
original_render_one = w.render_one
|
original_render_one = w.render_one
|
||||||
original_subprocess_run = w.subprocess.run
|
original_subprocess_run = w.subprocess.run
|
||||||
|
w.detect_installed_contract = lambda args, _password: (
|
||||||
|
w.CONTRACT_STATE_V3 if args.contract_version == 3 else w.CONTRACT_STATE_V2
|
||||||
|
)
|
||||||
yield
|
yield
|
||||||
w.ap.load_password = original_load_password
|
w.ap.load_password = original_load_password
|
||||||
|
w.ap.run_psql = original_run_psql
|
||||||
w.load_failure_state = original_load_failure_state
|
w.load_failure_state = original_load_failure_state
|
||||||
w.fetch_candidates = original_fetch_candidates
|
w.fetch_candidates = original_fetch_candidates
|
||||||
|
w.detect_installed_contract = original_detect_installed_contract
|
||||||
w.apply_one = original_apply_one
|
w.apply_one = original_apply_one
|
||||||
w.render_one = original_render_one
|
w.render_one = original_render_one
|
||||||
w.subprocess.run = original_subprocess_run
|
w.subprocess.run = original_subprocess_run
|
||||||
|
|
@ -86,6 +95,80 @@ def test_candidate_query_requires_apply_payload():
|
||||||
assert "payload ? 'apply_payload'" in sql
|
assert "payload ? 'apply_payload'" in sql
|
||||||
|
|
||||||
|
|
||||||
|
def test_candidate_query_is_pinned_to_v2_by_default() -> None:
|
||||||
|
sql = w.build_candidate_query()
|
||||||
|
assert "contract_version' = '2'" in sql
|
||||||
|
assert "contract_version' = '3'" not in sql
|
||||||
|
|
||||||
|
|
||||||
|
def test_candidate_query_can_be_pinned_to_v3_without_selecting_v2() -> None:
|
||||||
|
sql = w.build_candidate_query(contract_version=3)
|
||||||
|
assert "contract_version' = '3'" in sql
|
||||||
|
assert "contract_version' = '2'" not in sql
|
||||||
|
assert "proposal_type = 'approve_claim'" in sql
|
||||||
|
assert "proposal_type in ('revise_strategy'" not in sql
|
||||||
|
assert "jsonb_typeof(payload) = 'object'" in sql
|
||||||
|
assert "jsonb_typeof(payload->'apply_payload') = 'object'" in sql
|
||||||
|
assert "jsonb_typeof(payload->'apply_payload'->'contract_version') = 'number'" in sql
|
||||||
|
|
||||||
|
|
||||||
|
def test_candidate_query_rejects_unknown_contract_version() -> None:
|
||||||
|
with pytest.raises(ValueError, match="contract_version"):
|
||||||
|
w.build_candidate_query(contract_version=4)
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_can_select_the_v3_worker_contract_explicitly() -> None:
|
||||||
|
assert w.parse_args(["--contract-version", "3"]).contract_version == 3
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_rejects_a_noncanonical_exact_proposal_uuid() -> None:
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
w.parse_args(["--proposal-id", "AAAAAAAA-AAAA-4AAA-8AAA-AAAAAAAAAAAA"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_candidate_query_can_select_one_exact_proposal_without_queue_fallback() -> None:
|
||||||
|
proposal_id = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"
|
||||||
|
|
||||||
|
sql = w.build_candidate_query(contract_version=3, proposal_id=proposal_id)
|
||||||
|
|
||||||
|
exact_clause = f"id = {w.ap.sql_literal(proposal_id)}::uuid"
|
||||||
|
assert exact_clause in sql
|
||||||
|
assert sql.index(exact_clause) < sql.index("limit 20")
|
||||||
|
|
||||||
|
|
||||||
|
def test_candidate_query_rejects_noncanonical_exact_proposal_id() -> None:
|
||||||
|
with pytest.raises(ValueError, match="canonical"):
|
||||||
|
w.build_candidate_query(contract_version=3, proposal_id="AAAAAAAA-AAAA-4AAA-8AAA-AAAAAAAAAAAA")
|
||||||
|
|
||||||
|
|
||||||
|
def test_installed_contract_detection_is_exact_and_tri_state() -> None:
|
||||||
|
seen: list[str] = []
|
||||||
|
w.ap.run_psql = lambda _args, sql, _password: seen.append(sql) or w.CONTRACT_STATE_V3
|
||||||
|
|
||||||
|
assert REAL_DETECT_INSTALLED_CONTRACT(_args(contract_version=3), "pw") == w.CONTRACT_STATE_V3
|
||||||
|
assert "then 'V2'" in seen[0]
|
||||||
|
assert "else 'INVALID'" in seen[0]
|
||||||
|
assert "actual.tgenabled = 'O'" in seen[0]
|
||||||
|
assert "actual.update_columns = expected.update_columns" in seen[0]
|
||||||
|
assert "actual.when_expression is not distinct from expected.when_expression" in seen[0]
|
||||||
|
assert "actual.argument_hex = expected.argument_hex" in seen[0]
|
||||||
|
assert "actual.tgfoid" in seen[0]
|
||||||
|
assert "owner_role.rolname = E'kb_gate_owner'" in seen[0]
|
||||||
|
assert "actual.function_security_definer = expected.function_security_definer" in seen[0]
|
||||||
|
assert "actual.function_source_sha256 = expected.function_source_sha256" in seen[0]
|
||||||
|
assert "search_path=pg_catalog, pg_temp" in seen[0]
|
||||||
|
assert "acl.grantee <> procedure.proowner" in seen[0]
|
||||||
|
assert "not owner_role.rolcanlogin" in seen[0]
|
||||||
|
assert "from pg_catalog.pg_auth_members membership" in seen[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_installed_contract_detection_rejects_unknown_readback() -> None:
|
||||||
|
w.ap.run_psql = lambda *_args, **_kwargs: "drifted"
|
||||||
|
|
||||||
|
with pytest.raises(RuntimeError, match="V2, V3, or INVALID"):
|
||||||
|
REAL_DETECT_INSTALLED_CONTRACT(_args(contract_version=3), "pw")
|
||||||
|
|
||||||
|
|
||||||
def test_candidate_query_lists_worker_types_only():
|
def test_candidate_query_lists_worker_types_only():
|
||||||
sql = w.build_candidate_query()
|
sql = w.build_candidate_query()
|
||||||
for t in w.WORKER_TYPES:
|
for t in w.WORKER_TYPES:
|
||||||
|
|
@ -116,6 +199,20 @@ def test_parse_candidates_skips_blank_lines():
|
||||||
assert rows[0]["id"] == "a"
|
assert rows[0]["id"] == "a"
|
||||||
|
|
||||||
|
|
||||||
|
def test_fetch_candidates_fails_closed_on_hybrid_query_result() -> None:
|
||||||
|
w.ap.run_psql = lambda *_args, **_kwargs: json.dumps(
|
||||||
|
{
|
||||||
|
"id": "hybrid",
|
||||||
|
"proposal_type": "add_edge",
|
||||||
|
"agent_id": None,
|
||||||
|
"contract_version": "3",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(RuntimeError, match="outside the exact worker contract"):
|
||||||
|
w.fetch_candidates(_args(contract_version=3), "pw", ())
|
||||||
|
|
||||||
|
|
||||||
# --- render hook ---------------------------------------------------------- #
|
# --- render hook ---------------------------------------------------------- #
|
||||||
def test_render_command_none_when_unset():
|
def test_render_command_none_when_unset():
|
||||||
assert w.build_render_command("", "agent-1") is None
|
assert w.build_render_command("", "agent-1") is None
|
||||||
|
|
@ -134,6 +231,8 @@ def test_render_command_expands_agent_id():
|
||||||
def _args(**over):
|
def _args(**over):
|
||||||
base = dict(
|
base = dict(
|
||||||
enable=False,
|
enable=False,
|
||||||
|
contract_version=2,
|
||||||
|
proposal_id=None,
|
||||||
limit=20,
|
limit=20,
|
||||||
max_per_tick=1,
|
max_per_tick=1,
|
||||||
max_attempts=3,
|
max_attempts=3,
|
||||||
|
|
@ -269,9 +368,28 @@ def test_apply_one_supplies_deterministic_private_receipt_path():
|
||||||
assert len(calls) == 1
|
assert len(calls) == 1
|
||||||
assert calls[0][calls[0].index("--receipt-dir") + 1] == str(expected.parent)
|
assert calls[0][calls[0].index("--receipt-dir") + 1] == str(expected.parent)
|
||||||
assert calls[0][calls[0].index("--receipt-out") + 1] == str(expected)
|
assert calls[0][calls[0].index("--receipt-out") + 1] == str(expected)
|
||||||
|
assert calls[0][calls[0].index("--expected-contract-version") + 1] == "2"
|
||||||
assert stat.S_IMODE(expected.stat().st_mode) == 0o600
|
assert stat.S_IMODE(expected.stat().st_mode) == 0o600
|
||||||
|
|
||||||
|
|
||||||
|
def test_apply_one_forwards_the_exact_fresh_preflight() -> None:
|
||||||
|
proposal_id = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"
|
||||||
|
receipt_dir = Path(tempfile.mkdtemp()) / "receipts"
|
||||||
|
preflight = Path(tempfile.mkdtemp()) / "fresh-preflight.json"
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def _run(cmd, **_kwargs):
|
||||||
|
calls.append(cmd)
|
||||||
|
receipt_path = Path(cmd[cmd.index("--receipt-out") + 1])
|
||||||
|
_write_valid_private_receipt(receipt_path, proposal_id)
|
||||||
|
return _completed(0, stdout='{"applied": true}')
|
||||||
|
|
||||||
|
w.subprocess.run = _run
|
||||||
|
w.apply_one(_args(receipt_dir=str(receipt_dir), fresh_preflight=preflight), proposal_id)
|
||||||
|
|
||||||
|
assert calls[0][calls[0].index("--fresh-preflight") + 1] == str(preflight)
|
||||||
|
|
||||||
|
|
||||||
def test_apply_one_recovers_postcommit_receipt_failure_without_reapplying():
|
def test_apply_one_recovers_postcommit_receipt_failure_without_reapplying():
|
||||||
proposal_id = "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb"
|
proposal_id = "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb"
|
||||||
receipt_dir = Path(tempfile.mkdtemp()) / "receipts"
|
receipt_dir = Path(tempfile.mkdtemp()) / "receipts"
|
||||||
|
|
@ -406,6 +524,61 @@ def test_postcommit_receipt_error_does_not_increment_apply_failure_count():
|
||||||
assert w.load_failure_state(path) == {}
|
assert w.load_failure_state(path) == {}
|
||||||
|
|
||||||
|
|
||||||
|
def test_installed_v3_contract_retains_and_skips_visible_v2_queue(capsys) -> None:
|
||||||
|
applied: list[str] = []
|
||||||
|
w.ap.load_password = lambda _f: "pw"
|
||||||
|
w.detect_installed_contract = lambda _args, _password: w.CONTRACT_STATE_V3
|
||||||
|
w.fetch_candidates = lambda _args, _password, excluded: [
|
||||||
|
{
|
||||||
|
"id": "legacy-v2",
|
||||||
|
"proposal_type": "add_edge",
|
||||||
|
"agent_id": None,
|
||||||
|
"contract_version": "2",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
w.apply_one = lambda _args, proposal_id: applied.append(proposal_id)
|
||||||
|
|
||||||
|
rc = w.run(_args(enable=True, contract_version=2))
|
||||||
|
|
||||||
|
captured = capsys.readouterr()
|
||||||
|
assert rc == 0
|
||||||
|
assert applied == []
|
||||||
|
assert "SKIP contract v2 execution" in captured.err
|
||||||
|
assert "remain review-visible" in captured.err
|
||||||
|
assert "migrate" in captured.err.lower()
|
||||||
|
assert "reject/cancel" in captured.err
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_worker_refuses_before_database_contract_install(capsys) -> None:
|
||||||
|
applied: list[str] = []
|
||||||
|
w.ap.load_password = lambda _f: "pw"
|
||||||
|
w.detect_installed_contract = lambda _args, _password: w.CONTRACT_STATE_V2
|
||||||
|
w.fetch_candidates = lambda *_args: (_ for _ in ()).throw(AssertionError("candidate selection must not run"))
|
||||||
|
w.apply_one = lambda _args, proposal_id: applied.append(proposal_id)
|
||||||
|
|
||||||
|
rc = w.run(_args(enable=True, contract_version=3))
|
||||||
|
|
||||||
|
assert rc == 1
|
||||||
|
assert applied == []
|
||||||
|
assert "V3 cutover is not installed" in capsys.readouterr().err
|
||||||
|
|
||||||
|
|
||||||
|
def test_invalid_contract_aborts_both_workers_before_candidate_selection(capsys) -> None:
|
||||||
|
selected: list[int] = []
|
||||||
|
applied: list[str] = []
|
||||||
|
w.ap.load_password = lambda _f: "pw"
|
||||||
|
w.detect_installed_contract = lambda _args, _password: w.CONTRACT_STATE_INVALID
|
||||||
|
w.fetch_candidates = lambda *_args: selected.append(1) or []
|
||||||
|
w.apply_one = lambda _args, proposal_id: applied.append(proposal_id)
|
||||||
|
|
||||||
|
for version in (2, 3):
|
||||||
|
assert w.run(_args(enable=True, contract_version=version)) == 1
|
||||||
|
|
||||||
|
assert selected == []
|
||||||
|
assert applied == []
|
||||||
|
assert capsys.readouterr().err.count("no candidate selection or writes attempted") == 2
|
||||||
|
|
||||||
|
|
||||||
def test_report_only_gate_does_not_apply(monkeypatch=None):
|
def test_report_only_gate_does_not_apply(monkeypatch=None):
|
||||||
calls = []
|
calls = []
|
||||||
w.ap.load_password = lambda _f: "pw"
|
w.ap.load_password = lambda _f: "pw"
|
||||||
|
|
@ -417,6 +590,45 @@ def test_report_only_gate_does_not_apply(monkeypatch=None):
|
||||||
assert calls == [] # disabled worker performs NO writes
|
assert calls == [] # disabled worker performs NO writes
|
||||||
|
|
||||||
|
|
||||||
|
def test_exact_proposal_selector_refuses_missing_target_without_applying(capsys) -> None:
|
||||||
|
proposal_id = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"
|
||||||
|
applied: list[str] = []
|
||||||
|
w.ap.load_password = lambda _f: "pw"
|
||||||
|
w.fetch_candidates = lambda *_args: []
|
||||||
|
w.apply_one = lambda _args, selected: applied.append(selected)
|
||||||
|
|
||||||
|
rc = w.run(_args(enable=True, contract_version=3, proposal_id=proposal_id))
|
||||||
|
|
||||||
|
captured = capsys.readouterr()
|
||||||
|
assert rc == 1
|
||||||
|
assert applied == []
|
||||||
|
assert proposal_id in captured.err
|
||||||
|
assert "no writes attempted" in captured.err
|
||||||
|
|
||||||
|
|
||||||
|
def test_required_exact_proposal_returns_distinct_missing_candidate_status(capsys) -> None:
|
||||||
|
proposal_id = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"
|
||||||
|
applied: list[str] = []
|
||||||
|
w.ap.load_password = lambda _f: "pw"
|
||||||
|
w.fetch_candidates = lambda *_args: []
|
||||||
|
w.apply_one = lambda _args, selected: applied.append(selected)
|
||||||
|
|
||||||
|
rc = w.run(
|
||||||
|
_args(
|
||||||
|
enable=True,
|
||||||
|
contract_version=3,
|
||||||
|
proposal_id=proposal_id,
|
||||||
|
require_candidate=True,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
captured = capsys.readouterr()
|
||||||
|
assert rc == 2
|
||||||
|
assert applied == []
|
||||||
|
assert proposal_id in captured.err
|
||||||
|
assert "no writes attempted" in captured.err
|
||||||
|
|
||||||
|
|
||||||
def test_enabled_worker_applies_and_renders():
|
def test_enabled_worker_applies_and_renders():
|
||||||
applied, rendered = [], []
|
applied, rendered = [], []
|
||||||
w.ap.load_password = lambda _f: "pw"
|
w.ap.load_password = lambda _f: "pw"
|
||||||
|
|
|
||||||
547
tests/test_build_v3_genesis_ledger_bundle.py
Normal file
547
tests/test_build_v3_genesis_ledger_bundle.py
Normal file
|
|
@ -0,0 +1,547 @@
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import stat
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from ops import build_v3_genesis_ledger_bundle as builder
|
||||||
|
from ops import run_local_genesis_ledger_rebuild as genesis_rebuild
|
||||||
|
from scripts import compile_kb_source_packet as compiler
|
||||||
|
|
||||||
|
REVIEWER_ID = "11111111-1111-4111-8111-111111111111"
|
||||||
|
SERVICE_ID = "44444444-4444-4444-8444-444444444444"
|
||||||
|
PRIVATE_MARKER = "PRIVATE-V3-MATERIAL-MUST-NOT-BE-PRINTED"
|
||||||
|
|
||||||
|
|
||||||
|
def _sha256(value: bytes) -> str:
|
||||||
|
return hashlib.sha256(value).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _write_manifest(path: Path) -> Path:
|
||||||
|
singleton_rows = [
|
||||||
|
{
|
||||||
|
"kind": "identity",
|
||||||
|
"database": "teleo_fixture",
|
||||||
|
"server_version_num": 160010,
|
||||||
|
"transaction_read_only": "on",
|
||||||
|
"server_address": None,
|
||||||
|
"server_port": None,
|
||||||
|
"ssl": False,
|
||||||
|
"ssl_version": None,
|
||||||
|
"database_collation": "C",
|
||||||
|
"database_ctype": "C",
|
||||||
|
},
|
||||||
|
{"kind": "schemas", "items": ["kb_stage", "public"]},
|
||||||
|
]
|
||||||
|
singleton_rows.extend(
|
||||||
|
{"kind": kind, "items": []}
|
||||||
|
for kind in sorted(genesis_rebuild.load_manifest.__globals__["SINGLETON_KINDS"] - {"identity", "schemas"})
|
||||||
|
)
|
||||||
|
rows = [
|
||||||
|
*singleton_rows,
|
||||||
|
{
|
||||||
|
"kind": "table",
|
||||||
|
"schema": "public",
|
||||||
|
"table": "claims",
|
||||||
|
"row_count": 1,
|
||||||
|
"rowset_md5": "f" * 32,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
path.write_text("".join(json.dumps(row, sort_keys=True) + "\n" for row in rows), encoding="utf-8")
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def _compiled_child(tmp_path: Path, *, salt: str) -> dict:
|
||||||
|
artifact_bytes = f"retained artifact {salt}\n".encode()
|
||||||
|
quote = f"The exact retained source {salt} requires review before canonical apply."
|
||||||
|
extracted_text = f"Header {salt}\n{quote}\n"
|
||||||
|
text_bytes = extracted_text.encode()
|
||||||
|
quote_bytes = quote.encode()
|
||||||
|
start = text_bytes.index(quote_bytes)
|
||||||
|
extractor_spec = f"fixture-extractor={salt}\n".encode()
|
||||||
|
artifact_sha256 = _sha256(artifact_bytes)
|
||||||
|
|
||||||
|
manifest = {
|
||||||
|
"schema": compiler.MANIFEST_SCHEMA_V3,
|
||||||
|
"artifact_sha256": artifact_sha256,
|
||||||
|
"extracted_text_sha256": _sha256(text_bytes),
|
||||||
|
"extractor": {
|
||||||
|
"name": "fixture-semantic-extractor",
|
||||||
|
"version": "3.1.0",
|
||||||
|
"spec_sha256": _sha256(extractor_spec),
|
||||||
|
},
|
||||||
|
"agent_id": REVIEWER_ID,
|
||||||
|
"source": {
|
||||||
|
"identity": f"document:sha256:{artifact_sha256}",
|
||||||
|
"source_key": f"source_{salt}",
|
||||||
|
"source_type": "article",
|
||||||
|
"title": f"Source {salt}",
|
||||||
|
"locator": f"artifact://sha256/{artifact_sha256}",
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"ingestion_origin": "operator_upload",
|
||||||
|
"provenance_status": "verified",
|
||||||
|
"source_class": "primary_record",
|
||||||
|
"auto_promotion_policy": "human_required",
|
||||||
|
"captured_at": "2026-07-18T10:00:00+00:00",
|
||||||
|
},
|
||||||
|
"claims": [
|
||||||
|
{
|
||||||
|
"claim_key": f"review_gate_{salt}",
|
||||||
|
"type": "empirical",
|
||||||
|
"proposition": f"Source {salt} remains non-canonical until guarded review and apply succeed.",
|
||||||
|
"body_markdown": f"The retained source {salt} explicitly preserves the review boundary.",
|
||||||
|
"scope": f"The exact retained source fixture {salt}.",
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"owner_agent_id": None,
|
||||||
|
"status": "active",
|
||||||
|
"revision_criteria": "Revise if the exact source bytes or review contract changes.",
|
||||||
|
"revision_kind": "original",
|
||||||
|
"supersedes_id": None,
|
||||||
|
"tags": ["genesis", "provenance", salt],
|
||||||
|
"evidence": [
|
||||||
|
{
|
||||||
|
"evidence_key": f"exact_quote_{salt}",
|
||||||
|
"quote": quote,
|
||||||
|
"polarity": "supports",
|
||||||
|
"locator": {
|
||||||
|
"unit": "utf8_bytes",
|
||||||
|
"start": start,
|
||||||
|
"end": start + len(quote_bytes),
|
||||||
|
},
|
||||||
|
"rationale": "The exact byte-bound sentence directly supports the bounded proposition.",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"assessment": {
|
||||||
|
"support_tier": "strong",
|
||||||
|
"challenge_tier": "none",
|
||||||
|
"overall_state": "support_dominant",
|
||||||
|
"rationale": "One exact verified primary-record sentence supports the claim.",
|
||||||
|
"supersedes_assessment_id": None,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
input_dir = tmp_path / f"compiler-{salt}"
|
||||||
|
input_dir.mkdir()
|
||||||
|
artifact = input_dir / "artifact.bin"
|
||||||
|
text = input_dir / "extracted.txt"
|
||||||
|
manifest_path = input_dir / "manifest.json"
|
||||||
|
extractor_path = input_dir / "extractor.spec"
|
||||||
|
artifact.write_bytes(artifact_bytes)
|
||||||
|
text.write_text(extracted_text, encoding="utf-8")
|
||||||
|
manifest_path.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||||
|
extractor_path.write_bytes(extractor_spec)
|
||||||
|
return compiler.compile_source_packet(artifact, text, manifest_path, extractor_path)["strict_child_proposal"]
|
||||||
|
|
||||||
|
|
||||||
|
def _applied_envelope(proposal: dict) -> dict:
|
||||||
|
fields = (
|
||||||
|
"id",
|
||||||
|
"proposal_type",
|
||||||
|
"status",
|
||||||
|
"payload",
|
||||||
|
"reviewed_by_handle",
|
||||||
|
"reviewed_by_agent_id",
|
||||||
|
"reviewed_at",
|
||||||
|
"review_note",
|
||||||
|
"applied_by_handle",
|
||||||
|
"applied_by_agent_id",
|
||||||
|
"applied_at",
|
||||||
|
)
|
||||||
|
return {field: proposal[field] for field in fields}
|
||||||
|
|
||||||
|
|
||||||
|
def _v3_material(tmp_path: Path, *, sequence: int, salt: str) -> dict:
|
||||||
|
child = _compiled_child(tmp_path, salt=salt)
|
||||||
|
approved = {
|
||||||
|
"id": child["id"],
|
||||||
|
"proposal_type": child["proposal_type"],
|
||||||
|
"status": "approved",
|
||||||
|
"proposed_by_handle": child["proposed_by_handle"],
|
||||||
|
"proposed_by_agent_id": child["proposed_by_agent_id"],
|
||||||
|
"channel": child["channel"],
|
||||||
|
"source_ref": child["source_ref"],
|
||||||
|
"rationale": PRIVATE_MARKER,
|
||||||
|
"payload": child["payload"],
|
||||||
|
"reviewed_by_handle": "m3taversal",
|
||||||
|
"reviewed_by_agent_id": REVIEWER_ID,
|
||||||
|
"reviewed_at": "2026-07-18T10:01:00.000000Z",
|
||||||
|
"review_note": "Reviewed the exact source-normalized V3 payload.",
|
||||||
|
"applied_by_handle": None,
|
||||||
|
"applied_by_agent_id": None,
|
||||||
|
"applied_at": None,
|
||||||
|
"created_at": "2026-07-18T10:00:30.000000Z",
|
||||||
|
"updated_at": "2026-07-18T10:01:00.000000Z",
|
||||||
|
}
|
||||||
|
applied = {
|
||||||
|
**approved,
|
||||||
|
"status": "applied",
|
||||||
|
"applied_by_handle": "kb-apply",
|
||||||
|
"applied_by_agent_id": SERVICE_ID,
|
||||||
|
"applied_at": "2026-07-18T10:02:00.000000Z",
|
||||||
|
"updated_at": "2026-07-18T10:02:00.000001Z",
|
||||||
|
}
|
||||||
|
approval = {
|
||||||
|
"proposal_id": approved["id"],
|
||||||
|
"proposal_type": approved["proposal_type"],
|
||||||
|
"payload": approved["payload"],
|
||||||
|
"reviewed_by_handle": approved["reviewed_by_handle"],
|
||||||
|
"reviewed_by_agent_id": approved["reviewed_by_agent_id"],
|
||||||
|
"reviewed_by_db_role": "kb_review",
|
||||||
|
"reviewed_at": approved["reviewed_at"],
|
||||||
|
"review_note": approved["review_note"],
|
||||||
|
}
|
||||||
|
|
||||||
|
proposal = _applied_envelope(applied)
|
||||||
|
apply_payload = proposal["payload"]["apply_payload"]
|
||||||
|
rows = genesis_rebuild.replay_receipt._expected_v3_semantic_rows(proposal, apply_payload)
|
||||||
|
created_at = "2026-07-18T10:01:30.000000Z"
|
||||||
|
for table, table_rows in rows.items():
|
||||||
|
for row in table_rows:
|
||||||
|
if table == "claims":
|
||||||
|
row.update(created_at=created_at, updated_at=created_at)
|
||||||
|
elif table in {
|
||||||
|
"sources",
|
||||||
|
"claim_evidence",
|
||||||
|
"claim_edges",
|
||||||
|
"claim_evidence_assessments",
|
||||||
|
}:
|
||||||
|
row["created_at"] = created_at
|
||||||
|
apply_sql = genesis_rebuild.apply_engine.build_apply_sql(proposal, applied["applied_by_handle"])
|
||||||
|
receipt = genesis_rebuild.replay_receipt.build_replay_receipt(
|
||||||
|
proposal,
|
||||||
|
rows,
|
||||||
|
apply_sql_sha256=genesis_rebuild.replay_receipt.sha256_text(apply_sql),
|
||||||
|
apply_sql_source="exact_executed_sql",
|
||||||
|
exported_at_utc="2026-07-18T10:03:00.000000Z",
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"artifact": genesis_rebuild.MATERIAL_ARTIFACT,
|
||||||
|
"contract_version": genesis_rebuild.MATERIAL_CONTRACT_VERSION,
|
||||||
|
"sequence": sequence,
|
||||||
|
"approved_proposal": approved,
|
||||||
|
"approval_snapshot": approval,
|
||||||
|
"applied_proposal": applied,
|
||||||
|
"replay_receipt": receipt,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _v2_material() -> dict:
|
||||||
|
proposal_id = "70000000-0000-4000-8000-000000000001"
|
||||||
|
claim_a = "70000000-0000-4000-8000-000000000002"
|
||||||
|
claim_b = "70000000-0000-4000-8000-000000000003"
|
||||||
|
payload = {"apply_payload": {"from_claim": claim_a, "to_claim": claim_b, "edge_type": "supports", "weight": 0.8}}
|
||||||
|
approved = {
|
||||||
|
"id": proposal_id,
|
||||||
|
"proposal_type": "add_edge",
|
||||||
|
"status": "approved",
|
||||||
|
"proposed_by_handle": "fixture",
|
||||||
|
"proposed_by_agent_id": REVIEWER_ID,
|
||||||
|
"channel": "fixture",
|
||||||
|
"source_ref": "private://fixture",
|
||||||
|
"rationale": PRIVATE_MARKER,
|
||||||
|
"payload": payload,
|
||||||
|
"reviewed_by_handle": "m3taversal",
|
||||||
|
"reviewed_by_agent_id": REVIEWER_ID,
|
||||||
|
"reviewed_at": "2026-07-18T10:01:00.000000Z",
|
||||||
|
"review_note": "Reviewed exact V2 fixture.",
|
||||||
|
"applied_by_handle": None,
|
||||||
|
"applied_by_agent_id": None,
|
||||||
|
"applied_at": None,
|
||||||
|
"created_at": "2026-07-18T10:00:30.000000Z",
|
||||||
|
"updated_at": "2026-07-18T10:01:00.000000Z",
|
||||||
|
}
|
||||||
|
applied = {
|
||||||
|
**approved,
|
||||||
|
"status": "applied",
|
||||||
|
"applied_by_handle": "kb-apply",
|
||||||
|
"applied_by_agent_id": SERVICE_ID,
|
||||||
|
"applied_at": "2026-07-18T10:02:00.000000Z",
|
||||||
|
"updated_at": "2026-07-18T10:02:00.000001Z",
|
||||||
|
}
|
||||||
|
approval = {
|
||||||
|
"proposal_id": proposal_id,
|
||||||
|
"proposal_type": "add_edge",
|
||||||
|
"payload": payload,
|
||||||
|
"reviewed_by_handle": "m3taversal",
|
||||||
|
"reviewed_by_agent_id": REVIEWER_ID,
|
||||||
|
"reviewed_by_db_role": "kb_review",
|
||||||
|
"reviewed_at": approved["reviewed_at"],
|
||||||
|
"review_note": approved["review_note"],
|
||||||
|
}
|
||||||
|
receipt = genesis_rebuild.replay_receipt.build_replay_receipt(
|
||||||
|
_applied_envelope(applied),
|
||||||
|
{
|
||||||
|
"claim_edges": [
|
||||||
|
{
|
||||||
|
"id": "70000000-0000-4000-8000-000000000004",
|
||||||
|
"from_claim": claim_a,
|
||||||
|
"to_claim": claim_b,
|
||||||
|
"edge_type": "supports",
|
||||||
|
"weight": 0.8,
|
||||||
|
"created_by": None,
|
||||||
|
"created_at": "2026-07-18T10:01:30.000000Z",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
apply_sql_sha256=genesis_rebuild.replay_receipt.sha256_text("exact V2 fixture SQL"),
|
||||||
|
apply_sql_source="exact_executed_sql",
|
||||||
|
exported_at_utc="2026-07-18T10:03:00.000000Z",
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"artifact": genesis_rebuild.MATERIAL_ARTIFACT,
|
||||||
|
"contract_version": genesis_rebuild.MATERIAL_CONTRACT_VERSION,
|
||||||
|
"sequence": 1,
|
||||||
|
"approved_proposal": approved,
|
||||||
|
"approval_snapshot": approval,
|
||||||
|
"applied_proposal": applied,
|
||||||
|
"replay_receipt": receipt,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _write_json(path: Path, value: object, *, private: bool = False) -> Path:
|
||||||
|
path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||||
|
if private:
|
||||||
|
path.chmod(0o600)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def bundle_inputs(tmp_path: Path) -> dict[str, object]:
|
||||||
|
dump = tmp_path / "genesis.dump"
|
||||||
|
dump.write_bytes(b"PGDMPfixture")
|
||||||
|
genesis_manifest = _write_manifest(tmp_path / "genesis-manifest.jsonl")
|
||||||
|
final_manifest = _write_manifest(tmp_path / "final-manifest.jsonl")
|
||||||
|
materials = [
|
||||||
|
_write_json(tmp_path / "material-0001.json", _v3_material(tmp_path, sequence=1, salt="alpha"), private=True),
|
||||||
|
_write_json(tmp_path / "material-0002.json", _v3_material(tmp_path, sequence=2, salt="beta"), private=True),
|
||||||
|
]
|
||||||
|
note = tmp_path / "review-note.txt"
|
||||||
|
note.write_text(f"Explicitly reviewed all source-backed entries. {PRIVATE_MARKER}\n", encoding="utf-8")
|
||||||
|
return {
|
||||||
|
"dump": dump,
|
||||||
|
"genesis_manifest": genesis_manifest,
|
||||||
|
"final_manifest": final_manifest,
|
||||||
|
"materials": materials,
|
||||||
|
"note": note,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _build_args(inputs: dict[str, object], ledger: Path, *, admission: Path | None = None) -> list[str]:
|
||||||
|
args = [
|
||||||
|
"--genesis-dump",
|
||||||
|
str(inputs["dump"]),
|
||||||
|
"--genesis-manifest",
|
||||||
|
str(inputs["genesis_manifest"]),
|
||||||
|
"--final-manifest",
|
||||||
|
str(inputs["final_manifest"]),
|
||||||
|
]
|
||||||
|
for material in inputs["materials"]:
|
||||||
|
args.extend(("--material", str(material)))
|
||||||
|
args.extend(("--output-ledger", str(ledger)))
|
||||||
|
if admission is not None:
|
||||||
|
args.extend(
|
||||||
|
(
|
||||||
|
"--emit-admission-manifest",
|
||||||
|
"--admission-output",
|
||||||
|
str(admission),
|
||||||
|
"--reviewed-by",
|
||||||
|
"m3taversal",
|
||||||
|
"--reviewed-by-agent-id",
|
||||||
|
REVIEWER_ID,
|
||||||
|
"--reviewed-at",
|
||||||
|
"2026-07-18T14:00:00+02:00",
|
||||||
|
"--review-note-file",
|
||||||
|
str(inputs["note"]),
|
||||||
|
"--admission-scope",
|
||||||
|
"collective_core",
|
||||||
|
"--admission-basis",
|
||||||
|
"forgejo_primary_source",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return args
|
||||||
|
|
||||||
|
|
||||||
|
def _verify_args(inputs: dict[str, object], ledger: Path, admission: Path | None = None) -> list[str]:
|
||||||
|
args = [
|
||||||
|
"--genesis-dump",
|
||||||
|
str(inputs["dump"]),
|
||||||
|
"--genesis-manifest",
|
||||||
|
str(inputs["genesis_manifest"]),
|
||||||
|
"--verify-ledger",
|
||||||
|
str(ledger),
|
||||||
|
"--ledger-sha256",
|
||||||
|
genesis_rebuild.canonical_rebuild.sha256_file(ledger),
|
||||||
|
]
|
||||||
|
if admission is not None:
|
||||||
|
args.extend(
|
||||||
|
(
|
||||||
|
"--admission-manifest",
|
||||||
|
str(admission),
|
||||||
|
"--admission-manifest-sha256",
|
||||||
|
genesis_rebuild.canonical_rebuild.sha256_file(admission),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return args
|
||||||
|
|
||||||
|
|
||||||
|
def test_deterministic_build_and_verify_every_pin_with_no_writes(
|
||||||
|
bundle_inputs: dict[str, object], tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||||
|
) -> None:
|
||||||
|
first_ledger = tmp_path / "ledger-first.json"
|
||||||
|
second_ledger = tmp_path / "ledger-second.json"
|
||||||
|
first_admission = tmp_path / "admission-first.json"
|
||||||
|
second_admission = tmp_path / "admission-second.json"
|
||||||
|
|
||||||
|
assert builder.main(_build_args(bundle_inputs, first_ledger, admission=first_admission)) == 0
|
||||||
|
first_output = capsys.readouterr()
|
||||||
|
assert builder.main(_build_args(bundle_inputs, second_ledger, admission=second_admission)) == 0
|
||||||
|
second_output = capsys.readouterr()
|
||||||
|
|
||||||
|
assert first_ledger.read_bytes() == second_ledger.read_bytes()
|
||||||
|
assert first_admission.read_bytes() == second_admission.read_bytes()
|
||||||
|
assert PRIVATE_MARKER not in first_output.out + first_output.err + second_output.out + second_output.err
|
||||||
|
before = {path: (path.read_bytes(), path.stat().st_mtime_ns) for path in (first_ledger, first_admission)}
|
||||||
|
|
||||||
|
assert builder.main(_verify_args(bundle_inputs, first_ledger, first_admission)) == 0
|
||||||
|
verification = json.loads(capsys.readouterr().out)
|
||||||
|
|
||||||
|
assert verification["action"] == "verify"
|
||||||
|
assert verification["entry_count"] == 2
|
||||||
|
assert verification["admission"]["review_metadata_provenance"] == builder.REVIEW_METADATA_PROVENANCE
|
||||||
|
assert verification["admission"]["cryptographically_authenticated"] is False
|
||||||
|
assert before == {path: (path.read_bytes(), path.stat().st_mtime_ns) for path in (first_ledger, first_admission)}
|
||||||
|
|
||||||
|
|
||||||
|
def test_reordered_materials_and_v2_material_are_refused(
|
||||||
|
bundle_inputs: dict[str, object], tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||||
|
) -> None:
|
||||||
|
reversed_inputs = {**bundle_inputs, "materials": list(reversed(bundle_inputs["materials"]))}
|
||||||
|
assert builder.main(_build_args(reversed_inputs, tmp_path / "reordered.json")) == 1
|
||||||
|
assert not (tmp_path / "reordered.json").exists()
|
||||||
|
assert json.loads(capsys.readouterr().err)["code"] == "invalid_exported_material"
|
||||||
|
|
||||||
|
v2_path = _write_json(tmp_path / "v2-material.json", _v2_material(), private=True)
|
||||||
|
v2_inputs = {**bundle_inputs, "materials": [v2_path]}
|
||||||
|
assert builder.main(_build_args(v2_inputs, tmp_path / "v2-ledger.json")) == 1
|
||||||
|
assert not (tmp_path / "v2-ledger.json").exists()
|
||||||
|
assert json.loads(capsys.readouterr().err)["code"] == "legacy_material_forbidden"
|
||||||
|
|
||||||
|
|
||||||
|
def test_hash_drift_engine_tamper_and_symlink_material_fail_closed(
|
||||||
|
bundle_inputs: dict[str, object], tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||||
|
) -> None:
|
||||||
|
ledger = tmp_path / "ledger.json"
|
||||||
|
assert builder.main(_build_args(bundle_inputs, ledger)) == 0
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
first_material = bundle_inputs["materials"][0]
|
||||||
|
first_material.write_bytes(first_material.read_bytes() + b" ")
|
||||||
|
assert builder.main(_verify_args(bundle_inputs, ledger)) == 1
|
||||||
|
assert json.loads(capsys.readouterr().err)["code"] == "genesis_preflight_refused"
|
||||||
|
|
||||||
|
first_material.write_bytes(first_material.read_bytes()[:-1])
|
||||||
|
tampered = json.loads(ledger.read_text(encoding="utf-8"))
|
||||||
|
tampered["engine"]["apply_engine_sha256"] = "0" * 64
|
||||||
|
_write_json(ledger, tampered, private=True)
|
||||||
|
assert builder.main(_verify_args(bundle_inputs, ledger)) == 1
|
||||||
|
assert json.loads(capsys.readouterr().err)["code"] == "genesis_preflight_refused"
|
||||||
|
|
||||||
|
symlink = tmp_path / "material-symlink.json"
|
||||||
|
symlink.symlink_to(bundle_inputs["materials"][0])
|
||||||
|
symlink_inputs = {**bundle_inputs, "materials": [symlink]}
|
||||||
|
assert builder.main(_build_args(symlink_inputs, tmp_path / "symlink-ledger.json")) == 1
|
||||||
|
assert json.loads(capsys.readouterr().err)["code"] == "unsafe_input"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("mutation", "expected_code"),
|
||||||
|
[
|
||||||
|
("missing_explicit_flag", "implicit_admission_forbidden"),
|
||||||
|
("missing_reviewer_id", "incomplete_admission_review"),
|
||||||
|
("alternate_reviewer", "invalid_reviewer_handle"),
|
||||||
|
("noncanonical_reviewer_id", "invalid_reviewer_identity"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_admission_requires_complete_exact_operator_review(
|
||||||
|
bundle_inputs: dict[str, object],
|
||||||
|
tmp_path: Path,
|
||||||
|
capsys: pytest.CaptureFixture[str],
|
||||||
|
mutation: str,
|
||||||
|
expected_code: str,
|
||||||
|
) -> None:
|
||||||
|
ledger = tmp_path / f"{mutation}-ledger.json"
|
||||||
|
admission = tmp_path / f"{mutation}-admission.json"
|
||||||
|
args = _build_args(bundle_inputs, ledger, admission=admission)
|
||||||
|
if mutation == "missing_explicit_flag":
|
||||||
|
args.remove("--emit-admission-manifest")
|
||||||
|
elif mutation == "missing_reviewer_id":
|
||||||
|
index = args.index("--reviewed-by-agent-id")
|
||||||
|
del args[index : index + 2]
|
||||||
|
elif mutation == "alternate_reviewer":
|
||||||
|
args[args.index("--reviewed-by") + 1] = "m3ta"
|
||||||
|
else:
|
||||||
|
args[args.index("--reviewed-by-agent-id") + 1] = "AAAAAAAA-AAAA-4AAA-8AAA-AAAAAAAAAAAA"
|
||||||
|
|
||||||
|
assert builder.main(args) == 1
|
||||||
|
output = capsys.readouterr()
|
||||||
|
assert json.loads(output.err)["code"] == expected_code
|
||||||
|
assert PRIVATE_MARKER not in output.out + output.err
|
||||||
|
assert not ledger.exists()
|
||||||
|
assert not admission.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_admission_always_selects_all_entries_and_partial_manifest_is_rejected(
|
||||||
|
bundle_inputs: dict[str, object], tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||||
|
) -> None:
|
||||||
|
ledger = tmp_path / "ledger.json"
|
||||||
|
admission = tmp_path / "admission.json"
|
||||||
|
assert builder.main(_build_args(bundle_inputs, ledger, admission=admission)) == 0
|
||||||
|
capsys.readouterr()
|
||||||
|
manifest = json.loads(admission.read_text(encoding="utf-8"))
|
||||||
|
assert [entry["sequence"] for entry in manifest["entries"]] == [1, 2]
|
||||||
|
|
||||||
|
manifest["entries"].pop()
|
||||||
|
_write_json(admission, manifest, private=True)
|
||||||
|
assert builder.main(_verify_args(bundle_inputs, ledger, admission)) == 1
|
||||||
|
assert json.loads(capsys.readouterr().err)["code"] == "genesis_preflight_refused"
|
||||||
|
|
||||||
|
|
||||||
|
def test_existing_outputs_are_never_overwritten_and_outputs_are_private(
|
||||||
|
bundle_inputs: dict[str, object], tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||||
|
) -> None:
|
||||||
|
existing = tmp_path / "existing-ledger.json"
|
||||||
|
existing.write_text("operator-owned\n", encoding="utf-8")
|
||||||
|
assert builder.main(_build_args(bundle_inputs, existing)) == 1
|
||||||
|
assert existing.read_text(encoding="utf-8") == "operator-owned\n"
|
||||||
|
assert json.loads(capsys.readouterr().err)["code"] == "existing_output"
|
||||||
|
|
||||||
|
ledger = tmp_path / "private-ledger.json"
|
||||||
|
admission = tmp_path / "private-admission.json"
|
||||||
|
assert builder.main(_build_args(bundle_inputs, ledger, admission=admission)) == 0
|
||||||
|
output = capsys.readouterr()
|
||||||
|
assert stat.S_IMODE(ledger.stat().st_mode) == 0o600
|
||||||
|
assert stat.S_IMODE(admission.stat().st_mode) == 0o600
|
||||||
|
assert ledger.stat().st_nlink == admission.stat().st_nlink == 1
|
||||||
|
assert PRIVATE_MARKER not in output.out + output.err
|
||||||
|
|
||||||
|
|
||||||
|
def test_material_tamper_is_redacted_and_publishes_nothing(
|
||||||
|
bundle_inputs: dict[str, object], tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||||
|
) -> None:
|
||||||
|
material_path = bundle_inputs["materials"][0]
|
||||||
|
material = json.loads(material_path.read_text(encoding="utf-8"))
|
||||||
|
material["approved_proposal"]["payload"]["private_secret"] = PRIVATE_MARKER
|
||||||
|
_write_json(material_path, material, private=True)
|
||||||
|
tampered_inputs = {**bundle_inputs, "materials": [material_path]}
|
||||||
|
ledger = tmp_path / "tampered-ledger.json"
|
||||||
|
|
||||||
|
assert builder.main(_build_args(tampered_inputs, ledger)) == 1
|
||||||
|
output = capsys.readouterr()
|
||||||
|
assert not ledger.exists()
|
||||||
|
assert PRIVATE_MARKER not in output.out + output.err
|
||||||
|
assert json.loads(output.err)["status"] == "refused"
|
||||||
|
|
@ -3,9 +3,12 @@ from __future__ import annotations
|
||||||
import copy
|
import copy
|
||||||
import hashlib
|
import hashlib
|
||||||
import json
|
import json
|
||||||
|
import os
|
||||||
import stat
|
import stat
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
|
import tarfile
|
||||||
|
import uuid
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
@ -24,6 +27,123 @@ EXTRACTED_TEXT = (
|
||||||
)
|
)
|
||||||
CLAIM_QUOTE = "A staged proposal remains non-canonical until a human review and guarded apply both succeed."
|
CLAIM_QUOTE = "A staged proposal remains non-canonical until a human review and guarded apply both succeed."
|
||||||
EVIDENCE_QUOTE = "Every proposed claim must retain an exact quote from the extracted source text."
|
EVIDENCE_QUOTE = "Every proposed claim must retain an exact quote from the extracted source text."
|
||||||
|
EXTRACTOR_SPEC_BYTES = (
|
||||||
|
b"livingip semantic extractor specification\n"
|
||||||
|
b"name=fixture-semantic-extractor\n"
|
||||||
|
b"version=3.1.0\n"
|
||||||
|
b"quotes=exact-utf8-byte-locators\n"
|
||||||
|
)
|
||||||
|
REPAIRED_WRITER_HEAD = "6fdaf19b96d027bc614e8ade0d475ba0954740b9"
|
||||||
|
V1_RENDERED_BUNDLE_SHA256 = "64795816f6b28e61df34cd00a62ffae12c7cc40bb26c475a56d47b4507d1bf73"
|
||||||
|
V2_RENDERED_BUNDLE_SHA256 = "e0a240371885a2c7fc714dd1b08099ef122d472e76ae1065542c1f305a27d816"
|
||||||
|
EXACT_WRITER_FILES = (
|
||||||
|
"scripts/apply_proposal.py",
|
||||||
|
"scripts/kb_apply_replay_receipt.py",
|
||||||
|
)
|
||||||
|
EXACT_WRITER_PROBE = r"""
|
||||||
|
import copy
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
scripts_dir = Path(sys.argv[1]).resolve()
|
||||||
|
child_path = Path(sys.argv[2]).resolve()
|
||||||
|
resolved_commit = sys.argv[3]
|
||||||
|
forbidden_checkout = Path(sys.argv[4]).resolve()
|
||||||
|
|
||||||
|
|
||||||
|
def canonical_bytes(value):
|
||||||
|
return json.dumps(value, ensure_ascii=True, separators=(",", ":"), sort_keys=True).encode("utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def sha256_bytes(value):
|
||||||
|
return hashlib.sha256(value).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
sys.path.insert(0, str(scripts_dir))
|
||||||
|
import apply_proposal as writer
|
||||||
|
import kb_apply_replay_receipt as replay_receipt
|
||||||
|
|
||||||
|
module_origins = {
|
||||||
|
"apply_proposal": str(Path(writer.__file__).resolve()),
|
||||||
|
"kb_apply_replay_receipt": str(Path(replay_receipt.__file__).resolve()),
|
||||||
|
}
|
||||||
|
expected_origins = {
|
||||||
|
"apply_proposal": str(scripts_dir / "apply_proposal.py"),
|
||||||
|
"kb_apply_replay_receipt": str(scripts_dir / "kb_apply_replay_receipt.py"),
|
||||||
|
}
|
||||||
|
assert module_origins == expected_origins
|
||||||
|
for origin in module_origins.values():
|
||||||
|
origin_path = Path(origin)
|
||||||
|
assert origin_path.is_relative_to(scripts_dir)
|
||||||
|
assert not origin_path.is_relative_to(forbidden_checkout)
|
||||||
|
|
||||||
|
|
||||||
|
def forbidden_process(*_args, **_kwargs):
|
||||||
|
raise AssertionError("exact-writer compatibility probe attempted process or database execution")
|
||||||
|
|
||||||
|
|
||||||
|
writer.subprocess.run = forbidden_process
|
||||||
|
raw_child = child_path.read_bytes()
|
||||||
|
child = json.loads(raw_child.decode("utf-8"))
|
||||||
|
payload_before = copy.deepcopy(child["payload"])
|
||||||
|
apply_payload = child["payload"]["apply_payload"]
|
||||||
|
apply_payload_before = canonical_bytes(apply_payload)
|
||||||
|
normalized = writer._normalize_v3_approve_claim(apply_payload)
|
||||||
|
|
||||||
|
roles = {"supports": "grounds", "challenges": "contradicts", "illustrates": "illustrates"}
|
||||||
|
expected_evidence = []
|
||||||
|
for row in apply_payload["evidence"]:
|
||||||
|
expected = copy.deepcopy(row)
|
||||||
|
expected["role"] = roles[row["polarity"]]
|
||||||
|
expected_evidence.append(expected)
|
||||||
|
assert normalized == {
|
||||||
|
"agent_id": apply_payload["agent_id"],
|
||||||
|
"claims": apply_payload["claims"],
|
||||||
|
"sources": apply_payload["sources"],
|
||||||
|
"evidence": expected_evidence,
|
||||||
|
"edges": apply_payload["edges"],
|
||||||
|
"assessments": apply_payload["assessments"],
|
||||||
|
}
|
||||||
|
assert normalized["sources"][0]["captured_at"] == apply_payload["sources"][0]["captured_at"]
|
||||||
|
assert apply_payload["sources"][0]["captured_at"].endswith(".000000Z")
|
||||||
|
assert canonical_bytes(apply_payload) == apply_payload_before
|
||||||
|
|
||||||
|
approved = copy.deepcopy(child)
|
||||||
|
approved.update(
|
||||||
|
status="approved",
|
||||||
|
reviewed_by_handle="source-compiler-v3-exact-writer-test",
|
||||||
|
reviewed_by_agent_id=None,
|
||||||
|
reviewed_at="2000-01-01T00:00:00.000000Z",
|
||||||
|
review_note="Build-only exact-writer compatibility validation.",
|
||||||
|
)
|
||||||
|
sql = writer.build_apply_sql(approved, writer.SERVICE_AGENT_HANDLE)
|
||||||
|
assert "begin;" in sql
|
||||||
|
assert approved["payload"] == payload_before
|
||||||
|
assert child["payload"] == payload_before
|
||||||
|
assert child_path.read_bytes() == raw_child
|
||||||
|
assert sha256_bytes(canonical_bytes(child["payload"])) == child["payload_sha256"]
|
||||||
|
|
||||||
|
receipt = {
|
||||||
|
"resolved_commit": resolved_commit,
|
||||||
|
"module_origins": module_origins,
|
||||||
|
"module_sha256": {
|
||||||
|
name: sha256_bytes(Path(origin).read_bytes()) for name, origin in module_origins.items()
|
||||||
|
},
|
||||||
|
"input_child_file_sha256": sha256_bytes(raw_child),
|
||||||
|
"strict_child_payload_sha256": child["payload_sha256"],
|
||||||
|
"apply_payload_sha256_before": sha256_bytes(apply_payload_before),
|
||||||
|
"apply_payload_sha256_after": sha256_bytes(canonical_bytes(apply_payload)),
|
||||||
|
"payload_file_bytes_preserved": child_path.read_bytes() == raw_child,
|
||||||
|
"payload_object_preserved": child["payload"] == payload_before,
|
||||||
|
"canonical_timestamp_equal": normalized["sources"][0]["captured_at"]
|
||||||
|
== apply_payload["sources"][0]["captured_at"],
|
||||||
|
"build_apply_sql_sha256": sha256_bytes(sql.encode("utf-8")),
|
||||||
|
"sql_executed": False,
|
||||||
|
}
|
||||||
|
print(json.dumps(receipt, ensure_ascii=True, sort_keys=True))
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
def _sha256(value: bytes) -> str:
|
def _sha256(value: bytes) -> str:
|
||||||
|
|
@ -59,6 +179,78 @@ def _manifest() -> dict:
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _byte_locator(quote: str, text: str = EXTRACTED_TEXT) -> dict:
|
||||||
|
text_bytes = text.encode("utf-8")
|
||||||
|
quote_bytes = quote.encode("utf-8")
|
||||||
|
start = text_bytes.index(quote_bytes)
|
||||||
|
return {"unit": "utf8_bytes", "start": start, "end": start + len(quote_bytes)}
|
||||||
|
|
||||||
|
|
||||||
|
def _manifest_v3(
|
||||||
|
*,
|
||||||
|
artifact_bytes: bytes = ARTIFACT_BYTES,
|
||||||
|
extracted_text: str = EXTRACTED_TEXT,
|
||||||
|
extractor_spec_bytes: bytes = EXTRACTOR_SPEC_BYTES,
|
||||||
|
) -> dict:
|
||||||
|
artifact_sha256 = _sha256(artifact_bytes)
|
||||||
|
return {
|
||||||
|
"schema": compiler.MANIFEST_SCHEMA_V3,
|
||||||
|
"artifact_sha256": artifact_sha256,
|
||||||
|
"extracted_text_sha256": _sha256(extracted_text.encode("utf-8")),
|
||||||
|
"extractor": {
|
||||||
|
"name": "fixture-semantic-extractor",
|
||||||
|
"version": "3.1.0",
|
||||||
|
"spec_sha256": _sha256(extractor_spec_bytes),
|
||||||
|
},
|
||||||
|
"agent_id": "11111111-1111-4111-8111-111111111111",
|
||||||
|
"source": {
|
||||||
|
"identity": f"document:sha256:{artifact_sha256}",
|
||||||
|
"source_key": "source_compiler_note_v3",
|
||||||
|
"source_type": "article",
|
||||||
|
"title": "Source packet compiler note",
|
||||||
|
"locator": f"artifact://sha256/{artifact_sha256}",
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"ingestion_origin": "operator_upload",
|
||||||
|
"provenance_status": "verified",
|
||||||
|
"source_class": "primary_record",
|
||||||
|
"auto_promotion_policy": "human_required",
|
||||||
|
"captured_at": "2026-07-18T10:00:00+00:00",
|
||||||
|
},
|
||||||
|
"claims": [
|
||||||
|
{
|
||||||
|
"claim_key": "review_gate_precedes_canonical_apply_v3",
|
||||||
|
"type": "empirical",
|
||||||
|
"proposition": "A staged proposal remains non-canonical until review and guarded apply succeed.",
|
||||||
|
"body_markdown": "The source explicitly separates staging from review and guarded canonical apply.",
|
||||||
|
"scope": "The retained source packet compiler note and its exact cited sentence.",
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"owner_agent_id": None,
|
||||||
|
"status": "active",
|
||||||
|
"revision_criteria": "Revise if the bound source sentence changes or no longer supports the proposition.",
|
||||||
|
"revision_kind": "original",
|
||||||
|
"supersedes_id": None,
|
||||||
|
"tags": ["review", "v3", "provenance"],
|
||||||
|
"evidence": [
|
||||||
|
{
|
||||||
|
"evidence_key": "review_gate_exact_sentence",
|
||||||
|
"quote": CLAIM_QUOTE,
|
||||||
|
"polarity": "supports",
|
||||||
|
"locator": _byte_locator(CLAIM_QUOTE, extracted_text),
|
||||||
|
"rationale": "The exact sentence directly supports the bounded proposition.",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"assessment": {
|
||||||
|
"support_tier": "strong",
|
||||||
|
"challenge_tier": "none",
|
||||||
|
"overall_state": "support_dominant",
|
||||||
|
"rationale": "One exact verified primary-record sentence directly supports the claim.",
|
||||||
|
"supersedes_assessment_id": None,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def _write_inputs(tmp_path: Path, manifest: dict | None = None, prefix: str = "input") -> tuple[Path, Path, Path]:
|
def _write_inputs(tmp_path: Path, manifest: dict | None = None, prefix: str = "input") -> tuple[Path, Path, Path]:
|
||||||
tmp_path.mkdir(parents=True, exist_ok=True)
|
tmp_path.mkdir(parents=True, exist_ok=True)
|
||||||
artifact = tmp_path / f"{prefix}.bin"
|
artifact = tmp_path / f"{prefix}.bin"
|
||||||
|
|
@ -75,6 +267,173 @@ def _compile(tmp_path: Path, manifest: dict | None = None) -> dict:
|
||||||
return compiler.compile_source_packet(*paths)
|
return compiler.compile_source_packet(*paths)
|
||||||
|
|
||||||
|
|
||||||
|
def _write_v3_inputs(
|
||||||
|
tmp_path: Path,
|
||||||
|
manifest: dict | None = None,
|
||||||
|
*,
|
||||||
|
artifact_bytes: bytes = ARTIFACT_BYTES,
|
||||||
|
extracted_text: str = EXTRACTED_TEXT,
|
||||||
|
extractor_spec_bytes: bytes = EXTRACTOR_SPEC_BYTES,
|
||||||
|
prefix: str = "v3-input",
|
||||||
|
) -> tuple[Path, Path, Path, Path]:
|
||||||
|
tmp_path.mkdir(parents=True, exist_ok=True)
|
||||||
|
artifact = tmp_path / f"{prefix}.bin"
|
||||||
|
text = tmp_path / f"{prefix}.txt"
|
||||||
|
manifest_path = tmp_path / f"{prefix}.json"
|
||||||
|
extractor_spec = tmp_path / f"{prefix}.extractor-spec"
|
||||||
|
artifact.write_bytes(artifact_bytes)
|
||||||
|
text.write_text(extracted_text, encoding="utf-8")
|
||||||
|
manifest_path.write_text(
|
||||||
|
json.dumps(
|
||||||
|
manifest
|
||||||
|
or _manifest_v3(
|
||||||
|
artifact_bytes=artifact_bytes,
|
||||||
|
extracted_text=extracted_text,
|
||||||
|
extractor_spec_bytes=extractor_spec_bytes,
|
||||||
|
),
|
||||||
|
indent=2,
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
extractor_spec.write_bytes(extractor_spec_bytes)
|
||||||
|
return artifact, text, manifest_path, extractor_spec
|
||||||
|
|
||||||
|
|
||||||
|
def _compile_v3(
|
||||||
|
tmp_path: Path,
|
||||||
|
manifest: dict | None = None,
|
||||||
|
*,
|
||||||
|
artifact_bytes: bytes = ARTIFACT_BYTES,
|
||||||
|
extracted_text: str = EXTRACTED_TEXT,
|
||||||
|
extractor_spec_bytes: bytes = EXTRACTOR_SPEC_BYTES,
|
||||||
|
) -> dict:
|
||||||
|
return compiler.compile_source_packet(
|
||||||
|
*_write_v3_inputs(
|
||||||
|
tmp_path,
|
||||||
|
manifest,
|
||||||
|
artifact_bytes=artifact_bytes,
|
||||||
|
extracted_text=extracted_text,
|
||||||
|
extractor_spec_bytes=extractor_spec_bytes,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _rendered_bundle_sha256(bundle: dict) -> str:
|
||||||
|
rendered = json.dumps(bundle, ensure_ascii=True, indent=2, sort_keys=True) + "\n"
|
||||||
|
return _sha256(rendered.encode("utf-8"))
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_exact_writer_commit(requested_head: str) -> str:
|
||||||
|
assert requested_head == REPAIRED_WRITER_HEAD, (
|
||||||
|
"KB_SOURCE_COMPILER_WRITER_COMPAT_HEAD must name the exact reviewed writer commit "
|
||||||
|
f"{REPAIRED_WRITER_HEAD}, got {requested_head!r}"
|
||||||
|
)
|
||||||
|
completed = subprocess.run(
|
||||||
|
["git", "rev-parse", "--verify", f"{requested_head}^{{commit}}"],
|
||||||
|
cwd=REPO_ROOT,
|
||||||
|
capture_output=True,
|
||||||
|
check=False,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
assert completed.returncode == 0, (
|
||||||
|
f"could not resolve exact writer commit {requested_head}: {completed.stderr.strip()}"
|
||||||
|
)
|
||||||
|
resolved = completed.stdout.strip()
|
||||||
|
assert resolved == requested_head, f"writer ref resolved to {resolved}, expected exact commit {requested_head}"
|
||||||
|
return resolved
|
||||||
|
|
||||||
|
|
||||||
|
def _git_file_sha256(commit: str, path: str) -> str:
|
||||||
|
completed = subprocess.run(
|
||||||
|
["git", "show", f"{commit}:{path}"],
|
||||||
|
cwd=REPO_ROOT,
|
||||||
|
capture_output=True,
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
assert completed.returncode == 0, f"could not read {path} from exact writer commit {commit}"
|
||||||
|
return _sha256(completed.stdout)
|
||||||
|
|
||||||
|
|
||||||
|
def _run_exact_writer_compatibility(child: dict, tmp_path: Path) -> dict:
|
||||||
|
tmp_path.mkdir(parents=True, exist_ok=True)
|
||||||
|
requested_head = os.environ.get("KB_SOURCE_COMPILER_WRITER_COMPAT_HEAD", REPAIRED_WRITER_HEAD)
|
||||||
|
resolved = _resolve_exact_writer_commit(requested_head)
|
||||||
|
writer_root = tmp_path / "exact-writer"
|
||||||
|
writer_root.mkdir()
|
||||||
|
archive_path = tmp_path / "exact-writer.tar"
|
||||||
|
archived = subprocess.run(
|
||||||
|
[
|
||||||
|
"git",
|
||||||
|
"archive",
|
||||||
|
"--format=tar",
|
||||||
|
f"--output={archive_path}",
|
||||||
|
resolved,
|
||||||
|
"--",
|
||||||
|
*EXACT_WRITER_FILES,
|
||||||
|
],
|
||||||
|
cwd=REPO_ROOT,
|
||||||
|
capture_output=True,
|
||||||
|
check=False,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
assert archived.returncode == 0, f"could not materialize exact writer commit {resolved}: {archived.stderr}"
|
||||||
|
with tarfile.open(archive_path, "r") as archive:
|
||||||
|
for member in archive.getmembers():
|
||||||
|
member_path = Path(member.name)
|
||||||
|
assert not member_path.is_absolute() and ".." not in member_path.parts
|
||||||
|
archive.extractall(writer_root)
|
||||||
|
|
||||||
|
writer_scripts = writer_root / "scripts"
|
||||||
|
expected_origins = {
|
||||||
|
"apply_proposal": str((writer_scripts / "apply_proposal.py").resolve()),
|
||||||
|
"kb_apply_replay_receipt": str((writer_scripts / "kb_apply_replay_receipt.py").resolve()),
|
||||||
|
}
|
||||||
|
expected_module_sha256 = {
|
||||||
|
"apply_proposal": _git_file_sha256(resolved, "scripts/apply_proposal.py"),
|
||||||
|
"kb_apply_replay_receipt": _git_file_sha256(resolved, "scripts/kb_apply_replay_receipt.py"),
|
||||||
|
}
|
||||||
|
child_path = tmp_path / "strict-child.json"
|
||||||
|
child_bytes = compiler.canonical_json_bytes(child)
|
||||||
|
child_path.write_bytes(child_bytes)
|
||||||
|
completed = subprocess.run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
"-I",
|
||||||
|
"-c",
|
||||||
|
EXACT_WRITER_PROBE,
|
||||||
|
str(writer_scripts),
|
||||||
|
str(child_path),
|
||||||
|
resolved,
|
||||||
|
str(REPO_ROOT),
|
||||||
|
],
|
||||||
|
cwd=tmp_path,
|
||||||
|
capture_output=True,
|
||||||
|
check=False,
|
||||||
|
text=True,
|
||||||
|
env={"PYTHONHASHSEED": "0"},
|
||||||
|
)
|
||||||
|
assert completed.returncode == 0, (
|
||||||
|
f"exact writer {resolved} compatibility execution failed\nstdout: {completed.stdout}\nstderr: {completed.stderr}"
|
||||||
|
)
|
||||||
|
receipt = json.loads(completed.stdout)
|
||||||
|
assert receipt["resolved_commit"] == resolved
|
||||||
|
assert receipt["module_origins"] == expected_origins
|
||||||
|
assert receipt["module_sha256"] == expected_module_sha256
|
||||||
|
assert receipt["input_child_file_sha256"] == _sha256(child_bytes)
|
||||||
|
assert receipt["strict_child_payload_sha256"] == child["payload_sha256"]
|
||||||
|
assert receipt["apply_payload_sha256_before"] == receipt["apply_payload_sha256_after"]
|
||||||
|
assert receipt["payload_file_bytes_preserved"] is True
|
||||||
|
assert receipt["payload_object_preserved"] is True
|
||||||
|
assert receipt["canonical_timestamp_equal"] is True
|
||||||
|
assert receipt["sql_executed"] is False
|
||||||
|
assert len(receipt["build_apply_sql_sha256"]) == 64
|
||||||
|
receipt["requested_commit"] = requested_head
|
||||||
|
receipt_path = tmp_path / "exact-writer-compatibility-receipt.json"
|
||||||
|
receipt_path.write_text(json.dumps(receipt, ensure_ascii=True, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||||
|
receipt["receipt_path"] = str(receipt_path)
|
||||||
|
return receipt
|
||||||
|
|
||||||
|
|
||||||
def test_compiles_deterministic_hash_bound_pending_review_bundle_without_db_write(
|
def test_compiles_deterministic_hash_bound_pending_review_bundle_without_db_write(
|
||||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
) -> None:
|
) -> None:
|
||||||
|
|
@ -162,6 +521,488 @@ def test_v2_manifest_carries_canonical_dedupe_candidates_into_review_packet(tmp_
|
||||||
assert staged_assessment == assessment
|
assert staged_assessment == assessment
|
||||||
|
|
||||||
|
|
||||||
|
def test_v1_v2_rendered_bundle_bytes_match_pre_repair_baselines(tmp_path: Path) -> None:
|
||||||
|
artifact, text, manifest_path = _write_inputs(tmp_path / "v1")
|
||||||
|
v1 = compiler.compile_source_packet(artifact, text, manifest_path)
|
||||||
|
|
||||||
|
v2_manifest = _manifest()
|
||||||
|
v2_manifest["schema"] = compiler.MANIFEST_SCHEMA_V2
|
||||||
|
v2_manifest["dedupe"] = {
|
||||||
|
"database_search_query": "test query",
|
||||||
|
"candidate_claims": [],
|
||||||
|
"duplicates": [],
|
||||||
|
}
|
||||||
|
artifact, text, manifest_path = _write_inputs(tmp_path / "v2", v2_manifest)
|
||||||
|
v2 = compiler.compile_source_packet(artifact, text, manifest_path)
|
||||||
|
|
||||||
|
assert _rendered_bundle_sha256(v1) == V1_RENDERED_BUNDLE_SHA256
|
||||||
|
assert _rendered_bundle_sha256(v2) == V2_RENDERED_BUNDLE_SHA256
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_compiles_exact_deterministic_pending_review_payload_without_db_write(
|
||||||
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
first_paths = _write_v3_inputs(tmp_path / "first", prefix="original-source")
|
||||||
|
|
||||||
|
def unexpected_db_call(*_args, **_kwargs):
|
||||||
|
raise AssertionError("V3 compiler attempted a database write")
|
||||||
|
|
||||||
|
monkeypatch.setattr(compiler.stage, "stage_normalized_child", unexpected_db_call)
|
||||||
|
monkeypatch.setattr(compiler.stage.bound, "_psql_json", unexpected_db_call)
|
||||||
|
first = compiler.compile_source_packet(*first_paths)
|
||||||
|
second = compiler.compile_source_packet(*_write_v3_inputs(tmp_path / "copied", prefix="copied-source"))
|
||||||
|
|
||||||
|
assert first == second
|
||||||
|
assert first["schema"] == compiler.BUNDLE_SCHEMA_V3
|
||||||
|
assert first["status"] == "pending_review"
|
||||||
|
assert first["build_only"] is True
|
||||||
|
assert first["database_write_performed"] is False
|
||||||
|
assert first["canonical_apply_performed"] is False
|
||||||
|
assert first["stage_preview"] == {
|
||||||
|
"target_table": "kb_stage.kb_proposals",
|
||||||
|
"target_status": "pending_review",
|
||||||
|
"executed": False,
|
||||||
|
"stage_sql_sha256": first["stage_preview"]["stage_sql_sha256"],
|
||||||
|
}
|
||||||
|
assert len(first["stage_preview"]["stage_sql_sha256"]) == 64
|
||||||
|
|
||||||
|
parent = first["parent_proposal"]
|
||||||
|
child = first["strict_child_proposal"]
|
||||||
|
stage_sql = compiler.stage.build_stage_sql(child)
|
||||||
|
assert stage_sql.count("insert into kb_stage.kb_proposals") == 1
|
||||||
|
assert "'pending_review'" in stage_sql
|
||||||
|
assert "public." not in stage_sql
|
||||||
|
assert first["stage_preview"]["stage_sql_sha256"] == _sha256(stage_sql.encode("utf-8"))
|
||||||
|
apply_payload = child["payload"]["apply_payload"]
|
||||||
|
assert parent["status"] == child["status"] == "pending_review"
|
||||||
|
assert child["proposal_type"] == "approve_claim"
|
||||||
|
assert child["payload_sha256"] == compiler.canonical_sha256(child["payload"])
|
||||||
|
assert apply_payload["contract_version"] == 3
|
||||||
|
assert apply_payload["source_proposal_id"] == parent["id"]
|
||||||
|
assert apply_payload["agent_id"] == _manifest_v3()["agent_id"]
|
||||||
|
assert apply_payload["edges"] == []
|
||||||
|
assert apply_payload["reasoning_tools"] == []
|
||||||
|
assert len(apply_payload["sources"]) == 1
|
||||||
|
assert len(apply_payload["claims"]) == 1
|
||||||
|
assert len(apply_payload["evidence"]) == 1
|
||||||
|
assert len(apply_payload["assessments"]) == 1
|
||||||
|
|
||||||
|
source = apply_payload["sources"][0]
|
||||||
|
assert source == {
|
||||||
|
"id": source["id"],
|
||||||
|
"source_type": "article",
|
||||||
|
"canonical_url": None,
|
||||||
|
"storage_uri": _manifest_v3()["source"]["locator"],
|
||||||
|
"excerpt": CLAIM_QUOTE,
|
||||||
|
"content_hash": _sha256(ARTIFACT_BYTES),
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"ingestion_origin": "operator_upload",
|
||||||
|
"provenance_status": "verified",
|
||||||
|
"source_class": "primary_record",
|
||||||
|
"auto_promotion_policy": "human_required",
|
||||||
|
"captured_at": "2026-07-18T10:00:00.000000Z",
|
||||||
|
"created_by": _manifest_v3()["agent_id"],
|
||||||
|
}
|
||||||
|
claim = apply_payload["claims"][0]
|
||||||
|
assert claim["proposition"] == _manifest_v3()["claims"][0]["proposition"]
|
||||||
|
assert claim["body_markdown"] == _manifest_v3()["claims"][0]["body_markdown"]
|
||||||
|
assert claim["scope"] == _manifest_v3()["claims"][0]["scope"]
|
||||||
|
assert claim["revision_criteria"] == _manifest_v3()["claims"][0]["revision_criteria"]
|
||||||
|
assert claim["revision_kind"] == "original"
|
||||||
|
assert claim["supersedes_id"] is None
|
||||||
|
assert claim["created_by"] == apply_payload["agent_id"]
|
||||||
|
|
||||||
|
evidence = apply_payload["evidence"][0]
|
||||||
|
assert evidence["claim_id"] == claim["id"]
|
||||||
|
assert evidence["source_id"] == source["id"]
|
||||||
|
assert evidence["excerpt"] == CLAIM_QUOTE
|
||||||
|
assert evidence["polarity"] == "supports"
|
||||||
|
assert evidence["source_content_hash"] == source["content_hash"]
|
||||||
|
locator = evidence["locator_json"]
|
||||||
|
assert locator["unit"] == "utf8_bytes"
|
||||||
|
assert EXTRACTED_TEXT.encode("utf-8")[locator["start"] : locator["end"]] == CLAIM_QUOTE.encode("utf-8")
|
||||||
|
assert locator["quote_sha256"] == _sha256(CLAIM_QUOTE.encode("utf-8"))
|
||||||
|
assert locator["artifact_sha256"] == _sha256(ARTIFACT_BYTES)
|
||||||
|
assert locator["extracted_text_sha256"] == _sha256(EXTRACTED_TEXT.encode("utf-8"))
|
||||||
|
assert locator["extractor"] == _manifest_v3()["extractor"]
|
||||||
|
|
||||||
|
assessment = apply_payload["assessments"][0]
|
||||||
|
assert assessment["claim_id"] == claim["id"]
|
||||||
|
assert assessment["evidence_set_hash"] == compiler.ap._v3_evidence_set_hash(claim["id"], [evidence])
|
||||||
|
assert assessment["supersedes_assessment_id"] is None
|
||||||
|
|
||||||
|
all_ids = [
|
||||||
|
parent["id"],
|
||||||
|
child["id"],
|
||||||
|
source["id"],
|
||||||
|
claim["id"],
|
||||||
|
evidence["id"],
|
||||||
|
assessment["id"],
|
||||||
|
]
|
||||||
|
assert len(all_ids) == len(set(all_ids))
|
||||||
|
assert all(value == str(uuid.UUID(value)) for value in all_ids)
|
||||||
|
identity = apply_payload["normalization_manifest"]["identity_material"]
|
||||||
|
assert identity == {
|
||||||
|
"artifact_sha256": _sha256(ARTIFACT_BYTES),
|
||||||
|
"extracted_text_sha256": _sha256(EXTRACTED_TEXT.encode("utf-8")),
|
||||||
|
"extractor": _manifest_v3()["extractor"],
|
||||||
|
"manifest_file_sha256": first["hashes"]["manifest_file_sha256"],
|
||||||
|
"source_identity": _manifest_v3()["source"]["identity"],
|
||||||
|
"source_locator": _manifest_v3()["source"]["locator"],
|
||||||
|
"normalized_candidate_content_sha256": first["hashes"]["normalized_candidate_content_sha256"],
|
||||||
|
}
|
||||||
|
assert apply_payload["normalization_manifest"]["semantic_extraction_is_retained_input"] is True
|
||||||
|
assert apply_payload["normalization_manifest"]["model_training_claim"] is False
|
||||||
|
assert "retained input" in apply_payload["normalization_manifest"]["semantic_extraction_statement"]
|
||||||
|
assert first["hashes"]["extractor_spec_sha256"] == _sha256(EXTRACTOR_SPEC_BYTES)
|
||||||
|
assert first["compiler_contract"]["writer_apply_invoked"] is False
|
||||||
|
|
||||||
|
without_bundle_hash = copy.deepcopy(first)
|
||||||
|
bundle_hash = without_bundle_hash["hashes"].pop("bundle_content_sha256")
|
||||||
|
assert bundle_hash == compiler.canonical_sha256(without_bundle_hash)
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_exact_writer_commit_preserves_payload_and_builds_apply_sql(tmp_path: Path) -> None:
|
||||||
|
bundle = _compile_v3(tmp_path)
|
||||||
|
apply_payload = bundle["strict_child_proposal"]["payload"]["apply_payload"]
|
||||||
|
assert apply_payload["sources"][0]["captured_at"] == "2026-07-18T10:00:00.000000Z"
|
||||||
|
receipt = _run_exact_writer_compatibility(bundle["strict_child_proposal"], tmp_path / "writer-proof")
|
||||||
|
|
||||||
|
assert receipt["resolved_commit"] == REPAIRED_WRITER_HEAD
|
||||||
|
assert receipt["apply_payload_sha256_before"] == receipt["apply_payload_sha256_after"]
|
||||||
|
assert Path(receipt["receipt_path"]).is_file()
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_exact_writer_gate_cannot_fall_back_to_branch_local_apply_module(
|
||||||
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
bundle = _compile_v3(tmp_path / "payload")
|
||||||
|
branch_local_origin = str(Path(compiler.ap.__file__).resolve())
|
||||||
|
|
||||||
|
def forbidden_branch_local_writer(*_args, **_kwargs):
|
||||||
|
raise AssertionError("exact writer gate used branch-local compiler.ap")
|
||||||
|
|
||||||
|
monkeypatch.setattr(compiler.ap, "_normalize_v3_approve_claim", forbidden_branch_local_writer)
|
||||||
|
monkeypatch.setattr(compiler.ap, "build_apply_sql", forbidden_branch_local_writer)
|
||||||
|
receipt = _run_exact_writer_compatibility(bundle["strict_child_proposal"], tmp_path / "writer-proof")
|
||||||
|
|
||||||
|
assert branch_local_origin not in receipt["module_origins"].values()
|
||||||
|
assert all(Path(origin).is_relative_to(tmp_path) for origin in receipt["module_origins"].values())
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_cli_rerun_is_byte_identical_and_emits_only_public_status(tmp_path: Path) -> None:
|
||||||
|
private_root = tmp_path / "private-v3-source"
|
||||||
|
artifact, text, manifest_path, extractor_spec = _write_v3_inputs(private_root)
|
||||||
|
outputs = [private_root / "compiled" / "first.json", private_root / "compiled" / "second.json"]
|
||||||
|
completed = []
|
||||||
|
for output in outputs:
|
||||||
|
completed.append(
|
||||||
|
subprocess.run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
str(REPO_ROOT / "scripts" / "compile_kb_source_packet.py"),
|
||||||
|
"--artifact",
|
||||||
|
str(artifact),
|
||||||
|
"--text",
|
||||||
|
str(text),
|
||||||
|
"--manifest",
|
||||||
|
str(manifest_path),
|
||||||
|
"--extractor-spec",
|
||||||
|
str(extractor_spec),
|
||||||
|
"--output",
|
||||||
|
str(output),
|
||||||
|
],
|
||||||
|
capture_output=True,
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
assert [item.returncode for item in completed] == [0, 0]
|
||||||
|
assert [item.stderr for item in completed] == [b"", b""]
|
||||||
|
assert completed[0].stdout == completed[1].stdout
|
||||||
|
assert outputs[0].read_bytes() == outputs[1].read_bytes()
|
||||||
|
assert json.loads(outputs[0].read_text(encoding="utf-8"))["schema"] == compiler.BUNDLE_SCHEMA_V3
|
||||||
|
public_status = json.loads(completed[0].stdout.decode("utf-8"))
|
||||||
|
assert public_status["status"] == "pending_review"
|
||||||
|
assert public_status["database_write_performed"] is False
|
||||||
|
assert public_status["canonical_apply_performed"] is False
|
||||||
|
assert EXTRACTED_TEXT.encode("utf-8") not in completed[0].stdout
|
||||||
|
assert str(private_root).encode("utf-8") not in completed[0].stdout
|
||||||
|
assert all(stat.S_IMODE(output.stat().st_mode) == 0o600 for output in outputs)
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_stale_artifact_fails_closed_and_rehashed_artifact_changes_identity(tmp_path: Path) -> None:
|
||||||
|
original = _compile_v3(tmp_path / "original")
|
||||||
|
mutated_bytes = ARTIFACT_BYTES + b"mutated"
|
||||||
|
stale_manifest = _manifest_v3()
|
||||||
|
with pytest.raises(compiler.CompilerError, match="artifact SHA-256 does not match"):
|
||||||
|
_compile_v3(tmp_path / "stale", stale_manifest, artifact_bytes=mutated_bytes)
|
||||||
|
|
||||||
|
rehashed_manifest = _manifest_v3(artifact_bytes=mutated_bytes)
|
||||||
|
stale_identity = copy.deepcopy(rehashed_manifest)
|
||||||
|
stale_identity["source"]["identity"] = _manifest_v3()["source"]["identity"]
|
||||||
|
with pytest.raises(compiler.CompilerError, match="content-addressed identity"):
|
||||||
|
_compile_v3(tmp_path / "stale-identity", stale_identity, artifact_bytes=mutated_bytes)
|
||||||
|
|
||||||
|
stale_locator = copy.deepcopy(rehashed_manifest)
|
||||||
|
stale_locator["source"]["locator"] = _manifest_v3()["source"]["locator"]
|
||||||
|
with pytest.raises(compiler.CompilerError, match="actual artifact SHA-256"):
|
||||||
|
_compile_v3(tmp_path / "stale-locator", stale_locator, artifact_bytes=mutated_bytes)
|
||||||
|
|
||||||
|
rehashed = _compile_v3(tmp_path / "rehashed", rehashed_manifest, artifact_bytes=mutated_bytes)
|
||||||
|
assert rehashed["strict_child_proposal"]["id"] != original["strict_child_proposal"]["id"]
|
||||||
|
assert rehashed["strict_child_proposal"]["payload_sha256"] != original["strict_child_proposal"]["payload_sha256"]
|
||||||
|
assert rehashed["hashes"]["packet_identity_sha256"] != original["hashes"]["packet_identity_sha256"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("target", ["extractor_name", "extractor_version", "claim"])
|
||||||
|
def test_v3_extraction_identity_and_normalized_candidate_mutations_change_proposal(tmp_path: Path, target: str) -> None:
|
||||||
|
original = _compile_v3(tmp_path / "original")
|
||||||
|
manifest = _manifest_v3()
|
||||||
|
if target == "extractor_name":
|
||||||
|
manifest["extractor"]["name"] = "fixture-semantic-extractor-revised"
|
||||||
|
elif target == "extractor_version":
|
||||||
|
manifest["extractor"]["version"] = "3.1.1"
|
||||||
|
else:
|
||||||
|
manifest["claims"][0]["proposition"] += " The proposal remains reviewable."
|
||||||
|
|
||||||
|
mutated = _compile_v3(tmp_path / target, manifest)
|
||||||
|
assert mutated["strict_child_proposal"]["id"] != original["strict_child_proposal"]["id"]
|
||||||
|
assert mutated["strict_child_proposal"]["payload_sha256"] != original["strict_child_proposal"]["payload_sha256"]
|
||||||
|
assert mutated["hashes"]["packet_identity_sha256"] != original["hashes"]["packet_identity_sha256"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_requires_exact_independent_extractor_spec_and_binds_spec_mutations(tmp_path: Path) -> None:
|
||||||
|
artifact, text, manifest_path, _extractor_spec = _write_v3_inputs(tmp_path / "missing")
|
||||||
|
with pytest.raises(compiler.CompilerError, match="independently supplied extractor specification"):
|
||||||
|
compiler.compile_source_packet(artifact, text, manifest_path)
|
||||||
|
|
||||||
|
mismatched_spec = EXTRACTOR_SPEC_BYTES + b"mismatch\n"
|
||||||
|
with pytest.raises(compiler.CompilerError, match="specification SHA-256 does not match"):
|
||||||
|
_compile_v3(
|
||||||
|
tmp_path / "mismatched",
|
||||||
|
_manifest_v3(),
|
||||||
|
extractor_spec_bytes=mismatched_spec,
|
||||||
|
)
|
||||||
|
|
||||||
|
original = _compile_v3(tmp_path / "original")
|
||||||
|
revised_spec = EXTRACTOR_SPEC_BYTES + b"normalization=revision-2\n"
|
||||||
|
revised_manifest = _manifest_v3(extractor_spec_bytes=revised_spec)
|
||||||
|
revised = _compile_v3(
|
||||||
|
tmp_path / "revised",
|
||||||
|
revised_manifest,
|
||||||
|
extractor_spec_bytes=revised_spec,
|
||||||
|
)
|
||||||
|
assert revised["hashes"]["extractor_spec_sha256"] == _sha256(revised_spec)
|
||||||
|
assert revised["strict_child_proposal"]["id"] != original["strict_child_proposal"]["id"]
|
||||||
|
assert revised["strict_child_proposal"]["payload_sha256"] != original["strict_child_proposal"]["payload_sha256"]
|
||||||
|
assert revised["hashes"]["packet_identity_sha256"] != original["hashes"]["packet_identity_sha256"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("field", "value", "expected"),
|
||||||
|
[
|
||||||
|
("identity", "document:alternate:source", "content-addressed identity"),
|
||||||
|
("locator", "artifact://mirror/source", "actual artifact SHA-256"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_v3_rejects_unbound_source_identity_or_locator(tmp_path: Path, field: str, value: str, expected: str) -> None:
|
||||||
|
manifest = _manifest_v3()
|
||||||
|
manifest["source"][field] = value
|
||||||
|
with pytest.raises(compiler.CompilerError, match=expected):
|
||||||
|
_compile_v3(tmp_path, manifest)
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_exact_manifest_bytes_are_identity_bound_but_paths_are_not(tmp_path: Path) -> None:
|
||||||
|
direct_paths = _write_v3_inputs(tmp_path / "direct", prefix="source")
|
||||||
|
copied_paths = _write_v3_inputs(tmp_path / "copied", prefix="renamed")
|
||||||
|
direct = compiler.compile_source_packet(*direct_paths)
|
||||||
|
copied = compiler.compile_source_packet(*copied_paths)
|
||||||
|
assert direct == copied
|
||||||
|
|
||||||
|
compact_paths = _write_v3_inputs(tmp_path / "compact", prefix="source")
|
||||||
|
compact_manifest = compact_paths[2]
|
||||||
|
manifest_object = json.loads(compact_manifest.read_text(encoding="utf-8"))
|
||||||
|
compact_manifest.write_text(
|
||||||
|
json.dumps(manifest_object, separators=(",", ":"), sort_keys=True),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
compact = compiler.compile_source_packet(*compact_paths)
|
||||||
|
|
||||||
|
assert compact["validated_manifest"] == direct["validated_manifest"]
|
||||||
|
assert compact["hashes"]["manifest_canonical_sha256"] == direct["hashes"]["manifest_canonical_sha256"]
|
||||||
|
assert compact["hashes"]["manifest_file_sha256"] != direct["hashes"]["manifest_file_sha256"]
|
||||||
|
assert compact["hashes"]["packet_identity_sha256"] != direct["hashes"]["packet_identity_sha256"]
|
||||||
|
assert compact["strict_child_proposal"]["id"] != direct["strict_child_proposal"]["id"]
|
||||||
|
assert compact["strict_child_proposal"]["payload_sha256"] != direct["strict_child_proposal"]["payload_sha256"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_valid_quote_binding_mutation_changes_identity_and_mismatches_fail_closed(tmp_path: Path) -> None:
|
||||||
|
original = _compile_v3(tmp_path / "original")
|
||||||
|
valid_mutation = _manifest_v3()
|
||||||
|
evidence = valid_mutation["claims"][0]["evidence"][0]
|
||||||
|
evidence["quote"] = EVIDENCE_QUOTE
|
||||||
|
evidence["locator"] = _byte_locator(EVIDENCE_QUOTE)
|
||||||
|
mutated = _compile_v3(tmp_path / "valid-mutation", valid_mutation)
|
||||||
|
assert mutated["strict_child_proposal"]["id"] != original["strict_child_proposal"]["id"]
|
||||||
|
assert mutated["strict_child_proposal"]["payload_sha256"] != original["strict_child_proposal"]["payload_sha256"]
|
||||||
|
|
||||||
|
quote_mismatch = _manifest_v3()
|
||||||
|
quote_mismatch["claims"][0]["evidence"][0]["quote"] = "Invented source sentence."
|
||||||
|
with pytest.raises(compiler.CompilerError, match="does not bind the exact quote bytes"):
|
||||||
|
_compile_v3(tmp_path / "quote-mismatch", quote_mismatch)
|
||||||
|
|
||||||
|
locator_mismatch = _manifest_v3()
|
||||||
|
locator_mismatch["claims"][0]["evidence"][0]["locator"]["start"] += 1
|
||||||
|
with pytest.raises(compiler.CompilerError, match="does not bind the exact quote bytes"):
|
||||||
|
_compile_v3(tmp_path / "locator-mismatch", locator_mismatch)
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_rejects_missing_assessment_relevant_evidence_and_tampered_set_hash(tmp_path: Path) -> None:
|
||||||
|
manifest = _manifest_v3()
|
||||||
|
manifest["claims"][0]["evidence"][0]["polarity"] = "illustrates"
|
||||||
|
manifest["claims"][0]["assessment"].update(
|
||||||
|
support_tier="none",
|
||||||
|
challenge_tier="none",
|
||||||
|
overall_state="insufficient",
|
||||||
|
)
|
||||||
|
with pytest.raises(compiler.CompilerError, match="requires at least one supports or challenges evidence"):
|
||||||
|
_compile_v3(tmp_path / "irrelevant", manifest)
|
||||||
|
|
||||||
|
bundle = _compile_v3(tmp_path / "valid")
|
||||||
|
child = copy.deepcopy(bundle["strict_child_proposal"])
|
||||||
|
child["payload"]["apply_payload"]["assessments"][0]["evidence_set_hash"] = "f" * 64
|
||||||
|
child["payload_sha256"] = compiler.canonical_sha256(child["payload"])
|
||||||
|
with pytest.raises(compiler.CompilerError, match="evidence_set_hash does not match claim-local evidence"):
|
||||||
|
compiler._validate_v3_compiled_child(child, bundle["validated_manifest"], bundle["quote_bindings"])
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("mutation", "expected"),
|
||||||
|
[
|
||||||
|
(
|
||||||
|
lambda manifest: manifest.__setitem__("agent_id", manifest["agent_id"].replace("-", "")),
|
||||||
|
"canonical lowercase",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
lambda manifest: manifest["source"].__setitem__("captured_at", "2026-07-18T10:00:00.0000000Z"),
|
||||||
|
"at most 6 fractional digits",
|
||||||
|
),
|
||||||
|
(lambda manifest: manifest["claims"][0].__setitem__("type", "structural"), "type must be one of"),
|
||||||
|
(
|
||||||
|
lambda manifest: manifest["claims"][0]["evidence"][0]["locator"].__setitem__("start", True),
|
||||||
|
"non-negative integer",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
lambda manifest: manifest["claims"][0]["assessment"].__setitem__("support_tier", 1),
|
||||||
|
"support_tier must be null or one of",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_v3_rejects_noncanonical_typed_manifest_fields(tmp_path: Path, mutation, expected: str) -> None:
|
||||||
|
manifest = _manifest_v3()
|
||||||
|
mutation(manifest)
|
||||||
|
with pytest.raises(compiler.CompilerError, match=expected):
|
||||||
|
_compile_v3(tmp_path, manifest)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("needle", "replacement"),
|
||||||
|
[
|
||||||
|
(
|
||||||
|
f'"schema": "{compiler.MANIFEST_SCHEMA_V3}",',
|
||||||
|
(f'"schema": "{compiler.MANIFEST_SCHEMA_V3}",\n "schema": "{compiler.MANIFEST_SCHEMA_V3}",'),
|
||||||
|
),
|
||||||
|
(
|
||||||
|
f'"schema": "{compiler.MANIFEST_SCHEMA_V3}",',
|
||||||
|
f'"schema": "{compiler.MANIFEST_SCHEMA_V3}",\n "schema": "{compiler.MANIFEST_SCHEMA}",',
|
||||||
|
),
|
||||||
|
(
|
||||||
|
f'"spec_sha256": "{_sha256(EXTRACTOR_SPEC_BYTES)}"',
|
||||||
|
(
|
||||||
|
f'"spec_sha256": "{_sha256(EXTRACTOR_SPEC_BYTES)}",\n'
|
||||||
|
f' "spec_sha256": "{_sha256(EXTRACTOR_SPEC_BYTES)}"'
|
||||||
|
),
|
||||||
|
),
|
||||||
|
(
|
||||||
|
'"overall_state": "support_dominant",',
|
||||||
|
'"overall_state": "support_dominant",\n "overall_state": "insufficient",',
|
||||||
|
),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_v3_rejects_identical_or_conflicting_duplicate_json_keys_at_any_depth(
|
||||||
|
tmp_path: Path, needle: str, replacement: str
|
||||||
|
) -> None:
|
||||||
|
paths = _write_v3_inputs(tmp_path)
|
||||||
|
manifest_path = paths[2]
|
||||||
|
raw_manifest = manifest_path.read_text(encoding="utf-8")
|
||||||
|
assert raw_manifest.count(needle) == 1
|
||||||
|
manifest_path.write_text(raw_manifest.replace(needle, replacement, 1), encoding="utf-8")
|
||||||
|
|
||||||
|
with pytest.raises(compiler.CompilerError, match="duplicate JSON object key"):
|
||||||
|
compiler.compile_source_packet(*paths)
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_duplicate_schema_key_fails_before_identity_or_output(
|
||||||
|
tmp_path: Path,
|
||||||
|
capsys: pytest.CaptureFixture,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
artifact, text, manifest_path, extractor_spec = _write_v3_inputs(tmp_path)
|
||||||
|
schema_line = f'"schema": "{compiler.MANIFEST_SCHEMA_V3}",'
|
||||||
|
raw_manifest = manifest_path.read_text(encoding="utf-8")
|
||||||
|
manifest_path.write_text(
|
||||||
|
raw_manifest.replace(schema_line, f'{schema_line}\n "schema": "{compiler.MANIFEST_SCHEMA_V3}",', 1),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
def unexpected_identity(*_args, **_kwargs):
|
||||||
|
raise AssertionError("duplicate-key manifest reached proposal identity")
|
||||||
|
|
||||||
|
monkeypatch.setattr(compiler, "_v3_identity_material", unexpected_identity)
|
||||||
|
output = tmp_path / "must-not-exist.json"
|
||||||
|
status = compiler.main(
|
||||||
|
[
|
||||||
|
"--artifact",
|
||||||
|
str(artifact),
|
||||||
|
"--text",
|
||||||
|
str(text),
|
||||||
|
"--manifest",
|
||||||
|
str(manifest_path),
|
||||||
|
"--extractor-spec",
|
||||||
|
str(extractor_spec),
|
||||||
|
"--output",
|
||||||
|
str(output),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
assert status == 2
|
||||||
|
assert json.loads(capsys.readouterr().out)["reason"] == "input_validation_failed"
|
||||||
|
assert not output.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_rejects_duplicate_semantic_keys_and_duplicate_generated_ids(
|
||||||
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
duplicate_claim = _manifest_v3()
|
||||||
|
duplicate_claim["claims"].append(copy.deepcopy(duplicate_claim["claims"][0]))
|
||||||
|
with pytest.raises(compiler.CompilerError, match="duplicate claim_key"):
|
||||||
|
_compile_v3(tmp_path / "claim-key", duplicate_claim)
|
||||||
|
|
||||||
|
duplicate_evidence = _manifest_v3()
|
||||||
|
duplicate_evidence["claims"][0]["evidence"].append(copy.deepcopy(duplicate_evidence["claims"][0]["evidence"][0]))
|
||||||
|
with pytest.raises(compiler.CompilerError, match="evidence_key duplicates"):
|
||||||
|
_compile_v3(tmp_path / "evidence-key", duplicate_evidence)
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
compiler,
|
||||||
|
"stable_uuid_v3",
|
||||||
|
lambda _row_kind, _packet_identity_sha256, _semantic_key: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
|
||||||
|
)
|
||||||
|
with pytest.raises(compiler.CompilerError, match="duplicate deterministic ids"):
|
||||||
|
_compile_v3(tmp_path / "generated-id")
|
||||||
|
|
||||||
|
|
||||||
def test_rejects_normalized_child_that_replaces_atomic_claim_text_with_quote(
|
def test_rejects_normalized_child_that_replaces_atomic_claim_text_with_quote(
|
||||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
) -> None:
|
) -> None:
|
||||||
|
|
|
||||||
232
tests/test_export_kb_transition_replay_material.py
Normal file
232
tests/test_export_kb_transition_replay_material.py
Normal file
|
|
@ -0,0 +1,232 @@
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import copy
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import stat
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from ops import run_local_genesis_ledger_rebuild as genesis_rebuild
|
||||||
|
|
||||||
|
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
sys.path.insert(0, str(REPO_ROOT / "scripts"))
|
||||||
|
|
||||||
|
import export_kb_transition_replay_material as exporter # noqa: E402
|
||||||
|
import kb_apply_replay_receipt as replay_receipt # noqa: E402
|
||||||
|
|
||||||
|
PROPOSAL_ID = "70000000-0000-4000-8000-000000000001"
|
||||||
|
REVIEWER_ID = "70000000-0000-4000-8000-000000000002"
|
||||||
|
SERVICE_ID = "70000000-0000-4000-8000-000000000003"
|
||||||
|
CLAIM_A = "70000000-0000-4000-8000-000000000004"
|
||||||
|
CLAIM_B = "70000000-0000-4000-8000-000000000005"
|
||||||
|
|
||||||
|
|
||||||
|
def _material() -> dict:
|
||||||
|
payload = {
|
||||||
|
"apply_payload": {
|
||||||
|
"from_claim": CLAIM_A,
|
||||||
|
"to_claim": CLAIM_B,
|
||||||
|
"edge_type": "supports",
|
||||||
|
"weight": 0.75,
|
||||||
|
},
|
||||||
|
"title": "Private exact transition packet",
|
||||||
|
}
|
||||||
|
approved = {
|
||||||
|
"id": PROPOSAL_ID,
|
||||||
|
"proposal_type": "add_edge",
|
||||||
|
"status": "approved",
|
||||||
|
"proposed_by_handle": "leo",
|
||||||
|
"proposed_by_agent_id": REVIEWER_ID,
|
||||||
|
"channel": "private_export_test",
|
||||||
|
"source_ref": "private://transition/source",
|
||||||
|
"rationale": "Retain the exact approved row for deterministic reconstruction.",
|
||||||
|
"payload": payload,
|
||||||
|
"reviewed_by_handle": "m3ta",
|
||||||
|
"reviewed_by_agent_id": REVIEWER_ID,
|
||||||
|
"reviewed_at": "2026-07-18T10:01:00+00:00",
|
||||||
|
"review_note": "Approved the exact strict packet.",
|
||||||
|
"applied_by_handle": None,
|
||||||
|
"applied_by_agent_id": None,
|
||||||
|
"applied_at": None,
|
||||||
|
"created_at": "2026-07-18T10:00:00+00:00",
|
||||||
|
"updated_at": "2026-07-18T10:01:00.000001+00:00",
|
||||||
|
}
|
||||||
|
applied = {
|
||||||
|
**approved,
|
||||||
|
"status": "applied",
|
||||||
|
"applied_by_handle": "kb-apply",
|
||||||
|
"applied_by_agent_id": SERVICE_ID,
|
||||||
|
"applied_at": "2026-07-18T10:02:00+00:00",
|
||||||
|
"updated_at": "2026-07-18T10:02:00.000001+00:00",
|
||||||
|
}
|
||||||
|
approval = {
|
||||||
|
"proposal_id": PROPOSAL_ID,
|
||||||
|
"proposal_type": "add_edge",
|
||||||
|
"payload": payload,
|
||||||
|
"reviewed_by_handle": "m3ta",
|
||||||
|
"reviewed_by_agent_id": REVIEWER_ID,
|
||||||
|
"reviewed_by_db_role": "kb_review",
|
||||||
|
"reviewed_at": "2026-07-18T10:01:00+00:00",
|
||||||
|
"review_note": "Approved the exact strict packet.",
|
||||||
|
}
|
||||||
|
receipt_proposal = {field: applied[field] for field in exporter.RECEIPT_PROPOSAL_FIELDS}
|
||||||
|
receipt = replay_receipt.build_replay_receipt(
|
||||||
|
receipt_proposal,
|
||||||
|
{
|
||||||
|
"claim_edges": [
|
||||||
|
{
|
||||||
|
"id": "70000000-0000-4000-8000-000000000006",
|
||||||
|
"from_claim": CLAIM_A,
|
||||||
|
"to_claim": CLAIM_B,
|
||||||
|
"edge_type": "supports",
|
||||||
|
"weight": 0.75,
|
||||||
|
"created_by": None,
|
||||||
|
"created_at": "2026-07-18T10:01:30+00:00",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
apply_sql_sha256=replay_receipt.sha256_text("exact guarded apply SQL"),
|
||||||
|
apply_sql_source="exact_executed_sql",
|
||||||
|
exported_at_utc="2026-07-18T10:03:00+00:00",
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"artifact": exporter.MATERIAL_ARTIFACT,
|
||||||
|
"contract_version": exporter.MATERIAL_CONTRACT_VERSION,
|
||||||
|
"sequence": 1,
|
||||||
|
"approved_proposal": approved,
|
||||||
|
"approval_snapshot": approval,
|
||||||
|
"applied_proposal": applied,
|
||||||
|
"replay_receipt": receipt,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_material_matches_genesis_replay_row_contract() -> None:
|
||||||
|
material = exporter.validate_material(
|
||||||
|
_material(),
|
||||||
|
expected_proposal_id=PROPOSAL_ID,
|
||||||
|
expected_sequence=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert exporter.MATERIAL_ARTIFACT == genesis_rebuild.MATERIAL_ARTIFACT
|
||||||
|
assert exporter.MATERIAL_CONTRACT_VERSION == genesis_rebuild.MATERIAL_CONTRACT_VERSION
|
||||||
|
assert exporter.PROPOSAL_FIELDS == genesis_rebuild.PROPOSAL_FIELDS
|
||||||
|
assert exporter.APPROVAL_FIELDS == genesis_rebuild.APPROVAL_FIELDS
|
||||||
|
genesis_rebuild._validate_proposal_rows(
|
||||||
|
material["approved_proposal"],
|
||||||
|
material["applied_proposal"],
|
||||||
|
material["approval_snapshot"],
|
||||||
|
material["replay_receipt"]["proposal"],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("mutation", "message"),
|
||||||
|
[
|
||||||
|
(lambda value: value["approved_proposal"].__setitem__("unexpected", True), "invalid fields"),
|
||||||
|
(
|
||||||
|
lambda value: value["applied_proposal"].__setitem__("rationale", "post-approval drift"),
|
||||||
|
"exact guarded apply transition",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
lambda value: value["approval_snapshot"].__setitem__("review_note", "different review"),
|
||||||
|
"does not match approved proposal",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
lambda value: value["replay_receipt"]["proposal"].__setitem__("id", CLAIM_A),
|
||||||
|
"worker replay receipt",
|
||||||
|
),
|
||||||
|
(lambda value: value.__setitem__("sequence", 0), "positive integer"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_material_validation_fails_closed_on_drift(mutation, message: str) -> None:
|
||||||
|
material = copy.deepcopy(_material())
|
||||||
|
mutation(material)
|
||||||
|
|
||||||
|
with pytest.raises(exporter.ExportError, match=message):
|
||||||
|
exporter.validate_material(material, expected_proposal_id=PROPOSAL_ID)
|
||||||
|
|
||||||
|
|
||||||
|
def test_json_parser_rejects_duplicate_and_nonfinite_fields() -> None:
|
||||||
|
with pytest.raises(exporter.ExportError, match="duplicate"):
|
||||||
|
exporter._parse_json_object('{"artifact":"a","artifact":"b"}', label="fixture")
|
||||||
|
with pytest.raises(exporter.ExportError, match="invalid numeric"):
|
||||||
|
exporter._parse_json_object('{"sequence":NaN}', label="fixture")
|
||||||
|
|
||||||
|
|
||||||
|
def test_private_writer_is_atomic_idempotent_and_refuses_alias_or_rerun_drift(tmp_path: Path) -> None:
|
||||||
|
material = exporter.validate_material(_material(), expected_proposal_id=PROPOSAL_ID)
|
||||||
|
protected = tmp_path / "worker-receipt.json"
|
||||||
|
protected.write_text("private worker receipt", encoding="utf-8")
|
||||||
|
protected.chmod(0o600)
|
||||||
|
output = tmp_path / "material.json"
|
||||||
|
|
||||||
|
written, disposition = exporter.write_private_material(output, material, protected_paths=(protected,))
|
||||||
|
assert written == output
|
||||||
|
assert disposition == "created"
|
||||||
|
assert stat.S_IMODE(output.stat().st_mode) == 0o600
|
||||||
|
assert output.stat().st_nlink == 1
|
||||||
|
assert json.loads(output.read_text(encoding="utf-8")) == material
|
||||||
|
|
||||||
|
_, rerun_disposition = exporter.write_private_material(output, material, protected_paths=(protected,))
|
||||||
|
assert rerun_disposition == "unchanged"
|
||||||
|
|
||||||
|
drifted = copy.deepcopy(material)
|
||||||
|
drifted["sequence"] = 2
|
||||||
|
with pytest.raises(exporter.ExportError, match="different replay material"):
|
||||||
|
exporter.write_private_material(output, drifted, protected_paths=(protected,))
|
||||||
|
with pytest.raises(exporter.ExportError, match="aliases a protected"):
|
||||||
|
exporter.write_private_material(protected, material, protected_paths=(protected,))
|
||||||
|
|
||||||
|
dangling = tmp_path / "dangling-output.json"
|
||||||
|
dangling.symlink_to(tmp_path / "missing-target.json")
|
||||||
|
with pytest.raises(exporter.ExportError, match="non-symlink"):
|
||||||
|
exporter.write_private_material(dangling, material, protected_paths=(protected,))
|
||||||
|
|
||||||
|
hardlink = tmp_path / "hardlink.json"
|
||||||
|
os.link(output, hardlink)
|
||||||
|
with pytest.raises(exporter.ExportError, match="hard-link"):
|
||||||
|
exporter.write_private_material(output, material, protected_paths=(protected,))
|
||||||
|
|
||||||
|
|
||||||
|
def test_private_receipt_loader_requires_exact_mode_and_rejects_symlinks(tmp_path: Path) -> None:
|
||||||
|
receipt = _material()["replay_receipt"]
|
||||||
|
receipt_path = tmp_path / "receipt.json"
|
||||||
|
receipt_path.write_text(json.dumps(receipt), encoding="utf-8")
|
||||||
|
receipt_path.chmod(0o600)
|
||||||
|
assert exporter._load_private_receipt(receipt_path) == receipt
|
||||||
|
|
||||||
|
receipt_path.chmod(0o640)
|
||||||
|
with pytest.raises(exporter.ExportError, match="exactly 0600"):
|
||||||
|
exporter._load_private_receipt(receipt_path)
|
||||||
|
|
||||||
|
receipt_path.chmod(0o600)
|
||||||
|
receipt_link = tmp_path / "receipt-link.json"
|
||||||
|
receipt_link.symlink_to(receipt_path)
|
||||||
|
with pytest.raises(exporter.ExportError, match="non-symlink"):
|
||||||
|
exporter._load_private_receipt(receipt_link)
|
||||||
|
|
||||||
|
|
||||||
|
def test_transition_fetch_uses_only_read_only_narrow_function(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
calls: list[tuple[str, bool]] = []
|
||||||
|
rows = {key: _material()[key] for key in ("approved_proposal", "approval_snapshot", "applied_proposal")}
|
||||||
|
|
||||||
|
def fake_run_psql(args, sql: str, password: str, *, redact_output_on_error: bool = False) -> str:
|
||||||
|
assert args.role == "kb_apply"
|
||||||
|
assert password == "private-password"
|
||||||
|
calls.append((sql, redact_output_on_error))
|
||||||
|
return json.dumps(rows)
|
||||||
|
|
||||||
|
monkeypatch.setattr(exporter.apply_engine, "run_psql", fake_run_psql)
|
||||||
|
args = argparse.Namespace(container="disposable", db="teleo", host="127.0.0.1", role="kb_apply")
|
||||||
|
|
||||||
|
assert exporter.fetch_transition_rows(args, "private-password", PROPOSAL_ID) == rows
|
||||||
|
assert len(calls) == 1
|
||||||
|
sql, redacted = calls[0]
|
||||||
|
assert "transaction isolation level repeatable read read only" in sql
|
||||||
|
assert "kb_stage.export_applied_proposal_replay_rows" in sql
|
||||||
|
assert not any(keyword in sql.lower() for keyword in ("insert ", "update ", "delete ", "truncate "))
|
||||||
|
assert redacted is True
|
||||||
|
|
@ -1022,6 +1022,37 @@ def _load_module(path: Path):
|
||||||
return module
|
return module
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("local", (True, False))
|
||||||
|
def test_vps_bridge_psql_fails_closed_on_sql_errors(
|
||||||
|
local: bool, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
module = _load_module(BRIDGE_DIR / "kb_tool.py")
|
||||||
|
captured: dict[str, object] = {}
|
||||||
|
|
||||||
|
def fake_run(command, **kwargs):
|
||||||
|
captured["command"] = command
|
||||||
|
captured["kwargs"] = kwargs
|
||||||
|
return SimpleNamespace(returncode=3, stdout="", stderr='ERROR: column "missing" does not exist')
|
||||||
|
|
||||||
|
monkeypatch.setattr(module.subprocess, "run", fake_run)
|
||||||
|
args = SimpleNamespace(
|
||||||
|
local=local,
|
||||||
|
container="teleo-pg",
|
||||||
|
db="teleo",
|
||||||
|
ssh="teleo@example.invalid",
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(SystemExit, match=r"psql command failed \(3\)"):
|
||||||
|
module.psql_json(args, "select missing from claims;")
|
||||||
|
|
||||||
|
command = captured["command"]
|
||||||
|
if local:
|
||||||
|
assert command[-3:] == ["-q", "-v", "ON_ERROR_STOP=1"]
|
||||||
|
else:
|
||||||
|
assert command[-1].endswith("-At -q -v ON_ERROR_STOP=1")
|
||||||
|
assert captured["kwargs"]["input"] == "select missing from claims;"
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
@pytest.mark.parametrize(
|
||||||
("value", "minimum", "maximum"),
|
("value", "minimum", "maximum"),
|
||||||
((True, 1, 2), ("1", 1, 2), (0, 1, 2), (3, 1, 2), (1, 2, 1)),
|
((True, 1, 2), ("1", 1, 2), (0, 1, 2), (3, 1, 2), (1, 2, 1)),
|
||||||
|
|
|
||||||
|
|
@ -1,18 +1,27 @@
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import time
|
import time
|
||||||
import uuid
|
import uuid
|
||||||
from collections.abc import Iterator
|
from collections.abc import Iterator
|
||||||
|
from contextlib import contextmanager
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||||
MIGRATION = REPO_ROOT / "scripts" / "kb_apply_prereqs.sql"
|
MIGRATION = REPO_ROOT / "scripts" / "kb_apply_prereqs.sql"
|
||||||
POSTGRES_IMAGE = "postgres:16-alpine"
|
POSTGRES_IMAGE = os.environ.get("LEOCLEAN_RUNTIME_POSTGRES_IMAGE", "postgres:16-alpine")
|
||||||
DATABASE = "teleo_clone"
|
DATABASE = "teleo_clone"
|
||||||
|
LEGACY_APPROVED_ID = "aaaaaaaa-0000-4000-8000-000000000001"
|
||||||
|
LEGACY_APPLIED_ID = "aaaaaaaa-0000-4000-8000-000000000002"
|
||||||
|
FRESH_PROPOSAL_ID = "aaaaaaaa-0000-4000-8000-000000000003"
|
||||||
|
REVIEWER_ID = "11111111-1111-1111-1111-111111111111"
|
||||||
|
STALE_REVIEWER_ID = "22222222-2222-2222-2222-222222222222"
|
||||||
|
LEGACY_REVIEWER_HANDLE = "m3ta"
|
||||||
|
CANONICAL_REVIEWER_HANDLE = "m3taversal"
|
||||||
|
|
||||||
BOOTSTRAP_SQL = r"""
|
BOOTSTRAP_SQL = r"""
|
||||||
create role kb_gate_owner nologin inherit;
|
create role kb_gate_owner nologin inherit;
|
||||||
|
|
@ -84,6 +93,42 @@ create table kb_stage.kb_proposal_approvals (
|
||||||
review_note text not null
|
review_note text not null
|
||||||
);
|
);
|
||||||
|
|
||||||
|
insert into kb_stage.kb_proposals (
|
||||||
|
id, proposal_type, status, payload, reviewed_by_handle, reviewed_by_agent_id,
|
||||||
|
reviewed_at, review_note, applied_by_handle, applied_by_agent_id, applied_at, updated_at
|
||||||
|
) values
|
||||||
|
(
|
||||||
|
'aaaaaaaa-0000-4000-8000-000000000001', 'add_edge', 'approved',
|
||||||
|
'{"apply_payload":{"from_claim":"aaaaaaaa-1000-4000-8000-000000000001"}}'::jsonb,
|
||||||
|
'm3ta', '11111111-1111-1111-1111-111111111111', '2026-07-18T10:00:00Z',
|
||||||
|
'Exact still-approved legacy row.', null, null, null, '2026-07-18T10:00:01Z'
|
||||||
|
),
|
||||||
|
(
|
||||||
|
'aaaaaaaa-0000-4000-8000-000000000002', 'add_edge', 'applied',
|
||||||
|
'{"apply_payload":{"from_claim":"aaaaaaaa-1000-4000-8000-000000000002"}}'::jsonb,
|
||||||
|
'm3ta', '11111111-1111-1111-1111-111111111111', '2026-07-18T09:00:00Z',
|
||||||
|
'Applied before exact approved rows were retained.', 'kb-apply',
|
||||||
|
'44444444-4444-4444-4444-444444444444', '2026-07-18T09:01:00Z',
|
||||||
|
'2026-07-18T09:01:00Z'
|
||||||
|
);
|
||||||
|
|
||||||
|
insert into kb_stage.kb_proposal_approvals (
|
||||||
|
proposal_id, proposal_type, payload, reviewed_by_handle, reviewed_by_agent_id,
|
||||||
|
reviewed_by_db_role, reviewed_at, review_note
|
||||||
|
) values
|
||||||
|
(
|
||||||
|
'aaaaaaaa-0000-4000-8000-000000000001', 'add_edge',
|
||||||
|
'{"apply_payload":{"from_claim":"aaaaaaaa-1000-4000-8000-000000000001"}}'::jsonb,
|
||||||
|
'm3ta', '11111111-1111-1111-1111-111111111111', 'kb_review',
|
||||||
|
'2026-07-18T10:00:00Z', 'Exact still-approved legacy row.'
|
||||||
|
),
|
||||||
|
(
|
||||||
|
'aaaaaaaa-0000-4000-8000-000000000002', 'add_edge',
|
||||||
|
'{"apply_payload":{"from_claim":"aaaaaaaa-1000-4000-8000-000000000002"}}'::jsonb,
|
||||||
|
'm3ta', '11111111-1111-1111-1111-111111111111', 'kb_review',
|
||||||
|
'2026-07-18T09:00:00Z', 'Applied before exact approved rows were retained.'
|
||||||
|
);
|
||||||
|
|
||||||
create function kb_stage.approve_strict_proposal(uuid, text, text)
|
create function kb_stage.approve_strict_proposal(uuid, text, text)
|
||||||
returns jsonb language sql security definer
|
returns jsonb language sql security definer
|
||||||
as $function$ select '{}'::jsonb $function$;
|
as $function$ select '{}'::jsonb $function$;
|
||||||
|
|
@ -145,8 +190,8 @@ def _apply_migration(container: str, *, check: bool = True) -> subprocess.Comple
|
||||||
return _psql(container, MIGRATION.read_text(), check=check)
|
return _psql(container, MIGRATION.read_text(), check=check)
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(scope="module")
|
@contextmanager
|
||||||
def migrated_postgres() -> Iterator[str]:
|
def _postgres_clone() -> Iterator[str]:
|
||||||
if shutil.which("docker") is None:
|
if shutil.which("docker") is None:
|
||||||
pytest.skip("Docker is required for the disposable PostgreSQL migration canary")
|
pytest.skip("Docker is required for the disposable PostgreSQL migration canary")
|
||||||
|
|
||||||
|
|
@ -155,6 +200,7 @@ def migrated_postgres() -> Iterator[str]:
|
||||||
[
|
[
|
||||||
"docker",
|
"docker",
|
||||||
"run",
|
"run",
|
||||||
|
"--pull=never",
|
||||||
"--rm",
|
"--rm",
|
||||||
"-d",
|
"-d",
|
||||||
"--name",
|
"--name",
|
||||||
|
|
@ -189,7 +235,6 @@ def migrated_postgres() -> Iterator[str]:
|
||||||
pytest.fail("disposable PostgreSQL did not become ready")
|
pytest.fail("disposable PostgreSQL did not become ready")
|
||||||
|
|
||||||
_psql(container, BOOTSTRAP_SQL)
|
_psql(container, BOOTSTRAP_SQL)
|
||||||
_apply_migration(container)
|
|
||||||
yield container
|
yield container
|
||||||
finally:
|
finally:
|
||||||
subprocess.run(
|
subprocess.run(
|
||||||
|
|
@ -199,10 +244,154 @@ def migrated_postgres() -> Iterator[str]:
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def migrated_postgres() -> Iterator[str]:
|
||||||
|
with _postgres_clone() as container:
|
||||||
|
_apply_migration(container)
|
||||||
|
yield container
|
||||||
|
|
||||||
|
|
||||||
def _lines(completed: subprocess.CompletedProcess[str]) -> set[str]:
|
def _lines(completed: subprocess.CompletedProcess[str]) -> set[str]:
|
||||||
return {line for line in completed.stdout.splitlines() if line}
|
return {line for line in completed.stdout.splitlines() if line}
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_only_reviewer_is_canonicalized_without_rewriting_history(
|
||||||
|
migrated_postgres: str,
|
||||||
|
) -> None:
|
||||||
|
identity_rows = _psql(
|
||||||
|
migrated_postgres,
|
||||||
|
f"""
|
||||||
|
select id::text || '|' || handle || '|' || kind
|
||||||
|
from public.agents
|
||||||
|
where handle in ('{LEGACY_REVIEWER_HANDLE}', '{CANONICAL_REVIEWER_HANDLE}')
|
||||||
|
order by handle;
|
||||||
|
""",
|
||||||
|
).stdout.splitlines()
|
||||||
|
assert identity_rows == [f"{REVIEWER_ID}|{CANONICAL_REVIEWER_HANDLE}|human"]
|
||||||
|
|
||||||
|
historical_rows = _psql(
|
||||||
|
migrated_postgres,
|
||||||
|
f"""
|
||||||
|
select 'proposal|' || id::text || '|' || reviewed_by_handle || '|' || reviewed_by_agent_id::text
|
||||||
|
from kb_stage.kb_proposals
|
||||||
|
where id in ('{LEGACY_APPROVED_ID}'::uuid, '{LEGACY_APPLIED_ID}'::uuid)
|
||||||
|
order by id;
|
||||||
|
select 'approval|' || proposal_id::text || '|' || reviewed_by_handle || '|' ||
|
||||||
|
reviewed_by_agent_id::text
|
||||||
|
from kb_stage.kb_proposal_approvals
|
||||||
|
where proposal_id in ('{LEGACY_APPROVED_ID}'::uuid, '{LEGACY_APPLIED_ID}'::uuid)
|
||||||
|
order by proposal_id;
|
||||||
|
""",
|
||||||
|
).stdout.splitlines()
|
||||||
|
assert historical_rows == [
|
||||||
|
f"proposal|{LEGACY_APPROVED_ID}|{LEGACY_REVIEWER_HANDLE}|{REVIEWER_ID}",
|
||||||
|
f"proposal|{LEGACY_APPLIED_ID}|{LEGACY_REVIEWER_HANDLE}|{REVIEWER_ID}",
|
||||||
|
f"approval|{LEGACY_APPROVED_ID}|{LEGACY_REVIEWER_HANDLE}|{REVIEWER_ID}",
|
||||||
|
f"approval|{LEGACY_APPLIED_ID}|{LEGACY_REVIEWER_HANDLE}|{REVIEWER_ID}",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_already_current_reviewer_migration_is_idempotent() -> None:
|
||||||
|
with _postgres_clone() as container:
|
||||||
|
_psql(
|
||||||
|
container,
|
||||||
|
f"update public.agents set handle = '{CANONICAL_REVIEWER_HANDLE}' "
|
||||||
|
f"where handle = '{LEGACY_REVIEWER_HANDLE}';",
|
||||||
|
)
|
||||||
|
_apply_migration(container)
|
||||||
|
state_sql = f"""
|
||||||
|
select principal.reviewed_by_handle || '|' || principal.reviewed_by_agent_id::text || '|' ||
|
||||||
|
principal.active::text || '|' || principal.created_at::text || '|' ||
|
||||||
|
(select count(*) from public.agents where handle = '{LEGACY_REVIEWER_HANDLE}')::text
|
||||||
|
from kb_stage.kb_review_principals principal
|
||||||
|
where principal.db_role = 'kb_review';
|
||||||
|
"""
|
||||||
|
first_state = _psql(container, state_sql).stdout.strip()
|
||||||
|
_apply_migration(container)
|
||||||
|
second_state = _psql(container, state_sql).stdout.strip()
|
||||||
|
|
||||||
|
expected_state = f"{CANONICAL_REVIEWER_HANDLE}|{REVIEWER_ID}|true|1970-01-01 00:00:00+00|0"
|
||||||
|
assert first_state == expected_state
|
||||||
|
assert second_state == first_state
|
||||||
|
|
||||||
|
|
||||||
|
def test_stale_reviewer_principal_is_repaired_without_resetting_created_at() -> None:
|
||||||
|
with _postgres_clone() as container:
|
||||||
|
_psql(
|
||||||
|
container,
|
||||||
|
f"""
|
||||||
|
insert into public.agents (id, handle, kind)
|
||||||
|
values ('{STALE_REVIEWER_ID}', 'stale-reviewer', 'human');
|
||||||
|
insert into kb_stage.kb_review_principals
|
||||||
|
(db_role, reviewed_by_handle, reviewed_by_agent_id, active, created_at)
|
||||||
|
values (
|
||||||
|
'kb_review', 'stale-reviewer', '{STALE_REVIEWER_ID}', false,
|
||||||
|
'2001-02-03T04:05:06Z'
|
||||||
|
);
|
||||||
|
""",
|
||||||
|
)
|
||||||
|
_apply_migration(container)
|
||||||
|
principal_state = _psql(
|
||||||
|
container,
|
||||||
|
"""
|
||||||
|
select reviewed_by_handle || '|' || reviewed_by_agent_id::text || '|' || active::text || '|' ||
|
||||||
|
created_at::text
|
||||||
|
from kb_stage.kb_review_principals
|
||||||
|
where db_role = 'kb_review';
|
||||||
|
""",
|
||||||
|
).stdout.strip()
|
||||||
|
|
||||||
|
expected_state = f"{CANONICAL_REVIEWER_HANDLE}|{REVIEWER_ID}|true|2001-02-03 04:05:06+00"
|
||||||
|
assert principal_state == expected_state
|
||||||
|
|
||||||
|
|
||||||
|
def test_migration_refuses_distinct_legacy_and_canonical_reviewers() -> None:
|
||||||
|
with _postgres_clone() as container:
|
||||||
|
_psql(
|
||||||
|
container,
|
||||||
|
f"""
|
||||||
|
insert into public.agents (id, handle, kind)
|
||||||
|
values ('{STALE_REVIEWER_ID}', '{CANONICAL_REVIEWER_HANDLE}', 'human');
|
||||||
|
""",
|
||||||
|
)
|
||||||
|
refused = _apply_migration(container, check=False)
|
||||||
|
identity_rows = _psql(
|
||||||
|
container,
|
||||||
|
f"""
|
||||||
|
select handle || '|' || id::text
|
||||||
|
from public.agents
|
||||||
|
where handle in ('{LEGACY_REVIEWER_HANDLE}', '{CANONICAL_REVIEWER_HANDLE}')
|
||||||
|
order by handle;
|
||||||
|
""",
|
||||||
|
).stdout.splitlines()
|
||||||
|
|
||||||
|
assert refused.returncode != 0
|
||||||
|
assert "both reviewer handles m3ta" in refused.stderr
|
||||||
|
assert "exist with different agent UUIDs" in refused.stderr
|
||||||
|
assert identity_rows == [
|
||||||
|
f"{LEGACY_REVIEWER_HANDLE}|{REVIEWER_ID}",
|
||||||
|
f"{CANONICAL_REVIEWER_HANDLE}|{STALE_REVIEWER_ID}",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_migration_refuses_to_fabricate_missing_human_reviewer() -> None:
|
||||||
|
with _postgres_clone() as container:
|
||||||
|
_psql(
|
||||||
|
container,
|
||||||
|
f"""
|
||||||
|
delete from kb_stage.kb_proposal_approvals;
|
||||||
|
delete from public.agents where handle = '{LEGACY_REVIEWER_HANDLE}';
|
||||||
|
""",
|
||||||
|
)
|
||||||
|
refused = _apply_migration(container, check=False)
|
||||||
|
remaining_agents = _psql(container, "select count(*) from public.agents;").stdout.strip()
|
||||||
|
|
||||||
|
assert refused.returncode != 0
|
||||||
|
assert "required human reviewer agent is missing" in refused.stderr
|
||||||
|
assert "handle m3ta or m3taversal" in refused.stderr
|
||||||
|
assert remaining_agents == "0"
|
||||||
|
|
||||||
|
|
||||||
def test_reviewer_principal_seed_timestamp_is_deterministic(migrated_postgres: str) -> None:
|
def test_reviewer_principal_seed_timestamp_is_deterministic(migrated_postgres: str) -> None:
|
||||||
created_at = _psql(
|
created_at = _psql(
|
||||||
migrated_postgres,
|
migrated_postgres,
|
||||||
|
|
@ -212,6 +401,161 @@ def test_reviewer_principal_seed_timestamp_is_deterministic(migrated_postgres: s
|
||||||
assert created_at == "1970-01-01 00:00:00+00"
|
assert created_at == "1970-01-01 00:00:00+00"
|
||||||
|
|
||||||
|
|
||||||
|
def test_upgrade_backfills_only_still_exact_approved_rows(migrated_postgres: str) -> None:
|
||||||
|
snapshot_state = _psql(
|
||||||
|
migrated_postgres,
|
||||||
|
f"""
|
||||||
|
select proposal_id::text || '|' || (approved_proposal_snapshot is not null)::text || '|' ||
|
||||||
|
coalesce(approved_proposal_snapshot->>'status', 'unavailable') || '|' ||
|
||||||
|
coalesce(approved_proposal_snapshot->>'reviewed_by_handle', 'unavailable')
|
||||||
|
from kb_stage.kb_proposal_approvals
|
||||||
|
where proposal_id in ('{LEGACY_APPROVED_ID}'::uuid, '{LEGACY_APPLIED_ID}'::uuid)
|
||||||
|
order by proposal_id;
|
||||||
|
""",
|
||||||
|
).stdout.splitlines()
|
||||||
|
|
||||||
|
assert snapshot_state == [
|
||||||
|
f"{LEGACY_APPROVED_ID}|true|approved|{LEGACY_REVIEWER_HANDLE}",
|
||||||
|
f"{LEGACY_APPLIED_ID}|false|unavailable|unavailable",
|
||||||
|
]
|
||||||
|
unavailable = _psql(
|
||||||
|
migrated_postgres,
|
||||||
|
f"""
|
||||||
|
set session authorization kb_apply;
|
||||||
|
select kb_stage.export_applied_proposal_replay_rows('{LEGACY_APPLIED_ID}'::uuid);
|
||||||
|
""",
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
assert unavailable.returncode != 0
|
||||||
|
assert "exact applied transition unavailable" in unavailable.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_new_approval_preserves_exact_post_approval_row_for_apply_export(migrated_postgres: str) -> None:
|
||||||
|
payload = '{"apply_payload":{"from_claim":"aaaaaaaa-1000-4000-8000-000000000003"}}'
|
||||||
|
_psql(
|
||||||
|
migrated_postgres,
|
||||||
|
f"""
|
||||||
|
insert into kb_stage.kb_proposals (id, proposal_type, status, payload, updated_at)
|
||||||
|
values ('{FRESH_PROPOSAL_ID}'::uuid, 'add_edge', 'pending_review', '{payload}'::jsonb,
|
||||||
|
'2026-07-18T11:00:00Z');
|
||||||
|
set session authorization kb_review;
|
||||||
|
select kb_stage.approve_strict_proposal(
|
||||||
|
'{FRESH_PROPOSAL_ID}'::uuid, 'add_edge', '{payload}'::jsonb, '{CANONICAL_REVIEWER_HANDLE}',
|
||||||
|
'Fresh exact approval snapshot.'
|
||||||
|
);
|
||||||
|
""",
|
||||||
|
)
|
||||||
|
approval_identity = _psql(
|
||||||
|
migrated_postgres,
|
||||||
|
f"""
|
||||||
|
select proposal.reviewed_by_handle || '|' || proposal.reviewed_by_agent_id::text || '|' ||
|
||||||
|
approval.reviewed_by_handle || '|' || approval.reviewed_by_agent_id::text || '|' ||
|
||||||
|
(approval.approved_proposal_snapshot->>'reviewed_by_handle') || '|' ||
|
||||||
|
(approval.approved_proposal_snapshot->>'reviewed_by_agent_id')
|
||||||
|
from kb_stage.kb_proposals proposal
|
||||||
|
join kb_stage.kb_proposal_approvals approval on approval.proposal_id = proposal.id
|
||||||
|
where proposal.id = '{FRESH_PROPOSAL_ID}'::uuid;
|
||||||
|
""",
|
||||||
|
).stdout.strip()
|
||||||
|
assert approval_identity == (
|
||||||
|
f"{CANONICAL_REVIEWER_HANDLE}|{REVIEWER_ID}|"
|
||||||
|
f"{CANONICAL_REVIEWER_HANDLE}|{REVIEWER_ID}|"
|
||||||
|
f"{CANONICAL_REVIEWER_HANDLE}|{REVIEWER_ID}"
|
||||||
|
)
|
||||||
|
approval_state = _psql(
|
||||||
|
migrated_postgres,
|
||||||
|
f"""
|
||||||
|
select jsonb_build_object(
|
||||||
|
'matches', approval.approved_proposal_snapshot = to_jsonb(proposal),
|
||||||
|
'status', approval.approved_proposal_snapshot->>'status',
|
||||||
|
'updated_at_matches', approval.approved_proposal_snapshot->'updated_at' = to_jsonb(proposal)->'updated_at'
|
||||||
|
)::text
|
||||||
|
from kb_stage.kb_proposal_approvals approval
|
||||||
|
join kb_stage.kb_proposals proposal on proposal.id = approval.proposal_id
|
||||||
|
where proposal.id = '{FRESH_PROPOSAL_ID}'::uuid;
|
||||||
|
""",
|
||||||
|
).stdout.strip()
|
||||||
|
assert approval_state == '{"status": "approved", "matches": true, "updated_at_matches": true}'
|
||||||
|
|
||||||
|
direct_approval_read = _psql(
|
||||||
|
migrated_postgres,
|
||||||
|
"set session authorization kb_apply; select * from kb_stage.kb_proposal_approvals;",
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
assert direct_approval_read.returncode != 0
|
||||||
|
assert "permission denied for table kb_proposal_approvals" in direct_approval_read.stderr
|
||||||
|
|
||||||
|
premature_export = _psql(
|
||||||
|
migrated_postgres,
|
||||||
|
f"""
|
||||||
|
set session authorization kb_apply;
|
||||||
|
select kb_stage.export_applied_proposal_replay_rows('{FRESH_PROPOSAL_ID}'::uuid);
|
||||||
|
""",
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
assert premature_export.returncode != 0
|
||||||
|
assert "exact applied transition unavailable" in premature_export.stderr
|
||||||
|
|
||||||
|
_psql(
|
||||||
|
migrated_postgres,
|
||||||
|
f"""
|
||||||
|
update kb_stage.kb_proposal_approvals
|
||||||
|
set approved_proposal_snapshot =
|
||||||
|
approved_proposal_snapshot || '{{"rationale":"tampered after approval"}}'::jsonb
|
||||||
|
where proposal_id = '{FRESH_PROPOSAL_ID}'::uuid;
|
||||||
|
""",
|
||||||
|
)
|
||||||
|
for guard_call in (
|
||||||
|
"kb_stage.assert_approved_proposal("
|
||||||
|
"proposal.id, proposal.proposal_type, proposal.payload, proposal.reviewed_by_handle, "
|
||||||
|
"proposal.reviewed_by_agent_id, proposal.reviewed_at, proposal.review_note)",
|
||||||
|
"kb_stage.finish_approved_proposal("
|
||||||
|
"proposal.id, proposal.proposal_type, proposal.payload, proposal.reviewed_by_handle, "
|
||||||
|
"proposal.reviewed_by_agent_id, proposal.reviewed_at, proposal.review_note, 'kb-apply')",
|
||||||
|
):
|
||||||
|
refused = _psql(
|
||||||
|
migrated_postgres,
|
||||||
|
f"""
|
||||||
|
set session authorization kb_apply;
|
||||||
|
select {guard_call}
|
||||||
|
from kb_stage.kb_proposals proposal where proposal.id = '{FRESH_PROPOSAL_ID}'::uuid;
|
||||||
|
""",
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
assert refused.returncode != 0
|
||||||
|
assert "immutable approved snapshot" in refused.stderr
|
||||||
|
_psql(
|
||||||
|
migrated_postgres,
|
||||||
|
f"""
|
||||||
|
update kb_stage.kb_proposal_approvals approval
|
||||||
|
set approved_proposal_snapshot = to_jsonb(proposal)
|
||||||
|
from kb_stage.kb_proposals proposal
|
||||||
|
where approval.proposal_id = proposal.id
|
||||||
|
and proposal.id = '{FRESH_PROPOSAL_ID}'::uuid;
|
||||||
|
""",
|
||||||
|
)
|
||||||
|
|
||||||
|
exported = _psql(
|
||||||
|
migrated_postgres,
|
||||||
|
f"""
|
||||||
|
set session authorization kb_apply;
|
||||||
|
select kb_stage.assert_approved_proposal(
|
||||||
|
proposal.id, proposal.proposal_type, proposal.payload, proposal.reviewed_by_handle,
|
||||||
|
proposal.reviewed_by_agent_id, proposal.reviewed_at, proposal.review_note
|
||||||
|
)
|
||||||
|
from kb_stage.kb_proposals proposal where proposal.id = '{FRESH_PROPOSAL_ID}'::uuid;
|
||||||
|
select kb_stage.finish_approved_proposal(
|
||||||
|
proposal.id, proposal.proposal_type, proposal.payload, proposal.reviewed_by_handle,
|
||||||
|
proposal.reviewed_by_agent_id, proposal.reviewed_at, proposal.review_note, 'kb-apply'
|
||||||
|
)
|
||||||
|
from kb_stage.kb_proposals proposal where proposal.id = '{FRESH_PROPOSAL_ID}'::uuid;
|
||||||
|
select jsonb_object_keys(kb_stage.export_applied_proposal_replay_rows('{FRESH_PROPOSAL_ID}'::uuid))
|
||||||
|
order by 1;
|
||||||
|
""",
|
||||||
|
).stdout.splitlines()
|
||||||
|
assert exported[-3:] == ["applied_proposal", "approval_snapshot", "approved_proposal"]
|
||||||
|
|
||||||
|
|
||||||
def _catalog_snapshot(container: str) -> str:
|
def _catalog_snapshot(container: str) -> str:
|
||||||
return _psql(
|
return _psql(
|
||||||
container,
|
container,
|
||||||
|
|
@ -239,9 +583,16 @@ select 'function|' || procedure.proname || '(' || pg_catalog.oidvectortypes(proc
|
||||||
join pg_catalog.pg_roles owner_role on owner_role.oid = procedure.proowner
|
join pg_catalog.pg_roles owner_role on owner_role.oid = procedure.proowner
|
||||||
where namespace.nspname = 'kb_stage'
|
where namespace.nspname = 'kb_stage'
|
||||||
and procedure.proname in (
|
and procedure.proname in (
|
||||||
'approve_strict_proposal', 'assert_approved_proposal', 'finish_approved_proposal'
|
'approve_strict_proposal', 'assert_approved_proposal', 'finish_approved_proposal',
|
||||||
|
'export_applied_proposal_replay_rows'
|
||||||
)
|
)
|
||||||
order by procedure.proname, procedure.proargtypes::text;
|
order by procedure.proname, procedure.proargtypes::text;
|
||||||
|
select 'column|' || attribute.attname || '|' || attribute.atttypid::regtype::text || '|' ||
|
||||||
|
attribute.attnotnull
|
||||||
|
from pg_catalog.pg_attribute attribute
|
||||||
|
where attribute.attrelid = 'kb_stage.kb_proposal_approvals'::regclass
|
||||||
|
and attribute.attname = 'approved_proposal_snapshot'
|
||||||
|
and not attribute.attisdropped;
|
||||||
select 'table_acl|' || namespace.nspname || '|' || relation.relname || '|' ||
|
select 'table_acl|' || namespace.nspname || '|' || relation.relname || '|' ||
|
||||||
case when acl.grantee = 0 then 'PUBLIC' else grantee_role.rolname end || '|' ||
|
case when acl.grantee = 0 then 'PUBLIC' else grantee_role.rolname end || '|' ||
|
||||||
pg_catalog.upper(acl.privilege_type) || '|' || acl.is_grantable
|
pg_catalog.upper(acl.privilege_type) || '|' || acl.is_grantable
|
||||||
|
|
@ -272,7 +623,8 @@ select 'function_acl|' || procedure.proname || '(' ||
|
||||||
) acl
|
) acl
|
||||||
where namespace.nspname = 'kb_stage'
|
where namespace.nspname = 'kb_stage'
|
||||||
and procedure.proname in (
|
and procedure.proname in (
|
||||||
'approve_strict_proposal', 'assert_approved_proposal', 'finish_approved_proposal'
|
'approve_strict_proposal', 'assert_approved_proposal', 'finish_approved_proposal',
|
||||||
|
'export_applied_proposal_replay_rows'
|
||||||
)
|
)
|
||||||
and acl.grantee <> procedure.proowner
|
and acl.grantee <> procedure.proowner
|
||||||
order by procedure.proname, procedure.proargtypes::text, acl.grantee;
|
order by procedure.proname, procedure.proargtypes::text, acl.grantee;
|
||||||
|
|
@ -343,7 +695,8 @@ select procedure.proname || '(' || pg_catalog.oidvectortypes(procedure.proargtyp
|
||||||
join pg_catalog.pg_roles owner_role on owner_role.oid = procedure.proowner
|
join pg_catalog.pg_roles owner_role on owner_role.oid = procedure.proowner
|
||||||
where namespace.nspname = 'kb_stage'
|
where namespace.nspname = 'kb_stage'
|
||||||
and procedure.proname in (
|
and procedure.proname in (
|
||||||
'approve_strict_proposal', 'assert_approved_proposal', 'finish_approved_proposal'
|
'approve_strict_proposal', 'assert_approved_proposal', 'finish_approved_proposal',
|
||||||
|
'export_applied_proposal_replay_rows'
|
||||||
);
|
);
|
||||||
""",
|
""",
|
||||||
)
|
)
|
||||||
|
|
@ -354,6 +707,7 @@ select procedure.proname || '(' || pg_catalog.oidvectortypes(procedure.proargtyp
|
||||||
'kb_gate_owner|true|{"search_path=pg_catalog, pg_temp"}',
|
'kb_gate_owner|true|{"search_path=pg_catalog, pg_temp"}',
|
||||||
"finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text, text)|"
|
"finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text, text)|"
|
||||||
'kb_gate_owner|true|{"search_path=pg_catalog, pg_temp"}',
|
'kb_gate_owner|true|{"search_path=pg_catalog, pg_temp"}',
|
||||||
|
'export_applied_proposal_replay_rows(uuid)|kb_gate_owner|true|{"search_path=pg_catalog, pg_temp"}',
|
||||||
}
|
}
|
||||||
assert (
|
assert (
|
||||||
_psql(
|
_psql(
|
||||||
|
|
@ -431,6 +785,7 @@ select count(*)
|
||||||
"kb_apply|EXECUTE|false",
|
"kb_apply|EXECUTE|false",
|
||||||
"function_acl|finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text, text)|"
|
"function_acl|finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text, text)|"
|
||||||
"kb_apply|EXECUTE|false",
|
"kb_apply|EXECUTE|false",
|
||||||
|
"function_acl|export_applied_proposal_replay_rows(uuid)|kb_apply|EXECUTE|false",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -470,6 +825,38 @@ def test_migration_fails_closed_on_unexpected_table_grantee(migrated_postgres: s
|
||||||
_psql(migrated_postgres, "revoke all on kb_stage.kb_proposal_approvals from rogue_grantee;")
|
_psql(migrated_postgres, "revoke all on kb_stage.kb_proposal_approvals from rogue_grantee;")
|
||||||
|
|
||||||
|
|
||||||
|
def test_migration_refuses_rogue_canonical_insert_before_install_and_normalizes_after_removal() -> None:
|
||||||
|
with _postgres_clone() as container:
|
||||||
|
_psql(container, "grant usage on schema public to rogue_grantee;")
|
||||||
|
for grant_sql, revoke_sql in (
|
||||||
|
(
|
||||||
|
"grant insert on public.claims to rogue_grantee;",
|
||||||
|
"revoke insert on public.claims from rogue_grantee;",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"grant insert (id) on public.claims to rogue_grantee;",
|
||||||
|
"revoke insert (id) on public.claims from rogue_grantee;",
|
||||||
|
),
|
||||||
|
):
|
||||||
|
_psql(container, grant_sql)
|
||||||
|
refused = _apply_migration(container, check=False)
|
||||||
|
assert refused.returncode != 0
|
||||||
|
assert "unexpected protected table grantee" in refused.stderr
|
||||||
|
assert _psql(container, "select count(*) from public.claims;").stdout.strip() == "0"
|
||||||
|
_psql(container, revoke_sql)
|
||||||
|
|
||||||
|
_apply_migration(container)
|
||||||
|
rogue_insert = _psql(
|
||||||
|
container,
|
||||||
|
"set session authorization rogue_grantee; "
|
||||||
|
"insert into public.claims (id) values ('99999999-9999-4999-8999-999999999999');",
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
assert rogue_insert.returncode != 0
|
||||||
|
assert "permission denied for table claims" in rogue_insert.stderr
|
||||||
|
assert _psql(container, "select count(*) from public.claims;").stdout.strip() == "0"
|
||||||
|
|
||||||
|
|
||||||
def test_migration_fails_closed_on_unexpected_column_grantee(migrated_postgres: str) -> None:
|
def test_migration_fails_closed_on_unexpected_column_grantee(migrated_postgres: str) -> None:
|
||||||
_psql(
|
_psql(
|
||||||
migrated_postgres,
|
migrated_postgres,
|
||||||
|
|
|
||||||
|
|
@ -70,8 +70,154 @@ def _edge_row() -> dict:
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _v3_bundle_and_rows() -> tuple[dict, dict]:
|
||||||
|
agent_id = "11111111-1111-4111-8111-111111111111"
|
||||||
|
evidence_id = "eeeeeeee-eeee-4eee-8eee-eeeeeeeeeeee"
|
||||||
|
assessment_id = "ffffffff-ffff-4fff-8fff-ffffffffffff"
|
||||||
|
evidence = {
|
||||||
|
"id": evidence_id,
|
||||||
|
"claim_id": CLAIM_A,
|
||||||
|
"source_id": SOURCE_A,
|
||||||
|
"polarity": "supports",
|
||||||
|
"excerpt": "The primary record supports the bounded claim.",
|
||||||
|
"locator_json": {"section": "results", "paragraph": 3},
|
||||||
|
"source_content_hash": "a" * 64,
|
||||||
|
"rationale": "The exact locator directly supports the proposition.",
|
||||||
|
"created_by": agent_id,
|
||||||
|
}
|
||||||
|
payload = {
|
||||||
|
"contract_version": 3,
|
||||||
|
"agent_id": agent_id,
|
||||||
|
"claims": [
|
||||||
|
{
|
||||||
|
"id": CLAIM_A,
|
||||||
|
"owner_agent_id": None,
|
||||||
|
"type": "empirical",
|
||||||
|
"proposition": "The V3 apply path retains exact provenance.",
|
||||||
|
"body_markdown": "This bounded claim is supported by one content-addressed source.",
|
||||||
|
"scope": "Disposable V3 writer verification.",
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"status": "active",
|
||||||
|
"revision_criteria": "Revise if the cited bytes or locator cannot be reproduced.",
|
||||||
|
"revision_kind": "original",
|
||||||
|
"supersedes_id": None,
|
||||||
|
"tags": ["v3", "replay"],
|
||||||
|
"created_by": agent_id,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"sources": [
|
||||||
|
{
|
||||||
|
"id": SOURCE_A,
|
||||||
|
"source_type": "paper",
|
||||||
|
"canonical_url": "https://example.test/v3-source",
|
||||||
|
"storage_uri": None,
|
||||||
|
"excerpt": "Content-addressed primary record.",
|
||||||
|
"content_hash": "a" * 64,
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"ingestion_origin": "operator_upload",
|
||||||
|
"provenance_status": "verified",
|
||||||
|
"source_class": "primary_record",
|
||||||
|
"auto_promotion_policy": "human_required",
|
||||||
|
"captured_at": "2026-07-18T10:00:00+00:00",
|
||||||
|
"created_by": agent_id,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"evidence": [evidence],
|
||||||
|
"edges": [],
|
||||||
|
"assessments": [
|
||||||
|
{
|
||||||
|
"id": assessment_id,
|
||||||
|
"claim_id": CLAIM_A,
|
||||||
|
"support_tier": "strong",
|
||||||
|
"challenge_tier": "none",
|
||||||
|
"overall_state": "support_dominant",
|
||||||
|
"rationale": "The exact primary record supports the bounded claim.",
|
||||||
|
"evidence_set_hash": ap._v3_evidence_set_hash(CLAIM_A, [evidence]),
|
||||||
|
"supersedes_assessment_id": None,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"reasoning_tools": [],
|
||||||
|
}
|
||||||
|
rows = {
|
||||||
|
"claims": [
|
||||||
|
{
|
||||||
|
"id": CLAIM_A,
|
||||||
|
"owner_agent_id": None,
|
||||||
|
"type": "empirical",
|
||||||
|
"text": payload["claims"][0]["proposition"],
|
||||||
|
"proposition": payload["claims"][0]["proposition"],
|
||||||
|
"body_markdown": payload["claims"][0]["body_markdown"],
|
||||||
|
"scope": payload["claims"][0]["scope"],
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"status": "active",
|
||||||
|
"admission_state": "admitted",
|
||||||
|
"revision_criteria": payload["claims"][0]["revision_criteria"],
|
||||||
|
"revision_kind": "original",
|
||||||
|
"supersedes_id": None,
|
||||||
|
"confidence": None,
|
||||||
|
"tags": ["v3", "replay"],
|
||||||
|
"created_by": agent_id,
|
||||||
|
"superseded_by": None,
|
||||||
|
"accepted_by_proposal_id": PROPOSAL_ID,
|
||||||
|
"created_at": "2026-07-18T10:01:00+00:00",
|
||||||
|
"updated_at": "2026-07-18T10:01:00+00:00",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"sources": [
|
||||||
|
{
|
||||||
|
"id": SOURCE_A,
|
||||||
|
"source_type": "paper",
|
||||||
|
"url": payload["sources"][0]["canonical_url"],
|
||||||
|
"storage_path": None,
|
||||||
|
"excerpt": payload["sources"][0]["excerpt"],
|
||||||
|
"hash": "a" * 64,
|
||||||
|
"created_by": agent_id,
|
||||||
|
"captured_at": payload["sources"][0]["captured_at"],
|
||||||
|
"canonical_url": payload["sources"][0]["canonical_url"],
|
||||||
|
"storage_uri": None,
|
||||||
|
"content_hash": "a" * 64,
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"ingestion_origin": "operator_upload",
|
||||||
|
"provenance_status": "verified",
|
||||||
|
"source_class": "primary_record",
|
||||||
|
"auto_promotion_policy": "human_required",
|
||||||
|
"accepted_by_proposal_id": PROPOSAL_ID,
|
||||||
|
"created_at": "2026-07-18T10:01:00+00:00",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"claim_evidence": [
|
||||||
|
{
|
||||||
|
"id": evidence_id,
|
||||||
|
"claim_id": CLAIM_A,
|
||||||
|
"source_id": SOURCE_A,
|
||||||
|
"role": "grounds",
|
||||||
|
"weight": None,
|
||||||
|
"created_by": agent_id,
|
||||||
|
"excerpt": evidence["excerpt"],
|
||||||
|
"locator_json": evidence["locator_json"],
|
||||||
|
"source_content_hash": "a" * 64,
|
||||||
|
"rationale": evidence["rationale"],
|
||||||
|
"polarity": "supports",
|
||||||
|
"accepted_from_proposal_id": PROPOSAL_ID,
|
||||||
|
"created_at": "2026-07-18T10:01:00+00:00",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"claim_edges": [],
|
||||||
|
"claim_evidence_assessments": [
|
||||||
|
{
|
||||||
|
**payload["assessments"][0],
|
||||||
|
"accepted_by_proposal_id": PROPOSAL_ID,
|
||||||
|
"created_at": "2026-07-18T10:01:00+00:00",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"reasoning_tools": [],
|
||||||
|
}
|
||||||
|
return payload, rows
|
||||||
|
|
||||||
|
|
||||||
def test_add_edge_postflight_query_selects_exact_semantic_key() -> None:
|
def test_add_edge_postflight_query_selects_exact_semantic_key() -> None:
|
||||||
sql = receipt.build_postflight_sql(_edge_proposal())
|
sql = receipt.build_postflight_sql(_edge_proposal())
|
||||||
|
assert sql.startswith("set timezone = 'UTC';")
|
||||||
assert "public.claim_edges" in sql
|
assert "public.claim_edges" in sql
|
||||||
assert CLAIM_A in sql
|
assert CLAIM_A in sql
|
||||||
assert CLAIM_B in sql
|
assert CLAIM_B in sql
|
||||||
|
|
@ -144,6 +290,42 @@ def test_receipt_hash_is_recoverable_and_independent_of_export_time() -> None:
|
||||||
assert first["exported_at_utc"] != second["exported_at_utc"]
|
assert first["exported_at_utc"] != second["exported_at_utc"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_fractional_timestamp_spellings_normalize_before_semantic_receipt_comparison() -> None:
|
||||||
|
payload, rows = _v3_bundle_and_rows()
|
||||||
|
payload["sources"][0]["captured_at"] = "2026-07-18T12:00:00.123400+02:00"
|
||||||
|
rows["sources"][0]["captured_at"] = "2026-07-18T10:00:00.1234Z"
|
||||||
|
proposal = _applied("approve_claim", payload)
|
||||||
|
proposal["reviewed_at"] = "2026-07-18T12:01:00.5+02:00"
|
||||||
|
proposal["applied_at"] = "2026-07-18T05:01:01.000001-05:00"
|
||||||
|
|
||||||
|
data = _build_receipt(
|
||||||
|
proposal,
|
||||||
|
rows,
|
||||||
|
exported_at_utc="2026-07-18T12:02:00.25+02:00",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert data["proposal"]["reviewed_at"] == "2026-07-18T10:01:00.500000Z"
|
||||||
|
assert data["proposal"]["applied_at"] == "2026-07-18T10:01:01.000001Z"
|
||||||
|
assert data["canonical_rows"]["sources"][0]["captured_at"] == "2026-07-18T10:00:00.123400Z"
|
||||||
|
assert data["exported_at_utc"] == "2026-07-18T10:02:00.250000Z"
|
||||||
|
assert receipt.validate_replay_receipt(data)["replay_ready"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_uuid_spellings_canonicalize_without_postcommit_receipt_loss() -> None:
|
||||||
|
proposal = _edge_proposal()
|
||||||
|
proposal["payload"]["apply_payload"]["from_claim"] = CLAIM_A.upper()
|
||||||
|
proposal["payload"]["apply_payload"]["to_claim"] = "{" + CLAIM_B + "}"
|
||||||
|
row = _edge_row()
|
||||||
|
row["from_claim"] = CLAIM_A
|
||||||
|
row["to_claim"] = CLAIM_B
|
||||||
|
|
||||||
|
data = _build_receipt(proposal, {"claim_edges": [row]})
|
||||||
|
|
||||||
|
assert data["canonical_rows"]["claim_edges"][0]["from_claim"] == CLAIM_A
|
||||||
|
assert data["canonical_rows"]["claim_edges"][0]["to_claim"] == CLAIM_B
|
||||||
|
assert receipt.validate_replay_receipt(data)["replay_ready"] is True
|
||||||
|
|
||||||
|
|
||||||
def test_attach_evidence_receipt_matches_every_payload_controlled_field() -> None:
|
def test_attach_evidence_receipt_matches_every_payload_controlled_field() -> None:
|
||||||
proposal = _applied(
|
proposal = _applied(
|
||||||
"attach_evidence",
|
"attach_evidence",
|
||||||
|
|
@ -271,6 +453,39 @@ def test_approve_claim_receipt_matches_all_supported_canonical_row_types() -> No
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_approve_claim_receipt_binds_every_admitted_row_family() -> None:
|
||||||
|
payload, rows = _v3_bundle_and_rows()
|
||||||
|
proposal = _applied("approve_claim", payload)
|
||||||
|
|
||||||
|
sql = receipt.build_postflight_sql(proposal)
|
||||||
|
for table in (
|
||||||
|
"public.claims",
|
||||||
|
"public.sources",
|
||||||
|
"public.claim_evidence",
|
||||||
|
"public.claim_edges",
|
||||||
|
"public.claim_evidence_assessments",
|
||||||
|
):
|
||||||
|
assert table in sql
|
||||||
|
|
||||||
|
data = _build_receipt(proposal, rows)
|
||||||
|
assert data["expected_row_counts"] == {
|
||||||
|
"claims": 1,
|
||||||
|
"sources": 1,
|
||||||
|
"claim_evidence": 1,
|
||||||
|
"claim_edges": 0,
|
||||||
|
"claim_evidence_assessments": 1,
|
||||||
|
"reasoning_tools": 0,
|
||||||
|
}
|
||||||
|
assert data["checks"]["semantic_rows_match_strict_payload"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_approve_claim_receipt_rejects_assessment_drift() -> None:
|
||||||
|
payload, rows = _v3_bundle_and_rows()
|
||||||
|
rows["claim_evidence_assessments"][0]["evidence_set_hash"] = "b" * 64
|
||||||
|
with pytest.raises(ValueError, match="semantic mismatch"):
|
||||||
|
_build_receipt(_applied("approve_claim", payload), rows)
|
||||||
|
|
||||||
|
|
||||||
def test_revise_strategy_receipt_requires_the_fresh_active_rows() -> None:
|
def test_revise_strategy_receipt_requires_the_fresh_active_rows() -> None:
|
||||||
agent_id = "11111111-1111-4111-8111-111111111111"
|
agent_id = "11111111-1111-4111-8111-111111111111"
|
||||||
payload = {
|
payload = {
|
||||||
|
|
@ -337,6 +552,14 @@ def test_receipt_refuses_legacy_applied_proposal_without_strict_payload() -> Non
|
||||||
receipt.build_postflight_sql(proposal)
|
receipt.build_postflight_sql(proposal)
|
||||||
|
|
||||||
|
|
||||||
|
def test_receipt_refuses_explicit_null_legacy_contract_version() -> None:
|
||||||
|
proposal = _edge_proposal()
|
||||||
|
proposal["payload"]["apply_payload"]["contract_version"] = None
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="must omit contract_version"):
|
||||||
|
receipt.build_postflight_sql(proposal)
|
||||||
|
|
||||||
|
|
||||||
def test_private_receipt_is_atomically_written_mode_0600(tmp_path: Path) -> None:
|
def test_private_receipt_is_atomically_written_mode_0600(tmp_path: Path) -> None:
|
||||||
data = _build_receipt(_edge_proposal(), {"claim_edges": [_edge_row()]})
|
data = _build_receipt(_edge_proposal(), {"claim_edges": [_edge_row()]})
|
||||||
output = tmp_path / "private" / "receipt.json"
|
output = tmp_path / "private" / "receipt.json"
|
||||||
|
|
@ -379,6 +602,7 @@ def test_apply_capture_writes_receipt_without_printing_canonical_rows(
|
||||||
assert data["replay_ready"] is True
|
assert data["replay_ready"] is True
|
||||||
assert data["apply_engine"]["source"] == "reconstructed_current_engine"
|
assert data["apply_engine"]["source"] == "reconstructed_current_engine"
|
||||||
assert len(seen) == 1
|
assert len(seen) == 1
|
||||||
|
assert seen[0].startswith("set timezone = 'UTC';")
|
||||||
assert "public.claim_edges" in seen[0]
|
assert "public.claim_edges" in seen[0]
|
||||||
|
|
||||||
args.receipt_out = str(tmp_path / "exact-executed.json")
|
args.receipt_out = str(tmp_path / "exact-executed.json")
|
||||||
|
|
|
||||||
|
|
@ -1,13 +1,16 @@
|
||||||
"""Tests for scripts/kb_proposal_review_packet.py."""
|
"""Tests for scripts/kb_proposal_review_packet.py."""
|
||||||
|
|
||||||
from pathlib import Path
|
import copy
|
||||||
import sys
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||||
sys.path.insert(0, str(REPO_ROOT / "scripts"))
|
sys.path.insert(0, str(REPO_ROOT / "scripts"))
|
||||||
|
|
||||||
import kb_proposal_review_packet as rp # noqa: E402
|
import kb_proposal_review_packet as rp # noqa: E402
|
||||||
|
|
||||||
|
from tests import test_compile_kb_source_packet as source_compiler_fixture # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
def test_approved_freeform_packet_is_not_worker_applyable():
|
def test_approved_freeform_packet_is_not_worker_applyable():
|
||||||
proposal = {
|
proposal = {
|
||||||
|
|
@ -73,3 +76,32 @@ def test_pending_packet_needs_human_review():
|
||||||
assert packet["review_state"] == "needs_human_review"
|
assert packet["review_state"] == "needs_human_review"
|
||||||
assert packet["worker_applyable"] is False
|
assert packet["worker_applyable"] is False
|
||||||
assert "approve, reject, or request edits" in packet["next_admin_action"]
|
assert "approve, reject, or request edits" in packet["next_admin_action"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_source_packet_exposes_exact_rows_normalization_and_hashes(tmp_path: Path):
|
||||||
|
bundle = source_compiler_fixture._compile_v3(tmp_path / "source")
|
||||||
|
proposal = copy.deepcopy(bundle["strict_child_proposal"])
|
||||||
|
|
||||||
|
packet = rp.classify_proposal(proposal)
|
||||||
|
|
||||||
|
apply_payload = proposal["payload"]["apply_payload"]
|
||||||
|
assert packet["review_state"] == "needs_human_review"
|
||||||
|
assert packet["payload_summary"]["contract_version"] == 3
|
||||||
|
assert packet["payload_summary"]["strict_row_counts"] == {
|
||||||
|
"assessments": 1,
|
||||||
|
"claims": 1,
|
||||||
|
"edges": 0,
|
||||||
|
"evidence": 1,
|
||||||
|
"reasoning_tools": 0,
|
||||||
|
"sources": 1,
|
||||||
|
}
|
||||||
|
assert packet["strict_review_material"]["rows"]["claims"] == apply_payload["claims"]
|
||||||
|
assert packet["strict_review_material"]["rows"]["sources"] == apply_payload["sources"]
|
||||||
|
assert packet["strict_review_material"]["rows"]["evidence"] == apply_payload["evidence"]
|
||||||
|
assert packet["strict_review_material"]["rows"]["assessments"] == apply_payload["assessments"]
|
||||||
|
assert (
|
||||||
|
packet["strict_review_material"]["normalization_manifest"]
|
||||||
|
== apply_payload["normalization_manifest"]
|
||||||
|
)
|
||||||
|
assert packet["strict_review_material"]["apply_payload_sha256"] == rp.canonical_sha256(apply_payload)
|
||||||
|
assert packet["strict_review_material"]["proposal_payload_sha256"] == rp.canonical_sha256(proposal["payload"])
|
||||||
|
|
|
||||||
|
|
@ -416,6 +416,13 @@ def test_fresh_preflight_requires_every_owned_target_to_be_absent_and_is_determi
|
||||||
lifecycle.build_fresh_preflight(proposal, incomplete)
|
lifecycle.build_fresh_preflight(proposal, incomplete)
|
||||||
|
|
||||||
|
|
||||||
|
def test_compensating_rollback_refuses_immutable_v3_admissions() -> None:
|
||||||
|
proposal = _proposal_before()
|
||||||
|
proposal["payload"]["apply_payload"]["contract_version"] = 3
|
||||||
|
with pytest.raises(ValueError, match="V3 admitted rows are immutable"):
|
||||||
|
lifecycle.build_fresh_preflight(proposal, _empty_targets())
|
||||||
|
|
||||||
|
|
||||||
def test_fresh_owned_apply_consumes_timestamped_preflight_and_never_adopts_conflicts():
|
def test_fresh_owned_apply_consumes_timestamped_preflight_and_never_adopts_conflicts():
|
||||||
proposal, _approval, preflight, _receipt, _rollback_sql = _materials()
|
proposal, _approval, preflight, _receipt, _rollback_sql = _materials()
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,9 +1,12 @@
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
import json
|
import json
|
||||||
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
|
import uuid
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
@ -45,6 +48,30 @@ def _docker_available() -> bool:
|
||||||
return result.returncode == 0
|
return result.returncode == 0
|
||||||
|
|
||||||
|
|
||||||
|
def _commit_test_repository(path: Path, files: dict[str, str]) -> None:
|
||||||
|
subprocess.run(["git", "init", "-q", str(path)], check=True)
|
||||||
|
for relative_path, contents in files.items():
|
||||||
|
target = path / relative_path
|
||||||
|
target.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
target.write_text(contents, encoding="utf-8")
|
||||||
|
subprocess.run(["git", "-C", str(path), "add", "."], check=True)
|
||||||
|
subprocess.run(
|
||||||
|
[
|
||||||
|
"git",
|
||||||
|
"-C",
|
||||||
|
str(path),
|
||||||
|
"-c",
|
||||||
|
"user.name=Lifecycle Test",
|
||||||
|
"-c",
|
||||||
|
"user.email=lifecycle-test@example.invalid",
|
||||||
|
"commit",
|
||||||
|
"-qm",
|
||||||
|
"test engine",
|
||||||
|
],
|
||||||
|
check=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_source_packet_recompiles_to_retained_expectation(tmp_path: Path) -> None:
|
def test_source_packet_recompiles_to_retained_expectation(tmp_path: Path) -> None:
|
||||||
packet = _write_source_packet(tmp_path)
|
packet = _write_source_packet(tmp_path)
|
||||||
|
|
||||||
|
|
@ -67,6 +94,34 @@ def test_source_packet_refuses_retained_packet_drift(tmp_path: Path) -> None:
|
||||||
lifecycle.compile_source_packet(packet)
|
lifecycle.compile_source_packet(packet)
|
||||||
|
|
||||||
|
|
||||||
|
def test_execution_identity_requires_exact_committed_dependency_bytes(
|
||||||
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
repo = tmp_path / "repo"
|
||||||
|
repo.mkdir()
|
||||||
|
_commit_test_repository(repo, {"scripts/engine.py": "print('committed')\n"})
|
||||||
|
engine = repo / "scripts" / "engine.py"
|
||||||
|
monkeypatch.setattr(lifecycle, "REPO_ROOT", repo)
|
||||||
|
monkeypatch.setattr(lifecycle, "ENGINE_INPUT_PATHS", {"engine.runner": engine})
|
||||||
|
|
||||||
|
identity = lifecycle.execution_identity()
|
||||||
|
assert identity["engine_paths_match_head"] is True
|
||||||
|
assert identity["engine_relpaths"] == ["scripts/engine.py"]
|
||||||
|
assert identity["engine_head_sha256"]["engine.runner"] == hashlib.sha256(engine.read_bytes()).hexdigest()
|
||||||
|
|
||||||
|
engine.write_text("print('dirty')\n", encoding="utf-8")
|
||||||
|
with pytest.raises(lifecycle.LifecycleError, match="does not match"):
|
||||||
|
lifecycle.execution_identity()
|
||||||
|
|
||||||
|
|
||||||
|
def test_engine_dependency_closure_includes_transitive_imports() -> None:
|
||||||
|
assert {
|
||||||
|
"engine.proposal_normalizer",
|
||||||
|
"engine.rich_proposal_plan",
|
||||||
|
"engine.clone_checkpoint",
|
||||||
|
} <= lifecycle.ENGINE_INPUT_PATHS.keys()
|
||||||
|
|
||||||
|
|
||||||
def test_exact_identities_adds_observed_evidence_ids(tmp_path: Path) -> None:
|
def test_exact_identities_adds_observed_evidence_ids(tmp_path: Path) -> None:
|
||||||
packet = _write_source_packet(tmp_path)
|
packet = _write_source_packet(tmp_path)
|
||||||
compiled, _expected, _paths = lifecycle.compile_source_packet(packet)
|
compiled, _expected, _paths = lifecycle.compile_source_packet(packet)
|
||||||
|
|
@ -106,6 +161,598 @@ def test_lifecycle_requires_explicit_approve_action(tmp_path: Path) -> None:
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _application_role(*, complete_settings: bool = True) -> dict:
|
||||||
|
role = {
|
||||||
|
"name": "kb_apply",
|
||||||
|
"can_login": True,
|
||||||
|
"superuser": False,
|
||||||
|
"create_db": False,
|
||||||
|
"create_role": False,
|
||||||
|
"inherit": False,
|
||||||
|
"replication": False,
|
||||||
|
"bypass_rls": False,
|
||||||
|
"connection_limit": -1,
|
||||||
|
"default_transaction_read_only": None,
|
||||||
|
}
|
||||||
|
if complete_settings:
|
||||||
|
role["settings"] = []
|
||||||
|
role["database_settings"] = []
|
||||||
|
return role
|
||||||
|
|
||||||
|
|
||||||
|
def _minimal_loaded_manifest(*, complete_settings: bool = True) -> dict:
|
||||||
|
return {
|
||||||
|
"singleton": {
|
||||||
|
"identity": {"database": lifecycle.DATABASE, "transaction_read_only": "on"},
|
||||||
|
"application_roles": {"items": [_application_role(complete_settings=complete_settings)]},
|
||||||
|
"role_memberships": {"items": []},
|
||||||
|
"object_acl": {"items": []},
|
||||||
|
},
|
||||||
|
"tables": {"public.fixture": {"row_count": 0, "rowset_md5": "0" * 32}},
|
||||||
|
"performance": {},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _write_exporter_receipt(
|
||||||
|
path: Path,
|
||||||
|
dump: Path,
|
||||||
|
manifest_path: Path,
|
||||||
|
loaded_manifest: dict,
|
||||||
|
*,
|
||||||
|
source: dict[str, str] | None = None,
|
||||||
|
) -> dict:
|
||||||
|
source = source or {
|
||||||
|
"ssh_target": "disposable-test",
|
||||||
|
"container": "teleo-test-source",
|
||||||
|
"database": lifecycle.DATABASE,
|
||||||
|
"service": "disposable-postgres-test",
|
||||||
|
}
|
||||||
|
snapshot = {
|
||||||
|
"captured_at_utc": "2026-07-18T00:00:00+00:00",
|
||||||
|
"exported_snapshot_id": "00000003-00000001-1",
|
||||||
|
"txid_snapshot": "1:1:",
|
||||||
|
"wal_lsn": "0/1",
|
||||||
|
"system_identifier": "123456789",
|
||||||
|
}
|
||||||
|
authorization_ref = "codex-delegation:disposable-genesis-test"
|
||||||
|
manifest_sha256 = lifecycle.sha256_file(manifest_path)
|
||||||
|
dump_sha256 = lifecycle.sha256_file(dump)
|
||||||
|
manifest_sql_sha256 = lifecycle.sha256_file(lifecycle.PARITY_SQL_PATH)
|
||||||
|
source_context_sha256 = hashlib.sha256(b"disposable-source-context").hexdigest()
|
||||||
|
binding = lifecycle.snapshot_capture.build_provenance_binding(
|
||||||
|
run_id="disposable-genesis-test",
|
||||||
|
authorization_ref=authorization_ref,
|
||||||
|
source=source,
|
||||||
|
snapshot=snapshot,
|
||||||
|
dump_sha256=dump_sha256,
|
||||||
|
manifest_sql_sha256=manifest_sql_sha256,
|
||||||
|
source_manifest_sha256=manifest_sha256,
|
||||||
|
source_context_sha256=source_context_sha256,
|
||||||
|
)
|
||||||
|
service_state = {
|
||||||
|
"ActiveState": "active",
|
||||||
|
"SubState": "running",
|
||||||
|
"MainPID": "123",
|
||||||
|
"NRestarts": "0",
|
||||||
|
"ActiveEnterTimestamp": "2026-07-18T00:00:00+00:00",
|
||||||
|
}
|
||||||
|
receipt = {
|
||||||
|
"artifact": lifecycle.TRUSTED_EXPORTER_ARTIFACT,
|
||||||
|
"schema": lifecycle.snapshot_capture.SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
|
||||||
|
"receipt_version": 2,
|
||||||
|
"status": "pass",
|
||||||
|
"run_id": "disposable-genesis-test",
|
||||||
|
"capture_authorization_ref": authorization_ref,
|
||||||
|
"source": source,
|
||||||
|
"snapshot_exported": True,
|
||||||
|
"snapshot": snapshot,
|
||||||
|
"dump": {"path": str(dump), "bytes": dump.stat().st_size, "sha256": dump_sha256},
|
||||||
|
"manifest": {
|
||||||
|
"path": str(manifest_path),
|
||||||
|
"sha256": manifest_sha256,
|
||||||
|
"manifest_sql_sha256": manifest_sql_sha256,
|
||||||
|
"table_count": len(loaded_manifest["tables"]),
|
||||||
|
"total_rows": sum(int(row["row_count"]) for row in loaded_manifest["tables"].values()),
|
||||||
|
"application_roles": loaded_manifest["singleton"]["application_roles"]["items"],
|
||||||
|
},
|
||||||
|
"source_context": {"path": "source-context.txt", "sha256": source_context_sha256},
|
||||||
|
"source_service": {"before": service_state, "after": service_state, "unchanged": True},
|
||||||
|
"provenance_binding": binding,
|
||||||
|
"service_unchanged": True,
|
||||||
|
"production_db_mutated": False,
|
||||||
|
"telegram_send_attempted": False,
|
||||||
|
}
|
||||||
|
path.write_text(json.dumps(receipt, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||||
|
path.chmod(0o600)
|
||||||
|
return receipt
|
||||||
|
|
||||||
|
|
||||||
|
def _trust_test_exporter_receipt(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
path: Path,
|
||||||
|
receipt: dict,
|
||||||
|
) -> None:
|
||||||
|
registry = dict(lifecycle.TRUSTED_EXPORTER_RECEIPTS)
|
||||||
|
registry[lifecycle.sha256_file(path)] = {
|
||||||
|
"run_id": receipt["run_id"],
|
||||||
|
"capture_authorization_ref": receipt["capture_authorization_ref"],
|
||||||
|
"source": receipt["source"],
|
||||||
|
"dump_sha256": receipt["dump"]["sha256"],
|
||||||
|
"manifest_sha256": receipt["manifest"]["sha256"],
|
||||||
|
"manifest_sql_sha256": receipt["manifest"]["manifest_sql_sha256"],
|
||||||
|
"provenance_binding_sha256": receipt["provenance_binding"]["sha256"],
|
||||||
|
}
|
||||||
|
monkeypatch.setattr(lifecycle, "TRUSTED_EXPORTER_RECEIPTS", registry)
|
||||||
|
|
||||||
|
|
||||||
|
def _genesis_cli(tmp_path: Path) -> list[str]:
|
||||||
|
return [
|
||||||
|
"--source-packet",
|
||||||
|
str(tmp_path / "packet"),
|
||||||
|
"--output",
|
||||||
|
str(tmp_path / "lifecycle-receipt.json"),
|
||||||
|
"--review-action",
|
||||||
|
"approve",
|
||||||
|
"--genesis-dump",
|
||||||
|
str(tmp_path / "genesis.dump"),
|
||||||
|
"--genesis-manifest",
|
||||||
|
str(tmp_path / "manifest.jsonl"),
|
||||||
|
"--genesis-exporter-receipt",
|
||||||
|
str(tmp_path / "exporter-receipt.json"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_genesis_cli_requires_complete_exporter_receipt_set(tmp_path: Path) -> None:
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
lifecycle.parse_args(_genesis_cli(tmp_path)[:-2])
|
||||||
|
|
||||||
|
|
||||||
|
def test_genesis_cli_rejects_caller_only_hash_flags(tmp_path: Path) -> None:
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
lifecycle.parse_args([*_genesis_cli(tmp_path), "--genesis-dump-sha256", "a" * 64])
|
||||||
|
|
||||||
|
|
||||||
|
def test_genesis_cli_rejects_moving_image_tag(tmp_path: Path) -> None:
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
lifecycle.parse_args([*_genesis_cli(tmp_path), "--image", "postgres:16-alpine"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_exporter_receipt_dump_drift_fails_before_docker(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
dump = tmp_path / "genesis.dump"
|
||||||
|
manifest = tmp_path / "manifest.jsonl"
|
||||||
|
exporter_receipt = tmp_path / "exporter-receipt.json"
|
||||||
|
dump.write_bytes(b"PGDMP-real-enough-for-signature-validation")
|
||||||
|
manifest.write_text("{}\n", encoding="utf-8")
|
||||||
|
loaded_manifest = _minimal_loaded_manifest()
|
||||||
|
receipt = _write_exporter_receipt(exporter_receipt, dump, manifest, loaded_manifest)
|
||||||
|
_trust_test_exporter_receipt(monkeypatch, exporter_receipt, receipt)
|
||||||
|
dump.write_bytes(dump.read_bytes() + b"drift")
|
||||||
|
monkeypatch.setattr(lifecycle, "load_manifest", lambda _path: loaded_manifest)
|
||||||
|
|
||||||
|
with pytest.raises(lifecycle.LifecycleError, match="dump_hash_from_exporter_receipt"):
|
||||||
|
lifecycle.validate_genesis_inputs(
|
||||||
|
dump,
|
||||||
|
manifest,
|
||||||
|
exporter_receipt,
|
||||||
|
lifecycle.DEFAULT_GENESIS_IMAGE,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_exporter_receipt_rebound_source_identity_fails_closed(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
dump = tmp_path / "genesis.dump"
|
||||||
|
manifest = tmp_path / "manifest.jsonl"
|
||||||
|
exporter_receipt = tmp_path / "exporter-receipt.json"
|
||||||
|
dump.write_bytes(b"PGDMP-real-enough-for-signature-validation")
|
||||||
|
manifest.write_text("{}\n", encoding="utf-8")
|
||||||
|
loaded_manifest = _minimal_loaded_manifest()
|
||||||
|
receipt = _write_exporter_receipt(exporter_receipt, dump, manifest, loaded_manifest)
|
||||||
|
_trust_test_exporter_receipt(monkeypatch, exporter_receipt, receipt)
|
||||||
|
receipt["source"]["database"] = "attacker_rebound"
|
||||||
|
receipt["provenance_binding"] = lifecycle.snapshot_capture.build_provenance_binding(
|
||||||
|
run_id=receipt["run_id"],
|
||||||
|
authorization_ref=receipt["capture_authorization_ref"],
|
||||||
|
source=receipt["source"],
|
||||||
|
snapshot=receipt["snapshot"],
|
||||||
|
dump_sha256=receipt["dump"]["sha256"],
|
||||||
|
manifest_sql_sha256=receipt["manifest"]["manifest_sql_sha256"],
|
||||||
|
source_manifest_sha256=receipt["manifest"]["sha256"],
|
||||||
|
source_context_sha256=receipt["source_context"]["sha256"],
|
||||||
|
)
|
||||||
|
exporter_receipt.write_text(json.dumps(receipt), encoding="utf-8")
|
||||||
|
exporter_receipt.chmod(0o600)
|
||||||
|
monkeypatch.setattr(lifecycle, "load_manifest", lambda _path: loaded_manifest)
|
||||||
|
|
||||||
|
with pytest.raises(lifecycle.LifecycleError, match="trusted_exporter_receipt_hash"):
|
||||||
|
lifecycle.validate_genesis_inputs(
|
||||||
|
dump,
|
||||||
|
manifest,
|
||||||
|
exporter_receipt,
|
||||||
|
lifecycle.DEFAULT_GENESIS_IMAGE,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_manifest_role_settings_projection_is_narrow_and_nonmutating() -> None:
|
||||||
|
manifest = {
|
||||||
|
"singleton": {
|
||||||
|
"application_roles": {
|
||||||
|
"items": [
|
||||||
|
{
|
||||||
|
"name": "kb_observatory_read",
|
||||||
|
"default_transaction_read_only": "on",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "kb_apply",
|
||||||
|
"default_transaction_read_only": None,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
projected = lifecycle.manifest_for_role_replay(manifest)
|
||||||
|
|
||||||
|
assert "settings" not in manifest["singleton"]["application_roles"]["items"][0]
|
||||||
|
assert projected["singleton"]["application_roles"]["items"][0]["settings"] == [
|
||||||
|
{"name": "default_transaction_read_only", "value": "on"}
|
||||||
|
]
|
||||||
|
assert projected["singleton"]["application_roles"]["items"][0]["database_settings"] == []
|
||||||
|
assert projected["singleton"]["application_roles"]["items"][1]["settings"] == []
|
||||||
|
assert lifecycle.role_settings_capture_complete(manifest) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_private_staging_is_read_only_and_original_path_drift_is_not_consumed(tmp_path: Path) -> None:
|
||||||
|
packet = _write_source_packet(tmp_path)
|
||||||
|
dump = tmp_path / "genesis.dump"
|
||||||
|
manifest = tmp_path / "manifest.jsonl"
|
||||||
|
exporter_receipt = tmp_path / "exporter-receipt.json"
|
||||||
|
dump.write_bytes(b"PGDMP-staged-input")
|
||||||
|
manifest.write_text("{}\n", encoding="utf-8")
|
||||||
|
exporter_receipt.write_text("{}\n", encoding="utf-8")
|
||||||
|
exporter_receipt.chmod(0o600)
|
||||||
|
|
||||||
|
staged = lifecycle.stage_lifecycle_inputs(packet, dump, manifest, exporter_receipt)
|
||||||
|
try:
|
||||||
|
before = lifecycle.verify_staged_inputs(staged)
|
||||||
|
dump.write_bytes(b"PGDMP-mutated-original")
|
||||||
|
(packet / "proposal-packet.json").write_text("{}\n", encoding="utf-8")
|
||||||
|
after = lifecycle.verify_staged_inputs(staged)
|
||||||
|
|
||||||
|
assert before == after
|
||||||
|
assert staged.genesis_dump.read_bytes() == b"PGDMP-staged-input"
|
||||||
|
assert staged.root.stat().st_mode & 0o777 == 0o500
|
||||||
|
assert all(
|
||||||
|
(staged.root / row["relative_path"]).stat().st_mode & 0o777 == 0o400 for row in staged.files.values()
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
cleanup = lifecycle.cleanup_staged_inputs(staged.root)
|
||||||
|
|
||||||
|
assert cleanup["root_absent"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_private_staging_detects_consumed_byte_mutation(tmp_path: Path) -> None:
|
||||||
|
packet = _write_source_packet(tmp_path)
|
||||||
|
dump = tmp_path / "genesis.dump"
|
||||||
|
manifest = tmp_path / "manifest.jsonl"
|
||||||
|
exporter_receipt = tmp_path / "exporter-receipt.json"
|
||||||
|
dump.write_bytes(b"PGDMP-staged-input")
|
||||||
|
manifest.write_text("{}\n", encoding="utf-8")
|
||||||
|
exporter_receipt.write_text("{}\n", encoding="utf-8")
|
||||||
|
exporter_receipt.chmod(0o600)
|
||||||
|
staged = lifecycle.stage_lifecycle_inputs(packet, dump, manifest, exporter_receipt)
|
||||||
|
staged_dump = staged.genesis_dump
|
||||||
|
try:
|
||||||
|
staged_dump.chmod(0o600)
|
||||||
|
staged_dump.write_bytes(b"PGDMP-mutated-staging")
|
||||||
|
staged_dump.chmod(0o400)
|
||||||
|
with pytest.raises(lifecycle.LifecycleError, match="staging bytes drifted"):
|
||||||
|
lifecycle.verify_staged_inputs(staged)
|
||||||
|
finally:
|
||||||
|
cleanup = lifecycle.cleanup_staged_inputs(staged.root)
|
||||||
|
|
||||||
|
assert cleanup["root_absent"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_genesis_initialization_cannot_call_synthetic_bootstrap(tmp_path: Path) -> None:
|
||||||
|
class RestoreOnlyPostgres:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.restored = False
|
||||||
|
|
||||||
|
def restore_genesis(self, _inputs):
|
||||||
|
self.restored = True
|
||||||
|
return {"parity": {"status": "pass"}}
|
||||||
|
|
||||||
|
def psql(self, sql: str) -> str:
|
||||||
|
assert sql.startswith("alter role kb_review password")
|
||||||
|
return ""
|
||||||
|
|
||||||
|
postgres = RestoreOnlyPostgres()
|
||||||
|
|
||||||
|
result = lifecycle.initialize_database(
|
||||||
|
postgres,
|
||||||
|
genesis=object(),
|
||||||
|
review_password="unused",
|
||||||
|
apply_password="unused",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert postgres.restored is True
|
||||||
|
assert result["mode"] == "exporter_receipt_bound_genesis_restore"
|
||||||
|
assert result["synthetic_schema_bootstrap_skipped"] is True
|
||||||
|
assert result["canonical_payload_preseed_performed"] is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_fresh_initial_state_rejects_proposal_and_payload_preseed() -> None:
|
||||||
|
empty_rows = {family: [] for family in lifecycle.apply_lifecycle.ROW_TABLES}
|
||||||
|
with pytest.raises(lifecycle.LifecycleError, match="proposal or approval preseed"):
|
||||||
|
lifecycle.require_fresh_initial_state(
|
||||||
|
{"state": {"proposal": {"id": "preseed"}, "immutable_approval": None, "canonical_rows": empty_rows}}
|
||||||
|
)
|
||||||
|
|
||||||
|
preseed_rows = {family: [] for family in lifecycle.apply_lifecycle.ROW_TABLES}
|
||||||
|
preseed_rows["claims"] = [{"id": "preseed"}]
|
||||||
|
with pytest.raises(lifecycle.LifecycleError, match="canonical payload target preseed"):
|
||||||
|
lifecycle.require_fresh_initial_state(
|
||||||
|
{"state": {"proposal": None, "immutable_approval": None, "canonical_rows": preseed_rows}}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_payload_controlled_fingerprint_ignores_generated_fields_but_detects_payload_drift() -> None:
|
||||||
|
rows = {family: [] for family in lifecycle.PAYLOAD_CONTROLLED_FIELDS}
|
||||||
|
rows["claims"] = [
|
||||||
|
{
|
||||||
|
"id": "11111111-1111-4111-8111-111111111111",
|
||||||
|
"type": "fact",
|
||||||
|
"text": "Stable payload text",
|
||||||
|
"status": "open",
|
||||||
|
"confidence": 0.7,
|
||||||
|
"tags": ["test"],
|
||||||
|
"created_by": None,
|
||||||
|
"superseded_by": None,
|
||||||
|
"created_at": "2026-01-01T00:00:00+00:00",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
first = lifecycle.payload_controlled_fingerprint(rows)
|
||||||
|
rows["claims"][0]["created_at"] = "2027-01-01T00:00:00+00:00"
|
||||||
|
rows["claims"][0]["updated_at"] = "2027-01-01T00:00:01+00:00"
|
||||||
|
second = lifecycle.payload_controlled_fingerprint(rows)
|
||||||
|
rows["claims"][0]["text"] = "Drifted payload text"
|
||||||
|
drifted = lifecycle.payload_controlled_fingerprint(rows)
|
||||||
|
|
||||||
|
assert first["sha256"] == second["sha256"]
|
||||||
|
assert drifted["sha256"] != first["sha256"]
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_genesis_run(run_number: int) -> dict:
|
||||||
|
controlled = {
|
||||||
|
"schema": "livingip.leoPayloadControlledRowsFingerprint.v1",
|
||||||
|
"sha256": "c" * 64,
|
||||||
|
"table_sha256": {family: "d" * 64 for family in lifecycle.PAYLOAD_CONTROLLED_FIELDS},
|
||||||
|
"row_counts": {family: 0 for family in lifecycle.PAYLOAD_CONTROLLED_FIELDS},
|
||||||
|
"rows": {family: [] for family in lifecycle.PAYLOAD_CONTROLLED_FIELDS},
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
"status": "pass",
|
||||||
|
"run_number": run_number,
|
||||||
|
"source": {
|
||||||
|
"artifact_sha256": "1" * 64,
|
||||||
|
"manifest_canonical_sha256": "2" * 64,
|
||||||
|
"retained_proposal_packet_file_sha256": "3" * 64,
|
||||||
|
"retained_proposal_packet_canonical_sha256": "4" * 64,
|
||||||
|
},
|
||||||
|
"compiled": {"bundle_sha256": "4" * 64},
|
||||||
|
"identities": {
|
||||||
|
"proposal_id": "11111111-1111-4111-8111-111111111111",
|
||||||
|
"claim_ids": [],
|
||||||
|
"source_ids": [],
|
||||||
|
"planned_evidence_keys": [],
|
||||||
|
"evidence_ids": [f"generated-{run_number}"],
|
||||||
|
"edge_ids": [],
|
||||||
|
},
|
||||||
|
"phases": {
|
||||||
|
"staged_pending_review": {"proposal": {"status": "pending_review"}},
|
||||||
|
"approved_unapplied": {"proposal": {"status": "approved"}},
|
||||||
|
"applied": {
|
||||||
|
"proposal": {"status": "applied", "applied_at": f"2026-01-0{run_number}T00:00:00+00:00"},
|
||||||
|
"payload_controlled_fingerprint": controlled,
|
||||||
|
"worker_action": {
|
||||||
|
"exact_candidate_id": "11111111-1111-4111-8111-111111111111",
|
||||||
|
"credential_role": "kb_apply",
|
||||||
|
"retry_failure_count": 1,
|
||||||
|
"failure_state_after_success": {},
|
||||||
|
"private_receipt_mode": "0600",
|
||||||
|
},
|
||||||
|
"apply_receipt": {
|
||||||
|
"hashes": {"apply_payload_sha256": "5" * 64},
|
||||||
|
"expected_row_counts": controlled["row_counts"],
|
||||||
|
"actual_row_counts": controlled["row_counts"],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"recompiled_readback": {
|
||||||
|
"bundle_sha256": "4" * 64,
|
||||||
|
"matches_initial_compile": True,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"database_initialization": {
|
||||||
|
"genesis": {
|
||||||
|
"dump": {"sha256": "6" * 64},
|
||||||
|
"manifest": {"sha256": "7" * 64},
|
||||||
|
"exporter_receipt": {
|
||||||
|
"sha256": "8" * 64,
|
||||||
|
"provenance_binding_sha256": "9" * 64,
|
||||||
|
},
|
||||||
|
"image": {"reference": lifecycle.DEFAULT_GENESIS_IMAGE},
|
||||||
|
"parity": {
|
||||||
|
"status": "pass",
|
||||||
|
"problems": [],
|
||||||
|
"claim_scope": "full_captured_manifest_including_role_and_database_settings",
|
||||||
|
"role_global_parity": {
|
||||||
|
"status": "proven",
|
||||||
|
"captured_source_settings_present": True,
|
||||||
|
"reason": None,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"key_counts": {"public.claims": 10},
|
||||||
|
"canonical_schema_table_count": 10,
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"cleanup": {"all": True},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _mocked_genesis_case(tmp_path: Path) -> tuple[Path, Path, Path, Path]:
|
||||||
|
packet = _write_source_packet(tmp_path)
|
||||||
|
dump = tmp_path / "genesis.dump"
|
||||||
|
manifest = tmp_path / "genesis-manifest.jsonl"
|
||||||
|
exporter_receipt = tmp_path / "exporter-receipt.json"
|
||||||
|
dump.write_bytes(b"PGDMPx")
|
||||||
|
manifest.write_text("{}\n", encoding="utf-8")
|
||||||
|
_write_exporter_receipt(exporter_receipt, dump, manifest, _minimal_loaded_manifest())
|
||||||
|
return packet, dump, manifest, exporter_receipt
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_validate_genesis(
|
||||||
|
dump: Path,
|
||||||
|
manifest: Path,
|
||||||
|
exporter_receipt: Path,
|
||||||
|
image: str,
|
||||||
|
**paths,
|
||||||
|
) -> lifecycle.GenesisInputs:
|
||||||
|
payload = json.loads(exporter_receipt.read_text(encoding="utf-8"))
|
||||||
|
return lifecycle.GenesisInputs(
|
||||||
|
dump=dump,
|
||||||
|
dump_sha256=lifecycle.sha256_file(dump),
|
||||||
|
dump_bytes=dump.stat().st_size,
|
||||||
|
manifest_path=manifest,
|
||||||
|
manifest_sha256=lifecycle.sha256_file(manifest),
|
||||||
|
manifest_bytes=manifest.stat().st_size,
|
||||||
|
manifest=_minimal_loaded_manifest(),
|
||||||
|
image=image,
|
||||||
|
exporter_receipt_path=exporter_receipt,
|
||||||
|
exporter_receipt_sha256=lifecycle.sha256_file(exporter_receipt),
|
||||||
|
exporter_receipt=payload,
|
||||||
|
source_identity=payload["source"],
|
||||||
|
capture_authorization_ref=payload["capture_authorization_ref"],
|
||||||
|
provenance_binding_sha256=payload["provenance_binding"]["sha256"],
|
||||||
|
trust_root_receipt_sha256=lifecycle.sha256_file(exporter_receipt),
|
||||||
|
role_settings_capture_complete=True,
|
||||||
|
parity_sql_path=paths["parity_sql_path"],
|
||||||
|
guard_prerequisites_path=paths["guard_prerequisites_path"],
|
||||||
|
approve_script_path=paths["approve_script_path"],
|
||||||
|
apply_worker_script_path=paths["apply_worker_script_path"],
|
||||||
|
apply_script_path=paths["apply_script_path"],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_genesis_lifecycle_repeats_semantically_and_writes_private_receipt(
|
||||||
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
packet, dump, manifest, exporter_receipt = _mocked_genesis_case(tmp_path)
|
||||||
|
calls: list[int] = []
|
||||||
|
|
||||||
|
monkeypatch.setattr(lifecycle, "validate_genesis_inputs", _fake_validate_genesis)
|
||||||
|
|
||||||
|
def fake_run(*_args, run_number: int, **_kwargs):
|
||||||
|
calls.append(run_number)
|
||||||
|
return _fake_genesis_run(run_number)
|
||||||
|
|
||||||
|
monkeypatch.setattr(lifecycle, "run_lifecycle", fake_run)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
lifecycle,
|
||||||
|
"verify_run_group_cleanup",
|
||||||
|
lambda run_group: {"checked": True, "run_group": run_group, "container_names": [], "all_absent": True},
|
||||||
|
)
|
||||||
|
output = tmp_path / "genesis-lifecycle-receipt.json"
|
||||||
|
|
||||||
|
receipt = lifecycle.run_genesis_lifecycle(
|
||||||
|
packet,
|
||||||
|
output,
|
||||||
|
review_action="approve",
|
||||||
|
genesis_dump=dump,
|
||||||
|
genesis_manifest=manifest,
|
||||||
|
genesis_exporter_receipt=exporter_receipt,
|
||||||
|
image=lifecycle.DEFAULT_GENESIS_IMAGE,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert calls == [1, 2]
|
||||||
|
assert receipt["status"] == "pass"
|
||||||
|
assert receipt["repeatability"]["semantic_match"] is True
|
||||||
|
assert len(set(receipt["repeatability"]["semantic_fingerprints_sha256"])) == 1
|
||||||
|
assert receipt["repeatability"]["identical_post_apply_fingerprints"] is True
|
||||||
|
assert receipt["checks"]["both_runs_consumed_preflight_bound_bytes"] is True
|
||||||
|
assert receipt["cleanup"]["zero_staging_leftovers"] is True
|
||||||
|
assert receipt["cleanup"]["zero_orphan_containers"] is True
|
||||||
|
assert output.stat().st_mode & 0o777 == 0o600
|
||||||
|
assert json.loads(output.read_text(encoding="utf-8")) == receipt
|
||||||
|
|
||||||
|
|
||||||
|
def test_genesis_lifecycle_fails_closed_on_semantic_repeatability_drift(
|
||||||
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
packet, dump, manifest, exporter_receipt = _mocked_genesis_case(tmp_path)
|
||||||
|
monkeypatch.setattr(lifecycle, "validate_genesis_inputs", _fake_validate_genesis)
|
||||||
|
|
||||||
|
def drifted_run(*_args, run_number: int, **_kwargs):
|
||||||
|
run = _fake_genesis_run(run_number)
|
||||||
|
if run_number == 2:
|
||||||
|
run["phases"]["applied"]["payload_controlled_fingerprint"]["sha256"] = "e" * 64
|
||||||
|
return run
|
||||||
|
|
||||||
|
monkeypatch.setattr(lifecycle, "run_lifecycle", drifted_run)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
lifecycle,
|
||||||
|
"verify_run_group_cleanup",
|
||||||
|
lambda run_group: {"checked": True, "run_group": run_group, "container_names": [], "all_absent": True},
|
||||||
|
)
|
||||||
|
|
||||||
|
receipt = lifecycle.run_genesis_lifecycle(
|
||||||
|
packet,
|
||||||
|
tmp_path / "repeatability-drift-receipt.json",
|
||||||
|
review_action="approve",
|
||||||
|
genesis_dump=dump,
|
||||||
|
genesis_manifest=manifest,
|
||||||
|
genesis_exporter_receipt=exporter_receipt,
|
||||||
|
image=lifecycle.DEFAULT_GENESIS_IMAGE,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert receipt["status"] == "fail"
|
||||||
|
assert receipt["checks"]["both_runs_passed"] is True
|
||||||
|
assert receipt["checks"]["both_runs_match_semantically"] is False
|
||||||
|
assert receipt["checks"]["identical_post_apply_fingerprints"] is False
|
||||||
|
assert len(set(receipt["repeatability"]["semantic_fingerprints_sha256"])) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_genesis_lifecycle_fails_closed_on_orphan_cleanup(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
packet, dump, manifest, exporter_receipt = _mocked_genesis_case(tmp_path)
|
||||||
|
monkeypatch.setattr(lifecycle, "validate_genesis_inputs", _fake_validate_genesis)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
lifecycle,
|
||||||
|
"run_lifecycle",
|
||||||
|
lambda *_args, run_number, **_kwargs: _fake_genesis_run(run_number),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
lifecycle,
|
||||||
|
"verify_run_group_cleanup",
|
||||||
|
lambda run_group: {
|
||||||
|
"checked": True,
|
||||||
|
"run_group": run_group,
|
||||||
|
"container_names": ["orphaned-container"],
|
||||||
|
"all_absent": False,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
receipt = lifecycle.run_genesis_lifecycle(
|
||||||
|
packet,
|
||||||
|
tmp_path / "failed-cleanup-receipt.json",
|
||||||
|
review_action="approve",
|
||||||
|
genesis_dump=dump,
|
||||||
|
genesis_manifest=manifest,
|
||||||
|
genesis_exporter_receipt=exporter_receipt,
|
||||||
|
image=lifecycle.DEFAULT_GENESIS_IMAGE,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert receipt["status"] == "fail"
|
||||||
|
assert receipt["checks"]["zero_orphan_containers"] is False
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.skipif(not _docker_available(), reason="Docker daemon is required for the isolated lifecycle")
|
@pytest.mark.skipif(not _docker_available(), reason="Docker daemon is required for the isolated lifecycle")
|
||||||
def test_disposable_postgres_lifecycle_end_to_end(tmp_path: Path) -> None:
|
def test_disposable_postgres_lifecycle_end_to_end(tmp_path: Path) -> None:
|
||||||
packet = _write_source_packet(tmp_path)
|
packet = _write_source_packet(tmp_path)
|
||||||
|
|
@ -124,10 +771,143 @@ def test_disposable_postgres_lifecycle_end_to_end(tmp_path: Path) -> None:
|
||||||
assert receipt["phases"]["approved_unapplied"]["proposal"]["status"] == "approved"
|
assert receipt["phases"]["approved_unapplied"]["proposal"]["status"] == "approved"
|
||||||
assert receipt["phases"]["approved_unapplied"]["proposal"]["applied_at"] is None
|
assert receipt["phases"]["approved_unapplied"]["proposal"]["applied_at"] is None
|
||||||
assert receipt["phases"]["applied"]["proposal"]["status"] == "applied"
|
assert receipt["phases"]["applied"]["proposal"]["status"] == "applied"
|
||||||
|
worker = receipt["phases"]["applied"]["worker_action"]
|
||||||
|
assert worker["report_only"]["expected_marker_present"] is True
|
||||||
|
assert worker["expected_retry_failure"]["returncode"] == 1
|
||||||
|
assert worker["retry_failure_count"] == 1
|
||||||
|
assert worker["failure_state_after_success"] == {}
|
||||||
|
assert worker["enabled_retry"]["expected_marker_present"] is True
|
||||||
|
assert worker["post_apply_exact_refusal"]["returncode"] == 1
|
||||||
|
assert worker["post_apply_exact_refusal"]["expected_refusal_marker_present"] is True
|
||||||
|
assert worker["private_receipt_mode"] == "0600"
|
||||||
assert receipt["phases"]["applied"]["apply_receipt"]["replay_ready"] is True
|
assert receipt["phases"]["applied"]["apply_receipt"]["replay_ready"] is True
|
||||||
assert receipt["phases"]["recompiled_readback"]["matches_initial_compile"] is True
|
assert receipt["phases"]["recompiled_readback"]["matches_initial_compile"] is True
|
||||||
assert receipt["identities"]["evidence_ids"]
|
assert receipt["identities"]["evidence_ids"]
|
||||||
assert len(receipt["checks"]) >= 20
|
assert len(receipt["checks"]) >= 20
|
||||||
assert all(receipt["checks"].values())
|
assert all(receipt["checks"].values())
|
||||||
assert receipt["cleanup"]["all"] is True
|
assert receipt["cleanup"]["all"] is True
|
||||||
|
assert output.stat().st_mode & 0o777 == 0o600
|
||||||
|
assert json.loads(output.read_text(encoding="utf-8")) == receipt
|
||||||
|
|
||||||
|
|
||||||
|
def _build_real_docker_genesis(tmp_path: Path) -> tuple[Path, Path, Path]:
|
||||||
|
run_id = uuid.uuid4().hex[:10]
|
||||||
|
source = lifecycle.DisposablePostgres(
|
||||||
|
f"leo-real-genesis-source-{os.getpid()}-{run_id}",
|
||||||
|
"disposable-admin-password",
|
||||||
|
image=lifecycle.DEFAULT_GENESIS_IMAGE,
|
||||||
|
genesis_backed=True,
|
||||||
|
run_group=f"leo-real-genesis-source-{run_id}",
|
||||||
|
)
|
||||||
|
dump = tmp_path / "real-genesis.dump"
|
||||||
|
manifest_path = tmp_path / "real-genesis-manifest.jsonl"
|
||||||
|
exporter_receipt = tmp_path / "real-genesis-exporter-receipt.json"
|
||||||
|
try:
|
||||||
|
source.start()
|
||||||
|
source.psql(lifecycle.bootstrap_sql("review-password", "apply-password"))
|
||||||
|
source.psql(
|
||||||
|
"alter role kb_apply set statement_timeout to '17s';\n"
|
||||||
|
"alter role kb_review in database teleo set lock_timeout to '19s';"
|
||||||
|
)
|
||||||
|
manifest_output = source.psql(lifecycle.PARITY_SQL_PATH.read_text(encoding="utf-8"))
|
||||||
|
manifest_path.write_text(manifest_output + "\n", encoding="utf-8")
|
||||||
|
manifest_path.chmod(0o600)
|
||||||
|
container_dump = "/tmp/leo-real-genesis.dump"
|
||||||
|
source._run(
|
||||||
|
[
|
||||||
|
"exec",
|
||||||
|
source.name,
|
||||||
|
"pg_dump",
|
||||||
|
"-U",
|
||||||
|
"postgres",
|
||||||
|
"-d",
|
||||||
|
lifecycle.DATABASE,
|
||||||
|
"--format=custom",
|
||||||
|
"--compress=9",
|
||||||
|
"--no-owner",
|
||||||
|
f"--file={container_dump}",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
source._run(["cp", f"{source.name}:{container_dump}", str(dump)])
|
||||||
|
dump.chmod(0o600)
|
||||||
|
loaded_manifest = lifecycle.load_manifest(manifest_path)
|
||||||
|
roles = {role["name"]: role for role in loaded_manifest["singleton"]["application_roles"]["items"]}
|
||||||
|
assert roles["kb_apply"]["settings"] == [{"name": "statement_timeout", "value": "17s"}]
|
||||||
|
assert roles["kb_review"]["database_settings"] == [{"name": "lock_timeout", "value": "19s"}]
|
||||||
|
_write_exporter_receipt(
|
||||||
|
exporter_receipt,
|
||||||
|
dump,
|
||||||
|
manifest_path,
|
||||||
|
loaded_manifest,
|
||||||
|
source={
|
||||||
|
"ssh_target": "disposable-docker-test",
|
||||||
|
"container": source.name,
|
||||||
|
"database": lifecycle.DATABASE,
|
||||||
|
"service": "disposable-postgres-test",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
cleanup = source.cleanup()
|
||||||
|
assert all(
|
||||||
|
(
|
||||||
|
cleanup["container_absent"],
|
||||||
|
cleanup["name_readback_clear"],
|
||||||
|
cleanup["instance_label_readback_clear"],
|
||||||
|
cleanup["run_group_readback_clear"],
|
||||||
|
cleanup["network_mode_was_none"],
|
||||||
|
cleanup["tmpfs_data_dir_was_present"],
|
||||||
|
cleanup["docker_volume_mounts_observed"] == [],
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return dump, manifest_path, exporter_receipt
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(not _docker_available(), reason="Docker daemon is required for the real genesis lifecycle")
|
||||||
|
def test_real_docker_genesis_restore_worker_parity_and_repeatability(
|
||||||
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
packet = _write_source_packet(tmp_path)
|
||||||
|
dump, manifest, exporter_receipt = _build_real_docker_genesis(tmp_path)
|
||||||
|
receipt_payload = json.loads(exporter_receipt.read_text(encoding="utf-8"))
|
||||||
|
_trust_test_exporter_receipt(monkeypatch, exporter_receipt, receipt_payload)
|
||||||
|
output = tmp_path / "real-genesis-lifecycle-receipt.json"
|
||||||
|
|
||||||
|
receipt = lifecycle.run_genesis_lifecycle(
|
||||||
|
packet,
|
||||||
|
output,
|
||||||
|
review_action="approve",
|
||||||
|
review_note="Approved exact Docker genesis payload for regression coverage.",
|
||||||
|
genesis_dump=dump,
|
||||||
|
genesis_manifest=manifest,
|
||||||
|
genesis_exporter_receipt=exporter_receipt,
|
||||||
|
image=lifecycle.DEFAULT_GENESIS_IMAGE,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert receipt["status"] == "pass", receipt.get("error")
|
||||||
|
assert receipt["execution"]["git_head"]
|
||||||
|
assert receipt["execution"]["engine_set_sha256"]
|
||||||
|
assert receipt["execution"]["engine_paths_match_head"] is True
|
||||||
|
assert receipt["execution"]["engine_sha256"] == receipt["execution"]["engine_head_sha256"]
|
||||||
|
assert receipt["pre_docker_validation"]["trusted_exporter_receipt_validated"] is True
|
||||||
|
assert receipt["pre_docker_validation"]["committed_engine_dependency_closure_verified"] is True
|
||||||
|
assert receipt["checks"]["both_runs_consumed_preflight_bound_bytes"] is True
|
||||||
|
assert receipt["checks"]["committed_engine_dependency_closure_verified"] is True
|
||||||
|
assert receipt["checks"]["portable_receipt_has_no_absolute_paths"] is True
|
||||||
|
assert lifecycle.absolute_path_value_pointers(receipt) == []
|
||||||
|
assert receipt["repeatability"]["identical_post_apply_fingerprints"] is True
|
||||||
|
assert len(set(receipt["repeatability"]["post_apply_fingerprints_sha256"])) == 1
|
||||||
|
assert receipt["cleanup"]["zero_orphan_containers"] is True
|
||||||
|
assert receipt["cleanup"]["zero_staging_leftovers"] is True
|
||||||
|
for run in receipt["runs"]:
|
||||||
|
assert run["status"] == "pass", run.get("error")
|
||||||
|
assert (
|
||||||
|
run["consumed_inputs"]["preflight_manifest_sha256"]
|
||||||
|
== receipt["immutable_staging"]["preflight_manifest_sha256"]
|
||||||
|
)
|
||||||
|
assert run["consumed_inputs"]["executing_git_head"] == receipt["execution"]["git_head"]
|
||||||
|
assert run["consumed_inputs"]["engine_set_sha256"] == receipt["execution"]["engine_set_sha256"]
|
||||||
|
assert run["database_initialization"]["genesis"]["parity"]["role_global_parity"]["status"] == "proven"
|
||||||
|
assert run["phases"]["applied"]["worker_action"]["retry_failure_count"] == 1
|
||||||
|
assert run["phases"]["applied"]["worker_action"]["private_receipt_mode"] == "0600"
|
||||||
|
assert output.stat().st_mode & 0o777 == 0o600
|
||||||
assert json.loads(output.read_text(encoding="utf-8")) == receipt
|
assert json.loads(output.read_text(encoding="utf-8")) == receipt
|
||||||
|
|
|
||||||
620
tests/test_run_local_corpus_knowledge_rebuild.py
Normal file
620
tests/test_run_local_corpus_knowledge_rebuild.py
Normal file
|
|
@ -0,0 +1,620 @@
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import copy
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import os
|
||||||
|
import stat
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from collections.abc import Callable, Iterator
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from ops import run_local_corpus_knowledge_rebuild as rebuild
|
||||||
|
|
||||||
|
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
CLI = REPO_ROOT / "ops" / "run_local_corpus_knowledge_rebuild.py"
|
||||||
|
FIXTURE_ROOT = REPO_ROOT / "tests" / "fixtures" / "teleo_corpus_rebuild"
|
||||||
|
|
||||||
|
|
||||||
|
def _sha256(value: bytes) -> str:
|
||||||
|
return hashlib.sha256(value).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _load_json(path: Path) -> dict:
|
||||||
|
value = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
assert isinstance(value, dict)
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _run_cli(source_root: Path, output: Path, *extra: str) -> subprocess.CompletedProcess[str]:
|
||||||
|
return subprocess.run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
str(CLI),
|
||||||
|
"--source-root",
|
||||||
|
str(source_root),
|
||||||
|
"--output",
|
||||||
|
str(output),
|
||||||
|
*extra,
|
||||||
|
],
|
||||||
|
cwd=REPO_ROOT,
|
||||||
|
check=False,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _run_verify(bundle: Path) -> subprocess.CompletedProcess[str]:
|
||||||
|
return subprocess.run(
|
||||||
|
[sys.executable, str(CLI), "--verify-bundle", str(bundle)],
|
||||||
|
cwd=REPO_ROOT,
|
||||||
|
check=False,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _tree_bytes(root: Path) -> dict[str, bytes]:
|
||||||
|
return {path.relative_to(root).as_posix(): path.read_bytes() for path in sorted(root.rglob("*")) if path.is_file()}
|
||||||
|
|
||||||
|
|
||||||
|
def _all_keys(value: object) -> Iterator[str]:
|
||||||
|
if isinstance(value, dict):
|
||||||
|
for key, child in value.items():
|
||||||
|
yield key
|
||||||
|
yield from _all_keys(child)
|
||||||
|
elif isinstance(value, list):
|
||||||
|
for child in value:
|
||||||
|
yield from _all_keys(child)
|
||||||
|
|
||||||
|
|
||||||
|
def _make_source(root: Path, name: str = "source.md", content: bytes = b"# Source\n\nExact text.\n") -> Path:
|
||||||
|
root.mkdir(parents=True, exist_ok=True)
|
||||||
|
source = root / name
|
||||||
|
source.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
source.write_bytes(content)
|
||||||
|
return source
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_builds_deterministic_source_inventory_without_claims_or_proposals(tmp_path: Path) -> None:
|
||||||
|
first_output = tmp_path / "bundle-a"
|
||||||
|
second_output = tmp_path / "bundle-b"
|
||||||
|
|
||||||
|
first = _run_cli(FIXTURE_ROOT, first_output, "--mode", "source-inventory-only")
|
||||||
|
second = _run_cli(FIXTURE_ROOT, second_output)
|
||||||
|
|
||||||
|
assert first.returncode == second.returncode == 0, (first.stderr, first.stdout, second.stderr, second.stdout)
|
||||||
|
first_status = json.loads(first.stdout)
|
||||||
|
second_status = json.loads(second.stdout)
|
||||||
|
assert first_status["status"] == second_status["status"] == "pass"
|
||||||
|
assert first_status["source_count"] == second_status["source_count"] == 2
|
||||||
|
assert first_status["work_item_count"] == second_status["work_item_count"] == 3
|
||||||
|
assert first_status["bundle_tree_sha256"] == second_status["bundle_tree_sha256"]
|
||||||
|
assert _tree_bytes(first_output) == _tree_bytes(second_output)
|
||||||
|
assert rebuild.INCOMPLETE_MARKER not in _tree_bytes(first_output)
|
||||||
|
|
||||||
|
source_manifest = _load_json(first_output / rebuild.SOURCE_MANIFEST_PATH)
|
||||||
|
work_index = _load_json(first_output / rebuild.EXTRACTION_WORK_INDEX_PATH)
|
||||||
|
receipt = _load_json(first_output / rebuild.RECEIPT_PATH)
|
||||||
|
assert source_manifest["schema"] == rebuild.SOURCE_MANIFEST_SCHEMA
|
||||||
|
assert work_index["schema"] == rebuild.EXTRACTION_WORK_INDEX_SCHEMA
|
||||||
|
assert receipt["schema"] == rebuild.RECEIPT_SCHEMA
|
||||||
|
assert source_manifest["source_snapshot_semantics"] == {
|
||||||
|
"concurrent_directory_additions_or_removals_may_be_absent": True,
|
||||||
|
"directory_snapshot_atomic": False,
|
||||||
|
"guarantee": "each listed path is bound to the exact bytes copied from one stable file descriptor",
|
||||||
|
}
|
||||||
|
assert [row["relative_path"] for row in source_manifest["sources"]] == [
|
||||||
|
"forgejo/project-record.md",
|
||||||
|
"operator-notes.txt",
|
||||||
|
]
|
||||||
|
assert [row["section_count"] for row in source_manifest["sources"]] == [2, 1]
|
||||||
|
assert work_index["work_type"] == rebuild.WORK_TYPE
|
||||||
|
assert all(item["work_type"] == rebuild.WORK_TYPE for item in work_index["items"])
|
||||||
|
|
||||||
|
forbidden_keys = {"candidate", "candidates", "claim", "claims", "proposal", "proposals", "proposition"}
|
||||||
|
for document in (source_manifest, work_index, receipt):
|
||||||
|
assert forbidden_keys.isdisjoint(_all_keys(document))
|
||||||
|
for key, expected in rebuild.NO_WRITE_FLAGS.items():
|
||||||
|
assert document[key] is expected
|
||||||
|
|
||||||
|
sources_by_path = {row["relative_path"]: row for row in source_manifest["sources"]}
|
||||||
|
for relative_path, source in sources_by_path.items():
|
||||||
|
raw = (FIXTURE_ROOT / relative_path).read_bytes()
|
||||||
|
snapshot = first_output / source["snapshot_path"]
|
||||||
|
assert snapshot.read_bytes() == raw
|
||||||
|
assert source["bytes"] == len(raw)
|
||||||
|
assert source["artifact_sha256"] == _sha256(raw)
|
||||||
|
assert source["format_syntax_validated"] is False
|
||||||
|
for locator in source["sections"]:
|
||||||
|
start = locator["utf8_byte_start"]
|
||||||
|
end = locator["utf8_byte_end"]
|
||||||
|
quote = raw[start:end]
|
||||||
|
assert quote
|
||||||
|
assert locator["quote_sha256"] == _sha256(quote)
|
||||||
|
assert locator["line_start"] == raw[:start].decode("utf-8").count("\n") + 1
|
||||||
|
assert locator["line_end"] == raw[: end - 1].decode("utf-8").count("\n") + 1
|
||||||
|
|
||||||
|
artifacts = receipt["outputs"]["artifacts"]
|
||||||
|
assert receipt["outputs"]["artifact_count"] == len(artifacts)
|
||||||
|
assert receipt["hashes"]["bundle_tree_sha256"] == rebuild.canonical_sha256(artifacts)
|
||||||
|
receipt_preimage = copy.deepcopy(receipt)
|
||||||
|
receipt_preimage["hashes"].pop("receipt_content_sha256")
|
||||||
|
assert receipt["hashes"]["receipt_content_sha256"] == rebuild.canonical_sha256(receipt_preimage)
|
||||||
|
assert rebuild.verify_bundle(first_output)["status"] == "pass"
|
||||||
|
|
||||||
|
persisted = b"".join(_tree_bytes(first_output).values())
|
||||||
|
assert str(FIXTURE_ROOT).encode() not in persisted
|
||||||
|
assert str(first_output).encode() not in persisted
|
||||||
|
for path in first_output.rglob("*"):
|
||||||
|
expected_mode = 0o600 if path.is_file() else 0o700
|
||||||
|
assert stat.S_IMODE(path.stat().st_mode) == expected_mode
|
||||||
|
|
||||||
|
|
||||||
|
def test_receipt_is_published_last(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
source_root = tmp_path / "sources"
|
||||||
|
_make_source(source_root)
|
||||||
|
output = tmp_path / "bundle"
|
||||||
|
original = rebuild._write_private_json
|
||||||
|
writes: list[str] = []
|
||||||
|
|
||||||
|
def record(path: Path, value: object) -> None:
|
||||||
|
writes.append(path.name)
|
||||||
|
if path.name == rebuild.RECEIPT_PATH:
|
||||||
|
assert (output / rebuild.INCOMPLETE_MARKER).exists()
|
||||||
|
original(path, value)
|
||||||
|
|
||||||
|
monkeypatch.setattr(rebuild, "_write_private_json", record)
|
||||||
|
rebuild.build_source_inventory_bundle(source_root, output)
|
||||||
|
|
||||||
|
assert writes[-1] == rebuild.RECEIPT_PATH
|
||||||
|
assert rebuild.INCOMPLETE_MARKER not in writes
|
||||||
|
assert not (output / rebuild.INCOMPLETE_MARKER).exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_verifier_rejects_rehashed_metadata_overclaim(tmp_path: Path) -> None:
|
||||||
|
output = tmp_path / "bundle"
|
||||||
|
rebuild.build_source_inventory_bundle(FIXTURE_ROOT, output)
|
||||||
|
receipt_path = output / rebuild.RECEIPT_PATH
|
||||||
|
receipt = _load_json(receipt_path)
|
||||||
|
receipt["canonical_database_rebuilt"] = True
|
||||||
|
receipt["hashes"].pop("receipt_content_sha256")
|
||||||
|
receipt["hashes"]["receipt_content_sha256"] = rebuild.canonical_sha256(receipt)
|
||||||
|
receipt_path.write_text(json.dumps(receipt), encoding="utf-8")
|
||||||
|
|
||||||
|
completed = _run_verify(output)
|
||||||
|
|
||||||
|
assert completed.returncode == 2
|
||||||
|
assert json.loads(completed.stdout)["reason"] == "invalid_bundle_schema"
|
||||||
|
|
||||||
|
|
||||||
|
METADATA_PATHS = (
|
||||||
|
rebuild.SOURCE_MANIFEST_PATH,
|
||||||
|
rebuild.EXTRACTION_WORK_INDEX_PATH,
|
||||||
|
rebuild.RECEIPT_PATH,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _inject_duplicate_top_level_key(raw: str, _metadata_path: str) -> str:
|
||||||
|
return raw.replace("{\n", '{\n "schema": "contradictory-shadow",\n', 1)
|
||||||
|
|
||||||
|
|
||||||
|
def _inject_duplicate_nested_key(raw: str, metadata_path: str) -> str:
|
||||||
|
anchors = {
|
||||||
|
rebuild.SOURCE_MANIFEST_PATH: (
|
||||||
|
'"source_snapshot_semantics": {\n',
|
||||||
|
' "directory_snapshot_atomic": true,\n',
|
||||||
|
),
|
||||||
|
rebuild.EXTRACTION_WORK_INDEX_PATH: (
|
||||||
|
'"source_manifest": {\n',
|
||||||
|
' "path": "contradictory-shadow.json",\n',
|
||||||
|
),
|
||||||
|
rebuild.RECEIPT_PATH: (
|
||||||
|
'"hashes": {\n',
|
||||||
|
f' "bundle_tree_sha256": "{"0" * 64}",\n',
|
||||||
|
),
|
||||||
|
}
|
||||||
|
anchor, duplicate = anchors[metadata_path]
|
||||||
|
assert anchor in raw
|
||||||
|
return raw.replace(anchor, anchor + duplicate, 1)
|
||||||
|
|
||||||
|
|
||||||
|
def _inject_nonfinite(raw: str, constant: str) -> str:
|
||||||
|
return raw.replace("{\n", f'{{\n "nonfinite_shadow": {constant},\n', 1)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("metadata_path", METADATA_PATHS)
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("mutator", "expected_reason"),
|
||||||
|
(
|
||||||
|
(_inject_duplicate_top_level_key, "duplicate_json_key"),
|
||||||
|
(_inject_duplicate_nested_key, "duplicate_json_key"),
|
||||||
|
(lambda raw, _path: _inject_nonfinite(raw, "NaN"), "invalid_json_value"),
|
||||||
|
(lambda raw, _path: _inject_nonfinite(raw, "Infinity"), "invalid_json_value"),
|
||||||
|
(lambda raw, _path: _inject_nonfinite(raw, "-Infinity"), "invalid_json_value"),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
def test_verifier_rejects_ambiguous_or_nonfinite_metadata_before_verification(
|
||||||
|
tmp_path: Path,
|
||||||
|
metadata_path: str,
|
||||||
|
mutator: Callable[[str, str], str],
|
||||||
|
expected_reason: str,
|
||||||
|
) -> None:
|
||||||
|
output = tmp_path / "bundle"
|
||||||
|
rebuild.build_source_inventory_bundle(FIXTURE_ROOT, output)
|
||||||
|
target = output / metadata_path
|
||||||
|
target.write_text(mutator(target.read_text(encoding="utf-8"), metadata_path), encoding="utf-8")
|
||||||
|
|
||||||
|
completed = _run_verify(output)
|
||||||
|
|
||||||
|
assert completed.returncode == 2
|
||||||
|
assert json.loads(completed.stdout)["reason"] == expected_reason
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("value", (math.nan, math.inf, -math.inf))
|
||||||
|
def test_canonical_inventory_json_writers_reject_nonfinite_numbers(value: float) -> None:
|
||||||
|
for writer in (rebuild.canonical_json_bytes, rebuild.rendered_json_bytes):
|
||||||
|
with pytest.raises(rebuild.RebuildError) as error:
|
||||||
|
writer({"value": value})
|
||||||
|
assert error.value.code == "invalid_json_value"
|
||||||
|
|
||||||
|
|
||||||
|
def test_verify_mode_is_read_only(tmp_path: Path) -> None:
|
||||||
|
output = tmp_path / "bundle"
|
||||||
|
completed = _run_cli(FIXTURE_ROOT, output)
|
||||||
|
assert completed.returncode == 0, completed.stdout
|
||||||
|
before = {path: (path.stat().st_mtime_ns, path.read_bytes()) for path in output.rglob("*") if path.is_file()}
|
||||||
|
|
||||||
|
verified = _run_verify(output)
|
||||||
|
|
||||||
|
assert verified.returncode == 0, (verified.stdout, verified.stderr)
|
||||||
|
assert json.loads(verified.stdout)["operation"] == "verify"
|
||||||
|
after = {path: (path.stat().st_mtime_ns, path.read_bytes()) for path in output.rglob("*") if path.is_file()}
|
||||||
|
assert after == before
|
||||||
|
|
||||||
|
|
||||||
|
def test_post_read_source_mutation_does_not_change_copied_snapshot_or_overclaim_atomicity(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
source_root = tmp_path / "sources"
|
||||||
|
source = _make_source(source_root, content=b"# Original\n\nCopied bytes.\n")
|
||||||
|
original = source.read_bytes()
|
||||||
|
output = tmp_path / "bundle"
|
||||||
|
|
||||||
|
def mutate(relative_path: str, _source_root: Path, _snapshot: Path) -> None:
|
||||||
|
assert relative_path == "source.md"
|
||||||
|
source.write_bytes(b"# Mutated\n\nLater bytes.\n")
|
||||||
|
|
||||||
|
monkeypatch.setattr(rebuild, "_after_source_snapshot", mutate)
|
||||||
|
result = rebuild.build_source_inventory_bundle(source_root, output)
|
||||||
|
|
||||||
|
manifest = _load_json(output / rebuild.SOURCE_MANIFEST_PATH)
|
||||||
|
row = manifest["sources"][0]
|
||||||
|
assert (output / row["snapshot_path"]).read_bytes() == original
|
||||||
|
assert source.read_bytes() != original
|
||||||
|
assert row["artifact_sha256"] == _sha256(original)
|
||||||
|
assert manifest["source_snapshot_semantics"]["directory_snapshot_atomic"] is False
|
||||||
|
assert result["verification"]["status"] == "pass"
|
||||||
|
|
||||||
|
|
||||||
|
def test_destination_reservation_race_never_replaces_or_cleans_other_directory(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
source_root = tmp_path / "sources"
|
||||||
|
_make_source(source_root)
|
||||||
|
output = tmp_path / "bundle"
|
||||||
|
original_mkdir = rebuild.os.mkdir
|
||||||
|
raced = False
|
||||||
|
|
||||||
|
def racing_mkdir(path: os.PathLike[str] | str, mode: int = 0o777, *, dir_fd: int | None = None) -> None:
|
||||||
|
nonlocal raced
|
||||||
|
if Path(path) == output and not raced and dir_fd is None:
|
||||||
|
raced = True
|
||||||
|
original_mkdir(output, mode)
|
||||||
|
(output / "other-owner.txt").write_text("retain", encoding="utf-8")
|
||||||
|
original_mkdir(path, mode, dir_fd=dir_fd)
|
||||||
|
|
||||||
|
monkeypatch.setattr(rebuild.os, "mkdir", racing_mkdir)
|
||||||
|
with pytest.raises(rebuild.RebuildError) as error:
|
||||||
|
rebuild.build_source_inventory_bundle(source_root, output)
|
||||||
|
|
||||||
|
assert error.value.code == "output_exists"
|
||||||
|
assert (output / "other-owner.txt").read_text(encoding="utf-8") == "retain"
|
||||||
|
assert set(path.name for path in output.iterdir()) == {"other-owner.txt"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_existing_destination_is_never_replaced(tmp_path: Path) -> None:
|
||||||
|
source_root = tmp_path / "sources"
|
||||||
|
_make_source(source_root)
|
||||||
|
output = tmp_path / "bundle"
|
||||||
|
output.mkdir()
|
||||||
|
marker = output / "retain.txt"
|
||||||
|
marker.write_text("owned elsewhere", encoding="utf-8")
|
||||||
|
|
||||||
|
with pytest.raises(rebuild.RebuildError) as error:
|
||||||
|
rebuild.build_source_inventory_bundle(source_root, output)
|
||||||
|
|
||||||
|
assert error.value.code == "output_exists"
|
||||||
|
assert marker.read_text(encoding="utf-8") == "owned elsewhere"
|
||||||
|
|
||||||
|
|
||||||
|
def test_output_identity_replacement_fails_before_writing_replacement(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
source_root = tmp_path / "sources"
|
||||||
|
_make_source(source_root)
|
||||||
|
output = tmp_path / "bundle"
|
||||||
|
moved = tmp_path / "moved-reservation"
|
||||||
|
|
||||||
|
def replace(reserved: rebuild.ReservedOutput) -> None:
|
||||||
|
assert reserved.path == output
|
||||||
|
output.rename(moved)
|
||||||
|
output.mkdir()
|
||||||
|
(output / "other-owner.txt").write_text("retain", encoding="utf-8")
|
||||||
|
|
||||||
|
monkeypatch.setattr(rebuild, "_after_output_reservation", replace)
|
||||||
|
with pytest.raises(rebuild.RebuildError) as error:
|
||||||
|
rebuild.build_source_inventory_bundle(source_root, output)
|
||||||
|
|
||||||
|
assert error.value.code == "output_binding_changed"
|
||||||
|
assert set(path.name for path in output.iterdir()) == {"other-owner.txt"}
|
||||||
|
assert set(path.name for path in moved.iterdir()) == {rebuild.INCOMPLETE_MARKER}
|
||||||
|
|
||||||
|
|
||||||
|
def test_failure_after_reservation_removes_only_reserved_output_and_marker(tmp_path: Path) -> None:
|
||||||
|
source_root = tmp_path / "sources"
|
||||||
|
_make_source(source_root, content=b"text\x00binary\n")
|
||||||
|
output = tmp_path / "bundle"
|
||||||
|
|
||||||
|
with pytest.raises(rebuild.RebuildError) as error:
|
||||||
|
rebuild.build_source_inventory_bundle(source_root, output)
|
||||||
|
|
||||||
|
assert error.value.code == "binary_control_character"
|
||||||
|
assert not output.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_lf_and_crlf_locators_are_exact_and_cr_only_is_rejected(tmp_path: Path) -> None:
|
||||||
|
source_root = tmp_path / "sources"
|
||||||
|
source_root.mkdir()
|
||||||
|
lf = b"# One\nalpha\n# Two\nbeta\n"
|
||||||
|
crlf = b"# One\r\nalpha\r\n# Two\r\nbeta\r\n"
|
||||||
|
(source_root / "lf.md").write_bytes(lf)
|
||||||
|
(source_root / "crlf.md").write_bytes(crlf)
|
||||||
|
output = tmp_path / "bundle"
|
||||||
|
|
||||||
|
rebuild.build_source_inventory_bundle(source_root, output)
|
||||||
|
manifest = _load_json(output / rebuild.SOURCE_MANIFEST_PATH)
|
||||||
|
by_path = {row["relative_path"]: row for row in manifest["sources"]}
|
||||||
|
assert by_path["lf.md"]["newline_style"] == "lf"
|
||||||
|
assert by_path["crlf.md"]["newline_style"] == "crlf"
|
||||||
|
for name, raw in (("lf.md", lf), ("crlf.md", crlf)):
|
||||||
|
assert [(row["line_start"], row["line_end"]) for row in by_path[name]["sections"]] == [(1, 2), (3, 4)]
|
||||||
|
for locator in by_path[name]["sections"]:
|
||||||
|
quote = raw[locator["utf8_byte_start"] : locator["utf8_byte_end"]]
|
||||||
|
assert _sha256(quote) == locator["quote_sha256"]
|
||||||
|
|
||||||
|
cr_root = tmp_path / "cr-only"
|
||||||
|
_make_source(cr_root, content=b"# One\ralpha\r# Two\rbeta\r")
|
||||||
|
with pytest.raises(rebuild.RebuildError) as error:
|
||||||
|
rebuild.build_source_inventory_bundle(cr_root, tmp_path / "cr-bundle")
|
||||||
|
assert error.value.code == "unsupported_newline"
|
||||||
|
assert not (tmp_path / "cr-bundle").exists()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("content", [b"nul\x00byte\n", b"escape\x1bbyte\n", b"delete\x7fbyte\n"])
|
||||||
|
def test_binary_control_characters_are_rejected(tmp_path: Path, content: bytes) -> None:
|
||||||
|
source_root = tmp_path / "sources"
|
||||||
|
_make_source(source_root, content=content)
|
||||||
|
with pytest.raises(rebuild.RebuildError) as error:
|
||||||
|
rebuild.build_source_inventory_bundle(source_root, tmp_path / "bundle")
|
||||||
|
assert error.value.code == "binary_control_character"
|
||||||
|
assert not (tmp_path / "bundle").exists()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("content", "limits", "expected_code"),
|
||||||
|
[
|
||||||
|
(b"123456\n", rebuild.ResourceLimits(max_source_bytes=4), "source_byte_limit_exceeded"),
|
||||||
|
(b"123456\n", rebuild.ResourceLimits(max_total_bytes=4, max_source_bytes=4), "source_byte_limit_exceeded"),
|
||||||
|
(
|
||||||
|
b"# One\na\n# Two\nb\n",
|
||||||
|
rebuild.ResourceLimits(max_sections_per_source=1, max_work_items=1),
|
||||||
|
"section_limit_exceeded",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_resource_limits_fail_closed(
|
||||||
|
tmp_path: Path,
|
||||||
|
content: bytes,
|
||||||
|
limits: rebuild.ResourceLimits,
|
||||||
|
expected_code: str,
|
||||||
|
) -> None:
|
||||||
|
source_root = tmp_path / "sources"
|
||||||
|
_make_source(source_root, content=content)
|
||||||
|
with pytest.raises(rebuild.RebuildError) as error:
|
||||||
|
rebuild.build_source_inventory_bundle(source_root, tmp_path / "bundle", limits=limits)
|
||||||
|
assert error.value.code == expected_code
|
||||||
|
assert not (tmp_path / "bundle").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_total_source_count_and_work_item_limits_fail_closed(tmp_path: Path) -> None:
|
||||||
|
source_root = tmp_path / "sources"
|
||||||
|
_make_source(source_root, "a.txt", b"a\n")
|
||||||
|
_make_source(source_root, "b.txt", b"b\n")
|
||||||
|
|
||||||
|
with pytest.raises(rebuild.RebuildError) as count_error:
|
||||||
|
rebuild.build_source_inventory_bundle(
|
||||||
|
source_root,
|
||||||
|
tmp_path / "count-bundle",
|
||||||
|
limits=rebuild.ResourceLimits(max_source_count=1),
|
||||||
|
)
|
||||||
|
assert count_error.value.code == "source_count_limit_exceeded"
|
||||||
|
|
||||||
|
with pytest.raises(rebuild.RebuildError) as work_error:
|
||||||
|
rebuild.build_source_inventory_bundle(
|
||||||
|
source_root,
|
||||||
|
tmp_path / "work-bundle",
|
||||||
|
limits=rebuild.ResourceLimits(max_work_items=1, max_sections_per_source=1),
|
||||||
|
)
|
||||||
|
assert work_error.value.code == "work_item_limit_exceeded"
|
||||||
|
|
||||||
|
with pytest.raises(rebuild.RebuildError) as total_error:
|
||||||
|
rebuild.build_source_inventory_bundle(
|
||||||
|
source_root,
|
||||||
|
tmp_path / "total-bundle",
|
||||||
|
limits=rebuild.ResourceLimits(max_source_bytes=2, max_total_bytes=3),
|
||||||
|
)
|
||||||
|
assert total_error.value.code == "total_byte_limit_exceeded"
|
||||||
|
|
||||||
|
|
||||||
|
def test_limit_overrides_remain_hard_bounded(tmp_path: Path) -> None:
|
||||||
|
source_root = tmp_path / "sources"
|
||||||
|
_make_source(source_root)
|
||||||
|
completed = _run_cli(
|
||||||
|
source_root,
|
||||||
|
tmp_path / "bundle",
|
||||||
|
"--max-source-bytes",
|
||||||
|
str(rebuild.HARD_MAX_SOURCE_BYTES + 1),
|
||||||
|
)
|
||||||
|
assert completed.returncode == 2
|
||||||
|
assert json.loads(completed.stdout)["reason"] == "invalid_resource_limit"
|
||||||
|
assert not (tmp_path / "bundle").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_identical_source_bytes_share_one_content_addressed_snapshot(tmp_path: Path) -> None:
|
||||||
|
source_root = tmp_path / "sources"
|
||||||
|
content = b"same exact bytes\n"
|
||||||
|
_make_source(source_root, "a.txt", content)
|
||||||
|
_make_source(source_root, "nested/b.txt", content)
|
||||||
|
output = tmp_path / "bundle"
|
||||||
|
|
||||||
|
rebuild.build_source_inventory_bundle(source_root, output)
|
||||||
|
manifest = _load_json(output / rebuild.SOURCE_MANIFEST_PATH)
|
||||||
|
assert manifest["source_count"] == 2
|
||||||
|
assert manifest["unique_snapshot_count"] == 1
|
||||||
|
assert manifest["duplicate_content_source_count"] == 1
|
||||||
|
assert len({row["snapshot_path"] for row in manifest["sources"]}) == 1
|
||||||
|
assert len(list((output / "snapshots" / "sha256").rglob("*.utf8"))) == 1
|
||||||
|
assert _load_json(output / rebuild.EXTRACTION_WORK_INDEX_PATH)["work_item_count"] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_source_rows_use_global_utf8_relative_path_order(tmp_path: Path) -> None:
|
||||||
|
source_root = tmp_path / "sources"
|
||||||
|
_make_source(source_root, "a-z.txt", b"sibling\n")
|
||||||
|
_make_source(source_root, "a/x.txt", b"nested\n")
|
||||||
|
_make_source(source_root, "aa.txt", b"later\n")
|
||||||
|
output = tmp_path / "bundle"
|
||||||
|
|
||||||
|
rebuild.build_source_inventory_bundle(source_root, output)
|
||||||
|
|
||||||
|
rows = _load_json(output / rebuild.SOURCE_MANIFEST_PATH)["sources"]
|
||||||
|
assert [row["relative_path"] for row in rows] == ["a-z.txt", "a/x.txt", "aa.txt"]
|
||||||
|
assert rebuild.verify_bundle(output)["status"] == "pass"
|
||||||
|
|
||||||
|
|
||||||
|
def test_mismatched_existing_content_addressed_snapshot_is_rejected(tmp_path: Path) -> None:
|
||||||
|
output = tmp_path / "bundle"
|
||||||
|
prefix = output / "snapshots" / "sha256"
|
||||||
|
prefix.mkdir(parents=True)
|
||||||
|
partial = output / "snapshots" / ".partial-00000001"
|
||||||
|
content = b"expected\n"
|
||||||
|
partial.write_bytes(content)
|
||||||
|
digest = _sha256(content)
|
||||||
|
destination = output / Path(rebuild._snapshot_path_for_hash(digest))
|
||||||
|
destination.parent.mkdir()
|
||||||
|
destination.write_bytes(b"mismatch\n")
|
||||||
|
|
||||||
|
with pytest.raises(rebuild.RebuildError) as error:
|
||||||
|
rebuild._install_snapshot(output, partial, digest, len(content))
|
||||||
|
|
||||||
|
assert error.value.code == "snapshot_collision"
|
||||||
|
assert destination.read_bytes() == b"mismatch\n"
|
||||||
|
assert partial.read_bytes() == content
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("target", ["snapshot", "manifest", "index", "receipt"])
|
||||||
|
def test_verifier_rejects_tampered_bundle(tmp_path: Path, target: str) -> None:
|
||||||
|
output = tmp_path / "bundle"
|
||||||
|
rebuild.build_source_inventory_bundle(FIXTURE_ROOT, output)
|
||||||
|
manifest = _load_json(output / rebuild.SOURCE_MANIFEST_PATH)
|
||||||
|
if target == "snapshot":
|
||||||
|
path = output / manifest["sources"][0]["snapshot_path"]
|
||||||
|
path.write_bytes(path.read_bytes() + b"tamper")
|
||||||
|
elif target == "manifest":
|
||||||
|
path = output / rebuild.SOURCE_MANIFEST_PATH
|
||||||
|
document = _load_json(path)
|
||||||
|
document["source_count"] += 1
|
||||||
|
path.write_text(json.dumps(document), encoding="utf-8")
|
||||||
|
elif target == "index":
|
||||||
|
path = output / rebuild.EXTRACTION_WORK_INDEX_PATH
|
||||||
|
document = _load_json(path)
|
||||||
|
document["items"][0]["work_type"] = "semantic_claim"
|
||||||
|
path.write_text(json.dumps(document), encoding="utf-8")
|
||||||
|
else:
|
||||||
|
path = output / rebuild.RECEIPT_PATH
|
||||||
|
document = _load_json(path)
|
||||||
|
document["database_write_performed"] = True
|
||||||
|
path.write_text(json.dumps(document), encoding="utf-8")
|
||||||
|
|
||||||
|
completed = _run_verify(output)
|
||||||
|
|
||||||
|
assert completed.returncode == 2
|
||||||
|
assert json.loads(completed.stdout)["status"] == "rejected"
|
||||||
|
|
||||||
|
|
||||||
|
def test_inventory_rejects_symlinks_unsupported_files_invalid_utf8_and_empty_text(tmp_path: Path) -> None:
|
||||||
|
source_root = tmp_path / "symlinks"
|
||||||
|
target = _make_source(source_root)
|
||||||
|
link = source_root / "alias.md"
|
||||||
|
try:
|
||||||
|
link.symlink_to(target.name)
|
||||||
|
except OSError as exc: # pragma: no cover - platform capability
|
||||||
|
pytest.skip(f"symlinks unavailable: {exc}")
|
||||||
|
with pytest.raises(rebuild.RebuildError) as symlink_error:
|
||||||
|
rebuild.build_source_inventory_bundle(source_root, tmp_path / "symlink-bundle")
|
||||||
|
assert symlink_error.value.code == "symlink_rejected"
|
||||||
|
|
||||||
|
cases = [
|
||||||
|
("unsupported", "unsafe.pdf", b"%PDF-1.7\n", "unsupported_source_file"),
|
||||||
|
("invalid", "invalid.md", b"invalid: \xff\n", "invalid_utf8_source"),
|
||||||
|
("empty", "empty.txt", b" \n\t\n", "empty_source_file"),
|
||||||
|
]
|
||||||
|
for directory, filename, content, expected in cases:
|
||||||
|
root = tmp_path / directory
|
||||||
|
_make_source(root, filename, content)
|
||||||
|
with pytest.raises(rebuild.RebuildError) as error:
|
||||||
|
rebuild.build_source_inventory_bundle(root, tmp_path / f"{directory}-bundle")
|
||||||
|
assert error.value.code == expected
|
||||||
|
|
||||||
|
|
||||||
|
def test_rejects_output_overlap_and_unsupported_mode(tmp_path: Path) -> None:
|
||||||
|
source_root = tmp_path / "sources"
|
||||||
|
_make_source(source_root)
|
||||||
|
with pytest.raises(rebuild.RebuildError) as overlap:
|
||||||
|
rebuild.build_source_inventory_bundle(source_root, source_root / "bundle")
|
||||||
|
assert overlap.value.code == "output_overlap"
|
||||||
|
|
||||||
|
completed = _run_cli(source_root, tmp_path / "bundle", "--mode", "candidate-only")
|
||||||
|
assert completed.returncode == 2
|
||||||
|
assert json.loads(completed.stdout)["reason"] == "unsupported_mode"
|
||||||
|
assert not (tmp_path / "bundle").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_structured_extensions_are_inventoried_as_utf8_not_claimed_as_format_validated(tmp_path: Path) -> None:
|
||||||
|
source_root = tmp_path / "sources"
|
||||||
|
_make_source(source_root, "opaque.json", b"not actually json\n")
|
||||||
|
output = tmp_path / "bundle"
|
||||||
|
|
||||||
|
rebuild.build_source_inventory_bundle(source_root, output)
|
||||||
|
|
||||||
|
row = _load_json(output / rebuild.SOURCE_MANIFEST_PATH)["sources"][0]
|
||||||
|
assert row["sectioning_strategy"] == "whole_utf8_source"
|
||||||
|
assert row["format_syntax_validated"] is False
|
||||||
|
assert row["section_count"] == 1
|
||||||
|
|
@ -37,6 +37,22 @@ RECEIPT_STRATEGY_ID = "14141414-1414-4414-8414-141414141414"
|
||||||
RECEIPT_NODE_A_ID = "15151515-1515-4515-8515-151515151515"
|
RECEIPT_NODE_A_ID = "15151515-1515-4515-8515-151515151515"
|
||||||
RECEIPT_NODE_B_ID = "16161616-1616-4616-8616-161616161616"
|
RECEIPT_NODE_B_ID = "16161616-1616-4616-8616-161616161616"
|
||||||
PRIVATE_MARKER = "PRIVATE-REPLAY-MATERIAL-MUST-NOT-LEAK"
|
PRIVATE_MARKER = "PRIVATE-REPLAY-MATERIAL-MUST-NOT-LEAK"
|
||||||
|
EXPORTER_MATERIAL_FIELDS = frozenset(
|
||||||
|
{
|
||||||
|
"artifact",
|
||||||
|
"contract_version",
|
||||||
|
"sequence",
|
||||||
|
"approved_proposal",
|
||||||
|
"approval_snapshot",
|
||||||
|
"applied_proposal",
|
||||||
|
"replay_receipt",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
V3_PROPOSAL_ID = "17171717-1717-4717-8717-171717171717"
|
||||||
|
V3_CLAIM_ID = "18181818-1818-4818-8818-181818181818"
|
||||||
|
V3_SOURCE_ID = "20202020-2020-4020-8020-202020202020"
|
||||||
|
V3_EVIDENCE_ID = "21212121-2121-4121-8121-212121212121"
|
||||||
|
V3_ASSESSMENT_ID = "23232323-2323-4323-8323-232323232323"
|
||||||
|
|
||||||
|
|
||||||
def sha256_file(path: Path) -> str:
|
def sha256_file(path: Path) -> str:
|
||||||
|
|
@ -45,6 +61,11 @@ def sha256_file(path: Path) -> str:
|
||||||
return digest.hexdigest()
|
return digest.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def sha256_json(value: object) -> str:
|
||||||
|
encoded = json.dumps(value, ensure_ascii=False, separators=(",", ":"), sort_keys=True).encode("utf-8")
|
||||||
|
return hashlib.sha256(encoded).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
def write_json(path: Path, value: object) -> Path:
|
def write_json(path: Path, value: object) -> Path:
|
||||||
path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||||
return path
|
return path
|
||||||
|
|
@ -270,6 +291,272 @@ def replay_material() -> dict:
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def v3_replay_material() -> dict:
|
||||||
|
evidence = {
|
||||||
|
"id": V3_EVIDENCE_ID,
|
||||||
|
"claim_id": V3_CLAIM_ID,
|
||||||
|
"source_id": V3_SOURCE_ID,
|
||||||
|
"polarity": "supports",
|
||||||
|
"excerpt": "The exact source bytes support this bounded reconstruction claim.",
|
||||||
|
"locator_json": {"section": "v3-rebuild", "paragraph": 1},
|
||||||
|
"source_content_hash": "a" * 64,
|
||||||
|
"rationale": "The content-addressed source and exact locator support the proposition.",
|
||||||
|
"created_by": REVIEWER_ID,
|
||||||
|
}
|
||||||
|
apply_payload = {
|
||||||
|
"contract_version": 3,
|
||||||
|
"agent_id": REVIEWER_ID,
|
||||||
|
"claims": [
|
||||||
|
{
|
||||||
|
"id": V3_CLAIM_ID,
|
||||||
|
"owner_agent_id": None,
|
||||||
|
"type": "empirical",
|
||||||
|
"proposition": "V3 knowledge can be reconstructed from genesis plus an exact proposal ledger.",
|
||||||
|
"body_markdown": "The reconstruction replays review, admission, provenance, and assessment rows.",
|
||||||
|
"scope": "Networkless disposable PostgreSQL reconstruction.",
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"status": "active",
|
||||||
|
"revision_criteria": "Revise if exact final parity or trigger restoration cannot be reproduced.",
|
||||||
|
"revision_kind": "original",
|
||||||
|
"supersedes_id": None,
|
||||||
|
"tags": ["v3", "reconstruction"],
|
||||||
|
"created_by": REVIEWER_ID,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"sources": [
|
||||||
|
{
|
||||||
|
"id": V3_SOURCE_ID,
|
||||||
|
"source_type": "paper",
|
||||||
|
"canonical_url": "https://example.test/v3-reconstruction",
|
||||||
|
"storage_uri": None,
|
||||||
|
"excerpt": "Exact source bytes for the disposable V3 reconstruction.",
|
||||||
|
"content_hash": "a" * 64,
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"ingestion_origin": "operator_upload",
|
||||||
|
"provenance_status": "verified",
|
||||||
|
"source_class": "primary_record",
|
||||||
|
"auto_promotion_policy": "human_required",
|
||||||
|
"captured_at": "2026-07-18T10:00:00+00:00",
|
||||||
|
"created_by": REVIEWER_ID,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"evidence": [evidence],
|
||||||
|
"edges": [],
|
||||||
|
"assessments": [
|
||||||
|
{
|
||||||
|
"id": V3_ASSESSMENT_ID,
|
||||||
|
"claim_id": V3_CLAIM_ID,
|
||||||
|
"support_tier": "strong",
|
||||||
|
"challenge_tier": "none",
|
||||||
|
"overall_state": "support_dominant",
|
||||||
|
"rationale": "The exact source and locator support the bounded reconstruction proposition.",
|
||||||
|
"evidence_set_hash": rebuild.apply_engine._v3_evidence_set_hash(V3_CLAIM_ID, [evidence]),
|
||||||
|
"supersedes_assessment_id": None,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"reasoning_tools": [],
|
||||||
|
}
|
||||||
|
payload = {"title": "V3 reconstruction fixture", "apply_payload": apply_payload}
|
||||||
|
approved = {
|
||||||
|
"id": V3_PROPOSAL_ID,
|
||||||
|
"proposal_type": "approve_claim",
|
||||||
|
"status": "approved",
|
||||||
|
"proposed_by_handle": "fixture-agent",
|
||||||
|
"proposed_by_agent_id": REVIEWER_ID,
|
||||||
|
"channel": "v3_reconstruction_fixture",
|
||||||
|
"source_ref": "private://fixture/v3-source",
|
||||||
|
"rationale": PRIVATE_MARKER,
|
||||||
|
"payload": payload,
|
||||||
|
"reviewed_by_handle": "m3ta",
|
||||||
|
"reviewed_by_agent_id": REVIEWER_ID,
|
||||||
|
"reviewed_at": "2026-07-18T10:01:00+00:00",
|
||||||
|
"review_note": "Reviewed the exact V3 reconstruction payload.",
|
||||||
|
"applied_by_handle": None,
|
||||||
|
"applied_by_agent_id": None,
|
||||||
|
"applied_at": None,
|
||||||
|
"created_at": "2026-07-18T10:00:30+00:00",
|
||||||
|
"updated_at": "2026-07-18T10:01:00+00:00",
|
||||||
|
}
|
||||||
|
applied = {
|
||||||
|
**approved,
|
||||||
|
"status": "applied",
|
||||||
|
"applied_by_handle": "kb-apply",
|
||||||
|
"applied_by_agent_id": SERVICE_ID,
|
||||||
|
"applied_at": "2026-07-18T10:02:00+00:00",
|
||||||
|
"updated_at": "2026-07-18T10:02:00.000001+00:00",
|
||||||
|
}
|
||||||
|
approval = {
|
||||||
|
"proposal_id": V3_PROPOSAL_ID,
|
||||||
|
"proposal_type": "approve_claim",
|
||||||
|
"payload": payload,
|
||||||
|
"reviewed_by_handle": approved["reviewed_by_handle"],
|
||||||
|
"reviewed_by_agent_id": REVIEWER_ID,
|
||||||
|
"reviewed_by_db_role": "kb_review",
|
||||||
|
"reviewed_at": approved["reviewed_at"],
|
||||||
|
"review_note": approved["review_note"],
|
||||||
|
}
|
||||||
|
created_at = "2026-07-18T10:01:30+00:00"
|
||||||
|
rows = {
|
||||||
|
"claims": [
|
||||||
|
{
|
||||||
|
"id": V3_CLAIM_ID,
|
||||||
|
"owner_agent_id": None,
|
||||||
|
"type": "empirical",
|
||||||
|
"text": apply_payload["claims"][0]["proposition"],
|
||||||
|
"proposition": apply_payload["claims"][0]["proposition"],
|
||||||
|
"body_markdown": apply_payload["claims"][0]["body_markdown"],
|
||||||
|
"scope": apply_payload["claims"][0]["scope"],
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"status": "active",
|
||||||
|
"admission_state": "admitted",
|
||||||
|
"revision_criteria": apply_payload["claims"][0]["revision_criteria"],
|
||||||
|
"revision_kind": "original",
|
||||||
|
"supersedes_id": None,
|
||||||
|
"confidence": None,
|
||||||
|
"tags": ["v3", "reconstruction"],
|
||||||
|
"created_by": REVIEWER_ID,
|
||||||
|
"superseded_by": None,
|
||||||
|
"accepted_by_proposal_id": V3_PROPOSAL_ID,
|
||||||
|
"created_at": created_at,
|
||||||
|
"updated_at": created_at,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"sources": [
|
||||||
|
{
|
||||||
|
"id": V3_SOURCE_ID,
|
||||||
|
"source_type": "paper",
|
||||||
|
"url": apply_payload["sources"][0]["canonical_url"],
|
||||||
|
"storage_path": None,
|
||||||
|
"excerpt": apply_payload["sources"][0]["excerpt"],
|
||||||
|
"hash": "a" * 64,
|
||||||
|
"created_by": REVIEWER_ID,
|
||||||
|
"captured_at": apply_payload["sources"][0]["captured_at"],
|
||||||
|
"canonical_url": apply_payload["sources"][0]["canonical_url"],
|
||||||
|
"storage_uri": None,
|
||||||
|
"content_hash": "a" * 64,
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"ingestion_origin": "operator_upload",
|
||||||
|
"provenance_status": "verified",
|
||||||
|
"source_class": "primary_record",
|
||||||
|
"auto_promotion_policy": "human_required",
|
||||||
|
"accepted_by_proposal_id": V3_PROPOSAL_ID,
|
||||||
|
"created_at": created_at,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"claim_evidence": [
|
||||||
|
{
|
||||||
|
"id": V3_EVIDENCE_ID,
|
||||||
|
"claim_id": V3_CLAIM_ID,
|
||||||
|
"source_id": V3_SOURCE_ID,
|
||||||
|
"role": "grounds",
|
||||||
|
"weight": None,
|
||||||
|
"created_by": REVIEWER_ID,
|
||||||
|
"excerpt": evidence["excerpt"],
|
||||||
|
"locator_json": evidence["locator_json"],
|
||||||
|
"source_content_hash": "a" * 64,
|
||||||
|
"rationale": evidence["rationale"],
|
||||||
|
"polarity": "supports",
|
||||||
|
"accepted_from_proposal_id": V3_PROPOSAL_ID,
|
||||||
|
"created_at": created_at,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"claim_edges": [],
|
||||||
|
"claim_evidence_assessments": [
|
||||||
|
{
|
||||||
|
**apply_payload["assessments"][0],
|
||||||
|
"accepted_by_proposal_id": V3_PROPOSAL_ID,
|
||||||
|
"created_at": created_at,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"reasoning_tools": [],
|
||||||
|
}
|
||||||
|
proposal = applied_envelope(applied)
|
||||||
|
apply_sql = rebuild.apply_engine.build_apply_sql(proposal, applied["applied_by_handle"])
|
||||||
|
receipt = rebuild.replay_receipt.build_replay_receipt(
|
||||||
|
proposal,
|
||||||
|
rows,
|
||||||
|
apply_sql_sha256=rebuild.replay_receipt.sha256_text(apply_sql),
|
||||||
|
apply_sql_source="exact_executed_sql",
|
||||||
|
exported_at_utc="2026-07-18T10:03:00+00:00",
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"artifact": rebuild.MATERIAL_ARTIFACT,
|
||||||
|
"contract_version": rebuild.MATERIAL_CONTRACT_VERSION,
|
||||||
|
"sequence": 1,
|
||||||
|
"approved_proposal": approved,
|
||||||
|
"approval_snapshot": approval,
|
||||||
|
"applied_proposal": applied,
|
||||||
|
"replay_receipt": receipt,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def exporter_shaped_v3_replay_material() -> dict:
|
||||||
|
material = copy.deepcopy(v3_replay_material())
|
||||||
|
payload_rows = (
|
||||||
|
material["approved_proposal"],
|
||||||
|
material["approval_snapshot"],
|
||||||
|
material["applied_proposal"],
|
||||||
|
material["replay_receipt"]["proposal"],
|
||||||
|
)
|
||||||
|
for row in payload_rows:
|
||||||
|
row["payload"]["apply_payload"]["sources"][0]["captured_at"] = "2026-07-18T12:00:00+02:00"
|
||||||
|
|
||||||
|
approved = material["approved_proposal"]
|
||||||
|
approved.update(
|
||||||
|
created_at="2026-07-18T10:00:30.000000Z",
|
||||||
|
reviewed_at="2026-07-18T10:01:00.000000Z",
|
||||||
|
updated_at="2026-07-18T10:01:00.000000Z",
|
||||||
|
)
|
||||||
|
applied = material["applied_proposal"]
|
||||||
|
applied.update(
|
||||||
|
created_at=approved["created_at"],
|
||||||
|
reviewed_at=approved["reviewed_at"],
|
||||||
|
applied_at="2026-07-18T10:02:00.000000Z",
|
||||||
|
updated_at="2026-07-18T10:02:00.000001Z",
|
||||||
|
)
|
||||||
|
material["approval_snapshot"]["reviewed_at"] = approved["reviewed_at"]
|
||||||
|
|
||||||
|
canonical_rows = material["replay_receipt"]["canonical_rows"]
|
||||||
|
for table_rows in canonical_rows.values():
|
||||||
|
for row in table_rows:
|
||||||
|
if "captured_at" in row:
|
||||||
|
row["captured_at"] = "2026-07-18T10:00:00.000000Z"
|
||||||
|
if "created_at" in row:
|
||||||
|
row["created_at"] = "2026-07-18T10:01:30.000000Z"
|
||||||
|
if "updated_at" in row:
|
||||||
|
row["updated_at"] = "2026-07-18T10:01:30.000000Z"
|
||||||
|
|
||||||
|
proposal = applied_envelope(applied)
|
||||||
|
semantic_proposal = copy.deepcopy(proposal)
|
||||||
|
semantic_proposal["payload"]["apply_payload"]["sources"][0]["captured_at"] = canonical_rows["sources"][0][
|
||||||
|
"captured_at"
|
||||||
|
]
|
||||||
|
receipt = rebuild.replay_receipt.build_replay_receipt(
|
||||||
|
semantic_proposal,
|
||||||
|
canonical_rows,
|
||||||
|
apply_sql_sha256=rebuild.replay_receipt.sha256_text("snapshot exporter fixture apply SQL"),
|
||||||
|
apply_sql_source="exact_executed_sql",
|
||||||
|
exported_at_utc="2026-07-18T10:03:00.000000Z",
|
||||||
|
)
|
||||||
|
receipt["proposal"]["payload"] = proposal["payload"]
|
||||||
|
replay_material = {
|
||||||
|
"receipt_contract_version": receipt["receipt_contract_version"],
|
||||||
|
"apply_engine": receipt["apply_engine"],
|
||||||
|
"proposal": receipt["proposal"],
|
||||||
|
"canonical_rows": receipt["canonical_rows"],
|
||||||
|
}
|
||||||
|
receipt["hashes"] = {
|
||||||
|
"apply_payload_sha256": sha256_json(proposal["payload"]["apply_payload"]),
|
||||||
|
"proposal_payload_sha256": sha256_json(proposal["payload"]),
|
||||||
|
"canonical_rows_sha256": sha256_json(receipt["canonical_rows"]),
|
||||||
|
"row_sha256": {table: [sha256_json(row) for row in rows] for table, rows in receipt["canonical_rows"].items()},
|
||||||
|
"table_sha256": {table: sha256_json(rows) for table, rows in receipt["canonical_rows"].items()},
|
||||||
|
"replay_material_sha256": sha256_json(replay_material),
|
||||||
|
}
|
||||||
|
material["replay_receipt"] = receipt
|
||||||
|
return material
|
||||||
|
|
||||||
|
|
||||||
def revise_strategy_replay_material(*, apply_sql_source: str = "exact_executed_sql", version: int = 2) -> dict:
|
def revise_strategy_replay_material(*, apply_sql_source: str = "exact_executed_sql", version: int = 2) -> dict:
|
||||||
approved, applied, approval = revise_strategy_proposal_rows()
|
approved, applied, approval = revise_strategy_proposal_rows()
|
||||||
proposal = applied_envelope(applied)
|
proposal = applied_envelope(applied)
|
||||||
|
|
@ -386,9 +673,45 @@ def write_bundle(tmp_path: Path, *, material: dict | None = None) -> argparse.Na
|
||||||
max_target_source_ratio=100.0,
|
max_target_source_ratio=100.0,
|
||||||
docker_bin="docker",
|
docker_bin="docker",
|
||||||
output=tmp_path / "receipt.json",
|
output=tmp_path / "receipt.json",
|
||||||
|
admission_manifest=None,
|
||||||
|
admission_manifest_sha256=None,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def attach_admission_manifest(
|
||||||
|
args: argparse.Namespace,
|
||||||
|
*,
|
||||||
|
admission_scope: str = "collective_core",
|
||||||
|
basis: str = "forgejo_primary_source",
|
||||||
|
) -> Path:
|
||||||
|
ledger = json.loads(args.ledger.read_text(encoding="utf-8"))
|
||||||
|
ledger_entry = ledger["entries"][0]
|
||||||
|
material = json.loads((args.ledger.parent / ledger_entry["material"]).read_text(encoding="utf-8"))
|
||||||
|
manifest = {
|
||||||
|
"artifact": rebuild.ADMISSION_ARTIFACT,
|
||||||
|
"contract_version": rebuild.ADMISSION_CONTRACT_VERSION,
|
||||||
|
"ledger_sha256": args.ledger_sha256,
|
||||||
|
"reviewed_by_handle": "m3taversal",
|
||||||
|
"reviewed_by_agent_id": REVIEWER_ID,
|
||||||
|
"reviewed_at": "2026-07-18T14:00:00+00:00",
|
||||||
|
"review_note": "Admit this exact source-backed V3 proposal into the clean genesis fixture.",
|
||||||
|
"entries": [
|
||||||
|
{
|
||||||
|
"sequence": 1,
|
||||||
|
"proposal_id": material["applied_proposal"]["id"],
|
||||||
|
"material_sha256": ledger_entry["sha256"],
|
||||||
|
"replay_material_sha256": ledger_entry["replay_material_sha256"],
|
||||||
|
"admission_scope": admission_scope,
|
||||||
|
"basis": basis,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
path = write_json(args.ledger.parent / "admission-manifest.json", manifest)
|
||||||
|
args.admission_manifest = path
|
||||||
|
args.admission_manifest_sha256 = sha256_file(path)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
def rewrite_material_and_ledger(args: argparse.Namespace, material: dict) -> None:
|
def rewrite_material_and_ledger(args: argparse.Namespace, material: dict) -> None:
|
||||||
material_path = args.ledger.parent / "entry-0001.json"
|
material_path = args.ledger.parent / "entry-0001.json"
|
||||||
write_json(material_path, material)
|
write_json(material_path, material)
|
||||||
|
|
@ -408,12 +731,77 @@ def test_load_bundle_validates_every_pin_and_current_apply_engine(tmp_path: Path
|
||||||
assert bundle.final_manifest_sha256 == sha256_file(tmp_path / "final-manifest.jsonl")
|
assert bundle.final_manifest_sha256 == sha256_file(tmp_path / "final-manifest.jsonl")
|
||||||
assert bundle.engine_hashes == rebuild._engine_hashes()
|
assert bundle.engine_hashes == rebuild._engine_hashes()
|
||||||
assert len(bundle.entries) == 1
|
assert len(bundle.entries) == 1
|
||||||
|
assert bundle.admission == {
|
||||||
|
"status": "forensic_only",
|
||||||
|
"canonical_retrieval_eligible": False,
|
||||||
|
"reason": "no explicit admission manifest was supplied",
|
||||||
|
"entry_count": 1,
|
||||||
|
}
|
||||||
entry = bundle.entries[0]
|
entry = bundle.entries[0]
|
||||||
assert entry.applied_proposal["proposal_type"] == "add_edge"
|
assert entry.applied_proposal["proposal_type"] == "add_edge"
|
||||||
assert entry.current_apply_sql_sha256 == hashlib.sha256(entry.current_apply_sql.encode("utf-8")).hexdigest()
|
assert entry.current_apply_sql_sha256 == hashlib.sha256(entry.current_apply_sql.encode("utf-8")).hexdigest()
|
||||||
assert "kb_stage.assert_approved_proposal" in entry.current_apply_sql
|
assert "kb_stage.assert_approved_proposal" in entry.current_apply_sql
|
||||||
|
|
||||||
|
|
||||||
|
def test_hash_pinned_v3_review_metadata_selects_material_without_granting_authority(tmp_path: Path) -> None:
|
||||||
|
args = write_bundle(tmp_path, material=exporter_shaped_v3_replay_material())
|
||||||
|
manifest_path = attach_admission_manifest(args)
|
||||||
|
|
||||||
|
bundle = rebuild.load_bundle(args)
|
||||||
|
|
||||||
|
assert bundle.admission_manifest_path == manifest_path
|
||||||
|
assert bundle.admission_manifest_sha256 == sha256_file(manifest_path)
|
||||||
|
assert bundle.admission["status"] == "review_metadata_supplied"
|
||||||
|
assert bundle.admission["canonical_retrieval_eligible"] is False
|
||||||
|
assert bundle.admission["cryptographically_authenticated"] is False
|
||||||
|
assert bundle.admission["review_metadata_provenance"] == rebuild.REVIEW_METADATA_PROVENANCE
|
||||||
|
assert "not authenticated" in bundle.admission["reason"]
|
||||||
|
assert bundle.admission["reviewed_by_handle"] == "m3taversal"
|
||||||
|
assert (
|
||||||
|
bundle.admission["review_note_sha256"]
|
||||||
|
== hashlib.sha256(b"Admit this exact source-backed V3 proposal into the clean genesis fixture.").hexdigest()
|
||||||
|
)
|
||||||
|
assert bundle.admission["entries"][0]["basis"] == "forgejo_primary_source"
|
||||||
|
|
||||||
|
|
||||||
|
def test_admission_rejects_forged_reviewer_identity(tmp_path: Path) -> None:
|
||||||
|
args = write_bundle(tmp_path, material=exporter_shaped_v3_replay_material())
|
||||||
|
manifest_path = attach_admission_manifest(args)
|
||||||
|
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||||
|
manifest["reviewed_by_handle"] = "forged-reviewer"
|
||||||
|
write_json(manifest_path, manifest)
|
||||||
|
args.admission_manifest_sha256 = sha256_file(manifest_path)
|
||||||
|
|
||||||
|
with pytest.raises(rebuild.ReconstructionError) as exc_info:
|
||||||
|
rebuild.load_bundle(args)
|
||||||
|
|
||||||
|
assert exc_info.value.code == "invalid_admission_review"
|
||||||
|
|
||||||
|
|
||||||
|
def test_admission_rejects_rehashed_duplicate_json_fields(tmp_path: Path) -> None:
|
||||||
|
args = write_bundle(tmp_path, material=exporter_shaped_v3_replay_material())
|
||||||
|
manifest_path = attach_admission_manifest(args)
|
||||||
|
raw = manifest_path.read_text(encoding="utf-8")
|
||||||
|
duplicate = '{"artifact":"teleo_ledger_admission_manifest",' + raw.lstrip()[1:]
|
||||||
|
manifest_path.write_text(duplicate, encoding="utf-8")
|
||||||
|
args.admission_manifest_sha256 = sha256_file(manifest_path)
|
||||||
|
|
||||||
|
with pytest.raises(rebuild.ReconstructionError) as exc_info:
|
||||||
|
rebuild.load_bundle(args)
|
||||||
|
|
||||||
|
assert exc_info.value.code == "invalid_json_artifact"
|
||||||
|
|
||||||
|
|
||||||
|
def test_admission_rejects_legacy_rows_instead_of_blessing_forensic_history(tmp_path: Path) -> None:
|
||||||
|
args = write_bundle(tmp_path)
|
||||||
|
attach_admission_manifest(args)
|
||||||
|
|
||||||
|
with pytest.raises(rebuild.ReconstructionError) as exc_info:
|
||||||
|
rebuild.load_bundle(args)
|
||||||
|
|
||||||
|
assert exc_info.value.code == "legacy_admission_forbidden"
|
||||||
|
|
||||||
|
|
||||||
def test_dump_hash_mismatch_fails_before_container_start(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
def test_dump_hash_mismatch_fails_before_container_start(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
args = write_bundle(tmp_path)
|
args = write_bundle(tmp_path)
|
||||||
args.genesis_dump.write_bytes(b"PGDMPtampered")
|
args.genesis_dump.write_bytes(b"PGDMPtampered")
|
||||||
|
|
@ -531,7 +919,9 @@ def test_revise_strategy_accepts_timezone_aware_closed_timestamp_boundaries(
|
||||||
|
|
||||||
entry = rebuild.load_bundle(write_bundle(tmp_path, material=material)).entries[0]
|
entry = rebuild.load_bundle(write_bundle(tmp_path, material=material)).entries[0]
|
||||||
|
|
||||||
assert entry.receipt["canonical_rows"]["strategies"][0]["created_at"] == transaction_timestamp
|
assert entry.receipt["canonical_rows"]["strategies"][0]["created_at"] == (
|
||||||
|
rebuild.replay_receipt.canonical_utc_timestamp(transaction_timestamp)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_revise_strategy_rejects_reviewed_at_without_timezone(tmp_path: Path) -> None:
|
def test_revise_strategy_rejects_reviewed_at_without_timezone(tmp_path: Path) -> None:
|
||||||
|
|
@ -540,13 +930,8 @@ def test_revise_strategy_rejects_reviewed_at_without_timezone(tmp_path: Path) ->
|
||||||
for proposal_row in (material["approved_proposal"], material["applied_proposal"]):
|
for proposal_row in (material["approved_proposal"], material["applied_proposal"]):
|
||||||
proposal_row["reviewed_at"] = reviewed_at
|
proposal_row["reviewed_at"] = reviewed_at
|
||||||
material["approval_snapshot"]["reviewed_at"] = reviewed_at
|
material["approval_snapshot"]["reviewed_at"] = reviewed_at
|
||||||
rehash_revise_strategy_receipt(material)
|
with pytest.raises(ValueError, match="reviewed_at must be a timezone-aware ISO-8601 timestamp"):
|
||||||
|
rehash_revise_strategy_receipt(material)
|
||||||
with pytest.raises(rebuild.ReconstructionError) as exc_info:
|
|
||||||
rebuild.load_bundle(write_bundle(tmp_path, material=material))
|
|
||||||
|
|
||||||
assert exc_info.value.code == "invalid_mutating_receipt"
|
|
||||||
assert "reviewed_at must include a timezone" in exc_info.value.safe_message
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
@pytest.mark.parametrize(
|
||||||
|
|
@ -598,7 +983,7 @@ def test_revise_strategy_transition_builders_preserve_historical_prestate(tmp_pa
|
||||||
assert PRIOR_RETIRED_NODE_ID not in sql
|
assert PRIOR_RETIRED_NODE_ID not in sql
|
||||||
assert PRIOR_STRATEGY_ID in sql
|
assert PRIOR_STRATEGY_ID in sql
|
||||||
assert RECEIPT_STRATEGY_ID in sql
|
assert RECEIPT_STRATEGY_ID in sql
|
||||||
assert "updated_at = E'2026-07-14T01:00:30+00:00'::timestamptz" in sql
|
assert "updated_at = E'2026-07-14T01:00:30.000000Z'::timestamptz" in sql
|
||||||
|
|
||||||
|
|
||||||
def test_revise_strategy_transition_builders_allow_empty_prestate(tmp_path: Path) -> None:
|
def test_revise_strategy_transition_builders_allow_empty_prestate(tmp_path: Path) -> None:
|
||||||
|
|
@ -722,6 +1107,89 @@ def test_seed_sql_preserves_exact_rows_and_uses_only_fixed_tables(tmp_path: Path
|
||||||
assert "public.strategies" not in sql
|
assert "public.strategies" not in sql
|
||||||
|
|
||||||
|
|
||||||
|
def test_seed_sql_retains_the_exact_approved_row_outside_the_v1_approval_contract(
|
||||||
|
tmp_path: Path,
|
||||||
|
) -> None:
|
||||||
|
entry = rebuild.load_bundle(write_bundle(tmp_path)).entries[0]
|
||||||
|
approval_seed = {
|
||||||
|
**entry.approval_snapshot,
|
||||||
|
"approved_proposal_snapshot": entry.approved_proposal,
|
||||||
|
}
|
||||||
|
|
||||||
|
assert rebuild._jsonb_literal(approval_seed) in rebuild.build_seed_sql(entry)
|
||||||
|
assert "jsonb_set" in rebuild.build_seed_sql(entry)
|
||||||
|
assert "to_jsonb(proposal)" in rebuild.build_seed_sql(entry)
|
||||||
|
readback_sql = rebuild.build_proposal_readback_sql(entry.approved_proposal["id"])
|
||||||
|
assert "to_jsonb(a) - 'approved_proposal_snapshot'" in readback_sql
|
||||||
|
assert "'approved_proposal_snapshot', (select a.approved_proposal_snapshot" in readback_sql
|
||||||
|
|
||||||
|
|
||||||
|
def test_v3_seed_and_timestamp_normalization_preserve_contract_guards(tmp_path: Path) -> None:
|
||||||
|
entry = rebuild.load_bundle(write_bundle(tmp_path, material=v3_replay_material())).entries[0]
|
||||||
|
|
||||||
|
seed_sql = rebuild.build_seed_sql(entry)
|
||||||
|
normalization_sql = rebuild.build_applied_timestamp_normalization_sql(entry)
|
||||||
|
canonical_normalization_sql = rebuild.build_v3_canonical_timestamp_normalization_sql(entry)
|
||||||
|
|
||||||
|
assert entry.contract_version == 3
|
||||||
|
assert "approved_proposal_snapshot" in seed_sql
|
||||||
|
assert "public.claims" not in seed_sql
|
||||||
|
assert "public.sources" not in seed_sql
|
||||||
|
assert "captured_at = pinned.captured_at" in canonical_normalization_sql
|
||||||
|
assert "created_at = pinned.created_at" in canonical_normalization_sql
|
||||||
|
assert f"accepted_by_proposal_id = E'{V3_PROPOSAL_ID}'::uuid" in canonical_normalization_sql
|
||||||
|
assert "disable trigger teleo_v3_source_immutable" in canonical_normalization_sql
|
||||||
|
assert "enable trigger teleo_v3_source_immutable" in canonical_normalization_sql
|
||||||
|
assert "disable trigger all" not in canonical_normalization_sql
|
||||||
|
assert "trigger.tgenabled = 'O'" in canonical_normalization_sql
|
||||||
|
assert "disable trigger teleo_v3_guard_linked_proposal_decision" in normalization_sql
|
||||||
|
assert "disable trigger all" not in normalization_sql
|
||||||
|
assert normalization_sql.index("disable trigger") < normalization_sql.index("update kb_stage.kb_proposals")
|
||||||
|
assert normalization_sql.index("update kb_stage.kb_proposals") < normalization_sql.index("enable trigger")
|
||||||
|
assert "trigger.tgenabled = 'O'" in normalization_sql
|
||||||
|
|
||||||
|
|
||||||
|
def test_exporter_shaped_material_validates_storage_provenance_and_typed_source_timestamp(
|
||||||
|
tmp_path: Path,
|
||||||
|
) -> None:
|
||||||
|
material = exporter_shaped_v3_replay_material()
|
||||||
|
entry = rebuild.load_bundle(write_bundle(tmp_path, material=material)).entries[0]
|
||||||
|
|
||||||
|
assert frozenset(material) == EXPORTER_MATERIAL_FIELDS
|
||||||
|
assert len(entry.approval_snapshot) == 8
|
||||||
|
assert entry.approved_proposal["payload"]["apply_payload"]["sources"][0]["captured_at"].endswith("+02:00")
|
||||||
|
assert entry.receipt["canonical_rows"]["sources"][0]["captured_at"] == "2026-07-18T10:00:00.000000Z"
|
||||||
|
|
||||||
|
approval_seed = {
|
||||||
|
**entry.approval_snapshot,
|
||||||
|
"approved_proposal_snapshot": entry.approved_proposal,
|
||||||
|
}
|
||||||
|
assert rebuild._jsonb_literal(approval_seed) in rebuild.build_seed_sql(entry)
|
||||||
|
assert "public.sources" not in rebuild.build_seed_sql(entry)
|
||||||
|
readback_sql = rebuild.build_proposal_readback_sql(entry.approved_proposal["id"])
|
||||||
|
assert "to_jsonb(a) - 'approved_proposal_snapshot'" in readback_sql
|
||||||
|
assert "'approved_proposal_snapshot', (select a.approved_proposal_snapshot" in readback_sql
|
||||||
|
|
||||||
|
actual_rows = copy.deepcopy(entry.receipt["canonical_rows"])
|
||||||
|
actual_rows["sources"][0]["captured_at"] = "2026-07-18T12:00:00+02:00"
|
||||||
|
rebuild._assert_exact_canonical_rows(
|
||||||
|
actual_rows,
|
||||||
|
entry.receipt["canonical_rows"],
|
||||||
|
code="fixture_mismatch",
|
||||||
|
action="exporter-shaped fixture",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_contract_order_rejects_v2_after_v3() -> None:
|
||||||
|
v3 = type("Entry", (), {"contract_version": 3})()
|
||||||
|
v2 = type("Entry", (), {"contract_version": 2})()
|
||||||
|
|
||||||
|
rebuild._validate_contract_order([v2, v3])
|
||||||
|
with pytest.raises(rebuild.ReconstructionError, match="cannot appear") as exc_info:
|
||||||
|
rebuild._validate_contract_order([v3, v2])
|
||||||
|
assert exc_info.value.code == "contract_order_mismatch"
|
||||||
|
|
||||||
|
|
||||||
def test_private_psql_failure_output_is_replaced_with_a_fingerprint(
|
def test_private_psql_failure_output_is_replaced_with_a_fingerprint(
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
|
|
@ -1069,6 +1537,11 @@ def source_postgres() -> Iterator[tuple[str, str]]:
|
||||||
)
|
)
|
||||||
if migration.returncode != 0:
|
if migration.returncode != 0:
|
||||||
pytest.fail(f"could not apply guarded prerequisites: {migration.stderr}")
|
pytest.fail(f"could not apply guarded prerequisites: {migration.stderr}")
|
||||||
|
docker_psql(
|
||||||
|
container,
|
||||||
|
database,
|
||||||
|
"alter table kb_stage.kb_proposal_approvals add column if not exists approved_proposal_snapshot jsonb;",
|
||||||
|
)
|
||||||
yield container, database
|
yield container, database
|
||||||
finally:
|
finally:
|
||||||
subprocess.run(
|
subprocess.run(
|
||||||
|
|
@ -1176,6 +1649,7 @@ def seed_proposal_and_approval(
|
||||||
approved: dict,
|
approved: dict,
|
||||||
approval: dict,
|
approval: dict,
|
||||||
) -> dict:
|
) -> dict:
|
||||||
|
approval_seed = rebuild._approval_seed_row(approved, approval)
|
||||||
sql = f"""begin;
|
sql = f"""begin;
|
||||||
set local standard_conforming_strings = on;
|
set local standard_conforming_strings = on;
|
||||||
insert into kb_stage.kb_proposals
|
insert into kb_stage.kb_proposals
|
||||||
|
|
@ -1184,7 +1658,7 @@ select seeded.* from jsonb_populate_record(
|
||||||
) seeded;
|
) seeded;
|
||||||
insert into kb_stage.kb_proposal_approvals
|
insert into kb_stage.kb_proposal_approvals
|
||||||
select seeded.* from jsonb_populate_record(
|
select seeded.* from jsonb_populate_record(
|
||||||
null::kb_stage.kb_proposal_approvals, {rebuild._jsonb_literal(approval)}
|
null::kb_stage.kb_proposal_approvals, {rebuild._jsonb_literal(approval_seed)}
|
||||||
) seeded;
|
) seeded;
|
||||||
commit;
|
commit;
|
||||||
"""
|
"""
|
||||||
|
|
@ -1208,6 +1682,7 @@ def run_genesis_ledger_command(
|
||||||
artifact_stem: str,
|
artifact_stem: str,
|
||||||
container_prefix: str,
|
container_prefix: str,
|
||||||
run_number: int,
|
run_number: int,
|
||||||
|
admit: bool = False,
|
||||||
) -> tuple[subprocess.CompletedProcess[str], dict, Path]:
|
) -> tuple[subprocess.CompletedProcess[str], dict, Path]:
|
||||||
ledger = {
|
ledger = {
|
||||||
"artifact": rebuild.LEDGER_ARTIFACT,
|
"artifact": rebuild.LEDGER_ARTIFACT,
|
||||||
|
|
@ -1233,6 +1708,32 @@ def run_genesis_ledger_command(
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
ledger_path = write_json(tmp_path / f"{artifact_stem}-ledger-run-{run_number}.json", ledger)
|
ledger_path = write_json(tmp_path / f"{artifact_stem}-ledger-run-{run_number}.json", ledger)
|
||||||
|
admission_path: Path | None = None
|
||||||
|
if admit:
|
||||||
|
material = json.loads(material_path.read_text(encoding="utf-8"))
|
||||||
|
ledger_entry = ledger["entries"][0]
|
||||||
|
admission_path = write_json(
|
||||||
|
tmp_path / f"{artifact_stem}-admission-run-{run_number}.json",
|
||||||
|
{
|
||||||
|
"artifact": rebuild.ADMISSION_ARTIFACT,
|
||||||
|
"contract_version": rebuild.ADMISSION_CONTRACT_VERSION,
|
||||||
|
"ledger_sha256": sha256_file(ledger_path),
|
||||||
|
"reviewed_by_handle": "m3taversal",
|
||||||
|
"reviewed_by_agent_id": REVIEWER_ID,
|
||||||
|
"reviewed_at": "2026-07-18T14:00:00+00:00",
|
||||||
|
"review_note": "Admit this exact source-backed lifecycle fixture.",
|
||||||
|
"entries": [
|
||||||
|
{
|
||||||
|
"sequence": 1,
|
||||||
|
"proposal_id": material["applied_proposal"]["id"],
|
||||||
|
"material_sha256": ledger_entry["sha256"],
|
||||||
|
"replay_material_sha256": ledger_entry["replay_material_sha256"],
|
||||||
|
"admission_scope": "collective_core",
|
||||||
|
"basis": "forgejo_primary_source",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
},
|
||||||
|
)
|
||||||
output = tmp_path / f"{artifact_stem}-reconstruction-receipt-run-{run_number}.json"
|
output = tmp_path / f"{artifact_stem}-reconstruction-receipt-run-{run_number}.json"
|
||||||
command = [
|
command = [
|
||||||
sys.executable,
|
sys.executable,
|
||||||
|
|
@ -1258,6 +1759,15 @@ def run_genesis_ledger_command(
|
||||||
"--output",
|
"--output",
|
||||||
str(output),
|
str(output),
|
||||||
]
|
]
|
||||||
|
if admission_path is not None:
|
||||||
|
command.extend(
|
||||||
|
[
|
||||||
|
"--admission-manifest",
|
||||||
|
str(admission_path),
|
||||||
|
"--admission-manifest-sha256",
|
||||||
|
sha256_file(admission_path),
|
||||||
|
]
|
||||||
|
)
|
||||||
completed = subprocess.run(
|
completed = subprocess.run(
|
||||||
command,
|
command,
|
||||||
cwd=rebuild.REPO_ROOT,
|
cwd=rebuild.REPO_ROOT,
|
||||||
|
|
@ -1468,3 +1978,110 @@ def test_live_revise_strategy_rebuild_is_exact_repeatable_and_leaves_no_containe
|
||||||
first["ledger"]["entries"][0]["replay_material_sha256"]
|
first["ledger"]["entries"][0]["replay_material_sha256"]
|
||||||
== second["ledger"]["entries"][0]["replay_material_sha256"]
|
== second["ledger"]["entries"][0]["replay_material_sha256"]
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_live_v3_exporter_material_genesis_plus_ledger_rebuild_is_exact_and_restores_guard(
|
||||||
|
tmp_path: Path,
|
||||||
|
source_postgres: tuple[str, str],
|
||||||
|
) -> None:
|
||||||
|
source_container, source_database = source_postgres
|
||||||
|
database = f"teleo_v3_{uuid.uuid4().hex[:12]}"
|
||||||
|
docker_psql(
|
||||||
|
source_container,
|
||||||
|
"postgres",
|
||||||
|
f'create database "{database}" template "{source_database}";',
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
genesis_dump, genesis_manifest = capture_source_snapshot(
|
||||||
|
source_container,
|
||||||
|
database,
|
||||||
|
tmp_path,
|
||||||
|
stem="v3-genesis",
|
||||||
|
)
|
||||||
|
docker_psql(source_container, database, "alter role kb_apply nologin;")
|
||||||
|
docker_psql(source_container, database, rebuild.V3_CONTRACT_PATH.read_text(encoding="utf-8"))
|
||||||
|
docker_psql(source_container, database, rebuild.GUARD_PREREQUISITES_PATH.read_text(encoding="utf-8"))
|
||||||
|
|
||||||
|
material = exporter_shaped_v3_replay_material()
|
||||||
|
material_path = write_json(tmp_path / "v3-entry-0001.json", material)
|
||||||
|
entry = rebuild._validate_entry_material(
|
||||||
|
material,
|
||||||
|
expected_sequence=1,
|
||||||
|
expected_replay_hash=material["replay_receipt"]["hashes"]["replay_material_sha256"],
|
||||||
|
material_path=material_path,
|
||||||
|
material_sha256=sha256_file(material_path),
|
||||||
|
)
|
||||||
|
docker_psql(source_container, database, rebuild.build_seed_sql(entry))
|
||||||
|
docker_psql(source_container, database, entry.current_apply_sql, role="kb_apply")
|
||||||
|
docker_psql(
|
||||||
|
source_container,
|
||||||
|
database,
|
||||||
|
rebuild.build_v3_canonical_timestamp_normalization_sql(entry),
|
||||||
|
)
|
||||||
|
docker_psql(source_container, database, rebuild.build_applied_timestamp_normalization_sql(entry))
|
||||||
|
source_readback = json.loads(
|
||||||
|
docker_psql(
|
||||||
|
source_container,
|
||||||
|
database,
|
||||||
|
rebuild.build_proposal_readback_sql(V3_PROPOSAL_ID),
|
||||||
|
).stdout.strip()
|
||||||
|
)
|
||||||
|
rebuild._assert_exact_proposal_readback(
|
||||||
|
source_readback,
|
||||||
|
{
|
||||||
|
"proposal": material["applied_proposal"],
|
||||||
|
"approval_snapshot": material["approval_snapshot"],
|
||||||
|
"approved_proposal_snapshot": material["approved_proposal"],
|
||||||
|
},
|
||||||
|
code="fixture_mismatch",
|
||||||
|
action="exporter-shaped source proposal",
|
||||||
|
)
|
||||||
|
assert len(source_readback["approval_snapshot"]) == 8
|
||||||
|
assert source_readback["approved_proposal_snapshot"]["payload"] == material["approved_proposal"]["payload"]
|
||||||
|
|
||||||
|
final_manifest = tmp_path / "v3-final-manifest.jsonl"
|
||||||
|
capture_manifest(source_container, database, final_manifest)
|
||||||
|
completed, receipt, output = run_genesis_ledger_command(
|
||||||
|
tmp_path=tmp_path,
|
||||||
|
database=database,
|
||||||
|
genesis_dump=genesis_dump,
|
||||||
|
genesis_manifest=genesis_manifest,
|
||||||
|
material_path=material_path,
|
||||||
|
final_manifest=final_manifest,
|
||||||
|
artifact_stem="v3",
|
||||||
|
container_prefix="teleo-genesis-ledger-v3",
|
||||||
|
run_number=1,
|
||||||
|
admit=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert receipt["status"] == "pass"
|
||||||
|
assert receipt["genesis_parity"]["status"] == "pass"
|
||||||
|
assert receipt["v3_contract"]["status"] == "pass"
|
||||||
|
assert receipt["admission"]["status"] == "review_metadata_supplied"
|
||||||
|
assert receipt["admission"]["canonical_retrieval_eligible"] is False
|
||||||
|
assert receipt["admission"]["cryptographically_authenticated"] is False
|
||||||
|
assert receipt["v3_contract"]["cutover"] == {
|
||||||
|
"active_apply_sessions": 0,
|
||||||
|
"apply_role_login_disabled": True,
|
||||||
|
}
|
||||||
|
assert receipt["v3_contract"]["catalog"] == {
|
||||||
|
"assessment_table_present": True,
|
||||||
|
"apply_role_login_restored": True,
|
||||||
|
"cutover_guard_enabled": True,
|
||||||
|
"decision_guard_enabled": True,
|
||||||
|
}
|
||||||
|
assert receipt["ledger"]["entries"][0]["contract_version"] == 3
|
||||||
|
assert receipt["ledger"]["entries"][0]["guarded_apply_executed"] is True
|
||||||
|
assert receipt["ledger"]["entries"][0]["canonical_rows_exact"] is True
|
||||||
|
assert receipt["final_parity"]["status"] == "pass"
|
||||||
|
assert receipt["cleanup"]["container_absent"] is True
|
||||||
|
assert PRIVATE_MARKER not in completed.stdout
|
||||||
|
assert PRIVATE_MARKER not in output.read_text(encoding="utf-8")
|
||||||
|
assert stat.S_IMODE(output.stat().st_mode) == 0o600
|
||||||
|
finally:
|
||||||
|
docker_psql(
|
||||||
|
source_container,
|
||||||
|
"postgres",
|
||||||
|
f'drop database if exists "{database}" with (force);',
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
|
|
||||||
442
tests/test_shard_corpus_extraction_work.py
Normal file
442
tests/test_shard_corpus_extraction_work.py
Normal file
|
|
@ -0,0 +1,442 @@
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import copy
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import stat
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from collections.abc import Callable, Iterator
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from ops import run_local_corpus_knowledge_rebuild as rebuild
|
||||||
|
from ops import shard_corpus_extraction_work as sharder
|
||||||
|
|
||||||
|
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
CLI = REPO_ROOT / "ops" / "shard_corpus_extraction_work.py"
|
||||||
|
FIXTURE_ROOT = REPO_ROOT / "tests" / "fixtures" / "teleo_corpus_rebuild"
|
||||||
|
PRIVATE_TEXT = "PRIVATE SOURCE BODY MUST NOT ENTER SHARDS"
|
||||||
|
|
||||||
|
|
||||||
|
def _load_json(path: Path) -> dict:
|
||||||
|
value = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
assert isinstance(value, dict)
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _write_json(path: Path, value: object) -> None:
|
||||||
|
path.write_bytes(sharder.rendered_json_bytes(value))
|
||||||
|
path.chmod(0o600)
|
||||||
|
|
||||||
|
|
||||||
|
def _tree_bytes(root: Path) -> dict[str, bytes]:
|
||||||
|
return {path.relative_to(root).as_posix(): path.read_bytes() for path in sorted(root.rglob("*")) if path.is_file()}
|
||||||
|
|
||||||
|
|
||||||
|
def _all_keys(value: object) -> Iterator[str]:
|
||||||
|
if isinstance(value, dict):
|
||||||
|
for key, child in value.items():
|
||||||
|
yield key
|
||||||
|
yield from _all_keys(child)
|
||||||
|
elif isinstance(value, list):
|
||||||
|
for child in value:
|
||||||
|
yield from _all_keys(child)
|
||||||
|
|
||||||
|
|
||||||
|
def _build_inventory(tmp_path: Path, source_root: Path = FIXTURE_ROOT, *, name: str = "inventory") -> Path:
|
||||||
|
output = tmp_path / name
|
||||||
|
rebuild.build_source_inventory_bundle(source_root, output)
|
||||||
|
return output
|
||||||
|
|
||||||
|
|
||||||
|
def _make_section_source(root: Path, count: int, *, private_text: str = PRIVATE_TEXT) -> Path:
|
||||||
|
root.mkdir(parents=True, exist_ok=True)
|
||||||
|
body = "".join(f"# Section {index:04d}\n\n{private_text} {index:04d}.\n\n" for index in range(count))
|
||||||
|
source = root / "source.md"
|
||||||
|
source.write_text(body, encoding="utf-8")
|
||||||
|
return source
|
||||||
|
|
||||||
|
|
||||||
|
def _run_cli(*args: str) -> subprocess.CompletedProcess[str]:
|
||||||
|
return subprocess.run(
|
||||||
|
[sys.executable, str(CLI), *args],
|
||||||
|
cwd=REPO_ROOT,
|
||||||
|
check=False,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _shard_documents(bundle: Path) -> list[tuple[Path, dict]]:
|
||||||
|
manifest = _load_json(bundle / sharder.MANIFEST_PATH)
|
||||||
|
return [(bundle / record["path"], _load_json(bundle / record["path"])) for record in manifest["shards"]]
|
||||||
|
|
||||||
|
|
||||||
|
def _rehash_bundle(bundle: Path) -> None:
|
||||||
|
manifest_path = bundle / sharder.MANIFEST_PATH
|
||||||
|
receipt_path = bundle / sharder.RECEIPT_PATH
|
||||||
|
manifest = _load_json(manifest_path)
|
||||||
|
for record in manifest["shards"]:
|
||||||
|
shard_path = bundle / record["path"]
|
||||||
|
document = _load_json(shard_path)
|
||||||
|
items = document["items"]
|
||||||
|
work_ids = [item["work_item_id"] for item in items]
|
||||||
|
document["work_item_count"] = len(items)
|
||||||
|
document["work_item_ids_sha256"] = sharder.canonical_sha256(work_ids)
|
||||||
|
_write_json(shard_path, document)
|
||||||
|
raw = shard_path.read_bytes()
|
||||||
|
record.update(
|
||||||
|
bytes=len(raw),
|
||||||
|
sha256=sharder.sha256_bytes(raw),
|
||||||
|
work_item_count=len(items),
|
||||||
|
work_item_ids_sha256=sharder.canonical_sha256(work_ids),
|
||||||
|
)
|
||||||
|
_write_json(manifest_path, manifest)
|
||||||
|
|
||||||
|
receipt = _load_json(receipt_path)
|
||||||
|
artifact_paths = [sharder.MANIFEST_PATH, *(record["path"] for record in manifest["shards"])]
|
||||||
|
artifacts = sharder._artifact_records(bundle, artifact_paths)
|
||||||
|
receipt["outputs"] = {
|
||||||
|
"artifact_count": len(artifacts),
|
||||||
|
"artifacts": artifacts,
|
||||||
|
"manifest_path": sharder.MANIFEST_PATH,
|
||||||
|
}
|
||||||
|
receipt["hashes"] = {
|
||||||
|
"bundle_tree_sha256": sharder.canonical_sha256(artifacts),
|
||||||
|
"manifest_file_sha256": sharder.sha256_bytes(manifest_path.read_bytes()),
|
||||||
|
}
|
||||||
|
receipt["hashes"]["receipt_content_sha256"] = sharder.canonical_sha256(receipt)
|
||||||
|
_write_json(receipt_path, receipt)
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_is_deterministic_private_complete_and_source_text_free(tmp_path: Path) -> None:
|
||||||
|
source_root = tmp_path / "source"
|
||||||
|
_make_section_source(source_root, 12)
|
||||||
|
inventory_bundle = _build_inventory(tmp_path, source_root)
|
||||||
|
first = tmp_path / "shards-a"
|
||||||
|
second = tmp_path / "shards-b"
|
||||||
|
|
||||||
|
first_result = sharder.build_shard_bundle(inventory_bundle, first, 4)["verification"]
|
||||||
|
second_result = sharder.build_shard_bundle(inventory_bundle, second, 4)["verification"]
|
||||||
|
|
||||||
|
assert first_result["status"] == second_result["status"] == "pass"
|
||||||
|
assert first_result["work_item_count"] == second_result["work_item_count"] == 12
|
||||||
|
assert first_result["shard_count"] == second_result["shard_count"] == 4
|
||||||
|
assert first_result["bundle_tree_sha256"] == second_result["bundle_tree_sha256"]
|
||||||
|
assert _tree_bytes(first) == _tree_bytes(second)
|
||||||
|
persisted = b"".join(_tree_bytes(first).values())
|
||||||
|
assert PRIVATE_TEXT.encode() not in persisted
|
||||||
|
assert str(source_root).encode() not in persisted
|
||||||
|
assert str(inventory_bundle).encode() not in persisted
|
||||||
|
assert str(first).encode() not in persisted
|
||||||
|
|
||||||
|
forbidden_keys = {"claim", "claims", "proposal", "proposals", "admitted", "database_rows"}
|
||||||
|
for document in [_load_json(first / sharder.MANIFEST_PATH), _load_json(first / sharder.RECEIPT_PATH)]:
|
||||||
|
assert forbidden_keys.isdisjoint(_all_keys(document))
|
||||||
|
for key, expected in sharder.NO_WRITE_FLAGS.items():
|
||||||
|
assert document[key] is expected
|
||||||
|
|
||||||
|
for path in first.rglob("*"):
|
||||||
|
expected_mode = 0o600 if path.is_file() else 0o700
|
||||||
|
assert stat.S_IMODE(path.stat().st_mode) == expected_mode
|
||||||
|
assert stat.S_IMODE(first.stat().st_mode) == 0o700
|
||||||
|
assert not (first / sharder.INCOMPLETE_MARKER).exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_hash_assignment_covers_every_exact_inventory_item_once(tmp_path: Path) -> None:
|
||||||
|
source_root = tmp_path / "source"
|
||||||
|
_make_section_source(source_root, 40)
|
||||||
|
inventory_bundle = _build_inventory(tmp_path, source_root)
|
||||||
|
output = tmp_path / "shards"
|
||||||
|
sharder.build_shard_bundle(inventory_bundle, output, 7)
|
||||||
|
inventory_items = _load_json(inventory_bundle / rebuild.EXTRACTION_WORK_INDEX_PATH)["items"]
|
||||||
|
|
||||||
|
assigned: list[dict] = []
|
||||||
|
for index, (_path, document) in enumerate(_shard_documents(output)):
|
||||||
|
ids = [item["work_item_id"] for item in document["items"]]
|
||||||
|
assert ids == sorted(ids, key=lambda value: value.encode("utf-8"))
|
||||||
|
assert all(sharder.assignment_for(work_id, 7) == index for work_id in ids)
|
||||||
|
assigned.extend(document["items"])
|
||||||
|
|
||||||
|
assert len(assigned) == len(inventory_items)
|
||||||
|
assert {item["work_item_id"] for item in assigned} == {item["work_item_id"] for item in inventory_items}
|
||||||
|
assert {item["work_item_id"]: item for item in assigned} == {item["work_item_id"]: item for item in inventory_items}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("shard_count", [0, 1025])
|
||||||
|
def test_shard_count_must_be_within_hard_bounds(tmp_path: Path, shard_count: int) -> None:
|
||||||
|
inventory_bundle = _build_inventory(tmp_path)
|
||||||
|
with pytest.raises(sharder.ShardError) as error:
|
||||||
|
sharder.build_shard_bundle(inventory_bundle, tmp_path / "shards", shard_count)
|
||||||
|
assert error.value.code == "invalid_shard_count"
|
||||||
|
assert not (tmp_path / "shards").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_shard_count_must_not_exceed_work_item_count(tmp_path: Path) -> None:
|
||||||
|
inventory_bundle = _build_inventory(tmp_path)
|
||||||
|
work_count = _load_json(inventory_bundle / rebuild.EXTRACTION_WORK_INDEX_PATH)["work_item_count"]
|
||||||
|
with pytest.raises(sharder.ShardError) as error:
|
||||||
|
sharder.build_shard_bundle(inventory_bundle, tmp_path / "shards", work_count + 1)
|
||||||
|
assert error.value.code == "invalid_shard_count"
|
||||||
|
|
||||||
|
|
||||||
|
def test_existing_output_is_never_overwritten(tmp_path: Path) -> None:
|
||||||
|
inventory_bundle = _build_inventory(tmp_path)
|
||||||
|
output = tmp_path / "shards"
|
||||||
|
output.mkdir(mode=0o700)
|
||||||
|
sentinel = output / "sentinel"
|
||||||
|
sentinel.write_text("preserve", encoding="utf-8")
|
||||||
|
|
||||||
|
with pytest.raises(sharder.ShardError) as error:
|
||||||
|
sharder.build_shard_bundle(inventory_bundle, output, 2)
|
||||||
|
|
||||||
|
assert error.value.code == "output_exists"
|
||||||
|
assert sentinel.read_text(encoding="utf-8") == "preserve"
|
||||||
|
|
||||||
|
|
||||||
|
def test_marker_write_failure_closes_and_removes_reserved_output(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
inventory_bundle = _build_inventory(tmp_path)
|
||||||
|
output = tmp_path / "shards"
|
||||||
|
original_write = sharder._write_bytes_at
|
||||||
|
|
||||||
|
def fail_after_write(directory_fd: int, name: str, payload: bytes) -> None:
|
||||||
|
original_write(directory_fd, name, payload)
|
||||||
|
raise OSError("synthetic marker publication failure")
|
||||||
|
|
||||||
|
monkeypatch.setattr(sharder, "_write_bytes_at", fail_after_write)
|
||||||
|
with pytest.raises(OSError, match="synthetic marker publication failure"):
|
||||||
|
sharder.build_shard_bundle(inventory_bundle, output, 2)
|
||||||
|
|
||||||
|
assert not output.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_receipt_is_published_while_incomplete_marker_exists(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
inventory_bundle = _build_inventory(tmp_path)
|
||||||
|
output = tmp_path / "shards"
|
||||||
|
|
||||||
|
def inspect(reserved: sharder.ReservedOutput) -> None:
|
||||||
|
marker = os.stat(sharder.INCOMPLETE_MARKER, dir_fd=reserved.root_fd, follow_symlinks=False)
|
||||||
|
assert stat.S_ISREG(marker.st_mode)
|
||||||
|
with pytest.raises(FileNotFoundError):
|
||||||
|
os.stat(sharder.RECEIPT_PATH, dir_fd=reserved.root_fd, follow_symlinks=False)
|
||||||
|
|
||||||
|
monkeypatch.setattr(sharder, "_before_receipt_publication", inspect)
|
||||||
|
sharder.build_shard_bundle(inventory_bundle, output, 2)
|
||||||
|
assert not (output / sharder.INCOMPLETE_MARKER).exists()
|
||||||
|
assert (output / sharder.RECEIPT_PATH).is_file()
|
||||||
|
|
||||||
|
|
||||||
|
def test_destination_swap_does_not_overwrite_or_cleanup_replacement(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
inventory_bundle = _build_inventory(tmp_path)
|
||||||
|
output = tmp_path / "shards"
|
||||||
|
moved = tmp_path / "reserved-moved"
|
||||||
|
|
||||||
|
def swap(_reserved: sharder.ReservedOutput) -> None:
|
||||||
|
output.rename(moved)
|
||||||
|
output.mkdir(mode=0o700)
|
||||||
|
sentinel = output / "replacement-sentinel"
|
||||||
|
sentinel.write_text("preserve replacement", encoding="utf-8")
|
||||||
|
sentinel.chmod(0o600)
|
||||||
|
|
||||||
|
monkeypatch.setattr(sharder, "_after_output_reservation", swap)
|
||||||
|
with pytest.raises(sharder.ShardError) as error:
|
||||||
|
sharder.build_shard_bundle(inventory_bundle, output, 2)
|
||||||
|
|
||||||
|
assert error.value.code == "output_binding_changed"
|
||||||
|
assert (output / "replacement-sentinel").read_text(encoding="utf-8") == "preserve replacement"
|
||||||
|
assert moved.is_dir()
|
||||||
|
assert list(moved.iterdir()) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_inventory_change_before_publication_rejects_and_cleans_owned_output(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
inventory_bundle = _build_inventory(tmp_path)
|
||||||
|
output = tmp_path / "shards"
|
||||||
|
|
||||||
|
def mutate(root: Path) -> None:
|
||||||
|
receipt = root / rebuild.RECEIPT_PATH
|
||||||
|
receipt.write_bytes(receipt.read_bytes() + b" ")
|
||||||
|
|
||||||
|
monkeypatch.setattr(sharder, "_before_final_publication", mutate)
|
||||||
|
with pytest.raises(sharder.ShardError) as error:
|
||||||
|
sharder.build_shard_bundle(inventory_bundle, output, 2)
|
||||||
|
|
||||||
|
assert error.value.code == "inventory_changed"
|
||||||
|
assert not output.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def _mutate_unknown(bundle: Path) -> None:
|
||||||
|
for path, document in _shard_documents(bundle):
|
||||||
|
if document["items"]:
|
||||||
|
document["items"][0]["work_item_id"] = "f" * 64
|
||||||
|
document["items"].sort(key=lambda item: item["work_item_id"].encode("utf-8"))
|
||||||
|
_write_json(path, document)
|
||||||
|
return
|
||||||
|
raise AssertionError("fixture unexpectedly produced no shard items")
|
||||||
|
|
||||||
|
|
||||||
|
def _mutate_missing(bundle: Path) -> None:
|
||||||
|
for path, document in _shard_documents(bundle):
|
||||||
|
if document["items"]:
|
||||||
|
document["items"].pop()
|
||||||
|
_write_json(path, document)
|
||||||
|
return
|
||||||
|
raise AssertionError("fixture unexpectedly produced no shard items")
|
||||||
|
|
||||||
|
|
||||||
|
def _mutate_duplicate(bundle: Path) -> None:
|
||||||
|
for path, document in _shard_documents(bundle):
|
||||||
|
if document["items"]:
|
||||||
|
document["items"].append(copy.deepcopy(document["items"][0]))
|
||||||
|
document["items"].sort(key=lambda item: item["work_item_id"].encode("utf-8"))
|
||||||
|
_write_json(path, document)
|
||||||
|
return
|
||||||
|
raise AssertionError("fixture unexpectedly produced no shard items")
|
||||||
|
|
||||||
|
|
||||||
|
def _mutate_overclaim(bundle: Path) -> None:
|
||||||
|
receipt_path = bundle / sharder.RECEIPT_PATH
|
||||||
|
receipt = _load_json(receipt_path)
|
||||||
|
receipt["database_write_performed"] = True
|
||||||
|
_write_json(receipt_path, receipt)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("mutator", "expected_codes"),
|
||||||
|
[
|
||||||
|
(_mutate_unknown, {"unknown_work_item"}),
|
||||||
|
(_mutate_missing, {"missing_work_item"}),
|
||||||
|
(_mutate_duplicate, {"duplicate_work_item", "invalid_shard_order"}),
|
||||||
|
(_mutate_overclaim, {"false_sharding_claim"}),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_verifier_rejects_fully_rehashed_semantic_tampering(
|
||||||
|
tmp_path: Path,
|
||||||
|
mutator: Callable[[Path], None],
|
||||||
|
expected_codes: set[str],
|
||||||
|
) -> None:
|
||||||
|
source_root = tmp_path / "source"
|
||||||
|
_make_section_source(source_root, 20)
|
||||||
|
inventory_bundle = _build_inventory(tmp_path, source_root)
|
||||||
|
output = tmp_path / "shards"
|
||||||
|
sharder.build_shard_bundle(inventory_bundle, output, 4)
|
||||||
|
|
||||||
|
mutator(output)
|
||||||
|
_rehash_bundle(output)
|
||||||
|
|
||||||
|
with pytest.raises(sharder.ShardError) as error:
|
||||||
|
sharder.verify_shard_bundle(inventory_bundle, output)
|
||||||
|
assert error.value.code in expected_codes
|
||||||
|
|
||||||
|
|
||||||
|
def test_verifier_rejects_wrong_inventory_even_with_same_cardinality(tmp_path: Path) -> None:
|
||||||
|
first_source = tmp_path / "source-a"
|
||||||
|
second_source = tmp_path / "source-b"
|
||||||
|
_make_section_source(first_source, 8, private_text="first corpus")
|
||||||
|
_make_section_source(second_source, 8, private_text="second corpus")
|
||||||
|
first_inventory = _build_inventory(tmp_path, first_source, name="inventory-a")
|
||||||
|
second_inventory = _build_inventory(tmp_path, second_source, name="inventory-b")
|
||||||
|
output = tmp_path / "shards"
|
||||||
|
sharder.build_shard_bundle(first_inventory, output, 3)
|
||||||
|
|
||||||
|
with pytest.raises(sharder.ShardError) as error:
|
||||||
|
sharder.verify_shard_bundle(second_inventory, output)
|
||||||
|
assert error.value.code == "inventory_mismatch"
|
||||||
|
|
||||||
|
|
||||||
|
def test_verifier_rejects_extra_file_and_incomplete_marker(tmp_path: Path) -> None:
|
||||||
|
inventory_bundle = _build_inventory(tmp_path)
|
||||||
|
extra_output = tmp_path / "extra-shards"
|
||||||
|
sharder.build_shard_bundle(inventory_bundle, extra_output, 2)
|
||||||
|
extra = extra_output / "extra.json"
|
||||||
|
extra.write_text("{}\n", encoding="utf-8")
|
||||||
|
extra.chmod(0o600)
|
||||||
|
with pytest.raises(sharder.ShardError) as extra_error:
|
||||||
|
sharder.verify_shard_bundle(inventory_bundle, extra_output)
|
||||||
|
assert extra_error.value.code == "unexpected_bundle_entry"
|
||||||
|
|
||||||
|
incomplete_output = tmp_path / "incomplete-shards"
|
||||||
|
sharder.build_shard_bundle(inventory_bundle, incomplete_output, 2)
|
||||||
|
marker = incomplete_output / sharder.INCOMPLETE_MARKER
|
||||||
|
marker.write_text("incomplete\n", encoding="utf-8")
|
||||||
|
marker.chmod(0o600)
|
||||||
|
with pytest.raises(sharder.ShardError) as incomplete_error:
|
||||||
|
sharder.verify_shard_bundle(inventory_bundle, incomplete_output)
|
||||||
|
assert incomplete_error.value.code == "incomplete_shard_bundle"
|
||||||
|
|
||||||
|
|
||||||
|
def test_verifier_rejects_symlinked_shard(tmp_path: Path) -> None:
|
||||||
|
inventory_bundle = _build_inventory(tmp_path)
|
||||||
|
output = tmp_path / "shards"
|
||||||
|
sharder.build_shard_bundle(inventory_bundle, output, 2)
|
||||||
|
manifest = _load_json(output / sharder.MANIFEST_PATH)
|
||||||
|
target = output / manifest["shards"][0]["path"]
|
||||||
|
retained = target.with_suffix(".retained")
|
||||||
|
target.rename(retained)
|
||||||
|
try:
|
||||||
|
target.symlink_to(retained.name)
|
||||||
|
except OSError as exc:
|
||||||
|
pytest.skip(f"symlinks unavailable: {exc}")
|
||||||
|
|
||||||
|
with pytest.raises(sharder.ShardError) as error:
|
||||||
|
sharder.verify_shard_bundle(inventory_bundle, output)
|
||||||
|
assert error.value.code == "unsafe_bundle_entry"
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_build_and_verify_return_path_free_receipts(tmp_path: Path) -> None:
|
||||||
|
inventory_bundle = _build_inventory(tmp_path)
|
||||||
|
output = tmp_path / "shards"
|
||||||
|
build = _run_cli(
|
||||||
|
"--inventory-bundle",
|
||||||
|
str(inventory_bundle),
|
||||||
|
"--output-dir",
|
||||||
|
str(output),
|
||||||
|
"--shard-count",
|
||||||
|
"2",
|
||||||
|
)
|
||||||
|
verify = _run_cli(
|
||||||
|
"--inventory-bundle",
|
||||||
|
str(inventory_bundle),
|
||||||
|
"--verify-bundle",
|
||||||
|
str(output),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert build.returncode == verify.returncode == 0, (build.stderr, build.stdout, verify.stderr, verify.stdout)
|
||||||
|
build_status = json.loads(build.stdout)
|
||||||
|
verify_status = json.loads(verify.stdout)
|
||||||
|
assert build_status["status"] == verify_status["status"] == "pass"
|
||||||
|
assert build_status["bundle_tree_sha256"] == verify_status["bundle_tree_sha256"]
|
||||||
|
assert build_status["work_item_count"] == verify_status["work_item_count"] == 3
|
||||||
|
assert str(inventory_bundle) not in build.stdout + verify.stdout
|
||||||
|
assert str(output) not in build.stdout + verify.stdout
|
||||||
|
for key, expected in sharder.NO_WRITE_FLAGS.items():
|
||||||
|
assert build_status[key] is verify_status[key] is expected
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_rejects_invalid_mode_arguments_without_output(tmp_path: Path) -> None:
|
||||||
|
inventory_bundle = _build_inventory(tmp_path)
|
||||||
|
result = _run_cli(
|
||||||
|
"--inventory-bundle",
|
||||||
|
str(inventory_bundle),
|
||||||
|
"--verify-bundle",
|
||||||
|
str(tmp_path / "missing"),
|
||||||
|
"--shard-count",
|
||||||
|
"2",
|
||||||
|
)
|
||||||
|
status = json.loads(result.stdout)
|
||||||
|
assert result.returncode == 2
|
||||||
|
assert status["status"] == "rejected"
|
||||||
|
assert status["reason"] == "invalid_arguments"
|
||||||
|
|
@ -13,6 +13,8 @@ sys.path.insert(0, str(REPO_ROOT / "scripts"))
|
||||||
|
|
||||||
import stage_normalized_proposal as stage # noqa: E402
|
import stage_normalized_proposal as stage # noqa: E402
|
||||||
|
|
||||||
|
from tests import test_compile_kb_source_packet as source_compiler_fixture # noqa: E402
|
||||||
|
|
||||||
PARENT_ID = "77777777-7777-4777-8777-777777777701"
|
PARENT_ID = "77777777-7777-4777-8777-777777777701"
|
||||||
AGENT_ID = "11111111-1111-4111-8111-111111111111"
|
AGENT_ID = "11111111-1111-4111-8111-111111111111"
|
||||||
DOCUMENT = (
|
DOCUMENT = (
|
||||||
|
|
@ -211,6 +213,131 @@ def test_stage_normalized_child_binds_to_full_disposable_container_id(
|
||||||
assert result["database_receipt"]["created"] is True
|
assert result["database_receipt"]["created"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_stage_normalized_child_does_not_require_an_ambient_production_container(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
target = {
|
||||||
|
"id": "clone-container-id",
|
||||||
|
"name": "composition-clone",
|
||||||
|
"labels": {stage.bound.DISPOSABLE_LABEL_KEY: stage.bound.DISPOSABLE_LABEL_VALUE},
|
||||||
|
"network_mode": "none",
|
||||||
|
"ports": {},
|
||||||
|
"mount_sources": [],
|
||||||
|
}
|
||||||
|
|
||||||
|
def identity(name: str) -> dict[str, Any]:
|
||||||
|
if name == "composition-clone":
|
||||||
|
return target
|
||||||
|
raise stage.bound.CheckpointError("docker inspect teleo-pg: no such object: teleo-pg")
|
||||||
|
|
||||||
|
monkeypatch.setattr(stage.bound, "container_identity", identity)
|
||||||
|
child = stage.prepare_normalized_child(_parent())
|
||||||
|
monkeypatch.setattr(
|
||||||
|
stage.bound,
|
||||||
|
"_psql_json",
|
||||||
|
lambda container, database, sql: {
|
||||||
|
"created": True,
|
||||||
|
"proposal_id": child["id"],
|
||||||
|
"proposal_type": child["proposal_type"],
|
||||||
|
"status": "pending_review",
|
||||||
|
"source_ref": child["source_ref"],
|
||||||
|
"payload_sha256": child["payload_sha256"],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
result = stage.stage_normalized_child("composition-clone", "teleo", _parent())
|
||||||
|
|
||||||
|
assert result["bound_container_id"] == target["id"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_stage_normalized_child_refuses_unexpected_production_inspection_failure(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
target = {
|
||||||
|
"id": "clone-container-id",
|
||||||
|
"name": "composition-clone",
|
||||||
|
"labels": {stage.bound.DISPOSABLE_LABEL_KEY: stage.bound.DISPOSABLE_LABEL_VALUE},
|
||||||
|
"network_mode": "none",
|
||||||
|
"ports": {},
|
||||||
|
"mount_sources": [],
|
||||||
|
}
|
||||||
|
|
||||||
|
def identity(name: str) -> dict[str, Any]:
|
||||||
|
if name == "composition-clone":
|
||||||
|
return target
|
||||||
|
raise stage.bound.CheckpointError("docker inspect teleo-pg: permission denied")
|
||||||
|
|
||||||
|
monkeypatch.setattr(stage.bound, "container_identity", identity)
|
||||||
|
|
||||||
|
with pytest.raises(stage.bound.CheckpointError, match="permission denied"):
|
||||||
|
stage.stage_normalized_child("composition-clone", "teleo", _parent())
|
||||||
|
|
||||||
|
|
||||||
|
def test_compiled_v3_child_revalidates_exact_bundle_and_rejects_tampering(tmp_path: Path) -> None:
|
||||||
|
bundle = source_compiler_fixture._compile_v3(tmp_path / "source")
|
||||||
|
|
||||||
|
child = stage.prepare_compiled_v3_child(bundle)
|
||||||
|
|
||||||
|
assert child == bundle["strict_child_proposal"]
|
||||||
|
assert child is not bundle["strict_child_proposal"]
|
||||||
|
assert child["payload"]["apply_payload"]["contract_version"] == 3
|
||||||
|
|
||||||
|
tampered = copy.deepcopy(bundle)
|
||||||
|
tampered["strict_child_proposal"]["payload"]["apply_payload"]["claims"][0]["proposition"] += " tampered"
|
||||||
|
with pytest.raises(stage.bound.CheckpointError, match="bundle content hash does not match"):
|
||||||
|
stage.prepare_compiled_v3_child(tampered)
|
||||||
|
|
||||||
|
|
||||||
|
def test_stage_compiled_v3_bundle_binds_to_disposable_container(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
bundle = source_compiler_fixture._compile_v3(tmp_path / "source")
|
||||||
|
child = bundle["strict_child_proposal"]
|
||||||
|
target = {
|
||||||
|
"id": "clone-container-id",
|
||||||
|
"name": "composition-clone",
|
||||||
|
"labels": {stage.bound.DISPOSABLE_LABEL_KEY: stage.bound.DISPOSABLE_LABEL_VALUE},
|
||||||
|
"network_mode": "none",
|
||||||
|
"ports": {},
|
||||||
|
"mount_sources": ["/clone-data"],
|
||||||
|
}
|
||||||
|
production = {
|
||||||
|
"id": "production-container-id",
|
||||||
|
"name": stage.bound.PRODUCTION_CONTAINER,
|
||||||
|
"labels": {},
|
||||||
|
"network_mode": "bridge",
|
||||||
|
"ports": {},
|
||||||
|
"mount_sources": ["/production-data"],
|
||||||
|
}
|
||||||
|
monkeypatch.setattr(
|
||||||
|
stage.bound,
|
||||||
|
"container_identity",
|
||||||
|
lambda name: target if name == "composition-clone" else production,
|
||||||
|
)
|
||||||
|
observed: dict[str, Any] = {}
|
||||||
|
|
||||||
|
def fake_psql(container: str, database: str, sql: str) -> dict[str, Any]:
|
||||||
|
observed.update({"container": container, "database": database, "sql": sql})
|
||||||
|
return {
|
||||||
|
"created": True,
|
||||||
|
"proposal_id": child["id"],
|
||||||
|
"proposal_type": child["proposal_type"],
|
||||||
|
"status": "pending_review",
|
||||||
|
"source_ref": child["source_ref"],
|
||||||
|
"payload_sha256": child["payload_sha256"],
|
||||||
|
}
|
||||||
|
|
||||||
|
monkeypatch.setattr(stage.bound, "_psql_json", fake_psql)
|
||||||
|
|
||||||
|
result = stage.stage_compiled_v3_bundle("composition-clone", "teleo", bundle)
|
||||||
|
|
||||||
|
assert observed["container"] == "clone-container-id"
|
||||||
|
assert observed["database"] == "teleo"
|
||||||
|
assert "insert into kb_stage.kb_proposals" in observed["sql"].lower()
|
||||||
|
assert result["child"]["id"] == child["id"]
|
||||||
|
|
||||||
|
|
||||||
def test_main_rejects_the_production_container_without_reading_the_packet(tmp_path: Path) -> None:
|
def test_main_rejects_the_production_container_without_reading_the_packet(tmp_path: Path) -> None:
|
||||||
missing = tmp_path / "missing.json"
|
missing = tmp_path / "missing.json"
|
||||||
|
|
||||||
|
|
|
||||||
799
tests/test_teleo_v3_epistemic_contract.py
Normal file
799
tests/test_teleo_v3_epistemic_contract.py
Normal file
|
|
@ -0,0 +1,799 @@
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from ops import verify_teleo_v3_epistemic_contract as verifier
|
||||||
|
|
||||||
|
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
MIGRATION_SQL = REPO_ROOT / "ops" / "teleo_v3_epistemic_contract.sql"
|
||||||
|
ROLLBACK_SQL = REPO_ROOT / "ops" / "rollback_teleo_v3_epistemic_contract.sql"
|
||||||
|
SECRET_STARTUP_MARKER = "secret-startup-marker"
|
||||||
|
|
||||||
|
|
||||||
|
def test_install_migration_is_additive_and_legacy_compatible() -> None:
|
||||||
|
sql = MIGRATION_SQL.read_text(encoding="utf-8").lower()
|
||||||
|
|
||||||
|
assert "add column if not exists" in sql
|
||||||
|
assert "create table if not exists public.claim_evidence_assessments" in sql
|
||||||
|
assert "admission_state text not null default 'legacy'" in sql
|
||||||
|
assert "admission_state = 'legacy'" in sql
|
||||||
|
assert "admission_state = 'admitted'" in sql
|
||||||
|
assert "accepted_by_proposal_id" in sql
|
||||||
|
assert "accepted_from_proposal_id" in sql
|
||||||
|
assert "teleo_v3_require_linked_insert" in sql
|
||||||
|
assert (
|
||||||
|
"create or replace function kb_stage.teleo_v3_require_accepted_proposal()\nreturns trigger\nlanguage plpgsql\nsecurity definer"
|
||||||
|
in sql
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
"create or replace function kb_stage.teleo_v3_enforce_apply_contract_cutover()\n"
|
||||||
|
"returns trigger\nlanguage plpgsql\nsecurity definer" in sql
|
||||||
|
)
|
||||||
|
assert "alter function kb_stage.teleo_v3_require_accepted_proposal() owner to kb_gate_owner" in sql
|
||||||
|
assert "revoke all on function kb_stage.teleo_v3_require_accepted_proposal()" in sql
|
||||||
|
assert "alter function kb_stage.teleo_v3_enforce_apply_contract_cutover() owner to kb_gate_owner" in sql
|
||||||
|
assert "revoke all on function kb_stage.teleo_v3_enforce_apply_contract_cutover()" in sql
|
||||||
|
assert "from public, kb_apply, kb_review" in sql
|
||||||
|
assert "security definer trigger invariants failed" in sql
|
||||||
|
assert "kb_gate_owner must be nologin noinherit and unprivileged" in sql
|
||||||
|
for attribute in (
|
||||||
|
"not role.rolcanlogin",
|
||||||
|
"not role.rolinherit",
|
||||||
|
"not role.rolsuper",
|
||||||
|
"not role.rolcreatedb",
|
||||||
|
"not role.rolcreaterole",
|
||||||
|
"not role.rolreplication",
|
||||||
|
"not role.rolbypassrls",
|
||||||
|
):
|
||||||
|
assert attribute in sql
|
||||||
|
assert "teleo_v3_reject_delete" in sql
|
||||||
|
assert "teleo_v3_proposal_apply_receipt_check" in sql
|
||||||
|
assert "create or replace trigger teleo_v3_claim_immutable" in sql
|
||||||
|
assert "create or replace trigger teleo_v3_guard_linked_proposal_decision" in sql
|
||||||
|
assert "'status', 'admission_state'" in sql
|
||||||
|
assert "'updated_at', 'accepted_by_proposal_id'" in sql
|
||||||
|
assert "legacy is reserved for rows present at migration" in sql
|
||||||
|
assert "cutover order: deploy the cutover-aware writer" in sql
|
||||||
|
assert "alter role kb_apply nologin" in sql
|
||||||
|
assert "active kb_apply sessions must exit" in sql
|
||||||
|
assert "retains % pending/approved v2 proposal(s) for explicit migration or rejection" in sql
|
||||||
|
assert "teleo_v3_00_enforce_apply_contract_cutover" in sql
|
||||||
|
assert "teleo_v3_enforce_apply_contract_cutover" in sql
|
||||||
|
assert sql.count("create or replace trigger teleo_v3_") == 24
|
||||||
|
assert re.search(r"(?m)^\s*create trigger teleo_v3_", sql) is None
|
||||||
|
assert "actual.tgenabled = 'o'" in sql
|
||||||
|
assert "actual.tgtype = expected.trigger_type" in sql
|
||||||
|
assert "actual.update_columns = expected.update_columns" in sql
|
||||||
|
assert "actual.when_expression is not distinct from expected.when_expression" in sql
|
||||||
|
assert "actual.argument_hex = expected.argument_hex" in sql
|
||||||
|
assert "actual.tgconstraint = 0::pg_catalog.oid" in sql
|
||||||
|
assert "actual.tgoldtable is null" in sql
|
||||||
|
assert "actual.tgnewtable is null" in sql
|
||||||
|
assert "actual.tgfoid" in sql
|
||||||
|
assert "actual.function_security_definer = expected.function_security_definer" in sql
|
||||||
|
assert "actual.function_source_sha256 = expected.function_source_sha256" in sql
|
||||||
|
assert "acl.grantee <> procedure.proowner" in sql
|
||||||
|
assert "protected role membership drift must be removed" in sql
|
||||||
|
assert "teleo_v3_contract_postflight_query_begin" in sql
|
||||||
|
assert "alter function %i.%i() owner to kb_gate_owner" in sql
|
||||||
|
assert "jsonb_typeof(new.payload->'apply_payload'->'contract_version') = 'number'" in sql
|
||||||
|
assert "migrate or reject it" in sql
|
||||||
|
assert "for share" in sql
|
||||||
|
assert "source_content_hash" in sql
|
||||||
|
assert "locator_json" in sql
|
||||||
|
assert "relation_type in ('supports', 'challenges', 'depends_on', 'contradicts')" in sql
|
||||||
|
assert "preexisting v3 schema objects lack installation ownership" in sql
|
||||||
|
assert "create table if not exists kb_stage.teleo_v3_installation_ownership" in sql
|
||||||
|
assert "installation_token = 'collision-free-install-v1'" in sql
|
||||||
|
assert "alter table kb_stage.teleo_v3_installation_ownership owner to kb_gate_owner" in sql
|
||||||
|
|
||||||
|
destructive_patterns = (
|
||||||
|
r"\bdrop\s+",
|
||||||
|
r"\btruncate\b",
|
||||||
|
r"\bdelete\s+from\b",
|
||||||
|
r"\bupdate\s+public\.",
|
||||||
|
r"\balter\s+type\b",
|
||||||
|
)
|
||||||
|
for pattern in destructive_patterns:
|
||||||
|
assert re.search(pattern, sql) is None, pattern
|
||||||
|
|
||||||
|
|
||||||
|
def test_rollback_refuses_silent_v3_data_loss_before_dropping_contract() -> None:
|
||||||
|
sql = ROLLBACK_SQL.read_text(encoding="utf-8").lower()
|
||||||
|
|
||||||
|
guard_position = sql.index("teleo_v3 rollback refused")
|
||||||
|
first_drop_position = sql.index("drop trigger")
|
||||||
|
assert guard_position < first_drop_position
|
||||||
|
assert "claim_evidence_assessments contains v3 data" in sql
|
||||||
|
assert "claims contains admitted or proposal-linked v3 data" in sql
|
||||||
|
assert "drop column if exists proposition" in sql
|
||||||
|
assert "drop function if exists kb_stage.teleo_v3_require_accepted_proposal" in sql
|
||||||
|
assert "drop function if exists kb_stage.teleo_v3_require_linked_insert" in sql
|
||||||
|
assert "drop function if exists kb_stage.teleo_v3_reject_delete" in sql
|
||||||
|
assert "drop function if exists kb_stage.teleo_v3_acquire_protected_relation_locks" in sql
|
||||||
|
assert "drop function if exists kb_stage.teleo_v3_enforce_apply_contract_cutover" in sql
|
||||||
|
assert "drop constraint if exists teleo_v3_proposal_apply_receipt_check" in sql
|
||||||
|
assert "installation ownership ledger is not exact" in sql
|
||||||
|
assert "v3-named objects lack exact installation ownership" in sql
|
||||||
|
assert "drop table if exists kb_stage.teleo_v3_installation_ownership" in sql
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("logs", "probe_returncode", "probe_stdout", "expected"),
|
||||||
|
(
|
||||||
|
(verifier.POSTGRES_READY_LOG_LINE, 0, "1\n", False),
|
||||||
|
(f"{verifier.POSTGRES_READY_LOG_LINE}\n{verifier.POSTGRES_READY_LOG_LINE}", 2, "", False),
|
||||||
|
(f"{verifier.POSTGRES_READY_LOG_LINE}\n{verifier.POSTGRES_READY_LOG_LINE}", 0, "", False),
|
||||||
|
(f"{verifier.POSTGRES_READY_LOG_LINE}\n{verifier.POSTGRES_READY_LOG_LINE}", 0, "1\n", True),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
def test_postgres_final_startup_requires_second_transition_and_target_database(
|
||||||
|
logs: str,
|
||||||
|
probe_returncode: int,
|
||||||
|
probe_stdout: str,
|
||||||
|
expected: bool,
|
||||||
|
) -> None:
|
||||||
|
probe = subprocess.CompletedProcess(["psql"], probe_returncode, stdout=probe_stdout, stderr="")
|
||||||
|
|
||||||
|
assert verifier.postgres_final_startup_ready(logs, probe) is expected
|
||||||
|
|
||||||
|
|
||||||
|
def _startup_test_postgres() -> verifier.DisposablePostgres:
|
||||||
|
postgres = object.__new__(verifier.DisposablePostgres)
|
||||||
|
postgres.docker = "docker"
|
||||||
|
postgres.name = "teleo-v3-contract-test"
|
||||||
|
postgres.password = SECRET_STARTUP_MARKER
|
||||||
|
postgres.started = True
|
||||||
|
postgres.cleanup_required = True
|
||||||
|
postgres.environment = {}
|
||||||
|
return postgres
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("as_bytes", (False, True))
|
||||||
|
def test_startup_detail_redacts_secret_before_tail_limit(as_bytes: bool) -> None:
|
||||||
|
postgres = _startup_test_postgres()
|
||||||
|
postgres.password = "K8VN3PQ7RM2TX6ZC9WB4HJ5DF1GS0LAE8YU3IO2N"
|
||||||
|
context = "context=database-unavailable|"
|
||||||
|
trailer = "|stage=target_database_probe|kind=timeout"
|
||||||
|
suffix_length = 500 - len(context) - len("<redacted>")
|
||||||
|
suffix = ("x" * (suffix_length - len(trailer))) + trailer
|
||||||
|
diagnostic = context + postgres.password + suffix
|
||||||
|
value: str | bytes = diagnostic.encode() if as_bytes else diagnostic
|
||||||
|
|
||||||
|
assert len(diagnostic) > 500
|
||||||
|
assert len(diagnostic) - 500 > len(context)
|
||||||
|
|
||||||
|
detail = postgres._redacted_startup_detail(value)
|
||||||
|
|
||||||
|
assert detail.startswith(f"{context}<redacted>")
|
||||||
|
assert detail.endswith(trailer)
|
||||||
|
assert postgres.password not in detail
|
||||||
|
assert all(postgres.password[offset : offset + 4] not in detail for offset in range(len(postgres.password) - 3))
|
||||||
|
|
||||||
|
|
||||||
|
def test_docker_run_timeout_is_secret_safe_and_reconciles_owned_side_effect(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
postgres = _startup_test_postgres()
|
||||||
|
postgres.password = "K8VN3PQ7RM2TX6ZC9WB4HJ5DF1GS0LAE8YU3IO2N"
|
||||||
|
postgres.started = False
|
||||||
|
postgres.cleanup_required = False
|
||||||
|
owned_container_id = "a" * 64
|
||||||
|
owned_container_exists = [True]
|
||||||
|
calls: list[tuple[list[str], dict[str, str] | None]] = []
|
||||||
|
context = "context=docker-daemon-accepted-request|"
|
||||||
|
trailer = "|stage=docker_run|kind=timeout"
|
||||||
|
suffix_length = 500 - len(context) - len("<redacted>")
|
||||||
|
timeout_stderr = context + postgres.password + ("x" * (suffix_length - len(trailer))) + trailer
|
||||||
|
|
||||||
|
def fake_run(
|
||||||
|
arguments: list[str],
|
||||||
|
*,
|
||||||
|
input_text: str | None = None,
|
||||||
|
check: bool = True,
|
||||||
|
timeout: float = 120,
|
||||||
|
environment: dict[str, str] | None = None,
|
||||||
|
) -> subprocess.CompletedProcess[str]:
|
||||||
|
del input_text, check
|
||||||
|
calls.append((list(arguments), environment))
|
||||||
|
if arguments[0] == "run":
|
||||||
|
assert postgres.cleanup_required is True
|
||||||
|
assert postgres.started is False
|
||||||
|
assert postgres.password not in " ".join(arguments)
|
||||||
|
assert environment is not None
|
||||||
|
assert environment["POSTGRES_PASSWORD"] == postgres.password
|
||||||
|
secret_bearing_command = ["docker", *arguments, f"adjacent={postgres.password}"]
|
||||||
|
raise subprocess.TimeoutExpired(
|
||||||
|
secret_bearing_command,
|
||||||
|
timeout,
|
||||||
|
stderr=timeout_stderr,
|
||||||
|
)
|
||||||
|
if arguments[:2] == ["container", "ls"] and arguments[-1] == "{{.ID}}":
|
||||||
|
stdout = f"{owned_container_id}\n" if owned_container_exists[0] else ""
|
||||||
|
return subprocess.CompletedProcess(arguments, 0, stdout=stdout, stderr="")
|
||||||
|
if arguments[:2] == ["rm", "--force"]:
|
||||||
|
assert arguments == ["rm", "--force", owned_container_id]
|
||||||
|
owned_container_exists[0] = False
|
||||||
|
return subprocess.CompletedProcess(arguments, 0, stdout=owned_container_id, stderr="")
|
||||||
|
if arguments[0] == "inspect":
|
||||||
|
return subprocess.CompletedProcess(
|
||||||
|
arguments,
|
||||||
|
0 if owned_container_exists[0] else 1,
|
||||||
|
stdout="[]" if owned_container_exists[0] else "",
|
||||||
|
stderr="" if owned_container_exists[0] else "not found",
|
||||||
|
)
|
||||||
|
if arguments[:2] == ["container", "ls"] and arguments[-1] == "{{.Names}}":
|
||||||
|
stdout = f"{postgres.name}\n" if owned_container_exists[0] else ""
|
||||||
|
return subprocess.CompletedProcess(arguments, 0, stdout=stdout, stderr="")
|
||||||
|
raise AssertionError(f"unexpected docker arguments: {arguments}")
|
||||||
|
|
||||||
|
monkeypatch.setattr(postgres, "run", fake_run)
|
||||||
|
monkeypatch.setattr(verifier.time, "sleep", lambda _seconds: None)
|
||||||
|
|
||||||
|
with pytest.raises(verifier.VerificationError) as exc_info:
|
||||||
|
postgres.start()
|
||||||
|
cleanup = postgres.cleanup()
|
||||||
|
|
||||||
|
error = str(exc_info.value)
|
||||||
|
assert "launch state is ambiguous; cleanup_required=true" in error
|
||||||
|
assert "stage=docker_run kind=timeout timeout_seconds=120" in error
|
||||||
|
assert error.endswith(trailer)
|
||||||
|
assert exc_info.value.__suppress_context__ is True
|
||||||
|
assert postgres.password not in error
|
||||||
|
assert all(postgres.password[offset : offset + 4] not in error for offset in range(len(postgres.password) - 3))
|
||||||
|
assert postgres.started is False
|
||||||
|
assert cleanup["cleanup_reconciliation_attempted"] is True
|
||||||
|
assert cleanup["remove_attempted"] is True
|
||||||
|
assert cleanup["remove_returncodes"] == [0]
|
||||||
|
assert cleanup["owned_container_ids_observed"] == [owned_container_id]
|
||||||
|
assert cleanup["container_absent"] is True
|
||||||
|
assert cleanup["name_readback_clear"] is True
|
||||||
|
assert cleanup["instance_label_readback_clear"] is True
|
||||||
|
assert cleanup["stable_absence_readback"] is True
|
||||||
|
ownership_call = next(arguments for arguments, _environment in calls if arguments[-1] == "{{.ID}}")
|
||||||
|
assert f"name=^/{postgres.name}$" in ownership_call
|
||||||
|
assert f"label={verifier.CONTAINER_LABEL}" in ownership_call
|
||||||
|
assert f"label=livingip.canary.instance={postgres.name}" in ownership_call
|
||||||
|
|
||||||
|
|
||||||
|
def test_docker_run_normal_launch_marks_cleanup_before_execution(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
postgres = _startup_test_postgres()
|
||||||
|
postgres.started = False
|
||||||
|
postgres.cleanup_required = False
|
||||||
|
container_id = "b" * 64
|
||||||
|
waited_for_startup: list[bool] = []
|
||||||
|
|
||||||
|
def fake_run(
|
||||||
|
arguments: list[str],
|
||||||
|
*,
|
||||||
|
input_text: str | None = None,
|
||||||
|
check: bool = True,
|
||||||
|
timeout: float = 120,
|
||||||
|
environment: dict[str, str] | None = None,
|
||||||
|
) -> subprocess.CompletedProcess[str]:
|
||||||
|
del input_text, check, timeout
|
||||||
|
if arguments[0] == "run":
|
||||||
|
assert postgres.cleanup_required is True
|
||||||
|
assert postgres.started is False
|
||||||
|
assert postgres.password not in " ".join(arguments)
|
||||||
|
assert environment is not None
|
||||||
|
assert environment["POSTGRES_PASSWORD"] == postgres.password
|
||||||
|
return subprocess.CompletedProcess(arguments, 0, stdout=f"{container_id}\n", stderr="")
|
||||||
|
assert arguments == ["inspect", postgres.name]
|
||||||
|
inspected = [
|
||||||
|
{
|
||||||
|
"Config": {
|
||||||
|
"Image": verifier.POSTGRES_IMAGE,
|
||||||
|
"Labels": {
|
||||||
|
"livingip.canary": "teleo-v3-epistemic-contract",
|
||||||
|
"livingip.canary.instance": postgres.name,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"HostConfig": {
|
||||||
|
"NetworkMode": "none",
|
||||||
|
"Tmpfs": {"/var/lib/postgresql/data": "rw,nosuid,nodev,size=384m"},
|
||||||
|
},
|
||||||
|
"Image": "sha256:postgres-image",
|
||||||
|
"Mounts": [],
|
||||||
|
}
|
||||||
|
]
|
||||||
|
return subprocess.CompletedProcess(arguments, 0, stdout=json.dumps(inspected), stderr="")
|
||||||
|
|
||||||
|
def fake_wait_for_final_startup() -> None:
|
||||||
|
assert postgres.cleanup_required is True
|
||||||
|
assert postgres.started is True
|
||||||
|
waited_for_startup.append(True)
|
||||||
|
|
||||||
|
monkeypatch.setattr(postgres, "run", fake_run)
|
||||||
|
monkeypatch.setattr(postgres, "_wait_for_final_startup", fake_wait_for_final_startup)
|
||||||
|
|
||||||
|
environment = postgres.start()
|
||||||
|
|
||||||
|
assert waited_for_startup == [True]
|
||||||
|
assert postgres.started is True
|
||||||
|
assert postgres.cleanup_required is True
|
||||||
|
assert environment["container_name"] == postgres.name
|
||||||
|
assert environment["network_mode"] == "none"
|
||||||
|
assert environment["tmpfs_data_dir"] is True
|
||||||
|
assert environment["docker_volume_mounts"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_cleanup_never_removes_foreign_exact_name_without_instance_labels(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
postgres = _startup_test_postgres()
|
||||||
|
postgres.started = False
|
||||||
|
postgres.cleanup_required = True
|
||||||
|
calls: list[list[str]] = []
|
||||||
|
|
||||||
|
def fake_run(
|
||||||
|
arguments: list[str],
|
||||||
|
*,
|
||||||
|
input_text: str | None = None,
|
||||||
|
check: bool = True,
|
||||||
|
timeout: float = 120,
|
||||||
|
environment: dict[str, str] | None = None,
|
||||||
|
) -> subprocess.CompletedProcess[str]:
|
||||||
|
del input_text, check, timeout, environment
|
||||||
|
calls.append(list(arguments))
|
||||||
|
if arguments[:2] == ["container", "ls"] and arguments[-1] == "{{.ID}}":
|
||||||
|
return subprocess.CompletedProcess(arguments, 0, stdout="", stderr="")
|
||||||
|
if arguments[0] == "inspect":
|
||||||
|
return subprocess.CompletedProcess(arguments, 0, stdout="[]", stderr="")
|
||||||
|
if arguments[:2] == ["container", "ls"] and arguments[-1] == "{{.Names}}":
|
||||||
|
has_instance_filter = f"label=livingip.canary.instance={postgres.name}" in arguments
|
||||||
|
stdout = "" if has_instance_filter else f"{postgres.name}\n"
|
||||||
|
return subprocess.CompletedProcess(arguments, 0, stdout=stdout, stderr="")
|
||||||
|
raise AssertionError(f"unexpected docker arguments: {arguments}")
|
||||||
|
|
||||||
|
monkeypatch.setattr(postgres, "run", fake_run)
|
||||||
|
monkeypatch.setattr(verifier.time, "sleep", lambda _seconds: None)
|
||||||
|
|
||||||
|
cleanup = postgres.cleanup()
|
||||||
|
|
||||||
|
assert not any(arguments[:2] == ["rm", "--force"] for arguments in calls)
|
||||||
|
assert cleanup["remove_attempted"] is False
|
||||||
|
assert cleanup["foreign_name_collision_detected"] is True
|
||||||
|
assert cleanup["container_absent"] is False
|
||||||
|
assert cleanup["name_readback_clear"] is False
|
||||||
|
assert cleanup["instance_label_readback_clear"] is True
|
||||||
|
assert cleanup["stable_absence_readback"] is False
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("hung_stage", ("docker_logs", "target_database_probe"))
|
||||||
|
def test_postgres_startup_commands_are_capped_to_remaining_deadline(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
hung_stage: str,
|
||||||
|
) -> None:
|
||||||
|
postgres = _startup_test_postgres()
|
||||||
|
now = [100.0]
|
||||||
|
observed_timeouts: list[float] = []
|
||||||
|
ready_logs = f"{verifier.POSTGRES_READY_LOG_LINE}\n{verifier.POSTGRES_READY_LOG_LINE}\n"
|
||||||
|
|
||||||
|
def fake_run(
|
||||||
|
arguments: list[str],
|
||||||
|
*,
|
||||||
|
input_text: str | None = None,
|
||||||
|
check: bool = True,
|
||||||
|
timeout: float = 120,
|
||||||
|
) -> subprocess.CompletedProcess[str]:
|
||||||
|
del input_text, check
|
||||||
|
assert arguments == ["logs", postgres.name]
|
||||||
|
if hung_stage == "docker_logs":
|
||||||
|
observed_timeouts.append(timeout)
|
||||||
|
now[0] += timeout
|
||||||
|
raise subprocess.TimeoutExpired(arguments, timeout, stderr=SECRET_STARTUP_MARKER)
|
||||||
|
return subprocess.CompletedProcess(arguments, 0, stdout=ready_logs, stderr="")
|
||||||
|
|
||||||
|
def fake_psql(
|
||||||
|
sql: str,
|
||||||
|
*,
|
||||||
|
check: bool = True,
|
||||||
|
timeout: float = 120,
|
||||||
|
) -> subprocess.CompletedProcess[str]:
|
||||||
|
del check
|
||||||
|
assert sql == "select 1;"
|
||||||
|
observed_timeouts.append(timeout)
|
||||||
|
now[0] += timeout
|
||||||
|
raise subprocess.TimeoutExpired(["psql"], timeout, stderr=SECRET_STARTUP_MARKER)
|
||||||
|
|
||||||
|
monkeypatch.setattr(verifier.time, "monotonic", lambda: now[0])
|
||||||
|
monkeypatch.setattr(postgres, "run", fake_run)
|
||||||
|
monkeypatch.setattr(postgres, "psql", fake_psql)
|
||||||
|
|
||||||
|
with pytest.raises(verifier.VerificationError) as exc_info:
|
||||||
|
postgres._wait_for_final_startup(startup_timeout=3.0, probe_timeout=120.0, poll_interval=0)
|
||||||
|
|
||||||
|
assert observed_timeouts == [3.0]
|
||||||
|
assert f"stage={hung_stage} kind=timeout timeout_seconds=3" in str(exc_info.value)
|
||||||
|
assert SECRET_STARTUP_MARKER not in str(exc_info.value)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("failed_stage", "expected_context"),
|
||||||
|
(
|
||||||
|
("docker_logs", "stage=docker_logs kind=nonzero returncode=17 detail=daemon <redacted>"),
|
||||||
|
(
|
||||||
|
"target_database_probe",
|
||||||
|
"stage=target_database_probe kind=nonzero returncode=2 detail=database missing <redacted>",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
def test_postgres_startup_failure_reports_sanitized_terminal_context(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
failed_stage: str,
|
||||||
|
expected_context: str,
|
||||||
|
) -> None:
|
||||||
|
postgres = _startup_test_postgres()
|
||||||
|
now = [200.0]
|
||||||
|
ready_logs = f"{verifier.POSTGRES_READY_LOG_LINE}\n{verifier.POSTGRES_READY_LOG_LINE}\n"
|
||||||
|
|
||||||
|
def fake_run(
|
||||||
|
arguments: list[str],
|
||||||
|
*,
|
||||||
|
input_text: str | None = None,
|
||||||
|
check: bool = True,
|
||||||
|
timeout: float = 120,
|
||||||
|
) -> subprocess.CompletedProcess[str]:
|
||||||
|
del input_text, check, timeout
|
||||||
|
assert arguments == ["logs", postgres.name]
|
||||||
|
if failed_stage == "docker_logs":
|
||||||
|
return subprocess.CompletedProcess(
|
||||||
|
arguments,
|
||||||
|
17,
|
||||||
|
stdout="",
|
||||||
|
stderr=f"daemon {SECRET_STARTUP_MARKER}",
|
||||||
|
)
|
||||||
|
return subprocess.CompletedProcess(arguments, 0, stdout=ready_logs, stderr="")
|
||||||
|
|
||||||
|
def fake_psql(
|
||||||
|
sql: str,
|
||||||
|
*,
|
||||||
|
check: bool = True,
|
||||||
|
timeout: float = 120,
|
||||||
|
) -> subprocess.CompletedProcess[str]:
|
||||||
|
del check, timeout
|
||||||
|
assert sql == "select 1;"
|
||||||
|
return subprocess.CompletedProcess(
|
||||||
|
["psql"],
|
||||||
|
2,
|
||||||
|
stdout="",
|
||||||
|
stderr=f"database missing {SECRET_STARTUP_MARKER}",
|
||||||
|
)
|
||||||
|
|
||||||
|
def fake_sleep(seconds: float) -> None:
|
||||||
|
now[0] += seconds
|
||||||
|
|
||||||
|
monkeypatch.setattr(verifier.time, "monotonic", lambda: now[0])
|
||||||
|
monkeypatch.setattr(verifier.time, "sleep", fake_sleep)
|
||||||
|
monkeypatch.setattr(postgres, "run", fake_run)
|
||||||
|
monkeypatch.setattr(postgres, "psql", fake_psql)
|
||||||
|
|
||||||
|
with pytest.raises(verifier.VerificationError) as exc_info:
|
||||||
|
postgres._wait_for_final_startup(startup_timeout=0.25, probe_timeout=10.0, poll_interval=0.25)
|
||||||
|
|
||||||
|
error = str(exc_info.value)
|
||||||
|
assert expected_context in error
|
||||||
|
assert "within 0.25s after 1 attempt(s)" in error
|
||||||
|
assert SECRET_STARTUP_MARKER not in error
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(not verifier.docker_available(), reason="Docker daemon is required")
|
||||||
|
def test_preexisting_v3_named_objects_are_never_adopted_or_rolled_back() -> None:
|
||||||
|
postgres = verifier.DisposablePostgres()
|
||||||
|
cleanup: dict[str, object] = {}
|
||||||
|
try:
|
||||||
|
postgres.start()
|
||||||
|
postgres.psql(verifier.BOOTSTRAP_SQL)
|
||||||
|
postgres.psql(
|
||||||
|
"""
|
||||||
|
alter table public.claims
|
||||||
|
add column owner_agent_id uuid,
|
||||||
|
add column proposition text,
|
||||||
|
add column body_markdown text,
|
||||||
|
add column scope text,
|
||||||
|
add column access_scope text,
|
||||||
|
add column admission_state text,
|
||||||
|
add column revision_criteria text,
|
||||||
|
add column revision_kind text,
|
||||||
|
add column supersedes_id uuid,
|
||||||
|
add column accepted_by_proposal_id uuid;
|
||||||
|
update public.claims
|
||||||
|
set owner_agent_id = '11111111-1111-4111-8111-111111111111'::uuid,
|
||||||
|
proposition = 'claims.proposition',
|
||||||
|
body_markdown = 'claims.body_markdown',
|
||||||
|
scope = 'claims.scope',
|
||||||
|
access_scope = 'claims.access_scope',
|
||||||
|
admission_state = 'claims.admission_state',
|
||||||
|
revision_criteria = 'claims.revision_criteria',
|
||||||
|
revision_kind = 'claims.revision_kind',
|
||||||
|
supersedes_id = '20000000-0000-4000-8000-000000000002'::uuid,
|
||||||
|
accepted_by_proposal_id = '30000000-0000-4000-8000-000000000003'::uuid
|
||||||
|
where id = '20000000-0000-4000-8000-000000000001'::uuid;
|
||||||
|
|
||||||
|
alter table public.sources
|
||||||
|
add column canonical_url text,
|
||||||
|
add column storage_uri text,
|
||||||
|
add column content_hash text,
|
||||||
|
add column access_scope text,
|
||||||
|
add column ingestion_origin text,
|
||||||
|
add column provenance_status text,
|
||||||
|
add column source_class text,
|
||||||
|
add column auto_promotion_policy text,
|
||||||
|
add column accepted_by_proposal_id uuid;
|
||||||
|
update public.sources
|
||||||
|
set canonical_url = 'sources.canonical_url',
|
||||||
|
storage_uri = 'sources.storage_uri',
|
||||||
|
content_hash = 'sources.content_hash',
|
||||||
|
access_scope = 'sources.access_scope',
|
||||||
|
ingestion_origin = 'sources.ingestion_origin',
|
||||||
|
provenance_status = 'sources.provenance_status',
|
||||||
|
source_class = 'sources.source_class',
|
||||||
|
auto_promotion_policy = 'sources.auto_promotion_policy',
|
||||||
|
accepted_by_proposal_id = '30000000-0000-4000-8000-000000000003'::uuid
|
||||||
|
where id = '20000000-0000-4000-8000-000000000003'::uuid;
|
||||||
|
|
||||||
|
alter table public.claim_evidence
|
||||||
|
add column excerpt text,
|
||||||
|
add column locator_json jsonb,
|
||||||
|
add column source_content_hash text,
|
||||||
|
add column rationale text,
|
||||||
|
add column polarity text,
|
||||||
|
add column accepted_from_proposal_id uuid;
|
||||||
|
update public.claim_evidence
|
||||||
|
set excerpt = 'claim_evidence.excerpt',
|
||||||
|
locator_json = '{"sentinel":"claim_evidence.locator_json"}'::jsonb,
|
||||||
|
source_content_hash = 'claim_evidence.source_content_hash',
|
||||||
|
rationale = 'claim_evidence.rationale',
|
||||||
|
polarity = 'claim_evidence.polarity',
|
||||||
|
accepted_from_proposal_id = '30000000-0000-4000-8000-000000000003'::uuid
|
||||||
|
where id = '20000000-0000-4000-8000-000000000004'::uuid;
|
||||||
|
|
||||||
|
alter table public.claim_edges
|
||||||
|
add column relation_type text,
|
||||||
|
add column rationale text,
|
||||||
|
add column scope_or_conditions text,
|
||||||
|
add column accepted_by_proposal_id uuid;
|
||||||
|
update public.claim_edges
|
||||||
|
set relation_type = 'claim_edges.relation_type',
|
||||||
|
rationale = 'claim_edges.rationale',
|
||||||
|
scope_or_conditions = 'claim_edges.scope_or_conditions',
|
||||||
|
accepted_by_proposal_id = '30000000-0000-4000-8000-000000000003'::uuid
|
||||||
|
where id = '20000000-0000-4000-8000-000000000005'::uuid;
|
||||||
|
|
||||||
|
create table public.claim_evidence_assessments (
|
||||||
|
id uuid primary key,
|
||||||
|
legacy_marker text
|
||||||
|
);
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
state_sql = """
|
||||||
|
select jsonb_build_object(
|
||||||
|
'claims', (
|
||||||
|
select to_jsonb(row_data) from (
|
||||||
|
select owner_agent_id, proposition, body_markdown, scope, access_scope, admission_state,
|
||||||
|
revision_criteria, revision_kind, supersedes_id, accepted_by_proposal_id
|
||||||
|
from public.claims
|
||||||
|
where id = '20000000-0000-4000-8000-000000000001'::uuid
|
||||||
|
) row_data
|
||||||
|
),
|
||||||
|
'sources', (
|
||||||
|
select to_jsonb(row_data) from (
|
||||||
|
select canonical_url, storage_uri, content_hash, access_scope, ingestion_origin,
|
||||||
|
provenance_status, source_class, auto_promotion_policy, accepted_by_proposal_id
|
||||||
|
from public.sources
|
||||||
|
where id = '20000000-0000-4000-8000-000000000003'::uuid
|
||||||
|
) row_data
|
||||||
|
),
|
||||||
|
'claim_evidence', (
|
||||||
|
select to_jsonb(row_data) from (
|
||||||
|
select excerpt, locator_json, source_content_hash, rationale, polarity, accepted_from_proposal_id
|
||||||
|
from public.claim_evidence
|
||||||
|
where id = '20000000-0000-4000-8000-000000000004'::uuid
|
||||||
|
) row_data
|
||||||
|
),
|
||||||
|
'claim_edges', (
|
||||||
|
select to_jsonb(row_data) from (
|
||||||
|
select relation_type, rationale, scope_or_conditions, accepted_by_proposal_id
|
||||||
|
from public.claim_edges
|
||||||
|
where id = '20000000-0000-4000-8000-000000000005'::uuid
|
||||||
|
) row_data
|
||||||
|
),
|
||||||
|
'assessment_columns', (
|
||||||
|
select jsonb_agg(column_name order by ordinal_position)
|
||||||
|
from information_schema.columns
|
||||||
|
where table_schema = 'public' and table_name = 'claim_evidence_assessments'
|
||||||
|
),
|
||||||
|
'assessment_rows', (select count(*) from public.claim_evidence_assessments),
|
||||||
|
'ownership_ledger', to_regclass('kb_stage.teleo_v3_installation_ownership')::text
|
||||||
|
)::text;
|
||||||
|
"""
|
||||||
|
before = postgres.psql_json(state_sql)
|
||||||
|
|
||||||
|
install = postgres.apply_file(MIGRATION_SQL, check=False)
|
||||||
|
rollback = postgres.apply_file(ROLLBACK_SQL, check=False)
|
||||||
|
after = postgres.psql_json(state_sql)
|
||||||
|
|
||||||
|
assert install.returncode != 0
|
||||||
|
assert "preexisting V3 schema objects lack installation ownership" in install.stderr
|
||||||
|
assert rollback.returncode != 0
|
||||||
|
assert "V3-named objects lack exact installation ownership" in rollback.stderr
|
||||||
|
assert before == after
|
||||||
|
assert set(before["claims"]) == {
|
||||||
|
"owner_agent_id",
|
||||||
|
"proposition",
|
||||||
|
"body_markdown",
|
||||||
|
"scope",
|
||||||
|
"access_scope",
|
||||||
|
"admission_state",
|
||||||
|
"revision_criteria",
|
||||||
|
"revision_kind",
|
||||||
|
"supersedes_id",
|
||||||
|
"accepted_by_proposal_id",
|
||||||
|
}
|
||||||
|
assert set(before["sources"]) == {
|
||||||
|
"canonical_url",
|
||||||
|
"storage_uri",
|
||||||
|
"content_hash",
|
||||||
|
"access_scope",
|
||||||
|
"ingestion_origin",
|
||||||
|
"provenance_status",
|
||||||
|
"source_class",
|
||||||
|
"auto_promotion_policy",
|
||||||
|
"accepted_by_proposal_id",
|
||||||
|
}
|
||||||
|
assert set(before["claim_evidence"]) == {
|
||||||
|
"excerpt",
|
||||||
|
"locator_json",
|
||||||
|
"source_content_hash",
|
||||||
|
"rationale",
|
||||||
|
"polarity",
|
||||||
|
"accepted_from_proposal_id",
|
||||||
|
}
|
||||||
|
assert set(before["claim_edges"]) == {
|
||||||
|
"relation_type",
|
||||||
|
"rationale",
|
||||||
|
"scope_or_conditions",
|
||||||
|
"accepted_by_proposal_id",
|
||||||
|
}
|
||||||
|
assert all(value is not None for value in before["claims"].values())
|
||||||
|
assert all(value is not None for value in before["sources"].values())
|
||||||
|
assert all(value is not None for value in before["claim_evidence"].values())
|
||||||
|
assert all(value is not None for value in before["claim_edges"].values())
|
||||||
|
assert before["assessment_columns"] == ["id", "legacy_marker"]
|
||||||
|
assert before["assessment_rows"] == 0
|
||||||
|
assert before["ownership_ledger"] is None
|
||||||
|
finally:
|
||||||
|
cleanup = postgres.cleanup()
|
||||||
|
|
||||||
|
assert cleanup["container_absent"] is True
|
||||||
|
assert cleanup["name_readback_clear"] is True
|
||||||
|
assert cleanup["instance_label_readback_clear"] is True
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(not verifier.docker_available(), reason="Docker daemon is required")
|
||||||
|
def test_partial_v3_contract_is_never_adopted_or_rolled_back() -> None:
|
||||||
|
postgres = verifier.DisposablePostgres()
|
||||||
|
cleanup: dict[str, object] = {}
|
||||||
|
try:
|
||||||
|
postgres.start()
|
||||||
|
postgres.psql(verifier.BOOTSTRAP_SQL)
|
||||||
|
postgres.psql(
|
||||||
|
"""
|
||||||
|
create function kb_stage.teleo_v3_partial_sentinel()
|
||||||
|
returns text
|
||||||
|
language sql
|
||||||
|
immutable
|
||||||
|
as $$ select 'legacy-partial-sentinel'::text $$;
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
state_sql = """
|
||||||
|
select jsonb_build_object(
|
||||||
|
'sentinel_result', kb_stage.teleo_v3_partial_sentinel(),
|
||||||
|
'sentinel_definition', pg_get_functiondef('kb_stage.teleo_v3_partial_sentinel()'::regprocedure),
|
||||||
|
'ownership_ledger', to_regclass('kb_stage.teleo_v3_installation_ownership')::text
|
||||||
|
)::text;
|
||||||
|
"""
|
||||||
|
before = postgres.psql_json(state_sql)
|
||||||
|
|
||||||
|
install = postgres.apply_file(MIGRATION_SQL, check=False)
|
||||||
|
rollback = postgres.apply_file(ROLLBACK_SQL, check=False)
|
||||||
|
after = postgres.psql_json(state_sql)
|
||||||
|
|
||||||
|
assert install.returncode != 0
|
||||||
|
assert "partial or altered V3 contract lacks exact installation ownership" in install.stderr
|
||||||
|
assert rollback.returncode != 0
|
||||||
|
assert "V3-named objects lack exact installation ownership" in rollback.stderr
|
||||||
|
assert before == after
|
||||||
|
assert before["sentinel_result"] == "legacy-partial-sentinel"
|
||||||
|
assert "teleo_v3_partial_sentinel" in before["sentinel_definition"]
|
||||||
|
assert before["ownership_ledger"] is None
|
||||||
|
finally:
|
||||||
|
cleanup = postgres.cleanup()
|
||||||
|
|
||||||
|
assert cleanup["container_absent"] is True
|
||||||
|
assert cleanup["name_readback_clear"] is True
|
||||||
|
assert cleanup["instance_label_readback_clear"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_verifier_has_no_external_database_target_surface(tmp_path: Path) -> None:
|
||||||
|
args = verifier.parse_args(["--output", str(tmp_path / "receipt.json")])
|
||||||
|
assert args.output == tmp_path / "receipt.json"
|
||||||
|
|
||||||
|
source = Path(verifier.__file__).read_text(encoding="utf-8")
|
||||||
|
assert "DATABASE_URL" not in source
|
||||||
|
assert "PGHOST" not in source
|
||||||
|
assert '"--network",\n "none"' in source
|
||||||
|
assert '"--tmpfs"' in source
|
||||||
|
assert 'production_access_attempted": False' in source
|
||||||
|
|
||||||
|
|
||||||
|
def test_failure_receipt_requires_exact_sqlstate() -> None:
|
||||||
|
completed = subprocess.CompletedProcess(
|
||||||
|
args=["psql"],
|
||||||
|
returncode=3,
|
||||||
|
stdout="",
|
||||||
|
stderr="ERROR: 23514: expected constraint failure\n",
|
||||||
|
)
|
||||||
|
receipt = verifier.failure_receipt(completed, "23514")
|
||||||
|
|
||||||
|
assert receipt["refused"] is True
|
||||||
|
assert receipt["observed_sqlstate"] == "23514"
|
||||||
|
assert receipt["expected_sqlstate_observed"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_verifier_exercises_concurrent_proposal_downgrade_lock() -> None:
|
||||||
|
source = Path(verifier.__file__).read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
assert "proposal_decision_lock_race_check" in source
|
||||||
|
assert "teleo_v3_proposal_lock_holder" in source
|
||||||
|
assert "set statement_timeout = '750ms'" in source
|
||||||
|
assert 'failure_receipt(downgrade, "57014")' in source
|
||||||
|
|
||||||
|
|
||||||
|
def test_verifier_asserts_security_definer_owner_search_path_and_acl() -> None:
|
||||||
|
source = Path(verifier.__file__).read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
assert '"owner_name": "kb_gate_owner"' in source
|
||||||
|
assert '"security_definer": True' in source
|
||||||
|
assert '"config": ["search_path=pg_catalog, pg_temp"]' in source
|
||||||
|
assert '"owner_only_execute": True' in source
|
||||||
|
|
||||||
|
|
||||||
|
def test_verifier_cleanup_bypass_is_bounded_to_disposable_superuser_transaction() -> None:
|
||||||
|
source = Path(verifier.__file__).read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
assert "set local session_replication_role = replica" in source
|
||||||
|
assert '"isolated-disposable-superuser-transaction"' in source
|
||||||
|
assert "Production writers must never receive" in source
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(not verifier.docker_available(), reason="Docker daemon is required")
|
||||||
|
def test_disposable_postgres_contract_end_to_end(tmp_path: Path) -> None:
|
||||||
|
output = tmp_path / "teleo-v3-epistemic-contract-receipt.json"
|
||||||
|
|
||||||
|
receipt = verifier.run_verification(output)
|
||||||
|
|
||||||
|
assert receipt["status"] == "pass", receipt.get("error")
|
||||||
|
assert receipt["required_tier"] == "T2_runtime"
|
||||||
|
assert receipt["current_tier"] == "T2_runtime_disposable_postgresql"
|
||||||
|
assert receipt["production_access_attempted"] is False
|
||||||
|
assert receipt["production_migration_shipped"] is False
|
||||||
|
assert receipt["environment"]["network_mode"] == "none"
|
||||||
|
assert receipt["environment"]["tmpfs_data_dir"] is True
|
||||||
|
assert receipt["environment"]["docker_volume_mounts"] == []
|
||||||
|
assert len(receipt["checks"]) >= 20
|
||||||
|
assert all(receipt["checks"].values()), receipt["checks"]
|
||||||
|
assert all(item["refused"] for item in receipt["adversarial"].values())
|
||||||
|
assert all(item["expected_sqlstate_observed"] for item in receipt["adversarial"].values())
|
||||||
|
assert receipt["rollback_with_data"]["refused"] is True
|
||||||
|
assert receipt["cleanup"]["container_absent"] is True
|
||||||
|
assert receipt["cleanup"]["instance_label_readback_clear"] is True
|
||||||
|
assert json.loads(output.read_text(encoding="utf-8")) == receipt
|
||||||
467
tests/test_v3_apply_catalog_serialization_postgres.py
Normal file
467
tests/test_v3_apply_catalog_serialization_postgres.py
Normal file
|
|
@ -0,0 +1,467 @@
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import subprocess
|
||||||
|
import time
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from ops import verify_teleo_v3_epistemic_contract as contract
|
||||||
|
from tests import test_v3_apply_worker_postgres as worker
|
||||||
|
|
||||||
|
CATALOG_DDL_ADVISORY_LOCK_KEY = 6072343533146485505
|
||||||
|
TEST_APPLY_BARRIER_LOCK_KEY = 6072343533146485599
|
||||||
|
|
||||||
|
|
||||||
|
def _start_psql(
|
||||||
|
postgres: contract.DisposablePostgres,
|
||||||
|
sql: str,
|
||||||
|
*,
|
||||||
|
role: str = "postgres",
|
||||||
|
password: str | None = None,
|
||||||
|
keep_stdin_open: bool = False,
|
||||||
|
) -> subprocess.Popen[str]:
|
||||||
|
assert postgres.docker is not None
|
||||||
|
command = [postgres.docker, "exec"]
|
||||||
|
if password is not None:
|
||||||
|
command.extend(["-e", f"PGPASSWORD={password}"])
|
||||||
|
command.extend(
|
||||||
|
[
|
||||||
|
"-i",
|
||||||
|
postgres.name,
|
||||||
|
"psql",
|
||||||
|
"--no-psqlrc",
|
||||||
|
"--set=ON_ERROR_STOP=1",
|
||||||
|
"--set=VERBOSITY=verbose",
|
||||||
|
"--username",
|
||||||
|
role,
|
||||||
|
f"--dbname={contract.DATABASE}",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
if password is not None:
|
||||||
|
command.extend(["--host=127.0.0.1"])
|
||||||
|
process = subprocess.Popen(
|
||||||
|
command,
|
||||||
|
stdin=subprocess.PIPE,
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
assert process.stdin is not None
|
||||||
|
process.stdin.write(sql)
|
||||||
|
if keep_stdin_open:
|
||||||
|
process.stdin.flush()
|
||||||
|
else:
|
||||||
|
process.stdin.close()
|
||||||
|
process.stdin = None
|
||||||
|
return process
|
||||||
|
|
||||||
|
|
||||||
|
def _finish(process: subprocess.Popen[str], *, timeout: float = 20) -> subprocess.CompletedProcess[str]:
|
||||||
|
stdout, stderr = process.communicate(timeout=timeout)
|
||||||
|
return subprocess.CompletedProcess(process.args, process.returncode, stdout, stderr)
|
||||||
|
|
||||||
|
|
||||||
|
def _stop(process: subprocess.Popen[str] | None) -> None:
|
||||||
|
if process is None or process.poll() is not None:
|
||||||
|
return
|
||||||
|
process.terminate()
|
||||||
|
try:
|
||||||
|
process.communicate(timeout=3)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
process.kill()
|
||||||
|
process.communicate(timeout=3)
|
||||||
|
|
||||||
|
|
||||||
|
def _wait_for_activity(
|
||||||
|
postgres: contract.DisposablePostgres,
|
||||||
|
application_name: str,
|
||||||
|
*,
|
||||||
|
wait_event_type: str,
|
||||||
|
wait_event: str,
|
||||||
|
timeout: float = 8,
|
||||||
|
) -> None:
|
||||||
|
deadline = time.monotonic() + timeout
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
waiting = postgres.psql(
|
||||||
|
f"""
|
||||||
|
select exists (
|
||||||
|
select 1
|
||||||
|
from pg_catalog.pg_stat_activity
|
||||||
|
where application_name = '{application_name}'
|
||||||
|
and wait_event_type = '{wait_event_type}'
|
||||||
|
and wait_event = '{wait_event}'
|
||||||
|
);
|
||||||
|
"""
|
||||||
|
).stdout.strip()
|
||||||
|
if waiting == "t":
|
||||||
|
return
|
||||||
|
time.sleep(0.05)
|
||||||
|
raise AssertionError(f"{application_name} did not wait on {wait_event_type}/{wait_event}")
|
||||||
|
|
||||||
|
|
||||||
|
def _approved_apply_sql(
|
||||||
|
postgres: contract.DisposablePostgres,
|
||||||
|
ids: dict[str, str],
|
||||||
|
) -> str:
|
||||||
|
payload = worker._payload(ids)
|
||||||
|
packet = worker._insert_pending_proposal(postgres, payload, ids)
|
||||||
|
worker._approve_proposal(postgres, ids["proposal"], packet)
|
||||||
|
proposal = postgres.psql_json(
|
||||||
|
f"""
|
||||||
|
select pg_catalog.jsonb_build_object(
|
||||||
|
'id', id::text,
|
||||||
|
'proposal_type', proposal_type,
|
||||||
|
'status', status,
|
||||||
|
'payload', payload,
|
||||||
|
'reviewed_by_handle', reviewed_by_handle,
|
||||||
|
'reviewed_by_agent_id', reviewed_by_agent_id::text,
|
||||||
|
'reviewed_at', pg_catalog.to_char(
|
||||||
|
reviewed_at at time zone 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"'
|
||||||
|
),
|
||||||
|
'review_note', review_note,
|
||||||
|
'applied_by_handle', applied_by_handle,
|
||||||
|
'applied_by_agent_id', applied_by_agent_id::text,
|
||||||
|
'applied_at', applied_at
|
||||||
|
)::text
|
||||||
|
from kb_stage.kb_proposals
|
||||||
|
where id = '{ids["proposal"]}'::uuid;
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
return worker.apply.build_apply_sql(proposal, worker.apply.SERVICE_AGENT_HANDLE)
|
||||||
|
|
||||||
|
|
||||||
|
def _instrument_apply_after_catalog_guard(sql: str, application_name: str) -> str:
|
||||||
|
guard_end = "$v3_contract_guard$;"
|
||||||
|
assert sql.count(guard_end) == 1
|
||||||
|
return sql.replace(
|
||||||
|
"begin;\n",
|
||||||
|
f"begin;\nset local application_name = '{application_name}';\n",
|
||||||
|
1,
|
||||||
|
).replace(
|
||||||
|
guard_end,
|
||||||
|
guard_end + f"\nselect pg_catalog.pg_advisory_xact_lock({TEST_APPLY_BARRIER_LOCK_KEY});",
|
||||||
|
1,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _instrument_apply_before_transaction(sql: str, application_name: str) -> str:
|
||||||
|
return f"set application_name = '{application_name}';\n{sql}"
|
||||||
|
|
||||||
|
|
||||||
|
def _start_barrier_holder(postgres: contract.DisposablePostgres) -> subprocess.Popen[str]:
|
||||||
|
holder = _start_psql(
|
||||||
|
postgres,
|
||||||
|
f"""
|
||||||
|
set application_name = 'teleo_v3_catalog_barrier_holder';
|
||||||
|
select pg_catalog.pg_advisory_lock({TEST_APPLY_BARRIER_LOCK_KEY});
|
||||||
|
""",
|
||||||
|
keep_stdin_open=True,
|
||||||
|
)
|
||||||
|
_wait_for_activity(
|
||||||
|
postgres,
|
||||||
|
"teleo_v3_catalog_barrier_holder",
|
||||||
|
wait_event_type="Client",
|
||||||
|
wait_event="ClientRead",
|
||||||
|
)
|
||||||
|
return holder
|
||||||
|
|
||||||
|
|
||||||
|
def _release_barrier(
|
||||||
|
postgres: contract.DisposablePostgres,
|
||||||
|
holder: subprocess.Popen[str],
|
||||||
|
) -> None:
|
||||||
|
released = postgres.psql(
|
||||||
|
"""
|
||||||
|
select pg_catalog.pg_terminate_backend(pid)
|
||||||
|
from pg_catalog.pg_stat_activity
|
||||||
|
where application_name = 'teleo_v3_catalog_barrier_holder';
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
assert released.stdout.strip() == "t"
|
||||||
|
if holder.stdin is not None:
|
||||||
|
holder.stdin.close()
|
||||||
|
holder.stdin = None
|
||||||
|
_finish(holder, timeout=5)
|
||||||
|
|
||||||
|
|
||||||
|
def _install_and_build_apply(
|
||||||
|
postgres: contract.DisposablePostgres,
|
||||||
|
ids: dict[str, str],
|
||||||
|
) -> str:
|
||||||
|
environment = postgres.start()
|
||||||
|
assert environment["network_mode"] == "none"
|
||||||
|
postgres.psql(contract.BOOTSTRAP_SQL)
|
||||||
|
postgres.psql(worker._supplemental_bootstrap())
|
||||||
|
postgres.apply_file(worker.PREREQUISITES)
|
||||||
|
worker._install_v3_contract(postgres)
|
||||||
|
return _approved_apply_sql(postgres, ids)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(not contract.docker_available(), reason="Docker daemon is required")
|
||||||
|
def test_trigger_ddl_waits_for_apply_transaction_relation_locks() -> None:
|
||||||
|
postgres = contract.DisposablePostgres()
|
||||||
|
holder: subprocess.Popen[str] | None = None
|
||||||
|
apply_process: subprocess.Popen[str] | None = None
|
||||||
|
ddl_process: subprocess.Popen[str] | None = None
|
||||||
|
try:
|
||||||
|
apply_sql = _install_and_build_apply(postgres, worker.RACE_IDS)
|
||||||
|
holder = _start_barrier_holder(postgres)
|
||||||
|
apply_process = _start_psql(
|
||||||
|
postgres,
|
||||||
|
_instrument_apply_after_catalog_guard(apply_sql, "teleo_v3_guarded_apply"),
|
||||||
|
role="kb_apply",
|
||||||
|
password=worker.APPLY_PASSWORD,
|
||||||
|
)
|
||||||
|
_wait_for_activity(
|
||||||
|
postgres,
|
||||||
|
"teleo_v3_guarded_apply",
|
||||||
|
wait_event_type="Lock",
|
||||||
|
wait_event="advisory",
|
||||||
|
)
|
||||||
|
|
||||||
|
ddl_process = _start_psql(
|
||||||
|
postgres,
|
||||||
|
"""
|
||||||
|
set application_name = 'teleo_v3_uncoordinated_trigger_ddl';
|
||||||
|
create or replace trigger teleo_v3_00_enforce_apply_contract_cutover
|
||||||
|
before insert or update of proposal_type, payload, status on kb_stage.kb_proposals
|
||||||
|
for each row when (false)
|
||||||
|
execute function kb_stage.teleo_v3_enforce_apply_contract_cutover();
|
||||||
|
""",
|
||||||
|
)
|
||||||
|
_wait_for_activity(
|
||||||
|
postgres,
|
||||||
|
"teleo_v3_uncoordinated_trigger_ddl",
|
||||||
|
wait_event_type="Lock",
|
||||||
|
wait_event="relation",
|
||||||
|
)
|
||||||
|
assert ddl_process.poll() is None
|
||||||
|
assert worker._contract_state(postgres) == "V3"
|
||||||
|
|
||||||
|
_release_barrier(postgres, holder)
|
||||||
|
holder = None
|
||||||
|
apply_result = _finish(apply_process)
|
||||||
|
apply_process = None
|
||||||
|
assert apply_result.returncode == 0, apply_result.stderr
|
||||||
|
ddl_result = _finish(ddl_process)
|
||||||
|
ddl_process = None
|
||||||
|
assert ddl_result.returncode == 0, ddl_result.stderr
|
||||||
|
assert postgres.psql_json(
|
||||||
|
f"""
|
||||||
|
select pg_catalog.jsonb_build_object(
|
||||||
|
'proposal_status', (
|
||||||
|
select status from kb_stage.kb_proposals
|
||||||
|
where id = '{worker.RACE_IDS["proposal"]}'::uuid
|
||||||
|
),
|
||||||
|
'contract_state', ({worker.apply.build_v3_contract_state_query()})
|
||||||
|
)::text;
|
||||||
|
"""
|
||||||
|
) == {"proposal_status": "applied", "contract_state": "INVALID"}
|
||||||
|
finally:
|
||||||
|
if holder is not None and holder.poll() is None:
|
||||||
|
_release_barrier(postgres, holder)
|
||||||
|
_stop(ddl_process)
|
||||||
|
_stop(apply_process)
|
||||||
|
_stop(holder)
|
||||||
|
cleanup = postgres.cleanup()
|
||||||
|
assert cleanup["container_absent"] is True
|
||||||
|
assert cleanup["instance_label_readback_clear"] is True
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(not contract.docker_available(), reason="Docker daemon is required")
|
||||||
|
def test_controlled_function_migration_waits_for_apply_shared_advisory_lock() -> None:
|
||||||
|
postgres = contract.DisposablePostgres()
|
||||||
|
holder: subprocess.Popen[str] | None = None
|
||||||
|
apply_process: subprocess.Popen[str] | None = None
|
||||||
|
migration_process: subprocess.Popen[str] | None = None
|
||||||
|
try:
|
||||||
|
apply_sql = _install_and_build_apply(postgres, worker.RACE_IDS)
|
||||||
|
holder = _start_barrier_holder(postgres)
|
||||||
|
apply_process = _start_psql(
|
||||||
|
postgres,
|
||||||
|
_instrument_apply_after_catalog_guard(apply_sql, "teleo_v3_shared_lock_apply"),
|
||||||
|
role="kb_apply",
|
||||||
|
password=worker.APPLY_PASSWORD,
|
||||||
|
)
|
||||||
|
_wait_for_activity(
|
||||||
|
postgres,
|
||||||
|
"teleo_v3_shared_lock_apply",
|
||||||
|
wait_event_type="Lock",
|
||||||
|
wait_event="advisory",
|
||||||
|
)
|
||||||
|
|
||||||
|
migration_process = _start_psql(
|
||||||
|
postgres,
|
||||||
|
f"""
|
||||||
|
begin;
|
||||||
|
set local application_name = 'teleo_v3_controlled_function_migration';
|
||||||
|
select pg_catalog.pg_advisory_xact_lock({CATALOG_DDL_ADVISORY_LOCK_KEY});
|
||||||
|
create or replace function kb_stage.teleo_v3_validate_claim_edge()
|
||||||
|
returns trigger
|
||||||
|
language plpgsql
|
||||||
|
set search_path = pg_catalog, pg_temp
|
||||||
|
as $function$
|
||||||
|
begin
|
||||||
|
return new;
|
||||||
|
end
|
||||||
|
$function$;
|
||||||
|
commit;
|
||||||
|
""",
|
||||||
|
)
|
||||||
|
_wait_for_activity(
|
||||||
|
postgres,
|
||||||
|
"teleo_v3_controlled_function_migration",
|
||||||
|
wait_event_type="Lock",
|
||||||
|
wait_event="advisory",
|
||||||
|
)
|
||||||
|
assert migration_process.poll() is None
|
||||||
|
assert worker._contract_state(postgres) == "V3"
|
||||||
|
|
||||||
|
_release_barrier(postgres, holder)
|
||||||
|
holder = None
|
||||||
|
apply_result = _finish(apply_process)
|
||||||
|
apply_process = None
|
||||||
|
assert apply_result.returncode == 0, apply_result.stderr
|
||||||
|
migration_result = _finish(migration_process)
|
||||||
|
migration_process = None
|
||||||
|
assert migration_result.returncode == 0, migration_result.stderr
|
||||||
|
assert worker._contract_state(postgres) == "INVALID"
|
||||||
|
finally:
|
||||||
|
if holder is not None and holder.poll() is None:
|
||||||
|
_release_barrier(postgres, holder)
|
||||||
|
_stop(migration_process)
|
||||||
|
_stop(apply_process)
|
||||||
|
_stop(holder)
|
||||||
|
cleanup = postgres.cleanup()
|
||||||
|
assert cleanup["container_absent"] is True
|
||||||
|
assert cleanup["instance_label_readback_clear"] is True
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(not contract.docker_available(), reason="Docker daemon is required")
|
||||||
|
def test_migration_first_apply_resnapshots_and_refuses_post_migration_catalog_drift() -> None:
|
||||||
|
postgres = contract.DisposablePostgres()
|
||||||
|
holder: subprocess.Popen[str] | None = None
|
||||||
|
apply_process: subprocess.Popen[str] | None = None
|
||||||
|
migration_process: subprocess.Popen[str] | None = None
|
||||||
|
try:
|
||||||
|
apply_sql = _install_and_build_apply(postgres, worker.RACE_IDS)
|
||||||
|
holder = _start_barrier_holder(postgres)
|
||||||
|
migration_process = _start_psql(
|
||||||
|
postgres,
|
||||||
|
f"""
|
||||||
|
begin;
|
||||||
|
set local application_name = 'teleo_v3_migration_first_holder';
|
||||||
|
select pg_catalog.pg_advisory_xact_lock({CATALOG_DDL_ADVISORY_LOCK_KEY});
|
||||||
|
select pg_catalog.pg_advisory_xact_lock({TEST_APPLY_BARRIER_LOCK_KEY});
|
||||||
|
create or replace function kb_stage.teleo_v3_validate_claim_edge()
|
||||||
|
returns trigger
|
||||||
|
language plpgsql
|
||||||
|
set search_path = pg_catalog, pg_temp
|
||||||
|
as $function$
|
||||||
|
begin
|
||||||
|
return new;
|
||||||
|
end
|
||||||
|
$function$;
|
||||||
|
commit;
|
||||||
|
""",
|
||||||
|
)
|
||||||
|
_wait_for_activity(
|
||||||
|
postgres,
|
||||||
|
"teleo_v3_migration_first_holder",
|
||||||
|
wait_event_type="Lock",
|
||||||
|
wait_event="advisory",
|
||||||
|
)
|
||||||
|
|
||||||
|
apply_process = _start_psql(
|
||||||
|
postgres,
|
||||||
|
_instrument_apply_before_transaction(apply_sql, "teleo_v3_migration_first_apply"),
|
||||||
|
role="kb_apply",
|
||||||
|
password=worker.APPLY_PASSWORD,
|
||||||
|
)
|
||||||
|
_wait_for_activity(
|
||||||
|
postgres,
|
||||||
|
"teleo_v3_migration_first_apply",
|
||||||
|
wait_event_type="Lock",
|
||||||
|
wait_event="advisory",
|
||||||
|
)
|
||||||
|
|
||||||
|
_release_barrier(postgres, holder)
|
||||||
|
holder = None
|
||||||
|
migration_result = _finish(migration_process)
|
||||||
|
migration_process = None
|
||||||
|
assert migration_result.returncode == 0, migration_result.stderr
|
||||||
|
apply_result = _finish(apply_process)
|
||||||
|
apply_process = None
|
||||||
|
assert apply_result.returncode != 0
|
||||||
|
assert "V3 catalog contract is INVALID" in apply_result.stderr
|
||||||
|
assert postgres.psql_json(
|
||||||
|
f"""
|
||||||
|
select pg_catalog.jsonb_build_object(
|
||||||
|
'proposal_status', (
|
||||||
|
select status from kb_stage.kb_proposals
|
||||||
|
where id = '{worker.RACE_IDS["proposal"]}'::uuid
|
||||||
|
),
|
||||||
|
'applied_at', (
|
||||||
|
select applied_at from kb_stage.kb_proposals
|
||||||
|
where id = '{worker.RACE_IDS["proposal"]}'::uuid
|
||||||
|
),
|
||||||
|
'claims', (
|
||||||
|
select count(*) from public.claims
|
||||||
|
where accepted_by_proposal_id = '{worker.RACE_IDS["proposal"]}'::uuid
|
||||||
|
),
|
||||||
|
'sources', (
|
||||||
|
select count(*) from public.sources
|
||||||
|
where accepted_by_proposal_id = '{worker.RACE_IDS["proposal"]}'::uuid
|
||||||
|
),
|
||||||
|
'evidence', (
|
||||||
|
select count(*) from public.claim_evidence
|
||||||
|
where accepted_from_proposal_id = '{worker.RACE_IDS["proposal"]}'::uuid
|
||||||
|
),
|
||||||
|
'edges', (
|
||||||
|
select count(*) from public.claim_edges
|
||||||
|
where accepted_by_proposal_id = '{worker.RACE_IDS["proposal"]}'::uuid
|
||||||
|
),
|
||||||
|
'assessments', (
|
||||||
|
select count(*) from public.claim_evidence_assessments
|
||||||
|
where accepted_by_proposal_id = '{worker.RACE_IDS["proposal"]}'::uuid
|
||||||
|
),
|
||||||
|
'contract_state', ({worker.apply.build_v3_contract_state_query()})
|
||||||
|
)::text;
|
||||||
|
"""
|
||||||
|
) == {
|
||||||
|
"proposal_status": "approved",
|
||||||
|
"applied_at": None,
|
||||||
|
"claims": 0,
|
||||||
|
"sources": 0,
|
||||||
|
"evidence": 0,
|
||||||
|
"edges": 0,
|
||||||
|
"assessments": 0,
|
||||||
|
"contract_state": "INVALID",
|
||||||
|
}
|
||||||
|
lock_cleanup = postgres.psql_json(
|
||||||
|
f"""
|
||||||
|
with acquired as (
|
||||||
|
select pg_catalog.pg_try_advisory_lock({CATALOG_DDL_ADVISORY_LOCK_KEY}) as value
|
||||||
|
), released as (
|
||||||
|
select pg_catalog.pg_advisory_unlock({CATALOG_DDL_ADVISORY_LOCK_KEY}) as value
|
||||||
|
from acquired where value
|
||||||
|
)
|
||||||
|
select pg_catalog.jsonb_build_object(
|
||||||
|
'exclusive_lock_acquired', (select value from acquired),
|
||||||
|
'exclusive_lock_released', coalesce((select value from released), false)
|
||||||
|
)::text;
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
assert lock_cleanup == {
|
||||||
|
"exclusive_lock_acquired": True,
|
||||||
|
"exclusive_lock_released": True,
|
||||||
|
}
|
||||||
|
finally:
|
||||||
|
if holder is not None and holder.poll() is None:
|
||||||
|
_release_barrier(postgres, holder)
|
||||||
|
_stop(migration_process)
|
||||||
|
_stop(apply_process)
|
||||||
|
_stop(holder)
|
||||||
|
cleanup = postgres.cleanup()
|
||||||
|
assert cleanup["container_absent"] is True
|
||||||
|
assert cleanup["instance_label_readback_clear"] is True
|
||||||
1545
tests/test_v3_apply_worker_postgres.py
Normal file
1545
tests/test_v3_apply_worker_postgres.py
Normal file
File diff suppressed because it is too large
Load diff
701
tests/test_v3_source_rebuild_lifecycle_postgres.py
Normal file
701
tests/test_v3_source_rebuild_lifecycle_postgres.py
Normal file
|
|
@ -0,0 +1,701 @@
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import secrets
|
||||||
|
import stat
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import uuid
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from ops import run_local_corpus_knowledge_rebuild as corpus_rebuild
|
||||||
|
from ops import run_local_genesis_ledger_rebuild as genesis_rebuild
|
||||||
|
from ops import verify_teleo_v3_epistemic_contract as v3_contract
|
||||||
|
from scripts import compile_kb_source_packet as compiler
|
||||||
|
from tests import test_run_local_genesis_ledger_rebuild as genesis_test
|
||||||
|
from tests import test_v3_apply_worker_postgres as worker_test
|
||||||
|
|
||||||
|
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
CORPUS_REBUILD = REPO_ROOT / "ops" / "run_local_corpus_knowledge_rebuild.py"
|
||||||
|
SOURCE_COMPILER = REPO_ROOT / "scripts" / "compile_kb_source_packet.py"
|
||||||
|
PROPOSAL_STAGER = REPO_ROOT / "scripts" / "stage_normalized_proposal.py"
|
||||||
|
REVIEW_PACKET = REPO_ROOT / "scripts" / "kb_proposal_review_packet.py"
|
||||||
|
APPROVER = REPO_ROOT / "scripts" / "approve_proposal.py"
|
||||||
|
APPLY_WORKER = REPO_ROOT / "scripts" / "apply_worker.py"
|
||||||
|
MATERIAL_EXPORTER = REPO_ROOT / "scripts" / "export_kb_transition_replay_material.py"
|
||||||
|
LEDGER_BUILDER = REPO_ROOT / "ops" / "build_v3_genesis_ledger_bundle.py"
|
||||||
|
GENESIS_REBUILDER = REPO_ROOT / "ops" / "run_local_genesis_ledger_rebuild.py"
|
||||||
|
|
||||||
|
SOURCE_CONTAINER_LABEL = "livingip.test.v3-source-rebuild-lifecycle"
|
||||||
|
STAGING_LABEL = "com.livingip.leo.checkpoint=disposable"
|
||||||
|
DATABASE = "teleo_v3_source_lifecycle"
|
||||||
|
SYNTHETIC_REVIEW_NOTE = "SYNTHETIC TEST ONLY: operator-supplied fixture approval; this is not human review proof."
|
||||||
|
|
||||||
|
|
||||||
|
def _sha256(path: Path) -> str:
|
||||||
|
return hashlib.sha256(path.read_bytes()).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _run(command: list[str], *, timeout: int = 240) -> subprocess.CompletedProcess[str]:
|
||||||
|
completed = subprocess.run(
|
||||||
|
command,
|
||||||
|
cwd=REPO_ROOT,
|
||||||
|
text=True,
|
||||||
|
capture_output=True,
|
||||||
|
check=False,
|
||||||
|
timeout=timeout,
|
||||||
|
)
|
||||||
|
assert completed.returncode == 0, (
|
||||||
|
f"command failed ({completed.returncode}): {command[0]} {Path(command[1]).name if len(command) > 1 else ''}\n"
|
||||||
|
f"stdout:\n{completed.stdout}\nstderr:\n{completed.stderr}"
|
||||||
|
)
|
||||||
|
return completed
|
||||||
|
|
||||||
|
|
||||||
|
def _docker(*arguments: str, check: bool = True, timeout: int = 120) -> subprocess.CompletedProcess[str]:
|
||||||
|
completed = subprocess.run(
|
||||||
|
["docker", *arguments],
|
||||||
|
text=True,
|
||||||
|
capture_output=True,
|
||||||
|
check=False,
|
||||||
|
timeout=timeout,
|
||||||
|
)
|
||||||
|
if check:
|
||||||
|
assert completed.returncode == 0, completed.stderr or completed.stdout
|
||||||
|
return completed
|
||||||
|
|
||||||
|
|
||||||
|
def _write_private(path: Path, content: str | bytes) -> Path:
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||||
|
path.parent.chmod(0o700)
|
||||||
|
if isinstance(content, bytes):
|
||||||
|
path.write_bytes(content)
|
||||||
|
else:
|
||||||
|
path.write_text(content, encoding="utf-8")
|
||||||
|
path.chmod(0o600)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def _start_postgres(name: str, database: str, run_label: str, *, staging_target: bool) -> str:
|
||||||
|
command = [
|
||||||
|
"run",
|
||||||
|
"--detach",
|
||||||
|
"--rm",
|
||||||
|
"--network",
|
||||||
|
"none",
|
||||||
|
"--name",
|
||||||
|
name,
|
||||||
|
"--label",
|
||||||
|
f"{SOURCE_CONTAINER_LABEL}={run_label}",
|
||||||
|
"--tmpfs",
|
||||||
|
"/var/lib/postgresql/data:rw,nosuid,nodev,noexec,size=384m",
|
||||||
|
"--env",
|
||||||
|
f"POSTGRES_DB={database}",
|
||||||
|
"--env",
|
||||||
|
"POSTGRES_HOST_AUTH_METHOD=trust",
|
||||||
|
]
|
||||||
|
if staging_target:
|
||||||
|
command.extend(("--label", STAGING_LABEL))
|
||||||
|
command.append(v3_contract.POSTGRES_IMAGE)
|
||||||
|
container_id = _docker(*command).stdout.strip()
|
||||||
|
assert container_id
|
||||||
|
|
||||||
|
for _ in range(180):
|
||||||
|
ready = _docker(
|
||||||
|
"exec",
|
||||||
|
name,
|
||||||
|
"pg_isready",
|
||||||
|
"-U",
|
||||||
|
"postgres",
|
||||||
|
"-h",
|
||||||
|
"127.0.0.1",
|
||||||
|
"-d",
|
||||||
|
database,
|
||||||
|
check=False,
|
||||||
|
timeout=10,
|
||||||
|
)
|
||||||
|
if ready.returncode == 0:
|
||||||
|
break
|
||||||
|
time.sleep(0.25)
|
||||||
|
else:
|
||||||
|
pytest.fail(f"disposable PostgreSQL did not become ready: {name}")
|
||||||
|
|
||||||
|
inspected = json.loads(_docker("inspect", name).stdout)[0]
|
||||||
|
assert inspected["HostConfig"]["NetworkMode"] == "none"
|
||||||
|
assert inspected["NetworkSettings"]["Ports"] == {}
|
||||||
|
assert inspected["Mounts"] == []
|
||||||
|
assert "/var/lib/postgresql/data" in inspected["HostConfig"]["Tmpfs"]
|
||||||
|
assert inspected["Config"]["Labels"][SOURCE_CONTAINER_LABEL] == run_label
|
||||||
|
if staging_target:
|
||||||
|
assert inspected["Config"]["Labels"]["com.livingip.leo.checkpoint"] == "disposable"
|
||||||
|
return container_id
|
||||||
|
|
||||||
|
|
||||||
|
def _remove_container(name: str) -> None:
|
||||||
|
_docker("rm", "--force", name, check=False)
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_container_absent(name: str) -> None:
|
||||||
|
assert _docker("inspect", name, check=False).returncode != 0
|
||||||
|
names = _docker(
|
||||||
|
"container",
|
||||||
|
"ls",
|
||||||
|
"--all",
|
||||||
|
"--filter",
|
||||||
|
f"name=^/{name}$",
|
||||||
|
"--format",
|
||||||
|
"{{.Names}}",
|
||||||
|
)
|
||||||
|
assert names.stdout.strip() == ""
|
||||||
|
|
||||||
|
|
||||||
|
def _capture_dump(container: str, database: str, destination: Path) -> None:
|
||||||
|
remote = f"/tmp/{destination.name}"
|
||||||
|
_docker(
|
||||||
|
"exec",
|
||||||
|
container,
|
||||||
|
"pg_dump",
|
||||||
|
"-U",
|
||||||
|
"postgres",
|
||||||
|
"-d",
|
||||||
|
database,
|
||||||
|
"--format=custom",
|
||||||
|
f"--file={remote}",
|
||||||
|
)
|
||||||
|
_docker("cp", f"{container}:{remote}", str(destination))
|
||||||
|
destination.chmod(0o600)
|
||||||
|
|
||||||
|
|
||||||
|
def _capture_manifest(container: str, database: str, destination: Path) -> None:
|
||||||
|
genesis_test.capture_manifest(container, database, destination)
|
||||||
|
destination.chmod(0o600)
|
||||||
|
|
||||||
|
|
||||||
|
def _json_stdout(completed: subprocess.CompletedProcess[str]) -> dict:
|
||||||
|
value = json.loads(completed.stdout)
|
||||||
|
assert isinstance(value, dict)
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_private_files(root: Path) -> None:
|
||||||
|
files = [path for path in root.rglob("*") if path.is_file()]
|
||||||
|
assert files
|
||||||
|
for path in files:
|
||||||
|
assert stat.S_IMODE(path.stat().st_mode) == 0o600, path
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(not v3_contract.docker_available(), reason="Docker daemon is required")
|
||||||
|
def test_v3_source_rebuild_lifecycle_is_exact_networkless_and_cleanup_proven(tmp_path: Path) -> None:
|
||||||
|
if _docker("image", "inspect", v3_contract.POSTGRES_IMAGE, check=False).returncode != 0:
|
||||||
|
pytest.skip("the pinned PostgreSQL image must already exist; this test never pulls over the network")
|
||||||
|
run_id = uuid.uuid4().hex[:12]
|
||||||
|
source_container = f"teleo-v3-source-{run_id}"
|
||||||
|
rebuild_prefix = f"teleo-v3-rebuild-{run_id[:8]}"
|
||||||
|
private_root = tmp_path / "private"
|
||||||
|
private_root.mkdir(mode=0o700)
|
||||||
|
password = secrets.token_hex(24)
|
||||||
|
source_started = False
|
||||||
|
rebuild_container: str | None = None
|
||||||
|
|
||||||
|
try:
|
||||||
|
corpus_root = private_root / "corpus"
|
||||||
|
source_text = (
|
||||||
|
"# Canonical review boundary\n\n"
|
||||||
|
"A staged knowledge proposal remains non-canonical until explicit review and guarded apply both succeed.\n"
|
||||||
|
)
|
||||||
|
source_path = _write_private(corpus_root / "review-boundary.md", source_text)
|
||||||
|
inventory = private_root / "source-inventory"
|
||||||
|
|
||||||
|
built_inventory = _run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
str(CORPUS_REBUILD),
|
||||||
|
"--source-root",
|
||||||
|
str(corpus_root),
|
||||||
|
"--output",
|
||||||
|
str(inventory),
|
||||||
|
"--mode",
|
||||||
|
"source-inventory-only",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
inventory_status = _json_stdout(built_inventory)
|
||||||
|
assert inventory_status["status"] == "pass"
|
||||||
|
assert inventory_status["source_count"] == 1
|
||||||
|
verified_inventory = _run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
str(CORPUS_REBUILD),
|
||||||
|
"--verify-bundle",
|
||||||
|
str(inventory),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
assert _json_stdout(verified_inventory)["status"] == "pass"
|
||||||
|
|
||||||
|
source_manifest = json.loads((inventory / corpus_rebuild.SOURCE_MANIFEST_PATH).read_text(encoding="utf-8"))
|
||||||
|
assert len(source_manifest["sources"]) == 1
|
||||||
|
source_row = source_manifest["sources"][0]
|
||||||
|
snapshot = inventory / source_row["snapshot_path"]
|
||||||
|
assert snapshot.read_bytes() == source_path.read_bytes()
|
||||||
|
assert source_row["artifact_sha256"] == _sha256(snapshot)
|
||||||
|
assert snapshot.name == f"{source_row['artifact_sha256']}.utf8"
|
||||||
|
|
||||||
|
quote = (
|
||||||
|
"A staged knowledge proposal remains non-canonical until explicit review and guarded apply both succeed."
|
||||||
|
)
|
||||||
|
text_bytes = snapshot.read_bytes()
|
||||||
|
quote_bytes = quote.encode("utf-8")
|
||||||
|
quote_start = text_bytes.index(quote_bytes)
|
||||||
|
artifact_sha256 = _sha256(snapshot)
|
||||||
|
extractor_spec_bytes = (
|
||||||
|
b"livingip synthetic lifecycle extractor specification\n"
|
||||||
|
b"name=synthetic-v3-source-lifecycle\n"
|
||||||
|
b"version=1.0.0\n"
|
||||||
|
b"quotes=exact-utf8-byte-locators\n"
|
||||||
|
b"approval=test-operator-supplied-not-human-proof\n"
|
||||||
|
)
|
||||||
|
extractor_spec = _write_private(private_root / "extractor.spec", extractor_spec_bytes)
|
||||||
|
extraction_manifest = {
|
||||||
|
"schema": compiler.MANIFEST_SCHEMA_V3,
|
||||||
|
"artifact_sha256": artifact_sha256,
|
||||||
|
"extracted_text_sha256": artifact_sha256,
|
||||||
|
"extractor": {
|
||||||
|
"name": "synthetic-v3-source-lifecycle",
|
||||||
|
"version": "1.0.0",
|
||||||
|
"spec_sha256": hashlib.sha256(extractor_spec_bytes).hexdigest(),
|
||||||
|
},
|
||||||
|
"agent_id": v3_contract.AGENT_ID,
|
||||||
|
"source": {
|
||||||
|
"identity": f"document:sha256:{artifact_sha256}",
|
||||||
|
"source_key": "canonical_review_boundary",
|
||||||
|
"source_type": "article",
|
||||||
|
"title": "Canonical review boundary",
|
||||||
|
"locator": f"artifact://sha256/{artifact_sha256}",
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"ingestion_origin": "operator_upload",
|
||||||
|
"provenance_status": "verified",
|
||||||
|
"source_class": "primary_record",
|
||||||
|
"auto_promotion_policy": "human_required",
|
||||||
|
"captured_at": "2026-07-19T00:00:00+00:00",
|
||||||
|
},
|
||||||
|
"claims": [
|
||||||
|
{
|
||||||
|
"claim_key": "review_and_apply_precede_canonical_state",
|
||||||
|
"type": "empirical",
|
||||||
|
"proposition": (
|
||||||
|
"A staged knowledge proposal is not canonical until explicit review and guarded apply succeed."
|
||||||
|
),
|
||||||
|
"body_markdown": "The exact retained source states both gates and their order.",
|
||||||
|
"scope": "The exact content-addressed source snapshot used by this disposable lifecycle.",
|
||||||
|
"access_scope": "collective_internal",
|
||||||
|
"owner_agent_id": None,
|
||||||
|
"status": "active",
|
||||||
|
"revision_criteria": "Revise if the exact source bytes or guarded apply contract changes.",
|
||||||
|
"revision_kind": "original",
|
||||||
|
"supersedes_id": None,
|
||||||
|
"tags": ["genesis", "provenance", "review"],
|
||||||
|
"evidence": [
|
||||||
|
{
|
||||||
|
"evidence_key": "exact_review_boundary_sentence",
|
||||||
|
"quote": quote,
|
||||||
|
"polarity": "supports",
|
||||||
|
"locator": {
|
||||||
|
"unit": "utf8_bytes",
|
||||||
|
"start": quote_start,
|
||||||
|
"end": quote_start + len(quote_bytes),
|
||||||
|
},
|
||||||
|
"rationale": "The exact byte-bound sentence directly supports the bounded proposition.",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"assessment": {
|
||||||
|
"support_tier": "strong",
|
||||||
|
"challenge_tier": "none",
|
||||||
|
"overall_state": "support_dominant",
|
||||||
|
"rationale": "One exact verified primary-record sentence directly supports the claim.",
|
||||||
|
"supersedes_assessment_id": None,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
manifest_path = _write_private(
|
||||||
|
private_root / "synthetic-extraction-manifest.json",
|
||||||
|
json.dumps(extraction_manifest, indent=2, sort_keys=True) + "\n",
|
||||||
|
)
|
||||||
|
compiled_bundle_path = private_root / "compiled-v3-bundle.json"
|
||||||
|
compiled = _run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
str(SOURCE_COMPILER),
|
||||||
|
"--artifact",
|
||||||
|
str(snapshot),
|
||||||
|
"--text",
|
||||||
|
str(snapshot),
|
||||||
|
"--manifest",
|
||||||
|
str(manifest_path),
|
||||||
|
"--extractor-spec",
|
||||||
|
str(extractor_spec),
|
||||||
|
"--output",
|
||||||
|
str(compiled_bundle_path),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
assert _json_stdout(compiled)["status"] == "pending_review"
|
||||||
|
compiled_bundle = json.loads(compiled_bundle_path.read_text(encoding="utf-8"))
|
||||||
|
child = compiled_bundle["strict_child_proposal"]
|
||||||
|
proposal_id = child["id"]
|
||||||
|
apply_payload = child["payload"]["apply_payload"]
|
||||||
|
assert compiled_bundle["hashes"]["artifact_sha256"] == artifact_sha256
|
||||||
|
assert apply_payload["sources"][0]["content_hash"] == artifact_sha256
|
||||||
|
|
||||||
|
source_container_id = _start_postgres(source_container, DATABASE, run_id, staging_target=True)
|
||||||
|
source_started = True
|
||||||
|
|
||||||
|
genesis_test.docker_psql(source_container, DATABASE, v3_contract.BOOTSTRAP_SQL)
|
||||||
|
genesis_test.docker_psql(source_container, DATABASE, worker_test._supplemental_bootstrap())
|
||||||
|
genesis_test.docker_psql(
|
||||||
|
source_container,
|
||||||
|
DATABASE,
|
||||||
|
(f"alter role kb_review login password '{password}';alter role kb_apply login password '{password}';"),
|
||||||
|
)
|
||||||
|
genesis_test.docker_psql(
|
||||||
|
source_container,
|
||||||
|
DATABASE,
|
||||||
|
worker_test.PREREQUISITES.read_text(encoding="utf-8"),
|
||||||
|
)
|
||||||
|
|
||||||
|
genesis_dump = private_root / "baseline-genesis.dump"
|
||||||
|
genesis_manifest = private_root / "baseline-genesis-manifest.jsonl"
|
||||||
|
_capture_dump(source_container, DATABASE, genesis_dump)
|
||||||
|
_capture_manifest(source_container, DATABASE, genesis_manifest)
|
||||||
|
|
||||||
|
genesis_test.docker_psql(source_container, DATABASE, "alter role kb_apply nologin;")
|
||||||
|
genesis_test.docker_psql(
|
||||||
|
source_container,
|
||||||
|
DATABASE,
|
||||||
|
worker_test.MIGRATION.read_text(encoding="utf-8"),
|
||||||
|
)
|
||||||
|
genesis_test.docker_psql(
|
||||||
|
source_container,
|
||||||
|
DATABASE,
|
||||||
|
worker_test.PREREQUISITES.read_text(encoding="utf-8"),
|
||||||
|
)
|
||||||
|
|
||||||
|
secrets_file = _write_private(
|
||||||
|
private_root / "database-secrets.env",
|
||||||
|
f"KB_APPLY_DB_PASSWORD={password}\nKB_REVIEW_DB_PASSWORD={password}\n",
|
||||||
|
)
|
||||||
|
staged = _run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
str(PROPOSAL_STAGER),
|
||||||
|
"--container",
|
||||||
|
source_container,
|
||||||
|
"--db",
|
||||||
|
DATABASE,
|
||||||
|
"--compiled-v3-bundle",
|
||||||
|
str(compiled_bundle_path),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
stage_status = _json_stdout(staged)
|
||||||
|
assert stage_status == {
|
||||||
|
"bound_container_id": source_container_id,
|
||||||
|
"contract_version": 3,
|
||||||
|
"created": True,
|
||||||
|
"payload_sha256": child["payload_sha256"],
|
||||||
|
"proposal_id": proposal_id,
|
||||||
|
"status": "pending_review",
|
||||||
|
}
|
||||||
|
|
||||||
|
reviewed = _run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
str(REVIEW_PACKET),
|
||||||
|
"--proposal-id",
|
||||||
|
proposal_id,
|
||||||
|
"--format",
|
||||||
|
"json",
|
||||||
|
"--secrets-file",
|
||||||
|
str(secrets_file),
|
||||||
|
"--container",
|
||||||
|
source_container,
|
||||||
|
"--db",
|
||||||
|
DATABASE,
|
||||||
|
"--host",
|
||||||
|
"127.0.0.1",
|
||||||
|
"--role",
|
||||||
|
"kb_review",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
packets = _json_stdout(reviewed)["packets"]
|
||||||
|
assert len(packets) == 1
|
||||||
|
packet = packets[0]
|
||||||
|
assert packet["proposal_id"] == proposal_id
|
||||||
|
assert packet["review_state"] == "needs_human_review"
|
||||||
|
assert packet["strict_review_material"]["rows"] == {
|
||||||
|
key: apply_payload[key]
|
||||||
|
for key in ("claims", "sources", "evidence", "edges", "assessments", "reasoning_tools")
|
||||||
|
}
|
||||||
|
assert packet["strict_review_material"]["normalization_manifest"] == apply_payload["normalization_manifest"]
|
||||||
|
|
||||||
|
approved = _run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
str(APPROVER),
|
||||||
|
proposal_id,
|
||||||
|
"--reviewed-by",
|
||||||
|
"m3taversal",
|
||||||
|
"--review-note",
|
||||||
|
SYNTHETIC_REVIEW_NOTE,
|
||||||
|
"--secrets-file",
|
||||||
|
str(secrets_file),
|
||||||
|
"--container",
|
||||||
|
source_container,
|
||||||
|
"--db",
|
||||||
|
DATABASE,
|
||||||
|
"--host",
|
||||||
|
"127.0.0.1",
|
||||||
|
"--role",
|
||||||
|
"kb_review",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
approval_status = _json_stdout(approved)
|
||||||
|
assert approval_status["id"] == proposal_id
|
||||||
|
assert approval_status["status"] == "approved"
|
||||||
|
assert approval_status["reviewed_by_handle"] == "m3taversal"
|
||||||
|
|
||||||
|
receipts = private_root / "apply-receipts"
|
||||||
|
failure_state = private_root / "apply-worker-state.json"
|
||||||
|
applied = _run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
str(APPLY_WORKER),
|
||||||
|
"--enable",
|
||||||
|
"--contract-version",
|
||||||
|
"3",
|
||||||
|
"--proposal-id",
|
||||||
|
proposal_id,
|
||||||
|
"--max-per-tick",
|
||||||
|
"1",
|
||||||
|
"--secrets-file",
|
||||||
|
str(secrets_file),
|
||||||
|
"--failure-state-file",
|
||||||
|
str(failure_state),
|
||||||
|
"--receipt-dir",
|
||||||
|
str(receipts),
|
||||||
|
"--container",
|
||||||
|
source_container,
|
||||||
|
"--db",
|
||||||
|
DATABASE,
|
||||||
|
"--host",
|
||||||
|
"127.0.0.1",
|
||||||
|
"--role",
|
||||||
|
"kb_apply",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
assert f"applied {proposal_id} (approve_claim)" in applied.stdout
|
||||||
|
assert json.loads(failure_state.read_text(encoding="utf-8")) == {}
|
||||||
|
failure_state.unlink()
|
||||||
|
replay_receipt = receipts / f"{proposal_id}.json"
|
||||||
|
receipt = json.loads(replay_receipt.read_text(encoding="utf-8"))
|
||||||
|
assert receipt["proposal"]["reviewed_by_handle"] == "m3taversal"
|
||||||
|
assert receipt["canonical_rows"]["sources"][0]["content_hash"] == artifact_sha256
|
||||||
|
assert receipt["canonical_rows"]["claims"][0]["proposition"] == extraction_manifest["claims"][0]["proposition"]
|
||||||
|
|
||||||
|
material_path = private_root / "entry-0001.json"
|
||||||
|
exported = _run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
str(MATERIAL_EXPORTER),
|
||||||
|
proposal_id,
|
||||||
|
"--replay-receipt",
|
||||||
|
str(replay_receipt),
|
||||||
|
"--sequence",
|
||||||
|
"1",
|
||||||
|
"--output",
|
||||||
|
str(material_path),
|
||||||
|
"--secrets-file",
|
||||||
|
str(secrets_file),
|
||||||
|
"--container",
|
||||||
|
source_container,
|
||||||
|
"--db",
|
||||||
|
DATABASE,
|
||||||
|
"--host",
|
||||||
|
"127.0.0.1",
|
||||||
|
"--role",
|
||||||
|
"kb_apply",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
assert exported.stdout.strip() == f"replay material created: proposal={proposal_id} sequence=1"
|
||||||
|
material = json.loads(material_path.read_text(encoding="utf-8"))
|
||||||
|
assert material["contract_version"] == genesis_rebuild.MATERIAL_CONTRACT_VERSION
|
||||||
|
assert material["applied_proposal"]["payload"]["apply_payload"]["contract_version"] == 3
|
||||||
|
assert material["approved_proposal"]["review_note"] == SYNTHETIC_REVIEW_NOTE
|
||||||
|
assert material["approval_snapshot"]["reviewed_by_db_role"] == "kb_review"
|
||||||
|
|
||||||
|
final_manifest = private_root / "final-parity-manifest.jsonl"
|
||||||
|
_capture_manifest(source_container, DATABASE, final_manifest)
|
||||||
|
|
||||||
|
_remove_container(source_container)
|
||||||
|
source_started = False
|
||||||
|
_assert_container_absent(source_container)
|
||||||
|
labeled_sources = _docker(
|
||||||
|
"container",
|
||||||
|
"ls",
|
||||||
|
"--all",
|
||||||
|
"--filter",
|
||||||
|
f"label={SOURCE_CONTAINER_LABEL}={run_id}",
|
||||||
|
"--format",
|
||||||
|
"{{.Names}}",
|
||||||
|
)
|
||||||
|
assert labeled_sources.stdout.strip() == ""
|
||||||
|
|
||||||
|
review_note_file = _write_private(private_root / "admission-review-note.txt", SYNTHETIC_REVIEW_NOTE + "\n")
|
||||||
|
ledger_path = private_root / "v3-genesis-ledger.json"
|
||||||
|
admission_path = private_root / "v3-admission-manifest.json"
|
||||||
|
built_ledger = _run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
str(LEDGER_BUILDER),
|
||||||
|
"--genesis-dump",
|
||||||
|
str(genesis_dump),
|
||||||
|
"--genesis-manifest",
|
||||||
|
str(genesis_manifest),
|
||||||
|
"--final-manifest",
|
||||||
|
str(final_manifest),
|
||||||
|
"--material",
|
||||||
|
str(material_path),
|
||||||
|
"--output-ledger",
|
||||||
|
str(ledger_path),
|
||||||
|
"--emit-admission-manifest",
|
||||||
|
"--admission-output",
|
||||||
|
str(admission_path),
|
||||||
|
"--reviewed-by",
|
||||||
|
"m3taversal",
|
||||||
|
"--reviewed-by-agent-id",
|
||||||
|
worker_test.REVIEWER_AGENT_ID,
|
||||||
|
"--reviewed-at",
|
||||||
|
"2026-07-19T00:00:00+00:00",
|
||||||
|
"--review-note-file",
|
||||||
|
str(review_note_file),
|
||||||
|
"--admission-scope",
|
||||||
|
"knowledge_commons",
|
||||||
|
"--admission-basis",
|
||||||
|
"reviewed_primary_source",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
ledger_status = _json_stdout(built_ledger)
|
||||||
|
assert ledger_status["status"] == "pass"
|
||||||
|
assert ledger_status["action"] == "build"
|
||||||
|
ledger_sha256 = _sha256(ledger_path)
|
||||||
|
admission_sha256 = _sha256(admission_path)
|
||||||
|
admission = json.loads(admission_path.read_text(encoding="utf-8"))
|
||||||
|
assert admission["reviewed_by_handle"] == "m3taversal"
|
||||||
|
assert admission["review_note"] == SYNTHETIC_REVIEW_NOTE + "\n"
|
||||||
|
|
||||||
|
verified_ledger = _run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
str(LEDGER_BUILDER),
|
||||||
|
"--genesis-dump",
|
||||||
|
str(genesis_dump),
|
||||||
|
"--genesis-manifest",
|
||||||
|
str(genesis_manifest),
|
||||||
|
"--verify-ledger",
|
||||||
|
str(ledger_path),
|
||||||
|
"--ledger-sha256",
|
||||||
|
ledger_sha256,
|
||||||
|
"--admission-manifest",
|
||||||
|
str(admission_path),
|
||||||
|
"--admission-manifest-sha256",
|
||||||
|
admission_sha256,
|
||||||
|
]
|
||||||
|
)
|
||||||
|
verify_status = _json_stdout(verified_ledger)
|
||||||
|
assert verify_status["status"] == "pass"
|
||||||
|
assert verify_status["action"] == "verify"
|
||||||
|
|
||||||
|
reconstruction_receipt = private_root / "reconstruction-receipt.json"
|
||||||
|
rebuilt = _run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
str(GENESIS_REBUILDER),
|
||||||
|
"--genesis-dump",
|
||||||
|
str(genesis_dump),
|
||||||
|
"--genesis-manifest",
|
||||||
|
str(genesis_manifest),
|
||||||
|
"--ledger",
|
||||||
|
str(ledger_path),
|
||||||
|
"--ledger-sha256",
|
||||||
|
ledger_sha256,
|
||||||
|
"--admission-manifest",
|
||||||
|
str(admission_path),
|
||||||
|
"--admission-manifest-sha256",
|
||||||
|
admission_sha256,
|
||||||
|
"--database",
|
||||||
|
DATABASE,
|
||||||
|
"--container-prefix",
|
||||||
|
rebuild_prefix,
|
||||||
|
"--tmpfs-mb",
|
||||||
|
"384",
|
||||||
|
"--max-target-query-ms",
|
||||||
|
"5000",
|
||||||
|
"--max-target-source-ratio",
|
||||||
|
"100",
|
||||||
|
"--output",
|
||||||
|
str(reconstruction_receipt),
|
||||||
|
],
|
||||||
|
timeout=360,
|
||||||
|
)
|
||||||
|
assert _json_stdout(rebuilt)["status"] == "pass"
|
||||||
|
reconstruction = json.loads(reconstruction_receipt.read_text(encoding="utf-8"))
|
||||||
|
rebuild_container = reconstruction["container"]["name"]
|
||||||
|
assert reconstruction["status"] == "pass"
|
||||||
|
assert reconstruction["genesis_parity"]["status"] == "pass"
|
||||||
|
assert reconstruction["genesis_parity"]["problems"] == []
|
||||||
|
assert reconstruction["v3_contract"]["status"] == "pass"
|
||||||
|
assert reconstruction["ledger"]["entries"][0]["proposal_exact"] is True
|
||||||
|
assert reconstruction["ledger"]["entries"][0]["canonical_rows_exact"] is True
|
||||||
|
assert reconstruction["admission"]["status"] == "review_metadata_supplied"
|
||||||
|
assert reconstruction["admission"]["canonical_retrieval_eligible"] is False
|
||||||
|
assert reconstruction["admission"]["cryptographically_authenticated"] is False
|
||||||
|
assert reconstruction["final_parity"]["status"] == "pass"
|
||||||
|
assert reconstruction["final_parity"]["problems"] == []
|
||||||
|
assert reconstruction["cleanup"]["container_absent"] is True
|
||||||
|
assert reconstruction["safety"] == {
|
||||||
|
"network_access_available_to_container": False,
|
||||||
|
"persistent_postgres_storage_used": False,
|
||||||
|
"private_replay_material_emitted": False,
|
||||||
|
"production_database_touched": False,
|
||||||
|
"vps_gcp_telegram_or_live_db_used": False,
|
||||||
|
}
|
||||||
|
_assert_container_absent(rebuild_container)
|
||||||
|
rebuilt_labels = _docker(
|
||||||
|
"container",
|
||||||
|
"ls",
|
||||||
|
"--all",
|
||||||
|
"--filter",
|
||||||
|
f"label={genesis_rebuild.canonical_rebuild.CANARY_LABEL}={rebuild_container}",
|
||||||
|
"--format",
|
||||||
|
"{{.Names}}",
|
||||||
|
)
|
||||||
|
assert rebuilt_labels.stdout.strip() == ""
|
||||||
|
_assert_private_files(private_root)
|
||||||
|
finally:
|
||||||
|
if source_started:
|
||||||
|
_remove_container(source_container)
|
||||||
|
|
||||||
|
_assert_container_absent(source_container)
|
||||||
|
if rebuild_container is not None:
|
||||||
|
_assert_container_absent(rebuild_container)
|
||||||
|
orphaned = _docker(
|
||||||
|
"container",
|
||||||
|
"ls",
|
||||||
|
"--all",
|
||||||
|
"--filter",
|
||||||
|
f"label={SOURCE_CONTAINER_LABEL}={run_id}",
|
||||||
|
"--format",
|
||||||
|
"{{.Names}}",
|
||||||
|
)
|
||||||
|
assert orphaned.stdout.strip() == ""
|
||||||
Loading…
Reference in a new issue