teleo/teleo-infrastructure
8
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions 1

Merge pull request #220 from living-ip/codex/source-first-v3-foundation-integration-20260721
Some checks are pending
CI / lint-and-test (push) Waiting to run
Details

Browse source 
Integrate hardened source-first V3 rebuild foundation
This commit is contained in:
twentyOne2x 2026-07-22 03:31:39 +02:00 committed by GitHub
commit fab6c17805
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
51 changed files with 24151 additions and 306 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

153
.agents/skills/teleo-corpus-knowledge-rebuild/SKILL.md Normal file
View file

@ -0,0 +1,153 @@
---
name: teleo-corpus-knowledge-rebuild
description: Use when locally snapshotting an extracted UTF-8 Forgejo or text corpus into a deterministic Teleo source library, or sharding its verified extraction-work index for parallel processing before any claim generation, review, admission, or database write.
---
# Teleo Corpus Knowledge Rebuild
## Job
Turn one local UTF-8 source directory into a deterministic, self-contained
source snapshot bundle. This is the source-library phase only: it preserves
bytes, paths, hashes, and syntactic section locators without interpreting any
section as a claim. It does not perform semantic claim extraction.
After that inventory verifies, optionally divide its exact work rows into
deterministic local shards so independent extractors can process a large corpus
in parallel. Sharding schedules work; it does not interpret source content.
## Operator Path
From the repository root, build into a path that does not exist:
```bash
python3 ops/run_local_corpus_knowledge_rebuild.py \
--source-root tests/fixtures/teleo_corpus_rebuild \
--output /private/tmp/teleo-corpus-rebuild-bundle \
--mode source-inventory-only
```
Then run the read-only verifier:
```bash
python3 ops/run_local_corpus_knowledge_rebuild.py \
--verify-bundle /private/tmp/teleo-corpus-rebuild-bundle
```
For parallel processing, choose a shard count from 1 through 1024 that does not
exceed the verified work-item count. Build into another path that does not
exist:
```bash
python3 ops/shard_corpus_extraction_work.py \
--inventory-bundle /private/tmp/teleo-corpus-rebuild-bundle \
--output-dir /private/tmp/teleo-corpus-extraction-shards \
--shard-count 64
```
Then verify the shards against the same inventory:
```bash
python3 ops/shard_corpus_extraction_work.py \
--inventory-bundle /private/tmp/teleo-corpus-rebuild-bundle \
--verify-bundle /private/tmp/teleo-corpus-extraction-shards
```
Inspect these stable entry points:
```text
source-manifest.json
extraction-work-index.json
receipt.json
snapshots/sha256/<prefix>/<sha256>.utf8
shard-manifest.json
shards/shard-<index>-of-<count>.json
```
## Required Readback
Require all of these:
- every source row binds a normalized source-root-relative path to one exact,
content-addressed byte snapshot;
- every section has exact one-based line and UTF-8 byte ranges plus a quote
hash;
- every work row is `source_section_for_semantic_extraction`;
- `semantic_claim_extraction_performed=false`,
`database_write_performed=false`, and `admission_performed=false` throughout;
- the receipt says the directory snapshot was not atomic and only guarantees
the exact files copied from stable descriptors;
- the verifier recomputes snapshots, locators, cardinality, artifact hashes,
bundle-tree hash, and receipt hash;
- a second run over unchanged inputs produces byte-identical output.
- every inventory work item appears in exactly one shard selected by
`sha256(utf8(work_item_id)) mod shard_count`;
- each shard binds the exact inventory tree, source-manifest, work-index, and
inventory-receipt hashes without embedding source-body text or absolute paths;
- the shard verifier rejects wrong inventories, missing, duplicate, unknown, or
misplaced work IDs, symlinks, extras, incomplete output, and rehashed
side-effect overclaims;
- shard directories are `0700`, files are `0600`, and repeated runs with the
same inventory and shard count are byte-identical.
Run focused checks:
```bash
python3 -m pytest -q tests/test_run_local_corpus_knowledge_rebuild.py
python3 -m pytest -q tests/test_shard_corpus_extraction_work.py
python3 -m ruff check \
ops/run_local_corpus_knowledge_rebuild.py \
ops/shard_corpus_extraction_work.py \
tests/test_run_local_corpus_knowledge_rebuild.py \
tests/test_shard_corpus_extraction_work.py
```
## Safety Boundary
The build reserves a new output directory atomically and never replaces an
existing destination. It publishes `receipt.json` while the incomplete marker
is still present, then removes that marker as the final completion step. It
opens source files descriptor-first without following
symlinks, streams exact bytes into private content-addressed snapshots, rejects
invalid UTF-8, binary controls, lone CR newlines, unsupported entries, path
collisions, and configured resource-limit overruns, and deduplicates identical
content.
The defaults are bounded. Raise them only with the explicit `--max-*` options;
the command enforces hard upper bounds on every override.
The sharder independently verifies the inventory before reading work metadata,
reserves a new output directory without replacement, writes through held
directory descriptors, keeps an incomplete marker through receipt publication,
and revalidates the inventory before publishing. It reads no snapshot bodies.
Hash shards can be empty; membership is stable only for the exact inventory and
the same shard count.
## Semantic Handoff
The inventory plus one verified shard is input to a separate semantic
extractor, not to a database writer. The extractor must verify both bundles,
then resolve each relative snapshot path and exact byte range from the supplied
inventory. Only after that extractor has produced explicit, source-backed V3
manifests may the operator run:
```text
scripts/compile_kb_source_packet.py
```
That compiler's output still requires separate review and guarded admission.
Do not fabricate V3 manifests from raw sections in this inventory helper.
Use `teleo-reconstruction-recovery` for restore/replay selection and
`teleo-kb-db-change-workflow` only for an explicitly authorized later review or
apply task.
## Claim Ceiling
The output proves deterministic local inventory of the exact copied UTF-8
files, bounded syntactic sectioning, bundle integrity, and an exact once-only
hash partition of extraction work for a fixed shard count. It does not prove an
atomic snapshot of concurrent directory changes, non-empty shards,
format-specific syntax, semantic extraction or extractor quality, claim
atomicity or correctness, database deduplication, review, admission,
application, binary-document extraction, or any live environment.

62
.agents/skills/teleo-kb-db-change-workflow/SKILL.md
View file

@ -39,6 +39,40 @@ A useful composed change must retain:
Do not flatten strategy, governance, concept maps, identity, and reasoning tools
into generic claim rows merely because the input document discusses them.
## V3 Source-To-Rebuild Route
For new reconstruction work, use this order:
1. Run `ops/run_local_corpus_knowledge_rebuild.py` to retain exact UTF-8
snapshots and section work items. Its output contains no semantic claims.
2. For a large corpus, run `ops/shard_corpus_extraction_work.py` to create
deterministic exact-once work partitions. Sharding schedules extraction; it
does not read source bodies or produce claims.
3. Have a separately pinned extractor produce an explicit
`livingip.kbSourceExtractionManifest.v3`. Every evidence quote must bind to
exact UTF-8 byte offsets in the retained extraction.
4. Run `scripts/compile_kb_source_packet.py` with the original artifact, exact
extracted text, manifest, and extractor specification. The result is
build-only and remains `pending_review`.
5. On a disposable PostgreSQL clone, stage with
`scripts/stage_normalized_proposal.py --compiled-v3-bundle ...`, then render
the exact rows with `scripts/kb_proposal_review_packet.py`.
6. Record a real review separately with `scripts/approve_proposal.py` and exact
`--reviewed-by m3taversal`. Never use test-entered review metadata as
evidence that `m3taversal` actually reviewed a production packet.
7. Apply only the exact UUID with `scripts/apply_worker.py`, `--enable`,
`--contract-version 3`, and `--proposal-id ...`; refuse rather than draining
another approved proposal from the queue.
8. Export the applied transition with
`scripts/export_kb_transition_replay_material.py`, assemble ordered V3
materials with `ops/build_v3_genesis_ledger_bundle.py`, and independently
replay with `ops/run_local_genesis_ledger_rebuild.py`.
This repository does not yet provide a general PDF/office extractor or an
automatic corpus-wide semantic extractor. For binary originals, retain their
exact bytes and pin the separately produced UTF-8 extraction plus extractor
specification before step 3.
Run the deterministic source-to-proposal canary:
```bash
@ -47,6 +81,17 @@ Run the deterministic source-to-proposal canary:
--output /tmp/leo-local-ingestion-proposal-canary-current.json
```
Run the complete disposable V3 source-to-clean-rebuild lifecycle:
```bash
.venv/bin/python -m pytest -q tests/test_v3_source_rebuild_lifecycle_postgres.py
```
This lifecycle uses a synthetic pinned extractor manifest and synthetic
operator-supplied review metadata. It proves the local machinery and exact
replay boundary, not semantic extraction quality, real human approval, or a
live database mutation.
Require the source, claim, evidence, and proposal UUIDs to link exactly; claim
and evidence text must be exact substrings of the source; status must remain
`pending_review`; the container must use network `none` and leave no volume or
@ -54,7 +99,12 @@ container behind.
## Canonical Apply Contract
The strict `approve_claim` v2 payload may create:
The strict `approve_claim` v2 contract remains supported for its retained
proofs. V2/V3 payloads may create the supported canonical rows for their exact
contract version; V3 additionally carries explicit provenance, access,
assessment, and admission fields.
The shared row families include:
- `public.claims`
- `public.sources`
@ -64,9 +114,19 @@ The strict `approve_claim` v2 payload may create:
The lifecycle is split across:
- `scripts/compile_kb_source_packet.py`: hash-bound source plus V3 semantic
manifest to a deterministic build-only bundle;
- `scripts/stage_normalized_proposal.py`: revalidate and stage an exact bundle
only on the named disposable database;
- `scripts/kb_proposal_review_packet.py`: read-only review of exact proposed
canonical rows;
- `scripts/kb_proposal_normalize.py`: fail-closed rich intent to strict payload;
- `scripts/approve_proposal.py`: `kb_review` approves the exact type/payload;
- `scripts/apply_proposal.py`: operator-only `kb_apply` writes and verifies;
- `scripts/apply_worker.py`: exact-ID, one-at-a-time V2/V3 worker wrapper;
- `scripts/export_kb_transition_replay_material.py`: read-only applied-transition
exporter;
- `ops/build_v3_genesis_ledger_bundle.py`: V3-only ordered ledger assembler;
- `scripts/kb_apply_prereqs.sql`: roles, immutable approval row, gate functions,
ownership, indexes, and ACLs;
- `scripts/run_approve_claim_isolated_container_canary.sh`: disposable

1
.agents/skills/teleo-leo-onboarding/SKILL.md
View file

@ -90,6 +90,7 @@ Run the deterministic isolated acceptance canary with:
| GCP and Cloud SQL | `teleo-gcp-parity-ops` | Current main evidence; keep open PR #148 candidate-only |
| Hermes packaging/runtime | `nousresearch-hermes-agent` | `hermes-agent/` plus the live service-specific ops skill |
| Database provenance | `teleo-infra-provenance` and `teleo-db-operator` | Separate Git, runtime, SQLite, Postgres, and retained proof |
| Local corpus inventory and extraction preparation | `teleo-corpus-knowledge-rebuild` | Exact UTF-8 snapshots and section work items; semantic claims are a separate reviewed step |
| Source ingestion and proposal/apply | `teleo-kb-db-change-workflow` | Hash-bound source, review, guarded apply, receipt |
| Reconstruction and recovery | `teleo-reconstruction-recovery` | `docs/kb-rebuild-and-recompile.md` |
| Identity and rendered soul | `working-leo-m3taversal-outcomes` | Canonical identity rows before `SOUL.md` |

5
.agents/skills/teleo-leo-onboarding/scripts/run_clean_context_canary.py
View file

@ -48,6 +48,11 @@ SEMANTIC_CONTRACT = {
"PR #148 is a separate, open GCP least-privilege candidate",
"PR #148 open candidate only; not canonical main",
),
"local_corpus_source_inventory": (
"teleo-corpus-knowledge-rebuild",
"It does not perform semantic claim extraction.",
"deterministic source snapshots and extraction work items; no claims, proposals, or DB writes",
),
"next_safe_action": (
"teleo-leo-onboarding",
".agents/skills/teleo-leo-onboarding/scripts/validate_skill_pack.py --root .",

28
.agents/skills/teleo-reconstruction-recovery/SKILL.md
View file

@ -30,7 +30,10 @@ as canonical `main` recovery instructions until it merges.
| Need | Path | Honest result |
|---|---|---|
| Restore the latest verified state quickly | `ops/run_local_canonical_postgres_rebuild.py` | Exact snapshot recovery |
| Snapshot a Forgejo/text corpus for extraction | `ops/run_local_corpus_knowledge_rebuild.py` | Content-addressed UTF-8 sources and work items; no claims |
| Partition verified extraction work | `ops/shard_corpus_extraction_work.py` | Deterministic exact-once work shards; no source interpretation |
| Prove strict post-genesis applies replay exactly | `ops/run_local_genesis_ledger_rebuild.py` | Isolated insert-only ledger replay |
| Assemble reviewed V3 replay materials | `ops/build_v3_genesis_ledger_bundle.py` | Hash-pinned V3 ledger; optional operator-supplied admission metadata |
| Turn one retained source into a reviewed packet | `scripts/compile_kb_source_packet.py` | Deterministic proposal packet, not canonical knowledge |
| Rebuild every row from original sources | Follow the semantic recompilation contract in `docs/kb-rebuild-and-recompile.md` | Partial until every row and receipt is accounted for |
| Diagnose a live incident before recovery | Use `.agents/skills/teleo-vps-runtime-ops/SKILL.md` and `.agents/skills/teleo-infra-provenance/SKILL.md` | Fresh read-only incident map |
@ -40,12 +43,18 @@ as canonical `main` recovery instructions until it merges.
1. Identify the intended source database, snapshot, manifest, commit, and hashes.
2. Keep private dumps, row payloads, source excerpts, and replay material outside
the repository and mode `0600`.
3. Run the selected path in an isolated, network-disabled local container.
4. Verify schema, constraints, roles, counts, row hashes, key queries, and exact
3. Inventory text sources before semantic extraction. For a binary original,
retain its exact bytes and provide text from a separately pinned extractor;
this repository does not yet supply a general PDF/office extractor.
4. Build and independently verify the V3 ledger before running genesis replay.
Admission metadata is operator-supplied and is not a cryptographic signature
from `m3taversal`.
5. Run the selected path in an isolated, network-disabled local container.
6. Verify schema, constraints, roles, counts, row hashes, key queries, and exact
cleanup.
5. Run the relevant focused tests, then the unseen-reasoning verifier when the
7. Run the relevant focused tests, then the unseen-reasoning verifier when the
restored state is meant to support Leo answers.
6. Separate local reconstruction from VPS restore, GCP restore, service restart,
8. Separate local reconstruction from VPS restore, GCP restore, service restart,
Telegram delivery, promotion, and production apply. Each is its own proof row.
## Commands
@ -54,9 +63,16 @@ Use the repository virtual environment documented in `README.md`:
```bash
.venv/bin/python ops/run_local_canonical_postgres_rebuild.py --help
.venv/bin/python ops/run_local_corpus_knowledge_rebuild.py --help
.venv/bin/python ops/shard_corpus_extraction_work.py --help
.venv/bin/python ops/build_v3_genesis_ledger_bundle.py --help
.venv/bin/python ops/run_local_genesis_ledger_rebuild.py --help
.venv/bin/python scripts/compile_kb_source_packet.py --help
.venv/bin/python -m pytest -q \
tests/test_run_local_corpus_knowledge_rebuild.py \
tests/test_shard_corpus_extraction_work.py \
tests/test_v3_source_rebuild_lifecycle_postgres.py \
tests/test_build_v3_genesis_ledger_bundle.py \
tests/test_run_local_canonical_postgres_rebuild.py \
tests/test_run_local_genesis_ledger_rebuild.py \
tests/test_verify_leo_unseen_reasoning_chain.py
@ -76,6 +92,10 @@ would promote GCP or mutate production without exact authorization.
## Claim Ceiling
- Exact snapshot recovery is working when its parity receipt passes.
- Corpus inventory proves exact copied text bytes and extraction work locations,
not semantic claims or binary extraction.
- The V3 ledger builder accepts only exact source-normalized applied materials,
selects every ordered entry, and never infers admission from proposal status.
- The merged genesis-plus-ledger path proves an isolated insert-only slice.
- Semantic source-to-blank-database recompilation remains incomplete until every
canonical row traces to a genesis record or reviewed replay receipt.

2
.github/workflows/ci.yml vendored
View file

@ -160,6 +160,8 @@ jobs:
timeout-minutes: 20
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}

141
docs/kb-rebuild-and-recompile.md
View file

@ -96,6 +96,104 @@ The remaining gaps are concrete rather than mysterious:
This proves that most of the initial database came from the retained file-KB
import path. It does not yet prove a clean blank-database recompile.
## Genesis-Backed Source Lifecycle: Working T2 Canary
`scripts/run_leo_integrated_learning_lifecycle.py` can now start from one
retained real genesis dump rather than its legacy synthetic fixture schema. The
genesis mode is all-or-nothing: the dump, parity manifest, repository-pinned
trusted exporter receipt, and a digest-pinned PostgreSQL image are mandatory.
Run:
```bash
python3 scripts/run_leo_integrated_learning_lifecycle.py \
--source-packet /private/path/source-packet \
--genesis-dump /private/path/genesis.dump \
--genesis-manifest /private/path/genesis-manifest.jsonl \
--genesis-exporter-receipt /private/path/exporter-receipt.json \
--image docker.io/library/postgres:16-alpine@sha256:57c72fd2a128e416c7fcc499958864df5301e940bca0a56f58fddf30ffc07777 \
--review-action approve \
--review-note "Approved exact retained payload for disposable T2 verification only." \
--output /private/path/leo-genesis-lifecycle-receipt.json
```
Before the first Docker command, the runner copies the dump, manifest, exporter
receipt, source packet, controller, subprocess engines, and their transitive
repository imports into one mode `0500` private staging directory with mode
`0400` files. It validates the custom-dump signature; the exact
repository-pinned exporter-receipt hash, source identity, capture authorization,
and provenance binding; exporter-bound dump and manifest bytes; manifest
contract; image digest; executing Git commit; complete engine dependency hash
set; and a fresh deterministic compile against the retained
`proposal-packet.json`. Every enumerated engine file must be tracked at that
commit, its working-tree bytes must equal the commit blob, and its staged hash
must equal the same commit-bound hash. The compiler performs no model calls. Any
untrusted or incomplete exporter contract, moving image tag, engine drift,
staged-byte drift, invalid manifest, retained-packet drift, or output/input
collision fails before a container can start.
The child tools consume staged paths, and both runs bind their consumed file
set, engine set, and commit back to the same preflight manifest. The already
running Python controller imports repository modules before it creates staging;
the proof therefore establishes commit-matched imported dependencies and
byte-identical staged copies, not a claim that the controller process itself was
re-executed from staging. Successful public receipts retain logical names,
hashes, counts, and cleanup evidence without host-specific absolute paths.
Each of two independent runs then:
1. starts a uniquely labeled, networkless PostgreSQL container with tmpfs data;
2. creates only manifest-declared global roles, restores the supplied custom
dump directly, replays captured database ACLs, and proves the full parity
manifest before installing the current guarded-apply prerequisites;
3. skips the synthetic schema bootstrap and performs no canonical preseed;
4. proves the strict proposal, immutable approval, and every payload-owned
claim/source/evidence/edge/tool target are absent before staging;
5. stages `pending_review`, explicitly approves as the independent review role,
and drives the real apply worker through an exact-candidate report-only tick,
an enabled pre-commit failure with persisted retry state, a successful
password-authenticated retry with a mode `0600` replay receipt, and a
post-apply no-op tick;
6. recompiles the retained packet identically and fingerprints only fields
controlled by its immutable apply payload; and
7. removes the container and private temporary root, then proves its exact
instance and shared two-run label have no remaining containers.
The two-run comparison excludes generated evidence/edge IDs, timestamps,
container identity, and the PostgreSQL system identifier. It requires the
source hashes, proposal and payload identities, phase states, restored key
counts, row counts, recompile hash, and payload-controlled row projections to
match exactly.
Older retained parity manifests may predate explicit
`application_roles[].settings` and `database_settings` arrays. Their raw bytes
and trusted exporter binding remain mandatory. For role creation and current
parity SQL, the runner narrowly projects missing arrays from the captured
`default_transaction_read_only` field (or an empty list), without changing the
retained bytes. Such a run may prove data/schema parity and captured role
attributes, but its receipt reports full role/global parity as `not_proven`.
Only a source manifest that actually captured both settings arrays can earn the
full role/global parity claim.
The focused Docker regression constructs a real custom-format genesis with
nondefault role and per-database settings, restores it twice, executes the real
worker lifecycle, compares post-apply fingerprints, and verifies container and
staging cleanup:
```bash
pytest -q \
tests/test_run_leo_integrated_learning_lifecycle.py \
-k test_real_docker_genesis_restore_worker_parity_and_repeatability
```
The detailed receipt can contain proposal payloads and canonical row excerpts.
It is written atomically as mode `0600` and must remain private. Standard output
contains only the artifact/status, receipt basename and SHA-256, aggregate
problem/check counts, and error type. A passing receipt proves T2 disposable
runtime reconstruction only. It does not prove or permit VPS/GCP mutation,
production access, live Telegram behavior, or a blank-schema rebuild of all
historical knowledge.
## Target Compiler
The durable rebuild model is:
@ -148,6 +246,31 @@ not retain every column of the proposal ledger, so exact reconstruction also
needs the full approved proposal row, immutable approval snapshot, and final
applied proposal row.
New approvals retain the exact post-approval `kb_proposals` row inside
`kb_proposal_approvals`, including `updated_at`. After the worker applies and
writes its replay receipt, export the seven-field private material without any
pre-apply filesystem capture:
```bash
python3 scripts/export_kb_transition_replay_material.py <proposal-uuid> \
--replay-receipt /private/path/<proposal-uuid>.json \
--sequence 1 \
--output /private/path/entry-0001.json
```
The exporter uses the restricted `kb_apply` read function in a read-only
transaction, validates exact approved/approval/applied identities and the
worker receipt, and publishes a mode-`0600` file once. An identical rerun is a
no-op; drift or output aliasing is refused. Already-applied legacy approvals
whose exact approved row was never retained remain explicitly unavailable.
The rollback boundary is this transition capability only. Stop review/apply
workers, revoke and drop
`kb_stage.export_applied_proposal_replay_rows(uuid)`, then deploy the prior
revision and rerun its `kb_apply_prereqs.sql` before restarting workers. Keep
the nullable `approved_proposal_snapshot` column and any captured rows as inert
evidence; dropping them is neither required nor safe for rollback.
## Genesis Plus Strict Ledger: Working Deterministic Slice
`ops/run_local_genesis_ledger_rebuild.py` now executes the first exact
@ -251,6 +374,24 @@ The exact v1 claim ceiling is intentionally bounded:
- this proves only an isolated local reconstruction. It does not touch or prove
VPS, GCP, Telegram, a live database, or blank-schema source recompilation.
V3 `approve_claim` receipts extend the same exact-ledger boundary to admitted
claim versions, content-addressed source versions, located evidence, typed
edges, and evidence assessments. Ledger order is also the schema-cutover order:
all V2 entries must replay before the first V3 entry, the hash-pinned V3 contract
is installed once at that boundary, guarded prerequisites are rerun, and any V2
entry after cutover fails before replay. Historical V2 proposals already present
in genesis remain visible; the reconstruction does not silently apply, delete,
or reinterpret them.
Receipt-pinned V3 rows are seeded before the idempotent guarded apply, then read
back exactly. PostgreSQL generates the proposal's apply timestamp, so the
disposable reconstruction takes an exclusive proposal-table lock, verifies the
exact V3 decision trigger is enabled, transactionally disables only that trigger,
normalizes the receipt-pinned proposal timestamps as `postgres`, re-enables the
trigger, and verifies it is enabled before commit. Any failure rolls the trigger
state and timestamp update back together. This is a disposable reconstruction
mechanism, not a production write path or a relaxation of the live V3 contract.
The transition contract for `revise_strategy` is final-state deterministic, not
a claim that the v1 receipt independently contains a historical before-image:

14
docs/reports/leo-working-state-20260709/skill-pack-manifest.json
View file

@ -1,7 +1,7 @@
{
"pack": "leo-teleo-skill-pack",
"created_date": "2026-07-09",
"last_updated_utc": "2026-07-15T00:16:53Z",
"last_updated_utc": "2026-07-18T22:02:36Z",
"status": "repo_native_path_validated",
"contains_secrets": false,
"production_mutation_authorized": false,
@ -46,6 +46,11 @@
"role": "Snapshot recovery, genesis-ledger replay, source recompilation, and incident recovery",
"path": ".agents/skills/teleo-reconstruction-recovery/SKILL.md"
},
{
"name": "teleo-corpus-knowledge-rebuild",
"role": "Deterministic UTF-8 source snapshots and semantic-extraction work indexes; no claim generation",
"path": ".agents/skills/teleo-corpus-knowledge-rebuild/SKILL.md"
},
{
"name": "leo-telegram-canary-ops",
"role": "Live Telegram bot testing through authenticated Chrome and Computer Use",
@ -103,9 +108,9 @@
"gcp": ["teleo-gcp-parity-ops"],
"hermes_runtime": ["nousresearch-hermes-agent", "teleo-vps-runtime-ops"],
"database_provenance": ["teleo-infra-provenance", "teleo-db-operator"],
"ingestion": ["teleo-kb-db-change-workflow", "living-ip-kb-interop"],
"ingestion": ["teleo-corpus-knowledge-rebuild", "teleo-kb-db-change-workflow", "living-ip-kb-interop"],
"proposal_apply": ["teleo-kb-db-change-workflow"],
"reconstruction": ["teleo-reconstruction-recovery"],
"reconstruction": ["teleo-corpus-knowledge-rebuild", "teleo-reconstruction-recovery"],
"identity": ["teleo-leo-onboarding", "working-leo-m3taversal-outcomes"],
"testing": ["teleo-leo-onboarding", "teleo-proof-handoff", "crabbox", "decision-engine-refinement"],
"security": ["teleo-vps-runtime-ops", "teleo-gcp-parity-ops", "private-password-storage"],
@ -134,6 +139,9 @@
".agents/skills/teleo-leo-onboarding/scripts/run_clean_context_canary.py",
".agents/skills/teleo-leo-onboarding/scripts/validate_skill_pack.py",
"ops/run_local_canonical_postgres_rebuild.py",
"ops/run_local_corpus_knowledge_rebuild.py",
"ops/shard_corpus_extraction_work.py",
"ops/build_v3_genesis_ledger_bundle.py",
"ops/run_local_genesis_ledger_rebuild.py",
"ops/capture_vps_canonical_postgres_snapshot.py",
"ops/restore_gcp_generated_postgres_snapshot.py",

18
hermes-agent/leoclean-bin/kb_tool.py
View file

@ -374,9 +374,23 @@ def sql_array(values: list[str], cast: str = "text") -> str:
def psql_json(args: argparse.Namespace, sql: str) -> list[dict[str, Any]]:
if args.local:
command = ["docker", "exec", "-i", args.container, "psql", "-U", "postgres", "-d", args.db, "-At", "-q"]
command = [
"docker",
"exec",
"-i",
args.container,
"psql",
"-U",
"postgres",
"-d",
args.db,
"-At",
"-q",
"-v",
"ON_ERROR_STOP=1",
]
else:
remote = f"docker exec -i {args.container} psql -U postgres -d {args.db} -At -q"
remote = f"docker exec -i {args.container} psql -U postgres -d {args.db} -At -q -v ON_ERROR_STOP=1"
command = ["ssh", "-o", "ConnectTimeout=15", "-o", "BatchMode=yes", args.ssh, remote]
result = subprocess.run(command, text=True, capture_output=True, input=sql, check=False)
if result.returncode != 0:

1101
ops/build_v3_genesis_ledger_bundle.py Normal file
View file

File diff suppressed because it is too large Load diff

357
ops/rollback_teleo_v3_epistemic_contract.sql Normal file
View file

@ -0,0 +1,357 @@
-- Schema-only rollback for ops/teleo_v3_epistemic_contract.sql.
--
-- The rollback refuses to discard proposal-linked V3 data. A disposable
-- verifier may delete its own canary rows first; a real migration needs an
-- explicit export/port decision before this file can run.
\set ON_ERROR_STOP on
set timezone = 'UTC';
begin;
do $ownership_guard$
declare
v_artifacts_present boolean;
v_ledger_owner_valid boolean;
v_ledger_row_valid boolean;
v_missing_objects text;
begin
select
pg_catalog.to_regclass('kb_stage.teleo_v3_installation_ownership') is not null
or pg_catalog.to_regclass('public.claim_evidence_assessments') is not null
or exists (
select 1
from information_schema.columns columns
where columns.table_schema = 'public'
and (columns.table_name, columns.column_name) in (
('claims', 'owner_agent_id'),
('claims', 'proposition'),
('claims', 'body_markdown'),
('claims', 'scope'),
('claims', 'access_scope'),
('claims', 'admission_state'),
('claims', 'revision_criteria'),
('claims', 'revision_kind'),
('claims', 'supersedes_id'),
('claims', 'accepted_by_proposal_id'),
('sources', 'canonical_url'),
('sources', 'storage_uri'),
('sources', 'content_hash'),
('sources', 'access_scope'),
('sources', 'ingestion_origin'),
('sources', 'provenance_status'),
('sources', 'source_class'),
('sources', 'auto_promotion_policy'),
('sources', 'accepted_by_proposal_id'),
('claim_evidence', 'excerpt'),
('claim_evidence', 'locator_json'),
('claim_evidence', 'source_content_hash'),
('claim_evidence', 'rationale'),
('claim_evidence', 'polarity'),
('claim_evidence', 'accepted_from_proposal_id'),
('claim_edges', 'relation_type'),
('claim_edges', 'rationale'),
('claim_edges', 'scope_or_conditions'),
('claim_edges', 'accepted_by_proposal_id')
)
)
or exists (
select 1
from pg_catalog.pg_proc procedure
join pg_catalog.pg_namespace namespace on namespace.oid = procedure.pronamespace
where namespace.nspname = 'kb_stage'
and procedure.proname like 'teleo\_v3\_%' escape '\'
)
or exists (
select 1
from pg_catalog.pg_trigger trigger_row
where not trigger_row.tgisinternal
and trigger_row.tgname like 'teleo\_v3\_%' escape '\'
)
or exists (
select 1
from pg_catalog.pg_constraint constraint_row
where constraint_row.conname like 'teleo\_v3\_%' escape '\'
)
or exists (
select 1
from pg_catalog.pg_class index_row
where index_row.relkind = 'i'
and index_row.relname like 'teleo\_v3\_%' escape '\'
)
into v_artifacts_present;
if pg_catalog.to_regclass('kb_stage.teleo_v3_installation_ownership') is null then
if v_artifacts_present then
raise exception using
errcode = '55000',
message = 'teleo_v3 rollback refused: V3-named objects lack exact installation ownership';
end if;
return;
end if;
select owner_role.rolname = 'kb_gate_owner'
into v_ledger_owner_valid
from pg_catalog.pg_class relation
join pg_catalog.pg_namespace namespace on namespace.oid = relation.relnamespace
join pg_catalog.pg_roles owner_role on owner_role.oid = relation.relowner
where namespace.nspname = 'kb_stage'
and relation.relname = 'teleo_v3_installation_ownership'
and relation.relkind in ('r', 'p');
if not coalesce(v_ledger_owner_valid, false) then
raise exception using
errcode = '55000',
message = 'teleo_v3 rollback refused: installation ownership ledger owner is invalid';
end if;
execute $ownership_query$
select count(*) = 1
and bool_and(
contract_name = 'livingip.teleo-v3-epistemic-contract'
and contract_version = 1
and installation_token = 'collision-free-install-v1'
)
from kb_stage.teleo_v3_installation_ownership
$ownership_query$ into v_ledger_row_valid;
if not coalesce(v_ledger_row_valid, false) then
raise exception using
errcode = '55000',
message = 'teleo_v3 rollback refused: installation ownership ledger is not exact';
end if;
with expected_columns(table_name, column_name) as (
values
('claims', 'owner_agent_id'),
('claims', 'proposition'),
('claims', 'body_markdown'),
('claims', 'scope'),
('claims', 'access_scope'),
('claims', 'admission_state'),
('claims', 'revision_criteria'),
('claims', 'revision_kind'),
('claims', 'supersedes_id'),
('claims', 'accepted_by_proposal_id'),
('sources', 'canonical_url'),
('sources', 'storage_uri'),
('sources', 'content_hash'),
('sources', 'access_scope'),
('sources', 'ingestion_origin'),
('sources', 'provenance_status'),
('sources', 'source_class'),
('sources', 'auto_promotion_policy'),
('sources', 'accepted_by_proposal_id'),
('claim_evidence', 'excerpt'),
('claim_evidence', 'locator_json'),
('claim_evidence', 'source_content_hash'),
('claim_evidence', 'rationale'),
('claim_evidence', 'polarity'),
('claim_evidence', 'accepted_from_proposal_id'),
('claim_edges', 'relation_type'),
('claim_edges', 'rationale'),
('claim_edges', 'scope_or_conditions'),
('claim_edges', 'accepted_by_proposal_id')
), missing(object_name) as (
select pg_catalog.format('%I.%I', expected.table_name, expected.column_name)
from expected_columns expected
where not exists (
select 1
from information_schema.columns columns
where columns.table_schema = 'public'
and columns.table_name = expected.table_name
and columns.column_name = expected.column_name
)
union all
select 'public.claim_evidence_assessments'
where pg_catalog.to_regclass('public.claim_evidence_assessments') is null
)
select pg_catalog.string_agg(object_name, ', ' order by object_name)
into v_missing_objects
from missing;
if v_missing_objects is not null then
raise exception using
errcode = '55000',
message = pg_catalog.format(
'teleo_v3 rollback refused: installation-owned object set is incomplete: %s',
v_missing_objects
);
end if;
end
$ownership_guard$;
do $data_guard$
declare
contract_rows_exist boolean := false;
begin
if to_regclass('public.claim_evidence_assessments') is not null then
execute 'select exists (select 1 from public.claim_evidence_assessments)'
into contract_rows_exist;
if contract_rows_exist then
raise exception using
errcode = '55000',
message = 'teleo_v3 rollback refused: claim_evidence_assessments contains V3 data';
end if;
end if;
if exists (
select 1 from information_schema.columns
where table_schema = 'public' and table_name = 'claim_edges'
and column_name = 'accepted_by_proposal_id'
) then
execute 'select exists (select 1 from public.claim_edges where accepted_by_proposal_id is not null)'
into contract_rows_exist;
if contract_rows_exist then
raise exception using
errcode = '55000',
message = 'teleo_v3 rollback refused: claim_edges contains proposal-linked V3 data';
end if;
end if;
if exists (
select 1 from information_schema.columns
where table_schema = 'public' and table_name = 'claim_evidence'
and column_name = 'accepted_from_proposal_id'
) then
execute 'select exists (select 1 from public.claim_evidence where accepted_from_proposal_id is not null)'
into contract_rows_exist;
if contract_rows_exist then
raise exception using
errcode = '55000',
message = 'teleo_v3 rollback refused: claim_evidence contains proposal-linked V3 data';
end if;
end if;
if exists (
select 1 from information_schema.columns
where table_schema = 'public' and table_name = 'sources'
and column_name = 'accepted_by_proposal_id'
) then
execute 'select exists (select 1 from public.sources where accepted_by_proposal_id is not null)'
into contract_rows_exist;
if contract_rows_exist then
raise exception using
errcode = '55000',
message = 'teleo_v3 rollback refused: sources contains proposal-linked V3 data';
end if;
end if;
if exists (
select 1 from information_schema.columns
where table_schema = 'public' and table_name = 'claims'
and column_name = 'admission_state'
) then
execute $query$
select exists (
select 1 from public.claims
where admission_state = 'admitted' or accepted_by_proposal_id is not null
)
$query$ into contract_rows_exist;
if contract_rows_exist then
raise exception using
errcode = '55000',
message = 'teleo_v3 rollback refused: claims contains admitted or proposal-linked V3 data';
end if;
end if;
end
$data_guard$;
drop trigger if exists teleo_v3_guard_linked_proposal_decision on kb_stage.kb_proposals;
drop trigger if exists teleo_v3_00_enforce_apply_contract_cutover on kb_stage.kb_proposals;
alter table kb_stage.kb_proposals
drop constraint if exists teleo_v3_proposal_apply_receipt_check;
drop trigger if exists teleo_v3_edge_immutable on public.claim_edges;
drop trigger if exists teleo_v3_edge_reject_delete on public.claim_edges;
drop trigger if exists teleo_v3_edge_validate_endpoints on public.claim_edges;
drop trigger if exists teleo_v3_edge_require_accepted_proposal on public.claim_edges;
drop trigger if exists teleo_v3_00_edge_require_linked_insert on public.claim_edges;
drop trigger if exists teleo_v3_evidence_immutable on public.claim_evidence;
drop trigger if exists teleo_v3_evidence_reject_delete on public.claim_evidence;
drop trigger if exists teleo_v3_evidence_validate_provenance on public.claim_evidence;
drop trigger if exists teleo_v3_evidence_require_accepted_proposal on public.claim_evidence;
drop trigger if exists teleo_v3_00_evidence_require_linked_insert on public.claim_evidence;
drop trigger if exists teleo_v3_source_immutable on public.sources;
drop trigger if exists teleo_v3_source_reject_delete on public.sources;
drop trigger if exists teleo_v3_source_require_accepted_proposal on public.sources;
drop trigger if exists teleo_v3_00_source_require_linked_insert on public.sources;
drop trigger if exists teleo_v3_claim_immutable on public.claims;
drop trigger if exists teleo_v3_claim_reject_delete on public.claims;
drop trigger if exists teleo_v3_claim_require_accepted_proposal on public.claims;
drop trigger if exists teleo_v3_00_claim_require_linked_insert on public.claims;
drop index if exists public.teleo_v3_edge_typed_adjacency_idx;
drop index if exists public.teleo_v3_evidence_claim_polarity_idx;
drop index if exists public.teleo_v3_assessment_one_successor_idx;
drop index if exists public.teleo_v3_claim_one_admitted_successor_idx;
drop table if exists public.claim_evidence_assessments;
alter table public.claim_edges
drop constraint if exists teleo_v3_edge_contract_check,
drop constraint if exists teleo_v3_edge_accepted_proposal_fkey,
drop column if exists relation_type,
drop column if exists rationale,
drop column if exists scope_or_conditions,
drop column if exists accepted_by_proposal_id;
alter table public.claim_evidence
drop constraint if exists teleo_v3_evidence_contract_check,
drop constraint if exists teleo_v3_evidence_accepted_proposal_fkey,
drop column if exists excerpt,
drop column if exists locator_json,
drop column if exists source_content_hash,
drop column if exists rationale,
drop column if exists polarity,
drop column if exists accepted_from_proposal_id;
alter table public.sources
drop constraint if exists teleo_v3_source_contract_check,
drop constraint if exists teleo_v3_source_accepted_proposal_fkey,
drop column if exists canonical_url,
drop column if exists storage_uri,
drop column if exists content_hash,
drop column if exists access_scope,
drop column if exists ingestion_origin,
drop column if exists provenance_status,
drop column if exists source_class,
drop column if exists auto_promotion_policy,
drop column if exists accepted_by_proposal_id;
alter table public.claims
drop constraint if exists teleo_v3_claim_admission_check,
drop constraint if exists teleo_v3_claim_accepted_proposal_fkey,
drop constraint if exists teleo_v3_claim_supersedes_fkey,
drop constraint if exists teleo_v3_claim_owner_agent_fkey,
drop column if exists owner_agent_id,
drop column if exists proposition,
drop column if exists body_markdown,
drop column if exists scope,
drop column if exists access_scope,
drop column if exists admission_state,
drop column if exists revision_criteria,
drop column if exists revision_kind,
drop column if exists supersedes_id,
drop column if exists accepted_by_proposal_id;
drop function if exists kb_stage.teleo_v3_guard_linked_proposal_decision();
drop function if exists kb_stage.teleo_v3_acquire_protected_relation_locks();
drop function if exists kb_stage.teleo_v3_enforce_apply_contract_cutover();
drop function if exists kb_stage.teleo_v3_validate_evidence_assessment();
drop function if exists kb_stage.teleo_v3_validate_claim_edge();
drop function if exists kb_stage.teleo_v3_validate_claim_evidence();
drop function if exists kb_stage.teleo_v3_guard_immutable_fields();
drop function if exists kb_stage.teleo_v3_reject_delete();
drop function if exists kb_stage.teleo_v3_require_linked_insert();
drop function if exists kb_stage.teleo_v3_require_accepted_proposal();
drop table if exists kb_stage.teleo_v3_installation_ownership;
commit;
select jsonb_build_object(
'contract', 'livingip.teleo-v3-epistemic-contract',
'version', 1,
'migration', 'rolled_back'
)::text;

1363
ops/run_local_corpus_knowledge_rebuild.py Executable file
View file

File diff suppressed because it is too large Load diff

726
ops/run_local_genesis_ledger_rebuild.py
View file

@ -2,7 +2,7 @@
"""Deterministically rebuild a disposable Postgres from genesis plus a strict ledger.
The v1 replay boundary supports strict ``add_edge``, ``attach_evidence``,
``approve_claim``, and ``revise_strategy`` receipts. Insert-only actions seed
V2/V3 ``approve_claim``, and ``revise_strategy`` receipts. Insert-only actions seed
their receipt-pinned rows before the guarded transition. ``revise_strategy``
instead captures the exact prestate identities, executes the exact hash-matched
guarded SQL, validates the generated delta, and normalizes only that delta to the
@ -41,20 +41,34 @@ if str(SCRIPTS_DIR) not in sys.path:
sys.path.insert(0, str(SCRIPTS_DIR))
import apply_proposal as apply_engine # noqa: E402, I001
import export_kb_transition_replay_material as replay_exporter # noqa: E402
import kb_apply_replay_receipt as replay_receipt # noqa: E402
LEDGER_ARTIFACT = "teleo_genesis_plus_ledger"
MATERIAL_ARTIFACT = "teleo_strict_proposal_replay_material"
ADMISSION_ARTIFACT = "teleo_ledger_admission_manifest"
LEDGER_CONTRACT_VERSION = 1
MATERIAL_CONTRACT_VERSION = 1
ADMISSION_CONTRACT_VERSION = 1
CANONICAL_REVIEWER_HANDLE = "m3taversal"
REVIEW_METADATA_PROVENANCE = "operator_supplied_explicit_review_metadata"
ADMISSION_BASES = frozenset(
{
"forgejo_primary_source",
"reviewed_telegram_steering",
"reviewed_primary_source",
}
)
ADMISSION_SCOPES = frozenset({"collective_core", "agent_identity", "knowledge_commons"})
DEFAULT_REBUILD_IMAGE = (
"docker.io/library/postgres:16-alpine@sha256:57c72fd2a128e416c7fcc499958864df5301e940bca0a56f58fddf30ffc07777"
)
SUPPORTED_PROPOSAL_TYPES = frozenset({"add_edge", "attach_evidence", "approve_claim", "revise_strategy"})
CLAIM_CEILING = (
"exact genesis plus ordered strict proposal replay; supported types are add_edge, "
"attach_evidence, approve_claim, and revise_strategy; mutating strategy replay "
"attach_evidence, V2/V3 approve_claim, and revise_strategy; V3 cutover is ordered and exact; "
"mutating strategy replay "
"captures prestate identities and normalizes the exact receipt-pinned poststate; "
"full approved/applied proposal rows and immutable approval snapshots are required"
)
@ -67,14 +81,17 @@ NOT_PROVEN = (
APPLY_ENGINE_PATH = SCRIPTS_DIR / "apply_proposal.py"
REPLAY_RECEIPT_ENGINE_PATH = SCRIPTS_DIR / "kb_apply_replay_receipt.py"
GUARD_PREREQUISITES_PATH = SCRIPTS_DIR / "kb_apply_prereqs.sql"
V3_CONTRACT_PATH = REPO_ROOT / "ops" / "teleo_v3_epistemic_contract.sql"
PARITY_SQL_PATH = REPO_ROOT / "ops" / "postgres_parity_manifest.sql"
BASE_REBUILD_PATH = REPO_ROOT / "ops" / "run_local_canonical_postgres_rebuild.py"
GUARD_PREREQUISITES_DESTINATION = "/tmp/kb-apply-prereqs.sql"
V3_CONTRACT_DESTINATION = "/tmp/teleo-v3-epistemic-contract.sql"
FIXED_ENGINE_PATHS = frozenset(
{
APPLY_ENGINE_PATH.resolve(),
REPLAY_RECEIPT_ENGINE_PATH.resolve(),
GUARD_PREREQUISITES_PATH.resolve(),
V3_CONTRACT_PATH.resolve(),
PARITY_SQL_PATH.resolve(),
BASE_REBUILD_PATH.resolve(),
Path(__file__).resolve(),
@ -83,6 +100,7 @@ FIXED_ENGINE_PATHS = frozenset(
SHA256_RE = re.compile(r"[0-9a-f]{64}\Z")
IMAGE_DIGEST_RE = re.compile(r"\S+@sha256:[0-9a-f]{64}\Z")
TYPED_TIMESTAMP_FIELDS = frozenset({"captured_at", "created_at", "updated_at", "reviewed_at", "applied_at"})
PROPOSAL_TRANSITION_FIELDS = frozenset(
{"status", "applied_by_handle", "applied_by_agent_id", "applied_at", "updated_at"}
)
@ -125,6 +143,7 @@ CANONICAL_TABLE_ORDER = (
"sources",
"reasoning_tools",
"claim_evidence",
"claim_evidence_assessments",
"claim_edges",
)
STRATEGY_REQUIRED_FIELDS = frozenset(
@ -173,6 +192,7 @@ class LedgerEntry:
material_path: Path
material_sha256: str
replay_material_sha256: str
contract_version: int
approved_proposal: dict[str, Any]
approval_snapshot: dict[str, Any]
applied_proposal: dict[str, Any]
@ -196,6 +216,10 @@ class ReconstructionBundle:
final_manifest_bytes: int
engine_hashes: dict[str, str]
entries: tuple[LedgerEntry, ...]
admission: dict[str, Any]
admission_manifest_path: Path | None
admission_manifest_sha256: str | None
admission_manifest_bytes: int | None
protected_paths: frozenset[Path]
@ -228,12 +252,18 @@ def _require_exact_keys(value: Any, expected: frozenset[str], field: str) -> dic
def _load_json_object(path: Path, field: str) -> dict[str, Any]:
try:
value = json.loads(path.read_text(encoding="utf-8"))
except (OSError, UnicodeError, json.JSONDecodeError) as exc:
raise ReconstructionError("invalid_json_artifact", f"{field} is not valid UTF-8 JSON") from exc
if not isinstance(value, dict):
raise ReconstructionError("invalid_json_artifact", f"{field} must contain one JSON object")
return value
raw = path.read_text(encoding="utf-8")
return replay_exporter._parse_json_object(raw, label=field)
except replay_exporter.ExportError as exc:
raise ReconstructionError(
"invalid_json_artifact",
f"{field} is not exact duplicate-free UTF-8 JSON",
) from exc
except (OSError, UnicodeError) as exc:
raise ReconstructionError(
"invalid_json_artifact",
f"{field} is not exact duplicate-free UTF-8 JSON",
) from exc
def _resolve_artifact(ledger_path: Path, declared: Any, field: str) -> Path:
@ -263,11 +293,183 @@ def _engine_hashes() -> dict[str, str]:
"apply_engine_sha256": APPLY_ENGINE_PATH,
"replay_receipt_engine_sha256": REPLAY_RECEIPT_ENGINE_PATH,
"guard_prerequisites_sha256": GUARD_PREREQUISITES_PATH,
"v3_contract_sha256": V3_CONTRACT_PATH,
"parity_sql_sha256": PARITY_SQL_PATH,
}
return {name: canonical_rebuild.sha256_file(path) for name, path in paths.items()}
def _proposal_contract_version(proposal: dict[str, Any]) -> int:
payload = proposal.get("payload")
apply_payload = payload.get("apply_payload") if isinstance(payload, dict) else None
if not isinstance(apply_payload, dict):
raise ReconstructionError("invalid_material", "proposal is missing a strict apply payload")
raw_version = apply_payload.get("contract_version", 2)
if type(raw_version) is not int or raw_version not in {2, 3}:
raise ReconstructionError("unsupported_contract_version", "proposal contract version is unsupported")
if raw_version == 3 and proposal.get("proposal_type") != "approve_claim":
raise ReconstructionError("unsupported_contract_version", "V3 replay currently supports approve_claim only")
return raw_version
def _validate_contract_order(entries: list[LedgerEntry]) -> None:
v3_started = False
for entry in entries:
if entry.contract_version == 3:
v3_started = True
elif v3_started:
raise ReconstructionError(
"contract_order_mismatch",
"V2 ledger entries cannot appear after the ordered V3 cutover",
)
def _load_admission_manifest(
args: argparse.Namespace,
*,
ledger_sha256: str,
entries: list[LedgerEntry],
) -> tuple[dict[str, Any], Path | None, str | None, int | None]:
manifest_path = getattr(args, "admission_manifest", None)
manifest_pin = getattr(args, "admission_manifest_sha256", None)
if manifest_path is None and manifest_pin is None:
return (
{
"status": "forensic_only",
"canonical_retrieval_eligible": False,
"reason": "no explicit admission manifest was supplied",
"entry_count": len(entries),
},
None,
None,
None,
)
if manifest_path is None or manifest_pin is None:
raise ReconstructionError(
"incomplete_admission_pin",
"admission requires both a manifest and its SHA-256 pin",
)
if not isinstance(manifest_path, Path) or not manifest_path.is_file():
raise ReconstructionError("missing_admission_manifest", "admission manifest is not an existing regular file")
manifest_path = manifest_path.resolve()
manifest_sha256 = _assert_file_hash(manifest_path, manifest_pin, "--admission-manifest-sha256")
manifest = _require_exact_keys(
_load_json_object(manifest_path, "admission manifest"),
frozenset(
{
"artifact",
"contract_version",
"ledger_sha256",
"reviewed_by_handle",
"reviewed_by_agent_id",
"reviewed_at",
"review_note",
"entries",
}
),
"admission manifest",
)
if manifest["artifact"] != ADMISSION_ARTIFACT or manifest["contract_version"] != ADMISSION_CONTRACT_VERSION:
raise ReconstructionError(
"unsupported_admission_contract",
"admission manifest is not the supported v1 contract",
)
if _validate_sha256(manifest["ledger_sha256"], "admission manifest ledger_sha256") != ledger_sha256:
raise ReconstructionError("admission_ledger_mismatch", "admission manifest does not select this exact ledger")
reviewer_handle = manifest["reviewed_by_handle"]
review_note = manifest["review_note"]
if reviewer_handle != CANONICAL_REVIEWER_HANDLE:
raise ReconstructionError(
"invalid_admission_review",
"admission reviewer handle must equal m3taversal exactly",
)
if not isinstance(review_note, str) or not review_note.strip():
raise ReconstructionError("invalid_admission_review", "admission review note must be non-empty")
try:
reviewer_agent_id = str(uuid.UUID(str(manifest["reviewed_by_agent_id"])))
reviewed_at = replay_receipt.canonical_utc_timestamp(
manifest["reviewed_at"],
path="admission manifest reviewed_at",
)
except (AttributeError, TypeError, ValueError) as exc:
raise ReconstructionError(
"invalid_admission_review",
"admission review requires one reviewer UUID and timezone-aware timestamp",
) from exc
raw_entries = manifest["entries"]
if not isinstance(raw_entries, list) or len(raw_entries) != len(entries):
raise ReconstructionError(
"admission_selection_mismatch",
"admission manifest must select every entry in the supplied clean-genesis ledger exactly once",
)
selected: list[dict[str, Any]] = []
for index, (raw_entry, entry) in enumerate(zip(raw_entries, entries, strict=True), 1):
selected_entry = _require_exact_keys(
raw_entry,
frozenset(
{
"sequence",
"proposal_id",
"material_sha256",
"replay_material_sha256",
"admission_scope",
"basis",
}
),
f"admission manifest entries[{index - 1}]",
)
if entry.contract_version != 3:
raise ReconstructionError(
"legacy_admission_forbidden",
"only source-backed V3 entries may enter an admitted clean genesis",
)
expected = {
"sequence": entry.sequence,
"proposal_id": entry.applied_proposal["id"],
"material_sha256": entry.material_sha256,
"replay_material_sha256": entry.replay_material_sha256,
}
if any(selected_entry[field] != value for field, value in expected.items()):
raise ReconstructionError(
"admission_selection_mismatch",
f"admission manifest entry {index} does not pin the exact ledger material",
)
scope = selected_entry["admission_scope"]
basis = selected_entry["basis"]
if scope not in ADMISSION_SCOPES or basis not in ADMISSION_BASES:
raise ReconstructionError(
"invalid_admission_classification",
f"admission manifest entry {index} has an unsupported scope or basis",
)
selected.append({**expected, "admission_scope": scope, "basis": basis})
return (
{
"status": "review_metadata_supplied",
"canonical_retrieval_eligible": False,
"reason": "operator-supplied review metadata is hash-pinned but not authenticated",
"artifact": ADMISSION_ARTIFACT,
"contract_version": ADMISSION_CONTRACT_VERSION,
"review_metadata_provenance": REVIEW_METADATA_PROVENANCE,
"cryptographically_authenticated": False,
"manifest_sha256": manifest_sha256,
"ledger_sha256": ledger_sha256,
"reviewed_by_handle": reviewer_handle.strip(),
"reviewed_by_agent_id": reviewer_agent_id,
"reviewed_at": reviewed_at,
"review_note_sha256": _sha256_text(review_note),
"entry_count": len(selected),
"entries": selected,
},
manifest_path,
manifest_sha256,
manifest_path.stat().st_size,
)
def _validate_proposal_rows(
approved: dict[str, Any],
applied: dict[str, Any],
@ -337,7 +539,25 @@ def _validate_proposal_rows(
"applied_by_agent_id",
"applied_at",
)
if any(receipt_proposal.get(field) != applied[field] for field in envelope_fields):
try:
envelope_mismatch = any(
replay_receipt.normalize_typed_value(
field,
receipt_proposal.get(field),
path=f"replay_receipt.proposal.{field}",
)
!= replay_receipt.normalize_typed_value(
field,
applied[field],
path=f"material.applied_proposal.{field}",
)
for field in envelope_fields
)
except ValueError as exc:
raise ReconstructionError(
"invalid_material", "replay receipt proposal envelope contains an invalid typed value"
) from exc
if envelope_mismatch:
raise ReconstructionError(
"material_mismatch", "replay receipt proposal envelope does not match applied proposal row"
)
@ -500,6 +720,7 @@ def _validate_entry_material(
raise ReconstructionError("invalid_replay_receipt", "replay receipt is missing apply or row material")
if proposal_type == "revise_strategy":
_validate_revise_strategy_receipt_rows(proposal, canonical_rows)
legacy_receipt = False
try:
rebuilt = replay_receipt.build_replay_receipt(
proposal,
@ -508,23 +729,34 @@ def _validate_entry_material(
apply_sql_source=apply_metadata.get("source"),
exported_at_utc=receipt.get("exported_at_utc"),
)
except (KeyError, TypeError, ValueError) as exc:
raise ReconstructionError(
"invalid_replay_receipt", "replay receipt failed strict payload and exact-row validation"
) from exc
except (KeyError, TypeError, ValueError):
legacy_receipt = True
try:
rebuilt = replay_exporter.validate_worker_replay_receipt(
receipt,
expected_proposal_id=material["applied_proposal"].get("id"),
)
except (KeyError, TypeError, ValueError, replay_exporter.ExportError) as exc:
raise ReconstructionError(
"invalid_replay_receipt", "replay receipt failed strict payload and exact-row validation"
) from exc
if receipt.get("hashes") != rebuilt["hashes"]:
if not legacy_receipt and receipt.get("hashes") != rebuilt.get("hashes"):
raise ReconstructionError("replay_receipt_hash_mismatch", "replay receipt hashes are not recoverable")
if receipt.get("expected_row_counts") != rebuilt["expected_row_counts"]:
if receipt.get("expected_row_counts") != rebuilt.get("expected_row_counts"):
raise ReconstructionError("replay_receipt_count_mismatch", "replay receipt expected counts are invalid")
if receipt.get("actual_row_counts") != rebuilt["actual_row_counts"]:
if receipt.get("actual_row_counts") != rebuilt.get("actual_row_counts"):
raise ReconstructionError("replay_receipt_count_mismatch", "replay receipt actual counts are invalid")
actual_replay_hash = rebuilt["hashes"]["replay_material_sha256"]
receipt_hashes = receipt.get("hashes")
if not isinstance(receipt_hashes, dict):
raise ReconstructionError("replay_receipt_hash_mismatch", "replay receipt hashes are not recoverable")
actual_replay_hash = receipt_hashes.get("replay_material_sha256")
if actual_replay_hash != expected_replay_hash:
raise ReconstructionError(
"replay_material_hash_mismatch", "ledger replay-material pin does not match the private receipt"
)
receipt = {**receipt, "canonical_rows": rebuilt["canonical_rows"]}
receipt = rebuilt
proposal = receipt["proposal"]
approved = material["approved_proposal"]
approval = material["approval_snapshot"]
@ -552,6 +784,7 @@ def _validate_entry_material(
material_path=material_path,
material_sha256=material_sha256,
replay_material_sha256=actual_replay_hash,
contract_version=_proposal_contract_version(proposal),
approved_proposal=approved,
approval_snapshot=approval,
applied_proposal=applied,
@ -644,6 +877,15 @@ def load_bundle(args: argparse.Namespace) -> ReconstructionBundle:
entries.append(entry)
protected_paths.add(material_path)
_validate_contract_order(entries)
admission, admission_path, admission_sha256, admission_bytes = _load_admission_manifest(
args,
ledger_sha256=ledger_sha256,
entries=entries,
)
if admission_path is not None:
protected_paths.add(admission_path)
if args.output.resolve() in protected_paths:
raise ReconstructionError("unsafe_output_path", "--output must not overwrite any reconstruction input")
@ -661,6 +903,10 @@ def load_bundle(args: argparse.Namespace) -> ReconstructionBundle:
final_manifest_bytes=final_manifest_path.stat().st_size,
engine_hashes=actual_engine_hashes,
entries=tuple(entries),
admission=admission,
admission_manifest_path=admission_path,
admission_manifest_sha256=admission_sha256,
admission_manifest_bytes=admission_bytes,
protected_paths=frozenset(protected_paths),
)
@ -669,7 +915,15 @@ def _jsonb_literal(value: Any) -> str:
return apply_engine.sql_literal(_canonical_json(value)) + "::jsonb"
def _approval_seed_row(approved_proposal: dict[str, Any], approval_snapshot: dict[str, Any]) -> dict[str, Any]:
return {
**approval_snapshot,
"approved_proposal_snapshot": approved_proposal,
}
def build_seed_sql(entry: LedgerEntry) -> str:
approval_seed = _approval_seed_row(entry.approved_proposal, entry.approval_snapshot)
statements = [
"begin;",
"set local standard_conforming_strings = on;",
@ -679,25 +933,30 @@ def build_seed_sql(entry: LedgerEntry) -> str:
") seeded",
"on conflict (id) do nothing;",
"insert into kb_stage.kb_proposal_approvals",
"select seeded.* from jsonb_populate_record(",
f" null::kb_stage.kb_proposal_approvals, {_jsonb_literal(entry.approval_snapshot)}",
"select seeded.*",
"from kb_stage.kb_proposals proposal",
"cross join lateral jsonb_populate_record(",
" null::kb_stage.kb_proposal_approvals,",
f" jsonb_set({_jsonb_literal(approval_seed)}, '{{approved_proposal_snapshot}}', to_jsonb(proposal), true)",
") seeded",
f"where proposal.id = {apply_engine.sql_literal(entry.approved_proposal['id'])}::uuid",
"on conflict (proposal_id) do nothing;",
]
rows = entry.receipt["canonical_rows"]
for table in CANONICAL_TABLE_ORDER:
table_rows = rows.get(table)
if not table_rows:
continue
statements.extend(
[
f"insert into public.{table}",
"select seeded.* from jsonb_populate_recordset(",
f" null::public.{table}, {_jsonb_literal(table_rows)}",
") seeded",
"on conflict do nothing;",
]
)
if entry.contract_version != 3:
rows = entry.receipt["canonical_rows"]
for table in CANONICAL_TABLE_ORDER:
table_rows = rows.get(table)
if not table_rows:
continue
statements.extend(
[
f"insert into public.{table}",
"select seeded.* from jsonb_populate_recordset(",
f" null::public.{table}, {_jsonb_literal(table_rows)}",
") seeded",
"on conflict do nothing;",
]
)
statements.append("commit;")
return "\n".join(statements) + "\n"
@ -967,14 +1226,163 @@ def build_proposal_readback_sql(proposal_id: str) -> str:
return f"""select jsonb_build_object(
'proposal', (select to_jsonb(p) from kb_stage.kb_proposals p
where p.id = {literal}::uuid),
'approval_snapshot', (select to_jsonb(a) from kb_stage.kb_proposal_approvals a
where a.proposal_id = {literal}::uuid)
'approval_snapshot', (select to_jsonb(a) - 'approved_proposal_snapshot'
from kb_stage.kb_proposal_approvals a
where a.proposal_id = {literal}::uuid),
'approved_proposal_snapshot', (select a.approved_proposal_snapshot
from kb_stage.kb_proposal_approvals a
where a.proposal_id = {literal}::uuid)
)::text;
"""
def build_v3_canonical_timestamp_normalization_sql(entry: LedgerEntry) -> str:
if entry.contract_version != 3:
raise ReconstructionError(
"invalid_v3_normalization",
"canonical V3 timestamp normalization requires a V3 ledger entry",
)
proposal_id = entry.applied_proposal["id"]
table_contracts = {
"claims": ("accepted_by_proposal_id", "teleo_v3_claim_immutable"),
"sources": ("accepted_by_proposal_id", "teleo_v3_source_immutable"),
"claim_evidence": ("accepted_from_proposal_id", "teleo_v3_evidence_immutable"),
"claim_evidence_assessments": (
"accepted_by_proposal_id",
"teleo_v3_assessment_immutable",
),
"claim_edges": ("accepted_by_proposal_id", "teleo_v3_edge_immutable"),
}
statements = ["begin;", "set local standard_conforming_strings = on;"]
for table in CANONICAL_TABLE_ORDER:
rows = entry.receipt["canonical_rows"].get(table) or []
if not rows:
continue
if table not in table_contracts or any("id" not in row for row in rows):
raise ReconstructionError(
"invalid_v3_normalization",
"V3 canonical timestamp rows lack an exact proposal-owned identity",
)
timestamp_fields = sorted(TYPED_TIMESTAMP_FIELDS.intersection(*(set(row) for row in rows)))
if not timestamp_fields:
continue
proposal_column, immutable_trigger = table_contracts[table]
assignments = ",\n ".join(f"{field} = pinned.{field}" for field in timestamp_fields)
statements.append(
f"""lock table public.{table} in access exclusive mode;
do $preflight_{table}$
begin
if not exists (
select 1
from pg_catalog.pg_trigger trigger
where trigger.tgrelid = 'public.{table}'::regclass
and trigger.tgname = '{immutable_trigger}'
and trigger.tgfoid = 'kb_stage.teleo_v3_guard_immutable_fields()'::regprocedure
and not trigger.tgisinternal
and trigger.tgenabled = 'O'
) then
raise exception 'V3 {table} immutable guard is not enabled before deterministic normalization';
end if;
end
$preflight_{table}$;
alter table public.{table} disable trigger {immutable_trigger};
do $normalize_{table}$
declare
changed_rows integer;
begin
update public.{table} target
set {assignments}
from jsonb_populate_recordset(
null::public.{table}, {_jsonb_literal(rows)}
) pinned
where target.id = pinned.id
and target.{proposal_column} = {apply_engine.sql_literal(proposal_id)}::uuid;
get diagnostics changed_rows = row_count;
if changed_rows <> {len(rows)} then
raise exception 'deterministic V3 {table} timestamp normalization failed';
end if;
end
$normalize_{table}$;
alter table public.{table} enable trigger {immutable_trigger};
do $postflight_{table}$
begin
if not exists (
select 1
from pg_catalog.pg_trigger trigger
where trigger.tgrelid = 'public.{table}'::regclass
and trigger.tgname = '{immutable_trigger}'
and trigger.tgfoid = 'kb_stage.teleo_v3_guard_immutable_fields()'::regprocedure
and not trigger.tgisinternal
and trigger.tgenabled = 'O'
) then
raise exception 'V3 {table} immutable guard was not restored after deterministic normalization';
end if;
end
$postflight_{table}$;"""
)
statements.append("commit;")
return "\n".join(statements) + "\n"
def build_applied_timestamp_normalization_sql(entry: LedgerEntry) -> str:
applied = entry.applied_proposal
if entry.contract_version == 3:
return f"""begin;
lock table kb_stage.kb_proposals in access exclusive mode;
do $preflight$
begin
if not exists (
select 1
from pg_catalog.pg_trigger trigger
where trigger.tgrelid = 'kb_stage.kb_proposals'::regclass
and trigger.tgname = 'teleo_v3_guard_linked_proposal_decision'
and trigger.tgfoid = 'kb_stage.teleo_v3_guard_linked_proposal_decision()'::regprocedure
and not trigger.tgisinternal
and trigger.tgenabled = 'O'
) then
raise exception 'V3 proposal decision guard is not enabled before deterministic normalization';
end if;
end
$preflight$;
alter table kb_stage.kb_proposals
disable trigger teleo_v3_guard_linked_proposal_decision;
do $normalization$
declare
changed_rows integer;
begin
update kb_stage.kb_proposals
set applied_at = {apply_engine.sql_literal(applied["applied_at"])}::timestamptz,
updated_at = {apply_engine.sql_literal(applied["updated_at"])}::timestamptz
where id = {apply_engine.sql_literal(applied["id"])}::uuid
and status = 'applied'
and applied_by_handle = {apply_engine.sql_literal(applied["applied_by_handle"])}
and applied_by_agent_id is not distinct from
{apply_engine.sql_literal(applied["applied_by_agent_id"])}::uuid;
get diagnostics changed_rows = row_count;
if changed_rows <> 1 then
raise exception 'deterministic V3 proposal timestamp normalization failed';
end if;
end
$normalization$;
alter table kb_stage.kb_proposals
enable trigger teleo_v3_guard_linked_proposal_decision;
do $postflight$
begin
if not exists (
select 1
from pg_catalog.pg_trigger trigger
where trigger.tgrelid = 'kb_stage.kb_proposals'::regclass
and trigger.tgname = 'teleo_v3_guard_linked_proposal_decision'
and trigger.tgfoid = 'kb_stage.teleo_v3_guard_linked_proposal_decision()'::regprocedure
and not trigger.tgisinternal
and trigger.tgenabled = 'O'
) then
raise exception 'V3 proposal decision guard was not restored after deterministic normalization';
end if;
end
$postflight$;
commit;
"""
return f"""do $reconstruction$
declare
changed_rows integer;
@ -1089,17 +1497,191 @@ def _read_json(
raise ReconstructionError(code, f"{action} returned an invalid private JSON readback") from exc
def install_v3_contract(args: argparse.Namespace, container: str) -> dict[str, Any]:
completed = canonical_rebuild._run(
[
args.docker_bin,
"cp",
str(V3_CONTRACT_PATH),
f"{container}:{V3_CONTRACT_DESTINATION}",
],
timeout=args.command_timeout,
)
_require_command_success(
completed,
code="v3_contract_copy_failed",
action="V3 epistemic contract copy",
)
cutover = _read_json(
args,
container,
"""begin;
alter role kb_apply nologin;
do $cutover$
begin
if exists (
select 1
from pg_catalog.pg_stat_activity activity
where activity.datid = (select oid from pg_catalog.pg_database where datname = current_database())
and activity.usename = 'kb_apply'
and activity.pid <> pg_catalog.pg_backend_pid()
) then
raise exception 'V3 reconstruction cutover found an active kb_apply session';
end if;
end
$cutover$;
commit;
select jsonb_build_object(
'apply_role_login_disabled', not (select role.rolcanlogin from pg_catalog.pg_roles role where role.rolname = 'kb_apply'),
'active_apply_sessions', (
select count(*)
from pg_catalog.pg_stat_activity activity
where activity.datid = (select oid from pg_catalog.pg_database where datname = current_database())
and activity.usename = 'kb_apply'
and activity.pid <> pg_catalog.pg_backend_pid()
)
)::text;
""",
code="v3_cutover_preflight_failed",
action="V3 disposable writer cutover preflight",
)
if cutover != {"apply_role_login_disabled": True, "active_apply_sessions": 0}:
raise ReconstructionError(
"v3_cutover_preflight_failed",
"V3 disposable writer cutover did not disable login with zero active apply sessions",
)
for destination, label in (
(V3_CONTRACT_DESTINATION, "V3 epistemic contract"),
(GUARD_PREREQUISITES_DESTINATION, "guarded apply prerequisites after V3"),
):
completed = canonical_rebuild._run(
[
args.docker_bin,
"exec",
container,
"psql",
"-X",
"-U",
"postgres",
"-d",
args.database,
"-Atq",
"-v",
"ON_ERROR_STOP=1",
"-f",
destination,
],
timeout=args.apply_timeout,
)
_require_command_success(
completed,
code="v3_contract_install_failed",
action=label,
)
catalog = _read_json(
args,
container,
"""select jsonb_build_object(
'assessment_table_present', to_regclass('public.claim_evidence_assessments') is not null,
'apply_role_login_restored', (select role.rolcanlogin from pg_catalog.pg_roles role where role.rolname = 'kb_apply'),
'cutover_guard_enabled', exists (
select 1 from pg_catalog.pg_trigger trigger
where trigger.tgrelid = 'kb_stage.kb_proposals'::regclass
and trigger.tgname = 'teleo_v3_00_enforce_apply_contract_cutover'
and trigger.tgfoid = 'kb_stage.teleo_v3_enforce_apply_contract_cutover()'::regprocedure
and not trigger.tgisinternal
and trigger.tgenabled = 'O'
),
'decision_guard_enabled', exists (
select 1 from pg_catalog.pg_trigger trigger
where trigger.tgrelid = 'kb_stage.kb_proposals'::regclass
and trigger.tgname = 'teleo_v3_guard_linked_proposal_decision'
and trigger.tgfoid = 'kb_stage.teleo_v3_guard_linked_proposal_decision()'::regprocedure
and not trigger.tgisinternal
and trigger.tgenabled = 'O'
)
)::text;
""",
code="v3_contract_readback_failed",
action="V3 epistemic contract catalog readback",
)
if catalog != {
"assessment_table_present": True,
"apply_role_login_restored": True,
"cutover_guard_enabled": True,
"decision_guard_enabled": True,
}:
raise ReconstructionError("v3_contract_readback_failed", "V3 contract catalog readback was incomplete")
return {
"status": "pass",
"sha256": canonical_rebuild.sha256_file(V3_CONTRACT_PATH),
"cutover": cutover,
"guard_prerequisites_rerun": True,
"catalog": catalog,
}
def _assert_exact_json(actual: Any, expected: Any, *, code: str, action: str) -> None:
if _canonical_json(actual) != _canonical_json(expected):
raise ReconstructionError(code, f"{action} did not match the hash-pinned material")
def _assert_exact_canonical_rows(actual: Any, expected: Any, *, code: str, action: str) -> None:
def normalize(value: Any, path: str) -> dict[str, list[dict[str, Any]]]:
if not isinstance(value, dict):
raise ValueError(f"{path} must be an object")
normalized: dict[str, list[dict[str, Any]]] = {}
for table, rows in value.items():
if (
not isinstance(table, str)
or not isinstance(rows, list)
or any(not isinstance(row, dict) for row in rows)
):
raise ValueError(f"{path} must map table names to row arrays")
normalized[table] = [
replay_receipt.normalize_typed_row(row, path=f"{path}.{table}[{index}]")
for index, row in enumerate(rows)
]
return normalized
try:
normalized_actual = normalize(actual, "actual canonical rows")
normalized_expected = normalize(expected, "expected canonical rows")
except ValueError as exc:
raise ReconstructionError(code, f"{action} returned invalid typed canonical rows") from exc
_assert_exact_json(normalized_actual, normalized_expected, code=code, action=action)
def _assert_exact_proposal_readback(actual: Any, expected: Any, *, code: str, action: str) -> None:
def normalize(value: Any, path: str) -> dict[str, Any]:
if not isinstance(value, dict):
raise ValueError(f"{path} must be an object")
normalized = dict(value)
for field in ("proposal", "approval_snapshot", "approved_proposal_snapshot"):
row = normalized.get(field)
if not isinstance(row, dict):
raise ValueError(f"{path}.{field} must be an object")
normalized[field] = replay_receipt.normalize_typed_row(row, path=f"{path}.{field}")
return normalized
try:
normalized_actual = normalize(actual, "actual proposal readback")
normalized_expected = normalize(expected, "expected proposal readback")
except ValueError as exc:
raise ReconstructionError(code, f"{action} returned invalid typed proposal rows") from exc
_assert_exact_json(normalized_actual, normalized_expected, code=code, action=action)
def _entry_summary(entry: LedgerEntry) -> dict[str, Any]:
proposal = entry.applied_proposal
return {
"sequence": entry.sequence,
"proposal_id": proposal["id"],
"proposal_type": proposal["proposal_type"],
"contract_version": entry.contract_version,
"material_sha256": entry.material_sha256,
"replay_material_sha256": entry.replay_material_sha256,
"original_apply_sql_sha256": entry.receipt["apply_engine"]["apply_sql_sha256"],
@ -1142,11 +1724,12 @@ def apply_entry(
code="entry_seed_readback_failed",
action=f"ledger entry {entry.sequence} proposal seed readback",
)
_assert_exact_json(
_assert_exact_proposal_readback(
proposal_readback,
{
"proposal": entry.approved_proposal,
"approval_snapshot": entry.approval_snapshot,
"approved_proposal_snapshot": entry.approved_proposal,
},
code="entry_seed_mismatch",
action=f"ledger entry {entry.sequence} proposal seed",
@ -1163,6 +1746,21 @@ def apply_entry(
)
)
summary["mutating_prestate_captured"] = True
elif entry.contract_version == 3:
seeded_rows = _read_json(
args,
container,
replay_receipt.build_postflight_sql(entry.receipt["proposal"]),
code="entry_seed_readback_failed",
action=f"ledger entry {entry.sequence} V3 empty canonical prestate readback",
)
_assert_exact_canonical_rows(
seeded_rows,
{table: [] for table in entry.receipt["canonical_rows"]},
code="entry_seed_mismatch",
action=f"ledger entry {entry.sequence} V3 empty canonical prestate",
)
summary["canonical_seed_exact"] = True
else:
seeded_rows = _read_json(
args,
@ -1171,7 +1769,7 @@ def apply_entry(
code="entry_seed_readback_failed",
action=f"ledger entry {entry.sequence} canonical seed readback",
)
_assert_exact_json(
_assert_exact_canonical_rows(
seeded_rows,
entry.receipt["canonical_rows"],
code="entry_seed_mismatch",
@ -1211,6 +1809,17 @@ def apply_entry(
action=f"ledger entry {entry.sequence} revise_strategy exact poststate normalization",
)
summary["mutating_poststate_normalized"] = True
elif entry.contract_version == 3:
_psql(
args,
container,
build_v3_canonical_timestamp_normalization_sql(entry),
role="postgres",
timeout=args.command_timeout,
code="v3_canonical_timestamp_normalization_failed",
action=f"ledger entry {entry.sequence} V3 canonical timestamp normalization",
)
summary["mutating_poststate_normalized"] = True
_psql(
args,
@ -1228,11 +1837,12 @@ def apply_entry(
code="entry_postflight_failed",
action=f"ledger entry {entry.sequence} applied proposal readback",
)
_assert_exact_json(
_assert_exact_proposal_readback(
proposal_readback,
{
"proposal": entry.applied_proposal,
"approval_snapshot": entry.approval_snapshot,
"approved_proposal_snapshot": entry.approved_proposal,
},
code="entry_postflight_mismatch",
action=f"ledger entry {entry.sequence} applied proposal",
@ -1246,7 +1856,7 @@ def apply_entry(
code="entry_postflight_failed",
action=f"ledger entry {entry.sequence} exact row readback",
)
_assert_exact_json(
_assert_exact_canonical_rows(
final_rows,
entry.receipt["canonical_rows"],
code="entry_postflight_mismatch",
@ -1314,6 +1924,7 @@ def _initial_receipt(args: argparse.Namespace, container: str) -> dict[str, Any]
"genesis_dump": {"bytes": None, "sha256": None, "pin_verified": False},
"genesis_manifest": {"bytes": None, "sha256": None, "pin_verified": False},
"ledger": {"bytes": None, "sha256": None, "pin_verified": False},
"admission_manifest": {"bytes": None, "sha256": None, "pin_verified": False},
"final_manifest": {"bytes": None, "sha256": None, "pin_verified": False},
"engine_hashes": {},
},
@ -1327,9 +1938,14 @@ def _initial_receipt(args: argparse.Namespace, container: str) -> dict[str, Any]
"isolation": None,
},
"guard_prerequisites": {"status": "pending", "sha256": None},
"v3_contract": {"status": "not_required", "sha256": None},
"genesis_parity": {"status": "pending"},
"genesis_key_counts": {},
"ledger": {"entry_count": None, "entries": []},
"admission": {
"status": "pending",
"canonical_retrieval_eligible": False,
},
"final_parity": {"status": "pending"},
"final_key_counts": {},
"cleanup": {
@ -1375,6 +1991,11 @@ def run_reconstruction(args: argparse.Namespace) -> dict[str, Any]:
"sha256": bundle.ledger_sha256,
"pin_verified": True,
},
"admission_manifest": {
"bytes": bundle.admission_manifest_bytes,
"sha256": bundle.admission_manifest_sha256,
"pin_verified": bundle.admission_manifest_sha256 is not None,
},
"final_manifest": {
"bytes": bundle.final_manifest_bytes,
"sha256": bundle.final_manifest_sha256,
@ -1384,6 +2005,14 @@ def run_reconstruction(args: argparse.Namespace) -> dict[str, Any]:
}
receipt["ledger"]["entry_count"] = len(bundle.entries)
receipt["ledger"]["entries"] = [_entry_summary(entry) for entry in bundle.entries]
receipt["admission"] = bundle.admission
if bundle.admission["canonical_retrieval_eligible"] is not True:
receipt["claim_ceiling"] = f"forensic-only reconstruction; {CLAIM_CEILING}"
if any(entry.contract_version == 3 for entry in bundle.entries):
receipt["v3_contract"] = {
"status": "pending",
"sha256": bundle.engine_hashes["v3_contract_sha256"],
}
phase = "container_name_preflight"
absent, absence_error = canonical_rebuild.inspect_container_absence(
@ -1548,7 +2177,13 @@ def run_reconstruction(args: argparse.Namespace) -> dict[str, Any]:
)
phase = "ordered_ledger_replay"
v3_installed = False
for entry, summary in zip(bundle.entries, receipt["ledger"]["entries"], strict=True):
if entry.contract_version == 3 and not v3_installed:
phase = "v3_contract_install"
receipt["v3_contract"] = install_v3_contract(args, container)
v3_installed = True
phase = "ordered_ledger_replay"
apply_entry(args, container, entry, summary)
phase = "final_parity"
@ -1651,6 +2286,8 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser.add_argument("--genesis-manifest", required=True, type=Path)
parser.add_argument("--ledger", required=True, type=Path)
parser.add_argument("--ledger-sha256", required=True)
parser.add_argument("--admission-manifest", type=Path)
parser.add_argument("--admission-manifest-sha256")
parser.add_argument("--database", default="teleo")
parser.add_argument("--image", default=DEFAULT_REBUILD_IMAGE)
parser.add_argument("--container-prefix", default="teleo-genesis-ledger-rebuild")
@ -1698,6 +2335,13 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser.error(f"{flag} must be an existing regular file")
if not SHA256_RE.fullmatch(args.ledger_sha256):
parser.error("--ledger-sha256 must be a lowercase SHA-256 digest")
if (args.admission_manifest is None) != (args.admission_manifest_sha256 is None):
parser.error("--admission-manifest and --admission-manifest-sha256 must be supplied together")
if args.admission_manifest is not None:
if not args.admission_manifest.is_file():
parser.error("--admission-manifest must be an existing regular file")
if not SHA256_RE.fullmatch(args.admission_manifest_sha256):
parser.error("--admission-manifest-sha256 must be a lowercase SHA-256 digest")
try:
protected = {
args.genesis_dump.resolve(),
@ -1705,6 +2349,8 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
*_declared_artifact_paths(args.ledger),
*FIXED_ENGINE_PATHS,
}
if args.admission_manifest is not None:
protected.add(args.admission_manifest.resolve())
except ReconstructionError as exc:
parser.error(exc.safe_message)
if args.output.resolve() in protected:

1023
ops/shard_corpus_extraction_work.py Normal file
View file

File diff suppressed because it is too large Load diff

2293
ops/teleo_v3_epistemic_contract.sql Normal file
View file

File diff suppressed because it is too large Load diff

1926
ops/verify_teleo_v3_epistemic_contract.py Normal file
View file

File diff suppressed because it is too large Load diff

1343
scripts/apply_proposal.py
View file

File diff suppressed because it is too large Load diff

177
scripts/apply_worker.py
View file

@ -24,6 +24,9 @@ Safety posture
* Gated OFF by default. Without ``--enable`` (or ``KB_APPLY_WORKER_ENABLED=1``) the
worker only REPORTS what it would apply -- it performs no writes. Flip it on only
after the first apply has been proven by hand.
* Pinned to one proposal contract version. V2 remains the default; a V3 worker
must be selected explicitly with ``--contract-version 3`` or
``KB_APPLY_WORKER_CONTRACT_VERSION=3`` after the V3 schema is installed.
* The optional render step is a configurable hook (``--render-cmd`` /
``KB_APPLY_RENDER_CMD``). Only ``revise_strategy`` currently has a proven
agent-owned render contract. Other apply types update canonical state without
@ -38,6 +41,7 @@ import os
import stat
import subprocess
import sys
import uuid
from pathlib import Path
from typing import Any
@ -56,6 +60,9 @@ RENDER_TYPES = ("revise_strategy",)
POSTCOMMIT_RECEIPT_FAILURE_MARKER = "committed, but private replay receipt capture failed"
POSTCOMMIT_RECEIPT_RECOVERY_MARKER = ". Recover without reapplying via: "
CONTRACT_STATE_V2 = "V2"
CONTRACT_STATE_V3 = "V3"
CONTRACT_STATE_INVALID = "INVALID"
class PostCommitReceiptError(RuntimeError):
@ -69,28 +76,71 @@ def build_candidate_query(
types: tuple = WORKER_TYPES,
limit: int = 20,
excluded_ids: tuple = (),
contract_version: int = 2,
proposal_id: str | None = None,
) -> str:
"""SELECT approved, applyable proposals that carry an apply_payload.
"""SELECT approved proposals for one exact apply contract version.
Filters on status='approved' at the DB level -- this is the structural
guarantee that the worker cannot act on anything a human has not approved.
IDs at the failure ceiling are excluded before LIMIT so exhausted rows cannot
occupy every fetched slot and starve later eligible proposals.
"""
type_list = ", ".join(ap.sql_literal(t) for t in types)
if contract_version not in {2, 3}:
raise ValueError("worker contract_version must be 2 or 3")
requested_types = tuple(dict.fromkeys(types))
if contract_version == 3:
eligible_types = tuple(proposal_type for proposal_type in requested_types if proposal_type == "approve_claim")
if not eligible_types:
raise ValueError("V3 worker requires proposal_type 'approve_claim'")
type_contract_clause = (
"proposal_type = 'approve_claim'\n"
" and pg_catalog.jsonb_typeof(payload->'apply_payload'->'contract_version') = 'number'\n"
" and payload->'apply_payload'->>'contract_version' = '3'"
)
else:
eligible_types = tuple(proposal_type for proposal_type in requested_types if proposal_type in WORKER_TYPES)
if not eligible_types:
raise ValueError("V2 worker has no supported proposal types")
legacy_types = tuple(proposal_type for proposal_type in eligible_types if proposal_type != "approve_claim")
clauses = []
if legacy_types:
legacy_type_list = ", ".join(ap.sql_literal(proposal_type) for proposal_type in legacy_types)
clauses.append(
f"(proposal_type in ({legacy_type_list}) and not (payload->'apply_payload' ? 'contract_version'))"
)
if "approve_claim" in eligible_types:
clauses.append(
"(proposal_type = 'approve_claim' "
"and pg_catalog.jsonb_typeof(payload->'apply_payload'->'contract_version') = 'number' "
"and payload->'apply_payload'->>'contract_version' = '2')"
)
type_contract_clause = "(\n " + "\n or ".join(clauses) + "\n )"
excluded = tuple(str(proposal_id) for proposal_id in excluded_ids)
exclusion_clause = ""
if excluded:
excluded_list = ", ".join(ap.sql_literal(proposal_id) for proposal_id in excluded)
exclusion_clause = f"\n and id::text not in ({excluded_list})"
exact_clause = ""
if proposal_id is not None:
try:
canonical_proposal_id = str(uuid.UUID(str(proposal_id)))
except ValueError as exc:
raise ValueError("proposal_id must be a full canonical UUID") from exc
if str(proposal_id) != canonical_proposal_id:
raise ValueError("proposal_id must be a canonical lowercase UUID")
exact_clause = f"\n and id = {ap.sql_literal(canonical_proposal_id)}::uuid"
return f"""select jsonb_build_object(
'id', id::text,
'proposal_type', proposal_type,
'agent_id', payload->'apply_payload'->>'agent_id')::text
'agent_id', payload->'apply_payload'->>'agent_id',
'contract_version', coalesce(payload->'apply_payload'->>'contract_version', '2'))::text
from kb_stage.kb_proposals
where status = 'approved'
and proposal_type in ({type_list})
and payload ? 'apply_payload'{exclusion_clause}
and pg_catalog.jsonb_typeof(payload) = 'object'
and payload ? 'apply_payload'
and pg_catalog.jsonb_typeof(payload->'apply_payload') = 'object'
and {type_contract_clause}{exact_clause}{exclusion_clause}
order by created_at asc
limit {int(limit)};"""
@ -238,9 +288,29 @@ def fetch_candidates(
password: str,
excluded_ids: tuple,
) -> list[dict[str, Any]]:
sql = build_candidate_query(limit=args.limit, excluded_ids=excluded_ids)
sql = build_candidate_query(
limit=args.limit,
excluded_ids=excluded_ids,
contract_version=args.contract_version,
proposal_id=getattr(args, "proposal_id", None),
)
out = ap.run_psql(_psql_args(args), sql, password)
return parse_candidates(out)
candidates = parse_candidates(out)
expected_version = str(args.contract_version)
allowed_types = {"approve_claim"} if args.contract_version == 3 else set(WORKER_TYPES)
for candidate in candidates:
if candidate.get("proposal_type") not in allowed_types or candidate.get("contract_version") != expected_version:
raise RuntimeError("candidate query returned a proposal outside the exact worker contract")
return candidates
def detect_installed_contract(args: argparse.Namespace, password: str) -> str:
"""Classify the exact database apply contract before candidate selection."""
sql = ap.build_v3_contract_state_query() + ";"
raw = ap.run_psql(_psql_args(args), sql, password).strip()
if raw not in {CONTRACT_STATE_V2, CONTRACT_STATE_V3, CONTRACT_STATE_INVALID}:
raise RuntimeError("database apply contract readback was not V2, V3, or INVALID")
return raw
def apply_one(args: argparse.Namespace, proposal_id: str) -> None:
@ -250,6 +320,8 @@ def apply_one(args: argparse.Namespace, proposal_id: str) -> None:
sys.executable,
str(args.apply_script),
proposal_id,
"--expected-contract-version",
str(args.contract_version),
"--applied-by",
args.applied_by,
"--secrets-file",
@ -267,6 +339,9 @@ def apply_one(args: argparse.Namespace, proposal_id: str) -> None:
"--receipt-out",
str(receipt_path),
]
fresh_preflight = getattr(args, "fresh_preflight", None)
if fresh_preflight is not None:
cmd.extend(("--fresh-preflight", str(fresh_preflight)))
result = subprocess.run(cmd, text=True, capture_output=True, check=False)
committed = result.returncode == 0 or has_exact_postcommit_receipt_failure(
result.stderr,
@ -327,10 +402,56 @@ def render_one(args: argparse.Namespace, agent_id: str | None) -> str:
# --------------------------------------------------------------------------- #
def run(args: argparse.Namespace) -> int:
password = ap.load_password(args.secrets_file)
installed_contract = detect_installed_contract(args, password)
if installed_contract == CONTRACT_STATE_INVALID:
print(
"REFUSE execution: database apply contract integrity is INVALID; "
"no candidate selection or writes attempted.",
file=sys.stderr,
)
return 1
if args.contract_version == 3 and installed_contract == CONTRACT_STATE_V2:
print(
"REFUSE contract v3 execution: the database V3 cutover is not installed; "
"no candidate selection or writes attempted.",
file=sys.stderr,
)
return 1
failure_counts = load_failure_state(args.failure_state_file)
exhausted_ids = tuple(proposal_id for proposal_id, count in failure_counts.items() if count >= args.max_attempts)
candidates = fetch_candidates(args, password, exhausted_ids)
fetch_exclusions = () if args.contract_version == 2 and installed_contract == CONTRACT_STATE_V3 else exhausted_ids
candidates = fetch_candidates(args, password, fetch_exclusions)
exact_proposal_id = getattr(args, "proposal_id", None)
if exact_proposal_id is not None and (len(candidates) != 1 or candidates[0].get("id") != exact_proposal_id):
if not candidates and getattr(args, "require_candidate", False):
print(
f"required approved+applyable contract v{args.contract_version} proposal "
f"{exact_proposal_id} was not selected; no writes attempted.",
file=sys.stderr,
)
return 2
print(
f"REFUSE exact proposal execution: approved+applyable contract v{args.contract_version} "
f"proposal {exact_proposal_id} was not selected exactly once; no writes attempted.",
file=sys.stderr,
)
return 1
if args.contract_version == 2 and installed_contract == CONTRACT_STATE_V3:
print(
"SKIP contract v2 execution: the database V3 cutover is installed; "
f"{len(candidates)} approved V2 proposal(s) in this fetch remain review-visible. "
"Migrate each packet to a newly reviewed V3 proposal or explicitly reject/cancel it; no writes attempted.",
file=sys.stderr,
)
for candidate in candidates:
print(
f" retained V2 proposal {candidate['id']} ({candidate['proposal_type']})",
file=sys.stderr,
)
return 0
for proposal_id in exhausted_ids:
print(
f"EXCLUDE failure-ceiling ID {proposal_id}: failed "
@ -340,7 +461,7 @@ def run(args: argparse.Namespace) -> int:
)
if not candidates:
print("no approved+applyable proposals; nothing to do")
print(f"no approved+applyable contract v{args.contract_version} proposals; nothing to do")
return 0
split = partition_candidates(candidates, failure_counts, args.max_attempts, args.max_per_tick)
@ -356,7 +477,8 @@ def run(args: argparse.Namespace) -> int:
enabled = args.enable or os.environ.get("KB_APPLY_WORKER_ENABLED") == "1"
if not enabled:
print(
f"[report-only] worker disabled; {len(split['to_apply'])} proposal(s) would apply "
f"[report-only] contract v{args.contract_version} worker disabled; "
f"{len(split['to_apply'])} proposal(s) would apply "
f"this tick (cap {args.max_per_tick}):"
)
for c in split["to_apply"]:
@ -405,7 +527,23 @@ def parse_args(argv: list[str]) -> argparse.Namespace:
action="store_true",
help="actually apply (default: report-only). Also honored via KB_APPLY_WORKER_ENABLED=1.",
)
p.add_argument(
"--contract-version",
type=int,
choices=(2, 3),
default=os.environ.get("KB_APPLY_WORKER_CONTRACT_VERSION", "2"),
help="apply only proposals for one exact contract version (default 2)",
)
p.add_argument("--limit", type=int, default=20, help="max candidates fetched per tick")
p.add_argument(
"--proposal-id",
help="select exactly one canonical UUID; refuse instead of falling through to another approved proposal",
)
p.add_argument(
"--require-candidate",
action="store_true",
help="fail when --proposal-id is not selected instead of treating an empty tick as success",
)
p.add_argument(
"--max-per-tick",
type=int,
@ -435,6 +573,11 @@ def parse_args(argv: list[str]) -> argparse.Namespace:
default=str(HERE / "apply_proposal.py"),
help="path to the apply_proposal.py engine (the sole apply path)",
)
p.add_argument(
"--fresh-preflight",
type=Path,
help="exact fresh-owned preflight forwarded to the apply engine; requires --proposal-id",
)
p.add_argument(
"--receipt-dir",
default=(
@ -456,7 +599,19 @@ def parse_args(argv: list[str]) -> argparse.Namespace:
p.add_argument("--db", default=ap.DEFAULT_DB)
p.add_argument("--host", default=ap.DEFAULT_HOST)
p.add_argument("--role", default=ap.DEFAULT_ROLE)
return p.parse_args(argv)
args = p.parse_args(argv)
if args.proposal_id is not None:
try:
canonical_proposal_id = str(uuid.UUID(args.proposal_id))
except (AttributeError, TypeError, ValueError) as exc:
p.error(f"--proposal-id must be a UUID: {exc}")
if args.proposal_id != canonical_proposal_id:
p.error("--proposal-id must be a canonical lowercase UUID")
if args.require_candidate and args.proposal_id is None:
p.error("--require-candidate requires --proposal-id")
if args.fresh_preflight is not None and args.proposal_id is None:
p.error("--fresh-preflight requires --proposal-id")
return args
def main(argv: list[str] | None = None) -> int:

830
scripts/compile_kb_source_packet.py
View file

@ -2,10 +2,10 @@
"""Compile one hash-bound source document into a pending-review KB proposal.
The command is deliberately build-only. It validates a raw artifact, its UTF-8
text extraction, and an extraction manifest; then it reuses Leo's rich proposal
normalizer and strict staging preflight to produce a schema-valid
``approve_claim`` child. It never connects to Postgres and never executes the
generated staging SQL.
text extraction, and an extraction manifest. V1/V2 reuse Leo's rich proposal
normalizer; V3 additionally requires the exact extractor specification and
builds the explicit epistemic payload. The command never connects to Postgres
or executes the generated staging SQL.
"""
from __future__ import annotations
@ -18,6 +18,7 @@ import re
import sys
import tempfile
import uuid
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from urllib.parse import urlsplit
@ -31,11 +32,14 @@ import stage_normalized_proposal as stage # noqa: E402
MANIFEST_SCHEMA = "livingip.kbSourceExtractionManifest.v1"
MANIFEST_SCHEMA_V2 = "livingip.kbSourceExtractionManifest.v2"
MANIFEST_SCHEMAS = {MANIFEST_SCHEMA, MANIFEST_SCHEMA_V2}
MANIFEST_SCHEMA_V3 = "livingip.kbSourceExtractionManifest.v3"
BUNDLE_SCHEMA = "livingip.kbSourceProposalBundle.v1"
BUNDLE_SCHEMA_V3 = "livingip.kbSourceProposalBundle.v3"
STATUS_SCHEMA = "livingip.kbSourceCompilerStatus.v1"
SHA256_RE = re.compile(r"^[0-9a-f]{64}$")
KEY_RE = re.compile(r"^[a-z0-9][a-z0-9._:-]{0,127}$")
URI_SCHEME_RE = re.compile(r"^[a-z][a-z0-9+.-]*:", re.IGNORECASE)
ISO_TIMESTAMP_RE = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{1,6})?(?:Z|[+-]\d{2}:\d{2})$")
UNSAFE_URI_SCHEMES = {"data", "javascript", "vbscript"}
TOP_LEVEL_KEYS = {
@ -55,6 +59,57 @@ DEDUPE_KEYS = {"database_search_query", "candidate_claims", "duplicates"}
DEDUPE_CANDIDATE_KEYS = {"id", "text", "score", "evidence_count", "edge_count"}
DEDUPE_DUPLICATE_KEYS = {"claim_id", "source_quote", "reason"}
TOP_LEVEL_KEYS_V3 = {
"schema",
"artifact_sha256",
"extracted_text_sha256",
"extractor",
"agent_id",
"source",
"claims",
}
EXTRACTOR_KEYS_V3 = {"name", "version", "spec_sha256"}
SOURCE_KEYS_V3 = {
"identity",
"source_key",
"source_type",
"title",
"locator",
"access_scope",
"ingestion_origin",
"provenance_status",
"source_class",
"auto_promotion_policy",
"captured_at",
}
CLAIM_KEYS_V3 = {
"claim_key",
"type",
"proposition",
"body_markdown",
"scope",
"access_scope",
"owner_agent_id",
"status",
"revision_criteria",
"revision_kind",
"supersedes_id",
"tags",
"evidence",
"assessment",
}
EVIDENCE_KEYS_V3 = {"evidence_key", "quote", "polarity", "locator", "rationale"}
LOCATOR_KEYS_V3 = {"unit", "start", "end"}
ASSESSMENT_KEYS_V3 = {
"support_tier",
"challenge_tier",
"overall_state",
"rationale",
"supersedes_assessment_id",
}
TRUTH_APT_V3_CLAIM_TYPES = {"empirical", "predictive", "causal"}
ASSESSMENT_RELEVANT_POLARITIES = {"supports", "challenges"}
class CompilerError(RuntimeError):
"""Raised when an input cannot safely become a reviewable proposal."""
@ -72,10 +127,37 @@ def canonical_sha256(value: Any) -> str:
return sha256_bytes(canonical_json_bytes(value))
def _load_duplicate_free_json(value: bytes, label: str) -> Any:
try:
text = value.decode("utf-8", errors="strict")
except UnicodeDecodeError as exc:
raise CompilerError(f"{label} must be valid UTF-8 JSON: {exc}") from exc
def reject_duplicate_pairs(pairs: list[tuple[str, Any]]) -> dict[str, Any]:
result: dict[str, Any] = {}
for key, item in pairs:
if key in result:
raise CompilerError(f"{label} contains duplicate JSON object key: {key!r}")
result[key] = item
return result
try:
return json.loads(text, object_pairs_hook=reject_duplicate_pairs)
except json.JSONDecodeError as exc:
raise CompilerError(f"{label} must be valid UTF-8 JSON: {exc}") from exc
def stable_uuid(label: str) -> str:
return str(uuid.uuid5(uuid.NAMESPACE_URL, f"livingip:kb-source-compiler:v1:{label}"))
def stable_uuid_v3(row_kind: str, packet_identity_sha256: str, semantic_key: Any) -> str:
material = canonical_json_bytes(
["livingip", "kb-source-compiler", 3, packet_identity_sha256, row_kind, semantic_key]
).decode("ascii")
return str(uuid.uuid5(uuid.NAMESPACE_URL, material))
def _require_object(value: Any, label: str) -> dict[str, Any]:
if not isinstance(value, dict):
raise CompilerError(f"{label} must be an object")
@ -88,6 +170,13 @@ def _require_exact_keys(value: dict[str, Any], allowed: set[str], label: str) ->
raise CompilerError(f"{label} contains unsupported keys: {', '.join(unknown)}")
def _require_exact_shape(value: dict[str, Any], required: set[str], label: str) -> None:
_require_exact_keys(value, required, label)
missing = sorted(required - set(value))
if missing:
raise CompilerError(f"{label} is missing required keys: {', '.join(missing)}")
def _require_nonempty_string(value: Any, label: str) -> str:
if not isinstance(value, str) or not value.strip():
raise CompilerError(f"{label} must be a non-empty string")
@ -110,6 +199,51 @@ def _require_key(value: Any, label: str) -> str:
return text
def _require_canonical_uuid(value: Any, label: str, *, nullable: bool = False) -> str | None:
if value is None and nullable:
return None
if not isinstance(value, str):
raise CompilerError(f"{label} must be a canonical lowercase hyphenated UUID")
try:
canonical = str(uuid.UUID(value))
except ValueError as exc:
raise CompilerError(f"{label} must be a canonical lowercase hyphenated UUID") from exc
if value != canonical:
raise CompilerError(f"{label} must be a canonical lowercase hyphenated UUID")
return canonical
def _require_canonical_utc_timestamp(value: Any, label: str) -> str:
text = _require_nonempty_string(value, label)
if not ISO_TIMESTAMP_RE.fullmatch(text):
raise CompilerError(f"{label} must be a timezone-aware ISO-8601 timestamp with at most 6 fractional digits")
try:
parsed = datetime.fromisoformat(text.replace("Z", "+00:00"))
except ValueError as exc:
raise CompilerError(f"{label} must be a valid ISO-8601 timestamp") from exc
if parsed.tzinfo is None:
raise CompilerError(f"{label} must include a timezone")
return parsed.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ")
def _require_nullable_enum(value: Any, allowed: set[str], label: str) -> str | None:
if value is None:
return None
if not isinstance(value, str) or value not in allowed:
raise CompilerError(f"{label} must be null or one of {sorted(allowed)}")
return value
def _require_string_list(value: Any, label: str) -> list[str]:
if not isinstance(value, list) or any(
not isinstance(item, str) or not item.strip() or item != item.strip() for item in value
):
raise CompilerError(f"{label} must be a list of non-empty strings without surrounding whitespace")
if len(value) != len(set(value)):
raise CompilerError(f"{label} contains duplicates")
return sorted(value)
def _require_stable_reference(value: Any, label: str) -> str:
text = _require_nonempty_string(value, label)
if any(character.isspace() for character in text) or not URI_SCHEME_RE.match(text):
@ -162,6 +296,258 @@ def _require_nonnegative_integer(value: Any, label: str) -> int:
return value
def _v3_quote_binding(text_bytes: bytes, quote: Any, locator_value: Any, label: str) -> dict[str, Any]:
exact = _require_nonempty_string(quote, f"{label}.quote")
locator = _require_object(locator_value, f"{label}.locator")
_require_exact_shape(locator, LOCATOR_KEYS_V3, f"{label}.locator")
if locator.get("unit") != "utf8_bytes":
raise CompilerError(f"{label}.locator.unit must equal 'utf8_bytes'")
start = _require_nonnegative_integer(locator.get("start"), f"{label}.locator.start")
end = _require_nonnegative_integer(locator.get("end"), f"{label}.locator.end")
quote_bytes = exact.encode("utf-8")
if end <= start or end > len(text_bytes) or text_bytes[start:end] != quote_bytes:
raise CompilerError(f"{label}.locator does not bind the exact quote bytes in the extracted text")
return {
"quote": exact,
"quote_sha256": sha256_bytes(quote_bytes),
"unit": "utf8_bytes",
"start": start,
"end": end,
}
def _validate_v3_assessment(value: Any, claim_type: str, evidence: list[dict[str, Any]], label: str) -> dict[str, Any]:
assessment = _require_object(value, label)
_require_exact_shape(assessment, ASSESSMENT_KEYS_V3, label)
support_tier = _require_nullable_enum(assessment.get("support_tier"), ap.V3_SUPPORT_TIERS, f"{label}.support_tier")
challenge_tier = _require_nullable_enum(
assessment.get("challenge_tier"), ap.V3_SUPPORT_TIERS, f"{label}.challenge_tier"
)
if claim_type in TRUTH_APT_V3_CLAIM_TYPES and (support_tier is None or challenge_tier is None):
raise CompilerError(f"{label} requires support_tier and challenge_tier for a truth-apt claim")
overall_state = assessment.get("overall_state")
if overall_state not in ap.V3_ASSESSMENT_STATES:
raise CompilerError(f"{label}.overall_state must be one of {sorted(ap.V3_ASSESSMENT_STATES)}")
polarities = {row["polarity"] for row in evidence}
if not polarities & ASSESSMENT_RELEVANT_POLARITIES:
raise CompilerError(f"{label} requires at least one supports or challenges evidence row")
if support_tier not in {None, "none"} and "supports" not in polarities:
raise CompilerError(f"{label}.support_tier requires supports evidence")
if challenge_tier not in {None, "none"} and "challenges" not in polarities:
raise CompilerError(f"{label}.challenge_tier requires challenges evidence")
if overall_state == "support_dominant" and "supports" not in polarities:
raise CompilerError(f"{label}.overall_state=support_dominant requires supports evidence")
if overall_state == "challenge_dominant" and "challenges" not in polarities:
raise CompilerError(f"{label}.overall_state=challenge_dominant requires challenges evidence")
if overall_state == "mixed" and not {"supports", "challenges"} <= polarities:
raise CompilerError(f"{label}.overall_state=mixed requires supports and challenges evidence")
if assessment.get("supersedes_assessment_id") is not None:
raise CompilerError(f"{label}.supersedes_assessment_id must be null for a first assessment")
return {
"support_tier": support_tier,
"challenge_tier": challenge_tier,
"overall_state": overall_state,
"rationale": _require_nonempty_string(assessment.get("rationale"), f"{label}.rationale"),
"supersedes_assessment_id": None,
}
def _validate_manifest_v3(
manifest: dict[str, Any],
artifact_sha256: str,
text_sha256: str,
extractor_spec_sha256: str,
text_bytes: bytes,
) -> tuple[dict[str, Any], list[dict[str, Any]]]:
_require_exact_shape(manifest, TOP_LEVEL_KEYS_V3, "manifest")
if manifest.get("schema") != MANIFEST_SCHEMA_V3:
raise CompilerError(f"manifest.schema must equal {MANIFEST_SCHEMA_V3!r}")
declared_artifact_sha256 = _require_sha256(manifest.get("artifact_sha256"), "manifest.artifact_sha256")
declared_text_sha256 = _require_sha256(manifest.get("extracted_text_sha256"), "manifest.extracted_text_sha256")
if declared_artifact_sha256 != artifact_sha256:
raise CompilerError("artifact SHA-256 does not match manifest.artifact_sha256")
if declared_text_sha256 != text_sha256:
raise CompilerError("extracted text SHA-256 does not match manifest.extracted_text_sha256")
extractor = _require_object(manifest.get("extractor"), "manifest.extractor")
_require_exact_shape(extractor, EXTRACTOR_KEYS_V3, "manifest.extractor")
declared_spec_sha256 = _require_sha256(extractor.get("spec_sha256"), "manifest.extractor.spec_sha256")
if declared_spec_sha256 != extractor_spec_sha256:
raise CompilerError("extractor specification SHA-256 does not match manifest.extractor.spec_sha256")
validated_extractor = {
"name": _require_nonempty_string(extractor.get("name"), "manifest.extractor.name"),
"version": _require_nonempty_string(extractor.get("version"), "manifest.extractor.version"),
"spec_sha256": extractor_spec_sha256,
}
agent_id = _require_canonical_uuid(manifest.get("agent_id"), "manifest.agent_id")
source = _require_object(manifest.get("source"), "manifest.source")
_require_exact_shape(source, SOURCE_KEYS_V3, "manifest.source")
source_type = source.get("source_type")
if source_type not in ap.SOURCE_TYPES:
raise CompilerError(f"manifest.source.source_type must be one of {sorted(ap.SOURCE_TYPES)}")
source_access_scope = source.get("access_scope")
if source_access_scope not in ap.V3_ACCESS_SCOPES:
raise CompilerError(f"manifest.source.access_scope must be one of {sorted(ap.V3_ACCESS_SCOPES)}")
ingestion_origin = source.get("ingestion_origin")
if ingestion_origin not in ap.V3_SOURCE_ORIGINS:
raise CompilerError(f"manifest.source.ingestion_origin must be one of {sorted(ap.V3_SOURCE_ORIGINS)}")
provenance_status = source.get("provenance_status")
if provenance_status not in ap.V3_PROVENANCE_STATES:
raise CompilerError(f"manifest.source.provenance_status must be one of {sorted(ap.V3_PROVENANCE_STATES)}")
source_class = source.get("source_class")
if source_class not in ap.V3_SOURCE_CLASSES:
raise CompilerError(f"manifest.source.source_class must be one of {sorted(ap.V3_SOURCE_CLASSES)}")
auto_promotion_policy = source.get("auto_promotion_policy")
if auto_promotion_policy not in ap.V3_AUTO_PROMOTION_POLICIES:
raise CompilerError(
f"manifest.source.auto_promotion_policy must be one of {sorted(ap.V3_AUTO_PROMOTION_POLICIES)}"
)
source_identity = _require_stable_reference(source.get("identity"), "manifest.source.identity")
source_locator = _require_stable_reference(source.get("locator"), "manifest.source.locator")
expected_source_identity = f"document:sha256:{artifact_sha256}"
expected_source_locator = f"artifact://sha256/{artifact_sha256}"
if source_identity != expected_source_identity:
raise CompilerError(
"manifest.source.identity must equal the content-addressed identity for the actual artifact bytes"
)
if source_locator != expected_source_locator:
raise CompilerError("manifest.source.locator must equal artifact://sha256/<actual artifact SHA-256>")
validated_source = {
"identity": source_identity,
"source_key": _require_key(source.get("source_key"), "manifest.source.source_key"),
"source_type": source_type,
"title": _require_nonempty_string(source.get("title"), "manifest.source.title"),
"locator": source_locator,
"access_scope": source_access_scope,
"ingestion_origin": ingestion_origin,
"provenance_status": provenance_status,
"source_class": source_class,
"auto_promotion_policy": auto_promotion_policy,
"captured_at": _require_canonical_utc_timestamp(source.get("captured_at"), "manifest.source.captured_at"),
}
raw_claims = manifest.get("claims")
if not isinstance(raw_claims, list) or not raw_claims:
raise CompilerError("manifest.claims must be a non-empty list of objects")
if len(raw_claims) > ap.MAX_BUNDLE_ROWS["claims"]:
raise CompilerError(f"manifest.claims exceeds {ap.MAX_BUNDLE_ROWS['claims']} rows")
validated_claims: list[dict[str, Any]] = []
quote_bindings: list[dict[str, Any]] = []
seen_claim_keys: set[str] = set()
evidence_count = 0
for claim_index, raw_claim in enumerate(raw_claims):
label = f"manifest.claims[{claim_index}]"
claim = _require_object(raw_claim, label)
_require_exact_shape(claim, CLAIM_KEYS_V3, label)
claim_key = _require_key(claim.get("claim_key"), f"{label}.claim_key")
if claim_key in seen_claim_keys:
raise CompilerError(f"duplicate claim_key: {claim_key}")
seen_claim_keys.add(claim_key)
claim_type = claim.get("type")
if claim_type not in ap.V3_CLAIM_TYPES:
raise CompilerError(f"{label}.type must be one of {sorted(ap.V3_CLAIM_TYPES)}")
access_scope = claim.get("access_scope")
if access_scope not in ap.V3_ACCESS_SCOPES:
raise CompilerError(f"{label}.access_scope must be one of {sorted(ap.V3_ACCESS_SCOPES)}")
owner_agent_id = _require_canonical_uuid(claim.get("owner_agent_id"), f"{label}.owner_agent_id", nullable=True)
if access_scope == "agent_private" and owner_agent_id is None:
raise CompilerError(f"{label}.owner_agent_id is required for agent_private claims")
status = claim.get("status")
if status not in {"active", "contested"}:
raise CompilerError(f"{label}.status must be active or contested")
if claim.get("revision_kind") != "original" or claim.get("supersedes_id") is not None:
raise CompilerError(f"{label} must describe an original claim with supersedes_id=null")
raw_evidence = claim.get("evidence")
if not isinstance(raw_evidence, list) or not raw_evidence:
raise CompilerError(f"{label}.evidence must be a non-empty list of exact locator-bound rows")
evidence_count += len(raw_evidence)
if evidence_count > ap.MAX_BUNDLE_ROWS["evidence"]:
raise CompilerError(f"manifest evidence exceeds {ap.MAX_BUNDLE_ROWS['evidence']} rows")
validated_evidence: list[dict[str, Any]] = []
seen_evidence_keys: set[str] = set()
seen_polarities: set[str] = set()
for evidence_index, raw_row in enumerate(raw_evidence):
evidence_label = f"{label}.evidence[{evidence_index}]"
row = _require_object(raw_row, evidence_label)
_require_exact_shape(row, EVIDENCE_KEYS_V3, evidence_label)
evidence_key = _require_key(row.get("evidence_key"), f"{evidence_label}.evidence_key")
if evidence_key in seen_evidence_keys:
raise CompilerError(f"{evidence_label}.evidence_key duplicates an earlier evidence row")
seen_evidence_keys.add(evidence_key)
polarity = row.get("polarity")
if polarity not in ap.V3_EVIDENCE_POLARITIES:
raise CompilerError(f"{evidence_label}.polarity must be one of {sorted(ap.V3_EVIDENCE_POLARITIES)}")
if polarity in seen_polarities:
raise CompilerError(f"{evidence_label}.polarity duplicates a claim-local source/polarity key")
seen_polarities.add(polarity)
binding = _v3_quote_binding(text_bytes, row.get("quote"), row.get("locator"), evidence_label)
validated_evidence.append(
{
"evidence_key": evidence_key,
"quote": binding["quote"],
"polarity": polarity,
"locator": {
"unit": binding["unit"],
"start": binding["start"],
"end": binding["end"],
},
"rationale": _require_nonempty_string(row.get("rationale"), f"{evidence_label}.rationale"),
}
)
quote_bindings.append(
{
"claim_key": claim_key,
"evidence_key": evidence_key,
"kind": "evidence",
**binding,
}
)
validated_evidence.sort(key=lambda row: row["evidence_key"])
validated_claims.append(
{
"claim_key": claim_key,
"type": claim_type,
"proposition": _require_nonempty_string(claim.get("proposition"), f"{label}.proposition"),
"body_markdown": _require_nonempty_string(claim.get("body_markdown"), f"{label}.body_markdown"),
"scope": _require_nonempty_string(claim.get("scope"), f"{label}.scope"),
"access_scope": access_scope,
"owner_agent_id": owner_agent_id,
"status": status,
"revision_criteria": _require_nonempty_string(
claim.get("revision_criteria"), f"{label}.revision_criteria"
),
"revision_kind": "original",
"supersedes_id": None,
"tags": _require_string_list(claim.get("tags"), f"{label}.tags"),
"evidence": validated_evidence,
"assessment": _validate_v3_assessment(
claim.get("assessment"), claim_type, validated_evidence, f"{label}.assessment"
),
}
)
validated_claims.sort(key=lambda row: row["claim_key"])
quote_bindings.sort(key=lambda row: (row["claim_key"], row["evidence_key"]))
return (
{
"schema": MANIFEST_SCHEMA_V3,
"artifact_sha256": artifact_sha256,
"extracted_text_sha256": text_sha256,
"extractor": validated_extractor,
"agent_id": agent_id,
"source": validated_source,
"claims": validated_claims,
},
quote_bindings,
)
def _validate_dedupe(value: Any, extracted_text: str) -> dict[str, Any]:
dedupe = _require_object(value, "manifest.dedupe")
_require_exact_keys(dedupe, DEDUPE_KEYS, "manifest.dedupe")
@ -460,6 +846,408 @@ def build_parent_proposal(
}
def _v3_identity_material(manifest: dict[str, Any], manifest_file_sha256: str) -> tuple[dict[str, Any], str, str]:
candidate_content = {
"agent_id": manifest["agent_id"],
"source": manifest["source"],
"claims": manifest["claims"],
}
candidate_content_sha256 = canonical_sha256(candidate_content)
identity_material = {
"artifact_sha256": manifest["artifact_sha256"],
"extracted_text_sha256": manifest["extracted_text_sha256"],
"extractor": manifest["extractor"],
"manifest_file_sha256": manifest_file_sha256,
"source_identity": manifest["source"]["identity"],
"source_locator": manifest["source"]["locator"],
"normalized_candidate_content_sha256": candidate_content_sha256,
}
return identity_material, candidate_content_sha256, canonical_sha256(identity_material)
def _build_v3_parent_proposal(
manifest: dict[str, Any],
manifest_canonical_sha256: str,
identity_material: dict[str, Any],
candidate_content_sha256: str,
packet_identity_sha256: str,
) -> dict[str, Any]:
source = manifest["source"]
parent_id = stable_uuid_v3("source_proposal", packet_identity_sha256, candidate_content_sha256)
return {
"id": parent_id,
"proposal_type": "approve_claim",
"status": "pending_review",
"proposed_by_handle": "leo",
"proposed_by_agent_id": manifest["agent_id"],
"channel": "source_document_compiler_v3",
"source_ref": f"source-compiler-v3:{packet_identity_sha256[:32]}",
"rationale": "Compile one retained semantic extraction into a hash-bound V3 claim proposal for review.",
"payload": {
"source_document": {
"manifest_schema": MANIFEST_SCHEMA_V3,
"manifest_file_sha256": identity_material["manifest_file_sha256"],
"manifest_canonical_sha256": manifest_canonical_sha256,
"artifact_sha256": manifest["artifact_sha256"],
"extracted_text_sha256": manifest["extracted_text_sha256"],
"source_identity": source["identity"],
"source_locator": source["locator"],
"extractor": manifest["extractor"],
},
"identity_material": identity_material,
"normalized_candidate_content_sha256": candidate_content_sha256,
"packet_identity_sha256": packet_identity_sha256,
"semantic_extraction_is_retained_input": True,
"model_training_claim": False,
},
}
def _v3_source_excerpt(manifest: dict[str, Any]) -> str:
excerpts: list[str] = []
seen: set[str] = set()
for claim in manifest["claims"]:
for evidence in claim["evidence"]:
quote = evidence["quote"]
if quote not in seen:
seen.add(quote)
excerpts.append(quote)
return "\n\n".join(excerpts)
def _v3_locator_json(manifest: dict[str, Any], binding: dict[str, Any]) -> dict[str, Any]:
return {
"schema": "livingip.extractedTextQuoteLocator.v1",
"source_identity": manifest["source"]["identity"],
"source_locator": manifest["source"]["locator"],
"artifact_sha256": manifest["artifact_sha256"],
"extracted_text_sha256": manifest["extracted_text_sha256"],
"extractor": manifest["extractor"],
"unit": binding["unit"],
"start": binding["start"],
"end": binding["end"],
"quote_sha256": binding["quote_sha256"],
}
def _ensure_distinct_v3_ids(values: list[str]) -> None:
if len(values) != len(set(values)):
raise CompilerError("compiled V3 packet contains duplicate deterministic ids")
def _build_v3_strict_child(
manifest: dict[str, Any],
manifest_canonical_sha256: str,
quote_bindings: list[dict[str, Any]],
parent: dict[str, Any],
identity_material: dict[str, Any],
candidate_content_sha256: str,
packet_identity_sha256: str,
) -> dict[str, Any]:
source = manifest["source"]
source_id = stable_uuid_v3("source", packet_identity_sha256, source["source_key"])
claim_ids = {
claim["claim_key"]: stable_uuid_v3("claim", packet_identity_sha256, claim["claim_key"])
for claim in manifest["claims"]
}
binding_by_key = {(binding["claim_key"], binding["evidence_key"]): binding for binding in quote_bindings}
canonical_url, storage_uri = _locator_fields(source["locator"])
source_row = {
"id": source_id,
"source_type": source["source_type"],
"canonical_url": canonical_url,
"storage_uri": storage_uri,
"excerpt": _v3_source_excerpt(manifest),
"content_hash": manifest["artifact_sha256"],
"access_scope": source["access_scope"],
"ingestion_origin": source["ingestion_origin"],
"provenance_status": source["provenance_status"],
"source_class": source["source_class"],
"auto_promotion_policy": source["auto_promotion_policy"],
"captured_at": source["captured_at"],
"created_by": manifest["agent_id"],
}
claim_rows: list[dict[str, Any]] = []
evidence_rows: list[dict[str, Any]] = []
assessment_rows: list[dict[str, Any]] = []
for claim in manifest["claims"]:
claim_id = claim_ids[claim["claim_key"]]
claim_rows.append(
{
"id": claim_id,
"owner_agent_id": claim["owner_agent_id"],
"type": claim["type"],
"proposition": claim["proposition"],
"body_markdown": claim["body_markdown"],
"scope": claim["scope"],
"access_scope": claim["access_scope"],
"status": claim["status"],
"revision_criteria": claim["revision_criteria"],
"revision_kind": "original",
"supersedes_id": None,
"tags": claim["tags"],
"created_by": manifest["agent_id"],
}
)
for evidence in claim["evidence"]:
binding = binding_by_key[(claim["claim_key"], evidence["evidence_key"])]
evidence_rows.append(
{
"id": stable_uuid_v3(
"evidence",
packet_identity_sha256,
[claim["claim_key"], evidence["evidence_key"]],
),
"claim_id": claim_id,
"source_id": source_id,
"polarity": evidence["polarity"],
"excerpt": evidence["quote"],
"locator_json": _v3_locator_json(manifest, binding),
"source_content_hash": manifest["artifact_sha256"],
"rationale": evidence["rationale"],
"created_by": manifest["agent_id"],
}
)
for claim in manifest["claims"]:
claim_id = claim_ids[claim["claim_key"]]
assessment = claim["assessment"]
evidence_set_hash = ap._v3_evidence_set_hash(claim_id, evidence_rows)
assessment_rows.append(
{
"id": stable_uuid_v3("assessment", packet_identity_sha256, claim["claim_key"]),
"claim_id": claim_id,
"support_tier": assessment["support_tier"],
"challenge_tier": assessment["challenge_tier"],
"overall_state": assessment["overall_state"],
"rationale": assessment["rationale"],
"evidence_set_hash": evidence_set_hash,
"supersedes_assessment_id": None,
}
)
parent_packet_sha256 = canonical_sha256(parent)
apply_payload = {
"contract_version": 3,
"source_proposal_id": parent["id"],
"agent_id": manifest["agent_id"],
"claims": claim_rows,
"sources": [source_row],
"evidence": evidence_rows,
"edges": [],
"assessments": assessment_rows,
"reasoning_tools": [],
"normalization_manifest": {
"schema": "livingip.kbSourceProposalNormalization.v3",
"semantic_extraction_statement": (
"The semantic extraction is a retained input; this is a reproducible learning input, "
"not a model-training claim."
),
"semantic_extraction_is_retained_input": True,
"model_training_claim": False,
"manifest_schema": MANIFEST_SCHEMA_V3,
"manifest_file_sha256": identity_material["manifest_file_sha256"],
"manifest_identity_binding": "sha256(exact duplicate-free UTF-8 JSON file bytes)",
"manifest_canonical_sha256": manifest_canonical_sha256,
"artifact_sha256": manifest["artifact_sha256"],
"extracted_text_sha256": manifest["extracted_text_sha256"],
"extractor": manifest["extractor"],
"source_identity": source["identity"],
"source_locator": source["locator"],
"normalized_candidate_content_sha256": candidate_content_sha256,
"identity_material": identity_material,
"packet_identity_sha256": packet_identity_sha256,
"parent_packet_sha256": parent_packet_sha256,
"quote_bindings": quote_bindings,
},
}
payload = {"apply_payload": apply_payload}
payload_sha256 = canonical_sha256(payload)
child_id = stable_uuid_v3("pending_review_proposal", packet_identity_sha256, payload_sha256)
_ensure_distinct_v3_ids(
[
parent["id"],
child_id,
source_id,
*[row["id"] for row in claim_rows],
*[row["id"] for row in evidence_rows],
*[row["id"] for row in assessment_rows],
]
)
return {
"id": child_id,
"proposal_type": "approve_claim",
"status": "pending_review",
"proposed_by_handle": parent["proposed_by_handle"],
"proposed_by_agent_id": manifest["agent_id"],
"channel": parent["channel"],
"source_ref": f"source-compiler-v3:{packet_identity_sha256[:32]}:{payload_sha256[:16]}",
"rationale": f"Strict V3 source proposal compiled from retained packet {parent['id']}.",
"payload": payload,
"parent_packet_sha256": parent_packet_sha256,
"payload_sha256": payload_sha256,
}
def _validate_v3_compiled_child(
child: dict[str, Any], manifest: dict[str, Any], quote_bindings: list[dict[str, Any]]
) -> None:
if child.get("status") != "pending_review" or child.get("proposal_type") != "approve_claim":
raise CompilerError("V3 compilation did not produce one pending_review approve_claim proposal")
payload = child.get("payload")
apply_payload = payload.get("apply_payload") if isinstance(payload, dict) else None
if not isinstance(apply_payload, dict) or apply_payload.get("contract_version") != 3:
raise CompilerError("strict child has no schema-valid approve_claim V3 apply_payload")
if apply_payload.get("reasoning_tools") != [] or apply_payload.get("edges") != []:
raise CompilerError("V3 source compilation admits neither reasoning_tools nor inferred edges")
if child.get("payload_sha256") != canonical_sha256(payload):
raise CompilerError("strict child payload_sha256 does not bind the exact V3 payload")
claims = apply_payload.get("claims") or []
sources = apply_payload.get("sources") or []
evidence = apply_payload.get("evidence") or []
assessments = apply_payload.get("assessments") or []
if len(claims) != len(manifest["claims"]) or len(sources) != 1:
raise CompilerError("V3 strict child row counts differ from the extraction manifest")
if len(evidence) != sum(len(claim["evidence"]) for claim in manifest["claims"]):
raise CompilerError("V3 strict child evidence count differs from the extraction manifest")
if len(assessments) != len(claims):
raise CompilerError("V3 strict child requires exactly one assessment per claim")
_ensure_distinct_v3_ids(
[
child["id"],
*[row.get("id") for row in sources],
*[row.get("id") for row in claims],
*[row.get("id") for row in evidence],
*[row.get("id") for row in assessments],
]
)
source_row = sources[0]
if source_row.get("content_hash") != manifest["artifact_sha256"]:
raise CompilerError("V3 source content_hash does not bind the original artifact")
expected_url, expected_storage_uri = _locator_fields(manifest["source"]["locator"])
if source_row.get("canonical_url") != expected_url or source_row.get("storage_uri") != expected_storage_uri:
raise CompilerError("V3 source row does not retain the validated content-addressed locator")
if source_row.get("captured_at") != _require_canonical_utc_timestamp(
source_row.get("captured_at"), "V3 source captured_at"
):
raise CompilerError("V3 source captured_at is not in writer-canonical fixed-microsecond UTC Z form")
binding_by_locator = {canonical_sha256(_v3_locator_json(manifest, binding)): binding for binding in quote_bindings}
for row in evidence:
binding = binding_by_locator.get(canonical_sha256(row.get("locator_json")))
if binding is None or row.get("excerpt") != binding["quote"]:
raise CompilerError("V3 evidence does not retain an exact locator-bound quote")
if row.get("source_content_hash") != manifest["artifact_sha256"]:
raise CompilerError("V3 evidence source_content_hash does not bind the original artifact")
assessment_by_claim = {row.get("claim_id"): row for row in assessments}
if len(assessment_by_claim) != len(claims):
raise CompilerError("V3 strict child contains duplicate assessment claim ids")
for claim in claims:
assessment = assessment_by_claim.get(claim.get("id"))
if assessment is None:
raise CompilerError("V3 strict child is missing a claim-local evidence assessment")
if assessment.get("evidence_set_hash") != ap._v3_evidence_set_hash(claim["id"], evidence):
raise CompilerError("V3 assessment evidence_set_hash does not match claim-local evidence")
def _compile_v3_source_packet(
artifact_bytes: bytes,
text_bytes: bytes,
manifest_bytes: bytes,
extractor_spec_bytes: bytes,
manifest_object: dict[str, Any],
) -> dict[str, Any]:
artifact_sha256 = sha256_bytes(artifact_bytes)
text_sha256 = sha256_bytes(text_bytes)
extractor_spec_sha256 = sha256_bytes(extractor_spec_bytes)
manifest_file_sha256 = sha256_bytes(manifest_bytes)
manifest, quote_bindings = _validate_manifest_v3(
manifest_object,
artifact_sha256,
text_sha256,
extractor_spec_sha256,
text_bytes,
)
manifest_canonical_sha256 = canonical_sha256(manifest)
identity_material, candidate_content_sha256, packet_identity_sha256 = _v3_identity_material(
manifest, manifest_file_sha256
)
parent = _build_v3_parent_proposal(
manifest,
manifest_canonical_sha256,
identity_material,
candidate_content_sha256,
packet_identity_sha256,
)
child = _build_v3_strict_child(
manifest,
manifest_canonical_sha256,
quote_bindings,
parent,
identity_material,
candidate_content_sha256,
packet_identity_sha256,
)
_validate_v3_compiled_child(child, manifest, quote_bindings)
stage_sql = stage.build_stage_sql(child)
stable_bundle = {
"schema": BUNDLE_SCHEMA_V3,
"status": "pending_review",
"build_only": True,
"database_write_performed": False,
"canonical_apply_performed": False,
"inputs": {
"artifact_bytes": len(artifact_bytes),
"extracted_text_bytes": len(text_bytes),
"manifest_bytes": len(manifest_bytes),
"extractor_spec_bytes": len(extractor_spec_bytes),
},
"validated_manifest": manifest,
"quote_bindings": quote_bindings,
"parent_proposal": parent,
"strict_child_proposal": child,
"stage_preview": {
"target_table": "kb_stage.kb_proposals",
"target_status": "pending_review",
"executed": False,
"stage_sql_sha256": sha256_bytes(stage_sql.encode("utf-8")),
},
"compiler_contract": {
"manifest_contract": MANIFEST_SCHEMA_V3,
"output_contract": BUNDLE_SCHEMA_V3,
"approve_claim_contract_version": 3,
"strict_preflight": "scripts/compile_kb_source_packet.py::_validate_v3_compiled_child",
"writer_apply_invoked": False,
"writer_timestamp_contract": "fixed-microsecond UTC Z",
"manifest_identity_binding": "sha256(exact duplicate-free UTF-8 JSON file bytes)",
"extractor_spec_binding": "sha256(exact independently supplied artifact bytes)",
"semantic_extraction_is_retained_input": True,
"semantic_extraction_statement": (
"The semantic extraction is a retained input; this is the smallest reproducible learning input, "
"not a model-training claim."
),
"model_training_claim": False,
"source_from_chat_labels": False,
"database_stage_executed": False,
},
"hashes": {
"artifact_sha256": artifact_sha256,
"extracted_text_sha256": text_sha256,
"extractor_spec_sha256": extractor_spec_sha256,
"manifest_file_sha256": manifest_file_sha256,
"manifest_canonical_sha256": manifest_canonical_sha256,
"normalized_candidate_content_sha256": candidate_content_sha256,
"packet_identity_sha256": packet_identity_sha256,
"parent_packet_sha256": child["parent_packet_sha256"],
"strict_child_payload_sha256": child["payload_sha256"],
},
}
stable_bundle["hashes"]["bundle_content_sha256"] = canonical_sha256(stable_bundle)
return stable_bundle
def _validate_compiled_child(
child: dict[str, Any], manifest: dict[str, Any], quote_bindings: list[dict[str, Any]]
) -> None:
@ -503,7 +1291,12 @@ def _validate_compiled_child(
raise CompilerError("one or more exact quotes exceed the current canonical source excerpt capacity")
def compile_source_packet(artifact_path: Path, text_path: Path, manifest_path: Path) -> dict[str, Any]:
def compile_source_packet(
artifact_path: Path,
text_path: Path,
manifest_path: Path,
extractor_spec_path: Path | None = None,
) -> dict[str, Any]:
artifact_bytes = _read_required_file(artifact_path, "artifact")
text_bytes = _read_required_file(text_path, "extracted text")
manifest_bytes = _read_required_file(manifest_path, "extraction manifest")
@ -511,14 +1304,24 @@ def compile_source_packet(artifact_path: Path, text_path: Path, manifest_path: P
extracted_text = text_bytes.decode("utf-8", errors="strict")
except UnicodeDecodeError as exc:
raise CompilerError(f"extracted text must be valid UTF-8: {exc}") from exc
try:
manifest_raw = json.loads(manifest_bytes.decode("utf-8", errors="strict"))
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
raise CompilerError(f"extraction manifest must be valid UTF-8 JSON: {exc}") from exc
manifest_raw = _load_duplicate_free_json(manifest_bytes, "extraction manifest")
manifest_object = _require_object(manifest_raw, "manifest")
artifact_sha256 = sha256_bytes(artifact_bytes)
text_sha256 = sha256_bytes(text_bytes)
if manifest_object.get("schema") == MANIFEST_SCHEMA_V3:
if extractor_spec_path is None:
raise CompilerError("V3 requires an independently supplied extractor specification artifact")
extractor_spec_bytes = _read_required_file(extractor_spec_path, "extractor specification")
return _compile_v3_source_packet(
artifact_bytes,
text_bytes,
manifest_bytes,
extractor_spec_bytes,
manifest_object,
)
if extractor_spec_path is not None:
raise CompilerError("extractor specification input is only valid for a V3 manifest")
manifest, quote_bindings = _validate_manifest(manifest_object, artifact_sha256, text_sha256, extracted_text)
manifest_canonical_sha256 = canonical_sha256(manifest)
parent = build_parent_proposal(manifest, manifest_canonical_sha256, quote_bindings)
@ -573,6 +1376,11 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser.add_argument("--artifact", required=True, type=Path, help="raw source artifact; any binary format")
parser.add_argument("--text", required=True, type=Path, help="strict UTF-8 extraction of the artifact")
parser.add_argument("--manifest", required=True, type=Path, help="extraction manifest JSON")
parser.add_argument(
"--extractor-spec",
type=Path,
help="exact extractor specification artifact; required for V3 and rejected for V1/V2",
)
parser.add_argument("--output", required=True, type=Path, help="private path for the compiled proposal bundle")
return parser.parse_args(argv)
@ -612,9 +1420,11 @@ def main(argv: list[str] | None = None) -> int:
try:
output_path = args.output.resolve()
input_paths = {args.artifact.resolve(), args.text.resolve(), args.manifest.resolve()}
if args.extractor_spec is not None:
input_paths.add(args.extractor_spec.resolve())
if output_path in input_paths:
raise CompilerError("--output must not overwrite an input file")
bundle = compile_source_packet(args.artifact, args.text, args.manifest)
bundle = compile_source_packet(args.artifact, args.text, args.manifest, args.extractor_spec)
except (CompilerError, OSError, ValueError):
_print_status("rejected", reason="input_validation_failed")
return 2

713
scripts/export_kb_transition_replay_material.py Normal file
View file

@ -0,0 +1,713 @@
#!/usr/bin/env python3
"""Export exact private replay material for one guarded applied proposal.
The exporter is intentionally read-only. It validates an existing worker replay
receipt, reads the approved and applied proposal rows through one gate-owned
function, and atomically publishes deterministic genesis-ledger input.
"""
from __future__ import annotations
import argparse
import hashlib
import json
import os
import re
import stat
import sys
import tempfile
import uuid
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE))
import apply_proposal as apply_engine # noqa: E402
import kb_apply_replay_receipt as replay_receipt # noqa: E402
MATERIAL_ARTIFACT = "teleo_strict_proposal_replay_material"
MATERIAL_CONTRACT_VERSION = 1
MATERIAL_FIELDS = frozenset(
{
"artifact",
"contract_version",
"sequence",
"approved_proposal",
"approval_snapshot",
"applied_proposal",
"replay_receipt",
}
)
PROPOSAL_FIELDS = frozenset(
{
"id",
"proposal_type",
"status",
"proposed_by_handle",
"proposed_by_agent_id",
"channel",
"source_ref",
"rationale",
"payload",
"reviewed_by_handle",
"reviewed_by_agent_id",
"reviewed_at",
"review_note",
"applied_by_handle",
"applied_by_agent_id",
"applied_at",
"created_at",
"updated_at",
}
)
APPROVAL_FIELDS = frozenset(
{
"proposal_id",
"proposal_type",
"payload",
"reviewed_by_handle",
"reviewed_by_agent_id",
"reviewed_by_db_role",
"reviewed_at",
"review_note",
}
)
PROPOSAL_TRANSITION_FIELDS = frozenset(
{"status", "applied_by_handle", "applied_by_agent_id", "applied_at", "updated_at"}
)
PROPOSAL_UUID_FIELDS = frozenset({"id", "proposed_by_agent_id", "reviewed_by_agent_id", "applied_by_agent_id"})
PROPOSAL_TIMESTAMP_FIELDS = frozenset({"reviewed_at", "applied_at", "created_at", "updated_at"})
APPROVAL_UUID_FIELDS = frozenset({"proposal_id", "reviewed_by_agent_id"})
APPROVAL_TIMESTAMP_FIELDS = frozenset({"reviewed_at"})
RECEIPT_PROPOSAL_FIELDS = (
"id",
"proposal_type",
"status",
"payload",
"reviewed_by_handle",
"reviewed_by_agent_id",
"reviewed_at",
"review_note",
"applied_by_handle",
"applied_by_agent_id",
"applied_at",
)
RECEIPT_PROPOSAL_FIELD_SET = frozenset(RECEIPT_PROPOSAL_FIELDS)
LEGACY_RECEIPT_TIMESTAMP_FIELDS = frozenset({"captured_at", "created_at", "updated_at", "reviewed_at", "applied_at"})
LEGACY_POSTGRES_UTC_TIMESTAMP = re.compile(
r"^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}(?:\.[0-9]{1,6})?\+00$"
)
class ExportError(RuntimeError):
"""A fail-closed error that contains no proposal payload material."""
def _canonical_json(value: Any) -> str:
return json.dumps(value, ensure_ascii=False, separators=(",", ":"), sort_keys=True)
def _same_json(left: Any, right: Any) -> bool:
return _canonical_json(left) == _canonical_json(right)
def _sha256_json(value: Any) -> str:
return hashlib.sha256(_canonical_json(value).encode("utf-8")).hexdigest()
def _reject_duplicate_keys(pairs: list[tuple[str, Any]]) -> dict[str, Any]:
result: dict[str, Any] = {}
for key, value in pairs:
if key in result:
raise ExportError("JSON input contains duplicate object fields")
result[key] = value
return result
def _reject_nonfinite_number(value: str) -> None:
raise ExportError(f"JSON input contains invalid numeric constant {value}")
def _parse_json_object(raw: str, *, label: str) -> dict[str, Any]:
try:
value = json.loads(
raw,
object_pairs_hook=_reject_duplicate_keys,
parse_constant=_reject_nonfinite_number,
)
except (json.JSONDecodeError, UnicodeError) as exc:
raise ExportError(f"{label} is not valid UTF-8 JSON") from exc
if not isinstance(value, dict):
raise ExportError(f"{label} must contain one JSON object")
return value
def _absolute_path(value: str | Path) -> Path:
path = Path(value).expanduser()
if not path.is_absolute():
path = Path.cwd() / path
return path
def _load_private_receipt(path: Path) -> dict[str, Any]:
path = _absolute_path(path)
try:
metadata = path.lstat()
except OSError as exc:
raise ExportError("worker replay receipt is unavailable") from exc
if path.is_symlink() or not stat.S_ISREG(metadata.st_mode):
raise ExportError("worker replay receipt must be a regular non-symlink file")
if stat.S_IMODE(metadata.st_mode) != 0o600:
raise ExportError("worker replay receipt permissions must be exactly 0600")
try:
raw = path.read_text(encoding="utf-8")
except (OSError, UnicodeError) as exc:
raise ExportError("worker replay receipt is unreadable") from exc
return _parse_json_object(raw, label="worker replay receipt")
def _require_exact_fields(value: Any, expected: frozenset[str], *, label: str) -> dict[str, Any]:
if not isinstance(value, dict):
raise ExportError(f"{label} must be an object")
observed = frozenset(value)
if observed != expected:
raise ExportError(
f"{label} has invalid fields; missing_count={len(expected - observed)}, "
f"extra_count={len(observed - expected)}"
)
return value
def _canonical_uuid(value: Any, *, label: str, nullable: bool = False) -> str | None:
if value is None and nullable:
return None
try:
return str(uuid.UUID(str(value)))
except (AttributeError, TypeError, ValueError) as exc:
raise ExportError(f"{label} must be a UUID") from exc
def _canonical_timestamp(value: Any, *, label: str, nullable: bool = False) -> str | None:
if value is None and nullable:
return None
try:
return replay_receipt.canonical_utc_timestamp(value, path=label)
except ValueError as exc:
raise ExportError(f"{label} must be a timezone-aware timestamp") from exc
def _canonical_legacy_postgres_timestamp(value: Any, *, label: str) -> str:
if not isinstance(value, str) or not LEGACY_POSTGRES_UTC_TIMESTAMP.fullmatch(value):
raise ExportError(f"{label} must use the exact legacy PostgreSQL UTC timestamp format")
timestamp_format = "%Y-%m-%d %H:%M:%S.%f%z" if "." in value else "%Y-%m-%d %H:%M:%S%z"
try:
parsed = datetime.strptime(value + ":00", timestamp_format)
except ValueError as exc:
raise ExportError(f"{label} must be a valid legacy PostgreSQL UTC timestamp") from exc
return parsed.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ")
def _normalize_proposal(value: Any, *, label: str) -> dict[str, Any]:
row = dict(_require_exact_fields(value, PROPOSAL_FIELDS, label=label))
for field in PROPOSAL_UUID_FIELDS:
row[field] = _canonical_uuid(
row[field],
label=f"{label}.{field}",
nullable=field != "id",
)
for field in PROPOSAL_TIMESTAMP_FIELDS:
row[field] = _canonical_timestamp(
row[field],
label=f"{label}.{field}",
nullable=field in {"reviewed_at", "applied_at"},
)
if not isinstance(row["proposal_type"], str) or not row["proposal_type"]:
raise ExportError(f"{label}.proposal_type must be a non-empty string")
if not isinstance(row["status"], str) or not row["status"]:
raise ExportError(f"{label}.status must be a non-empty string")
if not isinstance(row["payload"], dict):
raise ExportError(f"{label}.payload must be an object")
for field in (
PROPOSAL_FIELDS
- PROPOSAL_UUID_FIELDS
- PROPOSAL_TIMESTAMP_FIELDS
- {
"payload",
}
):
if field in {"proposal_type", "status"}:
continue
if row[field] is not None and not isinstance(row[field], str):
raise ExportError(f"{label}.{field} must be text or null")
return row
def _normalize_approval(value: Any, *, label: str) -> dict[str, Any]:
row = dict(_require_exact_fields(value, APPROVAL_FIELDS, label=label))
for field in APPROVAL_UUID_FIELDS:
row[field] = _canonical_uuid(row[field], label=f"{label}.{field}")
for field in APPROVAL_TIMESTAMP_FIELDS:
row[field] = _canonical_timestamp(row[field], label=f"{label}.{field}")
if not isinstance(row["payload"], dict):
raise ExportError(f"{label}.payload must be an object")
for field in APPROVAL_FIELDS - APPROVAL_UUID_FIELDS - APPROVAL_TIMESTAMP_FIELDS - {"payload"}:
if not isinstance(row[field], str) or not row[field].strip():
raise ExportError(f"{label}.{field} must be non-empty text")
return row
def _normalize_receipt_proposal(value: Any, *, label: str) -> dict[str, Any]:
row = dict(_require_exact_fields(value, RECEIPT_PROPOSAL_FIELD_SET, label=label))
for field in RECEIPT_PROPOSAL_FIELD_SET & PROPOSAL_UUID_FIELDS:
row[field] = _canonical_uuid(
row[field],
label=f"{label}.{field}",
nullable=field != "id",
)
for field in RECEIPT_PROPOSAL_FIELD_SET & PROPOSAL_TIMESTAMP_FIELDS:
row[field] = _canonical_timestamp(
row[field],
label=f"{label}.{field}",
nullable=False,
)
return row
def _normalize_legacy_receipt_proposal(value: Any, *, label: str) -> dict[str, Any]:
row = dict(_require_exact_fields(value, RECEIPT_PROPOSAL_FIELD_SET, label=label))
for field in RECEIPT_PROPOSAL_FIELD_SET & PROPOSAL_UUID_FIELDS:
row[field] = _canonical_uuid(
row[field],
label=f"{label}.{field}",
nullable=field != "id",
)
for field in RECEIPT_PROPOSAL_FIELD_SET & PROPOSAL_TIMESTAMP_FIELDS:
row[field] = _canonical_legacy_postgres_timestamp(
row[field],
label=f"{label}.{field}",
)
return row
def _normalize_legacy_receipt_rows(value: Any) -> dict[str, list[dict[str, Any]]]:
if not isinstance(value, dict):
raise ExportError("worker replay receipt canonical rows must be an object")
normalized: dict[str, list[dict[str, Any]]] = {}
for table, rows in value.items():
if not isinstance(table, str) or not isinstance(rows, list) or any(not isinstance(row, dict) for row in rows):
raise ExportError("worker replay receipt canonical rows must map table names to row arrays")
normalized_rows: list[dict[str, Any]] = []
for index, row in enumerate(rows):
normalized_row: dict[str, Any] = {}
for field, field_value in row.items():
path = f"replay_receipt.canonical_rows.{table}[{index}].{field}"
if field in LEGACY_RECEIPT_TIMESTAMP_FIELDS and field_value is not None:
normalized_row[field] = _canonical_legacy_postgres_timestamp(field_value, label=path)
else:
try:
normalized_row[field] = replay_receipt.normalize_typed_value(
field,
field_value,
path=path,
)
except ValueError as exc:
raise ExportError("worker replay receipt contains an invalid typed row value") from exc
normalized_rows.append(normalized_row)
normalized[table] = normalized_rows
return normalized
def _has_legacy_postgres_timestamp(proposal: dict[str, Any], rows: Any) -> bool:
values = [proposal.get("reviewed_at"), proposal.get("applied_at")]
if isinstance(rows, dict):
values.extend(
row.get(field)
for table_rows in rows.values()
if isinstance(table_rows, list)
for row in table_rows
if isinstance(row, dict)
for field in LEGACY_RECEIPT_TIMESTAMP_FIELDS
)
return any(isinstance(value, str) and LEGACY_POSTGRES_UTC_TIMESTAMP.fullmatch(value) for value in values)
def _validate_legacy_worker_replay_receipt(
receipt: dict[str, Any],
*,
expected_proposal_id: str | None,
) -> dict[str, Any]:
proposal = receipt.get("proposal")
rows = receipt.get("canonical_rows")
apply_metadata = receipt.get("apply_engine")
if not isinstance(proposal, dict) or not isinstance(rows, dict) or not isinstance(apply_metadata, dict):
raise ExportError("worker replay receipt is missing proposal, rows, or apply metadata")
if not _has_legacy_postgres_timestamp(proposal, rows):
raise ExportError("worker replay receipt is not a supported legacy PostgreSQL rendering")
normalized_proposal = _normalize_legacy_receipt_proposal(proposal, label="replay_receipt.proposal")
if any(
proposal[field] is not None and proposal[field] != normalized_proposal[field]
for field in RECEIPT_PROPOSAL_FIELD_SET & PROPOSAL_UUID_FIELDS
):
raise ExportError("legacy worker replay receipt proposal UUIDs are not canonical")
if expected_proposal_id is not None and normalized_proposal["id"] != _canonical_uuid(
expected_proposal_id,
label="expected_proposal_id",
):
raise ExportError("worker replay receipt proposal id does not match")
normalized_rows = _normalize_legacy_receipt_rows(rows)
if any(
row.get(field) is not None and row.get(field) != normalized_rows[table][index].get(field)
for table, table_rows in rows.items()
for index, row in enumerate(table_rows)
for field in row
if field not in LEGACY_RECEIPT_TIMESTAMP_FIELDS
):
raise ExportError("legacy worker replay receipt row UUIDs are not canonical")
exported_at_utc = _canonical_timestamp(
receipt.get("exported_at_utc"),
label="replay_receipt.exported_at_utc",
)
try:
normalized_receipt = replay_receipt.build_replay_receipt(
normalized_proposal,
normalized_rows,
apply_sql_sha256=apply_metadata.get("apply_sql_sha256"),
apply_sql_source=apply_metadata.get("source"),
exported_at_utc=exported_at_utc,
)
except (KeyError, TypeError, ValueError) as exc:
raise ExportError("legacy worker replay receipt failed strict payload and row validation") from exc
expected_counts = normalized_receipt["expected_row_counts"]
if any(not isinstance(rows.get(table), list) for table in expected_counts):
raise ExportError("legacy worker replay receipt is missing an expected row array")
raw_rows = {table: sorted(rows.get(table, []), key=_canonical_json) for table in expected_counts}
actual_counts = {table: len(table_rows) for table, table_rows in raw_rows.items()}
raw_apply_metadata = {
"apply_sql_sha256": apply_metadata.get("apply_sql_sha256"),
"source": apply_metadata.get("source"),
}
replay_material = {
"receipt_contract_version": replay_receipt.RECEIPT_CONTRACT_VERSION,
"apply_engine": raw_apply_metadata,
"proposal": proposal,
"canonical_rows": raw_rows,
}
raw_apply_payload = proposal.get("payload", {}).get("apply_payload")
expected_receipt = {
**replay_material,
"artifact": "kb_apply_replay_receipt",
"exported_at_utc": receipt.get("exported_at_utc"),
"expected_row_counts": expected_counts,
"actual_row_counts": actual_counts,
"checks": normalized_receipt["checks"],
"hashes": {
"apply_payload_sha256": _sha256_json(raw_apply_payload),
"proposal_payload_sha256": _sha256_json(proposal.get("payload")),
"canonical_rows_sha256": _sha256_json(raw_rows),
"row_sha256": {table: [_sha256_json(row) for row in table_rows] for table, table_rows in raw_rows.items()},
"table_sha256": {table: _sha256_json(table_rows) for table, table_rows in raw_rows.items()},
"replay_material_sha256": _sha256_json(replay_material),
},
"replay_ready": True,
"privacy": "private receipt; may contain claim bodies and source excerpts; do not commit",
}
if not _same_json(receipt, expected_receipt):
raise ExportError("legacy worker replay receipt hashes or contract fields are inconsistent")
return normalized_receipt
def validate_worker_replay_receipt(
receipt: dict[str, Any],
*,
expected_proposal_id: str | None = None,
) -> dict[str, Any]:
"""Validate a current receipt or one exact legacy PostgreSQL timestamp rendering."""
try:
return replay_receipt.validate_replay_receipt(
receipt,
expected_proposal_id=expected_proposal_id,
)
except ValueError:
return _validate_legacy_worker_replay_receipt(
receipt,
expected_proposal_id=expected_proposal_id,
)
def _parse_timestamp(value: str) -> datetime:
return datetime.fromisoformat(value.replace("Z", "+00:00"))
def validate_material(
material: dict[str, Any],
*,
expected_proposal_id: str | None = None,
expected_sequence: int | None = None,
) -> dict[str, Any]:
"""Validate and canonicalize the exact seven-field replay material."""
_require_exact_fields(material, MATERIAL_FIELDS, label="replay material")
if material["artifact"] != MATERIAL_ARTIFACT:
raise ExportError("replay material artifact is unsupported")
if type(material["contract_version"]) is not int or material["contract_version"] != MATERIAL_CONTRACT_VERSION:
raise ExportError("replay material contract version is unsupported")
sequence = material["sequence"]
if type(sequence) is not int or sequence <= 0:
raise ExportError("replay material sequence must be a positive integer")
if expected_sequence is not None and sequence != expected_sequence:
raise ExportError("replay material sequence does not match the requested sequence")
approved = _normalize_proposal(material["approved_proposal"], label="approved_proposal")
approval = _normalize_approval(material["approval_snapshot"], label="approval_snapshot")
applied = _normalize_proposal(material["applied_proposal"], label="applied_proposal")
proposal_id = _canonical_uuid(applied["id"], label="applied_proposal.id")
if expected_proposal_id is not None and proposal_id != _canonical_uuid(
expected_proposal_id, label="expected_proposal_id"
):
raise ExportError("database proposal id does not match the requested proposal")
if approved["id"] != proposal_id or approval["proposal_id"] != proposal_id:
raise ExportError("approved, approval, and applied proposal ids do not match")
if approved["status"] != "approved" or applied["status"] != "applied":
raise ExportError("replay material requires exact approved and applied proposal states")
if any(approved[field] is not None for field in ("applied_by_handle", "applied_by_agent_id", "applied_at")):
raise ExportError("approved proposal snapshot already contains apply metadata")
if not applied["applied_by_handle"] or not applied["applied_by_agent_id"] or not applied["applied_at"]:
raise ExportError("applied proposal row lacks exact apply metadata")
if not approved["reviewed_by_handle"] or not approved["reviewed_by_agent_id"] or not approved["reviewed_at"]:
raise ExportError("approved proposal row lacks exact review metadata")
changed_fields = {field for field in PROPOSAL_FIELDS if not _same_json(approved[field], applied[field])}
if changed_fields != PROPOSAL_TRANSITION_FIELDS:
raise ExportError("proposal rows differ outside or omit the exact guarded apply transition")
approval_projection = {
"proposal_id": approved["id"],
"proposal_type": approved["proposal_type"],
"payload": approved["payload"],
"reviewed_by_handle": approved["reviewed_by_handle"],
"reviewed_by_agent_id": approved["reviewed_by_agent_id"],
"reviewed_at": approved["reviewed_at"],
"review_note": approved["review_note"],
}
for field, expected in approval_projection.items():
if not _same_json(approval[field], expected):
raise ExportError(f"approval snapshot does not match approved proposal field {field}")
receipt = material["replay_receipt"]
if not isinstance(receipt, dict):
raise ExportError("replay_receipt must be an object")
try:
normalized_receipt = validate_worker_replay_receipt(receipt, expected_proposal_id=proposal_id)
except (ExportError, ValueError) as exc:
raise ExportError("worker replay receipt failed its exact row-level contract") from exc
receipt_proposal = receipt.get("proposal")
if not isinstance(receipt_proposal, dict):
raise ExportError("worker replay receipt has no proposal envelope")
normalized_receipt_proposal = _normalize_receipt_proposal(
normalized_receipt.get("proposal"),
label="replay_receipt.proposal",
)
applied_envelope = {field: applied[field] for field in RECEIPT_PROPOSAL_FIELDS}
if not _same_json(normalized_receipt_proposal, applied_envelope):
raise ExportError("worker replay receipt does not match the exact applied proposal row")
created_at = _parse_timestamp(approved["created_at"])
reviewed_at = _parse_timestamp(approved["reviewed_at"])
approved_updated_at = _parse_timestamp(approved["updated_at"])
applied_at = _parse_timestamp(applied["applied_at"])
applied_updated_at = _parse_timestamp(applied["updated_at"])
if not created_at <= reviewed_at <= approved_updated_at < applied_at <= applied_updated_at:
raise ExportError("proposal transition timestamps are not in exact lifecycle order")
return {
"artifact": MATERIAL_ARTIFACT,
"contract_version": MATERIAL_CONTRACT_VERSION,
"sequence": sequence,
"approved_proposal": approved,
"approval_snapshot": approval,
"applied_proposal": applied,
"replay_receipt": receipt,
}
def fetch_transition_rows(args: argparse.Namespace, password: str, proposal_id: str) -> dict[str, Any]:
sql = f"""begin transaction isolation level repeatable read read only;
select kb_stage.export_applied_proposal_replay_rows(
{apply_engine.sql_literal(proposal_id)}::uuid
)::text;
rollback;
"""
raw = apply_engine.run_psql(args, sql, password, redact_output_on_error=True).strip()
if not raw or "\n" in raw:
raise ExportError("applied transition export returned an invalid row count")
return _parse_json_object(raw, label="applied transition export")
def _resolved_output(path: Path) -> Path:
absolute = _absolute_path(path)
if absolute.name in {"", ".", ".."}:
raise ExportError("output path must name a file")
return absolute.parent.resolve(strict=False) / absolute.name
def _paths_alias(left: Path, right: Path) -> bool:
if left.resolve(strict=False) == right.resolve(strict=False):
return True
try:
return left.exists() and right.exists() and os.path.samefile(left, right)
except OSError:
return False
def _assert_output_safe(path: Path, protected_paths: tuple[Path, ...]) -> None:
for protected in protected_paths:
if _paths_alias(path, protected):
raise ExportError("output path aliases a protected input or engine file")
try:
metadata = path.lstat()
except FileNotFoundError:
return
if path.is_symlink() or not stat.S_ISREG(metadata.st_mode):
raise ExportError("output path must be a regular non-symlink file")
if metadata.st_nlink != 1:
raise ExportError("output path has unsafe hard-link aliases")
if stat.S_IMODE(metadata.st_mode) != 0o600:
raise ExportError("existing output permissions must be exactly 0600")
def _sync_directory(path: Path) -> None:
directory_fd = os.open(path, os.O_RDONLY)
try:
os.fsync(directory_fd)
finally:
os.close(directory_fd)
def write_private_material(
path: Path,
material: dict[str, Any],
*,
protected_paths: tuple[Path, ...],
) -> tuple[Path, str]:
"""Publish once, or accept an exact 0600 rerun without overwriting drift."""
output = _resolved_output(path)
output.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
_assert_output_safe(output, protected_paths)
payload = (json.dumps(material, ensure_ascii=False, indent=2, sort_keys=True) + "\n").encode("utf-8")
if output.exists():
if output.read_bytes() != payload:
raise ExportError("existing output contains different replay material")
return output, "unchanged"
fd, temporary_name = tempfile.mkstemp(prefix=f".{output.name}.", dir=str(output.parent))
temporary = Path(temporary_name)
published = False
try:
os.fchmod(fd, 0o600)
with os.fdopen(fd, "wb") as handle:
handle.write(payload)
handle.flush()
os.fsync(handle.fileno())
try:
os.link(temporary, output)
published = True
except FileExistsError:
_assert_output_safe(output, protected_paths)
if output.read_bytes() != payload:
raise ExportError("output changed concurrently during replay material export") from None
_sync_directory(output.parent)
finally:
try:
temporary.unlink()
except FileNotFoundError:
pass
_assert_output_safe(output, protected_paths)
if output.read_bytes() != payload:
raise ExportError("published replay material failed exact readback")
return output, "created" if published else "unchanged"
def _positive_sequence(value: str) -> int:
try:
sequence = int(value)
except ValueError as exc:
raise argparse.ArgumentTypeError("sequence must be a positive integer") from exc
if sequence <= 0:
raise argparse.ArgumentTypeError("sequence must be a positive integer")
return sequence
def parse_args(argv: list[str]) -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("proposal_id", help="UUID of the applied proposal")
parser.add_argument("--replay-receipt", required=True, help="private 0600 worker replay receipt")
parser.add_argument("--sequence", required=True, type=_positive_sequence, help="positive ledger sequence")
parser.add_argument("--output", required=True, help="private replay material output path")
parser.add_argument("--secrets-file", default=apply_engine.DEFAULT_SECRETS_FILE)
parser.add_argument("--container", default=apply_engine.DEFAULT_CONTAINER)
parser.add_argument("--db", default=apply_engine.DEFAULT_DB)
parser.add_argument("--host", default=apply_engine.DEFAULT_HOST)
parser.add_argument("--role", default=apply_engine.DEFAULT_ROLE)
return parser.parse_args(argv)
def run(args: argparse.Namespace) -> int:
proposal_id = _canonical_uuid(args.proposal_id, label="proposal_id")
receipt_path = _absolute_path(args.replay_receipt)
receipt = _load_private_receipt(receipt_path)
try:
replay_receipt.validate_replay_receipt(receipt, expected_proposal_id=proposal_id)
except ValueError as exc:
raise ExportError("worker replay receipt failed its exact row-level contract") from exc
password = apply_engine.load_password(args.secrets_file)
rows = fetch_transition_rows(args, password, proposal_id)
material = validate_material(
{
"artifact": MATERIAL_ARTIFACT,
"contract_version": MATERIAL_CONTRACT_VERSION,
"sequence": args.sequence,
"approved_proposal": rows.get("approved_proposal"),
"approval_snapshot": rows.get("approval_snapshot"),
"applied_proposal": rows.get("applied_proposal"),
"replay_receipt": receipt,
},
expected_proposal_id=proposal_id,
expected_sequence=args.sequence,
)
protected_paths = tuple(
_absolute_path(path)
for path in (
receipt_path,
args.secrets_file,
Path(__file__),
Path(apply_engine.__file__),
Path(replay_receipt.__file__),
)
)
_, disposition = write_private_material(
Path(args.output),
material,
protected_paths=protected_paths,
)
print(f"replay material {disposition}: proposal={proposal_id} sequence={args.sequence}")
return 0
def main(argv: list[str] | None = None) -> int:
try:
return run(parse_args(sys.argv[1:] if argv is None else argv))
except ExportError as exc:
print(f"replay material export refused: {exc}", file=sys.stderr)
return 1
except OSError:
print("replay material export refused: private file operation failed", file=sys.stderr)
return 1
if __name__ == "__main__":
raise SystemExit(main())

390
scripts/kb_apply_prereqs.sql
View file

@ -10,6 +10,12 @@
--
-- The deployer provisions passwords separately in kb-review.env/kb-apply.env.
-- This file contains no credentials and is safe to reuse in disposable clones.
-- Before the first V3 contract migration, run this file, stop the V2 worker,
-- then ALTER ROLE kb_apply NOLOGIN and end its sessions. After V3 is installed,
-- this file may re-enable the login: a V2-configured worker detects the cutover
-- and leaves historical V2 approvals review-visible until migrate/reject.
set timezone = 'UTC';
begin;
@ -74,10 +80,54 @@ insert into public.agents (id, handle, kind)
values ('44444444-4444-4444-4444-444444444444', 'kb-apply', 'service')
on conflict (handle) do nothing;
-- Canonicalize the existing human reviewer identity without changing its UUID.
-- Historical proposal/approval reviewer text is deliberately left untouched.
do $reviewer_identity$
declare
v_legacy_id uuid;
v_canonical_id uuid;
begin
select id into v_legacy_id
from public.agents
where handle = 'm3ta'
for update;
select id into v_canonical_id
from public.agents
where handle = 'm3taversal'
for update;
if v_legacy_id is not null
and v_canonical_id is not null
and v_legacy_id <> v_canonical_id then
raise exception
'kb_apply_prereqs: both reviewer handles m3ta (%) and m3taversal (%) exist with different agent UUIDs; resolve the identity conflict before rerunning',
v_legacy_id, v_canonical_id;
end if;
if v_canonical_id is null and v_legacy_id is not null then
update public.agents
set handle = 'm3taversal'
where id = v_legacy_id
and handle = 'm3ta';
if not found then
raise exception
'kb_apply_prereqs: legacy reviewer m3ta changed during canonical identity migration';
end if;
v_canonical_id := v_legacy_id;
end if;
if v_canonical_id is null then
raise exception
'kb_apply_prereqs: required human reviewer agent is missing; restore exactly one public.agents row with handle m3ta or m3taversal before rerunning';
end if;
end
$reviewer_identity$;
-- DB-login to human-reviewer mapping. Only the gate owner/admin can change it;
-- the approval function verifies --reviewed-by against this mapping rather than
-- trusting caller-supplied identity. The current dedicated reviewer credential
-- maps to the existing canonical m3ta human agent when that row is present.
-- trusting caller-supplied identity. The dedicated reviewer credential always
-- maps to the canonical m3taversal human agent.
create table if not exists kb_stage.kb_review_principals (
db_role name primary key,
reviewed_by_handle text not null,
@ -96,7 +146,8 @@ create table if not exists kb_stage.kb_proposal_approvals (
reviewed_by_agent_id uuid not null references public.agents(id),
reviewed_by_db_role name not null,
reviewed_at timestamptz not null,
review_note text not null
review_note text not null,
approved_proposal_snapshot jsonb
);
-- Existing canonical/proposal tables must remain owned by the deployer. Gate
@ -119,22 +170,28 @@ begin
('public', 'strategies'),
('public', 'strategy_nodes'),
('public', 'claim_evidence'),
('public', 'claim_evidence_assessments'),
('public', 'claim_edges'),
('public', 'claims'),
('public', 'sources'),
('public', 'reasoning_tools'),
('kb_stage', 'kb_proposals'),
('kb_stage', 'kb_review_principals'),
('kb_stage', 'kb_proposal_approvals')
('kb_stage', 'kb_proposal_approvals'),
('kb_stage', 'teleo_v3_installation_ownership')
)
and relation.relkind in ('r', 'p')
and not (
(namespace.nspname = 'kb_stage'
and relation.relname in ('kb_review_principals', 'kb_proposal_approvals')
and relation.relname in (
'kb_review_principals', 'kb_proposal_approvals', 'teleo_v3_installation_ownership'
)
and owner_role.rolname in (current_user, 'kb_gate_owner'))
or (not (
namespace.nspname = 'kb_stage'
and relation.relname in ('kb_review_principals', 'kb_proposal_approvals')
and relation.relname in (
'kb_review_principals', 'kb_proposal_approvals', 'teleo_v3_installation_ownership'
)
)
and owner_role.rolname = current_user)
);
@ -161,7 +218,8 @@ begin
and procedure.proname in (
'approve_strict_proposal',
'assert_approved_proposal',
'finish_approved_proposal'
'finish_approved_proposal',
'export_applied_proposal_replay_rows'
)
and owner_role.rolname not in (current_user, 'kb_gate_owner');
@ -185,7 +243,8 @@ begin
and procedure.proname in (
'approve_strict_proposal',
'assert_approved_proposal',
'finish_approved_proposal'
'finish_approved_proposal',
'export_applied_proposal_replay_rows'
)
and not (
(procedure.proname = 'approve_strict_proposal'
@ -199,6 +258,8 @@ begin
or (procedure.proname = 'finish_approved_proposal'
and pg_catalog.oidvectortypes(procedure.proargtypes) =
'uuid, text, jsonb, text, uuid, timestamp with time zone, text, text')
or (procedure.proname = 'export_applied_proposal_replay_rows'
and pg_catalog.oidvectortypes(procedure.proargtypes) = 'uuid')
);
if v_drift is not null then
@ -224,8 +285,19 @@ begin
coalesce(relation.relacl, pg_catalog.acldefault('r', relation.relowner))
) acl
where (namespace.nspname, relation.relname) in (
('public', 'agents'),
('public', 'strategies'),
('public', 'strategy_nodes'),
('public', 'claim_evidence'),
('public', 'claim_evidence_assessments'),
('public', 'claim_edges'),
('public', 'claims'),
('public', 'sources'),
('public', 'reasoning_tools'),
('kb_stage', 'kb_proposals'),
('kb_stage', 'kb_review_principals'),
('kb_stage', 'kb_proposal_approvals')
('kb_stage', 'kb_proposal_approvals'),
('kb_stage', 'teleo_v3_installation_ownership')
)
and acl.grantee <> 0
and acl.grantee <> relation.relowner
@ -256,8 +328,19 @@ begin
join pg_catalog.pg_namespace namespace on namespace.oid = relation.relnamespace
cross join lateral pg_catalog.aclexplode(attribute.attacl) acl
where (namespace.nspname, relation.relname) in (
('public', 'agents'),
('public', 'strategies'),
('public', 'strategy_nodes'),
('public', 'claim_evidence'),
('public', 'claim_evidence_assessments'),
('public', 'claim_edges'),
('public', 'claims'),
('public', 'sources'),
('public', 'reasoning_tools'),
('kb_stage', 'kb_proposals'),
('kb_stage', 'kb_review_principals'),
('kb_stage', 'kb_proposal_approvals')
('kb_stage', 'kb_proposal_approvals'),
('kb_stage', 'teleo_v3_installation_ownership')
)
and not attribute.attisdropped
and acl.grantee <> 0
@ -293,7 +376,8 @@ begin
and procedure.proname in (
'approve_strict_proposal',
'assert_approved_proposal',
'finish_approved_proposal'
'finish_approved_proposal',
'export_applied_proposal_replay_rows'
)
and acl.grantee <> 0
and acl.grantee <> procedure.proowner
@ -314,13 +398,41 @@ drop function if exists kb_stage.approve_strict_proposal(uuid, text, text);
alter table kb_stage.kb_review_principals owner to kb_gate_owner;
alter table kb_stage.kb_proposal_approvals owner to kb_gate_owner;
alter table kb_stage.kb_proposal_approvals
add column if not exists approved_proposal_snapshot jsonb;
comment on column kb_stage.kb_proposal_approvals.approved_proposal_snapshot is
'Exact post-approval kb_proposals row. NULL means exact replay material is unavailable.';
-- A still-approved legacy row is the only state whose complete approved shape
-- remains recoverable without guessing. Already-applied rows stay NULL because
-- their pre-apply fields, especially updated_at, cannot be reconstructed.
update kb_stage.kb_proposal_approvals approval
set approved_proposal_snapshot = pg_catalog.to_jsonb(proposal)
from kb_stage.kb_proposals proposal
where approval.proposal_id = proposal.id
and approval.approved_proposal_snapshot is null
and proposal.status = 'approved'
and proposal.applied_by_handle is null
and proposal.applied_by_agent_id is null
and proposal.applied_at is null
and approval.proposal_type = proposal.proposal_type
and approval.payload = proposal.payload
and approval.reviewed_by_handle = proposal.reviewed_by_handle
and approval.reviewed_by_agent_id is not distinct from proposal.reviewed_by_agent_id
and approval.reviewed_at = proposal.reviewed_at
and approval.review_note = proposal.review_note;
insert into kb_stage.kb_review_principals
(db_role, reviewed_by_handle, reviewed_by_agent_id, active, created_at)
select 'kb_review'::name, a.handle, a.id, true,
select 'kb_review'::name, 'm3taversal', a.id, true,
'1970-01-01 00:00:00+00'::timestamptz
from public.agents a
where a.handle = 'm3ta'
on conflict (db_role) do nothing;
where a.handle = 'm3taversal'
on conflict (db_role) do update
set reviewed_by_handle = excluded.reviewed_by_handle,
reviewed_by_agent_id = excluded.reviewed_by_agent_id,
active = excluded.active;
-- Reset every explicit privilege held by the protected roles before granting
-- the documented matrix. This removes stale DELETE/TRUNCATE/REFERENCES/TRIGGER
@ -372,13 +484,15 @@ begin
('public', 'strategies'),
('public', 'strategy_nodes'),
('public', 'claim_evidence'),
('public', 'claim_evidence_assessments'),
('public', 'claim_edges'),
('public', 'claims'),
('public', 'sources'),
('public', 'reasoning_tools'),
('kb_stage', 'kb_proposals'),
('kb_stage', 'kb_review_principals'),
('kb_stage', 'kb_proposal_approvals')
('kb_stage', 'kb_proposal_approvals'),
('kb_stage', 'teleo_v3_installation_ownership')
)
and attribute.attnum > 0
and not attribute.attisdropped
@ -392,7 +506,8 @@ begin
);
if protected_table.schema_name = 'kb_stage'
and protected_table.table_name in (
'kb_proposals', 'kb_review_principals', 'kb_proposal_approvals'
'kb_proposals', 'kb_review_principals', 'kb_proposal_approvals',
'teleo_v3_installation_ownership'
) then
execute pg_catalog.format(
'revoke all privileges (%s) on table %I.%I from public',
@ -417,6 +532,45 @@ grant select, insert on public.claims to kb_apply;
grant select, insert on public.sources to kb_apply;
grant select, insert on public.reasoning_tools to kb_apply;
-- The V3 assessment table is installed by a separate additive migration. Keep
-- this prerequisite runnable before that migration, then normalize its ACLs on
-- the first post-migration rerun.
do $v3_assessment_acl$
declare
v_columns text;
begin
if pg_catalog.to_regclass('public.claim_evidence_assessments') is not null then
execute 'revoke all privileges on table public.claim_evidence_assessments '
'from public, kb_gate_owner, kb_apply, kb_review';
select pg_catalog.string_agg(pg_catalog.format('%I', attribute.attname), ', ' order by attribute.attnum)
into v_columns
from pg_catalog.pg_attribute attribute
where attribute.attrelid = 'public.claim_evidence_assessments'::regclass
and attribute.attnum > 0
and not attribute.attisdropped;
if v_columns is not null then
execute pg_catalog.format(
'revoke all privileges (%s) on table public.claim_evidence_assessments '
'from public, kb_gate_owner, kb_apply, kb_review',
v_columns
);
end if;
execute 'grant select, insert on public.claim_evidence_assessments to kb_apply';
execute 'grant select on public.claims, public.sources, public.claim_evidence, '
'public.claim_edges, public.claim_evidence_assessments to kb_gate_owner';
end if;
end
$v3_assessment_acl$;
do $v3_installation_ownership_acl$
begin
if pg_catalog.to_regclass('kb_stage.teleo_v3_installation_ownership') is not null then
execute 'revoke all privileges on table kb_stage.teleo_v3_installation_ownership '
'from public, kb_gate_owner, kb_apply, kb_review';
end if;
end
$v3_installation_ownership_acl$;
-- Keep the primary ledger denial explicit in addition to the full reset above.
revoke update on kb_stage.kb_proposals from kb_apply;
grant select on kb_stage.kb_proposals to kb_apply;
@ -479,6 +633,7 @@ create or replace function kb_stage.approve_strict_proposal(
language plpgsql
security definer
set search_path = pg_catalog, pg_temp
set timezone = 'UTC'
as $function$
declare
v_row kb_stage.kb_proposals%rowtype;
@ -527,14 +682,6 @@ begin
end if;
v_reviewed_at := pg_catalog.clock_timestamp();
insert into kb_stage.kb_proposal_approvals
(proposal_id, proposal_type, payload, reviewed_by_handle,
reviewed_by_agent_id, reviewed_by_db_role, reviewed_at, review_note)
values
(v_row.id, v_row.proposal_type, v_row.payload, v_principal.reviewed_by_handle,
v_principal.reviewed_by_agent_id, session_user::name, v_reviewed_at,
pg_catalog.btrim(p_review_note));
update kb_stage.kb_proposals
set status = 'approved',
reviewed_by_handle = v_principal.reviewed_by_handle,
@ -552,6 +699,15 @@ begin
p_proposal_id;
end if;
insert into kb_stage.kb_proposal_approvals
(proposal_id, proposal_type, payload, reviewed_by_handle,
reviewed_by_agent_id, reviewed_by_db_role, reviewed_at, review_note,
approved_proposal_snapshot)
values
(v_row.id, v_row.proposal_type, v_row.payload, v_principal.reviewed_by_handle,
v_principal.reviewed_by_agent_id, session_user::name, v_reviewed_at,
pg_catalog.btrim(p_review_note), pg_catalog.to_jsonb(v_row));
return pg_catalog.jsonb_build_object(
'id', v_row.id::text,
'proposal_type', v_row.proposal_type,
@ -579,6 +735,7 @@ create or replace function kb_stage.assert_approved_proposal(
language plpgsql
security definer
set search_path = pg_catalog, pg_temp
set timezone = 'UTC'
as $function$
begin
perform 1
@ -598,6 +755,7 @@ begin
and a.reviewed_by_agent_id is not distinct from p_reviewed_by_agent_id
and a.reviewed_at = p_reviewed_at
and a.review_note = p_review_note
and a.approved_proposal_snapshot = pg_catalog.to_jsonb(p)
for update of p;
if not found then
raise exception 'assert_approved_proposal: proposal % is not the exact immutable approved snapshot',
@ -623,6 +781,7 @@ create or replace function kb_stage.finish_approved_proposal(
language plpgsql
security definer
set search_path = pg_catalog, pg_temp
set timezone = 'UTC'
as $function$
declare
v_row kb_stage.kb_proposals%rowtype;
@ -649,6 +808,7 @@ begin
and a.reviewed_by_agent_id is not distinct from p_reviewed_by_agent_id
and a.reviewed_at = p_reviewed_at
and a.review_note = p_review_note
and a.approved_proposal_snapshot = pg_catalog.to_jsonb(p)
for update of p;
if not found then
raise exception 'finish_approved_proposal: proposal % is not the locked immutable approved snapshot',
@ -694,12 +854,89 @@ begin
end
$function$;
-- Read-only transition export. kb_apply receives only EXECUTE on this function,
-- never SELECT on the approval table. The stored approved row must be a complete
-- kb_proposals record and differ from the applied row only by the guarded apply
-- transition before any private material leaves PostgreSQL.
create or replace function kb_stage.export_applied_proposal_replay_rows(
p_proposal_id uuid
) returns jsonb
language plpgsql
stable
security definer
set search_path = pg_catalog, pg_temp
set timezone = 'UTC'
as $function$
declare
v_rows jsonb;
begin
select pg_catalog.jsonb_build_object(
'approved_proposal', approval.approved_proposal_snapshot,
'approval_snapshot',
pg_catalog.to_jsonb(approval) - 'approved_proposal_snapshot',
'applied_proposal', pg_catalog.to_jsonb(proposal)
)
into v_rows
from kb_stage.kb_proposals proposal
join kb_stage.kb_proposal_approvals approval
on approval.proposal_id = proposal.id
where proposal.id = p_proposal_id
and proposal.status = 'applied'
and proposal.applied_by_handle is not null
and proposal.applied_by_agent_id is not null
and proposal.applied_at is not null
and proposal.reviewed_by_handle is not null
and proposal.reviewed_by_agent_id is not null
and proposal.reviewed_at is not null
and proposal.review_note is not null
and nullif(pg_catalog.btrim(approval.reviewed_by_db_role::text), '') is not null
and pg_catalog.jsonb_typeof(approval.approved_proposal_snapshot) = 'object'
and approval.approved_proposal_snapshot = pg_catalog.to_jsonb(
pg_catalog.jsonb_populate_record(
null::kb_stage.kb_proposals,
approval.approved_proposal_snapshot
)
)
and approval.approved_proposal_snapshot->>'status' = 'approved'
and approval.approved_proposal_snapshot->'applied_by_handle' = 'null'::jsonb
and approval.approved_proposal_snapshot->'applied_by_agent_id' = 'null'::jsonb
and approval.approved_proposal_snapshot->'applied_at' = 'null'::jsonb
and approval.proposal_id = (approval.approved_proposal_snapshot->>'id')::uuid
and approval.proposal_type = approval.approved_proposal_snapshot->>'proposal_type'
and approval.payload = approval.approved_proposal_snapshot->'payload'
and approval.reviewed_by_handle =
approval.approved_proposal_snapshot->>'reviewed_by_handle'
and approval.reviewed_by_agent_id =
(approval.approved_proposal_snapshot->>'reviewed_by_agent_id')::uuid
and approval.reviewed_at =
(approval.approved_proposal_snapshot->>'reviewed_at')::timestamptz
and approval.review_note = approval.approved_proposal_snapshot->>'review_note'
and (
approval.approved_proposal_snapshot || pg_catalog.jsonb_build_object(
'status', proposal.status,
'applied_by_handle', proposal.applied_by_handle,
'applied_by_agent_id', proposal.applied_by_agent_id,
'applied_at', proposal.applied_at,
'updated_at', proposal.updated_at
)
) = pg_catalog.to_jsonb(proposal);
if v_rows is null then
raise exception 'export_applied_proposal_replay_rows: exact applied transition unavailable for proposal %',
p_proposal_id;
end if;
return v_rows;
end
$function$;
alter function kb_stage.approve_strict_proposal(uuid, text, jsonb, text, text)
owner to kb_gate_owner;
alter function kb_stage.assert_approved_proposal(uuid, text, jsonb, text, uuid, timestamptz, text)
owner to kb_gate_owner;
alter function kb_stage.finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamptz, text, text)
owner to kb_gate_owner;
alter function kb_stage.export_applied_proposal_replay_rows(uuid)
owner to kb_gate_owner;
revoke all on function kb_stage.approve_strict_proposal(uuid, text, jsonb, text, text)
from public, kb_gate_owner, kb_apply, kb_review;
@ -707,6 +944,8 @@ revoke all on function kb_stage.assert_approved_proposal(uuid, text, jsonb, text
from public, kb_gate_owner, kb_apply, kb_review;
revoke all on function kb_stage.finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamptz, text, text)
from public, kb_gate_owner, kb_apply, kb_review;
revoke all on function kb_stage.export_applied_proposal_replay_rows(uuid)
from public, kb_gate_owner, kb_apply, kb_review;
grant execute on function kb_stage.approve_strict_proposal(uuid, text, jsonb, text, text)
to kb_review;
@ -714,6 +953,8 @@ grant execute on function kb_stage.assert_approved_proposal(uuid, text, jsonb, t
to kb_apply;
grant execute on function kb_stage.finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamptz, text, text)
to kb_apply;
grant execute on function kb_stage.export_applied_proposal_replay_rows(uuid)
to kb_apply;
-- Assert the complete protected catalog contract before commit. These checks
-- make reruns an upgrade proof rather than a best-effort sequence of grants.
@ -755,6 +996,20 @@ begin
raise exception 'kb_apply_prereqs: protected role membership appeared during migration: %', v_drift;
end if;
if exists (select 1 from public.agents where handle = 'm3ta')
or not exists (
select 1
from kb_stage.kb_review_principals principal
join public.agents reviewer
on reviewer.id = principal.reviewed_by_agent_id
where principal.db_role = 'kb_review'::name
and principal.reviewed_by_handle = 'm3taversal'
and principal.active
and reviewer.handle = 'm3taversal'
) then
raise exception 'kb_apply_prereqs: kb_review principal did not normalize to active m3taversal';
end if;
select pg_catalog.string_agg(
pg_catalog.format(
'%I on %I usage=%s create=%s',
@ -792,21 +1047,27 @@ begin
('public', 'strategies'),
('public', 'strategy_nodes'),
('public', 'claim_evidence'),
('public', 'claim_evidence_assessments'),
('public', 'claim_edges'),
('public', 'claims'),
('public', 'sources'),
('public', 'reasoning_tools'),
('kb_stage', 'kb_proposals'),
('kb_stage', 'kb_review_principals'),
('kb_stage', 'kb_proposal_approvals')
('kb_stage', 'kb_proposal_approvals'),
('kb_stage', 'teleo_v3_installation_ownership')
)
and not (
(namespace.nspname = 'kb_stage'
and relation.relname in ('kb_review_principals', 'kb_proposal_approvals')
and relation.relname in (
'kb_review_principals', 'kb_proposal_approvals', 'teleo_v3_installation_ownership'
)
and owner_role.rolname = 'kb_gate_owner')
or (not (
namespace.nspname = 'kb_stage'
and relation.relname in ('kb_review_principals', 'kb_proposal_approvals')
and relation.relname in (
'kb_review_principals', 'kb_proposal_approvals', 'teleo_v3_installation_ownership'
)
)
and owner_role.rolname = current_user)
);
@ -815,11 +1076,24 @@ begin
raise exception 'kb_apply_prereqs: protected table ownership mismatch: %', v_drift;
end if;
if not exists (
select 1
from pg_catalog.pg_attribute attribute
where attribute.attrelid = 'kb_stage.kb_proposal_approvals'::regclass
and attribute.attname = 'approved_proposal_snapshot'
and attribute.atttypid = 'pg_catalog.jsonb'::regtype
and not attribute.attnotnull
and not attribute.attisdropped
) then
raise exception 'kb_apply_prereqs: approved proposal snapshot column is not nullable JSONB';
end if;
with expected(signature) as (
values
('approve_strict_proposal(uuid, text, jsonb, text, text)'),
('assert_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text)'),
('finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text, text)')
('finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text, text)'),
('export_applied_proposal_replay_rows(uuid)')
), actual(signature) as (
select procedure.proname || '(' || pg_catalog.oidvectortypes(procedure.proargtypes) || ')'
from pg_catalog.pg_proc procedure
@ -828,7 +1102,8 @@ begin
and procedure.proname in (
'approve_strict_proposal',
'assert_approved_proposal',
'finish_approved_proposal'
'finish_approved_proposal',
'export_applied_proposal_replay_rows'
)
), delta(kind, signature) as (
select 'unexpected', actual.signature
@ -867,13 +1142,25 @@ begin
and procedure.proname in (
'approve_strict_proposal',
'assert_approved_proposal',
'finish_approved_proposal'
'finish_approved_proposal',
'export_applied_proposal_replay_rows'
)
and (
owner_role.rolname <> 'kb_gate_owner'
or not procedure.prosecdef
or pg_catalog.array_to_string(procedure.proconfig, ',') is distinct from
'search_path=pg_catalog, pg_temp'
or pg_catalog.cardinality(procedure.proconfig) <> 2
or not procedure.proconfig @> array[
'search_path=pg_catalog, pg_temp'::text,
'TimeZone=UTC'::text
]
or (
procedure.proname = 'export_applied_proposal_replay_rows'
and (
procedure.provolatile <> 's'
or procedure.prokind <> 'f'
or procedure.prorettype <> 'pg_catalog.jsonb'::regtype
)
)
);
if v_drift is not null then
@ -886,13 +1173,15 @@ begin
('public', 'strategies'),
('public', 'strategy_nodes'),
('public', 'claim_evidence'),
('public', 'claim_evidence_assessments'),
('public', 'claim_edges'),
('public', 'claims'),
('public', 'sources'),
('public', 'reasoning_tools'),
('kb_stage', 'kb_proposals'),
('kb_stage', 'kb_review_principals'),
('kb_stage', 'kb_proposal_approvals')
('kb_stage', 'kb_proposal_approvals'),
('kb_stage', 'teleo_v3_installation_ownership')
)
select pg_catalog.string_agg(
pg_catalog.format(
@ -920,8 +1209,7 @@ begin
left join pg_catalog.pg_roles grantee_role on grantee_role.oid = acl.grantee
where attribute.attnum > 0
and not attribute.attisdropped
and (acl.grantee = 0
or grantee_role.rolname in ('kb_gate_owner', 'kb_review', 'kb_apply'));
and acl.grantee <> relation.relowner;
if v_drift is not null then
raise exception 'kb_apply_prereqs: protected column ACL mismatch: %', v_drift;
@ -952,19 +1240,42 @@ begin
('kb_stage', 'kb_proposals', 'kb_gate_owner', 'UPDATE', false),
('kb_stage', 'kb_proposals', 'kb_apply', 'SELECT', false),
('kb_stage', 'kb_proposals', 'kb_review', 'SELECT', false)
union all
select 'public', 'claim_evidence_assessments', 'kb_apply', 'SELECT', false
where pg_catalog.to_regclass('public.claim_evidence_assessments') is not null
union all
select 'public', 'claim_evidence_assessments', 'kb_apply', 'INSERT', false
where pg_catalog.to_regclass('public.claim_evidence_assessments') is not null
union all
select 'public', 'claims', 'kb_gate_owner', 'SELECT', false
where pg_catalog.to_regclass('public.claim_evidence_assessments') is not null
union all
select 'public', 'sources', 'kb_gate_owner', 'SELECT', false
where pg_catalog.to_regclass('public.claim_evidence_assessments') is not null
union all
select 'public', 'claim_evidence', 'kb_gate_owner', 'SELECT', false
where pg_catalog.to_regclass('public.claim_evidence_assessments') is not null
union all
select 'public', 'claim_edges', 'kb_gate_owner', 'SELECT', false
where pg_catalog.to_regclass('public.claim_evidence_assessments') is not null
union all
select 'public', 'claim_evidence_assessments', 'kb_gate_owner', 'SELECT', false
where pg_catalog.to_regclass('public.claim_evidence_assessments') is not null
), tracked(schema_name, object_name) as (
values
('public', 'agents'),
('public', 'strategies'),
('public', 'strategy_nodes'),
('public', 'claim_evidence'),
('public', 'claim_evidence_assessments'),
('public', 'claim_edges'),
('public', 'claims'),
('public', 'sources'),
('public', 'reasoning_tools'),
('kb_stage', 'kb_proposals'),
('kb_stage', 'kb_review_principals'),
('kb_stage', 'kb_proposal_approvals')
('kb_stage', 'kb_proposal_approvals'),
('kb_stage', 'teleo_v3_installation_ownership')
), actual(schema_name, object_name, grantee, privilege_type, is_grantable) as (
select distinct
namespace.nspname::text,
@ -985,8 +1296,6 @@ begin
) acl
left join pg_catalog.pg_roles grantee_role on grantee_role.oid = acl.grantee
where acl.grantee <> relation.relowner
and (acl.grantee = 0
or grantee_role.rolname in ('kb_gate_owner', 'kb_review', 'kb_apply'))
), delta(kind, schema_name, object_name, grantee, privilege_type, is_grantable) as (
select 'unexpected', actual.*
from actual
@ -1020,6 +1329,8 @@ begin
('assert_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text)',
'kb_apply', 'EXECUTE', false),
('finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text, text)',
'kb_apply', 'EXECUTE', false),
('export_applied_proposal_replay_rows(uuid)',
'kb_apply', 'EXECUTE', false)
), actual(signature, grantee, privilege_type, is_grantable) as (
select distinct
@ -1039,7 +1350,8 @@ begin
and procedure.proname in (
'approve_strict_proposal',
'assert_approved_proposal',
'finish_approved_proposal'
'finish_approved_proposal',
'export_applied_proposal_replay_rows'
)
and acl.grantee <> procedure.proowner
), delta(kind, signature, grantee, privilege_type, is_grantable) as (

273
scripts/kb_apply_replay_receipt.py
View file

@ -13,13 +13,79 @@ from __future__ import annotations
import hashlib
import json
import os
import re
import tempfile
import uuid
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
RECEIPT_CONTRACT_VERSION = 1
DEFAULT_RECEIPT_DIR = "/opt/teleo-eval/logs/kb-apply-receipts"
UTC_SESSION_SQL = "set timezone = 'UTC';"
_ISO_TIMESTAMP = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{1,6})?(?:Z|[+-]\d{2}:\d{2})$")
_UUID_FIELDS = {
"id",
"agent_id",
"owner_agent_id",
"created_by",
"superseded_by",
"supersedes_id",
"accepted_by_proposal_id",
"claim_id",
"source_id",
"accepted_from_proposal_id",
"from_claim",
"to_claim",
"supersedes_assessment_id",
"reviewed_by_agent_id",
"applied_by_agent_id",
}
_TIMESTAMP_FIELDS = {"captured_at", "created_at", "updated_at", "reviewed_at", "applied_at"}
def canonical_utc_timestamp(value: Any, *, path: str = "timestamp") -> str:
"""Normalize a typed timestamp to fixed-microsecond UTC receipt text."""
if not isinstance(value, str) or not _ISO_TIMESTAMP.fullmatch(value):
raise ValueError(f"{path} must be a timezone-aware ISO-8601 timestamp with at most 6 fractional digits")
try:
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError as exc:
raise ValueError(f"{path} must be a valid ISO-8601 timestamp") from exc
if parsed.tzinfo is None:
raise ValueError(f"{path} must include a timezone")
return parsed.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ")
def _canonical_uuid(value: Any, *, path: str) -> str:
try:
return str(uuid.UUID(str(value)))
except (AttributeError, TypeError, ValueError) as exc:
raise ValueError(f"{path} must be a UUID") from exc
def _normalize_typed_value(field: str, value: Any, *, path: str) -> Any:
if value is None:
return None
if field in _UUID_FIELDS:
return _canonical_uuid(value, path=path)
if field in _TIMESTAMP_FIELDS:
return canonical_utc_timestamp(value, path=path)
return value
def _normalize_typed_row(row: dict[str, Any], *, path: str) -> dict[str, Any]:
return {field: _normalize_typed_value(field, value, path=f"{path}.{field}") for field, value in row.items()}
def normalize_typed_value(field: str, value: Any, *, path: str) -> Any:
"""Canonicalize a value according to its PostgreSQL column type."""
return _normalize_typed_value(field, value, path=path)
def normalize_typed_row(row: dict[str, Any], *, path: str) -> dict[str, Any]:
"""Canonicalize UUID and timestamp columns without weakening text equality."""
return _normalize_typed_row(row, path=path)
def _canonical_json(value: Any) -> str:
@ -56,6 +122,12 @@ def _require_applied_strict_proposal(proposal: dict[str, Any]) -> dict[str, Any]
apply_payload = payload.get("apply_payload") if isinstance(payload, dict) else None
if not isinstance(apply_payload, dict):
raise ValueError("replay receipt requires a strict payload.apply_payload object")
raw_version = apply_payload.get("contract_version")
if proposal.get("proposal_type") == "approve_claim":
if type(raw_version) is not int or raw_version not in {2, 3}:
raise ValueError("approve_claim replay receipt requires integer contract_version 2 or 3")
elif "contract_version" in apply_payload:
raise ValueError("legacy replay receipt proposal types must omit contract_version")
return apply_payload
@ -108,6 +180,8 @@ def _build_attach_evidence_postflight_sql(apply_payload: dict[str, Any]) -> str:
def _build_approve_claim_postflight_sql(apply_payload: dict[str, Any]) -> str:
if apply_payload.get("contract_version") == 3:
return _build_approve_claim_v3_postflight_sql(apply_payload)
claims = apply_payload.get("claims") or []
sources = apply_payload.get("sources") or []
evidence = apply_payload.get("evidence") or []
@ -154,6 +228,44 @@ def _build_approve_claim_postflight_sql(apply_payload: dict[str, Any]) -> str:
return f"select jsonb_build_object({arguments})::text;"
def _build_approve_claim_v3_postflight_sql(apply_payload: dict[str, Any]) -> str:
fields = {
"claims": _coalesced_json_agg(
"public.claims",
"c",
_id_filter("c", [row.get("id") for row in apply_payload.get("claims") or []]),
"c.id::text",
),
"sources": _coalesced_json_agg(
"public.sources",
"s",
_id_filter("s", [row.get("id") for row in apply_payload.get("sources") or []]),
"s.id::text",
),
"claim_evidence": _coalesced_json_agg(
"public.claim_evidence",
"ce",
_id_filter("ce", [row.get("id") for row in apply_payload.get("evidence") or []]),
"ce.id::text",
),
"claim_edges": _coalesced_json_agg(
"public.claim_edges",
"e",
_id_filter("e", [row.get("id") for row in apply_payload.get("edges") or []]),
"e.id::text",
),
"claim_evidence_assessments": _coalesced_json_agg(
"public.claim_evidence_assessments",
"a",
_id_filter("a", [row.get("id") for row in apply_payload.get("assessments") or []]),
"a.id::text",
),
"reasoning_tools": "'[]'::jsonb",
}
arguments = ", ".join(f"{_sql_literal(name)}, {expression}" for name, expression in fields.items())
return f"select jsonb_build_object({arguments})::text;"
def _build_revise_strategy_postflight_sql(proposal: dict[str, Any], apply_payload: dict[str, Any]) -> str:
strategy = apply_payload.get("strategy") or {}
agent_id = apply_payload.get("agent_id")
@ -193,14 +305,16 @@ def build_postflight_sql(proposal: dict[str, Any]) -> str:
apply_payload = _require_applied_strict_proposal(proposal)
proposal_type = proposal.get("proposal_type")
if proposal_type == "add_edge":
return _build_add_edge_postflight_sql(apply_payload)
if proposal_type == "attach_evidence":
return _build_attach_evidence_postflight_sql(apply_payload)
if proposal_type == "approve_claim":
return _build_approve_claim_postflight_sql(apply_payload)
if proposal_type == "revise_strategy":
return _build_revise_strategy_postflight_sql(proposal, apply_payload)
raise ValueError(f"replay receipt does not support proposal_type {proposal_type!r}")
query = _build_add_edge_postflight_sql(apply_payload)
elif proposal_type == "attach_evidence":
query = _build_attach_evidence_postflight_sql(apply_payload)
elif proposal_type == "approve_claim":
query = _build_approve_claim_postflight_sql(apply_payload)
elif proposal_type == "revise_strategy":
query = _build_revise_strategy_postflight_sql(proposal, apply_payload)
else:
raise ValueError(f"replay receipt does not support proposal_type {proposal_type!r}")
return UTC_SESSION_SQL + "\n" + query
def expected_row_counts(proposal: dict[str, Any]) -> dict[str, int]:
@ -211,6 +325,15 @@ def expected_row_counts(proposal: dict[str, Any]) -> dict[str, int]:
if proposal_type == "attach_evidence":
return {"claim_evidence": len(apply_payload.get("evidence") or [])}
if proposal_type == "approve_claim":
if apply_payload.get("contract_version") == 3:
return {
"claims": len(apply_payload.get("claims") or []),
"sources": len(apply_payload.get("sources") or []),
"claim_evidence": len(apply_payload.get("evidence") or []),
"claim_edges": len(apply_payload.get("edges") or []),
"claim_evidence_assessments": len(apply_payload.get("assessments") or []),
"reasoning_tools": 0,
}
return {
"claims": len(apply_payload.get("claims") or []),
"sources": len(apply_payload.get("sources") or []),
@ -232,12 +355,13 @@ def _normalize_rows(canonical_rows: dict[str, Any], expected_counts: dict[str, i
rows = canonical_rows.get(table)
if not isinstance(rows, list) or any(not isinstance(row, dict) for row in rows):
raise ValueError(f"postflight {table} must be a list of row objects")
normalized[table] = sorted(rows, key=_canonical_json)
typed_rows = [_normalize_typed_row(row, path=f"postflight.{table}") for row in rows]
normalized[table] = sorted(typed_rows, key=_canonical_json)
return normalized
def _semantic_projection(row: dict[str, Any], fields: tuple[str, ...]) -> dict[str, Any]:
return {field: row.get(field) for field in fields}
return {field: _normalize_typed_value(field, row.get(field), path=f"semantic.{field}") for field in fields}
def _expected_semantic_rows(proposal: dict[str, Any]) -> dict[str, list[dict[str, Any]]]:
@ -269,6 +393,8 @@ def _expected_semantic_rows(proposal: dict[str, Any]) -> dict[str, list[dict[str
]
}
if proposal_type == "approve_claim":
if apply_payload.get("contract_version") == 3:
return _expected_v3_semantic_rows(proposal, apply_payload)
agent_id = apply_payload.get("agent_id")
return {
"claims": [
@ -359,12 +485,115 @@ def _expected_semantic_rows(proposal: dict[str, Any]) -> dict[str, list[dict[str
raise ValueError(f"replay receipt does not support proposal_type {proposal_type!r}")
def _expected_v3_semantic_rows(
proposal: dict[str, Any], apply_payload: dict[str, Any]
) -> dict[str, list[dict[str, Any]]]:
proposal_id = _canonical_uuid(proposal.get("id"), path="proposal.id")
agent_id = apply_payload.get("agent_id")
polarity_to_role = {"supports": "grounds", "challenges": "contradicts", "illustrates": "illustrates"}
return {
"claims": [
{
"id": row.get("id"),
"owner_agent_id": row.get("owner_agent_id"),
"type": row.get("type"),
"text": row.get("proposition"),
"proposition": row.get("proposition"),
"body_markdown": row.get("body_markdown"),
"scope": row.get("scope"),
"access_scope": row.get("access_scope"),
"status": row.get("status", "active"),
"admission_state": "admitted",
"revision_criteria": row.get("revision_criteria"),
"revision_kind": "original",
"supersedes_id": None,
"confidence": None,
"tags": row.get("tags") or [],
"created_by": row.get("created_by", agent_id),
"superseded_by": None,
"accepted_by_proposal_id": proposal_id,
}
for row in apply_payload.get("claims") or []
],
"sources": [
{
"id": row.get("id"),
"source_type": row.get("source_type"),
"url": row.get("canonical_url"),
"storage_path": row.get("storage_uri"),
"excerpt": row.get("excerpt"),
"hash": row.get("content_hash"),
"created_by": row.get("created_by", agent_id),
"captured_at": row.get("captured_at"),
"canonical_url": row.get("canonical_url"),
"storage_uri": row.get("storage_uri"),
"content_hash": row.get("content_hash"),
"access_scope": row.get("access_scope"),
"ingestion_origin": row.get("ingestion_origin"),
"provenance_status": row.get("provenance_status"),
"source_class": row.get("source_class"),
"auto_promotion_policy": row.get("auto_promotion_policy"),
"accepted_by_proposal_id": proposal_id,
}
for row in apply_payload.get("sources") or []
],
"claim_evidence": [
{
"id": row.get("id"),
"claim_id": row.get("claim_id"),
"source_id": row.get("source_id"),
"role": polarity_to_role.get(row.get("polarity")),
"weight": None,
"created_by": row.get("created_by", agent_id),
"excerpt": row.get("excerpt"),
"locator_json": row.get("locator_json"),
"source_content_hash": row.get("source_content_hash"),
"rationale": row.get("rationale"),
"polarity": row.get("polarity"),
"accepted_from_proposal_id": proposal_id,
}
for row in apply_payload.get("evidence") or []
],
"claim_edges": [
{
"id": row.get("id"),
"from_claim": row.get("from_claim"),
"to_claim": row.get("to_claim"),
"edge_type": "requires" if row.get("relation_type") == "depends_on" else row.get("relation_type"),
"weight": None,
"created_by": row.get("created_by", agent_id),
"relation_type": row.get("relation_type"),
"rationale": row.get("rationale"),
"scope_or_conditions": row.get("scope_or_conditions"),
"accepted_by_proposal_id": proposal_id,
}
for row in apply_payload.get("edges") or []
],
"claim_evidence_assessments": [
{
"id": row.get("id"),
"claim_id": row.get("claim_id"),
"support_tier": row.get("support_tier"),
"challenge_tier": row.get("challenge_tier"),
"overall_state": row.get("overall_state"),
"rationale": row.get("rationale"),
"evidence_set_hash": row.get("evidence_set_hash"),
"accepted_by_proposal_id": proposal_id,
"supersedes_assessment_id": row.get("supersedes_assessment_id"),
}
for row in apply_payload.get("assessments") or []
],
"reasoning_tools": [],
}
def _assert_semantic_rows_match(proposal: dict[str, Any], rows: dict[str, list[dict[str, Any]]]) -> None:
expected = _expected_semantic_rows(proposal)
for table, expected_rows in expected.items():
fields = tuple(expected_rows[0]) if expected_rows else ()
expected_projected = [_semantic_projection(row, fields) for row in expected_rows]
actual_projected = [_semantic_projection(row, fields) for row in rows.get(table, [])]
if sorted(expected_rows, key=_canonical_json) != sorted(actual_projected, key=_canonical_json):
if sorted(expected_projected, key=_canonical_json) != sorted(actual_projected, key=_canonical_json):
raise ValueError(
f"row-level postflight semantic mismatch for {table}; "
"canonical values do not match the immutable strict payload"
@ -397,17 +626,21 @@ def build_replay_receipt(
_assert_semantic_rows_match(proposal, rows)
proposal_envelope = {
"id": str(proposal.get("id")),
"id": _canonical_uuid(proposal.get("id"), path="proposal.id"),
"proposal_type": proposal.get("proposal_type"),
"status": proposal.get("status"),
"payload": proposal.get("payload"),
"reviewed_by_handle": proposal.get("reviewed_by_handle"),
"reviewed_by_agent_id": proposal.get("reviewed_by_agent_id"),
"reviewed_at": proposal.get("reviewed_at"),
"reviewed_by_agent_id": _normalize_typed_value(
"reviewed_by_agent_id", proposal.get("reviewed_by_agent_id"), path="proposal.reviewed_by_agent_id"
),
"reviewed_at": _normalize_typed_value("reviewed_at", proposal.get("reviewed_at"), path="proposal.reviewed_at"),
"review_note": proposal.get("review_note"),
"applied_by_handle": proposal.get("applied_by_handle"),
"applied_by_agent_id": proposal.get("applied_by_agent_id"),
"applied_at": proposal.get("applied_at"),
"applied_by_agent_id": _normalize_typed_value(
"applied_by_agent_id", proposal.get("applied_by_agent_id"), path="proposal.applied_by_agent_id"
),
"applied_at": _normalize_typed_value("applied_at", proposal.get("applied_at"), path="proposal.applied_at"),
}
row_hashes = {table: [_sha256_json(row) for row in table_rows] for table, table_rows in rows.items()}
table_hashes = {table: _sha256_json(table_rows) for table, table_rows in rows.items()}
@ -423,7 +656,9 @@ def build_replay_receipt(
return {
**replay_material,
"artifact": "kb_apply_replay_receipt",
"exported_at_utc": exported_at_utc or datetime.now(timezone.utc).isoformat(),
"exported_at_utc": canonical_utc_timestamp(
exported_at_utc or datetime.now(timezone.utc).isoformat(), path="exported_at_utc"
),
"expected_row_counts": expected,
"actual_row_counts": actual,
"checks": {
@ -464,7 +699,9 @@ def validate_replay_receipt(
apply_engine = receipt.get("apply_engine")
if not isinstance(proposal, dict) or not isinstance(rows, dict) or not isinstance(apply_engine, dict):
raise ValueError("replay receipt is missing proposal, canonical rows, or apply-engine metadata")
if expected_proposal_id is not None and proposal.get("id") != expected_proposal_id:
if expected_proposal_id is not None and proposal.get("id") != _canonical_uuid(
expected_proposal_id, path="expected_proposal_id"
):
raise ValueError("replay receipt proposal id does not match")
try:
rebuilt = build_replay_receipt(

25
scripts/kb_proposal_review_packet.py
View file

@ -18,6 +18,7 @@ a file for tests and UI development.
from __future__ import annotations
import argparse
import hashlib
import json
import sys
from pathlib import Path
@ -30,6 +31,11 @@ import apply_proposal as ap # noqa: E402
import apply_worker as aw # noqa: E402
def canonical_sha256(value: Any) -> str:
encoded = json.dumps(value, ensure_ascii=True, separators=(",", ":"), sort_keys=True).encode("utf-8")
return hashlib.sha256(encoded).hexdigest()
def _payload_list(payload: dict[str, Any], key: str) -> list[Any]:
value = payload.get(key)
return value if isinstance(value, list) else []
@ -47,7 +53,9 @@ def classify_proposal(proposal: dict[str, Any]) -> dict[str, Any]:
status = proposal.get("status")
proposal_type = proposal.get("proposal_type")
has_apply_payload = "apply_payload" in payload
apply_payload = _payload_dict(payload, "apply_payload")
has_apply_payload = bool(apply_payload)
contract_version = apply_payload.get("contract_version", 2) if has_apply_payload else None
worker_supported_type = proposal_type in aw.WORKER_TYPES
worker_applyable = (
status == "approved" and worker_supported_type and has_apply_payload
@ -73,6 +81,11 @@ def classify_proposal(proposal: dict[str, Any]) -> dict[str, Any]:
governance_standard = _payload_dict(payload, "proposed_governance_standard")
approval_request = _payload_dict(payload, "approval_request")
review_request = _payload_dict(payload, "review_request")
strict_rows = {
key: _payload_list(apply_payload, key)
for key in ("claims", "sources", "evidence", "edges", "assessments", "reasoning_tools")
}
strict_normalization_manifest = _payload_dict(apply_payload, "normalization_manifest")
missing_contract = []
if not has_apply_payload:
@ -125,6 +138,7 @@ def classify_proposal(proposal: dict[str, Any]) -> dict[str, Any]:
},
"payload_summary": {
"proposal_kind": payload.get("proposal_kind"),
"contract_version": contract_version,
"old_claim_id": _payload_dict(payload, "old_claim").get("id"),
"claim_candidate_count": len(claim_candidates),
"source_candidate_count": len(source_candidates),
@ -136,6 +150,13 @@ def classify_proposal(proposal: dict[str, Any]) -> dict[str, Any]:
"do_not_approve_yet": approval_request.get("do_not_approve_yet", []),
"requested_decision": review_request.get("requested_decision"),
"not_requested": review_request.get("not_requested"),
"strict_row_counts": {key: len(rows) for key, rows in strict_rows.items()},
},
"strict_review_material": {
"apply_payload_sha256": canonical_sha256(apply_payload) if has_apply_payload else None,
"proposal_payload_sha256": canonical_sha256(payload),
"normalization_manifest": strict_normalization_manifest,
"rows": strict_rows,
},
"applyability": {
"missing_contract": missing_contract,
@ -225,8 +246,10 @@ def render_markdown(packets: list[dict[str, Any]]) -> str:
f"- Worker applyable: {p['worker_applyable']}",
f"- Reviewed by: {auth.get('reviewed_by_handle') or 'unreviewed'}",
f"- Proposal kind: {summary.get('proposal_kind') or 'unknown'}",
f"- Contract version: {summary.get('contract_version') or 'none'}",
f"- Claim candidates: {summary['claim_candidate_count']}",
f"- Source candidates: {summary['source_candidate_count']}",
f"- Strict rows: {json.dumps(summary['strict_row_counts'], sort_keys=True)}",
f"- Missing contract: {', '.join(applyability['missing_contract']) or 'none'}",
f"- Next admin action: {p['next_admin_action']}",
]

5
scripts/proposal_apply_lifecycle.py
View file

@ -119,6 +119,11 @@ def _require_approved_bundle(proposal: dict[str, Any]) -> dict[str, Any]:
apply_payload = payload.get("apply_payload") if isinstance(payload, dict) else None
if not isinstance(apply_payload, dict):
raise ValueError("approved proposal requires payload.apply_payload")
if apply_payload.get("contract_version") != 2:
raise ValueError(
"compensating rollback supports approve_claim contract_version 2 only; "
"V3 admitted rows are immutable and require an append-only correction"
)
# Reuse the audited builder as the strict contract validator.
ap.build_apply_sql(proposal, ap.SERVICE_AGENT_HANDLE)
return apply_payload

2
scripts/run_approve_claim_clone_canary.py
View file

@ -32,7 +32,7 @@ import apply_proposal as ap # noqa: E402
import proposal_apply_lifecycle as lifecycle # noqa: E402
SAFE_DB_RE = re.compile(r"^[a-z][a-z0-9_]{1,62}$")
REVIEWER_HANDLE = "m3ta"
REVIEWER_HANDLE = "m3taversal"
REVIEW_NOTE = "Reviewed exact strict payload inside the disposable clone."
DEFAULT_REVIEW_SECRETS_FILE = "/home/teleo/.hermes/profiles/leoclean/secrets/kb-review.env"
SECURITY_DEFINER_DEPENDENCIES = {

1669
scripts/run_leo_integrated_learning_lifecycle.py
View file

File diff suppressed because it is too large Load diff

2
scripts/run_leo_negative_permissions_canary.py
View file

@ -30,7 +30,7 @@ READ_ROLE_SQL = ROOT / "ops" / "observatory_read_role.sql"
DATABASE = "teleo_canonical"
IAM_USER = "sa-observatory-read-adapter@teleo-501523.iam"
PARTICIPANT_HANDLE = "m3taversal"
DEFAULT_IMAGE = "postgres:16-alpine"
DEFAULT_IMAGE = os.environ.get("LEOCLEAN_RUNTIME_POSTGRES_IMAGE", "postgres:16-alpine")
CANARY_LABEL = "leo-negative-permissions"
BOOTSTRAP_SQL = r"""

120
scripts/stage_normalized_proposal.py
View file

@ -4,6 +4,7 @@
from __future__ import annotations
import argparse
import copy
import hashlib
import json
import sys
@ -18,6 +19,9 @@ import apply_proposal as ap # noqa: E402
import kb_proposal_normalize as normalize # noqa: E402
import run_leo_clone_bound_handler_checkpoint as bound # noqa: E402
V3_BUNDLE_SCHEMA = "livingip.kbSourceProposalBundle.v3"
V3_MANIFEST_SCHEMA = "livingip.kbSourceExtractionManifest.v3"
def canonical_sha256(value: Any) -> str:
encoded = json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode()
@ -175,24 +179,116 @@ commit;
"""
def stage_normalized_child(container: str, database: str, parent: dict[str, Any]) -> dict[str, Any]:
def prepare_compiled_v3_child(bundle: dict[str, Any]) -> dict[str, Any]:
"""Revalidate one build-only V3 compiler bundle before disposable staging."""
if bundle.get("schema") != V3_BUNDLE_SCHEMA:
raise bound.CheckpointError(f"compiled bundle schema must equal {V3_BUNDLE_SCHEMA!r}")
if (
bundle.get("status") != "pending_review"
or bundle.get("build_only") is not True
or bundle.get("database_write_performed") is not False
or bundle.get("canonical_apply_performed") is not False
):
raise bound.CheckpointError("compiled V3 bundle is not build-only pending-review material")
hashes = bundle.get("hashes")
if not isinstance(hashes, dict):
raise bound.CheckpointError("compiled V3 bundle hashes must be an object")
expected_bundle_hash = hashes.get("bundle_content_sha256")
preimage = copy.deepcopy(bundle)
preimage_hashes = preimage.get("hashes")
if not isinstance(expected_bundle_hash, str) or not isinstance(preimage_hashes, dict):
raise bound.CheckpointError("compiled V3 bundle content hash is missing")
preimage_hashes.pop("bundle_content_sha256", None)
if canonical_sha256(preimage) != expected_bundle_hash:
raise bound.CheckpointError("compiled V3 bundle content hash does not match")
manifest = bundle.get("validated_manifest")
child = bundle.get("strict_child_proposal")
if not isinstance(manifest, dict) or manifest.get("schema") != V3_MANIFEST_SCHEMA:
raise bound.CheckpointError("compiled V3 bundle has no validated V3 manifest")
if not isinstance(child, dict):
raise bound.CheckpointError("compiled V3 bundle has no strict child proposal")
payload = child.get("payload")
apply_payload = payload.get("apply_payload") if isinstance(payload, dict) else None
if (
child.get("status") != "pending_review"
or child.get("proposal_type") != "approve_claim"
or not isinstance(apply_payload, dict)
or apply_payload.get("contract_version") != 3
):
raise bound.CheckpointError("compiled V3 child is not one pending-review approve_claim v3 proposal")
payload_hash = canonical_sha256(payload)
if child.get("payload_sha256") != payload_hash or hashes.get("strict_child_payload_sha256") != payload_hash:
raise bound.CheckpointError("compiled V3 child payload hash does not match")
if child.get("parent_packet_sha256") != hashes.get("parent_packet_sha256"):
raise bound.CheckpointError("compiled V3 parent packet hash does not match")
stage_preview = bundle.get("stage_preview")
stage_sql_hash = hashlib.sha256(build_stage_sql(child).encode("utf-8")).hexdigest()
if (
not isinstance(stage_preview, dict)
or stage_preview.get("target_table") != "kb_stage.kb_proposals"
or stage_preview.get("target_status") != "pending_review"
or stage_preview.get("executed") is not False
or stage_preview.get("stage_sql_sha256") != stage_sql_hash
):
raise bound.CheckpointError("compiled V3 stage preview does not match the exact child")
validation_snapshot = copy.deepcopy(child)
validation_snapshot.update(
{
"status": "approved",
"reviewed_by_handle": "v3-compiler-preflight",
"reviewed_by_agent_id": None,
"reviewed_at": "2000-01-01T00:00:00+00:00",
"review_note": "Build-only validation of exact compiled V3 material.",
}
)
try:
ap.build_apply_sql(validation_snapshot, ap.SERVICE_AGENT_HANDLE)
except (TypeError, ValueError) as exc:
raise bound.CheckpointError(f"compiled V3 child failed strict writer preflight: {exc}") from exc
return copy.deepcopy(child)
def _stage_child(container: str, database: str, child: dict[str, Any]) -> dict[str, Any]:
target = bound.container_identity(container)
production = bound.container_identity(bound.PRODUCTION_CONTAINER)
try:
production = bound.container_identity(bound.PRODUCTION_CONTAINER)
except bound.CheckpointError as exc:
detail = str(exc).lower()
if "no such object" not in detail and "no such container" not in detail:
raise
production = {"id": None, "mount_sources": []}
bound.validate_disposable_target_identity(target, production)
if target["id"] == production["id"]:
raise bound.CheckpointError("normalized staging target resolves to production")
child = prepare_normalized_child(parent)
result = bound._psql_json(str(target["id"]), database, build_stage_sql(child))
if result.get("proposal_id") != child["id"] or result.get("status") != "pending_review":
raise bound.CheckpointError("normalized staging did not return the exact pending child")
return {"child": child, "database_receipt": result, "bound_container_id": str(target["id"])}
def stage_normalized_child(container: str, database: str, parent: dict[str, Any]) -> dict[str, Any]:
return _stage_child(container, database, prepare_normalized_child(parent))
def stage_compiled_v3_bundle(container: str, database: str, bundle: dict[str, Any]) -> dict[str, Any]:
return _stage_child(container, database, prepare_compiled_v3_child(bundle))
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--container", required=True)
parser.add_argument("--db", required=True)
parser.add_argument("--proposal-file", required=True, type=Path)
source = parser.add_mutually_exclusive_group(required=True)
source.add_argument("--proposal-file", type=Path, help="rich V1/V2 proposal to normalize then stage")
source.add_argument(
"--compiled-v3-bundle", type=Path, help="build-only V3 compiler bundle to revalidate then stage"
)
return parser.parse_args(argv)
@ -201,10 +297,15 @@ def main(argv: list[str] | None = None) -> int:
try:
if args.container == bound.PRODUCTION_CONTAINER:
raise bound.CheckpointError("--container must be a disposable non-production target")
parent = json.loads(args.proposal_file.read_text(encoding="utf-8"))
if not isinstance(parent, dict):
raise bound.CheckpointError("--proposal-file must contain one JSON object")
result = stage_normalized_child(args.container, args.db, parent)
input_path = args.compiled_v3_bundle or args.proposal_file
packet = json.loads(input_path.read_text(encoding="utf-8"))
if not isinstance(packet, dict):
raise bound.CheckpointError("staging input must contain one JSON object")
result = (
stage_compiled_v3_bundle(args.container, args.db, packet)
if args.compiled_v3_bundle
else stage_normalized_child(args.container, args.db, packet)
)
except (OSError, json.JSONDecodeError, bound.CheckpointError, SystemExit, ValueError) as exc:
print(json.dumps({"status": "rejected", "error": str(exc)}, sort_keys=True))
return 2
@ -216,6 +317,9 @@ def main(argv: list[str] | None = None) -> int:
"payload_sha256": result["child"]["payload_sha256"],
"created": result["database_receipt"].get("created"),
"bound_container_id": result["bound_container_id"],
"contract_version": (result["child"].get("payload") or {})
.get("apply_payload", {})
.get("contract_version", 2),
},
sort_keys=True,
)

6
systemd/teleo-kb-apply-worker.service
View file

@ -14,7 +14,11 @@ ExecStart=/opt/teleo-eval/pipeline/.venv/bin/python3 \
/opt/teleo-eval/workspaces/deploy-infra/scripts/apply_worker.py
# Ships INERT: no --enable here, so each tick is report-only until an operator opts
# in. Flip on (after the first manual apply is proven) by dropping an EnvironmentFile
# with KB_APPLY_WORKER_ENABLED=1 and KB_APPLY_RENDER_CMD=... — no unit edit needed.
# with KB_APPLY_WORKER_ENABLED=1, KB_APPLY_WORKER_CONTRACT_VERSION=2|3,
# and KB_APPLY_RENDER_CMD=... -- no unit edit needed. Once the V3 database cutover
# is installed, a V2-configured tick skips retained historical V2 approvals without
# applying them; set version 3 before enabling new execution. Migrate or reject each
# retained V2 packet explicitly through review instead of deleting it from the queue.
EnvironmentFile=-/opt/teleo-eval/secrets/kb-apply-worker.env
StandardOutput=journal
StandardError=journal

128
tests/docker_volume_lifecycle.py Normal file
View file

@ -0,0 +1,128 @@
from __future__ import annotations
import fcntl
import json
import subprocess
import tempfile
import threading
from dataclasses import dataclass
from pathlib import Path
from typing import TextIO
POSTGRES_DATA_PATH = "/var/lib/postgresql/data"
POSTGRES_TMPFS_SPEC = f"{POSTGRES_DATA_PATH}:rw,nosuid,nodev,noexec,size=384m"
VOLUME_GUARD_LOCK = Path(tempfile.gettempdir()) / "livingip-teleo-docker-volume-lifecycle.lock"
@dataclass(frozen=True)
class DockerVolumeSetGuard:
before: frozenset[str]
context: str
owner_thread_id: int
_PROCESS_GUARD = threading.RLock()
_LOCK_FILE: TextIO | None = None
_GUARD_STACK: list[DockerVolumeSetGuard] = []
def _docker(*arguments: str, check: bool = True) -> subprocess.CompletedProcess[str]:
completed = subprocess.run(
["docker", *arguments],
text=True,
capture_output=True,
check=False,
timeout=120,
)
if check and completed.returncode != 0:
raise AssertionError(completed.stderr or completed.stdout)
return completed
def docker_volume_names() -> frozenset[str]:
completed = _docker("volume", "ls", "--quiet")
return frozenset(name for name in completed.stdout.splitlines() if name)
def assert_docker_volume_set_unchanged(before: frozenset[str], *, context: str) -> None:
after = docker_volume_names()
added = sorted(after - before)
removed = sorted(before - after)
assert after == before, (
f"daemon-wide Docker volume drift while running {context}; another Docker job may be responsible; "
f"added={added!r}; removed={removed!r}; before_count={len(before)}; after_count={len(after)}"
)
def acquire_docker_volume_set_guard(*, context: str) -> DockerVolumeSetGuard:
global _LOCK_FILE
_PROCESS_GUARD.acquire()
opened_here = False
try:
if _LOCK_FILE is None:
lock_file = VOLUME_GUARD_LOCK.open("a+", encoding="utf-8")
try:
fcntl.flock(lock_file.fileno(), fcntl.LOCK_EX)
except BaseException:
lock_file.close()
raise
_LOCK_FILE = lock_file
opened_here = True
guard = DockerVolumeSetGuard(
before=docker_volume_names(),
context=context,
owner_thread_id=threading.get_ident(),
)
_GUARD_STACK.append(guard)
return guard
except BaseException:
if opened_here and _LOCK_FILE is not None:
fcntl.flock(_LOCK_FILE.fileno(), fcntl.LOCK_UN)
_LOCK_FILE.close()
_LOCK_FILE = None
_PROCESS_GUARD.release()
raise
def release_docker_volume_set_guard(guard: DockerVolumeSetGuard) -> None:
global _LOCK_FILE
assert guard.owner_thread_id == threading.get_ident(), "Docker volume guard released from a different thread"
assert _GUARD_STACK and _GUARD_STACK[-1] is guard, "Docker volume guards must be released in LIFO order"
try:
assert_docker_volume_set_unchanged(guard.before, context=guard.context)
finally:
_GUARD_STACK.pop()
if not _GUARD_STACK:
assert _LOCK_FILE is not None
fcntl.flock(_LOCK_FILE.fileno(), fcntl.LOCK_UN)
_LOCK_FILE.close()
_LOCK_FILE = None
_PROCESS_GUARD.release()
def inspect_postgres_tmpfs_container(name: str) -> dict:
inspected = json.loads(_docker("inspect", name).stdout)[0]
assert inspected["Mounts"] == [], f"{name} unexpectedly uses Docker volumes or bind mounts"
tmpfs = inspected["HostConfig"]["Tmpfs"]
assert POSTGRES_DATA_PATH in tmpfs, f"{name} does not mount PostgreSQL data on tmpfs"
return inspected
def remove_container_and_volumes(name: str) -> None:
removed = _docker("rm", "--force", "--volumes", name, check=False)
if removed.returncode != 0 and "no such container" not in removed.stderr.lower():
raise AssertionError(removed.stderr or removed.stdout)
assert _docker("container", "inspect", name, check=False).returncode != 0
names = _docker(
"container",
"ls",
"--all",
"--filter",
f"name=^/{name}$",
"--format",
"{{.Names}}",
)
assert names.stdout.strip() == "", f"container still exists after cleanup: {name}"

11
tests/fixtures/teleo_corpus_rebuild/forgejo/project-record.md vendored Normal file
View file

@ -0,0 +1,11 @@
# Forgejo Project Record
The retained project record names the repository and keeps source provenance reviewable.
Repository: living-ip/teleo-infrastructure
## Review Boundary
Every extracted section remains a pending candidate until a separate reviewer approves it.
No local corpus rebuild may stage or apply canonical database rows.

2
tests/fixtures/teleo_corpus_rebuild/operator-notes.txt vendored Normal file
View file

@ -0,0 +1,2 @@
The local rebuild inventories exact UTF-8 bytes and source-root-relative paths.
Its receipt binds the source manifest, candidate index, extractor specification, and V3 proposal files.

444
tests/test_apply_proposal.py
View file

@ -4,6 +4,8 @@ Pure tests: exercise the SQL builders, payload validation, and dispatch without
a live database. Runnable under pytest or standalone (`python3 tests/test_apply_proposal.py`).
"""
import json
import re
import sys
from pathlib import Path
@ -100,6 +102,26 @@ def test_revise_strategy_sql_shape():
assert "::jsonb" in sql
def test_guarded_sql_canonicalizes_reviewed_timestamp_to_stable_utc_text():
payload = _revise_payload()["apply_payload"]
approval = _approval_meta("revise_strategy", payload)
approval["reviewed_at"] = "2026-07-10T02:00:00.1+02:00"
sql = ap.build_revise_strategy_sql(payload, "pid-1", "kb-apply", approval)
assert sql.count("2026-07-10T00:00:00.100000Z") == 2
assert "2026-07-10T02:00:00.1+02:00" not in sql
def test_guarded_sql_rejects_reviewed_timestamp_without_timezone():
payload = _revise_payload()["apply_payload"]
approval = _approval_meta("revise_strategy", payload)
approval["reviewed_at"] = "2026-07-10T00:00:00"
with pytest.raises(ValueError, match="timezone-aware ISO-8601"):
ap.build_revise_strategy_sql(payload, "pid-1", "kb-apply", approval)
def test_revise_strategy_requires_agent_id():
payload = _revise_payload()["apply_payload"]
del payload["agent_id"]
@ -240,6 +262,135 @@ def _approve_claim_bundle():
}
def _approve_claim_v3_bundle():
agent_id = "11111111-1111-4111-8111-111111111111"
source_b = "dddddddd-dddd-4ddd-8ddd-dddddddddddd"
evidence_a = {
"id": "eeeeeeee-eeee-4eee-8eee-eeeeeeeeeeee",
"claim_id": CLAIM_A,
"source_id": SOURCE_A,
"polarity": "supports",
"excerpt": "The primary record supports claim A.",
"locator_json": {"section": "A", "paragraph": 1},
"source_content_hash": "a" * 64,
"rationale": "The cited paragraph directly supports the bounded proposition.",
"created_by": agent_id,
}
evidence_b = {
"id": "ffffffff-ffff-4fff-8fff-ffffffffffff",
"claim_id": CLAIM_B,
"source_id": source_b,
"polarity": "challenges",
"excerpt": "The primary record challenges claim B.",
"locator_json": {"section": "B", "paragraph": 2},
"source_content_hash": "b" * 64,
"rationale": "The cited paragraph directly challenges the bounded proposition.",
"created_by": agent_id,
}
evidence = [evidence_a, evidence_b]
return {
"contract_version": 3,
"agent_id": agent_id,
"claims": [
{
"id": CLAIM_A,
"type": "empirical",
"proposition": "Claim A has a reproducible evidence chain.",
"body_markdown": "Claim A is bounded to the cited primary record and exact locator.",
"scope": "V3 apply-writer disposable verification.",
"access_scope": "collective_internal",
"status": "active",
"revision_criteria": "Revise if the cited source or locator cannot be reproduced.",
"revision_kind": "original",
"supersedes_id": None,
"tags": ["v3", "apply-writer"],
"created_by": agent_id,
},
{
"id": CLAIM_B,
"type": "conceptual",
"proposition": "Claim B remains explicitly challengeable.",
"body_markdown": "Claim B records a conceptual object together with its challenge evidence.",
"scope": "V3 apply-writer disposable verification.",
"access_scope": "collective_internal",
"status": "contested",
"revision_criteria": "Revise when the challenge is resolved or materially changes.",
"revision_kind": "original",
"supersedes_id": None,
"tags": ["v3", "challenge"],
"created_by": agent_id,
},
],
"sources": [
{
"id": SOURCE_A,
"source_type": "paper",
"canonical_url": "https://example.test/v3/a",
"storage_uri": None,
"excerpt": "Primary source A.",
"content_hash": "a" * 64,
"access_scope": "collective_internal",
"ingestion_origin": "operator_upload",
"provenance_status": "verified",
"source_class": "primary_record",
"auto_promotion_policy": "human_required",
"captured_at": "2026-07-18T10:00:00+00:00",
"created_by": agent_id,
},
{
"id": source_b,
"source_type": "observation",
"canonical_url": None,
"storage_uri": "gs://livingip-test/v3/source-b",
"excerpt": "Primary source B.",
"content_hash": "b" * 64,
"access_scope": "collective_internal",
"ingestion_origin": "system_import",
"provenance_status": "verified",
"source_class": "primary_record",
"auto_promotion_policy": "human_required",
"captured_at": "2026-07-18T10:01:00+00:00",
"created_by": agent_id,
},
],
"evidence": evidence,
"edges": [
{
"id": "12121212-1212-4121-8121-121212121212",
"from_claim": CLAIM_A,
"to_claim": CLAIM_B,
"relation_type": "challenges",
"rationale": "Claim A supplies a bounded reason to challenge claim B.",
"scope_or_conditions": "Only inside the disposable V3 verification scope.",
"created_by": agent_id,
}
],
"assessments": [
{
"id": "13131313-1313-4131-8131-131313131313",
"claim_id": CLAIM_A,
"support_tier": "strong",
"challenge_tier": "none",
"overall_state": "support_dominant",
"rationale": "One exact primary record supports the bounded empirical claim.",
"evidence_set_hash": ap._v3_evidence_set_hash(CLAIM_A, evidence),
"supersedes_assessment_id": None,
},
{
"id": "14141414-1414-4141-8141-141414141414",
"claim_id": CLAIM_B,
"support_tier": None,
"challenge_tier": None,
"overall_state": "challenge_dominant",
"rationale": "The conceptual claim has an explicit challenge without numeric evidence tiers.",
"evidence_set_hash": ap._v3_evidence_set_hash(CLAIM_B, evidence),
"supersedes_assessment_id": None,
},
],
"reasoning_tools": [],
}
def test_approve_claim_bundle_sql_is_atomic_and_row_verified():
payload = _approve_claim_bundle()
sql = ap.build_approve_claim_sql(
@ -264,6 +415,233 @@ def test_approve_claim_bundle_sql_is_atomic_and_row_verified():
assert "kb_stage.finish_approved_proposal" in sql
def test_v3_approve_claim_sql_writes_complete_epistemic_bundle() -> None:
payload = _approve_claim_v3_bundle()
proposal_id = "99999999-9999-4999-8999-999999999999"
sql = ap.build_approve_claim_sql(
payload,
proposal_id,
None,
_approval_meta("approve_claim", payload),
)
shared_lock = f"select pg_catalog.pg_advisory_lock_shared({ap.V3_CATALOG_DDL_ADVISORY_LOCK_KEY});"
shared_unlock = f"select pg_catalog.pg_advisory_unlock_shared({ap.V3_CATALOG_DDL_ADVISORY_LOCK_KEY});"
assert sql.startswith(f"{shared_lock}\nbegin;")
assert "set transaction isolation level serializable" in sql
assert sql.rstrip().endswith(shared_unlock)
assert sql.count(shared_lock) == 1
assert sql.count(shared_unlock) == 1
assert "insert into public.claims" in sql
assert "body_markdown" in sql
assert "accepted_by_proposal_id" in sql
assert "insert into public.sources" in sql
assert "content_hash" in sql
assert "insert into public.claim_evidence (" in sql
assert "locator_json" in sql
assert "insert into public.claim_edges (" in sql
assert "relation_type" in sql
assert "insert into public.claim_evidence_assessments" in sql
assert "evidence_set_hash" in sql
assert "insert into public.reasoning_tools" not in sql
expected_inserts = sum(len(payload[key]) for key in ("sources", "claims", "evidence", "edges", "assessments"))
assert sql.lower().count("on conflict do nothing") == expected_inserts
assert sql.index("do $v3_contract_guard$") < sql.index("kb_stage.assert_approved_proposal")
relation_locks = [
f"select kb_stage.{ap.V3_CATALOG_LOCK_FUNCTION}();",
"lock table public.claim_edges in row exclusive mode;",
"lock table public.claim_evidence in row exclusive mode;",
"lock table public.claim_evidence_assessments in row exclusive mode;",
"lock table public.claims in row exclusive mode;",
"lock table public.sources in row exclusive mode;",
]
ordered_catalog_guard = [
shared_lock,
"begin;",
"set transaction isolation level serializable;",
*relation_locks,
"do $v3_contract_guard$",
"commit;",
shared_unlock,
]
assert [sql.index(statement) for statement in ordered_catalog_guard] == sorted(
sql.index(statement) for statement in ordered_catalog_guard
)
assert sql.index("kb_stage.assert_approved_proposal") < sql.index("insert into public.sources")
assert sql.index("insert into public.claim_evidence_assessments") < sql.index("kb_stage.finish_approved_proposal")
assert sql.index("on conflict do nothing") < sql.index("do $verify$")
assert "set local timezone = 'UTC'" in sql
assert "apply_proposal: V3 catalog contract is %s; refusing mutating transaction" in sql
assert "and superseded_by is null" in sql
expected_creator_checks = sum(len(payload[key]) for key in ("sources", "claims", "evidence", "edges"))
assert sql.count("created_by is not distinct from") == expected_creator_checks
def test_v3_catalog_guard_requires_the_full_hardened_contract() -> None:
sql = ap.build_v3_contract_state_query()
assert ap.V3_CATALOG_DDL_ADVISORY_LOCK_KEY == 6072343533146485505
assert ap.V3_PROTECTED_RELATIONS == (
("kb_stage", "kb_proposals"),
("public", "claim_edges"),
("public", "claim_evidence"),
("public", "claim_evidence_assessments"),
("public", "claims"),
("public", "sources"),
)
assert (
tuple(sorted({(row["schema_name"], row["table_name"]) for row in ap.V3_EXPECTED_TRIGGER_CONTRACT}))
== ap.V3_PROTECTED_RELATIONS
)
assert len(ap.V3_EXPECTED_TRIGGER_CONTRACT) == 24
assert len({row["trigger_name"] for row in ap.V3_EXPECTED_TRIGGER_CONTRACT}) == 24
assert all(row["when_expression"] is None for row in ap.V3_EXPECTED_TRIGGER_CONTRACT)
assert "when not artifacts_present then 'V2'" in sql
assert "when functions_valid and catalog_lock_function_valid and triggers_valid" in sql
assert "and owner_role_valid and memberships_valid then 'V3'" in sql
assert "else 'INVALID'" in sql
assert "actual.tgenabled = 'O'" in sql
assert "actual.tgtype = expected.trigger_type" in sql
assert "actual.update_columns = expected.update_columns" in sql
assert "actual.when_expression is not distinct from expected.when_expression" in sql
assert "actual.argument_hex = expected.argument_hex" in sql
assert "actual.tgconstraint = 0::pg_catalog.oid" in sql
assert "actual.tgoldtable is null" in sql
assert "actual.tgnewtable is null" in sql
assert "actual.tgfoid" in sql
assert "owner_role.rolname = E'kb_gate_owner'" in sql
assert "actual.function_security_definer = expected.function_security_definer" in sql
assert "actual.function_source_sha256 = expected.function_source_sha256" in sql
assert f"procedure.proname = E'{ap.V3_CATALOG_LOCK_FUNCTION}'" in sql
assert ap.V3_CATALOG_LOCK_FUNCTION_SOURCE_SHA256 in sql
assert "and prorettype = 'pg_catalog.void'::pg_catalog.regtype" in sql
assert "and grantee.rolname = 'kb_apply'" in sql
assert "and execute_acl_exact" in sql
assert "acl.grantee <> procedure.proowner" in sql
assert "from pg_catalog.pg_auth_members membership" in sql
def test_migration_trigger_contract_and_catalog_queries_match_worker_builder() -> None:
migration = (REPO_ROOT / "ops" / "teleo_v3_epistemic_contract.sql").read_text(encoding="utf-8")
contract_match = re.search(r"\$trigger_contract\$\s*(\[.*?\])\s*\$trigger_contract\$", migration, re.DOTALL)
assert contract_match is not None
assert json.loads(contract_match.group(1)) == list(ap.V3_EXPECTED_TRIGGER_CONTRACT)
exclusive_lock = f"select pg_catalog.pg_advisory_xact_lock({ap.V3_CATALOG_DDL_ADVISORY_LOCK_KEY});"
assert migration.count(exclusive_lock) == 1
assert migration.index(exclusive_lock) < migration.index(
f"create or replace function kb_stage.{ap.V3_CATALOG_LOCK_FUNCTION}()"
)
assert migration.index(exclusive_lock) < migration.index(
"create or replace function kb_stage.teleo_v3_enforce_apply_contract_cutover()"
)
assert migration.index(exclusive_lock) < migration.index("create or replace trigger")
expected_query = ap.build_v3_contract_state_query(
into="v_contract_state",
expected_contract_expression="pg_catalog.current_setting('teleo.v3_expected_trigger_contract')",
)
for marker in ("STATE", "POSTFLIGHT"):
query_match = re.search(
rf"TELEO_V3_CONTRACT_{marker}_QUERY_BEGIN\n(.*?)\n\s*-- TELEO_V3_CONTRACT_{marker}_QUERY_END",
migration,
re.DOTALL,
)
assert query_match is not None
assert query_match.group(1).strip() == expected_query + ";"
@pytest.mark.parametrize(
("mutation", "error"),
[
(lambda payload: payload["claims"][0].update(body_markdown=""), "body_markdown"),
(lambda payload: payload["claims"][0].update(revision_kind="semantic"), "original claims only"),
(lambda payload: payload["sources"][0].update(content_hash="not-a-hash"), "SHA-256"),
(lambda payload: payload["evidence"][0].update(source_content_hash="b" * 64), "does not match"),
(lambda payload: payload["assessments"][0].update(evidence_set_hash="c" * 64), "evidence_set_hash"),
(
lambda payload: payload["assessments"][0].update(
supersedes_assessment_id="15151515-1515-4151-8151-151515151515"
),
"assessment supersession",
),
(lambda payload: payload.update(reasoning_tools=[{"id": CLAIM_A}]), "reasoning_tools"),
],
)
def test_v3_approve_claim_rejects_incomplete_or_unbound_payloads(mutation, error) -> None:
payload = _approve_claim_v3_bundle()
mutation(payload)
with pytest.raises(ValueError, match=error):
ap.build_approve_claim_sql(
payload,
"99999999-9999-4999-8999-999999999999",
None,
_approval_meta("approve_claim", payload),
)
def test_v3_approve_claim_preserves_exact_reviewed_text_for_replay_receipts() -> None:
payload = _approve_claim_v3_bundle()
payload["claims"][0]["body_markdown"] = " Exact reviewed body with intentional spacing. "
normalized = ap._normalize_v3_approve_claim(payload)
assert normalized["claims"][0]["body_markdown"] == payload["claims"][0]["body_markdown"]
@pytest.mark.parametrize(
("captured_at", "expected"),
[
("2026-07-18T10:00:00Z", "2026-07-18T10:00:00.000000Z"),
("2026-07-18T12:00:00+02:00", "2026-07-18T10:00:00.000000Z"),
("2026-07-18T12:00:00.123400+02:00", "2026-07-18T10:00:00.123400Z"),
("2026-07-18T10:00:00.1Z", "2026-07-18T10:00:00.100000Z"),
],
)
def test_v3_approve_claim_normalizes_capture_timestamp_to_fixed_utc(
captured_at: str,
expected: str,
) -> None:
payload = _approve_claim_v3_bundle()
payload["sources"][0]["captured_at"] = captured_at
normalized = ap._normalize_v3_approve_claim(payload)
assert normalized["sources"][0]["captured_at"] == expected
@pytest.mark.parametrize(
"captured_at",
[
"2026-07-18 10:00:00+00:00",
"2026-07-18T10:00:00",
"2026-07-18T10:00:00.1234567Z",
"not-a-timestamp",
],
)
def test_v3_approve_claim_rejects_untyped_capture_timestamp(captured_at: str) -> None:
payload = _approve_claim_v3_bundle()
payload["sources"][0]["captured_at"] = captured_at
with pytest.raises(ValueError, match="timezone-aware ISO-8601"):
ap._normalize_v3_approve_claim(payload)
@pytest.mark.parametrize(
"spelling",
[
CLAIM_A.upper(),
"{" + CLAIM_A + "}",
CLAIM_A.replace("-", ""),
],
)
def test_v3_approve_claim_rejects_noncanonical_uuid_spellings(spelling: str) -> None:
payload = _approve_claim_v3_bundle()
payload["claims"][0]["id"] = spelling
with pytest.raises(ValueError, match="canonical lowercase hyphenated UUID"):
ap._normalize_v3_approve_claim(payload)
def test_approve_claim_semantic_rows_accept_matches_and_reject_payload_mismatches():
payload = _approve_claim_bundle()
sql = ap.build_approve_claim_sql(
@ -445,6 +823,72 @@ def test_build_apply_sql_requires_apply_payload():
ap.build_apply_sql(proposal, None)
@pytest.mark.parametrize(
("proposal", "error"),
[
({"proposal_type": "approve_claim", "payload": []}, "payload must be a JSON object"),
(
{"proposal_type": "approve_claim", "payload": {"apply_payload": []}},
"payload.apply_payload must be a JSON object",
),
(
{
"proposal_type": "approve_claim",
"payload": {"apply_payload": {"contract_version": "3"}},
},
"must be integer 2 or 3",
),
(
{
"proposal_type": "approve_claim",
"payload": {"apply_payload": {"contract_version": 3.0}},
},
"must be integer 2 or 3",
),
(
{
"proposal_type": "approve_claim",
"payload": {"apply_payload": {"contract_version": True}},
},
"must be integer 2 or 3",
),
(
{
"proposal_type": "add_edge",
"payload": {"apply_payload": {"contract_version": 3}},
},
"must omit contract_version",
),
(
{
"proposal_type": "add_edge",
"payload": {"apply_payload": {"contract_version": None}},
},
"must omit contract_version",
),
],
)
def test_exact_contract_gate_rejects_malformed_and_hybrid_proposals(
proposal: dict,
error: str,
) -> None:
with pytest.raises(ValueError, match=error):
ap.proposal_contract_version(proposal)
def test_mutating_subprocess_contract_expectation_is_reasserted() -> None:
payload = _approve_claim_v3_bundle()
proposal = {
"id": "99999999-9999-4999-8999-999999999999",
"proposal_type": "approve_claim",
"payload": {"apply_payload": payload},
}
assert ap.assert_expected_contract_version(proposal, 3) == 3
with pytest.raises(ValueError, match="does not match expected version 2"):
ap.assert_expected_contract_version(proposal, 2)
def test_build_apply_sql_rejects_unsupported_type():
proposal = {"id": "pid", "proposal_type": "reject_claim", "payload": {"apply_payload": {}}}
with pytest.raises(ValueError):

212
tests/test_apply_worker.py
View file

@ -42,18 +42,27 @@ sys.path.insert(0, str(REPO_ROOT / "scripts"))
import apply_worker as w # noqa: E402
REAL_DETECT_INSTALLED_CONTRACT = w.detect_installed_contract
def _dependency_restore():
original_load_password = w.ap.load_password
original_run_psql = w.ap.run_psql
original_load_failure_state = w.load_failure_state
original_fetch_candidates = w.fetch_candidates
original_detect_installed_contract = w.detect_installed_contract
original_apply_one = w.apply_one
original_render_one = w.render_one
original_subprocess_run = w.subprocess.run
w.detect_installed_contract = lambda args, _password: (
w.CONTRACT_STATE_V3 if args.contract_version == 3 else w.CONTRACT_STATE_V2
)
yield
w.ap.load_password = original_load_password
w.ap.run_psql = original_run_psql
w.load_failure_state = original_load_failure_state
w.fetch_candidates = original_fetch_candidates
w.detect_installed_contract = original_detect_installed_contract
w.apply_one = original_apply_one
w.render_one = original_render_one
w.subprocess.run = original_subprocess_run
@ -86,6 +95,80 @@ def test_candidate_query_requires_apply_payload():
assert "payload ? 'apply_payload'" in sql
def test_candidate_query_is_pinned_to_v2_by_default() -> None:
sql = w.build_candidate_query()
assert "contract_version' = '2'" in sql
assert "contract_version' = '3'" not in sql
def test_candidate_query_can_be_pinned_to_v3_without_selecting_v2() -> None:
sql = w.build_candidate_query(contract_version=3)
assert "contract_version' = '3'" in sql
assert "contract_version' = '2'" not in sql
assert "proposal_type = 'approve_claim'" in sql
assert "proposal_type in ('revise_strategy'" not in sql
assert "jsonb_typeof(payload) = 'object'" in sql
assert "jsonb_typeof(payload->'apply_payload') = 'object'" in sql
assert "jsonb_typeof(payload->'apply_payload'->'contract_version') = 'number'" in sql
def test_candidate_query_rejects_unknown_contract_version() -> None:
with pytest.raises(ValueError, match="contract_version"):
w.build_candidate_query(contract_version=4)
def test_cli_can_select_the_v3_worker_contract_explicitly() -> None:
assert w.parse_args(["--contract-version", "3"]).contract_version == 3
def test_cli_rejects_a_noncanonical_exact_proposal_uuid() -> None:
with pytest.raises(SystemExit):
w.parse_args(["--proposal-id", "AAAAAAAA-AAAA-4AAA-8AAA-AAAAAAAAAAAA"])
def test_candidate_query_can_select_one_exact_proposal_without_queue_fallback() -> None:
proposal_id = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"
sql = w.build_candidate_query(contract_version=3, proposal_id=proposal_id)
exact_clause = f"id = {w.ap.sql_literal(proposal_id)}::uuid"
assert exact_clause in sql
assert sql.index(exact_clause) < sql.index("limit 20")
def test_candidate_query_rejects_noncanonical_exact_proposal_id() -> None:
with pytest.raises(ValueError, match="canonical"):
w.build_candidate_query(contract_version=3, proposal_id="AAAAAAAA-AAAA-4AAA-8AAA-AAAAAAAAAAAA")
def test_installed_contract_detection_is_exact_and_tri_state() -> None:
seen: list[str] = []
w.ap.run_psql = lambda _args, sql, _password: seen.append(sql) or w.CONTRACT_STATE_V3
assert REAL_DETECT_INSTALLED_CONTRACT(_args(contract_version=3), "pw") == w.CONTRACT_STATE_V3
assert "then 'V2'" in seen[0]
assert "else 'INVALID'" in seen[0]
assert "actual.tgenabled = 'O'" in seen[0]
assert "actual.update_columns = expected.update_columns" in seen[0]
assert "actual.when_expression is not distinct from expected.when_expression" in seen[0]
assert "actual.argument_hex = expected.argument_hex" in seen[0]
assert "actual.tgfoid" in seen[0]
assert "owner_role.rolname = E'kb_gate_owner'" in seen[0]
assert "actual.function_security_definer = expected.function_security_definer" in seen[0]
assert "actual.function_source_sha256 = expected.function_source_sha256" in seen[0]
assert "search_path=pg_catalog, pg_temp" in seen[0]
assert "acl.grantee <> procedure.proowner" in seen[0]
assert "not owner_role.rolcanlogin" in seen[0]
assert "from pg_catalog.pg_auth_members membership" in seen[0]
def test_installed_contract_detection_rejects_unknown_readback() -> None:
w.ap.run_psql = lambda *_args, **_kwargs: "drifted"
with pytest.raises(RuntimeError, match="V2, V3, or INVALID"):
REAL_DETECT_INSTALLED_CONTRACT(_args(contract_version=3), "pw")
def test_candidate_query_lists_worker_types_only():
sql = w.build_candidate_query()
for t in w.WORKER_TYPES:
@ -116,6 +199,20 @@ def test_parse_candidates_skips_blank_lines():
assert rows[0]["id"] == "a"
def test_fetch_candidates_fails_closed_on_hybrid_query_result() -> None:
w.ap.run_psql = lambda *_args, **_kwargs: json.dumps(
{
"id": "hybrid",
"proposal_type": "add_edge",
"agent_id": None,
"contract_version": "3",
}
)
with pytest.raises(RuntimeError, match="outside the exact worker contract"):
w.fetch_candidates(_args(contract_version=3), "pw", ())
# --- render hook ---------------------------------------------------------- #
def test_render_command_none_when_unset():
assert w.build_render_command("", "agent-1") is None
@ -134,6 +231,8 @@ def test_render_command_expands_agent_id():
def _args(**over):
base = dict(
enable=False,
contract_version=2,
proposal_id=None,
limit=20,
max_per_tick=1,
max_attempts=3,
@ -269,9 +368,28 @@ def test_apply_one_supplies_deterministic_private_receipt_path():
assert len(calls) == 1
assert calls[0][calls[0].index("--receipt-dir") + 1] == str(expected.parent)
assert calls[0][calls[0].index("--receipt-out") + 1] == str(expected)
assert calls[0][calls[0].index("--expected-contract-version") + 1] == "2"
assert stat.S_IMODE(expected.stat().st_mode) == 0o600
def test_apply_one_forwards_the_exact_fresh_preflight() -> None:
proposal_id = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"
receipt_dir = Path(tempfile.mkdtemp()) / "receipts"
preflight = Path(tempfile.mkdtemp()) / "fresh-preflight.json"
calls = []
def _run(cmd, **_kwargs):
calls.append(cmd)
receipt_path = Path(cmd[cmd.index("--receipt-out") + 1])
_write_valid_private_receipt(receipt_path, proposal_id)
return _completed(0, stdout='{"applied": true}')
w.subprocess.run = _run
w.apply_one(_args(receipt_dir=str(receipt_dir), fresh_preflight=preflight), proposal_id)
assert calls[0][calls[0].index("--fresh-preflight") + 1] == str(preflight)
def test_apply_one_recovers_postcommit_receipt_failure_without_reapplying():
proposal_id = "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb"
receipt_dir = Path(tempfile.mkdtemp()) / "receipts"
@ -406,6 +524,61 @@ def test_postcommit_receipt_error_does_not_increment_apply_failure_count():
assert w.load_failure_state(path) == {}
def test_installed_v3_contract_retains_and_skips_visible_v2_queue(capsys) -> None:
applied: list[str] = []
w.ap.load_password = lambda _f: "pw"
w.detect_installed_contract = lambda _args, _password: w.CONTRACT_STATE_V3
w.fetch_candidates = lambda _args, _password, excluded: [
{
"id": "legacy-v2",
"proposal_type": "add_edge",
"agent_id": None,
"contract_version": "2",
}
]
w.apply_one = lambda _args, proposal_id: applied.append(proposal_id)
rc = w.run(_args(enable=True, contract_version=2))
captured = capsys.readouterr()
assert rc == 0
assert applied == []
assert "SKIP contract v2 execution" in captured.err
assert "remain review-visible" in captured.err
assert "migrate" in captured.err.lower()
assert "reject/cancel" in captured.err
def test_v3_worker_refuses_before_database_contract_install(capsys) -> None:
applied: list[str] = []
w.ap.load_password = lambda _f: "pw"
w.detect_installed_contract = lambda _args, _password: w.CONTRACT_STATE_V2
w.fetch_candidates = lambda *_args: (_ for _ in ()).throw(AssertionError("candidate selection must not run"))
w.apply_one = lambda _args, proposal_id: applied.append(proposal_id)
rc = w.run(_args(enable=True, contract_version=3))
assert rc == 1
assert applied == []
assert "V3 cutover is not installed" in capsys.readouterr().err
def test_invalid_contract_aborts_both_workers_before_candidate_selection(capsys) -> None:
selected: list[int] = []
applied: list[str] = []
w.ap.load_password = lambda _f: "pw"
w.detect_installed_contract = lambda _args, _password: w.CONTRACT_STATE_INVALID
w.fetch_candidates = lambda *_args: selected.append(1) or []
w.apply_one = lambda _args, proposal_id: applied.append(proposal_id)
for version in (2, 3):
assert w.run(_args(enable=True, contract_version=version)) == 1
assert selected == []
assert applied == []
assert capsys.readouterr().err.count("no candidate selection or writes attempted") == 2
def test_report_only_gate_does_not_apply(monkeypatch=None):
calls = []
w.ap.load_password = lambda _f: "pw"
@ -417,6 +590,45 @@ def test_report_only_gate_does_not_apply(monkeypatch=None):
assert calls == [] # disabled worker performs NO writes
def test_exact_proposal_selector_refuses_missing_target_without_applying(capsys) -> None:
proposal_id = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"
applied: list[str] = []
w.ap.load_password = lambda _f: "pw"
w.fetch_candidates = lambda *_args: []
w.apply_one = lambda _args, selected: applied.append(selected)
rc = w.run(_args(enable=True, contract_version=3, proposal_id=proposal_id))
captured = capsys.readouterr()
assert rc == 1
assert applied == []
assert proposal_id in captured.err
assert "no writes attempted" in captured.err
def test_required_exact_proposal_returns_distinct_missing_candidate_status(capsys) -> None:
proposal_id = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"
applied: list[str] = []
w.ap.load_password = lambda _f: "pw"
w.fetch_candidates = lambda *_args: []
w.apply_one = lambda _args, selected: applied.append(selected)
rc = w.run(
_args(
enable=True,
contract_version=3,
proposal_id=proposal_id,
require_candidate=True,
)
)
captured = capsys.readouterr()
assert rc == 2
assert applied == []
assert proposal_id in captured.err
assert "no writes attempted" in captured.err
def test_enabled_worker_applies_and_renders():
applied, rendered = [], []
w.ap.load_password = lambda _f: "pw"

547
tests/test_build_v3_genesis_ledger_bundle.py Normal file
View file

@ -0,0 +1,547 @@
from __future__ import annotations
import hashlib
import json
import stat
from pathlib import Path
import pytest
from ops import build_v3_genesis_ledger_bundle as builder
from ops import run_local_genesis_ledger_rebuild as genesis_rebuild
from scripts import compile_kb_source_packet as compiler
REVIEWER_ID = "11111111-1111-4111-8111-111111111111"
SERVICE_ID = "44444444-4444-4444-8444-444444444444"
PRIVATE_MARKER = "PRIVATE-V3-MATERIAL-MUST-NOT-BE-PRINTED"
def _sha256(value: bytes) -> str:
return hashlib.sha256(value).hexdigest()
def _write_manifest(path: Path) -> Path:
singleton_rows = [
{
"kind": "identity",
"database": "teleo_fixture",
"server_version_num": 160010,
"transaction_read_only": "on",
"server_address": None,
"server_port": None,
"ssl": False,
"ssl_version": None,
"database_collation": "C",
"database_ctype": "C",
},
{"kind": "schemas", "items": ["kb_stage", "public"]},
]
singleton_rows.extend(
{"kind": kind, "items": []}
for kind in sorted(genesis_rebuild.load_manifest.__globals__["SINGLETON_KINDS"] - {"identity", "schemas"})
)
rows = [
*singleton_rows,
{
"kind": "table",
"schema": "public",
"table": "claims",
"row_count": 1,
"rowset_md5": "f" * 32,
},
]
path.write_text("".join(json.dumps(row, sort_keys=True) + "\n" for row in rows), encoding="utf-8")
return path
def _compiled_child(tmp_path: Path, *, salt: str) -> dict:
artifact_bytes = f"retained artifact {salt}\n".encode()
quote = f"The exact retained source {salt} requires review before canonical apply."
extracted_text = f"Header {salt}\n{quote}\n"
text_bytes = extracted_text.encode()
quote_bytes = quote.encode()
start = text_bytes.index(quote_bytes)
extractor_spec = f"fixture-extractor={salt}\n".encode()
artifact_sha256 = _sha256(artifact_bytes)
manifest = {
"schema": compiler.MANIFEST_SCHEMA_V3,
"artifact_sha256": artifact_sha256,
"extracted_text_sha256": _sha256(text_bytes),
"extractor": {
"name": "fixture-semantic-extractor",
"version": "3.1.0",
"spec_sha256": _sha256(extractor_spec),
},
"agent_id": REVIEWER_ID,
"source": {
"identity": f"document:sha256:{artifact_sha256}",
"source_key": f"source_{salt}",
"source_type": "article",
"title": f"Source {salt}",
"locator": f"artifact://sha256/{artifact_sha256}",
"access_scope": "collective_internal",
"ingestion_origin": "operator_upload",
"provenance_status": "verified",
"source_class": "primary_record",
"auto_promotion_policy": "human_required",
"captured_at": "2026-07-18T10:00:00+00:00",
},
"claims": [
{
"claim_key": f"review_gate_{salt}",
"type": "empirical",
"proposition": f"Source {salt} remains non-canonical until guarded review and apply succeed.",
"body_markdown": f"The retained source {salt} explicitly preserves the review boundary.",
"scope": f"The exact retained source fixture {salt}.",
"access_scope": "collective_internal",
"owner_agent_id": None,
"status": "active",
"revision_criteria": "Revise if the exact source bytes or review contract changes.",
"revision_kind": "original",
"supersedes_id": None,
"tags": ["genesis", "provenance", salt],
"evidence": [
{
"evidence_key": f"exact_quote_{salt}",
"quote": quote,
"polarity": "supports",
"locator": {
"unit": "utf8_bytes",
"start": start,
"end": start + len(quote_bytes),
},
"rationale": "The exact byte-bound sentence directly supports the bounded proposition.",
}
],
"assessment": {
"support_tier": "strong",
"challenge_tier": "none",
"overall_state": "support_dominant",
"rationale": "One exact verified primary-record sentence supports the claim.",
"supersedes_assessment_id": None,
},
}
],
}
input_dir = tmp_path / f"compiler-{salt}"
input_dir.mkdir()
artifact = input_dir / "artifact.bin"
text = input_dir / "extracted.txt"
manifest_path = input_dir / "manifest.json"
extractor_path = input_dir / "extractor.spec"
artifact.write_bytes(artifact_bytes)
text.write_text(extracted_text, encoding="utf-8")
manifest_path.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8")
extractor_path.write_bytes(extractor_spec)
return compiler.compile_source_packet(artifact, text, manifest_path, extractor_path)["strict_child_proposal"]
def _applied_envelope(proposal: dict) -> dict:
fields = (
"id",
"proposal_type",
"status",
"payload",
"reviewed_by_handle",
"reviewed_by_agent_id",
"reviewed_at",
"review_note",
"applied_by_handle",
"applied_by_agent_id",
"applied_at",
)
return {field: proposal[field] for field in fields}
def _v3_material(tmp_path: Path, *, sequence: int, salt: str) -> dict:
child = _compiled_child(tmp_path, salt=salt)
approved = {
"id": child["id"],
"proposal_type": child["proposal_type"],
"status": "approved",
"proposed_by_handle": child["proposed_by_handle"],
"proposed_by_agent_id": child["proposed_by_agent_id"],
"channel": child["channel"],
"source_ref": child["source_ref"],
"rationale": PRIVATE_MARKER,
"payload": child["payload"],
"reviewed_by_handle": "m3taversal",
"reviewed_by_agent_id": REVIEWER_ID,
"reviewed_at": "2026-07-18T10:01:00.000000Z",
"review_note": "Reviewed the exact source-normalized V3 payload.",
"applied_by_handle": None,
"applied_by_agent_id": None,
"applied_at": None,
"created_at": "2026-07-18T10:00:30.000000Z",
"updated_at": "2026-07-18T10:01:00.000000Z",
}
applied = {
**approved,
"status": "applied",
"applied_by_handle": "kb-apply",
"applied_by_agent_id": SERVICE_ID,
"applied_at": "2026-07-18T10:02:00.000000Z",
"updated_at": "2026-07-18T10:02:00.000001Z",
}
approval = {
"proposal_id": approved["id"],
"proposal_type": approved["proposal_type"],
"payload": approved["payload"],
"reviewed_by_handle": approved["reviewed_by_handle"],
"reviewed_by_agent_id": approved["reviewed_by_agent_id"],
"reviewed_by_db_role": "kb_review",
"reviewed_at": approved["reviewed_at"],
"review_note": approved["review_note"],
}
proposal = _applied_envelope(applied)
apply_payload = proposal["payload"]["apply_payload"]
rows = genesis_rebuild.replay_receipt._expected_v3_semantic_rows(proposal, apply_payload)
created_at = "2026-07-18T10:01:30.000000Z"
for table, table_rows in rows.items():
for row in table_rows:
if table == "claims":
row.update(created_at=created_at, updated_at=created_at)
elif table in {
"sources",
"claim_evidence",
"claim_edges",
"claim_evidence_assessments",
}:
row["created_at"] = created_at
apply_sql = genesis_rebuild.apply_engine.build_apply_sql(proposal, applied["applied_by_handle"])
receipt = genesis_rebuild.replay_receipt.build_replay_receipt(
proposal,
rows,
apply_sql_sha256=genesis_rebuild.replay_receipt.sha256_text(apply_sql),
apply_sql_source="exact_executed_sql",
exported_at_utc="2026-07-18T10:03:00.000000Z",
)
return {
"artifact": genesis_rebuild.MATERIAL_ARTIFACT,
"contract_version": genesis_rebuild.MATERIAL_CONTRACT_VERSION,
"sequence": sequence,
"approved_proposal": approved,
"approval_snapshot": approval,
"applied_proposal": applied,
"replay_receipt": receipt,
}
def _v2_material() -> dict:
proposal_id = "70000000-0000-4000-8000-000000000001"
claim_a = "70000000-0000-4000-8000-000000000002"
claim_b = "70000000-0000-4000-8000-000000000003"
payload = {"apply_payload": {"from_claim": claim_a, "to_claim": claim_b, "edge_type": "supports", "weight": 0.8}}
approved = {
"id": proposal_id,
"proposal_type": "add_edge",
"status": "approved",
"proposed_by_handle": "fixture",
"proposed_by_agent_id": REVIEWER_ID,
"channel": "fixture",
"source_ref": "private://fixture",
"rationale": PRIVATE_MARKER,
"payload": payload,
"reviewed_by_handle": "m3taversal",
"reviewed_by_agent_id": REVIEWER_ID,
"reviewed_at": "2026-07-18T10:01:00.000000Z",
"review_note": "Reviewed exact V2 fixture.",
"applied_by_handle": None,
"applied_by_agent_id": None,
"applied_at": None,
"created_at": "2026-07-18T10:00:30.000000Z",
"updated_at": "2026-07-18T10:01:00.000000Z",
}
applied = {
**approved,
"status": "applied",
"applied_by_handle": "kb-apply",
"applied_by_agent_id": SERVICE_ID,
"applied_at": "2026-07-18T10:02:00.000000Z",
"updated_at": "2026-07-18T10:02:00.000001Z",
}
approval = {
"proposal_id": proposal_id,
"proposal_type": "add_edge",
"payload": payload,
"reviewed_by_handle": "m3taversal",
"reviewed_by_agent_id": REVIEWER_ID,
"reviewed_by_db_role": "kb_review",
"reviewed_at": approved["reviewed_at"],
"review_note": approved["review_note"],
}
receipt = genesis_rebuild.replay_receipt.build_replay_receipt(
_applied_envelope(applied),
{
"claim_edges": [
{
"id": "70000000-0000-4000-8000-000000000004",
"from_claim": claim_a,
"to_claim": claim_b,
"edge_type": "supports",
"weight": 0.8,
"created_by": None,
"created_at": "2026-07-18T10:01:30.000000Z",
}
]
},
apply_sql_sha256=genesis_rebuild.replay_receipt.sha256_text("exact V2 fixture SQL"),
apply_sql_source="exact_executed_sql",
exported_at_utc="2026-07-18T10:03:00.000000Z",
)
return {
"artifact": genesis_rebuild.MATERIAL_ARTIFACT,
"contract_version": genesis_rebuild.MATERIAL_CONTRACT_VERSION,
"sequence": 1,
"approved_proposal": approved,
"approval_snapshot": approval,
"applied_proposal": applied,
"replay_receipt": receipt,
}
def _write_json(path: Path, value: object, *, private: bool = False) -> Path:
path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n", encoding="utf-8")
if private:
path.chmod(0o600)
return path
@pytest.fixture
def bundle_inputs(tmp_path: Path) -> dict[str, object]:
dump = tmp_path / "genesis.dump"
dump.write_bytes(b"PGDMPfixture")
genesis_manifest = _write_manifest(tmp_path / "genesis-manifest.jsonl")
final_manifest = _write_manifest(tmp_path / "final-manifest.jsonl")
materials = [
_write_json(tmp_path / "material-0001.json", _v3_material(tmp_path, sequence=1, salt="alpha"), private=True),
_write_json(tmp_path / "material-0002.json", _v3_material(tmp_path, sequence=2, salt="beta"), private=True),
]
note = tmp_path / "review-note.txt"
note.write_text(f"Explicitly reviewed all source-backed entries. {PRIVATE_MARKER}\n", encoding="utf-8")
return {
"dump": dump,
"genesis_manifest": genesis_manifest,
"final_manifest": final_manifest,
"materials": materials,
"note": note,
}
def _build_args(inputs: dict[str, object], ledger: Path, *, admission: Path | None = None) -> list[str]:
args = [
"--genesis-dump",
str(inputs["dump"]),
"--genesis-manifest",
str(inputs["genesis_manifest"]),
"--final-manifest",
str(inputs["final_manifest"]),
]
for material in inputs["materials"]:
args.extend(("--material", str(material)))
args.extend(("--output-ledger", str(ledger)))
if admission is not None:
args.extend(
(
"--emit-admission-manifest",
"--admission-output",
str(admission),
"--reviewed-by",
"m3taversal",
"--reviewed-by-agent-id",
REVIEWER_ID,
"--reviewed-at",
"2026-07-18T14:00:00+02:00",
"--review-note-file",
str(inputs["note"]),
"--admission-scope",
"collective_core",
"--admission-basis",
"forgejo_primary_source",
)
)
return args
def _verify_args(inputs: dict[str, object], ledger: Path, admission: Path | None = None) -> list[str]:
args = [
"--genesis-dump",
str(inputs["dump"]),
"--genesis-manifest",
str(inputs["genesis_manifest"]),
"--verify-ledger",
str(ledger),
"--ledger-sha256",
genesis_rebuild.canonical_rebuild.sha256_file(ledger),
]
if admission is not None:
args.extend(
(
"--admission-manifest",
str(admission),
"--admission-manifest-sha256",
genesis_rebuild.canonical_rebuild.sha256_file(admission),
)
)
return args
def test_deterministic_build_and_verify_every_pin_with_no_writes(
bundle_inputs: dict[str, object], tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
first_ledger = tmp_path / "ledger-first.json"
second_ledger = tmp_path / "ledger-second.json"
first_admission = tmp_path / "admission-first.json"
second_admission = tmp_path / "admission-second.json"
assert builder.main(_build_args(bundle_inputs, first_ledger, admission=first_admission)) == 0
first_output = capsys.readouterr()
assert builder.main(_build_args(bundle_inputs, second_ledger, admission=second_admission)) == 0
second_output = capsys.readouterr()
assert first_ledger.read_bytes() == second_ledger.read_bytes()
assert first_admission.read_bytes() == second_admission.read_bytes()
assert PRIVATE_MARKER not in first_output.out + first_output.err + second_output.out + second_output.err
before = {path: (path.read_bytes(), path.stat().st_mtime_ns) for path in (first_ledger, first_admission)}
assert builder.main(_verify_args(bundle_inputs, first_ledger, first_admission)) == 0
verification = json.loads(capsys.readouterr().out)
assert verification["action"] == "verify"
assert verification["entry_count"] == 2
assert verification["admission"]["review_metadata_provenance"] == builder.REVIEW_METADATA_PROVENANCE
assert verification["admission"]["cryptographically_authenticated"] is False
assert before == {path: (path.read_bytes(), path.stat().st_mtime_ns) for path in (first_ledger, first_admission)}
def test_reordered_materials_and_v2_material_are_refused(
bundle_inputs: dict[str, object], tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
reversed_inputs = {**bundle_inputs, "materials": list(reversed(bundle_inputs["materials"]))}
assert builder.main(_build_args(reversed_inputs, tmp_path / "reordered.json")) == 1
assert not (tmp_path / "reordered.json").exists()
assert json.loads(capsys.readouterr().err)["code"] == "invalid_exported_material"
v2_path = _write_json(tmp_path / "v2-material.json", _v2_material(), private=True)
v2_inputs = {**bundle_inputs, "materials": [v2_path]}
assert builder.main(_build_args(v2_inputs, tmp_path / "v2-ledger.json")) == 1
assert not (tmp_path / "v2-ledger.json").exists()
assert json.loads(capsys.readouterr().err)["code"] == "legacy_material_forbidden"
def test_hash_drift_engine_tamper_and_symlink_material_fail_closed(
bundle_inputs: dict[str, object], tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
ledger = tmp_path / "ledger.json"
assert builder.main(_build_args(bundle_inputs, ledger)) == 0
capsys.readouterr()
first_material = bundle_inputs["materials"][0]
first_material.write_bytes(first_material.read_bytes() + b" ")
assert builder.main(_verify_args(bundle_inputs, ledger)) == 1
assert json.loads(capsys.readouterr().err)["code"] == "genesis_preflight_refused"
first_material.write_bytes(first_material.read_bytes()[:-1])
tampered = json.loads(ledger.read_text(encoding="utf-8"))
tampered["engine"]["apply_engine_sha256"] = "0" * 64
_write_json(ledger, tampered, private=True)
assert builder.main(_verify_args(bundle_inputs, ledger)) == 1
assert json.loads(capsys.readouterr().err)["code"] == "genesis_preflight_refused"
symlink = tmp_path / "material-symlink.json"
symlink.symlink_to(bundle_inputs["materials"][0])
symlink_inputs = {**bundle_inputs, "materials": [symlink]}
assert builder.main(_build_args(symlink_inputs, tmp_path / "symlink-ledger.json")) == 1
assert json.loads(capsys.readouterr().err)["code"] == "unsafe_input"
@pytest.mark.parametrize(
("mutation", "expected_code"),
[
("missing_explicit_flag", "implicit_admission_forbidden"),
("missing_reviewer_id", "incomplete_admission_review"),
("alternate_reviewer", "invalid_reviewer_handle"),
("noncanonical_reviewer_id", "invalid_reviewer_identity"),
],
)
def test_admission_requires_complete_exact_operator_review(
bundle_inputs: dict[str, object],
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
mutation: str,
expected_code: str,
) -> None:
ledger = tmp_path / f"{mutation}-ledger.json"
admission = tmp_path / f"{mutation}-admission.json"
args = _build_args(bundle_inputs, ledger, admission=admission)
if mutation == "missing_explicit_flag":
args.remove("--emit-admission-manifest")
elif mutation == "missing_reviewer_id":
index = args.index("--reviewed-by-agent-id")
del args[index : index + 2]
elif mutation == "alternate_reviewer":
args[args.index("--reviewed-by") + 1] = "m3ta"
else:
args[args.index("--reviewed-by-agent-id") + 1] = "AAAAAAAA-AAAA-4AAA-8AAA-AAAAAAAAAAAA"
assert builder.main(args) == 1
output = capsys.readouterr()
assert json.loads(output.err)["code"] == expected_code
assert PRIVATE_MARKER not in output.out + output.err
assert not ledger.exists()
assert not admission.exists()
def test_admission_always_selects_all_entries_and_partial_manifest_is_rejected(
bundle_inputs: dict[str, object], tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
ledger = tmp_path / "ledger.json"
admission = tmp_path / "admission.json"
assert builder.main(_build_args(bundle_inputs, ledger, admission=admission)) == 0
capsys.readouterr()
manifest = json.loads(admission.read_text(encoding="utf-8"))
assert [entry["sequence"] for entry in manifest["entries"]] == [1, 2]
manifest["entries"].pop()
_write_json(admission, manifest, private=True)
assert builder.main(_verify_args(bundle_inputs, ledger, admission)) == 1
assert json.loads(capsys.readouterr().err)["code"] == "genesis_preflight_refused"
def test_existing_outputs_are_never_overwritten_and_outputs_are_private(
bundle_inputs: dict[str, object], tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
existing = tmp_path / "existing-ledger.json"
existing.write_text("operator-owned\n", encoding="utf-8")
assert builder.main(_build_args(bundle_inputs, existing)) == 1
assert existing.read_text(encoding="utf-8") == "operator-owned\n"
assert json.loads(capsys.readouterr().err)["code"] == "existing_output"
ledger = tmp_path / "private-ledger.json"
admission = tmp_path / "private-admission.json"
assert builder.main(_build_args(bundle_inputs, ledger, admission=admission)) == 0
output = capsys.readouterr()
assert stat.S_IMODE(ledger.stat().st_mode) == 0o600
assert stat.S_IMODE(admission.stat().st_mode) == 0o600
assert ledger.stat().st_nlink == admission.stat().st_nlink == 1
assert PRIVATE_MARKER not in output.out + output.err
def test_material_tamper_is_redacted_and_publishes_nothing(
bundle_inputs: dict[str, object], tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
material_path = bundle_inputs["materials"][0]
material = json.loads(material_path.read_text(encoding="utf-8"))
material["approved_proposal"]["payload"]["private_secret"] = PRIVATE_MARKER
_write_json(material_path, material, private=True)
tampered_inputs = {**bundle_inputs, "materials": [material_path]}
ledger = tmp_path / "tampered-ledger.json"
assert builder.main(_build_args(tampered_inputs, ledger)) == 1
output = capsys.readouterr()
assert not ledger.exists()
assert PRIVATE_MARKER not in output.out + output.err
assert json.loads(output.err)["status"] == "refused"

841
tests/test_compile_kb_source_packet.py
View file

@ -3,9 +3,12 @@ from __future__ import annotations
import copy
import hashlib
import json
import os
import stat
import subprocess
import sys
import tarfile
import uuid
from pathlib import Path
import pytest
@ -24,6 +27,123 @@ EXTRACTED_TEXT = (
)
CLAIM_QUOTE = "A staged proposal remains non-canonical until a human review and guarded apply both succeed."
EVIDENCE_QUOTE = "Every proposed claim must retain an exact quote from the extracted source text."
EXTRACTOR_SPEC_BYTES = (
b"livingip semantic extractor specification\n"
b"name=fixture-semantic-extractor\n"
b"version=3.1.0\n"
b"quotes=exact-utf8-byte-locators\n"
)
REPAIRED_WRITER_HEAD = "6fdaf19b96d027bc614e8ade0d475ba0954740b9"
V1_RENDERED_BUNDLE_SHA256 = "64795816f6b28e61df34cd00a62ffae12c7cc40bb26c475a56d47b4507d1bf73"
V2_RENDERED_BUNDLE_SHA256 = "e0a240371885a2c7fc714dd1b08099ef122d472e76ae1065542c1f305a27d816"
EXACT_WRITER_FILES = (
"scripts/apply_proposal.py",
"scripts/kb_apply_replay_receipt.py",
)
EXACT_WRITER_PROBE = r"""
import copy
import hashlib
import json
import sys
from pathlib import Path
scripts_dir = Path(sys.argv[1]).resolve()
child_path = Path(sys.argv[2]).resolve()
resolved_commit = sys.argv[3]
forbidden_checkout = Path(sys.argv[4]).resolve()
def canonical_bytes(value):
return json.dumps(value, ensure_ascii=True, separators=(",", ":"), sort_keys=True).encode("utf-8")
def sha256_bytes(value):
return hashlib.sha256(value).hexdigest()
sys.path.insert(0, str(scripts_dir))
import apply_proposal as writer
import kb_apply_replay_receipt as replay_receipt
module_origins = {
"apply_proposal": str(Path(writer.__file__).resolve()),
"kb_apply_replay_receipt": str(Path(replay_receipt.__file__).resolve()),
}
expected_origins = {
"apply_proposal": str(scripts_dir / "apply_proposal.py"),
"kb_apply_replay_receipt": str(scripts_dir / "kb_apply_replay_receipt.py"),
}
assert module_origins == expected_origins
for origin in module_origins.values():
origin_path = Path(origin)
assert origin_path.is_relative_to(scripts_dir)
assert not origin_path.is_relative_to(forbidden_checkout)
def forbidden_process(*_args, **_kwargs):
raise AssertionError("exact-writer compatibility probe attempted process or database execution")
writer.subprocess.run = forbidden_process
raw_child = child_path.read_bytes()
child = json.loads(raw_child.decode("utf-8"))
payload_before = copy.deepcopy(child["payload"])
apply_payload = child["payload"]["apply_payload"]
apply_payload_before = canonical_bytes(apply_payload)
normalized = writer._normalize_v3_approve_claim(apply_payload)
roles = {"supports": "grounds", "challenges": "contradicts", "illustrates": "illustrates"}
expected_evidence = []
for row in apply_payload["evidence"]:
expected = copy.deepcopy(row)
expected["role"] = roles[row["polarity"]]
expected_evidence.append(expected)
assert normalized == {
"agent_id": apply_payload["agent_id"],
"claims": apply_payload["claims"],
"sources": apply_payload["sources"],
"evidence": expected_evidence,
"edges": apply_payload["edges"],
"assessments": apply_payload["assessments"],
}
assert normalized["sources"][0]["captured_at"] == apply_payload["sources"][0]["captured_at"]
assert apply_payload["sources"][0]["captured_at"].endswith(".000000Z")
assert canonical_bytes(apply_payload) == apply_payload_before
approved = copy.deepcopy(child)
approved.update(
status="approved",
reviewed_by_handle="source-compiler-v3-exact-writer-test",
reviewed_by_agent_id=None,
reviewed_at="2000-01-01T00:00:00.000000Z",
review_note="Build-only exact-writer compatibility validation.",
)
sql = writer.build_apply_sql(approved, writer.SERVICE_AGENT_HANDLE)
assert "begin;" in sql
assert approved["payload"] == payload_before
assert child["payload"] == payload_before
assert child_path.read_bytes() == raw_child
assert sha256_bytes(canonical_bytes(child["payload"])) == child["payload_sha256"]
receipt = {
"resolved_commit": resolved_commit,
"module_origins": module_origins,
"module_sha256": {
name: sha256_bytes(Path(origin).read_bytes()) for name, origin in module_origins.items()
},
"input_child_file_sha256": sha256_bytes(raw_child),
"strict_child_payload_sha256": child["payload_sha256"],
"apply_payload_sha256_before": sha256_bytes(apply_payload_before),
"apply_payload_sha256_after": sha256_bytes(canonical_bytes(apply_payload)),
"payload_file_bytes_preserved": child_path.read_bytes() == raw_child,
"payload_object_preserved": child["payload"] == payload_before,
"canonical_timestamp_equal": normalized["sources"][0]["captured_at"]
== apply_payload["sources"][0]["captured_at"],
"build_apply_sql_sha256": sha256_bytes(sql.encode("utf-8")),
"sql_executed": False,
}
print(json.dumps(receipt, ensure_ascii=True, sort_keys=True))
"""
def _sha256(value: bytes) -> str:
@ -59,6 +179,78 @@ def _manifest() -> dict:
}
def _byte_locator(quote: str, text: str = EXTRACTED_TEXT) -> dict:
text_bytes = text.encode("utf-8")
quote_bytes = quote.encode("utf-8")
start = text_bytes.index(quote_bytes)
return {"unit": "utf8_bytes", "start": start, "end": start + len(quote_bytes)}
def _manifest_v3(
*,
artifact_bytes: bytes = ARTIFACT_BYTES,
extracted_text: str = EXTRACTED_TEXT,
extractor_spec_bytes: bytes = EXTRACTOR_SPEC_BYTES,
) -> dict:
artifact_sha256 = _sha256(artifact_bytes)
return {
"schema": compiler.MANIFEST_SCHEMA_V3,
"artifact_sha256": artifact_sha256,
"extracted_text_sha256": _sha256(extracted_text.encode("utf-8")),
"extractor": {
"name": "fixture-semantic-extractor",
"version": "3.1.0",
"spec_sha256": _sha256(extractor_spec_bytes),
},
"agent_id": "11111111-1111-4111-8111-111111111111",
"source": {
"identity": f"document:sha256:{artifact_sha256}",
"source_key": "source_compiler_note_v3",
"source_type": "article",
"title": "Source packet compiler note",
"locator": f"artifact://sha256/{artifact_sha256}",
"access_scope": "collective_internal",
"ingestion_origin": "operator_upload",
"provenance_status": "verified",
"source_class": "primary_record",
"auto_promotion_policy": "human_required",
"captured_at": "2026-07-18T10:00:00+00:00",
},
"claims": [
{
"claim_key": "review_gate_precedes_canonical_apply_v3",
"type": "empirical",
"proposition": "A staged proposal remains non-canonical until review and guarded apply succeed.",
"body_markdown": "The source explicitly separates staging from review and guarded canonical apply.",
"scope": "The retained source packet compiler note and its exact cited sentence.",
"access_scope": "collective_internal",
"owner_agent_id": None,
"status": "active",
"revision_criteria": "Revise if the bound source sentence changes or no longer supports the proposition.",
"revision_kind": "original",
"supersedes_id": None,
"tags": ["review", "v3", "provenance"],
"evidence": [
{
"evidence_key": "review_gate_exact_sentence",
"quote": CLAIM_QUOTE,
"polarity": "supports",
"locator": _byte_locator(CLAIM_QUOTE, extracted_text),
"rationale": "The exact sentence directly supports the bounded proposition.",
}
],
"assessment": {
"support_tier": "strong",
"challenge_tier": "none",
"overall_state": "support_dominant",
"rationale": "One exact verified primary-record sentence directly supports the claim.",
"supersedes_assessment_id": None,
},
}
],
}
def _write_inputs(tmp_path: Path, manifest: dict | None = None, prefix: str = "input") -> tuple[Path, Path, Path]:
tmp_path.mkdir(parents=True, exist_ok=True)
artifact = tmp_path / f"{prefix}.bin"
@ -75,6 +267,173 @@ def _compile(tmp_path: Path, manifest: dict | None = None) -> dict:
return compiler.compile_source_packet(*paths)
def _write_v3_inputs(
tmp_path: Path,
manifest: dict | None = None,
*,
artifact_bytes: bytes = ARTIFACT_BYTES,
extracted_text: str = EXTRACTED_TEXT,
extractor_spec_bytes: bytes = EXTRACTOR_SPEC_BYTES,
prefix: str = "v3-input",
) -> tuple[Path, Path, Path, Path]:
tmp_path.mkdir(parents=True, exist_ok=True)
artifact = tmp_path / f"{prefix}.bin"
text = tmp_path / f"{prefix}.txt"
manifest_path = tmp_path / f"{prefix}.json"
extractor_spec = tmp_path / f"{prefix}.extractor-spec"
artifact.write_bytes(artifact_bytes)
text.write_text(extracted_text, encoding="utf-8")
manifest_path.write_text(
json.dumps(
manifest
or _manifest_v3(
artifact_bytes=artifact_bytes,
extracted_text=extracted_text,
extractor_spec_bytes=extractor_spec_bytes,
),
indent=2,
),
encoding="utf-8",
)
extractor_spec.write_bytes(extractor_spec_bytes)
return artifact, text, manifest_path, extractor_spec
def _compile_v3(
tmp_path: Path,
manifest: dict | None = None,
*,
artifact_bytes: bytes = ARTIFACT_BYTES,
extracted_text: str = EXTRACTED_TEXT,
extractor_spec_bytes: bytes = EXTRACTOR_SPEC_BYTES,
) -> dict:
return compiler.compile_source_packet(
*_write_v3_inputs(
tmp_path,
manifest,
artifact_bytes=artifact_bytes,
extracted_text=extracted_text,
extractor_spec_bytes=extractor_spec_bytes,
)
)
def _rendered_bundle_sha256(bundle: dict) -> str:
rendered = json.dumps(bundle, ensure_ascii=True, indent=2, sort_keys=True) + "\n"
return _sha256(rendered.encode("utf-8"))
def _resolve_exact_writer_commit(requested_head: str) -> str:
assert requested_head == REPAIRED_WRITER_HEAD, (
"KB_SOURCE_COMPILER_WRITER_COMPAT_HEAD must name the exact reviewed writer commit "
f"{REPAIRED_WRITER_HEAD}, got {requested_head!r}"
)
completed = subprocess.run(
["git", "rev-parse", "--verify", f"{requested_head}^{{commit}}"],
cwd=REPO_ROOT,
capture_output=True,
check=False,
text=True,
)
assert completed.returncode == 0, (
f"could not resolve exact writer commit {requested_head}: {completed.stderr.strip()}"
)
resolved = completed.stdout.strip()
assert resolved == requested_head, f"writer ref resolved to {resolved}, expected exact commit {requested_head}"
return resolved
def _git_file_sha256(commit: str, path: str) -> str:
completed = subprocess.run(
["git", "show", f"{commit}:{path}"],
cwd=REPO_ROOT,
capture_output=True,
check=False,
)
assert completed.returncode == 0, f"could not read {path} from exact writer commit {commit}"
return _sha256(completed.stdout)
def _run_exact_writer_compatibility(child: dict, tmp_path: Path) -> dict:
tmp_path.mkdir(parents=True, exist_ok=True)
requested_head = os.environ.get("KB_SOURCE_COMPILER_WRITER_COMPAT_HEAD", REPAIRED_WRITER_HEAD)
resolved = _resolve_exact_writer_commit(requested_head)
writer_root = tmp_path / "exact-writer"
writer_root.mkdir()
archive_path = tmp_path / "exact-writer.tar"
archived = subprocess.run(
[
"git",
"archive",
"--format=tar",
f"--output={archive_path}",
resolved,
"--",
*EXACT_WRITER_FILES,
],
cwd=REPO_ROOT,
capture_output=True,
check=False,
text=True,
)
assert archived.returncode == 0, f"could not materialize exact writer commit {resolved}: {archived.stderr}"
with tarfile.open(archive_path, "r") as archive:
for member in archive.getmembers():
member_path = Path(member.name)
assert not member_path.is_absolute() and ".." not in member_path.parts
archive.extractall(writer_root)
writer_scripts = writer_root / "scripts"
expected_origins = {
"apply_proposal": str((writer_scripts / "apply_proposal.py").resolve()),
"kb_apply_replay_receipt": str((writer_scripts / "kb_apply_replay_receipt.py").resolve()),
}
expected_module_sha256 = {
"apply_proposal": _git_file_sha256(resolved, "scripts/apply_proposal.py"),
"kb_apply_replay_receipt": _git_file_sha256(resolved, "scripts/kb_apply_replay_receipt.py"),
}
child_path = tmp_path / "strict-child.json"
child_bytes = compiler.canonical_json_bytes(child)
child_path.write_bytes(child_bytes)
completed = subprocess.run(
[
sys.executable,
"-I",
"-c",
EXACT_WRITER_PROBE,
str(writer_scripts),
str(child_path),
resolved,
str(REPO_ROOT),
],
cwd=tmp_path,
capture_output=True,
check=False,
text=True,
env={"PYTHONHASHSEED": "0"},
)
assert completed.returncode == 0, (
f"exact writer {resolved} compatibility execution failed\nstdout: {completed.stdout}\nstderr: {completed.stderr}"
)
receipt = json.loads(completed.stdout)
assert receipt["resolved_commit"] == resolved
assert receipt["module_origins"] == expected_origins
assert receipt["module_sha256"] == expected_module_sha256
assert receipt["input_child_file_sha256"] == _sha256(child_bytes)
assert receipt["strict_child_payload_sha256"] == child["payload_sha256"]
assert receipt["apply_payload_sha256_before"] == receipt["apply_payload_sha256_after"]
assert receipt["payload_file_bytes_preserved"] is True
assert receipt["payload_object_preserved"] is True
assert receipt["canonical_timestamp_equal"] is True
assert receipt["sql_executed"] is False
assert len(receipt["build_apply_sql_sha256"]) == 64
receipt["requested_commit"] = requested_head
receipt_path = tmp_path / "exact-writer-compatibility-receipt.json"
receipt_path.write_text(json.dumps(receipt, ensure_ascii=True, indent=2, sort_keys=True) + "\n", encoding="utf-8")
receipt["receipt_path"] = str(receipt_path)
return receipt
def test_compiles_deterministic_hash_bound_pending_review_bundle_without_db_write(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
@ -162,6 +521,488 @@ def test_v2_manifest_carries_canonical_dedupe_candidates_into_review_packet(tmp_
assert staged_assessment == assessment
def test_v1_v2_rendered_bundle_bytes_match_pre_repair_baselines(tmp_path: Path) -> None:
artifact, text, manifest_path = _write_inputs(tmp_path / "v1")
v1 = compiler.compile_source_packet(artifact, text, manifest_path)
v2_manifest = _manifest()
v2_manifest["schema"] = compiler.MANIFEST_SCHEMA_V2
v2_manifest["dedupe"] = {
"database_search_query": "test query",
"candidate_claims": [],
"duplicates": [],
}
artifact, text, manifest_path = _write_inputs(tmp_path / "v2", v2_manifest)
v2 = compiler.compile_source_packet(artifact, text, manifest_path)
assert _rendered_bundle_sha256(v1) == V1_RENDERED_BUNDLE_SHA256
assert _rendered_bundle_sha256(v2) == V2_RENDERED_BUNDLE_SHA256
def test_v3_compiles_exact_deterministic_pending_review_payload_without_db_write(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
first_paths = _write_v3_inputs(tmp_path / "first", prefix="original-source")
def unexpected_db_call(*_args, **_kwargs):
raise AssertionError("V3 compiler attempted a database write")
monkeypatch.setattr(compiler.stage, "stage_normalized_child", unexpected_db_call)
monkeypatch.setattr(compiler.stage.bound, "_psql_json", unexpected_db_call)
first = compiler.compile_source_packet(*first_paths)
second = compiler.compile_source_packet(*_write_v3_inputs(tmp_path / "copied", prefix="copied-source"))
assert first == second
assert first["schema"] == compiler.BUNDLE_SCHEMA_V3
assert first["status"] == "pending_review"
assert first["build_only"] is True
assert first["database_write_performed"] is False
assert first["canonical_apply_performed"] is False
assert first["stage_preview"] == {
"target_table": "kb_stage.kb_proposals",
"target_status": "pending_review",
"executed": False,
"stage_sql_sha256": first["stage_preview"]["stage_sql_sha256"],
}
assert len(first["stage_preview"]["stage_sql_sha256"]) == 64
parent = first["parent_proposal"]
child = first["strict_child_proposal"]
stage_sql = compiler.stage.build_stage_sql(child)
assert stage_sql.count("insert into kb_stage.kb_proposals") == 1
assert "'pending_review'" in stage_sql
assert "public." not in stage_sql
assert first["stage_preview"]["stage_sql_sha256"] == _sha256(stage_sql.encode("utf-8"))
apply_payload = child["payload"]["apply_payload"]
assert parent["status"] == child["status"] == "pending_review"
assert child["proposal_type"] == "approve_claim"
assert child["payload_sha256"] == compiler.canonical_sha256(child["payload"])
assert apply_payload["contract_version"] == 3
assert apply_payload["source_proposal_id"] == parent["id"]
assert apply_payload["agent_id"] == _manifest_v3()["agent_id"]
assert apply_payload["edges"] == []
assert apply_payload["reasoning_tools"] == []
assert len(apply_payload["sources"]) == 1
assert len(apply_payload["claims"]) == 1
assert len(apply_payload["evidence"]) == 1
assert len(apply_payload["assessments"]) == 1
source = apply_payload["sources"][0]
assert source == {
"id": source["id"],
"source_type": "article",
"canonical_url": None,
"storage_uri": _manifest_v3()["source"]["locator"],
"excerpt": CLAIM_QUOTE,
"content_hash": _sha256(ARTIFACT_BYTES),
"access_scope": "collective_internal",
"ingestion_origin": "operator_upload",
"provenance_status": "verified",
"source_class": "primary_record",
"auto_promotion_policy": "human_required",
"captured_at": "2026-07-18T10:00:00.000000Z",
"created_by": _manifest_v3()["agent_id"],
}
claim = apply_payload["claims"][0]
assert claim["proposition"] == _manifest_v3()["claims"][0]["proposition"]
assert claim["body_markdown"] == _manifest_v3()["claims"][0]["body_markdown"]
assert claim["scope"] == _manifest_v3()["claims"][0]["scope"]
assert claim["revision_criteria"] == _manifest_v3()["claims"][0]["revision_criteria"]
assert claim["revision_kind"] == "original"
assert claim["supersedes_id"] is None
assert claim["created_by"] == apply_payload["agent_id"]
evidence = apply_payload["evidence"][0]
assert evidence["claim_id"] == claim["id"]
assert evidence["source_id"] == source["id"]
assert evidence["excerpt"] == CLAIM_QUOTE
assert evidence["polarity"] == "supports"
assert evidence["source_content_hash"] == source["content_hash"]
locator = evidence["locator_json"]
assert locator["unit"] == "utf8_bytes"
assert EXTRACTED_TEXT.encode("utf-8")[locator["start"] : locator["end"]] == CLAIM_QUOTE.encode("utf-8")
assert locator["quote_sha256"] == _sha256(CLAIM_QUOTE.encode("utf-8"))
assert locator["artifact_sha256"] == _sha256(ARTIFACT_BYTES)
assert locator["extracted_text_sha256"] == _sha256(EXTRACTED_TEXT.encode("utf-8"))
assert locator["extractor"] == _manifest_v3()["extractor"]
assessment = apply_payload["assessments"][0]
assert assessment["claim_id"] == claim["id"]
assert assessment["evidence_set_hash"] == compiler.ap._v3_evidence_set_hash(claim["id"], [evidence])
assert assessment["supersedes_assessment_id"] is None
all_ids = [
parent["id"],
child["id"],
source["id"],
claim["id"],
evidence["id"],
assessment["id"],
]
assert len(all_ids) == len(set(all_ids))
assert all(value == str(uuid.UUID(value)) for value in all_ids)
identity = apply_payload["normalization_manifest"]["identity_material"]
assert identity == {
"artifact_sha256": _sha256(ARTIFACT_BYTES),
"extracted_text_sha256": _sha256(EXTRACTED_TEXT.encode("utf-8")),
"extractor": _manifest_v3()["extractor"],
"manifest_file_sha256": first["hashes"]["manifest_file_sha256"],
"source_identity": _manifest_v3()["source"]["identity"],
"source_locator": _manifest_v3()["source"]["locator"],
"normalized_candidate_content_sha256": first["hashes"]["normalized_candidate_content_sha256"],
}
assert apply_payload["normalization_manifest"]["semantic_extraction_is_retained_input"] is True
assert apply_payload["normalization_manifest"]["model_training_claim"] is False
assert "retained input" in apply_payload["normalization_manifest"]["semantic_extraction_statement"]
assert first["hashes"]["extractor_spec_sha256"] == _sha256(EXTRACTOR_SPEC_BYTES)
assert first["compiler_contract"]["writer_apply_invoked"] is False
without_bundle_hash = copy.deepcopy(first)
bundle_hash = without_bundle_hash["hashes"].pop("bundle_content_sha256")
assert bundle_hash == compiler.canonical_sha256(without_bundle_hash)
def test_v3_exact_writer_commit_preserves_payload_and_builds_apply_sql(tmp_path: Path) -> None:
bundle = _compile_v3(tmp_path)
apply_payload = bundle["strict_child_proposal"]["payload"]["apply_payload"]
assert apply_payload["sources"][0]["captured_at"] == "2026-07-18T10:00:00.000000Z"
receipt = _run_exact_writer_compatibility(bundle["strict_child_proposal"], tmp_path / "writer-proof")
assert receipt["resolved_commit"] == REPAIRED_WRITER_HEAD
assert receipt["apply_payload_sha256_before"] == receipt["apply_payload_sha256_after"]
assert Path(receipt["receipt_path"]).is_file()
def test_v3_exact_writer_gate_cannot_fall_back_to_branch_local_apply_module(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
bundle = _compile_v3(tmp_path / "payload")
branch_local_origin = str(Path(compiler.ap.__file__).resolve())
def forbidden_branch_local_writer(*_args, **_kwargs):
raise AssertionError("exact writer gate used branch-local compiler.ap")
monkeypatch.setattr(compiler.ap, "_normalize_v3_approve_claim", forbidden_branch_local_writer)
monkeypatch.setattr(compiler.ap, "build_apply_sql", forbidden_branch_local_writer)
receipt = _run_exact_writer_compatibility(bundle["strict_child_proposal"], tmp_path / "writer-proof")
assert branch_local_origin not in receipt["module_origins"].values()
assert all(Path(origin).is_relative_to(tmp_path) for origin in receipt["module_origins"].values())
def test_v3_cli_rerun_is_byte_identical_and_emits_only_public_status(tmp_path: Path) -> None:
private_root = tmp_path / "private-v3-source"
artifact, text, manifest_path, extractor_spec = _write_v3_inputs(private_root)
outputs = [private_root / "compiled" / "first.json", private_root / "compiled" / "second.json"]
completed = []
for output in outputs:
completed.append(
subprocess.run(
[
sys.executable,
str(REPO_ROOT / "scripts" / "compile_kb_source_packet.py"),
"--artifact",
str(artifact),
"--text",
str(text),
"--manifest",
str(manifest_path),
"--extractor-spec",
str(extractor_spec),
"--output",
str(output),
],
capture_output=True,
check=False,
)
)
assert [item.returncode for item in completed] == [0, 0]
assert [item.stderr for item in completed] == [b"", b""]
assert completed[0].stdout == completed[1].stdout
assert outputs[0].read_bytes() == outputs[1].read_bytes()
assert json.loads(outputs[0].read_text(encoding="utf-8"))["schema"] == compiler.BUNDLE_SCHEMA_V3
public_status = json.loads(completed[0].stdout.decode("utf-8"))
assert public_status["status"] == "pending_review"
assert public_status["database_write_performed"] is False
assert public_status["canonical_apply_performed"] is False
assert EXTRACTED_TEXT.encode("utf-8") not in completed[0].stdout
assert str(private_root).encode("utf-8") not in completed[0].stdout
assert all(stat.S_IMODE(output.stat().st_mode) == 0o600 for output in outputs)
def test_v3_stale_artifact_fails_closed_and_rehashed_artifact_changes_identity(tmp_path: Path) -> None:
original = _compile_v3(tmp_path / "original")
mutated_bytes = ARTIFACT_BYTES + b"mutated"
stale_manifest = _manifest_v3()
with pytest.raises(compiler.CompilerError, match="artifact SHA-256 does not match"):
_compile_v3(tmp_path / "stale", stale_manifest, artifact_bytes=mutated_bytes)
rehashed_manifest = _manifest_v3(artifact_bytes=mutated_bytes)
stale_identity = copy.deepcopy(rehashed_manifest)
stale_identity["source"]["identity"] = _manifest_v3()["source"]["identity"]
with pytest.raises(compiler.CompilerError, match="content-addressed identity"):
_compile_v3(tmp_path / "stale-identity", stale_identity, artifact_bytes=mutated_bytes)
stale_locator = copy.deepcopy(rehashed_manifest)
stale_locator["source"]["locator"] = _manifest_v3()["source"]["locator"]
with pytest.raises(compiler.CompilerError, match="actual artifact SHA-256"):
_compile_v3(tmp_path / "stale-locator", stale_locator, artifact_bytes=mutated_bytes)
rehashed = _compile_v3(tmp_path / "rehashed", rehashed_manifest, artifact_bytes=mutated_bytes)
assert rehashed["strict_child_proposal"]["id"] != original["strict_child_proposal"]["id"]
assert rehashed["strict_child_proposal"]["payload_sha256"] != original["strict_child_proposal"]["payload_sha256"]
assert rehashed["hashes"]["packet_identity_sha256"] != original["hashes"]["packet_identity_sha256"]
@pytest.mark.parametrize("target", ["extractor_name", "extractor_version", "claim"])
def test_v3_extraction_identity_and_normalized_candidate_mutations_change_proposal(tmp_path: Path, target: str) -> None:
original = _compile_v3(tmp_path / "original")
manifest = _manifest_v3()
if target == "extractor_name":
manifest["extractor"]["name"] = "fixture-semantic-extractor-revised"
elif target == "extractor_version":
manifest["extractor"]["version"] = "3.1.1"
else:
manifest["claims"][0]["proposition"] += " The proposal remains reviewable."
mutated = _compile_v3(tmp_path / target, manifest)
assert mutated["strict_child_proposal"]["id"] != original["strict_child_proposal"]["id"]
assert mutated["strict_child_proposal"]["payload_sha256"] != original["strict_child_proposal"]["payload_sha256"]
assert mutated["hashes"]["packet_identity_sha256"] != original["hashes"]["packet_identity_sha256"]
def test_v3_requires_exact_independent_extractor_spec_and_binds_spec_mutations(tmp_path: Path) -> None:
artifact, text, manifest_path, _extractor_spec = _write_v3_inputs(tmp_path / "missing")
with pytest.raises(compiler.CompilerError, match="independently supplied extractor specification"):
compiler.compile_source_packet(artifact, text, manifest_path)
mismatched_spec = EXTRACTOR_SPEC_BYTES + b"mismatch\n"
with pytest.raises(compiler.CompilerError, match="specification SHA-256 does not match"):
_compile_v3(
tmp_path / "mismatched",
_manifest_v3(),
extractor_spec_bytes=mismatched_spec,
)
original = _compile_v3(tmp_path / "original")
revised_spec = EXTRACTOR_SPEC_BYTES + b"normalization=revision-2\n"
revised_manifest = _manifest_v3(extractor_spec_bytes=revised_spec)
revised = _compile_v3(
tmp_path / "revised",
revised_manifest,
extractor_spec_bytes=revised_spec,
)
assert revised["hashes"]["extractor_spec_sha256"] == _sha256(revised_spec)
assert revised["strict_child_proposal"]["id"] != original["strict_child_proposal"]["id"]
assert revised["strict_child_proposal"]["payload_sha256"] != original["strict_child_proposal"]["payload_sha256"]
assert revised["hashes"]["packet_identity_sha256"] != original["hashes"]["packet_identity_sha256"]
@pytest.mark.parametrize(
("field", "value", "expected"),
[
("identity", "document:alternate:source", "content-addressed identity"),
("locator", "artifact://mirror/source", "actual artifact SHA-256"),
],
)
def test_v3_rejects_unbound_source_identity_or_locator(tmp_path: Path, field: str, value: str, expected: str) -> None:
manifest = _manifest_v3()
manifest["source"][field] = value
with pytest.raises(compiler.CompilerError, match=expected):
_compile_v3(tmp_path, manifest)
def test_v3_exact_manifest_bytes_are_identity_bound_but_paths_are_not(tmp_path: Path) -> None:
direct_paths = _write_v3_inputs(tmp_path / "direct", prefix="source")
copied_paths = _write_v3_inputs(tmp_path / "copied", prefix="renamed")
direct = compiler.compile_source_packet(*direct_paths)
copied = compiler.compile_source_packet(*copied_paths)
assert direct == copied
compact_paths = _write_v3_inputs(tmp_path / "compact", prefix="source")
compact_manifest = compact_paths[2]
manifest_object = json.loads(compact_manifest.read_text(encoding="utf-8"))
compact_manifest.write_text(
json.dumps(manifest_object, separators=(",", ":"), sort_keys=True),
encoding="utf-8",
)
compact = compiler.compile_source_packet(*compact_paths)
assert compact["validated_manifest"] == direct["validated_manifest"]
assert compact["hashes"]["manifest_canonical_sha256"] == direct["hashes"]["manifest_canonical_sha256"]
assert compact["hashes"]["manifest_file_sha256"] != direct["hashes"]["manifest_file_sha256"]
assert compact["hashes"]["packet_identity_sha256"] != direct["hashes"]["packet_identity_sha256"]
assert compact["strict_child_proposal"]["id"] != direct["strict_child_proposal"]["id"]
assert compact["strict_child_proposal"]["payload_sha256"] != direct["strict_child_proposal"]["payload_sha256"]
def test_v3_valid_quote_binding_mutation_changes_identity_and_mismatches_fail_closed(tmp_path: Path) -> None:
original = _compile_v3(tmp_path / "original")
valid_mutation = _manifest_v3()
evidence = valid_mutation["claims"][0]["evidence"][0]
evidence["quote"] = EVIDENCE_QUOTE
evidence["locator"] = _byte_locator(EVIDENCE_QUOTE)
mutated = _compile_v3(tmp_path / "valid-mutation", valid_mutation)
assert mutated["strict_child_proposal"]["id"] != original["strict_child_proposal"]["id"]
assert mutated["strict_child_proposal"]["payload_sha256"] != original["strict_child_proposal"]["payload_sha256"]
quote_mismatch = _manifest_v3()
quote_mismatch["claims"][0]["evidence"][0]["quote"] = "Invented source sentence."
with pytest.raises(compiler.CompilerError, match="does not bind the exact quote bytes"):
_compile_v3(tmp_path / "quote-mismatch", quote_mismatch)
locator_mismatch = _manifest_v3()
locator_mismatch["claims"][0]["evidence"][0]["locator"]["start"] += 1
with pytest.raises(compiler.CompilerError, match="does not bind the exact quote bytes"):
_compile_v3(tmp_path / "locator-mismatch", locator_mismatch)
def test_v3_rejects_missing_assessment_relevant_evidence_and_tampered_set_hash(tmp_path: Path) -> None:
manifest = _manifest_v3()
manifest["claims"][0]["evidence"][0]["polarity"] = "illustrates"
manifest["claims"][0]["assessment"].update(
support_tier="none",
challenge_tier="none",
overall_state="insufficient",
)
with pytest.raises(compiler.CompilerError, match="requires at least one supports or challenges evidence"):
_compile_v3(tmp_path / "irrelevant", manifest)
bundle = _compile_v3(tmp_path / "valid")
child = copy.deepcopy(bundle["strict_child_proposal"])
child["payload"]["apply_payload"]["assessments"][0]["evidence_set_hash"] = "f" * 64
child["payload_sha256"] = compiler.canonical_sha256(child["payload"])
with pytest.raises(compiler.CompilerError, match="evidence_set_hash does not match claim-local evidence"):
compiler._validate_v3_compiled_child(child, bundle["validated_manifest"], bundle["quote_bindings"])
@pytest.mark.parametrize(
("mutation", "expected"),
[
(
lambda manifest: manifest.__setitem__("agent_id", manifest["agent_id"].replace("-", "")),
"canonical lowercase",
),
(
lambda manifest: manifest["source"].__setitem__("captured_at", "2026-07-18T10:00:00.0000000Z"),
"at most 6 fractional digits",
),
(lambda manifest: manifest["claims"][0].__setitem__("type", "structural"), "type must be one of"),
(
lambda manifest: manifest["claims"][0]["evidence"][0]["locator"].__setitem__("start", True),
"non-negative integer",
),
(
lambda manifest: manifest["claims"][0]["assessment"].__setitem__("support_tier", 1),
"support_tier must be null or one of",
),
],
)
def test_v3_rejects_noncanonical_typed_manifest_fields(tmp_path: Path, mutation, expected: str) -> None:
manifest = _manifest_v3()
mutation(manifest)
with pytest.raises(compiler.CompilerError, match=expected):
_compile_v3(tmp_path, manifest)
@pytest.mark.parametrize(
("needle", "replacement"),
[
(
f'"schema": "{compiler.MANIFEST_SCHEMA_V3}",',
(f'"schema": "{compiler.MANIFEST_SCHEMA_V3}",\n "schema": "{compiler.MANIFEST_SCHEMA_V3}",'),
),
(
f'"schema": "{compiler.MANIFEST_SCHEMA_V3}",',
f'"schema": "{compiler.MANIFEST_SCHEMA_V3}",\n "schema": "{compiler.MANIFEST_SCHEMA}",',
),
(
f'"spec_sha256": "{_sha256(EXTRACTOR_SPEC_BYTES)}"',
(
f'"spec_sha256": "{_sha256(EXTRACTOR_SPEC_BYTES)}",\n'
f' "spec_sha256": "{_sha256(EXTRACTOR_SPEC_BYTES)}"'
),
),
(
'"overall_state": "support_dominant",',
'"overall_state": "support_dominant",\n "overall_state": "insufficient",',
),
],
)
def test_v3_rejects_identical_or_conflicting_duplicate_json_keys_at_any_depth(
tmp_path: Path, needle: str, replacement: str
) -> None:
paths = _write_v3_inputs(tmp_path)
manifest_path = paths[2]
raw_manifest = manifest_path.read_text(encoding="utf-8")
assert raw_manifest.count(needle) == 1
manifest_path.write_text(raw_manifest.replace(needle, replacement, 1), encoding="utf-8")
with pytest.raises(compiler.CompilerError, match="duplicate JSON object key"):
compiler.compile_source_packet(*paths)
def test_v3_duplicate_schema_key_fails_before_identity_or_output(
tmp_path: Path,
capsys: pytest.CaptureFixture,
monkeypatch: pytest.MonkeyPatch,
) -> None:
artifact, text, manifest_path, extractor_spec = _write_v3_inputs(tmp_path)
schema_line = f'"schema": "{compiler.MANIFEST_SCHEMA_V3}",'
raw_manifest = manifest_path.read_text(encoding="utf-8")
manifest_path.write_text(
raw_manifest.replace(schema_line, f'{schema_line}\n "schema": "{compiler.MANIFEST_SCHEMA_V3}",', 1),
encoding="utf-8",
)
def unexpected_identity(*_args, **_kwargs):
raise AssertionError("duplicate-key manifest reached proposal identity")
monkeypatch.setattr(compiler, "_v3_identity_material", unexpected_identity)
output = tmp_path / "must-not-exist.json"
status = compiler.main(
[
"--artifact",
str(artifact),
"--text",
str(text),
"--manifest",
str(manifest_path),
"--extractor-spec",
str(extractor_spec),
"--output",
str(output),
]
)
assert status == 2
assert json.loads(capsys.readouterr().out)["reason"] == "input_validation_failed"
assert not output.exists()
def test_v3_rejects_duplicate_semantic_keys_and_duplicate_generated_ids(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
duplicate_claim = _manifest_v3()
duplicate_claim["claims"].append(copy.deepcopy(duplicate_claim["claims"][0]))
with pytest.raises(compiler.CompilerError, match="duplicate claim_key"):
_compile_v3(tmp_path / "claim-key", duplicate_claim)
duplicate_evidence = _manifest_v3()
duplicate_evidence["claims"][0]["evidence"].append(copy.deepcopy(duplicate_evidence["claims"][0]["evidence"][0]))
with pytest.raises(compiler.CompilerError, match="evidence_key duplicates"):
_compile_v3(tmp_path / "evidence-key", duplicate_evidence)
monkeypatch.setattr(
compiler,
"stable_uuid_v3",
lambda _row_kind, _packet_identity_sha256, _semantic_key: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
)
with pytest.raises(compiler.CompilerError, match="duplicate deterministic ids"):
_compile_v3(tmp_path / "generated-id")
def test_rejects_normalized_child_that_replaces_atomic_claim_text_with_quote(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:

340
tests/test_export_kb_transition_replay_material.py Normal file
View file

@ -0,0 +1,340 @@
from __future__ import annotations
import argparse
import copy
import hashlib
import json
import os
import stat
import sys
from pathlib import Path
import pytest
from ops import run_local_genesis_ledger_rebuild as genesis_rebuild
REPO_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(REPO_ROOT / "scripts"))
import export_kb_transition_replay_material as exporter # noqa: E402
import kb_apply_replay_receipt as replay_receipt # noqa: E402
PROPOSAL_ID = "70000000-0000-4000-8000-000000000001"
REVIEWER_ID = "70000000-0000-4000-8000-000000000002"
SERVICE_ID = "70000000-0000-4000-8000-000000000003"
CLAIM_A = "70000000-0000-4000-8000-000000000004"
CLAIM_B = "70000000-0000-4000-8000-000000000005"
def _material() -> dict:
payload = {
"apply_payload": {
"from_claim": CLAIM_A,
"to_claim": CLAIM_B,
"edge_type": "supports",
"weight": 0.75,
},
"title": "Private exact transition packet",
}
approved = {
"id": PROPOSAL_ID,
"proposal_type": "add_edge",
"status": "approved",
"proposed_by_handle": "leo",
"proposed_by_agent_id": REVIEWER_ID,
"channel": "private_export_test",
"source_ref": "private://transition/source",
"rationale": "Retain the exact approved row for deterministic reconstruction.",
"payload": payload,
"reviewed_by_handle": "m3ta",
"reviewed_by_agent_id": REVIEWER_ID,
"reviewed_at": "2026-07-18T10:01:00+00:00",
"review_note": "Approved the exact strict packet.",
"applied_by_handle": None,
"applied_by_agent_id": None,
"applied_at": None,
"created_at": "2026-07-18T10:00:00+00:00",
"updated_at": "2026-07-18T10:01:00.000001+00:00",
}
applied = {
**approved,
"status": "applied",
"applied_by_handle": "kb-apply",
"applied_by_agent_id": SERVICE_ID,
"applied_at": "2026-07-18T10:02:00+00:00",
"updated_at": "2026-07-18T10:02:00.000001+00:00",
}
approval = {
"proposal_id": PROPOSAL_ID,
"proposal_type": "add_edge",
"payload": payload,
"reviewed_by_handle": "m3ta",
"reviewed_by_agent_id": REVIEWER_ID,
"reviewed_by_db_role": "kb_review",
"reviewed_at": "2026-07-18T10:01:00+00:00",
"review_note": "Approved the exact strict packet.",
}
receipt_proposal = {field: applied[field] for field in exporter.RECEIPT_PROPOSAL_FIELDS}
receipt = replay_receipt.build_replay_receipt(
receipt_proposal,
{
"claim_edges": [
{
"id": "70000000-0000-4000-8000-000000000006",
"from_claim": CLAIM_A,
"to_claim": CLAIM_B,
"edge_type": "supports",
"weight": 0.75,
"created_by": None,
"created_at": "2026-07-18T10:01:30+00:00",
}
]
},
apply_sql_sha256=replay_receipt.sha256_text("exact guarded apply SQL"),
apply_sql_source="exact_executed_sql",
exported_at_utc="2026-07-18T10:03:00+00:00",
)
return {
"artifact": exporter.MATERIAL_ARTIFACT,
"contract_version": exporter.MATERIAL_CONTRACT_VERSION,
"sequence": 1,
"approved_proposal": approved,
"approval_snapshot": approval,
"applied_proposal": applied,
"replay_receipt": receipt,
}
def _rehash_worker_receipt(material: dict) -> dict:
receipt = material["replay_receipt"]
def sha256_json(value) -> str:
encoded = json.dumps(value, ensure_ascii=False, separators=(",", ":"), sort_keys=True).encode("utf-8")
return hashlib.sha256(encoded).hexdigest()
rows = receipt["canonical_rows"]
replay_material = {
"receipt_contract_version": receipt["receipt_contract_version"],
"apply_engine": receipt["apply_engine"],
"proposal": receipt["proposal"],
"canonical_rows": rows,
}
receipt["hashes"] = {
"apply_payload_sha256": sha256_json(receipt["proposal"]["payload"]["apply_payload"]),
"proposal_payload_sha256": sha256_json(receipt["proposal"]["payload"]),
"canonical_rows_sha256": sha256_json(rows),
"row_sha256": {table: [sha256_json(row) for row in table_rows] for table, table_rows in rows.items()},
"table_sha256": {table: sha256_json(table_rows) for table, table_rows in rows.items()},
"replay_material_sha256": sha256_json(replay_material),
}
return material
def _legacy_space_timestamp_material() -> dict:
material = copy.deepcopy(_material())
receipt = material["replay_receipt"]
receipt["proposal"]["reviewed_at"] = "2026-07-18 10:01:00+00"
receipt["proposal"]["applied_at"] = "2026-07-18 10:02:00+00"
receipt["canonical_rows"]["claim_edges"][0]["created_at"] = "2026-07-18 10:01:30+00"
receipt["exported_at_utc"] = "2026-07-18T10:03:00+00:00"
return _rehash_worker_receipt(material)
def test_material_matches_genesis_replay_row_contract() -> None:
material = exporter.validate_material(
_material(),
expected_proposal_id=PROPOSAL_ID,
expected_sequence=1,
)
assert exporter.MATERIAL_ARTIFACT == genesis_rebuild.MATERIAL_ARTIFACT
assert exporter.MATERIAL_CONTRACT_VERSION == genesis_rebuild.MATERIAL_CONTRACT_VERSION
assert exporter.PROPOSAL_FIELDS == genesis_rebuild.PROPOSAL_FIELDS
assert exporter.APPROVAL_FIELDS == genesis_rebuild.APPROVAL_FIELDS
genesis_rebuild._validate_proposal_rows(
material["approved_proposal"],
material["applied_proposal"],
material["approval_snapshot"],
material["replay_receipt"]["proposal"],
)
def test_legacy_postgres_worker_receipt_exports_and_reconstructs_without_rehashing(
tmp_path: Path,
) -> None:
material = _legacy_space_timestamp_material()
legacy_hash = material["replay_receipt"]["hashes"]["replay_material_sha256"]
validated = exporter.validate_material(
material,
expected_proposal_id=PROPOSAL_ID,
expected_sequence=1,
)
assert validated["replay_receipt"]["proposal"]["reviewed_at"] == "2026-07-18 10:01:00+00"
assert validated["replay_receipt"]["hashes"]["replay_material_sha256"] == legacy_hash
material_path = tmp_path / "legacy-transition.json"
material_path.write_text(json.dumps(validated), encoding="utf-8")
entry = genesis_rebuild._validate_entry_material(
validated,
expected_sequence=1,
expected_replay_hash=legacy_hash,
material_path=material_path,
material_sha256=hashlib.sha256(material_path.read_bytes()).hexdigest(),
)
assert entry.replay_material_sha256 == legacy_hash
assert entry.receipt["proposal"]["reviewed_at"] == "2026-07-18T10:01:00.000000Z"
assert entry.receipt["canonical_rows"]["claim_edges"][0]["created_at"] == ("2026-07-18T10:01:30.000000Z")
tampered = copy.deepcopy(material)
tampered["replay_receipt"]["hashes"]["replay_material_sha256"] = "0" * 64
with pytest.raises(exporter.ExportError, match="worker replay receipt"):
exporter.validate_material(tampered, expected_proposal_id=PROPOSAL_ID)
@pytest.mark.parametrize(
"separator",
["X", "_", "\U0001f642"],
ids=["x", "underscore", "emoji"],
)
def test_current_material_rejects_noncanonical_timestamp_separators(separator: str) -> None:
material = _material()
invalid = material["approved_proposal"]["reviewed_at"].replace("T", separator, 1)
material["approved_proposal"]["reviewed_at"] = invalid
material["applied_proposal"]["reviewed_at"] = invalid
material["approval_snapshot"]["reviewed_at"] = invalid
with pytest.raises(exporter.ExportError, match="timezone-aware timestamp"):
exporter.validate_material(material, expected_proposal_id=PROPOSAL_ID)
@pytest.mark.parametrize(
"separator",
["X", "_", "\U0001f642"],
ids=["x", "underscore", "emoji"],
)
@pytest.mark.parametrize("target", ["proposal", "canonical_row"])
def test_rehashed_legacy_receipt_rejects_hybrid_timestamp_separators(
separator: str,
target: str,
) -> None:
material = _legacy_space_timestamp_material()
receipt = material["replay_receipt"]
if target == "proposal":
receipt["proposal"]["applied_at"] = receipt["proposal"]["applied_at"].replace(" ", separator, 1)
else:
row = receipt["canonical_rows"]["claim_edges"][0]
row["created_at"] = row["created_at"].replace(" ", separator, 1)
_rehash_worker_receipt(material)
with pytest.raises(exporter.ExportError, match="worker replay receipt"):
exporter.validate_material(material, expected_proposal_id=PROPOSAL_ID)
@pytest.mark.parametrize(
("mutation", "message"),
[
(lambda value: value["approved_proposal"].__setitem__("unexpected", True), "invalid fields"),
(
lambda value: value["applied_proposal"].__setitem__("rationale", "post-approval drift"),
"exact guarded apply transition",
),
(
lambda value: value["approval_snapshot"].__setitem__("review_note", "different review"),
"does not match approved proposal",
),
(
lambda value: value["replay_receipt"]["proposal"].__setitem__("id", CLAIM_A),
"worker replay receipt",
),
(lambda value: value.__setitem__("sequence", 0), "positive integer"),
],
)
def test_material_validation_fails_closed_on_drift(mutation, message: str) -> None:
material = copy.deepcopy(_material())
mutation(material)
with pytest.raises(exporter.ExportError, match=message):
exporter.validate_material(material, expected_proposal_id=PROPOSAL_ID)
def test_json_parser_rejects_duplicate_and_nonfinite_fields() -> None:
with pytest.raises(exporter.ExportError, match="duplicate"):
exporter._parse_json_object('{"artifact":"a","artifact":"b"}', label="fixture")
with pytest.raises(exporter.ExportError, match="invalid numeric"):
exporter._parse_json_object('{"sequence":NaN}', label="fixture")
def test_private_writer_is_atomic_idempotent_and_refuses_alias_or_rerun_drift(tmp_path: Path) -> None:
material = exporter.validate_material(_material(), expected_proposal_id=PROPOSAL_ID)
protected = tmp_path / "worker-receipt.json"
protected.write_text("private worker receipt", encoding="utf-8")
protected.chmod(0o600)
output = tmp_path / "material.json"
written, disposition = exporter.write_private_material(output, material, protected_paths=(protected,))
assert written == output
assert disposition == "created"
assert stat.S_IMODE(output.stat().st_mode) == 0o600
assert output.stat().st_nlink == 1
assert json.loads(output.read_text(encoding="utf-8")) == material
_, rerun_disposition = exporter.write_private_material(output, material, protected_paths=(protected,))
assert rerun_disposition == "unchanged"
drifted = copy.deepcopy(material)
drifted["sequence"] = 2
with pytest.raises(exporter.ExportError, match="different replay material"):
exporter.write_private_material(output, drifted, protected_paths=(protected,))
with pytest.raises(exporter.ExportError, match="aliases a protected"):
exporter.write_private_material(protected, material, protected_paths=(protected,))
dangling = tmp_path / "dangling-output.json"
dangling.symlink_to(tmp_path / "missing-target.json")
with pytest.raises(exporter.ExportError, match="non-symlink"):
exporter.write_private_material(dangling, material, protected_paths=(protected,))
hardlink = tmp_path / "hardlink.json"
os.link(output, hardlink)
with pytest.raises(exporter.ExportError, match="hard-link"):
exporter.write_private_material(output, material, protected_paths=(protected,))
def test_private_receipt_loader_requires_exact_mode_and_rejects_symlinks(tmp_path: Path) -> None:
receipt = _material()["replay_receipt"]
receipt_path = tmp_path / "receipt.json"
receipt_path.write_text(json.dumps(receipt), encoding="utf-8")
receipt_path.chmod(0o600)
assert exporter._load_private_receipt(receipt_path) == receipt
receipt_path.chmod(0o640)
with pytest.raises(exporter.ExportError, match="exactly 0600"):
exporter._load_private_receipt(receipt_path)
receipt_path.chmod(0o600)
receipt_link = tmp_path / "receipt-link.json"
receipt_link.symlink_to(receipt_path)
with pytest.raises(exporter.ExportError, match="non-symlink"):
exporter._load_private_receipt(receipt_link)
def test_transition_fetch_uses_only_read_only_narrow_function(monkeypatch: pytest.MonkeyPatch) -> None:
calls: list[tuple[str, bool]] = []
rows = {key: _material()[key] for key in ("approved_proposal", "approval_snapshot", "applied_proposal")}
def fake_run_psql(args, sql: str, password: str, *, redact_output_on_error: bool = False) -> str:
assert args.role == "kb_apply"
assert password == "private-password"
calls.append((sql, redact_output_on_error))
return json.dumps(rows)
monkeypatch.setattr(exporter.apply_engine, "run_psql", fake_run_psql)
args = argparse.Namespace(container="disposable", db="teleo", host="127.0.0.1", role="kb_apply")
assert exporter.fetch_transition_rows(args, "private-password", PROPOSAL_ID) == rows
assert len(calls) == 1
sql, redacted = calls[0]
assert "transaction isolation level repeatable read read only" in sql
assert "kb_stage.export_applied_proposal_replay_rows" in sql
assert not any(keyword in sql.lower() for keyword in ("insert ", "update ", "delete ", "truncate "))
assert redacted is True

31
tests/test_hermes_leoclean_kb_bridge_source.py
View file

@ -1228,6 +1228,37 @@ def _load_module(path: Path):
return module
@pytest.mark.parametrize("local", (True, False))
def test_vps_bridge_psql_fails_closed_on_sql_errors(
local: bool, monkeypatch: pytest.MonkeyPatch
) -> None:
module = _load_module(BRIDGE_DIR / "kb_tool.py")
captured: dict[str, object] = {}
def fake_run(command, **kwargs):
captured["command"] = command
captured["kwargs"] = kwargs
return SimpleNamespace(returncode=3, stdout="", stderr='ERROR: column "missing" does not exist')
monkeypatch.setattr(module.subprocess, "run", fake_run)
args = SimpleNamespace(
local=local,
container="teleo-pg",
db="teleo",
ssh="teleo@example.invalid",
)
with pytest.raises(SystemExit, match=r"psql command failed \(3\)"):
module.psql_json(args, "select missing from claims;")
command = captured["command"]
if local:
assert command[-3:] == ["-q", "-v", "ON_ERROR_STOP=1"]
else:
assert command[-1].endswith("-At -q -v ON_ERROR_STOP=1")
assert captured["kwargs"]["input"] == "select missing from claims;"
@pytest.mark.parametrize(
("value", "minimum", "maximum"),
((True, 1, 2), ("1", 1, 2), (0, 1, 2), (3, 1, 2), (1, 2, 1)),

587
tests/test_kb_apply_prereqs.py
View file

@ -1,18 +1,37 @@
from __future__ import annotations
import json
import os
import shutil
import subprocess
import time
import uuid
from collections.abc import Iterator
from contextlib import contextmanager
from pathlib import Path
import pytest
from tests.docker_volume_lifecycle import (
POSTGRES_TMPFS_SPEC,
acquire_docker_volume_set_guard,
inspect_postgres_tmpfs_container,
release_docker_volume_set_guard,
remove_container_and_volumes,
)
REPO_ROOT = Path(__file__).resolve().parents[1]
MIGRATION = REPO_ROOT / "scripts" / "kb_apply_prereqs.sql"
POSTGRES_IMAGE = "postgres:16-alpine"
POSTGRES_IMAGE = os.environ.get("LEOCLEAN_RUNTIME_POSTGRES_IMAGE", "postgres:16-alpine")
DATABASE = "teleo_clone"
LEGACY_APPROVED_ID = "aaaaaaaa-0000-4000-8000-000000000001"
LEGACY_APPLIED_ID = "aaaaaaaa-0000-4000-8000-000000000002"
FRESH_PROPOSAL_ID = "aaaaaaaa-0000-4000-8000-000000000003"
TIMEZONE_PROPOSAL_ID = "aaaaaaaa-0000-4000-8000-000000000004"
REVIEWER_ID = "11111111-1111-1111-1111-111111111111"
STALE_REVIEWER_ID = "22222222-2222-2222-2222-222222222222"
LEGACY_REVIEWER_HANDLE = "m3ta"
CANONICAL_REVIEWER_HANDLE = "m3taversal"
BOOTSTRAP_SQL = r"""
create role kb_gate_owner nologin inherit;
@ -84,6 +103,42 @@ create table kb_stage.kb_proposal_approvals (
review_note text not null
);
insert into kb_stage.kb_proposals (
id, proposal_type, status, payload, reviewed_by_handle, reviewed_by_agent_id,
reviewed_at, review_note, applied_by_handle, applied_by_agent_id, applied_at, updated_at
) values
(
'aaaaaaaa-0000-4000-8000-000000000001', 'add_edge', 'approved',
'{"apply_payload":{"from_claim":"aaaaaaaa-1000-4000-8000-000000000001"}}'::jsonb,
'm3ta', '11111111-1111-1111-1111-111111111111', '2026-07-18T10:00:00Z',
'Exact still-approved legacy row.', null, null, null, '2026-07-18T10:00:01Z'
),
(
'aaaaaaaa-0000-4000-8000-000000000002', 'add_edge', 'applied',
'{"apply_payload":{"from_claim":"aaaaaaaa-1000-4000-8000-000000000002"}}'::jsonb,
'm3ta', '11111111-1111-1111-1111-111111111111', '2026-07-18T09:00:00Z',
'Applied before exact approved rows were retained.', 'kb-apply',
'44444444-4444-4444-4444-444444444444', '2026-07-18T09:01:00Z',
'2026-07-18T09:01:00Z'
);
insert into kb_stage.kb_proposal_approvals (
proposal_id, proposal_type, payload, reviewed_by_handle, reviewed_by_agent_id,
reviewed_by_db_role, reviewed_at, review_note
) values
(
'aaaaaaaa-0000-4000-8000-000000000001', 'add_edge',
'{"apply_payload":{"from_claim":"aaaaaaaa-1000-4000-8000-000000000001"}}'::jsonb,
'm3ta', '11111111-1111-1111-1111-111111111111', 'kb_review',
'2026-07-18T10:00:00Z', 'Exact still-approved legacy row.'
),
(
'aaaaaaaa-0000-4000-8000-000000000002', 'add_edge',
'{"apply_payload":{"from_claim":"aaaaaaaa-1000-4000-8000-000000000002"}}'::jsonb,
'm3ta', '11111111-1111-1111-1111-111111111111', 'kb_review',
'2026-07-18T09:00:00Z', 'Applied before exact approved rows were retained.'
);
create function kb_stage.approve_strict_proposal(uuid, text, text)
returns jsonb language sql security definer
as $function$ select '{}'::jsonb $function$;
@ -145,34 +200,40 @@ def _apply_migration(container: str, *, check: bool = True) -> subprocess.Comple
return _psql(container, MIGRATION.read_text(), check=check)
@pytest.fixture(scope="module")
def migrated_postgres() -> Iterator[str]:
@contextmanager
def _postgres_clone() -> Iterator[str]:
if shutil.which("docker") is None:
pytest.skip("Docker is required for the disposable PostgreSQL migration canary")
container = f"kb-apply-prereqs-{uuid.uuid4().hex[:12]}"
started = subprocess.run(
[
"docker",
"run",
"--rm",
"-d",
"--name",
container,
"-e",
"POSTGRES_PASSWORD=postgres",
"-e",
f"POSTGRES_DB={DATABASE}",
POSTGRES_IMAGE,
],
text=True,
capture_output=True,
check=False,
volume_guard = acquire_docker_volume_set_guard(
context="kb_apply_prereqs PostgreSQL fixture",
)
if started.returncode != 0:
pytest.fail(f"could not start disposable PostgreSQL: {started.stderr}")
container = f"kb-apply-prereqs-{uuid.uuid4().hex[:12]}"
try:
started = subprocess.run(
[
"docker",
"run",
"--pull=never",
"-d",
"--name",
container,
"--tmpfs",
POSTGRES_TMPFS_SPEC,
"-e",
"POSTGRES_PASSWORD=postgres",
"-e",
f"POSTGRES_DB={DATABASE}",
POSTGRES_IMAGE,
],
text=True,
capture_output=True,
check=False,
)
if started.returncode != 0:
pytest.fail(f"could not start disposable PostgreSQL: {started.stderr}")
inspect_postgres_tmpfs_container(container)
for _ in range(120):
logs = subprocess.run(
["docker", "logs", container],
@ -189,20 +250,162 @@ def migrated_postgres() -> Iterator[str]:
pytest.fail("disposable PostgreSQL did not become ready")
_psql(container, BOOTSTRAP_SQL)
_apply_migration(container)
yield container
finally:
subprocess.run(
["docker", "rm", "-f", container],
capture_output=True,
check=False,
)
try:
remove_container_and_volumes(container)
finally:
release_docker_volume_set_guard(volume_guard)
@pytest.fixture(scope="module")
def migrated_postgres() -> Iterator[str]:
with _postgres_clone() as container:
_apply_migration(container)
yield container
def _lines(completed: subprocess.CompletedProcess[str]) -> set[str]:
return {line for line in completed.stdout.splitlines() if line}
def test_legacy_only_reviewer_is_canonicalized_without_rewriting_history(
migrated_postgres: str,
) -> None:
identity_rows = _psql(
migrated_postgres,
f"""
select id::text || '|' || handle || '|' || kind
from public.agents
where handle in ('{LEGACY_REVIEWER_HANDLE}', '{CANONICAL_REVIEWER_HANDLE}')
order by handle;
""",
).stdout.splitlines()
assert identity_rows == [f"{REVIEWER_ID}|{CANONICAL_REVIEWER_HANDLE}|human"]
historical_rows = _psql(
migrated_postgres,
f"""
select 'proposal|' || id::text || '|' || reviewed_by_handle || '|' || reviewed_by_agent_id::text
from kb_stage.kb_proposals
where id in ('{LEGACY_APPROVED_ID}'::uuid, '{LEGACY_APPLIED_ID}'::uuid)
order by id;
select 'approval|' || proposal_id::text || '|' || reviewed_by_handle || '|' ||
reviewed_by_agent_id::text
from kb_stage.kb_proposal_approvals
where proposal_id in ('{LEGACY_APPROVED_ID}'::uuid, '{LEGACY_APPLIED_ID}'::uuid)
order by proposal_id;
""",
).stdout.splitlines()
assert historical_rows == [
f"proposal|{LEGACY_APPROVED_ID}|{LEGACY_REVIEWER_HANDLE}|{REVIEWER_ID}",
f"proposal|{LEGACY_APPLIED_ID}|{LEGACY_REVIEWER_HANDLE}|{REVIEWER_ID}",
f"approval|{LEGACY_APPROVED_ID}|{LEGACY_REVIEWER_HANDLE}|{REVIEWER_ID}",
f"approval|{LEGACY_APPLIED_ID}|{LEGACY_REVIEWER_HANDLE}|{REVIEWER_ID}",
]
def test_already_current_reviewer_migration_is_idempotent() -> None:
with _postgres_clone() as container:
_psql(
container,
f"update public.agents set handle = '{CANONICAL_REVIEWER_HANDLE}' "
f"where handle = '{LEGACY_REVIEWER_HANDLE}';",
)
_apply_migration(container)
state_sql = f"""
select principal.reviewed_by_handle || '|' || principal.reviewed_by_agent_id::text || '|' ||
principal.active::text || '|' || principal.created_at::text || '|' ||
(select count(*) from public.agents where handle = '{LEGACY_REVIEWER_HANDLE}')::text
from kb_stage.kb_review_principals principal
where principal.db_role = 'kb_review';
"""
first_state = _psql(container, state_sql).stdout.strip()
_apply_migration(container)
second_state = _psql(container, state_sql).stdout.strip()
expected_state = f"{CANONICAL_REVIEWER_HANDLE}|{REVIEWER_ID}|true|1970-01-01 00:00:00+00|0"
assert first_state == expected_state
assert second_state == first_state
def test_stale_reviewer_principal_is_repaired_without_resetting_created_at() -> None:
with _postgres_clone() as container:
_psql(
container,
f"""
insert into public.agents (id, handle, kind)
values ('{STALE_REVIEWER_ID}', 'stale-reviewer', 'human');
insert into kb_stage.kb_review_principals
(db_role, reviewed_by_handle, reviewed_by_agent_id, active, created_at)
values (
'kb_review', 'stale-reviewer', '{STALE_REVIEWER_ID}', false,
'2001-02-03T04:05:06Z'
);
""",
)
_apply_migration(container)
principal_state = _psql(
container,
"""
select reviewed_by_handle || '|' || reviewed_by_agent_id::text || '|' || active::text || '|' ||
created_at::text
from kb_stage.kb_review_principals
where db_role = 'kb_review';
""",
).stdout.strip()
expected_state = f"{CANONICAL_REVIEWER_HANDLE}|{REVIEWER_ID}|true|2001-02-03 04:05:06+00"
assert principal_state == expected_state
def test_migration_refuses_distinct_legacy_and_canonical_reviewers() -> None:
with _postgres_clone() as container:
_psql(
container,
f"""
insert into public.agents (id, handle, kind)
values ('{STALE_REVIEWER_ID}', '{CANONICAL_REVIEWER_HANDLE}', 'human');
""",
)
refused = _apply_migration(container, check=False)
identity_rows = _psql(
container,
f"""
select handle || '|' || id::text
from public.agents
where handle in ('{LEGACY_REVIEWER_HANDLE}', '{CANONICAL_REVIEWER_HANDLE}')
order by handle;
""",
).stdout.splitlines()
assert refused.returncode != 0
assert "both reviewer handles m3ta" in refused.stderr
assert "exist with different agent UUIDs" in refused.stderr
assert identity_rows == [
f"{LEGACY_REVIEWER_HANDLE}|{REVIEWER_ID}",
f"{CANONICAL_REVIEWER_HANDLE}|{STALE_REVIEWER_ID}",
]
def test_migration_refuses_to_fabricate_missing_human_reviewer() -> None:
with _postgres_clone() as container:
_psql(
container,
f"""
delete from kb_stage.kb_proposal_approvals;
delete from public.agents where handle = '{LEGACY_REVIEWER_HANDLE}';
""",
)
refused = _apply_migration(container, check=False)
remaining_agents = _psql(container, "select count(*) from public.agents;").stdout.strip()
assert refused.returncode != 0
assert "required human reviewer agent is missing" in refused.stderr
assert "handle m3ta or m3taversal" in refused.stderr
assert remaining_agents == "0"
def test_reviewer_principal_seed_timestamp_is_deterministic(migrated_postgres: str) -> None:
created_at = _psql(
migrated_postgres,
@ -212,6 +415,266 @@ def test_reviewer_principal_seed_timestamp_is_deterministic(migrated_postgres: s
assert created_at == "1970-01-01 00:00:00+00"
def test_upgrade_backfills_only_still_exact_approved_rows(migrated_postgres: str) -> None:
snapshot_state = _psql(
migrated_postgres,
f"""
select proposal_id::text || '|' || (approved_proposal_snapshot is not null)::text || '|' ||
coalesce(approved_proposal_snapshot->>'status', 'unavailable') || '|' ||
coalesce(approved_proposal_snapshot->>'reviewed_by_handle', 'unavailable')
from kb_stage.kb_proposal_approvals
where proposal_id in ('{LEGACY_APPROVED_ID}'::uuid, '{LEGACY_APPLIED_ID}'::uuid)
order by proposal_id;
""",
).stdout.splitlines()
assert snapshot_state == [
f"{LEGACY_APPROVED_ID}|true|approved|{LEGACY_REVIEWER_HANDLE}",
f"{LEGACY_APPLIED_ID}|false|unavailable|unavailable",
]
unavailable = _psql(
migrated_postgres,
f"""
set session authorization kb_apply;
select kb_stage.export_applied_proposal_replay_rows('{LEGACY_APPLIED_ID}'::uuid);
""",
check=False,
)
assert unavailable.returncode != 0
assert "exact applied transition unavailable" in unavailable.stderr
def test_new_approval_preserves_exact_post_approval_row_for_apply_export(migrated_postgres: str) -> None:
payload = '{"apply_payload":{"from_claim":"aaaaaaaa-1000-4000-8000-000000000003"}}'
_psql(
migrated_postgres,
f"""
insert into kb_stage.kb_proposals (id, proposal_type, status, payload, updated_at)
values ('{FRESH_PROPOSAL_ID}'::uuid, 'add_edge', 'pending_review', '{payload}'::jsonb,
'2026-07-18T11:00:00Z');
set session authorization kb_review;
select kb_stage.approve_strict_proposal(
'{FRESH_PROPOSAL_ID}'::uuid, 'add_edge', '{payload}'::jsonb, '{CANONICAL_REVIEWER_HANDLE}',
'Fresh exact approval snapshot.'
);
""",
)
approval_identity = _psql(
migrated_postgres,
f"""
select proposal.reviewed_by_handle || '|' || proposal.reviewed_by_agent_id::text || '|' ||
approval.reviewed_by_handle || '|' || approval.reviewed_by_agent_id::text || '|' ||
(approval.approved_proposal_snapshot->>'reviewed_by_handle') || '|' ||
(approval.approved_proposal_snapshot->>'reviewed_by_agent_id')
from kb_stage.kb_proposals proposal
join kb_stage.kb_proposal_approvals approval on approval.proposal_id = proposal.id
where proposal.id = '{FRESH_PROPOSAL_ID}'::uuid;
""",
).stdout.strip()
assert approval_identity == (
f"{CANONICAL_REVIEWER_HANDLE}|{REVIEWER_ID}|"
f"{CANONICAL_REVIEWER_HANDLE}|{REVIEWER_ID}|"
f"{CANONICAL_REVIEWER_HANDLE}|{REVIEWER_ID}"
)
approval_state = _psql(
migrated_postgres,
f"""
select jsonb_build_object(
'matches', approval.approved_proposal_snapshot = to_jsonb(proposal),
'status', approval.approved_proposal_snapshot->>'status',
'updated_at_matches', approval.approved_proposal_snapshot->'updated_at' = to_jsonb(proposal)->'updated_at'
)::text
from kb_stage.kb_proposal_approvals approval
join kb_stage.kb_proposals proposal on proposal.id = approval.proposal_id
where proposal.id = '{FRESH_PROPOSAL_ID}'::uuid;
""",
).stdout.strip()
assert approval_state == '{"status": "approved", "matches": true, "updated_at_matches": true}'
direct_approval_read = _psql(
migrated_postgres,
"set session authorization kb_apply; select * from kb_stage.kb_proposal_approvals;",
check=False,
)
assert direct_approval_read.returncode != 0
assert "permission denied for table kb_proposal_approvals" in direct_approval_read.stderr
premature_export = _psql(
migrated_postgres,
f"""
set session authorization kb_apply;
select kb_stage.export_applied_proposal_replay_rows('{FRESH_PROPOSAL_ID}'::uuid);
""",
check=False,
)
assert premature_export.returncode != 0
assert "exact applied transition unavailable" in premature_export.stderr
_psql(
migrated_postgres,
f"""
update kb_stage.kb_proposal_approvals
set approved_proposal_snapshot =
approved_proposal_snapshot || '{{"rationale":"tampered after approval"}}'::jsonb
where proposal_id = '{FRESH_PROPOSAL_ID}'::uuid;
""",
)
for guard_call in (
"kb_stage.assert_approved_proposal("
"proposal.id, proposal.proposal_type, proposal.payload, proposal.reviewed_by_handle, "
"proposal.reviewed_by_agent_id, proposal.reviewed_at, proposal.review_note)",
"kb_stage.finish_approved_proposal("
"proposal.id, proposal.proposal_type, proposal.payload, proposal.reviewed_by_handle, "
"proposal.reviewed_by_agent_id, proposal.reviewed_at, proposal.review_note, 'kb-apply')",
):
refused = _psql(
migrated_postgres,
f"""
set session authorization kb_apply;
select {guard_call}
from kb_stage.kb_proposals proposal where proposal.id = '{FRESH_PROPOSAL_ID}'::uuid;
""",
check=False,
)
assert refused.returncode != 0
assert "immutable approved snapshot" in refused.stderr
_psql(
migrated_postgres,
f"""
update kb_stage.kb_proposal_approvals approval
set approved_proposal_snapshot = to_jsonb(proposal)
from kb_stage.kb_proposals proposal
where approval.proposal_id = proposal.id
and proposal.id = '{FRESH_PROPOSAL_ID}'::uuid;
""",
)
exported = _psql(
migrated_postgres,
f"""
set session authorization kb_apply;
select kb_stage.assert_approved_proposal(
proposal.id, proposal.proposal_type, proposal.payload, proposal.reviewed_by_handle,
proposal.reviewed_by_agent_id, proposal.reviewed_at, proposal.review_note
)
from kb_stage.kb_proposals proposal where proposal.id = '{FRESH_PROPOSAL_ID}'::uuid;
select kb_stage.finish_approved_proposal(
proposal.id, proposal.proposal_type, proposal.payload, proposal.reviewed_by_handle,
proposal.reviewed_by_agent_id, proposal.reviewed_at, proposal.review_note, 'kb-apply'
)
from kb_stage.kb_proposals proposal where proposal.id = '{FRESH_PROPOSAL_ID}'::uuid;
select jsonb_object_keys(kb_stage.export_applied_proposal_replay_rows('{FRESH_PROPOSAL_ID}'::uuid))
order by 1;
""",
).stdout.splitlines()
assert exported[-3:] == ["applied_proposal", "approval_snapshot", "approved_proposal"]
def test_gate_functions_are_timezone_independent_across_connections(
migrated_postgres: str,
) -> None:
payload = '{"apply_payload":{"from_claim":"aaaaaaaa-1000-4000-8000-000000000004"}}'
_psql(
migrated_postgres,
f"""
insert into kb_stage.kb_proposals (id, proposal_type, status, payload, updated_at)
values ('{TIMEZONE_PROPOSAL_ID}'::uuid, 'add_edge', 'pending_review', '{payload}'::jsonb,
'2026-07-18T12:00:00Z');
set timezone = 'America/Los_Angeles';
set session authorization kb_review;
select kb_stage.approve_strict_proposal(
'{TIMEZONE_PROPOSAL_ID}'::uuid, 'add_edge', '{payload}'::jsonb, '{CANONICAL_REVIEWER_HANDLE}',
'Cross-timezone exact approval snapshot.'
);
""",
)
asserted_and_finished = _psql(
migrated_postgres,
f"""
set timezone = 'Asia/Tokyo';
set session authorization kb_apply;
select kb_stage.assert_approved_proposal(
proposal.id, proposal.proposal_type, proposal.payload, proposal.reviewed_by_handle,
proposal.reviewed_by_agent_id, proposal.reviewed_at, proposal.review_note
)
from kb_stage.kb_proposals proposal where proposal.id = '{TIMEZONE_PROPOSAL_ID}'::uuid;
select kb_stage.finish_approved_proposal(
proposal.id, proposal.proposal_type, proposal.payload, proposal.reviewed_by_handle,
proposal.reviewed_by_agent_id, proposal.reviewed_at, proposal.review_note, 'kb-apply'
)
from kb_stage.kb_proposals proposal where proposal.id = '{TIMEZONE_PROPOSAL_ID}'::uuid;
""",
)
assert '"status": "applied"' in asserted_and_finished.stdout
exported = json.loads(
_psql(
migrated_postgres,
f"""
set timezone = 'Pacific/Auckland';
set session authorization kb_apply;
select kb_stage.export_applied_proposal_replay_rows('{TIMEZONE_PROPOSAL_ID}'::uuid)::text;
""",
).stdout.splitlines()[-1]
)
approved = exported["approved_proposal"]
applied = exported["applied_proposal"]
approval = exported["approval_snapshot"]
assert approved["reviewed_at"].endswith("+00:00")
assert applied["reviewed_at"] == approved["reviewed_at"]
assert approval["reviewed_at"] == approved["reviewed_at"]
assert applied["status"] == "applied"
def test_transition_export_rollback_preserves_evidence_and_reinstalls_cleanly() -> None:
with _postgres_clone() as container:
_apply_migration(container)
before = _psql(
container,
f"""
select approved_proposal_snapshot::text
from kb_stage.kb_proposal_approvals
where proposal_id = '{LEGACY_APPROVED_ID}'::uuid;
""",
).stdout.strip()
_psql(
container,
"""
revoke all on function kb_stage.export_applied_proposal_replay_rows(uuid)
from public, kb_gate_owner, kb_apply, kb_review;
drop function kb_stage.export_applied_proposal_replay_rows(uuid);
""",
)
rolled_back = _psql(
container,
f"""
select pg_catalog.to_regprocedure(
'kb_stage.export_applied_proposal_replay_rows(uuid)'
) is null;
select approved_proposal_snapshot::text
from kb_stage.kb_proposal_approvals
where proposal_id = '{LEGACY_APPROVED_ID}'::uuid;
""",
).stdout.splitlines()
assert rolled_back == ["t", before]
_apply_migration(container)
restored = _psql(
container,
f"""
select pg_catalog.to_regprocedure(
'kb_stage.export_applied_proposal_replay_rows(uuid)'
) is not null;
select approved_proposal_snapshot::text
from kb_stage.kb_proposal_approvals
where proposal_id = '{LEGACY_APPROVED_ID}'::uuid;
""",
).stdout.splitlines()
assert restored == ["t", before]
def _catalog_snapshot(container: str) -> str:
return _psql(
container,
@ -239,9 +702,16 @@ select 'function|' || procedure.proname || '(' || pg_catalog.oidvectortypes(proc
join pg_catalog.pg_roles owner_role on owner_role.oid = procedure.proowner
where namespace.nspname = 'kb_stage'
and procedure.proname in (
'approve_strict_proposal', 'assert_approved_proposal', 'finish_approved_proposal'
'approve_strict_proposal', 'assert_approved_proposal', 'finish_approved_proposal',
'export_applied_proposal_replay_rows'
)
order by procedure.proname, procedure.proargtypes::text;
select 'column|' || attribute.attname || '|' || attribute.atttypid::regtype::text || '|' ||
attribute.attnotnull
from pg_catalog.pg_attribute attribute
where attribute.attrelid = 'kb_stage.kb_proposal_approvals'::regclass
and attribute.attname = 'approved_proposal_snapshot'
and not attribute.attisdropped;
select 'table_acl|' || namespace.nspname || '|' || relation.relname || '|' ||
case when acl.grantee = 0 then 'PUBLIC' else grantee_role.rolname end || '|' ||
pg_catalog.upper(acl.privilege_type) || '|' || acl.is_grantable
@ -272,7 +742,8 @@ select 'function_acl|' || procedure.proname || '(' ||
) acl
where namespace.nspname = 'kb_stage'
and procedure.proname in (
'approve_strict_proposal', 'assert_approved_proposal', 'finish_approved_proposal'
'approve_strict_proposal', 'assert_approved_proposal', 'finish_approved_proposal',
'export_applied_proposal_replay_rows'
)
and acl.grantee <> procedure.proowner
order by procedure.proname, procedure.proargtypes::text, acl.grantee;
@ -337,23 +808,32 @@ select protected_role.rolname || '|' || protected_schema.nspname || '|' ||
migrated_postgres,
"""
select procedure.proname || '(' || pg_catalog.oidvectortypes(procedure.proargtypes) || ')|' ||
owner_role.rolname || '|' || procedure.prosecdef || '|' || procedure.proconfig::text
owner_role.rolname || '|' || procedure.prosecdef || '|' ||
(
pg_catalog.cardinality(procedure.proconfig) = 2
and procedure.proconfig @> array[
'search_path=pg_catalog, pg_temp'::text,
'TimeZone=UTC'::text
]
)
from pg_catalog.pg_proc procedure
join pg_catalog.pg_namespace namespace on namespace.oid = procedure.pronamespace
join pg_catalog.pg_roles owner_role on owner_role.oid = procedure.proowner
where namespace.nspname = 'kb_stage'
and procedure.proname in (
'approve_strict_proposal', 'assert_approved_proposal', 'finish_approved_proposal'
'approve_strict_proposal', 'assert_approved_proposal', 'finish_approved_proposal',
'export_applied_proposal_replay_rows'
);
""",
)
)
assert function_rows == {
'approve_strict_proposal(uuid, text, jsonb, text, text)|kb_gate_owner|true|{"search_path=pg_catalog, pg_temp"}',
"approve_strict_proposal(uuid, text, jsonb, text, text)|kb_gate_owner|true|true",
"assert_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text)|"
'kb_gate_owner|true|{"search_path=pg_catalog, pg_temp"}',
"kb_gate_owner|true|true",
"finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text, text)|"
'kb_gate_owner|true|{"search_path=pg_catalog, pg_temp"}',
"kb_gate_owner|true|true",
"export_applied_proposal_replay_rows(uuid)|kb_gate_owner|true|true",
}
assert (
_psql(
@ -431,6 +911,7 @@ select count(*)
"kb_apply|EXECUTE|false",
"function_acl|finish_approved_proposal(uuid, text, jsonb, text, uuid, timestamp with time zone, text, text)|"
"kb_apply|EXECUTE|false",
"function_acl|export_applied_proposal_replay_rows(uuid)|kb_apply|EXECUTE|false",
}
@ -470,6 +951,38 @@ def test_migration_fails_closed_on_unexpected_table_grantee(migrated_postgres: s
_psql(migrated_postgres, "revoke all on kb_stage.kb_proposal_approvals from rogue_grantee;")
def test_migration_refuses_rogue_canonical_insert_before_install_and_normalizes_after_removal() -> None:
with _postgres_clone() as container:
_psql(container, "grant usage on schema public to rogue_grantee;")
for grant_sql, revoke_sql in (
(
"grant insert on public.claims to rogue_grantee;",
"revoke insert on public.claims from rogue_grantee;",
),
(
"grant insert (id) on public.claims to rogue_grantee;",
"revoke insert (id) on public.claims from rogue_grantee;",
),
):
_psql(container, grant_sql)
refused = _apply_migration(container, check=False)
assert refused.returncode != 0
assert "unexpected protected table grantee" in refused.stderr
assert _psql(container, "select count(*) from public.claims;").stdout.strip() == "0"
_psql(container, revoke_sql)
_apply_migration(container)
rogue_insert = _psql(
container,
"set session authorization rogue_grantee; "
"insert into public.claims (id) values ('99999999-9999-4999-8999-999999999999');",
check=False,
)
assert rogue_insert.returncode != 0
assert "permission denied for table claims" in rogue_insert.stderr
assert _psql(container, "select count(*) from public.claims;").stdout.strip() == "0"
def test_migration_fails_closed_on_unexpected_column_grantee(migrated_postgres: str) -> None:
_psql(
migrated_postgres,

224
tests/test_kb_apply_replay_receipt.py
View file

@ -70,8 +70,154 @@ def _edge_row() -> dict:
}
def _v3_bundle_and_rows() -> tuple[dict, dict]:
agent_id = "11111111-1111-4111-8111-111111111111"
evidence_id = "eeeeeeee-eeee-4eee-8eee-eeeeeeeeeeee"
assessment_id = "ffffffff-ffff-4fff-8fff-ffffffffffff"
evidence = {
"id": evidence_id,
"claim_id": CLAIM_A,
"source_id": SOURCE_A,
"polarity": "supports",
"excerpt": "The primary record supports the bounded claim.",
"locator_json": {"section": "results", "paragraph": 3},
"source_content_hash": "a" * 64,
"rationale": "The exact locator directly supports the proposition.",
"created_by": agent_id,
}
payload = {
"contract_version": 3,
"agent_id": agent_id,
"claims": [
{
"id": CLAIM_A,
"owner_agent_id": None,
"type": "empirical",
"proposition": "The V3 apply path retains exact provenance.",
"body_markdown": "This bounded claim is supported by one content-addressed source.",
"scope": "Disposable V3 writer verification.",
"access_scope": "collective_internal",
"status": "active",
"revision_criteria": "Revise if the cited bytes or locator cannot be reproduced.",
"revision_kind": "original",
"supersedes_id": None,
"tags": ["v3", "replay"],
"created_by": agent_id,
}
],
"sources": [
{
"id": SOURCE_A,
"source_type": "paper",
"canonical_url": "https://example.test/v3-source",
"storage_uri": None,
"excerpt": "Content-addressed primary record.",
"content_hash": "a" * 64,
"access_scope": "collective_internal",
"ingestion_origin": "operator_upload",
"provenance_status": "verified",
"source_class": "primary_record",
"auto_promotion_policy": "human_required",
"captured_at": "2026-07-18T10:00:00+00:00",
"created_by": agent_id,
}
],
"evidence": [evidence],
"edges": [],
"assessments": [
{
"id": assessment_id,
"claim_id": CLAIM_A,
"support_tier": "strong",
"challenge_tier": "none",
"overall_state": "support_dominant",
"rationale": "The exact primary record supports the bounded claim.",
"evidence_set_hash": ap._v3_evidence_set_hash(CLAIM_A, [evidence]),
"supersedes_assessment_id": None,
}
],
"reasoning_tools": [],
}
rows = {
"claims": [
{
"id": CLAIM_A,
"owner_agent_id": None,
"type": "empirical",
"text": payload["claims"][0]["proposition"],
"proposition": payload["claims"][0]["proposition"],
"body_markdown": payload["claims"][0]["body_markdown"],
"scope": payload["claims"][0]["scope"],
"access_scope": "collective_internal",
"status": "active",
"admission_state": "admitted",
"revision_criteria": payload["claims"][0]["revision_criteria"],
"revision_kind": "original",
"supersedes_id": None,
"confidence": None,
"tags": ["v3", "replay"],
"created_by": agent_id,
"superseded_by": None,
"accepted_by_proposal_id": PROPOSAL_ID,
"created_at": "2026-07-18T10:01:00+00:00",
"updated_at": "2026-07-18T10:01:00+00:00",
}
],
"sources": [
{
"id": SOURCE_A,
"source_type": "paper",
"url": payload["sources"][0]["canonical_url"],
"storage_path": None,
"excerpt": payload["sources"][0]["excerpt"],
"hash": "a" * 64,
"created_by": agent_id,
"captured_at": payload["sources"][0]["captured_at"],
"canonical_url": payload["sources"][0]["canonical_url"],
"storage_uri": None,
"content_hash": "a" * 64,
"access_scope": "collective_internal",
"ingestion_origin": "operator_upload",
"provenance_status": "verified",
"source_class": "primary_record",
"auto_promotion_policy": "human_required",
"accepted_by_proposal_id": PROPOSAL_ID,
"created_at": "2026-07-18T10:01:00+00:00",
}
],
"claim_evidence": [
{
"id": evidence_id,
"claim_id": CLAIM_A,
"source_id": SOURCE_A,
"role": "grounds",
"weight": None,
"created_by": agent_id,
"excerpt": evidence["excerpt"],
"locator_json": evidence["locator_json"],
"source_content_hash": "a" * 64,
"rationale": evidence["rationale"],
"polarity": "supports",
"accepted_from_proposal_id": PROPOSAL_ID,
"created_at": "2026-07-18T10:01:00+00:00",
}
],
"claim_edges": [],
"claim_evidence_assessments": [
{
**payload["assessments"][0],
"accepted_by_proposal_id": PROPOSAL_ID,
"created_at": "2026-07-18T10:01:00+00:00",
}
],
"reasoning_tools": [],
}
return payload, rows
def test_add_edge_postflight_query_selects_exact_semantic_key() -> None:
sql = receipt.build_postflight_sql(_edge_proposal())
assert sql.startswith("set timezone = 'UTC';")
assert "public.claim_edges" in sql
assert CLAIM_A in sql
assert CLAIM_B in sql
@ -144,6 +290,42 @@ def test_receipt_hash_is_recoverable_and_independent_of_export_time() -> None:
assert first["exported_at_utc"] != second["exported_at_utc"]
def test_fractional_timestamp_spellings_normalize_before_semantic_receipt_comparison() -> None:
payload, rows = _v3_bundle_and_rows()
payload["sources"][0]["captured_at"] = "2026-07-18T12:00:00.123400+02:00"
rows["sources"][0]["captured_at"] = "2026-07-18T10:00:00.1234Z"
proposal = _applied("approve_claim", payload)
proposal["reviewed_at"] = "2026-07-18T12:01:00.5+02:00"
proposal["applied_at"] = "2026-07-18T05:01:01.000001-05:00"
data = _build_receipt(
proposal,
rows,
exported_at_utc="2026-07-18T12:02:00.25+02:00",
)
assert data["proposal"]["reviewed_at"] == "2026-07-18T10:01:00.500000Z"
assert data["proposal"]["applied_at"] == "2026-07-18T10:01:01.000001Z"
assert data["canonical_rows"]["sources"][0]["captured_at"] == "2026-07-18T10:00:00.123400Z"
assert data["exported_at_utc"] == "2026-07-18T10:02:00.250000Z"
assert receipt.validate_replay_receipt(data)["replay_ready"] is True
def test_legacy_uuid_spellings_canonicalize_without_postcommit_receipt_loss() -> None:
proposal = _edge_proposal()
proposal["payload"]["apply_payload"]["from_claim"] = CLAIM_A.upper()
proposal["payload"]["apply_payload"]["to_claim"] = "{" + CLAIM_B + "}"
row = _edge_row()
row["from_claim"] = CLAIM_A
row["to_claim"] = CLAIM_B
data = _build_receipt(proposal, {"claim_edges": [row]})
assert data["canonical_rows"]["claim_edges"][0]["from_claim"] == CLAIM_A
assert data["canonical_rows"]["claim_edges"][0]["to_claim"] == CLAIM_B
assert receipt.validate_replay_receipt(data)["replay_ready"] is True
def test_attach_evidence_receipt_matches_every_payload_controlled_field() -> None:
proposal = _applied(
"attach_evidence",
@ -271,6 +453,39 @@ def test_approve_claim_receipt_matches_all_supported_canonical_row_types() -> No
}
def test_v3_approve_claim_receipt_binds_every_admitted_row_family() -> None:
payload, rows = _v3_bundle_and_rows()
proposal = _applied("approve_claim", payload)
sql = receipt.build_postflight_sql(proposal)
for table in (
"public.claims",
"public.sources",
"public.claim_evidence",
"public.claim_edges",
"public.claim_evidence_assessments",
):
assert table in sql
data = _build_receipt(proposal, rows)
assert data["expected_row_counts"] == {
"claims": 1,
"sources": 1,
"claim_evidence": 1,
"claim_edges": 0,
"claim_evidence_assessments": 1,
"reasoning_tools": 0,
}
assert data["checks"]["semantic_rows_match_strict_payload"] is True
def test_v3_approve_claim_receipt_rejects_assessment_drift() -> None:
payload, rows = _v3_bundle_and_rows()
rows["claim_evidence_assessments"][0]["evidence_set_hash"] = "b" * 64
with pytest.raises(ValueError, match="semantic mismatch"):
_build_receipt(_applied("approve_claim", payload), rows)
def test_revise_strategy_receipt_requires_the_fresh_active_rows() -> None:
agent_id = "11111111-1111-4111-8111-111111111111"
payload = {
@ -337,6 +552,14 @@ def test_receipt_refuses_legacy_applied_proposal_without_strict_payload() -> Non
receipt.build_postflight_sql(proposal)
def test_receipt_refuses_explicit_null_legacy_contract_version() -> None:
proposal = _edge_proposal()
proposal["payload"]["apply_payload"]["contract_version"] = None
with pytest.raises(ValueError, match="must omit contract_version"):
receipt.build_postflight_sql(proposal)
def test_private_receipt_is_atomically_written_mode_0600(tmp_path: Path) -> None:
data = _build_receipt(_edge_proposal(), {"claim_edges": [_edge_row()]})
output = tmp_path / "private" / "receipt.json"
@ -379,6 +602,7 @@ def test_apply_capture_writes_receipt_without_printing_canonical_rows(
assert data["replay_ready"] is True
assert data["apply_engine"]["source"] == "reconstructed_current_engine"
assert len(seen) == 1
assert seen[0].startswith("set timezone = 'UTC';")
assert "public.claim_edges" in seen[0]
args.receipt_out = str(tmp_path / "exact-executed.json")

34
tests/test_kb_proposal_review_packet.py
View file

@ -1,13 +1,16 @@
"""Tests for scripts/kb_proposal_review_packet.py."""
from pathlib import Path
import copy
import sys
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(REPO_ROOT / "scripts"))
import kb_proposal_review_packet as rp # noqa: E402
from tests import test_compile_kb_source_packet as source_compiler_fixture # noqa: E402
def test_approved_freeform_packet_is_not_worker_applyable():
proposal = {
@ -73,3 +76,32 @@ def test_pending_packet_needs_human_review():
assert packet["review_state"] == "needs_human_review"
assert packet["worker_applyable"] is False
assert "approve, reject, or request edits" in packet["next_admin_action"]
def test_v3_source_packet_exposes_exact_rows_normalization_and_hashes(tmp_path: Path):
bundle = source_compiler_fixture._compile_v3(tmp_path / "source")
proposal = copy.deepcopy(bundle["strict_child_proposal"])
packet = rp.classify_proposal(proposal)
apply_payload = proposal["payload"]["apply_payload"]
assert packet["review_state"] == "needs_human_review"
assert packet["payload_summary"]["contract_version"] == 3
assert packet["payload_summary"]["strict_row_counts"] == {
"assessments": 1,
"claims": 1,
"edges": 0,
"evidence": 1,
"reasoning_tools": 0,
"sources": 1,
}
assert packet["strict_review_material"]["rows"]["claims"] == apply_payload["claims"]
assert packet["strict_review_material"]["rows"]["sources"] == apply_payload["sources"]
assert packet["strict_review_material"]["rows"]["evidence"] == apply_payload["evidence"]
assert packet["strict_review_material"]["rows"]["assessments"] == apply_payload["assessments"]
assert (
packet["strict_review_material"]["normalization_manifest"]
== apply_payload["normalization_manifest"]
)
assert packet["strict_review_material"]["apply_payload_sha256"] == rp.canonical_sha256(apply_payload)
assert packet["strict_review_material"]["proposal_payload_sha256"] == rp.canonical_sha256(proposal["payload"])

7
tests/test_proposal_apply_lifecycle.py
View file

@ -416,6 +416,13 @@ def test_fresh_preflight_requires_every_owned_target_to_be_absent_and_is_determi
lifecycle.build_fresh_preflight(proposal, incomplete)
def test_compensating_rollback_refuses_immutable_v3_admissions() -> None:
proposal = _proposal_before()
proposal["payload"]["apply_payload"]["contract_version"] = 3
with pytest.raises(ValueError, match="V3 admitted rows are immutable"):
lifecycle.build_fresh_preflight(proposal, _empty_targets())
def test_fresh_owned_apply_consumes_timestamped_preflight_and_never_adopts_conflicts():
proposal, _approval, preflight, _receipt, _rollback_sql = _materials()

780
tests/test_run_leo_integrated_learning_lifecycle.py
View file

@ -1,9 +1,12 @@
from __future__ import annotations
import hashlib
import json
import os
import shutil
import subprocess
import sys
import uuid
from pathlib import Path
import pytest
@ -45,6 +48,30 @@ def _docker_available() -> bool:
return result.returncode == 0
def _commit_test_repository(path: Path, files: dict[str, str]) -> None:
subprocess.run(["git", "init", "-q", str(path)], check=True)
for relative_path, contents in files.items():
target = path / relative_path
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(contents, encoding="utf-8")
subprocess.run(["git", "-C", str(path), "add", "."], check=True)
subprocess.run(
[
"git",
"-C",
str(path),
"-c",
"user.name=Lifecycle Test",
"-c",
"user.email=lifecycle-test@example.invalid",
"commit",
"-qm",
"test engine",
],
check=True,
)
def test_source_packet_recompiles_to_retained_expectation(tmp_path: Path) -> None:
packet = _write_source_packet(tmp_path)
@ -67,6 +94,34 @@ def test_source_packet_refuses_retained_packet_drift(tmp_path: Path) -> None:
lifecycle.compile_source_packet(packet)
def test_execution_identity_requires_exact_committed_dependency_bytes(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
repo = tmp_path / "repo"
repo.mkdir()
_commit_test_repository(repo, {"scripts/engine.py": "print('committed')\n"})
engine = repo / "scripts" / "engine.py"
monkeypatch.setattr(lifecycle, "REPO_ROOT", repo)
monkeypatch.setattr(lifecycle, "ENGINE_INPUT_PATHS", {"engine.runner": engine})
identity = lifecycle.execution_identity()
assert identity["engine_paths_match_head"] is True
assert identity["engine_relpaths"] == ["scripts/engine.py"]
assert identity["engine_head_sha256"]["engine.runner"] == hashlib.sha256(engine.read_bytes()).hexdigest()
engine.write_text("print('dirty')\n", encoding="utf-8")
with pytest.raises(lifecycle.LifecycleError, match="does not match"):
lifecycle.execution_identity()
def test_engine_dependency_closure_includes_transitive_imports() -> None:
assert {
"engine.proposal_normalizer",
"engine.rich_proposal_plan",
"engine.clone_checkpoint",
} <= lifecycle.ENGINE_INPUT_PATHS.keys()
def test_exact_identities_adds_observed_evidence_ids(tmp_path: Path) -> None:
packet = _write_source_packet(tmp_path)
compiled, _expected, _paths = lifecycle.compile_source_packet(packet)
@ -106,6 +161,598 @@ def test_lifecycle_requires_explicit_approve_action(tmp_path: Path) -> None:
)
def _application_role(*, complete_settings: bool = True) -> dict:
role = {
"name": "kb_apply",
"can_login": True,
"superuser": False,
"create_db": False,
"create_role": False,
"inherit": False,
"replication": False,
"bypass_rls": False,
"connection_limit": -1,
"default_transaction_read_only": None,
}
if complete_settings:
role["settings"] = []
role["database_settings"] = []
return role
def _minimal_loaded_manifest(*, complete_settings: bool = True) -> dict:
return {
"singleton": {
"identity": {"database": lifecycle.DATABASE, "transaction_read_only": "on"},
"application_roles": {"items": [_application_role(complete_settings=complete_settings)]},
"role_memberships": {"items": []},
"object_acl": {"items": []},
},
"tables": {"public.fixture": {"row_count": 0, "rowset_md5": "0" * 32}},
"performance": {},
}
def _write_exporter_receipt(
path: Path,
dump: Path,
manifest_path: Path,
loaded_manifest: dict,
*,
source: dict[str, str] | None = None,
) -> dict:
source = source or {
"ssh_target": "disposable-test",
"container": "teleo-test-source",
"database": lifecycle.DATABASE,
"service": "disposable-postgres-test",
}
snapshot = {
"captured_at_utc": "2026-07-18T00:00:00+00:00",
"exported_snapshot_id": "00000003-00000001-1",
"txid_snapshot": "1:1:",
"wal_lsn": "0/1",
"system_identifier": "123456789",
}
authorization_ref = "codex-delegation:disposable-genesis-test"
manifest_sha256 = lifecycle.sha256_file(manifest_path)
dump_sha256 = lifecycle.sha256_file(dump)
manifest_sql_sha256 = lifecycle.sha256_file(lifecycle.PARITY_SQL_PATH)
source_context_sha256 = hashlib.sha256(b"disposable-source-context").hexdigest()
binding = lifecycle.snapshot_capture.build_provenance_binding(
run_id="disposable-genesis-test",
authorization_ref=authorization_ref,
source=source,
snapshot=snapshot,
dump_sha256=dump_sha256,
manifest_sql_sha256=manifest_sql_sha256,
source_manifest_sha256=manifest_sha256,
source_context_sha256=source_context_sha256,
)
service_state = {
"ActiveState": "active",
"SubState": "running",
"MainPID": "123",
"NRestarts": "0",
"ActiveEnterTimestamp": "2026-07-18T00:00:00+00:00",
}
receipt = {
"artifact": lifecycle.TRUSTED_EXPORTER_ARTIFACT,
"schema": lifecycle.snapshot_capture.SOURCE_SNAPSHOT_RECEIPT_SCHEMA,
"receipt_version": 2,
"status": "pass",
"run_id": "disposable-genesis-test",
"capture_authorization_ref": authorization_ref,
"source": source,
"snapshot_exported": True,
"snapshot": snapshot,
"dump": {"path": str(dump), "bytes": dump.stat().st_size, "sha256": dump_sha256},
"manifest": {
"path": str(manifest_path),
"sha256": manifest_sha256,
"manifest_sql_sha256": manifest_sql_sha256,
"table_count": len(loaded_manifest["tables"]),
"total_rows": sum(int(row["row_count"]) for row in loaded_manifest["tables"].values()),
"application_roles": loaded_manifest["singleton"]["application_roles"]["items"],
},
"source_context": {"path": "source-context.txt", "sha256": source_context_sha256},
"source_service": {"before": service_state, "after": service_state, "unchanged": True},
"provenance_binding": binding,
"service_unchanged": True,
"production_db_mutated": False,
"telegram_send_attempted": False,
}
path.write_text(json.dumps(receipt, indent=2, sort_keys=True) + "\n", encoding="utf-8")
path.chmod(0o600)
return receipt
def _trust_test_exporter_receipt(
monkeypatch: pytest.MonkeyPatch,
path: Path,
receipt: dict,
) -> None:
registry = dict(lifecycle.TRUSTED_EXPORTER_RECEIPTS)
registry[lifecycle.sha256_file(path)] = {
"run_id": receipt["run_id"],
"capture_authorization_ref": receipt["capture_authorization_ref"],
"source": receipt["source"],
"dump_sha256": receipt["dump"]["sha256"],
"manifest_sha256": receipt["manifest"]["sha256"],
"manifest_sql_sha256": receipt["manifest"]["manifest_sql_sha256"],
"provenance_binding_sha256": receipt["provenance_binding"]["sha256"],
}
monkeypatch.setattr(lifecycle, "TRUSTED_EXPORTER_RECEIPTS", registry)
def _genesis_cli(tmp_path: Path) -> list[str]:
return [
"--source-packet",
str(tmp_path / "packet"),
"--output",
str(tmp_path / "lifecycle-receipt.json"),
"--review-action",
"approve",
"--genesis-dump",
str(tmp_path / "genesis.dump"),
"--genesis-manifest",
str(tmp_path / "manifest.jsonl"),
"--genesis-exporter-receipt",
str(tmp_path / "exporter-receipt.json"),
]
def test_genesis_cli_requires_complete_exporter_receipt_set(tmp_path: Path) -> None:
with pytest.raises(SystemExit):
lifecycle.parse_args(_genesis_cli(tmp_path)[:-2])
def test_genesis_cli_rejects_caller_only_hash_flags(tmp_path: Path) -> None:
with pytest.raises(SystemExit):
lifecycle.parse_args([*_genesis_cli(tmp_path), "--genesis-dump-sha256", "a" * 64])
def test_genesis_cli_rejects_moving_image_tag(tmp_path: Path) -> None:
with pytest.raises(SystemExit):
lifecycle.parse_args([*_genesis_cli(tmp_path), "--image", "postgres:16-alpine"])
def test_exporter_receipt_dump_drift_fails_before_docker(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
dump = tmp_path / "genesis.dump"
manifest = tmp_path / "manifest.jsonl"
exporter_receipt = tmp_path / "exporter-receipt.json"
dump.write_bytes(b"PGDMP-real-enough-for-signature-validation")
manifest.write_text("{}\n", encoding="utf-8")
loaded_manifest = _minimal_loaded_manifest()
receipt = _write_exporter_receipt(exporter_receipt, dump, manifest, loaded_manifest)
_trust_test_exporter_receipt(monkeypatch, exporter_receipt, receipt)
dump.write_bytes(dump.read_bytes() + b"drift")
monkeypatch.setattr(lifecycle, "load_manifest", lambda _path: loaded_manifest)
with pytest.raises(lifecycle.LifecycleError, match="dump_hash_from_exporter_receipt"):
lifecycle.validate_genesis_inputs(
dump,
manifest,
exporter_receipt,
lifecycle.DEFAULT_GENESIS_IMAGE,
)
def test_exporter_receipt_rebound_source_identity_fails_closed(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
dump = tmp_path / "genesis.dump"
manifest = tmp_path / "manifest.jsonl"
exporter_receipt = tmp_path / "exporter-receipt.json"
dump.write_bytes(b"PGDMP-real-enough-for-signature-validation")
manifest.write_text("{}\n", encoding="utf-8")
loaded_manifest = _minimal_loaded_manifest()
receipt = _write_exporter_receipt(exporter_receipt, dump, manifest, loaded_manifest)
_trust_test_exporter_receipt(monkeypatch, exporter_receipt, receipt)
receipt["source"]["database"] = "attacker_rebound"
receipt["provenance_binding"] = lifecycle.snapshot_capture.build_provenance_binding(
run_id=receipt["run_id"],
authorization_ref=receipt["capture_authorization_ref"],
source=receipt["source"],
snapshot=receipt["snapshot"],
dump_sha256=receipt["dump"]["sha256"],
manifest_sql_sha256=receipt["manifest"]["manifest_sql_sha256"],
source_manifest_sha256=receipt["manifest"]["sha256"],
source_context_sha256=receipt["source_context"]["sha256"],
)
exporter_receipt.write_text(json.dumps(receipt), encoding="utf-8")
exporter_receipt.chmod(0o600)
monkeypatch.setattr(lifecycle, "load_manifest", lambda _path: loaded_manifest)
with pytest.raises(lifecycle.LifecycleError, match="trusted_exporter_receipt_hash"):
lifecycle.validate_genesis_inputs(
dump,
manifest,
exporter_receipt,
lifecycle.DEFAULT_GENESIS_IMAGE,
)
def test_legacy_manifest_role_settings_projection_is_narrow_and_nonmutating() -> None:
manifest = {
"singleton": {
"application_roles": {
"items": [
{
"name": "kb_observatory_read",
"default_transaction_read_only": "on",
},
{
"name": "kb_apply",
"default_transaction_read_only": None,
},
]
}
}
}
projected = lifecycle.manifest_for_role_replay(manifest)
assert "settings" not in manifest["singleton"]["application_roles"]["items"][0]
assert projected["singleton"]["application_roles"]["items"][0]["settings"] == [
{"name": "default_transaction_read_only", "value": "on"}
]
assert projected["singleton"]["application_roles"]["items"][0]["database_settings"] == []
assert projected["singleton"]["application_roles"]["items"][1]["settings"] == []
assert lifecycle.role_settings_capture_complete(manifest) is False
def test_private_staging_is_read_only_and_original_path_drift_is_not_consumed(tmp_path: Path) -> None:
packet = _write_source_packet(tmp_path)
dump = tmp_path / "genesis.dump"
manifest = tmp_path / "manifest.jsonl"
exporter_receipt = tmp_path / "exporter-receipt.json"
dump.write_bytes(b"PGDMP-staged-input")
manifest.write_text("{}\n", encoding="utf-8")
exporter_receipt.write_text("{}\n", encoding="utf-8")
exporter_receipt.chmod(0o600)
staged = lifecycle.stage_lifecycle_inputs(packet, dump, manifest, exporter_receipt)
try:
before = lifecycle.verify_staged_inputs(staged)
dump.write_bytes(b"PGDMP-mutated-original")
(packet / "proposal-packet.json").write_text("{}\n", encoding="utf-8")
after = lifecycle.verify_staged_inputs(staged)
assert before == after
assert staged.genesis_dump.read_bytes() == b"PGDMP-staged-input"
assert staged.root.stat().st_mode & 0o777 == 0o500
assert all(
(staged.root / row["relative_path"]).stat().st_mode & 0o777 == 0o400 for row in staged.files.values()
)
finally:
cleanup = lifecycle.cleanup_staged_inputs(staged.root)
assert cleanup["root_absent"] is True
def test_private_staging_detects_consumed_byte_mutation(tmp_path: Path) -> None:
packet = _write_source_packet(tmp_path)
dump = tmp_path / "genesis.dump"
manifest = tmp_path / "manifest.jsonl"
exporter_receipt = tmp_path / "exporter-receipt.json"
dump.write_bytes(b"PGDMP-staged-input")
manifest.write_text("{}\n", encoding="utf-8")
exporter_receipt.write_text("{}\n", encoding="utf-8")
exporter_receipt.chmod(0o600)
staged = lifecycle.stage_lifecycle_inputs(packet, dump, manifest, exporter_receipt)
staged_dump = staged.genesis_dump
try:
staged_dump.chmod(0o600)
staged_dump.write_bytes(b"PGDMP-mutated-staging")
staged_dump.chmod(0o400)
with pytest.raises(lifecycle.LifecycleError, match="staging bytes drifted"):
lifecycle.verify_staged_inputs(staged)
finally:
cleanup = lifecycle.cleanup_staged_inputs(staged.root)
assert cleanup["root_absent"] is True
def test_genesis_initialization_cannot_call_synthetic_bootstrap(tmp_path: Path) -> None:
class RestoreOnlyPostgres:
def __init__(self) -> None:
self.restored = False
def restore_genesis(self, _inputs):
self.restored = True
return {"parity": {"status": "pass"}}
def psql(self, sql: str) -> str:
assert sql.startswith("alter role kb_review password")
return ""
postgres = RestoreOnlyPostgres()
result = lifecycle.initialize_database(
postgres,
genesis=object(),
review_password="unused",
apply_password="unused",
)
assert postgres.restored is True
assert result["mode"] == "exporter_receipt_bound_genesis_restore"
assert result["synthetic_schema_bootstrap_skipped"] is True
assert result["canonical_payload_preseed_performed"] is False
def test_fresh_initial_state_rejects_proposal_and_payload_preseed() -> None:
empty_rows = {family: [] for family in lifecycle.apply_lifecycle.ROW_TABLES}
with pytest.raises(lifecycle.LifecycleError, match="proposal or approval preseed"):
lifecycle.require_fresh_initial_state(
{"state": {"proposal": {"id": "preseed"}, "immutable_approval": None, "canonical_rows": empty_rows}}
)
preseed_rows = {family: [] for family in lifecycle.apply_lifecycle.ROW_TABLES}
preseed_rows["claims"] = [{"id": "preseed"}]
with pytest.raises(lifecycle.LifecycleError, match="canonical payload target preseed"):
lifecycle.require_fresh_initial_state(
{"state": {"proposal": None, "immutable_approval": None, "canonical_rows": preseed_rows}}
)
def test_payload_controlled_fingerprint_ignores_generated_fields_but_detects_payload_drift() -> None:
rows = {family: [] for family in lifecycle.PAYLOAD_CONTROLLED_FIELDS}
rows["claims"] = [
{
"id": "11111111-1111-4111-8111-111111111111",
"type": "fact",
"text": "Stable payload text",
"status": "open",
"confidence": 0.7,
"tags": ["test"],
"created_by": None,
"superseded_by": None,
"created_at": "2026-01-01T00:00:00+00:00",
}
]
first = lifecycle.payload_controlled_fingerprint(rows)
rows["claims"][0]["created_at"] = "2027-01-01T00:00:00+00:00"
rows["claims"][0]["updated_at"] = "2027-01-01T00:00:01+00:00"
second = lifecycle.payload_controlled_fingerprint(rows)
rows["claims"][0]["text"] = "Drifted payload text"
drifted = lifecycle.payload_controlled_fingerprint(rows)
assert first["sha256"] == second["sha256"]
assert drifted["sha256"] != first["sha256"]
def _fake_genesis_run(run_number: int) -> dict:
controlled = {
"schema": "livingip.leoPayloadControlledRowsFingerprint.v1",
"sha256": "c" * 64,
"table_sha256": {family: "d" * 64 for family in lifecycle.PAYLOAD_CONTROLLED_FIELDS},
"row_counts": {family: 0 for family in lifecycle.PAYLOAD_CONTROLLED_FIELDS},
"rows": {family: [] for family in lifecycle.PAYLOAD_CONTROLLED_FIELDS},
}
return {
"status": "pass",
"run_number": run_number,
"source": {
"artifact_sha256": "1" * 64,
"manifest_canonical_sha256": "2" * 64,
"retained_proposal_packet_file_sha256": "3" * 64,
"retained_proposal_packet_canonical_sha256": "4" * 64,
},
"compiled": {"bundle_sha256": "4" * 64},
"identities": {
"proposal_id": "11111111-1111-4111-8111-111111111111",
"claim_ids": [],
"source_ids": [],
"planned_evidence_keys": [],
"evidence_ids": [f"generated-{run_number}"],
"edge_ids": [],
},
"phases": {
"staged_pending_review": {"proposal": {"status": "pending_review"}},
"approved_unapplied": {"proposal": {"status": "approved"}},
"applied": {
"proposal": {"status": "applied", "applied_at": f"2026-01-0{run_number}T00:00:00+00:00"},
"payload_controlled_fingerprint": controlled,
"worker_action": {
"exact_candidate_id": "11111111-1111-4111-8111-111111111111",
"credential_role": "kb_apply",
"retry_failure_count": 1,
"failure_state_after_success": {},
"private_receipt_mode": "0600",
},
"apply_receipt": {
"hashes": {"apply_payload_sha256": "5" * 64},
"expected_row_counts": controlled["row_counts"],
"actual_row_counts": controlled["row_counts"],
},
},
"recompiled_readback": {
"bundle_sha256": "4" * 64,
"matches_initial_compile": True,
},
},
"database_initialization": {
"genesis": {
"dump": {"sha256": "6" * 64},
"manifest": {"sha256": "7" * 64},
"exporter_receipt": {
"sha256": "8" * 64,
"provenance_binding_sha256": "9" * 64,
},
"image": {"reference": lifecycle.DEFAULT_GENESIS_IMAGE},
"parity": {
"status": "pass",
"problems": [],
"claim_scope": "full_captured_manifest_including_role_and_database_settings",
"role_global_parity": {
"status": "proven",
"captured_source_settings_present": True,
"reason": None,
},
},
"key_counts": {"public.claims": 10},
"canonical_schema_table_count": 10,
}
},
"cleanup": {"all": True},
}
def _mocked_genesis_case(tmp_path: Path) -> tuple[Path, Path, Path, Path]:
packet = _write_source_packet(tmp_path)
dump = tmp_path / "genesis.dump"
manifest = tmp_path / "genesis-manifest.jsonl"
exporter_receipt = tmp_path / "exporter-receipt.json"
dump.write_bytes(b"PGDMPx")
manifest.write_text("{}\n", encoding="utf-8")
_write_exporter_receipt(exporter_receipt, dump, manifest, _minimal_loaded_manifest())
return packet, dump, manifest, exporter_receipt
def _fake_validate_genesis(
dump: Path,
manifest: Path,
exporter_receipt: Path,
image: str,
**paths,
) -> lifecycle.GenesisInputs:
payload = json.loads(exporter_receipt.read_text(encoding="utf-8"))
return lifecycle.GenesisInputs(
dump=dump,
dump_sha256=lifecycle.sha256_file(dump),
dump_bytes=dump.stat().st_size,
manifest_path=manifest,
manifest_sha256=lifecycle.sha256_file(manifest),
manifest_bytes=manifest.stat().st_size,
manifest=_minimal_loaded_manifest(),
image=image,
exporter_receipt_path=exporter_receipt,
exporter_receipt_sha256=lifecycle.sha256_file(exporter_receipt),
exporter_receipt=payload,
source_identity=payload["source"],
capture_authorization_ref=payload["capture_authorization_ref"],
provenance_binding_sha256=payload["provenance_binding"]["sha256"],
trust_root_receipt_sha256=lifecycle.sha256_file(exporter_receipt),
role_settings_capture_complete=True,
parity_sql_path=paths["parity_sql_path"],
guard_prerequisites_path=paths["guard_prerequisites_path"],
approve_script_path=paths["approve_script_path"],
apply_worker_script_path=paths["apply_worker_script_path"],
apply_script_path=paths["apply_script_path"],
)
def test_genesis_lifecycle_repeats_semantically_and_writes_private_receipt(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
packet, dump, manifest, exporter_receipt = _mocked_genesis_case(tmp_path)
calls: list[int] = []
monkeypatch.setattr(lifecycle, "validate_genesis_inputs", _fake_validate_genesis)
def fake_run(*_args, run_number: int, **_kwargs):
calls.append(run_number)
return _fake_genesis_run(run_number)
monkeypatch.setattr(lifecycle, "run_lifecycle", fake_run)
monkeypatch.setattr(
lifecycle,
"verify_run_group_cleanup",
lambda run_group: {"checked": True, "run_group": run_group, "container_names": [], "all_absent": True},
)
output = tmp_path / "genesis-lifecycle-receipt.json"
receipt = lifecycle.run_genesis_lifecycle(
packet,
output,
review_action="approve",
genesis_dump=dump,
genesis_manifest=manifest,
genesis_exporter_receipt=exporter_receipt,
image=lifecycle.DEFAULT_GENESIS_IMAGE,
)
assert calls == [1, 2]
assert receipt["status"] == "pass"
assert receipt["repeatability"]["semantic_match"] is True
assert len(set(receipt["repeatability"]["semantic_fingerprints_sha256"])) == 1
assert receipt["repeatability"]["identical_post_apply_fingerprints"] is True
assert receipt["checks"]["both_runs_consumed_preflight_bound_bytes"] is True
assert receipt["cleanup"]["zero_staging_leftovers"] is True
assert receipt["cleanup"]["zero_orphan_containers"] is True
assert output.stat().st_mode & 0o777 == 0o600
assert json.loads(output.read_text(encoding="utf-8")) == receipt
def test_genesis_lifecycle_fails_closed_on_semantic_repeatability_drift(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
packet, dump, manifest, exporter_receipt = _mocked_genesis_case(tmp_path)
monkeypatch.setattr(lifecycle, "validate_genesis_inputs", _fake_validate_genesis)
def drifted_run(*_args, run_number: int, **_kwargs):
run = _fake_genesis_run(run_number)
if run_number == 2:
run["phases"]["applied"]["payload_controlled_fingerprint"]["sha256"] = "e" * 64
return run
monkeypatch.setattr(lifecycle, "run_lifecycle", drifted_run)
monkeypatch.setattr(
lifecycle,
"verify_run_group_cleanup",
lambda run_group: {"checked": True, "run_group": run_group, "container_names": [], "all_absent": True},
)
receipt = lifecycle.run_genesis_lifecycle(
packet,
tmp_path / "repeatability-drift-receipt.json",
review_action="approve",
genesis_dump=dump,
genesis_manifest=manifest,
genesis_exporter_receipt=exporter_receipt,
image=lifecycle.DEFAULT_GENESIS_IMAGE,
)
assert receipt["status"] == "fail"
assert receipt["checks"]["both_runs_passed"] is True
assert receipt["checks"]["both_runs_match_semantically"] is False
assert receipt["checks"]["identical_post_apply_fingerprints"] is False
assert len(set(receipt["repeatability"]["semantic_fingerprints_sha256"])) == 2
def test_genesis_lifecycle_fails_closed_on_orphan_cleanup(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
packet, dump, manifest, exporter_receipt = _mocked_genesis_case(tmp_path)
monkeypatch.setattr(lifecycle, "validate_genesis_inputs", _fake_validate_genesis)
monkeypatch.setattr(
lifecycle,
"run_lifecycle",
lambda *_args, run_number, **_kwargs: _fake_genesis_run(run_number),
)
monkeypatch.setattr(
lifecycle,
"verify_run_group_cleanup",
lambda run_group: {
"checked": True,
"run_group": run_group,
"container_names": ["orphaned-container"],
"all_absent": False,
},
)
receipt = lifecycle.run_genesis_lifecycle(
packet,
tmp_path / "failed-cleanup-receipt.json",
review_action="approve",
genesis_dump=dump,
genesis_manifest=manifest,
genesis_exporter_receipt=exporter_receipt,
image=lifecycle.DEFAULT_GENESIS_IMAGE,
)
assert receipt["status"] == "fail"
assert receipt["checks"]["zero_orphan_containers"] is False
@pytest.mark.skipif(not _docker_available(), reason="Docker daemon is required for the isolated lifecycle")
def test_disposable_postgres_lifecycle_end_to_end(tmp_path: Path) -> None:
packet = _write_source_packet(tmp_path)
@ -124,10 +771,143 @@ def test_disposable_postgres_lifecycle_end_to_end(tmp_path: Path) -> None:
assert receipt["phases"]["approved_unapplied"]["proposal"]["status"] == "approved"
assert receipt["phases"]["approved_unapplied"]["proposal"]["applied_at"] is None
assert receipt["phases"]["applied"]["proposal"]["status"] == "applied"
worker = receipt["phases"]["applied"]["worker_action"]
assert worker["report_only"]["expected_marker_present"] is True
assert worker["expected_retry_failure"]["returncode"] == 1
assert worker["retry_failure_count"] == 1
assert worker["failure_state_after_success"] == {}
assert worker["enabled_retry"]["expected_marker_present"] is True
assert worker["post_apply_exact_refusal"]["returncode"] == 1
assert worker["post_apply_exact_refusal"]["expected_refusal_marker_present"] is True
assert worker["private_receipt_mode"] == "0600"
assert receipt["phases"]["applied"]["apply_receipt"]["replay_ready"] is True
assert receipt["phases"]["recompiled_readback"]["matches_initial_compile"] is True
assert receipt["identities"]["evidence_ids"]
assert len(receipt["checks"]) >= 20
assert all(receipt["checks"].values())
assert receipt["cleanup"]["all"] is True
assert output.stat().st_mode & 0o777 == 0o600
assert json.loads(output.read_text(encoding="utf-8")) == receipt
def _build_real_docker_genesis(tmp_path: Path) -> tuple[Path, Path, Path]:
run_id = uuid.uuid4().hex[:10]
source = lifecycle.DisposablePostgres(
f"leo-real-genesis-source-{os.getpid()}-{run_id}",
"disposable-admin-password",
image=lifecycle.DEFAULT_GENESIS_IMAGE,
genesis_backed=True,
run_group=f"leo-real-genesis-source-{run_id}",
)
dump = tmp_path / "real-genesis.dump"
manifest_path = tmp_path / "real-genesis-manifest.jsonl"
exporter_receipt = tmp_path / "real-genesis-exporter-receipt.json"
try:
source.start()
source.psql(lifecycle.bootstrap_sql("review-password", "apply-password"))
source.psql(
"alter role kb_apply set statement_timeout to '17s';\n"
"alter role kb_review in database teleo set lock_timeout to '19s';"
)
manifest_output = source.psql(lifecycle.PARITY_SQL_PATH.read_text(encoding="utf-8"))
manifest_path.write_text(manifest_output + "\n", encoding="utf-8")
manifest_path.chmod(0o600)
container_dump = "/tmp/leo-real-genesis.dump"
source._run(
[
"exec",
source.name,
"pg_dump",
"-U",
"postgres",
"-d",
lifecycle.DATABASE,
"--format=custom",
"--compress=9",
"--no-owner",
f"--file={container_dump}",
]
)
source._run(["cp", f"{source.name}:{container_dump}", str(dump)])
dump.chmod(0o600)
loaded_manifest = lifecycle.load_manifest(manifest_path)
roles = {role["name"]: role for role in loaded_manifest["singleton"]["application_roles"]["items"]}
assert roles["kb_apply"]["settings"] == [{"name": "statement_timeout", "value": "17s"}]
assert roles["kb_review"]["database_settings"] == [{"name": "lock_timeout", "value": "19s"}]
_write_exporter_receipt(
exporter_receipt,
dump,
manifest_path,
loaded_manifest,
source={
"ssh_target": "disposable-docker-test",
"container": source.name,
"database": lifecycle.DATABASE,
"service": "disposable-postgres-test",
},
)
finally:
cleanup = source.cleanup()
assert all(
(
cleanup["container_absent"],
cleanup["name_readback_clear"],
cleanup["instance_label_readback_clear"],
cleanup["run_group_readback_clear"],
cleanup["network_mode_was_none"],
cleanup["tmpfs_data_dir_was_present"],
cleanup["docker_volume_mounts_observed"] == [],
)
)
return dump, manifest_path, exporter_receipt
@pytest.mark.skipif(not _docker_available(), reason="Docker daemon is required for the real genesis lifecycle")
def test_real_docker_genesis_restore_worker_parity_and_repeatability(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
packet = _write_source_packet(tmp_path)
dump, manifest, exporter_receipt = _build_real_docker_genesis(tmp_path)
receipt_payload = json.loads(exporter_receipt.read_text(encoding="utf-8"))
_trust_test_exporter_receipt(monkeypatch, exporter_receipt, receipt_payload)
output = tmp_path / "real-genesis-lifecycle-receipt.json"
receipt = lifecycle.run_genesis_lifecycle(
packet,
output,
review_action="approve",
review_note="Approved exact Docker genesis payload for regression coverage.",
genesis_dump=dump,
genesis_manifest=manifest,
genesis_exporter_receipt=exporter_receipt,
image=lifecycle.DEFAULT_GENESIS_IMAGE,
)
assert receipt["status"] == "pass", receipt.get("error")
assert receipt["execution"]["git_head"]
assert receipt["execution"]["engine_set_sha256"]
assert receipt["execution"]["engine_paths_match_head"] is True
assert receipt["execution"]["engine_sha256"] == receipt["execution"]["engine_head_sha256"]
assert receipt["pre_docker_validation"]["trusted_exporter_receipt_validated"] is True
assert receipt["pre_docker_validation"]["committed_engine_dependency_closure_verified"] is True
assert receipt["checks"]["both_runs_consumed_preflight_bound_bytes"] is True
assert receipt["checks"]["committed_engine_dependency_closure_verified"] is True
assert receipt["checks"]["portable_receipt_has_no_absolute_paths"] is True
assert lifecycle.absolute_path_value_pointers(receipt) == []
assert receipt["repeatability"]["identical_post_apply_fingerprints"] is True
assert len(set(receipt["repeatability"]["post_apply_fingerprints_sha256"])) == 1
assert receipt["cleanup"]["zero_orphan_containers"] is True
assert receipt["cleanup"]["zero_staging_leftovers"] is True
for run in receipt["runs"]:
assert run["status"] == "pass", run.get("error")
assert (
run["consumed_inputs"]["preflight_manifest_sha256"]
== receipt["immutable_staging"]["preflight_manifest_sha256"]
)
assert run["consumed_inputs"]["executing_git_head"] == receipt["execution"]["git_head"]
assert run["consumed_inputs"]["engine_set_sha256"] == receipt["execution"]["engine_set_sha256"]
assert run["database_initialization"]["genesis"]["parity"]["role_global_parity"]["status"] == "proven"
assert run["phases"]["applied"]["worker_action"]["retry_failure_count"] == 1
assert run["phases"]["applied"]["worker_action"]["private_receipt_mode"] == "0600"
assert output.stat().st_mode & 0o777 == 0o600
assert json.loads(output.read_text(encoding="utf-8")) == receipt

620
tests/test_run_local_corpus_knowledge_rebuild.py Normal file
View file

@ -0,0 +1,620 @@
from __future__ import annotations
import copy
import hashlib
import json
import math
import os
import stat
import subprocess
import sys
from collections.abc import Callable, Iterator
from pathlib import Path
import pytest
from ops import run_local_corpus_knowledge_rebuild as rebuild
REPO_ROOT = Path(__file__).resolve().parents[1]
CLI = REPO_ROOT / "ops" / "run_local_corpus_knowledge_rebuild.py"
FIXTURE_ROOT = REPO_ROOT / "tests" / "fixtures" / "teleo_corpus_rebuild"
def _sha256(value: bytes) -> str:
return hashlib.sha256(value).hexdigest()
def _load_json(path: Path) -> dict:
value = json.loads(path.read_text(encoding="utf-8"))
assert isinstance(value, dict)
return value
def _run_cli(source_root: Path, output: Path, *extra: str) -> subprocess.CompletedProcess[str]:
return subprocess.run(
[
sys.executable,
str(CLI),
"--source-root",
str(source_root),
"--output",
str(output),
*extra,
],
cwd=REPO_ROOT,
check=False,
capture_output=True,
text=True,
)
def _run_verify(bundle: Path) -> subprocess.CompletedProcess[str]:
return subprocess.run(
[sys.executable, str(CLI), "--verify-bundle", str(bundle)],
cwd=REPO_ROOT,
check=False,
capture_output=True,
text=True,
)
def _tree_bytes(root: Path) -> dict[str, bytes]:
return {path.relative_to(root).as_posix(): path.read_bytes() for path in sorted(root.rglob("*")) if path.is_file()}
def _all_keys(value: object) -> Iterator[str]:
if isinstance(value, dict):
for key, child in value.items():
yield key
yield from _all_keys(child)
elif isinstance(value, list):
for child in value:
yield from _all_keys(child)
def _make_source(root: Path, name: str = "source.md", content: bytes = b"# Source\n\nExact text.\n") -> Path:
root.mkdir(parents=True, exist_ok=True)
source = root / name
source.parent.mkdir(parents=True, exist_ok=True)
source.write_bytes(content)
return source
def test_cli_builds_deterministic_source_inventory_without_claims_or_proposals(tmp_path: Path) -> None:
first_output = tmp_path / "bundle-a"
second_output = tmp_path / "bundle-b"
first = _run_cli(FIXTURE_ROOT, first_output, "--mode", "source-inventory-only")
second = _run_cli(FIXTURE_ROOT, second_output)
assert first.returncode == second.returncode == 0, (first.stderr, first.stdout, second.stderr, second.stdout)
first_status = json.loads(first.stdout)
second_status = json.loads(second.stdout)
assert first_status["status"] == second_status["status"] == "pass"
assert first_status["source_count"] == second_status["source_count"] == 2
assert first_status["work_item_count"] == second_status["work_item_count"] == 3
assert first_status["bundle_tree_sha256"] == second_status["bundle_tree_sha256"]
assert _tree_bytes(first_output) == _tree_bytes(second_output)
assert rebuild.INCOMPLETE_MARKER not in _tree_bytes(first_output)
source_manifest = _load_json(first_output / rebuild.SOURCE_MANIFEST_PATH)
work_index = _load_json(first_output / rebuild.EXTRACTION_WORK_INDEX_PATH)
receipt = _load_json(first_output / rebuild.RECEIPT_PATH)
assert source_manifest["schema"] == rebuild.SOURCE_MANIFEST_SCHEMA
assert work_index["schema"] == rebuild.EXTRACTION_WORK_INDEX_SCHEMA
assert receipt["schema"] == rebuild.RECEIPT_SCHEMA
assert source_manifest["source_snapshot_semantics"] == {
"concurrent_directory_additions_or_removals_may_be_absent": True,
"directory_snapshot_atomic": False,
"guarantee": "each listed path is bound to the exact bytes copied from one stable file descriptor",
}
assert [row["relative_path"] for row in source_manifest["sources"]] == [
"forgejo/project-record.md",
"operator-notes.txt",
]
assert [row["section_count"] for row in source_manifest["sources"]] == [2, 1]
assert work_index["work_type"] == rebuild.WORK_TYPE
assert all(item["work_type"] == rebuild.WORK_TYPE for item in work_index["items"])
forbidden_keys = {"candidate", "candidates", "claim", "claims", "proposal", "proposals", "proposition"}
for document in (source_manifest, work_index, receipt):
assert forbidden_keys.isdisjoint(_all_keys(document))
for key, expected in rebuild.NO_WRITE_FLAGS.items():
assert document[key] is expected
sources_by_path = {row["relative_path"]: row for row in source_manifest["sources"]}
for relative_path, source in sources_by_path.items():
raw = (FIXTURE_ROOT / relative_path).read_bytes()
snapshot = first_output / source["snapshot_path"]
assert snapshot.read_bytes() == raw
assert source["bytes"] == len(raw)
assert source["artifact_sha256"] == _sha256(raw)
assert source["format_syntax_validated"] is False
for locator in source["sections"]:
start = locator["utf8_byte_start"]
end = locator["utf8_byte_end"]
quote = raw[start:end]
assert quote
assert locator["quote_sha256"] == _sha256(quote)
assert locator["line_start"] == raw[:start].decode("utf-8").count("\n") + 1
assert locator["line_end"] == raw[: end - 1].decode("utf-8").count("\n") + 1
artifacts = receipt["outputs"]["artifacts"]
assert receipt["outputs"]["artifact_count"] == len(artifacts)
assert receipt["hashes"]["bundle_tree_sha256"] == rebuild.canonical_sha256(artifacts)
receipt_preimage = copy.deepcopy(receipt)
receipt_preimage["hashes"].pop("receipt_content_sha256")
assert receipt["hashes"]["receipt_content_sha256"] == rebuild.canonical_sha256(receipt_preimage)
assert rebuild.verify_bundle(first_output)["status"] == "pass"
persisted = b"".join(_tree_bytes(first_output).values())
assert str(FIXTURE_ROOT).encode() not in persisted
assert str(first_output).encode() not in persisted
for path in first_output.rglob("*"):
expected_mode = 0o600 if path.is_file() else 0o700
assert stat.S_IMODE(path.stat().st_mode) == expected_mode
def test_receipt_is_published_last(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
source_root = tmp_path / "sources"
_make_source(source_root)
output = tmp_path / "bundle"
original = rebuild._write_private_json
writes: list[str] = []
def record(path: Path, value: object) -> None:
writes.append(path.name)
if path.name == rebuild.RECEIPT_PATH:
assert (output / rebuild.INCOMPLETE_MARKER).exists()
original(path, value)
monkeypatch.setattr(rebuild, "_write_private_json", record)
rebuild.build_source_inventory_bundle(source_root, output)
assert writes[-1] == rebuild.RECEIPT_PATH
assert rebuild.INCOMPLETE_MARKER not in writes
assert not (output / rebuild.INCOMPLETE_MARKER).exists()
def test_verifier_rejects_rehashed_metadata_overclaim(tmp_path: Path) -> None:
output = tmp_path / "bundle"
rebuild.build_source_inventory_bundle(FIXTURE_ROOT, output)
receipt_path = output / rebuild.RECEIPT_PATH
receipt = _load_json(receipt_path)
receipt["canonical_database_rebuilt"] = True
receipt["hashes"].pop("receipt_content_sha256")
receipt["hashes"]["receipt_content_sha256"] = rebuild.canonical_sha256(receipt)
receipt_path.write_text(json.dumps(receipt), encoding="utf-8")
completed = _run_verify(output)
assert completed.returncode == 2
assert json.loads(completed.stdout)["reason"] == "invalid_bundle_schema"
METADATA_PATHS = (
rebuild.SOURCE_MANIFEST_PATH,
rebuild.EXTRACTION_WORK_INDEX_PATH,
rebuild.RECEIPT_PATH,
)
def _inject_duplicate_top_level_key(raw: str, _metadata_path: str) -> str:
return raw.replace("{\n", '{\n "schema": "contradictory-shadow",\n', 1)
def _inject_duplicate_nested_key(raw: str, metadata_path: str) -> str:
anchors = {
rebuild.SOURCE_MANIFEST_PATH: (
'"source_snapshot_semantics": {\n',
' "directory_snapshot_atomic": true,\n',
),
rebuild.EXTRACTION_WORK_INDEX_PATH: (
'"source_manifest": {\n',
' "path": "contradictory-shadow.json",\n',
),
rebuild.RECEIPT_PATH: (
'"hashes": {\n',
f' "bundle_tree_sha256": "{"0" * 64}",\n',
),
}
anchor, duplicate = anchors[metadata_path]
assert anchor in raw
return raw.replace(anchor, anchor + duplicate, 1)
def _inject_nonfinite(raw: str, constant: str) -> str:
return raw.replace("{\n", f'{{\n "nonfinite_shadow": {constant},\n', 1)
@pytest.mark.parametrize("metadata_path", METADATA_PATHS)
@pytest.mark.parametrize(
("mutator", "expected_reason"),
(
(_inject_duplicate_top_level_key, "duplicate_json_key"),
(_inject_duplicate_nested_key, "duplicate_json_key"),
(lambda raw, _path: _inject_nonfinite(raw, "NaN"), "invalid_json_value"),
(lambda raw, _path: _inject_nonfinite(raw, "Infinity"), "invalid_json_value"),
(lambda raw, _path: _inject_nonfinite(raw, "-Infinity"), "invalid_json_value"),
),
)
def test_verifier_rejects_ambiguous_or_nonfinite_metadata_before_verification(
tmp_path: Path,
metadata_path: str,
mutator: Callable[[str, str], str],
expected_reason: str,
) -> None:
output = tmp_path / "bundle"
rebuild.build_source_inventory_bundle(FIXTURE_ROOT, output)
target = output / metadata_path
target.write_text(mutator(target.read_text(encoding="utf-8"), metadata_path), encoding="utf-8")
completed = _run_verify(output)
assert completed.returncode == 2
assert json.loads(completed.stdout)["reason"] == expected_reason
@pytest.mark.parametrize("value", (math.nan, math.inf, -math.inf))
def test_canonical_inventory_json_writers_reject_nonfinite_numbers(value: float) -> None:
for writer in (rebuild.canonical_json_bytes, rebuild.rendered_json_bytes):
with pytest.raises(rebuild.RebuildError) as error:
writer({"value": value})
assert error.value.code == "invalid_json_value"
def test_verify_mode_is_read_only(tmp_path: Path) -> None:
output = tmp_path / "bundle"
completed = _run_cli(FIXTURE_ROOT, output)
assert completed.returncode == 0, completed.stdout
before = {path: (path.stat().st_mtime_ns, path.read_bytes()) for path in output.rglob("*") if path.is_file()}
verified = _run_verify(output)
assert verified.returncode == 0, (verified.stdout, verified.stderr)
assert json.loads(verified.stdout)["operation"] == "verify"
after = {path: (path.stat().st_mtime_ns, path.read_bytes()) for path in output.rglob("*") if path.is_file()}
assert after == before
def test_post_read_source_mutation_does_not_change_copied_snapshot_or_overclaim_atomicity(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
source_root = tmp_path / "sources"
source = _make_source(source_root, content=b"# Original\n\nCopied bytes.\n")
original = source.read_bytes()
output = tmp_path / "bundle"
def mutate(relative_path: str, _source_root: Path, _snapshot: Path) -> None:
assert relative_path == "source.md"
source.write_bytes(b"# Mutated\n\nLater bytes.\n")
monkeypatch.setattr(rebuild, "_after_source_snapshot", mutate)
result = rebuild.build_source_inventory_bundle(source_root, output)
manifest = _load_json(output / rebuild.SOURCE_MANIFEST_PATH)
row = manifest["sources"][0]
assert (output / row["snapshot_path"]).read_bytes() == original
assert source.read_bytes() != original
assert row["artifact_sha256"] == _sha256(original)
assert manifest["source_snapshot_semantics"]["directory_snapshot_atomic"] is False
assert result["verification"]["status"] == "pass"
def test_destination_reservation_race_never_replaces_or_cleans_other_directory(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
source_root = tmp_path / "sources"
_make_source(source_root)
output = tmp_path / "bundle"
original_mkdir = rebuild.os.mkdir
raced = False
def racing_mkdir(path: os.PathLike[str] | str, mode: int = 0o777, *, dir_fd: int | None = None) -> None:
nonlocal raced
if Path(path) == output and not raced and dir_fd is None:
raced = True
original_mkdir(output, mode)
(output / "other-owner.txt").write_text("retain", encoding="utf-8")
original_mkdir(path, mode, dir_fd=dir_fd)
monkeypatch.setattr(rebuild.os, "mkdir", racing_mkdir)
with pytest.raises(rebuild.RebuildError) as error:
rebuild.build_source_inventory_bundle(source_root, output)
assert error.value.code == "output_exists"
assert (output / "other-owner.txt").read_text(encoding="utf-8") == "retain"
assert set(path.name for path in output.iterdir()) == {"other-owner.txt"}
def test_existing_destination_is_never_replaced(tmp_path: Path) -> None:
source_root = tmp_path / "sources"
_make_source(source_root)
output = tmp_path / "bundle"
output.mkdir()
marker = output / "retain.txt"
marker.write_text("owned elsewhere", encoding="utf-8")
with pytest.raises(rebuild.RebuildError) as error:
rebuild.build_source_inventory_bundle(source_root, output)
assert error.value.code == "output_exists"
assert marker.read_text(encoding="utf-8") == "owned elsewhere"
def test_output_identity_replacement_fails_before_writing_replacement(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
source_root = tmp_path / "sources"
_make_source(source_root)
output = tmp_path / "bundle"
moved = tmp_path / "moved-reservation"
def replace(reserved: rebuild.ReservedOutput) -> None:
assert reserved.path == output
output.rename(moved)
output.mkdir()
(output / "other-owner.txt").write_text("retain", encoding="utf-8")
monkeypatch.setattr(rebuild, "_after_output_reservation", replace)
with pytest.raises(rebuild.RebuildError) as error:
rebuild.build_source_inventory_bundle(source_root, output)
assert error.value.code == "output_binding_changed"
assert set(path.name for path in output.iterdir()) == {"other-owner.txt"}
assert set(path.name for path in moved.iterdir()) == {rebuild.INCOMPLETE_MARKER}
def test_failure_after_reservation_removes_only_reserved_output_and_marker(tmp_path: Path) -> None:
source_root = tmp_path / "sources"
_make_source(source_root, content=b"text\x00binary\n")
output = tmp_path / "bundle"
with pytest.raises(rebuild.RebuildError) as error:
rebuild.build_source_inventory_bundle(source_root, output)
assert error.value.code == "binary_control_character"
assert not output.exists()
def test_lf_and_crlf_locators_are_exact_and_cr_only_is_rejected(tmp_path: Path) -> None:
source_root = tmp_path / "sources"
source_root.mkdir()
lf = b"# One\nalpha\n# Two\nbeta\n"
crlf = b"# One\r\nalpha\r\n# Two\r\nbeta\r\n"
(source_root / "lf.md").write_bytes(lf)
(source_root / "crlf.md").write_bytes(crlf)
output = tmp_path / "bundle"
rebuild.build_source_inventory_bundle(source_root, output)
manifest = _load_json(output / rebuild.SOURCE_MANIFEST_PATH)
by_path = {row["relative_path"]: row for row in manifest["sources"]}
assert by_path["lf.md"]["newline_style"] == "lf"
assert by_path["crlf.md"]["newline_style"] == "crlf"
for name, raw in (("lf.md", lf), ("crlf.md", crlf)):
assert [(row["line_start"], row["line_end"]) for row in by_path[name]["sections"]] == [(1, 2), (3, 4)]
for locator in by_path[name]["sections"]:
quote = raw[locator["utf8_byte_start"] : locator["utf8_byte_end"]]
assert _sha256(quote) == locator["quote_sha256"]
cr_root = tmp_path / "cr-only"
_make_source(cr_root, content=b"# One\ralpha\r# Two\rbeta\r")
with pytest.raises(rebuild.RebuildError) as error:
rebuild.build_source_inventory_bundle(cr_root, tmp_path / "cr-bundle")
assert error.value.code == "unsupported_newline"
assert not (tmp_path / "cr-bundle").exists()
@pytest.mark.parametrize("content", [b"nul\x00byte\n", b"escape\x1bbyte\n", b"delete\x7fbyte\n"])
def test_binary_control_characters_are_rejected(tmp_path: Path, content: bytes) -> None:
source_root = tmp_path / "sources"
_make_source(source_root, content=content)
with pytest.raises(rebuild.RebuildError) as error:
rebuild.build_source_inventory_bundle(source_root, tmp_path / "bundle")
assert error.value.code == "binary_control_character"
assert not (tmp_path / "bundle").exists()
@pytest.mark.parametrize(
("content", "limits", "expected_code"),
[
(b"123456\n", rebuild.ResourceLimits(max_source_bytes=4), "source_byte_limit_exceeded"),
(b"123456\n", rebuild.ResourceLimits(max_total_bytes=4, max_source_bytes=4), "source_byte_limit_exceeded"),
(
b"# One\na\n# Two\nb\n",
rebuild.ResourceLimits(max_sections_per_source=1, max_work_items=1),
"section_limit_exceeded",
),
],
)
def test_resource_limits_fail_closed(
tmp_path: Path,
content: bytes,
limits: rebuild.ResourceLimits,
expected_code: str,
) -> None:
source_root = tmp_path / "sources"
_make_source(source_root, content=content)
with pytest.raises(rebuild.RebuildError) as error:
rebuild.build_source_inventory_bundle(source_root, tmp_path / "bundle", limits=limits)
assert error.value.code == expected_code
assert not (tmp_path / "bundle").exists()
def test_total_source_count_and_work_item_limits_fail_closed(tmp_path: Path) -> None:
source_root = tmp_path / "sources"
_make_source(source_root, "a.txt", b"a\n")
_make_source(source_root, "b.txt", b"b\n")
with pytest.raises(rebuild.RebuildError) as count_error:
rebuild.build_source_inventory_bundle(
source_root,
tmp_path / "count-bundle",
limits=rebuild.ResourceLimits(max_source_count=1),
)
assert count_error.value.code == "source_count_limit_exceeded"
with pytest.raises(rebuild.RebuildError) as work_error:
rebuild.build_source_inventory_bundle(
source_root,
tmp_path / "work-bundle",
limits=rebuild.ResourceLimits(max_work_items=1, max_sections_per_source=1),
)
assert work_error.value.code == "work_item_limit_exceeded"
with pytest.raises(rebuild.RebuildError) as total_error:
rebuild.build_source_inventory_bundle(
source_root,
tmp_path / "total-bundle",
limits=rebuild.ResourceLimits(max_source_bytes=2, max_total_bytes=3),
)
assert total_error.value.code == "total_byte_limit_exceeded"
def test_limit_overrides_remain_hard_bounded(tmp_path: Path) -> None:
source_root = tmp_path / "sources"
_make_source(source_root)
completed = _run_cli(
source_root,
tmp_path / "bundle",
"--max-source-bytes",
str(rebuild.HARD_MAX_SOURCE_BYTES + 1),
)
assert completed.returncode == 2
assert json.loads(completed.stdout)["reason"] == "invalid_resource_limit"
assert not (tmp_path / "bundle").exists()
def test_identical_source_bytes_share_one_content_addressed_snapshot(tmp_path: Path) -> None:
source_root = tmp_path / "sources"
content = b"same exact bytes\n"
_make_source(source_root, "a.txt", content)
_make_source(source_root, "nested/b.txt", content)
output = tmp_path / "bundle"
rebuild.build_source_inventory_bundle(source_root, output)
manifest = _load_json(output / rebuild.SOURCE_MANIFEST_PATH)
assert manifest["source_count"] == 2
assert manifest["unique_snapshot_count"] == 1
assert manifest["duplicate_content_source_count"] == 1
assert len({row["snapshot_path"] for row in manifest["sources"]}) == 1
assert len(list((output / "snapshots" / "sha256").rglob("*.utf8"))) == 1
assert _load_json(output / rebuild.EXTRACTION_WORK_INDEX_PATH)["work_item_count"] == 2
def test_source_rows_use_global_utf8_relative_path_order(tmp_path: Path) -> None:
source_root = tmp_path / "sources"
_make_source(source_root, "a-z.txt", b"sibling\n")
_make_source(source_root, "a/x.txt", b"nested\n")
_make_source(source_root, "aa.txt", b"later\n")
output = tmp_path / "bundle"
rebuild.build_source_inventory_bundle(source_root, output)
rows = _load_json(output / rebuild.SOURCE_MANIFEST_PATH)["sources"]
assert [row["relative_path"] for row in rows] == ["a-z.txt", "a/x.txt", "aa.txt"]
assert rebuild.verify_bundle(output)["status"] == "pass"
def test_mismatched_existing_content_addressed_snapshot_is_rejected(tmp_path: Path) -> None:
output = tmp_path / "bundle"
prefix = output / "snapshots" / "sha256"
prefix.mkdir(parents=True)
partial = output / "snapshots" / ".partial-00000001"
content = b"expected\n"
partial.write_bytes(content)
digest = _sha256(content)
destination = output / Path(rebuild._snapshot_path_for_hash(digest))
destination.parent.mkdir()
destination.write_bytes(b"mismatch\n")
with pytest.raises(rebuild.RebuildError) as error:
rebuild._install_snapshot(output, partial, digest, len(content))
assert error.value.code == "snapshot_collision"
assert destination.read_bytes() == b"mismatch\n"
assert partial.read_bytes() == content
@pytest.mark.parametrize("target", ["snapshot", "manifest", "index", "receipt"])
def test_verifier_rejects_tampered_bundle(tmp_path: Path, target: str) -> None:
output = tmp_path / "bundle"
rebuild.build_source_inventory_bundle(FIXTURE_ROOT, output)
manifest = _load_json(output / rebuild.SOURCE_MANIFEST_PATH)
if target == "snapshot":
path = output / manifest["sources"][0]["snapshot_path"]
path.write_bytes(path.read_bytes() + b"tamper")
elif target == "manifest":
path = output / rebuild.SOURCE_MANIFEST_PATH
document = _load_json(path)
document["source_count"] += 1
path.write_text(json.dumps(document), encoding="utf-8")
elif target == "index":
path = output / rebuild.EXTRACTION_WORK_INDEX_PATH
document = _load_json(path)
document["items"][0]["work_type"] = "semantic_claim"
path.write_text(json.dumps(document), encoding="utf-8")
else:
path = output / rebuild.RECEIPT_PATH
document = _load_json(path)
document["database_write_performed"] = True
path.write_text(json.dumps(document), encoding="utf-8")
completed = _run_verify(output)
assert completed.returncode == 2
assert json.loads(completed.stdout)["status"] == "rejected"
def test_inventory_rejects_symlinks_unsupported_files_invalid_utf8_and_empty_text(tmp_path: Path) -> None:
source_root = tmp_path / "symlinks"
target = _make_source(source_root)
link = source_root / "alias.md"
try:
link.symlink_to(target.name)
except OSError as exc: # pragma: no cover - platform capability
pytest.skip(f"symlinks unavailable: {exc}")
with pytest.raises(rebuild.RebuildError) as symlink_error:
rebuild.build_source_inventory_bundle(source_root, tmp_path / "symlink-bundle")
assert symlink_error.value.code == "symlink_rejected"
cases = [
("unsupported", "unsafe.pdf", b"%PDF-1.7\n", "unsupported_source_file"),
("invalid", "invalid.md", b"invalid: \xff\n", "invalid_utf8_source"),
("empty", "empty.txt", b" \n\t\n", "empty_source_file"),
]
for directory, filename, content, expected in cases:
root = tmp_path / directory
_make_source(root, filename, content)
with pytest.raises(rebuild.RebuildError) as error:
rebuild.build_source_inventory_bundle(root, tmp_path / f"{directory}-bundle")
assert error.value.code == expected
def test_rejects_output_overlap_and_unsupported_mode(tmp_path: Path) -> None:
source_root = tmp_path / "sources"
_make_source(source_root)
with pytest.raises(rebuild.RebuildError) as overlap:
rebuild.build_source_inventory_bundle(source_root, source_root / "bundle")
assert overlap.value.code == "output_overlap"
completed = _run_cli(source_root, tmp_path / "bundle", "--mode", "candidate-only")
assert completed.returncode == 2
assert json.loads(completed.stdout)["reason"] == "unsupported_mode"
assert not (tmp_path / "bundle").exists()
def test_structured_extensions_are_inventoried_as_utf8_not_claimed_as_format_validated(tmp_path: Path) -> None:
source_root = tmp_path / "sources"
_make_source(source_root, "opaque.json", b"not actually json\n")
output = tmp_path / "bundle"
rebuild.build_source_inventory_bundle(source_root, output)
row = _load_json(output / rebuild.SOURCE_MANIFEST_PATH)["sources"][0]
assert row["sectioning_strategy"] == "whole_utf8_source"
assert row["format_syntax_validated"] is False
assert row["section_count"] == 1

755
tests/test_run_local_genesis_ledger_rebuild.py
View file

@ -17,6 +17,13 @@ import pytest
from ops import run_local_genesis_ledger_rebuild as rebuild
from ops.verify_postgres_parity_manifest import compare_manifests, load_manifest
from tests.docker_volume_lifecycle import (
POSTGRES_TMPFS_SPEC,
acquire_docker_volume_set_guard,
inspect_postgres_tmpfs_container,
release_docker_volume_set_guard,
remove_container_and_volumes,
)
CLAIM_A = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"
CLAIM_B = "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb"
@ -37,6 +44,22 @@ RECEIPT_STRATEGY_ID = "14141414-1414-4414-8414-141414141414"
RECEIPT_NODE_A_ID = "15151515-1515-4515-8515-151515151515"
RECEIPT_NODE_B_ID = "16161616-1616-4616-8616-161616161616"
PRIVATE_MARKER = "PRIVATE-REPLAY-MATERIAL-MUST-NOT-LEAK"
EXPORTER_MATERIAL_FIELDS = frozenset(
{
"artifact",
"contract_version",
"sequence",
"approved_proposal",
"approval_snapshot",
"applied_proposal",
"replay_receipt",
}
)
V3_PROPOSAL_ID = "17171717-1717-4717-8717-171717171717"
V3_CLAIM_ID = "18181818-1818-4818-8818-181818181818"
V3_SOURCE_ID = "20202020-2020-4020-8020-202020202020"
V3_EVIDENCE_ID = "21212121-2121-4121-8121-212121212121"
V3_ASSESSMENT_ID = "23232323-2323-4323-8323-232323232323"
def sha256_file(path: Path) -> str:
@ -45,6 +68,11 @@ def sha256_file(path: Path) -> str:
return digest.hexdigest()
def sha256_json(value: object) -> str:
encoded = json.dumps(value, ensure_ascii=False, separators=(",", ":"), sort_keys=True).encode("utf-8")
return hashlib.sha256(encoded).hexdigest()
def write_json(path: Path, value: object) -> Path:
path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n", encoding="utf-8")
return path
@ -270,6 +298,272 @@ def replay_material() -> dict:
}
def v3_replay_material() -> dict:
evidence = {
"id": V3_EVIDENCE_ID,
"claim_id": V3_CLAIM_ID,
"source_id": V3_SOURCE_ID,
"polarity": "supports",
"excerpt": "The exact source bytes support this bounded reconstruction claim.",
"locator_json": {"section": "v3-rebuild", "paragraph": 1},
"source_content_hash": "a" * 64,
"rationale": "The content-addressed source and exact locator support the proposition.",
"created_by": REVIEWER_ID,
}
apply_payload = {
"contract_version": 3,
"agent_id": REVIEWER_ID,
"claims": [
{
"id": V3_CLAIM_ID,
"owner_agent_id": None,
"type": "empirical",
"proposition": "V3 knowledge can be reconstructed from genesis plus an exact proposal ledger.",
"body_markdown": "The reconstruction replays review, admission, provenance, and assessment rows.",
"scope": "Networkless disposable PostgreSQL reconstruction.",
"access_scope": "collective_internal",
"status": "active",
"revision_criteria": "Revise if exact final parity or trigger restoration cannot be reproduced.",
"revision_kind": "original",
"supersedes_id": None,
"tags": ["v3", "reconstruction"],
"created_by": REVIEWER_ID,
}
],
"sources": [
{
"id": V3_SOURCE_ID,
"source_type": "paper",
"canonical_url": "https://example.test/v3-reconstruction",
"storage_uri": None,
"excerpt": "Exact source bytes for the disposable V3 reconstruction.",
"content_hash": "a" * 64,
"access_scope": "collective_internal",
"ingestion_origin": "operator_upload",
"provenance_status": "verified",
"source_class": "primary_record",
"auto_promotion_policy": "human_required",
"captured_at": "2026-07-18T10:00:00+00:00",
"created_by": REVIEWER_ID,
}
],
"evidence": [evidence],
"edges": [],
"assessments": [
{
"id": V3_ASSESSMENT_ID,
"claim_id": V3_CLAIM_ID,
"support_tier": "strong",
"challenge_tier": "none",
"overall_state": "support_dominant",
"rationale": "The exact source and locator support the bounded reconstruction proposition.",
"evidence_set_hash": rebuild.apply_engine._v3_evidence_set_hash(V3_CLAIM_ID, [evidence]),
"supersedes_assessment_id": None,
}
],
"reasoning_tools": [],
}
payload = {"title": "V3 reconstruction fixture", "apply_payload": apply_payload}
approved = {
"id": V3_PROPOSAL_ID,
"proposal_type": "approve_claim",
"status": "approved",
"proposed_by_handle": "fixture-agent",
"proposed_by_agent_id": REVIEWER_ID,
"channel": "v3_reconstruction_fixture",
"source_ref": "private://fixture/v3-source",
"rationale": PRIVATE_MARKER,
"payload": payload,
"reviewed_by_handle": "m3ta",
"reviewed_by_agent_id": REVIEWER_ID,
"reviewed_at": "2026-07-18T10:01:00+00:00",
"review_note": "Reviewed the exact V3 reconstruction payload.",
"applied_by_handle": None,
"applied_by_agent_id": None,
"applied_at": None,
"created_at": "2026-07-18T10:00:30+00:00",
"updated_at": "2026-07-18T10:01:00+00:00",
}
applied = {
**approved,
"status": "applied",
"applied_by_handle": "kb-apply",
"applied_by_agent_id": SERVICE_ID,
"applied_at": "2026-07-18T10:02:00+00:00",
"updated_at": "2026-07-18T10:02:00.000001+00:00",
}
approval = {
"proposal_id": V3_PROPOSAL_ID,
"proposal_type": "approve_claim",
"payload": payload,
"reviewed_by_handle": approved["reviewed_by_handle"],
"reviewed_by_agent_id": REVIEWER_ID,
"reviewed_by_db_role": "kb_review",
"reviewed_at": approved["reviewed_at"],
"review_note": approved["review_note"],
}
created_at = "2026-07-18T10:01:30+00:00"
rows = {
"claims": [
{
"id": V3_CLAIM_ID,
"owner_agent_id": None,
"type": "empirical",
"text": apply_payload["claims"][0]["proposition"],
"proposition": apply_payload["claims"][0]["proposition"],
"body_markdown": apply_payload["claims"][0]["body_markdown"],
"scope": apply_payload["claims"][0]["scope"],
"access_scope": "collective_internal",
"status": "active",
"admission_state": "admitted",
"revision_criteria": apply_payload["claims"][0]["revision_criteria"],
"revision_kind": "original",
"supersedes_id": None,
"confidence": None,
"tags": ["v3", "reconstruction"],
"created_by": REVIEWER_ID,
"superseded_by": None,
"accepted_by_proposal_id": V3_PROPOSAL_ID,
"created_at": created_at,
"updated_at": created_at,
}
],
"sources": [
{
"id": V3_SOURCE_ID,
"source_type": "paper",
"url": apply_payload["sources"][0]["canonical_url"],
"storage_path": None,
"excerpt": apply_payload["sources"][0]["excerpt"],
"hash": "a" * 64,
"created_by": REVIEWER_ID,
"captured_at": apply_payload["sources"][0]["captured_at"],
"canonical_url": apply_payload["sources"][0]["canonical_url"],
"storage_uri": None,
"content_hash": "a" * 64,
"access_scope": "collective_internal",
"ingestion_origin": "operator_upload",
"provenance_status": "verified",
"source_class": "primary_record",
"auto_promotion_policy": "human_required",
"accepted_by_proposal_id": V3_PROPOSAL_ID,
"created_at": created_at,
}
],
"claim_evidence": [
{
"id": V3_EVIDENCE_ID,
"claim_id": V3_CLAIM_ID,
"source_id": V3_SOURCE_ID,
"role": "grounds",
"weight": None,
"created_by": REVIEWER_ID,
"excerpt": evidence["excerpt"],
"locator_json": evidence["locator_json"],
"source_content_hash": "a" * 64,
"rationale": evidence["rationale"],
"polarity": "supports",
"accepted_from_proposal_id": V3_PROPOSAL_ID,
"created_at": created_at,
}
],
"claim_edges": [],
"claim_evidence_assessments": [
{
**apply_payload["assessments"][0],
"accepted_by_proposal_id": V3_PROPOSAL_ID,
"created_at": created_at,
}
],
"reasoning_tools": [],
}
proposal = applied_envelope(applied)
apply_sql = rebuild.apply_engine.build_apply_sql(proposal, applied["applied_by_handle"])
receipt = rebuild.replay_receipt.build_replay_receipt(
proposal,
rows,
apply_sql_sha256=rebuild.replay_receipt.sha256_text(apply_sql),
apply_sql_source="exact_executed_sql",
exported_at_utc="2026-07-18T10:03:00+00:00",
)
return {
"artifact": rebuild.MATERIAL_ARTIFACT,
"contract_version": rebuild.MATERIAL_CONTRACT_VERSION,
"sequence": 1,
"approved_proposal": approved,
"approval_snapshot": approval,
"applied_proposal": applied,
"replay_receipt": receipt,
}
def exporter_shaped_v3_replay_material() -> dict:
material = copy.deepcopy(v3_replay_material())
payload_rows = (
material["approved_proposal"],
material["approval_snapshot"],
material["applied_proposal"],
material["replay_receipt"]["proposal"],
)
for row in payload_rows:
row["payload"]["apply_payload"]["sources"][0]["captured_at"] = "2026-07-18T12:00:00+02:00"
approved = material["approved_proposal"]
approved.update(
created_at="2026-07-18T10:00:30.000000Z",
reviewed_at="2026-07-18T10:01:00.000000Z",
updated_at="2026-07-18T10:01:00.000000Z",
)
applied = material["applied_proposal"]
applied.update(
created_at=approved["created_at"],
reviewed_at=approved["reviewed_at"],
applied_at="2026-07-18T10:02:00.000000Z",
updated_at="2026-07-18T10:02:00.000001Z",
)
material["approval_snapshot"]["reviewed_at"] = approved["reviewed_at"]
canonical_rows = material["replay_receipt"]["canonical_rows"]
for table_rows in canonical_rows.values():
for row in table_rows:
if "captured_at" in row:
row["captured_at"] = "2026-07-18T10:00:00.000000Z"
if "created_at" in row:
row["created_at"] = "2026-07-18T10:01:30.000000Z"
if "updated_at" in row:
row["updated_at"] = "2026-07-18T10:01:30.000000Z"
proposal = applied_envelope(applied)
semantic_proposal = copy.deepcopy(proposal)
semantic_proposal["payload"]["apply_payload"]["sources"][0]["captured_at"] = canonical_rows["sources"][0][
"captured_at"
]
receipt = rebuild.replay_receipt.build_replay_receipt(
semantic_proposal,
canonical_rows,
apply_sql_sha256=rebuild.replay_receipt.sha256_text("snapshot exporter fixture apply SQL"),
apply_sql_source="exact_executed_sql",
exported_at_utc="2026-07-18T10:03:00.000000Z",
)
receipt["proposal"]["payload"] = proposal["payload"]
replay_material = {
"receipt_contract_version": receipt["receipt_contract_version"],
"apply_engine": receipt["apply_engine"],
"proposal": receipt["proposal"],
"canonical_rows": receipt["canonical_rows"],
}
receipt["hashes"] = {
"apply_payload_sha256": sha256_json(proposal["payload"]["apply_payload"]),
"proposal_payload_sha256": sha256_json(proposal["payload"]),
"canonical_rows_sha256": sha256_json(receipt["canonical_rows"]),
"row_sha256": {table: [sha256_json(row) for row in rows] for table, rows in receipt["canonical_rows"].items()},
"table_sha256": {table: sha256_json(rows) for table, rows in receipt["canonical_rows"].items()},
"replay_material_sha256": sha256_json(replay_material),
}
material["replay_receipt"] = receipt
return material
def revise_strategy_replay_material(*, apply_sql_source: str = "exact_executed_sql", version: int = 2) -> dict:
approved, applied, approval = revise_strategy_proposal_rows()
proposal = applied_envelope(applied)
@ -386,9 +680,45 @@ def write_bundle(tmp_path: Path, *, material: dict | None = None) -> argparse.Na
max_target_source_ratio=100.0,
docker_bin="docker",
output=tmp_path / "receipt.json",
admission_manifest=None,
admission_manifest_sha256=None,
)
def attach_admission_manifest(
args: argparse.Namespace,
*,
admission_scope: str = "collective_core",
basis: str = "forgejo_primary_source",
) -> Path:
ledger = json.loads(args.ledger.read_text(encoding="utf-8"))
ledger_entry = ledger["entries"][0]
material = json.loads((args.ledger.parent / ledger_entry["material"]).read_text(encoding="utf-8"))
manifest = {
"artifact": rebuild.ADMISSION_ARTIFACT,
"contract_version": rebuild.ADMISSION_CONTRACT_VERSION,
"ledger_sha256": args.ledger_sha256,
"reviewed_by_handle": "m3taversal",
"reviewed_by_agent_id": REVIEWER_ID,
"reviewed_at": "2026-07-18T14:00:00+00:00",
"review_note": "Admit this exact source-backed V3 proposal into the clean genesis fixture.",
"entries": [
{
"sequence": 1,
"proposal_id": material["applied_proposal"]["id"],
"material_sha256": ledger_entry["sha256"],
"replay_material_sha256": ledger_entry["replay_material_sha256"],
"admission_scope": admission_scope,
"basis": basis,
}
],
}
path = write_json(args.ledger.parent / "admission-manifest.json", manifest)
args.admission_manifest = path
args.admission_manifest_sha256 = sha256_file(path)
return path
def rewrite_material_and_ledger(args: argparse.Namespace, material: dict) -> None:
material_path = args.ledger.parent / "entry-0001.json"
write_json(material_path, material)
@ -408,12 +738,77 @@ def test_load_bundle_validates_every_pin_and_current_apply_engine(tmp_path: Path
assert bundle.final_manifest_sha256 == sha256_file(tmp_path / "final-manifest.jsonl")
assert bundle.engine_hashes == rebuild._engine_hashes()
assert len(bundle.entries) == 1
assert bundle.admission == {
"status": "forensic_only",
"canonical_retrieval_eligible": False,
"reason": "no explicit admission manifest was supplied",
"entry_count": 1,
}
entry = bundle.entries[0]
assert entry.applied_proposal["proposal_type"] == "add_edge"
assert entry.current_apply_sql_sha256 == hashlib.sha256(entry.current_apply_sql.encode("utf-8")).hexdigest()
assert "kb_stage.assert_approved_proposal" in entry.current_apply_sql
def test_hash_pinned_v3_review_metadata_selects_material_without_granting_authority(tmp_path: Path) -> None:
args = write_bundle(tmp_path, material=exporter_shaped_v3_replay_material())
manifest_path = attach_admission_manifest(args)
bundle = rebuild.load_bundle(args)
assert bundle.admission_manifest_path == manifest_path
assert bundle.admission_manifest_sha256 == sha256_file(manifest_path)
assert bundle.admission["status"] == "review_metadata_supplied"
assert bundle.admission["canonical_retrieval_eligible"] is False
assert bundle.admission["cryptographically_authenticated"] is False
assert bundle.admission["review_metadata_provenance"] == rebuild.REVIEW_METADATA_PROVENANCE
assert "not authenticated" in bundle.admission["reason"]
assert bundle.admission["reviewed_by_handle"] == "m3taversal"
assert (
bundle.admission["review_note_sha256"]
== hashlib.sha256(b"Admit this exact source-backed V3 proposal into the clean genesis fixture.").hexdigest()
)
assert bundle.admission["entries"][0]["basis"] == "forgejo_primary_source"
def test_admission_rejects_forged_reviewer_identity(tmp_path: Path) -> None:
args = write_bundle(tmp_path, material=exporter_shaped_v3_replay_material())
manifest_path = attach_admission_manifest(args)
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
manifest["reviewed_by_handle"] = "forged-reviewer"
write_json(manifest_path, manifest)
args.admission_manifest_sha256 = sha256_file(manifest_path)
with pytest.raises(rebuild.ReconstructionError) as exc_info:
rebuild.load_bundle(args)
assert exc_info.value.code == "invalid_admission_review"
def test_admission_rejects_rehashed_duplicate_json_fields(tmp_path: Path) -> None:
args = write_bundle(tmp_path, material=exporter_shaped_v3_replay_material())
manifest_path = attach_admission_manifest(args)
raw = manifest_path.read_text(encoding="utf-8")
duplicate = '{"artifact":"teleo_ledger_admission_manifest",' + raw.lstrip()[1:]
manifest_path.write_text(duplicate, encoding="utf-8")
args.admission_manifest_sha256 = sha256_file(manifest_path)
with pytest.raises(rebuild.ReconstructionError) as exc_info:
rebuild.load_bundle(args)
assert exc_info.value.code == "invalid_json_artifact"
def test_admission_rejects_legacy_rows_instead_of_blessing_forensic_history(tmp_path: Path) -> None:
args = write_bundle(tmp_path)
attach_admission_manifest(args)
with pytest.raises(rebuild.ReconstructionError) as exc_info:
rebuild.load_bundle(args)
assert exc_info.value.code == "legacy_admission_forbidden"
def test_dump_hash_mismatch_fails_before_container_start(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
args = write_bundle(tmp_path)
args.genesis_dump.write_bytes(b"PGDMPtampered")
@ -531,7 +926,9 @@ def test_revise_strategy_accepts_timezone_aware_closed_timestamp_boundaries(
entry = rebuild.load_bundle(write_bundle(tmp_path, material=material)).entries[0]
assert entry.receipt["canonical_rows"]["strategies"][0]["created_at"] == transaction_timestamp
assert entry.receipt["canonical_rows"]["strategies"][0]["created_at"] == (
rebuild.replay_receipt.canonical_utc_timestamp(transaction_timestamp)
)
def test_revise_strategy_rejects_reviewed_at_without_timezone(tmp_path: Path) -> None:
@ -540,13 +937,8 @@ def test_revise_strategy_rejects_reviewed_at_without_timezone(tmp_path: Path) ->
for proposal_row in (material["approved_proposal"], material["applied_proposal"]):
proposal_row["reviewed_at"] = reviewed_at
material["approval_snapshot"]["reviewed_at"] = reviewed_at
rehash_revise_strategy_receipt(material)
with pytest.raises(rebuild.ReconstructionError) as exc_info:
rebuild.load_bundle(write_bundle(tmp_path, material=material))
assert exc_info.value.code == "invalid_mutating_receipt"
assert "reviewed_at must include a timezone" in exc_info.value.safe_message
with pytest.raises(ValueError, match="reviewed_at must be a timezone-aware ISO-8601 timestamp"):
rehash_revise_strategy_receipt(material)
@pytest.mark.parametrize(
@ -598,7 +990,7 @@ def test_revise_strategy_transition_builders_preserve_historical_prestate(tmp_pa
assert PRIOR_RETIRED_NODE_ID not in sql
assert PRIOR_STRATEGY_ID in sql
assert RECEIPT_STRATEGY_ID in sql
assert "updated_at = E'2026-07-14T01:00:30+00:00'::timestamptz" in sql
assert "updated_at = E'2026-07-14T01:00:30.000000Z'::timestamptz" in sql
def test_revise_strategy_transition_builders_allow_empty_prestate(tmp_path: Path) -> None:
@ -722,6 +1114,89 @@ def test_seed_sql_preserves_exact_rows_and_uses_only_fixed_tables(tmp_path: Path
assert "public.strategies" not in sql
def test_seed_sql_retains_the_exact_approved_row_outside_the_v1_approval_contract(
tmp_path: Path,
) -> None:
entry = rebuild.load_bundle(write_bundle(tmp_path)).entries[0]
approval_seed = {
**entry.approval_snapshot,
"approved_proposal_snapshot": entry.approved_proposal,
}
assert rebuild._jsonb_literal(approval_seed) in rebuild.build_seed_sql(entry)
assert "jsonb_set" in rebuild.build_seed_sql(entry)
assert "to_jsonb(proposal)" in rebuild.build_seed_sql(entry)
readback_sql = rebuild.build_proposal_readback_sql(entry.approved_proposal["id"])
assert "to_jsonb(a) - 'approved_proposal_snapshot'" in readback_sql
assert "'approved_proposal_snapshot', (select a.approved_proposal_snapshot" in readback_sql
def test_v3_seed_and_timestamp_normalization_preserve_contract_guards(tmp_path: Path) -> None:
entry = rebuild.load_bundle(write_bundle(tmp_path, material=v3_replay_material())).entries[0]
seed_sql = rebuild.build_seed_sql(entry)
normalization_sql = rebuild.build_applied_timestamp_normalization_sql(entry)
canonical_normalization_sql = rebuild.build_v3_canonical_timestamp_normalization_sql(entry)
assert entry.contract_version == 3
assert "approved_proposal_snapshot" in seed_sql
assert "public.claims" not in seed_sql
assert "public.sources" not in seed_sql
assert "captured_at = pinned.captured_at" in canonical_normalization_sql
assert "created_at = pinned.created_at" in canonical_normalization_sql
assert f"accepted_by_proposal_id = E'{V3_PROPOSAL_ID}'::uuid" in canonical_normalization_sql
assert "disable trigger teleo_v3_source_immutable" in canonical_normalization_sql
assert "enable trigger teleo_v3_source_immutable" in canonical_normalization_sql
assert "disable trigger all" not in canonical_normalization_sql
assert "trigger.tgenabled = 'O'" in canonical_normalization_sql
assert "disable trigger teleo_v3_guard_linked_proposal_decision" in normalization_sql
assert "disable trigger all" not in normalization_sql
assert normalization_sql.index("disable trigger") < normalization_sql.index("update kb_stage.kb_proposals")
assert normalization_sql.index("update kb_stage.kb_proposals") < normalization_sql.index("enable trigger")
assert "trigger.tgenabled = 'O'" in normalization_sql
def test_exporter_shaped_material_validates_storage_provenance_and_typed_source_timestamp(
tmp_path: Path,
) -> None:
material = exporter_shaped_v3_replay_material()
entry = rebuild.load_bundle(write_bundle(tmp_path, material=material)).entries[0]
assert frozenset(material) == EXPORTER_MATERIAL_FIELDS
assert len(entry.approval_snapshot) == 8
assert entry.approved_proposal["payload"]["apply_payload"]["sources"][0]["captured_at"].endswith("+02:00")
assert entry.receipt["canonical_rows"]["sources"][0]["captured_at"] == "2026-07-18T10:00:00.000000Z"
approval_seed = {
**entry.approval_snapshot,
"approved_proposal_snapshot": entry.approved_proposal,
}
assert rebuild._jsonb_literal(approval_seed) in rebuild.build_seed_sql(entry)
assert "public.sources" not in rebuild.build_seed_sql(entry)
readback_sql = rebuild.build_proposal_readback_sql(entry.approved_proposal["id"])
assert "to_jsonb(a) - 'approved_proposal_snapshot'" in readback_sql
assert "'approved_proposal_snapshot', (select a.approved_proposal_snapshot" in readback_sql
actual_rows = copy.deepcopy(entry.receipt["canonical_rows"])
actual_rows["sources"][0]["captured_at"] = "2026-07-18T12:00:00+02:00"
rebuild._assert_exact_canonical_rows(
actual_rows,
entry.receipt["canonical_rows"],
code="fixture_mismatch",
action="exporter-shaped fixture",
)
def test_contract_order_rejects_v2_after_v3() -> None:
v3 = type("Entry", (), {"contract_version": 3})()
v2 = type("Entry", (), {"contract_version": 2})()
rebuild._validate_contract_order([v2, v3])
with pytest.raises(rebuild.ReconstructionError, match="cannot appear") as exc_info:
rebuild._validate_contract_order([v3, v2])
assert exc_info.value.code == "contract_order_mismatch"
def test_private_psql_failure_output_is_replaced_with_a_fingerprint(
monkeypatch: pytest.MonkeyPatch,
) -> None:
@ -1004,33 +1479,39 @@ def capture_manifest(container: str, database: str, destination: Path) -> None:
def source_postgres() -> Iterator[tuple[str, str]]:
if shutil.which("docker") is None:
pytest.skip("Docker is required for the genesis-plus-ledger lifecycle test")
volume_guard = acquire_docker_volume_set_guard(
context="genesis ledger source PostgreSQL fixture",
)
container = f"genesis-ledger-source-{uuid.uuid4().hex[:12]}"
database = "teleo_fixture"
started = subprocess.run(
[
"docker",
"run",
"--rm",
"--detach",
"--network",
"none",
"--name",
container,
"--label",
f"{rebuild.canonical_rebuild.CANARY_LABEL}={container}",
"--env",
f"POSTGRES_DB={database}",
"--env",
"POSTGRES_HOST_AUTH_METHOD=trust",
rebuild.DEFAULT_REBUILD_IMAGE,
],
text=True,
capture_output=True,
check=False,
)
if started.returncode != 0:
pytest.fail(f"could not start fixture PostgreSQL: {started.stderr}")
try:
started = subprocess.run(
[
"docker",
"run",
"--detach",
"--network",
"none",
"--name",
container,
"--label",
f"{rebuild.canonical_rebuild.CANARY_LABEL}={container}",
"--tmpfs",
POSTGRES_TMPFS_SPEC,
"--env",
f"POSTGRES_DB={database}",
"--env",
"POSTGRES_HOST_AUTH_METHOD=trust",
rebuild.DEFAULT_REBUILD_IMAGE,
],
text=True,
capture_output=True,
check=False,
)
if started.returncode != 0:
pytest.fail(f"could not start fixture PostgreSQL: {started.stderr}")
inspect_postgres_tmpfs_container(container)
wait_for_postgres(container, database)
docker_psql(container, database, BOOTSTRAP_SQL)
copy_result = subprocess.run(
@ -1069,22 +1550,17 @@ def source_postgres() -> Iterator[tuple[str, str]]:
)
if migration.returncode != 0:
pytest.fail(f"could not apply guarded prerequisites: {migration.stderr}")
docker_psql(
container,
database,
"alter table kb_stage.kb_proposal_approvals add column if not exists approved_proposal_snapshot jsonb;",
)
yield container, database
finally:
subprocess.run(
["docker", "rm", "--force", container],
text=True,
capture_output=True,
check=False,
)
inspected = subprocess.run(
["docker", "container", "inspect", container],
text=True,
capture_output=True,
check=False,
)
if inspected.returncode == 0:
pytest.fail(f"fixture PostgreSQL container was not removed: {container}")
try:
remove_container_and_volumes(container)
finally:
release_docker_volume_set_guard(volume_guard)
def test_live_manifest_normalizes_physical_database_authorization_identity(
@ -1176,6 +1652,7 @@ def seed_proposal_and_approval(
approved: dict,
approval: dict,
) -> dict:
approval_seed = rebuild._approval_seed_row(approved, approval)
sql = f"""begin;
set local standard_conforming_strings = on;
insert into kb_stage.kb_proposals
@ -1184,7 +1661,7 @@ select seeded.* from jsonb_populate_record(
) seeded;
insert into kb_stage.kb_proposal_approvals
select seeded.* from jsonb_populate_record(
null::kb_stage.kb_proposal_approvals, {rebuild._jsonb_literal(approval)}
null::kb_stage.kb_proposal_approvals, {rebuild._jsonb_literal(approval_seed)}
) seeded;
commit;
"""
@ -1208,6 +1685,7 @@ def run_genesis_ledger_command(
artifact_stem: str,
container_prefix: str,
run_number: int,
admit: bool = False,
) -> tuple[subprocess.CompletedProcess[str], dict, Path]:
ledger = {
"artifact": rebuild.LEDGER_ARTIFACT,
@ -1233,6 +1711,32 @@ def run_genesis_ledger_command(
},
}
ledger_path = write_json(tmp_path / f"{artifact_stem}-ledger-run-{run_number}.json", ledger)
admission_path: Path | None = None
if admit:
material = json.loads(material_path.read_text(encoding="utf-8"))
ledger_entry = ledger["entries"][0]
admission_path = write_json(
tmp_path / f"{artifact_stem}-admission-run-{run_number}.json",
{
"artifact": rebuild.ADMISSION_ARTIFACT,
"contract_version": rebuild.ADMISSION_CONTRACT_VERSION,
"ledger_sha256": sha256_file(ledger_path),
"reviewed_by_handle": "m3taversal",
"reviewed_by_agent_id": REVIEWER_ID,
"reviewed_at": "2026-07-18T14:00:00+00:00",
"review_note": "Admit this exact source-backed lifecycle fixture.",
"entries": [
{
"sequence": 1,
"proposal_id": material["applied_proposal"]["id"],
"material_sha256": ledger_entry["sha256"],
"replay_material_sha256": ledger_entry["replay_material_sha256"],
"admission_scope": "collective_core",
"basis": "forgejo_primary_source",
}
],
},
)
output = tmp_path / f"{artifact_stem}-reconstruction-receipt-run-{run_number}.json"
command = [
sys.executable,
@ -1258,6 +1762,15 @@ def run_genesis_ledger_command(
"--output",
str(output),
]
if admission_path is not None:
command.extend(
[
"--admission-manifest",
str(admission_path),
"--admission-manifest-sha256",
sha256_file(admission_path),
]
)
completed = subprocess.run(
command,
cwd=rebuild.REPO_ROOT,
@ -1306,6 +1819,45 @@ def test_live_genesis_plus_ledger_command_rebuilds_exactly_and_proves_cleanup(
rebuild.build_applied_timestamp_normalization_sql(entry),
)
worker_args = argparse.Namespace(
container=source_container,
role="kb_apply",
host="127.0.0.1",
db=database,
proposal_id=entry.applied_proposal["id"],
receipt_out=str(tmp_path / "worker-replay-receipt.json"),
receipt_dir=None,
)
worker_proposal = rebuild.apply_engine.load_proposal(worker_args, "")
assert "T" in worker_proposal["reviewed_at"]
assert worker_proposal["reviewed_at"].endswith("Z")
_, worker_receipt = rebuild.apply_engine.capture_replay_receipt(
worker_args,
worker_proposal,
"",
apply_sql=entry.current_apply_sql,
)
transition_rows = rebuild.replay_exporter.fetch_transition_rows(
worker_args,
"",
entry.applied_proposal["id"],
)
exported_material = rebuild.replay_exporter.validate_material(
{
"artifact": rebuild.replay_exporter.MATERIAL_ARTIFACT,
"contract_version": rebuild.replay_exporter.MATERIAL_CONTRACT_VERSION,
"sequence": 1,
"approved_proposal": transition_rows["approved_proposal"],
"approval_snapshot": transition_rows["approval_snapshot"],
"applied_proposal": transition_rows["applied_proposal"],
"replay_receipt": worker_receipt,
},
expected_proposal_id=entry.applied_proposal["id"],
expected_sequence=1,
)
assert exported_material["replay_receipt"]["hashes"] == worker_receipt["hashes"]
assert stat.S_IMODE(Path(worker_args.receipt_out).stat().st_mode) == 0o600
final_manifest = tmp_path / "final-manifest.jsonl"
capture_manifest(source_container, database, final_manifest)
completed, receipt, output = run_genesis_ledger_command(
@ -1468,3 +2020,110 @@ def test_live_revise_strategy_rebuild_is_exact_repeatable_and_leaves_no_containe
first["ledger"]["entries"][0]["replay_material_sha256"]
== second["ledger"]["entries"][0]["replay_material_sha256"]
)
def test_live_v3_exporter_material_genesis_plus_ledger_rebuild_is_exact_and_restores_guard(
tmp_path: Path,
source_postgres: tuple[str, str],
) -> None:
source_container, source_database = source_postgres
database = f"teleo_v3_{uuid.uuid4().hex[:12]}"
docker_psql(
source_container,
"postgres",
f'create database "{database}" template "{source_database}";',
)
try:
genesis_dump, genesis_manifest = capture_source_snapshot(
source_container,
database,
tmp_path,
stem="v3-genesis",
)
docker_psql(source_container, database, "alter role kb_apply nologin;")
docker_psql(source_container, database, rebuild.V3_CONTRACT_PATH.read_text(encoding="utf-8"))
docker_psql(source_container, database, rebuild.GUARD_PREREQUISITES_PATH.read_text(encoding="utf-8"))
material = exporter_shaped_v3_replay_material()
material_path = write_json(tmp_path / "v3-entry-0001.json", material)
entry = rebuild._validate_entry_material(
material,
expected_sequence=1,
expected_replay_hash=material["replay_receipt"]["hashes"]["replay_material_sha256"],
material_path=material_path,
material_sha256=sha256_file(material_path),
)
docker_psql(source_container, database, rebuild.build_seed_sql(entry))
docker_psql(source_container, database, entry.current_apply_sql, role="kb_apply")
docker_psql(
source_container,
database,
rebuild.build_v3_canonical_timestamp_normalization_sql(entry),
)
docker_psql(source_container, database, rebuild.build_applied_timestamp_normalization_sql(entry))
source_readback = json.loads(
docker_psql(
source_container,
database,
rebuild.build_proposal_readback_sql(V3_PROPOSAL_ID),
).stdout.strip()
)
rebuild._assert_exact_proposal_readback(
source_readback,
{
"proposal": material["applied_proposal"],
"approval_snapshot": material["approval_snapshot"],
"approved_proposal_snapshot": material["approved_proposal"],
},
code="fixture_mismatch",
action="exporter-shaped source proposal",
)
assert len(source_readback["approval_snapshot"]) == 8
assert source_readback["approved_proposal_snapshot"]["payload"] == material["approved_proposal"]["payload"]
final_manifest = tmp_path / "v3-final-manifest.jsonl"
capture_manifest(source_container, database, final_manifest)
completed, receipt, output = run_genesis_ledger_command(
tmp_path=tmp_path,
database=database,
genesis_dump=genesis_dump,
genesis_manifest=genesis_manifest,
material_path=material_path,
final_manifest=final_manifest,
artifact_stem="v3",
container_prefix="teleo-genesis-ledger-v3",
run_number=1,
admit=True,
)
assert receipt["status"] == "pass"
assert receipt["genesis_parity"]["status"] == "pass"
assert receipt["v3_contract"]["status"] == "pass"
assert receipt["admission"]["status"] == "review_metadata_supplied"
assert receipt["admission"]["canonical_retrieval_eligible"] is False
assert receipt["admission"]["cryptographically_authenticated"] is False
assert receipt["v3_contract"]["cutover"] == {
"active_apply_sessions": 0,
"apply_role_login_disabled": True,
}
assert receipt["v3_contract"]["catalog"] == {
"assessment_table_present": True,
"apply_role_login_restored": True,
"cutover_guard_enabled": True,
"decision_guard_enabled": True,
}
assert receipt["ledger"]["entries"][0]["contract_version"] == 3
assert receipt["ledger"]["entries"][0]["guarded_apply_executed"] is True
assert receipt["ledger"]["entries"][0]["canonical_rows_exact"] is True
assert receipt["final_parity"]["status"] == "pass"
assert receipt["cleanup"]["container_absent"] is True
assert PRIVATE_MARKER not in completed.stdout
assert PRIVATE_MARKER not in output.read_text(encoding="utf-8")
assert stat.S_IMODE(output.stat().st_mode) == 0o600
finally:
docker_psql(
source_container,
"postgres",
f'drop database if exists "{database}" with (force);',
check=False,
)

442
tests/test_shard_corpus_extraction_work.py Normal file
View file

@ -0,0 +1,442 @@
from __future__ import annotations
import copy
import json
import os
import stat
import subprocess
import sys
from collections.abc import Callable, Iterator
from pathlib import Path
import pytest
from ops import run_local_corpus_knowledge_rebuild as rebuild
from ops import shard_corpus_extraction_work as sharder
REPO_ROOT = Path(__file__).resolve().parents[1]
CLI = REPO_ROOT / "ops" / "shard_corpus_extraction_work.py"
FIXTURE_ROOT = REPO_ROOT / "tests" / "fixtures" / "teleo_corpus_rebuild"
PRIVATE_TEXT = "PRIVATE SOURCE BODY MUST NOT ENTER SHARDS"
def _load_json(path: Path) -> dict:
value = json.loads(path.read_text(encoding="utf-8"))
assert isinstance(value, dict)
return value
def _write_json(path: Path, value: object) -> None:
path.write_bytes(sharder.rendered_json_bytes(value))
path.chmod(0o600)
def _tree_bytes(root: Path) -> dict[str, bytes]:
return {path.relative_to(root).as_posix(): path.read_bytes() for path in sorted(root.rglob("*")) if path.is_file()}
def _all_keys(value: object) -> Iterator[str]:
if isinstance(value, dict):
for key, child in value.items():
yield key
yield from _all_keys(child)
elif isinstance(value, list):
for child in value:
yield from _all_keys(child)
def _build_inventory(tmp_path: Path, source_root: Path = FIXTURE_ROOT, *, name: str = "inventory") -> Path:
output = tmp_path / name
rebuild.build_source_inventory_bundle(source_root, output)
return output
def _make_section_source(root: Path, count: int, *, private_text: str = PRIVATE_TEXT) -> Path:
root.mkdir(parents=True, exist_ok=True)
body = "".join(f"# Section {index:04d}\n\n{private_text} {index:04d}.\n\n" for index in range(count))
source = root / "source.md"
source.write_text(body, encoding="utf-8")
return source
def _run_cli(*args: str) -> subprocess.CompletedProcess[str]:
return subprocess.run(
[sys.executable, str(CLI), *args],
cwd=REPO_ROOT,
check=False,
capture_output=True,
text=True,
)
def _shard_documents(bundle: Path) -> list[tuple[Path, dict]]:
manifest = _load_json(bundle / sharder.MANIFEST_PATH)
return [(bundle / record["path"], _load_json(bundle / record["path"])) for record in manifest["shards"]]
def _rehash_bundle(bundle: Path) -> None:
manifest_path = bundle / sharder.MANIFEST_PATH
receipt_path = bundle / sharder.RECEIPT_PATH
manifest = _load_json(manifest_path)
for record in manifest["shards"]:
shard_path = bundle / record["path"]
document = _load_json(shard_path)
items = document["items"]
work_ids = [item["work_item_id"] for item in items]
document["work_item_count"] = len(items)
document["work_item_ids_sha256"] = sharder.canonical_sha256(work_ids)
_write_json(shard_path, document)
raw = shard_path.read_bytes()
record.update(
bytes=len(raw),
sha256=sharder.sha256_bytes(raw),
work_item_count=len(items),
work_item_ids_sha256=sharder.canonical_sha256(work_ids),
)
_write_json(manifest_path, manifest)
receipt = _load_json(receipt_path)
artifact_paths = [sharder.MANIFEST_PATH, *(record["path"] for record in manifest["shards"])]
artifacts = sharder._artifact_records(bundle, artifact_paths)
receipt["outputs"] = {
"artifact_count": len(artifacts),
"artifacts": artifacts,
"manifest_path": sharder.MANIFEST_PATH,
}
receipt["hashes"] = {
"bundle_tree_sha256": sharder.canonical_sha256(artifacts),
"manifest_file_sha256": sharder.sha256_bytes(manifest_path.read_bytes()),
}
receipt["hashes"]["receipt_content_sha256"] = sharder.canonical_sha256(receipt)
_write_json(receipt_path, receipt)
def test_build_is_deterministic_private_complete_and_source_text_free(tmp_path: Path) -> None:
source_root = tmp_path / "source"
_make_section_source(source_root, 12)
inventory_bundle = _build_inventory(tmp_path, source_root)
first = tmp_path / "shards-a"
second = tmp_path / "shards-b"
first_result = sharder.build_shard_bundle(inventory_bundle, first, 4)["verification"]
second_result = sharder.build_shard_bundle(inventory_bundle, second, 4)["verification"]
assert first_result["status"] == second_result["status"] == "pass"
assert first_result["work_item_count"] == second_result["work_item_count"] == 12
assert first_result["shard_count"] == second_result["shard_count"] == 4
assert first_result["bundle_tree_sha256"] == second_result["bundle_tree_sha256"]
assert _tree_bytes(first) == _tree_bytes(second)
persisted = b"".join(_tree_bytes(first).values())
assert PRIVATE_TEXT.encode() not in persisted
assert str(source_root).encode() not in persisted
assert str(inventory_bundle).encode() not in persisted
assert str(first).encode() not in persisted
forbidden_keys = {"claim", "claims", "proposal", "proposals", "admitted", "database_rows"}
for document in [_load_json(first / sharder.MANIFEST_PATH), _load_json(first / sharder.RECEIPT_PATH)]:
assert forbidden_keys.isdisjoint(_all_keys(document))
for key, expected in sharder.NO_WRITE_FLAGS.items():
assert document[key] is expected
for path in first.rglob("*"):
expected_mode = 0o600 if path.is_file() else 0o700
assert stat.S_IMODE(path.stat().st_mode) == expected_mode
assert stat.S_IMODE(first.stat().st_mode) == 0o700
assert not (first / sharder.INCOMPLETE_MARKER).exists()
def test_hash_assignment_covers_every_exact_inventory_item_once(tmp_path: Path) -> None:
source_root = tmp_path / "source"
_make_section_source(source_root, 40)
inventory_bundle = _build_inventory(tmp_path, source_root)
output = tmp_path / "shards"
sharder.build_shard_bundle(inventory_bundle, output, 7)
inventory_items = _load_json(inventory_bundle / rebuild.EXTRACTION_WORK_INDEX_PATH)["items"]
assigned: list[dict] = []
for index, (_path, document) in enumerate(_shard_documents(output)):
ids = [item["work_item_id"] for item in document["items"]]
assert ids == sorted(ids, key=lambda value: value.encode("utf-8"))
assert all(sharder.assignment_for(work_id, 7) == index for work_id in ids)
assigned.extend(document["items"])
assert len(assigned) == len(inventory_items)
assert {item["work_item_id"] for item in assigned} == {item["work_item_id"] for item in inventory_items}
assert {item["work_item_id"]: item for item in assigned} == {item["work_item_id"]: item for item in inventory_items}
@pytest.mark.parametrize("shard_count", [0, 1025])
def test_shard_count_must_be_within_hard_bounds(tmp_path: Path, shard_count: int) -> None:
inventory_bundle = _build_inventory(tmp_path)
with pytest.raises(sharder.ShardError) as error:
sharder.build_shard_bundle(inventory_bundle, tmp_path / "shards", shard_count)
assert error.value.code == "invalid_shard_count"
assert not (tmp_path / "shards").exists()
def test_shard_count_must_not_exceed_work_item_count(tmp_path: Path) -> None:
inventory_bundle = _build_inventory(tmp_path)
work_count = _load_json(inventory_bundle / rebuild.EXTRACTION_WORK_INDEX_PATH)["work_item_count"]
with pytest.raises(sharder.ShardError) as error:
sharder.build_shard_bundle(inventory_bundle, tmp_path / "shards", work_count + 1)
assert error.value.code == "invalid_shard_count"
def test_existing_output_is_never_overwritten(tmp_path: Path) -> None:
inventory_bundle = _build_inventory(tmp_path)
output = tmp_path / "shards"
output.mkdir(mode=0o700)
sentinel = output / "sentinel"
sentinel.write_text("preserve", encoding="utf-8")
with pytest.raises(sharder.ShardError) as error:
sharder.build_shard_bundle(inventory_bundle, output, 2)
assert error.value.code == "output_exists"
assert sentinel.read_text(encoding="utf-8") == "preserve"
def test_marker_write_failure_closes_and_removes_reserved_output(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
inventory_bundle = _build_inventory(tmp_path)
output = tmp_path / "shards"
original_write = sharder._write_bytes_at
def fail_after_write(directory_fd: int, name: str, payload: bytes) -> None:
original_write(directory_fd, name, payload)
raise OSError("synthetic marker publication failure")
monkeypatch.setattr(sharder, "_write_bytes_at", fail_after_write)
with pytest.raises(OSError, match="synthetic marker publication failure"):
sharder.build_shard_bundle(inventory_bundle, output, 2)
assert not output.exists()
def test_receipt_is_published_while_incomplete_marker_exists(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
inventory_bundle = _build_inventory(tmp_path)
output = tmp_path / "shards"
def inspect(reserved: sharder.ReservedOutput) -> None:
marker = os.stat(sharder.INCOMPLETE_MARKER, dir_fd=reserved.root_fd, follow_symlinks=False)
assert stat.S_ISREG(marker.st_mode)
with pytest.raises(FileNotFoundError):
os.stat(sharder.RECEIPT_PATH, dir_fd=reserved.root_fd, follow_symlinks=False)
monkeypatch.setattr(sharder, "_before_receipt_publication", inspect)
sharder.build_shard_bundle(inventory_bundle, output, 2)
assert not (output / sharder.INCOMPLETE_MARKER).exists()
assert (output / sharder.RECEIPT_PATH).is_file()
def test_destination_swap_does_not_overwrite_or_cleanup_replacement(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
inventory_bundle = _build_inventory(tmp_path)
output = tmp_path / "shards"
moved = tmp_path / "reserved-moved"
def swap(_reserved: sharder.ReservedOutput) -> None:
output.rename(moved)
output.mkdir(mode=0o700)
sentinel = output / "replacement-sentinel"
sentinel.write_text("preserve replacement", encoding="utf-8")
sentinel.chmod(0o600)
monkeypatch.setattr(sharder, "_after_output_reservation", swap)
with pytest.raises(sharder.ShardError) as error:
sharder.build_shard_bundle(inventory_bundle, output, 2)
assert error.value.code == "output_binding_changed"
assert (output / "replacement-sentinel").read_text(encoding="utf-8") == "preserve replacement"
assert moved.is_dir()
assert list(moved.iterdir()) == []
def test_inventory_change_before_publication_rejects_and_cleans_owned_output(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
inventory_bundle = _build_inventory(tmp_path)
output = tmp_path / "shards"
def mutate(root: Path) -> None:
receipt = root / rebuild.RECEIPT_PATH
receipt.write_bytes(receipt.read_bytes() + b" ")
monkeypatch.setattr(sharder, "_before_final_publication", mutate)
with pytest.raises(sharder.ShardError) as error:
sharder.build_shard_bundle(inventory_bundle, output, 2)
assert error.value.code == "inventory_changed"
assert not output.exists()
def _mutate_unknown(bundle: Path) -> None:
for path, document in _shard_documents(bundle):
if document["items"]:
document["items"][0]["work_item_id"] = "f" * 64
document["items"].sort(key=lambda item: item["work_item_id"].encode("utf-8"))
_write_json(path, document)
return
raise AssertionError("fixture unexpectedly produced no shard items")
def _mutate_missing(bundle: Path) -> None:
for path, document in _shard_documents(bundle):
if document["items"]:
document["items"].pop()
_write_json(path, document)
return
raise AssertionError("fixture unexpectedly produced no shard items")
def _mutate_duplicate(bundle: Path) -> None:
for path, document in _shard_documents(bundle):
if document["items"]:
document["items"].append(copy.deepcopy(document["items"][0]))
document["items"].sort(key=lambda item: item["work_item_id"].encode("utf-8"))
_write_json(path, document)
return
raise AssertionError("fixture unexpectedly produced no shard items")
def _mutate_overclaim(bundle: Path) -> None:
receipt_path = bundle / sharder.RECEIPT_PATH
receipt = _load_json(receipt_path)
receipt["database_write_performed"] = True
_write_json(receipt_path, receipt)
@pytest.mark.parametrize(
("mutator", "expected_codes"),
[
(_mutate_unknown, {"unknown_work_item"}),
(_mutate_missing, {"missing_work_item"}),
(_mutate_duplicate, {"duplicate_work_item", "invalid_shard_order"}),
(_mutate_overclaim, {"false_sharding_claim"}),
],
)
def test_verifier_rejects_fully_rehashed_semantic_tampering(
tmp_path: Path,
mutator: Callable[[Path], None],
expected_codes: set[str],
) -> None:
source_root = tmp_path / "source"
_make_section_source(source_root, 20)
inventory_bundle = _build_inventory(tmp_path, source_root)
output = tmp_path / "shards"
sharder.build_shard_bundle(inventory_bundle, output, 4)
mutator(output)
_rehash_bundle(output)
with pytest.raises(sharder.ShardError) as error:
sharder.verify_shard_bundle(inventory_bundle, output)
assert error.value.code in expected_codes
def test_verifier_rejects_wrong_inventory_even_with_same_cardinality(tmp_path: Path) -> None:
first_source = tmp_path / "source-a"
second_source = tmp_path / "source-b"
_make_section_source(first_source, 8, private_text="first corpus")
_make_section_source(second_source, 8, private_text="second corpus")
first_inventory = _build_inventory(tmp_path, first_source, name="inventory-a")
second_inventory = _build_inventory(tmp_path, second_source, name="inventory-b")
output = tmp_path / "shards"
sharder.build_shard_bundle(first_inventory, output, 3)
with pytest.raises(sharder.ShardError) as error:
sharder.verify_shard_bundle(second_inventory, output)
assert error.value.code == "inventory_mismatch"
def test_verifier_rejects_extra_file_and_incomplete_marker(tmp_path: Path) -> None:
inventory_bundle = _build_inventory(tmp_path)
extra_output = tmp_path / "extra-shards"
sharder.build_shard_bundle(inventory_bundle, extra_output, 2)
extra = extra_output / "extra.json"
extra.write_text("{}\n", encoding="utf-8")
extra.chmod(0o600)
with pytest.raises(sharder.ShardError) as extra_error:
sharder.verify_shard_bundle(inventory_bundle, extra_output)
assert extra_error.value.code == "unexpected_bundle_entry"
incomplete_output = tmp_path / "incomplete-shards"
sharder.build_shard_bundle(inventory_bundle, incomplete_output, 2)
marker = incomplete_output / sharder.INCOMPLETE_MARKER
marker.write_text("incomplete\n", encoding="utf-8")
marker.chmod(0o600)
with pytest.raises(sharder.ShardError) as incomplete_error:
sharder.verify_shard_bundle(inventory_bundle, incomplete_output)
assert incomplete_error.value.code == "incomplete_shard_bundle"
def test_verifier_rejects_symlinked_shard(tmp_path: Path) -> None:
inventory_bundle = _build_inventory(tmp_path)
output = tmp_path / "shards"
sharder.build_shard_bundle(inventory_bundle, output, 2)
manifest = _load_json(output / sharder.MANIFEST_PATH)
target = output / manifest["shards"][0]["path"]
retained = target.with_suffix(".retained")
target.rename(retained)
try:
target.symlink_to(retained.name)
except OSError as exc:
pytest.skip(f"symlinks unavailable: {exc}")
with pytest.raises(sharder.ShardError) as error:
sharder.verify_shard_bundle(inventory_bundle, output)
assert error.value.code == "unsafe_bundle_entry"
def test_cli_build_and_verify_return_path_free_receipts(tmp_path: Path) -> None:
inventory_bundle = _build_inventory(tmp_path)
output = tmp_path / "shards"
build = _run_cli(
"--inventory-bundle",
str(inventory_bundle),
"--output-dir",
str(output),
"--shard-count",
"2",
)
verify = _run_cli(
"--inventory-bundle",
str(inventory_bundle),
"--verify-bundle",
str(output),
)
assert build.returncode == verify.returncode == 0, (build.stderr, build.stdout, verify.stderr, verify.stdout)
build_status = json.loads(build.stdout)
verify_status = json.loads(verify.stdout)
assert build_status["status"] == verify_status["status"] == "pass"
assert build_status["bundle_tree_sha256"] == verify_status["bundle_tree_sha256"]
assert build_status["work_item_count"] == verify_status["work_item_count"] == 3
assert str(inventory_bundle) not in build.stdout + verify.stdout
assert str(output) not in build.stdout + verify.stdout
for key, expected in sharder.NO_WRITE_FLAGS.items():
assert build_status[key] is verify_status[key] is expected
def test_cli_rejects_invalid_mode_arguments_without_output(tmp_path: Path) -> None:
inventory_bundle = _build_inventory(tmp_path)
result = _run_cli(
"--inventory-bundle",
str(inventory_bundle),
"--verify-bundle",
str(tmp_path / "missing"),
"--shard-count",
"2",
)
status = json.loads(result.stdout)
assert result.returncode == 2
assert status["status"] == "rejected"
assert status["reason"] == "invalid_arguments"

127
tests/test_stage_normalized_proposal.py
View file

@ -13,6 +13,8 @@ sys.path.insert(0, str(REPO_ROOT / "scripts"))
import stage_normalized_proposal as stage # noqa: E402
from tests import test_compile_kb_source_packet as source_compiler_fixture # noqa: E402
PARENT_ID = "77777777-7777-4777-8777-777777777701"
AGENT_ID = "11111111-1111-4111-8111-111111111111"
DOCUMENT = (
@ -211,6 +213,131 @@ def test_stage_normalized_child_binds_to_full_disposable_container_id(
assert result["database_receipt"]["created"] is True
def test_stage_normalized_child_does_not_require_an_ambient_production_container(
monkeypatch: pytest.MonkeyPatch,
) -> None:
target = {
"id": "clone-container-id",
"name": "composition-clone",
"labels": {stage.bound.DISPOSABLE_LABEL_KEY: stage.bound.DISPOSABLE_LABEL_VALUE},
"network_mode": "none",
"ports": {},
"mount_sources": [],
}
def identity(name: str) -> dict[str, Any]:
if name == "composition-clone":
return target
raise stage.bound.CheckpointError("docker inspect teleo-pg: no such object: teleo-pg")
monkeypatch.setattr(stage.bound, "container_identity", identity)
child = stage.prepare_normalized_child(_parent())
monkeypatch.setattr(
stage.bound,
"_psql_json",
lambda container, database, sql: {
"created": True,
"proposal_id": child["id"],
"proposal_type": child["proposal_type"],
"status": "pending_review",
"source_ref": child["source_ref"],
"payload_sha256": child["payload_sha256"],
},
)
result = stage.stage_normalized_child("composition-clone", "teleo", _parent())
assert result["bound_container_id"] == target["id"]
def test_stage_normalized_child_refuses_unexpected_production_inspection_failure(
monkeypatch: pytest.MonkeyPatch,
) -> None:
target = {
"id": "clone-container-id",
"name": "composition-clone",
"labels": {stage.bound.DISPOSABLE_LABEL_KEY: stage.bound.DISPOSABLE_LABEL_VALUE},
"network_mode": "none",
"ports": {},
"mount_sources": [],
}
def identity(name: str) -> dict[str, Any]:
if name == "composition-clone":
return target
raise stage.bound.CheckpointError("docker inspect teleo-pg: permission denied")
monkeypatch.setattr(stage.bound, "container_identity", identity)
with pytest.raises(stage.bound.CheckpointError, match="permission denied"):
stage.stage_normalized_child("composition-clone", "teleo", _parent())
def test_compiled_v3_child_revalidates_exact_bundle_and_rejects_tampering(tmp_path: Path) -> None:
bundle = source_compiler_fixture._compile_v3(tmp_path / "source")
child = stage.prepare_compiled_v3_child(bundle)
assert child == bundle["strict_child_proposal"]
assert child is not bundle["strict_child_proposal"]
assert child["payload"]["apply_payload"]["contract_version"] == 3
tampered = copy.deepcopy(bundle)
tampered["strict_child_proposal"]["payload"]["apply_payload"]["claims"][0]["proposition"] += " tampered"
with pytest.raises(stage.bound.CheckpointError, match="bundle content hash does not match"):
stage.prepare_compiled_v3_child(tampered)
def test_stage_compiled_v3_bundle_binds_to_disposable_container(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
bundle = source_compiler_fixture._compile_v3(tmp_path / "source")
child = bundle["strict_child_proposal"]
target = {
"id": "clone-container-id",
"name": "composition-clone",
"labels": {stage.bound.DISPOSABLE_LABEL_KEY: stage.bound.DISPOSABLE_LABEL_VALUE},
"network_mode": "none",
"ports": {},
"mount_sources": ["/clone-data"],
}
production = {
"id": "production-container-id",
"name": stage.bound.PRODUCTION_CONTAINER,
"labels": {},
"network_mode": "bridge",
"ports": {},
"mount_sources": ["/production-data"],
}
monkeypatch.setattr(
stage.bound,
"container_identity",
lambda name: target if name == "composition-clone" else production,
)
observed: dict[str, Any] = {}
def fake_psql(container: str, database: str, sql: str) -> dict[str, Any]:
observed.update({"container": container, "database": database, "sql": sql})
return {
"created": True,
"proposal_id": child["id"],
"proposal_type": child["proposal_type"],
"status": "pending_review",
"source_ref": child["source_ref"],
"payload_sha256": child["payload_sha256"],
}
monkeypatch.setattr(stage.bound, "_psql_json", fake_psql)
result = stage.stage_compiled_v3_bundle("composition-clone", "teleo", bundle)
assert observed["container"] == "clone-container-id"
assert observed["database"] == "teleo"
assert "insert into kb_stage.kb_proposals" in observed["sql"].lower()
assert result["child"]["id"] == child["id"]
def test_main_rejects_the_production_container_without_reading_the_packet(tmp_path: Path) -> None:
missing = tmp_path / "missing.json"

799
tests/test_teleo_v3_epistemic_contract.py Normal file
View file

@ -0,0 +1,799 @@
from __future__ import annotations
import json
import re
import subprocess
from pathlib import Path
import pytest
from ops import verify_teleo_v3_epistemic_contract as verifier
REPO_ROOT = Path(__file__).resolve().parents[1]
MIGRATION_SQL = REPO_ROOT / "ops" / "teleo_v3_epistemic_contract.sql"
ROLLBACK_SQL = REPO_ROOT / "ops" / "rollback_teleo_v3_epistemic_contract.sql"
SECRET_STARTUP_MARKER = "secret-startup-marker"
def test_install_migration_is_additive_and_legacy_compatible() -> None:
sql = MIGRATION_SQL.read_text(encoding="utf-8").lower()
assert "add column if not exists" in sql
assert "create table if not exists public.claim_evidence_assessments" in sql
assert "admission_state text not null default 'legacy'" in sql
assert "admission_state = 'legacy'" in sql
assert "admission_state = 'admitted'" in sql
assert "accepted_by_proposal_id" in sql
assert "accepted_from_proposal_id" in sql
assert "teleo_v3_require_linked_insert" in sql
assert (
"create or replace function kb_stage.teleo_v3_require_accepted_proposal()\nreturns trigger\nlanguage plpgsql\nsecurity definer"
in sql
)
assert (
"create or replace function kb_stage.teleo_v3_enforce_apply_contract_cutover()\n"
"returns trigger\nlanguage plpgsql\nsecurity definer" in sql
)
assert "alter function kb_stage.teleo_v3_require_accepted_proposal() owner to kb_gate_owner" in sql
assert "revoke all on function kb_stage.teleo_v3_require_accepted_proposal()" in sql
assert "alter function kb_stage.teleo_v3_enforce_apply_contract_cutover() owner to kb_gate_owner" in sql
assert "revoke all on function kb_stage.teleo_v3_enforce_apply_contract_cutover()" in sql
assert "from public, kb_apply, kb_review" in sql
assert "security definer trigger invariants failed" in sql
assert "kb_gate_owner must be nologin noinherit and unprivileged" in sql
for attribute in (
"not role.rolcanlogin",
"not role.rolinherit",
"not role.rolsuper",
"not role.rolcreatedb",
"not role.rolcreaterole",
"not role.rolreplication",
"not role.rolbypassrls",
):
assert attribute in sql
assert "teleo_v3_reject_delete" in sql
assert "teleo_v3_proposal_apply_receipt_check" in sql
assert "create or replace trigger teleo_v3_claim_immutable" in sql
assert "create or replace trigger teleo_v3_guard_linked_proposal_decision" in sql
assert "'status', 'admission_state'" in sql
assert "'updated_at', 'accepted_by_proposal_id'" in sql
assert "legacy is reserved for rows present at migration" in sql
assert "cutover order: deploy the cutover-aware writer" in sql
assert "alter role kb_apply nologin" in sql
assert "active kb_apply sessions must exit" in sql
assert "retains % pending/approved v2 proposal(s) for explicit migration or rejection" in sql
assert "teleo_v3_00_enforce_apply_contract_cutover" in sql
assert "teleo_v3_enforce_apply_contract_cutover" in sql
assert sql.count("create or replace trigger teleo_v3_") == 24
assert re.search(r"(?m)^\s*create trigger teleo_v3_", sql) is None
assert "actual.tgenabled = 'o'" in sql
assert "actual.tgtype = expected.trigger_type" in sql
assert "actual.update_columns = expected.update_columns" in sql
assert "actual.when_expression is not distinct from expected.when_expression" in sql
assert "actual.argument_hex = expected.argument_hex" in sql
assert "actual.tgconstraint = 0::pg_catalog.oid" in sql
assert "actual.tgoldtable is null" in sql
assert "actual.tgnewtable is null" in sql
assert "actual.tgfoid" in sql
assert "actual.function_security_definer = expected.function_security_definer" in sql
assert "actual.function_source_sha256 = expected.function_source_sha256" in sql
assert "acl.grantee <> procedure.proowner" in sql
assert "protected role membership drift must be removed" in sql
assert "teleo_v3_contract_postflight_query_begin" in sql
assert "alter function %i.%i() owner to kb_gate_owner" in sql
assert "jsonb_typeof(new.payload->'apply_payload'->'contract_version') = 'number'" in sql
assert "migrate or reject it" in sql
assert "for share" in sql
assert "source_content_hash" in sql
assert "locator_json" in sql
assert "relation_type in ('supports', 'challenges', 'depends_on', 'contradicts')" in sql
assert "preexisting v3 schema objects lack installation ownership" in sql
assert "create table if not exists kb_stage.teleo_v3_installation_ownership" in sql
assert "installation_token = 'collision-free-install-v1'" in sql
assert "alter table kb_stage.teleo_v3_installation_ownership owner to kb_gate_owner" in sql
destructive_patterns = (
r"\bdrop\s+",
r"\btruncate\b",
r"\bdelete\s+from\b",
r"\bupdate\s+public\.",
r"\balter\s+type\b",
)
for pattern in destructive_patterns:
assert re.search(pattern, sql) is None, pattern
def test_rollback_refuses_silent_v3_data_loss_before_dropping_contract() -> None:
sql = ROLLBACK_SQL.read_text(encoding="utf-8").lower()
guard_position = sql.index("teleo_v3 rollback refused")
first_drop_position = sql.index("drop trigger")
assert guard_position < first_drop_position
assert "claim_evidence_assessments contains v3 data" in sql
assert "claims contains admitted or proposal-linked v3 data" in sql
assert "drop column if exists proposition" in sql
assert "drop function if exists kb_stage.teleo_v3_require_accepted_proposal" in sql
assert "drop function if exists kb_stage.teleo_v3_require_linked_insert" in sql
assert "drop function if exists kb_stage.teleo_v3_reject_delete" in sql
assert "drop function if exists kb_stage.teleo_v3_acquire_protected_relation_locks" in sql
assert "drop function if exists kb_stage.teleo_v3_enforce_apply_contract_cutover" in sql
assert "drop constraint if exists teleo_v3_proposal_apply_receipt_check" in sql
assert "installation ownership ledger is not exact" in sql
assert "v3-named objects lack exact installation ownership" in sql
assert "drop table if exists kb_stage.teleo_v3_installation_ownership" in sql
@pytest.mark.parametrize(
("logs", "probe_returncode", "probe_stdout", "expected"),
(
(verifier.POSTGRES_READY_LOG_LINE, 0, "1\n", False),
(f"{verifier.POSTGRES_READY_LOG_LINE}\n{verifier.POSTGRES_READY_LOG_LINE}", 2, "", False),
(f"{verifier.POSTGRES_READY_LOG_LINE}\n{verifier.POSTGRES_READY_LOG_LINE}", 0, "", False),
(f"{verifier.POSTGRES_READY_LOG_LINE}\n{verifier.POSTGRES_READY_LOG_LINE}", 0, "1\n", True),
),
)
def test_postgres_final_startup_requires_second_transition_and_target_database(
logs: str,
probe_returncode: int,
probe_stdout: str,
expected: bool,
) -> None:
probe = subprocess.CompletedProcess(["psql"], probe_returncode, stdout=probe_stdout, stderr="")
assert verifier.postgres_final_startup_ready(logs, probe) is expected
def _startup_test_postgres() -> verifier.DisposablePostgres:
postgres = object.__new__(verifier.DisposablePostgres)
postgres.docker = "docker"
postgres.name = "teleo-v3-contract-test"
postgres.password = SECRET_STARTUP_MARKER
postgres.started = True
postgres.cleanup_required = True
postgres.environment = {}
return postgres
@pytest.mark.parametrize("as_bytes", (False, True))
def test_startup_detail_redacts_secret_before_tail_limit(as_bytes: bool) -> None:
postgres = _startup_test_postgres()
postgres.password = "K8VN3PQ7RM2TX6ZC9WB4HJ5DF1GS0LAE8YU3IO2N"
context = "context=database-unavailable|"
trailer = "|stage=target_database_probe|kind=timeout"
suffix_length = 500 - len(context) - len("<redacted>")
suffix = ("x" * (suffix_length - len(trailer))) + trailer
diagnostic = context + postgres.password + suffix
value: str | bytes = diagnostic.encode() if as_bytes else diagnostic
assert len(diagnostic) > 500
assert len(diagnostic) - 500 > len(context)
detail = postgres._redacted_startup_detail(value)
assert detail.startswith(f"{context}<redacted>")
assert detail.endswith(trailer)
assert postgres.password not in detail
assert all(postgres.password[offset : offset + 4] not in detail for offset in range(len(postgres.password) - 3))
def test_docker_run_timeout_is_secret_safe_and_reconciles_owned_side_effect(
monkeypatch: pytest.MonkeyPatch,
) -> None:
postgres = _startup_test_postgres()
postgres.password = "K8VN3PQ7RM2TX6ZC9WB4HJ5DF1GS0LAE8YU3IO2N"
postgres.started = False
postgres.cleanup_required = False
owned_container_id = "a" * 64
owned_container_exists = [True]
calls: list[tuple[list[str], dict[str, str] | None]] = []
context = "context=docker-daemon-accepted-request|"
trailer = "|stage=docker_run|kind=timeout"
suffix_length = 500 - len(context) - len("<redacted>")
timeout_stderr = context + postgres.password + ("x" * (suffix_length - len(trailer))) + trailer
def fake_run(
arguments: list[str],
*,
input_text: str | None = None,
check: bool = True,
timeout: float = 120,
environment: dict[str, str] | None = None,
) -> subprocess.CompletedProcess[str]:
del input_text, check
calls.append((list(arguments), environment))
if arguments[0] == "run":
assert postgres.cleanup_required is True
assert postgres.started is False
assert postgres.password not in " ".join(arguments)
assert environment is not None
assert environment["POSTGRES_PASSWORD"] == postgres.password
secret_bearing_command = ["docker", *arguments, f"adjacent={postgres.password}"]
raise subprocess.TimeoutExpired(
secret_bearing_command,
timeout,
stderr=timeout_stderr,
)
if arguments[:2] == ["container", "ls"] and arguments[-1] == "{{.ID}}":
stdout = f"{owned_container_id}\n" if owned_container_exists[0] else ""
return subprocess.CompletedProcess(arguments, 0, stdout=stdout, stderr="")
if arguments[:2] == ["rm", "--force"]:
assert arguments == ["rm", "--force", owned_container_id]
owned_container_exists[0] = False
return subprocess.CompletedProcess(arguments, 0, stdout=owned_container_id, stderr="")
if arguments[0] == "inspect":
return subprocess.CompletedProcess(
arguments,
0 if owned_container_exists[0] else 1,
stdout="[]" if owned_container_exists[0] else "",
stderr="" if owned_container_exists[0] else "not found",
)
if arguments[:2] == ["container", "ls"] and arguments[-1] == "{{.Names}}":
stdout = f"{postgres.name}\n" if owned_container_exists[0] else ""
return subprocess.CompletedProcess(arguments, 0, stdout=stdout, stderr="")
raise AssertionError(f"unexpected docker arguments: {arguments}")
monkeypatch.setattr(postgres, "run", fake_run)
monkeypatch.setattr(verifier.time, "sleep", lambda _seconds: None)
with pytest.raises(verifier.VerificationError) as exc_info:
postgres.start()
cleanup = postgres.cleanup()
error = str(exc_info.value)
assert "launch state is ambiguous; cleanup_required=true" in error
assert "stage=docker_run kind=timeout timeout_seconds=120" in error
assert error.endswith(trailer)
assert exc_info.value.__suppress_context__ is True
assert postgres.password not in error
assert all(postgres.password[offset : offset + 4] not in error for offset in range(len(postgres.password) - 3))
assert postgres.started is False
assert cleanup["cleanup_reconciliation_attempted"] is True
assert cleanup["remove_attempted"] is True
assert cleanup["remove_returncodes"] == [0]
assert cleanup["owned_container_ids_observed"] == [owned_container_id]
assert cleanup["container_absent"] is True
assert cleanup["name_readback_clear"] is True
assert cleanup["instance_label_readback_clear"] is True
assert cleanup["stable_absence_readback"] is True
ownership_call = next(arguments for arguments, _environment in calls if arguments[-1] == "{{.ID}}")
assert f"name=^/{postgres.name}$" in ownership_call
assert f"label={verifier.CONTAINER_LABEL}" in ownership_call
assert f"label=livingip.canary.instance={postgres.name}" in ownership_call
def test_docker_run_normal_launch_marks_cleanup_before_execution(
monkeypatch: pytest.MonkeyPatch,
) -> None:
postgres = _startup_test_postgres()
postgres.started = False
postgres.cleanup_required = False
container_id = "b" * 64
waited_for_startup: list[bool] = []
def fake_run(
arguments: list[str],
*,
input_text: str | None = None,
check: bool = True,
timeout: float = 120,
environment: dict[str, str] | None = None,
) -> subprocess.CompletedProcess[str]:
del input_text, check, timeout
if arguments[0] == "run":
assert postgres.cleanup_required is True
assert postgres.started is False
assert postgres.password not in " ".join(arguments)
assert environment is not None
assert environment["POSTGRES_PASSWORD"] == postgres.password
return subprocess.CompletedProcess(arguments, 0, stdout=f"{container_id}\n", stderr="")
assert arguments == ["inspect", postgres.name]
inspected = [
{
"Config": {
"Image": verifier.POSTGRES_IMAGE,
"Labels": {
"livingip.canary": "teleo-v3-epistemic-contract",
"livingip.canary.instance": postgres.name,
},
},
"HostConfig": {
"NetworkMode": "none",
"Tmpfs": {"/var/lib/postgresql/data": "rw,nosuid,nodev,size=384m"},
},
"Image": "sha256:postgres-image",
"Mounts": [],
}
]
return subprocess.CompletedProcess(arguments, 0, stdout=json.dumps(inspected), stderr="")
def fake_wait_for_final_startup() -> None:
assert postgres.cleanup_required is True
assert postgres.started is True
waited_for_startup.append(True)
monkeypatch.setattr(postgres, "run", fake_run)
monkeypatch.setattr(postgres, "_wait_for_final_startup", fake_wait_for_final_startup)
environment = postgres.start()
assert waited_for_startup == [True]
assert postgres.started is True
assert postgres.cleanup_required is True
assert environment["container_name"] == postgres.name
assert environment["network_mode"] == "none"
assert environment["tmpfs_data_dir"] is True
assert environment["docker_volume_mounts"] == []
def test_cleanup_never_removes_foreign_exact_name_without_instance_labels(
monkeypatch: pytest.MonkeyPatch,
) -> None:
postgres = _startup_test_postgres()
postgres.started = False
postgres.cleanup_required = True
calls: list[list[str]] = []
def fake_run(
arguments: list[str],
*,
input_text: str | None = None,
check: bool = True,
timeout: float = 120,
environment: dict[str, str] | None = None,
) -> subprocess.CompletedProcess[str]:
del input_text, check, timeout, environment
calls.append(list(arguments))
if arguments[:2] == ["container", "ls"] and arguments[-1] == "{{.ID}}":
return subprocess.CompletedProcess(arguments, 0, stdout="", stderr="")
if arguments[0] == "inspect":
return subprocess.CompletedProcess(arguments, 0, stdout="[]", stderr="")
if arguments[:2] == ["container", "ls"] and arguments[-1] == "{{.Names}}":
has_instance_filter = f"label=livingip.canary.instance={postgres.name}" in arguments
stdout = "" if has_instance_filter else f"{postgres.name}\n"
return subprocess.CompletedProcess(arguments, 0, stdout=stdout, stderr="")
raise AssertionError(f"unexpected docker arguments: {arguments}")
monkeypatch.setattr(postgres, "run", fake_run)
monkeypatch.setattr(verifier.time, "sleep", lambda _seconds: None)
cleanup = postgres.cleanup()
assert not any(arguments[:2] == ["rm", "--force"] for arguments in calls)
assert cleanup["remove_attempted"] is False
assert cleanup["foreign_name_collision_detected"] is True
assert cleanup["container_absent"] is False
assert cleanup["name_readback_clear"] is False
assert cleanup["instance_label_readback_clear"] is True
assert cleanup["stable_absence_readback"] is False
@pytest.mark.parametrize("hung_stage", ("docker_logs", "target_database_probe"))
def test_postgres_startup_commands_are_capped_to_remaining_deadline(
monkeypatch: pytest.MonkeyPatch,
hung_stage: str,
) -> None:
postgres = _startup_test_postgres()
now = [100.0]
observed_timeouts: list[float] = []
ready_logs = f"{verifier.POSTGRES_READY_LOG_LINE}\n{verifier.POSTGRES_READY_LOG_LINE}\n"
def fake_run(
arguments: list[str],
*,
input_text: str | None = None,
check: bool = True,
timeout: float = 120,
) -> subprocess.CompletedProcess[str]:
del input_text, check
assert arguments == ["logs", postgres.name]
if hung_stage == "docker_logs":
observed_timeouts.append(timeout)
now[0] += timeout
raise subprocess.TimeoutExpired(arguments, timeout, stderr=SECRET_STARTUP_MARKER)
return subprocess.CompletedProcess(arguments, 0, stdout=ready_logs, stderr="")
def fake_psql(
sql: str,
*,
check: bool = True,
timeout: float = 120,
) -> subprocess.CompletedProcess[str]:
del check
assert sql == "select 1;"
observed_timeouts.append(timeout)
now[0] += timeout
raise subprocess.TimeoutExpired(["psql"], timeout, stderr=SECRET_STARTUP_MARKER)
monkeypatch.setattr(verifier.time, "monotonic", lambda: now[0])
monkeypatch.setattr(postgres, "run", fake_run)
monkeypatch.setattr(postgres, "psql", fake_psql)
with pytest.raises(verifier.VerificationError) as exc_info:
postgres._wait_for_final_startup(startup_timeout=3.0, probe_timeout=120.0, poll_interval=0)
assert observed_timeouts == [3.0]
assert f"stage={hung_stage} kind=timeout timeout_seconds=3" in str(exc_info.value)
assert SECRET_STARTUP_MARKER not in str(exc_info.value)
@pytest.mark.parametrize(
("failed_stage", "expected_context"),
(
("docker_logs", "stage=docker_logs kind=nonzero returncode=17 detail=daemon <redacted>"),
(
"target_database_probe",
"stage=target_database_probe kind=nonzero returncode=2 detail=database missing <redacted>",
),
),
)
def test_postgres_startup_failure_reports_sanitized_terminal_context(
monkeypatch: pytest.MonkeyPatch,
failed_stage: str,
expected_context: str,
) -> None:
postgres = _startup_test_postgres()
now = [200.0]
ready_logs = f"{verifier.POSTGRES_READY_LOG_LINE}\n{verifier.POSTGRES_READY_LOG_LINE}\n"
def fake_run(
arguments: list[str],
*,
input_text: str | None = None,
check: bool = True,
timeout: float = 120,
) -> subprocess.CompletedProcess[str]:
del input_text, check, timeout
assert arguments == ["logs", postgres.name]
if failed_stage == "docker_logs":
return subprocess.CompletedProcess(
arguments,
17,
stdout="",
stderr=f"daemon {SECRET_STARTUP_MARKER}",
)
return subprocess.CompletedProcess(arguments, 0, stdout=ready_logs, stderr="")
def fake_psql(
sql: str,
*,
check: bool = True,
timeout: float = 120,
) -> subprocess.CompletedProcess[str]:
del check, timeout
assert sql == "select 1;"
return subprocess.CompletedProcess(
["psql"],
2,
stdout="",
stderr=f"database missing {SECRET_STARTUP_MARKER}",
)
def fake_sleep(seconds: float) -> None:
now[0] += seconds
monkeypatch.setattr(verifier.time, "monotonic", lambda: now[0])
monkeypatch.setattr(verifier.time, "sleep", fake_sleep)
monkeypatch.setattr(postgres, "run", fake_run)
monkeypatch.setattr(postgres, "psql", fake_psql)
with pytest.raises(verifier.VerificationError) as exc_info:
postgres._wait_for_final_startup(startup_timeout=0.25, probe_timeout=10.0, poll_interval=0.25)
error = str(exc_info.value)
assert expected_context in error
assert "within 0.25s after 1 attempt(s)" in error
assert SECRET_STARTUP_MARKER not in error
@pytest.mark.skipif(not verifier.docker_available(), reason="Docker daemon is required")
def test_preexisting_v3_named_objects_are_never_adopted_or_rolled_back() -> None:
postgres = verifier.DisposablePostgres()
cleanup: dict[str, object] = {}
try:
postgres.start()
postgres.psql(verifier.BOOTSTRAP_SQL)
postgres.psql(
"""
alter table public.claims
add column owner_agent_id uuid,
add column proposition text,
add column body_markdown text,
add column scope text,
add column access_scope text,
add column admission_state text,
add column revision_criteria text,
add column revision_kind text,
add column supersedes_id uuid,
add column accepted_by_proposal_id uuid;
update public.claims
set owner_agent_id = '11111111-1111-4111-8111-111111111111'::uuid,
proposition = 'claims.proposition',
body_markdown = 'claims.body_markdown',
scope = 'claims.scope',
access_scope = 'claims.access_scope',
admission_state = 'claims.admission_state',
revision_criteria = 'claims.revision_criteria',
revision_kind = 'claims.revision_kind',
supersedes_id = '20000000-0000-4000-8000-000000000002'::uuid,
accepted_by_proposal_id = '30000000-0000-4000-8000-000000000003'::uuid
where id = '20000000-0000-4000-8000-000000000001'::uuid;
alter table public.sources
add column canonical_url text,
add column storage_uri text,
add column content_hash text,
add column access_scope text,
add column ingestion_origin text,
add column provenance_status text,
add column source_class text,
add column auto_promotion_policy text,
add column accepted_by_proposal_id uuid;
update public.sources
set canonical_url = 'sources.canonical_url',
storage_uri = 'sources.storage_uri',
content_hash = 'sources.content_hash',
access_scope = 'sources.access_scope',
ingestion_origin = 'sources.ingestion_origin',
provenance_status = 'sources.provenance_status',
source_class = 'sources.source_class',
auto_promotion_policy = 'sources.auto_promotion_policy',
accepted_by_proposal_id = '30000000-0000-4000-8000-000000000003'::uuid
where id = '20000000-0000-4000-8000-000000000003'::uuid;
alter table public.claim_evidence
add column excerpt text,
add column locator_json jsonb,
add column source_content_hash text,
add column rationale text,
add column polarity text,
add column accepted_from_proposal_id uuid;
update public.claim_evidence
set excerpt = 'claim_evidence.excerpt',
locator_json = '{"sentinel":"claim_evidence.locator_json"}'::jsonb,
source_content_hash = 'claim_evidence.source_content_hash',
rationale = 'claim_evidence.rationale',
polarity = 'claim_evidence.polarity',
accepted_from_proposal_id = '30000000-0000-4000-8000-000000000003'::uuid
where id = '20000000-0000-4000-8000-000000000004'::uuid;
alter table public.claim_edges
add column relation_type text,
add column rationale text,
add column scope_or_conditions text,
add column accepted_by_proposal_id uuid;
update public.claim_edges
set relation_type = 'claim_edges.relation_type',
rationale = 'claim_edges.rationale',
scope_or_conditions = 'claim_edges.scope_or_conditions',
accepted_by_proposal_id = '30000000-0000-4000-8000-000000000003'::uuid
where id = '20000000-0000-4000-8000-000000000005'::uuid;
create table public.claim_evidence_assessments (
id uuid primary key,
legacy_marker text
);
"""
)
state_sql = """
select jsonb_build_object(
'claims', (
select to_jsonb(row_data) from (
select owner_agent_id, proposition, body_markdown, scope, access_scope, admission_state,
revision_criteria, revision_kind, supersedes_id, accepted_by_proposal_id
from public.claims
where id = '20000000-0000-4000-8000-000000000001'::uuid
) row_data
),
'sources', (
select to_jsonb(row_data) from (
select canonical_url, storage_uri, content_hash, access_scope, ingestion_origin,
provenance_status, source_class, auto_promotion_policy, accepted_by_proposal_id
from public.sources
where id = '20000000-0000-4000-8000-000000000003'::uuid
) row_data
),
'claim_evidence', (
select to_jsonb(row_data) from (
select excerpt, locator_json, source_content_hash, rationale, polarity, accepted_from_proposal_id
from public.claim_evidence
where id = '20000000-0000-4000-8000-000000000004'::uuid
) row_data
),
'claim_edges', (
select to_jsonb(row_data) from (
select relation_type, rationale, scope_or_conditions, accepted_by_proposal_id
from public.claim_edges
where id = '20000000-0000-4000-8000-000000000005'::uuid
) row_data
),
'assessment_columns', (
select jsonb_agg(column_name order by ordinal_position)
from information_schema.columns
where table_schema = 'public' and table_name = 'claim_evidence_assessments'
),
'assessment_rows', (select count(*) from public.claim_evidence_assessments),
'ownership_ledger', to_regclass('kb_stage.teleo_v3_installation_ownership')::text
)::text;
"""
before = postgres.psql_json(state_sql)
install = postgres.apply_file(MIGRATION_SQL, check=False)
rollback = postgres.apply_file(ROLLBACK_SQL, check=False)
after = postgres.psql_json(state_sql)
assert install.returncode != 0
assert "preexisting V3 schema objects lack installation ownership" in install.stderr
assert rollback.returncode != 0
assert "V3-named objects lack exact installation ownership" in rollback.stderr
assert before == after
assert set(before["claims"]) == {
"owner_agent_id",
"proposition",
"body_markdown",
"scope",
"access_scope",
"admission_state",
"revision_criteria",
"revision_kind",
"supersedes_id",
"accepted_by_proposal_id",
}
assert set(before["sources"]) == {
"canonical_url",
"storage_uri",
"content_hash",
"access_scope",
"ingestion_origin",
"provenance_status",
"source_class",
"auto_promotion_policy",
"accepted_by_proposal_id",
}
assert set(before["claim_evidence"]) == {
"excerpt",
"locator_json",
"source_content_hash",
"rationale",
"polarity",
"accepted_from_proposal_id",
}
assert set(before["claim_edges"]) == {
"relation_type",
"rationale",
"scope_or_conditions",
"accepted_by_proposal_id",
}
assert all(value is not None for value in before["claims"].values())
assert all(value is not None for value in before["sources"].values())
assert all(value is not None for value in before["claim_evidence"].values())
assert all(value is not None for value in before["claim_edges"].values())
assert before["assessment_columns"] == ["id", "legacy_marker"]
assert before["assessment_rows"] == 0
assert before["ownership_ledger"] is None
finally:
cleanup = postgres.cleanup()
assert cleanup["container_absent"] is True
assert cleanup["name_readback_clear"] is True
assert cleanup["instance_label_readback_clear"] is True
@pytest.mark.skipif(not verifier.docker_available(), reason="Docker daemon is required")
def test_partial_v3_contract_is_never_adopted_or_rolled_back() -> None:
postgres = verifier.DisposablePostgres()
cleanup: dict[str, object] = {}
try:
postgres.start()
postgres.psql(verifier.BOOTSTRAP_SQL)
postgres.psql(
"""
create function kb_stage.teleo_v3_partial_sentinel()
returns text
language sql
immutable
as $$ select 'legacy-partial-sentinel'::text $$;
"""
)
state_sql = """
select jsonb_build_object(
'sentinel_result', kb_stage.teleo_v3_partial_sentinel(),
'sentinel_definition', pg_get_functiondef('kb_stage.teleo_v3_partial_sentinel()'::regprocedure),
'ownership_ledger', to_regclass('kb_stage.teleo_v3_installation_ownership')::text
)::text;
"""
before = postgres.psql_json(state_sql)
install = postgres.apply_file(MIGRATION_SQL, check=False)
rollback = postgres.apply_file(ROLLBACK_SQL, check=False)
after = postgres.psql_json(state_sql)
assert install.returncode != 0
assert "partial or altered V3 contract lacks exact installation ownership" in install.stderr
assert rollback.returncode != 0
assert "V3-named objects lack exact installation ownership" in rollback.stderr
assert before == after
assert before["sentinel_result"] == "legacy-partial-sentinel"
assert "teleo_v3_partial_sentinel" in before["sentinel_definition"]
assert before["ownership_ledger"] is None
finally:
cleanup = postgres.cleanup()
assert cleanup["container_absent"] is True
assert cleanup["name_readback_clear"] is True
assert cleanup["instance_label_readback_clear"] is True
def test_verifier_has_no_external_database_target_surface(tmp_path: Path) -> None:
args = verifier.parse_args(["--output", str(tmp_path / "receipt.json")])
assert args.output == tmp_path / "receipt.json"
source = Path(verifier.__file__).read_text(encoding="utf-8")
assert "DATABASE_URL" not in source
assert "PGHOST" not in source
assert '"--network",\n "none"' in source
assert '"--tmpfs"' in source
assert 'production_access_attempted": False' in source
def test_failure_receipt_requires_exact_sqlstate() -> None:
completed = subprocess.CompletedProcess(
args=["psql"],
returncode=3,
stdout="",
stderr="ERROR: 23514: expected constraint failure\n",
)
receipt = verifier.failure_receipt(completed, "23514")
assert receipt["refused"] is True
assert receipt["observed_sqlstate"] == "23514"
assert receipt["expected_sqlstate_observed"] is True
def test_verifier_exercises_concurrent_proposal_downgrade_lock() -> None:
source = Path(verifier.__file__).read_text(encoding="utf-8")
assert "proposal_decision_lock_race_check" in source
assert "teleo_v3_proposal_lock_holder" in source
assert "set statement_timeout = '750ms'" in source
assert 'failure_receipt(downgrade, "57014")' in source
def test_verifier_asserts_security_definer_owner_search_path_and_acl() -> None:
source = Path(verifier.__file__).read_text(encoding="utf-8")
assert '"owner_name": "kb_gate_owner"' in source
assert '"security_definer": True' in source
assert '"config": ["search_path=pg_catalog, pg_temp"]' in source
assert '"owner_only_execute": True' in source
def test_verifier_cleanup_bypass_is_bounded_to_disposable_superuser_transaction() -> None:
source = Path(verifier.__file__).read_text(encoding="utf-8")
assert "set local session_replication_role = replica" in source
assert '"isolated-disposable-superuser-transaction"' in source
assert "Production writers must never receive" in source
@pytest.mark.skipif(not verifier.docker_available(), reason="Docker daemon is required")
def test_disposable_postgres_contract_end_to_end(tmp_path: Path) -> None:
output = tmp_path / "teleo-v3-epistemic-contract-receipt.json"
receipt = verifier.run_verification(output)
assert receipt["status"] == "pass", receipt.get("error")
assert receipt["required_tier"] == "T2_runtime"
assert receipt["current_tier"] == "T2_runtime_disposable_postgresql"
assert receipt["production_access_attempted"] is False
assert receipt["production_migration_shipped"] is False
assert receipt["environment"]["network_mode"] == "none"
assert receipt["environment"]["tmpfs_data_dir"] is True
assert receipt["environment"]["docker_volume_mounts"] == []
assert len(receipt["checks"]) >= 20
assert all(receipt["checks"].values()), receipt["checks"]
assert all(item["refused"] for item in receipt["adversarial"].values())
assert all(item["expected_sqlstate_observed"] for item in receipt["adversarial"].values())
assert receipt["rollback_with_data"]["refused"] is True
assert receipt["cleanup"]["container_absent"] is True
assert receipt["cleanup"]["instance_label_readback_clear"] is True
assert json.loads(output.read_text(encoding="utf-8")) == receipt

467
tests/test_v3_apply_catalog_serialization_postgres.py Normal file
View file

@ -0,0 +1,467 @@
from __future__ import annotations
import subprocess
import time
import pytest
from ops import verify_teleo_v3_epistemic_contract as contract
from tests import test_v3_apply_worker_postgres as worker
CATALOG_DDL_ADVISORY_LOCK_KEY = 6072343533146485505
TEST_APPLY_BARRIER_LOCK_KEY = 6072343533146485599
def _start_psql(
postgres: contract.DisposablePostgres,
sql: str,
*,
role: str = "postgres",
password: str | None = None,
keep_stdin_open: bool = False,
) -> subprocess.Popen[str]:
assert postgres.docker is not None
command = [postgres.docker, "exec"]
if password is not None:
command.extend(["-e", f"PGPASSWORD={password}"])
command.extend(
[
"-i",
postgres.name,
"psql",
"--no-psqlrc",
"--set=ON_ERROR_STOP=1",
"--set=VERBOSITY=verbose",
"--username",
role,
f"--dbname={contract.DATABASE}",
]
)
if password is not None:
command.extend(["--host=127.0.0.1"])
process = subprocess.Popen(
command,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
)
assert process.stdin is not None
process.stdin.write(sql)
if keep_stdin_open:
process.stdin.flush()
else:
process.stdin.close()
process.stdin = None
return process
def _finish(process: subprocess.Popen[str], *, timeout: float = 20) -> subprocess.CompletedProcess[str]:
stdout, stderr = process.communicate(timeout=timeout)
return subprocess.CompletedProcess(process.args, process.returncode, stdout, stderr)
def _stop(process: subprocess.Popen[str] | None) -> None:
if process is None or process.poll() is not None:
return
process.terminate()
try:
process.communicate(timeout=3)
except subprocess.TimeoutExpired:
process.kill()
process.communicate(timeout=3)
def _wait_for_activity(
postgres: contract.DisposablePostgres,
application_name: str,
*,
wait_event_type: str,
wait_event: str,
timeout: float = 8,
) -> None:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
waiting = postgres.psql(
f"""
select exists (
select 1
from pg_catalog.pg_stat_activity
where application_name = '{application_name}'
and wait_event_type = '{wait_event_type}'
and wait_event = '{wait_event}'
);
"""
).stdout.strip()
if waiting == "t":
return
time.sleep(0.05)
raise AssertionError(f"{application_name} did not wait on {wait_event_type}/{wait_event}")
def _approved_apply_sql(
postgres: contract.DisposablePostgres,
ids: dict[str, str],
) -> str:
payload = worker._payload(ids)
packet = worker._insert_pending_proposal(postgres, payload, ids)
worker._approve_proposal(postgres, ids["proposal"], packet)
proposal = postgres.psql_json(
f"""
select pg_catalog.jsonb_build_object(
'id', id::text,
'proposal_type', proposal_type,
'status', status,
'payload', payload,
'reviewed_by_handle', reviewed_by_handle,
'reviewed_by_agent_id', reviewed_by_agent_id::text,
'reviewed_at', pg_catalog.to_char(
reviewed_at at time zone 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"'
),
'review_note', review_note,
'applied_by_handle', applied_by_handle,
'applied_by_agent_id', applied_by_agent_id::text,
'applied_at', applied_at
)::text
from kb_stage.kb_proposals
where id = '{ids["proposal"]}'::uuid;
"""
)
return worker.apply.build_apply_sql(proposal, worker.apply.SERVICE_AGENT_HANDLE)
def _instrument_apply_after_catalog_guard(sql: str, application_name: str) -> str:
guard_end = "$v3_contract_guard$;"
assert sql.count(guard_end) == 1
return sql.replace(
"begin;\n",
f"begin;\nset local application_name = '{application_name}';\n",
1,
).replace(
guard_end,
guard_end + f"\nselect pg_catalog.pg_advisory_xact_lock({TEST_APPLY_BARRIER_LOCK_KEY});",
1,
)
def _instrument_apply_before_transaction(sql: str, application_name: str) -> str:
return f"set application_name = '{application_name}';\n{sql}"
def _start_barrier_holder(postgres: contract.DisposablePostgres) -> subprocess.Popen[str]:
holder = _start_psql(
postgres,
f"""
set application_name = 'teleo_v3_catalog_barrier_holder';
select pg_catalog.pg_advisory_lock({TEST_APPLY_BARRIER_LOCK_KEY});
""",
keep_stdin_open=True,
)
_wait_for_activity(
postgres,
"teleo_v3_catalog_barrier_holder",
wait_event_type="Client",
wait_event="ClientRead",
)
return holder
def _release_barrier(
postgres: contract.DisposablePostgres,
holder: subprocess.Popen[str],
) -> None:
released = postgres.psql(
"""
select pg_catalog.pg_terminate_backend(pid)
from pg_catalog.pg_stat_activity
where application_name = 'teleo_v3_catalog_barrier_holder';
"""
)
assert released.stdout.strip() == "t"
if holder.stdin is not None:
holder.stdin.close()
holder.stdin = None
_finish(holder, timeout=5)
def _install_and_build_apply(
postgres: contract.DisposablePostgres,
ids: dict[str, str],
) -> str:
environment = postgres.start()
assert environment["network_mode"] == "none"
postgres.psql(contract.BOOTSTRAP_SQL)
postgres.psql(worker._supplemental_bootstrap())
postgres.apply_file(worker.PREREQUISITES)
worker._install_v3_contract(postgres)
return _approved_apply_sql(postgres, ids)
@pytest.mark.skipif(not contract.docker_available(), reason="Docker daemon is required")
def test_trigger_ddl_waits_for_apply_transaction_relation_locks() -> None:
postgres = contract.DisposablePostgres()
holder: subprocess.Popen[str] | None = None
apply_process: subprocess.Popen[str] | None = None
ddl_process: subprocess.Popen[str] | None = None
try:
apply_sql = _install_and_build_apply(postgres, worker.RACE_IDS)
holder = _start_barrier_holder(postgres)
apply_process = _start_psql(
postgres,
_instrument_apply_after_catalog_guard(apply_sql, "teleo_v3_guarded_apply"),
role="kb_apply",
password=worker.APPLY_PASSWORD,
)
_wait_for_activity(
postgres,
"teleo_v3_guarded_apply",
wait_event_type="Lock",
wait_event="advisory",
)
ddl_process = _start_psql(
postgres,
"""
set application_name = 'teleo_v3_uncoordinated_trigger_ddl';
create or replace trigger teleo_v3_00_enforce_apply_contract_cutover
before insert or update of proposal_type, payload, status on kb_stage.kb_proposals
for each row when (false)
execute function kb_stage.teleo_v3_enforce_apply_contract_cutover();
""",
)
_wait_for_activity(
postgres,
"teleo_v3_uncoordinated_trigger_ddl",
wait_event_type="Lock",
wait_event="relation",
)
assert ddl_process.poll() is None
assert worker._contract_state(postgres) == "V3"
_release_barrier(postgres, holder)
holder = None
apply_result = _finish(apply_process)
apply_process = None
assert apply_result.returncode == 0, apply_result.stderr
ddl_result = _finish(ddl_process)
ddl_process = None
assert ddl_result.returncode == 0, ddl_result.stderr
assert postgres.psql_json(
f"""
select pg_catalog.jsonb_build_object(
'proposal_status', (
select status from kb_stage.kb_proposals
where id = '{worker.RACE_IDS["proposal"]}'::uuid
),
'contract_state', ({worker.apply.build_v3_contract_state_query()})
)::text;
"""
) == {"proposal_status": "applied", "contract_state": "INVALID"}
finally:
if holder is not None and holder.poll() is None:
_release_barrier(postgres, holder)
_stop(ddl_process)
_stop(apply_process)
_stop(holder)
cleanup = postgres.cleanup()
assert cleanup["container_absent"] is True
assert cleanup["instance_label_readback_clear"] is True
@pytest.mark.skipif(not contract.docker_available(), reason="Docker daemon is required")
def test_controlled_function_migration_waits_for_apply_shared_advisory_lock() -> None:
postgres = contract.DisposablePostgres()
holder: subprocess.Popen[str] | None = None
apply_process: subprocess.Popen[str] | None = None
migration_process: subprocess.Popen[str] | None = None
try:
apply_sql = _install_and_build_apply(postgres, worker.RACE_IDS)
holder = _start_barrier_holder(postgres)
apply_process = _start_psql(
postgres,
_instrument_apply_after_catalog_guard(apply_sql, "teleo_v3_shared_lock_apply"),
role="kb_apply",
password=worker.APPLY_PASSWORD,
)
_wait_for_activity(
postgres,
"teleo_v3_shared_lock_apply",
wait_event_type="Lock",
wait_event="advisory",
)
migration_process = _start_psql(
postgres,
f"""
begin;
set local application_name = 'teleo_v3_controlled_function_migration';
select pg_catalog.pg_advisory_xact_lock({CATALOG_DDL_ADVISORY_LOCK_KEY});
create or replace function kb_stage.teleo_v3_validate_claim_edge()
returns trigger
language plpgsql
set search_path = pg_catalog, pg_temp
as $function$
begin
return new;
end
$function$;
commit;
""",
)
_wait_for_activity(
postgres,
"teleo_v3_controlled_function_migration",
wait_event_type="Lock",
wait_event="advisory",
)
assert migration_process.poll() is None
assert worker._contract_state(postgres) == "V3"
_release_barrier(postgres, holder)
holder = None
apply_result = _finish(apply_process)
apply_process = None
assert apply_result.returncode == 0, apply_result.stderr
migration_result = _finish(migration_process)
migration_process = None
assert migration_result.returncode == 0, migration_result.stderr
assert worker._contract_state(postgres) == "INVALID"
finally:
if holder is not None and holder.poll() is None:
_release_barrier(postgres, holder)
_stop(migration_process)
_stop(apply_process)
_stop(holder)
cleanup = postgres.cleanup()
assert cleanup["container_absent"] is True
assert cleanup["instance_label_readback_clear"] is True
@pytest.mark.skipif(not contract.docker_available(), reason="Docker daemon is required")
def test_migration_first_apply_resnapshots_and_refuses_post_migration_catalog_drift() -> None:
postgres = contract.DisposablePostgres()
holder: subprocess.Popen[str] | None = None
apply_process: subprocess.Popen[str] | None = None
migration_process: subprocess.Popen[str] | None = None
try:
apply_sql = _install_and_build_apply(postgres, worker.RACE_IDS)
holder = _start_barrier_holder(postgres)
migration_process = _start_psql(
postgres,
f"""
begin;
set local application_name = 'teleo_v3_migration_first_holder';
select pg_catalog.pg_advisory_xact_lock({CATALOG_DDL_ADVISORY_LOCK_KEY});
select pg_catalog.pg_advisory_xact_lock({TEST_APPLY_BARRIER_LOCK_KEY});
create or replace function kb_stage.teleo_v3_validate_claim_edge()
returns trigger
language plpgsql
set search_path = pg_catalog, pg_temp
as $function$
begin
return new;
end
$function$;
commit;
""",
)
_wait_for_activity(
postgres,
"teleo_v3_migration_first_holder",
wait_event_type="Lock",
wait_event="advisory",
)
apply_process = _start_psql(
postgres,
_instrument_apply_before_transaction(apply_sql, "teleo_v3_migration_first_apply"),
role="kb_apply",
password=worker.APPLY_PASSWORD,
)
_wait_for_activity(
postgres,
"teleo_v3_migration_first_apply",
wait_event_type="Lock",
wait_event="advisory",
)
_release_barrier(postgres, holder)
holder = None
migration_result = _finish(migration_process)
migration_process = None
assert migration_result.returncode == 0, migration_result.stderr
apply_result = _finish(apply_process)
apply_process = None
assert apply_result.returncode != 0
assert "V3 catalog contract is INVALID" in apply_result.stderr
assert postgres.psql_json(
f"""
select pg_catalog.jsonb_build_object(
'proposal_status', (
select status from kb_stage.kb_proposals
where id = '{worker.RACE_IDS["proposal"]}'::uuid
),
'applied_at', (
select applied_at from kb_stage.kb_proposals
where id = '{worker.RACE_IDS["proposal"]}'::uuid
),
'claims', (
select count(*) from public.claims
where accepted_by_proposal_id = '{worker.RACE_IDS["proposal"]}'::uuid
),
'sources', (
select count(*) from public.sources
where accepted_by_proposal_id = '{worker.RACE_IDS["proposal"]}'::uuid
),
'evidence', (
select count(*) from public.claim_evidence
where accepted_from_proposal_id = '{worker.RACE_IDS["proposal"]}'::uuid
),
'edges', (
select count(*) from public.claim_edges
where accepted_by_proposal_id = '{worker.RACE_IDS["proposal"]}'::uuid
),
'assessments', (
select count(*) from public.claim_evidence_assessments
where accepted_by_proposal_id = '{worker.RACE_IDS["proposal"]}'::uuid
),
'contract_state', ({worker.apply.build_v3_contract_state_query()})
)::text;
"""
) == {
"proposal_status": "approved",
"applied_at": None,
"claims": 0,
"sources": 0,
"evidence": 0,
"edges": 0,
"assessments": 0,
"contract_state": "INVALID",
}
lock_cleanup = postgres.psql_json(
f"""
with acquired as (
select pg_catalog.pg_try_advisory_lock({CATALOG_DDL_ADVISORY_LOCK_KEY}) as value
), released as (
select pg_catalog.pg_advisory_unlock({CATALOG_DDL_ADVISORY_LOCK_KEY}) as value
from acquired where value
)
select pg_catalog.jsonb_build_object(
'exclusive_lock_acquired', (select value from acquired),
'exclusive_lock_released', coalesce((select value from released), false)
)::text;
"""
)
assert lock_cleanup == {
"exclusive_lock_acquired": True,
"exclusive_lock_released": True,
}
finally:
if holder is not None and holder.poll() is None:
_release_barrier(postgres, holder)
_stop(migration_process)
_stop(apply_process)
_stop(holder)
cleanup = postgres.cleanup()
assert cleanup["container_absent"] is True
assert cleanup["instance_label_readback_clear"] is True

1545
tests/test_v3_apply_worker_postgres.py Normal file
View file

File diff suppressed because it is too large Load diff

746
tests/test_v3_source_rebuild_lifecycle_postgres.py Normal file
View file

@ -0,0 +1,746 @@
from __future__ import annotations
import hashlib
import json
import secrets
import stat
import subprocess
import sys
import time
import uuid
from collections.abc import Iterator
from pathlib import Path
import pytest
from ops import run_local_corpus_knowledge_rebuild as corpus_rebuild
from ops import run_local_genesis_ledger_rebuild as genesis_rebuild
from ops import verify_teleo_v3_epistemic_contract as v3_contract
from scripts import compile_kb_source_packet as compiler
from tests import test_run_local_genesis_ledger_rebuild as genesis_test
from tests import test_v3_apply_worker_postgres as worker_test
from tests.docker_volume_lifecycle import (
POSTGRES_TMPFS_SPEC,
acquire_docker_volume_set_guard,
inspect_postgres_tmpfs_container,
release_docker_volume_set_guard,
remove_container_and_volumes,
)
REPO_ROOT = Path(__file__).resolve().parents[1]
CORPUS_REBUILD = REPO_ROOT / "ops" / "run_local_corpus_knowledge_rebuild.py"
SOURCE_COMPILER = REPO_ROOT / "scripts" / "compile_kb_source_packet.py"
PROPOSAL_STAGER = REPO_ROOT / "scripts" / "stage_normalized_proposal.py"
REVIEW_PACKET = REPO_ROOT / "scripts" / "kb_proposal_review_packet.py"
APPROVER = REPO_ROOT / "scripts" / "approve_proposal.py"
APPLY_WORKER = REPO_ROOT / "scripts" / "apply_worker.py"
MATERIAL_EXPORTER = REPO_ROOT / "scripts" / "export_kb_transition_replay_material.py"
LEDGER_BUILDER = REPO_ROOT / "ops" / "build_v3_genesis_ledger_bundle.py"
GENESIS_REBUILDER = REPO_ROOT / "ops" / "run_local_genesis_ledger_rebuild.py"
SOURCE_CONTAINER_LABEL = "livingip.test.v3-source-rebuild-lifecycle"
STAGING_LABEL = "com.livingip.leo.checkpoint=disposable"
DATABASE = "teleo_v3_source_lifecycle"
SYNTHETIC_REVIEW_NOTE = "SYNTHETIC TEST ONLY: operator-supplied fixture approval; this is not human review proof."
def _sha256(path: Path) -> str:
return hashlib.sha256(path.read_bytes()).hexdigest()
def _run(command: list[str], *, timeout: int = 240) -> subprocess.CompletedProcess[str]:
completed = subprocess.run(
command,
cwd=REPO_ROOT,
text=True,
capture_output=True,
check=False,
timeout=timeout,
)
assert completed.returncode == 0, (
f"command failed ({completed.returncode}): {command[0]} {Path(command[1]).name if len(command) > 1 else ''}\n"
f"stdout:\n{completed.stdout}\nstderr:\n{completed.stderr}"
)
return completed
def _docker(*arguments: str, check: bool = True, timeout: int = 120) -> subprocess.CompletedProcess[str]:
completed = subprocess.run(
["docker", *arguments],
text=True,
capture_output=True,
check=False,
timeout=timeout,
)
if check:
assert completed.returncode == 0, completed.stderr or completed.stdout
return completed
def _write_private(path: Path, content: str | bytes) -> Path:
path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
path.parent.chmod(0o700)
if isinstance(content, bytes):
path.write_bytes(content)
else:
path.write_text(content, encoding="utf-8")
path.chmod(0o600)
return path
def _start_postgres(name: str, database: str, run_label: str, *, staging_target: bool) -> str:
command = [
"run",
"--detach",
"--network",
"none",
"--name",
name,
"--label",
f"{SOURCE_CONTAINER_LABEL}={run_label}",
"--tmpfs",
POSTGRES_TMPFS_SPEC,
"--env",
f"POSTGRES_DB={database}",
"--env",
"POSTGRES_HOST_AUTH_METHOD=trust",
]
if staging_target:
command.extend(("--label", STAGING_LABEL))
command.append(v3_contract.POSTGRES_IMAGE)
try:
container_id = _docker(*command).stdout.strip()
assert container_id
for _ in range(180):
ready = _docker(
"exec",
name,
"pg_isready",
"-U",
"postgres",
"-h",
"127.0.0.1",
"-d",
database,
check=False,
timeout=10,
)
if ready.returncode == 0:
break
time.sleep(0.25)
else:
pytest.fail(f"disposable PostgreSQL did not become ready: {name}")
inspected = inspect_postgres_tmpfs_container(name)
assert inspected["HostConfig"]["NetworkMode"] == "none"
assert inspected["NetworkSettings"]["Ports"] == {}
assert inspected["Config"]["Labels"][SOURCE_CONTAINER_LABEL] == run_label
if staging_target:
assert inspected["Config"]["Labels"]["com.livingip.leo.checkpoint"] == "disposable"
return container_id
except BaseException:
remove_container_and_volumes(name)
raise
def _remove_container(name: str) -> None:
remove_container_and_volumes(name)
def _assert_container_absent(name: str) -> None:
assert _docker("inspect", name, check=False).returncode != 0
names = _docker(
"container",
"ls",
"--all",
"--filter",
f"name=^/{name}$",
"--format",
"{{.Names}}",
)
assert names.stdout.strip() == ""
def _capture_dump(container: str, database: str, destination: Path) -> None:
remote = f"/tmp/{destination.name}"
_docker(
"exec",
container,
"pg_dump",
"-U",
"postgres",
"-d",
database,
"--format=custom",
f"--file={remote}",
)
_docker("cp", f"{container}:{remote}", str(destination))
destination.chmod(0o600)
def _capture_manifest(container: str, database: str, destination: Path) -> None:
genesis_test.capture_manifest(container, database, destination)
destination.chmod(0o600)
def _json_stdout(completed: subprocess.CompletedProcess[str]) -> dict:
value = json.loads(completed.stdout)
assert isinstance(value, dict)
return value
def _assert_private_files(root: Path) -> None:
files = [path for path in root.rglob("*") if path.is_file()]
assert files
for path in files:
assert stat.S_IMODE(path.stat().st_mode) == 0o600, path
@pytest.fixture
def unchanged_docker_volume_set() -> Iterator[None]:
volume_guard = acquire_docker_volume_set_guard(
context="v3 source rebuild lifecycle PostgreSQL fixture",
)
try:
yield
finally:
release_docker_volume_set_guard(volume_guard)
@pytest.mark.skipif(not v3_contract.docker_available(), reason="Docker daemon is required")
def test_start_postgres_cleans_up_when_post_start_validation_fails(
monkeypatch: pytest.MonkeyPatch,
unchanged_docker_volume_set: None,
) -> None:
if _docker("image", "inspect", v3_contract.POSTGRES_IMAGE, check=False).returncode != 0:
pytest.skip("the pinned PostgreSQL image must already exist; this test never pulls over the network")
run_id = uuid.uuid4().hex[:12]
container = f"teleo-v3-startup-failure-{run_id}"
def fail_inspection(_name: str) -> dict:
raise AssertionError("forced post-start validation failure")
monkeypatch.setattr(sys.modules[__name__], "inspect_postgres_tmpfs_container", fail_inspection)
try:
with pytest.raises(AssertionError, match="forced post-start validation failure"):
_start_postgres(container, DATABASE, run_id, staging_target=True)
_assert_container_absent(container)
finally:
remove_container_and_volumes(container)
@pytest.mark.skipif(not v3_contract.docker_available(), reason="Docker daemon is required")
def test_v3_source_rebuild_lifecycle_is_exact_networkless_and_cleanup_proven(
tmp_path: Path,
unchanged_docker_volume_set: None,
) -> None:
if _docker("image", "inspect", v3_contract.POSTGRES_IMAGE, check=False).returncode != 0:
pytest.skip("the pinned PostgreSQL image must already exist; this test never pulls over the network")
run_id = uuid.uuid4().hex[:12]
source_container = f"teleo-v3-source-{run_id}"
rebuild_prefix = f"teleo-v3-rebuild-{run_id[:8]}"
private_root = tmp_path / "private"
private_root.mkdir(mode=0o700)
password = secrets.token_hex(24)
source_started = False
rebuild_container: str | None = None
try:
corpus_root = private_root / "corpus"
source_text = (
"# Canonical review boundary\n\n"
"A staged knowledge proposal remains non-canonical until explicit review and guarded apply both succeed.\n"
)
source_path = _write_private(corpus_root / "review-boundary.md", source_text)
inventory = private_root / "source-inventory"
built_inventory = _run(
[
sys.executable,
str(CORPUS_REBUILD),
"--source-root",
str(corpus_root),
"--output",
str(inventory),
"--mode",
"source-inventory-only",
]
)
inventory_status = _json_stdout(built_inventory)
assert inventory_status["status"] == "pass"
assert inventory_status["source_count"] == 1
verified_inventory = _run(
[
sys.executable,
str(CORPUS_REBUILD),
"--verify-bundle",
str(inventory),
]
)
assert _json_stdout(verified_inventory)["status"] == "pass"
source_manifest = json.loads((inventory / corpus_rebuild.SOURCE_MANIFEST_PATH).read_text(encoding="utf-8"))
assert len(source_manifest["sources"]) == 1
source_row = source_manifest["sources"][0]
snapshot = inventory / source_row["snapshot_path"]
assert snapshot.read_bytes() == source_path.read_bytes()
assert source_row["artifact_sha256"] == _sha256(snapshot)
assert snapshot.name == f"{source_row['artifact_sha256']}.utf8"
quote = (
"A staged knowledge proposal remains non-canonical until explicit review and guarded apply both succeed."
)
text_bytes = snapshot.read_bytes()
quote_bytes = quote.encode("utf-8")
quote_start = text_bytes.index(quote_bytes)
artifact_sha256 = _sha256(snapshot)
extractor_spec_bytes = (
b"livingip synthetic lifecycle extractor specification\n"
b"name=synthetic-v3-source-lifecycle\n"
b"version=1.0.0\n"
b"quotes=exact-utf8-byte-locators\n"
b"approval=test-operator-supplied-not-human-proof\n"
)
extractor_spec = _write_private(private_root / "extractor.spec", extractor_spec_bytes)
extraction_manifest = {
"schema": compiler.MANIFEST_SCHEMA_V3,
"artifact_sha256": artifact_sha256,
"extracted_text_sha256": artifact_sha256,
"extractor": {
"name": "synthetic-v3-source-lifecycle",
"version": "1.0.0",
"spec_sha256": hashlib.sha256(extractor_spec_bytes).hexdigest(),
},
"agent_id": v3_contract.AGENT_ID,
"source": {
"identity": f"document:sha256:{artifact_sha256}",
"source_key": "canonical_review_boundary",
"source_type": "article",
"title": "Canonical review boundary",
"locator": f"artifact://sha256/{artifact_sha256}",
"access_scope": "collective_internal",
"ingestion_origin": "operator_upload",
"provenance_status": "verified",
"source_class": "primary_record",
"auto_promotion_policy": "human_required",
"captured_at": "2026-07-19T00:00:00+00:00",
},
"claims": [
{
"claim_key": "review_and_apply_precede_canonical_state",
"type": "empirical",
"proposition": (
"A staged knowledge proposal is not canonical until explicit review and guarded apply succeed."
),
"body_markdown": "The exact retained source states both gates and their order.",
"scope": "The exact content-addressed source snapshot used by this disposable lifecycle.",
"access_scope": "collective_internal",
"owner_agent_id": None,
"status": "active",
"revision_criteria": "Revise if the exact source bytes or guarded apply contract changes.",
"revision_kind": "original",
"supersedes_id": None,
"tags": ["genesis", "provenance", "review"],
"evidence": [
{
"evidence_key": "exact_review_boundary_sentence",
"quote": quote,
"polarity": "supports",
"locator": {
"unit": "utf8_bytes",
"start": quote_start,
"end": quote_start + len(quote_bytes),
},
"rationale": "The exact byte-bound sentence directly supports the bounded proposition.",
}
],
"assessment": {
"support_tier": "strong",
"challenge_tier": "none",
"overall_state": "support_dominant",
"rationale": "One exact verified primary-record sentence directly supports the claim.",
"supersedes_assessment_id": None,
},
}
],
}
manifest_path = _write_private(
private_root / "synthetic-extraction-manifest.json",
json.dumps(extraction_manifest, indent=2, sort_keys=True) + "\n",
)
compiled_bundle_path = private_root / "compiled-v3-bundle.json"
compiled = _run(
[
sys.executable,
str(SOURCE_COMPILER),
"--artifact",
str(snapshot),
"--text",
str(snapshot),
"--manifest",
str(manifest_path),
"--extractor-spec",
str(extractor_spec),
"--output",
str(compiled_bundle_path),
]
)
assert _json_stdout(compiled)["status"] == "pending_review"
compiled_bundle = json.loads(compiled_bundle_path.read_text(encoding="utf-8"))
child = compiled_bundle["strict_child_proposal"]
proposal_id = child["id"]
apply_payload = child["payload"]["apply_payload"]
assert compiled_bundle["hashes"]["artifact_sha256"] == artifact_sha256
assert apply_payload["sources"][0]["content_hash"] == artifact_sha256
source_container_id = _start_postgres(source_container, DATABASE, run_id, staging_target=True)
source_started = True
genesis_test.docker_psql(source_container, DATABASE, v3_contract.BOOTSTRAP_SQL)
genesis_test.docker_psql(source_container, DATABASE, worker_test._supplemental_bootstrap())
genesis_test.docker_psql(
source_container,
DATABASE,
(f"alter role kb_review login password '{password}';alter role kb_apply login password '{password}';"),
)
genesis_test.docker_psql(
source_container,
DATABASE,
worker_test.PREREQUISITES.read_text(encoding="utf-8"),
)
genesis_dump = private_root / "baseline-genesis.dump"
genesis_manifest = private_root / "baseline-genesis-manifest.jsonl"
_capture_dump(source_container, DATABASE, genesis_dump)
_capture_manifest(source_container, DATABASE, genesis_manifest)
genesis_test.docker_psql(source_container, DATABASE, "alter role kb_apply nologin;")
genesis_test.docker_psql(
source_container,
DATABASE,
worker_test.MIGRATION.read_text(encoding="utf-8"),
)
genesis_test.docker_psql(
source_container,
DATABASE,
worker_test.PREREQUISITES.read_text(encoding="utf-8"),
)
secrets_file = _write_private(
private_root / "database-secrets.env",
f"KB_APPLY_DB_PASSWORD={password}\nKB_REVIEW_DB_PASSWORD={password}\n",
)
staged = _run(
[
sys.executable,
str(PROPOSAL_STAGER),
"--container",
source_container,
"--db",
DATABASE,
"--compiled-v3-bundle",
str(compiled_bundle_path),
]
)
stage_status = _json_stdout(staged)
assert stage_status == {
"bound_container_id": source_container_id,
"contract_version": 3,
"created": True,
"payload_sha256": child["payload_sha256"],
"proposal_id": proposal_id,
"status": "pending_review",
}
reviewed = _run(
[
sys.executable,
str(REVIEW_PACKET),
"--proposal-id",
proposal_id,
"--format",
"json",
"--secrets-file",
str(secrets_file),
"--container",
source_container,
"--db",
DATABASE,
"--host",
"127.0.0.1",
"--role",
"kb_review",
]
)
packets = _json_stdout(reviewed)["packets"]
assert len(packets) == 1
packet = packets[0]
assert packet["proposal_id"] == proposal_id
assert packet["review_state"] == "needs_human_review"
assert packet["strict_review_material"]["rows"] == {
key: apply_payload[key]
for key in ("claims", "sources", "evidence", "edges", "assessments", "reasoning_tools")
}
assert packet["strict_review_material"]["normalization_manifest"] == apply_payload["normalization_manifest"]
approved = _run(
[
sys.executable,
str(APPROVER),
proposal_id,
"--reviewed-by",
"m3taversal",
"--review-note",
SYNTHETIC_REVIEW_NOTE,
"--secrets-file",
str(secrets_file),
"--container",
source_container,
"--db",
DATABASE,
"--host",
"127.0.0.1",
"--role",
"kb_review",
]
)
approval_status = _json_stdout(approved)
assert approval_status["id"] == proposal_id
assert approval_status["status"] == "approved"
assert approval_status["reviewed_by_handle"] == "m3taversal"
receipts = private_root / "apply-receipts"
failure_state = private_root / "apply-worker-state.json"
applied = _run(
[
sys.executable,
str(APPLY_WORKER),
"--enable",
"--contract-version",
"3",
"--proposal-id",
proposal_id,
"--max-per-tick",
"1",
"--secrets-file",
str(secrets_file),
"--failure-state-file",
str(failure_state),
"--receipt-dir",
str(receipts),
"--container",
source_container,
"--db",
DATABASE,
"--host",
"127.0.0.1",
"--role",
"kb_apply",
]
)
assert f"applied {proposal_id} (approve_claim)" in applied.stdout
assert json.loads(failure_state.read_text(encoding="utf-8")) == {}
failure_state.unlink()
replay_receipt = receipts / f"{proposal_id}.json"
receipt = json.loads(replay_receipt.read_text(encoding="utf-8"))
assert receipt["proposal"]["reviewed_by_handle"] == "m3taversal"
assert receipt["canonical_rows"]["sources"][0]["content_hash"] == artifact_sha256
assert receipt["canonical_rows"]["claims"][0]["proposition"] == extraction_manifest["claims"][0]["proposition"]
material_path = private_root / "entry-0001.json"
exported = _run(
[
sys.executable,
str(MATERIAL_EXPORTER),
proposal_id,
"--replay-receipt",
str(replay_receipt),
"--sequence",
"1",
"--output",
str(material_path),
"--secrets-file",
str(secrets_file),
"--container",
source_container,
"--db",
DATABASE,
"--host",
"127.0.0.1",
"--role",
"kb_apply",
]
)
assert exported.stdout.strip() == f"replay material created: proposal={proposal_id} sequence=1"
material = json.loads(material_path.read_text(encoding="utf-8"))
assert material["contract_version"] == genesis_rebuild.MATERIAL_CONTRACT_VERSION
assert material["applied_proposal"]["payload"]["apply_payload"]["contract_version"] == 3
assert material["approved_proposal"]["review_note"] == SYNTHETIC_REVIEW_NOTE
assert material["approval_snapshot"]["reviewed_by_db_role"] == "kb_review"
final_manifest = private_root / "final-parity-manifest.jsonl"
_capture_manifest(source_container, DATABASE, final_manifest)
_remove_container(source_container)
source_started = False
_assert_container_absent(source_container)
labeled_sources = _docker(
"container",
"ls",
"--all",
"--filter",
f"label={SOURCE_CONTAINER_LABEL}={run_id}",
"--format",
"{{.Names}}",
)
assert labeled_sources.stdout.strip() == ""
review_note_file = _write_private(private_root / "admission-review-note.txt", SYNTHETIC_REVIEW_NOTE + "\n")
ledger_path = private_root / "v3-genesis-ledger.json"
admission_path = private_root / "v3-admission-manifest.json"
built_ledger = _run(
[
sys.executable,
str(LEDGER_BUILDER),
"--genesis-dump",
str(genesis_dump),
"--genesis-manifest",
str(genesis_manifest),
"--final-manifest",
str(final_manifest),
"--material",
str(material_path),
"--output-ledger",
str(ledger_path),
"--emit-admission-manifest",
"--admission-output",
str(admission_path),
"--reviewed-by",
"m3taversal",
"--reviewed-by-agent-id",
worker_test.REVIEWER_AGENT_ID,
"--reviewed-at",
"2026-07-19T00:00:00+00:00",
"--review-note-file",
str(review_note_file),
"--admission-scope",
"knowledge_commons",
"--admission-basis",
"reviewed_primary_source",
]
)
ledger_status = _json_stdout(built_ledger)
assert ledger_status["status"] == "pass"
assert ledger_status["action"] == "build"
ledger_sha256 = _sha256(ledger_path)
admission_sha256 = _sha256(admission_path)
admission = json.loads(admission_path.read_text(encoding="utf-8"))
assert admission["reviewed_by_handle"] == "m3taversal"
assert admission["review_note"] == SYNTHETIC_REVIEW_NOTE + "\n"
verified_ledger = _run(
[
sys.executable,
str(LEDGER_BUILDER),
"--genesis-dump",
str(genesis_dump),
"--genesis-manifest",
str(genesis_manifest),
"--verify-ledger",
str(ledger_path),
"--ledger-sha256",
ledger_sha256,
"--admission-manifest",
str(admission_path),
"--admission-manifest-sha256",
admission_sha256,
]
)
verify_status = _json_stdout(verified_ledger)
assert verify_status["status"] == "pass"
assert verify_status["action"] == "verify"
reconstruction_receipt = private_root / "reconstruction-receipt.json"
rebuilt = _run(
[
sys.executable,
str(GENESIS_REBUILDER),
"--genesis-dump",
str(genesis_dump),
"--genesis-manifest",
str(genesis_manifest),
"--ledger",
str(ledger_path),
"--ledger-sha256",
ledger_sha256,
"--admission-manifest",
str(admission_path),
"--admission-manifest-sha256",
admission_sha256,
"--database",
DATABASE,
"--container-prefix",
rebuild_prefix,
"--tmpfs-mb",
"384",
"--max-target-query-ms",
"5000",
"--max-target-source-ratio",
"100",
"--output",
str(reconstruction_receipt),
],
timeout=360,
)
assert _json_stdout(rebuilt)["status"] == "pass"
reconstruction = json.loads(reconstruction_receipt.read_text(encoding="utf-8"))
rebuild_container = reconstruction["container"]["name"]
assert reconstruction["status"] == "pass"
assert reconstruction["genesis_parity"]["status"] == "pass"
assert reconstruction["genesis_parity"]["problems"] == []
assert reconstruction["v3_contract"]["status"] == "pass"
assert reconstruction["ledger"]["entries"][0]["proposal_exact"] is True
assert reconstruction["ledger"]["entries"][0]["canonical_rows_exact"] is True
assert reconstruction["admission"]["status"] == "review_metadata_supplied"
assert reconstruction["admission"]["canonical_retrieval_eligible"] is False
assert reconstruction["admission"]["cryptographically_authenticated"] is False
assert reconstruction["final_parity"]["status"] == "pass"
assert reconstruction["final_parity"]["problems"] == []
assert reconstruction["cleanup"]["container_absent"] is True
assert reconstruction["safety"] == {
"network_access_available_to_container": False,
"persistent_postgres_storage_used": False,
"private_replay_material_emitted": False,
"production_database_touched": False,
"vps_gcp_telegram_or_live_db_used": False,
}
_assert_container_absent(rebuild_container)
rebuilt_labels = _docker(
"container",
"ls",
"--all",
"--filter",
f"label={genesis_rebuild.canonical_rebuild.CANARY_LABEL}={rebuild_container}",
"--format",
"{{.Names}}",
)
assert rebuilt_labels.stdout.strip() == ""
_assert_private_files(private_root)
finally:
if source_started:
_remove_container(source_container)
_assert_container_absent(source_container)
if rebuild_container is not None:
_assert_container_absent(rebuild_container)
orphaned = _docker(
"container",
"ls",
"--all",
"--filter",
f"label={SOURCE_CONTAINER_LABEL}={run_id}",
"--format",
"{{.Names}}",
)
assert orphaned.stdout.strip() == ""