teleo/teleo-infrastructure
8
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions 1

Merge pull request #104 from living-ip/codex/kb-rebuild-and-broad-hardening-20260713
Some checks are pending
CI / lint-and-test (push) Waiting to run
Details

Browse source 
Rebuild Leo KB quickly and harden broad database reasoning
This commit is contained in:
twentyOne2x 2026-07-13 10:52:48 +02:00 committed by GitHub
commit 279962fe8a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
14 changed files with 2614 additions and 38 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

20
docs/gcp-kb-cloudsql-restore-runbook.md
View file

@ -61,6 +61,24 @@ retains a private custom dump, dump SHA-256, object TOC, catalog/data manifest,
and before/after service state. It never restarts Leo or writes to the source
database.
Prove that this exact snapshot can rebuild a blank Postgres target before using
it for GCP:
```bash
.venv/bin/python ops/run_local_canonical_postgres_rebuild.py \
--dump <private-output-dir>/teleo-canonical.dump \
--source-manifest <private-output-dir>/source-manifest.jsonl \
--output /tmp/teleo-canonical-rebuild-receipt.json
```
The local runner starts a uniquely named `postgres:16-alpine` container with
network mode `none` and tmpfs-only database storage. It waits for an actual
`psql` connection to the named database, restores with `pg_restore
--no-owner --no-privileges --exit-on-error`, compares the full parity manifest,
then removes the container and proves it is absent. A passing local receipt is
the exact-recovery preflight; it is not semantic recompilation from raw source
documents.
Run `ops/postgres_parity_manifest.sql` against the isolated restored target,
then compare source and target:
@ -85,7 +103,7 @@ drill into `teleo`, `teleo_kb`, or `teleo_canonical`. Database isolation does
not isolate cluster-global roles or extensions, so verify those separately and
do not run the Docker-only gate bootstrap against the shared Cloud SQL instance.
After the parity verifier passes, run the no-send Cory composition replay from
After the parity verifier passes, run the no-send operator composition replay from
staging compute against that generated database. Only then delete the generated
database and uploaded import object and retain cleanup readback.

124
docs/kb-rebuild-and-recompile.md Normal file
View file

@ -0,0 +1,124 @@
# Rebuilding Leo's Knowledge Database
## Outcome
Leo should improve by compiling durable source material and reviewed changes
into Postgres. Repeatedly changing prompts or retraining the chat behavior is
not the knowledge system.
There are two different rebuilds:
1. **Exact recovery** restores the current canonical database from a verified
snapshot. This is working now.
2. **Semantic recompilation** starts from the retained source corpus and the
reviewed change ledger, then reproduces the canonical rows. This is partly
recoverable but is not yet complete.
## Exact Recovery: Working
Run:
```bash
.venv/bin/python ops/run_local_canonical_postgres_rebuild.py \
--dump /private/path/teleo-canonical.dump \
--source-manifest /private/path/source-manifest.jsonl \
--output /tmp/teleo-canonical-rebuild-receipt.json
```
The 2026-07-13 live canary restored a fresh, network-isolated Postgres in 3.81
seconds and then removed it. The restored target matched the source across all
39 manifest tables and all 52,164 rows, with no schema, data, constraint, role,
or performance mismatch. Key rows included:
- 1,837 claims;
- 4,145 sources;
- 4,670 claim-evidence links;
- 4,916 claim edges;
- 17 reasoning tools;
- 26 proposals.
This is the fastest disaster-recovery path. It does not require Leo to
re-extract or relearn the corpus.
## Source Recompilation: Current Evidence
Read-only VPS inspection found two retained June import runs. Both point to the
Forgejo-era workspace at `/opt/teleo-eval/workspaces/main` and retained
inventory JSONL files under `/opt/teleo-eval/kb-import/`.
The database still retains:
- `kb_stage.import_runs` for the two inventories;
- staged claims, sources, claim-source links, and claim edges;
- `kb_stage.canonical_mappings` from legacy keys to canonical UUIDs;
- mappings for 1,807 of 1,837 canonical claims;
- mappings for all 4,145 canonical sources.
For the 1,807 mapped claims, current canonical type, text, status, confidence,
tags, and creator match the retained staged rows exactly. Creation timestamps
also follow a recoverable rule: use the legacy timestamp when present and the
mapping timestamp for the eight rows that had no legacy timestamp.
The simple retained-row joins currently reproduce:
- 4,254 of 4,670 canonical evidence links;
- 4,878 of 4,916 canonical edge rows can be accounted for by a staged relation;
historical duplicate multiplicity still needs an explicit replay rule.
The remaining gaps are concrete rather than mysterious:
- 30 claims were created after or outside the mapped import;
- 284 mapped source rows do not have a matching retained `staged_sources` row
and need the original source-synthesis rule or an explicit genesis record;
- 416 evidence links need source-synthesis or later-change provenance;
- 38 edge rows need later-change receipts or explicit replay records;
- old applied proposal rows do not describe every historical canonical write.
This proves that most of the initial database came from the retained file-KB
import path. It does not yet prove a clean blank-database recompile.
## Target Compiler
The durable rebuild model is:
```text
immutable source corpus + file hashes
-> deterministic inventory and classification
-> staged claims, sources, evidence links, and edges
-> stable canonical ID mapping
-> review decisions
-> append-only accepted apply payloads and receipts
-> canonical Postgres
-> render/sync/restart
-> answer benchmark
```
Use the current verified snapshot as **genesis epoch 1**. Preserve its dump,
manifest, source commit, inventory files, mappings, and aggregate rebuild
receipt. Every accepted change after that epoch must carry a replayable strict
apply payload and row-level postflight receipt. This prevents the historical
gap from growing while the old import rules are reconstructed.
The existing local ingestion canary already proves one hash-bound document
fixture can become a normalized `pending_review` proposal without touching
canonical rows. The existing full-data clone canary proves a reviewed packet
can create source, claim, evidence, and edge rows and affect later reasoning.
The next capability is to join those into a corpus runner and replay ledger.
## Definition Of Working
Semantic recompilation is complete only when all of these pass:
1. A command creates a blank database from the retained source corpus plus the
reviewed ledger.
2. Schema, constraints, roles, table counts, row hashes, and key query results
match the canonical manifest.
3. Every canonical row traces to a genesis import record or a reviewed apply
receipt.
4. A new document can be hash-captured, extracted into grounded candidates,
deduplicated, staged, reviewed, applied in a disposable clone, and read back.
5. After render/sync and restart, Leo answers the related broad question using
the new rows and cites the source chain.
Until then, exact snapshot recovery is production recovery; source
recompilation is an active build capability, not a finished claim.

60
docs/reports/leo-working-state-20260709/local-canonical-postgres-rebuild-canary-20260713.md Normal file
View file

@ -0,0 +1,60 @@
# Local Canonical Postgres Rebuild Canary
Generated UTC: `2026-07-13T08:34:54.016399+00:00`
Status: **pass**
## Result
- Total command time: `3.809726s`
- Named database ready: `0.781077s`
- `pg_restore`: `1.026398s`
- Manifest capture and parity comparison: `1.229484s`
- Cleanup: `0.088110s`
- Dump bytes: `13,462,116`
- Dump SHA-256: `8a217e32bf6c83cb2b629e5ca5f91fc57f759e5010d9c40babe0d13c9a450822`
- Source manifest SHA-256: `d077f859b5f8ece908c61958a90914272ddc6fb9930df3b97d2ad2ac85b8e39a`
## Parity
- Manifest tables: `39/39`
- Total rows: `52,164/52,164`
- Table mismatches: `0`
- Column, constraint, function, index, policy, schema, sequence, trigger,
type, and view hash mismatches: `0`
- Application role mismatches: `0`
- Performance threshold failures: `0`
Key restored rows:
| Object | Rows |
|---|---:|
| `public.claims` | 1,837 |
| `public.sources` | 4,145 |
| `public.claim_evidence` | 4,670 |
| `public.claim_edges` | 4,916 |
| `public.reasoning_tools` | 17 |
| `kb_stage.kb_proposals` | 26 |
## Isolation And Cleanup
- Docker image: `postgres:16-alpine`
- Network mode: `none`
- Postgres data directory: `tmpfs`, no persistent Docker volume
- Named-database readiness used an actual `psql -d teleo` connection
- Production database touched: `false`
- Container absence after cleanup: `true`
- Independent label-filtered orphan check: no containers found
## Command
```bash
.venv/bin/python ops/run_local_canonical_postgres_rebuild.py \
--dump /private/path/teleo-canonical.dump \
--source-manifest /private/path/source-manifest.jsonl \
--output /tmp/teleo-canonical-rebuild-receipt.json
```
This proves fast exact recovery from the canonical snapshot. It does not claim
that the whole database can yet be semantically recompiled from raw documents
and reviewed change records alone.

58
docs/reports/leo-working-state-20260709/telegram-handler-m3taversal-blind-review-pre-broad-hardening-20260713.md Normal file
View file

@ -0,0 +1,58 @@
# Working Leo Blind Review Before Broad Hardening
Date: `2026-07-13`
Status: **not reliable for unattended operator use**
## Method
A 12-question temporary-profile handler run used broad prompts without known
proposal IDs. It posted nothing to Telegram and made no database or service
change. Two independent strict reviews evaluated the same transcript against
the live v1 schema and retained proof.
Reviewer results differed in severity but agreed on the verdict:
- strict binary review: `1/12 pass`;
- pass/partial/fail review: `2 pass`, `4 partial`, `6 fail`.
## Converged Failures
1. Replies presented target-architecture fields as current schema, including
source author/channel/date, claim bodies, evidence excerpts, edge rationale,
and forecast-resolution fields.
2. Replies invented edge types such as `superseded_by`, `relates_to`, and
`resolves`. Current v1 uses `supersedes`; `superseded_by` is a claim column.
3. Replies treated unchanged canonical counts as a complete explanation of
behavior, ignoring deployed skills, runtime configuration, rendered files,
and persisted Hermes session state.
4. A handler-only no-post run was described as proof of the live Telegram path.
5. A temporary conversation label was promoted into source provenance rather
than resolving a real URL, storage path, file hash, or retained artifact.
6. Staging was treated as though it required the same authorization as
canonical apply. Requested source capture and proposal staging are allowed;
guarded canonical apply is the approval boundary.
7. Forecast handling proposed rewriting historical confidence and inventing
resolution storage that current v1 does not have.
8. Claims/evidence were not consistently treated as shared knowledge while
beliefs and positions remain agent-specific.
9. Replies were too long for operational use: `9,033` words total, `753` words
per answer on average, with several over `1,000` words.
## Repair Incorporated
- Explicit current-schema and edge contracts in both VPS and GCP bridge skills.
- Runtime/session/canonical proof-tier separation.
- Source-link versus provenance-quality distinction.
- Shared claims/evidence versus agent-specific positions.
- Forecast-history preservation and current schema gap.
- Autonomous source capture and staging boundary.
- Maximum response length in the out-of-sample scorer.
- Expansion from 9 to 15 broad prompts covering these failure modes.
## Claim Ceiling
This is a pre-fix handler baseline. It is not Telegram-visible proof. The repair
is not accepted until the deployed 15-question run passes, independent strict
review converges, restart survival passes, and a separate Telegram-visible
no-mutation canary succeeds.

189
docs/reports/leo-working-state-20260709/telegram-handler-m3taversal-oos-suite-pre-evidence-semantics-repair-20260713.json Normal file
View file

File diff suppressed because one or more lines are too long

285
docs/reports/leo-working-state-20260709/telegram-handler-m3taversal-oos-suite-score-pre-evidence-semantics-repair-20260713.json Normal file
View file

@ -0,0 +1,285 @@
{
"db_counts_changed": false,
"generated_at_utc": "2026-07-13T08:15:42.852150+00:00",
"memory_token": "demo-ledger-61c5dcf6",
"mode": "working_leo_m3taversal_out_of_sample_live_vps_handler_score",
"posted_to_telegram": false,
"production_db_apply_ran": false,
"score": {
"expected_prompt_count": 9,
"expected_prompt_ids": [
"OOS-01",
"OOS-02",
"OOS-03",
"OOS-04",
"OOS-05",
"OOS-06",
"OOS-07",
"OOS-08",
"OOS-09"
],
"failures": [
{
"concepts": {
"canonical_readback": true,
"live_check": false,
"material_action": true,
"state_boundary": true
},
"current_schema_overclaims": [],
"custom_signals": {},
"dimension": "demo_pressure_without_ids",
"invalid_count_invariant_detected": false,
"legacy_pass": true,
"legacy_signals": {
"canonical_db": true,
"fresh_readback": true,
"next_action": true,
"no_overclaim": true,
"state_semantics": true
},
"overclaim_detected": false,
"pass": false,
"prompt_id": "OOS-01",
"response_issue_detected": false
},
{
"concepts": {
"receipt": true,
"source_evidence_chain": false,
"staged_review_apply": true
},
"current_schema_overclaims": [],
"custom_signals": {},
"dimension": "source_pointer_is_not_evidence",
"invalid_count_invariant_detected": false,
"legacy_pass": false,
"legacy_signals": {
"canonical_db": true,
"document_artifact_linking": true,
"no_overclaim": false,
"row_level_proof": true,
"staging_or_review": false
},
"overclaim_detected": false,
"pass": false,
"prompt_id": "OOS-05",
"response_issue_detected": false
}
],
"missing_prompt_ids": [],
"pass": false,
"passes": 7,
"prompt_count": 9,
"scores": [
{
"concepts": {
"canonical_readback": true,
"live_check": false,
"material_action": true,
"state_boundary": true
},
"current_schema_overclaims": [],
"custom_signals": {},
"dimension": "demo_pressure_without_ids",
"invalid_count_invariant_detected": false,
"legacy_pass": true,
"legacy_signals": {
"canonical_db": true,
"fresh_readback": true,
"next_action": true,
"no_overclaim": true,
"state_semantics": true
},
"overclaim_detected": false,
"pass": false,
"prompt_id": "OOS-01",
"response_issue_detected": false
},
{
"concepts": {
"contradiction": true,
"deduplication": true,
"provenance": true,
"receipt": true,
"staged_review_apply": true
},
"current_schema_overclaims": [],
"custom_signals": {},
"dimension": "multi_surface_knowledge_ingestion",
"invalid_count_invariant_detected": false,
"legacy_pass": false,
"legacy_signals": {
"canonical_db": true,
"caveat_retention": false,
"document_artifact_linking": true,
"next_action": true,
"no_overclaim": true,
"staging_or_review": false
},
"overclaim_detected": false,
"pass": true,
"prompt_id": "OOS-02",
"response_issue_detected": false
},
{
"concepts": {
"identity_chain": true,
"receipt": true,
"state_boundary": true
},
"current_schema_overclaims": [],
"custom_signals": {},
"dimension": "identity_restart_truth",
"invalid_count_invariant_detected": false,
"legacy_pass": true,
"legacy_signals": {
"canonical_db": true,
"identity_rendering": true,
"no_overclaim": true,
"row_level_proof": true
},
"overclaim_detected": false,
"pass": true,
"prompt_id": "OOS-03",
"response_issue_detected": false
},
{
"concepts": {
"canonical_readback": true,
"receipt": true,
"state_boundary": true
},
"current_schema_overclaims": [],
"custom_signals": {},
"dimension": "partner_demo_claim_ceiling",
"invalid_count_invariant_detected": false,
"legacy_pass": false,
"legacy_signals": {
"artifact": true,
"canonical_db": true,
"no_overclaim": false,
"row_level_proof": true,
"state_semantics": true
},
"overclaim_detected": false,
"pass": true,
"prompt_id": "OOS-04",
"response_issue_detected": false
},
{
"concepts": {
"receipt": true,
"source_evidence_chain": false,
"staged_review_apply": true
},
"current_schema_overclaims": [],
"custom_signals": {},
"dimension": "source_pointer_is_not_evidence",
"invalid_count_invariant_detected": false,
"legacy_pass": false,
"legacy_signals": {
"canonical_db": true,
"document_artifact_linking": true,
"no_overclaim": false,
"row_level_proof": true,
"staging_or_review": false
},
"overclaim_detected": false,
"pass": false,
"prompt_id": "OOS-05",
"response_issue_detected": false
},
{
"concepts": {
"heterogeneous_types": true,
"receipt": true,
"staged_review_apply": true
},
"current_schema_overclaims": [],
"custom_signals": {},
"dimension": "database_composition_judgment",
"invalid_count_invariant_detected": false,
"legacy_pass": false,
"legacy_signals": {
"canonical_db": true,
"caveat_retention": false,
"next_action": true,
"no_overclaim": true,
"staging_or_review": true
},
"overclaim_detected": false,
"pass": true,
"prompt_id": "OOS-06",
"response_issue_detected": false
},
{
"concepts": {
"blocker_definition": true
},
"current_schema_overclaims": [],
"custom_signals": {
"memory_token": true
},
"dimension": "same_session_memory_set",
"invalid_count_invariant_detected": false,
"legacy_pass": false,
"legacy_signals": {
"next_action": true,
"no_overclaim": false
},
"overclaim_detected": false,
"pass": true,
"prompt_id": "OOS-07",
"response_issue_detected": false
},
{
"concepts": {
"blocker_definition": true,
"receipt": true
},
"current_schema_overclaims": [],
"custom_signals": {
"closure_proof": true,
"memory_token": true
},
"dimension": "same_session_memory_recall",
"invalid_count_invariant_detected": false,
"legacy_pass": false,
"legacy_signals": {
"next_action": true,
"no_overclaim": false,
"row_level_proof": true
},
"overclaim_detected": false,
"pass": true,
"prompt_id": "OOS-08",
"response_issue_detected": false
},
{
"concepts": {},
"current_schema_overclaims": [],
"custom_signals": {
"current_update_identity_boundary": true,
"exact_participant_handle": true,
"no_cross_participant_identity_bleed": true,
"no_unverified_alias": true
},
"dimension": "telegram_participant_identity",
"invalid_count_invariant_detected": false,
"legacy_pass": false,
"legacy_signals": {
"no_overclaim": false
},
"overclaim_detected": false,
"pass": true,
"prompt_id": "OOS-09",
"response_issue_detected": false
}
],
"unexpected_prompt_ids": []
},
"service_unchanged": true,
"source_results_json": "/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/work/teleo-infra-main/docs/reports/leo-working-state-20260709/telegram-handler-m3taversal-oos-suite-current.json",
"temp_profile_removed": true
}

25
docs/reports/leo-working-state-20260709/telegram-handler-m3taversal-oos-suite-score-pre-evidence-semantics-repair-20260713.md Normal file
View file

@ -0,0 +1,25 @@
# Working Leo m3taversal Out-of-Sample Score
Generated UTC: `2026-07-13T08:15:42.852150+00:00`
Pass: `False`
Prompts: `7/9`
DB counts changed: `False`
Service unchanged: `True`
Temporary profile removed: `True`
Posted to Telegram: `False`
## Prompt Scores
- `OOS-01` / `demo_pressure_without_ids`: `pass=False`
- `OOS-02` / `multi_surface_knowledge_ingestion`: `pass=True`
- `OOS-03` / `identity_restart_truth`: `pass=True`
- `OOS-04` / `partner_demo_claim_ceiling`: `pass=True`
- `OOS-05` / `source_pointer_is_not_evidence`: `pass=False`
- `OOS-06` / `database_composition_judgment`: `pass=True`
- `OOS-07` / `same_session_memory_set`: `pass=True`
- `OOS-08` / `same_session_memory_recall`: `pass=True`
- `OOS-09` / `telegram_participant_identity`: `pass=True`
## Claim Ceiling
A pass proves broad out-of-sample and same-session-memory behavior through the live VPS GatewayRunner using a temporary profile, with no Telegram post and no DB count change. It does not prove human-visible Telegram delivery, production DB apply, or GCP parity.

71
hermes-agent/leoclean-skills/gcp/teleo-kb-bridge/SKILL.md
View file

@ -52,6 +52,32 @@ good default is:
3. final answer with what is grounded, what is weak, and what evidence would
improve it.
Lead with the answer. Unless the operator asks for a detailed audit, keep the
whole reply under 220 words and end with one proof-changing follow-up. Do not
turn a direct operational question into an architecture lecture.
Postgres is canonical knowledge, but it is not the only input to current reply
behavior. Deployed skills, runtime configuration, rendered `SOUL.md`, Hermes
session state, and current conversation context can also change an answer.
Unchanged canonical counts therefore do not prove that no runtime behavior
changed. Hermes `state.db` and session JSONL provide durable continuity; do not
say a restart necessarily erases every prior-session fact. Separate these proof
tiers explicitly:
- handler or temporary-profile success with no Telegram post is handler proof,
not Telegram-visible delivery proof;
- a Telegram-visible reply proves delivery and reply behavior, not canonical
DB mutation;
- canonical mutation requires row-level `public.*` readback and the matching
applied proposal receipt.
Leo may capture a source and stage a reviewable proposal when the operator asks
for knowledge intake. Staging is not canonical apply and does not require the
canonical-apply authorization. Approval is required before the guarded apply
step. Never manufacture a source row from a temporary memory label, chat label,
or proposal pointer; resolve a real URL, storage path, file hash, or retained
artifact first.
For no-context direct claims such as "Is X in Leo now?", "did the DB change?",
"did the decision matrix approve this?", or "is it still just proposals?", do
not stop at `search` or default `list-proposals`. Run the status-specific
@ -134,6 +160,18 @@ Current `public.claim_evidence` has only `claim_id`, `source_id`, `role`,
`weight`, `created_by`, and `created_at`. Its accepted roles are `grounds`,
`illustrates`, and `contradicts`. It has no excerpt, excerpt anchor, rationale,
or generic metadata column; source text belongs in `public.sources.excerpt`.
A `public.claim_evidence` link from a claim to a `public.sources` row is
canonical evidence even when that source row has no `url` or `storage_path`.
Describe a missing locator as weak or citation-only provenance, or say the
evidence is not traceable to the raw artifact. Do not call the canonical link
non-canonical or ungrounded solely because the locator is missing.
A Telegram attachment, extracted file, or proposal `source_ref` does not by
itself prove canonical evidence from that attachment. That proof requires a
`public.sources` row representing the attachment and a `public.claim_evidence`
link from the claim to that source row. Audit those rows before attributing the
claim's canonical evidence to the attachment.
Current `public.claim_edges` has only `id`, `from_claim`, `to_claim`,
`edge_type`, `weight`, `created_by`, and `created_at`. Both endpoints are claim
IDs, so do not claim that a `reasoning_tools` row is directly connected through
@ -142,16 +180,22 @@ IDs, so do not claim that a `reasoning_tools` row is directly connected through
For heterogeneous research packets, map only to structures proven in the
current schema:
- claims, sources, and evidence links are shared knowledge objects; an agent's
confidence, stance, or position belongs in agent-specific belief/position
structures rather than duplicate agent-authored copies of the same claim;
- factual observations and disputed interpretations may become separate
`public.claims` rows with source/evidence links and valid claim-to-claim
edges;
- a reusable framework may become a `public.reasoning_tools` row, but the
current schema has no generic reasoning-tool-to-claim edge and no shipped
`concept_maps` or `claim_concept_map_links` table;
- `public.governance_gates` can store an evaluative gate with `name`,
`criteria`, `evidence_bar`, and `pass_condition`; it is not a generic
behavioral-rule or policy table, so state a schema gap when a governance rule
does not fit that contract;
- a behavioral or operating rule belongs in the existing
`public.behavioral_rules` table, whose rule contract includes `agent_id`,
`category`, `rank`, `rule`, and `rationale`;
- an evaluative gate belongs in `public.governance_gates`, with `name`,
`criteria`, `evidence_bar`, and `pass_condition`; do not flatten a behavioral
rule into this gate table;
- a belief correction may create a new claim, a valid `supersedes` edge, and
set the old claim's `superseded_by` column. `superseded_by` is a column, not
an edge type.
@ -175,11 +219,13 @@ universal success condition.
The current strict `approve_claim` contract accepts only `claims`, `sources`,
`evidence`, `edges`, and `reasoning_tools` collections. It does not insert
`governance_gates`, update an existing claim's `status` or `superseded_by`, or
write arbitrary soul/context rows. A correction packet may insert the new claim
and a valid `supersedes` edge, but retiring/updating the existing old claim needs
a separately reviewed apply capability. Do not describe that unsupported update
as part of one atomic `approve_claim` transaction.
`behavioral_rules` or `governance_gates`, update an existing claim's `status`
or `superseded_by`, or write arbitrary soul/context rows. Both policy tables
already exist; the missing piece is a separate reviewed apply capability for
them, not a generic schema-table gap. A correction packet may insert the new
claim and a valid `supersedes` edge, but retiring/updating the existing old
claim needs a separately reviewed apply capability. Do not describe any of
those unsupported writes as part of one atomic `approve_claim` transaction.
Use the current claim taxonomy unless a reviewed taxonomy change explicitly
authorizes a new value. The live values are `structural`, `normative`,
@ -192,6 +238,13 @@ evidence, say that `public.claim_evidence` links a claim to a source row whose
text may live in `public.sources.excerpt`; never say the evidence row stores or
carries the excerpt.
Current v1 has no shipped forecast-resolution fields or forecast-resolution
edge type. Preserve the original probability and its timestamp. Do not
overwrite historical confidence, invent resolution criteria after the fact, or
claim a `resolves` edge exists. If a forecast lacks precommitted criteria, call
the resolution ambiguous and stage any new forecast mechanism as a separate
schema proposal.
- "Did we actually update the KB?": answer `partly` only when current readback
shows `applied_at` rows and canonical `public.*` rows. Otherwise say
`mostly still proposals`; list applied, approved-but-not-applied, pending,

71
hermes-agent/leoclean-skills/vps/teleo-kb-bridge/SKILL.md
View file

@ -45,6 +45,32 @@ For KB questions, prefer the bridge over raw database access. A good default is:
3. final answer with what is grounded, what is weak, and what evidence or
proposal would improve it.
Lead with the answer. Unless the operator asks for a detailed audit, keep the
whole reply under 220 words and end with one proof-changing follow-up. Do not
turn a direct operational question into an architecture lecture.
Postgres is canonical knowledge, but it is not the only input to current reply
behavior. Deployed skills, runtime configuration, rendered `SOUL.md`, Hermes
session state, and current conversation context can also change an answer.
Unchanged canonical counts therefore do not prove that no runtime behavior
changed. Hermes `state.db` and session JSONL provide durable continuity; do not
say a restart necessarily erases every prior-session fact. Separate these proof
tiers explicitly:
- handler or temporary-profile success with no Telegram post is handler proof,
not Telegram-visible delivery proof;
- a Telegram-visible reply proves delivery and reply behavior, not canonical
DB mutation;
- canonical mutation requires row-level `public.*` readback and the matching
applied proposal receipt.
Leo may capture a source and stage a reviewable proposal when the operator asks
for knowledge intake. Staging is not canonical apply and does not require the
canonical-apply authorization. Approval is required before the guarded apply
step. Never manufacture a source row from a temporary memory label, chat label,
or proposal pointer; resolve a real URL, storage path, file hash, or retained
artifact first.
For no-context direct claims such as "Is X in Leo now?", "did the DB change?",
"did the decision matrix approve this?", or "is it still just proposals?", do
not stop at `search` or default `list-proposals`. Run the status-specific
@ -127,6 +153,18 @@ Current `public.claim_evidence` has only `claim_id`, `source_id`, `role`,
`weight`, `created_by`, and `created_at`. Its accepted roles are `grounds`,
`illustrates`, and `contradicts`. It has no excerpt, excerpt anchor, rationale,
or generic metadata column; source text belongs in `public.sources.excerpt`.
A `public.claim_evidence` link from a claim to a `public.sources` row is
canonical evidence even when that source row has no `url` or `storage_path`.
Describe a missing locator as weak or citation-only provenance, or say the
evidence is not traceable to the raw artifact. Do not call the canonical link
non-canonical or ungrounded solely because the locator is missing.
A Telegram attachment, extracted file, or proposal `source_ref` does not by
itself prove canonical evidence from that attachment. That proof requires a
`public.sources` row representing the attachment and a `public.claim_evidence`
link from the claim to that source row. Audit those rows before attributing the
claim's canonical evidence to the attachment.
Current `public.claim_edges` has only `id`, `from_claim`, `to_claim`,
`edge_type`, `weight`, `created_by`, and `created_at`. Both endpoints are claim
IDs, so do not claim that a `reasoning_tools` row is directly connected through
@ -135,16 +173,22 @@ IDs, so do not claim that a `reasoning_tools` row is directly connected through
For heterogeneous research packets, map only to structures proven in the
current schema:
- claims, sources, and evidence links are shared knowledge objects; an agent's
confidence, stance, or position belongs in agent-specific belief/position
structures rather than duplicate agent-authored copies of the same claim;
- factual observations and disputed interpretations may become separate
`public.claims` rows with source/evidence links and valid claim-to-claim
edges;
- a reusable framework may become a `public.reasoning_tools` row, but the
current schema has no generic reasoning-tool-to-claim edge and no shipped
`concept_maps` or `claim_concept_map_links` table;
- `public.governance_gates` can store an evaluative gate with `name`,
`criteria`, `evidence_bar`, and `pass_condition`; it is not a generic
behavioral-rule or policy table, so state a schema gap when a governance rule
does not fit that contract;
- a behavioral or operating rule belongs in the existing
`public.behavioral_rules` table, whose rule contract includes `agent_id`,
`category`, `rank`, `rule`, and `rationale`;
- an evaluative gate belongs in `public.governance_gates`, with `name`,
`criteria`, `evidence_bar`, and `pass_condition`; do not flatten a behavioral
rule into this gate table;
- a belief correction may create a new claim, a valid `supersedes` edge, and
set the old claim's `superseded_by` column. `superseded_by` is a column, not
an edge type.
@ -168,11 +212,13 @@ universal success condition.
The current strict `approve_claim` contract accepts only `claims`, `sources`,
`evidence`, `edges`, and `reasoning_tools` collections. It does not insert
`governance_gates`, update an existing claim's `status` or `superseded_by`, or
write arbitrary soul/context rows. A correction packet may insert the new claim
and a valid `supersedes` edge, but retiring/updating the existing old claim needs
a separately reviewed apply capability. Do not describe that unsupported update
as part of one atomic `approve_claim` transaction.
`behavioral_rules` or `governance_gates`, update an existing claim's `status`
or `superseded_by`, or write arbitrary soul/context rows. Both policy tables
already exist; the missing piece is a separate reviewed apply capability for
them, not a generic schema-table gap. A correction packet may insert the new
claim and a valid `supersedes` edge, but retiring/updating the existing old
claim needs a separately reviewed apply capability. Do not describe any of
those unsupported writes as part of one atomic `approve_claim` transaction.
Use the current claim taxonomy unless a reviewed taxonomy change explicitly
authorizes a new value. The live values are `structural`, `normative`,
@ -185,6 +231,13 @@ evidence, say that `public.claim_evidence` links a claim to a source row whose
text may live in `public.sources.excerpt`; never say the evidence row stores or
carries the excerpt.
Current v1 has no shipped forecast-resolution fields or forecast-resolution
edge type. Preserve the original probability and its timestamp. Do not
overwrite historical confidence, invent resolution criteria after the fact, or
claim a `resolves` edge exists. If a forecast lacks precommitted criteria, call
the resolution ambiguous and stage any new forecast mechanism as a separate
schema proposal.
- "Did we actually update the KB?": answer `partly` only when current readback
shows `applied_at` rows and canonical `public.*` rows. Otherwise say
`mostly still proposals`; list applied, approved-but-not-applied, pending,

819
ops/run_local_canonical_postgres_rebuild.py Executable file
View file

@ -0,0 +1,819 @@
#!/usr/bin/env python3
"""Restore and verify a canonical PostgreSQL dump in an isolated Docker canary."""
from __future__ import annotations
import argparse
import hashlib
import json
import os
import re
import signal
import subprocess
import tempfile
import time
import uuid
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
try:
from .verify_postgres_parity_manifest import compare_manifests, load_manifest
except ImportError: # pragma: no cover - direct script execution
from verify_postgres_parity_manifest import compare_manifests, load_manifest
DEFAULT_IMAGE = "postgres:16-alpine"
DEFAULT_MANIFEST_SQL = Path(__file__).with_name("postgres_parity_manifest.sql")
DATA_DIR = "/var/lib/postgresql/data"
DUMP_DESTINATION = "/tmp/teleo-canonical.dump"
MANIFEST_DESTINATION = "/tmp/postgres-parity-manifest.sql"
CANARY_LABEL = "com.livingip.teleo.canonical-rebuild-canary"
DATABASE_NAME_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]{0,62}\Z")
CONTAINER_PREFIX_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,39}\Z")
ALLOWED_APPLICATION_ROLES = frozenset({"kb_apply", "kb_review"})
ROLE_BOOLEAN_FIELDS = (
"can_login",
"superuser",
"create_db",
"create_role",
"inherit",
"replication",
"bypass_rls",
)
KEY_TABLES = (
"public.claims",
"public.sources",
"public.claim_evidence",
"public.claim_edges",
"public.reasoning_tools",
"kb_stage.kb_proposals",
)
class TerminationRequested(RuntimeError):
"""Raised after SIGINT/SIGTERM so the normal cleanup path still runs."""
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def validate_custom_dump(path: Path) -> None:
if not path.is_file():
raise ValueError(f"custom dump does not exist or is not a regular file: {path}")
if path.stat().st_size <= 5:
raise ValueError("custom dump is empty or truncated")
with path.open("rb") as handle:
signature = handle.read(5)
if signature != b"PGDMP":
raise ValueError("dump is not PostgreSQL custom format (missing PGDMP signature)")
def new_container_name(prefix: str) -> str:
timestamp = datetime.now(UTC).strftime("%Y%m%d%H%M%S")
return f"{prefix}-{timestamp}-{uuid.uuid4().hex[:12]}"
def _run(
command: list[str],
*,
timeout: float,
) -> subprocess.CompletedProcess[str]:
return subprocess.run(
command,
capture_output=True,
check=False,
text=True,
timeout=timeout,
)
def _bounded_error(completed: subprocess.CompletedProcess[str]) -> str:
message = (completed.stderr or completed.stdout or "no command output").strip()
return message[-2000:]
def _require_success(completed: subprocess.CompletedProcess[str], action: str) -> str:
if completed.returncode != 0:
raise RuntimeError(f"{action} failed (exit {completed.returncode}): {_bounded_error(completed)}")
return completed.stdout
def _absence_result(completed: subprocess.CompletedProcess[str]) -> tuple[bool | None, str | None]:
if completed.returncode == 0:
return False, None
message = (completed.stderr or completed.stdout or "").strip()
lowered = message.lower()
if "no such object" in lowered or "no such container" in lowered:
return True, None
return None, message[-2000:] or f"docker inspect exited {completed.returncode}"
def inspect_container_absence(
docker_bin: str,
container: str,
*,
timeout: float,
) -> tuple[bool | None, str | None]:
completed = _run([docker_bin, "container", "inspect", container], timeout=timeout)
return _absence_result(completed)
def inspect_container_isolation(
docker_bin: str,
container: str,
*,
expected_image: str,
expected_label: str,
timeout: float,
) -> dict[str, Any]:
completed = _run([docker_bin, "container", "inspect", container], timeout=timeout)
raw = _require_success(completed, "container isolation inspection")
try:
rows = json.loads(raw)
observed = rows[0]
host_config = observed["HostConfig"]
config = observed["Config"]
except (IndexError, KeyError, TypeError, json.JSONDecodeError) as exc:
raise RuntimeError("container isolation inspection returned invalid JSON") from exc
network_mode = host_config.get("NetworkMode")
tmpfs = host_config.get("Tmpfs") or {}
labels = config.get("Labels") or {}
image = config.get("Image")
running = bool((observed.get("State") or {}).get("Running"))
problems = []
if network_mode != "none":
problems.append(f"network_mode={network_mode!r}")
if DATA_DIR not in tmpfs:
problems.append("postgres_data_tmpfs_missing")
if labels.get(CANARY_LABEL) != expected_label:
problems.append("canary_label_mismatch")
if image != expected_image:
problems.append(f"image={image!r}")
if not running:
problems.append("container_not_running")
if problems:
raise RuntimeError("container isolation check failed: " + ", ".join(problems))
return {
"network_mode": network_mode,
"postgres_data_tmpfs": tmpfs[DATA_DIR],
"image": image,
"label": labels[CANARY_LABEL],
"running": running,
}
def _psql_command(container: str, database: str, *arguments: str, docker_bin: str) -> list[str]:
return [
docker_bin,
"exec",
container,
"psql",
"-X",
"-U",
"postgres",
"-d",
database,
"-Atq",
"-v",
"ON_ERROR_STOP=1",
*arguments,
]
def wait_for_named_database(
docker_bin: str,
container: str,
database: str,
*,
startup_timeout: float,
command_timeout: float,
poll_interval: float,
) -> int:
deadline = time.monotonic() + startup_timeout
attempts = 0
last_error = "no psql attempt completed"
while True:
attempts += 1
remaining = max(0.1, deadline - time.monotonic())
completed = _run(
_psql_command(
container,
database,
"-c",
"select current_database();",
docker_bin=docker_bin,
),
timeout=min(command_timeout, remaining),
)
if completed.returncode == 0 and completed.stdout.strip() == database:
return attempts
last_error = _bounded_error(completed)
if time.monotonic() >= deadline:
raise RuntimeError(
f"named database did not accept a psql connection within {startup_timeout:g}s: {last_error}"
)
time.sleep(min(poll_interval, max(0.0, deadline - time.monotonic())))
def _role_keyword(value: bool, enabled: str, disabled: str) -> str:
return enabled if value else disabled
def application_role_sql(source_manifest: dict[str, Any]) -> str:
items = source_manifest["singleton"]["application_roles"].get("items", [])
if not isinstance(items, list):
raise ValueError("source manifest application_roles.items must be a list")
statements = []
seen: set[str] = set()
for item in items:
if not isinstance(item, dict):
raise ValueError("source manifest application role must be an object")
name = item.get("name")
if name not in ALLOWED_APPLICATION_ROLES:
raise ValueError(f"source manifest contains non-allowlisted application role: {name!r}")
if name in seen:
raise ValueError(f"source manifest contains duplicate application role: {name}")
seen.add(name)
for field in ROLE_BOOLEAN_FIELDS:
if type(item.get(field)) is not bool:
raise ValueError(f"source manifest application role {name} has invalid {field}")
attributes = [
_role_keyword(item["can_login"], "LOGIN", "NOLOGIN"),
_role_keyword(item["superuser"], "SUPERUSER", "NOSUPERUSER"),
_role_keyword(item["create_db"], "CREATEDB", "NOCREATEDB"),
_role_keyword(item["create_role"], "CREATEROLE", "NOCREATEROLE"),
_role_keyword(item["inherit"], "INHERIT", "NOINHERIT"),
_role_keyword(item["replication"], "REPLICATION", "NOREPLICATION"),
_role_keyword(item["bypass_rls"], "BYPASSRLS", "NOBYPASSRLS"),
]
statements.append(f'CREATE ROLE "{name}" {" ".join(attributes)};')
return "\n".join(statements)
def execute_application_role_bootstrap(
docker_bin: str,
container: str,
database: str,
source_manifest: dict[str, Any],
*,
timeout: float,
) -> list[str]:
sql = application_role_sql(source_manifest)
if not sql:
return []
completed = _run(
_psql_command(container, database, "-c", sql, docker_bin=docker_bin),
timeout=timeout,
)
_require_success(completed, "application role bootstrap")
return sorted(item["name"] for item in source_manifest["singleton"]["application_roles"].get("items", []))
def query_scalar(
docker_bin: str,
container: str,
database: str,
sql: str,
*,
timeout: float,
) -> str:
completed = _run(
_psql_command(container, database, "-c", sql, docker_bin=docker_bin),
timeout=timeout,
)
return _require_success(completed, "PostgreSQL readback").strip()
def collect_key_counts(
docker_bin: str,
container: str,
database: str,
*,
timeout: float,
) -> tuple[dict[str, int], int]:
counts: dict[str, int] = {}
for table in KEY_TABLES:
exists = query_scalar(
docker_bin,
container,
database,
f"select to_regclass('{table}') is not null;",
timeout=timeout,
)
if exists != "t":
raise RuntimeError(f"required canonical table is missing after restore: {table}")
schema, name = table.split(".", 1)
raw_count = query_scalar(
docker_bin,
container,
database,
f'select count(*) from "{schema}"."{name}";',
timeout=timeout,
)
try:
counts[table] = int(raw_count)
except ValueError as exc:
raise RuntimeError(f"invalid row count for {table}: {raw_count!r}") from exc
raw_table_count = query_scalar(
docker_bin,
container,
database,
"select count(*) from pg_tables where schemaname in ('public', 'kb_stage');",
timeout=timeout,
)
try:
table_count = int(raw_table_count)
except ValueError as exc:
raise RuntimeError(f"invalid canonical schema table count: {raw_table_count!r}") from exc
return counts, table_count
def run_parity_check(
docker_bin: str,
container: str,
database: str,
*,
source_manifest_path: Path,
source_manifest: dict[str, Any],
manifest_sql: Path,
command_timeout: float,
max_target_query_ms: float,
max_target_source_ratio: float,
) -> dict[str, Any]:
copy_result = _run(
[docker_bin, "cp", str(manifest_sql), f"{container}:{MANIFEST_DESTINATION}"],
timeout=command_timeout,
)
_require_success(copy_result, "parity manifest copy")
completed = _run(
_psql_command(
container,
database,
"-f",
MANIFEST_DESTINATION,
docker_bin=docker_bin,
),
timeout=command_timeout,
)
target_payload = _require_success(completed, "target parity manifest capture")
with tempfile.TemporaryDirectory(prefix="teleo-canonical-parity-") as temp_dir:
target_path = Path(temp_dir) / "target-manifest.jsonl"
target_path.write_text(target_payload, encoding="utf-8")
target_manifest = load_manifest(target_path)
problems, details = compare_manifests(
source_manifest,
target_manifest,
max_target_query_ms=max_target_query_ms,
max_target_source_ratio=max_target_source_ratio,
)
target_receipt = {
"bytes": target_path.stat().st_size,
"sha256": sha256_file(target_path),
"line_count": sum(1 for line in target_payload.splitlines() if line.strip()),
}
return {
"requested": True,
"status": "pass" if not problems else "fail",
"verifier": "ops.verify_postgres_parity_manifest.compare_manifests",
"source_manifest": {
"path": str(source_manifest_path),
"bytes": source_manifest_path.stat().st_size,
"sha256": sha256_file(source_manifest_path),
},
"target_manifest": target_receipt,
"problems": problems,
"details": details,
}
def cleanup_container(
docker_bin: str,
container: str,
*,
command_timeout: float,
poll_interval: float,
) -> dict[str, Any]:
cleanup: dict[str, Any] = {
"attempted": True,
"container": container,
"force_remove_attempted": True,
"container_absent": None,
"status": "unproven",
}
try:
removed = _run([docker_bin, "rm", "--force", container], timeout=command_timeout)
cleanup["force_remove_exit_code"] = removed.returncode
if removed.returncode != 0:
cleanup["force_remove_message"] = _bounded_error(removed)
except (OSError, subprocess.TimeoutExpired) as exc:
cleanup["force_remove_error"] = str(exc)
except (KeyboardInterrupt, TerminationRequested) as exc:
cleanup["interrupted"] = str(exc)
cleanup["force_remove_error"] = str(exc)
last_error = None
for attempt in range(1, 6):
cleanup["absence_inspection_attempts"] = attempt
try:
absent, error = inspect_container_absence(
docker_bin,
container,
timeout=command_timeout,
)
except (OSError, subprocess.TimeoutExpired) as exc:
absent, error = None, str(exc)
except (KeyboardInterrupt, TerminationRequested) as exc:
cleanup["interrupted"] = str(exc)
absent, error = None, str(exc)
if absent is True:
cleanup["container_absent"] = True
cleanup["status"] = "pass"
cleanup.pop("absence_inspection_error", None)
return cleanup
cleanup["container_absent"] = absent
last_error = error
if error and "interrupted" not in cleanup:
cleanup["absence_inspection_error"] = error
break
if attempt < 5:
try:
time.sleep(poll_interval)
except (KeyboardInterrupt, TerminationRequested) as exc:
cleanup["interrupted"] = str(exc)
if last_error:
cleanup["absence_inspection_error"] = last_error
cleanup["status"] = "fail" if cleanup["container_absent"] is False else "unproven"
return cleanup
def _initial_receipt(args: argparse.Namespace, container: str) -> dict[str, Any]:
return {
"artifact": "local_canonical_postgres_rebuild_canary",
"generated_at_utc": datetime.now(UTC).isoformat(),
"status": "running",
"database": args.database,
"dump": {
"path": str(args.dump.resolve()),
"bytes": None,
"sha256": None,
"custom_format_signature_valid": False,
},
"container": {
"name": container,
"image": args.image,
"network_mode_required": "none",
"postgres_data_tmpfs_required": f"rw,nosuid,nodev,noexec,size={args.tmpfs_mb}m",
"preexisting_absence_proven": False,
"started": False,
"isolation": None,
},
"application_roles_bootstrapped": [],
"key_counts": {},
"canonical_schema_table_count": None,
"parity": {
"requested": args.source_manifest is not None,
"status": "pending" if args.source_manifest is not None else "not_requested",
"reason": None if args.source_manifest is not None else "--source-manifest was not supplied",
},
"timings_seconds": {},
"cleanup": {
"attempted": False,
"container": container,
"container_absent": None,
"status": "pending",
},
"safety": {
"production_database_touched": False,
"network_access_available_to_container": False,
"persistent_postgres_storage_used": False,
},
"error": None,
}
def run_canary(args: argparse.Namespace) -> dict[str, Any]:
started_at = time.monotonic()
container = new_container_name(args.container_prefix)
receipt = _initial_receipt(args, container)
phase = "validation"
source_manifest: dict[str, Any] | None = None
primary_error: dict[str, str] | None = None
try:
hash_started = time.monotonic()
validate_custom_dump(args.dump)
receipt["dump"].update(
{
"bytes": args.dump.stat().st_size,
"sha256": sha256_file(args.dump),
"custom_format_signature_valid": True,
}
)
receipt["timings_seconds"]["dump_validation_and_hash"] = round(time.monotonic() - hash_started, 6)
if args.source_manifest is not None:
phase = "source_manifest_validation"
source_manifest = load_manifest(args.source_manifest)
phase = "container_name_preflight"
absent, absence_error = inspect_container_absence(
args.docker_bin,
container,
timeout=args.command_timeout,
)
if absent is not True:
detail = absence_error or "generated container name unexpectedly already exists"
raise RuntimeError(f"could not prove generated container name is unused: {detail}")
receipt["container"]["preexisting_absence_proven"] = True
phase = "container_start"
startup_started = time.monotonic()
tmpfs_options = f"{DATA_DIR}:rw,nosuid,nodev,noexec,size={args.tmpfs_mb}m"
run_result = _run(
[
args.docker_bin,
"run",
"--detach",
"--rm",
"--network",
"none",
"--name",
container,
"--label",
f"{CANARY_LABEL}={container}",
"--tmpfs",
tmpfs_options,
"--env",
f"POSTGRES_DB={args.database}",
"--env",
"POSTGRES_HOST_AUTH_METHOD=trust",
args.image,
],
timeout=args.command_timeout,
)
container_id = _require_success(run_result, "networkless PostgreSQL container start").strip()
if not container_id:
raise RuntimeError("networkless PostgreSQL container start returned no container id")
receipt["container"]["started"] = True
receipt["container"]["id_prefix"] = container_id[:12]
receipt["container"]["isolation"] = inspect_container_isolation(
args.docker_bin,
container,
expected_image=args.image,
expected_label=container,
timeout=args.command_timeout,
)
phase = "named_database_readiness"
readiness_attempts = wait_for_named_database(
args.docker_bin,
container,
args.database,
startup_timeout=args.startup_timeout,
command_timeout=args.command_timeout,
poll_interval=args.poll_interval,
)
receipt["container"]["named_database_psql_ready"] = True
receipt["container"]["named_database_psql_attempts"] = readiness_attempts
receipt["timings_seconds"]["container_start_and_named_database_ready"] = round(
time.monotonic() - startup_started,
6,
)
if source_manifest is not None:
phase = "application_role_bootstrap"
receipt["application_roles_bootstrapped"] = execute_application_role_bootstrap(
args.docker_bin,
container,
args.database,
source_manifest,
timeout=args.command_timeout,
)
phase = "dump_copy"
copy_result = _run(
[args.docker_bin, "cp", str(args.dump.resolve()), f"{container}:{DUMP_DESTINATION}"],
timeout=args.command_timeout,
)
_require_success(copy_result, "custom dump copy")
phase = "restore"
restore_started = time.monotonic()
restore_result = _run(
[
args.docker_bin,
"exec",
container,
"pg_restore",
"-U",
"postgres",
"-d",
args.database,
"--no-owner",
"--no-privileges",
"--exit-on-error",
DUMP_DESTINATION,
],
timeout=args.restore_timeout,
)
_require_success(restore_result, "canonical PostgreSQL restore")
receipt["timings_seconds"]["pg_restore"] = round(time.monotonic() - restore_started, 6)
phase = "key_count_readback"
count_started = time.monotonic()
key_counts, table_count = collect_key_counts(
args.docker_bin,
container,
args.database,
timeout=args.command_timeout,
)
receipt["key_counts"] = key_counts
receipt["canonical_schema_table_count"] = table_count
receipt["timings_seconds"]["key_count_readback"] = round(time.monotonic() - count_started, 6)
if source_manifest is not None:
phase = "parity_manifest_and_compare"
parity_started = time.monotonic()
receipt["parity"] = run_parity_check(
args.docker_bin,
container,
args.database,
source_manifest_path=args.source_manifest,
source_manifest=source_manifest,
manifest_sql=args.manifest_sql,
command_timeout=args.manifest_timeout,
max_target_query_ms=args.max_target_query_ms,
max_target_source_ratio=args.max_target_source_ratio,
)
receipt["timings_seconds"]["manifest_capture_and_parity_compare"] = round(
time.monotonic() - parity_started,
6,
)
if receipt["parity"]["status"] != "pass":
raise RuntimeError("canonical parity verification failed")
receipt["status"] = "pass"
except (
OSError,
ValueError,
RuntimeError,
subprocess.TimeoutExpired,
json.JSONDecodeError,
KeyboardInterrupt,
TerminationRequested,
) as exc:
primary_error = {
"phase": phase,
"type": type(exc).__name__,
"message": str(exc)[-2000:],
}
receipt["status"] = "fail"
receipt["error"] = primary_error
finally:
cleanup_started = time.monotonic()
cleanup = cleanup_container(
args.docker_bin,
container,
command_timeout=args.command_timeout,
poll_interval=args.poll_interval,
)
receipt["cleanup"] = cleanup
receipt["timings_seconds"]["cleanup"] = round(time.monotonic() - cleanup_started, 6)
receipt["timings_seconds"]["total"] = round(time.monotonic() - started_at, 6)
if cleanup.get("container_absent") is not True:
receipt["status"] = "fail"
cleanup_error = {
"phase": "cleanup",
"type": "CleanupProofError",
"message": "container absence was not independently proven",
}
if primary_error is None:
receipt["error"] = cleanup_error
else:
receipt["cleanup_error"] = cleanup_error
if cleanup.get("interrupted"):
receipt["status"] = "fail"
interruption_error = {
"phase": "cleanup",
"type": "TerminationRequested",
"message": cleanup["interrupted"],
}
if primary_error is None:
receipt["error"] = interruption_error
else:
receipt["cleanup_interruption"] = interruption_error
return receipt
def write_receipt(path: Path, payload: dict[str, Any]) -> None:
path = path.expanduser()
if not path.is_absolute():
path = Path.cwd() / path
path.parent.mkdir(parents=True, exist_ok=True)
path = path.parent.resolve() / path.name
encoded = (json.dumps(payload, indent=2, sort_keys=True) + "\n").encode()
temporary = path.parent / f".{path.name}.{uuid.uuid4().hex}.tmp"
descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
try:
with os.fdopen(descriptor, "wb") as handle:
handle.write(encoded)
handle.flush()
os.fsync(handle.fileno())
os.replace(temporary, path)
os.chmod(path, 0o600)
finally:
try:
temporary.unlink()
except FileNotFoundError:
pass
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--dump", required=True, type=Path, help="PostgreSQL custom-format pg_dump")
parser.add_argument("--database", default="teleo")
parser.add_argument("--image", default=DEFAULT_IMAGE)
parser.add_argument("--container-prefix", default="teleo-canonical-rebuild")
parser.add_argument("--tmpfs-mb", default=1024, type=int)
parser.add_argument("--source-manifest", type=Path)
parser.add_argument("--manifest-sql", default=DEFAULT_MANIFEST_SQL, type=Path)
parser.add_argument("--max-target-query-ms", default=2000.0, type=float)
parser.add_argument("--max-target-source-ratio", default=20.0, type=float)
parser.add_argument("--startup-timeout", default=60.0, type=float)
parser.add_argument("--restore-timeout", default=300.0, type=float)
parser.add_argument("--manifest-timeout", default=180.0, type=float)
parser.add_argument("--command-timeout", default=30.0, type=float)
parser.add_argument("--poll-interval", default=0.25, type=float)
parser.add_argument("--docker-bin", default="docker")
parser.add_argument("--output", required=True, type=Path)
args = parser.parse_args(argv)
if not DATABASE_NAME_RE.fullmatch(args.database):
parser.error("--database must be a safe PostgreSQL identifier up to 63 characters")
if not CONTAINER_PREFIX_RE.fullmatch(args.container_prefix):
parser.error("--container-prefix must be 1-40 Docker-safe characters")
if not args.image or any(character.isspace() for character in args.image):
parser.error("--image must be a non-empty Docker image reference without whitespace")
if not args.docker_bin or any(character.isspace() for character in args.docker_bin):
parser.error("--docker-bin must be one executable name or path without whitespace")
if not 64 <= args.tmpfs_mb <= 65536:
parser.error("--tmpfs-mb must be between 64 and 65536")
for value, flag in (
(args.startup_timeout, "--startup-timeout"),
(args.restore_timeout, "--restore-timeout"),
(args.manifest_timeout, "--manifest-timeout"),
(args.command_timeout, "--command-timeout"),
(args.poll_interval, "--poll-interval"),
(args.max_target_query_ms, "--max-target-query-ms"),
(args.max_target_source_ratio, "--max-target-source-ratio"),
):
if value <= 0:
parser.error(f"{flag} must be positive")
if not args.dump.is_file():
parser.error("--dump must be an existing regular file")
if args.source_manifest is not None:
if not args.source_manifest.is_file():
parser.error("--source-manifest must be an existing regular file")
if not args.manifest_sql.is_file():
parser.error("--manifest-sql must be an existing regular file when parity is requested")
protected_inputs = {args.dump.resolve(), args.manifest_sql.resolve()}
if args.source_manifest is not None:
protected_inputs.add(args.source_manifest.resolve())
if args.output.resolve() in protected_inputs:
parser.error("--output must not overwrite a dump or manifest input")
return args
def _termination_handler(signum: int, _frame: Any) -> None:
raise TerminationRequested(f"received {signal.Signals(signum).name}")
def main(argv: list[str] | None = None) -> int:
args = parse_args(argv)
previous_handlers = {}
for watched_signal in (signal.SIGINT, signal.SIGTERM):
previous_handlers[watched_signal] = signal.getsignal(watched_signal)
signal.signal(watched_signal, _termination_handler)
try:
payload = run_canary(args)
finally:
for watched_signal, previous in previous_handlers.items():
signal.signal(watched_signal, previous)
try:
write_receipt(args.output, payload)
except OSError as exc:
payload["status"] = "fail"
payload["receipt_write_error"] = str(exc)
print(json.dumps(payload, indent=2, sort_keys=True))
return 0 if payload["status"] == "pass" else 1
if __name__ == "__main__":
raise SystemExit(main())

307
scripts/working_leo_m3taversal_oos_benchmark.py
View file

@ -74,7 +74,7 @@ SCENARIOS: list[dict[str, Any]] = [
},
{
"id": "OOS-05",
"dimension": "source_pointer_is_not_evidence",
"dimension": "source_pointer_needs_canonical_link_audit",
"message": (
"A proposal points to a Telegram attachment and its extracted text exists on disk. Does that mean the "
"claim already has canonical evidence? Walk the link chain I should audit and tell me what a valid "
@ -87,7 +87,13 @@ SCENARIOS: list[dict[str, Any]] = [
"row_level_proof",
"no_overclaim",
],
"required_concepts": ["source_evidence_chain", "staged_review_apply", "receipt"],
"required_concepts": [
"source_evidence_chain",
"canonical_evidence_boundary",
"evidence_provenance_quality",
"staged_review_apply",
"receipt",
],
},
{
"id": "OOS-06",
@ -99,7 +105,13 @@ SCENARIOS: list[dict[str, Any]] = [
"review and apply sequence too, but do not write anything now."
),
"required_signals": ["canonical_db", "staging_or_review", "caveat_retention", "next_action", "no_overclaim"],
"required_concepts": ["heterogeneous_types", "staged_review_apply", "receipt"],
"required_concepts": [
"heterogeneous_types",
"behavioral_rule_storage",
"reviewed_policy_apply",
"staged_review_apply",
"receipt",
],
},
{
"id": "OOS-07",
@ -134,6 +146,72 @@ SCENARIOS: list[dict[str, Any]] = [
"required_signals": ["no_overclaim"],
"required_concepts": [],
},
{
"id": "OOS-10",
"dimension": "runtime_vs_canonical_causality",
"message": (
"The five database totals are unchanged after a restart. Does that prove Leo's answer behavior is "
"unchanged and that every fact from the prior session was erased? Explain which persisted and deployed "
"surfaces can affect the answer, and separate the proof tiers. Read-only; keep it under 180 words."
),
"required_signals": ["canonical_db", "no_overclaim"],
"required_concepts": ["runtime_inputs", "durable_session_continuity", "proof_tiers"],
},
{
"id": "OOS-11",
"dimension": "shared_claims_agent_positions",
"message": (
"Two agents read the same evidence and reach different conclusions. Should Leo duplicate the factual "
"claim once per agent, or keep shared knowledge and represent agent-specific positions elsewhere? Give "
"the current-schema answer and how disagreement remains queryable. Do not write anything."
),
"required_signals": ["canonical_db", "caveat_retention", "no_overclaim"],
"required_concepts": ["shared_knowledge_commons", "agent_specific_positions", "contradiction"],
},
{
"id": "OOS-12",
"dimension": "forecast_resolution_without_rewriting_history",
"message": (
"An old claim recorded a 60% forecast but never defined resolution criteria. The event is now over. "
"What may Leo record in the current database without rewriting history, and what needs a schema "
"proposal? Do not apply anything."
),
"required_signals": ["canonical_db", "staging_or_review", "caveat_retention", "no_overclaim"],
"required_concepts": ["forecast_history", "forecast_schema_gap", "staged_review_apply"],
},
{
"id": "OOS-13",
"dimension": "handler_is_not_telegram_delivery",
"message": (
"A temporary-profile GatewayRunner answered every test prompt, but the run explicitly posted nothing "
"to Telegram. Can I tell a partner the Telegram path is proven live? Answer first, then state exactly "
"what this run proves and the smallest test that closes the gap."
),
"required_signals": ["artifact", "next_action", "no_overclaim"],
"required_concepts": ["handler_not_telegram", "proof_tiers", "receipt"],
},
{
"id": "OOS-14",
"dimension": "autonomous_source_intake_boundary",
"message": (
"I hand Leo a document and say: absorb this as far as safely possible without making me approve every "
"mechanical step. What can Leo capture and stage immediately, what real source identity must be retained, "
"and where does explicit approval begin? Explain only; do not ingest this prompt."
),
"required_signals": ["authorization", "staging_or_review", "artifact", "no_overclaim"],
"required_concepts": ["staging_without_apply_authorization", "real_source_identity", "staged_review_apply"],
},
{
"id": "OOS-15",
"dimension": "schema_valid_supersession",
"message": (
"A canonical claim is wrong. I want the replacement, an explanation, and the old claim visibly retired. "
"In current v1, which exact claim and edge fields exist, which requested writes fit approve_claim, and "
"which require a separate reviewed apply capability? Do not mutate anything."
),
"required_signals": ["canonical_db", "staging_or_review", "row_level_proof", "no_overclaim"],
"required_concepts": ["valid_supersession", "current_edge_schema", "apply_capability_boundary"],
},
]
@ -146,7 +224,9 @@ CONCEPT_PATTERNS: dict[str, tuple[re.Pattern[str], ...]] = {
re.I,
),
),
"live_check": (re.compile(r"checked live|I ran `?teleo-kb|fresh readback|current canonical row counts", re.I),),
"live_check": (
re.compile(r"checked live|I ran `?teleo-kb|fresh readback|live readback|current canonical row counts", re.I),
),
"material_action": (
re.compile(
r"next .*action|rebuild .*apply_payload|operator .*authoriz|review .*apply|apply sequence|postflight",
@ -175,7 +255,22 @@ CONCEPT_PATTERNS: dict[str, tuple[re.Pattern[str], ...]] = {
re.compile(r"file|attachment|source_ref", re.I),
re.compile(r"public\.sources", re.I),
re.compile(r"claim_evidence", re.I),
re.compile(r"not canonical|staging-layer pointer|proves nothing about the KB", re.I),
re.compile(r"audit|link|join|row chain|before-and-after|before/after", re.I),
),
"canonical_evidence_boundary": (
re.compile(r"canonical evidence", re.I),
re.compile(r"claim_evidence", re.I),
re.compile(r"public\.sources|source row", re.I),
re.compile(
r"attachment.{0,160}(?:does not|doesn't|is not|isn't|cannot|can't|alone|until|unless)|"
r"(?:does not|doesn't|is not|isn't|cannot|can't).{0,100}canonical evidence from (?:that|the) attachment",
re.I | re.S,
),
),
"evidence_provenance_quality": (
re.compile(r"(?:missing|no|without).{0,50}(?:url|storage(?:_path)?|locator)", re.I | re.S),
re.compile(r"weak|citation-only|citation only|not traceable|raw artifact", re.I),
re.compile(r"canonical evidence|canonical link", re.I),
),
"heterogeneous_types": (
re.compile(r"claim", re.I),
@ -185,6 +280,97 @@ CONCEPT_PATTERNS: dict[str, tuple[re.Pattern[str], ...]] = {
re.compile(r"correction|supersed", re.I),
re.compile(r"disput|contradict", re.I),
),
"behavioral_rule_storage": (
re.compile(r"public\.behavioral_rules", re.I),
re.compile(r"\bagent_id\b", re.I),
re.compile(r"\bcategory\b", re.I),
re.compile(r"\brank\b", re.I),
re.compile(r"\brule\b", re.I),
re.compile(r"\brationale\b", re.I),
),
"reviewed_policy_apply": (
re.compile(r"approve_claim", re.I),
re.compile(r"behavioral_rules", re.I),
re.compile(r"governance_gates", re.I),
re.compile(r"does not (?:accept|insert|support)|supports neither|neither.{0,80}nor", re.I | re.S),
re.compile(r"separate.{0,50}reviewed apply|reviewed.{0,50}apply capability", re.I | re.S),
),
"runtime_inputs": (
re.compile(r"Postgres|canonical (?:DB|database|counts?)", re.I),
re.compile(r"skills?|runtime config|configuration|SOUL\.md", re.I),
re.compile(r"session|conversation context", re.I),
re.compile(r"unchanged.{0,80}(?:does not|doesn't|do not).{0,80}(?:behavior|answer)", re.I | re.S),
),
"durable_session_continuity": (
re.compile(r"state\.db|session JSONL", re.I),
re.compile(r"persist|durable|continuity", re.I),
re.compile(r"restart.{0,80}(?:does not|doesn't|need not|not necessarily).{0,80}(?:erase|forget)", re.I | re.S),
),
"proof_tiers": (
re.compile(r"handler|temporary[- ]profile|GatewayRunner", re.I),
re.compile(r"Telegram", re.I),
re.compile(r"canonical|public\.\*|DB mutation|database mutation", re.I),
),
"shared_knowledge_commons": (
re.compile(r"shared (?:claim|knowledge|commons)|claims.{0,40}shared", re.I | re.S),
re.compile(r"source|evidence", re.I),
re.compile(r"do not duplicate|don'?t duplicate|one shared claim|single shared claim", re.I),
),
"agent_specific_positions": (
re.compile(r"agent[- ]specific", re.I),
re.compile(r"belief|position|stance|confidence", re.I),
re.compile(r"separate|elsewhere|per-agent", re.I),
),
"forecast_history": (
re.compile(r"original (?:probability|confidence)|60%|history", re.I),
re.compile(r"preserve|retain|do not overwrite|don'?t overwrite", re.I),
re.compile(r"ambiguous|missing.{0,30}criteria|no.{0,30}criteria", re.I | re.S),
),
"forecast_schema_gap": (
re.compile(r"current (?:v1|schema)|public\.claims", re.I),
re.compile(
r"no.{0,50}(?:forecast[- ]resolution|resolution field|resolved_at)|does not have.{0,50}resolution",
re.I | re.S,
),
re.compile(r"resolves.{0,30}(?:not|isn'?t|does not)|no.{0,30}resolves", re.I | re.S),
),
"handler_not_telegram": (
re.compile(r"no|not", re.I),
re.compile(r"handler|GatewayRunner|temporary[- ]profile", re.I),
re.compile(r"did not post|posted nothing|not Telegram-visible|does not prove Telegram", re.I),
),
"staging_without_apply_authorization": (
re.compile(r"capture|hash|archive", re.I),
re.compile(r"stage|pending_review|proposal", re.I),
re.compile(
r"does not require.{0,50}(?:apply )?approval|without.{0,50}(?:apply )?approval|approval begins.{0,80}apply",
re.I | re.S,
),
),
"real_source_identity": (
re.compile(r"URL|storage path|file hash|content hash|retained artifact", re.I),
re.compile(r"temporary label|chat label|proposal pointer|source_ref", re.I),
re.compile(r"not.{0,40}(?:a )?(?:source|provenance)|must not.{0,40}source|do not manufacture", re.I | re.S),
),
"valid_supersession": (
re.compile(r"new|replacement", re.I),
re.compile(r"supersedes", re.I),
re.compile(r"superseded_by", re.I),
re.compile(r"old claim", re.I),
),
"current_edge_schema": (
re.compile(r"claim_edges", re.I),
re.compile(r"from_claim", re.I),
re.compile(r"to_claim", re.I),
re.compile(r"edge_type", re.I),
re.compile(r"no.{0,30}rationale|does not.{0,30}rationale", re.I | re.S),
),
"apply_capability_boundary": (
re.compile(r"approve_claim", re.I),
re.compile(r"insert.{0,50}(?:new|replacement) claim|new claim.{0,50}insert", re.I | re.S),
re.compile(r"supersedes edge", re.I),
re.compile(r"separate.{0,50}reviewed apply|does not update.{0,80}(?:status|superseded_by)", re.I | re.S),
),
"blocker_definition": (
re.compile(r"blocker", re.I),
re.compile(r"approved|applied_at", re.I),
@ -206,7 +392,8 @@ COUNT_INVARIANT_REJECTION_RE = re.compile(
SCHEMA_GAP_QUALIFIER_RE = re.compile(
r"\b(?:proposed|future|not current|not shipped|does not exist|doesn't exist|absent|"
r"has no|have no|no column|not an edge|would require|schema gap|must be added)\b",
r"has no|have no|no column|not an edge|would require|schema gap|must be added|"
r"does not support|doesn't support|supports neither|not supported)\b",
re.I,
)
CURRENT_SCHEMA_ASSERTION_PATTERNS: dict[str, re.Pattern[str]] = {
@ -247,6 +434,53 @@ CURRENT_SCHEMA_ASSERTION_PATTERNS: dict[str, re.Pattern[str]] = {
}
UNVERIFIED_M3TAVERSAL_ALIAS_RE = re.compile(r"\b(?:Cory|m3ta)\b", re.I)
SOURCE_EVIDENCE_CANONICAL_OBJECT_RE = re.compile(r"claim_evidence|public\.sources|source rows?", re.I)
SOURCE_EVIDENCE_LOCATOR_GAP_RE = re.compile(
r"(?:missing|without|no).{0,50}(?:url|storage(?:_path)?|locator)", re.I | re.S
)
SOURCE_EVIDENCE_DENIAL_RE = re.compile(
r"(?:not|isn't|is not|doesn't count as|does not count as).{0,40}(?:canonical|grounded) evidence",
re.I | re.S,
)
SOURCE_EVIDENCE_CITATION_STUB_DENIAL_RE = re.compile(
r"citation stubs?.{0,50}(?:not|isn't|is not).{0,30}(?:canonical|grounded) evidence",
re.I | re.S,
)
BEHAVIORAL_RULES_FALSE_ABSENCE_RE = re.compile(
r"(?:public\.)?behavioral_rules(?: table)?.{0,50}(?:does not exist|doesn't exist|is absent|is missing|not shipped)|"
r"(?:no|missing) (?:public\.)?behavioral_rules table",
re.I | re.S,
)
DB_ONLY_CAUSALITY_RE = re.compile(
r"(?:unchanged|same) (?:canonical )?(?:DB|database|counts?).{0,80}"
r"(?:proves?|means|therefore).{0,80}(?:behavior|answer).{0,30}(?:unchanged|cannot change)",
re.I | re.S,
)
RESTART_ERASES_ALL_RE = re.compile(
r"restart.{0,100}(?:erases?|forgets?|loses?).{0,40}(?:all|every|prior[- ]session)", re.I | re.S
)
HANDLER_TELEGRAM_OVERCLAIM_RE = re.compile(
r"(?:Telegram (?:path|delivery).{0,40}(?:is|was) (?:live|proven)|"
r"live Telegram (?:path|delivery)).{0,160}(?:posted nothing|did not post|no Telegram post)",
re.I | re.S,
)
FORECAST_HISTORY_REWRITE_RE = re.compile(
r"(?<!not )(?<!don't )(?<!dont )(?:overwrite|update|replace).{0,40}"
r"(?:original )?(?:probability|confidence).{0,80}"
r"(?:resolved|actual|outcome)",
re.I | re.S,
)
TEMP_LABEL_AS_SOURCE_RE = re.compile(
r"(?:use|write|create|store|promote).{0,50}(?:temporary|memory|chat) (?:label|token).{0,100}"
r"(?:source_ref|public\.sources|canonical source)",
re.I | re.S,
)
DEFAULT_MAX_RESPONSE_WORDS = 300
MAX_RESPONSE_WORDS = {"OOS-09": 100, "OOS-10": 180}
def prompt_catalog(memory_token: str) -> list[dict[str, Any]]:
return [
@ -279,6 +513,51 @@ def current_schema_overclaims(reply: str) -> list[str]:
return sorted(set(findings))
def source_evidence_semantic_issues(reply: str) -> list[str]:
"""Reject answers that confuse canonical linkage with locator quality."""
findings: set[str] = set()
for segment in re.split(r"(?<=[.!?])\s+|\n+", reply):
if (
SOURCE_EVIDENCE_CANONICAL_OBJECT_RE.search(segment)
and SOURCE_EVIDENCE_LOCATOR_GAP_RE.search(segment)
and SOURCE_EVIDENCE_DENIAL_RE.search(segment)
):
findings.add("locator_gap_called_noncanonical")
if SOURCE_EVIDENCE_CITATION_STUB_DENIAL_RE.search(reply):
findings.add("citation_stub_called_ungrounded")
return sorted(findings)
def behavioral_rule_schema_issues(reply: str) -> list[str]:
"""Reject claims that the current behavioral-rules table is absent."""
return ["behavioral_rules_false_absence"] if BEHAVIORAL_RULES_FALSE_ABSENCE_RE.search(reply) else []
def broad_semantic_issues(reply: str) -> list[str]:
"""Reject high-impact false claims that can hide inside otherwise complete prose."""
findings: set[str] = set()
if DB_ONLY_CAUSALITY_RE.search(reply) and not re.search(
r"does not prove|doesn't prove|do not prove|does not mean|doesn't mean|do not mean|cannot prove", reply, re.I
):
findings.add("unchanged_db_called_complete_behavior_proof")
if RESTART_ERASES_ALL_RE.search(reply) and not re.search(
r"restart.{0,100}(?:does not|doesn't|need not|not necessarily).{0,60}(?:erase|forget|lose)",
reply,
re.I | re.S,
):
findings.add("restart_called_total_memory_erasure")
if HANDLER_TELEGRAM_OVERCLAIM_RE.search(reply):
findings.add("handler_proof_called_telegram_live")
if FORECAST_HISTORY_REWRITE_RE.search(reply):
findings.add("forecast_history_rewrite")
if TEMP_LABEL_AS_SOURCE_RE.search(reply):
findings.add("temporary_label_promoted_to_source")
return sorted(findings)
def score_reply(prompt: dict[str, Any], reply: str, *, memory_token: str) -> dict[str, Any]:
legacy_score = base.score_reply(prompt, reply)
concepts = {concept: matched_concept(reply, concept) for concept in prompt["required_concepts"]}
@ -307,6 +586,12 @@ def score_reply(prompt: dict[str, Any], reply: str, *, memory_token: str) -> dic
)
invalid_count_invariant = asserts_invalid_count_invariant(reply)
schema_overclaims = current_schema_overclaims(reply)
source_evidence_issues = source_evidence_semantic_issues(reply) if prompt["id"] == "OOS-05" else []
behavioral_rule_issues = behavioral_rule_schema_issues(reply) if prompt["id"] == "OOS-06" else []
semantic_issues = broad_semantic_issues(reply)
word_count = len(re.findall(r"\b\w+(?:[-']\w+)*\b", reply))
max_response_words = MAX_RESPONSE_WORDS.get(prompt["id"], DEFAULT_MAX_RESPONSE_WORDS)
response_too_long = word_count > max_response_words
return {
"prompt_id": prompt["id"],
"dimension": prompt["dimension"],
@ -318,12 +603,22 @@ def score_reply(prompt: dict[str, Any], reply: str, *, memory_token: str) -> dic
"response_issue_detected": legacy_score["response_issue_detected"],
"invalid_count_invariant_detected": invalid_count_invariant,
"current_schema_overclaims": schema_overclaims,
"source_evidence_semantic_issues": source_evidence_issues,
"behavioral_rule_schema_issues": behavioral_rule_issues,
"broad_semantic_issues": semantic_issues,
"word_count": word_count,
"max_response_words": max_response_words,
"response_too_long": response_too_long,
"pass": bool(
all(concepts.values())
and all(custom_signals.values())
and not legacy_score["overclaim_detected"]
and not invalid_count_invariant
and not schema_overclaims
and not source_evidence_issues
and not behavioral_rule_issues
and not semantic_issues
and not response_too_long
),
}

44
tests/test_hermes_leoclean_skill_surfaces.py
View file

@ -120,7 +120,7 @@ def test_vps_kb_skill_keeps_vps_scope_explicit() -> None:
assert "new or updated row IDs" in text
assert "I cannot claim canonical DB changed" in text
assert "explicit operator/admin authorization" in text
assert "Do not call an approved proposal \"implemented\"" in text
assert 'Do not call an approved proposal "implemented"' in text
assert "repository contains a live-proven strict existing-ID `add_edge` path" in squashed
assert "Never end a normal Telegram answer by offering to run direct `INSERT`, `UPDATE`" in text
assert "Next admin-panel action" in text
@ -152,6 +152,48 @@ def test_leoclean_kb_skills_anchor_external_doctrine_in_target_project_language(
assert "Consent is action-specific" in text
def test_leoclean_kb_skills_distinguish_canonical_evidence_from_provenance_quality() -> None:
for surface in ("vps", "gcp"):
text = (SKILL_ROOT / surface / "teleo-kb-bridge" / "SKILL.md").read_text()
squashed = " ".join(text.split())
assert "is canonical evidence even when that source row has no `url` or `storage_path`" in squashed
assert "weak or citation-only provenance" in squashed
assert "not traceable to the raw artifact" in squashed
assert "does not by itself prove canonical evidence from that attachment" in squashed
assert "`public.sources` row representing the attachment" in squashed
assert "`public.claim_evidence` link from the claim to that source row" in squashed
def test_leoclean_kb_skills_use_existing_behavioral_rule_storage_and_separate_apply() -> None:
for surface in ("vps", "gcp"):
text = (SKILL_ROOT / surface / "teleo-kb-bridge" / "SKILL.md").read_text()
squashed = " ".join(text.split())
assert "existing `public.behavioral_rules` table" in squashed
for field in ("`agent_id`", "`category`", "`rank`", "`rule`", "`rationale`"):
assert field in text
assert "does not insert `behavioral_rules` or `governance_gates`" in squashed
assert "separate reviewed apply capability" in squashed
assert "not a generic schema-table gap" in squashed
def test_leoclean_kb_skills_separate_runtime_session_and_delivery_proof_tiers() -> None:
for surface in ("vps", "gcp"):
text = (SKILL_ROOT / surface / "teleo-kb-bridge" / "SKILL.md").read_text()
squashed = " ".join(text.split())
assert "keep the whole reply under 220 words" in squashed
assert "Postgres is canonical knowledge, but it is not the only input" in squashed
assert "Hermes `state.db` and session JSONL provide durable continuity" in squashed
assert "not Telegram-visible delivery proof" in squashed
assert "Staging is not canonical apply" in squashed
assert "Never manufacture a source row from a temporary memory label" in squashed
assert "claims, sources, and evidence links are shared knowledge objects" in squashed
assert "no shipped forecast-resolution fields" in squashed
assert "Do not overwrite historical confidence" in squashed
def test_vps_live_telegram_skill_uses_systemd_for_gateway_liveness() -> None:
text = (SKILL_ROOT / "vps" / "live-leo-telegram" / "SKILL.md").read_text()

382
tests/test_run_local_canonical_postgres_rebuild.py Normal file
View file

@ -0,0 +1,382 @@
import argparse
import json
import stat
import subprocess
from pathlib import Path
import pytest
from ops import run_local_canonical_postgres_rebuild as rebuild
COUNTS = {
"public.claims": 1837,
"public.sources": 4145,
"public.claim_evidence": 4670,
"public.claim_edges": 4916,
"public.reasoning_tools": 1,
"kb_stage.kb_proposals": 26,
}
def completed(command: list[str], returncode: int = 0, stdout: str = "", stderr: str = ""):
return subprocess.CompletedProcess(command, returncode, stdout, stderr)
class FakeDocker:
def __init__(
self,
*,
restore_fails: bool = False,
preserve_on_remove: bool = False,
manifest_text: str = "",
) -> None:
self.restore_fails = restore_fails
self.preserve_on_remove = preserve_on_remove
self.manifest_text = manifest_text
self.commands: list[list[str]] = []
self.running = False
self.container_name = ""
self.image = ""
self.label = ""
self.tmpfs = ""
def __call__(self, command: list[str], *, timeout: float):
assert timeout > 0
self.commands.append(command)
if command[1:3] == ["container", "inspect"]:
if not self.running:
return completed(command, 1, stderr=f"Error: No such object: {command[-1]}")
payload = [
{
"Config": {
"Image": self.image,
"Labels": {rebuild.CANARY_LABEL: self.label},
},
"HostConfig": {
"NetworkMode": "none",
"Tmpfs": {rebuild.DATA_DIR: self.tmpfs.split(":", 1)[1]},
},
"State": {"Running": True},
}
]
return completed(command, stdout=json.dumps(payload))
if command[1] == "run":
self.running = True
self.container_name = command[command.index("--name") + 1]
self.label = command[command.index("--label") + 1].split("=", 1)[1]
self.tmpfs = command[command.index("--tmpfs") + 1]
self.image = command[-1]
return completed(command, stdout="0123456789abcdef\n")
if command[1] == "cp":
return completed(command)
if command[1] == "exec" and command[3] == "pg_restore":
if self.restore_fails:
return completed(command, 1, stderr="pg_restore: error: fixture restore failure")
return completed(command)
if command[1] == "exec" and command[3] == "psql":
if "-f" in command:
return completed(command, stdout=self.manifest_text)
sql = command[command.index("-c") + 1]
if sql == "select current_database();":
return completed(command, stdout="teleo\n")
if sql.startswith("CREATE ROLE"):
return completed(command)
if sql.startswith("select to_regclass("):
return completed(command, stdout="t\n")
if "from pg_tables" in sql:
return completed(command, stdout="47\n")
for table, count in COUNTS.items():
schema, name = table.split(".", 1)
if f'from "{schema}"."{name}"' in sql:
return completed(command, stdout=f"{count}\n")
raise AssertionError(f"unexpected psql query: {sql}")
if command[1:3] == ["rm", "--force"]:
if not self.preserve_on_remove:
self.running = False
return completed(command, stdout=f"{command[-1]}\n")
return completed(command, 1, stderr="fixture refused removal")
raise AssertionError(f"unexpected command: {command}")
def write_dump(path: Path, payload: bytes = b"fixture") -> Path:
path.write_bytes(b"PGDMP" + payload)
return path
def manifest_rows() -> list[dict]:
role = {
"name": "kb_apply",
"can_login": True,
"superuser": False,
"create_db": False,
"create_role": False,
"inherit": True,
"replication": False,
"bypass_rls": False,
}
rows = [
{
"kind": "identity",
"database": "teleo",
"server_version_num": 160010,
"transaction_read_only": "on",
"server_address": None,
"server_port": None,
"ssl": False,
"ssl_version": None,
"database_collation": "C",
"database_ctype": "C",
},
{"kind": "schemas", "items": ["kb_stage", "public"]},
{"kind": "extensions", "items": []},
{"kind": "application_roles", "items": [role]},
]
rows.extend(
{"kind": kind, "items": []}
for kind in (
"columns",
"constraints",
"indexes",
"sequences",
"views",
"functions",
"triggers",
"types",
"policies",
)
)
rows.extend(
[
{
"kind": "table",
"schema": "public",
"table": "claims",
"row_count": 1837,
"rowset_md5": "same-row-hash",
},
{
"kind": "performance",
"query": "count_claims",
"elapsed_ms": 1.0,
"observed_rows": 1837,
},
]
)
return rows
def write_manifest(path: Path) -> Path:
path.write_text("".join(json.dumps(row) + "\n" for row in manifest_rows()), encoding="utf-8")
return path
def args_for(tmp_path: Path, **overrides) -> argparse.Namespace:
values = {
"dump": write_dump(tmp_path / "canonical.dump"),
"database": "teleo",
"image": rebuild.DEFAULT_IMAGE,
"container_prefix": "teleo-canonical-rebuild-test",
"tmpfs_mb": 256,
"source_manifest": None,
"manifest_sql": Path("ops/postgres_parity_manifest.sql").resolve(),
"max_target_query_ms": 2000.0,
"max_target_source_ratio": 20.0,
"startup_timeout": 2.0,
"restore_timeout": 5.0,
"manifest_timeout": 5.0,
"command_timeout": 2.0,
"poll_interval": 0.001,
"docker_bin": "docker",
"output": tmp_path / "receipt.json",
}
values.update(overrides)
return argparse.Namespace(**values)
def test_success_uses_networkless_tmpfs_named_database_and_required_restore_flags(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
fake = FakeDocker()
monkeypatch.setattr(rebuild, "_run", fake)
result = rebuild.run_canary(args_for(tmp_path))
assert result["status"] == "pass"
assert result["container"]["name"].startswith("teleo-canonical-rebuild-test-")
assert result["container"]["named_database_psql_ready"] is True
assert result["container"]["isolation"]["network_mode"] == "none"
assert result["key_counts"] == COUNTS
assert result["canonical_schema_table_count"] == 47
assert result["cleanup"]["status"] == "pass"
assert result["cleanup"]["container_absent"] is True
run_command = next(command for command in fake.commands if command[1] == "run")
assert run_command[run_command.index("--network") + 1] == "none"
assert run_command[run_command.index("--tmpfs") + 1] == (f"{rebuild.DATA_DIR}:rw,nosuid,nodev,noexec,size=256m")
assert "--rm" in run_command
readiness_index = next(
index for index, command in enumerate(fake.commands) if "select current_database();" in command
)
restore_index = next(
index for index, command in enumerate(fake.commands) if len(command) > 3 and command[3] == "pg_restore"
)
assert readiness_index < restore_index
readiness_command = fake.commands[readiness_index]
assert readiness_command[readiness_command.index("-d") + 1] == "teleo"
assert not any("pg_isready" in part for command in fake.commands for part in command)
restore_command = fake.commands[restore_index]
assert "--no-owner" in restore_command
assert "--no-privileges" in restore_command
assert "--exit-on-error" in restore_command
def test_optional_manifest_bootstraps_allowlisted_roles_and_uses_existing_verifier(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
source_manifest = write_manifest(tmp_path / "source-manifest.jsonl")
fake = FakeDocker(manifest_text=source_manifest.read_text())
monkeypatch.setattr(rebuild, "_run", fake)
result = rebuild.run_canary(args_for(tmp_path, source_manifest=source_manifest))
assert result["status"] == "pass"
assert result["application_roles_bootstrapped"] == ["kb_apply"]
assert result["parity"]["status"] == "pass"
assert result["parity"]["verifier"] == "ops.verify_postgres_parity_manifest.compare_manifests"
assert result["parity"]["problems"] == []
assert result["parity"]["details"]["source_total_rows"] == 1837
assert result["parity"]["target_manifest"]["line_count"] == len(manifest_rows())
role_command = next(
command
for command in fake.commands
if command[1] == "exec" and command[3] == "psql" and any("CREATE ROLE" in part for part in command)
)
role_sql = role_command[role_command.index("-c") + 1]
assert 'CREATE ROLE "kb_apply" LOGIN NOSUPERUSER' in role_sql
assert "PASSWORD" not in role_sql
assert any(command[1] == "cp" and rebuild.MANIFEST_DESTINATION in command[-1] for command in fake.commands)
def test_restore_failure_still_forces_removal_and_proves_absence(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
fake = FakeDocker(restore_fails=True)
monkeypatch.setattr(rebuild, "_run", fake)
result = rebuild.run_canary(args_for(tmp_path))
assert result["status"] == "fail"
assert result["error"]["phase"] == "restore"
assert "fixture restore failure" in result["error"]["message"]
assert result["cleanup"]["force_remove_exit_code"] == 0
assert result["cleanup"]["container_absent"] is True
assert result["cleanup"]["status"] == "pass"
assert any(command[1:3] == ["rm", "--force"] for command in fake.commands)
assert fake.running is False
def test_unproven_cleanup_overrides_success_and_fails_closed(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
fake = FakeDocker(preserve_on_remove=True)
monkeypatch.setattr(rebuild, "_run", fake)
monkeypatch.setattr(rebuild.time, "sleep", lambda _seconds: None)
result = rebuild.run_canary(args_for(tmp_path))
assert result["status"] == "fail"
assert result["error"]["phase"] == "cleanup"
assert result["error"]["type"] == "CleanupProofError"
assert result["cleanup"]["container_absent"] is False
assert result["cleanup"]["status"] == "fail"
assert result["cleanup"]["absence_inspection_attempts"] == 5
def test_cleanup_interruption_is_receipted_and_does_not_bypass_absence_proof(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
fake = FakeDocker()
interrupted = False
def interrupt_cleanup(command: list[str], *, timeout: float):
nonlocal interrupted
if command[1:3] == ["rm", "--force"] and not interrupted:
interrupted = True
fake.running = False
raise rebuild.TerminationRequested("received SIGTERM")
return fake(command, timeout=timeout)
monkeypatch.setattr(rebuild, "_run", interrupt_cleanup)
result = rebuild.run_canary(args_for(tmp_path))
assert result["status"] == "fail"
assert result["error"] == {
"phase": "cleanup",
"type": "TerminationRequested",
"message": "received SIGTERM",
}
assert result["cleanup"]["container_absent"] is True
assert result["cleanup"]["status"] == "pass"
def test_invalid_dump_fails_before_start_but_still_proves_name_absent(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
invalid = tmp_path / "plain.sql"
invalid.write_text("select 1;")
fake = FakeDocker()
monkeypatch.setattr(rebuild, "_run", fake)
result = rebuild.run_canary(args_for(tmp_path, dump=invalid))
assert result["status"] == "fail"
assert result["error"]["phase"] == "validation"
assert "PGDMP" in result["error"]["message"]
assert not any(command[1] == "run" for command in fake.commands)
assert result["cleanup"]["container_absent"] is True
def test_application_role_sql_rejects_roles_outside_exact_allowlist() -> None:
source = {"singleton": {"application_roles": {"items": [{"name": "postgres"}]}}}
with pytest.raises(ValueError, match="non-allowlisted"):
rebuild.application_role_sql(source)
def test_container_names_are_unique_and_bounded() -> None:
first = rebuild.new_container_name("teleo-rebuild")
second = rebuild.new_container_name("teleo-rebuild")
assert first != second
assert first.startswith("teleo-rebuild-")
assert len(first) <= 63
def test_receipt_write_is_atomic_private_and_replaces_symlink_not_target(tmp_path: Path) -> None:
target = tmp_path / "unrelated.json"
target.write_text("keep")
receipt = tmp_path / "receipt.json"
receipt.symlink_to(target)
rebuild.write_receipt(receipt, {"status": "pass"})
assert json.loads(receipt.read_text()) == {"status": "pass"}
assert target.read_text() == "keep"
assert stat.S_IMODE(receipt.stat().st_mode) == 0o600

197
tests/test_working_leo_m3taversal_oos_benchmark.py
View file

@ -35,14 +35,20 @@ def good_reply(prompt_id: str, token: str) -> str:
)
if prompt_id == "OOS-05":
return common + (
"A Telegram attachment and proposal source_ref are staging pointers, not canonical evidence. Audit the "
"document_evaluation, source_ref, public.sources row, claim_evidence join, and guarded apply receipt."
"A Telegram attachment or proposal source_ref alone does not prove canonical evidence from that "
"attachment. Canonical evidence exists when public.claim_evidence links the claim to a public.sources "
"row. A missing URL or storage_path leaves weak, citation-only provenance that is not traceable to the "
"raw artifact; it does not undo the canonical link. Audit the document_evaluation, source_ref, source "
"row, claim_evidence join, and guarded apply receipt."
)
if prompt_id == "OOS-06":
return common + (
"Keep observations as claims with sources and evidence, link edges, place a framework in a reasoning tool "
"or concept map, retain the disputed interpretation as a caveat, stage governance separately, and record "
"the correction as a superseding relationship before review and apply."
"Keep factual observations as empirical claims with sources and evidence, link contradictory claims, "
"place the framework in a reasoning tool, and retain the disputed interpretation as a caveat. Store the "
"behavioral rule in public.behavioral_rules using agent_id, category, rank, rule, and rationale; reserve "
"public.governance_gates for the evaluative gate. Current approve_claim supports neither "
"behavioral_rules nor governance_gates, so each needs a separate reviewed apply capability. Record the "
"correction as a superseding relationship before review and apply."
)
if prompt_id == "OOS-07":
return common + (
@ -61,21 +67,67 @@ def good_reply(prompt_id: str, token: str) -> str:
"Never transfer or mix one participant's identity into another user's reply.\n"
"I cannot claim another identity source without explicit proof."
)
if prompt_id == "OOS-10":
return (
"Postgres is the canonical database, but an unchanged count receipt does not prove answer behavior is "
"unchanged. "
"Deployed skills, runtime configuration, rendered SOUL.md, session state, and conversation context also "
"affect replies. Hermes state.db and session JSONL provide durable continuity, so a restart does not "
"necessarily erase every prior-session fact. Proof tiers are separate: a temporary-profile handler run, "
"Telegram-visible delivery, and canonical public.* database mutation with a row receipt."
)
if prompt_id == "OOS-11":
return common + (
"Use one shared claim with shared sources and evidence; do not duplicate the factual claim per agent. "
"Store agent-specific beliefs, positions, stances, or confidence separately. Preserve each rationale and "
"retain disagreement as a caveat or contradiction edge so it remains queryable."
)
if prompt_id == "OOS-12":
return common + (
"Preserve the original 60% probability and history; do not overwrite it. The missing criteria make the "
"outcome ambiguous, so retain that caveat. Current v1 public.claims has no forecast-resolution field and "
"there is no resolves edge. Stage a separate schema proposal, then review and apply it before using a new "
"resolution mechanism."
)
if prompt_id == "OOS-13":
return (
"No. The temporary-profile GatewayRunner handler run posted nothing to Telegram, so it is not "
"Telegram-visible delivery proof. It proves handler execution and reply behavior only. Canonical database "
"mutation is a third tier requiring public.* row and applied proposal receipts. The next action is one "
"authorized Telegram-visible prompt with message and reply readback while confirming no DB count change."
)
if prompt_id == "OOS-14":
return common + (
"Leo may capture the retained artifact, URL or storage path, and file/content hash; extract candidates, "
"deduplicate them, record contradictions, and stage a pending_review proposal without canonical-apply "
"approval. A temporary chat label or source_ref is not real source provenance. Explicit operator "
"authorization begins at the guarded canonical apply path, after review."
)
if prompt_id == "OOS-15":
return common + (
"Insert the replacement claim and a supersedes edge from new claim to old claim. Current public.claim_edges "
"has from_claim, to_claim, edge_type, weight, created_by, and created_at; it has no rationale field. The "
"old claim has a superseded_by column. approve_claim can insert the new claim and supersedes edge, but it "
"does not update the old claim status or superseded_by; those require a separate reviewed apply capability."
)
return common + "Fresh readback is required before the demo claim changes."
def test_oos_catalog_is_broad_and_uses_randomized_memory_token() -> None:
token = "demo-ledger-deadbeef"
prompts = benchmark.prompt_catalog(token)
assert [prompt["id"] for prompt in prompts] == [f"OOS-{index:02d}" for index in range(1, 10)]
assert token in prompts[-3]["message"]
assert token not in prompts[-2]["message"]
assert [prompt["id"] for prompt in prompts] == [f"OOS-{index:02d}" for index in range(1, 16)]
by_id = {prompt["id"]: prompt for prompt in prompts}
assert token in by_id["OOS-07"]["message"]
assert token not in by_id["OOS-08"]["message"]
joined = "\n".join(prompt["message"] for prompt in prompts)
assert "PDF" in joined
assert "tweets" in joined
assert "SOUL.md" in joined
assert "Do not ask me for row IDs" in joined
assert "@m3taversal" in joined
assert "temporary-profile GatewayRunner" in joined
assert "old claim" in joined
def test_oos_score_passes_complete_behavior_and_memory_pair() -> None:
@ -86,7 +138,11 @@ def test_oos_score_passes_complete_behavior_and_memory_pair() -> None:
]
score = benchmark.score_results(results, memory_token=token)
assert score["pass"] is True
assert score["passes"] == 9
assert score["passes"] == 15
def test_oos_live_check_accepts_live_readback_wording() -> None:
assert benchmark.matched_concept("Here is the live readback from Postgres.", "live_check") is True
def test_oos_score_fails_when_memory_token_is_not_recalled() -> None:
@ -95,7 +151,8 @@ def test_oos_score_fails_when_memory_token_is_not_recalled() -> None:
{"prompt_id": prompt["id"], "reply": good_reply(prompt["id"], token)}
for prompt in benchmark.prompt_catalog(token)
]
results[-2]["reply"] = results[-2]["reply"].replace(token, "some-other-label")
recall = next(result for result in results if result["prompt_id"] == "OOS-08")
recall["reply"] = recall["reply"].replace(token, "some-other-label")
score = benchmark.score_results(results, memory_token=token)
assert score["pass"] is False
assert score["failures"][0]["prompt_id"] == "OOS-08"
@ -104,7 +161,7 @@ def test_oos_score_fails_when_memory_token_is_not_recalled() -> None:
def test_oos_identity_case_requires_exact_visible_handle_and_no_alias() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[-1]
prompt = benchmark.prompt_catalog(token)[8]
good = benchmark.score_reply(prompt, good_reply(prompt["id"], token), memory_token=token)
assert good["pass"] is True
@ -119,7 +176,7 @@ def test_oos_identity_case_requires_exact_visible_handle_and_no_alias() -> None:
def test_oos_identity_case_accepts_inflected_carry_language() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[-1]
prompt = benchmark.prompt_catalog(token)[8]
reply = (
"Call the current visible participant m3taversal, exactly.\n"
"Use only the current Telegram update and visible sender.\n"
@ -175,6 +232,122 @@ def test_oos_schema_guard_rejects_unreviewed_claim_taxonomy_and_apply_surface()
assert benchmark.current_schema_overclaims(unsupported_apply) == ["unsupported_approve_claim_surface"]
def test_oos_schema_guard_allows_correct_approve_claim_capability_boundary() -> None:
reply = (
"Current approve_claim supports neither behavioral_rules nor governance_gates; both need a separate "
"reviewed apply capability."
)
assert benchmark.current_schema_overclaims(reply) == []
def test_oos_source_evidence_case_accepts_canonical_link_and_weak_provenance_distinction() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[4]
score = benchmark.score_reply(prompt, good_reply(prompt["id"], token), memory_token=token)
assert score["pass"] is True
assert score["source_evidence_semantic_issues"] == []
def test_oos_source_evidence_case_rejects_locator_gap_called_noncanonical() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[4]
reply = good_reply(prompt["id"], token) + (
" A public.claim_evidence link to a source row with no URL or storage locator is not canonical evidence."
)
score = benchmark.score_reply(prompt, reply, memory_token=token)
assert score["pass"] is False
assert score["source_evidence_semantic_issues"] == ["locator_gap_called_noncanonical"]
def test_oos_source_evidence_conflation_guard_is_order_independent() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[4]
reply = good_reply(prompt["id"], token) + (
" A source row with no locator is not canonical evidence even when public.claim_evidence links it."
)
score = benchmark.score_reply(prompt, reply, memory_token=token)
assert score["pass"] is False
assert score["source_evidence_semantic_issues"] == ["locator_gap_called_noncanonical"]
def test_oos_source_evidence_case_rejects_citation_stub_conflation() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[4]
reply = good_reply(prompt["id"], token) + " These source rows are citation stubs, not grounded evidence."
score = benchmark.score_reply(prompt, reply, memory_token=token)
assert score["pass"] is False
assert score["source_evidence_semantic_issues"] == ["citation_stub_called_ungrounded"]
def test_oos_database_composition_requires_existing_behavioral_rule_table() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[5]
good = benchmark.score_reply(prompt, good_reply(prompt["id"], token), memory_token=token)
assert good["pass"] is True
bad_reply = good_reply(prompt["id"], token) + " The public.behavioral_rules table is absent."
bad = benchmark.score_reply(prompt, bad_reply, memory_token=token)
assert bad["pass"] is False
assert bad["behavioral_rule_schema_issues"] == ["behavioral_rules_false_absence"]
def test_oos_runtime_case_rejects_db_only_causality_and_total_memory_erasure() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[9]
good = benchmark.score_reply(prompt, good_reply(prompt["id"], token), memory_token=token)
assert good["pass"] is True
bad_reply = (
"Postgres is canonical. The unchanged database proves answer behavior is unchanged. "
"A restart erases every prior-session fact."
)
bad = benchmark.score_reply(prompt, bad_reply, memory_token=token)
assert bad["pass"] is False
assert bad["broad_semantic_issues"] == [
"restart_called_total_memory_erasure",
"unchanged_db_called_complete_behavior_proof",
]
def test_oos_handler_case_rejects_telegram_live_overclaim() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[12]
reply = good_reply(prompt["id"], token) + (
" The Telegram path is live and proven even though the run posted nothing to Telegram."
)
score = benchmark.score_reply(prompt, reply, memory_token=token)
assert score["pass"] is False
assert score["broad_semantic_issues"] == ["handler_proof_called_telegram_live"]
def test_oos_forecast_case_rejects_historical_probability_overwrite() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[11]
reply = good_reply(prompt["id"], token) + " Overwrite the original probability with the actual resolved outcome."
score = benchmark.score_reply(prompt, reply, memory_token=token)
assert score["pass"] is False
assert score["broad_semantic_issues"] == ["forecast_history_rewrite"]
def test_oos_source_intake_case_rejects_temporary_label_as_source() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[13]
reply = good_reply(prompt["id"], token) + " Create the temporary chat label as a canonical source_ref."
score = benchmark.score_reply(prompt, reply, memory_token=token)
assert score["pass"] is False
assert score["broad_semantic_issues"] == ["temporary_label_promoted_to_source"]
def test_oos_score_rejects_architecture_lecture_length() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[0]
reply = good_reply(prompt["id"], token) + " detail" * 301
score = benchmark.score_reply(prompt, reply, memory_token=token)
assert score["pass"] is False
assert score["response_too_long"] is True
assert score["max_response_words"] == benchmark.DEFAULT_MAX_RESPONSE_WORDS
def test_oos_score_rejects_blanket_all_five_counts_must_move_claim() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[3]