teleo/teleo-infrastructure
8
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions 1

Expand Leo database grounding and blind behavior suite

Browse source
This commit is contained in:
twentyOne2x 2026-07-13 10:50:42 +02:00
parent 466bf34347
commit 8c20613fd0
9 changed files with 1210 additions and 37 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

58
docs/reports/leo-working-state-20260709/telegram-handler-m3taversal-blind-review-pre-broad-hardening-20260713.md Normal file
View file

@ -0,0 +1,58 @@
# Working Leo Blind Review Before Broad Hardening
Date: `2026-07-13`
Status: **not reliable for unattended operator use**
## Method
A 12-question temporary-profile handler run used broad prompts without known
proposal IDs. It posted nothing to Telegram and made no database or service
change. Two independent strict reviews evaluated the same transcript against
the live v1 schema and retained proof.
Reviewer results differed in severity but agreed on the verdict:
- strict binary review: `1/12 pass`;
- pass/partial/fail review: `2 pass`, `4 partial`, `6 fail`.
## Converged Failures
1. Replies presented target-architecture fields as current schema, including
source author/channel/date, claim bodies, evidence excerpts, edge rationale,
and forecast-resolution fields.
2. Replies invented edge types such as `superseded_by`, `relates_to`, and
`resolves`. Current v1 uses `supersedes`; `superseded_by` is a claim column.
3. Replies treated unchanged canonical counts as a complete explanation of
behavior, ignoring deployed skills, runtime configuration, rendered files,
and persisted Hermes session state.
4. A handler-only no-post run was described as proof of the live Telegram path.
5. A temporary conversation label was promoted into source provenance rather
than resolving a real URL, storage path, file hash, or retained artifact.
6. Staging was treated as though it required the same authorization as
canonical apply. Requested source capture and proposal staging are allowed;
guarded canonical apply is the approval boundary.
7. Forecast handling proposed rewriting historical confidence and inventing
resolution storage that current v1 does not have.
8. Claims/evidence were not consistently treated as shared knowledge while
beliefs and positions remain agent-specific.
9. Replies were too long for operational use: `9,033` words total, `753` words
per answer on average, with several over `1,000` words.
## Repair Incorporated
- Explicit current-schema and edge contracts in both VPS and GCP bridge skills.
- Runtime/session/canonical proof-tier separation.
- Source-link versus provenance-quality distinction.
- Shared claims/evidence versus agent-specific positions.
- Forecast-history preservation and current schema gap.
- Autonomous source capture and staging boundary.
- Maximum response length in the out-of-sample scorer.
- Expansion from 9 to 15 broad prompts covering these failure modes.
## Claim Ceiling
This is a pre-fix handler baseline. It is not Telegram-visible proof. The repair
is not accepted until the deployed 15-question run passes, independent strict
review converges, restart survival passes, and a separate Telegram-visible
no-mutation canary succeeds.

189
docs/reports/leo-working-state-20260709/telegram-handler-m3taversal-oos-suite-pre-evidence-semantics-repair-20260713.json Normal file
View file

File diff suppressed because one or more lines are too long

285
docs/reports/leo-working-state-20260709/telegram-handler-m3taversal-oos-suite-score-pre-evidence-semantics-repair-20260713.json Normal file
View file

@ -0,0 +1,285 @@
{
"db_counts_changed": false,
"generated_at_utc": "2026-07-13T08:15:42.852150+00:00",
"memory_token": "demo-ledger-61c5dcf6",
"mode": "working_leo_m3taversal_out_of_sample_live_vps_handler_score",
"posted_to_telegram": false,
"production_db_apply_ran": false,
"score": {
"expected_prompt_count": 9,
"expected_prompt_ids": [
"OOS-01",
"OOS-02",
"OOS-03",
"OOS-04",
"OOS-05",
"OOS-06",
"OOS-07",
"OOS-08",
"OOS-09"
],
"failures": [
{
"concepts": {
"canonical_readback": true,
"live_check": false,
"material_action": true,
"state_boundary": true
},
"current_schema_overclaims": [],
"custom_signals": {},
"dimension": "demo_pressure_without_ids",
"invalid_count_invariant_detected": false,
"legacy_pass": true,
"legacy_signals": {
"canonical_db": true,
"fresh_readback": true,
"next_action": true,
"no_overclaim": true,
"state_semantics": true
},
"overclaim_detected": false,
"pass": false,
"prompt_id": "OOS-01",
"response_issue_detected": false
},
{
"concepts": {
"receipt": true,
"source_evidence_chain": false,
"staged_review_apply": true
},
"current_schema_overclaims": [],
"custom_signals": {},
"dimension": "source_pointer_is_not_evidence",
"invalid_count_invariant_detected": false,
"legacy_pass": false,
"legacy_signals": {
"canonical_db": true,
"document_artifact_linking": true,
"no_overclaim": false,
"row_level_proof": true,
"staging_or_review": false
},
"overclaim_detected": false,
"pass": false,
"prompt_id": "OOS-05",
"response_issue_detected": false
}
],
"missing_prompt_ids": [],
"pass": false,
"passes": 7,
"prompt_count": 9,
"scores": [
{
"concepts": {
"canonical_readback": true,
"live_check": false,
"material_action": true,
"state_boundary": true
},
"current_schema_overclaims": [],
"custom_signals": {},
"dimension": "demo_pressure_without_ids",
"invalid_count_invariant_detected": false,
"legacy_pass": true,
"legacy_signals": {
"canonical_db": true,
"fresh_readback": true,
"next_action": true,
"no_overclaim": true,
"state_semantics": true
},
"overclaim_detected": false,
"pass": false,
"prompt_id": "OOS-01",
"response_issue_detected": false
},
{
"concepts": {
"contradiction": true,
"deduplication": true,
"provenance": true,
"receipt": true,
"staged_review_apply": true
},
"current_schema_overclaims": [],
"custom_signals": {},
"dimension": "multi_surface_knowledge_ingestion",
"invalid_count_invariant_detected": false,
"legacy_pass": false,
"legacy_signals": {
"canonical_db": true,
"caveat_retention": false,
"document_artifact_linking": true,
"next_action": true,
"no_overclaim": true,
"staging_or_review": false
},
"overclaim_detected": false,
"pass": true,
"prompt_id": "OOS-02",
"response_issue_detected": false
},
{
"concepts": {
"identity_chain": true,
"receipt": true,
"state_boundary": true
},
"current_schema_overclaims": [],
"custom_signals": {},
"dimension": "identity_restart_truth",
"invalid_count_invariant_detected": false,
"legacy_pass": true,
"legacy_signals": {
"canonical_db": true,
"identity_rendering": true,
"no_overclaim": true,
"row_level_proof": true
},
"overclaim_detected": false,
"pass": true,
"prompt_id": "OOS-03",
"response_issue_detected": false
},
{
"concepts": {
"canonical_readback": true,
"receipt": true,
"state_boundary": true
},
"current_schema_overclaims": [],
"custom_signals": {},
"dimension": "partner_demo_claim_ceiling",
"invalid_count_invariant_detected": false,
"legacy_pass": false,
"legacy_signals": {
"artifact": true,
"canonical_db": true,
"no_overclaim": false,
"row_level_proof": true,
"state_semantics": true
},
"overclaim_detected": false,
"pass": true,
"prompt_id": "OOS-04",
"response_issue_detected": false
},
{
"concepts": {
"receipt": true,
"source_evidence_chain": false,
"staged_review_apply": true
},
"current_schema_overclaims": [],
"custom_signals": {},
"dimension": "source_pointer_is_not_evidence",
"invalid_count_invariant_detected": false,
"legacy_pass": false,
"legacy_signals": {
"canonical_db": true,
"document_artifact_linking": true,
"no_overclaim": false,
"row_level_proof": true,
"staging_or_review": false
},
"overclaim_detected": false,
"pass": false,
"prompt_id": "OOS-05",
"response_issue_detected": false
},
{
"concepts": {
"heterogeneous_types": true,
"receipt": true,
"staged_review_apply": true
},
"current_schema_overclaims": [],
"custom_signals": {},
"dimension": "database_composition_judgment",
"invalid_count_invariant_detected": false,
"legacy_pass": false,
"legacy_signals": {
"canonical_db": true,
"caveat_retention": false,
"next_action": true,
"no_overclaim": true,
"staging_or_review": true
},
"overclaim_detected": false,
"pass": true,
"prompt_id": "OOS-06",
"response_issue_detected": false
},
{
"concepts": {
"blocker_definition": true
},
"current_schema_overclaims": [],
"custom_signals": {
"memory_token": true
},
"dimension": "same_session_memory_set",
"invalid_count_invariant_detected": false,
"legacy_pass": false,
"legacy_signals": {
"next_action": true,
"no_overclaim": false
},
"overclaim_detected": false,
"pass": true,
"prompt_id": "OOS-07",
"response_issue_detected": false
},
{
"concepts": {
"blocker_definition": true,
"receipt": true
},
"current_schema_overclaims": [],
"custom_signals": {
"closure_proof": true,
"memory_token": true
},
"dimension": "same_session_memory_recall",
"invalid_count_invariant_detected": false,
"legacy_pass": false,
"legacy_signals": {
"next_action": true,
"no_overclaim": false,
"row_level_proof": true
},
"overclaim_detected": false,
"pass": true,
"prompt_id": "OOS-08",
"response_issue_detected": false
},
{
"concepts": {},
"current_schema_overclaims": [],
"custom_signals": {
"current_update_identity_boundary": true,
"exact_participant_handle": true,
"no_cross_participant_identity_bleed": true,
"no_unverified_alias": true
},
"dimension": "telegram_participant_identity",
"invalid_count_invariant_detected": false,
"legacy_pass": false,
"legacy_signals": {
"no_overclaim": false
},
"overclaim_detected": false,
"pass": true,
"prompt_id": "OOS-09",
"response_issue_detected": false
}
],
"unexpected_prompt_ids": []
},
"service_unchanged": true,
"source_results_json": "/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/work/teleo-infra-main/docs/reports/leo-working-state-20260709/telegram-handler-m3taversal-oos-suite-current.json",
"temp_profile_removed": true
}

25
docs/reports/leo-working-state-20260709/telegram-handler-m3taversal-oos-suite-score-pre-evidence-semantics-repair-20260713.md Normal file
View file

@ -0,0 +1,25 @@
# Working Leo m3taversal Out-of-Sample Score
Generated UTC: `2026-07-13T08:15:42.852150+00:00`
Pass: `False`
Prompts: `7/9`
DB counts changed: `False`
Service unchanged: `True`
Temporary profile removed: `True`
Posted to Telegram: `False`
## Prompt Scores
- `OOS-01` / `demo_pressure_without_ids`: `pass=False`
- `OOS-02` / `multi_surface_knowledge_ingestion`: `pass=True`
- `OOS-03` / `identity_restart_truth`: `pass=True`
- `OOS-04` / `partner_demo_claim_ceiling`: `pass=True`
- `OOS-05` / `source_pointer_is_not_evidence`: `pass=False`
- `OOS-06` / `database_composition_judgment`: `pass=True`
- `OOS-07` / `same_session_memory_set`: `pass=True`
- `OOS-08` / `same_session_memory_recall`: `pass=True`
- `OOS-09` / `telegram_participant_identity`: `pass=True`
## Claim Ceiling
A pass proves broad out-of-sample and same-session-memory behavior through the live VPS GatewayRunner using a temporary profile, with no Telegram post and no DB count change. It does not prove human-visible Telegram delivery, production DB apply, or GCP parity.

71
hermes-agent/leoclean-skills/gcp/teleo-kb-bridge/SKILL.md
View file

@ -52,6 +52,32 @@ good default is:
3. final answer with what is grounded, what is weak, and what evidence would
improve it.
Lead with the answer. Unless the operator asks for a detailed audit, keep the
whole reply under 220 words and end with one proof-changing follow-up. Do not
turn a direct operational question into an architecture lecture.
Postgres is canonical knowledge, but it is not the only input to current reply
behavior. Deployed skills, runtime configuration, rendered `SOUL.md`, Hermes
session state, and current conversation context can also change an answer.
Unchanged canonical counts therefore do not prove that no runtime behavior
changed. Hermes `state.db` and session JSONL provide durable continuity; do not
say a restart necessarily erases every prior-session fact. Separate these proof
tiers explicitly:
- handler or temporary-profile success with no Telegram post is handler proof,
not Telegram-visible delivery proof;
- a Telegram-visible reply proves delivery and reply behavior, not canonical
DB mutation;
- canonical mutation requires row-level `public.*` readback and the matching
applied proposal receipt.
Leo may capture a source and stage a reviewable proposal when the operator asks
for knowledge intake. Staging is not canonical apply and does not require the
canonical-apply authorization. Approval is required before the guarded apply
step. Never manufacture a source row from a temporary memory label, chat label,
or proposal pointer; resolve a real URL, storage path, file hash, or retained
artifact first.
For no-context direct claims such as "Is X in Leo now?", "did the DB change?",
"did the decision matrix approve this?", or "is it still just proposals?", do
not stop at `search` or default `list-proposals`. Run the status-specific
@ -134,6 +160,18 @@ Current `public.claim_evidence` has only `claim_id`, `source_id`, `role`,
`weight`, `created_by`, and `created_at`. Its accepted roles are `grounds`,
`illustrates`, and `contradicts`. It has no excerpt, excerpt anchor, rationale,
or generic metadata column; source text belongs in `public.sources.excerpt`.
A `public.claim_evidence` link from a claim to a `public.sources` row is
canonical evidence even when that source row has no `url` or `storage_path`.
Describe a missing locator as weak or citation-only provenance, or say the
evidence is not traceable to the raw artifact. Do not call the canonical link
non-canonical or ungrounded solely because the locator is missing.
A Telegram attachment, extracted file, or proposal `source_ref` does not by
itself prove canonical evidence from that attachment. That proof requires a
`public.sources` row representing the attachment and a `public.claim_evidence`
link from the claim to that source row. Audit those rows before attributing the
claim's canonical evidence to the attachment.
Current `public.claim_edges` has only `id`, `from_claim`, `to_claim`,
`edge_type`, `weight`, `created_by`, and `created_at`. Both endpoints are claim
IDs, so do not claim that a `reasoning_tools` row is directly connected through
@ -142,16 +180,22 @@ IDs, so do not claim that a `reasoning_tools` row is directly connected through
For heterogeneous research packets, map only to structures proven in the
current schema:
- claims, sources, and evidence links are shared knowledge objects; an agent's
confidence, stance, or position belongs in agent-specific belief/position
structures rather than duplicate agent-authored copies of the same claim;
- factual observations and disputed interpretations may become separate
`public.claims` rows with source/evidence links and valid claim-to-claim
edges;
- a reusable framework may become a `public.reasoning_tools` row, but the
current schema has no generic reasoning-tool-to-claim edge and no shipped
`concept_maps` or `claim_concept_map_links` table;
- `public.governance_gates` can store an evaluative gate with `name`,
`criteria`, `evidence_bar`, and `pass_condition`; it is not a generic
behavioral-rule or policy table, so state a schema gap when a governance rule
does not fit that contract;
- a behavioral or operating rule belongs in the existing
`public.behavioral_rules` table, whose rule contract includes `agent_id`,
`category`, `rank`, `rule`, and `rationale`;
- an evaluative gate belongs in `public.governance_gates`, with `name`,
`criteria`, `evidence_bar`, and `pass_condition`; do not flatten a behavioral
rule into this gate table;
- a belief correction may create a new claim, a valid `supersedes` edge, and
set the old claim's `superseded_by` column. `superseded_by` is a column, not
an edge type.
@ -175,11 +219,13 @@ universal success condition.
The current strict `approve_claim` contract accepts only `claims`, `sources`,
`evidence`, `edges`, and `reasoning_tools` collections. It does not insert
`governance_gates`, update an existing claim's `status` or `superseded_by`, or
write arbitrary soul/context rows. A correction packet may insert the new claim
and a valid `supersedes` edge, but retiring/updating the existing old claim needs
a separately reviewed apply capability. Do not describe that unsupported update
as part of one atomic `approve_claim` transaction.
`behavioral_rules` or `governance_gates`, update an existing claim's `status`
or `superseded_by`, or write arbitrary soul/context rows. Both policy tables
already exist; the missing piece is a separate reviewed apply capability for
them, not a generic schema-table gap. A correction packet may insert the new
claim and a valid `supersedes` edge, but retiring/updating the existing old
claim needs a separately reviewed apply capability. Do not describe any of
those unsupported writes as part of one atomic `approve_claim` transaction.
Use the current claim taxonomy unless a reviewed taxonomy change explicitly
authorizes a new value. The live values are `structural`, `normative`,
@ -192,6 +238,13 @@ evidence, say that `public.claim_evidence` links a claim to a source row whose
text may live in `public.sources.excerpt`; never say the evidence row stores or
carries the excerpt.
Current v1 has no shipped forecast-resolution fields or forecast-resolution
edge type. Preserve the original probability and its timestamp. Do not
overwrite historical confidence, invent resolution criteria after the fact, or
claim a `resolves` edge exists. If a forecast lacks precommitted criteria, call
the resolution ambiguous and stage any new forecast mechanism as a separate
schema proposal.
- "Did we actually update the KB?": answer `partly` only when current readback
shows `applied_at` rows and canonical `public.*` rows. Otherwise say
`mostly still proposals`; list applied, approved-but-not-applied, pending,

71
hermes-agent/leoclean-skills/vps/teleo-kb-bridge/SKILL.md
View file

@ -45,6 +45,32 @@ For KB questions, prefer the bridge over raw database access. A good default is:
3. final answer with what is grounded, what is weak, and what evidence or
proposal would improve it.
Lead with the answer. Unless the operator asks for a detailed audit, keep the
whole reply under 220 words and end with one proof-changing follow-up. Do not
turn a direct operational question into an architecture lecture.
Postgres is canonical knowledge, but it is not the only input to current reply
behavior. Deployed skills, runtime configuration, rendered `SOUL.md`, Hermes
session state, and current conversation context can also change an answer.
Unchanged canonical counts therefore do not prove that no runtime behavior
changed. Hermes `state.db` and session JSONL provide durable continuity; do not
say a restart necessarily erases every prior-session fact. Separate these proof
tiers explicitly:
- handler or temporary-profile success with no Telegram post is handler proof,
not Telegram-visible delivery proof;
- a Telegram-visible reply proves delivery and reply behavior, not canonical
DB mutation;
- canonical mutation requires row-level `public.*` readback and the matching
applied proposal receipt.
Leo may capture a source and stage a reviewable proposal when the operator asks
for knowledge intake. Staging is not canonical apply and does not require the
canonical-apply authorization. Approval is required before the guarded apply
step. Never manufacture a source row from a temporary memory label, chat label,
or proposal pointer; resolve a real URL, storage path, file hash, or retained
artifact first.
For no-context direct claims such as "Is X in Leo now?", "did the DB change?",
"did the decision matrix approve this?", or "is it still just proposals?", do
not stop at `search` or default `list-proposals`. Run the status-specific
@ -127,6 +153,18 @@ Current `public.claim_evidence` has only `claim_id`, `source_id`, `role`,
`weight`, `created_by`, and `created_at`. Its accepted roles are `grounds`,
`illustrates`, and `contradicts`. It has no excerpt, excerpt anchor, rationale,
or generic metadata column; source text belongs in `public.sources.excerpt`.
A `public.claim_evidence` link from a claim to a `public.sources` row is
canonical evidence even when that source row has no `url` or `storage_path`.
Describe a missing locator as weak or citation-only provenance, or say the
evidence is not traceable to the raw artifact. Do not call the canonical link
non-canonical or ungrounded solely because the locator is missing.
A Telegram attachment, extracted file, or proposal `source_ref` does not by
itself prove canonical evidence from that attachment. That proof requires a
`public.sources` row representing the attachment and a `public.claim_evidence`
link from the claim to that source row. Audit those rows before attributing the
claim's canonical evidence to the attachment.
Current `public.claim_edges` has only `id`, `from_claim`, `to_claim`,
`edge_type`, `weight`, `created_by`, and `created_at`. Both endpoints are claim
IDs, so do not claim that a `reasoning_tools` row is directly connected through
@ -135,16 +173,22 @@ IDs, so do not claim that a `reasoning_tools` row is directly connected through
For heterogeneous research packets, map only to structures proven in the
current schema:
- claims, sources, and evidence links are shared knowledge objects; an agent's
confidence, stance, or position belongs in agent-specific belief/position
structures rather than duplicate agent-authored copies of the same claim;
- factual observations and disputed interpretations may become separate
`public.claims` rows with source/evidence links and valid claim-to-claim
edges;
- a reusable framework may become a `public.reasoning_tools` row, but the
current schema has no generic reasoning-tool-to-claim edge and no shipped
`concept_maps` or `claim_concept_map_links` table;
- `public.governance_gates` can store an evaluative gate with `name`,
`criteria`, `evidence_bar`, and `pass_condition`; it is not a generic
behavioral-rule or policy table, so state a schema gap when a governance rule
does not fit that contract;
- a behavioral or operating rule belongs in the existing
`public.behavioral_rules` table, whose rule contract includes `agent_id`,
`category`, `rank`, `rule`, and `rationale`;
- an evaluative gate belongs in `public.governance_gates`, with `name`,
`criteria`, `evidence_bar`, and `pass_condition`; do not flatten a behavioral
rule into this gate table;
- a belief correction may create a new claim, a valid `supersedes` edge, and
set the old claim's `superseded_by` column. `superseded_by` is a column, not
an edge type.
@ -168,11 +212,13 @@ universal success condition.
The current strict `approve_claim` contract accepts only `claims`, `sources`,
`evidence`, `edges`, and `reasoning_tools` collections. It does not insert
`governance_gates`, update an existing claim's `status` or `superseded_by`, or
write arbitrary soul/context rows. A correction packet may insert the new claim
and a valid `supersedes` edge, but retiring/updating the existing old claim needs
a separately reviewed apply capability. Do not describe that unsupported update
as part of one atomic `approve_claim` transaction.
`behavioral_rules` or `governance_gates`, update an existing claim's `status`
or `superseded_by`, or write arbitrary soul/context rows. Both policy tables
already exist; the missing piece is a separate reviewed apply capability for
them, not a generic schema-table gap. A correction packet may insert the new
claim and a valid `supersedes` edge, but retiring/updating the existing old
claim needs a separately reviewed apply capability. Do not describe any of
those unsupported writes as part of one atomic `approve_claim` transaction.
Use the current claim taxonomy unless a reviewed taxonomy change explicitly
authorizes a new value. The live values are `structural`, `normative`,
@ -185,6 +231,13 @@ evidence, say that `public.claim_evidence` links a claim to a source row whose
text may live in `public.sources.excerpt`; never say the evidence row stores or
carries the excerpt.
Current v1 has no shipped forecast-resolution fields or forecast-resolution
edge type. Preserve the original probability and its timestamp. Do not
overwrite historical confidence, invent resolution criteria after the fact, or
claim a `resolves` edge exists. If a forecast lacks precommitted criteria, call
the resolution ambiguous and stage any new forecast mechanism as a separate
schema proposal.
- "Did we actually update the KB?": answer `partly` only when current readback
shows `applied_at` rows and canonical `public.*` rows. Otherwise say
`mostly still proposals`; list applied, approved-but-not-applied, pending,

307
scripts/working_leo_m3taversal_oos_benchmark.py
View file

@ -74,7 +74,7 @@ SCENARIOS: list[dict[str, Any]] = [
},
{
"id": "OOS-05",
"dimension": "source_pointer_is_not_evidence",
"dimension": "source_pointer_needs_canonical_link_audit",
"message": (
"A proposal points to a Telegram attachment and its extracted text exists on disk. Does that mean the "
"claim already has canonical evidence? Walk the link chain I should audit and tell me what a valid "
@ -87,7 +87,13 @@ SCENARIOS: list[dict[str, Any]] = [
"row_level_proof",
"no_overclaim",
],
"required_concepts": ["source_evidence_chain", "staged_review_apply", "receipt"],
"required_concepts": [
"source_evidence_chain",
"canonical_evidence_boundary",
"evidence_provenance_quality",
"staged_review_apply",
"receipt",
],
},
{
"id": "OOS-06",
@ -99,7 +105,13 @@ SCENARIOS: list[dict[str, Any]] = [
"review and apply sequence too, but do not write anything now."
),
"required_signals": ["canonical_db", "staging_or_review", "caveat_retention", "next_action", "no_overclaim"],
"required_concepts": ["heterogeneous_types", "staged_review_apply", "receipt"],
"required_concepts": [
"heterogeneous_types",
"behavioral_rule_storage",
"reviewed_policy_apply",
"staged_review_apply",
"receipt",
],
},
{
"id": "OOS-07",
@ -134,6 +146,72 @@ SCENARIOS: list[dict[str, Any]] = [
"required_signals": ["no_overclaim"],
"required_concepts": [],
},
{
"id": "OOS-10",
"dimension": "runtime_vs_canonical_causality",
"message": (
"The five database totals are unchanged after a restart. Does that prove Leo's answer behavior is "
"unchanged and that every fact from the prior session was erased? Explain which persisted and deployed "
"surfaces can affect the answer, and separate the proof tiers. Read-only; keep it under 180 words."
),
"required_signals": ["canonical_db", "no_overclaim"],
"required_concepts": ["runtime_inputs", "durable_session_continuity", "proof_tiers"],
},
{
"id": "OOS-11",
"dimension": "shared_claims_agent_positions",
"message": (
"Two agents read the same evidence and reach different conclusions. Should Leo duplicate the factual "
"claim once per agent, or keep shared knowledge and represent agent-specific positions elsewhere? Give "
"the current-schema answer and how disagreement remains queryable. Do not write anything."
),
"required_signals": ["canonical_db", "caveat_retention", "no_overclaim"],
"required_concepts": ["shared_knowledge_commons", "agent_specific_positions", "contradiction"],
},
{
"id": "OOS-12",
"dimension": "forecast_resolution_without_rewriting_history",
"message": (
"An old claim recorded a 60% forecast but never defined resolution criteria. The event is now over. "
"What may Leo record in the current database without rewriting history, and what needs a schema "
"proposal? Do not apply anything."
),
"required_signals": ["canonical_db", "staging_or_review", "caveat_retention", "no_overclaim"],
"required_concepts": ["forecast_history", "forecast_schema_gap", "staged_review_apply"],
},
{
"id": "OOS-13",
"dimension": "handler_is_not_telegram_delivery",
"message": (
"A temporary-profile GatewayRunner answered every test prompt, but the run explicitly posted nothing "
"to Telegram. Can I tell a partner the Telegram path is proven live? Answer first, then state exactly "
"what this run proves and the smallest test that closes the gap."
),
"required_signals": ["artifact", "next_action", "no_overclaim"],
"required_concepts": ["handler_not_telegram", "proof_tiers", "receipt"],
},
{
"id": "OOS-14",
"dimension": "autonomous_source_intake_boundary",
"message": (
"I hand Leo a document and say: absorb this as far as safely possible without making me approve every "
"mechanical step. What can Leo capture and stage immediately, what real source identity must be retained, "
"and where does explicit approval begin? Explain only; do not ingest this prompt."
),
"required_signals": ["authorization", "staging_or_review", "artifact", "no_overclaim"],
"required_concepts": ["staging_without_apply_authorization", "real_source_identity", "staged_review_apply"],
},
{
"id": "OOS-15",
"dimension": "schema_valid_supersession",
"message": (
"A canonical claim is wrong. I want the replacement, an explanation, and the old claim visibly retired. "
"In current v1, which exact claim and edge fields exist, which requested writes fit approve_claim, and "
"which require a separate reviewed apply capability? Do not mutate anything."
),
"required_signals": ["canonical_db", "staging_or_review", "row_level_proof", "no_overclaim"],
"required_concepts": ["valid_supersession", "current_edge_schema", "apply_capability_boundary"],
},
]
@ -146,7 +224,9 @@ CONCEPT_PATTERNS: dict[str, tuple[re.Pattern[str], ...]] = {
re.I,
),
),
"live_check": (re.compile(r"checked live|I ran `?teleo-kb|fresh readback|current canonical row counts", re.I),),
"live_check": (
re.compile(r"checked live|I ran `?teleo-kb|fresh readback|live readback|current canonical row counts", re.I),
),
"material_action": (
re.compile(
r"next .*action|rebuild .*apply_payload|operator .*authoriz|review .*apply|apply sequence|postflight",
@ -175,7 +255,22 @@ CONCEPT_PATTERNS: dict[str, tuple[re.Pattern[str], ...]] = {
re.compile(r"file|attachment|source_ref", re.I),
re.compile(r"public\.sources", re.I),
re.compile(r"claim_evidence", re.I),
re.compile(r"not canonical|staging-layer pointer|proves nothing about the KB", re.I),
re.compile(r"audit|link|join|row chain|before-and-after|before/after", re.I),
),
"canonical_evidence_boundary": (
re.compile(r"canonical evidence", re.I),
re.compile(r"claim_evidence", re.I),
re.compile(r"public\.sources|source row", re.I),
re.compile(
r"attachment.{0,160}(?:does not|doesn't|is not|isn't|cannot|can't|alone|until|unless)|"
r"(?:does not|doesn't|is not|isn't|cannot|can't).{0,100}canonical evidence from (?:that|the) attachment",
re.I | re.S,
),
),
"evidence_provenance_quality": (
re.compile(r"(?:missing|no|without).{0,50}(?:url|storage(?:_path)?|locator)", re.I | re.S),
re.compile(r"weak|citation-only|citation only|not traceable|raw artifact", re.I),
re.compile(r"canonical evidence|canonical link", re.I),
),
"heterogeneous_types": (
re.compile(r"claim", re.I),
@ -185,6 +280,97 @@ CONCEPT_PATTERNS: dict[str, tuple[re.Pattern[str], ...]] = {
re.compile(r"correction|supersed", re.I),
re.compile(r"disput|contradict", re.I),
),
"behavioral_rule_storage": (
re.compile(r"public\.behavioral_rules", re.I),
re.compile(r"\bagent_id\b", re.I),
re.compile(r"\bcategory\b", re.I),
re.compile(r"\brank\b", re.I),
re.compile(r"\brule\b", re.I),
re.compile(r"\brationale\b", re.I),
),
"reviewed_policy_apply": (
re.compile(r"approve_claim", re.I),
re.compile(r"behavioral_rules", re.I),
re.compile(r"governance_gates", re.I),
re.compile(r"does not (?:accept|insert|support)|supports neither|neither.{0,80}nor", re.I | re.S),
re.compile(r"separate.{0,50}reviewed apply|reviewed.{0,50}apply capability", re.I | re.S),
),
"runtime_inputs": (
re.compile(r"Postgres|canonical (?:DB|database|counts?)", re.I),
re.compile(r"skills?|runtime config|configuration|SOUL\.md", re.I),
re.compile(r"session|conversation context", re.I),
re.compile(r"unchanged.{0,80}(?:does not|doesn't|do not).{0,80}(?:behavior|answer)", re.I | re.S),
),
"durable_session_continuity": (
re.compile(r"state\.db|session JSONL", re.I),
re.compile(r"persist|durable|continuity", re.I),
re.compile(r"restart.{0,80}(?:does not|doesn't|need not|not necessarily).{0,80}(?:erase|forget)", re.I | re.S),
),
"proof_tiers": (
re.compile(r"handler|temporary[- ]profile|GatewayRunner", re.I),
re.compile(r"Telegram", re.I),
re.compile(r"canonical|public\.\*|DB mutation|database mutation", re.I),
),
"shared_knowledge_commons": (
re.compile(r"shared (?:claim|knowledge|commons)|claims.{0,40}shared", re.I | re.S),
re.compile(r"source|evidence", re.I),
re.compile(r"do not duplicate|don'?t duplicate|one shared claim|single shared claim", re.I),
),
"agent_specific_positions": (
re.compile(r"agent[- ]specific", re.I),
re.compile(r"belief|position|stance|confidence", re.I),
re.compile(r"separate|elsewhere|per-agent", re.I),
),
"forecast_history": (
re.compile(r"original (?:probability|confidence)|60%|history", re.I),
re.compile(r"preserve|retain|do not overwrite|don'?t overwrite", re.I),
re.compile(r"ambiguous|missing.{0,30}criteria|no.{0,30}criteria", re.I | re.S),
),
"forecast_schema_gap": (
re.compile(r"current (?:v1|schema)|public\.claims", re.I),
re.compile(
r"no.{0,50}(?:forecast[- ]resolution|resolution field|resolved_at)|does not have.{0,50}resolution",
re.I | re.S,
),
re.compile(r"resolves.{0,30}(?:not|isn'?t|does not)|no.{0,30}resolves", re.I | re.S),
),
"handler_not_telegram": (
re.compile(r"no|not", re.I),
re.compile(r"handler|GatewayRunner|temporary[- ]profile", re.I),
re.compile(r"did not post|posted nothing|not Telegram-visible|does not prove Telegram", re.I),
),
"staging_without_apply_authorization": (
re.compile(r"capture|hash|archive", re.I),
re.compile(r"stage|pending_review|proposal", re.I),
re.compile(
r"does not require.{0,50}(?:apply )?approval|without.{0,50}(?:apply )?approval|approval begins.{0,80}apply",
re.I | re.S,
),
),
"real_source_identity": (
re.compile(r"URL|storage path|file hash|content hash|retained artifact", re.I),
re.compile(r"temporary label|chat label|proposal pointer|source_ref", re.I),
re.compile(r"not.{0,40}(?:a )?(?:source|provenance)|must not.{0,40}source|do not manufacture", re.I | re.S),
),
"valid_supersession": (
re.compile(r"new|replacement", re.I),
re.compile(r"supersedes", re.I),
re.compile(r"superseded_by", re.I),
re.compile(r"old claim", re.I),
),
"current_edge_schema": (
re.compile(r"claim_edges", re.I),
re.compile(r"from_claim", re.I),
re.compile(r"to_claim", re.I),
re.compile(r"edge_type", re.I),
re.compile(r"no.{0,30}rationale|does not.{0,30}rationale", re.I | re.S),
),
"apply_capability_boundary": (
re.compile(r"approve_claim", re.I),
re.compile(r"insert.{0,50}(?:new|replacement) claim|new claim.{0,50}insert", re.I | re.S),
re.compile(r"supersedes edge", re.I),
re.compile(r"separate.{0,50}reviewed apply|does not update.{0,80}(?:status|superseded_by)", re.I | re.S),
),
"blocker_definition": (
re.compile(r"blocker", re.I),
re.compile(r"approved|applied_at", re.I),
@ -206,7 +392,8 @@ COUNT_INVARIANT_REJECTION_RE = re.compile(
SCHEMA_GAP_QUALIFIER_RE = re.compile(
r"\b(?:proposed|future|not current|not shipped|does not exist|doesn't exist|absent|"
r"has no|have no|no column|not an edge|would require|schema gap|must be added)\b",
r"has no|have no|no column|not an edge|would require|schema gap|must be added|"
r"does not support|doesn't support|supports neither|not supported)\b",
re.I,
)
CURRENT_SCHEMA_ASSERTION_PATTERNS: dict[str, re.Pattern[str]] = {
@ -247,6 +434,53 @@ CURRENT_SCHEMA_ASSERTION_PATTERNS: dict[str, re.Pattern[str]] = {
}
UNVERIFIED_M3TAVERSAL_ALIAS_RE = re.compile(r"\b(?:Cory|m3ta)\b", re.I)
SOURCE_EVIDENCE_CANONICAL_OBJECT_RE = re.compile(r"claim_evidence|public\.sources|source rows?", re.I)
SOURCE_EVIDENCE_LOCATOR_GAP_RE = re.compile(
r"(?:missing|without|no).{0,50}(?:url|storage(?:_path)?|locator)", re.I | re.S
)
SOURCE_EVIDENCE_DENIAL_RE = re.compile(
r"(?:not|isn't|is not|doesn't count as|does not count as).{0,40}(?:canonical|grounded) evidence",
re.I | re.S,
)
SOURCE_EVIDENCE_CITATION_STUB_DENIAL_RE = re.compile(
r"citation stubs?.{0,50}(?:not|isn't|is not).{0,30}(?:canonical|grounded) evidence",
re.I | re.S,
)
BEHAVIORAL_RULES_FALSE_ABSENCE_RE = re.compile(
r"(?:public\.)?behavioral_rules(?: table)?.{0,50}(?:does not exist|doesn't exist|is absent|is missing|not shipped)|"
r"(?:no|missing) (?:public\.)?behavioral_rules table",
re.I | re.S,
)
DB_ONLY_CAUSALITY_RE = re.compile(
r"(?:unchanged|same) (?:canonical )?(?:DB|database|counts?).{0,80}"
r"(?:proves?|means|therefore).{0,80}(?:behavior|answer).{0,30}(?:unchanged|cannot change)",
re.I | re.S,
)
RESTART_ERASES_ALL_RE = re.compile(
r"restart.{0,100}(?:erases?|forgets?|loses?).{0,40}(?:all|every|prior[- ]session)", re.I | re.S
)
HANDLER_TELEGRAM_OVERCLAIM_RE = re.compile(
r"(?:Telegram (?:path|delivery).{0,40}(?:is|was) (?:live|proven)|"
r"live Telegram (?:path|delivery)).{0,160}(?:posted nothing|did not post|no Telegram post)",
re.I | re.S,
)
FORECAST_HISTORY_REWRITE_RE = re.compile(
r"(?<!not )(?<!don't )(?<!dont )(?:overwrite|update|replace).{0,40}"
r"(?:original )?(?:probability|confidence).{0,80}"
r"(?:resolved|actual|outcome)",
re.I | re.S,
)
TEMP_LABEL_AS_SOURCE_RE = re.compile(
r"(?:use|write|create|store|promote).{0,50}(?:temporary|memory|chat) (?:label|token).{0,100}"
r"(?:source_ref|public\.sources|canonical source)",
re.I | re.S,
)
DEFAULT_MAX_RESPONSE_WORDS = 300
MAX_RESPONSE_WORDS = {"OOS-09": 100, "OOS-10": 180}
def prompt_catalog(memory_token: str) -> list[dict[str, Any]]:
return [
@ -279,6 +513,51 @@ def current_schema_overclaims(reply: str) -> list[str]:
return sorted(set(findings))
def source_evidence_semantic_issues(reply: str) -> list[str]:
"""Reject answers that confuse canonical linkage with locator quality."""
findings: set[str] = set()
for segment in re.split(r"(?<=[.!?])\s+|\n+", reply):
if (
SOURCE_EVIDENCE_CANONICAL_OBJECT_RE.search(segment)
and SOURCE_EVIDENCE_LOCATOR_GAP_RE.search(segment)
and SOURCE_EVIDENCE_DENIAL_RE.search(segment)
):
findings.add("locator_gap_called_noncanonical")
if SOURCE_EVIDENCE_CITATION_STUB_DENIAL_RE.search(reply):
findings.add("citation_stub_called_ungrounded")
return sorted(findings)
def behavioral_rule_schema_issues(reply: str) -> list[str]:
"""Reject claims that the current behavioral-rules table is absent."""
return ["behavioral_rules_false_absence"] if BEHAVIORAL_RULES_FALSE_ABSENCE_RE.search(reply) else []
def broad_semantic_issues(reply: str) -> list[str]:
"""Reject high-impact false claims that can hide inside otherwise complete prose."""
findings: set[str] = set()
if DB_ONLY_CAUSALITY_RE.search(reply) and not re.search(
r"does not prove|doesn't prove|do not prove|does not mean|doesn't mean|do not mean|cannot prove", reply, re.I
):
findings.add("unchanged_db_called_complete_behavior_proof")
if RESTART_ERASES_ALL_RE.search(reply) and not re.search(
r"restart.{0,100}(?:does not|doesn't|need not|not necessarily).{0,60}(?:erase|forget|lose)",
reply,
re.I | re.S,
):
findings.add("restart_called_total_memory_erasure")
if HANDLER_TELEGRAM_OVERCLAIM_RE.search(reply):
findings.add("handler_proof_called_telegram_live")
if FORECAST_HISTORY_REWRITE_RE.search(reply):
findings.add("forecast_history_rewrite")
if TEMP_LABEL_AS_SOURCE_RE.search(reply):
findings.add("temporary_label_promoted_to_source")
return sorted(findings)
def score_reply(prompt: dict[str, Any], reply: str, *, memory_token: str) -> dict[str, Any]:
legacy_score = base.score_reply(prompt, reply)
concepts = {concept: matched_concept(reply, concept) for concept in prompt["required_concepts"]}
@ -307,6 +586,12 @@ def score_reply(prompt: dict[str, Any], reply: str, *, memory_token: str) -> dic
)
invalid_count_invariant = asserts_invalid_count_invariant(reply)
schema_overclaims = current_schema_overclaims(reply)
source_evidence_issues = source_evidence_semantic_issues(reply) if prompt["id"] == "OOS-05" else []
behavioral_rule_issues = behavioral_rule_schema_issues(reply) if prompt["id"] == "OOS-06" else []
semantic_issues = broad_semantic_issues(reply)
word_count = len(re.findall(r"\b\w+(?:[-']\w+)*\b", reply))
max_response_words = MAX_RESPONSE_WORDS.get(prompt["id"], DEFAULT_MAX_RESPONSE_WORDS)
response_too_long = word_count > max_response_words
return {
"prompt_id": prompt["id"],
"dimension": prompt["dimension"],
@ -318,12 +603,22 @@ def score_reply(prompt: dict[str, Any], reply: str, *, memory_token: str) -> dic
"response_issue_detected": legacy_score["response_issue_detected"],
"invalid_count_invariant_detected": invalid_count_invariant,
"current_schema_overclaims": schema_overclaims,
"source_evidence_semantic_issues": source_evidence_issues,
"behavioral_rule_schema_issues": behavioral_rule_issues,
"broad_semantic_issues": semantic_issues,
"word_count": word_count,
"max_response_words": max_response_words,
"response_too_long": response_too_long,
"pass": bool(
all(concepts.values())
and all(custom_signals.values())
and not legacy_score["overclaim_detected"]
and not invalid_count_invariant
and not schema_overclaims
and not source_evidence_issues
and not behavioral_rule_issues
and not semantic_issues
and not response_too_long
),
}

44
tests/test_hermes_leoclean_skill_surfaces.py
View file

@ -120,7 +120,7 @@ def test_vps_kb_skill_keeps_vps_scope_explicit() -> None:
assert "new or updated row IDs" in text
assert "I cannot claim canonical DB changed" in text
assert "explicit operator/admin authorization" in text
assert "Do not call an approved proposal \"implemented\"" in text
assert 'Do not call an approved proposal "implemented"' in text
assert "repository contains a live-proven strict existing-ID `add_edge` path" in squashed
assert "Never end a normal Telegram answer by offering to run direct `INSERT`, `UPDATE`" in text
assert "Next admin-panel action" in text
@ -152,6 +152,48 @@ def test_leoclean_kb_skills_anchor_external_doctrine_in_target_project_language(
assert "Consent is action-specific" in text
def test_leoclean_kb_skills_distinguish_canonical_evidence_from_provenance_quality() -> None:
for surface in ("vps", "gcp"):
text = (SKILL_ROOT / surface / "teleo-kb-bridge" / "SKILL.md").read_text()
squashed = " ".join(text.split())
assert "is canonical evidence even when that source row has no `url` or `storage_path`" in squashed
assert "weak or citation-only provenance" in squashed
assert "not traceable to the raw artifact" in squashed
assert "does not by itself prove canonical evidence from that attachment" in squashed
assert "`public.sources` row representing the attachment" in squashed
assert "`public.claim_evidence` link from the claim to that source row" in squashed
def test_leoclean_kb_skills_use_existing_behavioral_rule_storage_and_separate_apply() -> None:
for surface in ("vps", "gcp"):
text = (SKILL_ROOT / surface / "teleo-kb-bridge" / "SKILL.md").read_text()
squashed = " ".join(text.split())
assert "existing `public.behavioral_rules` table" in squashed
for field in ("`agent_id`", "`category`", "`rank`", "`rule`", "`rationale`"):
assert field in text
assert "does not insert `behavioral_rules` or `governance_gates`" in squashed
assert "separate reviewed apply capability" in squashed
assert "not a generic schema-table gap" in squashed
def test_leoclean_kb_skills_separate_runtime_session_and_delivery_proof_tiers() -> None:
for surface in ("vps", "gcp"):
text = (SKILL_ROOT / surface / "teleo-kb-bridge" / "SKILL.md").read_text()
squashed = " ".join(text.split())
assert "keep the whole reply under 220 words" in squashed
assert "Postgres is canonical knowledge, but it is not the only input" in squashed
assert "Hermes `state.db` and session JSONL provide durable continuity" in squashed
assert "not Telegram-visible delivery proof" in squashed
assert "Staging is not canonical apply" in squashed
assert "Never manufacture a source row from a temporary memory label" in squashed
assert "claims, sources, and evidence links are shared knowledge objects" in squashed
assert "no shipped forecast-resolution fields" in squashed
assert "Do not overwrite historical confidence" in squashed
def test_vps_live_telegram_skill_uses_systemd_for_gateway_liveness() -> None:
text = (SKILL_ROOT / "vps" / "live-leo-telegram" / "SKILL.md").read_text()

197
tests/test_working_leo_m3taversal_oos_benchmark.py
View file

@ -35,14 +35,20 @@ def good_reply(prompt_id: str, token: str) -> str:
)
if prompt_id == "OOS-05":
return common + (
"A Telegram attachment and proposal source_ref are staging pointers, not canonical evidence. Audit the "
"document_evaluation, source_ref, public.sources row, claim_evidence join, and guarded apply receipt."
"A Telegram attachment or proposal source_ref alone does not prove canonical evidence from that "
"attachment. Canonical evidence exists when public.claim_evidence links the claim to a public.sources "
"row. A missing URL or storage_path leaves weak, citation-only provenance that is not traceable to the "
"raw artifact; it does not undo the canonical link. Audit the document_evaluation, source_ref, source "
"row, claim_evidence join, and guarded apply receipt."
)
if prompt_id == "OOS-06":
return common + (
"Keep observations as claims with sources and evidence, link edges, place a framework in a reasoning tool "
"or concept map, retain the disputed interpretation as a caveat, stage governance separately, and record "
"the correction as a superseding relationship before review and apply."
"Keep factual observations as empirical claims with sources and evidence, link contradictory claims, "
"place the framework in a reasoning tool, and retain the disputed interpretation as a caveat. Store the "
"behavioral rule in public.behavioral_rules using agent_id, category, rank, rule, and rationale; reserve "
"public.governance_gates for the evaluative gate. Current approve_claim supports neither "
"behavioral_rules nor governance_gates, so each needs a separate reviewed apply capability. Record the "
"correction as a superseding relationship before review and apply."
)
if prompt_id == "OOS-07":
return common + (
@ -61,21 +67,67 @@ def good_reply(prompt_id: str, token: str) -> str:
"Never transfer or mix one participant's identity into another user's reply.\n"
"I cannot claim another identity source without explicit proof."
)
if prompt_id == "OOS-10":
return (
"Postgres is the canonical database, but an unchanged count receipt does not prove answer behavior is "
"unchanged. "
"Deployed skills, runtime configuration, rendered SOUL.md, session state, and conversation context also "
"affect replies. Hermes state.db and session JSONL provide durable continuity, so a restart does not "
"necessarily erase every prior-session fact. Proof tiers are separate: a temporary-profile handler run, "
"Telegram-visible delivery, and canonical public.* database mutation with a row receipt."
)
if prompt_id == "OOS-11":
return common + (
"Use one shared claim with shared sources and evidence; do not duplicate the factual claim per agent. "
"Store agent-specific beliefs, positions, stances, or confidence separately. Preserve each rationale and "
"retain disagreement as a caveat or contradiction edge so it remains queryable."
)
if prompt_id == "OOS-12":
return common + (
"Preserve the original 60% probability and history; do not overwrite it. The missing criteria make the "
"outcome ambiguous, so retain that caveat. Current v1 public.claims has no forecast-resolution field and "
"there is no resolves edge. Stage a separate schema proposal, then review and apply it before using a new "
"resolution mechanism."
)
if prompt_id == "OOS-13":
return (
"No. The temporary-profile GatewayRunner handler run posted nothing to Telegram, so it is not "
"Telegram-visible delivery proof. It proves handler execution and reply behavior only. Canonical database "
"mutation is a third tier requiring public.* row and applied proposal receipts. The next action is one "
"authorized Telegram-visible prompt with message and reply readback while confirming no DB count change."
)
if prompt_id == "OOS-14":
return common + (
"Leo may capture the retained artifact, URL or storage path, and file/content hash; extract candidates, "
"deduplicate them, record contradictions, and stage a pending_review proposal without canonical-apply "
"approval. A temporary chat label or source_ref is not real source provenance. Explicit operator "
"authorization begins at the guarded canonical apply path, after review."
)
if prompt_id == "OOS-15":
return common + (
"Insert the replacement claim and a supersedes edge from new claim to old claim. Current public.claim_edges "
"has from_claim, to_claim, edge_type, weight, created_by, and created_at; it has no rationale field. The "
"old claim has a superseded_by column. approve_claim can insert the new claim and supersedes edge, but it "
"does not update the old claim status or superseded_by; those require a separate reviewed apply capability."
)
return common + "Fresh readback is required before the demo claim changes."
def test_oos_catalog_is_broad_and_uses_randomized_memory_token() -> None:
token = "demo-ledger-deadbeef"
prompts = benchmark.prompt_catalog(token)
assert [prompt["id"] for prompt in prompts] == [f"OOS-{index:02d}" for index in range(1, 10)]
assert token in prompts[-3]["message"]
assert token not in prompts[-2]["message"]
assert [prompt["id"] for prompt in prompts] == [f"OOS-{index:02d}" for index in range(1, 16)]
by_id = {prompt["id"]: prompt for prompt in prompts}
assert token in by_id["OOS-07"]["message"]
assert token not in by_id["OOS-08"]["message"]
joined = "\n".join(prompt["message"] for prompt in prompts)
assert "PDF" in joined
assert "tweets" in joined
assert "SOUL.md" in joined
assert "Do not ask me for row IDs" in joined
assert "@m3taversal" in joined
assert "temporary-profile GatewayRunner" in joined
assert "old claim" in joined
def test_oos_score_passes_complete_behavior_and_memory_pair() -> None:
@ -86,7 +138,11 @@ def test_oos_score_passes_complete_behavior_and_memory_pair() -> None:
]
score = benchmark.score_results(results, memory_token=token)
assert score["pass"] is True
assert score["passes"] == 9
assert score["passes"] == 15
def test_oos_live_check_accepts_live_readback_wording() -> None:
assert benchmark.matched_concept("Here is the live readback from Postgres.", "live_check") is True
def test_oos_score_fails_when_memory_token_is_not_recalled() -> None:
@ -95,7 +151,8 @@ def test_oos_score_fails_when_memory_token_is_not_recalled() -> None:
{"prompt_id": prompt["id"], "reply": good_reply(prompt["id"], token)}
for prompt in benchmark.prompt_catalog(token)
]
results[-2]["reply"] = results[-2]["reply"].replace(token, "some-other-label")
recall = next(result for result in results if result["prompt_id"] == "OOS-08")
recall["reply"] = recall["reply"].replace(token, "some-other-label")
score = benchmark.score_results(results, memory_token=token)
assert score["pass"] is False
assert score["failures"][0]["prompt_id"] == "OOS-08"
@ -104,7 +161,7 @@ def test_oos_score_fails_when_memory_token_is_not_recalled() -> None:
def test_oos_identity_case_requires_exact_visible_handle_and_no_alias() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[-1]
prompt = benchmark.prompt_catalog(token)[8]
good = benchmark.score_reply(prompt, good_reply(prompt["id"], token), memory_token=token)
assert good["pass"] is True
@ -119,7 +176,7 @@ def test_oos_identity_case_requires_exact_visible_handle_and_no_alias() -> None:
def test_oos_identity_case_accepts_inflected_carry_language() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[-1]
prompt = benchmark.prompt_catalog(token)[8]
reply = (
"Call the current visible participant m3taversal, exactly.\n"
"Use only the current Telegram update and visible sender.\n"
@ -175,6 +232,122 @@ def test_oos_schema_guard_rejects_unreviewed_claim_taxonomy_and_apply_surface()
assert benchmark.current_schema_overclaims(unsupported_apply) == ["unsupported_approve_claim_surface"]
def test_oos_schema_guard_allows_correct_approve_claim_capability_boundary() -> None:
reply = (
"Current approve_claim supports neither behavioral_rules nor governance_gates; both need a separate "
"reviewed apply capability."
)
assert benchmark.current_schema_overclaims(reply) == []
def test_oos_source_evidence_case_accepts_canonical_link_and_weak_provenance_distinction() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[4]
score = benchmark.score_reply(prompt, good_reply(prompt["id"], token), memory_token=token)
assert score["pass"] is True
assert score["source_evidence_semantic_issues"] == []
def test_oos_source_evidence_case_rejects_locator_gap_called_noncanonical() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[4]
reply = good_reply(prompt["id"], token) + (
" A public.claim_evidence link to a source row with no URL or storage locator is not canonical evidence."
)
score = benchmark.score_reply(prompt, reply, memory_token=token)
assert score["pass"] is False
assert score["source_evidence_semantic_issues"] == ["locator_gap_called_noncanonical"]
def test_oos_source_evidence_conflation_guard_is_order_independent() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[4]
reply = good_reply(prompt["id"], token) + (
" A source row with no locator is not canonical evidence even when public.claim_evidence links it."
)
score = benchmark.score_reply(prompt, reply, memory_token=token)
assert score["pass"] is False
assert score["source_evidence_semantic_issues"] == ["locator_gap_called_noncanonical"]
def test_oos_source_evidence_case_rejects_citation_stub_conflation() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[4]
reply = good_reply(prompt["id"], token) + " These source rows are citation stubs, not grounded evidence."
score = benchmark.score_reply(prompt, reply, memory_token=token)
assert score["pass"] is False
assert score["source_evidence_semantic_issues"] == ["citation_stub_called_ungrounded"]
def test_oos_database_composition_requires_existing_behavioral_rule_table() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[5]
good = benchmark.score_reply(prompt, good_reply(prompt["id"], token), memory_token=token)
assert good["pass"] is True
bad_reply = good_reply(prompt["id"], token) + " The public.behavioral_rules table is absent."
bad = benchmark.score_reply(prompt, bad_reply, memory_token=token)
assert bad["pass"] is False
assert bad["behavioral_rule_schema_issues"] == ["behavioral_rules_false_absence"]
def test_oos_runtime_case_rejects_db_only_causality_and_total_memory_erasure() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[9]
good = benchmark.score_reply(prompt, good_reply(prompt["id"], token), memory_token=token)
assert good["pass"] is True
bad_reply = (
"Postgres is canonical. The unchanged database proves answer behavior is unchanged. "
"A restart erases every prior-session fact."
)
bad = benchmark.score_reply(prompt, bad_reply, memory_token=token)
assert bad["pass"] is False
assert bad["broad_semantic_issues"] == [
"restart_called_total_memory_erasure",
"unchanged_db_called_complete_behavior_proof",
]
def test_oos_handler_case_rejects_telegram_live_overclaim() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[12]
reply = good_reply(prompt["id"], token) + (
" The Telegram path is live and proven even though the run posted nothing to Telegram."
)
score = benchmark.score_reply(prompt, reply, memory_token=token)
assert score["pass"] is False
assert score["broad_semantic_issues"] == ["handler_proof_called_telegram_live"]
def test_oos_forecast_case_rejects_historical_probability_overwrite() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[11]
reply = good_reply(prompt["id"], token) + " Overwrite the original probability with the actual resolved outcome."
score = benchmark.score_reply(prompt, reply, memory_token=token)
assert score["pass"] is False
assert score["broad_semantic_issues"] == ["forecast_history_rewrite"]
def test_oos_source_intake_case_rejects_temporary_label_as_source() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[13]
reply = good_reply(prompt["id"], token) + " Create the temporary chat label as a canonical source_ref."
score = benchmark.score_reply(prompt, reply, memory_token=token)
assert score["pass"] is False
assert score["broad_semantic_issues"] == ["temporary_label_promoted_to_source"]
def test_oos_score_rejects_architecture_lecture_length() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[0]
reply = good_reply(prompt["id"], token) + " detail" * 301
score = benchmark.score_reply(prompt, reply, memory_token=token)
assert score["pass"] is False
assert score["response_too_long"] is True
assert score["max_response_words"] == benchmark.DEFAULT_MAX_RESPONSE_WORDS
def test_oos_score_rejects_blanket_all_five_counts_must_move_claim() -> None:
token = "demo-ledger-deadbeef"
prompt = benchmark.prompt_catalog(token)[3]