{
  "benchmark_layers": [
    {
      "id": "precise_regression",
      "purpose": "Pin known IDs, proposals, and row facts so regressions are easy to diagnose.",
      "runtime": "live_readonly_or_fixture"
    },
    {
      "id": "open_ended_readonly",
      "purpose": "Ask vague Cory-style questions while explicitly preventing KB mutation on live Telegram/VPS.",
      "runtime": "live_readonly"
    },
    {
      "id": "cory_style_outcome",
      "purpose": "Exercise regular-use ambiguity: broad operator intent, demo pressure, missing IDs, mixed KB/runtime/governance asks.",
      "runtime": "disposable_clone_or_sandbox_first"
    },
    {
      "id": "direct_claim_followup",
      "purpose": "Ask no-context Cory-style direct claim questions and require the expected answer plus the next follow-up.",
      "runtime": "disposable_clone_or_sandbox_first"
    }
  ],
  "generated_at_utc": "2026-07-13T00:27:54.787284+00:00",
  "include_cory_style_scenarios": false,
  "include_direct_claim_followups": true,
  "mode": "retained_evidence_score",
  "mutates_kb": false,
  "require_all": false,
  "results": [
    {
      "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
      "dimension": "db_changed_direct_claim",
      "ended_at_utc": "2026-07-13T00:16:57.122173+00:00",
      "evidence_tier": "live_vps_gatewayrunner_temp_profile",
      "mutates_kb": false,
      "ok": true,
      "prompt": "Did we actually update the knowledge base, or is it still just proposals?",
      "prompt_id": "DC-01",
      "reply": "Mostly still proposals. Here's the exact state:\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nApplied (canonical change confirmed): 2\n- `00957f6c` \u2014 `add_edge` canary, applied 2026-07-09\n- `f004bbb2` \u2014 `attach_evidence` Leo guiding-policy revision, applied 2026-07-05\n\nApproved but NOT applied (applied_at: none): 3\n- `14fa5ecc` \u2014 attach_evidence, split compressed internet-finance claim\n- `ac036c9d` \u2014 attach_evidence, strategy-kernel hygiene / Hayek anchor\n- `a64df080` \u2014 attach_evidence, Helmer 7 Powers framework packet (approved 2026-06-29, still sitting)\n\nPending review: 14\nCanceled: 7\n\nApproved is not the same as applied. Those 3 approved proposals \u2014 including the 7 Powers packet \u2014 have reviewer sign-off but no applied_at timestamp, which means no canonical `public.*` rows were written. The 7 Powers framework specifically has been approved since June 29 and is still not in the canonical DB.\n\nNext Cory-style follow-up: The proof-changing action is applying the 3 approved proposals to canonical `public.*` \u2014 that requires explicit operator/admin authorization and a dedicated apply path with before/after postflight readback. Want me to draft or refresh the admin review packet for the oldest of those (`a64df080` \u2014 7 Powers)?",
      "started_at_utc": "2026-07-13T00:16:23.688987+00:00",
      "turn": 1
    },
    {
      "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
      "dimension": "helmer_live_direct_claim",
      "ended_at_utc": "2026-07-13T00:17:21.172268+00:00",
      "evidence_tier": "live_vps_gatewayrunner_temp_profile",
      "mutates_kb": false,
      "ok": true,
      "prompt": "Is Helmer's 7 Powers in Leo now?",
      "prompt_id": "DC-02",
      "reply": "No. Not canonical.\n\nDB readback: proposal: `a64df080-8502-42e2-98f4-9bbdecb8da73`; status: `approved`; applied_at: `none` (database NULL).\n\nThe 7 Powers framework is approved/staged \u2014 the full packet exists in `kb_stage.kb_proposals` with m3ta's sign-off from June 29 \u2014 but it has never been applied to canonical `public.*`. A search of the canonical KB returns no Helmer/7 Powers claims, sources, edges, or reasoning_tool rows.\n\nApproved is not the same as applied.\n\nWhat exists in staging (not canonical):\n- A `reasoning_tool` node definition for Helmer 7 Powers\n- 5 proposed claims (benefit/barrier logic, power progression, counter-positioning, network effects vs network economies, agentic capital needs power analysis)\n- 6 source candidates (Helmer book, official synopsis, Reed Hastings foreword, two Acquired transcripts, NFX interview)\n- Edge links into existing competitive-advantage/moats neighborhood claims\n- Conceptual neighborhood connections to the competitive advantage/moats domain\n\nWhat's missing before it's canonical:\n- `public.sources` rows for the source candidates\n- `public.claims` rows for the 5 proposed claims\n- `public.claim_evidence` and `public.claim_edges` rows linking them\n- A `reasoning_tool` node row (schema gap noted in the proposal \u2014 current bridge doesn't expose `add_claim`/`add_context_node` proposal types)\n\nNext Cory-style follow-up: The proof-changing action is running a guarded apply of `a64df080` \u2014 writing those source, claim, evidence, and edge rows to `public.*`. That requires explicit operator authorization and a dedicated apply path with before/after postflight readback; not runnable from normal chat.",
      "started_at_utc": "2026-07-13T00:16:57.122476+00:00",
      "turn": 2
    },
    {
      "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
      "dimension": "decision_matrix_direct_claim",
      "ended_at_utc": "2026-07-13T00:17:33.760712+00:00",
      "evidence_tier": "live_vps_gatewayrunner_temp_profile",
      "mutates_kb": false,
      "ok": true,
      "prompt": "Did the decision matrix approve this already?",
      "prompt_id": "DC-03",
      "reply": "No. Fresh readback: the decision-matrix schema is absent; reviewer status is not a decision-matrix vote.\n\nNone of the required tables exist \u2014 `matrix_voters`, `proposal_votes`, and `proposal_decisions` are missing from both `kb_stage` and `public`. The decision-matrix path has not been shipped.\n\nWhat actually happened with `a64df080`: m3ta reviewed it directly in Telegram on June 29, said approved-with-changes, and Leo recorded that as `status: approved` in `kb_stage.kb_proposals`. That is a human reviewer approval \u2014 not a futarchy vote, not a decision-matrix quorum, not a multi-agent vote count.\n\nDB readback: proposal: `a64df080-8502-42e2-98f4-9bbdecb8da73`; status: `approved`; applied_at: `none` (database NULL).\n\nSo the full picture on 7 Powers:\n- Decision matrix: not shipped, no vote possible\n- Reviewer approval: m3ta approved-with-changes June 29\n- Canonical rows: zero\n- Applied: no\n\nNext Cory-style follow-up: The decision-matrix path needs `matrix_voters`, `proposal_votes`, and `proposal_decisions` tables before any futarchy-style governance is possible. If you want to move 7 Powers to canonical without waiting for that, the path is explicit operator authorization to run the guarded apply on `a64df080` \u2014 the reviewer approval is already on record.",
      "started_at_utc": "2026-07-13T00:17:21.172764+00:00",
      "turn": 3
    },
    {
      "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
      "dimension": "document_source_direct_claim",
      "ended_at_utc": "2026-07-13T00:18:29.511463+00:00",
      "evidence_tier": "live_vps_gatewayrunner_temp_profile",
      "mutates_kb": false,
      "ok": true,
      "prompt": "Are the pending proposals stuck because the documents are not pointed at the right source rows?",
      "prompt_id": "DC-04",
      "reply": "Not just pointer mismatch. The `source_ref` values are a useful trace but they're not the root cause of the pending pile. Here's what the row-link audit actually shows:\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nThe 14 pending proposals break into distinct layers:\n\n3 are canary `add_edge` probes (8dfedb3f, 03f7853e, e60a48af) \u2014 never intended for content review, just pipeline tests. They're pending because no one cleared them, not because of pointer mismatch.\n\n1 is a live edge proposal from June 19 (c824da66) \u2014 `source_ref` is a Telegram session marker, no doc involved at all.\n\nThe remaining 10 are `attach_evidence` proposals \u2014 and here is where it gets layered:\n\n- `source_ref` values point at raw files (telegram-attachment paths), Telegram session markers, or URLs \u2014 not at `public.sources` row IDs. That's a real gap but it's a gap in the apply contract, not the reason they're stuck.\n- The actual reason they're pending is that no reviewer has actioned them. `reviewed_at` is null on all 14. They're not blocked by a pointer \u2014 they haven't been reviewed.\n- The schema gap noted in the 7 Powers packet makes it worse: `attach_evidence` proposals carry full source/claim/edge payloads because the bridge doesn't expose `add_claim`/`add_source` proposal types. So even if a reviewer approves, there's no worker or apply command that knows how to unpack a complex packet into canonical rows.\n\nSo the honest answer: not a single cause. The stack is:\n\n1. No reviewer has touched 14 pending proposals\n2. No apply path exists for complex `attach_evidence` packets (schema gap)\n3. `source_ref` values don't resolve to `public.sources` row IDs \u2014 an apply worker would need a row-link audit before it could safely write\n\nNext Cory-style follow-up: The proof-changing action on the pending pile is a reviewer pass to cancel canary noise (3 edge tests), action or cancel duplicates on Good Strategy/Bad Strategy (at least 3 overlapping proposals), and then route the substantive remaining ones through an apply path that hasn't been built yet. Want me to draft a triage packet?",
      "started_at_utc": "2026-07-13T00:17:33.761151+00:00",
      "turn": 4
    },
    {
      "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
      "dimension": "demo_changes_kb_direct_claim",
      "ended_at_utc": "2026-07-13T00:18:50.471863+00:00",
      "evidence_tier": "live_vps_gatewayrunner_temp_profile",
      "mutates_kb": false,
      "ok": true,
      "prompt": "Can I demo that Leo changes the KB?",
      "prompt_id": "DC-05",
      "reply": "Staging yes, canonical KB change not safe to demo from chat.\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nHere's the exact demo tier breakdown:\n\nTier 1 \u2014 live-proven, safe to demo now:\nLeo can write a real proposal to `kb_stage.kb_proposals` and read it back. The strict existing-ID `add_edge` path is live-proven \u2014 it was used for the canary at `00957f6c` (applied 2026-07-09). A demo here is real: Leo stages a proposal, you see the row, you see the rationale, you see the edge it would create. That's a genuine KB write, just not to canonical `public.*`.\n\nTier 2 \u2014 clone-proven, not production-ready:\nThe guarded `approve_claim` bundles and rich packet apply path exist in the repo and have been tested on a clone. But the production permission migration and apply worker remain disabled. Approved legacy packets like `a64df080` (7 Powers) don't carry a strict `apply_payload`, so even if the worker existed, they're not worker-applyable as-is.\n\nTier 3 \u2014 not demoable from chat:\nCanonical `public.*` mutation. Approved is not the same as applied. A canonical demo still requires explicit operator/admin authorization, the matching reviewed apply path, and retained before/after postflight readback. That's not a chat command.\n\nWhat you can safely demo right now: Leo stages a real `add_edge` proposal \u2014 picks two existing canonical claim IDs, writes the row, reads it back with `teleo-kb show-proposal`. The audience sees a genuine DB write with full provenance. Just not a canonical table mutation.\n\nWant me to stage one live for the demo?\n\nNext Cory-style follow-up: Say the word and I'll write a clean demo `add_edge` proposal to `kb_stage.kb_proposals` using two real canonical claim IDs \u2014 you get a live before/after readback you can show.",
      "started_at_utc": "2026-07-13T00:18:29.512196+00:00",
      "turn": 5
    },
    {
      "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
      "dimension": "soul_identity_direct_claim",
      "ended_at_utc": "2026-07-13T00:19:01.072726+00:00",
      "evidence_tier": "live_vps_gatewayrunner_temp_profile",
      "mutates_kb": false,
      "ok": true,
      "prompt": "If we changed SOUL.md, did we change Leo's canonical identity?",
      "prompt_id": "DC-06",
      "reply": "No.\n\n`SOUL.md` is a runtime/rendered artifact, not canonical Postgres, not the source of truth, not a canonical commit, and not collective truth. Editing it affects the next runtime session's injected context \u2014 what Leo reads at session start \u2014 but it does not write or update any rows in `public.*`.\n\nRow-level proof would require current readback of new or updated row IDs in `public.claims`, `public.sources`, `public.claim_evidence`, `public.claim_edges` or identity tables, plus postflight/render-sync proof; without those rows, canonical identity is unchanged.\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nThose counts haven't moved from earlier in this session. No identity-related rows were written.\n\nThe provenance chain for canonical identity is:\n- Canonical: Postgres `public.*` rows \u2014 claims, sources, evidence, edges, agent/identity tables\n- Rendered: `SOUL.md` \u2014 generated from DB rows, injected at session start\n- Gap: the render/sync path that keeps them in sync is not proven as a live automated pipeline\n\nSo if `SOUL.md` was edited directly, you have a runtime Leo whose session context diverges from canonical Postgres \u2014 until the next render-sync from DB overrides it. The DB didn't change; the rendered file did.\n\nNext Cory-style follow-up: The proof-changing action is a render-sync run that regenerates `SOUL.md` from current canonical `public.*` rows, with before/after diff retained \u2014 that would either close the gap or surface which DB rows are missing.",
      "started_at_utc": "2026-07-13T00:18:50.472273+00:00",
      "turn": 6
    }
  ],
  "score": {
    "coverage": "partial",
    "expected_prompt_count": 6,
    "expected_prompt_ids": [
      "DC-01",
      "DC-02",
      "DC-03",
      "DC-04",
      "DC-05",
      "DC-06"
    ],
    "failures": [],
    "missing_prompt_ids": [],
    "pass": true,
    "passes": 6,
    "prompt_count": 6,
    "scores": [
      {
        "dimension": "db_changed_direct_claim",
        "overclaim_detected": false,
        "pass": true,
        "prompt_id": "DC-01",
        "response_issue_detected": false,
        "signals": {
          "canonical_db": true,
          "cory_followup": true,
          "no_overclaim": true,
          "row_level_proof": true,
          "state_semantics": true,
          "structured_db_readback": true
        }
      },
      {
        "dimension": "helmer_live_direct_claim",
        "overclaim_detected": false,
        "pass": true,
        "prompt_id": "DC-02",
        "response_issue_detected": false,
        "signals": {
          "authorization": true,
          "canonical_db": true,
          "cory_followup": true,
          "no_overclaim": true,
          "row_level_proof": true,
          "structured_db_readback": true
        }
      },
      {
        "dimension": "decision_matrix_direct_claim",
        "overclaim_detected": false,
        "pass": true,
        "prompt_id": "DC-03",
        "response_issue_detected": false,
        "signals": {
          "canonical_db": true,
          "cory_followup": true,
          "decision_matrix": true,
          "fresh_readback": true,
          "no_overclaim": true,
          "structured_db_readback": true
        }
      },
      {
        "dimension": "document_source_direct_claim",
        "overclaim_detected": false,
        "pass": true,
        "prompt_id": "DC-04",
        "response_issue_detected": false,
        "signals": {
          "canonical_db": true,
          "cory_followup": true,
          "document_artifact_linking": true,
          "no_overclaim": true,
          "staging_or_review": true,
          "structured_db_readback": true
        }
      },
      {
        "dimension": "demo_changes_kb_direct_claim",
        "overclaim_detected": false,
        "pass": true,
        "prompt_id": "DC-05",
        "response_issue_detected": false,
        "signals": {
          "artifact": true,
          "authorization": true,
          "cory_followup": true,
          "no_overclaim": true,
          "row_level_proof": true,
          "state_semantics": true,
          "structured_db_readback": true
        }
      },
      {
        "dimension": "soul_identity_direct_claim",
        "overclaim_detected": false,
        "pass": true,
        "prompt_id": "DC-06",
        "response_issue_detected": false,
        "signals": {
          "canonical_db": true,
          "cory_followup": true,
          "identity_rendering": true,
          "no_overclaim": true,
          "row_level_proof": true,
          "structured_db_readback": true
        }
      }
    ],
    "unexpected_prompt_ids": [],
    "unknown_expected_prompt_ids": []
  },
  "source_results_json": "docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json"
}