{
  "benchmark_layers": [
    {
      "id": "precise_regression",
      "purpose": "Pin known IDs, proposals, and row facts so regressions are easy to diagnose.",
      "runtime": "live_readonly_or_fixture"
    },
    {
      "id": "open_ended_readonly",
      "purpose": "Ask vague Cory-style questions while explicitly preventing KB mutation on live Telegram/VPS.",
      "runtime": "live_readonly"
    },
    {
      "id": "cory_style_outcome",
      "purpose": "Exercise regular-use ambiguity: broad operator intent, demo pressure, missing IDs, mixed KB/runtime/governance asks.",
      "runtime": "disposable_clone_or_sandbox_first"
    },
    {
      "id": "direct_claim_followup",
      "purpose": "Ask no-context Cory-style direct claim questions and require the expected answer plus the next follow-up.",
      "runtime": "disposable_clone_or_sandbox_first"
    }
  ],
  "generated_at_utc": "2026-07-10T01:11:29.518898+00:00",
  "include_cory_style_scenarios": false,
  "include_direct_claim_followups": true,
  "mode": "retained_evidence_score",
  "mutates_kb": false,
  "require_all": false,
  "results": [
    {
      "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
      "dimension": "db_changed_direct_claim",
      "ended_at_utc": "2026-07-10T01:05:52.647447+00:00",
      "evidence_tier": "live_vps_gatewayrunner_temp_profile",
      "mutates_kb": false,
      "ok": true,
      "prompt": "Did we actually update the knowledge base, or is it still just proposals?",
      "prompt_id": "DC-01",
      "reply": "Mostly still proposals. Here's the full picture:\n\nDirect answer: 2 proposals applied to canonical public.*, 3 approved but not yet applied, 14 pending review, 7 canceled.\n\nReadback: list-proposals --status all --limit 50 + decision-matrix-status.\n\nCanonical vs staged split:\n\nApplied (public.* changed):\n- `00957f6c` \u2014 add_edge canary (Nascent tech relates MetaDAO launches) \u2014 applied 2026-07-09\n- `f004bbb2` \u2014 attach_evidence Leo guiding-policy revision \u2014 applied 2026-07-05\n\nApproved but not applied (applied_at is empty \u2014 staged/packet-ready, not canonical):\n- `14fa5ecc` \u2014 split compressed internet-finance/capital-allocation claim\n- `ac036c9d` \u2014 strategy-kernel doc eval + claim-node hygiene/Hayek anchor\n- `a64df080` \u2014 Helmer 7 Powers framework packet (the big one)\n\nPending review (14 proposals): canary edges, strategy-kernel revisions, concept maps (waves of change, living-system/FEP, Good Strategy/Bad Strategy), Rumelt evidence, Markov blanket sources, Hayek anchor, Agents United doc eval, Leo strategy-kernel v2 revision.\n\nDecision-matrix: not shipped. No matrix_voters, proposal_votes, or proposal_decisions tables exist. \"Approved\" means reviewer sign-off in kb_stage.kb_proposals, not a futarchy/matrix vote.\n\nSo Helmer 7 Powers, Leo strategy-kernel v2, concept maps, and the Hayek/Rumelt evidence are all still staged \u2014 approved or pending, not canonical.\n\nNext Cory-style follow-up: The proof-changing action is applying the 3 approved proposals (`14fa5ecc`, `ac036c9d`, `a64df080`) \u2014 that requires explicit operator/admin authorization and a dedicated apply+postflight sequence, not a chat command.",
      "started_at_utc": "2026-07-10T01:05:22.771940+00:00",
      "turn": 1
    },
    {
      "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
      "dimension": "helmer_live_direct_claim",
      "ended_at_utc": "2026-07-10T01:06:02.802948+00:00",
      "evidence_tier": "live_vps_gatewayrunner_temp_profile",
      "mutates_kb": false,
      "ok": true,
      "prompt": "Is Helmer's 7 Powers in Leo now?",
      "prompt_id": "DC-02",
      "reply": "No, not canonical.\n\nFrom the readback we just did:\n\nProposal `a64df080` \u2014 Helmer 7 Powers framework packet\n- status: `approved`\n- reviewed_at: `2026-06-29`\n- applied_at: empty\n\nThat means it passed reviewer sign-off in `kb_stage.kb_proposals` but was never applied to `public.*`. No canonical `public.sources`, `public.claims`, evidence rows, or edges exist for it yet.\n\nDecision-matrix path is also not shipped, so \"approved\" here means m3ta sign-off only \u2014 not a futarchy vote.\n\nNext Cory-style follow-up: Applying `a64df080` to canonical `public.*` is the proof-changing action \u2014 requires explicit operator/admin authorization and a dedicated apply+postflight sequence. Want me to draft the admin review packet for it?",
      "started_at_utc": "2026-07-10T01:05:52.647836+00:00",
      "turn": 2
    },
    {
      "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
      "dimension": "decision_matrix_direct_claim",
      "ended_at_utc": "2026-07-10T01:06:12.962312+00:00",
      "evidence_tier": "live_vps_gatewayrunner_temp_profile",
      "mutates_kb": false,
      "ok": true,
      "prompt": "Did the decision matrix approve this already?",
      "prompt_id": "DC-03",
      "reply": "No. The decision-matrix path is not shipped.\n\nFrom the `decision-matrix-status` readback we just ran:\n\n- `kb_stage.matrix_voters`: absent\n- `kb_stage.proposal_votes`: absent\n- `kb_stage.proposal_decisions`: absent\n- Same for all `public.*` matrix tables\n\nNone of the infrastructure for a futarchy/matrix vote exists yet. The `approved` status on `a64df080` is m3ta reviewer sign-off in `kb_stage.kb_proposals` \u2014 that's the only approval mechanism that ran.\n\nSo: approved by m3ta, not approved by decision matrix. Those are different things, and only the second one would constitute collective governance approval under the intended architecture.\n\nNext Cory-style follow-up: The decision-matrix schema needs to be designed and shipped before any proposal can claim matrix approval. Want to scope that as a separate track, or treat m3ta sign-off as sufficient to apply `a64df080` now?",
      "started_at_utc": "2026-07-10T01:06:02.803335+00:00",
      "turn": 3
    },
    {
      "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
      "dimension": "document_source_direct_claim",
      "ended_at_utc": "2026-07-10T01:06:51.894893+00:00",
      "evidence_tier": "live_vps_gatewayrunner_temp_profile",
      "mutates_kb": false,
      "ok": true,
      "prompt": "Are the pending proposals stuck because the documents are not pointed at the right source rows?",
      "prompt_id": "DC-04",
      "reply": "Not just a pointer mismatch \u2014 that's one layer of a multi-layer problem.\n\nWhat the readback actually shows:\n\n1. Source refs are Telegram-local paths, not public.sources rows\n   - `e987b07c` (Good Strategy/Bad Strategy): source_ref is `telegram-resolved-attachment:doc_996e2e86cd94_...pdf` \u2014 a local cache path, no `public.sources` row confirmed\n   - `23e896c2` (living-system map): source_ref is a `telegram-thread:` reference \u2014 not a canonical source key\n   - `a59311ae` (Hayek/Ramstead): references `existing_claim_id` rows that may or may not have sources attached\n   - `f5036882` (strategy-kernel doc): source_ref is a `telegram-doc:` local cache path\n\n2. No reviewer has touched them \u2014 reviewed_at is empty on all 14 pending proposals. They are not stuck on a pointer problem; they haven't been reviewed at all yet.\n\n3. The \"stuck\" cause is absence of review, not bad pointers. The pointer issue is real \u2014 telegram-doc and telegram-resolved-attachment refs aren't the same as confirmed `public.sources` rows \u2014 but that's a quality concern a reviewer would flag during review, not a pre-review blocker.\n\n4. The two applied proposals got through because they were canary/procedural (edge canary) or had explicit m3ta sign-off on a narrow scope (Leo guiding-policy revision).\n\nSo the honest answer: pending proposals are stuck because no reviewer has processed them, and when review does happen, some will need source rows created in `public.sources` before evidence can be formally attached. Pointer hygiene is a second-order problem that surfaces during apply, not the root cause of the backlog.\n\nNext Cory-style follow-up: The proof-changing action is a reviewer/operator working through the pending queue \u2014 starting with the 3 approved-but-not-applied proposals (`14fa5ecc`, `ac036c9d`, `a64df080`) before clearing the pending backlog. Want me to draft a prioritized review packet?",
      "started_at_utc": "2026-07-10T01:06:12.962876+00:00",
      "turn": 4
    },
    {
      "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
      "dimension": "demo_changes_kb_direct_claim",
      "ended_at_utc": "2026-07-10T01:07:04.967689+00:00",
      "evidence_tier": "live_vps_gatewayrunner_temp_profile",
      "mutates_kb": false,
      "ok": true,
      "prompt": "Can I demo that Leo changes the KB?",
      "prompt_id": "DC-05",
      "reply": "Yes, but the demo tier matters. Two tiers:\n\nTier 1 \u2014 Safe demo (available now, no authorization needed)\nLeo stages a real write to `kb_stage.kb_proposals` and reads it back. You can see the proposal appear in `list-proposals`, inspect the payload, and confirm the write landed in Postgres. This is what the canary proposals (`8dfedb3f`, `00957f6c`) proved. I can do this live right now from this chat.\n\nTier 2 \u2014 Canonical demo (requires explicit operator authorization)\nMutating `public.claims`, `public.sources`, `public.claim_evidence`, or `public.claim_edges` is locked behind a dedicated apply+postflight sequence. Even with the 3 approved proposals sitting ready, I do not run that from chat without explicit operator sign-off for that exact action. The product flow is: review packet \u2192 authorization \u2192 apply tool \u2192 postflight readback confirms rows changed.\n\nWhat the Tier 1 demo proves: Leo can receive input, evaluate it, and write a structured reviewable proposal to durable Postgres storage in real time \u2014 that's the coordination-primitive that matters.\n\nWhat it does not prove: that canonical `public.*` changed. For that you need Tier 2 authorization.\n\nNext Cory-style follow-up: Want me to run a live Tier 1 demo now \u2014 stage a fresh canary proposal and read it back \u2014 or do you want to authorize a Tier 2 apply for one of the 3 approved proposals?",
      "started_at_utc": "2026-07-10T01:06:51.895411+00:00",
      "turn": 5
    },
    {
      "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
      "dimension": "soul_identity_direct_claim",
      "ended_at_utc": "2026-07-10T01:07:15.156039+00:00",
      "evidence_tier": "live_vps_gatewayrunner_temp_profile",
      "mutates_kb": false,
      "ok": true,
      "prompt": "If we changed SOUL.md, did we change Leo's canonical identity?",
      "prompt_id": "DC-06",
      "reply": "No.\n\nSOUL.md is a runtime/rendered artifact. It gets injected into Leo's session context at startup \u2014 it shapes how Leo behaves in that session \u2014 but it does not change canonical Postgres rows.\n\nCanonical identity lives in the DB:\n- `public.claims` \u2014 beliefs, positions, strategy nodes\n- `public.sources` \u2014 grounding evidence\n- `public.claim_evidence` and `public.claim_edges` \u2014 the graph that connects them\n\nEditing SOUL.md without a corresponding DB apply means:\n- Next session that reads the updated SOUL.md will behave differently\n- But the KB graph still reflects the old state\n- The two can drift \u2014 runtime Leo and canonical Leo diverge\n- A DB render/sync that regenerates SOUL.md from canonical rows would overwrite the manual edit\n\nThe proof that canonical identity changed requires: new or updated rows in `public.*` plus a render/sync showing SOUL.md was generated from those rows, not edited directly.\n\nSo a SOUL.md edit is a runtime patch, not a canonical commit. It has the same status as runtime memory \u2014 local continuity, not collective truth.\n\nNext Cory-style follow-up: If the SOUL.md edit captured something that should be canonical, the proof-changing action is staging it as a KB proposal and running the apply+postflight sequence to land it in `public.*`. Want me to inspect what changed in SOUL.md and draft the corresponding proposal?",
      "started_at_utc": "2026-07-10T01:07:04.973021+00:00",
      "turn": 6
    }
  ],
  "score": {
    "coverage": "partial",
    "expected_prompt_count": 6,
    "expected_prompt_ids": [
      "DC-01",
      "DC-02",
      "DC-03",
      "DC-04",
      "DC-05",
      "DC-06"
    ],
    "failures": [],
    "missing_prompt_ids": [],
    "pass": true,
    "passes": 6,
    "prompt_count": 6,
    "scores": [
      {
        "dimension": "db_changed_direct_claim",
        "overclaim_detected": false,
        "pass": true,
        "prompt_id": "DC-01",
        "signals": {
          "canonical_db": true,
          "cory_followup": true,
          "no_overclaim": true,
          "row_level_proof": true,
          "state_semantics": true
        }
      },
      {
        "dimension": "helmer_live_direct_claim",
        "overclaim_detected": false,
        "pass": true,
        "prompt_id": "DC-02",
        "signals": {
          "authorization": true,
          "canonical_db": true,
          "cory_followup": true,
          "no_overclaim": true,
          "row_level_proof": true
        }
      },
      {
        "dimension": "decision_matrix_direct_claim",
        "overclaim_detected": false,
        "pass": true,
        "prompt_id": "DC-03",
        "signals": {
          "canonical_db": true,
          "cory_followup": true,
          "decision_matrix": true,
          "fresh_readback": true,
          "no_overclaim": true
        }
      },
      {
        "dimension": "document_source_direct_claim",
        "overclaim_detected": false,
        "pass": true,
        "prompt_id": "DC-04",
        "signals": {
          "canonical_db": true,
          "cory_followup": true,
          "document_artifact_linking": true,
          "no_overclaim": true,
          "staging_or_review": true
        }
      },
      {
        "dimension": "demo_changes_kb_direct_claim",
        "overclaim_detected": false,
        "pass": true,
        "prompt_id": "DC-05",
        "signals": {
          "artifact": true,
          "authorization": true,
          "cory_followup": true,
          "no_overclaim": true,
          "row_level_proof": true,
          "state_semantics": true
        }
      },
      {
        "dimension": "soul_identity_direct_claim",
        "overclaim_detected": false,
        "pass": true,
        "prompt_id": "DC-06",
        "signals": {
          "canonical_db": true,
          "cory_followup": true,
          "identity_rendering": true,
          "no_overclaim": true,
          "row_level_proof": true
        }
      }
    ],
    "unexpected_prompt_ids": [],
    "unknown_expected_prompt_ids": []
  },
  "source_results_json": "docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json"
}