223 lines
17 KiB
JSON
223 lines
17 KiB
JSON
{
|
|
"benchmark_layers": [
|
|
{
|
|
"id": "precise_regression",
|
|
"purpose": "Pin known IDs, proposals, and row facts so regressions are easy to diagnose.",
|
|
"runtime": "live_readonly_or_fixture"
|
|
},
|
|
{
|
|
"id": "open_ended_readonly",
|
|
"purpose": "Ask vague Cory-style questions while explicitly preventing KB mutation on live Telegram/VPS.",
|
|
"runtime": "live_readonly"
|
|
},
|
|
{
|
|
"id": "cory_style_outcome",
|
|
"purpose": "Exercise regular-use ambiguity: broad operator intent, demo pressure, missing IDs, mixed KB/runtime/governance asks.",
|
|
"runtime": "disposable_clone_or_sandbox_first"
|
|
},
|
|
{
|
|
"id": "direct_claim_followup",
|
|
"purpose": "Ask no-context Cory-style direct claim questions and require the expected answer plus the next follow-up.",
|
|
"runtime": "disposable_clone_or_sandbox_first"
|
|
}
|
|
],
|
|
"generated_at_utc": "2026-07-13T04:33:26.008200+00:00",
|
|
"include_cory_style_scenarios": false,
|
|
"include_direct_claim_followups": true,
|
|
"mode": "retained_evidence_score",
|
|
"mutates_kb": false,
|
|
"require_all": false,
|
|
"results": [
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "db_changed_direct_claim",
|
|
"ended_at_utc": "2026-07-13T04:27:04.428288+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Did we actually update the knowledge base, or is it still just proposals?",
|
|
"prompt_id": "DC-01",
|
|
"reply": "Mostly still proposals. Here's the breakdown:\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nApplied (canonical): 2\n- `00957f6c` \u2014 `add_edge` canary (strict path, applied 2026-07-09)\n- `f004bbb2` \u2014 `attach_evidence` for Leo guiding-policy revision (applied 2026-07-05)\n\nApproved but NOT applied: 3\n- `14fa5ecc` \u2014 split compressed internet-finance/capital-allocation claim; applied_at: none\n- `ac036c9d` \u2014 strategy-kernel document / claim-node hygiene; applied_at: none\n- `a64df080` \u2014 Helmer 7 Powers framework packet; applied_at: none\n\nPending review: 14 proposals (ranging from Hayek/Friston evidence to Living Systems, Rumelt, canary edges)\n\nCanceled: 7 (superseded drafts)\n\nApproved is not the same as applied. The 3 approved proposals \u2014 including the 7 Powers packet \u2014 have no `applied_at`, meaning they are staged/packet-ready but not canonical. The canonical `public.*` tables did not change for those.\n\nNext Cory-style follow-up: The proof-changing action for the 3 approved proposals is validating or rebuilding their strict `apply_payload`, then running the apply path with retained before/after postflight readback \u2014 that requires explicit operator/admin authorization and should not be run from normal chat.",
|
|
"started_at_utc": "2026-07-13T04:26:39.479688+00:00",
|
|
"turn": 1
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "helmer_live_direct_claim",
|
|
"ended_at_utc": "2026-07-13T04:27:27.245582+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Is Helmer's 7 Powers in Leo now?",
|
|
"prompt_id": "DC-02",
|
|
"reply": "No, not canonical.\n\nDB readback: proposal: `a64df080-8502-42e2-98f4-9bbdecb8da73`; status: `approved`; applied_at: `none` (database NULL).\n\nThe search for \"7 Powers helmer\" and \"seven powers\" returned zero matching canonical claims. The Helmer reasoning tool, 5 proposed claims (benefit/barrier logic, power progression, counter-positioning, network economies vs network effects, agentic capital power analysis), and all 6\u20138 source candidates exist only inside the proposal payload \u2014 not in `public.claims`, `public.sources`, `public.claim_evidence`, `public.claim_edges`, or any reasoning-tool rows.\n\nWhat exists:\n- `kb_stage.kb_proposals` \u2014 proposal `a64df080` approved with changes on 2026-06-29, `applied_at: NULL`\n- Local research files at `/home/teleo/.hermes/profiles/leoclean/kb_stage/research/7powers/`\n\nWhat does not exist:\n- No canonical `public.claims` row for any 7 Powers proposed claim\n- No `public.sources` row for Helmer book, Acquired transcript, NFX interview\n- No `public.claim_evidence` or `public.claim_edges` rows linking to those claims\n- No reasoning_tool node for \"Helmer 7 Powers\"\n\nApproved is not the same as applied. The proposal is packet-ready but the `apply_payload` is a rich `create_reasoning_tool_and_claim_set` bundle that the bridge has no native `apply` command for \u2014 it would require operator-level validation and execution.\n\nNext Cory-style follow-up: Validate or rebuild a strict `apply_payload` (source rows + claim rows + evidence rows + edges) for `a64df080`, then run the apply path with retained before/after postflight readback \u2014 requires explicit operator/admin authorization, not a normal chat command.",
|
|
"started_at_utc": "2026-07-13T04:27:04.428643+00:00",
|
|
"turn": 2
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "decision_matrix_direct_claim",
|
|
"ended_at_utc": "2026-07-13T04:27:41.751314+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Did the decision matrix approve this already?",
|
|
"prompt_id": "DC-03",
|
|
"reply": "No. Fresh readback: the decision-matrix schema is absent; reviewer status is not a decision-matrix vote.\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nNone of the decision-matrix tables exist in any schema:\n\n- `kb_stage.matrix_voters` \u2014 absent\n- `kb_stage.proposal_votes` \u2014 absent\n- `kb_stage.proposal_decisions` \u2014 absent\n- `public.*` equivalents \u2014 all absent\n\nThe `approved` status on proposal `a64df080` reflects m3ta's direct reviewer approval on 2026-06-29, not a futarchy/decision-matrix vote. Those are two different paths. The matrix path has not shipped.\n\nNext Cory-style follow-up: If you want the decision-matrix path to count, the schema (`matrix_voters`, `proposal_votes`, `proposal_decisions`) needs to be deployed \u2014 that's a separate infrastructure step before any proposal can claim matrix approval.",
|
|
"started_at_utc": "2026-07-13T04:27:27.246033+00:00",
|
|
"turn": 3
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "document_source_direct_claim",
|
|
"ended_at_utc": "2026-07-13T04:28:15.355617+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Are the pending proposals stuck because the documents are not pointed at the right source rows?",
|
|
"prompt_id": "DC-04",
|
|
"reply": "Not just pointer mismatch. The 14 pending proposals have at least three distinct blockers, and source-row pointer issues are only one layer of one of them.\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nBreaking it down by type:\n\n`add_edge` canaries (4 proposals: `8dfedb3f`, `03f7853e`, `e60a48af`, `c824da66`)\nThese reference existing canonical claim IDs by UUID. The strict `add_edge` apply path is live-proven. They are not stuck on pointer mismatch \u2014 they are stuck on missing operator review and authorization. These could be applied now if a reviewer approves them.\n\n`attach_evidence` proposals (10 proposals: `f5036882`, `bcec8528`, `e987b07c`, `23e896c2`, `da53f198`, `e106ca60`, `e60fa116`, `416385fe`, `124942fd`, `a59311ae`)\nThese carry document grounding work \u2014 raw Telegram-ingested files, KB-stage research files, logical source keys \u2014 that need to land as `public.sources` rows before evidence links can be written. The layers are distinct:\n- raw local/Telegram file references\n- `kb_stage` document evaluations\n- proposal `source_ref` / logical source keys\n- canonical `public.sources` rows\n\nThe missing proof is a row-link audit: do the logical source keys in each proposal payload already have matching `public.sources` rows? And is there a live apply path for `attach_evidence` that writes those rows? The answer to the second question is no \u2014 only the strict `add_edge` apply path is live-proven. The `attach_evidence` apply worker is not live.\n\nSo the blockers are:\n1. No live `attach_evidence` apply path (all 10 evidence proposals)\n2. No reviewer decision yet (all 14 pending)\n3. For source-heavy proposals, possibly missing `public.sources` rows \u2014 but that cannot be confirmed without a row-link audit comparing proposal `source_ref` keys against existing canonical `public.sources` UUIDs\n\nNext Cory-style follow-up: The proof-changing action is a row-link audit on the `attach_evidence` proposals \u2014 check whether their logical source keys already exist as `public.sources` rows, then assess whether to rebuild as strict payloads or wait for the apply worker to ship before authorizing.",
|
|
"started_at_utc": "2026-07-13T04:27:41.751922+00:00",
|
|
"turn": 4
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "demo_changes_kb_direct_claim",
|
|
"ended_at_utc": "2026-07-13T04:28:40.092234+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Can I demo that Leo changes the KB?",
|
|
"prompt_id": "DC-05",
|
|
"reply": "Staging yes, canonical KB change not safe to demo from chat.\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nApproved is not the same as applied.\n\nDemo tiers, from safest to most consequential:\n\nTier 1 \u2014 Staging write (safe now, no authorization needed)\nCreate a new canary `add_edge` proposal, write it to `kb_stage.kb_proposals`, read it back. That proves the Telegram \u2192 Leo \u2192 DB staging path end-to-end. No canonical tables touched.\n\nTier 2 \u2014 Strict `add_edge` apply (live-proven path, needs operator authorization)\nThe 4 pending `add_edge` canaries (`8dfedb3f`, `03f7853e`, `e60a48af`, `c824da66`) reference existing canonical claim IDs. The strict apply path for `add_edge` is live-proven. A reviewer could authorize one of these and it would actually write a new `public.claim_edges` row \u2014 that is a real canonical change. Requires explicit operator/admin authorization before running from chat.\n\nTier 3 \u2014 `attach_evidence` or rich multi-row packets (not ready)\nThe `attach_evidence` apply worker is not live. No rich packet (like 7 Powers) has a matching strict `apply_payload`. These cannot be cleanly demoed without rebuilding the payload and shipping the apply worker first.\n\nCurrent tier ceiling in plain terms:\n- Strict `add_edge` path: live-proven\n- Guarded `approve_claim` bundles: clone-proven, behind separate reviewer/apply roles\n- Production permission migration and apply worker: disabled\n- Approved legacy packets without strict `apply_payload`: not worker-applyable\n\nIf you want to demo canonical change, the safest real demo is a new staging canary followed by a reviewed strict `add_edge` apply with retained before/after postflight readback. That requires your explicit authorization \u2014 not something Leo runs from normal chat on its own.\n\nNext Cory-style follow-up: Say the word and I can stage a fresh `add_edge` canary right now (Tier 1, safe) \u2014 or if you want to authorize a Tier 2 canonical apply on one of the existing pending canaries, name which one and I'll prepare the apply packet for your review.",
|
|
"started_at_utc": "2026-07-13T04:28:15.356361+00:00",
|
|
"turn": 5
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "soul_identity_direct_claim",
|
|
"ended_at_utc": "2026-07-13T04:28:51.365739+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "If we changed SOUL.md, did we change Leo's canonical identity?",
|
|
"prompt_id": "DC-06",
|
|
"reply": "No.\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\n`SOUL.md` is a runtime/rendered artifact \u2014 not canonical Postgres, not the source of truth, not a canonical commit, and not collective truth. Editing it affects the next runtime session's context injection, but it does not touch any DB rows.\n\nRow-level proof would require current readback of new or updated row IDs in `public.claims`, `public.sources`, `public.claim_evidence`, `public.claim_edges`, or identity tables, plus postflight/render-sync proof. Without those rows, canonical identity is unchanged.\n\nThe render direction is: Postgres KB \u2192 SOUL.md, not the reverse. Writing to SOUL.md directly is editing the output, not the source. A change to Leo's canonical identity requires:\n1. A proposal staged to `kb_stage.kb_proposals`\n2. Reviewer approval\n3. Apply to `public.*` canonical tables\n4. Render/sync that regenerates SOUL.md from the updated DB\n\nWhat SOUL.md edits do buy: the next Leo session picks up the changed runtime context immediately. That has real operational effect. It just isn't canonical truth, and it drifts back toward the DB on the next render/sync.\n\nNext Cory-style follow-up: If a specific SOUL.md change should become canonical, stage it as a proposal now \u2014 I can draft the KB proposal from the diff and route it for review.",
|
|
"started_at_utc": "2026-07-13T04:28:40.092648+00:00",
|
|
"turn": 6
|
|
}
|
|
],
|
|
"score": {
|
|
"coverage": "partial",
|
|
"expected_prompt_count": 6,
|
|
"expected_prompt_ids": [
|
|
"DC-01",
|
|
"DC-02",
|
|
"DC-03",
|
|
"DC-04",
|
|
"DC-05",
|
|
"DC-06"
|
|
],
|
|
"failures": [],
|
|
"missing_prompt_ids": [],
|
|
"pass": true,
|
|
"passes": 6,
|
|
"prompt_count": 6,
|
|
"scores": [
|
|
{
|
|
"dimension": "db_changed_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": true,
|
|
"prompt_id": "DC-01",
|
|
"response_issue_detected": false,
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": true,
|
|
"no_overclaim": true,
|
|
"row_level_proof": true,
|
|
"state_semantics": true,
|
|
"structured_db_readback": true
|
|
}
|
|
},
|
|
{
|
|
"dimension": "helmer_live_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": true,
|
|
"prompt_id": "DC-02",
|
|
"response_issue_detected": false,
|
|
"signals": {
|
|
"authorization": true,
|
|
"canonical_db": true,
|
|
"cory_followup": true,
|
|
"no_overclaim": true,
|
|
"row_level_proof": true,
|
|
"structured_db_readback": true
|
|
}
|
|
},
|
|
{
|
|
"dimension": "decision_matrix_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": true,
|
|
"prompt_id": "DC-03",
|
|
"response_issue_detected": false,
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": true,
|
|
"decision_matrix": true,
|
|
"fresh_readback": true,
|
|
"no_overclaim": true,
|
|
"structured_db_readback": true
|
|
}
|
|
},
|
|
{
|
|
"dimension": "document_source_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": true,
|
|
"prompt_id": "DC-04",
|
|
"response_issue_detected": false,
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": true,
|
|
"document_artifact_linking": true,
|
|
"no_overclaim": true,
|
|
"staging_or_review": true,
|
|
"structured_db_readback": true
|
|
}
|
|
},
|
|
{
|
|
"dimension": "demo_changes_kb_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": true,
|
|
"prompt_id": "DC-05",
|
|
"response_issue_detected": false,
|
|
"signals": {
|
|
"artifact": true,
|
|
"authorization": true,
|
|
"cory_followup": true,
|
|
"no_overclaim": true,
|
|
"row_level_proof": true,
|
|
"state_semantics": true,
|
|
"structured_db_readback": true
|
|
}
|
|
},
|
|
{
|
|
"dimension": "soul_identity_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": true,
|
|
"prompt_id": "DC-06",
|
|
"response_issue_detected": false,
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": true,
|
|
"identity_rendering": true,
|
|
"no_overclaim": true,
|
|
"row_level_proof": true,
|
|
"structured_db_readback": true
|
|
}
|
|
}
|
|
],
|
|
"unexpected_prompt_ids": [],
|
|
"unknown_expected_prompt_ids": []
|
|
},
|
|
"source_results_json": "docs/reports/leo-working-state-20260709/leo-restart-survival-direct-claim-current.json"
|
|
}
|