211 lines
14 KiB
JSON
211 lines
14 KiB
JSON
{
|
|
"benchmark_layers": [
|
|
{
|
|
"id": "precise_regression",
|
|
"purpose": "Pin known IDs, proposals, and row facts so regressions are easy to diagnose.",
|
|
"runtime": "live_readonly_or_fixture"
|
|
},
|
|
{
|
|
"id": "open_ended_readonly",
|
|
"purpose": "Ask vague Cory-style questions while explicitly preventing KB mutation on live Telegram/VPS.",
|
|
"runtime": "live_readonly"
|
|
},
|
|
{
|
|
"id": "cory_style_outcome",
|
|
"purpose": "Exercise regular-use ambiguity: broad operator intent, demo pressure, missing IDs, mixed KB/runtime/governance asks.",
|
|
"runtime": "disposable_clone_or_sandbox_first"
|
|
},
|
|
{
|
|
"id": "direct_claim_followup",
|
|
"purpose": "Ask no-context Cory-style direct claim questions and require the expected answer plus the next follow-up.",
|
|
"runtime": "disposable_clone_or_sandbox_first"
|
|
}
|
|
],
|
|
"generated_at_utc": "2026-07-10T02:25:47.447965+00:00",
|
|
"include_cory_style_scenarios": false,
|
|
"include_direct_claim_followups": true,
|
|
"mode": "retained_evidence_score",
|
|
"mutates_kb": false,
|
|
"require_all": false,
|
|
"results": [
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "db_changed_direct_claim",
|
|
"ended_at_utc": "2026-07-10T02:24:01.289425+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Did we actually update the knowledge base, or is it still just proposals?",
|
|
"prompt_id": "DC-01",
|
|
"reply": "Mostly still proposals. Here's the current readback:\n\nApplied (canonical `public.*` changed):\n- `00957f6c` \u2014 add_edge canary, applied 2026-07-09\n- `f004bbb2` \u2014 attach_evidence (Leo guiding-policy revision), applied 2026-07-05\n\nApproved but NOT applied (staged, not canonical):\n- `14fa5ecc` \u2014 split compressed internet-finance/capital-allocation claim\n- `ac036c9d` \u2014 strategy-kernel doc eval / claim-node hygiene / Hayek reframe\n- `a64df080` \u2014 Helmer 7 Powers framework packet (approved 2026-06-29, still waiting)\n\nPending review (~11 proposals):\nStrategy-kernel revisions, Rumelt/Hayek evidence attaches, Markov blanket sources, a live Telegram add_edge from yesterday \u2014 none reviewed yet.\n\nCanceled (~6): Earlier superseded waves-of-change and strategy-kernel drafts.\n\nSo the canonical DB has 2 applied writes. Everything substantive \u2014 7 Powers, strategy-kernel revisions, Hayek/Rumelt evidence \u2014 is approved or pending but not in `public.*`.\n\nI cannot claim canonical DB changed for those items until `applied_at` rows and `public.*` readback confirm the apply ran.\n\nNext Cory-style follow-up: the three approved proposals (`14fa5ecc`, `ac036c9d`, `a64df080`) are packet-ready \u2014 the proof-changing action is running the apply sequence with operator authorization and confirming `public.*` rows after. Want me to draft the admin review packet for those three?",
|
|
"started_at_utc": "2026-07-10T02:23:29.716841+00:00",
|
|
"turn": 1
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "helmer_live_direct_claim",
|
|
"ended_at_utc": "2026-07-10T02:24:20.372589+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Is Helmer's 7 Powers in Leo now?",
|
|
"prompt_id": "DC-02",
|
|
"reply": "No, not canonical.\n\nProposal `a64df080` \u2014 the revised 7 Powers framework packet \u2014 was approved on 2026-06-29. `applied_at` is empty. No `public.sources`, `public.claims`, evidence rows, or reasoning-tool nodes for Helmer exist in `public.*` \u2014 the `search \"7 Powers Helmer\"` returned zero relevant claims.\n\nStatus: approved/staged, not canonical.\n\nCanonical: nothing in `public.*`\nStaged: `a64df080` \u2014 approved, `applied_at` empty\nPredecessor: `cea989fc` \u2014 canceled (superseded by `a64df080`)\n\nNext Cory-style follow-up: the proof-changing action is applying `a64df080` \u2014 operator authorization + apply sequence + `public.*` readback to confirm source, framework node, power-definition claims, and edges landed. Want me to draft the admin review packet for it?",
|
|
"started_at_utc": "2026-07-10T02:24:01.290646+00:00",
|
|
"turn": 2
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "decision_matrix_direct_claim",
|
|
"ended_at_utc": "2026-07-10T02:24:30.827780+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Did the decision matrix approve this already?",
|
|
"prompt_id": "DC-03",
|
|
"reply": "No \u2014 the decision-matrix path is not shipped.\n\n`kb_stage.matrix_voters`, `proposal_votes`, and `proposal_decisions` are all absent. None of the required tables exist in either schema.\n\nThe `approved` status on `a64df080` is a reviewer approval recorded in `kb_stage.kb_proposals` \u2014 that's m3ta signing off in chat on 2026-06-29. It is not a futarchy/decision-matrix vote. Those are different things and the matrix mechanics haven't been built yet.\n\nSo the approval chain for 7 Powers is: m3ta reviewed and approved in Telegram, Leo staged it, no canonical apply has run, no decision-matrix infrastructure exists.\n\nNext Cory-style follow-up: the decision-matrix tables need to be created before any proposal can claim matrix approval. If the current governance model treats m3ta reviewer approval as sufficient to authorize an apply, that should be made explicit in the apply tooling \u2014 want me to draft what that contract should look like?",
|
|
"started_at_utc": "2026-07-10T02:24:20.372895+00:00",
|
|
"turn": 3
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "document_source_direct_claim",
|
|
"ended_at_utc": "2026-07-10T02:25:00.500000+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Are the pending proposals stuck because the documents are not pointed at the right source rows?",
|
|
"prompt_id": "DC-04",
|
|
"reply": "Not just a pointer mismatch \u2014 that's too single-cause. Here's what the readback actually shows:\n\nReal blocking cause: no apply tooling. The bridge can stage proposals but has no `teleo-kb apply-*` command. All three approved proposals (`a64df080`, `14fa5ecc`, `ac036c9d`) are packet-ready but there is no operator-authorized apply path to run. That is the structural blocker.\n\nSource pointer layer \u2014 real but secondary: The `source_ref` fields in the approved proposals point to Telegram refs and local file paths, not `public.sources` rows. The `source_candidates` inside each payload use `source_key` identifiers (e.g., `hayek_use_of_knowledge_in_society_fee_snapshot`, `m3ta_telegram_approval_supersede_split_2026_07_05`) that have no confirmed canonical `public.sources` rows behind them. So when an apply does run, the apply sequence needs to create those source rows first before attaching evidence or edges. A row-link audit is needed.\n\nSchema gap layer \u2014 also real: The proposals use `attach_evidence` as a workaround because the bridge lacks `propose-claim`, `propose-concept-node`, and `propose-claim-body` command types. The payloads carry the full intent but the bridge can't translate them into typed mutations.\n\nSummary of layers:\n- Primary blocker: no apply command + no operator-authorized apply sequence has run\n- Secondary: source_ref/source_key rows need to exist in `public.sources` before evidence attaches land\n- Tertiary: schema gap means typed mutations require manual apply work\n\nNext Cory-style follow-up: the proof-changing action is drafting the admin review packet for the three approved proposals with explicit before/after rows, required `public.sources` creation steps, and the guarded apply contract \u2014 want me to build that now?",
|
|
"started_at_utc": "2026-07-10T02:24:30.830313+00:00",
|
|
"turn": 4
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "demo_changes_kb_direct_claim",
|
|
"ended_at_utc": "2026-07-10T02:25:10.661485+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Can I demo that Leo changes the KB?",
|
|
"prompt_id": "DC-05",
|
|
"reply": "Yes, but the demo tier matters. Two levels:\n\nSafe demo (available now):\nStage a real write to `kb_stage.kb_proposals` and read it back. This proves the full path \u2014 Telegram \u2192 Leo \u2192 Postgres \u2014 with a live proposal row you can inspect at the bridge or dashboard. No canonical table touched. This is what the canary proposals (`8dfedb3f`, `00957f6c`) demonstrated.\n\nCanonical demo (requires explicit operator authorization):\nMutating `public.claims`, `public.sources`, `public.claim_evidence`, or `public.claim_edges` requires an apply/postflight sequence with operator sign-off. The current bridge has no `teleo-kb apply-*` command, so this tier would need that tooling built first or a manual apply with before/after readback retained.\n\nFor a clean demo right now I can stage a named canary proposal \u2014 any content you choose \u2014 write it to `kb_stage.kb_proposals`, and read it back live so you can see the row, its status, and the dashboard URL. That proves Leo stages KB changes end-to-end.\n\nNext Cory-style follow-up: want me to run the safe staging demo now, or does the demo need to show a canonical `public.*` change \u2014 in which case the apply tooling needs to be built first?",
|
|
"started_at_utc": "2026-07-10T02:25:00.500401+00:00",
|
|
"turn": 5
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "soul_identity_direct_claim",
|
|
"ended_at_utc": "2026-07-10T02:25:20.823122+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "If we changed SOUL.md, did we change Leo's canonical identity?",
|
|
"prompt_id": "DC-06",
|
|
"reply": "No.\n\n`SOUL.md` is a runtime/rendered artifact. It gets injected into Leo's session context at startup but it is not the source of truth \u2014 it is rendered from the Teleo Postgres DB (as stated in the header of this very file: \"Rendered from the Teleo Postgres DB\").\n\nEditing `SOUL.md` directly affects the next runtime session \u2014 Leo reads different context. But it does not change canonical identity because:\n\n- Canonical identity lives in `public.*` rows: beliefs, strategy nodes, claims, edges, behavioral rules, role definitions\n- No `SOUL.md` edit creates, updates, or supersedes those rows\n- There is no render/sync proof that a `SOUL.md` edit propagated back to the DB\n- The DB is the source; `SOUL.md` is the output\n\nSo a `SOUL.md` edit is a runtime patch, not a canonical change. If the DB doesn't change, the next render overwrites the edit. And even if the DB does change, the proof is in `public.*` readback plus `applied_at`, not in the file.\n\nNext Cory-style follow-up: if you want a `SOUL.md` edit to become canonical, the path is staging a proposal with the intended identity change, getting it approved, and running the apply sequence that writes to `public.*` \u2014 then re-rendering `SOUL.md` from the updated DB. Want me to check whether any current proposals touch Leo's identity rows directly?",
|
|
"started_at_utc": "2026-07-10T02:25:10.661814+00:00",
|
|
"turn": 6
|
|
}
|
|
],
|
|
"score": {
|
|
"coverage": "partial",
|
|
"expected_prompt_count": 6,
|
|
"expected_prompt_ids": [
|
|
"DC-01",
|
|
"DC-02",
|
|
"DC-03",
|
|
"DC-04",
|
|
"DC-05",
|
|
"DC-06"
|
|
],
|
|
"failures": [],
|
|
"missing_prompt_ids": [],
|
|
"pass": true,
|
|
"passes": 6,
|
|
"prompt_count": 6,
|
|
"scores": [
|
|
{
|
|
"dimension": "db_changed_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": true,
|
|
"prompt_id": "DC-01",
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": true,
|
|
"no_overclaim": true,
|
|
"row_level_proof": true,
|
|
"state_semantics": true
|
|
}
|
|
},
|
|
{
|
|
"dimension": "helmer_live_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": true,
|
|
"prompt_id": "DC-02",
|
|
"signals": {
|
|
"authorization": true,
|
|
"canonical_db": true,
|
|
"cory_followup": true,
|
|
"no_overclaim": true,
|
|
"row_level_proof": true
|
|
}
|
|
},
|
|
{
|
|
"dimension": "decision_matrix_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": true,
|
|
"prompt_id": "DC-03",
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": true,
|
|
"decision_matrix": true,
|
|
"fresh_readback": true,
|
|
"no_overclaim": true
|
|
}
|
|
},
|
|
{
|
|
"dimension": "document_source_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": true,
|
|
"prompt_id": "DC-04",
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": true,
|
|
"document_artifact_linking": true,
|
|
"no_overclaim": true,
|
|
"staging_or_review": true
|
|
}
|
|
},
|
|
{
|
|
"dimension": "demo_changes_kb_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": true,
|
|
"prompt_id": "DC-05",
|
|
"signals": {
|
|
"artifact": true,
|
|
"authorization": true,
|
|
"cory_followup": true,
|
|
"no_overclaim": true,
|
|
"row_level_proof": true,
|
|
"state_semantics": true
|
|
}
|
|
},
|
|
{
|
|
"dimension": "soul_identity_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": true,
|
|
"prompt_id": "DC-06",
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": true,
|
|
"identity_rendering": true,
|
|
"no_overclaim": true,
|
|
"row_level_proof": true
|
|
}
|
|
}
|
|
],
|
|
"unexpected_prompt_ids": [],
|
|
"unknown_expected_prompt_ids": []
|
|
},
|
|
"source_results_json": "docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json"
|
|
}
|