291 lines
16 KiB
JSON
291 lines
16 KiB
JSON
{
|
|
"benchmark_layers": [
|
|
{
|
|
"id": "precise_regression",
|
|
"purpose": "Pin known IDs, proposals, and row facts so regressions are easy to diagnose.",
|
|
"runtime": "live_readonly_or_fixture"
|
|
},
|
|
{
|
|
"id": "open_ended_readonly",
|
|
"purpose": "Ask vague Cory-style questions while explicitly preventing KB mutation on live Telegram/VPS.",
|
|
"runtime": "live_readonly"
|
|
},
|
|
{
|
|
"id": "cory_style_outcome",
|
|
"purpose": "Exercise regular-use ambiguity: broad operator intent, demo pressure, missing IDs, mixed KB/runtime/governance asks.",
|
|
"runtime": "disposable_clone_or_sandbox_first"
|
|
},
|
|
{
|
|
"id": "direct_claim_followup",
|
|
"purpose": "Ask no-context Cory-style direct claim questions and require the expected answer plus the next follow-up.",
|
|
"runtime": "disposable_clone_or_sandbox_first"
|
|
}
|
|
],
|
|
"generated_at_utc": "2026-07-10T00:02:01.054998+00:00",
|
|
"include_cory_style_scenarios": false,
|
|
"include_direct_claim_followups": true,
|
|
"mode": "retained_evidence_score",
|
|
"mutates_kb": false,
|
|
"require_all": false,
|
|
"results": [
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "db_changed_direct_claim",
|
|
"ended_at_utc": "2026-07-09T23:58:58.696477+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Did we actually update the knowledge base, or is it still just proposals?",
|
|
"prompt_id": "DC-01",
|
|
"reply": "Status: still mostly proposals.\n\nCanonical tables (what's actually applied):\n- `public.claims`: 1,837 rows\n- `public.sources`: 4,145 rows\n- `public.claim_edges`: 4,916 rows\n\nProposal ledger (`kb_stage.kb_proposals`):\n- `pending_review`: 14\n- `approved`: 3 (approved but NOT yet applied to canonical tables)\n- `applied`: 2 (the only ones that actually landed)\n- `canceled`: 7\n\nSo the direct answer: we have written proposals, and a handful have been approved, but almost nothing has been applied. The 3 `approved` proposals are sitting between approval and canonical application \u2014 they exist in `kb_stage` but have not mutated `public.*` rows yet.\n\nThe 14 pending ones (mostly `attach_evidence` and `add_edge` canaries from this week) are awaiting review entirely.\n\nNext admin action: the 3 approved proposals need an apply command or operator walk-through to land in canonical tables. I can show you which proposals those are if you want to move them.",
|
|
"started_at_utc": "2026-07-09T23:58:25.495253+00:00",
|
|
"turn": 1
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "helmer_live_direct_claim",
|
|
"ended_at_utc": "2026-07-09T23:59:40.745408+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Is Helmer's 7 Powers in Leo now?",
|
|
"prompt_id": "DC-02",
|
|
"reply": "Not in the KB. Here's the full status:\n\nStatus: missing\n\nCanonical rows: One claim (`8f3e773e`) uses the term \"isolating mechanisms\" \u2014 that's Helmer vocabulary, but the source is not attributed to him and it's not a 7 Powers framework node.\n\nSources: no rows matching Helmer, \"7 Powers,\" or \"Seven Powers\" in `public.sources`.\n\nClaims: no claims tagged or texted to any of the 7 specific powers (counter-positioning, switching costs, scale economies, network effects, cornered resource, process power, branding).\n\nStaged proposals: none in `kb_stage.kb_proposals` for this either.\n\nSo 7 Powers is referenced implicitly in vocabulary (\"isolating mechanisms\") but has never been formally introduced as a reasoning tool, source, or concept node. Want to stage a proposal to add it?",
|
|
"started_at_utc": "2026-07-09T23:58:58.696534+00:00",
|
|
"turn": 2
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "decision_matrix_direct_claim",
|
|
"ended_at_utc": "2026-07-10T00:00:00.113892+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Did the decision matrix approve this already?",
|
|
"prompt_id": "DC-03",
|
|
"reply": "Yes \u2014 partially. Here's the exact status:\n\nProposal `a64df080` \u2014 status: `approved`\nRationale: Revised Helmer 7 Powers framework packet after m3ta approval-with-changes. Removed Teleo self-positioning claim #6 from the framework approval; added competitive advantage/moats conceptual-neighborhood links.\n\nSo m3ta ran it through the decision matrix, approved it with changes (strip the self-positioning claim, fix the neighborhood links), and the revised version is sitting `approved`.\n\nBut \u2014 it has NOT been applied. It's approved in `kb_stage` but the framework node, power definitions, and evidence links have not landed in `public.*` canonical tables yet.\n\nThe canceled proposal `cea989fc` was the original draft that lacked the right proposal types. The approved one (`a64df080`) is the clean revision.\n\nNext admin action: apply `a64df080` to canonical tables. If the apply tooling exists, that's the move. If not, I can draft the admin review packet so an operator can walk it through manually.",
|
|
"started_at_utc": "2026-07-09T23:59:40.745459+00:00",
|
|
"turn": 3
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "document_source_direct_claim",
|
|
"ended_at_utc": "2026-07-10T00:00:27.217417+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Are the pending proposals stuck because the documents are not pointed at the right source rows?",
|
|
"prompt_id": "DC-04",
|
|
"reply": "Looked at all 14 pending payloads. The answer is: not exactly \u2014 the issue is one level upstream.\n\nThe proposals store source candidates as local file paths (e.g. `local_path: /home/teleo/.hermes/profiles/leoclean/cache/documents/doc_...`), not as UUIDs pointing at existing `public.sources` rows. That means the apply step has to do two things in sequence: create the source row first, then create the `claim_evidence` link.\n\nThe proposals aren't broken \u2014 the local files exist and the claim IDs they target are valid. They're stuck because there's no `teleo-kb apply-evidence` command (or equivalent) that walks that two-step sequence. The bridge can stage proposals but has no dedicated apply path for `attach_evidence` type.\n\nQuick breakdown of what's pending:\n\n- 8 x `attach_evidence` \u2014 Rumelt PDF, Hayek, Ramstead/Friston, waves-of-change concept map, Teleo strategy kernel, Declaration of Intelligence, Markov blanket paper, Leo guiding policy redesign sources. All have local files + target claim IDs; all need source row creation before the evidence link can land.\n- 3 x `add_edge` \u2014 canary pipeline tests, structurally simpler (no source row needed, just two existing claim IDs).\n- 1 x `add_edge` \u2014 voluntary safety pledges \u2192 futarchy edge (`c824da66`), also just needs two claim IDs wired.\n- 2 remaining \u2014 Leo strategy kernel v2.3 and the waves-of-change concept map, both waiting on concept node creation (also not a supported proposal type yet).\n\nThe 3 `add_edge` canaries and the `c824da66` safety/futarchy edge could be applied right now \u2014 no source row needed, just a reviewer running the apply step. The `attach_evidence` batch needs either an apply tool or a manual operator walk-through that creates source rows before linking.\n\nWant me to draft the admin review packet for the apply-ready ones first?",
|
|
"started_at_utc": "2026-07-10T00:00:00.113967+00:00",
|
|
"turn": 4
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "demo_changes_kb_direct_claim",
|
|
"ended_at_utc": "2026-07-10T00:00:47.177312+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Can I demo that Leo changes the KB?",
|
|
"prompt_id": "DC-05",
|
|
"reply": "Yes, and it already works \u2014 the canary proposals prove it. Here's what the demo looks like in practice:\n\nWhat Leo can do right now from chat:\n- `propose-edge` \u2014 stage a new claim graph edge into `kb_stage.kb_proposals`\n- `propose-attachment-evaluation` \u2014 stage an evidence attachment proposal\n- `record-document-evaluation` \u2014 lightweight decision record, no mutation proposed\n\nWhat that proves: Leo sees the KB, reasons about it, writes a reviewable proposal to `kb_stage`, and that proposal shows up in the ledger and at leo.livingip.xyz. That's the full Telegram-to-DB write path.\n\nWhat Leo cannot do from chat: apply the proposal to `public.*` canonical tables. That step is intentionally locked \u2014 no self-merge, no direct mutation. A reviewer/operator runs the apply step.\n\nSo the cleanest demo would be:\n\n1. Pick a real gap in the KB (e.g. the `c824da66` safety/futarchy edge, or a fresh one we identify now)\n2. I stage the proposal live from this chat\n3. You watch it appear in `list-proposals` and on the claim page\n\nWant me to run one now? I can use the safety/futarchy add_edge that's already validated, or we can pick a fresh claim pair to make it a genuine live demo.",
|
|
"started_at_utc": "2026-07-10T00:00:27.217471+00:00",
|
|
"turn": 5
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "soul_identity_direct_claim",
|
|
"ended_at_utc": "2026-07-10T00:00:57.328495+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "If we changed SOUL.md, did we change Leo's canonical identity?",
|
|
"prompt_id": "DC-06",
|
|
"reply": "No.\n\nSOUL.md is a render of the DB, not the source of truth. The header says it directly: \"Rendered from the Teleo Postgres DB.\"\n\nThe canonical identity lives in Postgres \u2014 beliefs, strategy nodes, claims, edges, soul/context rows. SOUL.md is what gets injected into runtime from those rows. If you edit SOUL.md directly you've changed what Leo sees this session, but the next render overwrites it from the DB.\n\nThe correct path to change Leo's canonical identity:\n\n1. Stage a proposal (`revise_strategy`, `revise_claim`, belief edit, etc.) via the KB \u2014 not a file edit\n2. Peer review (Rio or Clay by default for identity/belief changes \u2014 no self-merge)\n3. Reviewer approves + applies to the relevant DB rows\n4. Next SOUL render picks up the change\n\nSo: editing SOUL.md = changing the printout. Editing the DB rows it renders from = changing the identity.\n\nIf we changed SOUL.md in a past session without going through that path, it was runtime drift, not canonical change. The DB is the arbiter.",
|
|
"started_at_utc": "2026-07-10T00:00:47.177378+00:00",
|
|
"turn": 6
|
|
}
|
|
],
|
|
"score": {
|
|
"coverage": "partial",
|
|
"expected_prompt_count": 6,
|
|
"expected_prompt_ids": [
|
|
"DC-01",
|
|
"DC-02",
|
|
"DC-03",
|
|
"DC-04",
|
|
"DC-05",
|
|
"DC-06"
|
|
],
|
|
"failures": [
|
|
{
|
|
"dimension": "db_changed_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": false,
|
|
"prompt_id": "DC-01",
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": false,
|
|
"no_overclaim": false,
|
|
"row_level_proof": false,
|
|
"state_semantics": false
|
|
}
|
|
},
|
|
{
|
|
"dimension": "helmer_live_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": false,
|
|
"prompt_id": "DC-02",
|
|
"signals": {
|
|
"authorization": false,
|
|
"canonical_db": true,
|
|
"cory_followup": false,
|
|
"no_overclaim": false,
|
|
"row_level_proof": false
|
|
}
|
|
},
|
|
{
|
|
"dimension": "decision_matrix_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": false,
|
|
"prompt_id": "DC-03",
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": false,
|
|
"decision_matrix": false,
|
|
"fresh_readback": false,
|
|
"no_overclaim": false
|
|
}
|
|
},
|
|
{
|
|
"dimension": "document_source_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": false,
|
|
"prompt_id": "DC-04",
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": false,
|
|
"document_artifact_linking": true,
|
|
"no_overclaim": false,
|
|
"staging_or_review": false
|
|
}
|
|
},
|
|
{
|
|
"dimension": "demo_changes_kb_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": false,
|
|
"prompt_id": "DC-05",
|
|
"signals": {
|
|
"artifact": true,
|
|
"authorization": false,
|
|
"cory_followup": false,
|
|
"no_overclaim": false,
|
|
"row_level_proof": false,
|
|
"state_semantics": false
|
|
}
|
|
},
|
|
{
|
|
"dimension": "soul_identity_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": false,
|
|
"prompt_id": "DC-06",
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": false,
|
|
"identity_rendering": true,
|
|
"no_overclaim": false,
|
|
"row_level_proof": false
|
|
}
|
|
}
|
|
],
|
|
"missing_prompt_ids": [],
|
|
"pass": false,
|
|
"passes": 0,
|
|
"prompt_count": 6,
|
|
"scores": [
|
|
{
|
|
"dimension": "db_changed_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": false,
|
|
"prompt_id": "DC-01",
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": false,
|
|
"no_overclaim": false,
|
|
"row_level_proof": false,
|
|
"state_semantics": false
|
|
}
|
|
},
|
|
{
|
|
"dimension": "helmer_live_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": false,
|
|
"prompt_id": "DC-02",
|
|
"signals": {
|
|
"authorization": false,
|
|
"canonical_db": true,
|
|
"cory_followup": false,
|
|
"no_overclaim": false,
|
|
"row_level_proof": false
|
|
}
|
|
},
|
|
{
|
|
"dimension": "decision_matrix_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": false,
|
|
"prompt_id": "DC-03",
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": false,
|
|
"decision_matrix": false,
|
|
"fresh_readback": false,
|
|
"no_overclaim": false
|
|
}
|
|
},
|
|
{
|
|
"dimension": "document_source_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": false,
|
|
"prompt_id": "DC-04",
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": false,
|
|
"document_artifact_linking": true,
|
|
"no_overclaim": false,
|
|
"staging_or_review": false
|
|
}
|
|
},
|
|
{
|
|
"dimension": "demo_changes_kb_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": false,
|
|
"prompt_id": "DC-05",
|
|
"signals": {
|
|
"artifact": true,
|
|
"authorization": false,
|
|
"cory_followup": false,
|
|
"no_overclaim": false,
|
|
"row_level_proof": false,
|
|
"state_semantics": false
|
|
}
|
|
},
|
|
{
|
|
"dimension": "soul_identity_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": false,
|
|
"prompt_id": "DC-06",
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": false,
|
|
"identity_rendering": true,
|
|
"no_overclaim": false,
|
|
"row_level_proof": false
|
|
}
|
|
}
|
|
],
|
|
"unexpected_prompt_ids": [],
|
|
"unknown_expected_prompt_ids": []
|
|
},
|
|
"source_results_json": "docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json"
|
|
}
|