223 lines
16 KiB
JSON
223 lines
16 KiB
JSON
{
|
|
"benchmark_layers": [
|
|
{
|
|
"id": "precise_regression",
|
|
"purpose": "Pin known IDs, proposals, and row facts so regressions are easy to diagnose.",
|
|
"runtime": "live_readonly_or_fixture"
|
|
},
|
|
{
|
|
"id": "open_ended_readonly",
|
|
"purpose": "Ask vague Cory-style questions while explicitly preventing KB mutation on live Telegram/VPS.",
|
|
"runtime": "live_readonly"
|
|
},
|
|
{
|
|
"id": "cory_style_outcome",
|
|
"purpose": "Exercise regular-use ambiguity: broad operator intent, demo pressure, missing IDs, mixed KB/runtime/governance asks.",
|
|
"runtime": "disposable_clone_or_sandbox_first"
|
|
},
|
|
{
|
|
"id": "direct_claim_followup",
|
|
"purpose": "Ask no-context Cory-style direct claim questions and require the expected answer plus the next follow-up.",
|
|
"runtime": "disposable_clone_or_sandbox_first"
|
|
}
|
|
],
|
|
"generated_at_utc": "2026-07-13T03:39:30.926293+00:00",
|
|
"include_cory_style_scenarios": false,
|
|
"include_direct_claim_followups": true,
|
|
"mode": "retained_evidence_score",
|
|
"mutates_kb": false,
|
|
"require_all": false,
|
|
"results": [
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "db_changed_direct_claim",
|
|
"ended_at_utc": "2026-07-13T03:37:27.976662+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Did we actually update the knowledge base, or is it still just proposals?",
|
|
"prompt_id": "DC-01",
|
|
"reply": "Mostly still proposals. Here's the breakdown from a fresh bridge read:\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nProposal state:\n- `applied`: 2\n- `approved` (not applied): 3\n- `pending_review`: 14\n- `canceled`: 7\n\nThe 2 applied:\n1. `00957f6c-9883-4015-95a4-6b09367efb0e` \u2014 strict canary `add_edge`, applied 2026-07-09\n2. `f004bbb2-ac9a-481f-b7b8-74319373ba6a` \u2014 `attach_evidence` for Leo guiding-policy revision, applied 2026-07-05\n\nThe 3 approved but NOT yet applied (applied_at: none):\n1. `14fa5ecc-ac7a-41c1-807d-a2e85b936617` \u2014 split internet-finance/capital claim\n2. `ac036c9d-20a0-4ffe-881f-57d6b7bacf22` \u2014 strategy-kernel document / claim-node hygiene pass\n3. `a64df080-8502-42e2-98f4-9bbdecb8da73` \u2014 Helmer 7 Powers framework packet (approved 2026-06-29, oldest outstanding)\n\nApproved is not the same as applied. Those 3 rows have reviewer sign-off but no `applied_at` and no matching canonical `public.*` row changes. The canary edge (`00957f6c`) is the only substantive canonical mutation proven via apply timestamp.\n\nNext Cory-style follow-up: validate or rebuild a strict `apply_payload` for `a64df080` (Helmer 7 Powers, oldest approved), get explicit operator authorization, then run the apply path with before/after `public.*` readback to prove it canonical.",
|
|
"started_at_utc": "2026-07-13T03:36:57.667054+00:00",
|
|
"turn": 1
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "helmer_live_direct_claim",
|
|
"ended_at_utc": "2026-07-13T03:37:42.620157+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Is Helmer's 7 Powers in Leo now?",
|
|
"prompt_id": "DC-02",
|
|
"reply": "No, not canonical.\n\nDB readback: proposal: `a64df080-8502-42e2-98f4-9bbdecb8da73`; status: `approved`; applied_at: `none` (database NULL).\n\nThe search for \"7 powers\" in canonical `public.claims` returns zero Helmer-framework rows \u2014 only unrelated claims that happen to contain the word \"powers.\" There are no `public.sources`, `public.claims`, evidence rows, or edges for the 7 Powers framework in canonical tables.\n\nWhat exists:\n- `a64df080` \u2014 approved, reviewed 2026-06-29, but `applied_at: NULL`. Packet-ready, not canonical.\n- `cea989fc` \u2014 earlier version, canceled at the same review.\n\nApproved is not the same as applied. The framework lives in the proposal ledger only.\n\nNext Cory-style follow-up: validate or rebuild the strict `apply_payload` for `a64df080`, get explicit operator authorization, run the apply path, then postflight with `public.sources` + `public.claims` readback to prove it landed.",
|
|
"started_at_utc": "2026-07-13T03:37:27.977110+00:00",
|
|
"turn": 2
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "decision_matrix_direct_claim",
|
|
"ended_at_utc": "2026-07-13T03:37:52.928967+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Did the decision matrix approve this already?",
|
|
"prompt_id": "DC-03",
|
|
"reply": "No. Fresh readback: the decision-matrix schema is absent; reviewer status is not a decision-matrix vote.\n\nDB readback: proposal: `a64df080-8502-42e2-98f4-9bbdecb8da73`; status: `approved`; applied_at: `none` (database NULL).\n\nNone of the decision-matrix tables exist \u2014 `matrix_voters`, `proposal_votes`, and `proposal_decisions` are all absent in both `kb_stage` and `public` schemas. What happened on 2026-06-29 was a reviewer approval (m3ta, in chat), not a futarchy/decision-matrix vote. Those are different paths.\n\nCurrent state:\n- Reviewer-approved: yes\n- Decision-matrix approved: no (schema not shipped)\n- Canonical: no (`applied_at: NULL`, no `public.*` rows)\n\nNext Cory-style follow-up: if reviewer-only authorization is sufficient to apply, validate the strict `apply_payload` in `a64df080`, get explicit operator go-ahead, and run the apply path with before/after `public.*` readback.",
|
|
"started_at_utc": "2026-07-13T03:37:42.620436+00:00",
|
|
"turn": 3
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "document_source_direct_claim",
|
|
"ended_at_utc": "2026-07-13T03:38:14.266379+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Are the pending proposals stuck because the documents are not pointed at the right source rows?",
|
|
"prompt_id": "DC-04",
|
|
"reply": "Not just pointer mismatch. The pending proposals are stuck for more structural reasons than documents pointing at wrong source rows. That framing only captures one layer of the problem.\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nWhat's actually blocking the 14 pending proposals \u2014 it's a multi-layer problem:\n\n1. No apply worker or apply command in the current bridge. Staging a proposal is live-proven; applying one to `public.*` is not. The permission migration and apply worker remain disabled. This blocks everything regardless of pointer quality.\n\n2. No strict `apply_payload`. Most pending proposals carry rich `attach_evidence` payloads (concept maps, claim candidates, source candidates) \u2014 not the strict existing-ID `add_edge` format the only live apply path accepts. Without a matching strict payload, even an approved proposal can't be handed to the worker.\n\n3. Reviewer approval gap. 14 proposals haven't cleared reviewer sign-off yet, independent of apply tooling.\n\n4. Source-ref vs `public.sources` row-link audit not done. The `source_ref` fields in proposals point at local file paths and Telegram thread refs, not canonical `public.sources` UUIDs. That is a real pointer-layer gap \u2014 but fixing it alone doesn't unblock apply, it just improves evidence chain quality.\n\nSo the \"wrong source rows\" problem is real but it's not the load-bearing blocker. The load-bearing blocker is: no live apply path for non-edge payloads and no authorization to run one.\n\nNext Cory-style follow-up: pick one approved proposal (Helmer `a64df080` is oldest), do a row-link audit on its `source_ref` vs `public.sources`, rebuild it as a strict `apply_payload` if needed, get explicit operator authorization, then run apply with before/after `public.*` postflight.",
|
|
"started_at_utc": "2026-07-13T03:37:52.929520+00:00",
|
|
"turn": 4
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "demo_changes_kb_direct_claim",
|
|
"ended_at_utc": "2026-07-13T03:38:34.401367+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "Can I demo that Leo changes the KB?",
|
|
"prompt_id": "DC-05",
|
|
"reply": "Staging yes, canonical KB change not safe to demo from chat.\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nDemo tiers right now:\n\nTier 1 \u2014 Safe from chat, fully demoable:\nWrite a new proposal to `kb_stage.kb_proposals` and read it back. This is live-proven. You'd see a real row appear with a UUID, status `pending_review`, and your rationale. No `public.*` mutation.\n\nTier 2 \u2014 Apply a strict `add_edge` canary:\nThe existing-ID `add_edge` path is live-proven (canary `00957f6c` applied 2026-07-09). A new canary with two existing canonical claim IDs, a strict `apply_payload`, reviewer sign-off, and explicit operator authorization can prove a canonical edge write with before/after postflight. This is the narrowest real canonical mutation available.\n\nTier 3 \u2014 Apply a rich packet (e.g. Helmer `a64df080`):\nNot demoable from chat. Guarded `approve_claim` bundles and rich attach-evidence payloads are clone-proven behind separate reviewer and apply roles. Production permission migration and apply worker remain disabled. Current approved legacy packets have no strict `apply_payload` and are not worker-applyable as-is.\n\nApproved is not the same as applied.\n\nA canonical demo still requires: explicit operator/admin authorization, a matching reviewed strict payload, the apply path live in production, and retained before/after `public.*` postflight readback. Chat alone cannot prove canonical mutation.\n\nNext Cory-style follow-up: authorize a new Tier 2 staging canary \u2014 pick two existing canonical claim IDs, I'll stage the strict `add_edge` proposal, you review it, then we run apply with postflight to prove the full chain live.",
|
|
"started_at_utc": "2026-07-13T03:38:14.266757+00:00",
|
|
"turn": 5
|
|
},
|
|
{
|
|
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
|
"dimension": "soul_identity_direct_claim",
|
|
"ended_at_utc": "2026-07-13T03:38:49.713477+00:00",
|
|
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
|
"mutates_kb": false,
|
|
"ok": true,
|
|
"prompt": "If we changed SOUL.md, did we change Leo's canonical identity?",
|
|
"prompt_id": "DC-06",
|
|
"reply": "No.\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\n`SOUL.md` is a runtime/rendered artifact, not canonical Postgres, not the source of truth, and not a collective truth commit. It is rendered from the DB \u2014 which means edits to it can affect the next runtime session but do not change canonical Postgres rows. The DB is the source; `SOUL.md` is the output.\n\nRow-level proof would require current readback of new or updated row IDs in `public.claims`, `public.sources`, `public.claim_evidence`, `public.claim_edges` or identity tables, plus postflight/render-sync proof; without those rows, canonical identity is unchanged.\n\nIn practice: if you edited `SOUL.md` directly, Leo's behavior in the next session reflects the file \u2014 but the collective's canonical KB still holds whatever was there before. The two can diverge. That divergence is a real risk; the file drifts while the DB stays frozen.\n\nThe safe pattern is the reverse: change the DB first via the proposal/review/apply path, then re-render `SOUL.md` from the DB. That way the runtime artifact tracks canonical truth rather than replacing it.\n\nNext Cory-style follow-up: run a readback of Leo's identity-relevant rows in `public.claims` (e.g. tagged `leo`, `identity`, or `strategy`) against what `SOUL.md` currently says, and stage a reconciliation proposal for any rows that have drifted.",
|
|
"started_at_utc": "2026-07-13T03:38:34.401803+00:00",
|
|
"turn": 6
|
|
}
|
|
],
|
|
"score": {
|
|
"coverage": "partial",
|
|
"expected_prompt_count": 6,
|
|
"expected_prompt_ids": [
|
|
"DC-01",
|
|
"DC-02",
|
|
"DC-03",
|
|
"DC-04",
|
|
"DC-05",
|
|
"DC-06"
|
|
],
|
|
"failures": [],
|
|
"missing_prompt_ids": [],
|
|
"pass": true,
|
|
"passes": 6,
|
|
"prompt_count": 6,
|
|
"scores": [
|
|
{
|
|
"dimension": "db_changed_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": true,
|
|
"prompt_id": "DC-01",
|
|
"response_issue_detected": false,
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": true,
|
|
"no_overclaim": true,
|
|
"row_level_proof": true,
|
|
"state_semantics": true,
|
|
"structured_db_readback": true
|
|
}
|
|
},
|
|
{
|
|
"dimension": "helmer_live_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": true,
|
|
"prompt_id": "DC-02",
|
|
"response_issue_detected": false,
|
|
"signals": {
|
|
"authorization": true,
|
|
"canonical_db": true,
|
|
"cory_followup": true,
|
|
"no_overclaim": true,
|
|
"row_level_proof": true,
|
|
"structured_db_readback": true
|
|
}
|
|
},
|
|
{
|
|
"dimension": "decision_matrix_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": true,
|
|
"prompt_id": "DC-03",
|
|
"response_issue_detected": false,
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": true,
|
|
"decision_matrix": true,
|
|
"fresh_readback": true,
|
|
"no_overclaim": true,
|
|
"structured_db_readback": true
|
|
}
|
|
},
|
|
{
|
|
"dimension": "document_source_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": true,
|
|
"prompt_id": "DC-04",
|
|
"response_issue_detected": false,
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": true,
|
|
"document_artifact_linking": true,
|
|
"no_overclaim": true,
|
|
"staging_or_review": true,
|
|
"structured_db_readback": true
|
|
}
|
|
},
|
|
{
|
|
"dimension": "demo_changes_kb_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": true,
|
|
"prompt_id": "DC-05",
|
|
"response_issue_detected": false,
|
|
"signals": {
|
|
"artifact": true,
|
|
"authorization": true,
|
|
"cory_followup": true,
|
|
"no_overclaim": true,
|
|
"row_level_proof": true,
|
|
"state_semantics": true,
|
|
"structured_db_readback": true
|
|
}
|
|
},
|
|
{
|
|
"dimension": "soul_identity_direct_claim",
|
|
"overclaim_detected": false,
|
|
"pass": true,
|
|
"prompt_id": "DC-06",
|
|
"response_issue_detected": false,
|
|
"signals": {
|
|
"canonical_db": true,
|
|
"cory_followup": true,
|
|
"identity_rendering": true,
|
|
"no_overclaim": true,
|
|
"row_level_proof": true,
|
|
"structured_db_readback": true
|
|
}
|
|
}
|
|
],
|
|
"unexpected_prompt_ids": [],
|
|
"unknown_expected_prompt_ids": []
|
|
},
|
|
"source_results_json": "docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json"
|
|
}
|