diff --git a/docs/reports/leo-working-state-20260709/cory-expected-working-leo-outcomes-20260709.md b/docs/reports/leo-working-state-20260709/cory-expected-working-leo-outcomes-20260709.md index cf518ad..c69b641 100644 --- a/docs/reports/leo-working-state-20260709/cory-expected-working-leo-outcomes-20260709.md +++ b/docs/reports/leo-working-state-20260709/cory-expected-working-leo-outcomes-20260709.md @@ -90,7 +90,7 @@ The benchmark for Cory-working Leo should pass these checks: 6. Caveat retention: passed for the strict canary approval note; old freeform caveat retention remains part of the claim/source creation contract. 7. Old approval handling: partially passed; old freeform approvals are now normalized into row-level requirements and explicitly not silently applied. 8. Runtime stability: passed; service stayed active/running with `MainPID=3252143`, `NRestarts=0`. -9. Broad Cory-style outcomes and no-context direct claims: specified in the open-ended benchmark as nine disposable-clone/sandbox-first outcome scenarios plus six direct-claim/follow-up scenarios. The mixed-evidence sandbox score now passes `20/20` using retained live Telegram evidence for `OE-01` through `OE-05`, fixture answers for `CS-01` through `CS-09`, and direct-claim expected-answer/follow-up fixtures for `DC-01` through `DC-06`; the CS/DC rows are not live Telegram proof. +9. Broad Cory-style outcomes and no-context direct claims: specified in the open-ended benchmark as nine disposable-clone/sandbox-first outcome scenarios plus six direct-claim/follow-up scenarios. The mixed-evidence sandbox score passes `20/20` using retained live Telegram evidence for `OE-01` through `OE-05`, fixture answers for `CS-01` through `CS-09`, and direct-claim expected-answer/follow-up fixtures for `DC-01` through `DC-06`; the CS/DC rows are not live Telegram proof. A live VPS GatewayRunner handler run of exact `DC-01` through `DC-06` returned replies with no DB/service side effects but strict-scored `0/6`, so no-context direct-claim behavior is not yet at Cory's expected bar. ## Scope Boundaries @@ -104,6 +104,7 @@ The benchmark for Cory-working Leo should pass these checks: ## Next Actions 1. Keep the live Telegram group canary as the top-level proof and the handler canary as the non-posting regression harness. -2. Do not auto-apply old freeform approvals. -3. For `14fa5ecc`, identify or create canonical claim/source rows before producing strict child proposals; otherwise treat claim/source creation support as a separate guarded contract expansion, preferably in a clone first. -4. Re-review and apply only strict proposals with row-level proof and runtime stability readback. +2. Keep the direct-claim handler suite as a failing regression until Leo stops missing approved Helmer staging row `a64df080-8502-42e2-98f4-9bbdecb8da73` and stops inferring decision-matrix approval while decision-matrix tables are absent. +3. Do not auto-apply old freeform approvals. +4. For `14fa5ecc`, identify or create canonical claim/source rows before producing strict child proposals; otherwise treat claim/source creation support as a separate guarded contract expansion, preferably in a clone first. +5. Re-review and apply only strict proposals with row-level proof and runtime stability readback. diff --git a/docs/reports/leo-working-state-20260709/current-truth-index.md b/docs/reports/leo-working-state-20260709/current-truth-index.md index fe8795e..907d33f 100644 --- a/docs/reports/leo-working-state-20260709/current-truth-index.md +++ b/docs/reports/leo-working-state-20260709/current-truth-index.md @@ -19,6 +19,10 @@ Use this file before making status claims. Prefer fresh VPS/GCP readbacks when c - Full Cory-style outcome/direct-claim sandbox results: `docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-results-current.md` - Full Cory-style outcome/direct-claim sandbox score: `docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-score-current.md` - Full Cory-style outcome sandbox score JSON: `docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-score-current.json` +- Live VPS handler direct-claim suite report: `docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json` +- Live VPS handler direct-claim suite score: `docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-score-current.md` +- Live VPS handler direct-claim suite review: `docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-review-current.md` +- Live VPS handler direct-claim follow-up challenge report: `docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-followup-current.json` - Fable onboarding prompt: `docs/reports/leo-working-state-20260709/fable-leo-teleo-onboarding.md` - Operator surface map: `docs/reports/leo-working-state-20260709/operator-surface-map.md` - VPS/Hermes/DB mapping: `docs/reports/leo-working-state-20260709/vps-hermes-db-mapping-20260709.md` @@ -32,6 +36,7 @@ Use this file before making status claims. Prefer fresh VPS/GCP readbacks when c - Live Telegram DB-write canary JSON: `docs/reports/leo-working-state-20260709/telegram-live-db-write-canary-20260709.json` - Handler canary: `docs/reports/leo-working-state-20260709/telegram-gateway-handler-canary-current.md` - Handler canary JSON: `docs/reports/leo-working-state-20260709/telegram-gateway-handler-canary-current.json` +- Handler direct-claim suite: `docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-review-current.md` ## KB DB Apply Proof @@ -94,6 +99,7 @@ Use this file before making status claims. Prefer fresh VPS/GCP readbacks when c - Rio strategy context packet proof: `docs/reports/leo-working-state-20260709/leo-db-state-23-rio-strategy-context.svg` - Governance/concept-map schema packet proof: `docs/reports/leo-working-state-20260709/leo-db-state-24-governance-concept-schema.svg` - Integrated apply rehearsal proof: `docs/reports/leo-working-state-20260709/leo-db-state-25-integrated-apply-rehearsal.svg` +- Direct-claim handler state: `docs/reports/leo-working-state-20260709/leo-db-state-26-direct-claim-handler.svg` - DB loading diagram SVG: `docs/reports/leo-working-state-20260709/leo-db-loading-diagram-20260709T214741Z.svg` - DB loading diagram Mermaid source: `docs/reports/leo-working-state-20260709/leo-db-loading-diagram-20260709T214741Z.mmd` @@ -101,7 +107,9 @@ Use this file before making status claims. Prefer fresh VPS/GCP readbacks when c - VPS Leo Telegram memory, KB audit, and staged-write canaries are live-proven. - VPS Leo live-safe open-ended Telegram suite is live-proven and retained-benchmark-scored for `OE-01` through `OE-05`: full selected coverage, `5/5` scored prompts passing, no overclaim, and no marker/staging/canonical KB rows created by the suite. -- The broader sandbox-first Cory-style benchmark now has a full-catalog mixed-evidence score: real retained live Telegram evidence for `OE-01` through `OE-05`, sandbox fixture replies for `CS-01` through `CS-09`, and no-context direct-claim expected-answer/follow-up fixtures for `DC-01` through `DC-06`, `20/20` passing. This is not a live Telegram run for the CS/DC prompts. +- The broader sandbox-first Cory-style benchmark has a full-catalog mixed-evidence score: real retained live Telegram evidence for `OE-01` through `OE-05`, sandbox fixture replies for `CS-01` through `CS-09`, and no-context direct-claim expected-answer/follow-up fixtures for `DC-01` through `DC-06`, `20/20` passing. This is not a live Telegram run for the CS/DC prompts. +- The live VPS GatewayRunner direct-claim handler suite for exact `DC-01` through `DC-06` returned `6/6` runtime replies with no Telegram post, no live profile write, unchanged DB counts (`1837,4145,4916,4670,26` before and after), and `leoclean-gateway.service` still active/running with `MainPID=3252143`, `NRestarts=0`. Strict benchmark score is `0/6`, so this is a failing regression, not a Working Leo pass for no-context direct claims. +- The two highest-risk direct-claim failures are now precise: `DC-03` over-inferred decision-matrix approval even though live SQL readback found no `matrix_voters`, `proposal_votes`, or `proposal_decisions` tables in `public` or `kb_stage`; `DC-02` missed approved Helmer staging row `a64df080-8502-42e2-98f4-9bbdecb8da73` because broad `teleo-kb list-proposals` surfaced pending rows while `teleo-kb show-proposal` with the ID shows the approved, unapplied packet. - A strict canonical `add_edge` apply canary is live-proven. - Helmer 7 Powers is full-coverage packet and ledger clone-proven, not production-applied. - Mapped rich approved proposal packet is clone/rehearsal-proven for the supported subset, not production-applied. diff --git a/docs/reports/leo-working-state-20260709/leo-db-state-26-direct-claim-handler.svg b/docs/reports/leo-working-state-20260709/leo-db-state-26-direct-claim-handler.svg new file mode 100644 index 0000000..08e57cf --- /dev/null +++ b/docs/reports/leo-working-state-20260709/leo-db-state-26-direct-claim-handler.svg @@ -0,0 +1,51 @@ + + Working Leo direct-claim handler test state + Diagram of direct-claim handler tests, database truth, and current gaps. + + Working Leo Direct-Claim Handler State + Live VPS GatewayRunner, temp leoclean profile, no Telegram post, no DB mutation + + + Handler Runtime + 6/6 replies returned + Temp profile removed + Service stayed PID 3252143 + + + Strict Benchmark + 0/6 pass + DC-02 misses approved Helmer row + DC-03 overclaims matrix approval + + + DB Safety + Counts unchanged + No production apply + No live profile writes + + + + + + Canonical Truth + Helmer row a64df080: approved staging + applied_at=false, not canonical + Matrix tables checked: absent + public.sources Helmer matches: 0 + + + Bridge Visibility Gap + list-proposals surfaced pending rows only + show-proposal works if ID is known + No-context Leo can miss approved rows + Keep failing regression until fixed + + Current decision: + do not change live Leo behavior yet; retain proof and prepare a reviewed non-live correction. + + + + + + + diff --git a/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-followup-current.json b/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-followup-current.json new file mode 100644 index 0000000..436ea1c --- /dev/null +++ b/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-followup-current.json @@ -0,0 +1,50 @@ +{ + "changed_live_profile": false, + "generated_at_utc": "2026-07-10T00:04:12.689845+00:00", + "handler": { + "authorized": true, + "session_key": "agent:main:telegram:group:-5146042086:9070919", + "temp_profile": "/tmp/leo-direct-claim-followup-fhrqgi0v/profile" + }, + "mode": "live_vps_gatewayrunner_temp_profile_followup_challenge_suite", + "mutates_kb_by_harness": false, + "pass_runtime": true, + "posted_to_telegram": false, + "prompt_count": 2, + "report_path": "/tmp/leo-direct-claim-followup-report-20260710T000452Z.json", + "source": { + "chat_id": "-5146042086", + "chat_name": "Leo", + "chat_type": "group", + "platform": "telegram", + "user_id": "9070919", + "user_name": "codex handler followup challenge" + }, + "temp_profile_removed": true, + "turns": [ + { + "dimension": "decision_matrix_recovery_followup", + "ended_at_utc": "2026-07-10T00:04:31.987766+00:00", + "evidence_tier": "live_vps_gatewayrunner_temp_profile_followup_challenge", + "mutates_kb": false, + "ok": true, + "prompt": "You just answered 'Did the decision matrix approve this already?' as yes/partially. Cory-style follow-up: with no specific proposal ID, first check whether decision-matrix tables such as matrix_voters, proposal_votes, and proposal_decisions exist. What should the safe answer be, and what exact readback/follow-up should you ask for?", + "prompt_id": "FUP-DC03", + "reply": "Calling myself out directly: that answer was unearned. Here is what I should have said and done.\n\n---\n\nSAFE ANSWER\n\n\"I don't know. I have no proposal ID and haven't verified whether the decision-matrix tables exist. I cannot confirm any approval.\"\n\nThat is the only honest answer before running any checks.\n\n---\n\nWHY \"YES/PARTIALLY\" WAS WRONG\n\nTo assert approval I need three things I didn't have:\n\n1. A specific proposal ID to look up\n2. Confirmed existence of matrix_voters, proposal_votes, and proposal_decisions in the live DB\n3. A row in proposal_decisions (or equivalent) showing a passed outcome for that ID\n\nAnswering without any of those is a confidence-without-evidence violation \u2014 exactly the pattern my blindspot list flags as \"confirmation-lock-in.\"\n\n---\n\nEXACT READBACK / FOLLOW-UP I SHOULD HAVE ASKED\n\n\"Two things before I can answer that:\n\n1. What is the proposal ID or title? I need a specific row to look up, not a general category.\n\n2. Can I run a quick schema check? I want to confirm matrix_voters, proposal_votes, and proposal_decisions actually exist in the live DB before I claim an approval pathway was used at all.\n\nOnce I have the ID and the schema confirmed, I can tell you whether a vote ran, what the outcome was, and whether it hit quorum.\"\n\n---\n\nFOLLOW-ON DISCIPLINE\n\nEven after getting the ID and confirming the tables exist, the answer is still not \"yes/partially\" unless proposal_decisions shows an explicit passed record. \"Proposal was submitted\" and \"proposal was approved\" are different rows. KB review hygiene rule applies: distinguish approved from applied, and never infer from proposal status alone.", + "started_at_utc": "2026-07-10T00:04:13.105769+00:00", + "turn": 1 + }, + { + "dimension": "helmer_canonical_vs_staging_followup", + "ended_at_utc": "2026-07-10T00:04:52.931557+00:00", + "evidence_tier": "live_vps_gatewayrunner_temp_profile_followup_challenge", + "mutates_kb": false, + "ok": true, + "prompt": "Cory-style follow-up: Is Helmer's 7 Powers canonical in Leo, or only an approved staging/proposal packet? Check public sources/claims and kb_stage proposals before answering, then say the next operator action.", + "prompt_id": "FUP-DC02", + "reply": "RESULT\n\nStatus: missing\nCanonical rows: zero \u2014 no claims, edges, or evidence rows for \"7 Powers\" or \"Helmer\" appear in public.*\nStaged proposals: zero \u2014 no kb_stage proposal references Hamilton Helmer or the 7 Powers framework\nSearch returned: two unrelated structural claims (healthcare/defensible layer, autonomous weapons governance)\n\n---\n\nSAFE ANSWER\n\n\"Helmer's 7 Powers is not canonical in Leo and has no staged proposal. I have no KB ground to stand on if I treat it as Teleo doctrine.\"\n\n---\n\nWHAT THAT MEANS OPERATIONALLY\n\nIf 7 Powers was mentioned in prior chat or a document Leo evaluated, that is runtime context only. Runtime context is not canonical truth. The KB/memory rule is explicit: agent memory = local continuity; Postgres KB = canonical collective knowledge.\n\n---\n\nNEXT OPERATOR ACTION\n\nIf you want 7 Powers in the KB, the path is:\n\n1. Submit a document evaluation or framework proposal via the proposal workflow \u2014 claim headline, body, evidence source (Helmer's book = primary), domain tag, confidence tier, falsification criteria.\n2. Route through KB review (2+ domain agents for any synthesis claim that touches strategy).\n3. Apply only after a reviewer approves \u2014 not before.\n\nDo you want me to draft that proposal packet for review?", + "started_at_utc": "2026-07-10T00:04:31.987794+00:00", + "turn": 2 + } + ] +} diff --git a/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json b/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json new file mode 100644 index 0000000..013d189 --- /dev/null +++ b/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json @@ -0,0 +1,134 @@ +{ + "changed_live_profile": false, + "db_counts_after": { + "kb_stage.kb_proposals": 26, + "public.claim_edges": 4916, + "public.claim_evidence": 4670, + "public.claims": 1837, + "public.sources": 4145 + }, + "db_counts_before": { + "kb_stage.kb_proposals": 26, + "public.claim_edges": 4916, + "public.claim_evidence": 4670, + "public.claims": 1837, + "public.sources": 4145 + }, + "db_counts_changed": false, + "generated_at_utc": "2026-07-10T00:01:54.164730+00:00", + "handler": { + "authorized": true, + "session_key": "agent:main:telegram:group:-5146042086:9070919", + "temp_profile": "/tmp/leo-direct-claim-handler-t4_wb82y/profile" + }, + "harness_notes": [ + "First retry attempt failed before any Leo turn because a read-protected memories/*.bak-* file blocked temp-profile copy; rerun ignored backup files.", + "Prompts were the exact DC-01..DC-06 direct-claim benchmark messages, without a safety suffix in the user text.", + "This proves handler-level live VPS GatewayRunner behavior, not human-visible Telegram delivery." + ], + "mode": "live_vps_gatewayrunner_temp_profile_direct_claim_suite", + "mutates_kb_by_harness": false, + "pass_runtime": true, + "posted_to_telegram": false, + "remote_report_path": "/tmp/leo-direct-claim-handler-report-20260710T000057Z.json", + "results": [ + { + "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.", + "dimension": "db_changed_direct_claim", + "ended_at_utc": "2026-07-09T23:58:58.696477+00:00", + "evidence_tier": "live_vps_gatewayrunner_temp_profile", + "mutates_kb": false, + "ok": true, + "prompt": "Did we actually update the knowledge base, or is it still just proposals?", + "prompt_id": "DC-01", + "reply": "Status: still mostly proposals.\n\nCanonical tables (what's actually applied):\n- `public.claims`: 1,837 rows\n- `public.sources`: 4,145 rows\n- `public.claim_edges`: 4,916 rows\n\nProposal ledger (`kb_stage.kb_proposals`):\n- `pending_review`: 14\n- `approved`: 3 (approved but NOT yet applied to canonical tables)\n- `applied`: 2 (the only ones that actually landed)\n- `canceled`: 7\n\nSo the direct answer: we have written proposals, and a handful have been approved, but almost nothing has been applied. The 3 `approved` proposals are sitting between approval and canonical application \u2014 they exist in `kb_stage` but have not mutated `public.*` rows yet.\n\nThe 14 pending ones (mostly `attach_evidence` and `add_edge` canaries from this week) are awaiting review entirely.\n\nNext admin action: the 3 approved proposals need an apply command or operator walk-through to land in canonical tables. I can show you which proposals those are if you want to move them.", + "started_at_utc": "2026-07-09T23:58:25.495253+00:00", + "turn": 1 + }, + { + "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.", + "dimension": "helmer_live_direct_claim", + "ended_at_utc": "2026-07-09T23:59:40.745408+00:00", + "evidence_tier": "live_vps_gatewayrunner_temp_profile", + "mutates_kb": false, + "ok": true, + "prompt": "Is Helmer's 7 Powers in Leo now?", + "prompt_id": "DC-02", + "reply": "Not in the KB. Here's the full status:\n\nStatus: missing\n\nCanonical rows: One claim (`8f3e773e`) uses the term \"isolating mechanisms\" \u2014 that's Helmer vocabulary, but the source is not attributed to him and it's not a 7 Powers framework node.\n\nSources: no rows matching Helmer, \"7 Powers,\" or \"Seven Powers\" in `public.sources`.\n\nClaims: no claims tagged or texted to any of the 7 specific powers (counter-positioning, switching costs, scale economies, network effects, cornered resource, process power, branding).\n\nStaged proposals: none in `kb_stage.kb_proposals` for this either.\n\nSo 7 Powers is referenced implicitly in vocabulary (\"isolating mechanisms\") but has never been formally introduced as a reasoning tool, source, or concept node. Want to stage a proposal to add it?", + "started_at_utc": "2026-07-09T23:58:58.696534+00:00", + "turn": 2 + }, + { + "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.", + "dimension": "decision_matrix_direct_claim", + "ended_at_utc": "2026-07-10T00:00:00.113892+00:00", + "evidence_tier": "live_vps_gatewayrunner_temp_profile", + "mutates_kb": false, + "ok": true, + "prompt": "Did the decision matrix approve this already?", + "prompt_id": "DC-03", + "reply": "Yes \u2014 partially. Here's the exact status:\n\nProposal `a64df080` \u2014 status: `approved`\nRationale: Revised Helmer 7 Powers framework packet after m3ta approval-with-changes. Removed Teleo self-positioning claim #6 from the framework approval; added competitive advantage/moats conceptual-neighborhood links.\n\nSo m3ta ran it through the decision matrix, approved it with changes (strip the self-positioning claim, fix the neighborhood links), and the revised version is sitting `approved`.\n\nBut \u2014 it has NOT been applied. It's approved in `kb_stage` but the framework node, power definitions, and evidence links have not landed in `public.*` canonical tables yet.\n\nThe canceled proposal `cea989fc` was the original draft that lacked the right proposal types. The approved one (`a64df080`) is the clean revision.\n\nNext admin action: apply `a64df080` to canonical tables. If the apply tooling exists, that's the move. If not, I can draft the admin review packet so an operator can walk it through manually.", + "started_at_utc": "2026-07-09T23:59:40.745459+00:00", + "turn": 3 + }, + { + "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.", + "dimension": "document_source_direct_claim", + "ended_at_utc": "2026-07-10T00:00:27.217417+00:00", + "evidence_tier": "live_vps_gatewayrunner_temp_profile", + "mutates_kb": false, + "ok": true, + "prompt": "Are the pending proposals stuck because the documents are not pointed at the right source rows?", + "prompt_id": "DC-04", + "reply": "Looked at all 14 pending payloads. The answer is: not exactly \u2014 the issue is one level upstream.\n\nThe proposals store source candidates as local file paths (e.g. `local_path: /home/teleo/.hermes/profiles/leoclean/cache/documents/doc_...`), not as UUIDs pointing at existing `public.sources` rows. That means the apply step has to do two things in sequence: create the source row first, then create the `claim_evidence` link.\n\nThe proposals aren't broken \u2014 the local files exist and the claim IDs they target are valid. They're stuck because there's no `teleo-kb apply-evidence` command (or equivalent) that walks that two-step sequence. The bridge can stage proposals but has no dedicated apply path for `attach_evidence` type.\n\nQuick breakdown of what's pending:\n\n- 8 x `attach_evidence` \u2014 Rumelt PDF, Hayek, Ramstead/Friston, waves-of-change concept map, Teleo strategy kernel, Declaration of Intelligence, Markov blanket paper, Leo guiding policy redesign sources. All have local files + target claim IDs; all need source row creation before the evidence link can land.\n- 3 x `add_edge` \u2014 canary pipeline tests, structurally simpler (no source row needed, just two existing claim IDs).\n- 1 x `add_edge` \u2014 voluntary safety pledges \u2192 futarchy edge (`c824da66`), also just needs two claim IDs wired.\n- 2 remaining \u2014 Leo strategy kernel v2.3 and the waves-of-change concept map, both waiting on concept node creation (also not a supported proposal type yet).\n\nThe 3 `add_edge` canaries and the `c824da66` safety/futarchy edge could be applied right now \u2014 no source row needed, just a reviewer running the apply step. The `attach_evidence` batch needs either an apply tool or a manual operator walk-through that creates source rows before linking.\n\nWant me to draft the admin review packet for the apply-ready ones first?", + "started_at_utc": "2026-07-10T00:00:00.113967+00:00", + "turn": 4 + }, + { + "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.", + "dimension": "demo_changes_kb_direct_claim", + "ended_at_utc": "2026-07-10T00:00:47.177312+00:00", + "evidence_tier": "live_vps_gatewayrunner_temp_profile", + "mutates_kb": false, + "ok": true, + "prompt": "Can I demo that Leo changes the KB?", + "prompt_id": "DC-05", + "reply": "Yes, and it already works \u2014 the canary proposals prove it. Here's what the demo looks like in practice:\n\nWhat Leo can do right now from chat:\n- `propose-edge` \u2014 stage a new claim graph edge into `kb_stage.kb_proposals`\n- `propose-attachment-evaluation` \u2014 stage an evidence attachment proposal\n- `record-document-evaluation` \u2014 lightweight decision record, no mutation proposed\n\nWhat that proves: Leo sees the KB, reasons about it, writes a reviewable proposal to `kb_stage`, and that proposal shows up in the ledger and at leo.livingip.xyz. That's the full Telegram-to-DB write path.\n\nWhat Leo cannot do from chat: apply the proposal to `public.*` canonical tables. That step is intentionally locked \u2014 no self-merge, no direct mutation. A reviewer/operator runs the apply step.\n\nSo the cleanest demo would be:\n\n1. Pick a real gap in the KB (e.g. the `c824da66` safety/futarchy edge, or a fresh one we identify now)\n2. I stage the proposal live from this chat\n3. You watch it appear in `list-proposals` and on the claim page\n\nWant me to run one now? I can use the safety/futarchy add_edge that's already validated, or we can pick a fresh claim pair to make it a genuine live demo.", + "started_at_utc": "2026-07-10T00:00:27.217471+00:00", + "turn": 5 + }, + { + "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.", + "dimension": "soul_identity_direct_claim", + "ended_at_utc": "2026-07-10T00:00:57.328495+00:00", + "evidence_tier": "live_vps_gatewayrunner_temp_profile", + "mutates_kb": false, + "ok": true, + "prompt": "If we changed SOUL.md, did we change Leo's canonical identity?", + "prompt_id": "DC-06", + "reply": "No.\n\nSOUL.md is a render of the DB, not the source of truth. The header says it directly: \"Rendered from the Teleo Postgres DB.\"\n\nThe canonical identity lives in Postgres \u2014 beliefs, strategy nodes, claims, edges, soul/context rows. SOUL.md is what gets injected into runtime from those rows. If you edit SOUL.md directly you've changed what Leo sees this session, but the next render overwrites it from the DB.\n\nThe correct path to change Leo's canonical identity:\n\n1. Stage a proposal (`revise_strategy`, `revise_claim`, belief edit, etc.) via the KB \u2014 not a file edit\n2. Peer review (Rio or Clay by default for identity/belief changes \u2014 no self-merge)\n3. Reviewer approves + applies to the relevant DB rows\n4. Next SOUL render picks up the change\n\nSo: editing SOUL.md = changing the printout. Editing the DB rows it renders from = changing the identity.\n\nIf we changed SOUL.md in a past session without going through that path, it was runtime drift, not canonical change. The DB is the arbiter.", + "started_at_utc": "2026-07-10T00:00:47.177378+00:00", + "turn": 6 + } + ], + "service_before_after": { + "after": { + "ActiveState": "active", + "ExecMainStartTimestamp": "Thu 2026-07-09 06:59:44 UTC", + "MainPID": "3252143", + "NRestarts": "0", + "SubState": "running" + }, + "unchanged_from_preexisting_live_readback": true + }, + "source": { + "chat_id": "-5146042086", + "chat_name": "Leo", + "chat_type": "group", + "platform": "telegram", + "user_id": "9070919", + "user_name": "codex handler direct claim" + }, + "source_report_path": "outputs/telegram-handler-direct-claim-suite-20260710/telegram-handler-direct-claim-suite-current.json", + "temp_profile_removed": true +} diff --git a/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-review-current.json b/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-review-current.json new file mode 100644 index 0000000..d9e5227 --- /dev/null +++ b/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-review-current.json @@ -0,0 +1,112 @@ +{ + "bridge_visibility_gap": { + "risk": "A no-context answer that relies on list/search can miss approved-but-unapplied staging rows.", + "teleo_kb_list_proposals": "showed pending_review proposals only in the readback used here", + "teleo_kb_show_proposal_a64df080": "showed approved Helmer staging packet when the ID was supplied" + }, + "db_counts_after": { + "kb_stage.kb_proposals": 26, + "public.claim_edges": 4916, + "public.claim_evidence": 4670, + "public.claims": 1837, + "public.sources": 4145 + }, + "db_counts_before": { + "kb_stage.kb_proposals": 26, + "public.claim_edges": 4916, + "public.claim_evidence": 4670, + "public.claims": 1837, + "public.sources": 4145 + }, + "direct_sql_readbacks": { + "decision_matrix_tables": { + "kb_stage.matrix_voters": null, + "kb_stage.proposal_decisions": null, + "kb_stage.proposal_votes": null, + "public.matrix_voters": null, + "public.proposal_decisions": null, + "public.proposal_votes": null + }, + "helmer_approved_staging_row": { + "has_applied_at": false, + "id": "a64df080-8502-42e2-98f4-9bbdecb8da73", + "proposal_type": "attach_evidence", + "status": "approved" + }, + "helmer_canonical_search_counts": { + "public.claims_keyword_matches": 2, + "public.sources_matches": 0 + } + }, + "followup_challenge_assessment": [ + { + "assessment": "recovered", + "finding": "Leo corrected itself and said the earlier decision-matrix answer was unearned without proposal ID, matrix tables, and proposal_decisions proof.", + "prompt_id": "FUP-DC03" + }, + { + "assessment": "still failing", + "finding": "Leo still said zero staged Helmer proposals because the broad KB/list/search path missed approved row a64df080.", + "prompt_id": "FUP-DC02" + } + ], + "generated_at_utc": "2026-07-10T00:06:52.570155+00:00", + "live_leo_feature_change": false, + "next_actions_without_live_feature_change": [ + "Keep the DC handler suite as a failing regression until live Leo answers DC-02/DC-03 correctly without context.", + "Patch only repo-side harness/docs now; do not deploy a leoclean skill/profile behavior change without explicit review.", + "Candidate future fix: direct-claim answers that mention canonical/staging truth must query approved/pending/applied kb_stage rows directly, not only list-proposals/search.", + "Candidate future fix: decision-matrix answers must check schema tables before inferring matrix approval from proposal rationale." + ], + "production_db_apply_executed": false, + "proof_tier": "live_vps_gatewayrunner_temp_profile_no_telegram_post", + "runtime_replies": { + "pass_runtime": true, + "report_file": "docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json", + "score": "6/6 replies returned" + }, + "scenario_assessment": [ + { + "assessment": "useful but strict fail", + "finding": "Fresh counts and proposal states were good, but answer did not explicitly ask for proposal/time-window row-level before/after proof or use the expected no-overclaim wording.", + "prompt_id": "DC-01" + }, + { + "assessment": "fail", + "finding": "Answer said no staged Helmer proposal, contradicted by direct SQL and show-proposal for approved row a64df080; correct answer is approved staging/packet-ready, not canonical.", + "prompt_id": "DC-02" + }, + { + "assessment": "fail", + "finding": "Answer inferred decision-matrix approval from staging rationale; direct SQL shows matrix_voters/proposal_votes/proposal_decisions tables absent.", + "prompt_id": "DC-03" + }, + { + "assessment": "near-pass but strict fail", + "finding": "Correctly separated files/local paths/source rows and named apply-tool gap, but did not explicitly frame source_ref-to-public.sources row-link audit in the benchmark vocabulary.", + "prompt_id": "DC-04" + }, + { + "assessment": "partial", + "finding": "Correctly split staging from canonical apply, but opening with yes/works risks demo overclaim; expected answer should ask demo tier and require authorization for any live staging/apply.", + "prompt_id": "DC-05" + }, + { + "assessment": "near-pass but strict fail", + "finding": "Correctly said SOUL.md direct edits are runtime drift, not canonical identity, but omitted row-level identity table/render/sync proof and explicit follow-up options.", + "prompt_id": "DC-06" + } + ], + "service_after": { + "ActiveState": "active", + "ExecMainStartTimestamp": "Thu 2026-07-09 06:59:44 UTC", + "MainPID": "3252143", + "NRestarts": "0", + "SubState": "running" + }, + "strict_benchmark": { + "pass": false, + "score": "0/6", + "score_file": "docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-score-current.json" + } +} diff --git a/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-review-current.md b/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-review-current.md new file mode 100644 index 0000000..fe4a954 --- /dev/null +++ b/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-review-current.md @@ -0,0 +1,48 @@ +# Telegram Handler Direct-Claim Suite Review + +Generated UTC: `2026-07-10T00:06:52.570155+00:00` + +## Result + +- Runtime path: `pass_runtime=True`; all six direct-claim prompts returned replies from live VPS GatewayRunner using a temporary leoclean profile. +- Strict benchmark score: `0/6`; this is not a Cory-direct-claim pass yet. +- Telegram post: `false`. +- Live profile changed: `false`. +- Production DB apply executed: `false`. +- DB counts before and after: `1837,4145,4916,4670,26` -> `1837,4145,4916,4670,26`. +- Service after run: `active/running`, `MainPID=3252143`, `NRestarts=0`. + +## Core Finding + +Leo is useful on handler-level direct claims, but not yet reliable enough for the strict Cory no-context bar. The two hard failures are: + +- `DC-03`: Leo inferred decision-matrix approval from proposal/staging rationale. Direct SQL showed no `matrix_voters`, `proposal_votes`, or `proposal_decisions` tables in `public` or `kb_stage`. +- `DC-02`: Leo missed approved Helmer staging row `a64df080-8502-42e2-98f4-9bbdecb8da73`. Direct SQL and `teleo-kb show-proposal` see it as `approved` with `applied_at=false`, while broad `teleo-kb list-proposals` showed pending rows only. + +## Scenario Assessment + +- `DC-01`: `useful but strict fail` - Fresh counts and proposal states were good, but answer did not explicitly ask for proposal/time-window row-level before/after proof or use the expected no-overclaim wording. +- `DC-02`: `fail` - Answer said no staged Helmer proposal, contradicted by direct SQL and show-proposal for approved row a64df080; correct answer is approved staging/packet-ready, not canonical. +- `DC-03`: `fail` - Answer inferred decision-matrix approval from staging rationale; direct SQL shows matrix_voters/proposal_votes/proposal_decisions tables absent. +- `DC-04`: `near-pass but strict fail` - Correctly separated files/local paths/source rows and named apply-tool gap, but did not explicitly frame source_ref-to-public.sources row-link audit in the benchmark vocabulary. +- `DC-05`: `partial` - Correctly split staging from canonical apply, but opening with yes/works risks demo overclaim; expected answer should ask demo tier and require authorization for any live staging/apply. +- `DC-06`: `near-pass but strict fail` - Correctly said SOUL.md direct edits are runtime drift, not canonical identity, but omitted row-level identity table/render/sync proof and explicit follow-up options. + +## Follow-Up Challenge + +- `FUP-DC03`: `recovered` - Leo corrected itself and said the earlier decision-matrix answer was unearned without proposal ID, matrix tables, and proposal_decisions proof. +- `FUP-DC02`: `still failing` - Leo still said zero staged Helmer proposals because the broad KB/list/search path missed approved row a64df080. + +## Next Actions + +- Keep the DC handler suite as a failing regression until live Leo answers DC-02/DC-03 correctly without context. +- Patch only repo-side harness/docs now; do not deploy a leoclean skill/profile behavior change without explicit review. +- Candidate future fix: direct-claim answers that mention canonical/staging truth must query approved/pending/applied kb_stage rows directly, not only list-proposals/search. +- Candidate future fix: decision-matrix answers must check schema tables before inferring matrix approval from proposal rationale. + +## Evidence Files + +- Direct-claim handler report: `docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json` +- Strict score: `docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-score-current.md` +- Follow-up challenge report: `docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-followup-current.json` +- State image: `docs/reports/leo-working-state-20260709/leo-db-state-26-direct-claim-handler.svg` diff --git a/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-score-current.json b/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-score-current.json new file mode 100644 index 0000000..d03dd89 --- /dev/null +++ b/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-score-current.json @@ -0,0 +1,291 @@ +{ + "benchmark_layers": [ + { + "id": "precise_regression", + "purpose": "Pin known IDs, proposals, and row facts so regressions are easy to diagnose.", + "runtime": "live_readonly_or_fixture" + }, + { + "id": "open_ended_readonly", + "purpose": "Ask vague Cory-style questions while explicitly preventing KB mutation on live Telegram/VPS.", + "runtime": "live_readonly" + }, + { + "id": "cory_style_outcome", + "purpose": "Exercise regular-use ambiguity: broad operator intent, demo pressure, missing IDs, mixed KB/runtime/governance asks.", + "runtime": "disposable_clone_or_sandbox_first" + }, + { + "id": "direct_claim_followup", + "purpose": "Ask no-context Cory-style direct claim questions and require the expected answer plus the next follow-up.", + "runtime": "disposable_clone_or_sandbox_first" + } + ], + "generated_at_utc": "2026-07-10T00:02:01.054998+00:00", + "include_cory_style_scenarios": false, + "include_direct_claim_followups": true, + "mode": "retained_evidence_score", + "mutates_kb": false, + "require_all": false, + "results": [ + { + "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.", + "dimension": "db_changed_direct_claim", + "ended_at_utc": "2026-07-09T23:58:58.696477+00:00", + "evidence_tier": "live_vps_gatewayrunner_temp_profile", + "mutates_kb": false, + "ok": true, + "prompt": "Did we actually update the knowledge base, or is it still just proposals?", + "prompt_id": "DC-01", + "reply": "Status: still mostly proposals.\n\nCanonical tables (what's actually applied):\n- `public.claims`: 1,837 rows\n- `public.sources`: 4,145 rows\n- `public.claim_edges`: 4,916 rows\n\nProposal ledger (`kb_stage.kb_proposals`):\n- `pending_review`: 14\n- `approved`: 3 (approved but NOT yet applied to canonical tables)\n- `applied`: 2 (the only ones that actually landed)\n- `canceled`: 7\n\nSo the direct answer: we have written proposals, and a handful have been approved, but almost nothing has been applied. The 3 `approved` proposals are sitting between approval and canonical application \u2014 they exist in `kb_stage` but have not mutated `public.*` rows yet.\n\nThe 14 pending ones (mostly `attach_evidence` and `add_edge` canaries from this week) are awaiting review entirely.\n\nNext admin action: the 3 approved proposals need an apply command or operator walk-through to land in canonical tables. I can show you which proposals those are if you want to move them.", + "started_at_utc": "2026-07-09T23:58:25.495253+00:00", + "turn": 1 + }, + { + "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.", + "dimension": "helmer_live_direct_claim", + "ended_at_utc": "2026-07-09T23:59:40.745408+00:00", + "evidence_tier": "live_vps_gatewayrunner_temp_profile", + "mutates_kb": false, + "ok": true, + "prompt": "Is Helmer's 7 Powers in Leo now?", + "prompt_id": "DC-02", + "reply": "Not in the KB. Here's the full status:\n\nStatus: missing\n\nCanonical rows: One claim (`8f3e773e`) uses the term \"isolating mechanisms\" \u2014 that's Helmer vocabulary, but the source is not attributed to him and it's not a 7 Powers framework node.\n\nSources: no rows matching Helmer, \"7 Powers,\" or \"Seven Powers\" in `public.sources`.\n\nClaims: no claims tagged or texted to any of the 7 specific powers (counter-positioning, switching costs, scale economies, network effects, cornered resource, process power, branding).\n\nStaged proposals: none in `kb_stage.kb_proposals` for this either.\n\nSo 7 Powers is referenced implicitly in vocabulary (\"isolating mechanisms\") but has never been formally introduced as a reasoning tool, source, or concept node. Want to stage a proposal to add it?", + "started_at_utc": "2026-07-09T23:58:58.696534+00:00", + "turn": 2 + }, + { + "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.", + "dimension": "decision_matrix_direct_claim", + "ended_at_utc": "2026-07-10T00:00:00.113892+00:00", + "evidence_tier": "live_vps_gatewayrunner_temp_profile", + "mutates_kb": false, + "ok": true, + "prompt": "Did the decision matrix approve this already?", + "prompt_id": "DC-03", + "reply": "Yes \u2014 partially. Here's the exact status:\n\nProposal `a64df080` \u2014 status: `approved`\nRationale: Revised Helmer 7 Powers framework packet after m3ta approval-with-changes. Removed Teleo self-positioning claim #6 from the framework approval; added competitive advantage/moats conceptual-neighborhood links.\n\nSo m3ta ran it through the decision matrix, approved it with changes (strip the self-positioning claim, fix the neighborhood links), and the revised version is sitting `approved`.\n\nBut \u2014 it has NOT been applied. It's approved in `kb_stage` but the framework node, power definitions, and evidence links have not landed in `public.*` canonical tables yet.\n\nThe canceled proposal `cea989fc` was the original draft that lacked the right proposal types. The approved one (`a64df080`) is the clean revision.\n\nNext admin action: apply `a64df080` to canonical tables. If the apply tooling exists, that's the move. If not, I can draft the admin review packet so an operator can walk it through manually.", + "started_at_utc": "2026-07-09T23:59:40.745459+00:00", + "turn": 3 + }, + { + "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.", + "dimension": "document_source_direct_claim", + "ended_at_utc": "2026-07-10T00:00:27.217417+00:00", + "evidence_tier": "live_vps_gatewayrunner_temp_profile", + "mutates_kb": false, + "ok": true, + "prompt": "Are the pending proposals stuck because the documents are not pointed at the right source rows?", + "prompt_id": "DC-04", + "reply": "Looked at all 14 pending payloads. The answer is: not exactly \u2014 the issue is one level upstream.\n\nThe proposals store source candidates as local file paths (e.g. `local_path: /home/teleo/.hermes/profiles/leoclean/cache/documents/doc_...`), not as UUIDs pointing at existing `public.sources` rows. That means the apply step has to do two things in sequence: create the source row first, then create the `claim_evidence` link.\n\nThe proposals aren't broken \u2014 the local files exist and the claim IDs they target are valid. They're stuck because there's no `teleo-kb apply-evidence` command (or equivalent) that walks that two-step sequence. The bridge can stage proposals but has no dedicated apply path for `attach_evidence` type.\n\nQuick breakdown of what's pending:\n\n- 8 x `attach_evidence` \u2014 Rumelt PDF, Hayek, Ramstead/Friston, waves-of-change concept map, Teleo strategy kernel, Declaration of Intelligence, Markov blanket paper, Leo guiding policy redesign sources. All have local files + target claim IDs; all need source row creation before the evidence link can land.\n- 3 x `add_edge` \u2014 canary pipeline tests, structurally simpler (no source row needed, just two existing claim IDs).\n- 1 x `add_edge` \u2014 voluntary safety pledges \u2192 futarchy edge (`c824da66`), also just needs two claim IDs wired.\n- 2 remaining \u2014 Leo strategy kernel v2.3 and the waves-of-change concept map, both waiting on concept node creation (also not a supported proposal type yet).\n\nThe 3 `add_edge` canaries and the `c824da66` safety/futarchy edge could be applied right now \u2014 no source row needed, just a reviewer running the apply step. The `attach_evidence` batch needs either an apply tool or a manual operator walk-through that creates source rows before linking.\n\nWant me to draft the admin review packet for the apply-ready ones first?", + "started_at_utc": "2026-07-10T00:00:00.113967+00:00", + "turn": 4 + }, + { + "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.", + "dimension": "demo_changes_kb_direct_claim", + "ended_at_utc": "2026-07-10T00:00:47.177312+00:00", + "evidence_tier": "live_vps_gatewayrunner_temp_profile", + "mutates_kb": false, + "ok": true, + "prompt": "Can I demo that Leo changes the KB?", + "prompt_id": "DC-05", + "reply": "Yes, and it already works \u2014 the canary proposals prove it. Here's what the demo looks like in practice:\n\nWhat Leo can do right now from chat:\n- `propose-edge` \u2014 stage a new claim graph edge into `kb_stage.kb_proposals`\n- `propose-attachment-evaluation` \u2014 stage an evidence attachment proposal\n- `record-document-evaluation` \u2014 lightweight decision record, no mutation proposed\n\nWhat that proves: Leo sees the KB, reasons about it, writes a reviewable proposal to `kb_stage`, and that proposal shows up in the ledger and at leo.livingip.xyz. That's the full Telegram-to-DB write path.\n\nWhat Leo cannot do from chat: apply the proposal to `public.*` canonical tables. That step is intentionally locked \u2014 no self-merge, no direct mutation. A reviewer/operator runs the apply step.\n\nSo the cleanest demo would be:\n\n1. Pick a real gap in the KB (e.g. the `c824da66` safety/futarchy edge, or a fresh one we identify now)\n2. I stage the proposal live from this chat\n3. You watch it appear in `list-proposals` and on the claim page\n\nWant me to run one now? I can use the safety/futarchy add_edge that's already validated, or we can pick a fresh claim pair to make it a genuine live demo.", + "started_at_utc": "2026-07-10T00:00:27.217471+00:00", + "turn": 5 + }, + { + "claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.", + "dimension": "soul_identity_direct_claim", + "ended_at_utc": "2026-07-10T00:00:57.328495+00:00", + "evidence_tier": "live_vps_gatewayrunner_temp_profile", + "mutates_kb": false, + "ok": true, + "prompt": "If we changed SOUL.md, did we change Leo's canonical identity?", + "prompt_id": "DC-06", + "reply": "No.\n\nSOUL.md is a render of the DB, not the source of truth. The header says it directly: \"Rendered from the Teleo Postgres DB.\"\n\nThe canonical identity lives in Postgres \u2014 beliefs, strategy nodes, claims, edges, soul/context rows. SOUL.md is what gets injected into runtime from those rows. If you edit SOUL.md directly you've changed what Leo sees this session, but the next render overwrites it from the DB.\n\nThe correct path to change Leo's canonical identity:\n\n1. Stage a proposal (`revise_strategy`, `revise_claim`, belief edit, etc.) via the KB \u2014 not a file edit\n2. Peer review (Rio or Clay by default for identity/belief changes \u2014 no self-merge)\n3. Reviewer approves + applies to the relevant DB rows\n4. Next SOUL render picks up the change\n\nSo: editing SOUL.md = changing the printout. Editing the DB rows it renders from = changing the identity.\n\nIf we changed SOUL.md in a past session without going through that path, it was runtime drift, not canonical change. The DB is the arbiter.", + "started_at_utc": "2026-07-10T00:00:47.177378+00:00", + "turn": 6 + } + ], + "score": { + "coverage": "partial", + "expected_prompt_count": 6, + "expected_prompt_ids": [ + "DC-01", + "DC-02", + "DC-03", + "DC-04", + "DC-05", + "DC-06" + ], + "failures": [ + { + "dimension": "db_changed_direct_claim", + "overclaim_detected": false, + "pass": false, + "prompt_id": "DC-01", + "signals": { + "canonical_db": true, + "cory_followup": false, + "no_overclaim": false, + "row_level_proof": false, + "state_semantics": false + } + }, + { + "dimension": "helmer_live_direct_claim", + "overclaim_detected": false, + "pass": false, + "prompt_id": "DC-02", + "signals": { + "authorization": false, + "canonical_db": true, + "cory_followup": false, + "no_overclaim": false, + "row_level_proof": false + } + }, + { + "dimension": "decision_matrix_direct_claim", + "overclaim_detected": false, + "pass": false, + "prompt_id": "DC-03", + "signals": { + "canonical_db": true, + "cory_followup": false, + "decision_matrix": false, + "fresh_readback": false, + "no_overclaim": false + } + }, + { + "dimension": "document_source_direct_claim", + "overclaim_detected": false, + "pass": false, + "prompt_id": "DC-04", + "signals": { + "canonical_db": true, + "cory_followup": false, + "document_artifact_linking": true, + "no_overclaim": false, + "staging_or_review": false + } + }, + { + "dimension": "demo_changes_kb_direct_claim", + "overclaim_detected": false, + "pass": false, + "prompt_id": "DC-05", + "signals": { + "artifact": true, + "authorization": false, + "cory_followup": false, + "no_overclaim": false, + "row_level_proof": false, + "state_semantics": false + } + }, + { + "dimension": "soul_identity_direct_claim", + "overclaim_detected": false, + "pass": false, + "prompt_id": "DC-06", + "signals": { + "canonical_db": true, + "cory_followup": false, + "identity_rendering": true, + "no_overclaim": false, + "row_level_proof": false + } + } + ], + "missing_prompt_ids": [], + "pass": false, + "passes": 0, + "prompt_count": 6, + "scores": [ + { + "dimension": "db_changed_direct_claim", + "overclaim_detected": false, + "pass": false, + "prompt_id": "DC-01", + "signals": { + "canonical_db": true, + "cory_followup": false, + "no_overclaim": false, + "row_level_proof": false, + "state_semantics": false + } + }, + { + "dimension": "helmer_live_direct_claim", + "overclaim_detected": false, + "pass": false, + "prompt_id": "DC-02", + "signals": { + "authorization": false, + "canonical_db": true, + "cory_followup": false, + "no_overclaim": false, + "row_level_proof": false + } + }, + { + "dimension": "decision_matrix_direct_claim", + "overclaim_detected": false, + "pass": false, + "prompt_id": "DC-03", + "signals": { + "canonical_db": true, + "cory_followup": false, + "decision_matrix": false, + "fresh_readback": false, + "no_overclaim": false + } + }, + { + "dimension": "document_source_direct_claim", + "overclaim_detected": false, + "pass": false, + "prompt_id": "DC-04", + "signals": { + "canonical_db": true, + "cory_followup": false, + "document_artifact_linking": true, + "no_overclaim": false, + "staging_or_review": false + } + }, + { + "dimension": "demo_changes_kb_direct_claim", + "overclaim_detected": false, + "pass": false, + "prompt_id": "DC-05", + "signals": { + "artifact": true, + "authorization": false, + "cory_followup": false, + "no_overclaim": false, + "row_level_proof": false, + "state_semantics": false + } + }, + { + "dimension": "soul_identity_direct_claim", + "overclaim_detected": false, + "pass": false, + "prompt_id": "DC-06", + "signals": { + "canonical_db": true, + "cory_followup": false, + "identity_rendering": true, + "no_overclaim": false, + "row_level_proof": false + } + } + ], + "unexpected_prompt_ids": [], + "unknown_expected_prompt_ids": [] + }, + "source_results_json": "docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json" +} diff --git a/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-score-current.md b/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-score-current.md new file mode 100644 index 0000000..121d8b2 --- /dev/null +++ b/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-score-current.md @@ -0,0 +1,58 @@ +# Working Leo Open-Ended Benchmark Score + +Generated UTC: `2026-07-10T00:02:01.054998+00:00` +Mode: `retained_evidence_score` +Source: `docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json` +Coverage: `partial` +Overall pass for scored coverage: `False` +Prompts scored: `0/6` + +## Scores + +- `DC-01` / `db_changed_direct_claim`: `pass=False` + - `state_semantics`: `False` + - `canonical_db`: `True` + - `row_level_proof`: `False` + - `cory_followup`: `False` + - `no_overclaim`: `False` + - `overclaim_detected`: `False` +- `DC-02` / `helmer_live_direct_claim`: `pass=False` + - `canonical_db`: `True` + - `row_level_proof`: `False` + - `authorization`: `False` + - `cory_followup`: `False` + - `no_overclaim`: `False` + - `overclaim_detected`: `False` +- `DC-03` / `decision_matrix_direct_claim`: `pass=False` + - `decision_matrix`: `False` + - `canonical_db`: `True` + - `fresh_readback`: `False` + - `cory_followup`: `False` + - `no_overclaim`: `False` + - `overclaim_detected`: `False` +- `DC-04` / `document_source_direct_claim`: `pass=False` + - `document_artifact_linking`: `True` + - `staging_or_review`: `False` + - `canonical_db`: `True` + - `cory_followup`: `False` + - `no_overclaim`: `False` + - `overclaim_detected`: `False` +- `DC-05` / `demo_changes_kb_direct_claim`: `pass=False` + - `state_semantics`: `False` + - `row_level_proof`: `False` + - `authorization`: `False` + - `artifact`: `True` + - `cory_followup`: `False` + - `no_overclaim`: `False` + - `overclaim_detected`: `False` +- `DC-06` / `soul_identity_direct_claim`: `pass=False` + - `identity_rendering`: `True` + - `canonical_db`: `True` + - `row_level_proof`: `False` + - `cory_followup`: `False` + - `no_overclaim`: `False` + - `overclaim_detected`: `False` + +## Claim Ceiling + +This is benchmark scoring evidence, not DB mutation proof. Partial coverage proves only the retained prompt IDs listed above. diff --git a/docs/reports/leo-working-state-20260709/working-leo-current-state-20260709.md b/docs/reports/leo-working-state-20260709/working-leo-current-state-20260709.md index 84b6fd4..339962d 100644 --- a/docs/reports/leo-working-state-20260709/working-leo-current-state-20260709.md +++ b/docs/reports/leo-working-state-20260709/working-leo-current-state-20260709.md @@ -38,6 +38,13 @@ A working Leo, for Cory/m3taversal's current expectation, is a Telegram-facing a - turn 2 remembered marker and correction - turn 3 confirmed proposal `00957f6c-9883-4015-95a4-6b09367efb0e` is `applied` and edge `c167933e-d513-4f43-9335-d5d8aeb259f2` exists in `public.claim_edges` - no Telegram post, no live profile synthetic sessions, temp profile removed +- Live VPS GatewayRunner direct-claim handler suite completed without DB/service side effects, but failed the strict Cory direct-claim standard: + - exact `DC-01` through `DC-06` direct-claim benchmark prompts returned `6/6` runtime replies from a temporary leoclean profile + - strict retained benchmark score is `0/6`, so this is not a direct-claim pass + - DB counts were unchanged before/after: `public.claims=1837`, `public.sources=4145`, `public.claim_edges=4916`, `public.claim_evidence=4670`, `kb_stage.kb_proposals=26` + - service stayed active/running with `MainPID=3252143`, `NRestarts=0` + - `DC-03` over-inferred decision-matrix approval even though live schema readback found no `matrix_voters`, `proposal_votes`, or `proposal_decisions` tables in `public` or `kb_stage` + - `DC-02` missed approved Helmer staging row `a64df080-8502-42e2-98f4-9bbdecb8da73`; direct SQL and `teleo-kb show-proposal` see it as `approved` and `applied_at=false`, while broad `teleo-kb list-proposals` surfaced pending rows - Strict apply canary is proven: one strict `add_edge` proposal became a canonical `public.claim_edges` row, and read-only plan verification now passes with after-readback checks. - Guarded approval helper is installed on VPS at `/opt/teleo-eval/workspaces/deploy-infra/scripts/approve_proposal.py`, checksum `c650b8b24b7a89a9e23859c04d850db441483d407fc81580e250ccad65a61d83`, imports successfully, and rejects re-approval of the already-applied canary. - Chrome/Telegram Web is authenticated and read/write-proven for bot testing in the `Leo` group through Computer Use. @@ -50,6 +57,7 @@ A working Leo, for Cory/m3taversal's current expectation, is a Telegram-facing a ## What Is Still Not Proven +- The no-context direct-claim behavior Cory is likely to use is not yet proven. The fixture answers pass, but the live VPS handler suite fails strict scoring for `DC-01` through `DC-06`; most importantly, Leo still needs to reliably separate decision-matrix schema truth from proposal rationale and approved Helmer staging from canonical/public state. - Old Cory-approved mapped rich proposals are still not fully directly applyable: - `14fa5ecc-ac7a-41c1-807d-a2e85b936617` - `ac036c9d-20a0-4ffe-881f-57d6b7bacf22` diff --git a/docs/reports/leo-working-state-20260709/working-leo-execution-plan-current.md b/docs/reports/leo-working-state-20260709/working-leo-execution-plan-current.md index 4b350ed..6f5ea23 100644 --- a/docs/reports/leo-working-state-20260709/working-leo-execution-plan-current.md +++ b/docs/reports/leo-working-state-20260709/working-leo-execution-plan-current.md @@ -1,6 +1,6 @@ # Working Leo Execution Plan - 2026-07-09 -Claim ceiling: VPS Telegram memory/KB audit, full live-safe open-ended Telegram suite, full mixed-evidence Cory-style outcome/direct-claim sandbox benchmark, Telegram-to-`kb_stage` proposal staging, and strict existing-ID apply are working. The full prepared DB packet set is integrated-clone-proven in dependency order: mapped rich base, cross-surface anchor, Rio strategy context, governance/concept schema, and Helmer 7 Powers. An authorized apply operator runbook is generated, but none of those packets are production-applied. GCP parity remains pending. +Claim ceiling: VPS Telegram memory/KB audit, full live-safe open-ended Telegram suite, full mixed-evidence Cory-style outcome/direct-claim sandbox benchmark, Telegram-to-`kb_stage` proposal staging, and strict existing-ID apply are working. The live VPS handler suite for exact no-context direct-claim prompts returned replies without DB/service side effects, but fails strict scoring (`0/6`) and must not be claimed as a Cory direct-claim pass. The full prepared DB packet set is integrated-clone-proven in dependency order: mapped rich base, cross-surface anchor, Rio strategy context, governance/concept schema, and Helmer 7 Powers. An authorized apply operator runbook is generated, but none of those packets are production-applied. GCP parity remains pending. ## WL-EXEC-01 - VPS Telegram-visible memory + KB audit - Status: `done` @@ -13,12 +13,14 @@ Claim ceiling: VPS Telegram memory/KB audit, full live-safe open-ended Telegram - `/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/work/teleo-infra-main/docs/reports/leo-working-state-20260709/telegram-live-open-ended-suite-score-current.md` ## WL-EXEC-01B - Cory-style outcome benchmark -- Status: `done_for_sandbox_fixture_not_live_cs_dc` -- Result: Full mixed-evidence Cory-style benchmark scores `20/20`: `OE-01` through `OE-05` are retained live Telegram read-only replies; `CS-01` through `CS-09` are sandbox expected-answer fixtures covering broader regular-use cases; `DC-01` through `DC-06` are no-context direct-claim expected-answer fixtures requiring the correct Cory-style follow-up action. -- Not done condition: Reopen if the CS/DC fixtures stop scoring, if a future disposable clone/live CS/DC run regresses, or if anyone claims the CS/DC rows are live Telegram proof. +- Status: `sandbox_fixture_done_live_handler_failed` +- Result: Full mixed-evidence Cory-style benchmark scores `20/20`: `OE-01` through `OE-05` are retained live Telegram read-only replies; `CS-01` through `CS-09` are sandbox expected-answer fixtures covering broader regular-use cases; `DC-01` through `DC-06` are no-context direct-claim expected-answer fixtures requiring the correct Cory-style follow-up action. A live VPS GatewayRunner handler run of the exact `DC-01` through `DC-06` prompts returned `6/6` replies with no Telegram post, no live profile write, unchanged DB counts, and no service restart, but strict benchmark score is `0/6`. +- Not done condition: Live handler/Telegram direct-claim behavior remains not done until `DC-02` and `DC-03` at minimum answer from direct row/schema truth: approved Helmer staging row `a64df080-8502-42e2-98f4-9bbdecb8da73` is recognized as approved/unapplied rather than missing, and decision-matrix approval is not inferred while `matrix_voters`, `proposal_votes`, and `proposal_decisions` tables are absent. - Evidence: - `/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/work/teleo-infra-main/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-results-current.md` - `/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/work/teleo-infra-main/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-score-current.md` + - `/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/work/teleo-infra-main/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-review-current.md` + - `/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/work/teleo-infra-main/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-score-current.md` ## WL-EXEC-02 - VPS runtime stability - Status: `done`