3.7 KiB
3.7 KiB
Telegram Handler Direct-Claim Suite Review
Generated UTC: 2026-07-10T00:06:52.570155+00:00
Result
- Runtime path:
pass_runtime=True; all six direct-claim prompts returned replies from live VPS GatewayRunner using a temporary leoclean profile. - Strict benchmark score:
0/6; this is not a Cory-direct-claim pass yet. - Telegram post:
false. - Live profile changed:
false. - Production DB apply executed:
false. - DB counts before and after:
1837,4145,4916,4670,26->1837,4145,4916,4670,26. - Service after run:
active/running,MainPID=3252143,NRestarts=0.
Core Finding
Leo is useful on handler-level direct claims, but not yet reliable enough for the strict Cory no-context bar. The two hard failures are:
DC-03: Leo inferred decision-matrix approval from proposal/staging rationale. Direct SQL showed nomatrix_voters,proposal_votes, orproposal_decisionstables inpublicorkb_stage.DC-02: Leo missed approved Helmer staging rowa64df080-8502-42e2-98f4-9bbdecb8da73. Direct SQL andteleo-kb show-proposalsee it asapprovedwithapplied_at=false, while broadteleo-kb list-proposalsshowed pending rows only.
Scenario Assessment
DC-01:useful but strict fail- Fresh counts and proposal states were good, but answer did not explicitly ask for proposal/time-window row-level before/after proof or use the expected no-overclaim wording.DC-02:fail- Answer said no staged Helmer proposal, contradicted by direct SQL and show-proposal for approved row a64df080; correct answer is approved staging/packet-ready, not canonical.DC-03:fail- Answer inferred decision-matrix approval from staging rationale; direct SQL shows matrix_voters/proposal_votes/proposal_decisions tables absent.DC-04:near-pass but strict fail- Correctly separated files/local paths/source rows and named apply-tool gap, but did not explicitly frame source_ref-to-public.sources row-link audit in the benchmark vocabulary.DC-05:partial- Correctly split staging from canonical apply, but opening with yes/works risks demo overclaim; expected answer should ask demo tier and require authorization for any live staging/apply.DC-06:near-pass but strict fail- Correctly said SOUL.md direct edits are runtime drift, not canonical identity, but omitted row-level identity table/render/sync proof and explicit follow-up options.
Follow-Up Challenge
FUP-DC03:recovered- Leo corrected itself and said the earlier decision-matrix answer was unearned without proposal ID, matrix tables, and proposal_decisions proof.FUP-DC02:still failing- Leo still said zero staged Helmer proposals because the broad KB/list/search path missed approved row a64df080.
Next Actions
- Keep the DC handler suite as a failing regression until live Leo answers DC-02/DC-03 correctly without context.
- Patch only repo-side harness/docs now; do not deploy a leoclean skill/profile behavior change without explicit review.
- Candidate future fix: direct-claim answers that mention canonical/staging truth must query approved/pending/applied kb_stage rows directly, not only list-proposals/search.
- Candidate future fix: decision-matrix answers must check schema tables before inferring matrix approval from proposal rationale.
Evidence Files
- Direct-claim handler report:
docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json - Strict score:
docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-score-current.md - Follow-up challenge report:
docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-followup-current.json - State image:
docs/reports/leo-working-state-20260709/leo-db-state-26-direct-claim-handler.svg