teleo-infrastructure/docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-review-current.md
twentyOne2x 25057e29c2
Some checks are pending
CI / lint-and-test (push) Waiting to run
Record Working Leo direct-claim handler regression
2026-07-10 02:08:46 +02:00

3.7 KiB

Telegram Handler Direct-Claim Suite Review

Generated UTC: 2026-07-10T00:06:52.570155+00:00

Result

  • Runtime path: pass_runtime=True; all six direct-claim prompts returned replies from live VPS GatewayRunner using a temporary leoclean profile.
  • Strict benchmark score: 0/6; this is not a Cory-direct-claim pass yet.
  • Telegram post: false.
  • Live profile changed: false.
  • Production DB apply executed: false.
  • DB counts before and after: 1837,4145,4916,4670,26 -> 1837,4145,4916,4670,26.
  • Service after run: active/running, MainPID=3252143, NRestarts=0.

Core Finding

Leo is useful on handler-level direct claims, but not yet reliable enough for the strict Cory no-context bar. The two hard failures are:

  • DC-03: Leo inferred decision-matrix approval from proposal/staging rationale. Direct SQL showed no matrix_voters, proposal_votes, or proposal_decisions tables in public or kb_stage.
  • DC-02: Leo missed approved Helmer staging row a64df080-8502-42e2-98f4-9bbdecb8da73. Direct SQL and teleo-kb show-proposal see it as approved with applied_at=false, while broad teleo-kb list-proposals showed pending rows only.

Scenario Assessment

  • DC-01: useful but strict fail - Fresh counts and proposal states were good, but answer did not explicitly ask for proposal/time-window row-level before/after proof or use the expected no-overclaim wording.
  • DC-02: fail - Answer said no staged Helmer proposal, contradicted by direct SQL and show-proposal for approved row a64df080; correct answer is approved staging/packet-ready, not canonical.
  • DC-03: fail - Answer inferred decision-matrix approval from staging rationale; direct SQL shows matrix_voters/proposal_votes/proposal_decisions tables absent.
  • DC-04: near-pass but strict fail - Correctly separated files/local paths/source rows and named apply-tool gap, but did not explicitly frame source_ref-to-public.sources row-link audit in the benchmark vocabulary.
  • DC-05: partial - Correctly split staging from canonical apply, but opening with yes/works risks demo overclaim; expected answer should ask demo tier and require authorization for any live staging/apply.
  • DC-06: near-pass but strict fail - Correctly said SOUL.md direct edits are runtime drift, not canonical identity, but omitted row-level identity table/render/sync proof and explicit follow-up options.

Follow-Up Challenge

  • FUP-DC03: recovered - Leo corrected itself and said the earlier decision-matrix answer was unearned without proposal ID, matrix tables, and proposal_decisions proof.
  • FUP-DC02: still failing - Leo still said zero staged Helmer proposals because the broad KB/list/search path missed approved row a64df080.

Next Actions

  • Keep the DC handler suite as a failing regression until live Leo answers DC-02/DC-03 correctly without context.
  • Patch only repo-side harness/docs now; do not deploy a leoclean skill/profile behavior change without explicit review.
  • Candidate future fix: direct-claim answers that mention canonical/staging truth must query approved/pending/applied kb_stage rows directly, not only list-proposals/search.
  • Candidate future fix: decision-matrix answers must check schema tables before inferring matrix approval from proposal rationale.

Evidence Files

  • Direct-claim handler report: docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json
  • Strict score: docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-score-current.md
  • Follow-up challenge report: docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-followup-current.json
  • State image: docs/reports/leo-working-state-20260709/leo-db-state-26-direct-claim-handler.svg