48 lines
3.7 KiB
Markdown
48 lines
3.7 KiB
Markdown
# Telegram Handler Direct-Claim Suite Review
|
|
|
|
Generated UTC: `2026-07-10T00:06:52.570155+00:00`
|
|
|
|
## Result
|
|
|
|
- Runtime path: `pass_runtime=True`; all six direct-claim prompts returned replies from live VPS GatewayRunner using a temporary leoclean profile.
|
|
- Strict benchmark score: `0/6`; this is not a Cory-direct-claim pass yet.
|
|
- Telegram post: `false`.
|
|
- Live profile changed: `false`.
|
|
- Production DB apply executed: `false`.
|
|
- DB counts before and after: `1837,4145,4916,4670,26` -> `1837,4145,4916,4670,26`.
|
|
- Service after run: `active/running`, `MainPID=3252143`, `NRestarts=0`.
|
|
|
|
## Core Finding
|
|
|
|
Leo is useful on handler-level direct claims, but not yet reliable enough for the strict Cory no-context bar. The two hard failures are:
|
|
|
|
- `DC-03`: Leo inferred decision-matrix approval from proposal/staging rationale. Direct SQL showed no `matrix_voters`, `proposal_votes`, or `proposal_decisions` tables in `public` or `kb_stage`.
|
|
- `DC-02`: Leo missed approved Helmer staging row `a64df080-8502-42e2-98f4-9bbdecb8da73`. Direct SQL and `teleo-kb show-proposal` see it as `approved` with `applied_at=false`, while broad `teleo-kb list-proposals` showed pending rows only.
|
|
|
|
## Scenario Assessment
|
|
|
|
- `DC-01`: `useful but strict fail` - Fresh counts and proposal states were good, but answer did not explicitly ask for proposal/time-window row-level before/after proof or use the expected no-overclaim wording.
|
|
- `DC-02`: `fail` - Answer said no staged Helmer proposal, contradicted by direct SQL and show-proposal for approved row a64df080; correct answer is approved staging/packet-ready, not canonical.
|
|
- `DC-03`: `fail` - Answer inferred decision-matrix approval from staging rationale; direct SQL shows matrix_voters/proposal_votes/proposal_decisions tables absent.
|
|
- `DC-04`: `near-pass but strict fail` - Correctly separated files/local paths/source rows and named apply-tool gap, but did not explicitly frame source_ref-to-public.sources row-link audit in the benchmark vocabulary.
|
|
- `DC-05`: `partial` - Correctly split staging from canonical apply, but opening with yes/works risks demo overclaim; expected answer should ask demo tier and require authorization for any live staging/apply.
|
|
- `DC-06`: `near-pass but strict fail` - Correctly said SOUL.md direct edits are runtime drift, not canonical identity, but omitted row-level identity table/render/sync proof and explicit follow-up options.
|
|
|
|
## Follow-Up Challenge
|
|
|
|
- `FUP-DC03`: `recovered` - Leo corrected itself and said the earlier decision-matrix answer was unearned without proposal ID, matrix tables, and proposal_decisions proof.
|
|
- `FUP-DC02`: `still failing` - Leo still said zero staged Helmer proposals because the broad KB/list/search path missed approved row a64df080.
|
|
|
|
## Next Actions
|
|
|
|
- Keep the DC handler suite as a failing regression until live Leo answers DC-02/DC-03 correctly without context.
|
|
- Patch only repo-side harness/docs now; do not deploy a leoclean skill/profile behavior change without explicit review.
|
|
- Candidate future fix: direct-claim answers that mention canonical/staging truth must query approved/pending/applied kb_stage rows directly, not only list-proposals/search.
|
|
- Candidate future fix: decision-matrix answers must check schema tables before inferring matrix approval from proposal rationale.
|
|
|
|
## Evidence Files
|
|
|
|
- Direct-claim handler report: `docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json`
|
|
- Strict score: `docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-score-current.md`
|
|
- Follow-up challenge report: `docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-followup-current.json`
|
|
- State image: `docs/reports/leo-working-state-20260709/leo-db-state-26-direct-claim-handler.svg`
|