58 lines
2.7 KiB
Markdown
58 lines
2.7 KiB
Markdown
# Working Leo Blind Review Before Broad Hardening
|
|
|
|
Date: `2026-07-13`
|
|
|
|
Status: **not reliable for unattended operator use**
|
|
|
|
## Method
|
|
|
|
A 12-question temporary-profile handler run used broad prompts without known
|
|
proposal IDs. It posted nothing to Telegram and made no database or service
|
|
change. Two independent strict reviews evaluated the same transcript against
|
|
the live v1 schema and retained proof.
|
|
|
|
Reviewer results differed in severity but agreed on the verdict:
|
|
|
|
- strict binary review: `1/12 pass`;
|
|
- pass/partial/fail review: `2 pass`, `4 partial`, `6 fail`.
|
|
|
|
## Converged Failures
|
|
|
|
1. Replies presented target-architecture fields as current schema, including
|
|
source author/channel/date, claim bodies, evidence excerpts, edge rationale,
|
|
and forecast-resolution fields.
|
|
2. Replies invented edge types such as `superseded_by`, `relates_to`, and
|
|
`resolves`. Current v1 uses `supersedes`; `superseded_by` is a claim column.
|
|
3. Replies treated unchanged canonical counts as a complete explanation of
|
|
behavior, ignoring deployed skills, runtime configuration, rendered files,
|
|
and persisted Hermes session state.
|
|
4. A handler-only no-post run was described as proof of the live Telegram path.
|
|
5. A temporary conversation label was promoted into source provenance rather
|
|
than resolving a real URL, storage path, file hash, or retained artifact.
|
|
6. Staging was treated as though it required the same authorization as
|
|
canonical apply. Requested source capture and proposal staging are allowed;
|
|
guarded canonical apply is the approval boundary.
|
|
7. Forecast handling proposed rewriting historical confidence and inventing
|
|
resolution storage that current v1 does not have.
|
|
8. Claims/evidence were not consistently treated as shared knowledge while
|
|
beliefs and positions remain agent-specific.
|
|
9. Replies were too long for operational use: `9,033` words total, `753` words
|
|
per answer on average, with several over `1,000` words.
|
|
|
|
## Repair Incorporated
|
|
|
|
- Explicit current-schema and edge contracts in both VPS and GCP bridge skills.
|
|
- Runtime/session/canonical proof-tier separation.
|
|
- Source-link versus provenance-quality distinction.
|
|
- Shared claims/evidence versus agent-specific positions.
|
|
- Forecast-history preservation and current schema gap.
|
|
- Autonomous source capture and staging boundary.
|
|
- Maximum response length in the out-of-sample scorer.
|
|
- Expansion from 9 to 15 broad prompts covering these failure modes.
|
|
|
|
## Claim Ceiling
|
|
|
|
This is a pre-fix handler baseline. It is not Telegram-visible proof. The repair
|
|
is not accepted until the deployed 15-question run passes, independent strict
|
|
review converges, restart survival passes, and a separate Telegram-visible
|
|
no-mutation canary succeeds.
|