212 lines
12 KiB
Markdown
212 lines
12 KiB
Markdown
# Working Leo: Three-Day Delivery Delta
|
|
|
|
Window: `2026-07-10 00:00` through `2026-07-13` current delivery wave.
|
|
|
|
## Executive Verdict
|
|
|
|
**No, Leo is not yet working to the full m3taversal standard.**
|
|
|
|
Leo on the VPS is now strong on narrow, proof-grounded operations: it can query
|
|
the canonical database, distinguish proposal state from canonical state, return
|
|
structured row/count receipts, survive a gateway restart, stage reviewable
|
|
changes, and exercise guarded apply/composition lifecycles in disposable clones.
|
|
The exact GCP database copy and adapter-free model replay are also proven.
|
|
|
|
The remaining product gap is broad unattended judgment. A fresh 12-question
|
|
blind suite returned every answer without changing the DB or service, but two
|
|
independent strict judges accepted only `1/12` and `2/12` outright. The failures
|
|
included invented current schema, invalid edge types, handler proof described as
|
|
Telegram-visible, temporary memory treated as provenance, and excessive answer
|
|
length. The latest Telegram conversation also exposed participant-name
|
|
hallucination and cross-session identity bleed. This delivery wave adds exact
|
|
`m3taversal` naming, neutral follow-up labels, current-v1 schema guards, and
|
|
regression tests; post-deploy blind and Telegram-visible proof is still required.
|
|
|
|
## Starting Point
|
|
|
|
Three days ago the phrase "Leo is broken" had no single operational meaning.
|
|
We had fragments of evidence, but not a reliable answer to these questions:
|
|
|
|
- Is Leo merely replying, or querying the canonical Postgres KB?
|
|
- Did reviewer approval create canonical rows, or only update
|
|
`kb_stage.kb_proposals`?
|
|
- Can Leo turn a new document/post into linked sources, evidence, claims, and
|
|
edges rather than a flat answer?
|
|
- Does a correction survive session and gateway restart boundaries?
|
|
- Is GCP an exact copy of VPS state, or merely a similar deployment?
|
|
- Can a broad operator question be answered correctly without supplying IDs and
|
|
schema hints?
|
|
|
|
The working lane delivered `28` merged PRs from `#72`, `#73`, and `#75` through
|
|
`#100` (PR `#90` was squash-merged rather than represented by a merge commit).
|
|
That count is delivery history, not proof that all 28 changes are user-visible
|
|
features.
|
|
|
|
## What Changed
|
|
|
|
### July 10: Make KB Truth Executable
|
|
|
|
- Built open-ended and no-context direct-claim benchmarks from real Telegram
|
|
questions.
|
|
- Added fresh VPS preflight, complete DB count/row receipts, overclaim guards,
|
|
and Telegram capture receipts.
|
|
- Proved an intentional gateway restart and a post-restart handler smoke.
|
|
- Added a guarded canonical claim/apply primitive and explicit authorization,
|
|
preflight, postflight, validation, rollback, and cleanup contracts.
|
|
- Shipped the first live-truth VPS/GCP/onboarding skill pack.
|
|
|
|
**Movement:** the standard changed from "the bot replies" to "the answer names
|
|
the actual proposal/canonical state and the one proof-changing next action."
|
|
|
|
### July 11: Prove Composition And Exact GCP Restore
|
|
|
|
- Ran source composition against a disposable full-data VPS clone: new source
|
|
bytes were hash-bound, conflicting atomic claims were extracted, evidence and
|
|
edges were linked, a strict proposal was staged, approval/apply authority was
|
|
separated, and the new graph was rediscovered after a new handler process.
|
|
- Captured an exact canonical Postgres snapshot and restored it to GCP staging.
|
|
- Verified schema, constraints, indexes, functions, types, triggers, views,
|
|
policies, roles, extensions, row counts, row-content hashes, and bounded
|
|
performance checks.
|
|
|
|
**Movement:** database composition and cloud restore stopped being architecture
|
|
claims. They became disposable, repeatable lifecycle proofs with cleanup.
|
|
|
|
### July 12: Harden Readback, Apply, And GCP Replay
|
|
|
|
- Added Cloud SQL-bound operator and full Working Leo benchmark paths.
|
|
- Repaired IAP workflow execution and retained sanitized failure receipts.
|
|
- Grounded direct claims in structured DB readback and required complete,
|
|
packet-specific receipts.
|
|
- Proved the composition/approved-apply lifecycle and corrected the false rule
|
|
that every table count must move on every valid apply.
|
|
- Hardened GCP replay after a nominal `6/6` falsely printed zero canonical
|
|
counts; printed counts must now equal the canonical status receipt.
|
|
- Replayed the real adapter-free GCP GatewayRunner with exact count consistency,
|
|
unchanged fingerprint/service/profile, no send/write, and cleanup of the
|
|
temporary profile.
|
|
|
|
**Movement:** a plausible answer can no longer pass merely because its shape
|
|
looks right. The harness checks the answer against the database receipt.
|
|
|
|
### July 13: Stabilize Deploy Proof And Expose The Broad Gap
|
|
|
|
- Added a pasteable private-password helper/skill without printing or committing
|
|
secrets.
|
|
- Repeated post-deploy VPS direct-claim tests, pinned apply refusal to the exact
|
|
deployed SHA, hardened replay bootstrap, and removed heredoc deadlocks.
|
|
- Confirmed VPS deploy SHA `48777fd984dcfc195e6c33a8d3f7d78bd0c2e344`,
|
|
active gateway PID `1105322`, `NRestarts=0`, and unchanged canonical counts
|
|
`1837/4145/4916/4670/26`.
|
|
- Ran a 12-question blind suite and a four-turn targeted schema correction
|
|
challenge. The correction challenge passed operationally; the blind suite did
|
|
not meet the semantic bar.
|
|
- Proved through GitHub runs `29227390073` and `29227520353` that the expected
|
|
GCP operator/readiness service accounts are absent (`404 Gaia id not found`).
|
|
Reusing the shared identity provider alone cannot repair durable access.
|
|
- Added the exact Telegram participant rule: address `@m3taversal` only as
|
|
`m3taversal`; do not infer or transfer names; keep standard labels neutral.
|
|
|
|
**Movement:** the work now has an honest acceptance frontier. Infrastructure and
|
|
narrow DB truth are green; broad reasoning and participant identity are red
|
|
until post-deploy live tests pass.
|
|
|
|
## Participant-Name Leak Root Cause
|
|
|
|
The unverified personal name came from the instruction and benchmark layer we
|
|
deployed while encoding expected operator behavior. It was not discovered from
|
|
a canonical person/profile row:
|
|
|
|
1. Both active bridge skills explicitly said the compact answer shape existed
|
|
"so Cory gets the expected follow-up" and required every direct answer to end
|
|
with `Next Cory-style follow-up:`.
|
|
2. The shared skill folder was named `working-leo-cory-outcomes` and repeatedly
|
|
described broad questions using that label.
|
|
3. Active benchmark prompts used the same label, reinforcing it in expected
|
|
answer fixtures and handler tests.
|
|
4. The separate shortened value `m3ta` is a real legacy
|
|
`reviewed_by_handle`/proposal attribution value. Leo incorrectly generalized
|
|
that stored row value into a Telegram form of address.
|
|
5. There was no explicit rule binding participant identity to the current
|
|
Telegram update, so a session-derived identity could bleed into another
|
|
participant's reply.
|
|
|
|
The repair changes the deployed VPS/GCP bridge skills, shared operator skill,
|
|
benchmark prompts, output labels, and tests. The only permitted address is the
|
|
exact visible handle `m3taversal`. The stored database value `m3ta` may be quoted
|
|
only when reporting the exact reviewer row, never as a nickname. Historical
|
|
artifact filenames and immutable receipts retain their old identifiers so
|
|
evidence is not rewritten.
|
|
|
|
## Outcome Matrix
|
|
|
|
| Dimension | Three days ago | Current evidence | Current status |
|
|
| --- | --- | --- | --- |
|
|
| Canonical lookup | Mixed memory/file/DB explanations | Complete structured VPS and GCP DB readback | Proven |
|
|
| State semantics | Approval often conflated with apply | `proposed/pending/approved/applied` plus `applied_at` and canonical rows | Proven for bounded questions |
|
|
| Restart survival | Inferred from uptime | Intentional restart, new PID, unchanged counts, successful handler smoke | Proven on VPS |
|
|
| Staging | Vague/manual | Telegram staging and clone staging receipts | Proven at bounded tier |
|
|
| Canonical apply | Packet/SQL discussion | Strict separated review/apply lifecycle with row-level postflight and rollback | Isolated proven; broad production apply not run |
|
|
| Source composition | Files and proposals not clearly connected | Full-data clone source/evidence/claim/edge composition and rediscovery | Clone-proven; arbitrary production breadth open |
|
|
| GCP copy | Similar-looking state | `39` tables, `52,164` rows, zero parity mismatches, private TLS | Proven |
|
|
| GCP model reasoning | Nominal score could hide false counts | Adapter-free `6/6` with exact count equality and unchanged state | Proven for direct-claim replay |
|
|
| GCP on-demand operation | Password/firewall/provider friction | Two missing service accounts proven; exact bootstrap gate known | Not durable; cleanup open |
|
|
| Broad reasoning | Narrow benchmark fixtures | Blind judges: `1/12` strict pass and `2 pass / 4 partial / 6 fail` | Not reliable |
|
|
| Conversation memory | Marker and restart cases | Same-session recall works, but one blind run misused memory as provenance | Partial |
|
|
| Telegram identity | No explicit participant rule | Exact `m3taversal` rule and regression tests added | Awaiting post-deploy visible proof |
|
|
| Identity composition | DB-first direction documented | Current answer contract distinguishes DB rows from rendered `SOUL.md` | Scheduled renderer lifecycle still open |
|
|
|
|
## Why The Work Felt Endless
|
|
|
|
Several proof tiers were previously summarized together. A passing unit suite, a
|
|
no-post GatewayRunner reply, a Telegram-visible reply, a clone apply, a
|
|
production apply, and a GCP parity receipt answer different questions. Repeating
|
|
tests without naming the tier made progress look circular.
|
|
|
|
The current control rule is:
|
|
|
|
1. `Runtime`: did a reply return and did the service remain stable?
|
|
2. `Truth`: did the reply match current schema and canonical rows?
|
|
3. `Delivery`: was it visible in the real Telegram group?
|
|
4. `Mutation`: were exact approved payload rows applied with postflight proof?
|
|
5. `Persistence`: did the result survive a fresh process/restart/render cycle?
|
|
6. `Parity`: does the same path work against the exact GCP copy?
|
|
|
|
A row is green only at the tier named in the outcome matrix.
|
|
|
|
## Current Repair Order
|
|
|
|
1. Merge and auto-deploy the exact participant-name, neutral-label, and current
|
|
schema rules to the VPS.
|
|
2. Verify deploy SHA, gateway restart, unchanged DB counts, and no orphan
|
|
handler/profile resources.
|
|
3. Send a concise naming question and broad out-of-sample questions through the
|
|
authenticated Telegram Chrome session; capture visible replies and check that
|
|
no alias or cross-user identity appears.
|
|
4. Rerun the blind handler suite with schema hallucination and identity scoring.
|
|
5. After privileged GCP reauthentication, recreate least-privilege operator
|
|
identities, prove passwordless status, and delete the retained replay clone
|
|
and run directory.
|
|
|
|
## Evidence
|
|
|
|
- `telegram-handler-blind-oos-audit-current.json`
|
|
- `leo-restart-survival-proof-current.json`
|
|
- `leo-post-deploy-direct-claim-repeat-current.json`
|
|
- `telegram-visible-direct-claim-suite-current.json`
|
|
- `leo-source-composition-clone-checkpoint-current.json`
|
|
- `approve-claim-clone-canary-current.json`
|
|
- `gcp-canonical-parity-live-20260712.json`
|
|
- `gcp-operator-access-blocker-current.json`
|
|
- Root retained copy proof:
|
|
`outputs/gcp-staging-canonical-parity-20260712T1905Z/final-receipt.json`
|
|
- Root retained model replay:
|
|
`outputs/gcp-cory-model-replay-20260712T1940Z/direct-claim-result-final.json`
|
|
|
|
## Completion Rule
|
|
|
|
Do not answer "yes" to the whole m3taversal-standard question until the
|
|
post-deploy Telegram identity test and a fresh broad blind suite pass, with
|
|
unchanged VPS DB/service state; and the GCP durable operator/cleanup lifecycle is
|
|
complete. Narrow VPS/GCP capabilities must continue to be reported as proven at
|
|
their actual tiers rather than being downgraded or rounded up.
|