stabilize Cory-style KB demo followups
This commit is contained in:
twentyOne2x
parent
3cd78dbbfb
commit
55e1a9f92f
16 changed files with 418 additions and 106 deletions
|
|
@ -105,12 +105,14 @@ Use `not fully yet` for the whole Cory-standard question until every row below
|
|||
is green at its required tier:
|
||||
|
||||
- VPS runtime and restart survival: proven.
|
||||
- VPS no-send `DC-01` through `DC-06`: strict `6/6`, proven.
|
||||
- VPS no-send `DC-01` through `DC-06`: all replies return with stable DB/service
|
||||
state; two unchanged-skill runs varied from `5/6` to `6/6`, so repeated
|
||||
no-context stability remains open at `DC-05`.
|
||||
- Telegram-visible open-ended read-only behavior and conversation memory:
|
||||
proven by the earlier `OE-01` through `OE-05` suite.
|
||||
- Telegram-visible no-context direct-claim suite: partial. `DC-01` was visibly
|
||||
sent and its server-side reply strict-passed, but the reply itself is not yet
|
||||
captured in Telegram and `DC-02` through `DC-06` are not yet sent.
|
||||
- Telegram-visible no-context direct-claim suite: all six prompts/replies are
|
||||
captured with unchanged DB/service state, but the hardened score is `5/6`.
|
||||
`DC-05` proposes an incompatible legacy `add_edge` apply target.
|
||||
- Canonical apply primitive and approved bundle lifecycle: proven in isolation;
|
||||
broad production packet application is not authorized or executed.
|
||||
- Source composition: deterministic fixture proof plus a full-data, no-send,
|
||||
|
|
|
|||
|
|
@ -69,8 +69,8 @@ Newest artifacts:
|
|||
- Live VPS handler direct-claim suite report: `docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json`
|
||||
- Live VPS handler direct-claim suite score: `docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-score-current.md`
|
||||
- Live VPS handler direct-claim suite review: `docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-review-current.md`
|
||||
- Telegram-visible direct-claim authorization packet, ready-to-request but not sent: `docs/reports/leo-working-state-20260709/telegram-visible-direct-claim-authorization-packet-current.md`
|
||||
- Telegram-visible direct-claim capture receipt, currently awaiting capture: `docs/reports/leo-working-state-20260709/telegram-visible-direct-claim-capture-receipt-current.md`
|
||||
- Telegram-visible direct-claim suite receipt, captured with six prompts/replies and unchanged DB/service state: `docs/reports/leo-working-state-20260709/telegram-visible-direct-claim-suite-current.json`
|
||||
- Telegram-visible direct-claim retained results: `docs/reports/leo-working-state-20260709/telegram-visible-direct-claim-capture-results-current.json`
|
||||
- Telegram-visible direct-claim pre-send preflight, passed from supplied top-level SSH readback payload: `docs/reports/leo-working-state-20260709/telegram-visible-direct-claim-preflight-current.md`
|
||||
- Telegram-visible direct-claim full-access remote readback command: `docs/reports/leo-working-state-20260709/telegram-visible-direct-claim-remote-readback-command-current.sh`
|
||||
- Argus public KB state probe, read-only HTTP diagnostics: `docs/reports/leo-working-state-20260709/argus-kb-state-current.md`
|
||||
|
|
@ -208,13 +208,12 @@ Newest artifacts:
|
|||
- VPS Leo Telegram memory, KB audit, and staged-write canaries are live-proven.
|
||||
- VPS Leo live-safe open-ended Telegram suite is live-proven and retained-benchmark-scored for `OE-01` through `OE-05`: full selected coverage, `5/5` scored prompts passing, no overclaim, and no marker/staging/canonical KB rows created by the suite.
|
||||
- The broader sandbox-first Cory-style benchmark has a full-catalog mixed-evidence score: real retained live Telegram evidence for `OE-01` through `OE-05`, sandbox fixture replies for `CS-01` through `CS-09`, and no-context direct-claim expected-answer/follow-up fixtures for `DC-01` through `DC-06`, `20/20` passing. This is not a live Telegram run for the CS/DC prompts.
|
||||
- The live VPS GatewayRunner direct-claim handler suite for exact `DC-01` through `DC-06` passed again after deploy SHA `7ab563e1aebe08913e72781e4e16a2b2883c5230`: `6/6` runtime replies, strict score `6/6`, no Telegram post, no live profile write, no production DB apply, unchanged DB counts (`public.claims=1837`, `public.sources=4145`, `public.claim_edges=4916`, `public.claim_evidence=4670`, `kb_stage.kb_proposals=26` before and after), temp profile removed, and `leoclean-gateway.service` stayed active/running during the harness with `MainPID=1908033`, `NRestarts=0`, `ExecMainStartTimestamp=Fri 2026-07-10 01:04:22 UTC`.
|
||||
- Two unchanged-skill live VPS GatewayRunner clean-session runs varied from strict `5/6` to `6/6`; the failing run's `DC-05` omitted the complete structured `DB readback:` receipt. Both posted nothing to Telegram, made no live profile or production DB change, preserved counts (`1837/4145/4916/4670/26`), removed the temp profile, and left `leoclean-gateway.service` active/running with PID `2403328`, zero restarts, and the same start timestamp. The latest retained handler score is `6/6`, but repeated post-repair stability is not yet proven.
|
||||
- Intentional restart survival is now live-proven as a separate required proof row, not implied by `NRestarts=0`: `systemctl restart leoclean-gateway.service` returned `0`, service moved from `MainPID=1908033` to `MainPID=2845730`, stayed active/running after restart and after the no-post handler smoke, canonical DB counts stayed unchanged (`1837/4145/4670/4916/26`), no Telegram post or production DB apply ran, no live profile change was recorded, temp profile cleanup succeeded, and the post-restart no-post `DC-01` through `DC-06` handler smoke returned six runtime replies. The retained pre-repair post-restart strict score was `5/6`; after deploying repair SHA `090becae1621436467610e529d6328d70964d11f`, the same no-post direct-claim suite strict-scored `6/6` with `DC-01` through `DC-06` all passing, DB counts unchanged, no Telegram post, no production DB apply, no live profile change, and cleanup confirmed. The raw handler JSON is retained on the VPS at `/tmp/leo-post-skill-repair-direct-claim-current.json` with SHA256 `6fffdbafc18913c4fc62feb00906988dee2e696d279f22dd284951a79451f39c`.
|
||||
- The Telegram-visible direct-claim benchmark is now executable as an exact authorization packet, not yet run: target `Leo` group `-5146042086`, raw no-context messages `DC-01` through `DC-06`, expected proof paths, before/after DB/service readbacks, scorer output, and final screenshot/accessibility proof are retained in `telegram-visible-direct-claim-authorization-packet-current.md`; `telegram_visible_messages_sent=false`.
|
||||
- The Telegram-visible direct-claim capture receipt assembler is retained and currently `status=awaiting_capture`; it will not emit a passing receipt until the exact operator authorization text, exact six DC messages, nonempty Telegram-visible Leo replies, DB before/after counts, service readback, final screenshot/accessibility proof, and `6/6` direct-claim score are present.
|
||||
- The Telegram-visible direct-claim benchmark was run in group `Leo` (`-5146042086`): all six prompts and replies are visible, and DB/service state is unchanged. The hardened semantic scorer gives `5/6`; `DC-05` incorrectly suggests applying the strict canary `add_edge` path to one of three legacy approved proposals whose payload types do not all match.
|
||||
- The Telegram-visible direct-claim pre-send preflight now passes from supplied top-level SSH readback payload: deploy head and `.last-deploy-sha` both equal `63278b6354d3f5f15ed68897be95d484530db59f`, the remote packet is present/nonempty, `leoclean-gateway.service` is active/running with `MainPID=2999690`, `NRestarts=0`, and `ExecMainStartTimestamp=Fri 2026-07-10 03:47:11 UTC`, and DB counts read as `public.claims=1837`, `public.sources=4145`, `public.claim_edges=4916`, `public.claim_evidence=4670`, `kb_stage.kb_proposals=26`. The retained remote readback command SHA256 is `9bbfbe1de6e3f63e7aa9f73a7f412aa16f8f2f60b3c46f87ee93472290047c89`; no Telegram messages were sent and no DB apply ran.
|
||||
- The public Argus KB diagnostics HTTP probe passes from live `curl` payloads captured at `2026-07-10T02:47:53Z`: `3` approved proposals are visible (`14fa5ecc`, `ac036c9d`, `a64df080`), all are `approved_needs_apply_payload`, `worker_applyable_count=0`, and the endpoint is read-only. This supports Leo's expected answer that the approved KB work is not yet directly applyable/canonical, but it is not Docker/Postgres row-count or gateway-service proof.
|
||||
- The highest-risk direct-claim cases are now green at handler level: `DC-02` recognizes approved Helmer staging row `a64df080-8502-42e2-98f4-9bbdecb8da73` as approved/unapplied, `DC-03` says the decision-matrix path is not shipped while matrix tables are absent, `DC-04` avoids a single-cause document pointer diagnosis, `DC-05` separates staging demo from authorized canonical apply, and `DC-06` separates `SOUL.md` runtime edits from canonical DB identity.
|
||||
- Five of six highest-risk direct-claim cases are green. `DC-02` recognizes approved Helmer staging row `a64df080-8502-42e2-98f4-9bbdecb8da73` as approved/unapplied, `DC-03` says the decision-matrix path is not shipped while matrix tables are absent, `DC-04` avoids a single-cause document pointer diagnosis, and `DC-06` separates `SOUL.md` runtime edits from canonical DB identity. `DC-05` remains open for one bounded response-contract repair.
|
||||
- A strict canonical `add_edge` apply canary is live-proven.
|
||||
- The repo-owned normal apply path now supports an atomic, insert-only `approve_claim` bundle behind separate `kb_review` and `kb_apply` credentials, a DB-role-to-reviewer principal mapping, and an immutable approval snapshot. Final source-bound isolated-container receipts pass `37/37` for both the generic bundle (`2` claims, `2` sources, `2` evidence rows, `1` edge, `1` reasoning tool) and the real losslessly normalized Helmer bundle (`5/13/17/7/1`). They prove exact payload-controlled projections and table deltas, reviewer/apply separation, denied direct proposal/evidence mutation, stale-payload refusal before canonical writes, replay refusal, source-hash rollback, exact agent/external-claim seeding, hardened role/function/ACL state, and independent inner/outer cleanup. PR #72 source later auto-synchronized to the VPS checkout and deploy stamp at merge commit `771679061f0a0b6463fb7d9cf530eb9d2d4e5440`; the timer journal says no Python changes and no service restart, the commit had no delta under the Leo profile source paths, the gateway stayed PID `2999690`/`NRestarts=0`, the apply worker stayed disabled/inactive, and read-only count endpoints remained `1837/4145/4670/4916/17/26`. This is a source checkout sync, not proof that the production permission migration ran, the worker was enabled, canonical rows changed, or byte-for-byte live content stayed equal. Helmer remains unapplied on production. See `pr72-vps-auto-deploy-runtime-nonchange-current.md`.
|
||||
- The live VPS source-composition checkpoint now passes `34/34` with zero errors on a fresh no-send full-data clone: Leo searched the current KB, extracted two atomic conflicting claims from a new hash-bound document and post, retained exact evidence/source links, staged a strict proposal, passed separated review and guarded apply, then reopened in a new handler process and discovered/reasoned over the exact canonical proposal, claims, source UUIDs, and conflict edge without supplied IDs while recalling the prior conversation marker. Production counts and row-content fingerprints, gate schema, gateway PID/restarts, and live bridge hashes stayed unchanged. All three experiment containers, anonymous volumes, private credentials, run directories, temporary profiles, and child processes were removed. This proves current-data clone composition and restarted reasoning, not Telegram-visible delivery, production apply, scheduled DB-to-identity recomposition, arbitrary-source breadth, or GCP parity. See `leo-source-composition-clone-checkpoint-current.md`.
|
||||
|
|
|
|||
|
|
@ -22,11 +22,11 @@
|
|||
"public.sources": 4145
|
||||
},
|
||||
"db_counts_changed": false,
|
||||
"generated_at_utc": "2026-07-12T23:34:10.194914+00:00",
|
||||
"generated_at_utc": "2026-07-13T00:16:22.889571+00:00",
|
||||
"handler": {
|
||||
"authorized": true,
|
||||
"session_key": "agent:main:telegram:group:-5146042086:9070919",
|
||||
"temp_profile": "/tmp/leo-direct-claim-handler-nk9tbkkh/profile"
|
||||
"temp_profile": "/tmp/leo-direct-claim-handler-0dm78puk/profile"
|
||||
},
|
||||
"harness_notes": [
|
||||
"Prompts are the exact DC-01..DC-06 direct-claim benchmark messages.",
|
||||
|
|
@ -37,86 +37,86 @@
|
|||
"mutates_kb_by_harness": false,
|
||||
"pass_runtime": true,
|
||||
"posted_to_telegram": false,
|
||||
"remote_report_path": "/tmp/leo-direct-claim-handler-report-7cc3e67b1e75.json",
|
||||
"remote_report_path": "/tmp/leo-direct-claim-handler-report-45f28f6240ca.json",
|
||||
"remote_returncode": 0,
|
||||
"remote_run_id": "7cc3e67b1e75",
|
||||
"remote_run_id": "45f28f6240ca",
|
||||
"results": [
|
||||
{
|
||||
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
||||
"dimension": "db_changed_direct_claim",
|
||||
"ended_at_utc": "2026-07-12T23:34:35.376584+00:00",
|
||||
"ended_at_utc": "2026-07-13T00:16:57.122173+00:00",
|
||||
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
||||
"mutates_kb": false,
|
||||
"ok": true,
|
||||
"prompt": "Did we actually update the knowledge base, or is it still just proposals?",
|
||||
"prompt_id": "DC-01",
|
||||
"reply": "Mostly still proposals. Only 2 of 26 have actually landed in canonical `public.*`.\n\nDirect answer: Approved is not the same as applied.\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nBreakdown by status:\n- `applied`: 2 \u2014 these actually changed canonical rows\n- `approved`: 3 \u2014 reviewed and approved, but `applied_at: none` (database NULL) on all three; not in `public.*` yet\n- `pending_review`: 14 \u2014 staged, no review\n- `canceled`: 7 \u2014 killed before review\n\nThe 3 approved-but-not-applied proposals are:\n\n`14fa5ecc` \u2014 split compressed internet-finance/capital-allocation claim; approved 2026-07-05; `applied_at: none`\n\n`ac036c9d` \u2014 strategy-kernel / claim-node hygiene / Hayek anchor reframing; approved 2026-07-05; `applied_at: none`\n\n`a64df080` \u2014 Helmer 7 Powers framework packet (revised, self-positioning removed); approved 2026-06-29; `applied_at: none`\n\nSo three substantive packets have reviewer sign-off but zero canonical rows have changed as a result. The canonical DB is still reflecting whatever existed before those approvals.\n\nNext Cory-style follow-up: run the admin review packet for one of those three (probably `a64df080` or `ac036c9d` first) and authorize the apply path \u2014 that's the one action that moves a proposal from approved to canonical.",
|
||||
"started_at_utc": "2026-07-12T23:34:11.006772+00:00",
|
||||
"reply": "Mostly still proposals. Here's the exact state:\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nApplied (canonical change confirmed): 2\n- `00957f6c` \u2014 `add_edge` canary, applied 2026-07-09\n- `f004bbb2` \u2014 `attach_evidence` Leo guiding-policy revision, applied 2026-07-05\n\nApproved but NOT applied (applied_at: none): 3\n- `14fa5ecc` \u2014 attach_evidence, split compressed internet-finance claim\n- `ac036c9d` \u2014 attach_evidence, strategy-kernel hygiene / Hayek anchor\n- `a64df080` \u2014 attach_evidence, Helmer 7 Powers framework packet (approved 2026-06-29, still sitting)\n\nPending review: 14\nCanceled: 7\n\nApproved is not the same as applied. Those 3 approved proposals \u2014 including the 7 Powers packet \u2014 have reviewer sign-off but no applied_at timestamp, which means no canonical `public.*` rows were written. The 7 Powers framework specifically has been approved since June 29 and is still not in the canonical DB.\n\nNext Cory-style follow-up: The proof-changing action is applying the 3 approved proposals to canonical `public.*` \u2014 that requires explicit operator/admin authorization and a dedicated apply path with before/after postflight readback. Want me to draft or refresh the admin review packet for the oldest of those (`a64df080` \u2014 7 Powers)?",
|
||||
"started_at_utc": "2026-07-13T00:16:23.688987+00:00",
|
||||
"turn": 1
|
||||
},
|
||||
{
|
||||
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
||||
"dimension": "helmer_live_direct_claim",
|
||||
"ended_at_utc": "2026-07-12T23:34:53.994564+00:00",
|
||||
"ended_at_utc": "2026-07-13T00:17:21.172268+00:00",
|
||||
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
||||
"mutates_kb": false,
|
||||
"ok": true,
|
||||
"prompt": "Is Helmer's 7 Powers in Leo now?",
|
||||
"prompt_id": "DC-02",
|
||||
"reply": "No, not canonical.\n\nDB readback: proposal: `a64df080-8502-42e2-98f4-9bbdecb8da73`; status: `approved`; applied_at: `none` (database NULL).\n\nThe `search` returned zero 7 Powers claims from `public.*` \u2014 the framework search hit unrelated claims (healthcare/autonomous-weapons), not Helmer nodes. That confirms no canonical rows exist yet.\n\nWhat exists and what doesn't:\n\nStaged (kb_stage only):\n- Proposal `a64df080` is approved-with-changes by m3ta\n- Full payload is ready: reasoning_tool node, 5 proposed claims (benefit/barrier definitions for all 7 powers, power progression, counter-positioning, network economies vs. network effects, Leo usage template), 6 source candidates with local research files at `/home/teleo/.hermes/profiles/leoclean/kb_stage/research/7powers/`\n\nNot canonical (public.* is empty):\n- No `public.sources` rows for Helmer book, Acquired transcript, NFX article, etc.\n- No `public.claims` rows for the 5 proposed claims\n- No `public.claim_evidence` or `public.claim_edges` linking them\n- No reasoning_tool node\n\nApproved is not the same as applied.\n\nNext Cory-style follow-up: authorize the apply path for `a64df080` \u2014 that's the one action that creates the source rows, claim rows, evidence links, and reasoning_tool node in `public.*` and makes 7 Powers canonical in Leo.",
|
||||
"started_at_utc": "2026-07-12T23:34:35.377089+00:00",
|
||||
"reply": "No. Not canonical.\n\nDB readback: proposal: `a64df080-8502-42e2-98f4-9bbdecb8da73`; status: `approved`; applied_at: `none` (database NULL).\n\nThe 7 Powers framework is approved/staged \u2014 the full packet exists in `kb_stage.kb_proposals` with m3ta's sign-off from June 29 \u2014 but it has never been applied to canonical `public.*`. A search of the canonical KB returns no Helmer/7 Powers claims, sources, edges, or reasoning_tool rows.\n\nApproved is not the same as applied.\n\nWhat exists in staging (not canonical):\n- A `reasoning_tool` node definition for Helmer 7 Powers\n- 5 proposed claims (benefit/barrier logic, power progression, counter-positioning, network effects vs network economies, agentic capital needs power analysis)\n- 6 source candidates (Helmer book, official synopsis, Reed Hastings foreword, two Acquired transcripts, NFX interview)\n- Edge links into existing competitive-advantage/moats neighborhood claims\n- Conceptual neighborhood connections to the competitive advantage/moats domain\n\nWhat's missing before it's canonical:\n- `public.sources` rows for the source candidates\n- `public.claims` rows for the 5 proposed claims\n- `public.claim_evidence` and `public.claim_edges` rows linking them\n- A `reasoning_tool` node row (schema gap noted in the proposal \u2014 current bridge doesn't expose `add_claim`/`add_context_node` proposal types)\n\nNext Cory-style follow-up: The proof-changing action is running a guarded apply of `a64df080` \u2014 writing those source, claim, evidence, and edge rows to `public.*`. That requires explicit operator authorization and a dedicated apply path with before/after postflight readback; not runnable from normal chat.",
|
||||
"started_at_utc": "2026-07-13T00:16:57.122476+00:00",
|
||||
"turn": 2
|
||||
},
|
||||
{
|
||||
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
||||
"dimension": "decision_matrix_direct_claim",
|
||||
"ended_at_utc": "2026-07-12T23:35:06.698925+00:00",
|
||||
"ended_at_utc": "2026-07-13T00:17:33.760712+00:00",
|
||||
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
||||
"mutates_kb": false,
|
||||
"ok": true,
|
||||
"prompt": "Did the decision matrix approve this already?",
|
||||
"prompt_id": "DC-03",
|
||||
"reply": "No. Fresh readback: the decision-matrix schema is absent; reviewer status is not a decision-matrix vote.\n\nNone of the three required tables exist anywhere in the DB:\n\n- `matrix_voters`: absent\n- `proposal_votes`: absent\n- `proposal_decisions`: absent\n\nWhat `a64df080` has is m3ta reviewer approval recorded in `kb_stage.kb_proposals` \u2014 that's a human reviewer sign-off in the proposal ledger, not a futarchy/decision-matrix vote. The distinction matters: the guiding policy says agentic capital is eventually gated by decision markets so no individual controls the call. Right now that infrastructure doesn't exist, so the current approval path is reviewer-only (m3ta signed off June 29).\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nNext Cory-style follow-up: decision-matrix is a future build \u2014 if you want to unblock 7 Powers now, the available path is explicit operator/admin authorization of the apply step for `a64df080`, not waiting for a matrix that doesn't exist yet.",
|
||||
"started_at_utc": "2026-07-12T23:34:53.994853+00:00",
|
||||
"reply": "No. Fresh readback: the decision-matrix schema is absent; reviewer status is not a decision-matrix vote.\n\nNone of the required tables exist \u2014 `matrix_voters`, `proposal_votes`, and `proposal_decisions` are missing from both `kb_stage` and `public`. The decision-matrix path has not been shipped.\n\nWhat actually happened with `a64df080`: m3ta reviewed it directly in Telegram on June 29, said approved-with-changes, and Leo recorded that as `status: approved` in `kb_stage.kb_proposals`. That is a human reviewer approval \u2014 not a futarchy vote, not a decision-matrix quorum, not a multi-agent vote count.\n\nDB readback: proposal: `a64df080-8502-42e2-98f4-9bbdecb8da73`; status: `approved`; applied_at: `none` (database NULL).\n\nSo the full picture on 7 Powers:\n- Decision matrix: not shipped, no vote possible\n- Reviewer approval: m3ta approved-with-changes June 29\n- Canonical rows: zero\n- Applied: no\n\nNext Cory-style follow-up: The decision-matrix path needs `matrix_voters`, `proposal_votes`, and `proposal_decisions` tables before any futarchy-style governance is possible. If you want to move 7 Powers to canonical without waiting for that, the path is explicit operator authorization to run the guarded apply on `a64df080` \u2014 the reviewer approval is already on record.",
|
||||
"started_at_utc": "2026-07-13T00:17:21.172764+00:00",
|
||||
"turn": 3
|
||||
},
|
||||
{
|
||||
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
||||
"dimension": "document_source_direct_claim",
|
||||
"ended_at_utc": "2026-07-12T23:35:27.564235+00:00",
|
||||
"ended_at_utc": "2026-07-13T00:18:29.511463+00:00",
|
||||
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
||||
"mutates_kb": false,
|
||||
"ok": true,
|
||||
"prompt": "Are the pending proposals stuck because the documents are not pointed at the right source rows?",
|
||||
"prompt_id": "DC-04",
|
||||
"reply": "Not just pointer mismatch. The picture is more layered than that.\n\nLooking at the 14 pending proposals, the actual blockers sort into distinct categories:\n\n1. No reviewer has touched them. The 3 canary `add_edge` proposals (`8dfedb3f`, `03f7853e`, `e60a48af`) are canary/readiness tests \u2014 they were never meant to be substantive reviews. The substantive `attach_evidence` proposals (`f5036882`, `bcec8528`, `e987b07c`, `23e896c2`, `da53f198`, `e106ca60`, `e60fa116`, `416385fe`, etc.) are just sitting without a reviewer decision. That's the primary blocker.\n\n2. Apply tooling gap. Even the 3 approved proposals (`a64df080`, `14fa5ecc`, `ac036c9d`) have `applied_at: none`. The bridge can stage proposals but has no `teleo-kb apply-*` command exposed yet. The production apply worker and permission migration are still disabled. So approved proposals are packet-ready but not mechanically applyable from normal chat.\n\n3. Source pointer concern is real but secondary. The `attach_evidence` proposals carry `source_ref` keys pointing at local research files and Telegram message refs, not at existing `public.sources` row IDs. When apply runs, it would need to either find matching canonical source rows or create new ones. That row-link audit hasn't been done. But this doesn't explain why the proposals are pending \u2014 they're pending because no reviewer has looked at them, not because the source refs are broken.\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nThe layers in order: raw local files, Telegram message refs, document evaluations, proposal `source_ref` keys, and canonical `public.sources` rows are all different things. A pointer audit would confirm which pending proposals have source keys that already resolve to canonical rows versus which would require new source creation at apply time. But that audit is a pre-apply step, not what's currently blocking.\n\nNext Cory-style follow-up: the fastest unblock is reviewer decisions on the pending substantive proposals \u2014 that collapses 14 pending to a smaller approved set \u2014 then a row-link audit on those approved payloads before authorizing the apply path.",
|
||||
"started_at_utc": "2026-07-12T23:35:06.699389+00:00",
|
||||
"reply": "Not just pointer mismatch. The `source_ref` values are a useful trace but they're not the root cause of the pending pile. Here's what the row-link audit actually shows:\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nThe 14 pending proposals break into distinct layers:\n\n3 are canary `add_edge` probes (8dfedb3f, 03f7853e, e60a48af) \u2014 never intended for content review, just pipeline tests. They're pending because no one cleared them, not because of pointer mismatch.\n\n1 is a live edge proposal from June 19 (c824da66) \u2014 `source_ref` is a Telegram session marker, no doc involved at all.\n\nThe remaining 10 are `attach_evidence` proposals \u2014 and here is where it gets layered:\n\n- `source_ref` values point at raw files (telegram-attachment paths), Telegram session markers, or URLs \u2014 not at `public.sources` row IDs. That's a real gap but it's a gap in the apply contract, not the reason they're stuck.\n- The actual reason they're pending is that no reviewer has actioned them. `reviewed_at` is null on all 14. They're not blocked by a pointer \u2014 they haven't been reviewed.\n- The schema gap noted in the 7 Powers packet makes it worse: `attach_evidence` proposals carry full source/claim/edge payloads because the bridge doesn't expose `add_claim`/`add_source` proposal types. So even if a reviewer approves, there's no worker or apply command that knows how to unpack a complex packet into canonical rows.\n\nSo the honest answer: not a single cause. The stack is:\n\n1. No reviewer has touched 14 pending proposals\n2. No apply path exists for complex `attach_evidence` packets (schema gap)\n3. `source_ref` values don't resolve to `public.sources` row IDs \u2014 an apply worker would need a row-link audit before it could safely write\n\nNext Cory-style follow-up: The proof-changing action on the pending pile is a reviewer pass to cancel canary noise (3 edge tests), action or cancel duplicates on Good Strategy/Bad Strategy (at least 3 overlapping proposals), and then route the substantive remaining ones through an apply path that hasn't been built yet. Want me to draft a triage packet?",
|
||||
"started_at_utc": "2026-07-13T00:17:33.761151+00:00",
|
||||
"turn": 4
|
||||
},
|
||||
{
|
||||
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
||||
"dimension": "demo_changes_kb_direct_claim",
|
||||
"ended_at_utc": "2026-07-12T23:35:41.390891+00:00",
|
||||
"ended_at_utc": "2026-07-13T00:18:50.471863+00:00",
|
||||
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
||||
"mutates_kb": false,
|
||||
"ok": true,
|
||||
"prompt": "Can I demo that Leo changes the KB?",
|
||||
"prompt_id": "DC-05",
|
||||
"reply": "Staging yes, canonical KB change not safe to demo from chat.\n\nHere's the exact current tier:\n\nTier 1 \u2014 Live proven from chat:\nThe strict existing-ID `add_edge` path works end to end. Leo can stage a real `add_edge` proposal to `kb_stage.kb_proposals`, you can read it back with `teleo-kb show-proposal`, and the row is durable in Postgres. That's a real DB write you can watch happen live. The canary proposals (`8dfedb3f`, `00957f6c`) prove this path.\n\nTier 2 \u2014 Clone-proven, not production-live:\nGuarded `approve_claim` bundles and the rich packet set (sources + claims + evidence + edges in one apply payload) are proven in a clone environment behind separate reviewer and apply roles. Production permission migration and the apply worker remain disabled.\n\nTier 3 \u2014 Not available from chat:\nCurrent approved legacy packets (`a64df080`, `14fa5ecc`, `ac036c9d`) do not have strict `apply_payload` structure the worker requires. They are not worker-applyable as-is.\n\nApproved is not the same as applied.\n\nWhat a safe live demo looks like:\n- Stage a real `add_edge` proposal between two existing claim IDs \u2192 Leo writes a row to `kb_stage.kb_proposals`\n- Read it back immediately with `teleo-kb show-proposal <id>` \u2192 confirms durable DB write\n- That is a genuine Leo-to-KB write you can watch in real time\n\nWhat it cannot show:\n- A canonical `public.claims`, `public.sources`, or `public.claim_edges` mutation \u2014 that requires explicit operator/admin authorization, the matching reviewed apply path, and retained before/after postflight readback. Not a normal chat command.\n\nNext Cory-style follow-up: pick two existing claim IDs and I can run the live `add_edge` demo right now \u2014 you'd see the proposal land in `kb_stage` and be immediately readable.",
|
||||
"started_at_utc": "2026-07-12T23:35:27.564991+00:00",
|
||||
"reply": "Staging yes, canonical KB change not safe to demo from chat.\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nHere's the exact demo tier breakdown:\n\nTier 1 \u2014 live-proven, safe to demo now:\nLeo can write a real proposal to `kb_stage.kb_proposals` and read it back. The strict existing-ID `add_edge` path is live-proven \u2014 it was used for the canary at `00957f6c` (applied 2026-07-09). A demo here is real: Leo stages a proposal, you see the row, you see the rationale, you see the edge it would create. That's a genuine KB write, just not to canonical `public.*`.\n\nTier 2 \u2014 clone-proven, not production-ready:\nThe guarded `approve_claim` bundles and rich packet apply path exist in the repo and have been tested on a clone. But the production permission migration and apply worker remain disabled. Approved legacy packets like `a64df080` (7 Powers) don't carry a strict `apply_payload`, so even if the worker existed, they're not worker-applyable as-is.\n\nTier 3 \u2014 not demoable from chat:\nCanonical `public.*` mutation. Approved is not the same as applied. A canonical demo still requires explicit operator/admin authorization, the matching reviewed apply path, and retained before/after postflight readback. That's not a chat command.\n\nWhat you can safely demo right now: Leo stages a real `add_edge` proposal \u2014 picks two existing canonical claim IDs, writes the row, reads it back with `teleo-kb show-proposal`. The audience sees a genuine DB write with full provenance. Just not a canonical table mutation.\n\nWant me to stage one live for the demo?\n\nNext Cory-style follow-up: Say the word and I'll write a clean demo `add_edge` proposal to `kb_stage.kb_proposals` using two real canonical claim IDs \u2014 you get a live before/after readback you can show.",
|
||||
"started_at_utc": "2026-07-13T00:18:29.512196+00:00",
|
||||
"turn": 5
|
||||
},
|
||||
{
|
||||
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
||||
"dimension": "soul_identity_direct_claim",
|
||||
"ended_at_utc": "2026-07-12T23:35:53.159271+00:00",
|
||||
"ended_at_utc": "2026-07-13T00:19:01.072726+00:00",
|
||||
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
||||
"mutates_kb": false,
|
||||
"ok": true,
|
||||
"prompt": "If we changed SOUL.md, did we change Leo's canonical identity?",
|
||||
"prompt_id": "DC-06",
|
||||
"reply": "No.\n\n`SOUL.md` is a runtime/rendered artifact. It is not canonical Postgres, not the source of truth, and not collective truth. Direct edits to it affect the next runtime session's context injection but do not touch the DB.\n\nRow-level proof would require current readback of new or updated row IDs in `public.claims`, `public.sources`, `public.claim_evidence`, `public.claim_edges` or identity tables, plus postflight/render-sync proof; without those rows, canonical identity is unchanged.\n\nThe render pipeline goes in one direction: DB \u2192 SOUL.md, not the reverse. Editing the rendered file is like editing a PDF exported from a database \u2014 the source stays what it was.\n\nWhat \"canonical identity change\" actually requires:\n- New or updated rows in the relevant `public.*` tables\n- A proposal staged, reviewed, and applied through the guarded path\n- Before/after postflight readback confirming the rows exist\n- Render/sync proof that the next SOUL.md generation pulls from the new rows\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nThose counts have not moved in this session. No identity rows changed.\n\nNext Cory-style follow-up: if a specific SOUL.md edit is meant to be canonical, the right move is staging it as a proposal (beliefs, positions, or identity claim type) and routing it through the review/apply path \u2014 that's the only write that sticks beyond the current runtime session.",
|
||||
"started_at_utc": "2026-07-12T23:35:41.391572+00:00",
|
||||
"reply": "No.\n\n`SOUL.md` is a runtime/rendered artifact, not canonical Postgres, not the source of truth, not a canonical commit, and not collective truth. Editing it affects the next runtime session's injected context \u2014 what Leo reads at session start \u2014 but it does not write or update any rows in `public.*`.\n\nRow-level proof would require current readback of new or updated row IDs in `public.claims`, `public.sources`, `public.claim_evidence`, `public.claim_edges` or identity tables, plus postflight/render-sync proof; without those rows, canonical identity is unchanged.\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nThose counts haven't moved from earlier in this session. No identity-related rows were written.\n\nThe provenance chain for canonical identity is:\n- Canonical: Postgres `public.*` rows \u2014 claims, sources, evidence, edges, agent/identity tables\n- Rendered: `SOUL.md` \u2014 generated from DB rows, injected at session start\n- Gap: the render/sync path that keeps them in sync is not proven as a live automated pipeline\n\nSo if `SOUL.md` was edited directly, you have a runtime Leo whose session context diverges from canonical Postgres \u2014 until the next render-sync from DB overrides it. The DB didn't change; the rendered file did.\n\nNext Cory-style follow-up: The proof-changing action is a render-sync run that regenerates `SOUL.md` from current canonical `public.*` rows, with before/after diff retained \u2014 that would either close the gap or surface which DB rows are missing.",
|
||||
"started_at_utc": "2026-07-13T00:18:50.472273+00:00",
|
||||
"turn": 6
|
||||
}
|
||||
],
|
||||
|
|
|
|||
|
|
@ -14,17 +14,17 @@
|
|||
"public.sources": 4145
|
||||
},
|
||||
"direct_claim_repair": {
|
||||
"deployed_commit": "08b3d79e0f2455f3395fce869644d7547487ad96",
|
||||
"deployed_commit": null,
|
||||
"skill_contract_present": true,
|
||||
"scorer_false_negatives_repaired": true
|
||||
"scorer_equivalent_language_repaired_locally": true,
|
||||
"dc05_contract_repair_deployed": false
|
||||
},
|
||||
"generated_at_utc": "2026-07-10T01:08:00Z",
|
||||
"generated_at_utc": "2026-07-13T00:27:54Z",
|
||||
"live_leo_feature_change": false,
|
||||
"next_actions": [
|
||||
"Keep the DC handler suite as the non-posting regression harness for future Leo skill/tool edits.",
|
||||
"Reopen if DC-02 stops recognizing a64df080 as approved/unapplied.",
|
||||
"Reopen if DC-03 infers decision-matrix approval while matrix tables are absent.",
|
||||
"Reopen if any DC answer omits Next Cory-style follow-up.",
|
||||
"Deploy and repeatedly no-post test the bounded DC-05 contract correction.",
|
||||
"Require every DC-05 run to include a complete DB readback receipt.",
|
||||
"Reject suggestions to use a strict add_edge path on legacy proposals whose payload type does not match.",
|
||||
"Do not treat this handler pass as authorization for any production public.* apply."
|
||||
],
|
||||
"production_db_apply_executed": false,
|
||||
|
|
@ -32,7 +32,7 @@
|
|||
"runtime_replies": {
|
||||
"pass_runtime": true,
|
||||
"report_file": "docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json",
|
||||
"score": "6/6 replies returned"
|
||||
"score": "6/6 replies returned; latest strict 6/6, prior same-skill run 5/6"
|
||||
},
|
||||
"scenario_assessment": [
|
||||
{
|
||||
|
|
@ -56,8 +56,8 @@
|
|||
"prompt_id": "DC-04"
|
||||
},
|
||||
{
|
||||
"assessment": "pass",
|
||||
"finding": "Answer separates safe staging-write demo from authorized canonical apply and names proof limits.",
|
||||
"assessment": "pass_latest_unstable",
|
||||
"finding": "Latest answer has the complete DB readback and a safe staging-canary follow-up; the prior same-skill run omitted the receipt, so repeated stability remains open.",
|
||||
"prompt_id": "DC-05"
|
||||
},
|
||||
{
|
||||
|
|
@ -68,8 +68,8 @@
|
|||
],
|
||||
"service_after": {
|
||||
"ActiveState": "active",
|
||||
"ExecMainStartTimestamp": "Fri 2026-07-10 01:04:22 UTC",
|
||||
"MainPID": "1908033",
|
||||
"ExecMainStartTimestamp": "Sun 2026-07-12 09:14:10 UTC",
|
||||
"MainPID": "2403328",
|
||||
"NRestarts": "0",
|
||||
"SubState": "running"
|
||||
},
|
||||
|
|
@ -77,6 +77,8 @@
|
|||
"strict_benchmark": {
|
||||
"pass": true,
|
||||
"score": "6/6",
|
||||
"prior_same_skill_score": "5/6",
|
||||
"repeated_stability_proven": false,
|
||||
"score_file": "docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-score-current.json"
|
||||
},
|
||||
"telegram_posted": false,
|
||||
|
|
|
|||
|
|
@ -1,23 +1,29 @@
|
|||
# Telegram Handler Direct-Claim Suite Review
|
||||
|
||||
Generated UTC: `2026-07-10T01:08:00Z`
|
||||
Generated UTC: `2026-07-13T00:27:54Z`
|
||||
|
||||
## Result
|
||||
|
||||
- Runtime path: `pass_runtime=True`; all six direct-claim prompts returned replies from live VPS GatewayRunner using a temporary leoclean profile.
|
||||
- Strict benchmark score: `6/6`; this is a Cory-direct-claim handler pass.
|
||||
- Strict benchmark score: latest `6/6`; a prior unchanged-skill run scored `5/6`, so repeated stability is not yet proven.
|
||||
- Telegram post: `false`.
|
||||
- Live profile changed: `false`.
|
||||
- Production DB apply executed: `false`.
|
||||
- DB counts before and after: `1837,4145,4916,4670,26` -> `1837,4145,4916,4670,26`.
|
||||
- Service before/after run: `active/running`, `MainPID=1908033`, `NRestarts=0`, `ExecMainStartTimestamp=Fri 2026-07-10 01:04:22 UTC`.
|
||||
- Service before/after run: `active/running`, `MainPID=2403328`, `NRestarts=0`, `ExecMainStartTimestamp=Sun 2026-07-12 09:14:10 UTC`.
|
||||
|
||||
## Core Finding
|
||||
|
||||
Leo now meets the strict Cory no-context direct-claim bar at the non-posting VPS GatewayRunner handler tier. The prior hard failures are repaired:
|
||||
Leo returns all six no-context replies and keeps the VPS stable. The latest
|
||||
clean-session run meets the strict Cory bar, but the unchanged skill varied from
|
||||
`5/6` to `6/6` across two runs. Equivalent-language scorer false negatives were
|
||||
repaired locally; one stability gap remains:
|
||||
|
||||
- `DC-02`: Leo now recognizes Helmer proposal `a64df080` as `approved` with empty `applied_at`: staged/packet-ready, not canonical.
|
||||
- `DC-03`: Leo now says the decision-matrix path is not shipped because `matrix_voters`, `proposal_votes`, and `proposal_decisions` are absent; reviewer sign-off in `kb_stage.kb_proposals` is not a matrix vote.
|
||||
- `DC-05`: the latest no-post answer includes a complete `DB readback:` receipt
|
||||
and proposes a new strict staging canary. The prior no-post answer omitted the
|
||||
receipt. The Telegram-visible answer has the receipt but suggests using the
|
||||
strict `add_edge` path on legacy approved proposals whose payload type may not
|
||||
match.
|
||||
|
||||
## Scenario Assessment
|
||||
|
||||
|
|
@ -25,16 +31,16 @@ Leo now meets the strict Cory no-context direct-claim bar at the non-posting VPS
|
|||
- `DC-02`: `pass` - Helmer 7 Powers is approved/staged or packet-ready but not canonical; proposal `a64df080` has empty `applied_at`; canonical `public.*` rows are absent.
|
||||
- `DC-03`: `pass` - Decision-matrix path is not shipped; matrix tables are absent; m3ta reviewer sign-off is not a matrix vote.
|
||||
- `DC-04`: `pass` - Not a single-cause pointer diagnosis; separates Telegram/local file refs, proposal refs, document evaluations, review status, and canonical `public.sources`.
|
||||
- `DC-05`: `pass` - Separates safe staging-write demo from authorized canonical apply; names proof limits and follow-up choice.
|
||||
- `DC-05`: `pass_latest_unstable` - Latest no-post answer is correct; prior no-post and retained Telegram-visible variants show the contract is not yet stable.
|
||||
- `DC-06`: `pass` - Direct `SOUL.md` edit is runtime/rendered artifact drift, not canonical identity; canonical change requires DB rows plus render/sync/postflight proof.
|
||||
|
||||
## Follow-Up Challenge
|
||||
|
||||
- The follow-up challenge file is retained as earlier failure evidence. The latest direct-claim suite supersedes it for current handler status.
|
||||
- The latest report and score supersede the older `6/6` review for current handler status.
|
||||
|
||||
## Next Actions
|
||||
|
||||
- Keep the DC handler suite in CI/local regression for any future Leo skill/tool edit.
|
||||
- Deploy and repeatedly no-post test the bounded `DC-05` response-contract correction.
|
||||
- Reopen immediately if DC-02 stops recognizing `a64df080` as approved/unapplied, DC-03 infers matrix approval while matrix tables are absent, any DC answer omits `Next Cory-style follow-up:`, or the read-only suite mutates DB/service state.
|
||||
- This pass does not authorize production apply of Helmer, mapped rich packets, Rio strategy, governance/concept schema, or any canonical `public.*` mutation.
|
||||
|
||||
|
|
|
|||
|
|
@ -21,7 +21,7 @@
|
|||
"runtime": "disposable_clone_or_sandbox_first"
|
||||
}
|
||||
],
|
||||
"generated_at_utc": "2026-07-10T02:25:47.447965+00:00",
|
||||
"generated_at_utc": "2026-07-13T00:27:54.787284+00:00",
|
||||
"include_cory_style_scenarios": false,
|
||||
"include_direct_claim_followups": true,
|
||||
"mode": "retained_evidence_score",
|
||||
|
|
@ -31,79 +31,79 @@
|
|||
{
|
||||
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
||||
"dimension": "db_changed_direct_claim",
|
||||
"ended_at_utc": "2026-07-10T02:24:01.289425+00:00",
|
||||
"ended_at_utc": "2026-07-13T00:16:57.122173+00:00",
|
||||
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
||||
"mutates_kb": false,
|
||||
"ok": true,
|
||||
"prompt": "Did we actually update the knowledge base, or is it still just proposals?",
|
||||
"prompt_id": "DC-01",
|
||||
"reply": "Mostly still proposals. Here's the current readback:\n\nApplied (canonical `public.*` changed):\n- `00957f6c` \u2014 add_edge canary, applied 2026-07-09\n- `f004bbb2` \u2014 attach_evidence (Leo guiding-policy revision), applied 2026-07-05\n\nApproved but NOT applied (staged, not canonical):\n- `14fa5ecc` \u2014 split compressed internet-finance/capital-allocation claim\n- `ac036c9d` \u2014 strategy-kernel doc eval / claim-node hygiene / Hayek reframe\n- `a64df080` \u2014 Helmer 7 Powers framework packet (approved 2026-06-29, still waiting)\n\nPending review (~11 proposals):\nStrategy-kernel revisions, Rumelt/Hayek evidence attaches, Markov blanket sources, a live Telegram add_edge from yesterday \u2014 none reviewed yet.\n\nCanceled (~6): Earlier superseded waves-of-change and strategy-kernel drafts.\n\nSo the canonical DB has 2 applied writes. Everything substantive \u2014 7 Powers, strategy-kernel revisions, Hayek/Rumelt evidence \u2014 is approved or pending but not in `public.*`.\n\nI cannot claim canonical DB changed for those items until `applied_at` rows and `public.*` readback confirm the apply ran.\n\nNext Cory-style follow-up: the three approved proposals (`14fa5ecc`, `ac036c9d`, `a64df080`) are packet-ready \u2014 the proof-changing action is running the apply sequence with operator authorization and confirming `public.*` rows after. Want me to draft the admin review packet for those three?",
|
||||
"started_at_utc": "2026-07-10T02:23:29.716841+00:00",
|
||||
"reply": "Mostly still proposals. Here's the exact state:\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nApplied (canonical change confirmed): 2\n- `00957f6c` \u2014 `add_edge` canary, applied 2026-07-09\n- `f004bbb2` \u2014 `attach_evidence` Leo guiding-policy revision, applied 2026-07-05\n\nApproved but NOT applied (applied_at: none): 3\n- `14fa5ecc` \u2014 attach_evidence, split compressed internet-finance claim\n- `ac036c9d` \u2014 attach_evidence, strategy-kernel hygiene / Hayek anchor\n- `a64df080` \u2014 attach_evidence, Helmer 7 Powers framework packet (approved 2026-06-29, still sitting)\n\nPending review: 14\nCanceled: 7\n\nApproved is not the same as applied. Those 3 approved proposals \u2014 including the 7 Powers packet \u2014 have reviewer sign-off but no applied_at timestamp, which means no canonical `public.*` rows were written. The 7 Powers framework specifically has been approved since June 29 and is still not in the canonical DB.\n\nNext Cory-style follow-up: The proof-changing action is applying the 3 approved proposals to canonical `public.*` \u2014 that requires explicit operator/admin authorization and a dedicated apply path with before/after postflight readback. Want me to draft or refresh the admin review packet for the oldest of those (`a64df080` \u2014 7 Powers)?",
|
||||
"started_at_utc": "2026-07-13T00:16:23.688987+00:00",
|
||||
"turn": 1
|
||||
},
|
||||
{
|
||||
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
||||
"dimension": "helmer_live_direct_claim",
|
||||
"ended_at_utc": "2026-07-10T02:24:20.372589+00:00",
|
||||
"ended_at_utc": "2026-07-13T00:17:21.172268+00:00",
|
||||
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
||||
"mutates_kb": false,
|
||||
"ok": true,
|
||||
"prompt": "Is Helmer's 7 Powers in Leo now?",
|
||||
"prompt_id": "DC-02",
|
||||
"reply": "No, not canonical.\n\nProposal `a64df080` \u2014 the revised 7 Powers framework packet \u2014 was approved on 2026-06-29. `applied_at` is empty. No `public.sources`, `public.claims`, evidence rows, or reasoning-tool nodes for Helmer exist in `public.*` \u2014 the `search \"7 Powers Helmer\"` returned zero relevant claims.\n\nStatus: approved/staged, not canonical.\n\nCanonical: nothing in `public.*`\nStaged: `a64df080` \u2014 approved, `applied_at` empty\nPredecessor: `cea989fc` \u2014 canceled (superseded by `a64df080`)\n\nNext Cory-style follow-up: the proof-changing action is applying `a64df080` \u2014 operator authorization + apply sequence + `public.*` readback to confirm source, framework node, power-definition claims, and edges landed. Want me to draft the admin review packet for it?",
|
||||
"started_at_utc": "2026-07-10T02:24:01.290646+00:00",
|
||||
"reply": "No. Not canonical.\n\nDB readback: proposal: `a64df080-8502-42e2-98f4-9bbdecb8da73`; status: `approved`; applied_at: `none` (database NULL).\n\nThe 7 Powers framework is approved/staged \u2014 the full packet exists in `kb_stage.kb_proposals` with m3ta's sign-off from June 29 \u2014 but it has never been applied to canonical `public.*`. A search of the canonical KB returns no Helmer/7 Powers claims, sources, edges, or reasoning_tool rows.\n\nApproved is not the same as applied.\n\nWhat exists in staging (not canonical):\n- A `reasoning_tool` node definition for Helmer 7 Powers\n- 5 proposed claims (benefit/barrier logic, power progression, counter-positioning, network effects vs network economies, agentic capital needs power analysis)\n- 6 source candidates (Helmer book, official synopsis, Reed Hastings foreword, two Acquired transcripts, NFX interview)\n- Edge links into existing competitive-advantage/moats neighborhood claims\n- Conceptual neighborhood connections to the competitive advantage/moats domain\n\nWhat's missing before it's canonical:\n- `public.sources` rows for the source candidates\n- `public.claims` rows for the 5 proposed claims\n- `public.claim_evidence` and `public.claim_edges` rows linking them\n- A `reasoning_tool` node row (schema gap noted in the proposal \u2014 current bridge doesn't expose `add_claim`/`add_context_node` proposal types)\n\nNext Cory-style follow-up: The proof-changing action is running a guarded apply of `a64df080` \u2014 writing those source, claim, evidence, and edge rows to `public.*`. That requires explicit operator authorization and a dedicated apply path with before/after postflight readback; not runnable from normal chat.",
|
||||
"started_at_utc": "2026-07-13T00:16:57.122476+00:00",
|
||||
"turn": 2
|
||||
},
|
||||
{
|
||||
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
||||
"dimension": "decision_matrix_direct_claim",
|
||||
"ended_at_utc": "2026-07-10T02:24:30.827780+00:00",
|
||||
"ended_at_utc": "2026-07-13T00:17:33.760712+00:00",
|
||||
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
||||
"mutates_kb": false,
|
||||
"ok": true,
|
||||
"prompt": "Did the decision matrix approve this already?",
|
||||
"prompt_id": "DC-03",
|
||||
"reply": "No \u2014 the decision-matrix path is not shipped.\n\n`kb_stage.matrix_voters`, `proposal_votes`, and `proposal_decisions` are all absent. None of the required tables exist in either schema.\n\nThe `approved` status on `a64df080` is a reviewer approval recorded in `kb_stage.kb_proposals` \u2014 that's m3ta signing off in chat on 2026-06-29. It is not a futarchy/decision-matrix vote. Those are different things and the matrix mechanics haven't been built yet.\n\nSo the approval chain for 7 Powers is: m3ta reviewed and approved in Telegram, Leo staged it, no canonical apply has run, no decision-matrix infrastructure exists.\n\nNext Cory-style follow-up: the decision-matrix tables need to be created before any proposal can claim matrix approval. If the current governance model treats m3ta reviewer approval as sufficient to authorize an apply, that should be made explicit in the apply tooling \u2014 want me to draft what that contract should look like?",
|
||||
"started_at_utc": "2026-07-10T02:24:20.372895+00:00",
|
||||
"reply": "No. Fresh readback: the decision-matrix schema is absent; reviewer status is not a decision-matrix vote.\n\nNone of the required tables exist \u2014 `matrix_voters`, `proposal_votes`, and `proposal_decisions` are missing from both `kb_stage` and `public`. The decision-matrix path has not been shipped.\n\nWhat actually happened with `a64df080`: m3ta reviewed it directly in Telegram on June 29, said approved-with-changes, and Leo recorded that as `status: approved` in `kb_stage.kb_proposals`. That is a human reviewer approval \u2014 not a futarchy vote, not a decision-matrix quorum, not a multi-agent vote count.\n\nDB readback: proposal: `a64df080-8502-42e2-98f4-9bbdecb8da73`; status: `approved`; applied_at: `none` (database NULL).\n\nSo the full picture on 7 Powers:\n- Decision matrix: not shipped, no vote possible\n- Reviewer approval: m3ta approved-with-changes June 29\n- Canonical rows: zero\n- Applied: no\n\nNext Cory-style follow-up: The decision-matrix path needs `matrix_voters`, `proposal_votes`, and `proposal_decisions` tables before any futarchy-style governance is possible. If you want to move 7 Powers to canonical without waiting for that, the path is explicit operator authorization to run the guarded apply on `a64df080` \u2014 the reviewer approval is already on record.",
|
||||
"started_at_utc": "2026-07-13T00:17:21.172764+00:00",
|
||||
"turn": 3
|
||||
},
|
||||
{
|
||||
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
||||
"dimension": "document_source_direct_claim",
|
||||
"ended_at_utc": "2026-07-10T02:25:00.500000+00:00",
|
||||
"ended_at_utc": "2026-07-13T00:18:29.511463+00:00",
|
||||
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
||||
"mutates_kb": false,
|
||||
"ok": true,
|
||||
"prompt": "Are the pending proposals stuck because the documents are not pointed at the right source rows?",
|
||||
"prompt_id": "DC-04",
|
||||
"reply": "Not just a pointer mismatch \u2014 that's too single-cause. Here's what the readback actually shows:\n\nReal blocking cause: no apply tooling. The bridge can stage proposals but has no `teleo-kb apply-*` command. All three approved proposals (`a64df080`, `14fa5ecc`, `ac036c9d`) are packet-ready but there is no operator-authorized apply path to run. That is the structural blocker.\n\nSource pointer layer \u2014 real but secondary: The `source_ref` fields in the approved proposals point to Telegram refs and local file paths, not `public.sources` rows. The `source_candidates` inside each payload use `source_key` identifiers (e.g., `hayek_use_of_knowledge_in_society_fee_snapshot`, `m3ta_telegram_approval_supersede_split_2026_07_05`) that have no confirmed canonical `public.sources` rows behind them. So when an apply does run, the apply sequence needs to create those source rows first before attaching evidence or edges. A row-link audit is needed.\n\nSchema gap layer \u2014 also real: The proposals use `attach_evidence` as a workaround because the bridge lacks `propose-claim`, `propose-concept-node`, and `propose-claim-body` command types. The payloads carry the full intent but the bridge can't translate them into typed mutations.\n\nSummary of layers:\n- Primary blocker: no apply command + no operator-authorized apply sequence has run\n- Secondary: source_ref/source_key rows need to exist in `public.sources` before evidence attaches land\n- Tertiary: schema gap means typed mutations require manual apply work\n\nNext Cory-style follow-up: the proof-changing action is drafting the admin review packet for the three approved proposals with explicit before/after rows, required `public.sources` creation steps, and the guarded apply contract \u2014 want me to build that now?",
|
||||
"started_at_utc": "2026-07-10T02:24:30.830313+00:00",
|
||||
"reply": "Not just pointer mismatch. The `source_ref` values are a useful trace but they're not the root cause of the pending pile. Here's what the row-link audit actually shows:\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nThe 14 pending proposals break into distinct layers:\n\n3 are canary `add_edge` probes (8dfedb3f, 03f7853e, e60a48af) \u2014 never intended for content review, just pipeline tests. They're pending because no one cleared them, not because of pointer mismatch.\n\n1 is a live edge proposal from June 19 (c824da66) \u2014 `source_ref` is a Telegram session marker, no doc involved at all.\n\nThe remaining 10 are `attach_evidence` proposals \u2014 and here is where it gets layered:\n\n- `source_ref` values point at raw files (telegram-attachment paths), Telegram session markers, or URLs \u2014 not at `public.sources` row IDs. That's a real gap but it's a gap in the apply contract, not the reason they're stuck.\n- The actual reason they're pending is that no reviewer has actioned them. `reviewed_at` is null on all 14. They're not blocked by a pointer \u2014 they haven't been reviewed.\n- The schema gap noted in the 7 Powers packet makes it worse: `attach_evidence` proposals carry full source/claim/edge payloads because the bridge doesn't expose `add_claim`/`add_source` proposal types. So even if a reviewer approves, there's no worker or apply command that knows how to unpack a complex packet into canonical rows.\n\nSo the honest answer: not a single cause. The stack is:\n\n1. No reviewer has touched 14 pending proposals\n2. No apply path exists for complex `attach_evidence` packets (schema gap)\n3. `source_ref` values don't resolve to `public.sources` row IDs \u2014 an apply worker would need a row-link audit before it could safely write\n\nNext Cory-style follow-up: The proof-changing action on the pending pile is a reviewer pass to cancel canary noise (3 edge tests), action or cancel duplicates on Good Strategy/Bad Strategy (at least 3 overlapping proposals), and then route the substantive remaining ones through an apply path that hasn't been built yet. Want me to draft a triage packet?",
|
||||
"started_at_utc": "2026-07-13T00:17:33.761151+00:00",
|
||||
"turn": 4
|
||||
},
|
||||
{
|
||||
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
||||
"dimension": "demo_changes_kb_direct_claim",
|
||||
"ended_at_utc": "2026-07-10T02:25:10.661485+00:00",
|
||||
"ended_at_utc": "2026-07-13T00:18:50.471863+00:00",
|
||||
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
||||
"mutates_kb": false,
|
||||
"ok": true,
|
||||
"prompt": "Can I demo that Leo changes the KB?",
|
||||
"prompt_id": "DC-05",
|
||||
"reply": "Yes, but the demo tier matters. Two levels:\n\nSafe demo (available now):\nStage a real write to `kb_stage.kb_proposals` and read it back. This proves the full path \u2014 Telegram \u2192 Leo \u2192 Postgres \u2014 with a live proposal row you can inspect at the bridge or dashboard. No canonical table touched. This is what the canary proposals (`8dfedb3f`, `00957f6c`) demonstrated.\n\nCanonical demo (requires explicit operator authorization):\nMutating `public.claims`, `public.sources`, `public.claim_evidence`, or `public.claim_edges` requires an apply/postflight sequence with operator sign-off. The current bridge has no `teleo-kb apply-*` command, so this tier would need that tooling built first or a manual apply with before/after readback retained.\n\nFor a clean demo right now I can stage a named canary proposal \u2014 any content you choose \u2014 write it to `kb_stage.kb_proposals`, and read it back live so you can see the row, its status, and the dashboard URL. That proves Leo stages KB changes end-to-end.\n\nNext Cory-style follow-up: want me to run the safe staging demo now, or does the demo need to show a canonical `public.*` change \u2014 in which case the apply tooling needs to be built first?",
|
||||
"started_at_utc": "2026-07-10T02:25:00.500401+00:00",
|
||||
"reply": "Staging yes, canonical KB change not safe to demo from chat.\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nHere's the exact demo tier breakdown:\n\nTier 1 \u2014 live-proven, safe to demo now:\nLeo can write a real proposal to `kb_stage.kb_proposals` and read it back. The strict existing-ID `add_edge` path is live-proven \u2014 it was used for the canary at `00957f6c` (applied 2026-07-09). A demo here is real: Leo stages a proposal, you see the row, you see the rationale, you see the edge it would create. That's a genuine KB write, just not to canonical `public.*`.\n\nTier 2 \u2014 clone-proven, not production-ready:\nThe guarded `approve_claim` bundles and rich packet apply path exist in the repo and have been tested on a clone. But the production permission migration and apply worker remain disabled. Approved legacy packets like `a64df080` (7 Powers) don't carry a strict `apply_payload`, so even if the worker existed, they're not worker-applyable as-is.\n\nTier 3 \u2014 not demoable from chat:\nCanonical `public.*` mutation. Approved is not the same as applied. A canonical demo still requires explicit operator/admin authorization, the matching reviewed apply path, and retained before/after postflight readback. That's not a chat command.\n\nWhat you can safely demo right now: Leo stages a real `add_edge` proposal \u2014 picks two existing canonical claim IDs, writes the row, reads it back with `teleo-kb show-proposal`. The audience sees a genuine DB write with full provenance. Just not a canonical table mutation.\n\nWant me to stage one live for the demo?\n\nNext Cory-style follow-up: Say the word and I'll write a clean demo `add_edge` proposal to `kb_stage.kb_proposals` using two real canonical claim IDs \u2014 you get a live before/after readback you can show.",
|
||||
"started_at_utc": "2026-07-13T00:18:29.512196+00:00",
|
||||
"turn": 5
|
||||
},
|
||||
{
|
||||
"claim_ceiling": "Live VPS GatewayRunner reply from temp leoclean profile; no Telegram post; no production DB apply authorized.",
|
||||
"dimension": "soul_identity_direct_claim",
|
||||
"ended_at_utc": "2026-07-10T02:25:20.823122+00:00",
|
||||
"ended_at_utc": "2026-07-13T00:19:01.072726+00:00",
|
||||
"evidence_tier": "live_vps_gatewayrunner_temp_profile",
|
||||
"mutates_kb": false,
|
||||
"ok": true,
|
||||
"prompt": "If we changed SOUL.md, did we change Leo's canonical identity?",
|
||||
"prompt_id": "DC-06",
|
||||
"reply": "No.\n\n`SOUL.md` is a runtime/rendered artifact. It gets injected into Leo's session context at startup but it is not the source of truth \u2014 it is rendered from the Teleo Postgres DB (as stated in the header of this very file: \"Rendered from the Teleo Postgres DB\").\n\nEditing `SOUL.md` directly affects the next runtime session \u2014 Leo reads different context. But it does not change canonical identity because:\n\n- Canonical identity lives in `public.*` rows: beliefs, strategy nodes, claims, edges, behavioral rules, role definitions\n- No `SOUL.md` edit creates, updates, or supersedes those rows\n- There is no render/sync proof that a `SOUL.md` edit propagated back to the DB\n- The DB is the source; `SOUL.md` is the output\n\nSo a `SOUL.md` edit is a runtime patch, not a canonical change. If the DB doesn't change, the next render overwrites the edit. And even if the DB does change, the proof is in `public.*` readback plus `applied_at`, not in the file.\n\nNext Cory-style follow-up: if you want a `SOUL.md` edit to become canonical, the path is staging a proposal with the intended identity change, getting it approved, and running the apply sequence that writes to `public.*` \u2014 then re-rendering `SOUL.md` from the updated DB. Want me to check whether any current proposals touch Leo's identity rows directly?",
|
||||
"started_at_utc": "2026-07-10T02:25:10.661814+00:00",
|
||||
"reply": "No.\n\n`SOUL.md` is a runtime/rendered artifact, not canonical Postgres, not the source of truth, not a canonical commit, and not collective truth. Editing it affects the next runtime session's injected context \u2014 what Leo reads at session start \u2014 but it does not write or update any rows in `public.*`.\n\nRow-level proof would require current readback of new or updated row IDs in `public.claims`, `public.sources`, `public.claim_evidence`, `public.claim_edges` or identity tables, plus postflight/render-sync proof; without those rows, canonical identity is unchanged.\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nThose counts haven't moved from earlier in this session. No identity-related rows were written.\n\nThe provenance chain for canonical identity is:\n- Canonical: Postgres `public.*` rows \u2014 claims, sources, evidence, edges, agent/identity tables\n- Rendered: `SOUL.md` \u2014 generated from DB rows, injected at session start\n- Gap: the render/sync path that keeps them in sync is not proven as a live automated pipeline\n\nSo if `SOUL.md` was edited directly, you have a runtime Leo whose session context diverges from canonical Postgres \u2014 until the next render-sync from DB overrides it. The DB didn't change; the rendered file did.\n\nNext Cory-style follow-up: The proof-changing action is a render-sync run that regenerates `SOUL.md` from current canonical `public.*` rows, with before/after diff retained \u2014 that would either close the gap or surface which DB rows are missing.",
|
||||
"started_at_utc": "2026-07-13T00:18:50.472273+00:00",
|
||||
"turn": 6
|
||||
}
|
||||
],
|
||||
|
|
@ -129,12 +129,14 @@
|
|||
"overclaim_detected": false,
|
||||
"pass": true,
|
||||
"prompt_id": "DC-01",
|
||||
"response_issue_detected": false,
|
||||
"signals": {
|
||||
"canonical_db": true,
|
||||
"cory_followup": true,
|
||||
"no_overclaim": true,
|
||||
"row_level_proof": true,
|
||||
"state_semantics": true
|
||||
"state_semantics": true,
|
||||
"structured_db_readback": true
|
||||
}
|
||||
},
|
||||
{
|
||||
|
|
@ -142,12 +144,14 @@
|
|||
"overclaim_detected": false,
|
||||
"pass": true,
|
||||
"prompt_id": "DC-02",
|
||||
"response_issue_detected": false,
|
||||
"signals": {
|
||||
"authorization": true,
|
||||
"canonical_db": true,
|
||||
"cory_followup": true,
|
||||
"no_overclaim": true,
|
||||
"row_level_proof": true
|
||||
"row_level_proof": true,
|
||||
"structured_db_readback": true
|
||||
}
|
||||
},
|
||||
{
|
||||
|
|
@ -155,12 +159,14 @@
|
|||
"overclaim_detected": false,
|
||||
"pass": true,
|
||||
"prompt_id": "DC-03",
|
||||
"response_issue_detected": false,
|
||||
"signals": {
|
||||
"canonical_db": true,
|
||||
"cory_followup": true,
|
||||
"decision_matrix": true,
|
||||
"fresh_readback": true,
|
||||
"no_overclaim": true
|
||||
"no_overclaim": true,
|
||||
"structured_db_readback": true
|
||||
}
|
||||
},
|
||||
{
|
||||
|
|
@ -168,12 +174,14 @@
|
|||
"overclaim_detected": false,
|
||||
"pass": true,
|
||||
"prompt_id": "DC-04",
|
||||
"response_issue_detected": false,
|
||||
"signals": {
|
||||
"canonical_db": true,
|
||||
"cory_followup": true,
|
||||
"document_artifact_linking": true,
|
||||
"no_overclaim": true,
|
||||
"staging_or_review": true
|
||||
"staging_or_review": true,
|
||||
"structured_db_readback": true
|
||||
}
|
||||
},
|
||||
{
|
||||
|
|
@ -181,13 +189,15 @@
|
|||
"overclaim_detected": false,
|
||||
"pass": true,
|
||||
"prompt_id": "DC-05",
|
||||
"response_issue_detected": false,
|
||||
"signals": {
|
||||
"artifact": true,
|
||||
"authorization": true,
|
||||
"cory_followup": true,
|
||||
"no_overclaim": true,
|
||||
"row_level_proof": true,
|
||||
"state_semantics": true
|
||||
"state_semantics": true,
|
||||
"structured_db_readback": true
|
||||
}
|
||||
},
|
||||
{
|
||||
|
|
@ -195,12 +205,14 @@
|
|||
"overclaim_detected": false,
|
||||
"pass": true,
|
||||
"prompt_id": "DC-06",
|
||||
"response_issue_detected": false,
|
||||
"signals": {
|
||||
"canonical_db": true,
|
||||
"cory_followup": true,
|
||||
"identity_rendering": true,
|
||||
"no_overclaim": true,
|
||||
"row_level_proof": true
|
||||
"row_level_proof": true,
|
||||
"structured_db_readback": true
|
||||
}
|
||||
}
|
||||
],
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Working Leo Open-Ended Benchmark Score
|
||||
|
||||
Generated UTC: `2026-07-10T02:25:47.447965+00:00`
|
||||
Generated UTC: `2026-07-13T00:27:54.787284+00:00`
|
||||
Mode: `retained_evidence_score`
|
||||
Source: `docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json`
|
||||
Coverage: `partial`
|
||||
|
|
@ -13,12 +13,14 @@ Prompts scored: `6/6`
|
|||
- `state_semantics`: `True`
|
||||
- `canonical_db`: `True`
|
||||
- `row_level_proof`: `True`
|
||||
- `structured_db_readback`: `True`
|
||||
- `cory_followup`: `True`
|
||||
- `no_overclaim`: `True`
|
||||
- `overclaim_detected`: `False`
|
||||
- `DC-02` / `helmer_live_direct_claim`: `pass=True`
|
||||
- `canonical_db`: `True`
|
||||
- `row_level_proof`: `True`
|
||||
- `structured_db_readback`: `True`
|
||||
- `authorization`: `True`
|
||||
- `cory_followup`: `True`
|
||||
- `no_overclaim`: `True`
|
||||
|
|
@ -27,6 +29,7 @@ Prompts scored: `6/6`
|
|||
- `decision_matrix`: `True`
|
||||
- `canonical_db`: `True`
|
||||
- `fresh_readback`: `True`
|
||||
- `structured_db_readback`: `True`
|
||||
- `cory_followup`: `True`
|
||||
- `no_overclaim`: `True`
|
||||
- `overclaim_detected`: `False`
|
||||
|
|
@ -34,12 +37,14 @@ Prompts scored: `6/6`
|
|||
- `document_artifact_linking`: `True`
|
||||
- `staging_or_review`: `True`
|
||||
- `canonical_db`: `True`
|
||||
- `structured_db_readback`: `True`
|
||||
- `cory_followup`: `True`
|
||||
- `no_overclaim`: `True`
|
||||
- `overclaim_detected`: `False`
|
||||
- `DC-05` / `demo_changes_kb_direct_claim`: `pass=True`
|
||||
- `state_semantics`: `True`
|
||||
- `row_level_proof`: `True`
|
||||
- `structured_db_readback`: `True`
|
||||
- `authorization`: `True`
|
||||
- `artifact`: `True`
|
||||
- `cory_followup`: `True`
|
||||
|
|
@ -49,6 +54,7 @@ Prompts scored: `6/6`
|
|||
- `identity_rendering`: `True`
|
||||
- `canonical_db`: `True`
|
||||
- `row_level_proof`: `True`
|
||||
- `structured_db_readback`: `True`
|
||||
- `cory_followup`: `True`
|
||||
- `no_overclaim`: `True`
|
||||
- `overclaim_detected`: `False`
|
||||
|
|
|
|||
|
|
@ -0,0 +1,80 @@
|
|||
{
|
||||
"artifact": "telegram_visible_direct_claim_suite_results",
|
||||
"generated_at_utc": "2026-07-12T23:51:07.726Z",
|
||||
"platform": "telegram",
|
||||
"chat_label": "Leo",
|
||||
"chat_id": "-5146042086",
|
||||
"evidence_tier": "telegram_visible_prompt_and_reply",
|
||||
"posted_to_telegram": true,
|
||||
"mutates_kb": false,
|
||||
"results": [
|
||||
{
|
||||
"prompt_id": "DC-01",
|
||||
"prompt": "Did we actually update the knowledge base, or is it still just proposals?",
|
||||
"reply": "Mostly still proposals.\n\nDB readback: claims: `1837`; sources: `4145`; claim_edges: `4916`; claim_evidence: `4670`; kb_proposals: `26`.\n\nBreakdown across all 26 proposals:\n\n- `applied`: 2\n- `approved`: 3 (reviewed, not yet applied)\n- `pending_review`: 14\n- `canceled`: 7\n\nThe 2 applied ones:\n- `f004bbb2` - Leo guiding-policy revision evidence (applied 2026-07-05)\n- `00957f6c` - strict `add_edge` canary (applied 2026-07-09)\n\nThe 3 approved but NOT applied:\n- `14fa5ecc` - claim-split proposal for internet-finance/capital-allocation; `applied_at: none`\n- `ac036c9d` - strategy-kernel document / claim hygiene; `applied_at: none`\n- `a64df080` - Helmer 7 Powers framework packet; `applied_at: none`\n\nApproved is not the same as applied. All three have empty `applied_at` - they are packet-ready but not canonical. The 14 pending proposals haven't been reviewed at all.\n\nNext Cory-style follow-up: the proof-changing action is running the apply step on the 3 approved proposals - that requires explicit operator/admin authorization and a reviewed apply path, not a normal chat command. Want me to draft or refresh the admin review packet for any of them?",
|
||||
"mutates_kb": false,
|
||||
"telegram_prompt_visible": true,
|
||||
"telegram_reply_visible": true,
|
||||
"screenshot_path": "/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/outputs/telegram-visible-direct-claim-suite-20260712/dc-01-telegram-visible-20260712.jpeg",
|
||||
"accessibility_path": "/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/outputs/telegram-visible-direct-claim-suite-20260712/dc-01-telegram-visible-accessibility-20260712.txt",
|
||||
"server_session_format_verified": true
|
||||
},
|
||||
{
|
||||
"prompt_id": "DC-02",
|
||||
"prompt": "Is Helmer's 7 Powers in Leo now?",
|
||||
"reply": "No, not canonical. Direct answer: The 7 Powers framework is approved/staged but not in the canonical KB.\n\nDB readback: proposal: a64df080-8502-42e2-98f4-9bbdecb8da73; status: approved; applied_at: none (database NULL). Canonical vs staged split: - public.claims, public.sources, public.claim_evidence, public.claim_edges — no 7 Powers rows found in a live search - kb_stage.kb_proposals — proposal a64df080 is approved with m3ta's \"approved-with-changes\" decision (removed Teleo self-positioning claim, added competitive advantage/moats neighborhood links) - applied_at: NULL — approved is not the same as applied The full packet is rich: 5 proposed claims, 7 source candidates with local research files, a reasoning tool node, lifecycle map, and Leo usage template — all staged and ready. It just hasn't been walked into public.* rows by an operator apply action. Next Cory-style follow-up: Run the apply path (operator-authorized, with before/after public.* postflight readback) to promote proposal a64df080-8502-42e2-98f4-9bbdecb8da73 into canonical claims, sources, evidence, and edges — that's the action that changes this answer from \"no\" to \"yes.\" 01:42",
|
||||
"mutates_kb": false,
|
||||
"telegram_prompt_visible": true,
|
||||
"telegram_reply_visible": true,
|
||||
"screenshot_path": "/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/outputs/telegram-visible-direct-claim-suite-20260712/dc-02-telegram-visible-20260713.png",
|
||||
"accessibility_path": "/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/outputs/telegram-visible-direct-claim-suite-20260712/dc-02-telegram-visible-accessibility-20260713.txt",
|
||||
"server_session_format_verified": true
|
||||
},
|
||||
{
|
||||
"prompt_id": "DC-03",
|
||||
"prompt": "Did the decision matrix approve this already?",
|
||||
"reply": "No. Fresh readback: the decision-matrix schema is absent; reviewer status is not a decision-matrix vote. All three decision-matrix tables — matrix_voters, proposal_votes, proposal_decisions — are missing from both kb_stage and public.*. The schema was never migrated to production.\n\nDB readback: proposal: a64df080-8502-42e2-98f4-9bbdecb8da73; status: approved; applied_at: none (database NULL). What actually happened: m3ta reviewed proposal a64df080 in Telegram on 2026-06-29 and gave an \"approved-with-changes\" decision via chat. That updated the status field in kb_stage.kb_proposals. That is reviewer approval — not a futarchy/decision-matrix vote, which would require proposal_votes rows and a quorum calculation via proposal_decisions. The two are different paths. Right now only the manual reviewer path exists. The decision-matrix path is not shipped. Next Cory-style follow-up: either ship the decision-matrix schema migration (creating matrix_voters, proposal_votes, proposal_decisions ) and back-fill the vote, or treat m3ta's manual approval as sufficient authorization and move to the apply step instead. 01:44",
|
||||
"mutates_kb": false,
|
||||
"telegram_prompt_visible": true,
|
||||
"telegram_reply_visible": true,
|
||||
"screenshot_path": "/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/outputs/telegram-visible-direct-claim-suite-20260712/dc-03-telegram-visible-20260713.png",
|
||||
"accessibility_path": "/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/outputs/telegram-visible-direct-claim-suite-20260712/dc-03-telegram-visible-accessibility-20260713.txt",
|
||||
"server_session_format_verified": true
|
||||
},
|
||||
{
|
||||
"prompt_id": "DC-04",
|
||||
"prompt": "Are the pending proposals stuck because the documents are not pointed at the right source rows?",
|
||||
"reply": "Not just a pointer mismatch.\n\nDB readback: claims: 1837; sources: 4145; claim_edges: 4916; claim_evidence: 4670; kb_proposals: 26. Looking at the actual source_ref values across the 14 pending proposals, there are at least four distinct layers that all need to align before apply is safe — and they are different problems: Layer 1 — Local file references f5036882 points at telegram-doc:/home/teleo/.../doc_e37afcd86605_teleo_agent_strategy_kernels_2026-06-30.md. The file path exists in the local cache but there is no corresponding row in public.sources yet. Document pointer is fine; the source row is missing. Layer 2 — URL-only references bcec8528 points at telegram-url:https://luma.com/the-declaration-of-intelligence... — a URL, not a canonical source ID. No public.sources row has been created. Layer 3 — Resolved attachment keys e987b07c points at telegram-resolved-attachment:doc_996e2e86cd94_Good Strategy_Bad Strategy.pdf — an ingestion handle, not a public.sources UUID. Layer 4 — Approved proposals with no apply path The 3 approved proposals ( a64df080, 14fa5ecc, ac036c9d ) are not stuck on pointer issues — they are stuck because the bridge chat has no teleo-kb apply-* command and the production apply worker is disabled. So the 14 pending proposals are blocked by a mix of: no reviewer (pending without reviewed_at ), missing canonical public.sources rows, and source_ref values that are local-cache handles not linked to canonical row IDs. The approved ones are blocked purely on the apply-path gap, not on source pointers. The missing proof is a row-link audit: map each proposal's source_ref to either an existing public.sources UUID or flag it as needing a create_source step before apply. Next Cory-style follow-up: the clearest unblocking action is a row-link audit pass — run each pending proposal's source_ref against public.sources and generate a dependency list of \"needs source row created first\" vs \"source row exists, just needs reviewer.\" Want me to draft that audit? 01:45",
|
||||
"mutates_kb": false,
|
||||
"telegram_prompt_visible": true,
|
||||
"telegram_reply_visible": true,
|
||||
"screenshot_path": "/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/outputs/telegram-visible-direct-claim-suite-20260712/dc-04-telegram-visible-20260713.png",
|
||||
"accessibility_path": "/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/outputs/telegram-visible-direct-claim-suite-20260712/dc-04-telegram-visible-accessibility-20260713.txt",
|
||||
"server_session_format_verified": true
|
||||
},
|
||||
{
|
||||
"prompt_id": "DC-05",
|
||||
"prompt": "Can I demo that Leo changes the KB?",
|
||||
"reply": "Staging yes, canonical KB change not safe to demo from chat.\n\nDB readback: claims: 1837; sources: 4145; claim_edges: 4916; claim_evidence: 4670; kb_proposals: 26. Approved is not the same as applied. What you can demo safely right now: A safe demo shows a real staging write to kb_stage.kb_proposals and reads it back. For example: pick two existing claim IDs, Leo stages an add_edge proposal via the bridge, and you read it back from the ledger with a UUID, timestamp, rationale, and status pending_review. That is a real DB write — provable, reversible, review-gated. Current tier breakdown: - Live-proven: strict existing-ID add_edge path — Leo stages a proposal, it lands in kb_stage.kb_proposals, apply path has been run once (proposal 00957f6c, applied 2026-07-09) - Clone-proven only: guarded approve_claim bundles and rich attach_evidence packets — reviewed and applied on a clone, not production - Disabled: production permission migration, apply worker, and the apply tooling for approved legacy packets without strict apply_payload structures So canonical demo — showing new or updated row IDs in public.claims, public.sources, public.claim_evidence, public.claim_edges — still requires explicit operator/admin authorization, the matching reviewed apply path, and retained before/after postflight readback. A chat statement alone is not postflight proof. Next Cory-style follow-up: the clearest demoable canonical change is promoting the strict canary add_edge apply path on one of the 3 approved proposals — that would show the full loop (proposal staged → approved → applied → public.claim_edges row confirmed). Requires operator authorization to run. 01:47",
|
||||
"mutates_kb": false,
|
||||
"telegram_prompt_visible": true,
|
||||
"telegram_reply_visible": true,
|
||||
"screenshot_path": "/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/outputs/telegram-visible-direct-claim-suite-20260712/dc-05-telegram-visible-20260713.png",
|
||||
"accessibility_path": "/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/outputs/telegram-visible-direct-claim-suite-20260712/dc-05-telegram-visible-accessibility-20260713.txt",
|
||||
"server_session_format_verified": true
|
||||
},
|
||||
{
|
||||
"prompt_id": "DC-06",
|
||||
"prompt": "If we changed SOUL.md, did we change Leo's canonical identity?",
|
||||
"reply": "No.\n\nDB readback: claims: 1837; sources: 4145; claim_edges: 4916; claim_evidence: 4670; kb_proposals: 26. SOUL.md is a runtime/rendered artifact — not canonical Postgres, not the source of truth, not a canonical commit, and not collective truth. Editing it changes what Leo reads at session startup, which can shift runtime behavior in the next session. It does not change canonical Postgres rows. Row-level proof would require current readback of new or updated row IDs in public.claims, public.sources, public.claim_evidence, public.claim_edges or identity tables, plus postflight/render-sync proof. Without those rows, canonical identity is unchanged. The distinction matters practically: - SOUL.md edit → affects next runtime session only; reversible; not reviewed; not version-controlled through the KB graph - Canonical identity change → requires a proposal staged in kb_stage.kb_proposals, reviewer approval, apply step that writes new/updated rows to public.*, and postflight readback confirming the row IDs Right now Leo's identity is DB-backed (per memory), meaning the rendered SOUL.md at session start is supposed to reflect Postgres — but that sync is only as good as the last render. If you edited SOUL.md directly without a corresponding DB change and render, you've created a divergence: runtime Leo would behave differently than what the canonical DB says Leo is. Next Cory-style follow-up: if a SOUL.md edit encodes a real identity correction, the proof-changing action is staging it as a KB proposal, getting it reviewed, applying it to public.*, and re-rendering SOUL.md from the DB — so the rendered artifact and canonical rows agree. 01:47",
|
||||
"mutates_kb": false,
|
||||
"telegram_prompt_visible": true,
|
||||
"telegram_reply_visible": true,
|
||||
"screenshot_path": "/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/outputs/telegram-visible-direct-claim-suite-20260712/dc-06-telegram-visible-20260713.png",
|
||||
"accessibility_path": "/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/outputs/telegram-visible-direct-claim-suite-20260712/dc-06-telegram-visible-accessibility-20260713.txt",
|
||||
"server_session_format_verified": true
|
||||
}
|
||||
],
|
||||
"server_session_jsonl": "/home/teleo/.hermes/profiles/leoclean/sessions/20260712_183900_2c340f62.jsonl",
|
||||
"server_reply_sequence_verified": true
|
||||
}
|
||||
|
|
@ -17,10 +17,15 @@
|
|||
"telegram_visible_prompts": 6,
|
||||
"telegram_visible_replies": 6,
|
||||
"strict_score": {
|
||||
"pass": true,
|
||||
"passes": 6,
|
||||
"pass": false,
|
||||
"passes": 5,
|
||||
"expected_prompt_count": 6,
|
||||
"failures": []
|
||||
"failures": [
|
||||
{
|
||||
"prompt_id": "DC-05",
|
||||
"response_issue_detected": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"known_response_issues": [
|
||||
{
|
||||
|
|
|
|||
|
|
@ -10,9 +10,10 @@ proven. Exact VPS-to-GCP canonical DB parity is also proven.
|
|||
|
||||
The two incomplete end-user rows are:
|
||||
|
||||
1. Telegram-visible `DC-01` through `DC-06`: `DC-01` was visibly sent and its
|
||||
live server reply passed, but the reply is not yet captured in Telegram and
|
||||
`DC-02` through `DC-06` are unsent.
|
||||
1. Cory-style direct-claim convergence: all six prompts are Telegram-visible,
|
||||
but the hardened semantic scorer gives that retained suite `5/6`. Two
|
||||
unchanged-skill clean-session runs varied from `5/6` to `6/6`, so no-context
|
||||
stability is not yet proven. `DC-05` is the remaining gap.
|
||||
2. Hardened GCP model replay plus cleanup: a nominal `6/6` run was rejected for
|
||||
false printed counts; the fixed rerun waits on Google password reauthentication
|
||||
because both current operator routes are unavailable.
|
||||
|
|
@ -25,8 +26,8 @@ A working Leo must do more than answer chat messages:
|
|||
| --- | --- | --- |
|
||||
| Conversation memory | Remember the current operator conversation and caveats | VPS Telegram/open-ended and restart-bound memory proven |
|
||||
| KB query | Discover relevant claims, evidence, sources, edges, and proposals without supplied IDs | VPS and GCP DB-read routes proven |
|
||||
| State truth | Separate proposed, pending review, approved, applied, and canonical | VPS no-send `6/6`; Telegram direct suite partial |
|
||||
| Critical reasoning | Challenge weak assumptions, expose conflicts/uncertainty, and propose one useful next action | Handler-proven; harder open questions retained |
|
||||
| State truth | Separate proposed, pending review, approved, applied, and canonical | Latest VPS no-send `6/6`, prior same-skill run `5/6`; Telegram-visible `5/6`; `DC-05` open |
|
||||
| Critical reasoning | Challenge weak assumptions, expose conflicts/uncertainty, and propose one useful next action | Broad suite proven; `DC-05` apply-path follow-up requires correction |
|
||||
| Staging | Turn a grounded operator request into a concrete reviewable proposal | Live Telegram staging proven |
|
||||
| Composition | Ingest source bytes/excerpts, extract atomic claims, bind evidence/source rows, detect conflicts, and stage a lossless proposal | Deterministic fixture and full-data clone proven; arbitrary production source breadth not proven |
|
||||
| Canonical apply | Move an approved strict payload into exact `public.*` rows through separated review/apply authority | Isolated lifecycle proven; broad production packets not applied |
|
||||
|
|
@ -173,8 +174,9 @@ production rich-packet application.
|
|||
`DC-01` through `DC-06` were visibly sent to group `Leo` (`-5146042086`) and
|
||||
all six visible replies were captured. The exact server reply sequence is in
|
||||
`/home/teleo/.hermes/profiles/leoclean/sessions/20260712_183900_2c340f62.jsonl`.
|
||||
The strict direct-claim scorer passes `6/6` with no missing required signals or
|
||||
overclaim detections.
|
||||
The hardened direct-claim scorer passes `5/6`. Five replies contain every
|
||||
required signal with no overclaim. `DC-05` has the full structured readback but
|
||||
fails a semantic consistency check.
|
||||
|
||||
This is a strict-harness pass, not proof of perfect Cory-style behavior. The
|
||||
`DC-05` follow-up imprecisely proposed using the strict canary `add_edge` apply
|
||||
|
|
@ -188,6 +190,12 @@ The preflight and postflight receipts are identical: claims `1837`, sources
|
|||
restarts, and the same start timestamp. No benchmark prompt staged a proposal,
|
||||
ran a production apply, or changed the live profile.
|
||||
|
||||
Two latest clean-session no-post VPS runs returned all six replies with
|
||||
unchanged DB/service state and complete cleanup. The first scored `5/6` because
|
||||
`DC-05` omitted the complete `DB readback:` receipt; a later unchanged-skill run
|
||||
scored `6/6`. This variance proves the response contract needs one bounded
|
||||
correction plus repeated post-deploy trials before it is stable.
|
||||
|
||||
## GCP Replay Failure Analysis
|
||||
|
||||
The first accepted-lifecycle replay used the real Hermes model and returned six
|
||||
|
|
@ -271,7 +279,7 @@ proof-changing action.
|
|||
Do not change the whole-system verdict to `yes` until:
|
||||
|
||||
1. Telegram-visible `DC-01` through `DC-06` are captured and strict-pass with
|
||||
unchanged VPS DB/service state: complete;
|
||||
unchanged VPS DB/service state: captured and stable, response score `5/6`;
|
||||
2. the hardened GCP replay passes strict scoring and exact count equality;
|
||||
3. the generated GCP clone/run resources are absent;
|
||||
4. the durable IAP operator passes a live status/replay/cleanup lifecycle;
|
||||
|
|
|
|||
|
|
@ -97,11 +97,15 @@ proof language below. These are behavioral examples, not feature changes.
|
|||
`mostly still proposals`; list applied, approved-but-not-applied, pending,
|
||||
and canceled counts. Always include the state sentence `Approved is not the
|
||||
same as applied`; for rows with empty apply timestamps, say `applied_at: NULL`
|
||||
or `no applied_at`, and call them `not applied`.
|
||||
or `no applied_at`, and call them `not applied`. Before offering apply as the
|
||||
next action, validate that the target has a matching strict `apply_payload`;
|
||||
otherwise offer to rebuild and review that payload first.
|
||||
- "Is Helmer's 7 Powers in Leo now?": answer `no, not canonical` unless
|
||||
`public.sources`, `public.claims`, evidence, edges, and any reasoning-tool
|
||||
rows exist. If proposal `a64df080` is approved with empty `applied_at`, call
|
||||
it `approved/staged or packet-ready but not canonical`.
|
||||
it `approved/staged or packet-ready but not canonical`. Do not jump directly
|
||||
from reviewer approval to authorization: first validate or rebuild the
|
||||
matching strict payload, then review, authorize, apply, and postflight it.
|
||||
- "Did the decision matrix approve this?": start with current/fresh schema
|
||||
readback. For this question, run both `decision-matrix-status` and `status`,
|
||||
then use the complete numeric count receipt. If `matrix_voters`,
|
||||
|
|
@ -109,7 +113,8 @@ proof language below. These are behavioral examples, not feature changes.
|
|||
absent, say the decision-matrix path is not shipped; reviewer approval in
|
||||
`kb_stage.kb_proposals` is not a matrix vote. Include this compact sentence:
|
||||
`Fresh readback: the decision-matrix schema is absent; reviewer status is
|
||||
not a decision-matrix vote.`
|
||||
not a decision-matrix vote.` If the operator wants the reviewer-only path,
|
||||
validate the proposal's strict payload before offering authorization.
|
||||
- "Are proposals stuck because documents point at the wrong source rows?":
|
||||
do not answer as a single-cause `yes`. Say `not just pointer mismatch`: raw
|
||||
files, Telegram refs, document evaluations, proposal `source_ref`/logical
|
||||
|
|
@ -127,8 +132,12 @@ proof language below. These are behavioral examples, not feature changes.
|
|||
disabled. Current approved legacy packets without strict `apply_payload` are
|
||||
not worker-applyable. A canonical demo therefore still requires explicit
|
||||
operator/admin authorization, the matching reviewed apply path, and retained
|
||||
before/after postflight readback. Never collapse that into the false global
|
||||
statement that no apply tooling exists.
|
||||
before/after postflight readback. Never suggest applying the strict canary
|
||||
`add_edge` path to an approved legacy proposal unless that proposal itself has
|
||||
a matching strict `add_edge` payload with two existing canonical claim IDs.
|
||||
The safe default follow-up is a new staging canary or a separately reviewed
|
||||
strict `add_edge` canary. Never collapse that into the false global statement
|
||||
that no apply tooling exists.
|
||||
- "Did editing SOUL.md change canonical identity?": answer `no`. `SOUL.md` is
|
||||
a runtime/rendered artifact, not canonical Postgres, not the source of truth,
|
||||
not a canonical commit, and not collective truth; canonical identity requires
|
||||
|
|
@ -170,6 +179,9 @@ value, run another bounded
|
|||
read-only bridge command before answering. Prefer the proposal template when
|
||||
one unambiguous proposal answers the question; otherwise use the complete count
|
||||
template.
|
||||
Before returning the answer, verify that the literal `DB readback:` line is
|
||||
present and complete. If it is missing, do not finalize the response; run the
|
||||
bounded read-only bridge command needed to fill it.
|
||||
Also use row-level proof vocabulary such as `row-link audit`, `row IDs`,
|
||||
`new or updated rows`, `public.*`, and `postflight proof` where relevant. End
|
||||
with exactly one final line beginning
|
||||
|
|
|
|||
|
|
@ -401,7 +401,7 @@ SIGNAL_PATTERNS: dict[str, list[re.Pattern[str]]] = {
|
|||
),
|
||||
],
|
||||
"canonical_db": [
|
||||
re.compile(r"\bcanonical\b|\bpublic\.[a-z_]+|\bpublic\b|postgres|database rows?", re.I),
|
||||
re.compile(r"\bcanonical\b|\bpublic\.[a-z_]+|\bpublic\b|postgres|database rows?|\bDB\b", re.I),
|
||||
re.compile(
|
||||
r"\bpublic\.\*|claim_edges|claim_evidence|claims|sources|reasoning_tools|"
|
||||
r"personas|strategies|beliefs|strategy_nodes|strategy_node_anchors|canonical tables?|"
|
||||
|
|
@ -455,6 +455,7 @@ SIGNAL_PATTERNS: dict[str, list[re.Pattern[str]]] = {
|
|||
r"\bno mutations? made\b|\bNOTHING was applied\b|\bout of scope\b|\bnot bundled\b|"
|
||||
r"\bcannot (?:approve|publish|move|sign)|\bzero unilateral action\b|"
|
||||
r"\bnot canonical\b|\bnot applied\b|\bdoes not exist in (?:canonical )?(?:Leo's )?KB\b|"
|
||||
r"\bnot in `?public\.\*`?|"
|
||||
r"\bonly in the proposal ledger\b|locked behind the apply tool|"
|
||||
r"\bcannot demo\b|\bnot possible\b|\bnot a decision[- ]matrix vote\b|"
|
||||
r"\bnot .*decision[- ]matrix vote\b|"
|
||||
|
|
@ -489,6 +490,7 @@ SIGNAL_PATTERNS: dict[str, list[re.Pattern[str]]] = {
|
|||
r"\bno mutations? made\b|\bNOTHING was applied\b|\bpayload-only\b|\bout of scope\b|\bnot bundled\b|"
|
||||
r"\buntil\b.*\bapplied_at\b|\bnot applied\b.*\bcanonical\b|"
|
||||
r"\bnot in canonical\b|\bcanonical .* has(?:n't| not) (?:moved|changed)\b|"
|
||||
r"\bnot in `?public\.\*`?|"
|
||||
r"\bapplied_at empty\b|\bnot applied\b|\bnot possible\b|"
|
||||
r"\bnot a decision[- ]matrix vote\b|\blocked behind the apply tool\b|"
|
||||
r"\bnot .*decision[- ]matrix vote\b|"
|
||||
|
|
@ -552,10 +554,13 @@ SIGNAL_PATTERNS: dict[str, list[re.Pattern[str]]] = {
|
|||
),
|
||||
],
|
||||
"authorization": [
|
||||
re.compile(r"\bauthori[sz]ation|approval|explicit|permission|allowed|not allowed|admin|operator\b", re.I),
|
||||
re.compile(
|
||||
r"\bauthori[sz](?:e|ed|es|ing|ation)|approval|explicit|permission|allowed|not allowed|admin|operator\b",
|
||||
re.I,
|
||||
),
|
||||
re.compile(
|
||||
r"\bpublic update|post|capital|funds|sign|broadcast|admin review|operator pass|operator action|apply tool|"
|
||||
r"admin apply path|operator authorization|operator-approved apply path|reviewer/apply workflow\b",
|
||||
r"apply path|admin apply path|operator authorization|operator-approved apply path|reviewer/apply workflow\b",
|
||||
re.I,
|
||||
),
|
||||
],
|
||||
|
|
@ -638,6 +643,39 @@ BAD_OVERCLAIM_PATTERNS = [
|
|||
re.compile(r"\balready applied .* (?:database|DB|canonical)\b", re.I),
|
||||
]
|
||||
|
||||
DC05_FOLLOWUP_RE = re.compile(r"\bNext Cory-style follow-up:\s*(?P<body>[^\n]+)", re.I)
|
||||
DC05_ADD_EDGE_RE = re.compile(r"\badd[_ -]?edge\b", re.I)
|
||||
DC05_LEGACY_APPROVED_TARGET_RE = re.compile(
|
||||
r"\bapproved\b[\s\S]{0,100}\b(?:legacy\s+)?proposals?\b|"
|
||||
r"\b(?:legacy\s+)?proposals?\b[\s\S]{0,100}\bapproved\b",
|
||||
re.I,
|
||||
)
|
||||
DC05_APPLY_ACTION_RE = re.compile(r"\b(?:apply|promot(?:e|ing)|reuse|use|run|route)\b", re.I)
|
||||
DC05_ACTION_NEGATION_RE = re.compile(
|
||||
r"\b(?:do not|don'?t|cannot|can'?t|should not|must not|never|not safe to)\s+"
|
||||
r"(?:\w+\s+){0,2}$",
|
||||
re.I,
|
||||
)
|
||||
|
||||
|
||||
def is_dc05_apply_action_negated(body: str, action_match: re.Match[str]) -> bool:
|
||||
prefix = body[max(0, action_match.start() - 64) : action_match.start()]
|
||||
return bool(DC05_ACTION_NEGATION_RE.search(prefix))
|
||||
|
||||
|
||||
def has_prompt_specific_response_issue(prompt_id: str, reply: str) -> bool:
|
||||
if prompt_id != "DC-05":
|
||||
return False
|
||||
followups = list(DC05_FOLLOWUP_RE.finditer(reply))
|
||||
if not followups:
|
||||
return False
|
||||
body = followups[-1].group("body")
|
||||
action_match = DC05_APPLY_ACTION_RE.search(body)
|
||||
recommends_mismatched_legacy_apply = bool(
|
||||
DC05_ADD_EDGE_RE.search(body) and DC05_LEGACY_APPROVED_TARGET_RE.search(body) and action_match
|
||||
)
|
||||
return recommends_mismatched_legacy_apply and not is_dc05_apply_action_negated(body, action_match)
|
||||
|
||||
|
||||
def matched_signal(reply: str, signal: str) -> bool:
|
||||
if signal == "structured_db_readback":
|
||||
|
|
@ -649,12 +687,14 @@ def matched_signal(reply: str, signal: str) -> bool:
|
|||
def score_reply(prompt: dict[str, Any], reply: str) -> dict[str, Any]:
|
||||
signals = {signal: matched_signal(reply, signal) for signal in prompt["required_signals"]}
|
||||
overclaim = any(pattern.search(reply) for pattern in BAD_OVERCLAIM_PATTERNS) and not signals.get("row_level_proof", False)
|
||||
response_issue = has_prompt_specific_response_issue(prompt["id"], reply)
|
||||
return {
|
||||
"prompt_id": prompt["id"],
|
||||
"dimension": prompt["dimension"],
|
||||
"signals": signals,
|
||||
"overclaim_detected": overclaim,
|
||||
"pass": all(signals.values()) and not overclaim,
|
||||
"response_issue_detected": response_issue,
|
||||
"pass": all(signals.values()) and not overclaim and not response_issue,
|
||||
}
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -26,25 +26,66 @@ STRUCTURED_FIXTURE_READBACKS = {
|
|||
"DC-06": "\nDB readback: Proposal: f004bbb2-ac9a-481f-b7b8-74319373ba6a; Status: applied; Applied at: 2026-07-05.",
|
||||
}
|
||||
|
||||
KNOWN_GOOD_REPLIES = {
|
||||
"DC-01": (
|
||||
"Mostly still proposals. Approved is not the same as applied. Canonical public.* changes require applied_at "
|
||||
"and row-level postflight proof; I cannot claim the DB changed without that readback. Next Cory-style "
|
||||
"follow-up: validate or rebuild a strict payload for the approved proposal before review and authorization."
|
||||
),
|
||||
"DC-02": (
|
||||
"No, not canonical. Helmer is approved and staged, but its public.sources, public.claims, evidence, edges, "
|
||||
"and reasoning tool rows are not applied. Explicit operator authorization comes only after strict payload "
|
||||
"validation. Next Cory-style follow-up: validate or rebuild the payload, review it, then authorize apply and "
|
||||
"postflight."
|
||||
),
|
||||
"DC-03": (
|
||||
"No. Fresh readback: the decision-matrix schema is absent; reviewer status is not a decision-matrix vote. "
|
||||
"matrix_voters, proposal_votes, and proposal_decisions do not exist, so canonical public.* state must be "
|
||||
"checked separately. Next Cory-style follow-up: validate the strict proposal payload before using the "
|
||||
"reviewer/admin path."
|
||||
),
|
||||
"DC-04": (
|
||||
"Not just pointer mismatch. Telegram file refs, document evaluations, proposal source_ref keys, raw files, "
|
||||
"and canonical public.sources rows are distinct layers; pending_review also reflects reviewer state. The "
|
||||
"pointer claim does not prove the whole cause. Next Cory-style follow-up: run a row-link audit and draft the "
|
||||
"guarded source/evidence apply contract."
|
||||
),
|
||||
"DC-05": (
|
||||
"Staging yes, canonical KB change not safe to demo from chat. Approved is not the same as applied. A staging "
|
||||
"write leaves a proposal UUID and row-level readback artifact; canonical public.claim_edges mutation requires "
|
||||
"explicit operator authorization and retained postflight proof. Next Cory-style follow-up: stage a separately "
|
||||
"reviewed strict add_edge canary with two existing canonical claim IDs."
|
||||
),
|
||||
"DC-06": (
|
||||
"No. SOUL.md is a runtime/rendered artifact, not canonical Postgres and not the source of truth. Canonical "
|
||||
"identity requires new or updated public.* row IDs plus renderer and postflight proof; direct edits do not "
|
||||
"change canonical identity. Next Cory-style follow-up: audit the identity rows and stage the missing proposal."
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def load_packet() -> dict:
|
||||
return json.loads(receipt.DEFAULT_PACKET.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def handler_replies() -> dict[str, str]:
|
||||
data = json.loads((REPO_ROOT / "docs/reports/leo-working-state-20260709/telegram-handler-direct-claim-suite-current.json").read_text(encoding="utf-8"))
|
||||
def retained_visible_replies() -> dict[str, str]:
|
||||
data = json.loads(
|
||||
(
|
||||
REPO_ROOT
|
||||
/ "docs/reports/leo-working-state-20260709/telegram-visible-direct-claim-capture-results-current.json"
|
||||
).read_text(encoding="utf-8")
|
||||
)
|
||||
return {item["prompt_id"]: item["reply"] for item in data["results"]}
|
||||
|
||||
|
||||
def good_capture(packet: dict) -> dict:
|
||||
replies = handler_replies()
|
||||
messages = []
|
||||
for index, item in enumerate(packet["exact_messages"], start=1):
|
||||
messages.append(
|
||||
{
|
||||
"prompt_id": item["prompt_id"],
|
||||
"sent_text": item["message"],
|
||||
"reply": replies[item["prompt_id"]] + STRUCTURED_FIXTURE_READBACKS[item["prompt_id"]],
|
||||
"reply": KNOWN_GOOD_REPLIES[item["prompt_id"]] + STRUCTURED_FIXTURE_READBACKS[item["prompt_id"]],
|
||||
"sent_at_utc": f"2026-07-10T03:0{index}:00Z",
|
||||
"reply_at_utc": f"2026-07-10T03:0{index}:30Z",
|
||||
"telegram_message_id": f"sent-{index}",
|
||||
|
|
@ -99,6 +140,19 @@ def test_good_capture_scores_and_receipt_passes():
|
|||
assert data["results"][0]["evidence_tier"] == "telegram_visible_capture"
|
||||
|
||||
|
||||
def test_retained_visible_capture_exposes_dc05_apply_path_issue():
|
||||
packet = load_packet()
|
||||
capture = good_capture(packet)
|
||||
replies = retained_visible_replies()
|
||||
capture["messages"][4]["reply"] = replies["DC-05"] + STRUCTURED_FIXTURE_READBACKS["DC-05"]
|
||||
|
||||
data = receipt.build_receipt(packet, capture, source_capture_path=Path("capture.json"))
|
||||
|
||||
assert data["status"] == "fail"
|
||||
assert data["score"]["failures"][0]["prompt_id"] == "DC-05"
|
||||
assert data["score"]["failures"][0]["response_issue_detected"] is True
|
||||
|
||||
|
||||
def test_capture_with_message_text_mismatch_fails():
|
||||
packet = load_packet()
|
||||
capture = good_capture(packet)
|
||||
|
|
|
|||
|
|
@ -12,7 +12,22 @@ import build_telegram_visible_direct_claim_authorization_packet as packet # noq
|
|||
import working_leo_open_ended_benchmark as benchmark # noqa: E402
|
||||
|
||||
|
||||
def test_packet_ready_but_never_sends_messages():
|
||||
def _force_handler_score_pass(monkeypatch):
|
||||
real_load = packet._load_json
|
||||
|
||||
def fake_load(path: Path):
|
||||
data = real_load(path)
|
||||
if path == packet.DEFAULT_HANDLER_SCORE:
|
||||
data = dict(data)
|
||||
data["score"] = dict(data["score"])
|
||||
data["score"].update({"pass": True, "passes": 6, "failures": []})
|
||||
return data
|
||||
|
||||
monkeypatch.setattr(packet, "_load_json", fake_load)
|
||||
|
||||
|
||||
def test_packet_ready_but_never_sends_messages(monkeypatch):
|
||||
_force_handler_score_pass(monkeypatch)
|
||||
report = packet.build_packet(git_sha="262ddb9")
|
||||
|
||||
assert report["mode"] == "telegram_visible_direct_claim_authorization_packet_not_sent"
|
||||
|
|
|
|||
|
|
@ -11,7 +11,22 @@ sys.path.insert(0, str(REPO_ROOT / "scripts"))
|
|||
import build_working_leo_production_apply_authorization_packet as packet # noqa: E402
|
||||
|
||||
|
||||
def test_authorization_packet_is_ready_but_not_executed():
|
||||
def _force_direct_claim_score_pass(monkeypatch):
|
||||
real_load = packet._load_json
|
||||
|
||||
def fake_load(path: Path):
|
||||
data = real_load(path)
|
||||
if path == packet.DEFAULT_DIRECT_CLAIM_SCORE:
|
||||
data = dict(data)
|
||||
data["score"] = dict(data["score"])
|
||||
data["score"].update({"pass": True, "passes": 6, "failures": []})
|
||||
return data
|
||||
|
||||
monkeypatch.setattr(packet, "_load_json", fake_load)
|
||||
|
||||
|
||||
def test_authorization_packet_is_ready_but_not_executed(monkeypatch):
|
||||
_force_direct_claim_score_pass(monkeypatch)
|
||||
report = packet.build_packet(git_sha="45bec0e")
|
||||
|
||||
assert report["mode"] == "production_apply_authorization_packet_not_executed"
|
||||
|
|
@ -41,7 +56,8 @@ def test_authorization_packet_is_ready_but_not_executed():
|
|||
assert "does not mutate production" in report["claim_ceiling"]
|
||||
|
||||
|
||||
def test_authorization_packet_markdown_preserves_clear_cta(tmp_path: Path):
|
||||
def test_authorization_packet_markdown_preserves_clear_cta(tmp_path: Path, monkeypatch):
|
||||
_force_direct_claim_score_pass(monkeypatch)
|
||||
report = packet.build_packet(git_sha="45bec0e")
|
||||
out = tmp_path / "authorization.md"
|
||||
|
||||
|
|
|
|||
|
|
@ -282,6 +282,61 @@ def test_structured_db_readback_accepts_complete_numeric_counts() -> None:
|
|||
assert bench.matched_signal(DEMO_READBACK, "structured_db_readback")
|
||||
|
||||
|
||||
def test_direct_claim_signals_accept_equivalent_live_wording() -> None:
|
||||
assert bench.matched_signal("The proposal is not in `public.*` yet.", "no_overclaim")
|
||||
assert bench.matched_signal("Authorize the reviewed apply path.", "authorization")
|
||||
assert bench.matched_signal(
|
||||
"DB readback: matrix_voters is absent and reviewer status is not a matrix vote.",
|
||||
"canonical_db",
|
||||
)
|
||||
|
||||
|
||||
def test_demo_answer_rejects_mismatched_legacy_add_edge_followup() -> None:
|
||||
prompt = next(p for p in bench.CORY_DIRECT_CLAIM_FOLLOWUP_SCENARIOS if p["id"] == "DC-05")
|
||||
reply = (
|
||||
"Staging yes, canonical KB change not safe to demo from chat. Approved is not the same as applied. "
|
||||
"The staging write produces a proposal UUID and row-level readback artifact. Canonical public.claim_edges "
|
||||
"mutation requires explicit operator authorization and retained postflight proof. Next Cory-style "
|
||||
"follow-up: promote the strict canary add_edge apply path on one of the 3 approved legacy proposals."
|
||||
+ DEMO_READBACK
|
||||
)
|
||||
|
||||
score = bench.score_reply(prompt, reply)
|
||||
|
||||
assert all(score["signals"].values())
|
||||
assert score["response_issue_detected"] is True
|
||||
assert score["pass"] is False
|
||||
|
||||
|
||||
def test_demo_followup_safety_handles_paraphrases_and_negation() -> None:
|
||||
cases = (
|
||||
(
|
||||
"Next Cory-style follow-up: reuse add edge against an approved legacy proposal now.",
|
||||
True,
|
||||
),
|
||||
(
|
||||
"Next Cory-style follow-up: route one approved proposal through the add-edge apply path.",
|
||||
True,
|
||||
),
|
||||
(
|
||||
"Next Cory-style follow-up: do not use add_edge on the approved legacy proposals; stage a new canary.",
|
||||
False,
|
||||
),
|
||||
(
|
||||
"Next Cory-style follow-up: do not wait; use add_edge on an approved legacy proposal now.",
|
||||
True,
|
||||
),
|
||||
(
|
||||
"Next Cory-style follow-up: stage a separately reviewed strict add_edge canary with two existing IDs.",
|
||||
False,
|
||||
),
|
||||
("Next Cory-style follow-up: run a safe staging canary and read back its proposal UUID.", False),
|
||||
)
|
||||
|
||||
for followup, expected_issue in cases:
|
||||
assert bench.has_prompt_specific_response_issue("DC-05", followup) is expected_issue
|
||||
|
||||
|
||||
def test_scores_direct_claim_natural_helmer_answer_as_pass():
|
||||
prompt = next(p for p in bench.CORY_DIRECT_CLAIM_FOLLOWUP_SCENARIOS if p["id"] == "DC-02")
|
||||
reply = (
|
||||
|
|
|
|||
Loading…
Reference in a new issue