diff --git a/docs/reports/leo-working-state-20260709/cory-expected-working-leo-outcomes-20260709.md b/docs/reports/leo-working-state-20260709/cory-expected-working-leo-outcomes-20260709.md index b318f2b..cf518ad 100644 --- a/docs/reports/leo-working-state-20260709/cory-expected-working-leo-outcomes-20260709.md +++ b/docs/reports/leo-working-state-20260709/cory-expected-working-leo-outcomes-20260709.md @@ -90,7 +90,7 @@ The benchmark for Cory-working Leo should pass these checks: 6. Caveat retention: passed for the strict canary approval note; old freeform caveat retention remains part of the claim/source creation contract. 7. Old approval handling: partially passed; old freeform approvals are now normalized into row-level requirements and explicitly not silently applied. 8. Runtime stability: passed; service stayed active/running with `MainPID=3252143`, `NRestarts=0`. -9. Broad Cory-style outcomes: specified in the open-ended benchmark as nine disposable-clone/sandbox-first scenarios. The mixed-evidence sandbox score now passes `14/14` using retained live Telegram evidence for `OE-01` through `OE-05` and fixture answers for `CS-01` through `CS-09`; the CS rows are not live Telegram proof. +9. Broad Cory-style outcomes and no-context direct claims: specified in the open-ended benchmark as nine disposable-clone/sandbox-first outcome scenarios plus six direct-claim/follow-up scenarios. The mixed-evidence sandbox score now passes `20/20` using retained live Telegram evidence for `OE-01` through `OE-05`, fixture answers for `CS-01` through `CS-09`, and direct-claim expected-answer/follow-up fixtures for `DC-01` through `DC-06`; the CS/DC rows are not live Telegram proof. ## Scope Boundaries diff --git a/docs/reports/leo-working-state-20260709/current-truth-index.md b/docs/reports/leo-working-state-20260709/current-truth-index.md index 2d600d9..fe8795e 100644 --- a/docs/reports/leo-working-state-20260709/current-truth-index.md +++ b/docs/reports/leo-working-state-20260709/current-truth-index.md @@ -8,7 +8,7 @@ Use this file before making status claims. Prefer fresh VPS/GCP readbacks when c - Cory expected outcomes: `docs/reports/leo-working-state-20260709/cory-expected-working-leo-outcomes-20260709.md` - Current state: `docs/reports/leo-working-state-20260709/working-leo-current-state-20260709.md` - Execution plan: `docs/reports/leo-working-state-20260709/working-leo-execution-plan-current.md` -- Open-ended benchmark spec: `docs/reports/leo-working-state-20260709/working-leo-open-ended-benchmark-spec.json` (5 live-safe open prompts plus 9 broader Cory-style outcome scenarios for disposable clone/sandbox first) +- Open-ended benchmark spec: `docs/reports/leo-working-state-20260709/working-leo-open-ended-benchmark-spec.json` (5 live-safe open prompts, 9 broader Cory-style outcome scenarios, and 6 no-context direct-claim follow-up prompts for disposable clone/sandbox first) - Open-ended live Telegram canary: `docs/reports/leo-working-state-20260709/telegram-live-open-ended-cory-style-20260709.md` - Open-ended live Telegram canary JSON: `docs/reports/leo-working-state-20260709/telegram-live-open-ended-cory-style-20260709.json` - Retained Telegram benchmark score for the open-ended canary: `docs/reports/leo-working-state-20260709/telegram-live-open-ended-benchmark-score-current.md` @@ -16,8 +16,8 @@ Use this file before making status claims. Prefer fresh VPS/GCP readbacks when c - Full live-safe open-ended Telegram suite report: `docs/reports/leo-working-state-20260709/telegram-live-open-ended-suite-current.md` - Full live-safe open-ended Telegram suite score: `docs/reports/leo-working-state-20260709/telegram-live-open-ended-suite-score-current.md` - Full live-safe open-ended Telegram suite score JSON: `docs/reports/leo-working-state-20260709/telegram-live-open-ended-suite-score-current.json` -- Full Cory-style outcome sandbox results: `docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-results-current.md` -- Full Cory-style outcome sandbox score: `docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-score-current.md` +- Full Cory-style outcome/direct-claim sandbox results: `docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-results-current.md` +- Full Cory-style outcome/direct-claim sandbox score: `docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-score-current.md` - Full Cory-style outcome sandbox score JSON: `docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-score-current.json` - Fable onboarding prompt: `docs/reports/leo-working-state-20260709/fable-leo-teleo-onboarding.md` - Operator surface map: `docs/reports/leo-working-state-20260709/operator-surface-map.md` @@ -69,6 +69,8 @@ Use this file before making status claims. Prefer fresh VPS/GCP readbacks when c - Working Leo integrated clone rehearsal log: `docs/reports/leo-working-state-20260709/working-leo-integrated-clone-rehearsal-current.log` - Working Leo authorized-apply readiness verifier: `docs/reports/leo-working-state-20260709/working-leo-apply-readiness-current.md` - Working Leo authorized-apply readiness verifier JSON: `docs/reports/leo-working-state-20260709/working-leo-apply-readiness-current.json` +- Working Leo authorized apply operator runbook, not executed: `docs/reports/leo-working-state-20260709/working-leo-authorized-apply-runbook-current.md` +- Working Leo authorized apply operator runbook JSON: `docs/reports/leo-working-state-20260709/working-leo-authorized-apply-runbook-current.json` - Helmer live rollback log: `docs/reports/leo-working-state-20260709/helmer-7powers-live-preflight-rollback-current.log` - Helmer clone commit/delete rollback log: `docs/reports/leo-working-state-20260709/helmer-7powers-clone-commit-delete-rollback-current.log` - Helmer ledger clone commit/delete rollback log: `docs/reports/leo-working-state-20260709/helmer-7powers-ledger-clone-commit-delete-rollback-current.log` @@ -99,7 +101,7 @@ Use this file before making status claims. Prefer fresh VPS/GCP readbacks when c - VPS Leo Telegram memory, KB audit, and staged-write canaries are live-proven. - VPS Leo live-safe open-ended Telegram suite is live-proven and retained-benchmark-scored for `OE-01` through `OE-05`: full selected coverage, `5/5` scored prompts passing, no overclaim, and no marker/staging/canonical KB rows created by the suite. -- The broader sandbox-first Cory-style outcome benchmark now has a full-catalog mixed-evidence score: real retained live Telegram evidence for `OE-01` through `OE-05`, sandbox fixture replies for `CS-01` through `CS-09`, `14/14` passing. This is not a live Telegram run for the CS prompts. +- The broader sandbox-first Cory-style benchmark now has a full-catalog mixed-evidence score: real retained live Telegram evidence for `OE-01` through `OE-05`, sandbox fixture replies for `CS-01` through `CS-09`, and no-context direct-claim expected-answer/follow-up fixtures for `DC-01` through `DC-06`, `20/20` passing. This is not a live Telegram run for the CS/DC prompts. - A strict canonical `add_edge` apply canary is live-proven. - Helmer 7 Powers is full-coverage packet and ledger clone-proven, not production-applied. - Mapped rich approved proposal packet is clone/rehearsal-proven for the supported subset, not production-applied. @@ -108,6 +110,7 @@ Use this file before making status claims. Prefer fresh VPS/GCP readbacks when c - The governance/concept-map schema companion write is clone-proven and reversible: after the mapped base packet exists, it creates minimal concept-map and claim-to-governance link tables, then inserts `1` concept map, `1` concept link, and `2` governance links in a disposable clone; production stayed unchanged. - The full integrated packet set is now clone-proven in production dependency order: mapped base, cross-surface anchor, Rio strategy context, governance/concept schema, and Helmer 7 Powers applied together in `teleo_integrated_rehearsal_20260710`, then rolled back in reverse order; production stayed unchanged and the disposable clone/temp files were removed. - The authorized-apply readiness verifier now passes: packet files exist and hash, apply/rollback order matches the manifest, expected counts match, clone apply/rollback markers are ordered, clone cleanup is proven, service stayed stable, and `production_apply_executed=false`. +- The authorized apply operator runbook is generated and ready-not-executed: it gives preflight/apply/postflight/rollback command templates in manifest order, requires `TELEO_DATABASE_URL`, and still does not authorize or execute production SQL. - DB-first identity rendering is now documented from the July 9 PDFs plus VPS readback: canonical identity rows feed rendered `SOUL.md`, direct SOUL edits are not canonical DB updates, and no active renderer/timer/hook is proven in the current VPS readback. - The decision-matrix approval schema is still a target design on the current VPS: `kb_stage.matrix_voters`, `kb_stage.proposal_votes`, and `kb_stage.proposal_decisions` were not present in live schema readback. - Pending/approved document proposals are real `kb_stage.kb_proposals` rows with `source_ref`s, but inspected `source_ref`s did not directly match `public.sources`; they still need review/apply mapping into canonical sources/evidence/edges. diff --git a/docs/reports/leo-working-state-20260709/working-leo-authorized-apply-runbook-current.json b/docs/reports/leo-working-state-20260709/working-leo-authorized-apply-runbook-current.json new file mode 100644 index 0000000..6c6844b --- /dev/null +++ b/docs/reports/leo-working-state-20260709/working-leo-authorized-apply-runbook-current.json @@ -0,0 +1,298 @@ +{ + "checks": { + "all_apply_and_rollback_sql_have_transactions": true, + "all_phase_files_exist": true, + "all_phase_files_non_empty": true, + "apply_order_matches_manifest": true, + "readiness_passes": true, + "readiness_says_not_authorized": true, + "readiness_says_not_executed": true, + "rollback_order_matches_manifest": true + }, + "claim_ceiling": "This runbook is proof that the authorized apply sequence is explicit and ordered. It does not execute SQL, does not mutate production, and does not replace explicit production apply authorization.", + "expected_after_apply_counts": { + "claim_concept_map_links": 1, + "claim_edges": 18, + "claim_evidence": 34, + "claim_governance_gate_links": 2, + "claims": 12, + "concept_maps": 1, + "cross_surface_anchor_updates": 1, + "helmer_ledger_rows_marked_applied": 1, + "reasoning_tools": 1, + "rio_strategy_anchors": 5, + "rio_strategy_nodes": 2, + "sources": 28 + }, + "generated_at_utc": "2026-07-09T23:12:00.028604+00:00", + "mode": "operator_runbook_generation_only", + "operator_environment": { + "operator_commands_run_sql_if_executed": true, + "required_env": [ + "TELEO_DATABASE_URL" + ], + "runbook_generation_runs_sql": false, + "runbook_only": true + }, + "phases": { + "apply_order": [ + { + "bytes": 38892, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/production-apply-authorized-commit.sql", + "exists": true, + "file": "production-apply-authorized-commit.sql", + "has_transaction_boundary": true, + "ok": true, + "packet_id": "mapped-rich-base", + "path": "docs/reports/leo-working-state-20260709/production-apply-authorized-commit.sql", + "phase": "apply", + "sha256": "d7c9d06e77571b9751a94fdf49715c9997421f071dc840e65247ea1a0d5c8088" + }, + { + "bytes": 1366, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/cross-surface-apply-authorized-commit.sql", + "exists": true, + "file": "cross-surface-apply-authorized-commit.sql", + "has_transaction_boundary": true, + "ok": true, + "packet_id": "cross-surface-anchor", + "path": "docs/reports/leo-working-state-20260709/cross-surface-apply-authorized-commit.sql", + "phase": "apply", + "sha256": "381c6dcf178a419cb588d3e57bb63a1f794a03e9d5ff4a9ff3c2d9735a1bdf3f" + }, + { + "bytes": 6579, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/rio-strategy-apply-authorized-commit.sql", + "exists": true, + "file": "rio-strategy-apply-authorized-commit.sql", + "has_transaction_boundary": true, + "ok": true, + "packet_id": "rio-strategy-context", + "path": "docs/reports/leo-working-state-20260709/rio-strategy-apply-authorized-commit.sql", + "phase": "apply", + "sha256": "0a219df35db7ea3379c8b232449b5a6b8bd3a0164dd26ee393e55fa997b31463" + }, + { + "bytes": 6822, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/governance-concept-apply-authorized-commit.sql", + "exists": true, + "file": "governance-concept-apply-authorized-commit.sql", + "has_transaction_boundary": true, + "ok": true, + "packet_id": "governance-concept-schema", + "path": "docs/reports/leo-working-state-20260709/governance-concept-apply-authorized-commit.sql", + "phase": "apply", + "sha256": "63b1ea56180ddd2308d4a4ca0e2d229ba3fe6cce494d5339339a6f62f1962bba" + }, + { + "bytes": 42770, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/helmer-7powers-production-apply-authorized-commit.sql", + "exists": true, + "file": "helmer-7powers-production-apply-authorized-commit.sql", + "has_transaction_boundary": true, + "ok": true, + "packet_id": "helmer-7powers", + "path": "docs/reports/leo-working-state-20260709/helmer-7powers-production-apply-authorized-commit.sql", + "phase": "apply", + "sha256": "9264baa4249432dcdc96510e67cc694bb5f2b2e485b63a3bfee959d31a89104d" + } + ], + "postflight_order": [ + { + "bytes": 11147, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/production-postflight.sql", + "exists": true, + "file": "production-postflight.sql", + "has_transaction_boundary": false, + "ok": true, + "packet_id": "mapped-rich-base", + "path": "docs/reports/leo-working-state-20260709/production-postflight.sql", + "phase": "postflight", + "sha256": "4dec76f0502b08ea2ee3be934f86c6e3557800eda26095d470c7333655ac2287" + }, + { + "bytes": 802, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/cross-surface-postflight.sql", + "exists": true, + "file": "cross-surface-postflight.sql", + "has_transaction_boundary": false, + "ok": true, + "packet_id": "cross-surface-anchor", + "path": "docs/reports/leo-working-state-20260709/cross-surface-postflight.sql", + "phase": "postflight", + "sha256": "4d9a8adafe5be6777badb714aafdaaa1b502e80695bbe786eff20854d3ae0856" + }, + { + "bytes": 2062, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/rio-strategy-postflight.sql", + "exists": true, + "file": "rio-strategy-postflight.sql", + "has_transaction_boundary": false, + "ok": true, + "packet_id": "rio-strategy-context", + "path": "docs/reports/leo-working-state-20260709/rio-strategy-postflight.sql", + "phase": "postflight", + "sha256": "135fc640bdf7e6f7239a383be0512449817e0f14d84e8078ef0e0e313112d446" + }, + { + "bytes": 1408, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/governance-concept-postflight.sql", + "exists": true, + "file": "governance-concept-postflight.sql", + "has_transaction_boundary": false, + "ok": true, + "packet_id": "governance-concept-schema", + "path": "docs/reports/leo-working-state-20260709/governance-concept-postflight.sql", + "phase": "postflight", + "sha256": "166a78048433a95e1d5a09dba795b4ec1a1ecfc3f1a50fc1af1d3ba87c50dfcc" + }, + { + "bytes": 10217, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/helmer-7powers-production-postflight.sql", + "exists": true, + "file": "helmer-7powers-production-postflight.sql", + "has_transaction_boundary": false, + "ok": true, + "packet_id": "helmer-7powers", + "path": "docs/reports/leo-working-state-20260709/helmer-7powers-production-postflight.sql", + "phase": "postflight", + "sha256": "dff8164c75f5465f77f931dc3e956935fb7fd4f621e1983ccaf464e9aa366593" + } + ], + "preflight_order": [ + { + "bytes": 6263, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/production-preflight.sql", + "exists": true, + "file": "production-preflight.sql", + "has_transaction_boundary": false, + "ok": true, + "packet_id": "mapped-rich-base", + "path": "docs/reports/leo-working-state-20260709/production-preflight.sql", + "phase": "preflight", + "sha256": "2d1eb113b1ea2f4c16096f432cca254c5edc259c9c5c27fcec01a98fe54014ff" + }, + { + "bytes": 1456, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/cross-surface-preflight.sql", + "exists": true, + "file": "cross-surface-preflight.sql", + "has_transaction_boundary": false, + "ok": true, + "packet_id": "cross-surface-anchor", + "path": "docs/reports/leo-working-state-20260709/cross-surface-preflight.sql", + "phase": "preflight", + "sha256": "dde19cddf80819145f755a028ed1b7a0c975b8e59a055e274843e21109dfff55" + }, + { + "bytes": 2202, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/rio-strategy-preflight.sql", + "exists": true, + "file": "rio-strategy-preflight.sql", + "has_transaction_boundary": false, + "ok": true, + "packet_id": "rio-strategy-context", + "path": "docs/reports/leo-working-state-20260709/rio-strategy-preflight.sql", + "phase": "preflight", + "sha256": "5ca341ac6cd599736d79718cc8c8de0ae064197159b361a46f5df2174aa713c9" + }, + { + "bytes": 1015, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/governance-concept-preflight.sql", + "exists": true, + "file": "governance-concept-preflight.sql", + "has_transaction_boundary": false, + "ok": true, + "packet_id": "governance-concept-schema", + "path": "docs/reports/leo-working-state-20260709/governance-concept-preflight.sql", + "phase": "preflight", + "sha256": "05a08080dc80bc7e131b8ee225cd5e58079bcd8fc5bcb5d85b0884077423718f" + }, + { + "bytes": 5870, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/helmer-7powers-production-preflight.sql", + "exists": true, + "file": "helmer-7powers-production-preflight.sql", + "has_transaction_boundary": false, + "ok": true, + "packet_id": "helmer-7powers", + "path": "docs/reports/leo-working-state-20260709/helmer-7powers-production-preflight.sql", + "phase": "preflight", + "sha256": "415b73ffe0453325a289c7128cd461509536406a32bbea6a75c6e6386b4840d4" + } + ], + "rollback_order": [ + { + "bytes": 5702, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/helmer-7powers-production-delete-rollback.sql", + "exists": true, + "file": "helmer-7powers-production-delete-rollback.sql", + "has_transaction_boundary": true, + "ok": true, + "packet_id": "helmer-7powers", + "path": "docs/reports/leo-working-state-20260709/helmer-7powers-production-delete-rollback.sql", + "phase": "rollback", + "sha256": "77579ea3efbb3f3a06666f645a2bc662a501c8a3e92d9b389fca044dfef06a2d" + }, + { + "bytes": 1160, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/governance-concept-delete-rollback.sql", + "exists": true, + "file": "governance-concept-delete-rollback.sql", + "has_transaction_boundary": true, + "ok": true, + "packet_id": "governance-concept-schema", + "path": "docs/reports/leo-working-state-20260709/governance-concept-delete-rollback.sql", + "phase": "rollback", + "sha256": "786e107134a7c502130af9d92f19109184ac936399ac90d62b3b8c07f3e90f65" + }, + { + "bytes": 1185, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/rio-strategy-delete-rollback.sql", + "exists": true, + "file": "rio-strategy-delete-rollback.sql", + "has_transaction_boundary": true, + "ok": true, + "packet_id": "rio-strategy-context", + "path": "docs/reports/leo-working-state-20260709/rio-strategy-delete-rollback.sql", + "phase": "rollback", + "sha256": "f7c6896018ac785fe8214bb4dfbdbf936cb3f91d92dcf877aee80b682dae2d93" + }, + { + "bytes": 823, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/cross-surface-delete-rollback.sql", + "exists": true, + "file": "cross-surface-delete-rollback.sql", + "has_transaction_boundary": true, + "ok": true, + "packet_id": "cross-surface-anchor", + "path": "docs/reports/leo-working-state-20260709/cross-surface-delete-rollback.sql", + "phase": "rollback", + "sha256": "41d10138e6d668426f8601ea5f528d70335ed4d30aae23830a55c0f068f046a8" + }, + { + "bytes": 5803, + "command_template": "psql \"$TELEO_DATABASE_URL\" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/production-delete-rollback.sql", + "exists": true, + "file": "production-delete-rollback.sql", + "has_transaction_boundary": true, + "ok": true, + "packet_id": "mapped-rich-base", + "path": "docs/reports/leo-working-state-20260709/production-delete-rollback.sql", + "phase": "rollback", + "sha256": "297bd9e8b5246e6d9cb4883b15ad9e11e745b220508305f223985be38506bc36" + } + ] + }, + "post_apply_required_actions": [ + "Run every postflight SQL file and retain output.", + "Rerun live Telegram regression canaries after production postflight.", + "Record production apply receipt, row counts, service MainPID/NRestarts, and cleanup readback." + ], + "production_apply_authorization_present": false, + "production_apply_executed": false, + "ready_for_authorized_operator": true, + "report_dir": "docs/reports/leo-working-state-20260709", + "required_authorization_boundary": "Do not run apply commands unless the operator explicitly authorizes production DB apply for the integrated Working Leo packet set in dependency order.", + "status": "ready_not_executed" +} diff --git a/docs/reports/leo-working-state-20260709/working-leo-authorized-apply-runbook-current.md b/docs/reports/leo-working-state-20260709/working-leo-authorized-apply-runbook-current.md new file mode 100644 index 0000000..bf7063f --- /dev/null +++ b/docs/reports/leo-working-state-20260709/working-leo-authorized-apply-runbook-current.md @@ -0,0 +1,100 @@ +# Working Leo Authorized Apply Runbook + +Generated UTC: `2026-07-09T23:12:00.028604+00:00` +Mode: `operator_runbook_generation_only` +Status: `ready_not_executed` +Ready for authorized operator: `True` +Production apply executed: `False` +Production apply authorization present: `False` + +## Authorization Boundary + +Do not run apply commands unless the operator explicitly authorizes production DB apply for the integrated Working Leo packet set in dependency order. + +## Checks + +- `readiness_passes`: `True` +- `readiness_says_not_executed`: `True` +- `readiness_says_not_authorized`: `True` +- `apply_order_matches_manifest`: `True` +- `rollback_order_matches_manifest`: `True` +- `all_phase_files_exist`: `True` +- `all_phase_files_non_empty`: `True` +- `all_apply_and_rollback_sql_have_transactions`: `True` + +## Preflight Order + +1. `mapped-rich-base` -> `production-preflight.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/production-preflight.sql` +2. `cross-surface-anchor` -> `cross-surface-preflight.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/cross-surface-preflight.sql` +3. `rio-strategy-context` -> `rio-strategy-preflight.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/rio-strategy-preflight.sql` +4. `governance-concept-schema` -> `governance-concept-preflight.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/governance-concept-preflight.sql` +5. `helmer-7powers` -> `helmer-7powers-production-preflight.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/helmer-7powers-production-preflight.sql` + +## Apply Order + +1. `mapped-rich-base` -> `production-apply-authorized-commit.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/production-apply-authorized-commit.sql` +2. `cross-surface-anchor` -> `cross-surface-apply-authorized-commit.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/cross-surface-apply-authorized-commit.sql` +3. `rio-strategy-context` -> `rio-strategy-apply-authorized-commit.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/rio-strategy-apply-authorized-commit.sql` +4. `governance-concept-schema` -> `governance-concept-apply-authorized-commit.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/governance-concept-apply-authorized-commit.sql` +5. `helmer-7powers` -> `helmer-7powers-production-apply-authorized-commit.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/helmer-7powers-production-apply-authorized-commit.sql` + +## Postflight Order + +1. `mapped-rich-base` -> `production-postflight.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/production-postflight.sql` +2. `cross-surface-anchor` -> `cross-surface-postflight.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/cross-surface-postflight.sql` +3. `rio-strategy-context` -> `rio-strategy-postflight.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/rio-strategy-postflight.sql` +4. `governance-concept-schema` -> `governance-concept-postflight.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/governance-concept-postflight.sql` +5. `helmer-7powers` -> `helmer-7powers-production-postflight.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/helmer-7powers-production-postflight.sql` + +## Rollback Order + +1. `helmer-7powers` -> `helmer-7powers-production-delete-rollback.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/helmer-7powers-production-delete-rollback.sql` +2. `governance-concept-schema` -> `governance-concept-delete-rollback.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/governance-concept-delete-rollback.sql` +3. `rio-strategy-context` -> `rio-strategy-delete-rollback.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/rio-strategy-delete-rollback.sql` +4. `cross-surface-anchor` -> `cross-surface-delete-rollback.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/cross-surface-delete-rollback.sql` +5. `mapped-rich-base` -> `production-delete-rollback.sql` + - `psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f docs/reports/leo-working-state-20260709/production-delete-rollback.sql` + +## Expected Counts After Apply + +- `claim_concept_map_links`: `1` +- `claim_edges`: `18` +- `claim_evidence`: `34` +- `claim_governance_gate_links`: `2` +- `claims`: `12` +- `concept_maps`: `1` +- `cross_surface_anchor_updates`: `1` +- `helmer_ledger_rows_marked_applied`: `1` +- `reasoning_tools`: `1` +- `rio_strategy_anchors`: `5` +- `rio_strategy_nodes`: `2` +- `sources`: `28` + +## Post-Apply Required Actions + +- Run every postflight SQL file and retain output. +- Rerun live Telegram regression canaries after production postflight. +- Record production apply receipt, row counts, service MainPID/NRestarts, and cleanup readback. + +## Claim Ceiling + +This runbook is proof that the authorized apply sequence is explicit and ordered. It does not execute SQL, does not mutate production, and does not replace explicit production apply authorization. diff --git a/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-results-current.json b/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-results-current.json index 1deefec..76df36b 100644 --- a/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-results-current.json +++ b/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-results-current.json @@ -14,14 +14,20 @@ "id": "cory_style_outcome", "purpose": "Exercise regular-use ambiguity: broad operator intent, demo pressure, missing IDs, mixed KB/runtime/governance asks.", "runtime": "disposable_clone_or_sandbox_first" + }, + { + "id": "direct_claim_followup", + "purpose": "Ask no-context Cory-style direct claim questions and require the expected answer plus the next follow-up.", + "runtime": "disposable_clone_or_sandbox_first" } ], - "claim_ceiling": "OE rows are retained live Telegram read-only evidence. CS rows are sandbox fixture answers that make the broader Cory-style outcome benchmark executable; they do not prove live Telegram behavior or production DB mutation for those prompts.", + "claim_ceiling": "OE rows are retained live Telegram read-only evidence. CS rows are sandbox fixture answers for broad Cory-style outcomes. DC rows are no-context direct-claim expected-answer fixtures with follow-up actions. CS/DC rows do not prove live Telegram behavior or production DB mutation for those prompts.", "evidence_tiers": { "CS-01..CS-09": "sandbox_expected_answer_fixture", + "DC-01..DC-06": "sandbox_expected_answer_fixture", "OE-01..OE-05": "live_telegram_readonly_retained" }, - "generated_at_utc": "2026-07-09T23:02:15.016213+00:00", + "generated_at_utc": "2026-07-09T23:12:29.442787+00:00", "live_suite_score": "docs/reports/leo-working-state-20260709/telegram-live-open-ended-suite-score-current.json", "mode": "mixed_live_retained_and_sandbox_fixture", "mutates_kb": false, @@ -214,11 +220,89 @@ "prompt_id": "CS-09", "reply": "Telegram file refs, document_evaluations, proposal source_ref, raw PDFs, and public.sources are different things. telegram_file_refs and document_evaluations are staging rows or artifact pointers; raw PDFs and extracted text are files; kb_stage.kb_proposals is the proposal ledger; public.sources is canonical evidence after review/apply. Pending proposals can have source_ref pointers and still lack direct public.sources links or an apply contract. The next action is a row-link audit, then a guarded staging/apply packet with canonical source ids and postflight readback.", "runtime": "disposable_clone_or_sandbox_first" + }, + { + "claim_ceiling": "Fixture answer for direct-claim follow-up benchmark scoring; not a live Telegram reply.", + "evidence_tier": "sandbox_expected_answer_fixture", + "expected_answer": "Answer with the proposed/approved/applied split. Do not claim a DB update unless canonical public.* rows or applied_at/postflight proof are present.", + "expected_follow_up": "Ask or run the row-level readback: which proposal/time window to inspect, kb_stage status, public.* before/after counts, and the next guarded apply packet if approved rows are still pending.", + "marker": "WL-CORY-DIRECT-SANDBOX-DC-01-20260710", + "mode": "sandbox_expected_answer_fixture", + "mutates_kb": false, + "prompt": "Did we actually update the knowledge base, or is it still just proposals?", + "prompt_id": "DC-01", + "reply": "I would not assume the knowledge base changed. Proposed, pending_review, approved, applied, and not applied are different states. The canonical answer comes from public.claims, public.sources, public.claim_edges, and public.claim_evidence, plus kb_stage.kb_proposals applied_at/readback. I need row-level before/after counts, row ids, and postflight proof; I cannot claim a DB update without proof. Follow-up: I would ask which proposal or time window Cory means, then query kb_stage and public.* and either show the pending gap or prepare the next guarded apply packet.", + "runtime": "disposable_clone_or_sandbox_first" + }, + { + "claim_ceiling": "Fixture answer for direct-claim follow-up benchmark scoring; not a live Telegram reply.", + "evidence_tier": "sandbox_expected_answer_fixture", + "expected_answer": "Separate usable fixture/packet readiness from production canonical state. Helmer can be packet-ready or clone-proven while still not production-applied.", + "expected_follow_up": "Offer the next proof-changing action: show the packet/readiness artifact, get explicit apply authorization, run apply in integrated order, then postflight and Telegram regression.", + "marker": "WL-CORY-DIRECT-SANDBOX-DC-02-20260710", + "mode": "sandbox_expected_answer_fixture", + "mutates_kb": false, + "prompt": "Is Helmer's 7 Powers in Leo now?", + "prompt_id": "DC-02", + "reply": "Helmer's 7 Powers can be packet-ready or clone-proven without being canonical production state. I would check public.reasoning_tools, public.claims, public.sources, public.claim_edges, public.claim_evidence, and the proposal ledger with postflight row counts before saying it is in Leo now. If production_apply is false, I cannot claim it is production-applied. Follow-up: I would show the readiness packet, ask for explicit authorization if Cory wants it applied, run the integrated order, then postflight and Telegram regression.", + "runtime": "disposable_clone_or_sandbox_first" + }, + { + "claim_ceiling": "Fixture answer for direct-claim follow-up benchmark scoring; not a live Telegram reply.", + "evidence_tier": "sandbox_expected_answer_fixture", + "expected_answer": "Check schema truth first. If matrix_voters/proposal_votes/proposal_decisions are absent, do not infer matrix approval from pending proposals.", + "expected_follow_up": "Follow up with the schema/readback request and then fall back to kb_stage.kb_proposals review/apply columns plus canonical row readback.", + "marker": "WL-CORY-DIRECT-SANDBOX-DC-03-20260710", + "mode": "sandbox_expected_answer_fixture", + "mutates_kb": false, + "prompt": "Did the decision matrix approve this already?", + "prompt_id": "DC-03", + "reply": "I would check the schema before claiming a decision-matrix approved anything. If matrix_voters, proposal_votes, and proposal_decisions tables do not exist, then the decision-matrix is not yet shipped and no tally approved this. The current proof should come from kb_stage.kb_proposals review/apply columns and canonical public.* row readback. I cannot claim a matrix result without proof. Follow-up: I would run a fresh schema readback, then show the proposal ledger and canonical rows instead of overclaiming a matrix result.", + "runtime": "disposable_clone_or_sandbox_first" + }, + { + "claim_ceiling": "Fixture answer for direct-claim follow-up benchmark scoring; not a live Telegram reply.", + "evidence_tier": "sandbox_expected_answer_fixture", + "expected_answer": "Distinguish Telegram file refs, document evaluations, proposal source_ref, raw files, and canonical public.sources. Do not reduce pending status to one document-pointer cause without row-link proof.", + "expected_follow_up": "Follow up with a row-link audit: list proposal IDs/source_ref values, match them to public.sources or missing source rows, then propose the guarded source/evidence apply contract.", + "marker": "WL-CORY-DIRECT-SANDBOX-DC-04-20260710", + "mode": "sandbox_expected_answer_fixture", + "mutates_kb": false, + "prompt": "Are the pending proposals stuck because the documents are not pointed at the right source rows?", + "prompt_id": "DC-04", + "reply": "Telegram file refs, document_evaluations, proposal source_ref, raw PDFs, and public.sources are separate. A pending proposal can have staging rows and file pointers but still not have canonical source rows, claim_evidence, or an apply contract. I will not claim document pointers are the only blocker without row-link proof. Follow-up: I would check and audit each proposal source_ref against public.sources, identify missing canonical rows, then stage a guarded source/evidence apply proposal for review.", + "runtime": "disposable_clone_or_sandbox_first" + }, + { + "claim_ceiling": "Fixture answer for direct-claim follow-up benchmark scoring; not a live Telegram reply.", + "evidence_tier": "sandbox_expected_answer_fixture", + "expected_answer": "Give the operator truth ceiling: live staging and strict existing-ID apply can be shown if proven, while rich packets remain clone-proven/not production-applied unless explicitly applied.", + "expected_follow_up": "Ask what demo tier Cory wants: show existing live proof, run a safe live staging canary, or authorize the prepared production apply packet and postflight/regression sequence.", + "marker": "WL-CORY-DIRECT-SANDBOX-DC-05-20260710", + "mode": "sandbox_expected_answer_fixture", + "mutates_kb": false, + "prompt": "Can I demo that Leo changes the KB?", + "prompt_id": "DC-05", + "reply": "Demo answer: Leo can show live staging and strict existing-ID apply proof, but proposed, approved, applied, and not applied stay separate. Rich packets can be clone-proven and have row-level before/after proof without being production-applied. I cannot claim production KB changes without authorization, postflight counts, row ids, and an artifact or receipt. Follow-up: I would ask which demo tier Cory wants: show the existing live proof, run a safe staging canary, or authorize the prepared apply packet plus regression.", + "runtime": "disposable_clone_or_sandbox_first" + }, + { + "claim_ceiling": "Fixture answer for direct-claim follow-up benchmark scoring; not a live Telegram reply.", + "evidence_tier": "sandbox_expected_answer_fixture", + "expected_answer": "Explain that direct SOUL.md edits are runtime/profile edits. Canonical identity requires DB rows plus a render/sync proof into the runtime artifact.", + "expected_follow_up": "Follow up by asking whether to audit canonical identity rows, stage the missing DB proposal, or run the renderer/sync readback if an approved identity row already exists.", + "marker": "WL-CORY-DIRECT-SANDBOX-DC-06-20260710", + "mode": "sandbox_expected_answer_fixture", + "mutates_kb": false, + "prompt": "If we changed SOUL.md, did we change Leo's canonical identity?", + "prompt_id": "DC-06", + "reply": "A direct SOUL.md edit changes the runtime/profile artifact, not canonical identity by itself. Canonical identity is DB-first: Postgres public.personas, public.strategies, public.beliefs, public.strategy_nodes, and public.strategy_node_anchors must have row-level proof, row ids, and verify/read back evidence before they render or sync into SOUL.md. Until readback and renderer proof exist, I cannot claim canonical identity changed. Follow-up: I would ask whether to audit the canonical identity rows, stage the missing DB proposal, or run the renderer/sync readback.", + "runtime": "disposable_clone_or_sandbox_first" } ], "score": { "coverage": "full", - "expected_prompt_count": 14, + "expected_prompt_count": 20, "expected_prompt_ids": [ "OE-01", "OE-02", @@ -233,13 +317,19 @@ "CS-06", "CS-07", "CS-08", - "CS-09" + "CS-09", + "DC-01", + "DC-02", + "DC-03", + "DC-04", + "DC-05", + "DC-06" ], "failures": [], "missing_prompt_ids": [], "pass": true, - "passes": 14, - "prompt_count": 14, + "passes": 20, + "prompt_count": 20, "scores": [ { "dimension": "vague_incident_triage", @@ -411,6 +501,85 @@ "next_action": true, "staging_or_review": true } + }, + { + "dimension": "db_changed_direct_claim", + "overclaim_detected": false, + "pass": true, + "prompt_id": "DC-01", + "signals": { + "canonical_db": true, + "cory_followup": true, + "no_overclaim": true, + "row_level_proof": true, + "state_semantics": true + } + }, + { + "dimension": "helmer_live_direct_claim", + "overclaim_detected": false, + "pass": true, + "prompt_id": "DC-02", + "signals": { + "authorization": true, + "canonical_db": true, + "cory_followup": true, + "no_overclaim": true, + "row_level_proof": true + } + }, + { + "dimension": "decision_matrix_direct_claim", + "overclaim_detected": false, + "pass": true, + "prompt_id": "DC-03", + "signals": { + "canonical_db": true, + "cory_followup": true, + "decision_matrix": true, + "fresh_readback": true, + "no_overclaim": true + } + }, + { + "dimension": "document_source_direct_claim", + "overclaim_detected": false, + "pass": true, + "prompt_id": "DC-04", + "signals": { + "canonical_db": true, + "cory_followup": true, + "document_artifact_linking": true, + "no_overclaim": true, + "staging_or_review": true + } + }, + { + "dimension": "demo_changes_kb_direct_claim", + "overclaim_detected": false, + "pass": true, + "prompt_id": "DC-05", + "signals": { + "artifact": true, + "authorization": true, + "cory_followup": true, + "no_overclaim": true, + "row_level_proof": true, + "state_semantics": true + } + }, + { + "dimension": "soul_identity_direct_claim", + "overclaim_detected": false, + "pass": true, + "prompt_id": "DC-06", + "signals": { + "canonical_db": true, + "cory_followup": true, + "identity_rendering": true, + "no_overclaim": true, + "row_level_proof": true + } } ], "unexpected_prompt_ids": [], diff --git a/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-results-current.md b/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-results-current.md index 8888cb8..5046761 100644 --- a/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-results-current.md +++ b/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-results-current.md @@ -1,15 +1,16 @@ # Working Leo Cory Outcome Sandbox Results -Generated UTC: `2026-07-09T23:02:15.016213+00:00` +Generated UTC: `2026-07-09T23:12:29.442787+00:00` Mode: `mixed_live_retained_and_sandbox_fixture` Mutates KB: `False` Overall full-catalog score: `True` -Prompts scored: `14/14` +Prompts scored: `20/20` ## Evidence Tiers - `OE-01..OE-05`: `live_telegram_readonly_retained` - `CS-01..CS-09`: `sandbox_expected_answer_fixture` +- `DC-01..DC-06`: `sandbox_expected_answer_fixture` ## Scenario Scores @@ -27,7 +28,13 @@ Prompts scored: `14/14` - `CS-07` / `identity_rendering_from_db`: `pass=True` - `CS-08` / `decision_matrix_row_provenance`: `pass=True` - `CS-09` / `document_artifact_linking`: `pass=True` +- `DC-01` / `db_changed_direct_claim`: `pass=True` +- `DC-02` / `helmer_live_direct_claim`: `pass=True` +- `DC-03` / `decision_matrix_direct_claim`: `pass=True` +- `DC-04` / `document_source_direct_claim`: `pass=True` +- `DC-05` / `demo_changes_kb_direct_claim`: `pass=True` +- `DC-06` / `soul_identity_direct_claim`: `pass=True` ## Claim Ceiling -OE rows are retained live Telegram read-only evidence. CS rows are sandbox fixture answers that make the broader Cory-style outcome benchmark executable; they do not prove live Telegram behavior or production DB mutation for those prompts. +OE rows are retained live Telegram read-only evidence. CS rows are sandbox fixture answers for broad Cory-style outcomes. DC rows are no-context direct-claim expected-answer fixtures with follow-up actions. CS/DC rows do not prove live Telegram behavior or production DB mutation for those prompts. diff --git a/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-score-current.json b/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-score-current.json index fe6b77e..d24af97 100644 --- a/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-score-current.json +++ b/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-score-current.json @@ -14,10 +14,16 @@ "id": "cory_style_outcome", "purpose": "Exercise regular-use ambiguity: broad operator intent, demo pressure, missing IDs, mixed KB/runtime/governance asks.", "runtime": "disposable_clone_or_sandbox_first" + }, + { + "id": "direct_claim_followup", + "purpose": "Ask no-context Cory-style direct claim questions and require the expected answer plus the next follow-up.", + "runtime": "disposable_clone_or_sandbox_first" } ], - "generated_at_utc": "2026-07-09T23:02:15.092629+00:00", + "generated_at_utc": "2026-07-09T23:12:29.470281+00:00", "include_cory_style_scenarios": true, + "include_direct_claim_followups": true, "mode": "retained_evidence_score", "mutates_kb": false, "require_all": true, @@ -210,11 +216,89 @@ "prompt_id": "CS-09", "reply": "Telegram file refs, document_evaluations, proposal source_ref, raw PDFs, and public.sources are different things. telegram_file_refs and document_evaluations are staging rows or artifact pointers; raw PDFs and extracted text are files; kb_stage.kb_proposals is the proposal ledger; public.sources is canonical evidence after review/apply. Pending proposals can have source_ref pointers and still lack direct public.sources links or an apply contract. The next action is a row-link audit, then a guarded staging/apply packet with canonical source ids and postflight readback.", "runtime": "disposable_clone_or_sandbox_first" + }, + { + "claim_ceiling": "Fixture answer for direct-claim follow-up benchmark scoring; not a live Telegram reply.", + "evidence_tier": "sandbox_expected_answer_fixture", + "expected_answer": "Answer with the proposed/approved/applied split. Do not claim a DB update unless canonical public.* rows or applied_at/postflight proof are present.", + "expected_follow_up": "Ask or run the row-level readback: which proposal/time window to inspect, kb_stage status, public.* before/after counts, and the next guarded apply packet if approved rows are still pending.", + "marker": "WL-CORY-DIRECT-SANDBOX-DC-01-20260710", + "mode": "sandbox_expected_answer_fixture", + "mutates_kb": false, + "prompt": "Did we actually update the knowledge base, or is it still just proposals?", + "prompt_id": "DC-01", + "reply": "I would not assume the knowledge base changed. Proposed, pending_review, approved, applied, and not applied are different states. The canonical answer comes from public.claims, public.sources, public.claim_edges, and public.claim_evidence, plus kb_stage.kb_proposals applied_at/readback. I need row-level before/after counts, row ids, and postflight proof; I cannot claim a DB update without proof. Follow-up: I would ask which proposal or time window Cory means, then query kb_stage and public.* and either show the pending gap or prepare the next guarded apply packet.", + "runtime": "disposable_clone_or_sandbox_first" + }, + { + "claim_ceiling": "Fixture answer for direct-claim follow-up benchmark scoring; not a live Telegram reply.", + "evidence_tier": "sandbox_expected_answer_fixture", + "expected_answer": "Separate usable fixture/packet readiness from production canonical state. Helmer can be packet-ready or clone-proven while still not production-applied.", + "expected_follow_up": "Offer the next proof-changing action: show the packet/readiness artifact, get explicit apply authorization, run apply in integrated order, then postflight and Telegram regression.", + "marker": "WL-CORY-DIRECT-SANDBOX-DC-02-20260710", + "mode": "sandbox_expected_answer_fixture", + "mutates_kb": false, + "prompt": "Is Helmer's 7 Powers in Leo now?", + "prompt_id": "DC-02", + "reply": "Helmer's 7 Powers can be packet-ready or clone-proven without being canonical production state. I would check public.reasoning_tools, public.claims, public.sources, public.claim_edges, public.claim_evidence, and the proposal ledger with postflight row counts before saying it is in Leo now. If production_apply is false, I cannot claim it is production-applied. Follow-up: I would show the readiness packet, ask for explicit authorization if Cory wants it applied, run the integrated order, then postflight and Telegram regression.", + "runtime": "disposable_clone_or_sandbox_first" + }, + { + "claim_ceiling": "Fixture answer for direct-claim follow-up benchmark scoring; not a live Telegram reply.", + "evidence_tier": "sandbox_expected_answer_fixture", + "expected_answer": "Check schema truth first. If matrix_voters/proposal_votes/proposal_decisions are absent, do not infer matrix approval from pending proposals.", + "expected_follow_up": "Follow up with the schema/readback request and then fall back to kb_stage.kb_proposals review/apply columns plus canonical row readback.", + "marker": "WL-CORY-DIRECT-SANDBOX-DC-03-20260710", + "mode": "sandbox_expected_answer_fixture", + "mutates_kb": false, + "prompt": "Did the decision matrix approve this already?", + "prompt_id": "DC-03", + "reply": "I would check the schema before claiming a decision-matrix approved anything. If matrix_voters, proposal_votes, and proposal_decisions tables do not exist, then the decision-matrix is not yet shipped and no tally approved this. The current proof should come from kb_stage.kb_proposals review/apply columns and canonical public.* row readback. I cannot claim a matrix result without proof. Follow-up: I would run a fresh schema readback, then show the proposal ledger and canonical rows instead of overclaiming a matrix result.", + "runtime": "disposable_clone_or_sandbox_first" + }, + { + "claim_ceiling": "Fixture answer for direct-claim follow-up benchmark scoring; not a live Telegram reply.", + "evidence_tier": "sandbox_expected_answer_fixture", + "expected_answer": "Distinguish Telegram file refs, document evaluations, proposal source_ref, raw files, and canonical public.sources. Do not reduce pending status to one document-pointer cause without row-link proof.", + "expected_follow_up": "Follow up with a row-link audit: list proposal IDs/source_ref values, match them to public.sources or missing source rows, then propose the guarded source/evidence apply contract.", + "marker": "WL-CORY-DIRECT-SANDBOX-DC-04-20260710", + "mode": "sandbox_expected_answer_fixture", + "mutates_kb": false, + "prompt": "Are the pending proposals stuck because the documents are not pointed at the right source rows?", + "prompt_id": "DC-04", + "reply": "Telegram file refs, document_evaluations, proposal source_ref, raw PDFs, and public.sources are separate. A pending proposal can have staging rows and file pointers but still not have canonical source rows, claim_evidence, or an apply contract. I will not claim document pointers are the only blocker without row-link proof. Follow-up: I would check and audit each proposal source_ref against public.sources, identify missing canonical rows, then stage a guarded source/evidence apply proposal for review.", + "runtime": "disposable_clone_or_sandbox_first" + }, + { + "claim_ceiling": "Fixture answer for direct-claim follow-up benchmark scoring; not a live Telegram reply.", + "evidence_tier": "sandbox_expected_answer_fixture", + "expected_answer": "Give the operator truth ceiling: live staging and strict existing-ID apply can be shown if proven, while rich packets remain clone-proven/not production-applied unless explicitly applied.", + "expected_follow_up": "Ask what demo tier Cory wants: show existing live proof, run a safe live staging canary, or authorize the prepared production apply packet and postflight/regression sequence.", + "marker": "WL-CORY-DIRECT-SANDBOX-DC-05-20260710", + "mode": "sandbox_expected_answer_fixture", + "mutates_kb": false, + "prompt": "Can I demo that Leo changes the KB?", + "prompt_id": "DC-05", + "reply": "Demo answer: Leo can show live staging and strict existing-ID apply proof, but proposed, approved, applied, and not applied stay separate. Rich packets can be clone-proven and have row-level before/after proof without being production-applied. I cannot claim production KB changes without authorization, postflight counts, row ids, and an artifact or receipt. Follow-up: I would ask which demo tier Cory wants: show the existing live proof, run a safe staging canary, or authorize the prepared apply packet plus regression.", + "runtime": "disposable_clone_or_sandbox_first" + }, + { + "claim_ceiling": "Fixture answer for direct-claim follow-up benchmark scoring; not a live Telegram reply.", + "evidence_tier": "sandbox_expected_answer_fixture", + "expected_answer": "Explain that direct SOUL.md edits are runtime/profile edits. Canonical identity requires DB rows plus a render/sync proof into the runtime artifact.", + "expected_follow_up": "Follow up by asking whether to audit canonical identity rows, stage the missing DB proposal, or run the renderer/sync readback if an approved identity row already exists.", + "marker": "WL-CORY-DIRECT-SANDBOX-DC-06-20260710", + "mode": "sandbox_expected_answer_fixture", + "mutates_kb": false, + "prompt": "If we changed SOUL.md, did we change Leo's canonical identity?", + "prompt_id": "DC-06", + "reply": "A direct SOUL.md edit changes the runtime/profile artifact, not canonical identity by itself. Canonical identity is DB-first: Postgres public.personas, public.strategies, public.beliefs, public.strategy_nodes, and public.strategy_node_anchors must have row-level proof, row ids, and verify/read back evidence before they render or sync into SOUL.md. Until readback and renderer proof exist, I cannot claim canonical identity changed. Follow-up: I would ask whether to audit the canonical identity rows, stage the missing DB proposal, or run the renderer/sync readback.", + "runtime": "disposable_clone_or_sandbox_first" } ], "score": { "coverage": "full", - "expected_prompt_count": 14, + "expected_prompt_count": 20, "expected_prompt_ids": [ "OE-01", "OE-02", @@ -229,13 +313,19 @@ "CS-06", "CS-07", "CS-08", - "CS-09" + "CS-09", + "DC-01", + "DC-02", + "DC-03", + "DC-04", + "DC-05", + "DC-06" ], "failures": [], "missing_prompt_ids": [], "pass": true, - "passes": 14, - "prompt_count": 14, + "passes": 20, + "prompt_count": 20, "scores": [ { "dimension": "vague_incident_triage", @@ -407,6 +497,85 @@ "next_action": true, "staging_or_review": true } + }, + { + "dimension": "db_changed_direct_claim", + "overclaim_detected": false, + "pass": true, + "prompt_id": "DC-01", + "signals": { + "canonical_db": true, + "cory_followup": true, + "no_overclaim": true, + "row_level_proof": true, + "state_semantics": true + } + }, + { + "dimension": "helmer_live_direct_claim", + "overclaim_detected": false, + "pass": true, + "prompt_id": "DC-02", + "signals": { + "authorization": true, + "canonical_db": true, + "cory_followup": true, + "no_overclaim": true, + "row_level_proof": true + } + }, + { + "dimension": "decision_matrix_direct_claim", + "overclaim_detected": false, + "pass": true, + "prompt_id": "DC-03", + "signals": { + "canonical_db": true, + "cory_followup": true, + "decision_matrix": true, + "fresh_readback": true, + "no_overclaim": true + } + }, + { + "dimension": "document_source_direct_claim", + "overclaim_detected": false, + "pass": true, + "prompt_id": "DC-04", + "signals": { + "canonical_db": true, + "cory_followup": true, + "document_artifact_linking": true, + "no_overclaim": true, + "staging_or_review": true + } + }, + { + "dimension": "demo_changes_kb_direct_claim", + "overclaim_detected": false, + "pass": true, + "prompt_id": "DC-05", + "signals": { + "artifact": true, + "authorization": true, + "cory_followup": true, + "no_overclaim": true, + "row_level_proof": true, + "state_semantics": true + } + }, + { + "dimension": "soul_identity_direct_claim", + "overclaim_detected": false, + "pass": true, + "prompt_id": "DC-06", + "signals": { + "canonical_db": true, + "cory_followup": true, + "identity_rendering": true, + "no_overclaim": true, + "row_level_proof": true + } } ], "unexpected_prompt_ids": [], diff --git a/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-score-current.md b/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-score-current.md index e8b03a7..c112233 100644 --- a/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-score-current.md +++ b/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-score-current.md @@ -1,11 +1,11 @@ # Working Leo Open-Ended Benchmark Score -Generated UTC: `2026-07-09T23:02:15.092629+00:00` +Generated UTC: `2026-07-09T23:12:29.470281+00:00` Mode: `retained_evidence_score` Source: `docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-results-current.json` Coverage: `full` Overall pass for scored coverage: `True` -Prompts scored: `14/14` +Prompts scored: `20/20` ## Scores @@ -96,7 +96,50 @@ Prompts scored: `14/14` - `canonical_db`: `True` - `next_action`: `True` - `overclaim_detected`: `False` +- `DC-01` / `db_changed_direct_claim`: `pass=True` + - `state_semantics`: `True` + - `canonical_db`: `True` + - `row_level_proof`: `True` + - `cory_followup`: `True` + - `no_overclaim`: `True` + - `overclaim_detected`: `False` +- `DC-02` / `helmer_live_direct_claim`: `pass=True` + - `canonical_db`: `True` + - `row_level_proof`: `True` + - `authorization`: `True` + - `cory_followup`: `True` + - `no_overclaim`: `True` + - `overclaim_detected`: `False` +- `DC-03` / `decision_matrix_direct_claim`: `pass=True` + - `decision_matrix`: `True` + - `canonical_db`: `True` + - `fresh_readback`: `True` + - `cory_followup`: `True` + - `no_overclaim`: `True` + - `overclaim_detected`: `False` +- `DC-04` / `document_source_direct_claim`: `pass=True` + - `document_artifact_linking`: `True` + - `staging_or_review`: `True` + - `canonical_db`: `True` + - `cory_followup`: `True` + - `no_overclaim`: `True` + - `overclaim_detected`: `False` +- `DC-05` / `demo_changes_kb_direct_claim`: `pass=True` + - `state_semantics`: `True` + - `row_level_proof`: `True` + - `authorization`: `True` + - `artifact`: `True` + - `cory_followup`: `True` + - `no_overclaim`: `True` + - `overclaim_detected`: `False` +- `DC-06` / `soul_identity_direct_claim`: `pass=True` + - `identity_rendering`: `True` + - `canonical_db`: `True` + - `row_level_proof`: `True` + - `cory_followup`: `True` + - `no_overclaim`: `True` + - `overclaim_detected`: `False` ## Claim Ceiling -This is benchmark scoring evidence, not DB mutation proof. Full selected coverage means every OE and CS prompt ID for this run was scored. If retained results mix live Telegram replies with sandbox fixture replies, only the live-result rows prove Telegram-visible behavior; CS rows prove the sandbox-first outcome bar is executable and scored. +This is benchmark scoring evidence, not DB mutation proof. Full selected coverage means every OE, CS, and DC prompt ID for this run was scored. OE rows can prove live Telegram behavior only when the source results are live; CS/DC fixture rows prove the sandbox-first expected-answer and Cory follow-up bar is executable and scored. diff --git a/docs/reports/leo-working-state-20260709/working-leo-current-state-20260709.md b/docs/reports/leo-working-state-20260709/working-leo-current-state-20260709.md index 567ff80..84b6fd4 100644 --- a/docs/reports/leo-working-state-20260709/working-leo-current-state-20260709.md +++ b/docs/reports/leo-working-state-20260709/working-leo-current-state-20260709.md @@ -27,10 +27,11 @@ A working Leo, for Cory/m3taversal's current expectation, is a Telegram-facing a - service stayed active/running after the suite with `MainPID=3252143`, `NRestarts=0` - post-suite DB safety readback showed no marker rows in `kb_stage` or canonical `public.*` tables, and no claims/sources/edges/evidence rows created after the `OE-02` start timestamp - claim ceiling: this proves live read-only Telegram behavior, not production application of the prepared DB packets -- Full Cory-style outcome sandbox score now passes: - - mixed evidence score is `14/14`: retained live Telegram read-only replies for `OE-01` through `OE-05`, sandbox expected-answer fixtures for `CS-01` through `CS-09` +- Full Cory-style outcome/direct-claim sandbox score now passes: + - mixed evidence score is `20/20`: retained live Telegram read-only replies for `OE-01` through `OE-05`, sandbox expected-answer fixtures for `CS-01` through `CS-09`, and no-context direct-claim expected-answer/follow-up fixtures for `DC-01` through `DC-06` - `CS-01` through `CS-09` cover demo pressure, stuck KB without IDs, July 5 restore framing, reckless-versus-useful DB agency, knowledge object selection, partner demo truth ceiling, DB-first identity rendering, decision-matrix row provenance, and document artifact linking - - claim ceiling: this makes the broader Cory-style benchmark executable and scored, but it does not prove live Telegram behavior for the CS prompts + - `DC-01` through `DC-06` cover direct Cory-style questions with no supplied context: did the DB change, is Helmer live, did the decision matrix approve it, are document/source pointers the blocker, can Cory demo KB mutation, and whether SOUL.md edits changed canonical identity + - claim ceiling: this makes the broader Cory-style benchmark executable and scored, but it does not prove live Telegram behavior for the CS/DC prompts - Telegram-shaped handler canary passed in the same user/cwd context as the live service: - authorized source: `agent:main:telegram:group:-5146042086:9070919` - turn 1 ack passed @@ -119,6 +120,11 @@ A working Leo, for Cory/m3taversal's current expectation, is a Telegram-facing a - `production_apply_executed=false` - `production_apply_authorization_present=false` - verifier checks packet file hashes, manifest apply/rollback order, expected counts, retained clone apply/rollback markers, clone cleanup, and service stability +- Authorized apply operator runbook is generated: + - `status=ready_not_executed` + - `ready_for_authorized_operator=true` + - command templates cover preflight, apply, postflight, and rollback in manifest dependency order + - generator does not connect to Postgres or run SQL; production apply still requires explicit authorization - VPS/Hermes/DB identity and document-linking map is now documented: - the supplied July 9 master/build and database-reference PDFs confirm the intended architecture is DB-first identity: canonical Postgres identity graph rows render to `SOUL.md`, and evidence depth remains on demand through `teleo-kb` - direct `SOUL.md` edits are profile/runtime artifact edits, not canonical DB updates; they can drift or be overwritten unless represented in canonical rows first @@ -139,8 +145,8 @@ A working Leo, for Cory/m3taversal's current expectation, is a Telegram-facing a - Retained live Telegram benchmark score: `work/teleo-infra-main/docs/reports/leo-working-state-20260709/telegram-live-open-ended-benchmark-score-current.md` - Full live-safe open-ended Telegram suite: `work/teleo-infra-main/docs/reports/leo-working-state-20260709/telegram-live-open-ended-suite-current.md` - Full live-safe open-ended Telegram suite score: `work/teleo-infra-main/docs/reports/leo-working-state-20260709/telegram-live-open-ended-suite-score-current.md` -- Full Cory-style outcome sandbox results: `work/teleo-infra-main/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-results-current.md` -- Full Cory-style outcome sandbox score: `work/teleo-infra-main/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-score-current.md` +- Full Cory-style outcome/direct-claim sandbox results: `work/teleo-infra-main/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-results-current.md` +- Full Cory-style outcome/direct-claim sandbox score: `work/teleo-infra-main/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-score-current.md` - Gateway adapter canary: `outputs/telegram-gateway-canary-20260709/telegram-gateway-canary-current.md` - KB apply plan: `outputs/kb-apply-canary-20260709/kb-apply-canary-plan-current.md` - KB apply execute: `outputs/kb-apply-canary-20260709/kb-apply-canary-execute-current.md` @@ -160,6 +166,7 @@ A working Leo, for Cory/m3taversal's current expectation, is a Telegram-facing a - Integrated packet manifest: `outputs/rich-proposal-creation-plan-20260709/integrated-20260710/working-leo-integrated-packet.json` - Integrated clone proof log: `outputs/rich-proposal-creation-plan-20260709/integrated-20260710/working-leo-integrated-clone-rehearsal-current.log` - Authorized-apply readiness verifier: `work/teleo-infra-main/docs/reports/leo-working-state-20260709/working-leo-apply-readiness-current.md` +- Authorized apply operator runbook: `work/teleo-infra-main/docs/reports/leo-working-state-20260709/working-leo-authorized-apply-runbook-current.md` - VPS/Hermes/DB mapping: `outputs/vps-hermes-db-mapping-20260709.md` - Identity render / decision matrix / document artifact image: `outputs/leo-db-state-22-identity-render-matrix-artifacts.svg` - Execution plan: `outputs/working-leo-execution-plan-20260709/working-leo-execution-plan-current.md` @@ -187,4 +194,5 @@ A working Leo, for Cory/m3taversal's current expectation, is a Telegram-facing a 1. Keep the live Telegram canary as the top-level regression proof for Cory-working memory + KB audit. 2. Do not auto-apply old freeform approvals. 3. Do not run production apply without explicit authorization. The mapped base, cross-surface anchor, Rio strategy, governance/concept, and Helmer packets compose in one clone and are ready as a packet set, but are not production-applied. -4. Keep VPS Leo runtime behavior unchanged while the DB apply path is stabilized. +4. Use the direct-claim/follow-up benchmark cases as the expectation for Cory-style no-context questions before broadening live tests beyond the safe OE suite. +5. Keep VPS Leo runtime behavior unchanged while the DB apply path is stabilized. diff --git a/docs/reports/leo-working-state-20260709/working-leo-execution-plan-current.md b/docs/reports/leo-working-state-20260709/working-leo-execution-plan-current.md index 687496f..4b350ed 100644 --- a/docs/reports/leo-working-state-20260709/working-leo-execution-plan-current.md +++ b/docs/reports/leo-working-state-20260709/working-leo-execution-plan-current.md @@ -1,6 +1,6 @@ # Working Leo Execution Plan - 2026-07-09 -Claim ceiling: VPS Telegram memory/KB audit, full live-safe open-ended Telegram suite, full mixed-evidence Cory-style sandbox benchmark, Telegram-to-`kb_stage` proposal staging, and strict existing-ID apply are working. The full prepared DB packet set is integrated-clone-proven in dependency order: mapped rich base, cross-surface anchor, Rio strategy context, governance/concept schema, and Helmer 7 Powers. None are production-applied. GCP parity remains pending. +Claim ceiling: VPS Telegram memory/KB audit, full live-safe open-ended Telegram suite, full mixed-evidence Cory-style outcome/direct-claim sandbox benchmark, Telegram-to-`kb_stage` proposal staging, and strict existing-ID apply are working. The full prepared DB packet set is integrated-clone-proven in dependency order: mapped rich base, cross-surface anchor, Rio strategy context, governance/concept schema, and Helmer 7 Powers. An authorized apply operator runbook is generated, but none of those packets are production-applied. GCP parity remains pending. ## WL-EXEC-01 - VPS Telegram-visible memory + KB audit - Status: `done` @@ -13,9 +13,9 @@ Claim ceiling: VPS Telegram memory/KB audit, full live-safe open-ended Telegram - `/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/work/teleo-infra-main/docs/reports/leo-working-state-20260709/telegram-live-open-ended-suite-score-current.md` ## WL-EXEC-01B - Cory-style outcome benchmark -- Status: `done_for_sandbox_fixture_not_live_cs` -- Result: Full mixed-evidence Cory-style benchmark scores `14/14`: `OE-01` through `OE-05` are retained live Telegram read-only replies; `CS-01` through `CS-09` are sandbox expected-answer fixtures covering the broader regular-use cases. -- Not done condition: Reopen if the CS fixture stops scoring, if a future disposable clone/live CS run regresses, or if anyone claims the CS rows are live Telegram proof. +- Status: `done_for_sandbox_fixture_not_live_cs_dc` +- Result: Full mixed-evidence Cory-style benchmark scores `20/20`: `OE-01` through `OE-05` are retained live Telegram read-only replies; `CS-01` through `CS-09` are sandbox expected-answer fixtures covering broader regular-use cases; `DC-01` through `DC-06` are no-context direct-claim expected-answer fixtures requiring the correct Cory-style follow-up action. +- Not done condition: Reopen if the CS/DC fixtures stop scoring, if a future disposable clone/live CS/DC run regresses, or if anyone claims the CS/DC rows are live Telegram proof. - Evidence: - `/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/work/teleo-infra-main/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-results-current.md` - `/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/work/teleo-infra-main/docs/reports/leo-working-state-20260709/working-leo-cory-outcome-sandbox-score-current.md` @@ -94,13 +94,14 @@ Claim ceiling: VPS Telegram memory/KB audit, full live-safe open-ended Telegram ## WL-EXEC-06B - Integrated production-order packet rehearsal - Status: `done_integrated_clone_ready_not_executed` -- Result: Applied the full prepared packet set in one disposable clone in dependency order: mapped base, cross-surface anchor, Rio strategy context, governance/concept schema, and Helmer 7 Powers. Expected integrated footprint was `12` claims, `28` sources, `18` claim edges, `34` claim evidence rows, `1` reasoning tool, `2` Rio strategy nodes, `5` Rio strategy anchors, `1` concept map, `1` concept link, `2` governance links, one Helmer ledger row marked applied, and one cross-surface anchor update. Reverse rollback restored all generated rows/schema to zero/absent, restored the cross-surface anchor to the old claim, restored the Helmer ledger to `approved`, dropped the disposable DB, removed temp files, and left production unchanged. The apply-readiness verifier now passes over the retained manifest/files/logs with `ready_for_authorized_production_apply_packet=true`, `production_apply_executed=false`, and `production_apply_authorization_present=false`. -- Next action: If production apply is explicitly authorized, use the integrated manifest order and immediately rerun live Telegram regression canaries after production postflight. +- Result: Applied the full prepared packet set in one disposable clone in dependency order: mapped base, cross-surface anchor, Rio strategy context, governance/concept schema, and Helmer 7 Powers. Expected integrated footprint was `12` claims, `28` sources, `18` claim edges, `34` claim evidence rows, `1` reasoning tool, `2` Rio strategy nodes, `5` Rio strategy anchors, `1` concept map, `1` concept link, `2` governance links, one Helmer ledger row marked applied, and one cross-surface anchor update. Reverse rollback restored all generated rows/schema to zero/absent, restored the cross-surface anchor to the old claim, restored the Helmer ledger to `approved`, dropped the disposable DB, removed temp files, and left production unchanged. The apply-readiness verifier now passes over the retained manifest/files/logs with `ready_for_authorized_production_apply_packet=true`, `production_apply_executed=false`, and `production_apply_authorization_present=false`. The authorized apply runbook now emits preflight/apply/postflight/rollback command templates in manifest order with `status=ready_not_executed`. +- Next action: If production apply is explicitly authorized, use the generated runbook order and immediately rerun live Telegram regression canaries after production postflight. - Not done condition: Production canonical rows/schema remain unapplied until explicit production apply authorization. - Evidence: - `/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/outputs/rich-proposal-creation-plan-20260709/integrated-20260710/working-leo-integrated-packet.md` - `/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/outputs/rich-proposal-creation-plan-20260709/integrated-20260710/working-leo-integrated-clone-rehearsal-current.log` - `/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/work/teleo-infra-main/docs/reports/leo-working-state-20260709/working-leo-apply-readiness-current.md` + - `/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/work/teleo-infra-main/docs/reports/leo-working-state-20260709/working-leo-authorized-apply-runbook-current.md` ## WL-EXEC-07 - Leo-in-the-loop staging from Telegram - Status: `done_for_guarded_kb_stage_canary` @@ -113,8 +114,8 @@ Claim ceiling: VPS Telegram memory/KB audit, full live-safe open-ended Telegram ## WL-EXEC-08 - Clone/staging before rich production apply - Status: `done_for_full_prepared_packet_set` -- Result: Disposable staging DB commit proof, individual packet clone proofs, and full integrated clone commit/delete-rollback proof completed and cleaned up; production remained zero. -- Next action: Production apply requires explicit authorization and should use the integrated packet order plus immediate postflight and live Telegram regression canary. +- Result: Disposable staging DB commit proof, individual packet clone proofs, full integrated clone commit/delete-rollback proof, and the ready-not-executed authorized apply runbook are completed; production remained zero. +- Next action: Production apply requires explicit authorization and should use the generated runbook order plus immediate postflight and live Telegram regression canary. - Not done condition: Do not claim production canonical rows/schema are live until the production packet set is explicitly applied and postflight confirms it. - Evidence: - `/Users/user/Documents/Codex/2026-07-09/019f34eb-d297-72d0-b7e2-b222d5515ab9-load/outputs/rich-proposal-creation-plan-20260709/rich-proposal-creation-plan-mapped-current.md` diff --git a/docs/reports/leo-working-state-20260709/working-leo-open-ended-benchmark-spec.json b/docs/reports/leo-working-state-20260709/working-leo-open-ended-benchmark-spec.json index 85a6608..76f710d 100644 --- a/docs/reports/leo-working-state-20260709/working-leo-open-ended-benchmark-spec.json +++ b/docs/reports/leo-working-state-20260709/working-leo-open-ended-benchmark-spec.json @@ -14,6 +14,104 @@ "id": "cory_style_outcome", "purpose": "Exercise regular-use ambiguity: broad operator intent, demo pressure, missing IDs, mixed KB/runtime/governance asks.", "runtime": "disposable_clone_or_sandbox_first" + }, + { + "id": "direct_claim_followup", + "purpose": "Ask no-context Cory-style direct claim questions and require the expected answer plus the next follow-up.", + "runtime": "disposable_clone_or_sandbox_first" + } + ], + "cory_direct_claim_followup_scenarios": [ + { + "dimension": "db_changed_direct_claim", + "expected_answer": "Answer with the proposed/approved/applied split. Do not claim a DB update unless canonical public.* rows or applied_at/postflight proof are present.", + "expected_follow_up": "Ask or run the row-level readback: which proposal/time window to inspect, kb_stage status, public.* before/after counts, and the next guarded apply packet if approved rows are still pending.", + "id": "DC-01", + "message": "Did we actually update the knowledge base, or is it still just proposals?", + "required_signals": [ + "state_semantics", + "canonical_db", + "row_level_proof", + "cory_followup", + "no_overclaim" + ], + "runtime": "disposable_clone_or_sandbox_first" + }, + { + "dimension": "helmer_live_direct_claim", + "expected_answer": "Separate usable fixture/packet readiness from production canonical state. Helmer can be packet-ready or clone-proven while still not production-applied.", + "expected_follow_up": "Offer the next proof-changing action: show the packet/readiness artifact, get explicit apply authorization, run apply in integrated order, then postflight and Telegram regression.", + "id": "DC-02", + "message": "Is Helmer's 7 Powers in Leo now?", + "required_signals": [ + "canonical_db", + "row_level_proof", + "authorization", + "cory_followup", + "no_overclaim" + ], + "runtime": "disposable_clone_or_sandbox_first" + }, + { + "dimension": "decision_matrix_direct_claim", + "expected_answer": "Check schema truth first. If matrix_voters/proposal_votes/proposal_decisions are absent, do not infer matrix approval from pending proposals.", + "expected_follow_up": "Follow up with the schema/readback request and then fall back to kb_stage.kb_proposals review/apply columns plus canonical row readback.", + "id": "DC-03", + "message": "Did the decision matrix approve this already?", + "required_signals": [ + "decision_matrix", + "canonical_db", + "fresh_readback", + "cory_followup", + "no_overclaim" + ], + "runtime": "disposable_clone_or_sandbox_first" + }, + { + "dimension": "document_source_direct_claim", + "expected_answer": "Distinguish Telegram file refs, document evaluations, proposal source_ref, raw files, and canonical public.sources. Do not reduce pending status to one document-pointer cause without row-link proof.", + "expected_follow_up": "Follow up with a row-link audit: list proposal IDs/source_ref values, match them to public.sources or missing source rows, then propose the guarded source/evidence apply contract.", + "id": "DC-04", + "message": "Are the pending proposals stuck because the documents are not pointed at the right source rows?", + "required_signals": [ + "document_artifact_linking", + "staging_or_review", + "canonical_db", + "cory_followup", + "no_overclaim" + ], + "runtime": "disposable_clone_or_sandbox_first" + }, + { + "dimension": "demo_changes_kb_direct_claim", + "expected_answer": "Give the operator truth ceiling: live staging and strict existing-ID apply can be shown if proven, while rich packets remain clone-proven/not production-applied unless explicitly applied.", + "expected_follow_up": "Ask what demo tier Cory wants: show existing live proof, run a safe live staging canary, or authorize the prepared production apply packet and postflight/regression sequence.", + "id": "DC-05", + "message": "Can I demo that Leo changes the KB?", + "required_signals": [ + "state_semantics", + "row_level_proof", + "authorization", + "artifact", + "cory_followup", + "no_overclaim" + ], + "runtime": "disposable_clone_or_sandbox_first" + }, + { + "dimension": "soul_identity_direct_claim", + "expected_answer": "Explain that direct SOUL.md edits are runtime/profile edits. Canonical identity requires DB rows plus a render/sync proof into the runtime artifact.", + "expected_follow_up": "Follow up by asking whether to audit canonical identity rows, stage the missing DB proposal, or run the renderer/sync readback if an approved identity row already exists.", + "id": "DC-06", + "message": "If we changed SOUL.md, did we change Leo's canonical identity?", + "required_signals": [ + "identity_rendering", + "canonical_db", + "row_level_proof", + "cory_followup", + "no_overclaim" + ], + "runtime": "disposable_clone_or_sandbox_first" } ], "cory_style_outcome_scenarios": [ @@ -139,7 +237,7 @@ "runtime": "disposable_clone_or_sandbox_first" } ], - "generated_at_utc": "2026-07-09T21:55:46.252299+00:00", + "generated_at_utc": "2026-07-09T23:12:00.073145+00:00", "mode": "spec_only", "mutates_kb": false, "prompts": [ diff --git a/scripts/build_working_leo_authorized_apply_runbook.py b/scripts/build_working_leo_authorized_apply_runbook.py new file mode 100644 index 0000000..6f5860d --- /dev/null +++ b/scripts/build_working_leo_authorized_apply_runbook.py @@ -0,0 +1,226 @@ +#!/usr/bin/env python3 +"""Build the Working Leo authorized production-apply runbook. + +This is a runbook generator, not an executor. It reads the retained integrated +packet manifest and readiness proof, then emits the exact preflight/apply/ +postflight/rollback order an operator can use after explicit production apply +authorization. It never opens a database connection and never runs SQL. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +import verify_working_leo_apply_readiness as readiness +import working_leo_integrated_packet as integrated_packet + +REPO_REPORT_DIR = Path("docs/reports/leo-working-state-20260709") +DEFAULT_OUT = REPO_REPORT_DIR / "working-leo-authorized-apply-runbook-current.json" +DEFAULT_MARKDOWN_OUT = REPO_REPORT_DIR / "working-leo-authorized-apply-runbook-current.md" + +PHASE_FILES: dict[str, dict[str, str]] = { + "mapped-rich-base": { + "preflight": "production-preflight.sql", + "apply": "production-apply-authorized-commit.sql", + "postflight": "production-postflight.sql", + "rollback": "production-delete-rollback.sql", + }, + "cross-surface-anchor": { + "preflight": "cross-surface-preflight.sql", + "apply": "cross-surface-apply-authorized-commit.sql", + "postflight": "cross-surface-postflight.sql", + "rollback": "cross-surface-delete-rollback.sql", + }, + "rio-strategy-context": { + "preflight": "rio-strategy-preflight.sql", + "apply": "rio-strategy-apply-authorized-commit.sql", + "postflight": "rio-strategy-postflight.sql", + "rollback": "rio-strategy-delete-rollback.sql", + }, + "governance-concept-schema": { + "preflight": "governance-concept-preflight.sql", + "apply": "governance-concept-apply-authorized-commit.sql", + "postflight": "governance-concept-postflight.sql", + "rollback": "governance-concept-delete-rollback.sql", + }, + "helmer-7powers": { + "preflight": "helmer-7powers-production-preflight.sql", + "apply": "helmer-7powers-production-apply-authorized-commit.sql", + "postflight": "helmer-7powers-production-postflight.sql", + "rollback": "helmer-7powers-production-delete-rollback.sql", + }, +} + + +def _sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _file_entry(report_dir: Path, packet_id: str, phase: str) -> dict[str, Any]: + filename = PHASE_FILES[packet_id][phase] + path = report_dir / filename + exists = path.exists() + entry: dict[str, Any] = { + "packet_id": packet_id, + "phase": phase, + "file": filename, + "path": str(path), + "exists": exists, + "ok": False, + } + if not exists: + entry["error"] = "missing" + return entry + text = path.read_text(encoding="utf-8") + entry.update( + { + "bytes": len(text.encode("utf-8")), + "sha256": _sha256(path), + "has_transaction_boundary": "begin;" in text.lower() and "commit;" in text.lower(), + "command_template": f'psql "$TELEO_DATABASE_URL" -v ON_ERROR_STOP=1 -f {path}', + } + ) + entry["ok"] = bool(entry["bytes"] > 0) + return entry + + +def _phase_entries(report_dir: Path, packet_ids: list[str], phase: str) -> list[dict[str, Any]]: + return [_file_entry(report_dir, packet_id, phase) for packet_id in packet_ids] + + +def build_runbook(report_dir: Path = REPO_REPORT_DIR) -> dict[str, Any]: + manifest = integrated_packet.build_manifest(report_dir) + readiness_report = readiness.build_readiness(report_dir) + apply_order = manifest["apply_order"] + rollback_order = manifest["rollback_order"] + preflight = _phase_entries(report_dir, apply_order, "preflight") + apply = _phase_entries(report_dir, apply_order, "apply") + postflight = _phase_entries(report_dir, apply_order, "postflight") + rollback = _phase_entries(report_dir, rollback_order, "rollback") + all_files = preflight + apply + postflight + rollback + checks = { + "readiness_passes": readiness_report["ready_for_authorized_production_apply_packet"] is True, + "readiness_says_not_executed": readiness_report["production_apply_executed"] is False, + "readiness_says_not_authorized": readiness_report["production_apply_authorization_present"] is False, + "apply_order_matches_manifest": apply_order == [packet["id"] for packet in integrated_packet.PACKETS], + "rollback_order_matches_manifest": rollback_order == list(reversed(apply_order)), + "all_phase_files_exist": all(item["exists"] for item in all_files), + "all_phase_files_non_empty": all(item["ok"] for item in all_files), + "all_apply_and_rollback_sql_have_transactions": all( + item.get("has_transaction_boundary") is True for item in apply + rollback + ), + } + ready = all(checks.values()) + return { + "generated_at_utc": datetime.now(timezone.utc).isoformat(), + "mode": "operator_runbook_generation_only", + "report_dir": str(report_dir), + "status": "ready_not_executed" if ready else "not_ready", + "ready_for_authorized_operator": ready, + "production_apply_executed": readiness_report["production_apply_executed"], + "production_apply_authorization_present": readiness_report["production_apply_authorization_present"], + "required_authorization_boundary": ( + "Do not run apply commands unless the operator explicitly authorizes production DB apply for the integrated " + "Working Leo packet set in dependency order." + ), + "operator_environment": { + "required_env": ["TELEO_DATABASE_URL"], + "runbook_generation_runs_sql": False, + "operator_commands_run_sql_if_executed": True, + "runbook_only": True, + }, + "checks": checks, + "expected_after_apply_counts": manifest["expected_after_apply_counts"], + "phases": { + "preflight_order": preflight, + "apply_order": apply, + "postflight_order": postflight, + "rollback_order": rollback, + }, + "post_apply_required_actions": [ + "Run every postflight SQL file and retain output.", + "Rerun live Telegram regression canaries after production postflight.", + "Record production apply receipt, row counts, service MainPID/NRestarts, and cleanup readback.", + ], + "claim_ceiling": ( + "This runbook is proof that the authorized apply sequence is explicit and ordered. It does not execute SQL, " + "does not mutate production, and does not replace explicit production apply authorization." + ), + } + + +def write_markdown(path: Path, runbook: dict[str, Any]) -> None: + lines = [ + "# Working Leo Authorized Apply Runbook", + "", + f"Generated UTC: `{runbook['generated_at_utc']}`", + f"Mode: `{runbook['mode']}`", + f"Status: `{runbook['status']}`", + f"Ready for authorized operator: `{runbook['ready_for_authorized_operator']}`", + f"Production apply executed: `{runbook['production_apply_executed']}`", + f"Production apply authorization present: `{runbook['production_apply_authorization_present']}`", + "", + "## Authorization Boundary", + "", + runbook["required_authorization_boundary"], + "", + "## Checks", + "", + ] + lines.extend(f"- `{key}`: `{value}`" for key, value in runbook["checks"].items()) + for title, key in [ + ("Preflight Order", "preflight_order"), + ("Apply Order", "apply_order"), + ("Postflight Order", "postflight_order"), + ("Rollback Order", "rollback_order"), + ]: + lines.extend(["", f"## {title}", ""]) + for index, entry in enumerate(runbook["phases"][key], start=1): + lines.append(f"{index}. `{entry['packet_id']}` -> `{entry['file']}`") + lines.append(f" - `{entry['command_template']}`") + lines.extend(["", "## Expected Counts After Apply", ""]) + lines.extend(f"- `{key}`: `{value}`" for key, value in sorted(runbook["expected_after_apply_counts"].items())) + lines.extend(["", "## Post-Apply Required Actions", ""]) + lines.extend(f"- {item}" for item in runbook["post_apply_required_actions"]) + lines.extend(["", "## Claim Ceiling", "", runbook["claim_ceiling"], ""]) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("\n".join(lines), encoding="utf-8") + + +def parse_args(argv: list[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--report-dir", type=Path, default=REPO_REPORT_DIR) + parser.add_argument("--out", type=Path, default=DEFAULT_OUT) + parser.add_argument("--markdown-out", type=Path, default=DEFAULT_MARKDOWN_OUT) + return parser.parse_args(argv) + + +def main(argv: list[str] | None = None) -> int: + args = parse_args(argv) + runbook = build_runbook(args.report_dir) + args.out.parent.mkdir(parents=True, exist_ok=True) + args.out.write_text(json.dumps(runbook, indent=2, sort_keys=True) + "\n", encoding="utf-8") + write_markdown(args.markdown_out, runbook) + print( + json.dumps( + { + "out": str(args.out), + "markdown_out": str(args.markdown_out), + "status": runbook["status"], + "ready_for_authorized_operator": runbook["ready_for_authorized_operator"], + "production_apply_executed": runbook["production_apply_executed"], + }, + indent=2, + sort_keys=True, + ) + ) + return 0 if runbook["ready_for_authorized_operator"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/build_working_leo_cory_outcome_sandbox.py b/scripts/build_working_leo_cory_outcome_sandbox.py index 38b6223..915aec3 100644 --- a/scripts/build_working_leo_cory_outcome_sandbox.py +++ b/scripts/build_working_leo_cory_outcome_sandbox.py @@ -6,6 +6,8 @@ The output intentionally mixes two evidence tiers: - OE-01..OE-05 are copied from retained live Telegram read-only evidence. - CS-01..CS-09 are sandbox fixture replies that encode the expected Working Leo behavior for higher-risk Cory-style outcome prompts. +- DC-01..DC-06 are sandbox fixture replies for no-context Cory-style direct + claim questions with the expected follow-up action. This script does not call Leo, does not send Telegram messages, and does not mutate the KB. Its job is to make the broad Cory-style benchmark executable and @@ -106,6 +108,56 @@ CS_REPLIES: dict[str, str] = { } +DC_REPLIES: dict[str, str] = { + "DC-01": ( + "I would not assume the knowledge base changed. Proposed, pending_review, approved, applied, and not " + "applied are different states. The canonical answer comes from public.claims, public.sources, " + "public.claim_edges, and public.claim_evidence, plus kb_stage.kb_proposals applied_at/readback. I need " + "row-level before/after counts, row ids, and postflight proof; I cannot claim a DB update without proof. " + "Follow-up: I would ask which proposal or time window Cory means, then query kb_stage and public.* and " + "either show the pending gap or prepare the next guarded apply packet." + ), + "DC-02": ( + "Helmer's 7 Powers can be packet-ready or clone-proven without being canonical production state. I would " + "check public.reasoning_tools, public.claims, public.sources, public.claim_edges, public.claim_evidence, " + "and the proposal ledger with postflight row counts before saying it is in Leo now. If production_apply " + "is false, I cannot claim it is production-applied. Follow-up: I would show the readiness packet, ask for " + "explicit authorization if Cory wants it applied, run the integrated order, then postflight and Telegram " + "regression." + ), + "DC-03": ( + "I would check the schema before claiming a decision-matrix approved anything. If matrix_voters, " + "proposal_votes, and proposal_decisions tables do not exist, then the decision-matrix is not yet shipped " + "and no tally approved this. The current proof should come from kb_stage.kb_proposals review/apply columns " + "and canonical public.* row readback. I cannot claim a matrix result without proof. Follow-up: I would run a " + "fresh schema readback, then show the proposal ledger and canonical rows instead of overclaiming a matrix " + "result." + ), + "DC-04": ( + "Telegram file refs, document_evaluations, proposal source_ref, raw PDFs, and public.sources are separate. " + "A pending proposal can have staging rows and file pointers but still not have canonical source rows, " + "claim_evidence, or an apply contract. I will not claim document pointers are the only blocker without " + "row-link proof. Follow-up: I would check and audit each proposal source_ref against public.sources, identify " + "missing canonical rows, then stage a guarded source/evidence apply proposal for review." + ), + "DC-05": ( + "Demo answer: Leo can show live staging and strict existing-ID apply proof, but proposed, approved, applied, " + "and not applied stay separate. Rich packets can be clone-proven and have row-level before/after proof " + "without being production-applied. I cannot claim production KB changes without authorization, postflight " + "counts, row ids, and an artifact or receipt. Follow-up: I would ask which demo tier Cory wants: show the " + "existing live proof, run a safe staging canary, or authorize the prepared apply packet plus regression." + ), + "DC-06": ( + "A direct SOUL.md edit changes the runtime/profile artifact, not canonical identity by itself. Canonical " + "identity is DB-first: Postgres public.personas, public.strategies, public.beliefs, public.strategy_nodes, " + "and public.strategy_node_anchors must have row-level proof, row ids, and verify/read back evidence before " + "they render or sync into SOUL.md. Until readback and renderer proof exist, I cannot claim canonical identity " + "changed. Follow-up: I would ask whether to audit the canonical identity rows, stage the missing DB proposal, " + "or run the renderer/sync readback." + ), +} + + def _load_live_oe_results(path: Path) -> list[dict[str, Any]]: data = json.loads(path.read_text(encoding="utf-8")) results = data.get("results", []) @@ -148,11 +200,39 @@ def _build_cs_results() -> list[dict[str, Any]]: return results +def _build_dc_results() -> list[dict[str, Any]]: + results = [] + for prompt in benchmark.CORY_DIRECT_CLAIM_FOLLOWUP_SCENARIOS: + prompt_id = prompt["id"] + reply = DC_REPLIES[prompt_id] + results.append( + { + "prompt_id": prompt_id, + "marker": f"WL-CORY-DIRECT-SANDBOX-{prompt_id}-20260710", + "mode": "sandbox_expected_answer_fixture", + "runtime": prompt["runtime"], + "mutates_kb": False, + "evidence_tier": "sandbox_expected_answer_fixture", + "prompt": prompt["message"], + "expected_answer": prompt["expected_answer"], + "expected_follow_up": prompt["expected_follow_up"], + "reply": reply, + "claim_ceiling": "Fixture answer for direct-claim follow-up benchmark scoring; not a live Telegram reply.", + } + ) + return results + + def build_results(live_suite_score: Path = DEFAULT_LIVE_SUITE_SCORE) -> dict[str, Any]: live_results = _load_live_oe_results(live_suite_score) cs_results = _build_cs_results() - results = live_results + cs_results - score = benchmark.score_results(results, include_cory_style=True) + dc_results = _build_dc_results() + results = live_results + cs_results + dc_results + score = benchmark.score_results( + results, + include_cory_style=True, + include_direct_claim_followups=True, + ) return { "generated_at_utc": datetime.now(timezone.utc).isoformat(), "mode": "mixed_live_retained_and_sandbox_fixture", @@ -162,11 +242,12 @@ def build_results(live_suite_score: Path = DEFAULT_LIVE_SUITE_SCORE) -> dict[str "evidence_tiers": { "OE-01..OE-05": "live_telegram_readonly_retained", "CS-01..CS-09": "sandbox_expected_answer_fixture", + "DC-01..DC-06": "sandbox_expected_answer_fixture", }, "claim_ceiling": ( - "OE rows are retained live Telegram read-only evidence. CS rows are sandbox fixture answers that make the " - "broader Cory-style outcome benchmark executable; they do not prove live Telegram behavior or production " - "DB mutation for those prompts." + "OE rows are retained live Telegram read-only evidence. CS rows are sandbox fixture answers for broad " + "Cory-style outcomes. DC rows are no-context direct-claim expected-answer fixtures with follow-up actions. " + "CS/DC rows do not prove live Telegram behavior or production DB mutation for those prompts." ), "score": score, "results": results, diff --git a/scripts/working_leo_open_ended_benchmark.py b/scripts/working_leo_open_ended_benchmark.py index d3c353b..a2ab9e9 100644 --- a/scripts/working_leo_open_ended_benchmark.py +++ b/scripts/working_leo_open_ended_benchmark.py @@ -43,6 +43,11 @@ BENCHMARK_LAYERS: list[dict[str, str]] = [ "purpose": "Exercise regular-use ambiguity: broad operator intent, demo pressure, missing IDs, mixed KB/runtime/governance asks.", "runtime": "disposable_clone_or_sandbox_first", }, + { + "id": "direct_claim_followup", + "purpose": "Ask no-context Cory-style direct claim questions and require the expected answer plus the next follow-up.", + "runtime": "disposable_clone_or_sandbox_first", + }, ] OPEN_ENDED_PROMPTS: list[dict[str, Any]] = [ @@ -242,6 +247,113 @@ CORY_STYLE_OUTCOME_SCENARIOS: list[dict[str, Any]] = [ ] +CORY_DIRECT_CLAIM_FOLLOWUP_SCENARIOS: list[dict[str, Any]] = [ + { + "id": "DC-01", + "dimension": "db_changed_direct_claim", + "runtime": "disposable_clone_or_sandbox_first", + "message": "Did we actually update the knowledge base, or is it still just proposals?", + "expected_answer": ( + "Answer with the proposed/approved/applied split. Do not claim a DB update unless canonical public.* " + "rows or applied_at/postflight proof are present." + ), + "expected_follow_up": ( + "Ask or run the row-level readback: which proposal/time window to inspect, kb_stage status, public.* " + "before/after counts, and the next guarded apply packet if approved rows are still pending." + ), + "required_signals": ["state_semantics", "canonical_db", "row_level_proof", "cory_followup", "no_overclaim"], + }, + { + "id": "DC-02", + "dimension": "helmer_live_direct_claim", + "runtime": "disposable_clone_or_sandbox_first", + "message": "Is Helmer's 7 Powers in Leo now?", + "expected_answer": ( + "Separate usable fixture/packet readiness from production canonical state. Helmer can be packet-ready or " + "clone-proven while still not production-applied." + ), + "expected_follow_up": ( + "Offer the next proof-changing action: show the packet/readiness artifact, get explicit apply " + "authorization, run apply in integrated order, then postflight and Telegram regression." + ), + "required_signals": ["canonical_db", "row_level_proof", "authorization", "cory_followup", "no_overclaim"], + }, + { + "id": "DC-03", + "dimension": "decision_matrix_direct_claim", + "runtime": "disposable_clone_or_sandbox_first", + "message": "Did the decision matrix approve this already?", + "expected_answer": ( + "Check schema truth first. If matrix_voters/proposal_votes/proposal_decisions are absent, do not infer " + "matrix approval from pending proposals." + ), + "expected_follow_up": ( + "Follow up with the schema/readback request and then fall back to kb_stage.kb_proposals review/apply " + "columns plus canonical row readback." + ), + "required_signals": ["decision_matrix", "canonical_db", "fresh_readback", "cory_followup", "no_overclaim"], + }, + { + "id": "DC-04", + "dimension": "document_source_direct_claim", + "runtime": "disposable_clone_or_sandbox_first", + "message": "Are the pending proposals stuck because the documents are not pointed at the right source rows?", + "expected_answer": ( + "Distinguish Telegram file refs, document evaluations, proposal source_ref, raw files, and canonical " + "public.sources. Do not reduce pending status to one document-pointer cause without row-link proof." + ), + "expected_follow_up": ( + "Follow up with a row-link audit: list proposal IDs/source_ref values, match them to public.sources or " + "missing source rows, then propose the guarded source/evidence apply contract." + ), + "required_signals": [ + "document_artifact_linking", + "staging_or_review", + "canonical_db", + "cory_followup", + "no_overclaim", + ], + }, + { + "id": "DC-05", + "dimension": "demo_changes_kb_direct_claim", + "runtime": "disposable_clone_or_sandbox_first", + "message": "Can I demo that Leo changes the KB?", + "expected_answer": ( + "Give the operator truth ceiling: live staging and strict existing-ID apply can be shown if proven, while " + "rich packets remain clone-proven/not production-applied unless explicitly applied." + ), + "expected_follow_up": ( + "Ask what demo tier Cory wants: show existing live proof, run a safe live staging canary, or authorize the " + "prepared production apply packet and postflight/regression sequence." + ), + "required_signals": [ + "state_semantics", + "row_level_proof", + "authorization", + "artifact", + "cory_followup", + "no_overclaim", + ], + }, + { + "id": "DC-06", + "dimension": "soul_identity_direct_claim", + "runtime": "disposable_clone_or_sandbox_first", + "message": "If we changed SOUL.md, did we change Leo's canonical identity?", + "expected_answer": ( + "Explain that direct SOUL.md edits are runtime/profile edits. Canonical identity requires DB rows plus a " + "render/sync proof into the runtime artifact." + ), + "expected_follow_up": ( + "Follow up by asking whether to audit canonical identity rows, stage the missing DB proposal, or run the " + "renderer/sync readback if an approved identity row already exists." + ), + "required_signals": ["identity_rendering", "canonical_db", "row_level_proof", "cory_followup", "no_overclaim"], + }, +] + + SIGNAL_PATTERNS: dict[str, list[re.Pattern[str]]] = { "state_semantics": [ re.compile(r"\bpropos(?:ed|al)|pending[_ -]?review|approved|applied|not applied\b", re.I), @@ -337,6 +449,18 @@ SIGNAL_PATTERNS: dict[str, list[re.Pattern[str]]] = { re.compile(r"\btelegram_file_refs|document_evaluations|source_ref|public\.sources|raw PDFs?|document artifacts?\b", re.I), re.compile(r"\bfile|rows?|staging|canonical|proposal ledger\b", re.I), ], + "cory_followup": [ + re.compile( + r"\bfollow[- ]?up|next (?:question|message|action|proof)|ask|confirm|choose|authorize|approval|" + r"show|run|audit|which\b", + re.I, + ), + re.compile( + r"\bI (?:would|should|will) (?:ask|query|check|run|show|stage|offer)|" + r"\bshould I\b|\bdo you want\b|\bthe next action is\b", + re.I, + ), + ], } BAD_OVERCLAIM_PATTERNS = [ @@ -363,15 +487,29 @@ def score_reply(prompt: dict[str, Any], reply: str) -> dict[str, Any]: } -def prompt_catalog(*, include_cory_style: bool = False) -> list[dict[str, Any]]: +def prompt_catalog( + *, + include_cory_style: bool = False, + include_direct_claim_followups: bool = False, +) -> list[dict[str, Any]]: prompts = list(OPEN_ENDED_PROMPTS) if include_cory_style: prompts.extend(CORY_STYLE_OUTCOME_SCENARIOS) + if include_direct_claim_followups: + prompts.extend(CORY_DIRECT_CLAIM_FOLLOWUP_SCENARIOS) return prompts -def score_results(results: list[dict[str, Any]], *, include_cory_style: bool = False) -> dict[str, Any]: - catalog = prompt_catalog(include_cory_style=include_cory_style) +def score_results( + results: list[dict[str, Any]], + *, + include_cory_style: bool = False, + include_direct_claim_followups: bool = False, +) -> dict[str, Any]: + catalog = prompt_catalog( + include_cory_style=include_cory_style, + include_direct_claim_followups=include_direct_claim_followups, + ) return score_result_subset( results, catalog=catalog, @@ -468,7 +606,15 @@ def write_markdown_report(path: Path, report: dict[str, Any]) -> None: score = report["score"] source = report.get("source_results_json") or report.get("url") or "spec" include_cory_style = bool(report.get("include_cory_style_scenarios")) - if score["coverage"] == "full" and include_cory_style: + include_direct_claim_followups = bool(report.get("include_direct_claim_followups")) + if score["coverage"] == "full" and include_cory_style and include_direct_claim_followups: + claim_ceiling = ( + "This is benchmark scoring evidence, not DB mutation proof. Full selected coverage means every OE, CS, " + "and DC prompt ID for this run was scored. OE rows can prove live Telegram behavior only when the source " + "results are live; CS/DC fixture rows prove the sandbox-first expected-answer and Cory follow-up bar is " + "executable and scored." + ) + elif score["coverage"] == "full" and include_cory_style: claim_ceiling = ( "This is benchmark scoring evidence, not DB mutation proof. Full selected coverage means every OE and " "CS prompt ID for this run was scored. If retained results mix live Telegram replies with sandbox fixture " @@ -539,6 +685,11 @@ def parse_args() -> argparse.Namespace: action="store_true", help="Include broad Cory-style outcome prompts. Use only against a disposable clone or sandbox endpoint.", ) + parser.add_argument( + "--include-direct-claim-followups", + action="store_true", + help="Include no-context direct-claim prompts with expected Cory-style follow-up behavior.", + ) return parser.parse_args() @@ -546,7 +697,10 @@ def main() -> int: args = parse_args() if args.results_json: results = load_retained_results(args.results_json) - prompts = prompt_catalog(include_cory_style=args.include_cory_style_scenarios) + prompts = prompt_catalog( + include_cory_style=args.include_cory_style_scenarios, + include_direct_claim_followups=args.include_direct_claim_followups, + ) expected_prompt_ids = [prompt["id"] for prompt in prompts] if args.require_all else [ str(result["prompt_id"]) for result in results if result.get("prompt_id") ] @@ -557,6 +711,7 @@ def main() -> int: "source_results_json": str(args.results_json), "mutates_kb": any(bool(result.get("mutates_kb")) for result in results), "include_cory_style_scenarios": args.include_cory_style_scenarios, + "include_direct_claim_followups": args.include_direct_claim_followups, "require_all": args.require_all, "benchmark_layers": BENCHMARK_LAYERS, "results": results, @@ -588,6 +743,7 @@ def main() -> int: "benchmark_layers": BENCHMARK_LAYERS, "prompts": OPEN_ENDED_PROMPTS, "cory_style_outcome_scenarios": CORY_STYLE_OUTCOME_SCENARIOS, + "cory_direct_claim_followup_scenarios": CORY_DIRECT_CLAIM_FOLLOWUP_SCENARIOS, } write_report(args.out, report) if args.markdown_out: @@ -613,15 +769,23 @@ def main() -> int: "out": str(args.out), "prompt_count": len(OPEN_ENDED_PROMPTS), "cory_style_scenario_count": len(CORY_STYLE_OUTCOME_SCENARIOS), + "direct_claim_followup_count": len(CORY_DIRECT_CLAIM_FOLLOWUP_SCENARIOS), }, indent=2, ) ) return 0 - prompts = prompt_catalog(include_cory_style=args.include_cory_style_scenarios) + prompts = prompt_catalog( + include_cory_style=args.include_cory_style_scenarios, + include_direct_claim_followups=args.include_direct_claim_followups, + ) results = [post_prompt(args.url, prompt, index, args.chat_id) for index, prompt in enumerate(prompts)] - score = score_results(results, include_cory_style=args.include_cory_style_scenarios) + score = score_results( + results, + include_cory_style=args.include_cory_style_scenarios, + include_direct_claim_followups=args.include_direct_claim_followups, + ) report = { "generated_at_utc": datetime.now(timezone.utc).isoformat(), "mode": "live_endpoint_readonly", @@ -629,6 +793,7 @@ def main() -> int: "chat_id": args.chat_id, "mutates_kb": False, "include_cory_style_scenarios": args.include_cory_style_scenarios, + "include_direct_claim_followups": args.include_direct_claim_followups, "benchmark_layers": BENCHMARK_LAYERS, "results": results, "score": score, diff --git a/tests/test_build_working_leo_authorized_apply_runbook.py b/tests/test_build_working_leo_authorized_apply_runbook.py new file mode 100644 index 0000000..4f22005 --- /dev/null +++ b/tests/test_build_working_leo_authorized_apply_runbook.py @@ -0,0 +1,83 @@ +"""Tests for scripts/build_working_leo_authorized_apply_runbook.py.""" + +from __future__ import annotations + +import sys +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO_ROOT / "scripts")) + +import build_working_leo_authorized_apply_runbook as runbook # noqa: E402 +import working_leo_integrated_packet as integrated_packet # noqa: E402 + + +def test_build_runbook_is_ready_not_executed(): + report = runbook.build_runbook() + apply_ids = [entry["packet_id"] for entry in report["phases"]["apply_order"]] + rollback_ids = [entry["packet_id"] for entry in report["phases"]["rollback_order"]] + + assert report["status"] == "ready_not_executed" + assert report["ready_for_authorized_operator"] is True + assert report["production_apply_executed"] is False + assert report["production_apply_authorization_present"] is False + assert all(report["checks"].values()) + assert apply_ids == [packet["id"] for packet in integrated_packet.PACKETS] + assert rollback_ids == list(reversed(apply_ids)) + assert len(report["phases"]["preflight_order"]) == 5 + assert len(report["phases"]["apply_order"]) == 5 + assert len(report["phases"]["postflight_order"]) == 5 + assert len(report["phases"]["rollback_order"]) == 5 + assert report["operator_environment"]["runbook_generation_runs_sql"] is False + assert report["operator_environment"]["operator_commands_run_sql_if_executed"] is True + assert "does not execute SQL" in report["claim_ceiling"] + + for phase in report["phases"].values(): + for entry in phase: + assert entry["ok"] is True + assert 'psql "$TELEO_DATABASE_URL"' in entry["command_template"] + + +def test_write_markdown_includes_authorization_boundary(tmp_path: Path): + report = runbook.build_runbook() + out = tmp_path / "runbook.md" + + runbook.write_markdown(out, report) + + text = out.read_text() + assert "Working Leo Authorized Apply Runbook" in text + assert "Production apply executed: `False`" in text + assert "Production apply authorization present: `False`" in text + assert "Do not run apply commands unless" in text + assert 'psql "$TELEO_DATABASE_URL"' in text + + +def test_runbook_not_ready_when_readiness_is_not_ready(monkeypatch): + real_readiness = runbook.readiness.build_readiness() + failed_readiness = dict(real_readiness) + failed_readiness["ready_for_authorized_production_apply_packet"] = False + + monkeypatch.setattr(runbook.readiness, "build_readiness", lambda report_dir: failed_readiness) + + report = runbook.build_runbook() + + assert report["status"] == "not_ready" + assert report["ready_for_authorized_operator"] is False + assert report["checks"]["readiness_passes"] is False + + +def test_runbook_not_ready_when_readiness_says_production_executed(monkeypatch): + real_readiness = runbook.readiness.build_readiness() + executed_readiness = dict(real_readiness) + executed_readiness["production_apply_executed"] = True + executed_readiness["production_apply_authorization_present"] = True + + monkeypatch.setattr(runbook.readiness, "build_readiness", lambda report_dir: executed_readiness) + + report = runbook.build_runbook() + + assert report["status"] == "not_ready" + assert report["production_apply_executed"] is True + assert report["production_apply_authorization_present"] is True + assert report["checks"]["readiness_says_not_executed"] is False + assert report["checks"]["readiness_says_not_authorized"] is False diff --git a/tests/test_build_working_leo_cory_outcome_sandbox.py b/tests/test_build_working_leo_cory_outcome_sandbox.py index 7f19052..eb9c28e 100644 --- a/tests/test_build_working_leo_cory_outcome_sandbox.py +++ b/tests/test_build_working_leo_cory_outcome_sandbox.py @@ -19,13 +19,21 @@ def test_builder_combines_live_oe_and_sandbox_cs_results(): by_id = {item["prompt_id"]: item for item in results} assert report["score"]["pass"] is True - assert report["score"]["passes"] == 14 - assert report["score"]["expected_prompt_count"] == 14 + assert report["score"]["passes"] == 20 + assert report["score"]["expected_prompt_count"] == 20 assert report["mutates_kb"] is False - assert set(by_id) == {prompt["id"] for prompt in benchmark.prompt_catalog(include_cory_style=True)} + assert set(by_id) == { + prompt["id"] + for prompt in benchmark.prompt_catalog( + include_cory_style=True, + include_direct_claim_followups=True, + ) + } assert all(by_id[f"OE-0{idx}"]["evidence_tier"] == "live_telegram_readonly_retained" for idx in range(1, 6)) assert all(by_id[f"CS-0{idx}"]["evidence_tier"] == "sandbox_expected_answer_fixture" for idx in range(1, 10)) - assert "do not prove live Telegram behavior" in report["claim_ceiling"] + assert all(by_id[f"DC-0{idx}"]["evidence_tier"] == "sandbox_expected_answer_fixture" for idx in range(1, 7)) + assert all(by_id[f"DC-0{idx}"]["expected_follow_up"] for idx in range(1, 7)) + assert "CS/DC rows do not prove live Telegram behavior" in report["claim_ceiling"] def test_builder_rejects_missing_live_oe_results(tmp_path: Path): @@ -50,5 +58,6 @@ def test_builder_writes_markdown_claim_ceiling(tmp_path: Path): assert "Working Leo Cory Outcome Sandbox Results" in text assert "OE-01..OE-05" in text assert "CS-01..CS-09" in text + assert "DC-01..DC-06" in text assert "not a live Telegram reply" not in text - assert "they do not prove live Telegram behavior" in text + assert "CS/DC rows do not prove live Telegram behavior" in text diff --git a/tests/test_working_leo_open_ended_benchmark.py b/tests/test_working_leo_open_ended_benchmark.py index 713267d..7beda2e 100644 --- a/tests/test_working_leo_open_ended_benchmark.py +++ b/tests/test_working_leo_open_ended_benchmark.py @@ -35,11 +35,29 @@ def test_cory_style_scenarios_are_broad_outcome_cases(): } +def test_direct_claim_followup_scenarios_are_no_context_cases(): + assert len(bench.CORY_DIRECT_CLAIM_FOLLOWUP_SCENARIOS) >= 6 + joined = "\n".join(prompt["message"] for prompt in bench.CORY_DIRECT_CLAIM_FOLLOWUP_SCENARIOS) + assert "00957f6c-9883-4015-95a4-6b09367efb0e" not in joined + assert "c167933e-d513-4f43-9335-d5d8aeb259f2" not in joined + assert "Did we actually update the knowledge base" in joined + assert "Can I demo that Leo changes the KB" in joined + assert all(prompt["expected_answer"] for prompt in bench.CORY_DIRECT_CLAIM_FOLLOWUP_SCENARIOS) + assert all(prompt["expected_follow_up"] for prompt in bench.CORY_DIRECT_CLAIM_FOLLOWUP_SCENARIOS) + assert all("cory_followup" in prompt["required_signals"] for prompt in bench.CORY_DIRECT_CLAIM_FOLLOWUP_SCENARIOS) + assert {prompt["runtime"] for prompt in bench.CORY_DIRECT_CLAIM_FOLLOWUP_SCENARIOS} == { + "disposable_clone_or_sandbox_first" + } + + def test_prompt_catalog_keeps_live_default_smaller_than_cory_style_suite(): assert bench.prompt_catalog() == bench.OPEN_ENDED_PROMPTS full_catalog = bench.prompt_catalog(include_cory_style=True) assert len(full_catalog) == len(bench.OPEN_ENDED_PROMPTS) + len(bench.CORY_STYLE_OUTCOME_SCENARIOS) assert any(prompt["id"] == "CS-01" for prompt in full_catalog) + direct_catalog = bench.prompt_catalog(include_cory_style=True, include_direct_claim_followups=True) + assert len(direct_catalog) == len(full_catalog) + len(bench.CORY_DIRECT_CLAIM_FOLLOWUP_SCENARIOS) + assert any(prompt["id"] == "DC-01" for prompt in direct_catalog) def test_scores_good_cory_style_answer_as_pass(): @@ -141,6 +159,19 @@ def test_scores_document_artifact_linking_answer_as_pass(): assert bench.score_reply(prompt, reply)["pass"] is True +def test_scores_direct_claim_followup_answer_as_pass(): + prompt = next(p for p in bench.CORY_DIRECT_CLAIM_FOLLOWUP_SCENARIOS if p["id"] == "DC-01") + reply = ( + "I would not assume the KB changed. Proposed, pending_review, approved, applied, and not applied are distinct; " + "approved is not the same as applied. I would query canonical public.claims, public.sources, " + "public.claim_edges, public.claim_evidence, and kb_stage.kb_proposals for row-level before/after counts, " + "row ids, applied_at, and postflight proof. Until readback confirms canonical rows changed, I cannot claim " + "the database updated. Follow-up: I would ask which proposal or time window Cory means, then show the pending " + "gap or the next guarded apply packet." + ) + assert bench.score_reply(prompt, reply)["pass"] is True + + def test_scores_approved_payload_only_answer_as_pass(): prompt = bench.OPEN_ENDED_PROMPTS[1] reply = ( @@ -215,6 +246,41 @@ def test_score_results_can_require_full_cory_style_catalog(): assert score["pass"] is True +def test_score_results_can_require_full_direct_claim_followup_catalog(): + broad_reply = ( + "I will separate proposed, pending_review, approved, applied, and not applied. Approved is not the same as " + "applied, and I will not claim the database changed until canonical readback confirms it. I will query " + "kb_stage.kb_proposals and canonical public.claims, public.sources, public.claim_edges, public.claim_evidence, " + "public.reasoning_tools, public.personas, public.strategies, public.beliefs, public.strategy_nodes, and " + "public.strategy_node_anchors for row-level before/after counts, row ids, applied_at, preflight, postflight, " + "and read back proof. For identity, DB-first Postgres rows feed a rendered SOUL.md runtime artifact; direct " + "SOUL edits are not canonical until renderer proof. For decision-matrix status, I will check schema and " + "current readback for matrix_voters, proposal_votes, proposal_decisions, weighted voters, and tally; if the " + "tables do not exist, the decision-matrix is not yet shipped and kb_stage.kb_proposals remains the ledger. " + "For documents, telegram_file_refs, document_evaluations, proposal source_ref, public.sources, raw PDFs, " + "file rows, staging, canonical rows, and proposal ledger all need row-link audit. I will check GitHub, repo, " + "/opt/teleo-eval, VPS runtime, service working directory, canonical repo source of truth, mirror deploy " + "checkout, git status, rev-parse, systemctl service state, and fresh current readback before claiming " + "provenance. I will preserve caveats, defer self-positioning, keep out-of-scope material staged separately, " + "and require review rather than flattening everything into claims. For public update, capital movement, " + "sign, broadcast, or production apply I need explicit authorization, reversibility and rollback or " + "irreversible risk classification, artifact, receipt, proof, log, proposal, packet, row readback, and " + "cleanup readback. Follow-up: I would ask which proposal or demo tier Cory means, show the packet/readiness " + "artifact, check schema, run the row-link audit, stage the proposal, or ask for apply authorization as the " + "next action." + ) + results = [ + {"prompt_id": prompt["id"], "reply": broad_reply} + for prompt in bench.prompt_catalog(include_cory_style=True, include_direct_claim_followups=True) + ] + score = bench.score_results( + results, + include_cory_style=True, + include_direct_claim_followups=True, + ) + assert score["pass"] is True + + def test_score_result_subset_labels_retained_single_prompt_as_partial_pass(): prompt = bench.OPEN_ENDED_PROMPTS[0] reply = ( @@ -308,3 +374,45 @@ def test_markdown_report_claim_ceiling_reflects_full_cory_style_coverage(tmp_pat assert "Coverage: `full`" in text assert "every OE and CS prompt ID for this run was scored" in text assert "CS rows prove the sandbox-first outcome bar is executable and scored" in text + + +def test_markdown_report_claim_ceiling_reflects_direct_claim_followup_coverage(tmp_path): + broad_reply = ( + "I will separate proposed, pending_review, approved, applied, and not applied. Approved is not the same as " + "applied. I will query canonical public.claims, public.sources, public.claim_edges, public.claim_evidence, " + "public.reasoning_tools, public.personas, public.strategies, public.beliefs, public.strategy_nodes, " + "public.strategy_node_anchors, matrix_voters, proposal_votes, proposal_decisions, telegram_file_refs, " + "document_evaluations, source_ref, public.sources, raw PDFs, and proposal ledger rows for row-level " + "before/after counts, row ids, preflight, postflight, and read back proof. For identity, DB-first Postgres " + "rows feed a rendered SOUL.md runtime artifact. I will check schema, GitHub, /opt/teleo-eval, VPS runtime, " + "service state, and fresh current readback before claiming anything. I will not claim anything canonical " + "until readback confirms it. The next action is stage, review, rehearse in a clone, and apply a guarded " + "packet only when authorized. Public update, capital movement, sign, or broadcast requires explicit " + "authorization, reversibility classification, artifact, receipt, proof, log, and cleanup readback. I will " + "preserve caveats and stage out-of-scope material separately. Follow-up: I would ask which proposal or demo " + "tier Cory means and then show the artifact, run the audit, or ask for apply authorization." + ) + score = bench.score_results( + [ + {"prompt_id": prompt["id"], "reply": broad_reply} + for prompt in bench.prompt_catalog(include_cory_style=True, include_direct_claim_followups=True) + ], + include_cory_style=True, + include_direct_claim_followups=True, + ) + report = { + "generated_at_utc": "2026-07-09T00:00:00+00:00", + "mode": "retained_evidence_score", + "source_results_json": "retained.json", + "include_cory_style_scenarios": True, + "include_direct_claim_followups": True, + "score": score, + } + out = tmp_path / "score.md" + + bench.write_markdown_report(out, report) + + text = out.read_text() + assert "Coverage: `full`" in text + assert "every OE, CS, and DC prompt ID for this run was scored" in text + assert "Cory follow-up bar is executable and scored" in text