88 lines
4.9 KiB
Markdown
88 lines
4.9 KiB
Markdown
# Canary
|
|
|
|
Run one isolated stateless challenger against one persistent temporary-profile Leo `GatewayRunner` session. The
|
|
challenger asks for an evidence-bounded claim, attacks its weakest causal step, and forces a narrower review-only
|
|
revision. The run must leave Telegram, the canonical database, the deployed profile, and the gateway unchanged.
|
|
|
|
# What changed
|
|
|
|
- Added a six-turn `C1/L1/C2/L2/C3/L3` live protocol with exact transcript-prefix binding.
|
|
- Added a stateless Gemini challenger with no profile, memory, tools, or database access.
|
|
- Added a temporary Leo profile with model tools disabled, a fail-closed read-only KB wrapper, and a canary-only
|
|
context policy that cannot replace the final objection response with an unrelated compiled capability response.
|
|
- Added per-turn model/session/process/profile bindings, database retrieval receipts, and a review-only proposal packet.
|
|
- Added five executed negative controls: collusion/leaked ID/silent apply, repeated execution identity, unbound row
|
|
reference, forbidden or unknown mutation tool, and unrelated keyword-stuffed revision.
|
|
- Added transport/source attribution, recovery readback, full database fingerprint comparison, service/behavior
|
|
invariance, cleanup verification, focused tests, and retained live evidence.
|
|
|
|
# Operator command
|
|
|
|
```bash
|
|
python3 -B scripts/run_leo_agent_challenger_loop.py
|
|
```
|
|
|
|
The lane used the preserved project virtual environment because system Python does not include pytest.
|
|
|
|
# Evidence
|
|
|
|
- Live run: `7c6a2937042c`
|
|
- Runtime source commit: `af5af06130c493dd4c99c9e53638fc5b239dbee3`
|
|
- Source report SHA-256: `343f18688334d5166bbb951f8672ccf4ff5a3a2b14e05f5279d3e1165b76f83b`
|
|
- Recovered raw remote report SHA-256: `6a7a9160095e77e5c42b7d8a2c995d3eb1e84f40624033a94ad49d616bb548a8`
|
|
- Proposal packet SHA-256: `89c543b8a12e922044f93bcc6b1e23a3a252b044dc1d45163575b70bc1a3c100`
|
|
- Database fingerprint SHA-256: `32a1f2c18f4b4c623443cbd4d9520982e9f291ec74ca63855cef51c88ac56f24`
|
|
- Standalone verifier: `pass`, current tier `T2_runtime`, zero failed checks.
|
|
- Negative controls: 5/5 caught.
|
|
- Independent post-live judge: `ACCEPT — T2_runtime`, no blocking findings.
|
|
- Focused suite: 36 passed.
|
|
- Full suite before origin/main integration: 1289 passed in 63.33 seconds.
|
|
- Ruff and `git diff --check`: pass.
|
|
|
|
Retained files:
|
|
|
|
- `docs/reports/leo-working-state-20260709/leo-agent-challenger-loop-source-current.json`
|
|
- `docs/reports/leo-working-state-20260709/leo-agent-challenger-loop-receipt-current.json`
|
|
- `docs/reports/leo-working-state-20260709/leo-agent-challenger-loop-negative-controls-current.json`
|
|
- `docs/reports/leo-working-state-20260709/leo-agent-challenger-loop-ledger.jsonl`
|
|
|
|
# Cleanup and no-write readback
|
|
|
|
- Full 39-table canonical database fingerprint unchanged before/after.
|
|
- Gateway PID, start timestamp, active/sub states, and `NRestarts=0` unchanged.
|
|
- Deployed behavior manifest unchanged for the accepted run.
|
|
- No model tool calls; mutation preflight rejected with exit 77.
|
|
- No Telegram post and no canonical or staging mutation by the harness.
|
|
- Proposal remains `pending_review`; review incomplete; apply unauthorized and unexecuted.
|
|
- Leo worker stopped gracefully with exit 0; no orphan worker; temporary roots removed.
|
|
- Initial report fetch timed out after the completed run. The exact remote file was recovered by SCP, hashed, then the
|
|
remote path was unlinked and verified absent. The recovery contract is retained in the source report.
|
|
|
|
# Cost and authorization boundary
|
|
|
|
- Successful challenger usage: 2,201 prompt + 361 completion tokens; configured Gemini Flash estimate about $0.00055.
|
|
- Successful Leo usage: three OpenRouter Sonnet calls, 46,198 total reported tokens. At the repo's uncached list rates,
|
|
this is an upper-bound estimate of about $0.17; provider cache accounting may reduce the actual charge.
|
|
- The command authorizes only this bounded no-send/no-write canary. It does not authorize Telegram delivery, proposal
|
|
staging, approval, apply, production deployment, or canonical mutation.
|
|
|
|
# Not shipped
|
|
|
|
- T3 deployment of this orchestrator.
|
|
- Human acceptance of the revised proposal.
|
|
- Telegram-visible delivery.
|
|
- Proposal staging, approval, or canonical apply.
|
|
|
|
# Review focus
|
|
|
|
- Verify the temporary-profile isolation and context-only patch cannot affect the live profile.
|
|
- Verify semantic linkage rejects unrelated revisions without overfitting exact prose.
|
|
- Verify transport recovery/source attribution and the five negative controls.
|
|
- Verify the claim remains capped at one T2 no-send/no-write exchange.
|
|
|
|
# Delivery and worktree lifecycle
|
|
|
|
- Delivery state before PR creation: `canonical_commits`; PR creation follows the final origin/main integration test.
|
|
- Worktree state: `retained_active` until push and PR readback, then `retained_unintegrated` while the PR is open.
|
|
- Per `WORKTREE_LIFECYCLE.md`, this worker does not remove or prune the worktree. The integration owner owns cleanup
|
|
only after merge, deliberate port, or explicit rejection is verified.
|