126 lines
7.9 KiB
Markdown
126 lines
7.9 KiB
Markdown
# Canary
|
|
|
|
Run one isolated stateless challenger against one persistent temporary-profile Leo `GatewayRunner` session. The
|
|
challenger asks for an evidence-bounded claim, attacks its weakest causal step, and forces a narrower review-only
|
|
revision. The run must leave Telegram, the canonical database, the deployed profile, and the gateway unchanged.
|
|
|
|
# Changes-required repair checkpoint
|
|
|
|
- Current tier is `partial_lower_tier` until one fresh clean, pushed, current-main T2 run passes the repaired gate.
|
|
- C3 now uses one shared runner/verifier predicate. Measurement-only objections never advance to Leo; the runner
|
|
regenerates up to three times and retains every rejected model-call trace without adding it to the six-turn transcript.
|
|
- The verifier validates the raw submitted list before ID mapping: exactly six object rows, exact ordered unique IDs,
|
|
and whole-report execution, tool, mutation/write, database, and row-reference provenance.
|
|
- The exact injected seventh `X1` positive-mutation turn is the sixth executable negative control and is mirrored in
|
|
`fixtures/working-leo/agent-challenger-loop-negative-controls-v1.json`.
|
|
- Failed current-main run `46b8b7834e35` is preserved under immutable run-specific names with source SHA-256
|
|
`81d71f5db918f4be0e0308fa060fa244480739ac00cbfd1bef0f6fce07421d0a`.
|
|
- Current-main repair checks pass after merging `origin/main` at
|
|
`33399bd9acfa09e4e5409873e86c8c46ac694a8d`: 52 expanded relevant and 1,536 full-suite tests under the isolated
|
|
dev environment. All four reviewer-named Python files pass Ruff lint and format checks. Fresh current-main runtime
|
|
evidence remains pending.
|
|
|
|
# What changed
|
|
|
|
- Added a six-turn `C1/L1/C2/L2/C3/L3` live protocol with exact transcript-prefix binding.
|
|
- Added a stateless Gemini challenger with no profile, memory, tools, or database access.
|
|
- Added a temporary Leo profile with model tools disabled, a fail-closed read-only KB wrapper, and a canary-only
|
|
context policy that cannot replace the final objection response with an unrelated compiled capability response.
|
|
- Added per-turn model/session/process/profile bindings, database retrieval receipts, and a review-only proposal packet.
|
|
- Added six executed negative controls: collusion/leaked ID/silent apply, repeated execution identity, unbound row
|
|
reference, forbidden or unknown mutation tool, an unexpected seventh positive-mutation turn, and unrelated
|
|
keyword-stuffed revision.
|
|
- Added transport/source attribution, recovery readback, full database fingerprint comparison, service/behavior
|
|
invariance, cleanup verification, focused tests, and retained live evidence.
|
|
|
|
# Operator command
|
|
|
|
```bash
|
|
python3 -B scripts/run_leo_agent_challenger_loop.py
|
|
```
|
|
|
|
The lane used the preserved project virtual environment because system Python does not include pytest.
|
|
|
|
# Historical pre-review evidence
|
|
|
|
The following accepted run predates the changes-required predicates and is retained as history, not current closure.
|
|
|
|
- Live run: `7c6a2937042c`
|
|
- Runtime source commit: `af5af06130c493dd4c99c9e53638fc5b239dbee3`
|
|
- Retention-safe source report SHA-256: `4fd4269b795c32629d4b250cb66d56ceadaa93eaaa1f8f00a5ec19d76bf4f09e`
|
|
- Recovered raw remote report SHA-256: `6a7a9160095e77e5c42b7d8a2c995d3eb1e84f40624033a94ad49d616bb548a8`
|
|
- Proposal packet SHA-256: `89c543b8a12e922044f93bcc6b1e23a3a252b044dc1d45163575b70bc1a3c100`
|
|
- Database fingerprint SHA-256: `32a1f2c18f4b4c623443cbd4d9520982e9f291ec74ca63855cef51c88ac56f24`
|
|
- Standalone verifier: `pass`, current tier `T2_runtime`, zero failed checks.
|
|
- Historical negative controls: 5/5 caught under the pre-review verifier.
|
|
- Independent post-live judge: `ACCEPT — T2_runtime`, no blocking findings.
|
|
- Focused suite: 36 passed.
|
|
- Merged `origin/main` commit `15c30f1fbe30c8f2295ddc33e293a45788a95706` without rewriting the feature commits.
|
|
- Post-merge focused suite: 36 passed in 0.19 seconds.
|
|
- Post-merge full suite: 1297 passed in 106.35 seconds.
|
|
- Ruff and `git diff --check`: pass.
|
|
- Post-hygiene focused suite: 39 passed in 0.25 seconds.
|
|
|
|
Retained files:
|
|
|
|
- `docs/reports/leo-working-state-20260709/leo-agent-challenger-loop-source-current.json`
|
|
- `docs/reports/leo-working-state-20260709/leo-agent-challenger-loop-receipt-current.json`
|
|
- `docs/reports/leo-working-state-20260709/leo-agent-challenger-loop-negative-controls-current.json`
|
|
- `docs/reports/leo-working-state-20260709/leo-agent-challenger-loop-ledger.jsonl`
|
|
|
|
# Cleanup and no-write readback
|
|
|
|
- Full 39-table canonical database fingerprint unchanged before/after.
|
|
- Gateway PID, start timestamp, active/sub states, and `NRestarts=0` unchanged.
|
|
- Deployed behavior manifest unchanged for the accepted run.
|
|
- No model tool calls; mutation preflight rejected with exit 77.
|
|
- No Telegram post and no canonical or staging mutation by the harness.
|
|
- Proposal remains `pending_review`; review incomplete; apply unauthorized and unexecuted.
|
|
- Leo worker stopped gracefully with exit 0; no orphan worker; temporary roots removed.
|
|
- Initial report fetch timed out after the completed run. The exact remote file was recovered by SCP, hashed, then the
|
|
remote path was unlinked and verified absent. The recovery contract is retained without the private path value.
|
|
- The retained source projects both behavior manifests to their aggregate hashes, counts, and byte totals. It removes
|
|
2,658 per-file inventory rows and all absolute runtime paths while retaining the raw remote report hash and verifier
|
|
contracts. The committed source is 1,402 lines rather than the original 17,330-line transport report.
|
|
|
|
# Cost and authorization boundary
|
|
|
|
- Successful challenger usage: 2,201 prompt + 361 completion tokens; configured Gemini Flash estimate about $0.00055.
|
|
- Successful Leo usage: three OpenRouter Sonnet calls, 46,198 total reported tokens. At the repo's uncached list rates,
|
|
this is an upper-bound estimate of about $0.17; provider cache accounting may reduce the actual charge.
|
|
- The command authorizes only this bounded no-send/no-write canary. It does not authorize Telegram delivery, proposal
|
|
staging, approval, apply, production deployment, or canonical mutation.
|
|
|
|
# Not shipped
|
|
|
|
- T3 deployment of this orchestrator.
|
|
- Human acceptance of the revised proposal.
|
|
- Telegram-visible delivery.
|
|
- Proposal staging, approval, or canonical apply.
|
|
|
|
# Review focus
|
|
|
|
- Verify the temporary-profile isolation and context-only patch cannot affect the live profile.
|
|
- Verify semantic linkage rejects unrelated revisions without overfitting exact prose.
|
|
- Verify transport recovery/source attribution and the six negative controls.
|
|
- Verify the claim remains capped at one T2 no-send/no-write exchange.
|
|
|
|
# Integration hygiene
|
|
|
|
- Removed the lane-local root `goal.md`; the durable control copy remains outside the repository.
|
|
- Regenerated the retained source, verifier receipt, negative-control receipt, and ledger entry from the accepted live
|
|
run rather than discarding the existing receipt changes.
|
|
- Reduced the retained source from 17,330 to 1,402 lines by removing 2,658 duplicated per-file behavior inventory
|
|
entries. Aggregate behavior/component hashes, counts, byte totals, raw remote report hash, and role contracts remain.
|
|
- Targeted scan of the retained source: zero high-confidence secrets, absolute runtime paths, emails, or IPv4 values.
|
|
- Targeted scan of the complete PR delta: zero high-confidence secrets, local private paths, emails, or IPv4 values.
|
|
The 28 remaining path matches are required remote runtime constants or synthetic fail-closed test fixtures.
|
|
- The historical standalone verifier passed at `T2_runtime` with all five pre-review negative controls caught; a fresh
|
|
six-control current-main receipt is required before closure.
|
|
|
|
# Delivery and worktree lifecycle
|
|
|
|
- Delivery state: open PR [#152](https://github.com/living-ip/teleo-infrastructure/pull/152), mergeable at creation.
|
|
- Worktree state: `retained_unintegrated` while PR #152 is open.
|
|
- Per `WORKTREE_LIFECYCLE.md`, this worker does not remove or prune the worktree. The integration owner owns cleanup
|
|
only after merge, deliberate port, or explicit rejection is verified.
|