teleo/teleo-infrastructure
8
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions 1
teleo-infrastructure/docs/reports/leo-working-state-20260709/leo-agent-challenger-loop-pr-current.md
twentyOne2x da84d4703e docs: record challenger delivery validation
2026-07-15 03:38:02 +02:00

5.1 KiB
Raw Blame History

Canary

Run one isolated stateless challenger against one persistent temporary-profile Leo GatewayRunner session. The challenger asks for an evidence-bounded claim, attacks its weakest causal step, and forces a narrower review-only revision. The run must leave Telegram, the canonical database, the deployed profile, and the gateway unchanged.

What changed

  • Added a six-turn C1/L1/C2/L2/C3/L3 live protocol with exact transcript-prefix binding.
  • Added a stateless Gemini challenger with no profile, memory, tools, or database access.
  • Added a temporary Leo profile with model tools disabled, a fail-closed read-only KB wrapper, and a canary-only context policy that cannot replace the final objection response with an unrelated compiled capability response.
  • Added per-turn model/session/process/profile bindings, database retrieval receipts, and a review-only proposal packet.
  • Added five executed negative controls: collusion/leaked ID/silent apply, repeated execution identity, unbound row reference, forbidden or unknown mutation tool, and unrelated keyword-stuffed revision.
  • Added transport/source attribution, recovery readback, full database fingerprint comparison, service/behavior invariance, cleanup verification, focused tests, and retained live evidence.

Operator command

python3 -B scripts/run_leo_agent_challenger_loop.py

The lane used the preserved project virtual environment because system Python does not include pytest.

Evidence

  • Live run: 7c6a2937042c
  • Runtime source commit: af5af06130c493dd4c99c9e53638fc5b239dbee3
  • Source report SHA-256: 343f18688334d5166bbb951f8672ccf4ff5a3a2b14e05f5279d3e1165b76f83b
  • Recovered raw remote report SHA-256: 6a7a9160095e77e5c42b7d8a2c995d3eb1e84f40624033a94ad49d616bb548a8
  • Proposal packet SHA-256: 89c543b8a12e922044f93bcc6b1e23a3a252b044dc1d45163575b70bc1a3c100
  • Database fingerprint SHA-256: 32a1f2c18f4b4c623443cbd4d9520982e9f291ec74ca63855cef51c88ac56f24
  • Standalone verifier: pass, current tier T2_runtime, zero failed checks.
  • Negative controls: 5/5 caught.
  • Independent post-live judge: ACCEPT — T2_runtime, no blocking findings.
  • Focused suite: 36 passed.
  • Merged origin/main commit 15c30f1fbe30c8f2295ddc33e293a45788a95706 without rewriting the feature commits.
  • Post-merge focused suite: 36 passed in 0.19 seconds.
  • Post-merge full suite: 1297 passed in 106.35 seconds.
  • Ruff and git diff --check: pass.

Retained files:

  • docs/reports/leo-working-state-20260709/leo-agent-challenger-loop-source-current.json
  • docs/reports/leo-working-state-20260709/leo-agent-challenger-loop-receipt-current.json
  • docs/reports/leo-working-state-20260709/leo-agent-challenger-loop-negative-controls-current.json
  • docs/reports/leo-working-state-20260709/leo-agent-challenger-loop-ledger.jsonl

Cleanup and no-write readback

  • Full 39-table canonical database fingerprint unchanged before/after.
  • Gateway PID, start timestamp, active/sub states, and NRestarts=0 unchanged.
  • Deployed behavior manifest unchanged for the accepted run.
  • No model tool calls; mutation preflight rejected with exit 77.
  • No Telegram post and no canonical or staging mutation by the harness.
  • Proposal remains pending_review; review incomplete; apply unauthorized and unexecuted.
  • Leo worker stopped gracefully with exit 0; no orphan worker; temporary roots removed.
  • Initial report fetch timed out after the completed run. The exact remote file was recovered by SCP, hashed, then the remote path was unlinked and verified absent. The recovery contract is retained in the source report.

Cost and authorization boundary

  • Successful challenger usage: 2,201 prompt + 361 completion tokens; configured Gemini Flash estimate about $0.00055.
  • Successful Leo usage: three OpenRouter Sonnet calls, 46,198 total reported tokens. At the repo's uncached list rates, this is an upper-bound estimate of about $0.17; provider cache accounting may reduce the actual charge.
  • The command authorizes only this bounded no-send/no-write canary. It does not authorize Telegram delivery, proposal staging, approval, apply, production deployment, or canonical mutation.

Not shipped

  • T3 deployment of this orchestrator.
  • Human acceptance of the revised proposal.
  • Telegram-visible delivery.
  • Proposal staging, approval, or canonical apply.

Review focus

  • Verify the temporary-profile isolation and context-only patch cannot affect the live profile.
  • Verify semantic linkage rejects unrelated revisions without overfitting exact prose.
  • Verify transport recovery/source attribution and the five negative controls.
  • Verify the claim remains capped at one T2 no-send/no-write exchange.

Delivery and worktree lifecycle

  • Delivery state before PR creation: canonical_commits; PR creation follows the final origin/main integration test.
  • Worktree state: retained_active until push and PR readback, then retained_unintegrated while the PR is open.
  • Per WORKTREE_LIFECYCLE.md, this worker does not remove or prune the worktree. The integration owner owns cleanup only after merge, deliberate port, or explicit rejection is verified.