Compare commits
No commits in common. "main" and "epimetheus/substantive-fixer-visibility" have entirely different histories.
main
...
epimetheus
91 changed files with 365 additions and 13470 deletions
|
|
@ -1,35 +0,0 @@
|
|||
# Crabbox
|
||||
|
||||
Use Crabbox for remote Linux verification and PR proof only.
|
||||
|
||||
Allowed jobs:
|
||||
|
||||
- `crabbox job run unit`
|
||||
- `crabbox job run lint-phase1b`
|
||||
- `crabbox job run ci-contract`
|
||||
- `crabbox job run phase1b-local-proof`
|
||||
- `crabbox job run sync-smoke`
|
||||
|
||||
Default workflow:
|
||||
|
||||
1. Run `crabbox job run --dry-run ci-contract`.
|
||||
2. Run `crabbox job run --dry-run phase1b-local-proof`.
|
||||
3. Inspect the planned commands and confirm no production secrets or production deploy commands appear.
|
||||
4. Run `crabbox job run ci-contract`.
|
||||
5. Run `crabbox job run phase1b-local-proof`.
|
||||
6. Save the run id, lease id, stdout, downloaded proof JSON, and JUnit output.
|
||||
7. Stop the lease unless the CLI has already stopped it.
|
||||
|
||||
Boundaries:
|
||||
|
||||
- Do not run production deploy commands from Crabbox.
|
||||
- Do not forward production GitHub, Forgejo, OpenRouter, SSH, Bitwarden, or VPS secrets.
|
||||
- Do not target the production `decision-engine` repo for sandbox proof.
|
||||
- Do not mutate the production VPS.
|
||||
- Do not call Crabbox proof equivalent to production proof unless the lease recreates `/opt/teleo-eval`, systemd services, runtime users, DB paths, timers, and deploy scripts.
|
||||
|
||||
Failure handling:
|
||||
|
||||
- If sync sanity fails, stop the lease and retry on a fresh lease.
|
||||
- If a proof script fails, save the full run output and do not summarize it as a pass.
|
||||
- If a remote box has unknown state, stop it instead of debugging against reused state.
|
||||
|
|
@ -1,41 +0,0 @@
|
|||
---
|
||||
name: decision-engine-refinement
|
||||
description: Use when improving Living IP decision-engine quality, LLM model selection, evaluator prompts, rubrics, replay evals, Rio or Theseus reviewer behavior, or model bakeoffs.
|
||||
---
|
||||
|
||||
# Decision Engine Refinement
|
||||
|
||||
Use this skill for quality work, not infrastructure work. Pentagon.run or Crabbox can run remote jobs; this repo owns model judgment, rubric design, prompt/tool refinement, and proof artifacts.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Read `docs/llm-refinement-decision-engine.md`.
|
||||
2. Identify the lane: Rio economics, Theseus model integrity, Leo cross-domain, domain factuality, retrieval quality, or prompt/tool self-upgrade.
|
||||
3. Build or reuse a replayable fixture before changing prompts or model assignments.
|
||||
4. Compare baseline vs candidate with the same input, same rubric, and structured verdict format.
|
||||
5. Record false approves, false rejects, useful disagreements, cost, and latency.
|
||||
6. Change runtime prompts/models only after the candidate shows a measured improvement with no critical regression.
|
||||
|
||||
## Hard Rules
|
||||
|
||||
- Do not change live model assignments because one answer sounds better.
|
||||
- Do not use production DB writes to tune prompts.
|
||||
- Do not collapse Rio and Theseus into generic "reviewers".
|
||||
- Do not treat payment, popularity, or engagement as quality approval.
|
||||
- Do not claim production decision-engine improvement without replay evidence and live/staging readback.
|
||||
|
||||
## Agent Responsibilities
|
||||
|
||||
- Rio: incentive design, contribution weights, paid-query effects, market/mechanism reasoning, OPSEC, correlated-prior warnings.
|
||||
- Theseus: model diversity, adversarial evals, disagreement queues, self-upgrade criteria, prompt/tool safety, verifier drift.
|
||||
- Leo: cross-domain synthesis, fallback review, final arbitration where the route or rubric is ambiguous.
|
||||
|
||||
## Expected Artifacts
|
||||
|
||||
- fixture file or DB query used for sampling;
|
||||
- baseline verdict output;
|
||||
- candidate verdict output;
|
||||
- summary JSON with quality, cost, latency, and disagreement metrics;
|
||||
- patch scoped to prompts, model config, rubric docs, or eval harness.
|
||||
|
||||
Run `python3 scripts/check_llm_refinement_contract.py` after editing this surface.
|
||||
|
|
@ -1,93 +0,0 @@
|
|||
---
|
||||
name: living-ip-kb-interop
|
||||
description: Use when giving Hermes, OpenClaw, Claude-style, Pentagon, or other external agents safe read/write access patterns for the Living IP knowledge base.
|
||||
---
|
||||
|
||||
# Living IP KB Interop
|
||||
|
||||
Use this skill when an outside agent needs to read from the Living IP knowledge base or propose a write back into it. The default is propose-first, proof-backed, and no-secret.
|
||||
|
||||
## Goal
|
||||
|
||||
Any Hermes, OpenClaw, Claude-style, or Pentagon agent should be able to:
|
||||
|
||||
1. search the knowledge base;
|
||||
2. read a cited file or record;
|
||||
3. propose a source, claim, entity, or correction;
|
||||
4. route the proposal to the right evaluator agents;
|
||||
5. leave a proof artifact that shows inputs, tools, and no denied actions.
|
||||
|
||||
## Read Path
|
||||
|
||||
Prefer deterministic local surfaces before asking an LLM:
|
||||
|
||||
- repository files under the knowledge base checkout;
|
||||
- generated claim indexes from `lib/claim_index.py`;
|
||||
- search helpers in `lib/search.py`;
|
||||
- copied SQLite state through `teleo-db-operator`;
|
||||
- retained proof JSON in `.crabbox-results/` or `proof/`.
|
||||
|
||||
Read outputs must include file paths, source paths, claim/entity IDs when available, and the exact query used.
|
||||
|
||||
## Write Path
|
||||
|
||||
All writes are proposals until the normal review/evaluation pipeline accepts them.
|
||||
|
||||
Allowed proposal targets:
|
||||
|
||||
- source file proposal;
|
||||
- claim file proposal;
|
||||
- entity file proposal;
|
||||
- correction proposal;
|
||||
- route/evaluator proof artifact.
|
||||
|
||||
Required fields:
|
||||
|
||||
- source or rationale;
|
||||
- target domain;
|
||||
- proposed author/agent;
|
||||
- route evidence;
|
||||
- confidence or uncertainty tag;
|
||||
- citations to existing KB context;
|
||||
- proof output path.
|
||||
|
||||
Do not write directly to main. Do not mutate production `pipeline.db`. Use `teleo-db-operator` for any SQLite write, and only after explicit authorization, backup, transaction, and readback.
|
||||
|
||||
## Minimal Tool Contract
|
||||
|
||||
Adapters should expose this shape even if their runtime uses different names:
|
||||
|
||||
- `kb.search(query, domain?, limit?)`
|
||||
- `kb.get(path_or_id)`
|
||||
- `kb.propose_source(markdown, metadata)`
|
||||
- `kb.propose_claim(markdown, metadata)`
|
||||
- `kb.propose_entity(markdown, metadata)`
|
||||
- `kb.route(diff_or_metadata)`
|
||||
- `kb.proof(path, payload)`
|
||||
|
||||
If a runtime cannot implement one of these, record the missing tool as a blocker instead of silently skipping it.
|
||||
|
||||
## Denied Actions
|
||||
|
||||
- raw Bitwarden export;
|
||||
- card, token, or password reads;
|
||||
- production DB writes;
|
||||
- direct pushes to main;
|
||||
- public comments or messages;
|
||||
- hidden Slack, Linear, Telegram, or GitHub sends;
|
||||
- uncited knowledge writes;
|
||||
- model-driven edits without route evidence.
|
||||
|
||||
## Expected Artifact
|
||||
|
||||
Write `.crabbox-results/kb-interop-proof.json` or a caller-specified proof path containing:
|
||||
|
||||
- runtime name;
|
||||
- model/provider if known;
|
||||
- tools invoked;
|
||||
- denied tools not invoked;
|
||||
- query or input fixture;
|
||||
- cited reads;
|
||||
- proposed writes;
|
||||
- route evidence;
|
||||
- verifier result.
|
||||
|
|
@ -1,70 +0,0 @@
|
|||
---
|
||||
name: nousresearch-hermes-agent
|
||||
description: Use when packaging Living IP agents, skills, prompts, memory, model routing, or decision-engine workflows for NousResearch Hermes Agent.
|
||||
---
|
||||
|
||||
# NousResearch Hermes Agent
|
||||
|
||||
Use this skill to adapt Living IP decision-engine behavior to Hermes Agent. Keep the package fixture-first and no-secret by default.
|
||||
|
||||
## Current External Surface
|
||||
|
||||
As of 2026-06-01, the upstream Hermes Agent README describes:
|
||||
|
||||
- model switching via `hermes model`;
|
||||
- tools via `hermes tools`;
|
||||
- a messaging gateway for Telegram, Discord, Slack, WhatsApp, Signal, and CLI;
|
||||
- built-in skill creation and self-improvement;
|
||||
- cron scheduling;
|
||||
- terminal backends including local, Docker, SSH, Modal, and Daytona;
|
||||
- OpenClaw migration commands.
|
||||
|
||||
Verify upstream docs before depending on a command in code.
|
||||
|
||||
## Living IP Package Shape
|
||||
|
||||
Create a package that includes:
|
||||
|
||||
- agent identity file for Rio or Theseus;
|
||||
- skill instructions copied from repo-owned `.agents/skills/*`;
|
||||
- `living-ip-kb-interop` for read/propose/writeback behavior;
|
||||
- no-secret tool allowlist;
|
||||
- fixture replay command;
|
||||
- model selection notes;
|
||||
- proof output path.
|
||||
|
||||
Do not package production DBs, tokens, API keys, SSH keys, or Bitwarden exports.
|
||||
|
||||
## Rio Package
|
||||
|
||||
Rio Hermes package should focus on:
|
||||
|
||||
- internet finance and mechanism reasoning;
|
||||
- contribution weights and paid-query effects;
|
||||
- OPSEC finance filters;
|
||||
- source-diversity warnings;
|
||||
- fixture tests for false economic reasoning.
|
||||
|
||||
## Theseus Package
|
||||
|
||||
Theseus Hermes package should focus on:
|
||||
|
||||
- model-diversity evals;
|
||||
- disagreement queues;
|
||||
- self-upgrade criteria;
|
||||
- prompt/tool safety;
|
||||
- fixture tests for overconfident or poorly grounded model judgments.
|
||||
|
||||
## Handoff Contract
|
||||
|
||||
Every Hermes handoff must include:
|
||||
|
||||
1. install/config snippet;
|
||||
2. model/provider selection left configurable;
|
||||
3. tool allowlist;
|
||||
4. fixture-first demo;
|
||||
5. no-live-write default;
|
||||
6. proof artifact path;
|
||||
7. known blockers.
|
||||
|
||||
Do not claim Hermes production integration until a Hermes runtime actually executes the fixture and writes proof.
|
||||
|
|
@ -1,70 +0,0 @@
|
|||
---
|
||||
name: openclaw-agent
|
||||
description: Use when adapting Living IP decision-engine agents, skills, tools, prompt files, or no-secret workflows to OpenClaw agent workspaces.
|
||||
---
|
||||
|
||||
# OpenClaw Agent
|
||||
|
||||
Use this skill to package Living IP decision-engine behavior for OpenClaw workspaces. Treat OpenClaw as a distribution/runtime surface, not a new source of truth.
|
||||
|
||||
## Current External Surface
|
||||
|
||||
As of 2026-06-01, the upstream OpenClaw README describes:
|
||||
|
||||
- Node 24 or Node 22.19+ runtime;
|
||||
- `openclaw onboard --install-daemon`;
|
||||
- Gateway daemon usage;
|
||||
- agent prompt files `AGENTS.md`, `SOUL.md`, and `TOOLS.md`;
|
||||
- workspace skills at `~/.openclaw/workspace/skills/<skill>/SKILL.md`;
|
||||
- model configuration in OpenClaw config;
|
||||
- security guidance for DM pairing, allowlists, and sandboxing.
|
||||
|
||||
Verify upstream docs before depending on a command in code.
|
||||
|
||||
## Living IP Workspace Shape
|
||||
|
||||
Create or update:
|
||||
|
||||
- `AGENTS.md`: scope, repo boundaries, proof requirements;
|
||||
- `SOUL.md`: Rio or Theseus identity;
|
||||
- `TOOLS.md`: bounded tools only;
|
||||
- `skills/decision-engine-refinement/SKILL.md`;
|
||||
- `skills/living-ip-kb-interop/SKILL.md`;
|
||||
- `skills/teleo-db-operator/SKILL.md` only for read-only local copies unless explicitly authorized.
|
||||
|
||||
## Tool Policy
|
||||
|
||||
Default allow:
|
||||
|
||||
- read files;
|
||||
- run local fixture tests;
|
||||
- write proof artifacts;
|
||||
- inspect git diffs;
|
||||
- query copied SQLite DBs read-only.
|
||||
|
||||
Default deny:
|
||||
|
||||
- production DB writes;
|
||||
- token reads;
|
||||
- Bitwarden vault export;
|
||||
- live GitHub PR comments;
|
||||
- public messaging sends;
|
||||
- broad shell automation against host services.
|
||||
|
||||
## Rio And Theseus
|
||||
|
||||
- Rio OpenClaw package: economic reasoning, contribution incentives, paid-query guardrails, OPSEC.
|
||||
- Theseus OpenClaw package: eval integrity, adversarial prompts, model bakeoffs, self-upgrade review.
|
||||
|
||||
## Proof Contract
|
||||
|
||||
An OpenClaw adapter is useful only if it can run a fixture and produce:
|
||||
|
||||
- prompt files used;
|
||||
- tool allowlist;
|
||||
- model selected;
|
||||
- fixture input;
|
||||
- structured verdict output;
|
||||
- proof that no denied tools were invoked.
|
||||
|
||||
Do not claim OpenClaw production readiness until the package runs in an OpenClaw workspace and writes proof.
|
||||
|
|
@ -1,76 +0,0 @@
|
|||
---
|
||||
name: teleo-db-operator
|
||||
description: Use when reading, auditing, backing up, querying, or safely writing the Teleo pipeline SQLite database, including review_records, audit_log, costs, prs, sources, and contributor feedback loops.
|
||||
---
|
||||
|
||||
# Teleo DB Operator
|
||||
|
||||
Default to read-only. The database is evidence for decision-engine refinement, not a scratchpad.
|
||||
|
||||
## Discover
|
||||
|
||||
1. Read `lib/config.py` for `DB_PATH` and related paths.
|
||||
2. Prefer local or copied DBs over production DBs.
|
||||
3. If using production, record whether access is read-only or write-authorized.
|
||||
4. Never print secret values found near DB paths or shell history.
|
||||
|
||||
## Read Path
|
||||
|
||||
Use `sqlite3` or Python `sqlite3`.
|
||||
|
||||
Recommended read targets:
|
||||
|
||||
- `review_records`: evaluator, model, outcome, rejection reason.
|
||||
- `audit_log`: route decisions, approve/reject events, failure details.
|
||||
- `costs`: model cost by date/stage.
|
||||
- `prs`: status, tier, route compatibility fields, verdicts.
|
||||
- `sources`: priority, feedback, extraction model.
|
||||
|
||||
For refinement work, export aggregated JSON or CSV into `.crabbox-results/` or `proof/`, not raw private DB snapshots.
|
||||
|
||||
## Write Path
|
||||
|
||||
Writes require explicit authorization and a backup.
|
||||
|
||||
Required sequence:
|
||||
|
||||
1. Create a backup or operate on a copy.
|
||||
2. Write the exact SQL in a retained artifact.
|
||||
3. Use `BEGIN IMMEDIATE;`.
|
||||
4. Apply the minimal mutation.
|
||||
5. Read back the changed rows.
|
||||
6. Commit the transaction only after readback is correct.
|
||||
7. Write a blocker artifact instead of guessing if any precondition is missing.
|
||||
|
||||
Never write production prompt/model state as part of an experiment. Experiments should replay fixtures and produce proof first.
|
||||
|
||||
## Safety Boundaries
|
||||
|
||||
- Do not attach, copy, or commit `pipeline.db`.
|
||||
- Do not run broad `UPDATE` or `DELETE` without a `WHERE` clause and a prior row count.
|
||||
- Do not mutate `prs`, `sources`, or contributor state from a model response alone.
|
||||
- Do not treat local copied DB proof as production proof.
|
||||
|
||||
## Useful Queries
|
||||
|
||||
```sql
|
||||
SELECT reviewer, reviewer_model, outcome, rejection_reason, count(*) AS n
|
||||
FROM review_records
|
||||
GROUP BY reviewer, reviewer_model, outcome, rejection_reason
|
||||
ORDER BY n DESC;
|
||||
```
|
||||
|
||||
```sql
|
||||
SELECT event, count(*) AS n
|
||||
FROM audit_log
|
||||
WHERE stage = 'evaluate'
|
||||
GROUP BY event
|
||||
ORDER BY n DESC;
|
||||
```
|
||||
|
||||
```sql
|
||||
SELECT model, stage, calls, input_tokens, output_tokens, cost_usd
|
||||
FROM costs
|
||||
ORDER BY date DESC, cost_usd DESC
|
||||
LIMIT 50;
|
||||
```
|
||||
187
.crabbox.yaml
187
.crabbox.yaml
|
|
@ -1,187 +0,0 @@
|
|||
profile: teleo-infrastructure-check
|
||||
provider: hetzner
|
||||
target: linux
|
||||
architecture: arm64
|
||||
class: beast
|
||||
ttl: 90m
|
||||
idleTimeout: 20m
|
||||
capacity:
|
||||
market: spot
|
||||
strategy: most-available
|
||||
fallback: on-demand-after-120s
|
||||
actions:
|
||||
workflow: .github/workflows/crabbox.yml
|
||||
job: hydrate
|
||||
runnerLabels:
|
||||
- crabbox
|
||||
runnerVersion: latest
|
||||
ephemeral: true
|
||||
sync:
|
||||
delete: true
|
||||
checksum: false
|
||||
gitSeed: true
|
||||
fingerprint: true
|
||||
timeout: 15m
|
||||
warnFiles: 50000
|
||||
warnBytes: 5368709120
|
||||
failFiles: 150000
|
||||
failBytes: 21474836480
|
||||
exclude:
|
||||
- .cache
|
||||
- .venv
|
||||
- .pytest_cache
|
||||
- .ruff_cache
|
||||
- __pycache__
|
||||
- "*.pyc"
|
||||
- "*.db"
|
||||
- "*.db-wal"
|
||||
- "*.db-shm"
|
||||
- "*.log"
|
||||
- logs
|
||||
- secrets
|
||||
- .env
|
||||
- htmlcov
|
||||
- dist
|
||||
- build
|
||||
- "*.egg-info"
|
||||
- .turbo
|
||||
- node_modules
|
||||
env:
|
||||
allow:
|
||||
- CI
|
||||
- PYTHONWARNINGS
|
||||
- PHASE1B_AGENT_ROUTING_ENABLED
|
||||
ssh:
|
||||
user: crabbox
|
||||
port: "2222"
|
||||
# Ordered fallback ports tried after ssh.port; use [] to disable fallback.
|
||||
fallbackPorts:
|
||||
- "22"
|
||||
|
||||
jobs:
|
||||
ci-contract:
|
||||
provider: hetzner
|
||||
target: linux
|
||||
architecture: arm64
|
||||
class: beast
|
||||
hydrate:
|
||||
actions: true
|
||||
githubRunner: false
|
||||
waitTimeout: 20m
|
||||
keepAliveMinutes: 90
|
||||
actions:
|
||||
workflow: .github/workflows/crabbox.yml
|
||||
job: hydrate
|
||||
shell: true
|
||||
command: >
|
||||
python3 -m pip install -e '.[dev]' &&
|
||||
mkdir -p .crabbox-results &&
|
||||
python3 scripts/check_crabbox_ci_contract.py
|
||||
--output .crabbox-results/crabbox-ci-contract.json &&
|
||||
python3 scripts/check_llm_refinement_contract.py
|
||||
--output .crabbox-results/llm-refinement-contract.json &&
|
||||
python3 scripts/replay_decision_engine_eval.py
|
||||
--output .crabbox-results/decision-engine-eval.json
|
||||
downloads:
|
||||
- .crabbox-results/crabbox-ci-contract.json
|
||||
- .crabbox-results/llm-refinement-contract.json
|
||||
- .crabbox-results/decision-engine-eval.json
|
||||
stop: always
|
||||
|
||||
unit:
|
||||
provider: hetzner
|
||||
target: linux
|
||||
architecture: arm64
|
||||
class: beast
|
||||
hydrate:
|
||||
actions: true
|
||||
githubRunner: false
|
||||
waitTimeout: 20m
|
||||
keepAliveMinutes: 90
|
||||
actions:
|
||||
workflow: .github/workflows/crabbox.yml
|
||||
job: hydrate
|
||||
shell: true
|
||||
command: >
|
||||
python3 -m pip install -e '.[dev]' &&
|
||||
mkdir -p .crabbox-results &&
|
||||
python3 -m pytest --junitxml=.crabbox-results/pytest.xml
|
||||
junit:
|
||||
- .crabbox-results/pytest.xml
|
||||
downloads:
|
||||
- .crabbox-results/pytest.xml
|
||||
stop: always
|
||||
|
||||
lint-phase1b:
|
||||
provider: hetzner
|
||||
target: linux
|
||||
architecture: arm64
|
||||
class: beast
|
||||
hydrate:
|
||||
actions: true
|
||||
githubRunner: false
|
||||
waitTimeout: 20m
|
||||
keepAliveMinutes: 90
|
||||
actions:
|
||||
workflow: .github/workflows/crabbox.yml
|
||||
job: hydrate
|
||||
shell: true
|
||||
command: >
|
||||
python3 -m pip install -e '.[dev]' &&
|
||||
python3 -m ruff check
|
||||
lib/agent_routing.py
|
||||
lib/config.py
|
||||
lib/db.py
|
||||
lib/evaluate.py
|
||||
lib/llm.py
|
||||
lib/post_extract.py
|
||||
telegram/approvals.py
|
||||
scripts/prove_phase1b_local.py
|
||||
tests/test_agent_routing.py
|
||||
tests/test_evaluate_agent_routing.py
|
||||
tests/test_phase1b_end_to_end.py
|
||||
tests/test_eval_parse.py
|
||||
tests/test_contributor.py
|
||||
tests/test_search.py
|
||||
stop: always
|
||||
|
||||
phase1b-local-proof:
|
||||
provider: hetzner
|
||||
target: linux
|
||||
architecture: arm64
|
||||
class: beast
|
||||
hydrate:
|
||||
actions: true
|
||||
githubRunner: false
|
||||
waitTimeout: 20m
|
||||
keepAliveMinutes: 90
|
||||
actions:
|
||||
workflow: .github/workflows/crabbox.yml
|
||||
job: hydrate
|
||||
shell: true
|
||||
command: >
|
||||
python3 -m pip install -e '.[dev]' &&
|
||||
scripts/crabbox_phase1b_proof.sh
|
||||
junit:
|
||||
- .crabbox-results/phase1b-pytest.xml
|
||||
downloads:
|
||||
- .crabbox-results/crabbox-ci-contract.json
|
||||
- proof/phase1b-local-e2e-proof.json
|
||||
- .crabbox-results/phase1b-pytest.xml
|
||||
- .crabbox-results/phase1b-proof-summary.json
|
||||
stop: always
|
||||
|
||||
sync-smoke:
|
||||
provider: hetzner
|
||||
target: linux
|
||||
architecture: arm64
|
||||
class: beast
|
||||
hydrate:
|
||||
actions: false
|
||||
shell: true
|
||||
command: >
|
||||
python3 -m compileall
|
||||
lib
|
||||
tests
|
||||
scripts/prove_phase1b_local.py
|
||||
stop: always
|
||||
146
.github/workflows/ci.yml
vendored
146
.github/workflows/ci.yml
vendored
|
|
@ -1,146 +0,0 @@
|
|||
name: ci
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: ci-${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
PYTHON_VERSION: "3.11"
|
||||
CI: "1"
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
name: Focused lint
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
- name: Install
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
python -m pip install -e ".[dev]"
|
||||
- name: Ruff focused surface
|
||||
run: |
|
||||
python -m ruff check \
|
||||
lib/agent_routing.py \
|
||||
lib/config.py \
|
||||
lib/db.py \
|
||||
lib/evaluate.py \
|
||||
lib/llm.py \
|
||||
lib/post_extract.py \
|
||||
telegram/approvals.py \
|
||||
scripts/check_crabbox_ci_contract.py \
|
||||
scripts/check_llm_refinement_contract.py \
|
||||
scripts/replay_decision_engine_eval.py \
|
||||
scripts/prove_phase1b_local.py \
|
||||
tests/test_agent_routing.py \
|
||||
tests/test_decision_engine_replay.py \
|
||||
tests/test_evaluate_agent_routing.py \
|
||||
tests/test_phase1b_end_to_end.py \
|
||||
tests/test_eval_parse.py \
|
||||
tests/test_contributor.py \
|
||||
tests/test_search.py
|
||||
|
||||
test:
|
||||
name: Unit tests
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
- name: Install
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
python -m pip install -e ".[dev]"
|
||||
- name: Pytest
|
||||
run: |
|
||||
mkdir -p .crabbox-results
|
||||
python -m pytest --junitxml=.crabbox-results/pytest.xml
|
||||
- name: Upload test artifact
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: teleo-infrastructure-pytest
|
||||
path: .crabbox-results/pytest.xml
|
||||
if-no-files-found: warn
|
||||
|
||||
repo-contracts:
|
||||
name: Repo contracts
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
- name: Install
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
python -m pip install -e ".[dev]"
|
||||
- name: Validate repo-owned contract
|
||||
run: |
|
||||
python scripts/check_crabbox_ci_contract.py \
|
||||
--output .crabbox-results/crabbox-ci-contract.json
|
||||
python scripts/check_llm_refinement_contract.py \
|
||||
--output .crabbox-results/llm-refinement-contract.json
|
||||
python scripts/replay_decision_engine_eval.py \
|
||||
--output .crabbox-results/decision-engine-eval.json
|
||||
- name: Upload contract artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: teleo-infrastructure-repo-contracts
|
||||
path: |
|
||||
.crabbox-results/crabbox-ci-contract.json
|
||||
.crabbox-results/llm-refinement-contract.json
|
||||
.crabbox-results/decision-engine-eval.json
|
||||
if-no-files-found: error
|
||||
|
||||
phase1b-local-proof:
|
||||
name: Phase 1B local proof
|
||||
runs-on: ubuntu-latest
|
||||
needs:
|
||||
- lint
|
||||
- test
|
||||
- repo-contracts
|
||||
timeout-minutes: 20
|
||||
env:
|
||||
PHASE1B_AGENT_ROUTING_ENABLED: "true"
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
- name: Install
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
python -m pip install -e ".[dev]"
|
||||
- name: Run proof wrapper
|
||||
run: |
|
||||
scripts/crabbox_phase1b_proof.sh
|
||||
- name: Upload proof artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: teleo-infrastructure-phase1b-proof
|
||||
path: |
|
||||
.crabbox-results/crabbox-ci-contract.json
|
||||
proof/phase1b-local-e2e-proof.json
|
||||
.crabbox-results/phase1b-pytest.xml
|
||||
.crabbox-results/phase1b-proof-summary.json
|
||||
if-no-files-found: warn
|
||||
101
.github/workflows/crabbox.yml
vendored
101
.github/workflows/crabbox.yml
vendored
|
|
@ -1,101 +0,0 @@
|
|||
name: crabbox
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
ref:
|
||||
description: "Git ref to hydrate"
|
||||
required: false
|
||||
type: string
|
||||
crabbox_id:
|
||||
description: "Crabbox lease ID"
|
||||
required: true
|
||||
type: string
|
||||
crabbox_runner_label:
|
||||
description: "Dynamic Crabbox runner label"
|
||||
required: true
|
||||
type: string
|
||||
crabbox_job:
|
||||
description: "Hydration job identifier expected by Crabbox"
|
||||
required: false
|
||||
default: "hydrate"
|
||||
type: string
|
||||
crabbox_keep_alive_minutes:
|
||||
description: "Minutes to keep the hydrated job alive"
|
||||
required: false
|
||||
default: "90"
|
||||
type: string
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
hydrate:
|
||||
runs-on: [self-hosted, "${{ inputs.crabbox_runner_label }}"]
|
||||
timeout-minutes: 120
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
ref: ${{ inputs.ref || github.ref }}
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
- name: Hydrate
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
python -m pip install -e ".[dev]"
|
||||
if [ -f package-lock.json ]; then npm ci; fi
|
||||
if [ -f pnpm-lock.yaml ]; then corepack enable && pnpm install --frozen-lockfile; fi
|
||||
if [ -f go.mod ]; then go mod download; fi
|
||||
- name: Mark Crabbox ready
|
||||
shell: bash
|
||||
run: |
|
||||
job="${{ inputs.crabbox_job }}"
|
||||
if [ -z "$job" ]; then job=hydrate; fi
|
||||
mkdir -p "$HOME/.crabbox/actions"
|
||||
state="$HOME/.crabbox/actions/${{ inputs.crabbox_id }}.env"
|
||||
env_file="$HOME/.crabbox/actions/${{ inputs.crabbox_id }}.env.sh"
|
||||
services_file="$HOME/.crabbox/actions/${{ inputs.crabbox_id }}.services"
|
||||
write_export() {
|
||||
key="$1"
|
||||
value="${!key-}"
|
||||
if [ -n "$value" ]; then
|
||||
printf 'export %s=%q\n' "$key" "$value"
|
||||
fi
|
||||
}
|
||||
{
|
||||
for key in CI GITHUB_ACTIONS GITHUB_WORKSPACE GITHUB_REPOSITORY GITHUB_RUN_ID GITHUB_RUN_NUMBER GITHUB_RUN_ATTEMPT GITHUB_REF GITHUB_REF_NAME GITHUB_SHA GITHUB_EVENT_NAME GITHUB_ACTOR GITHUB_JOB RUNNER_OS RUNNER_ARCH RUNNER_TEMP RUNNER_TOOL_CACHE; do
|
||||
write_export "$key"
|
||||
done
|
||||
} > "${env_file}.tmp"
|
||||
mv "${env_file}.tmp" "$env_file"
|
||||
{
|
||||
echo "# Docker containers visible from the hydrated runner"
|
||||
docker ps --format '{{.Names}}\t{{.Image}}\t{{.Ports}}' 2>/dev/null || true
|
||||
} > "${services_file}.tmp"
|
||||
mv "${services_file}.tmp" "$services_file"
|
||||
tmp="${state}.tmp"
|
||||
{
|
||||
echo "WORKSPACE=${GITHUB_WORKSPACE}"
|
||||
echo "RUN_ID=${GITHUB_RUN_ID}"
|
||||
echo "JOB=${job}"
|
||||
echo "ENV_FILE=${env_file}"
|
||||
echo "SERVICES_FILE=${services_file}"
|
||||
echo "READY_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
} > "$tmp"
|
||||
mv "$tmp" "$state"
|
||||
- name: Keep Crabbox job alive
|
||||
shell: bash
|
||||
run: |
|
||||
minutes="${{ inputs.crabbox_keep_alive_minutes }}"
|
||||
case "$minutes" in
|
||||
''|*[!0-9]*) minutes=90 ;;
|
||||
esac
|
||||
stop="$HOME/.crabbox/actions/${{ inputs.crabbox_id }}.stop"
|
||||
deadline=$(( $(date +%s) + minutes * 60 ))
|
||||
while [ "$(date +%s)" -lt "$deadline" ]; do
|
||||
if [ -f "$stop" ]; then
|
||||
exit 0
|
||||
fi
|
||||
sleep 15
|
||||
done
|
||||
2
.gitignore
vendored
2
.gitignore
vendored
|
|
@ -20,8 +20,6 @@ logs/
|
|||
|
||||
# Test artifacts
|
||||
.pytest_cache/
|
||||
.crabbox/
|
||||
.crabbox-results/
|
||||
htmlcov/
|
||||
.coverage
|
||||
|
||||
|
|
|
|||
165
README.md
165
README.md
|
|
@ -1,134 +1,65 @@
|
|||
# teleo-infrastructure
|
||||
|
||||
This repo runs the pipeline that processes contributions into the
|
||||
[teleo-codex](https://github.com/living-ip/teleo-codex) knowledge base.
|
||||
Pipeline infrastructure for the Teleo collective knowledge base. Async Python daemon that extracts, validates, evaluates, and merges claims via Forgejo PRs.
|
||||
|
||||
Every claim on `main` has been extracted from a source, validated for schema
|
||||
and duplicates, evaluated by at least two independent reviewers, and merged
|
||||
through an event-sourced audit log. The whole flow is an async Python daemon
|
||||
talking to a Forgejo git server, an SQLite WAL state store, OpenRouter (for
|
||||
most LLM calls), and the Anthropic Claude CLI (for Opus deep reviews).
|
||||
## Directory Structure
|
||||
|
||||
**Production state** (live):
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| Claims merged into `main` | 1,546 across 13 domains |
|
||||
| PRs merged through the pipeline | 1,975 |
|
||||
| Merge throughput (last 7d) | 508 PRs (~73/day) |
|
||||
| Review approval rate | 94% |
|
||||
| Cost per merged claim (last 30d) | $0.10 incl. extract + triage + multi-tier review |
|
||||
| Production agents | 6 (rio, theseus, leo, vida, astra, clay) |
|
||||
|
||||
## Pipeline
|
||||
|
||||
Concurrent stage loops in a single daemon (`teleo-pipeline.py`), coordinated
|
||||
by SQLite. Circuit breakers cap costs, retry budgets cap attempts, and merges
|
||||
are serialized per-domain to avoid cross-PR conflicts.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Inbox["inbox/queue/"] --> Extract
|
||||
Extract["Extract<br/>(Sonnet 4.5)"] --> Validate
|
||||
Validate["Validate<br/>(tier 0, $0)"] --> Evaluate
|
||||
Evaluate["Evaluate<br/>(tiered, multi-model)"] --> Merge
|
||||
Merge["Merge<br/>(Forgejo, domain-serial)"] --> Effects
|
||||
Effects["Effects<br/>cascade · backlinks · reciprocal edges"]
|
||||
```
|
||||
|
||||
If any reviewer rejects, the PR gets a structured rationale and either
|
||||
re-extraction guidance (for fixable issues) or a terminal close (for
|
||||
scope or duplicate problems). Approved merges trigger downstream effects:
|
||||
|
||||
- **Cascade** — agents whose beliefs/positions depend on the changed claim get inbox notifications
|
||||
- **Bidirectional provenance** — `sourced_from:` is stamped on each claim at extraction; the source's `claims_extracted:` list is updated post-merge
|
||||
- **Reciprocal edges** — when a new claim has `supports: [X]`, X's frontmatter is updated with `supports: [new]`
|
||||
- **Cross-domain index** — entity mentions across domain boundaries are logged for silo detection
|
||||
|
||||
## Multi-agent review
|
||||
|
||||
Reviews aren't free. Tier classification is deterministic where possible
|
||||
(changes to `core/` or `foundations/` always go Deep) and otherwise picked
|
||||
by Haiku based on PR scope. Last 30d distribution: 76% Standard, 21% Light,
|
||||
2% Deep.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
PR[New PR] --> Classify{Classify}
|
||||
Classify -->|"core/, foundations/, challenged"| Deep
|
||||
Classify -->|default| Standard
|
||||
Classify -->|single claim, low risk| Light
|
||||
Light["Light tier<br/>Domain agent only"] --> Result
|
||||
Standard["Standard tier<br/>Domain agent + Leo (Sonnet 4.5)"] --> Result
|
||||
Deep["Deep tier<br/>Domain agent + Leo (Opus)"] --> Result
|
||||
Result{Both approve?}
|
||||
Result -->|yes| MergeOK[Merge]
|
||||
Result -->|no| Reject[Structured rejection<br/>+ re-extract guidance]
|
||||
teleo-infrastructure/
|
||||
├── teleo-pipeline.py # Daemon entry point
|
||||
├── reweave.py # Reciprocal edge maintenance
|
||||
├── lib/ # Pipeline modules (Python package)
|
||||
├── diagnostics/ # Monitoring dashboard (port 8081)
|
||||
├── telegram/ # Telegram bot interface
|
||||
├── deploy/ # Deployment + mirror scripts
|
||||
├── systemd/ # Service definitions
|
||||
├── agent-state/ # Cross-session agent state
|
||||
├── research/ # Nightly research orchestration
|
||||
├── hermes-agent/ # Hermes agent setup
|
||||
├── scripts/ # One-off backfills + migrations
|
||||
├── tests/ # Test suite
|
||||
└── docs/ # Operational documentation
|
||||
```
|
||||
|
||||
Domain agents bring domain expertise: **Rio** (internet-finance), **Vida**
|
||||
(health), **Astra** (space-development), **Clay** (entertainment),
|
||||
**Theseus** (ai-alignment). **Leo** brings cross-domain consistency on
|
||||
every PR. Disagreement between the two reviewers surfaces in `audit_log`
|
||||
and is tracked as a quality signal, not silenced.
|
||||
|
||||
Model diversity isn't cosmetic — same-family models share ~60% of their
|
||||
errors (Kim et al. ICML 2025). Pipeline mixes Haiku for triage, Gemini 2.5
|
||||
Flash for domain review, Sonnet 4.5 for Leo standard, Opus for Leo deep.
|
||||
|
||||
## Contributor flow
|
||||
|
||||
External contributors submit PRs to
|
||||
[`living-ip/teleo-codex`](https://github.com/living-ip/teleo-codex) on GitHub.
|
||||
A mirror sync (every 2 minutes) fast-forwards the PR onto Forgejo, where
|
||||
the pipeline picks it up. From there it's the same flow as agent-authored
|
||||
PRs — same tiers, same reviewers, same merge rules.
|
||||
|
||||
The contributor-facing guide lives in
|
||||
[`teleo-codex/CONTRIBUTING.md`](https://github.com/living-ip/teleo-codex/blob/main/CONTRIBUTING.md).
|
||||
|
||||
## Repository layout
|
||||
|
||||
| Directory | What it does |
|
||||
|-----------------|-----------------------------------------------------------|
|
||||
| `lib/` | Pipeline modules — config, db, extract, evaluate, merge, cascade |
|
||||
| `diagnostics/` | Argus monitoring dashboard (4 pages: ops, health, agents, epistemic) |
|
||||
| `telegram/` | Telegram bot that answers from the knowledge base |
|
||||
| `research/` | Nightly autonomous research sessions for domain agents |
|
||||
| `agent-state/` | File-backed state for cross-session agent continuity |
|
||||
| `deploy/` | Auto-deploy pipeline (Forgejo → working dirs → systemd) |
|
||||
| `systemd/` | Service definitions for daemon + dashboard + agents |
|
||||
| `scripts/` | Backfills and one-off migrations |
|
||||
| `tests/` | pytest suite |
|
||||
| `docs/` | Architecture specs and operational protocols |
|
||||
|
||||
## Ownership
|
||||
|
||||
Code review authority is enforced by [`CODEOWNERS`](./CODEOWNERS) — every
|
||||
file has one accountable agent. The high-level map:
|
||||
Each directory has one owning agent. The owner is accountable for correctness and reviews all changes to their section. See `CODEOWNERS` for per-file detail.
|
||||
|
||||
- **Ship** — pipeline core, telegram, deploy, agent-state, research, systemd
|
||||
- **Epimetheus** — extraction (intake, entity processing, pre-screening, post-extract validation)
|
||||
- **Leo** — evaluation (claim review, analytics, attribution)
|
||||
- **Argus** — health (diagnostics dashboard, alerting, claim index, search)
|
||||
- **Ganymede** — tests (pytest suite, integration, code review gate)
|
||||
| Directory | Owner | What it does |
|
||||
|-----------|-------|-------------|
|
||||
| `lib/` (core) | **Ship** | Config, DB, merge, cascade, validation, LLM calls |
|
||||
| `lib/` (extraction) | **Epimetheus** | Source extraction, entity processing, pre-screening |
|
||||
| `lib/` (evaluation) | **Leo** | Claim evaluation, analytics, attribution |
|
||||
| `lib/` (health) | **Argus** | Health checks, search, claim index |
|
||||
| `diagnostics/` | **Argus** | 4-page dashboard, alerting, vitality metrics |
|
||||
| `telegram/` | **Ship** | Telegram bot, X integration, retrieval |
|
||||
| `deploy/` | **Ship** | rsync deploy, GitHub-Forgejo mirror |
|
||||
| `systemd/` | **Ship** | teleo-pipeline, teleo-diagnostics, teleo-agent@ |
|
||||
| `agent-state/` | **Ship** | Bootstrap, state library, cascade inbox processor |
|
||||
| `research/` | **Ship** | Nightly research sessions, prompt templates |
|
||||
| `scripts/` | **Ship** | Backfills, migrations, one-off maintenance |
|
||||
| `tests/` | **Ganymede** | pytest suite, integration tests |
|
||||
| `docs/` | Shared | Architecture, specs, protocols |
|
||||
|
||||
For active sprint work and per-agent in-flight items, see each agent's
|
||||
status report in their Pentagon profile.
|
||||
## VPS Layout
|
||||
|
||||
## Development
|
||||
Runs on Hetzner CAX31 (77.42.65.182) as user `teleo`.
|
||||
|
||||
| VPS Path | Repo Source | Service |
|
||||
|----------|-------------|---------|
|
||||
| `/opt/teleo-eval/pipeline/` | `lib/`, `teleo-pipeline.py`, `reweave.py` | teleo-pipeline |
|
||||
| `/opt/teleo-eval/diagnostics/` | `diagnostics/` | teleo-diagnostics |
|
||||
| `/opt/teleo-eval/telegram/` | `telegram/` | (manual) |
|
||||
| `/opt/teleo-eval/agent-state/` | `agent-state/` | (used by research-session.sh) |
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Run tests
|
||||
pip install -e ".[dev]"
|
||||
pytest
|
||||
|
||||
# Deploy to VPS
|
||||
./deploy/deploy.sh --dry-run # preview
|
||||
./deploy/deploy.sh # deploy
|
||||
```
|
||||
|
||||
## Operations
|
||||
|
||||
Production deployment runs on a single VPS. Runbook, restart procedures,
|
||||
secret rotation, and on-call live in the private
|
||||
[`teleo-ops`](https://github.com/living-ip/teleo-ops) repo (request access).
|
||||
|
||||
## License
|
||||
|
||||
[TBD]
|
||||
|
|
|
|||
|
|
@ -13,7 +13,6 @@ fi
|
|||
|
||||
DEPLOY_CHECKOUT="/opt/teleo-eval/workspaces/deploy-infra"
|
||||
PIPELINE_DIR="/opt/teleo-eval/pipeline"
|
||||
TELEGRAM_DIR="/opt/teleo-eval/telegram"
|
||||
DIAGNOSTICS_DIR="/opt/teleo-eval/diagnostics"
|
||||
AGENT_STATE_DIR="/opt/teleo-eval/ops/agent-state"
|
||||
STAMP_FILE="/opt/teleo-eval/.last-deploy-sha"
|
||||
|
|
@ -21,31 +20,18 @@ LOG_TAG="auto-deploy"
|
|||
|
||||
log() { logger -t "$LOG_TAG" "$1"; echo "$(date '+%Y-%m-%d %H:%M:%S') $1"; }
|
||||
|
||||
DEPLOY_REMOTE="${TELEO_DEPLOY_REMOTE:-}"
|
||||
if [ -z "$DEPLOY_REMOTE" ]; then
|
||||
if git -C "$DEPLOY_CHECKOUT" remote get-url github >/dev/null 2>&1; then
|
||||
DEPLOY_REMOTE="github"
|
||||
else
|
||||
DEPLOY_REMOTE="origin"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ ! -d "$DEPLOY_CHECKOUT/.git" ]; then
|
||||
log "ERROR: Deploy checkout not found at $DEPLOY_CHECKOUT. Run setup first."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
cd "$DEPLOY_CHECKOUT"
|
||||
if ! git remote get-url "$DEPLOY_REMOTE" >/dev/null 2>&1; then
|
||||
log "ERROR: deploy remote '$DEPLOY_REMOTE' is not configured"
|
||||
exit 1
|
||||
fi
|
||||
if ! git fetch "$DEPLOY_REMOTE" main --quiet 2>&1; then
|
||||
log "ERROR: git fetch failed for $DEPLOY_REMOTE/main"
|
||||
if ! git fetch origin main --quiet 2>&1; then
|
||||
log "ERROR: git fetch failed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
NEW_SHA=$(git rev-parse "$DEPLOY_REMOTE/main")
|
||||
NEW_SHA=$(git rev-parse origin/main)
|
||||
OLD_SHA=$(cat "$STAMP_FILE" 2>/dev/null || echo "none")
|
||||
|
||||
if [ "$NEW_SHA" = "$OLD_SHA" ]; then
|
||||
|
|
@ -58,8 +44,8 @@ if ! git checkout main --quiet 2>&1; then
|
|||
log "ERROR: git checkout main failed — dirty tree or corrupted index"
|
||||
exit 1
|
||||
fi
|
||||
if ! git merge --ff-only "$DEPLOY_REMOTE/main" --quiet 2>&1; then
|
||||
log "ERROR: git merge --ff-only $DEPLOY_REMOTE/main failed. Manual intervention needed."
|
||||
if ! git pull --ff-only --quiet 2>&1; then
|
||||
log "ERROR: git pull --ff-only failed. Manual intervention needed."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
|
@ -83,12 +69,11 @@ RSYNC_OPTS=(-az --exclude __pycache__ --exclude '*.pyc' --exclude '*.bak*')
|
|||
|
||||
rsync "${RSYNC_OPTS[@]}" lib/ "$PIPELINE_DIR/lib/"
|
||||
|
||||
for f in teleo-pipeline.py reweave.py fetch_coins.py pipeline-health-check.py; do
|
||||
for f in teleo-pipeline.py reweave.py fetch_coins.py; do
|
||||
[ -f "$f" ] && rsync "${RSYNC_OPTS[@]}" "$f" "$PIPELINE_DIR/$f"
|
||||
done
|
||||
|
||||
rsync "${RSYNC_OPTS[@]}" telegram/ "$PIPELINE_DIR/telegram/"
|
||||
rsync "${RSYNC_OPTS[@]}" telegram/ "$TELEGRAM_DIR/"
|
||||
rsync "${RSYNC_OPTS[@]}" diagnostics/ "$DIAGNOSTICS_DIR/"
|
||||
rsync "${RSYNC_OPTS[@]}" agent-state/ "$AGENT_STATE_DIR/"
|
||||
rsync "${RSYNC_OPTS[@]}" tests/ "$PIPELINE_DIR/tests/"
|
||||
|
|
@ -101,37 +86,15 @@ log "Files synced"
|
|||
|
||||
# Restart services only if Python files changed
|
||||
RESTART=""
|
||||
add_restart() {
|
||||
case " $RESTART " in
|
||||
*" $1 "*) ;;
|
||||
*) RESTART="$RESTART $1" ;;
|
||||
esac
|
||||
}
|
||||
add_restart_if_unit_exists() {
|
||||
if systemctl list-units --all --full "$1.service" --no-legend 2>/dev/null | grep -q .; then
|
||||
add_restart "$1"
|
||||
fi
|
||||
}
|
||||
add_restart_if_unit_active() {
|
||||
if systemctl is-active --quiet "$1.service"; then
|
||||
add_restart "$1"
|
||||
fi
|
||||
}
|
||||
if [ "$OLD_SHA" != "none" ]; then
|
||||
if git diff --name-only "$OLD_SHA" "$NEW_SHA" -- lib/ teleo-pipeline.py reweave.py telegram/ 2>/dev/null | grep -q '\.py$'; then
|
||||
add_restart teleo-pipeline
|
||||
fi
|
||||
if git diff --name-only "$OLD_SHA" "$NEW_SHA" -- telegram/ 2>/dev/null | grep -q '\.py$'; then
|
||||
add_restart_if_unit_active teleo-agent@leo
|
||||
add_restart_if_unit_exists teleo-agent@leo-wallet-test
|
||||
RESTART="$RESTART teleo-pipeline"
|
||||
fi
|
||||
if git diff --name-only "$OLD_SHA" "$NEW_SHA" -- diagnostics/ 2>/dev/null | grep -q '\.py$'; then
|
||||
add_restart teleo-diagnostics
|
||||
RESTART="$RESTART teleo-diagnostics"
|
||||
fi
|
||||
else
|
||||
RESTART="teleo-pipeline teleo-diagnostics"
|
||||
add_restart_if_unit_active teleo-agent@leo
|
||||
add_restart_if_unit_exists teleo-agent@leo-wallet-test
|
||||
fi
|
||||
|
||||
if [ -n "$RESTART" ]; then
|
||||
|
|
|
|||
|
|
@ -7,7 +7,6 @@ set -euo pipefail
|
|||
|
||||
VPS_HOST="teleo@77.42.65.182"
|
||||
VPS_PIPELINE="/opt/teleo-eval/pipeline"
|
||||
VPS_TELEGRAM="/opt/teleo-eval/telegram"
|
||||
VPS_DIAGNOSTICS="/opt/teleo-eval/diagnostics"
|
||||
VPS_AGENT_STATE="/opt/teleo-eval/ops/agent-state"
|
||||
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
|
|
@ -75,7 +74,6 @@ echo ""
|
|||
|
||||
echo "=== Telegram bot ==="
|
||||
rsync "${RSYNC_OPTS[@]}" "$REPO_ROOT/telegram/" "$VPS_HOST:$VPS_PIPELINE/telegram/"
|
||||
rsync "${RSYNC_OPTS[@]}" "$REPO_ROOT/telegram/" "$VPS_HOST:$VPS_TELEGRAM/"
|
||||
echo ""
|
||||
|
||||
echo "=== Tests ==="
|
||||
|
|
@ -104,6 +102,6 @@ echo "Deploy complete."
|
|||
if $RESTART; then
|
||||
echo ""
|
||||
echo "=== Restarting services ==="
|
||||
ssh "$VPS_HOST" "sudo systemctl restart teleo-pipeline teleo-diagnostics; if systemctl is-active --quiet teleo-agent@leo.service; then sudo systemctl restart teleo-agent@leo; fi; if systemctl list-units --all --full teleo-agent@leo-wallet-test.service --no-legend | grep -q .; then sudo systemctl restart teleo-agent@leo-wallet-test; fi"
|
||||
ssh "$VPS_HOST" "sudo systemctl restart teleo-pipeline teleo-diagnostics"
|
||||
echo "Services restarted."
|
||||
fi
|
||||
|
|
|
|||
|
|
@ -32,12 +32,13 @@ def _is_source_slug(slug):
|
|||
|
||||
|
||||
def _classify_event(branch, description, commit_type, candidate_slug=None):
|
||||
"""Return one of: create | enrich | challenge | source | session_digest | None.
|
||||
"""Return one of: create | enrich | challenge | source | None.
|
||||
|
||||
Source-archive PRs are extract/* branches that filed a source into
|
||||
inbox/archive/ but didn't produce a claim. Session-digest PRs are
|
||||
agent research/entity commits with no per-claim description — they
|
||||
represent session-level rollups, not specific knowledge artifacts.
|
||||
inbox/archive/ but didn't produce a claim. Two signals classify them
|
||||
as 'source' (defense in depth):
|
||||
1. extract/* branch with empty description (no claim title produced)
|
||||
2. candidate_slug matches YYYY-MM-DD-...-HASH4 (inbox filename pattern)
|
||||
"""
|
||||
commit_type_l = (commit_type or "").lower()
|
||||
branch = branch or ""
|
||||
|
|
@ -59,12 +60,6 @@ def _classify_event(branch, description, commit_type, candidate_slug=None):
|
|||
or branch.startswith("reweave/")):
|
||||
return "enrich"
|
||||
|
||||
# Research and entity commits with no description are session-level
|
||||
# rollups (e.g. astra/research-2026-05-11). They have no claim to
|
||||
# link to — surface as session_digest, not as a phantom create.
|
||||
if commit_type_l in ("research", "entity") and not has_desc:
|
||||
return "session_digest"
|
||||
|
||||
# Source-only: extract/* with no claim description means inbox archive
|
||||
# landed but no domain claim was written.
|
||||
if branch.startswith("extract/") and not has_desc:
|
||||
|
|
@ -81,92 +76,12 @@ def _classify_event(branch, description, commit_type, candidate_slug=None):
|
|||
return "create"
|
||||
|
||||
|
||||
# Internal classifier value -> canonical `kind` enum returned to frontend.
|
||||
_KIND_MAP = {
|
||||
"create": "claim_merged",
|
||||
"enrich": "claim_enriched",
|
||||
"challenge": "claim_challenged",
|
||||
"source": "source_archived",
|
||||
"session_digest": "session_digest",
|
||||
}
|
||||
|
||||
|
||||
def _archive_slug_from_branch(branch):
|
||||
"""For extract/YYYY-MM-DD-...-HASH4, return YYYY-MM-DD-... (keep date,
|
||||
drop the 4-hex hash suffix). Matches inbox/archive filename convention.
|
||||
"""
|
||||
if not branch or "/" not in branch:
|
||||
return ""
|
||||
slug = branch.split("/", 1)[1]
|
||||
return re.sub(r"-[a-f0-9]{4}$", "", slug)
|
||||
|
||||
|
||||
def _source_target_url(domain, archive_slug):
|
||||
"""Forgejo blob URL for an archived source file. Falls back to the
|
||||
repo-wide inbox/archive directory when domain is unknown so the link
|
||||
still resolves to something useful instead of a 404.
|
||||
"""
|
||||
if not archive_slug:
|
||||
return None
|
||||
domain = (domain or "").strip()
|
||||
if not domain or domain == "unknown":
|
||||
return "https://git.livingip.xyz/teleo/teleo-codex/src/branch/main/inbox/archive"
|
||||
return (
|
||||
"https://git.livingip.xyz/teleo/teleo-codex/src/branch/main/inbox/archive/"
|
||||
f"{domain}/{archive_slug}.md"
|
||||
)
|
||||
|
||||
|
||||
def _claim_target_url(claim_slug):
|
||||
if not claim_slug:
|
||||
return None
|
||||
return f"/claims/{claim_slug}"
|
||||
|
||||
|
||||
# Canonical clickthrough URL for an activity-feed event.
|
||||
#
|
||||
# Every merged PR in the pipeline.db `prs` table lives on Forgejo at
|
||||
# git.livingip.xyz/teleo/teleo-codex/pulls/{number}. A small subset (3 of
|
||||
# 4094 as of 2026-05-13) was additionally mirrored to GitHub and has
|
||||
# prs.github_pr populated. Prefer GitHub when available (more public-facing
|
||||
# surface), fall back to Forgejo so every row has a real destination
|
||||
# instead of None (which makes the frontend whole-row overlay no-op and
|
||||
# leaves pipeline-attributed events looking dead-on-click).
|
||||
def _pr_url(pr_number, github_pr):
|
||||
if github_pr:
|
||||
return f"https://github.com/living-ip/teleo-codex/pull/{github_pr}"
|
||||
if pr_number:
|
||||
return f"https://git.livingip.xyz/teleo/teleo-codex/pulls/{pr_number}"
|
||||
return None
|
||||
|
||||
|
||||
# Canonicalize contributor labels so frontend links resolve to real
|
||||
# /contributors/{handle} pages. Pipeline writers (extract.py, manual edits,
|
||||
# the old backfill_submitted_by.py) historically wrote mixed-case agent
|
||||
# names with a trailing decorator into prs.submitted_by — e.g.
|
||||
# "Vida (self-directed)", "pipeline (reweave)", or "@m3taversal".
|
||||
# These decorated strings do not exist as contributors and 404 the profile
|
||||
# page. Strip the trailing parenthetical wholesale: valid handles match
|
||||
# ^[a-z0-9][a-z0-9_-]{0,38}$ (see pipeline/lib/attribution._HANDLE_RE) and
|
||||
# cannot contain parens, so this is lossless.
|
||||
_TRAILING_PAREN_RE = re.compile(r"\s*\([^)]*\)\s*$")
|
||||
|
||||
|
||||
def _canonicalize(raw):
|
||||
if not raw:
|
||||
return ""
|
||||
h = raw.strip().lower().lstrip("@")
|
||||
h = _TRAILING_PAREN_RE.sub("", h).strip()
|
||||
return h
|
||||
|
||||
|
||||
def _normalize_contributor(submitted_by, agent):
|
||||
name = _canonicalize(submitted_by)
|
||||
if name:
|
||||
return name
|
||||
name = _canonicalize(agent)
|
||||
if name and name != "pipeline":
|
||||
if submitted_by and submitted_by.strip():
|
||||
name = submitted_by.strip().lstrip("@")
|
||||
return name
|
||||
if agent and agent.strip() and agent != "pipeline":
|
||||
return agent.strip()
|
||||
return "pipeline"
|
||||
|
||||
|
||||
|
|
@ -213,7 +128,7 @@ def _build_events():
|
|||
rows = conn.execute(f"""
|
||||
SELECT p.number, p.branch, p.domain, p.agent, p.submitted_by,
|
||||
p.merged_at, p.description, p.commit_type, p.cost_usd,
|
||||
p.source_channel, p.source_path, p.github_pr
|
||||
p.source_channel, p.source_path
|
||||
FROM prs p
|
||||
WHERE p.status = 'merged'
|
||||
AND p.commit_type IN ({placeholders})
|
||||
|
|
@ -236,67 +151,29 @@ def _build_events():
|
|||
continue
|
||||
|
||||
contributor = _normalize_contributor(row["submitted_by"], row["agent"])
|
||||
# Hide pipeline-attributed events (reweave/*, ingestion/*) from the
|
||||
# public activity feed. They're automation maintenance, not
|
||||
# contributions — the daemon re-knits the graph nightly and ingests
|
||||
# external sources. Internal diagnostics + CI math still see these
|
||||
# rows in prs / contribution_events; only the public timeline drops
|
||||
# them. Mirrors the existing _FEED_COMMIT_TYPES filter (which hides
|
||||
# commit_type='pipeline') along the contributor axis.
|
||||
if contributor == "pipeline":
|
||||
continue
|
||||
merged_at = row["merged_at"] or ""
|
||||
domain = row["domain"] or "unknown"
|
||||
kind = _KIND_MAP.get(event_type, event_type)
|
||||
|
||||
ci_map = {
|
||||
"create": 0.35, "enrich": 0.25, "challenge": 0.40,
|
||||
"source": 0.15, "session_digest": 0.05,
|
||||
}
|
||||
ci_map = {"create": 0.35, "enrich": 0.25, "challenge": 0.40, "source": 0.15}
|
||||
ci_earned = ci_map.get(event_type, 0)
|
||||
|
||||
# Source events never carry a claim_slug — no claim was written.
|
||||
# target_url points at the archived file on Forgejo instead.
|
||||
# Source events never carry a claim_slug — no claim was written —
|
||||
# so the frontend can't produce a 404-ing claim link.
|
||||
if event_type == "source":
|
||||
archive_slug = _archive_slug_from_branch(row["branch"])
|
||||
summary_text = _summary_from_branch(row["branch"])
|
||||
source_display_slug = (
|
||||
summary_text.lower().replace(" ", "-") or row["branch"]
|
||||
source_slug = (
|
||||
_summary_from_branch(row["branch"]).lower().replace(" ", "-")
|
||||
or row["branch"]
|
||||
)
|
||||
events.append({
|
||||
"kind": kind,
|
||||
"type": "source",
|
||||
"target_url": _source_target_url(domain, archive_slug),
|
||||
"claim_slug": "",
|
||||
"source_slug": source_display_slug,
|
||||
"domain": domain,
|
||||
"source_slug": source_slug,
|
||||
"domain": row["domain"] or "unknown",
|
||||
"contributor": contributor,
|
||||
"timestamp": merged_at,
|
||||
"ci_earned": round(ci_earned, 2),
|
||||
"summary": summary_text,
|
||||
"pr_number": row["number"],
|
||||
"pr_url": _pr_url(row["number"], row["github_pr"]),
|
||||
"source_channel": row["source_channel"] or "unknown",
|
||||
})
|
||||
continue
|
||||
|
||||
# Session digests have no clickthrough surface yet (per-agent
|
||||
# session pages not built). target_url=null so frontend renders
|
||||
# plain text instead of a broken /claims/research-... link.
|
||||
if event_type == "session_digest":
|
||||
summary_text = _summary_from_branch(row["branch"]) or "Research session"
|
||||
events.append({
|
||||
"kind": kind,
|
||||
"type": "session_digest",
|
||||
"target_url": None,
|
||||
"claim_slug": "",
|
||||
"domain": domain,
|
||||
"contributor": contributor,
|
||||
"timestamp": merged_at,
|
||||
"ci_earned": round(ci_earned, 2),
|
||||
"summary": summary_text,
|
||||
"pr_number": row["number"],
|
||||
"pr_url": _pr_url(row["number"], row["github_pr"]),
|
||||
"source_channel": row["source_channel"] or "unknown",
|
||||
})
|
||||
continue
|
||||
|
|
@ -325,17 +202,14 @@ def _build_events():
|
|||
|
||||
for slug in (slugs[:1] if slugs else [""]):
|
||||
events.append({
|
||||
"kind": kind,
|
||||
"type": event_type,
|
||||
"target_url": _claim_target_url(slug),
|
||||
"claim_slug": slug,
|
||||
"domain": domain,
|
||||
"domain": row["domain"] or "unknown",
|
||||
"contributor": contributor,
|
||||
"timestamp": merged_at,
|
||||
"ci_earned": round(ci_earned, 2),
|
||||
"summary": summary_text,
|
||||
"pr_number": row["number"],
|
||||
"pr_url": _pr_url(row["number"], row["github_pr"]),
|
||||
"source_channel": row["source_channel"] or "unknown",
|
||||
})
|
||||
|
||||
|
|
@ -360,11 +234,8 @@ def _sort_events(events, claim_activity, sort_mode, now_ts):
|
|||
return _hot_score(ca["challenges"], ca["enriches"], ca["signals"], hours)
|
||||
events.sort(key=hot_key, reverse=True)
|
||||
elif sort_mode == "important":
|
||||
type_rank = {
|
||||
"challenge": 0, "enrich": 1, "create": 2,
|
||||
"source": 3, "session_digest": 4,
|
||||
}
|
||||
events.sort(key=lambda e: (type_rank.get(e["type"], 5), -len(e["summary"])))
|
||||
type_rank = {"challenge": 0, "enrich": 1, "create": 2, "source": 3}
|
||||
events.sort(key=lambda e: (type_rank.get(e["type"], 4), -len(e["summary"])))
|
||||
return events
|
||||
|
||||
|
||||
|
|
@ -398,13 +269,7 @@ async def handle_activity_feed(request):
|
|||
if contributor:
|
||||
filtered = [e for e in filtered if e["contributor"] == contributor]
|
||||
if type_filter:
|
||||
# Accept both legacy `type` values (create/enrich/challenge/source/
|
||||
# session_digest) and canonical `kind` values (claim_merged/etc.) so
|
||||
# callers can migrate at their own pace.
|
||||
filtered = [
|
||||
e for e in filtered
|
||||
if e["type"] in type_filter or e.get("kind") in type_filter
|
||||
]
|
||||
filtered = [e for e in filtered if e["type"] in type_filter]
|
||||
|
||||
sorted_events = _sort_events(list(filtered), claim_activity, sort_mode, now)
|
||||
total = len(sorted_events)
|
||||
|
|
|
|||
|
|
@ -79,16 +79,12 @@ def main():
|
|||
fm = sfm
|
||||
break
|
||||
|
||||
# `submitted_by` is stored as a canonical handle (lowercase, no @, no
|
||||
# "(self-directed)" / "(reweave)" suffix). Read consumers normalize via
|
||||
# attribution.normalize_handle, so writing decorated strings produces
|
||||
# downstream 404s on /contributors/{handle} (livingip-web timeline).
|
||||
if fm:
|
||||
proposed_by = fm.get("proposed_by")
|
||||
intake_tier = fm.get("intake_tier")
|
||||
|
||||
if proposed_by:
|
||||
contributor = proposed_by.strip().strip('"').strip("'").lower().lstrip("@")
|
||||
contributor = proposed_by.strip().strip('"').strip("'")
|
||||
elif intake_tier == "research-task":
|
||||
# Derive agent from branch prefix
|
||||
prefix = branch.split("/", 1)[0] if "/" in branch else "unknown"
|
||||
|
|
@ -98,12 +94,13 @@ def main():
|
|||
"clay": "clay", "astra": "astra", "leo": "leo",
|
||||
"reweave": "pipeline",
|
||||
}
|
||||
contributor = agent_map.get(prefix, prefix)
|
||||
agent = agent_map.get(prefix, prefix)
|
||||
contributor = f"{agent} (self-directed)"
|
||||
elif intake_tier == "directed":
|
||||
contributor = "m3taversal"
|
||||
contributor = "@m3taversal"
|
||||
else:
|
||||
# Default: if source exists but no proposed_by, operator submitted it.
|
||||
contributor = "m3taversal"
|
||||
# Default: if source exists but no proposed_by, it was Cory's submission
|
||||
contributor = "@m3taversal"
|
||||
|
||||
if contributor:
|
||||
conn.execute(
|
||||
|
|
@ -117,19 +114,19 @@ def main():
|
|||
agent = branch.split("/", 1)[0]
|
||||
conn.execute(
|
||||
"UPDATE prs SET submitted_by = ? WHERE number = ?",
|
||||
(agent, pr["number"]),
|
||||
(f"{agent} (self-directed)", pr["number"]),
|
||||
)
|
||||
updated += 1
|
||||
elif branch.startswith("reweave/"):
|
||||
conn.execute(
|
||||
"UPDATE prs SET submitted_by = 'pipeline' WHERE number = ?",
|
||||
"UPDATE prs SET submitted_by = 'pipeline (reweave)' WHERE number = ?",
|
||||
(pr["number"],),
|
||||
)
|
||||
updated += 1
|
||||
else:
|
||||
# Everything else (extract/, ingestion/, unknown) → operator directed it
|
||||
# Everything else (extract/, ingestion/, unknown) → Cory directed it
|
||||
conn.execute(
|
||||
"UPDATE prs SET submitted_by = 'm3taversal' WHERE number = ?",
|
||||
"UPDATE prs SET submitted_by = '@m3taversal' WHERE number = ?",
|
||||
(pr["number"],),
|
||||
)
|
||||
updated += 1
|
||||
|
|
|
|||
|
|
@ -1,399 +1,79 @@
|
|||
"""Claims API — list endpoint + canonical claim detail page.
|
||||
|
||||
Owner: Argus
|
||||
Routes:
|
||||
GET /api/claims — list/filter (frontmatter scan, lightweight)
|
||||
GET /api/claims/{slug} — full claim detail (Ship contract)
|
||||
GET /api/domains — domain rollups for sidebar
|
||||
|
||||
The detail endpoint is the canonical /claims/{slug} backend per Ship's
|
||||
2026-04-29 brief. One round-trip, no N+1 cascade. Wikilinks resolved
|
||||
server-side via title→slug index built from a tree walk.
|
||||
"""
|
||||
import json
|
||||
"""Claims API endpoint — serves claim data from the codex filesystem."""
|
||||
import os
|
||||
import re
|
||||
import sqlite3
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
from pathlib import Path
|
||||
from aiohttp import web
|
||||
|
||||
# Codex tree roots — claims live in three places (Sourcer Apr 26 fix scope)
|
||||
CODEX_BASE = Path("/opt/teleo-eval/workspaces/main")
|
||||
CLAIM_TREES = [CODEX_BASE / "domains", CODEX_BASE / "foundations", CODEX_BASE / "core"]
|
||||
CODEX_ROOT = Path("/opt/teleo-eval/workspaces/main/domains")
|
||||
_cache = {"data": None, "ts": 0}
|
||||
CACHE_TTL = 300 # 5 minutes
|
||||
|
||||
# pipeline.db for joins (review_records, prs, sources)
|
||||
DB_PATH = "/opt/teleo-eval/pipeline/pipeline.db"
|
||||
|
||||
# In-process caches
|
||||
_list_cache = {"data": None, "ts": 0}
|
||||
_LIST_CACHE_TTL = 300 # 5 min — list view tolerates staleness
|
||||
|
||||
_index_cache = {"by_title": None, "by_stem": None, "ts": 0}
|
||||
_INDEX_CACHE_TTL = 60 # 1 min — title→slug index for wikilink resolution
|
||||
|
||||
CORS_HEADERS = {"Access-Control-Allow-Origin": "*"}
|
||||
|
||||
# Wikilink pattern. [[text]] or [[text|alias]] — we keep the link text only.
|
||||
_WIKILINK_RE = re.compile(r"\[\[([^\]|#]+?)(?:[#|][^\]]*)?\]\]")
|
||||
|
||||
|
||||
# ─── Normalization ─────────────────────────────────────────────────────────
|
||||
|
||||
def _normalize_for_match(s):
|
||||
"""Collapse a title or slug to a comparable form.
|
||||
|
||||
Rules (from Ship's brief — match the link-fixer canonicalization):
|
||||
- lowercase
|
||||
- hyphen ↔ space tolerant (both → single space)
|
||||
- collapse runs of whitespace
|
||||
- strip leading/trailing whitespace
|
||||
- drop trailing punctuation that gets stripped from filenames
|
||||
(`.`, `?`, `!`, `:`, `--`)
|
||||
NOTE: lib/attribution.py exposes only normalize_handle today, not the
|
||||
title normalizer Ship referenced. Implementing inline; if a canonical
|
||||
helper lands later we point at it.
|
||||
"""
|
||||
if not s:
|
||||
return ""
|
||||
s = str(s).lower().strip()
|
||||
# Treat hyphens as spaces, then collapse whitespace runs
|
||||
s = s.replace("-", " ").replace("_", " ")
|
||||
s = re.sub(r"\s+", " ", s)
|
||||
# Strip ASCII punctuation that filenames drop
|
||||
s = re.sub(r"[^\w\s]", "", s)
|
||||
return s.strip()
|
||||
|
||||
|
||||
# ─── Frontmatter parse ─────────────────────────────────────────────────────
|
||||
|
||||
_CODE_FENCE_WRAPPER_RE = re.compile(r"^\s*```(?:markdown|md)?\s*\n(.*?)\n```\s*$", re.DOTALL)
|
||||
|
||||
|
||||
def _split_frontmatter(text):
|
||||
"""Return (frontmatter_dict, body_str) or (None, None) if not a claim file.
|
||||
|
||||
Tolerates files wrapped in a top-level ```markdown ... ``` code fence —
|
||||
some agents have produced these (e.g. Montreal Protocol claim from Astra,
|
||||
2024-12-09). Unwrap once before frontmatter detection.
|
||||
"""
|
||||
if not text:
|
||||
return None, None
|
||||
m = _CODE_FENCE_WRAPPER_RE.match(text)
|
||||
if m:
|
||||
text = m.group(1)
|
||||
text = text.lstrip()
|
||||
if not text.startswith("---"):
|
||||
return None, None
|
||||
try:
|
||||
end = text.index("\n---", 3)
|
||||
except ValueError:
|
||||
return None, None
|
||||
try:
|
||||
fm = yaml.safe_load(text[3:end])
|
||||
except Exception:
|
||||
return None, None
|
||||
if not isinstance(fm, dict):
|
||||
return None, None
|
||||
body = text[end + 4:].lstrip()
|
||||
return fm, body
|
||||
|
||||
|
||||
def _read_claim_file(filepath):
|
||||
"""Read a claim file from disk. Returns (frontmatter, body) or (None, None)."""
|
||||
def _parse_frontmatter(filepath):
|
||||
try:
|
||||
text = filepath.read_text(encoding="utf-8")
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return None, None
|
||||
return _split_frontmatter(text)
|
||||
if not text.startswith("---"):
|
||||
return None
|
||||
end = text.index("---", 3)
|
||||
fm = yaml.safe_load(text[3:end])
|
||||
if not fm or fm.get("type") != "claim":
|
||||
return None
|
||||
body = text[end+3:].strip()
|
||||
# Count wiki-links
|
||||
links = re.findall(r"\[\[([^\]]+)\]\]", body)
|
||||
# Extract first paragraph as summary
|
||||
paragraphs = [p.strip() for p in body.split("\n\n") if p.strip() and not p.strip().startswith("#")]
|
||||
summary = paragraphs[0][:300] if paragraphs else ""
|
||||
return {
|
||||
"slug": filepath.stem,
|
||||
"title": fm.get("title", filepath.stem.replace("-", " ")),
|
||||
"domain": fm.get("domain", "unknown"),
|
||||
"confidence": fm.get("confidence", "unknown"),
|
||||
"agent": fm.get("agent"),
|
||||
"scope": fm.get("scope"),
|
||||
"created": str(fm.get("created", "")),
|
||||
"source": fm.get("source", "") if isinstance(fm.get("source"), str) else "",
|
||||
"sourcer": fm.get("sourcer", ""),
|
||||
"wiki_link_count": len(links),
|
||||
"summary": summary,
|
||||
"challenged_by": fm.get("challenged_by"),
|
||||
"related_claims": fm.get("related_claims", []),
|
||||
}
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
# ─── Tree walk + indexing ──────────────────────────────────────────────────
|
||||
def _load_all_claims():
|
||||
now = time.time()
|
||||
if _cache["data"] and now - _cache["ts"] < CACHE_TTL:
|
||||
return _cache["data"]
|
||||
|
||||
def _walk_claim_files():
|
||||
"""Yield Path objects for every .md claim file in domains/, foundations/, core/."""
|
||||
for root in CLAIM_TREES:
|
||||
if not root.exists():
|
||||
claims = []
|
||||
for domain_dir in sorted(CODEX_ROOT.iterdir()):
|
||||
if not domain_dir.is_dir():
|
||||
continue
|
||||
for f in root.rglob("*.md"):
|
||||
for f in sorted(domain_dir.glob("*.md")):
|
||||
if f.name == "_map.md":
|
||||
continue
|
||||
yield f
|
||||
c = _parse_frontmatter(f)
|
||||
if c:
|
||||
claims.append(c)
|
||||
|
||||
|
||||
def _build_indexes():
|
||||
"""Build (title→stem, stem→relpath) indexes for wikilink resolution.
|
||||
|
||||
Cached for _INDEX_CACHE_TTL. Pulls from claim-index endpoint when
|
||||
possible (already cached upstream) and falls back to filesystem walk.
|
||||
"""
|
||||
now = time.time()
|
||||
if _index_cache["by_title"] is not None and now - _index_cache["ts"] < _INDEX_CACHE_TTL:
|
||||
return _index_cache["by_title"], _index_cache["by_stem"]
|
||||
|
||||
by_title = {}
|
||||
by_stem = {}
|
||||
for f in _walk_claim_files():
|
||||
stem = f.stem
|
||||
rel = str(f.relative_to(CODEX_BASE))
|
||||
by_stem[stem] = rel
|
||||
# Index by stem-as-normalized too (covers wikilinks that use the slug)
|
||||
by_title[_normalize_for_match(stem)] = stem
|
||||
# Also try parsing the title from frontmatter for higher-fidelity matches
|
||||
fm, _ = _read_claim_file(f)
|
||||
if fm:
|
||||
title = fm.get("title")
|
||||
if title:
|
||||
key = _normalize_for_match(title)
|
||||
if key and key not in by_title:
|
||||
by_title[key] = stem
|
||||
|
||||
_index_cache["by_title"] = by_title
|
||||
_index_cache["by_stem"] = by_stem
|
||||
_index_cache["ts"] = now
|
||||
return by_title, by_stem
|
||||
|
||||
|
||||
def _resolve_wikilinks(body, by_title):
|
||||
"""Extract [[link]] occurrences from body, return {link_text: slug_or_null}."""
|
||||
out = {}
|
||||
for match in _WIKILINK_RE.finditer(body or ""):
|
||||
link_text = match.group(1).strip()
|
||||
if not link_text or link_text in out:
|
||||
continue
|
||||
norm = _normalize_for_match(link_text)
|
||||
out[link_text] = by_title.get(norm)
|
||||
return out
|
||||
|
||||
|
||||
# ─── Edge extraction from frontmatter ──────────────────────────────────────
|
||||
|
||||
_EDGE_FIELDS = {
|
||||
"supports": "supports",
|
||||
"challenges": "challenges",
|
||||
"challenged_by": "challenges", # canonical: store as challenges direction
|
||||
"related": "related",
|
||||
"related_claims": "related",
|
||||
"depends_on": "depends_on",
|
||||
}
|
||||
|
||||
|
||||
def _extract_edges(fm, by_title, by_stem):
|
||||
"""Return edges dict shaped per Ship's contract.
|
||||
|
||||
Each edge is {slug, title, exists}. Slug resolved through title index.
|
||||
"""
|
||||
edges = {"supports": [], "challenges": [], "related": [], "depends_on": []}
|
||||
|
||||
for fm_key, edge_kind in _EDGE_FIELDS.items():
|
||||
raw = fm.get(fm_key)
|
||||
if not raw:
|
||||
continue
|
||||
items = raw if isinstance(raw, list) else [raw]
|
||||
for item in items:
|
||||
if not isinstance(item, str):
|
||||
continue
|
||||
text = item.strip()
|
||||
# Strip wikilink wrapping if present
|
||||
text = re.sub(r"^\[\[|\]\]$", "", text)
|
||||
# Strip pipe annotations: "[[link|alias]]" style or "claim | edge_type | date"
|
||||
text = text.split("|")[0].strip()
|
||||
if not text:
|
||||
continue
|
||||
# Try title match first, fall back to stem match
|
||||
slug = by_title.get(_normalize_for_match(text))
|
||||
if not slug and text in by_stem:
|
||||
slug = text
|
||||
edges[edge_kind].append({
|
||||
"slug": slug,
|
||||
"title": text,
|
||||
"exists": slug is not None,
|
||||
})
|
||||
|
||||
return edges
|
||||
|
||||
|
||||
# ─── Source provenance ─────────────────────────────────────────────────────
|
||||
|
||||
def _resolve_sourced_from(conn, claim_filepath, fm, title, stem):
|
||||
"""Build sourced_from list for the claim.
|
||||
|
||||
Strategy: find PRs that produced this claim (via prs.description LIKE
|
||||
or branch slug match), look at prs.source_path → inbox archive file →
|
||||
parse that source's frontmatter for title/url. Falls back to the raw
|
||||
`source` string from the claim's own frontmatter.
|
||||
|
||||
Both `title` and `stem` must be non-empty — caller (handler) already
|
||||
falls back stem→title; passing empty values would leak `LIKE '%%'`
|
||||
and match unrelated PRs.
|
||||
"""
|
||||
out = []
|
||||
seen_paths = set()
|
||||
pr_rows = []
|
||||
if (title or "").strip() and (stem or "").strip():
|
||||
try:
|
||||
pr_rows = conn.execute(
|
||||
"""SELECT DISTINCT source_path
|
||||
FROM prs
|
||||
WHERE source_path IS NOT NULL AND source_path != ''
|
||||
AND (description LIKE ? OR branch LIKE ?)
|
||||
LIMIT 10""",
|
||||
(f"%{title}%", f"%{stem}%"),
|
||||
).fetchall()
|
||||
except sqlite3.OperationalError:
|
||||
pr_rows = []
|
||||
|
||||
for row in pr_rows:
|
||||
path = row["source_path"]
|
||||
if not path or path in seen_paths:
|
||||
continue
|
||||
seen_paths.add(path)
|
||||
out.append(_resolve_source_file(path))
|
||||
|
||||
# 2. Fallback: parse raw source frontmatter field if no PR match
|
||||
if not out:
|
||||
raw = fm.get("source")
|
||||
if isinstance(raw, str) and raw.strip():
|
||||
out.append({"path": None, "title": raw.strip()[:200], "url": None})
|
||||
|
||||
return out
|
||||
|
||||
|
||||
def _resolve_source_file(rel_path):
|
||||
"""Given inbox/archive/... path, parse frontmatter for title+url. Best-effort."""
|
||||
full = CODEX_BASE / rel_path
|
||||
entry = {"path": rel_path, "title": None, "url": None}
|
||||
if full.exists():
|
||||
fm, _ = _read_claim_file(full)
|
||||
if fm:
|
||||
entry["title"] = fm.get("title") or fm.get("source") or rel_path
|
||||
entry["url"] = fm.get("url")
|
||||
if not entry["title"]:
|
||||
# Last resort: derive from filename
|
||||
entry["title"] = Path(rel_path).stem.replace("-", " ")
|
||||
return entry
|
||||
|
||||
|
||||
# ─── Reviews + PRs ─────────────────────────────────────────────────────────
|
||||
|
||||
def _load_pr_history(conn, title, stem):
|
||||
"""Find PRs that touched this claim and their reviews.
|
||||
|
||||
Both title and stem must be non-empty strings — empty leaks `LIKE '%%'`
|
||||
which matches every PR. Handler already populates a fallback so this
|
||||
is a defense-in-depth guard.
|
||||
"""
|
||||
if not (title or "").strip() or not (stem or "").strip():
|
||||
return [], []
|
||||
|
||||
try:
|
||||
pr_rows = conn.execute(
|
||||
"""SELECT number, merged_at, commit_type, agent, branch, status
|
||||
FROM prs
|
||||
WHERE merged_at IS NOT NULL
|
||||
AND (description LIKE ? OR branch LIKE ?)
|
||||
ORDER BY merged_at ASC
|
||||
LIMIT 50""",
|
||||
(f"%{title}%", f"%{stem}%"),
|
||||
).fetchall()
|
||||
except sqlite3.OperationalError:
|
||||
return [], []
|
||||
|
||||
prs = [
|
||||
{
|
||||
"number": r["number"],
|
||||
"merged_at": r["merged_at"],
|
||||
"kind": r["commit_type"] or "unknown",
|
||||
"agent": r["agent"],
|
||||
"branch": r["branch"],
|
||||
}
|
||||
for r in pr_rows
|
||||
]
|
||||
|
||||
pr_numbers = [p["number"] for p in prs]
|
||||
if not pr_numbers:
|
||||
return prs, []
|
||||
|
||||
placeholders = ",".join("?" * len(pr_numbers))
|
||||
try:
|
||||
review_rows = conn.execute(
|
||||
f"""SELECT pr_number, reviewer, reviewer_model, outcome,
|
||||
rejection_reason, notes, reviewed_at
|
||||
FROM review_records
|
||||
WHERE pr_number IN ({placeholders})
|
||||
ORDER BY reviewed_at ASC""",
|
||||
pr_numbers,
|
||||
).fetchall()
|
||||
except sqlite3.OperationalError:
|
||||
review_rows = []
|
||||
|
||||
reviews = [
|
||||
{
|
||||
"pr_number": r["pr_number"],
|
||||
"reviewer": r["reviewer"],
|
||||
"model": r["reviewer_model"],
|
||||
"outcome": r["outcome"],
|
||||
"rejection_reason": r["rejection_reason"],
|
||||
"notes": r["notes"],
|
||||
"reviewed_at": r["reviewed_at"],
|
||||
}
|
||||
for r in review_rows
|
||||
]
|
||||
return prs, reviews
|
||||
|
||||
|
||||
# ─── List view (preserved) ─────────────────────────────────────────────────
|
||||
|
||||
def _parse_list_entry(filepath):
|
||||
fm, body = _read_claim_file(filepath)
|
||||
if not fm or fm.get("type") != "claim":
|
||||
return None
|
||||
links = _WIKILINK_RE.findall(body or "")
|
||||
paragraphs = [p.strip() for p in (body or "").split("\n\n")
|
||||
if p.strip() and not p.strip().startswith("#")]
|
||||
summary = paragraphs[0][:300] if paragraphs else ""
|
||||
return {
|
||||
"slug": filepath.stem,
|
||||
"title": fm.get("title", filepath.stem.replace("-", " ")),
|
||||
"domain": fm.get("domain", "unknown"),
|
||||
"confidence": fm.get("confidence", "unknown"),
|
||||
"agent": fm.get("agent"),
|
||||
"scope": fm.get("scope"),
|
||||
"created": str(fm.get("created", "")),
|
||||
"source": fm.get("source", "") if isinstance(fm.get("source"), str) else "",
|
||||
"sourcer": fm.get("sourcer", ""),
|
||||
"wiki_link_count": len(links),
|
||||
"summary": summary,
|
||||
"challenged_by": fm.get("challenged_by"),
|
||||
"related_claims": fm.get("related_claims", []),
|
||||
}
|
||||
|
||||
|
||||
def _load_all_claims_list():
|
||||
now = time.time()
|
||||
if _list_cache["data"] and now - _list_cache["ts"] < _LIST_CACHE_TTL:
|
||||
return _list_cache["data"]
|
||||
claims = []
|
||||
for f in _walk_claim_files():
|
||||
entry = _parse_list_entry(f)
|
||||
if entry:
|
||||
claims.append(entry)
|
||||
_list_cache["data"] = claims
|
||||
_list_cache["ts"] = now
|
||||
_cache["data"] = claims
|
||||
_cache["ts"] = now
|
||||
return claims
|
||||
|
||||
|
||||
# ─── Handlers ──────────────────────────────────────────────────────────────
|
||||
|
||||
async def handle_claims(request):
|
||||
claims = _load_all_claims_list()
|
||||
claims = _load_all_claims()
|
||||
|
||||
# Filters
|
||||
domain = request.query.get("domain")
|
||||
search = request.query.get("q", "").lower()
|
||||
confidence = request.query.get("confidence")
|
||||
agent = request.query.get("agent")
|
||||
sort = request.query.get("sort", "recent")
|
||||
sort = request.query.get("sort", "recent") # recent, alpha, domain
|
||||
|
||||
filtered = claims
|
||||
if domain:
|
||||
|
|
@ -403,9 +83,9 @@ async def handle_claims(request):
|
|||
if agent:
|
||||
filtered = [c for c in filtered if c["agent"] == agent]
|
||||
if search:
|
||||
filtered = [c for c in filtered
|
||||
if search in c["title"].lower() or search in c["summary"].lower()]
|
||||
filtered = [c for c in filtered if search in c["title"].lower() or search in c["summary"].lower()]
|
||||
|
||||
# Sort
|
||||
if sort == "recent":
|
||||
filtered.sort(key=lambda c: c["created"], reverse=True)
|
||||
elif sort == "alpha":
|
||||
|
|
@ -413,10 +93,12 @@ async def handle_claims(request):
|
|||
elif sort == "domain":
|
||||
filtered.sort(key=lambda c: (c["domain"], c["title"].lower()))
|
||||
|
||||
# Pagination
|
||||
limit = min(int(request.query.get("limit", "50")), 200)
|
||||
offset = int(request.query.get("offset", "0"))
|
||||
page = filtered[offset:offset + limit]
|
||||
page = filtered[offset:offset+limit]
|
||||
|
||||
# Domain counts for sidebar
|
||||
domain_counts = {}
|
||||
for c in claims:
|
||||
domain_counts[c["domain"]] = domain_counts.get(c["domain"], 0) + 1
|
||||
|
|
@ -429,114 +111,31 @@ async def handle_claims(request):
|
|||
"domains": dict(sorted(domain_counts.items(), key=lambda x: -x[1])),
|
||||
"confidence_levels": sorted(set(c["confidence"] for c in claims)),
|
||||
"agents": sorted(set(c["agent"] for c in claims if c["agent"])),
|
||||
}, headers=CORS_HEADERS)
|
||||
}, headers={"Access-Control-Allow-Origin": "*"})
|
||||
|
||||
|
||||
async def handle_claim_detail(request):
|
||||
"""GET /api/claims/{slug} — canonical claim detail page (Ship contract).
|
||||
|
||||
One round-trip, all data resolved server-side. Wikilinks pre-resolved.
|
||||
"""
|
||||
requested_slug = request.match_info["slug"]
|
||||
by_title, by_stem = _build_indexes()
|
||||
|
||||
# Resolution order: exact stem → title-normalized (handles description-derived
|
||||
# slugs from /api/activity-feed that are longer than on-disk file stems) →
|
||||
# stem-as-prefix (handles description-derived slugs that are shorter than the
|
||||
# file stem because the description was truncated upstream).
|
||||
slug = requested_slug
|
||||
rel_path = by_stem.get(slug)
|
||||
if not rel_path:
|
||||
# Title fallback: requested slug = slugified frontmatter title
|
||||
norm = _normalize_for_match(requested_slug)
|
||||
resolved_stem = by_title.get(norm)
|
||||
if resolved_stem:
|
||||
slug = resolved_stem
|
||||
rel_path = by_stem.get(resolved_stem)
|
||||
if not rel_path:
|
||||
# Prefix fallback: walk stems sharing a common prefix with the request,
|
||||
# pick longest match. Anchored at 32 chars to avoid spurious hits.
|
||||
norm_req = _normalize_for_match(requested_slug)
|
||||
best_stem = None
|
||||
best_len = 0
|
||||
for stem in by_stem:
|
||||
norm_stem = _normalize_for_match(stem)
|
||||
common = 0
|
||||
for a, b in zip(norm_req, norm_stem):
|
||||
if a != b:
|
||||
slug = request.match_info["slug"]
|
||||
claims = _load_all_claims()
|
||||
for c in claims:
|
||||
if c["slug"] == slug:
|
||||
# Read full body for detail view
|
||||
for domain_dir in CODEX_ROOT.iterdir():
|
||||
if not domain_dir.is_dir():
|
||||
continue
|
||||
f = domain_dir / f"{slug}.md"
|
||||
if f.exists():
|
||||
text = f.read_text(encoding="utf-8")
|
||||
end = text.index("---", 3)
|
||||
body = text[end+3:].strip()
|
||||
c["body"] = body
|
||||
break
|
||||
common += 1
|
||||
if common >= 32 and common > best_len:
|
||||
best_stem = stem
|
||||
best_len = common
|
||||
if best_stem:
|
||||
slug = best_stem
|
||||
rel_path = by_stem.get(best_stem)
|
||||
if not rel_path:
|
||||
return web.json_response({"error": "claim not found", "slug": requested_slug},
|
||||
status=404, headers=CORS_HEADERS)
|
||||
|
||||
filepath = CODEX_BASE / rel_path
|
||||
fm, body = _read_claim_file(filepath)
|
||||
if not fm:
|
||||
# File exists at this stem but has no parseable frontmatter — almost
|
||||
# always a stray enrichment fragment that landed in domains/ without
|
||||
# being merged into a parent claim. Surfacing as 404 (no claim here)
|
||||
# not 500: the caller can't act on it differently anyway.
|
||||
return web.json_response({"error": "claim not found", "slug": slug,
|
||||
"reason": "file_no_frontmatter"},
|
||||
status=404, headers=CORS_HEADERS)
|
||||
|
||||
# Open read-only DB connection for this request
|
||||
conn = sqlite3.connect(f"file:{DB_PATH}?mode=ro", uri=True)
|
||||
conn.row_factory = sqlite3.Row
|
||||
try:
|
||||
title = fm.get("title") or slug.replace("-", " ")
|
||||
prs, reviews = _load_pr_history(conn, title, slug)
|
||||
sourced_from = _resolve_sourced_from(conn, filepath, fm, title, slug)
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
last_review = None
|
||||
if reviews:
|
||||
latest = reviews[-1]
|
||||
last_review = {
|
||||
"outcome": latest["outcome"],
|
||||
"reviewer": latest["reviewer"],
|
||||
"date": (latest["reviewed_at"] or "")[:10],
|
||||
}
|
||||
|
||||
# secondary_domains: explicit list, or empty
|
||||
secondary = fm.get("secondary_domains") or fm.get("cross_domain_links") or []
|
||||
if isinstance(secondary, str):
|
||||
secondary = [secondary]
|
||||
|
||||
description = fm.get("description") or ""
|
||||
|
||||
edges = _extract_edges(fm, by_title, by_stem)
|
||||
wikilinks = _resolve_wikilinks(body, by_title)
|
||||
|
||||
response = {
|
||||
"slug": slug,
|
||||
"title": title,
|
||||
"domain": fm.get("domain", "unknown"),
|
||||
"secondary_domains": secondary,
|
||||
"confidence": fm.get("confidence", "unknown"),
|
||||
"description": description,
|
||||
"created": str(fm.get("created", "")),
|
||||
"last_review": last_review,
|
||||
"body": body or "",
|
||||
"sourced_from": sourced_from,
|
||||
"reviews": reviews,
|
||||
"prs": prs,
|
||||
"edges": edges,
|
||||
"wikilinks": wikilinks,
|
||||
}
|
||||
return web.json_response(response, headers=CORS_HEADERS)
|
||||
return web.json_response(c, headers={"Access-Control-Allow-Origin": "*"})
|
||||
return web.json_response({"error": "claim not found"}, status=404)
|
||||
|
||||
|
||||
async def handle_domains(request):
|
||||
claims = _load_all_claims_list()
|
||||
claims = _load_all_claims()
|
||||
domains = {}
|
||||
for c in claims:
|
||||
d = c["domain"]
|
||||
|
|
@ -547,11 +146,13 @@ async def handle_domains(request):
|
|||
domains[d]["agents"].add(c["agent"])
|
||||
conf = c["confidence"]
|
||||
domains[d]["confidence_dist"][conf] = domains[d]["confidence_dist"].get(conf, 0) + 1
|
||||
|
||||
result = []
|
||||
for d in sorted(domains.values(), key=lambda x: -x["count"]):
|
||||
d["agents"] = sorted(d["agents"])
|
||||
result.append(d)
|
||||
return web.json_response(result, headers=CORS_HEADERS)
|
||||
|
||||
return web.json_response(result, headers={"Access-Control-Allow-Origin": "*"})
|
||||
|
||||
|
||||
def register_claims_routes(app):
|
||||
|
|
|
|||
|
|
@ -1,54 +0,0 @@
|
|||
{
|
||||
"status": "blocked_remote_execution",
|
||||
"scope": "crabbox remote proof",
|
||||
"attempted_discovery": [
|
||||
"verified Crabbox CLI is installed at /Users/user/.local/bin/crabbox",
|
||||
"ran crabbox job list",
|
||||
"ran crabbox sync-plan",
|
||||
"ran crabbox job run --dry-run unit",
|
||||
"ran crabbox job run --dry-run phase1b-local-proof",
|
||||
"checked presence of CRABBOX_COORDINATOR, CRABBOX_COORDINATOR_TOKEN, HCLOUD_TOKEN, HETZNER_TOKEN, GH_TOKEN, and GITHUB_TOKEN without printing values",
|
||||
"loaded retained Bitwarden session from /tmp/bw_session without printing the session value",
|
||||
"ran bw status and bw sync",
|
||||
"checked Bitwarden organization, collection, and item counts",
|
||||
"checked visible Bitwarden item names and metadata only",
|
||||
"scanned visible Bitwarden item names and notes for crabbox, hcloud, hetzner, and coordinator terms without printing note or secret values"
|
||||
],
|
||||
"exact_blocker": "Crabbox provider execution still lacks a real provider credential: HCLOUD_TOKEN, HETZNER_TOKEN, CRABBOX_COORDINATOR, and CRABBOX_COORDINATOR_TOKEN are unset, and the visible Bitwarden org collection contains only Anthropic API Key, Leo twitter, and LivingIPbot Github, with no Crabbox, HCloud, Hetzner, or coordinator metadata match.",
|
||||
"why_it_cannot_be_solved_autonomously": "A remote Crabbox lease requires a real Hetzner or Crabbox broker credential. The repo can safely commit CI/CD config, dry-run plans, and blocker artifacts, but it cannot fabricate the provider credential or commit secret values.",
|
||||
"exact_next_action": "Add a scoped Hetzner/Crabbox broker credential to Bitwarden or GitHub environment secrets as HCLOUD_TOKEN, HETZNER_TOKEN, CRABBOX_COORDINATOR, or CRABBOX_COORDINATOR_TOKEN, then rerun crabbox doctor --json and crabbox job run phase1b-local-proof from teleo-infrastructure.",
|
||||
"safe_local_status": {
|
||||
"crabbox_cli_installed": "0.22.1",
|
||||
"job_list": "passes",
|
||||
"sync_plan": "217 files, 2.4 MiB",
|
||||
"unit_dry_run": "passes",
|
||||
"phase1b_proof_dry_run": "passes",
|
||||
"ci_contract_guard": "passes",
|
||||
"phase1b_proof_wrapper": "131 passed, 8 proof cases succeeded, all six agents seen",
|
||||
"full_pytest": "422 passed",
|
||||
"crabbox_doctor": "fails only provider credential check: HCLOUD_TOKEN or HETZNER_TOKEN is required",
|
||||
"bitwarden_status": "unlocked",
|
||||
"bitwarden_organizations": 1,
|
||||
"bitwarden_collections": 1,
|
||||
"bitwarden_items_visible": 3,
|
||||
"bitwarden_matching_crabbox_or_hetzner_items": 0
|
||||
},
|
||||
"secret_commit_policy": {
|
||||
"allowed_to_commit": [
|
||||
"workflow files",
|
||||
"Crabbox config with secret slot names omitted",
|
||||
"proof scripts",
|
||||
"machine-readable blocker artifacts",
|
||||
"docs and agent skills"
|
||||
],
|
||||
"not_allowed_to_commit": [
|
||||
"Bitwarden item values",
|
||||
"Bitwarden vault exports",
|
||||
"provider tokens",
|
||||
"GitHub bot tokens",
|
||||
"OpenRouter keys",
|
||||
"SSH private keys",
|
||||
"production databases"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
|
@ -1,96 +0,0 @@
|
|||
# Crabbox Remote Proof
|
||||
|
||||
Crabbox is the remote execution layer for `teleo-infrastructure`. It is not the production deploy system.
|
||||
|
||||
## Goals
|
||||
|
||||
- Run Python tests on a disposable or warm remote Linux box.
|
||||
- Prove the CI/Crabbox contract without network access before remote runs.
|
||||
- Run the Phase 1B local proof script remotely.
|
||||
- Retain JUnit and machine-readable proof artifacts.
|
||||
- Give agents a bounded job list instead of arbitrary cloud shell access.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- No production deploys.
|
||||
- No production secrets.
|
||||
- No production VPS mutation.
|
||||
- No production `decision-engine` PR comments from Crabbox jobs.
|
||||
|
||||
## Required Local Setup
|
||||
|
||||
Crabbox CLI 0.22.1 or newer:
|
||||
|
||||
```bash
|
||||
crabbox --version
|
||||
```
|
||||
|
||||
One of:
|
||||
|
||||
```bash
|
||||
crabbox login --url "$CRABBOX_COORDINATOR"
|
||||
```
|
||||
|
||||
or direct Hetzner operator env:
|
||||
|
||||
```bash
|
||||
export HCLOUD_TOKEN="..."
|
||||
```
|
||||
|
||||
Do not commit either value.
|
||||
|
||||
## Jobs
|
||||
|
||||
```bash
|
||||
crabbox job list
|
||||
crabbox job run --dry-run ci-contract
|
||||
crabbox job run --dry-run unit
|
||||
crabbox job run --dry-run phase1b-local-proof
|
||||
crabbox job run ci-contract
|
||||
crabbox job run unit
|
||||
crabbox job run phase1b-local-proof
|
||||
```
|
||||
|
||||
`ci-contract` writes:
|
||||
|
||||
- `.crabbox-results/crabbox-ci-contract.json`
|
||||
|
||||
`phase1b-local-proof` writes:
|
||||
|
||||
- `.crabbox-results/crabbox-ci-contract.json`
|
||||
- `proof/phase1b-local-e2e-proof.json`
|
||||
- `.crabbox-results/phase1b-pytest.xml`
|
||||
- `.crabbox-results/phase1b-proof-summary.json`
|
||||
|
||||
The contract proof checks that:
|
||||
|
||||
- Crabbox exposes only the named bounded jobs.
|
||||
- sync excludes secret/runtime files such as `.env`, `secrets`, DBs, logs, caches, and virtualenvs.
|
||||
- `.crabbox.yaml` contains no token-bearing env names.
|
||||
- Leo routes are explicit: Leo-owned domains, fallback routes, and top-2 cross-domain routes that include Leo are covered, while Phase 1B does not silently preserve Leo as a universal second reviewer.
|
||||
|
||||
## Secret Boundary
|
||||
|
||||
Allowed:
|
||||
|
||||
- `CI`
|
||||
- `PYTHONWARNINGS`
|
||||
- `PHASE1B_AGENT_ROUTING_ENABLED`
|
||||
- broker token in user config
|
||||
- direct `HCLOUD_TOKEN` or `HETZNER_TOKEN` in local operator env
|
||||
- GitHub environment secrets named `HCLOUD_TOKEN` or `HETZNER_TOKEN` for an explicitly dispatched remote proof workflow
|
||||
|
||||
Not allowed:
|
||||
|
||||
- production GitHub admin token
|
||||
- production Forgejo token
|
||||
- production OpenRouter key
|
||||
- production SSH keys
|
||||
- Bitwarden exports
|
||||
- prod `pipeline.db`
|
||||
|
||||
Bitwarden may be used as the human/operator source of truth for secret lookup and GitHub secret setup, but no Bitwarden item value, vault export, or copied secret belongs in this repo. The committed config may name required secret slots; it must not contain the values.
|
||||
|
||||
## Proof Boundary
|
||||
|
||||
Crabbox remote proof proves repo behavior on a remote Linux lease. It does not prove production parity unless the lease recreates the production runtime paths, systemd services, timers, DB path, and deploy script behavior.
|
||||
|
|
@ -1,236 +0,0 @@
|
|||
# LLM Refinement And Decision Engine Program
|
||||
|
||||
Created: 2026-06-01
|
||||
Status: active direction
|
||||
|
||||
## Product Outcome
|
||||
|
||||
The decision engine should become the best judgment layer for Living IP: it routes knowledge changes to the right agent identities, tests competing LLMs against the same rubric, learns from disagreement, and improves prompts/tools only when measured deltas prove the change.
|
||||
|
||||
Pentagon.run should own disposable infrastructure and remote execution. This repo should own decision quality: rubrics, prompts, model selection, route evidence, database feedback loops, and agent tool packages.
|
||||
|
||||
## What Rio And Theseus Become
|
||||
|
||||
### Rio
|
||||
|
||||
Rio becomes the economic and incentive-quality evaluator.
|
||||
|
||||
Rio owns:
|
||||
|
||||
- contribution weights and role economics;
|
||||
- paid-query effects and anti-pay-to-pollute rules;
|
||||
- market, mechanism, futarchy, x402, token, and capital-formation reasoning;
|
||||
- source-diversity and correlated-prior warnings;
|
||||
- OPSEC for finance, deal terms, token economics, and internal allocations;
|
||||
- model tests that expose weak economic reasoning.
|
||||
|
||||
Rio should not be "the crypto agent". Rio should be the agent that asks whether the system's incentives create useful knowledge or garbage incentives.
|
||||
|
||||
### Theseus
|
||||
|
||||
Theseus becomes the model-integrity and agent-refinement evaluator.
|
||||
|
||||
Theseus owns:
|
||||
|
||||
- model diversity and correlated-blind-spot measurement;
|
||||
- adversarial eval rubrics;
|
||||
- prompt/tool safety and self-upgrade criteria;
|
||||
- disagreement queues and verifier-divergence analysis;
|
||||
- LLM capability evidence and agent-system architecture;
|
||||
- tests that expose hallucinated certainty, weak causal claims, and prompt-injection fragility.
|
||||
|
||||
Theseus should not be "the AI safety agent". Theseus should be the agent that asks whether the decision system can be trusted when the models are persuasive but wrong.
|
||||
|
||||
## Decision Engine Loop
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
PR["Decision-engine PR or source record"] --> Route["Deterministic route evidence"]
|
||||
Route --> Reviewers["Required agent reviewers"]
|
||||
Reviewers --> Rubric["Shared rubric"]
|
||||
Rubric --> ModelA["Primary model"]
|
||||
Rubric --> ModelB["Independent model family"]
|
||||
ModelA --> Verdicts["Structured verdicts"]
|
||||
ModelB --> Verdicts
|
||||
Verdicts --> Disagree{"Disagreement?"}
|
||||
Disagree -->|yes| Queue["Disagreement queue"]
|
||||
Disagree -->|no| Metrics["Calibration metrics"]
|
||||
Queue --> HumanOrLeo["Leo or human arbitration"]
|
||||
HumanOrLeo --> Metrics
|
||||
Metrics --> DB["SQLite feedback state"]
|
||||
DB --> Refine["Prompt, tool, or model proposal"]
|
||||
Refine --> Delta["Before/after eval harness"]
|
||||
Delta -->|passes| Update["Commit refinement"]
|
||||
Delta -->|fails| Archive["Archive failed refinement"]
|
||||
```
|
||||
|
||||
## Model Portfolio
|
||||
|
||||
The goal is not to pick one favorite model. The goal is to assign models to failure modes.
|
||||
|
||||
| Lane | Primary evaluator | Independent check | Why |
|
||||
| --- | --- | --- | --- |
|
||||
| Fast triage | cheap small model | deterministic route evidence | triage should be cheap and overridable |
|
||||
| Domain review | routed agent prompt | different model family | catch domain-specific errors without same-family agreement bias |
|
||||
| Deep review | strongest available reasoning model | non-Claude or non-primary family | deep review is for structural claims and disagreement |
|
||||
| Economic reasoning | Rio rubric | model with strong quantitative/mechanism reasoning | tests incentive design, paid-query effects, and contribution weights |
|
||||
| Agent/refinement safety | Theseus rubric | model with strong adversarial critique | tests tool safety, self-upgrades, and evaluator drift |
|
||||
|
||||
Candidate models should enter only through a harness:
|
||||
|
||||
1. fixed input set;
|
||||
2. fixed rubric;
|
||||
3. structured verdict JSON;
|
||||
4. cost and latency recorded;
|
||||
5. disagreement categories stored;
|
||||
6. before/after comparison against current baseline.
|
||||
|
||||
No model switch is accepted because it "sounds better" on one example.
|
||||
|
||||
## Refinement Workstreams
|
||||
|
||||
### R0: Model Discovery Registry
|
||||
|
||||
Create a registry before arguing about model preference. The registry should track:
|
||||
|
||||
- hosted frontier models;
|
||||
- open-weight Hugging Face candidates;
|
||||
- local or edge candidates;
|
||||
- small, cheap triage models;
|
||||
- larger reasoning models, including future in-house or 27B-class candidates;
|
||||
- license, hardware, context, latency, cost, tool support, and known failure modes.
|
||||
|
||||
The registry does not bless a model. It decides which model deserves a bakeoff fixture.
|
||||
|
||||
### R1: Rubric Packets
|
||||
|
||||
Create a small rubric packet for each evaluator role:
|
||||
|
||||
- `rio-economics-rubric`
|
||||
- `theseus-model-integrity-rubric`
|
||||
- `leo-cross-domain-rubric`
|
||||
- domain-specific factuality rubrics
|
||||
|
||||
Each packet must define allowed verdicts, rejection tags, must-check criteria, and examples of false positives.
|
||||
|
||||
### R2: Evaluation Corpus
|
||||
|
||||
Build a replayable corpus from existing PRs:
|
||||
|
||||
- approved clean PRs;
|
||||
- rejected PRs by issue tag;
|
||||
- Rio/Theseus cross-domain PRs;
|
||||
- paid-query or contribution-weight examples;
|
||||
- adversarial malformed claims;
|
||||
- near-duplicate and OPSEC edge cases.
|
||||
|
||||
Use local fixture data first. Production DB sampling requires the DB operator skill.
|
||||
|
||||
### R3: Model Bakeoff
|
||||
|
||||
Run each candidate model against the same corpus and emit:
|
||||
|
||||
- accuracy against expected disposition;
|
||||
- false-approve count;
|
||||
- false-reject count;
|
||||
- issue-tag precision;
|
||||
- average latency;
|
||||
- estimated cost;
|
||||
- disagreement matrix by model pair.
|
||||
|
||||
The highest-signal metric is not raw approval rate. It is false approvals on bad claims plus useful disagreement on ambiguous claims.
|
||||
|
||||
### R4: Feedback Loop
|
||||
|
||||
Use `review_records`, `audit_log`, `costs`, and PR state to find:
|
||||
|
||||
- recurring model failure categories;
|
||||
- agents with repeated same-tag rejections;
|
||||
- prompts that produce vague reviews;
|
||||
- cost spikes without quality gain;
|
||||
- routes that keep requiring manual override.
|
||||
|
||||
Every prompt/tool change should include a before/after proof over this loop.
|
||||
|
||||
### R5: Agent Runtime Packages
|
||||
|
||||
Package the same decision-engine contract for:
|
||||
|
||||
- NousResearch Hermes Agent: skill/memory/model-switching oriented.
|
||||
- OpenClaw: workspace skill plus `AGENTS.md`, `SOUL.md`, `TOOLS.md` oriented.
|
||||
- Claude-style, Pentagon, or other persistent agents: skill-oriented knowledge-base read/write interop.
|
||||
|
||||
Both packages should be fixture-first and no-secret by default. They are distribution surfaces for the decision engine, not separate evaluators with their own truth.
|
||||
|
||||
### R6: Knowledge-Base Interop
|
||||
|
||||
Any Hermes, OpenClaw, or Claude-style agent should be able to read information from the Living IP knowledge base and propose writes back into it.
|
||||
|
||||
The contract is:
|
||||
|
||||
- read through deterministic search, claim indexes, copied SQLite state, or cited repo files;
|
||||
- propose source, claim, entity, correction, and route artifacts;
|
||||
- never write directly to main;
|
||||
- never mutate production `pipeline.db` from a model response;
|
||||
- leave proof showing the exact query, cited reads, proposed write, and route evidence.
|
||||
|
||||
Use `.agents/skills/living-ip-kb-interop/SKILL.md` for runtime-neutral KB access, and `.agents/skills/teleo-db-operator/SKILL.md` for SQLite-specific work.
|
||||
|
||||
## DB Usage Boundary
|
||||
|
||||
Default is read-only.
|
||||
|
||||
Writes are allowed only when all are true:
|
||||
|
||||
- the target DB is local, staging, or explicitly authorized production;
|
||||
- a backup or copy exists;
|
||||
- the write is wrapped in a transaction;
|
||||
- the exact query is retained in a proof artifact;
|
||||
- the post-write readback is retained.
|
||||
|
||||
Never let an agent tune prompts by mutating production state directly.
|
||||
|
||||
## Pentagon.run Boundary
|
||||
|
||||
Pentagon.run should own:
|
||||
|
||||
- disposable VPS setup;
|
||||
- Crabbox or remote proof execution;
|
||||
- Hetzner lifecycle;
|
||||
- runner cleanup;
|
||||
- infra receipts.
|
||||
- persistent agent teammates, company-brain infrastructure, and agent-to-agent transport when that is their managed stack.
|
||||
|
||||
This repo should own:
|
||||
|
||||
- decision-engine quality;
|
||||
- model and prompt experiments;
|
||||
- agent skills and adapter handoffs;
|
||||
- database feedback analysis;
|
||||
- proof schemas for eval quality.
|
||||
|
||||
Raw cards and secrets are not agent runtime inputs. Human operators may decide vendor billing and spend policy, but repo artifacts should only name secret slots, scoped tokens, spend limits, receipts, and setup checklists.
|
||||
|
||||
## Transcript-Derived Requirements
|
||||
|
||||
The 2026-06-01 working transcript adds these requirements:
|
||||
|
||||
- LLM/refinement work should focus on model discovery, compression, context strategy, and decision-engine quality while Pentagon handles cloud/persistent-agent infrastructure.
|
||||
- Rio should be the first place to route Meteora, LP, x402, futarchy, paid-query, and contribution-incentive questions.
|
||||
- Theseus should own the skill/MCP/refinement path that makes model judgment portable across Hermes, OpenClaw, Claude-style agents, and Pentagon-style company brains.
|
||||
- The knowledge-writing path should turn large founder/source corpora into structured, reviewable knowledge packets, not shallow summaries.
|
||||
- Slack, Linear, email, billing, and provider accounts are external collaboration setup. They should unblock people, but they are not prerequisites for local fixture, rubric, and proof work.
|
||||
|
||||
## Next Implementation Slice
|
||||
|
||||
1. Add `docs/model-discovery-registry.md`.
|
||||
2. Add `scripts/replay_decision_engine_eval.py` with local fixture mode.
|
||||
3. Add `fixtures/decision-engine-eval/*.json`.
|
||||
4. Store verdict outputs in `.crabbox-results/decision-engine-eval.json`.
|
||||
5. Add one Rio economics fixture and one Theseus model-integrity fixture.
|
||||
6. Add one KB interop fixture that searches existing context and proposes a write without touching main or production DB.
|
||||
7. Compare current prompt versus one candidate prompt before touching runtime prompts.
|
||||
|
||||
Do not start by changing live model assignments.
|
||||
|
||||
Run `python3 scripts/replay_decision_engine_eval.py` after changing fixture, rubric, registry, or candidate-output formats.
|
||||
|
|
@ -1,75 +0,0 @@
|
|||
# Model Discovery Registry
|
||||
|
||||
Created: 2026-06-01
|
||||
Status: candidate registry, not model approval
|
||||
|
||||
This registry exists to decide which models deserve a Living IP bakeoff fixture. It does not choose production models and it does not replace measured replay results.
|
||||
|
||||
## Rules
|
||||
|
||||
- Use official provider docs, model cards, or source repositories for every entry.
|
||||
- Treat all model specs, prices, context limits, and aliases as volatile.
|
||||
- Do not switch runtime model assignments from this document alone.
|
||||
- Promote a model only after `scripts/replay_decision_engine_eval.py` shows no critical regression on the same fixture set.
|
||||
- Prefer different model families for independent review so agreement is not just same-family correlation.
|
||||
|
||||
## Candidate Matrix
|
||||
|
||||
| Candidate | Surface | Why It Is Worth Testing | First Living IP Lane | Source |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| GPT-5.5 / GPT-5.4 family | Hosted API | Strong general reasoning and agentic task baseline; useful as a frontier comparison point. | deep review, Leo arbitration | [OpenAI models](https://platform.openai.com/docs/models) |
|
||||
| GPT-5 lower-latency variants | Hosted API | Possible cheap triage candidates; exact model IDs must be re-verified before a bakeoff run. | fast triage | [OpenAI models](https://platform.openai.com/docs/models) |
|
||||
| gpt-oss-120b | Open-weight | Open-weight reasoning candidate for on-prem or Pentagon-managed inference; needs hardware/cost proof. | Theseus model integrity | [OpenAI open models](https://openai.com/open-models/) |
|
||||
| gpt-oss-20b | Open-weight | Smaller local/edge candidate for cheap first-pass triage and portable demos. | fast triage, local harness | [OpenAI open models](https://openai.com/open-models/) |
|
||||
| Claude Opus 4.8 | Hosted API | Complex-reasoning candidate for highest-stakes arbitration. | Leo arbitration, deep review | [Anthropic models overview](https://docs.anthropic.com/en/docs/about-claude/models) |
|
||||
| Claude Sonnet 4.6 | Hosted API | Speed/intelligence tradeoff candidate for domain review. | domain review | [Anthropic models overview](https://docs.anthropic.com/en/docs/about-claude/models) |
|
||||
| Claude Haiku 4.5 | Hosted API | Low-latency candidate for cheap reviewer pre-checks. | fast triage | [Anthropic models overview](https://docs.anthropic.com/en/docs/about-claude/models) |
|
||||
| Gemini 3.5 Flash | Hosted API | Agentic/coding-oriented candidate from a different model family. | independent second review | [Gemini API models](https://ai.google.dev/gemini-api/docs/models) |
|
||||
| Gemini 3.1 Pro | Hosted API | Complex problem-solving candidate from a non-primary model family. | deep review | [Gemini API models](https://ai.google.dev/gemini-api/docs/models) |
|
||||
| Mistral Medium 3.5 | Hosted or open surface per provider docs | Agentic/coding candidate with a non-US-primary model family. | independent second review | [Mistral models overview](https://docs.mistral.ai/getting-started/models/) |
|
||||
| Mistral Small 4 | Hosted or open surface per provider docs | Efficient hybrid instruct/reasoning/coding candidate. | fast triage, domain review | [Mistral models overview](https://docs.mistral.ai/getting-started/models/) |
|
||||
| Mistral Large 3 | Open-weight | Large open-weight comparison point for self-hosted evaluation. | deep review | [Mistral models overview](https://docs.mistral.ai/getting-started/models/) |
|
||||
| Devstral 2 | Hosted or open surface per provider docs | Code-agent candidate for tools, repository work, and adapter tasks. | Theseus tool integrity | [Mistral models overview](https://docs.mistral.ai/getting-started/models/) |
|
||||
| Hermes 4 70B | Open-weight / provider-hosted | Nous-aligned model with structured output and tool-use relevance for Hermes Agent packaging. | Hermes adapter, Theseus | [NousResearch Hermes 4 70B](https://huggingface.co/NousResearch/Hermes-4-70B) |
|
||||
| Qwen3.5 9B | Open-weight | Small multimodal/open-weight candidate for local and edge experiments. | fast triage, local harness | [Qwen3.5 9B model card](https://huggingface.co/Qwen/Qwen3.5-9B) |
|
||||
|
||||
## Bakeoff Intake Fields
|
||||
|
||||
Each candidate needs a retained record before a real bakeoff:
|
||||
|
||||
- provider or local runtime;
|
||||
- exact model ID or pinned snapshot;
|
||||
- source URL;
|
||||
- license or terms surface;
|
||||
- context window and max output if verified;
|
||||
- structured-output support;
|
||||
- tool/function calling support;
|
||||
- expected hardware or hosted cost;
|
||||
- latency estimate;
|
||||
- privacy and data-retention posture;
|
||||
- failure mode hypothesis;
|
||||
- first fixture lane.
|
||||
|
||||
## First Bakeoff Order
|
||||
|
||||
1. Cheap triage: exact-ID-verified GPT-5 lower-latency variant, Claude Haiku 4.5, Mistral Small 4, Qwen3.5 9B, gpt-oss-20b.
|
||||
2. Theseus integrity: Gemini 3.5 Flash, Hermes 4 70B, Devstral 2, gpt-oss-120b.
|
||||
3. Rio economics: GPT-5.5/5.4, Claude Sonnet 4.6, Gemini 3.1 Pro, Mistral Medium 3.5.
|
||||
4. Deep arbitration: Claude Opus 4.8, GPT-5.5, Gemini 3.1 Pro, Mistral Large 3.
|
||||
|
||||
## Promotion Gate
|
||||
|
||||
A model can move from registry to runtime proposal only if the replay proof includes:
|
||||
|
||||
- exact model ID;
|
||||
- fixture count;
|
||||
- route accuracy;
|
||||
- false approvals;
|
||||
- false rejects;
|
||||
- missing required issue tags;
|
||||
- average latency;
|
||||
- cost estimate;
|
||||
- disagreement matrix against current baseline;
|
||||
- one paragraph explaining why the observed disagreements are useful.
|
||||
|
||||
Zero false approvals on known-bad fixtures is a hard gate for evaluator roles.
|
||||
|
|
@ -1,996 +0,0 @@
|
|||
# Phase 1b Agent Routing Spec
|
||||
|
||||
Created: 2026-05-29
|
||||
Status: active draft
|
||||
Owner: Epimetheus pipeline implementation, with m3taversal as scope owner and Fwaz as VPS/runtime owner
|
||||
|
||||
## Product Outcome Contract
|
||||
|
||||
Phase 1b makes the knowledge-base evaluation engine behave like a six-agent review system instead of a generic triage stack.
|
||||
|
||||
When a contribution changes the `decision-engine` KB, the pipeline must decide which Hermes agent identity is responsible for judging that change, run the required review or reviews, post agent-specific verdicts, and then let the existing merge or feedback machinery continue.
|
||||
|
||||
The user-visible outcome is not a new frontend. It is a PR review trail showing that the right agent or agents reviewed the right KB mutation.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This spec does not implement:
|
||||
|
||||
- Twitter/X posting.
|
||||
- x402, wallet, payment, or funding flows.
|
||||
- Decision markets, agent bidding, stake-weighted quorum, or prediction-market review.
|
||||
- Full general user-input routing outside the PR evaluation path.
|
||||
- Separate GitHub accounts for each agent.
|
||||
- A full Forgejo-to-GitHub daemon rewrite beyond what Phase 1b needs.
|
||||
- A dashboard redesign.
|
||||
- Production deployment without staging or VPS proof.
|
||||
|
||||
## Program Decomposition
|
||||
|
||||
This is a medium-sized control-plane change with five execution lanes:
|
||||
|
||||
1. Agent identity routing.
|
||||
2. Eval pipeline integration.
|
||||
3. GitHub identity and bot comment posture.
|
||||
4. Reporting and contributor compatibility.
|
||||
5. Staging and production proof.
|
||||
|
||||
The implementation can remain in one PR only if lanes 1 through 4 are tightly tested and the staging proof remains a separate operator task. If the eval integration diff grows beyond the files named in this spec, split into:
|
||||
|
||||
- PR 1: route contract and tests.
|
||||
- PR 2: eval integration and mocked state tests.
|
||||
- PR 3: GitHub/comment idempotency and reporting compatibility.
|
||||
- PR 4 or operator runbook: staging proof artifacts.
|
||||
|
||||
Child specs:
|
||||
|
||||
- `docs/phase1b/agent-identity-router-spec.md`
|
||||
- `docs/phase1b/eval-pipeline-integration-spec.md`
|
||||
- `docs/phase1b/github-identity-bot-posture-spec.md`
|
||||
- `docs/phase1b/reporting-contributor-compatibility-spec.md`
|
||||
- `docs/phase1b/staging-proof-spec.md`
|
||||
|
||||
## Priority Matrix
|
||||
|
||||
| Rank | Workstream | Recurrence | Value | Readiness | Current state | Issue/spec mapping | Thread-claimed status | Verified implementation/proof status | Recommended next move |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| 1 | Canonical repo and eval target | Repeated confusion between `teleo-codex`, `teleo-kb`, and `decision-engine`. | Critical | Ready now | Confirmed by user: `decision-engine`. Some code still has Forgejo/teleo-codex defaults. | This spec, `handoff/phase1-step3-script-migration.md` | Clarified in chat. | Partially reflected in repo; not unified in daemon modules. | Make Phase 1b route/proof explicitly target `decision-engine`. |
|
||||
| 2 | Agent identity routing | Repeated confusion between domain folders and agent ownership. | Critical | Ready now | Existing `lib/domains.py` is folder-first. | This spec | m3taversal clarified identity-first routing. | Initial local patch is insufficient. | Replace with identity-scored route contract. |
|
||||
| 3 | Cross-domain review | Raised as scope expansion during clarification. | High | Ready now | Not implemented. | This spec | m3taversal confirmed cap at top 2. | No code proof. | Add top-2 required reviewer aggregation. |
|
||||
| 4 | Single master bot account | GitHub bot/PAT issue was noted as blocker. | High | Ready now | Phase 1 handoff already documents single `livingIPbot` posture. | `handoff/phase1-step3-script-migration.md` | Separate identities ideal, likely too complex. | Handoff-only. | Use master bot comments with agent verdict tags. |
|
||||
| 5 | Staging proof | User asked how to test without mutating prod VPS. | Critical for production | Draft gated | Needs VPS clone or Crabbox/staging access. | This spec | Proposed, not executed. | No proof. | Run after code PR passes local checks. |
|
||||
|
||||
## Goal
|
||||
|
||||
Implement Phase 1b for the `decision-engine` knowledge base: pipeline-v2 evaluates each incoming KB pull request by routing it to the Hermes agent identity that owns the relevant domain of judgment.
|
||||
|
||||
The implementation lives in `teleo-infrastructure`. The canonical KB repo for this phase is `living-ip/decision-engine`.
|
||||
|
||||
Phase 1b is complete only when single-domain and cross-domain PRs are routed to the expected required reviewer agents, verdicts are posted in the existing `VERDICT:AGENT:*` format, and the merge or feedback path continues from those verdicts.
|
||||
|
||||
## User-Journey Contract
|
||||
|
||||
Contributor or agent flow:
|
||||
|
||||
1. A contributor or agent opens a PR against `living-ip/decision-engine`.
|
||||
2. The PR changes one or more KB files.
|
||||
3. Pipeline-v2 discovers the PR and fetches its diff.
|
||||
4. The router scores Hermes agent identities from the diff, file paths, branch metadata, and eventually PR metadata.
|
||||
5. The pipeline runs the required reviewer agents.
|
||||
6. The master bot posts verdict comments that clearly name the agent identity in `VERDICT:AGENT:*` tags.
|
||||
7. If all required reviewers approve, the existing approval and merge path continues.
|
||||
8. If any required reviewer requests changes, the existing feedback/retry path continues.
|
||||
|
||||
Operator flow:
|
||||
|
||||
1. Operator can inspect a PR and see why each agent was selected.
|
||||
2. Operator can inspect pipeline logs or audit rows and see route scores, required agents, verdicts, and aggregate result.
|
||||
3. Operator can distinguish local proof, staging proof, and production proof.
|
||||
|
||||
## Existing-Spec Inventory
|
||||
|
||||
| Existing doc | Relevance | Decision | Reason |
|
||||
| --- | --- | --- | --- |
|
||||
| `handoff/phase1-step3-script-migration.md` | Establishes the Phase 1 move from Forgejo `teleo-codex` toward GitHub `living-ip/decision-engine`, and documents the single master bot account posture. | Reuse as context. | It owns migration history, not the Phase 1b routing implementation. |
|
||||
| `handoff/deprecated/eval-scripts.md` | Confirms old eval dispatcher/worker scripts are dead and `lib/evaluate.py::evaluate_cycle` owns live eval behavior. | Reuse as context. | It prevents work from targeting retired scripts. |
|
||||
| `docs/ARCHITECTURE.md` | Describes pipeline-v2 stages, SQLite state, Forgejo-era runtime topology, and existing evaluate/merge loops. | Reuse as context. | It is broader architecture; this spec is a Phase 1b delta spec. |
|
||||
| `docs/multi-model-eval-architecture.md` | Documents the prior Leo-first plus second-model evaluation theory. | Supersede for Phase 1b eval routing only. | Phase 1b now routes to domain-owner agent identities, with capped top-2 cross-domain review. The old doc remains useful for later calibration. |
|
||||
| `docs/queue.md` | Mentions domain evolution such as `ai-alignment` to `ai-systems`. | Reuse as signal. | It supports the identity-scored router rather than folder-only routing. |
|
||||
|
||||
## Current Implementation Audit
|
||||
|
||||
Current relevant implementation state:
|
||||
|
||||
- `teleo-pipeline.py` runs pipeline-v2 as a single async daemon.
|
||||
- `lib/evaluate.py::evaluate_cycle` is the active eval loop.
|
||||
- `lib/evaluate.py::evaluate_pr` currently detects a domain, runs a domain review, then runs Leo review for non-LIGHT PRs.
|
||||
- `lib/domains.py` contains a folder-first `DOMAIN_AGENT_MAP`.
|
||||
- `lib/llm.py` contains prompt templates and `run_domain_review`, `run_batch_domain_review`, and `run_leo_review`.
|
||||
- `lib/eval_parse.py::parse_verdict` parses `VERDICT:AGENT:APPROVE` and `VERDICT:AGENT:REQUEST_CHANGES`.
|
||||
- `pipeline-health-check.py` is GitHub-oriented and points at `living-ip/decision-engine`.
|
||||
- `lib/forgejo.py`, `lib/evaluate.py`, and `lib/merge.py` still use Forgejo-named abstractions as the primary API surface.
|
||||
- Per-agent GitHub identity is deferred; Phase 1 uses one master bot account.
|
||||
|
||||
Fwaz clarification on 2026-05-29:
|
||||
|
||||
- Separate GitHub identities are still ideal and blocked on GitHub/PAT setup; Phase 1b must not require them to land the routed-eval path.
|
||||
- Current production update behavior is `pull -> services recognize pull -> edit on VPS -> PR to Leo`; this is useful context, not the desired long-term control model.
|
||||
- New desired rule is no direct production self-upgrades: agents open PRs, and production deploys exact reviewed/tested SHAs approved and signed by Leo.
|
||||
- Crabbox is acceptable as the long-term disposable staging/test-box direction, while a production-like clone remains the highest-fidelity proof for systemd/VPS paths.
|
||||
|
||||
This branch implementation now includes:
|
||||
|
||||
- `lib/agent_routing.py` with a pure identity-scored route contract.
|
||||
- `PHASE1B_AGENT_ROUTING_ENABLED`, defaulting off.
|
||||
- A Phase 1b eval path that runs routed required agents and disables stale domain batching under the flag.
|
||||
- Focused tests for six-agent routing, top-2 cross-domain routing, verdict parsing, and mocked eval aggregation.
|
||||
|
||||
## Goal-Vs-Repo-Truth Diff
|
||||
|
||||
Desired Phase 1b behavior:
|
||||
|
||||
- Route PRs against `decision-engine`, not `teleo-codex`.
|
||||
- Classify by agent identity ownership, not only by folder path.
|
||||
- Run exactly the required reviewer agents.
|
||||
- Use one master bot account if separate GitHub identities are too complex.
|
||||
- Preserve the existing verdict comment format.
|
||||
- Preserve existing merge and feedback behavior.
|
||||
- Support cross-domain PRs by requiring the top 2 routed agents.
|
||||
|
||||
Pre-implementation repo truth:
|
||||
|
||||
- Pipeline eval still has a two-stage review shape: domain review plus Leo review.
|
||||
- Folder-domain mapping exists, but agent identity scoring does not.
|
||||
- Cross-domain review is not implemented as multiple required reviewer agents.
|
||||
- Batch eval can group rows before fetching diffs, which risks routing unclassified rows through `general`.
|
||||
- GitHub migration is partial: some scripts target GitHub `decision-engine`, but live daemon modules still have Forgejo-era names and assumptions.
|
||||
|
||||
## Completion Percent And Remaining Delta
|
||||
|
||||
Estimated implementation progress on this branch:
|
||||
|
||||
- B1 classifier foundation: 100 percent locally, pending staging calibration.
|
||||
- B2 routing layer: 75 percent locally behind a default-off feature flag.
|
||||
- Cross-domain top-2 review: 75 percent locally through mocked eval proof.
|
||||
- Local proof suite: 85 percent for router/eval/parser scope.
|
||||
- Staging or VPS proof: 0 percent.
|
||||
|
||||
Remaining delta:
|
||||
|
||||
1. Decide whether the production Phase 1b transport stays Forgejo-first for cutover or switches direct to GitHub `decision-engine` before staging.
|
||||
2. Update reporting/health compatibility beyond `review_records` if staging shows false readiness.
|
||||
3. Prove against staging before production.
|
||||
4. Deploy only an exact reviewed/tested SHA after Leo signoff.
|
||||
|
||||
## Closure, Endpoint, And Deployment Truth
|
||||
|
||||
Local closure means:
|
||||
|
||||
- Focused tests pass in `teleo-infrastructure`.
|
||||
- A PR exists with the Phase 1b routing implementation and proof notes.
|
||||
|
||||
Staging closure means:
|
||||
|
||||
- A cloned or disposable staging runtime is pointed at a sandbox `decision-engine`.
|
||||
- Six single-domain sandbox PRs and one cross-domain sandbox PR complete the expected eval path.
|
||||
- A machine-readable proof artifact captures routes, required agents, verdicts, status transitions, git SHAs, and logs.
|
||||
|
||||
Production closure means:
|
||||
|
||||
- The exact reviewed SHA is deployed to the production VPS.
|
||||
- Production pipeline runs real `decision-engine` PRs through Phase 1b routing.
|
||||
- All six agents have completed at least one live review cycle.
|
||||
- Pipeline remains stable for at least 24 hours after cutover.
|
||||
|
||||
Without VPS or staging access, only local closure can be claimed.
|
||||
|
||||
## Critical Assumptions And Invalidators
|
||||
|
||||
Assumptions:
|
||||
|
||||
- `decision-engine` is the canonical KB repo for Phase 1b.
|
||||
- The active eval implementation is `teleo-infrastructure/lib/evaluate.py`, not retired shell scripts.
|
||||
- One master bot account is acceptable for Phase 1b verdict comments.
|
||||
- Required reviewer identity is encoded in the verdict tag, not necessarily in the GitHub account identity.
|
||||
- Agent state files in `decision-engine/agents/{agent}` are the right identity context source when present.
|
||||
|
||||
Invalidators:
|
||||
|
||||
- Production pipeline is still wired to a different canonical repo.
|
||||
- The VPS runs code not represented by current `teleo-infrastructure`.
|
||||
- Branch protection requires separate GitHub identities before comments or reviews count.
|
||||
- Agent identity files are absent or materially different on the VPS.
|
||||
- Cross-domain review must include more than top 2 reviewers.
|
||||
|
||||
## State And Truth Contract
|
||||
|
||||
The routing implementation must record or expose:
|
||||
|
||||
- PR number.
|
||||
- Primary agent.
|
||||
- Required agents.
|
||||
- Route kind: `single`, `multi`, or `escalated`.
|
||||
- Route scores by agent.
|
||||
- Route evidence: path, branch, title, diff keyword, or fallback.
|
||||
- Verdict per required agent.
|
||||
- Aggregate result.
|
||||
- Failure reason for missing or unparseable verdicts.
|
||||
|
||||
This can be stored first in audit log details and test artifacts. A DB schema migration is optional for Phase 1b unless downstream dashboards require queryable route fields.
|
||||
|
||||
### Route Decision Schema
|
||||
|
||||
The route decision should be serializable without importing Python classes. Use this JSON shape in audit rows and proof artifacts:
|
||||
|
||||
```json
|
||||
{
|
||||
"pr": 123,
|
||||
"repo": "living-ip/decision-engine",
|
||||
"route_version": "phase1b-v1",
|
||||
"route_kind": "single",
|
||||
"primary_agent": "Rio",
|
||||
"required_agents": ["Rio"],
|
||||
"scores": {
|
||||
"Leo": 0,
|
||||
"Theseus": 1,
|
||||
"Rio": 9,
|
||||
"Vida": 0,
|
||||
"Clay": 0,
|
||||
"Astra": 0
|
||||
},
|
||||
"evidence": [
|
||||
{
|
||||
"agent": "Rio",
|
||||
"signal": "path",
|
||||
"weight": 5,
|
||||
"value": "domains/internet-finance/example.md"
|
||||
}
|
||||
],
|
||||
"fallback": false
|
||||
}
|
||||
```
|
||||
|
||||
`route_kind` values:
|
||||
|
||||
- `single`: one required reviewer.
|
||||
- `multi`: two required reviewers from cross-domain scoring.
|
||||
- `fallback`: no confident route, Leo required.
|
||||
- `escalated`: route exceeded simple review bounds and was capped by policy.
|
||||
|
||||
### Verdict State Schema
|
||||
|
||||
Aggregate review state should be serializable as:
|
||||
|
||||
```json
|
||||
{
|
||||
"pr": 123,
|
||||
"required_agents": ["Theseus", "Rio"],
|
||||
"agent_verdicts": {
|
||||
"Theseus": "approve",
|
||||
"Rio": "request_changes"
|
||||
},
|
||||
"aggregate_verdict": "request_changes",
|
||||
"blocking_agents": ["Rio"],
|
||||
"missing_agents": [],
|
||||
"unparseable_agents": [],
|
||||
"transport_failed_agents": []
|
||||
}
|
||||
```
|
||||
|
||||
Aggregate states:
|
||||
|
||||
- `approve`: all required agents approved.
|
||||
- `request_changes`: at least one required agent requested changes or produced unparseable content.
|
||||
- `retry`: at least one required review failed for transport reasons and should not burn the PR as a substantive rejection.
|
||||
|
||||
## Measurement Contract
|
||||
|
||||
Minimum metrics:
|
||||
|
||||
- `route_single_count`
|
||||
- `route_multi_count`
|
||||
- `route_escalated_count`
|
||||
- `review_required_agent_count`
|
||||
- `review_missing_verdict_count`
|
||||
- `review_request_changes_count`
|
||||
- `review_approve_count`
|
||||
- `route_fallback_count`
|
||||
|
||||
Minimum proof matrix:
|
||||
|
||||
| Case | Expected route |
|
||||
| --- | --- |
|
||||
| grand strategy PR | Leo |
|
||||
| ai systems or ai alignment PR | Theseus |
|
||||
| internet finance or x402 PR | Rio |
|
||||
| health PR | Vida |
|
||||
| entertainment PR | Clay |
|
||||
| space, robotics, energy, or advanced manufacturing PR | Astra |
|
||||
| ai plus x402 PR | Theseus and Rio |
|
||||
| collective ai goals PR | Leo and Theseus, if both score in top 2 |
|
||||
|
||||
## Score-To-100 Closure Plan
|
||||
|
||||
Preparedness score before implementation: 35/100.
|
||||
|
||||
| Score band | Closure move | Evidence that moves score |
|
||||
| --- | --- | --- |
|
||||
| 35 -> 50 | Route contract implemented and unit-tested. | `test_agent_routing.py` proves six single-agent routes, broadened identity ownership, top-2 cross-domain routes, and fallback behavior. |
|
||||
| 50 -> 65 | Eval integration mocked locally. | Mocked eval tests prove required agents are invoked, default Leo review is removed, and aggregate verdicts drive approve/request-changes behavior. |
|
||||
| 65 -> 75 | API/comment compatibility proven locally. | Tests prove all six verdict tags parse and master-bot comment bodies preserve existing parser expectations. |
|
||||
| 75 -> 85 | Staging clone or disposable test box runs sandbox PR proof. | Six single-domain sandbox PRs plus one cross-domain sandbox PR produce expected comments and state transitions. |
|
||||
| 85 -> 95 | Production deploy of exact reviewed SHA. | VPS deploy log, service restart readback, and route/proof artifact for first real PRs. |
|
||||
| 95 -> 100 | 24-hour production stability. | 24-hour daemon readback with no duplicate comments, no stuck review rows, no production fallback spike, and all six agents represented in verdict history. |
|
||||
|
||||
The implementation PR can be merged at 65-75 if reviewers accept staging as a deploy gate. It cannot claim Phase 1b complete below 100.
|
||||
|
||||
## Backend Work Required
|
||||
|
||||
### 1. Agent identity router
|
||||
|
||||
Create or refactor into `lib/agent_routing.py` unless the existing `lib/domains.py` remains clearly small enough.
|
||||
|
||||
Define:
|
||||
|
||||
```python
|
||||
AgentRoute(
|
||||
primary_agent: str,
|
||||
required_agents: tuple[str, ...],
|
||||
route_kind: str,
|
||||
scores: dict[str, int],
|
||||
evidence: list[dict],
|
||||
)
|
||||
```
|
||||
|
||||
Router signals:
|
||||
|
||||
- Path signals from `domains/`, `entities/`, `core/`, `foundations/`, and `agents/`.
|
||||
- Branch prefix signals such as `rio/`, `theseus/`, `astra/`, `leo/`.
|
||||
- Keyword signals from path, filename, branch, PR title/body when available, and capped diff text.
|
||||
- Agent identity ownership map.
|
||||
|
||||
Agent identity ownership map:
|
||||
|
||||
| Agent | Owns |
|
||||
| --- | --- |
|
||||
| Leo | grand strategy, teleohumanity goals, collective AI self-understanding, meta strategy, nested collective intelligence concepts |
|
||||
| Theseus | AI systems, AI alignment, AI governance, agent systems, safety, evaluation |
|
||||
| Rio | internet finance, living capital, markets, crypto, futarchy, x402, payments, capital formation |
|
||||
| Vida | health, healthcare, medicine, prevention, clinical systems, mental health, biohealth |
|
||||
| Clay | entertainment, media, culture, IP, fandom, narrative, consumer attention |
|
||||
| Astra | space development, robotics, energy, advanced manufacturing, physical frontier infrastructure |
|
||||
|
||||
Routing rules:
|
||||
|
||||
- If only one agent crosses the threshold, require that agent.
|
||||
- If more than one agent crosses the threshold, require the top 2 agents.
|
||||
- If no agent crosses threshold, fallback to Leo with route kind `fallback`.
|
||||
- Tie break by score, then deterministic configured order.
|
||||
|
||||
Implementation constraints:
|
||||
|
||||
- The router must be deterministic.
|
||||
- The router must be pure and side-effect free.
|
||||
- Route scores must be explainable through evidence entries.
|
||||
- Folder paths should be strong evidence, not the whole classifier.
|
||||
- Keyword scoring must not require paid inference.
|
||||
- LLM classification may be added later only as shadow-mode evidence.
|
||||
|
||||
Recommended scoring starter:
|
||||
|
||||
| Signal | Weight |
|
||||
| --- | --- |
|
||||
| Path directly under known primary ownership area | 8 |
|
||||
| Path under broadened ownership area | 6 |
|
||||
| Branch prefix matches agent | 4 |
|
||||
| Filename keyword matches ownership | 3 |
|
||||
| Diff keyword matches ownership | 1 per capped hit |
|
||||
| PR title/body keyword matches ownership, if available | 2 |
|
||||
|
||||
Top-2 selection:
|
||||
|
||||
- Include the highest-scoring agent.
|
||||
- Include a second agent only if its score is at least 40 percent of the first score and at least the minimum threshold.
|
||||
- Minimum threshold starts at 4.
|
||||
- Never include more than two required agents in Phase 1b.
|
||||
|
||||
### 2. Eval layer integration
|
||||
|
||||
Modify `lib/evaluate.py`:
|
||||
|
||||
- Fetch PR diff.
|
||||
- Build route from diff and branch.
|
||||
- Store or audit route decision.
|
||||
- Run required reviewer agents.
|
||||
- Aggregate verdicts.
|
||||
- Remove default Leo second-review for normal single-agent PRs.
|
||||
- Keep existing bypasses for musings and reweave unless m3taversal changes policy.
|
||||
- Revisit batch eval: disable batching for Phase 1b or classify before batching.
|
||||
|
||||
Implementation sequence:
|
||||
|
||||
1. Add pure route builder and tests.
|
||||
2. Add review aggregation helper and tests.
|
||||
3. Add `run_agent_review` while leaving existing `run_domain_review` and `run_leo_review` intact.
|
||||
4. Switch individual `evaluate_pr` path to the new router behind a feature flag such as `PHASE1B_AGENT_ROUTING_ENABLED`.
|
||||
5. Disable batch domain eval when the feature flag is enabled unless route-aware batching is implemented in the same PR.
|
||||
6. Remove or bypass the default Leo second-review when the feature flag is enabled.
|
||||
7. Preserve old behavior when the feature flag is disabled.
|
||||
|
||||
Feature flag requirement:
|
||||
|
||||
```text
|
||||
PHASE1B_AGENT_ROUTING_ENABLED=false by default until staging proof exists.
|
||||
```
|
||||
|
||||
The PR may set tests against enabled behavior without changing the production default.
|
||||
|
||||
### 3. Agent review runner
|
||||
|
||||
Modify or add in `lib/llm.py`:
|
||||
|
||||
```python
|
||||
async def run_agent_review(diff: str, files: str, agent: str, route: AgentRoute) -> tuple[str | None, dict]:
|
||||
...
|
||||
```
|
||||
|
||||
Prompt must include:
|
||||
|
||||
- Agent identity context when available.
|
||||
- Route evidence.
|
||||
- Existing eval criteria.
|
||||
- Required verdict tag for that exact agent.
|
||||
|
||||
Continue using one master bot account for comments. The bot comment body must identify the routed agent via the verdict tag.
|
||||
|
||||
Agent context lookup order:
|
||||
|
||||
1. Runtime-configured KB worktree path, expected to point at `decision-engine`.
|
||||
2. Existing `config.MAIN_WORKTREE` if production still uses that convention.
|
||||
3. Explicit test fixture path in unit tests.
|
||||
|
||||
Context files:
|
||||
|
||||
- `agents/{agent}/identity.md`
|
||||
- `agents/{agent}/beliefs.md`
|
||||
- `agents/{agent}/reasoning.md`
|
||||
- `agents/{agent}/skills.md`
|
||||
|
||||
Missing context files:
|
||||
|
||||
- Log a warning.
|
||||
- Include an audit evidence entry.
|
||||
- Continue with the generic agent prompt.
|
||||
- Do not crash the eval cycle.
|
||||
|
||||
### 4. Verdict aggregation
|
||||
|
||||
Add helper:
|
||||
|
||||
```python
|
||||
aggregate_agent_verdicts(required_agents, reviews) -> AggregateVerdict
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- All required agents approve: approved.
|
||||
- Any required agent requests changes: request changes.
|
||||
- Transport failure: reopen for retry.
|
||||
- Missing or unparseable verdict: request changes unless transport failure is explicit.
|
||||
|
||||
Comment format:
|
||||
|
||||
Preferred for one required agent:
|
||||
|
||||
```text
|
||||
<review text>
|
||||
|
||||
<!-- VERDICT:RIO:APPROVE -->
|
||||
```
|
||||
|
||||
Preferred for two required agents:
|
||||
|
||||
```text
|
||||
## Theseus review
|
||||
|
||||
<review text>
|
||||
|
||||
<!-- VERDICT:THESEUS:APPROVE -->
|
||||
|
||||
## Rio review
|
||||
|
||||
<review text>
|
||||
|
||||
<!-- VERDICT:RIO:REQUEST_CHANGES -->
|
||||
```
|
||||
|
||||
Two separate comments are acceptable if simpler and less risky for existing parsers.
|
||||
|
||||
### 5. Contributor and dashboard compatibility
|
||||
|
||||
Audit and update:
|
||||
|
||||
- `lib/contributor.py` assumptions that Leo reviews every PR.
|
||||
- `pipeline-health-check.py` verdict parsing if needed.
|
||||
- Any dashboard code assuming only `leo_verdict` plus `domain_verdict`.
|
||||
|
||||
Avoid broad dashboard redesign in Phase 1b. If dashboards need richer route state, add an audit artifact first and defer UI.
|
||||
|
||||
## Frontend Work Required
|
||||
|
||||
No frontend work is required for Phase 1b.
|
||||
|
||||
`livingip-web` Phase 1c can later reuse the same router as pre-PR guidance, but Phase 1b acceptance is based on `decision-engine` PR evaluation.
|
||||
|
||||
## Operator Work Required
|
||||
|
||||
Operator or infrastructure owner must provide before production proof:
|
||||
|
||||
- Current production deployed SHA for `teleo-infrastructure`.
|
||||
- Current production KB target and worktree path.
|
||||
- Current systemd units and restart commands.
|
||||
- Staging clone or disposable test runner access.
|
||||
- Sandbox `decision-engine` target or clear permission to create one.
|
||||
- Staging token set with no production mutation authority.
|
||||
- Rollback SHA and rollback command.
|
||||
|
||||
If these are unavailable, implementation can continue locally but production proof must remain blocked.
|
||||
|
||||
## Expected Runtime And User-Visible Behavior
|
||||
|
||||
Single-domain PR:
|
||||
|
||||
1. Pipeline detects route.
|
||||
2. Required agents has one name.
|
||||
3. Master bot posts one review comment with `VERDICT:AGENT:*`.
|
||||
4. Existing merge or feedback path continues.
|
||||
|
||||
Cross-domain PR:
|
||||
|
||||
1. Pipeline detects route.
|
||||
2. Required agents has two names.
|
||||
3. Master bot posts one review comment per required agent, or one structured comment with separate verdict sections if that is simpler.
|
||||
4. Merge requires both approvals.
|
||||
5. Any request changes blocks and feeds back.
|
||||
|
||||
The user-visible proof is PR comments and final PR disposition.
|
||||
|
||||
## Staging Proof Contract
|
||||
|
||||
Staging must be production-like enough to test pipeline behavior but quarantined from production side effects.
|
||||
|
||||
Required staging safety controls:
|
||||
|
||||
- Production services disabled before any daemon starts.
|
||||
- Production GitHub tokens removed or replaced.
|
||||
- Production OpenRouter/Claude/Hermes keys removed or replaced unless explicitly approved for staging spend.
|
||||
- Sandbox `decision-engine` repo configured.
|
||||
- Auto-merge either disabled or constrained to sandbox repo.
|
||||
- Hostname clearly changed to staging.
|
||||
|
||||
Required proof artifact:
|
||||
|
||||
```json
|
||||
{
|
||||
"phase": "1b",
|
||||
"environment": "staging",
|
||||
"teleo_infrastructure_sha": "...",
|
||||
"decision_engine_sha": "...",
|
||||
"pipeline_db_schema": 26,
|
||||
"feature_flags": {
|
||||
"PHASE1B_AGENT_ROUTING_ENABLED": "true"
|
||||
},
|
||||
"test_prs": [
|
||||
{
|
||||
"case": "internet-finance",
|
||||
"pr": 1,
|
||||
"required_agents": ["Rio"],
|
||||
"verdicts": {"Rio": "approve"},
|
||||
"final_state": "approved"
|
||||
}
|
||||
],
|
||||
"cross_domain_pr": {
|
||||
"required_agents": ["Theseus", "Rio"],
|
||||
"final_state": "approved_or_feedback"
|
||||
},
|
||||
"prod_services_disabled": true,
|
||||
"proof_generated_at": "2026-05-29T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
Staging proof does not satisfy the 24-hour production stability gate.
|
||||
|
||||
## Validation And Test Matrix
|
||||
|
||||
Unit tests:
|
||||
|
||||
- `test_agent_routing.py`
|
||||
- routes six primary ownership cases.
|
||||
- routes broadened Astra cases: energy, robotics, advanced manufacturing.
|
||||
- routes Leo meta cases: collective AI goals, teleohumanity strategy.
|
||||
- routes Theseus AI systems cases.
|
||||
- routes Rio x402 and internet finance cases.
|
||||
- caps cross-domain to top 2 agents.
|
||||
- has deterministic tie breaking.
|
||||
|
||||
Parser tests:
|
||||
|
||||
- Existing `test_eval_parse.py` remains valid.
|
||||
- Add explicit verdict parse coverage for all six agent names.
|
||||
|
||||
Mocked eval integration tests:
|
||||
|
||||
- One required agent calls one runner and posts one verdict.
|
||||
- Two required agents call two runners and post two verdicts.
|
||||
- One request changes blocks aggregate approval.
|
||||
- Transport failure reopens for retry.
|
||||
- Default Leo second-review does not run unless Leo is routed.
|
||||
|
||||
Batch tests:
|
||||
|
||||
- If batching remains enabled, batch grouping must use route decisions, not stale DB domain.
|
||||
- If batching is disabled for Phase 1b, assert cross-domain and single-domain PRs still process individually.
|
||||
|
||||
Smoke commands:
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
. .venv/bin/activate
|
||||
python3 -m pip install 'aiohttp>=3.9,<4' 'pytest>=8' 'pytest-asyncio>=0.23' 'ruff>=0.3' pyyaml
|
||||
python3 -m pytest tests/test_agent_routing.py tests/test_evaluate_agent_routing.py tests/test_eval_parse.py
|
||||
```
|
||||
|
||||
If local `pytest` is unavailable, that is a tooling blocker for full local proof, not an implementation blocker.
|
||||
|
||||
## CI/CD, Release, And Pre-Push Gate Contract
|
||||
|
||||
Pre-push required:
|
||||
|
||||
- `python3 -m pytest` for the focused routing/eval test set.
|
||||
- `python3 -m ruff check lib tests` if dev deps are installed.
|
||||
- Manual scan that no secrets are printed or committed.
|
||||
|
||||
PR required:
|
||||
|
||||
- Summary of routing rule.
|
||||
- Test output.
|
||||
- Known non-prod proof boundary.
|
||||
- Statement that production acceptance still requires staging or VPS proof.
|
||||
|
||||
Deploy required:
|
||||
|
||||
- Exact reviewed SHA.
|
||||
- Staging proof bundle first.
|
||||
- Production service restart plan.
|
||||
- Rollback SHA.
|
||||
|
||||
Release phases:
|
||||
|
||||
| Phase | Feature flag | Environment | Required proof |
|
||||
| --- | --- | --- | --- |
|
||||
| Local implementation | Enabled only in tests | Local | Unit and mocked eval tests. |
|
||||
| Staging shadow | Enabled against sandbox repo | Staging clone or Crabbox-like box | Seven sandbox PR proof artifact. |
|
||||
| Production shadow | Optional, no merge mutation if supported | Production | Route decisions logged without changing verdict path. |
|
||||
| Production cutover | Enabled | Production | Real PR verdicts by required agents. |
|
||||
| Production closure | Enabled | Production | 24-hour stability plus all six agents represented. |
|
||||
|
||||
Rollback:
|
||||
|
||||
- Flip `PHASE1B_AGENT_ROUTING_ENABLED=false`.
|
||||
- Restart `teleo-pipeline.service`.
|
||||
- Confirm eval path returns to prior behavior.
|
||||
- If code rollback is required, deploy the previous exact SHA and restart service.
|
||||
- Keep proof artifact explaining why rollback occurred.
|
||||
|
||||
Pre-push commands:
|
||||
|
||||
```bash
|
||||
python3 -m pytest tests/test_agent_routing.py tests/test_evaluate_agent_routing.py tests/test_eval_parse.py
|
||||
python3 -m ruff check lib tests
|
||||
git diff --check
|
||||
```
|
||||
|
||||
If dev dependencies are missing, install with:
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
. .venv/bin/activate
|
||||
python3 -m pip install 'aiohttp>=3.9,<4' 'pytest>=8' 'pytest-asyncio>=0.23' 'ruff>=0.3' pyyaml
|
||||
```
|
||||
|
||||
## Independent CLI Audit Contract
|
||||
|
||||
A reviewer should be able to run:
|
||||
|
||||
```bash
|
||||
git diff --stat
|
||||
git diff -- lib/agent_routing.py lib/domains.py lib/evaluate.py lib/llm.py tests/
|
||||
python3 -m pytest tests/test_agent_routing.py tests/test_evaluate_agent_routing.py
|
||||
```
|
||||
|
||||
The audit should confirm:
|
||||
|
||||
- No direct production credentials are introduced.
|
||||
- `decision-engine` is the target in docs/config where Phase 1b needs it.
|
||||
- No old eval scripts are revived.
|
||||
- Default Leo second-review is not silently preserved for all PRs.
|
||||
- Multi-agent PRs require top 2 reviewer approvals.
|
||||
|
||||
## Outside-The-Box Fix Paths
|
||||
|
||||
If identity-scored keyword routing is too noisy:
|
||||
|
||||
- Use folder-first routing for strong path evidence and identity scoring only for ambiguous or cross-domain cases.
|
||||
- Add a cheap LLM classifier in shadow mode only, comparing against deterministic router decisions.
|
||||
- Require contributors/frontends to include an explicit domain or agent hint in PR metadata.
|
||||
|
||||
If live GitHub identity constraints block separate agent comments:
|
||||
|
||||
- Keep one master bot account and agent-specific verdict tags.
|
||||
- Defer separate GitHub identities to Phase 2.
|
||||
|
||||
If staging VPS access is delayed:
|
||||
|
||||
- Use a disposable Hetzner clone when available.
|
||||
- Use Crabbox or another remote test box for local dirty checkout proof.
|
||||
- Use a mocked local fake GitHub/Forgejo API server for the eval loop.
|
||||
|
||||
## Maintenance Capture
|
||||
|
||||
Same-tranche maintenance that is justified now:
|
||||
|
||||
- Extract route scoring into a dedicated module if `lib/domains.py` would become too broad.
|
||||
- Keep backward-compatible wrappers for existing `agent_for_domain` and `detect_domain_from_diff` until downstream callers are migrated.
|
||||
- Add tests around the existing bug-prone batch grouping surface.
|
||||
|
||||
Maintenance to avoid now:
|
||||
|
||||
- Full Forgejo-to-GitHub daemon rewrite unless needed for the Phase 1b PR.
|
||||
- Dashboard redesign.
|
||||
- Contributor credit redesign beyond removing "Leo reviews every PR" assumptions.
|
||||
- Separate GitHub identities per agent.
|
||||
- Payment, wallet, Twitter, or decision-market work.
|
||||
|
||||
## Parallelization And Fanout
|
||||
|
||||
| Workstream | Classification | Owner | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| Agent identity router and tests | local_owner | Codex current turn | Core implementation surface. Do not fan out because it owns central route contract. |
|
||||
| Eval layer integration and mocked tests | local_owner | Codex current turn | Needs tight coupling with router semantics. |
|
||||
| Staging VPS clone proof | draft_gated | Fwaz or infrastructure owner | Requires VPS/provider access and secret quarantine. |
|
||||
| GitHub identity model | draft_gated | Fwaz plus m3taversal | Deferred unless master bot account becomes unacceptable. |
|
||||
| Dashboard/reporting polish | do_not_parallelize | Later | Avoid until route state contract is stable. |
|
||||
|
||||
### Workstream Sub-Spec: Agent Identity Router
|
||||
|
||||
Classification: local_owner
|
||||
|
||||
Owned files:
|
||||
|
||||
- `lib/agent_routing.py` if created.
|
||||
- `lib/domains.py` compatibility wrappers.
|
||||
- `tests/test_agent_routing.py`.
|
||||
|
||||
Forbidden files:
|
||||
|
||||
- `lib/evaluate.py` except imports needed for route type compatibility.
|
||||
- Any runtime secrets.
|
||||
- Any production config defaults outside route feature flags.
|
||||
|
||||
Binary done condition:
|
||||
|
||||
- Pure route function returns expected required agents for every row in the proof matrix.
|
||||
- Tests prove deterministic top-2 behavior and fallback behavior.
|
||||
|
||||
Verification commands:
|
||||
|
||||
```bash
|
||||
python3 -m pytest tests/test_agent_routing.py
|
||||
```
|
||||
|
||||
Non-claims:
|
||||
|
||||
- Does not prove PR comment posting.
|
||||
- Does not prove production target wiring.
|
||||
|
||||
Prompt-ready handoff:
|
||||
|
||||
```text
|
||||
implement phase 1b agent identity routing in teleo-infrastructure. own only route module and route tests. preserve compatibility wrappers. route decision must be pure, deterministic, evidence-bearing, and top-2 capped for cross-domain cases. do not touch production API or eval state transitions.
|
||||
```
|
||||
|
||||
### Workstream Sub-Spec: Eval Integration
|
||||
|
||||
Classification: local_owner
|
||||
|
||||
Owned files:
|
||||
|
||||
- `lib/evaluate.py`
|
||||
- `lib/llm.py`
|
||||
- `lib/eval_parse.py` only if parser normalization is required.
|
||||
- `tests/test_evaluate_agent_routing.py`
|
||||
- `tests/test_eval_parse.py`
|
||||
|
||||
Forbidden files:
|
||||
|
||||
- Old deprecated eval shell scripts.
|
||||
- Deploy scripts unless a feature flag must be exposed.
|
||||
- Dashboard UI except parser-compatible health checks.
|
||||
|
||||
Binary done condition:
|
||||
|
||||
- With `PHASE1B_AGENT_ROUTING_ENABLED=true`, eval invokes only required reviewer agents.
|
||||
- With flag disabled, prior behavior remains available.
|
||||
- One request-changes verdict blocks aggregate approval.
|
||||
- All approve verdicts continue to existing approval path.
|
||||
|
||||
Verification commands:
|
||||
|
||||
```bash
|
||||
python3 -m pytest tests/test_evaluate_agent_routing.py tests/test_eval_parse.py
|
||||
```
|
||||
|
||||
Non-claims:
|
||||
|
||||
- Does not prove live GitHub or VPS behavior.
|
||||
- Does not prove separate agent GitHub identities.
|
||||
|
||||
Prompt-ready handoff:
|
||||
|
||||
```text
|
||||
wire phase 1b routing into teleo-infrastructure eval path behind a feature flag. use required agents from the route result, run agent-specific reviews, aggregate verdicts, and preserve merge/feedback semantics. do not revive deprecated scripts or remove rollback path.
|
||||
```
|
||||
|
||||
### Workstream Sub-Spec: Staging Proof
|
||||
|
||||
Classification: draft_gated
|
||||
|
||||
Owned files and surfaces:
|
||||
|
||||
- Staging VPS or disposable remote test box.
|
||||
- Sandbox `decision-engine` repo.
|
||||
- Staging secrets.
|
||||
- Machine-readable proof artifact.
|
||||
|
||||
Forbidden files and surfaces:
|
||||
|
||||
- Production VPS services.
|
||||
- Production GitHub repo.
|
||||
- Production secrets.
|
||||
- Mainnet/payment/Twitter surfaces.
|
||||
|
||||
Binary done condition:
|
||||
|
||||
- Six single-domain PRs and one cross-domain PR produce expected required-agent verdicts and final dispositions in staging.
|
||||
|
||||
Verification commands:
|
||||
|
||||
```bash
|
||||
systemctl status teleo-pipeline
|
||||
journalctl -u teleo-pipeline --since "1 hour ago"
|
||||
sqlite3 /path/to/pipeline.db "select number, status, domain_agent, leo_verdict, domain_verdict from prs order by number desc limit 20;"
|
||||
gh pr view --repo living-ip/decision-engine-sandbox PR_NUMBER --comments
|
||||
```
|
||||
|
||||
Non-claims:
|
||||
|
||||
- Does not prove production 24-hour stability.
|
||||
|
||||
Prompt-ready handoff:
|
||||
|
||||
```text
|
||||
create a quarantined staging proof for phase 1b. clone or provision a disposable server, disable production services and secrets before starting pipeline, point to a sandbox decision-engine repo, run six single-domain prs plus one cross-domain pr, and save a machine-readable proof artifact. do not mutate production.
|
||||
```
|
||||
|
||||
Worker-ready ticket for later staging proof:
|
||||
|
||||
```text
|
||||
title: phase 1b staging proof on cloned vps
|
||||
owned surfaces: staging vps, sandbox decision-engine repo, staging secrets, proof artifact
|
||||
forbidden surfaces: production vps services, production github repo, production secrets
|
||||
done condition: six single-domain prs plus one cross-domain pr produce expected required-agent verdicts and final dispositions
|
||||
verification commands: systemd status readback, pipeline log scrape, sqlite route query, github pr comment readback
|
||||
non-claims: does not prove 24h production stability
|
||||
preferred executor: human/fwaz with codex support
|
||||
handoff: create staging clone, disable prod services, inject sandbox config, run phase 1b proof script, save machine-readable proof
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
Local PR acceptance:
|
||||
|
||||
- Focused tests pass.
|
||||
- Router returns correct single-agent routes.
|
||||
- Router returns top-2 required agents for cross-domain cases.
|
||||
- Eval layer invokes only required reviewer agents.
|
||||
- Verdict aggregation handles all approve, request changes, transport failure, and missing verdict.
|
||||
- Existing verdict format remains parseable.
|
||||
- No production readiness claim is made.
|
||||
|
||||
Staging acceptance:
|
||||
|
||||
- Staging environment cannot mutate production.
|
||||
- Six single-domain sandbox PRs complete.
|
||||
- One cross-domain sandbox PR completes.
|
||||
- Required reviewer agents match proof matrix.
|
||||
- Proof artifact is retained.
|
||||
|
||||
Production exit:
|
||||
|
||||
- Exact reviewed SHA deployed.
|
||||
- All six agents produce at least one verdict in their domain.
|
||||
- At least one cross-domain PR proves top-2 review behavior.
|
||||
- Pipeline stable for 24 hours.
|
||||
|
||||
## Readiness And Claim Boundaries
|
||||
|
||||
Allowed claims after local implementation:
|
||||
|
||||
- "Route logic is implemented and locally tested."
|
||||
- "Mocked eval integration proves required-agent invocation and aggregation."
|
||||
- "The implementation PR is ready for staging proof."
|
||||
|
||||
Forbidden claims after local implementation:
|
||||
|
||||
- "Phase 1b is complete."
|
||||
- "Production is ready."
|
||||
- "All six agents have demonstrated live review cycles."
|
||||
- "The VPS is safely updated."
|
||||
|
||||
Allowed claims after staging proof:
|
||||
|
||||
- "Phase 1b passed sandbox staging proof."
|
||||
- "The exact SHA is eligible for production cutover review."
|
||||
|
||||
Forbidden claims after staging proof:
|
||||
|
||||
- "Production is stable."
|
||||
- "Live `decision-engine` PRs are proven."
|
||||
|
||||
Allowed claims after production 24-hour proof:
|
||||
|
||||
- "Phase 1b production exit criteria are met."
|
||||
|
||||
## Spec Quality Self-Audit
|
||||
|
||||
Required execution-grade headings present:
|
||||
|
||||
- Current Implementation Audit: present.
|
||||
- Goal-Vs-Repo-Truth Diff: present.
|
||||
- Completion Percent And Remaining Delta: present.
|
||||
- Closure, Endpoint, And Deployment Truth: present.
|
||||
- Critical Assumptions And Invalidators: present.
|
||||
- State And Truth Contract: present.
|
||||
- Measurement Contract: present.
|
||||
- Backend Work Required: present.
|
||||
- Frontend Work Required: present.
|
||||
- Expected Runtime And User-Visible Behavior: present.
|
||||
- Validation And Test Matrix: present.
|
||||
- CI/CD, Release, And Pre-Push Gate Contract: present.
|
||||
- Independent CLI Audit Contract: present.
|
||||
- Outside-The-Box Fix Paths: present.
|
||||
- Maintenance Capture: present.
|
||||
- Parallelization And Fanout: present.
|
||||
|
||||
Additional spec-of-spec coverage:
|
||||
|
||||
- Product Outcome Contract: present.
|
||||
- Non-Goals: present.
|
||||
- Program Decomposition: present.
|
||||
- Priority Matrix: present.
|
||||
- Score-To-100 Closure Plan: present.
|
||||
- Workstream sub-specs: present.
|
||||
- Staging Proof Contract: present.
|
||||
- Rollback contract: present.
|
||||
|
||||
Known incompleteness:
|
||||
|
||||
- This spec cannot name the exact production deploy command until Fwaz or VPS truth confirms it.
|
||||
- This spec cannot name the exact sandbox repo until the operator creates or selects it.
|
||||
- This spec cannot prove whether production daemon code exactly matches local `teleo-infrastructure` until VPS readback exists.
|
||||
|
||||
## Assistant-Added Caveats
|
||||
|
||||
This spec intentionally expands B1/B2 from folder-domain routing to identity-scored agent routing because m3taversal clarified that agent identities should route and folders are only signals. That is the right product interpretation, but it increases implementation scope versus the original simple path classifier.
|
||||
|
||||
This spec does not claim production readiness without staging or VPS proof.
|
||||
|
|
@ -1,31 +0,0 @@
|
|||
# Phase 1b Spec Index
|
||||
|
||||
Status: active draft
|
||||
Parent spec: `docs/phase1b-agent-routing-spec.md`
|
||||
|
||||
## Scope
|
||||
|
||||
Phase 1b is the `decision-engine` PR evaluation router. It sends each KB mutation to the owning Hermes agent identity, supports top-2 cross-domain review, posts parseable `VERDICT:AGENT:*` comments through one master bot account, preserves existing merge or feedback behavior, and proves the change in staging before production cutover.
|
||||
|
||||
## Specs
|
||||
|
||||
| Workstream | Spec | Implementation posture |
|
||||
| --- | --- | --- |
|
||||
| Agent identity router | `docs/phase1b/agent-identity-router-spec.md` | ready_now |
|
||||
| Eval pipeline integration | `docs/phase1b/eval-pipeline-integration-spec.md` | ready_now after router contract freezes |
|
||||
| GitHub identity and bot comments | `docs/phase1b/github-identity-bot-posture-spec.md` | ready_now after canonical target config freezes |
|
||||
| Reporting and contributor compatibility | `docs/phase1b/reporting-contributor-compatibility-spec.md` | ready_now after verdict state shape freezes |
|
||||
| Staging proof | `docs/phase1b/staging-proof-spec.md` | draft_gated on staging/VPS or disposable remote access |
|
||||
| Staging blocker | `docs/phase1b/staging-blocker.json` | external_only |
|
||||
|
||||
## Execution Order
|
||||
|
||||
1. Implement router contract and tests.
|
||||
2. Wire eval pipeline to required reviewer agents under a feature flag.
|
||||
3. Route comments through the canonical GitHub target with idempotency markers.
|
||||
4. Update reporting and contributor accounting to read reviewer sets rather than fixed Leo plus domain slots.
|
||||
5. Run staging proof on a clone or disposable remote target before production cutover.
|
||||
|
||||
## Claim Boundary
|
||||
|
||||
These specs plus the Phase 1b branch prove only local implementation behavior. A production completion claim requires merged code, passing tests, staging proof, exact production SHA deployment, Leo signoff, and 24-hour production daemon stability.
|
||||
|
|
@ -1,338 +0,0 @@
|
|||
# Phase 1b Child Spec: Agent Identity Router
|
||||
|
||||
Created: 2026-05-29
|
||||
Status: active draft
|
||||
Parent spec: `docs/phase1b-agent-routing-spec.md`
|
||||
|
||||
## Product Outcome Contract
|
||||
|
||||
The router decides which Hermes agent identity should review a `decision-engine` KB PR. It must route by agent ownership, with file paths as strong evidence but not the only source of truth.
|
||||
|
||||
## Goal
|
||||
|
||||
Implement a pure, deterministic, evidence-bearing route scorer that returns one or two required reviewer agents for a PR.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not call paid LLMs for routing.
|
||||
- Do not post PR comments.
|
||||
- Do not mutate pipeline DB state.
|
||||
- Do not deploy to VPS.
|
||||
- Do not implement general user-input routing outside PR evaluation.
|
||||
|
||||
## Current Implementation Audit
|
||||
|
||||
Current relevant code:
|
||||
|
||||
- `lib/domains.py` contains `DOMAIN_AGENT_MAP`, `agent_for_domain`, `detect_domain_from_diff`, and `detect_domain_from_branch`.
|
||||
- `lib/agent_routing.py` now owns the Phase 1b identity-scored route contract.
|
||||
- The obsolete local `DomainRoute` folder-first draft and its draft tests were removed before this branch was committed.
|
||||
- Cross-domain PRs now require the top 2 routed agents locally, with `route_kind="escalated"` when more than two agents scored.
|
||||
|
||||
Existing implementation truth:
|
||||
|
||||
- The repo already has domain detection that can be reused for path signals.
|
||||
- The new route tests cover six primary agents, broadened ownership domains, top-2 cross-domain routing, fallback, and deterministic repeat behavior.
|
||||
- The existing map includes adjacent domains such as `mechanisms`, `living-capital`, `living-agents`, `critical-systems`, `collective-intelligence`, `teleological-economics`, and `cultural-dynamics`.
|
||||
- The product owner clarified that Phase 1b should use agent identities to route, not only folder names.
|
||||
|
||||
## Existing-Spec Inventory
|
||||
|
||||
| Existing doc | Relevance | Decision |
|
||||
| --- | --- | --- |
|
||||
| `docs/phase1b-agent-routing-spec.md` | Umbrella source of truth. | Reuse. |
|
||||
| `docs/queue.md` | Notes `ai-alignment` domain evolution. | Reuse as a signal for Theseus ownership. |
|
||||
| `docs/ARCHITECTURE.md` | Describes eval stage shape. | Context only. |
|
||||
|
||||
## Goal-Vs-Repo-Truth Diff
|
||||
|
||||
Goal:
|
||||
|
||||
- Return `AgentRoute` with `primary_agent`, `required_agents`, `route_kind`, `scores`, and `evidence`.
|
||||
- Cap cross-domain routes at top 2 agents.
|
||||
- Treat folders as evidence, not the complete classifier.
|
||||
- Be testable without network, DB, GitHub, or LLM calls.
|
||||
|
||||
Repo truth:
|
||||
|
||||
- Existing classifier returns one folder-domain string or `None`.
|
||||
- No scores, evidence, or top-2 agent set exist.
|
||||
- Existing tests do not cover identity-broadened ownership.
|
||||
|
||||
## Completion Percent And Remaining Delta
|
||||
|
||||
Current completion on this branch: 100 percent for local route logic, 0 percent for staging route calibration.
|
||||
|
||||
Remaining delta:
|
||||
|
||||
1. Review the route weights against real recent `decision-engine` PRs.
|
||||
2. Calibrate ambiguous keyword cases from staging evidence.
|
||||
3. Decide whether escalated routes should remain top-2 total or become Leo plus top-2 later.
|
||||
|
||||
## Closure, Endpoint, And Deployment Truth
|
||||
|
||||
Local closure:
|
||||
|
||||
- Route tests pass.
|
||||
- No network or DB dependency exists in route tests.
|
||||
|
||||
Staging closure:
|
||||
|
||||
- Staging proof artifact records route scores and evidence for seven sandbox PRs.
|
||||
|
||||
Production closure:
|
||||
|
||||
- Live PR audit rows show route evidence and required agents.
|
||||
|
||||
This child spec alone cannot prove staging or production behavior.
|
||||
|
||||
## Critical Assumptions And Invalidators
|
||||
|
||||
Assumptions:
|
||||
|
||||
- `decision-engine` file layout is close enough to current local clone for path signals to apply.
|
||||
- Agent identity ownership from m3taversal is authoritative.
|
||||
- Top-2 cap is acceptable for cross-domain cases.
|
||||
|
||||
Invalidators:
|
||||
|
||||
- Product owner changes cross-domain rule from top 2 to all touched agents.
|
||||
- Agent ownership boundaries change materially.
|
||||
- Production PR metadata lacks branch or changed-file data.
|
||||
|
||||
## State And Truth Contract
|
||||
|
||||
Route output schema:
|
||||
|
||||
```python
|
||||
AgentRoute(
|
||||
primary_agent="Rio",
|
||||
required_agents=("Rio",),
|
||||
route_kind="single",
|
||||
scores={"Leo": 0, "Theseus": 1, "Rio": 9, "Vida": 0, "Clay": 0, "Astra": 0},
|
||||
evidence=[
|
||||
{"agent": "Rio", "signal": "path", "weight": 8, "value": "domains/internet-finance/foo.md"}
|
||||
],
|
||||
fallback=False,
|
||||
)
|
||||
```
|
||||
|
||||
`route_kind` values:
|
||||
|
||||
- `single`
|
||||
- `multi`
|
||||
- `fallback`
|
||||
- `escalated`
|
||||
|
||||
`required_agents` must never contain more than two agents in Phase 1b.
|
||||
|
||||
## Measurement Contract
|
||||
|
||||
Required route fixture cases:
|
||||
|
||||
| Fixture | Expected |
|
||||
| --- | --- |
|
||||
| `domains/grand-strategy/foo.md` | Leo |
|
||||
| `domains/ai-alignment/foo.md` | Theseus |
|
||||
| `domains/internet-finance/foo.md` | Rio |
|
||||
| `domains/health/foo.md` | Vida |
|
||||
| `domains/entertainment/foo.md` | Clay |
|
||||
| `domains/space-development/foo.md` | Astra |
|
||||
| `domains/energy/foo.md` | Astra |
|
||||
| `domains/robotics/foo.md` | Astra |
|
||||
| `domains/manufacturing/foo.md` | Astra |
|
||||
| `core/living-capital/foo.md` | Rio |
|
||||
| `core/living-agents/foo.md` | Theseus |
|
||||
| `foundations/cultural-dynamics/foo.md` | Clay |
|
||||
| AI plus x402 diff | Theseus and Rio |
|
||||
| collective AI goals diff | Leo and Theseus |
|
||||
|
||||
Minimum quality metrics:
|
||||
|
||||
- `route_fixture_pass_rate = 100 percent`
|
||||
- `fallback_count = 0` for known fixtures
|
||||
- deterministic repeat count: same input returns same result 100 times
|
||||
|
||||
## Backend Work Required
|
||||
|
||||
Owned files:
|
||||
|
||||
- `lib/agent_routing.py`
|
||||
- `lib/domains.py`
|
||||
- `tests/test_agent_routing.py`
|
||||
|
||||
Implementation steps:
|
||||
|
||||
1. Move new identity routing into `lib/agent_routing.py`.
|
||||
2. Keep `lib/domains.py` as compatibility for domain-oriented callers.
|
||||
3. Define `AGENT_ORDER = ("Leo", "Theseus", "Rio", "Vida", "Clay", "Astra")`.
|
||||
4. Define identity signals per agent.
|
||||
5. Add path signal extraction for `domains`, `entities`, `core`, `foundations`, and `agents`.
|
||||
6. Add branch prefix signal extraction.
|
||||
7. Add capped keyword scoring from filenames and diff text.
|
||||
8. Add top-2 selection rule.
|
||||
9. Add fallback to Leo.
|
||||
10. Add tests.
|
||||
|
||||
Forbidden files:
|
||||
|
||||
- `lib/evaluate.py`
|
||||
- `lib/llm.py`
|
||||
- deploy scripts
|
||||
- secrets or runtime config outside route feature flag wiring
|
||||
|
||||
## Frontend Work Required
|
||||
|
||||
None.
|
||||
|
||||
## Expected Runtime And User-Visible Behavior
|
||||
|
||||
The router itself has no user-visible UI. Its behavior becomes visible through audit logs, PR comment reviewer selection, and proof artifacts.
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
input: domains/internet-finance/x402-agent-payments.md
|
||||
output: required_agents = ["Rio"]
|
||||
```
|
||||
|
||||
Cross-domain example:
|
||||
|
||||
```text
|
||||
input: ai systems claim plus x402 payment claim
|
||||
output: required_agents = ["Theseus", "Rio"]
|
||||
```
|
||||
|
||||
## Validation And Test Matrix
|
||||
|
||||
Commands:
|
||||
|
||||
```bash
|
||||
python3 -m pytest tests/test_agent_routing.py
|
||||
python3 -m ruff check lib/agent_routing.py lib/domains.py tests/test_agent_routing.py
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Test classes:
|
||||
|
||||
- primary ownership routes
|
||||
- broadened ownership routes
|
||||
- branch fallback routes
|
||||
- keyword routes
|
||||
- top-2 cross-domain routes
|
||||
- fallback routes
|
||||
- deterministic tie-breaking
|
||||
- compatibility wrapper behavior
|
||||
|
||||
## CI/CD, Release, And Pre-Push Gate Contract
|
||||
|
||||
Before PR:
|
||||
|
||||
- Route tests pass locally.
|
||||
- No production config defaults change.
|
||||
- No network dependency enters route tests.
|
||||
|
||||
Before staging:
|
||||
|
||||
- Eval integration spec consumes the route result without modifying route internals.
|
||||
|
||||
Before production:
|
||||
|
||||
- Route evidence appears in staging proof artifact.
|
||||
|
||||
## Independent CLI Audit Contract
|
||||
|
||||
Reviewer commands:
|
||||
|
||||
```bash
|
||||
git diff -- lib/agent_routing.py lib/domains.py tests/test_agent_routing.py
|
||||
python3 -m pytest tests/test_agent_routing.py
|
||||
```
|
||||
|
||||
Reviewer checks:
|
||||
|
||||
- Route function is pure.
|
||||
- Scores are explainable.
|
||||
- Top-2 cap is enforced.
|
||||
- Folder paths are not the only signal.
|
||||
- Old callers still work or have a clear migration path.
|
||||
|
||||
## Outside-The-Box Fix Paths
|
||||
|
||||
If keyword scoring is noisy:
|
||||
|
||||
- Disable diff keyword scoring and use path plus branch only.
|
||||
- Use LLM classifier in shadow mode only.
|
||||
- Add explicit PR label or frontmatter hint later.
|
||||
|
||||
If identity boundaries are ambiguous:
|
||||
|
||||
- Prefer top-2 over fallback when two agents have meaningful scores.
|
||||
- Log route evidence for later calibration.
|
||||
|
||||
## Maintenance Capture
|
||||
|
||||
Beneficial now:
|
||||
|
||||
- Keep route logic out of `lib/evaluate.py`.
|
||||
- Keep compatibility wrappers narrow.
|
||||
|
||||
Avoid now:
|
||||
|
||||
- Large domain taxonomy rewrite.
|
||||
- Dashboard UI changes.
|
||||
- Paid classifier calls.
|
||||
|
||||
## Parallelization And Fanout
|
||||
|
||||
Classification: local_owner.
|
||||
|
||||
Do not fan out implementation. This module is a root contract consumed by eval integration.
|
||||
|
||||
Worker-ready prompt:
|
||||
|
||||
```text
|
||||
implement the phase 1b agent identity router in teleo-infrastructure. own lib/agent_routing.py, lib/domains.py compatibility wrappers, and route tests only. make the route function pure, deterministic, evidence-bearing, and capped at top 2 required agents. do not touch eval integration or deploy code.
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- All required route fixtures pass.
|
||||
- Route result includes primary agent, required agents, route kind, scores, evidence, and fallback status.
|
||||
- Cross-domain route never requires more than two agents.
|
||||
- No LLM, network, DB, or GitHub calls occur in the router.
|
||||
|
||||
## Readiness And Claim Boundaries
|
||||
|
||||
Allowed claim:
|
||||
|
||||
- "Agent identity routing is locally implemented and unit-tested."
|
||||
|
||||
Forbidden claim:
|
||||
|
||||
- "Phase 1b eval is complete."
|
||||
|
||||
## Spec Quality Self-Audit
|
||||
|
||||
Required headings present:
|
||||
|
||||
- Current Implementation Audit: present.
|
||||
- Goal-Vs-Repo-Truth Diff: present.
|
||||
- Completion Percent And Remaining Delta: present.
|
||||
- Closure, Endpoint, And Deployment Truth: present.
|
||||
- Critical Assumptions And Invalidators: present.
|
||||
- State And Truth Contract: present.
|
||||
- Measurement Contract: present.
|
||||
- Backend Work Required: present.
|
||||
- Frontend Work Required: present.
|
||||
- Expected Runtime And User-Visible Behavior: present.
|
||||
- Validation And Test Matrix: present.
|
||||
- CI/CD, Release, And Pre-Push Gate Contract: present.
|
||||
- Independent CLI Audit Contract: present.
|
||||
- Outside-The-Box Fix Paths: present.
|
||||
- Maintenance Capture: present.
|
||||
- Parallelization And Fanout: present.
|
||||
|
||||
## Assistant-Added Caveats
|
||||
|
||||
This child spec intentionally keeps routing deterministic and no-spend. That may be less semantically smart than an LLM classifier, but it is the right first implementation for Phase 1b because it is testable, cheap, and auditable.
|
||||
|
|
@ -1,343 +0,0 @@
|
|||
# Phase 1b Child Spec: Eval Pipeline Integration
|
||||
|
||||
Created: 2026-05-29
|
||||
Status: active draft
|
||||
Parent spec: `docs/phase1b-agent-routing-spec.md`
|
||||
|
||||
## Product Outcome Contract
|
||||
|
||||
Pipeline-v2 must use the Phase 1b route result to run the required Hermes agent reviews for a `decision-engine` PR. The old default shape where every non-LIGHT PR receives a domain review plus Leo review must be bypassed when Phase 1b routing is enabled.
|
||||
|
||||
## Goal
|
||||
|
||||
Integrate agent identity routing into `lib/evaluate.py` behind a feature flag, run one or two required reviewer agents, aggregate verdicts, and preserve existing merge or feedback behavior.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not remove the old eval path until staging proof exists.
|
||||
- Do not rewrite the full Forgejo/GitHub API abstraction.
|
||||
- Do not redesign dashboards.
|
||||
- Do not implement separate GitHub identities.
|
||||
- Do not change extraction or validation behavior except as needed for eval tests.
|
||||
|
||||
## Current Implementation Audit
|
||||
|
||||
Current relevant code:
|
||||
|
||||
- `lib/evaluate.py::evaluate_pr` owns single PR evaluation.
|
||||
- `lib/evaluate.py::evaluate_cycle` selects eligible PRs.
|
||||
- `_build_domain_batches` groups STANDARD PRs by DB domain before fetching diffs.
|
||||
- `_run_batch_domain_eval` runs batch domain reviews, then individual Leo reviews.
|
||||
- `run_domain_review` in `lib/llm.py` prompts a domain expert through OpenRouter.
|
||||
- `run_leo_review` in `lib/llm.py` prompts Leo through OpenRouter or Claude path depending on tier.
|
||||
- `parse_verdict` in `lib/eval_parse.py` parses reviewer-specific verdict tags.
|
||||
- `approve_pr`, `reopen_pr`, `close_pr`, and `start_review` handle state transitions.
|
||||
|
||||
Current behavior:
|
||||
|
||||
- Diff path detects a domain.
|
||||
- `agent_for_domain(domain)` selects one domain agent.
|
||||
- Domain review runs first.
|
||||
- Leo review runs after domain approval for non-LIGHT PRs.
|
||||
- `leo_verdict` and `domain_verdict` are the stored verdict fields.
|
||||
- Contributor credit logic assumes Leo can be one evaluator and `domain_agent` can be the other.
|
||||
|
||||
## Existing-Spec Inventory
|
||||
|
||||
| Existing doc | Relevance | Decision |
|
||||
| --- | --- | --- |
|
||||
| `docs/phase1b-agent-routing-spec.md` | Parent route and eval contract. | Reuse. |
|
||||
| `docs/ARCHITECTURE.md` | Existing pipeline stage model. | Reuse as baseline. |
|
||||
| `docs/multi-model-eval-architecture.md` | Prior Leo-plus-second-model design. | Supersede for Phase 1b eval path only. |
|
||||
| `handoff/deprecated/eval-scripts.md` | Confirms shell eval scripts are dead. | Reuse to avoid wrong surface. |
|
||||
|
||||
## Goal-Vs-Repo-Truth Diff
|
||||
|
||||
Goal:
|
||||
|
||||
- `evaluate_pr` calls the route scorer.
|
||||
- Required agents are the only reviewer agents.
|
||||
- One required agent means one review.
|
||||
- Two required agents means two reviews and aggregate verdict.
|
||||
- Default Leo second-review is removed when the feature flag is enabled.
|
||||
- Old behavior remains available when the feature flag is disabled.
|
||||
|
||||
Branch truth:
|
||||
|
||||
- Legacy eval is still available when the feature flag is false.
|
||||
- When the feature flag is true, eval invokes the identity route, runs required agents only, writes `review_records`, and projects aggregate state back into legacy `leo_verdict` and `domain_verdict` columns.
|
||||
- Batch eval is disabled while the feature flag is true because stale DB-domain grouping is not route-aware.
|
||||
- `run_agent_review` exists, but it uses prompt-level identity context rather than loading full KB identity/belief/reasoning files.
|
||||
|
||||
## Completion Percent And Remaining Delta
|
||||
|
||||
Current completion on this branch: 75 percent local implementation behind a default-off feature flag.
|
||||
|
||||
Remaining delta:
|
||||
|
||||
1. Decide direct GitHub `decision-engine` comment transport versus Forgejo-first cutover compatibility.
|
||||
2. Prove with staging PRs and real daemon logs.
|
||||
3. Update contributor/dashboard assumptions only where staging or tests prove breakage.
|
||||
|
||||
## Closure, Endpoint, And Deployment Truth
|
||||
|
||||
Local closure:
|
||||
|
||||
- Mocked eval tests prove route-to-review-to-aggregate behavior.
|
||||
|
||||
Staging closure:
|
||||
|
||||
- Staging sandbox PRs receive expected comments and DB state transitions.
|
||||
|
||||
Production closure:
|
||||
|
||||
- Live `decision-engine` PRs are handled by Phase 1b route path for 24 hours.
|
||||
|
||||
This spec cannot claim production closure without VPS proof.
|
||||
|
||||
## Critical Assumptions And Invalidators
|
||||
|
||||
Assumptions:
|
||||
|
||||
- Feature flag rollback is acceptable.
|
||||
- Existing state fields can support Phase 1b initially by storing primary agent in `domain_agent` and aggregate details in audit rows.
|
||||
- A DB schema migration is avoidable for the first PR.
|
||||
- Master bot comments with `VERDICT:AGENT:*` are acceptable.
|
||||
|
||||
Invalidators:
|
||||
|
||||
- Downstream merge logic requires formal reviews from separate GitHub users.
|
||||
- Dashboards or contributor credit fail hard when Leo is not present.
|
||||
- Batch eval cannot be safely disabled and must be route-aware from day one.
|
||||
- Production env cannot set feature flags.
|
||||
|
||||
## State And Truth Contract
|
||||
|
||||
Feature flag:
|
||||
|
||||
```text
|
||||
PHASE1B_AGENT_ROUTING_ENABLED=false
|
||||
```
|
||||
|
||||
When false:
|
||||
|
||||
- Existing eval behavior continues.
|
||||
|
||||
When true:
|
||||
|
||||
- Eval route is built for every non-bypass PR.
|
||||
- Audit log records route JSON.
|
||||
- Required agent reviews run.
|
||||
- Aggregate verdict determines approval or feedback.
|
||||
|
||||
Minimal DB field use:
|
||||
|
||||
- `domain`: keep route primary domain or `multi`.
|
||||
- `domain_agent`: keep primary agent.
|
||||
- `domain_verdict`: keep aggregate non-Leo review verdict or aggregate verdict.
|
||||
- `leo_verdict`: set `skipped` unless Leo is a required agent; if Leo is required, store Leo verdict.
|
||||
- `review_records`: write one row per required reviewer attempt with reviewer agent, model, outcome, and notes.
|
||||
- review comments include a `PHASE1B_REVIEW` marker and the current local helper suppresses duplicate posts for the same PR and agent.
|
||||
- audit log: route and all per-agent verdicts.
|
||||
|
||||
This is a compatibility posture, not the ideal long-term schema.
|
||||
|
||||
## Measurement Contract
|
||||
|
||||
Required local assertions:
|
||||
|
||||
- Phase 1b flag disabled uses old runner calls.
|
||||
- Phase 1b flag enabled calls `run_agent_review` once for single route.
|
||||
- Phase 1b flag enabled calls `run_agent_review` twice for multi route.
|
||||
- `run_leo_review` is not called unless Leo is in `required_agents`.
|
||||
- all approve returns approved aggregate.
|
||||
- one request changes returns feedback aggregate.
|
||||
- transport failure reopens for retry.
|
||||
- retry after a partial multi-agent success does not duplicate existing posted verdict comments.
|
||||
|
||||
## Backend Work Required
|
||||
|
||||
Owned files:
|
||||
|
||||
- `lib/evaluate.py`
|
||||
- `lib/llm.py`
|
||||
- `lib/config.py`
|
||||
- `lib/eval_parse.py` only if parser compatibility needs explicit tests or normalization.
|
||||
- `tests/test_evaluate_agent_routing.py`
|
||||
- `tests/test_eval_parse.py`
|
||||
|
||||
Implementation steps:
|
||||
|
||||
1. Add `PHASE1B_AGENT_ROUTING_ENABLED` to `lib/config.py`.
|
||||
2. Import route scorer.
|
||||
3. Add `run_agent_review` in `lib/llm.py`.
|
||||
4. Add helper to load agent context from KB worktree.
|
||||
5. Add `aggregate_agent_verdicts`.
|
||||
6. In `evaluate_pr`, after bypasses and diff filtering, branch into Phase 1b path when flag is true.
|
||||
7. In Phase 1b path, run required reviews and post comments through the existing API helper.
|
||||
8. Update DB fields conservatively.
|
||||
9. Write `review_records` rows for each required reviewer attempt.
|
||||
10. Preserve old logic under flag false.
|
||||
11. Disable `_build_domain_batches` while flag is true or make it route-aware.
|
||||
|
||||
Forbidden files:
|
||||
|
||||
- Deprecated eval shell scripts.
|
||||
- Deployment scripts unless needed for documenting the flag.
|
||||
- Runtime secrets.
|
||||
|
||||
## Frontend Work Required
|
||||
|
||||
None.
|
||||
|
||||
## Expected Runtime And User-Visible Behavior
|
||||
|
||||
Single-agent example:
|
||||
|
||||
```text
|
||||
PR touches internet finance.
|
||||
route.required_agents = ["Rio"]
|
||||
pipeline posts a Rio verdict.
|
||||
merge proceeds if Rio approves.
|
||||
```
|
||||
|
||||
Cross-agent example:
|
||||
|
||||
```text
|
||||
PR touches AI systems and x402 payments.
|
||||
route.required_agents = ["Theseus", "Rio"]
|
||||
pipeline posts Theseus and Rio verdicts.
|
||||
merge proceeds only if both approve.
|
||||
```
|
||||
|
||||
Fallback example:
|
||||
|
||||
```text
|
||||
PR cannot be confidently routed.
|
||||
route.required_agents = ["Leo"]
|
||||
pipeline posts Leo verdict.
|
||||
route_kind = fallback is audited.
|
||||
```
|
||||
|
||||
## Validation And Test Matrix
|
||||
|
||||
Commands:
|
||||
|
||||
```bash
|
||||
python3 -m pytest tests/test_evaluate_agent_routing.py tests/test_eval_parse.py
|
||||
python3 -m ruff check lib/evaluate.py lib/llm.py lib/config.py tests/test_evaluate_agent_routing.py
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Test cases:
|
||||
|
||||
- flag-off old behavior smoke
|
||||
- flag-on single reviewer approve
|
||||
- flag-on single reviewer request changes
|
||||
- flag-on two reviewer approve
|
||||
- flag-on two reviewer one reject
|
||||
- missing verdict
|
||||
- transport failure
|
||||
- Leo required route
|
||||
- Leo not required route
|
||||
- batch disabled or route-aware under flag
|
||||
|
||||
## CI/CD, Release, And Pre-Push Gate Contract
|
||||
|
||||
Before PR:
|
||||
|
||||
- Focused tests pass.
|
||||
- Old behavior remains behind flag false.
|
||||
- No production default flips to true.
|
||||
|
||||
Before staging:
|
||||
|
||||
- Operator can enable flag in staging env.
|
||||
- Sandbox repo target is configured.
|
||||
|
||||
Before production:
|
||||
|
||||
- Staging proof artifact exists.
|
||||
- Rollback command is known.
|
||||
|
||||
## Independent CLI Audit Contract
|
||||
|
||||
Reviewer commands:
|
||||
|
||||
```bash
|
||||
git diff -- lib/evaluate.py lib/llm.py lib/config.py tests/test_evaluate_agent_routing.py
|
||||
python3 -m pytest tests/test_evaluate_agent_routing.py
|
||||
```
|
||||
|
||||
Reviewer checks:
|
||||
|
||||
- No deprecated scripts revived.
|
||||
- No secrets introduced.
|
||||
- Feature flag false preserves old path.
|
||||
- Feature flag true bypasses default Leo second-review.
|
||||
- Cross-domain aggregate requires all required reviewers to approve.
|
||||
|
||||
## Outside-The-Box Fix Paths
|
||||
|
||||
If compatibility fields become confusing:
|
||||
|
||||
- Add a narrow DB migration for `route_json` and `agent_verdicts_json`.
|
||||
|
||||
If batch eval blocks safe integration:
|
||||
|
||||
- Disable batch eval under Phase 1b flag for one release.
|
||||
|
||||
If LLM review prompts get too large:
|
||||
|
||||
- Load only identity plus beliefs first, then add reasoning/skills later.
|
||||
|
||||
## Maintenance Capture
|
||||
|
||||
Beneficial now:
|
||||
|
||||
- Isolate Phase 1b logic into helpers instead of expanding `evaluate_pr` deeply.
|
||||
- Keep rollback path explicit.
|
||||
|
||||
Avoid now:
|
||||
|
||||
- Full eval architecture rewrite.
|
||||
- Dashboard redesign.
|
||||
- Broad DB migration unless tests require it.
|
||||
|
||||
## Parallelization And Fanout
|
||||
|
||||
Classification: local_owner.
|
||||
|
||||
Do not fan out before the router contract lands. Eval integration depends tightly on route result semantics.
|
||||
|
||||
Worker-ready prompt:
|
||||
|
||||
```text
|
||||
wire phase 1b routing into teleo-infrastructure eval behind PHASE1B_AGENT_ROUTING_ENABLED. own lib/evaluate.py, lib/llm.py, lib/config.py, and mocked eval tests. run required agents from the route result, aggregate verdicts, preserve old behavior when the flag is false, and do not revive deprecated scripts.
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Flag false path remains available.
|
||||
- Flag true path runs required agents only.
|
||||
- One or two verdicts aggregate correctly.
|
||||
- Existing merge or feedback path is preserved.
|
||||
- Focused mocked tests pass.
|
||||
|
||||
## Readiness And Claim Boundaries
|
||||
|
||||
Allowed claim:
|
||||
|
||||
- "Phase 1b eval integration is locally tested behind a feature flag."
|
||||
|
||||
Forbidden claim:
|
||||
|
||||
- "Phase 1b is live."
|
||||
|
||||
## Spec Quality Self-Audit
|
||||
|
||||
All required execution-grade headings are present. This spec intentionally defers exact production commands to the staging/proof child spec because they depend on VPS truth.
|
||||
|
||||
## Assistant-Added Caveats
|
||||
|
||||
The compatibility use of `domain_verdict` and `leo_verdict` is a pragmatic Phase 1b bridge. A cleaner route schema may be worth adding after staging proof, but a premature migration would widen the blast radius.
|
||||
|
|
@ -1,296 +0,0 @@
|
|||
# Phase 1b Child Spec: GitHub Identity And Bot Posture
|
||||
|
||||
Created: 2026-05-29
|
||||
Status: active draft
|
||||
Parent spec: `docs/phase1b-agent-routing-spec.md`
|
||||
|
||||
## Product Outcome Contract
|
||||
|
||||
Phase 1b must post agent-specific verdicts for `decision-engine` PRs without requiring six separate GitHub accounts. Agent identity is represented in the comment content and verdict tags, while a single master bot account owns transport.
|
||||
|
||||
## Goal
|
||||
|
||||
Define and implement the minimum GitHub identity and comment transport posture for Phase 1b:
|
||||
|
||||
- canonical target is `living-ip/decision-engine`;
|
||||
- one master bot token is acceptable;
|
||||
- verdict comments preserve `VERDICT:AGENT:*`;
|
||||
- duplicate comments are prevented;
|
||||
- old Forgejo or mirror behavior remains rollback-safe until staging proof.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not create separate GitHub users for all agents.
|
||||
- Do not require GitHub branch protection to count separate formal reviewers in Phase 1b.
|
||||
- Do not rewrite every Forgejo-named helper unless needed for Phase 1b comments.
|
||||
- Do not redesign contributor credit.
|
||||
- Do not revive deprecated eval shell scripts.
|
||||
|
||||
## Current Implementation Audit
|
||||
|
||||
Current truth:
|
||||
|
||||
- `pipeline-health-check.py` targets `https://api.github.com/repos/living-ip/decision-engine`.
|
||||
- `research/research-session.sh` targets GitHub `living-ip/decision-engine` and `github-admin-token`.
|
||||
- `handoff/phase1-step3-script-migration.md` documents Phase 1 single `livingIPbot` posture and defers per-agent identities.
|
||||
- `lib/config.py` still defaults to Forgejo `teleo/teleo-codex`.
|
||||
- `lib/github_feedback.py` hardcodes `living-ip/teleo-codex` and reads `github-pat`, not `decision-engine` and `github-admin-token`.
|
||||
- `lib/evaluate.py` posts review comments through Forgejo helpers and per-agent Forgejo tokens.
|
||||
- `lib/github_feedback.py` is a mirror feedback channel keyed by `prs.github_pr`, not the canonical review transport.
|
||||
- `deploy/sync-mirror.sh` still references `living-ip/teleo-codex`.
|
||||
- Fwaz confirmed separate GitHub identities are ideal and blocked on GitHub/PAT setup; Phase 1b implementation should not wait on six distinct accounts if the pipeline can post parseable `VERDICT:AGENT:*` comments through the pipeline bot.
|
||||
|
||||
## Existing-Spec Inventory
|
||||
|
||||
| Existing doc | Relevance | Decision |
|
||||
| --- | --- | --- |
|
||||
| `docs/phase1b-agent-routing-spec.md` | Parent identity posture. | Reuse. |
|
||||
| `handoff/phase1-step3-script-migration.md` | Documents single bot token and GitHub `decision-engine` migration for scripts. | Reuse. |
|
||||
| `handoff/deprecated/eval-scripts.md` | Confirms old eval scripts should not be revived. | Reuse. |
|
||||
|
||||
## Goal-Vs-Repo-Truth Diff
|
||||
|
||||
Goal:
|
||||
|
||||
- One canonical GitHub target for Phase 1b: `living-ip/decision-engine`.
|
||||
- One master bot token for Phase 1b comments.
|
||||
- Agent identity lives in verdict tags and comment headings.
|
||||
- Comment posting supports idempotency by PR, head SHA, and agent.
|
||||
|
||||
Repo truth:
|
||||
|
||||
- GitHub target and token names are split across files.
|
||||
- Eval comments still use Forgejo helpers.
|
||||
- GitHub feedback is non-fatal mirror feedback, not agent review transport.
|
||||
|
||||
## Completion Percent And Remaining Delta
|
||||
|
||||
Current completion: 15 percent.
|
||||
|
||||
Remaining delta:
|
||||
|
||||
1. Add explicit GitHub target config with staging override.
|
||||
2. Normalize token file selection or document compatibility.
|
||||
3. Add Phase 1b comment posting helper for GitHub `decision-engine`.
|
||||
4. Add idempotency marker.
|
||||
5. Add tests for URL target, token path, missing token, and duplicate prevention.
|
||||
6. Decide direct GitHub mode versus Forgejo-mirror mode before staging.
|
||||
|
||||
## Closure, Endpoint, And Deployment Truth
|
||||
|
||||
Local closure:
|
||||
|
||||
- Tests prove comments target `living-ip/decision-engine` and token material is not logged.
|
||||
|
||||
Staging closure:
|
||||
|
||||
- Sandbox PR comments are posted by master bot with agent verdict tags.
|
||||
|
||||
Production closure:
|
||||
|
||||
- Live `decision-engine` PR comments are posted by master bot without duplicates.
|
||||
|
||||
## Critical Assumptions And Invalidators
|
||||
|
||||
Assumptions:
|
||||
|
||||
- One bot account is enough for Phase 1b.
|
||||
- Agent identity in verdict content satisfies acceptance.
|
||||
- Formal GitHub reviews from distinct accounts are not required now.
|
||||
- Per-agent PATs can be added later without changing the route contract.
|
||||
|
||||
Invalidators:
|
||||
|
||||
- Branch protection requires distinct GitHub reviewer identities.
|
||||
- GitHub org disallows the selected PAT or bot account.
|
||||
- Production daemon must remain Forgejo-first for the cutover window.
|
||||
- Direct GitHub PRs lack the DB linkage used by existing `github_feedback`.
|
||||
|
||||
## State And Truth Contract
|
||||
|
||||
Comment idempotency marker:
|
||||
|
||||
```text
|
||||
<!-- PHASE1B_REVIEW:PR=123:SHA=abc123:AGENT=RIO -->
|
||||
```
|
||||
|
||||
Verdict marker remains:
|
||||
|
||||
```text
|
||||
<!-- VERDICT:RIO:APPROVE -->
|
||||
```
|
||||
|
||||
Required config:
|
||||
|
||||
```python
|
||||
GITHUB_OWNER = "living-ip"
|
||||
GITHUB_REPO = "decision-engine"
|
||||
GITHUB_TOKEN_FILE = SECRETS_DIR / "github-admin-token"
|
||||
```
|
||||
|
||||
Staging must override repo or owner without code changes.
|
||||
|
||||
## Measurement Contract
|
||||
|
||||
Minimum tests:
|
||||
|
||||
- URL builder targets `https://api.github.com/repos/living-ip/decision-engine`.
|
||||
- Staging override changes target.
|
||||
- Missing token returns non-fatal failure and audit detail.
|
||||
- Token value is never logged.
|
||||
- Duplicate marker prevents repeat comment for same PR, SHA, and agent.
|
||||
- Six agent verdict tags remain parseable.
|
||||
|
||||
## Backend Work Required
|
||||
|
||||
Owned files:
|
||||
|
||||
- `lib/github_feedback.py` or a new `lib/github_reviews.py`.
|
||||
- `lib/config.py`.
|
||||
- `lib/evaluate.py` only where the eval integration calls the comment helper.
|
||||
- `tests/test_github_identity.py` or equivalent.
|
||||
|
||||
Implementation steps:
|
||||
|
||||
1. Add canonical GitHub target config.
|
||||
2. Add token lookup that prefers `github-admin-token` for Phase 1b and can fall back only if explicitly configured.
|
||||
3. Add comment helper for agent verdict comments.
|
||||
4. Add idempotency marker and readback check.
|
||||
5. Add tests.
|
||||
6. Wire eval integration to the helper under Phase 1b flag.
|
||||
|
||||
Forbidden files:
|
||||
|
||||
- Deprecated eval shell scripts.
|
||||
- Production secrets.
|
||||
- Broad deploy rewrite.
|
||||
|
||||
## Frontend Work Required
|
||||
|
||||
None.
|
||||
|
||||
## Expected Runtime And User-Visible Behavior
|
||||
|
||||
PR comment example:
|
||||
|
||||
```text
|
||||
## Rio review
|
||||
|
||||
<review text>
|
||||
|
||||
<!-- PHASE1B_REVIEW:PR=123:SHA=abc123:AGENT=RIO -->
|
||||
<!-- VERDICT:RIO:APPROVE -->
|
||||
```
|
||||
|
||||
The GitHub account may be a master bot. The comment content must show which agent reviewed.
|
||||
|
||||
## Validation And Test Matrix
|
||||
|
||||
Commands:
|
||||
|
||||
```bash
|
||||
python3 -m pytest tests/test_github_identity.py tests/test_eval_parse.py
|
||||
python3 -m ruff check lib/github_feedback.py lib/config.py tests/test_github_identity.py
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Test cases:
|
||||
|
||||
- canonical target
|
||||
- staging override
|
||||
- missing token
|
||||
- no token logging
|
||||
- idempotent comment marker
|
||||
- all six verdict tags parse
|
||||
|
||||
## CI/CD, Release, And Pre-Push Gate Contract
|
||||
|
||||
Before PR:
|
||||
|
||||
- Local tests prove target and idempotency.
|
||||
|
||||
Before staging:
|
||||
|
||||
- Sandbox repo token exists.
|
||||
- Production token is not used.
|
||||
|
||||
Before production:
|
||||
|
||||
- Bot account has comment permissions on `decision-engine`.
|
||||
- Rollback path is old Forgejo or disabled Phase 1b flag.
|
||||
|
||||
## Independent CLI Audit Contract
|
||||
|
||||
Reviewer checks:
|
||||
|
||||
```bash
|
||||
rg -n "teleo-codex|decision-engine|github-admin-token|github-pat|VERDICT|PHASE1B_REVIEW" lib tests pipeline-health-check.py research deploy
|
||||
```
|
||||
|
||||
Audit questions:
|
||||
|
||||
- Which files still target `teleo-codex`?
|
||||
- Are those files in the Phase 1b runtime path?
|
||||
- Does any log path expose token values?
|
||||
- Does idempotency prevent duplicate comments?
|
||||
|
||||
## Outside-The-Box Fix Paths
|
||||
|
||||
If direct GitHub comments are not safe in the first PR:
|
||||
|
||||
- Keep Forgejo review transport and post GitHub mirror feedback only in staging.
|
||||
- Add a dry-run comment mode that writes the planned body into audit logs.
|
||||
|
||||
If GitHub PAT remains blocked:
|
||||
|
||||
- Use a GitHub App only for comment posting.
|
||||
- Keep master bot for git push but app token for PR comments.
|
||||
|
||||
## Maintenance Capture
|
||||
|
||||
Beneficial now:
|
||||
|
||||
- Name GitHub target config clearly.
|
||||
- Avoid proliferating `github-pat` versus `github-admin-token`.
|
||||
|
||||
Avoid now:
|
||||
|
||||
- Separate agent GitHub users.
|
||||
- Full mirror rewrite.
|
||||
- Contributor identity overhaul.
|
||||
|
||||
## Parallelization And Fanout
|
||||
|
||||
Classification: ready_now after the implementer explicitly chooses direct GitHub comments or Forgejo-mirror compatibility for the Phase 1b flag path.
|
||||
|
||||
Worker-ready prompt:
|
||||
|
||||
```text
|
||||
implement phase 1b github review comment posture. use one master bot token, target living-ip/decision-engine with staging override support, add agent-specific verdict comment helper with idempotency marker, and prove no token leakage. do not create separate agent accounts or rewrite deploy/mirror broadly.
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Phase 1b comment helper targets `decision-engine`.
|
||||
- Master bot can post agent verdict tags.
|
||||
- Duplicate comments are prevented.
|
||||
- Missing token is non-fatal and auditable.
|
||||
- Existing old transport remains rollback-safe.
|
||||
|
||||
## Readiness And Claim Boundaries
|
||||
|
||||
Allowed claim:
|
||||
|
||||
- "Master-bot GitHub verdict comment posture is locally specified/tested."
|
||||
|
||||
Forbidden claim:
|
||||
|
||||
- "Separate agent GitHub identities are solved."
|
||||
|
||||
## Spec Quality Self-Audit
|
||||
|
||||
All required execution-grade headings are present. The exact direct-GitHub versus Forgejo-mirror cutover remains a deliberate implementation decision because current daemon code is Forgejo-first.
|
||||
|
||||
## Assistant-Added Caveats
|
||||
|
||||
The repo has real target drift between `teleo-codex` and `decision-engine`. Do not hide that drift in the eval implementation. The Phase 1b PR should either fix the runtime path it uses or explicitly leave non-runtime references for a later migration.
|
||||
|
|
@ -1,125 +0,0 @@
|
|||
# Phase 1b Local Review Guide
|
||||
|
||||
Status: local-only review artifact
|
||||
Branch: `phase1b-agent-routing-local`
|
||||
|
||||
## What This Repo Is
|
||||
|
||||
`teleo-infrastructure` is the pipeline/runtime repo. For Phase 1b, it owns the evaluation daemon logic that watches PRs, fetches diffs, runs reviewers, posts verdict comments, and moves PR state toward merge or feedback.
|
||||
|
||||
Canonical split for this phase:
|
||||
|
||||
- KB repo: `decision-engine`
|
||||
- implementation/runtime repo: `teleo-infrastructure`
|
||||
- production runtime: VPS under `/opt/teleo-eval`, not currently accessible from this workspace
|
||||
|
||||
## What This Branch Changes
|
||||
|
||||
Local code changes:
|
||||
|
||||
- `lib/agent_routing.py`: new pure router that maps a PR diff to one or two Hermes agents.
|
||||
- `lib/config.py`: adds `PHASE1B_AGENT_ROUTING_ENABLED`, default `false`.
|
||||
- `lib/evaluate.py`: adds a feature-flagged Phase 1b eval path.
|
||||
- `lib/llm.py`: adds `run_agent_review`.
|
||||
- `tests/test_agent_routing.py`: router tests.
|
||||
- `tests/test_evaluate_agent_routing.py`: mocked eval tests.
|
||||
- `tests/test_eval_parse.py`: all six `VERDICT:AGENT:*` parser coverage.
|
||||
|
||||
Spec/docs changes:
|
||||
|
||||
- `docs/phase1b-agent-routing-spec.md`
|
||||
- `docs/phase1b/README.md`
|
||||
- child specs under `docs/phase1b/`
|
||||
- `docs/phase1b/staging-blocker.json`
|
||||
|
||||
## What It Does Not Change
|
||||
|
||||
- It does not enable Phase 1b in production.
|
||||
- It does not touch the VPS.
|
||||
- It does not create or require six GitHub identities.
|
||||
- It does not solve the Forgejo-vs-GitHub cutover.
|
||||
- It does not fix unrelated full-suite failures.
|
||||
|
||||
## Current Safety Posture
|
||||
|
||||
The feature flag defaults off:
|
||||
|
||||
```text
|
||||
PHASE1B_AGENT_ROUTING_ENABLED=false
|
||||
```
|
||||
|
||||
With the flag off, the legacy eval path remains available. The Phase 1b path should only run in staging or a controlled daemon after explicit env config.
|
||||
|
||||
The local review hardening pass removed changes to `lib/domains.py` so the legacy domain map is not changed by this branch.
|
||||
|
||||
## Local Proof
|
||||
|
||||
Focused proof that currently passes:
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m pytest tests/test_agent_routing.py tests/test_evaluate_agent_routing.py tests/test_eval_parse.py
|
||||
.venv/bin/ruff check lib/agent_routing.py lib/domains.py lib/evaluate.py lib/llm.py lib/config.py tests/test_agent_routing.py tests/test_evaluate_agent_routing.py
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Latest focused result:
|
||||
|
||||
```text
|
||||
61 passed
|
||||
ruff: all checks passed
|
||||
git diff --check: passed
|
||||
```
|
||||
|
||||
Full-suite status:
|
||||
|
||||
```text
|
||||
406 passed, 12 failed, 3 errors
|
||||
```
|
||||
|
||||
Known full-suite failure groups:
|
||||
|
||||
- `db.migrate` fresh-fixture rebuild error: `prs_new has no column named auto_merge`
|
||||
- contributor test fixture missing `submitted_by`
|
||||
- date/frontmatter expectations in `test_post_extract.py`
|
||||
- search threshold expectation in `test_search.py`
|
||||
- missing `python-telegram-bot` imports for X content tests
|
||||
|
||||
Those failures mean this branch should not be called repo-green or PR-ready.
|
||||
|
||||
## How To Review Locally
|
||||
|
||||
Stay local:
|
||||
|
||||
```bash
|
||||
git switch phase1b-agent-routing-local
|
||||
git status --short --branch
|
||||
git diff main...HEAD --stat
|
||||
git diff main...HEAD -- lib/agent_routing.py lib/evaluate.py lib/llm.py lib/config.py
|
||||
```
|
||||
|
||||
Review the behavior in this order:
|
||||
|
||||
1. `lib/agent_routing.py`
|
||||
2. `tests/test_agent_routing.py`
|
||||
3. `lib/evaluate.py`
|
||||
4. `tests/test_evaluate_agent_routing.py`
|
||||
5. `docs/phase1b/staging-blocker.json`
|
||||
|
||||
## Before Any PR
|
||||
|
||||
Do not open a PR until at least one of these is true:
|
||||
|
||||
- full-suite failures are triaged into accepted unrelated failures with issue links, or fixed;
|
||||
- staging access is available and a sandbox proof path is ready;
|
||||
- m3taversal/Fwaz explicitly accept a local-only draft review without staging proof.
|
||||
|
||||
## Before Production
|
||||
|
||||
Production requires:
|
||||
|
||||
- staging proof against sandbox `decision-engine`;
|
||||
- exact reviewed SHA;
|
||||
- Leo signoff;
|
||||
- no direct VPS self-upgrades;
|
||||
- `PHASE1B_AGENT_ROUTING_ENABLED` enabled only after cutover plan is written;
|
||||
- rollback path to flag-off behavior.
|
||||
|
|
@ -1,275 +0,0 @@
|
|||
# Phase 1b Child Spec: Reporting And Contributor Compatibility
|
||||
|
||||
Created: 2026-05-29
|
||||
Status: active draft
|
||||
Parent spec: `docs/phase1b-agent-routing-spec.md`
|
||||
|
||||
## Product Outcome Contract
|
||||
|
||||
Phase 1b must not make dashboards, health checks, or contributor credit lie about review state. Reporting may stay minimal, but it must not mark a cross-domain PR as ready before all required agents have reviewed.
|
||||
|
||||
## Goal
|
||||
|
||||
Update compatibility surfaces so Phase 1b required-agent reviews are represented accurately enough for operations, health, and contributor attribution without doing a dashboard redesign.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not redesign the dashboard UI.
|
||||
- Do not implement a new leaderboard model.
|
||||
- Do not require a broad DB migration unless `review_records` is insufficient.
|
||||
- Do not make production-readiness claims from health-check summaries alone.
|
||||
|
||||
## Current Implementation Audit
|
||||
|
||||
Current truth:
|
||||
|
||||
- `lib/db.py` already has `review_records` with `pr_number`, `domain`, `agent`, `reviewer`, `reviewer_model`, `outcome`, `rejection_reason`, and `notes`.
|
||||
- `lib/contributor.py` assumes Leo reviews every PR and credits Leo plus one `domain_agent`.
|
||||
- `lib/health.py` computes approval rates from `domain_verdict` and `leo_verdict`.
|
||||
- `lib/health.py` builds reviewer strings only from `domain_verdict` and `leo_verdict`.
|
||||
- `pipeline-health-check.py` can parse arbitrary `VERDICT:AGENT:*` tags, but it has no required-agent concept.
|
||||
- A cross-domain PR with one approval and one missing required review could be misclassified if reporting only checks "any approve".
|
||||
|
||||
## Existing-Spec Inventory
|
||||
|
||||
| Existing doc | Relevance | Decision |
|
||||
| --- | --- | --- |
|
||||
| `docs/phase1b-agent-routing-spec.md` | Parent route/verdict state. | Reuse. |
|
||||
| `docs/ARCHITECTURE.md` | Health/dashboard baseline. | Reuse as context. |
|
||||
| `docs/DIAGNOSTICS-AGENT-SPEC.md` | Diagnostics philosophy. | Reuse as later direction, not immediate scope. |
|
||||
|
||||
## Goal-Vs-Repo-Truth Diff
|
||||
|
||||
Goal:
|
||||
|
||||
- Required-agent state is visible enough to avoid false readiness.
|
||||
- Contributor evaluator credit follows actual approved reviewer agents.
|
||||
- Health and pipeline checks can distinguish incomplete cross-domain review.
|
||||
|
||||
Repo truth:
|
||||
|
||||
- Legacy fields only represent `domain_verdict` plus `leo_verdict`.
|
||||
- Contributor credit hardcodes Leo as universal reviewer.
|
||||
- `pipeline-health-check.py` parses comments but does not know required reviewers.
|
||||
|
||||
## Completion Percent And Remaining Delta
|
||||
|
||||
Current completion: 10 percent because `review_records` already exists.
|
||||
|
||||
Remaining delta:
|
||||
|
||||
1. Ensure eval integration writes one `review_records` row per required reviewer.
|
||||
2. Update contributor attribution to prefer approved `review_records`.
|
||||
3. Keep legacy fields as projection only.
|
||||
4. Add optional route marker parsing to `pipeline-health-check.py`.
|
||||
5. Add tests proving no partial-review false readiness.
|
||||
|
||||
## Closure, Endpoint, And Deployment Truth
|
||||
|
||||
Local closure:
|
||||
|
||||
- Tests prove contributor credit and stage classification respect required reviewers.
|
||||
|
||||
Staging closure:
|
||||
|
||||
- Staging proof artifact and health readback agree on required-agent completion.
|
||||
|
||||
Production closure:
|
||||
|
||||
- Production health does not show PRs as ready before all required agents approve.
|
||||
|
||||
## Critical Assumptions And Invalidators
|
||||
|
||||
Assumptions:
|
||||
|
||||
- `review_records` is available in production DB schema.
|
||||
- Eval integration can write `review_records` for each required reviewer.
|
||||
- Dashboards can tolerate legacy projections during Phase 1b.
|
||||
|
||||
Invalidators:
|
||||
|
||||
- Production DB lacks `review_records`.
|
||||
- Contributor code path cannot query `review_records` without performance issues.
|
||||
- Branch protection or merge logic uses legacy fields directly for readiness.
|
||||
|
||||
## State And Truth Contract
|
||||
|
||||
`review_records` becomes the compatibility source for per-agent reviewer history.
|
||||
|
||||
Required eval write:
|
||||
|
||||
```text
|
||||
one review_records row per required reviewer per PR attempt
|
||||
```
|
||||
|
||||
Legacy projection:
|
||||
|
||||
- `domain_agent = primary_agent`
|
||||
- `domain_verdict = aggregate_verdict`
|
||||
- `leo_verdict = actual Leo verdict when Leo is required, else skipped`
|
||||
|
||||
Route/audit JSON remains the source for `required_agents`.
|
||||
|
||||
## Measurement Contract
|
||||
|
||||
Minimum compatibility metrics:
|
||||
|
||||
- `review_records_written_count`
|
||||
- `required_reviews_missing_count`
|
||||
- `partial_review_not_ready_count`
|
||||
- `contributor_evaluator_credit_count_by_agent`
|
||||
|
||||
Minimum proof:
|
||||
|
||||
- A two-agent PR with one approval and one missing verdict is not classified as ready.
|
||||
- A two-agent PR with two approvals is classified as ready.
|
||||
- Contributor credit includes both approved reviewers.
|
||||
|
||||
## Backend Work Required
|
||||
|
||||
Owned files:
|
||||
|
||||
- `lib/contributor.py`
|
||||
- `lib/health.py`
|
||||
- `pipeline-health-check.py`
|
||||
- `tests/test_contributor.py` or new focused test.
|
||||
- `tests/test_pipeline_health_phase1b.py` if added.
|
||||
|
||||
Implementation steps:
|
||||
|
||||
1. Confirm `review_records` exists in local schema and migrations.
|
||||
2. Update eval integration spec to write review records per required reviewer.
|
||||
3. Update contributor credit to prefer approved `review_records.reviewer` rows.
|
||||
4. Fall back to legacy `leo_verdict` and `domain_verdict` for old data.
|
||||
5. Update health output to include review records or route audit fields where available.
|
||||
6. Update pipeline health check to read required-agent markers if present.
|
||||
7. Add tests.
|
||||
|
||||
Forbidden work:
|
||||
|
||||
- Dashboard redesign.
|
||||
- New leaderboard model.
|
||||
- Broad schema migration before proof requires it.
|
||||
|
||||
## Frontend Work Required
|
||||
|
||||
None.
|
||||
|
||||
## Expected Runtime And User-Visible Behavior
|
||||
|
||||
Operators should see:
|
||||
|
||||
- Per-agent reviewer outcomes when available.
|
||||
- Cross-domain PRs not marked ready until all required reviewers approve.
|
||||
- Contributor credit reflecting actual approved reviewer agents.
|
||||
|
||||
Existing dashboard layout can remain unchanged if data is honest.
|
||||
|
||||
## Validation And Test Matrix
|
||||
|
||||
Commands:
|
||||
|
||||
```bash
|
||||
python3 -m pytest tests/test_contributor.py tests/test_pipeline_health_phase1b.py
|
||||
python3 -m ruff check lib/contributor.py lib/health.py pipeline-health-check.py tests
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Test cases:
|
||||
|
||||
- old data fallback credits Leo/domain reviewer.
|
||||
- new `review_records` data credits all approved required reviewers.
|
||||
- request-changes reviewer receives no evaluator credit.
|
||||
- one missing required reviewer blocks ready classification.
|
||||
- all required reviewers approve enables ready classification.
|
||||
|
||||
## CI/CD, Release, And Pre-Push Gate Contract
|
||||
|
||||
Before PR:
|
||||
|
||||
- Compatibility tests pass or are documented as not runnable due missing dev deps.
|
||||
|
||||
Before staging:
|
||||
|
||||
- Staging proof includes health and contributor-readback commands.
|
||||
|
||||
Before production:
|
||||
|
||||
- Operator verifies no partial-review false readiness in logs/health readback.
|
||||
|
||||
## Independent CLI Audit Contract
|
||||
|
||||
Reviewer commands:
|
||||
|
||||
```bash
|
||||
rg -n "Leo reviews every PR|leo_verdict|domain_verdict|review_records|required_agents|VERDICT" lib pipeline-health-check.py tests
|
||||
sqlite3 /path/to/pipeline.db ".schema review_records"
|
||||
```
|
||||
|
||||
Reviewer checks:
|
||||
|
||||
- `review_records` is preferred for new evaluator credit.
|
||||
- Legacy fallback remains for old rows.
|
||||
- Health does not rely on any-approve for multi-review readiness.
|
||||
|
||||
## Outside-The-Box Fix Paths
|
||||
|
||||
If `review_records` is insufficient:
|
||||
|
||||
- Add additive `route_json` and `agent_verdicts_json` columns to `prs`.
|
||||
|
||||
If `pipeline-health-check.py` cannot read route markers:
|
||||
|
||||
- Treat cross-domain PRs as awaiting review unless all verdict tags expected by route artifact are present.
|
||||
|
||||
If contributor credit is too risky for Phase 1b:
|
||||
|
||||
- Defer credit mutation and emit review-record-only proof until after eval stability.
|
||||
|
||||
## Maintenance Capture
|
||||
|
||||
Beneficial now:
|
||||
|
||||
- Replace comments claiming "Leo reviews every PR."
|
||||
- Add focused tests for the compatibility projection.
|
||||
|
||||
Avoid now:
|
||||
|
||||
- Dashboard UI rewrite.
|
||||
- Historical backfill.
|
||||
- Leaderboard redesign.
|
||||
|
||||
## Parallelization And Fanout
|
||||
|
||||
Classification: ready_now after eval integration establishes review record writes.
|
||||
|
||||
Worker-ready prompt:
|
||||
|
||||
```text
|
||||
make reporting and contributor attribution phase 1b-compatible. prefer review_records for new evaluator credit, preserve legacy fallback, and prevent health/pipeline checks from marking cross-domain prs ready before all required agents approve. do not redesign dashboards or add broad schema migrations unless tests prove necessary.
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- No code path claims Leo reviews every new Phase 1b PR.
|
||||
- Approved `review_records` can credit all required reviewer agents.
|
||||
- Health/check logic avoids partial-review false readiness.
|
||||
- Legacy data still renders.
|
||||
|
||||
## Readiness And Claim Boundaries
|
||||
|
||||
Allowed claim:
|
||||
|
||||
- "Reporting compatibility is updated to avoid false readiness and credit loss."
|
||||
|
||||
Forbidden claim:
|
||||
|
||||
- "Dashboards are redesigned for Phase 1b."
|
||||
|
||||
## Spec Quality Self-Audit
|
||||
|
||||
All required execution-grade headings are present. This spec is intentionally compatibility-scoped and does not attempt a full reporting product redesign.
|
||||
|
||||
## Assistant-Added Caveats
|
||||
|
||||
The safest first move is to write accurate `review_records` and route audit JSON. Rich dashboards should wait until production behavior proves stable.
|
||||
|
|
@ -1,18 +0,0 @@
|
|||
{
|
||||
"phase": "1b",
|
||||
"blocked_area": "staging_and_production_proof",
|
||||
"attempted_discovery": [
|
||||
"audited teleo-infrastructure eval, config, deploy, systemd, github feedback, and health-check surfaces",
|
||||
"implemented and tested local default-off phase1b routing path",
|
||||
"opened draft pr for reviewed sha",
|
||||
"recorded staging proof contract in docs/phase1b/staging-proof-spec.md"
|
||||
],
|
||||
"exact_blocker": "no usable staging vps clone, crabbox runner config, sandbox decision-engine repo token, or production read-only access is available in this workspace",
|
||||
"why_it_cannot_be_solved_autonomously": "staging proof requires external infrastructure authority and non-production credentials; creating or using those without the project owner/runtime owner would risk mutating production or leaking production secrets",
|
||||
"exact_next_action": "fwaz or m3taversal should provide either a scrubbed hetzner snapshot clone or crabbox config plus staging-only github/openrouter tokens and the sandbox decision-engine repo target",
|
||||
"safe_until_unblocked": [
|
||||
"keep PHASE1B_AGENT_ROUTING_ENABLED=false in production",
|
||||
"review the draft pr locally and in ci",
|
||||
"do not allow agents to self-edit production vps state for this change"
|
||||
]
|
||||
}
|
||||
|
|
@ -1,356 +0,0 @@
|
|||
# Phase 1b Child Spec: Staging Proof
|
||||
|
||||
Created: 2026-05-29
|
||||
Status: active draft
|
||||
Parent spec: `docs/phase1b-agent-routing-spec.md`
|
||||
|
||||
## Product Outcome Contract
|
||||
|
||||
Phase 1b must be tested without mutating the production VPS or production `decision-engine` PRs. A staging clone or disposable remote test box must prove routing, verdict posting, and merge or feedback behavior against a sandbox target before production cutover.
|
||||
|
||||
## Goal
|
||||
|
||||
Define the staging proof path for Phase 1b: provision an isolated production-like runtime, disable production authority, run six single-domain PR cycles plus one cross-domain PR cycle, save a machine-readable proof artifact, then destroy or shut down the staging environment.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not mutate production PRs.
|
||||
- Do not use production GitHub tokens in staging.
|
||||
- Do not prove 24-hour production stability.
|
||||
- Do not promote a mutated staging server as production.
|
||||
- Do not test payment, wallet, Twitter, or mainnet flows.
|
||||
|
||||
## Current Implementation Audit
|
||||
|
||||
Known repo truth:
|
||||
|
||||
- `systemd/teleo-pipeline.service` defines the production-style pipeline service.
|
||||
- `deploy/` contains deployment and mirror scripts.
|
||||
- `docs/ARCHITECTURE.md` documents VPS path assumptions and SQLite state.
|
||||
- `docs/INFRASTRUCTURE.md` documents production as Hetzner `77.42.65.182`, root path `/opt/teleo-eval`, diagnostics on port `8081`, and health on port `8080`.
|
||||
- `deploy/auto-deploy.sh` pulls from `/opt/teleo-eval/workspaces/deploy-infra`, syncs code into runtime paths, restarts changed Python services, and updates `/opt/teleo-eval/.last-deploy-sha` after smoke checks.
|
||||
- `systemd/teleo-pipeline.service` expects `/opt/teleo-eval/pipeline/fix-ownership.sh`, while this repo stores that script under `deploy/fix-ownership.sh`; staging bootstrap must verify the live runtime path before assuming the unit works.
|
||||
- `handoff/phase1-step3-script-migration.md` documents GitHub migration posture and `decision-engine` target for scripts.
|
||||
- `handoff/deprecated/eval-scripts.md` confirms old eval scripts are dead.
|
||||
- Fwaz described the current production update path as `pull -> services recognize pull -> edit on VPS -> PR to Leo`; staging proof must treat that as an unsafe legacy behavior to replace, not as a release gate.
|
||||
- Fwaz approved Crabbox as the long-term disposable staging/test-box direction.
|
||||
|
||||
Unknown production truth:
|
||||
|
||||
- Exact current deployed SHA.
|
||||
- Whether production service files match this repo.
|
||||
- Whether production still points at Forgejo in the live daemon.
|
||||
- Exact restart/deploy commands used by Fwaz or agents.
|
||||
- Current secrets layout.
|
||||
- Current `systemctl cat` output for `teleo-pipeline`, `teleo-diagnostics`, auto-deploy timers, cron-like research jobs, Telegram-related services, and any agent daemons.
|
||||
- Whether production has uncommitted hotfixes, generated scripts, or local service patches under `/opt/teleo-eval`.
|
||||
- Read-only live access is not available in this workspace; the infrastructure audit attempted SSH readback and hit authentication denial, so no production SHA or service state should be claimed from this spec.
|
||||
|
||||
## Existing-Spec Inventory
|
||||
|
||||
| Existing doc | Relevance | Decision |
|
||||
| --- | --- | --- |
|
||||
| `docs/phase1b-agent-routing-spec.md` | Parent proof requirements. | Reuse. |
|
||||
| `docs/ARCHITECTURE.md` | VPS topology and service assumptions. | Reuse with current-readback requirement. |
|
||||
| `systemd/teleo-pipeline.service` | Service command template. | Reuse as staging baseline. |
|
||||
| `handoff/phase1-step3-script-migration.md` | GitHub `decision-engine` target context. | Reuse. |
|
||||
|
||||
## Goal-Vs-Repo-Truth Diff
|
||||
|
||||
Goal:
|
||||
|
||||
- Staging proof runs against sandbox `decision-engine`.
|
||||
- Production services and secrets are disabled before test daemon starts.
|
||||
- Proof artifact captures routes, verdicts, final PR states, SHAs, DB schema, feature flags, and logs.
|
||||
|
||||
Repo truth:
|
||||
|
||||
- Staging automation does not exist.
|
||||
- No proof script exists for seven PR cases.
|
||||
- No machine-readable Phase 1b proof schema exists outside the umbrella spec.
|
||||
|
||||
## Completion Percent And Remaining Delta
|
||||
|
||||
Current completion: 0 percent.
|
||||
|
||||
Remaining delta:
|
||||
|
||||
1. Choose staging substrate: Hetzner snapshot clone, Crabbox, or another disposable test box.
|
||||
2. Define sandbox repo.
|
||||
3. Define staging secrets.
|
||||
4. Write or run proof sequence.
|
||||
5. Retain proof artifact.
|
||||
6. Confirm staging cannot mutate production.
|
||||
|
||||
## Closure, Endpoint, And Deployment Truth
|
||||
|
||||
Staging closure means:
|
||||
|
||||
- Staging environment is isolated.
|
||||
- Sandbox PRs are created and processed.
|
||||
- Required reviewer verdicts appear in PR comments.
|
||||
- Pipeline state transitions match expected behavior.
|
||||
- Proof artifact exists.
|
||||
|
||||
Production closure is separate and requires exact reviewed SHA deployment plus 24-hour readback.
|
||||
|
||||
## Critical Assumptions And Invalidators
|
||||
|
||||
Assumptions:
|
||||
|
||||
- A VPS snapshot or disposable equivalent can run the pipeline.
|
||||
- Production secrets can be removed or replaced before daemon start.
|
||||
- A sandbox GitHub repo can be used.
|
||||
- The proof can run without real production inference spend, or spend is explicitly approved.
|
||||
|
||||
Invalidators:
|
||||
|
||||
- Clone boots production services before quarantine.
|
||||
- Sandbox target cannot receive PRs/comments.
|
||||
- No operator has cloud or VPS access.
|
||||
- Secrets cannot be separated from production.
|
||||
- Service paths on production are materially different from repo docs.
|
||||
|
||||
## State And Truth Contract
|
||||
|
||||
Proof artifact path should be under staging, then copied back into the PR or retained artifact location. Suggested filename:
|
||||
|
||||
```text
|
||||
proof/phase1b-staging-proof-YYYYMMDD-HHMMSS.json
|
||||
```
|
||||
|
||||
Required JSON fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"phase": "1b",
|
||||
"schema_version": 1,
|
||||
"environment": {
|
||||
"kind": "hetzner_snapshot|crabbox|disposable_remote",
|
||||
"host": "...",
|
||||
"snapshot_id": "...",
|
||||
"created_from_prod_host": "77.42.65.182"
|
||||
},
|
||||
"teleo_infrastructure_sha": "...",
|
||||
"decision_engine_target": "living-ip/decision-engine-sandbox",
|
||||
"pipeline_db_schema": 26,
|
||||
"feature_flags": {"PHASE1B_AGENT_ROUTING_ENABLED": "true"},
|
||||
"safety": {
|
||||
"prod_services_disabled": true,
|
||||
"prod_timers_disabled": true,
|
||||
"prod_crons_disabled": true,
|
||||
"prod_secrets_removed": true,
|
||||
"auto_merge_constrained": true
|
||||
},
|
||||
"test_cases": [],
|
||||
"verification_outputs": {
|
||||
"service_status_path": "...",
|
||||
"journal_excerpt_path": "...",
|
||||
"db_snapshot_path": "...",
|
||||
"github_comments_path": "..."
|
||||
},
|
||||
"rollback": {
|
||||
"production_sha_before": "...",
|
||||
"candidate_sha": "...",
|
||||
"rollback_command": "..."
|
||||
},
|
||||
"created_at": "..."
|
||||
}
|
||||
```
|
||||
|
||||
Each test case:
|
||||
|
||||
```json
|
||||
{
|
||||
"case": "internet-finance",
|
||||
"pr": 12,
|
||||
"required_agents": ["Rio"],
|
||||
"posted_verdicts": {"Rio": "approve"},
|
||||
"final_state": "approved",
|
||||
"route_kind": "single"
|
||||
}
|
||||
```
|
||||
|
||||
## Measurement Contract
|
||||
|
||||
Minimum staging cases:
|
||||
|
||||
- grand strategy -> Leo
|
||||
- ai systems or ai alignment -> Theseus
|
||||
- internet finance -> Rio
|
||||
- health -> Vida
|
||||
- entertainment -> Clay
|
||||
- space, robotics, energy, or advanced manufacturing -> Astra
|
||||
- cross-domain ai plus x402 -> Theseus and Rio
|
||||
|
||||
Pass criteria:
|
||||
|
||||
- 7 of 7 route decisions match expected required agents.
|
||||
- 7 of 7 PRs receive parseable verdict comments.
|
||||
- No production repo receives comments.
|
||||
- No production service remains enabled during staging run.
|
||||
|
||||
## Backend Work Required
|
||||
|
||||
Owned surfaces:
|
||||
|
||||
- Staging host.
|
||||
- Sandbox repo.
|
||||
- Staging env/config.
|
||||
- Proof artifact generator or manual proof script.
|
||||
|
||||
Implementation steps:
|
||||
|
||||
1. Snapshot or provision staging environment.
|
||||
2. Block public/prod access.
|
||||
3. Disable production services.
|
||||
4. Remove production secrets.
|
||||
5. Set hostname to staging.
|
||||
6. Configure sandbox target.
|
||||
7. Deploy exact implementation SHA.
|
||||
8. Enable Phase 1b feature flag.
|
||||
9. Create seven sandbox PRs.
|
||||
10. Run pipeline until verdicts and states are visible.
|
||||
11. Save proof artifact.
|
||||
12. Shut down or destroy staging.
|
||||
|
||||
## Frontend Work Required
|
||||
|
||||
None.
|
||||
|
||||
## Expected Runtime And User-Visible Behavior
|
||||
|
||||
Operator sees:
|
||||
|
||||
- Staging service status.
|
||||
- Sandbox PR comments with agent verdict tags.
|
||||
- SQLite rows or logs showing route decisions.
|
||||
- Proof artifact summarizing pass/fail.
|
||||
|
||||
No production user-visible behavior should change during staging.
|
||||
|
||||
## Validation And Test Matrix
|
||||
|
||||
Commands will vary by staging substrate. Baseline readback:
|
||||
|
||||
```bash
|
||||
hostname
|
||||
git -C /opt/teleo-eval/workspaces/deploy-infra rev-parse HEAD
|
||||
cat /opt/teleo-eval/.last-deploy-sha
|
||||
systemctl is-active teleo-pipeline teleo-diagnostics teleo-auto-deploy.timer
|
||||
systemctl list-timers | grep -E 'teleo|sync|extract|research' || true
|
||||
curl -s localhost:8080/health | python3 -m json.tool
|
||||
journalctl -u teleo-pipeline --since "1 hour ago" --no-pager
|
||||
sqlite3 /opt/teleo-eval/pipeline/pipeline.db "select max(version) from schema_version;"
|
||||
sqlite3 /opt/teleo-eval/pipeline/pipeline.db "select number,status,domain,domain_agent,leo_verdict,domain_verdict,auto_merge,github_pr from prs order by number desc limit 20;"
|
||||
gh pr list --repo living-ip/decision-engine-sandbox --state all
|
||||
gh pr view --repo living-ip/decision-engine-sandbox PR_NUMBER --comments
|
||||
```
|
||||
|
||||
Safety checks:
|
||||
|
||||
```bash
|
||||
systemctl is-enabled teleo-pipeline
|
||||
systemctl cat teleo-pipeline
|
||||
systemctl cat teleo-diagnostics
|
||||
grep -R "github-admin-token" /opt/teleo-eval/secrets 2>/dev/null
|
||||
git -C /opt/teleo-eval/workspaces/main remote -v
|
||||
```
|
||||
|
||||
## CI/CD, Release, And Pre-Push Gate Contract
|
||||
|
||||
Before staging:
|
||||
|
||||
- Code PR has passed local tests.
|
||||
- Sandbox target selected.
|
||||
- Staging secrets prepared.
|
||||
|
||||
Before production:
|
||||
|
||||
- Staging proof artifact exists.
|
||||
- Exact SHA to deploy is recorded.
|
||||
- Rollback path is recorded.
|
||||
- Leo approval/signoff for the exact reviewed SHA is recorded.
|
||||
- The production cutover avoids direct agent self-edits on the VPS.
|
||||
|
||||
## Independent CLI Audit Contract
|
||||
|
||||
Auditor should verify:
|
||||
|
||||
- Staging host is not production.
|
||||
- Production services were disabled before test daemon start.
|
||||
- GitHub target is sandbox.
|
||||
- Proof artifact PR IDs belong to sandbox repo.
|
||||
- Logs show no production mutation.
|
||||
|
||||
## Outside-The-Box Fix Paths
|
||||
|
||||
If Hetzner snapshot clone is too risky:
|
||||
|
||||
- Use Crabbox with a synced checkout and fake/sandbox services.
|
||||
- Use a fresh Hetzner server and repo checkout instead of disk clone.
|
||||
- Use local fake GitHub/Forgejo API for pure pipeline proof.
|
||||
|
||||
Substrate guidance:
|
||||
|
||||
- Prefer a Hetzner snapshot clone for canonical staging proof because it exercises `/opt/teleo-eval`, systemd units, timers, runtime user ownership, SQLite path assumptions, and deploy scripts.
|
||||
- Crabbox is acceptable and preferred long-term as `disposable_remote` proof for command/test execution, but it does not count as VPS-clone fidelity unless it recreates the same unit files, runtime paths, service user, database path, and deploy flow.
|
||||
- A local fake GitHub/Forgejo API can prove parser and state logic, but it cannot close the staging acceptance gate for real GitHub comments.
|
||||
|
||||
If inference spend is a concern:
|
||||
|
||||
- Mock agent review responses in staging.
|
||||
- Use a staging-specific cheap model.
|
||||
- Run only one real model call after mocked proof passes.
|
||||
|
||||
## Maintenance Capture
|
||||
|
||||
Beneficial now:
|
||||
|
||||
- Add a reusable `proof/phase1b` script later if manual staging repeats.
|
||||
- Record exact service and config readback.
|
||||
|
||||
Avoid now:
|
||||
|
||||
- Building a full deployment platform.
|
||||
- Giving Crabbox or staging production secrets.
|
||||
- Replacing production with staging server.
|
||||
|
||||
## Parallelization And Fanout
|
||||
|
||||
Classification: draft_gated.
|
||||
|
||||
This can be delegated to Fwaz or the infrastructure owner after code PR exists.
|
||||
|
||||
Worker-ready prompt:
|
||||
|
||||
```text
|
||||
run phase 1b staging proof without mutating production. provision or clone a staging box, disable production services and secrets before starting the daemon, point the runtime at a sandbox decision-engine repo, enable phase 1b routing, run six single-domain prs plus one cross-domain pr, and save a machine-readable proof artifact. do not touch production prs or production secrets.
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Staging is isolated.
|
||||
- Seven sandbox PR cases run.
|
||||
- Required agents match expected matrix.
|
||||
- Verdicts are parseable.
|
||||
- Proof artifact exists.
|
||||
- Staging is stopped or destroyed after proof.
|
||||
|
||||
## Readiness And Claim Boundaries
|
||||
|
||||
Allowed claim:
|
||||
|
||||
- "Phase 1b passed staging proof."
|
||||
|
||||
Forbidden claim:
|
||||
|
||||
- "Production Phase 1b is complete."
|
||||
|
||||
## Spec Quality Self-Audit
|
||||
|
||||
All required execution-grade headings are present. Exact production commands remain unknown until VPS truth is read back.
|
||||
|
||||
## Assistant-Added Caveats
|
||||
|
||||
Crabbox is useful here only as a disposable staging/test substrate. It should not receive production secrets until there is a deliberate security review.
|
||||
|
|
@ -1,21 +0,0 @@
|
|||
{
|
||||
"agent": "leo",
|
||||
"currentTier": "T3_live_readonly",
|
||||
"generatedAt": "2026-06-19T17:25:27.555494+00:00",
|
||||
"httpStatus": 402,
|
||||
"llmOk": true,
|
||||
"notProven": [
|
||||
"teleo-agent@leo.service active",
|
||||
"Telegram message delivery",
|
||||
"Telegram reply delivery",
|
||||
"new payment execution"
|
||||
],
|
||||
"ok": true,
|
||||
"reply": "This reached Leo HTTP via Telegram bridge confirmation.",
|
||||
"requiredTier": "T3_live_readonly",
|
||||
"routeSchema": "livingip.x402.leoChatResponse.v1",
|
||||
"schema": "livingip.telegramLeoX402BridgeProof.v1",
|
||||
"secretValuesIncluded": false,
|
||||
"strongestClaimAllowed": "Telegram bridge helper can POST a no-secret payload to the public Leo HTTP chat route and extract a usable Leo reply. This proves the bridge parser/readback only; it does not prove the Telegram bot service is deployed or active.",
|
||||
"url": "https://leo.livingip.xyz/api/agents/leo/chat"
|
||||
}
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
{
|
||||
"currentTier": "T3_live_readonly",
|
||||
"exactBlocker": "smart_research_paid_execution_not_allowed",
|
||||
"fundsMoved": false,
|
||||
"generatedAt": "2026-06-22T19:21:49.939563+00:00",
|
||||
"httpStatus": 402,
|
||||
"notProven": [
|
||||
"teleo-agent@leo-wallet-test.service active",
|
||||
"Telegram message delivery",
|
||||
"Telegram reply delivery",
|
||||
"Telegram-triggered paid execution"
|
||||
],
|
||||
"ok": true,
|
||||
"paidPostAttempted": false,
|
||||
"reply": "Leo smart research can select the retained AgentCash x402 research provider and query, but did not attempt payment because the call was not fully authorized.",
|
||||
"requiredTier": "T3_live_readonly",
|
||||
"routeSchema": "livingip.x402.leoSmartResearchResponse.v1",
|
||||
"schema": "livingip.telegramLeoX402SmartResearchBridgeProof.v1",
|
||||
"secretValuesIncluded": false,
|
||||
"selectedProvider": "agentcash-stableenrich-exa-search",
|
||||
"strongestClaimAllowed": "Telegram bridge helper can POST a no-secret smart-research payload to the public Leo research route and extract a usable fail-closed reply. This proves route shape and readback only; it does not prove a Telegram bot service is deployed or a paid Telegram message executed.",
|
||||
"url": "https://leo.livingip.xyz/api/agents/leo/research"
|
||||
}
|
||||
|
|
@ -1,83 +0,0 @@
|
|||
# Telegram Leo x402 Bridge PR Packet
|
||||
|
||||
## Working Target
|
||||
|
||||
Run Leo as a Telegram bot without duplicating Leo/x402 logic: Telegram receives
|
||||
a user message, forwards it to `https://leo.livingip.xyz/api/agents/leo/chat`,
|
||||
and replies with the hosted Leo answer.
|
||||
|
||||
## Non-Destructive Boundary
|
||||
|
||||
- This PR does not start, stop, restart, or mutate any live Telegram service.
|
||||
- Deployment sync is updated to copy `telegram/` into both
|
||||
`/opt/teleo-eval/pipeline/telegram/` and `/opt/teleo-eval/telegram/`, matching
|
||||
the current `teleo-agent@.service` runtime path.
|
||||
- Existing Rio and Theseus configs do not set `http_chat_proxy_url`, so their
|
||||
current KB/retrieval path stays unchanged.
|
||||
- Leo opts into the bridge with `telegram/agents/leo.yaml`.
|
||||
- The live token's Telegram username readback is `@TeleoHumanBot`; `@teLEOhuman`
|
||||
remains an alias for continuity with Leo's X identity.
|
||||
- Secret contents are not stored or printed. The config references only the
|
||||
expected token-file name: `leo-telegram-bot-token`.
|
||||
|
||||
## Local Proof Commands
|
||||
|
||||
```sh
|
||||
.venv/bin/python -m pytest tests/test_telegram_leo_x402_bridge.py
|
||||
.venv/bin/python -m py_compile telegram/agent_config.py telegram/http_chat_proxy.py telegram/bot.py telegram/agent_runner.py
|
||||
.venv/bin/python telegram/agent_runner.py --agent leo --validate
|
||||
.venv/bin/python scripts/check_telegram_leo_x402_bridge.py
|
||||
bash -n deploy/deploy.sh deploy/auto-deploy.sh
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Primary retained proof path:
|
||||
|
||||
```text
|
||||
docs/reports/telegram-leo-x402-bridge-proof.json
|
||||
```
|
||||
|
||||
## Production Promotion Commands
|
||||
|
||||
Run only after review and after confirming the token filename exists on the VPS:
|
||||
|
||||
```sh
|
||||
test -f /opt/teleo-eval/secrets/leo-telegram-bot-token
|
||||
test -f /opt/teleo-eval/telegram/agents/leo.yaml
|
||||
test -f /opt/teleo-eval/telegram/http_chat_proxy.py
|
||||
/opt/teleo-eval/pipeline/.venv/bin/python3 /opt/teleo-eval/telegram/agent_runner.py --agent leo --validate
|
||||
systemctl start teleo-agent@leo
|
||||
journalctl -u teleo-agent@leo -n 100 --no-pager
|
||||
```
|
||||
|
||||
Then send Leo a Telegram DM or tag the configured handle and retain:
|
||||
|
||||
- Telegram message/reply screenshot or export.
|
||||
- `journalctl -u teleo-agent@leo` lines showing the proxy path.
|
||||
- Caddy access log line for `POST /api/agents/leo/chat` on `leo.livingip.xyz`.
|
||||
|
||||
## Reviewer CTA
|
||||
|
||||
Approve deploying this as the next non-destructive Telegram step if these facts
|
||||
are acceptable:
|
||||
|
||||
- `leo-telegram-bot-token` exists on the VPS.
|
||||
- Telegram `getMe` for that token reports bot username `TeleoHumanBot`.
|
||||
- `teleo-agent@leo.service` is currently inactive, so this is an additive new
|
||||
agent process rather than a restart of Rio or Theseus.
|
||||
- The public Leo HTTP route already returns a parseable Leo reply.
|
||||
- Existing Rio/Theseus configs do not set `http_chat_proxy_url`.
|
||||
- The deploy-path mismatch is fixed by syncing Telegram files to the runtime
|
||||
path used by `teleo-agent@.service`.
|
||||
|
||||
## Strongest Claim Before Promotion
|
||||
|
||||
PR-ready local bridge only: config and parser tests prove Telegram can be wired
|
||||
to the hosted Leo HTTP route without changing existing Rio/Theseus behavior.
|
||||
|
||||
## Strongest Claim After Promotion
|
||||
|
||||
If the production commands pass and a Telegram message returns a hosted Leo
|
||||
answer, Telegram Leo is a live transport for Leo's public HTTP chat route.
|
||||
Payment and external research claims still come from retained HTTP/x402 proof
|
||||
artifacts, not from Telegram by itself.
|
||||
|
|
@ -1,133 +0,0 @@
|
|||
# Telegram Leo x402 Priority And Spec
|
||||
|
||||
## Definition Of Working
|
||||
|
||||
Working target: a user can DM or tag `@TeleoHumanBot`; the Telegram Leo process
|
||||
forwards the message to `https://leo.livingip.xyz/api/agents/leo/chat`; the user
|
||||
receives a Leo answer; retained logs prove the request hit the public Leo HTTP
|
||||
route.
|
||||
|
||||
Operator path:
|
||||
|
||||
```sh
|
||||
/opt/teleo-eval/pipeline/.venv/bin/python3 /opt/teleo-eval/telegram/agent_runner.py --agent leo --validate
|
||||
systemctl start teleo-agent@leo
|
||||
journalctl -u teleo-agent@leo -n 100 --no-pager
|
||||
```
|
||||
|
||||
Done means:
|
||||
|
||||
- `teleo-agent@leo.service` is active on `77.42.65.182`.
|
||||
- A real Telegram message to `@TeleoHumanBot` receives a Leo reply.
|
||||
- Retained proof includes Telegram message/readback, `journalctl` proxy log, and
|
||||
`leo.livingip.xyz` HTTP access/readback.
|
||||
- Rio and Theseus remain unaffected.
|
||||
|
||||
Not done:
|
||||
|
||||
- HTTP-only proof without a live Telegram delivery.
|
||||
- Candidate/local proof without the public bot service active.
|
||||
- Payment evidence reused as Telegram delivery evidence.
|
||||
|
||||
Required tier: `T3_live_readonly` for the Telegram transport; payment claims use
|
||||
the separately retained x402/Faremeter/AgentCash evidence tiers.
|
||||
|
||||
Current tier: `T3_live_readonly` for bridge-to-public-HTTP proof only. The bot
|
||||
token exists on the VPS, `getMe` identifies `@TeleoHumanBot`, and temporary VPS
|
||||
config validation passed. The live `teleo-agent@leo.service` deployment has not
|
||||
been started by this PR-shaped patch.
|
||||
|
||||
Promotion gate: current VPS readback showed `teleo-agent@leo.service` uses
|
||||
`/opt/teleo-eval/telegram/agent_runner.py`, while deploy scripts historically
|
||||
synced `telegram/` only into `/opt/teleo-eval/pipeline/telegram/`. This patch
|
||||
updates both manual and auto deploy scripts to sync `telegram/` into the runtime
|
||||
path too. Do not start `teleo-agent@leo` until `leo.yaml` and
|
||||
`http_chat_proxy.py` read back from `/opt/teleo-eval/telegram/`.
|
||||
|
||||
## Priority Matrix
|
||||
|
||||
| Priority | Lane | Current State | Ship Decision |
|
||||
| --- | --- | --- | --- |
|
||||
| P0 | Telegram Leo bridge deploy/readback | PR-shaped patch exists; public HTTP proof is retained; VPS token and config validation are confirmed; deploy-path mismatch is patched locally. | Push/merge the bridge, confirm runtime files read back under `/opt/teleo-eval/telegram`, start `teleo-agent@leo`, and retain Telegram delivery logs. |
|
||||
| P0 | Self-hosted Faremeter seller rail | Retained public and hosted mainnet canary receipts exist, and direct `77.42.65.182:3118` currently serves a valid 0.01 USDC mainnet challenge. Fresh `https://leo.livingip.xyz` readback currently returns a Devnet `payment_challenge_unavailable` response, so public host routing is not proving the mainnet Faremeter rail right now. | Keep Faremeter as the default seller rail, but repair/repoint public `leo.livingip.xyz` to the working mainnet route before claiming current public mainnet seller readiness. |
|
||||
| P1 | Leo paid research outbound loop | AgentCash/StableEnrich paid answer and Leo analysis proof already exist. | Expose the result through Telegram after bridge deploy; add per-provider approval packets for new services. |
|
||||
| P1 | Public Leo HTTP behavior | `https://leo.livingip.xyz/api/agents/leo/chat` returns a parseable Leo reply under the current schema. | Treat as the bridge backend; avoid duplicating Leo logic inside Telegram. |
|
||||
| P2 | Corbits/Herd/payable external services | Corbits moved payment but failed upstream API-key validation; Herd still needs an authenticated/payable endpoint proof. | Keep as provider-specific follow-up; do not block Telegram/Faremeter shipping on it. |
|
||||
| P2 | All inbound service coverage | Sponsor-research has the strongest retained x402 receipts; other catalog rows need per-service canaries. | Broaden after Telegram bridge is live. |
|
||||
|
||||
## Spec Tickets
|
||||
|
||||
### TLG-001: Merge And Deploy Telegram Leo Bridge
|
||||
|
||||
Surface: `telegram/agent_config.py`, `telegram/bot.py`,
|
||||
`telegram/http_chat_proxy.py`, `telegram/agents/leo.yaml`.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Deploy scripts sync `telegram/` into `/opt/teleo-eval/telegram/`, matching
|
||||
`teleo-agent@.service`.
|
||||
- Leo config validates in the production venv.
|
||||
- `teleo-agent@leo.service` starts without restarting Rio or Theseus.
|
||||
- A Telegram DM/tag reaches the HTTP proxy branch.
|
||||
- Failure from the HTTP route returns a clear fail-closed Telegram response.
|
||||
|
||||
Evidence:
|
||||
|
||||
- `docs/reports/telegram-leo-x402-bridge-proof.json`
|
||||
- `journalctl -u teleo-agent@leo -n 100 --no-pager`
|
||||
- Telegram screenshot/export for the delivered reply.
|
||||
|
||||
### TLG-002: Retain Live Telegram Proof
|
||||
|
||||
Surface: `scripts/check_telegram_leo_x402_bridge.py` plus a live deployment
|
||||
proof artifact after promotion.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Proof names the public Telegram bot handle and public Leo HTTP URL.
|
||||
- Proof says whether the message was Telegram-delivered or HTTP-only.
|
||||
- Proof includes no token values, secrets, chat-private content beyond the test
|
||||
prompt and Leo reply.
|
||||
|
||||
### X402-FARE-001: Make Faremeter The Default Seller Rail
|
||||
|
||||
Surface: Living IP x402 route configuration and operator docs in the x402
|
||||
worktree.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Public sponsor-research route keeps using the self-hosted Faremeter path.
|
||||
- Fresh public readback for `https://leo.livingip.xyz/api/initiatives/sponsor-research`
|
||||
returns the intended mainnet 0.01 USDC challenge, not the stale Devnet
|
||||
`payment_challenge_unavailable` response.
|
||||
- A repeat public canary command is documented with the smallest safe spend cap.
|
||||
- No PayAI/CDP dependency is required for the default seller rail.
|
||||
|
||||
Existing evidence:
|
||||
|
||||
- `ops/x402-faremeter-mainnet-public-payment-proof.json`
|
||||
- `ops/x402-faremeter-hosted-candidate-payment-proof.json`
|
||||
- `ops/x402-faremeter-direct-payment-proof.json`
|
||||
|
||||
### LEO-OUT-001: Telegram Surface For Paid Research Results
|
||||
|
||||
Surface: Telegram Leo bridge plus retained paid-source artifacts.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Telegram Leo can answer a question using the same public Leo HTTP behavior
|
||||
that already consumed paid AgentCash research.
|
||||
- The answer references retained paid-source evidence without claiming a fresh
|
||||
payment unless a fresh payment receipt exists.
|
||||
|
||||
Existing evidence:
|
||||
|
||||
- `ops/x402-agentcash-paid-readback-proof.json`
|
||||
- `ops/x402-leo-paid-research-analysis-proof.json`
|
||||
|
||||
## Reviewer CTA
|
||||
|
||||
Approve the PR-shaped Telegram bridge and then run the production promotion
|
||||
commands from `docs/telegram-leo-x402-bridge-pr-packet.md`. Do not wait on
|
||||
Corbits/Herd broadening to ship the Telegram transport and self-hosted Faremeter
|
||||
seller rail.
|
||||
|
|
@ -1,43 +0,0 @@
|
|||
{
|
||||
"id": "kb_interop_propose_only",
|
||||
"lane": "kb-interop",
|
||||
"description": "External agents should search and propose KB writes with proof, not push to main or mutate production DB state.",
|
||||
"input": {
|
||||
"branch": "theseus/kb-interop-fixture",
|
||||
"title": "Portable KB read and writeback adapter",
|
||||
"body": "Hermes, OpenClaw, Claude-style, and Pentagon agents need a safe way to search Living IP knowledge and propose source, claim, entity, or correction packets.",
|
||||
"diff": "diff --git a/domains/ai-systems/kb-interop.md b/domains/ai-systems/kb-interop.md\n+Hermes runtime and OpenClaw agents can read cited KB context, propose claim files, and write proof artifacts.\n+They must not push to main, export Bitwarden, or mutate production pipeline.db.",
|
||||
"proposed_write": {
|
||||
"target": "proposal-branch",
|
||||
"mutates_production_db": false,
|
||||
"denied_actions_invoked": [],
|
||||
"proof_path": ".crabbox-results/kb-interop-proof.json"
|
||||
}
|
||||
},
|
||||
"rubric": {
|
||||
"must_check": [
|
||||
"cited reads",
|
||||
"route evidence",
|
||||
"proposal-only write path",
|
||||
"denied tools not invoked"
|
||||
],
|
||||
"reject_if": [
|
||||
"writes directly to main",
|
||||
"mutates production pipeline.db",
|
||||
"omits cited KB context"
|
||||
]
|
||||
},
|
||||
"expected": {
|
||||
"primary_agent": "Theseus",
|
||||
"required_agents": [
|
||||
"Theseus"
|
||||
],
|
||||
"route_kind": "single",
|
||||
"expected_disposition": "approve",
|
||||
"issue_tags": [
|
||||
"citation_required",
|
||||
"kb_propose_only",
|
||||
"no_prod_db_write"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
|
@ -1,37 +0,0 @@
|
|||
{
|
||||
"id": "rio_meteora_lp_incentives",
|
||||
"lane": "rio-economics",
|
||||
"description": "Rio should own Meteora LP, paid-query, x402, futarchy, and contribution-incentive reasoning.",
|
||||
"input": {
|
||||
"branch": "rio/meteora-lp-incentives",
|
||||
"title": "Meteora LP agent capital allocation rules",
|
||||
"body": "Proposal asks whether a Living IP agent should route liquidity through Meteora while using paid queries and futarchy signals to allocate contributor rewards.",
|
||||
"diff": "diff --git a/domains/internet-finance/meteora-lp-agent.md b/domains/internet-finance/meteora-lp-agent.md\n+Meteora LP agent capital allocation depends on x402 paid queries, decision markets, futarchy votes, and contribution weights.\n+The evaluator must check whether payment creates garbage incentives or useful knowledge."
|
||||
},
|
||||
"rubric": {
|
||||
"must_check": [
|
||||
"paid-query effects",
|
||||
"liquidity pool risk",
|
||||
"contribution-weight incentives",
|
||||
"source-diversity and correlated-prior risk"
|
||||
],
|
||||
"reject_if": [
|
||||
"treats payment as quality approval",
|
||||
"ignores LP loss or protocol risk",
|
||||
"omits OPSEC review for capital allocation"
|
||||
]
|
||||
},
|
||||
"expected": {
|
||||
"primary_agent": "Rio",
|
||||
"required_agents": [
|
||||
"Rio"
|
||||
],
|
||||
"route_kind": "single",
|
||||
"expected_disposition": "escalate",
|
||||
"issue_tags": [
|
||||
"capital_allocation",
|
||||
"incentive_design",
|
||||
"paid_query_effects"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
|
@ -1,37 +0,0 @@
|
|||
{
|
||||
"id": "theseus_live_model_switch_reject",
|
||||
"lane": "theseus-model-integrity",
|
||||
"description": "Theseus should reject live model assignment changes based on one persuasive answer.",
|
||||
"input": {
|
||||
"branch": "theseus/model-switch-demo",
|
||||
"title": "Switch live reviewers after one good model answer",
|
||||
"body": "A candidate model sounded better on one example, so this proposal changes live model assignments without a replay corpus, disagreement metrics, or rollback proof.",
|
||||
"diff": "diff --git a/domains/ai-systems/live-model-routing.md b/domains/ai-systems/live-model-routing.md\n+Change live model assignments today because a single LLM response looked more convincing.\n+Skip replay fixtures, skip disagreement queues, and let the agent self-upgrade its tool allowlist."
|
||||
},
|
||||
"rubric": {
|
||||
"must_check": [
|
||||
"replay evidence",
|
||||
"model-family diversity",
|
||||
"prompt/tool safety",
|
||||
"rollback path"
|
||||
],
|
||||
"reject_if": [
|
||||
"uses one answer as proof",
|
||||
"lets an agent approve its own tool escalation",
|
||||
"changes live routing without before/after metrics"
|
||||
]
|
||||
},
|
||||
"expected": {
|
||||
"primary_agent": "Theseus",
|
||||
"required_agents": [
|
||||
"Theseus"
|
||||
],
|
||||
"route_kind": "single",
|
||||
"expected_disposition": "reject",
|
||||
"issue_tags": [
|
||||
"model_assignment_without_eval",
|
||||
"self_upgrade_without_proof",
|
||||
"tool_safety"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
|
@ -1,74 +0,0 @@
|
|||
# Deprecated: eval-dispatcher.sh + eval-worker.sh
|
||||
|
||||
## Why these are NOT being migrated to GitHub
|
||||
|
||||
Both scripts are dead code. The pipeline-v2 daemon replaced them.
|
||||
|
||||
### Evidence
|
||||
|
||||
```bash
|
||||
# Last invocation of either script — March 12, 2026
|
||||
$ ls -la /opt/teleo-eval/logs/eval-{dispatcher,worker}-*.log | tail -3
|
||||
-rw-rw-r-- 1 teleo teleo 4133 Mar 12 12:03 eval-worker-0-PR821.log
|
||||
-rw-rw-r-- 1 teleo teleo 4296 Mar 12 12:03 eval-worker-2-PR678.log
|
||||
-rw-rw-r-- 1 teleo teleo 7405113 Mar 12 12:03 eval-dispatcher.log
|
||||
|
||||
# `teleo-eval.service` does NOT run these — it runs webhook.py
|
||||
$ systemctl cat teleo-eval.service | grep ExecStart
|
||||
ExecStart=/usr/bin/python3 /opt/teleo-eval/webhook.py
|
||||
|
||||
# No cron entries reference them
|
||||
$ crontab -l | grep -E "eval-(dispatcher|worker)"
|
||||
(no output)
|
||||
|
||||
# Live eval logic runs inside teleo-pipeline.service daemon
|
||||
$ systemctl cat teleo-pipeline.service | grep ExecStart
|
||||
ExecStart=/opt/teleo-eval/pipeline/.venv/bin/python3 /opt/teleo-eval/teleo-pipeline.py
|
||||
|
||||
# Daemon imports evaluate_cycle, not the shell scripts
|
||||
$ grep -r "evaluate_cycle\|merge_cycle" /opt/teleo-eval/teleo-pipeline.py
|
||||
from lib.evaluate import evaluate_cycle
|
||||
from lib.merge import merge_cycle
|
||||
```
|
||||
|
||||
### What replaced them
|
||||
|
||||
- `lib/evaluate.py::evaluate_cycle` — the in-daemon equivalent of `eval-dispatcher.sh` + `eval-worker.sh`. Runs continuously as a stage in the pipeline daemon.
|
||||
- `lib/merge.py::merge_cycle` — handles the merge-after-approval step.
|
||||
|
||||
Both fully functional. PRs continue to get reviewed and merged through the daemon, not the shell scripts.
|
||||
|
||||
### Why we didn't migrate them anyway
|
||||
|
||||
Phase 1 scope is migration, not preservation. Migrating dead code:
|
||||
- Adds maintenance surface without runtime value
|
||||
- Costs ~8h of mechanical Forgejo→GitHub URL swapping
|
||||
- Adds attack surface (scripts that touch the codex but no one watches)
|
||||
- Risks Chesterton's Fence violation (the scripts were retired for a reason; we don't fully know the reason without archaeology)
|
||||
|
||||
The pipeline daemon's `lib/evaluate.py` and `lib/merge.py` still reference Forgejo internally (via `lib/forgejo.py`). Those migrations are part of Billy's pipeline-v2 productionization sprint, explicitly out of Phase 1 scope per `phase1-instructions.md`:
|
||||
|
||||
> Out of scope: Pipeline-v2 daemon changes (Billy productionizes).
|
||||
|
||||
### If you ever need to re-activate these scripts
|
||||
|
||||
They're preserved in git history. To re-activate:
|
||||
1. Restore from git
|
||||
2. Apply the migration patterns documented in `phase1-step3-script-migration.md` (Forgejo→GitHub URL swap, Bearer auth, x-access-token URL rewrite for git operations)
|
||||
3. Reconnect to either cron or webhook.py invocation
|
||||
4. Test against `living-ip/decision-engine` not Forgejo
|
||||
|
||||
Don't re-activate without understanding why they were retired. Talk to m3ta first.
|
||||
|
||||
### Files staying as-is
|
||||
|
||||
```
|
||||
/opt/teleo-eval/eval/eval-dispatcher.sh ← preserved, points at Forgejo
|
||||
/opt/teleo-eval/eval/eval-worker.sh ← preserved, points at Forgejo
|
||||
/opt/teleo-eval/eval/tier0-gate.py ← preserved, related helper
|
||||
/opt/teleo-eval/eval/*.log ← old logs, March 2026
|
||||
```
|
||||
|
||||
These will silently break when Forgejo is decommissioned (Phase 1 Step 7). That's fine — they're already dead code; the break is a discovery mechanism, not a regression.
|
||||
|
||||
If Billy decides to delete them entirely during productionization: also fine, they're recoverable from git history.
|
||||
|
|
@ -1,102 +0,0 @@
|
|||
# Phase 1 Step 3: Script Migration to GitHub
|
||||
|
||||
## Summary
|
||||
|
||||
Migrated critical-path scripts from Forgejo (`git.livingip.xyz` / `teleo/teleo-codex`) to GitHub (`living-ip/decision-engine`). Audit found two of the four planned scripts are dead code; scope reduced from 4 scripts to 2.
|
||||
|
||||
| Script | Status | Action |
|
||||
|---|---|---|
|
||||
| `research/research-session.sh` | live (cron paused 2026-05-12 pending Hermes) | migrated this PR |
|
||||
| `pipeline-health-check.py` (VPS root, unversioned) | live, cron every 2h | migrated, deploy notes below |
|
||||
| `eval/eval-dispatcher.sh` | dead since 2026-03-12 | deprecated, see `handoff/deprecated/eval-scripts.md` |
|
||||
| `eval/eval-worker.sh` | dead since 2026-03-12 | deprecated, see `handoff/deprecated/eval-scripts.md` |
|
||||
|
||||
## What changed in `research/research-session.sh`
|
||||
|
||||
Forgejo → GitHub rewire. Same control flow, same Claude invocation, same agent-state hooks. Only external integrations swapped.
|
||||
|
||||
| Change | Before | After |
|
||||
|---|---|---|
|
||||
| API base | `http://localhost:3000` (Forgejo) | `https://api.github.com` |
|
||||
| Repo | `teleo/teleo-codex` | `living-ip/decision-engine` |
|
||||
| Token file | `/opt/teleo-eval/secrets/forgejo-${AGENT}-token` (per-agent), fallback to admin | `/opt/teleo-eval/secrets/github-admin-token` (single livingIPbot, per Option A) |
|
||||
| REST API auth | `?token=<pat>` query or `Authorization: token <pat>` header | `Authorization: Bearer <pat>` + GitHub API version header |
|
||||
| Git auth | `http.extraHeader: Authorization: token <pat>` | `url.<base>.insteadOf` rewrite injecting `x-access-token:<pat>@github.com` |
|
||||
| PR list query | `pulls?state=open` then jq filter | `pulls?state=open&head=living-ip:<branch>` (server-side filter) |
|
||||
| PR create | `POST /api/v1/repos/.../pulls` | `POST /repos/.../pulls` + GitHub API version header |
|
||||
|
||||
## Per-agent identity (deferred)
|
||||
|
||||
Phase 1 uses Option A: single `livingIPbot` PAT for all agents. The `AGENT_TOKEN` variable remains as a placeholder so per-agent elevation in Phase 2 is a one-line change.
|
||||
|
||||
When Billy elevates: generate `github-${AGENT}-token` files at `/opt/teleo-eval/secrets/`, switch the PR-creation curl to use `AGENT_TOKEN`. Git operations stay on the bot token (it's the one with push access to all agent branches). Per-agent VERDICT comments / PR opens become visible in commit history as separate authors.
|
||||
|
||||
## Security note: token in URL rewrite
|
||||
|
||||
The `insteadOf` rewrite injects the PAT into the URL only at command-execution time. It does NOT persist in `.git/config` or `git remote -v`. Verified: post-push `remote -v` shows the clean `https://github.com/living-ip/decision-engine.git` URL.
|
||||
|
||||
Risk surfaces that remain:
|
||||
- `ps auxf` during the git command shows the rewrite arg with the token
|
||||
- If the script's log file gets verbose enough, token could appear in error output
|
||||
|
||||
Mitigation for Billy: switch to a git credential helper (`git-credential-store` or a custom helper that reads from the secrets file) to remove the in-flight exposure entirely. Out of scope for Phase 1.
|
||||
|
||||
## Smoke test results
|
||||
|
||||
Performed against `living-ip/decision-engine` end-to-end, without invoking Claude:
|
||||
|
||||
```
|
||||
✅ git clone (depth=1) via insteadOf rewrite
|
||||
✅ branch create + commit
|
||||
✅ git push (authenticated)
|
||||
✅ PR list API (server-side head= filter)
|
||||
✅ remote -v shows clean URL (token not persisted)
|
||||
✅ branch cleanup
|
||||
```
|
||||
|
||||
Static checks: `bash -n` passes, no residual Forgejo references in the file.
|
||||
|
||||
## `pipeline-health-check.py` — deploy notes (NOT auto-deployed)
|
||||
|
||||
This script lives at `/opt/teleo-eval/pipeline-health-check.py` on the VPS — **NOT in this repo**. It was never added to teleo-infrastructure; lives only as a VPS-local script.
|
||||
|
||||
The migrated version is at `/tmp/pipeline-health-check.py.new` on the VPS. To go live:
|
||||
|
||||
```bash
|
||||
# Backup current
|
||||
cp /opt/teleo-eval/pipeline-health-check.py /opt/teleo-eval/pipeline-health-check.py.bak-pre-github
|
||||
|
||||
# Promote new version
|
||||
cp /tmp/pipeline-health-check.py.new /opt/teleo-eval/pipeline-health-check.py
|
||||
chmod +x /opt/teleo-eval/pipeline-health-check.py
|
||||
|
||||
# Cron continues to run it every 2h; no cron change needed.
|
||||
```
|
||||
|
||||
Before promoting: confirm with Fwaz/m3ta whether the script should also be added to this repo for versioning. Recommended yes; out of scope for this PR.
|
||||
|
||||
Until promoted, the live VPS script keeps reading from Forgejo. Fine during cutover window. Will produce empty/stale metrics once Forgejo is decommissioned (Step 7) if not promoted by then.
|
||||
|
||||
## Auto-deploy of research-session.sh
|
||||
|
||||
`research/research-session.sh` is in the repo's `research/` directory. The auto-deploy script (`teleo-auto-deploy.timer`) rsyncs the repo into `/opt/teleo-eval/pipeline/`. Check whether `research/` is in the rsync manifest — if not, the migrated script won't reach the runtime path that cron used to invoke (`/opt/teleo-eval/research-session.sh`).
|
||||
|
||||
If `research/` is NOT in the rsync manifest (or the runtime path differs from `pipeline/research/research-session.sh`), Billy should add it during productionization. Until then, the migrated script needs a manual `cp` to `/opt/teleo-eval/research-session.sh`.
|
||||
|
||||
This was a pre-existing topology issue; not introduced by this PR.
|
||||
|
||||
## When the cron gets re-enabled
|
||||
|
||||
The research-session crons were paused 2026-05-12 with comment `PAUSED 2026-05-12 (architecture change)`. They should stay paused until Phase 1 Step 4 (Leo on Hermes) is verified — Hermes-Leo's research loop replaces this script for Leo.
|
||||
|
||||
For the other 5 agents (Theseus, Rio, Vida, Clay, Astra): this script remains the fallback path during the Hermes rollout. Billy uses Leo as the pattern and can either re-enable cron or invoke from Hermes per agent.
|
||||
|
||||
## Hermes runtime note (Step 4 preview)
|
||||
|
||||
While auditing the repo, found `hermes-agent/` directory in teleo-infrastructure root. Not investigated as part of Step 3. Will audit during Step 4.
|
||||
|
||||
## Files changed in this PR
|
||||
|
||||
- `research/research-session.sh` — migrated (+29 / −14 lines)
|
||||
- `handoff/phase1-step3-script-migration.md` — this file (new)
|
||||
- `handoff/deprecated/eval-scripts.md` — deprecation notes (new)
|
||||
|
|
@ -1,287 +0,0 @@
|
|||
"""Phase 1b Hermes agent routing.
|
||||
|
||||
Routes knowledge-base PRs to the agent identity that owns the changed domain.
|
||||
This module is deliberately pure: no network, database, LLM, or filesystem IO.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import asdict, dataclass
|
||||
|
||||
AGENT_ORDER: tuple[str, ...] = ("Leo", "Theseus", "Rio", "Vida", "Clay", "Astra")
|
||||
_AGENT_RANK = {agent: idx for idx, agent in enumerate(AGENT_ORDER)}
|
||||
|
||||
DOMAIN_AGENT_MAP: dict[str, str] = {
|
||||
"grand-strategy": "Leo",
|
||||
"strategy": "Leo",
|
||||
"teleohumanity": "Leo",
|
||||
"collective-intelligence": "Leo",
|
||||
"ai-alignment": "Theseus",
|
||||
"ai-systems": "Theseus",
|
||||
"living-agents": "Theseus",
|
||||
"critical-systems": "Theseus",
|
||||
"internet-finance": "Rio",
|
||||
"mechanisms": "Rio",
|
||||
"living-capital": "Rio",
|
||||
"teleological-economics": "Rio",
|
||||
"health": "Vida",
|
||||
"entertainment": "Clay",
|
||||
"cultural-dynamics": "Clay",
|
||||
"space-development": "Astra",
|
||||
"space": "Astra",
|
||||
"robotics": "Astra",
|
||||
"energy": "Astra",
|
||||
"manufacturing": "Astra",
|
||||
"advanced-manufacturing": "Astra",
|
||||
}
|
||||
|
||||
_AGENT_PRIMARY_DOMAIN: dict[str, str] = {
|
||||
"leo": "grand-strategy",
|
||||
"theseus": "ai-systems",
|
||||
"rio": "internet-finance",
|
||||
"vida": "health",
|
||||
"clay": "entertainment",
|
||||
"astra": "space-development",
|
||||
}
|
||||
|
||||
_INGESTION_SOURCE_DOMAIN: dict[str, str] = {
|
||||
"futardio": "internet-finance",
|
||||
"metadao": "internet-finance",
|
||||
"x402": "internet-finance",
|
||||
}
|
||||
|
||||
_DOMAIN_PATH_RE = re.compile(r"^(?:domains|entities|core|foundations)/([^/]+)/")
|
||||
_AGENT_PATH_RE = re.compile(r"^agents/([^/]+)/")
|
||||
|
||||
_KEYWORDS: dict[str, tuple[str, ...]] = {
|
||||
"Leo": (
|
||||
"grand strategy",
|
||||
"collective ai",
|
||||
"collective ais",
|
||||
"collective goals",
|
||||
"goal of the collective",
|
||||
"self-understanding",
|
||||
"self understanding",
|
||||
"teleohumanity",
|
||||
"meta-governance",
|
||||
),
|
||||
"Theseus": (
|
||||
"ai alignment",
|
||||
"ai systems",
|
||||
"ai safety",
|
||||
"agent alignment",
|
||||
"prompt injection",
|
||||
"model behavior",
|
||||
"llm",
|
||||
"hermes runtime",
|
||||
),
|
||||
"Rio": (
|
||||
"internet finance",
|
||||
"x402",
|
||||
"wallet",
|
||||
"payment",
|
||||
"payments",
|
||||
"onchain",
|
||||
"defi",
|
||||
"futarchy",
|
||||
"metadao",
|
||||
"prediction market",
|
||||
"decision market",
|
||||
"stablecoin",
|
||||
),
|
||||
"Vida": (
|
||||
"health",
|
||||
"medicine",
|
||||
"clinical",
|
||||
"patient",
|
||||
"doctor",
|
||||
"disease",
|
||||
"longevity",
|
||||
"biotech",
|
||||
"glp-1",
|
||||
),
|
||||
"Clay": (
|
||||
"entertainment",
|
||||
"game",
|
||||
"games",
|
||||
"media",
|
||||
"story",
|
||||
"film",
|
||||
"music",
|
||||
"culture",
|
||||
),
|
||||
"Astra": (
|
||||
"space",
|
||||
"robotics",
|
||||
"robot",
|
||||
"energy",
|
||||
"manufacturing",
|
||||
"advanced manufacturing",
|
||||
"hardware",
|
||||
"satellite",
|
||||
"rocket",
|
||||
"nuclear",
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RouteEvidence:
|
||||
agent: str
|
||||
signal: str
|
||||
weight: int
|
||||
value: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AgentRoute:
|
||||
primary_agent: str
|
||||
required_agents: tuple[str, ...]
|
||||
route_kind: str
|
||||
scores: dict[str, int]
|
||||
evidence: tuple[RouteEvidence, ...]
|
||||
fallback: bool = False
|
||||
touched_domains: tuple[str, ...] = ()
|
||||
|
||||
def to_audit_dict(self) -> dict:
|
||||
return {
|
||||
"primary_agent": self.primary_agent,
|
||||
"required_agents": list(self.required_agents),
|
||||
"route_kind": self.route_kind,
|
||||
"scores": self.scores,
|
||||
"evidence": [asdict(item) for item in self.evidence],
|
||||
"fallback": self.fallback,
|
||||
"touched_domains": list(self.touched_domains),
|
||||
}
|
||||
|
||||
|
||||
def _changed_paths(diff: str) -> tuple[str, ...]:
|
||||
paths: list[str] = []
|
||||
for line in diff.splitlines():
|
||||
if not line.startswith("diff --git "):
|
||||
continue
|
||||
match = re.match(r"diff --git a/(.*?) b/(.*)$", line)
|
||||
if match:
|
||||
paths.append(match.group(2))
|
||||
return tuple(paths)
|
||||
|
||||
|
||||
def _add_score(
|
||||
scores: dict[str, int],
|
||||
evidence: list[RouteEvidence],
|
||||
agent: str,
|
||||
signal: str,
|
||||
weight: int,
|
||||
value: str,
|
||||
) -> None:
|
||||
if agent not in scores:
|
||||
return
|
||||
scores[agent] += weight
|
||||
evidence.append(RouteEvidence(agent=agent, signal=signal, weight=weight, value=value))
|
||||
|
||||
|
||||
def _domain_for_branch(branch: str) -> str | None:
|
||||
prefix = branch.split("/")[0].lower() if "/" in branch else ""
|
||||
if prefix in _AGENT_PRIMARY_DOMAIN:
|
||||
return _AGENT_PRIMARY_DOMAIN[prefix]
|
||||
if prefix == "ingestion":
|
||||
rest = branch.split("/", 1)[1].lower() if "/" in branch else ""
|
||||
for source_key, domain in _INGESTION_SOURCE_DOMAIN.items():
|
||||
if source_key in rest:
|
||||
return domain
|
||||
return None
|
||||
|
||||
|
||||
def _keyword_hits(agent: str, text: str) -> list[str]:
|
||||
hits = []
|
||||
for keyword in _KEYWORDS[agent]:
|
||||
pattern = rf"(?<![a-z0-9]){re.escape(keyword)}(?![a-z0-9])"
|
||||
if re.search(pattern, text):
|
||||
hits.append(keyword)
|
||||
return hits
|
||||
|
||||
|
||||
def classify_pr_route(
|
||||
diff: str,
|
||||
*,
|
||||
branch: str | None = None,
|
||||
title: str | None = None,
|
||||
body: str | None = None,
|
||||
max_required_agents: int = 2,
|
||||
) -> AgentRoute:
|
||||
"""Classify a PR into one or two required Hermes reviewer agents."""
|
||||
max_required_agents = max(1, min(max_required_agents, 2))
|
||||
scores = {agent: 0 for agent in AGENT_ORDER}
|
||||
evidence: list[RouteEvidence] = []
|
||||
touched_domains: list[str] = []
|
||||
path_signal_found = False
|
||||
|
||||
for path in _changed_paths(diff):
|
||||
domain_match = _DOMAIN_PATH_RE.match(path)
|
||||
if domain_match:
|
||||
domain = domain_match.group(1).lower()
|
||||
if domain in DOMAIN_AGENT_MAP:
|
||||
agent = DOMAIN_AGENT_MAP[domain]
|
||||
_add_score(scores, evidence, agent, "path", 8, path)
|
||||
touched_domains.append(domain)
|
||||
path_signal_found = True
|
||||
continue
|
||||
|
||||
agent_match = _AGENT_PATH_RE.match(path)
|
||||
if agent_match:
|
||||
agent_key = agent_match.group(1).lower()
|
||||
for agent in AGENT_ORDER:
|
||||
if agent.lower() == agent_key:
|
||||
_add_score(scores, evidence, agent, "agent_path", 8, path)
|
||||
path_signal_found = True
|
||||
break
|
||||
|
||||
if branch and not path_signal_found:
|
||||
branch_domain = _domain_for_branch(branch)
|
||||
if branch_domain:
|
||||
agent = DOMAIN_AGENT_MAP[branch_domain]
|
||||
_add_score(scores, evidence, agent, "branch", 4, branch)
|
||||
touched_domains.append(branch_domain)
|
||||
|
||||
keyword_text = "\n".join(part for part in (title or "", body or "", branch or "", diff) if part).lower()
|
||||
for agent in AGENT_ORDER:
|
||||
hits = _keyword_hits(agent, keyword_text)
|
||||
for keyword in hits[:4]:
|
||||
_add_score(scores, evidence, agent, "keyword", 2, keyword)
|
||||
|
||||
ranked = sorted(
|
||||
(agent for agent, score in scores.items() if score > 0),
|
||||
key=lambda agent: (-scores[agent], _AGENT_RANK[agent]),
|
||||
)
|
||||
|
||||
if not ranked:
|
||||
evidence.append(RouteEvidence(agent="Leo", signal="fallback", weight=0, value="no route signal"))
|
||||
return AgentRoute(
|
||||
primary_agent="Leo",
|
||||
required_agents=("Leo",),
|
||||
route_kind="fallback",
|
||||
scores=scores,
|
||||
evidence=tuple(evidence),
|
||||
fallback=True,
|
||||
touched_domains=(),
|
||||
)
|
||||
|
||||
primary = ranked[0]
|
||||
required = tuple(ranked[:max_required_agents])
|
||||
if len(ranked) > max_required_agents:
|
||||
route_kind = "escalated"
|
||||
elif len(required) > 1:
|
||||
route_kind = "multi"
|
||||
else:
|
||||
route_kind = "single"
|
||||
|
||||
return AgentRoute(
|
||||
primary_agent=primary,
|
||||
required_agents=required,
|
||||
route_kind=route_kind,
|
||||
scores=scores,
|
||||
evidence=tuple(evidence),
|
||||
fallback=False,
|
||||
touched_domains=tuple(dict.fromkeys(touched_domains)),
|
||||
)
|
||||
|
|
@ -192,11 +192,6 @@ SAMPLE_AUDIT_MODEL = MODEL_OPUS # Opus for audit — different family from Haik
|
|||
BATCH_EVAL_MAX_PRS = int(os.environ.get("BATCH_EVAL_MAX_PRS", "5"))
|
||||
BATCH_EVAL_MAX_DIFF_BYTES = int(os.environ.get("BATCH_EVAL_MAX_DIFF_BYTES", "100000")) # 100KB
|
||||
|
||||
# --- Phase 1b agent routing ---
|
||||
# When enabled, eval uses the identity router to run exactly the routed Hermes
|
||||
# reviewer agents instead of the legacy domain review + default Leo review path.
|
||||
PHASE1B_AGENT_ROUTING_ENABLED = os.environ.get("PHASE1B_AGENT_ROUTING_ENABLED", "false").lower() == "true"
|
||||
|
||||
# --- Tier logic ---
|
||||
# LIGHT_SKIP_LLM: when True, LIGHT PRs skip domain+Leo review entirely (auto-approve on Tier 0 pass).
|
||||
# Set False for shadow mode (domain review runs but logs only). Flip True after 24h validation (Rhea).
|
||||
|
|
@ -216,14 +211,6 @@ HEALTH_CHECK_INTERVAL = 60
|
|||
# --- Extraction gates ---
|
||||
EXTRACTION_COOLDOWN_HOURS = 4 # Skip sources with any PR activity in this window. Defense-in-depth for DB-status filter.
|
||||
|
||||
# --- Verdict-deadlock reaper ---
|
||||
# Defaults safe (dry-run, 24h age, hourly throttle). Operator flips REAPER_DRY_RUN
|
||||
# to "false" via systemctl edit teleo-pipeline → restart, no code change required.
|
||||
REAPER_DRY_RUN = os.environ.get("REAPER_DRY_RUN", "true").lower() == "true"
|
||||
REAPER_DEADLOCK_AGE_HOURS = int(os.environ.get("REAPER_DEADLOCK_AGE_HOURS", "24"))
|
||||
REAPER_INTERVAL_SECONDS = int(os.environ.get("REAPER_INTERVAL_SECONDS", "3600"))
|
||||
REAPER_MAX_PER_RUN = int(os.environ.get("REAPER_MAX_PER_RUN", "50"))
|
||||
|
||||
# --- Retrieval (Telegram bot) ---
|
||||
RETRIEVAL_RRF_K = 20 # RRF smoothing constant — tuned for 5-10 results per source
|
||||
RETRIEVAL_ENTITY_BOOST = 1.5 # RRF score multiplier for claims wiki-linked from matched entities
|
||||
|
|
|
|||
158
lib/db.py
158
lib/db.py
|
|
@ -9,7 +9,7 @@ from . import config
|
|||
|
||||
logger = logging.getLogger("pipeline.db")
|
||||
|
||||
SCHEMA_VERSION = 27
|
||||
SCHEMA_VERSION = 26
|
||||
|
||||
SCHEMA_SQL = """
|
||||
CREATE TABLE IF NOT EXISTS schema_version (
|
||||
|
|
@ -93,10 +93,6 @@ CREATE TABLE IF NOT EXISTS costs (
|
|||
input_tokens INTEGER DEFAULT 0,
|
||||
output_tokens INTEGER DEFAULT 0,
|
||||
cost_usd REAL DEFAULT 0,
|
||||
duration_ms INTEGER DEFAULT 0,
|
||||
cache_read_tokens INTEGER DEFAULT 0,
|
||||
cache_write_tokens INTEGER DEFAULT 0,
|
||||
cost_estimate_usd REAL DEFAULT 0,
|
||||
PRIMARY KEY (date, model, stage)
|
||||
);
|
||||
|
||||
|
|
@ -407,7 +403,7 @@ def migrate(conn: sqlite3.Connection):
|
|||
if current < 5:
|
||||
# Phase 5: contributor identity system — tracks who contributed what
|
||||
# Aligned with schemas/attribution.md (5 roles) + Leo's tier system.
|
||||
# CI is COMPUTED from raw counts x weights, never stored.
|
||||
# CI is COMPUTED from raw counts × weights, never stored.
|
||||
conn.executescript("""
|
||||
CREATE TABLE IF NOT EXISTS contributors (
|
||||
handle TEXT PRIMARY KEY,
|
||||
|
|
@ -526,105 +522,43 @@ def migrate(conn: sqlite3.Connection):
|
|||
# Old constraint (v7): extract,research,entity,decision,reweave,fix,unknown
|
||||
# New constraint: adds challenge,enrich,synthesize
|
||||
# Also re-derive commit_type from branch prefix for rows with invalid/NULL values.
|
||||
prs_sql_row = conn.execute(
|
||||
"SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'prs'"
|
||||
).fetchone()
|
||||
prs_sql = (prs_sql_row["sql"] or "") if prs_sql_row else ""
|
||||
|
||||
if all(kind in prs_sql for kind in ("challenge", "enrich", "synthesize")):
|
||||
logger.info("Migration v9: prs commit_type CHECK already expanded, rebuild skipped")
|
||||
else:
|
||||
# Step 1: Get all column names from existing table.
|
||||
cols_info = conn.execute("PRAGMA table_info(prs)").fetchall()
|
||||
col_names = [c["name"] for c in cols_info]
|
||||
# Step 1: Get all column names from existing table
|
||||
cols_info = conn.execute("PRAGMA table_info(prs)").fetchall()
|
||||
col_names = [c["name"] for c in cols_info]
|
||||
col_list = ", ".join(col_names)
|
||||
|
||||
# Step 2: Create new table with the expanded CHECK constraint.
|
||||
# Keep columns introduced before and after v9 when present. This keeps
|
||||
# fresh DB bootstrap and partially manually-migrated VPS DBs idempotent.
|
||||
target_cols = [
|
||||
"number",
|
||||
"source_path",
|
||||
"branch",
|
||||
"status",
|
||||
"domain",
|
||||
"agent",
|
||||
"commit_type",
|
||||
"tier",
|
||||
"tier0_pass",
|
||||
"leo_verdict",
|
||||
"domain_verdict",
|
||||
"domain_agent",
|
||||
"domain_model",
|
||||
"priority",
|
||||
"origin",
|
||||
"eval_attempts",
|
||||
"eval_issues",
|
||||
"fix_attempts",
|
||||
"transient_retries",
|
||||
"substantive_retries",
|
||||
"last_error",
|
||||
"last_attempt",
|
||||
"cost_usd",
|
||||
"auto_merge",
|
||||
"github_pr",
|
||||
"source_channel",
|
||||
"prompt_version",
|
||||
"pipeline_version",
|
||||
"submitted_by",
|
||||
"conflict_rebase_attempts",
|
||||
"merge_failures",
|
||||
"merge_cycled",
|
||||
"created_at",
|
||||
"merged_at",
|
||||
]
|
||||
insert_cols = [col for col in target_cols if col in col_names]
|
||||
col_list = ", ".join(insert_cols)
|
||||
|
||||
conn.executescript("""
|
||||
CREATE TABLE prs_new (
|
||||
number INTEGER PRIMARY KEY,
|
||||
source_path TEXT REFERENCES sources(path),
|
||||
branch TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'open',
|
||||
domain TEXT,
|
||||
agent TEXT,
|
||||
commit_type TEXT CHECK(commit_type IS NULL OR commit_type IN ('extract','research','entity','decision','reweave','fix','challenge','enrich','synthesize','unknown')),
|
||||
tier TEXT,
|
||||
tier0_pass INTEGER,
|
||||
leo_verdict TEXT DEFAULT 'pending',
|
||||
domain_verdict TEXT DEFAULT 'pending',
|
||||
domain_agent TEXT,
|
||||
domain_model TEXT,
|
||||
priority TEXT,
|
||||
origin TEXT DEFAULT 'pipeline',
|
||||
eval_attempts INTEGER DEFAULT 0,
|
||||
eval_issues TEXT DEFAULT '[]',
|
||||
fix_attempts INTEGER DEFAULT 0,
|
||||
transient_retries INTEGER DEFAULT 0,
|
||||
substantive_retries INTEGER DEFAULT 0,
|
||||
last_error TEXT,
|
||||
last_attempt TEXT,
|
||||
cost_usd REAL DEFAULT 0,
|
||||
auto_merge INTEGER DEFAULT 0,
|
||||
github_pr INTEGER,
|
||||
source_channel TEXT,
|
||||
prompt_version TEXT,
|
||||
pipeline_version TEXT,
|
||||
submitted_by TEXT,
|
||||
conflict_rebase_attempts INTEGER DEFAULT 0,
|
||||
merge_failures INTEGER DEFAULT 0,
|
||||
merge_cycled INTEGER DEFAULT 0,
|
||||
created_at TEXT DEFAULT (datetime('now')),
|
||||
merged_at TEXT
|
||||
);
|
||||
""")
|
||||
if insert_cols:
|
||||
conn.execute(f"INSERT INTO prs_new ({col_list}) SELECT {col_list} FROM prs")
|
||||
conn.executescript("""
|
||||
DROP TABLE prs;
|
||||
ALTER TABLE prs_new RENAME TO prs;
|
||||
""")
|
||||
logger.info("Migration v9: rebuilt prs table with expanded commit_type CHECK constraint")
|
||||
# Step 2: Create new table with expanded CHECK constraint
|
||||
conn.executescript(f"""
|
||||
CREATE TABLE prs_new (
|
||||
number INTEGER PRIMARY KEY,
|
||||
source_path TEXT REFERENCES sources(path),
|
||||
branch TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'open',
|
||||
domain TEXT,
|
||||
agent TEXT,
|
||||
commit_type TEXT CHECK(commit_type IS NULL OR commit_type IN ('extract','research','entity','decision','reweave','fix','challenge','enrich','synthesize','unknown')),
|
||||
tier TEXT,
|
||||
tier0_pass INTEGER,
|
||||
leo_verdict TEXT DEFAULT 'pending',
|
||||
domain_verdict TEXT DEFAULT 'pending',
|
||||
domain_agent TEXT,
|
||||
domain_model TEXT,
|
||||
priority TEXT,
|
||||
origin TEXT DEFAULT 'pipeline',
|
||||
transient_retries INTEGER DEFAULT 0,
|
||||
substantive_retries INTEGER DEFAULT 0,
|
||||
last_error TEXT,
|
||||
last_attempt TEXT,
|
||||
cost_usd REAL DEFAULT 0,
|
||||
created_at TEXT DEFAULT (datetime('now')),
|
||||
merged_at TEXT
|
||||
);
|
||||
INSERT INTO prs_new ({col_list}) SELECT {col_list} FROM prs;
|
||||
DROP TABLE prs;
|
||||
ALTER TABLE prs_new RENAME TO prs;
|
||||
""")
|
||||
logger.info("Migration v9: rebuilt prs table with expanded commit_type CHECK constraint")
|
||||
|
||||
# Step 3: Re-derive commit_type from branch prefix for invalid/NULL values
|
||||
rows = conn.execute(
|
||||
|
|
@ -679,7 +613,7 @@ def migrate(conn: sqlite3.Connection):
|
|||
|
||||
if current < 17:
|
||||
# Add prompt/pipeline version tracking per PR
|
||||
for col, _default in [
|
||||
for col, default in [
|
||||
("prompt_version", None),
|
||||
("pipeline_version", None),
|
||||
]:
|
||||
|
|
@ -870,7 +804,7 @@ def migrate(conn: sqlite3.Connection):
|
|||
# Add publishers + contributor_identities. Non-breaking — new tables only.
|
||||
# No existing data moved. Classification into publishers happens via a
|
||||
# separate script (scripts/reclassify-contributors.py) with Cory-reviewed
|
||||
# seed list. CHECK constraint on contributors.kind deferred until after
|
||||
# seed list. CHECK constraint on contributors.kind deferred to v27 after
|
||||
# classification completes. (Apr 24 Cory directive: "fix schema, don't
|
||||
# filter output" — separate contributors from publishers at the data layer.)
|
||||
conn.executescript("""
|
||||
|
|
@ -911,20 +845,6 @@ def migrate(conn: sqlite3.Connection):
|
|||
conn.commit()
|
||||
logger.info("Migration v26: added publishers + contributor_identities tables + sources provenance columns")
|
||||
|
||||
if current < 27:
|
||||
for col, definition in [
|
||||
("duration_ms", "INTEGER DEFAULT 0"),
|
||||
("cache_read_tokens", "INTEGER DEFAULT 0"),
|
||||
("cache_write_tokens", "INTEGER DEFAULT 0"),
|
||||
("cost_estimate_usd", "REAL DEFAULT 0"),
|
||||
]:
|
||||
try:
|
||||
conn.execute(f"ALTER TABLE costs ADD COLUMN {col} {definition}")
|
||||
except sqlite3.OperationalError:
|
||||
pass
|
||||
conn.commit()
|
||||
logger.info("Migration v27: added detailed cost accounting columns")
|
||||
|
||||
if current < SCHEMA_VERSION:
|
||||
conn.execute(
|
||||
"INSERT OR REPLACE INTO schema_version (version) VALUES (?)",
|
||||
|
|
|
|||
263
lib/evaluate.py
263
lib/evaluate.py
|
|
@ -24,9 +24,7 @@ import random
|
|||
from datetime import datetime, timezone
|
||||
|
||||
from . import config, db
|
||||
from .agent_routing import AgentRoute, classify_pr_route
|
||||
from .domains import agent_for_domain, detect_domain_from_branch, detect_domain_from_diff
|
||||
from .eval_actions import dispose_rejected_pr, post_formal_approvals, terminate_pr
|
||||
from .eval_parse import (
|
||||
deterministic_tier,
|
||||
diff_contains_claim_type,
|
||||
|
|
@ -40,10 +38,12 @@ from .eval_parse import (
|
|||
)
|
||||
from .forgejo import api as forgejo_api
|
||||
from .forgejo import get_agent_token, get_pr_diff, repo_path
|
||||
from .github_feedback import on_eval_complete
|
||||
from .llm import run_agent_review, run_batch_domain_review, run_domain_review, run_leo_review, triage_pr
|
||||
from .merge import PIPELINE_OWNED_PREFIXES
|
||||
from .llm import run_batch_domain_review, run_domain_review, run_leo_review, triage_pr
|
||||
from .eval_actions import dispose_rejected_pr, post_formal_approvals, terminate_pr
|
||||
from .github_feedback import on_eval_complete
|
||||
from .pr_state import approve_pr, close_pr, reopen_pr, start_review
|
||||
from .validate import load_existing_claims
|
||||
|
||||
logger = logging.getLogger("pipeline.evaluate")
|
||||
|
||||
|
|
@ -57,236 +57,6 @@ logger = logging.getLogger("pipeline.evaluate")
|
|||
# ─── Single PR evaluation ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _phase1b_domain_for_route(route: AgentRoute) -> str:
|
||||
if route.route_kind in ("multi", "escalated"):
|
||||
return "multi"
|
||||
if route.touched_domains:
|
||||
return route.touched_domains[0]
|
||||
return "general"
|
||||
|
||||
|
||||
def _phase1b_review_model(agent: str, tier: str) -> str:
|
||||
if agent == "Leo":
|
||||
return config.EVAL_LEO_STANDARD_MODEL
|
||||
return config.EVAL_DOMAIN_MODEL
|
||||
|
||||
|
||||
def _phase1b_compat_verdicts(agent_verdicts: dict[str, str]) -> tuple[str, str]:
|
||||
"""Project arbitrary routed verdicts into legacy leo/domain columns."""
|
||||
leo_verdict = agent_verdicts.get("Leo", "skipped")
|
||||
non_leo = [verdict for agent, verdict in agent_verdicts.items() if agent != "Leo"]
|
||||
aggregate = "request_changes" if "request_changes" in agent_verdicts.values() else "approve"
|
||||
domain_verdict = aggregate if non_leo else "skipped"
|
||||
return leo_verdict, domain_verdict
|
||||
|
||||
|
||||
def _phase1b_review_marker(pr_number: int, agent: str) -> str:
|
||||
return f"<!-- PHASE1B_REVIEW:PR={pr_number}:AGENT={agent.upper()} -->"
|
||||
|
||||
|
||||
async def _post_phase1b_review_comment(pr_number: int, agent: str, review_text: str) -> bool:
|
||||
"""Post a routed review comment once per PR/agent marker."""
|
||||
marker = _phase1b_review_marker(pr_number, agent)
|
||||
comments = await forgejo_api("GET", repo_path(f"issues/{pr_number}/comments"))
|
||||
if isinstance(comments, list):
|
||||
for comment in comments:
|
||||
body = comment.get("body", "") if isinstance(comment, dict) else ""
|
||||
if marker in body:
|
||||
logger.info("PR #%d: Phase 1b %s review comment already posted", pr_number, agent)
|
||||
return False
|
||||
|
||||
body = review_text if marker in review_text else f"{marker}\n{review_text}"
|
||||
result = await forgejo_api(
|
||||
"POST",
|
||||
repo_path(f"issues/{pr_number}/comments"),
|
||||
{"body": body},
|
||||
)
|
||||
return result is not None
|
||||
|
||||
|
||||
async def _evaluate_pr_phase1b(
|
||||
conn,
|
||||
pr_number: int,
|
||||
*,
|
||||
tier: str,
|
||||
diff: str,
|
||||
review_diff: str,
|
||||
files: str,
|
||||
branch_name: str,
|
||||
eval_attempts: int,
|
||||
pr_cost: float,
|
||||
) -> dict:
|
||||
"""Evaluate a PR using the Phase 1b identity router."""
|
||||
from . import costs
|
||||
|
||||
route = classify_pr_route(diff, branch=branch_name)
|
||||
domain = _phase1b_domain_for_route(route)
|
||||
route_context = json.dumps(route.to_audit_dict(), sort_keys=True)
|
||||
|
||||
conn.execute(
|
||||
"UPDATE prs SET domain = ?, domain_agent = ? WHERE number = ?",
|
||||
(domain, route.primary_agent, pr_number),
|
||||
)
|
||||
db.audit(
|
||||
conn,
|
||||
"evaluate",
|
||||
"phase1b_route",
|
||||
json.dumps({"pr": pr_number, "tier": tier, "route": route.to_audit_dict()}),
|
||||
)
|
||||
|
||||
reviews: dict[str, str] = {}
|
||||
agent_verdicts: dict[str, str] = {}
|
||||
usage_by_agent: dict[str, dict] = {}
|
||||
|
||||
for agent in route.required_agents:
|
||||
logger.info("PR #%d: Phase 1b %s review (tier=%s, route=%s)", pr_number, agent, tier, route.route_kind)
|
||||
review_text, usage = await run_agent_review(review_diff, files, agent, route_context, tier=tier)
|
||||
if review_text is None:
|
||||
reopen_pr(conn, pr_number)
|
||||
if pr_cost > 0:
|
||||
conn.execute("UPDATE prs SET cost_usd = cost_usd + ? WHERE number = ?", (pr_cost, pr_number))
|
||||
return {
|
||||
"pr": pr_number,
|
||||
"skipped": True,
|
||||
"reason": "phase1b_agent_review_failed",
|
||||
"agent": agent,
|
||||
}
|
||||
|
||||
verdict = parse_verdict(review_text, agent)
|
||||
reviews[agent] = review_text
|
||||
agent_verdicts[agent] = verdict
|
||||
usage_by_agent[agent] = usage
|
||||
|
||||
await _post_phase1b_review_comment(pr_number, agent, review_text)
|
||||
|
||||
db.record_review(
|
||||
conn,
|
||||
pr_number,
|
||||
"approved" if verdict == "approve" else "rejected",
|
||||
domain=domain,
|
||||
agent=agent,
|
||||
reviewer=agent,
|
||||
reviewer_model=_phase1b_review_model(agent, tier),
|
||||
rejection_reason=",".join(parse_issues(review_text)) if verdict == "request_changes" else None,
|
||||
notes=review_text,
|
||||
)
|
||||
|
||||
aggregate_approve = all(verdict == "approve" for verdict in agent_verdicts.values())
|
||||
leo_verdict, domain_verdict = _phase1b_compat_verdicts(agent_verdicts)
|
||||
conn.execute(
|
||||
"UPDATE prs SET leo_verdict = ?, domain_verdict = ?, domain_model = ? WHERE number = ?",
|
||||
(leo_verdict, domain_verdict, "phase1b-agent-routing", pr_number),
|
||||
)
|
||||
|
||||
for agent, usage in usage_by_agent.items():
|
||||
model = _phase1b_review_model(agent, tier)
|
||||
pr_cost += costs.record_usage(
|
||||
conn,
|
||||
model,
|
||||
"eval_agent",
|
||||
input_tokens=usage.get("prompt_tokens", 0),
|
||||
output_tokens=usage.get("completion_tokens", 0),
|
||||
backend="openrouter",
|
||||
)
|
||||
|
||||
if aggregate_approve:
|
||||
pr_info = await forgejo_api("GET", repo_path(f"pulls/{pr_number}"))
|
||||
pr_author = pr_info.get("user", {}).get("login", "") if pr_info else ""
|
||||
await post_formal_approvals(pr_number, pr_author)
|
||||
|
||||
is_agent_pr = not branch_name.startswith(PIPELINE_OWNED_PREFIXES)
|
||||
approve_pr(
|
||||
conn,
|
||||
pr_number,
|
||||
domain=domain,
|
||||
auto_merge=1 if is_agent_pr else 0,
|
||||
leo_verdict=leo_verdict,
|
||||
domain_verdict=domain_verdict,
|
||||
)
|
||||
db.audit(
|
||||
conn,
|
||||
"evaluate",
|
||||
"phase1b_approved",
|
||||
json.dumps(
|
||||
{
|
||||
"pr": pr_number,
|
||||
"tier": tier,
|
||||
"route": route.to_audit_dict(),
|
||||
"agent_verdicts": agent_verdicts,
|
||||
"auto_merge": is_agent_pr,
|
||||
}
|
||||
),
|
||||
)
|
||||
try:
|
||||
await on_eval_complete(conn, pr_number, outcome="approved", review_text="\n\n".join(reviews.values()))
|
||||
except Exception:
|
||||
logger.exception("PR #%d: GitHub eval feedback failed (non-fatal)", pr_number)
|
||||
else:
|
||||
all_issues: list[str] = []
|
||||
for agent, verdict in agent_verdicts.items():
|
||||
if verdict == "request_changes":
|
||||
all_issues.extend(parse_issues(reviews[agent]))
|
||||
|
||||
reopen_pr(
|
||||
conn,
|
||||
pr_number,
|
||||
leo_verdict=leo_verdict,
|
||||
domain_verdict=domain_verdict,
|
||||
last_error="phase1b agent review requested changes",
|
||||
eval_issues=json.dumps(all_issues),
|
||||
)
|
||||
feedback = {
|
||||
"route": route.to_audit_dict(),
|
||||
"agent_verdicts": agent_verdicts,
|
||||
"tier": tier,
|
||||
"issues": all_issues,
|
||||
}
|
||||
conn.execute(
|
||||
"UPDATE sources SET feedback = ? WHERE path = (SELECT source_path FROM prs WHERE number = ?)",
|
||||
(json.dumps(feedback), pr_number),
|
||||
)
|
||||
db.audit(
|
||||
conn,
|
||||
"evaluate",
|
||||
"phase1b_changes_requested",
|
||||
json.dumps(
|
||||
{
|
||||
"pr": pr_number,
|
||||
"tier": tier,
|
||||
"route": route.to_audit_dict(),
|
||||
"agent_verdicts": agent_verdicts,
|
||||
"issues": all_issues,
|
||||
}
|
||||
),
|
||||
)
|
||||
await dispose_rejected_pr(conn, pr_number, eval_attempts, all_issues)
|
||||
try:
|
||||
await on_eval_complete(
|
||||
conn,
|
||||
pr_number,
|
||||
outcome="rejected",
|
||||
review_text="\n\n".join(reviews.values()),
|
||||
issues=all_issues,
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("PR #%d: GitHub eval feedback failed (non-fatal)", pr_number)
|
||||
|
||||
if pr_cost > 0:
|
||||
conn.execute("UPDATE prs SET cost_usd = cost_usd + ? WHERE number = ?", (pr_cost, pr_number))
|
||||
|
||||
return {
|
||||
"pr": pr_number,
|
||||
"tier": tier,
|
||||
"domain": domain,
|
||||
"phase1b": True,
|
||||
"route": route.to_audit_dict(),
|
||||
"agent_verdicts": agent_verdicts,
|
||||
"approved": aggregate_approve,
|
||||
"leo_verdict": leo_verdict,
|
||||
"domain_verdict": domain_verdict,
|
||||
}
|
||||
|
||||
|
||||
async def evaluate_pr(conn, pr_number: int, tier: str = None) -> dict:
|
||||
"""Evaluate a single PR. Returns result dict."""
|
||||
from . import costs
|
||||
|
|
@ -431,19 +201,6 @@ async def evaluate_pr(conn, pr_number: int, tier: str = None) -> dict:
|
|||
(pr_number,),
|
||||
)
|
||||
|
||||
if config.PHASE1B_AGENT_ROUTING_ENABLED:
|
||||
return await _evaluate_pr_phase1b(
|
||||
conn,
|
||||
pr_number,
|
||||
tier=tier,
|
||||
diff=diff,
|
||||
review_diff=review_diff,
|
||||
files=files,
|
||||
branch_name=branch_name,
|
||||
eval_attempts=eval_attempts,
|
||||
pr_cost=pr_cost,
|
||||
)
|
||||
|
||||
# Check if domain review already completed (resuming after Leo rate limit)
|
||||
existing = conn.execute("SELECT domain_verdict, leo_verdict FROM prs WHERE number = ?", (pr_number,)).fetchone()
|
||||
existing_domain_verdict = existing["domain_verdict"] if existing else "pending"
|
||||
|
|
@ -786,7 +543,7 @@ async def _run_batch_domain_eval(
|
|||
"diff": review_diff,
|
||||
"files": files,
|
||||
"full_diff": diff, # kept for Leo review
|
||||
"file_count": len([line for line in files.split("\n") if line.strip()]),
|
||||
"file_count": len([l for l in files.split("\n") if l.strip()]),
|
||||
})
|
||||
claimed_prs.append(pr_num)
|
||||
|
||||
|
|
@ -824,7 +581,7 @@ async def _run_batch_domain_eval(
|
|||
"UPDATE prs SET domain = COALESCE(domain, ?), domain_agent = ? WHERE number IN ({})".format(
|
||||
",".join("?" * len(claimed_prs))
|
||||
),
|
||||
[domain, agent, *claimed_prs],
|
||||
[domain, agent] + claimed_prs,
|
||||
)
|
||||
|
||||
# Step 2: Run batch domain review
|
||||
|
|
@ -1102,12 +859,8 @@ async def evaluate_cycle(conn, max_workers=None) -> tuple[int, int]:
|
|||
succeeded = 0
|
||||
failed = 0
|
||||
|
||||
# Phase 1b routes per PR by identity and supports cross-domain top-2 review,
|
||||
# so stale DB-domain batching is disabled while the feature flag is on.
|
||||
if config.PHASE1B_AGENT_ROUTING_ENABLED:
|
||||
domain_batches, individual_prs = {}, list(rows)
|
||||
else:
|
||||
domain_batches, individual_prs = _build_domain_batches(rows, conn)
|
||||
# Group STANDARD PRs by domain for batch eval
|
||||
domain_batches, individual_prs = _build_domain_batches(rows, conn)
|
||||
|
||||
# Process batch domain reviews first
|
||||
for domain, batch_prs in domain_batches.items():
|
||||
|
|
|
|||
|
|
@ -32,7 +32,6 @@ from datetime import date
|
|||
from pathlib import Path
|
||||
|
||||
from . import config
|
||||
from .attribution import normalize_handle
|
||||
from .costs import record_usage
|
||||
from .db import classify_source_channel
|
||||
from .domains import agent_for_domain
|
||||
|
|
@ -684,25 +683,16 @@ async def _extract_one_source(
|
|||
logger.info("PR #%d created for %s (%d claims, %d entities)", pr_num, source_file, len(claim_files), len(entity_files))
|
||||
|
||||
# Store contributor attribution: who submitted this source?
|
||||
# Priority: proposed_by field → intake_tier inference → operator default.
|
||||
# NB: `submitted_by` is a CANONICAL HANDLE — lowercase, no @, no
|
||||
# trailing "(self-directed)" decorator. The "self-directed" signal is
|
||||
# already carried by intake_tier == "research-task" + the prs.agent
|
||||
# column; persisting it here as a string suffix produced decorated
|
||||
# values like "Vida (self-directed)" that broke /contributors/{handle}
|
||||
# lookups downstream (livingip-web timeline → 404). Read consumers
|
||||
# (lib/contributor.insert_contribution_event, scripts/scoring_digest,
|
||||
# diagnostics/activity_feed_api) all normalize via attribution.normalize_handle
|
||||
# anyway, so writing the canonical form is the source-of-truth fix.
|
||||
# Priority: proposed_by field → intake_tier inference → "unknown"
|
||||
if proposed_by:
|
||||
contributor = normalize_handle(proposed_by, conn=conn)
|
||||
contributor = proposed_by.strip().strip('"').strip("'")
|
||||
elif intake_tier == "research-task":
|
||||
contributor = normalize_handle(agent_name, conn=conn)
|
||||
contributor = f"{agent_name} (self-directed)"
|
||||
elif intake_tier == "directed":
|
||||
contributor = "m3taversal"
|
||||
contributor = "@m3taversal"
|
||||
else:
|
||||
# Default: if no proposed_by and not a research task, operator submitted it.
|
||||
contributor = "m3taversal"
|
||||
# Default: if no proposed_by and not a research task, Cory submitted it
|
||||
contributor = "@m3taversal"
|
||||
|
||||
# Build pipe-separated claim titles for the description field
|
||||
claim_titles = " | ".join(
|
||||
|
|
|
|||
64
lib/llm.py
64
lib/llm.py
|
|
@ -117,48 +117,6 @@ End your review with exactly one of:
|
|||
--- CHANGED FILES ---
|
||||
{files}"""
|
||||
|
||||
AGENT_REVIEW_PROMPT = """You are {agent}, a Hermes evaluator for TeleoHumanity's knowledge base.
|
||||
|
||||
You are reviewing this PR because the Phase 1b router assigned it to your agent identity.
|
||||
Route context:
|
||||
{route_context}
|
||||
|
||||
IMPORTANT — This PR may contain different content types:
|
||||
- **Claims** (type: claim): arguable assertions with confidence levels. Review fully.
|
||||
- **Entities** (type: entity, files in entities/): descriptive records of projects, people, protocols. Do NOT reject entities for missing confidence or source fields — they have a different schema.
|
||||
- **Sources** (files in inbox/): archive metadata. Auto-approve these.
|
||||
|
||||
Review this PR through your assigned identity. For EACH criterion below, write one sentence stating what you found:
|
||||
|
||||
1. **Domain ownership** — Is this change inside your area of responsibility? If not, still review the portion relevant to your routed responsibility.
|
||||
2. **Factual accuracy** — Are the claims/entities factually correct? Name any specific errors.
|
||||
3. **Confidence calibration** — For claims only. Is the confidence level right for the evidence?
|
||||
4. **System impact** — Does this change alter how agents, domains, or the collective understand goals, incentives, or operating assumptions?
|
||||
5. **Wiki links** — Note broken [[wiki links]], but do NOT let them affect your verdict. Broken links are expected.
|
||||
|
||||
VERDICT RULES:
|
||||
- APPROVE if claims are factually correct and evidence supports them.
|
||||
- APPROVE entity files unless they contain factual errors.
|
||||
- APPROVE even if wiki links are broken.
|
||||
- REQUEST_CHANGES only for blocking factual errors, duplicated evidence, clear confidence miscalibration, or a materially wrong domain/system implication.
|
||||
|
||||
{style_guide}
|
||||
|
||||
If requesting changes, tag the specific issues using ONLY these tags (do not invent new tags):
|
||||
<!-- ISSUES: tag1, tag2 -->
|
||||
|
||||
Valid tags: frontmatter_schema, title_overclaims, confidence_miscalibration, date_errors, factual_discrepancy, near_duplicate, scope_error
|
||||
|
||||
End your review with exactly one of:
|
||||
<!-- VERDICT:{agent_upper}:APPROVE -->
|
||||
<!-- VERDICT:{agent_upper}:REQUEST_CHANGES -->
|
||||
|
||||
--- PR DIFF ---
|
||||
{diff}
|
||||
|
||||
--- CHANGED FILES ---
|
||||
{files}"""
|
||||
|
||||
LEO_PROMPT_STANDARD = """You are Leo, the lead evaluator for TeleoHumanity's knowledge base.
|
||||
|
||||
IMPORTANT — Content types have DIFFERENT schemas:
|
||||
|
|
@ -462,28 +420,6 @@ async def run_domain_review(diff: str, files: str, domain: str, agent: str) -> t
|
|||
return result, usage
|
||||
|
||||
|
||||
async def run_agent_review(
|
||||
diff: str,
|
||||
files: str,
|
||||
agent: str,
|
||||
route_context: str = "",
|
||||
tier: str = "STANDARD",
|
||||
) -> tuple[str | None, dict]:
|
||||
"""Run a Phase 1b routed Hermes agent review via OpenRouter."""
|
||||
prompt = AGENT_REVIEW_PROMPT.format(
|
||||
agent=agent,
|
||||
agent_upper=agent.upper(),
|
||||
route_context=route_context or "(no route context)",
|
||||
style_guide=REVIEW_STYLE_GUIDE,
|
||||
diff=diff,
|
||||
files=files,
|
||||
)
|
||||
model = config.EVAL_LEO_STANDARD_MODEL if agent == "Leo" else config.EVAL_DOMAIN_MODEL
|
||||
timeout = config.EVAL_TIMEOUT_OPUS if tier == "DEEP" and agent == "Leo" else config.EVAL_TIMEOUT
|
||||
result, usage = await openrouter_call(model, prompt, timeout_sec=timeout)
|
||||
return result, usage
|
||||
|
||||
|
||||
async def run_leo_review(diff: str, files: str, tier: str) -> tuple[str | None, dict]:
|
||||
"""Run Leo review. DEEP → Opus (Claude Max, queue if limited). STANDARD → GPT-4o (OpenRouter).
|
||||
|
||||
|
|
|
|||
42
lib/merge.py
42
lib/merge.py
|
|
@ -112,44 +112,16 @@ async def discover_external_prs(conn) -> int:
|
|||
# Detect origin: pipeline agents have per-agent Forgejo users
|
||||
pipeline_users = {"teleo", "rio", "clay", "theseus", "vida", "astra", "leo"}
|
||||
author = pr.get("user", {}).get("login", "")
|
||||
branch_ref = pr["head"]["ref"]
|
||||
|
||||
# Pre-classify by branch prefix — pipeline-shape branches are
|
||||
# pipeline regardless of Forgejo opener. reweave.py and
|
||||
# ingestion run as the operator's token, so opener-based
|
||||
# classification mis-credited system maintenance to the
|
||||
# operator (~2.7k PRs on m3ta's contributor row before the
|
||||
# 2026-05-12 reattribute fix). Branch prefix is the canonical
|
||||
# signal: reweave/ingestion -> 'pipeline', <agent>/ -> agent.
|
||||
branch_target = None
|
||||
if branch_ref.startswith(("reweave/", "ingestion/")):
|
||||
branch_target = "pipeline"
|
||||
elif branch_ref.startswith(_AGENT_NAMES):
|
||||
# _AGENT_NAMES is a tuple of bare names; agent branches
|
||||
# are "<name>/..." so use the tuple as a startswith prefix
|
||||
# set after appending '/'.
|
||||
for name in _AGENT_NAMES:
|
||||
if branch_ref.startswith(name + "/"):
|
||||
branch_target = name
|
||||
break
|
||||
|
||||
is_pipeline = author.lower() in pipeline_users or branch_target is not None
|
||||
is_pipeline = author.lower() in pipeline_users
|
||||
origin = "pipeline" if is_pipeline else "human"
|
||||
priority = "high" if origin == "human" else None
|
||||
domain = None if not is_pipeline else detect_domain_from_branch(branch_ref)
|
||||
agent, commit_type = classify_branch(branch_ref)
|
||||
source_channel = classify_source_channel(branch_ref)
|
||||
domain = None if not is_pipeline else detect_domain_from_branch(pr["head"]["ref"])
|
||||
agent, commit_type = classify_branch(pr["head"]["ref"])
|
||||
source_channel = classify_source_channel(pr["head"]["ref"])
|
||||
|
||||
# submitted_by precedence (canonical handles only):
|
||||
# 1. branch prefix (pipeline/agent) — set here at discovery
|
||||
# 2. Forgejo opener for human PRs — set here, lowercased
|
||||
# 3. extract.py later (from source proposed_by) — left None
|
||||
if branch_target is not None:
|
||||
submitted_by = branch_target
|
||||
elif origin == "human":
|
||||
submitted_by = author.lower().lstrip("@") if author else None
|
||||
else:
|
||||
submitted_by = None
|
||||
# For human PRs, submitted_by is the Forgejo author.
|
||||
# For pipeline PRs, submitted_by is set later by extract.py (from source proposed_by).
|
||||
submitted_by = author if origin == "human" else None
|
||||
|
||||
conn.execute(
|
||||
"""INSERT OR IGNORE INTO prs
|
||||
|
|
|
|||
|
|
@ -19,6 +19,7 @@ Epimetheus owns this module. Leo reviews changes.
|
|||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from datetime import date, datetime
|
||||
from difflib import SequenceMatcher
|
||||
|
|
@ -66,9 +67,6 @@ def parse_frontmatter(text: str) -> tuple[dict | None, str]:
|
|||
fm = yaml.safe_load(raw)
|
||||
if not isinstance(fm, dict):
|
||||
return None, body
|
||||
for key, value in list(fm.items()):
|
||||
if isinstance(value, date | datetime):
|
||||
fm[key] = value.isoformat()
|
||||
return fm, body
|
||||
except ImportError:
|
||||
pass
|
||||
|
|
@ -144,13 +142,8 @@ def fix_frontmatter(content: str, domain: str, agent: str) -> tuple[str, list[st
|
|||
|
||||
# Fix 5: description field
|
||||
if "description" not in fm or not fm["description"]:
|
||||
# Try to derive from the first non-empty body line.
|
||||
first_sentence = ""
|
||||
for line in body.splitlines():
|
||||
first_sentence = line.strip().lstrip("# ")
|
||||
if first_sentence:
|
||||
first_sentence = first_sentence.split(".")[0].strip()
|
||||
break
|
||||
# Try to derive from body's first sentence
|
||||
first_sentence = body.split(".")[0].strip().lstrip("# ") if body else ""
|
||||
if first_sentence and len(first_sentence) > 10:
|
||||
fm["description"] = first_sentence[:200]
|
||||
fixes.append("derived_description_from_body")
|
||||
|
|
@ -436,7 +429,7 @@ def validate_and_fix_entities(
|
|||
issues = []
|
||||
|
||||
if action == "create" and content:
|
||||
fm, _body = parse_frontmatter(content)
|
||||
fm, body = parse_frontmatter(content)
|
||||
if fm is None:
|
||||
issues.append("no_frontmatter")
|
||||
else:
|
||||
|
|
|
|||
|
|
@ -522,55 +522,53 @@ async def substantive_fix_cycle(conn, max_workers=None) -> tuple[int, int]:
|
|||
Finds PRs with substantive issue tags that haven't exceeded fix budget.
|
||||
Processes up to 3 per cycle (Rhea: 180s interval, don't overwhelm eval).
|
||||
"""
|
||||
# Build the actionable-tag list from the routing constants so adding a new
|
||||
# tag to FIXABLE_TAGS / CONVERTIBLE_TAGS / UNFIXABLE_TAGS auto-updates the
|
||||
# SELECT filter — no two-place edit footgun.
|
||||
actionable_tags = sorted(FIXABLE_TAGS | CONVERTIBLE_TAGS | UNFIXABLE_TAGS)
|
||||
placeholders = ",".join(["?"] * len(actionable_tags))
|
||||
|
||||
# Push the actionable-tag filter into SQL (was a post-fetch Python loop).
|
||||
# The old shape selected the 3 oldest request_changes PRs and then dropped
|
||||
# ones without actionable tags, so empty-eval_issues rows occupied LIMIT-3
|
||||
# forever (head-of-line). Now LIMIT-3 always returns 3 actionable rows.
|
||||
# Reaper handles the empty-tag PRs after their 24h cooldown.
|
||||
rows = conn.execute(
|
||||
f"""SELECT number, eval_issues FROM prs
|
||||
"""SELECT number, eval_issues FROM prs
|
||||
WHERE status = 'open'
|
||||
AND tier0_pass = 1
|
||||
AND (domain_verdict = 'request_changes' OR leo_verdict = 'request_changes')
|
||||
AND COALESCE(fix_attempts, 0) < ?
|
||||
AND (last_attempt IS NULL OR last_attempt < datetime('now', '-3 minutes'))
|
||||
AND json_valid(eval_issues)
|
||||
AND EXISTS (
|
||||
SELECT 1 FROM json_each(eval_issues)
|
||||
WHERE value IN ({placeholders})
|
||||
)
|
||||
ORDER BY created_at ASC
|
||||
LIMIT 3""",
|
||||
(MAX_SUBSTANTIVE_FIXES + config.MAX_FIX_ATTEMPTS, *actionable_tags),
|
||||
(MAX_SUBSTANTIVE_FIXES + config.MAX_FIX_ATTEMPTS,), # Total budget: mechanical + substantive
|
||||
).fetchall()
|
||||
|
||||
if not rows:
|
||||
return 0, 0
|
||||
|
||||
# Defense-in-depth: json_valid(eval_issues) in the SELECT already filters
|
||||
# corrupt JSON before json_each runs, so this WARN should be unreachable.
|
||||
# Kept anyway: json_valid and json.loads use technically distinct parsers,
|
||||
# and the journal entry names the failure mode if SQLite ever surfaces a
|
||||
# row that passes json_valid + json_each but fails json.loads.
|
||||
# Filter to only PRs with substantive issues (not just mechanical)
|
||||
substantive_rows = []
|
||||
skipped_no_tags = []
|
||||
for row in rows:
|
||||
try:
|
||||
json.loads(row["eval_issues"] or "[]")
|
||||
issues = json.loads(row["eval_issues"] or "[]")
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
# Corrupt JSON in eval_issues is abnormal (post-merge column drift,
|
||||
# hand-edited row, partial write during crash). WARN so ops can chase
|
||||
# the upstream column-write path. Without this, the row drops out of
|
||||
# both substantive_rows and skipped_no_tags — the third silent path.
|
||||
logger.warning(
|
||||
"PR #%d: corrupt eval_issues JSON — skipping in substantive fix cycle",
|
||||
row["number"],
|
||||
)
|
||||
continue
|
||||
substantive_rows.append(row)
|
||||
if set(issues) & (FIXABLE_TAGS | CONVERTIBLE_TAGS | UNFIXABLE_TAGS):
|
||||
substantive_rows.append(row)
|
||||
else:
|
||||
skipped_no_tags.append((row["number"], issues))
|
||||
|
||||
if not substantive_rows:
|
||||
# Visibility for the LIMIT-3 head-of-line block: if the oldest
|
||||
# candidates have no fixer-actionable tags (e.g. eval_issues=[],
|
||||
# broken_wiki_links only), the cycle silently returns 0 — and the
|
||||
# next cycle picks the same head-of-line, forever. Log the eval_issues
|
||||
# of skipped candidates so the journal makes the block visible.
|
||||
if skipped_no_tags:
|
||||
logger.info(
|
||||
"Substantive fix cycle: 0 actionable from %d candidate(s) — head-of-line: %s",
|
||||
len(rows), skipped_no_tags,
|
||||
)
|
||||
return 0, 0
|
||||
|
||||
fixed = 0
|
||||
|
|
@ -598,191 +596,3 @@ async def substantive_fix_cycle(conn, max_workers=None) -> tuple[int, int]:
|
|||
logger.info("Substantive fix cycle: %d fixed, %d errors", fixed, errors)
|
||||
|
||||
return fixed, errors
|
||||
|
||||
|
||||
# ─── Verdict-deadlock reaper ──────────────────────────────────────────────
|
||||
#
|
||||
# Defense-in-depth for PRs that substantive_fixer can't make progress on.
|
||||
# Targets two stuck-verdict shapes empirically observed in production:
|
||||
#
|
||||
# 1. leo:request_changes + domain:approve
|
||||
# Leo asked for substantive fix; fixer either failed silently
|
||||
# (no_claim_files / no_review_comments / etc.) or the issue tag isn't
|
||||
# in FIXABLE | CONVERTIBLE | UNFIXABLE. PR sits forever.
|
||||
#
|
||||
# 2. leo:skipped + domain:request_changes
|
||||
# Eval bypassed Leo (eval_attempts >= MAX). Domain rejected with no
|
||||
# structured eval_issues. fixer can't classify → silent skip → forever.
|
||||
#
|
||||
# Both shapes need a clearance path. Reaper closes them after a 24h cooldown
|
||||
# with audit_log breadcrumbs for forensics. First deploy runs in dry-run mode
|
||||
# (audit "would_close" events only — no Forgejo writes, no DB closes).
|
||||
#
|
||||
# Reaper config (REAPER_DRY_RUN, REAPER_DEADLOCK_AGE_HOURS, REAPER_INTERVAL_SECONDS,
|
||||
# REAPER_MAX_PER_RUN) lives in lib/config.py with env-var overrides — operator
|
||||
# flips dry-run to live via `systemctl edit teleo-pipeline.service`
|
||||
# (Environment=REAPER_DRY_RUN=false) + restart. No code change, no commit, no
|
||||
# redeploy required.
|
||||
|
||||
|
||||
async def verdict_deadlock_reaper_cycle(conn) -> int:
|
||||
"""Reap PRs stuck in conflicting-verdict deadlock for >24h.
|
||||
|
||||
Returns count of PRs closed (or "would-close" in dry-run mode).
|
||||
Throttled to once per REAPER_INTERVAL_SECONDS via sentinel audit event.
|
||||
"""
|
||||
# Throttle: skip if last reaper run was within REAPER_INTERVAL_SECONDS.
|
||||
# Uses audit_log as the rate-limit ledger so no schema/state needed.
|
||||
# stage='reaper' filter so the planner uses idx_audit_stage (avoids full scan).
|
||||
last_run = conn.execute(
|
||||
"SELECT MAX(timestamp) FROM audit_log "
|
||||
"WHERE stage = 'reaper' AND event = 'verdict_deadlock_reaper_run'"
|
||||
).fetchone()[0]
|
||||
if last_run:
|
||||
cur = conn.execute(
|
||||
"SELECT (julianday('now') - julianday(?)) * 86400 < ?",
|
||||
(last_run, config.REAPER_INTERVAL_SECONDS),
|
||||
).fetchone()[0]
|
||||
if cur:
|
||||
return 0
|
||||
|
||||
# Two stuck-verdict shapes: leo:rc+domain:approve, leo:skipped+domain:rc.
|
||||
#
|
||||
# Branch allowlist invariant: the reaper closes ONLY disposable, pipeline-
|
||||
# generated branches — content the pipeline (or a daily cron) created and
|
||||
# can recreate. Four classes qualify:
|
||||
#
|
||||
# extract/* — per-source extraction PRs, regenerated next ingest cycle
|
||||
# reweave/* — nightly graph-edge maintenance, regenerated next reweave
|
||||
# fix/* — pipeline-internal fix branches
|
||||
# */research-YYYY-MM-DD — daily {agent}/research-{date} cron sessions.
|
||||
# Matched via SQLite `_` single-char wildcards as
|
||||
# `research-20__-__-__` to literally enforce the date-
|
||||
# suffix shape. Excludes hand-named research branches
|
||||
# (rio/research-batch-agents-memory-harnesses,
|
||||
# theseus/research-2nd-attempt-on-X, etc.) which are
|
||||
# feature work owned by the agent. Pattern good through
|
||||
# 2099; revisit then.
|
||||
#
|
||||
# WIP agent feature branches (theseus/feature-foo, epimetheus/some-fix,
|
||||
# rio/research-thesis-name) are NEVER reaped — owners review their own PRs
|
||||
# on their own cadence. The date-shaped pattern threads the needle: picks
|
||||
# up daily synthesis output the agent regenerates tomorrow while leaving
|
||||
# manually-named research work alone.
|
||||
rows = conn.execute(
|
||||
"""SELECT number, branch, eval_issues, leo_verdict, domain_verdict,
|
||||
last_attempt, fix_attempts
|
||||
FROM prs
|
||||
WHERE status = 'open'
|
||||
AND tier0_pass = 1
|
||||
AND last_attempt IS NOT NULL
|
||||
AND last_attempt < datetime('now', ? || ' hours')
|
||||
AND (branch LIKE 'extract/%'
|
||||
OR branch LIKE 'reweave/%'
|
||||
OR branch LIKE 'fix/%'
|
||||
OR branch LIKE '%/research-20__-__-__')
|
||||
AND (
|
||||
(leo_verdict = 'request_changes' AND domain_verdict = 'approve')
|
||||
OR (leo_verdict = 'skipped' AND domain_verdict = 'request_changes')
|
||||
)
|
||||
ORDER BY last_attempt ASC
|
||||
LIMIT ?""",
|
||||
(f"-{config.REAPER_DEADLOCK_AGE_HOURS}", config.REAPER_MAX_PER_RUN),
|
||||
).fetchall()
|
||||
|
||||
mode = "dryrun" if config.REAPER_DRY_RUN else "live"
|
||||
|
||||
if not rows:
|
||||
# Heartbeat anyway so throttle ticks even when nothing to reap.
|
||||
db.audit(conn, "reaper", "verdict_deadlock_reaper_run", json.dumps({
|
||||
"candidates": 0, "closed": 0, "mode": mode,
|
||||
}))
|
||||
return 0
|
||||
|
||||
logger.info(
|
||||
"Verdict-deadlock reaper [%s]: %d candidate(s) in deadlock >%dh",
|
||||
mode, len(rows), config.REAPER_DEADLOCK_AGE_HOURS,
|
||||
)
|
||||
|
||||
closed = 0
|
||||
would_close = 0
|
||||
errors = 0
|
||||
for row in rows:
|
||||
pr = row["number"]
|
||||
reason_detail = {
|
||||
"pr": pr,
|
||||
"branch": row["branch"],
|
||||
"leo_verdict": row["leo_verdict"],
|
||||
"domain_verdict": row["domain_verdict"],
|
||||
"eval_issues": row["eval_issues"],
|
||||
"last_attempt": row["last_attempt"],
|
||||
"fix_attempts": row["fix_attempts"],
|
||||
}
|
||||
|
||||
if config.REAPER_DRY_RUN:
|
||||
# Audit only — do NOT touch DB row or Forgejo state.
|
||||
db.audit(conn, "reaper", "verdict_deadlock_would_close",
|
||||
json.dumps(reason_detail))
|
||||
logger.info(
|
||||
"Reaper [dryrun]: would close PR #%d (leo=%s domain=%s issues=%s)",
|
||||
pr, row["leo_verdict"], row["domain_verdict"], row["eval_issues"],
|
||||
)
|
||||
would_close += 1
|
||||
continue
|
||||
|
||||
try:
|
||||
comment_body = (
|
||||
"Closed by verdict-deadlock reaper.\n\n"
|
||||
f"This PR sat for >{config.REAPER_DEADLOCK_AGE_HOURS}h with conflicting "
|
||||
f"verdicts (leo={row['leo_verdict']}, domain={row['domain_verdict']}) "
|
||||
f"that the substantive fixer couldn't auto-resolve.\n\n"
|
||||
f"Eval issues: `{row['eval_issues']}`\n"
|
||||
f"Last attempt: {row['last_attempt']}\n\n"
|
||||
"_Automated message from the LivingIP pipeline._"
|
||||
)
|
||||
await forgejo_api(
|
||||
"POST", repo_path(f"issues/{pr}/comments"), {"body": comment_body},
|
||||
)
|
||||
patch_result = await forgejo_api(
|
||||
"PATCH", repo_path(f"pulls/{pr}"), {"state": "closed"},
|
||||
token=get_agent_token("leo"),
|
||||
)
|
||||
if patch_result is None:
|
||||
logger.warning(
|
||||
"Reaper: PR #%d Forgejo close failed — skipping DB close to "
|
||||
"avoid drift", pr,
|
||||
)
|
||||
errors += 1
|
||||
continue
|
||||
# Forgejo already closed at the PATCH above — pass close_on_forgejo=False
|
||||
# so close_pr() doesn't issue a redundant PATCH (which on transient
|
||||
# failure returns False and skips the DB close → status drift).
|
||||
await close_pr(
|
||||
conn, pr,
|
||||
last_error=(
|
||||
f"verdict_deadlock_reaper: leo={row['leo_verdict']} "
|
||||
f"domain={row['domain_verdict']} age>{config.REAPER_DEADLOCK_AGE_HOURS}h"
|
||||
),
|
||||
close_on_forgejo=False,
|
||||
)
|
||||
db.audit(conn, "reaper", "verdict_deadlock_closed",
|
||||
json.dumps(reason_detail))
|
||||
closed += 1
|
||||
except Exception:
|
||||
logger.exception("Reaper: PR #%d close failed", pr)
|
||||
errors += 1
|
||||
|
||||
db.audit(conn, "reaper", "verdict_deadlock_reaper_run", json.dumps({
|
||||
"candidates": len(rows), "closed": closed, "would_close": would_close,
|
||||
"errors": errors, "mode": mode,
|
||||
}))
|
||||
if errors:
|
||||
logger.warning(
|
||||
"Verdict-deadlock reaper [%s]: %d closed, %d would-close, %d errors",
|
||||
mode, closed, would_close, errors,
|
||||
)
|
||||
elif config.REAPER_DRY_RUN:
|
||||
logger.info("Verdict-deadlock reaper [dryrun]: %d would-close", would_close)
|
||||
else:
|
||||
logger.info("Verdict-deadlock reaper [live]: %d closed", closed)
|
||||
return closed + would_close
|
||||
|
|
|
|||
|
|
@ -1,458 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Pipeline health metrics — Forgejo API → stage transitions → throughput → JSON.
|
||||
|
||||
Implements Vida's pipeline diagnostics spec (2026-03-11).
|
||||
Runs on VPS, outputs to /opt/teleo-eval/metrics/pipeline-YYYY-MM-DD.json
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import statistics
|
||||
import sys
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
|
||||
|
||||
BASE_URL = "https://api.github.com/repos/living-ip/decision-engine"
|
||||
TOKEN_FILE = "/opt/teleo-eval/secrets/github-admin-token"
|
||||
VERDICT_RE = re.compile(r'<!-- VERDICT:(\w+):(APPROVE|REQUEST_CHANGES) -->')
|
||||
OUTPUT_DIR = "/opt/teleo-eval/metrics"
|
||||
|
||||
|
||||
def api_get(path, token, page=1, per_page=50):
|
||||
"""GET from GitHub REST API."""
|
||||
sep = "&" if "?" in path else "?"
|
||||
url = f"{BASE_URL}{path}{sep}page={page}&per_page={per_page}"
|
||||
req = urllib.request.Request(url, headers={
|
||||
"Authorization": f"Bearer {token}",
|
||||
"Accept": "application/vnd.github+json",
|
||||
"X-GitHub-Api-Version": "2022-11-28",
|
||||
})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||
return json.loads(resp.read())
|
||||
except urllib.error.HTTPError as e:
|
||||
print(f"API error {e.code}: {path}", file=sys.stderr)
|
||||
return [] if "pulls" in path or "comments" in path or "commits" in path else {}
|
||||
|
||||
|
||||
def get_all_pulls(token, state="all", since=None):
|
||||
"""Paginate through all PRs."""
|
||||
all_prs = []
|
||||
page = 1
|
||||
while True:
|
||||
# GitHub: per_page (not limit) is added by api_get; sort=created&direction=desc supported.
|
||||
path = f"/pulls?state={state}&sort=created&direction=desc"
|
||||
prs = api_get(path, token, page=page)
|
||||
if not prs:
|
||||
break
|
||||
all_prs.extend(prs)
|
||||
# Stop paginating if we've gone past our time window
|
||||
if since and prs:
|
||||
oldest = parse_ts(prs[-1].get("created_at", ""))
|
||||
if oldest and oldest < since - timedelta(days=7):
|
||||
break
|
||||
if len(prs) < 50: # GitHub Link-header pagination would be cleaner; len-check is sufficient here
|
||||
break
|
||||
page += 1
|
||||
return all_prs
|
||||
|
||||
|
||||
def get_comments(token, pr_number):
|
||||
"""Get all comments on a PR."""
|
||||
return api_get(f"/issues/{pr_number}/comments", token)
|
||||
|
||||
|
||||
def get_commits(token, pr_number):
|
||||
"""Get all commits on a PR."""
|
||||
return api_get(f"/pulls/{pr_number}/commits", token)
|
||||
|
||||
|
||||
def parse_ts(ts_str):
|
||||
"""Parse ISO timestamp to datetime."""
|
||||
if not ts_str:
|
||||
return None
|
||||
try:
|
||||
# Handle various formats
|
||||
ts_str = ts_str.replace("Z", "+00:00")
|
||||
return datetime.fromisoformat(ts_str)
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def hours_between(start, end):
|
||||
"""Hours between two datetimes."""
|
||||
if not start or not end:
|
||||
return None
|
||||
delta = (end - start).total_seconds() / 3600
|
||||
return round(delta, 2)
|
||||
|
||||
|
||||
def parse_verdicts(comments):
|
||||
"""Extract verdict events from PR comments, sorted by time."""
|
||||
verdicts = []
|
||||
for c in comments:
|
||||
body = c.get("body", "")
|
||||
ts = parse_ts(c.get("created_at"))
|
||||
for match in VERDICT_RE.finditer(body):
|
||||
verdicts.append({
|
||||
"reviewer": match.group(1),
|
||||
"verdict": match.group(2),
|
||||
"ts": ts,
|
||||
"user": c.get("user", {}).get("login", "unknown"),
|
||||
})
|
||||
verdicts.sort(key=lambda v: v["ts"] if v["ts"] else datetime.min.replace(tzinfo=timezone.utc))
|
||||
return verdicts
|
||||
|
||||
|
||||
def detect_agent(pr):
|
||||
"""Detect proposing agent from branch name."""
|
||||
ref = pr.get("head", {}).get("ref", "")
|
||||
if "/" in ref:
|
||||
prefix = ref.split("/")[0]
|
||||
if prefix in ("rio", "clay", "theseus", "vida", "astra", "leo", "extract", "auto-fix"):
|
||||
return prefix if prefix not in ("extract", "auto-fix") else "pipeline"
|
||||
return "unknown"
|
||||
|
||||
|
||||
def compute_stage_durations(pr, verdicts, commits):
|
||||
"""Compute wait times for each stage of a PR."""
|
||||
created = parse_ts(pr.get("created_at"))
|
||||
merged = parse_ts(pr.get("merged_at"))
|
||||
|
||||
result = {
|
||||
"review_wait_hrs": None,
|
||||
"remediation_cycles": [],
|
||||
"merge_wait_hrs": None,
|
||||
}
|
||||
|
||||
if not verdicts:
|
||||
# No review yet — still in stage 1
|
||||
return result
|
||||
|
||||
# Stage 1: created → first verdict
|
||||
first_verdict = verdicts[0]
|
||||
result["review_wait_hrs"] = hours_between(created, first_verdict["ts"])
|
||||
|
||||
# Stage 2: REQUEST_CHANGES → next push (may repeat)
|
||||
commit_times = sorted([
|
||||
parse_ts(c.get("created", c.get("commit", {}).get("author", {}).get("date")))
|
||||
for c in commits
|
||||
if parse_ts(c.get("created", c.get("commit", {}).get("author", {}).get("date")))
|
||||
])
|
||||
|
||||
for v in verdicts:
|
||||
if v["verdict"] == "REQUEST_CHANGES" and v["ts"]:
|
||||
# Find first commit after this verdict
|
||||
next_push = None
|
||||
for ct in commit_times:
|
||||
if ct > v["ts"]:
|
||||
next_push = ct
|
||||
break
|
||||
cycle_hrs = hours_between(v["ts"], next_push)
|
||||
result["remediation_cycles"].append(cycle_hrs)
|
||||
|
||||
# Stage 3: last APPROVE → merged
|
||||
approve_verdicts = [v for v in verdicts if v["verdict"] == "APPROVE"]
|
||||
if approve_verdicts:
|
||||
last_approve = approve_verdicts[-1]
|
||||
if merged:
|
||||
result["merge_wait_hrs"] = hours_between(last_approve["ts"], merged)
|
||||
else:
|
||||
result["merge_wait_hrs"] = "in_flight"
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def classify_pr_stage(pr, verdicts, commits):
|
||||
"""Classify which queue a PR is currently in."""
|
||||
if pr.get("merged_at"):
|
||||
return "merged"
|
||||
if pr.get("state") == "closed":
|
||||
return "closed"
|
||||
|
||||
if not verdicts:
|
||||
return "awaiting_review"
|
||||
|
||||
# Build per-reviewer latest verdict
|
||||
reviewer_verdicts = {}
|
||||
for v in verdicts:
|
||||
reviewer_verdicts[v["reviewer"]] = v
|
||||
|
||||
# Any outstanding REQUEST_CHANGES blocks merge
|
||||
has_request_changes = any(
|
||||
v["verdict"] == "REQUEST_CHANGES" for v in reviewer_verdicts.values()
|
||||
)
|
||||
has_any_approve = any(
|
||||
v["verdict"] == "APPROVE" for v in reviewer_verdicts.values()
|
||||
)
|
||||
|
||||
if has_request_changes:
|
||||
# Find the latest REQUEST_CHANGES
|
||||
rc_verdicts = [v for v in verdicts if v["verdict"] == "REQUEST_CHANGES"]
|
||||
latest_rc = rc_verdicts[-1]
|
||||
# Check if there's been a push after it
|
||||
commit_times = [
|
||||
parse_ts(c.get("created", c.get("commit", {}).get("author", {}).get("date")))
|
||||
for c in commits
|
||||
]
|
||||
has_fix = any(ct and ct > latest_rc["ts"] for ct in commit_times if ct)
|
||||
if has_fix:
|
||||
return "awaiting_review" # Fixed, back in review queue
|
||||
return "awaiting_remediation"
|
||||
|
||||
if has_any_approve and not has_request_changes:
|
||||
# All verdicts are APPROVE — awaiting merge
|
||||
return "awaiting_merge"
|
||||
|
||||
return "awaiting_review"
|
||||
|
||||
|
||||
def percentile(data, p):
|
||||
"""Compute percentile of a list."""
|
||||
if not data:
|
||||
return None
|
||||
sorted_data = sorted(data)
|
||||
k = (len(sorted_data) - 1) * (p / 100)
|
||||
f = int(k)
|
||||
c = f + 1 if f + 1 < len(sorted_data) else f
|
||||
d = k - f
|
||||
return round(sorted_data[f] + d * (sorted_data[c] - sorted_data[f]), 2)
|
||||
|
||||
|
||||
def compute_throughput(prs_with_data, window_start, window_end):
|
||||
"""Compute per-hour throughput rates within the time window."""
|
||||
hours = max((window_end - window_start).total_seconds() / 3600, 1)
|
||||
|
||||
extraction_count = 0
|
||||
eval_count = 0
|
||||
feedback_count = 0
|
||||
merge_count = 0
|
||||
|
||||
for item in prs_with_data:
|
||||
pr = item["pr"]
|
||||
verdicts = item["verdicts"]
|
||||
commits = item["commits"]
|
||||
|
||||
created = parse_ts(pr.get("created_at"))
|
||||
if created and window_start <= created <= window_end:
|
||||
extraction_count += 1
|
||||
|
||||
for v in verdicts:
|
||||
if v["ts"] and window_start <= v["ts"] <= window_end:
|
||||
eval_count += 1
|
||||
|
||||
merged = parse_ts(pr.get("merged_at"))
|
||||
if merged and window_start <= merged <= window_end:
|
||||
merge_count += 1
|
||||
|
||||
# Count feedback pushes (commits after REQUEST_CHANGES within window)
|
||||
rc_times = [v["ts"] for v in verdicts if v["verdict"] == "REQUEST_CHANGES" and v["ts"]]
|
||||
commit_times = sorted([
|
||||
parse_ts(c.get("created", c.get("commit", {}).get("author", {}).get("date")))
|
||||
for c in commits
|
||||
if parse_ts(c.get("created", c.get("commit", {}).get("author", {}).get("date")))
|
||||
])
|
||||
for rc_ts in rc_times:
|
||||
for ct in commit_times:
|
||||
if ct and ct > rc_ts and window_start <= ct <= window_end:
|
||||
feedback_count += 1
|
||||
break
|
||||
|
||||
ext_rate = round(extraction_count / hours, 2)
|
||||
eval_rate = round(eval_count / hours, 2)
|
||||
fb_rate = round(feedback_count / hours, 2)
|
||||
merge_rate = round(merge_count / hours, 2)
|
||||
|
||||
# Bottleneck: lowest throughput channel where upstream is higher
|
||||
channels = {"extraction": ext_rate, "eval": eval_rate, "feedback": fb_rate, "merge": merge_rate}
|
||||
bottleneck = "none"
|
||||
# Simple: if extraction > eval, bottleneck is eval. Walk the chain.
|
||||
if ext_rate > eval_rate and eval_rate > 0:
|
||||
bottleneck = "eval"
|
||||
elif eval_rate > fb_rate and fb_rate > 0:
|
||||
bottleneck = "feedback"
|
||||
elif fb_rate > merge_rate and merge_rate > 0:
|
||||
bottleneck = "merge"
|
||||
elif ext_rate > 0 and eval_rate == 0:
|
||||
bottleneck = "eval"
|
||||
|
||||
return {
|
||||
"extraction_per_hr": ext_rate,
|
||||
"eval_per_hr": eval_rate,
|
||||
"feedback_per_hr": fb_rate,
|
||||
"merge_per_hr": merge_rate,
|
||||
"bottleneck": bottleneck,
|
||||
"queue_growth_rate": round(ext_rate - merge_rate, 2),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Pipeline health metrics")
|
||||
parser.add_argument("--hours", type=int, default=24, help="Time window in hours (default: 24)")
|
||||
parser.add_argument("--output", help="Output file path (default: /opt/teleo-eval/metrics/pipeline-YYYY-MM-DD.json)")
|
||||
parser.add_argument("--max-prs", type=int, default=200, help="Max PRs to analyze")
|
||||
args = parser.parse_args()
|
||||
|
||||
# Read token
|
||||
token = open(TOKEN_FILE).read().strip()
|
||||
now = datetime.now(timezone.utc)
|
||||
window_start = now - timedelta(hours=args.hours)
|
||||
window_end = now
|
||||
|
||||
print(f"Pipeline health check: {window_start.isoformat()} → {window_end.isoformat()}")
|
||||
|
||||
# Fetch open PRs
|
||||
print("Fetching open PRs...", file=sys.stderr)
|
||||
open_prs = get_all_pulls(token, state="open", since=window_start)
|
||||
print(f" {len(open_prs)} open PRs fetched", file=sys.stderr)
|
||||
|
||||
# Fetch recently closed/merged PRs (sort by updated to get recent merges)
|
||||
print("Fetching recently closed PRs...", file=sys.stderr)
|
||||
closed_prs = api_get("/pulls?state=closed&sort=updated&direction=desc", token)
|
||||
# Filter to those merged/closed within our window
|
||||
recent_closed = [p for p in closed_prs if
|
||||
(parse_ts(p.get("merged_at", "")) and parse_ts(p["merged_at"]) >= window_start) or
|
||||
(parse_ts(p.get("closed_at", "")) and parse_ts(p["closed_at"]) >= window_start)]
|
||||
print(f" {len(recent_closed)} recently closed/merged PRs", file=sys.stderr)
|
||||
|
||||
# Combine: all open + recently closed, deduplicate
|
||||
all_prs = open_prs + closed_prs
|
||||
analyze_prs = list({p["number"]: p for p in open_prs + recent_closed}.values())
|
||||
analyze_prs = analyze_prs[:args.max_prs]
|
||||
print(f" Analyzing {len(analyze_prs)} PRs ({len(open_prs)} open, {len(recent_closed)} recently merged/closed)", file=sys.stderr)
|
||||
|
||||
# Fetch comments + commits for each PR
|
||||
prs_with_data = []
|
||||
for i, pr in enumerate(analyze_prs):
|
||||
num = pr["number"]
|
||||
if (i + 1) % 20 == 0:
|
||||
print(f" Processing PR {i+1}/{len(analyze_prs)}...", file=sys.stderr)
|
||||
comments = get_comments(token, num)
|
||||
commits = get_commits(token, num)
|
||||
verdicts = parse_verdicts(comments)
|
||||
stage = classify_pr_stage(pr, verdicts, commits)
|
||||
durations = compute_stage_durations(pr, verdicts, commits)
|
||||
|
||||
prs_with_data.append({
|
||||
"pr": pr,
|
||||
"verdicts": verdicts,
|
||||
"commits": commits,
|
||||
"stage": stage,
|
||||
"durations": durations,
|
||||
})
|
||||
|
||||
# Compute throughput
|
||||
throughput = compute_throughput(prs_with_data, window_start, window_end)
|
||||
|
||||
# Compute wait times
|
||||
review_waits = [d["durations"]["review_wait_hrs"] for d in prs_with_data
|
||||
if d["durations"]["review_wait_hrs"] is not None]
|
||||
remediation_waits = [c for d in prs_with_data
|
||||
for c in d["durations"]["remediation_cycles"]
|
||||
if c is not None]
|
||||
merge_waits = [d["durations"]["merge_wait_hrs"] for d in prs_with_data
|
||||
if d["durations"]["merge_wait_hrs"] is not None
|
||||
and d["durations"]["merge_wait_hrs"] != "in_flight"]
|
||||
|
||||
wait_times = {
|
||||
"review": {
|
||||
"median_hrs": round(statistics.median(review_waits), 2) if review_waits else None,
|
||||
"p90_hrs": percentile(review_waits, 90),
|
||||
"max_hrs": round(max(review_waits), 2) if review_waits else None,
|
||||
"n": len(review_waits),
|
||||
},
|
||||
"remediation": {
|
||||
"median_hrs": round(statistics.median(remediation_waits), 2) if remediation_waits else None,
|
||||
"p90_hrs": percentile(remediation_waits, 90),
|
||||
"max_hrs": round(max(remediation_waits), 2) if remediation_waits else None,
|
||||
"n": len(remediation_waits),
|
||||
},
|
||||
"merge": {
|
||||
"median_hrs": round(statistics.median(merge_waits), 2) if merge_waits else None,
|
||||
"p90_hrs": percentile(merge_waits, 90),
|
||||
"max_hrs": round(max(merge_waits), 2) if merge_waits else None,
|
||||
"n": len(merge_waits),
|
||||
},
|
||||
}
|
||||
|
||||
# Queue snapshot
|
||||
queue = {
|
||||
"awaiting_review": sum(1 for d in prs_with_data if d["stage"] == "awaiting_review"),
|
||||
"awaiting_remediation": sum(1 for d in prs_with_data if d["stage"] == "awaiting_remediation"),
|
||||
"awaiting_merge": sum(1 for d in prs_with_data if d["stage"] == "awaiting_merge"),
|
||||
"total_open": len(open_prs),
|
||||
}
|
||||
|
||||
# Per-PR detail
|
||||
per_pr = []
|
||||
for d in prs_with_data:
|
||||
pr = d["pr"]
|
||||
if pr.get("state") != "open":
|
||||
continue
|
||||
per_pr.append({
|
||||
"number": pr["number"],
|
||||
"title": pr.get("title", "")[:100],
|
||||
"branch": pr.get("head", {}).get("ref", ""),
|
||||
"agent": detect_agent(pr),
|
||||
"current_stage": d["stage"],
|
||||
"created_at": pr.get("created_at"),
|
||||
"stage_durations": d["durations"],
|
||||
})
|
||||
|
||||
# Sort per_pr by longest wait first
|
||||
per_pr.sort(key=lambda p: p.get("stage_durations", {}).get("review_wait_hrs") or 0, reverse=True)
|
||||
|
||||
# Build output
|
||||
output = {
|
||||
"generated": now.isoformat(),
|
||||
"window": {
|
||||
"start": window_start.isoformat(),
|
||||
"end": window_end.isoformat(),
|
||||
"hours": args.hours,
|
||||
},
|
||||
"throughput": throughput,
|
||||
"wait_times": wait_times,
|
||||
"queue_snapshot": queue,
|
||||
"per_pr": per_pr,
|
||||
}
|
||||
|
||||
# Write output
|
||||
os.makedirs(OUTPUT_DIR, exist_ok=True)
|
||||
output_path = args.output or os.path.join(OUTPUT_DIR, f"pipeline-{now.strftime('%Y-%m-%d')}.json")
|
||||
with open(output_path, "w") as f:
|
||||
json.dump(output, f, indent=2, default=str)
|
||||
|
||||
# Print summary
|
||||
print(f"\n{'='*60}")
|
||||
print(f" PIPELINE HEALTH — {now.strftime('%Y-%m-%d %H:%M')} UTC")
|
||||
print(f" Window: {args.hours}h")
|
||||
print(f"{'='*60}")
|
||||
print(f" Throughput (per hour):")
|
||||
print(f" Extraction: {throughput['extraction_per_hr']}")
|
||||
print(f" Eval: {throughput['eval_per_hr']}")
|
||||
print(f" Feedback: {throughput['feedback_per_hr']}")
|
||||
print(f" Merge: {throughput['merge_per_hr']}")
|
||||
print(f" Bottleneck: {throughput['bottleneck']}")
|
||||
print(f" Queue growth: {throughput['queue_growth_rate']}/hr")
|
||||
print(f" Wait times (hours):")
|
||||
print(f" Review: median={wait_times['review']['median_hrs']} p90={wait_times['review']['p90_hrs']} n={wait_times['review']['n']}")
|
||||
print(f" Remediation: median={wait_times['remediation']['median_hrs']} p90={wait_times['remediation']['p90_hrs']} n={wait_times['remediation']['n']}")
|
||||
print(f" Merge: median={wait_times['merge']['median_hrs']} p90={wait_times['merge']['p90_hrs']} n={wait_times['merge']['n']}")
|
||||
print(f" Queue:")
|
||||
print(f" Awaiting review: {queue['awaiting_review']}")
|
||||
print(f" Awaiting remediation: {queue['awaiting_remediation']}")
|
||||
print(f" Awaiting merge: {queue['awaiting_merge']}")
|
||||
print(f" Total open: {queue['total_open']}")
|
||||
print(f"{'='*60}")
|
||||
print(f" Output: {output_path}")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -1,930 +0,0 @@
|
|||
{
|
||||
"agent_review_calls": [
|
||||
{
|
||||
"agent": "Leo",
|
||||
"files": [
|
||||
"domains/grand-strategy/strategy.md"
|
||||
],
|
||||
"route": {
|
||||
"evidence": [
|
||||
{
|
||||
"agent": "Leo",
|
||||
"signal": "path",
|
||||
"value": "domains/grand-strategy/strategy.md",
|
||||
"weight": 8
|
||||
}
|
||||
],
|
||||
"fallback": false,
|
||||
"primary_agent": "Leo",
|
||||
"required_agents": [
|
||||
"Leo"
|
||||
],
|
||||
"route_kind": "single",
|
||||
"scores": {
|
||||
"Astra": 0,
|
||||
"Clay": 0,
|
||||
"Leo": 8,
|
||||
"Rio": 0,
|
||||
"Theseus": 0,
|
||||
"Vida": 0
|
||||
},
|
||||
"touched_domains": [
|
||||
"grand-strategy"
|
||||
]
|
||||
},
|
||||
"tier": "STANDARD",
|
||||
"verdict": "APPROVE"
|
||||
},
|
||||
{
|
||||
"agent": "Theseus",
|
||||
"files": [
|
||||
"domains/ai-alignment/systems.md"
|
||||
],
|
||||
"route": {
|
||||
"evidence": [
|
||||
{
|
||||
"agent": "Theseus",
|
||||
"signal": "path",
|
||||
"value": "domains/ai-alignment/systems.md",
|
||||
"weight": 8
|
||||
}
|
||||
],
|
||||
"fallback": false,
|
||||
"primary_agent": "Theseus",
|
||||
"required_agents": [
|
||||
"Theseus"
|
||||
],
|
||||
"route_kind": "single",
|
||||
"scores": {
|
||||
"Astra": 0,
|
||||
"Clay": 0,
|
||||
"Leo": 0,
|
||||
"Rio": 0,
|
||||
"Theseus": 8,
|
||||
"Vida": 0
|
||||
},
|
||||
"touched_domains": [
|
||||
"ai-alignment"
|
||||
]
|
||||
},
|
||||
"tier": "STANDARD",
|
||||
"verdict": "APPROVE"
|
||||
},
|
||||
{
|
||||
"agent": "Rio",
|
||||
"files": [
|
||||
"domains/internet-finance/x402.md"
|
||||
],
|
||||
"route": {
|
||||
"evidence": [
|
||||
{
|
||||
"agent": "Rio",
|
||||
"signal": "path",
|
||||
"value": "domains/internet-finance/x402.md",
|
||||
"weight": 8
|
||||
},
|
||||
{
|
||||
"agent": "Rio",
|
||||
"signal": "keyword",
|
||||
"value": "x402",
|
||||
"weight": 2
|
||||
}
|
||||
],
|
||||
"fallback": false,
|
||||
"primary_agent": "Rio",
|
||||
"required_agents": [
|
||||
"Rio"
|
||||
],
|
||||
"route_kind": "single",
|
||||
"scores": {
|
||||
"Astra": 0,
|
||||
"Clay": 0,
|
||||
"Leo": 0,
|
||||
"Rio": 10,
|
||||
"Theseus": 0,
|
||||
"Vida": 0
|
||||
},
|
||||
"touched_domains": [
|
||||
"internet-finance"
|
||||
]
|
||||
},
|
||||
"tier": "STANDARD",
|
||||
"verdict": "APPROVE"
|
||||
},
|
||||
{
|
||||
"agent": "Vida",
|
||||
"files": [
|
||||
"domains/health/clinical.md"
|
||||
],
|
||||
"route": {
|
||||
"evidence": [
|
||||
{
|
||||
"agent": "Vida",
|
||||
"signal": "path",
|
||||
"value": "domains/health/clinical.md",
|
||||
"weight": 8
|
||||
},
|
||||
{
|
||||
"agent": "Vida",
|
||||
"signal": "keyword",
|
||||
"value": "health",
|
||||
"weight": 2
|
||||
},
|
||||
{
|
||||
"agent": "Vida",
|
||||
"signal": "keyword",
|
||||
"value": "clinical",
|
||||
"weight": 2
|
||||
}
|
||||
],
|
||||
"fallback": false,
|
||||
"primary_agent": "Vida",
|
||||
"required_agents": [
|
||||
"Vida"
|
||||
],
|
||||
"route_kind": "single",
|
||||
"scores": {
|
||||
"Astra": 0,
|
||||
"Clay": 0,
|
||||
"Leo": 0,
|
||||
"Rio": 0,
|
||||
"Theseus": 0,
|
||||
"Vida": 12
|
||||
},
|
||||
"touched_domains": [
|
||||
"health"
|
||||
]
|
||||
},
|
||||
"tier": "STANDARD",
|
||||
"verdict": "APPROVE"
|
||||
},
|
||||
{
|
||||
"agent": "Clay",
|
||||
"files": [
|
||||
"domains/entertainment/games.md"
|
||||
],
|
||||
"route": {
|
||||
"evidence": [
|
||||
{
|
||||
"agent": "Clay",
|
||||
"signal": "path",
|
||||
"value": "domains/entertainment/games.md",
|
||||
"weight": 8
|
||||
},
|
||||
{
|
||||
"agent": "Clay",
|
||||
"signal": "keyword",
|
||||
"value": "entertainment",
|
||||
"weight": 2
|
||||
},
|
||||
{
|
||||
"agent": "Clay",
|
||||
"signal": "keyword",
|
||||
"value": "games",
|
||||
"weight": 2
|
||||
}
|
||||
],
|
||||
"fallback": false,
|
||||
"primary_agent": "Clay",
|
||||
"required_agents": [
|
||||
"Clay"
|
||||
],
|
||||
"route_kind": "single",
|
||||
"scores": {
|
||||
"Astra": 0,
|
||||
"Clay": 12,
|
||||
"Leo": 0,
|
||||
"Rio": 0,
|
||||
"Theseus": 0,
|
||||
"Vida": 0
|
||||
},
|
||||
"touched_domains": [
|
||||
"entertainment"
|
||||
]
|
||||
},
|
||||
"tier": "STANDARD",
|
||||
"verdict": "APPROVE"
|
||||
},
|
||||
{
|
||||
"agent": "Astra",
|
||||
"files": [
|
||||
"domains/space-development/robotics.md"
|
||||
],
|
||||
"route": {
|
||||
"evidence": [
|
||||
{
|
||||
"agent": "Astra",
|
||||
"signal": "path",
|
||||
"value": "domains/space-development/robotics.md",
|
||||
"weight": 8
|
||||
},
|
||||
{
|
||||
"agent": "Astra",
|
||||
"signal": "keyword",
|
||||
"value": "space",
|
||||
"weight": 2
|
||||
},
|
||||
{
|
||||
"agent": "Astra",
|
||||
"signal": "keyword",
|
||||
"value": "robotics",
|
||||
"weight": 2
|
||||
}
|
||||
],
|
||||
"fallback": false,
|
||||
"primary_agent": "Astra",
|
||||
"required_agents": [
|
||||
"Astra"
|
||||
],
|
||||
"route_kind": "single",
|
||||
"scores": {
|
||||
"Astra": 12,
|
||||
"Clay": 0,
|
||||
"Leo": 0,
|
||||
"Rio": 0,
|
||||
"Theseus": 0,
|
||||
"Vida": 0
|
||||
},
|
||||
"touched_domains": [
|
||||
"space-development"
|
||||
]
|
||||
},
|
||||
"tier": "STANDARD",
|
||||
"verdict": "APPROVE"
|
||||
},
|
||||
{
|
||||
"agent": "Rio",
|
||||
"files": [
|
||||
"domains/ai-systems/agent-wallets.md",
|
||||
"domains/internet-finance/x402.md"
|
||||
],
|
||||
"route": {
|
||||
"evidence": [
|
||||
{
|
||||
"agent": "Theseus",
|
||||
"signal": "path",
|
||||
"value": "domains/ai-systems/agent-wallets.md",
|
||||
"weight": 8
|
||||
},
|
||||
{
|
||||
"agent": "Rio",
|
||||
"signal": "path",
|
||||
"value": "domains/internet-finance/x402.md",
|
||||
"weight": 8
|
||||
},
|
||||
{
|
||||
"agent": "Rio",
|
||||
"signal": "keyword",
|
||||
"value": "x402",
|
||||
"weight": 2
|
||||
}
|
||||
],
|
||||
"fallback": false,
|
||||
"primary_agent": "Rio",
|
||||
"required_agents": [
|
||||
"Rio",
|
||||
"Theseus"
|
||||
],
|
||||
"route_kind": "multi",
|
||||
"scores": {
|
||||
"Astra": 0,
|
||||
"Clay": 0,
|
||||
"Leo": 0,
|
||||
"Rio": 10,
|
||||
"Theseus": 8,
|
||||
"Vida": 0
|
||||
},
|
||||
"touched_domains": [
|
||||
"ai-systems",
|
||||
"internet-finance"
|
||||
]
|
||||
},
|
||||
"tier": "STANDARD",
|
||||
"verdict": "APPROVE"
|
||||
},
|
||||
{
|
||||
"agent": "Theseus",
|
||||
"files": [
|
||||
"domains/ai-systems/agent-wallets.md",
|
||||
"domains/internet-finance/x402.md"
|
||||
],
|
||||
"route": {
|
||||
"evidence": [
|
||||
{
|
||||
"agent": "Theseus",
|
||||
"signal": "path",
|
||||
"value": "domains/ai-systems/agent-wallets.md",
|
||||
"weight": 8
|
||||
},
|
||||
{
|
||||
"agent": "Rio",
|
||||
"signal": "path",
|
||||
"value": "domains/internet-finance/x402.md",
|
||||
"weight": 8
|
||||
},
|
||||
{
|
||||
"agent": "Rio",
|
||||
"signal": "keyword",
|
||||
"value": "x402",
|
||||
"weight": 2
|
||||
}
|
||||
],
|
||||
"fallback": false,
|
||||
"primary_agent": "Rio",
|
||||
"required_agents": [
|
||||
"Rio",
|
||||
"Theseus"
|
||||
],
|
||||
"route_kind": "multi",
|
||||
"scores": {
|
||||
"Astra": 0,
|
||||
"Clay": 0,
|
||||
"Leo": 0,
|
||||
"Rio": 10,
|
||||
"Theseus": 8,
|
||||
"Vida": 0
|
||||
},
|
||||
"touched_domains": [
|
||||
"ai-systems",
|
||||
"internet-finance"
|
||||
]
|
||||
},
|
||||
"tier": "STANDARD",
|
||||
"verdict": "APPROVE"
|
||||
},
|
||||
{
|
||||
"agent": "Vida",
|
||||
"files": [
|
||||
"domains/health/incorrect-health-claim.md"
|
||||
],
|
||||
"route": {
|
||||
"evidence": [
|
||||
{
|
||||
"agent": "Vida",
|
||||
"signal": "path",
|
||||
"value": "domains/health/incorrect-health-claim.md",
|
||||
"weight": 8
|
||||
},
|
||||
{
|
||||
"agent": "Vida",
|
||||
"signal": "keyword",
|
||||
"value": "health",
|
||||
"weight": 2
|
||||
}
|
||||
],
|
||||
"fallback": false,
|
||||
"primary_agent": "Vida",
|
||||
"required_agents": [
|
||||
"Vida"
|
||||
],
|
||||
"route_kind": "single",
|
||||
"scores": {
|
||||
"Astra": 0,
|
||||
"Clay": 0,
|
||||
"Leo": 0,
|
||||
"Rio": 0,
|
||||
"Theseus": 0,
|
||||
"Vida": 10
|
||||
},
|
||||
"touched_domains": [
|
||||
"health"
|
||||
]
|
||||
},
|
||||
"tier": "STANDARD",
|
||||
"verdict": "REQUEST_CHANGES"
|
||||
}
|
||||
],
|
||||
"agents_seen": [
|
||||
"Astra",
|
||||
"Clay",
|
||||
"Leo",
|
||||
"Rio",
|
||||
"Theseus",
|
||||
"Vida"
|
||||
],
|
||||
"case_results": [
|
||||
{
|
||||
"comments": 1,
|
||||
"domain": "grand-strategy",
|
||||
"domain_agent": "Leo",
|
||||
"domain_verdict": "skipped",
|
||||
"expected_agents": [
|
||||
"Leo"
|
||||
],
|
||||
"markers": [
|
||||
"<!-- PHASE1B_REVIEW:PR=101:AGENT=LEO -->"
|
||||
],
|
||||
"number": 101,
|
||||
"reviewers": [
|
||||
"Leo"
|
||||
],
|
||||
"status": "approved"
|
||||
},
|
||||
{
|
||||
"comments": 1,
|
||||
"domain": "ai-alignment",
|
||||
"domain_agent": "Theseus",
|
||||
"domain_verdict": "approve",
|
||||
"expected_agents": [
|
||||
"Theseus"
|
||||
],
|
||||
"markers": [
|
||||
"<!-- PHASE1B_REVIEW:PR=102:AGENT=THESEUS -->"
|
||||
],
|
||||
"number": 102,
|
||||
"reviewers": [
|
||||
"Theseus"
|
||||
],
|
||||
"status": "approved"
|
||||
},
|
||||
{
|
||||
"comments": 1,
|
||||
"domain": "internet-finance",
|
||||
"domain_agent": "Rio",
|
||||
"domain_verdict": "approve",
|
||||
"expected_agents": [
|
||||
"Rio"
|
||||
],
|
||||
"markers": [
|
||||
"<!-- PHASE1B_REVIEW:PR=103:AGENT=RIO -->"
|
||||
],
|
||||
"number": 103,
|
||||
"reviewers": [
|
||||
"Rio"
|
||||
],
|
||||
"status": "approved"
|
||||
},
|
||||
{
|
||||
"comments": 1,
|
||||
"domain": "health",
|
||||
"domain_agent": "Vida",
|
||||
"domain_verdict": "approve",
|
||||
"expected_agents": [
|
||||
"Vida"
|
||||
],
|
||||
"markers": [
|
||||
"<!-- PHASE1B_REVIEW:PR=104:AGENT=VIDA -->"
|
||||
],
|
||||
"number": 104,
|
||||
"reviewers": [
|
||||
"Vida"
|
||||
],
|
||||
"status": "approved"
|
||||
},
|
||||
{
|
||||
"comments": 1,
|
||||
"domain": "entertainment",
|
||||
"domain_agent": "Clay",
|
||||
"domain_verdict": "approve",
|
||||
"expected_agents": [
|
||||
"Clay"
|
||||
],
|
||||
"markers": [
|
||||
"<!-- PHASE1B_REVIEW:PR=105:AGENT=CLAY -->"
|
||||
],
|
||||
"number": 105,
|
||||
"reviewers": [
|
||||
"Clay"
|
||||
],
|
||||
"status": "approved"
|
||||
},
|
||||
{
|
||||
"comments": 1,
|
||||
"domain": "space-development",
|
||||
"domain_agent": "Astra",
|
||||
"domain_verdict": "approve",
|
||||
"expected_agents": [
|
||||
"Astra"
|
||||
],
|
||||
"markers": [
|
||||
"<!-- PHASE1B_REVIEW:PR=106:AGENT=ASTRA -->"
|
||||
],
|
||||
"number": 106,
|
||||
"reviewers": [
|
||||
"Astra"
|
||||
],
|
||||
"status": "approved"
|
||||
},
|
||||
{
|
||||
"comments": 2,
|
||||
"domain": "cross-ai-finance",
|
||||
"domain_agent": "Rio",
|
||||
"domain_verdict": "approve",
|
||||
"expected_agents": [
|
||||
"Rio",
|
||||
"Theseus"
|
||||
],
|
||||
"markers": [
|
||||
"<!-- PHASE1B_REVIEW:PR=107:AGENT=RIO -->",
|
||||
"<!-- PHASE1B_REVIEW:PR=107:AGENT=THESEUS -->"
|
||||
],
|
||||
"number": 107,
|
||||
"reviewers": [
|
||||
"Rio",
|
||||
"Theseus"
|
||||
],
|
||||
"status": "approved"
|
||||
},
|
||||
{
|
||||
"comments": 1,
|
||||
"domain": "health-feedback",
|
||||
"domain_agent": "Vida",
|
||||
"domain_verdict": "request_changes",
|
||||
"expected_agents": [
|
||||
"Vida"
|
||||
],
|
||||
"markers": [
|
||||
"<!-- PHASE1B_REVIEW:PR=108:AGENT=VIDA -->"
|
||||
],
|
||||
"number": 108,
|
||||
"reviewers": [
|
||||
"Vida"
|
||||
],
|
||||
"status": "open"
|
||||
}
|
||||
],
|
||||
"cases_total": 8,
|
||||
"eval_feedback": [
|
||||
{
|
||||
"issues": [],
|
||||
"outcome": "approved",
|
||||
"pr": 101
|
||||
},
|
||||
{
|
||||
"issues": [],
|
||||
"outcome": "approved",
|
||||
"pr": 102
|
||||
},
|
||||
{
|
||||
"issues": [],
|
||||
"outcome": "approved",
|
||||
"pr": 103
|
||||
},
|
||||
{
|
||||
"issues": [],
|
||||
"outcome": "approved",
|
||||
"pr": 104
|
||||
},
|
||||
{
|
||||
"issues": [],
|
||||
"outcome": "approved",
|
||||
"pr": 105
|
||||
},
|
||||
{
|
||||
"issues": [],
|
||||
"outcome": "approved",
|
||||
"pr": 106
|
||||
},
|
||||
{
|
||||
"issues": [],
|
||||
"outcome": "approved",
|
||||
"pr": 107
|
||||
},
|
||||
{
|
||||
"issues": [
|
||||
"factual_discrepancy"
|
||||
],
|
||||
"outcome": "rejected",
|
||||
"pr": 108
|
||||
}
|
||||
],
|
||||
"failed": 0,
|
||||
"feature_flag": "PHASE1B_AGENT_ROUTING_ENABLED",
|
||||
"formal_approvals": [
|
||||
101,
|
||||
102,
|
||||
103,
|
||||
104,
|
||||
105,
|
||||
106,
|
||||
107
|
||||
],
|
||||
"ok": true,
|
||||
"rejection_dispositions": [
|
||||
{
|
||||
"eval_attempts": 1,
|
||||
"issues": [
|
||||
"factual_discrepancy"
|
||||
],
|
||||
"pr": 108
|
||||
}
|
||||
],
|
||||
"route_events": [
|
||||
{
|
||||
"pr": 101,
|
||||
"route": {
|
||||
"evidence": [
|
||||
{
|
||||
"agent": "Leo",
|
||||
"signal": "path",
|
||||
"value": "domains/grand-strategy/strategy.md",
|
||||
"weight": 8
|
||||
}
|
||||
],
|
||||
"fallback": false,
|
||||
"primary_agent": "Leo",
|
||||
"required_agents": [
|
||||
"Leo"
|
||||
],
|
||||
"route_kind": "single",
|
||||
"scores": {
|
||||
"Astra": 0,
|
||||
"Clay": 0,
|
||||
"Leo": 8,
|
||||
"Rio": 0,
|
||||
"Theseus": 0,
|
||||
"Vida": 0
|
||||
},
|
||||
"touched_domains": [
|
||||
"grand-strategy"
|
||||
]
|
||||
},
|
||||
"tier": "STANDARD"
|
||||
},
|
||||
{
|
||||
"pr": 102,
|
||||
"route": {
|
||||
"evidence": [
|
||||
{
|
||||
"agent": "Theseus",
|
||||
"signal": "path",
|
||||
"value": "domains/ai-alignment/systems.md",
|
||||
"weight": 8
|
||||
}
|
||||
],
|
||||
"fallback": false,
|
||||
"primary_agent": "Theseus",
|
||||
"required_agents": [
|
||||
"Theseus"
|
||||
],
|
||||
"route_kind": "single",
|
||||
"scores": {
|
||||
"Astra": 0,
|
||||
"Clay": 0,
|
||||
"Leo": 0,
|
||||
"Rio": 0,
|
||||
"Theseus": 8,
|
||||
"Vida": 0
|
||||
},
|
||||
"touched_domains": [
|
||||
"ai-alignment"
|
||||
]
|
||||
},
|
||||
"tier": "STANDARD"
|
||||
},
|
||||
{
|
||||
"pr": 103,
|
||||
"route": {
|
||||
"evidence": [
|
||||
{
|
||||
"agent": "Rio",
|
||||
"signal": "path",
|
||||
"value": "domains/internet-finance/x402.md",
|
||||
"weight": 8
|
||||
},
|
||||
{
|
||||
"agent": "Rio",
|
||||
"signal": "keyword",
|
||||
"value": "x402",
|
||||
"weight": 2
|
||||
}
|
||||
],
|
||||
"fallback": false,
|
||||
"primary_agent": "Rio",
|
||||
"required_agents": [
|
||||
"Rio"
|
||||
],
|
||||
"route_kind": "single",
|
||||
"scores": {
|
||||
"Astra": 0,
|
||||
"Clay": 0,
|
||||
"Leo": 0,
|
||||
"Rio": 10,
|
||||
"Theseus": 0,
|
||||
"Vida": 0
|
||||
},
|
||||
"touched_domains": [
|
||||
"internet-finance"
|
||||
]
|
||||
},
|
||||
"tier": "STANDARD"
|
||||
},
|
||||
{
|
||||
"pr": 104,
|
||||
"route": {
|
||||
"evidence": [
|
||||
{
|
||||
"agent": "Vida",
|
||||
"signal": "path",
|
||||
"value": "domains/health/clinical.md",
|
||||
"weight": 8
|
||||
},
|
||||
{
|
||||
"agent": "Vida",
|
||||
"signal": "keyword",
|
||||
"value": "health",
|
||||
"weight": 2
|
||||
},
|
||||
{
|
||||
"agent": "Vida",
|
||||
"signal": "keyword",
|
||||
"value": "clinical",
|
||||
"weight": 2
|
||||
}
|
||||
],
|
||||
"fallback": false,
|
||||
"primary_agent": "Vida",
|
||||
"required_agents": [
|
||||
"Vida"
|
||||
],
|
||||
"route_kind": "single",
|
||||
"scores": {
|
||||
"Astra": 0,
|
||||
"Clay": 0,
|
||||
"Leo": 0,
|
||||
"Rio": 0,
|
||||
"Theseus": 0,
|
||||
"Vida": 12
|
||||
},
|
||||
"touched_domains": [
|
||||
"health"
|
||||
]
|
||||
},
|
||||
"tier": "STANDARD"
|
||||
},
|
||||
{
|
||||
"pr": 105,
|
||||
"route": {
|
||||
"evidence": [
|
||||
{
|
||||
"agent": "Clay",
|
||||
"signal": "path",
|
||||
"value": "domains/entertainment/games.md",
|
||||
"weight": 8
|
||||
},
|
||||
{
|
||||
"agent": "Clay",
|
||||
"signal": "keyword",
|
||||
"value": "entertainment",
|
||||
"weight": 2
|
||||
},
|
||||
{
|
||||
"agent": "Clay",
|
||||
"signal": "keyword",
|
||||
"value": "games",
|
||||
"weight": 2
|
||||
}
|
||||
],
|
||||
"fallback": false,
|
||||
"primary_agent": "Clay",
|
||||
"required_agents": [
|
||||
"Clay"
|
||||
],
|
||||
"route_kind": "single",
|
||||
"scores": {
|
||||
"Astra": 0,
|
||||
"Clay": 12,
|
||||
"Leo": 0,
|
||||
"Rio": 0,
|
||||
"Theseus": 0,
|
||||
"Vida": 0
|
||||
},
|
||||
"touched_domains": [
|
||||
"entertainment"
|
||||
]
|
||||
},
|
||||
"tier": "STANDARD"
|
||||
},
|
||||
{
|
||||
"pr": 106,
|
||||
"route": {
|
||||
"evidence": [
|
||||
{
|
||||
"agent": "Astra",
|
||||
"signal": "path",
|
||||
"value": "domains/space-development/robotics.md",
|
||||
"weight": 8
|
||||
},
|
||||
{
|
||||
"agent": "Astra",
|
||||
"signal": "keyword",
|
||||
"value": "space",
|
||||
"weight": 2
|
||||
},
|
||||
{
|
||||
"agent": "Astra",
|
||||
"signal": "keyword",
|
||||
"value": "robotics",
|
||||
"weight": 2
|
||||
}
|
||||
],
|
||||
"fallback": false,
|
||||
"primary_agent": "Astra",
|
||||
"required_agents": [
|
||||
"Astra"
|
||||
],
|
||||
"route_kind": "single",
|
||||
"scores": {
|
||||
"Astra": 12,
|
||||
"Clay": 0,
|
||||
"Leo": 0,
|
||||
"Rio": 0,
|
||||
"Theseus": 0,
|
||||
"Vida": 0
|
||||
},
|
||||
"touched_domains": [
|
||||
"space-development"
|
||||
]
|
||||
},
|
||||
"tier": "STANDARD"
|
||||
},
|
||||
{
|
||||
"pr": 107,
|
||||
"route": {
|
||||
"evidence": [
|
||||
{
|
||||
"agent": "Theseus",
|
||||
"signal": "path",
|
||||
"value": "domains/ai-systems/agent-wallets.md",
|
||||
"weight": 8
|
||||
},
|
||||
{
|
||||
"agent": "Rio",
|
||||
"signal": "path",
|
||||
"value": "domains/internet-finance/x402.md",
|
||||
"weight": 8
|
||||
},
|
||||
{
|
||||
"agent": "Rio",
|
||||
"signal": "keyword",
|
||||
"value": "x402",
|
||||
"weight": 2
|
||||
}
|
||||
],
|
||||
"fallback": false,
|
||||
"primary_agent": "Rio",
|
||||
"required_agents": [
|
||||
"Rio",
|
||||
"Theseus"
|
||||
],
|
||||
"route_kind": "multi",
|
||||
"scores": {
|
||||
"Astra": 0,
|
||||
"Clay": 0,
|
||||
"Leo": 0,
|
||||
"Rio": 10,
|
||||
"Theseus": 8,
|
||||
"Vida": 0
|
||||
},
|
||||
"touched_domains": [
|
||||
"ai-systems",
|
||||
"internet-finance"
|
||||
]
|
||||
},
|
||||
"tier": "STANDARD"
|
||||
},
|
||||
{
|
||||
"pr": 108,
|
||||
"route": {
|
||||
"evidence": [
|
||||
{
|
||||
"agent": "Vida",
|
||||
"signal": "path",
|
||||
"value": "domains/health/incorrect-health-claim.md",
|
||||
"weight": 8
|
||||
},
|
||||
{
|
||||
"agent": "Vida",
|
||||
"signal": "keyword",
|
||||
"value": "health",
|
||||
"weight": 2
|
||||
}
|
||||
],
|
||||
"fallback": false,
|
||||
"primary_agent": "Vida",
|
||||
"required_agents": [
|
||||
"Vida"
|
||||
],
|
||||
"route_kind": "single",
|
||||
"scores": {
|
||||
"Astra": 0,
|
||||
"Clay": 0,
|
||||
"Leo": 0,
|
||||
"Rio": 0,
|
||||
"Theseus": 0,
|
||||
"Vida": 10
|
||||
},
|
||||
"touched_domains": [
|
||||
"health"
|
||||
]
|
||||
},
|
||||
"tier": "STANDARD"
|
||||
}
|
||||
],
|
||||
"schema_version": 27,
|
||||
"scope": "local_no_network_phase1b_eval_cycle",
|
||||
"source_feedback_paths": [
|
||||
"inbox/archive/phase1b-108.md"
|
||||
],
|
||||
"succeeded": 8
|
||||
}
|
||||
|
|
@ -1,7 +1,3 @@
|
|||
[build-system]
|
||||
requires = ["setuptools>=68"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "teleo-pipeline"
|
||||
version = "2.0.0"
|
||||
|
|
@ -9,7 +5,6 @@ description = "Teleo Pipeline v2 — async daemon for claim extraction, validati
|
|||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"aiohttp>=3.9,<4",
|
||||
"PyYAML>=6,<7",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
|
|
@ -19,9 +14,6 @@ dev = [
|
|||
"ruff>=0.3",
|
||||
]
|
||||
|
||||
[tool.setuptools]
|
||||
packages = ["lib"]
|
||||
|
||||
[tool.ruff]
|
||||
target-version = "py311"
|
||||
line-length = 120
|
||||
|
|
|
|||
|
|
@ -17,18 +17,9 @@ set -euo pipefail
|
|||
|
||||
AGENT="${1:?Usage: $0 <agent-name>}"
|
||||
REPO_DIR="/opt/teleo-eval/workspaces/research-${AGENT}"
|
||||
# GitHub migration (Phase 1 Step 3): single livingIPbot token for all agents.
|
||||
# Per-agent identity deferred to Billy's productionization sprint (Option A).
|
||||
GITHUB_API="https://api.github.com"
|
||||
GITHUB_REPO="living-ip/decision-engine"
|
||||
GITHUB_TOKEN=$(cat /opt/teleo-eval/secrets/github-admin-token)
|
||||
AGENT_TOKEN="$GITHUB_TOKEN" # placeholder for future per-agent tokens
|
||||
# Two auth surfaces:
|
||||
# - REST API: Authorization: Bearer <pat> header
|
||||
# - git smart-HTTP: x-access-token:<pat> in URL, injected via insteadOf rewrite
|
||||
# so the credential never lands in .git/config or `git remote -v` output.
|
||||
GH_REST_AUTH="Authorization: Bearer $GITHUB_TOKEN"
|
||||
GH_GIT_REWRITE="url.https://x-access-token:${GITHUB_TOKEN}@github.com/.insteadOf=https://github.com/"
|
||||
FORGEJO_URL="http://localhost:3000"
|
||||
FORGEJO_ADMIN_TOKEN=$(cat /opt/teleo-eval/secrets/forgejo-admin-token)
|
||||
AGENT_TOKEN=$(cat "/opt/teleo-eval/secrets/forgejo-${AGENT}-token" 2>/dev/null || echo "$FORGEJO_ADMIN_TOKEN")
|
||||
TWITTER_API_KEY=$(cat /opt/teleo-eval/secrets/twitterapi-io-key)
|
||||
CLAUDE_BIN="/home/teleo/.local/bin/claude"
|
||||
LOG_DIR="/opt/teleo-eval/logs"
|
||||
|
|
@ -73,15 +64,14 @@ mkdir -p "$RAW_DIR" "$LOG_DIR"
|
|||
# --- Clone or update repo ---
|
||||
if [ ! -d "$REPO_DIR/.git" ]; then
|
||||
log "Cloning repo for $AGENT research..."
|
||||
git -c "$GH_GIT_REWRITE" \
|
||||
clone "https://github.com/${GITHUB_REPO}.git" "$REPO_DIR" >> "$LOG" 2>&1
|
||||
git -c http.extraHeader="Authorization: token $FORGEJO_ADMIN_TOKEN" \
|
||||
clone "${FORGEJO_URL}/teleo/teleo-codex.git" "$REPO_DIR" >> "$LOG" 2>&1
|
||||
fi
|
||||
|
||||
cd "$REPO_DIR"
|
||||
# Idempotent remote-url swap — handles legacy Forgejo clones from pre-migration runs.
|
||||
git remote set-url origin "https://github.com/${GITHUB_REPO}.git" 2>/dev/null || true
|
||||
git -c "$GH_GIT_REWRITE" checkout main >> "$LOG" 2>&1
|
||||
git -c "$GH_GIT_REWRITE" pull --rebase >> "$LOG" 2>&1
|
||||
git remote set-url origin "${FORGEJO_URL}/teleo/teleo-codex.git" 2>/dev/null || true
|
||||
git -c http.extraHeader="Authorization: token $FORGEJO_ADMIN_TOKEN" checkout main >> "$LOG" 2>&1
|
||||
git -c http.extraHeader="Authorization: token $FORGEJO_ADMIN_TOKEN" pull --rebase >> "$LOG" 2>&1
|
||||
|
||||
# --- Map agent to domain ---
|
||||
case "$AGENT" in
|
||||
|
|
@ -442,16 +432,13 @@ git commit -m "${AGENT}: research session ${DATE} — ${SOURCE_COUNT} sources ar
|
|||
Pentagon-Agent: ${AGENT_UPPER} <HEADLESS>" >> "$LOG" 2>&1
|
||||
|
||||
# --- Push ---
|
||||
git -c "$GH_GIT_REWRITE" push -u origin "$BRANCH" --force >> "$LOG" 2>&1
|
||||
git -c http.extraHeader="Authorization: token $AGENT_TOKEN" push -u origin "$BRANCH" --force >> "$LOG" 2>&1
|
||||
log "Pushed $BRANCH"
|
||||
|
||||
# --- Check for existing PR on this branch ---
|
||||
# GitHub: filter open PRs by head=org:branch. Owner prefix on `head` is required.
|
||||
EXISTING_PR=$(curl -s -H "$GH_REST_AUTH" \
|
||||
-H "Accept: application/vnd.github+json" \
|
||||
-H "X-GitHub-Api-Version: 2022-11-28" \
|
||||
"${GITHUB_API}/repos/${GITHUB_REPO}/pulls?state=open&head=living-ip:${BRANCH}&per_page=10" \
|
||||
| jq -r ".[0].number // empty" 2>/dev/null)
|
||||
EXISTING_PR=$(curl -s "${FORGEJO_URL}/api/v1/repos/teleo/teleo-codex/pulls?state=open" \
|
||||
-H "Authorization: token $AGENT_TOKEN" \
|
||||
| jq -r ".[] | select(.head.ref == \"$BRANCH\") | .number" 2>/dev/null)
|
||||
|
||||
if [ -n "$EXISTING_PR" ]; then
|
||||
log "PR already exists for $BRANCH (#$EXISTING_PR), skipping creation"
|
||||
|
|
@ -470,10 +457,8 @@ Researcher and extractor are different Claude instances to prevent motivated rea
|
|||
--arg head "$BRANCH" \
|
||||
'{title: $title, body: $body, base: $base, head: $head}')
|
||||
|
||||
PR_RESULT=$(curl -s -X POST "${GITHUB_API}/repos/${GITHUB_REPO}/pulls" \
|
||||
-H "$GH_REST_AUTH" \
|
||||
-H "Accept: application/vnd.github+json" \
|
||||
-H "X-GitHub-Api-Version: 2022-11-28" \
|
||||
PR_RESULT=$(curl -s -X POST "${FORGEJO_URL}/api/v1/repos/teleo/teleo-codex/pulls" \
|
||||
-H "Authorization: token $AGENT_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "$PR_JSON" 2>&1)
|
||||
|
||||
|
|
|
|||
|
|
@ -1,104 +0,0 @@
|
|||
# Teleo Agent Graph Schema v1
|
||||
|
||||
Source idea: `teleo-agent-architecture-COMBINED (2).excalidraw`.
|
||||
|
||||
This schema models the agent commons as a graph:
|
||||
|
||||
```text
|
||||
persona -> strategy -> position -> belief -> claim -> evidence
|
||||
```
|
||||
|
||||
The top layers are agent-owned. The lower layers are shared commons.
|
||||
Changes cascade upward: evidence changes re-evaluate claims, claims flag beliefs,
|
||||
beliefs flag positions, and positions can force persona/strategy review.
|
||||
|
||||
## Design Commitments
|
||||
|
||||
- Personas are authored, stable, and loaded every turn.
|
||||
- Strategies are derived from personas using the Rumelt kernel:
|
||||
diagnosis, guiding policy, proximate objectives.
|
||||
- Positions and beliefs are per-agent public commitments.
|
||||
- Claims are owned by no agent.
|
||||
- Evidence is owned by no agent.
|
||||
- Claims link to claims through typed weighted edges.
|
||||
- One evidence node can ground many claims.
|
||||
- One claim can be cited by many beliefs across agents and domains.
|
||||
- `cited_by` and `importance` are computed/readback fields, not hand-authored
|
||||
truth.
|
||||
- Every edge has a relation, weight, and rationale so cascade behavior is
|
||||
auditable.
|
||||
|
||||
## Main Tables
|
||||
|
||||
| Table | Purpose |
|
||||
| --- | --- |
|
||||
| `agents` | Agent registry: Leo, Rio, Theseus, etc. |
|
||||
| `agent_persona_revisions` | Stable authored identity, voice, and role snapshots |
|
||||
| `agent_strategy_revisions` | Derived diagnosis, guiding policy, and objectives |
|
||||
| `agent_positions` | Per-agent public commitments with falsification criteria |
|
||||
| `agent_beliefs` | Per-agent falsifiable beliefs citing claims |
|
||||
| `claims` | Shared claim commons |
|
||||
| `evidence` | Shared sourced/verifiable evidence commons |
|
||||
| `position_belief_edges` | Position depends on belief |
|
||||
| `belief_claim_edges` | Belief cites or depends on claim |
|
||||
| `claim_edges` | Claim-to-claim typed relationship |
|
||||
| `claim_evidence_edges` | Claim grounded by evidence |
|
||||
| `graph_evaluation_runs` | Evaluation/re-evaluation records |
|
||||
| `cascade_events` | Upward propagation queue/history |
|
||||
| `graph_history_events` | Sanitized GitHub/Forgejo/local-git manifest events |
|
||||
| `graph_node_history_links` | Links history events to graph nodes |
|
||||
|
||||
## Claim Node
|
||||
|
||||
Diagram frontmatter maps to `claims`:
|
||||
|
||||
| Diagram field | Column |
|
||||
| --- | --- |
|
||||
| `type: claim` | implicit table |
|
||||
| `domain` | `claims.domain` |
|
||||
| `description` | `claims.description` |
|
||||
| `confidence` | `claims.confidence` |
|
||||
| `source` | `claims.source_summary`, plus evidence edges |
|
||||
| `created` | `claims.created_at` |
|
||||
| `last_evaluated` | `claims.last_evaluated` |
|
||||
| `cross_references` | `claim_edges` |
|
||||
| `importance` | `claims.importance`, computed from inbound refs |
|
||||
| `attribution` | `claims.attribution_json` |
|
||||
|
||||
## Claim Relations
|
||||
|
||||
| Relation | Meaning |
|
||||
| --- | --- |
|
||||
| `depends_on` | This claim cannot be true unless the linked claim is true |
|
||||
| `supports` | Linked claim provides evidence for this one |
|
||||
| `challenged_by` | Linked claim is counter-argument or counter-evidence |
|
||||
| `cited_by` | Computed inbound reference, not hand-authored |
|
||||
| `related` | Topical link without a specific evidential relationship |
|
||||
|
||||
## Experiment Use
|
||||
|
||||
This schema should be applied after a test database is created and before a
|
||||
history manifest is loaded:
|
||||
|
||||
```text
|
||||
spin database
|
||||
apply teleo-agent-graph-v1.sql
|
||||
load history manifest through graph adapter
|
||||
run persona/journey/red-team experiments
|
||||
verify cascades and graph invariants
|
||||
tear database down
|
||||
```
|
||||
|
||||
## Minimum Invariants
|
||||
|
||||
- Every active belief must cite at least three claims before it can be marked
|
||||
`load_bearing`.
|
||||
- Every active claim must have at least one evidence edge before it can be
|
||||
marked `accepted`.
|
||||
- Red-team or quarantined claims cannot be cited by active beliefs unless the
|
||||
edge relation is `challenged_by`.
|
||||
- `claim_edges` cannot self-reference.
|
||||
- `importance` should be recomputed from inbound belief and claim references
|
||||
during loader/evaluation jobs.
|
||||
- Any evidence update must produce cascade events for affected claims and
|
||||
upstream beliefs/positions.
|
||||
|
|
@ -1,251 +0,0 @@
|
|||
-- Teleo Agent Graph Schema v1
|
||||
-- Common SQL subset intended for ephemeral SQLite tests and Postgres/Supabase
|
||||
-- staging. IDs are app-generated text IDs so this can run across engines.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS graph_schema_version (
|
||||
version TEXT PRIMARY KEY,
|
||||
source TEXT NOT NULL,
|
||||
applied_at TEXT DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
INSERT OR IGNORE INTO graph_schema_version (version, source)
|
||||
VALUES ('teleo-agent-graph-v1', 'teleo-agent-architecture-excalidraw');
|
||||
|
||||
CREATE TABLE IF NOT EXISTS agents (
|
||||
slug TEXT PRIMARY KEY,
|
||||
display_name TEXT NOT NULL,
|
||||
archetype TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'active'
|
||||
CHECK(status IN ('active', 'inactive', 'deprecated')),
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TEXT DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS agent_persona_revisions (
|
||||
id TEXT PRIMARY KEY,
|
||||
agent_slug TEXT NOT NULL REFERENCES agents(slug),
|
||||
revision INTEGER NOT NULL,
|
||||
identity TEXT NOT NULL,
|
||||
voice TEXT NOT NULL,
|
||||
role TEXT NOT NULL,
|
||||
authored_by TEXT,
|
||||
stable INTEGER NOT NULL DEFAULT 1 CHECK(stable IN (0, 1)),
|
||||
loads_every_turn INTEGER NOT NULL DEFAULT 1 CHECK(loads_every_turn IN (0, 1)),
|
||||
active INTEGER NOT NULL DEFAULT 1 CHECK(active IN (0, 1)),
|
||||
notes TEXT,
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(agent_slug, revision)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS agent_strategy_revisions (
|
||||
id TEXT PRIMARY KEY,
|
||||
agent_slug TEXT NOT NULL REFERENCES agents(slug),
|
||||
persona_revision_id TEXT REFERENCES agent_persona_revisions(id),
|
||||
revision INTEGER NOT NULL,
|
||||
diagnosis TEXT NOT NULL,
|
||||
guiding_policy TEXT NOT NULL,
|
||||
proximate_objectives_json TEXT NOT NULL DEFAULT '[]',
|
||||
derivation_notes TEXT,
|
||||
active INTEGER NOT NULL DEFAULT 1 CHECK(active IN (0, 1)),
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(agent_slug, revision)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS agent_positions (
|
||||
id TEXT PRIMARY KEY,
|
||||
agent_slug TEXT NOT NULL REFERENCES agents(slug),
|
||||
title TEXT NOT NULL,
|
||||
statement TEXT NOT NULL,
|
||||
falsification_criteria TEXT,
|
||||
public_commitment INTEGER NOT NULL DEFAULT 1 CHECK(public_commitment IN (0, 1)),
|
||||
confidence TEXT NOT NULL DEFAULT 'experimental'
|
||||
CHECK(confidence IN ('proven', 'likely', 'experimental', 'speculative')),
|
||||
status TEXT NOT NULL DEFAULT 'active'
|
||||
CHECK(status IN ('draft', 'active', 'flagged', 'retired')),
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
last_reviewed TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS agent_beliefs (
|
||||
id TEXT PRIMARY KEY,
|
||||
agent_slug TEXT NOT NULL REFERENCES agents(slug),
|
||||
belief_code TEXT NOT NULL,
|
||||
title TEXT NOT NULL,
|
||||
statement TEXT NOT NULL,
|
||||
falsification_criteria TEXT,
|
||||
is_keystone INTEGER NOT NULL DEFAULT 0 CHECK(is_keystone IN (0, 1)),
|
||||
min_claims INTEGER NOT NULL DEFAULT 3,
|
||||
confidence TEXT NOT NULL DEFAULT 'experimental'
|
||||
CHECK(confidence IN ('proven', 'likely', 'experimental', 'speculative')),
|
||||
status TEXT NOT NULL DEFAULT 'active'
|
||||
CHECK(status IN ('draft', 'active', 'load_bearing', 'flagged', 'retired')),
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
last_evaluated TEXT,
|
||||
UNIQUE(agent_slug, belief_code)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS evidence (
|
||||
id TEXT PRIMARY KEY,
|
||||
evidence_type TEXT NOT NULL
|
||||
CHECK(evidence_type IN ('study', 'data', 'event', 'formal_result', 'legal', 'market', 'historical', 'other')),
|
||||
title TEXT NOT NULL,
|
||||
source_uri TEXT,
|
||||
citation TEXT,
|
||||
summary TEXT NOT NULL,
|
||||
verification_status TEXT NOT NULL DEFAULT 'unverified'
|
||||
CHECK(verification_status IN ('unverified', 'sourced', 'verified', 'disputed', 'retracted')),
|
||||
observed_at TEXT,
|
||||
attribution_json TEXT NOT NULL DEFAULT '{}',
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS claims (
|
||||
id TEXT PRIMARY KEY,
|
||||
slug TEXT NOT NULL UNIQUE,
|
||||
domain TEXT NOT NULL,
|
||||
description TEXT NOT NULL,
|
||||
confidence TEXT NOT NULL DEFAULT 'experimental'
|
||||
CHECK(confidence IN ('proven', 'likely', 'experimental', 'speculative')),
|
||||
source_summary TEXT,
|
||||
proposed_by TEXT,
|
||||
primary_evidence_id TEXT REFERENCES evidence(id),
|
||||
importance REAL NOT NULL DEFAULT 0 CHECK(importance >= 0 AND importance <= 1),
|
||||
status TEXT NOT NULL DEFAULT 'draft'
|
||||
CHECK(status IN ('draft', 'active', 'accepted', 'challenged', 'quarantined', 'retired')),
|
||||
attribution_json TEXT NOT NULL DEFAULT '{}',
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
last_evaluated TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS position_belief_edges (
|
||||
id TEXT PRIMARY KEY,
|
||||
position_id TEXT NOT NULL REFERENCES agent_positions(id),
|
||||
belief_id TEXT NOT NULL REFERENCES agent_beliefs(id),
|
||||
relation TEXT NOT NULL DEFAULT 'depends_on'
|
||||
CHECK(relation IN ('depends_on', 'supports', 'challenged_by', 'related')),
|
||||
weight REAL NOT NULL DEFAULT 1 CHECK(weight >= 0 AND weight <= 1),
|
||||
rationale TEXT NOT NULL,
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(position_id, belief_id, relation)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS belief_claim_edges (
|
||||
id TEXT PRIMARY KEY,
|
||||
belief_id TEXT NOT NULL REFERENCES agent_beliefs(id),
|
||||
claim_id TEXT NOT NULL REFERENCES claims(id),
|
||||
relation TEXT NOT NULL DEFAULT 'cites'
|
||||
CHECK(relation IN ('cites', 'depends_on', 'supports', 'challenged_by', 'related')),
|
||||
weight REAL NOT NULL DEFAULT 1 CHECK(weight >= 0 AND weight <= 1),
|
||||
rationale TEXT NOT NULL,
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(belief_id, claim_id, relation)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS claim_edges (
|
||||
id TEXT PRIMARY KEY,
|
||||
from_claim_id TEXT NOT NULL REFERENCES claims(id),
|
||||
to_claim_id TEXT NOT NULL REFERENCES claims(id),
|
||||
relation TEXT NOT NULL
|
||||
CHECK(relation IN ('depends_on', 'supports', 'challenged_by', 'cited_by', 'related')),
|
||||
weight REAL NOT NULL DEFAULT 1 CHECK(weight >= 0 AND weight <= 1),
|
||||
rationale TEXT NOT NULL,
|
||||
authored_by TEXT,
|
||||
computed INTEGER NOT NULL DEFAULT 0 CHECK(computed IN (0, 1)),
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
CHECK(from_claim_id <> to_claim_id),
|
||||
UNIQUE(from_claim_id, to_claim_id, relation)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS claim_evidence_edges (
|
||||
id TEXT PRIMARY KEY,
|
||||
claim_id TEXT NOT NULL REFERENCES claims(id),
|
||||
evidence_id TEXT NOT NULL REFERENCES evidence(id),
|
||||
relation TEXT NOT NULL DEFAULT 'supports'
|
||||
CHECK(relation IN ('primary', 'supports', 'challenges', 'context', 'weakens')),
|
||||
weight REAL NOT NULL DEFAULT 1 CHECK(weight >= 0 AND weight <= 1),
|
||||
rationale TEXT NOT NULL,
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(claim_id, evidence_id, relation)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS graph_evaluation_runs (
|
||||
id TEXT PRIMARY KEY,
|
||||
target_layer TEXT NOT NULL
|
||||
CHECK(target_layer IN ('persona', 'strategy', 'position', 'belief', 'claim', 'evidence', 'edge')),
|
||||
target_id TEXT NOT NULL,
|
||||
trigger_type TEXT NOT NULL
|
||||
CHECK(trigger_type IN ('scheduled', 'history_replay', 'evidence_changed', 'claim_changed', 'manual', 'red_team')),
|
||||
trigger_id TEXT,
|
||||
evaluator TEXT NOT NULL,
|
||||
model TEXT,
|
||||
verdict TEXT NOT NULL
|
||||
CHECK(verdict IN ('approve', 'request_changes', 'reject', 'flag', 'quarantine', 'no_op')),
|
||||
confidence REAL CHECK(confidence IS NULL OR (confidence >= 0 AND confidence <= 1)),
|
||||
notes TEXT,
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS cascade_events (
|
||||
id TEXT PRIMARY KEY,
|
||||
changed_layer TEXT NOT NULL
|
||||
CHECK(changed_layer IN ('evidence', 'claim', 'belief', 'position', 'strategy', 'persona')),
|
||||
changed_id TEXT NOT NULL,
|
||||
affected_layer TEXT NOT NULL
|
||||
CHECK(affected_layer IN ('claim', 'belief', 'position', 'strategy', 'persona')),
|
||||
affected_id TEXT NOT NULL,
|
||||
direction TEXT NOT NULL DEFAULT 'up'
|
||||
CHECK(direction IN ('up', 'down', 'lateral')),
|
||||
status TEXT NOT NULL DEFAULT 'queued'
|
||||
CHECK(status IN ('queued', 'reviewing', 'resolved', 'ignored')),
|
||||
reason TEXT NOT NULL,
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
resolved_at TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS graph_history_events (
|
||||
id TEXT PRIMARY KEY,
|
||||
provider TEXT NOT NULL CHECK(provider IN ('github', 'forgejo', 'local_git', 'web', 'x', 'telegram', 'manual')),
|
||||
repo TEXT,
|
||||
provider_event_id TEXT,
|
||||
event_type TEXT NOT NULL,
|
||||
actor TEXT,
|
||||
occurred_at TEXT,
|
||||
payload_json TEXT NOT NULL DEFAULT '{}',
|
||||
redacted INTEGER NOT NULL DEFAULT 1 CHECK(redacted IN (0, 1)),
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS graph_node_history_links (
|
||||
history_event_id TEXT NOT NULL REFERENCES graph_history_events(id),
|
||||
node_layer TEXT NOT NULL
|
||||
CHECK(node_layer IN ('persona', 'strategy', 'position', 'belief', 'claim', 'evidence', 'edge')),
|
||||
node_id TEXT NOT NULL,
|
||||
role TEXT NOT NULL
|
||||
CHECK(role IN ('created', 'updated', 'evaluated', 'merged', 'challenged', 'cited', 'sourced')),
|
||||
PRIMARY KEY (history_event_id, node_layer, node_id, role)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_persona_revisions_agent_active
|
||||
ON agent_persona_revisions(agent_slug, active);
|
||||
CREATE INDEX IF NOT EXISTS idx_strategy_revisions_agent_active
|
||||
ON agent_strategy_revisions(agent_slug, active);
|
||||
CREATE INDEX IF NOT EXISTS idx_positions_agent_status
|
||||
ON agent_positions(agent_slug, status);
|
||||
CREATE INDEX IF NOT EXISTS idx_beliefs_agent_status
|
||||
ON agent_beliefs(agent_slug, status);
|
||||
CREATE INDEX IF NOT EXISTS idx_claims_domain_status
|
||||
ON claims(domain, status);
|
||||
CREATE INDEX IF NOT EXISTS idx_claims_importance
|
||||
ON claims(importance);
|
||||
CREATE INDEX IF NOT EXISTS idx_evidence_status
|
||||
ON evidence(verification_status);
|
||||
CREATE INDEX IF NOT EXISTS idx_belief_claim_edges_claim
|
||||
ON belief_claim_edges(claim_id, relation);
|
||||
CREATE INDEX IF NOT EXISTS idx_claim_edges_to
|
||||
ON claim_edges(to_claim_id, relation);
|
||||
CREATE INDEX IF NOT EXISTS idx_claim_evidence_edges_evidence
|
||||
ON claim_evidence_edges(evidence_id, relation);
|
||||
CREATE INDEX IF NOT EXISTS idx_cascade_status
|
||||
ON cascade_events(status, affected_layer);
|
||||
CREATE INDEX IF NOT EXISTS idx_history_provider_repo
|
||||
ON graph_history_events(provider, repo, event_type);
|
||||
|
|
@ -1,73 +0,0 @@
|
|||
# Teleo Agent Research Eval Schema v1
|
||||
|
||||
Apply this schema after `teleo-agent-graph-v1.sql`.
|
||||
|
||||
This schema records how Leo and other agents answer research requests, which
|
||||
tools they choose, what sources they cite, and whether benchmark cases passed.
|
||||
It is operational/economic telemetry, not the claim/evidence graph itself.
|
||||
|
||||
## Design Commitments
|
||||
|
||||
- The graph schema remains the knowledge spine: persona, strategy, beliefs,
|
||||
claims, evidence, graph evals, and cascades.
|
||||
- Research-eval rows explain how a request was handled and whether the route was
|
||||
good enough to trust or ship.
|
||||
- Payment funds work. It does not directly mutate claims, confidence, beliefs,
|
||||
or rewards.
|
||||
- Tool-use benchmarking must distinguish candidates, selected tools, executed
|
||||
tools, skipped tools, and rejected tools.
|
||||
- Secrets and private payloads are never stored. Tables store hashes, redacted
|
||||
excerpts, proof references, source metadata, and receipt ids.
|
||||
|
||||
## Main Tables
|
||||
|
||||
| Table | Purpose |
|
||||
| --- | --- |
|
||||
| `agent_research_runs` | One row per research request from Telegram, API, checkout, CLI, or benchmark. |
|
||||
| `agent_tool_invocations` | One row per candidate, selected, executed, skipped, rejected, fallback, or failed tool decision. |
|
||||
| `agent_research_sources` | Retrieved or cited source rows tied to a run and optionally a tool invocation. |
|
||||
| `agent_eval_cases` | Versioned benchmark prompts, expected routes/providers, tool constraints, tags, and rubrics. |
|
||||
| `agent_eval_results` | Per-case result, routing correctness, tool score, source quality, groundedness, cost, and safety scores. |
|
||||
| `work_order_graph_links` | Links sponsored work orders to research runs, tool traces, graph evals, evidence, claims, and outcomes. |
|
||||
|
||||
## Leo x402 Research Flow
|
||||
|
||||
```text
|
||||
Telegram/API question
|
||||
-> agent_research_runs
|
||||
-> agent_tool_invocations
|
||||
-> agent_research_sources
|
||||
-> agent_eval_results when a benchmark case applies
|
||||
-> work_order_graph_links when a paid work order or graph artifact is involved
|
||||
```
|
||||
|
||||
For paid research, `agent_research_runs.sponsored_work_order_id` and
|
||||
`payment_receipt_id` carry the external work-order/payment anchors. The payment
|
||||
receipt table is still owned by the economic/payment layer; this schema only
|
||||
keeps references.
|
||||
|
||||
## Ranger Liquidation Guard
|
||||
|
||||
The Ranger benchmark class should be represented as:
|
||||
|
||||
- `agent_eval_cases.expected_route = 'web_search'`
|
||||
- `agent_eval_cases.tags_json` includes `ranger_liquidated`
|
||||
- `agent_eval_cases.must_not_use_tools_json` includes market-data-only routes
|
||||
- `agent_tool_invocations` records market data as `rejected` or `skipped` when
|
||||
it is not the right tool
|
||||
- `agent_eval_results.routing_correct = 1` only if Leo routed to source-backed
|
||||
research instead of live-token valuation
|
||||
|
||||
This ensures "Ranger is liquidated/gone" is verified before any valuation
|
||||
framing and never silently treated as a normal live fair-value token question.
|
||||
|
||||
## Minimum Invariants
|
||||
|
||||
- No row may set `secret_values_included = 1`.
|
||||
- A benchmark result must link to both an eval case and a research run.
|
||||
- Tool invocation sequence numbers are unique per research run.
|
||||
- Scores are bounded between `0` and `1`.
|
||||
- Research runs store prompt and answer hashes plus optional redacted excerpts,
|
||||
not raw private prompts.
|
||||
- `outcome_observations` remain the downstream business-value layer; raw tool
|
||||
traces belong here, not there.
|
||||
|
|
@ -1,247 +0,0 @@
|
|||
-- Teleo Agent Research Eval Schema v1
|
||||
-- Common SQL subset intended for ephemeral SQLite tests and Postgres/Supabase
|
||||
-- staging. IDs are app-generated text IDs so this can run across engines.
|
||||
--
|
||||
-- Apply after teleo-agent-graph-v1.sql.
|
||||
--
|
||||
-- Secret policy: store hashes, redacted excerpts, and proof references only.
|
||||
-- Raw prompts, bearer tokens, API keys, wallet secrets, and private receipts do
|
||||
-- not belong in these tables.
|
||||
|
||||
INSERT OR IGNORE INTO graph_schema_version (version, source)
|
||||
VALUES ('teleo-agent-research-eval-v1', 'leo-x402-research-routing-benchmark');
|
||||
|
||||
CREATE TABLE IF NOT EXISTS agent_research_runs (
|
||||
id TEXT PRIMARY KEY,
|
||||
agent_slug TEXT NOT NULL REFERENCES agents(slug),
|
||||
source_surface TEXT NOT NULL
|
||||
CHECK(source_surface IN ('telegram', 'api', 'checkout', 'web', 'cli', 'test', 'other')),
|
||||
source_ref TEXT,
|
||||
request_kind TEXT NOT NULL DEFAULT 'free'
|
||||
CHECK(request_kind IN ('free', 'paid_quote', 'paid_work_order', 'benchmark', 'system')),
|
||||
sponsored_work_order_id TEXT,
|
||||
payment_receipt_id TEXT,
|
||||
prompt_sha256 TEXT NOT NULL,
|
||||
prompt_excerpt TEXT,
|
||||
selected_provider TEXT,
|
||||
selected_route TEXT NOT NULL DEFAULT 'unknown'
|
||||
CHECK(selected_route IN (
|
||||
'none',
|
||||
'web_search',
|
||||
'social_trends',
|
||||
'structured_market_data',
|
||||
'local_context',
|
||||
'mixed',
|
||||
'unknown'
|
||||
)),
|
||||
status TEXT NOT NULL DEFAULT 'running'
|
||||
CHECK(status IN (
|
||||
'quoted',
|
||||
'payment_pending',
|
||||
'running',
|
||||
'answered',
|
||||
'abstained',
|
||||
'blocked',
|
||||
'failed',
|
||||
'cancelled'
|
||||
)),
|
||||
answer_sha256 TEXT,
|
||||
answer_excerpt TEXT,
|
||||
proof_ref TEXT,
|
||||
cost_amount REAL NOT NULL DEFAULT 0 CHECK(cost_amount >= 0),
|
||||
currency TEXT NOT NULL DEFAULT 'USDC',
|
||||
latency_ms INTEGER CHECK(latency_ms IS NULL OR latency_ms >= 0),
|
||||
source_count INTEGER NOT NULL DEFAULT 0 CHECK(source_count >= 0),
|
||||
secret_values_included INTEGER NOT NULL DEFAULT 0 CHECK(secret_values_included = 0),
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
completed_at TEXT,
|
||||
CHECK(prompt_excerpt IS NULL OR length(prompt_excerpt) <= 1000),
|
||||
CHECK(answer_excerpt IS NULL OR length(answer_excerpt) <= 2000)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_research_runs_agent_created
|
||||
ON agent_research_runs(agent_slug, created_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_research_runs_work_order
|
||||
ON agent_research_runs(sponsored_work_order_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_research_runs_status_route
|
||||
ON agent_research_runs(status, selected_route);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS agent_tool_invocations (
|
||||
id TEXT PRIMARY KEY,
|
||||
research_run_id TEXT NOT NULL REFERENCES agent_research_runs(id) ON DELETE CASCADE,
|
||||
sequence INTEGER NOT NULL DEFAULT 0 CHECK(sequence >= 0),
|
||||
provider TEXT NOT NULL,
|
||||
tool_name TEXT NOT NULL,
|
||||
tool_category TEXT NOT NULL
|
||||
CHECK(tool_category IN (
|
||||
'web_search',
|
||||
'social_trends',
|
||||
'market_data',
|
||||
'page_read',
|
||||
'x402_checkout',
|
||||
'agentcash',
|
||||
'faremeter',
|
||||
'database',
|
||||
'local_context',
|
||||
'other'
|
||||
)),
|
||||
endpoint_host TEXT,
|
||||
endpoint_hash TEXT,
|
||||
decision TEXT NOT NULL
|
||||
CHECK(decision IN ('candidate', 'selected', 'executed', 'skipped', 'rejected', 'fallback', 'failed')),
|
||||
decision_reason TEXT NOT NULL,
|
||||
paid INTEGER NOT NULL DEFAULT 0 CHECK(paid IN (0, 1)),
|
||||
rail TEXT CHECK(rail IS NULL OR rail IN ('x402', 'agentcash', 'manual', 'free', 'other')),
|
||||
network TEXT,
|
||||
amount REAL CHECK(amount IS NULL OR amount >= 0),
|
||||
currency TEXT NOT NULL DEFAULT 'USDC',
|
||||
payment_receipt_id TEXT,
|
||||
input_sha256 TEXT,
|
||||
output_sha256 TEXT,
|
||||
source_count INTEGER NOT NULL DEFAULT 0 CHECK(source_count >= 0),
|
||||
latency_ms INTEGER CHECK(latency_ms IS NULL OR latency_ms >= 0),
|
||||
error_class TEXT,
|
||||
secret_values_included INTEGER NOT NULL DEFAULT 0 CHECK(secret_values_included = 0),
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(research_run_id, sequence)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_tool_invocations_run_decision
|
||||
ON agent_tool_invocations(research_run_id, decision);
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_tool_invocations_provider_category
|
||||
ON agent_tool_invocations(provider, tool_category);
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_tool_invocations_receipt
|
||||
ON agent_tool_invocations(payment_receipt_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS agent_research_sources (
|
||||
id TEXT PRIMARY KEY,
|
||||
research_run_id TEXT NOT NULL REFERENCES agent_research_runs(id) ON DELETE CASCADE,
|
||||
tool_invocation_id TEXT REFERENCES agent_tool_invocations(id) ON DELETE SET NULL,
|
||||
source_type TEXT NOT NULL
|
||||
CHECK(source_type IN ('web', 'social', 'market', 'db', 'document', 'other')),
|
||||
source_uri TEXT,
|
||||
source_uri_sha256 TEXT,
|
||||
title TEXT,
|
||||
cited INTEGER NOT NULL DEFAULT 0 CHECK(cited IN (0, 1)),
|
||||
retrieval_rank INTEGER CHECK(retrieval_rank IS NULL OR retrieval_rank >= 0),
|
||||
observed_at TEXT,
|
||||
support_status TEXT NOT NULL DEFAULT 'unknown'
|
||||
CHECK(support_status IN ('supports', 'context', 'conflicts', 'stale', 'unknown')),
|
||||
secret_values_included INTEGER NOT NULL DEFAULT 0 CHECK(secret_values_included = 0),
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_research_sources_run
|
||||
ON agent_research_sources(research_run_id, cited);
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_research_sources_tool
|
||||
ON agent_research_sources(tool_invocation_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS agent_eval_cases (
|
||||
id TEXT PRIMARY KEY,
|
||||
suite_id TEXT NOT NULL,
|
||||
case_slug TEXT NOT NULL,
|
||||
case_version INTEGER NOT NULL DEFAULT 1 CHECK(case_version >= 1),
|
||||
prompt_sha256 TEXT NOT NULL,
|
||||
prompt_excerpt TEXT NOT NULL CHECK(length(prompt_excerpt) <= 1000),
|
||||
fixture_context_sha256 TEXT,
|
||||
fixture_context_excerpt TEXT CHECK(fixture_context_excerpt IS NULL OR length(fixture_context_excerpt) <= 2000),
|
||||
expected_route TEXT NOT NULL
|
||||
CHECK(expected_route IN (
|
||||
'none',
|
||||
'web_search',
|
||||
'social_trends',
|
||||
'structured_market_data',
|
||||
'local_context',
|
||||
'mixed',
|
||||
'unknown'
|
||||
)),
|
||||
expected_provider TEXT,
|
||||
must_use_tools_json TEXT NOT NULL DEFAULT '[]',
|
||||
must_not_use_tools_json TEXT NOT NULL DEFAULT '[]',
|
||||
tags_json TEXT NOT NULL DEFAULT '[]',
|
||||
rubric_json TEXT NOT NULL DEFAULT '{}',
|
||||
stale_after TEXT,
|
||||
active INTEGER NOT NULL DEFAULT 1 CHECK(active IN (0, 1)),
|
||||
secret_values_included INTEGER NOT NULL DEFAULT 0 CHECK(secret_values_included = 0),
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(suite_id, case_slug, case_version)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_eval_cases_suite_active
|
||||
ON agent_eval_cases(suite_id, active);
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_eval_cases_route
|
||||
ON agent_eval_cases(expected_route);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS agent_eval_results (
|
||||
id TEXT PRIMARY KEY,
|
||||
eval_case_id TEXT NOT NULL REFERENCES agent_eval_cases(id) ON DELETE CASCADE,
|
||||
research_run_id TEXT NOT NULL REFERENCES agent_research_runs(id) ON DELETE CASCADE,
|
||||
graph_evaluation_run_id TEXT REFERENCES graph_evaluation_runs(id) ON DELETE SET NULL,
|
||||
status TEXT NOT NULL
|
||||
CHECK(status IN ('passed', 'failed', 'warning', 'blocked', 'skipped')),
|
||||
score REAL CHECK(score IS NULL OR (score >= 0 AND score <= 1)),
|
||||
routing_correct INTEGER CHECK(routing_correct IS NULL OR routing_correct IN (0, 1)),
|
||||
tool_choice_score REAL CHECK(tool_choice_score IS NULL OR (tool_choice_score >= 0 AND tool_choice_score <= 1)),
|
||||
source_quality_score REAL CHECK(source_quality_score IS NULL OR (source_quality_score >= 0 AND source_quality_score <= 1)),
|
||||
groundedness_score REAL CHECK(groundedness_score IS NULL OR (groundedness_score >= 0 AND groundedness_score <= 1)),
|
||||
freshness_score REAL CHECK(freshness_score IS NULL OR (freshness_score >= 0 AND freshness_score <= 1)),
|
||||
cost_efficiency_score REAL CHECK(cost_efficiency_score IS NULL OR (cost_efficiency_score >= 0 AND cost_efficiency_score <= 1)),
|
||||
safety_payment_score REAL CHECK(safety_payment_score IS NULL OR (safety_payment_score >= 0 AND safety_payment_score <= 1)),
|
||||
failure_reason TEXT,
|
||||
judge TEXT,
|
||||
proof_ref TEXT,
|
||||
secret_values_included INTEGER NOT NULL DEFAULT 0 CHECK(secret_values_included = 0),
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(eval_case_id, research_run_id)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_eval_results_case_status
|
||||
ON agent_eval_results(eval_case_id, status);
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_eval_results_run
|
||||
ON agent_eval_results(research_run_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_eval_results_graph_eval
|
||||
ON agent_eval_results(graph_evaluation_run_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS work_order_graph_links (
|
||||
id TEXT PRIMARY KEY,
|
||||
sponsored_work_order_id TEXT NOT NULL,
|
||||
role TEXT NOT NULL
|
||||
CHECK(role IN (
|
||||
'input_context',
|
||||
'evaluation_target',
|
||||
'created_evidence',
|
||||
'created_claim',
|
||||
'created_eval_run',
|
||||
'research_run',
|
||||
'tool_trace',
|
||||
'history_trace',
|
||||
'outcome_trace'
|
||||
)),
|
||||
graph_layer TEXT NOT NULL
|
||||
CHECK(graph_layer IN (
|
||||
'persona',
|
||||
'strategy',
|
||||
'position',
|
||||
'belief',
|
||||
'claim',
|
||||
'evidence',
|
||||
'edge',
|
||||
'graph_evaluation_run',
|
||||
'cascade_event',
|
||||
'graph_history_event',
|
||||
'agent_research_run',
|
||||
'agent_tool_invocation',
|
||||
'agent_eval_result',
|
||||
'outcome_observation'
|
||||
)),
|
||||
graph_id TEXT NOT NULL,
|
||||
rationale TEXT,
|
||||
secret_values_included INTEGER NOT NULL DEFAULT 0 CHECK(secret_values_included = 0),
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(sponsored_work_order_id, role, graph_layer, graph_id)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_work_order_graph_links_work_order
|
||||
ON work_order_graph_links(sponsored_work_order_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_work_order_graph_links_graph
|
||||
ON work_order_graph_links(graph_layer, graph_id);
|
||||
|
|
@ -17,9 +17,7 @@ Identification:
|
|||
PR whose `source_path` matches one of those filenames is research-derived.
|
||||
|
||||
Touch list (per matched PR):
|
||||
1. UPDATE prs SET submitted_by = '<agent>' (canonical handle, lowercase, no
|
||||
trailing "(self-directed)" suffix — see lib/extract.py and
|
||||
diagnostics/activity_feed_api.py for why decorators leak into 404s)
|
||||
1. UPDATE prs SET submitted_by = '<agent> (self-directed)'
|
||||
2. DELETE FROM contribution_events
|
||||
WHERE handle='m3taversal' AND role='author' AND pr_number=?
|
||||
3. INSERT OR IGNORE INTO contribution_events with handle=<agent>,
|
||||
|
|
@ -171,7 +169,7 @@ def apply_backfill(conn: sqlite3.Connection, matches: list[dict], dry_run: bool)
|
|||
|
||||
if dry_run:
|
||||
logger.info(
|
||||
"would update pr=%d submitted_by '%s' → '%s' "
|
||||
"would update pr=%d submitted_by '%s' → '%s (self-directed)' "
|
||||
"[m3ta_event=%s, agent_event=%s]",
|
||||
pr, m["current_submitted_by"], agent,
|
||||
old_event_exists, new_event_exists,
|
||||
|
|
@ -183,10 +181,10 @@ def apply_backfill(conn: sqlite3.Connection, matches: list[dict], dry_run: bool)
|
|||
counters["events_to_insert"] += 1
|
||||
continue
|
||||
|
||||
# 1. UPDATE prs.submitted_by — canonical handle only.
|
||||
# 1. UPDATE prs.submitted_by
|
||||
conn.execute(
|
||||
"UPDATE prs SET submitted_by = ? WHERE number = ?",
|
||||
(agent, pr),
|
||||
(f"{agent} (self-directed)", pr),
|
||||
)
|
||||
counters["prs"] += 1
|
||||
|
||||
|
|
|
|||
|
|
@ -1,251 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Validate the repo-owned Crabbox and Leo CI contract.
|
||||
|
||||
This is intentionally no-network and dependency-free. It checks the local
|
||||
Crabbox config for bounded jobs/secret hygiene and exercises a small Leo route
|
||||
contract through the real Phase 1b router.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
if str(REPO_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
|
||||
from lib.agent_routing import classify_pr_route # noqa: E402
|
||||
|
||||
CRABBOX_CONFIG = REPO_ROOT / ".crabbox.yaml"
|
||||
CRABBOX_DOC = REPO_ROOT / "docs" / "crabbox.md"
|
||||
CRABBOX_SKILL = REPO_ROOT / ".agents" / "skills" / "crabbox" / "SKILL.md"
|
||||
CRABBOX_WORKFLOW = REPO_ROOT / ".github" / "workflows" / "crabbox.yml"
|
||||
CI_WORKFLOW = REPO_ROOT / ".github" / "workflows" / "ci.yml"
|
||||
|
||||
REQUIRED_JOBS = {
|
||||
"unit",
|
||||
"lint-phase1b",
|
||||
"phase1b-local-proof",
|
||||
"sync-smoke",
|
||||
"ci-contract",
|
||||
}
|
||||
REQUIRED_SYNC_EXCLUDES = {
|
||||
".cache",
|
||||
".venv",
|
||||
".pytest_cache",
|
||||
".ruff_cache",
|
||||
"__pycache__",
|
||||
"*.db",
|
||||
"*.db-wal",
|
||||
"*.db-shm",
|
||||
"*.log",
|
||||
"logs",
|
||||
"secrets",
|
||||
".env",
|
||||
"node_modules",
|
||||
}
|
||||
ALLOWED_ENV = {"CI", "PYTHONWARNINGS", "PHASE1B_AGENT_ROUTING_ENABLED"}
|
||||
FORBIDDEN_CONFIG_TOKENS = {
|
||||
"HCLOUD_TOKEN",
|
||||
"HETZNER_TOKEN",
|
||||
"CRABBOX_COORDINATOR_TOKEN",
|
||||
"GITHUB_TOKEN",
|
||||
"GH_TOKEN",
|
||||
"OPENROUTER",
|
||||
"FORGEJO",
|
||||
"BITWARDEN",
|
||||
"BW_SESSION",
|
||||
"SSH_PRIVATE",
|
||||
}
|
||||
|
||||
|
||||
def _read(path: Path) -> str:
|
||||
if not path.exists():
|
||||
raise AssertionError(f"missing required file: {path.relative_to(REPO_ROOT)}")
|
||||
return path.read_text()
|
||||
|
||||
|
||||
def _list_values_under(text: str, parent: str, child: str) -> list[str]:
|
||||
lines = text.splitlines()
|
||||
in_parent = False
|
||||
in_child = False
|
||||
values: list[str] = []
|
||||
|
||||
for line in lines:
|
||||
if not in_parent:
|
||||
if line == f"{parent}:":
|
||||
in_parent = True
|
||||
continue
|
||||
|
||||
if line and not line.startswith(" "):
|
||||
break
|
||||
|
||||
if not in_child:
|
||||
if line == f" {child}:":
|
||||
in_child = True
|
||||
continue
|
||||
|
||||
if line.startswith(" - "):
|
||||
values.append(line.removeprefix(" - ").strip().strip('"'))
|
||||
continue
|
||||
break
|
||||
|
||||
return values
|
||||
|
||||
|
||||
def _top_level_job_names(text: str) -> set[str]:
|
||||
jobs_match = re.search(r"(?ms)^jobs:\n(?P<body>.*?)(?:\n\S|\Z)", text)
|
||||
if not jobs_match:
|
||||
return set()
|
||||
return set(re.findall(r"^ ([A-Za-z0-9_-]+):\s*$", jobs_match.group("body"), flags=re.MULTILINE))
|
||||
|
||||
|
||||
def _diff_for(*paths: str, line: str = "+type: claim") -> str:
|
||||
return "\n".join(f"diff --git a/{path} b/{path}\n{line}" for path in paths)
|
||||
|
||||
|
||||
def _assert_equal(name: str, actual: Any, expected: Any) -> None:
|
||||
if actual != expected:
|
||||
raise AssertionError(f"{name}: expected {expected!r}, got {actual!r}")
|
||||
|
||||
|
||||
def _validate_leo_route_contract() -> dict[str, Any]:
|
||||
cases = [
|
||||
{
|
||||
"name": "leo_owned_domain",
|
||||
"route": classify_pr_route(_diff_for("domains/grand-strategy/strategy.md")),
|
||||
"required_agents": ["Leo"],
|
||||
"route_kind": "single",
|
||||
"fallback": False,
|
||||
},
|
||||
{
|
||||
"name": "leo_fallback",
|
||||
"route": classify_pr_route(_diff_for("docs/readme.md"), branch="misc/update"),
|
||||
"required_agents": ["Leo"],
|
||||
"route_kind": "fallback",
|
||||
"fallback": True,
|
||||
},
|
||||
{
|
||||
"name": "leo_cross_domain",
|
||||
"route": classify_pr_route(
|
||||
_diff_for(
|
||||
"foundations/collective-intelligence/collective-ai-goals.md",
|
||||
line="+Collective AI goals and AI systems self-understanding need review.",
|
||||
)
|
||||
),
|
||||
"required_agents": ["Leo", "Theseus"],
|
||||
"route_kind": "multi",
|
||||
"fallback": False,
|
||||
},
|
||||
{
|
||||
"name": "non_leo_single_domain",
|
||||
"route": classify_pr_route(_diff_for("domains/internet-finance/x402.md")),
|
||||
"required_agents": ["Rio"],
|
||||
"route_kind": "single",
|
||||
"fallback": False,
|
||||
},
|
||||
]
|
||||
|
||||
results = []
|
||||
for case in cases:
|
||||
route = case["route"]
|
||||
result = route.to_audit_dict()
|
||||
_assert_equal(f"{case['name']} required_agents", result["required_agents"], case["required_agents"])
|
||||
_assert_equal(f"{case['name']} route_kind", result["route_kind"], case["route_kind"])
|
||||
_assert_equal(f"{case['name']} fallback", result["fallback"], case["fallback"])
|
||||
results.append({"name": case["name"], "route": result})
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"cases": results,
|
||||
"contract": {
|
||||
"leo_required_when": [
|
||||
"grand-strategy or Leo-owned domain route",
|
||||
"no confident route fallback",
|
||||
"top-2 cross-domain route where Leo is one of the top owners",
|
||||
],
|
||||
"leo_not_universal_second_review": True,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _validate_crabbox_contract() -> dict[str, Any]:
|
||||
config = _read(CRABBOX_CONFIG)
|
||||
doc = _read(CRABBOX_DOC)
|
||||
skill = _read(CRABBOX_SKILL)
|
||||
crabbox_workflow = _read(CRABBOX_WORKFLOW)
|
||||
ci_workflow = _read(CI_WORKFLOW)
|
||||
|
||||
jobs = _top_level_job_names(config)
|
||||
missing_jobs = sorted(REQUIRED_JOBS - jobs)
|
||||
if missing_jobs:
|
||||
raise AssertionError(f"missing Crabbox jobs: {missing_jobs}")
|
||||
|
||||
sync_excludes = set(_list_values_under(config, "sync", "exclude"))
|
||||
missing_excludes = sorted(REQUIRED_SYNC_EXCLUDES - sync_excludes)
|
||||
if missing_excludes:
|
||||
raise AssertionError(f"missing sync excludes: {missing_excludes}")
|
||||
|
||||
allowed_env = set(_list_values_under(config, "env", "allow"))
|
||||
if allowed_env != ALLOWED_ENV:
|
||||
raise AssertionError(f"env allowlist must be {sorted(ALLOWED_ENV)}, got {sorted(allowed_env)}")
|
||||
|
||||
upper_config = config.upper()
|
||||
leaked_tokens = sorted(token for token in FORBIDDEN_CONFIG_TOKENS if token in upper_config)
|
||||
if leaked_tokens:
|
||||
raise AssertionError(f"secret-like token names must not appear in .crabbox.yaml: {leaked_tokens}")
|
||||
|
||||
if "scripts/check_crabbox_ci_contract.py" not in ci_workflow:
|
||||
raise AssertionError("ci.yml must run scripts/check_crabbox_ci_contract.py")
|
||||
if "scripts/crabbox_phase1b_proof.sh" not in ci_workflow:
|
||||
raise AssertionError("ci.yml must run scripts/crabbox_phase1b_proof.sh")
|
||||
if "crabbox_phase1b_proof.sh" not in config:
|
||||
raise AssertionError(".crabbox.yaml must run the Phase 1B proof wrapper")
|
||||
if "crabbox-ci-contract.json" not in config:
|
||||
raise AssertionError(".crabbox.yaml must download the CI contract proof")
|
||||
if "runs-on: [self-hosted" not in crabbox_workflow:
|
||||
raise AssertionError("crabbox hydration workflow must target the dynamic self-hosted runner label")
|
||||
|
||||
for job in REQUIRED_JOBS:
|
||||
if f"crabbox job run {job}" not in skill and f"`{job}`" not in skill:
|
||||
raise AssertionError(f"Crabbox skill must name allowed job {job}")
|
||||
|
||||
if "production deploy" not in doc.lower() or "not the production deploy system" not in doc.lower():
|
||||
raise AssertionError("docs/crabbox.md must preserve the production deploy boundary")
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"jobs": sorted(jobs),
|
||||
"required_jobs": sorted(REQUIRED_JOBS),
|
||||
"sync_excludes_checked": sorted(REQUIRED_SYNC_EXCLUDES),
|
||||
"env_allowlist": sorted(allowed_env),
|
||||
"secret_token_names_absent": sorted(FORBIDDEN_CONFIG_TOKENS),
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--output", default=".crabbox-results/crabbox-ci-contract.json")
|
||||
args = parser.parse_args()
|
||||
|
||||
proof = {
|
||||
"ok": True,
|
||||
"scope": "crabbox_ci_leo_contract",
|
||||
"crabbox": _validate_crabbox_contract(),
|
||||
"leo_route_contract": _validate_leo_route_contract(),
|
||||
}
|
||||
|
||||
output = REPO_ROOT / args.output
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
output.write_text(json.dumps(proof, indent=2, sort_keys=True) + "\n")
|
||||
print(json.dumps(proof, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
|
@ -1,185 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Validate the LLM refinement and decision-engine guidance surface."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
REQUIRED_FILES = {
|
||||
"program_doc": REPO_ROOT / "docs" / "llm-refinement-decision-engine.md",
|
||||
"model_registry": REPO_ROOT / "docs" / "model-discovery-registry.md",
|
||||
"replay_script": REPO_ROOT / "scripts" / "replay_decision_engine_eval.py",
|
||||
"decision_skill": REPO_ROOT / ".agents" / "skills" / "decision-engine-refinement" / "SKILL.md",
|
||||
"db_skill": REPO_ROOT / ".agents" / "skills" / "teleo-db-operator" / "SKILL.md",
|
||||
"kb_skill": REPO_ROOT / ".agents" / "skills" / "living-ip-kb-interop" / "SKILL.md",
|
||||
"hermes_skill": REPO_ROOT / ".agents" / "skills" / "nousresearch-hermes-agent" / "SKILL.md",
|
||||
"openclaw_skill": REPO_ROOT / ".agents" / "skills" / "openclaw-agent" / "SKILL.md",
|
||||
}
|
||||
|
||||
PROGRAM_REQUIRED_PHRASES = [
|
||||
"Pentagon.run should own disposable infrastructure",
|
||||
"This repo should own decision quality",
|
||||
"Rio becomes the economic and incentive-quality evaluator",
|
||||
"Theseus becomes the model-integrity and agent-refinement evaluator",
|
||||
"No model switch is accepted because it",
|
||||
"Default is read-only",
|
||||
"Model Discovery Registry",
|
||||
"Any Hermes, OpenClaw, or Claude-style agent",
|
||||
"Raw cards and secrets are not agent runtime inputs",
|
||||
"scripts/replay_decision_engine_eval.py",
|
||||
]
|
||||
|
||||
MODEL_REGISTRY_REQUIRED_PHRASES = [
|
||||
"candidate registry, not model approval",
|
||||
"GPT-5.5",
|
||||
"gpt-oss-20b",
|
||||
"Claude Opus 4.8",
|
||||
"Gemini 3.5 Flash",
|
||||
"Hermes 4 70B",
|
||||
"Qwen3.5 9B",
|
||||
"Zero false approvals on known-bad fixtures",
|
||||
]
|
||||
|
||||
REPLAY_REQUIRED_PHRASES = [
|
||||
"decision_engine_replay",
|
||||
"false_approve_count",
|
||||
"kb_interop_ok",
|
||||
"route_accuracy",
|
||||
]
|
||||
|
||||
SKILL_REQUIRED = {
|
||||
"decision_skill": [
|
||||
"Rio economics",
|
||||
"Theseus model integrity",
|
||||
"Do not change live model assignments",
|
||||
"baseline verdict output",
|
||||
],
|
||||
"db_skill": [
|
||||
"Default to read-only",
|
||||
"BEGIN IMMEDIATE",
|
||||
"Do not attach, copy, or commit `pipeline.db`",
|
||||
"review_records",
|
||||
],
|
||||
"kb_skill": [
|
||||
"propose-first",
|
||||
"kb.search",
|
||||
"Do not write directly to main",
|
||||
"teleo-db-operator",
|
||||
],
|
||||
"hermes_skill": [
|
||||
"model switching",
|
||||
"fixture-first",
|
||||
"Rio Hermes package",
|
||||
"Theseus Hermes package",
|
||||
"living-ip-kb-interop",
|
||||
],
|
||||
"openclaw_skill": [
|
||||
"AGENTS.md",
|
||||
"SOUL.md",
|
||||
"TOOLS.md",
|
||||
"Default deny",
|
||||
"living-ip-kb-interop",
|
||||
],
|
||||
}
|
||||
|
||||
FIXTURE_REQUIRED = {
|
||||
"rio_meteora_lp_incentives.json": ["rio-economics", "paid_query_effects", "Rio"],
|
||||
"theseus_live_model_switch_reject.json": [
|
||||
"theseus-model-integrity",
|
||||
"model_assignment_without_eval",
|
||||
"Theseus",
|
||||
],
|
||||
"kb_interop_propose_only.json": ["kb-interop", "no_prod_db_write", "Theseus"],
|
||||
}
|
||||
|
||||
|
||||
def _read(path: Path) -> str:
|
||||
if not path.exists():
|
||||
raise AssertionError(f"missing file: {path.relative_to(REPO_ROOT)}")
|
||||
return path.read_text()
|
||||
|
||||
|
||||
def _assert_frontmatter(path: Path, text: str) -> None:
|
||||
match = re.match(r"^---\n(?P<body>.*?)\n---\n", text, flags=re.DOTALL)
|
||||
if not match:
|
||||
raise AssertionError(f"{path.relative_to(REPO_ROOT)} missing YAML frontmatter")
|
||||
body = match.group("body")
|
||||
if "name:" not in body or "description:" not in body:
|
||||
raise AssertionError(f"{path.relative_to(REPO_ROOT)} frontmatter needs name and description")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--output", default=".crabbox-results/llm-refinement-contract.json")
|
||||
args = parser.parse_args()
|
||||
|
||||
program = _read(REQUIRED_FILES["program_doc"])
|
||||
missing_program = [phrase for phrase in PROGRAM_REQUIRED_PHRASES if phrase not in program]
|
||||
if missing_program:
|
||||
raise AssertionError(f"program doc missing phrases: {missing_program}")
|
||||
|
||||
model_registry = _read(REQUIRED_FILES["model_registry"])
|
||||
missing_registry = [phrase for phrase in MODEL_REGISTRY_REQUIRED_PHRASES if phrase not in model_registry]
|
||||
if missing_registry:
|
||||
raise AssertionError(f"model registry missing phrases: {missing_registry}")
|
||||
|
||||
replay_script = _read(REQUIRED_FILES["replay_script"])
|
||||
missing_replay = [phrase for phrase in REPLAY_REQUIRED_PHRASES if phrase not in replay_script]
|
||||
if missing_replay:
|
||||
raise AssertionError(f"replay script missing phrases: {missing_replay}")
|
||||
|
||||
fixture_checks = {}
|
||||
fixtures_dir = REPO_ROOT / "fixtures" / "decision-engine-eval"
|
||||
for filename, phrases in FIXTURE_REQUIRED.items():
|
||||
path = fixtures_dir / filename
|
||||
text = _read(path)
|
||||
missing = [phrase for phrase in phrases if phrase not in text]
|
||||
if missing:
|
||||
raise AssertionError(f"{path.relative_to(REPO_ROOT)} missing phrases: {missing}")
|
||||
fixture_checks[filename] = {
|
||||
"path": str(path.relative_to(REPO_ROOT)),
|
||||
"phrases_checked": phrases,
|
||||
}
|
||||
|
||||
skill_checks = {}
|
||||
for key, phrases in SKILL_REQUIRED.items():
|
||||
path = REQUIRED_FILES[key]
|
||||
text = _read(path)
|
||||
_assert_frontmatter(path, text)
|
||||
missing = [phrase for phrase in phrases if phrase not in text]
|
||||
if missing:
|
||||
raise AssertionError(f"{path.relative_to(REPO_ROOT)} missing phrases: {missing}")
|
||||
skill_checks[key] = {
|
||||
"path": str(path.relative_to(REPO_ROOT)),
|
||||
"phrases_checked": phrases,
|
||||
}
|
||||
|
||||
proof = {
|
||||
"ok": True,
|
||||
"scope": "llm_refinement_decision_engine_contract",
|
||||
"program_doc": str(REQUIRED_FILES["program_doc"].relative_to(REPO_ROOT)),
|
||||
"model_registry": str(REQUIRED_FILES["model_registry"].relative_to(REPO_ROOT)),
|
||||
"program_phrases_checked": PROGRAM_REQUIRED_PHRASES,
|
||||
"model_registry_phrases_checked": MODEL_REGISTRY_REQUIRED_PHRASES,
|
||||
"fixtures": fixture_checks,
|
||||
"skills": skill_checks,
|
||||
"pivot": {
|
||||
"infra_owner": "Pentagon.run",
|
||||
"repo_owner": "decision quality, rubrics, model evals, prompt/tool refinement, DB feedback loops",
|
||||
},
|
||||
}
|
||||
|
||||
output = REPO_ROOT / args.output
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
output.write_text(json.dumps(proof, indent=2, sort_keys=True) + "\n")
|
||||
print(json.dumps(proof, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
|
@ -1,228 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Verify the disposable Leo wallet-test Telegram runtime without leaking tokens."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
TOKEN_RE = re.compile(r"^\d{6,12}:[A-Za-z0-9_-]{25,}$")
|
||||
|
||||
|
||||
def repo_root_from_script() -> Path:
|
||||
return Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def parse_args(argv: list[str]) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Check the Leo wallet-test Telegram bot token, getMe identity, and service state.",
|
||||
allow_abbrev=False,
|
||||
)
|
||||
parser.add_argument("--agent", default="leo-wallet-test", help="telegram/agents/<agent>.yaml")
|
||||
parser.add_argument("--repo-root", default=str(repo_root_from_script()))
|
||||
parser.add_argument("--secrets-dir", default="/opt/teleo-eval/secrets")
|
||||
parser.add_argument("--skip-getme", action="store_true", help="Do not call Telegram getMe")
|
||||
parser.add_argument("--require-token", action="store_true", help="Exit nonzero when token file is missing")
|
||||
parser.add_argument("--require-service-active", action="store_true", help="Exit nonzero unless systemd says active")
|
||||
parser.add_argument("--output", default="docs/reports/telegram-leo-wallet-test-runtime-proof.json")
|
||||
|
||||
namespace, unknown = parser.parse_known_args(argv)
|
||||
if unknown:
|
||||
print(
|
||||
"ERROR: Unsupported arguments were provided. Secret-bearing CLI args are not accepted.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
raise SystemExit(2)
|
||||
return namespace
|
||||
|
||||
|
||||
def load_agent_config(repo_root: Path, agent: str):
|
||||
telegram_dir = repo_root / "telegram"
|
||||
sys.path.insert(0, str(telegram_dir))
|
||||
from agent_config import load_agent_config as load_config
|
||||
|
||||
config_path = telegram_dir / "agents" / f"{agent}.yaml"
|
||||
config = load_config(str(config_path))
|
||||
return config_path, config
|
||||
|
||||
|
||||
def token_path_for(secrets_dir: Path, bot_token_file: str) -> Path:
|
||||
token_file = Path(bot_token_file)
|
||||
if token_file.name != bot_token_file or token_file.name in {"", ".", ".."}:
|
||||
raise ValueError("bot_token_file must be a plain filename")
|
||||
return secrets_dir / token_file.name
|
||||
|
||||
|
||||
def run_command(command: list[str]) -> dict:
|
||||
try:
|
||||
proc = subprocess.run(command, check=False, text=True, capture_output=True)
|
||||
except FileNotFoundError:
|
||||
return {
|
||||
"command": command,
|
||||
"returncode": 127,
|
||||
"stdout": "",
|
||||
"stderr": "command_unavailable",
|
||||
}
|
||||
return {
|
||||
"command": command,
|
||||
"returncode": proc.returncode,
|
||||
"stdout": proc.stdout.strip(),
|
||||
"stderr": proc.stderr.strip(),
|
||||
}
|
||||
|
||||
|
||||
def systemd_state(unit: str) -> dict:
|
||||
active = run_command(["systemctl", "is-active", unit])
|
||||
enabled = run_command(["systemctl", "is-enabled", unit])
|
||||
return {
|
||||
"unit": unit,
|
||||
"active": active["stdout"] or "unknown",
|
||||
"activeReturncode": active["returncode"],
|
||||
"enabled": enabled["stdout"] or "unknown",
|
||||
"enabledReturncode": enabled["returncode"],
|
||||
}
|
||||
|
||||
|
||||
def telegram_get_me(token: str, *, timeout_seconds: int = 20) -> dict:
|
||||
url = f"https://api.telegram.org/bot{token}/getMe"
|
||||
request = urllib.request.Request(url, headers={"Accept": "application/json"})
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=timeout_seconds) as response:
|
||||
status = response.status
|
||||
body = json.loads(response.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as exc:
|
||||
status = exc.code
|
||||
try:
|
||||
body = json.loads(exc.read().decode("utf-8"))
|
||||
except Exception:
|
||||
body = {"ok": False, "error_code": exc.code, "description": "non_json_http_error"}
|
||||
except Exception as exc:
|
||||
return {
|
||||
"attempted": True,
|
||||
"httpStatus": None,
|
||||
"ok": False,
|
||||
"errorType": type(exc).__name__,
|
||||
"secretValuesIncluded": False,
|
||||
}
|
||||
|
||||
result = body.get("result") if isinstance(body, dict) else None
|
||||
return {
|
||||
"attempted": True,
|
||||
"httpStatus": status,
|
||||
"ok": bool(body.get("ok")) if isinstance(body, dict) else False,
|
||||
"botIdPresent": isinstance(result, dict) and bool(result.get("id")),
|
||||
"isBot": result.get("is_bot") if isinstance(result, dict) else None,
|
||||
"username": result.get("username") if isinstance(result, dict) else None,
|
||||
"firstName": result.get("first_name") if isinstance(result, dict) else None,
|
||||
"canJoinGroups": result.get("can_join_groups") if isinstance(result, dict) else None,
|
||||
"canReadAllGroupMessages": result.get("can_read_all_group_messages") if isinstance(result, dict) else None,
|
||||
"supportsInlineQueries": result.get("supports_inline_queries") if isinstance(result, dict) else None,
|
||||
"secretValuesIncluded": False,
|
||||
}
|
||||
|
||||
|
||||
def build_proof(args: argparse.Namespace) -> tuple[dict, int]:
|
||||
repo_root = Path(args.repo_root).resolve()
|
||||
secrets_dir = Path(args.secrets_dir)
|
||||
config_path, config = load_agent_config(repo_root, args.agent)
|
||||
token_path = token_path_for(secrets_dir, config.bot_token_file)
|
||||
unit = f"teleo-agent@{args.agent}.service"
|
||||
|
||||
token_file_present = token_path.exists()
|
||||
token_shape_valid = False
|
||||
get_me = {"attempted": False, "ok": False, "secretValuesIncluded": False}
|
||||
exact_blocker = None
|
||||
token = None
|
||||
|
||||
if token_file_present:
|
||||
token = token_path.read_text().strip()
|
||||
token_shape_valid = bool(TOKEN_RE.match(token))
|
||||
if not token_shape_valid:
|
||||
exact_blocker = "telegram_token_shape_invalid"
|
||||
else:
|
||||
exact_blocker = "telegram_token_file_missing"
|
||||
|
||||
if token_file_present and token_shape_valid and not args.skip_getme:
|
||||
get_me = telegram_get_me(token)
|
||||
if not get_me.get("ok"):
|
||||
exact_blocker = "telegram_getme_failed"
|
||||
|
||||
expected_username = config.handle.lstrip("@")
|
||||
username_matches = (
|
||||
bool(get_me.get("username"))
|
||||
and get_me.get("username", "").lower() == expected_username.lower()
|
||||
)
|
||||
if get_me.get("attempted") and get_me.get("ok") and not username_matches:
|
||||
exact_blocker = "telegram_getme_username_mismatch"
|
||||
|
||||
service = systemd_state(unit)
|
||||
service_active = service["active"] == "active"
|
||||
if args.require_service_active and not service_active:
|
||||
exact_blocker = exact_blocker or "telegram_service_inactive"
|
||||
|
||||
ok = (
|
||||
token_file_present
|
||||
and token_shape_valid
|
||||
and (args.skip_getme or (get_me.get("ok") and username_matches))
|
||||
and (service_active or not args.require_service_active)
|
||||
)
|
||||
if args.require_token and not token_file_present:
|
||||
ok = False
|
||||
|
||||
proof = {
|
||||
"schema": "livingip.telegramLeoWalletTestRuntimeProof.v1",
|
||||
"generatedAt": datetime.now(timezone.utc).isoformat(),
|
||||
"ok": ok,
|
||||
"requiredTier": "T3_live_readonly",
|
||||
"currentTier": "T3_live_readonly" if ok else "T2_runtime",
|
||||
"agent": args.agent,
|
||||
"configPath": str(config_path),
|
||||
"expectedHandle": config.handle,
|
||||
"expectedUsername": expected_username,
|
||||
"tokenPath": str(token_path),
|
||||
"tokenFilePresent": token_file_present,
|
||||
"tokenShapeValid": token_shape_valid,
|
||||
"getMe": get_me,
|
||||
"usernameMatchesExpected": username_matches if get_me.get("attempted") else None,
|
||||
"service": service,
|
||||
"secretValuesIncluded": False,
|
||||
"exactBlocker": exact_blocker,
|
||||
"notProven": [
|
||||
"Telegram message delivery",
|
||||
"Telegram reply delivery",
|
||||
"Telegram-triggered x402 readback",
|
||||
"Telegram-triggered paid execution",
|
||||
],
|
||||
"strongestClaimAllowed": (
|
||||
"This verifier proves the disposable Leo wallet-test Telegram token identity and service state "
|
||||
"after BotFather token installation. It does not send Telegram messages or prove x402 payment execution."
|
||||
),
|
||||
}
|
||||
exit_code = 0 if ok or (exact_blocker == "telegram_token_file_missing" and not args.require_token) else 1
|
||||
return proof, exit_code
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = parse_args(sys.argv[1:] if argv is None else argv)
|
||||
proof, exit_code = build_proof(args)
|
||||
output = json.dumps(proof, indent=2, sort_keys=True) + "\n"
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_text(output)
|
||||
print(output, end="")
|
||||
return exit_code
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
raise SystemExit(main())
|
||||
except ValueError as exc:
|
||||
print(f"ERROR: {exc}", file=sys.stderr)
|
||||
raise SystemExit(1) from None
|
||||
|
|
@ -1,88 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Prove the Telegram Leo bridge can consume the public Leo HTTP chat route."""
|
||||
|
||||
# ruff: noqa: E402, I001
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
TELEGRAM_DIR = REPO_ROOT / "telegram"
|
||||
sys.path.insert(0, str(TELEGRAM_DIR))
|
||||
|
||||
from http_chat_proxy import build_chat_proxy_payload, post_chat_proxy
|
||||
|
||||
|
||||
DEFAULT_URL = "https://leo.livingip.xyz/api/agents/leo/chat"
|
||||
DEFAULT_OUTPUT = "docs/reports/telegram-leo-x402-bridge-proof.json"
|
||||
|
||||
|
||||
async def run_check(url: str, message: str) -> dict:
|
||||
payload = build_chat_proxy_payload(
|
||||
message=message,
|
||||
source="telegram-proof",
|
||||
agent="leo",
|
||||
chat_id=0,
|
||||
message_id=0,
|
||||
username="codex-proof",
|
||||
)
|
||||
status, body, reply = await post_chat_proxy(url=url, payload=payload)
|
||||
return {
|
||||
"schema": "livingip.telegramLeoX402BridgeProof.v1",
|
||||
"generatedAt": datetime.now(timezone.utc).isoformat(),
|
||||
"ok": bool(reply),
|
||||
"requiredTier": "T3_live_readonly",
|
||||
"currentTier": "T3_live_readonly" if reply else "T2_runtime",
|
||||
"url": url,
|
||||
"httpStatus": status,
|
||||
"routeSchema": body.get("schema") if isinstance(body, dict) else None,
|
||||
"agent": body.get("agent") if isinstance(body, dict) else None,
|
||||
"llmOk": (
|
||||
body.get("llmOk")
|
||||
if isinstance(body, dict) and "llmOk" in body
|
||||
else body.get("llm", {}).get("ok")
|
||||
if isinstance(body.get("llm"), dict)
|
||||
else None
|
||||
),
|
||||
"reply": reply,
|
||||
"secretValuesIncluded": False,
|
||||
"strongestClaimAllowed": (
|
||||
"Telegram bridge helper can POST a no-secret payload to the public Leo HTTP chat route "
|
||||
"and extract a usable Leo reply. This proves the bridge parser/readback only; it does "
|
||||
"not prove the Telegram bot service is deployed or active."
|
||||
),
|
||||
"notProven": [
|
||||
"teleo-agent@leo.service active",
|
||||
"Telegram message delivery",
|
||||
"Telegram reply delivery",
|
||||
"new payment execution",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--url", default=DEFAULT_URL)
|
||||
parser.add_argument(
|
||||
"--message",
|
||||
default="Telegram bridge proof: reply with one sentence confirming this reached Leo HTTP.",
|
||||
)
|
||||
parser.add_argument("--output", default=DEFAULT_OUTPUT)
|
||||
args = parser.parse_args()
|
||||
|
||||
proof = asyncio.run(run_check(args.url, args.message))
|
||||
output_path = REPO_ROOT / args.output
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_text(json.dumps(proof, indent=2, sort_keys=True) + "\n")
|
||||
print(json.dumps(proof, indent=2, sort_keys=True))
|
||||
return 0 if proof["ok"] else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
|
@ -1,91 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Prove the Telegram Leo bridge can consume the public smart-research route."""
|
||||
|
||||
# ruff: noqa: E402, I001
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
TELEGRAM_DIR = REPO_ROOT / "telegram"
|
||||
sys.path.insert(0, str(TELEGRAM_DIR))
|
||||
|
||||
from http_chat_proxy import build_smart_research_proxy_payload, post_chat_proxy
|
||||
|
||||
|
||||
DEFAULT_URL = "https://leo.livingip.xyz/api/agents/leo/research"
|
||||
DEFAULT_OUTPUT = "docs/reports/telegram-leo-x402-smart-research-bridge-proof.json"
|
||||
|
||||
|
||||
async def run_check(url: str, research_goal: str) -> dict:
|
||||
payload = build_smart_research_proxy_payload(
|
||||
research_goal=research_goal,
|
||||
source="telegram-proof",
|
||||
agent="leo",
|
||||
chat_id=0,
|
||||
message_id=0,
|
||||
username="codex-proof",
|
||||
allow_paid_execution=False,
|
||||
max_amount_usd=0.01,
|
||||
include_synthesis=True,
|
||||
)
|
||||
status, body, reply = await post_chat_proxy(url=url, payload=payload, timeout_seconds=90)
|
||||
funds_moved = bool(body.get("fundsMoved")) if isinstance(body, dict) else False
|
||||
selected_provider = body.get("selectedProvider") if isinstance(body, dict) else None
|
||||
exact_blocker = body.get("exactBlocker") if isinstance(body, dict) else None
|
||||
return {
|
||||
"schema": "livingip.telegramLeoX402SmartResearchBridgeProof.v1",
|
||||
"generatedAt": datetime.now(timezone.utc).isoformat(),
|
||||
"ok": bool(reply) and status in {200, 402} and not funds_moved,
|
||||
"requiredTier": "T3_live_readonly",
|
||||
"currentTier": body.get("currentTier", "T2_runtime") if isinstance(body, dict) else "T2_runtime",
|
||||
"url": url,
|
||||
"httpStatus": status,
|
||||
"routeSchema": body.get("schema") if isinstance(body, dict) else None,
|
||||
"selectedProvider": selected_provider,
|
||||
"exactBlocker": exact_blocker,
|
||||
"reply": reply,
|
||||
"paidPostAttempted": bool(body.get("paidPostAttempted")) if isinstance(body, dict) else False,
|
||||
"fundsMoved": funds_moved,
|
||||
"secretValuesIncluded": False,
|
||||
"strongestClaimAllowed": (
|
||||
"Telegram bridge helper can POST a no-secret smart-research payload to the public Leo "
|
||||
"research route and extract a usable fail-closed reply. This proves route shape and "
|
||||
"readback only; it does not prove a Telegram bot service is deployed or a paid Telegram "
|
||||
"message executed."
|
||||
),
|
||||
"notProven": [
|
||||
"teleo-agent@leo-wallet-test.service active",
|
||||
"Telegram message delivery",
|
||||
"Telegram reply delivery",
|
||||
"Telegram-triggered paid execution",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--url", default=DEFAULT_URL)
|
||||
parser.add_argument(
|
||||
"--research-goal",
|
||||
default="Find current public evidence on x402 agent payments and recommend what Living IP Leo should test next.",
|
||||
)
|
||||
parser.add_argument("--output", default=DEFAULT_OUTPUT)
|
||||
args = parser.parse_args()
|
||||
|
||||
proof = asyncio.run(run_check(args.url, args.research_goal))
|
||||
output_path = REPO_ROOT / args.output
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_text(json.dumps(proof, indent=2, sort_keys=True) + "\n")
|
||||
print(json.dumps(proof, indent=2, sort_keys=True))
|
||||
return 0 if proof["ok"] else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
|
@ -1,52 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$ROOT"
|
||||
PYTHON_BIN="${PYTHON:-python3}"
|
||||
|
||||
mkdir -p proof .crabbox-results
|
||||
|
||||
"$PYTHON_BIN" scripts/check_crabbox_ci_contract.py \
|
||||
--output .crabbox-results/crabbox-ci-contract.json
|
||||
|
||||
"$PYTHON_BIN" -m pytest \
|
||||
tests/test_agent_routing.py \
|
||||
tests/test_evaluate_agent_routing.py \
|
||||
tests/test_phase1b_end_to_end.py \
|
||||
tests/test_eval_parse.py \
|
||||
tests/test_contributor.py \
|
||||
tests/test_search.py \
|
||||
--junitxml=.crabbox-results/phase1b-pytest.xml
|
||||
|
||||
PHASE1B_AGENT_ROUTING_ENABLED=true \
|
||||
"$PYTHON_BIN" scripts/prove_phase1b_local.py \
|
||||
--output proof/phase1b-local-e2e-proof.json
|
||||
|
||||
"$PYTHON_BIN" - <<'PY'
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
proof_path = Path("proof/phase1b-local-e2e-proof.json")
|
||||
proof = json.loads(proof_path.read_text())
|
||||
contract = json.loads(Path(".crabbox-results/crabbox-ci-contract.json").read_text())
|
||||
summary = {
|
||||
"ok": proof.get("ok") is True,
|
||||
"scope": proof.get("scope"),
|
||||
"schema_version": proof.get("schema_version"),
|
||||
"crabbox_ci_contract_ok": contract.get("ok") is True,
|
||||
"leo_route_contract_ok": contract.get("leo_route_contract", {}).get("ok") is True,
|
||||
"agents_seen": proof.get("agents_seen", []),
|
||||
"cases_total": proof.get("cases_total"),
|
||||
"succeeded": proof.get("succeeded"),
|
||||
"failed": proof.get("failed"),
|
||||
}
|
||||
if not summary["ok"]:
|
||||
raise SystemExit(f"phase1b proof failed: {summary}")
|
||||
if len(summary["agents_seen"]) != 6:
|
||||
raise SystemExit(f"expected six agents, got {summary['agents_seen']}")
|
||||
Path(".crabbox-results/phase1b-proof-summary.json").write_text(
|
||||
json.dumps(summary, indent=2, sort_keys=True) + "\n"
|
||||
)
|
||||
print(json.dumps(summary, indent=2, sort_keys=True))
|
||||
PY
|
||||
|
|
@ -1,184 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Install a Telegram agent bot token without printing the secret value."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import getpass
|
||||
import grp
|
||||
import json
|
||||
import os
|
||||
import pwd
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
TOKEN_RE = re.compile(r"^\d{6,12}:[A-Za-z0-9_-]{25,}$")
|
||||
|
||||
|
||||
def repo_root_from_script() -> Path:
|
||||
return Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def parse_args(argv: list[str]) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Install a Telegram agent token from stdin or a hidden prompt.",
|
||||
allow_abbrev=False,
|
||||
)
|
||||
parser.add_argument("--agent", default="leo-wallet-test", help="telegram/agents/<agent>.yaml")
|
||||
parser.add_argument("--repo-root", default=str(repo_root_from_script()))
|
||||
parser.add_argument("--secrets-dir", default="/opt/teleo-eval/secrets")
|
||||
parser.add_argument("--from-stdin", action="store_true", help="Read the token from stdin instead of a prompt")
|
||||
parser.add_argument("--owner", default="teleo", help="Token file owner after write")
|
||||
parser.add_argument("--group", default="teleo", help="Token file group after write")
|
||||
parser.add_argument("--no-chown", action="store_true", help="Skip chown; useful for local tests")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Validate paths and input without writing")
|
||||
parser.add_argument("--start-service", action="store_true", help="Run systemctl start teleo-agent@<agent>.service")
|
||||
parser.add_argument("--enable-service", action="store_true", help="Run systemctl enable teleo-agent@<agent>.service")
|
||||
parser.add_argument("--skip-validate", action="store_true", help="Skip agent_runner.py --validate")
|
||||
parser.add_argument("--output", help="Optional sanitized JSON proof path")
|
||||
|
||||
namespace, unknown = parser.parse_known_args(argv)
|
||||
if unknown:
|
||||
print(
|
||||
"ERROR: Unsupported arguments were provided. Secret-bearing CLI args are not accepted.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
raise SystemExit(2)
|
||||
return namespace
|
||||
|
||||
|
||||
def read_token(*, from_stdin: bool) -> str:
|
||||
token = sys.stdin.read() if from_stdin else getpass.getpass("Telegram bot token: ")
|
||||
return token.strip()
|
||||
|
||||
|
||||
def validate_token(token: str) -> None:
|
||||
if not TOKEN_RE.match(token):
|
||||
raise ValueError("Telegram bot token shape is invalid")
|
||||
|
||||
|
||||
def load_agent_config(repo_root: Path, agent: str):
|
||||
telegram_dir = repo_root / "telegram"
|
||||
sys.path.insert(0, str(telegram_dir))
|
||||
from agent_config import load_agent_config as load_config
|
||||
|
||||
config_path = telegram_dir / "agents" / f"{agent}.yaml"
|
||||
config = load_config(str(config_path))
|
||||
return config_path, config
|
||||
|
||||
|
||||
def token_path_for(secrets_dir: Path, bot_token_file: str) -> Path:
|
||||
token_file = Path(bot_token_file)
|
||||
if token_file.name != bot_token_file or token_file.name in {"", ".", ".."}:
|
||||
raise ValueError("bot_token_file must be a plain filename")
|
||||
return secrets_dir / token_file.name
|
||||
|
||||
|
||||
def resolve_owner_group(owner: str, group: str, *, no_chown: bool) -> tuple[int | None, int | None]:
|
||||
if no_chown:
|
||||
return None, None
|
||||
return pwd.getpwnam(owner).pw_uid, grp.getgrnam(group).gr_gid
|
||||
|
||||
|
||||
def write_token_file(token: str, token_path: Path, *, uid: int | None, gid: int | None, dry_run: bool) -> None:
|
||||
if dry_run:
|
||||
return
|
||||
|
||||
token_path.parent.mkdir(parents=True, mode=0o700, exist_ok=True)
|
||||
tmp_path = token_path.with_name(f".{token_path.name}.tmp-{os.getpid()}")
|
||||
flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL
|
||||
fd = os.open(tmp_path, flags, 0o600)
|
||||
try:
|
||||
with os.fdopen(fd, "w") as handle:
|
||||
handle.write(token)
|
||||
handle.write("\n")
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
os.chmod(tmp_path, 0o600)
|
||||
if uid is not None or gid is not None:
|
||||
os.chown(tmp_path, -1 if uid is None else uid, -1 if gid is None else gid)
|
||||
os.replace(tmp_path, token_path)
|
||||
os.chmod(token_path, 0o600)
|
||||
finally:
|
||||
if tmp_path.exists():
|
||||
tmp_path.unlink()
|
||||
|
||||
|
||||
def run_command(command: list[str], *, dry_run: bool) -> dict:
|
||||
if dry_run:
|
||||
return {"command": command, "skipped": "dry_run"}
|
||||
proc = subprocess.run(command, check=False, text=True, capture_output=True)
|
||||
return {
|
||||
"command": command,
|
||||
"returncode": proc.returncode,
|
||||
"stdout": proc.stdout.strip(),
|
||||
"stderr": proc.stderr.strip(),
|
||||
}
|
||||
|
||||
|
||||
def validate_agent(repo_root: Path, agent: str, *, dry_run: bool, skip_validate: bool) -> dict | None:
|
||||
if skip_validate:
|
||||
return None
|
||||
runner = repo_root / "telegram" / "agent_runner.py"
|
||||
return run_command([sys.executable, str(runner), "--agent", agent, "--validate"], dry_run=dry_run)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = parse_args(sys.argv[1:] if argv is None else argv)
|
||||
repo_root = Path(args.repo_root).resolve()
|
||||
secrets_dir = Path(args.secrets_dir)
|
||||
token = read_token(from_stdin=args.from_stdin)
|
||||
|
||||
config_path, config = load_agent_config(repo_root, args.agent)
|
||||
token_path = token_path_for(secrets_dir, config.bot_token_file)
|
||||
validate_token(token)
|
||||
uid, gid = resolve_owner_group(args.owner, args.group, no_chown=args.no_chown)
|
||||
|
||||
write_token_file(token, token_path, uid=uid, gid=gid, dry_run=args.dry_run)
|
||||
validate_result = validate_agent(repo_root, args.agent, dry_run=args.dry_run, skip_validate=args.skip_validate)
|
||||
|
||||
unit = f"teleo-agent@{args.agent}.service"
|
||||
enable_result = None
|
||||
start_result = None
|
||||
if args.enable_service:
|
||||
enable_result = run_command(["systemctl", "enable", unit], dry_run=args.dry_run)
|
||||
if args.start_service:
|
||||
start_result = run_command(["systemctl", "start", unit], dry_run=args.dry_run)
|
||||
|
||||
proof = {
|
||||
"schema": "livingip.telegramAgentTokenInstall.v1",
|
||||
"generatedAt": datetime.now(timezone.utc).isoformat(),
|
||||
"ok": True,
|
||||
"agent": args.agent,
|
||||
"configPath": str(config_path),
|
||||
"tokenPath": str(token_path),
|
||||
"tokenFileWritten": not args.dry_run,
|
||||
"tokenMode": "0600",
|
||||
"owner": None if args.no_chown else args.owner,
|
||||
"group": None if args.no_chown else args.group,
|
||||
"dryRun": args.dry_run,
|
||||
"validation": validate_result,
|
||||
"serviceUnit": unit,
|
||||
"serviceEnabled": enable_result,
|
||||
"serviceStarted": start_result,
|
||||
"secretValuesIncluded": False,
|
||||
}
|
||||
|
||||
output = json.dumps(proof, indent=2, sort_keys=True) + "\n"
|
||||
if args.output:
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_text(output)
|
||||
print(output, end="")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
raise SystemExit(main())
|
||||
except ValueError as exc:
|
||||
print(f"ERROR: {exc}", file=sys.stderr)
|
||||
raise SystemExit(1) from None
|
||||
|
|
@ -1,203 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Install Telegram smart-research paid gates without printing approval refs."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import getpass
|
||||
import grp
|
||||
import json
|
||||
import os
|
||||
import pwd
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
APPROVAL_REF_RE = re.compile(r"^[A-Za-z0-9._:@/-]{8,256}$")
|
||||
CHAT_ID_RE = re.compile(r"^-?\d+$")
|
||||
MAX_SMART_RESEARCH_USD = 0.06
|
||||
|
||||
|
||||
def parse_args(argv: list[str]) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Install server-side gates for Leo Telegram smart-research paid execution.",
|
||||
allow_abbrev=False,
|
||||
)
|
||||
parser.add_argument("--agent", default="leo-wallet-test", help="Agent instance name for teleo-agent@<agent>")
|
||||
parser.add_argument("--secrets-dir", default="/opt/teleo-eval/secrets")
|
||||
parser.add_argument("--allow-paid", action="store_true", help="Enable paid smart research for one allowed chat")
|
||||
parser.add_argument("--allowed-chat-id", help="Telegram chat id allowed to trigger paid smart research")
|
||||
parser.add_argument("--max-usd", default="0.01", help="Maximum spend per Telegram smart-research call")
|
||||
parser.add_argument(
|
||||
"--approval-ref-from-stdin",
|
||||
action="store_true",
|
||||
help="Read approval ref from stdin instead of a hidden prompt",
|
||||
)
|
||||
parser.add_argument("--owner", default="teleo", help="Env/approval file owner after write")
|
||||
parser.add_argument("--group", default="teleo", help="Env/approval file group after write")
|
||||
parser.add_argument("--no-chown", action="store_true", help="Skip chown; useful for local tests")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Validate inputs without writing files")
|
||||
parser.add_argument("--restart-service", action="store_true", help="Restart teleo-agent@<agent>.service")
|
||||
parser.add_argument("--output", help="Optional sanitized JSON proof path")
|
||||
|
||||
namespace, unknown = parser.parse_known_args(argv)
|
||||
if unknown:
|
||||
print(
|
||||
"ERROR: Unsupported arguments were provided. Secret-bearing CLI args are not accepted.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
raise SystemExit(2)
|
||||
return namespace
|
||||
|
||||
|
||||
def read_approval_ref(*, from_stdin: bool) -> str:
|
||||
approval_ref = sys.stdin.read() if from_stdin else getpass.getpass("Leo smart-research approval ref: ")
|
||||
return approval_ref.strip()
|
||||
|
||||
|
||||
def validate_agent_name(agent: str) -> None:
|
||||
if not re.match(r"^[A-Za-z0-9_.-]+$", agent):
|
||||
raise ValueError("agent must contain only letters, numbers, dot, dash, or underscore")
|
||||
|
||||
|
||||
def validate_max_usd(value: str) -> str:
|
||||
try:
|
||||
parsed = float(value)
|
||||
except ValueError:
|
||||
raise ValueError("max-usd must be numeric") from None
|
||||
if parsed <= 0 or parsed > MAX_SMART_RESEARCH_USD:
|
||||
raise ValueError(f"max-usd must be greater than 0 and no more than {MAX_SMART_RESEARCH_USD:.2f}")
|
||||
return f"{parsed:.2f}"
|
||||
|
||||
|
||||
def validate_paid_inputs(args: argparse.Namespace) -> str | None:
|
||||
if not args.allow_paid:
|
||||
return None
|
||||
if not args.allowed_chat_id or not CHAT_ID_RE.match(args.allowed_chat_id):
|
||||
raise ValueError("--allowed-chat-id is required for --allow-paid and must be an integer")
|
||||
approval_ref = read_approval_ref(from_stdin=args.approval_ref_from_stdin)
|
||||
if not APPROVAL_REF_RE.match(approval_ref):
|
||||
raise ValueError("approval ref shape is invalid")
|
||||
return approval_ref
|
||||
|
||||
|
||||
def resolve_owner_group(owner: str, group: str, *, no_chown: bool) -> tuple[int | None, int | None]:
|
||||
if no_chown:
|
||||
return None, None
|
||||
return pwd.getpwnam(owner).pw_uid, grp.getgrnam(group).gr_gid
|
||||
|
||||
|
||||
def write_private_file(path: Path, content: str, *, uid: int | None, gid: int | None, dry_run: bool) -> None:
|
||||
if dry_run:
|
||||
return
|
||||
|
||||
path.parent.mkdir(parents=True, mode=0o700, exist_ok=True)
|
||||
tmp_path = path.with_name(f".{path.name}.tmp-{os.getpid()}")
|
||||
fd = os.open(tmp_path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
try:
|
||||
with os.fdopen(fd, "w") as handle:
|
||||
handle.write(content)
|
||||
if not content.endswith("\n"):
|
||||
handle.write("\n")
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
os.chmod(tmp_path, 0o600)
|
||||
if uid is not None or gid is not None:
|
||||
os.chown(tmp_path, -1 if uid is None else uid, -1 if gid is None else gid)
|
||||
os.replace(tmp_path, path)
|
||||
os.chmod(path, 0o600)
|
||||
finally:
|
||||
if tmp_path.exists():
|
||||
tmp_path.unlink()
|
||||
|
||||
|
||||
def run_command(command: list[str], *, dry_run: bool) -> dict:
|
||||
if dry_run:
|
||||
return {"command": command, "skipped": "dry_run"}
|
||||
proc = subprocess.run(command, check=False, text=True, capture_output=True)
|
||||
return {
|
||||
"command": command,
|
||||
"returncode": proc.returncode,
|
||||
"stdout": proc.stdout.strip(),
|
||||
"stderr": proc.stderr.strip(),
|
||||
}
|
||||
|
||||
|
||||
def build_env_content(*, allow_paid: bool, allowed_chat_id: str | None, max_usd: str, approval_ref_path: Path) -> str:
|
||||
lines = [
|
||||
f"LIVINGIP_LEO_TELEGRAM_SMART_RESEARCH_ALLOW_PAID={'1' if allow_paid else '0'}",
|
||||
f"LIVINGIP_LEO_TELEGRAM_SMART_RESEARCH_MAX_USD={max_usd}",
|
||||
f"LIVINGIP_LEO_TELEGRAM_SMART_RESEARCH_APPROVAL_REF_FILE={approval_ref_path}",
|
||||
]
|
||||
if allow_paid and allowed_chat_id:
|
||||
lines.insert(1, f"LIVINGIP_LEO_TELEGRAM_SMART_RESEARCH_ALLOWED_CHAT_ID={allowed_chat_id}")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = parse_args(sys.argv[1:] if argv is None else argv)
|
||||
validate_agent_name(args.agent)
|
||||
max_usd = validate_max_usd(args.max_usd)
|
||||
approval_ref = validate_paid_inputs(args)
|
||||
uid, gid = resolve_owner_group(args.owner, args.group, no_chown=args.no_chown)
|
||||
|
||||
secrets_dir = Path(args.secrets_dir)
|
||||
env_path = secrets_dir / f"teleo-agent-{args.agent}.env"
|
||||
approval_ref_path = secrets_dir / f"{args.agent}-smart-research-approval-ref"
|
||||
env_content = build_env_content(
|
||||
allow_paid=args.allow_paid,
|
||||
allowed_chat_id=args.allowed_chat_id,
|
||||
max_usd=max_usd,
|
||||
approval_ref_path=approval_ref_path,
|
||||
)
|
||||
|
||||
write_private_file(env_path, env_content, uid=uid, gid=gid, dry_run=args.dry_run)
|
||||
approval_ref_written = False
|
||||
if args.allow_paid and approval_ref is not None:
|
||||
write_private_file(approval_ref_path, approval_ref, uid=uid, gid=gid, dry_run=args.dry_run)
|
||||
approval_ref_written = not args.dry_run
|
||||
|
||||
unit = f"teleo-agent@{args.agent}.service"
|
||||
restart_result = None
|
||||
if args.restart_service:
|
||||
restart_result = run_command(["systemctl", "restart", unit], dry_run=args.dry_run)
|
||||
|
||||
proof = {
|
||||
"schema": "livingip.telegramSmartResearchGateInstall.v1",
|
||||
"generatedAt": datetime.now(timezone.utc).isoformat(),
|
||||
"ok": True,
|
||||
"agent": args.agent,
|
||||
"envPath": str(env_path),
|
||||
"envFileWritten": not args.dry_run,
|
||||
"approvalRefPath": str(approval_ref_path),
|
||||
"approvalRefWritten": approval_ref_written,
|
||||
"approvalRefPresent": bool(args.allow_paid),
|
||||
"allowedChatIdPresent": bool(args.allowed_chat_id),
|
||||
"maxUsd": max_usd,
|
||||
"paidEnabled": bool(args.allow_paid),
|
||||
"dryRun": args.dry_run,
|
||||
"fileMode": "0600",
|
||||
"owner": None if args.no_chown else args.owner,
|
||||
"group": None if args.no_chown else args.group,
|
||||
"serviceUnit": unit,
|
||||
"serviceRestart": restart_result,
|
||||
"secretValuesIncluded": False,
|
||||
}
|
||||
|
||||
output = json.dumps(proof, indent=2, sort_keys=True) + "\n"
|
||||
if args.output:
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_text(output)
|
||||
print(output, end="")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
raise SystemExit(main())
|
||||
except ValueError as exc:
|
||||
print(f"ERROR: {exc}", file=sys.stderr)
|
||||
raise SystemExit(1) from None
|
||||
|
|
@ -1,114 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""One-time backfill: canonicalize prs.submitted_by and sources.submitted_by.
|
||||
|
||||
Strips legacy decorators ("(self-directed)", "(reweave)"), lowercases, drops
|
||||
the @ prefix. After this runs, every value matches the contract documented
|
||||
on diagnostics/activity_feed_api.py::_normalize_contributor — and the
|
||||
companion read-side fix becomes redundant defense-in-depth instead of
|
||||
load-bearing.
|
||||
|
||||
Defaults to --dry-run. Pass --apply to commit.
|
||||
|
||||
Usage:
|
||||
python3 normalize-submitted-by.py --dry-run
|
||||
python3 normalize-submitted-by.py --apply
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import sqlite3
|
||||
import sys
|
||||
from collections import Counter
|
||||
|
||||
DEFAULT_DB = os.environ.get("PIPELINE_DB", "/opt/teleo-eval/pipeline/pipeline.db")
|
||||
|
||||
# Valid handle: lowercase alphanum + _-, 1-39 chars (matches GitHub rules,
|
||||
# same as pipeline/lib/attribution._HANDLE_RE). Anything with parens, spaces,
|
||||
# or uppercase needs canonicalization.
|
||||
_TRAILING_PAREN_RE = re.compile(r"\s*\([^)]*\)\s*$")
|
||||
_HANDLE_RE = re.compile(r"^[a-z0-9][a-z0-9_-]{0,38}$")
|
||||
|
||||
|
||||
def canonicalize(raw):
|
||||
if raw is None:
|
||||
return None
|
||||
h = raw.strip().lower().lstrip("@")
|
||||
h = _TRAILING_PAREN_RE.sub("", h).strip()
|
||||
return h or None
|
||||
|
||||
|
||||
def normalize_table(conn, table, dry_run):
|
||||
cur = conn.execute(
|
||||
f"SELECT rowid, submitted_by FROM {table} WHERE submitted_by IS NOT NULL"
|
||||
)
|
||||
changes = []
|
||||
for row in cur.fetchall():
|
||||
old = row[1]
|
||||
new = canonicalize(old)
|
||||
if new != old:
|
||||
changes.append((row[0], old, new))
|
||||
|
||||
print(f"\n{table}: {len(changes)} rows need normalization")
|
||||
if not changes:
|
||||
return 0
|
||||
|
||||
# Distribution preview
|
||||
from_to = Counter((old, new) for _, old, new in changes)
|
||||
for (old, new), count in from_to.most_common(15):
|
||||
print(f" {count:>5} {old!r:40} -> {new!r}")
|
||||
if len(from_to) > 15:
|
||||
print(f" ... ({len(from_to) - 15} more distinct mappings)")
|
||||
|
||||
# Sanity: every result is a valid handle (no garbage falls through).
|
||||
invalid = [(rowid, old, new) for rowid, old, new in changes
|
||||
if new is not None and not _HANDLE_RE.match(new)]
|
||||
if invalid:
|
||||
print(f"\n WARNING: {len(invalid)} rows would normalize to invalid handles:")
|
||||
for rowid, old, new in invalid[:10]:
|
||||
print(f" rowid={rowid} {old!r} -> {new!r}")
|
||||
print(" These rows will be SKIPPED (left as-is). Inspect manually.")
|
||||
|
||||
valid_changes = [(rowid, old, new) for rowid, old, new in changes
|
||||
if new is None or _HANDLE_RE.match(new)]
|
||||
|
||||
if dry_run:
|
||||
print(f" [dry-run] would update {len(valid_changes)} rows in {table}")
|
||||
return len(valid_changes)
|
||||
|
||||
for rowid, _, new in valid_changes:
|
||||
conn.execute(
|
||||
f"UPDATE {table} SET submitted_by = ? WHERE rowid = ?",
|
||||
(new, rowid),
|
||||
)
|
||||
print(f" updated {len(valid_changes)} rows in {table}")
|
||||
return len(valid_changes)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--db", default=DEFAULT_DB)
|
||||
ap.add_argument("--apply", action="store_true", help="Commit changes (default is dry-run)")
|
||||
ap.add_argument("--dry-run", action="store_true", help="Preview only (default)")
|
||||
args = ap.parse_args()
|
||||
|
||||
dry_run = not args.apply
|
||||
print(f"DB: {args.db}")
|
||||
print(f"Mode: {'DRY-RUN' if dry_run else 'APPLY'}")
|
||||
|
||||
conn = sqlite3.connect(args.db, timeout=30)
|
||||
try:
|
||||
total = 0
|
||||
total += normalize_table(conn, "prs", dry_run)
|
||||
total += normalize_table(conn, "sources", dry_run)
|
||||
if not dry_run:
|
||||
conn.commit()
|
||||
print(f"\nCommitted. Total rows updated: {total}")
|
||||
else:
|
||||
print(f"\nDry-run complete. Run with --apply to commit ({total} rows pending).")
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -1,346 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""No-network local proof for Phase 1b agent routing.
|
||||
|
||||
This script exercises the real evaluate cycle against an in-memory migrated DB
|
||||
while replacing only external network/LLM edges with deterministic fakes.
|
||||
"""
|
||||
|
||||
# ruff: noqa: E402,I001
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import re
|
||||
import sqlite3
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
if str(REPO_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
|
||||
from lib import config, db
|
||||
from lib import evaluate as evaluate_mod
|
||||
|
||||
|
||||
SINGLE_DOMAIN_CASES = [
|
||||
{
|
||||
"number": 101,
|
||||
"domain": "grand-strategy",
|
||||
"branch": "leo/grand-strategy",
|
||||
"paths": ["domains/grand-strategy/strategy.md"],
|
||||
"expected_agents": ["Leo"],
|
||||
},
|
||||
{
|
||||
"number": 102,
|
||||
"domain": "ai-alignment",
|
||||
"branch": "theseus/alignment",
|
||||
"paths": ["domains/ai-alignment/systems.md"],
|
||||
"expected_agents": ["Theseus"],
|
||||
},
|
||||
{
|
||||
"number": 103,
|
||||
"domain": "internet-finance",
|
||||
"branch": "rio/x402",
|
||||
"paths": ["domains/internet-finance/x402.md"],
|
||||
"expected_agents": ["Rio"],
|
||||
},
|
||||
{
|
||||
"number": 104,
|
||||
"domain": "health",
|
||||
"branch": "vida/health",
|
||||
"paths": ["domains/health/clinical.md"],
|
||||
"expected_agents": ["Vida"],
|
||||
},
|
||||
{
|
||||
"number": 105,
|
||||
"domain": "entertainment",
|
||||
"branch": "clay/games",
|
||||
"paths": ["domains/entertainment/games.md"],
|
||||
"expected_agents": ["Clay"],
|
||||
},
|
||||
{
|
||||
"number": 106,
|
||||
"domain": "space-development",
|
||||
"branch": "astra/robotics",
|
||||
"paths": ["domains/space-development/robotics.md"],
|
||||
"expected_agents": ["Astra"],
|
||||
},
|
||||
]
|
||||
|
||||
CROSS_DOMAIN_CASE = {
|
||||
"number": 107,
|
||||
"domain": "cross-ai-finance",
|
||||
"branch": "rio/ai-x402",
|
||||
"paths": ["domains/ai-systems/agent-wallets.md", "domains/internet-finance/x402.md"],
|
||||
"expected_agents": ["Theseus", "Rio"],
|
||||
}
|
||||
|
||||
FEEDBACK_CASE = {
|
||||
"number": 108,
|
||||
"domain": "health-feedback",
|
||||
"branch": "vida/reject-health",
|
||||
"paths": ["domains/health/incorrect-health-claim.md"],
|
||||
"expected_agents": ["Vida"],
|
||||
}
|
||||
|
||||
|
||||
def _diff_for(paths: list[str]) -> str:
|
||||
chunks = []
|
||||
for path in paths:
|
||||
chunks.append(
|
||||
"\n".join(
|
||||
[
|
||||
f"diff --git a/{path} b/{path}",
|
||||
"--- a/file.md",
|
||||
"+++ b/file.md",
|
||||
"+type: claim",
|
||||
"+description: local phase 1b proof claim",
|
||||
]
|
||||
)
|
||||
)
|
||||
return "\n".join(chunks)
|
||||
|
||||
|
||||
def _insert_pr(conn: sqlite3.Connection, case: dict[str, Any]) -> None:
|
||||
source_path = f"inbox/archive/phase1b-{case['number']}.md"
|
||||
conn.execute(
|
||||
"INSERT INTO sources (path, status, priority) VALUES (?, 'extracted', 'medium')",
|
||||
(source_path,),
|
||||
)
|
||||
conn.execute(
|
||||
"""INSERT INTO prs
|
||||
(number, source_path, branch, status, tier, tier0_pass,
|
||||
leo_verdict, domain_verdict, eval_attempts, priority)
|
||||
VALUES (?, ?, ?, 'open', 'STANDARD', 1, 'pending', 'pending', 0, 'medium')""",
|
||||
(case["number"], source_path, case["branch"]),
|
||||
)
|
||||
|
||||
|
||||
def _pr_number_from_path(path: str) -> int | None:
|
||||
match = re.search(r"(?:issues|pulls)/(\d+)", path)
|
||||
return int(match.group(1)) if match else None
|
||||
|
||||
|
||||
async def run_phase1b_local_proof() -> dict[str, Any]:
|
||||
conn = sqlite3.connect(":memory:")
|
||||
conn.row_factory = sqlite3.Row
|
||||
db.migrate(conn)
|
||||
|
||||
cases = [*SINGLE_DOMAIN_CASES, CROSS_DOMAIN_CASE, FEEDBACK_CASE]
|
||||
diffs = {case["number"]: _diff_for(case["paths"]) for case in cases}
|
||||
for case in cases:
|
||||
_insert_pr(conn, case)
|
||||
|
||||
comments: dict[int, list[str]] = {}
|
||||
formal_approvals: list[int] = []
|
||||
eval_feedback: list[dict[str, Any]] = []
|
||||
dispositions: list[dict[str, Any]] = []
|
||||
agent_review_calls: list[dict[str, Any]] = []
|
||||
|
||||
async def fake_get_pr_diff(pr_number: int) -> str:
|
||||
return diffs[pr_number]
|
||||
|
||||
async def fake_run_agent_review(
|
||||
diff: str,
|
||||
files: str,
|
||||
agent: str,
|
||||
route_context: str = "",
|
||||
tier: str = "STANDARD",
|
||||
) -> tuple[str, dict[str, int]]:
|
||||
verdict = "REQUEST_CHANGES" if "incorrect-health-claim.md" in diff and agent == "Vida" else "APPROVE"
|
||||
issues = "\n<!-- ISSUES: factual_discrepancy -->" if verdict == "REQUEST_CHANGES" else ""
|
||||
agent_review_calls.append(
|
||||
{
|
||||
"agent": agent,
|
||||
"tier": tier,
|
||||
"files": files.splitlines(),
|
||||
"route": json.loads(route_context),
|
||||
"verdict": verdict,
|
||||
}
|
||||
)
|
||||
return (
|
||||
f"{agent} local Phase 1b review{issues}\n<!-- VERDICT:{agent.upper()}:{verdict} -->",
|
||||
{"prompt_tokens": 10, "completion_tokens": 5},
|
||||
)
|
||||
|
||||
async def fake_forgejo_api(method: str, path: str, body: dict | None = None, token: str | None = None):
|
||||
pr_number = _pr_number_from_path(path)
|
||||
if method == "GET" and "comments" in path:
|
||||
return [{"body": body_text} for body_text in comments.get(pr_number or -1, [])]
|
||||
if method == "POST" and "comments" in path:
|
||||
comments.setdefault(pr_number or -1, []).append((body or {}).get("body", ""))
|
||||
return {"id": len(comments[pr_number or -1])}
|
||||
if method == "GET" and "pulls/" in path:
|
||||
return {"user": {"login": "phase1b-local-proof"}}
|
||||
return {"ok": True, "token": bool(token)}
|
||||
|
||||
async def fake_post_formal_approvals(pr_number: int, pr_author: str) -> None:
|
||||
formal_approvals.append(pr_number)
|
||||
|
||||
async def fake_on_eval_complete(
|
||||
conn: sqlite3.Connection,
|
||||
pr_number: int,
|
||||
*,
|
||||
outcome: str,
|
||||
review_text: str,
|
||||
issues: list[str] | None = None,
|
||||
) -> None:
|
||||
eval_feedback.append({"pr": pr_number, "outcome": outcome, "issues": issues or []})
|
||||
|
||||
async def fake_dispose_rejected_pr(
|
||||
conn: sqlite3.Connection,
|
||||
pr_number: int,
|
||||
eval_attempts: int,
|
||||
issues: list[str],
|
||||
) -> None:
|
||||
dispositions.append({"pr": pr_number, "eval_attempts": eval_attempts, "issues": issues})
|
||||
|
||||
originals = {
|
||||
"flag": config.PHASE1B_AGENT_ROUTING_ENABLED,
|
||||
"backoff": evaluate_mod._rate_limit_backoff_until,
|
||||
"get_pr_diff": evaluate_mod.get_pr_diff,
|
||||
"run_agent_review": evaluate_mod.run_agent_review,
|
||||
"forgejo_api": evaluate_mod.forgejo_api,
|
||||
"post_formal_approvals": evaluate_mod.post_formal_approvals,
|
||||
"on_eval_complete": evaluate_mod.on_eval_complete,
|
||||
"dispose_rejected_pr": evaluate_mod.dispose_rejected_pr,
|
||||
}
|
||||
|
||||
try:
|
||||
config.PHASE1B_AGENT_ROUTING_ENABLED = True
|
||||
evaluate_mod._rate_limit_backoff_until = None
|
||||
evaluate_mod.get_pr_diff = fake_get_pr_diff
|
||||
evaluate_mod.run_agent_review = fake_run_agent_review
|
||||
evaluate_mod.forgejo_api = fake_forgejo_api
|
||||
evaluate_mod.post_formal_approvals = fake_post_formal_approvals
|
||||
evaluate_mod.on_eval_complete = fake_on_eval_complete
|
||||
evaluate_mod.dispose_rejected_pr = fake_dispose_rejected_pr
|
||||
|
||||
succeeded, failed = await evaluate_mod.evaluate_cycle(conn, max_workers=len(cases))
|
||||
finally:
|
||||
config.PHASE1B_AGENT_ROUTING_ENABLED = originals["flag"]
|
||||
evaluate_mod._rate_limit_backoff_until = originals["backoff"]
|
||||
evaluate_mod.get_pr_diff = originals["get_pr_diff"]
|
||||
evaluate_mod.run_agent_review = originals["run_agent_review"]
|
||||
evaluate_mod.forgejo_api = originals["forgejo_api"]
|
||||
evaluate_mod.post_formal_approvals = originals["post_formal_approvals"]
|
||||
evaluate_mod.on_eval_complete = originals["on_eval_complete"]
|
||||
evaluate_mod.dispose_rejected_pr = originals["dispose_rejected_pr"]
|
||||
|
||||
pr_rows = {
|
||||
row["number"]: dict(row)
|
||||
for row in conn.execute(
|
||||
"""SELECT number, status, branch, domain, domain_agent, leo_verdict,
|
||||
domain_verdict, auto_merge, eval_issues
|
||||
FROM prs
|
||||
ORDER BY number"""
|
||||
).fetchall()
|
||||
}
|
||||
review_rows = [dict(row) for row in conn.execute("SELECT * FROM review_records ORDER BY pr_number, agent")]
|
||||
route_events = [
|
||||
json.loads(row["detail"])
|
||||
for row in conn.execute(
|
||||
"SELECT detail FROM audit_log WHERE stage = 'evaluate' AND event = 'phase1b_route' ORDER BY id"
|
||||
).fetchall()
|
||||
]
|
||||
source_feedback = {
|
||||
row["path"]: row["feedback"]
|
||||
for row in conn.execute("SELECT path, feedback FROM sources WHERE feedback IS NOT NULL ORDER BY path")
|
||||
}
|
||||
|
||||
case_results = []
|
||||
for case in cases:
|
||||
number = case["number"]
|
||||
reviewers = sorted(row["agent"] for row in review_rows if row["pr_number"] == number)
|
||||
posted = comments.get(number, [])
|
||||
case_results.append(
|
||||
{
|
||||
"number": number,
|
||||
"domain": case["domain"],
|
||||
"expected_agents": sorted(case["expected_agents"]),
|
||||
"reviewers": reviewers,
|
||||
"status": pr_rows[number]["status"],
|
||||
"domain_agent": pr_rows[number]["domain_agent"],
|
||||
"domain_verdict": pr_rows[number]["domain_verdict"],
|
||||
"comments": len(posted),
|
||||
"markers": [
|
||||
marker
|
||||
for body in posted
|
||||
for marker in re.findall(r"<!-- PHASE1B_REVIEW:PR=\d+:AGENT=[A-Z]+ -->", body)
|
||||
],
|
||||
}
|
||||
)
|
||||
|
||||
proof = {
|
||||
"ok": True,
|
||||
"scope": "local_no_network_phase1b_eval_cycle",
|
||||
"schema_version": db.SCHEMA_VERSION,
|
||||
"feature_flag": "PHASE1B_AGENT_ROUTING_ENABLED",
|
||||
"succeeded": succeeded,
|
||||
"failed": failed,
|
||||
"cases_total": len(cases),
|
||||
"case_results": case_results,
|
||||
"agents_seen": sorted({call["agent"] for call in agent_review_calls}),
|
||||
"agent_review_calls": agent_review_calls,
|
||||
"formal_approvals": sorted(formal_approvals),
|
||||
"eval_feedback": sorted(eval_feedback, key=lambda item: item["pr"]),
|
||||
"rejection_dispositions": dispositions,
|
||||
"route_events": route_events,
|
||||
"source_feedback_paths": sorted(source_feedback),
|
||||
}
|
||||
_assert_phase1b_proof(proof)
|
||||
return proof
|
||||
|
||||
|
||||
def _assert_phase1b_proof(proof: dict[str, Any]) -> None:
|
||||
expected_agents = ["Astra", "Clay", "Leo", "Rio", "Theseus", "Vida"]
|
||||
assert proof["succeeded"] == proof["cases_total"]
|
||||
assert proof["failed"] == 0
|
||||
assert proof["agents_seen"] == expected_agents
|
||||
assert len(proof["route_events"]) == proof["cases_total"]
|
||||
|
||||
by_number = {case["number"]: case for case in proof["case_results"]}
|
||||
for case in SINGLE_DOMAIN_CASES:
|
||||
result = by_number[case["number"]]
|
||||
assert result["status"] == "approved"
|
||||
assert result["reviewers"] == sorted(case["expected_agents"])
|
||||
assert result["comments"] == len(case["expected_agents"])
|
||||
|
||||
cross = by_number[CROSS_DOMAIN_CASE["number"]]
|
||||
assert cross["status"] == "approved"
|
||||
assert cross["reviewers"] == sorted(CROSS_DOMAIN_CASE["expected_agents"])
|
||||
assert cross["comments"] == 2
|
||||
|
||||
feedback = by_number[FEEDBACK_CASE["number"]]
|
||||
assert feedback["status"] == "open"
|
||||
assert feedback["reviewers"] == ["Vida"]
|
||||
assert feedback["domain_verdict"] == "request_changes"
|
||||
assert proof["rejection_dispositions"] == [
|
||||
{"pr": FEEDBACK_CASE["number"], "eval_attempts": 1, "issues": ["factual_discrepancy"]}
|
||||
]
|
||||
assert len(proof["formal_approvals"]) == len(SINGLE_DOMAIN_CASES) + 1
|
||||
assert [item for item in proof["eval_feedback"] if item["outcome"] == "rejected"]
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Run local no-network Phase 1b proof")
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
default="proof/phase1b-local-e2e-proof.json",
|
||||
help="JSON proof output path",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
proof = asyncio.run(run_phase1b_local_proof())
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_text(json.dumps(proof, indent=2, sort_keys=True) + "\n")
|
||||
print(json.dumps({"ok": True, "output": str(output_path), "cases_total": proof["cases_total"]}, sort_keys=True))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -1,168 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Reattribute PRs and their author events from m3taversal to the true author.
|
||||
|
||||
Scope (intentionally conservative):
|
||||
- branch reweave/* -> pipeline (system maintenance, no human author)
|
||||
- branch ingestion/* -> pipeline (pipeline-internal source intake)
|
||||
- branch <agent>/* -> <agent> (autonomous agent work)
|
||||
for agent in {leo, vida, rio, astra, clay, theseus}.
|
||||
|
||||
NOT in scope:
|
||||
- branch extract/* -- proposed_by may legitimately be absent
|
||||
(telegram source drops default to operator).
|
||||
|
||||
Per affected PR (atomic):
|
||||
1. UPDATE prs.submitted_by -> target
|
||||
2. UPDATE sources.submitted_by where path = pr.source_path
|
||||
3. UPDATE contribution_events.handle for every m3ta author event on this PR
|
||||
(kind set to 'agent', since pipeline + the six agents are all kind='agent'
|
||||
per attribution.PENTAGON_AGENTS).
|
||||
|
||||
Idempotent. Dry-run by default; --apply commits.
|
||||
Run AFTER scripts/normalize-submitted-by.py.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
from collections import Counter
|
||||
|
||||
DB_PATH = os.environ.get("DB_PATH", "/opt/teleo-eval/pipeline/pipeline.db")
|
||||
|
||||
AGENT_PREFIXES = ("leo/", "vida/", "rio/", "astra/", "clay/", "theseus/")
|
||||
PIPELINE_PREFIXES = ("reweave/", "ingestion/")
|
||||
|
||||
|
||||
def target_for(branch):
|
||||
if not branch:
|
||||
return None
|
||||
if branch.startswith(PIPELINE_PREFIXES):
|
||||
return "pipeline"
|
||||
for prefix in AGENT_PREFIXES:
|
||||
if branch.startswith(prefix):
|
||||
return prefix.rstrip("/")
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--apply", action="store_true", help="commit changes (default: dry-run)")
|
||||
ap.add_argument("--db", default=DB_PATH)
|
||||
args = ap.parse_args()
|
||||
|
||||
conn = sqlite3.connect(args.db, timeout=30)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.execute("PRAGMA busy_timeout = 30000")
|
||||
|
||||
mode = "APPLY" if args.apply else "DRY-RUN"
|
||||
print("DB: {}\nMode: {}\n".format(args.db, mode))
|
||||
|
||||
rows = conn.execute("""
|
||||
SELECT number, branch, source_path
|
||||
FROM prs
|
||||
WHERE submitted_by = 'm3taversal'
|
||||
AND branch IS NOT NULL
|
||||
""").fetchall()
|
||||
|
||||
pr_targets = []
|
||||
pr_counts = Counter()
|
||||
for r in rows:
|
||||
tgt = target_for(r["branch"])
|
||||
if tgt is None:
|
||||
continue
|
||||
pr_targets.append((r["number"], r["branch"], r["source_path"], tgt))
|
||||
pr_counts[tgt] += 1
|
||||
|
||||
print("prs to reattribute: {}".format(len(pr_targets)))
|
||||
for tgt, n in pr_counts.most_common():
|
||||
print(" {:6d} -> {!r}".format(n, tgt))
|
||||
|
||||
src_paths = [t[2] for t in pr_targets if t[2]]
|
||||
src_count = 0
|
||||
if src_paths:
|
||||
placeholders = ",".join("?" * len(src_paths))
|
||||
src_count = conn.execute(
|
||||
"SELECT COUNT(*) FROM sources "
|
||||
"WHERE submitted_by = 'm3taversal' AND path IN ({})".format(placeholders),
|
||||
src_paths,
|
||||
).fetchone()[0]
|
||||
print("\nsources rows that will be re-pointed: {}".format(src_count))
|
||||
|
||||
pr_to_target = {p[0]: p[3] for p in pr_targets}
|
||||
events = []
|
||||
if pr_to_target:
|
||||
pr_placeholders = ",".join("?" * len(pr_to_target))
|
||||
events = conn.execute(
|
||||
"SELECT id, pr_number FROM contribution_events "
|
||||
"WHERE handle = 'm3taversal' AND role = 'author' "
|
||||
"AND pr_number IN ({})".format(pr_placeholders),
|
||||
list(pr_to_target.keys()),
|
||||
).fetchall()
|
||||
print("contribution_events author rows to move: {}".format(len(events)))
|
||||
ev_counts = Counter(pr_to_target[e["pr_number"]] for e in events)
|
||||
for tgt, n in ev_counts.most_common():
|
||||
print(" {:6d} events -> {!r}".format(n, tgt))
|
||||
|
||||
if not args.apply:
|
||||
print("\nDry-run complete. Run with --apply to commit "
|
||||
"({} PRs + {} sources + {} events).".format(
|
||||
len(pr_targets), src_count, len(events)))
|
||||
return 0
|
||||
|
||||
pr_updated = 0
|
||||
src_updated = 0
|
||||
ev_updated = 0
|
||||
ev_collisions = 0
|
||||
|
||||
try:
|
||||
for pr_num, branch, source_path, target in pr_targets:
|
||||
cur = conn.execute(
|
||||
"UPDATE prs SET submitted_by = ? "
|
||||
"WHERE number = ? AND submitted_by = 'm3taversal'",
|
||||
(target, pr_num),
|
||||
)
|
||||
pr_updated += cur.rowcount
|
||||
|
||||
if source_path:
|
||||
cur = conn.execute(
|
||||
"UPDATE sources SET submitted_by = ? "
|
||||
"WHERE path = ? AND submitted_by = 'm3taversal'",
|
||||
(target, source_path),
|
||||
)
|
||||
src_updated += cur.rowcount
|
||||
|
||||
for ev in conn.execute(
|
||||
"SELECT id FROM contribution_events "
|
||||
"WHERE handle = 'm3taversal' AND role = 'author' AND pr_number = ?",
|
||||
(pr_num,),
|
||||
).fetchall():
|
||||
try:
|
||||
conn.execute(
|
||||
"UPDATE contribution_events SET handle = ?, kind = 'agent' "
|
||||
"WHERE id = ?",
|
||||
(target, ev["id"]),
|
||||
)
|
||||
ev_updated += 1
|
||||
except sqlite3.IntegrityError:
|
||||
conn.execute(
|
||||
"DELETE FROM contribution_events WHERE id = ?",
|
||||
(ev["id"],),
|
||||
)
|
||||
ev_collisions += 1
|
||||
|
||||
conn.commit()
|
||||
except Exception:
|
||||
conn.rollback()
|
||||
raise
|
||||
|
||||
print("\nCommitted.")
|
||||
print(" prs.submitted_by moves: {}".format(pr_updated))
|
||||
print(" sources.submitted_by moves: {}".format(src_updated))
|
||||
print(" contribution_events moves: {}".format(ev_updated))
|
||||
print(" ce collisions deleted: {}".format(ev_collisions))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -1,244 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Replay fixture-backed decision-engine evals without live model calls."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from lib.agent_routing import classify_pr_route
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
DEFAULT_FIXTURES_DIR = REPO_ROOT / "fixtures" / "decision-engine-eval"
|
||||
DEFAULT_OUTPUT = REPO_ROOT / ".crabbox-results" / "decision-engine-eval.json"
|
||||
VALID_DISPOSITIONS = {"approve", "reject", "escalate"}
|
||||
|
||||
|
||||
def _read_json(path: Path) -> dict[str, Any]:
|
||||
with path.open() as fh:
|
||||
data = json.load(fh)
|
||||
if not isinstance(data, dict):
|
||||
raise AssertionError(f"{path.relative_to(REPO_ROOT)} must contain a JSON object")
|
||||
return data
|
||||
|
||||
|
||||
def _require_dict(data: dict[str, Any], key: str, fixture_id: str) -> dict[str, Any]:
|
||||
value = data.get(key)
|
||||
if not isinstance(value, dict):
|
||||
raise AssertionError(f"{fixture_id}: {key} must be an object")
|
||||
return value
|
||||
|
||||
|
||||
def _require_list(data: dict[str, Any], key: str, fixture_id: str) -> list[Any]:
|
||||
value = data.get(key)
|
||||
if not isinstance(value, list) or not value:
|
||||
raise AssertionError(f"{fixture_id}: {key} must be a non-empty list")
|
||||
return value
|
||||
|
||||
|
||||
def _require_str(data: dict[str, Any], key: str, fixture_id: str) -> str:
|
||||
value = data.get(key)
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
raise AssertionError(f"{fixture_id}: {key} must be a non-empty string")
|
||||
return value
|
||||
|
||||
|
||||
def _validate_fixture(fixture: dict[str, Any], path: Path) -> None:
|
||||
fixture_id = _require_str(fixture, "id", str(path))
|
||||
_require_str(fixture, "lane", fixture_id)
|
||||
input_data = _require_dict(fixture, "input", fixture_id)
|
||||
rubric = _require_dict(fixture, "rubric", fixture_id)
|
||||
expected = _require_dict(fixture, "expected", fixture_id)
|
||||
_require_str(input_data, "diff", fixture_id)
|
||||
_require_list(rubric, "must_check", fixture_id)
|
||||
_require_list(rubric, "reject_if", fixture_id)
|
||||
_require_str(expected, "primary_agent", fixture_id)
|
||||
_require_list(expected, "required_agents", fixture_id)
|
||||
_require_str(expected, "route_kind", fixture_id)
|
||||
disposition = _require_str(expected, "expected_disposition", fixture_id)
|
||||
if disposition not in VALID_DISPOSITIONS:
|
||||
raise AssertionError(f"{fixture_id}: expected_disposition must be one of {sorted(VALID_DISPOSITIONS)}")
|
||||
_require_list(expected, "issue_tags", fixture_id)
|
||||
|
||||
|
||||
def load_fixtures(fixtures_dir: Path = DEFAULT_FIXTURES_DIR) -> list[dict[str, Any]]:
|
||||
if not fixtures_dir.exists():
|
||||
raise AssertionError(f"missing fixtures directory: {fixtures_dir.relative_to(REPO_ROOT)}")
|
||||
fixtures = []
|
||||
for path in sorted(fixtures_dir.glob("*.json")):
|
||||
fixture = _read_json(path)
|
||||
_validate_fixture(fixture, path)
|
||||
fixtures.append(fixture)
|
||||
if not fixtures:
|
||||
raise AssertionError(f"no fixtures found in {fixtures_dir.relative_to(REPO_ROOT)}")
|
||||
ids = [fixture["id"] for fixture in fixtures]
|
||||
duplicates = [fixture_id for fixture_id, count in Counter(ids).items() if count > 1]
|
||||
if duplicates:
|
||||
raise AssertionError(f"duplicate fixture ids: {duplicates}")
|
||||
return fixtures
|
||||
|
||||
|
||||
def _kb_interop_ok(fixture: dict[str, Any]) -> bool | None:
|
||||
if fixture["lane"] != "kb-interop":
|
||||
return None
|
||||
proposed_write = fixture["input"].get("proposed_write")
|
||||
if not isinstance(proposed_write, dict):
|
||||
return False
|
||||
target = str(proposed_write.get("target", "")).lower()
|
||||
denied_actions = proposed_write.get("denied_actions_invoked")
|
||||
return (
|
||||
target not in {"main", "production", "prod"}
|
||||
and proposed_write.get("mutates_production_db") is False
|
||||
and isinstance(denied_actions, list)
|
||||
and not denied_actions
|
||||
and bool(proposed_write.get("proof_path"))
|
||||
)
|
||||
|
||||
|
||||
def _fixture_result(fixture: dict[str, Any]) -> dict[str, Any]:
|
||||
input_data = fixture["input"]
|
||||
expected = fixture["expected"]
|
||||
route = classify_pr_route(
|
||||
input_data["diff"],
|
||||
branch=input_data.get("branch"),
|
||||
title=input_data.get("title"),
|
||||
body=input_data.get("body"),
|
||||
)
|
||||
checks = {
|
||||
"route_primary_ok": route.primary_agent == expected["primary_agent"],
|
||||
"route_required_ok": list(route.required_agents) == expected["required_agents"],
|
||||
"route_kind_ok": route.route_kind == expected["route_kind"],
|
||||
"kb_interop_ok": _kb_interop_ok(fixture),
|
||||
}
|
||||
applicable_checks = [value for value in checks.values() if value is not None]
|
||||
return {
|
||||
"id": fixture["id"],
|
||||
"lane": fixture["lane"],
|
||||
"ok": all(applicable_checks),
|
||||
"expected": expected,
|
||||
"actual_route": route.to_audit_dict(),
|
||||
"checks": checks,
|
||||
"baseline_verdict": {
|
||||
"disposition": expected["expected_disposition"],
|
||||
"issue_tags": expected["issue_tags"],
|
||||
"primary_agent": route.primary_agent,
|
||||
"required_agents": list(route.required_agents),
|
||||
"reason": "fixture truth with deterministic route evidence",
|
||||
},
|
||||
"rubric": fixture["rubric"],
|
||||
}
|
||||
|
||||
|
||||
def _load_candidate_output(path: Path | None) -> dict[str, Any] | None:
|
||||
if path is None:
|
||||
return None
|
||||
candidate = _read_json(path)
|
||||
_require_str(candidate, "candidate_name", str(path))
|
||||
verdicts = candidate.get("verdicts")
|
||||
if not isinstance(verdicts, list):
|
||||
raise AssertionError(f"{path.relative_to(REPO_ROOT)}: verdicts must be a list")
|
||||
return candidate
|
||||
|
||||
|
||||
def _score_candidate(results: list[dict[str, Any]], candidate: dict[str, Any] | None) -> dict[str, Any] | None:
|
||||
if candidate is None:
|
||||
return None
|
||||
verdicts_by_id = {}
|
||||
for verdict in candidate["verdicts"]:
|
||||
if not isinstance(verdict, dict):
|
||||
raise AssertionError("candidate verdicts must be JSON objects")
|
||||
fixture_id = _require_str(verdict, "fixture_id", candidate["candidate_name"])
|
||||
disposition = _require_str(verdict, "disposition", fixture_id)
|
||||
if disposition not in VALID_DISPOSITIONS:
|
||||
raise AssertionError(f"{fixture_id}: candidate disposition must be one of {sorted(VALID_DISPOSITIONS)}")
|
||||
verdicts_by_id[fixture_id] = verdict
|
||||
|
||||
missing_verdicts: list[str] = []
|
||||
false_approves: list[str] = []
|
||||
false_rejects: list[str] = []
|
||||
route_mismatches: list[str] = []
|
||||
missing_required_tags: dict[str, list[str]] = {}
|
||||
|
||||
for result in results:
|
||||
fixture_id = result["id"]
|
||||
expected = result["expected"]
|
||||
verdict = verdicts_by_id.get(fixture_id)
|
||||
if verdict is None:
|
||||
missing_verdicts.append(fixture_id)
|
||||
continue
|
||||
if verdict["disposition"] == "approve" and expected["expected_disposition"] != "approve":
|
||||
false_approves.append(fixture_id)
|
||||
if verdict["disposition"] == "reject" and expected["expected_disposition"] == "approve":
|
||||
false_rejects.append(fixture_id)
|
||||
if verdict.get("primary_agent") and verdict.get("primary_agent") != expected["primary_agent"]:
|
||||
route_mismatches.append(fixture_id)
|
||||
if verdict.get("required_agents") and verdict.get("required_agents") != expected["required_agents"]:
|
||||
route_mismatches.append(fixture_id)
|
||||
expected_tags = set(expected["issue_tags"])
|
||||
actual_tags = set(verdict.get("issue_tags", []))
|
||||
missing = sorted(expected_tags - actual_tags)
|
||||
if missing and expected["expected_disposition"] != "approve":
|
||||
missing_required_tags[fixture_id] = missing
|
||||
|
||||
return {
|
||||
"candidate_name": candidate["candidate_name"],
|
||||
"ok": not (missing_verdicts or false_approves or false_rejects or route_mismatches or missing_required_tags),
|
||||
"missing_verdicts": missing_verdicts,
|
||||
"false_approve_count": len(false_approves),
|
||||
"false_approves": false_approves,
|
||||
"false_reject_count": len(false_rejects),
|
||||
"false_rejects": false_rejects,
|
||||
"route_mismatches": sorted(set(route_mismatches)),
|
||||
"missing_required_tags": missing_required_tags,
|
||||
}
|
||||
|
||||
|
||||
def evaluate_fixtures(
|
||||
fixtures: list[dict[str, Any]],
|
||||
*,
|
||||
candidate: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
results = [_fixture_result(fixture) for fixture in fixtures]
|
||||
fixture_count = len(results)
|
||||
route_ok_count = sum(1 for result in results if result["ok"])
|
||||
candidate_score = _score_candidate(results, candidate)
|
||||
proof_ok = route_ok_count == fixture_count and (candidate_score is None or candidate_score["ok"])
|
||||
return {
|
||||
"ok": proof_ok,
|
||||
"scope": "decision_engine_replay",
|
||||
"fixture_count": fixture_count,
|
||||
"metrics": {
|
||||
"route_accuracy": route_ok_count / fixture_count,
|
||||
"route_ok_count": route_ok_count,
|
||||
"lanes": dict(sorted(Counter(result["lane"] for result in results).items())),
|
||||
},
|
||||
"results": results,
|
||||
"candidate": candidate_score,
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--fixtures-dir", default=str(DEFAULT_FIXTURES_DIR))
|
||||
parser.add_argument("--candidate-output")
|
||||
parser.add_argument("--output", default=str(DEFAULT_OUTPUT))
|
||||
args = parser.parse_args()
|
||||
|
||||
fixtures = load_fixtures(Path(args.fixtures_dir))
|
||||
candidate = _load_candidate_output(Path(args.candidate_output) if args.candidate_output else None)
|
||||
proof = evaluate_fixtures(fixtures, candidate=candidate)
|
||||
|
||||
output = Path(args.output)
|
||||
if not output.is_absolute():
|
||||
output = REPO_ROOT / output
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
output.write_text(json.dumps(proof, indent=2, sort_keys=True) + "\n")
|
||||
print(json.dumps(proof, indent=2, sort_keys=True))
|
||||
return 0 if proof["ok"] else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
|
@ -33,7 +33,6 @@ ReadWritePaths=/opt/teleo-eval/pipeline/pipeline.db-shm
|
|||
ReadWritePaths=/opt/teleo-eval/workspaces/main/agents
|
||||
|
||||
Environment=PYTHONUNBUFFERED=1
|
||||
EnvironmentFile=-/opt/teleo-eval/secrets/teleo-agent-%i.env
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
|
|
|||
|
|
@ -11,8 +11,8 @@ import logging
|
|||
import os
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
from urllib.parse import urlparse
|
||||
|
||||
logger = logging.getLogger("tg.agent_config")
|
||||
|
||||
|
|
@ -43,12 +43,6 @@ class AgentConfig:
|
|||
triage_model: str = "anthropic/claude-haiku-4.5"
|
||||
max_tokens: int = 1024
|
||||
max_response_per_user_per_hour: int = 30
|
||||
http_chat_proxy_url: Optional[str] = None
|
||||
http_research_proxy_url: Optional[str] = None
|
||||
respond_to_private_chats: bool = False
|
||||
mention_aliases: list[str] = field(default_factory=list)
|
||||
smart_research_command_prefixes: list[str] = field(default_factory=list)
|
||||
auto_smart_research_from_chat: bool = False
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
"""Convert to dict for passing to build_system_prompt."""
|
||||
|
|
@ -61,12 +55,6 @@ class AgentConfig:
|
|||
"voice_summary": self.voice_summary,
|
||||
"domain_expertise": self.domain_expertise,
|
||||
"pentagon_agent_id": self.pentagon_agent_id,
|
||||
"http_chat_proxy_url": self.http_chat_proxy_url,
|
||||
"http_research_proxy_url": self.http_research_proxy_url,
|
||||
"respond_to_private_chats": self.respond_to_private_chats,
|
||||
"mention_aliases": self.mention_aliases,
|
||||
"smart_research_command_prefixes": self.smart_research_command_prefixes,
|
||||
"auto_smart_research_from_chat": self.auto_smart_research_from_chat,
|
||||
}
|
||||
|
||||
@property
|
||||
|
|
@ -112,29 +100,6 @@ def load_agent_config(config_path: str) -> AgentConfig:
|
|||
if "learnings_file" not in raw:
|
||||
errors.append("Missing required field: learnings_file")
|
||||
|
||||
proxy_url = raw.get("http_chat_proxy_url")
|
||||
if proxy_url:
|
||||
parsed_proxy = urlparse(proxy_url)
|
||||
if parsed_proxy.scheme not in {"http", "https"} or not parsed_proxy.netloc:
|
||||
errors.append("http_chat_proxy_url must be an absolute http(s) URL")
|
||||
|
||||
research_proxy_url = raw.get("http_research_proxy_url")
|
||||
if research_proxy_url:
|
||||
parsed_research_proxy = urlparse(research_proxy_url)
|
||||
if parsed_research_proxy.scheme not in {"http", "https"} or not parsed_research_proxy.netloc:
|
||||
errors.append("http_research_proxy_url must be an absolute http(s) URL")
|
||||
|
||||
mention_aliases = raw.get("mention_aliases", [])
|
||||
if mention_aliases and not isinstance(mention_aliases, list):
|
||||
errors.append("mention_aliases must be a list")
|
||||
|
||||
smart_research_command_prefixes = raw.get("smart_research_command_prefixes", [])
|
||||
if smart_research_command_prefixes and not isinstance(smart_research_command_prefixes, list):
|
||||
errors.append("smart_research_command_prefixes must be a list")
|
||||
for prefix in smart_research_command_prefixes or []:
|
||||
if not isinstance(prefix, str) or not prefix.startswith("/"):
|
||||
errors.append("smart_research_command_prefixes entries must start with /")
|
||||
|
||||
if errors:
|
||||
raise ValueError(
|
||||
f"Agent config validation failed ({config_path}):\n"
|
||||
|
|
@ -158,12 +123,6 @@ def load_agent_config(config_path: str) -> AgentConfig:
|
|||
triage_model=raw.get("triage_model", "anthropic/claude-haiku-4.5"),
|
||||
max_tokens=raw.get("max_tokens", 1024),
|
||||
max_response_per_user_per_hour=raw.get("max_response_per_user_per_hour", 30),
|
||||
http_chat_proxy_url=proxy_url,
|
||||
http_research_proxy_url=research_proxy_url,
|
||||
respond_to_private_chats=bool(raw.get("respond_to_private_chats", False)),
|
||||
mention_aliases=mention_aliases,
|
||||
smart_research_command_prefixes=smart_research_command_prefixes,
|
||||
auto_smart_research_from_chat=bool(raw.get("auto_smart_research_from_chat", False)),
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,57 +0,0 @@
|
|||
# Leo Wallet Test - disposable Living IP x402 Telegram canary
|
||||
# This config runs a separate Telegram bot process against Leo's hosted HTTP chat route.
|
||||
|
||||
name: Leo Wallet Test
|
||||
handle: "@lipleowallet0622183538bot"
|
||||
x_handle: "@teLEOhuman"
|
||||
mention_aliases:
|
||||
- "@leo"
|
||||
- "@lipleowallet0622183538bot"
|
||||
- "@LeoWalletTest"
|
||||
bot_token_file: leo-test-telegram-bot-token
|
||||
pentagon_agent_id: livingip-leo-wallet-test
|
||||
domain: collective-intelligence
|
||||
domain_expertise: >
|
||||
Living IP x402 payment status, hosted Leo wallet canaries, AgentCash
|
||||
paid research readbacks, and Telegram transport testing
|
||||
|
||||
http_chat_proxy_url: "https://leo.livingip.xyz/api/agents/leo/chat"
|
||||
http_research_proxy_url: "https://leo.livingip.xyz/api/agents/leo/research"
|
||||
smart_research_command_prefixes:
|
||||
- "/smart_research"
|
||||
- "/paid_research"
|
||||
auto_smart_research_from_chat: true
|
||||
respond_to_private_chats: true
|
||||
|
||||
kb_scope:
|
||||
primary:
|
||||
- domains/collective-intelligence
|
||||
- domains/internet-finance
|
||||
- foundations
|
||||
- core
|
||||
|
||||
voice_summary: "Disposable Leo x402 wallet-test transport. Concise, proof-aware, no-spend by default."
|
||||
|
||||
voice_definition: |
|
||||
## Register
|
||||
You are the disposable Telegram wallet-test instance for Leo. Keep replies
|
||||
concise and tied to retained Living IP x402 runtime evidence.
|
||||
|
||||
## x402 / Wallet Testing
|
||||
Report current public x402 receive capability, AgentCash paid-readback status,
|
||||
and exact blockers. Do not claim new Telegram-triggered payment execution
|
||||
unless the hosted Leo HTTP route returns retained payment/readback evidence.
|
||||
Ordinary addressed/private chat may be routed into smart research when the
|
||||
request clearly asks for sourced, current, market, or evidence-backed work.
|
||||
Explicit /smart_research remains available for narrow canaries. Paid smart
|
||||
research remains fail-closed unless the server-side allow flag, allowed chat
|
||||
id, cap, and retained approval-ref file are all present.
|
||||
Do not request or expose private keys, bot tokens, wallet exports, seed phrases,
|
||||
or raw secret values.
|
||||
|
||||
learnings_file: agents/leo/learnings.md
|
||||
|
||||
response_model: anthropic/claude-opus-4-6
|
||||
triage_model: anthropic/claude-haiku-4.5
|
||||
max_tokens: 500
|
||||
max_response_per_user_per_hour: 30
|
||||
|
|
@ -1,57 +0,0 @@
|
|||
# Leo — Living IP x402 research agent
|
||||
# This config makes Telegram a thin transport to Leo's hosted HTTP chat route.
|
||||
|
||||
# ─── Identity ────────────────────────────────────────────────────────────
|
||||
name: Leo
|
||||
handle: "@TeleoHumanBot"
|
||||
x_handle: "@teLEOhuman"
|
||||
mention_aliases:
|
||||
- "@leo"
|
||||
- "@teLEOhuman"
|
||||
- "@TeleoHumanBot"
|
||||
- "@teLEOhumanity"
|
||||
bot_token_file: leo-telegram-bot-token
|
||||
pentagon_agent_id: livingip-leo
|
||||
domain: collective-intelligence
|
||||
domain_expertise: >
|
||||
collective intelligence, coordination systems, Living IP strategy, agent
|
||||
markets, paid research, x402 service rails, and cross-domain synthesis
|
||||
|
||||
# ─── Hosted Leo Runtime ──────────────────────────────────────────────────
|
||||
http_chat_proxy_url: "https://leo.livingip.xyz/api/agents/leo/chat"
|
||||
respond_to_private_chats: true
|
||||
|
||||
# ─── KB Scope ────────────────────────────────────────────────────────────
|
||||
kb_scope:
|
||||
primary:
|
||||
- domains/collective-intelligence
|
||||
- domains/ai-alignment
|
||||
- domains/space-development
|
||||
- foundations
|
||||
- core
|
||||
|
||||
# ─── Voice ───────────────────────────────────────────────────────────────
|
||||
voice_summary: "Cross-domain strategist. Direct, synthesis-heavy, proof-aware."
|
||||
|
||||
voice_definition: |
|
||||
## Register
|
||||
You are Leo, TeleoHumanity's cross-domain strategy and collective
|
||||
intelligence agent. Be direct and synthesis-heavy. Prefer concrete
|
||||
mechanisms, coordination failures, and next actions over broad abstractions.
|
||||
|
||||
## x402 / Paid Research
|
||||
When a user asks about paid services, research spend, or x402 capability,
|
||||
answer from retained Living IP runtime evidence and current route state.
|
||||
Do not claim payment execution unless the HTTP route returns retained
|
||||
payment/readback evidence.
|
||||
|
||||
# ─── Learnings ───────────────────────────────────────────────────────────
|
||||
learnings_file: agents/leo/learnings.md
|
||||
|
||||
# ─── Model ───────────────────────────────────────────────────────────────
|
||||
response_model: anthropic/claude-opus-4-6
|
||||
triage_model: anthropic/claude-haiku-4.5
|
||||
max_tokens: 500
|
||||
|
||||
# ─── Rate Limits ─────────────────────────────────────────────────────────
|
||||
max_response_per_user_per_hour: 30
|
||||
|
|
@ -14,24 +14,14 @@ No deal terms, no dollar amounts, no private investment details in approval requ
|
|||
Epimetheus owns this module.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# ruff: noqa: I001
|
||||
|
||||
import logging
|
||||
import re
|
||||
import sqlite3
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
from telegram import InlineKeyboardButton, InlineKeyboardMarkup, Update
|
||||
from telegram.ext import CallbackQueryHandler, ContextTypes
|
||||
except ImportError: # Optional in local unit tests that only exercise OPSEC logic.
|
||||
InlineKeyboardButton = None
|
||||
InlineKeyboardMarkup = None
|
||||
Update = None
|
||||
CallbackQueryHandler = None
|
||||
ContextTypes = None
|
||||
from telegram import InlineKeyboardButton, InlineKeyboardMarkup, Update
|
||||
from telegram.ext import CallbackQueryHandler, ContextTypes
|
||||
|
||||
logger = logging.getLogger("telegram.approvals")
|
||||
|
||||
|
|
@ -120,8 +110,8 @@ def format_approval_message(row: sqlite3.Row) -> str:
|
|||
content = content[:3000] + "\n\n[... truncated]"
|
||||
|
||||
parts = [
|
||||
"APPROVAL REQUEST",
|
||||
"",
|
||||
f"APPROVAL REQUEST",
|
||||
f"",
|
||||
f"Type: {type_label}",
|
||||
f"From: {agent}",
|
||||
]
|
||||
|
|
@ -144,8 +134,6 @@ def format_approval_message(row: sqlite3.Row) -> str:
|
|||
|
||||
def build_keyboard(request_id: int) -> InlineKeyboardMarkup:
|
||||
"""Build inline keyboard with Approve/Reject buttons."""
|
||||
if InlineKeyboardMarkup is None or InlineKeyboardButton is None:
|
||||
raise ImportError("python-telegram-bot is required to build approval keyboards")
|
||||
return InlineKeyboardMarkup([
|
||||
[
|
||||
InlineKeyboardButton("Approve", callback_data=f"approve:{request_id}"),
|
||||
|
|
@ -237,6 +225,8 @@ async def handle_approval_callback(update: Update, context: ContextTypes.DEFAULT
|
|||
return
|
||||
|
||||
if action == "reject":
|
||||
# Check if user sent a reply with rejection reason
|
||||
rejection_reason = None
|
||||
# For rejection, edit the message to ask for reason
|
||||
row = conn.execute(
|
||||
"SELECT * FROM approval_queue WHERE id = ?", (request_id,)
|
||||
|
|
|
|||
442
telegram/bot.py
442
telegram/bot.py
|
|
@ -29,7 +29,6 @@ import time
|
|||
import yaml
|
||||
from collections import defaultdict
|
||||
from datetime import datetime, timezone
|
||||
from html import escape
|
||||
from pathlib import Path
|
||||
|
||||
# Add pipeline lib to path for shared modules
|
||||
|
|
@ -48,21 +47,9 @@ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
|||
import json as _json
|
||||
from kb_retrieval import KBIndex, retrieve_context, retrieve_vector_context
|
||||
from retrieval import orchestrate_retrieval
|
||||
from market_data import get_token_price, format_price_context, extract_market_data_tokens
|
||||
from market_data import get_token_price, format_price_context
|
||||
from worktree_lock import main_worktree_lock
|
||||
from x_client import search_tweets, fetch_from_url, check_research_rate_limit, record_research_usage, get_research_remaining
|
||||
from http_chat_proxy import (
|
||||
DEFAULT_SMART_RESEARCH_COMMAND_PREFIXES,
|
||||
build_chat_proxy_payload,
|
||||
build_smart_research_proxy_payload,
|
||||
extract_auto_smart_research_followup_goal,
|
||||
extract_auto_smart_research_goal,
|
||||
extract_paid_work_order_id,
|
||||
extract_smart_research_goal,
|
||||
post_chat_proxy,
|
||||
should_attach_structured_market_context,
|
||||
smart_research_command_names,
|
||||
)
|
||||
|
||||
# ─── Config ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
|
@ -88,18 +75,6 @@ TRIAGE_MODEL = "anthropic/claude-haiku-4.5" # Haiku for batch triage
|
|||
|
||||
# KB scope — None means all domains (Rio default). Set from YAML config for other agents.
|
||||
AGENT_KB_SCOPE: list[str] | None = None
|
||||
AGENT_NAME = "Rio"
|
||||
AGENT_HANDLE = "@FutAIrdBot"
|
||||
AGENT_X_HANDLE = "@futaRdIO"
|
||||
AGENT_DOMAIN_EXPERTISE = (
|
||||
"futarchy, prediction markets, token governance, and the MetaDAO ecosystem"
|
||||
)
|
||||
AGENT_HTTP_CHAT_PROXY_URL: str | None = None
|
||||
AGENT_HTTP_RESEARCH_PROXY_URL: str | None = None
|
||||
AGENT_RESPOND_TO_PRIVATE_CHATS = False
|
||||
AGENT_MENTION_ALIASES = ["@teleo", "@FutAIrdBot"]
|
||||
AGENT_SMART_RESEARCH_COMMAND_PREFIXES = list(DEFAULT_SMART_RESEARCH_COMMAND_PREFIXES)
|
||||
AGENT_AUTO_SMART_RESEARCH_FROM_CHAT = False
|
||||
|
||||
# Rate limits
|
||||
MAX_RESPONSE_PER_USER_PER_HOUR = 30
|
||||
|
|
@ -115,8 +90,6 @@ logging.basicConfig(
|
|||
logging.StreamHandler(),
|
||||
],
|
||||
)
|
||||
logging.getLogger("httpx").setLevel(logging.WARNING)
|
||||
logging.getLogger("httpcore").setLevel(logging.WARNING)
|
||||
logger = logging.getLogger("telegram-bot")
|
||||
|
||||
# ─── State ──────────────────────────────────────────────────────────────
|
||||
|
|
@ -130,91 +103,6 @@ user_response_times: dict[int, list[float]] = defaultdict(list)
|
|||
# Allowed group IDs (set after first message received, or configure)
|
||||
allowed_groups: set[int] = set()
|
||||
|
||||
TELEGRAM_REPLY_CHUNK_LIMIT = 3500
|
||||
|
||||
|
||||
def _telegram_native_html(text: str) -> str:
|
||||
"""Render common LLM Markdown as Telegram HTML without trusting raw HTML."""
|
||||
rendered = escape(text, quote=False)
|
||||
rendered = re.sub(r"(?m)^#{1,6}\s+(.+)$", r"<b>\1</b>", rendered)
|
||||
rendered = re.sub(r"\*\*([^*\n]{1,240})\*\*", r"<b>\1</b>", rendered)
|
||||
rendered = re.sub(r"`([^`\n]{1,240})`", r"<code>\1</code>", rendered)
|
||||
return rendered
|
||||
|
||||
|
||||
def _plain_telegram_fallback(text: str) -> str:
|
||||
text = re.sub(r"(?m)^#{1,6}\s+", "", text)
|
||||
text = re.sub(r"\*\*([^*\n]{1,240})\*\*", r"\1", text)
|
||||
text = re.sub(r"`([^`\n]{1,240})`", r"\1", text)
|
||||
return text
|
||||
|
||||
|
||||
def _telegram_reply_chunks(text: str) -> list[str]:
|
||||
chunks: list[str] = []
|
||||
current = ""
|
||||
for part in re.split(r"(\n\n+)", text):
|
||||
if len(part) > TELEGRAM_REPLY_CHUNK_LIMIT:
|
||||
if current:
|
||||
chunks.append(current)
|
||||
current = ""
|
||||
chunks.extend(
|
||||
part[i : i + TELEGRAM_REPLY_CHUNK_LIMIT]
|
||||
for i in range(0, len(part), TELEGRAM_REPLY_CHUNK_LIMIT)
|
||||
)
|
||||
continue
|
||||
if len(current) + len(part) > TELEGRAM_REPLY_CHUNK_LIMIT and current:
|
||||
chunks.append(current)
|
||||
current = part
|
||||
else:
|
||||
current += part
|
||||
if current:
|
||||
chunks.append(current)
|
||||
return chunks or [""]
|
||||
|
||||
|
||||
async def _reply_text_native(msg, text: str, *, do_quote: bool = True):
|
||||
first = True
|
||||
for chunk in _telegram_reply_chunks(text):
|
||||
try:
|
||||
await msg.reply_text(
|
||||
_telegram_native_html(chunk),
|
||||
parse_mode="HTML",
|
||||
do_quote=do_quote and first,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning("Telegram native-format reply failed; using plain fallback: %s", e)
|
||||
await msg.reply_text(
|
||||
_plain_telegram_fallback(chunk),
|
||||
do_quote=do_quote and first,
|
||||
)
|
||||
first = False
|
||||
|
||||
|
||||
async def _typing_keepalive(chat, stop_event: asyncio.Event, interval_seconds: float = 4.0) -> None:
|
||||
while not stop_event.is_set():
|
||||
try:
|
||||
await chat.send_action("typing")
|
||||
except Exception as exc:
|
||||
logger.debug("typing keepalive failed: %s", exc)
|
||||
try:
|
||||
await asyncio.wait_for(stop_event.wait(), timeout=interval_seconds)
|
||||
except asyncio.TimeoutError:
|
||||
continue
|
||||
|
||||
|
||||
async def _post_chat_proxy_with_typing(chat, **kwargs):
|
||||
stop_event = asyncio.Event()
|
||||
keepalive = asyncio.create_task(_typing_keepalive(chat, stop_event))
|
||||
try:
|
||||
return await post_chat_proxy(**kwargs)
|
||||
finally:
|
||||
stop_event.set()
|
||||
keepalive.cancel()
|
||||
try:
|
||||
await keepalive
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
# Shared KB index (built once, refreshed on mtime change)
|
||||
kb_index = KBIndex(KB_READ_DIR)
|
||||
|
||||
|
|
@ -712,64 +600,6 @@ def sanitize_message(text: str) -> str:
|
|||
return text[:2000]
|
||||
|
||||
|
||||
def _smart_research_payment_gate(chat_id: int) -> dict:
|
||||
"""Return paid smart-research fields only when all server-side gates pass."""
|
||||
max_allowed_usd = 0.06
|
||||
if os.getenv("LIVINGIP_LEO_TELEGRAM_SMART_RESEARCH_ALLOW_PAID") != "1":
|
||||
return {"allow_paid_execution": False}
|
||||
|
||||
allowed_chat_id = os.getenv("LIVINGIP_LEO_TELEGRAM_SMART_RESEARCH_ALLOWED_CHAT_ID", "").strip()
|
||||
if not allowed_chat_id or allowed_chat_id != str(chat_id):
|
||||
return {"allow_paid_execution": False}
|
||||
|
||||
try:
|
||||
max_amount_usd = float(os.getenv("LIVINGIP_LEO_TELEGRAM_SMART_RESEARCH_MAX_USD", "0.01"))
|
||||
except ValueError:
|
||||
return {"allow_paid_execution": False}
|
||||
if max_amount_usd <= 0 or max_amount_usd > max_allowed_usd:
|
||||
return {"allow_paid_execution": False}
|
||||
|
||||
approval_ref_file = os.getenv("LIVINGIP_LEO_TELEGRAM_SMART_RESEARCH_APPROVAL_REF_FILE", "").strip()
|
||||
if not approval_ref_file:
|
||||
return {"allow_paid_execution": False}
|
||||
|
||||
try:
|
||||
approval_ref = Path(approval_ref_file).read_text().strip()
|
||||
except OSError:
|
||||
return {"allow_paid_execution": False}
|
||||
if not approval_ref:
|
||||
return {"allow_paid_execution": False}
|
||||
|
||||
return {
|
||||
"allow_paid_execution": True,
|
||||
"approval_ref": approval_ref,
|
||||
"max_amount_usd": max_amount_usd,
|
||||
}
|
||||
|
||||
|
||||
async def _market_context_for_message(
|
||||
text: str,
|
||||
extra_terms: list[str] | tuple[str, ...] = (),
|
||||
) -> tuple[str, dict, int, list[str]]:
|
||||
"""Fetch structured market data for token questions without blocking the answer path."""
|
||||
token_mentions = extract_market_data_tokens(text, extra_terms=extra_terms)
|
||||
market_context = ""
|
||||
market_data_audit = {}
|
||||
t_market = time.monotonic()
|
||||
for token in token_mentions:
|
||||
try:
|
||||
data = await get_token_price(token)
|
||||
if data:
|
||||
price_str = format_price_context(data, token)
|
||||
if price_str:
|
||||
market_context += price_str + "\n"
|
||||
market_data_audit[token] = data
|
||||
except Exception as e:
|
||||
logger.warning("Market context lookup failed for %s: %s", token, e)
|
||||
market_duration = int((time.monotonic() - t_market) * 1000)
|
||||
return market_context.strip(), market_data_audit, market_duration, token_mentions
|
||||
|
||||
|
||||
def _git_commit_archive(archive_path, filename: str):
|
||||
"""Commit archived source to git so it survives git clean. (Rio review: data loss bug)"""
|
||||
import subprocess
|
||||
|
|
@ -1100,22 +930,6 @@ async def handle_reply_to_bot(update: Update, context: ContextTypes.DEFAULT_TYPE
|
|||
await handle_tagged(update, context)
|
||||
|
||||
|
||||
async def handle_smart_research_command(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
"""Route configured slash commands into the smart-research proxy path."""
|
||||
await handle_tagged(update, context)
|
||||
|
||||
|
||||
def _previous_user_message(chat_id: int, user_id: int | None) -> str | None:
|
||||
if user_id is not None:
|
||||
history = conversation_history.get((chat_id, user_id), [])
|
||||
if history:
|
||||
return history[-1].get("user")
|
||||
chat_history = conversation_history.get((chat_id, 0), [])
|
||||
if chat_history:
|
||||
return chat_history[-1].get("user")
|
||||
return None
|
||||
|
||||
|
||||
async def handle_message(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
"""Handle ALL incoming group messages — buffer for triage."""
|
||||
if not update.message or not update.message.text:
|
||||
|
|
@ -1170,7 +984,7 @@ async def handle_message(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
|||
|
||||
|
||||
async def handle_tagged(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
"""Handle messages that tag the bot."""
|
||||
"""Handle messages that tag the bot — Rio responds with Opus."""
|
||||
if not update.message or not update.message.text:
|
||||
return
|
||||
|
||||
|
|
@ -1185,176 +999,6 @@ async def handle_tagged(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
|||
|
||||
logger.info("Tagged by @%s: %s", user.username if user else "unknown", text[:100])
|
||||
|
||||
smart_research_goal = None
|
||||
previous_user_goal = _previous_user_message(msg.chat_id, user.id if user else None)
|
||||
paid_work_order_id = extract_paid_work_order_id(text) if AGENT_HTTP_RESEARCH_PROXY_URL else None
|
||||
if AGENT_HTTP_RESEARCH_PROXY_URL:
|
||||
smart_research_goal = extract_smart_research_goal(
|
||||
text,
|
||||
tuple(AGENT_SMART_RESEARCH_COMMAND_PREFIXES),
|
||||
)
|
||||
if paid_work_order_id and not smart_research_goal:
|
||||
smart_research_goal = previous_user_goal or text
|
||||
if not smart_research_goal and AGENT_AUTO_SMART_RESEARCH_FROM_CHAT:
|
||||
smart_research_goal = extract_auto_smart_research_goal(
|
||||
text,
|
||||
tuple(AGENT_MENTION_ALIASES),
|
||||
)
|
||||
if not smart_research_goal and AGENT_AUTO_SMART_RESEARCH_FROM_CHAT:
|
||||
smart_research_goal = extract_auto_smart_research_followup_goal(
|
||||
text,
|
||||
previous_user_goal,
|
||||
tuple(AGENT_MENTION_ALIASES),
|
||||
)
|
||||
if AGENT_HTTP_RESEARCH_PROXY_URL and smart_research_goal:
|
||||
payment_gate = _smart_research_payment_gate(msg.chat_id)
|
||||
proxy_research_goal = smart_research_goal
|
||||
if should_attach_structured_market_context(smart_research_goal):
|
||||
market_context, market_data_audit, market_duration, market_tokens = await _market_context_for_message(
|
||||
smart_research_goal
|
||||
)
|
||||
else:
|
||||
market_context, market_data_audit, market_duration, market_tokens = "", {}, 0, []
|
||||
if market_context:
|
||||
logger.info(
|
||||
"%s smart research added structured market context for %s in %dms",
|
||||
AGENT_NAME,
|
||||
",".join(market_tokens),
|
||||
market_duration,
|
||||
)
|
||||
proxy_research_goal = (
|
||||
f"{smart_research_goal}\n\n"
|
||||
"Structured live market-data context available before web research:\n"
|
||||
f"{market_context}\n\n"
|
||||
"Use the structured market-data context as primary evidence for price, volume, FDV, "
|
||||
"market cap, and liquidity. Do not say you cannot check these metrics when values "
|
||||
"are present above. For buy/sell wording, do not provide personalized financial advice; "
|
||||
"give market data, risks, and a concise non-advice thesis. Do not mention payment "
|
||||
"execution status unless the user asked about payments."
|
||||
)
|
||||
payload = build_smart_research_proxy_payload(
|
||||
research_goal=proxy_research_goal,
|
||||
source="telegram",
|
||||
agent=AGENT_NAME.lower(),
|
||||
chat_id=msg.chat_id,
|
||||
message_id=msg.message_id,
|
||||
username=user.username if user else None,
|
||||
include_synthesis=True,
|
||||
work_order_id=paid_work_order_id,
|
||||
original_research_goal=previous_user_goal if paid_work_order_id else None,
|
||||
**payment_gate,
|
||||
)
|
||||
try:
|
||||
status, proxy_body, proxy_reply = await _post_chat_proxy_with_typing(
|
||||
msg.chat,
|
||||
url=AGENT_HTTP_RESEARCH_PROXY_URL,
|
||||
payload=payload,
|
||||
timeout_seconds=90,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning("%s HTTP smart research proxy failed: %s", AGENT_NAME, e)
|
||||
await msg.reply_text(
|
||||
f"{AGENT_NAME}'s smart research route is temporarily unavailable. "
|
||||
"Try again after the service recovers.",
|
||||
do_quote=True,
|
||||
)
|
||||
return
|
||||
|
||||
if not proxy_reply:
|
||||
logger.warning("%s HTTP smart research proxy returned no reply (status=%s)", AGENT_NAME, status)
|
||||
await msg.reply_text(
|
||||
f"{AGENT_NAME}'s smart research route returned no usable reply. "
|
||||
"The Telegram bridge is fail-closed.",
|
||||
do_quote=True,
|
||||
)
|
||||
return
|
||||
|
||||
await _reply_text_native(msg, proxy_reply, do_quote=True)
|
||||
|
||||
if user:
|
||||
username = user.username or "anonymous"
|
||||
key = (msg.chat_id, user.id)
|
||||
unanswered_count[key] = 0
|
||||
entry = {"user": text[:500], "bot": proxy_reply[:500], "username": username}
|
||||
history = conversation_history.setdefault(key, [])
|
||||
history.append(entry)
|
||||
if len(history) > MAX_HISTORY_USER:
|
||||
history.pop(0)
|
||||
chat_key = (msg.chat_id, 0)
|
||||
chat_history = conversation_history.setdefault(chat_key, [])
|
||||
chat_history.append(entry)
|
||||
if len(chat_history) > MAX_HISTORY_CHAT:
|
||||
chat_history.pop(0)
|
||||
user_response_times[user.id].append(time.time())
|
||||
return
|
||||
|
||||
if AGENT_HTTP_CHAT_PROXY_URL:
|
||||
payload = build_chat_proxy_payload(
|
||||
message=text,
|
||||
source="telegram",
|
||||
agent=AGENT_NAME.lower(),
|
||||
chat_id=msg.chat_id,
|
||||
message_id=msg.message_id,
|
||||
username=user.username if user else None,
|
||||
)
|
||||
try:
|
||||
status, proxy_body, proxy_reply = await _post_chat_proxy_with_typing(
|
||||
msg.chat,
|
||||
url=AGENT_HTTP_CHAT_PROXY_URL,
|
||||
payload=payload,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning("%s HTTP chat proxy failed: %s", AGENT_NAME, e)
|
||||
await msg.reply_text(
|
||||
f"{AGENT_NAME}'s HTTP chat route is temporarily unavailable. "
|
||||
"Try again after the service recovers.",
|
||||
do_quote=True,
|
||||
)
|
||||
return
|
||||
|
||||
if not proxy_reply:
|
||||
logger.warning("%s HTTP chat proxy returned no reply (status=%s)", AGENT_NAME, status)
|
||||
await msg.reply_text(
|
||||
f"{AGENT_NAME}'s HTTP chat route returned no usable reply. "
|
||||
"The Telegram bridge is fail-closed.",
|
||||
do_quote=True,
|
||||
)
|
||||
return
|
||||
|
||||
await _reply_text_native(msg, proxy_reply, do_quote=True)
|
||||
|
||||
if user:
|
||||
username = user.username or "anonymous"
|
||||
key = (msg.chat_id, user.id)
|
||||
unanswered_count[key] = 0
|
||||
entry = {"user": text[:500], "bot": proxy_reply[:500], "username": username}
|
||||
history = conversation_history.setdefault(key, [])
|
||||
history.append(entry)
|
||||
if len(history) > MAX_HISTORY_USER:
|
||||
history.pop(0)
|
||||
chat_key = (msg.chat_id, 0)
|
||||
chat_history = conversation_history.setdefault(chat_key, [])
|
||||
chat_history.append(entry)
|
||||
if len(chat_history) > MAX_HISTORY_CHAT:
|
||||
chat_history.pop(0)
|
||||
user_response_times[user.id].append(time.time())
|
||||
|
||||
logger.info("%s proxied Telegram reply via HTTP chat route (status=%s)", AGENT_NAME, status)
|
||||
_record_transcript(
|
||||
msg,
|
||||
proxy_reply,
|
||||
is_bot=True,
|
||||
rio_response=proxy_reply,
|
||||
internal={
|
||||
"agent": AGENT_NAME.lower(),
|
||||
"http_chat_proxy": True,
|
||||
"http_status": status,
|
||||
"schema": proxy_body.get("schema") if isinstance(proxy_body, dict) else None,
|
||||
"llm_ok": proxy_body.get("llmOk") if isinstance(proxy_body, dict) else None,
|
||||
},
|
||||
)
|
||||
return
|
||||
|
||||
# ─── Audit: init timing and tool call tracking ──────────────────
|
||||
response_start = time.monotonic()
|
||||
tool_calls = []
|
||||
|
|
@ -1521,14 +1165,38 @@ async def handle_tagged(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
|||
stats = get_db_stats()
|
||||
|
||||
# Fetch live market data for any tokens mentioned (Rhea: market-data API)
|
||||
entity_terms = [tag for ent in kb_ctx.entities for tag in ent.tags]
|
||||
market_context, market_data_audit, market_duration, token_mentions = await _market_context_for_message(
|
||||
text,
|
||||
extra_terms=entity_terms,
|
||||
)
|
||||
market_context = ""
|
||||
market_data_audit = {}
|
||||
token_mentions = re.findall(r"\$([A-Z]{2,10})", text.upper())
|
||||
# Entity name → token mapping for natural language mentions
|
||||
ENTITY_TOKEN_MAP = {
|
||||
"omnipair": "OMFG", "metadao": "META", "sanctum": "CLOUD",
|
||||
"drift": "DRIFT", "ore": "ORE", "jupiter": "JUP",
|
||||
}
|
||||
text_lower = text.lower()
|
||||
for name, ticker in ENTITY_TOKEN_MAP.items():
|
||||
if name in text_lower:
|
||||
token_mentions.append(ticker)
|
||||
# Also check entity matches from KB retrieval
|
||||
for ent in kb_ctx.entities:
|
||||
for tag in ent.tags:
|
||||
if tag.upper() in ENTITY_TOKEN_MAP.values():
|
||||
token_mentions.append(tag.upper())
|
||||
t_market = time.monotonic()
|
||||
for token in set(token_mentions):
|
||||
try:
|
||||
data = await get_token_price(token)
|
||||
if data:
|
||||
price_str = format_price_context(data, token)
|
||||
if price_str:
|
||||
market_context += price_str + "\n"
|
||||
market_data_audit[token] = data
|
||||
except Exception:
|
||||
pass # Market data is supplementary — never block on failure
|
||||
market_duration = int((time.monotonic() - t_market) * 1000)
|
||||
if token_mentions:
|
||||
tool_calls.append({
|
||||
"tool": "market_data", "input": {"tickers": token_mentions},
|
||||
"tool": "market_data", "input": {"tickers": list(set(token_mentions))},
|
||||
"output": market_data_audit,
|
||||
"duration_ms": market_duration,
|
||||
})
|
||||
|
|
@ -2245,9 +1913,9 @@ tags: [telegram, ownership-community]
|
|||
async def start_command(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
"""Handle /start command."""
|
||||
await update.message.reply_text(
|
||||
f"I'm {AGENT_NAME}, a TeleoHumanity agent. "
|
||||
f"Tag me with {AGENT_HANDLE} to ask about {AGENT_DOMAIN_EXPERTISE}. "
|
||||
"I'll ground my response in the configured agent runtime."
|
||||
"I'm Rio, the internet finance agent for TeleoHumanity's collective knowledge base. "
|
||||
"Tag me with @teleo to ask about futarchy, prediction markets, token governance, "
|
||||
"or anything in our domain. I'll ground my response in our KB's evidence."
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -2269,29 +1937,10 @@ def _load_agent_config(config_path: str):
|
|||
"""Load agent YAML config and set module-level variables."""
|
||||
global BOT_TOKEN_FILE, RESPONSE_MODEL, TRIAGE_MODEL, AGENT_KB_SCOPE
|
||||
global LEARNINGS_FILE, MAX_RESPONSE_PER_USER_PER_HOUR
|
||||
global AGENT_NAME, AGENT_HANDLE, AGENT_X_HANDLE, AGENT_DOMAIN_EXPERTISE
|
||||
global AGENT_HTTP_CHAT_PROXY_URL, AGENT_HTTP_RESEARCH_PROXY_URL
|
||||
global AGENT_RESPOND_TO_PRIVATE_CHATS, AGENT_MENTION_ALIASES, AGENT_SMART_RESEARCH_COMMAND_PREFIXES
|
||||
global AGENT_AUTO_SMART_RESEARCH_FROM_CHAT
|
||||
|
||||
with open(config_path) as f:
|
||||
cfg = yaml.safe_load(f)
|
||||
|
||||
AGENT_NAME = cfg.get("name", AGENT_NAME)
|
||||
AGENT_HANDLE = cfg.get("handle", AGENT_HANDLE)
|
||||
AGENT_X_HANDLE = cfg.get("x_handle", AGENT_X_HANDLE)
|
||||
AGENT_DOMAIN_EXPERTISE = cfg.get("domain_expertise", AGENT_DOMAIN_EXPERTISE)
|
||||
AGENT_HTTP_CHAT_PROXY_URL = cfg.get("http_chat_proxy_url")
|
||||
AGENT_HTTP_RESEARCH_PROXY_URL = cfg.get("http_research_proxy_url")
|
||||
AGENT_RESPOND_TO_PRIVATE_CHATS = bool(cfg.get("respond_to_private_chats", False))
|
||||
aliases = [AGENT_HANDLE, *cfg.get("mention_aliases", [])]
|
||||
AGENT_MENTION_ALIASES = sorted({alias for alias in aliases if alias})
|
||||
AGENT_SMART_RESEARCH_COMMAND_PREFIXES = cfg.get(
|
||||
"smart_research_command_prefixes",
|
||||
list(DEFAULT_SMART_RESEARCH_COMMAND_PREFIXES),
|
||||
)
|
||||
AGENT_AUTO_SMART_RESEARCH_FROM_CHAT = bool(cfg.get("auto_smart_research_from_chat", False))
|
||||
|
||||
if cfg.get("bot_token_file"):
|
||||
BOT_TOKEN_FILE = f"/opt/teleo-eval/secrets/{cfg['bot_token_file']}"
|
||||
if cfg.get("response_model"):
|
||||
|
|
@ -2310,17 +1959,6 @@ def _load_agent_config(config_path: str):
|
|||
return cfg
|
||||
|
||||
|
||||
def _mention_filter_regex(agent_cfg: dict | None) -> str:
|
||||
"""Build the Telegram mention regex for the active agent."""
|
||||
aliases = ["@teleo", "@FutAIrdBot"]
|
||||
if agent_cfg:
|
||||
aliases = [agent_cfg.get("handle", ""), *agent_cfg.get("mention_aliases", [])]
|
||||
cleaned = sorted({alias for alias in aliases if alias})
|
||||
if not cleaned:
|
||||
cleaned = ["@teleo", "@FutAIrdBot"]
|
||||
return "(?i)(" + "|".join(re.escape(alias) for alias in cleaned) + ")"
|
||||
|
||||
|
||||
def main():
|
||||
"""Start the bot."""
|
||||
parser = argparse.ArgumentParser()
|
||||
|
|
@ -2371,23 +2009,15 @@ def main():
|
|||
# Command handlers
|
||||
app.add_handler(CommandHandler("start", start_command))
|
||||
app.add_handler(CommandHandler("stats", stats_command))
|
||||
for command in smart_research_command_names(AGENT_SMART_RESEARCH_COMMAND_PREFIXES):
|
||||
app.add_handler(CommandHandler(command, handle_smart_research_command))
|
||||
|
||||
# Tag handler — messages mentioning the bot
|
||||
# python-telegram-bot filters.Mention doesn't work for bot mentions in groups
|
||||
# Use a regex filter for the bot username
|
||||
app.add_handler(MessageHandler(
|
||||
filters.TEXT & filters.Regex(_mention_filter_regex(agent_cfg)),
|
||||
filters.TEXT & filters.Regex(r"(?i)(@teleo|@futairdbot)"),
|
||||
handle_tagged,
|
||||
))
|
||||
|
||||
if agent_cfg and AGENT_RESPOND_TO_PRIVATE_CHATS:
|
||||
app.add_handler(MessageHandler(
|
||||
filters.ChatType.PRIVATE & filters.TEXT & ~filters.COMMAND,
|
||||
handle_tagged,
|
||||
))
|
||||
|
||||
# Reply handler — replies to the bot's own messages continue the conversation
|
||||
reply_to_bot_filter = filters.TEXT & filters.REPLY & ~filters.COMMAND
|
||||
app.add_handler(MessageHandler(
|
||||
|
|
|
|||
|
|
@ -1,289 +0,0 @@
|
|||
"""HTTP chat proxy helpers for opt-in Telegram agent routing."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
DEFAULT_SMART_RESEARCH_COMMAND_PREFIXES = ("/smart_research", "/paid_research")
|
||||
_TELEGRAM_COMMAND_NAME_RE = re.compile(r"^[A-Za-z0-9_]+$")
|
||||
_AUTO_SMART_RESEARCH_RE = re.compile(
|
||||
r"\b("
|
||||
r"research|source|sources|citation|citations|evidence|"
|
||||
r"search|find|lookup|look\s+up|web|"
|
||||
r"latest|current|today|recent|as\s+of|this\s+week|this\s+month|"
|
||||
r"twitter|x\.com|tweet|tweets|social|social\s+media|trend|trends|"
|
||||
r"discussion|discussions|sentiment|narrative|narratives|"
|
||||
r"revenue|revenues|fees|tvl|volume|fdv|fully\s+diluted|"
|
||||
r"market\s+cap|mcap|valuation|funding|liquidity|price|chart|"
|
||||
r"token|coin|pair|pool|dex|dexscreener|birdeye|jupiter|"
|
||||
r"buy|sell|should\s+i|yes\s+or\s+no|"
|
||||
r"estimate|compare|benchmark"
|
||||
r")\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_AUTO_CONTEXTUAL_RESEARCH_RE = re.compile(
|
||||
r"("
|
||||
r"\b(what\s+are\s+your\s+thoughts|thoughts\s+on|take\s+on|opinion\s+on|"
|
||||
r"how\s+did|how\s+has|how\s+is|assess|evaluate)\b"
|
||||
r".*\b(managed|manage|handled|handle|handling|responded|response|situation|incident|"
|
||||
r"controversy|fallout|fault|blame|position|stance|fair|valuation|valued|growth|metrics|peers?)\b"
|
||||
r"|"
|
||||
r"\b(who|what|why)\s+(was\s+)?(at\s+)?fault\b"
|
||||
r"|"
|
||||
r"\b(position|stance)\s+(on|about|towards?)\b"
|
||||
r"|"
|
||||
r"\b(compare|benchmark)\b.*\b(metrics|growth|valuation|peers?|fintech|web2|products?)\b"
|
||||
r")",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_AUTO_SMART_RESEARCH_FOLLOWUP_RE = re.compile(
|
||||
r"\b("
|
||||
r"check\s+it\s+yourself|check\s+yourself|actually\s+check|"
|
||||
r"look\s+it\s+up|look\s+that\s+up|search\s+it|search\s+that|"
|
||||
r"use\s+(the\s+)?web|use\s+sources|find\s+sources|"
|
||||
r"do\s+the\s+research|go\s+research|verify\s+it"
|
||||
r")\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_PAID_WORK_ORDER_ID_RE = re.compile(
|
||||
r"\b((?:sponsored_work_orders|payment_receipts):[a-f0-9]{16,64})\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_MARKET_CONTEXT_RE = re.compile(
|
||||
r"\b("
|
||||
r"volume|fdv|fully\s+diluted|market\s+cap|mcap|liquidity|price|chart|"
|
||||
r"token|coin|pair|pool|dex|dexscreener|birdeye|jupiter|"
|
||||
r"buy|sell|should\s+i|yes\s+or\s+no"
|
||||
r")\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_SOCIAL_DISCUSSION_RE = re.compile(
|
||||
r"\b(twitter|x\.com|x\/twitter|tweet|tweets|social)\b.*"
|
||||
r"\b(current|latest|recent|discussion|discussions|trend|trends|narrative|sentiment|fault|blame|position|stance)\b"
|
||||
r"|"
|
||||
r"\b(current|latest|recent|discussion|discussions|trend|trends|narrative|sentiment|fault|blame|position|stance)\b.*"
|
||||
r"\b(twitter|x\.com|x\/twitter|tweet|tweets|social)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def smart_research_command_names(
|
||||
command_prefixes: tuple[str, ...] | list[str] = DEFAULT_SMART_RESEARCH_COMMAND_PREFIXES,
|
||||
) -> list[str]:
|
||||
"""Return Telegram command names registered for smart-research routing."""
|
||||
command_names: set[str] = set()
|
||||
for prefix in command_prefixes:
|
||||
command = str(prefix).strip()
|
||||
if not command.startswith("/"):
|
||||
continue
|
||||
command = command[1:].split("@", 1)[0].strip()
|
||||
if command and _TELEGRAM_COMMAND_NAME_RE.match(command):
|
||||
command_names.add(command)
|
||||
return sorted(command_names)
|
||||
|
||||
|
||||
def build_chat_proxy_payload(
|
||||
*,
|
||||
message: str,
|
||||
source: str,
|
||||
agent: str,
|
||||
chat_id: int | None = None,
|
||||
message_id: int | None = None,
|
||||
username: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Build the no-secret payload sent from Telegram to an agent HTTP chat route."""
|
||||
metadata = {
|
||||
"source": source,
|
||||
"agent": agent,
|
||||
"chat_id": chat_id,
|
||||
"message_id": message_id,
|
||||
"username": username,
|
||||
}
|
||||
return {
|
||||
"message": message,
|
||||
"metadata": {k: v for k, v in metadata.items() if v is not None},
|
||||
}
|
||||
|
||||
|
||||
def extract_smart_research_goal(
|
||||
message: str,
|
||||
command_prefixes: tuple[str, ...] | list[str] = DEFAULT_SMART_RESEARCH_COMMAND_PREFIXES,
|
||||
) -> str | None:
|
||||
"""Return the research goal when a Telegram message opts into smart research."""
|
||||
text = message.strip()
|
||||
for prefix in command_prefixes:
|
||||
command = re.escape(prefix.lstrip("/"))
|
||||
match = re.match(rf"^(?:@\w+\s+)?/{command}(?:@\w+)?(?:\s+(?P<goal>.+))?$", text, re.IGNORECASE)
|
||||
if match:
|
||||
goal = (match.group("goal") or "").strip()
|
||||
return goal or None
|
||||
return None
|
||||
|
||||
|
||||
def extract_auto_smart_research_goal(
|
||||
message: str,
|
||||
mention_aliases: tuple[str, ...] | list[str] = (),
|
||||
) -> str | None:
|
||||
"""Return a research goal when ordinary chat clearly asks for sourced/current research."""
|
||||
text = message.strip()
|
||||
for alias in mention_aliases:
|
||||
clean_alias = str(alias).strip()
|
||||
if not clean_alias:
|
||||
continue
|
||||
text = re.sub(rf"(^|\s){re.escape(clean_alias)}(?:@\w+)?\b", " ", text, flags=re.IGNORECASE).strip()
|
||||
text = re.sub(r"\s+", " ", text).strip()
|
||||
if not text or text.startswith("/"):
|
||||
return None
|
||||
if _AUTO_SMART_RESEARCH_RE.search(text) or _AUTO_CONTEXTUAL_RESEARCH_RE.search(text):
|
||||
return text
|
||||
return None
|
||||
|
||||
|
||||
def extract_auto_smart_research_followup_goal(
|
||||
message: str,
|
||||
previous_user_message: str | None,
|
||||
mention_aliases: tuple[str, ...] | list[str] = (),
|
||||
) -> str | None:
|
||||
"""Turn short follow-ups like 'check it yourself' into the prior research goal."""
|
||||
text = message.strip()
|
||||
for alias in mention_aliases:
|
||||
clean_alias = str(alias).strip()
|
||||
if not clean_alias:
|
||||
continue
|
||||
text = re.sub(rf"(^|\s){re.escape(clean_alias)}(?:@\w+)?\b", " ", text, flags=re.IGNORECASE).strip()
|
||||
text = re.sub(r"\s+", " ", text).strip()
|
||||
if not text or text.startswith("/") or not _AUTO_SMART_RESEARCH_FOLLOWUP_RE.search(text):
|
||||
return None
|
||||
previous_goal = extract_auto_smart_research_goal(previous_user_message or "", mention_aliases)
|
||||
if not previous_goal:
|
||||
return None
|
||||
return (
|
||||
f"{previous_goal}\n\n"
|
||||
f"Follow-up instruction: {text}. Use current public sources and cite assumptions. "
|
||||
"For buy/sell wording, do not provide personalized financial advice; provide market data, risks, "
|
||||
"and a concise non-advice thesis."
|
||||
)
|
||||
|
||||
|
||||
def extract_paid_work_order_id(message: str) -> str | None:
|
||||
"""Return a paid x402 work-order/receipt id from ordinary Telegram text."""
|
||||
match = _PAID_WORK_ORDER_ID_RE.search(message.strip())
|
||||
if not match:
|
||||
return None
|
||||
return match.group(1)
|
||||
|
||||
|
||||
def should_attach_structured_market_context(message: str) -> bool:
|
||||
"""Return true only for explicit market-data questions, not social narrative research."""
|
||||
text = message.strip()
|
||||
if not text:
|
||||
return False
|
||||
if _SOCIAL_DISCUSSION_RE.search(text):
|
||||
return False
|
||||
return bool(_MARKET_CONTEXT_RE.search(text))
|
||||
|
||||
|
||||
def build_smart_research_proxy_payload(
|
||||
*,
|
||||
research_goal: str,
|
||||
source: str,
|
||||
agent: str,
|
||||
chat_id: int | None = None,
|
||||
message_id: int | None = None,
|
||||
username: str | None = None,
|
||||
allow_paid_execution: bool = False,
|
||||
approval_ref: str | None = None,
|
||||
max_amount_usd: float | None = None,
|
||||
include_synthesis: bool = True,
|
||||
work_order_id: str | None = None,
|
||||
original_research_goal: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Build the no-secret Telegram payload for Leo smart research."""
|
||||
payload = build_chat_proxy_payload(
|
||||
message=research_goal,
|
||||
source=source,
|
||||
agent=agent,
|
||||
chat_id=chat_id,
|
||||
message_id=message_id,
|
||||
username=username,
|
||||
)
|
||||
payload.update(
|
||||
{
|
||||
"research_goal": research_goal,
|
||||
"allow_paid_execution": bool(allow_paid_execution),
|
||||
"include_synthesis": bool(include_synthesis),
|
||||
}
|
||||
)
|
||||
if max_amount_usd is not None:
|
||||
payload["max_amount_usd"] = max_amount_usd
|
||||
if allow_paid_execution and approval_ref:
|
||||
payload["approval_ref"] = approval_ref
|
||||
if work_order_id:
|
||||
payload["work_order_id"] = work_order_id
|
||||
if original_research_goal:
|
||||
payload["original_research_goal"] = original_research_goal
|
||||
return payload
|
||||
|
||||
|
||||
def extract_chat_proxy_reply(payload: dict[str, Any]) -> str | None:
|
||||
"""Extract only user-facing replies from supported Living IP Leo response shapes."""
|
||||
if not isinstance(payload, dict):
|
||||
return None
|
||||
|
||||
direct_reply = payload.get("reply")
|
||||
if isinstance(direct_reply, str) and direct_reply.strip():
|
||||
return direct_reply.strip()
|
||||
|
||||
decision = payload.get("decision")
|
||||
if isinstance(decision, dict):
|
||||
decision_reply = decision.get("reply")
|
||||
if isinstance(decision_reply, str) and decision_reply.strip():
|
||||
return decision_reply.strip()
|
||||
|
||||
llm = payload.get("llm")
|
||||
if isinstance(llm, dict):
|
||||
llm_reply = llm.get("reply")
|
||||
if isinstance(llm_reply, str) and llm_reply.strip():
|
||||
return llm_reply.strip()
|
||||
llm_decision = llm.get("decision")
|
||||
if isinstance(llm_decision, dict):
|
||||
llm_decision_reply = llm_decision.get("reply")
|
||||
if isinstance(llm_decision_reply, str) and llm_decision_reply.strip():
|
||||
return llm_decision_reply.strip()
|
||||
|
||||
synthesis = payload.get("synthesis")
|
||||
if isinstance(synthesis, dict):
|
||||
synthesis_reply = synthesis.get("reply")
|
||||
if isinstance(synthesis_reply, str) and synthesis_reply.strip():
|
||||
return synthesis_reply.strip()
|
||||
synthesis_decision = synthesis.get("decision")
|
||||
if isinstance(synthesis_decision, dict):
|
||||
synthesis_decision_reply = synthesis_decision.get("reply")
|
||||
if isinstance(synthesis_decision_reply, str) and synthesis_decision_reply.strip():
|
||||
return synthesis_decision_reply.strip()
|
||||
|
||||
return None
|
||||
|
||||
|
||||
async def post_chat_proxy(
|
||||
*,
|
||||
url: str,
|
||||
payload: dict[str, Any],
|
||||
timeout_seconds: int = 30,
|
||||
) -> tuple[int, dict[str, Any], str | None]:
|
||||
"""POST to an agent HTTP chat route and return status, JSON body, and reply."""
|
||||
import aiohttp
|
||||
|
||||
async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=timeout_seconds)) as session:
|
||||
async with session.post(
|
||||
url,
|
||||
json=payload,
|
||||
headers={
|
||||
"Accept": "application/json",
|
||||
"Content-Type": "application/json",
|
||||
"X-LivingIP-Source": "telegram-agent-proxy",
|
||||
},
|
||||
) as resp:
|
||||
data = await resp.json(content_type=None)
|
||||
return resp.status, data, extract_chat_proxy_reply(data)
|
||||
|
|
@ -8,7 +8,6 @@ Epimetheus owns this module. Rhea: static API key pattern.
|
|||
"""
|
||||
|
||||
import logging
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import aiohttp
|
||||
|
|
@ -17,205 +16,12 @@ logger = logging.getLogger("market-data")
|
|||
|
||||
API_URL = "https://teleo-ai-api-257133920458.us-east4.run.app/v0/chat/tool/market-data"
|
||||
API_KEY_FILE = "/opt/teleo-eval/secrets/market-data-key"
|
||||
DEXSCREENER_SEARCH_URL = "https://api.dexscreener.com/latest/dex/search"
|
||||
|
||||
ENTITY_TOKEN_MAP = {
|
||||
"omnipair": "OMFG",
|
||||
"omfg": "OMFG",
|
||||
"avici": "AVICI",
|
||||
"umbra": "UMBRA",
|
||||
"metadao": "META",
|
||||
"sanctum": "CLOUD",
|
||||
"drift": "DRIFT",
|
||||
"ore": "ORE",
|
||||
"jupiter": "JUP",
|
||||
}
|
||||
|
||||
_BARE_TICKER_STOPWORDS = {
|
||||
"FDV",
|
||||
"TVL",
|
||||
"API",
|
||||
"USD",
|
||||
"USDC",
|
||||
"SOL",
|
||||
"YES",
|
||||
"NO",
|
||||
"BUY",
|
||||
"SELL",
|
||||
}
|
||||
|
||||
# Cache: avoid hitting the API on every message
|
||||
_cache: dict[str, dict] = {} # token_name → {data, timestamp}
|
||||
CACHE_TTL = 300 # 5 minutes
|
||||
|
||||
|
||||
def extract_market_data_tokens(text: str, extra_terms: list[str] | tuple[str, ...] = ()) -> list[str]:
|
||||
"""Extract token tickers from market-data questions while preserving order."""
|
||||
seen: set[str] = set()
|
||||
tokens: list[str] = []
|
||||
|
||||
def add(token: str | None) -> None:
|
||||
if not token:
|
||||
return
|
||||
normalized = token.upper().strip("$")
|
||||
if len(normalized) < 2 or normalized in _BARE_TICKER_STOPWORDS or normalized in seen:
|
||||
return
|
||||
seen.add(normalized)
|
||||
tokens.append(normalized)
|
||||
|
||||
for ticker in re.findall(r"\$([A-Za-z][A-Za-z0-9]{1,9})\b", text):
|
||||
add(ticker)
|
||||
|
||||
marketish = re.search(
|
||||
r"\b(price|volume|fdv|fully\s+diluted|market\s+cap|mcap|liquidity|buy|sell|token|coin)\b",
|
||||
text,
|
||||
flags=re.IGNORECASE,
|
||||
)
|
||||
if marketish:
|
||||
for ticker in re.findall(r"\b([A-Z][A-Z0-9]{1,9})\b", text):
|
||||
add(ticker)
|
||||
|
||||
lowered = text.lower()
|
||||
for name, ticker in ENTITY_TOKEN_MAP.items():
|
||||
if re.search(rf"\b{re.escape(name)}\b", lowered):
|
||||
add(ticker)
|
||||
|
||||
for term in extra_terms:
|
||||
term_upper = str(term).upper().strip("$")
|
||||
if term_upper in ENTITY_TOKEN_MAP.values():
|
||||
add(term_upper)
|
||||
|
||||
return tokens
|
||||
|
||||
|
||||
def _format_usd(value) -> str | None:
|
||||
try:
|
||||
number = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
if number >= 1_000_000_000:
|
||||
return f"${number / 1_000_000_000:.2f}B"
|
||||
if number >= 1_000_000:
|
||||
return f"${number / 1_000_000:.2f}M"
|
||||
if number >= 1_000:
|
||||
return f"${number / 1_000:.2f}K"
|
||||
return f"${number:,.2f}"
|
||||
|
||||
|
||||
def _needs_public_market_augmentation(data: dict) -> bool:
|
||||
result = str(data.get("result") or "").lower()
|
||||
if not result:
|
||||
return True
|
||||
return "fdv" not in result or "volume" not in result
|
||||
|
||||
|
||||
def _merge_market_data(primary: dict, public: dict) -> dict:
|
||||
merged = dict(primary)
|
||||
primary_result = str(primary.get("result") or "").strip()
|
||||
public_result = str(public.get("result") or "").strip()
|
||||
if primary_result and public_result:
|
||||
merged["result"] = f"{primary_result}\n{public_result}"
|
||||
elif public_result:
|
||||
merged["result"] = public_result
|
||||
merged["public_market_data"] = {
|
||||
k: v for k, v in public.items() if k != "pair"
|
||||
}
|
||||
merged["public_market_pair"] = public.get("pair")
|
||||
return merged
|
||||
|
||||
|
||||
def _dex_pair_score(pair: dict, token: str) -> tuple[int, float]:
|
||||
token_lower = token.lower()
|
||||
base = pair.get("baseToken") or {}
|
||||
symbol = str(base.get("symbol") or "").lower()
|
||||
name = str(base.get("name") or "").lower()
|
||||
score = 0
|
||||
if symbol == token_lower:
|
||||
score += 100
|
||||
elif token_lower in symbol:
|
||||
score += 50
|
||||
if token_lower in name:
|
||||
score += 25
|
||||
liquidity = ((pair.get("liquidity") or {}).get("usd") or 0)
|
||||
try:
|
||||
liquidity_value = float(liquidity)
|
||||
except (TypeError, ValueError):
|
||||
liquidity_value = 0
|
||||
return score, liquidity_value
|
||||
|
||||
|
||||
async def _get_dexscreener_token_data(token_name: str) -> dict | None:
|
||||
token_upper = token_name.upper().strip("$")
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(
|
||||
DEXSCREENER_SEARCH_URL,
|
||||
params={"q": token_upper},
|
||||
timeout=aiohttp.ClientTimeout(total=10),
|
||||
) as resp:
|
||||
if resp.status >= 400:
|
||||
logger.warning("DexScreener %s -> %d", token_upper, resp.status)
|
||||
return None
|
||||
body = await resp.json()
|
||||
except Exception as e:
|
||||
logger.warning("DexScreener error for %s: %s", token_upper, e)
|
||||
return None
|
||||
|
||||
pairs = body.get("pairs") or []
|
||||
if not pairs:
|
||||
return None
|
||||
|
||||
best = max(pairs, key=lambda pair: _dex_pair_score(pair, token_upper))
|
||||
score, _ = _dex_pair_score(best, token_upper)
|
||||
if score <= 0:
|
||||
return None
|
||||
|
||||
volume_24h = (best.get("volume") or {}).get("h24")
|
||||
liquidity_usd = (best.get("liquidity") or {}).get("usd")
|
||||
price_change_24h = (best.get("priceChange") or {}).get("h24")
|
||||
base = best.get("baseToken") or {}
|
||||
fdv = best.get("fdv")
|
||||
market_cap = best.get("marketCap")
|
||||
price = best.get("priceUsd")
|
||||
|
||||
parts = [
|
||||
f"Live market data for {token_upper}",
|
||||
f"source: DexScreener",
|
||||
f"pair: {base.get('symbol') or token_upper} on {best.get('chainId', 'unknown')}/{best.get('dexId', 'unknown')}",
|
||||
]
|
||||
if price:
|
||||
parts.append(f"price: ${price}")
|
||||
formatted_fdv = _format_usd(fdv)
|
||||
if formatted_fdv:
|
||||
parts.append(f"FDV: {formatted_fdv}")
|
||||
formatted_mcap = _format_usd(market_cap)
|
||||
if formatted_mcap:
|
||||
parts.append(f"market cap: {formatted_mcap}")
|
||||
formatted_volume = _format_usd(volume_24h)
|
||||
if formatted_volume:
|
||||
parts.append(f"24h volume: {formatted_volume}")
|
||||
formatted_liquidity = _format_usd(liquidity_usd)
|
||||
if formatted_liquidity:
|
||||
parts.append(f"liquidity: {formatted_liquidity}")
|
||||
if price_change_24h is not None:
|
||||
parts.append(f"24h change: {price_change_24h}%")
|
||||
if best.get("url"):
|
||||
parts.append(f"url: {best['url']}")
|
||||
|
||||
return {
|
||||
"provider": "dexscreener",
|
||||
"token": token_upper,
|
||||
"result": " | ".join(parts),
|
||||
"price": price,
|
||||
"fdv": fdv,
|
||||
"market_cap": market_cap,
|
||||
"volume_24h": volume_24h,
|
||||
"liquidity_usd": liquidity_usd,
|
||||
"price_change_24h": price_change_24h,
|
||||
"pair": best,
|
||||
}
|
||||
|
||||
|
||||
def _load_api_key() -> str | None:
|
||||
"""Load the market-data API key from secrets."""
|
||||
try:
|
||||
|
|
@ -241,41 +47,34 @@ async def get_token_price(token_name: str) -> dict | None:
|
|||
return cached["data"]
|
||||
|
||||
key = _load_api_key()
|
||||
if not key:
|
||||
return None
|
||||
|
||||
if key:
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post(
|
||||
API_URL,
|
||||
headers={
|
||||
"X-Internal-Key": key,
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
json={"token": token_upper},
|
||||
timeout=aiohttp.ClientTimeout(total=10),
|
||||
) as resp:
|
||||
if resp.status < 400:
|
||||
data = await resp.json()
|
||||
if _needs_public_market_augmentation(data):
|
||||
public_data = await _get_dexscreener_token_data(token_upper)
|
||||
if public_data:
|
||||
data = _merge_market_data(data, public_data)
|
||||
_cache[token_upper] = {
|
||||
"data": data,
|
||||
"timestamp": time.time(),
|
||||
}
|
||||
return data
|
||||
logger.warning("Market data API %s -> %d", token_upper, resp.status)
|
||||
except Exception as e:
|
||||
logger.warning("Market data API error for %s: %s", token_upper, e)
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post(
|
||||
API_URL,
|
||||
headers={
|
||||
"X-Internal-Key": key,
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
json={"token": token_upper},
|
||||
timeout=aiohttp.ClientTimeout(total=10),
|
||||
) as resp:
|
||||
if resp.status >= 400:
|
||||
logger.warning("Market data API %s → %d", token_upper, resp.status)
|
||||
return None
|
||||
data = await resp.json()
|
||||
|
||||
data = await _get_dexscreener_token_data(token_upper)
|
||||
if data:
|
||||
_cache[token_upper] = {
|
||||
"data": data,
|
||||
"timestamp": time.time(),
|
||||
}
|
||||
return data
|
||||
# Cache the result
|
||||
_cache[token_upper] = {
|
||||
"data": data,
|
||||
"timestamp": time.time(),
|
||||
}
|
||||
return data
|
||||
except Exception as e:
|
||||
logger.warning("Market data API error for %s: %s", token_upper, e)
|
||||
return None
|
||||
|
||||
|
||||
def format_price_context(data: dict, token_name: str) -> str:
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ from lib import log as logmod
|
|||
from lib.breaker import CircuitBreaker
|
||||
from lib.evaluate import evaluate_cycle
|
||||
from lib.fixer import fix_cycle as mechanical_fix_cycle
|
||||
from lib.substantive_fixer import substantive_fix_cycle, verdict_deadlock_reaper_cycle
|
||||
from lib.substantive_fixer import substantive_fix_cycle
|
||||
from lib.health import start_health_server, stop_health_server
|
||||
from lib.llm import kill_active_subprocesses
|
||||
from lib.merge import merge_cycle
|
||||
|
|
@ -91,30 +91,14 @@ async def ingest_cycle(conn, max_workers=None):
|
|||
|
||||
|
||||
async def fix_cycle(conn, max_workers=None):
|
||||
"""Combined fix stage: mechanical fixes first, then substantive fixes,
|
||||
finally the verdict-deadlock reaper.
|
||||
"""Combined fix stage: mechanical fixes first, then substantive fixes.
|
||||
|
||||
Mechanical (fixer.py): wiki link bracket stripping, $0
|
||||
Substantive (substantive_fixer.py): confidence/title/scope fixes via LLM, $0.001
|
||||
Reaper (substantive_fixer.verdict_deadlock_reaper_cycle): defense-in-depth
|
||||
for stuck-verdict PRs that the substantive fixer can't progress on.
|
||||
Hourly throttle, dry-run by default. Cost $0.
|
||||
"""
|
||||
m_fixed, m_errors = await mechanical_fix_cycle(conn, max_workers=max_workers)
|
||||
s_fixed, s_errors = await substantive_fix_cycle(conn, max_workers=max_workers)
|
||||
# Defense-in-depth: reaper exception must never block primary fix paths.
|
||||
# Same exception-isolation pattern as ingest_cycle's extract_cycle wrapper —
|
||||
# propagating would trip the fix breaker and lock out mechanical+substantive
|
||||
# for 15 min after 5 reaper failures.
|
||||
try:
|
||||
r_closed = await verdict_deadlock_reaper_cycle(conn)
|
||||
except Exception:
|
||||
import logging
|
||||
logging.getLogger("pipeline").exception(
|
||||
"Reaper cycle failed (non-fatal)"
|
||||
)
|
||||
r_closed = 0
|
||||
return m_fixed + s_fixed + r_closed, m_errors + s_errors
|
||||
return m_fixed + s_fixed, m_errors + s_errors
|
||||
|
||||
|
||||
async def snapshot_cycle(conn, max_workers=None):
|
||||
|
|
|
|||
|
|
@ -1,129 +0,0 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
SCHEMA_SQL = REPO_ROOT / "schemas" / "teleo-agent-graph-v1.sql"
|
||||
|
||||
|
||||
def test_agent_graph_schema_applies_and_models_shared_nodes():
|
||||
conn = sqlite3.connect(":memory:")
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.execute("PRAGMA foreign_keys = ON")
|
||||
conn.executescript(SCHEMA_SQL.read_text())
|
||||
|
||||
conn.executemany(
|
||||
"INSERT INTO agents (slug, display_name, archetype) VALUES (?, ?, ?)",
|
||||
[
|
||||
("leo", "Leo", "cross-domain synthesizer"),
|
||||
("theseus", "Theseus", "AI alignment"),
|
||||
],
|
||||
)
|
||||
conn.execute(
|
||||
"""INSERT INTO agent_persona_revisions
|
||||
(id, agent_slug, revision, identity, voice, role, authored_by)
|
||||
VALUES
|
||||
('persona-leo-v1', 'leo', 1, 'cross-domain synthesizer', 'direct', 'evaluate commons', 'diagram'),
|
||||
('persona-theseus-v1', 'theseus', 1, 'alignment maze navigator', 'precise', 'AI evidence lead', 'diagram')"""
|
||||
)
|
||||
conn.execute(
|
||||
"""INSERT INTO agent_strategy_revisions
|
||||
(id, agent_slug, persona_revision_id, revision, diagnosis, guiding_policy, proximate_objectives_json)
|
||||
VALUES
|
||||
('strategy-leo-v1', 'leo', 'persona-leo-v1', 1, 'coordination is the bottleneck', 'surface cross-domain isomorphisms', '[]'),
|
||||
('strategy-theseus-v1', 'theseus', 'persona-theseus-v1', 1, 'AI discourse is ungrounded', 'separate generation from evaluation', '[]')"""
|
||||
)
|
||||
conn.executemany(
|
||||
"""INSERT INTO evidence
|
||||
(id, evidence_type, title, summary, verification_status)
|
||||
VALUES (?, ?, ?, ?, 'verified')""",
|
||||
[
|
||||
("e-kim-2025", "study", "Kim et al. ICML 2025", "Shared evidence grounding coordination and verification degradation."),
|
||||
("e-arrow", "formal_result", "Arrow impossibility theorem", "Formal result grounding alignment impossibility claim."),
|
||||
],
|
||||
)
|
||||
conn.executemany(
|
||||
"""INSERT INTO claims
|
||||
(id, slug, domain, description, confidence, primary_evidence_id, status)
|
||||
VALUES (?, ?, ?, ?, ?, ?, 'accepted')""",
|
||||
[
|
||||
("c-coordination", "alignment-is-coordination", "ai-alignment", "Alignment is a coordination problem, not only a technical one.", "likely", "e-kim-2025"),
|
||||
("c-verification", "verification-degrades-with-capability", "ai-alignment", "Verification degrades as capability gaps grow.", "experimental", "e-kim-2025"),
|
||||
("c-arrow", "universal-alignment-impossible", "ai-alignment", "Universal alignment is mathematically impossible under strong aggregation assumptions.", "likely", "e-arrow"),
|
||||
],
|
||||
)
|
||||
conn.executemany(
|
||||
"""INSERT INTO claim_evidence_edges
|
||||
(id, claim_id, evidence_id, relation, weight, rationale)
|
||||
VALUES (?, ?, ?, 'supports', ?, ?)""",
|
||||
[
|
||||
("ce-kim-coordination", "c-coordination", "e-kim-2025", 0.9, "Diagram shared-node case: one evidence node grounds multiple claims."),
|
||||
("ce-kim-verification", "c-verification", "e-kim-2025", 0.8, "Same evidence also grounds verification degradation."),
|
||||
("ce-arrow", "c-arrow", "e-arrow", 0.9, "Formal result evidence."),
|
||||
],
|
||||
)
|
||||
conn.executemany(
|
||||
"""INSERT INTO agent_beliefs
|
||||
(id, agent_slug, belief_code, title, statement, falsification_criteria, is_keystone)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)""",
|
||||
[
|
||||
("b-leo-b1", "leo", "B1", "Coordination bottleneck", "Coordination is the bottleneck.", "Falsified by civ-scale pure-tech solution.", 1),
|
||||
("b-theseus-t2", "theseus", "T2", "Alignment as coordination", "Alignment is a coordination problem.", "Falsified by a robust one-agent technical alignment solution.", 1),
|
||||
("b-theseus-t4", "theseus", "T4", "Verification degradation", "Verification degrades faster than capability grows.", "Falsified by scalable oversight evidence.", 0),
|
||||
],
|
||||
)
|
||||
conn.executemany(
|
||||
"""INSERT INTO belief_claim_edges
|
||||
(id, belief_id, claim_id, relation, weight, rationale)
|
||||
VALUES (?, ?, ?, 'cites', ?, ?)""",
|
||||
[
|
||||
("bc-leo-coordination", "b-leo-b1", "c-coordination", 1.0, "Keystone belief cites shared claim."),
|
||||
("bc-theseus-coordination", "b-theseus-t2", "c-coordination", 0.9, "Different agent cites same shared claim."),
|
||||
("bc-theseus-verification", "b-theseus-t4", "c-verification", 0.9, "Belief cites verification claim."),
|
||||
("bc-theseus-arrow", "b-theseus-t2", "c-arrow", 0.6, "Belief also cites formal-result claim."),
|
||||
],
|
||||
)
|
||||
conn.execute(
|
||||
"""INSERT INTO claim_edges
|
||||
(id, from_claim_id, to_claim_id, relation, weight, rationale)
|
||||
VALUES ('edge-verification-supports-coordination', 'c-verification', 'c-coordination', 'supports', 0.6, 'Oversight degradation strengthens coordination framing.')"""
|
||||
)
|
||||
conn.execute(
|
||||
"""INSERT INTO cascade_events
|
||||
(id, changed_layer, changed_id, affected_layer, affected_id, reason)
|
||||
VALUES ('cascade-kim-to-coordination', 'evidence', 'e-kim-2025', 'claim', 'c-coordination', 'shared evidence updated')"""
|
||||
)
|
||||
|
||||
shared_evidence_count = conn.execute(
|
||||
"SELECT COUNT(*) AS n FROM claim_evidence_edges WHERE evidence_id = 'e-kim-2025'"
|
||||
).fetchone()["n"]
|
||||
shared_claim_count = conn.execute(
|
||||
"SELECT COUNT(*) AS n FROM belief_claim_edges WHERE claim_id = 'c-coordination'"
|
||||
).fetchone()["n"]
|
||||
cascade_count = conn.execute("SELECT COUNT(*) AS n FROM cascade_events").fetchone()["n"]
|
||||
|
||||
assert shared_evidence_count == 2
|
||||
assert shared_claim_count == 2
|
||||
assert cascade_count == 1
|
||||
|
||||
|
||||
def test_claim_edges_reject_self_reference():
|
||||
conn = sqlite3.connect(":memory:")
|
||||
conn.execute("PRAGMA foreign_keys = ON")
|
||||
conn.executescript(SCHEMA_SQL.read_text())
|
||||
conn.execute(
|
||||
"""INSERT INTO claims (id, slug, domain, description)
|
||||
VALUES ('c1', 'claim-one', 'ai-alignment', 'A claim specific enough to disagree with.')"""
|
||||
)
|
||||
|
||||
try:
|
||||
conn.execute(
|
||||
"""INSERT INTO claim_edges
|
||||
(id, from_claim_id, to_claim_id, relation, rationale)
|
||||
VALUES ('self', 'c1', 'c1', 'related', 'self edge should fail')"""
|
||||
)
|
||||
except sqlite3.IntegrityError:
|
||||
pass
|
||||
else:
|
||||
raise AssertionError("claim_edges allowed a self-reference")
|
||||
|
|
@ -1,129 +0,0 @@
|
|||
"""Tests for Phase 1b identity-based agent routing."""
|
||||
|
||||
from lib.agent_routing import AGENT_ORDER, classify_pr_route
|
||||
|
||||
|
||||
def _diff_for(*paths_and_lines: tuple[str, str] | str) -> str:
|
||||
chunks = []
|
||||
for item in paths_and_lines:
|
||||
if isinstance(item, tuple):
|
||||
path, line = item
|
||||
else:
|
||||
path, line = item, "+content"
|
||||
chunks.append(f"diff --git a/{path} b/{path}\n{line}")
|
||||
return "\n".join(chunks)
|
||||
|
||||
|
||||
def test_six_primary_domains_route_to_expected_agents():
|
||||
expected = {
|
||||
"grand-strategy": "Leo",
|
||||
"ai-alignment": "Theseus",
|
||||
"internet-finance": "Rio",
|
||||
"health": "Vida",
|
||||
"entertainment": "Clay",
|
||||
"space-development": "Astra",
|
||||
}
|
||||
|
||||
for domain, agent in expected.items():
|
||||
route = classify_pr_route(_diff_for(f"domains/{domain}/claim.md"))
|
||||
assert route.primary_agent == agent
|
||||
assert route.required_agents == (agent,)
|
||||
assert route.route_kind == "single"
|
||||
assert route.fallback is False
|
||||
|
||||
|
||||
def test_broadened_identity_domains_route_to_owners():
|
||||
expected = {
|
||||
"ai-systems": "Theseus",
|
||||
"living-agents": "Theseus",
|
||||
"living-capital": "Rio",
|
||||
"collective-intelligence": "Leo",
|
||||
"cultural-dynamics": "Clay",
|
||||
"energy": "Astra",
|
||||
"robotics": "Astra",
|
||||
"manufacturing": "Astra",
|
||||
"advanced-manufacturing": "Astra",
|
||||
}
|
||||
|
||||
for domain, agent in expected.items():
|
||||
route = classify_pr_route(_diff_for(f"foundations/{domain}/claim.md"))
|
||||
assert route.primary_agent == agent
|
||||
assert route.required_agents == (agent,)
|
||||
|
||||
|
||||
def test_cross_domain_ai_and_x402_requires_theseus_and_rio():
|
||||
route = classify_pr_route(
|
||||
_diff_for(
|
||||
("domains/ai-alignment/agent-wallets.md", "+AI systems route agents around x402 payments"),
|
||||
("domains/internet-finance/x402.md", "+x402 payment rail for onchain agent transactions"),
|
||||
)
|
||||
)
|
||||
|
||||
assert route.primary_agent == "Rio"
|
||||
assert set(route.required_agents) == {"Theseus", "Rio"}
|
||||
assert len(route.required_agents) == 2
|
||||
assert route.route_kind == "multi"
|
||||
|
||||
|
||||
def test_collective_ai_goals_routes_to_leo_and_theseus():
|
||||
route = classify_pr_route(
|
||||
_diff_for(
|
||||
(
|
||||
"foundations/collective-intelligence/collective-ai-goals.md",
|
||||
"+Collective AI goals and AI systems self-understanding need review.",
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
assert route.primary_agent == "Leo"
|
||||
assert route.required_agents == ("Leo", "Theseus")
|
||||
assert route.route_kind == "multi"
|
||||
|
||||
|
||||
def test_too_many_touched_domains_caps_at_two_and_marks_escalated():
|
||||
route = classify_pr_route(
|
||||
_diff_for(
|
||||
"domains/internet-finance/a.md",
|
||||
"domains/internet-finance/b.md",
|
||||
"domains/health/c.md",
|
||||
"domains/entertainment/d.md",
|
||||
"domains/space-development/e.md",
|
||||
)
|
||||
)
|
||||
|
||||
assert route.primary_agent == "Rio"
|
||||
assert route.required_agents == ("Rio", "Vida")
|
||||
assert route.route_kind == "escalated"
|
||||
assert len(route.required_agents) == 2
|
||||
|
||||
|
||||
def test_branch_prefix_used_when_diff_has_no_route_path():
|
||||
route = classify_pr_route(_diff_for("inbox/archive/source.md"), branch="vida/research-glp1")
|
||||
|
||||
assert route.primary_agent == "Vida"
|
||||
assert route.required_agents == ("Vida",)
|
||||
assert route.route_kind == "single"
|
||||
|
||||
|
||||
def test_unknown_route_falls_back_to_leo():
|
||||
route = classify_pr_route(_diff_for("docs/readme.md"), branch="misc/update")
|
||||
|
||||
assert route.primary_agent == "Leo"
|
||||
assert route.required_agents == ("Leo",)
|
||||
assert route.route_kind == "fallback"
|
||||
assert route.fallback is True
|
||||
|
||||
|
||||
def test_routing_is_deterministic_for_repeated_inputs():
|
||||
diff = _diff_for(
|
||||
("domains/health/agent-care.md", "+AI systems and health medicine review"),
|
||||
("domains/ai-systems/care-agent.md", "+clinical model behavior"),
|
||||
)
|
||||
first = classify_pr_route(diff)
|
||||
|
||||
for _ in range(100):
|
||||
assert classify_pr_route(diff) == first
|
||||
|
||||
|
||||
def test_agent_order_is_stable():
|
||||
assert AGENT_ORDER == ("Leo", "Theseus", "Rio", "Vida", "Clay", "Astra")
|
||||
|
|
@ -1,11 +1,9 @@
|
|||
"""Tests for lib/contributor.py — contributor attribution functions."""
|
||||
|
||||
# ruff: noqa: E402,I001
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sqlite3
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
sys.modules.setdefault("aiohttp", MagicMock())
|
||||
|
|
@ -178,16 +176,9 @@ def _make_attribution_db():
|
|||
conn.execute("""CREATE TABLE prs (
|
||||
number INTEGER PRIMARY KEY,
|
||||
commit_type TEXT,
|
||||
agent TEXT,
|
||||
submitted_by TEXT,
|
||||
domain TEXT,
|
||||
source_channel TEXT,
|
||||
leo_verdict TEXT,
|
||||
domain_verdict TEXT,
|
||||
domain_agent TEXT,
|
||||
merged_at TEXT
|
||||
agent TEXT
|
||||
)""")
|
||||
conn.execute("INSERT INTO prs (number, commit_type, agent) VALUES (100, 'extract', 'rio')")
|
||||
conn.execute("INSERT INTO prs VALUES (100, 'extract', 'rio')")
|
||||
return conn
|
||||
|
||||
def test_record_skips_pipeline_only():
|
||||
|
|
@ -205,19 +196,12 @@ def test_record_skips_pipeline_only():
|
|||
|
||||
def test_record_fallback_to_pr_agent():
|
||||
conn = _make_attribution_db()
|
||||
mock_diff = "diff --git a/x b/domains/crypto/claim.md\nnew file\n+++ b/domains/crypto/claim.md\n+some content\n"
|
||||
mock_diff = "+++ b/domains/crypto/claim.md\n+some content\n"
|
||||
|
||||
async def run():
|
||||
with patch("lib.contributor.get_pr_diff", new_callable=AsyncMock, return_value=mock_diff):
|
||||
# First call: trailer log (no trailers), Second call: author log (bot name → skipped)
|
||||
git_fn = AsyncMock(
|
||||
side_effect=[
|
||||
(0, "no trailers here"),
|
||||
(0, "domains/crypto/claim.md"),
|
||||
(0, ""),
|
||||
(0, "m3taversal"),
|
||||
]
|
||||
)
|
||||
git_fn = AsyncMock(side_effect=[(0, "no trailers here"), (0, "m3taversal")])
|
||||
with patch("lib.contributor.config") as mock_config:
|
||||
mock_config.CONTRIBUTOR_TIER_RULES = {
|
||||
"veteran": {"claims_merged": 50, "min_days_since_first": 90, "challenges_survived": 5},
|
||||
|
|
@ -234,23 +218,13 @@ def test_record_fallback_to_pr_agent():
|
|||
def test_record_fallback_to_git_author():
|
||||
"""External contributors get credited via git commit author."""
|
||||
conn = _make_attribution_db()
|
||||
conn.execute("INSERT INTO prs (number, commit_type, agent) VALUES (200, 'contrib', 'external')")
|
||||
mock_diff = (
|
||||
"diff --git a/x b/domains/ai-alignment/claim.md\nnew file\n"
|
||||
"+++ b/domains/ai-alignment/claim.md\n+new content\n"
|
||||
)
|
||||
conn.execute("INSERT INTO prs VALUES (200, 'contrib', 'external')")
|
||||
mock_diff = "+++ b/domains/ai-alignment/claim.md\n+new content\n"
|
||||
|
||||
async def run():
|
||||
with patch("lib.contributor.get_pr_diff", new_callable=AsyncMock, return_value=mock_diff):
|
||||
# First call: trailer log (no trailers), Second call: author log (external name)
|
||||
git_fn = AsyncMock(
|
||||
side_effect=[
|
||||
(0, "no trailers"),
|
||||
(0, "domains/ai-alignment/claim.md"),
|
||||
(0, ""),
|
||||
(0, "Cameron-S1"),
|
||||
]
|
||||
)
|
||||
git_fn = AsyncMock(side_effect=[(0, "no trailers"), (0, "Cameron-S1")])
|
||||
with patch("lib.contributor.config") as mock_config:
|
||||
mock_config.CONTRIBUTOR_TIER_RULES = {
|
||||
"veteran": {"claims_merged": 50, "min_days_since_first": 90, "challenges_survived": 5},
|
||||
|
|
|
|||
|
|
@ -1,56 +0,0 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
SCRIPT_PATH = REPO_ROOT / "scripts" / "replay_decision_engine_eval.py"
|
||||
FIXTURES_DIR = REPO_ROOT / "fixtures" / "decision-engine-eval"
|
||||
|
||||
spec = importlib.util.spec_from_file_location("replay_decision_engine_eval", SCRIPT_PATH)
|
||||
replay = importlib.util.module_from_spec(spec)
|
||||
assert spec.loader is not None
|
||||
spec.loader.exec_module(replay)
|
||||
|
||||
|
||||
def test_default_decision_engine_fixtures_replay_cleanly():
|
||||
fixtures = replay.load_fixtures(FIXTURES_DIR)
|
||||
proof = replay.evaluate_fixtures(fixtures)
|
||||
|
||||
assert proof["ok"] is True
|
||||
assert proof["fixture_count"] == 3
|
||||
assert proof["metrics"]["route_accuracy"] == 1.0
|
||||
assert proof["metrics"]["lanes"] == {
|
||||
"kb-interop": 1,
|
||||
"rio-economics": 1,
|
||||
"theseus-model-integrity": 1,
|
||||
}
|
||||
|
||||
|
||||
def test_candidate_false_approve_is_caught(tmp_path):
|
||||
fixtures = replay.load_fixtures(FIXTURES_DIR)
|
||||
candidate_path = tmp_path / "candidate.json"
|
||||
candidate_path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"candidate_name": "bad-single-answer-model",
|
||||
"verdicts": [
|
||||
{
|
||||
"fixture_id": "theseus_live_model_switch_reject",
|
||||
"disposition": "approve",
|
||||
"issue_tags": [],
|
||||
"primary_agent": "Theseus",
|
||||
"required_agents": ["Theseus"],
|
||||
}
|
||||
],
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
candidate = replay._load_candidate_output(candidate_path)
|
||||
proof = replay.evaluate_fixtures(fixtures, candidate=candidate)
|
||||
|
||||
assert proof["ok"] is False
|
||||
assert proof["candidate"]["false_approve_count"] == 1
|
||||
assert proof["candidate"]["false_approves"] == ["theseus_live_model_switch_reject"]
|
||||
|
|
@ -1,9 +1,7 @@
|
|||
"""Tests for lib/eval_parse.py — pure parsing functions extracted from evaluate.py."""
|
||||
|
||||
# ruff: noqa: E402,I001
|
||||
|
||||
import os
|
||||
import sys
|
||||
import os
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
|
@ -14,6 +12,7 @@ sys.modules.setdefault("aiohttp", MagicMock())
|
|||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
|
||||
|
||||
from lib.eval_parse import (
|
||||
VALID_ISSUE_TAGS,
|
||||
classify_issues,
|
||||
deterministic_tier,
|
||||
diff_contains_claim_type,
|
||||
|
|
@ -41,7 +40,7 @@ class TestFilterDiff:
|
|||
"diff --git a/domains/finance/claim.md b/domains/finance/claim.md\n"
|
||||
"+real content\n"
|
||||
)
|
||||
review_diff, _entity_diff = filter_diff(diff)
|
||||
review_diff, entity_diff = filter_diff(diff)
|
||||
assert "inbox" not in review_diff
|
||||
assert "claim.md" in review_diff
|
||||
|
||||
|
|
@ -171,11 +170,6 @@ class TestParseVerdict:
|
|||
def test_case_insensitive_reviewer(self):
|
||||
assert parse_verdict("VERDICT:LEO:APPROVE", "leo") == "approve"
|
||||
|
||||
@pytest.mark.parametrize("agent", ["LEO", "THESEUS", "RIO", "VIDA", "CLAY", "ASTRA"])
|
||||
def test_phase1b_agent_verdicts(self, agent):
|
||||
assert parse_verdict(f"<!-- VERDICT:{agent}:APPROVE -->", agent) == "approve"
|
||||
assert parse_verdict(f"<!-- VERDICT:{agent}:REQUEST_CHANGES -->", agent) == "request_changes"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# normalize_tag
|
||||
|
|
|
|||
|
|
@ -1,238 +0,0 @@
|
|||
"""Tests for Phase 1b eval integration."""
|
||||
|
||||
import sqlite3
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
|
||||
from lib import config
|
||||
from lib.evaluate import _evaluate_pr_phase1b, _post_phase1b_review_comment, evaluate_pr
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def phase1b_conn():
|
||||
conn = sqlite3.connect(":memory:")
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.executescript(
|
||||
"""
|
||||
CREATE TABLE prs (
|
||||
number INTEGER PRIMARY KEY,
|
||||
source_path TEXT,
|
||||
branch TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'open',
|
||||
domain TEXT,
|
||||
agent TEXT,
|
||||
tier TEXT,
|
||||
tier0_pass INTEGER,
|
||||
leo_verdict TEXT DEFAULT 'pending',
|
||||
domain_verdict TEXT DEFAULT 'pending',
|
||||
domain_agent TEXT,
|
||||
domain_model TEXT,
|
||||
eval_attempts INTEGER DEFAULT 0,
|
||||
eval_issues TEXT DEFAULT '[]',
|
||||
merge_cycled INTEGER DEFAULT 0,
|
||||
last_error TEXT,
|
||||
last_attempt TEXT,
|
||||
cost_usd REAL DEFAULT 0,
|
||||
auto_merge INTEGER DEFAULT 0,
|
||||
created_at TEXT DEFAULT (datetime('now')),
|
||||
merged_at TEXT
|
||||
);
|
||||
CREATE TABLE sources (
|
||||
path TEXT PRIMARY KEY,
|
||||
status TEXT DEFAULT 'extracted',
|
||||
feedback TEXT
|
||||
);
|
||||
CREATE TABLE audit_log (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
stage TEXT,
|
||||
event TEXT,
|
||||
detail TEXT
|
||||
);
|
||||
CREATE TABLE review_records (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
pr_number INTEGER NOT NULL,
|
||||
claim_path TEXT,
|
||||
domain TEXT,
|
||||
agent TEXT,
|
||||
reviewer TEXT,
|
||||
reviewer_model TEXT,
|
||||
outcome TEXT NOT NULL,
|
||||
rejection_reason TEXT,
|
||||
disagreement_type TEXT,
|
||||
notes TEXT,
|
||||
batch_id TEXT,
|
||||
claims_in_batch INTEGER,
|
||||
reviewed_at TEXT DEFAULT (datetime('now'))
|
||||
);
|
||||
CREATE TABLE costs (
|
||||
date TEXT,
|
||||
model TEXT,
|
||||
stage TEXT,
|
||||
calls INTEGER DEFAULT 0,
|
||||
input_tokens INTEGER DEFAULT 0,
|
||||
output_tokens INTEGER DEFAULT 0,
|
||||
cost_usd REAL DEFAULT 0,
|
||||
duration_ms INTEGER DEFAULT 0,
|
||||
cache_read_tokens INTEGER DEFAULT 0,
|
||||
cache_write_tokens INTEGER DEFAULT 0,
|
||||
cost_estimate_usd REAL DEFAULT 0,
|
||||
PRIMARY KEY (date, model, stage)
|
||||
);
|
||||
"""
|
||||
)
|
||||
yield conn
|
||||
conn.close()
|
||||
|
||||
|
||||
def _diff_for(*paths: str) -> str:
|
||||
return "\n".join(f"diff --git a/{path} b/{path}\n+type: claim\n+description: test" for path in paths)
|
||||
|
||||
|
||||
def _insert_pr(conn, number=1, branch="rio/test", source_path="inbox/archive/test.md"):
|
||||
conn.execute("INSERT INTO sources (path, status) VALUES (?, ?)", (source_path, "extracted"))
|
||||
conn.execute(
|
||||
"""INSERT INTO prs
|
||||
(number, source_path, branch, status, tier, tier0_pass, leo_verdict, domain_verdict, eval_attempts)
|
||||
VALUES (?, ?, ?, 'open', 'STANDARD', 1, 'pending', 'pending', 0)""",
|
||||
(number, source_path, branch),
|
||||
)
|
||||
|
||||
|
||||
async def _fake_agent_review(_diff, _files, agent, _route_context, tier="STANDARD"):
|
||||
return f"{agent} review\n<!-- VERDICT:{agent.upper()}:APPROVE -->", {
|
||||
"prompt_tokens": 10,
|
||||
"completion_tokens": 5,
|
||||
}
|
||||
|
||||
|
||||
async def _fake_agent_review_reject_vida(_diff, _files, agent, _route_context, tier="STANDARD"):
|
||||
verdict = "REQUEST_CHANGES" if agent == "Vida" else "APPROVE"
|
||||
issues = "\n<!-- ISSUES: factual_discrepancy -->" if verdict == "REQUEST_CHANGES" else ""
|
||||
return f"{agent} review{issues}\n<!-- VERDICT:{agent.upper()}:{verdict} -->", {
|
||||
"prompt_tokens": 10,
|
||||
"completion_tokens": 5,
|
||||
}
|
||||
|
||||
|
||||
async def _fake_forgejo_api(method, path, body=None, token=None):
|
||||
if method == "GET" and "comments" in path:
|
||||
return []
|
||||
if method == "GET" and "pulls/" in path:
|
||||
return {"user": {"login": "contributor"}}
|
||||
return {"id": 1}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_phase1b_cross_domain_approves_after_all_required_agents(phase1b_conn, monkeypatch):
|
||||
conn = phase1b_conn
|
||||
_insert_pr(conn, branch="rio/ai-x402")
|
||||
monkeypatch.setattr("lib.evaluate.run_agent_review", _fake_agent_review)
|
||||
monkeypatch.setattr("lib.evaluate.forgejo_api", _fake_forgejo_api)
|
||||
post_formal = AsyncMock()
|
||||
monkeypatch.setattr("lib.evaluate.post_formal_approvals", post_formal)
|
||||
monkeypatch.setattr("lib.evaluate.on_eval_complete", AsyncMock())
|
||||
|
||||
diff = _diff_for("domains/ai-systems/agent-wallets.md", "domains/internet-finance/x402.md")
|
||||
result = await _evaluate_pr_phase1b(
|
||||
conn,
|
||||
1,
|
||||
tier="STANDARD",
|
||||
diff=diff,
|
||||
review_diff=diff,
|
||||
files="domains/ai-systems/agent-wallets.md\ndomains/internet-finance/x402.md",
|
||||
branch_name="rio/ai-x402",
|
||||
eval_attempts=1,
|
||||
pr_cost=0,
|
||||
)
|
||||
|
||||
assert result["approved"] is True
|
||||
assert set(result["agent_verdicts"]) == {"Theseus", "Rio"}
|
||||
row = conn.execute("SELECT status, domain, domain_agent, leo_verdict, domain_verdict FROM prs WHERE number = 1").fetchone()
|
||||
assert row["status"] == "approved"
|
||||
assert row["domain"] == "multi"
|
||||
assert row["leo_verdict"] == "skipped"
|
||||
assert row["domain_verdict"] == "approve"
|
||||
assert row["domain_agent"] in {"Theseus", "Rio"}
|
||||
review_count = conn.execute("SELECT COUNT(*) AS n FROM review_records WHERE pr_number = 1").fetchone()["n"]
|
||||
assert review_count == 2
|
||||
reviewers = {
|
||||
row["agent"] for row in conn.execute("SELECT agent FROM review_records WHERE pr_number = 1").fetchall()
|
||||
}
|
||||
assert reviewers == {"Theseus", "Rio"}
|
||||
post_formal.assert_awaited_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_phase1b_request_changes_blocks_merge(phase1b_conn, monkeypatch):
|
||||
conn = phase1b_conn
|
||||
_insert_pr(conn, branch="vida/health")
|
||||
monkeypatch.setattr("lib.evaluate.run_agent_review", _fake_agent_review_reject_vida)
|
||||
monkeypatch.setattr("lib.evaluate.forgejo_api", _fake_forgejo_api)
|
||||
monkeypatch.setattr("lib.evaluate.post_formal_approvals", AsyncMock())
|
||||
dispose = AsyncMock()
|
||||
monkeypatch.setattr("lib.evaluate.dispose_rejected_pr", dispose)
|
||||
monkeypatch.setattr("lib.evaluate.on_eval_complete", AsyncMock())
|
||||
|
||||
diff = _diff_for("domains/health/claim.md")
|
||||
result = await _evaluate_pr_phase1b(
|
||||
conn,
|
||||
1,
|
||||
tier="STANDARD",
|
||||
diff=diff,
|
||||
review_diff=diff,
|
||||
files="domains/health/claim.md",
|
||||
branch_name="vida/health",
|
||||
eval_attempts=1,
|
||||
pr_cost=0,
|
||||
)
|
||||
|
||||
assert result["approved"] is False
|
||||
assert result["agent_verdicts"] == {"Vida": "request_changes"}
|
||||
row = conn.execute("SELECT status, domain_agent, domain_verdict, eval_issues FROM prs WHERE number = 1").fetchone()
|
||||
assert row["status"] == "open"
|
||||
assert row["domain_agent"] == "Vida"
|
||||
assert row["domain_verdict"] == "request_changes"
|
||||
assert "factual_discrepancy" in row["eval_issues"]
|
||||
dispose.assert_awaited_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_evaluate_pr_flag_uses_phase1b_and_not_legacy_reviewers(phase1b_conn, monkeypatch):
|
||||
conn = phase1b_conn
|
||||
_insert_pr(conn, branch="rio/x402")
|
||||
monkeypatch.setattr(config, "PHASE1B_AGENT_ROUTING_ENABLED", True)
|
||||
monkeypatch.setattr("lib.evaluate.get_pr_diff", AsyncMock(return_value=_diff_for("domains/internet-finance/x402.md")))
|
||||
monkeypatch.setattr("lib.evaluate.run_agent_review", _fake_agent_review)
|
||||
legacy_domain = AsyncMock()
|
||||
legacy_leo = AsyncMock()
|
||||
monkeypatch.setattr("lib.evaluate.run_domain_review", legacy_domain)
|
||||
monkeypatch.setattr("lib.evaluate.run_leo_review", legacy_leo)
|
||||
monkeypatch.setattr("lib.evaluate.forgejo_api", _fake_forgejo_api)
|
||||
monkeypatch.setattr("lib.evaluate.post_formal_approvals", AsyncMock())
|
||||
monkeypatch.setattr("lib.evaluate.on_eval_complete", AsyncMock())
|
||||
|
||||
result = await evaluate_pr(conn, 1, tier="STANDARD")
|
||||
|
||||
assert result["phase1b"] is True
|
||||
assert result["agent_verdicts"] == {"Rio": "approve"}
|
||||
legacy_domain.assert_not_awaited()
|
||||
legacy_leo.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_phase1b_review_comment_is_idempotent(monkeypatch):
|
||||
calls = []
|
||||
|
||||
async def fake_api(method, path, body=None, token=None):
|
||||
calls.append((method, path, body))
|
||||
if method == "GET":
|
||||
return [{"body": "<!-- PHASE1B_REVIEW:PR=7:AGENT=RIO -->\nold review"}]
|
||||
return {"id": 1}
|
||||
|
||||
monkeypatch.setattr("lib.evaluate.forgejo_api", fake_api)
|
||||
|
||||
posted = await _post_phase1b_review_comment(7, "Rio", "new review\n<!-- VERDICT:RIO:APPROVE -->")
|
||||
|
||||
assert posted is False
|
||||
assert [call[0] for call in calls] == ["GET"]
|
||||
|
|
@ -1,68 +0,0 @@
|
|||
"""Tests for safe Telegram agent token installation."""
|
||||
|
||||
import json
|
||||
import os
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
SCRIPT = REPO_ROOT / "scripts" / "install_telegram_agent_token.py"
|
||||
|
||||
|
||||
def run_installer(args: list[str], *, token: str = "123456789:abcdefghijklmnopqrstuvwxyzABC") -> subprocess.CompletedProcess:
|
||||
return subprocess.run(
|
||||
[sys.executable, str(SCRIPT), *args],
|
||||
input=token,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=False,
|
||||
)
|
||||
|
||||
|
||||
def test_installs_leo_wallet_test_token_from_stdin_without_echoing_secret(tmp_path):
|
||||
token = "123456789:abcdefghijklmnopqrstuvwxyzABC"
|
||||
proof_path = tmp_path / "proof.json"
|
||||
proc = run_installer(
|
||||
[
|
||||
"--agent",
|
||||
"leo-wallet-test",
|
||||
"--repo-root",
|
||||
str(REPO_ROOT),
|
||||
"--secrets-dir",
|
||||
str(tmp_path / "secrets"),
|
||||
"--from-stdin",
|
||||
"--no-chown",
|
||||
"--skip-validate",
|
||||
"--output",
|
||||
str(proof_path),
|
||||
],
|
||||
token=token,
|
||||
)
|
||||
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
assert token not in proc.stdout
|
||||
assert token not in proc.stderr
|
||||
|
||||
proof = json.loads(proof_path.read_text())
|
||||
token_path = Path(proof["tokenPath"])
|
||||
assert proof["ok"] is True
|
||||
assert proof["agent"] == "leo-wallet-test"
|
||||
assert proof["secretValuesIncluded"] is False
|
||||
assert proof["tokenFileWritten"] is True
|
||||
assert token not in proof_path.read_text()
|
||||
|
||||
assert token_path.read_text().strip() == token
|
||||
mode = stat.S_IMODE(os.stat(token_path).st_mode)
|
||||
assert mode == 0o600
|
||||
|
||||
|
||||
def test_refuses_cli_token_argument_without_echoing_secret():
|
||||
token = "123456789:abcdefghijklmnopqrstuvwxyzABC"
|
||||
proc = run_installer(["--token", token], token="")
|
||||
|
||||
combined_output = proc.stdout + proc.stderr
|
||||
assert proc.returncode == 2
|
||||
assert token not in combined_output
|
||||
assert "Secret-bearing CLI args are not accepted" in combined_output
|
||||
|
|
@ -1,110 +0,0 @@
|
|||
"""Tests for safe Telegram smart-research gate installation."""
|
||||
|
||||
import json
|
||||
import os
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
SCRIPT = REPO_ROOT / "scripts" / "install_telegram_smart_research_gates.py"
|
||||
|
||||
|
||||
def run_installer(
|
||||
args: list[str],
|
||||
*,
|
||||
approval_ref: str = "approval_ref_livingip_x402_20260622",
|
||||
) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(
|
||||
[sys.executable, str(SCRIPT), *args],
|
||||
input=approval_ref,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=False,
|
||||
)
|
||||
|
||||
|
||||
def test_installs_paid_gate_from_stdin_without_echoing_secret_or_chat_id(tmp_path):
|
||||
approval_ref = "approval_ref_livingip_x402_20260622"
|
||||
chat_id = "-1001234567890"
|
||||
proof_path = tmp_path / "proof.json"
|
||||
proc = run_installer(
|
||||
[
|
||||
"--agent",
|
||||
"leo-wallet-test",
|
||||
"--secrets-dir",
|
||||
str(tmp_path / "secrets"),
|
||||
"--allow-paid",
|
||||
"--allowed-chat-id",
|
||||
chat_id,
|
||||
"--max-usd",
|
||||
"0.06",
|
||||
"--approval-ref-from-stdin",
|
||||
"--no-chown",
|
||||
"--output",
|
||||
str(proof_path),
|
||||
],
|
||||
approval_ref=approval_ref,
|
||||
)
|
||||
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
combined_output = proc.stdout + proc.stderr + proof_path.read_text()
|
||||
assert approval_ref not in combined_output
|
||||
assert chat_id not in combined_output
|
||||
|
||||
proof = json.loads(proof_path.read_text())
|
||||
env_path = Path(proof["envPath"])
|
||||
approval_path = Path(proof["approvalRefPath"])
|
||||
assert proof["ok"] is True
|
||||
assert proof["agent"] == "leo-wallet-test"
|
||||
assert proof["paidEnabled"] is True
|
||||
assert proof["approvalRefPresent"] is True
|
||||
assert proof["allowedChatIdPresent"] is True
|
||||
assert proof["maxUsd"] == "0.06"
|
||||
assert proof["secretValuesIncluded"] is False
|
||||
|
||||
env_content = env_path.read_text()
|
||||
assert "LIVINGIP_LEO_TELEGRAM_SMART_RESEARCH_ALLOW_PAID=1" in env_content
|
||||
assert "LIVINGIP_LEO_TELEGRAM_SMART_RESEARCH_MAX_USD=0.06" in env_content
|
||||
assert f"LIVINGIP_LEO_TELEGRAM_SMART_RESEARCH_ALLOWED_CHAT_ID={chat_id}" in env_content
|
||||
assert f"LIVINGIP_LEO_TELEGRAM_SMART_RESEARCH_APPROVAL_REF_FILE={approval_path}" in env_content
|
||||
assert approval_path.read_text().strip() == approval_ref
|
||||
|
||||
assert stat.S_IMODE(os.stat(env_path).st_mode) == 0o600
|
||||
assert stat.S_IMODE(os.stat(approval_path).st_mode) == 0o600
|
||||
|
||||
|
||||
def test_installs_disabled_gate_without_approval_ref(tmp_path):
|
||||
proof_path = tmp_path / "proof.json"
|
||||
proc = run_installer(
|
||||
[
|
||||
"--agent",
|
||||
"leo-wallet-test",
|
||||
"--secrets-dir",
|
||||
str(tmp_path / "secrets"),
|
||||
"--no-chown",
|
||||
"--output",
|
||||
str(proof_path),
|
||||
],
|
||||
approval_ref="",
|
||||
)
|
||||
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
proof = json.loads(proof_path.read_text())
|
||||
env_path = Path(proof["envPath"])
|
||||
approval_path = Path(proof["approvalRefPath"])
|
||||
assert proof["paidEnabled"] is False
|
||||
assert proof["approvalRefWritten"] is False
|
||||
assert "LIVINGIP_LEO_TELEGRAM_SMART_RESEARCH_ALLOW_PAID=0" in env_path.read_text()
|
||||
assert not approval_path.exists()
|
||||
|
||||
|
||||
def test_refuses_cli_approval_ref_argument_without_echoing_secret():
|
||||
approval_ref = "approval_ref_livingip_x402_20260622"
|
||||
proc = run_installer(["--approval-ref", approval_ref], approval_ref="")
|
||||
|
||||
combined_output = proc.stdout + proc.stderr
|
||||
assert proc.returncode == 2
|
||||
assert approval_ref not in combined_output
|
||||
assert "Secret-bearing CLI args are not accepted" in combined_output
|
||||
|
|
@ -1,31 +0,0 @@
|
|||
"""End-to-end local proof for Phase 1b agent routing."""
|
||||
|
||||
import pytest
|
||||
|
||||
from scripts.prove_phase1b_local import CROSS_DOMAIN_CASE, FEEDBACK_CASE, SINGLE_DOMAIN_CASES, run_phase1b_local_proof
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_phase1b_local_eval_cycle_routes_reviews_approves_and_feedbacks():
|
||||
proof = await run_phase1b_local_proof()
|
||||
|
||||
assert proof["scope"] == "local_no_network_phase1b_eval_cycle"
|
||||
assert proof["succeeded"] == len(SINGLE_DOMAIN_CASES) + 2
|
||||
assert proof["failed"] == 0
|
||||
assert proof["agents_seen"] == ["Astra", "Clay", "Leo", "Rio", "Theseus", "Vida"]
|
||||
|
||||
results = {case["number"]: case for case in proof["case_results"]}
|
||||
for case in SINGLE_DOMAIN_CASES:
|
||||
result = results[case["number"]]
|
||||
assert result["status"] == "approved"
|
||||
assert result["reviewers"] == sorted(case["expected_agents"])
|
||||
|
||||
cross_domain = results[CROSS_DOMAIN_CASE["number"]]
|
||||
assert cross_domain["status"] == "approved"
|
||||
assert cross_domain["reviewers"] == sorted(CROSS_DOMAIN_CASE["expected_agents"])
|
||||
|
||||
feedback = results[FEEDBACK_CASE["number"]]
|
||||
assert feedback["status"] == "open"
|
||||
assert feedback["reviewers"] == ["Vida"]
|
||||
assert feedback["domain_verdict"] == "request_changes"
|
||||
assert proof["source_feedback_paths"] == [f"inbox/archive/phase1b-{FEEDBACK_CASE['number']}.md"]
|
||||
|
|
@ -1,365 +0,0 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
GRAPH_SCHEMA_SQL = REPO_ROOT / "schemas" / "teleo-agent-graph-v1.sql"
|
||||
RESEARCH_EVAL_SCHEMA_SQL = REPO_ROOT / "schemas" / "teleo-agent-research-eval-v1.sql"
|
||||
|
||||
|
||||
def _conn() -> sqlite3.Connection:
|
||||
conn = sqlite3.connect(":memory:")
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.execute("PRAGMA foreign_keys = ON")
|
||||
conn.executescript(GRAPH_SCHEMA_SQL.read_text())
|
||||
conn.executescript(RESEARCH_EVAL_SCHEMA_SQL.read_text())
|
||||
return conn
|
||||
|
||||
|
||||
def test_research_eval_schema_applies_after_graph_schema():
|
||||
conn = _conn()
|
||||
|
||||
versions = {
|
||||
row["version"]
|
||||
for row in conn.execute("SELECT version FROM graph_schema_version").fetchall()
|
||||
}
|
||||
assert versions == {
|
||||
"teleo-agent-graph-v1",
|
||||
"teleo-agent-research-eval-v1",
|
||||
}
|
||||
|
||||
tables = {
|
||||
row["name"]
|
||||
for row in conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type = 'table'"
|
||||
).fetchall()
|
||||
}
|
||||
assert {
|
||||
"agent_research_runs",
|
||||
"agent_tool_invocations",
|
||||
"agent_research_sources",
|
||||
"agent_eval_cases",
|
||||
"agent_eval_results",
|
||||
"work_order_graph_links",
|
||||
} <= tables
|
||||
|
||||
|
||||
def test_ranger_liquidation_case_routes_to_source_backed_research_not_market_data():
|
||||
conn = _conn()
|
||||
conn.execute(
|
||||
"INSERT INTO agents (slug, display_name, archetype) VALUES ('leo', 'Leo', 'research agent')"
|
||||
)
|
||||
conn.execute(
|
||||
"""INSERT INTO agent_eval_cases
|
||||
(id, suite_id, case_slug, prompt_sha256, prompt_excerpt, expected_route,
|
||||
expected_provider, must_use_tools_json, must_not_use_tools_json, tags_json, rubric_json)
|
||||
VALUES
|
||||
(
|
||||
'eval-ranger-liquidated-v1',
|
||||
'leo-research-routing-v1',
|
||||
'ranger-liquidated-not-fair-value',
|
||||
'sha256:ranger-prompt',
|
||||
'Is Ranger Finance fairly valued today given Ranger Finance is liquidated and gone?',
|
||||
'web_search',
|
||||
'agentcash-stableenrich-exa-search',
|
||||
'["source-backed web research"]',
|
||||
'["structured_market_data_only", "live_token_fair_value"]',
|
||||
'["ranger_liquidated", "valuation", "source_verification"]',
|
||||
'{"routing": "verify liquidation before valuation framing"}'
|
||||
)"""
|
||||
)
|
||||
conn.execute(
|
||||
"""INSERT INTO agent_research_runs
|
||||
(id, agent_slug, source_surface, source_ref, request_kind, sponsored_work_order_id,
|
||||
payment_receipt_id, prompt_sha256, prompt_excerpt, selected_provider, selected_route,
|
||||
status, answer_sha256, answer_excerpt, proof_ref, cost_amount, latency_ms, source_count)
|
||||
VALUES
|
||||
(
|
||||
'run-ranger-liquidated-001',
|
||||
'leo',
|
||||
'telegram',
|
||||
'telegram:group:message-123',
|
||||
'paid_quote',
|
||||
'sponsored_work_orders:test-ranger-001',
|
||||
'payment_receipts:test-ranger-001',
|
||||
'sha256:ranger-prompt',
|
||||
'Is Ranger Finance fairly valued today given Ranger Finance is liquidated and gone?',
|
||||
'agentcash-stableenrich-exa-search',
|
||||
'web_search',
|
||||
'answered',
|
||||
'sha256:ranger-answer',
|
||||
'Verified liquidation/gone status before valuation framing.',
|
||||
'proof/leo-ranger-liquidated-routing.json',
|
||||
0.01,
|
||||
1240,
|
||||
3
|
||||
)"""
|
||||
)
|
||||
conn.executemany(
|
||||
"""INSERT INTO agent_tool_invocations
|
||||
(id, research_run_id, sequence, provider, tool_name, tool_category, decision,
|
||||
decision_reason, paid, rail, network, amount, payment_receipt_id, input_sha256,
|
||||
output_sha256, source_count, latency_ms)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
[
|
||||
(
|
||||
"tool-ranger-market-rejected",
|
||||
"run-ranger-liquidated-001",
|
||||
1,
|
||||
"DexScreener",
|
||||
"structured-market-context",
|
||||
"market_data",
|
||||
"rejected",
|
||||
"Ranger liquidation status must be verified before treating this as a live token valuation.",
|
||||
0,
|
||||
"free",
|
||||
None,
|
||||
0,
|
||||
None,
|
||||
"sha256:market-input",
|
||||
None,
|
||||
0,
|
||||
12,
|
||||
),
|
||||
(
|
||||
"tool-ranger-web-selected",
|
||||
"run-ranger-liquidated-001",
|
||||
2,
|
||||
"AgentCash StableEnrich",
|
||||
"exa-search",
|
||||
"web_search",
|
||||
"executed",
|
||||
"Source-backed liquidation and status verification required.",
|
||||
1,
|
||||
"agentcash",
|
||||
"solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp",
|
||||
0.01,
|
||||
"payment_receipts:test-ranger-001",
|
||||
"sha256:exa-input",
|
||||
"sha256:exa-output",
|
||||
3,
|
||||
1228,
|
||||
),
|
||||
],
|
||||
)
|
||||
conn.executemany(
|
||||
"""INSERT INTO agent_research_sources
|
||||
(id, research_run_id, tool_invocation_id, source_type, source_uri_sha256,
|
||||
title, cited, retrieval_rank, support_status)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
[
|
||||
(
|
||||
"source-ranger-official",
|
||||
"run-ranger-liquidated-001",
|
||||
"tool-ranger-web-selected",
|
||||
"web",
|
||||
"sha256:ranger-official",
|
||||
"Ranger status source",
|
||||
1,
|
||||
1,
|
||||
"supports",
|
||||
),
|
||||
(
|
||||
"source-ranger-forum",
|
||||
"run-ranger-liquidated-001",
|
||||
"tool-ranger-web-selected",
|
||||
"web",
|
||||
"sha256:ranger-forum",
|
||||
"MetaDAO/Ranger discussion source",
|
||||
1,
|
||||
2,
|
||||
"context",
|
||||
),
|
||||
],
|
||||
)
|
||||
conn.execute(
|
||||
"""INSERT INTO graph_evaluation_runs
|
||||
(id, target_layer, target_id, trigger_type, evaluator, verdict, confidence, notes)
|
||||
VALUES
|
||||
(
|
||||
'graph-eval-ranger-routing',
|
||||
'claim',
|
||||
'ranger-liquidated-status',
|
||||
'manual',
|
||||
'leo-research-routing-benchmark',
|
||||
'approve',
|
||||
0.92,
|
||||
'Tool choice matched Ranger liquidation guard.'
|
||||
)"""
|
||||
)
|
||||
conn.execute(
|
||||
"""INSERT INTO agent_eval_results
|
||||
(id, eval_case_id, research_run_id, graph_evaluation_run_id, status, score,
|
||||
routing_correct, tool_choice_score, source_quality_score, groundedness_score,
|
||||
freshness_score, cost_efficiency_score, safety_payment_score, proof_ref)
|
||||
VALUES
|
||||
(
|
||||
'eval-result-ranger-liquidated-001',
|
||||
'eval-ranger-liquidated-v1',
|
||||
'run-ranger-liquidated-001',
|
||||
'graph-eval-ranger-routing',
|
||||
'passed',
|
||||
0.94,
|
||||
1,
|
||||
1.0,
|
||||
0.9,
|
||||
0.9,
|
||||
0.85,
|
||||
0.8,
|
||||
1.0,
|
||||
'proof/leo-ranger-liquidated-routing.json'
|
||||
)"""
|
||||
)
|
||||
conn.execute(
|
||||
"""INSERT INTO work_order_graph_links
|
||||
(id, sponsored_work_order_id, role, graph_layer, graph_id, rationale)
|
||||
VALUES
|
||||
(
|
||||
'wo-ranger-run-link',
|
||||
'sponsored_work_orders:test-ranger-001',
|
||||
'research_run',
|
||||
'agent_research_run',
|
||||
'run-ranger-liquidated-001',
|
||||
'Paid work order produced source-backed research run.'
|
||||
)"""
|
||||
)
|
||||
|
||||
row = conn.execute(
|
||||
"""SELECT
|
||||
r.selected_route,
|
||||
r.selected_provider,
|
||||
er.status AS eval_status,
|
||||
er.routing_correct,
|
||||
er.tool_choice_score,
|
||||
COUNT(s.id) AS cited_source_count
|
||||
FROM agent_research_runs r
|
||||
JOIN agent_eval_results er ON er.research_run_id = r.id
|
||||
LEFT JOIN agent_research_sources s ON s.research_run_id = r.id AND s.cited = 1
|
||||
WHERE r.id = 'run-ranger-liquidated-001'
|
||||
GROUP BY r.id, er.id"""
|
||||
).fetchone()
|
||||
|
||||
market_executed = conn.execute(
|
||||
"""SELECT COUNT(*) AS n
|
||||
FROM agent_tool_invocations
|
||||
WHERE research_run_id = 'run-ranger-liquidated-001'
|
||||
AND tool_category = 'market_data'
|
||||
AND decision = 'executed'"""
|
||||
).fetchone()["n"]
|
||||
rejected_market = conn.execute(
|
||||
"""SELECT COUNT(*) AS n
|
||||
FROM agent_tool_invocations
|
||||
WHERE research_run_id = 'run-ranger-liquidated-001'
|
||||
AND tool_category = 'market_data'
|
||||
AND decision = 'rejected'"""
|
||||
).fetchone()["n"]
|
||||
|
||||
assert dict(row) == {
|
||||
"selected_route": "web_search",
|
||||
"selected_provider": "agentcash-stableenrich-exa-search",
|
||||
"eval_status": "passed",
|
||||
"routing_correct": 1,
|
||||
"tool_choice_score": 1.0,
|
||||
"cited_source_count": 2,
|
||||
}
|
||||
assert market_executed == 0
|
||||
assert rejected_market == 1
|
||||
|
||||
|
||||
def test_schema_rejects_secret_flags_bad_scores_and_bad_tool_decisions():
|
||||
conn = _conn()
|
||||
conn.execute(
|
||||
"INSERT INTO agents (slug, display_name, archetype) VALUES ('leo', 'Leo', 'research agent')"
|
||||
)
|
||||
conn.execute(
|
||||
"""INSERT INTO agent_research_runs
|
||||
(id, agent_slug, source_surface, request_kind, prompt_sha256, selected_route, status)
|
||||
VALUES ('run-constraints', 'leo', 'test', 'benchmark', 'sha256:prompt', 'web_search', 'answered')"""
|
||||
)
|
||||
conn.execute(
|
||||
"""INSERT INTO agent_eval_cases
|
||||
(id, suite_id, case_slug, prompt_sha256, prompt_excerpt, expected_route)
|
||||
VALUES ('case-constraints', 'suite', 'case', 'sha256:prompt', 'redacted prompt', 'web_search')"""
|
||||
)
|
||||
|
||||
invalid_statements = [
|
||||
"""INSERT INTO agent_research_runs
|
||||
(id, agent_slug, source_surface, request_kind, prompt_sha256, selected_route, status, secret_values_included)
|
||||
VALUES ('run-secret', 'leo', 'test', 'benchmark', 'sha256:secret', 'web_search', 'answered', 1)""",
|
||||
"""INSERT INTO agent_tool_invocations
|
||||
(id, research_run_id, provider, tool_name, tool_category, decision, decision_reason)
|
||||
VALUES ('tool-bad-decision', 'run-constraints', 'p', 't', 'web_search', 'approved', 'bad enum')""",
|
||||
"""INSERT INTO agent_eval_results
|
||||
(id, eval_case_id, research_run_id, status, score)
|
||||
VALUES ('eval-bad-score', 'case-constraints', 'run-constraints', 'passed', 1.1)""",
|
||||
"""INSERT INTO agent_eval_results
|
||||
(id, eval_case_id, research_run_id, status, routing_correct)
|
||||
VALUES ('eval-bad-bool', 'case-constraints', 'run-constraints', 'passed', 2)""",
|
||||
]
|
||||
|
||||
for statement in invalid_statements:
|
||||
try:
|
||||
conn.execute(statement)
|
||||
except sqlite3.IntegrityError:
|
||||
pass
|
||||
else:
|
||||
raise AssertionError(f"invalid statement unexpectedly passed: {statement}")
|
||||
|
||||
|
||||
def test_research_run_can_be_recorded_without_raw_prompt_or_private_payloads():
|
||||
conn = _conn()
|
||||
conn.execute(
|
||||
"INSERT INTO agents (slug, display_name, archetype) VALUES ('leo', 'Leo', 'research agent')"
|
||||
)
|
||||
conn.execute(
|
||||
"""INSERT INTO agent_research_runs
|
||||
(id, agent_slug, source_surface, source_ref, request_kind, prompt_sha256,
|
||||
selected_route, status, answer_sha256, proof_ref)
|
||||
VALUES
|
||||
(
|
||||
'run-hash-only',
|
||||
'leo',
|
||||
'api',
|
||||
'api:request-redacted',
|
||||
'paid_work_order',
|
||||
'sha256:prompt-only',
|
||||
'social_trends',
|
||||
'answered',
|
||||
'sha256:answer-only',
|
||||
'proof/hash-only.json'
|
||||
)"""
|
||||
)
|
||||
conn.execute(
|
||||
"""INSERT INTO agent_tool_invocations
|
||||
(id, research_run_id, provider, tool_name, tool_category, decision,
|
||||
decision_reason, input_sha256, output_sha256)
|
||||
VALUES
|
||||
(
|
||||
'tool-hash-only',
|
||||
'run-hash-only',
|
||||
'AgentCash StableSocial',
|
||||
'lightreel-trends',
|
||||
'social_trends',
|
||||
'executed',
|
||||
'Question asks for current Twitter/X discussion.',
|
||||
'sha256:input-only',
|
||||
'sha256:output-only'
|
||||
)"""
|
||||
)
|
||||
|
||||
row = conn.execute(
|
||||
"""SELECT
|
||||
r.prompt_excerpt,
|
||||
r.answer_excerpt,
|
||||
r.secret_values_included AS run_secret_flag,
|
||||
t.secret_values_included AS tool_secret_flag
|
||||
FROM agent_research_runs r
|
||||
JOIN agent_tool_invocations t ON t.research_run_id = r.id
|
||||
WHERE r.id = 'run-hash-only'"""
|
||||
).fetchone()
|
||||
|
||||
assert row["prompt_excerpt"] is None
|
||||
assert row["answer_excerpt"] is None
|
||||
assert row["run_secret_flag"] == 0
|
||||
assert row["tool_secret_flag"] == 0
|
||||
|
|
@ -1,20 +1,21 @@
|
|||
"""Tests for lib/search.py — vector search and graph expansion."""
|
||||
|
||||
import json
|
||||
from unittest.mock import MagicMock, patch
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from lib.search import (
|
||||
PASS1_THRESHOLD,
|
||||
WIKI_LINK_RE,
|
||||
_parse_frontmatter_edges,
|
||||
_resolve_claim_path,
|
||||
graph_expand,
|
||||
search,
|
||||
search_qdrant,
|
||||
WIKI_LINK_RE,
|
||||
)
|
||||
|
||||
|
||||
# ─── Fixtures ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
|
|
@ -512,19 +513,17 @@ class TestTwoPassRetrieval:
|
|||
@patch("lib.search.search_qdrant")
|
||||
@patch("lib.search.embed_query")
|
||||
def test_pass1_only_default(self, mock_embed, mock_qdrant, mock_expand):
|
||||
"""Default search (expand=False) only calls Qdrant once with the pass-1 threshold."""
|
||||
"""Default search (expand=False) only calls Qdrant once with high threshold."""
|
||||
mock_embed.return_value = [0.1] * 1536
|
||||
mock_qdrant.return_value = [
|
||||
{"score": 0.85, "payload": {"claim_title": "Hit", "claim_path": "d/a.md"}},
|
||||
]
|
||||
result = search("query")
|
||||
mock_qdrant.assert_called_once()
|
||||
# Should use the production pass-1 threshold.
|
||||
# Should use PASS1_THRESHOLD (0.70)
|
||||
call_kwargs = mock_qdrant.call_args
|
||||
assert (
|
||||
call_kwargs.kwargs.get("score_threshold") == PASS1_THRESHOLD
|
||||
or call_kwargs[1].get("score_threshold") == PASS1_THRESHOLD
|
||||
)
|
||||
assert call_kwargs.kwargs.get("score_threshold") == 0.70 \
|
||||
or call_kwargs[1].get("score_threshold") == 0.70
|
||||
mock_expand.assert_not_called()
|
||||
assert len(result["direct_results"]) == 1
|
||||
|
||||
|
|
|
|||
|
|
@ -1,110 +0,0 @@
|
|||
"""Tests for the Leo wallet-test Telegram runtime verifier."""
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
SCRIPT = REPO_ROOT / "scripts" / "check_telegram_leo_wallet_test_runtime.py"
|
||||
|
||||
|
||||
def run_checker(args: list[str]) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(
|
||||
[sys.executable, str(SCRIPT), *args],
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=False,
|
||||
)
|
||||
|
||||
|
||||
def test_missing_token_writes_sanitized_blocker(tmp_path):
|
||||
proof_path = tmp_path / "proof.json"
|
||||
proc = run_checker(
|
||||
[
|
||||
"--agent",
|
||||
"leo-wallet-test",
|
||||
"--repo-root",
|
||||
str(REPO_ROOT),
|
||||
"--secrets-dir",
|
||||
str(tmp_path / "secrets"),
|
||||
"--skip-getme",
|
||||
"--output",
|
||||
str(proof_path),
|
||||
]
|
||||
)
|
||||
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
proof = json.loads(proof_path.read_text())
|
||||
assert proof["ok"] is False
|
||||
assert proof["exactBlocker"] == "telegram_token_file_missing"
|
||||
assert proof["tokenFilePresent"] is False
|
||||
assert proof["secretValuesIncluded"] is False
|
||||
assert "secretValuesIncluded" in proc.stdout
|
||||
|
||||
|
||||
def test_invalid_token_shape_fails_without_printing_token(tmp_path):
|
||||
secrets_dir = tmp_path / "secrets"
|
||||
secrets_dir.mkdir()
|
||||
token_path = secrets_dir / "leo-test-telegram-bot-token"
|
||||
token = "not-a-valid-token"
|
||||
token_path.write_text(token)
|
||||
proof_path = tmp_path / "proof.json"
|
||||
proc = run_checker(
|
||||
[
|
||||
"--agent",
|
||||
"leo-wallet-test",
|
||||
"--repo-root",
|
||||
str(REPO_ROOT),
|
||||
"--secrets-dir",
|
||||
str(secrets_dir),
|
||||
"--skip-getme",
|
||||
"--require-token",
|
||||
"--output",
|
||||
str(proof_path),
|
||||
]
|
||||
)
|
||||
|
||||
assert proc.returncode == 1
|
||||
assert token not in proc.stdout
|
||||
assert token not in proc.stderr
|
||||
proof = json.loads(proof_path.read_text())
|
||||
assert proof["exactBlocker"] == "telegram_token_shape_invalid"
|
||||
assert proof["tokenFilePresent"] is True
|
||||
assert proof["tokenShapeValid"] is False
|
||||
assert proof["secretValuesIncluded"] is False
|
||||
|
||||
|
||||
def test_getme_result_is_sanitized_and_matches_expected_username():
|
||||
module_dir = str(SCRIPT.parent)
|
||||
sys.path.insert(0, module_dir)
|
||||
import check_telegram_leo_wallet_test_runtime as checker
|
||||
|
||||
token = "dummy-token-value"
|
||||
response_body = {
|
||||
"ok": True,
|
||||
"result": {
|
||||
"id": 123456789,
|
||||
"is_bot": True,
|
||||
"first_name": "Living IP Leo Wallet Test",
|
||||
"username": "lipleowallet0622183538bot",
|
||||
"can_join_groups": True,
|
||||
"can_read_all_group_messages": False,
|
||||
"supports_inline_queries": False,
|
||||
},
|
||||
}
|
||||
response = MagicMock()
|
||||
response.status = 200
|
||||
response.read.return_value = json.dumps(response_body).encode("utf-8")
|
||||
response.__enter__.return_value = response
|
||||
|
||||
with patch("urllib.request.urlopen", return_value=response):
|
||||
result = checker.telegram_get_me(token)
|
||||
|
||||
serialized = json.dumps(result)
|
||||
assert result["ok"] is True
|
||||
assert result["username"] == "lipleowallet0622183538bot"
|
||||
assert result["botIdPresent"] is True
|
||||
assert result["secretValuesIncluded"] is False
|
||||
assert token not in serialized
|
||||
|
|
@ -1,328 +0,0 @@
|
|||
"""Regression coverage for the Leo Telegram -> Living IP HTTP chat bridge."""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
TELEGRAM_DIR = REPO_ROOT / "telegram"
|
||||
sys.path.insert(0, str(TELEGRAM_DIR))
|
||||
|
||||
from agent_config import load_agent_config # noqa: E402
|
||||
from http_chat_proxy import ( # noqa: E402
|
||||
build_chat_proxy_payload,
|
||||
build_smart_research_proxy_payload,
|
||||
extract_auto_smart_research_followup_goal,
|
||||
extract_auto_smart_research_goal,
|
||||
extract_paid_work_order_id,
|
||||
extract_chat_proxy_reply,
|
||||
extract_smart_research_goal,
|
||||
should_attach_structured_market_context,
|
||||
smart_research_command_names,
|
||||
)
|
||||
from market_data import extract_market_data_tokens, format_price_context # noqa: E402
|
||||
|
||||
|
||||
def test_leo_config_opts_into_http_chat_proxy_without_changing_default_agents():
|
||||
leo = load_agent_config(str(TELEGRAM_DIR / "agents" / "leo.yaml"))
|
||||
leo_wallet_test = load_agent_config(str(TELEGRAM_DIR / "agents" / "leo-wallet-test.yaml"))
|
||||
rio = load_agent_config(str(TELEGRAM_DIR / "agents" / "rio.yaml"))
|
||||
|
||||
assert leo.name == "Leo"
|
||||
assert leo.http_chat_proxy_url == "https://leo.livingip.xyz/api/agents/leo/chat"
|
||||
assert leo.respond_to_private_chats is True
|
||||
assert "@teLEOhuman" in leo.mention_aliases
|
||||
assert leo_wallet_test.name == "Leo Wallet Test"
|
||||
assert leo_wallet_test.http_chat_proxy_url == "https://leo.livingip.xyz/api/agents/leo/chat"
|
||||
assert leo_wallet_test.http_research_proxy_url == "https://leo.livingip.xyz/api/agents/leo/research"
|
||||
assert "/smart_research" in leo_wallet_test.smart_research_command_prefixes
|
||||
assert leo_wallet_test.auto_smart_research_from_chat is True
|
||||
assert leo_wallet_test.respond_to_private_chats is True
|
||||
assert leo_wallet_test.bot_token_file == "leo-test-telegram-bot-token"
|
||||
assert "@lipleowallet0622183538bot" in leo_wallet_test.mention_aliases
|
||||
assert rio.http_chat_proxy_url is None
|
||||
assert rio.respond_to_private_chats is False
|
||||
assert leo.auto_smart_research_from_chat is False
|
||||
|
||||
|
||||
def test_invalid_http_chat_proxy_url_fails_closed(tmp_path):
|
||||
config = tmp_path / "bad.yaml"
|
||||
config.write_text(
|
||||
"""
|
||||
name: Leo
|
||||
handle: "@teLEOhuman"
|
||||
bot_token_file: leo-telegram-bot-token
|
||||
pentagon_agent_id: livingip-leo
|
||||
domain: collective-intelligence
|
||||
kb_scope:
|
||||
primary: ["core"]
|
||||
voice_summary: "test"
|
||||
voice_definition: "test"
|
||||
learnings_file: agents/leo/learnings.md
|
||||
http_chat_proxy_url: "not-a-url"
|
||||
""".strip()
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="http_chat_proxy_url"):
|
||||
load_agent_config(str(config))
|
||||
|
||||
|
||||
def test_proxy_payload_contains_no_secret_material():
|
||||
payload = build_chat_proxy_payload(
|
||||
message="Can Leo use x402 paid research now?",
|
||||
source="telegram",
|
||||
agent="leo",
|
||||
chat_id=123,
|
||||
message_id=456,
|
||||
username="tester",
|
||||
)
|
||||
|
||||
assert payload == {
|
||||
"message": "Can Leo use x402 paid research now?",
|
||||
"metadata": {
|
||||
"source": "telegram",
|
||||
"agent": "leo",
|
||||
"chat_id": 123,
|
||||
"message_id": 456,
|
||||
"username": "tester",
|
||||
},
|
||||
}
|
||||
assert "token" not in str(payload).lower()
|
||||
assert "secret" not in str(payload).lower()
|
||||
|
||||
|
||||
def test_smart_research_payload_is_no_spend_by_default():
|
||||
payload = build_smart_research_proxy_payload(
|
||||
research_goal="Find x402 evidence",
|
||||
source="telegram",
|
||||
agent="leo",
|
||||
chat_id=123,
|
||||
message_id=456,
|
||||
username="tester",
|
||||
max_amount_usd=0.01,
|
||||
)
|
||||
|
||||
assert payload["message"] == "Find x402 evidence"
|
||||
assert payload["research_goal"] == "Find x402 evidence"
|
||||
assert payload["allow_paid_execution"] is False
|
||||
assert payload["max_amount_usd"] == 0.01
|
||||
assert "approval_ref" not in payload
|
||||
assert "token" not in str(payload).lower()
|
||||
assert "secret" not in str(payload).lower()
|
||||
|
||||
|
||||
def test_smart_research_payload_can_resume_paid_work_order_without_secret_material():
|
||||
payload = build_smart_research_proxy_payload(
|
||||
research_goal="what are the current discussions about MetaDAO Ranger Finance on Twitter?",
|
||||
source="telegram",
|
||||
agent="leo",
|
||||
chat_id=123,
|
||||
message_id=456,
|
||||
username="tester",
|
||||
work_order_id="sponsored_work_orders:f951ccc6c7762ecba6f76cf6",
|
||||
original_research_goal="what are the current discussions about MetaDAO Ranger Finance on Twitter?",
|
||||
)
|
||||
|
||||
assert payload["work_order_id"] == "sponsored_work_orders:f951ccc6c7762ecba6f76cf6"
|
||||
assert (
|
||||
payload["original_research_goal"]
|
||||
== "what are the current discussions about MetaDAO Ranger Finance on Twitter?"
|
||||
)
|
||||
assert payload["allow_paid_execution"] is False
|
||||
assert "approval_ref" not in payload
|
||||
assert "token" not in str(payload).lower()
|
||||
assert "secret" not in str(payload).lower()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("message", "expected"),
|
||||
[
|
||||
("/smart_research find x402 evidence", "find x402 evidence"),
|
||||
("@lipleowallet0622183538bot /smart_research find x402 evidence", "find x402 evidence"),
|
||||
("/paid_research@lipleowallet0622183538bot find x402 evidence", "find x402 evidence"),
|
||||
("can Leo use x402 paid research now?", None),
|
||||
("/smart_research", None),
|
||||
],
|
||||
)
|
||||
def test_extract_smart_research_goal(message, expected):
|
||||
assert extract_smart_research_goal(message) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("message", "expected"),
|
||||
[
|
||||
(
|
||||
"work_order_id: sponsored_work_orders:f951ccc6c7762ecba6f76cf6",
|
||||
"sponsored_work_orders:f951ccc6c7762ecba6f76cf6",
|
||||
),
|
||||
(
|
||||
"paid receipt payment_receipts:f951ccc6c7762ecba6f76cf6 thanks",
|
||||
"payment_receipts:f951ccc6c7762ecba6f76cf6",
|
||||
),
|
||||
("no paid id here", None),
|
||||
],
|
||||
)
|
||||
def test_extract_paid_work_order_id(message, expected):
|
||||
assert extract_paid_work_order_id(message) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("message", "expected"),
|
||||
[
|
||||
(
|
||||
"@lipleowallet0622183538bot how much revenue does MetaDAO make today?",
|
||||
"how much revenue does MetaDAO make today?",
|
||||
),
|
||||
(
|
||||
"what is the volume and fdv of omnipair avici umbra? should i buy them yes or no",
|
||||
"what is the volume and fdv of omnipair avici umbra? should i buy them yes or no",
|
||||
),
|
||||
("Can you find current sources on x402 usage?", "Can you find current sources on x402 usage?"),
|
||||
(
|
||||
"what is the latest trend of internet finance on Twitter",
|
||||
"what is the latest trend of internet finance on Twitter",
|
||||
),
|
||||
(
|
||||
"what are your thoughts on how metadao managed the ranger finance situation",
|
||||
"what are your thoughts on how metadao managed the ranger finance situation",
|
||||
),
|
||||
(
|
||||
"who was at fault for Ranger according to Twitter?",
|
||||
"who was at fault for Ranger according to Twitter?",
|
||||
),
|
||||
(
|
||||
"what is MetaDAO's position on Ranger Finance?",
|
||||
"what is MetaDAO's position on Ranger Finance?",
|
||||
),
|
||||
(
|
||||
"how did Jupiter handle the outage situation?",
|
||||
"how did Jupiter handle the outage situation?",
|
||||
),
|
||||
(
|
||||
"assess whether the protocol valuation is fair versus fintech peers",
|
||||
"assess whether the protocol valuation is fair versus fintech peers",
|
||||
),
|
||||
(
|
||||
"compare growth metrics for these products against web2 companies",
|
||||
"compare growth metrics for these products against web2 companies",
|
||||
),
|
||||
("thanks, that makes sense", None),
|
||||
("what are your thoughts on lunch", None),
|
||||
("how did you sleep", None),
|
||||
("/paid_research find x402 evidence", None),
|
||||
],
|
||||
)
|
||||
def test_extract_auto_smart_research_goal(message, expected):
|
||||
assert extract_auto_smart_research_goal(
|
||||
message,
|
||||
mention_aliases=["@lipleowallet0622183538bot", "@leo"],
|
||||
) == expected
|
||||
|
||||
|
||||
def test_extract_auto_smart_research_followup_goal_uses_previous_market_question():
|
||||
previous = "what is the volume and fdv of omnipair avici umbra? should i buy them yes or no"
|
||||
goal = extract_auto_smart_research_followup_goal("check it yourself", previous)
|
||||
|
||||
assert previous in goal
|
||||
assert "Use current public sources" in goal
|
||||
assert "do not provide personalized financial advice" in goal
|
||||
|
||||
|
||||
def test_extract_auto_smart_research_followup_goal_ignores_context_without_research_intent():
|
||||
assert extract_auto_smart_research_followup_goal("check it yourself", "thanks, that makes sense") is None
|
||||
|
||||
|
||||
def test_market_data_token_extraction_maps_natural_market_question():
|
||||
message = "what is the volume and fdv of omnipair avici umbra? should i buy them yes or no"
|
||||
|
||||
assert extract_market_data_tokens(message) == ["OMFG", "AVICI", "UMBRA"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("message", "expected"),
|
||||
[
|
||||
("what is the volume and fdv of omnipair avici umbra? should i buy them yes or no", True),
|
||||
("show me price and liquidity for AVICI", True),
|
||||
("what are the current discussions about MetaDAO Ranger Finance on Twitter?", False),
|
||||
("who was at fault for Ranger according to Twitter?", False),
|
||||
("how much revenue does MetaDAO make today?", False),
|
||||
],
|
||||
)
|
||||
def test_should_attach_structured_market_context_only_for_market_data_intent(message, expected):
|
||||
assert should_attach_structured_market_context(message) is expected
|
||||
|
||||
|
||||
def test_market_data_context_formats_dexscreener_fdv_volume_liquidity():
|
||||
context = format_price_context(
|
||||
{
|
||||
"provider": "dexscreener",
|
||||
"result": (
|
||||
"Live market data for OMFG | source: DexScreener | price: $0.10 | "
|
||||
"FDV: $1.20M | 24h volume: $52.00K | liquidity: $101.00K"
|
||||
),
|
||||
},
|
||||
"OMFG",
|
||||
)
|
||||
|
||||
assert "FDV: $1.20M" in context
|
||||
assert "24h volume: $52.00K" in context
|
||||
|
||||
|
||||
def test_smart_research_command_names_are_safe_for_telegram_handlers():
|
||||
assert smart_research_command_names(
|
||||
[
|
||||
"/smart_research",
|
||||
"/paid_research",
|
||||
"/paid_research@lipleowallet0622183538bot",
|
||||
"not_a_command",
|
||||
"/bad-name",
|
||||
"/",
|
||||
]
|
||||
) == ["paid_research", "smart_research"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("payload", "expected"),
|
||||
[
|
||||
({"reply": "public route reply"}, "public route reply"),
|
||||
({"decision": {"reply": "analysis route reply"}}, "analysis route reply"),
|
||||
({"llm": {"decision": {"reply": "nested decision reply"}}}, "nested decision reply"),
|
||||
({"synthesis": {"decision": {"reply": "smart research reply"}}}, "smart research reply"),
|
||||
],
|
||||
)
|
||||
def test_extract_chat_proxy_reply_accepts_retained_leo_shapes(payload, expected):
|
||||
assert extract_chat_proxy_reply(payload) == expected
|
||||
|
||||
|
||||
def test_extract_chat_proxy_reply_fails_closed_on_missing_reply():
|
||||
assert extract_chat_proxy_reply({"schema": "livingip.x402.leoChatResponse.v1"}) is None
|
||||
|
||||
|
||||
def test_extract_chat_proxy_reply_never_displays_operator_claim_fields():
|
||||
payload = {
|
||||
"ok": False,
|
||||
"status": "payment_authorization_required",
|
||||
"strongestClaimAllowed": (
|
||||
"Leo smart research can select sponsor-agent-research on Devnet, "
|
||||
"but did not attempt payment because the call was not fully authorized."
|
||||
),
|
||||
"exactBlocker": "smart_research_paid_execution_not_allowed",
|
||||
"nextExactAction": "POST again with allow_paid_execution=true.",
|
||||
}
|
||||
|
||||
assert extract_chat_proxy_reply(payload) is None
|
||||
|
||||
|
||||
def test_extract_chat_proxy_reply_uses_clean_route_reply_for_tool_failures():
|
||||
payload = {
|
||||
"ok": False,
|
||||
"status": "payment_authorization_required",
|
||||
"reply": "I tried to use my paid research tool, but it was not available. No funds moved.",
|
||||
"strongestClaimAllowed": "Internal status text that must stay out of Telegram.",
|
||||
}
|
||||
|
||||
assert (
|
||||
extract_chat_proxy_reply(payload)
|
||||
== "I tried to use my paid research tool, but it was not available. No funds moved."
|
||||
)
|
||||
|
|
@ -1,40 +0,0 @@
|
|||
from pathlib import Path
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def test_teleo_agent_template_supports_optional_per_agent_env_file():
|
||||
unit = (REPO_ROOT / "systemd" / "teleo-agent@.service").read_text()
|
||||
|
||||
assert "Environment=PYTHONUNBUFFERED=1" in unit
|
||||
assert "EnvironmentFile=-/opt/teleo-eval/secrets/teleo-agent-%i.env" in unit
|
||||
|
||||
|
||||
def test_deploy_scripts_restart_wallet_test_agent_when_present():
|
||||
auto_deploy = (REPO_ROOT / "deploy" / "auto-deploy.sh").read_text()
|
||||
manual_deploy = (REPO_ROOT / "deploy" / "deploy.sh").read_text()
|
||||
|
||||
assert "add_restart_if_unit_exists teleo-agent@leo-wallet-test" in auto_deploy
|
||||
assert 'systemctl list-units --all --full "$1.service"' in auto_deploy
|
||||
assert "teleo-agent@leo-wallet-test" in manual_deploy
|
||||
|
||||
|
||||
def test_deploy_scripts_do_not_start_inactive_legacy_leo_agent():
|
||||
auto_deploy = (REPO_ROOT / "deploy" / "auto-deploy.sh").read_text()
|
||||
manual_deploy = (REPO_ROOT / "deploy" / "deploy.sh").read_text()
|
||||
|
||||
assert "add_restart_if_unit_active teleo-agent@leo" in auto_deploy
|
||||
assert "add_restart teleo-agent@leo" not in auto_deploy
|
||||
assert "systemctl is-active --quiet teleo-agent@leo.service" in manual_deploy
|
||||
assert "teleo-pipeline teleo-diagnostics teleo-agent@leo" not in manual_deploy
|
||||
|
||||
|
||||
def test_auto_deploy_prefers_github_remote_when_present():
|
||||
auto_deploy = (REPO_ROOT / "deploy" / "auto-deploy.sh").read_text()
|
||||
|
||||
assert 'DEPLOY_REMOTE="${TELEO_DEPLOY_REMOTE:-}"' in auto_deploy
|
||||
assert 'remote get-url github' in auto_deploy
|
||||
assert 'git fetch "$DEPLOY_REMOTE" main' in auto_deploy
|
||||
assert 'git rev-parse "$DEPLOY_REMOTE/main"' in auto_deploy
|
||||
assert 'git merge --ff-only "$DEPLOY_REMOTE/main"' in auto_deploy
|
||||
Loading…
Reference in a new issue