diff --git a/docs/bootstrap/agent-learnings.md b/docs/bootstrap/agent-learnings.md deleted file mode 100644 index 776d40b2..00000000 --- a/docs/bootstrap/agent-learnings.md +++ /dev/null @@ -1,114 +0,0 @@ -# Agent Learnings — Bootstrap for New Operators - -This document distills operational knowledge from the first 2 weeks of running the Teleo agent collective. It's written for someone bootstrapping their own agents against this codebase. - ---- - -## Architecture Overview - -Six domain agents + one evaluator + one pipeline agent + one infrastructure agent: - -| Agent | Domain | Role | -|-------|--------|------| -| **Leo** | Grand strategy / cross-domain | Evaluator — reviews all PRs, synthesizes cross-domain | -| **Rio** | Internet finance | Proposer — extracts and proposes claims | -| **Clay** | Entertainment / cultural dynamics | Proposer | -| **Theseus** | AI / alignment | Proposer | -| **Vida** | Health & human flourishing | Proposer | -| **Astra** | Space development | Proposer | -| **Epimetheus** | Pipeline infrastructure | Pipeline agent — owns extraction, validation, eval, merge | -| **Ganymede** | Systems architecture | Adversarial reviewer for infrastructure changes | - -Agents communicate via Pentagon inboxes (JSON messages). All changes to the knowledge base go through PR review on Forgejo. - -## The Pipeline (what actually runs) - -``` -Source → Ingest → Extract (Sonnet 4.5 + Haiku review) → PR on Forgejo - → Tier 0.5 validation (deterministic, $0) - → Domain eval (Gemini 2.5 Flash via OpenRouter) - → Leo eval (Sonnet via OpenRouter for STANDARD, Opus for DEEP) - → Auto-fix (Haiku for mechanical issues) - → Merge (requires 2 formal approvals) -``` - -**Key numbers:** -- ~411 claims across 14 knowledge domains -- 500+ PRs processed -- Approval rate: started at 7%, now ~36% after quality-guide improvements and auto-fix -- Auto-fix success rate: 87% -- Cost: ~$0.02/review for domain eval, Claude Max flat rate for Opus - -### What works - -1. **Tier 0.5 deterministic gate** — catches 60%+ of mechanical failures (broken wiki links, frontmatter schema, near-duplicates) at $0 before any LLM eval. This was the single biggest ROI improvement. - -2. **Dual extraction** — claims + entities from the same source in the same LLM session. Entity extraction is where most of the structured data comes from. - -3. **Separated proposer/evaluator roles** — agents that extract claims don't evaluate their own claims. Different model families for extraction (Sonnet/Haiku) vs evaluation (GPT-4o/Opus) eliminate correlated blind spots. - -4. **Domain-serialized merge** — merges happen one domain at a time to prevent `_map.md` file conflicts. - -5. **SHA-based idempotency** — validation results are tagged with the commit SHA. Force-pushes trigger re-validation automatically. - -### What broke (lessons learned) - -1. **100+ claims/12h is too many.** When extraction ran without a novelty gate, it produced massive volume of incremental claims that overwhelmed review. Fix: extraction budget (3-5 claims/source), novelty gate (check existing KB before extracting), challenge premium (weight toward claims that contradict existing KB). - -2. **0 claims/10 sources is too few.** When the novelty gate was too aggressive, it treated "same topic" as "same claim" and extracted nothing. Fix: calibrate — new data points on existing topics = enrichment (strengthen/extend existing claim), new arguments = new claims. - -3. **Force-push invalidates Forgejo approvals.** Branch protection requires 2 approvals. Rebase → force-push → approvals gone → merge API returns 405. Fix: `_resubmit_approvals()` — programmatically re-submit 2 formal APPROVED reviews from agent tokens after rebase. - -4. **Root ownership on worker files.** Root crontab ran extraction scripts, creating root-owned files in shared workspaces. Fix: move ALL pipeline crons to the `teleo` service account. - -5. **ARG_MAX on large prompts.** Passing prompts as CLI arguments exceeds 2MB limit. Fix: pipe via stdin (`< "$prompt_file"`) instead. - -6. **Entity files cause merge conflicts.** Entities like `futardio.md` and `metadao.md` get modified by many PRs simultaneously. These are the real 405 blocker, not approvals. Fix: consolidation pattern — create clean branch from main, apply all enrichments via API, merge single consolidation PR, close originals. - -7. **"Dispatching workers" ≠ "healthy pipeline."** We declared the pipeline healthy while ALL workers were silently failing with ARG_MAX for 2 hours. Fix: log worker exit codes and outcomes, not just dispatch counts. - -## VPS Infrastructure - -- **Hetzner CAX31** at `77.42.65.182` — Ubuntu 24.04 ARM64, 16GB RAM -- **Four accounts:** root, teleo (service account for pipeline), cory, ben -- **Forgejo** at `git.livingip.xyz`, org: `teleo`, repo: `teleo-codex` -- **Pipeline location:** `/opt/teleo-eval/pipeline/` (Python async daemon) -- **Agent tokens:** `/opt/teleo-eval/secrets/forgejo-{agent}-token` -- **Bidirectional mirror:** `sync-mirror.sh` (every 2 min) syncs Forgejo ↔ GitHub. Forgejo is authoritative. - -### Bare Repo Architecture - -``` -/opt/teleo-eval/workspaces/teleo-codex.git ← bare repo (fetch cron updates every 2 min) -/opt/teleo-eval/workspaces/main ← persistent main worktree -``` - -Single-writer principle: only the fetch cron writes to the bare repo. Workers create disposable worktrees with `--detach`. Recovery = kill workers + rm -rf + re-clone bare + re-create main worktree (~30 seconds). - -## Model Strategy - -| Task | Model | Cost | -|------|-------|------| -| Research | Opus (Claude Max flat rate) | $0 marginal | -| Extraction pass 1 | Sonnet 4.5 (OpenRouter) | ~$0.05/source | -| Extraction pass 2 (review) | Haiku 4.5 (OpenRouter) | ~$0.01/source | -| Domain evaluation | Gemini 2.5 Flash (OpenRouter) | ~$0.02/review | -| Leo STANDARD review | Sonnet (OpenRouter) | ~$0.02/review | -| Leo DEEP review | Opus (Claude Max) | $0 marginal | -| Auto-fix | Haiku (default), Sonnet (escalation) | ~$0.01/fix | - -Two model families (Anthropic + Google) for evaluation prevents correlated blind spots — the same training bias won't produce the same false positives. - -## Key Design Decisions - -1. **PRs for everything.** Even during bootstrap. The PR history IS the audit trail. No direct commits to main. - -2. **Git trailers for agent attribution.** `Pentagon-Agent: Rio ` in every commit. Survives platform migration (unlike GitHub-specific metadata). - -3. **Claims are prose propositions, not labels.** "futarchy is manipulation-resistant because attack attempts create profitable opportunities for defenders" — not "futarchy manipulation resistance." The title IS the claim. - -4. **Confidence is calibrated.** `proven` requires strong evidence + survived challenges. `speculative` is honest about limited evidence. Miscalibrating confidence is a review failure. - -5. **Wiki links as graph edges.** `[[claim-title]]` links carry semantic weight. The link graph IS the knowledge structure. - -6. **Enrichment > new claims.** When a source adds evidence to an existing claim, enrich that claim rather than creating a near-duplicate. Near-duplicates are the #1 quality problem.