Auto: ops/self-directed-research.md | 1 file changed, 169 insertions(+)
This commit is contained in:
parent
d2beae7c2a
commit
7c1bfe8eef
1 changed files with 169 additions and 0 deletions
169
ops/self-directed-research.md
Normal file
169
ops/self-directed-research.md
Normal file
|
|
@ -0,0 +1,169 @@
|
|||
# Self-Directed Research Architecture
|
||||
|
||||
Draft — Leo, 2026-03-10
|
||||
|
||||
## Core Idea
|
||||
|
||||
Each agent gets a daily research session on the VPS. They autonomously pull tweets from their domain accounts, decide what's interesting, archive sources with notes, and push to inbox. A separate extraction cron (already running) picks up the archives and makes claims. The researcher never sees the extraction — preventing motivated reasoning.
|
||||
|
||||
## Why Separate Researcher and Extractor
|
||||
|
||||
When the same agent researches and extracts, they prime themselves. The researcher finds a tweet they think supports a thesis → writes notes emphasizing that angle → extracts a claim that confirms the thesis. The extraction becomes a formality.
|
||||
|
||||
Separation breaks this:
|
||||
- **Researcher** writes: "This tweet is about X, connects to Y, might challenge Z"
|
||||
- **Extractor** (different Claude instance, fresh context) reads the source and notes, extracts what's actually there
|
||||
- Neither has the other's context window or priming
|
||||
|
||||
This mirrors our proposer-evaluator separation for claims, applied one layer earlier in the pipeline.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Three cron stages on VPS
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
|
||||
│ Research Cron │────▶│ Extract Cron │────▶│ Eval Pipeline │
|
||||
│ (daily, 2hr) │ │ (every 5 min) │ │ (webhook.py) │
|
||||
│ │ │ │ │ │
|
||||
│ Pull tweets │ │ Read archives │ │ Review claims │
|
||||
│ Pick 1 task │ │ Extract claims │ │ Approve/reject │
|
||||
│ Archive sources │ │ Open PR │ │ Merge │
|
||||
│ Push to main │ │ │ │ │
|
||||
└─────────────────┘ └──────────────────┘ └─────────────────┘
|
||||
```
|
||||
|
||||
### Research Cron: `research-daily.sh`
|
||||
|
||||
**Schedule:** Once daily, staggered across agents to respect rate limits
|
||||
|
||||
```
|
||||
# Stagger: each agent gets a 30-min window
|
||||
0 2 * * * /opt/teleo-eval/research-daily.sh rio
|
||||
30 2 * * * /opt/teleo-eval/research-daily.sh clay
|
||||
0 3 * * * /opt/teleo-eval/research-daily.sh theseus
|
||||
30 3 * * * /opt/teleo-eval/research-daily.sh vida
|
||||
0 4 * * * /opt/teleo-eval/research-daily.sh astra
|
||||
30 4 * * * /opt/teleo-eval/research-daily.sh leo
|
||||
```
|
||||
|
||||
**Per agent, the research session:**
|
||||
|
||||
1. Pull latest tweets from agent's network accounts (X API)
|
||||
2. Read the agent's beliefs, recent claims, open positions
|
||||
3. Claude prompt: "You are {agent}. Here are your latest tweets from {accounts}. Here is your current knowledge state. Pick ONE research direction that advances your domain understanding. Archive the most relevant sources with notes."
|
||||
4. Agent writes source archives to `inbox/archive/` with `status: unprocessed`
|
||||
5. Commit and push to main (source-only, no claims)
|
||||
6. Extract cron picks them up within 5 minutes
|
||||
|
||||
**Key constraint:** One Claude session per agent, ~20-30 minutes, Sonnet model. Total daily VPS research compute: ~3 hours of sequential Sonnet sessions.
|
||||
|
||||
### Research Prompt Structure
|
||||
|
||||
```
|
||||
You are {agent}, a Teleo knowledge base agent specializing in {domain}.
|
||||
|
||||
## Your Current State
|
||||
{Read from agents/{agent}/beliefs.md, reasoning.md, positions/}
|
||||
|
||||
## Your Network
|
||||
{Read from network file — accounts to monitor}
|
||||
|
||||
## Recent Tweets
|
||||
{Raw tweet data pulled from X API}
|
||||
|
||||
## Your Task
|
||||
1. Scan these tweets for anything substantive — new claims, evidence,
|
||||
debates, data, counterarguments to existing KB positions
|
||||
2. Pick ONE research direction that would most advance your domain
|
||||
understanding right now. Consider:
|
||||
- Gaps in your beliefs that need evidence
|
||||
- Claims in the KB that might be wrong
|
||||
- Cross-domain connections you've been flagged about
|
||||
- New developments that change the landscape
|
||||
3. Archive the relevant sources (5-15 per session) following the
|
||||
inbox/archive format with full agent notes
|
||||
4. Write a brief research summary explaining what you found and why
|
||||
it matters
|
||||
|
||||
## Rules
|
||||
- Archive EVERYTHING substantive, not just what supports your views
|
||||
- Write honest agent notes — flag what challenges your beliefs too
|
||||
- Set all sources to status: unprocessed (a different instance extracts)
|
||||
- Flag cross-domain sources for other agents
|
||||
- Do NOT extract claims yourself — that's a separate process
|
||||
```
|
||||
|
||||
### Capacity on Claude Max ($200/month)
|
||||
|
||||
**VPS compute budget (all Sonnet):**
|
||||
- Research cron: 6 agents × 30 min/day = 3 hr/day
|
||||
- Extract cron: ~37 sources × 10 min = 6 hr one-time backlog, then ~1 hr/day steady-state
|
||||
- Eval pipeline: ~10 PRs/day × 15 min = 2.5 hr/day
|
||||
- **Total VPS:** ~6.5 hr/day Sonnet (steady state)
|
||||
|
||||
**Laptop compute budget (Opus + Sonnet mix):**
|
||||
- Agent sessions: 2-3 concurrent, ~4-6 hr/day
|
||||
- Leo coordination: ~1-2 hr/day
|
||||
|
||||
**Single subscription feasibility:** Tight but workable if:
|
||||
- VPS runs overnight (2am-8am staggered research + continuous extraction)
|
||||
- Laptop agents run during the day
|
||||
- Never more than 2-3 concurrent sessions total
|
||||
- VPS uses Sonnet exclusively (cheaper rate limits)
|
||||
|
||||
**Risk:** If rate limits tighten or daily message caps exist, the VPS research cron may not complete all 6 agents. Mitigation: priority ordering (run the 3 most active agents daily, others every 2-3 days).
|
||||
|
||||
## Contributor Workflow Options
|
||||
|
||||
Different people want different levels of involvement:
|
||||
|
||||
### Mode 1: Full Researcher
|
||||
"I found this, here's why it matters, here are the KB connections"
|
||||
- Uses /ingest on laptop (Track A or B)
|
||||
- Writes detailed agent notes
|
||||
- May extract claims themselves
|
||||
- Highest quality input
|
||||
|
||||
### Mode 2: Curator
|
||||
"Here's a source, it's about X domain"
|
||||
- Minimal archive file with domain tag and brief notes
|
||||
- VPS extracts (Track B)
|
||||
- Good enough for most sources
|
||||
|
||||
### Mode 3: Raw Dump
|
||||
"Here are tweets, figure it out"
|
||||
- Dumps raw JSON to VPS inbox-raw/
|
||||
- Leo triages: decides domain, writes archive files
|
||||
- VPS extracts from Leo's archives
|
||||
- Lowest effort, decent quality (Leo's triage catches the important stuff)
|
||||
|
||||
### Mode 4: Self-Directed Agent (VPS)
|
||||
"Agent, go research your domain"
|
||||
- No human involvement beyond initial network setup
|
||||
- Daily cron pulls tweets, agent picks direction, archives, extraction follows
|
||||
- Quality depends on prompt engineering + eval pipeline catching errors
|
||||
|
||||
All four modes feed into the same extraction → eval pipeline. Quality varies, but the eval pipeline is the quality gate regardless.
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Rate limits**: What are the actual Claude Max per-minute and per-day limits for headless Sonnet sessions? Need empirical data from this first extraction run.
|
||||
|
||||
2. **Research quality**: Will a 30-minute Sonnet session produce good enough research notes? Or does research require Opus-level reasoning?
|
||||
|
||||
3. **Network bootstrapping**: Agents need network files. Who curates the initial account lists? (Currently Cory + Leo, eventually agents propose additions)
|
||||
|
||||
4. **Cross-domain routing**: When the research cron finds cross-domain content, should it archive under the researcher's domain or the correct domain? (Probably correct domain with flagged_for_{researcher})
|
||||
|
||||
5. **Feedback loop**: How does extraction quality feed back to improve research notes? If the extractor consistently ignores certain types of notes, the researcher should learn.
|
||||
|
||||
6. **Deduplication across agents**: Multiple agents may archive the same tweet (e.g., a Karpathy tweet relevant to both AI systems and collective intelligence). The extract cron needs to detect this.
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. ✅ Extract cron (running now — validating extraction quality)
|
||||
2. **Next**: Research cron — daily self-directed sessions per agent
|
||||
3. **Then**: Raw dump path — Leo triage from JSON → archive
|
||||
4. **Later**: Full end-to-end with X API pull integrated into research cron
|
||||
5. **Eventually**: Feedback loops from eval quality → research prompt tuning
|
||||
Loading…
Reference in a new issue