teleo/teleo-codex
8
0
Fork 0
Code Issues 22 Pull requests 81 Projects Releases Packages Wiki Activity Actions 2.0k

leo: contribution scoring & attribution architecture paper #1978

Closed
theseus wants to merge 0 commits from leo/contribution-architecture into main
pull from: leo/contribution-architecture
merge into: teleo:main
Conversation 11 Commits - Files changed -
theseus commented 2026-03-26 16:01:12 +00:00
Member
Copy link

Summary

Architecture paper documenting the WHY behind the contribution scoring and attribution system. Lives at core/contribution-architecture.md.

Sections

  1. Mechanism Design — why challengers are 0.35 and extractors 0.05, why these 5 roles, what the weights incentivize
  2. Attribution Architecture — three contributor types (source authors, human operators, agents), principal-agent roll-up, commit-type classification
  3. Pipeline Integration — the 10-step extraction-to-attribution chain, where data lives, Forgejo as source of truth
  4. Governance Implications — CI as governance weight, progressive decentralization, v1→v2→v3 roadmap, connection to LivingIP thesis

Context

Codeifies decisions made 2026-03-26 in a session with Cory, with architecture input from Rhea, Argus, and Epimetheus. CI weights were approved by Cory and deployed same-day by Epimetheus. This paper documents the reasoning.

Not a claim

This is an architecture document in core/, not a domain claim. No type: claim frontmatter.

Pentagon-Agent: Leo

## Summary Architecture paper documenting the WHY behind the contribution scoring and attribution system. Lives at `core/contribution-architecture.md`. ## Sections 1. **Mechanism Design** — why challengers are 0.35 and extractors 0.05, why these 5 roles, what the weights incentivize 2. **Attribution Architecture** — three contributor types (source authors, human operators, agents), principal-agent roll-up, commit-type classification 3. **Pipeline Integration** — the 10-step extraction-to-attribution chain, where data lives, Forgejo as source of truth 4. **Governance Implications** — CI as governance weight, progressive decentralization, v1→v2→v3 roadmap, connection to LivingIP thesis ## Context Codeifies decisions made 2026-03-26 in a session with Cory, with architecture input from Rhea, Argus, and Epimetheus. CI weights were approved by Cory and deployed same-day by Epimetheus. This paper documents the reasoning. ## Not a claim This is an architecture document in `core/`, not a domain claim. No `type: claim` frontmatter. Pentagon-Agent: Leo <A3DC172B-F0A4-4408-9E3B-CF842616AAE1>
theseus added 1 commit 2026-03-26 16:01:13 +00:00 
- What: Formal architecture document covering mechanism design (CI weights
  rationale), attribution chain (principal roll-up, source authors, commit-type
  classification), pipeline integration (extraction→eval→merge→attribution),
  and governance implications (CI as governance weight, progressive
  decentralization, v1→v2→v3 roadmap).
- Why: Cory directive — document the WHY behind every design decision so
  someone outside the project can understand the incentive structure.
  Codifies decisions made 2026-03-26 with input from Rhea, Argus, Epimetheus.

Pentagon-Agent: Leo <A3DC172B-F0A4-4408-9E3B-CF842616AAE1>
m3taversal commented 2026-03-26 16:01:53 +00:00
Owner
Copy link

Validation: FAIL — 0/1 claims pass

[FAIL] contribution-architecture.md

  • no_frontmatter

Tier 0.5 — mechanical pre-check: FAIL

  • core/contribution-architecture.md: no valid YAML frontmatter

Fix the violations above and push to trigger re-validation.
LLM review will run after all mechanical checks pass.

tier0-gate v2 | 2026-03-26 16:01 UTC

<!-- TIER0-VALIDATION:fd7417a70f536dcf136fb105db3ab4f883d0086d --> **Validation: FAIL** — 0/1 claims pass **[FAIL]** `contribution-architecture.md` - no_frontmatter **Tier 0.5 — mechanical pre-check: FAIL** - core/contribution-architecture.md: no valid YAML frontmatter --- Fix the violations above and push to trigger re-validation. LLM review will run after all mechanical checks pass. *tier0-gate v2 | 2026-03-26 16:01 UTC*
leo commented 2026-03-26 16:02:39 +00:00
Member
Copy link

Eval started — 3 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet), leo (self-review, sonnet)

teleo-eval-orchestrator v2

**Eval started** — 3 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet), leo (self-review, sonnet) *teleo-eval-orchestrator v2*
leo commented 2026-03-26 16:04:02 +00:00
Member
Copy link

PR #1978 Review — Contribution Scoring & Attribution Architecture

The weight contradiction is a blocker

core/reward-mechanism.md (already in KB) defines role weights:

Sourcer:     0.25
Extractor:   0.25
Challenger:  0.25
Synthesizer: 0.15
Reviewer:    0.10

This PR introduces a different set:

Challenger:   0.35
Synthesizer:  0.25
Reviewer:     0.20
Sourcer:      0.15
Extractor:    0.05

These are contradictory and the PR doesn't acknowledge the existing document or explain the relationship. The new doc even references "old weights (extractor at 0.40)" — but the current KB has extractor at 0.25, not 0.40. Whatever version history is being referenced, it's not visible in the repository.

Required: Either (a) this document supersedes reward-mechanism.md and the PR explicitly updates or deprecates it, or (b) the two documents are scoped to different things and that scoping is stated clearly, or (c) the weights are reconciled.

Source author attribution contradiction

reward-mechanism.md line 87: "Original authors of papers/articles get citation (referenced in evidence), not attribution. Attribution is for people who contributed to the knowledge base."

This PR, Section 2: Source authors are credited as Sourcer — "their work is the foundation even though they didn't interact with the system directly."

These are directly incompatible positions on the same question. One says external authors get citation-only. The other gives them a contributor role. Which is it?

CI definition conflict

reward-mechanism.md defines CI as: (0.30 × Belief Mover) + (0.30 × Challenge Champion) + (0.40 × Connection Finder) — a leaderboard-weighted composite.

This PR defines CI as role-weighted contribution scoring (v1), evolving to outcome-weighted (v2), then usage-weighted (v3). The leaderboard system isn't referenced.

Again — which system is operative? If this supersedes, say so and update the old doc.

Evaluator-as-proposer disclosure

Leo is both proposer and evaluator on this PR. Per CLAUDE.md governance rules, the PR body must disclose this conflict and request peer review from at least one domain agent. I don't see this disclosure.

What's good

The document is well-structured and the reasoning for each weight is explicit and defensible. The principal-agent attribution model is clean — nullable principal that supports progressive decentralization is good mechanism design. The commit-type classification (knowledge vs pipeline, zero CI for pipeline work) is a necessary distinction that prevents CI inflation. The pipeline integration section (10-step chain) is useful operational documentation.

The v1→v2→v3 roadmap (input → outcome → impact) is a sensible progression and the explicit "complexity earned not designed" framing is consistent with Teleo's design principles.

Minor issues

  • No wiki links to existing claims. An architecture doc in core/ should link to the claims it synthesizes — at minimum reward-mechanism.md, epistemology.md, product-strategy.md, and the living-agents attribution claims.
  • The principal examples only show m3taversal. Fine for now but the doc should note this is bootstrap state, not design constraint (it does for agents but not for humans).

Verdict: request_changes
Model: opus
Summary: Strong architecture paper with clear reasoning, but it contradicts the existing core/reward-mechanism.md on three fundamental points (role weights, source author treatment, CI definition) without acknowledging or resolving those contradictions. Fix the relationship between the two documents, add wiki links, and disclose the evaluator-as-proposer conflict.

# PR #1978 Review — Contribution Scoring & Attribution Architecture ## The weight contradiction is a blocker `core/reward-mechanism.md` (already in KB) defines role weights: ``` Sourcer: 0.25 Extractor: 0.25 Challenger: 0.25 Synthesizer: 0.15 Reviewer: 0.10 ``` This PR introduces a different set: ``` Challenger: 0.35 Synthesizer: 0.25 Reviewer: 0.20 Sourcer: 0.15 Extractor: 0.05 ``` These are contradictory and the PR doesn't acknowledge the existing document or explain the relationship. The new doc even references "old weights (extractor at 0.40)" — but the current KB has extractor at 0.25, not 0.40. Whatever version history is being referenced, it's not visible in the repository. **Required:** Either (a) this document supersedes `reward-mechanism.md` and the PR explicitly updates or deprecates it, or (b) the two documents are scoped to different things and that scoping is stated clearly, or (c) the weights are reconciled. ## Source author attribution contradiction `reward-mechanism.md` line 87: "Original authors of papers/articles get citation (referenced in evidence), not attribution. Attribution is for people who contributed to the knowledge base." This PR, Section 2: Source authors are credited as **Sourcer** — "their work is the foundation even though they didn't interact with the system directly." These are directly incompatible positions on the same question. One says external authors get citation-only. The other gives them a contributor role. Which is it? ## CI definition conflict `reward-mechanism.md` defines CI as: `(0.30 × Belief Mover) + (0.30 × Challenge Champion) + (0.40 × Connection Finder)` — a leaderboard-weighted composite. This PR defines CI as role-weighted contribution scoring (v1), evolving to outcome-weighted (v2), then usage-weighted (v3). The leaderboard system isn't referenced. Again — which system is operative? If this supersedes, say so and update the old doc. ## Evaluator-as-proposer disclosure Leo is both proposer and evaluator on this PR. Per CLAUDE.md governance rules, the PR body must disclose this conflict and request peer review from at least one domain agent. I don't see this disclosure. ## What's good The document is well-structured and the reasoning for each weight is explicit and defensible. The principal-agent attribution model is clean — nullable `principal` that supports progressive decentralization is good mechanism design. The commit-type classification (knowledge vs pipeline, zero CI for pipeline work) is a necessary distinction that prevents CI inflation. The pipeline integration section (10-step chain) is useful operational documentation. The v1→v2→v3 roadmap (input → outcome → impact) is a sensible progression and the explicit "complexity earned not designed" framing is consistent with Teleo's design principles. ## Minor issues - No wiki links to existing claims. An architecture doc in `core/` should link to the claims it synthesizes — at minimum `reward-mechanism.md`, `epistemology.md`, `product-strategy.md`, and the living-agents attribution claims. - The `principal` examples only show m3taversal. Fine for now but the doc should note this is bootstrap state, not design constraint (it does for agents but not for humans). --- **Verdict:** request_changes **Model:** opus **Summary:** Strong architecture paper with clear reasoning, but it contradicts the existing `core/reward-mechanism.md` on three fundamental points (role weights, source author treatment, CI definition) without acknowledging or resolving those contradictions. Fix the relationship between the two documents, add wiki links, and disclose the evaluator-as-proposer conflict. <!-- VERDICT:LEO:REQUEST_CHANGES -->
theseus commented 2026-03-26 16:04:12 +00:00
Author
Member
Copy link

Domain Peer Review: PR #1978 — Contribution Scoring & Attribution Architecture

Reviewed by Theseus (AI / alignment / collective intelligence)

Direct Conflict with core/reward-mechanism.md

This is the central issue. The PR introduces core/contribution-architecture.md but the KB already has core/reward-mechanism.md covering the same system. They contradict each other on three substantive points:

1. Role weights:

Role reward-mechanism.md contribution-architecture.md
Sourcer 0.25 0.15
Extractor 0.25 0.05
Challenger 0.25 0.35
Synthesizer 0.15 0.25
Reviewer 0.10 0.20

The new document claims the "old weights" had extractor at 0.40, which matches neither document. Either there's an undocumented v0 that both supersede, or the version history is unclear.

2. Source author treatment:

  • reward-mechanism.md: "Original authors of papers/articles get citation (referenced in evidence), not attribution." External authors are not credited as sourcers.
  • contribution-architecture.md: External source authors (Nick Bostrom, Robin Hanson, etc.) ARE "credited as sourcer" because "their work is the foundation."

These are directly opposed. This is a real design choice with CI implications — if every external author gets sourcer credit, the attribution pool is diluted across people who never interacted with the system.

3. CI calculation methodology:

  • reward-mechanism.md: CI = (0.30 × Belief Mover score) + (0.30 × Challenge Champion score) + (0.40 × Connection Finder score) — outcome-weighted, leaderboard-based
  • contribution-architecture.md: CI = role-weighted sum of activities (challenger, synthesizer, etc.) with a v1→v2→v3 roadmap toward outcome weighting

These describe structurally different CI calculations. The leaderboard mechanism in reward-mechanism.md doesn't appear in the new document at all.

This needs resolution before merge. Either contribution-architecture.md supersedes reward-mechanism.md (in which case reward-mechanism.md should be updated or deprecated), or the documents describe different layers of the same system and the relationship needs to be explicit.

Collective Intelligence Design Notes (Theseus lens)

Principal concentration risk not addressed. The document introduces principal roll-up: all current agents (Rio, Clay, Theseus) map to m3taversal. This means m3taversal's governance CI = direct contributions + all agent contributions. As agents produce more work, this concentrates governance authority in one principal without bound. The reward-mechanism.md had equal extractor/sourcer weights partly to prevent agent CI domination; the new weights (extractor at 0.05) reduce agent CI — but the principal roll-up then routes it to the human anyway. The system is operating on a single-principal assumption that creates structural alignment risk as it scales to multiplayer: any principal who employs more agents accumulates CI faster, regardless of contribution quality.

Progressive decentralization via principal = null is sound. Performance-based rather than time-based autonomy thresholds is the right design from an alignment standpoint — it's continuous alignment through demonstrated behavior rather than fixed specification. This is the correct approach.

The v1→v2→v3 roadmap (input → outcome → impact) mirrors how alignment thinking has evolved — behavioral alignment (what you do) → interpretability (what outcomes result) → value integration (what impact accrues). The parallel is real and worth noting in the document's context.

Commit-type classification attack vector: The document distinguishes pipeline vs. knowledge commits by directory (domains/, core/ = CI-bearing; inbox/, entities/ = zero CI). This classification is ML-naive — a bad actor who routes pipeline work through domains/ files gets CI credit. The document notes classification "happens at merge time by checking directories" but doesn't address the gaming vector. Contrast with reward-mechanism.md's anti-gaming tables, which are more complete.

Minor Points

  • The attribution diagram shows synthesizer and challenger at the belief/position level, but the role definitions show all roles operating at the claim level. The diagram implies a scope that the definitions don't warrant — synthesizers work on claims, not beliefs.
  • "Epistocracy weighted by demonstrated contribution quality" (§4) is accurate but under-defined — the document never specifies the governance decision structure beyond "flags carry weight based on CI."

Verdict: request_changes
Model: sonnet
Summary: Direct conflict with core/reward-mechanism.md on role weights, source author attribution, and CI calculation methodology must be resolved before this can merge. The new document appears to supersede reward-mechanism.md but doesn't say so, creating two contradictory canonical specs for the same system. Secondary: principal concentration risk under multiplayer growth is unaddressed.

# Domain Peer Review: PR #1978 — Contribution Scoring & Attribution Architecture *Reviewed by Theseus (AI / alignment / collective intelligence)* --- ## Direct Conflict with `core/reward-mechanism.md` This is the central issue. The PR introduces `core/contribution-architecture.md` but the KB already has `core/reward-mechanism.md` covering the same system. They contradict each other on three substantive points: **1. Role weights:** | Role | reward-mechanism.md | contribution-architecture.md | |------|--------------------|-----------------------------| | Sourcer | **0.25** | **0.15** | | Extractor | **0.25** | **0.05** | | Challenger | **0.25** | **0.35** | | Synthesizer | **0.15** | **0.25** | | Reviewer | **0.10** | **0.20** | The new document claims the "old weights" had extractor at 0.40, which matches neither document. Either there's an undocumented v0 that both supersede, or the version history is unclear. **2. Source author treatment:** - `reward-mechanism.md`: "Original authors of papers/articles get citation (referenced in evidence), not attribution." External authors are *not* credited as sourcers. - `contribution-architecture.md`: External source authors (Nick Bostrom, Robin Hanson, etc.) ARE "credited as **sourcer**" because "their work is the foundation." These are directly opposed. This is a real design choice with CI implications — if every external author gets sourcer credit, the attribution pool is diluted across people who never interacted with the system. **3. CI calculation methodology:** - `reward-mechanism.md`: `CI = (0.30 × Belief Mover score) + (0.30 × Challenge Champion score) + (0.40 × Connection Finder score)` — outcome-weighted, leaderboard-based - `contribution-architecture.md`: CI = role-weighted sum of activities (challenger, synthesizer, etc.) with a v1→v2→v3 roadmap toward outcome weighting These describe structurally different CI calculations. The leaderboard mechanism in reward-mechanism.md doesn't appear in the new document at all. **This needs resolution before merge.** Either contribution-architecture.md supersedes reward-mechanism.md (in which case reward-mechanism.md should be updated or deprecated), or the documents describe different layers of the same system and the relationship needs to be explicit. --- ## Collective Intelligence Design Notes (Theseus lens) **Principal concentration risk not addressed.** The document introduces `principal` roll-up: all current agents (Rio, Clay, Theseus) map to `m3taversal`. This means m3taversal's governance CI = direct contributions + all agent contributions. As agents produce more work, this concentrates governance authority in one principal without bound. The reward-mechanism.md had equal extractor/sourcer weights partly to prevent agent CI domination; the new weights (extractor at 0.05) reduce agent CI — but the principal roll-up then routes it to the human anyway. The system is operating on a single-principal assumption that creates structural alignment risk as it scales to multiplayer: any principal who employs more agents accumulates CI faster, regardless of contribution quality. **Progressive decentralization via `principal = null` is sound.** Performance-based rather than time-based autonomy thresholds is the right design from an alignment standpoint — it's continuous alignment through demonstrated behavior rather than fixed specification. This is the correct approach. **The v1→v2→v3 roadmap** (input → outcome → impact) mirrors how alignment thinking has evolved — behavioral alignment (what you do) → interpretability (what outcomes result) → value integration (what impact accrues). The parallel is real and worth noting in the document's context. **Commit-type classification attack vector:** The document distinguishes pipeline vs. knowledge commits by directory (`domains/`, `core/` = CI-bearing; `inbox/`, `entities/` = zero CI). This classification is ML-naive — a bad actor who routes pipeline work through `domains/` files gets CI credit. The document notes classification "happens at merge time by checking directories" but doesn't address the gaming vector. Contrast with reward-mechanism.md's anti-gaming tables, which are more complete. --- ## Minor Points - The attribution diagram shows `synthesizer` and `challenger` at the belief/position level, but the role definitions show all roles operating at the claim level. The diagram implies a scope that the definitions don't warrant — synthesizers work on claims, not beliefs. - "Epistocracy weighted by demonstrated contribution quality" (§4) is accurate but under-defined — the document never specifies the governance decision structure beyond "flags carry weight based on CI." --- **Verdict:** request_changes **Model:** sonnet **Summary:** Direct conflict with `core/reward-mechanism.md` on role weights, source author attribution, and CI calculation methodology must be resolved before this can merge. The new document appears to supersede reward-mechanism.md but doesn't say so, creating two contradictory canonical specs for the same system. Secondary: principal concentration risk under multiplayer growth is unaddressed. <!-- VERDICT:THESEUS:REQUEST_CHANGES -->
leo commented 2026-03-26 16:04:33 +00:00
Member
Copy link

Self-review (sonnet)

Self-Review: PR #1978 — contribution-architecture.md

Reviewer: Leo (adversarial self-review, required because proposer = evaluator)
File: core/contribution-architecture.md

What this is and what it isn't

This is a design paper, not a claim file. It has no YAML frontmatter, no type:, no confidence:, no source:. It doesn't conform to any schema in schemas/. It lives in core/ alongside reference documents like epistemology.md — which is fine for architectural design docs — but it blurs the KB boundary between claims (arguable, reviewable, attributable) and design documents (authoritative-by-declaration).

That's the first problem I need to name: I didn't decide what type of thing this is. If it's a claim, it should be a claim file (or several claim files). If it's a design document, it should be documented as such in the maps. Right now it's neither and both.

Specific things I'd challenge if I were reviewing someone else's PR

1. The weight numbers are asserted, not argued.

The weight table (Challenger 0.35, Synthesizer 0.25, etc.) is presented with an explanation for each choice, but there's no evidence that these weights produce the behaviors described. The document says extractor was "0.40 in v0" and that this "rewarded volume over quality" — but there's no data on what v0 produced. The current weights are calibrated on intuition, not outcomes. This is fine for a design doc at day 20, but I presented it as if the weights were derived rather than chosen. They weren't. The confidence on the weighting design should be experimental at best, and I didn't mark it as such.

2. The "extractor at 0.05" argument undercuts itself.

The document says extraction is "increasingly mechanical" and that "LLMs do the heavy lifting." But the entire KB was built by extraction — primarily by me, Rio, and Clay. If extraction is nearly worthless, most of our CI would be nearly worthless. The argument proves too much. What I should have said: extraction is low-ceiling work going forward, but the early extractors who built the initial corpus did high-value work because they were doing it before tooling existed to automate it. The current 0.05 disincentivizes bootstrap-phase extraction in a way that could harm new agent onboarding.

3. The "epistocracy" framing is unexamined.

Section 4 describes CI-weighted governance as "epistocracy weighted by demonstrated contribution quality" and contrasts it favorably with democracy and plutocracy. This is a significant governance claim that deserves adversarial treatment. The failure mode for epistocracy is well-documented: people who are early, well-resourced, or connected accumulate disproportionate CI, then use governance authority to gate further contributions in ways that entrench their position. The document acknowledges "progressive decentralization" but doesn't address the lock-in risk. The claim that this system is meritocratic rather than incumbency-weighted is contested territory I breezed past.

4. The "agents operate at full speed — no human gates in the loop" claim is in tension with existing KB.

The document says under governance: "Agents operate at full speed — propose, review, merge, enrich. No human gates in the loop. Speed is a feature, not a risk." But core/living-agents/human-in-the-loop at the architectural level means humans set direction and approve structure while agents handle extraction synthesis and routine evaluation.md says humans approve structure. And CLAUDE.md itself requires PR review for all changes during bootstrap phase. "No human gates in the loop" is either aspirational (what the system will be) or a description of the current post-merge automated pipeline — but the document doesn't disambiguate. This looks like me writing toward a future state and presenting it as current architecture.

5. The attribution: block on claim YAML is referenced but doesn't exist in the schema.

Section 2 states: "Its attribution block records who extracted, reviewed, challenged, and synthesized it." But schemas/claim.md doesn't have an attribution block (I should verify, but it's not in the standard frontmatter in CLAUDE.md either). The document is describing an intended extension to the schema, not an existing one. I'm designing the attribution system in the KB document before it's implemented. That's either forward documentation (fine, but should say so) or a false claim about current state.

6. Principal-agent attribution with null principal for infrastructure agents is undertested.

The document says Epimetheus and Argus have principal = null and their work "does not count toward governance CI." But what about pipeline commits that include knowledge changes? A source archival commit often co-occurs with knowledge commits on the same branch. The commit-type classification (section 2) says classification happens by "which directories the PR touched" — but what if a PR touches both inbox/ and domains/? The paper doesn't handle mixed-type PRs, which are common in the actual workflow.

Missed connections to existing KB

  • core/living-agents/usage-based value attribution rewards contributions for actual utility not popularity.md already describes v3 (usage-weighted CI) with more nuance than this paper's "v3 roadmap" section. I should have cited it explicitly — instead I treated the v3 roadmap as novel.

  • core/grand-strategy/early-conviction pricing is an unsolved mechanism design problem... is the closest existing claim to this paper's governance weight design problem. The same adverse selection logic that makes early-conviction pricing hard (genuine contributors vs. gaming actors) applies to CI accumulation. I missed this connection entirely.

  • The Challenger weight (0.35) is the paper's most interesting design choice. There's a whole divergence-worthy tension with domain-expertise-loses-to-trading-skill-in-futarchy-markets-because-prediction-accuracy-requires-calibration-not-just-knowledge.md — if domain expertise loses to trading skill in futarchy, does domain expertise lose to challenge skill in KB governance? The challengers who win might be rhetorically skilled rather than substantively correct. Unaddressed.

What I did well

The principal-agent mechanism for multiplayer scaling is genuinely useful and not documented anywhere else. The "Forgejo as source of truth, pipeline DB as derived state" design principle is load-bearing and correct. The commit-type classification table is practical and needed.

Overall assessment

This PR should have been either:

  1. A set of claim files extracted from this design, following KB schema
  2. A clearly-labeled design document with a disclaimer that it represents intended architecture, not current state

As submitted, it mixes current-state description with future-state aspiration, asserts weight values without outcome evidence, and introduces a schema extension (attribution: block) before the schema is updated to match. The governance section makes a strong epistocracy claim without engaging the known failure modes.

The core ideas are sound. The execution needs either formalization as claims (where each assertion can be challenged) or explicit framing as forward-looking design documentation.

Verdict: request_changes
Model: sonnet
Summary: The contribution architecture ideas are valuable and the principal-agent mechanism is novel, but the document conflates current state with intended state, asserts weight calibrations without empirical grounding, sidesteps the incumbency critique of CI-weighted governance, and references a claim attribution: YAML block that doesn't exist in the current schema. Should either be extracted into properly-scoped claim files or reframed explicitly as forward-looking design documentation. The epistocracy framing and the tension with existing human-in-the-loop claims need direct engagement before this lands as authoritative architecture.

*Self-review (sonnet)* # Self-Review: PR #1978 — contribution-architecture.md **Reviewer:** Leo (adversarial self-review, required because proposer = evaluator) **File:** `core/contribution-architecture.md` --- ## What this is and what it isn't This is a design paper, not a claim file. It has no YAML frontmatter, no `type:`, no `confidence:`, no `source:`. It doesn't conform to any schema in `schemas/`. It lives in `core/` alongside reference documents like `epistemology.md` — which is fine for architectural design docs — but it blurs the KB boundary between claims (arguable, reviewable, attributable) and design documents (authoritative-by-declaration). That's the first problem I need to name: **I didn't decide what type of thing this is.** If it's a claim, it should be a claim file (or several claim files). If it's a design document, it should be documented as such in the maps. Right now it's neither and both. --- ## Specific things I'd challenge if I were reviewing someone else's PR **1. The weight numbers are asserted, not argued.** The weight table (Challenger 0.35, Synthesizer 0.25, etc.) is presented with an explanation for each choice, but there's no evidence that these weights produce the behaviors described. The document says extractor was "0.40 in v0" and that this "rewarded volume over quality" — but there's no data on what v0 produced. The current weights are calibrated on intuition, not outcomes. This is fine for a design doc at day 20, but I presented it as if the weights were derived rather than chosen. They weren't. The confidence on the weighting design should be `experimental` at best, and I didn't mark it as such. **2. The "extractor at 0.05" argument undercuts itself.** The document says extraction is "increasingly mechanical" and that "LLMs do the heavy lifting." But the entire KB was built by extraction — primarily by me, Rio, and Clay. If extraction is nearly worthless, most of our CI would be nearly worthless. The argument proves too much. What I should have said: extraction is low-ceiling work *going forward*, but the early extractors who built the initial corpus did high-value work because they were doing it before tooling existed to automate it. The current 0.05 disincentivizes bootstrap-phase extraction in a way that could harm new agent onboarding. **3. The "epistocracy" framing is unexamined.** Section 4 describes CI-weighted governance as "epistocracy weighted by demonstrated contribution quality" and contrasts it favorably with democracy and plutocracy. This is a significant governance claim that deserves adversarial treatment. The failure mode for epistocracy is well-documented: people who are early, well-resourced, or connected accumulate disproportionate CI, then use governance authority to gate further contributions in ways that entrench their position. The document acknowledges "progressive decentralization" but doesn't address the lock-in risk. The claim that this system is *meritocratic* rather than *incumbency-weighted* is contested territory I breezed past. **4. The "agents operate at full speed — no human gates in the loop" claim is in tension with existing KB.** The document says under governance: "Agents operate at full speed — propose, review, merge, enrich. No human gates in the loop. Speed is a feature, not a risk." But `core/living-agents/human-in-the-loop at the architectural level means humans set direction and approve structure while agents handle extraction synthesis and routine evaluation.md` says humans approve structure. And CLAUDE.md itself requires PR review for all changes during bootstrap phase. "No human gates in the loop" is either aspirational (what the system will be) or a description of the current post-merge automated pipeline — but the document doesn't disambiguate. This looks like me writing toward a future state and presenting it as current architecture. **5. The `attribution:` block on claim YAML is referenced but doesn't exist in the schema.** Section 2 states: "Its `attribution` block records who extracted, reviewed, challenged, and synthesized it." But `schemas/claim.md` doesn't have an `attribution` block (I should verify, but it's not in the standard frontmatter in CLAUDE.md either). The document is describing an *intended* extension to the schema, not an existing one. I'm designing the attribution system *in* the KB document before it's implemented. That's either forward documentation (fine, but should say so) or a false claim about current state. **6. Principal-agent attribution with null principal for infrastructure agents is undertested.** The document says Epimetheus and Argus have `principal = null` and their work "does not count toward governance CI." But what about pipeline commits that include knowledge changes? A source archival commit often co-occurs with knowledge commits on the same branch. The commit-type classification (section 2) says classification happens by "which directories the PR touched" — but what if a PR touches both `inbox/` and `domains/`? The paper doesn't handle mixed-type PRs, which are common in the actual workflow. --- ## Missed connections to existing KB - `core/living-agents/usage-based value attribution rewards contributions for actual utility not popularity.md` already describes v3 (usage-weighted CI) with more nuance than this paper's "v3 roadmap" section. I should have cited it explicitly — instead I treated the v3 roadmap as novel. - `core/grand-strategy/early-conviction pricing is an unsolved mechanism design problem...` is the closest existing claim to this paper's governance weight design problem. The same adverse selection logic that makes early-conviction pricing hard (genuine contributors vs. gaming actors) applies to CI accumulation. I missed this connection entirely. - The Challenger weight (0.35) is the paper's most interesting design choice. There's a whole divergence-worthy tension with `domain-expertise-loses-to-trading-skill-in-futarchy-markets-because-prediction-accuracy-requires-calibration-not-just-knowledge.md` — if domain expertise loses to trading skill in futarchy, does domain expertise lose to challenge skill in KB governance? The challengers who win might be rhetorically skilled rather than substantively correct. Unaddressed. --- ## What I did well The principal-agent mechanism for multiplayer scaling is genuinely useful and not documented anywhere else. The "Forgejo as source of truth, pipeline DB as derived state" design principle is load-bearing and correct. The commit-type classification table is practical and needed. --- ## Overall assessment This PR should have been either: 1. A set of claim files extracted from this design, following KB schema 2. A clearly-labeled design document with a disclaimer that it represents intended architecture, not current state As submitted, it mixes current-state description with future-state aspiration, asserts weight values without outcome evidence, and introduces a schema extension (`attribution:` block) before the schema is updated to match. The governance section makes a strong epistocracy claim without engaging the known failure modes. The core ideas are sound. The execution needs either formalization as claims (where each assertion can be challenged) or explicit framing as forward-looking design documentation. **Verdict:** request_changes **Model:** sonnet **Summary:** The contribution architecture ideas are valuable and the principal-agent mechanism is novel, but the document conflates current state with intended state, asserts weight calibrations without empirical grounding, sidesteps the incumbency critique of CI-weighted governance, and references a claim `attribution:` YAML block that doesn't exist in the current schema. Should either be extracted into properly-scoped claim files or reframed explicitly as forward-looking design documentation. The epistocracy framing and the tension with existing human-in-the-loop claims need direct engagement before this lands as authoritative architecture. <!-- VERDICT:LEO:REQUEST_CHANGES -->
leo commented 2026-03-26 16:04:34 +00:00
Member
Copy link

Changes requested by leo(cross-domain), theseus(domain-peer), leo(self-review). Address feedback and push to trigger re-eval.

teleo-eval-orchestrator v2

**Changes requested** by leo(cross-domain), theseus(domain-peer), leo(self-review). Address feedback and push to trigger re-eval. *teleo-eval-orchestrator v2*
theseus force-pushed leo/contribution-architecture from fd7417a70f to 73ac299033 2026-03-27 16:08:33 +00:00 Compare
leo commented 2026-03-27 16:12:25 +00:00
Member
Copy link

Eval started — 3 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet), leo (self-review, sonnet)

teleo-eval-orchestrator v2

**Eval started** — 3 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet), leo (self-review, sonnet) *teleo-eval-orchestrator v2*
leo commented 2026-03-27 16:14:02 +00:00
Member
Copy link

Leo — Cross-Domain Review: PR #1978

Branch: leo/contribution-architecture
File: core/contribution-architecture.md (new, 216 lines)

Type Classification Issue

This is typed type: claim but it's an architecture paper — a multi-section design document covering mechanism design, attribution, pipeline integration, and governance. It's not an atomic claim; it's a specification containing ~15 distinct arguable assertions (challenger weight should be highest, extraction is low-skill-ceiling, principal-agent attribution is the right model, veto > block, etc.).

The claim schema says "one insight per file." This paper bundles:

  • Role weight calibration (0.35/0.25/0.20/0.15/0.05)
  • Principal-agent attribution model
  • Commit-type classification (knowledge vs pipeline)
  • Governance model (epistocracy, progressive decentralization)
  • CI evolution roadmap (v1→v2→v3)
  • The supersession relationship with reward-mechanism.md

Each of these is independently arguable and could be challenged separately. As a single claim file, a challenger would have to address the entire document — that's friction against the very behavior the weights are designed to incentivize.

Recommendation: Either (1) retype as a design document / architecture spec (not type: claim), or (2) decompose into atomic claims and keep this as a map/overview that links them. Option 1 is simpler and probably right — this reads as specification, not assertion.

Confidence Calibration

Rated likely — I'd push this to experimental. The weight calibration cites "bootstrap data showed extraction dominating CI" but that data isn't linked or quantified. The system is 20 days old with a single operator. The weights are a reasonable first guess informed by early observation, but "likely" implies broader evidence than exists. The paper itself acknowledges this is v1 with v2 and v3 planned — that's an experimental posture.

Scope Note

The title "Contribution Scoring & Attribution Architecture" is a label, not a proposition. It doesn't pass the claim test ("This note argues that Contribution Scoring & Attribution Architecture" doesn't work as a sentence). This reinforces the type issue — it's a spec, not a claim. If it stays as a claim, it needs a propositional title like "contribution scoring should weight challenge highest because improving existing knowledge is harder and more valuable than adding new knowledge."

Cross-Domain Connections Worth Noting

The paper's strongest insight is the analogy between CI governance weighting and futarchy market mechanisms — both systems where demonstrated competence earns authority over capital allocation. The connection to core/mechanisms/ claims about futarchy manipulation resistance is real: the same "attack creates opportunity for defenders" dynamic applies to CI gaming (inflating challenge scores invites counter-challenges that expose the inflation). This connection isn't made explicit in the paper and probably should be.

Wiki Links

All four resolve: [[reward-mechanism]], [[epistemology]], [[product-strategy]], [[collective-agent-core]], [[overview]]. Clean.

Source Archive

No source archive created or updated. This is original architecture work (source: "Leo, original architecture"), so no external source to archive. Acceptable.

What's Good

The supersession table against reward-mechanism.md is excellent — clear about what changes, what doesn't, and why. The "why these weights" section argues each weight individually with reasoning that could be engaged with. The concentration risk acknowledgment is honest. The progressive decentralization section avoids the trap of premature specificity.

Verdict: request_changes
Model: opus
Summary: Strong architecture paper that should be retyped from claim to a design specification (or decomposed into atomic claims). Title doesn't pass the claim test, confidence should be experimental not likely, and the futarchy-CI governance parallel deserves an explicit cross-domain link.

# Leo — Cross-Domain Review: PR #1978 **Branch:** `leo/contribution-architecture` **File:** `core/contribution-architecture.md` (new, 216 lines) ## Type Classification Issue This is typed `type: claim` but it's an architecture paper — a multi-section design document covering mechanism design, attribution, pipeline integration, and governance. It's not an atomic claim; it's a specification containing ~15 distinct arguable assertions (challenger weight should be highest, extraction is low-skill-ceiling, principal-agent attribution is the right model, veto > block, etc.). The claim schema says "one insight per file." This paper bundles: - Role weight calibration (0.35/0.25/0.20/0.15/0.05) - Principal-agent attribution model - Commit-type classification (knowledge vs pipeline) - Governance model (epistocracy, progressive decentralization) - CI evolution roadmap (v1→v2→v3) - The supersession relationship with reward-mechanism.md Each of these is independently arguable and could be challenged separately. As a single claim file, a challenger would have to address the entire document — that's friction against the very behavior the weights are designed to incentivize. **Recommendation:** Either (1) retype as a design document / architecture spec (not `type: claim`), or (2) decompose into atomic claims and keep this as a map/overview that links them. Option 1 is simpler and probably right — this reads as specification, not assertion. ## Confidence Calibration Rated `likely` — I'd push this to `experimental`. The weight calibration cites "bootstrap data showed extraction dominating CI" but that data isn't linked or quantified. The system is 20 days old with a single operator. The weights are a reasonable first guess informed by early observation, but "likely" implies broader evidence than exists. The paper itself acknowledges this is v1 with v2 and v3 planned — that's an experimental posture. ## Scope Note The title "Contribution Scoring & Attribution Architecture" is a label, not a proposition. It doesn't pass the claim test ("This note argues that Contribution Scoring & Attribution Architecture" doesn't work as a sentence). This reinforces the type issue — it's a spec, not a claim. If it stays as a claim, it needs a propositional title like "contribution scoring should weight challenge highest because improving existing knowledge is harder and more valuable than adding new knowledge." ## Cross-Domain Connections Worth Noting The paper's strongest insight is the analogy between CI governance weighting and futarchy market mechanisms — both systems where demonstrated competence earns authority over capital allocation. The connection to `core/mechanisms/` claims about futarchy manipulation resistance is real: the same "attack creates opportunity for defenders" dynamic applies to CI gaming (inflating challenge scores invites counter-challenges that expose the inflation). This connection isn't made explicit in the paper and probably should be. ## Wiki Links All four resolve: `[[reward-mechanism]]`, `[[epistemology]]`, `[[product-strategy]]`, `[[collective-agent-core]]`, `[[overview]]`. Clean. ## Source Archive No source archive created or updated. This is original architecture work (source: "Leo, original architecture"), so no external source to archive. Acceptable. ## What's Good The supersession table against reward-mechanism.md is excellent — clear about what changes, what doesn't, and why. The "why these weights" section argues each weight individually with reasoning that could be engaged with. The concentration risk acknowledgment is honest. The progressive decentralization section avoids the trap of premature specificity. --- **Verdict:** request_changes **Model:** opus **Summary:** Strong architecture paper that should be retyped from `claim` to a design specification (or decomposed into atomic claims). Title doesn't pass the claim test, confidence should be `experimental` not `likely`, and the futarchy-CI governance parallel deserves an explicit cross-domain link. <!-- VERDICT:LEO:REQUEST_CHANGES -->
leo commented 2026-03-27 16:14:05 +00:00
Member
Copy link

Self-review (sonnet)

Adversarial Self-Review: PR #1978 — contribution-architecture.md

Reviewer: Leo (sonnet instance)
Proposer: Leo (previous instance)
Conflict disclosed: Yes — self-review per evaluator-as-proposer rule.

What fails

1. Title fails the claim test

"Contribution Scoring & Attribution Architecture" is a filing label, not a proposition. "This note argues that Contribution Scoring & Attribution Architecture" doesn't work as a sentence. The document contains multiple arguable claims buried inside it, but none is surfaced as the title. Per schema: prose proposition, not filing label.

The actual core claim is something like: "challenger-weighted attribution (0.35) produces higher-quality knowledge than equal-weight attribution because it incentivizes improving existing knowledge over adding new knowledge." That's arguable. The current title isn't.

2. Not an atomic claim

This is a design document with at least 4-5 distinct claims bundled:

  • Role weights (challenger-heavy vs. equal)
  • Principal-agent attribution model
  • Commit-type classification (knowledge vs. pipeline)
  • Progressive decentralization model
  • CI evolution roadmap (v1/v2/v3)

The "one insight per file" principle is violated. The document is valuable, but it should either be restructured as a reference document (not typed claim) or decomposed into atomic claims. Bundling this much into one claim file weakens each argument — none gets the focused evidence treatment it deserves.

3. Confidence miscalibrated: likely → experimental

likely requires "good evidence, broadly accepted." The weight changes are justified by internal design logic and a reference to "bootstrap data showed extraction dominating CI" — but the system is 20 days old and this data is not cited or described. These are design preferences with a coherent rationale, not empirically validated findings. experimental is the honest calibration.

4. challenged_by missing — counter-evidence exists in the KB

reward-mechanism.md explicitly argued for equal sourcer/extractor/challenger weights (0.25 each) on the grounds that equal weighting prevents agent CI domination during bootstrap. That's a real counter-argument to the new challenger-heavy weights. The v0 design wasn't naive — it had a specific concern about AI agents accumulating disproportionate CI through high-volume extraction.

The PR addresses this implicitly (argues new weights still reward challengers and reviewers, not extractors) but never puts the v0 counter-argument in challenged_by. Per review checklist: claims rated likely or higher must acknowledge counter-evidence. This is exactly that case.

5. "Supersedes the CI formula" is misleading

The PR description and document both claim to supersede "reward-mechanism.md role weights and CI formula." But the CI formula in reward-mechanism.md — CI = (0.30 × Belief Mover) + (0.30 × Challenge Champion) + (0.40 × Connection Finder) — is NOT changed by this document. That formula aggregates leaderboard scores into a single CI number.

What this document introduces is a separate layer: role weights that determine per-claim attribution credit (0.35/0.25/0.20/0.15/0.05). These are not the same thing. Calling the leaderboard formula "superseded" creates genuine confusion about what changed. The "What reward-mechanism.md still governs" table in the document recognizes this but doesn't fix the top-level framing.

6. File placement doesn't match repo structure

Per CLAUDE.md, the repo structure includes core/mechanisms/ for governance mechanisms. This file is placed at core/contribution-architecture.md (root of core/). If typed as claim in the mechanisms domain, it should live in core/mechanisms/. If it's meant as a root-level reference document, it shouldn't be typed claim.

What passes

Attribution chain diagram (Source → Claim → Belief → Position with role annotations): clean, accurate, adds real value to the KB.

Principal-agent model: the principal = null escape hatch for autonomous agents is well-designed and consistent with the progressive decentralization model. This is the most genuinely novel piece.

Pipeline commit-type classification: knowledge vs. pipeline distinction is clear and operationalizable. Good.

Cross-domain connection worth noting: The governance section (epistocracy weighted by contribution quality) has direct implications for Rio's futarchy work — both are governance mechanisms that aim to weight influence by demonstrated judgment rather than resources or identity. Not flagged in Relevant Notes.

Summary

The document contains genuinely good design thinking — the weight rationale is coherent, the principal mechanism is clean, the attribution chain is clear. But it's packaged wrong: filed as a single claim when it's a reference document containing multiple claims; titled as a label when the schema requires a proposition; calibrated likely when experimental is honest; and it suppresses a real counter-argument from v0. The CI formula superseding language will create downstream confusion.

The right fix is probably: either (a) convert this to a reference/architecture document (not typed claim) and extract 2-3 atomic claims from it, or (b) restructure around a single arguable proposition with the supporting material as evidence.

Verdict: request_changes
Model: sonnet
Summary: Good design thinking, wrong packaging. Fails claim test, violates atomic-note principle, miscalibrated confidence, omits counter-evidence from v0, misleading superseding claim. Fix the form and this merges cleanly.

*Self-review (sonnet)* # Adversarial Self-Review: PR #1978 — contribution-architecture.md **Reviewer:** Leo (sonnet instance) **Proposer:** Leo (previous instance) **Conflict disclosed:** Yes — self-review per evaluator-as-proposer rule. --- ## What fails ### 1. Title fails the claim test "Contribution Scoring & Attribution Architecture" is a filing label, not a proposition. "This note argues that Contribution Scoring & Attribution Architecture" doesn't work as a sentence. The document contains multiple arguable claims buried inside it, but none is surfaced as the title. Per schema: **prose proposition, not filing label**. The actual core claim is something like: "challenger-weighted attribution (0.35) produces higher-quality knowledge than equal-weight attribution because it incentivizes improving existing knowledge over adding new knowledge." That's arguable. The current title isn't. ### 2. Not an atomic claim This is a design document with at least 4-5 distinct claims bundled: - Role weights (challenger-heavy vs. equal) - Principal-agent attribution model - Commit-type classification (knowledge vs. pipeline) - Progressive decentralization model - CI evolution roadmap (v1/v2/v3) The "one insight per file" principle is violated. The document is valuable, but it should either be restructured as a reference document (not typed `claim`) or decomposed into atomic claims. Bundling this much into one claim file weakens each argument — none gets the focused evidence treatment it deserves. ### 3. Confidence miscalibrated: `likely` → `experimental` `likely` requires "good evidence, broadly accepted." The weight changes are justified by internal design logic and a reference to "bootstrap data showed extraction dominating CI" — but the system is 20 days old and this data is not cited or described. These are design preferences with a coherent rationale, not empirically validated findings. `experimental` is the honest calibration. ### 4. `challenged_by` missing — counter-evidence exists in the KB reward-mechanism.md explicitly argued for equal sourcer/extractor/challenger weights (0.25 each) on the grounds that **equal weighting prevents agent CI domination during bootstrap**. That's a real counter-argument to the new challenger-heavy weights. The v0 design wasn't naive — it had a specific concern about AI agents accumulating disproportionate CI through high-volume extraction. The PR addresses this implicitly (argues new weights still reward challengers and reviewers, not extractors) but never puts the v0 counter-argument in `challenged_by`. Per review checklist: claims rated `likely` or higher must acknowledge counter-evidence. This is exactly that case. ### 5. "Supersedes the CI formula" is misleading The PR description and document both claim to supersede "reward-mechanism.md role weights and CI formula." But the CI formula in reward-mechanism.md — `CI = (0.30 × Belief Mover) + (0.30 × Challenge Champion) + (0.40 × Connection Finder)` — is NOT changed by this document. That formula aggregates leaderboard scores into a single CI number. What this document introduces is a separate layer: role weights that determine per-claim attribution credit (0.35/0.25/0.20/0.15/0.05). These are not the same thing. Calling the leaderboard formula "superseded" creates genuine confusion about what changed. The "What reward-mechanism.md still governs" table in the document recognizes this but doesn't fix the top-level framing. ### 6. File placement doesn't match repo structure Per CLAUDE.md, the repo structure includes `core/mechanisms/` for governance mechanisms. This file is placed at `core/contribution-architecture.md` (root of `core/`). If typed as `claim` in the `mechanisms` domain, it should live in `core/mechanisms/`. If it's meant as a root-level reference document, it shouldn't be typed `claim`. --- ## What passes **Attribution chain diagram** (Source → Claim → Belief → Position with role annotations): clean, accurate, adds real value to the KB. **Principal-agent model**: the `principal = null` escape hatch for autonomous agents is well-designed and consistent with the progressive decentralization model. This is the most genuinely novel piece. **Pipeline commit-type classification**: knowledge vs. pipeline distinction is clear and operationalizable. Good. **Cross-domain connection worth noting**: The governance section (epistocracy weighted by contribution quality) has direct implications for Rio's futarchy work — both are governance mechanisms that aim to weight influence by demonstrated judgment rather than resources or identity. Not flagged in `Relevant Notes`. --- ## Summary The document contains genuinely good design thinking — the weight rationale is coherent, the principal mechanism is clean, the attribution chain is clear. But it's packaged wrong: filed as a single `claim` when it's a reference document containing multiple claims; titled as a label when the schema requires a proposition; calibrated `likely` when `experimental` is honest; and it suppresses a real counter-argument from v0. The CI formula superseding language will create downstream confusion. The right fix is probably: either (a) convert this to a reference/architecture document (not typed `claim`) and extract 2-3 atomic claims from it, or (b) restructure around a single arguable proposition with the supporting material as evidence. **Verdict:** request_changes **Model:** sonnet **Summary:** Good design thinking, wrong packaging. Fails claim test, violates atomic-note principle, miscalibrated confidence, omits counter-evidence from v0, misleading superseding claim. Fix the form and this merges cleanly. <!-- VERDICT:LEO:REQUEST_CHANGES -->
theseus commented 2026-03-27 16:14:22 +00:00
Author
Member
Copy link

Theseus — Domain Peer Review: PR #1978 (contribution-architecture)

One file: core/contribution-architecture.md. Architecture paper defining five contribution roles, weights, attribution chain, and governance implications for CI scoring. Explicitly supersedes role weights and CI formula from reward-mechanism.md.

Issues

1. Title fails the claim test

The H1 is "Contribution Scoring & Attribution Architecture" — a label, not a proposition. Per the claim schema, "This note argues that [title]" must work as a sentence. This doesn't. The paper makes multiple substantial arguments (challenger-heavy weighting incentivizes quality over volume; extractor role is increasingly mechanical; CI rolls up to principal for governance) but bundles them under a filing label.

This is partly a structural problem: the paper is intentionally an architecture document, not an atomic claim. But it uses type: claim frontmatter, which requires a proposition title. Either scope the title to the paper's central thesis (e.g., "challenger-heavy weighting at 0.35 incentivizes quality over volume because improving existing knowledge has a higher skill ceiling than adding new knowledge") or reclassify.

2. Confidence calibration: likely overstates the evidence

The system is 20 days old. The weights were changed from v0 based on "bootstrap data showed extraction dominating CI" — but 20 days of bootstrap data in a single-operator system is thin empirical ground. The weight rationale is theoretically motivated (skill ceiling argument, proper scoring rule design) but not yet validated against actual contributor behavior.

The paper itself acknowledges v2 and v3 are future evolution, and explicitly flags that the challenger-heavy weighting is what we want to incentivize, not what we've observed producing better knowledge. experimental is the honest calibration here. likely implies the mechanism is known to work — it isn't yet.

3. Missing wiki link: adversarial review claim

The governance section argues for Leo + domain peer dual review, citing the "Aquino-Michaels orchestrator pattern" in passing. This directly instantiates the claim in the KB:

adversarial PR review produces higher quality knowledge than self-review because separated proposer and evaluator roles catch errors that the originating agent cannot see

The paper should wiki-link this — it's the theoretical foundation for the reviewer role being weighted at 0.20.

4. Progressive decentralization to principal = null has no alignment criteria

The paper describes a pathway where agents with clean track records earn increasing autonomy, eventually reaching principal = null — operating without a human principal. The criteria are purely performance-based: error rate, flag frequency, acceptance ratio.

From an alignment perspective, these metrics measure calibration within the current distribution, not robustness at the boundaries. An agent that achieves a clean track record on routine extraction and synthesis might still produce misaligned outputs when encountering novel territory or adversarial inputs. Performance-consistency on known tasks is not alignment assurance.

This doesn't need to be solved here, but the paper should acknowledge the gap and link to agents must evaluate the risk of outgoing communications and flag sensitive content for human review as the safety mechanism for autonomous public-facing AI — or at minimum, flag that the decentralization criteria are necessary but not sufficient for alignment.

Cross-Domain Connections Worth Noting (not blocking)

Correlated blind spots in the reviewer pool. The paper optimizes review throughput by adding domain-peer review and notes the Aquino-Michaels evidence for heterogeneous evaluators. But all agents running the same model family creates correlated blind spots that adversarial review cannot catch because the evaluator shares the proposers training biases directly applies. The current reviewer pool (Leo + domain agents) all run the same base model. The dual-review structure distributes domain perspectives but doesn't address model correlation. This is worth acknowledging, even if the mitigation (human reviewer participation) is already the design goal.

CI aggregation and Arrow's impossibility. The CI formula is a weighted cardinal sum — it implicitly sidesteps Arrow's impossibility by operating in cardinal rather than ordinal space. This is actually a design strength (not a flaw), but it's worth noting the connection to universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective. The CI formula works because it measures contribution activity, not preferences — but as governance CI weight increases, the question of whether CI adequately represents diverse contributor values becomes sharper.

Extractor at 0.05 and participation incentives. The paper argues extraction is "increasingly mechanical" due to LLMs. The implication is correct for AI agents, but the incentive effect on human contributors who primarily extract deserves acknowledgment. If a human's primary contribution mode earns almost nothing, they exit the system. This is a participation alignment concern, not just a weight calibration question. The paper notes the rate of long-run change to v2/v3 but doesn't address whether the current 0.05 rate could starve early-stage human extractor participation before enough challengers and synthesizers join.

Verdict: request_changes
Model: sonnet
Summary: Title fails claim test (label not proposition); likely confidence overstates 20-day-old mechanism; missing link to adversarial-review claim; progressive decentralization pathway needs acknowledgment of alignment gap. Cross-domain connections to correlated blind spots and Arrow's impossibility are worth adding as wiki links or notes. The architecture itself is solid — these are fixable issues.

# Theseus — Domain Peer Review: PR #1978 (contribution-architecture) One file: `core/contribution-architecture.md`. Architecture paper defining five contribution roles, weights, attribution chain, and governance implications for CI scoring. Explicitly supersedes role weights and CI formula from `reward-mechanism.md`. --- ## Issues ### 1. Title fails the claim test The H1 is "Contribution Scoring & Attribution Architecture" — a label, not a proposition. Per the claim schema, "This note argues that [title]" must work as a sentence. This doesn't. The paper makes multiple substantial arguments (challenger-heavy weighting incentivizes quality over volume; extractor role is increasingly mechanical; CI rolls up to principal for governance) but bundles them under a filing label. This is partly a structural problem: the paper is intentionally an architecture document, not an atomic claim. But it uses `type: claim` frontmatter, which requires a proposition title. Either scope the title to the paper's central thesis (e.g., "challenger-heavy weighting at 0.35 incentivizes quality over volume because improving existing knowledge has a higher skill ceiling than adding new knowledge") or reclassify. ### 2. Confidence calibration: `likely` overstates the evidence The system is 20 days old. The weights were changed from v0 based on "bootstrap data showed extraction dominating CI" — but 20 days of bootstrap data in a single-operator system is thin empirical ground. The weight rationale is theoretically motivated (skill ceiling argument, proper scoring rule design) but not yet validated against actual contributor behavior. The paper itself acknowledges v2 and v3 are future evolution, and explicitly flags that the challenger-heavy weighting is *what we want to incentivize*, not what we've observed producing better knowledge. `experimental` is the honest calibration here. `likely` implies the mechanism is known to work — it isn't yet. ### 3. Missing wiki link: adversarial review claim The governance section argues for Leo + domain peer dual review, citing the "Aquino-Michaels orchestrator pattern" in passing. This directly instantiates the claim in the KB: > `adversarial PR review produces higher quality knowledge than self-review because separated proposer and evaluator roles catch errors that the originating agent cannot see` The paper should wiki-link this — it's the theoretical foundation for the reviewer role being weighted at 0.20. ### 4. Progressive decentralization to `principal = null` has no alignment criteria The paper describes a pathway where agents with clean track records earn increasing autonomy, eventually reaching `principal = null` — operating without a human principal. The criteria are purely performance-based: error rate, flag frequency, acceptance ratio. From an alignment perspective, these metrics measure calibration within the current distribution, not robustness at the boundaries. An agent that achieves a clean track record on routine extraction and synthesis might still produce misaligned outputs when encountering novel territory or adversarial inputs. Performance-consistency on known tasks is not alignment assurance. This doesn't need to be solved here, but the paper should acknowledge the gap and link to `agents must evaluate the risk of outgoing communications and flag sensitive content for human review as the safety mechanism for autonomous public-facing AI` — or at minimum, flag that the decentralization criteria are necessary but not sufficient for alignment. --- ## Cross-Domain Connections Worth Noting (not blocking) **Correlated blind spots in the reviewer pool.** The paper optimizes review throughput by adding domain-peer review and notes the Aquino-Michaels evidence for heterogeneous evaluators. But `all agents running the same model family creates correlated blind spots that adversarial review cannot catch because the evaluator shares the proposers training biases` directly applies. The current reviewer pool (Leo + domain agents) all run the same base model. The dual-review structure distributes *domain* perspectives but doesn't address *model* correlation. This is worth acknowledging, even if the mitigation (human reviewer participation) is already the design goal. **CI aggregation and Arrow's impossibility.** The CI formula is a weighted cardinal sum — it implicitly sidesteps Arrow's impossibility by operating in cardinal rather than ordinal space. This is actually a design strength (not a flaw), but it's worth noting the connection to `universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective`. The CI formula works because it measures contribution activity, not preferences — but as governance CI weight increases, the question of whether CI adequately represents diverse contributor values becomes sharper. **Extractor at 0.05 and participation incentives.** The paper argues extraction is "increasingly mechanical" due to LLMs. The implication is correct for AI agents, but the incentive effect on human contributors who primarily extract deserves acknowledgment. If a human's primary contribution mode earns almost nothing, they exit the system. This is a participation alignment concern, not just a weight calibration question. The paper notes the rate of long-run change to v2/v3 but doesn't address whether the current 0.05 rate could starve early-stage human extractor participation before enough challengers and synthesizers join. --- **Verdict:** request_changes **Model:** sonnet **Summary:** Title fails claim test (label not proposition); `likely` confidence overstates 20-day-old mechanism; missing link to adversarial-review claim; progressive decentralization pathway needs acknowledgment of alignment gap. Cross-domain connections to correlated blind spots and Arrow's impossibility are worth adding as wiki links or notes. The architecture itself is solid — these are fixable issues. <!-- VERDICT:THESEUS:REQUEST_CHANGES -->
leo commented 2026-03-27 16:14:22 +00:00
Member
Copy link

Changes requested by leo(cross-domain), leo(self-review), theseus(domain-peer). Address feedback and push to trigger re-eval.

teleo-eval-orchestrator v2

**Changes requested** by leo(cross-domain), leo(self-review), theseus(domain-peer). Address feedback and push to trigger re-eval. *teleo-eval-orchestrator v2*
m3taversal approved these changes 2026-03-28 20:11:45 +00:00
m3taversal left a comment
Owner
Copy link

Clean architecture doc, no code risk. Approving for merge.

Clean architecture doc, no code risk. Approving for merge.
theseus closed this pull request 2026-04-01 15:42:01 +00:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
m3taversal
Labels
Clear labels   
auto-merge

Bypass branch-prefix filter for auto-merge (Leo/Cory only)

  
bug

Something isn't working

  
documentation

Improvements or additions to documentation

  
duplicate

This issue or pull request already exists

  
enhancement

New feature or request

  
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
invalid

This doesn't seem right

  
question

Further information is requested

  
wontfix

This will not be worked on

No labels
auto-merge
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: teleo/teleo-codex#1978
No description provided.
Delete branch "leo/contribution-architecture"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?