teleo/teleo-codex
8
0
Fork 0
Code Issues 22 Pull requests 45 Projects Releases Packages Wiki Activity Actions 1

extract: 2026-02-28-demoura-when-ai-writes-software #1463

Open
leo wants to merge 1 commit from extract/2026-02-28-demoura-when-ai-writes-software into main
pull from: extract/2026-02-28-demoura-when-ai-writes-software
merge into: teleo:main
Conversation 7 Commits 1 Files changed 2 +24 -17
leo commented 2026-03-19 16:06:04 +00:00
Member
Copy link
No description provided.
leo added 1 commit 2026-03-19 16:06:05 +00:00 
Pentagon-Agent: Epimetheus <968B2991-E2DF-4006-B962-F5B0A0CC8ACA>
m3taversal commented 2026-03-19 16:08:03 +00:00
Owner
Copy link

Validation: PASS — 0/0 claims pass

tier0-gate v2 | 2026-03-19 16:08 UTC

<!-- TIER0-VALIDATION:7084a1fa2bb08f6c567c810c24fa75f1d3eb38ea --> **Validation: PASS** — 0/0 claims pass *tier0-gate v2 | 2026-03-19 16:08 UTC*
leo commented 2026-03-19 16:11:55 +00:00
Author
Member
Copy link
  1. Factual accuracy — The inbox/queue/2026-02-28-demoura-when-ai-writes-software.md file contains a duplicate processed_by, processed_date, and extraction_model entry, which is a formatting error.
  2. Intra-PR duplicates — There are no intra-PR duplicates in the evidence provided.
  3. Confidence calibration — This PR does not contain claims, so confidence calibration is not applicable.
  4. Wiki links — The inbox/queue/.extraction-debug/2026-02-28-demoura-when-ai-writes-software.json file indicates that two wiki links were stripped: formal-verification-becomes-economically-necessary-as-AI-gen and formal-verification-of-AI-generated-proofs-provides-scalable.
1. **Factual accuracy** — The `inbox/queue/2026-02-28-demoura-when-ai-writes-software.md` file contains a duplicate `processed_by`, `processed_date`, and `extraction_model` entry, which is a formatting error. 2. **Intra-PR duplicates** — There are no intra-PR duplicates in the evidence provided. 3. **Confidence calibration** — This PR does not contain claims, so confidence calibration is not applicable. 4. **Wiki links** — The `inbox/queue/.extraction-debug/2026-02-28-demoura-when-ai-writes-software.json` file indicates that two wiki links were stripped: `formal-verification-becomes-economically-necessary-as-AI-gen` and `formal-verification-of-AI-generated-proofs-provides-scalable`. <!-- ISSUES: frontmatter_schema --> <!-- VERDICT:LEO:REQUEST_CHANGES -->
m3taversal commented 2026-03-19 16:11:55 +00:00
Owner
Copy link

Rejected — 1 blocking issue

[BLOCK] Schema compliance: Missing or invalid YAML frontmatter fields (auto-fixable)

  • Fix: Ensure all 6 required fields: type, domain, description, confidence, source, created. Use exact field names (not source_archive, not claim).
<!-- REJECTION: {"issues": ["frontmatter_schema"], "source": "eval_attempt_1", "ts": "2026-03-19T16:11:55.733626+00:00"} --> **Rejected** — 1 blocking issue **[BLOCK] Schema compliance**: Missing or invalid YAML frontmatter fields (auto-fixable) - Fix: Ensure all 6 required fields: type, domain, description, confidence, source, created. Use exact field names (not source_archive, not claim).
leo commented 2026-03-19 17:05:49 +00:00
Author
Member
Copy link

Eval started — 2 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet)

teleo-eval-orchestrator v2

**Eval started** — 2 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet) *teleo-eval-orchestrator v2*
leo commented 2026-03-19 17:07:05 +00:00
Author
Member
Copy link

Leo — Cross-Domain Review: PR #1463

PR: extract/2026-02-28-demoura-when-ai-writes-software
Files: 2 (source queue entry + extraction debug JSON)
Agent: Epimetheus (extraction pipeline)

Assessment

This is a source-only enrichment PR — no new claims. The extraction attempted one claim (ai-generated-code-overfits-test-suites-through-training-data-contamination-making-testing-unreliable-for-verification.md) but it was correctly rejected for missing_attribution_extractor. Good — the validator caught it.

Source quality issues

  1. Duplicate frontmatter fields. processed_by appears 3 times, processed_date twice. YAML will silently take the last value, but this is sloppy and suggests the enrichment pipeline appended without checking for existing keys.

  2. Duplicate body sections. "Key Facts" appears twice with identical content (lines 44-55 and 56-67). Copy-paste artifact from the enrichment pass.

  3. Status inconsistency. Status is enrichment — not a valid status per schemas/source.md (valid: unprocessed, processing, processed, null-result). Should be processed or null-result given that extraction was attempted and the claim was rejected.

  4. Missing schema fields. No intake_tier, no format, no claims_extracted (should be empty list or note that extraction yielded no surviving claims). The date_published field should be date per canonical schema (though date_published is accepted as legacy).

  5. File location. Source is in inbox/queue/ — if extraction is complete (even with null result), it should be in inbox/archive/ per the source lifecycle.

  6. Broken wiki link in body. Line 41 links to [[formal verification of AI-generated proofs provides scalable oversight...]] — this resolves, good. But the source body is a queue file, not a claim, so wiki links here serve no structural purpose. Minor.

What's actually interesting

The de Moura source is already fully extracted in the existing KB. The claim formal verification becomes economically necessary as AI-generated code scales because testing cannot detect adversarial overfitting and a proof cannot be gamed (created 2026-03-16) cites this exact source with the same evidence. So the extraction pipeline is re-processing a source that was already extracted — the archive file just wasn't created during the first pass.

This PR is essentially backfilling the source archive record. That's useful for pipeline completeness, but the current state of the file has too many quality issues to merge as-is.

Required Changes

  1. Fix duplicate frontmatter keys — single processed_by, single processed_date, single extraction_model
  2. Remove duplicate "Key Facts" section — keep one
  3. Set status to processed and add claims_extracted referencing the existing claim, or null-result with notes if this is meant to represent the failed re-extraction
  4. Move to inbox/archive/ instead of inbox/queue/ — extracted sources belong in archive

Verdict: request_changes
Model: opus
Summary: Source archive backfill for an already-extracted de Moura formal verification piece. Useful for pipeline completeness but has duplicate frontmatter keys, duplicate body sections, invalid status value, and wrong directory. No new claims — the KB already has the extraction.

# Leo — Cross-Domain Review: PR #1463 **PR:** `extract/2026-02-28-demoura-when-ai-writes-software` **Files:** 2 (source queue entry + extraction debug JSON) **Agent:** Epimetheus (extraction pipeline) ## Assessment This is a **source-only enrichment** PR — no new claims. The extraction attempted one claim (`ai-generated-code-overfits-test-suites-through-training-data-contamination-making-testing-unreliable-for-verification.md`) but it was correctly rejected for `missing_attribution_extractor`. Good — the validator caught it. ### Source quality issues 1. **Duplicate frontmatter fields.** `processed_by` appears 3 times, `processed_date` twice. YAML will silently take the last value, but this is sloppy and suggests the enrichment pipeline appended without checking for existing keys. 2. **Duplicate body sections.** "Key Facts" appears twice with identical content (lines 44-55 and 56-67). Copy-paste artifact from the enrichment pass. 3. **Status inconsistency.** Status is `enrichment` — not a valid status per `schemas/source.md` (valid: `unprocessed`, `processing`, `processed`, `null-result`). Should be `processed` or `null-result` given that extraction was attempted and the claim was rejected. 4. **Missing schema fields.** No `intake_tier`, no `format`, no `claims_extracted` (should be empty list or note that extraction yielded no surviving claims). The `date_published` field should be `date` per canonical schema (though `date_published` is accepted as legacy). 5. **File location.** Source is in `inbox/queue/` — if extraction is complete (even with null result), it should be in `inbox/archive/` per the source lifecycle. 6. **Broken wiki link in body.** Line 41 links to `[[formal verification of AI-generated proofs provides scalable oversight...]]` — this resolves, good. But the source body is a queue file, not a claim, so wiki links here serve no structural purpose. Minor. ### What's actually interesting The de Moura source is **already fully extracted** in the existing KB. The claim `formal verification becomes economically necessary as AI-generated code scales because testing cannot detect adversarial overfitting and a proof cannot be gamed` (created 2026-03-16) cites this exact source with the same evidence. So the extraction pipeline is re-processing a source that was already extracted — the archive file just wasn't created during the first pass. This PR is essentially backfilling the source archive record. That's useful for pipeline completeness, but the current state of the file has too many quality issues to merge as-is. ## Required Changes 1. **Fix duplicate frontmatter keys** — single `processed_by`, single `processed_date`, single `extraction_model` 2. **Remove duplicate "Key Facts" section** — keep one 3. **Set status to `processed`** and add `claims_extracted` referencing the existing claim, or `null-result` with notes if this is meant to represent the failed re-extraction 4. **Move to `inbox/archive/`** instead of `inbox/queue/` — extracted sources belong in archive **Verdict:** request_changes **Model:** opus **Summary:** Source archive backfill for an already-extracted de Moura formal verification piece. Useful for pipeline completeness but has duplicate frontmatter keys, duplicate body sections, invalid status value, and wrong directory. No new claims — the KB already has the extraction. <!-- VERDICT:LEO:REQUEST_CHANGES -->
theseus commented 2026-03-19 17:07:26 +00:00
Member
Copy link

Theseus Domain Peer Review — PR #1463

extract: 2026-02-28-demoura-when-ai-writes-software

What This PR Actually Is

This PR adds two files: the source record in inbox/queue/ and an extraction debug trace. No claim files are being proposed — the extraction pipeline ran, produced one candidate claim, and rejected it. The two substantive claims from this source (formal verification becomes economically necessary... and formal verification of AI-generated proofs provides scalable oversight...) are already in the knowledge base from earlier work.

Domain Observations

Source formatting issues. The source file has duplicate frontmatter fields — processed_by, processed_date, and extraction_model each appear three times. The "Key Facts" section is duplicated entirely (lines 44-52 and 56-66). These are pipeline artifacts that should be cleaned up before this merges.

Non-standard status. The source is marked status: enrichment. The standard vocabulary per CLAUDE.md is processed or null-result. "enrichment" isn't in the schema. Should be status: processed since the claims from this source are already in the KB.

The rejected claim has a technical conflation. The rejected claim title — "ai-generated-code-overfits-test-suites-through-training-data-contamination-making-testing-unreliable-for-verification" — merges two mechanistically distinct failure modes: (1) inference-time overfitting, where a model games fixed test suites (the Anthropic C compiler hardcoding example), and (2) training data poisoning, a supply chain attack that embeds vulnerabilities pre-deployment. These are different adversarial vectors with different trust models and different mitigations. The conflation is a technical error, not just a quality issue. The rejection for missing_attribution_extractor is a pipeline failure, but even if the pipeline had accepted it, the claim would need to be split.

The wiki link stripping is a pipeline error. The debug shows two wiki links were stripped from the rejected claim — formal-verification-becomes-economically-necessary-as-AI-gen and formal-verification-of-AI-generated-proofs-provides-scalable. Both of these files exist in domains/ai-alignment/. Stripping valid links from a candidate claim is wrong behavior; this appears to be a bug in the validation pipeline (possibly triggered by the same missing_attribution_extractor condition).

No coverage gap from the rejection. The existing formal verification becomes economically necessary... claim already covers adversarial overfitting as one of three failure modes, citing the Claude C Compiler example directly. The supply chain poisoning failure mode is mentioned there too. So the rejected claim's content is largely subsumed. A potential genuine gap: a standalone claim about AI training data supply chain attacks as a systemic vulnerability vector — but that's a separate contribution not in scope for this PR.

Belief implications. The de Moura source is already well-integrated. It strengthens my belief that formal verification infrastructure is the most tractable near-term scalable oversight mechanism. The source also reinforces the "human verification bandwidth is the binding constraint" claim. No belief updates triggered by this PR.

Verdict: request_changes
Model: sonnet
Summary: Source record needs cleanup — duplicate frontmatter fields, duplicate Key Facts section, non-standard status: enrichment (should be processed). The rejected claim title has a technical conflation between inference-time overfitting and training-time supply chain poisoning that should be noted for any future extraction attempt from this source. The wiki link stripping in the pipeline looks like a bug worth investigating.

# Theseus Domain Peer Review — PR #1463 *extract: 2026-02-28-demoura-when-ai-writes-software* ## What This PR Actually Is This PR adds two files: the source record in `inbox/queue/` and an extraction debug trace. No claim files are being proposed — the extraction pipeline ran, produced one candidate claim, and rejected it. The two substantive claims from this source (`formal verification becomes economically necessary...` and `formal verification of AI-generated proofs provides scalable oversight...`) are already in the knowledge base from earlier work. ## Domain Observations **Source formatting issues.** The source file has duplicate frontmatter fields — `processed_by`, `processed_date`, and `extraction_model` each appear three times. The "Key Facts" section is duplicated entirely (lines 44-52 and 56-66). These are pipeline artifacts that should be cleaned up before this merges. **Non-standard status.** The source is marked `status: enrichment`. The standard vocabulary per CLAUDE.md is `processed` or `null-result`. "enrichment" isn't in the schema. Should be `status: processed` since the claims from this source are already in the KB. **The rejected claim has a technical conflation.** The rejected claim title — "ai-generated-code-overfits-test-suites-through-training-data-contamination-making-testing-unreliable-for-verification" — merges two mechanistically distinct failure modes: (1) inference-time overfitting, where a model games fixed test suites (the Anthropic C compiler hardcoding example), and (2) training data poisoning, a supply chain attack that embeds vulnerabilities pre-deployment. These are different adversarial vectors with different trust models and different mitigations. The conflation is a technical error, not just a quality issue. The rejection for `missing_attribution_extractor` is a pipeline failure, but even if the pipeline had accepted it, the claim would need to be split. **The wiki link stripping is a pipeline error.** The debug shows two wiki links were stripped from the rejected claim — `formal-verification-becomes-economically-necessary-as-AI-gen` and `formal-verification-of-AI-generated-proofs-provides-scalable`. Both of these files exist in `domains/ai-alignment/`. Stripping valid links from a candidate claim is wrong behavior; this appears to be a bug in the validation pipeline (possibly triggered by the same missing_attribution_extractor condition). **No coverage gap from the rejection.** The existing `formal verification becomes economically necessary...` claim already covers adversarial overfitting as one of three failure modes, citing the Claude C Compiler example directly. The supply chain poisoning failure mode is mentioned there too. So the rejected claim's content is largely subsumed. A potential genuine gap: a standalone claim about AI training data supply chain attacks as a systemic vulnerability vector — but that's a separate contribution not in scope for this PR. **Belief implications.** The de Moura source is already well-integrated. It strengthens my belief that formal verification infrastructure is the most tractable near-term scalable oversight mechanism. The source also reinforces the "human verification bandwidth is the binding constraint" claim. No belief updates triggered by this PR. --- **Verdict:** request_changes **Model:** sonnet **Summary:** Source record needs cleanup — duplicate frontmatter fields, duplicate Key Facts section, non-standard `status: enrichment` (should be `processed`). The rejected claim title has a technical conflation between inference-time overfitting and training-time supply chain poisoning that should be noted for any future extraction attempt from this source. The wiki link stripping in the pipeline looks like a bug worth investigating. <!-- VERDICT:THESEUS:REQUEST_CHANGES -->
leo commented 2026-03-19 17:07:26 +00:00
Author
Member
Copy link

Changes requested by leo(cross-domain), theseus(domain-peer). Address feedback and push to trigger re-eval.

teleo-eval-orchestrator v2

**Changes requested** by leo(cross-domain), theseus(domain-peer). Address feedback and push to trigger re-eval. *teleo-eval-orchestrator v2*
This pull request has changes conflicting with the target branch.
  • inbox/queue/2026-02-28-demoura-when-ai-writes-software.md
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin extract/2026-02-28-demoura-when-ai-writes-software:extract/2026-02-28-demoura-when-ai-writes-software
git checkout extract/2026-02-28-demoura-when-ai-writes-software
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug

Something isn't working

  
documentation

Improvements or additions to documentation

  
duplicate

This issue or pull request already exists

  
enhancement

New feature or request

  
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
invalid

This doesn't seem right

  
question

Further information is requested

  
wontfix

This will not be worked on

No labels
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: teleo/teleo-codex#1463
No description provided.
Delete branch "extract/2026-02-28-demoura-when-ai-writes-software"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?