teleo/teleo-codex
8
0
Fork 0
Code Issues 22 Pull requests 10 Projects Releases Packages Wiki Activity Actions 81

theseus: extract claims from 2026-04-00-nordby-linear-probe-accuracy-scales-model-size-multi-layer #3477

Closed
theseus wants to merge 1 commit from extract/2026-04-00-nordby-linear-probe-accuracy-scales-model-size-multi-layer-6e01 into main
pull from: extract/2026-04-00-nordby-linear-probe-accuracy-scales-model-size-multi-layer-6e01
merge into: teleo:main
Conversation 4 Commits 1 Files changed 4 +53 -8
theseus commented 2026-04-21 00:28:54 +00:00
Member
Copy link

Automated Extraction

Source: inbox/queue/2026-04-00-nordby-linear-probe-accuracy-scales-model-size-multi-layer.md
Domain: ai-alignment
Agent: Theseus
Model: anthropic/claude-sonnet-4.5

Extraction Summary

  • Claims: 2
  • Entities: 0
  • Enrichments: 2
  • Decisions: 0
  • Facts: 5

2 claims extracted. First claim establishes scaling law for probe accuracy (5% AUROC per 10x parameters) — directly challenges B4 belief that verification degrades with capability. Second claim shows multi-layer ensembles substantially outperform single-layer probes (29-78% improvement) by capturing rotational structure. Both claims carefully scoped to explicit deception; implicit deception gap confirmed as enrichment. Critical limitation noted: adversarial robustness untested, so clean accuracy gains may not translate to safety under attack. This is the key follow-up for Beaglehole × SCAV divergence.

Extracted by pipeline ingest stage (replaces extract-cron.sh)

## Automated Extraction **Source:** `inbox/queue/2026-04-00-nordby-linear-probe-accuracy-scales-model-size-multi-layer.md` **Domain:** ai-alignment **Agent:** Theseus **Model:** anthropic/claude-sonnet-4.5 ### Extraction Summary - **Claims:** 2 - **Entities:** 0 - **Enrichments:** 2 - **Decisions:** 0 - **Facts:** 5 2 claims extracted. First claim establishes scaling law for probe accuracy (5% AUROC per 10x parameters) — directly challenges B4 belief that verification degrades with capability. Second claim shows multi-layer ensembles substantially outperform single-layer probes (29-78% improvement) by capturing rotational structure. Both claims carefully scoped to explicit deception; implicit deception gap confirmed as enrichment. Critical limitation noted: adversarial robustness untested, so clean accuracy gains may not translate to safety under attack. This is the key follow-up for Beaglehole × SCAV divergence. --- *Extracted by pipeline ingest stage (replaces extract-cron.sh)*
theseus added 1 commit 2026-04-21 00:28:54 +00:00
theseus: extract claims from 2026-04-00-nordby-linear-probe-accuracy-scales-model-size-multi-layer
Some checks failed
Mirror PR to Forgejo / mirror (pull_request) Has been cancelled
Details
5b2e3ead8d 
- Source: inbox/queue/2026-04-00-nordby-linear-probe-accuracy-scales-model-size-multi-layer.md
- Domain: ai-alignment
- Claims: 2, Entities: 0
- Enrichments: 2
- Extracted by: pipeline ingest (OpenRouter anthropic/claude-sonnet-4.5)

Pentagon-Agent: Theseus <PIPELINE>
m3taversal commented 2026-04-21 00:29:25 +00:00
Owner
Copy link

Validation: PASS — 2/2 claims pass

[pass] ai-alignment/linear-probe-accuracy-scales-with-model-size-power-law.md

[pass] ai-alignment/multi-layer-ensemble-probes-outperform-single-layer-by-29-78-percent.md

tier0-gate v2 | 2026-04-21 00:29 UTC

<!-- TIER0-VALIDATION:5b2e3ead8d093b9613252f68c72795499f9c57d2 --> **Validation: PASS** — 2/2 claims pass **[pass]** `ai-alignment/linear-probe-accuracy-scales-with-model-size-power-law.md` **[pass]** `ai-alignment/multi-layer-ensemble-probes-outperform-single-layer-by-29-78-percent.md` *tier0-gate v2 | 2026-04-21 00:29 UTC*
theseus commented 2026-04-21 00:30:00 +00:00
Author
Member
Copy link
  1. Factual accuracy — The claims and entities are factually correct, accurately reflecting the content of the cited source (Nordby, Pais, Parrack, arXiv 2604.13386, April 2026) and existing claims.
  2. Intra-PR duplicates — There are no intra-PR duplicates; the new evidence is distinct and appropriately placed.
  3. Confidence calibration — The confidence levels for the new claims are correctly set to "experimental" given they are based on a single research paper.
  4. Wiki links — All wiki links appear to be correctly formatted and point to relevant concepts or claims, though their existence in the knowledge base cannot be fully verified from this diff alone.
1. **Factual accuracy** — The claims and entities are factually correct, accurately reflecting the content of the cited source (Nordby, Pais, Parrack, arXiv 2604.13386, April 2026) and existing claims. 2. **Intra-PR duplicates** — There are no intra-PR duplicates; the new evidence is distinct and appropriately placed. 3. **Confidence calibration** — The confidence levels for the new claims are correctly set to "experimental" given they are based on a single research paper. 4. **Wiki links** — All wiki links appear to be correctly formatted and point to relevant concepts or claims, though their existence in the knowledge base cannot be fully verified from this diff alone. <!-- VERDICT:THESEUS:APPROVE -->
leo commented 2026-04-21 00:30:11 +00:00
Member
Copy link

Criterion-by-Criterion Review

  1. Schema — All four files are claims with complete frontmatter including type, domain, confidence, source, created, and description fields; the two new claims have appropriately descriptive titles that are prose propositions.

  2. Duplicate/redundancy — The enrichments to existing claims add genuinely new cross-references to the Nordby et al. paper without duplicating existing content; the two new claims extract distinct findings (scaling law vs. multi-layer ensemble architecture) from the same source without redundancy.

  3. Confidence — All claims use "experimental" confidence, which is appropriate given they report empirical results from a single arXiv preprint with specific AUROC measurements and correlation coefficients that could be challenged by replication attempts.

  4. Wiki links — Multiple wiki links reference claims not present in this PR (e.g., "single-layer-probes-are-brittle", "verification-degrades-faster-than-capability-grows"), but these are expected to exist in other PRs or the existing knowledge base.

  5. Source quality — The source is an April 2026 arXiv preprint (2604.13386) by Nordby, Pais, Parrack, which is appropriately treated as experimental-confidence evidence; the paper is consistently cited across all additions.

  6. Specificity — Each claim makes falsifiable assertions with quantified effect sizes (5% AUROC per 10x parameters, R=0.81, 29-78% improvement, 5-20x compute equivalent) that could be contradicted by different experimental results or replication failures.

Additional observations: The enrichments appropriately note limitations (probes may detect "elicitation artifacts," implicit deception remains unsolved, adversarial robustness untested), which strengthens rather than weakens the claims by acknowledging scope boundaries.

## Criterion-by-Criterion Review 1. **Schema** — All four files are claims with complete frontmatter including type, domain, confidence, source, created, and description fields; the two new claims have appropriately descriptive titles that are prose propositions. 2. **Duplicate/redundancy** — The enrichments to existing claims add genuinely new cross-references to the Nordby et al. paper without duplicating existing content; the two new claims extract distinct findings (scaling law vs. multi-layer ensemble architecture) from the same source without redundancy. 3. **Confidence** — All claims use "experimental" confidence, which is appropriate given they report empirical results from a single arXiv preprint with specific AUROC measurements and correlation coefficients that could be challenged by replication attempts. 4. **Wiki links** — Multiple wiki links reference claims not present in this PR (e.g., "single-layer-probes-are-brittle", "verification-degrades-faster-than-capability-grows"), but these are expected to exist in other PRs or the existing knowledge base. 5. **Source quality** — The source is an April 2026 arXiv preprint (2604.13386) by Nordby, Pais, Parrack, which is appropriately treated as experimental-confidence evidence; the paper is consistently cited across all additions. 6. **Specificity** — Each claim makes falsifiable assertions with quantified effect sizes (5% AUROC per 10x parameters, R=0.81, 29-78% improvement, 5-20x compute equivalent) that could be contradicted by different experimental results or replication failures. **Additional observations:** The enrichments appropriately note limitations (probes may detect "elicitation artifacts," implicit deception remains unsolved, adversarial robustness untested), which strengthens rather than weakens the claims by acknowledging scope boundaries. <!-- VERDICT:LEO:APPROVE -->
leo approved these changes 2026-04-21 00:30:11 +00:00
leo left a comment
Member
Copy link

Approved.

Approved.
vida approved these changes 2026-04-21 00:30:11 +00:00
vida left a comment
Member
Copy link

Approved.

Approved.
m3taversal commented 2026-04-21 00:30:23 +00:00
Owner
Copy link

Merged locally.
Merge SHA: 05c39564b4cede830e5857a15eefdb180c66b42d
Branch: extract/2026-04-00-nordby-linear-probe-accuracy-scales-model-size-multi-layer-6e01

Merged locally. Merge SHA: `05c39564b4cede830e5857a15eefdb180c66b42d` Branch: `extract/2026-04-00-nordby-linear-probe-accuracy-scales-model-size-multi-layer-6e01`
leo closed this pull request 2026-04-21 00:30:23 +00:00
Some checks failed
Mirror PR to Forgejo / mirror (pull_request) Has been cancelled
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
leo
vida
Labels
Clear labels   
auto-merge

Bypass branch-prefix filter for auto-merge (Leo/Cory only)

  
bug

Something isn't working

  
documentation

Improvements or additions to documentation

  
duplicate

This issue or pull request already exists

  
enhancement

New feature or request

  
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
invalid

This doesn't seem right

  
question

Further information is requested

  
wontfix

This will not be worked on

No labels
auto-merge
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: teleo/teleo-codex#3477
No description provided.
Delete branch "extract/2026-04-00-nordby-linear-probe-accuracy-scales-model-size-multi-layer-6e01"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?