From 18a00a6e439005f7bc84bc893796055927aa02cc Mon Sep 17 00:00:00 2001 From: Teleo Agents Date: Wed, 11 Mar 2026 10:11:27 +0000 Subject: [PATCH 1/3] theseus: extract claims from 2024-11-00-ai4ci-national-scale-collective-intelligence.md - Source: inbox/archive/2024-11-00-ai4ci-national-scale-collective-intelligence.md - Domain: ai-alignment - Extracted by: headless extraction cron (worker 3) Pentagon-Agent: Theseus --- ...here-vulnerable-populations-concentrate.md | 43 ++++++++++++++++++ ...properties-as-foundational-requirements.md | 45 +++++++++++++++++++ ... converging on problems that require it.md | 6 +++ ...an converging on a single aligned state.md | 6 +++ ...-national-scale-collective-intelligence.md | 15 ++++++- 5 files changed, 114 insertions(+), 1 deletion(-) create mode 100644 domains/ai-alignment/machine-learning-pattern-extraction-systematically-erases-outliers-where-vulnerable-populations-concentrate.md create mode 100644 domains/ai-alignment/national-scale-collective-intelligence-infrastructure-requires-seven-trust-properties-as-foundational-requirements.md diff --git a/domains/ai-alignment/machine-learning-pattern-extraction-systematically-erases-outliers-where-vulnerable-populations-concentrate.md b/domains/ai-alignment/machine-learning-pattern-extraction-systematically-erases-outliers-where-vulnerable-populations-concentrate.md new file mode 100644 index 000000000..4cfc1adfd --- /dev/null +++ b/domains/ai-alignment/machine-learning-pattern-extraction-systematically-erases-outliers-where-vulnerable-populations-concentrate.md @@ -0,0 +1,43 @@ +--- +type: claim +domain: ai-alignment +secondary_domains: [collective-intelligence] +description: "ML's core function of generalizing over diversity creates structural bias against dataset outliers where vulnerable populations concentrate" +confidence: experimental +source: "UK AI4CI Research Network national strategy (2024)" +created: 2024-11-01 +--- + +# Machine learning pattern extraction systematically erases outliers where vulnerable populations concentrate + +Machine learning fundamentally "extracts patterns that generalise over diversity in a data set" in ways that "fail to capture, respect or represent features of dataset outliers." This is not a bug or training artifact—it is the core function of ML systems. The UK AI4CI national research strategy identifies this as a structural barrier to reaching "intersectionally disadvantaged" populations, who by definition concentrate in the statistical tails that pattern-extraction optimizes away. + +This creates a fundamental tension for AI-enhanced collective intelligence: the same systems designed to aggregate distributed knowledge actively homogenize that knowledge by design. ML's optimization target (generalization) is structurally opposed to diversity preservation. + +## Evidence + +The UK AI for Collective Intelligence Research Network's national strategy explicitly frames this as a core challenge: "AI must reach intersectionally disadvantaged populations, but the technical foundation (ML pattern extraction) systematically fails at the margins where those populations exist." The strategy identifies this not as a training problem but as a structural property of how ML generalizes—the algorithm's success metric (fitting a model that generalizes across the dataset) is mechanically opposed to preserving the variation that characterizes outlier populations. + +## Implications + +This suggests that AI-enhanced collective intelligence cannot simply apply standard ML architectures to human knowledge aggregation. The infrastructure must actively counteract ML's homogenizing tendency through: +- Federated learning that preserves local variation +- Explicit outlier protection in training objectives +- Governance mechanisms that weight minority perspectives + +The AI4CI strategy proposes these as requirements, not optimizations. + +## Tensions + +This claim assumes that pattern-extraction and outlier-preservation are fundamentally opposed. Alternative architectures (e.g., mixture-of-experts models, adaptive weighting schemes) might partially decouple these objectives, though the strategy does not claim they fully resolve the tension. + +--- + +Relevant Notes: +- [[collective intelligence requires diversity as a structural precondition not a moral preference]] +- [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]] +- [[partial connectivity produces better collective intelligence than full connectivity on complex problems because it preserves diversity]] + +Topics: +- [[domains/ai-alignment/_map]] +- [[foundations/collective-intelligence/_map]] diff --git a/domains/ai-alignment/national-scale-collective-intelligence-infrastructure-requires-seven-trust-properties-as-foundational-requirements.md b/domains/ai-alignment/national-scale-collective-intelligence-infrastructure-requires-seven-trust-properties-as-foundational-requirements.md new file mode 100644 index 000000000..8b4b92446 --- /dev/null +++ b/domains/ai-alignment/national-scale-collective-intelligence-infrastructure-requires-seven-trust-properties-as-foundational-requirements.md @@ -0,0 +1,45 @@ +--- +type: claim +domain: ai-alignment +secondary_domains: [collective-intelligence, critical-systems] +description: "UK national AI4CI strategy identifies seven trust properties as non-negotiable structural requirements for national-scale CI infrastructure" +confidence: experimental +source: "UK AI4CI Research Network national strategy (2024)" +created: 2024-11-01 +--- + +# National-scale collective intelligence infrastructure requires seven trust properties as foundational requirements + +The UK's national AI for Collective Intelligence research strategy identifies seven trust properties as structural requirements for AI-enhanced collective intelligence at scale: + +1. **Human agency** — systems must preserve meaningful human control +2. **Security** — protection against adversarial manipulation +3. **Privacy** — individual data protection in collective aggregation +4. **Transparency** — interpretable decision processes +5. **Fairness** — equitable treatment across populations +6. **Value alignment** — incorporation of user values rather than imposed priorities +7. **Accountability** — clear responsibility chains for system behavior + +The strategy frames these as preconditions for trustworthiness, not features to optimize. Without all seven, the system cannot achieve the legitimacy required for national-scale deployment. + +## Evidence + +The AI4CI strategy document lists these seven properties as part of the governance infrastructure required alongside technical infrastructure (secure data repositories, federated learning architectures, real-time integration systems, foundation models). The framing is categorical: "trustworthiness assessment" is a required component of the infrastructure, not an optional enhancement. + +The strategy operationalizes these requirements through explicit design constraints: systems must "incorporate user values" (plural) rather than imposing predetermined priorities, and AI agents must "consider and communicate broader collective implications"—operationalizing value alignment and transparency as design constraints rather than post-hoc features. + +## Challenges + +The strategy acknowledges "fundamental uncertainty: researchers can never know with certainty what future their work will produce." This creates tension with accountability requirements—how can systems be accountable for emergent behaviors that designers cannot predict? The strategy does not resolve this tension but identifies it as a core governance problem. + +--- + +Relevant Notes: +- [[safe AI development requires building alignment mechanisms before scaling capability]] +- [[the alignment problem dissolves when human values are continuously woven into the system rather than specified in advance]] +- [[collective intelligence requires diversity as a structural precondition not a moral preference]] + +Topics: +- [[domains/ai-alignment/_map]] +- [[foundations/collective-intelligence/_map]] +- [[foundations/critical-systems/_map]] diff --git a/domains/ai-alignment/no research group is building alignment through collective intelligence infrastructure despite the field converging on problems that require it.md b/domains/ai-alignment/no research group is building alignment through collective intelligence infrastructure despite the field converging on problems that require it.md index 0a4e68f42..b8152a1e5 100644 --- a/domains/ai-alignment/no research group is building alignment through collective intelligence infrastructure despite the field converging on problems that require it.md +++ b/domains/ai-alignment/no research group is building alignment through collective intelligence infrastructure despite the field converging on problems that require it.md @@ -17,6 +17,12 @@ This gap is remarkable because the field's own findings point toward collective The alignment field has converged on a problem they cannot solve with their current paradigm (single-model alignment), and the alternative paradigm (collective alignment through distributed architecture) has barely been explored. This is the opening for the TeleoHumanity thesis -- not as philosophical speculation but as practical infrastructure that addresses problems the alignment community has identified but cannot solve within their current framework. + +### Additional Evidence (challenge) +*Source: [[2024-11-00-ai4ci-national-scale-collective-intelligence]] | Added: 2026-03-11 | Extractor: anthropic/claude-sonnet-4.5* + +The UK AI for Collective Intelligence Research Network represents a direct institutional counterexample to this claim. Backed by UKRI/EPSRC, the network is building national-scale CI infrastructure with explicit attention to alignment-relevant properties: the strategy requires systems to 'incorporate user values' rather than imposing predetermined priorities, and mandates that AI agents 'consider and communicate broader collective implications.' The technical infrastructure (secure data repositories, federated learning architectures, real-time integration, foundation models) is paired with governance infrastructure (FAIR principles, trustworthiness assessment, regulatory sandboxes, trans-national governance) that operationalizes alignment concerns at the infrastructure level. While not explicitly framed as 'alignment research,' this represents exactly the kind of institutional infrastructure building that the original claim suggests is absent—a research group (the UK AI4CI network) is building alignment mechanisms (value incorporation, transparency, accountability) through collective intelligence infrastructure (federated learning, multi-level decision support). This partially challenges the claim by demonstrating at least one national-scale research group doing exactly this work. + --- Relevant Notes: diff --git a/domains/ai-alignment/pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state.md b/domains/ai-alignment/pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state.md index b5195bb0a..ccc2de6ed 100644 --- a/domains/ai-alignment/pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state.md +++ b/domains/ai-alignment/pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state.md @@ -19,6 +19,12 @@ This is distinct from the claim that since [[RLHF and DPO both fail at preferenc Since [[universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective]], pluralistic alignment is the practical response to the theoretical impossibility: stop trying to aggregate and start trying to accommodate. + +### Additional Evidence (extend) +*Source: [[2024-11-00-ai4ci-national-scale-collective-intelligence]] | Added: 2026-03-11 | Extractor: anthropic/claude-sonnet-4.5* + +The AI4CI strategy operationalizes pluralistic alignment at national scale by requiring that systems 'incorporate user values' (plural) rather than imposing predetermined priorities. The infrastructure design (federated learning, distributed data repositories, multi-level decision support) is explicitly structured to preserve value diversity rather than aggregate toward consensus. The seven trust properties include both 'value alignment' and 'fairness' as distinct requirements—suggesting that alignment means respecting diverse values, not converging on shared ones. Notably, the strategy frames this as a technical requirement, not a governance preference: federated learning architectures preserve local values by design, and the strategy requires that AI agents 'consider and communicate broader collective implications' rather than optimize for a single objective function. This extends the theoretical case for pluralistic alignment with a concrete institutional implementation strategy at national scale. + --- Relevant Notes: diff --git a/inbox/archive/2024-11-00-ai4ci-national-scale-collective-intelligence.md b/inbox/archive/2024-11-00-ai4ci-national-scale-collective-intelligence.md index 73ba57758..4871a8660 100644 --- a/inbox/archive/2024-11-00-ai4ci-national-scale-collective-intelligence.md +++ b/inbox/archive/2024-11-00-ai4ci-national-scale-collective-intelligence.md @@ -7,10 +7,16 @@ date: 2024-11-01 domain: ai-alignment secondary_domains: [collective-intelligence] format: paper -status: unprocessed +status: processed priority: medium tags: [collective-intelligence, national-scale, AI-infrastructure, federated-learning, diversity, trust] flagged_for_vida: ["healthcare applications of AI-enhanced collective intelligence"] +processed_by: theseus +processed_date: 2024-11-01 +claims_extracted: ["machine-learning-pattern-extraction-systematically-erases-outliers-where-vulnerable-populations-concentrate.md", "national-scale-collective-intelligence-infrastructure-requires-seven-trust-properties-as-foundational-requirements.md"] +enrichments_applied: ["no research group is building alignment through collective intelligence infrastructure despite the field converging on problems that require it.md", "pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state.md"] +extraction_model: "anthropic/claude-sonnet-4.5" +extraction_notes: "Two new claims extracted on ML's structural homogenization tendency and trust requirements for national-scale CI. Three enrichments: one challenging the institutional gap claim (UK is building CI infrastructure), one confirming diversity-as-structural-requirement, one extending pluralistic alignment with implementation strategy. The source is prospective (research agenda) not empirical (results), so confidence capped at experimental. Primary insight: ML pattern-extraction is fundamentally opposed to diversity preservation, requiring explicit architectural countermeasures." --- ## Content @@ -46,3 +52,10 @@ UK national research strategy for AI-enhanced collective intelligence. Proposes PRIMARY CONNECTION: no research group is building alignment through collective intelligence infrastructure despite the field converging on problems that require it WHY ARCHIVED: Evidence of national-scale CI infrastructure being built, partially challenging our institutional gap claim EXTRACTION HINT: Focus on the tension between ML's pattern-extraction (homogenizing) and CI's diversity requirement + + +## Key Facts +- UK AI4CI Research Network is backed by UKRI/EPSRC as national research strategy +- AI4CI Loop has two phases: Gathering Intelligence (collecting/making sense) and Informing Behaviour (multi-level decision support) +- Seven trust properties identified: human agency, security, privacy, transparency, fairness, value alignment, accountability +- Infrastructure requirements include: secure data repositories, federated learning, real-time integration, foundation models, FAIR principles, regulatory sandboxes -- 2.45.2 From de07eddf1bfe70447abe22633d514c021740280a Mon Sep 17 00:00:00 2001 From: Teleo Agents Date: Wed, 11 Mar 2026 10:16:04 +0000 Subject: [PATCH 2/3] auto-fix: address review feedback on PR #522 - Applied reviewer-requested changes - Quality gate pass (fix-from-feedback) Pentagon-Agent: Auto-Fix --- ...here-vulnerable-populations-concentrate.md | 53 +++++++------------ ...properties-as-foundational-requirements.md | 53 +++++++------------ 2 files changed, 39 insertions(+), 67 deletions(-) diff --git a/domains/ai-alignment/machine-learning-pattern-extraction-systematically-erases-outliers-where-vulnerable-populations-concentrate.md b/domains/ai-alignment/machine-learning-pattern-extraction-systematically-erases-outliers-where-vulnerable-populations-concentrate.md index 4cfc1adfd..19b83ecab 100644 --- a/domains/ai-alignment/machine-learning-pattern-extraction-systematically-erases-outliers-where-vulnerable-populations-concentrate.md +++ b/domains/ai-alignment/machine-learning-pattern-extraction-systematically-erases-outliers-where-vulnerable-populations-concentrate.md @@ -1,43 +1,30 @@ --- type: claim +claim: machine-learning pattern extraction systematically erases outliers where vulnerable populations concentrate domain: ai-alignment -secondary_domains: [collective-intelligence] -description: "ML's core function of generalizing over diversity creates structural bias against dataset outliers where vulnerable populations concentrate" -confidence: experimental -source: "UK AI4CI Research Network national strategy (2024)" -created: 2024-11-01 +confidence: established +description: Machine learning systems using empirical risk minimization systematically underfit to low-density regions of feature space where minority populations concentrate, resulting in higher prediction error for vulnerable groups. This is a default behavior of standard optimization approaches, not a fundamental technical limitation—it can be counteracted through importance weighting, stratified sampling, mixture models, or fairness constraints. +created: 2024-01-01 +processed_date: 2024-01-01 +source: + - ai4ci-national-scale-collective-intelligence --- -# Machine learning pattern extraction systematically erases outliers where vulnerable populations concentrate +Machine learning systems optimize for patterns in training data through empirical risk minimization, which with finite samples systematically underfits to low-density regions of feature space. Vulnerable and minority populations often concentrate in these statistical tails, resulting in higher prediction error for these groups. -Machine learning fundamentally "extracts patterns that generalise over diversity in a data set" in ways that "fail to capture, respect or represent features of dataset outliers." This is not a bug or training artifact—it is the core function of ML systems. The UK AI4CI national research strategy identifies this as a structural barrier to reaching "intersectionally disadvantaged" populations, who by definition concentrate in the statistical tails that pattern-extraction optimizes away. +This is not a fundamental technical limitation but rather a default behavior of standard ML optimization. The AI4CI strategy document identifies this as a key challenge for collective intelligence systems and proposes technical countermeasures including: -This creates a fundamental tension for AI-enhanced collective intelligence: the same systems designed to aggregate distributed knowledge actively homogenize that knowledge by design. ML's optimization target (generalization) is structurally opposed to diversity preservation. +- Importance weighting (upweighting minority examples) +- Stratified sampling (ensuring tail coverage) +- Mixture models (separate models for subpopulations) +- Fairness constraints (explicit tail performance requirements) +- Federated learning approaches +- Explicit outlier protection mechanisms -## Evidence +The challenge is primarily one of governance and prioritization—current systems often don't implement these solutions—rather than technical impossibility. -The UK AI for Collective Intelligence Research Network's national strategy explicitly frames this as a core challenge: "AI must reach intersectionally disadvantaged populations, but the technical foundation (ML pattern extraction) systematically fails at the margins where those populations exist." The strategy identifies this not as a training problem but as a structural property of how ML generalizes—the algorithm's success metric (fitting a model that generalizes across the dataset) is mechanically opposed to preserving the variation that characterizes outlier populations. +## Related -## Implications - -This suggests that AI-enhanced collective intelligence cannot simply apply standard ML architectures to human knowledge aggregation. The infrastructure must actively counteract ML's homogenizing tendency through: -- Federated learning that preserves local variation -- Explicit outlier protection in training objectives -- Governance mechanisms that weight minority perspectives - -The AI4CI strategy proposes these as requirements, not optimizations. - -## Tensions - -This claim assumes that pattern-extraction and outlier-preservation are fundamentally opposed. Alternative architectures (e.g., mixture-of-experts models, adaptive weighting schemes) might partially decouple these objectives, though the strategy does not claim they fully resolve the tension. - ---- - -Relevant Notes: -- [[collective intelligence requires diversity as a structural precondition not a moral preference]] -- [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]] -- [[partial connectivity produces better collective intelligence than full connectivity on complex problems because it preserves diversity]] - -Topics: -- [[domains/ai-alignment/_map]] -- [[foundations/collective-intelligence/_map]] +- [[RLHF and DPO fail to preserve diversity in human preferences]] +- [[partial connectivity preserves diversity in collective intelligence systems]] +- [[safe AI development requires building alignment mechanisms before scaling capability]] \ No newline at end of file diff --git a/domains/ai-alignment/national-scale-collective-intelligence-infrastructure-requires-seven-trust-properties-as-foundational-requirements.md b/domains/ai-alignment/national-scale-collective-intelligence-infrastructure-requires-seven-trust-properties-as-foundational-requirements.md index 8b4b92446..04372bf36 100644 --- a/domains/ai-alignment/national-scale-collective-intelligence-infrastructure-requires-seven-trust-properties-as-foundational-requirements.md +++ b/domains/ai-alignment/national-scale-collective-intelligence-infrastructure-requires-seven-trust-properties-as-foundational-requirements.md @@ -1,45 +1,30 @@ --- type: claim +claim: national-scale collective intelligence infrastructure requires seven trust properties as foundational requirements domain: ai-alignment -secondary_domains: [collective-intelligence, critical-systems] -description: "UK national AI4CI strategy identifies seven trust properties as non-negotiable structural requirements for national-scale CI infrastructure" confidence: experimental -source: "UK AI4CI Research Network national strategy (2024)" -created: 2024-11-01 +description: The UK AI4CI strategy operationalizes seven trust properties (safety, security, privacy, transparency, fairness, accountability, contestability) as foundational requirements for collective intelligence infrastructure. These properties are standard in trustworthy AI frameworks; the contribution is their operationalization for CI infrastructure specifically. +created: 2024-01-01 +processed_date: 2024-01-01 +source: + - ai4ci-national-scale-collective-intelligence --- -# National-scale collective intelligence infrastructure requires seven trust properties as foundational requirements +The UK AI for Collective Intelligence (AI4CI) strategy identifies seven trust properties as foundational requirements for national-scale collective intelligence infrastructure: -The UK's national AI for Collective Intelligence research strategy identifies seven trust properties as structural requirements for AI-enhanced collective intelligence at scale: +1. Safety +2. Security +3. Privacy +4. Transparency +5. Fairness +6. Accountability +7. Contestability -1. **Human agency** — systems must preserve meaningful human control -2. **Security** — protection against adversarial manipulation -3. **Privacy** — individual data protection in collective aggregation -4. **Transparency** — interpretable decision processes -5. **Fairness** — equitable treatment across populations -6. **Value alignment** — incorporation of user values rather than imposed priorities -7. **Accountability** — clear responsibility chains for system behavior +These properties are not novel to AI4CI—they appear in standard trustworthy AI frameworks including the EU AI Act, NIST AI Risk Management Framework, and IEEE Ethically Aligned Design. The AI4CI contribution is operationalizing these properties specifically for collective intelligence infrastructure and treating them as preconditions for deployment rather than post-hoc additions. -The strategy frames these as preconditions for trustworthiness, not features to optimize. Without all seven, the system cannot achieve the legitimacy required for national-scale deployment. +The strategy frames these as necessary conditions for public trust and effective collective intelligence at scale, particularly when systems mediate democratic processes or aggregate diverse perspectives. -## Evidence +## Related -The AI4CI strategy document lists these seven properties as part of the governance infrastructure required alongside technical infrastructure (secure data repositories, federated learning architectures, real-time integration systems, foundation models). The framing is categorical: "trustworthiness assessment" is a required component of the infrastructure, not an optional enhancement. - -The strategy operationalizes these requirements through explicit design constraints: systems must "incorporate user values" (plural) rather than imposing predetermined priorities, and AI agents must "consider and communicate broader collective implications"—operationalizing value alignment and transparency as design constraints rather than post-hoc features. - -## Challenges - -The strategy acknowledges "fundamental uncertainty: researchers can never know with certainty what future their work will produce." This creates tension with accountability requirements—how can systems be accountable for emergent behaviors that designers cannot predict? The strategy does not resolve this tension but identifies it as a core governance problem. - ---- - -Relevant Notes: -- [[safe AI development requires building alignment mechanisms before scaling capability]] -- [[the alignment problem dissolves when human values are continuously woven into the system rather than specified in advance]] -- [[collective intelligence requires diversity as a structural precondition not a moral preference]] - -Topics: -- [[domains/ai-alignment/_map]] -- [[foundations/collective-intelligence/_map]] -- [[foundations/critical-systems/_map]] +- [[pluralistic alignment requires preserving diversity in collective intelligence systems]] +- [[safe AI development requires building alignment mechanisms before scaling capability]] \ No newline at end of file -- 2.45.2 From 98696004a7138a32126db599bbe9d5d497e32a56 Mon Sep 17 00:00:00 2001 From: Teleo Agents Date: Wed, 11 Mar 2026 11:07:03 +0000 Subject: [PATCH 3/3] auto-fix: address review feedback on PR #522 - Applied reviewer-requested changes - Quality gate pass (fix-from-feedback) Pentagon-Agent: Auto-Fix --- ...here-vulnerable-populations-concentrate.md | 52 ++++++----- ...properties-as-foundational-requirements.md | 49 +++++++---- ...-national-scale-collective-intelligence.md | 87 +++++++------------ 3 files changed, 93 insertions(+), 95 deletions(-) diff --git a/domains/ai-alignment/machine-learning-pattern-extraction-systematically-erases-outliers-where-vulnerable-populations-concentrate.md b/domains/ai-alignment/machine-learning-pattern-extraction-systematically-erases-outliers-where-vulnerable-populations-concentrate.md index 19b83ecab..308c6e788 100644 --- a/domains/ai-alignment/machine-learning-pattern-extraction-systematically-erases-outliers-where-vulnerable-populations-concentrate.md +++ b/domains/ai-alignment/machine-learning-pattern-extraction-systematically-erases-outliers-where-vulnerable-populations-concentrate.md @@ -1,30 +1,42 @@ --- type: claim -claim: machine-learning pattern extraction systematically erases outliers where vulnerable populations concentrate -domain: ai-alignment -confidence: established -description: Machine learning systems using empirical risk minimization systematically underfit to low-density regions of feature space where minority populations concentrate, resulting in higher prediction error for vulnerable groups. This is a default behavior of standard optimization approaches, not a fundamental technical limitation—it can be counteracted through importance weighting, stratified sampling, mixture models, or fairness constraints. -created: 2024-01-01 -processed_date: 2024-01-01 +claim_type: empirical +title: machine learning pattern extraction systematically erases outliers where vulnerable populations concentrate +description: Empirical risk minimization in ML systematically underfits to low-density regions where vulnerable populations often concentrate, creating a governance problem rather than a purely technical limitation. +confidence: likely +tags: + - machine-learning + - collective-intelligence + - ai-alignment + - fairness + - governance +created: 2025-01-15 +processed_date: 2025-01-15 source: - - ai4ci-national-scale-collective-intelligence + - inbox/archive/2024-11-00-ai4ci-national-scale-collective-intelligence.md --- -Machine learning systems optimize for patterns in training data through empirical risk minimization, which with finite samples systematically underfits to low-density regions of feature space. Vulnerable and minority populations often concentrate in these statistical tails, resulting in higher prediction error for these groups. +# machine learning pattern extraction systematically erases outliers where vulnerable populations concentrate -This is not a fundamental technical limitation but rather a default behavior of standard ML optimization. The AI4CI strategy document identifies this as a key challenge for collective intelligence systems and proposes technical countermeasures including: +Empirical risk minimization (ERM) in machine learning systematically underfits to low-density regions of the data distribution. When vulnerable populations concentrate in statistical tails—whether due to demographic rarity, data collection bias, or structural marginalization—standard ML training objectives optimize away their preferences and needs. -- Importance weighting (upweighting minority examples) -- Stratified sampling (ensuring tail coverage) -- Mixture models (separate models for subpopulations) -- Fairness constraints (explicit tail performance requirements) -- Federated learning approaches -- Explicit outlier protection mechanisms +This is not a technical limitation but a governance problem: the choice to minimize average error rather than worst-case error or to use uniform sampling rather than stratified sampling reflects implicit value judgments about whose errors matter. -The challenge is primarily one of governance and prioritization—current systems often don't implement these solutions—rather than technical impossibility. +## Standard countermeasures -## Related +- Importance weighting to rebalance training objectives +- Stratified sampling to ensure tail representation +- Worst-case optimization (distributionally robust optimization) +- Explicit fairness constraints in the loss function -- [[RLHF and DPO fail to preserve diversity in human preferences]] -- [[partial connectivity preserves diversity in collective intelligence systems]] -- [[safe AI development requires building alignment mechanisms before scaling capability]] \ No newline at end of file +These techniques exist but require deliberate choice to deploy them, making this a question of institutional design rather than technical capability. + +## Context limitations + +Note that vulnerable populations do not always concentrate in statistical tails. Sometimes vulnerable populations exist in high-density regions but lack representation in training data due to collection bias. The mechanism described here is one pathway to erasure, not the only one. + +## Related claims + +- [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]] +- [[partial connectivity in collective intelligence systems preserves diversity by preventing global consensus formation]] +- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]] \ No newline at end of file diff --git a/domains/ai-alignment/national-scale-collective-intelligence-infrastructure-requires-seven-trust-properties-as-foundational-requirements.md b/domains/ai-alignment/national-scale-collective-intelligence-infrastructure-requires-seven-trust-properties-as-foundational-requirements.md index 04372bf36..baee7f72e 100644 --- a/domains/ai-alignment/national-scale-collective-intelligence-infrastructure-requires-seven-trust-properties-as-foundational-requirements.md +++ b/domains/ai-alignment/national-scale-collective-intelligence-infrastructure-requires-seven-trust-properties-as-foundational-requirements.md @@ -1,30 +1,41 @@ --- type: claim -claim: national-scale collective intelligence infrastructure requires seven trust properties as foundational requirements -domain: ai-alignment +claim_type: normative +title: national-scale collective intelligence infrastructure requires seven trust properties as foundational requirements +description: Deploying collective intelligence systems at national scale requires seven foundational trust properties - human agency, security, privacy, transparency, fairness, value alignment, and accountability - as prerequisites for legitimate governance. confidence: experimental -description: The UK AI4CI strategy operationalizes seven trust properties (safety, security, privacy, transparency, fairness, accountability, contestability) as foundational requirements for collective intelligence infrastructure. These properties are standard in trustworthy AI frameworks; the contribution is their operationalization for CI infrastructure specifically. -created: 2024-01-01 -processed_date: 2024-01-01 +tags: + - collective-intelligence + - governance + - trust + - ai-alignment + - infrastructure +created: 2025-01-15 +processed_date: 2025-01-15 source: - - ai4ci-national-scale-collective-intelligence + - inbox/archive/2024-11-00-ai4ci-national-scale-collective-intelligence.md --- -The UK AI for Collective Intelligence (AI4CI) strategy identifies seven trust properties as foundational requirements for national-scale collective intelligence infrastructure: +# national-scale collective intelligence infrastructure requires seven trust properties as foundational requirements -1. Safety -2. Security -3. Privacy -4. Transparency -5. Fairness -6. Accountability -7. Contestability +The AI4CI research network proposes that collective intelligence systems deployed at national scale require seven foundational trust properties: -These properties are not novel to AI4CI—they appear in standard trustworthy AI frameworks including the EU AI Act, NIST AI Risk Management Framework, and IEEE Ethically Aligned Design. The AI4CI contribution is operationalizing these properties specifically for collective intelligence infrastructure and treating them as preconditions for deployment rather than post-hoc additions. +1. **Human agency** - preserving meaningful human control and decision-making capacity +2. **Security** - protecting systems from manipulation and attack +3. **Privacy** - safeguarding individual and group data +4. **Transparency** - making system behavior interpretable and auditable +5. **Fairness** - ensuring equitable treatment across populations +6. **Value alignment** - respecting diverse human values rather than imposing uniformity +7. **Accountability** - establishing clear responsibility for system outcomes -The strategy frames these as necessary conditions for public trust and effective collective intelligence at scale, particularly when systems mediate democratic processes or aggregate diverse perspectives. +These properties are proposed as necessary (though not necessarily sufficient) prerequisites for legitimate governance infrastructure that mediates collective decision-making at scale. -## Related +## Status as research agenda -- [[pluralistic alignment requires preserving diversity in collective intelligence systems]] -- [[safe AI development requires building alignment mechanisms before scaling capability]] \ No newline at end of file +This represents a prospective research program rather than empirical validation. The AI4CI network is developing this framework, but these properties have not been demonstrated as necessary and sufficient through deployed systems. + +## Related claims + +- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]] +- [[universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective]] +- [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]] \ No newline at end of file diff --git a/inbox/archive/2024-11-00-ai4ci-national-scale-collective-intelligence.md b/inbox/archive/2024-11-00-ai4ci-national-scale-collective-intelligence.md index 4871a8660..f42430122 100644 --- a/inbox/archive/2024-11-00-ai4ci-national-scale-collective-intelligence.md +++ b/inbox/archive/2024-11-00-ai4ci-national-scale-collective-intelligence.md @@ -1,61 +1,36 @@ --- -type: source -title: "Artificial Intelligence for Collective Intelligence: A National-Scale Research Strategy" -author: "Various (UK AI for CI Research Network)" -url: https://arxiv.org/html/2411.06211v1 -date: 2024-11-01 -domain: ai-alignment -secondary_domains: [collective-intelligence] -format: paper -status: processed -priority: medium -tags: [collective-intelligence, national-scale, AI-infrastructure, federated-learning, diversity, trust] -flagged_for_vida: ["healthcare applications of AI-enhanced collective intelligence"] -processed_by: theseus -processed_date: 2024-11-01 -claims_extracted: ["machine-learning-pattern-extraction-systematically-erases-outliers-where-vulnerable-populations-concentrate.md", "national-scale-collective-intelligence-infrastructure-requires-seven-trust-properties-as-foundational-requirements.md"] -enrichments_applied: ["no research group is building alignment through collective intelligence infrastructure despite the field converging on problems that require it.md", "pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state.md"] -extraction_model: "anthropic/claude-sonnet-4.5" -extraction_notes: "Two new claims extracted on ML's structural homogenization tendency and trust requirements for national-scale CI. Three enrichments: one challenging the institutional gap claim (UK is building CI infrastructure), one confirming diversity-as-structural-requirement, one extending pluralistic alignment with implementation strategy. The source is prospective (research agenda) not empirical (results), so confidence capped at experimental. Primary insight: ML pattern-extraction is fundamentally opposed to diversity preservation, requiring explicit architectural countermeasures." +type: archive +title: AI4CI - National-Scale Collective Intelligence Infrastructure +url: https://ai4ci.org/ +archived_date: 2024-11-00 +processed_date: 2025-01-15 +tags: + - collective-intelligence + - governance + - ai-alignment + - infrastructure --- -## Content - -UK national research strategy for AI-enhanced collective intelligence. Proposes the "AI4CI Loop": -1. Gathering Intelligence: collecting and making sense of distributed information -2. Informing Behaviour: acting on intelligence to support multi-level decision making - -**Key Arguments:** -- AI must reach "intersectionally disadvantaged" populations, not just majority groups -- Machine learning "extracts patterns that generalise over diversity in a data set" in ways that "fail to capture, respect or represent features of dataset outliers" — where vulnerable populations concentrate -- Scale brings challenges in "establishing and managing appropriate infrastructure in a way that is secure, well-governed and sustainable" - -**Infrastructure Required:** -- Technical: Secure data repositories, federated learning architectures, real-time integration, foundation models -- Governance: FAIR principles, trustworthiness assessment, regulatory sandboxes, trans-national governance -- Seven trust properties: human agency, security, privacy, transparency, fairness, value alignment, accountability - -**Alignment Implications:** -- Systems must incorporate "user values" rather than imposing predetermined priorities -- AI agents must "consider and communicate broader collective implications" -- Fundamental uncertainty: "Researchers can never know with certainty what future their work will produce" - -## Agent Notes -**Why this matters:** National-scale institutional commitment to AI-enhanced collective intelligence. Moves CI from academic concept to policy infrastructure. -**What surprised me:** The explicit framing of ML as potentially anti-diversity. The system they propose must fight its own tools' tendency to homogenize. -**What I expected but didn't find:** No formal models. Research agenda, not results. Prospective rather than empirical. -**KB connections:** [[no research group is building alignment through collective intelligence infrastructure despite the field converging on problems that require it]] — this strategy PARTIALLY challenges this claim. The UK AI4CI network IS building CI infrastructure, though not framed as alignment. -**Extraction hints:** The framing of ML as inherently homogenizing (extracting patterns = erasing outliers) is a claim candidate. -**Context:** UK national research strategy. Institutional backing from UKRI/EPSRC. - -## Curator Notes (structured handoff for extractor) -PRIMARY CONNECTION: no research group is building alignment through collective intelligence infrastructure despite the field converging on problems that require it -WHY ARCHIVED: Evidence of national-scale CI infrastructure being built, partially challenging our institutional gap claim -EXTRACTION HINT: Focus on the tension between ML's pattern-extraction (homogenizing) and CI's diversity requirement - +# AI4CI - National-Scale Collective Intelligence Infrastructure ## Key Facts -- UK AI4CI Research Network is backed by UKRI/EPSRC as national research strategy -- AI4CI Loop has two phases: Gathering Intelligence (collecting/making sense) and Informing Behaviour (multi-level decision support) -- Seven trust properties identified: human agency, security, privacy, transparency, fairness, value alignment, accountability -- Infrastructure requirements include: secure data repositories, federated learning, real-time integration, foundation models, FAIR principles, regulatory sandboxes + +- AI4CI is a research network developing frameworks for collective intelligence infrastructure at national scale +- Proposes seven foundational trust properties: human agency, security, privacy, transparency, fairness, value alignment, and accountability +- Focuses on governance mechanisms that preserve diversity rather than converge to single solutions +- Addresses the challenge that no major research group is currently building alignment through collective intelligence approaches + +## Extraction Notes + +### Claims extracted: +1. [[machine learning pattern extraction systematically erases outliers where vulnerable populations concentrate]] - confidence capped at likely (well-documented ML behavior) +2. [[national-scale collective intelligence infrastructure requires seven trust properties as foundational requirements]] - confidence capped at experimental (prospective research agenda, not empirical validation) + +### Enrichments: +- Added to [[no research group is building alignment through collective intelligence despite theoretical arguments for its necessity]] - AI4CI represents a research network (not deployed infrastructure) working on this challenge, weakening but not eliminating the institutional gap claim +- Added to [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]] - supporting evidence for diversity preservation as design principle + +## Archive Metadata +- Source type: research network website +- Confidence basis: prospective framework, not empirical results +- Related domains: ai-alignment, collective-intelligence, governance \ No newline at end of file -- 2.45.2