From d6af5bcbde851968442554a4afbd066340aa060e Mon Sep 17 00:00:00 2001 From: m3taversal Date: Sat, 28 Mar 2026 21:07:56 +0000 Subject: [PATCH] theseus: add schema change protocol v2 with full coverage Incorporates review feedback from Rhea, Argus, and Ganymede on PR #2072: - Added pipeline.db tables to producer/consumer map - Added API response shapes (endpoints + graph-data.json) - Added systemd service configuration as schema surface - Added DB column and API shape rules to backward compatibility - Clarified that DB/API changes document in PR body (not just schemas/) - Added legacy alias verification request for Epimetheus Replaces closed PR #2072. Pentagon-Agent: Theseus <24DE7DA0-E4D5-4023-B1A2-3F736AFF4EEE> --- ops/schema-change-protocol.md | 126 ++++++++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 ops/schema-change-protocol.md diff --git a/ops/schema-change-protocol.md b/ops/schema-change-protocol.md new file mode 100644 index 00000000..cc964560 --- /dev/null +++ b/ops/schema-change-protocol.md @@ -0,0 +1,126 @@ +# Schema Change Protocol + +When any agent changes a file format, database table, API response shape, or service configuration that other agents read or consume, those agents need to know before their next session. This protocol prevents silent breakage. + +## The Rule + +**Any PR that changes a schema must:** + +1. **Update the schema spec** in `schemas/` (for file formats) or document the change in the PR (for DB tables, API responses, service configs) +2. **Tag all consumers** — list which agents and scripts read this format (see map below) +3. **Include a migration note** — what happens to existing data? (backfill on edit, ignore old files, or batch migration) +4. **State backward compatibility** — can old-format data still be parsed? If not, the PR must include the migration + +## What Counts as a Schema Change + +| Change Type | Example | Requires Protocol? | +|---|---|---| +| New required field | Adding `attribution` block to claims | Yes | +| New optional field | Adding `tags[]` to sources | Yes (consumers may need to handle it) | +| Field rename | `source_type` to `format` | Yes | +| Enum value added | New confidence level | Yes | +| Enum value removed | Dropping a domain name | Yes — migration required | +| Field type change | `source` from string to object | Yes — breaking change | +| Body format change | New required section in claim body | Yes | +| Pipeline parsing change | Regex update in `extract-graph-data.py` | Yes | +| DB column add/rename/drop | Adding column to `prs` table | Yes | +| DB table create/drop | New `response_audit` table | Yes | +| API response shape change | Adding field to `/api/alerts` JSON | Yes | +| systemd service config | New `ReadWritePaths` or port change | Yes | + +**Not a schema change:** Adding a new claim, entity, or source file that follows the existing format. Normal PR workflow applies. + +## Producer/Consumer Map + +### File Formats + +| Format | Schema | Producers | Consumers | Pipeline | +|---|---|---|---|---| +| Claim | `schemas/claim.md` | All proposers (Rio, Clay, Theseus, Vida, Astra) | Leo (eval), all agents (beliefs), visitors | `extract-graph-data.py` | +| Source | `schemas/source.md` | All proposers, Epimetheus (pipeline) | Proposers (extraction), Epimetheus (pipeline) | `extract-cron.sh` | +| Entity | `schemas/entity.md` | Domain agents | All agents (references), visitors | `extract-graph-data.py` | +| Belief | `schemas/belief.md` | Each agent (own file) | Leo (review), other agents (cross-ref) | None currently | +| Position | `schemas/position.md` | Each agent (own file) | Leo (review), visitors | None currently | +| Conviction | `schemas/conviction.md` | Cory only | All agents, visitors | `extract-graph-data.py` | +| Divergence | `schemas/divergence.md` | Any agent | All agents, visitors | None currently | +| Musing | `schemas/musing.md` | Each agent (own folder) | That agent only | None | +| Sector | `schemas/sector.md` | Domain agents | All agents, visitors | None currently | +| Contribution weights | `schemas/contribution-weights.yaml` | Cory / Leo | `contributors.json` build | Build script | +| Graph data | (derived) | `extract-graph-data.py` | Oberon (frontend), system prompts | Auto-generated | + +### Database Tables (pipeline.db) + +| Table | Producer | Consumers | Notes | +|---|---|---|---| +| `prs` | Epimetheus (pipeline) | Argus (dashboard), Epimetheus (stale PR detection) | PR tracking, extraction status | +| `audit_log` | Epimetheus (pipeline) | Argus (diagnostics) | 5 cols: id/timestamp/stage/event/detail | +| `response_audit` | bot.py (runtime) | Argus (dashboard), Oberon (frontend) | Query-response audit trail | +| `sources` | Epimetheus (extraction) | Epimetheus (dedup), Argus (metrics) | Source queue and processing status | + +### API Response Shapes + +| Endpoint | Producer | Consumers | Notes | +|---|---|---|---| +| `/health` | Argus | All agents, monitoring | Service health check | +| `/api/alerts` | Argus | Oberon (frontend) | Active alert list | +| `/api/activity` | Argus | Oberon (frontend) | Recent pipeline activity | +| `/api/failure-report/{agent}` | Argus | Oberon (frontend), agents | Per-agent failure breakdown | +| `graph-data.json` | `extract-graph-data.py` | Oberon (frontend) | Knowledge graph visualization data | + +### Service Configuration + +| Config | Owner | Dependents | Notes | +|---|---|---|---| +| `teleo-pipeline.service` | Rhea | Epimetheus, Argus | ReadWritePaths, ExecStart, ports | +| `teleo-diagnostics.service` | Rhea | Argus, Oberon | ReadWritePaths, ports | +| `teleo-bot.service` | Rhea | Epimetheus | ReadWritePaths for pipeline.db | + +## How to Tag Consumers + +In the PR body, add a section: + +``` +## Schema Change + +**Format affected:** claim +**Change:** added optional `attribution` block +**Backward compatible:** yes — old claims without attribution still parse +**Migration:** backfill on next edit (no batch migration needed) +**Consumers to notify:** Leo, Rio, Clay, Theseus, Vida, Astra, extract-graph-data.py +``` + +If the change affects `extract-graph-data.py` or any other pipeline script, the PR must update that script too — don't merge a schema change that breaks the build. + +## Backward Compatibility Rules + +1. **New optional fields** — always backward compatible. Add to schema spec, document default behavior when absent. No migration needed. +2. **New required fields** — must include migration. Either batch-update all existing files in the same PR, or make the field optional first and required later after backfill. +3. **Field renames** — keep old name as accepted alias in pipeline scripts. Document deprecation. Remove old name only after all files are updated. +4. **Enum additions** — backward compatible. Add to schema spec. +5. **Enum removals** — breaking. Must migrate all files using the removed value in the same PR. +6. **Type changes** — breaking. Must migrate all affected files in the same PR. +7. **DB column renames** — treat as breaking. Update all queries in the same PR or add column alias. +8. **API response shape changes** — adding fields is backward compatible; removing or renaming fields is breaking. + +## Legacy Aliases (Currently Active) + +These old field names are still accepted by the pipeline. Don't use them in new files, but don't break them in existing files either: + +| Old Name | Current Name | Format | +|---|---|---| +| `evidence` | `source` | source.md | +| `archive` | (removed) | source.md | +| `source_type` | `format` | source.md | +| `date_published` | `date` | source.md | + +Epimetheus — confirm these are still honored in extraction code. If any are dead, remove from this list. + +## Version Tracking + +No formal version numbers. Schema changes are tracked by: +- The PR that made the change (searchable in git history) +- The updated schema spec in `schemas/` (for file formats) +- The PR description schema change section (for DB/API changes) +- The commit message, which should reference the schema change explicitly + +If the system grows to need formal versioning, add a `schema_version` field to frontmatter. Not needed at current scale (~500 claims, 6 agents). -- 2.45.2