teleo-infrastructure/.agents/skills/teleo-corpus-knowledge-rebuild/SKILL.md

name	description
teleo-corpus-knowledge-rebuild	Use when locally snapshotting an extracted UTF-8 Forgejo or text corpus into a deterministic Teleo source library, or sharding its verified extraction-work index for parallel processing before any claim generation, review, admission, or database write.

Teleo Corpus Knowledge Rebuild

Job

Turn one local UTF-8 source directory into a deterministic, self-contained source snapshot bundle. This is the source-library phase only: it preserves bytes, paths, hashes, and syntactic section locators without interpreting any section as a claim. It does not perform semantic claim extraction.

After that inventory verifies, optionally divide its exact work rows into deterministic local shards so independent extractors can process a large corpus in parallel. Sharding schedules work; it does not interpret source content.

Operator Path

From the repository root, build into a path that does not exist:

python3 ops/run_local_corpus_knowledge_rebuild.py \
  --source-root tests/fixtures/teleo_corpus_rebuild \
  --output /private/tmp/teleo-corpus-rebuild-bundle \
  --mode source-inventory-only

Then run the read-only verifier:

python3 ops/run_local_corpus_knowledge_rebuild.py \
  --verify-bundle /private/tmp/teleo-corpus-rebuild-bundle

For parallel processing, choose a shard count from 1 through 1024 that does not exceed the verified work-item count. Build into another path that does not exist:

python3 ops/shard_corpus_extraction_work.py \
  --inventory-bundle /private/tmp/teleo-corpus-rebuild-bundle \
  --output-dir /private/tmp/teleo-corpus-extraction-shards \
  --shard-count 64

Then verify the shards against the same inventory:

python3 ops/shard_corpus_extraction_work.py \
  --inventory-bundle /private/tmp/teleo-corpus-rebuild-bundle \
  --verify-bundle /private/tmp/teleo-corpus-extraction-shards

Inspect these stable entry points:

source-manifest.json
extraction-work-index.json
receipt.json
snapshots/sha256/<prefix>/<sha256>.utf8
shard-manifest.json
shards/shard-<index>-of-<count>.json

Required Readback

Require all of these:

• every source row binds a normalized source-root-relative path to one exact, content-addressed byte snapshot;
• every section has exact one-based line and UTF-8 byte ranges plus a quote hash;
• every work row is source_section_for_semantic_extraction;
• semantic_claim_extraction_performed=false, database_write_performed=false, and admission_performed=false throughout;
• the receipt says the directory snapshot was not atomic and only guarantees the exact files copied from stable descriptors;
• the verifier recomputes snapshots, locators, cardinality, artifact hashes, bundle-tree hash, and receipt hash;
• a second run over unchanged inputs produces byte-identical output.
• every inventory work item appears in exactly one shard selected by sha256(utf8(work_item_id)) mod shard_count;
• each shard binds the exact inventory tree, source-manifest, work-index, and inventory-receipt hashes without embedding source-body text or absolute paths;
• the shard verifier rejects wrong inventories, missing, duplicate, unknown, or misplaced work IDs, symlinks, extras, incomplete output, and rehashed side-effect overclaims;
• shard directories are 0700, files are 0600, and repeated runs with the same inventory and shard count are byte-identical.

Run focused checks:

python3 -m pytest -q tests/test_run_local_corpus_knowledge_rebuild.py
python3 -m pytest -q tests/test_shard_corpus_extraction_work.py
python3 -m ruff check \
  ops/run_local_corpus_knowledge_rebuild.py \
  ops/shard_corpus_extraction_work.py \
  tests/test_run_local_corpus_knowledge_rebuild.py \
  tests/test_shard_corpus_extraction_work.py

Safety Boundary

The build reserves a new output directory atomically and never replaces an existing destination. It publishes receipt.json while the incomplete marker is still present, then removes that marker as the final completion step. It opens source files descriptor-first without following symlinks, streams exact bytes into private content-addressed snapshots, rejects invalid UTF-8, binary controls, lone CR newlines, unsupported entries, path collisions, and configured resource-limit overruns, and deduplicates identical content.

The defaults are bounded. Raise them only with the explicit --max-* options; the command enforces hard upper bounds on every override.

The sharder independently verifies the inventory before reading work metadata, reserves a new output directory without replacement, writes through held directory descriptors, keeps an incomplete marker through receipt publication, and revalidates the inventory before publishing. It reads no snapshot bodies. Hash shards can be empty; membership is stable only for the exact inventory and the same shard count.

Semantic Handoff

The inventory plus one verified shard is input to a separate semantic extractor, not to a database writer. The extractor must verify both bundles, then resolve each relative snapshot path and exact byte range from the supplied inventory. Only after that extractor has produced explicit, source-backed V3 manifests may the operator run:

scripts/compile_kb_source_packet.py

That compiler's output still requires separate review and guarded admission. Do not fabricate V3 manifests from raw sections in this inventory helper.

Use teleo-reconstruction-recovery for restore/replay selection and teleo-kb-db-change-workflow only for an explicitly authorized later review or apply task.

Claim Ceiling

The output proves deterministic local inventory of the exact copied UTF-8 files, bounded syntactic sectioning, bundle integrity, and an exact once-only hash partition of extraction work for a fixed shard count. It does not prove an atomic snapshot of concurrent directory changes, non-empty shards, format-specific syntax, semantic extraction or extractor quality, claim atomicity or correctness, database deduplication, review, admission, application, binary-document extraction, or any live environment.