---
name: teleo-corpus-knowledge-rebuild
description: Use when locally snapshotting an extracted UTF-8 Forgejo or text corpus into a deterministic Teleo source library, or sharding its verified extraction-work index for parallel processing before any claim generation, review, admission, or database write.
---

# Teleo Corpus Knowledge Rebuild

## Job

Turn one local UTF-8 source directory into a deterministic, self-contained
source snapshot bundle. This is the source-library phase only: it preserves
bytes, paths, hashes, and syntactic section locators without interpreting any
section as a claim. It does not perform semantic claim extraction.

After that inventory verifies, optionally divide its exact work rows into
deterministic local shards so independent extractors can process a large corpus
in parallel. Sharding schedules work; it does not interpret source content.

## Operator Path

From the repository root, build into a path that does not exist:

```bash
python3 ops/run_local_corpus_knowledge_rebuild.py \
  --source-root tests/fixtures/teleo_corpus_rebuild \
  --output /private/tmp/teleo-corpus-rebuild-bundle \
  --mode source-inventory-only
```

Then run the read-only verifier:

```bash
python3 ops/run_local_corpus_knowledge_rebuild.py \
  --verify-bundle /private/tmp/teleo-corpus-rebuild-bundle
```

For parallel processing, choose a shard count from 1 through 1024 that does not
exceed the verified work-item count. Build into another path that does not
exist:

```bash
python3 ops/shard_corpus_extraction_work.py \
  --inventory-bundle /private/tmp/teleo-corpus-rebuild-bundle \
  --output-dir /private/tmp/teleo-corpus-extraction-shards \
  --shard-count 64
```

Then verify the shards against the same inventory:

```bash
python3 ops/shard_corpus_extraction_work.py \
  --inventory-bundle /private/tmp/teleo-corpus-rebuild-bundle \
  --verify-bundle /private/tmp/teleo-corpus-extraction-shards
```

Inspect these stable entry points:

```text
source-manifest.json
extraction-work-index.json
receipt.json
snapshots/sha256/<prefix>/<sha256>.utf8
shard-manifest.json
shards/shard-<index>-of-<count>.json
```

## Required Readback

Require all of these:

- every source row binds a normalized source-root-relative path to one exact,
  content-addressed byte snapshot;
- every section has exact one-based line and UTF-8 byte ranges plus a quote
  hash;
- every work row is `source_section_for_semantic_extraction`;
- `semantic_claim_extraction_performed=false`,
  `database_write_performed=false`, and `admission_performed=false` throughout;
- the receipt says the directory snapshot was not atomic and only guarantees
  the exact files copied from stable descriptors;
- the verifier recomputes snapshots, locators, cardinality, artifact hashes,
  bundle-tree hash, and receipt hash;
- a second run over unchanged inputs produces byte-identical output.
- every inventory work item appears in exactly one shard selected by
  `sha256(utf8(work_item_id)) mod shard_count`;
- each shard binds the exact inventory tree, source-manifest, work-index, and
  inventory-receipt hashes without embedding source-body text or absolute paths;
- the shard verifier rejects wrong inventories, missing, duplicate, unknown, or
  misplaced work IDs, symlinks, extras, incomplete output, and rehashed
  side-effect overclaims;
- shard directories are `0700`, files are `0600`, and repeated runs with the
  same inventory and shard count are byte-identical.

Run focused checks:

```bash
python3 -m pytest -q tests/test_run_local_corpus_knowledge_rebuild.py
python3 -m pytest -q tests/test_shard_corpus_extraction_work.py
python3 -m ruff check \
  ops/run_local_corpus_knowledge_rebuild.py \
  ops/shard_corpus_extraction_work.py \
  tests/test_run_local_corpus_knowledge_rebuild.py \
  tests/test_shard_corpus_extraction_work.py
```

## Safety Boundary

The build reserves a new output directory atomically and never replaces an
existing destination. It publishes `receipt.json` while the incomplete marker
is still present, then removes that marker as the final completion step. It
opens source files descriptor-first without following
symlinks, streams exact bytes into private content-addressed snapshots, rejects
invalid UTF-8, binary controls, lone CR newlines, unsupported entries, path
collisions, and configured resource-limit overruns, and deduplicates identical
content.

The defaults are bounded. Raise them only with the explicit `--max-*` options;
the command enforces hard upper bounds on every override.

The sharder independently verifies the inventory before reading work metadata,
reserves a new output directory without replacement, writes through held
directory descriptors, keeps an incomplete marker through receipt publication,
and revalidates the inventory before publishing. It reads no snapshot bodies.
Hash shards can be empty; membership is stable only for the exact inventory and
the same shard count.

## Semantic Handoff

The inventory plus one verified shard is input to a separate semantic
extractor, not to a database writer. The extractor must verify both bundles,
then resolve each relative snapshot path and exact byte range from the supplied
inventory. Only after that extractor has produced explicit, source-backed V3
manifests may the operator run:

```text
scripts/compile_kb_source_packet.py
```

That compiler's output still requires separate review and guarded admission.
Do not fabricate V3 manifests from raw sections in this inventory helper.

Use `teleo-reconstruction-recovery` for restore/replay selection and
`teleo-kb-db-change-workflow` only for an explicitly authorized later review or
apply task.

## Claim Ceiling

The output proves deterministic local inventory of the exact copied UTF-8
files, bounded syntactic sectioning, bundle integrity, and an exact once-only
hash partition of extraction work for a fixed shard count. It does not prove an
atomic snapshot of concurrent directory changes, non-empty shards,
format-specific syntax, semantic extraction or extractor quality, claim
atomicity or correctness, database deduplication, review, admission,
application, binary-document extraction, or any live environment.
