153 lines
6 KiB
Markdown
153 lines
6 KiB
Markdown
---
|
|
name: teleo-corpus-knowledge-rebuild
|
|
description: Use when locally snapshotting an extracted UTF-8 Forgejo or text corpus into a deterministic Teleo source library, or sharding its verified extraction-work index for parallel processing before any claim generation, review, admission, or database write.
|
|
---
|
|
|
|
# Teleo Corpus Knowledge Rebuild
|
|
|
|
## Job
|
|
|
|
Turn one local UTF-8 source directory into a deterministic, self-contained
|
|
source snapshot bundle. This is the source-library phase only: it preserves
|
|
bytes, paths, hashes, and syntactic section locators without interpreting any
|
|
section as a claim. It does not perform semantic claim extraction.
|
|
|
|
After that inventory verifies, optionally divide its exact work rows into
|
|
deterministic local shards so independent extractors can process a large corpus
|
|
in parallel. Sharding schedules work; it does not interpret source content.
|
|
|
|
## Operator Path
|
|
|
|
From the repository root, build into a path that does not exist:
|
|
|
|
```bash
|
|
python3 ops/run_local_corpus_knowledge_rebuild.py \
|
|
--source-root tests/fixtures/teleo_corpus_rebuild \
|
|
--output /private/tmp/teleo-corpus-rebuild-bundle \
|
|
--mode source-inventory-only
|
|
```
|
|
|
|
Then run the read-only verifier:
|
|
|
|
```bash
|
|
python3 ops/run_local_corpus_knowledge_rebuild.py \
|
|
--verify-bundle /private/tmp/teleo-corpus-rebuild-bundle
|
|
```
|
|
|
|
For parallel processing, choose a shard count from 1 through 1024 that does not
|
|
exceed the verified work-item count. Build into another path that does not
|
|
exist:
|
|
|
|
```bash
|
|
python3 ops/shard_corpus_extraction_work.py \
|
|
--inventory-bundle /private/tmp/teleo-corpus-rebuild-bundle \
|
|
--output-dir /private/tmp/teleo-corpus-extraction-shards \
|
|
--shard-count 64
|
|
```
|
|
|
|
Then verify the shards against the same inventory:
|
|
|
|
```bash
|
|
python3 ops/shard_corpus_extraction_work.py \
|
|
--inventory-bundle /private/tmp/teleo-corpus-rebuild-bundle \
|
|
--verify-bundle /private/tmp/teleo-corpus-extraction-shards
|
|
```
|
|
|
|
Inspect these stable entry points:
|
|
|
|
```text
|
|
source-manifest.json
|
|
extraction-work-index.json
|
|
receipt.json
|
|
snapshots/sha256/<prefix>/<sha256>.utf8
|
|
shard-manifest.json
|
|
shards/shard-<index>-of-<count>.json
|
|
```
|
|
|
|
## Required Readback
|
|
|
|
Require all of these:
|
|
|
|
- every source row binds a normalized source-root-relative path to one exact,
|
|
content-addressed byte snapshot;
|
|
- every section has exact one-based line and UTF-8 byte ranges plus a quote
|
|
hash;
|
|
- every work row is `source_section_for_semantic_extraction`;
|
|
- `semantic_claim_extraction_performed=false`,
|
|
`database_write_performed=false`, and `admission_performed=false` throughout;
|
|
- the receipt says the directory snapshot was not atomic and only guarantees
|
|
the exact files copied from stable descriptors;
|
|
- the verifier recomputes snapshots, locators, cardinality, artifact hashes,
|
|
bundle-tree hash, and receipt hash;
|
|
- a second run over unchanged inputs produces byte-identical output.
|
|
- every inventory work item appears in exactly one shard selected by
|
|
`sha256(utf8(work_item_id)) mod shard_count`;
|
|
- each shard binds the exact inventory tree, source-manifest, work-index, and
|
|
inventory-receipt hashes without embedding source-body text or absolute paths;
|
|
- the shard verifier rejects wrong inventories, missing, duplicate, unknown, or
|
|
misplaced work IDs, symlinks, extras, incomplete output, and rehashed
|
|
side-effect overclaims;
|
|
- shard directories are `0700`, files are `0600`, and repeated runs with the
|
|
same inventory and shard count are byte-identical.
|
|
|
|
Run focused checks:
|
|
|
|
```bash
|
|
python3 -m pytest -q tests/test_run_local_corpus_knowledge_rebuild.py
|
|
python3 -m pytest -q tests/test_shard_corpus_extraction_work.py
|
|
python3 -m ruff check \
|
|
ops/run_local_corpus_knowledge_rebuild.py \
|
|
ops/shard_corpus_extraction_work.py \
|
|
tests/test_run_local_corpus_knowledge_rebuild.py \
|
|
tests/test_shard_corpus_extraction_work.py
|
|
```
|
|
|
|
## Safety Boundary
|
|
|
|
The build reserves a new output directory atomically and never replaces an
|
|
existing destination. It publishes `receipt.json` while the incomplete marker
|
|
is still present, then removes that marker as the final completion step. It
|
|
opens source files descriptor-first without following
|
|
symlinks, streams exact bytes into private content-addressed snapshots, rejects
|
|
invalid UTF-8, binary controls, lone CR newlines, unsupported entries, path
|
|
collisions, and configured resource-limit overruns, and deduplicates identical
|
|
content.
|
|
|
|
The defaults are bounded. Raise them only with the explicit `--max-*` options;
|
|
the command enforces hard upper bounds on every override.
|
|
|
|
The sharder independently verifies the inventory before reading work metadata,
|
|
reserves a new output directory without replacement, writes through held
|
|
directory descriptors, keeps an incomplete marker through receipt publication,
|
|
and revalidates the inventory before publishing. It reads no snapshot bodies.
|
|
Hash shards can be empty; membership is stable only for the exact inventory and
|
|
the same shard count.
|
|
|
|
## Semantic Handoff
|
|
|
|
The inventory plus one verified shard is input to a separate semantic
|
|
extractor, not to a database writer. The extractor must verify both bundles,
|
|
then resolve each relative snapshot path and exact byte range from the supplied
|
|
inventory. Only after that extractor has produced explicit, source-backed V3
|
|
manifests may the operator run:
|
|
|
|
```text
|
|
scripts/compile_kb_source_packet.py
|
|
```
|
|
|
|
That compiler's output still requires separate review and guarded admission.
|
|
Do not fabricate V3 manifests from raw sections in this inventory helper.
|
|
|
|
Use `teleo-reconstruction-recovery` for restore/replay selection and
|
|
`teleo-kb-db-change-workflow` only for an explicitly authorized later review or
|
|
apply task.
|
|
|
|
## Claim Ceiling
|
|
|
|
The output proves deterministic local inventory of the exact copied UTF-8
|
|
files, bounded syntactic sectioning, bundle integrity, and an exact once-only
|
|
hash partition of extraction work for a fixed shard count. It does not prove an
|
|
atomic snapshot of concurrent directory changes, non-empty shards,
|
|
format-specific syntax, semantic extraction or extractor quality, claim
|
|
atomicity or correctness, database deduplication, review, admission,
|
|
application, binary-document extraction, or any live environment.
|