m3taversal
22b6ebb6f6
Some checks are pending
CI / lint-and-test (push) Waiting to run
Orphan ratio at 39.6% (443/1118 claims) vs <15% target. Root cause: reweave threshold 0.70 too strict for text-embedding-3-small — 56% of orphans found "no neighbors." At 0.55, dry-run shows 0% no-neighbor skips. Batch size 200 clears backlog in ~3-4 nights at ~$0.20/run. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
48 lines
1.6 KiB
Bash
Executable file
48 lines
1.6 KiB
Bash
Executable file
#!/bin/bash
|
|
# Nightly reweave — connect orphan claims via vector similarity + Haiku classification.
|
|
# Creates a PR per run for Leo to review.
|
|
#
|
|
# Cron: 0 1 * * * /opt/teleo-eval/pipeline/ops/nightly-reweave.sh >> /opt/teleo-eval/logs/reweave.log 2>&1
|
|
#
|
|
# Pentagon-Agent: Epimetheus <0144398e-4ed3-4fe2-95a3-3d72e1abf887>
|
|
|
|
set -euo pipefail
|
|
|
|
PIPELINE_DIR="/opt/teleo-eval/pipeline"
|
|
EMBED_SCRIPT="/opt/teleo-eval/embed-claims.py"
|
|
REWEAVE_SCRIPT="${PIPELINE_DIR}/reweave.py"
|
|
LOG_DIR="/opt/teleo-eval/logs"
|
|
LOCK_FILE="/opt/teleo-eval/workspaces/.reweave-nightly.lock"
|
|
|
|
# Batch size per night — 200 orphans is ~$0.20 in Haiku calls
|
|
BATCH_SIZE=200
|
|
|
|
echo "=== Nightly reweave started at $(date -u +%Y-%m-%dT%H:%M:%SZ) ==="
|
|
|
|
# Prevent concurrent runs
|
|
if [ -f "$LOCK_FILE" ]; then
|
|
LOCK_AGE=$(( $(date +%s) - $(stat -c %Y "$LOCK_FILE" 2>/dev/null || stat -f %m "$LOCK_FILE") ))
|
|
if [ "$LOCK_AGE" -lt 3600 ]; then
|
|
echo "Lock file exists and is ${LOCK_AGE}s old — another reweave is running. Exiting."
|
|
exit 0
|
|
fi
|
|
echo "Stale lock (${LOCK_AGE}s old) — removing."
|
|
rm -f "$LOCK_FILE"
|
|
fi
|
|
trap 'rm -f "$LOCK_FILE"' EXIT
|
|
touch "$LOCK_FILE"
|
|
|
|
# Step 1: Backfill missing embeddings
|
|
echo "--- Step 1: Embedding backfill ---"
|
|
if [ -f "$EMBED_SCRIPT" ]; then
|
|
python3 "$EMBED_SCRIPT" 2>&1 | tail -5
|
|
echo "Embedding backfill complete."
|
|
else
|
|
echo "WARNING: embed-claims.py not found at ${EMBED_SCRIPT} — skipping backfill"
|
|
fi
|
|
|
|
# Step 2: Run reweave
|
|
echo "--- Step 2: Reweave (batch=${BATCH_SIZE}) ---"
|
|
python3 "$REWEAVE_SCRIPT" --max-orphans "$BATCH_SIZE" 2>&1
|
|
|
|
echo "=== Nightly reweave finished at $(date -u +%Y-%m-%dT%H:%M:%SZ) ==="
|