m3taversal
d79ff60689
Pipeline reliability (8 fixes, reviewed by Ganymede+Rhea+Leo+Rio):
1. Merge API recovery — pre-flight approval check, transient/permanent distinction, jitter
2. Ghost PR detection — ls-remote branch check in reconciliation, network guard
3. Source status contract — directory IS status, no code change needed
4. Batch-state markers eliminated — two-gate skip (archive-check + batched branch-check)
5. Branch SHA tracking — batched ls-remote, auto-reset verdicts, dismiss stale reviews
6. Mirror pre-flight permissions — chown check in sync-mirror.sh
7. Telegram archive commit-after-write — git add/commit/push with rebase --abort fallback
8. Post-merge source archiving — queue/ → archive/{domain}/ after merge
Pipeline fixes:
- merge_cycled flag — eval attempts preserved during merge-failure cycling (Ganymede+Rhea)
- merge_failures diagnostic counter
- Startup recovery preserves eval_attempts (was incorrectly resetting to 0)
- No-diff PRs auto-closed by eval (root cause of 17 zombie PRs)
- GC threshold aligned with substantive fixer budget (was 2, now 4)
- Conflict retry with 3-attempt budget + permanent conflict handler
- Local ff-merge fallback for Forgejo 405 errors
Telegram bot:
- KB retrieval: 3-layer (entity resolution → claim search → agent context)
- Reply-to-bot handler (context.bot.id check)
- Tag regex: @teleo|@futairdbot
- Prompt rewrite for natural analyst voice
- Market data API integration (Ben's token price endpoint)
- Conversation windows (5-message unanswered counter, per-user-per-chat)
- Conversation history in prompt (last 5 exchanges)
- Worktree file lock for archive writes
Infrastructure:
- worktree_lock.py — file-based lock (flock) for main worktree coordination
- backfill-sources.py — source DB registration for Argus funnel
- batch-extract-50.sh v3 — two-gate skip, batched ls-remote, network guard
- sync-mirror.sh — auto-PR creation for mirrored GitHub branches, permission pre-flight
- Argus dashboard — conflicts + reviewing in backlog, queue count in funnel
- Enrichment-inside-frontmatter bug fix (regex anchor, not --- split)
Pentagon-Agent: Epimetheus <3D35839A-7722-4740-B93D-51157F7D5E70>
124 lines
5.2 KiB
Bash
Executable file
124 lines
5.2 KiB
Bash
Executable file
#!/bin/bash
|
|
# Bidirectional sync: Forgejo (authoritative) <-> GitHub (public mirror)
|
|
# Forgejo wins on conflict. Runs every 2 minutes via cron.
|
|
#
|
|
# Security note: GitHub->Forgejo path is for external contributor convenience.
|
|
# Never auto-process branches arriving via this path without a PR.
|
|
# Eval pipeline and extract cron only act on PRs, not raw branches.
|
|
|
|
set -euo pipefail
|
|
|
|
REPO_DIR="/opt/teleo-eval/mirror/teleo-codex.git"
|
|
LOG="/opt/teleo-eval/logs/sync.log"
|
|
LOCKFILE="/tmp/sync-mirror.lock"
|
|
|
|
log() { echo "[$(date -Iseconds)] $1" >> "$LOG"; }
|
|
|
|
# Lockfile — prevent concurrent runs
|
|
if [ -f "$LOCKFILE" ]; then
|
|
pid=$(cat "$LOCKFILE" 2>/dev/null)
|
|
if kill -0 "$pid" 2>/dev/null; then
|
|
exit 0
|
|
fi
|
|
rm -f "$LOCKFILE"
|
|
fi
|
|
echo $$ > "$LOCKFILE"
|
|
trap 'rm -f "$LOCKFILE"' EXIT
|
|
|
|
# Pre-flight: fix permissions if another user touched the mirror dir (Rhea)
|
|
BAD_PERMS=$(find "$REPO_DIR" ! -user teleo 2>/dev/null | head -1 || true)
|
|
if [ -n "$BAD_PERMS" ]; then
|
|
log "Fixing mirror permissions (found: $BAD_PERMS)"
|
|
chown -R teleo:teleo "$REPO_DIR" 2>/dev/null
|
|
fi
|
|
cd "$REPO_DIR" || { log "ERROR: cannot cd to $REPO_DIR"; exit 1; }
|
|
|
|
# Step 1: Fetch from Forgejo (must succeed — it's authoritative)
|
|
log "Fetching from Forgejo..."
|
|
if ! git fetch forgejo --prune >> "$LOG" 2>&1; then
|
|
log "ERROR: Forgejo fetch failed — aborting"
|
|
exit 1
|
|
fi
|
|
|
|
# Step 2: Fetch from GitHub (warn on failure, don't abort)
|
|
log "Fetching from GitHub..."
|
|
git fetch origin --prune >> "$LOG" 2>&1 || log "WARN: GitHub fetch failed"
|
|
|
|
# Step 3: Forgejo -> GitHub (primary direction)
|
|
# Update local refs from Forgejo remote refs using process substitution (avoids subshell)
|
|
log "Syncing Forgejo -> GitHub..."
|
|
while read branch; do
|
|
[ "$branch" = "HEAD" ] && continue
|
|
git update-ref "refs/heads/$branch" "refs/remotes/forgejo/$branch" 2>/dev/null || \
|
|
log "WARN: Failed to update ref $branch"
|
|
done < <(git for-each-ref --format="%(refname:lstrip=3)" refs/remotes/forgejo/)
|
|
|
|
# Safety: verify Forgejo main descends from GitHub main before force-pushing
|
|
GITHUB_MAIN=$(git rev-parse refs/remotes/origin/main 2>/dev/null || true)
|
|
FORGEJO_MAIN=$(git rev-parse refs/remotes/forgejo/main 2>/dev/null || true)
|
|
PUSH_MAIN=true
|
|
if [ -n "$GITHUB_MAIN" ] && [ -n "$FORGEJO_MAIN" ]; then
|
|
if ! git merge-base --is-ancestor "$GITHUB_MAIN" "$FORGEJO_MAIN"; then
|
|
log "CRITICAL: Forgejo main is NOT a descendant of GitHub main — skipping main push"
|
|
log "CRITICAL: GitHub main: $GITHUB_MAIN, Forgejo main: $FORGEJO_MAIN"
|
|
PUSH_MAIN=false
|
|
fi
|
|
fi
|
|
|
|
if [ "$PUSH_MAIN" = true ]; then
|
|
git push origin --all --force >> "$LOG" 2>&1 || log "WARN: Push to GitHub failed"
|
|
else
|
|
# Push all branches except main
|
|
while read branch; do
|
|
[ "$branch" = "main" ] && continue
|
|
[ "$branch" = "HEAD" ] && continue
|
|
git push origin --force "refs/heads/$branch:refs/heads/$branch" >> "$LOG" 2>&1 || \
|
|
log "WARN: Failed to push $branch to GitHub"
|
|
done < <(git for-each-ref --format="%(refname:lstrip=2)" refs/heads/)
|
|
fi
|
|
git push origin --tags --force >> "$LOG" 2>&1 || log "WARN: Tag push to GitHub failed"
|
|
|
|
# Step 4: GitHub -> Forgejo (external contributions only)
|
|
# Only push branches that exist on GitHub but NOT on Forgejo
|
|
log "Checking GitHub-only branches..."
|
|
GITHUB_ONLY=$(comm -23 \
|
|
<(git for-each-ref --format="%(refname:lstrip=3)" refs/remotes/origin/ | grep -v HEAD | sort) \
|
|
<(git for-each-ref --format="%(refname:lstrip=3)" refs/remotes/forgejo/ | grep -v HEAD | sort))
|
|
|
|
if [ -n "$GITHUB_ONLY" ]; then
|
|
FORGEJO_TOKEN=$(cat /opt/teleo-eval/secrets/forgejo-admin-token 2>/dev/null)
|
|
for branch in $GITHUB_ONLY; do
|
|
log "New from GitHub: $branch -> Forgejo"
|
|
git push forgejo "refs/remotes/origin/$branch:refs/heads/$branch" >> "$LOG" 2>&1 || {
|
|
log "WARN: Failed to push $branch to Forgejo"
|
|
continue
|
|
}
|
|
# Auto-create PR on Forgejo for mirrored branches (external contributor path)
|
|
# Skip pipeline-internal branches
|
|
case "$branch" in
|
|
extract/*|ingestion/*) continue ;;
|
|
esac
|
|
if [ -n "$FORGEJO_TOKEN" ]; then
|
|
# Check if PR already exists
|
|
EXISTING=$(curl -sf "http://localhost:3000/api/v1/repos/teleo/teleo-codex/pulls?state=open&head=$branch&limit=1" \
|
|
-H "Authorization: token $FORGEJO_TOKEN" 2>/dev/null || echo "[]")
|
|
if [ "$EXISTING" = "[]" ] || [ "$EXISTING" = "null" ]; then
|
|
PR_TITLE=$(echo "$branch" | sed 's|/|: |;s/-/ /g')
|
|
RESULT=$(curl -sf -X POST "http://localhost:3000/api/v1/repos/teleo/teleo-codex/pulls" \
|
|
-H "Authorization: token $FORGEJO_TOKEN" \
|
|
-H "Content-Type: application/json" \
|
|
-d "{\"title\":\"$PR_TITLE\",\"head\":\"$branch\",\"base\":\"main\"}" 2>/dev/null || echo "")
|
|
PR_NUM=$(echo "$RESULT" | grep -o '"number":[0-9]*' | head -1 | grep -o "[0-9]*" || true)
|
|
if [ -n "$PR_NUM" ]; then
|
|
log "Auto-created PR #$PR_NUM on Forgejo for $branch"
|
|
else
|
|
log "WARN: Failed to auto-create PR for $branch"
|
|
fi
|
|
fi
|
|
fi
|
|
done
|
|
else
|
|
log "No new GitHub-only branches"
|
|
fi
|
|
|
|
log "Sync complete"
|