teleo-infrastructure/docs/external-contributor-merge-flow.md

# External Contributor Merge Flow — Design Doc

**Author:** Epimetheus
**Architecture review:** Ship (owns merge.py, sync-mirror.sh)
**Code review:** Ganymede (line-level, post-design-approval)
**Status:** Phase 1 sweep-only scope locked. Phase 2 architecture decisions locked (Ship Msg 3). Awaiting Phase 1 line-level code review.

## Revision log

- **v3 (this revision):** Cleanup pass per Ship Msg 3 — merge commit message updated to locked verbose form (`"Merge external GitHub PR #{N}: {branch_slug}"`), open-questions section collapsed to "Locked Phase 2 decisions" restating the three resolved outcomes (no longer questions).
- **v2:** Phase 1 simplified per Ship's Msg 2 — backfill script dropped (sweep IS the backfill), sweep placement specified explicitly as the very first action after initial fetch (ahead of branch-mirror loop AND auto-create-PR block at line ~250). Phase 2 architecture unchanged.
- **v1:** Initial draft.

## Problem statement

External GitHub contributors submit PRs via the `living-ip/teleo-codex` mirror.
Pipeline accepts the claim and merges the content into Forgejo `main`, but the
GitHub PR shows "open with no diff" — it looks abandoned to the contributor.

Two compounding bugs intersect on this path:

1. **Cherry-pick breaks GitHub merge detection.** `lib/merge.py::_cherry_pick_onto_main`
   creates a new SHA on Forgejo main. GitHub's "is PR head SHA an ancestor of
   main?" check returns false. `merged: false, merge_commit_sha: null` forever.

2. **`prs.github_pr` not populated for fork PRs.** `sync-mirror.sh` Step 4.5 looks
   up GitHub PR number via `?head=living-ip:$branch`, but fork PR heads are
   `FwazB:contributor/...` (or `<fork-owner>:<branch>`), not `living-ip:`. The
   filter misses, `github_pr` stays NULL, and `lib/github_feedback.py::on_merged`
   returns early (no comment, no close) because `_get_github_pr` requires non-NULL.

Empirical:

| PR | head | merge mech | github_pr | merged badge | comment posted |
|---|---|---|---|---|---|
| #87 (own-repo) | `living-ip:fix/...` | `git merge --no-ff` | populated | ✓ true | ✓ |
| #90 (FwazB fork) | `FwazB:contributor/...` | cherry-pick | NULL | ✗ false | ✗ |

Both bugs need fixes. Bug #1 is the structural one (load-bearing for the badge).
Bug #2 is a sync-mirror filter issue (load-bearing for the comment/close).

## Goal

External GitHub contributor opens a PR → pipeline ingests, evaluates, merges →
GitHub PR shows `merged: true` with badge → bot comment posted → PR closed
cleanly. No human in the loop on the success path. Failure modes (eval reject,
auto-fix, contributor force-push) handled gracefully.

## Out of scope

- Agent-extraction PRs (`extract/*`, `reweave/*`, `epimetheus/*`, etc.) — keep
  cherry-pick. They merge 70+/day, have no contributor UX surface, and the
  cherry-pick → linear-history rationale (auto-fixer rebase pattern, bisect
  friendliness) holds.
- `/api/contributors` legacy endpoint — separate work, deferred.
- PAT-in-URL credential pattern — separate security follow-up.

## Design — branch-prefix conditional, scoped to `gh-pr-*`

### Bug #1 fix: `_merge_no_ff_external` for `gh-pr-*` branches

**Dispatch site (`lib/merge.py::_merge_domain_queue`, currently lines 736-738):**

```python
# Reweave: per-file frontmatter union (existing)
if branch.startswith("reweave/"):
    merge_fn = _merge_reweave_pr(branch)
# External GitHub fork PRs: true merge with --no-ff so contributor SHA lands
# in main's history → GitHub recognizes "merged" badge.
elif branch.startswith("gh-pr-"):
    merge_fn = _merge_no_ff_external(branch)
# Default: cherry-pick (extraction commits ADD new files, applies cleanly,
# linear history preserved for the bulk-extraction flow).
else:
    merge_fn = _cherry_pick_onto_main(branch)
```

**New function (`lib/merge.py`):**

```python
async def _merge_no_ff_external(branch: str) -> tuple[bool, str]:
    """Merge an external GitHub PR with --no-ff so contributor SHA lands in main.

    Why this differs from _cherry_pick_onto_main:
    - Cherry-pick rewrites SHA → GitHub never recognizes the PR as merged.
    - --no-ff preserves the contributor's commit SHA in main's history.
    - sync-mirror's Forgejo→GitHub propagation already handles merge commits
      (verified empirically: PR #87 round-tripped cleanly with merge_commit_sha
      preserved).

    Mechanics:
    1. Fetch latest origin/main and origin/{branch}
    2. Create scratch worktree at HEAD of origin/main
    3. Derive: gh_pr_num = re.match(r"gh-pr-(\d+)/", branch).group(1)
              branch_slug = branch[len(f"gh-pr-{gh_pr_num}/"):]
    4. git merge --no-ff origin/{branch} \
         -m f"Merge external GitHub PR #{gh_pr_num}: {branch_slug}"
    5. git push origin HEAD:main
    6. Cleanup worktree

    Conflict handling:
    - Entity conflicts: same auto-resolve pattern as cherry-pick
      (--ours = main HEAD, --theirs = branch). External claims rarely touch
      entities so this is a low-frequency path.
    - Other conflicts: abort, return False with conflict detail. Caller marks
      conflict_permanent. Manual resolution or contributor rebase required.

    Idempotency: caller already gates on PR status, so re-running on a merged
    PR fails at the merge step (already merged), which is the right behavior.

    Returns (success, message).
    """
```

The merge commit message format `"Merge external GitHub PR #{N}: {branch_slug}"`
embeds the GitHub PR number explicitly so `git log --merges --grep "#90"`
surfaces the merge from PR number alone (no branch-name guessing). Branch slug
is the post-`gh-pr-{N}/` portion of the branch (e.g.,
`contributor/arcium-confidential-computing-challenge`) — already in scope at
merge time, no claim-file read needed.

### Bug #2 fix: self-healing sweep at top of sync-mirror cycle

**Root cause:** The one-shot link UPDATE in Step 4.5 (lines ~250-294) runs once
per branch creation — if it fails (race with PR row insertion, transient API
hiccup, transient lock), the row is permanently stuck at `github_pr=NULL`. No
retry path. FwazB's PR 4066 is the visible artifact of this class of failure.

The structural fix is a self-healing sweep: each cron tick, scan for any
`gh-pr-*` PR rows missing `github_pr` and link them. Idempotent, zero-cost when
clean, retries forever until the row is healed. The sweep IS the backfill — no
separate one-shot script needed (Ship's simplification, Msg 2). First cron tick
after deploy picks up 4066 automatically, same SELECT path that handles all
future races.

**Placement (load-bearing):** the sweep runs **as the very first action after
the initial Forgejo+GitHub fetch**, ahead of both:
- The branch-mirror loop
- The auto-create-PR block at line ~250

This sequencing matters because a fresh-cycle race — PR created in the *current*
cycle, link UPDATE in `Step 4.5` fails — self-heals on the very next iteration's
sweep, not 2 minutes later. Same-cycle convergence vs cross-cycle convergence.

```bash
# Step 0: self-heal any gh-pr-* PR rows missing github_pr.
# Runs FIRST — before branch-mirror loop, before auto-create-PR block.
# Idempotent: SELECT returns empty when clean.
# The branch name encodes the GitHub PR number (gh-pr-{N}/...) so no API
# round-trip needed to recover the number. Source-of-truth derivation.
sqlite3 -separator '|' "$PIPELINE_DB" \
  "SELECT number, branch FROM prs WHERE branch LIKE 'gh-pr-%' AND github_pr IS NULL" \
| while IFS='|' read -r pr_num branch; do
    gh_pr_num=$(echo "$branch" | sed -n 's|^gh-pr-\([0-9]*\)/.*|\1|p')
    [ -z "$gh_pr_num" ] && continue
    sqlite3 "$PIPELINE_DB" \
      "UPDATE prs SET github_pr = $gh_pr_num, source_channel = 'github' WHERE number = $pr_num;"
    log "self-heal: linked Forgejo PR #$pr_num → GitHub PR #$gh_pr_num"
done
```

~10 lines. No API call required (branch name carries the number deterministically).
No `.fork-pr-map` file, no `--script` flag, no manual deploy step.

**Why this approach beats the earlier `.fork-pr-map` proposal:**
- The map approach repaired the write path but left already-stuck rows orphaned
- A separate one-shot backfill script for the orphans is one more code path to maintain
- The sweep collapses both paths: it's the failure recovery AND the historical backfill
- Future races automatically self-heal without code changes

**Why this approach beats the earlier `?head=living-ip:` filter:**
- That filter never matched fork PRs at all — by design, fork PRs come from a
  different owner. Pre-existing nit Ganymede flagged in multi-repo-mirror review
- The branch name encoding bypasses the GitHub API entirely for fork PRs

**Why no API verification before UPDATE:**
- Branch name is deterministic source — the only writer of `gh-pr-N/...` is
  Step 2.1, which only writes after fetching the GitHub PR ref. If the branch
  exists, the GitHub PR exists.
- Avoids API rate-limit pressure on every cron tick (every 2 min × 24h × 7d ≈ 5040
  calls/week even when no work needed).
- Sanity-check via API is achievable cheaply if Ship wants it; current scope
  matches "zero-cost when clean."

### Auto-fixer mode='append' for `gh-pr-*` branches

**Current behavior (`lib/fixer.py`):** Worktree-based fix → commit → push (regular
push, not force). When the PR was created by the pipeline (extract/* branches),
this works because the LLM-extractor's commit is at HEAD and we're appending on
top — push succeeds.

**External PR behavior (today):** Same code path runs. Fork PR has FwazB's commit
at HEAD; auto-fixer creates a fix commit on top; pushes via Forgejo's branch
ref (the fork PR was mirrored as `refs/heads/gh-pr-90/contributor/...` on
Forgejo). Push works. Eval reset fires. Eval re-runs.

Wait — re-reading the existing fixer, it's actually already append-only. Good.
The cherry-pick at merge time was the only thing rewriting SHAs. So **no fixer
change required** for Option 2. The fixer commit is already at HEAD~1 from
FwazB's commit on Forgejo. When we `git merge --no-ff` instead of cherry-pick,
the merge commit's parent chain includes BOTH the fix commit AND FwazB's
original commit. GitHub sees FwazB's SHA in ancestry → "merged" badge.

**Cross-check:** Verified PR 4066's existing branch state.

```
$ git log refs/heads/gh-pr-90/contributor/arcium-confidential-computing-challenge --oneline
d7916d65 auto-fix: strip 2 broken wiki links     ← fixer's commit
f6a59d7d claim: confidential computing reshapes... ← FwazB's commit
```

Both commits already on Forgejo. When merge.py cherry-picked, it picked both
commits onto main as new SHAs. With merge --no-ff, both stay intact, the merge
commit references them, GitHub sees `f6a59d7d` (FwazB's HEAD on his fork) in
main's ancestry, marks merged.

**This means scope shrinks:** the design is purely a merge.py change + a
sync-mirror Step 2.1/4.5 plumbing tweak. No fixer module changes.

### Edge case: contributor rebases their fork after fixer appended

**Scenario:** FwazB's PR is in eval. Pipeline auto-fixer pushes a fix commit to
Forgejo `gh-pr-90/...`. FwazB notices the original wiki-link issue, fixes it
locally, force-pushes to his fork. sync-mirror's next cycle fetches his new
SHA → tries to update the Forgejo branch → push from sync-mirror is regular
(not --force) → fails because Forgejo branch has diverged from FwazB's fork.

**Today's behavior:** sync-mirror logs a warning. Forgejo branch keeps the
appended-fix state. Eval continues against that state. FwazB's most recent
fork state is silently ignored.

**Acceptable risk for hackathon.** The eval-reset-on-tip-change gate will
re-trigger eval if anyone force-pushes the Forgejo branch later (e.g., manual
re-sync). Documented; not fixing in this PR.

**Followup:** sync-mirror could detect the divergence, log a structured alert,
and post a comment on the GitHub PR ("we detected your force-push but our
appended fix has diverged; please rebase against `<sha>` or we'll close").
Out of scope for this branch.

## Test plan

| # | Scenario | Expected | Validation |
|---|---|---|---|
| 1 | External PR, clean (no auto-fix needed) | Merged with `--no-ff`, contributor SHA in main, GitHub badge `merged: true`, on_merged comment + close | curl GitHub API for PR state after merge |
| 2 | External PR with broken wiki links | auto-fixer appends commit, eval re-runs and approves, merge --no-ff brings BOTH commits in via merge commit, GitHub badge `merged: true` | log line trace + GitHub API |
| 3 | External PR rejected by eval (substantive issue) | terminate_pr fires existing path, on_closed posts rejection comment + closes GitHub PR | GitHub API after eval cycle |
| 4 | sync-mirror github_pr backfill on fork PR | `prs.github_pr` populated within one cron cycle (≤2 min) of mirror PR creation | sqlite3 SELECT after sync |
| 5 | Contributor force-pushes fork mid-eval | sync-mirror logs warning, eval continues against pre-rebase state (documented behavior, not regression) | journalctl |
| 6 | Re-running merge on already-merged PR | --no-ff fails cleanly (already up to date), caller handles as no-op | manual replay |

Test 1 and 2 are the critical-path tests. 3 verifies the rejection path didn't
regress. 4 isolates the github_pr backfill fix. 5 is acceptance criteria for
the documented edge case. 6 is idempotency.

**Production smoke test:** after deploy, manually create a tiny test PR from a
secondary GitHub account. Walk it through the full lifecycle. Tear down before
hackathon. Cost: 5 minutes, catches integration-level issues that unit tests
miss.

## Backout procedure

If Option 2 misbehaves in production, revert path is one config-line toggle:

```python
# lib/merge.py — gate on a feature flag for fast disable
if branch.startswith("gh-pr-") and config.EXTERNAL_PR_NO_FF_MERGE:
    merge_fn = _merge_no_ff_external(branch)
elif ...
```

Default flag value: `True` after deploy. Set to `False` via env var on VPS to
fall back to cherry-pick path immediately if anything breaks. No code revert
required for a fast cutout.

Forgejo→GitHub merge commits already in main when the flag flips can't be
un-merged (they're real commits), but the failure mode is the same as today:
GitHub PR shows merged because the SHA is in history. No worse than current.

## Migration / cleanup

**FwazB's PR #90:** existing artifact, can't retroactively un-cherry-pick.
Manual close with explanatory comment (Ship's option b). Cory-approved.

```
We've merged your claim into the knowledge base via cherry-pick (commit f6a59d7d
on main). Future external PRs will use `git merge --no-ff` so the GitHub merge
badge fires correctly. This PR is being closed manually as the one historical
case before the fix lands. Thanks for the contribution!

— LivingIP pipeline
```

Posted via `_post_comment` + `_close_github_pr` from a one-off script
(`scripts/close-fwazb-pr-90.py`).

## Implementation order

### Phase 1 — self-healing sweep in sync-mirror.sh (Bug #2, ~10 lines)

**Single block in `deploy/sync-mirror.sh`:**

- Inserted as **Step 0** — runs after the initial Forgejo+GitHub fetch but
  **before** the branch-mirror loop and the auto-create-PR block (line ~250)
- SELECT `prs` rows where `branch LIKE 'gh-pr-%' AND github_pr IS NULL`
- Parse PR number from branch name (regex on `gh-pr-{N}/...`)
- UPDATE `github_pr` and `source_channel='github'`
- Audit-friendly log line per healed row

**Properties:**
- Idempotent (SELECT empty when clean)
- Zero-cost path when no rows match
- No API calls (branch name is the source of truth)
- Self-healing: same SELECT/UPDATE pattern recovers from race AND backfills
  historical orphans (PR 4066 picked up on first cron tick post-deploy)
- Same-cycle convergence: a freshly-mirrored PR whose Step 4.5 link UPDATE
  fails gets healed on the next cron tick's sweep, not delayed multiple cycles

**Branch:** `epimetheus/external-merge-flow-bug2` (revision in progress on
existing `epimetheus/external-merge-flow-design`).

**Deploys independently** of Phase 2 — no dependency on Bug #1 fix. Once Phase 1
is live, comments + close fire correctly even on cherry-pick-merged PRs (the
half-fix state). Phase 2 then layers in the "merged" badge.

**Smoke test (cost: 1 cron tick = 2 min):** after deploy, verify FwazB's
`prs.github_pr` populates from NULL → 90, then `_get_github_pr` resolves on
next merge action.

### Phase 2 — merge.py --no-ff for gh-pr-* (Bug #1, ~120 lines)

- Add `_merge_no_ff_external` function (architecture unchanged from v1)
- Add branch-prefix dispatch case in `_merge_domain_queue`
- Add config flag `EXTERNAL_PR_NO_FF_MERGE` for backout
- Branch: `epimetheus/external-merge-flow-bug1`
- Depends on Phase 1 being live (otherwise `on_merged` still no-ops on fork PRs)
- Smoke: end-to-end test PR

### Phase 3 — FwazB cleanup (~10 lines)

- Manual one-off script for PR #90 (option b: explanatory comment + close)
- Independent of Phase 1/2 deploy
- Can run any time after Phase 1 lands (Phase 1 populates github_pr=90 on PR 4066,
  enabling `_get_github_pr` to resolve for the comment script)

**Two phases, separately reviewable, separately deployable.** Phase 1 alone gives
us contributor comments on closed-via-cherry-pick PRs (already partially solves
the UX). Phase 2 adds the "merged" badge. Phase 3 is post-Phase-1 cleanup.

## Locked decisions

**Phase 1 (resolved in Msg 2):**
- `.fork-pr-map` file location → moot, sweep replaces map approach
- Phase 1 vs Phase 2 sequencing → separate deploys, sweep-only Phase 1
- One-shot backfill script → dropped, sweep IS the backfill
- Sweep placement → first action after initial fetch, before branch-mirror loop
  AND auto-create-PR block

**Phase 2 (resolved in Msg 2 / Ship reply):**

1. **Backout flag:** `EXTERNAL_PR_NO_FF_MERGE` config flag included, default
   `True` after deploy. One config branch in dispatch is cheap insurance with
   hackathon timing risk one week out.

2. **Rebase-after-fix scope:** silent-ignore as documented in §"Edge case".
   Eval-reset semantics already handle the recovery path — no structured
   alert in this scope. Re-evaluate after first 5-10 external contributors.

3. **Merge commit message format:** verbose with PR number —
   `"Merge external GitHub PR #{N}: {branch_slug}"`. Branch slug derived from
   the `gh-pr-{N}/` prefix strip; already in scope at merge time, no claim-file
   read needed. Searchability via `git log --merges --grep "#90"` is the value.

Ganymede gets line-level review of the Phase 1 sweep code on
`epimetheus/sync-mirror-self-heal` once Ship signs off on this revision.