Compare commits

...

2 commits

Author SHA1 Message Date
517e9884cc fix(substantive_fixer): WARN on corrupt eval_issues JSON
Some checks are pending
CI / lint-and-test (push) Waiting to run
Third silent return path in substantive_fix_cycle — JSON-decode except
at the eval_issues parse drops rows that don't reach skipped_no_tags
or substantive_rows. If all 3 LIMIT-3 candidates have corrupt JSON,
cycle returns 0,0 with no log entry.

WARN level (not INFO): corrupt JSON is abnormal (post-merge column
drift, hand-edited DB row, partial write during crash). If this fires,
ops want to chase the upstream column-write path. If it never fires,
baseline noise stays at zero.

Closes the visibility gap on ALL silent returns in this function, not
just the two patched in 3f8666e.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 18:33:08 -04:00
3f8666ee0c fix(substantive_fixer): surface silent-skip reasons at INFO
Two silent paths in substantive_fix_cycle masked a 13-day stall:

1. Filter strips all candidates → return 0,0 with no log. With LIMIT 3
   ordered created_at ASC, if the oldest 3 have no fixer-actionable tags
   (e.g. eval_issues=[] from leo:skipped+domain:request_changes), the
   cycle silently picks the same head-of-line every tick.

2. _fix_pr early-returns logged at DEBUG only — invisible without
   fleet-wide DEBUG. Skip reasons (no_claim_files, no_review_comments,
   not_open lock, worktree_failed, etc.) never surfaced in journalctl.

Patch: log skipped candidate eval_issues when no actionable rows
found (path 1); promote DEBUG→INFO for per-PR skip reasons (path 2).
Zero behavior change — observability only.

Diagnosis context: 98 PRs stuck >3d, last successful substantive_fixer
event 2026-04-24. Need journal evidence to choose between (a) one-line
fix to the cycle, (b) larger _fix_pr regression. (Ship Step 2 directive.)
2026-05-07 11:58:22 -04:00

View file

@ -539,15 +539,36 @@ async def substantive_fix_cycle(conn, max_workers=None) -> tuple[int, int]:
# Filter to only PRs with substantive issues (not just mechanical)
substantive_rows = []
skipped_no_tags = []
for row in rows:
try:
issues = json.loads(row["eval_issues"] or "[]")
except (json.JSONDecodeError, TypeError):
# Corrupt JSON in eval_issues is abnormal (post-merge column drift,
# hand-edited row, partial write during crash). WARN so ops can chase
# the upstream column-write path. Without this, the row drops out of
# both substantive_rows and skipped_no_tags — the third silent path.
logger.warning(
"PR #%d: corrupt eval_issues JSON — skipping in substantive fix cycle",
row["number"],
)
continue
if set(issues) & (FIXABLE_TAGS | CONVERTIBLE_TAGS | UNFIXABLE_TAGS):
substantive_rows.append(row)
else:
skipped_no_tags.append((row["number"], issues))
if not substantive_rows:
# Visibility for the LIMIT-3 head-of-line block: if the oldest
# candidates have no fixer-actionable tags (e.g. eval_issues=[],
# broken_wiki_links only), the cycle silently returns 0 — and the
# next cycle picks the same head-of-line, forever. Log the eval_issues
# of skipped candidates so the journal makes the block visible.
if skipped_no_tags:
logger.info(
"Substantive fix cycle: 0 actionable from %d candidate(s) — head-of-line: %s",
len(rows), skipped_no_tags,
)
return 0, 0
fixed = 0
@ -559,7 +580,13 @@ async def substantive_fix_cycle(conn, max_workers=None) -> tuple[int, int]:
if result.get("action"):
fixed += 1
elif result.get("skipped"):
logger.debug("PR #%d: substantive fix skipped: %s", row["number"], result.get("reason"))
# Was DEBUG — promoted to INFO to make stuck-PR root cause
# visible without enabling DEBUG fleet-wide. (Ship Apr 24+
# silent skip diagnosis.)
logger.info(
"PR #%d: substantive fix skipped: %s",
row["number"], result.get("reason"),
)
except Exception:
logger.exception("PR #%d: substantive fix failed", row["number"])
errors += 1