extraction quality: trust hierarchy + verified tagging + telegram review endpoint
Some checks are pending
CI / lint-and-test (push) Waiting to run

Three fixes for conversation-sourced claim quality:

1. Trust hierarchy in extraction prompt: bot-generated numbers are
   flagged as unverified context, not evidence. Directional claims
   are extractable but specific figures require external verification.
   Prevents laundering bot guesses into the KB as evidence.

2. Conversation-sourced claims tagged with verified: false and
   source_type: conversation in frontmatter. Downstream consumers
   (Leo, dashboard) can filter/flag these for verification.

3. GET /api/telegram-extractions endpoint for daily spot-checking.
   Shows recent Telegram-sourced PRs with claim titles, status,
   merge rate, and eval issues. Quick review surface.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
m3taversal 2026-04-16 12:38:31 +01:00
parent c8a08023f9
commit 716cc43890
3 changed files with 99 additions and 2 deletions

View file

@ -1109,6 +1109,79 @@ async def handle_pr_lifecycle(request):
conn.close() conn.close()
# ─── GET /api/telegram-extractions ───────────────────────────────────────
async def handle_telegram_extractions(request):
"""Review surface for Telegram conversation extractions.
Shows recent PRs sourced from Telegram conversations with claim titles,
status, and source info. Designed for quick daily spot-checking.
Query params:
days (int): lookback window (default 7, max 90)
"""
conn = request.app["_get_conn"]()
try:
days = min(int(request.query.get("days", "7")), 90)
day_filter = f"-{days}"
# Find PRs from Telegram sources (source_path contains 'telegram' or submitted_by is @m3taversal via bot)
rows = conn.execute(
"""SELECT p.number, p.agent, p.domain, p.tier, p.status,
p.created_at, p.merged_at, p.description, p.source_path,
p.submitted_by, p.branch, p.eval_issues, p.leo_verdict
FROM prs p
WHERE (p.source_path LIKE '%telegram%' OR p.source_path LIKE '%futardio%')
AND p.created_at > datetime('now', ? || ' days')
ORDER BY p.number DESC""",
(day_filter,),
).fetchall()
prs = []
for r in rows:
desc = r["description"] or ""
claim_titles = [t.strip() for t in desc.split("|") if t.strip()] if desc.strip() else []
issues = None
if r["eval_issues"]:
try:
issues = json.loads(r["eval_issues"]) if isinstance(r["eval_issues"], str) else r["eval_issues"]
except (json.JSONDecodeError, TypeError):
pass
prs.append({
"number": r["number"],
"agent": r["agent"],
"domain": r["domain"],
"tier": r["tier"],
"status": r["status"],
"created_at": r["created_at"],
"merged_at": r["merged_at"],
"claim_titles": claim_titles,
"source_path": r["source_path"],
"submitted_by": r["submitted_by"],
"eval_issues": issues,
"leo_verdict": r["leo_verdict"],
})
# Summary stats
merged = sum(1 for p in prs if p["status"] == "merged")
closed = sum(1 for p in prs if p["status"] == "closed")
open_prs = sum(1 for p in prs if p["status"] == "open")
return web.json_response({
"days": days,
"total": len(prs),
"merged": merged,
"closed": closed,
"open": open_prs,
"merge_rate": round(merged / len(prs) * 100, 1) if prs else 0,
"prs": prs,
})
finally:
conn.close()
# ─── Registration ────────────────────────────────────────────────────────── # ─── Registration ──────────────────────────────────────────────────────────
def register_dashboard_routes(app: web.Application, get_conn): def register_dashboard_routes(app: web.Application, get_conn):
@ -1125,3 +1198,4 @@ def register_dashboard_routes(app: web.Application, get_conn):
app.router.add_get("/api/trace/{trace_id}", handle_trace) app.router.add_get("/api/trace/{trace_id}", handle_trace)
app.router.add_get("/api/growth", handle_growth) app.router.add_get("/api/growth", handle_growth)
app.router.add_get("/api/pr-lifecycle", handle_pr_lifecycle) app.router.add_get("/api/pr-lifecycle", handle_pr_lifecycle)
app.router.add_get("/api/telegram-extractions", handle_telegram_extractions)

View file

@ -215,7 +215,7 @@ def _parse_extraction_json(text: str) -> dict | None:
return None return None
def _build_claim_content(claim: dict, agent: str) -> str: def _build_claim_content(claim: dict, agent: str, source_format: str | None = None) -> str:
"""Build claim markdown file content from extraction JSON.""" """Build claim markdown file content from extraction JSON."""
today = date.today().isoformat() today = date.today().isoformat()
domain = claim.get("domain", "") domain = claim.get("domain", "")
@ -265,6 +265,9 @@ def _build_claim_content(claim: dict, agent: str) -> str:
lines.append(f"scope: {scope}") lines.append(f"scope: {scope}")
if sourcer: if sourcer:
lines.append(f'sourcer: "{sourcer}"') lines.append(f'sourcer: "{sourcer}"')
if source_format and source_format.lower() == "conversation":
lines.append("verified: false")
lines.append("source_type: conversation")
lines.extend(edge_lines) lines.extend(edge_lines)
lines.append("---") lines.append("---")
lines.append("") lines.append("")
@ -401,7 +404,7 @@ async def _extract_one_source(
filename = Path(filename).name # Strip directory components — LLM output may contain path traversal filename = Path(filename).name # Strip directory components — LLM output may contain path traversal
if not filename.endswith(".md"): if not filename.endswith(".md"):
filename += ".md" filename += ".md"
content = _build_claim_content(c, agent_lower) content = _build_claim_content(c, agent_lower, source_format=source_format)
claim_files.append({"filename": filename, "domain": c.get("domain", domain), "content": content}) claim_files.append({"filename": filename, "domain": c.get("domain", domain), "content": content})
# Build entity file contents # Build entity file contents

View file

@ -178,6 +178,26 @@ casual or too specific to the conversation context).
When the AI agent drops its confidence score after a correction, that CONFIRMS the human When the AI agent drops its confidence score after a correction, that CONFIRMS the human
was right. Low confidence (0.3-0.5) after pushback = strong signal the correction is valid. was right. Low confidence (0.3-0.5) after pushback = strong signal the correction is valid.
### Trust hierarchy for numbers and specifics
**CRITICAL:** Neither the human NOR the AI agent should be treated as authoritative sources
for specific numbers, dates, dollar amounts, or statistics UNLESS they cite a verifiable
external source (on-chain data, official announcements, published reports).
- **Bot-generated numbers are ALWAYS unverified.** When the AI agent says "$25.6M committed
capital" or "15x oversubscription" — these are the bot's best guess, NOT verified data.
NEVER extract bot-generated numbers as evidence in a claim.
- **Human-asserted numbers are ALSO unverified** unless they cite a source. "It raised $11.4M"
from the human is a claim about a number, not proof of the number.
- **Extract the DIRECTIONAL insight, not the specific figures.** "Curated launches attracted
significantly more committed capital than permissionless launches" is extractable.
"$25.6M vs $11.4M" is not unless the conversation cites where those numbers come from.
- **If specific figures are important to the claim, flag them.** Add a note in the claim body:
"Note: specific figures cited in conversation require verification against on-chain data."
The goal: capture WHAT the human is asserting (the mechanism, the direction, the pattern)
without laundering unverified numbers into the knowledge base as if they were evidence.
### Anti-circularity rule ### Anti-circularity rule
If the AI agent is simply reflecting the human's thesis back (restating what the human said If the AI agent is simply reflecting the human's thesis back (restating what the human said