How Vegapunk works
Vegapunk takes a GitHub issue URL and produces a pull request end-to-end. A LangGraph pipeline in a FastAPI backend runs seven steps and streams typed events to this frontend over Server-Sent Events. Everything below is what makes that work — no code reading required.
The 7 pipeline steps
Both retry loops (Tester → Coder on new test failures, Reviewer → Coder on rejection) cap at max_retries so the pipeline can't spin forever.
1. End-to-end sequence
The request timeline from click to PR link, across every process boundary. Coder invokes K parallel LLM calls (Best-of-N — detail in the next diagram).
2. Best-of-N Coder — the quality-lifting mechanic
Instead of trusting a single LLM sample, the Coder generates K parallel candidates at different temperatures, verifies each against real tests in an isolated git worktree, and keeps the winner. Rooted in the DeepSWE and ACECoder research on execution-verified rewards.
K is configurable via CODER_BON_K (default 3; set to 1 to disable and use the fast path).
3. Repo graph — the retrieval mechanic
Before the Planner writes anything, we build a tree-sitter graph of the workspace and rank files by a hybrid of keyword overlap and PageRank on the reference graph. Grounded in a 2026 study reporting ~10× token reduction over regex scans on real repos.
What runs where
Frontend (this app)
Next.js 16 + React 19 on Vercel. Consumes the backend's SSE stream and renders each step as a live-updating card. No stateful backend on the frontend itself.
Backend (FastAPI on Render)
Hosts the LangGraph pipeline, the event bus, and the MCP server. Free tier sleeps after 15 min of inactivity — first request after wake takes ~30 s.
LLM providers
NVIDIA NIM primary, Google Gemini fallback. Auto-failover on rate-limit or 5xx so a single provider going down doesn't kill a run.
MCP server
Optional. Exposes tools + the repo graph as MCP resources so Claude Code / Cursor / Cline can drive Vegapunk without a custom SDK. Launched via vegapunk-mcp.
Evidence — measured performance
Every number below comes from a reproducible script in benchmarks/ — run make bench to regenerate them locally. Values here are from the run at git 3105092. Full detail, per-query breakdowns, and statistical framing in docs/benchmarks.md.
Retrieval — tree-sitter graph vs regex baseline
9 hand-annotated queries across the mini_py fixture and the Vegapunk repo itself. Graph is compared against the legacy ripgrep-based retrieval the pipeline used before Phase 1.
Graph build performance
Cold-build wall-clock (cache cleared) over 30 runs per workspace. 95% CI via student-t.
End-to-end pipeline determinism
10 fresh runs of pytest tests/test_pipeline_e2e.py. Real git worktrees + tree-sitter + pytest subprocess; only LLM / GitHub API / git push mocked.
Static gates
Enforced on every push + PR via GitHub Actions matrix on Python 3.11 + 3.12 (backend) and Node 20 (frontend).
Comparison to cited research
| Claim | Source | Our measurement | Verdict |
|---|---|---|---|
| ~10× token reduction (tree-sitter graph vs regex) | Codebase-Memory, 2026 | 2.92× mean, 12.85× max | smaller but significant (p = 0.015) |
| Sub-second repo-graph build on typical repos | Aider repomap docs | 68.5 ms mean on Vegapunk | 15× better |
| Best-of-N beats single-shot via execution-verified reward | DeepSWE, ACECoder | selector 12 / 12 correct on unit tests + real E2E | mechanic verified (see caveats) |
| MCP as universal tool plane for coding agents | MCP protocol adoption (~9k servers) | 7 tools + 6 URI resources; 21 / 21 tests pass | shipped |
Honest read: the 10× token headline from Codebase-Memory 2026 was measured on 31 diverse repos. Our 9-query sample on this single codebase produces a smaller mean but the reduction is statistically significant and the best case matches the research ballpark. Direct SWE-bench-style cross-repo evaluation is a documented follow-up.