Language Recursive Self-Improvement: Grinding Code Intelligence Across ~280 Languages

Numbers in this post reflect the system at publication (May 2026). See our team page for current figures.

Maguyva extracts symbols, references, and a dependency graph from source code in roughly 280 languages. Each language is a custom tree-sitter handler — queries, heuristics, edge cases — and every handler can be subtly wrong in its own way. A method call emitted as a read. A function attributed to the wrong enclosing scope. A relationship that simply isn’t there.

You cannot hand-audit that. No team can read extraction output across 280 grammars and spot the bad edges. So the interesting question isn’t “is our extraction correct” — it’s “how do you find out that it’s wrong, at this breadth, without a human in every loop.” Our answer is language recursive self-improvement: a quality loop driven by language agents, an LLM acting as judge, and one rule we keep relearning: green is not the same as correct.

Green is not correct

Every language has a fixture suite, and a release gate scores it across five dimensions — accuracy, structural integrity, completeness, quality, and performance. A language goes GREEN only when, on its fixtures, precision ≥ 0.95, recall ≥ 0.99, and F1 ≥ 0.97, with at least 20 expected edges for statistical confidence. Below that it’s YELLOW or RED, and it doesn’t ship.

That gate is necessary and not sufficient. Fixtures validate against fixtures we wrote. They encode the cases we already thought of. A handler can be flawless on its fixtures and still mangle a pattern that only shows up in real code — a macro idiom, a generic-bound method, a language feature nobody wrote a fixture for. GREEN means the fixtures pass. It does not mean a real repository extracts cleanly. So the loop has to leave the fixtures behind and look at the wild.

The inner loop: spot-check, judge, fix, prove

The core loop runs one language at a time:

        ┌──────────────────────────────────────────────────────────┐
        │                                                          │
        ▼                                                          │
  1. corpus run ── clone real-world repos, extract relationships   │
        │                                                          │
        ▼                                                          │
  2. spot-check 100 edges (seed 42, then seed 123 to cross-check)  │
        │                                                          │
        ▼                                                          │
  3. LLM judge classifies every sampled edge:                      │
        CORRECT · FALSE_POSITIVE · TYPE_ERROR ·                    │
        SCOPE_ERROR · METADATA_ERROR                               │
        │                                                          │
        ▼                                                          │
  4. fix ONE thing — handler .py, .scm query, or config           │
        │                                                          │
        ▼                                                          │
  5. re-validate — F1 + re-classify + manifest diff (no regressions)│
        │                                                          │
   better? ──no──► revert, try a different fix ────────────────────┤
        │ yes                                                       │
        ▼                                                           │
  6. promote the fix into fixtures (a permanent regression guard) ──┘

A few things make this work rather than thrash.

The judge is the agent, not an API call. When we say “LLM-as-judge,” we mean the language agent itself reads each sampled edge against the real source and classifies it with a fixed five-category rubric: is this edge correct, a false positive, the right relationship with the wrong type, attached to the wrong scope, or carrying wrong metadata? That rubric is the whole game — “23% error rate” is meaningless until you know whether those are real defects or the judge miscounting.

Fix one thing, then prove it. Each iteration changes exactly one editable asset, then re-runs against a fixed harness and keeps the change only if F1 improves and the re-classification looks better. If it doesn’t, it reverts. No batch of speculative edits, no “should be better.” A change earns its place or it’s gone. And when a fix sticks, it’s promoted into the fixture suite — so the bug it fixed can never silently come back. That promotion step is what makes the loop recursive rather than merely repetitive: every pass hardens the oracle the next pass validates against.

The lesson we keep relearning: metrics over-report

Here’s the trap, and we walked straight into it. The secondary corpus metrics — how often an extracted target has no resolvable symbol, how many symbols look “orphaned,” and so on — massively over-report problems. They are mostly paradigm artifacts, not bugs.

The cleanest example: llvm once showed a 73% “source-without-symbol” rate and got branded catastrophic. We dug in. Real accuracy was 98.5%. The “missing symbols” were almost all legitimate external references — calls into the standard library, into frameworks, into code that lives outside the repo. The metric was measuring a property of the language, not a defect in the handler. Languages like Zig, COBOL, and Odin show 65–70% “orphan” rates and are entirely correct; COBOL had zero real errors.

If we had let those numbers drive the work, we’d have spent weeks “fixing” handlers that were already right and ignoring the languages with quiet, real bugs. The conclusion is blunt: the aggregate metrics are a rough triage signal at best. The real quality signal is the spot-check with edge classification — looking at actual edges in actual repositories and judging them one by one. Data over intuition, but only once you know which data is telling the truth.

The outer loop: a fleet, not a marathon

One language at a time would take forever across 280 of them, so the inner loop is wrapped in an outer one that runs many in parallel.

   pick a wave of near-GREEN / high-error languages
        │
        ▼
   fan out 10–15 agents, each ISOLATED in its own git worktree,
   each grinding ONE language, committing to its own branch
        │
        ▼
   orchestrator integrates serially: file-scoped apply, then
   `manifest diff` — any cross-language regression blocks the batch
        │
        ▼
   gated push (reviewed and confirmed) ──► rotate to the next wave

Each agent works in a throwaway worktree so they can’t step on each other. The orchestrator integrates their fixes one at a time, each behind a full-manifest regression check: a change that helps its own language but quietly breaks three others doesn’t land. Integration is gated and confirmed before anything is pushed — the regression gate decides what’s safe, a human still decides what ships. Then the pool rotates to the next set of languages and the whole thing runs again.

What’s still imperfect

The judge can be wrong, and wrongness has a direction: an agent running with too little context over-reports. In one batch, eight languages were flagged at 5–20% error; on inspection only one was a real per-language bug — the rest were judge mistakes from missing the language’s own semantics (one source line legitimately emitting several edges, parameter bindings modeled as assignments, reference-per-identifier languages). That’s why we sample with two seeds and cross-check, and why a disagreement between the judge and the fixtures is treated as the most interesting signal, not a settled verdict — sometimes the fixture is the thing that’s wrong.

We’re also honest about the target. The goal is zero real errors, full stop — but “zero” is a direction we grind toward, language by language, not a box that’s checked. There is always another repo with another idiom.

Earned, Not Asserted

Everything Maguyva does for an agent — finding a symbol, tracing a dependency, answering a question with cited code — rests on the extraction underneath being right. Across 280 languages, “right” can’t be asserted; it has to be continuously earned against real code. The loop is how we earn it: an autonomous spot-check-and-fix cycle that treats its own metrics with suspicion, proves every change, and turns each fix into a guard against the next regression. It’s not glamorous. It’s the work that lets us say “we support your language” and mean it. Green is easy. Correct is earned.