How we measure

1. Why This Page Exists

Skeptical buyers should not have to reverse-engineer marketing copy. Maguyva indexes repositories and surfaces precision, language coverage, and cost claims across the site. Those claims need a methodology surface that is honest about measurement scope: what is fixture-backed, what is judgment-backed, and what is framing rather than independent certification.

2. What We Measure

Code intelligence quality is measured primarily on the language engine — extraction of symbols, relationships, and graphs from source — not on subjective “agent happiness” scores.

• Per-language fixture suites: expected edges and symbols the handler must extract correctly
• Release gates: languages ship only when fixture precision, recall, and F1 clear published thresholds (precision ≥ 0.95, recall ≥ 0.99, F1 ≥ 0.97 on fixtures, with a minimum edge count for statistical confidence)
• Language-audit dimensions: accuracy, structural integrity, completeness, quality, and performance on the validation board
• Corpus and spot-check loops: sampled real-repository edges classified by rubric (correct, false positive, type/scope/metadata error) — described in our language-grind blog posts
• Product capability flags: what the server advertises (AST, locals, graph extraction) separate from catalog size

Important: Fixtures validate against fixtures we wrote. GREEN means the known cases pass. It does not automatically mean every real-world idiom extracts cleanly. That distinction is deliberate and public.

3. Green vs Independently Verified

The language-audit board uses multiple axes so a single green light cannot be read as “proven perfect.” Headline numbers on the board are typically split:

• overall_green — regression-clean against self-snapshot fixtures and related corpus gates (necessary, not sufficient)
• independently_verified — has a strong judgment signal such as a spot-check seed (includes languages that still show judged errors)
• verified_clean / zero judged error on sampled edges — a stricter subset of judged languages
• curation / oracle trust — whether fixtures themselves are treated as trusted oracles
• structural flags — graph extraction and AST capability are not identical to catalog membership

Marketing language count (for example “279+ languages”) is catalog size: configured languages and server entries. Catalog size is not an AST quality SLA. Prefer tiered reporting over a single vanity headcount. For the current product surface, see Compatibility and Language Guides; for narrative detail, see the language recursive self-improvement post on the blog.

4. Search and Retrieval Quality

Semantic search quality is multi-modal: text, AST, graph, and embeddings are fused. We document deliberate engineering trades rather than claiming unbeatable retrieval:

• Embeddings use a commercial model family (voyage-4-large as of the June 2026 blog snapshot) chosen against public retrieval leaderboards at the time of decision
• Vectors are binary-quantized for storage and cost; that deliberately trades some retrieval precision for cheaper, faster Hamming search without a separate vector database
• Intent routing and fusion weights are engineered heuristics with measured operational effects (for example lower zero-result rates after intent routing), not a published independent IR evaluation suite on customer corpora
• Blog posts include “what’s still imperfect” sections — imperfect signals are part of the record, not footnotes to hide

5. Cost Claims

Cost language on Maguyva is about pricing structure and operating transparency, not a formal TCO study certified by a third party.

• Workspace pricing: billed by repositories, indexed lines, and rebuild frequency — not per human or agent seat. FAQ and plan copy spell out the dimensions.
• “Roughly 10–30x less” style comparisons on Pricing are illustrative math against typical per-seat ranges, depending on which seat-priced tool you compare. They are not a locked independent competitive benchmark package.
• Operating cost honesty: the /team page publishes a real monthly software burn breakdown (subscriptions, MCP/search tooling, usage-shaped costs). That is customer-zero transparency, not an audited financial statement.
• Embedding and infra costs are accepted product costs (premium embeddings, storage, graph rebuilds). We pay them on purpose and say so in the voyage-4-large post and pricing narrative.

6. What We Do Not Claim

This page is also a list of non-claims. If something is not on the measurement board, do not treat marketing tone as proof.

• No blanket absolute-precision guarantee for every language. Fixture thresholds apply per language on known cases; residual real-world error is expected and ground.
• No claim that overall_green equals production-perfect extraction for every repository idiom
• No third-party compliance certification package asserted as a methodology artifact on this surface (see Security for data-handling facts, not compliance badges)
• No independent multi-vendor bake-off with shared corpora published as a standing scorecard
• Search results and analysis remain best-effort under Terms of Service — Maguyva is not a substitute for code review, testing, or security audits

7. How to Verify Yourself

The intended buyer motion is still: index a repo you already understand, ask one real question, and inspect citations.

• Start free: small representative repos beat a whole-company index on day one
• Use MCP tools (intelligent_search, find_symbol, dependency_search) and open the cited paths
• Read How It Works for the ingestion and retrieval architecture
• Read Compatibility and Language Guides for capability tiers, not just catalog size
• Read Security and Privacy for data handling; this page does not replace them