The Aider Polyglot benchmark evaluates AI models on the task that most closely mirrors how developers actually use AI coding assistants: editing existing code across multiple programming languages. Created and maintained by Paul Gauthier as part of the Aider open-source coding assistant project, the benchmark tests 133 Exercism coding exercises spanning Python, JavaScript, TypeScript, C#, Java, Go, Rust, C++, PHP, Ruby, Swift, and other languages. Each task requires the model to modify existing code files using Aider’s structured diff format, then verifies correctness by running the exercise’s test suite. This combination of multi-language coverage, code editing (not just generation), and structured output adherence makes Aider Polyglot the most practically relevant coding benchmark for developers choosing an AI assistant for daily work.
What Aider Polyglot Measures
Aider Polyglot measures three capabilities simultaneously: the ability to write correct code across diverse programming languages, the ability to edit existing code rather than generating from scratch, and the ability to express those edits in a structured diff format that tooling can apply. This combination is significant because it mirrors the actual workflow of AI-assisted development — a developer asks their coding assistant to modify an existing file, and the assistant must produce changes in a format that can be cleanly applied.
The 133 exercises are drawn from Exercism, a platform for practicing programming skills. Each exercise provides a code file that needs modification and a test suite that the modified code must pass. Exercises cover a range of programming concepts: string manipulation, data structures, algorithms, object-oriented design, error handling, and language-specific idioms. The difficulty ranges from straightforward (implement a simple function) to moderately challenging (implement a complex data structure with specific performance characteristics).
The polyglot dimension is what sets this benchmark apart from Python-only evaluations like HumanEval or SWE-bench. Models must handle statically-typed languages (Rust, Go, C++, Java, C#) where the type system and memory model constrain valid implementations, dynamically-typed languages (Python, Ruby, PHP, JavaScript) where different idioms apply, and languages with unique features (Swift’s optionals, Rust’s ownership model, Go’s goroutines) that require language-specific understanding. A model’s aggregate score masks significant per-language variation — many models that perform well on Python struggle with Rust or C++.
The structured edit format requirement adds another dimension of evaluation. Models must not only produce correct code but express their changes in Aider’s diff format — specifying which file to modify, which lines to replace, and what to replace them with. A model that generates correct code but fails to format it as a proper diff scores zero on that task. This tests instruction following and format compliance alongside coding ability.
Current Leaderboard Analysis
The Aider Polyglot leaderboard shows Anthropic’s models in a dominant position. Claude Opus 4 leads at 82.7%, with Claude Sonnet 4 in second place at 76.4% — a 6.3-point gap that is significant but smaller than the gap between Sonnet and the rest of the field. Gemini 2.5 Pro (72.9%) and GPT-4.1 (70.5%) form a competitive third tier, followed by DeepSeek V3 (65.4%), GPT-4o (62.8%), and Grok-3 (60.1%).
The 22.6-point spread from top to bottom is the widest of any major coding benchmark, even more discriminative than SWE-bench Verified (34-point spread, but fewer tracked models above the floor). This spread confirms that multi-language code editing remains a challenging and differentiating capability.
Anthropic’s lead is not marginal — Claude Opus 4’s 82.7% is 9.8 points ahead of third-place Gemini 2.5 Pro, and even the “smaller” Claude Sonnet 4 outperforms every non-Anthropic model. This gap reflects consistent strengths in two areas: structured output adherence (correctly formatting edits as diffs) and multi-language code understanding (generating correct implementations across diverse languages and paradigms).
The ordering on Aider Polyglot diverges meaningfully from other benchmarks. GPT-4.1 leads MMLU and is competitive on most benchmarks, but falls to fourth place here. Grok-3, which scores well on HellaSwag and MT-Bench, trails significantly at 60.1%. These divergences indicate that code editing across languages tests a distinct capability dimension that does not simply track with general model quality.
Methodology Deep Dive
The benchmark uses 133 Exercism exercises distributed across 12+ programming languages. The exact distribution varies somewhat between benchmark runs as Gauthier updates the exercise selection, but it typically includes approximately 10-15 exercises per language, covering Python, JavaScript, TypeScript, C#, Java, Go, Rust, C++, PHP, Ruby, and Swift, with occasional additional languages.
Each exercise provides: the existing code file(s) that need modification, the exercise instructions describing the required functionality, and a test suite that the modified code must pass. The model receives the code and instructions as context and must produce edits in Aider’s diff format — either the “search/replace” format (specifying exact text to find and replace) or the “whole file” format (providing the complete modified file).
Scoring is binary per exercise: the model’s edits are applied to the code files, the test suite is run, and the exercise passes if all tests succeed. The aggregate score is the percentage of exercises where all tests pass. There is no partial credit for nearly-correct implementations or for edits that pass some but not all tests.
The benchmark is run through the Aider tool itself, using each model’s API. This means the evaluation captures not just the model’s coding ability but its performance through the specific prompt templates and interaction patterns that Aider uses. Since Aider is one of the most popular open-source coding assistants (with millions of downloads), this evaluation setting directly predicts the experience developers have when using these models through Aider.
Gauthier runs and publishes results regularly, updating scores as new models are released. The benchmark is fully reproducible — the Exercism exercises, test suites, and Aider tool are all open source, and any developer can run the evaluation independently to verify results.
Why This Benchmark Matters
Aider Polyglot matters because it is the most direct measure of how well a model performs as an AI coding assistant in real developer workflows. Unlike HumanEval (which tests isolated function generation in Python) or SWE-bench (which tests bug fixing in Python repositories), Aider Polyglot tests the specific task that AI coding tools do most often: editing existing code across the languages developers actually use.
The multi-language dimension is critical. Professional software development rarely involves a single language. A typical project might use TypeScript for the frontend, Go or Rust for the backend, Python for data processing, and C++ for performance-critical components. A coding assistant that excels at Python but stumbles on Rust or Go is significantly less useful than one that performs consistently across the polyglot stack. Aider Polyglot quantifies this cross-language consistency in a way no other benchmark does.
The edit format requirement captures a capability that pure code generation benchmarks miss. In practice, AI coding assistants must produce changes that integrate cleanly into existing codebases — they cannot simply output a complete new file for every change. The ability to express precise, minimal edits in a structured format is essential for practical AI-assisted development, and Aider Polyglot tests it explicitly.
Because the benchmark runs through the Aider tool — one of the most widely used AI coding assistants — its scores directly predict developer experience. A model scoring 82.7% on Aider Polyglot will successfully complete roughly four out of five code editing tasks on the first attempt, while a model scoring 60% will fail two out of five. This difference is large enough to meaningfully affect developer productivity and satisfaction.
Known Limitations and Criticisms
The benchmark’s dependence on Aider’s specific prompt templates and diff format means that scores reflect compatibility with Aider as much as raw coding ability. A model that produces excellent code but struggles with Aider’s particular edit format will score lower than one that produces adequate code in the right format. This is a feature (it tests practical tool compatibility) but also a limitation (it may not generalize to other coding tools that use different formats).
The Exercism exercises, while diverse, are self-contained programming exercises rather than real-world coding tasks. They do not test understanding of large codebases, navigating complex dependency graphs, or making changes that span multiple files — all of which are critical capabilities for professional software engineering. SWE-bench covers this dimension more thoroughly, though only in Python.
The language distribution is not perfectly balanced. Some languages have more exercises than others, and the difficulty varies by language. A model might score well overall by dominating the Python and JavaScript exercises while performing poorly on less-represented languages like Swift or Rust. Per-language breakdowns would be more informative but are not always published.
The binary scoring system (pass/fail per exercise) discards useful information. A model that fails an exercise because of a single edge case is scored the same as one that produces completely non-functional code. A more granular scoring system that counted passing tests as a fraction of total tests would provide finer-grained comparison.
The benchmark is maintained by a single individual (Paul Gauthier), which introduces concentration risk and potential bias. Exercise selection, evaluation methodology, and result publication are all under one person’s control. While Gauthier has maintained the benchmark transparently and reproducibly, the lack of institutional backing means the benchmark could change or disappear based on one person’s decisions.
How Scores Have Changed Over Time
| Year | Top Model | Score | Key Insight |
|---|---|---|---|
| 2023 | GPT-4 | 47.3% | Multi-language editing was a major challenge |
| 2024 | Claude 3.5 Sonnet | 64.0% | Anthropic models showed early strength |
| 2024 | GPT-4o | 62.8% | Competitive but trailed Claude on editing |
| 2025 | Claude Sonnet 4 | 76.4% | Major jump, Anthropic extended lead |
| 2025 | Claude Opus 4 | 82.7% | First model to exceed 80%, dominant position |
The progression from 47.3% to 82.7% in roughly two years shows steady improvement, with Anthropic models driving the most significant gains. The jump from Claude 3.5 Sonnet (64%) to Claude Opus 4 (82.7%) — an 18.7-point improvement in roughly one year — is one of the largest single-generation capability gains on any coding benchmark. This suggests that code editing ability is advancing rapidly and that further improvements are likely.
Aider Polyglot vs Other Benchmarks
| Benchmark | Languages | Task Type | Problems | Key Difference |
|---|---|---|---|---|
| Aider Polyglot | 12+ languages | Edit existing code | 133 | Multi-language editing with format compliance |
| HumanEval | Python only | Generate functions | 164 | Single-language, generation-only, saturated |
| SWE-bench Verified | Python only | Fix real-world bugs | 500 | Real repos but Python-only |
| MBPP | Python only | Simple generation | 974 | Easier than HumanEval, saturated |
| CodeContests | Multiple | Algorithmic problems | 165 | Competition-style, not practical coding |
Aider Polyglot and SWE-bench are complementary benchmarks that together provide the most complete picture of AI coding capability. SWE-bench tests deep engineering within complex Python codebases — understanding architecture, navigating large repositories, producing patches for real bugs. Aider Polyglot tests breadth — can the model write and edit correct code across the range of languages developers use? A model that scores well on both is a strong all-around coding assistant; one that excels on only one has a significant blind spot.
HumanEval has been largely superseded by Aider Polyglot for frontier model comparison. With top models scoring 92-94% on HumanEval, it provides minimal separation, while Aider Polyglot’s 22-point spread still offers strong discriminative power. The multi-language and editing dimensions make Aider Polyglot a strictly more informative benchmark for evaluating practical coding capability.
Practical Implications
For developers choosing a model for their AI coding workflow, Aider Polyglot is the most directly predictive benchmark. If you use Aider or a similar AI coding tool, the Aider Polyglot score is a near-direct prediction of your experience. Claude Opus 4 at 82.7% means approximately four out of five code edits will succeed on the first attempt. Grok-3 at 60.1% means roughly three out of five — a difference that translates to significantly more time spent reviewing, correcting, and re-prompting.
For teams working in multiple languages, the polyglot dimension is essential. A Python-focused team might not see much difference between models that all score well on Python benchmarks, but a team maintaining TypeScript, Go, and Rust codebases will feel the difference acutely. Models with lower Aider Polyglot scores tend to fall off most steeply on statically-typed and systems-level languages, precisely where code correctness matters most.
For enterprises evaluating AI coding tools for developer productivity, Aider Polyglot scores provide the most actionable metric for comparing model backends. The 12.2-point gap between Claude Opus 4 (82.7%) and GPT-4.1 (70.5%) represents a meaningful difference in developer experience — enough to affect tool adoption, developer satisfaction, and the ROI of AI coding assistant investments.
The structured edit format requirement also has practical relevance for tool developers. Models that score well on Aider Polyglot are models that can follow structured output formats reliably — a capability that matters for any tool that needs models to produce machine-parseable responses, not just for coding assistants.
Frequently Asked Questions
Why do Anthropic models dominate Aider Polyglot?
Anthropic models consistently lead on Aider Polyglot for two likely reasons: superior adherence to structured output formats (diff specifications) and strong cross-language code understanding. The Aider benchmark specifically penalizes models that generate correct code but fail to express it in the required edit format, and Claude models have demonstrated particularly strong instruction-following for structured outputs. Additionally, Anthropic appears to have invested significantly in multi-language code training, which manifests in consistent performance across Python, Rust, Go, TypeScript, and other languages.
How does Aider Polyglot relate to my daily coding experience?
Aider Polyglot is the most directly predictive benchmark for AI-assisted coding workflows. Each exercise mirrors a real coding task: you have existing code, you need to modify it, and the result needs to pass tests. If you use Aider, the benchmark score directly predicts your first-attempt success rate. For other coding tools that use different prompting or formatting, the signal is somewhat less direct but still strongly correlated with practical coding assistance quality.
Why is multi-language performance important?
Professional software projects rarely use a single language. A typical stack might include TypeScript for the frontend, Python for ML services, Go for microservices, and Rust for performance-critical components. A coding assistant that only works well in Python (which is all that HumanEval and SWE-bench test) covers only part of most developers’ needs. Aider Polyglot reveals which models can assist across the full language stack, which is essential information for teams working in polyglot environments.
Is Aider Polyglot harder than HumanEval?
Yes, significantly. The top HumanEval score is 94.5% while the top Aider Polyglot score is 82.7%. The additional difficulty comes from three sources: the code editing requirement (producing correct diffs, not just correct functions), the multi-language coverage (including languages like Rust and C++ that are harder for models than Python), and the structured format compliance (edits must be in the exact format Aider expects). A model scoring 93% on HumanEval but 60% on Aider Polyglot has strong Python generation ability but weak editing and multi-language skills.
How often is the Aider Polyglot benchmark updated?
Paul Gauthier updates the benchmark regularly as new models are released and as the Aider tool evolves. The exercise set may be adjusted between runs to improve coverage or address issues with specific exercises. Because the benchmark is maintained by one individual rather than a research institution, updates follow an informal schedule tied to major model releases. All results and methodology are published on the Aider documentation site, and the benchmark can be independently reproduced using the open-source Aider tool.