Full Breakdown
Android Bench 2.0 Expands AI Evaluation to Multi-Day Development Tasks
By Drooid · · How we work
Core Upgrade: Long-Horizon Tasks and Agentic Evaluation
Google has launched Android Bench 2.0, a major revision of its benchmark for large-language models (LLMs) that assist Android developers. The new version adds “long-horizon tasks” (LHTs) that typically require an engineer several days or a week to complete, such as adding new features, building apps from scratch, upgrading dependencies, or converting cross-platform applications to Android. In parallel, the benchmark now incorporates “agentic evaluation,” running each model with its corresponding development agent (e.g., Gemini 3.8 Flash on Google Antigravity, GPT 5.6 Sol on Codex).
Expanded Scope Compared with the Original Benchmark
The first iteration of Android Bench focused on incremental changes—bug fixes and small feature requests—reflecting the capabilities of early AI coding assistants. By contrast, Android Bench 2.0 targets tasks that involve extensive architectural decisions, multiple files, and runtime validation. The dataset spans more than 125 files and over 8,000 lines of code, allowing the benchmark to surface strengths such as deterministic transformations (e.g., Java-to-Kotlin conversion, Retrofit-to-Ktor swaps, ViewModel layer insertion) and weaknesses such as handling missing dependency-injection graphs, breaking framework changes, or unreleased libraries.
Continuous Scoring Replaces Binary Pass/Fail
Google explains that binary pass/fail grading obscures partial successes on complex tasks. Android Bench 2.0 therefore uses a continuous “completion rate” that aggregates functionality, visual fidelity, regression avoidance, and penalties for deviating from evaluation instructions or structural constraints. This approach yields more granular signals for both model developers and Android engineers.
Early Performance Results
The updated leaderboard lists several frontier models. OpenAI’s GPT-6 Astra leads with a 28 % pass rate, while other top models—including Gemini 3.8 Flash, Gemini 3.7 Flash, Anthropic’s Fable 5.1, Kimi K3, and Qwen 3.8 Max—reach at most an 80 % completion rate on the most demanding tasks. No model achieves a 100 % pass rate on cross-platform app porting, underscoring the ongoing challenge of fully automated migration.
Outlook and Community Involvement
Matthew McCullough, VP of Product Management for Android Developer, notes that the benchmark will continue to evolve, with plans to add more model-agent combinations and update the leaderboard in the coming weeks. Google invites developers to provide feedback via GitHub and social channels such as X and LinkedIn, emphasizing that community input directly shapes future iterations of Android Bench.