The agent starts from an empty project and has to build a whole product, one feature at a time. That is how Replit measures ViBench 2.0.
Fact
The facts: on 28 October 2026, Replit published ViBench 2.0, a benchmark for long-horizon software development. It has 17 products, 241 feature requests and 544 test plans. A grading agent runs the plans in a real browser. By Replit's data, GPT-6.1 Sol leads with a pass@1 of 65%, followed by GPT-6 Astra at 60% and Opus 5.5 at 49%; pass@1 is the share of builds in which every test plan passes. DeepSeek V4.1 Flash passes 77% of its test plans at about 2.50 dollars per build, and no open-weight model passes a full build. 8 of the 17 products are public on GitHub (the vibench-public repository), 9 are held out.
Two numbers in this story must not be mixed. At first glance DeepSeek V4.1 Flash, with its 77%, beats GPT-6.1 Sol at 65%. It doesn't. 65% is the share of whole builds where everything passed, and 77% is a share of single test plans.
77% of the plans is not 77% of the product.
Eight of the 17 products are public on GitHub, nine are held out. Anyone can run the public ones. Until someone else does, the numbers are Replit's.
Official primary sources
Replit - ViBench 2.0: A Benchmark for Long-Horizon Software Building, 28.10.2026The visual is generated code-art, no third-party images.