AI Is Still Not as Good at Using Computers as Many Think — But It’s Improving Fast
If you’ve ever watched a demo of AI opening a browser, filling out forms, moving files, or editing documents on its own, you might assume AI is close to working on a computer the way a human does.
But how do you know whether AI is actually capable—or just made to look capable in a carefully staged demo?
The answer is benchmarking. And the most credible benchmark for testing AI on real computers is OSWorld, published at NeurIPS 2024 and now widely used across the AI industry as a standard evaluation framework.
What Is OSWorld?
OSWorld (Open-ended tasks in real computer environments) is a benchmark designed to test AI agents on real computers—not simulated environments, not just multiple-choice questions, and not coding in a sandbox. Instead, the agent must perform real tasks on a computer screen the way a human would.
How is it different from other benchmarks?
Most benchmarks test AI in a "question-and-answer" format: give the model a prompt and check whether the answer is correct. OSWorld tests AI in a "do the work" format: give it a goal, and the AI must:
- See the screen (via screenshots or an accessibility tree)
- Reason about what to do to achieve the goal
- Control the mouse and keyboard to execute actions
- Verify the result and adjust its plan if needed
All of this happens on real operating systems—Ubuntu, Windows, and macOS.
How Comprehensive Is the Benchmark?
OSWorld includes 369 tasks across four main categories:
- Web Application — searching for information, filling out forms, managing email
- Desktop Application — editing documents, creating spreadsheets, adjusting images
- OS File I/O — managing files, moving folders, changing system settings
- Cross-Application Workflow — tasks that require multiple apps, such as pulling data from the web into a spreadsheet and then creating a chart
Each task includes an initial setup script to ensure every test starts from the same state, along with an execution-based evaluation script that checks not just whether the right area was clicked, but whether the final outcome is actually correct.
Surprising Results — and Progress Worth Watching
The starting point (2024): AI could do only 12%
When OSWorld launched in 2024, the results were eye-opening: the best AI system completed only 12.24% of all tasks, while humans achieved 72.36%.
That 60-point gap made one thing clear: even though AI had become strong at question answering and code generation, actually "using a computer" was still far from human capability.
Today (2026): the gap is closing
Two years later, the picture has changed significantly. On OSWorld-Verified—an improved version that fixes issues and increases reliability—the leaderboard looks like this:
| Rank |
System |
Score |
| 1 |
CoACT-1 |
60.76% |
| 2 |
Agent S2.5 (w/ o3) |
56.0% |
| 3 |
UiPath Screen Agent (Claude Opus 4.5) |
53.6% |
| 4 |
GTA1 (w/ o3) |
53.1% |
| 5 |
Claude Sonnet 4.5 |
43.9% |
| 6 |
UI-TARS |
40.0% |
| — |
Human |
~72% |
From 12% to 60% in two years—nearly a 5x improvement.
What OSWorld Reveals About AI’s Weaknesses
A detailed analysis of the results highlights several critical weaknesses:
GUI grounding is still a problem
AI still struggles to "understand" what it sees on the screen—which button is which, where menus are located, and exactly where it needs to click—especially when the UI is highly complex.
Practical know-how is still lacking
AI may "know" what LibreOffice Calc can do, but it may not know which button to press, where it is, and in what sequence to accomplish the task. The gap between theoretical knowledge and practical execution remains significant.
It is not robust to UI changes
When the screen layout changes even slightly—such as a button moving position or a window being resized—AI performance drops significantly, while humans adapt with little effort.
Resolution matters
The higher the screenshot resolution, the better AI performs. This indicates that AI still depends heavily on visual perception, and image compression has a direct impact on performance.
Agentic Frameworks Outperform Standalone Foundation Models
One of the most interesting takeaways from the latest leaderboard is this: the top-ranked systems are not single AI models, but frameworks that coordinate multiple components together.
CoACT-1, ranked first, combines a reasoning model with an agentic framework that plans, acts, and checks results in a loop. Agent S2.5 uses Experience-augmented Hierarchical Planning—learning from past task execution to improve how it plans the next one.
This confirms that "using a computer" is not just about having a smarter AI model—it requires a complete system that can plan, act, learn, and adapt.
What This Means for Organizations
Automation is about to reach a new level
As AI becomes better at "using a computer," more work that once required a person sitting at a screen will start being handled by AI agents. This goes beyond tasks with available APIs and includes work that requires clicking, typing, and drag-and-drop actions inside applications that were never designed for AI use.
UiPath, ranked third, is a clear example. The RPA giant, once known for rule-based automation, is now shifting toward AI as the core engine.
Choose AI based on evidence, not marketing
OSWorld gives organizations a way to compare AI models objectively instead of relying on vendor demos. Look at the OSWorld scores of the models you are evaluating—it will give you a much clearer view of how well that AI can actually operate on a real computer.
There is still plenty of headroom
Despite the rapid improvement, the gap between 60% and the human score of 72% still remains—and even 72% is not 100%. Some tasks are difficult enough that even humans do not always complete them successfully. That tells us AI is not yet ready to replace humans entirely, but it is already well positioned to work alongside humans as a highly effective assistant.
Looking Ahead
The jump from 12% to 60% in just two years tells us that human-level computer-using AI is no longer a question of "if," but "when."
Organizations that begin understanding the limitations and potential of AI agents today will have a clear advantage when the technology is ready—not just because of the model itself, but because workflows, data, and processes must all be prepared to support agents that can truly do the work.
Sources
