Anthropic’s latest AI model, Claude 3.5 Sonnet (New), can now do something remarkably, use a computer like humans.
We’re talking about clicking buttons, moving the cursor, and even filling out forms – all by itself.
How Does It Actually Work?
Let’s break down what’s happening under the hood. Claude 3.5 isn’t just a Language Model anymore – it’s what we call an “AI Agent.” Instead of just understanding and generating text, it can:
- See and understand computer interfaces
- Move cursors and click buttons
- Fill out forms
- Navigate through applications
- Perform complex sequences of actions
A Real-World Example
Imagine asking Claude to “clean up unnecessary files.” Here’s how it breaks down the task:
- Opens File Explorer
- Navigates to the right folders
- Identifies unnecessary files
- Selects and deletes them
- Empties the Recycle Bin
And it does all this by actually interacting with your computer’s interface, just like you would.
The Numbers Don’t Lie
In benchmark tests by OSWorld (which measures how well AI can use computers like humans):
- Claude 3.5 scored 14.9% on basic computer tasks
- This is nearly double the score of the next best AI model (7.8%)
- For complex operations, it reached 22.0%
While these numbers might seem low compared to human performance, they represent a significant breakthrough in AI capabilities.
Current Limitations
Let’s be real – Claude isn’t perfect yet. It struggles with:
- Scrolling smoothly
- Dragging and dropping
- Some complex mouse movements
- Certain types of interactive elements
But remember: this is just the beginning.
Want to Try It Yourself?
Developers can access these features through the Claude 3.5 Computer Use API. While it’s recommended to start with low-risk tasks, the possibilities for innovation are endless.