Sidecar Blog

How AI Learned to Use Your Computer (And Why That Changes Everything)

Written by Mallory Mejias | Oct 7, 2025 10:30:00 AM

Here's a problem many associations know intimately: you have software that works, sort of. Maybe it's an AMS from 2008. Maybe it's a conference management system that lacks basic features like bulk session uploads. Maybe it's that mini computer with terminal applications that predates Windows—yes, those still exist in association land, running critical functions that would be catastrophic if they stopped working.

The traditional solutions have been expensive and unsatisfying. Replace the entire system and migrate years of data. Build custom integrations if the software even has an API. Or just accept that someone on your team will spend hours each week moving data between systems by hand because there's no other option.

Until now. AI can now control computers the way humans do—moving the mouse, clicking buttons, typing text, navigating between applications. It's not perfect yet, but it's getting good enough to automate processes that were previously impossible without rebuilding everything.

Computer Use: AI That Works With What You Already Have

Claude 4.5 Sonnet just hit a 61% success rate on computer use benchmarks, up from 42% with the previous model. That's not an A+, but hey—it's passing! 

Computer use means AI can interact with software through the same interface you use. It sees what's on your screen through screenshots. It figures out where to click based on what it sees and what you asked it to do. It moves the cursor, clicks, types, waits for the screen to update, takes another screenshot, and continues until it completes the task.

Think about the humanoid robot phenomenon for a second. Companies are building human-shaped robots not to be creepy, but because we built the entire physical world for human bodies. Doorknobs are at hand height. Stairs are sized for human legs. Cars have pedals positioned for human feet. A humanoid robot can navigate all of that without requiring us to redesign everything.

Computer use works the same way. We built software interfaces for humans using keyboards and mice. Buttons are clickable. Forms have fields. Navigation makes visual sense. AI that can see and interact like a human can use any of this software without requiring special APIs or integrations.

This matters because associations are running a lot of software that lacks modern integration capabilities. Those terminal applications connecting to mini computers from the 1970s? Computer use AI can interact with those. Windows applications that have no API at all? Computer use AI can click through those interfaces. Web-based vendor platforms that only expose 20% of their functionality through APIs? Computer use AI can access the other 80% through the actual user interface.

How It Actually Works

The technical process is more sophisticated than "AI clicks buttons," though that's the practical result.

Here's what happens: you give the AI instructions and point it at a piece of software. The AI takes a screenshot and analyzes what's on screen. It's a multimodal model, which means it can process images and understand what they represent in context.

Based on your instructions and what it sees, the AI decides what action to take next. Maybe it needs to find and click a search box. It looks at the screenshot the way your eyes would—visually scanning for elements that look like search boxes—and issues a command to move the cursor and click at specific coordinates.

Then it takes another screenshot to see what changed. Did a menu open? Did a new page load? Based on that, it decides the next action. This continues in a loop until the task is complete.

Amazon released a model called NOVA Act earlier this year that demonstrated similar capabilities. It was actually quite good for certain use cases, probably comparable to that earlier 42% benchmark. The speed and accuracy improvements in Claude 4.5 Sonnet make this technology viable for production use in ways it wasn't before.

AI Browsers: The Other Approach

While computer use represents AI adapting to existing tools, AI browsers take the opposite approach—rebuilding tools to be AI-native from the start.

Opera just launched Neon at $19.99/month, positioning itself as a power-user browser with built-in agentic capabilities. It can handle multi-step tasks, create reusable prompt cards, and consolidate search, writing, and workflow automation into a single interface. Perplexity has its own browser. Microsoft keeps enhancing Edge with Copilot features. Brave has AI capabilities built in. Chrome just added a Gemini extension with native integration.

These browsers have AI designed into their core architecture. They track context across tabs and sessions. They understand what you're working on holistically rather than treating each page as isolated. They can execute complex workflows without leaving the browser environment.

The promise is a unified workspace where you're not constantly switching between tools and manually connecting information. You're watching a cooking video and the browser automatically captures the recipe. You ask it to order the ingredients on Instacart and it handles the entire transaction. You're reading an article and can immediately ask questions about it while also pulling in context from other tabs you have open.

Why the Distinction Won't Matter Soon

Here's the thing: computer use and AI browsers aren't competing approaches that will duke it out until one wins. They're going to blend together.

Browsers and operating systems have already been blurring for years. Chromebooks made the browser essentially become the OS. Mac and Windows both added web widgets and browser-like features into the OS itself. The technical boundaries are permeable.

The same blurring will happen with AI capabilities. You might use a computer use agent to automate something in your legacy AMS. That agent might trigger a workflow in an AI-native browser. That browser might hand control back to a general-purpose AI assistant that works across all your applications. The user experience will be seamless even though multiple technical approaches are working together behind the scenes.

The shift we're seeing in consumer tools is toward AI just being assumed. You probably don't say "internet browser" anymore—you just say "browser" because of course it's connected to the internet. Soon you won't say "AI browser" either. It'll just be your browser, and of course it has AI built in.

Practical Applications 

Let's get practical about what becomes possible:

Legacy system automation. That AMS without modern APIs? You can now build automation around it. Computer use AI can log in, navigate through screens, extract data, update records—all the steps your staff currently does by hand. It won't be 100% reliable at first, but for well-defined repetitive tasks, it's becoming genuinely usable.

Conference management workflows. Remember entering session information into your event app one by one because bulk upload wasn't supported? Computer use AI can handle that. It can open the form, fill in the title, paste the description, add speaker names, select the track, save, and move to the next session. All the clicking you used to do.

Document routing processes. Those routines where someone exports data from one system, opens it in Excel, manipulates it, then imports it somewhere else? That's a perfect candidate for automation. The AI can execute each step the same way the human would, but consistently and without getting bored.

Testing and quality assurance. Before you launch a new member portal feature, you want to test it like your members will use it. Computer use AI can run through test scenarios, clicking through processes, filling out forms, and documenting results. This catches usability issues that might not show up in technical testing.

Southwest flight check-ins. Okay, this one's personal, but if you're someone who sets calendar reminders to check in for flights exactly 24 hours in advance to get a better boarding position, computer use AI can handle that for you. It's the kind of repetitive, time-sensitive task that's annoying for humans but perfect for automation.

The pattern across all these examples is manual steps that someone currently has to do because the software doesn't provide another way. Computer use doesn't require the software vendor to change anything. It works with what you already have.

Starting Carefully and Scaling Thoughtfully

A 61% benchmark score means computer use isn't perfect. For some processes it might be 95% accurate. For others it might be closer to 10%. You won't know until you test your specific use case.

Start with tasks that are:

  • Repetitive and clearly defined
  • Time-consuming but not mission-critical
  • Based on software that's unlikely to change dramatically
  • Easy to verify and review

What you probably shouldn't do yet is give computer use AI unsupervised access to financial transactions, member data modifications, or anything where an error would be difficult to reverse. The technology will get there, but 61% accuracy isn't good enough for those stakes.

The approach is similar to how you might work with a new intern. You give them clearly defined tasks. You check their work. As they prove reliable, you give them more autonomy. Computer use AI follows the same progression.

Start With One Annoying Thing

You probably have a routine right now that makes you wince every time someone has to do it. The task isn't difficult, but it's tedious. It takes 30 minutes of clicking through screens. It has to be done weekly or monthly. Nobody enjoys it.

That's your starting point.

Claude 4.5 Sonnet's computer use feature is currently available through the API, with a Chrome extension on a waitlist. The broad consumer release is coming soon. Now is the time to start thinking through your basic use cases so you're ready when access expands.

Focus on the repetitive, not the rare. A task you do once a year probably isn't worth automating, even if it takes two hours. But something you do weekly that takes 20 minutes? That's 17 hours annually—and that's just one person. Multiply that across your team and the time savings become real.

Describe your target process in detail. List out the steps. Define what success looks like. Identify what could go wrong. Take screenshots of how a human does it currently.

When you get access, try computer use AI on it. Set it up, supervise it, see what happens. Maybe it works great. Maybe it fails hilariously. Either way, you learn something about this technology's capabilities and limitations for your specific systems.

Document your findings. Record what worked and what didn't. Note how long setup took. Track how much supervision was needed. Decide whether you'd use this for real or if it was too unreliable.

That documentation becomes your institutional knowledge. When the next generation of models releases, you can test the same routine and measure improvement. You can try a slightly more complex task using what you learned. You build capability incrementally.

The technology isn't perfect, but it's accessible now. Associations experimenting today are learning what works while the tools are still evolving. Start small. See what happens.