Copilot Is Finally Catching Up With Other Chatbots

Copilot was initially conceived as an AI component to Bing, but it has since evolved into a comprehensive AI present in pretty much every Microsoft product right now. And it’s now getting a bunch of improvements that will bring it a little more in line with both ChatGPT and Google’s Gemini—and bring Bing in line with Google, too, on that subject.

Microsoft has announced a big expansion of Copilot’s capabilities. The bigger feature here is perhaps the introduction of a new feature called Actions. This feature enables the AI assistant to perform specific online tasks based on user instructions provided through chat prompts. According to Microsoft, these tasks can include booking restaurant reservations, purchasing event tickets, or ordering items for delivery. The system is designed to handle these operations in the background while users attend to other matters.

It’s kind of like ChatGPT’s Operator available on the chatbot’s most expensive $200 tier, except it only works with a limited list of curated websites. This is probably the better approach for features like this at this moment—the agentic operator announced by Amazon today for “buying stuff for you” also works with a curated list of stores. It’s easier to train an AI to know its way around a specific list of websites than to trust it to do everything out of the box.

Initial partners for Copilot Actions include travel and booking services like Booking.com, Expedia, Kayak, Tripadvisor, Skyscanner, Viater, Vrbo, and Priceline. Dining reservations via Open Table and floral orders through 1-800-Flowers.com are also supported at launch. Microsoft indicates that the feature is intended to eventually work with a wide range of websites and suggested it might handle transportation bookings, potentially integrating with ride-sharing services. But for now, it’s only for a specific list of curated partners. While Copilot is probably well-tested to work on these specific websites, you should still watch as it inputs your info there just in case it does make a mistake. It’s always possible—you’d be surprised by how frequently AI messes up, even with how advanced models have gotten.

Microsoft is also broadening the reach of “Copilot Vision,” its visual analysis feature. It was previously only available within the Edge browser, but Vision is now being integrated into the Windows operating system and the Copilot mobile apps (iOS and Android). On mobile, Copilot Vision can analyze real-time video from the device’s camera, allowing it to offer contextual help, such as plant care tips or home decorating advice. For Windows users, you can just share your screen with Vision and have the AI analyze what’s on screen. This enables scenarios like letting you get help on specific apps or helping you analyze viewed photos and webpages. The Windows implementation of Vision will land for Windows Insiders first, followed by a wider rollout.

Microsoft is also adding its own take on Google’s AI Mode by launching “Copilot Search” within Bing. This would basically replace the search results for an AI-only result, just like Google’s AI Mode for search. When a user performs a search, Copilot Search can generate summaries of key information, provide direct answers, or present information in structured layouts, depending on the query. Microsoft emphasizes that these AI-generated responses will prominently cite their sources, including inline links within the text and lists of URLs used, allowing users to verify information and navigate to the original content owners easily.

All of these features should roll out for everyone soon, except for Vision on Windows, which might take however much features tend to take to release for Windows users.

Source: Microsoft via The Verge