Microsoft expands Copilot across the Windows ecosystem, with new Vision and Copilot PC features

Microsoft has announced sweeping updates for its Copilot offering, aimed at making Copilot easier to use on Windows PCs and integrated within apps like the Edge browser.

Copilot Vision is a new feature built into the Microsoft Edge browser as a native feature, allowing users to ask questions about the web pages they’re navigating. Users can use voice inputs to ask Copilot Vision questions in real-time, with low-latency responses derived from the content on their screen.

Microsoft clarified that users need to actively give Copilot permission to view their active webpages and that there is a clear indication when Copilot is active. It added that no data is stored beyond the active session or used for training, meaning a user’s history is not stored within the app.

This is a new era of technology that doesn’t just “solve problems”, it’s there to support you, teach you, help you,” wrote Mustafa Suleyman, CEO at Microsoft AI. “In this sense, Copilots really are different to that last wave of the web and mobile.

“This is the beginning of a fundamental shift in what’s possible for all of us. It’s a long journey that will take years. With our latest updates to Copilot, you are seeing only the first careful steps in this direction.”

Another new feature for the Copilot mobile app is Copilot Voice, which aims to streamline interactions with the chatbot. In a demonstration, a user was shown speaking in a natural way to the app, with Copilot responding in kind – even with realistic pauses and “ums” in between words

Finally, Microsoft has announced a new mode for the Copilot browser chat called ‘Think Deeper’. Available via a ‘think deep’ button in the Copilot chat interface, it allows Copilot to process more complex information, with the demo giving the example of planning a family reunion taking into account where people are arriving from, their ages, tastes, interests, and more to feed into the overall plan for the day.

It’s unclear yet if this has dedicated enterprise potential, or if it represents a shift toward AI ‘reasoning’ as with OpenAI’s o1-preview. Microsoft does not comment on the exact model used for each Copilot feature, though select offerings such GitHub Copilot have already been testing using o1 as the backing model for more detailed outputs.

Copilot Voice is in English available now in the US, UK, Australia, Canada, and New Zealand. Microsoft has said that while it is limited to English and these regions at present, more will follow in the near future.

Copilot Vision and Think Deeper, meanwhile, will be limited to select Copilot Pro subscribers via a new channel for early access features called Copilot Labs. Microsoft has not laid out a release roadmap for either feature beyond this.

OS-wide AI via Copilot PCs

Alongside the new software features in Edge and on the app, Microsoft unveiled a range of new AI capabilities for Windows operating systems running on the range of Copilot+ PCs. This includes machines from manufacturers such as Acer, HP, Dell, Lenovo, and Microsoft’s own Surface laptops.

A new feature dubbed ‘Click to Do’ will be made available for any application, document, or media. Users can press the Windows key at the same time as clicking to activate the feature, allowing them to select text on the screen, share links to media, or reverse image search elements on their screen.

Microsoft has also updated Windows Search with embedded AI, with the intention of making it easier for users to find files on their PC. Using natural language processing, the search bar will be able to recommend settings based on user search terms such as ‘add my headphones’ or find files and images using descriptive language.

The number of developers creating third-party apps specifically for Copilot is also expanding.

“Because over 90% of the time, Copilot+ PC customers are using ARM-native applications, the best and most used applications are already on Windows including Adobe, Chrome, Google Drive, Zoom, Brace, Opera, Slack, Spotify, OpenVPN, and so many more,” siad Yusuf Mehdi, EVP & Consumer CMO at Microsoft.

The Windows PC news comes alongside a new update to Recall, intended to convince those who had feared the service carried major security risks.