Will AI be the new UI?

Long predicted, the end of the smartphone as we know it now seems to be gradually approaching. The reason: instead of laboriously moving from app to app, artificial intelligence (AI) will enable AI assistants to carry out cross-app tasks in the future, including by voice command.

A look back at the development of user interfaces shows just how significant this step is: While the command line still dominated in the early 1970s, it was gradually replaced by the graphical user interface (GUI) from the 1980s onwards. Touch operation followed in the 2000s and voice input was gradually added. Nevertheless, apps are still silos in which users operate in relative isolation — a situation that is now slowly disappearing with the triumph of generative AI (genAI) and AI agents.

Off to a shaky start

Pioneers in the field of AI devices include the Humane AI Pin and the Rabbit r1. These devices broke away from the classic form factor of a smartphone and were primarily operated by voice. But contrary to what the marketing promised, the technology was not ready for practical use and the commands were not well implemented, disappointing both reviewers and users. Although the Rabbit r1 is still available, the more expensive Humane AI Pin was discontinued in early 2025, less than a year after its launch.

Deutsche Telekom also presented a concept AI phone at MWC 2024, in which the countless apps on a smartphone were to be replaced by an AI-based assistant.

The idea: like a concierge, the AI understands the user’s wishes and then takes care of the details. Instead of searching for a flight to the World Cup quarter-finals in various portals or airline apps, for example, the user should be able to order this simply by voice command via their smartphone. This is at least one scenario that appears to have been borrowed from the connected car sector.

Deutsche Telekom intends to stick with this idea. The company presented an updated version of its AI phone in Barcelona this year and announced that it would be available “at an affordable price” in the second half of the year.

While the unassuming exterior of the Android smartphone has remained the same, Deutsche Telekom’s technology partner has changed. Instead of Natural AI from Brain.ai, a genAI solution from Perplexity (also cloud-based) is now used. The Telekom phone will offer additional AI features in the form of applications from Google Cloud AI (object recognition), Elevenlabs (a podcast generator) and Picsart (a genAI design tool).

Users can get an impression of how it works with the Perplexity Assistant (not to be confused with the AI-supported search engine), which is freely available as an app. On Android devices, the AI tool is able to open various apps, send messages, reserve tables in restaurants via OpenTable, and translate texts on the screen, among other things.

On the iPhone, the app’s functionality is somewhat limited but can easily outshine Siri when accessing third-party apps. The iOS version of Perplexity also offers a function that the company calls “Persistent Listening.” If this is activated, the voice assistant continues to listen to the user’s voice, even if they leave or switch apps. This is a workaround, as Apple naturally does not allow a “wake-up word” such as “Hey Siri.”

OpenAI and Google are getting involved

In view of the current trend towards AI agents, or agentic AI, similar approaches can also be found among other providers. In January, for example, OpenAI presented Operator, an AI agent that can independently perform a range of repetitive tasks on the web. According to the provider, the agent, which is currently only available in the US (as a preview for Pro users of ChatGPT), can be tasked with a variety of repetitive browser-based tasks, such as filling out forms or ordering food. Operator is powered by OpenAI’s Computer-Using Agent (CUA), an AI model that combines the visual capabilities of GPT-4o with reasoning.

According to OpenAI, CUA is trained to interact with graphical user interfaces (GUIs) in the same way that humans do. This is done with the help of numerous “screenshots” of the website, which are evaluated by the AI in order to initiate the next step. According to the company, this makes the agent flexible enough to carry out digital tasks without using operating system or web-specific APIs.

It’s worth noting, too, that OpenAI is showing strong interest in developing AI devices with its $6.5 billion purchase of io, an AI hardware startup founded by former Apple design wunderkind Jony Ive.

Google is working on the topic of AI assistants in several projects. At its I/O 2025 developer conference, the tech giant demonstrated the latest advances in Project Astra, among other things. The most revolutionary innovation: Astra can now act proactively. Instead of only reacting to direct commands, the assistant continuously observes its surroundings and decides independently when it should intervene.

Project Mariner, which was first presented in December 2024, now comprises a system of agents that can perform up to 10 different tasks simultaneously. Among other things, they are able to look up information, make bookings, and make purchases — all at the same time.

In a demo at I/O, Google showed how a simple command such as “Plan a weekend trip to Berlin” led to a cascade of actions: the agent researched flights, hotels, and activities and presented a complete itinerary, all without further user interaction. As with the other tools, the principle of “Human in the Loop” (HITL) applies to such steps: the agents explain their actions, ask questions when important decisions are made, and can be interrupted by the user at any time.

On-device AI or cloud AI — or both?

While cloud AI promises further AI expansion and — with good connectivity — better performance, hardware manufacturers such as Apple, Samsung, and Qualcomm are focusing on “on-device AI” to differentiate themselves and create incentives to buy new, high-quality devices.

At the launch of its new flagship smartphone, the Galaxy S25, Samsung proudly announced that the corresponding features had almost doubled compared to its predecessor. As an example of this, the Korean company cited the ability to activate settings such as dark mode via voice command. And Google’s Gemini assistant can be activated via the function key to use Samsung and Google apps without having to select them individually. Some third-party apps such as Spotify and WhatsApp can also be controlled in this way.

Qualcomm, which provides the necessary AI computing power for high-end smartphones such as the S25 with its Snapdragon 8 Elite chipset (or the Hexagon NPU built into it), is also driving the trend towards on-device AI. The company provides a large number of small language models (SLMs) on its AI Hub. Although the language models are trained in the cloud, inferencing takes place on the device.

And Apple? The iPhone manufacturer is lagging far behind in the field of AI agents, although the original Siri app was already quite advanced in 2014 before Apple bought it and was able to make dinner reservations, find a movie theater, or call a cab. After the death of Steve Jobs, the development team had a hard time convincing software boss Craig Federighi to take AI seriously, Bloomberg reports, citing informed sources and quoting an insider: “When ChatGPT was publicly unveiled in November 2022, Apple Intelligence didn’t even exist as an idea, although it was no secret what OpenAI was doing.”

Apple is now working on “LLM Siri,” but the launch planned for 2025 is still delayed for technical reasons. According to Bloomberg, the biggest problem is that Apple had to split Siri’s infrastructure in half in order to bring the new functions to market as quickly as possible. The old code was used for old functions (such as setting an alarm) and the new code for requests that use personal data. However, this trick caused integration problems that led to delays.

How much personalization is needed?

In order to create context for some tasks, it is of course an advantage if the agents can incorporate personal knowledge about the user, such as gender and age, but also things like location and place of residence or culinary preferences. A classic marketing example is a scenario in which a user wants to eat something on the way home. He asks the AI assistant to find him a restaurant and reserve a table.

When it comes to finding a restaurant that suits the user’s eating habits and is on the route, classic data collectors such as Google have a clear advantage. However, a device such as the smartphone, as a constant companion, is also able to collect a lot of personal data about the user. At the same time, the inhibition threshold for sharing information should not be as high.

Announcements by companies such as OPPO prove that such a personal knowledge graph is not fiction. The smartphone manufacturer recently announced that it is developing a new knowledge system that will serve as a central repository for user data. “The system is designed to learn and adapt from users’ activities, interests, data and memories. This should enable highly personalized AI experiences and thus progress towards a future vision of intelligent, AI-driven operating systems,” the announcement states.

Samsung, for its part, laid the foundation for such a knowledge graph back in 2024 with the acquisition of Oxford Semantic and has now integrated it into its Galaxy S25 family. “RDFox is the technology behind Samsung’s Personal Data Engine, which creates hyper-personalized user experiences while ensuring the privacy and security of data on the device,” Oxford Semantic announced at the launch of the new Samsung flagship. By using knowledge graph technology to integrate and link data, RDFox improves the understanding of how people use a product or service and enables relevant information to be retrieved quickly and recommendations to be made.

Additional considerations

Once such a personal knowledge graph is up and running, the problems are only just beginning: To prevent misuse, it must be stored in a secure enclave in a highly encrypted form. In the case of the Galaxy S25, for example, they are stored in Samsung’s Knox Vault and protected by the blockchain-based Knox Matrix security system.

At the same time, however, numerous other questions arise, some of which still need to be answered, such as:

• To what extent is this information passed on when devices are changed? What scenarios are there when switching to the successor model, a model from the same manufacturer, a model with the same operating system or even a different operating system?
• Is there a backup in the cloud for migration or loss scenarios?
• Do multiple devices — such as a tablet and smartphone — share knowledge?
• Is there a cross-device knowledge graph for several agents or several separate ones?
• Are there different databases for different personas (e.g., private or professional)?

But more technical aspects also need to be addressed if the AI is constantly listening — such as energy management. One option is to use several AIs: a power-saving AI module waits in the background for commands and then passes them on to other, specialized AIs.

Possible app applications

If the AI agents primarily access functions and services via APIs, this naturally also has consequences for the user interface of apps. “Apps will not become meaningless, but different,” comments the Fraunhofer Institute for Experimental Software Engineering (IESE) on the development in a blog post.

A future-proof software concept must therefore be designed in such a way that it also works when users do not even see the software interface. “The ability to function seamlessly in different contexts and across different interfaces is becoming the decisive factor for success,” Fraunhofer IESE states.

“The challenge and the art of native app development will be to see these developments not as competition, but as an opportunity,” adds Mark Zimmermann, Head of CoE for Mobile Application Development at EnBW. “Successful apps of the future will not only rely on classic UI elements, but will also be intelligently interlinked with AI-controlled systems.”

EnBW

According to Zimmermann, the key to success lies in integration: “Customers need to be met where they expect to be — be it through voice interaction, multimodal interfaces, or directly in existing ecosystems. So it’s not about the end of the UI, but about its transformation. Those who get to grips with these technologies early on and integrate them will remain competitive in the long term.”

This article was originally published on Computerwoche.