The Future of Intelligent Cockpits as Smart Entities

With the rapid development of AI technology, the interaction methods of intelligent cockpits have undergone multiple evolutions. From the initial single touch interaction, to simple command-based interaction, and then advancing to complex voice interactions, the frequency of user interactions has also increased rapidly, from less than 10 times a day to over 60 times.

“But is a higher daily interaction count necessarily better?” On August 9, at the high-end seminar “Development Path of Intelligent Vehicle Industry Ecosystem” hosted by the China Electric Vehicle 100 People Association, Li Tao, General Manager of Baidu Apollo’s Intelligent Cockpit Division, pointed out: “This may be because the vehicle system does not truly meet user needs. For example, if a user talks to the vehicle 100 times in a day, it may indicate that the vehicle system is user-friendly, or it could mean that the vehicle system is too clumsy to understand or predict user needs, forcing users to repeat their dialogues multiple times.”

Furthermore, Li Tao stated that the trend of vehicle systems resembling tablets is very serious. “Putting an iPad or Android tablet at the center of the vehicle system and installing all applications, are these programs really what users need?” He believes that according to the Pareto distribution law (the 80/20 principle), 80% of the functions are rarely used. An excess of programs not only increases the user’s understanding and cognitive burden, occupying valuable vehicle system resources, but also means that manufacturers need to incur substantial costs.

So, what direction should intelligent cockpits develop towards? What changes can powerful models with strong semantic understanding, text generation, logical reasoning, and multimodal capabilities bring to intelligent cockpits? What practices has Baidu Apollo implemented? Li Tao provided answers during his speech.

Li Tao, General Manager of Baidu Apollo’s Intelligent Cockpit Division

The following is the transcript of his speech:

Currently, we are in an era of deep integration between AI and automobiles. Let’s take a look at the current development status of intelligent vehicles, illustrated by a few sets of data: In the field of new energy vehicles, the penetration rate of L2 intelligent driving is about 40%, while the installation rate of intelligent cockpits has exceeded 60%, and it is expected to reach over 70% this year. On the other hand, we also see a trend where consumer satisfaction with interactions has declined. This may be because the vehicle system does not truly meet user needs.

In recent years, the control methods for cockpits have undergone some evolution. Initially, control was achieved through precise commands, i.e., saying A means A, saying B means B, with no generalization. Later, preliminary generalized command-based interactions emerged; previously, the daily interaction count was below ten, but now some vehicle models have reached over sixty daily voice interactions. The industry has begun to discuss whether the daily interaction count will continue to increase as we evolve towards natural language dialogue.

But is a higher daily interaction count necessarily better? For instance, if a user talks to the vehicle 100 times in a day, it may indicate that the vehicle system is user-friendly, but it could also mean that the vehicle system is too clumsy to understand or predict user needs. Therefore, when users need something, they can only repeat their dialogues with the vehicle.

Additionally, the trend of tablet-like systems is very serious. The current implementation method is to install an iPad or Android tablet in the center of the vehicle system and install all applications. Are these programs really what users need? An excess of programs not only increases the user’s understanding and cognitive burden, occupying valuable vehicle system resources, but also means that manufacturers need to incur substantial costs. This follows the Pareto distribution, where 80% of the functions are rarely used.

In this context, we need to consider which direction intelligent cockpits will ultimately take. From the perspective of large models, we believe that intelligent cockpits will evolve towards smart entities that can understand contextual information, naturally comprehend user needs, generate contextual solutions, and execute them. Therefore, we launched the Apollo Super Cockpit series products. It is a type of smart entity that can achieve full sensory integration, global planning, and comprehensive execution.

Users hope that vehicles can understand their needs, record their habits, and provide the most suitable in-car environment or application configuration based on the current context. This is precisely what large models excel at—understanding and memory, logic and generation. We have other specialized model analysis architectures. Within this framework, the intelligent cockpit can automatically understand, construct, and generate corresponding models. For vehicle manufacturers, this can significantly reduce the adaptation costs for various scenarios, ultimately achieving comprehensive execution. We also have a foundational large model that can truly schedule the capabilities of the entire vehicle, genuinely understand user needs, and proactively execute to provide users with a better experience.

We have achieved breakthroughs in certain scenarios, such as integrating in-car DMS, OMS, and the entire cockpit’s voice capabilities. We have also collected external data, such as speed bump predictions, internal music playback on highways, and multi-person detection. For instance, in high-speed and open-window scenarios, due to excessive noise, human voices can be drowned out, making it very difficult to detect actual speech. Through audio and video speech enhancement technology, the success rate of speech detection in high-speed and open-window scenarios can be improved to 99%, which is even better than the case of ordinary vehicles with closed windows.

The above are some of our thoughts on generative cockpits. In the future, we will continue to push forward in this direction. Thank you all.