How to Achieve Broad AI? Insights from Sepp Hochreiter

Human intelligence, encompassing consciousness, cognition, decision-making, and more, seems to have captivated philosophers since the dawn of recorded history. Similarly, since the birth of AI, scientists have been pondering: How can AI achieve human-like intelligence?

Recently, Professor Sepp Hochreiter, one of the founders of LSTM and the founder of the Austrian Institute of Artificial Intelligence Research (IARAI), who received the IEEE CIS Neural Networks Pioneer Award in 2021, shared his views on the current level of AI intelligence in the ACM Communications.

Professor Hochreiter pointed out that the current development of AI is aimed at achieving Broad AI. He emphasized that the combination of traditional logic-based symbolic AI and the existing data-driven neural AI into a bilateral AI is the most promising way to realize Broad AI.

Professor Hochreiter believes that existing architectures, networks, or methods have reached the requirements of Broad AI to some extent. He cited the use of contrastive learning for self-supervised training to demonstrate transferability; the Modern Hopfield networks that fully leverage context and prior experience; and the neural-symbolic computing models that integrate knowledge and reasoning.

One effective model learning method to enhance network transferability is few-shot learning. It can achieve good performance with only a small amount of training data, thanks to its existing “prior knowledge” or “experience.” This prior knowledge often benefits from pre-training tasks that involve large-scale data and foundation models. Such data is typically extracted using contrastive learning or self-supervised training to obtain useful representations. Once the pre-trained model is well-trained, the posterior knowledge acquired in that task becomes the prior knowledge for new downstream tasks, enabling the model to effectively transfer to new contexts, customers, products, processes, and data.

Professor Hochreiter specifically mentioned the contrastive learning pre-training method in the vision-language cross-modal field—CLIP. CLIP is a work published by OpenAI last year at ICML, using a simple pre-training task, namely image-text matching, to learn more powerful image representations through contrastive learning. This task was trained using 40 million image-text pairs collected from the internet.

The surprising outcome is that it can achieve performance comparable to previous fully supervised methods on over 30 computer vision tasks without any additional data training, achieving remarkable “zero-shot learning”. As Professor Hochreiter praised, this high degree of transferability and robustness is a highly favored characteristic in industries that deal with real data.

In fact, since the invention of CLIP, many subsequent transfer learning works have been based on CLIP. It is not an exaggeration to say that it is an indispensable part of the unification of vision and language, somewhat akin to how Bert is to NLP tasks. Many contemporary methods of Prompt learning also draw inspiration from CLIP or operate within the CLIP framework.

Broad AI should also fully utilize the context and prior experiences, which is closely related to memory. Professor Hochreiter, as one of the inventors of Long Short-Term Memory networks, is well-positioned to speak on this matter. In cognitive science, some scholars have proposed the concept of short-term memory, which describes how humans quickly transform stimuli, such as an image or a word, into an abstract conceptual category and associate it with information related to long-term memory. The process of “stimulus-conceptualization-association” occurs almost unconsciously but plays a crucial role in our understanding of everyday things and language communication.

Combining neural networks with symbolic systems can better facilitate AI models’ integration of world knowledge and abstract reasoning capabilities.

Rationalism-based symbolic systems rely on logic and symbolic representation, directly encoding human reasoning methods into machines. Their advantage lies in strong abstraction capabilities and achieving good results with relatively little data. However, due to the complexity and diversity of real-world knowledge and its unstructured nature, it is challenging to perfectly encode these into machine-readable rules.

On the other hand, empiricism-based neural networks directly use large amounts of data to implicitly (unsupervised) or explicitly (supervised) guide models to learn useful representations from the data without the need to design complex rules, achieving astonishing performance. However, neural networks also face challenges such as weak interpretability and data hunger.

Organically combining the two has been a frequent topic of contemplation among AI scholars. Interestingly, this can also be related to the historical debate between rationalism and empiricism in human history, which is similarly a contentious topic.

Professor Hochreiter believes that the graph neural networks (GNN), which have been developed for some time, represent this direction. This is also based on an opinion from a survey published at IJCAI’20, which classifies GNN as a type 1 neural-symbolic system. The paper argues that both seek rich vectorized representations of neural network inputs and use tree and graph structures to represent data and their relationships. Professor Hochreiter believes they perform well in dynamic interaction and reasoning, emphasizing molecular properties, social network modeling, and engineering predictions, among others.

Hochreiter emphasized that achieving Broad AI requires the combination of neural-symbolic systems to reach a form of bilateral AI. AI researchers should also strive towards AI systems with stronger skill acquisition and problem-solving capabilities. He further envisioned that Europe has traditional advantages in both areas, thus should leverage these advantages and actively seek solutions for Broad AI.

References:

Hochreiter, Sepp. “Toward a broad AI.” Communications of the ACM 65.4 (2022): 56-57.

Chollet, F. On the Measure of Intelligence (2019); ArXiv:1911.01547.

Luís C. Lamb, Artur d’Avila Garcez, Marco Gori, Marcelo O.R. Prates, Pedro H.C. Avelar, and Moshe Y. Vardi. 2021. Graph neural networks meet neural-symbolic computing: a survey and perspective. In IJCAI’20. Article 679, 4877–4884.