Explaining The Development of Explainable AI and Deep Learning

【Introduction】Hello everyone, my name is Zhang Quanshi, a postdoctoral researcher at UCLA. Currently, I lead a team in Professor Zhu Songchun’s lab, focusing on explainable AI. The title of this article is quite grand; in this short essay, I will briefly share my personal feelings about the development of deep learning and my recent research topics on explanatory graphs for CNNs and interpretable CNNs. I hope everyone will provide feedback.

When deep learning first emerged in the computer vision (CV) community, I didn’t pay enough attention immediately. It wasn’t until months later that the waves of transformation hit, and the speed of the old directions fading away and the birth of new technologies exceeded my expectations. I believe many people have had similar feelings. On one hand, the strong performance of deep learning has ended a batch of old algorithms. On the other hand, compared to traditional graph-based methods, deep learning has greatly reduced algorithm diversity and simplified the complexity of algorithm design. Suddenly, the approach to CV research became very clear: design a new loss function, propose a new network structure, and hard-code traditional heuristic methods into the network structure for end-to-end learning. Just one or two technologies can transform the CV field to such an extent; deep learning has brought about tremendous changes to AI.

However, as end-to-end learning neural networks rapidly developed like fire meeting oil, I and many surrounding scholars occasionally felt a hint of concern: will training a black-box model end-to-end continue to develop smoothly? As the designs of network structures and loss functions become increasingly complex, will neural networks truly express the knowledge that people hope they will convey? With such anxieties, many scholars are dedicated to the visualization of CNN knowledge, making the knowledge of each unit in CNN clear to people. Furthermore, @Zhou Bolai has defined a series of standards to evaluate the interpretability of CNN knowledge.

But ultimately, I believe that beyond end-to-end learning, we need to find a new set of neural network operation tools that allow neural networks to possess clear symbolic internal knowledge representations that match human knowledge frameworks. This way, people can diagnose and modify neural networks on a semantic level. From logic-based expert systems to graphical models, and then to deep neural networks, the flexibility and performance of models have gradually improved. However, from the opposite direction, transforming the internal logic of a neural network into graphical representations or logic-based rules can enhance the interpretability of knowledge expression. With clear internal representations, can training neural networks not only be end-to-end but also end-to-middle and middle-to-middle? When certain units within the network possess a specific semantics, could transfer learning be directly assigned at the semantic level, eliminating the need for large datasets for training? If network training can delve into the internal semantics of the network, perhaps the future development of deep learning will hold more possibilities.

I hope a CNN not only tells me that it has detected a bird in a certain image, but also clearly indicates that it uses the first filter to monitor the bird’s head and the second filter to detect the bird’s tail. Because these two filters are triggered by this image, it concludes that there is a bird in the image. Furthermore, when I know the classification score for the bird is 0.7, I also hope the CNN indicates that the bird’s head contributed 0.3 to the score, while the tail contributed 0.2. When the internal logic of the CNN is sufficiently clear, do we still need to rely on large datasets for end-to-end training? Can we directly debug CNNs at the semantic level?

Following this line of thought, in the paper “Interpreting CNN Knowledge via an Explanatory Graph”, I mainly introduce how to transform the internal knowledge of a pre-trained CNN (for object classification) conv-layer into a graphical model. The algorithm automatically learns an explanatory graph with tens of thousands of nodes to explain the hierarchical knowledge structure within the CNN. Each node in the explanatory graph strictly represents a certain object part pattern in a specific conv-layer of the CNN. This way, I can break down the chaotic knowledge of the CNN into hundreds of thousands of object parts sub-patterns. Each sub-pattern has strong transferability; for example, it can reduce errors by 1/3 to 2/3 in multi-shot part localization.

Furthermore, based on the semantic expression of the explanatory graph, can we naturally integrate traditional graph-based technologies into CNN learning? I don’t know.

In another article “Interpretable Convolutional Neural Networks”, I introduce how to learn a CNN end-to-end such that each filter in its high-level conv-layer automatically represents a specific object part. The algorithm does not require manual labeling of object parts or textures as additional supervision; instead, it adds a prior constraint to these filters, allowing them to automatically regress toward certain object parts during training.

I will write two more short articles specifically detailing the technical aspects of these two papers.

In the face of the waves of the era, where will deep learning develop in the future? I don’t know. I can only cross the river by feeling the stones, moving forward while observing.