Neuromorphic computing is a brain-like computing paradigm, generally referring to running Spiking Neural Networks (SNN) on neuromorphic chips.
Essentially, neuromorphic computing is a design paradigm driven by algorithms. With its low-power advantages, neuromorphic computing is also considered a “potential substitute” for traditional AI.
Understanding neuromorphic computing should be approached from a system level, rather than isolating algorithms or hardware.
The term “brain-like” in neuromorphic computing refers to the ability of spiking neurons to simulate the structure and function of biological neurons.
Through this simulation: on one hand, neuromorphic computing possesses biomimetic complex spatiotemporal dynamics; on the other hand, it can utilize spike signals to convey information.
The former allows the expressive capability of the spiking neuron model to theoretically exceed that of artificial neuron models based on traditional Artificial Neural Networks (ANN); the latter enables spiking neurons to have spike-driven computational characteristics.
When spiking neural networks operate on neuromorphic chips, sparse computation is triggered only when input spike signals are present. Otherwise, neurons remain in a resting state. Therefore, to achieve low-power neuromorphic systems, spike-driven mechanisms are essential.
Currently, the field of neuromorphic computing faces a severe reality: compared to traditional ANN algorithms, spiking neural network algorithms fall far short in task performance, making it difficult to meet various complex scenario demands.
For edge computing scenarios, low power consumption and low latency are often required. Once the performance issues of spiking neural networks at the algorithmic level are resolved, combined with the advantages of neuromorphic chips, the strengths of neuromorphic computing can be greatly highlighted.
Researcher Li Guoqi and his team from the Institute of Automation, Chinese Academy of Sciences believe that the performance potential of neuromorphic computing has yet to be fully realized.
For example, in terms of neural network architecture, most current applications of neuromorphic computing revolve around spiking convolutional neural networks (CNN), and current neuromorphic chips can only support spiking CNNs.
In contrast, traditional deep learning has already achieved significant breakthroughs in various tasks using the Transformer architecture. It was only after this team proposed a series of Spike-driven Transformer models that the neuromorphic computing field integrated the spike-driven paradigm into the Transformer architecture.
Figure | Li Guoqi (Source: Li Guoqi)
How should spiking neural networks and Transformers be combined?
For Li Guoqi, the work surrounding spiking neural networks can be traced back to a publication from 2018 when he was working with Professor Shi Leping’s team at Tsinghua University’s Brain-like Computing Center.
He stated: “Professor Shi’s team proposed a backpropagation algorithm to replace gradient-based spatiotemporal learning, addressing fundamental training issues in the field of spiking neural networks.”
However, due to issues such as a lack of basic programming frameworks, non-differentiable binary spikes, and the degradation of deep networks, spiking neural networks had at most only a dozen layers until 2021.
This small scale resulted in spiking neural networks performing far worse than traditional deep learning. Later, spiking neural networks began to develop towards deeper architectures.
For instance, in 2021, Li Guoqi’s team published a paper at the Association for the Advancement of Artificial Intelligence (AAAI) conference that addressed the deep training issues of spiking neural networks.
After joining the Institute of Automation, Chinese Academy of Sciences, Li Guoqi and Professor Tian Yonghong from Peking University co-authored a paper on the open-source training framework for spiking neural networks called SpikingJelly in Science Advances.
This paper solved the issue of the lack of training frameworks in the field, significantly lowering the learning threshold for spiking neural networks.
Meanwhile, Li Guoqi’s team and Tian Yonghong’s team proposed two different residual depths for spiking networks, which have now become standard residual architectures in the field.
These architectures allow spiking neural networks to achieve hundreds of layers in depth while avoiding spike degradation issues, overcoming the technical bottleneck in training large-scale spiking neural networks in terms of depth and scale.
Although the performance gap between spiking neural networks and artificial neural networks has been significantly narrowed, it is still not enough. The Transformer architecture is a milestone in deep learning and has attracted the interest of scholars in the field of spiking neural networks.
Starting around 2022, related work on spiking Transformers began to emerge. These studies typically replace some artificial neurons in the Transformer architecture with spiking neurons.
Key operations such as self-attention mechanisms are retained, ensuring task performance.
These early works inspired Li Guoqi’s team. However, they felt this was more of a heterogeneous combination of artificial and spiking neural networks.
Thus, the research team posed the question: “How should spiking neural networks and Transformers be combined to leverage the advantages of both?”
After repeated contemplation and discussion, the team ultimately chose the “spike-driven self-attention operator” as the breakthrough point for this issue.
The reason is: currently in the field of spiking neural networks, there are only convolutional and fully connected spike-driven operators.
Since the self-attention mechanism is key to the success of Transformers, could it be modified to be spike-driven?
Once this idea was confirmed, they conducted repeated experiments and ultimately designed several spike-driven self-attention operators that functioned correctly.
The results showed that the spike-driven self-attention operator possesses many excellent characteristics, such as being a natural linear operator and enabling sparse computations.
After the spike-driven Transformer could operate normally, they attempted to further improve performance by modifying the architecture.
However, there are too many variants of the Transformer architecture, which can be overwhelming.
Thus, they began to consider: could they design a meta-architecture for spiking neural networks? This would significantly reduce the architectural gap between spiking neural networks and artificial neural networks.
Subsequently, the team divided this series of work into two main steps:
First step: propose the spike-driven self-attention operator. This is the third type of operator in the field of spiking neural networks, allowing the entire spike-driven Transformer to consist only of sparse additions.
Second step: explore the meta-architecture of spiking neural networks. This can narrow the design gap between spiking neural networks and traditional artificial neural networks.
After completing the above steps,they successfully introduced new operators and architectures to the field of spiking neural networks, allowing neuromorphic computing to achieve a new level of task performance while maintaining its low-power advantage.
The research team believes that if they continue to progress in this direction over the next two years, the performance of spiking neural networks will be able to match that of artificial neural networks, and the former’s energy efficiency advantages will become even more pronounced.
In currently mainstream visual tasks, natural language processing tasks, and generative tasks, if neuromorphic computing can resolve performance bottlenecks at the algorithm level, it will certainly inspire the design of neuromorphic chips based on new spike-driven operators and architectures. Additionally, it is of great significance for the realization of low-power artificial intelligence.
Recently, a paper related to this research was accepted for the 2024 International Conference on Learning Representations (ICLR 2024) titled “Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips.”
Assistant researcher Yao Man from the Institute of Automation, Chinese Academy of Sciences is the first author, and researcher Li Guoqi is the corresponding author.
Figure | Related Paper (Source: ICLR2024; Paper link: https://openreview.net/forum?id=1SIBN5Xyw7)
On one hand, this achievement can be used in edge neuromorphic computing scenarios, such as a combination of “neuromorphic vision + neuromorphic computing.”
Here, neuromorphic vision refers to perceiving brightness changes in visual scenes through Dynamic Vision Sensors (DVS), thereby outputting an asynchronous sparse event stream, mimicking biological vision.
For neuromorphic computing, it naturally possesses event-driven computing characteristics, making it well-suited for processing such sparse event streams.
Recently, the team also collaborated with a brain-like startup to deploy spiking neural networks onto an asynchronous sensing computation integrated chip.
The resting power consumption of the chip processor is only 0.42mW, and its power consumption in typical neuromorphic vision task scenarios is also below 10mW.
This gives the chip an “always-on” characteristic, providing significant advantages in some low-power edge computing scenarios.
If the Spike-driven Transformer architecture can be integrated into asynchronous brain-like chips, it will not only maintain its low-power characteristics but also, with improved model expressive capabilities, be applicable in more scenarios.
On the other hand, this achievement provides technical support for the design of ultra-large-scale networks based on neuromorphic computing.
Currently, most large models based on artificial neural networks are designed based on the Transformer architecture. This work integrates the spike-driven paradigm into the Transformer architecture, resulting in a purely additive Transformer.
Moreover, the designed operators are linear with respect to the number of input tokens and feature dimensions. Therefore, as the model size increases, the energy consumption advantages become more apparent.
As we all know, artificial intelligence has entered the era of large models, which are expected to become the foundational service infrastructure for human society in the future.
However, with the increase in user numbers and usage frequency, the high energy consumption of AI will become an issue that cannot be ignored.
In this context, exploring a new generation of linear spiking neural network architectures that incorporate brain-like spatiotemporal dynamics becomes particularly important. This also means that this achievement can provide technical support for low-power brain-like spiking large models.
(Source: ICLR2024)
The neuromorphic computing field is expected to witness significant development.
Reflecting on the journey, Li Guoqi feels it has not been easy. He said: “Whether for outsiders or insiders, the field of spiking neural networks has always faced skepticism. Even some members of our team are not very confident, as they often see doubts from netizens about this direction.”
He also expressed understanding. Although spiking neural networks have advantages such as brain-like structures and low power consumption, these advantages can only be reflected at the system level.
As mentioned earlier, compared to mature artificial neural networks, spiking neural networks still have certain gaps in various aspects, making the future direction of spiking neural networks unclear.
Fortunately, the field of spiking neural networks has made significant progress in recent years, and his confidence in this field is growing.
He stated: “I am personally optimistic about the development of the neuromorphic computing field and expect significant advancements in the coming years.”
Especially with the arrival of the large model era, AI cannot ignore the huge energy consumption issue if it wants to become the foundational infrastructure for human society.
Therefore, he and his team hope that this achievement can advance the practical application of spiking neural networks and inspire the design of the next generation of neuromorphic chips.
Overall, there are still many challenges to be overcome in the field of neuromorphic computing, which requires collective efforts from the entire field.
Based on this achievement, they will continue to work on the following aspects:
First, to realize larger-scale spiking neural network models. Due to the complex spatiotemporal dynamics of spiking neural networks, they are more challenging to train than artificial neural networks, necessitating the design of new training methods for efficient training.
Second, to apply spiking neural networks to more task types. This work mainly focuses on computer vision tasks, and in the future, they also want to try using the designed structures for more tasks, such as long-sequence tasks.
Third, to propose brain-like spiking large model architectures based on spiking neural networks. It can be anticipated that this will be a daunting task requiring systematic breakthroughs in training speed, architectural design, model scale, task performance, and long-distance dependency modeling.
Fourth, to design hardware computing architectures that adapt to brain-like spiking large models. Currently, the team has begun some explorations regarding hardware implementations based on this work.
If efficient spike-driven self-attention operators can be realized in hardware, combined with the sparse computational characteristics of large-scale spiking neural networks, it will certainly lead to more functionalities.
05/ Chinese scientists solve the laser emission problem of blue phase liquid crystals below 0°C, achieving an ultra-wide temperature range from -180°C to 240°C, opening the door for the development of low-temperature blue phase liquid laser devices.
