On September 19, local time, Musk’s brain-machine interface company Neuralink releasedrecruitment information for its first human clinical trial, which has attracted widespread attention and reporting.
After years of clinical trial applications, the U.S. Food and Drug Administration (FDA) finally approved Neuralink’s device clinical trial application (investigational device exemption, IDE) this May, and the upcoming clinical study named PRIME will be launched. This study aims to evaluate the safety of Neuralink’s fully implanted wireless brain-machine interface device and surgical robots, and assess the performance of the brain-machine interface to help paralyzed patients control external devices through their brains.
Image source: Every Day
So, what is a brain-machine interface? What are its technical challenges and development directions? What are its applications?
What is a Brain-Machine Interface?
As the name suggests, a brain-machine interface is a system used to connect the brain with external devices, it controls external devices by converting brain signals into control instructions. This concept originated from scientists’ understanding of how the brain works:
The Italian anatomist Galvani first discovered the phenomenon of bioelectricity;
The British physiologist Richard Caton recorded electrical signals on the surface of brain tissue in rabbits and monkeys;
The German doctor H. Berger invented the electroencephalogram;
The German-American neurologist E. Fetz achieved control of instruments using single-neuron signals, which can be considered the earliest prototype of a brain-machine interface;
The American computer scientist J. Vidal first proposed the concept of a brain-machine interface.
We can classify brain-machine interfaces from various perspectives.
Based on whether the signal acquisition process requires surgery, they can be divided into invasive (invasive) and non-invasive (noninvasive) brain-machine interfaces.
Devices like Blackrock’s Utah array electrodes and Neuralink’s Thread require craniotomy to implant electrodes into the cerebral cortex to collect neuronal action potentials, thus they are called invasive.
Utah Array Electrode & Thread Electrode｜Image source: Neuralink official website
On the other hand, non-invasive methods of signal acquisition such as EEG caps, near-infrared headbands, and functional MRI are referred to as non-invasive.
EEG Cap｜Image source: neuracle
The method of signal acquisition determines the type of signal.
Invasive methods typically collect spike potential (spike) and local field potential (LFP) signals; non-invasive methods generally collect scalp EEG (EEG) signals; and electrocorticography (ECoG) signals fall between the two.
Common brain-machine interface signal acquisition methods and types｜Image source: Reference 1
The difference in signal quality collected by invasive and non-invasive methods is like the difference between sitting in the front row at a concert and being outside the venue; the former has a significant advantage in spatial-temporal resolution.
Based on the function of the brain-machine interface, they can be classified into motor, sensory, and cognitive types. For example, assisting paralyzed patients in recovering upper and lower limb movements, sensory restoration for the blind and deaf, and brain-machine interfaces for detecting attention and emotions.
Based on the cause of signal generation, they can be divided into endogenous and exogenous types. Endogenous signals are spontaneously generated by the subjects, such as motor imagery; exogenous signals are brain signals induced by external stimuli, for example, when we see a fixed frequency flashing image, the brain’s visual cortex will produce oscillation signals at the corresponding frequency, which depend on the stimulation of the external flash, hence they are called exogenous signals.
Steady-state visual evoked potentials are a common type of exogenous brain-machine interface｜Image source: Tencent Cloud
Additionally, based on the direction of information flow between the brain and machines, they can be divided into unidirectional and bidirectional brain-machine interfaces. Unidirectional brain-machine interfaces only achieve one-way information transfer from brain decoding to external control, while bidirectional or reverse brain-machine interfaces also feedback signals felt by external devices to the brain through electrical stimulation, hence they are also referred to as writing brain-machine interfaces.
Most brain-machine interface systems can be summarized into three parts: signal acquisition, signal decoding, and external control, with the decoding part being the most critical. The basis of decoding is the selectivity of neurons.
Imagine a neuron in the brain that loves apples and hates bananas; when we see an apple, it will shout, “I want it! I want it!” When it sees a banana, it remains silent. We can easily classify apples and bananas based on the strength of its neuronal activity.
This is a simple description of the principle of brain-machine interfaces; in fact, as early as 1986, the Georgopoulos laboratory had already achieved decoding of monkey hand movements based on the directional preference of motor cortex neurons[2].
What Are the Challenges of Brain-Machine Interfaces?
Some may wonder, since the decoding principle of brain-machine interfaces is so simple, and many functions such as controlling computers, robotic arms, wheelchairs, and typing have already been achieved, why is Musk’s clinical trial receiving so much attention? Why haven’t brain-machine interfaces entered everyday life?
Currently, the achievements of brain-machine interfaces are basically results obtained in laboratory environments, under specific scenarios and tasks, and on individual patients. From various aspects such as safety, accuracy, usability, generality, and cost, there is still a long way to go before they can be considered commercial products.
For example, invasive electrodes can cause inflammatory reactions in the brain and cannot be used long-term; recording a large number of neural signals can significantly enhance brain-machine interface performance, but it requires chips with more powerful processing capabilities; the variability of neural signals is high, and they belong to small sample data, making decoder training very difficult.
Brain-machine interfaces are an interdisciplinary field that requires the integration of neuroscience, materials science, mathematics, machine learning, artificial intelligence, integrated circuits, and other areas to create a complete product, which is one of the reasons why Neuralink is highly regarded.
What Are the Applications of Brain-Machine Interfaces?
Let’s illustrate the current development directions of brain-machine interfaces through two examples.
Reconstructing What Is Seen
A research team from Osaka University led by Shinji Nishimoto published a work accepted at CVPR 2023 (IEEE International Conference on Computer Vision and Pattern Recognition), which achieved the reconstruction of brain activity into high-resolution images through Stable Diffusion[3].
The images seen by the subjects (first row) and the images generated by Stable Diffusion (second row)｜Image source: Reference 3
Scanning the MRI can reveal the images in the human brain, isn’t it fascinating? It seems that mind reading and dream probing are no longer distant. In fact, this is an excellent example of the combination of brain-machine interface decoding ideas with AI.
Stable Diffusion is a image generation model based on the latent diffusion model (Latent Diffusion Model), which can generate images or modify images based on pictures or text.
Latent Diffusion Model｜Image source: Reference 3
Stable Diffusion obtains the encoder ε and decoder D through training an autoencoder model. The encoder can reduce the dimensionality of the image X to the latent representation space z, and the text information is represented semantically through the encoder τ to obtain representation c, which is then combined with zr after the diffusion process to obtain zc, and finally the image X’ is generated through the decoder D.
Thus, it can be seen that the LDM model follows the process of “image/text → latent representation → image”, while the fMRI signal acquisition is the brain’s activity when the subject sees the image, which is the process of “image → brain → fMRI”.
Nishimoto et al. constructed the mapping between fMRI and latent representations for the same image using a simple linear model, decoding the image representation z and semantic representation c through the primary and higher areas of the brain’s visual pathway, as input, allowing the model to generate images similar to those seen by the subjects.
Decoding diagram｜Image source: Reference 3
In decoder training, subjects often find it difficult to complete a large number of tasks, so the training set samples are usually small; however, AI models can be trained with large data sets to achieve excellent generative capabilities. Combining the two can help address the small sample training and generalization issues faced by brain-machine interfaces to some extent.
Touching the World with a Robotic Hand
A significant application scenario for invasive brain-machine interfaces is helping paralyzed patients control robotic arms to regain mobility. However, controlling the robotic arm is far from enough; patients also need to feel the robotic arm.
Motor control includes both feedforward and feedback components; quickly achieving target movements without corrections is feedforward, while continuously correcting movements based on sensation is feedback. For example, when our eyes are closed and we lose visual feedback, we can still recall and pick up a cup on the table, but without tactile feedback, we find it difficult to hold the cup steadily.
A study published in the journal Science in 2021 achieved writing tactile information from the robotic hand into the brain, allowing patients to control the robotic hand more dexterously[4].
Collinger et al. decoded motor intentions to control the robotic hand using an array of electrodes implanted in the patient’s motor cortex, while the tactile information sensed by the robotic hand was encoded into different patterns of microcurrent stimulation, fed back to the patient’s brain through electrodes implanted in the sensory cortex. This addition of tactile feedback improved the patients’ performance in upper limb motor ability assessments (ARAT).
Bidirectional brain-machine interface diagram｜Image source: Reference 4
Brain-machine interfaces are not a fantasy nor a brand new concept that appeared out of nowhere; they are a technology that has undergone decades of laboratory research and achieved numerous functions. We hope that in the near future, they can help more patients live better lives.
[1] Thakor, N. V. Translating the brain-machine interface. Sci Transl Med 5, 210ps217, doi:10.1126/scitranslmed.3007303 (2013).
[2] Georgopoulos, A. P., Schwartz, A. B. & Kettner, R. E. Neuronal population coding of movement direction. Science 233, 1416-1419 (1986).
[3] Takagi, Y. & Nishimoto, S. High-resolution image reconstruction with latent diffusion models from human brain activity. 2022.2011.2018.517004, doi:10.1101/2022.11.18.517004 %J bioRxiv (2022).
[4] Flesher, S. N. et al. A brain-computer interface that evokes tactile sensations improves robotic arm control. 372, 831-836, doi:doi:10.1126/science.abd0380 (2021).
Author：Ye Jian, PhD, Institute of Neuroscience, Chinese Academy of Sciences
— END —
Reproduced content only represents the author’s views
It does not represent the position of the Institute of Physics, Chinese Academy of Sciences
If you need to reprint, please contact the original public account
Source: Shanghai Science and Technology Museum
Editor: wnkwef
Recent Popular Articles Top10
↓ Click the title to view ↓
1.She is not in “Oppenheimer,” but the physics community cannot do without her
2.People who fall asleep as soon as they get in the car may have this problem!
3.15 equations to test your physics level
4.“Pre-packaged meals enter campus” trending! How harmful is it to eat pre-packaged meals regularly?
5.Why do solitary cats hold regular meetings, what are they plotting?
6.What is the essence of time?
7.Why did a nose hair count win a 10 trillion prize, this year’s humorous Nobel Prize is too heavy…
8.Why do most plastic stools have a hole in the middle? | No.377
9.With a big bug on the cover, you probably can’t find it
10.When we photograph a black hole, what are we actually photographing?
Click here to view all past popular articles
