In the past decade, deep learning-based algorithms have been widely applied in remote sensing image analysis. The transformer, initially introduced to the NLP field, has also permeated the computer vision domain. The remote sensing community has witnessed an increase in the use of visual transformers for various tasks. However, many surveys have focused on transformers in computer vision, with very few addressing remote sensing. This paper systematically reviews the latest advancements in transformer-based methods used in remote sensing, covering over 60 methods that can address different remote sensing issues in subfields such as Very High Resolution (VHR), Hyperspectral Imaging (HSI), and Synthetic Aperture Radar (SAR) images.
Remote sensing imaging technology has made significant progress over the past few decades. Modern airborne sensors have continuously improved in spatial, spectral, and resolution capabilities, enabling coverage of most of the Earth’s surface. Therefore, remote sensing technology plays a crucial role in numerous research fields, including ecology, environmental science, soil science, water pollution, glaciology, land surveying, and analysis. The unique challenges posed by the multimodal nature of remote sensing data, its geographical positioning, and the global scale of data continue to grow.
In many areas of computer vision, such as object recognition, detection, and segmentation, deep learning, particularly Convolutional Neural Networks (CNNs), has become mainstream. CNNs typically take RGB images as input and perform a series of convolutions, local normalization, and pooling operations. CNNs often rely on large amounts of training data and use the resulting pretrained models as general feature extractors for various downstream applications. The success of deep learning-based computer vision techniques has also inspired the remote sensing field, achieving significant progress in many remote sensing tasks, such as hyperspectral image classification, change detection, and ultra-high-resolution satellite instance segmentation.
One of the main foundations of CNNs is the convolution operation, which captures local interactions between elements in the input image (such as contours and edge information). CNNs encode biases such as spatial connectivity and translational equivalence, which help build general and efficient architectures. The local receptive fields in CNNs limit modeling of long-range dependencies in the image (such as relationships between distant parts). Convolution is content-agnostic, as the weights of the convolution filters are fixed and apply the same weights to all inputs regardless of their nature. Visual Transformers (ViTs) have demonstrated impressive performance across various tasks in computer vision. ViTs effectively capture global interactions by learning relationships between sequence elements through a self-attention mechanism. Recent studies have shown that ViTs can model content-dependent long-range interactions and flexibly adjust their receptive fields to counteract noise in the data and learn effective feature representations. As a result, ViTs and their variants have been successfully applied to many computer vision tasks, including classification, detection, and segmentation.
The success of ViTs in the computer vision domain has led to a significant increase in the use of transformer-based frameworks in remote sensing analysis (see Figure 1), with applications in ultra-high-resolution image classification, change detection, pansharpening, building detection, and image captioning. This has opened a new era in remote sensing analysis, where researchers adopt various approaches, such as leveraging ImageNet pretraining or using visual transformers for remote sensing pretraining.
Similarly, there are methods in the literature based on pure transformer designs or hybrid approaches that utilize both transformers and CNNs. With the rapid emergence of transformer-based methods for various remote sensing issues, keeping up with the latest advancements has become increasingly challenging.
In this article, the authors review the progress made in remote sensing analysis and introduce popular transformer-based methods in the field, with the main contributions of the article as follows:
Providing a comprehensive overview of the applications of transformer-based models in remote sensing imaging, and the authors are the first to conduct a survey on the use of transformers in remote sensing analysis, bridging the gap between the latest advancements in computer vision and remote sensing in this rapidly evolving and popular field.
-
Overview of CNNs and Transformers, discussing their respective advantages and disadvantages.
-
Review of over 60 transformer-based research works in the literature, discussing the latest advancements in the remote sensing field.
-
Exploration of the different challenges and research directions of transformers in remote sensing analysis.
The rest of the article is organized as follows: Section 2 discusses other relevant surveys on remote sensing imaging; Section 3 provides an overview of different imaging modalities in remote sensing; Section 4 briefly outlines CNNs and visual transformers; Section 5 reviews Very High Resolution (VHR) imaging; Section 6 introduces hyperspectral image analysis; Section 7 presents advancements in transformer-based methods in Synthetic Aperture Radar (SAR); Section 8 discusses future research directions.
For more details, please refer to the original paper.
-
Paper link: https://arxiv.org/pdf/2209.01206.pdf
-
GitHub address: https://github.com/VIROBO-15/Transformer-in-Remote-Sensing
Scan the QR code to add the assistant on WeChat
About Us
