Click on the above “Beginner Learning Visuals” to select “Bookmark” or “Pin”
Heavyweight content delivered at the first moment
This article is reprinted from: AI Algorithms and Image Processing
Recently, researchers from Harbin Institute of Technology, Guangdong University of Technology, Tsinghua University, and National Tsing Hua University in Taiwan jointly authored a comprehensive review on deep learning in image denoising, published on arXiv. This review systematically summarizes the importance of image denoising, the development of image denoising techniques, the advantages and disadvantages of traditional machine learning and deep learning image denoising techniques, and analyzes the challenges and potential research points faced by image denoising techniques. This review is of significant guidance for both academia and industry and is worth studying.Deep Learning on Image Denoising: An Overview Paper link: https://arxiv.org/abs/1912.13171 Related code link: https://github.com/hellloxiaotian
1 Background and MotivationDigital imaging devices have been applied in various fields such as weather forecasting, disaster rescue, security monitoring, and medical diagnosis. However, digital devices are often affected by camera shake, moving objects, low light, and noise, resulting in unclear captured photos. Therefore, research on image denoising techniques holds significant theoretical and practical value.Image denoising techniques became a research hotspot in the 1990s. For example, using non-local similarity to optimize sparse methods can improve denoising performance. Dictionary learning helps to quickly remove noise. Prior knowledge can recover details of the potential clean image by smoothing the noisy image. More competitive denoising methods include MRF, WNNM, LSSC, CSF, TNRD, and GHEP.Although most of these methods achieve good performance in image denoising, they have the following drawbacks:(1) Involves complex optimization methods during the testing phase,(2) Manual parameter settings, (3) A fixed model to handle a single denoising task. Deep learning techniques, with flexible structures and strong self-learning capabilities, can be used to address these shortcomings.2 Research Framework of This PaperThis paper introduces deep learning applications in image denoising from shallow to deep. First, it introduces the basic framework of deep learning in image processing, including supervised and unsupervised machine learning, convolutional networks, main structures of deep learning in image denoising (such as VGG, ResNet, GoogLeNet, and GAN), and commonly used software and hardware for deep learning techniques; secondly, it focuses on the applications of deep learning techniques in image denoising, as shown in Figure 1:
Applications of deep learning techniques in image denoising include deep learning techniques for denoising added white noise images, real noisy image denoising, blind denoising, and hybrid noisy image denoising.2.1 Deep Learning Techniques for Denoising Added White Noise Images:CNN/NN for AWNI denoising, CNN/NN and common feature extraction methods for AWNI denoising, and the combination of optimization method and CNN/NN for AWNI denoising.2.1.1 CNN/NN for AWNI denoising:Designing different network structures based on noise attributes is crucial. There are the following ways to design network structures:(1) Use multiple views to design the network;(2) Change the Loss function;(3) Increase the width or depth of CNN;(4) Add arbitrary plugins in CNN;(5) Use skip connections or cascaded operations in CNN.Additional notes:The first method includes three types: a noisy image as input for multiple sub-networks; different angles of a single sample as input for the network; different channels of a network as input.The fourth method: arbitrary plugins include activation functions, dilated convolutions, fully connected layers, and pooling layers.The fifth method includes skip connections and cascaded operations. Table 1 provides a summary of CNNs/NNs for AWNI denoising.
2.1.2 CNN/NN and Common Feature Extraction Methods for AWNI Denoising:Weak edge-information, non-linear, high-dimensional, and non-salient noisy images and high computational costs. For weak edge-information noisy images, CNN with transformation domain methods is very effective in removing noise. For non-linear noisy images, CNN with kernel methods is very effective in recovering potential clean images. This type of method generally has three steps:The first step is to extract features using CNN,The second step is to convert non-linear features to linear features using kernel methods,The third step is to reconstruct the potential clean image using residual techniques.For high-dimensional noisy images, the combination of CNN and dimensionality reduction methods is a common denoising method. For non-salient noisy images, signal processing methods can guide CNN to extract significant features. For high computational costs, combining CNN with image attributes can effectively reduce complexity. More information about the methods mentioned above is shown in Table 2.
2.1.3 The Combination of Optimization Method and CNN/NN for AWNI Denoising:(1) Improve denoising speed,(2) Improve denoising performance.To improve denoising efficiency, embedding optimization methods into CNN to find optimal solutions is a good tool. Additionally, mapping noise and noisy image blocks as input for CNN can also improve the speed of noise prediction. To improve denoising performance, combining CNN with prior knowledge can effectively remove noise. Table 3 shows detailed information about the combination of optimization method and CNN/NN for AWNI denoising.
2.2 Deep Learning Techniques for Real Noisy Image Denoising:Single end-to-end CNN and the combination of CNN and prior knowledge. For the first category, designing network structures for processing real noisy images is popular. Integrating multi-scale, skip connections, batch renormalization, dilated convolutions, and attention mechanisms into CNN can effectively handle real noisy images. Detailed information about these methods is shown in Table 4:
For the second aspect, combining CNN with prior knowledge can effectively solve real noisy images. Verification knowledge includes HQS, TV, and channel prior. Table 5 shows detailed information about these methods.
2.3 Deep Learning Techniques for Blind Denoising:Using image devices and soft shrinkage combined with CNN/NN can perform blind denoising well. More methods are presented in Table 6.
2.4 Deep Learning Techniques for Hybrid Noisy Image Denoising:Using warped guidance combined with CNN, single CNN, and the combination of CNN and iterative algorithms can effectively remove hybrid noise. More information is shown in Table 7:
3 Experimental Results3.1 Database:3.1.1 Training Set:BSD400, Waterloo Exploration Database, and PolyU-Real-World Noisy Images.3.1.2 Test Dataset:Set12, BSD68, CBSD68, Kodak24, McMaster, CC, DND, NC12, SIDD, and Nam.3.2 Results of Deep Learning Techniques for Additive White Noisy-Image Denoising
3.3 Results of Deep Learning Techniques for Real Noisy Image Denoising
Deep Learning Techniques for Blind Denoising Results
3.4 Results of Deep Learning Techniques for Hybrid Noisy Image Denoising
4 DiscussionDeep learning generally improves image performance, denoising efficiency, and complex noisy images in image denoising.4.1 Improving Denoising Performance, Solutions Include:(1) Increasing the receptive field of the network can capture more contextual information to improve denoising performance. Among them, increasing the width and depth of the network is the most common way to increase the receptive field, however, they lead to high computational costs and more memory consumption. Dilated convolutions can effectively solve this problem.(2) Combining CNN with prior knowledge can extract more robust features.(3) Combining local and global information can enhance the network’s memory capability.(4) Integrating signal processing mechanisms into CNN can better suppress noise.(5) Data augmentation can improve image denoising performance.(6) Transfer learning, graph learning, and network search can effectively handle noisy images.4.2 Improving Denoising EfficiencyCompressing networks can effectively improve denoising speed. Reducing network width and depth, using smaller convolution kernels, and group convolutions can effectively increase denoising speed.4.3 Solving Complex Noisy ImagesUsing distribution mechanisms is very popular. The first step is to use CNN to estimate noise levels as ground truth or to recover high-resolution images. The second step is to recover potential clean images.4.4 Challenges(1) Deeper networks require more memory.(2) Deeper denoising networks cannot stably train models for real noisy images or unlabeled noisy images.(3) Real noisy images are not easy to obtain.(4) Deeper networks face difficulties in solving unsupervised denoising tasks.(5) Finding more accurate denoising metrics.5 ConclusionIn this paper, we conducted a comprehensive study and systematic summary of different denoising networks. First, we presented the basic framework of deep learning for image denoising. Then, we provided deep learning techniques for different denoising tasks (such as added white noise, blind noise, real noise, and hybrid noise images). Next, we analyzed the motivations and theories of denoising networks for different tasks. Finally, we compared the denoising results, efficiency, and visualization effects of different methods. Additionally, we pointed out potential research points and challenges for deep learning techniques in image denoising.Download 1: OpenCV-Contrib Extended Module Chinese TutorialReply in the background of the “Beginner Learning Visuals” public account:Extended Module Chinese Tutorial, you can download the first OpenCV extended module tutorial in Chinese online, covering installation of extended modules, SFM algorithms, stereo vision, target tracking, biological vision, super-resolution processing, and more than twenty chapters of content.Download 2: Python Visual Practical Projects 52 LecturesReply in the background of the “Beginner Learning Visuals” public account: Python Visual Practical Projects, you can download 31 visual practical projects including image segmentation, mask detection, lane line detection, vehicle counting, adding eyeliner, license plate recognition, character recognition, emotion detection, text content extraction, face recognition, etc., to help quickly learn computer vision.Download 3: OpenCV Practical Projects 20 LecturesReply in the background of the “Beginner Learning Visuals” public account: OpenCV Practical Projects 20 Lectures, you can download 20 practical projects based on OpenCV to advance your OpenCV learning.
Group Chat
Welcome to join the reader group of the public account to communicate with peers. Currently, there are WeChat groups for SLAM, 3D vision, sensor, autonomous driving, computational photography, detection, segmentation, recognition, medical imaging, GAN, algorithm competitions and more (will gradually be subdivided in the future). Please scan the WeChat ID below to join the group, and note: “nickname + school/company + research direction”, for example: “Zhang San + Shanghai Jiao Tong University + Visual SLAM”. Please follow the format to note, otherwise, it will not be approved. After successful addition, you will be invited into relevant WeChat groups based on research direction. Please do not send advertisements in the group, otherwise you will be removed from the group, thank you for your understanding~
