IntroductionThis issue introduces a multimodal fusion classification model based on 1D-GRU+2D-MTF-ResNet-CBAM, which has shown remarkable results in fault diagnosis tasks!
1 Model Overview and Innovations1.1 Model Overview
This model combines time-frequency images and one-dimensional time series signals, utilizing a ResNet optimized with the CBAM attention mechanism and a GRU multimodal feature fusion model for fault signal classification. It effectively integrates spatial features of time-frequency images and temporal features of one-dimensional signals, fully leveraging the advantages of multimodal features.● Dataset: CWRU Case Western Reserve University bearing dataset● Environmental Framework: Python 3.9, PyTorch 2.1 or above versions are supported● Time-Frequency Image Transformations: Provides 5 methods for time-frequency image transformations● Accuracy: 100% on the test set● Target Audience: Thesis and graduation project seekers● Code Assurance: The code is well-commented and can be run directly.Note: We also provide supporting model explanations (for learning network structure) and parameter tuning explanations! Essential for students with graduation projects or thesis needs, rich modules, high innovation, and superior performance!!Code Link: https://mbd.pub/o/bread/Z5iZl5py1.2 InnovationsInnovation 1: Multimodal FusionThis model performs multimodal fusion of time-frequency images and one-dimensional time series signals, fully utilizing the complementarity of these two types of data. Time-frequency images visualize the frequency and time characteristics of signals through a Markov Transition Field (MTF), while one-dimensional time series signals retain the original temporal dependencies, making them suitable for processing with Recurrent Neural Networks (RNNs) or GRUs. By integrating these two features:
- Time-frequency images capture high-frequency and low-frequency trends in the signal, aiding in the identification of fault features in the frequency domain.
- One-dimensional time series signals retain the temporal dependencies of the signal, reflecting the dynamic evolution of faults over time.
This dual-channel data fusion allows the model to leverage temporal, frequency, and image features simultaneously, significantly enhancing the accuracy of fault classification.Innovation 2: ResNet Optimized with CBAM Attention MechanismCBAM (Convolutional Block Attention Module) is an attention mechanism for Convolutional Neural Networks (CNNs) designed to enhance the model’s representational capacity. CBAM enhances feature representation by combining Channel Attention and Spatial Attention, thereby improving model performance.CBAM modules are inserted after each residual block of ResNet to balance computational load and performance enhancement:
(1) Enhanced Feature Representation
The combination of channel attention and spatial attention: CBAM provides two modules for channel attention and spatial attention. These modules enhance feature representation from the channel and spatial dimensions, respectively. By inserting CBAM into the basic units of ResNet, the network can better capture the importance of different features, enabling effective information extraction and aggregation at deeper levels.
(2) Fine-Grained Feature Selection
Fine-grained feature focus: CBAM allocates weights to every channel and spatial location of the feature map, allowing for fine-grained selection and emphasis of important features. This mechanism enables ResNet to intelligently ignore irrelevant background information and focus on key features for fault classification.
(3) Seamless Integration with Residual Connections
Compatibility with residual structures: CBAM can be seamlessly integrated into the residual blocks of ResNet. Due to its simple and efficient design, CBAM can be directly inserted into each residual block of ResNet without significantly increasing computational complexity. This integration allows ResNet to retain its original advantages (such as gradient flow and training stability) while gaining stronger feature expression capabilities.
(4) Performance Improvement
Performance improvement: After adding CBAM to ResNet, accuracy improvements are generally observed in tasks such as image classification and object detection. By dynamically adjusting the feature maps, CBAM helps the network better adapt to the distribution characteristics of the data and task requirements.
(5) Computational Efficiency and Flexibility
- Computational efficiency: CBAM has a simple design and low computational overhead, making it suitable for enhancing model performance without significantly increasing network complexity.
- Flexibility: CBAM can be flexibly applied to different variants of ResNet (such as ResNet-18, ResNet-34, ResNet-50, etc.) and can be adjusted and optimized according to specific task requirements.
Combining CBAM with ResNet (Residual Network) is an effective method to enhance the expressiveness of ResNet. This combination strengthens feature representation while maintaining the depth and width of the network by introducing attention mechanisms.Innovation 3: Temporal Feature Extraction Based on Stacked One-Dimensional Signal SequencesIn processing one-dimensional time series signals, we stack one-dimensional signal sequence data and use GRU (Gated Recurrent Unit) to extract temporal features. This innovative design accelerates the computational efficiency of GRU, enabling more effective extraction of key features during fault occurrence, significantly improving the accuracy of signal classification.Innovation 4: Advantages of Feature FusionThe multimodal fusion part of the model combines the time-frequency image features extracted by ResNet-CBAM and the one-dimensional signal features processed by GRU through feature concatenation. Compared to traditional models that only use single-modal features, the fused features exhibit superior performance in classification tasks, with key advantages including:
- Time-frequency images and time series signals provide different perspectives of feature information; the former offers frequency domain features while the latter retains temporal dependencies, and their combination can more comprehensively reflect the fault characteristics of the signal.
- Through feature fusion, the model becomes more robust in capturing fault features under different modalities, especially in complex fault signal environments, where the fused features can better cope with noise interference and signal variations.
This feature fusion strategy enables our model to maintain efficient and accurate classification performance across various fault modes, enhancing the model’s robustness and generalization capability in practical applications.2 Preprocessing of Bearing Fault Data2.1 Importing DataRefer to previous articles for preprocessing of 10-class fault classification, Case Western Reserve University bearing 10-class dataset:
train_set, val_set, and test_set are divided into training, validation, and test sets in a 7:2:1 ratio, and the data is saved at the end.
The above image shows the data reading format and preprocessing ideas.2.2 Data Preprocessing, Time-Frequency Image TransformationsWe provide five methods for time-frequency image transformations: Markov Transition Field (MTF), Recurrence Plot (RP), Gramian Angular Field (GAF), Continuous Wavelet Transform (CWT), and Short-Time Fourier Transform (STFT), allowing flexible replacement of the time-frequency image types in multimodal features!
This article uses the Markov Transition Field (MTF) as the processing method for time-frequency image transformations, and the generated time-frequency images are shown below:
3 Innovative Model Performance Display(1) Model Training Visualization
(2) Model Evaluation
After 50 epochs, accuracy reached 98%, with significant classification results using the 1D-GRU+2D-MTF-ResNet-CBAM network. The model effectively extracts spatial and temporal features of bearing fault signals, converges quickly, and performs excellently with high accuracy!(3) Confusion Matrix
Other Visualizations:(1) Original Data t-SNE Feature Visualization
(2) t-SNE Feature Visualization After Model Training:
If you have modeling needs or require thesis guidance, please follow the public account and contact the author.Click to Download: Complete Data and Python Codehttps://mbd.pub/o/bread/Z5iZl5py
Previous Exciting Content:Python Bearing Fault Diagnosis | From Beginner to PublicationPython Bearing Fault Diagnosis (14) High-Innovation Fault Identification ModelExclusive Original | SCI Q1 High-Innovation Bearing Fault Diagnosis Model!Attention Modification | Super Strong Bearing Fault Diagnosis Model!Bearing Fault Diagnosis Model Based on K-NN + GCN27 Bearing Fault Diagnosis Models | The Most Cost-Effective Bundle OnlineExclusive First Release | Bearing Fault Diagnosis Model Based on KAN and KAN ConvolutionTime-Frequency Image Update | Bearing Fault Diagnosis Based on Visual Top Conferences + Time-Frequency ImagesFault Diagnosis | Code Explanation + Live Course, Hot Ongoing!Quick Publication | Parallel Fault Diagnosis Model Based on 2D-SWinTransformer + 1D-CNN-SENetPosition Encoding Demystified | Detailed Explanation of Positional Encoding in TransformersResource Sharing | Intelligent Fault Diagnosis and Prediction Based on Graph Neural NetworksInnovation Points | Fault Diagnosis Model Based on Fast Fourier Convolution (FFC)Open Source Code! | Domain-Adversarial Graph Convolutional Network Fault Diagnosis Under Varying ConditionsMore Than 30 Open Source Datasets | No More Worries About Datasets for Fault Diagnosis!Fault Diagnosis | Essential for Graduate Students to Publish Papers!!!Super Strong! Top Conference Innovative Fusion! Parallel Classification Network Based on 2D-SWinTransformerMultimodal Fault Diagnosis | Large Kernel Convolution Opens a New Era of Vision!Transformer Structural Advantages, How Much Attention Do You Need?Fault Diagnosis | A Small Innovation: Feature Extraction + KAN ClassificationSuper Strong! Directly Writing in Q1! Fault Diagnosis Model Based on SSA + Informer-SENetFault Diagnosis | Signal Denoising Algorithm CollectionGraph Convolutional Fault Diagnosis, New GAT, SGCN, GIN Classification Models AddedHeavyweight! Bearing Life Prediction Bundle is Here!Don’t Miss! Major Update on Fault Diagnosis + Time-Frequency Image Classification!Recommended Journals for Intelligent Fault Diagnosis and Life PredictionFault Diagnosis Directly Written in Q1, Graph Convolution + BiGRU-Attention Parallel Diagnosis ModelEND1. Copyright Statement: Original content is not easy, please indicate the source when reprinting from the WeChat public account [Modeling Pioneer]2. If you like it, please follow, give the article a “look”, share it in your circle of friends or discussion group3. Follow the WeChat public account [Modeling Pioneer], reply “Bearing Fault” to get the Case Western Reserve University bearing dataset for free