Time Series Prediction Method Based on LSTM and Attention Mechanism

Traditional time series prediction methods include autoregression, moving average, and autoregressive moving average, mainly based on statistical methods, which assume the stationarity of time series and perform linear weighted predictions on historical values. Traditional time series prediction methods are only suitable for small-scale univariate predictions, as they struggle with nonlinear modeling of time series data, leading to low prediction accuracy.

With the continuous development of artificial intelligence technologies, machine learning models such as KNN, GBRT, RF, and XGBoost have been successfully applied in the field of time series prediction. Compared to traditional time series prediction methods, machine learning techniques have stronger capabilities in nonlinear modeling of time series data, which can enhance prediction accuracy. However, machine learning techniques build models through feature engineering without considering the dependencies between time series data.

In the digital age, with the continuous improvement of computer computing power, deep learning technologies have been widely applied.Deep learning technologies have strong nonlinear modeling capabilities, while recurrent neural networks and their variants can effectively model the long-term dependencies of time series data. The attention mechanism can effectively capture the correlations between time series data, which plays an important role in improving prediction accuracy^[2].Academia and industry have applied Long Short-Term Memory Networks (LSTM) based on attention mechanisms for time series prediction.

LSTM is a special type of recurrent neural network designed to address the issues of gradient vanishing and explosion during the training process of long sequences. Figure 1 shows the structure of LSTM, with four neural network layers inside each neuron.

Figure 1 LSTM Structure

Figure 2 represents the neuron state of LSTM, indicated by the horizontal line running across the top. LSTM uses a gating structure to delete or add information to the neuron state. A gate is an information selection mechanism composed of a Sigmoid function and pointwise multiplication operator, which determines how much information passes through.

Figure 2 LSTM Neuron State

Figure 3 shows the forget gate, which controls how much information from the previous moment’s internal state needs to be forgotten.

Figure 3 LSTM Forget Gate

Figure 4 shows the input gate, which controls how much information from the current moment’s candidate state needs to be saved.

Figure 4 LSTM Input Gate

Figure 5 shows the output gate, which controls how much information from the current moment’s internal state needs to be output to the external state.

Figure 5 LSTM Output Gate

The attention mechanism in deep learning is an optimization method. Its basic idea is inspired by the human visual processing mechanism. When humans gather information with their eyes, they prioritize the most important information and focus more attention on it, thereby extracting key information from complex data.

Attention mechanisms can be divided into soft attention mechanisms and hard attention mechanisms. The weight distribution of the soft attention mechanism ranges from [0, 1], while the hard attention mechanism has weights of either 0 or 1.

In recent years, the attention mechanism has significantly improved accuracy metrics in fields such as computer vision and natural language processing. Consequently, research has emerged applying attention mechanisms to time series prediction to enhance prediction accuracy.

In the field of time series prediction, many researchers often combine LSTM with attention mechanisms.This combination typically occurs in two ways:based on attention mechanisms at different times and based on attention mechanisms for different features.The attention mechanism based on different times assigns different weights to the hidden layer outputs at different times, and then obtains a context vector for LSTM through weighted summation;the attention mechanism based on different features assigns different attention weights to different dimensions of the output vector^[3].

LSTM can effectively learn the temporal correlations in the sequence, and the attention mechanism can effectively extract the dynamic change features of the data, making the analysis of correlations in time series data more accurate, thus leading to more precise prediction results.

The banking industry is a service-oriented industry, where high-quality service is the foundation for gaining customers’ lasting trust. To provide better services, banks need to accurately predict future cash levels, business volumes, transaction volumes, and workloads related to relevant business scenarios. Therefore, the time series prediction methods based on LSTM and attention mechanisms can be applied to the following banking business scenarios:

• Cash level prediction for self-service ATMs. Banks predict the cash levels stored in each self-service ATM based on business volumes and related management requirements, and formulate cash replenishment plans.

• Branch cash level prediction. Predict the cash usage for branch cash business, formulate cash allocation plans for branches, and request the cash center for cash preparation and allocation.

• Remote banking center business volume prediction. Due to the large number of agents in various business lines at remote banking centers, unreasonable scheduling can occur based on human experience; this can be resolved through business volume predictions.

• Optimizing human resource allocation at branches. By predicting the transaction volumes and workloads at various positions in branches, reasonable staff scheduling can be achieved, optimizing human resource allocation.

• Reasonable pricing of mortgage-backed securities (MBS). Accurate predictions of prepayments can guide the reasonable pricing of MBS.