Understanding Deep Learning: Basics of Artificial Neural Networks

Reprinted from Yunqi Community as required

ID: yunqiinsight

Author | doflamingo

I have touched on deep learning during my studies, but only superficially. In this era of data and algorithms, it is necessary to get closer to the algorithms. Therefore, from the perspective of an engineer, I hope to record the basics of deep learning through a few articles, which also serves as a summary and accumulation of my learning. The overall idea is ANN-CNN-DNN, and I will add anything I remember in between.

This is a typical diagram of an artificial neural network. The nodes in the diagram are called neurons, and the diagram is divided into three layers: the first layer is the input layer, the second layer is the hidden layer, and the third layer is the output layer. The input layer receives inputs from the external world, represented as pixel values of images, feature values of entities, etc. The output layer predicts probability results, visualized as whether the image is a portrait or the entity is a potential merchant.

A neuron combines multiple inputs and their weights into a single input for the next layer. For example:

Neurons generally use the sigmoid function. The reason for using the sigmoid function is a topic worth discussing. For more details, you can refer to this article to understand the characteristics of sigmoid: http://www.tuicool.com/articles/uMraAb.

Here, w represents the weight vector, x represents the input vector, and b is the threshold of this node.

So the next question is how to choose suitable weights and thresholds to construct an appropriate network.

The network structure often determines the algorithm’s complexity and model adjustability. The output layer is mainly determined by vectors, while the output layer is mainly determined by the prediction type. The main issue lies in the selection of the number of hidden layers and nodes. More nodes and layers mean stronger model adjustability and finer prediction granularity, but it also means higher computational complexity. In practice, 1-2 hidden layers are generally chosen, and the number of nodes is treated as an adjustable parameter.

First, define the loss function. The significance of the loss function is to evaluate the difference between the predicted results and the actual results for the training set:

This loss function is actually the variance between the predicted results and the actual results.

We hope to adjust the weights w and the threshold b to minimize the difference between the predicted results and the actual results. This is equivalent to finding the optimal solution in a solution space. There are many methods, such as gradient descent, quasi-Newton methods, etc.

From the above formula, we can see that the change in the loss function can be described as the sum of the changes in loss across each dimension v, represented as a vector:

To make the loss smaller rather than larger, the change in loss should be less than 0, so we take:

Then, the decrease in loss can be expressed as:

Backpropagation is actually about evaluating the contribution of each parameter to the prediction error after a prediction is completed and adjusting it accordingly. The adjustment method can be obtained by taking the derivative of the loss function with respect to the weights:

Through multiple iterations, the minimum value of the loss function is obtained. The step size determines the convergence speed of the function.

Artificial Neural Networks are like a child lost in an unfamiliar city, evaluating each step to calculate the likelihood of reaching the destination, gradually making their way there. The three important components of artificial neural networks are:

1. Structure: Hierarchical Network 2. Solving Method: Gradient Descent 3. Solving Philosophy: Backpropagation