(Click the official account above, you can quickly follow)
Source: taogang
Link: http://my.oschina.net/taogang/blog/686568?fromerr=DzNrW1c1
Introduction to Neural Networks
A neural network attempts to simulate the relationships between neurons in the brain to process information. Its computational model typically requires a large number of interconnected nodes. Each neuron processes weighted input values from other neighboring neurons through a special output function.
The strength of information transfer between neurons is defined by the so-called weight values, and the algorithm continuously adjusts these weights to achieve a self-learning process.
Neural networks are divided into multiple layers, as shown in the image above, including an input layer, hidden layers, and an output layer.
JS Linear Algebra Library
The calculations in neural networks involve a large number of matrix computations, and there are many open-source linear algebra libraries available. Python has the famous numpy, which is very well-known. JavaScript also has several options:
-
http://numericjs.com/index.php
-
https://github.com/jstat/jstat
-
https://mkaz.tech/javascript-linear-algebra-calculator.html
I used numericjs, and it works quite well. I recommend everyone to give it a try.
Two-Layer Neural Network
We have some simple input-output data to train the neural network. Here, each row represents a data point. There are three input parameters and one output parameter.
First, we implement the simplest neural network with no hidden layers, where the input is directly connected to the output.
Since there are three input parameters and one output, our neural network will have three nodes in the input layer and one in the output layer.
// Sigmoid function
function nonlin(x, deriv) {
if(deriv) {
return numeric.mul(x, numeric.sub(1, x));
}
return numeric.div(1, numeric.add(1, numeric.exp(numeric.neg(x))));
}
function train_neural(X, y, iteration) {
// initialize weights
var syn0 = numeric.sub(numeric.mul(2, numeric.random([3, 1])), 1);
//Training loop
var i = 0;
for(; i < iteration; i++) {
var l0 = X;
var l1 = nonlin(numeric.dot(l0, syn0));
var l1_error = numeric.sub(y, l1);
var l1_delta = numeric.mul(l1_error, nonlin(l1, true));
syn0 = numeric.add(syn0, numeric.dot(numeric.transpose(l0), l1_delta));
}
}
//Initial input/output values
var X = [
[0, 0, 1],
[0, 1, 1],
[1, 0, 1],
[1, 1, 1]
];
var y = [
[0],
[0],
[1],
[1]
];
train_neural(X, y, 1000);
Let’s briefly introduce the training code and process:
-
X input data
-
y output data
-
nonlin, sigmoid function
-
l0, the first layer of the network, which equals the input data
-
l1, the second layer of the network, which is the output layer
-
syn0, the weights of the first layer of the network
The training iteration process starts with an initial weight, computes an output value using this weight, subtracts this value from the target result to obtain a difference value, and then uses this difference to adjust the weights.
After 1000 iterations, the network output is: [0.03, 0.02, 0.979, 0.974]
After 1000 iterations, the syn0 weight values are: [7.266, -0.221, -3.415]
Here we find that the weight of the first node is larger, which is consistent with our data. By observing the data, we can also find that the output value is strongly correlated with the input value in the first column. If we increase the number of iterations, this value will be even larger.
Three-Layer Neural Network
Now we have a new set of data. Upon observation, we find that the third column is completely unrelated to the result. When the first and second columns are the same, the result is 0; otherwise, it is 1. This is a non-linear relationship, and to learn effectively, we add another layer, making the network look like this.
// Sigmoid function
function nonlin(x, deriv) {
if(deriv) {
return numeric.mul(x, numeric.sub(1, x));
}
return numeric.div(1, numeric.add(1, numeric.exp(numeric.neg(x))));
}
function train_neural(X, y, iteration) {
// initialize weights
var syn0 = [
[-0.1653904, 0.11737966, -0.71922612, -0.60379702],
[0.60148914, 0.93652315, -0.37315164, 0.38464523],
[0.7527783, 0.78921333, -0.82991158, -0.92189043]
];
var syn1 = [
[-0.66033916],
[0.75628501],
[-0.80330633],
[-0.15778475]
];
//Training loop
var i = 0;
for(; i < 1000; i++) {
var l0 = X;
var l1 = nonlin(numeric.dot(l0, syn0));
var l2 = nonlin(numeric.dot(l1, syn1));
var l2_error = numeric.sub(y, l2);
var l2_delta = numeric.mul(l2_error, nonlin(l2, true));
var l1_error = numeric.dot(l2_delta, numeric.transpose(syn1));
var l1_delta = numeric.mul(l1_error, nonlin(l1, true));
syn1 = numeric.add(syn1, numeric.dot(numeric.transpose(l1), l2_delta));
syn0 = numeric.add(syn0, numeric.dot(numeric.transpose(l0), l1_delta));
}
}
//Initial input/output values
var X = [
[0, 0, 1],
[0, 1, 1],
[1, 0, 1],
[1, 1, 1]
];
var y = [
[0],
[1],
[1],
[0]
];
train_neural(X, y, 1000);
The training process is not much different from the previous two layers, just that there is one more layer. By adding this layer, we can effectively learn the complex non-linear relationships in the data.
After 1000 iterations, the output value is: [0.02, 0.95, 0.94, 0.05]
syn0 :
If anyone is interested in the training process, you can run my code:
-
http://codepen.io/gangtao/pen/yJLdXG
-
http://codepen.io/gangtao/pen/jrOjpX
