Implementing a Simple Neural Network Algorithm in JS

(Click the official account above, you can quickly follow)

Source: taogang

Link: http://my.oschina.net/taogang/blog/686568?fromerr=DzNrW1c1

Introduction to Neural Networks

A neural network attempts to simulate the relationships between neurons in the brain to process information. Its computational model typically requires a large number of interconnected nodes. Each neuron processes weighted input values from other neighboring neurons through a special output function.

The strength of information transfer between neurons is defined by the so-called weight values, and the algorithm continuously adjusts these weights to achieve a self-learning process.

Implementing a Simple Neural Network Algorithm in JS

Neural networks are divided into multiple layers, as shown in the image above, including an input layer, hidden layers, and an output layer.

JS Linear Algebra Library

The calculations in neural networks involve a large number of matrix computations, and there are many open-source linear algebra libraries available. Python has the famous numpy, which is very well-known. JavaScript also has several options:

  • http://numericjs.com/index.php

  • https://github.com/jstat/jstat

  • https://mkaz.tech/javascript-linear-algebra-calculator.html

I used numericjs, and it works quite well. I recommend everyone to give it a try.

Two-Layer Neural Network

We have some simple input-output data to train the neural network. Here, each row represents a data point. There are three input parameters and one output parameter.

Implementing a Simple Neural Network Algorithm in JS

First, we implement the simplest neural network with no hidden layers, where the input is directly connected to the output.

Implementing a Simple Neural Network Algorithm in JS

Since there are three input parameters and one output, our neural network will have three nodes in the input layer and one in the output layer.

// Sigmoid function

function nonlin(x, deriv) {

if(deriv) {

return numeric.mul(x, numeric.sub(1, x));

}

return numeric.div(1, numeric.add(1, numeric.exp(numeric.neg(x))));

}

function train_neural(X, y, iteration) {

// initialize weights

var syn0 = numeric.sub(numeric.mul(2, numeric.random([3, 1])), 1);

//Training loop

var i = 0;

for(; i < iteration; i++) {

var l0 = X;

var l1 = nonlin(numeric.dot(l0, syn0));

var l1_error = numeric.sub(y, l1);

var l1_delta = numeric.mul(l1_error, nonlin(l1, true));

syn0 = numeric.add(syn0, numeric.dot(numeric.transpose(l0), l1_delta));

}

}

//Initial input/output values

var X = [

[0, 0, 1],

[0, 1, 1],

[1, 0, 1],

[1, 1, 1]

];

var y = [

[0],

[0],

[1],

[1]

];

train_neural(X, y, 1000);

Let’s briefly introduce the training code and process:

  • X input data

  • y output data

  • nonlin, sigmoid function

  • l0, the first layer of the network, which equals the input data

  • l1, the second layer of the network, which is the output layer

  • syn0, the weights of the first layer of the network

The training iteration process starts with an initial weight, computes an output value using this weight, subtracts this value from the target result to obtain a difference value, and then uses this difference to adjust the weights.

After 1000 iterations, the network output is: [0.03, 0.02, 0.979, 0.974]

After 1000 iterations, the syn0 weight values are: [7.266, -0.221, -3.415]

Here we find that the weight of the first node is larger, which is consistent with our data. By observing the data, we can also find that the output value is strongly correlated with the input value in the first column. If we increase the number of iterations, this value will be even larger.

Three-Layer Neural Network

Implementing a Simple Neural Network Algorithm in JS

Now we have a new set of data. Upon observation, we find that the third column is completely unrelated to the result. When the first and second columns are the same, the result is 0; otherwise, it is 1. This is a non-linear relationship, and to learn effectively, we add another layer, making the network look like this.

Implementing a Simple Neural Network Algorithm in JS

// Sigmoid function

function nonlin(x, deriv) {

if(deriv) {

return numeric.mul(x, numeric.sub(1, x));

}

return numeric.div(1, numeric.add(1, numeric.exp(numeric.neg(x))));

}

function train_neural(X, y, iteration) {

// initialize weights

var syn0 = [

[-0.1653904, 0.11737966, -0.71922612, -0.60379702],

[0.60148914, 0.93652315, -0.37315164, 0.38464523],

[0.7527783, 0.78921333, -0.82991158, -0.92189043]

];

var syn1 = [

[-0.66033916],

[0.75628501],

[-0.80330633],

[-0.15778475]

];

//Training loop

var i = 0;

for(; i < 1000; i++) {

var l0 = X;

var l1 = nonlin(numeric.dot(l0, syn0));

var l2 = nonlin(numeric.dot(l1, syn1));

var l2_error = numeric.sub(y, l2);

var l2_delta = numeric.mul(l2_error, nonlin(l2, true));

var l1_error = numeric.dot(l2_delta, numeric.transpose(syn1));

var l1_delta = numeric.mul(l1_error, nonlin(l1, true));

syn1 = numeric.add(syn1, numeric.dot(numeric.transpose(l1), l2_delta));

syn0 = numeric.add(syn0, numeric.dot(numeric.transpose(l0), l1_delta));

}

}

//Initial input/output values

var X = [

[0, 0, 1],

[0, 1, 1],

[1, 0, 1],

[1, 1, 1]

];

var y = [

[0],

[1],

[1],

[0]

];

train_neural(X, y, 1000);

The training process is not much different from the previous two layers, just that there is one more layer. By adding this layer, we can effectively learn the complex non-linear relationships in the data.

After 1000 iterations, the output value is: [0.02, 0.95, 0.94, 0.05]

syn0 :

Implementing a Simple Neural Network Algorithm in JS

If anyone is interested in the training process, you can run my code:

  • http://codepen.io/gangtao/pen/yJLdXG

  • http://codepen.io/gangtao/pen/jrOjpX

Implementing a Simple Neural Network Algorithm in JS

Leave a Comment