Build Machine Learning Models in the Browser with TensorFlow.js and Python

Build Machine Learning Models in the Browser with TensorFlow.js and Python

by

Build Machine Learning Models in the Browser with TensorFlow.js and Python

Author: MOHD SANAD ZAKI RIZVI

Translator: Wu Jindi

Proofreader: Ding Nanya

This article is about 5500 words, recommended reading time 15 minutes.

This article first introduces the importance of TensorFlow.js and its components, and then describes how to build machine learning models in the browser using it. Next, we will build an application that uses the computer’s webcam to detect body posture.

Overview

  • TensorFlow.js (formerly deeplearn.js) allows us to build machine learning and deep learning models in the browser without any complicated installation steps.

  • The two components of TensorFlow.js are the Core API and the Layer API.

  • Learn how to build a great model that classifies images from the webcam using TensorFlow.js.

Introduction

What tools do you prefer to write machine learning models? Data scientists will give various answers to this eternal question. Some prefer RStudio, while others prefer Jupyter Notebooks. I definitely belong to the latter.

So when I first encountered TensorFlow.js (formerly deeplearn.js), my heart exploded. Building machine learning models in the browser? Using JavaScript? It sounded too good to be true!

Over 4.3 billion people use web browsers—about 55% of the world’s population. — Wikipedia (March 2019)

Google’s TensorFlow.js not only brings machine learning into the browser and democratizes it, but it is also a perfect machine learning gateway for developers who frequently use JavaScript.

Build Machine Learning Models in the Browser with TensorFlow.js and Python

Our web browsers are among the most accessible platforms. That is why it makes sense to build applications that can not only train machine learning models but also ‘learn’ or ‘transfer learn’ right in the browser itself.

In this article, we will first understand the importance of using TensorFlow.js and its different components. Then we will delve into how to build our own machine learning models in the browser using TensorFlow.js. Finally, we will build an application to detect your body posture using the computer’s webcam!

If you are new to TensorFlow, you can learn more in the articles below:

  • TensorFlow 101: Understanding Tensors and Graphs to get you started with Deep Learning

  • Introduction to Implementing Neural Networks using TensorFlow

Table of Contents

1. Why You Should Use TensorFlow.js?

1.1 Image Classification in the Browser with Webcam

1.2 Features of TensorFlow.js

2. Understanding Machine Learning in the Browser

2.1 Core API: Working with Tensors

2.2 Layer API: Building Models like Keras

3. Utilizing Google’s Pre-trained Model: PoseNet

1. Why Use TensorFlow.js?

I will answer this question in a unique way. I won’t delve into the theoretical aspects of TensorFlow.js, nor will I list why it is such an incredible tool.

Instead, I will simply show you what you would miss if you didn’t use TensorFlow.js. So, let’s build an application in 5 minutes that uses your webcam to classify images. Yes, we will dive straight into the code!

This is the best part—you don’t need to install anything to do this! Just a text editor and a web browser. The animated GIF below shows the application we are going to build:

Build Machine Learning Models in the Browser with TensorFlow.js and Python

How cool is that! I completed it in minutes in the browser. So let’s look at the steps and the code to help you build your own image classification model in the web browser.

1.1 Build an Image Classification Model in the Browser with Webcam

Open your preferred text editor and create a file named index.html. Save the following code in this file:

<!DOCTYPE html>  <html>  <head>      <meta charset="UTF-8">      <meta http-equiv="X-UA-Compatible" content="IE=edge">      <meta name="viewport" content="width=device-width, initial-scale=1">      <!-- title of the page -->      <title>image_classification</title>      <!-- load processing library-->      <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.8.0/p5.min.js"></script>      <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.8.0/addons/p5.dom.min.js"></script>      <!-- load ml5.js -->      <script src="https://unpkg.com/[email protected]/dist/ml5.min.js"></script>      <!-- load index.js -->      <script src="index.js"></script>  </head>  <body>      <!-- this is where the video will be shown -->      <video id="video"></video>  </body>  </html>

Next, create another file named index.js and write the following code in it:

let mobilenet;    let video;    let label = '';        // when model is ready make predictions    function modelReady() {        console.log('Model is ready!!!');        mobilenet.predict(gotResults);    }    function gotResults(error, results) {      if (error) {          console.error(error);      } else {          label = results[0].className;          // loop the inference by calling itself          mobilenet.predict(gotResults);      }  }    // setup function  function setup() {      createCanvas(640, 550);      // ml5 to create video capture      video = createCapture(VIDEO);      video.hide();      background(0);      // load the MobileNet and apply it on video feed      mobilenet = ml5.imageClassifier('MobileNet', video, modelReady);  }    function draw() {      background(0);      // show video       image(video, 0, 0);      fill(255);      textSize(32);      // show prediction label       text(label, 10, height - 20);  }

Save these two files and open the index.html file in a browser such as Google Chrome or Mozilla Firefox.That’s it! You have now created an application that can classify images in real-time using your webcam in the browser!Here’s what it looks like on my computer:

Video Link:

https://s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2019/05/mobilenet_demo.mp4?_=1

Key points to note in this example:

  • In the above example, we used a pre-trained image classification model called MobileNet (https://ai.googleblog.com/2017/06/mobilenets-open-source-models-for.html)

  • We used ml5.js (https://ml5js.org/), a library built on TensorFlow. It loads the MobileNet model into the browser and performs inference on the video stream.

  • We also utilized the P5.js (https://p5js.org/) library to handle video input and display labels on the video itself.

I don’t need to install anything on my computer. This example should work on any modern system, whether it is Linux, Windows, or macOS—that’s the power of building models on the web using JavaScript.

Now, let’s look at the powerful features TensorFlow.js offers and how to leverage them to deploy machine learning models in the browser.

1.2 Features of TensorFlow.js

TensorFlow.js is a library for developing and training ML models in JavaScript and deploying them in the browser or Node.js.

TensorFlow.js provides many functionalities for us to use.

It is an extension of TensorFlow in JavaScript, which is the programming language behind almost all websites, browsers, or application logic we use on the internet. JavaScript, like Python, is versatile, so using it to develop machine learning models brings us many benefits:

  • It is easier to deploy if the ML model is written in a web language.

  • Since all major browsers support JavaScript, you can use it everywhere without worrying about platform types or other compatibility issues. The same goes for your users.

  • TensorFlow.js is a client-side library, meaning it can train or run ML models in the user’s browser. This alleviates any concerns related to data privacy.

  • Running real-time inference on your client makes your applications more interactive as they can respond to user input immediately (like the webcam application we built earlier).

Build Machine Learning Models in the Browser with TensorFlow.js and Python

TensorFlow.js currently offers the following main features:

  • Machine Learning in the Browser:You can create and train ML models in the browser using TensorFlow.js.

  • Google’s Pre-trained Models:TensorFlow.js comes with a set of pre-trained models from Google for tasks like object detection, image segmentation, speech recognition, and text toxicity classification.

  • Transfer Learning:You can perform transfer learning by retraining parts of already trained models, such as MobileNet in TensorFlow.js.

  • Deploy Python Models: Models trained with Keras or TensorFlow can be easily imported into the browser for deployment using TensorFlow.js.

In this article, we will focus on the first two features. In the second part of this series (coming soon!), we will discuss how to perform transfer learning and deploy our models in Python.

2. Understanding Machine Learning in the Browser

TensorFlow.js provides two ways to train models (very similar to TensorFlow):

  • The first method is to define models using low-level tensor operations with the Core API.

  • The second method is to define models using the Layers API, similar to Keras.

Let’s understand these two methods through a few examples. After all, the best way to learn a concept is to put it into practice!

First, set up your HTML file:

Create a new index.html file on your computer and write the following code in it:

<html lang="en"><head>  <meta charset="UTF-8">  <meta name="viewport" content="width=device-width, initial-scale=1.0">  <meta http-equiv="X-UA-Compatible" content="ie=edge">  <!-- load Tensorflow.js -->  <script src="https://cdn.jsdelivr.net/npm/@tensorflow/[email protected]/dist/tf.min.js"></script></head><body>  <h1>Tensorflow.js Core API</h1>  <!-- <script src="index.js"></script> -->  <script type="text/javascript"></script></body></html>

We have created a basic HTML page and loaded TensorFlow.js from a cloud URL (line 7).

Note on Installing TensorFlow.js (deeplearn.js):

Since TensorFlow.js is designed for the browser, the easiest way to install and use TensorFlow.js is to not install it at all. You can simply load it from the URL in your HTML.

What if you want to work locally? In fact, you can use TensorFlow.js in Jupyter Notebook, just like you would normally do in Python or R. This is a solution that works for everyone!

This local method is a bit longer and takes some time, so this article will not use it. If you do want to learn how to do it, you can start by installing the ijavascript kernel for Jupyter. Here’s a screenshot of my Jupyter Notebook:

Build Machine Learning Models in the Browser with TensorFlow.js and Python

Now, the recommended way to use TensorFlow.js is to load it directly from the official URL of the library. You just need to add the following line to your HTML file:

<script src="https://cdn.jsdelivr.net/npm/@tensorflow/[email protected]/dist/tf.min.js"></script>

Done! It’s really that simple.

2.1 Core API: Working with Tensors

The Core API is quite similar to TensorFlow Core, where we can define models using low-level tensor operations and linear algebra.

This is very useful if we want to build custom models or want to build neural networks from scratch. Let’s take an example of using tensors in the browser.

First, add the following code between the <script> </script> tags in the index.html file:

const a = tf.tensor([1, 2, 3, 4]);const b = tf.tensor([10, 20, 30, 40]);const y = a.add(b); // equivalent to tf.add(a, b)const z = a.mul(b);y.print();z.print();

The <script> tag basically indicates JavaScript. Anything we write between these tags will be executed as JavaScript code. Here’s what the index.html now looks like:

<html lang="en"><head>  <meta charset="UTF-8">  <meta name="viewport" content="width=device-width, initial-scale=1.0">  <meta http-equiv="X-UA-Compatible" content="ie=edge">  <!-- load Tensorflow.js -->  <script src="https://cdn.jsdelivr.net/npm/@tensorflow/[email protected]/dist/tf.min.js"></script></head><body>  <h1>Tensorflow.js Core API</h1>  <!-- <script src="index.js"></script> -->  <script type="text/javascript">    const a = tf.tensor([1, 2, 3, 4]);    const b = tf.tensor([10, 20, 30, 40]);    const y = a.add(b);  // equivalent to tf.add(a, b)    const z = a.mul(b);  // equivalent to tf.mul(a, b)    y.print();    z.print();</script></body></html>

In the code above, we perform basic addition and multiplication operations on two tensors, a and b, and print the results in the browser. Now, go to the terminal, open the project folder, and start the Python server with the following command:

python3 -m http.server

Then open the following address in your browser:

http://localhost:8000/

When you see a page displaying “Tensorflow.js Core API”, use Ctrl+Shift+I to open the console. This should work in both Chrome and Firefox. We should see the output of the operations above in the console:

Build Machine Learning Models in the Browser with TensorFlow.js and Python

If you want to read more about the Core API, I recommend reading the official documentation of the Core API.

Core API Documentation:

https://www.tensorflow.org/js/guide/tensors_operations

2.2 Layer API: Building Models like Keras

The Layers API is very similar to Keras in Python. Like Keras, you can create models using sequential and functional methods.

Let’s take a closer look at the sequential method through an example. We will train a regression model on these data points:

Build Machine Learning Models in the Browser with TensorFlow.js and Python

Here, X and Y have a linear relationship—each Y corresponds to X + i (where i is 0, 1, 2, 3…n+1). Let’s train a simple regression model on this dataset. You can write the following code between the <script></script> tags in the index.html file:

const callbacks = {        onEpochEnd: async (epoch, logs) => {          console.log("epoch: " + epoch + JSON.stringify(logs))        }      };    // Generate some synthetic data for training.    const xs = tf.tensor2d([[1], [2], [3], [4]], [4, 1]);    const ys = tf.tensor2d([[1], [3], [5], [7]], [4, 1]);    // Build and compile model.    async function basicRegression(){      // Build a sequential model      const model = tf.sequential();      model.add(tf.layers.dense({units: 1, inputShape: [1]}));      model.add(tf.layers.dense({units: 1, inputShape: [1]}));      model.compile({optimizer: 'sgd', loss: 'meanSquaredError'});      // Train model with fit().      await model.fit(xs, ys, {epochs: 100, validationSplit: 0.1, callbacks: callbacks});      // Run inference with predict().      model.predict(tf.tensor2d([[5]], [1, 1])).print();  }  // Create a basic regression model  basicRegression();

Sharp-eyed readers must have noticed that the syntax above is very similar to the Keras syntax for building sequential models in Python. When we return to the browser console, we will get predictions.

Build Machine Learning Models in the Browser with TensorFlow.js and Python

Our simple regression model predicts 7.556, which is very close to the expected value of 8. This is a basic example, but we can clearly see how easy and useful it is to build machine learning models directly in the browser.

TensorFlow.js allows you to build machine learning and deep learning models in the browser. It also automatically leverages the power of GPUs if available during model training on your system.

Below are some examples of deep learning models trained on some standard datasets using TensorFlow.js:

Build Machine Learning Models in the Browser with TensorFlow.js and Python

You can browse these examples in the tfjs-examples repository.

tfjs-examples Repository:

https://github.com/tensorflow/tfjs-examples

3. Utilizing Google’s Pre-trained Model: PoseNet

TensorFlow.js provides a plethora of pre-trained models from Google for many useful tasks, such as object detection, speech recognition, image segmentation, etc. The advantage of pre-trained models is that we can use them without any significant dependencies or installations, and they are ready to use out of the box.

It is widely expected that Google will release more models in the coming months. You can check the available pre-trained models at the link below:

Related Links:

https://www.tensorflow.org/js/models

Build Machine Learning Models in the Browser with TensorFlow.js and Python

We will use PoseNet in this article. PoseNet is a vision model that estimates a person’s pose in an image or video by estimating the position of key joints in the body.

How Does PoseNet Work?

This is a fascinating concept. Pose estimation is a computer vision technique used to detect people in images and videos. For example, this can help us determine where someone’s elbow appears in an image.

Just to be clear—pose estimation is not about recognizing who is in an image. The algorithm simply estimates the position of key body joints.

The detected key points are set with “Part” and “ID” indices, and the confidence score ranges from 0.0 to 1.0 (1.0 being the highest).

Build Machine Learning Models in the Browser with TensorFlow.js and Python

Below is an example of the output types given by PoseNet:

Build Machine Learning Models in the Browser with TensorFlow.js and Python

Incredible, right?! We will use the ml5.js library to utilize PoseNet. ml5.js is a library built on TensorFlow.js and p5.js. p5.js is another library that makes it easier to access the webcam in the browser.

ml5.js is designed to make machine learning accessible to a broad audience of artists, creative coders, and students. The library is built on top of TensorFlow.js and provides access to machine learning algorithms and models in the browser through simple syntax.

For example, you can create an image classification model using MobileNet in just 5 lines of code with ml5.js, as shown below:

Build Machine Learning Models in the Browser with TensorFlow.js and Python

It is precisely the simplicity of ml5.js that makes it very suitable for quickly prototyping in the browser, which is why we are using it in this project.

Let’s return to PoseNet. Create a new file named index.html and add the following code:

<html lang="en"><head>  <meta charset="UTF-8">  <meta name="viewport" content="width=device-width, initial-scale=1.0">  <meta http-equiv="X-UA-Compatible" content="ie=edge">  <!-- load p5.js -->  <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.7.3/p5.min.js"></script>  <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.7.3/addons/p5.dom.min.js"></script>  <!-- load ml5.js -->  <script src="https://unpkg.com/[email protected]/dist/ml5.min.js" type="text/javascript"></script>  <!-- keep the video in center of browser -->  <style type="text/css">    body{      text-align: center;    }</style></head><body>  <h1>PoseNet demo with Ml5.js</h1>  <p id="status">Loading Model...</p>  <div id="videoContainer"></div>  <!-- load the posenet.js file -->  <script src="posenet.js"></script></body></html>

This will create a basic HTML webpage and load the necessary files:

  • ml5.js and p5.js are loaded through their official URLs.

  • posenet.js is the file where we will write the code to use PoseNet.

Now, we will write the JavaScript code to use PoseNet. Create a new file named posenet.js in the same folder as index.html. Here are the steps needed to accomplish this:

  1. Load the PoseNet model and capture video from the webcam

  2. Detect key points of body joints

  3. Display the detected body joints

  4. Draw the estimated body skeleton

Let’s start with the first step.

Step 1: Load the PoseNet Model and Capture Video from the Webcam

We will use ml5.js to load PoseNet. Meanwhile, p5.js allows us to capture video from the webcam with just a few lines of code:

let video;let poseNet;let poses = [];function setup() {  const canvas = createCanvas(640, 480);  canvas.parent('videoContainer');  // Video capture  video = createCapture(VIDEO);  video.size(width, height);  // Create a new poseNet method with a single detection  poseNet = ml5.poseNet(video, modelReady);  // This sets up an event that fills the global variable "poses"  // with an array every time new poses are detected  poseNet.on('pose', function(results) {    poses = results;  });    function modelReady(){  select('#status').html('model Loaded')}

The most important parts of the above code are:

  • createCapture(VIDEO): It is a p5.js function that creates a video element by capturing video through the webcam.

  • ml5.poseNet(video, modelRead): We use ml5.js to load the poseNet model. By passing in the video, we tell the model to process the video input.

  • PoseNet.on(): This function is executed whenever a new pose is detected.

  • modelReady(): When PoseNet is finished loading, we call this function to display the model status.

Step 2: Detect Key Points of Body Joints

The next step is to detect the pose. You may have noticed that in the previous step, we saved each detected pose into the poses variable by calling poseNet.on(). This function runs continuously in the background. Whenever a new pose is found, it gives the positions of the body joints in the following format:

Build Machine Learning Models in the Browser with TensorFlow.js and Python

  • ‘score’ refers to the confidence of the model

  • ‘part’ indicates the detected body joint/keypoint

  • ‘position’ contains the x and y positions of the detected part

We don’t need to write code for this part as it is automatically generated.

Step 3: Display the Detected Body Joints

We know the detected body joints and their x and y positions. Now we just need to draw them on the video to display the detected body joints. We have seen that PoseNet provides a list of detected body joints, along with the confidence score of each joint and its x and y position.

We will use a threshold of 20% (keypoint.score > 0.2) confidence score to draw a keypoint. Here’s the code to implement this:

// A function to draw ellipses over the detected keypointsfunction drawKeypoints()  {  // Loop through all the poses detected  for (let i = 0; i < poses.length; i++) {    // For each pose detected, loop through all the keypoints    let pose = poses[i].pose;    for (let j = 0; j < pose.keypoints.length; j++) {      // A keypoint is an object describing a body part (like rightArm or leftShoulder)      let keypoint = pose.keypoints[j];      // Only draw an ellipse is the pose probability is bigger than 0.2      if (keypoint.score > 0.2) {        fill(255, 0, 0);        noStroke();        ellipse(keypoint.position.x, keypoint.position.y, 10, 10);      }    }  }}

Step 4: Draw the Estimated Body Skeleton

In addition to key points or body joints, PoseNet can also detect the estimated body skeleton. We can use the poses variable to draw the skeleton:

// A function to draw the skeletonsfunction drawSkeleton() {  // Loop through all the skeletons detected  for (let i = 0; i < poses.length; i++) {    let skeleton = poses[i].skeleton;    // For every skeleton, loop through all body connections    for (let j = 0; j < skeleton.length; j++) {      let partA = skeleton[j][0];      let partB = skeleton[j][1];      stroke(255, 0, 0);      line(partA.position.x, partA.position.y, partB.position.x, partB.position.y);    }  }}

Here, we loop through the detected skeletons and create lines connecting the key points. The code is still quite simple.

Now, the last step is to repeatedly call the drawSkeleton() and drawKeypoints() functions, along with the video source we captured from the webcam. We can achieve this using the draw() function of p5.js, which is called directly after setup() and executes repeatedly:

function draw() {  image(video, 0, 0, width, height);  // We can call both functions to draw all keypoints and the skeletons  drawKeypoints();  drawSkeleton();}

Next, go to the terminal window, navigate to the project folder, and start the Python server:

python3 -m http.server

Then go to your browser and open the following address:

http://localhost:8000/

Build Machine Learning Models in the Browser with TensorFlow.js and Python

Look! Your PoseNet should be detecting your body posture well (if you have followed all the steps correctly). Here’s what my model looks like:

Build Machine Learning Models in the Browser with TensorFlow.js and Python

Conclusion

You can see why I love TensorFlow.js. It is extremely efficient and doesn’t even require you to worry about complicated installation steps when building models.

TensorFlow.js demonstrates many prospects of making machine learning more accessible by bringing it into the browser. At the same time, it has advantages like data privacy and interactivity. This combination makes it a very powerful tool in the data scientist’s toolbox, especially if you want to deploy your machine learning applications.

In the next article, we will explore how to apply transfer learning in the browser and use TensorFlow.js to deploy machine learning or deep learning models.

The project we did with PoseNet can be taken further by training another classifier to build a posture recognition application. I encourage you to give it a try!

Original Title:

Build a Machine Learning Model in your Browser using TensorFlow.js and Python

Original Link:

https://www.analyticsvidhya.com/blog/2019/06/build-machine-learning-model-in-your-browser-tensorflow-js-deeplearn-js/

Translator Profile

Build Machine Learning Models in the Browser with TensorFlow.js and Python

Translation Team Recruitment Information

Job Description: A meticulous heart is needed to translate selected foreign articles into fluent Chinese. If you are a data science/statistics/computer major studying abroad, or working in related fields overseas, or confident in your foreign language skills, you are welcome to join the translation team.

What You Gain: Regular translation training to improve volunteers’ translation skills, increase awareness of cutting-edge data science, and overseas friends can stay in touch with domestic technological application development. The Data Party THU’s academic-industry-research background provides good development opportunities for volunteers.

Other Benefits: You will have the chance to work with data scientists from renowned companies and students from prestigious universities such as Peking University, Tsinghua University, and other overseas institutions who will become your partners in the translation team.

Click on the end of the article “Read the Original” to join the Data Party team~

Build Machine Learning Models in the Browser with TensorFlow.js and PythonBuild Machine Learning Models in the Browser with TensorFlow.js and Python

Click “Read the Original” to embrace the organization

Leave a Comment