Shivangi Das: Machine Learning

What?

We are going to build an image classifier with MobileNet in TensorFlow js from scratch (installing python included)

Why?

I wanted to learn how to use a pre-trained model in javascript.

How?

Find complete code here and demo

Installing Python and TensorFlow

Follow this https://www.tensorflow.org/install/install_windows for windows installation.

Download python 3.5 or 3.6. Version 3.7 will NOT work out as of July 2018. It doesn't download tensorflow js (henceforth referred to as tfjs)
Run pip3 install --upgrade tensorflow (for CPU only)
Or pip3 install --upgrade tensorflow-gpu (for GPU)

Convert MobileNet model to tfjs model

Get the MobileNet model from here.

This model is in tensorflow and needs to be converted to a tfjs model. Refer https://github.com/tensorflow/tfjs-converter

tensorflowjs_converter \
    --input_format=tf_frozen_model \
    --output_node_names='MobilenetV1/Predictions/Reshape_1' \
    --saved_model_tags=serve \
    F:/image/image-
    classifier/mobilenet_v1_1.0_224/mobilenet_v1_1.0_224_frozen.pb \
    F:/image/image-classifier/modelv1

Load the frozen model

I use GitHub as a fast and dirty way to host static files. Upload your model to a GitHub repository and use the raw path in your code.

const MODEL_URL =
  "https://raw.githubusercontent.com/shivangidas/image-
  classifier/master/modelv1/tensorflowjs_model.pb";
const WEIGHTS_URL =
  "https://raw.githubusercontent.com/shivangidas/image-
  classifier/master/modelv1/weights_manifest.json";
let model;
(async () => {
  model = await tf.loadFrozenModel(MODEL_URL, WEIGHTS_URL);
})();

Get ImageNet classes

ImageNet Dataset has 1000 classes. You can read more here.

let IMAGENET_CLASSES = [];
$.getJSON(
    "https://raw.githubusercontent.com/shivangidas/image-
     classifier/master/mobilenet/imagenet_classes.json",
    function(data) {
      $.each(data, function(key, val) {
        IMAGENET_CLASSES.push(val);
      });
    }
  );

Make Predictions

We get the image from the image tag. Refer full code for image selection part.
```
let imageData = document.getElementById("imageSrc");
```
Make tensor from the image using tfjs inbuilt function fromPixels()
Resize the tensor to 224x224 and make the datatype float.
The MobileNet version we are using (MobileNet_v1_1.0_224) takes an input of size [batchSize,224,224, 3]

let pixels = tf.fromPixels(imageData)
               .resizeNearestNeighbor([224, 224])
               .toFloat();

Normalize tensors to values between -1 and 1. Add a dimension as we have just one image to predict, no batch.
```
let offset = tf.scalar(128);
```
```
pixels = pixels.sub(offset).div(offset).expandDims();
```

Call predict on the tensor

const output = await model.predict(pixels);

We get probabilities of 1000 classes.
Sort and map to the classes we imported at the beginning.
Here we are showing the top ten classes.

const predictions = Array.from(output.dataSync())
                       .map(function(p, i) {
                          return {
                            probabilty: p,
                            classname: IMAGENET_CLASSES[i]
                          };
                       })
                       .sort((a, b) => b.probabilty - a.probabilty)
                       .slice(0, 10);
console.log(predictions);

Putting it together:

let offset = tf.scalar(128);
let imageData = document.getElementById("imageSrc");
let pixels = tf.fromPixels(imageData)
               .resizeNearestNeighbor([224, 224])
               .toFloat();

pixels = pixels.sub(offset).div(offset).expandDims();
      
const output = await model.predict(pixels);
const predictions = Array.from(output.dataSync())
                         .map(function(p, i) {
                          return {
                            probabilty: p,
                            classname: IMAGENET_CLASSES[i]
                          };
                       })
                       .sort((a, b) => b.probabilty - a.probabilty)
                       .slice(0, 10);

console.log(predictions);

That's it!

Well not really. We are making 7 extra tensors that we need to dispose to prevent a memory leak. That means the tensors created will persist in GPU and take up space. I went to great lengths to dispose of them individually but if you come up with a better solution (maybe using tidy), do let me know.

References

All the links above

Disclaimer: I am not qualified to teach you. But while you're here, let's learn something together.

What is XOR?

XOR or exclusive-OR is a logical operation that returns true if only one of the inputs is true. (only considering for two input combination)
https://en.wikipedia.org/wiki/Exclusive_or

Behold my bad handwriting : )

It's difficult to teach a computer XOR because the output cannot be separated by a straight line. However, now that we have the power of Machine Learning, we are going to do exactly that.

Visualize the Neural Network using TensorFlow Playground

I really enjoy trying out different Neural Networks(NN) on https://playground.tensorflow.org
Here is the NN I use for XOR playground.tensorflow.org

Bring out the power of Tensorflow.js

(Find the full code on github )

Here we are building a 2 layer NN, 1 hidden layer with 4 units and an output layer with 1 unit (the output should be between 0 and 1)

Get tensorflow.js

<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@0.11.6">

Input :

const train_xs = tf.tensor2d([

 [0, 0],

 [0, 1],

 [1, 0],

 [1, 1]

]);

Output:

const train_ys = tf.tensor2d([0, 1, 1, 0], [4, 1]);

These will stay the same :)

We'll use a sequential model - that means the output of one layer is fed into the input for the next layer.

const model = tf.sequential();

Add a fully connected hidden layer. It is mandatory to give the input shape. Our inputs will be two numbers between 0 and 1. (Fully connected- each input unit is mapped to each output unit). I'll probably write about neural networks, activations, weights, and biases soon (or not), till then follow the references.

model.add(

tf.layers.dense({

inputShape: [2],

units: 4,

activation: "sigmoid",

useBias: true })

);

The activation function sigmoid was giving much better results than relu :D

Add a fully connected output layer

model.add(

tf.layers.dense({

units: 1,

activation: "sigmoid" })

);

Our model is ready! Time to compile.
Learning rate should be small, you can experiment with different values. 0.1 worked for me.

const LEARNING_RATE = 0.1;

const optimizer = tf.train.adam(LEARNING_RATE);

model.compile({

optimizer: optimizer,

loss: tf.losses.meanSquaredError,

metrics: ["accuracy"]

});

I was trying to make things work with sgd(stochastic gradient descent) optimizer, but adam worked so much better for XOR. It's a lot about hit and trial.

To fit the training data we train the model on our input multiple times. Ask the fit function to shuffle the input.

You'll notice the loss decreasing and the accuracy increasing after each round. That's it! You're done!

Make predictions

Input can be as simple as

var input = tf.tensor2d([[0, 1]]);

Use the predict function of model to make predictions

var output = model.predict(input);

console.log(output.dataSync());

I am using a canvas to depict my predictions for different inputs.
(I should write a blog on that alone because man, is that difficult!)
Find the full code on github

My canvas looked like this after 20 epochs

Inspiration

I was inspired to build a model for XOR on watching a coding challenge on 'The Coding Train' by Daniel Shiffman (https://twitter.com/shiffman)

While you're on twitter, here's my handle @iShivangiDas

Did not understand everything?

If you want to learn about TensorFlow APIs go to https://js.tensorflow.org/api/0.11.6/
Neural Networks can be learned from Coursera - https://www.coursera.org/learn/neural-networks-deep-learning (I loved these courses and would highly recommend to everyone)

Shivangi Das

Monday, July 23, 2018

Using MobileNet with TensorFlow.js