Question

JAVA Program: Train a 3 layer neural network to recognize MNIST handwritten digit set. The network must use back propagation and Stochastic Gradient algorithm. NO third party libraries allowed(from scratch). Assume the mnist data is in two csv files: mnist_train.csv and mnist_test.csv. After each training epoch, The program should (1) print out statistics for each of the ten digits, the number of recognized inputs over total number of occurences of that digit (2) the overall accuracy considering all ten digits.

Answer 1

The MNIST database is a large database of handwritten digits. It is widely used for training and testing machine learning algorithms.
The database consists of 60,000 training images. Each image has a size of 28 x 28 pixels and depicts a digit between 0 and 9. The collection also provides 10,000 images in the same format for testing the network. This is a typical pattern in machine learning, test and validate the network with a different set of data to prevent overfitting.

The main programming language of machine learning is Python, and so all the examples in Tariq's book are written in this language. But the examples in the book don't depend on a machine learning framework, and so it is quite easy to implement them in another language. For playing with the algorithms, I wrote a Java.There are many matrix calculations involved, so I had to add external libraries to the projects because neither Java nor JavaScript has built-in support for matrices.

The network consists of three layers: input, hidden, and output layer. The input layer contains 784 nodes. One node represents one pixel of the scanned image
(28 x 28 = 784)

The hidden layer contains 200 nodes, and the output layer 10 nodes. The purpose of this neural network is to detect the digit that the scanned image contains; therefore it uses 10 output nodes. For instance if we feed the image of the digit 1 to the network we want as output [0, 1, 0, 0, 0, 0, 0, 0, 0, 0].

Each node between the input and hidden and between the hidden and output layer are connected. To each connection, a weight is assigned. The weight is what changes over time during the learning process.

During the learning process, each image runs through the following steps.

Forward propagation:-

Each pixel of the image is assigned to an input node.For each connection between the input and hidden layer, multiply the value of the input node with the weight and apply an activation function (this example utilizes sigmoid). This value is the result of the hidden layer. Do the same for each connection between the hidden and output layer. Multiply the value of the hidden node with the weight and apply the activation function.

Backpropagation:-

Compare the values in the output layer with the target values. For instance if we feed the scanned image of a 7 we want [0, 0, 0, 0, 0, 0, 0, 1, 0, 0] as the output.To "learn", the network calculates how "wrong" the calculated values are and then adjusts the weight accordingly. The weight adjustment occurs on all layers,output->hidden and hidden-> input. After step 5, the application feeds the next image into the network and starts over with step 1. It does that for each of the 60,000 images, and it runs the whole process 10 times (epochs).

This is the way how a neural network learns; somebody feeds data into it, compares the output results with the expected results, and tries to minimize the difference by adjusting the weights between the nodes.
The Java application reads the MNIST dataset and trains the network.

The train files contain the training dataset (60,000 images), and the t10k file the 10,000 images of the test dataset.

The images files contain the pixel data of the images, and the label files contain the label for each image. A label is a number between 0 and 9 in string form and corresponds to the scanned image in the images file.

As mentioned before, a lot of the calculations can be quite efficiently handled with matrix calculations. Java does not have built-in support for matrices; therefore, I added the Apache commons-math library to the project.

<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-math3</artifactId>
<version>3.6.1</version>
</dependency>

pom.xml

The application first creates the matrices for the connection between input and hidden and hidden and output layer and then assigns a random weight to each of the connections.

wInputHidden = createRealMatrix(hnodes, inodes);
wHiddenOutput = createRealMatrix(onodes, hnodes);
wInputHidden = initRandom(wInputHidden, Math.pow(inodes, -0.5));
wHiddenOutput = initRandom(wHiddenOutput, Math.pow(hnodes, -0.5));

Mnist.java

Next, the application reads the labels and image data from the MNIST files. For each image, it creates two additional images. One rotated 10 degrees clockwise and 10 degrees anticlockwise. This was one of the suggestions in Tariq's book to improve the accuracy of the model.

int[] labels = MnistReader.getLabels(Paths.get("./train-labels-idx1-ubyte.gz"));
List<int[][]> images = MnistReader
.getImages(Paths.get("./train-images-idx3-ubyte.gz"));

double[][] scaledImages = new double[images.size()][];
for (int i = 0; i < images.size(); i++) {
scaledImages[i] = scale(Util.flat(images.get(i)));
}

double[][] roated1ScaledImages = new double[images.size()][];
for (int i = 0; i < images.size(); i++) {
roated1ScaledImages[i] = scale(Util.flat(rotate(images.get(i), 10)));
}

double[][] roated2scaledImages = new double[images.size()][];
for (int i = 0; i < images.size(); i++) {
roated2scaledImages[i] = scale(Util.flat(rotate(images.get(i), -10)));
}
Mnist.java

And then, it feeds the three images into the neural network. The application multiplies the input values with the weights and applies the sigmoid function and does the same again from the hidden to the output layer.

private static void train(RealMatrix inputs, RealMatrix targets) {
// forward
RealMatrix hiddenInputs = wInputHidden.multiply(inputs);
RealMatrix hiddenOutputs = scalar(hiddenInputs, Util::sigmoid);

RealMatrix finalInputs = wHiddenOutput.multiply(hiddenOutputs);
RealMatrix finalOutputs = scalar(finalInputs, Util::sigmoid);

Mnist.java

During backpropagation, the application calculates the error between the calculated values and the desired target values and applies a correction value to the weights between output->hidden and hidden->input layer.

// back
RealMatrix outputErrors = targets.subtract(finalOutputs);
RealMatrix t1 = multiplyElements(outputErrors, finalOutputs);
RealMatrix t2 = multiplyElements(t1, scalar(finalOutputs, in -> 1.0 - in));
RealMatrix t3 = t2.multiply(hiddenOutputs.transpose());
wHiddenOutput = wHiddenOutput.add(scalar(t3, in -> learning_rate * in));

RealMatrix hiddenErrors = wHiddenOutput.transpose().multiply(outputErrors);
t1 = multiplyElements(hiddenErrors, hiddenOutputs);
t2 = multiplyElements(t1, scalar(hiddenOutputs, in -> 1.0 - in));
t3 = t2.multiply(inputs.transpose());
wInputHidden = wInputHidden.add(scalar(t3, in -> learning_rate * in));

Mnist.java

After running all scanned images through the learning process, the application writes the weights into two files

writeJson(wInputHidden, "./weights-input-hidden.json");
writeJson(wHiddenOutput, "./weights-hidden-output.json");
Mnist.java

This is what makes up the "knowledge" of our neural network, the weights between the different layers. We don't need to store the topology of the network, because in this trivial example it is hardcoded (784 input, 200 hidden and 10 output nodes)

The JavaScript application is going to read these two files and utilizes the pre-trained model for detecting user-generated input.

After the learning phase, the program runs the 10,000 test images through the network and calculates an accuracy. In my tests, I got an accuracy of about 97,6 %. Not bad for such a trivial neural network implementation.