CNN Assignment - Milestone 2

Overview

Assignment

1. For this milestone, you will implement and test the backpropagtion pass.
2. Please create a document that is the lab manual. Please add your name to the top of it.
3. Continue working with the "LeNet5.java" file and place all the MNIST testing into a "MNISTCNN.java" You should be able to copy it from the FFNet MNIST problem.
4. For milestone 1, I asked you to have bias weights for the pooling layer. It turns out they do not affect the performance of the network. Please remove them.
5. The network gets good training performance if the weights are initialize to the range [-0.1..0.1[, i.e. through Math.random() * 0.2 - 0.1;
6. For your reference, you should have the following trainable parameters:
   • C1: 156
   • S2: 0
   • C3: 1516
   • S4: 0
   • C5/F5: 48120
   • F6: 10164
   • Output: 850
7. The learning rate should be 0.01
8. Implement the backprop functionality by determining the gradients as well as update the weights. Use the information from the slides to implement this pass.
9. Here is an annotated sheet giving you a sense of the flow of gradients.
10. Testing:
    • Synthetic testing. Please use the following synthetic test code. Please paste the results into the lab manual.
    • Inspection of processing of the different levels of the CNN forward pass for the first digit of the training set, after you have successfullly trained the network to 97% precision on the testing set. Paste those images into your lab manual. If you cannot get to 97% precision then use the best run you were able to get.

      The imaging code requires that you add the following getters to your network class:

      
      // To be added
      public double[][] getInputLayer(){ return inputLayer; }
      
      public double[][][] getConv1Output(){ return conv1Output; }
      
      public double[][][] getPool1Output(){ return pool1Output; }
      
      public double[][][] getConv2Output(){ return conv2Output; }
      
      public double[][][] getPool2Output(){ return pool2Output; }
      
      public double[] getFc1Output(){ return fc1Output; }
      
      public double[] getFc2Output(){ return fc2Output; }
      
      public double[] getOutputLayer(){ return outputLayer; }
      
      

      Here is the code for displaying the activations of the LeNet5: NetworkVisualizer.java
    • Here is some code to compute the loss:

      public double computeLoss(int label) {
        // Cross-entropy loss
        return -Math.log(outputLayer[label] + 1e-10);
      }
      

    • Testing on MNIST: How many epochs does your network need to be trained so as to get 97% precision on the testing set. Hint: it is ridiculously low.
    • Here is some code for testing, to insure we are all on the same page:
    • To determine the networks best guess as to the digit presented, please use this code:

      public int getPrediction() {
        int maxIndex = 0;
        double maxValue = outputLayer[0];
        for (int i = 1; i < OUTPUT_NUM_NEURONS; i++) {
          if (outputLayer[i] > maxValue) {
            maxValue = outputLayer[i];
            maxIndex = i;
          }
        }
        return maxIndex;
      }
      

    • Here is some code for loading and then evaluating on MNIST. Please paste the output of the code into the lab manual.
11. Submission: Please submit a zipped copy of your lab manual and the .java code for this assignment.