Implementing, Tuning and Data Curation of Perceptrons and Feedforward Networks

Overview

Assignment

1. Review the materials from the days 4, 5, 7, and 8.
2. Throughout this assignment, complete the lab manual
3. Engineered Perceptron [5 pts] As a warm-up exercise, install the EngineeredPerceptron.java file and the Testing.java file. It is supposed to implement the Boolean AND function. Run it. It should fail to implement AND. Your job is to identify the lowest positive values of the weights, using one decimal and the lowest value of the threshold, again, using one decimal so that the network implements AND. Write down the three values in the lab-manual.
4. Learning Perceptron [20 pts] Continuing with Perceptrons, install Perceptron.java. Above, you engineered a Perceptron to recognize AND. You will now experiment with learning AND. Implement a sigmoid activation function as well as function that calculates the derivative of the sigmoid activation function. Then, complete the perceptron learning algorithm and a testing procedure.

   For initial testing, keep the initial weights at 1, set the learning rate to 0.7, run the training algorithm for one epoch and print the weights at the end of the run. The weights after running one epoch should be:

   Weights, left to right are:
   0.911496 0.911496
   

   Now, change the perceptron to have small random initial weights and experiment with different learning rates and number of epochs. Attempt to learn boolean AND. Print out the activations after the training run. You should consider any activation >= 0.75 as 1 and any value <= 0.25 as zero. You may also wish to look at the weights.

   Copy and paste the activations for what you consider your best run into the lab manual. Also enter the hyper parameters into the lab manual.

5. FFnet [50 pts] Implement a feed-forward network and test it. Name the file "FFnet.java" and feel free to copy and paste from the "Perceptron.java" file. Your network only needs to have a single hidden layer.

   For future use, ensure you have a constructor into which you can pass the data as well as the learning rate and the size of the hidden layer. The sizes of the input and output layers as well as the training set size should be inferred from the training data passed to it. Implement a training procedure which takes the number of epochs as a parameter. It is also helpful to have a procedure that prints the weights.

   Test your network in the "Testing.java" file. For testing purposes, initialize the weights to 1. Do this in the FFnet.java file. Use the data from the XOR FFnet worksheet to verify your code works correctly.

   Once your code works correctly, change it so as to initialize the weights to small random values.

   Now, attempt to learn AND. Again, consider any activation >= 0.75 as 1 and any value <= 0.25 as zero. Using this criteria, where you able to learn AND? Are there any criteria for which it can be said that the network learned AND? Answer these questions in the lab manual.

   Copy and paste the activations for what you consider your best runs for the AND the experiments into the lab manual. Also enter the hyper parameters into the lab manual.

   Now, attempt to learn XOR. Again, consider any activation >= 0.75 as 1 and any value <= 0.25 as zero. Using this criteria, where you able to learn XOR? Are there any criteria for which it can be said that the network learned XOR? Answer these questions in the lab manual.

   Copy and paste the activations for what you consider your best runs for each of the experiments into the lab manual. Also enter the hyper parameters into the lab manual.

6. Data Curation Light [20 pts] Modify the "Testing.java" file so that it reads in all of the training and testing data for MNIST. Here is the MNIST training data An explanation of it can be found here. Here are some hints for curating the data. Here is a set procedures that visualizes the data after you read it in properly. Here is the PixelGrid.java file that goes with it. Paste the visualization of the first three images from the training set in the lab manual.

7. Training an FFnet on MNSIT [35 points] Modify your "Testing.java" file to train an FFnet on MNIST.

   Write a procedure that determines the number of errors. It needs to have access to the trained network. It will be passed either the training data (including the desired outputs) or the testing data (including the desired outputs). Since you are using a sigmoid activation function, you need to be slightly creative in determining when your network determined the correct answer. Please consider any output activation greater or equal to .75 as a postitive output and any output activation less than or equal to .25 as negative.

   Please maintain two counts: The number of items your network gets wrong in the 10 neuron output layer. Notice that your network may get all 10 items wrong. The second count only tallies whether you get the desired output wrong. Run these statistics both for the training set and the testing set.

   Experiment with the following values of the hyper-parameters.

   • Size of hidden layer: less than 50, 50-500, larger than 500.
   • Initial random weight are all positive, all negative and both, positive and negative.
   • Learning rates: 0.3, 0.5, 0.7.
   • Additional curation of the data. Right now, your data probably consists of a grey-scale image. Try removing some of the precision of the data.

   Based on your experiments, what are the values of the hyperparameters that ensure the fewest number of epochs to successfully train the network? By successful, we mean at most 10% wrong, but it should be less than 5%, ideally less than 3%.

   Answer the questions in the lab manual.

Submission

1. The lab manual.
2. Your Perceptron.java, FFnet.java and Testing.java files.