Implementing, Tuning and Data Curation of Feedforward Networks

Overview

The purpose of this assignment is to learn the underpinnings of feedforward networks. You will implement a perceptron network and a single hidden layer feedforward network with a step-activation function. You will experiment with key hyper parameters as well as key activation functions so as to understand how they interact and how they influence the performance of the networks. You will additionally learn about the task of curating data so that it may be used to successfully train neural networks. You will be working with code that implements perceptrons, feed-forward networks and the backpropagation algorithm.

Basics

  1. Review the materials from the days 4 and 5.
  2. Throughout this assignment, complete the lab manual
  3. Here is a link to a zipped copy of all software and the lab manual for this assignment.
  4. For this portion, work with Perceptron.java. It is a perceptron network with a single output node. It uses a step activation function. Currently the training data is set up to learn the Boolean and function.
  5. Implement the trainNetwork() method with a fixed number of ten training episodes.
  6. Complete the experiments for questions 1-5 in the lab manual and enter your results.

XOR Experiments

  1. For this portion, work with XOR.java. It is a feedforward network with one output node. It uses a sigmoid activation function. The weights are initialized to random values in the range [0..1[.
  2. Study the code and conduct experiments 6 and 7 from the lab manual.

Parity Bit

  1. Use the given ParityNet.java and TrainingParity.java files to engineer a network that calculates an even parity bit for 7 bits. Please notice that one can implement a 2-bit parity net with XOR. A three bit parity bit and any n-bit parity net can be implemented through a cascading net of XOR "circuits." Here is a paper that has some figures with architectures for parity bit NN architectures. Here is a figure for an XOR net that uses just 1 and -1 for weights.

Data Curation and Digit Recognition

  1. For this portion, work with FeedForwardNetwork.java and Training.java. Study the code. Notice that "Training.java" is set-up to train the feed-forward network to train XOR. It serves to test your installation. Please notice that there is some testing code in the "FeedForwardNetwork.java" file. Feel free to modify it to your needs. Modify the training file so that it successfully recognizes handwritten digits. Here is the MNIST training data An explanation of it can be found here.
  2. There are two portions to this assignment. At first, you need to curate the data from the MNIST data set so that it can be used for training purposes. We recommend that you:
    1. Learn about the byte level layout of an idx-file.
    2. Study which Java classes and functions you will be using to read bytes from an idx-file.
    3. Format and arrange the bytes so that they can be used to train the given neural network.
    4. Here are some hints for curating the data.
  3. For the second part of this assignment, conduct and document the experiments of question 8.

Submission

Please submit a zipped copy of the following items to the appropriate drop-box on Moodle.
  1. Your report for the experiments your conducted.
  2. Your modified Perceptron.java, XOR.java, ParityNet.java, TrainingParity.java and Training.java files