Project 4: Neural Radiance Fields

Overview

In this assignment, you’ll implement a (slightly simplified) version of the technique described in the influential 2020 paper NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis by Mildenhall et al. Given a collection of images taken from multiple perspectives, along with pose information for those cameras, NeRF is an elegant technique that encodes a 3D volumetric representation of the scene in a neural network.

Setup

Skeleton Code

Accept the Project 4 assignment on Github Classroom and clone your repository. This repository contains the assignment’s single Jupyter notebook, which contains detailed instructions as well as skeleton code and TODOs for you to complete.

Hardware

This project requires the use of a GPU to train NeRF models using the Pytorch framework. The recommended approach is to run on one of the departmental GPU servers, as you did on project 3.

Alternative approaches that should work but I cannot guarantee support for include:

• running on your own machine with an NVIDIA GPU
• running on Google Colab, which provides free GPU notebook hosting but has unspecified usage limits

Software

This project has similar software requirements to the prior projects, but also requires a few additional packages. The necessary Python requirements are given in requirements.txt, included in your repo. To set up an environment specifically for p4, go to your repo directory and run:

python3 -m venv p4env
source p4env/bin/activate
pip install -r requirements.txt

It may take a few minutes to install everything. If you want to augment your existing environment, just activate that environment and pip install -r requirements.txt there.

Data

The input and test data for this project will be downloaded (and cached) by the notebook as needed. I recommend running the notebook from the same place when possible (or moving all the downloads along with it) to avoid having to re-download files. Do not commit the data files to your Github repository.

Tasks - Overview

There are several pieces we ask you to implement in this assignment:

1. TODO 1: Fit a single 2D image using an MLP with and without positional encoding;
2. TODO 2: Compute the origin and direction of rays through all pixels of an image
3. TODO 3: Sample the 3D points on the camera rays
4. Implemented for you: Compute compositing weight of samples on camera ray
5. TODO 4: Render an image with NeRF

The instructions about individual TODOs are present in detail in the notebook.

Testing

To help you verify the correctness of your solutions, we provide tests at the end of each TODO block, as well as qualitative and quantitative evaluation at the end of the notebook.

Expected Outputs

2D Image Fitting

With no positional encoding:

With 3 frequencies:

With 6 frequencies:

NeRF Training:

The optimization for this scene takes around 1000 to 3000 iterations to converge on a single GPU (in 10 to 30 minutes). We use peak signal-to-noise ratio (PSNR) to measure structual similarity between the prediction and target image, and we expect a converged model to reach a PSNR score higher than 20, and produce a reasonable depth map. Here’s an output after 1000 iterations of training with default settings:

Here’s the resulting 360 video:

Submission

Execute your completed notebook in Jupyter and commit the resulting version of your .ipynb file to your github repository; make sure the cell outputs are contained in the notebook. Also commit the 360 video output (lego_spiral_001000_rgb.mp4) to your repository, and push to Github by the deadline.

Rubric

Points are awarded for correctness and efficiency, and deducted for issues with clarity or submission mechanics:

• 10 points for each correctly passed test case in the notebook (for TODOs 1-4)
• 5 points for your answer to TODO 1(c)
• 5 points for efficiency: training on a 360 scene with default parameters takes under 10 minutes on a GPU.

Clarity Deductions for poor coding style may be made. Please see the syllabus for general coding guidelines. Points may be deducted for any of the following:

• Methods should be written as concisely and clearly as possible
• Methods should not be too long - use helper methods to break code into sensible subroutines
• Code should not be cryptic and terse - explain nontrivial blocks with comments
• Methods you introduce should be accompanied by a precise specification
• Variable and function names should be informative but not overly verbose

Acknowledgements

This assignment is based on an assignment from Noah Snavely; thanks to Noah and numerous underappreciated TAs.