For historical reasons, this assignment was designed with a normal mode and a hard mode. You are only responsible for normal difficulty. Feel free to tackle hard mode if you’d like. Look for [HARD MODE] tags below. Otherwise, feel free to skip these sections.
You may work on this assignment solo or in groups of two. If you would like to work in a pair, you need to complete the following steps:
A couple other relevant policies:
In this project, you will implement a system to combine a series of horizontally overlapping photographs into a single panoramic image. We’ll use the built-in ORB feature detector and descriptor from the opencv library; on [HARD MODE] you will also implement an alternate homegrown feature matching pipeline. Given the feature correspondences, you will automatically align the photographs (determine their overlap and relative positions) using RANSAC to find an outlier-robust motion model and then blend the resulting images into a single seamless panorama.
You are provided with a GUI that lets you test and visualize the functionality and intermediate results of the various stages of the pipeline that ultimately produces the final panorama output. We have also provided you with some test images and unit tests to help you debug.
The high-level steps required to create a panorama are listed below. You will implement panorama stitching using translation and homography motion models. The [HARD MODE] tasks also would have you implement a translational model with spherically-warped input images, which allows for full 360-degree panoramas.
Take a sequence of photos with horizontal overlap
[HARD MODE] Warp each image to spherical coordinates
Extract features from each image
Match features among neighboring pairs of images
Align neighboring pairs using RANSAC
Write out list of transformations that relate each image to a single coordinate system
[HARD MODE] Correct for drift, if the panorama is 360 degrees
Warp the images into the output panorama and blend them together
Crop the panorama and admire the beautiful result
Skeleton code is provided in the repository created by Github Classroom. The invitation link is found in the Project 2 assignment on Moodle.
Test sets: See the resources subdirectory in your repo. You will find four datasets: yosemite, campus, melbourne, and melbourne_small.
Software environment: The software listed for Project 1 should be all you need for this project as well (please let me know if you find this not to be the case!). The GUI for this project is written using TK, much like the prior project. Let me know if you’re having trouble running the project.
Warp each image to spherical coordinates.
warp.pycomputeSphericalWarpMappings[TODO 1 - HARD MODE] Compute the inverse map to warp the image by filling in the skeleton code in the computeSphericalWarpMappings routine to:
Convert the given spherical image coordinates into the corresponding planar image coordinates
Apply radial distortion using the radial distortion model described in lecture
Align neighboring pairs.
alignment.pyalignPair, getInliers, computeHomography, leastSquaresFitThe computeHomography function takes two feature sets from image 1 and image 2 (f1 and f2) and a list of feature matches (containing pairs of indices into f1 and f2) and estimates a homography from image 1 to image 2.
[TODO 2] Set up the A matrix that defines to the system Ax that computes the residuals for a given homography unrolled into a vector h.
[TODO 3a] Implement minimizeAx to find the unit-length vector x that minimizes ||Ax|| for a given A.
[TODO 3b] Call minimizeAx on the matrix you set up in TODO 2 and use its result to fill in the 3x3 homography matrix H. Don’t forget to return the homography in its normalized form, with a 1 as the bottom right entry.
[TODO 4] alignPair is where you will implement RANSAC. It takes two feature sets, f1 and f2, the list of feature matches, and a motion model, m (described below) as parameters. For this project, we support two motion models, represented by the two possible values of the enum MotionModel: eTranslate and eHomography. alignPair estimates and returns the inter-image transform matrix M as follows:
getInliers to get the indices of inlier feature matches (i.e., indices into matches) that agree with the current motion estimate.After repeated trials, the entire inlier set from the M with the largest number of inliers is used to compute a final least squares estimate for the motion, which is returned as the matrix M.
[TODO 5] getInliers computes the indices of the matches that have a Euclidean distance below RANSACthresh given features f1 andf2 from image 1 and image 2 and an inter-image transformation matrix from image 1 to image 2.
[TODO 6, 7] leastSquaresFit computes a least squares estimate for the translation or homography using all of the matches previously estimated as inliers. It returns the resulting translation or homography output transform M. For translation estimation, I recommend simply averaging the translations rather than taking the heavy-handed linear algebra approach. For homographies, you’ve already implemented computeHomography to do the heavy lifting.
Warp and blend the aligned image pairs into a single output image to create the final panorama.
blend.pyimageBoundingBox, blendImages, accumulateBlend, normalizeBlend[TODO 8] imageBoundingBox: Given an image and a homography, figure out the box bounding the image after applying the homography.
[TODO 9] getAccSize: Given the warped images and their relative displacements, figure out how large the final stitched image needs to be in order to fit all the warped image. This method also augments each per-image transformation with a translation that moves the output image coordinate system into a numpy-array-friendly world where (0, 0) is at the top left.
[TODO 10] blendImages: Warp each image into the output image’s coordinate system and add its pixel content into the accumulator. You will need to use inverse warping to calculate values at integer output pixel coordinates. To allow the images to blend smoothly, use the fourth channel to represent the weight of the contribution of a pixel. Using the linear blending scheme described in lecture, the weight varies linearly from 0 to 1 from the left side of the image over a distance of blendWidth pixels, then ramps down correspondingly on the right side of the image. Other, fancier blending schemes are possible - you may experiment with some for extra credit.
TODO 10 implementation notes:
When working with homogeneous coordinates, don’t forget to normalize when converting them back to Cartesian coordinates.
When doing inverse warping, use bilinear interpolation for the source image pixels. First try to work out the code by looping over each pixel. Later you can optimize your code using array instructions and numpy tricks. My approach does vectorized bilinear interpolation using array operations; another approach uses cv2.remap to warp the image. In either case, you may find numpy.meshgrid useful. Optimizing this function is worth only a couple points, so prioritize this lowest.
[TODO 11] normalizeBlend: Having accumulated weighted pixels from all the source images, this function normalizes the image so each pixel has unit weight by dividing by the weight at each pixel. Be careful not to divide by zero. Remember to make sure the alpha (fourth) channel of the resulting panorama is opaque (1)!
[TODO 12 - HARD MODE] blendImages: To make a 360 panorama, you need to do a couple extra things. First, you’ll want to include the first image again at the end so you can put the seam in the middle of that image. Second, you’ll need to correct for vertical drift to make the left and right edges line up perfectly. The getDriftParams function computes the position of the top left and top right corners of the un-corrected panorama, accounting for cutting out the left half of the left image and the right half of the right image. Given these two points, build a shearing transformation that maps these top two corners to the same y value.
The base project uses built-in ORB feature detection and description functionality from OpenCV. The GUI (gui.py) accepts a --MOPS flag; if this is set, the program should use your own custom-written feature matching pipeline. Implement functionality to detect, desribe, and match features using Harris, MOPS, and SSD+ratio (methods for this likely fit best in alignment.py, but I haven’t given you any skeleton for this). Your pipeline should follow the code we wrote in class, but should be generalized to multiple scales by running on a Gaussian pyramid. Feel free to use OpenCV’s pyrDown or import relevant code from Project 1.
[TODO 13 - HARD MODE] Make appropriate calls to your own feature matching functionality in gui.py in the computeMapping function to replace ORB if the--MOPS flag is set.
The skeleton code that we provide comes with a graphical interface, with the module gui.py, which makes it easy for you to do the following:
You can use the GUI visualizations to check whether your program is running correctly.
Testing the warping routines:
In the campus test set, the camera parameters used for these examples are:
f = 595 k1 = -0.15 k2 = 0.00
In the yosemite test set, a few example warped images are provided for testing purposes. The camera parameters used for these examples are:
f = 678 k1 = -0.21 k2 = 0.26
See if your program produces the same output. Note that if you use Yosemite with the translation motion model, you might get slightly blurry panoramas in the blending region (as you can also see from the example results). This is because the translation model isn’t flexible enough to describe the true transformation.
Testing the alignment routines:
Note that the campus images are only suitable for the translational motion model! The yosemite images are suitable for both motion models. To test alignPair, load two images in the alignment tab of the GUI. Clicking ‘Align Images’, displays a pair, the left and right images, with the right image transformed according to the inter-image transformation matrix and overlaid over the left image. This enables visually analyzing the accuracy of the transformation matrix. Note that blending is not performed at this stage.
Testing the blending routines:
When debugging your blending routines, you may find it helpful for the sake of efficiency to use the melbourne_small dataset, which is simply a downsampled version of the Melbourne dataset. Example panoramas are included in the yosemite and the campus directories. Compare the resulting panorama with these images. Note that it’s important to use the specified f, k1, k2 parameters to get the same image. If you’re doing [HARD MODE], you should use the 360 degree checkbox to ensure you get the same result for campus dataset.
Each partner must submit their own artifact via Moodle: Take a series of images with a digital camera mounted on a tripod or a handheld camera, and stitch a panorama using your code. This panorama can be with any of the ways that you implemented. For best results, overlap each image by 50% with the previous one, and keep the camera level. In order to use your camera for a spherically warped translation-aligned panorama, you have to estimate the focal length. The simplest way to do this is through the EXIF tags of the images, as described here. You may also be able to find the focal length (in mm) and sensor width by searching for your camera or phone model. Alternatively, you can use a camera calibration toolkit to get more precise focal length and radial distortion coefficients.
For inspiration, check out some of the following links:
Code Submit your code by committing and pushing your changes to Github before the deadline.
Artifact Every student must submit their own artifact. If you are working in a pair, each group member must submit an artifact. Submit your panorama artifact to Moodle in JPG format.
Your project will be graded based on the quality of the panoramas generated. An approximate point breakdown is given below. Keep in mind that later code depends on earlier code, so partial credit may be hard to assign if something early on is broken. If you’re short on time, optimize for having working code for image alignment with homographies.
Correctness:
Efficiency:
Artifact:
Clarity: Deductions for poor coding style may be made. Please see the syllabus for general coding guidelines. Up to two points may be deducted for each of the following:
Many thanks are due to those who developed and refined prior versions of this assignment, including Scott Wehrwein, Steve Seitz, Kavita Bala, Noah Snavely, and many underappreciated TAs.