Computational Photography, Spring 2016

Project #2: Gradient-Domain Fusion

Click here for source codes!

Date Submitted: 26 Feb 2016

445167 (carol (Xu) Han)
446216 (Ying Wang)

Project Description

This project explores gradient-domain processing, a simple technique with a broad set of applications including blending, tone-mapping, and non-photorealistic rendering. For the core project, we will focus on "Poisson blending"; tone-mapping and NPR can be investigated as bells and whistles. The primary goal of this assignment is to seamlessly blend an object or texture from a source image into a target image.
-- Courtesy of Course Website

In this assignment we use matlab to implement the required algorithm: Gradient-domain fusion.

Toy Problem

Concept

In this initial excersize, we tried to match the gradients, plus one pixel intensity, on a toy problem given in the assigmnent. Since we mininimized the energy between an image itself, this way the optimum solution is trivial; it is the image itself. We can check if the module of sparse matrix and the way to formulate the constraints are good.

No.ObjectivesDescription
1Minimize ((v(x+1,y)-v(x,y))–(s(x+1,y)-s(x,y)))^2 The x-gradients of v and s should be matched closely
2Minimize ((v(x,y+1)-v(x,y))–(s(x,y+1)-s(x,y)))^2 The y-gradients of v and s should be matched closely
3Minimize (v(1,1)–(s(1,1))^2 The top left corner of the two images should be the same value

Result

Here are the image we used as shown below, we did get back the origional image, and our sum of squared error between the images was very low. In the table, a map of pixel differences is shown. The error is negligibly small(mean:1.5517e-15,std:1.6817e-16).

input output time RSS error map of pixel differences
0.848341s 2.1329e-13

Poisson Blending

Concept

The key concept is to solve the following optimization:

which means let the gradient of final image be close to the gradient of source image with the constraint that the region of target image must be the same as the region of source image.

Inplementation

In this part, we need to blend part of an image(foreground image) into another image(background image). However the sharp seam associated with directly pasting images on top of one another corresponds to an “unnatural” gradient value. To prevent this, we need to figure out an algorithm which can “fit” the foreground part into the background image.

This is solved by discrete Poisson equations. The pixel colors we want to transplant into an image will be unknown initially, since they will be adjusted (e.g. for color constancy, edges) if we want the composite image to look right. We form a system of linear equations describing these unknowns, using our known data (unmasked colors that are unchanged in the target image) and solve in Matlab. One way to go is to maintain the original gradient of the background image along the border location ,and preserve all other gradients. The blending constraints are given in the Perez paper.

After inplementing the MATLAB code, we tested on several picture pairs and the results are given below. You can see that most of the output pictures is visually pretty “natual”.

Successful results

target source mask raw Poisson Run time
1.011s
0.991s
1.524s
1.392s
1.207s

Blending works best when the background of the source image matches the color and general texture of the area it is being pasted into in the background, as above five examples show.

Failed results

target source mask raw Poisson Run time
7.621s
8.209s

Here we got an example that is not so successful. When we put Ronald McDonald near KFC restaurant, it seems that he was “welcomed” by a boom to his head. Why is that? Commercial conflict?Not really. It’s because the background’s affects a lot to the foreground part and the color of Ronald McDonald’s upper body is influenced by the color of KFC’s brick wall and glass window.

Conclusions

From the results we can see, Poisson blending can change the color of our source image. The changing of color is the algorithm tends to blend the source image into target image to smooth the seam. If two images's color is different too much, the blending results could be bad because the change of color make our object unrealistic.

Ideally, the background of the object in the source region and the surrounding area of the target region will be of similar color. However, if the difference between them is too large, the result image exists weird color and blurred boundaries in the blending area.

Mixed Gradients

Concept

Here we impliment a similar method to the previous section, but instead use a mixed-gradient approach, which uses the gradient in either the source or target image as a guide depending on the magnitude of the gradients. The formula is:



Here, d_ij is the gradient from the source image if the magnitude of the gradient of the source image is larger than that of the target image's. If not, then d_ij is the gradient from the target image. Firstly, we do mixed gradients blending. Then, we do poisson blending in order to compare the output images of two blending methods.

Results

target source mask Poisson MixedGradients

Conclusions>

We note that the result of poisson blending is not satisfying with its noticeable backgroung color from the source image, while the mixed gradients blending provides a wonderful result of words blending into the wall but without any noticeable artificial effects. Although Poission Blending is much better than just copy and paste the pixels from one image directly into the other which create very noticeable seams, it lost all the pixel intensity of the background in the blending area.

Some problems with mixed gradients: Foreground object get transparent in the background. It happens in Mixed Gradient Blending. The reason might be since Mixed Gradient Blending consider both gradient from source and target image, it will prevent the "Object in the target image is replaced/hidden" issue but will change the foreground color and make it transparent.

Bell & Whistles(Extra Points)

Color2Gray

When converting an image to grayscale, we can lose a lot of information about the contrast in the image. In this part, we’re going to turn a color picture to gray without loss of contrast and gradient. First I converted the image from RGB to HSV, then looked at the gradient in each channel. Use Mixed Gradient Blending method at each pixel to select the one with larger magnitude, in the mean time keep the grayscale intensity the same. Here you can see some examples.

input RGB HSV

Laplacian pyramid blending

Implementation Steps:
1) Build Laplacian pyramids for each image.
2) Build a Gaussian pyramid of region mask.
3) Blend each level of pyramid using region mask from the same level.
4) Collapse the pyramid to get the final blended image.

Here are some examples from both the slide and flickr website (using keywords "same place different time") implemented by ourselves, using pyramid level N=5:

source1 source2 raw blend laplacian pyramids poisson blending mixed gradients

From the results we can't tell which method performs the best. There is no such an ideal method that can handle all kind of image blending and do the best job in every detail at the same time.

Laplatian blending preserves the best color feature among the three methods. However, people can clearly notice a huge overlapping region with stong low frequency mixing, which might be a bad thing if we really want to have a seamless blending. However, it just smooth the edge, didn't remove the seam at all.

Poisson blending and mixed gradients blending do seamlessly blend two images, but the color of the masked image is totally different from the origin because of obvious mismatch between this two images. Moreover, because there contains both strong gradients in both source and background images, such as trees, the mixed gradient result shows double image effect that the tree trunk in the winter scene also appears in the masked autumn scene.

In a word, one should know the pros and cons for each blending method, and choose the best one for each specific case.

Color transfer

We implement the color transfer algorithm described in this paper. Here are some steps and details.

1. Change the source and target images to L*a*b space. This space is similar to how people perceive color.
2. Find the means and standard deviations of each channel for both images.
3. For each channel in the target image, subtract the mean of the target image for that channel.
4. Scale each channel in the target image by (std deviation of target channel)/(std deviation of source channel)
5. For each channel in the target image, add the mean of the source image for that channel.
6. Combine the target channels together again.

target source Color Tansfer

Thoughts

We both have so much fun to learn about how to utilize the gradients in an image, and experiment and compare different blending techniques. However, the biggest takeaway for us is the use of sparse matrices in MATLAB. Following the sparse matrices construction method taught in class, our blending algorithms can be done in just several seconds.

Reference

  • [1] P. Perez, M. Gangnet, Poisson Image Editing, ACM SIGGAPH 2003.