Activity 13 – Color Image Segmentation

Image borrowed from Dr. Freemason
“Were you listening to me, Neo? Or were you looking at the woman in the red dress?”

One of my favorite scenes in the Matrix (besides the spoon bending, bullet evading, reality warping scenes) is the scene wherein Neo is being taught the basics of the Matrix. Neo and Morpheus were walking along a bustling street of seemingly hostile and preoccupied people clad in black. Morpheus was explaining something to Neo when a lady in a red dress walks by and catches Neo’s attention. And then we realize that the woman in red was just to test how cautious Neo, or even the viewer is with respect to his surroundings. Morpheus calls Neo’s attention and tells him to “look again,” only to find a gun point-blank towards his head.

Of course, this was just a simulation to test our hero, so Neo is still well and alive.

But you can’t blame Neo. In a sea of black coats, black ties and black socks (even black sunglasses) a beautiful woman wearing red is a very welcome distraction. If I were him I’d also be turning my head in admiration.

We human beings can easily distinguish prominent colors such as the red of the woman’s dress with the help of our eyes and brains. In a sea of black and white, it is easy to point out which is red. But how about our computers? How can we teach them to automatically isolate different colors?

Today we learn about color image segmentation using two different methods: parametric image segmentation and non-parametric image segmentation.

But before we start isolating the colors, we first talk about brightness and chromaticity. A 3-dimensional object illuminated at a direction can have different colors at its different parts due to differing brightness along the object. That is why we just want to get the normalized chromaticity coordinates to help us in further segmentation. Getting the normalized chromaticity coordinates eliminates the effects of differing brightness, allowing us to analyze the object based on its color alone.

To get the normalized chromaticity coordinates, we first get the sum of the values of the three red, green and blue channels. Next is we divide each channel with the sum. Mathematically it is explained by the following equation:

where R, G, B are the red, green, and blue channels, I is the sum of these channels, and r, g, and b are the normalized chromaticities.

Notice that r+g+b = 1. We can then simply represent b as b=1-r-g to simplify our dimensions into 2.

Representing this in 2D space, we get

where the x-axis represents the normalized red chromaticity, while the y-axis represents the normalized green chromaticity. Notice that when we get equal parts of red, green and blue (values of 33, 33, and 33 each), we get the color white.

Now that we are ready to segment our images using the two different methods.

To segment based on color, from the image we first get a region of interest (ROI) containing the color that we want to segment. This ROI we convert to normalized chromaticity coordinates for processing. Now we can choose what kind of segmentation we want.

Parametric
Parametric segmentation is basically assuming that the color of interest we want is a Gaussian probability along the r and g axes of the normalized chromaticity coordinates. Thus, what we do is get the mean and the standard deviations along the r and g coordinates to get two different Gaussians along each axes. These Gaussians we multiply to get a probability distribution along the normalized chromaticity coordinates to get a Gaussian function containing similar colors.

In layman’s terms, what this does is it projects a Gaussian probability distribution in the normalized chromaticity coordinate plane, with mean and standard deviations coming from the ROI. This denotes the colors similar to our ROI.

This method effectively eliminates pixels of color not in the Gaussian probability distribution from the original image.

Non-parametric
Non-parametric segmentation on the other hand, utilizes the ROI’s histogram along the normalized chromaticity coordinates. It takes values of the colors from the ROI and stores it in a histogram. This method stores the color information specific to the ROI. This means that it does not project different points in the normalized chromaticity coordinates. The advantage of this method is that it does not encapsulate colors that are not present in the ROI.

The obtained histogram is then projected back to the image. Each pixel is scanned and if the r,g, and b values of this pixel are within the histogram, the value of this coordinate is projected on to the image.

Both image segmentation methods are demonstrated in the image below.

From the image, we can see that the parametric method segmented the whole iPod very cleanly, selecting the whole iPod on the process. On the other hand the non-parametric method segmented only points near the region of interest, and looked dirty compared to the parametric segmentation.

People may say that they prefer the parametric method due to its cleanliness, segmenting the whole object of interest. But sometimes this is not the case, when in the image there are other, err, uninteresting objects of similar color of our region of interest.

Pictured above is the segmentation of the pink iPod along a laptop screen with a picture of an aurora. Since the aurora exhibits colors that are similar to the iPod, it too is segmented when we segment the image using the parametric method.

What if we want to segment the iPod only, not the screen?

With non-parametric segmentation, we can isolate the colors truly belonging to the iPod since we fill the histograms with the values found only in the ROI. We don’t overestimate some values, as the parametric segmentation sometimes does. Because we get only the colors inside the ROI, we can separate the iPod from the screen.

More interesting is that different ROIs yield different segmentations. At first glance of the unprocessed image the iPod’s color seems to be similar all throughout. Checking the histograms and the backprojections of different ROIs says otherwise.

So when do we choose one over the other?
It depends on the application on which type of segmentation we would use. If we want to get the whole object, say, for applications of area counting, we can use the parametric segmentation since we can isolate the whole object using only a small patch. Though this process is in danger of segmenting from other parts of the image, you can still get the whole object through segmentation.

On the other hand if you strictly want just the color in or around the ROI, we can use the non-parametric image segmentation. As we see above, the pink iPod was separated from the pink aurora. Also, another application of this method is you can use multiple patches and store them in the same histogram to isolate objects of different colors at the same time (say you want to isolate a yellow ball at the same time with an orange ball).

Conclusion
We have demonstrated image segmentation through two different methods: parametric and non-parametric segmentation. Advantages and disadvantages of each method were discussed. Applications for each method were also presented. For this activity, I give myself a grade of 11/10 12/10. for thoroughly demonstrating knowledge about the subject and for giving the required output. Plus 1 for discussing and demonstrating when one method is more advantageous compared to the other. Another plus 1 for demonstrating both masking and NCC in the comment below.

Acknowledgments

• Miss Loren Tusara, for teaching me how to segment images 2 years ago. She greatly helped me in determining how to code the histogram backprojection method which I use for research up to this day.
• Ma’am Jing for teaching me how to vectorize the backprojection code in an RM late last year.

References
M. Soriano, “Activity 13 – Color Image Segmentation”