Finding Color Objects in a Scene

Let’s put some of this into practice with a real world image. I am going to load an image of a scene that comprises some tomatoes on a bush in a lab, and I am going to convert the image so that the pixels are stored inside MATLAB as double precision numbers. So we can have a look at the image as advertised: it contains some tomatoes on a bush inside a lab — at MIT, actually. If I click around on the image we can see that the pixels in the tomato have got quite a large amount of red, 65 per cent red, not much green, not much blue.

Now the leaves, the stems for instance, contain more green than they do red or blue. The tomato images are interesting; the tomatoes are actually quite shiny and if I click around here we see that the value is actually closer to white than it is to red. This region here we refer to as a specular highlight. What has happened is that bright light within the room is reflecting off the surface of the tomato in a specular rather than a defuse fashion and in this case the light has not picked up the color of the fruit, it is basically what we are seeing here is the color of the light source itself.

We have talked a couple of times already about the process of gamma encoding the information within an image in a file. But what I would like to do is to undo that gamma encoding; I would like to apply a gamma correction or gamma decoding and I can do that with the iread function by passing in an extra argument and passing in the gamma value that I would like to have applied to every pixel. So this will eliminate the non-linearity between the luminance that the camera observed and the pixel values that I have in my MATLAB workspace.

If I display this gamma corrected image we will see that it looks rather unusual, it looks actually quite dark and as explained before what has happened here is that the gamma correction has now been applied twice. I have applied it once in software and the monitor that I am looking at the image on is also applying the gamma correction. So it doesn’t look right that the values that I have in my workspace are now linearly related to the light that went into the lens of the camera.

So the next thing that I would like to do is just to display one of the color planes in this image. So let’s display the red plane and here it is here, and we can see that the tomato fruits are looking quite bright here because they reflect a lot of red light, whereas in the green plane we see that the fruit looks very, very, dark.

Let’s separate out the color values. I am going to create a new workspace variable called R which contains pixels in every row and every column that belong to plane one. I am going to do a similar thing for the green plane and I am going to do a similar thing for the blue plane, which is plane number 3. So, for instance, I can just display again the green plane that I just computed. Now the next thing that I want to do is to compute the chromaticity coordinates and to do that I need to compute the sum of the red and the green and the blue and I am going to put that into the local variable called Y.

Y is a common symbol used for what is called the luminance image; it’s a monochrome image actually. If we look at this now it will look something like a monochrome or black and white picture of the scene. It is the average of all of the color channels.

Now the chromaticity coordinates computes the ratio between all of the pixels in the red plane and the sum and I can compute this element wise division by using the MATLAB operator dot slash, so what it is going to do is to compute a new image, call it little R, and its dimensions will be the same as the dimensions of capital R and capital Y. Each element of the image little R is the corresponding pixel in capital R divided by the corresponding pixel in capital Y.
And let me do the same thing for the green chromaticity channel; let’s have a look at these. These are stored in the MATLAB workspaces matrices, but of course I can display a matrix as an image and an image is represented as a matrix, so here is little R and I can create another figure, figure two, and in figure two I am going to display little G.

So these are the chromaticity values, and if we poke around some of the values here we can see that where the fruit is, it has got quite a large value of R, it has a very small value of little G. If we look at where some of the leaves are we see that they have got a modest value of little R—about 0.39—and little G has got a slightly bigger value, so these two images represent the color in this scene: we have factored out the intensity.

So let’s say now that we wanted to make a rule that would select a pixel that belonged to the fruit and didn’t belong to anything else.

Well clearly we want a rule that said something about the fact that R is quite big, it seems to be at least greater than 0.8 and mostly it is above 0.9, but let’s say, conservatively, it has to be above 0.8 and in this case we want the G value has to be quite small. Has to be sort of, you know, less than 0.1 even. But to be conservative, let’s say it has to be less than 0.2. So I could create a new variable, it is going to be a logical variable, so I am going to say anywhere that R is greater than 0.8 and G is less than 0.2 it will be true.

So let’s display this image I just created called Tom and there we have it. The background here, these values are all zero, they are all false and in here they all have values of one, which is MATLAB’s way of representing the concept of true. We have created now a binary image where the pixels are one if they are the tomato fruit and they are zero if it is not the fruit, background, people and leaves.

Now that we have a binary image we can do a lot of the things that we demonstrated earlier when we were talking about binary vision. So I could look for all of the blobs that are within this image so I apply the iblobs function to the binary image we just computed and here it is, that there are ten blobs in here. Now, there are clearly a couple of big blobs and we can see there, there are some blobs with quite a large area and there is also a lot of very, very small blobs around the edge of one of the fruit.

Now we can go back to the original image, let’s have a look at figure one and there I am going to display the original image and then I am going to overlay some bounding boxes and I will overlay them in the color green.

So there we have it; quite simply, we have taken a color image, we have computed the essential color at each of the pixels in terms of chromaticity coordinates. We have made a very simple rule to choose pixels that belong to the fruit and not to anything else; that led us to a binary image. Once we have a binary image, then we can apply all of the techniques that we have already learnt and in very quick fashion I can outline the particular fruit. Now the problem with this is that the fruit are kind of misshapen, part of that is because of the specular reflection that I mentioned earlier and also some of the fruit is obscured by leaves and whatever, still the bounding box is a good place to start, we could look at some of the mathematical morphology techniques that we used before, we could apply the techniques of dilation and erosion perhaps to fill in some of the missing part of those fruit.