How to find only leaves in a gray scale image acquired using IR camera for Thermal imaging

I get only a gray scale image like this from IR camera and I need to write a code that would detect ONLY LEAVES from each plant.

I have no idea what sort of algorithms should I go for in order to that because of complex background. Any suggestions on what I should look for would be greatly appreciated.

Answer 1

[Image Analyst]

You could use image analysis. If the leaves are a temperature different than the rest of the scene, simply threshold and call regionprops().

See my Image Segmentation Tutorial for a full demo. http://www.mathworks.com/matlabcentral/fileexchange/?term=authorid%3A31862

[Pavan Basavaraj]

I have edited the post with actual image I need to work on. As you can see the background is not homogenous. It's Complex background. Do you still think that object (leaves) detection can be done by just Thresholding.?

[Image Analyst]

No, especially with all the shadows and background clutter. Why are you using a thermal camera instead of a visible light camera? I'd think a color camera or a monochrome camera with the proper color filter in front of it would be better.

[Pavan Basavaraj]

The aim is to find the temperatures of leaves for further study thats why IR camera. And for this experiement we cannot use a color camera as its not feasible for the current conditions we are having at lab. I can try to change the background making it more homogeneous.

[Image Analyst]

Is it an IR camera or a thermal camera? Thermal cameras operate in the long/far IR region, not the near IR region. Many infrared cameras will not be able to report temperatures. It has to be a special thermal camera. It looks like the leaves are cooler, if that's a thermal camera, and it looks like the sheets reflect highly - perhaps being at a different temperature or perhaps not, just being more reflective or having a different emissivity. You might try a metalized mylar sheet behind the plants to maximize the reflectivity. Or try different materials since different materials will have different thermal emissivities. And it should be as flat as possible to minimize "shadows" due to varying shape of the background.

[Pavan Basavaraj]

Its an FLIR A65sc Thermal camera. Would it be easier if I'm able to find a solution to get both RGB and Thermal images combined using image registration without changing the background?

[Image Analyst]

Yes that would be easier because the green plants could easily be segmented out based on their green hue - much easier than from the thermal grayscale image.

[Pavan Basavaraj]

Thanks a lot. You have been a real help.

[Pavan Basavaraj]

Hello, As per your guide I did manage to fit a webcamera and perform "Image registration". Thermal image as "Fixed" and RGB Image as "Moving". And I would like to ask who to proceed further. So what I would like to do is

1. Find green pixels from the registered image which would lead to find leaves.

2. As you can see there are 4 plants. I would like to put a box for each plant saying I found all 4 plants.

3. Extract a average pixel value of 3 measuring points from each plant ( Pixel vaue of thermal image ).

4. And then I can calculate temperature using these pixel values.

Any input is really appreciated. <<

<<

>>

>>

[Image Analyst]

See my color segmentation demos in my File Exchange: http://www.mathworks.com/matlabcentral/fileexchange/?term=authorid%3A31862. I'd probably start with the HSV one and threshold both on hue and saturation to get the green things as a mask, "greenMask". Split it up into a separate mask for each plant if you want. Then to get the temperature of the leaves simply do

meanTemperature = mean(thermalImage(greenMask));

[Pavan Basavaraj]

What I did not get is whether to do this color segmentation on only RGB Image or Registered image. Or find green pixels in RGB image and then use those pixel position on Registered Image to find temperature. And then do the the conversion of those pixels to temperature?. Kindly need help.

[Image Analyst]

Once the images are registered, they are aligned and the same (row, column) location points to the very same point on the plant in both the RGB image and the thermal image, so first you register them - it doesn't matter which is fixed and which is moving. Then use the RGB image to find green pixels (a mask), then extract the temperatures from the thermal image.

greenPixelMask = DetermineMask(rgbImage); % You write this function.
% Find temperatures within the mask from the thermal image
temps = thermalImage(greenPixelMask); % List of pixel values from thermal image.
meanTemperature = mean(temps);

[Pavan Basavaraj]

But for image registration we have converted RGB image to Grayscale image. Do you mean I have to convert back that Grayimage to RGB image and then find the green mask? I also got one more requirement recently. If I have 4 plants i need three measurements from each plant i.e max value and its position, minimum pixel value and its position and mean value. As in the night time we cannot see the plants at all. So, I can use these position to simply collect temperature signal value without doing any detection. Your inputs are highly appreciated.

[Image Analyst]

Use imregtform() to get the transform fro aligning the gray scale images. Then you can apply that to the color image.

I wouldn't worry about night. These are plants and they don't change very fast. If you really want to monitor them like every few minutes, then put on a small night light in the room.

[Pavan Basavaraj]

Yes I already suggested that we can put LED light but we cant see the plants at night at all in thermal images to do image registration. I will show a pic how it looks. We also want to monitor in night so I thought using the position from morning I will just extract signal in night.

[Image Analyst]

You probably do not have a thermal camera is your image looks different in the light and dark. You probably just have a near infrared camera, not a thermal infrared camera. What is the camera model? Like a FLIR, model xxxx or something.

[Pavan Basavaraj]

our camera is FLIR A65sc. At night plants cool down and they gradually become like this as we are maintaining room temperature at constant 22 degrees.

[Walter Roberson]

How often (or how quickly) do the plants move? Is the movement predictable?

If the only time they move is when the researchers move them (and the cleaning people do not move them or dust them), then if you can restrict the movement to times at which there is light, then you can use the last known position (based upon color image) to extract the thermal image at night. Create an ROI, mask out the background.

Is your FLIR set for automatic gain controls, adjusting the temperature range according to the minimum and maximum in its field of view? If so then that could be a nuisance. If you do use an ROI then you would prefer a situation where any particular sensor reading represents an absolute temperature rather than according to the surroundings; in such a situation the background at night would be irrelevant.

[Pavan Basavaraj]

Plants wont be moved at all once we place them at a position. Camera temperature range is in between -20 to 500 degrees. I believe that at night there wont be temperature difference between plants and surroundings and that s why we are not able to see plants. Can you please elaborate how I can extract the last known position out of color image? I have done image registration and then found out plants by their Hue and I have masked them in one image but I would like to mask them individually. How can I accomplish that?

And then how do I proceed further with extracting last known position automatically?

Thanks a lot

[Image Analyst]

Maybe you could track the number of pixel in the mask and when it starts to dramatically change, consider that as when "night" starts and as far as masking goes, use the last mask before that point in time.

Do you have any theory as to why the plants' temperatures go down when there is no visible light in your CTCH room?

[Pavan Basavaraj]

As far as I know its because only in visible light photosynthesis happens but not in the night. That could be the reason.

As you told me to create separate mask for individual plant. Can we create this individual mask automatically and then refer that in registered thermal image to find temperature? Because requirement is to find temperature of plant individually. Is there anywhere I could look specifically. Kind regards, Pavan

[Image Analyst]

Yes, you could. You could convert from rgb to hsv with rgb2hsv() use saturation to threshold on and create a mask. Use bwareafilt() to keep blobs only bigger than a certain size. You can then call regionprops to get the intensity of each plant

hsvImage = rgb2hsv(rgbImage);
sImage = hsvImage(:,:,2);
mask = sImage > someThresholdValue;
mask = bwareafilt(mask, [100, inf]);
labeledImage = bwlabel(mask);
props = regionprops(labeledImage, thermalImage, 'MeanIntensity');

[Pavan Basavaraj]

As I only need to find green pixels you initially told me to find object based on H,S and V values which is actually yielding me better results. So why just saturation values now?

And would it be possible to find the max and min temperature of each plant?

[Image Analyst]

You can use whatever channels you need to. Try the Color Thresholder app on the Apps tab of the tool ribbon. Maybe saturation is enough but maybe you need to include others - I don't know until I try it. If the background is white, and the pots and dirt are black or white, then saturation might be enough to get the green. Can't hurt to also threshold the hue to the green range though, but it might not be required.

Having a mask for green, then you can use that in the thermal image to get the max and min:

minTemp = min(thermalImage(greenMask));
maxTemp = max(thermalImage(greenMask));

This will get the min and max of the thermal image but ONLY within the green mask.

[Pavan Basavaraj]

Thanks a lot Image Analyst. Everything works. Couldn't have achieved without your guidance.

[sangeetha N]

Why we are checking only on green mask? Why not with other primary colors- Red and yellow?

Is it because green mask is related with some temperature difference indication?

Other colors will not help in the same?

[pavan basavaraj doddajjappanavar]

The colors only doesn’t mean anything. The goal was to find the temperature difference in leaves using infrared camera. Which gives out only grey scale images. And as my background was complex. Image analyst gave an idea to combine with RGB camera. Do the image registration and mask for green so that I can find respective temperature pixels from IR camera.

[Xiaoxia Shen]

Hello, I also met this kind of problem, may I ask that have you solved this problem or not, may I get your solution if possible