Can anyone suggest what operations should I perform to make my input image better?

I am trying to count the no. of fingers by feeding the input through webcam.But I am encountering a problem with the binary image which is getting worst with holes and discontinuities. Can anyone suggest some operations that can be made on my binary image so that I could make it better.

I've used skin pixel segmentation method to get the binary image and applied morphological operations to get the second binary image and I've written the code for finger detection to get the final image.