BWCONNCOMP for sparse coordinate data

I have a list of image pixel coordinates as a 2xN matrix where N is much less than the total number of pixels in the image. I would like a fast/efficient way of dividing them into connected components. I could use the coordinates to populate a black and white image, and then use BWCONNCOMP, but I already have the white pixel coordinates as a list, which is a more memory-efficient form. Creating a non-sparse image just to satisfy the input requirements of BWCONNCOMP seems like a step backward in efficiency.

Does anyone know of a superior approach that will work with the coordinate data in its raw form and doesn't require building an intermediate image? I can't see why working with the data in image form would be part of what makes the computation efficient.