Spatio-temporal video segmentation using k-means clustering

Hi everyone,

I am new to the forum and relatively new to Matlab so I hope I'll be on topic and I won't sound too naive.

I've actually been trying to implement a spatio-temporal video segmentation in two steps: (a)photometric based frame segmentation obtained by quickshift ;(b)spatio-temporal segmentation of the (previously segmented#)individual frames over time, obtained by k-means clustering. So far, I have succesfully implemented step (a), but I am now stuck on step (b), as I can't get how the output of my k-means clustering should be interpreted to be honest, so mostly I've faced a conceptual problem.

Once I have segmented every frame (intra-frame) of my video, on the base of their photometric characteristics (position and intensity), then having found several regions each one described by its centroid and mean intensity, I need to find its correspondent over time (inter-frame), and meanwhile reducing the number of regions (down to K=30 clusters specifically). Now, considering the kmeans matlab function :[match, centre]= kmeans(data,K), K = 30 and my data matrix will be composed by M= feature dimension rows and N=number of frames columns. My actual problem is defining the data vectors for every frame, as I think I haven't grasped well how kmeans work itself. Is it correct, I describe every frame region by a feature vector which is given by centroid postition and mean intensity, then concatenate all this regions in a column vector and eventually, try to compute the k-means algorithm among every column of my data (frame of my video)?

Note that, every frame can be composed by a different number of regions, and that I ordered the regions on the base of their centroid before computing the final result...

This is how my code for step (b) looks like:

%feature_vec is the cell array which will contain my visual features
feature_vec = cell(num_frames-2,1); 

for i = 2:num_frames-1
   region      = labels{1,i};  % labels is the cell array which contain the    region names for every frame

      s         = regionprops(region,'centroid');  % extract region centroid
      centroids    = cat(1, s.Centroid);
      centroids    = round(centroids);
      feature_v    = zeros(size(centroids,1),5);

      feature_v(:,1:2)    = centroids;
      image               = Iseg{1,i};  % is the segmented frame

   for j =1:size(centroids,1)
        feature_v(j,3)       = image(centroids(j,2),centroids(j,1),1);  % find R value of the particular region
        feature_v(j,4)       = image(centroids(j,2),centroids(j,1),2);  % find G value of the particular region
        feature_v(j,5)       = image(centroids(j,2),centroids(j,1),3);  % find B value of the particular region
    end

    feature_vec{i-1,1}    = feature_v;

end

feature_vec_sorted = cellfun (@(x) sortrows(x,[1 2]),feature_vec,'UniformOutput',false);  % order features on the base of centroids

maxNRegions = cellfun(@(x) max(size(x,1)),feature_vec,'UniformOutput',false);
data        = zeros(num_frames-2,maxNRegions);

% organise data for kmeans processing

for i = 1:num_frames-2
    frame_features = feature_vec_sorted{i};
    nCol           = size(frame_features,1)*5;
    data(i,1:nCol) = reshape(frame_features',[1 nCol]);
end

[centre,match] = vl_kmeans(data,30);

end

Where am I conceptually wrong?

Thanks a lot for your patience in reading.

Regards.

ps: I know the code is not efficient this way. Please do suggest efficient ways of implementing it if you feel so.