How to store the coordinates of a bounding box in a video?

Hello. I am working on a project where i use a camera to detect moving objects. Now what i want to do is to create a code in order to save the x and y coordinates of a (moving) bounding box (in all the frames of my video that is visible). I tried the code below but it saves only the last x and y k-times, where k is the number of frames of my video.

   ....
      while ~isDone(hVidReader)  % Stop when end of file is reached
       frame  = step(hVidReader);  % Read input video frame

       grayFrame = rgb2gray(frame);

       %The optical flow vectors are stored as complex numbers.    
       ofVectors = step(hOpticalFlow1, grayFrame); % Estimate optical flow
      %Compute their  magnitude squared which will later be used for thresholding.
      y1 = ofVectors .* conj(ofVectors);
      % Compute the velocity threshold from the matrix of complex velocities.
      vel_th = 0.5 * step(hMean2, step(hMean1, y1));
      % Threshold the image and then filter it to remove speckle noise.
      segmentedObjects = step(hMedianFilt, y1 >= vel_th);
      % Thin-out the parts of the road and fill holes in the blobs.
      segmentedObjects = step(hclose, step(herode, segmentedObjects));
      % Estimatethe area and bounding box of the blobs.
      [area, bbox] = step(hblob, segmentedObjects);
      % Select boxes inside ROI.
      Idx = (bbox(:,1) > lineRow1 & bbox(:,1) < lineRow2)&(bbox(:,2) > lineColumn1 & bbox(:,2) <  
      lineColumn2);
    % Based on blob sizes, filter out objects which can not be cars.
    % When the ratio between the area of the blob and the area of the
    % bounding box is above 0.4 (40%), classify it as a car.
    ratio = zeros(length(Idx), 1);
    ratio(Idx) = single(area(Idx,1))./single(bbox(Idx,3).*bbox(Idx,4));
    ratiob = ratio > 0.4;
    count = int32(sum(ratiob));    % Number of cars
    bbox(~ratiob, :) = int32(-1);
    % Draw bounding boxes around the tracked cars.
    y2 = step(hshapeins1, frame, bbox);
    % Display the number of cars tracked and  white lines showing the ROI.
    y2(22:25,250:1097,:)   = 1;   % The top white line.
    y2(230:233,250:1097,:) = 1;   % The bottom white line.
    y2(25:230,247:250,:) = 1;   % The left white line.
    y2(25:230,1097:1100,:) = 1; % The right white line.
    y2(1:15,1:30,:) = 0;   % Background for displaying count
    result = step(htextins, y2, count);

     for jj = 1 : k
       if bbox ~ [];
           if bbox(1,1)>0
             xbbox(jj) = bbox(1,1);
             ybbox(jj) = bbox(1,2);
             zz(:,:,jj) = [xcentroid,ycentroid];

            else
               xbbox(jj) = 0;
               ybbox(jj) =0;
                zz(:,:,jj) = 0;
           end
      end
end

    % Generate coordinates for plotting motion vectors.
    if firstTime
      [R , C] = size(ofVectors);            % Height and width in pixels
      RV = borderOffset:decimFactorRow:(R-borderOffset);
      CV = borderOffset:decimFactorCol:(C-borderOffset);
      [Y ,X] = meshgrid(CV,RV);
      firstTime = false;

    end

    % Calculate and draw the motion vectors.
    tmp = ofVectors(RV,CV) .* motionVecGain;
    lines = [Y(:), X(:), Y(:) + real(tmp(:)), X(:) + imag(tmp(:))];
    motionVectors = step(hshapeins2, frame, lines);
    % Display the results

    step(hVideo4, result);           % Video with bounding boxes

    end    
release(hVidReader);

The lines i was talking about are: for jj = 1 : k if bbox ~ []; if bbox(1,1)>0 xbbox(jj) = bbox(1,1); ybbox(jj) = bbox(1,2); zz(:,:,jj) = [xcentroid,ycentroid];

            else
               xbbox(jj) = 0;
               ybbox(jj) =0;
                zz(:,:,jj) = 0;
           end
      end
end

I also tried other things but the outcome was about the same. I don't know if i gave you adequate iinformation, so please ask me anything you want. Thank you all in advance.

[Peter Turner]

Hi! trying to do something similar to you for my uni project, just wondering how you're getting a threshold-ed binary image from the flow vectors? What are 'hmean1' and 'hmean2'? Would really help me a lot

[Peter Turner]

Ok I worked out the binary image stuff....DO you mind elaborating on this though: '% Select boxes inside ROI. Idx = (bbox(:,1) > lineRow1 & bbox(:,1) < lineRow2)&(bbox(:,2) > lineColumn1 & bbox(:,2) < lineColumn2);'

What are linecolumn 1 and 2?

Answer 1

[Image Analyst]

I'd probably make a cell array, call it caBoxes. It would be numFrames cells long. Each cell of the cell array has one 2D array for x1, x2, y1, and y2 for the four sides of the box for the cars/boxes in that frame. If frame k has N cars in it, then caBoxes{k} would have boundingBox be a N rows by 4 columns double array where the 4 columns define x1, x2, y1, and y2. N might be different from frame to frame, that's why we use a cell array instead of a regular rectangular numerical array. See the FAQ on cell arrays: http://matlab.wikia.com/wiki/FAQ#What_is_a_cell_array.3F

[Nikolaos]

Hello Image Analyst. I only want the x and y coordinates of the upper left corner of the bounding box, because as i noticed the length and the height of bounding box change throughout the video. So i don't want to base my code on them. I finally reached to the following code:

if bbox ~ [];
         if bbox(1,1)>0;
         caBoxes{jj} = bbox;
        else
         caBoxes{jj} =0;
         end
     else
        caBoxes{jj} = 0;
     end

Could you tell me your opinion? Is there any way to do it better?

[Image Analyst]

bbox(1,1) will never be zero if you got it from a MATLAB function, (I don't even know if it will be empty), so I think you'd want

if ~isempty(bbox)
  caBoxes{jj} = {[bbox(1,1), bbox(1,2)}; % Store [x,y];
else
  caBoxes{jj} = {[1, 1]};
end

[Nikolaos]

According to these lines:

% Select boxes inside ROI.
      Idx = (bbox(:,1) > lineRow1 & bbox(:,1) < lineRow2)&(bbox(:,2) > lineColumn1 & bbox(:,2) <  
      lineColumn2);
    % Based on blob sizes, filter out objects which can not be cars.
    % When the ratio between the area of the blob and the area of the
    % bounding box is above 0.4 (40%), classify it as a car.
    ratio = zeros(length(Idx), 1);
    ratio(Idx) = single(area(Idx,1))./single(bbox(Idx,3).*bbox(Idx,4));
    ratiob = ratio > 0.4;
    count = int32(sum(ratiob));    % Number of cars
    bbox(~ratiob, :) = int32(-1);

When the moving object isn't in the ROI or ratio isn't >0.4 , bbox =[-1 -1 -1 -1]. I run the code you gave and i stop the video(in 18 frame). Here are some of the results:

>> caBoxes{1}

ans =

       0     0

>> caBoxes{10}

ans =

            -1          -1

That's why i wrote "if bbox(1,1)>0". Also i changed the syntax after "else". I want if is empty or -1 to store it as zero.

[Image Analyst]

OK, not sure how you got bbox, but if that's what it gives, then that's what it gives, and you have to deal with it.

[Nikolaos]

Thank you very much for your time and your advices

Answer 2

[Dima Lisin]

Hi Nikolaos,

The problem here is that the second output of step(hblob, segmentedObjects) is the centroids, not the bounding boxes. For the bounding boxes, you would have to get the 3rd output:

 [area, centroid, bbox] = step(hblob, segmentedObjects);

Now bbox will be an M-by-4 matrix, where each row is of the form [x, y, w, h].

[Nikolaos]

Hello Dima,

I am really sorry for the delay. As you can see below, in hblob definition the "CentroidOutputPort" is set to false , while "AreaOutputPort" and "BoundingBoxOutputPort" are set to true:

hblob = vision.BlobAnalysis('CentroidOutputPort', false, 'AreaOutputPort', true, 'BoundingBoxOutputPort', true, 'OutputDataType', 'double','MinimumBlobArea', 250, 'MaximumBlobArea', 1800, 'MaximumCount', 1);

So (i believe) bbox is the 2nd output and as you correctly wrote is M-by-4 matrix, where each row is of the form [x, y, w, h].