Removing a character from a table (that's within a struct)

Hi,

I have a .mat file with the name faceDatasetGroundTruth.mat (a struct) that contains a table (called 'faceDataset') with two columns.

Each value in the second column starts and ends with a ' character, which I'd like to remove so that I'm only left with [95,71,226,313] in the first row, for example.

Thanks in advance.

[C B]

Is there any specific reason for doing so ?

because when i run following it shows me that first char is '[' and not '''

>> faceDataset.face{1}(1)

ans =

    '['

>> faceDataset
faceDataset =
  3×2 table
    filename       face    
    ________    ___________

      'ab'      '[1 2 2 4]'
      'bc'      '[1 2 2 4]'
      'cd'      '[1 2 2 4]'
>> faceDataset.face{1}
ans =
    '[1 2 2 4]'

>> faceDataset.face{1}(1)
ans =
    '['
>> faceDataset.face{1}(end)
ans =
    ']'

Answer 1

[Stephen23]

As far as I understand, you want to convert one column/variable of the table from cell array of character vectors to numeric. This is easy with convertvars:

S = load('faceDatasetGroundTruth.mat')

S = struct with fields:
    faceDataset: [202599×2 table]

T = S.faceDataset

T = 202599×2 table
          imageFilename                  face         
    _________________________    _____________________

    {'faceImages/000001.jpg'}    {'[95,71,226,313]'  }
    {'faceImages/000002.jpg'}    {'[72,94,221,306]'  }
    {'faceImages/000003.jpg'}    {'[216,59,91,126]'  }
    {'faceImages/000004.jpg'}    {'[622,257,564,781]'}
    {'faceImages/000005.jpg'}    {'[236,109,120,166]'}
    {'faceImages/000006.jpg'}    {'[146,67,182,252]' }
    {'faceImages/000007.jpg'}    {'[64,93,211,292]'  }
    {'faceImages/000008.jpg'}    {'[212,89,218,302]' }
    {'faceImages/000009.jpg'}    {'[600,274,343,475]'}
    {'faceImages/000010.jpg'}    {'[113,110,211,292]'}
    {'faceImages/000011.jpg'}    {'[166,68,125,173]' }
    {'faceImages/000012.jpg'}    {'[102,31,104,144]' }
    {'faceImages/000013.jpg'}    {'[89,132,247,342]' }
    {'faceImages/000014.jpg'}    {'[110,122,234,324]'}
    {'faceImages/000015.jpg'}    {'[93,86,190,263]'  }
    {'faceImages/000016.jpg'}    {'[39,89,283,392]'  }

F = @(c)sscanf([c{:}],'[%f,%f,%f,%f]',[4,Inf]).';
T = convertvars(T,'face',F);
T = convertvars(T,'imageFilename',@string) % optional, a bit slow.

T = 202599×2 table
         imageFilename                   face          
    _______________________    ________________________

    "faceImages/000001.jpg"     95     71    226    313
    "faceImages/000002.jpg"     72     94    221    306
    "faceImages/000003.jpg"    216     59     91    126
    "faceImages/000004.jpg"    622    257    564    781
    "faceImages/000005.jpg"    236    109    120    166
    "faceImages/000006.jpg"    146     67    182    252
    "faceImages/000007.jpg"     64     93    211    292
    "faceImages/000008.jpg"    212     89    218    302
    "faceImages/000009.jpg"    600    274    343    475
    "faceImages/000010.jpg"    113    110    211    292
    "faceImages/000011.jpg"    166     68    125    173
    "faceImages/000012.jpg"    102     31    104    144
    "faceImages/000013.jpg"     89    132    247    342
    "faceImages/000014.jpg"    110    122    234    324
    "faceImages/000015.jpg"     93     86    190    263
    "faceImages/000016.jpg"     39     89    283    392

[Yonathan Zarkovian]

Hi,

Thank you. What I got from running your code is a column with the numbers being separate but not in parantheses and not separated with commas.

Here's what the column should look like (disregard the first column and the 'vehicle' label above the second column):

Thanks again.

[Stephen23]

@Yonathan Zarkovian: you are confusing how data are stored with how data are displayed.

Character vectors within a cell array within a table are displayed with single quote marks, but those do NOT indicate that there are actually single quote characters at the ends of the character vectors. This is trivial to demonstrate:

C = {'cat';'hat'};
T = table(C)

T = 2×1 table
       C   
    _______

    {'cat'}
    {'hat'}

T.C{1}(1) % the first character is 'c', not a single quote.

ans = 'c'

So your question is basically asking how to remove something that is not there... but such a task is impossible: you cannot remove what is not there. If you want to display tables differently then you could write your own routine to do so (but this would not be a trivial task and not many people would waste their time on that).

Because you asked how to remove non-existent characters, I interpreted your question as asking how to convert those character vectors (in a cell array) into actual numeric data.

Perhaps that is what you really need, who knows.

"Here's what the column should look like"

The important question is: why should it look like that? Your character vectors already DO NOT have single quotes (that you are trying to remove, even though they do not exist), so the actual stored characters are already exactly as you show.

Do you actually need numeric data or character/text data?

https://xyproblem.info

https://en.wikipedia.org/wiki/XY_problem

https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem

[Yonathan Zarkovian]

Hi,

Thank you for the clarification. The example I showed in the screenshot above is of an existing dataset that works with the algorithm, so I'm assuming that's what my dataset should like like as well, for it to work.

The example is taken from this guide:

Object Detection Using YOLO v3 Deep Learning - MATLAB & Simulink (mathworks.com)

The first column should be text and the second numeric.

Here's the full code and the error message with the faceDatasetGroundTruth.mat file:

%% download pretrained network (set to 'true' to train the network)
doTraining = false;

if ~doTraining
    preTrainedDetector = downloadPretrainedYOLOv3Detector();    
end

%% load data (modify to faces dataset)
data = load('faceDatasetGroundTruth.mat')
faceDataset = data.faceDataset;

% Add the full path to the local face data folder.
faceDataset.imageFilename = fullfile(pwd, faceDataset.imageFilename);

% Split the data set into a training set for training the network, and a test set for evaluating the network. Use 60% of the data for training set and the rest for the test set
rng(0);
shuffledIndices = randperm(height(faceDataset));
idx = floor(0.6 * length(shuffledIndices));
trainingDataTbl = faceDataset(shuffledIndices(1:idx), :);
testDataTbl = faceDataset(shuffledIndices(idx+1:end), :);

% Create an image datastore for loading the images
imdsTrain = imageDatastore(trainingDataTbl.imageFilename);
imdsTest = imageDatastore(testDataTbl.imageFilename);

% Create a datastore for the ground truth bounding boxes
bldsTrain = boxLabelDatastore(trainingDataTbl(:, 2:end));
bldsTest = boxLabelDatastore(testDataTbl(:, 2:end));

% Combine the image and box label datastores
trainingData = combine(imdsTrain, bldsTrain);
testData = combine(imdsTest, bldsTest);

% detect invalid images, bounding boxes or labels - discard invalid images
% manually
% validateInputData(trainingData);
% validateInputData(testData);

%% Data Augmentation (using random transformation)
augmentedTrainingData = transform(trainingData, @augmentData);

%% Define YOLO v3 Object Detector (use 227 227 3 for lower computational time. standard size for v3 is 416 416 3)
networkInputSize = [450 450 3];

rng(0)
trainingDataForEstimation = transform(trainingData, @(data)preprocessData(data, networkInputSize));
numAnchors = 6;
[anchors, meanIoU] = estimateAnchorBoxes(trainingDataForEstimation, numAnchors)

area = anchors(:, 1).*anchors(:, 2);
[~, idx] = sort(area, 'descend');
anchors = anchors(idx, :);
anchorBoxes = {anchors(1:3,:)
    anchors(4:6,:)
    };

baseNetwork = squeezenet;
classNames = trainingDataTbl.Properties.VariableNames(2:end);

yolov3Detector = yolov3ObjectDetector(baseNetwork, classNames, anchorBoxes, 'DetectionNetworkSource', {'fire9-concat', 'fire5-concat'});

%% Preprocess Training Data
preprocessedTrainingData = transform(augmentedTrainingData, @(data)preprocess(yolov3Detector, data));
data = read(preprocessedTrainingData);
reset(preprocessedTrainingData);

%% Training options
numEpochs = 80;
miniBatchSize = 8;
learningRate = 0.001;
warmupPeriod = 1000;
l2Regularization = 0.0005;
penaltyThreshold = 0.5;
velocity = [];

%% Train Model
if canUseParallelPool
   dispatchInBackground = true;
else
   dispatchInBackground = false;
end

mbqTrain = minibatchqueue(preprocessedTrainingData, 2,...
        "MiniBatchSize", miniBatchSize,...
        "MiniBatchFcn", @(images, boxes, labels) createBatchData(images, boxes, labels, classNames), ...
        "MiniBatchFormat", ["SSCB", ""],...
        "DispatchInBackground", dispatchInBackground,...
        "OutputCast", ["", "double"]);

if doTraining
    
    % Create subplots for the learning rate and mini-batch loss.
    fig = figure;
    [lossPlotter, learningRatePlotter] = configureTrainingProgressPlotter(fig);

    iteration = 0;
    % Custom training loop.
    for epoch = 1:numEpochs
          
        reset(mbqTrain);
        shuffle(mbqTrain);
        
        while(hasdata(mbqTrain))
            iteration = iteration + 1;
           
            [XTrain, YTrain] = next(mbqTrain);
            
            % Evaluate the model gradients and loss using dlfeval and the
            % modelGradients function.
            [gradients, state, lossInfo] = dlfeval(@modelGradients, yolov3Detector, XTrain, YTrain, penaltyThreshold);
    
            % Apply L2 regularization.
            gradients = dlupdate(@(g,w) g + l2Regularization*w, gradients, yolov3Detector.Learnables);
    
            % Determine the current learning rate value.
            currentLR = piecewiseLearningRateWithWarmup(iteration, epoch, learningRate, warmupPeriod, numEpochs);
    
            % Update the detector learnable parameters using the SGDM optimizer.
            [yolov3Detector.Learnables, velocity] = sgdmupdate(yolov3Detector.Learnables, gradients, velocity, currentLR);
    
            % Update the state parameters of dlnetwork.
            yolov3Detector.State = state;
              
            % Display progress.
            displayLossInfo(epoch, iteration, currentLR, lossInfo);  
                
            % Update training plot with new points.
            updatePlots(lossPlotter, learningRatePlotter, iteration, currentLR, lossInfo.totalLoss);
        end        
    end
else
    yolov3Detector = preTrainedDetector;
end

%% Evaluation
% results = detect(yolov3Detector,testData,'MiniBatchSize',8);
% 
% % Evaluate the object detector using Average Precision metric.
% [ap,recall,precision] = evaluateDetectionPrecision(results,testData);
% 
% % Plot precision-recall curve.
% figure
% plot(recall,precision)
% xlabel('Recall')
% ylabel('Precision')
% grid on
% title(sprintf('Average Precision = %.2f', ap))

%% Detect Objects
% Read the datastore.
% data = read(testData);
% 
% % Get the image.
% I = data{1};
% 
% [bboxes,scores,labels] = detect(yolov3Detector,I);
% 
% % Display the detections on image.
% I = insertObjectAnnotation(I,'rectangle',bboxes,scores);
% 
% figure
% imshow(I)

%% test with single image file
img = imread('test.jpg');
[bboxes,scores,labels] = detect(yolov3Detector,img);
img = insertObjectAnnotation(img,'rectangle',bboxes,scores);
figure
imshow(img)

%% webcam setup
% cam = webcam('EpocCam Camera');
% cam.resolution = '640x480';
% img = snapshot(cam);
% [bboxes,scores,labels] = detect(yolov3Detector,img);
% img = insertObjectAnnotation(img,'rectangle',bboxes,scores);
% figure
% imshow(img)

%% object detection in video
% v = VideoReader('cars.mp4');
% img = read(v,1);
% [bboxes,scores,labels] = detect(yolov3Detector,img);
% img = insertObjectAnnotation(img,'rectangle',bboxes,scores);
% figure
% imshow(img)
%%
% for c = 1:25
%     img = read(v,c);
%     [bboxes,scores,labels] = detect(yolov3Detector,img);
%     img = insertObjectAnnotation(img,'rectangle',bboxes,scores);
%     figure
%     imshow(img)
% end

    
>> yolov3_Copy

data = 

  struct with fields:

    faceDataset: [202599×2 table]

Error using boxLabelDatastore (line 234)
Invalid data in column 1 of input table 1.

Error in yolov3_Copy (line 27)
bldsTrain = boxLabelDatastore(trainingDataTbl(:, 2:end));

Caused by:
    Error using boxLabelDatastore>iAssertValidBBoxFormat (line 870)
    The size of bounding box data must be M-by-4, M-by-5, or M-by-9, where M is the number of boxes in each table element. The column
    in the training data table that contains the bounding boxes must be a cell array.
 
>> 

[Stephen23]

"The first column should be text and the second numeric. "

If the second column should be numeric then use the code I gave in my answer: as I already wrote, my code converts your text data to numeric. Which you have now informed us, is actually what you need.

Your approach (trying to remove single quotes that do not even exist) does not convert that data to numeric.

[Yonathan Zarkovian]

I did use your code. Unfortunately it results in this error, as mentioned in my previous reply:

Error using boxLabelDatastore (line 234)
Invalid data in column 1 of input table 1.
Error in yolov3_Copy (line 27)
bldsTrain = boxLabelDatastore(trainingDataTbl(:, 2:end));
Caused by:
Error using boxLabelDatastore>iAssertValidBBoxFormat (line 870)
The size of bounding box data must be M-by-4, M-by-5, or M-by-9, where M is the number of boxes in each table element. The column
in the training data table that contains the bounding boxes must be a cell array.

Thanks again.

[Stephen23]

"Unfortunately it results in this error"

Yes, because you are not supplying that tool with the data in the format that it requires. You are using the tool, which means that you have its documentation, which means that you can read its documentation and find out what are the specific needs for the data.

The error message also gives some hints, perhaps something like this might be what that tool needs:

S = load('faceDatasetGroundTruth.mat')

S = struct with fields:
    faceDataset: [202599×2 table]

T = S.faceDataset

T = 202599×2 table
          imageFilename                  face         
    _________________________    _____________________

    {'faceImages/000001.jpg'}    {'[95,71,226,313]'  }
    {'faceImages/000002.jpg'}    {'[72,94,221,306]'  }
    {'faceImages/000003.jpg'}    {'[216,59,91,126]'  }
    {'faceImages/000004.jpg'}    {'[622,257,564,781]'}
    {'faceImages/000005.jpg'}    {'[236,109,120,166]'}
    {'faceImages/000006.jpg'}    {'[146,67,182,252]' }
    {'faceImages/000007.jpg'}    {'[64,93,211,292]'  }
    {'faceImages/000008.jpg'}    {'[212,89,218,302]' }
    {'faceImages/000009.jpg'}    {'[600,274,343,475]'}
    {'faceImages/000010.jpg'}    {'[113,110,211,292]'}
    {'faceImages/000011.jpg'}    {'[166,68,125,173]' }
    {'faceImages/000012.jpg'}    {'[102,31,104,144]' }
    {'faceImages/000013.jpg'}    {'[89,132,247,342]' }
    {'faceImages/000014.jpg'}    {'[110,122,234,324]'}
    {'faceImages/000015.jpg'}    {'[93,86,190,263]'  }
    {'faceImages/000016.jpg'}    {'[39,89,283,392]'  }

F = @(c)num2cell(sscanf([c{:}],'[%f,%f,%f,%f]',[4,Inf]).',2);
T = convertvars(T,'face',F)

T = 202599×2 table
          imageFilename                 face        
    _________________________    ___________________

    {'faceImages/000001.jpg'}    {[  95 71 226 313]}
    {'faceImages/000002.jpg'}    {[  72 94 221 306]}
    {'faceImages/000003.jpg'}    {[  216 59 91 126]}
    {'faceImages/000004.jpg'}    {[622 257 564 781]}
    {'faceImages/000005.jpg'}    {[236 109 120 166]}
    {'faceImages/000006.jpg'}    {[ 146 67 182 252]}
    {'faceImages/000007.jpg'}    {[  64 93 211 292]}
    {'faceImages/000008.jpg'}    {[ 212 89 218 302]}
    {'faceImages/000009.jpg'}    {[600 274 343 475]}
    {'faceImages/000010.jpg'}    {[113 110 211 292]}
    {'faceImages/000011.jpg'}    {[ 166 68 125 173]}
    {'faceImages/000012.jpg'}    {[ 102 31 104 144]}
    {'faceImages/000013.jpg'}    {[ 89 132 247 342]}
    {'faceImages/000014.jpg'}    {[110 122 234 324]}
    {'faceImages/000015.jpg'}    {[  93 86 190 263]}
    {'faceImages/000016.jpg'}    {[  39 89 283 392]}

[Yonathan Zarkovian]

Thank you, this method seems to work.