Automated image labelling with Attributes

Dear all,

I'm trying to automate image labelling. I went throught several available examples (Examples), and I think I'm nearly done, but I can't automate the attributes.

My problem is the following, I want to detect goats in images and then, classify the goats depending on their color (white, red, brown and black).

I created one rectangle label, with one attribute, the color, taking value into {'white','red','brown','black'}

I manually labelled more than 3000 images and know I have trained CNNs able to do, approximately, the job for me. I want to label more images to possibly improve my CNN accuracy, but it's a shame I have to label new images manually, when I have a method that is correct at 90%

Using the example liste above, for automatic car and distance labelling, I was able to design a code that automatically detect the goats. To be more precise, if I comment everything that is related to the attributes, my code run and I have bounding boxes around the detected animals.

But when the codes about the attributes is un-comment, I have error message: "Expected automated labels to be a valid label from among the existing label definitions".

I can understand the error message, but I can't find why the definition of my label is not correct in my code ...

I hope somebody will have any idea. Best would be that MAtlab provides more examples on automated labelling :)

Below is my code:

% This class defines a template for creating a custom label automation
% algorithm, to be used in labeling apps (Image Labeler, Video Labeler and
% Ground Truth Labeler). To access help for this class, enter the following
% at the MATLAB command prompt:
%
%   >> doc vision.labeler.AutomationAlgorithm
%
% For reference, see the following AutomationAlgorithm class:
%
%   >> edit vision.labeler.PeopleDetectorACF
%
%
% To use this algorithm from within the Image Labeler, Video Labeler or
% Ground Truth Labeler App, follow the steps outlined below and complete
% the class definition. Then, save this file as follows.
%
%  Create a +vision/+labeler folder within a folder that is already
%  on the MATLAB path. For example, if the folder /local/MyProject is on
%  the MATLAB path, then create a +vision/+labeler folder hierarchy as
%  follows:
%
%           projectFolder = fullfile('local','MyProject');
%           automationFolder = fullfile('+vision','+labeler');
%
%           mkdir(projectFolder, automationFolder)
%
%  Saving the file to the package directory is required to use your custom
%  algorithm from within the labeling apps. You can add a folder to the
%  path using the ADDPATH function.
%
%  Save your algorithm class in the folder created in step 1. Refresh the
%  algorithm list from within the app to start using your custom algorithm.

classdef CabrisCouleurs < vision.labeler.AutomationAlgorithm
    
    %----------------------------------------------------------------------
    % Step 1: Define required properties describing the algorithm. This
    %         includes Name, Description and UserDirections.
    properties(Constant)
        
        % Name: Give a name for your algorithm.
        Name = 'Detection des cabris et de leurs couleurs';
        
        % Description: Provide a one-line description for your algorithm.
        Description = 'Use it to detect cabris with Brinno time lapse cameras';
        
        % UserDirections: Provide a set of directions that are displayed
        %                 when this algorithm is invoked. The directions
        %                 are to be provided as a cell array of character
        %                 vectors, with each element of the cell array
        %                 representing a step in the list of directions.
        UserDirections = {...
            ['Automation algorithms are a way to automate manual labeling ' ...
            'tasks. This AutomationAlgorithm is a template for creating ' ...
            'user-defined automation algorithms. Below are typical steps' ...
            'involved in running an automation algorithm.'], ...
            ['Run: Press RUN to run the automation algorithm. '], ...
            ['Review and Modify: Review automated labels over the interval ', ...
            'using playback controls. Modify/delete/add ROIs that were not ' ...
            'satisfactorily automated at this stage. If the results are ' ...
            'satisfactory, click Accept to accept the automated labels.'], ...
            ['Change Settings and Rerun: If automated results are not ' ...
            'satisfactory, you can try to re-run the algorithm with ' ...
            'different settings. In order to do so, click Undo Run to undo ' ...
            'current automation run, click Settings and make changes to ' ...
            'Settings, and press Run again.'], ...
            ['Accept/Cancel: If results of automation are satisfactory, ' ...
            'click Accept to accept all automated labels and return to ' ...
            'manual labeling. If results of automation are not ' ...
            'satisfactory, click Cancel to return to manual labeling ' ...
            'without saving automated labels.']};
    end
    
    %---------------------------------------------------------------------
    % Step 2: Define properties to be used during the algorithm. These are
    % user-defined properties that can be defined to manage algorithm
    % execution.
    properties
        
        %% Couleur du cabris
        couleur
        
        %% Detector
        detector
        
        %% Couleur classification
        net
        
    end
    
    properties (Constant, Access = private)

       % Flag to enable Distance attribute estimation automation
       AutomatecouleurAttribute = true;

       % Supported Distance attribute name. 
       % The label must have an attribute with the name specified.
       SupportedcouleurAttribName = 'couleur';
    end

   properties (Access = private)

       % Actual attribute name for distance
       couleurAttributeName;

       % Flag to check if attribute specified is a valid distance
       % attribute
       HasValidcouleurAttribute = false;
   end

    
    %----------------------------------------------------------------------
    % Step 3: Define methods used for setting up the algorithm.
    methods
        % a) Use the checkLabelDefinition method to specify whether a label
        %    definition is valid for the algorithm. This method is invoked
        %    on each ROI and Scene label definition to determine whether it
        %    is valid for the specified algorithm.
        %
        %    For more help,
        %    >> doc vision.labeler.AutomationAlgorithm.checkLabelDefinition
        %
        function isValid = checkLabelDefinition(~, labelDef)
            
            
            isValid = labelDef.Type == labelType.Rectangle;
            
            
        end
        
        % b) Use the checkSetup method to specify whether the algorithm is
        %    ready and all required set up is complete. If your algorithm
        %    requires no setup from the user, remove this method.
        %
        %    For more help,
        %    >> doc vision.labeler.AutomationAlgorithm.checkSetup
        %
        function isReady = checkSetup(algObj)
            
            isReady = ~isempty(algObj.SelectedLabelDefinitions);
            
        end
        
        % c) Optionally, specify what settings the algorithm requires by
        %    implementing the settingsDialog method. This method is invoked
        %    when the user clicks the Settings button. If your algorithm
        %    requires no settings, remove this method.
        %
        %    For more help,
        %    >> doc vision.labeler.AutomationAlgorithm.settingsDialog
        %
        function settingsDialog(algObj)
            
            disp('Executing settingsDialog')
            
            %--------------------------------------------------------------
            % Place your code here
            %--------------------------------------------------------------
            
            
        end
    end
    
    %----------------------------------------------------------------------
    % Step 4: Specify algorithm execution. This controls what happens when
    %         the user presses RUN. Algorithm execution proceeds by first
    %         executing initialize on the first frame, followed by run on
    %         every frame, and terminate on the last frame.
    methods
        % a) Specify the initialize method to initialize the state of your
        %    algorithm. If your algorithm requires no initialization,
        %    remove this method.
        %
        %    For more help,
        %    >> doc vision.labeler.AutomationAlgorithm.initialize
        %
        function initialize(algObj, ~)
            
            disp('Executing initialize on the first image frame')
            
            % Couleur du cabris
            algObj.couleur = algObj.SelectedLabelDefinitions.Name;
            
            % Detector
            load Detector_27_08.mat
            algObj.detector = detectorc;
            
            % Couleur classification
            load ResNet_September2021.mat
            algObj.net = net;
            
            % Initialize parameters to compute color
            if algObj.AutomatecouleurAttribute
                initializeAttributeParams(algObj);
            end
            
            %-------------------------------------------------------------
            % Place your code here
            %--------------------------------------------------------------
            
            
            
        end
        
        function initializeAttributeParams(algObj)
            % Initialize properties relevant to attribute automation.
            
            % The label must have an attribute with name Distance and type
            % Numeric Value.
            hasAttribute = isfield(algObj.ValidLabelDefinitions, 'Attributes') && ...
                isstruct(algObj.ValidLabelDefinitions.Attributes);
            if hasAttribute
                attributeNames = fieldnames(algObj.ValidLabelDefinitions.Attributes);
                idx = find(contains(attributeNames, algObj.SupportedcouleurAttribName));
                if ~isempty(idx)
                    algObj.couleurAttributeName = attributeNames{idx};
                    algObj.HasValidcouleurAttribute = validatecouleurType(algObj);
                end
            end
        end
        
       function tf = validatecouleurType(algObj)
           % Validate the attribute type. 

           tf = isfield(algObj.ValidLabelDefinitions.Attributes, algObj.couleurAttributeName);% && ...
                %isfield(algObj.ValidLabelDefinitions.Attributes.(algObj.couleurAttributeName), 'DefaultValue') && ...
                %isnumeric(algObj.ValidLabelDefinitions.Attributes.(algObj.couleurAttributeName).DefaultValue);
       end

        
        % b) Specify the run method to process an image frame and execute
        %    the algorithm. Algorithm execution begins at the first image
        %    frame and is invoked on all image frames selected for
        %    automation. Algorithm execution can produce a set of labels
        %    which are to be returned in autoLabels.
        %
        %    For more help,
        %    >> doc vision.labeler.AutomationAlgorithm.run
        %
        function autoLabels = run(algObj, I)
            img = I;
            disp('Executing run on image frame')
            ordre = cell(4,2);
            ordre{1,1} = 1:4;
            ordre{1,2} = transpose(1:4);
            for n = 2:4
                col = [];
                C = nchoosek(1:4,n);
                
                for i = 1:size(C,1)
                    
                    col = [col ; perms(C(i,:))];
                end
                ordre{n,2} = col;
                lig = repmat(1:n,[size(col,1) , 1]);
                ind = zeros(size(lig,1),n);
                for i = 1:n
                    ind(:,i) = sub2ind([n , 4],lig(:,i),col(:,i));
                end
                ordre{n,1} = ind;
            end
            
            autoLabels = [];
            [bbox , probad , class] = algObj.detector.detect(img);
            [bbox , probad ] = selectStrongestBbox(bbox,probad,'NumStrongest',4);
            %             bbox = algObj.detector.detect(I);
            
            if ~isempty(probad)
                
                proba = [];
                
                for j = 1:numel(probad)
                    
                    [~ , p] = algObj.net.classify(imresize(imcrop(img,bbox(j,:)),[224 224]));
                    %             proba = [proba ; probad(j).*(p')];
                    proba = [proba ; p];
                end
                
                
                n = numel(probad);
                ind = ordre{n,1};
                c = ordre{n,2};
                s = 0;
                for j = 1:n
                    s = s + proba(j,c(:,j))';
                end
                [~ , m] = max(s);
                couleurnum = c(m,:);
                
                
                autoLabels.Name = 'goat';
                autoLabels.Type = labelType.Rectangle;
                autoLabels.Position = bbox;
                
                if (algObj.AutomatecouleurAttribute && algObj.HasValidcouleurAttribute)
                    attribName = algObj.couleurAttributeName;
                    % Attribute value is of type 'Numeric Value'
                    for z = 1:size(bbox,1)
                        autoLabels(z).(attribName) = algObj.net.Layers(end).Classes(couleurnum(z));
                    end
                end
            end
        end
        
        % c) Specify the terminate method to clean up state of the executed
        %    algorithm. If your method requires no clean up, remove this
        %    method.
        %
        %    For more help,
        %    >> doc vision.labeler.AutomationAlgorithm.terminate
        %
        function terminate(algObj)
            
            disp('Executing terminate')
            
            %--------------------------------------------------------------
            % Place your code here
            %--------------------------------------------------------------
            
            
            
        end
    end
end