Main Content

evaluateAnomalyDetection

Evaluate anomaly detection results against ground truth

    Description

    example

    metrics = evaluateAnomalyDetection(predLabels,trueLabels,truePosLabels) calculates various metrics to evaluate the quality of the anomaly detection results predLabels against the ground truth trueLabels. truePosLabels lists the class labels in trueLabels that correspond to positive detections.

    Note

    This function requires the Computer Vision Toolbox™ Automated Visual Inspection Library. You can install the Computer Vision Toolbox Automated Visual Inspection Library from Add-On Explorer. For more information about installing add-ons, see Get and Manage Add-Ons. To run this function, you will require the Deep Learning Toolbox™.

    metrics = evaluateAnomalyDetection(predLabels,trueLabels,truePosLabels,Verbose=tf) also displays the evaluation progress information in the command window.

    Examples

    collapse all

    Load test images and corresponding labels, then create a datastore that reads the test data. The data set consists of grayscale images of handwritten digits 0–9.

    [Xtest,catLabels] = digitTest4DArrayData;
    dsTest = arrayDatastore(Xtest,IterationDimension=4);

    Load a pretrained FCDD anomaly detector. This detector has been trained to classify the digit 8 as normal and all other digits as anomalies. Therefore, specify the set of positive labels as the set of digits between 0 and 9, excluding 8.

    load("digit8AnomalyDetector.mat");
    posLabels = setdiff(string(0:9),"8");

    Classify each test image as normal or anomalous.

    labelsPred = classify(detector,dsTest);

    Calculate metrics for the anomaly detection results.

    metrics = evaluateAnomalyDetection(labelsPred,catLabels,posLabels)
    Evaluating anomaly detection results
    ------------------------------------
    * Finalizing... Done.
    * Data set metrics:
    
        GlobalAccuracy    MeanAccuracy    Precision    Recall    Specificity    F1Score    FalsePositiveRate    FalseNegativeRate
        ______________    ____________    _________    ______    ___________    _______    _________________    _________________
    
            0.7662           0.839         0.98971     0.748        0.93        0.85204          0.07                 0.252      
    
    metrics = 
      anomalyDetectionMetrics with properties:
    
                  ConfusionMatrix: [2x2 table]
        NormalizedConfusionMatrix: [2x2 table]
                   DataSetMetrics: [1x8 table]
                     ClassMetrics: [2x2 table]
    
    

    Display the confusion matrix.

    cm = metrics.ConfusionMatrix
    cm=2×2 table
                   Normal    Anomaly
                   ______    _______
    
        Normal       465        35  
        Anomaly     1134      3366  
    
    

    Input Arguments

    collapse all

    Predicted labels, specified as a logical vector. The label value 0 (false) corresponds to a negative detection (the "Normal" class). The value 1 (true) corresponds to a positive detection (the "Anomaly" class). You can get the predicted labels using the classify function.

    Data Types: logical

    Ground truth labels for each image, specified as a numeric vector, logical vector, or categorical vector of the same length as predLabels. In order to calculate metrics, the evaluateAnomalyDetection function converts the labels into a logical vector according to the set of positive labels in truePosLabels.

    Ground truth positive labels, specified as a numeric vector, logical vector, string vector, or categorical vector.

    • When trueLabels is categorical, truePosLabels can be string or categorical.

    • When trueLabels is numeric or logical, truePosLabels can be numeric or logical.

    The evaluateAnomalyDetection function converts all ground truth labels in trueLabels that are in the set of positive labels to a logical true, indicating a positive detection. The function converts all other ground truth labels to a logical false, indicating a negative detection.

    Display evaluation progress information in the command window, specified as a numeric or logical 1 (true) or 0 (false). The displayed information includes a progress bar, elapsed time, estimated time remaining, and data set metrics.

    Data Types: logical

    Output Arguments

    collapse all

    Anomaly detection metrics, returned as an anomalyDetectionMetrics object.

    Version History

    Introduced in R2022b