evaluateObjectDetection

Evaluate object detection data set against ground truth

Since R2023b

Syntax

metrics = evaluateObjectDetection(detectionResults,groundTruthData)

metrics = evaluateObjectDetection(detectionResults,groundTruthData,threshold)

metrics = evaluateObjectDetection(___,Name=Value)

Description

metrics = evaluateObjectDetection(detectionResults,groundTruthData) evaluates the quality of the object detection results detectionResults against the labeled ground truth groundTruthData and returns various metrics.

metrics = evaluateObjectDetection(detectionResults,groundTruthData,threshold) specifies the overlap threshold for assigning an output bounding box to a ground truth bounding box.

metrics = evaluateObjectDetection(___,Name=Value) specifies one or more name-value arguments, in addition to any combination of input arguments from previous syntaxes, to configure the object detection results evaluation. For example, AdditionalMetrics="AOS" includes average orientation similarity metrics in the output.

Examples

Plot Precision-Recall Curve for Object Detection

Load a table containing images and ground truth bounding box labels. The first column contains the images, and the remaining columns contain the labeled bounding boxes.

data = load("vehicleTrainingData.mat");
trainingData = data.vehicleTrainingData;

Set the value of the dataDir variable as the location where the vehicleTrainingData.mat file is located. Load the test data into a local vehicle data folder.

dataDir = fullfile(toolboxdir("vision"),"visiondata");
trainingData.imageFilename = fullfile(dataDir,trainingData.imageFilename);

Create an imageDatastore using the files from the table.

imds = imageDatastore(trainingData.imageFilename);

Create a boxLabelDatastore using the label columns from the table.

blds = boxLabelDatastore(trainingData(:,2:end));

Load Pretrained Object Detector

Load a pretrained YOLO v2 object detector trained to detect vehicles into the workspace.

vehicleDetector = load("yolov2VehicleDetector.mat");
detector = vehicleDetector.detector;

Evaluate and Plot Object Detection Metrics

Run the detector on the test images. Set the detection threshold to a low value to detect as many objects as possible. This helps you evaluate the detector precision across the full range of recall values.

results = detect(detector,imds,Threshold=0.01);

Use evaluateObjectDetection to compute metrics for evaluating the performance of an object detector.

metrics = evaluateObjectDetection(results,blds);

Return the precision and recall metrics for the vehicle class using the precisionRecall object function.

[recall,precision,scores] = precisionRecall(metrics);
ap = averagePrecision(metrics);

Plot the precision-recall curve for the vehicle class, the only class in the data set. Compute the average precision (AP) using the averagePrecision object function.

figure
plot(recall{1},precision{1})
grid on
title("Average Precision = " + ap);
xlabel("Recall");
ylabel("Precision");

Compute the summary of the object detection metrics for the data set using the summarize object function.

[summaryDataset,summaryClass] = summarize(metrics);
summaryDataset

summaryDataset=1×3 table
    NumObjects    mAPOverlapAvg    mAP0.5 
    __________    _____________    _______

       336           0.99096       0.99096

Input Arguments

detectionResults — Predicted object detection results

table

Predicted object detection results, specified as a three-column table containing the bounding box, predicted label, and score for each detected object. The table describes the formats of the required elements.

Bounding Boxes	Labels	Scores
Predicted 2-D bounding boxes for M objects, specified as an M-by-4 or M-by-5 numeric array. Each 2-D bounding box must be in the format [x y width height] if axis-aligned, or in the format [xcenter ycenter width height yaw] if rotated.	Predicted object labels, specified as an M-by-1 categorical vector.	Predicted scores, specified as an M-by-1 numeric vector.

The order of the elements does not matter. When the datastore returns a cell array with more than three elements, the evaluateObjectDetection function assumes that the first element with an M-by-4 or M-by-5 numeric array contains the bounding boxes, the first element with categorical data contains the label data, and the first element with M-by-1 numeric data contains the scores.

You can create the detection results table using the detect function associated with your object detector..

groundTruthData — Labeled ground truth

datastore | table

Labeled ground truth, specified as a datastore with two outputs or a table with two columns. This table describes the formats of the required elements.

boxes	labels
Ground truth 2-D bounding boxes for M objects, specified as an M-by-4 or M-by-5 numeric array. Each 2-D bounding box must be in the format [x y width height] if axis-aligned, or in the format [xcenter ycenter width height yaw] if rotated.	Ground truth object labels, specified as an M-by-1 categorical vector.

The order of the elements does not matter. When the datastore returns a cell array with more than three elements, the evaluateObjectDetection function assumes that the first element with an M-by-4 or M-by-5 numeric array contains the bounding boxes, the first element with categorical data contains the label data, and the first element with M-by-1 numeric data contains the scores.

threshold — Overlap threshold

0.5 (default) | numeric scalar | numeric vector

Overlap threshold for assigning a detection to a ground truth box, specified as a numeric scalar in the range [0, 1] or a numeric vector. The overlap ratio is the intersection over union (IoU) ratio of two bounding boxes, or the ratio of the bounding box overlap area to the combined area of the predicted boxes and ground truth (minus the overlap).

IoU = Area of Overlap / (Ground Truth Area + Predicted Box Area – Intersection Area)

When you specify a numeric vector, the evaluateObjectDetection function calculates metrics for each threshold.

Tip

To reduce the number of false negatives, you can lower the IoU threshold at the possible expense of increasing the number of false positives. To learn about the role of IoU in detection evaluation, see Overlap Threshold (Intersection Over Union).

Tip

To optimize model performance for your application, you can evaluate the object detection metrics across a range of overlap thresholds. To learn how to fine-tune object detector performance using multiple overlap thresholds, see Tips.

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: evaluateObjectDetection(__,AdditionalMetrics=["LAMR","AOS"]) additionally returns LAMR and AOS.

Verbose — Evaluation progress display toggle

true or 1 (default) | false or 0

Evaluation progress display toggle, specified as a numeric or logical 1 (true) or 0 (false). If you specify Verbose as true, the function displays progress information in the Command Window. The displayed information includes a progress bar, elapsed time, estimated time remaining, and data set metrics.

AdditionalMetrics — Additional metrics

[] (default) | "LAMR" | "AOS" | string array

Additional metrics, specified as "LAMR", "AOS", or a string array. The function returns the specified metrics as additional columns in the ClassMetrics and ImageMetrics tables of the objectDetectionMetrics object, as shown in this table below.

AdditionalMetrics	ClassMetrics	ImageMetrics
"LAMR"	LAMROverlapAverage: Log-average miss rate for each class, averaged over all specified thresholds, as a numeric scalar. LAMR: Log-average miss rate for each class, computed at each threshold, as a numThresh-by-1 numeric vector. MR: Miss rate vector for each class as a numThresh-by-numPredictions matrix, where numPredictions is the number of predicted objects. FPPI: False positives per image vector for each class as a numThresh-by-numPredictions matrix.	LAMROverlapAverage: Log-average miss rate for each image, averaged over all specified overlap thresholds, as a numeric scalar. LAMR: Log-average miss rate for each image, computed at all specified overlap thresholds, as a numThresh-by-1 numeric vector.
"AOS"	AOSOverlapAvg: Average orientation similarity (AOS) for each class averaged over all the specified overlap thresholds, as a numeric scalar. AOS: Average orientation (AOS) similarity for each class computed at each overlap threshold as a numThresh-by-1 numeric vector. OrientationSimilarity: Orientation similarity vector for each class as a numThresh-by-(numPredictions+1) matrix.	AOSOverlapAvg: Average orientation similarity (AOS) for each image, averaged over all the specified overlap thresholds, as a numeric scalar. AOS: Average orientation (AOS) similarity for each image computed at each overlap threshold as a numThresh-by-1 numeric vector.

To specify "AOS" as an additional metric, your input data must contain rotated bounding boxes.

Output Arguments

metrics — Object detection metrics

objectDetectionMetrics object

Object detection metrics, returned as an objectDetectionMetrics object. The OverlapThreshold property of the object corresponds to the value of the threshold argument.

More About

Overlap Threshold (Intersection Over Union)

Use overlap (or IoU) threshold to determine whether a detection is a true positive or false positive.

A detection is a true positive (TP) when both of these conditions are met:

• The predicted class matches the class of a ground truth (indicated by the confidence score exceeding the confidence threshold).

• The predicted bounding box has an IoU greater than or equal to the specified overlap threshold.

A detection is a false positive (FP) when at least one of these conditions is met:

• The predicted class does not match the class of a ground truth.

• The predicted bounding box has an IoU less than the specified overlap threshold.

A missed detection is a false negative (FN) when one or more of these conditions is met:

• The confidence score of the detection is lower than the confidence score threshold, resulting in a missed detection.

• The detection has an IoU less than the specified threshold, resulting in the predicted bounding box does not sufficiently overlapping with the ground truth, leading to the detection being discarded.

Quantify the accuracy of an object detector on a particular dataset by evaluating the overlap between the predicted bounding boxes and the ground truth bounding boxes.

Tips

To evaluate model performance across different levels of localization accuracy and fine-tune an object detector, compute metrics at multiple overlap thresholds. Overlap thresholds vary typically from 0.5 to 0.95 during evaluation. For example, set the overlap (IoU) threshold value to 0.5 to accept moderate overlap accuracy, and set the value to 0.95 to ensure very high localization accuracy. There are many ways to use the overlap threshold to tune object detector performance.

• By evaluating common metrics, such as precision and recall, at different overlap thresholds, you can evaluate the trade-off between detecting more objects (higher recall) and ensuring those detections are accurate (higher precision).

• By evaluating model performance at a lower IoU threshold, such as 0.5, and at higher thresholds, such as 0.95, you can determine how good the detector is at identifying objects (detection) at the expense of precise localization (bounding box accuracy).

• By computing metrics such as the mean average precision (mAP) over multiple overlap thresholds, you can produce a single performance figure that accounts for both detection and localization accuracy, enabling you to compare different detection models.

To learn how to compute metrics at a range of overlap thresholds to tune detector performance, see the "Precision and Recall for a Single Class" section of the Multiclass Object Detection Using YOLO v2 Deep Learning example.

Version History

Introduced in R2023b

R2023b: Recommended over evaluateDetectionPrecision, evaluateDetectionMissRate, evaluateDetectionAOS

The evaluateObjectDetection function enables you to simultaneously calculate the average precision, miss rate, and AOS (if applicable) to evaluate detector performance. Therefore, evaluateObjectDetection is recommended over evaluateDetectionPrecision, evaluateDetectionMissRate, and evaluateDetectionAOS, which will be removed in a future release. To update your code, ensure that your ground truth data is in the correct format. See the groundTruthData argument for more details.

evaluateObjectDetection improves on evaluateDetectionPrecision by enabling you to compute the object detection performance metrics for the entire data set over all classes and at specified overlap thresholds, use rotated bounding boxes, and calculate the confusion matrix. The corresponding objectDetectionMetrics object also enables you to evaluate object detection metrics across various object size ranges using the metricsByArea function.