Getting Started with Anomaly Detection Using Deep Learning
Anomaly detection using deep learning is an increasingly popular approach to automating visual inspection tasks. The goal of anomaly detection is to perform a binary classification of images as normal or anomalous. For example, in the manufacturing industry, anomaly detection can determine which manufactured units meet quality standards and which units are defective.
You can train an anomaly detector using semi-supervised training. A complete semi-supervised workflow consists of training a model on normal image data and determining an anomaly threshold that separates normal images from anomalous images. Anomaly detection techniques generally follow these steps:
Prepare training data and calibration data.
Train a model to produce anomaly scores.
Calibrate the model by determining an anomaly score threshold, and evaluate the performance of the model by classifying metrics and visualizing the classification results.
Classify test images using the trained model and the optimal anomaly threshold.
Deploy the model onto a production line.
An alternative to anomaly detection is anomaly classification. Classification can differentiate between normal images and multiple known anomalous patterns. Classification can provide more insight on the type of anomaly, but it requires a greater amount of training data of the anomaly classes. For an example, see Classify Defects on Wafer Maps Using Deep Learning.
The Computer Vision Toolbox™ Automated Visual Inspection Library offers functions that enable you to train, calibrate, and evaluate anomaly detection networks. You can install the Computer Vision Toolbox Automated Visual Inspection Library from Add-On Explorer. For more information about installing add-ons, see Get and Manage Add-Ons. The functionality also requires Deep Learning Toolbox™.
Prepare Training and Calibration Data
You can train an anomaly detector using one-class learning, which means that the network trains on data consisting only of normal images. Despite training on samples only of normal images, the model learns how to distinguish between normal and anomalous images. One-class learning offers many advantages for anomaly detection problems:
Representations of anomalies can be scarce.
Anomalies can represent expensive or catastrophic outcomes.
There can be many types of anomalies, and the types of anomalies can change over the lifetime of the model. Describing what a normal image looks like is often more feasible than providing data that represents all possible anomalies in real world settings.
You can use the Image Labeler
app to interactively label images with scene labels. The app returns the ground truth
labeled data as a
groundTruth object. Convert the ground
truth scene labels to an image datastore for training by using the
sceneLabelTrainingData function. For more information, see Get Started with the Image Labeler.
In semi-supervised learning, you can tune the performance of the trained model using
calibration data. The calibration data set consists of labeled samples of normal and
anomalous images. Avoid overfitting the model by using different images in the training
and calibration data sets. You can use functions such as
partition to split a datastore into separate datastores for training and
Some techniques, such as Outlier Exposure, add a small number of anomalous images to the training data set.
Choose the Anomaly Detection Model
The following table distinguishes the characteristics and performance of anomaly detection methods available through the Computer Vision Toolbox Automated Visual Inspection Library during training and inference. The available anomaly detectors include fully convolutional data description (FCDD) , FastFlow , and PatchCore . Choose an anomaly detector based on the requirements of your workflow.
Consider the training requirements.
Available Anomaly Detection Models: Training
|Input image size
|Small to medium preferred (due to memory limitation for large images)
|Small to medium preferred (due to memory limitation for large images)
|Small to large (high-resolution images)
|Moderate to large (can vary with value of the compression ratio)
|Moderate to large
|Small (lightest weight model)
|Pretrained model as base feature extraction network
|Flexible (supports ResNet-18 or ResNet-50)
|Flexible (supports custom network creation)
|Low-shot training regime
Consider the inference and testing requirements.
Available Anomaly Detection Models: Inference
|Different image size at training and inference
|Not supported, as image size at inference must match image size at training (due to being a feature similarity-based technique)
|Medium to fast
Train the Model
The Computer Vision Toolbox Automated Visual Inspection Library provides framework for training anomaly detection networks. Start by creating an untrained network object. Then, train the network by passing the network and the training data to the appropriate training function. The table shows the supported networks and training functions.
|Fully convolutional data description (FCDD) 
If you want to train an anomaly detection network that uses a different framework, then design a model that accepts images as input and returns scalar valued anomaly scores. The score represents the likelihood that an image is anomalous. Larger scores indicate a higher probability that an image in anomalous samples. Optionally, you can design a network that returns localized anomaly scores in addition to a scalar aggregate score for the image. Localized score information enables you to subjectively understand the network classification decisions.
Calibrate and Evaluate the Model
Calibrate the model by determining an anomaly score threshold that separates the scores of good samples from the scores of anomaly samples. Unfortunately, most data sets cannot be cleanly separated into these two categories, and will have misclassifications in the form of false positives (normal images classified as anomalous) and false negatives (anomalous images classified as normal). By adjusting the threshold, you can tune the false positive and false negative rates to satisfy your operating requirements.
To select a threshold, you can use the
anomalyThreshold function. The function supports different optimization
methods based on statistical techniques such as receiver operating characteristic (ROC)
and precision-recall (PR) curves. The function optionally returns the performance
metrics as an
rocmetrics (Deep Learning Toolbox)
function, which you can use to plot the ROC and PR curves.
evaluateAnomalyDetection function to evaluate the quality of the anomaly
detection results using metrics such as the confusion matrix and average precision. The
anomalyDetectionMetrics object stores the metrics.
You can visualize the localized anomaly scores to understand the classification
results. For example, you can display an anomaly score map as a heatmap overlayed on a
calibration or test image using the
anomalyMapOverlay function. You can display an interactive figure that
displays the classification results and score maps for an entire data set using the
viewAnomalyDetectionResults function. Visualizing localized anomaly
scores is useful for identifying patterns in false negatives and false positives. The
patterns can indicate strategies to further improve the classifier performance.
Perform Classification Using the Model
Deploy the Model
The end goal of visual inspection is often to deploy trained models for use in making business decisions in real-time production systems on assembly lines. The model determines acceptance or rejection given an image. The MATLAB® Coder™ and GPU Coder™ products are effective tools for deploying visual inspection systems to CPU and NVIDIA® GPU targets.
 Liznerski, Philipp, Lukas Ruff, Robert A. Vandermeulen, Billy Joe Franks, Marius Kloft, and Klaus-Robert Müller. "Explainable Deep One-Class Classification." Preprint, submitted March 18, 2021. https://arxiv.org/abs/2007.01760.
 Yu, Jiawei, Ye Zheng, Xiang Wang, Wei Li, Yushuang Wu, Rui Zhao, and Liwei Wu. "FastFlow: Unsupervised Anomaly Detection and Localization via 2D Normalizing Flows." arXiv, November 16, 2021. https://doi.org/10.48550/arXiv.2111.07677.
 Roth, Karsten, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. “Towards Total Recall in Industrial Anomaly Detection.” arXiv, May 5, 2022. https://arxiv.org/abs/2106.08265.
- Detect Image Anomalies Using Pretrained ResNet-18 Feature Embeddings
- Detect Image Anomalies Using Explainable FCDD Network
- ROC Curve and Performance Metrics (Deep Learning Toolbox)