From ADC to AI: A Radar Data Deep Learning Tutorial

Since R2025a

Introduction

This example demonstrates a full deep learning workflow for collected radar data. It shows how to process the raw ADC data, create a labeled dataset, and finally, how to train a neural network in MATLAB® on radar data. After this example, you should be more familiar with how to incorporate radar data into your deep learning pipelines.

This example assumes that the tasks of target detection and/or tracking are performed separately and focuses solely on the task of classifying detections in the spatial domain based on the shape and magnitude of a target's response. Furthermore, we perform classification only after we project the range-azimuth response onto Cartesian coordinates so that target sizes remain relatively uniform for the ranges under consideration. Projecting onto Cartesian coordinates also allows the neural network to learn the physical sizes of each target class and have a fixed-size input. Otherwise, targets that are close to the radar could spread across many angle bins and require the input image to be very large so that it contains the entire target. The workflow can be succinctly written as:

1. Compute range-azimuth response from raw ADC data.

2. Threshold and project the response into Cartesian space.

3. Label reflections.

4. Export and aggregate all labeled patches into a single dataset.

5. Train and test a neural network that classifies these patches.

Data Collection

This example uses the Texas Instruments IWR1642BOOST automotive radar paired with a data acquisition board (DCA1000EVM). Together, these boards export raw I/Q samples that can be processed and visualized using MATLAB. For this example, we collected pedestrian data indoors, and bicycle data outdoors. The pedestrian data is in an office setting and contains a pedestrian walking around in an open area. In the outdoor data, a bicyclist rides around in circles in a parking lot. The radar was positioned so that it had angular resolution in the horizontal axis (azimuth) and no resolution in the vertical axis (elevation). I/Q Data Collection and Detection Generation with Texas Instruments (TI) millimeter-wave (mmWave) Radar explains how to use MATLAB® and the TI radars to collect, save, and process radar data.

Data Processing

This section shows how to compute the spatial heatmap from the radar's ADC samples. For convenience, we already preprocessed all of the recordings and only process a small subset of the data below for illustrative purposes. First, load the configuration file used to run the radar during data collection. The contents of this configuration file are used to configure the processing pipeline that outputs the range-azimuth response. To create the processing components for this example, instantiate an object of the helperRadarAndProcessor class. This class contains processing functions and a radar transceiver modeled after the TI radar used for the pedestrian and bicyclist data collection.

addpath("tiRadar")
% Create Radar from configuration file
radarCfg = helperParseRadarConfig('config_Fri_11_1.cfg');
radarSystem = helperRadarAndProcessor(radarCfg);

Next, load the radar data.

frames = load('radarFramesWalkingTI.mat').frameShort;
nFrame = size(frames,4);

Then, for each coherent processing interval (CPI), also known as a frame, compute the associated spatial heatmap using the helperMVDRHeatmap function. This function outputs the spatial response using phased.MVDREstimator for azimuth angles between +/- 70 degrees, thresholds the MVDR output, and projects the output onto Cartesian coordinates. Two example outputs are shown below. They depict a pedestrian moving from left to right. The left animation uses a magnitude threshold of 38 dB and performs no clutter suppression. This threshold is user selected and was chosen visually based on the images. The right animation uses the same threshold but also suppresses the clutter by subtracting the average pulse (the signal averaged over all the pulses in a CPI) to suppress the clutter.

Process the data below by choosing a threshold and whether to suppress the clutter. Furthermore, this is a convenient location in the workflow to augment the data by adding targets or simulating antenna phase and amplitude noise as described in Augmenting Radar Data to Create More Diverse Datasets.

saveFullSceneFlag = true; % Set to true to save the data
removeClutterFlag = false;
MVDRthreshold = 38;
rngBins = radarCfg.RangeResolution*(0:radarCfg.SamplesPerChirp-1);
figure;
set(gcf,'Visible','on');
if saveFullSceneFlag && ~exist('testdir')
    mkdir('testdir')
end
for frameIdx = 1:nFrame
    sig = frames(:,:,:,frameIdx);
    sig = reformatData(sig);
    patch = helperMVDRHeatmap(sig,radarSystem,rngBins,removeClutterFlag,MVDRthreshold);
    title(['Frame : ' num2str(frameIdx)],'Interpreter', 'none')
    colormap('gray')
    if saveFullSceneFlag
        helperSavePatch(flipud(patch),['testdir/frame'],frameIdx)
    end
    pause(.1)
end

Labeling the Data

After generating spatial heatmaps for each CPI, label it using the Image Labeler app in the Computer Vision Toolbox™. This app allows you to easily create labels for detection and segmentation tasks by providing a variety of labeling options. To get started with labeling radar data, see Get Started with the Image Labeler (Computer Vision Toolbox). Here is a brief summary of the steps used to label the datasets in this example:

1. Create a new session in the imageLabeler.

2. Import the saved images produced in the Data Processing section using the import button in the top left.

3. Add label definitions for each class by clicking Add Label. For the selected frames, create a rectangular label for class human.

4. Click on the label definition on the left and then draw the bounding box.

5. Repeat for each target of interest.

Below is an example CPI where a human is marked with a green bounding box.

After labeling the imported images, click the Export button and export the labels to a file. This is already done for your convenience and saved in the file testlabels.mat. Then, using those labels, create the cropped input patches based on the bounding boxes by running the code below. The following code saves patches of size 128-by-128 pixels in the directory testdir/dataset. The 128-by-128 pixel patches have an equivalent size of 4-by-4 meters in the scene.

labelPath = 'testlabels.mat';
[filepath,name,ext] = fileparts(labelPath);
load(labelPath);
imgPaths = gTruth.DataSource.Source;
patchLen = 128;

folderpath = fullfile("testdir","dataset","human");
if ~isfolder(folderpath)
    mkdir(folderpath)
end

for imgPathIdx = 1:numel(imgPaths)
    imgPath = imgPaths{imgPathIdx};
    img = imread(imgPath);
    allLabels = gTruth.LabelData(imgPathIdx,:);
    for classIdx = 1:size(allLabels,2)
        labels = allLabels(:,classIdx);
        className = labels.Properties.VariableNames{1};
        bboxCell = labels{1,1};
        bbox = bboxCell{1};
        for imgIdx = 1:size(bbox,1)
            patch = imcrop(img,bbox(imgIdx,:));
            patch = cropOrPad2Square(patch,patchLen);
            helperSavePatch(patch,['testdir\dataset\' className '\frame_' num2str(imgPathIdx)],randi(1000))
        end
    end
end

Training a Neural Network on Radar Data

Preparing the Dataset

The inputs to the network are single-channel images that are 128-by-128 pixels. Each image contains a cropped image of a radar return as shown below. Using a dataset composed of these patches, the network will learn how to distinguish between radar returns from a pedestrian and from a bicyclist. For the purpose of this example, we provide a labeled dataset from multiple locations and with multiple people. This dataset was processed and saved using the code in the Labeling Data section and a 38 dB threshold for the MVDR output. Load the dataset and display a few of the images.

% Load the full dataset
if ~exist('BikeHumanDataset','dir')
    unzip('https://ssd.mathworks.com/supportfiles/timmwaveradar/BikeHumanDataset.zip')
end
trainFolder = 'BikeHumanDataset/train';
valFolder = 'BikeHumanDataset/val';
testFolder = 'BikeHumanDataset/test';

% Create Image Datastore, include subfolders, Use folder name as label
imdsTrain = imageDatastore(trainFolder,"IncludeSubfolders",1,"LabelSource","foldernames");
imdsVal = imageDatastore(valFolder,"IncludeSubfolders",1,"LabelSource","foldernames");
imdsTest = imageDatastore(testFolder,"IncludeSubfolders",1,"LabelSource","foldernames");

classNames = categories(imdsTrain.Labels);

% Display some sample images.
rng('default') % for reproducible results
numImages = numel(imdsTrain.Labels);
idx = randperm(numImages,16);
I = imtile(imdsTrain,Frames=idx,BorderSize=[3 3],BackgroundColor="w");

figure;
imshow(I);
title('Sample of Images from Dataset')

Visualize the class distribution of the training dataset using the countEachLabel and histogram functions. The class distributions are nearly perfectly balanced so there is no need to use any further processing or augmentation to balance this dataset.

% Visualize Class Distribution
figure
tbl=countEachLabel(imdsTrain);
histogram('Categories',tbl.Label,'BinCounts',tbl.Count)
title('Training Dataset Class Distribution')
ylabel('Number of Labeled Instances')

Next, define the data augmentation techniques to be used during training. Apply random horizontal reflections, vertical and horizontal translations, and rotations to the images. These augmentations are common in computer vision applications. For additional radar-specific augmentation techniques that can be applied during the processing of the raw ADC data, see Augmenting Radar Data to Create More Diverse Datasets. Lastly, define a transform to convert the image from type uint8 to single precision float.

inputSize = [128 128 1]; % Image size

% Apply random horizontal reflections, vertical and horizontal translations, and rotations
imageAugmenter = imageDataAugmenter( ...
    RandXReflection=true, ...
    RandXTranslation=[-25 25], ...
    RandYTranslation=[-25 25],...
    RandRotation=[-30 30]);

% Augment training data using the above imageAugmenter object
augimdsTrain = augmentedImageDatastore(inputSize(1:2),imdsTrain,DataAugmentation=imageAugmenter);

% Create augmented datastores to simplify converting the images to doubles
% This does NOT apply the imageAugmenter to the validation and test data
imdsVal = augmentedImageDatastore(inputSize(1:2),imdsVal);
testLabels = imdsTest.Labels; % Used as ground truth labels for testing later
imdsTest = augmentedImageDatastore(inputSize(1:2),imdsTest);

% Convert images from uint8 to single
augimdsTrain = transform(augimdsTrain,@(x) helperIm2Single(x));
imdsVal = transform(imdsVal,@(x) helperIm2Single(x));
imdsTest = transform(imdsTest,@(x) helperIm2Single(x));

Using a Pretrained Network

In this example, we use the pretrained Mobilenetv2 [1] network as a basis for transfer learning due to its favorable size, efficiency, and accuracy for image classification. However, because our data differs from the data that Mobilenetv2 was trained on, we need to make three modifications to the network to accommodate our radar dataset.

1. Change the imageInputLayer of the network to accommodate the 128-by-128-by-1 input. Otherwise, the network would expect images of size 224×224×3.

2. Change the first convolution2dLayer to accommodate the change in input size.

3. Change the last fullyConnectedLayer to output two values, one for each of the classes we would like to classify.

% Original pretrained network
net = imagePretrainedNetwork("mobilenetv2");

% Replace the input layer
replaceThisLayer = net.Layers(1).Name;
newLayer = imageInputLayer(inputSize);
net = replaceLayer(net,replaceThisLayer,newLayer);

% Replace the first convolution layer
replaceThisLayer = net.Layers(2).Name;
newLayer = convolution2dLayer([3,3],32,"Stride",[2,2],"Padding",[1 1]);
net = replaceLayer(net,replaceThisLayer,newLayer);

% Replace the output layer
replaceThisLayer = net.Layers(end-1).Name;
newLayer = fullyConnectedLayer(2);
net = replaceLayer(net,replaceThisLayer,newLayer);

Training

Finally, set the training options and train the network. This network converges rapidly, so run it for only three epochs (after which, we observed that the model begins to overfit).

% Specify the training options
options = trainingOptions("adam", ...
    ValidationData=imdsVal, ...
    ValidationFrequency=29, ...
    Plots="training-progress", ...
    Metrics="accuracy", ...
    Verbose=false, ...
    InitialLearnRate=1e-4,...
    MaxEpochs=3, ...
    MiniBatchSize=32, ...
    Shuffle="every-epoch");

% Train the network
net = trainnet(augimdsTrain,net,"crossentropy",options);

Test the Neural Network on Unseen Data

Use the saved labels as input into minibatchpredict and scores2label to perform inference on the testing partition of the dataset. Then, compute the accuracy of the model and visualize the results in a confusion matrix using confusionchart.

% Test network
YTest = minibatchpredict(net,imdsTest);
YTest = scores2label(YTest,classNames);

% Evaluate the classification accuracy for the test data. The accuracy is the percentage of correct predictions.
TTest = testLabels;
acc = mean(TTest==YTest)

acc = 
0.9150

% Visualize the classification accuracy in a confusion matrix
figure;
confusionchart(TTest,YTest)
title('Confusion Matrix')

Visualizing Radar Features with High Impact for Classification

The Deep Learning Toolbox™ allows you to visualize the input features that the network focuses on during inference using gradCAM. This can help guide network design and provide intuition on which features of the radar data are most useful for distinguishing between types of targets. Below, visualize the gradCAM output of the trained network along with the confidence for each prediction. Note that for lower confidence predictions, the gradCAM results show that the network is not as focused on the targets reflections.

Truth = TTest; Pred = YTest;

% Read data from imds
dataAll = read(imdsTest);
human = dataAll(dataAll.response=='human',:);
bike = dataAll(dataAll.response=='bike',:);

f = figure;
f.Position(3:4) = [1000,400];
set(gcf,'Visible','on')

for imgIdx = 1:6
    if imgIdx>3  
        data = human;
        val = randi([1 height(data)]);
    else
        data = bike;
        val = randi([1 height(data)]);
    end  
    
    img = data.input{val};
    scores = predict(net,img);
    [label,score] = scores2label(scores,classNames);
    
    leftSubplot=subplot(2,6,2*imgIdx-1);
    imshow(img)
    colormap(leftSubplot,'gray')
    title({"Pred: " + string(label) + " (Score: " + gather(score) + ")", ...
        "Truth: " + string(data.response(val))})
    
    % Show Location of Interest for Prediction
    scoreMap = gradCAM(net,img,label);
    
    rightSubplot = subplot(2,6,2*imgIdx);
    imshow(img)
    hold on
    imagesc(scoreMap,AlphaData=.5)
    colormap(rightSubplot,'jet')
end

Summary

This example demonstrates a full deep learning workflow for collected radar data. It shows how to process the raw ADC data, create a labeled dataset, and finally, how to train a neural network. You should now be more familiar with how to incorporate radar data into your deep learning pipelines.

References

[1] Sandler, Mark, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. "Mobilenetv2: Inverted residuals and linear bottlenecks." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520. 2018.

Helper Functions

% This function computes the size Cartesian image of the whole scene given
% a scale from pixels to meters. This example uses 128 pixels to 4 meters.
function [imgWidth, imgHeight] = computeSceneImgSize(targetPatchPixelLen,targetPatchMeterLen,horizontalMeterLen,verticalMeterLen)
    pixelsPerMeter = targetPatchPixelLen / targetPatchMeterLen;
    imgWidth = round(horizontalMeterLen * pixelsPerMeter);
    imgHeight = round(verticalMeterLen * pixelsPerMeter);
end

% This function computes the MVDR spatial heatmap for the scene
function patch = helperMVDRHeatmap(sig,TIradar,rngBins,subtractClutterFlag,threshold)
    tgtAnglesMVDR = [-70 70]; % azimuth angles +/- 70 deg. of boresight
    mvdrSysObj = phased.MVDREstimator( ...
            SensorArray=TIradar.VirtualArray,OperatingFrequency=TIradar.Fc, ...
            ScanAngles=(max(tgtAnglesMVDR(1),-90):min(tgtAnglesMVDR(2),90)), ...
            ForwardBackwardAveraging=true);
    
    % Simply concatenate received chirps from different Tx antennas
    sigVirt = cat(2,sig(:,:,1:2:end),sig(:,:,2:2:end)); 
    
    if subtractClutterFlag
        meanChirp = mean(sigVirt,3);
        sigVirt = sigVirt - meanChirp; % get rid of clutter
    end

    sigRFFT = TIradar.RangeProcessor(sigVirt);
    rangeBinIdxs = 5:numel(rngBins); % ignore the first few range bins
    rngBins = rngBins(rangeBinIdxs);
    sigRFFT = sigRFFT(rangeBinIdxs,:,:); % get relevant range bins after range FFT
    
    mvdrOut = zeros([numel(rangeBinIdxs),numel(mvdrSysObj.ScanAngles)]);
    for rangeIdx = 1:numel(rangeBinIdxs) % MVDR for each range bin
        mvdrOut(rangeIdx,:) = mvdrSysObj(transpose(squeeze(sigRFFT(rangeIdx,:,:))));
    end
    
    mvdrOut = mag2db(mvdrOut);
    mvdrOut(mvdrOut < threshold) = threshold; % Lowers range of values for exporting img file
    
    % Bounds of image
    xBoundsMVDR = rngBins(end)*sind(tgtAnglesMVDR);
    yBoundsMVDR = [rngBins(1)*cosd(70), rngBins(end)];
    
    % Compute the output image size and then the cartesian image
    [imgWidth, imgHeight] = computeSceneImgSize(128,4,abs(diff(xBoundsMVDR)),abs(diff(yBoundsMVDR)));
    [cartDataMVDR, xBoundsMVDR, yBoundsMVDR] = helperHeatmapPolar2Cart(mvdrOut,rngBins([1,end]),tgtAnglesMVDR,imgWidth,imgHeight,false,[]);
    gcf;
    patch = cartDataMVDR;
    set(gcf,'Visible','on'); imagesc(xBoundsMVDR,yBoundsMVDR,cartDataMVDR); axis xy; axis equal; axis tight; title('MVDR'); colorbar; clim([threshold,threshold+15]);
    xlabel('Lateral Position (m)'); ylabel('Longitudinal Position (m)')
end

% Reformats data so that the number of channels equals the number of
% receive antennas. The input's number of channels is the size of the
% virtual array (2x the number of antennas)
function dataOut = reformatData(dataIn)
    dataOut = zeros(size(dataIn,1),.5*size(dataIn,2),2*size(dataIn,3));
    dataOut(:,:,1:2:end) = dataIn(:,1:4,:);
    dataOut(:,:,2:2:end) = dataIn(:,5:end,:);
end

% Converts input images to singles for neural network
function imgOut = helperIm2Single(img)
    imgOut = img;
    for i = 1:height(imgOut.input)    
        imgIn = imgOut.input{i};
        imgOut.input{i} = im2single(imgIn);
    end
end

% Crops or zeropads the input patch to have a fixed size equal to 
% squareLen x squareLen
function patch = cropOrPad2Square(patch,squareLen)
    padSize = floor(([squareLen squareLen] - size(patch))/2);
    padSize(padSize < 0) = 0; % prevents impossible negative padding
    patch = padarray(patch,padSize,0);
    patch = paddata(patch,[squareLen,squareLen]); % ensures size at least squareLen x squareLen
    center = round(size(patch)/2);
    patch = patch(1+center(1)-squareLen/2:center(1)+squareLen/2,...
                  1+center(2)-squareLen/2:center(2)+squareLen/2);
end

% Saves the scene's spatial heatmaps into images to be labeled later
function helperSavePatch(patch,header,idx)
    patch(isnan(patch))=min(patch,[],'all');
    patch(isinf(patch))=min(patch,[],'all');
    if class(patch) ~= "uint8"
        patch = 255*(patch - min(patch(:))) ./ (max(patch(:)) - min(patch(:))); %scale values between 0 and 255
        patch = cast(patch,'uint8');
    end
    imwrite(patch,[char(header) '_' num2str(idx) '.png'])
end