Create Deep Learning Processor Configuration for Custom Layers

Deep learning networks use custom layers to perform actions such as resizing 2-D inputs by a scale factor, performing element-wise multiplications, and so on. If your network requires layers to perform certain actions and the layers are not provided by Deep Learning Toolbox™, create a custom layer. Rapidly prototype, validate and deploy your networks that have custom layers by:

Creating and registering your custom layer function and Simulink^® model.
Validating your custom layer
Generating a custom bitstream

Deploy the network that has custom layers to a target board by using the custom bitstream

Deploy Custom Layer Networks

Create a custom processor configuration object by using the dlhdl.ProcessorConfig object.
For layers that use a custom function, create a MATLAB^® function and Simulink model that replicates your custom layer function.
Register your custom layer function and Simulink model by using the registerCustomLayer method.
Enable the registered custom layers in your custom deep learning processor configuration.
Simulate and verify your custom layer by using a generated verification model. Generate a verification model by using the openCustomLayerModel method. Verify the custom layer by using the verifyCustomLayerModel method. This step is optional.
Generate a custom bitstream by using the dlhdl.buildProcessor function.
Create a workflow object that has your custom layer network and custom bitstream by using the dlhdl.Workflow object.
Compile and deploy the workflow object by using the compile and deploy methods.
Retrieve the prediction results from the deployed network by using the predict method.

Tip

If you are creating a layer with multiple inputs, then you must set the NumInputs properties in the layer constructor.

Retrieve the prediction results from the deployed network by using MATLAB.

Create a Deep learning Processor Configuration

To generate a custom processor configuration, use the dlhdl.ProcessorConfig object. The generated deep learning processor configuration object has a custom module that contains the preconfigured custom layers. Save the deep learning processor configuration to a variable named hPC.

hPC = dlhdl.ProcessorConfig

hPC = 

                    Processing Module "conv"
                            ModuleGeneration: 'on'
                          LRNBlockGeneration: 'off'
                 SegmentationBlockGeneration: 'on'
                            ConvThreadNumber: 16
                             InputMemorySize: [227 227 3]
                            OutputMemorySize: [227 227 3]
                            FeatureSizeLimit: 2048

                      Processing Module "fc"
                            ModuleGeneration: 'on'
                      SoftmaxBlockGeneration: 'off'
                      SigmoidBlockGeneration: 'off'
                              FCThreadNumber: 4
                             InputMemorySize: 25088
                            OutputMemorySize: 4096

                  Processing Module "custom"
                            ModuleGeneration: 'on'
                                    Addition: 'on'
                              Multiplication: 'on'
                             InputMemorySize: 40
                            OutputMemorySize: 40

              Processor Top Level Properties
                              RunTimeControl: 'register'
                               RunTimeStatus: 'register'
                          InputStreamControl: 'register'
                         OutputStreamControl: 'register'
                           ProcessorDataType: 'single'

                     System Level Properties
                              TargetPlatform: 'Xilinx Zynq UltraScale+ MPSoC ZCU102 Evaluation Kit'
                             TargetFrequency: 200
                               SynthesisTool: 'Xilinx Vivado'
                             ReferenceDesign: 'AXI-Stream DDR Memory Access : 3-AXIM'
                     SynthesisToolChipFamily: 'Zynq UltraScale+'
                     SynthesisToolDeviceName: 'xczu9eg-ffvb1156-2-e'
                    SynthesisToolPackageName: ''
                     SynthesisToolSpeedValue: ''

Create Custom Layer MATLAB Function

Design the custom layer function by using a MATLAB function. The custom layer function must:

Have a maximum of two inputs and one output.
Use only element-wise operations. These operations are not element-wise operations:
- Matrix multiplication
- Flatten
- Reshape
- Concatenation
- Batch normalization

This example code shows the MATLAB function for a custom signum layer.

classdef SignumLayer < nnet.layer.Layer
    % Example custom Signum layer.
    
    properties
        testPropertyValue1 single = 3;
        testPropertyValue2 single = 4;
    end
    
    methods
        function layer = SignumLayer(name)
            % Set layer name.
            layer.Name = name;
            % Set layer description.
            layer.Description = "custom signum layer";
        end
        
        function Z = predict(layer, X)
            % Z = predict(layer, X) forwards the input data X through the
            % layer and outputs the result Z.
            
            Z = sign(X) + layer.testPropertyValue1 + layer.testPropertyValue2;
           
        end
    end
end

Create Custom Layer MATLAB Function with `inputParser`

When you call the openCustomLayerModel method without a Network name-value argument for a custom layer with no properties the custom layer must take the layer name as its only argument. For example,

classdef Exp1Layer < nnet.layer.Layer
    methods
        function layer = Exp1Layer(varargin)
            p = inputParser;
 
            addParameter(p,'Name', 'exp');
            parse(p,varargin{:});
            layer.Name = p.Results.Name;
            ...
        end
        ...
    end
end

When you call the openCustomLayerModel method without a Network name-value argument for a custom layer with properties you must use an inputParser object to parse the properties and the properties should be a property-value pair list that includes the layer name and the other properties. For example,

classdef sek3Layer < nnet.layer.Layer
    properties
        G = 0.95;
    end
 
    methods
        function layer = sek3Layer(varargin)
            p = inputParser;
            addParameter(p,'Name', 'sek');
            addParameter(p,'G',  0.95);
            parse(p,varargin{:});
            layer.Name = p.Results.Name;
            layer.G = p.Results.G;
            ...
        end
        ...
    end
end

Create Custom Layer Simulink Function

Design the custom layer model in Simulink. Your model design must:

Use subsystem reference blocks only. Model reference blocks are not supported.
Model the inputValid and outputValid signals.
Have the same inputs and outputs as the custom layer MATLAB function.

This image shows the Simulink model for the custom signum layer.

Simulink model representation of custom signum layer

Register Custom Layer and Model

hSignum = SignumLayer('Signum1');
registerCustomLayer(hPC, Layer = hSignum, Model = 'mySignumModel.slx');
hPC

The custom deep learning processor configuration has a Signum layer under the custom processing module. The custom signum layer is enabled for bitstream generation.

hPC

hPC = 

                    Processing Module "conv"
                            ModuleGeneration: 'on'
                          LRNBlockGeneration: 'off'
                 SegmentationBlockGeneration: 'on'
                            ConvThreadNumber: 16
                             InputMemorySize: [227 227 3]
                            OutputMemorySize: [227 227 3]
                            FeatureSizeLimit: 2048

                      Processing Module "fc"
                            ModuleGeneration: 'on'
                      SoftmaxBlockGeneration: 'off'
                      SigmoidBlockGeneration: 'off'
                              FCThreadNumber: 4
                             InputMemorySize: 25088
                            OutputMemorySize: 4096

                  Processing Module "custom"
                            ModuleGeneration: 'on'
                                    Addition: 'on'
                              Multiplication: 'on'
                                 SignumLayer: 'on'
                             InputMemorySize: 40
                            OutputMemorySize: 40

              Processor Top Level Properties
                              RunTimeControl: 'register'
                               RunTimeStatus: 'register'
                          InputStreamControl: 'register'
                         OutputStreamControl: 'register'
                           ProcessorDataType: 'single'

                     System Level Properties
                              TargetPlatform: 'Xilinx Zynq UltraScale+ MPSoC ZCU102 Evaluation Kit'
                             TargetFrequency: 200
                               SynthesisTool: 'Xilinx Vivado'
                             ReferenceDesign: 'AXI-Stream DDR Memory Access : 3-AXIM'
                     SynthesisToolChipFamily: 'Zynq UltraScale+'
                     SynthesisToolDeviceName: 'xczu9eg-ffvb1156-2-e'
                    SynthesisToolPackageName: ''
                     SynthesisToolSpeedValue: ''

Generate Verification Model for Custom Layer

Generate a verification model for your custom layer by using the openCustomLayerModel method. Generate a test network and test image for your custom layer network by specifying blank arguments for the Network and InputImages arguments of the openCustomLayerModel method. The size of the test image matches the input layer size of the created test network.

openCustomLayerModel(hPC)

### The 'Network' property is empty for the given object. An auto-generated network is provided.
### Custom layer verification model generation begin.
### Compiling network for Deep Learning FPGA prototyping ...
### Custom layer verification model generation complete.

An input image of size two-by-two-by four is created for the generated verification model. This image shows the auto-generated network for the custom layer model.

Network layer drawing for auto-generated test network

The openCustomLayerModel method generates a verification model file called dnnfpgaCustomLayerVerificationModel.slx for your custom layer. The generated verification model consists of a test bench block TB and a design under test block DUT. The testbench block contains tests signals that are applied to your custom layer model which is a part of the design under test block to verify the functionality of the custom layer and prediction accuracy of the network that has the custom layer. This image shows the generated verification model blocks.

Generated custom layer verification model

Simulate and Validate Custom Layer Model

Before you verify your custom layer model by using the verifyCustomLayerModel method, open the dnnfpgaCustomLayerVerificationModel.slx verification model. The verifyCustomLayerModel method verifies the functionality of the custom layer and prediction accuracy of the network that has the custom layer.

verifyCustomLayerModel(hPC)

### Custom layer verification model simulation and validation begin.
### Compiling Simulink model 'dnnfpgaCustomLayerVerificationModel' ...
### Complete Simulink model 'dnnfpgaCustomLayerVerificationModel' compilation.
Verification passed.
### Custom layer verification model simulation and validation complete.

Use the generated verification model to simulates, test, iterate and develop your custom kernel Simulink model. This image shows the custom kernel development process.

Custom layer kernel development process

Generate Custom Bitstream

Generate a custom bitstream that has the name myCustomLayer.bit by using the dlhdl.buildProcessor function. Save the generated bitstream to the myCustomLayer_prj folder.

dlhdl.buildProcessor(hPC, ProjectFolder = 'myCustomLayer_prj', ProcessorName = 'myCustomLayer');

Deploy and Predict Custom Layer Network on Hardware

Deploy the custom layer network by creating a dlhdl.Workflow object with the custom layer network as the Network argument and the custom bitstream myCustomLayer.bit as the Bitstream argument. To retrieve the prediction results from the deployed network use MATLAB and the predict method.

hTarget = dlhdl.Target('Xilinx','Interface','JTAG');
hW = dlhdl.Workflow(Network = myCustomNet, Bitstream =
'myCustomLayer.bit'
,
...
Target = hTarget);
hW.compile;
hW.deploy;
image = randi(255, [2,2,4]);
hW.predict(single(image),Profile =
'on'
);

Custom Layer Registration File

To reuse your verified custom layers, register them by using a custom layer registration file. Custom registration layer files must be named dlhdl_customLayer_registration.m. The custom layer registration file contains a list of dlhdl.CustomLayer objects. A specific board can have multiple custom layer registration files on the MATLAB path. Do not list dlhdl.CustomLayer objects in more than one custom layer registration file.

When the processor configuration object is created, Deep Learning HDL Toolbox™ searches the MATLAB path for files named dlhdl_customLayer_registration.m, and uses the information in the files to populate the registered custom layer information. List only custom layers in the custom layer registration file after they have been verified by using the verifyCustomLayerModel method.

This script is an example of a dlhdl_customLayer_registration.m file.

function customLayerList = dlhdl_customLayer_registration
% Custom layer registration file
% 1. Any registration file with this name on MATLAB path will be picked up.
% 2. Registration file returns a cell array of dlhdl.CustomLayer
% object which are used to register custom layer information for Deep 
% Learning HDL Toolbox workflow
% 3. Use dlhdl.CustomLayer object to register a layer class, and a
% model file path relative to the location of this registration file

%   Copyright 2021 The MathWorks, Inc.

customLayerList = { ...
    dlhdl.CustomLayer('Name', 'Addition',       'Layer', additionLayer(2),       'Model', 'model/customLayers/dnnfpgaAdditionLayerModel.slx'), ...
    dlhdl.CustomLayer('Name', 'Multiplication', 'Layer', multiplicationLayer(2), 'Model', 'model/customLayers/dnnfpgaMultiplicationLayerModel.slx'), ...
    };

end

To register the custom signum layer for reuse, create this dlhdl_customLayer_registration.m file and place it on the MATLAB path.

function customLayerList = dlhdl_customLayer_registration
% Custom layer registration file
% 1. Any registration file with this name on MATLAB path will be picked up.
% 2. Registration file returns a cell array of dlhdl.CustomLayer
% object which are used to register custom layer information for Deep 
% Learning HDL Toolbox workflow
% 3. Use dlhdl.CustomLayer object to register a layer class, and a
% model file path relative to the location of this registration file

%   Copyright 2021 The MathWorks, Inc.

customLayerList = { ...
    dlhdl.CustomLayer('Name','Signum','Layer',SignumLayer('Signum'),'Model','C:\myCustomLayer\dnnfpgaSignumLayerModel.slx'), ...
   };

end

Create a dlhdl.ProcessorConfig object. The custom signum layer now appears in the default processor configuration object under the custom processing module.

hPC = dlhdl.ProcessorConfig

hPC = 

                    Processing Module "conv"
                            ModuleGeneration: 'on'
                          LRNBlockGeneration: 'off'
                 SegmentationBlockGeneration: 'on'
                            ConvThreadNumber: 16
                             InputMemorySize: [227 227 3]
                            OutputMemorySize: [227 227 3]
                            FeatureSizeLimit: 2048

                      Processing Module "fc"
                            ModuleGeneration: 'on'
                      SoftmaxBlockGeneration: 'off'
                      SigmoidBlockGeneration: 'off'
                              FCThreadNumber: 4
                             InputMemorySize: 25088
                            OutputMemorySize: 4096

                  Processing Module "custom"
                            ModuleGeneration: 'on'
                                    Addition: 'on'
                              Multiplication: 'on'
                                      Signum: 'on'
                             InputMemorySize: 40
                            OutputMemorySize: 40

              Processor Top Level Properties
                              RunTimeControl: 'register'
                               RunTimeStatus: 'register'
                          InputStreamControl: 'register'
                         OutputStreamControl: 'register'
                           ProcessorDataType: 'single'

                     System Level Properties
                              TargetPlatform: 'Xilinx Zynq UltraScale+ MPSoC ZCU102 Evaluation Kit'
                             TargetFrequency: 200
                               SynthesisTool: 'Xilinx Vivado'
                             ReferenceDesign: 'AXI-Stream DDR Memory Access : 3-AXIM'
                     SynthesisToolChipFamily: 'Zynq UltraScale+'
                     SynthesisToolDeviceName: 'xczu9eg-ffvb1156-2-e'
                    SynthesisToolPackageName: ''
                     SynthesisToolSpeedValue: '

For an example of how to create, register, validate, and deploy a network with a custom log layer, see Register, Validate, and Deploy Custom Natural Logarithm Layer Network to FPGA.