Main Content

Generate Pixel-Streaming HDL Code from Frame-Based Models

Frame-Based vs Pixel-Streaming Modeling

In hardware, processing an entire frame of video at one time has a high cost in memory and area. To save resources and reduce the latency of the design, serial processing is preferable in hardware designs. Vision HDL Toolbox™ blocks and System objects operate on a pixel, line, or neighborhood rather than a frame. The blocks and objects accept and generate video data as a serial stream of pixel data and control signals. Another option for designing video processing hardware is to model the frame-based algorithm, and then use tools that convert the algorithm to serial processing for hardware.

The frame-to-sample conversion feature in HDL Coder™ transforms your frame-based algorithm into synthesizable HDL code with a streaming interface. The interface includes control signals and the logic to handle streaming data, and is compatible with an AXI streaming interface. You can use the frame-to-sample conversion with a Simulink® model or a MATLAB® function. The generated HDL code includes the necessary logic to store samples in line buffers, align streams, and balance data paths. To develop frame-based algorithms that are compatible with frame-to-sample conversion, you can use modeling patterns such as element-wise operations, neighborhood operations, and iterative and reduction operations.

For more information on frame-to-samples conversion, see HDL Code Generation from Frame-Based Algorithms (HDL Coder).

The next section shows how to generate HDL code from a frame-based optical flow algorithm by using modeling techniques from either MATLAB or Simulink.

Generate HDL Code from Frame-Based Models by Using Neighborhood Modeling Methods

If you have a frame-based model, you can model an optical flow algorithm by using multiple neighborhood processing methods. In this example, you use the HDL Coder™ frame-to-sample conversion optimization to generate synthesizable pixel-based HDL code from two frame-based models that demonstrate different modeling patterns.

This example shows two different modeling patterns for a neighborhood processing algorithm for optical flow. One of the patterns uses a MATLAB Function block with the neighborhood processing function hdl.npufun. The other pattern uses the Neighborhood Processing Subsystem block.

Model Algorithm with MATLAB Function Block

In the hdlFrame_OpticalFlow_LK_MLFB model, there is a single MATLAB Function block inside the device under test (DUT) that contains neighborhood processing operations needed for the Lucas Kanade method for optical flow. To generate synthesizable HDL code from the frame-based model, HDL Coder uses the frame-to-sample conversion supported function, hdl.npufun to create a streaming sample-based neighborhood processing algorithm. Using this function in a frame-based model enables you to contain the entire image processing algorithm in a single MATLAB Function block.

Open the hdlFrame_OpticalFlow_LK_MLFB/DUT_LK/MATLAB Function LK subsystem to see the optical flow algorithm.

load_system("hdlFrame_OpticalFlow_LK_MLFB");
open_system("hdlFrame_OpticalFlow_LK_MLFB/DUT_LK/MATLAB Function LK");
function flowVector  = opticalFlowForHDL_lk(I, Idelay)
% Lucas Kanade

%   Copyright 2021 The MathWorks, Inc.

Ix = hdl.npufun(@imfilter_ComputIx, [1 5], I);
Iy = hdl.npufun(@imfilter_ComputIy, [5 1], I);
It = I - Idelay;

WIxx = hdl.npufun(@imfilter_kernel, [5 5], Ix.*Ix);
WIxy = hdl.npufun(@imfilter_kernel, [5 5], Ix.*Iy);
WIyy = hdl.npufun(@imfilter_kernel, [5 5], Iy.*Iy);
WIxt = hdl.npufun(@imfilter_kernel, [5 5], Ix.*It);
WIyt = hdl.npufun(@imfilter_kernel, [5 5], Iy.*It);

[Vx, Vy] = hdl.npufun(@calc_roots_pixel, [1 1], WIxx, WIxy, WIyy, WIxt, WIyt);
flowVector = complex(Vx,Vy);

end

function WI_1x1 = imfilter_ComputIx(I_1x5)

coder.inline('always');

d5 = [-1 8 0 -8 1]/12;
d5 = fliplr(d5(:)');
[WI_1x1] = sum(d5 .* I_1x5);

end

function WI_1x1 = imfilter_ComputIy(I_5x1)

coder.inline('always');

d5 = [-1 8 0 -8 1]/12;
d5 = flipud(d5(:));
[WI_1x1] = sum(d5(:) .* I_5x1);

end

function WI_1x1 = imfilter_kernel(I_5x5)

coder.inline('always');

p5 = [1 4 6 4 1]/16;
W = p5(:)*p5(:)';
[WI_1x1] = sum(W(:) .* I_5x5(:));

end

function [VxPix, VyPix] = calc_roots_pixel(aPix, bPix, cPix, ...
    txPix, tyPix)

coder.inline('always');

r = aPix + cPix;
s = sqrt(single(4 * bPix * bPix + (aPix - cPix) * (aPix - cPix)));
eig1 = (single(r) + s);
eig2 = (single(r) - s);

vx = single(0);
vy = single(0);
% thresh = single(196.6);  %98.3040*2
thresh = 0.0039;
% thresh2 = single(98.304);
thresh2 = 0;

if (eig1 > thresh && eig2 > thresh)
    d = bPix * bPix - aPix * cPix;
    iDelta = 1/d;
    vx =  single( -(tyPix * bPix - txPix * cPix)*iDelta);
    vy =  single( -(txPix * bPix - aPix * tyPix)*iDelta);
elseif (eig1 > thresh && eig2 < thresh)
    rr = bPix+aPix;
    cc = cPix+bPix;
    norm = cc*cc + rr*rr;

    if (norm > thresh2)
        invnorm = 1/norm;
        tmp = -(txPix + tyPix) * invnorm;
        vx =  single(rr * tmp);
        vy =  single(cc * tmp);
    end
end

VxPix = vx;
VyPix = vy;

end

Run the Model

The input video is split into a previous frame and current frame. The input ports of the DUT, CurrFrame and PrevFrame, connect to the 2-D matrices that contain the data for the current frame and previous frame. Each input signal is a frame composed of 360x640 pixels. Simulate the model to see the frame size and simulation results of the optical flow output.

sim("hdlFrame_OpticalFlow_LK_MLFB");

Generate HDL Code

Generate synthesizable HDL code by using the frame-to-sample conversion. Set the HDL block property ConvertToSamples on the Inport blocks of the DUT that connect to the frame-input signals to convert the input signals from frame-based to sample-based inputs.

hdlset_param('hdlFrame_OpticalFlow_LK_MLFB/DUT_LK/PrevFrame','ConvertToSamples','on');
hdlset_param('hdlFrame_OpticalFlow_LK_MLFB/DUT_LK/CurrFrame','ConvertToSamples','on');

For the MATLAB Function block that contains the optical flow algorithm, set the HDL block property Architecture to MATLAB Datapath. Enable the frame-to-sample conversion optimization and generate HDL code using the makehdl command. For more information on the frame-to-sample conversion optimization, see HDL Code Generation from Frame-Based Algorithms (HDL Coder).

hdlset_param('hdlFrame_OpticalFlow_LK_MLFB/DUT_LK/MATLAB Function LK','Architecture','MATLAB Datapath');
hdlset_param('hdlFrame_OpticalFlow_LK_MLFB','FrameToSampleConversion','on');
makehdl('hdlFrame_OpticalFlow_LK_MLFB/DUT_LK');

The frame-to-sample conversion separates the frame-based inputs into sample, valid, and ready signals for a sample-based hardware-targeted interface.

Model Algorithm with the Neighborhood Processing Subsystem Block

In the hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem model, the DUT contains multiple Neighborhood Processing Subsystem blocks that contain the neighborhood processing operations needed for the Lucas Kanade optical flow algorithm. Separating the optical flow algorithm into various Simulink blocks enables you to visualize and model the different aspects of the algorithm in a more modular way than with a single MATLAB Function block.

Open the hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem/DUT subsystem to see the optical flow algorithm.

load_system("hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem");
open_system("hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem/DUT");

Run the Model

The input video is split into a previous frame and current frame. The input ports of the DUT, CurrFrame and PrevFrame, connect to the 2-D matrices that contain the data for the current frame and previous frame. Each input signal is a frame composed of 360x640 pixels. Simulate the model to see the frame size and simulation results of the optical flow output.

sim("hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem");

Although the hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem and hdlFrame_OpticalFlow_LK_MLFB models differ in design, the output is the same. Both models can also generate synthesizable HDL code for sample-based hardware.

Generate HDL Code

Generate synthesizable HDL code by using the frame-to-sample conversion. Set the HDL block property ConvertToSamples on the Inport blocks of the DUT that connect to the frame-input signals to convert the input signals from frame-based to sample-based inputs.

hdlset_param('hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem/DUT/PrevFrame','ConvertToSamples','on');
hdlset_param('hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem/DUT/CurrFrame','ConvertToSamples','on');

Enable the frame-to-sample conversion optimization and generate HDL code using the makehdl command.

hdlset_param('hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem','FrameToSampleConversion','on');
makehdl('hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem/DUT');

The frame-to-sample conversion separates the frame-based inputs into sample, valid, and ready signals for a sample-based hardware-targeted interface.

See Also

(HDL Coder)

Related Topics