Depth and Semantic Segmentation Visualization Using Unreal Engine Simulation
This example shows how to visualize depth and semantic segmentation data captured from a camera sensor in a simulation environment. This environment is rendered using the Unreal Engine® from Epic Games®.
You can use depth visualizations to validate depth estimation algorithms for your sensors. You can use semantic segmentation visualizations to analyze the classification scheme used for generating synthetic semantic segmentation data from the Unreal Engine environment.
The model used in this example simulates a vehicle driving in a city scene.
A Simulation 3D Scene Configuration block sets up simulation with the US City Block scene.
A Simulation 3D UAV Vehicle block specifies the driving route of the vehicle.
A Simulation 3D Camera block mounted to the quadrotor captures data from the flight. This block outputs the camera, depth, and semantic segmentation displays by using To Video Display (Computer Vision Toolbox) (Computer Vision Toolbox) blocks.
A depth map is a grayscale representation of camera sensor output. These maps visualize camera images in grayscale, with brighter pixels indicating objects that are farther away from the sensor. You can use depth maps to validate depth estimation algorithms for your sensors.
The Depth port of the Simulation 3D Camera block outputs a depth map of values in the range of 0 to 1000 meters. In this model, for better visibility, a Saturation block saturates the depth output to a maximum of 150 meters. Then, a Gain block scales the depth map to the range [0, 1] so that the To Video Display block can visualize the depth map in grayscale.
Semantic Segmentation Visualization
Semantic segmentation describes the process of associating each pixel of an image with a class label, such as road, building, or traffic sign. In the 3D simulation environment, you generate synthetic semantic segmentation data according to a label classification scheme. You can then use these labels to train a neural network for UAV flight applications, such as landing zone identification. By visualizing the semantic segmentation data, you can verify your classification scheme.
The Labels port of the Simulation 3D Camera block outputs a set of labels for each pixel in the output camera image. Each label corresponds to an object class. For example, in the default classification scheme used by the block,
1 corresponds to buildings. A label of
0 refers to objects of an unknown class and appears as black. For a complete list of label IDs and their corresponding object descriptions, see the Labels port description on the Simulation 3D Camera block reference page.
The MATLAB® Function block uses the
label2rgb (Image Processing Toolbox) function to convert the labels to a matrix of RGB triplets for visualization. The colormap is based on the colors used in the CamVid dataset, as shown in the Semantic Segmentation Using Deep Learning (Computer Vision Toolbox) example. The colors are mapped to the predefined label IDs used in the default Unreal Engine simulation scenes. The helper function
sim3dColormap defines the colormap. Inspect these colormap values.
Run the model.
When the simulation begins, it can take a few seconds for the visualization engine to initialize, especially when you are running it for the first time. The
AutoVirlEnv window displays the scene from behind the UAV. In this scene, the quadrotor UAV flies a short distance down one city block.
The Camera Display, Depth Display, and Semantic Segmentation Display blocks display the outputs from the camera sensor.
To change the visualization range of the output depth data, try updating the values in the Saturation and Gain blocks.
To change the semantic segmentation colors, try modifying the color values defined in the
sim3dColormap function. Alternatively, in the
label2rgb MATLAB Function block, try replacing the input colormap with your own colormap or a predefined colormap. See
Simulation 3D Scene Configuration | Simulation 3D Camera | Simulation 3D UAV Vehicle