UAV Toolbox - How to run a Simulink-Unreal cosimulation with thermal (IR) cameras?

Hello,

I want to start working on a project where I am aiming to simulate a UAV with a thermal (IR) camera payload in a Simulink and Unreal Engine cosimulation environment and the UAV Toolbox really impressed me in some regards. I understand that both Simulink and Unreal natively support standard and fisheye cameras (including depth cameras and camera segmentation for ML), however I have not found built-in support for thermal cameras.

My question is: Is it possible to integrate thermal cameras into this cosimulation workflow? If so, how can I implement or simulate the thermal imaging of the UAV's camera payload in this environment? Is this something that would be more easily coded within the Unreal Engine scene (if it is even possible), or would it be better to tackle this from the Simulink side, potentially with the help of additional toolboxes, custom code?

Additionally, I assume that at least the thermal data of the scene actors would need to be passed somehow to Simulink for accurate simulation and control, since I want to have an accurate model. Could you clarify the best way to handle the thermal information in this cosimulation context?

Any insights, references, or suggestions would be greatly appreciated. Thank you.

Answer 1

[Nishan Nekoo]

Hi Eugenio,

Unfortunately there is no native support for thermal imaging in Unreal Engine and by extension in Simulink 3D Animation. If this were to be done using workarounds and custom post-processed materials, it would likely need to be tackled on the Unreal side instead of through Simulink.

I do think that getting an accurate model will be very challenging - since Unreal Engine is primarily for gaming applications, any notion of "thermal imaging" will only be done through applying custom materials/textures. To get something approximate, there are tutorials on YouTube if you search "Unreal Thermal Vision". Once again, Unreal itself has no notion of thermal energy or heat transfer, so anything built will likely be very game-like. It will take a lot of effort to create something realistic using thermal modeling capabilities in Simulink and translating that into custom materials and textures in Unreal.

If you do go down the "approximate" route, note that thermal images captured can still be sent to Simulink for any post-processing.

I hope that helps.

Nishan