How to set up a deep neural network input so that it inputs both an image (X by Y greyscale) AND 5 numerical values for each output.

Hello, I am trying to train a deep neural network that classifies based on the data present in an [X by Y] grayscale image. However, prior to this, 5 regression values are being calculated from the image as well. I am trying to make a network that takes both the image, and the 5 regression values as inputs for each output.

I currently am using a table where the first column is the path to each image, and the second column is the classifier. However, I believe I learned that the trainNetwork function can take an arbitrary number of inputs paired with each output.

However, how should I format the network when the first column is an image file, and the next 5 are regression values, followed finally by the classifier? The image input layer can only take the image of course, so how do I also tell the network to input the 5 regression values?

I would really like to avoid breaking down the image into a sequence of values followed by the regression values for use in a sequenceInputLyaer - the images are large and there are hundreds of thousands of them.

Thanks for the advice!

Answer 1

[Niccolò Dal Santo]

Hi Thomas,

Currently trainNetwork does not support networks which have multiple heterogeneous inputs like your case (an image and a set of feature). However you can achieve what you want with a custom training loop.

This example might be helpful to start with: Train Network on Image and Feature Data

It trains a classification network with two heterogeneous inputs: a feature input (which can be used for your 5 regression values) and an image input.

Hope this helps.

Cheers,

Niccolò