Training a Variational Autoencoder (VAE) on sine waves

Hi,

I am trying to run a variational autoencoder according to the script in https://se.mathworks.com/help/deeplearning/examples/train-a-variational-autoencoder-vae-to-generate-images.html

Basically, I am testing the autoencoder on sine waves. I have a training set and a testing set each having 100 sine waves of length 1100 samples (they are all similar). However, when I try to run the code, I get the following error:

Error using nnet.internal.cnn.dlnetwork/forward (line 194)

Layer 'fc_encoder': Invalid input data. The number of weights (17600) for each output feature must match the number of elements (204800) in each observation

of the first argument.

Error in dlnetwork/forward (line 165)

[varargout{1:nargout}] = forward(net.PrivateNetwork, x, layerIndices, layerOutputIndices);

Error in sampling (line 2)

compressed = forward(encoderNet, x);

Error in modelGradients (line 2)

[z, zMean, zLogvar] = sampling(encoderNet, x);

Error in deep.internal.dlfeval (line 18)

[varargout{1:nout}] = fun(x{:});

Error in dlfeval (line 40)

[varargout{1:nout}] = deep.internal.dlfeval(fun,varargin{:});

Error in ConvAE (line 57)

[infGrad, genGrad] = dlfeval(...

When I run the same code with XBatch = XTrain instead, I get the same error but with number of elements 440000 instead of 204800.

When I run the same code with XBatch = XTrain(idx,:) instead, I get the error: Index in position 1 exceeds array bounds (must not exceed 100).

Can anyone help? I have used the exact same Helper Functions as in the link.

Thanks!

latentDim = 50;

encoderLG = layerGraph([
    imageInputLayer([1 1100],'Name','input_encoder','Normalization','none')
    convolution2dLayer([1 100], 32, 'Padding','same', 'Stride', 2, 'Name', 'conv1')
    reluLayer('Name','relu1')
    convolution2dLayer([1 100], 64, 'Padding','same', 'Stride', 2, 'Name', 'conv2')
    reluLayer('Name','relu2')
    fullyConnectedLayer(2 * latentDim, 'Name', 'fc_encoder')
    ]);

decoderLG = layerGraph([
    imageInputLayer([1 1 latentDim],'Name','i','Normalization','none')
    transposedConv2dLayer([1 100], 32, 'Cropping', 'same', 'Stride', 2, 'Name', 'transpose1')
    reluLayer('Name','relu1')
    transposedConv2dLayer([1 100], 64, 'Cropping', 'same', 'Stride', 2, 'Name', 'transpose2')
    reluLayer('Name','relu2')
    transposedConv2dLayer([1 100], 32, 'Cropping', 'same', 'Stride', 2, 'Name', 'transpose3')
    reluLayer('Name','relu3')
    transposedConv2dLayer([1 100], 1, 'Cropping', 'same', 'Name', 'transpose4')
    ]);

encoderNet = dlnetwork(encoderLG);
decoderNet = dlnetwork(decoderLG);

executionEnvironment = "auto";

XTrain = sineTrain;
XTest = sineTest;


numTrainImages = 1100;

numEpochs = 50;
miniBatchSize = 512;
lr = 1e-3;
numIterations = floor(numTrainImages/miniBatchSize);
iteration = 0;

avgGradientsEncoder = [];
avgGradientsSquaredEncoder = [];
avgGradientsDecoder = [];
avgGradientsSquaredDecoder = [];

for epoch = 1:numEpochs
    tic;
    for i = 1:numIterations
        iteration = iteration + 1;
        idx = (i-1)*miniBatchSize+1:i*miniBatchSize;
        XBatch = XTrain(:,idx);
        XBatch = dlarray(single(XBatch), 'SSCB');
        
        if (executionEnvironment == "auto" && canUseGPU) || executionEnvironment == "gpu"
            XBatch = gpuArray(XBatch);           
        end 
            
        [infGrad, genGrad] = dlfeval(...
            @modelGradients, encoderNet, decoderNet, XBatch);
        
        [decoderNet.Learnables, avgGradientsDecoder, avgGradientsSquaredDecoder] = ...
            adamupdate(decoderNet.Learnables, ...
                genGrad, avgGradientsDecoder, avgGradientsSquaredDecoder, iteration, lr);
        [encoderNet.Learnables, avgGradientsEncoder, avgGradientsSquaredEncoder] = ...
            adamupdate(encoderNet.Learnables, ...
                infGrad, avgGradientsEncoder, avgGradientsSquaredEncoder, iteration, lr);
    end
    elapsedTime = toc;
    
    [z, zMean, zLogvar] = sampling(encoderNet, XTest);
    xPred = sigmoid(forward(decoderNet, z));
    elbo = ELBOloss(XTest, xPred, zMean, zLogvar);
    disp("Epoch : "+epoch+" Test ELBO loss = "+gather(extractdata(elbo))+...
        ". Time taken for epoch = "+ elapsedTime + "s")    
end

Answer 1

[Joss Knight]

It looks like your input data size is wrong. Your formatting says that the 4th dimension is the batch dimension, but actually it appears that the batch dim is the second dimension. You could try labelling as 'SBCS' instead, but I can't be sure because I don't know what's in sineTrain. It may be you need to permute your data or all your filters to match the convolution dimension to the correct input dimension.

[Uerm]

Hi, thanks for the answer. I will give it a try. sineTrain and sineTest are both 100x1100 doubles, 100 identical sine waves of length 1100 samples. Regarding the 4th dimension... Does it mean that the training and testing data should be 4D arrays?

[Joss Knight]

No, but I'm trying to label your data in a way that your network will understand. You have created a 2-D convolutional layer that expects to see data with 2 spatial dimensions and a channel dimension. So I've labelled two trailing singleton dimensions with C and S. However, you probably also need to permute your two spatial dimensions, since your convolution kernels are row vectors, which means, I believe, that they are expecting your first spatial dimension to be singleton. dlarray can't do that for you on construction so you'll need to force that by permuting or reshaping data. You could just adapt the line where you create your dlarray to

XBatch = dlarray(reshape(single(XBatch), 1, size(XBatch,1), 1, size(XBatch,2)), 'SSCB');

That should do the trick.

[Uerm]

Hi Joss,

I have just tried with the line above but it still gives me this error:

Error using nnet.internal.cnn.dlnetwork/forward (line 194)

Layer 'fc_encoder': Invalid input data. The number of weights (17600) for each output feature must match the number of elements (1600) in each observation of

the first argument.

Can you see where the problem is?

When I try to run their script with the same data, I can see that XTrain is a 4-D dlarray with size(XTrain) = 28 28 1 60000.

[Joss Knight]

You say sineTrain is 100-by-1100 so clearly it is dimension 1 that is the batch dimension. But you are indexing into dimension 2 to create your mini-batch. So you need to change that. Then you can change the line I gave you to adapt to this as appropriate.

[Uerm]

Hi again,

I have done that now, but I run into another problem now: Array dimensions must match for binary array op.

The error occurs in the ELBOloss Helper Function. When inspecting the code, size(xPred) = 8 8 1 100 and size(sineTrain) = 1 1100 1 100. I changed sineTrain to a 4D array according to your code above. I don't quite see how the 8x8 part in xPred occurs... Do you know why? And how should I then change the network structure?

[Joss Knight]

Adapt your indexing code to index the 4th dimension instead of the 2nd. Return the code that creates the dlarray input to what it was before. Modify sineTest the same way as you did sineTrain. Use the debugger to trace the shape of variables through your code.

If you need someone to interactively help you fix your code, you might consider phoning MathWorks technical support.

[Uerm]

Hi, I have done the things you suggested and it seems to be able to run. Thank you for that! Unfortunately, I do not have access to the technical support. However, there is something wrong with the training part. The ELBO loss is identical through all the epochs which is strange. When I run the code using the same data from the mathworks page (MNIST database), it works and the ELBO loss is reduced through the epochs. But that does not work when I run the code with my own data. I attach two of the codes. Sinus.m is the code for the sine waves that I use as my data. ConvAE.m is the main code. I have added two more layers and adjusted some numbers to match with my own data. I have not attached the other helper functions as they are identical to the ones on the webpage.

Can you spot the problem?

[Joss Knight]

I suggest you ask a new question, to see if anyone wants to help you with this new problem.