A single hidden layered NN but the weight matrix is a diagonal matrix (with a bias) -- how to design that?

I am wondering if I can create a single hidden layered NN but the weight matrix is a diagonal matrix (with a bias). How to design that?

---------------------------------------------------------

Background

Let's say that a single layered hidden model (with m neurons) is given as

, where

• is input vector,
• is a first weight matrix connecting input with the hidden layer with m nodes,
• is a bias
• is an activation function which operates component wise, say $\tanh$
• is second weight matrix connecting the hidden layer with the output,
• is a bias, and
• is an output vector.

My model wants the weight matrices and as only diagonal matrices. How to do that?

Answer 1

[Srivardhan Gadila]

Refer to Weight and Bias Values, Input Weights, Layer Weights & net.layerWeights{i,j}.learn.

You can access the layer weights as follows:

net.LW{i,j}

You can set any values to the above weights and set the net.layerWeights{i,j}.learn to 0 so that the weights won't be altered during the training & adaption. In this case setting a specific weight for a connection is not possible since the property net.layerWeights{i,j}.learn is defined for the entire connections between layers i and j.

net.layerWeights{i,j}.learn = 0
net.LW{i,j} = ones(size(net.LW{i,j})) % any weights of size(net.LW{i,j})

If your network architecture is defined and trained already:

Then you can set weight of a connection between nodes k & l of layers i & j as follows:

net.LW{i,j}(k,l) = 1

and then use the network.

The above things can be done to Input wieghts too.

[Shashi Kant]

Thank you for your reply. I am a little confused with your suggestion.

When we set

net.layerWeights{i,j}.learn = 0

does this mean that the "weights" will not be adapted during learning?

Example, the size(net.IW{1}) == [m, n].My intention is to adapt/learn only the diagonal components of the weight matrix, that is, adapt only diag(net.IW{1}) and similarly for net.LW{2}.

I fail to see how your proposal would only adapt the diagonal elements of the weight matrices. Please suggest. Thank you in advance.

[Shashi Kant]

Would it be possible to create a toy example? Thank you.

[Srivardhan Gadila]

@Shashi Kant, as per my knowledge on shallow neural networks:

1. I listed the possible things you can do w.r.t the weights of layers of shallow nerual networks in the Answer.
2. The property net.layerWeights{i,j}.learn is defined for the entire connections between layers i and j hence you cannot set the diagonal weights to learn only & non-diagonal weights to not learn.
3. You can instead define custom Deep Learning layer to achieve your functionality. Refer to Define Custom Deep Learning Layers & Define Custom Deep Learning Layer with Learnable Parameters

[Shashi Kant]

@Srivardhan Gadila Thank you for your suggestion. I will look into that. I am not very familar with the deep learning toolbox. But I will dig into it.