How to set initial estimate for mean in PPO actor critic network

I am using a PPO actor critic network. I created the actor following this example

https://www.mathworks.com/help/reinforcement-learning/ref/rl.function.rlcontinuousgaussianactor.html

How can I set an initial guess for the mean? Currently the actor always starts with an intial mean at time zero of zero.

Answer 1

[Aneela]

Hi Jason Butler,

To set an initial guess for the mean in PPO actor network, modify the initial weights or biases of the layers that contribute to calculating the mean.

• Set an initial guess for the mean in the bias of the “fullyConnectedLayer” in the mean calculation.
• However, because of the non-linearities like the “tanhLayer”, directly setting the bias to achieve a specific mean after scaling and non-linear transformations can be complex.

Assuming the desired initial mean as 5, here’s a workaround:

desiredInitialMean = 5; % Adjust this value as needed
% Since you have 3 actions, create a bias vector with 3 elements
biasForDesiredMean = repmat(desiredInitialMean / actInfo.UpperLimit, [prod(actInfo.Dimension), 1]);
% Modify the meanPath definition to include the bias initialization as a vector
meanPath = [ 
    tanhLayer(Name="tanhMean");
    fullyConnectedLayer(prod(actInfo.Dimension), ...
    'Bias', biasForDesiredMean, ... 
    Name="fcMean");
    scalingLayer(Name="scale", ...
    'Scale', actInfo.UpperLimit)
];

For more information on “Bias” in the “fullyConnectedLayer”, refer to the following MathWorks documentation: https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.fullyconnectedlayer.html?s_tid=doc_ta#:~:text=Layer%20biases%2C%20specified,single%20%7C%20double