PPO agent applied to ACC model

I'm applying the PPO algorithm to the ACC model in this DDPG example:Train DDPG Agent for Adaptive Cruise Control

How ever the agent went wrong at 50th episode with error:

Error using rl.env.AbstractEnv/simWithPolicy (line 70)

An error occurred while simulating "rlACCMdl" with the agent "agent".

Error in rl.task.SeriesTrainTask/runImpl (line 33)

[varargout{1},varargout{2}] = simWithPolicy(this.Env,this.Agent,simOpts);

Error in rl.task.Task/run (line 21)

[varargout{1:nargout}] = runImpl(this);

Error in rl.task.TaskSpec/internal_run (line 159)

[varargout{1:nargout}] = run(task);

Error in rl.task.TaskSpec/runDirect (line 163)

[this.Outputs{1:getNumOutputs(this)}] = internal_run(this);

Error in rl.task.TaskSpec/runScalarTask (line 187)

runDirect(this);

Error in rl.task.TaskSpec/run (line 69)

runScalarTask(task);

Error in rl.train.SeriesTrainer/run (line 24)

run(seriestaskspec);

Error in rl.train.TrainingManager/train (line 291)

run(trainer);

Error in rl.train.TrainingManager/run (line 160)

train(this);

Error in rl.agent.AbstractAgent/train (line 54)

TrainingStatistics = run(trainMgr);

Caused by:

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 689)

Invalid input argument type or size such as observation, reward, isdone or loggedSignals.

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 689)

Standard deviation must be nonnegative. Ensure your representation always outputs nonnegative values for outputs that correspond to the standard deviation.

Is there anybody could solve this problem? Thanks for your help!

[qun wang]

您好，请问这个问题你解决了嘛？我最近也在用PPO遇到类似的问题，可否讨论一下？期待回复

Answer 1

[Emmanouil Tzorakoleftherakis]

Hello,

Can you make sure that you set up your actor following a structure similar to this one? It seems that your variance path is not set up properly and gives negative values.