Create policy evaluation function for RL agent

Hello,

I'm using the Reinforcement Learning Toolbox to train an agent for a particular task. Once the training is complete, I want to generate a policy evaluation function so I can use it in a simulink file. I can create the policy function using "generatePolicyFuncion" command. The matlab script that is generated is the following:

function action1 = evaluatePolicy(observation1)
%#codegen
 
% Reinforcement Learning Toolbox
% Generated on: 23-Sep-2019 09:27:39
 
action1 = localEvaluate(observation1);
end
%% Local Functions
function action1 = localEvaluate(observation1)
persistent policy
if isempty(policy)
                policy = coder.loadDeepLearningNetwork('agentData.mat','policy');
end
action1 = predict(policy,observation1);
end

When I run the simulink file, the following errors pop up:

Simulink does not have enough information to determine output sizes for this block. If you think the errors below are inaccurate, try specifying types for the block inputs and/or sizes for the block outputs.

Component:MATLAB Function | Category:Coder error

Dot indexing is not supported for variables of this type.

Function 'MATLAB Function' (#171.286.341), line 13, column 11:

"coder.loadDeepLearningNetwork('agentData.mat','policy')"

Launch diagnostic report.

Component:MATLAB Function | Category:Coder error

Persistent variable 'policy' must be assigned before it is used. The only exception is a check using 'isempty(policy)' that can be performed prior to assignment.

Function 'MATLAB Function' (#171.365.371), line 15, column 19:

"policy"

Launch diagnostic report.

Component:MATLAB Function | Category:Coder error

Function call failed.

Function 'MATLAB Function' (#171.140.167), line 7, column 11:

"localEvaluate(observation1)"

Launch diagnostic report.

Component:MATLAB Function | Category:Coder error

Errors occurred during parsing of MATLAB function 'test_RL/MATLAB Function'

Component:MATLAB Function | Category:Coder error

Simulink cannot determine sizes and/or types of the outputs for block 'test_RL/MATLAB Function' due to errors in the block body, or limitations of the underlying analysis. The errors might be inaccurate. Fix the indicated errors, or explicitly specify sizes and/or types for all block outputs.

Component:MATLAB Function | Category:Coder error

Dot indexing is not supported for variables of this type.

Function 'MATLAB Function' (#171.286.341), line 13, column 11:

"coder.loadDeepLearningNetwork('agentData.mat','policy')"

Launch diagnostic report.

Component:MATLAB Function | Category:Coder error

Persistent variable 'policy' must be assigned before it is used. The only exception is a check using 'isempty(policy)' that can be performed prior to assignment.

Function 'MATLAB Function' (#171.365.371), line 15, column 19:

"policy"

Launch diagnostic report.

Component:MATLAB Function | Category:Coder error

Function call failed.

Function 'MATLAB Function' (#171.140.167), line 7, column 11:

"localEvaluate(observation1)"

Launch diagnostic report.

Component:MATLAB Function | Category:Coder error

Errors occurred during parsing of MATLAB function 'test_RL/MATLAB Function'

How can I solve this issue?

Thanks

Answer 1

[Emmanouil Tzorakoleftherakis]

Can you try defining the size of inputs and outputs in the MATLAB Function block? This seems to be coming up a lot in the error message

[Juan Nunez]

Hello Emmanouil,

I'm sorry for the late reply. I tried your suggestion but I'm still getting the same error. Please see attached screenshot.

Thanks.

[J]

Hi there,

Were you able to resolve this issuse? I am facing the same problem as you and haven't been able to figure it out...

[SomeMatlabUser]

Any Update? I am having the same problem. Also there is no documentation by Mathworks itself how to setup an Simulinkmodel executing the policy for code generation.

Answer 2

[Sviatoslav Klos]

Hello.

I experienced the same issue. If you do not need code generation (only normal mode simulation is requested) I might suggest to substitute your two functions with the following code and insert it as a MATLAB User-defined fcn. Please, respond if it suits you.

function action1 = evaluatePolicy(observation1)%, policy) %#codegen
coder.extrinsic('evalin', 'predict');
action1 = single(0);
policy = evalin('base', 'policy') ;
action1 = predict(policy,observation1);
end

[Juan Nunez]

Hello Sviatoslav,

Thnk your for your response.

I'm sorry for my late reply. I tried your suggestion, but the following error keeps popping up:

"Variable 'action1' has conflicting type definitions. To resolve, verify that all preceding definitions of 'action1' are consistent.

Function 'MATLAB Function' (#420.371.409), line 14, column 1:

"action1 = predict(policy,observation1)"

Launch diagnostic report.

Component:MATLAB Function | Category:Coder error

Errors occurred during parsing of MATLAB function 'Copy_of_RF_FOC6/MATLAB Function'

Component:MATLAB Function | Category:Coder error

Errors occurred during parsing of MATLAB function 'Copy_of_RF_FOC6/MATLAB Function'"

[Sviatoslav Klos]

I think this is your project specific error. You might check the action of the RL agent. I might suggest to test this function with some RL example from MATLAB and the try to go into your application with obtained conclusions.