Triggered Subsystem for RL Agent possible?

In my system I am comparing target and current RPM. Starting from a defined deviation, 10 time steps are to be taken to generate a state vector from it; this is at an arbitrary point in time x. At time x the state vector is to serve the RL agent as observation.

The reward, the summed value of the target speed minus the current speed (Sum(|n_ref - n_act|)) shall be transmitted to the RL agent at the end of the episode, i.e. at time y.

How can I implement this intention? So:

1. How can I trigger the RL agent at that very point in time? (triggered subsystem?)

2. Is the RL agent waiting for a reward? Or does the agent take the present value at the end of the sample time?

3. How does the RL agent then receive the reward within a triggered subsystem?

Does the agent wait for a reward that differs from the default value (which in my case is 0) before ending the actual step?

Many thanks in advance!

Answer 1

[Hari]

Hi Tobias,

I understand that you want to implement a triggered subsystem in MATLAB/Simulink for your RL agent and you want to trigger the RL agent at a specific point in time, generate a state vector, and provide a reward to the agent at the end of an episode.

I am assuming you are using Reinforcement Learning Toolbox and have already setup your RL environment.

1. To implement a triggered subsystem to trigger the RL agent at a specific point in time, you can use an event-based trigger or a custom event. Here is an example of how you can create a custom event:

% Define the custom event function
function eventTrigger(block)

% Define your custom logic to trigger the agent at time x
if (current_time == x)
    block.Events = [];
    rl.agentStep(); % Trigger the RL agent step
end
end

2. The RL agent typically waits for a reward signal to update its policy. By default, the agent uses a value of 0 as the baseline reward. When you provide a non-zero reward, the agent learns from it.

3. To provide the reward within a triggered subsystem, you can use the same custom event approach as mentioned in (1). At time 'y', you can update the reward and transmit it to the RL agent. Here's an example:

% Define the custom event function to provide a reward at time y
function eventTrigger(block)

% Define your custom logic to trigger the agent at time y
if (current_time == y)
    block.Events = [];
    reward = calculateReward(); % Calculate the reward
    rl.setStepResult(reward);  % Provide the reward to the agent 
end
end

By implementing custom event triggers in your Simulink model as described, you can precisely control your RL agent to get it triggered at the time ‘x’ and receive rewards for the actions taken at time ‘y’.

Refer to the documentation of custom events in Simulink for information on how to create custom event triggers. https://www.mathworks.com/help/simulink/ug/define-custom-events.html

For information on how to work with RL agents, states, rewards, and steps, refer to the documentation of the Reinforcement Learning Toolbox in MATLAB.

https://www.mathworks.com/help/reinforcement-learning/index.html

Hope this helps!