How a varying PI parameter output by Reinforcement Learning Agent help to tune the static PI controller

Tune PI Controller using Reinforcement Learning - MATLAB & Simulink - MathWorks Benelux

I read this example and I am confused by the methodologic of this algorithme.

When I train the agent, the PI controller parameters represented by neural network will change every sampletime (0.1s for this example) to find the best setting. After the training, we can deploy the parameters to our PI controller with the parameter areadly fixed.

Why is this reasonable ? For example, in the dynamic process, when the error is large, the agent will tend to increase the parameters of the neural network, and when it converges, the agent will tend to reduce the parameters. The agent can find control the system well with a small cumulate error, but a PI controller with the parameters fixed can not perform like the agent.

[Sam Chak]

Why it is reasonable ?

Hi @轩, I'm unsure if I understand. What is "it" being reasonable to you? Are you referring to the increment and reduction of the values of the parameters?

[轩]

@Sam Chak Thank you for your comment. I am referring the online parameters tuning doing by the agent, I think it is unreasonable.

Answer 1

[Sam Chak]

Hi @轩

I believe it is scientifically logical that, when the difference between the actual level and the reference level (error) is substantial, the agent will tend to increase the control effort (resulting in a larger water flow to fill up the tank faster). Conversely, as the actual level approaches the reference level, the agent will tend to reduce the control effort (resulting in a smaller water flow to prevent overflow). This behavior is observed in both CST-tuned PI control and TD3-tuned PI control, as illustrated below.

In fact, the Control System Tuner (CST) performs an iterative tuning of the PI gains to meet tracking and stability requirements, while the TD3 agents undergo an iterative learning process to update their policy, maximizing the expected cumulative reward in the given environment.

Kp_CST  = 9.80199999804512;
Ki_CST  = 1.00019996230706e-06; 
Kp_TD3  = 8.0822;
Ki_TD3  = 0.3958;

[轩]

Hello Sam !

I think I understand what you said, but what confuses me is that: when the agent is trained, the PI parameters are regulated every 0.1s. However, when the PI setting is deployed in actual controller, it is fixed, it can't change every 0.1s.

After several days thinking, I think I have figuered it out, and I'd like to talk about my thinking:

The actor in the agent is changing its parameters, that is doing a gradient ascent to find the optimal point of the critic network, instead of finding the best action to handle this specific state. And the latter thing, finding the best action according to the specific state is what critic supposes to do. After training, critic network marks down the good and bad of different actions in different states, and actor network is supposed to find out the optimal parameters setting to the network, that means the most appropriate, the most suitable and compromise parameters to all condition (system's operating point).

[Luka]

Hello everyone,

I understood what @轩 was trying to say, and I have the same question. In the given example, shouldn't the network parameters (PI gains) be adjusted only after one episode of training has been completed, and not during the episode itself?

Because in reality, the controller will work with fixed gains (which we want to determine using RL), so I think it is correct to change these gains during the training process only after the individual episode is finished and after the reward for that particular episode is received.

[轩]

@Luka Thank you for you comment, I have a new comprehension after posting this question.

Copy one more time here:

The actor in the agent is changing its parameters, that is doing a gradient ascent to find the optimal point of the critic network, instead of finding the best action to handle this specific state. And the latter thing, finding the best action according to the specific state is what critic supposes to do. After training, critic network marks down the good and bad of different actions in different states, and actor network is supposed to find out the optimal parameters setting to the network, that means the most appropriate, the most suitable and compromise parameters to all condition (system's operating point).