The reward is reduced after certain episode on the DQN
11 ビュー (過去 30 日間)
古いコメントを表示
I don't know why the reward is reduced after 240episode.
I attach episode training progress and code.
my state(observation) is 2, continous and action space is 5,
anyone who knows what the problem is, please advise me why the algorithm is not working properly
nI = observationInfo.Dimension(1); % number of inputs
nL1 = 12;
nL2 = 24;
nL3 = 24; % number of neurons
nO = numel(actionInfo.Elements); %11
dnn = [
featureInputLayer(nI,'Normalization','none','Name','state')
fullyConnectedLayer(nL1,'Name','fc1')
reluLayer('Name','relu1')
fullyConnectedLayer(nL2,'Name','fc2')
reluLayer('Name','relu2')
fullyConnectedLayer(nL3,'Name','fc3')
reluLayer('Name','relu3')
fullyConnectedLayer(nO,'Name','out')];
%
% figureDFQ
% plot(layerGraph(dnn))
criticOptions = rlRepresentationOptions('LearnRate',0.0001,'GradientThreshold',1, 'UseDevice', "gpu");
critic = rlQValueRepresentation(dnn,observationInfo,actionInfo,'Observation',{'state'}, criticOptions);
agentOptions = rlDQNAgentOptions(...
'SampleTime', Ts,...
'UseDoubleDQN', true,...
'TargetSmoothFactor',1e-2,...
'TargetUpdateFrequency', 20,...
'DiscountFactor',0.99,... % 0.
'ExperienceBufferLength',1e8);
agentOptions.EpsilonGreedyExploration.EpsilonMin = 0.01;
agentOptions.EpsilonGreedyExploration.EpsilonDecay = 0.0001;
agentOptions.EpsilonGreedyExploration.Epsilon = 1;
agent = rlDQNAgent(critic,agentOptions);
%% Train agent
maxsteps = ceil(trun/Ts);
trainingOpts = rlTrainingOptions(...
'MaxEpisodes',maxepisodes,...
'MaxStepsPerEpisode',maxsteps,...
'ScoreAveragingWindowLength',5,...
'Verbose',true,... % display training porgress in the command line
'UseParallel',false,...
'StopTrainingCriteria','EpisodeCount',... % AverageReward, EpisodeCount
'StopTrainingValue',300,...
'SaveAgentCriteria','EpisodeCount',...
'SaveAgentValue', 300,...
'SaveAgentDirectory', "/Agent_5ac2ob_.mat",'agent');
% 'Plots','training-progress',...
trainingStats = train(agent,env,trainingOpts);
0 件のコメント
回答 (1 件)
Emmanouil Tzorakoleftherakis
2020 年 11 月 10 日
Hello,
There is no guarantee that the reward will keep going up always when using RL (after all there is certain stochastisity involved when exploring). I would recommend stopping training at the peak and check if the agent is good enough. If it's not then you would need to increase your exploration settings (maybe use smaller epsilon decay value and larger min epsilon value).
0 件のコメント
参考
カテゴリ
Help Center および File Exchange で Training and Simulation についてさらに検索
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!