Create and Train DQN Agent with just a State Path and Not Action Path

Every example I have seen of a DQN on MATLAB is with two inputs, the state and action. However, it is possible for DQN RL to be done with just one input, the state but there are no examples for that case. How can that be done on MATLAB? My Input would basically be a binary vector and my output would be that I can do two actions?

Basically I am trying to recreate this: http://cwnlab.eecs.ucf.edu/wp-content/uploads/2019/12/2019_MLSP_ANCS_NAZMUL.pdf

Answer 1

[Emmanouil Tzorakoleftherakis]

Hello,

This page shows how this can be done in 20a. We will have examples that show this workflow in the next release.

Hope that helps.

[Huzaifah Shamim]

Thank You Emmanouil! I have follow up regarding that page. When its says actInfo = rlFiniteSetSpec([7 5 3]), I understand that means we have 3 possible actions and we have named them that. In the environment then, would we set up something like: i

If action == 7:
    do this
else if action == 5: 
    do this
else 
    do this
end

[Emmanouil Tzorakoleftherakis]

Sounds right. Nothing changes on the definition of you action space - what changes is the structure of your neural net, i.e., you do not need a separate input channel for actions, but you need to change the number of neurons in the last layer of the critic to be equal to the #of actions you have.

[Huzaifah Shamim]

Ah perfect thank you so much. One last question which kind of derails from the topic but still has to do with RL and NN. If you could answer it, I would greatly appreciate it.

How would I approach a problem where I want to have K different actions each with p possible values. An example of this I was thinking would be something like having a GridWorld Environment with Two Agents exploring the maze? Is there a link or resource I should check out for that (not necessarily grid world Env, but multiple actions each with multiple values).

[Emmanouil Tzorakoleftherakis]

If you are looking for multiple agents training simultaneously (ori multi-agent RL), this functionality is coming in 20b.

If you are looking to train a single agent that can choose between combinations of multiple discrete actions, this example shows how you would do it.

Hope that helps.

[Huzaifah Shamim]

So currently (in 20a) is it not possible to do something like this where we have multiple actions and potentially multiple agents?

Additionally is parametrized actions something that will also be coming to 20b?

Sorry for barraging you with questions, and thank you for your answers.

[Emmanouil Tzorakoleftherakis]

We are looking into parameterized actions but I do not have a timeline for that.

The link you shared I believe only refers to multiple discrete actions for a single agent, which is already doable (the link I sent shows how to do that). Multiple agents training simultaneously is a different story.

In 20a you would probably have to implement your own custom training loop for multi-agent RL.

[Huzaifah Shamim]

Ok perfect I will look into the custom training loop. Is there any advice you have regarding that (to adjust/configure it for multiple actions)?

A question based off of that is also: Could I treat my agent as one (even if many) but it have multiple actions related to it, and based off of that it could be a sort of multi agent problem?

[Emmanouil Tzorakoleftherakis]

This sounds doable.You may even be able to do this without custom loops using built-in agents (something like centralized multi-agent RL). You can use a single agent and at each step extract the appropriate action and apply it to the appropriate part of the environment. The tricky part is (typical of multi-agent RL) to pick the right amount of observation to make sure your process is Markov. This will likely require observations from each 'subagent' etc.

[Huzaifah Shamim]

Oh ok interesting.

One last question (Hopefully), in this link, LoggedSignals is used as the State. However lets say I wanted LoggedSignals to contain three matrices or three vectors, one of them being the actual state. Could I contain those three matrices in LoggedSignal and then just let observation have the actual next state. Additionally if that is allowed, how would I define my observation info, would I use rlFiniteSetSpec or rlNumericSpec (My offical state would just be a binary vector of size N but I want to have two other matrices, x and y lets say, so that I can utlilze them in myStepFunction.m)