rlFiniteSetSpec

Create specifications object for a finite-set action or observation channel

Description

An rlFiniteSetSpec object contains specifications for a channel that carries an action or observation belonging to a finite set.

Creation

Syntax

spec = rlFiniteSetSpec(elements)

spec = rlFiniteSetSpec(elements,Name=Value)

Description

spec = rlFiniteSetSpec(elements) creates a data specification object for a finite-set action or observation channel, setting the Elements property.

spec = rlFiniteSetSpec(elements,Name=Value) creates the specification object spec and sets its Properties using one or more name-value arguments.

Properties

Elements — Set of valid actions or observations

vector | cell array

Set of valid actions or observations for the environment, specified as one of the following:

• Vector — Specify valid numeric values for a single action or single observation.

• Cell array — Specify valid numeric value combinations when you have more than one action or observation. Each entry of the cell array must have the same dimensions.

Example: Elements=[-2 -1 0 1 2]

Name — Name of the rlFiniteSetSpec object

string

Name of the rlFiniteSetSpec object, specified as a string. Use this property to set a meaningful name for the signal carried by this data channel. This property is used by the RL Agent block to match the bus signals with their corresponding environment channels.

Example: Name="Action"

Description — Description of the rlFiniteSetSpec object

string

Description of the rlFiniteSetSpec object, specified as a string. You can use this property to specify a meaningful description of the signal carried by this environment channel.

Example: Description="Applied force in N"

Dimension — Size of each element

vector (default)

This property is read-only.

Size of each element, specified as a vector.

If you specify Elements as a vector, then Dimension is [1 1]. Otherwise, if you specify a cell array, then Dimension indicates the size of the entries in Elements. This property is essential for creating agents and function approximators objects that work with a given environment.

Example: Dimension=[1 1]

DataType — Information about the type of data

"double" (default) | string

This property is read-only.

Information about the type of data, specified as a string, such as "double" or "single". The software uses this property to enforce data type consistency for observations and actions.

Example: DataType="single"

Object Functions

rlSimulinkEnv	Create environment object from a Simulink model already containing agent and environment
rlFunctionEnv	Create custom reinforcement learning environment using your reset and step functions
rlValueFunction	Value function approximator object for reinforcement learning agents
rlQValueFunction	Q-Value function approximator with a continuous or discrete action space reinforcement learning agents
rlVectorQValueFunction	Vector Q-value function approximator with hybrid or discrete action space for reinforcement learning agents
rlContinuousDeterministicActor	Deterministic actor with a continuous action space for reinforcement learning agents
rlDiscreteCategoricalActor	Stochastic categorical actor with a discrete action space for reinforcement learning agents
rlContinuousGaussianActor	Stochastic Gaussian actor with a continuous action space for reinforcement learning agents

Examples

Create Reinforcement Learning Environment for Simulink Model

For this example, consider the rlSimplePendulumModel Simulink® model. The model is a simple frictionless pendulum that initially hangs in a downward position.

Open the model.

mdl = "rlSimplePendulumModel";
open_system(mdl)

Create rlNumericSpec and rlFiniteSetSpec objects for the observation and action specifications, respectively.

The observation is a vector containing three signals: the sine, cosine, and time derivative of the angle.

obsInfo = rlNumericSpec([3 1]) 

obsInfo = 
  rlNumericSpec with properties:

     LowerLimit: -Inf
     UpperLimit: Inf
           Name: [0x0 string]
    Description: [0x0 string]
      Dimension: [3 1]
       DataType: "double"

The action is a scalar expressing the torque and can be one of three possible values, -2 Nm, 0 Nm and 2 Nm.

actInfo = rlFiniteSetSpec([-2 0 2])

actInfo = 
  rlFiniteSetSpec with properties:

       Elements: [3x1 double]
           Name: [0x0 string]
    Description: [0x0 string]
      Dimension: [1 1]
       DataType: "double"

You can use dot notation to assign property values for the rlNumericSpec and rlFiniteSetSpec objects.

obsInfo.Name = "observations";
actInfo.Name = "torque";

Assign the agent block path information, and create the reinforcement learning environment for the Simulink model using the information extracted in the previous steps.

agentBlk = mdl + "/RL Agent";
env = rlSimulinkEnv(mdl,agentBlk,obsInfo,actInfo)

env = 
SimulinkEnvWithAgent with properties:

           Model : rlSimplePendulumModel
      AgentBlock : rlSimplePendulumModel/RL Agent
        ResetFcn : []
  UseFastRestart : on

You can also specify a reset function using dot notation. For this example, randomly initialize theta0 in the model workspace.

env.ResetFcn = @(in) setVariable(in,"theta0",randn,"Workspace",mdl)

env = 
SimulinkEnvWithAgent with properties:

           Model : rlSimplePendulumModel
      AgentBlock : rlSimplePendulumModel/RL Agent
        ResetFcn : @(in)setVariable(in,"theta0",randn,"Workspace",mdl)
  UseFastRestart : on

Create Specification Vector for Mixed Continuous-Discrete Observation Space

If your environment has an observation space consisting of multiple channels, some continuous, some discrete, use a vector of specification objects (each defining a single channel) to define the observation space.

For example, define an observation space as consisting of four channels. The first one carries a single number labeled 7, 9, 19 or -2. The second one carries a vector over a continuous three-dimensional space. The third channel carries a two by two matrix that can be either the zero matrix (that is zeros(2)) or the identity matrix (that is eye(2)). Finally, the fourth channel carries a continuous matrix with four rows and three columns.

obsInfo = [  rlFiniteSetSpec([7 9 19 -2])
             rlNumericSpec([3 1])
             rlFiniteSetSpec({zeros(2), eye(2)})
             rlNumericSpec([4 3]) ]

obsInfo=4×1 heterogeneous RLDataSpec (rlFiniteSetSpec, rlNumericSpec) array with properties:
    Name
    Description
    Dimension
    DataType

You can access each channel specification using dot notation.

obsInfo(1)

ans = 
  rlFiniteSetSpec with properties:

       Elements: [4x1 double]
           Name: [0x0 string]
    Description: [0x0 string]
      Dimension: [1 1]
       DataType: "double"

obsInfo(2).Name = "Velocity";
obsInfo(2).Description = "Velocity vector in m/s in body reference frame";
obsInfo(2)

ans = 
  rlNumericSpec with properties:

     LowerLimit: -Inf
     UpperLimit: Inf
           Name: "Velocity"
    Description: "Velocity vector in m/s in body reference frame"
      Dimension: [3 1]
       DataType: "double"

Convert Multiple Discrete Action Channels to Single Discrete Action Channel

Within Reinforcement Learning Toolbox™ software, agents can only have one action channel. However, if your desired design involves multiple discrete action channels, you can convert them into a single action channel with a number of elements equal to the product of the number of elements of each action channel of your original design. Specifically, each element of the single action channel corresponds to a particular combination of elements in the action channels of your original design.

For example, suppose that the valid values for a two-output system are [1 2] for the first output and [10 20 30] for the second output. Create a discrete action space specification for all possible output combinations.

actionSpec = rlFiniteSetSpec({[1 10],[1 20],[1 30],...
                              [2 10],[2 20],[2 30]})

actionSpec = 
  rlFiniteSetSpec with properties:

       Elements: {6x1 cell}
           Name: [0x0 string]
    Description: [0x0 string]
      Dimension: [1 2]
       DataType: "double"

Version History

Introduced in R2019a