Deploy Trained Reinforcement Learning Policy as Microservice Docker Image

This example uses:

Supported platform: Linux®, Windows®, macOS

This example shows how to create a microservice Docker® image from a MATLAB® reinforcement learning policy. The microservice image created by MATLAB Compiler SDK™ provides an HTTP/HTTPS endpoint to access MATLAB code.

You package a MATLAB function into a deployable archive, and then create a Docker image that contains the archive and a minimal MATLAB Runtime package. You can then run the image in Docker and make calls to the service using any of the MATLAB Production Server™ client APIs.

Required Products

Type ver at the MATLAB command prompt to verify whether the following products are installed:

MATLAB
Reinforcement Learning Toolbox™
Deep Learning Toolbox™
MATLAB Compiler™
MATLAB Compiler SDK

Prerequisites

Verify that you have MATLAB Compiler SDK installed on the development machine.
Verify that you have Docker installed and configured on the development machine by typing [~,msg] = system('docker version') in a MATLAB command window. Note: If you are using WSL, use the command [~,msg] = system('wsl docker version') instead.
If you do not have Docker installed, follow the instructions on the Docker website to install and set up Docker. docs.docker.com/engine/install/
To build microservice images on Windows, you must install either Docker Desktop or Docker on Windows Subsystem for Linux v2 (WSL2). To install Docker Desktop, see docs.docker.com/desktop/install/windows-install/. For instructions on how to install Docker on WSL2, see https://www.mathworks.com/matlabcentral/answers/1758410-how-do-i-install-docker-on-wsl2.
If the computer you are using is not connected to the Internet, you must download the MATLAB Runtime installer for Linux from a computer that is connected to the Internet and transfer the installer to the computer that is not connected to the Internet. Then, on the offline machine, run the command compiler.runtime.createInstallerDockerImage(filepath), where filepath is the path to the MATLAB Runtime installer archive. You can download the installer from the MathWorks website. https://www.mathworks.com/products/compiler/matlab-runtime.html

Create MATLAB Function to Evaluate the Policy

For this example, we are using a pretrained DQN agent to balance a cart-pole system. See Train DQN Agent to Balance Discrete Cart-Pole System (Reinforcement Learning Toolbox) for more details. Write a policy-evaluation function called evaluatePolicy.m using the following code.

function action = evaluatePolicy(observation)
% Generate action from policy

% Load the policy from a MAT file
% policy.mat may contain an agent or policy with the name "agent"
persistent policy
if isempty(policy)
    policy = load("policy.mat");
end

% Evaluate the policy
action = getAction(policy.agent,observation);
end

Test the function from the MATLAB command line:

load("policy.mat")
evaluatePolicy(rand(agent.ObservationInfo.Dimension))

ans =

1×1 cell array

{[10]}

Create Deployable Archive

Package the evaluatePolicy function into a deployable archive using the compiler.build.productionServerArchive function.

You can specify additional options in the compiler.build command by using name-value arguments.

buildResults = compiler.build.productionServerArchive('evaluatePolicy.m',...
    'ArchiveName','cartPoleDQN','Verbose',true);

buildResults = 
  Results with properties:
                  BuildType: 'productionServerArchive'
                      Files: {'\home\mluser\work\cartPoleDQNproductionServerArchive\cartPoleDQN.ctf'}
    IncludedSupportPackages: {}
                    Options: [1×1 compiler.build.ProductionServerArchiveOptions]

The compiler.build.Results object buildResults contains information on the build type, generated files, included support packages, and build options.

Once the build is complete, the function creates a folder named cartPoleDQNproductionServerArchive in your current directory to store the deployable archive.

Package Archive into Microservice Docker Image

Build the microservice Docker image using the buildResults object that you created. You can specify additional options in the compiler.build command by using name-value arguments. For details, see compiler.package.microserviceDockerImage

compiler.package.microserviceDockerImage(buildResults,...
    'ImageName','cartpoledqn-microservice',...
    'DockerContext',fullfile(pwd,'microserviceDockerContext'));

The function generates the following files within a folder named microserviceDockerContext in your current working directory:

applicationFilesForMATLABCompiler/cartPoleDQN.ctf — Deployable archive file.
Dockerfile — Docker file that specifies Docker run-time options.
GettingStarted.txt — Text file that contains deployment information.

Test Docker Image

In a system command window, verify that your cartPoleDQN-microservice image is in your list of Docker images.

docker images

This command should return a list of Docker images, including your microservice:

REPOSITORY                                                               TAG               IMAGE ID       CREATED          SIZE
cartpoledqn-microservice                                                 latest            57f43b6811ce   22 seconds ago   7.31GB
matlabruntime/r2023a/release/update4/f08180002000003010                  latest            3d8fedb4189b   5 weeks ago      7.31GB

Run the cartpoledqn-microservice microservice image from the system command prompt.

docker run --rm -p 9900:9910 cartpoledqn-microservice -l trace &

Port 9910 is the default port exposed by the microservice within the Docker container. You can map it to any available port on your host machine. For this example, it is mapped to port 9900.

You can specify additional options in the Docker command. For a complete list of options, see Microservice Command Arguments.

Once the microservice container is running in Docker, you can check the status of the service by going to the following URL in a web browser:

http://<hostname>:9900/api/health

Note: Use localhost as the hostname if Docker is running on the same machine as the browser.

If the service is ready to receive requests, you see the following message:

"status:  ok"

Test the running service. In the terminal, use the curl command to send a JSON query to the service through port 9900. For more information on constructing JSON requests, see JSON Representation of MATLAB Data Types (MATLAB Production Server).

curl --location 'http://localhost:9900/cartPoleDQN/evaluatePolicy' \
--header 'Content-Type: application/json' \
--data '{"nargout":1,"rhs":[{"mwdata":[0,0,0,0],"mwsize":[4,1],"mwtype":"double"}]}'

The output is:

{"lhs":[{"mwdata":[{"mwdata":[-10],"mwsize":[1,1],"mwtype":"double"}],"mwsize":[1,1],"mwtype":"cell"}]}

You can also test from the MATLAB desktop:

%% Import MATLAB HTTP interface packages
import matlab.net.*
import matlab.net.http.*
import matlab.net.http.fields.*

%% Setup message body
body = MessageBody;
body.Payload = ...
    '{"nargout":1,"rhs":[{"mwdata":[0,0,0,0],"mwsize":[4,1],"mwtype":"double"}]}';

%% Setup request
requestUri = URI('http://hostname:9900/cartPoleDQN/evaluatePolicy');
options = matlab.net.http.HTTPOptions('ConnectTimeout',20,...
    'ConvertResponse',false);
request = RequestMessage;
request.Header = HeaderField('Content-Type','application/json');
request.Method = 'POST';
request.Body = body;

%% Send request & view raw response
response = request.send(requestUri, options);
disp(response.Body.Data)

%% Decode JSON
lhs = mps.json.decoderesponse(response.Body.Data);

%% Display response as a table
T = cell2table(lhs{:},'VariableNames',{'Action'})

Note: Replace hostname in the code above with the appropriate host. Since Docker is running locally, use localhost.

The output is:

T =
  table
    Action
    ______
      -10

To stop the service, use the following command to display the container id.

docker ps

CONTAINER ID   IMAGE                      COMMAND                  CREATED          STATUS          PORTS                                       NAMES
0662c1e1fa85   cartpoledqn-microservice   "/opt/matlabruntime/…"   17 minutes ago   Up 17 minutes   0.0.0.0:9900->9910/tcp, :::9900->9910/tcp   ecstatic_torvalds

Stop the service using the specified container id.

docker stop 0662c1e1fa85

Share Docker Image

You can share your Docker image in various ways.

Push your image to the Docker central registry DockerHub, or to your private registry. This is the most common workflow.
Save your image as a tar archive and share it with others. This workflow is suitable for immediate testing.

For details about pushing your image to DockerHub or your private registry, consult the Docker documentation.

Save Docker Image as Tar Archive

To save your Docker image as a tar archive, open a system command window, navigate to the Docker context folder, and type the following.

docker save cartpoledqn-microservice -o cartpoledqn-microservice.tar

This command creates a file named cartpoledqn-microservice.tar in the current folder. Set the appropriate permissions (for example, using chmod) prior to sharing the tarball with other users.

Load Docker Image from Tar Archive

Load the image contained in the tarball on the end user machine.

docker load --input cartpoledqn-microservice.tar

Verify that the image is loaded.

docker images

Run Docker Image

docker run --rm -p 9900:9910 cartpoledqn-microservice