Experiment Manager

Design and run experiments by using your MATLAB code

Since R2023b

Description

You can use the Experiment Manager app to create experiments to run your MATLAB^® code using various parameter values and compare the results. For example, you can use Experiment Manager to explore how the solution to a system of differential equations responds to different coefficient values or how it evolves from different initial conditions.

Experiment Manager provides visualizations, filters, and annotations to help you manage your experiment results and record your observations. To improve reproducibility, Experiment Manager stores a copy of the experiment definition every time that you run an experiment. You can access past experiment definitions to keep track of the combinations of parameters that produce each of your results.

Experiment Manager organizes your experiments and results in projects.

You can store several experiments in the same project.
Each experiment contains a set of results for each time that you run the experiment.
Each set of results consists of one or more trials that correspond to a different combination of parameters.

The Experiment Browser pane displays the hierarchy of experiments and results in the project. For example, this project has three experiments, each of which has several sets of results.

The orange round-bottom flask indicates a general-purpose experiment. The blue Erlenmeyer flask indicates a built-in training experiment for deep learning. The green beaker indicates a custom training experiment for deep learning or machine learning. For more information about experiments for your AI workflows, see Experiment Manager (Deep Learning Toolbox) and Experiment Manager (Statistics and Machine Learning Toolbox).

By default, Experiment Manager runs one trial at a time. If you have Parallel Computing Toolbox™, you can run multiple trials at the same time or run a single trial on multiple GPUs, on a cluster, or in the cloud. If you have MATLAB Parallel Server™, you can also offload experiments as batch jobs in a remote cluster so that you can continue working or close your MATLAB session while your experiment runs. For more information, see Run Experiments in Parallel and Offload Experiments as Batch Jobs to a Cluster.

Required Products

Deep Learning Toolbox™ to run built-in or custom training experiments for deep learning and to view confusion matrices for these experiments
Statistics and Machine Learning Toolbox™ to run custom training experiments for machine learning and experiments that use Bayesian optimization
Parallel Computing Toolbox to run multiple trials at the same time or a single trial at a time on multiple GPUs, on a cluster, or in the cloud
MATLAB Parallel Server to offload experiments as batch jobs in a remote cluster

Open the Experiment Manager App

MATLAB Toolstrip: On the Apps tab, under MATLAB, click the Experiment Manager icon.
MATLAB command prompt: Enter experimentManager.

Examples

expand all

Convert MATLAB Code into Experiment

Open Script

This example shows how to convert your existing MATLAB code into an experiment that you can run using the Experiment Manager app.

This script creates a histogram that shows that the 13th day of the month is more likely to fall on a Friday than on any other day of the week. For more information, see Chapter 3 of Experiments with MATLAB by Cleve Moler.

date = 13;

daysOfWeek = ["Sunday","Monday","Tuesday","Wednesday", ...
    "Thursday","Friday","Saturday"];
values = zeros(1,7);

for year = 1601:2000
    for month = 1:12
        d = datetime(year,month,date);
        w = weekday(d);
        values(w) = values(w) + 1;
    end
end

[minValue,maxValue] = bounds(values);
avgValue = mean(values);

figure(Name="Histogram")
bar(values)
axis([0 8 floor((minValue-1)/10)*10 ceil((maxValue+1)/10)*10])
line([0 8],[avgValue avgValue],linewidth=4,color="black")
set(gca,xticklabel=daysOfWeek)

You can convert this script into an experiment by following these steps. Alternatively, open the example to skip the conversion steps and load a preconfigured experiment that runs a converted version of the script.

1. Close any open projects and open the Experiment Manager app.

2. A dialog box provides links to the getting started tutorials and your recent projects, as well as buttons to create a new project or open an example from the documentation. Under New, select Blank Project.

3. If you have Deep Learning Toolbox or Statistics and Machine Learning Toolbox, Experiment Manager opens a second dialog box that lists several templates to support your AI workflows. Under Blank Experiments, select General Purpose.

4. Specify the name and location for the new project. Experiment Manager opens a new experiment in the project. The experiment definition tab displays the description, parameters, and experiment function that define the experiment. For more information, see Configure General-Purpose Experiment.

5. In the Description field, enter a description of the experiment:

Count the number of times that a given day and month falls on each day of the week.
To scan all months, set the value of Month to 0.

6. Under Parameters, add a parameter called Day with a value of 21 and a parameter called Month with a value of 0:3:12.

7. Under Experiment Function, click Edit. A blank experiment function called Experiment1Function1 opens in MATLAB Editor. The experiment function has an input argument called params and two output arguments called output1 and output2.

8. Copy and paste your MATLAB code into the body of the experiment function.

9. Replace the hard-coded value for the variable date with the expression params.Day. This expression uses dot notation to access the parameter values that you specified in step 6.

date = params.Day;

10. Add a new variable called monthRange that accesses the value of the parameter Month. If this value equals zero, set monthRange to the vector 1:12.

monthRange = params.Month;
if monthRange == 0
    monthRange = 1:12;
end

11. Use monthRange as the range for the for loop with counter month. Additionally, use the day function to account for months with fewer than 31 days.

for year = 1601:2000
    for month = monthRange
        d = datetime(year,month,date);
        if day(d) == date
            w = weekday(d);
            values(w) = values(w) + 1;
        end
    end
end

12. Rename the output arguments to MostLikelyDay and LeastLikelyDay. Use this code to compute these outputs after you calculate the values of maxValue, minValue, and avgValue:

maxIndex = ~(maxValue-values);
maxIndex = maxIndex.*(1:1:7);
maxIndex = nonzeros(maxIndex)';
MostLikelyDay = join(daysOfWeek(maxIndex));
 
minIndex = ~(values-minValue);
minIndex = minIndex.*(1:1:7);
minIndex = nonzeros(minIndex)';
LeastLikelyDay = join(daysOfWeek(minIndex));

After these steps, your experiment function contains this code:

function [MostLikelyDay,LeastLikelyDay] = Experiment1Function1(params)

date = params.Day;
 
monthRange = params.Month;
if monthRange == 0
    monthRange = 1:12;
end
 
daysOfWeek = ["Sunday","Monday","Tuesday","Wednesday", ...
    "Thursday","Friday","Saturday"];
values = zeros(1,7);
 
for year = 1601:2000
    for month = monthRange
        d = datetime(year,month,date);
        if day(d) == date
            w = weekday(d);
            values(w) = values(w) + 1;
        end
    end
end
 
[minValue,maxValue] = bounds(values);
avgValue = mean(values);
 
maxIndex = ~(maxValue-values);
maxIndex = maxIndex.*(1:1:7);
maxIndex = nonzeros(maxIndex)';
MostLikelyDay = join(daysOfWeek(maxIndex));
 
minIndex = ~(values-minValue);
minIndex = minIndex.*(1:1:7);
minIndex = nonzeros(minIndex)';
LeastLikelyDay = join(daysOfWeek(minIndex));
 
figure(Name="Histogram")
bar(values)
axis([0 8 floor((minValue-1)/10)*10 ceil((maxValue+1)/10)*10])
line([0 8],[avgValue avgValue],linewidth=4,color="black")
set(gca,xticklabel=daysOfWeek)
 
end

To run the experiment, on the Experiment Manager toolstrip, click Run. Experiment Manager runs the experiment function five times, each time using a different combination of parameter values. A table of results displays the output values for each trial.

To display a histogram for each completed trial, under Review Results, click Histogram.

The results of the experiment show that the 21st day of the month is more likely to fall on a Saturday than on any other day of the week. However, the summer solstice, June 21, is more likely to fall on a Sunday, Tuesday, or Thursday.

Configure General-Purpose Experiment

This example shows how to set up a general-purpose experiment using Experiment Manager.

General-purpose experiments consist of a description, a table of parameters, and an experiment function.

Default configuration for general-purpose experiment.

In the Description field, enter a description of the experiment.

Under Parameters, enter the names and values of the parameters used in the experiment. Parameter names must start with a letter, followed by letters, digits, or underscores. Parameter values must be scalars or vectors with numeric, logical, or string values, or cell arrays of character vectors. For example, these are valid parameter specifications:

0.01
0.01:0.01:0.05
[0.01 0.02 0.04 0.08]
["alpha" "beta" "gamma"]
{'delta' 'epsilon' 'zeta'}

Experiment Manager runs multiple trials of your experiment using a different combination of parameters for each trial.

The Experiment Function section specifies the procedure used by the experiment. The input to the experiment function is a structure with fields from the parameter table. The experiment function can return multiple outputs. The names of the output variables appear as column headers in the top of the results table. Each output value must be a numeric, logical, or string scalar.

Tip

When writing your experiment function, follow these guidelines:

Access the parameter values for the experiment by using dot notation. For more information, see Structure Arrays.
Load data for your experiment from a location that is accessible to all your parallel workers. For example, store your data outside the project and access the data by using an absolute path. Alternatively, create a datastore object that can access the data on another machine by setting up the AlternateFileSystemRoots property of the datastore. For more information, see Set Up Datastore for Processing on Different Machines or Clusters.
When your experiment finishes running, the Review Results gallery in the toolstrip displays a button for each figure that you create in your experiment function. To display a figure in the Visualizations pane, click the corresponding button in the Custom Plot section of the gallery. Specify the name of the button by using the Name property of the figure. If you do not name your figure, Experiment Manager derives the name of the button from the axes or figure title.

Sort, Filter, and Annotate Experiment Results

Experiment Manager runs multiple trials of your experiment using a different combination of parameters for each trial. A table of results displays the parameter and output values for each trial. To compare your results, you can use these values to sort the results table and filter trials.

To sort the trials in the results table, use the drop-down list on a column header.

Point to the header of a column by which you want to sort.
Click the triangle icon.
Select Sort in Ascending Order or Sort in Descending Order.

To filter trials from the results table, use the Filters pane:

On the Experiment Manager toolstrip, select Filters.
The Filters pane shows a histogram for each column in the results table that has numeric values. To remove a histogram, in the results table, open the drop-down list for the corresponding column and clear the Show Filter check box.
Adjust the sliders under the histogram for the column by which you want to filter.
The results table shows only the trials with a value in the selected range.
To restore all of the trials in the results table, close the experiment result tab and reopen the results from the Experiment Browser pane.

To record observations about the results of your experiment, add an annotation:

Right-click a cell in the results table and select Add Annotation. Alternatively, select a cell in the results table and, on the Experiment Manager toolstrip, select Annotations > Add Annotation.
In the Annotations pane, enter your observations in the text box. You can add multiple annotations for each cell in the results table.
To sort annotations, use the Sort By drop-down list. You can sort by creation time or trial number.
To highlight the cell that corresponds to an annotation, click the link above the annotation.
To delete an annotation, click the delete button to the right of the annotation.

View Source of Past Experiment Definitions

Experiment Manager stores a read-only copy of the parameter values and MATLAB code that produce each set of results for your experiment. You can run an experiment multiple times, each time using a different version of your code but always using the same function name. If you decide that you want to revert to an earlier version of your code, you can access it by opening the experiment source for the earlier result. To see this information:

On the Experiment Browser pane, double-click the name of the set of results you want to inspect.
On the experiment result tab, click View Experiment Source.
In the experiment source tab, inspect the experiment description, parameter values, and functions that produced the set of results.
To open the functions used by the experiment, click the links at the bottom of the tab. These functions are read-only, but you can copy them to the project folder, rerun the experiment, and reproduce your results.

Related Examples

Tips

To navigate Experiment Manager when using a mouse is not an option, use shortcut keyboards. For more information, see Keyboard Shortcuts for Experiment Manager.
To reduce the size of your experiments, discard the results and visualizations of any trial that is no longer relevant. In the Actions column of the results table, click the Discard button for the trial.
If you have Deep Learning Toolbox or Statistics and Machine Learning Toolbox, you can use Experiment Manager for your AI workflows. For more information, see Experiment Manager (Deep Learning Toolbox) or Experiment Manager (Statistics and Machine Learning Toolbox).

Version History

Introduced in R2023b