matlab.perftest.FrequentistTimeExperiment Class

Namespace: matlab.perftest
Superclasses: matlab.perftest.TimeExperiment

TimeExperiment that collects variable number of measurements

Description

A FrequentistTimeExperiment is a subclass of the TimeExperiment class that collects a variable number of measurements. The testing framework collects a variable number of measurements to reach a specified relative margin of error within a specified confidence level.

To create a FrequentistTimeExperiment instance, use the matlab.perftest.TimeExperiment.limitingSamplingError static method.

Properties

NumWarmups — Number of warm-up measurements

5 (default) | nonnegative integer

Number of warm-up measurements, specified as a nonnegative integer. NumWarmups defines the number of times that the framework runs the test code to warm it up.

MinSamples — Minimum number of samples

4 (default) | integer greater than 1

Minimum number of sample measurements, specified as an integer greater than 1. MinSamples defines the minimum number of times that the framework runs the test code after NumWarmups. The framework exercises the test code at least MinSamples times, regardless of whether the experiment meets the statistical objectives.

MaxSamples — Maximum number of samples

256 (default) | integer greater than or equal to MinSamples

Maximum number of sample measurements, specified as an integer greater than or equal to MinSamples. MaxSamples defines the maximum number of times that the framework runs the test code after NumWarmups. The framework will stop after it takes MaxSamples samples, even if the experiment does not meet the statistical objectives.

RelativeMarginOfError — Goal relative margin of error for samples

0.05 (default) | positive number

Goal relative margin of error for samples, specified as a positive number.

The framework calculates the Relative Margin of Error for a sample X using the equation

$$\text{relMoE}=\frac{\text{T}\ast \text{std(X)}}{\text{mean(X)}\ast \sqrt{\text{length(X)}}}$$

where T is the T-score from Student's T distribution using the specified ConfidenceLevel and length(X)-1 degrees of freedom.

ConfidenceLevel — Confidence level for samples to be within relative margin of error

0.95 (default) | number between 0 and 1

Confidence level for the samples to be within the relative margin of error, specified as a number between 0 and 1.

Copy Semantics

Handle. To learn how handle classes affect copy operations, see Copying Objects.

Version History

Introduced in R2016a

R2023a: Default number of warm-up measurements has increased

The default number of times that the framework exercises the test code to warm it up in a frequentist time experiment has increased from four to five. This change results in typically fewer samples required to meet the objective relative margin of error.

If you want to use the previous default value, explicitly specify the value in your code. This table shows an example of how to update code that runs tests using four warm-up measurements.

Before	After
import matlab.perftest.TimeExperiment experiment = TimeExperiment.limitingSamplingError; results = run(experiment,mySuite);	import matlab.perftest.TimeExperiment experiment = TimeExperiment.limitingSamplingError("NumWarmups",4); results = run(experiment,mySuite);