RegressionSVM Class

Superclasses: CompactRegressionSVM

Support vector machine regression model

Description

RegressionSVM is a support vector machine (SVM) regression model. Train a RegressionSVM model using fitrsvm and the sample data.

RegressionSVM models store data, parameter values, support vectors, and algorithmic implementation information. You can use these models to:

• Estimate resubstitution predictions. For details, see resubPredict.

• Predict values for new data. For details, see predict.

• Compute resubstitution loss. For details, see resubLoss.

• Compute the mean square error or epsilon-insensitive loss. For details, see loss.

Construction

Create a RegressionSVM object by using fitrsvm.

Properties

Alpha — Dual problem coefficients

vector of numeric values

Dual problem coefficients, specified as a vector of numeric values. Alpha contains m elements, where m is the number of support vectors in the trained SVM regression model. The dual problem introduces two Lagrange multipliers for each support vector. The values of Alpha are the differences between the two estimated Lagrange multipliers for the support vectors. For more details, see Understanding Support Vector Machine Regression.

If you specified to remove duplicates using RemoveDuplicates, then, for a particular set of duplicate observations that are support vectors, Alpha contains one coefficient corresponding to the entire set. That is, MATLAB^® attributes a nonzero coefficient to one observation from the set of duplicates and a coefficient of 0 to all other duplicate observations in the set.

Data Types: single | double

Beta — Primal linear problem coefficients

vector of numeric values | '[]'

Primal linear problem coefficients, stored as a numeric vector of length p, where p is the number of predictors in the SVM regression model.

The values in Beta are the linear coefficients for the primal optimization problem.

If the model is obtained using a kernel function other than 'linear', this property is empty ('[]').

The predict method computes predicted response values for the model as YFIT = (X/S)×Beta + Bias, where S is the value of the kernel scale stored in the KernelParameters.Scale property.

Data Types: single | double

Bias — Bias term

scalar value

Bias term in the SVM regression model, stored as a scalar value.

Data Types: single | double

BoxConstraints — Box constraints for dual problem coefficients

vector of numeric values

Box constraints for dual problem alpha coefficients, stored as a numeric vector containing n elements, where n is the number of observations in X (Mdl.NumObservations).

The absolute value of the dual coefficient Alpha for observation i cannot exceed BoxConstraints(i).

If you specify removing duplicates using 'RemoveDuplicates', then for a given set of duplicate observations, MATLAB sums the box constraints, and then attributes the sum to one observation and box constraints of 0 to all other observations in the set.

Data Types: single | double

CacheInfo — Caching information

structure

Caching information, stored as a structure with the following fields.

Field	Description
'Size'	Positive scalar value indicating the cache size (in MB) that the software reserves to store entries of the Gram matrix. Set the cache size by using the 'CacheSize' name-value pair argument in fitrsvm.
'Algorithm'	Character vector containing the name of the algorithm used to remove entries from the cache when its capacity is exceeded. Currently, the only available caching algorithm is 'Queue'. You cannot set the caching algorithm.

Data Types: struct

CategoricalPredictors — Indices of categorical predictors

vector of positive integers | []

This property is read-only.

Categorical predictor indices, specified as a vector of positive integers. CategoricalPredictors contains index values indicating that the corresponding predictors are categorical. The index values are between 1 and p, where p is the number of predictors used to train the model. If none of the predictors are categorical, then this property is empty ([]).

Data Types: single | double

ConvergenceInfo — Convergence information

structure

Convergence information, stored as a structure with the following fields.

Field	Description
Converged	Logical flag indicating whether the algorithm converged. A value of 1 indicates convergence.
ReasonForConvergence	Character vector indicating the criterion the software used to detect convergence.
Gap	Scalar feasibility gap between the dual and primal objective functions.
GapTolerance	Scalar tolerance for the feasibility gap. You can set this tolerance using the 'GapTolerance' name-value pair argument in fitrsvm.
DeltaGradient	Scalar gradient difference between upper and lower violators.
DeltaGradientTolerance	Scalar tolerance for the gradient difference. You can set this tolerance using the DeltaGradientTolerance name-value pair argument in fitrsvm.
LargestKKTViolation	Maximal scalar Karush-Kuhn-Tucker (KKT) violation value.
KKTTolerance	Scalar tolerance for the largest KKT violation. You can set this tolerance using the 'KKTTolerance' name-value pair argument in fitrsvm.
History	Structure containing convergence information recorded at periodic intervals during the model training process. This structure contains the following fields: NumIterations — Array of iteration indices at which the software recorded convergence criteria. Gap — Gap values at these iterations. DeltaGradient — DeltaGradient values at these iterations. LargestKKTViolation — LargestKKTViolation values at these iterations. NumSupportVectors — Number of support vectors at these iterations. Objective — Objective values at these iterations.
Objective	Numeric value of the dual objective.

Data Types: struct

Epsilon — Half the width of the epsilon-insensitive band

nonnegative scalar value

Half the width of the epsilon-insensitive band, stored as a nonnegative scalar value.

Data Types: single | double

ExpandedPredictorNames — Expanded predictor names

cell array of character vectors

Expanded predictor names, stored as a cell array of character vectors.

If the model uses encoding for categorical variables, then ExpandedPredictorNames includes the names that describe the expanded variables. Otherwise, ExpandedPredictorNames is the same as PredictorNames.

Data Types: cell

Gradient — Gradient values in training data

vector of numeric values

Gradient values in training data, stored as a numeric vector containing 2n elements, where n is the number of observations in the training data.

Element i of Gradient contains the gradient value for the Alpha coefficient that corresponds to the upper boundary of the epsilon-insensitive band at observation i at the end of the optimization.

Element i + NumObservations of Gradient contains the gradient value for the Alpha coefficient that corresponds to the lower boundary of the epsilon-insensitive band at observation i at the end of the optimization.

Data Types: single | double

HyperparameterOptimizationResults — Cross-validation optimization of hyperparameters

BayesianOptimization object | table

This property is read-only.

Cross-validation optimization of hyperparameters, specified as a BayesianOptimization object or a table of hyperparameters and associated values. This property is nonempty if the 'OptimizeHyperparameters' name-value pair argument is nonempty when you create the model. The value of HyperparameterOptimizationResults depends on the setting of the Optimizer field in the HyperparameterOptimizationOptions structure when you create the model.

Value of Optimizer Option	Value of HyperparameterOptimizationResults
"bayesopt" (default)	Object of class BayesianOptimization
"gridsearch" or "randomsearch"	Table of hyperparameters used, observed objective function values (cross-validation loss), and rank of observations from lowest (best) to highest (worst)

IsSupportVector — Flag indicating whether observation is support vector

logical vector

Flag indicating whether an observation is a support vector, stored as an n-by-1 logical vector. n is the number of observations in X (see NumObservations). A value of 1 indicates that the corresponding observation in the training data is a support vector.

If you specify removing duplicates using RemoveDuplicates, then for a given set of duplicate observations that are support vectors, IsSupportVector flags only one as a support vector.

Data Types: logical

KernelParameters — Kernel function parameters

structure

Kernel function parameters, stored as a structure with the following fields.

Field	Description
Function	Kernel function name (a character vector).
Scale	Numeric scale factor used to divide predictor values.

You can specify values for KernelParameters.Function and KernelParameters.Scale by using the KernelFunction and KernelScale name-value pair arguments in fitrsvm, respectively.

Data Types: struct

ModelParameters — Parameter values

model parameter object

Parameter values used to train the SVM regression model, stored as a model parameter object. Access the properties of ModelParameters using dot notation. For example, access the value of Epsilon used to train the model as Mdl.ModelParameters.Epsilon.

Mu — Predictor means

vector of numeric values | '[]'

Predictor means, stored as a vector of numeric values.

If the predictors are standardized, then Mu is a numeric vector of length p, where p is the number of predictors used to train the model. In this case, the predict method centers predictor matrix X by subtracting the corresponding element of Mu from each column.

If the predictors are not standardized, then Mu is empty ('[]').

If the data contains categorical predictors, then Mu includes elements for the dummy variables for those predictors. The corresponding entries in Mu are 0 because dummy variables are not centered or scaled.

Data Types: single | double

NumIterations — Number of iterations required for convergence

positive integer value

Number of iterations required for the optimization routine to reach convergence, stored as a positive integer value.

To set a limit on the number of iterations, use the 'IterationLimit' name-value pair argument of fitrsvm.

Data Types: single | double

NumObservations — Number of observations

positive integer value

Number of observations in the training data, stored as a positive integer value.

Data Types: single | double

PredictorNames — Predictor names

cell array of character vectors

Predictor names, stored as a cell array of character vectors containing the name of each predictor in the order they appear in X. PredictorNames has a length equal to the number of columns in X.

Data Types: cell

OutlierFraction — Expected fraction of outliers

scalar value in the range [0,1]

Expected fraction of outliers in the training set, stored as a scalar value in the range [0,1]. You can specify the expected fraction of outliers using the 'OutlierFraction' name-value pair argument in fitrsvm.

Data Types: double

ResponseName — Response variable name

character vector

Response variable name, stored as a character vector.

Data Types: char

ResponseTransform — Response transformation function

'none' | function handle

Response transformation function, specified as 'none' or a function handle. ResponseTransform describes how the software transforms raw response values.

For a MATLAB function or a function that you define, enter its function handle. For example, you can enter Mdl.ResponseTransform = @function, where function accepts a numeric vector of the original responses and returns a numeric vector of the same size containing the transformed responses.

Data Types: char | function_handle

RowsUsed — Rows of original training data stored

logical vector | []

Rows of the original training data stored in the model, specified as a logical vector. This property is empty if all rows are stored in X and Y.

Data Types: logical

ShrinkagePeriod — Number of iterations between reductions of active set

nonnegative integer value

Number of iterations between reductions of the active set during optimization, stored as a nonnegative integer value.

You can set the shrinkage period by using the 'ShrinkagePeriod' name-value pair argument in fitrsvm.

Data Types: single | double

Sigma — Predictor standard deviations

vector of numeric values | '[]'

Predictor standard deviations, stored as a vector of numeric values.

If the predictors are standardized, then Sigma is a numeric vector of length p, where p is the number of predictors used to train the model. In this case, the predict method scales the predictor matrix X by dividing each column by the corresponding element of Sigma, after centering each element using Mu.

If the predictors are not standardized, then Sigma is empty ('[]').

If the data contains categorical predictors, Sigma includes elements for the dummy variables for those predictors. The corresponding entries in Sigma are 1, because dummy variables are not centered or scaled.

Data Types: single | double

Solver — Name of solver algorithm

character vector

Name of the solver algorithm used to solve the optimization problem, stored as a value in this table.

Value	Description
'SMO'	Sequential Minimal Optimization
'ISDA'	Iterative Single Data Algorithm
'L1QP'	L1 soft-margin minimization by quadratic programming (requires an Optimization Toolbox™ license).

SupportVectors — Support vectors

matrix of numeric values

Support vectors, stored as an m-by-p matrix of numeric values. m is the number of support vectors (sum(Mdl.IsSupportVector)), and p is the number of predictors in X.

If you specified to remove duplicates using RemoveDuplicates, then for a given set of duplicate observations that are support vectors, SupportVectors contains one unique support vector.

Data Types: single | double

W — Observation weights

vector of numeric values

Observation weights used to train the model, stored as a numeric vector containing NumObservation number of elements. fitrsvm normalizes the weights used for training so that they sum to 1.

Data Types: single | double

X — Predictor values

matrix of numeric values | table of numeric values

Predictor values used to train the model, stored as a matrix of numeric values if the model is trained on a matrix, or a table if the model is trained on a table. X has size n-by-p, where n is the number of rows and p is the number of predictor variables or columns in the training data.

Data Types: single | double | table

Y — Observed response values

vector of numeric values

Observed response values, stored as a numeric vector containing NumObservations number of elements.

Data Types: single | double

Object Functions

compact	Compact support vector machine regression model
crossval	Cross-validate machine learning model
discardSupportVectors	Discard support vectors for linear support vector machine (SVM) regression model
gather	Gather properties of Statistics and Machine Learning Toolbox object from GPU
incrementalLearner	Convert support vector machine (SVM) regression model to incremental learner
lime	Local interpretable model-agnostic explanations (LIME)
loss	Regression error for support vector machine regression model
partialDependence	Compute partial dependence
plotPartialDependence	Create partial dependence plot (PDP) and individual conditional expectation (ICE) plots
predict	Predict responses using support vector machine regression model
resubLoss	Resubstitution loss for support vector machine regression model
resubPredict	Predict resubstitution response of support vector machine regression model
resume	Resume training support vector machine regression model
shapley	Shapley values

Copy Semantics

Value. To learn how value classes affect copy operations, see Copying Objects.

Examples

Train Linear Support Vector Machine Regression Model

This example shows how to train a linear support vector machine (SVM) regression model using sample data stored in matrices.

Load the carsmall data set.

load carsmall
rng default  % for reproducibility

Specify Horsepower and Weight as the predictor variables (X) and MPG as the response variable (Y).

X = [Horsepower,Weight];
Y = MPG;

Train a linear SVM regression model.

Mdl = fitrsvm(X,Y)

Mdl = 
  RegressionSVM
             ResponseName: 'Y'
    CategoricalPredictors: []
        ResponseTransform: 'none'
                    Alpha: [75x1 double]
                     Bias: 57.3800
         KernelParameters: [1x1 struct]
          NumObservations: 94
           BoxConstraints: [94x1 double]
          ConvergenceInfo: [1x1 struct]
          IsSupportVector: [94x1 logical]
                   Solver: 'SMO'

The Command Window shows that Mdl is a trained RegressionSVM model and a list of its properties.

Check the model for convergence.

Mdl.ConvergenceInfo.Converged

ans = logical
   0

0 indicates that the model did not converge.

MdlStd = fitrsvm(X,Y,'Standardize',true)

MdlStd = 
  RegressionSVM
             ResponseName: 'Y'
    CategoricalPredictors: []
        ResponseTransform: 'none'
                    Alpha: [77x1 double]
                     Bias: 22.9131
         KernelParameters: [1x1 struct]
                       Mu: [109.3441 2.9625e+03]
                    Sigma: [45.3545 805.9668]
          NumObservations: 94
           BoxConstraints: [94x1 double]
          ConvergenceInfo: [1x1 struct]
          IsSupportVector: [94x1 logical]
                   Solver: 'SMO'

Check the model for convergence.

MdlStd.ConvergenceInfo.Converged

ans = logical
   1

1 indicates that the model did converge.

Compute the resubstitution mean squared error for the new model.

lStd = resubLoss(MdlStd)

lStd = 
16.8551

Train Support Vector Machine Regression Model

Train a support vector machine regression model using the abalone data from the UCI Machine Learning Repository.

Download the data and save it in your current folder with the name 'abalone.csv'.

url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.data';
websave('abalone.csv',url);

Read the data into a table. Specify the variable names.

varnames = {'Sex'; 'Length'; 'Diameter'; 'Height'; 'Whole_weight';...
    'Shucked_weight'; 'Viscera_weight'; 'Shell_weight'; 'Rings'};
Tbl = readtable('abalone.csv','Filetype','text','ReadVariableNames',false);
Tbl.Properties.VariableNames = varnames;

The sample data contains 4177 observations. All the predictor variables are continuous except for Sex, which is a categorical variable with possible values 'M' (for males), 'F' (for females), and 'I' (for infants). The goal is to predict the number of rings (stored in Rings) on the abalone and determine its age using physical measurements.

Train an SVM regression model, using a Gaussian kernel function with an automatic kernel scale. Standardize the data.

rng default  % For reproducibility
Mdl = fitrsvm(Tbl,'Rings','KernelFunction','gaussian','KernelScale','auto',...
    'Standardize',true)

Mdl = 
  RegressionSVM
           PredictorNames: {'Sex'  'Length'  'Diameter'  'Height'  'Whole_weight'  'Shucked_weight'  'Viscera_weight'  'Shell_weight'}
             ResponseName: 'Rings'
    CategoricalPredictors: 1
        ResponseTransform: 'none'
                    Alpha: [3635×1 double]
                     Bias: 10.8144
         KernelParameters: [1×1 struct]
                       Mu: [0 0 0 0.5240 0.4079 0.1395 0.8287 0.3594 0.1806 0.2388]
                    Sigma: [1 1 1 0.1201 0.0992 0.0418 0.4904 0.2220 0.1096 0.1392]
          NumObservations: 4177
           BoxConstraints: [4177×1 double]
          ConvergenceInfo: [1×1 struct]
          IsSupportVector: [4177×1 logical]
                   Solver: 'SMO'


  Properties, Methods

The Command Window shows that Mdl is a trained RegressionSVM model and displays a property list.

Display the properties of Mdl using dot notation. For example, check to confirm whether the model converged and how many iterations it completed.

conv = Mdl.ConvergenceInfo.Converged

conv = logical
   1

iter = Mdl.NumIterations

iter = 2759

The returned results indicate that the model converged after 2759 iterations.

References

[1] Nash, W.J., T. L. Sellers, S. R. Talbot, A. J. Cawthorn, and W. B. Ford. "The Population Biology of Abalone (Haliotis species) in Tasmania. I. Blacklip Abalone (H. rubra) from the North Coast and Islands of Bass Strait." Sea Fisheries Division, Technical Report No. 48, 1994.

[2] Waugh, S. "Extending and Benchmarking Cascade-Correlation: Extensions to the Cascade-Correlation Architecture and Benchmarking of Feed-forward Supervised Artificial Neural Networks." University of Tasmania Department of Computer Science thesis, 1995.

[3] Clark, D., Z. Schreter, A. Adams. "A Quantitative Comparison of Dystal and Backpropagation." submitted to the Australian Conference on Neural Networks, 1996.

[4] Lichman, M. UCI Machine Learning Repository, [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.

Extended Capabilities

C/C++ Code Generation

Generate C and C++ code using MATLAB® Coder™.

Usage notes and limitations:

• The predict and update functions support code generation.

• To integrate the prediction of an SVM regression model into Simulink^®, you can use the RegressionSVM Predict block in the Statistics and Machine Learning Toolbox™ library or a MATLAB Function block with the predict function.

• When you train an SVM regression model by using fitrsvm, the following restrictions apply.

  • The value of the ResponseTransform name-value argument cannot be an anonymous function. For fixed-point code generation, the value must be 'none' (default).

  • For fixed-point code generation, the value of the KernelFunction name-value argument must be 'gaussian', 'linear', or 'polynomial'.

  • Fixed-point code generation and code generation with a coder configurer do not support categorical predictors (logical, categorical, char, string, or cell). You cannot use the CategoricalPredictors name-value argument. To include categorical predictors in a model, preprocess them by using dummyvar before fitting the model.

For more information, see Introduction to Code Generation.

GPU Arrays

Accelerate code by running on a graphics processing unit (GPU) using Parallel Computing Toolbox™. (since R2023a)

Usage notes and limitations:

• The following object functions fully support GPU arrays:

  • compact

  • crossval

  • discardSupportVectors

  • gather

  • loss

  • partialDependence

  • plotPartialDependence

  • predict

  • resubLoss

  • resubPredict

  • resume

• The object functions execute on a GPU if at least one of the following applies:

  • The model was fitted with GPU arrays.

  • The predictor data that you pass to the object function is a GPU array.

  • The response data that you pass to the object function is a GPU array.

For more information, see Run MATLAB Functions on a GPU (Parallel Computing Toolbox).

Version History

Introduced in R2015b

R2023b: Model stores observations with missing predictor values

Starting in R2023b, training observations with missing predictor values are included in the X, Y, and W data properties. The RowsUsed property indicates the training observations stored in the model, rather than those used for training. Observations with missing predictor values continue to be omitted from the model training process.

In previous releases, the software omitted training observations that contained missing predictor values from the data properties of the model.

R2023a: GPU array support for object functions (requires Parallel Computing Toolbox)

Starting in R2023a, you can fit a RegressionSVM object on a GPU by using fitrsvm. Most RegressionSVM object functions now support GPU array input arguments so that they can execute on a GPU. The object functions that do not support GPU array inputs are incrementalLearner, lime, and shapley.