regularize

Find weights to minimize resubstitution error plus penalty term

Syntax

ens1 = regularize(ens)
ens1 = regularize(ens,Name,Value)

Description

ens1 = regularize(ens) finds optimal weights for learners in ens by lasso regularization. regularize returns a regression ensemble identical to ens, but with a populated Regularization property.

ens1 = regularize(ens,Name,Value) computes optimal weights with additional options specified by one or more Name,Value pair arguments. You can specify several name-value pair arguments in any order as Name1,Value1,…,NameN,ValueN.

Input Arguments

 ens A regression ensemble, created by fitrensemble.

Name-Value Arguments

Specify optional comma-separated pairs of Name,Value arguments. Name is the argument name and Value is the corresponding value. Name must appear inside quotes. You can specify several name and value pair arguments in any order as Name1,Value1,...,NameN,ValueN.

 lambda Vector of nonnegative regularization parameter values for lasso. For the default setting of lambda, regularize calculates the smallest value lambda_max for which all optimal weights for learners are 0. The default value of lambda is a vector including 0 and nine exponentially-spaced numbers from lambda_max/1000 to lambda_max. Default: [0 logspace(log10(lambda_max/1000),log10(lambda_max),9)] MaxIter Maximum number of iterations allowed, specified as a positive integer. If the algorithm executes MaxIter iterations before reaching the convergence tolerance, then the function stops iterating and returns a warning message. The function can return more than one warning when either npass or the number of lambda values is greater than 1. Default: 1e3 npass Maximal number of passes for lasso optimization, a positive integer. Default: 10 reltol Relative tolerance on the regularized loss for lasso, a numeric positive scalar. Default: 1e-3 verbose Verbosity level, either 0 or 1. When set to 1, regularize displays more information as it runs. Default: 0

Output Arguments

 ens1 A regression ensemble. Usually you set ens1 to the same name as ens.

Examples

expand all

Regularize an ensemble of bagged trees.

Generate sample data.

rng(10,'twister') % For reproducibility
X = rand(2000,20);
Y = repmat(-1,2000,1);
Y(sum(X(:,1:5),2)>2.5) = 1;

You can create a bagged classification ensemble of 300 trees from the sample data.

bag = fitrensemble(X,Y,'Method','Bag','NumLearningCycles',300);

fitrensemble uses a default template tree object templateTree() as a weak learner when 'Method' is 'Bag'. In this example, for reproducibility, specify 'Reproducible',true when you create a tree template object, and then use the object as a weak learner.

t = templateTree('Reproducible',true); % For reproducibiliy of random predictor selections
bag = fitrensemble(X,Y,'Method','Bag','NumLearningCycles',300,'Learners',t);

Regularize the ensemble of bagged regression trees.

bag = regularize(bag,'lambda',[0.001 0.1],'verbose',1);
Starting lasso minimization for Lambda=0.001. Initial MSE=0.109923.
Lasso minimization completed pass 1 for Lambda=0.001
MSE = 0.086912
Relative change in MSE = 0.264768
Number of learners with non-zero weights = 15
Lasso minimization completed pass 2 for Lambda=0.001
MSE = 0.0670602
Relative change in MSE = 0.296029
Number of learners with non-zero weights = 34
Lasso minimization completed pass 3 for Lambda=0.001
MSE = 0.0623931
Relative change in MSE = 0.0748019
Number of learners with non-zero weights = 51
Lasso minimization completed pass 4 for Lambda=0.001
MSE = 0.0605444
Relative change in MSE = 0.0305348
Number of learners with non-zero weights = 70
Lasso minimization completed pass 5 for Lambda=0.001
MSE = 0.0599666
Relative change in MSE = 0.00963517
Number of learners with non-zero weights = 94
Lasso minimization completed pass 6 for Lambda=0.001
MSE = 0.0598835
Relative change in MSE = 0.00138719
Number of learners with non-zero weights = 105
Lasso minimization completed pass 7 for Lambda=0.001
MSE = 0.0598608
Relative change in MSE = 0.000379227
Number of learners with non-zero weights = 113
Lasso minimization completed pass 8 for Lambda=0.001
MSE = 0.0598586
Relative change in MSE = 3.72856e-05
Number of learners with non-zero weights = 115
Lasso minimization completed pass 9 for Lambda=0.001
MSE = 0.0598587
Relative change in MSE = 6.42954e-07
Number of learners with non-zero weights = 115
Lasso minimization completed pass 10 for Lambda=0.001
MSE = 0.0598587
Relative change in MSE = 4.53658e-08
Number of learners with non-zero weights = 115
Completed lasso minimization for Lambda=0.001.
Resubstitution MSE changed from 0.109923 to 0.0598587.
Number of learners reduced from 300 to 115.
Starting lasso minimization for Lambda=0.1. Initial MSE=0.109923.
Lasso minimization completed pass 1 for Lambda=0.1
MSE = 0.104917
Relative change in MSE = 0.0477191
Number of learners with non-zero weights = 12
Lasso minimization completed pass 2 for Lambda=0.1
MSE = 0.0851031
Relative change in MSE = 0.232821
Number of learners with non-zero weights = 30
Lasso minimization completed pass 3 for Lambda=0.1
MSE = 0.081245
Relative change in MSE = 0.0474877
Number of learners with non-zero weights = 40
Lasso minimization completed pass 4 for Lambda=0.1
MSE = 0.0796749
Relative change in MSE = 0.0197067
Number of learners with non-zero weights = 53
Lasso minimization completed pass 5 for Lambda=0.1
MSE = 0.0788411
Relative change in MSE = 0.0105746
Number of learners with non-zero weights = 64
Lasso minimization completed pass 6 for Lambda=0.1
MSE = 0.0784959
Relative change in MSE = 0.00439793
Number of learners with non-zero weights = 81
Lasso minimization completed pass 7 for Lambda=0.1
MSE = 0.0784429
Relative change in MSE = 0.000676468
Number of learners with non-zero weights = 88
Lasso minimization completed pass 8 for Lambda=0.1
MSE = 0.078447
Relative change in MSE = 5.24449e-05
Number of learners with non-zero weights = 88
Completed lasso minimization for Lambda=0.1.
Resubstitution MSE changed from 0.109923 to 0.078447.
Number of learners reduced from 300 to 88.

regularize reports on its progress.

Inspect the resulting regularization structure.

bag.Regularization
ans = struct with fields:
Method: 'Lasso'
TrainedWeights: [300x2 double]
Lambda: [1.0000e-03 0.1000]
ResubstitutionMSE: [0.0599 0.0784]
CombineWeights: @classreg.learning.combiner.WeightedSum

Check how many learners in the regularized ensemble have positive weights. These are the learners included in a shrunken ensemble.

sum(bag.Regularization.TrainedWeights > 0)
ans = 1×2

115    88

Shrink the ensemble using the weights from Lambda = 0.1.

cmp = shrink(bag,'weightcolumn',2)
cmp =
CompactRegressionEnsemble
ResponseName: 'Y'
CategoricalPredictors: []
ResponseTransform: 'none'
NumTrained: 88

Properties, Methods

The compact ensemble contains 87 members, less than 1/3 of the original 300.