Something wrong in fitrlinear with ridge regularization?

There is something wrong in the attached code. I couldn't figure out where it went wrong. Any help is highly appreciated!

[Prince Kumar]

Can you share the error as well?

[Peter He]

Hi Prince,

Thank you for looking into this.

There is no syntax error but rather a semantic error maybe. I explained the error in the live script - the result of setting "Lambda" to 0 for ridge regression should return a model exactly the same as the linear model w/o regularization/shrinkage. But I get a much worse model instead. Here is the line that I tested the above point in the attached code:

mdl2=fitrlinear(X(idxTrain,:),y(idxTrain),'Learner','leastsquares','Regularization','ridge','Lambda',0);

I got mse = 54.3139

if I use the following:

mdl = fitlm(X(idxTrain,:),y(idxTrain));

I got mse = 16.8294, which is significantly better. I also plotted the observed vs. fitted, which also shows very poor fitting for ridge regression.

[Peter He]

I'm not sure if there is any scaling involved for the predictors - I couldn't find any information from the document of this function. But it seems that the response is not scaled. Just a potential issue I have thought about but haven't tested.

[Biral Pradhan]

The "fitrlinear" method uses iterative solvers that converge subject to tolerances on beta coefficients and gradient. In this case, optimization doesn't occur because the first step does not move the beta estimates away from the initial zero values far enough. Kindly set the "BetaTolerance" parameter to a lower value, say 1e-6 in order to resolve this issue. For more information, kindly follow the below documentation link:

https://in.mathworks.com/help/stats/fitrlinear.html#d124e418872

I would also recommend setting "Verbose" to 1 in order to obtain more information throughout the optimization process.

[Peter He]

Thank you Biral. That indeed was the problem. Here is a link to a related question that I asked and got a similar answer to it. But I was able to verify the problem using that example.

https://www.mathworks.com/matlabcentral/answers/1689650-another-example-where-fitrlinear-performs-badly-and-i-don-t-know-where-the-problem-is#answer_1380091

Thanks again.

Peter

Answer 1

[Peter He]

Thank you Biral. That indeed was the problem. Here is a link to a related question that I asked and got a similar answer to it. But I was able to verify the problem using that example.

https://www.mathworks.com/matlabcentral/answers/1689650-another-example-where-fitrlinear-performs-badly-and-i-don-t-know-where-the-problem-is#answer_1380091

Thanks again.

Peter