How to use custom loss function with hyperparameter optimization with fit functions?

In MATLAB documentation there are several hyperparameter optimization options that can be tuned such as acquisition function, maximum time, cross-validation method etc. These can be applied to several fit functions for regression and classification tasks.

Example for regression ensembles: https://mathworks.com/help/stats/fitrensemble.html

I would like to use different Cross-validation loss than default one, which to my best understanding is MSE for regression. Is this possible by using fit functions or should I use bayesopt? I found at least one example of bayesopt in this case but I was wondering if there is easier way to do this if change of loss function is the only goal: https://se.mathworks.com/matlabcentral/answers/332524-change-objective-function-for-hyperparameter-optimization-in-regression-ensembles

Answer 1

[Akash]

Hi,

I understand that you would like to explore alternative methods for changing the loss function without relying on the "bayesopt" function. Additionally, you are interested in using a different cross-validation loss function instead of the default one.

In MATLAB, you can indeed use different loss functions for cross-validation by utilizing the "fit" functions available for "regression" and "classification" tasks. When fitting data with one or more models, you have the flexibility to evaluate the "goodness of fit" using various metrics. "Goodness of fit" offers valuable information on different metrics such as "Sum of Squared Errors (SSE)", "R-Square", "Adjusted R-Square", and "Root Mean Square Error (RMSE)". The documentation link given below will help you understand better.

https://www.mathworks.com/help/curvefit/evaluating-goodness-of-fit.html

You can explore the advanced options to change the loss function available in the Hyperparameter Optimization in "Regression Learner App". The documentation link for the same is given below. This app allows you to select different types of models, such as "decision trees" or "support vector machines (SVMs)” and tune their hyperparameters. By adjusting these hyperparameters, which are internal to the model, you can strongly influence its performance.

https://www.mathworks.com/help/stats/hyperparameter-optimization-in-regression-learner-app.html

The link given below gives an example on weighted nonlinear regression that demonstrates how to fit a nonlinear regression model for data with nonconstant error variance. This example showcases the flexibility of MATLAB in handling different scenarios and customizing the fitting process according to specific requirements.

https://www.mathworks.com/help/stats/weighted-nonlinear-regression.html

By utilizing these resources and exploring the advanced options in the “Regression Learner App”, you can easily change the loss function and optimize your models for better performance.