Classification Learner app Hyperparameter Optimization

If no validation scheme is used in the Classification Learner app, what objective function is used in hyperparameter optimization? Is it cross-validation loss described in the Bayesian Optimization Workflow or a different loss function?

Answer 1

[Shubham]

Hi rp,

In the MATLAB Classification Learner app, when you perform hyperparameter optimization without explicitly specifying a validation scheme, the app typically uses cross-validation as the default method to evaluate model performance during the optimization process. Specifically, it uses k-fold cross-validation to estimate the generalization error of the models being evaluated.

Objective Function in Hyperparameter Optimization

When hyperparameter optimization is performed in the Classification Learner app, the objective function that is minimized is typically the cross-validation loss. This loss is computed during the k-fold cross-validation process, which involves:

1. Partitioning the Data: The dataset is divided into k subsets or "folds."
2. Training and Validation: For each fold, the model is trained on k-1 folds and validated on the remaining fold. This process is repeated k times, with each fold serving as the validation set once.
3. Averaging the Results: The cross-validation loss is the average loss across all k iterations.

Bayesian Optimization

The Classification Learner app uses Bayesian optimization as the strategy for hyperparameter tuning. Bayesian optimization is a probabilistic model-based optimization technique that efficiently searches the hyperparameter space by building a surrogate model (usually a Gaussian process) of the objective function. The cross-validation loss serves as the feedback for the optimization process, guiding the search towards the best hyperparameter values.

Loss Function

The specific loss function used during cross-validation depends on the type of classification problem and the settings you choose. Common loss functions include:

• Classification Error: The fraction of misclassified observations.
• Logarithmic Loss: Useful for probabilistic classifiers, measuring the accuracy of predicted probabilities.
• Hinge Loss: Often used for SVM models.

In summary, when no specific validation scheme is chosen, the Classification Learner app defaults to using k-fold cross-validation to evaluate the model's performance during hyperparameter optimization, with the cross-validation loss serving as the objective function for Bayesian optimization. This approach helps ensure that the selected hyperparameters generalize well to unseen data.