Please tell us about feature selection and enlighten us.

Please tell us about feature selection and enlighten us.

One of the built-in feature selection algorithms, out-of-bag for random forests in classification trees, was employed to select features for machine learning (Selecting predictors for random forests - MATLAB & Simulink - MathWorks Japan) to indicate the importance of the predictors.

A histogram was created and sorted in order of the predictors' values, and from the histogram, the predictors were fed into the machine learning model by identifying where the importance of the predictors differed significantly and selecting only those that were greater than or equal to the histogram.

Here, can we consider the method of finding the histograms, the parts that differ significantly visually, and selecting thresholds as one of the various options for narrowing down the number of features?

We would be very grateful if you could provide us with some guidance.

Thank you very much in advance.

Answer 1

[Prasanna]

Hi Takeharu,

Feature selection is a critical step in the machine learning pipeline that involves selecting a subset of relevant features (predictors) for use in model construction. One of the built-in feature selection algorithms in MATLAB is the out-of-bag (OOB) error estimation for random forests in classification trees. This method evaluates the importance of each predictor by measuring how much the prediction error increases when the values of that predictor are permuted while all others are left unchanged.

The method of using histograms and visual inspection to set a threshold for feature selection can indeed be considered a valid approach. It is a practical way to narrow down the number of features by leveraging the importance scores provided by the random forest algorithm. While it may not be as rigorous as some other statistical methods, it offers a straightforward and intuitive means of feature selection, especially when dealing with many predictors. However, it is essential to note that this approach has its limitations. The choice of threshold may vary depending on the specific dataset and problem at hand. Additionally, visual inspection might not always capture subtle but significant differences in predictor importance.

To enhance the robustness of your feature selection process, you might consider combining this method with other techniques such as

• cross-validation
• recursive feature elimination
• regularization methods.

By combining the visual inspection method with these additional techniques, you can create a more robust and reliable feature selection process that enhances the performance and interpretability of your machine learning models.

Refer the following links for more information on feature selection and improving classification trees:

• https://www.mathworks.com/help/stats/improving-classification-trees-and-regression-trees.html
• https://www.mathworks.com/help/stats/classreg.learning.regr.regressionbaggedensemble.oobpermutedpredictorimportance.html

Hope this helps.