Can you train a classification model to prioritise precision?

I'm working on a binary classification problem where the precision of classifying a particular class is far more valuable than overall classification accuracy. For example, a case where 5 observations are classifed as class 1 correclty and 95 observations of class 1 are incorreclty identified as class 0, would be more desirable than the opposite. i.e. I don't mind misclassifying class 1 as 0, as long as the observations classed as 1 are highly accurate.

Is there a way to train a model, for example using fitcsvm, to obtain this behaviour? Also any examples/methods/keywords related to this kind of objective would be highly appreceated.

Many thanks

Answer 1

[Shubham]

Hi Jack,

In the scenario you've described, you are prioritizing the precision for a particular class, which means you want to minimize false positives for that class. In other words, when the model predicts class 1, you want to be very sure that it is indeed class 1, even if that means accepting more false negatives (class 1 instances predicted as class 0).

To train a model with a focus on precision for a particular class using MATLAB's fitcsvm, you can use several techniques:

1. Class Weights: You can assign a higher weight to the class you are interested in during training, which will make the classifier more sensitive to misclassifying that class. This doesn't directly optimize for precision but can help to shift the decision boundary in a way that might improve precision for the weighted class.
2. Custom Cost Matrix: You can define a custom cost matrix that penalizes the model more for false positives for the class of interest. This directly relates to precision, as it will make the model more conservative about predicting the positive class.
3. Precision-Recall Trade-off: After training the model, you can adjust the decision threshold to be more conservative about predicting class 1. This doesn't change the training process but allows you to control the trade-off between precision and recall after the model is trained.
4. Optimize for Precision: Use a custom performance metric for model selection during cross-validation. You can use crossval with a custom scoring function that focuses on precision for the class of interest.

Note: Optimizing for precision often comes at the cost of recall (or sensitivity), meaning that you may miss more true positives for class 1. It's important to find a balance that meets your specific needs.