How to do a ROC analysis using Matlab build-in SVM (Not LibSVM)

Hi all,

Just wondering anyone knows how to do a ROC analysis using Matlab build-in SVM? This question has been asked by millions of times on the web, but no answer.

svmStruct                               = svmtrain(featureSelcted, groundTruthGroup, 'Kernel_Function', 'rbf', 'Method', 'QP');
svmClassified                           = svmclassify(svmStruct,featureSelcted);

% Predict resubstitution response of SVM classifier
SVMScore ???

% Fit probabilities for scores
[FPR, TPR, Thr, AUC, OPTROCPT]          = perfcurve(groundTruth(:,1), SVMScore(:,1), 1);

Essentially, we need a function to get the '*scores*' of the SVM classifier (SVMScore). Thanks!

A.

Answer 1

[Ilya]

Well, there is an answer here http://www.mathworks.com/matlabcentral/answers/64475-does-anybody-have-expertise-with-matlab-svm-classifier, with a reference to another thread. I am collecting all the pieces in one place.

Assume your class labels are -1 and +1, assume that you have trained with 'autoscale' set to true by default, let svm be the struct for the trained SVM model, and let Xnew be the new data for which you need to compute the soft scores.

shift = svm.ScaleData.shift;
scale = svm.ScaleData.scaleFactor;

Xnew = bsxfun(@plus,Xnew,shift);
Xnew = bsxfun(@times,Xnew,scale);

sv = svm.SupportVectors;
alphaHat = svm.Alpha;
bias = svm.Bias;
kfun = svm.KernelFunction;
kfunargs = svm.KernelFunctionArgs;

f = kfun(sv,Xnew,kfunargs{:})'*alphaHat(:) + bias;
f = -f; % flip the sign to get the score for the +1 class

Then call perfcurve(true_labels,f,1).

[Aaronne]

Hi Ilya,

That's quite handy. Thanks so much. But more detailed questions:

1. I have noticed that the predict scores are posterior probabilities for classes obj predicts for the discriminant classification methods in Matlab like KNN and LDA, for example:

%%Construct a LDA classifier
LDAClassifierObject = ClassificationDiscriminant.fit(featureSelcted, groundTruthGroup, 'DiscrimType', 'linear');

% Predict resubstitution response of LDA classifier
[LDALabel, LDAScore] = resubPredict(LDAClassifierObject);

% Fit probabilities for scores
[FPR, TPR, Thr, AUC, OPTROCPT] = perfcurve(groundTruthNumericalLable(:,1), LDAScore(:,1), 1);

Then the scores, ie, LDAScore stores all the posterior probabilities ranged within [0,1].

However, the scores calculated by the svmStruct (the struct for the trained SVM model) has many values larger than 1 and some negitive values. Should we do some normalization for this? For example:

SVMScore = kfun(sv,Xnew,kfunargs{:})'*alphaHat(:) + bias;
SVMScore = SVMScore./max(SVMScore);

2. I am not sure why we need this flipping step:

SVMScore = kfun(sv,Xnew,kfunargs{:})'*alphaHat(:) + bias;
SVMScore = -SVMScore; % flip the sign to get the score for the +1 class

It seems to me that I define the ground truth of the first class to label 1 and the second class to label -1. I don't need the flipping.

3. Also want to make sure that the 'Xnew' variable you used is the 'training data'. For the cross-validation case, 'Xnew' is the training data in each fold, am I right?

4. If you have time, could you please explain a little bit more about this equation:

SVMScore = kfun(sv,Xnew,kfunargs{:})'*alphaHat(:) + bias;

Where did Matlab documents mentioned about this? Otherwise, I will google to see if I can fully understand this.

Looking forward to your replies. Thanks very much again.

A.

[Ilya]

1. Your observation that SVM scores do not represent posterior probabilities is correct. Whether you need to normalize them or not depends on what you want to do with them. For perfcurve, only score ranks matter and normalization is not needed.

2. I said in my answer "Assume your class labels are -1 and +1". If you pass groundTruthGroup as an array of -1 and +1, svmtrain takes -1 to be the positive class. That's why you need to flip. If you use different labels, figure out what the positive class is assumed by svmtrain.

3. Xnew is whatever you want it to be - it's the data for which you want to compute predictions.

4. This is not described in the doc. Because this behavior is undocumented, it may change in a future release. You could google, but a better approach could be to read an introductory text on SVM. Compare this line of code with the expression for the SVM score. Hint: alphaHat here are coefficients alpha multiplied by the class labels (-1 or +1).

[nur zawatil isqi]

May I know does this technique applicable for multi class svm?I'm also looking for plotting ROC curve too. Thank you.

[Ilya]

svmtrain does not support more than two classes, and ROC curve is not a well-defined concept for more than two classes.

[FARHAD]

Hi, I have specified poslabel=[poslabel 1] and neglabel=[neglabel -1]. So why svmtrain considers -1 as negative label?

[Tahir]

Hi, i just use the code ...but i have confusion about the line

f = -f; % flip the sign to get the score for the +1 class
why this flipping?? i use labels 0 and 1...still i should flip the f.??

[Farag Zbeda]

Hi Ilya

What do yo mean by true_labels when calling perfcurve(true_labels,f,1)? I guess they are the test instances labels. If so, how to calculate them?

Thanks

Faraj

Answer 2

[Mehrdad Heydarzadeh]

Hi Aaronne, It seems that your problem has been answered. But there is a shorter way of doing that using Matlab functions. Just have a look at the second example in following link: http://www.mathworks.com/help/stats/perfcurve.html