Why does Support Vector Regression (fitrsvm) optimization result in poor performance ?

Hi

I am working on a numerical prediction problem (load forecasting). I have a predictor matrix which consists of around 20 numerical variables (X_train: including historical lags, calendar information, temperature etc) and an output vector of real/target load values (Load_train). The data consists around 10.000 points

I am following the below documentation on Support Vector Regression, in particular the section 'Optimize SVM Regression'. https://matlabacademy.mathworks.com/R2016b/portal.html?course=mlml#chapter=4&lesson=7&section=3

However after the exhaustive 'hyperparameter optimization' I get poor prediction performance, especially compared to a simpler SVR which uses Gaussian Kernel, Kernel Scale 'auto' and all other parameters set to default without any optimization. I do standardize the input and output matrices before the training. Please find below the lines for two training procedure:

"Simple SVR"

mdl_simple=fitrsvm(X_train,Load_train,'Standardize',true,
    'KernelFunction','gaussian','KernelScale','auto');

"Optimized SVR"

Mdl_optimized = fitrsvm(X,Y,'Standardize','true','OptimizeHyperparameters','auto',...
    'HyperparameterOptimizationOptions',struct('AcquisitionFunctionName',
    'expected-improvement-plus'))

Would anyone have any advice with regards to the optimization procedure or any ideas why optimized SVR might be giving worse results than simpler SVR?

Best regards

Baran

[Walter Roberson]

Generally speaking, when more sophisticated fitting procedures end up giving worse results in practice, the problem can be due to overfitting.

[antlhem]

Could take a look into my question? https://uk.mathworks.com/matlabcentral/answers/842800-why-matlab-svr-is-not-working-for-exponential-data-and-works-well-with-data-that-fluctuates?s_tid=prof_contriblnk

Answer 1

[Don Mathis]

Maybe it's because the optimized SVR uses the default kernel function, which is 'linear'. If you include 'KernelFunction','gaussian' in your second call to fitrsvm, as you did in your first, it might perform better.

Secondarily, you could try running the optimization longer by adding the field 'MaxObjectiveEvaluations' to your struct:

Mdl_optimized = fitrsvm(X,Y,'KernelFunction','gaussian','Standardize',true,'OptimizeHyperparameters','auto',... 'HyperparameterOptimizationOptions',struct('AcquisitionFunctionName', 'expected-improvement-plus', 'MaxObjectiveEvaluations', 60))

[Baran Yildiz]

Hi Don Thanks for the answer. I had previously included gaussian as the Kernel Function type. But your suggestion of running the optimization longer with 'MaxObjectiveEvaluations' argument worked this time. The results are slightly better than the non-optimized SVR... For my problem, i am not sure if this slight improvement worths the extra time to spent on the optimization.

Cheers Baran

[Don Mathis]

I just noticed your reply. If you have some time on your hands you could try

    Mdl_optimized = fitrsvm(X,Y,'OptimizeHyperparameters','all', 'HyperparameterOptimizationOptions',struct('MaxObjectiveEvaluations', Inf, 'SaveIntermediateResults',true))

This optimizes over 6 hyperparameters until you press Control-C, at which point you'll find an object called BayesoptResults in your workspace. The best hyperparameters are obtained using

    bestPoint(BayesoptResults)

which you will then need to pass to a new call to fitrsvm manually.