Prediction of SVM with custom kernel extremely slow

I'm using the Matlab function [fitcsvm][1] for training a SVM with a RBF kernel. I'm using the following calls:

SVMModel = fitcsvm(X_train,labels,'KernelFunction','rbf','KernelScale',0.2087,'BoxConstraint',2.8779);
[~,scores] = predict(SVMModel,X_test);

X_train is a NxD matrix with training data, labels is a Nx1 vector with the labels for the training data and X_test is a MxD matrix with test data points.

Now I would like to use custom kernels. To start with, I decided to try the RBF kernel. The implementation goes as follows:

SVMModel = fitcsvm(X_train,labels,'KernelFunction','rbfKernel','BoxConstraint', 2.8779);
[~,scores] = predict(SVMModel,X_test);

function K = rbfKernel(U,V)
   sigma = 0.2087;
   gamma = 1 ./ (2*(sigma ^2));
   K = exp(-gamma .* pdist2(U,V,'euclidean').^2);
end

I stored the function rbfKernel in a rbfKernel.m file.

The result of the built-in kernel as well as the custom kernel is very similar and the fitcsvm method runs for both approaches very fast.

The problem is that the predict method is extremely slow when using the custom kernel. It takes around 1 minute, compared to 5 seconds with the built-in kernel.

Why is this? Is there a mistake I made?