How robust is the Bayesian Optimization implementation on Matlab. Does the Gaussian Process Regression have a Maximum LIkelihood Selector for Kernel Parameter's and Mean Parameter similar to the sklearn Gaussian Process Regressio?

Gaussian Process Regression (<http://scikit-learn.org/stable/modules/gaussian_process.html>) as mentioned on the site has a maximum likelihood implementation i.e.

"The GP prior mean is assumed to be zero. The prior’s covariance is specified by a passing a kernel object. The hyperparameters of the kernel are optimized during fitting of GaussianProcessRegressor by maximizing the log-marginal-likelihood (LML) based on the passed optimizer. As the LML may have multiple local optima, the optimizer can be started repeatedly by specifying n_restarts_optimizer. The first run is always conducted starting from the initial hyperparameter values of the kernel; subsequent runs are conducted from hyperparameter values that have been chosen randomly from the range of allowed values. If the initial hyperparameters should be kept fixed, None can be passed as optimizer." can you please elaborate on the gaussian process implementation in sklearn and if we have such a facility while implementing the bayesopt function

Answer 1

[Don Mathis]

The 'bayesopt' function uses 'fitrgp' to do Gaussian Process regression. You can read all about it here: https://www.mathworks.com/help/stats/fitrgp.html . It supports holding constant any subset of the kernel parameters you choose, among many other features.

'bayesopt' does not perform multiple restarts of fitrgp in a given iteration. Sometimes there are local minima in the GP solution, often when there are both good high-noise and low-noise solutions, and you can observe bayesopt choosing different minima on different iterations. But choosing different minima on different iterations can be a good thing, because ideally we would like to try all good GP solutions. Sampling from multiple solutions each iteration would be desirable but it is very expensive.

Also, 'bayesopt' does not start each call to fitrgp from the current values of the kernel parameters, because that can inhibit its ability to find other minima of the LML.