Saving output at each function iteration for a minimise function

hyp2 = minimize(hyp2, @gp, -1000000000, inf, meanfunc, covfunc, likfunc, x, y)

i use this command for optimising the hyper parameters present in meanfunc, covfunc, likfunc, which sometimes lead up to around 165 parameters to optimise. this takes a lot of time to simulate and takes around 2 to 3 months and while it is simulating the iteration doesnt get saved and only the last value gets saved. so when we stop at the middle due to unexpected circumstances such as power failure, system crash etc, i would have to start again from the beginning. hence is there any solution where for each iteration the solution gets autosaved and then the next iteration gets performed?

Answer 1

[Deepak]

Hi Avinash,

To address the issue of long optimization times and potential data loss due to unexpected interruptions, we can implement a checkpointing mechanism. This involves saving the current state of your optimization process at regular intervals, it can be resumed from the last saved state.

To implement checkpointing in the optimization process, wrap the “minimize” function call in smaller chunks of iterations instead of running it for large number of iterations at once, saving the state after each chunk.

Use “save” and “load” functions of MATLAB to store and retrieve the current state of the hyperparameters and any other necessary information.

Before starting optimization, check if a saved state exists; if it does, load it and resume from that point to ensure continuity in case of interruptions.

Below is a sample MATLAB code to achieve the same:

function optimize_with_checkpointing()
% Define your input data and functions
meanfunc = ...; % Define your mean function
    covfunc = ...;  % Define your covariance function
    likfunc = ...;  % Define your likelihood function
    x = ...;        % Your input data
    y = ...;        % Your output data
    
% Check if a checkpoint file exists
checkpoint_file = 'checkpoint.mat';
if isfile(checkpoint_file)
    load(checkpoint_file, 'hyp2', 'start_iteration');
    fprintf('Resuming from iteration %d\n', start_iteration);
else
    hyp2 = ...; % Initialize your hyperparameters
        start_iteration = 0;
end

total_iterations = 1000000000; % Total number of iterations you want
chunk_size = 1000; % Number of iterations per chunk
save_interval = 1000; % Save after every 1000 iterations

for iteration = start_iteration:chunk_size:total_iterations
    % Perform optimization for a chunk of iterations
    hyp2 = minimize(hyp2, @gp, -chunk_size, inf, meanfunc, covfunc, likfunc, x, y);
    
    % Update the start iteration for the next loop
    start_iteration = iteration + chunk_size;
    
    % Save the current state
    save(checkpoint_file, 'hyp2', 'start_iteration');
    fprintf('Checkpoint saved at iteration %d\n', start_iteration);
end

% Save final results
save('final_results.mat', 'hyp2');
fprintf('Optimization completed.\n');
end

Please find attached the documentation of functions used for reference:

minimize: www.mathworks.com/matlabcentral/fileexchange/24298-minimize

save: www.mathworks.com/help/matlab/ref/save.html

load: www.mathworks.com/help/matlab/ref/load.html

I hope this will help in resolving the issue.