Surrogateopt does not execute objective function in parallel and still uses 120GB of RAM

Hi all,

I am using surrogateopt to optimize an expensive objective function. One evaluation takes approx. 6 minutes.

I run it on a Win 10 PC with 28 cores and 128GB RAM and Matlab 2023a with the Global Optimization Toolbox and the Parallel Computing Toolbox.

Each evaluation of the objective function calls a legacy fortran executable twice. The function value is an average of the two fortran outputs. (see pseudo code below)

The fortran executable is 32 bit and needs ca. 1.5GB of RAM. By my math, the PC should be good for 27 parallel executions of that fortran file.

Each time surrogateopt starts, it does indeed start Poolsize number of parallel fortrans, however, it reverts to serial execution afterwards. At the same time, Matlab uses all available 128GB of RAM. See screenshot attached.

Does anyone have an idea of how to make surrogateopt work "more parallel?" Does surrogateopt have such a high overhead in terms of RAM?

As a sidenote: if I use patternsearch in parallel, I am able to execute 10 (2*length(x)) instances consuming approx. 15 GB of RAM. However, many cores sit idle and I hoped surrogate could help.

Thank you very much!

Florian

lb=[.5  0  0   0  ];
x0=[.9 .5  .2  .25];
ub=[ 1  2  .5  10 ];

opts = optimoptions('surrogateopt',UseParallel=true,UseVectorized=false,Display='iter',PlotFcn='surrogateoptplot',...
            InitialPoints=x0,MaxFunctionEvaluations=1e9,MaxTime=3*24*60^2,ObjectiveLimit=1e-4); 

fixed_para = [1 2]; %some fixed parameter values

xopt = patternsearch(@(x) objective(x),lb,ub,opts);

function val = objective(x)

    for i = 1:2
        
        paraset = [fixed_para(i) , x];  %full set of parameters
        
        %create unique subfolder for each worker
        wrkr = getCurrentTask();
        folder = [num2str(wrkr.ID),"_",num2str(i)]; 
        if ~isfolder(folder)
            copyfile('PricingRevisedDown.exe',folder)   
        end
        
        %write input file for executable
        fid = fopen([folder,'\input.txt'], 'wt'); 
        fprintf(fid,'%s', paraset);
        fclose(fid);
        
        %execute the fortran code using input.txt in folder and wait for termination
        system(['pushd ',folder,' && START /high /wait PricingRevisedDown.exe && popd']);
        
        %read in results
        valtemp(i) = load([folder,'\fortranout.txt']);
    end
    
    val = 2.\(valtemp(1) + valtemp(2));
    
end

Answer 1

[Ashutosh Thakur]

Hi Florian,

I can understand that you want "surrogateopt" to use less RAM and produce better performance. This may be due to limitations of optimization algorithm or a parallel pool configuration. You can follow these suggestions to improve the performance:

• You can explicitly create a parallel pool with desired number of workers using the "parpool" function in MATLAB. Try to keep number of workers as maximum.
• Try to monitor the process usage during execution of the program. Also MATLAB profiler can be used to find the bottlenecks in the program.
• Also try using different optimization algorithms provided by the Global Optimization Toolbox as some algorithms might be better for your usecase.
• Try to use different functions such as "parfor", "parfeval" from the Parallel Computing Toolbox to have a better performance.

Please refer to the following documentation links to have a better understanding of the above suggestions:

• https://www.mathworks.com/help/gads/surrogateopt.html
• https://www.mathworks.com/help/parallel-computing/parpool.html
• https://www.mathworks.com/help/parallel-computing/parfor.html

I hope this helps you in improving the performance.