GA is suddenly much slower (no other visible effects)

Out of the bluw, a code I use to run a GA optimization has become much (>ten-fold) slower. The specifics are as follows:

Ubuntu 20.04, 64Gb/2.4 GHz RAM, i9-7900X CPU @ 3.30GHz. Not using GPUs for any of my MATLAB (2018a) operations.

I won't paste the whole code, but a short pseudocode is provided for the function:

fopt = @(x)MyFun(x,pars);

tic;[x,fval] = ga(fopt,N,[],[],[],[],Lower,Upper,[],[],opt.options);toc

function f = MyFun(x,pars)

%LOTS OF STUFF

for j = 1:numel(pars(1))

%LOTS OF STUFF

end

%f = result of lots of stuffy calcuations

Let's have a look at opt.options:

opt.options

ans = 

  ga options:

   Set properties:
             ConstraintTolerance: 1.0000e-08
               CrossoverFraction: 0.5000
                         Display: 'iter'
                      EliteCount: 55
               FunctionTolerance: 3.0000e-31
         InitialPopulationMatrix: [65×7 double]
                  MaxGenerations: 444
             MaxStallGenerations: 1
    NonlinearConstraintAlgorithm: 'penalty'
                       OutputFcn: @getoutput
                  PopulationSize: 400
                     UseParallel: 1

   Default properties:
                     CreationFcn: @gacreationuniform
                    CrossoverFcn: @crossoverscattered
                    FitnessLimit: -Inf
               FitnessScalingFcn: @fitscalingrank
                       HybridFcn: []
          InitialPopulationRange: []
             InitialScoresMatrix: []
                    MaxStallTime: Inf
                         MaxTime: Inf
                     MutationFcn: {@mutationgaussian  [1]  [1]}
                         PlotFcn: []
                  PopulationType: 'doubleVector'
                    SelectionFcn: @selectionstochunif
                   UseVectorized: 0

This code used to run @1.3 s per GA generation with a popualtion of 400. One day, out of the blue, from one run to the next, without even restarting MATLAB, the run became much slower (over a minute per generation). I couldn't think of anything, so instead of reinstalling MATLAB, I recovered an earlier OS image. The code now ran again @1.3 s per GA generation for two times, but the third time I ran it.... again, about ten times slower.

The things to note:

1. The time for GA generation seems to be independent of population size! I have never encountered anything like this. I've checked population sizes in the range 66 to 400, and all now run at around 18 s per generation.
2. Other calculations, including using parpools, are unaffected. For instance, with an active parpool, the command:

                   tic;parfor j = 1:30; inv(rand(3e3));end;toc

used to be resolved on average in around seven seconds. This calculation time is not affected by whatever is plaguing the GA module.

Any suggestions would be greatly appreciated.

[Tom Brenner]

Comment: when recovering images of my OS, the behavior of GA is fixed. When the system asks to be updated, though, the GA becomes very slow again.

On a sidenote: I'm a veteran user of MATLAB. Oh how I yearn for the old days when literally not a single question was left unanswered on this forum.....

[Alan Weiss]

I really don't know what is going on. I suggest that you raise the issue with technical support.

Alan Weiss

MATLAB mathematical toolbox documentation

[Catalytic]

"On a sidenote: I'm a veteran user of MATLAB. Oh how I yearn for the old days when literally not a single question was left unanswered on this forum....."

As a veteran, you should know that you haven't provided nearly the info that would be needed to investigate this problem. Does the problem occur for every fitness function you try to optimize, or just this particular one? If it's just one particular problem, then obviously the details of the fitness function would be important, and you haven't given us any info about it, other than that it contains a loop.

[Tom Brenner]

Dear Alan Weiss: Thank you for peeking in. It is rather peculiar.

Dear Catalytic: I play around very little with MATLAB nowadays (my actual work involves python). I don't have detailed records of performance for different functions before and after the update. To prepare such now would require me to once again recover an earlier OS image, then run a bunch of optimizations before and after updating. I'm not sure I'm willing to go on such a goose hunt for this esoteric issue. Thank you for the suggestion, though; doing that could indeed narrow down the problem to something generic affecting the GA module vs. something that affects only MyFun. Thanks again.

[Matt J]

One day, out of the blue, from one run to the next, without even restarting MATLAB, the run became much slower

What if you do restart Matlab? Do things improve? What if you reboot?

[Tom Brenner]

Matt J: nope and nope. The only way to get back the original performace is by recovering the OS image from before the recent update. As soon as ubuntu is updated, though, it reverts back to very slow execution.

[Tom Brenner]

Matt J's first comment on his answer forms a solution of sorts, even if unintentional.

Answer 1

[Matt J]

As a troubleshooting strategy, I suggest putting a tic...toc as the first and last lines of the fitness function. This way you can see if the slowdown is in the execution of the fitness function itself or something else. If the fitness function is slowing down, maybe the slowdown occurs for very specific population members.

[Tom Brenner]

Thank you for the suggestion. Is there a way to collect the tic/toc results of the function for each population member, from each generation in a GA (say, in the same way that final population is saved, cf. OutputFcn: @getoutput) run? Because otherwise, I'd be printing the results of each population member of each run to the screen, and I'd have to scroll through all of them to search for outlier values.

[Matt J]

If you nest the fitness function, you can save data to the parent workspace like in the following example:

Times=main()

Optimization terminated: maximum number of generations exceeded.

Times = 1×4753

1.0e-03 * 0.1190 0.0240 0.0120 0.0100 0.0960 0.0130 0.0030 0.0010 0.0010 0.0010 0.0010 0.0020 0.0010 0.0010 0.0010 0.0010 0 0.0010 0.0010 0.0010 0.0010 0.0010 0.0010 0.0010 0.0010 0.0010 0.0010 0.0010 0.0010 0.0010

function Times=main

    Times=[];
    
    xopt=ga(@fitnessFunc,1);


          function f=fitnessFunc(x)
             tic; 
              f=norm(x);
             t=toc;
             
             Times=[Times,t];
             
            end

end

[Matt J]

You can also insert keyboard() commands or breakpoints to pause execution and examine the workspace of the fitness function when the execution time exceeds a certain threshold, e.g,

          function f=fitnessFunc(x)
             tic; 
              f=norm(x);
             t=toc;
             
             if t>threshold
                keyboard 
             end
            end

[Tom Brenner]

Because the offending function requires additional input parameters, I haven't found to way to implement your code to obtain the Times parameters including individual tocs. My code looks like this:

function Times=main(pars,N,Lower,Upper,opt)
    Times=[];
    
    xopt=ga(@(x)MyFun(x,pars),N,[],[],[],[],Lower,Upper,[],[],opt.options);
    function f=MyFun(x,pars)
             tic; 
              %%%Lots of code
             t=toc;
             
             Times=[Times,t];
             
            end
end

This implementation does not build up the Times parameter, but instead retains only a single value (just to be clear: I've managed to get it to build up the Times parameter with all t values when running it for other functions that don't require additional input parameters, using your original code).

While I can't use your method, however, the running time per generation has now gone back down to about 1.3 s, instead of 18 s!!!! This is truly baffling, but since it is doing what I want it to do, I will accept your answer. To be clear, running the code directly without nesting in within the main function still runs at about 18 s per generation.

Thanks again!