How can I decrease the runtime for a function (using cellfun or parfor)?

I am trying to speed some code up. Right now each iteration takes about 6 seconds to run and over the course of many iterations the runtime drastically increases. I tried two different running strategies: using cellfun and parallelization. Will paralization ever be slower than using cellfun? I get hugely varying times for the parfor on different computers due to different number of cores. Will the cellfun be more consistent (but slower) across computers? Below is my code. Thank you in advance.

Here is the function call:

%First way
testOut = cellfun(@testDFF_trial, num2cell(fReflect, 2), num2cell(fReflect2, 2), ...
    num2cell(fSom, 2), num2cell(ones(size(fReflect, 1), 1) .* numFrames), 'UniformOutput', false);

%Second way
parfor numCell = 1:size(fReflect, 1)
    dffAll{numCell} = testDFF(fReflect, fReflect2, fSom, numFrames, numCell); %slightly altered function for parfor compatibility
end

Here is the function that I am trying to speed up:

function dffTest = testDFF(fReflect, fReflect2, fSom, numFrames)

stepSize = 1;
windowSize = 900 / 2; %12

startLoc = 451; %20
endLoc = numFrames + 450; %80

%Gets the starting numbers
usedRange = startLoc:stepSize:endLoc;

%Gets the index that matter for each window
aOut = arrayfun(@(a, window) a-window:a+window, usedRange, ones(size(fReflect(usedRange))) .* windowSize, 'UniformOutput', false);
aOut2 = arrayfun(@(a, window) a-window:a+window, usedRange, ones(size(fReflect2(usedRange))) .* windowSize, 'UniformOutput', false);

%Makes ones array for later input
onesAll = (ones(1, length(fReflect)) .* fReflect);
onesAll2 = (ones(1, length(fReflect2)) .* fReflect2);

%Gets the values within each index
myfun = @(input, index) input(index);
datOut = cellfun(@(index) myfun(onesAll, index), aOut, 'UniformOutput', false);
datOut2 = cellfun(@(index) myfun(onesAll2, index), aOut2, 'UniformOutput', false);

%Find the 8th percentile
prcDat = cellfun(@prctile, datOut, num2cell(ones(1, length(datOut)) .* 8));
prcDat2 = cellfun(@prctile, datOut2, num2cell(ones(1, length(datOut2)) .* 8));

%Find the dff
dffFunct = @(rawF, prcF, baseline) (rawF - prcF) ./ baseline;
dffTest = arrayfun(dffFunct, fSom, prcDat, prcDat2);

tInter = toc;
display(['Cell processed - time: ', num2str(tInter / 60, '%0.2f')]);
end

[Stephen23]

"I am trying to speed some code up."

Then get rid of all of those ARRAYFUN and CELLFUN calls. One FOR-loop would be faster. Then you could also avoid things like NUM2CELL: rather than forcing MATLAB to duplicate the data in lots of separate arrays, just use a simple FOR-loop and indexing into some numeric arrays.

"I tried two different running strategies: using cellfun..."

In general CELLFUN is slower than a well-written FOR-loop. Ans you have lots of CELLFUN :(

"... and parallelization"

is no replacement for writing more efficient code in the first place: e.g. replacing all of those ARRAYFUN, CELLFUN, NUM2CELL, etc with a FOR-loop.

[AES]

Thank you for the feedback! I will see what I can do.

[Steven Lord]

FYI, another tool to add to your programming arsenal is the Profiler. This can help you understand what the most expensive location or segment of your code is. Using that you can also experiment with different approaches to determine if a change is better, worse, or the same in terms of performance. [That reminds me of my last eye exam ;)]

[AES]

That sounds familiar, but I have never used it before. Definitely will take a look!

Answer 1

[Voss]

Before worrying about cellfun vs parfor to call the function, try to make the function itself more efficient.

Compare the function testDFF_modified below with the original testDFF.

I don't know what typical inputs to the function are, but with these made-up inputs:

fReflect = rand(1,1000);
fReflect2 = rand(1,1100);
fSom = rand(1,50);
numFrames = 50;

The result is the same:

out1 = testDFF(fReflect, fReflect2, fSom, numFrames);
out2 = testDFF_modified(fReflect, fReflect2, fSom, numFrames);
isequal(out1,out2)

ans = logical
   1

And the modified function is about 3-4 times faster:

t1 = timeit(@()testDFF(fReflect, fReflect2, fSom, numFrames));
t2 = timeit(@()testDFF_modified(fReflect, fReflect2, fSom, numFrames));
fprintf('old: %5.3f ms\nnew: %5.3f ms\n',t1*1000,t2*1000);

old: 7.947 ms
new: 2.114 ms

function dffTest = testDFF_modified(fReflect, fReflect2, fSom, numFrames)

stepSize = 1;
windowSize = 900 / 2; %12

startLoc = 451; %20
endLoc = numFrames + 450; %80

%Gets the starting numbers
usedRange = startLoc:stepSize:endLoc;

% matrix of indices:
idx = usedRange+(-windowSize:windowSize).';

% simple indexing, and taking advantage of the fact that prctile
% given a matrix operates by column:
dffTest = (fSom-prctile(fReflect(idx),8))./prctile(fReflect2(idx),8);

end



function dffTest = testDFF(fReflect, fReflect2, fSom, numFrames)

stepSize = 1;
windowSize = 900 / 2; %12

startLoc = 451; %20
endLoc = numFrames + 450; %80

%Gets the starting numbers
usedRange = startLoc:stepSize:endLoc;

%Gets the index that matter for each window
aOut = arrayfun(@(a, window) a-window:a+window, usedRange, ones(size(fReflect(usedRange))) .* windowSize, 'UniformOutput', false);
aOut2 = arrayfun(@(a, window) a-window:a+window, usedRange, ones(size(fReflect2(usedRange))) .* windowSize, 'UniformOutput', false);

%Makes ones array for later input
onesAll = (ones(1, length(fReflect)) .* fReflect);
onesAll2 = (ones(1, length(fReflect2)) .* fReflect2);

%Gets the values within each index
myfun = @(input, index) input(index);
datOut = cellfun(@(index) myfun(onesAll, index), aOut, 'UniformOutput', false);
datOut2 = cellfun(@(index) myfun(onesAll2, index), aOut2, 'UniformOutput', false);

%Find the 8th percentile
prcDat = cellfun(@prctile, datOut, num2cell(ones(1, length(datOut)) .* 8));
prcDat2 = cellfun(@prctile, datOut2, num2cell(ones(1, length(datOut2)) .* 8));

%Find the dff
dffFunct = @(rawF, prcF, baseline) (rawF - prcF) ./ baseline;
dffTest = arrayfun(dffFunct, fSom, prcDat, prcDat2);

% tInter = toc;
% display(['Cell processed - time: ', num2str(tInter / 60, '%0.2f')]);

end

[AES]

Thank you for this. I understand the code for a input with a single row. But if inputs are a 2xN array the indexes no longer work in fReflect(idx). I know that this is different than my first attempt since that only handles inputs that are 1XN. I was previously using a loop to handle multiple rows. Is there a way to make the index work for inputs that are MXN? I appreciate any insight!

[Voss]

Please share the inputs for testDFF; maybe save them in a mat file and upload the mat file (using the paperclip button).

[AES]

Here are a small chunk of the inputs (due to uploading size limits)

Edit: the numFrames variable should be the size(fSom, 2). I mistakingly saved it incorrectly

[Voss]

Thanks. Here's one way to do it:

function dffTest = testDFF_modified(fReflect, fReflect2, fSom, numFrames)
    
stepSize = 1;
windowSize = 900 / 2; %12

startLoc = 451; %20
endLoc = numFrames + 450; %80

%Gets the starting numbers
usedRange = startLoc:stepSize:endLoc;

% matrix of indices:
idx = usedRange+(-windowSize:windowSize).';

m = size(fReflect,1);
dffTest = zeros(m,numFrames);
for ii = 1:m
    row1 = fReflect(ii,:);
    row2 = fReflect2(ii,:);
    dffTest(ii,:) = (fSom(ii,:)-prctile(row1(idx),8))./prctile(row2(idx),8);
end

end

[AES]

Thank you. This is very helpul and learned a lot,.

[Voss]

You're welcome!