How to calculate number of floating-point operations (FLOPs) of a CNN model in MatLab

How to calculate number of floating-point operations (FLOPs) of a CNN model in MatLab?

[Walter Roberson]

The number of FLOPs needed to train the model, or the number of FLOPs needed to classify or predict ?

FLOPs is not used much anymore except for academic discussions as an approximation to algorithm costs similar to 'Big-O notation". There are a lot of things that FLOPs do not take into account. For example if you are computing on a GPU, then the common operation A*x + B might well be converted into a single hardware instruction to do Fused Muitiply And Add; https://en.wikipedia.org/wiki/Multiply%E2%80%93accumulate_operation replacing two floating point operations with a single hardware instruction.

[debojit sharma]

Sir, actually I need to calculate FLOPS of my DL model to measure time complexity. This is required in my thesis. So, is there any MATLAB function with which I can calculate this? or please suggest any other method

Answer 1

[Walter Roberson]

Time complexity cannot be measured in FLOPs. Time complexity needs to be measured in "Big-O" notation. For a thesis you would typically measure average case and worst case (unless those are the same.)

Suppose that your program starts with a loop that initializes 1000 one-byte locations to 0 (for example for some kind of histogram, so independent of data size.) How many FLOPs is that? Is it 1000? Is it 1000 for the zeroing, plus 1000 increments for the for loop, plus 1000 comparisons? Is it 125 of each of those because the compiler unrolls the loop and then combines 8 adjacent byte zeros into a single floating point zero assignment? Is it "zero" because of Demand Page Zero hardware?

Whatever it is, for a given compiler it is constant, and that means that the complexity is constant, O(1) in big-O notation. But FLOPs cares about that.

If the algorithm involves a value to a positive integer power, then is the compiler going to use binary decomposition of the power to determine the optimal sequence of multiplicationqq? Is the compiler going with substitute log(), multiply, exponential and if so does that count as 3 FLOPs? Does the compiler emit some kind of hardware exponential instruction and so only count as one FLOP?

None of these are important to time complexity, so no-one measures FLOPs for that purpose.

FLOPs are a hold over from the days when instructions were performed entirely sequentially, with no pipelining and when there was only one ALU (Arithmetic Logic Unit), and were used to calculate actual runtime. FLOPs divided by MIPS (millions of instructions per second) to get seconds of run time. That is not time complexity.

[debojit sharma]

Is there any equivalent function/code in matlab that can perform the following operation as shown in the following python code:

from keras.applications.resnet50 import ResNet50

from keras.applications.vgg16 import VGG16

from keras.applications.mobilenet import MobileNet

from net_flops import net_flops

model = VGG16(weights=None, include_top=True, pooling=None,input_shape=(224,224,3))

model.summary()

#Prints a table with the FLOPS at each layer and total FLOPs

net_flops(model,table=True)

[Walter Roberson]

@debojit sharma

No, there is not.

You could experiment with https://www.mathworks.com/matlabcentral/fileexchange/50608-counting-the-floating-point-operations-flops

There used to be a different File Exchange contribution that allowed you to annotate your code to get approximate flops counts, but I am having difficultly locating it at the moment.

I suggest you also read https://www.mathworks.com/matlabcentral/answers/839375-flops-of-dag-neural-network#answer_709465

In theory it is possible to read the CPU Cycle Counters from the processors -- but those are not necessarily meaningful. See https://devblogs.microsoft.com/oldnewthing/20160429-00/?p=93385

On Windows (not sure about Linux, but not on MacOS) you might be able to use .NET to access performance counters; see https://michaelscodingspot.com/performance-counters/