Use Parfor to split a 4D matrix into several 3D matrices

Hey,

so I to change my current code:

for i=1:d(4)
    save_file_name=sprintf('layer_%i.mat',i); %name of the save file
    variable_name=sprintf('layer_%i',i); %name of the variable
    reduced_dimension=squeeze(reducedData(:,:,:,i)); %reduces the size of the matrix down to 3 since

      eval(['layer_' num2str(i) '=reduced_dimension;']);%renames the variable 
      save(save_file_name,variable_name);%saves the files

       eval(['clear ', 'layer_' num2str(i) ])
  end

This code is supposed to split a giant 4D Matrix into several smaller 3D matrices while automatically renaming and saving them separatly. I am aware that using eval is considered to be bad, but so far I have not found any other solution.

I want to speed up the whole process by using a parfor loop, but apparently "eval" and "save" are not allowed in this kind of loop. Has anyone an idea how to solve this kind of problem ?

[OCDER]

What you're trying to do seems to be going in the wrong direction in terms of code efficiency... Seems like you already have the code to split a 4D to 3D, and you're already aware that using eval to rename variables is a bad practice. Note that using parfor won't help much because you are limited by the hard drive write speed. It'll be better if you tell us:

• What is the real task you'd like to accomplish AFTER you split the 4D matrix?
• Why do you need to split this 4D matrix?

[Stephen23]

"Use Parfor to split a 4D matrix into several 3D matrices"

You can already access the data in the 4D array very efficiently using indexing (and indexing is very efficient). In contrast, what you are trying to do with eval will only force you into writing slow, complex, buggy code. For example, by using eval you remove any ability of the MATLAB JIT engine to speed up your code, so you automatically force your code into being pointlessly slow, all of the time:

https://www.mathworks.com/matlabcentral/answers/304528-tutorial-why-variables-should-not-be-named-dynamically-eval

If you used simple indexing then MATLAB would speed up your code automatically.

"I want to speed up the whole process by using a parfor loop, but apparently "eval" and "save" are not allowed in this kind of loop"

If you are planning on writing fast and efficient code (which is usually what beginners intend when they start talking about parfor) then your approach is in entirely the wrong direction. If you really want to write efficient code:

• do NOT make unnecessary duplicates of data in memory.
• do NOT use eval or anything else to magically access variable names.
• follow the guidelines in the MATLAB documentation:

https://www.mathworks.com/help/matlab/matlab_prog/techniques-for-improving-performance.html

Answer 1

[Titus Edelhofer]

Hi,

I agree with OCDER: parfor will only do huge memory movements from the Client to the workers but I doubt you will get much speedup.

If the "save" is indeed too slow, you might experiment with saving without compression: use "-v6" as flag when saving. The files are then probably much bigger but often saving is faster.

And by the way: if you have the file name with the layer number, why bothering what the variable name is? Why don't simply use "layer" for all of them?

Titus

Answer 2

[Stephen23]

Firstly it is important to note that processing a sequence of .mat files is much simpler when the variable names inside each file are exactly the same. This applies to both reading and writing them. Then you could trivially do this:

for slice = 1:d(4)
    file = sprintf('layer_%d.mat',slice);
    data = squeeze(reducedData(:,:,:,slice));
    save(file,'data','slice')
end

Note how I wrote simpler, much more efficient code than what you have in your question, and that I saved the slice value too, so you know exactly which slice it is when you load that data:

S = load(...)
S.slice
S.data

Simpler to write, simpler to read, much more efficient: why waste your time fighting eval ?

If you really want to have lots of numbered variables (which are a sign of bad data design, and bad code), then you could split the data into the fields of a structure, and save that structure using the '-struct' option. But of course that would be slow, complex, and not recommended.