How to use parfor with a range from a very large object and cause memory problems

How can I achieve the following without gettting the broadcast variable warning and minimising memory usage by Matlab trying to copy the full variable R?

This is a simplified example of what I am trying to achieve, but the memory requirements explode when I try to do this (I am using 48 processors in parallel on a high spec AWS box with lots of memory).

Thanks!

T = 10000;
S = 100; %S is actually ~1000
P = 10;
M = 60;
R = rand(T,S,P,M); % This is an example, R and C below get populated from another process, but this is to illustrate the size of R.
C = rand(T,M);
resultsM = zeros(T,S,P);
resultsSD = zeros(T,S,P);

parfor t=50:T
    
    r = R(1:t,:,:,:); % Broadcast warning here
    % Do some stuff with r
    c = C(1:t,:);
    
    for s=1:S
        
        for p=1:P
            
            rr = squeeze(r(:,s,p,:));
            tmpResults = sum(c.*rr,2);
            resultsM(t,s,p) = mean(tmpResults); % Mean and below sd are examples, I apply other custom functions to tmpResults
            resultsSD(t,s,p) = std(tmpResults);
            
        end
        
    end
    
end

[Matt J]

The solution would depend on the "Do some stuff with r" part of the code.

[Jon Ward]

Thanks, i have just edited the code to hopefully show more of the type of thing I am trying to do.

Answer 1

[Raymond Norris]

Have you considered assigning R in the parfor?

parfor t = 1:T
    R = rand(T,S,P,M)
    ...
end

It's less intuitive for MATLAB serial code, but may be a better practice for MATLAB parallel code. R is now a temporary variable, which can't be referenced after the parfor.

Also look at ticBytes and tocBytes to see how much data is truely getting passed back and forth.

[Jon Ward]

Unfortunately not, I was using rand to show the size of R only, it actually gets generated from another process. I have edited the code above to show what I am trying to do.