How to use parfor with a range from a very large object and cause memory problems

1 ビュー (過去 30 日間)
Jon Ward
Jon Ward 2021 年 11 月 18 日
編集済み: Jon Ward 2021 年 11 月 18 日
How can I achieve the following without gettting the broadcast variable warning and minimising memory usage by Matlab trying to copy the full variable R?
This is a simplified example of what I am trying to achieve, but the memory requirements explode when I try to do this (I am using 48 processors in parallel on a high spec AWS box with lots of memory).
T = 10000;
S = 100; %S is actually ~1000
P = 10;
M = 60;
R = rand(T,S,P,M); % This is an example, R and C below get populated from another process, but this is to illustrate the size of R.
C = rand(T,M);
resultsM = zeros(T,S,P);
resultsSD = zeros(T,S,P);
parfor t=50:T
r = R(1:t,:,:,:); % Broadcast warning here
% Do some stuff with r
c = C(1:t,:);
for s=1:S
for p=1:P
rr = squeeze(r(:,s,p,:));
tmpResults = sum(c.*rr,2);
resultsM(t,s,p) = mean(tmpResults); % Mean and below sd are examples, I apply other custom functions to tmpResults
resultsSD(t,s,p) = std(tmpResults);
  2 件のコメント
Jon Ward
Jon Ward 2021 年 11 月 18 日
Thanks, i have just edited the code to hopefully show more of the type of thing I am trying to do.


回答 (1 件)

Raymond Norris
Raymond Norris 2021 年 11 月 18 日
Have you considered assigning R in the parfor?
parfor t = 1:T
R = rand(T,S,P,M)
It's less intuitive for MATLAB serial code, but may be a better practice for MATLAB parallel code. R is now a temporary variable, which can't be referenced after the parfor.
Also look at ticBytes and tocBytes to see how much data is truely getting passed back and forth.
  1 件のコメント
Jon Ward
Jon Ward 2021 年 11 月 18 日
Unfortunately not, I was using rand to show the size of R only, it actually gets generated from another process. I have edited the code above to show what I am trying to do.



Find more on Loops and Conditional Statements in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by