Writing error in parfor loop

4 ビュー (過去 30 日間)
J T 2015 年 6 月 5 日
コメント済み: Walter Roberson 2015 年 6 月 8 日
Hi everyone,
I have a script that randomly removes and repredicts links in networks 100 times. Of course the bigger the networks are the longer it takes so, I made the 100 iterations into a parfor loop. It works fine for smaller networks but with the big ones I get the error
A write error occurred while sending to worker xy.
I assumed its due to memory problems but I can check in the task manager that the error occurs before the memory of my pc is used up. Also it says writing error...So does anybody have an idea what this error message refers to? Im using windows 7 pro, 64bit and matlab R2014a and I have 196GB of ram(I know quite a lot (:).
Any help would be appreciated! Thanks a lot, Josephine
  3 件のコメント
J T 2015 年 6 月 5 日
Dear Adam, Im sorry Im not so sure what I can post or not, I will have to wait until my supervisor is back on monday. But maybe you can be so kind, to comment the following: The code runs without parfor(just so slow it will probably take years) and needs about 100gb. So if it is that for two workers then u need 200gb of course I get the error. But I tried running the parfor with only one worker, even that does not work. I tried printing out something in the parfor and it seems it doesnt even enter the first iteration. Is it possible, that the transferring alone needs so much memory?? Thanks! Josephine


回答 (2 件)

Thomas Koelen
Thomas Koelen 2015 年 6 月 5 日
Try writing a function that writes the data, then call this function in your parfor loop.
  1 件のコメント
J T 2015 年 6 月 5 日
Dear Thomas, so simplified u mean I should do parfor 1:x; a=f(x); where f=x**2 or so instead of parfor 1:x; a=x**2; ? The function that is doing the major work Im already using like this, I put the rest also in a function now, but unfortunately I get the same error..


Walter Roberson
Walter Roberson 2015 年 6 月 5 日
The data needs to be transferred to even the one worker so Yes you can run out of memory even with one worker.
Have a look at Worker Object Wrapper as it might help in your situation.
  2 件のコメント
Walter Roberson
Walter Roberson 2015 年 6 月 8 日
The documentation is the File Exchange Contribution page and the code itself which is there. It works using handle objects.
There is one part of it that invokes "spmd" as part of the initialization. That suggests that you should open your parfor pool first before using the worker object wrapper.



Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by