Matlab parfor saves and loads temporary variables during execution!

2 ビュー (過去 30 日間)
SK 2018 年 7 月 21 日
編集済み: Matt J 2018 年 7 月 29 日
The parallel pool implementation appears to save and load variables using effectively the normal "save" and "load" channels when making copies to be passed to the workers. This is really bad because if someone has a saveobj() implemented in a class object the state of the variable could be modified during execution and depending on the circumstances could lead to "unexplained" crashes.
Lets say I have a class object that needs a large amount of temporary data during execution. When saved I would naturally want to get rid of this temporary data. A convenient way to do this is to write a saveobj that clears the temporary data within the class. If, however, the object is saved and reloaded during execution as with parfor, there are problems.
Why is the copy not made internally?
  1 件のコメント
Matt J
Matt J 2018 年 7 月 29 日
編集済み: Matt J 2018 年 7 月 29 日
Another situation where this will be an issue is when calling PACK. The documentation doesn't mention that saveobj is called when pack() is invoked. But it is.



Matt J
Matt J 2018 年 7 月 21 日
編集済み: Matt J 2018 年 7 月 21 日
I don't know why authoritatively, but it seems to me that this is only a danger if you are unaware of this behavior on the part of parfor. If you are aware of it, you would surely write a loadobj() method to restore the temporary data when reloaded. And, this would be better than broadcasting copies of the temporary data to all the workers. The latter is done serially and would take a lot more time.
Incidentally, have you verified that saveobj() is triggered in this scenario just as in an ordinary call to save()?
  7 件のコメント
SK 2018 年 7 月 29 日
Unfortunately, it doesn't look like ClassificationXXX classes let you
do this, but that might be what ClassificationEnsemble classes are
intended for.
Yes, but you may want to roll your own ensemble classes (as in my case) in a way that doesn't fit into the ClassificationEnsemble mould.
Regarding your other remarks, you are essentially asking me to "put up or shut up". I've been doing that for a while with Matlab but I do make the occasional post with my criticisms which I think are valid ones.
In my opinion, if save() can be customized via saveobj(), then the parallel pool should not be using it as a means of passing data between processes. Data transfer can be done in some other way, avoiding the custom saveobj(). The effect should be to make a deep copy of the object and save its bits and nothing else. Of course it would involve a little more effort on the part of implementors.
Anyway, I don't want to sound like I'm giving advice to those who write the internals of Matlab since I have no idea of the problems they face in the context of the overall design of Matlab. Moreover, even if they are doing something wrong, the software is theirs to mess up and not mine. So I would prefer to end the discussion here.
Thank you for your comments.


その他の回答 (1 件)

Edric Ellis
Edric Ellis 2018 年 7 月 23 日
The workers executing the body of any parfor loop are separate MATLAB processes, so the only reliable way that variables existing on the client can be sent to those workers is by doing something equivalent to calling save on the client, and then load on the workers. (The same procedure is used, but no files are created on disk).
I must admit, it's not clear to me what the benefit is of writing a saveobj that cannot be reversed by a loadobj. (Also, have you considered using Transient fields in your class?)
  1 件のコメント
SK 2018 年 7 月 28 日
編集済み: SK 2018 年 7 月 28 日
Sorry, I didn't see this post earlier.
Let's say a class needs to do some long computation where its (typically private) methods need to share a number of variables that are not needed after the long computation. Instead of passing these variables back and forth between methods, these common variables are stored in the class object. This issue is particularly relevant for a language designed for computation: It is not unusual for a class to be designed to do a few complex computations on any given piece of data - the results of interest are often best stored in a struct so that they can be easily shared with other people or with other classes that do not need to know about how the results were created. The actual class object is only used during the course of the computation which can be long and complex.
Another concrete example, but different in nature from the previous one, involving one of the Matlab toolboxes: Look at any of the ClassificationXXX classes in Matlab. They have a a method called compact(). This method empties the data from the variable X in the class object. If multiple such classes are created with the same data (this happens often), then a natural thing to do before saving an object of this class would be to save one instance of X somewhere and call compact() on all the objects. However you don't want this to happen in the middle of a computation.
'Transient' is convenient: Thanks for the tip (I had seen it before but had forgotten about it). However it doesn't help in this context, because the object state needs to be restored.
For the moment my gut_on_save flag seems to do the job.



Find more on Data Type Identification in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by