Continue to next iteration in for-loop while saving data

6 ビュー (過去 30 日間)
Weixin Wang
Weixin Wang 2021 年 4 月 20 日
回答済み: Thomas Falch 2021 年 10 月 1 日
I have a for-loop and in each iteration it produces about 1Gb data. Since it takes some time to save this data to hard drive, I want to use a different worker to save the data and let the for-loop continue to the next iteration at the same time on the original worker. Is there any way to do this? Thank you.
Edit: The calculations in each iteration before saving data takes longer time than saving to disk, so the current worker won't reach the save command in the next iteration before the other worker finishes save for the current iteration.
  3 件のコメント
Weixin Wang
Weixin Wang 2021 年 4 月 20 日
@Image Analyst Thank you for your comment. I'm sorry I made some confusions. The calculations in each iteration before saving data takes much longer than saving data, so it won't reach the next save command if it continues to the next iteration and leaves another worker to perform save.
messaoudi nada
messaoudi nada 2021 年 9 月 24 日
dear @Image Analyst , i need your help please! , im working about machine learning models , so after extracting features from images i have a 12 structure data ,i want to save the data in a reference matrix and test matrix ,i tried this loop but it didn't work it just save the features of the last image in structure ! i don't know what's wrong heare ! hope u can give help to me !
thanks in advance
for i=1:12
for iref=1:10
for jref=1:40
F1=data1_rf.(['p',num2str(iref)]).(['image',num2str(jref)]).Features;
for itst=1:2
for jtst=1:40
TSF=data1_T.(['p',num2str(itst)]).(['image',num2str(jtst)]).Features;
end
end
end
end
end

サインインしてコメントする。

回答 (1 件)

Thomas Falch
Thomas Falch 2021 年 10 月 1 日
You can use parfeval to run save in the background on a worker like this:
pool = parpool('local', 1) % Start pool with one worker
for i = 1:100
data = generateData()
parfeval(@() save(data), 0); % Sends data to worker to be saved in the background
end

カテゴリ

Help Center および File ExchangeParallel Computing についてさらに検索

製品


リリース

R2021a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by