Building Distributed/Codistributed Array with .mat files
3 ビュー (過去 30 日間)
古いコメントを表示
I'm trying to run a truncated SVD on a very large (wide) dataset. The dataset will be ~1000x100000, and broken up into multiple pieces by row (25x100000) and saved in different locations/workers as .mat files. If I have a list of these .mat files, is there a way to create a distributed or codistributed array from these files?
I've already tried creating a fileDatastore and converting the results into a distributed, but this just gets me a datastore containing multiple cells. Is there a function similar to cellUnderlying() for distributed arrays? I'm using Matlab R2017a.
0 件のコメント
回答 (1 件)
Oli Tissot
2020 年 8 月 13 日
The following should do what you want:
ds = datastore('A_rowchunk_*.mat', 'Type', 'file', 'ReadFcn', @importdata, 'UniformRead', true);
dA = distributed(ds);
[U, S, V] = svd(dA);
For this to work the files must be located in a (network) location accessible from all the workers, but this seems to already be the case for you.
0 件のコメント
参考
カテゴリ
Help Center および File Exchange で Distributed Arrays についてさらに検索
製品
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!