What's the upper limit for how much data can be stored in a structure?

26 ビュー (過去 30 日間)
Ibro Tutic
Ibro Tutic 2017 年 5 月 15 日
コメント済み: Ibro Tutic 2017 年 5 月 15 日
I am working with about 20 datasets, each with a size of about 2500x300. Is it possible to open these datasets, store them, and save off the resulting structure as a mat file without running into memory issues? This is my current code, and I run into an "Out of memory" error when I run it.
for i=1:length(fieldnames(dat))
a=fieldnames(dat); %files to work with
a=a{i}; %selects upper structure
a2=fieldnames(foldername); %foldername where .dat files are held
a2=a2{i}; %select which folder
for j=1:length(fieldnames(dat.(strcat(a))))
b=fieldnames(dat.(strcat(a))); %list of filenames in upper structure
b=b{j}; %select file to open
for k=1:length(dat.(strcat(a)).(strcat(b)))
filepath = strcat(foldername.(strcat(a2)),'\', dat.(strcat(a)).(strcat(b))); %sets file path
fid = fopen(filepath{k}); %opens file
fmt = repmat('%s', 1, 360); %reads file, text string delimter
C= textscan(fid,fmt,'Delimiter','\t','CollectOutput',true); %records csv data to cell array C
fclose(fid); %closes file
DLG.(strcat(a)).(strcat(b)){k}=C{1}; %writes data to structure
end
end
end
  4 件のコメント
Guillaume
Guillaume 2017 年 5 月 15 日
編集済み: Guillaume 2017 年 5 月 15 日
"I can't call Dat.a"
Indeed, the proper syntax is simply:
dat.(a)
No need for the strcat that does absolutely nothing.
Ibro Tutic
Ibro Tutic 2017 年 5 月 15 日
Thanks for that, works well.

サインインしてコメントする。

採用された回答

Guillaume
Guillaume 2017 年 5 月 15 日
Your code is a bit confusing with all these apparently useless strcat (see comment to the question).
Anyway, to answer your question, you at least need 2500*300*20*8 bytes ~= 120 MB to store your datasets. Add to that the overhead of the structure fields (around 20*(numberofcharactersinfieldname + constant) and of the structure itself (not much).
Unless you've got very little memory on your computer or the data size is vastly different from what you've stated, it should all fit in memory.
  1 件のコメント
Ibro Tutic
Ibro Tutic 2017 年 5 月 15 日
Just found a way to shrink the data sets even further, thanks for the reply, it was helpful.

サインインしてコメントする。

その他の回答 (1 件)

Jan
Jan 2017 年 5 月 15 日
A cleaner version of the code:
fmt = repmat('%s', 1, 360); %reads file, text string delimter
datFields = fieldnames(dat);
folderFields = fieldnames(foldername); % "foldername" is a bad name for a struct
for i = 1:length(datFields)
a = datFields{i}; %selects upper structure
a2 = folderFields{i}; %select which folder
aFields = fieldnames(dat.(a));
for j = 1:length(aFields)
b = aFields{j}; %select file to open
filepath = fullfile(foldername.(a2), dat.(a).(b)); %sets file path
DLG.(a).(b) = cell(1, length(dat.(a).(b))); % Pre-allocate
for k = 1:length(dat.(a).(b))
fid = fopen(filepath{k}); %opens file
C = textscan(fid,fmt,'Delimiter','\t','CollectOutput',true); %records csv data to cell array C
fclose(fid); %closes file
DLG.(a).(b){k} = C{1}; %writes data to structure
end
end
end
I tried to move all repeated work before the loops, removed the useless strcat and there is still much potential to improve the readability. Fieldnames like "a" and "b" are not clear.

カテゴリ

Help Center および File ExchangeLarge Files and Big Data についてさらに検索

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by