MATLAB Answers

Translated by

このページのコンテンツは英語から自動翻訳されています。自動翻訳をオフにする場合は「<a class="turn_off_mt" href="#">ここ</a>」をクリックしてください。

0

HELP running large code.. having storage issues

Julian Jesso さんによって質問されました 2019 年 2 月 20 日
最新アクティビティ Adam
さんによって コメントされました 2019 年 2 月 20 日
I am attempting to sort through csv data points in order to filter a certain data type out to put into a new csv. The large csv contains 55 columns and approximately 43 million rows. I am running into storage issues from Matlab and my computer, even though I have a 500GB adidtional storage drive that I am working through. It takes about 4 1/2 hours to run this code, but I always seem to have an error when I check back. It is not a java heap error, but just states "out of memory. Type HELP MEMORY" and I get a popup box that says low on storage and shuts matlab down. This is haulting my project and I am not sure where to go from here. Any recommendations are appreciated...
clc;clear;
myfile = fopen('file_2009.csv');
whichSite = input('Which site? ');
year = 2009;
recs{10e4,1} = [];
ctr = 1;
while ~feof(myfile)
linestring = fgetl(myfile);
supercell=strsplit(linestring, ',','CollapseDelimiters',false);
if str2double(supercell{3}(2:5)) == whichSite
recs{ctr,1} = supercell;
ctr = ctr+1;
end
end
fclose(myfile);
mycsvfile = sprintf('Site%d_%d.csv',whichSite, year);
recstable = cell2table(recs);
writetable(recstable,mycsvfile);
recs = [];
recstable = [];

  1 件のコメント

Adam
2019 年 2 月 20 日
Surely you can just read and write your data in sensible-sized chunks instead of trying to store 10^4 cells in memory at once? It seems to just be a sequential operation so I would assume you can just append results to those previously saved.

サインイン to comment.

0 件の回答


Translated by