HELP running large code.. having storage issues

1 回表示 (過去 30 日間)
Julian Jesso
Julian Jesso 2019 年 2 月 20 日
コメント済み: Adam 2019 年 2 月 20 日
I am attempting to sort through csv data points in order to filter a certain data type out to put into a new csv. The large csv contains 55 columns and approximately 43 million rows. I am running into storage issues from Matlab and my computer, even though I have a 500GB adidtional storage drive that I am working through. It takes about 4 1/2 hours to run this code, but I always seem to have an error when I check back. It is not a java heap error, but just states "out of memory. Type HELP MEMORY" and I get a popup box that says low on storage and shuts matlab down. This is haulting my project and I am not sure where to go from here. Any recommendations are appreciated...
clc;clear;
myfile = fopen('file_2009.csv');
whichSite = input('Which site? ');
year = 2009;
recs{10e4,1} = [];
ctr = 1;
while ~feof(myfile)
linestring = fgetl(myfile);
supercell=strsplit(linestring, ',','CollapseDelimiters',false);
if str2double(supercell{3}(2:5)) == whichSite
recs{ctr,1} = supercell;
ctr = ctr+1;
end
end
fclose(myfile);
mycsvfile = sprintf('Site%d_%d.csv',whichSite, year);
recstable = cell2table(recs);
writetable(recstable,mycsvfile);
recs = [];
recstable = [];
  1 件のコメント
Adam
Adam 2019 年 2 月 20 日
Surely you can just read and write your data in sensible-sized chunks instead of trying to store 10^4 cells in memory at once? It seems to just be a sequential operation so I would assume you can just append results to those previously saved.

サインインしてコメントする。

回答 (0 件)

カテゴリ

Help Center および File ExchangeStartup and Shutdown についてさらに検索

製品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by