Speeding up writing a very large text file
62 ビュー (過去 30 日間)
古いコメントを表示
I have to write very large text files (1-4 million lines; 100-400 MB) to disk. Writing binary output is NOT an option. To solve this problem I broke the output into blocks of 10000 lines and then used sprintf to write the formatted lines (each ending with a '\n') to a block of 10000 strings (dataop=string(10000,1)).
I have already opened the output file with the 'W' option and then successively write each block of strings using the command: fprintf(fid,dataop(:));
It is still taking an inordinate amount of time - each 1 million lines takes 60 minutes!
The machine is high-end: running Windows 10, 128GB RAM, dual Xeon 10 core processors, and WD Black 6TB drive. There are 5 other hard drives with are not active during the write process. I'm running 2017b
So - why is it so slow? Am I doing something stupid? (I'm more used to C++ than MATLAB)
2 件のコメント
Stephen23
2018 年 9 月 17 日
@davidwriter: are the data numeric, strings, or char vectors? Do you really need to call sprintf as an intermediate step, why not just call fprintf directly?
採用された回答
Stephen23
2018 年 9 月 17 日
編集済み: Stephen23
2018 年 9 月 17 日
"To solve this problem I broke the output into blocks ..."
What problem? MATLAB has no problem with fprintf used on millions of elements. I don't see any reason, based on your explanation so far, why you need to process this in "blocks".
The problem is likely to be how you have coded this, but note that you forgot to actually upload your code, thus making it impossible for us to know what you are actually doing.
I just wrote a short script to test how long MATLAB requires to print 1 million character vectors (each char vector is arbitrarily sized 1x128) to a file:
C = cellstr(char(randi([33,126],1e6,128)));
tic
fid = fopen('temp4A.txt','wt');
fprintf(fid,'%s\n',C{:});
fclose(fid);
toc
It created a 127 megabyte file in
Elapsed time is 3.06831 seconds.
Then I tried a script with 16 floating point numbers per row, for 1 million rows:
N = 16;
M = rand(1e6,N);
F = repmat(',%.6f',1,N);
F = [F(2:end),'\n'];
tic
fid = fopen('temp4B.txt','wt');
fprintf(fid,F,M.');
fclose(fid);
toc
It created a 141 megabyte file in
Elapsed time is 28.429 seconds.
I can't get anywhere close to "each 1 million lines takes 60 minutes!"
3 件のコメント
Jan
2018 年 9 月 19 日
@davidwriter: In "fprintf(fid,%7.1f \t %8.2f \t,..." the "..." might be the most interesting part. If you post some running code, it would be much easier to suggest improvements. It is not clear to me, what the inputs are. But converting numerical values to a string and again to a cell string is an indirection. See sprintfc or compose to create the required output format directly.
Creating a C-Mex function might be an option. But without knowing exactly what the inputs are, writing some code would include too much guessing.
その他の回答 (1 件)
KSSV
2018 年 9 月 17 日
How about this approach?
S = rand(10000,3) ;
S = cellstr(num2str(S)) ;
fid = fopen('data.txt','w') ;
fprintf(fid,'%s\n',S{:});
fclose(fid) ;
0 件のコメント
参考
カテゴリ
Help Center および File Exchange で Startup and Shutdown についてさらに検索
製品
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!