how to extract data from the fixed-width-field format using fscanf or textscan
14 ビュー (過去 30 日間)
古いコメントを表示
Dmitrii Semikin
2019 年 6 月 16 日
コメント済み: Dmitrii Semikin
2019 年 6 月 18 日
I am trying to parase large file, containing among others numerical data fixed in fixed-width-fields format without whitespaces (or any other separators) between the fields. Consider the example:
1223244
55 6 788
1122 3 4
In this text each line contains four numbers each - two digits wide. Expected result after parsing this flie is:
1 22 33 44
55 6 7 88
11 22 3 4
The prerformance is important for me, as the files I read may be hunreds of MBs large. From my experiments I discovered, that fgetl() is drammatically slower, than fscanf (I did not check the performance, but I expect, that textscan should also be significantly faster.
The problem is that I cannot find the way to parse data like this with fscanf or textscan. Could someone tell, if it is possible at all? If not, is there any other way to parse such a text file with good performance?
P.S. format of actual string is, of course, somewhat more complicated. Example string is:
1101332.18685714711829.064533733772535.874264373485 0 0
this string should be parsed into the numbers with the following widths: 8, 16, 16, 16, 8, 8, which in this particular case would result in the following numbers:
110
1332.18685714711
829.064533733772
535.874264373485
0
0
P.S.2: Differnence between the performance of fgetl and fscanf: On my laptop for the file, which contains 35597 lines the following code with fscanf complete in 0.253614 seconds
function [data] = test_fscanf_nodes_only_01()
file_name = 'myfile.txt';
file_id = fopen (file_name, 'rt');
cleanup_obj = onCleanup(@() fclose(file_id));
data = fscanf(file_id, '%8d%16f%16f%16f%8f%8f', [6, Inf]);
end
while the following code with fgetl needs 6.343209 seconds to complete, even though it does much less
function [data] = test_fscanf_nodes_only_02()
file_name = 'myfile.txt';
file_id = fopen (file_name, 'rt');
cleanup_obj = onCleanup(@() fclose(file_id));
lines_count = 0;
while ~feof(file_id)
current_line = fgetl(file_id);
lines_count = lines_count + 1;
end
data = 1;
fprintf('Lines count: %d', lines_count);
end
The main problem with the first snippet is that it returns wrong result (because for fscanf the width of the field counts from the first digit it finds and not from the current position of the file pointer (which means, that the leading whitespaces are not counted as field width).
EDIT: The dimension in fscanf is changed from [5, Inf] to [6, Inf] according to Jan's comment.
8 件のコメント
Walter Roberson
2019 年 6 月 17 日
textscan() is not the same as fscanf(): with textscan it is possible to change the Whitespace and Delimiter properties in a way that does not skip leading whitespace. because of the way that the numeric fields parse numbers, a leading whitespace in a numeric field in such an arrangement would be counted as an invalid character, so this turns out to only be useful for fixed-width character fields.
採用された回答
Walter Roberson
2019 年 6 月 17 日
fileread into a character vector. regexprep to insert a space before every field. textscan the character vector.
4 件のコメント
Walter Roberson
2019 年 6 月 17 日
newS = regexprep(S, '^(.{8})(.{16})(.{16})(.{16})(.{8})(.{8})', '$1 $2 $3 $4 $5 $6', 'lineanchors');
その他の回答 (1 件)
Stephen23
2019 年 6 月 17 日
A simple solution based on regeprep and sscanf. Before the file-reading loop:
vec = [8,16,16,16,8,8];
rgx = sprintf('(.{%d})',vec);
rpl = sprintf('$%d,',1:numel(vec));
In the file-reading loop:
str = fileread('temp2.txt');
str = regexprep(str,rgx,rpl,'dotexceptnewline');
mat = sscanf(str,'%f,',[numel(vec),Inf]).'
Giving (the test file is attached):
mat =
110.00000 1332.18686 829.06453 535.87426 0.00000 0.00000
220.00000 2332.28686 829.06453 535.87426 0.00000 0.00000
330.00000 3332.38686 829.06453 535.87426 0.00000 0.00000
440.00000 4332.48686 829.06453 535.87426 0.00000 0.00000
550.00000 5332.58686 829.06453 535.87426 0.00000 0.00000
参考
カテゴリ
Help Center および File Exchange で String Parsing についてさらに検索
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!