How do i extract specific row data whose value is changing from .dat file ?

Hi, I have a large .dat file (attached) and I am only interested in rows with specific values of constraint data of that redesing as shown below.

This is the data in .dat file as shown below:

$CONSTRAINT DATA NAME = D_Tstep1_OP_01

1.95063723e+00

$CHANGED TOPODRV NAME = M_OP_01

15932 2 1.77575504e-03 -3.88161658e-01

17231 2 1.80259062e-03 -3.91727572e-01

15291 2 -1.13913637e-02 2.21583925e+00

..

..

$CONSTRAINT DATA NAME = D_Tstep2_OP_01

3.95063723e+00

..

..

From the above .dat file, I only need data of D_Tstep1_OP_01 = 1.95063723e+00 with its values for 90 steps (like D_Tstep1_OP_01, D_Tstep2_OP_01,.......,D_Tstep90_OP_01). How do I extract the said data which has different delimiters from the .dat file and save it in excel? I need the data in the following format in excel:

NAME CONSTRAINT DATA

D_Tstep1_OP_01 1.95063723e+00

D_Tstep2_OP_01 3.95063723e+00

..

..

D_Tstep90_OP_01 5.95063723e+00 % this is the last value in the file

Your assistance is most appreciated. Thank you in advance.

Answer 1

[Mathieu NOE]

hello

I wrote this little code for you !

check it

% main code  

[ string_out, data_out ]  = retrieve_data('Redesign2-0_OP_01.dat');
C = [{'  NAME  '}  {'CONSTRAINT DATA'} ;string_out' data_out'];

writecell(C, 'C_out.xlsx');

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function [ string_out,data_out ] = retrieve_data(Filename)

fid = fopen(Filename);
tline = fgetl(fid);

% initialization
k = 0;
p = 0;
data_line = [];


while ischar(tline)
    
    k = k+1;    % loop over line index
    tmp1 = contains(tline,'$CONSTRAINT DATA     NAME = D_TStep'); 
    if tmp1  &&   ~isempty(tline) 
        p = p+1;
        ind_eq = strfind(tline,'=');
        string_out{p} = (tline(ind_eq+2:end));    % 
        data_line = k+1;
        
    end
    
    if ~isempty(data_line) && k  == data_line
        data_out{p} = str2num(tline);    % 
    end
    
    tline = fgetl(fid);
    
end

fclose(fid);

end

[Shubham Gorde]

Thank you so much for your reply. I need to check it in the uni tommorrow. Will let you know:)

[Mathieu NOE]

ok

FYI, this is the excel output file - hope you get the same result ! :)

[Shubham Gorde]

Yeah, I checked it today, it is what I expected. Actually, there is one column missing of timestep which I have to add manually: it is (3.7E-05*n) where n is from 1 to 90 steps and there are lot of such files where the first D_TStep column remains same but the second column of constraint data changes according to design, is it possible to extract multiple files and put it into one sheet as first column will be same for all but second column will be constraint data for design1, then for design2....?

[Mathieu NOE]

hello again

I am not sure to have understood the issue with :

Actually, there is one column missing of timestep which I have to add manually: it is (3.7E-05*n) where n is from 1 to 90 steps

do you ean there are other data from the dat file that we need to retrieve as well ? could not find anything that looks like 3.7E-05 in the file (???)

otherwise, the code is now able to do the work on all dat files in a folder ; to test it I created a dummy second file with values a bit changed.

the data columns title may be change to get either the filename or an iteration index...

the code uses this FEX submission to get the file names sorted in natural order , something matlab does not do by default.

https://fr.mathworks.com/matlabcentral/fileexchange/47434-natural-order-filename-sort

for convenience the two files are also in attachement

clc
clearvars

%%% main code %%%

fileDir = pwd;
outfile = 'C_out.xlsx'; % output file name 

fileNames = dir(fullfile(fileDir,'*.dat')); % get list of data files in directory
fileNames_sorted = natsortfiles({fileNames.name}); % sort file names into order (https://fr.mathworks.com/matlabcentral/fileexchange/47434-natural-order-filename-sort)
M= length (fileNames_sorted);

out_data = [];
str = []; 
for f = 1:M 
    %   retrieve data per file 
    [ string_out, data_out ]  = retrieve_data(fullfile(fileDir, fileNames_sorted{f}));
    % contatenation of all individual files data
    out_data = [out_data data_out];
    str = [ str {'CONSTRAINT DATA'}]; 
end

% store out_data in excel file 
C = [{'  NAME  '}  str ;string_out out_data];

% writematrix(out_data,fullfile(fileDir,outfile)); 
writecell(C, outfile);


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function [ string_out,data_out ] = retrieve_data(Filename)

fid = fopen(Filename);
tline = fgetl(fid);

% initialization
k = 0;
p = 0;
data_line = [];


while ischar(tline)
    
    k = k+1;    % loop over line index
    tmp1 = contains(tline,'$CONSTRAINT DATA     NAME = D_TStep'); 
    if tmp1  &&   ~isempty(tline) 
        p = p+1;
        ind_eq = strfind(tline,'=');
        string_out{p,1} = (tline(ind_eq+2:end));    % 
        data_line = k+1;
        
    end
    
    if ~isempty(data_line) && k  == data_line
        data_out{p,1} = str2num(tline);    % 
    end
    
    tline = fgetl(fid);
    
end

fclose(fid);

end

[Mathieu NOE]

hello

here you are

the time vector is now in second column

updated code :

clc
clearvars

%%% main code %%%

dt = 3.7E-05; % sampling time

fileDir = pwd;
outfile = 'C_out.xlsx'; % output file name 

fileNames = dir(fullfile(fileDir,'*.dat')); % get list of data files in directory
fileNames_sorted = natsortfiles({fileNames.name}); % sort file names into order (https://fr.mathworks.com/matlabcentral/fileexchange/47434-natural-order-filename-sort)
M= length (fileNames_sorted);

out_data = [];
str = []; 
for f = 1:M 
    %   retrieve data per file 
    [ string_out, data_out ]  = retrieve_data(fullfile(fileDir, fileNames_sorted{f}));
    % contatenation of all individual files data
    out_data = [out_data data_out];
    str = [ str {'CONSTRAINT DATA'}]; 
end
% time vector
t = (0:length(data_out)-1)'*dt;
out_data = [t out_data]; % add t vector in first column before data
% store out_data in excel file 
C = [{'  NAME  '}  {'  TIME  '} str ;string_out num2cell(out_data)];

writecell(C, fullfile(fileDir,outfile));


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function [ string_out,data_out ] = retrieve_data(Filename)

fid = fopen(Filename);
tline = fgetl(fid);

% initialization
k = 0;
p = 0;
data_line = [];


while ischar(tline)
    
    k = k+1;    % loop over line index
    tmp1 = contains(tline,'$CONSTRAINT DATA     NAME = D_TStep'); 
    if tmp1  &&   ~isempty(tline) 
        p = p+1;
        ind_eq = strfind(tline,'=');
        string_out{p,1} = (tline(ind_eq+2:end));    % 
        data_line = k+1;
        
    end
    
    if ~isempty(data_line) && k  == data_line
        data_out(p,1) = str2num(tline);    % 
    end
    
    tline = fgetl(fid);
    
end

fclose(fid);

end

[Shubham Gorde]

Thank you so much, I will implement it:)

[Mathieu NOE]

My pleasure !