Pull specific data from multiple text files and export it to an excel file.

I'm trying to figure out how to pull multiple text files into matlab. Scan each file and pull specific data that I want and then export that data to an excel file.

Specific data I want from each file: Note the files are very large and they have multiple OSIZ components that I want not just 1 row.

1) element # that matches the heading component "OSIZ": "element = some number"

2) flow data: "scfm = 380.0 or whatever the flow is"

3) duct size: "D_duct = some number"

4) orifice size: "D_orif = some number"

What I have so far:

see attached matlab file & example text file.

Thank you fo any help.

[Image Analyst]

Does what you have now work, or not?

[dpb]

How large is "very"???

If will fit in memory, I'd suck up the whole thing with readlines and then select the wanted data in memory. Otherwise, the historic fgetl route still works and is only a little more trouble (and well may be as fast or faster, depending).

Is there any way to know a priori how many sections are in the file or a way to know when there can be no more sections so can abort early? That could influence using fgetl as could short circuit when found/done; readlines sucks up the whole file, regardless.

[Joshua Keller]

The files are typically (12,000 + lines) long or 700-800 kb. The text files are an output from another model and depending on the changes in the model it will change the amount of 'sections in the text output file.' The idea was to have hundreds of the output files and capture the output flow and diameter of each configuration or run and then plot that in excel. I can try readlines and see thanks. I tried doing this in excel where I pulled the file in and then did a vlookup of the "OSIZ" heading to find the section I wanted but it's just to many files to pull into excel.

[dpb]

That's not that big in memory footprint; big in terms of hand-processing, yes, but not a memory issues...

Answer 1

[dpb]

That's not that big in memory footprint; big in terms of hand-processing, yes, but not a memory issues...

data=readlines(websave('testrun.txt','https://www.mathworks.com/matlabcentral/answers/uploaded_files/1158723/testrun.txt'));
ELEMENT= 101122;        % set this somehow -- could do a search for all records and present to user???
% engine
matchstr='OSIZ Component data for element =';
isWanted=find(contains(data,matchstr));             % indices to all the OSIZ sections
for i=1:numel(isWanted)
  element=str2double(extractBetween(data(isWanted(i)),matchstr,':'));
  if element==ELEMENT
    flow=str2double(strtrim(extractBetween(data(isWanted(i)+1),'scfm =',':')));
    D_duct=str2double(strtrim(extractBetween(data(isWanted(i)+2),'D_duct =','inches')));
    D_orif=str2double(strtrim(extractBetween(data(isWanted(i)+3),'D_orif =','inches')));
  end
end

What we found is

disp([element])

      101122

disp([flow D_duct D_orif])

  380.0000    5.5000    2.5713

I didn't show array appending or insertion here; you'll need to increment a counter and insert the values into a preallocated array for each value -- and then wrap the whole thing in a loop iterating over the files in the collection.

Above is the "deadahead" solution using the high-level MATLAB search/matching functions; if this is too slow, can then work on making it more elegant/faster.

[Joshua Keller]

Hey DB,

Thanks for the insight. So I used this format and it works. But I wanted to use it to pull from multiple txt files. Example of the above 'ELEMENT', is the same in hundreds of text files but their (flows/diameters) are different between the hundreds of files so I want to pull all of their values into matlab so I can add them to an excel file but this format only returns the last txt file in the selected folder. It looks like it went through all the files but only returned the last one. I'm not sure how to distinguish the files if they have the same element number but different properties (flow, dia, etc).

[dpb]

"...; you'll need to increment a counter and insert the values into a preallocated array for each value" is what I said before that I had not shown specifically.

How you structure the rest of the code to call the above and your choice of how you want to hold the values for later use determines the precise use; creating a table for the overall end object might be useful or you can go the other end and just return separate arrays for each...