csv file reading limits question

Hello, I'm trying to import the data from multiple csv files. My csv files have different numbers of heading lines. for example:

File one:

DATE = 20020414

TIME = 0723

Data 1, Data 2,

1,2

3,4...

File two:

DATE = 20020414

Data 1, Data 2,

1,2

3,4

4,5...

I have a code that works for a single csv file, using csvread/dlmread where I manually assign the row and column limits, but I would like to write a general code that can search for the number of headers in a specific file and adapt limits for file reading accordingly. Is there an easy way to do this? Should I be using a different function to read my data? (textscan, etc?)

[Jan]

How do you distinct header lines from data lines? Do header lines never start with a number character, but data lines always do?

[Namaka]

yes, that is the main difference.

Answer 1

[Walter Roberson]

Provided that you have the same number of columns in each of the files:

fid = fopen(CurrentFileName,'rt');
failed = 0;
if fid < 0
  disp(sprintf('Failed to open file: %s', CurrentFileName))
  failed = 2;
else
  while true
    pos_at_line = fseek(fid);
    this_line = fgetl(fid);
    if ~ischar(this_line)
      disp(sprintf('File has no dataset: %s', CurrentFileName))
      failed = 1;
      break
    elseif any(this_line == '=')
      continue
    else
      fseek(fid,pos_at_line,'bof');
      break
    end
end

if failed == 0
  inputdata = textscan(fid, ...... );  %use an appropriate format
end

if failed <= 1
  fclose(fid);
end

Answer 2

[Namaka]

A solution! Since the last header in all my files contains the word "FLAG" I found this line by using the code:

for i = 1:(numfiles)

str=['!grep -n ''FLAG'' ',filefolder,'/',d(i).name,' > hold'];

eval(str);

str=d(i).name;

fid=fopen('hold');

y=fscanf(fid,'%g',[1 1]); %finds the line number at FLAG using g

fclose(fid);

c=csvread(d(i).name,y+1,0,[y+1 0 x-2 8]);

% x is the location of the end of the file I found using a similar method

end