Why will Matlab read only 2000 of my 5000 files?

Hi. I have 5000 gridded ascii files that are output from a Fortran program. One is attached as an example ("Run2000.txt"). They are all in the same ESRI-ASCII format, and I have written the attached function "readESRI_ASCII_DepositFile.m" to separate the header values from the gridded data. That function is called repeatedly by the attached script, "read_arrivaltimes.m" to read each of the 5000 files and create a .mat file with the combined output. Some of the files are bad, so there is a "try . . . catch" loop. If a file fails to load, the script writes out "couldn't open xxxx".

The script runs fine for a certain number of (n) files. After that, it says "couldn't open xxxxx" for all the rest. Sometimes n=1982, other times n=1750, or some other number. Most of the remaining files are in the proper format and are readable if I start the loop at, for example, file #2000.

I thought perhaps I was exceeding some memory limit, but my Window Performance Monitor shows only 73% of my computer memory is being used.

My last attempt ended with the error message "Unrecognized function or variable: 'internal.matlab.datatoolsservices.WSBUtil.getWorkspaceDisplay'. Attached is a screenshot. Oddly, my workspace values do no display.

Do you have any idea what the problem could be? Thanks.

[dpb]

Attach the m file scripat as text and format so can read.

A guess of the cause is you're not releasing the file handles of the read-in files and are running out of system resources.

Answer 1

[Stephen23]

Yeah, dpb is likely on the right track.

"Why will Matlab read only 2000 of my 5000 files?"

What is likely happening is that your code opens a file in readESRI_ASCII_DepositFile, starts parsing it and then somewhere within readESRI_ASCII_DepositFile an error occurs (which is very likely, given the fragile way that code is written), and this error is then caught (and hidden) by the TRY within read_arrivaltimes, which means that you will accumulate more and more open files with each error (because you do not fclose those ones) until your OS cannot handle all of those open files any more (this depends on the OS, not MATLAB), after which all new fopen requests will fail. There is nothing in your code design to prevent this.

Using TRY to hide all errors like that is https://en.wikipedia.org/wiki/Anti-pattern and has a strong https://en.wikipedia.org/wiki/Code_smell . You are finding out why.

Best solution: get rid of those try-catches around everything.

Or if you want to continue writing shotgun code, use fclose('all')

[Larry Mastin]

Thanks, Stephen23. After skimming over all your pejoratives, I realized I hadn't added an "fclose" to the catch block. Adding that solved the problem. Including a try-catch block is necessary in this code because some input files are missing.

[Stephen23]

The code structure itself causes a resource leak due to the improper use of try/catch for file handling. Here's a breakdown of what's wrong and how to fix it without using try at all.

What’s Wrong with the Original Code:

• readESRI_ASCII_DepositFile wraps fopen in a try, but:
• fopen doesn't throw errors—it returns -1
• So the catch block will not catch file-open failures.
• read_arrivaltimes.m uses a try/catch to call readESRI_ASCII_DepositFile, but:
• If readESRI_ASCII_DepositFile throws an error after opening the file and before fclose(fid), then the file stays open.
• This leads to open file handle accumulation, eventually exhausting system resources.

✅ Better Design Without try/catch around the file handling

Here’s how to fix it cleanly

🔁 Revised readESRI_ASCII_DepositFile.m

function [headervals, output_grid, errstat] = readESRI_ASCII_DepositFile(infile)
% Read an ESRI ASCII Grid file safely, returning header and data.
% Returns errstat = 0 on success, 1 on failure.
errstat = 1;
headervals = [];
output_grid = [];
%
fid = fopen(infile, 'r');
if fid<0
    return;
end
cleanup = onCleanup(@() fclose(fid));  
try % this only catches bad file content (not the _existence_ of any file)
    headervals = NaN(6,1);
    headervals(1) = sscanf(fgetl(fid), 'NCOLS %f');
    headervals(2) = sscanf(fgetl(fid), 'NROWS %f');
    headervals(3) = sscanf(fgetl(fid), 'XLLCORNER %f');
    headervals(4) = sscanf(fgetl(fid), 'YLLCORNER %f');
    headervals(5) = sscanf(fgetl(fid), 'CELLSIZE %f');
    headervals(6) = sscanf(fgetl(fid), 'NODATA_VALUE %f');
    NCOLS = headervals(1);
    NROWS = headervals(2);
    NODATA_VALUE = headervals(6);
    output_grid = fscanf(fid, '%f', [NCOLS, NROWS]).';
    output_grid(output_grid <= NODATA_VALUE) = NaN;
    errstat = 0;
end

• onCleanup guarantees that fclose is always called, even on parse errors.
• This allows graceful continuation without leaking file handles.
• It handles this within the function where the error occurs, i.e. leakage cannot occur!

🔁 Revised read_arrivaltimes.m

nfilesread = 0;

for irun = 1:nruns
    infile = sprintf('ArrivalTimes\\Run%05d.txt', irun);
    [headervals, output_grid, errstat] = readESRI_ASCII_DepositFile(infile);
    if errstat==0
        fprintf('reading %s\n', infile);
        ArrivalTimes(irun,:,:) = output_grid;
        FilesRead(irun) = 1;
        nfilesread = nfilesread + 1;
    else
        fprintf('***couldn''t read %s\n', infile);
    end
end

• No try/catch is needed.
• Each file is handled deterministically.
• readESRI_ASCII_DepositFile encapsulates the responsibility of managing file access properly, just as it should!

✅ Final Advice

• Avoid try/catch for control flow—it's meant for truly exceptional errors, not for catching expected states which have flags specifically designed for handling them.
• Always check fid and use onCleanup to ensure fclose is called.
• Make your file-reading functions return clear status indicators (errstat), rather than relying on throwing errors which get caught somewhere else... that is just asking for leaks.

"Including a try-catch block is necessary in this code because some input files are missing."

Nope, there is nothing about missing files which makes TRY-blocks "necessary". I just demonstrated this above.

[dpb]

"Including a try-catch block is necessary in this code because some input files are missing...."

A cleaner approach that would avoid that issue would be to instead of using a hard count as the loop control that means you have to change the code if the number changes, use

% Script that reads arrival times and makes a .mat file of them
%% Variables
%nruns = 5000;
nrows = 320;
ncols = 600;
xLL   = -130.;
yLL   = 40.;
dxy   = 0.05;

%% calculate other variables
xUR = xLL+ncols*dxy;
yUR = yLL+nrows*dxy;

infile=fullfile('ArrivalTimes','Run*.txt');     % set file path lookup string
d=dir(infile);                                  % find the files that match
nruns=numel(d);                                 % how many there are...
ArrivalTimes = NaN(nruns,nrows,ncols);
FilesRead=zeros(nruns,1);
%% Start opening and reading files
nfilesread=0;
for i=1:nruns
  infile=fullfile(d(i).folder,d(i).name);       % build qualified filename
  [headervals,output_grid,errstat] = readESRI_ASCII_DepositFile(infile);
  if errstat, continue, end                     % skip this one, go on
  fprintf('reading %s\n',infile);
  ArrivalTimes(i,:,:) = output_grid;
  nfilesread=nfilesread+1;
end

%% All finished
fprintf('All finished.  nfilesread=%d\n',nfilesread);

Even better would be to wrap the script into a function to which you can pass a working directory and the wildcard string and then remove all the internal variables from the workspace, returning only the needed data.

The one place a try...catch...end construct could be of use would be inside function readESRI_ASCII_DepositFile() after successfully opened the file but if something goes wrong in reading the data from the file -- there is no error-checking/handling there at present.

[Larry Mastin]

Thanks, dbp and Stephen23. This is really helpful.

[dpb]

In similar veitn,

function [headervals,output_grid,errstat]=readESRI_ASCII_DepositFile(infile)
%readESRI_ASCII_DepositFile:  Function that reads an ESRI_ASCII
%deposit file and returns the contents of that file
%   infile=name of file to be read
%   headervals(6) = values of parameters in header:
%      headervals(1) = NCOLS
%      headervals(2) = NROWS
%      headervals(3) = XLLCORNER
%      headervals(4) = YLLCORNER
%      headervals(5) = CELLSIZE
%      headervals(6) = NODATA_VALUE
%  output_grid   = gridded data
%  errstat       = error status: 0=success, -1=failure

fid=fopen(infile);
if fid==-1                  % documented fopen() failure return
  errstat=fid;
  fprintf("failed opening input file %s\n",infile);
  return
end

%If I can open it
headervals = NaN(6,1);
input_line = fgetl(fid);
headervals(1) = str2double(input_line(6:end));
input_line = fgetl(fid);
headervals(2) = str2double(input_line(6:end));
input_line = fgetl(fid);
headervals(3) = str2double(input_line(10:end));
input_line = fgetl(fid);
headervals(4) = str2double(input_line(10:end));
input_line = fgetl(fid);
headervals(5) = str2double(input_line(9:25));
input_line = fgetl(fid);
headervals(6) = str2double(input_line(13:end));

NCOLS = headervals(1);
NROWS = headervals(2);
NODATA_VALUE = headervals(6);

output_grid = NaN(NROWS,NCOLS);
for i=1:NROWS
    output_grid(i,:) = fscanf(fid,'%f',[1 NCOLS]);
end
fclose(fid);

%Convert all elements that meet the NODATA_VALUE to NaNs
output_grid(output_grid<=NODATA_VALUE)=NaN;
return