Why doesn't the read(Tabul​arTextData​Store variable) function return the correct number of rows set by ReadSize property?

When using read(TabularTextDataStore) the function returns the correct number of lines determined by the ReadSize property, however occasionally it will return a set of data with less rows and break my script. Why does it not follow the ReadSize property every time I call the read function?

Answer 1

[Walter Roberson]

https://www.mathworks.com/help/matlab/ref/matlab.io.datastore.tabulartextdatastore.html

"If ReadSize is a positive integer, then each call to read reads at most ReadSize rows."

There has never been any guarantee that it will read ReadSize rows if they are available. It is permitted to return fewer rows for any internal reason, including potentially switching between files in the datastore, and including potentially reaching a limit of some internal buffer.

[John Nolan]

Is there a way to clear the buffer periodically so that I can ensure max ReadSize rows are read?

[Walter Roberson]

"Is there a way to clear the buffer periodically so that I can ensure max ReadSize rows are read?"

No.

You can implement your own buffering routines, keeping an internal buffer and looping doing read() operations until you have accumulated the size you want, extracting that and leaving the rest in your internal buffer. Watch out for the case of end of data store where there might not be enough data to meet the target.