Matlab and parallel computing

Hello,

I am facing some performance issues while reading data stored inside a wide list of txt files (around 1000000) and each file contains huge matrix data. I do not have PCT and I was thinking to use the java.util.concurrent package for multi-threading stuff (in this case I am looking to the BlockingQueue and ArrayBlockingQueue). Is there any expert that already used this package directly in a .m file? If yes, could you please share it? Do you see any limitation of MATLAB on using this package? Thank you in advance Best regards Paolo

[Jan]

You have 1 million text files, which contain numerical data? This would be a really bad method to store the data, because many import and export methods work much more efficient with binary data.

Note that parallel processing does not help, when the single resource of the hard disk's hardware is the bottleneck.

Answer 1

[John D'Errico]

As Jan says, you are probably wasting your time trying to use parallel processing to solve a problem that is not CPU bound, but disk bound.

I'd suggest a better tool is a RAM drive. Speed up the access time instead.

And having them as text files just means that your read time will be terribly slow. That may be something beyond your control.

Finally, I'm not sure this is a good task for MATLAB to solve. If you need the information in MATLAB eventually, then collect it all using some other tool, then read the entire mess to MATLAB in one piece.