importdata - HELP MEMORY

I am trying to use importdata to import a .dat file which is approximately 349000KB in size. If I import about 75% of the data, MATLAB keeps his cool... but when I try to import the whole things he gets angry and tells me:

N=importdata('Respiratory_Modulation.dat',' ',1);

??? Error using ==> importdata at 214 Unable to load file. Use TEXTSCAN or FREAD for more complex formats.

Caused by: Error using ==> fileread at 36 Out of memory. Type HELP MEMORY for your options.

Why is he angry with me? What have I done?

Thanks,

Linford

[Eric]

What happens when you try to create a single variable of this size? For instance, try

A = ones(6700,6700);

(don't forget the semicolon!). If this fails then there really does seem to be a memory issue rather than a problem reading in the DAT file.

Answer 1

[Andreas Goser]

Like mentioned in an other answer. A 64 Bit MATLAB on a 64 Bit machine should be tested.

There are however many smart methods to reduce memory. And very often the method depends on your data and application.

In this case, I assume, your data is of type DOUBLE and as you are more than halfway through importing, I like suggesting to convert the data into SINGLE. Most applications do not need DOUBLE data. But as I said, it depends what you do after the data is imported and how often you need to import new data sets.

Assuming, you like to try it:

• Cut the data into half
• import it
• Convert it using SINGLE()
• save it
• Do it with the other half
• Import both

Answer 2

[Walter Roberson]

Not very many systems can store 349 gigabytes of data in RAM. Are you certain you need to load all 349 gigabytes at one time??

Answer 3

[Linford Briant]

I'm not being persnickerty, but you may have misread: don't you mean 349 megabytes? I guess they are quite long simulations, but I wouldn't have thought MATLAB would be limited to this.. or, as you said, perhaps its a limitation of my laptop. You are right though, I can possibly split the file up, and do the analysis in chunks. A bit of an inconvenience though!

Any other suggestions?

[Walter Roberson]

Sorry, yes, I was off by a suffix.

[Walter Roberson]

Just before the importdata() call, for debugging please put in a call

memstats

to see how much memory your system has and how much is available.

(Note: this is only valid for MS Windows.)

Are you using a 32 bit version of MATLAB or a 64 bit version? Would switching to a 64 bit version be practical on your system?