Long time to load 40 gig file

Hi, I have a gaming laptop which I recently bought for 4000€. Massive CPU and lots of ram. However when I want to load a 30 gig matlab file from my hard drive which consists of variables with huge cell array tables, I still need a lot of time. How can I make this process faster? Is data store useful?

[Star Strider]

The easy solution is to use a SSD to store it, rather than a mechanical HDD.

[AA]

It is on an ssd

[Jan]

Please mention the details: How long is "a lot of time"? What is the acceptable speed for you? Which MAT format is used?

[Steven Lord]

There's some additional information that would be useful in trying to determine what's consuming most of the time and whether there's a better approach you can use.

• Is this a text file, a binary file, a MAT-file, etc?
• Do you need all the variables in the file at once or could you do what you need accessing each variable in turn, keeping only one at a time in memory?
• Can you post a small sample (not all 30-40 GB, but 5-10 lines) of what the data looks like in the file, and whether the file has a consistent format throughout or if the format changes periodically
• Copy and paste the command that you're using to try to read in the file, or describe the interactive process you're using (if you're using the Import Tool, for example.)
• Can you clarify what "a lot of time" means: a couple minutes, half an hour, an hour, etc.?
• Can you clarify what "lots of ram" means: how many GB?

Answer 1

[Edric Ellis]

You don't mention how you're loading the data right now. If you have Parallel Computing Toolbox, you can read the data in parallel using parfor together with datastore. Or, better still, use tall arrays, which can automatically take advantage of Parallel Computing Toolbox.

Answer 2

[Jeremy Hughes]

"cell array tables"

A good bet is that the cell arrays are the issue. Each cell of the array takes a 114 bytes of overhead. Without knowing anything about your data, except that you described them as tables, I suggest looking at the MATLAB datatype, table.

Data can be efficiently stored as tables for many common usages. I wish I could give a better answer, but I'd need to see what's in the file.

Cheers,

Jeremy