implement matrix with 2 million row in MATLAB!

hi

I have a matrix(that grows in my code) . I konw that in last section, the matrix will have 2 million Rows and 9000 columns. But , MATLAB cannot calculate that , and after some hours say : "out of memory . Type Help Memory for Your option"

i used sparse matrix or int32,int16 and int8 .. but i saw the error again! what can i do ? it's very important to me.

the memory(RAM)is 6 GB.

Answer 1

[John D'Errico]

You want to create a huge matrix. It is not sparse, else you would have said that most of the elements are zeros. Sparse won't help you otherwise, and for there to be any gain at all, you want only a small percentage of the values to be non-zero.

How big is the matrix? This is a computation you should do yourself!

2e6*9000 = 1.8e10

So roughly 18 billion elements. If they are doubles, then each element uses 8 bytes. Therefore that matrix will require roughly 144 gigabytes of memory. How can you be surprised if you ran out of memory?

Even for a uint8 array, it will still require 18 gigabytes of RAM. You have 6 GB, of which MATLAB will use only a fraction of that for array storage.

You don't say which version of MATLAB you are using. Is this a 64 bit version?

People think their computers are infinitely large and infinitely fast. Computers have a finite amount of RAM. Sorry, but they do. If you want to work with huge arrays, then get more memory. Use the 64 bit version of MATLAB for your computer, so it can address all that memory. Note that even if you have 64 bit MATLAB, trying to work with that huge of an array will be incredibly slow UNLESS you have sufficient RAM. Big problems require big computers.

[Oleg Komarov]

I recommend a local MapReduce. You can store compressed data on disk, read-in chunks and perform your operations and reduce the data, then consolidate across blocks.

In my work I handle more than 1 TB of data this way.

Caveat: as a rule of thumb, you can implement this approach if you can transform your problem in a linearly additive one. For example: the mean of a huge vector can be rewritten as the total sum of elements divided by total count. Since the sum and the count are linearly additive you can split those two operations in blocks.

[Isay]

john Thanks.

i use 64 bit version of Matlab ! i know it's very big,but i have no choice! do you have any suggest?

[Isay]

@Oleg, can you show me an example about what do you do?

[Oleg Komarov]

I think it will be more helpful if you read the Matlab docs about organizing a datastore() and using mapreduce.

[Isay]

@oleg , i can use map-reduce when i have matrix , but now, i don't have the matrix!

I said my matrix grows in my program and in last section the matrix will have 2 million Rows and 9000 columns.

do you have any suggest?

[Oleg Komarov]

MapReduce trades-off I/O operations on disk with memory. This means that the datastore will point to a collection of files on you hard drive. The MapReduce pulls small blocks that fit in memory, applies the mapping and a first consolidation/reduction, and then finally reduces the mid results to the final statistic. So, I recommend to read the docs more carefully. There is no way out with big data, it is tedious and not simple to grasp at first. I cannot simply write an example because it is not simpler than the doc already show. So, I am not gonna reinvent the wheel.

Answer 2

[matt dash]

Take a deep breath, and think about whether you REALLY need that matrix. At 144Gb, basically NO ONE'S computer could handle this data and still have enough memory to do something useful with it. (Figure 256Gb is about the top of the top end that you're likely to come across in nice computers today) And yet, we've gotten pretty far as a society. So somehow we're all getting by with smaller matrices. Instead of taking for granted that you NEED this matrix, try explaining the problem you're trying to solve and maybe people can help suggest ways you can accomplish it without needing such a large matrix.

(Also you don't want a big matrix to be growing in your code. If you know how big it needs to be at the end, just preallocate it to that size so you get the out of memory error right away. No point in waiting several hours just to run out of memory.)