185k1*185k Matrix

Hi everyone, I'd like to build a 185k*185k matrix in matlab but my memory doesn't allow me to do it...What can I do to solve my problem?

Thanks in advance

[Daniel M]

This is ~273 GB in double precision or ~34 GB with type uint8. You can do any of the following:

1. buy more memory for your computer
2. ask yourself if you really need to store this much information
3. use sparse matrices
4. break it into chunks or loops.

[Luis Sancho]

I will try sparse. The thing is that i have an entire code composed by 'for's" to build this matrix. I start my script by building a matrix of zeros 185k per 185k but it doesnt run. Thanks for your answer

Answer 1

[John D'Errico]

Think of it like this:

If all of the elements in that matrix are non-zeros, then you have

185000^2
ans =
   3.4225e+10

So 34 billion elements in that array. Since each will be a double precision number

185000^2*8

ans =

2.738e+11

That means 273 gigabytes, just to store the array. Do you have that much contiguous, addressable memory? Unlikely, unless you work for a well heeled employer. And anything you do with it, you will probably end up needing to make temporary copies of the array, etc. So expect to need at least 3x that much, so lets just call it a terabyte of main memory for kicks.

That does not mean nothing can be done of course. Computer memory is relatively cheap these days. And you can always get a large SSD drive. Solving your problem using brute force is rarely that great of an idea though. And by next year, I expect that your problem will have doubled in size.

That means you cannot do anything with such a large array, unless it is perhaps a sparse array. In fact, much of the time, when you are working with such huge matrices, they are often sparse ones. So almost all zeros. MATLAB has sparse arrays just for that purpose, and computations with them are vastly more efficient. So I would strongly consider using sparse storage, IF your array is mostly zero. I do mean seriously mostly zeros. If your array is only 50% non-zero, don't bother. Sparse storage won't gain you anything then.

There are other things you can do of course. It may well be possible to reorganize your computations such that this matrix never needed to be computed in the first place. That is a very good idea, IF possible.

There are other possibiities, for example tall arrays. (Your array is tall in one sense, but it is pretty squat too.)

[Luis Sancho]

Thanks for your answer...in fact the matrix is composed mostly by non-zero elements.