How do I fill a histogram in very high dimension if the array creating overflows memory?

I was trying to collect statistics of a 6D vector and plot a histogram for each coordinate. I get 729000000 version of this vector (each 6 dimensional). For this I create an array of zeros of size 729000000x6 which seems to be a problem in matlab since it says:

Error using zeros
Requested 729000000x6 (32.6GB) array exceeds maximum array size preference. Creation of arrays
greater than this limit may take a long time and cause MATLAB to become unresponsive. See array
size limit or preference panel for more information.

usually what I did is just create create a W array with all the history of W as in:

W_history = zeros(iter,D);

and then feed a specific coordinate history to the histogram function in matlab:

histogram(W_history(:,d),nbins,'Normalization','probability')

however this seems problematic since it requires me to create a zeros array that is really large before hand. Does someone know ways to get around this that are efficient?

Answer 1

[Nanda Gupta]

I understand that you are trying to create a matrix of size 729000000x6 which may not be possible due to RAM limitations.

By looking at the "histogram" function, I assume that you are just using 729000000x1 matrix inside it. Try the following suggestions:

1. Instead of creating a zeros matrix of size 729000000x6 for the entire data, create separate vectors for each dimension, each of size 729000000x1 and work on them respectively.
2. Creating a Cell Array having 6 elements, each for different dimension having the size 729000000x1.
3. Creating a Structure array with the field-value pairs for each dimension.
4. Also, try saving the data into a .mat file locally and try accessing its variables directly without loading it into the memory. In this case, this might not be an efficient solution, but the "matfile" function can do this.