generate more random numbers

Hi everyone, when i try to generate more 100 million random sample, I memory issue with matlab.

Is there a way to generate more than 100 million samples?

[JL]

Let say I want to generate 1,000,000,000,000 samples? I tried using A=rand(100000000,5) and its still okay. But is there to generate more than 100 million samples? Say, 1 Billion?

[dpb]

Whether they're random or not has no bearing on the size of array you can hold in memory -- or at least only very marginally based on size of code loaded. You'll have same problem if use zeros or nan or repelem.

If you don't need the full precision of double you could cast to single but you may run into issue that has to generate as double first...not sure if the optional input argument 'single' causes singles to be used internally or just casts the output. For RNG, one would presume it's throughout so you might almost double the size possible. But 10X or more likely ain't agonna happen.

[James Tursa]

What are you doing with these random numbers? Do you really need all of them in memory at the same time?

[Adam Danz]

You could process whatever algorithm you're running in chunks or you could use tall arrays. A 3rd idea would be to generate chunks of random vars, store them in a text file (or multiple text files if needed) and read them in as needed.

[dpb]

"generate chunks of random vars, store them in a text file (or multiple text files if needed) and read them in as needed."

If doing this, don't use a text file, use stream (aka "binary") files.

[JL]

Is there a way to do imporant sampling? Yes, in memory so I could all the state that appears count how many time they appear

[Adam Danz]

Nice, I wasn't familiar with that method. Thanks, dpb!

https://www.mathworks.com/discovery/stream-processing.html

[dpb]

For importance sampling you would have to have the translation adjustment for the normalization from the importance-sampled distribution.

Unless it takes multiple samples to actually compute the results, there's nothing preventing doing the accumulation dynamically or from processing output disk file. Would, of course, require binning to a number of bins that is computable but even there the bins could theoretically be maintained in tall array that isn't all in memory at once.