how to store large data during the execution of a program without getting out of memory

I am working on partitions. while i run program that generates partition it get out of memory. What i want to do is to store these partitions in a data store and use tall arrays to work on chunks to filter required partitions. partitions are stored in cell arrays. Thanks

[Walter Roberson]

Do you have a generating function, that given an integer will return a particular partition ?

[asim nadeem]

Like for n=20 and m=4 my function generates all the partitions of length m=4 but partitions are in billions so want to store them in data store and use them later for other function. Thanks

[Walter Roberson]

If I were to ask you for partition #107 would you be able to calculate it without calculating all the ones before it in the same call?

[asim nadeem]

No the function I have creates a cell array of size stirling no of 2nd kind and stores partitions in it. I can restrict the size to be less than the above size say like 10 partitions but then remaining will not be calculated. Thanks

Answer 1

[Walter Roberson]

You will need to go through calculating the all partitions and writing them to a file, but keeping only a limited working set in memory as you do so. After you have written them all to file, you can create a datastore from the data file and create a tall array from that and start working on it.

There are some methods of building partitions that work recursively, creating partitions of size 1, extending to size 2, then to size 3, and so on, with (for example) all the partitions of size 19 needed to calculate the partitions of size 20. You would want to create as large a partition in memory as your memory could afford before writing that to a file and proceeding through tall array methods to addition additional columns, creating a file as you go, going back and making that file a datastore, and so on. It would probably be a bit of a nuisance, but It should be possible... I think.

Matters would be easier to arrange if you had an algorithm that could translate from an index to a particular partition; perhaps Knuth volume 4A; https://math.stackexchange.com/questions/222780/enumeration-of-set-partitions

[asim nadeem]

Thanks for your time.