Contain large&hetergeneous datasets : Cell vs Table vs Others

load dryer2;
z = iddata(y2,u2,0.08,'Tstart',0);
na = 2:4;
nc = 1:2;
nk = 0:2;
models = cell(18,5);
ct = 1;
for i = 1:3
    na_ = na(i);
    nb_ = na_;
    for j = 1:2
        nc_ = nc(j);
        for k = 1:3
            nk_ = nk(k); 
            [models{ct, :}] = deal(armax(z,[na_ nb_ nc_ nk_]), na_, nb_, nc_, nk_);
            ct = ct+1;
        end
    end
end

Above codes are from documentation of ARMAX. https://kr.mathworks.com/help/ident/ref/armax.html

I made some modifications to variable 'models'

In my problem, the number of parameters is much larger(e.g. na, nb, ..., nz), and their range is much broader.

I want to save all parameters(na, nb,..) and their output(like output of armax functin in above codes), which are heterogeneous.

Which data types are the most fit to contain these datasets? Cell? Table? other else?

[Hyeong Seok Lee]

The number of datasets is more than 100 million.

Answer 1

[Gouri Chennuru]

Hi,

In order to work with large data sets, MATLAB has number of tools for accessing and processing large data in many ways,

• You can use datastore function that creates a datastore, which is a repository for collections of data that are too large to fit in memory.

• You can use Large MAT files access and change variables without loading into memory.

• You can create tall numeric arrays, cell arrays, categoricals, strings, datetimes, durations, or calendar durations, and you can use any of these tall types as variables in a tall table or tall timetable.

You can refer to these links for more information,

https://www.mathworks.com/help/matlab/tall-arrays.html

https://www.mathworks.com/help/matlab/large-files-and-big-data.html

Hope this Helps!