array exceeds maximum array size preference.

I am getting the error called : Error using string/cat. And in the second line, Its written that,

Requested 3x3x3x3x3x3x3x3x3x3x3x3x3x3x3x3x3x3x18 (52.0GB) array exceeds maximum array size preference.

Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive.

See array size limit or preference panel for more information.

Error in allcomb (line 42)

A = reshape(cat(NC+1,A{:}), [], NC) ;

This error is a result of a separate function called allcomb, which I have generated for getting the result faster.

I am attaching the code below with the function, allcomb as well.

I am struggling to get this error fixed somehow but, unfortunately I dont know the advanced steps or the solution. Any help in this code would be really greatful and appreciated.

CostLowerRange =  1 
CostHighRange =  10 
UtilityLowerRange =  1 
UtilityHighRange =  10 
%Define architecture decisions 
arch_deci = ["a";"b";"c";"d";"e";"f";"g";"h";"i";"j";"k";"l";"m";"n";"o";"p";"q";"r";]
%Define architecture decision options 
arch_deci_options = [
"HPP" "PA" "EPP" ;
"HLS" "ELS" "PC" ;
"HFS" "FA" "EFS" ;
"MINT" "ELMINT" "PR" ;
"NMINT" "ELNMINT" "Ma" ;
"AB" "BC" "HK" ;
"PY" "QY" "GH" ;
"RMINT" "NRM" "ZX" ;
"TRAC" "IMPL" "WE" ;
"ISO1" "FF" "UR" ;
"COMM1" "LEVER1" "REMOTE1" ;
"COMM2" "LEVER2" "REMOTE2" ;
"COMM3" "LEVER3" "REMOTE3" ;
"ISO2" "ANA" "JD" ;
"ISO3" "EHEC21" "SeP1" ;
"KA" "AI" "" ;
"SU" "UB" "" ;
"SA" "AY" "" ;
]
%Define architecture decision options combinations
archs = allcomb(arch_deci_options(1, :), arch_deci_options(2, :), arch_deci_options(3, :), arch_deci_options(4, :), arch_deci_options(5, :), arch_deci_options(6, :), arch_deci_options(7, :), arch_deci_options(8, :), arch_deci_options(9, :), arch_deci_options(10, :), arch_deci_options(11, :), arch_deci_options(12, :), arch_deci_options(13, :), arch_deci_options(14, :), arch_deci_options(15, :), arch_deci_options(16, :), arch_deci_options(17, :), arch_deci_options(18, :));
%Evaluating the total combinations without the empty cells
archs = archs(~any(cellfun('isempty', archs), 2), :);

And the function: allcomb:

function A = allcomb(varargin)
narginchk(1,Inf);
NC = nargin ;
% check if we should flip the order
if ischar(varargin{end}) && (strcmpi(varargin{end}, 'matlab') || strcmpi(varargin{end}, 'john'))
    % based on a suggestion by JD on the FEX
    NC = NC-1 ;
    ii = 1:NC ; % now first argument will change fastest
    %varargin(end)=[];
else
    % default: enter arguments backwards, so last one (AN) is changing fastest
    ii = NC:-1:1 ;
end
args = varargin(1:NC) ;
if any(cellfun('isempty', args)) % check for empty inputs
    warning('ALLCOMB:EmptyInput','One of more empty inputs result in an empty output.') ;
    A = zeros(0, NC) ;
elseif NC == 0 % no inputs
    A = zeros(0,0) ;
elseif NC == 1 % a single input, nothing to combine
    A = args{1}(:) ;
else
    isCellInput = cellfun(@iscell, args) ;
    if any(isCellInput)
        if ~all(isCellInput)
            error('ALLCOMB:InvalidCellInput', 'For cell input, all arguments should be cell arrays.') ;
        end
        % for cell input, we use to indices to get all combinations
        ix = cellfun(@(c) 1:numel(c), args, 'un', 0) ;
        % flip using ii if last column is changing fastest
        [ix{ii}] = ndgrid(ix{ii}) ;
        A = cell(numel(ix{1}), NC) ; % pre-allocate the output
        for k = 1:NC
            % combine
            A(:,k) = reshape(args{k}(ix{k}), [], 1) ;
        end
    else
        % non-cell input, assuming all numerical values or strings
        % flip using ii if last column is changing fastest
        [A{ii}] = ndgrid(args{ii}) ;
        % concatenate
        A = reshape(cat(NC+1,A{:}), [], NC) ;
    end
end

As I am at intermidiate level, could anyone provide me the solution to get this error solved?

Thanking you.

Answer 1

[Steven Lord]

You probably want to find an alternative to brute force to solve your problem.

How many combinations are you trying to create?

>> 3^16*2^3
ans =
   344373768

That's slightly larger than the current population of the United States. If you tried to process one combination a second, how long would it take?

>> years(seconds(3^16*2^3))
ans =

          10.9127702827573

What's the underlying problem you're trying to solve? We may be able to suggest an alternate approach that doesn't require creating this huge an array or taking multiple years to solve.

[kaivlya patel]

I want to plot the pareto frontier of all the possible combinations on the matlab graph for the tradespace analysis of the system with requirements and their respective variants.

My problem is to find a way so that matlab can access the result for 17 requirements and 3 variants for each.

so that it will be 3^17.

I want to create a cloud of all the combinations and then highlight the closest combinations to the Optimum point.

I have studied about breaking this large arrays into sections so that Matlab can process it. But as I told, I am new and this information is lil deep for me to understand, I would like to get the help in breaking the arrays in sections (dividing or distributing arrays).

[kaivlya patel]

this is the cloud of all the combinations with highlighted ones, closest to the optimum point. I am getting the result of 51 millions of combinations but want to exceed it to 100 millions.

[Steven Lord]

On average how long does it take to evalate one of your combinations? Fill in the blank below and see how long it'll take to perform your calculation serially. [My original estimates were off by a factor of 3; for some reason I thought you had 16 parameters that took on one of three possible values and 3 parameters that took on one of two, but instead you have 15 and 3.]

timeForOne = seconds(   );
timeForAll = years(3^15*2^3*timeForOne)

[kaivlya patel]

For generating the array of 3^15*2^3, it is taking 140 minutes to generate the result of 114 millions of combinations.

I need to figure out a way that can we distribute this much large set of arrays into sections so that matlab can process it easily?

or any other alternative with respect to the above code and problem?

Thanking you.