Speeding up the manipulation of large matrices

Hello there,

I am trying to manipulate (specifically delete rows with a certain condition and sort rows) a matrix with over 30 million rows. But the system almost craches and disk usage reaches %100 almost all the time during execution. Now I know there must be something I can do to make it run faster but since I'm fairly new to matlab, I wonder if anyone knows how I can somehow optimize my code to run faster.

I use a one-line code simmilar to one "the cyclist" suggested in one of the questions here, which sets rows with a certain condition (identical values in a row) to zero.

temp=sort(temp,2);
temp(any(diff(temp,[],2)==0,2),:)=0;

now this part of the code comsumes the most amount of time (usually more than 300~400 seconds) while my laptop is completely unsable for the time.

Is there a way I can run this code faster, maybe without having my pc totaly frozenor is there another way of writing it?

Thank you.

[per isakson]

How long are the rows?

[Hamed Moghaddam]

Currently they're at 10 or 11, but it could be more depending on my data.

[per isakson]

I fail to reproduce your long execution time

>> cssm( 30e6, 24, 2 )
Elapsed time is 6.122413 seconds.
>> cssm( 30e6, 24, 2 )
Elapsed time is 6.161369 seconds.
>> 

where

function    cssm( h, w, dim )
    temp = randi(4,h,w);
    tic
    temp = sort(temp,dim);
    if dim == 1
        temp(:,any(diff(temp,[],dim)==0,dim))=0;
    else
        temp(any(diff(temp,[],dim)==0,dim),:)=0;
    end
    toc
end

What do I miss?

[Hamed Moghaddam]

thank you for your response, and no you didn't miss anything. That is almost what I have is mind. It seems I may have a problem with my memory or hard disk since i failed to execute your code. I recieved this message:

Error using sort

Out of memory. Type "help memory" for your options.

and by the way it took like 30 seconds or so for the code to stop with this error while the system was unusable. I have 7.5 Gb of RAM available. Isn't that enough?!

[per isakson]

Make sure there isn't a spurious function named sort that shadows Matlab's sort

which sort -all

will produce a huge list

>> which sort -all
built-in (C:\Program Files\MATLAB\R2018b\toolbox\matlab\datafun\sort)
built-in (C:\Program Files\MATLAB\R2018b\toolbox\matlab\datafun\@cell\sort)     % Shadowed cell method
built-in (C:\Program Files\MATLAB\R2018b\toolbox\matlab\datafun\@char\sort)     % Shadowed char method
...

Yes, memory might be a problem on your system. (I have 32GB ram.)

>> 30e6*24*8/1e9
ans =
    5.7600    

Swapping might explain your problem.

[Hamed Moghaddam]

Is there a way I can alter my code to make it less RAM-intensive? maybe spilt the matrix into several pieces and execute the same function?

Answer 1

[per isakson]

Your code, "diff(temp,[],2)==0,2)", made me think that temp holds whole numbers. If so, you could convert temp to an appropriate interger type. See cast, Cast variable to different data type and intmax, Largest value of specified integer type

>> z1 = randi( 5, 1e6,3 );
>> z2 = cast( z1, 'uint8' );
>> whos z*
  Name            Size               Bytes  Class     Attributes
  z1        1000000x3             24000000  double              
  z2        1000000x3              3000000  uint8      

See In-place Operations on Data

If there is a free memory slot in your computer you might want to add ram.

"spilt the matrix into several pieces" piecewise sorting with Matlab - no that's not an option for you - IMO.

[Hamed Moghaddam]

Casting worked just fine. I changed the matrix to int8 and the amount of consumed RAM almoste halved.

Thnak you so much for your help!