Finding consecutive data with non zero in array

Hi all;

I want to find consecutive data with non zero value in array. However, as long as the data with 0 value is less than 3 points, it is still considered as part of the previous data set. The data with the same length is classified into same array.

For example:

xx = [1 2 3 0 1 2 0 0 0 1 2 3 4 0 1 0 0 0 1 1 1];

The expected result

xx1 =[1 2 3 0 1 2; 1 2 3 4 0 1];

xx2=[1 1 1];

I try to find it using looping procedure but it takes only onsecutive data with non zero value and I can not find the solution for this criteria "as long as the data with 0 value is less than 3 points".

Hope some help.

Thank you.

Edward

[Rik]

Numbered variables are a bad idea. Also, it should be feasible to adapt your code to also work for the shorter runs of zeros. Can you share your code here? Attach it in an m file if it is more than about 20 lines long.

Otherwise, you could try the RunLength FEX submission by Jan to find the run lengths and continue from there.

[Rik]

Comment posted as answer by edward kabanyas:

Thank you Rik. The following is my code:

threshold = 0;
transitions = diff([0, xx > threshold, 0]);
runstarts = find(transitions == 1);
runends = find(transitions == -1) - 1;
blocks = arrayfun(@(s, e) xx(s:e), runstarts, runends, 'UniformOutput', false);
celldisp(blocks)

However, it takes only onsecutive data with non zero value and I can not find the solution for the criteria "as long as the data with 0 value is less than 3 points". Then, the cell with same size in blocks is not merged into one cell as I need above.

Answer 1

[Andrei Bobrov]

for R2013a (and for your data from all_data.txt)

f = fopen('all_data.txt');
Data = textscan(f,'%f %f %f %f %f %f %f','CollectOutput',1);
fclose(f);
Data = Data{:};
lo = any(Data(:,6:7) ~= 0,2);
d = find(lo);
ii = [true;diff(d) > 3];
i2 = ii([2:end,1]);
i = zeros(size(Data,1),1);
i(d(ii)) = 1;
i(d(i2) + 1) = -1;
j = cumsum(i);
jj = cumsum([false;diff(j) == 1]).*j;
out = accumarray(jj+1,(1:size(Data,1))',[],@(x){Data(sort(x),:)});
out = out(2:end);  

[edward kabanyas]

Thank you Andrei. I tried your code, but I got error because repelem is not found in Matlab. I tried another option such as kron, but I get error also. Thank you

[Rik]

What release are you using? repelem was introduced in R2015a. If you are using a release that is not the latest, it is good practice to mention this.

[edward kabanyas]

I have old one, 2013a

[Andrei Bobrov]

Please see my answer, after words: "for R2013a".

[Rik]

Note that this answer linearizes the input, which may or may not be what you want for array inputs.

[edward kabanyas]

Hi Andrei and Rik;

I run the code:

data=load('all_data.txt');

xx=data(:,6);

lo = xx(:)' ~= 0;

i = strfind(lo,[0 0 0 1]);

p = zeros(size(xx));

p([1,i + 3]) = 1;

p(i) = -1;

loo = cumsum(p);

j = cumsum(diff([0; loo]) == 1).*loo;

out = accumarray(j(:) + 1,xx(:),[],@(x){x'});

out = out(2:end)

I send the data as attched file. The result of the current code is strange, for example the first cell of out = 1x193 double and the cell contains zero values. As my first example:

xx = [1 2 3 0 1 2 0 0 0 1 2 3 4 0 1 0 0 0 1 1 1];

The expected result

xx1 =[1 2 3 0 1 2]

xx2= [1 2 3 4 0 1];

xx2=[1 1 1];

Thus, the first and the last value of result must be non-zero, thus the following is not expected result:

xx1=[ 2 3 0 1 2 0 0]

Finally, I wanto also to take all columns of the result. Thank you for help.

[edward kabanyas]

Thank you Andrei for modified code, but it seems the readtable is not available in R2013a. I asked my frind who is using R2014a to test the code, some errors is found:

Warning: Variable names were modified to make them valid MATLAB identifiers.

> In makeValidName at 25

In @table\private\setVarNames at 48

In table.readTextFile at 278

In table.readFromFile at 33

In readtable at 118

In test_1 at 3

Error using table/subsasgnDot (line 56)

The VariableNames property must contain one name for each variable in the table.

Error in table/subsasgn (line 66)

t = subsasgnDot(t,s,b);

Error in test_1 (line 5)

T.Properties.VariableNames = {'year','month','day','hour','munute','data1','data2'};

[edward kabanyas]

It seems the following script is ok for R2013a:

T=load('all_data.txt');

m = size(T,1);

lo=any(T(:,6)~= 0,2);

d = find(lo);

ii = [true;diff(d) > 3];

i2 = ii([2:end,1]);

i = zeros(size(T,1),1);

i(d(ii)) = 1;

i(d(i2) + 1) = -1;

j = cumsum(i);

jj = cumsum([false;diff(j) == 1]).*j;

out = accumarray(jj+1,(1:m)',[],@(x){T(sort(x),:)});

out = out(2:end);

[Andrei Bobrov]

I'm fix.

[edward kabanyas]

Thank you very much Andrei. It is perfect, thank you again.

Answer 2

[Rik]

I used the mfile version of Jan's FEX submission RunLength, because I can't get my compiler to work right now. With the runlegth determined it is relatively easy to find the groups that need to be merged. Then you can still use the code you proposed to find the transitions.

xx = [1 2 3 0 1 2 0 0 0 1 2 3 4 0 1 0 0 0 1 1 1];
[a,b]=RunLength_M(xx~=0);

L= a(:)==false & b<3;%to be merged
%checks for egde cases should be perfomed here:
%-first group is a small group of zeros
%-last group is a small group of zeros
L_prev=[L(2:end);false];
L_next=[false;L(1:(end-1))];
b(L_prev)=b(L_prev)+b(L)+b(L_next);
a(L | L_next)=[];%remove merged parts
b(L | L_next)=[];%remove merged parts
x=RunLength_M(a,b);

transitions = diff([0, RunLength_M(a,b), 0]);
runstarts = find(transitions == 1);
runends = find(transitions == -1) - 1;
blocks = arrayfun(@(s, e) xx(s:e), runstarts, runends, 'UniformOutput', false);
celldisp(blocks)

[edward kabanyas]

Thank you Rik. I tried your code for larger data set, and I got the following error:

"Error using &

Matrix dimensions must agree". The erro comes from:

L= a(:)==false & b<3;

Thank you

[Rik]

What size is your input?

[edward kabanyas]

I have some files and the file sizes are different, but I take one as an example, the sizie is 9886

[Rik]

What I meant to ask is if you were using a matrix or a vector a input. Attaching the data as a mat file would probably work best.

[Rik]

It turns out you also need to use (:) for b if your input is a column vector. I just tested this on R2011a, so it should also works on ancient releases. I'll edit my answer.

Edit: except that your example is integer only and your data isn't, which seems to be causing problems.