Identifying duplicates and deleting duplicates

v1 =[300 350 350 350 360 370 380 380 380 390 395 405 450 450 450 450 450 450 500 500 500 500 500 500]

v2 = [NaN 1 2 3 3 NaN 1 2 3 NaN 2 NaN 1 2 3 1 2 3 1 2 3 1 2 1 ]

I am trying to make a script which identifies data which is repeated in v1 and v2 (when they are combined) and then deletes the repeated data in both v1 and v2.

I've been looking at this problem for a long time, I really don't know where to start. I started of using a = find(diff(v1)==0) but that didn't get me anywhere. I'm very bad at explaining my problems, here is the desired output of the script(may help with understanding the problem):

v1 =[300 350 350 350 360 370 380 380 380 390 395 405 450 450 450 500 500 500]

v2 = [NaN 1 2 3 3 NaN 1 2 3 NaN 2 NaN 1 2 3 1 2 3 ]

[Matt]

I'm looking into concatenation, i think it may help

[Sean de Wolski]

The output isn't clear to me. So you want to find values that are repeated in both v1 &v2?

So, why are there fewer 500s in the output of v1? There's not repeated... Is it a sequence you want rid of? [1 2 3] is repeated three times in v2 at the end but is only one + the end is removed. Sorry, not clear at all....

[Matt]

I'm sorry, I'll try to make it clearer.

It's not a sequence I want rid of it is duplicates the reason the 500's were deleted was because there was 500-1, 500-2, 500-3, 500-1, 500-2 and 500-1 therefore two of the 500-1's are repeats and one of the 500-1's is a repeat, therefore they are removed.

Answer 1

[Fangjun Jiang]

Use unique( ,'rows')

v1 =[300 350 350 350 360 370 380 380 380 390 395 405 450 450 450 450 450 450 500 500 500 500 500 500]
v2 = [NaN 1 2 3 3 NaN 1 2 3 NaN 2 NaN 1 2 3 1 2 3 1 2 3 1 2 1 ]

v1_out =[300 350 350 350 360 370 380 380 380 390 395 405 450 450 450 500 500 500]
v2_out = [NaN 1 2 3 3 NaN 1 2 3 NaN 2 NaN 1 2 3 1 2 3 ]

out=unique([v1' v2'],'rows');
z1=transpose(out(:,1));
z2=transpose(out(:,2));

isequal(v1_out,z1)
isequalwithequalnans(v2_out,z2)

If v2 is cell array of strings, then NaN is not an issue.

v1=[1 1 1 11 12];
v2={'a','ab','a','b','a'};

str=[num2str(v1'),char(v2)];
[str,index]=unique(cellstr(str));

v1=v1(index)
v2=v2(index)

[Sean de Wolski]

You need to do something if there are equal rows with nans;

unique([7 nan;7 nan],'rows')

since nans aren't equal to themselves

[Fangjun Jiang]

Thank you, Sean! Something I learned today!

[Matt]

Works great thanks guys

[Fangjun Jiang]

@Matt, you should use Sean's solution, which considered the case for comparing NaNs.

[Matt]

Thanks

Answer 2

[Sean de Wolski]

vcat = [v1', v2'];
vcat(isnan(vcat)) = inf;  %assuming no infs!
vu = unique(vcat,'rows')
vu(isinf(vu))= nan;
v1 = vu(:,1).';
v2 = vu(:,2).';

[Fangjun Jiang]

Good to consider the case for nan

[Matt]

This works, only problem is I forgot to mention that v2 is a string variable! I put it like that to make the explanation easier, I can't get your solution to work when one of them is a string variable!

Any ideas, silly of me not to mention it at the start I know

Answer 3

[Friedrich]

Hi,

are you looking for the unique command?

>> a = [1 2 2 3 1 4]

a =

       1     2     2     3     1     4

>> unique(a)

ans =

       1     2     3     4

[Matt]

i'm looking into concatenation, i think it may help