Is it possible to detect and replace abnormal/wrong numbers in an array?

I have a dataset which is an array containing values, and which are plotted in the figure below. A section of the array for the figure is: A = [.... 25.9, 25.9, 26.2, 27, 28, 29, 29.3, 29.6, 3, 30.4, 30.5, 30.4, 30.3, 30.3, ....]; Here is number 3 an abnormality in the array. Is it possible to detect these abnormal numbers and replace them with the average value of the number before and after? So for the number 3 in A I want it to be (29.6+30.4)/2.

Answer 1

[Kevin Phung]

A = [25.9, 25.9, 26.2, 27, 28, 29, 29.3, 29.6, 3, 30.4, 30.5, 30.4, 30.3, 30.3]
abnorm =mean(A) -  3*std(A); % you can set however many standard deviations to constitute 'abnormal'
n = find(A<abnorm); %locations of abnormals
for i = 1:numel(n)
    if or(n(i) == 1, n(i)==numel(A)) %if first or last index, replace with NaN;
        A(n(i)) = nan;
    else
            A(n(i)) = (A(n(i) - 1) + A(n(i) + 1)) / 2;
    end
end
A(isnan(A)) = []; %remove the endpoints that are NaN

let me know if this works for you

[Rikke]

Thanks for the repond! Unfortunatly it did not work for the whole array as the abnormal value for one section of the array may be normal for another section of the array.

[Rikke]

The figure i plotted above represent velocity of a vehicle and the abnormality is not so easy to detect by looking at the figure, however when I convert the velocity to acceleration the abnormal values are more easiliy spotted, shown as large peaks in the figure below.

[Rikke]

Actually it did work if i applied your code on the last figure i plotted, so I am able to correct the acceleration at least.

[Kevin Phung]

So after correcting the acceleration, would taking the integral give you appropriate velocity values?

[Rikke]

I got the solution of correcting the velocity values by using almost all the code you sent, just adjusted the first code lines. I got the 'isoutlier' from the link Walter sent written below this answer. Thanks for your help!

TF = isoutlier(A,'movmedian',5);
abnorm = 0;
n = find(TF>abnorm); %locations of abnormals
for i = 1:numel(n)
    if or(n(i) == 1, n(i)==numel(A)) %if first or last index, replace with NaN;
        A(n(i)) = nan;
    else
            A(n(i)) = (A(n(i) - 1) + A(n(i) + 1)) / 2;
    end
end
A(isnan(A)) = []; %remove the endpoints that are NaN

[Kevin Phung]

happy to help!

edit: if number of lines of code is important to you at all, you can actually condense the first 3 lines to one

n = find(isoutlier(A,'movmedian',5)); %locations of abnormals
for i = 1:numel(n)
    if or(n(i) == 1, n(i)==numel(A)) %if first or last index, replace with NaN;
        A(n(i)) = nan;
    else
            A(n(i)) = (A(n(i) - 1) + A(n(i) + 1)) / 2;
    end
end
A(isnan(A)) = []; %remove the endpoints that are NaN

[Rikke]

Aha, perfect!

Answer 2

[Walter Roberson]

https://www.mathworks.com/help/matlab/ref/isoutlier.html

[Rikke]

Thanks!

[Chris Turnes]

You may also be interested in the filloutliers function, which lets you not only identify outliers but replace them as well.