Outlier Detection of a matrix depending on spatio coordinates
6 ビュー (過去 30 日間)
古いコメントを表示
Hi, I have thousands of matrices, each of (130x160) with values (range of -25 up to +25) consisting of outliers.The values of the matrix depending on spatial data x(1:130) and Y(1:160). Outliers are often within the range and sometimes not only single values, clustering across 2 or 3 values across the raws and/or columns (X or Y) of the matrix.
I already tried "filloutliers" and deleteoutliers (https://de.mathworks.com/matlabcentral/fileexchange/3961-deleteoutliers)
with rather average level of satisfaction.
Can anyone recommend an appropriate method to detect the outliers? What kind of ML algorithms could I try?
Thanks,
Chris
14 件のコメント
Alan
2023 年 9 月 4 日
I had intended to post my answer in the "Answers" section instead of the "Coments" section. Therefore I shall repost it in the "Answers" section. I hope that the OP and others looking for a solution to a similar problem will find the resources to be helpful.
採用された回答
Alan
2023 年 9 月 4 日
編集済み: Alan
2023 年 9 月 4 日
(This is a repost from the comments section)
If you want to detect outliers within each matrix rather than considering each matrix as datapoints.
In that case, you can try to fit a surface to each matrix and evaluate the z-score of the matrix (considering the fitted surface as the mean). You could exclude the points whose absolute z-score crosses a certain threshold. Here is an example in 1-D: https://www.mathworks.com/help/curvefit/removing-outliers.html?s_tid=srchtitle_support_results_2_outliers#RemoveOutliersExample-2
Here is how you can fit a surface using the fit function instead of fitting a curve as shown in the above example:
x = 1:130;
y = 1:160;
[X Y] = meshgrid(x, y);
% Generating a matrix for this example
Z = 25 * sin(2 * pi / 130 / 160 * (X.^2 + Y.^2));
% Fitting surface
xdata = reshape(X, 1, [])';
ydata = reshape(Y, 1, [])';
zdata = reshape(Z, 1, [])';
fitted_surface = fit([xdata ydata], zdata, "poly23");
% Visualize fitted surface against the matrix
plot(fitted_surface, [xdata ydata], zdata)
legend(["Fitted Surface"]);
% Continue to calculate z-score and exclude outliers
In case you are considering each matrix as a datapoint, consider combining the matrices and find the outliers along the z-axis. The following functions could be utilized:
- isoutlier() - https://www.mathworks.com/help/matlab/ref/isoutlier.html
- rmoutliers() - https://www.mathworks.com/help/matlab/ref/rmoutliers.html
- filoutliers() - https://www.mathworks.com/help/matlab/ref/filloutliers.html
Some parameters to experiment with could be:
- dim : Dimension along which the outliers should be detected.
- movemethod : To detect outliers using a moving window.
- percentiles : To identify the extreme percentile of data.
As for machine learning methods, you could check out this resource: https://www.mathworks.com/help/stats/unsupervised-anomaly-detection.html . It contains various supervised methods for multivariate data.
0 件のコメント
その他の回答 (0 件)
参考
カテゴリ
Help Center および File Exchange で Statistics and Machine Learning Toolbox についてさらに検索
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!