Scattered interpolation unexpected peaks in extrapolation results

Hello,

I am trying to create a surface through scattered interpolation of experimental data. The results during extrapolation give some very high unexpected peaks.

The data that I am interpolating is shown in blue dots. The surface that I am trying to create should be "covering" the blue dots and extend as shown with the organge lines I drew on top.

Instead, when extrapolating towards zero (below the last known data curve in blue), huge peaks appear.

If I extend even further towards zero, the peaks are orders of magnitute larger than the rest of the data.

What could be the reason for that? Any ideas for possible solutions?

Thanks in advance,

John

[Bruno Luong]

Please post the interpolation options/code and data you are using.

[Giannis Zavos]

scatteredInterpolant(x,y,z,'linear','linear')

[Bruno Luong]

Can you post your data (especially the blue one), it looks a slope should go down hen approachin 0, so it's odd that you get the extrapolation that actually goes up.

[Giannis Zavos]

You can find my data attached. First column is y, second is x, third is z.

What I have done is use two for loops to interpolate through all values of x and y to create a full surface of z.

Answer 1

[Bruno Luong]

You might appropriately rescale your x/y data before interpolation

A=load('PV_curve.txt');
x=A(:,1);
y=A(:,2);
z=A(:,3);

xscale = 1/10; % <= experiment with it
yscale = 1;
xn = x*(xscale/max(x(:)));
yn = y*(yscale/max(y(:)));

xi = linspace(0,xscale);
yi = linspace(0,yscale);
[X,Y] = meshgrid(xi,yi);
I = scatteredInterpolant(xn,yn,z,'linear','linear');
Z = I(X,Y);
xi = xi*max(x(:))/xscale;
yi = yi*max(y(:))/yscale;

close all
figure
surf(xi,yi,Z);
hold on
plot3(x,y,z,'.r');

[Giannis Zavos]

Hi Bruno,

Yes, this solves the problem!

The negative values are unwanted due to the nature of the problem, but those I removed after extrapolation.

Thank you for your time!

John

final result :

the red * is the where the max y*x is found in each vertical plane (max power point of solar panel). :)

Answer 2

[Walter Roberson]

• scatteredInterpolant does a triangulation, and it is not uncommon for it to turn out that one of the three closest points to a given point can be from a different "layer" of Z. Furthermore, when you do your joining "along" the data, some of the points must be joined with a different Z layer, in order to be able to provide the surface triangles leading up the slope. Interpolation can get a bit odd.
• If you have edge data surrounded by a flat section, then the interpolation scheme does not just say, "Oh yes, you have a slope down and that joins onto a plain." Instead, it fits a polynomial through the points, and you are very likely to get overshoot when dealing with sharp angles.

             A  A
             +
      0  0  ?+

at point ? the interpolation does not understand that you have a "cliff": it has to fit a curve that goes through both A and the two zeros. That is likely going to involve a curve that rises steeply up through the right A and peaks and goes down through the left A and steep down and then back up through the right 0... so at the ? point, the value could be quite large (or quite large negative.)

Accounting for edges with an interpolating polynomial can lead to steep curves.

[Giannis Zavos]

Hi Walter,

Thank you for taking the time to explain this. I understand that the program does not see the "whole picture" and tries to fit with the nearest points. However, in this case I am using linear interpolation (and extrapolation) so I don't think it tries to fit to a polynomial. Even so, now that I think of it, it might be selecting the "wrong" points between the different known curves (200 and then 400) resulting to these peaks.

In any case, do you know any other extrapolation methods that might work better with this?

Answer 3

[John D'Errico]

Walter is correct, in that depending on the scattered data, if some points fall in just the right orientation (or perhaps, in the wrong orientation from your point of view), you can get strange looking artifacts.

Remember that a tiny differentiation is a noise amplifying operator. So a tiny amount of noise in the wrong place, and you get garbage.

Worse, around the edges of a region, strange things happen. These tools use what is called a delaunay triangulation. That is, they take your data in the (x,y) plane, and find a set of triangles that will completely cover the region. Inside the region, things are good. Delaunay triangulations are well-behaved in the interior. But things get confusing around the edges, because a Delaunay triangulation covers the entire convex hull of the data.

You have given us no access to the data you have, so it is impossible to show you exactly what did happen, with your data. And that means if will be difficult to prove to you what are the issues. I cannot possibly convince you of the facts without knowing the complete facts. But I can offer a simple example.

xy = rand(1000,2);
tri = delaunayn(xy);
triplot(tri,xy(:,1),xy(:,2))

Perfectly random data in the plane. If we assume some function is defined, thus z = f(x,y), and the function is smooth and relatively well defined, then if I pick any point in the interior of that domain, then there are always three points near it. Any point inside the convex hull will lie in exactly one triangle. (Or it will lie on an edge, or it may be coincident with one of the points.) But in any of those cases, interpolation is a well defined thing. Since the function is presumed to be smooth, with a relatively low noise level, then all is good. We get some nice smooth predicted surface.

However, what happens near the edges? Everything gets nasty there. Do you see what appear to be long, thin triagles? You see correctly. This is what must happen, since the delaunay triagulation fills the ENTIRE convex hull of the data. You get well behaved triangles inside, but along the edges, long thin stuff.

That means if you try to interpolate inside one of those triangles, the information that is used to do so uses information from points that are VERY far apart. And that is a bad thing! Because now you have strange stuff happening. Two points that are close together should have similar behavior. But if you try to join stuff together that does not belong together? GARBAGE RESULTS.

Now, think about also trying to use the very triangles where crap is happening, with extrapolation? OUCH. You get amplified crapola. And I will confidently predict that is what you see here. Can I prove it? Well, not without the real data. Bruno has requested you provide the data several times. Do I need the real data to be confident? Now it only becomes an educated guess as to the problem.

Is there a fix for a problem where I cannot conclusively KNOW what the issue is, and what caused the problem? SIGH. My head hurts. But there are options.

The use of interpolation methods to intelligently extrapolate is just dangerous a thing as you can do. They simply don't extrapolate well in general. And extrapolation is about the most difficult thing you can do, because it is controlled by tiny variations at the boundaries of your data. Tiny amounts of noise can result in chaotic predictions as you move away. If you have ever heard of the butterfly effect, you might appreciate this.

https://en.wikipedia.org/wiki/Butterfly_effect

Again, can you deal with it? One idea is to use a smoothing tool, that tries to reduce the random occurrences by assuming you have some smooth well-behaved underlying function behind the data. Such a tool does not need to use these problematic long thin triangles near the edge of your data.

I would suggest my gridfit tool. It is available for free download from the file exchange, here:

https://www.mathworks.com/matlabcentral/fileexchange/8998-surface-fitting-using-gridfit

It tries to fit a smooth surface to scattered data. There are no long thin triangles. And it does allow extrapolation, where the extrapolation is done smoothly. Of course, any extrapolation can be dangerous. Just ask those who try to forecast the weather. Sometimes they even come close. Other times, well, nobody is perfect.

Without the real data, I cannot go much more deeply, without trying to hack together an example to replicate what you saw. And that could take far more time than just looking at the real data.

==============

Edit: you have now provided your data.

You have 5 distinct values of x, with y varying for each level of x.

unique(x)
ans =
         200
         400
         600
         800
        1000

z goes all the way down to zero, which may in itself be a problem, since any extrapolation will surely produce negative values of z in those farily signifcant region of the extrapolation.

How will interpolation be done?

tri = delaunayn([x,y]);
triplot(tri,x,y)

Zooming into one corner of that figure, we see these triangles:

If a point lies inside the convex hull of the triangulation, we can interpolate linearly within the triangle that contains that point.

xy = [x,y];
[ind,barycentriccoef] = tsearchn(xy,tri,[500,25])
ind =
   513
barycentriccoef =
                       0.5         0.371503660596959         0.128496339403041

So the triangle #513, which has vertices #

tri(ind,:)
ans =
   359   491   490

That point lies within the triangle with (x,y) coordinates

xy(tri(ind,:),:)
ans =
                       600                   24.8613
                       400                   25.2756
                       400                   24.7429

What happens when we use this same scheme to interpolate a point at the location (300, 50)?

[ind,coord] = tsearchn([x,y],tri,[300 50])
ind =
   NaN
coord =
   NaN   NaN   NaN

It lies outside of the convex hull of your data. It lies inside no triangle in that triangulation. Now extrapolation is used, based on the one triangle at the edge of your region. Any noise, any probem in the data, and you can get garbage.

Now, if I look carefully at triangle 2 in the set I just produced, I see this:

 [xy(tri(2,:),:),z(tri(2,:))]
ans =
                       400                   49.9667                    0.0692
                       400                   50.0777                         0
                       200                   48.4277                    0.0474

And that appears interesting to me. Interesting, since one of the z values seems as if it was artificially clipped at zero, and that is the ony point where a zero appears in z, but it is also where you observed crapola happening.

I think what happened is your measurement apparatus clips there. Anyway, somehow garbage is introduced into the surface. And that introduces problems into that triangle, since it is inconsistent in shape with the rest of the surface. Now extrapolation produces crapola. Extrapolate far, and you get big crapola. You do not even need to extrapolate too far, since these are long thin triangles. You have very little data in the one variable compared to the other.

[Giannis Zavos]

Hi John D'Errico,

This was really insightfull!

I had heard about the butterfly effect but only now I read more about it.

About your observation on the artificial zero, you are right, I actually added zeros at the end of the curves (200, 400, 600) trying to fix "edge errors", but the extrapolation problem was the same before I did that.

This data comes from the digitization of some curves I found in image format, so it is defininetly not as smooth as if I had the original data. It represents the current-voltage curve of a solar panel for different radiation intensities (200, 400, ...).

I will try to use your tool later and see how it performs! For now, the problem is solved by Bruno Luong's suggestion of rescaling the x/y data, as seen below.

Many thanks!

John

[John D'Errico]

Ah. I had wondered about that, if you added zero points.

Sadly, gridfit also introduces interpolation artifacts on thus data, of a differnt kind, due to rhe sharp crease in the shape of your function. So you would not be happy with the result of that tool on your data. If nothing else, I hope I was helpful in your understanding how those interpolation tools work.

[Giannis Zavos]

You were definitely helpful :)