I don't have your data, while they say a picture is worth a thousand words, it is still just words, not numbers. Anyway, I think you are asking to find the cancave region where the points live, but not including those triangular chaped voids.
For this purpose, an alpha shape is often the best approach. The idea is to visualize a circle or ball, of radius r. Now imagine that ball rolling around the perimeter of your points, in a way where the ball cannot cross any data point. Allow the alpha ball to erode a triangulation to the extent that it can. So any triangles around the perimeter with long edges will be deleted. Again, since I lack any data, I'll make some up.
xy = rand(5000,2);
k = (sqrt(sum((xy - [1 .5]).^2,2)) < 0.3) | ...
(sqrt(sum((xy - [0 .5]).^2,2)) < 0.3) | ...
(sqrt(sum((xy - [0.5 1]).^2,2)) < 0.3) | ...
(sqrt(sum((xy - [0.5 0]).^2,2)) < 0.3);
xy(k,:) = [];
plot(xy(:,1),xy(:,2),'.')
grid on
Admittedly, not a very good approximation to what you had. But it is early in the morning, and I don't care about the difference.
S = alphaShape(xy(:,1),xy(:,2),.20)
S =
alphaShape with properties:
Points: [2163×2 double]
Alpha: 0.2
HoleThreshold: 0
RegionThreshold: 0
plot(S)
I choise 0.2 for the alpha radius arbitrarily, assuming I did not know the diameter of the circle I had actually used. Anyway, in real life, you will never have exactly circular cutouts as I did here. Regardless, it is not a poor approximation, but only that. See that if I chose the alpha radius to be too large, it will start to leave large triangles aound the perimeter. Too small an alpha radius, and we will get too much erosion of the real domain we are interested in. Luckily with alpha shapes, there tends to be a fuzzy region where you can be a little wrong and it won't matter much. A problem can arise where you have sharp internal corners, but thats life.
What is the area contained though?
Lets see, since I know the actual shape of the region, I can easily compute the area as:
1 - 4*(pi*0.3^2/2)
ans =
0.43451
Not bad at all.
By way of comparison, consider what a Delaunay triangulation does on the same data.
D = delaunayTriangulation(xy);
triplot(D)
As you can see, it is those triangles with the long edges around the pererimeter that we need to erode away, and the alpha shape did that spectacularly well. Remember that a Delaunay triangulation will always create a convex domain.
I would point out that with fewer points, the underlying delaunay triangulation that is eroded to create the alpha shape will be a coarse one, and therefore you may have some issues if you make alpha too small. But since you do have moderately sharp internal corners there, you want a small alpha ball. Hey, the real world is like that, and I've only once ever met a client in my career as a consultant who offered more data than I really wanted to do the best job possible. Remember that your eyes/brain form a very good pattern recognition utility. They do easily what computers often need real work to do well.