Why not do the obvious?
xy = rand(1000,2);
>> ch = convhull(xy)
What does the output of convhull tell you? What do those numbers mean? You started with a list of 1000 points. Which points are not part of the convex hull? If they are not used in the convex hull, what does that tell you about them?
I suppose, since you figured out how to compute the area, then you are actually computing with the delaunayTriangulation tool first?
T = delaunayTriangulation(xy(:,1),xy(:,2))
delaunayTriangulation with properties:
Points: [1000×2 double]
ConnectivityList: [1982×3 double]
ch = convexHull(T);
You will get to the same place. So now, you need to think about what that list of numbers means, and how it tells you how those points which are internal to the convex hull are removed.
As far as knowing when a point lies on the surface of the convex hull, we might try this:
Methods for class delaunayTriangulation:
barycentricToCartesian delaunayTriangulation featureEdges isInterior size
cartesianToBarycentric edgeAttachments freeBoundary nearestNeighbor vertexAttachments
circumcenter edges incenter neighbors vertexNormal
convexHull faceNormal isConnected pointLocation voronoiDiagram
Do any of the methods you see listed there give you any ideas? For example, might the isInterior method help you?
Another idea is to consider if the inpolygon function could have helped you, given the output of convhull? How could you use ch as I computed it in several ways above, to compute something that inpolygon can use? Or, perhaps you might use polyshape?
x = [0 1 .5 0 .25];
>> y = [0 0 .5 1 .25];
>> ch = convhull(x,y)
>> S = polyshape(x(ch),y(ch));
Warning: Polyshape has duplicate vertices, intersections, or other inconsistencies that may produce inaccurate or unexpected results. Input data has been
modified to create a well-defined polyshape.
> In polyshape/checkAndSimplify (line 481)
In polyshape (line 175)
polyshape with properties:
Vertices: [3×2 double]
Get your hands dirty. Play around. try a few things. Read the help for some of these methods, some functions. I'd wager you will learn a lot along the way.