If I understand your question correctly, you're trying to determine if a coordinate (x,y) lies within a polygon.
First try matlab's inpolygon().
Another way to achieve that is by "ray casting". The idea is to draw a line from the (x,y) coordinate and extend it rightward until it passes all of the possible polygon edges. Then count the number of times that ray intersects with an edge of the polygon. If that number is non-zero and odd, then the coordinate (x,y) is within the polygon.
To achieve that, you'll need to loop through each edge of your polygon and determine if each line segment intersects with the virtual ray. Here's a nice tutorial on how to detect line segment intersection. And here's more info on the ray casting technique.