The convex hull of a 2-d data set will be simply a polygon. So the convex hull code returns the set of points in the polygon, in order.
polynodes = convhull(XY(:,1),XY(:,2))
So there were 5 points in the convex hull. You can traverse around that polygon in that sequence.
However, in a higher number of dimensions, you cannot simply traverse around a polygon. In 3-d or higher, we would have a polyhedron. So in 3-d, it would be composed of triangular facets. And it simply would make no sense to try to write that as a simple piecewise linear path.
So look at the output of convhulln, on the same data.
We now see a list of the 5 edges. Each row of the result is an edge. If you look carefully at those edges, we can go from node 1 to 2, to 3, to 4, to 5, and then back to node 1, in that sequence.
However, the convechull will be composed of facets in a higher number of dimensions.
What happens in 3-d? Here, they will be a list of triangles.
Now convhull cannot return a simply polygon. A polygonal path would be meningless in this context, since a polygonal path could not describe the SURFACE of an object in 3-d or N-d for higher dimensions. Instead, it will return essentially the same result as convhulln, though the facets need not be listed in the same order. They need not even be in the same orientation.
So here I see facet [1 6 9] listed, but it appears as facet [1 9 6] in the latter call.
A final note is that convhull was written some years before convhulln, as I recall. So the use of convhull to produce a simple polygon has been often used in legacy code. For backwards compatibility, they surely left convhull alone, then added the convhulln code. convhulln is FULLY consistent in how it returns an output in 2 or more dimensions.
Anyway, if what you want is a consistent output, then just always use convhulln. I fail to see any problem. (Personally, I sometimes use each of these codes, because SOMETIMES, I want to see the path style output that convhull produces in 2-d.)