The problem is, even a simple function as you have written is is not as simple as you think. This is a problem of understanding the mathematics, before you just throw it into a tool like MATLAB or Mathematica.
If you want to maximize that function over an interval, then the solution will occur either at a root of the derivative, OR it will be at one of the end points of the interval. here the interval is semi-infinite, thus x>0, so open at the left end and unbounded on the right.
With 0 < b < 1, we will have a derivative singularity at x==0, but even so, at x==0, the function itself will be zero. We can look at an example curve, here with a=2, and b=0.25.
That sample curve is actually rather common for the members of this family, with positive a and b in the interval (0,1). All such curves will have the same general shape.
The maximum will therefore lie at the root of the first derivative:
xmax = solve(diff(a*x^b - x,x,1) == 0)
Of course, I could have done that by hand as easily. But this is a MATLAB forum.
Sadly, this really does not solve your problem, which is surely vastly different than this trivial example. But my point is, what you need here is to spend some time with the mathematics that would be employed. No, you may not be easily able to just throw it into a general calculating tool of any sort. Of course, I cannot know, since we have not seen your real problem.