I see a few issues that are preventing your code from running. For starters, the code at the end of your script is your function definition. The function definition just tells MATLAB what you want it to do when you use c_muscle. You need to actually call the function in order to generate the variable x. This call would happen earlier in your script, before the call to plot(o,x) to avoid the Undefined function or variable error.
The second is that you're using a variable in your function that you don't pass to it. Unless you add o to the inputs list, the c_muscle function won't know about the variable o in your script.
Finally, you don't assign a value to your output, p, in the function. If you want the output to be x, you could change the function definition to output x as the variable.
Here's a modified version of your script that runs fine for me:
t = 0.50;
o = 0:0.5:3;
x = c_muscle(t,o);
title('x vs. o','FontWeight','bold');
x_max = max(x)
function x = c_muscle(t,o)
x = (t.*o).*(1-o)./(t+o)
I hope this helps!