You are trying to use a tool not designed to solve that class of problem. You cannot round the results AFTER fmincon returns a result, because it may not be optimal. In fact, the rounded solution might not even be feasible, satisfying some of the constraints.
Nor can you round the unknowns inside your objective function. That would effectively create a discontinuous objective, something that fmincon is NOT designed to work with.
Problems with one or more variables that must be an integer are called integer programming problems. If only some of them are integer constraints, it still effectively counts as such, though then it would probably be called mixed integer programming. If the variable is constrained to beeither 0 or 1, that is usually called a binary integer programming problem. This is not something that fmincon does. (Though I do recall a tool called fminconset, that was on the file exchange. I'm not sure if it is still there, or if it still works. You might look.)
In MATLAB, look in the Global Optimization toolbox, where you can find GA. It is designed to solve that class of problem.
Finally, if you really have only ONE variable that is binary, then just solve the problem twice. First, with the variable replaced by 0. Then with it replaced by 1. Take the better solution from the two runs, and you are done. As Walter points out, this approach even works if you have a few binary variables, as long as you do not have too many of them.