FMINCON REQUIRES a continuous, differentiable objective function (and constraints).
If you use mod inside the objective function, or discretize something inside like that in any way (perhaps using round or floor), then your function is not at all continuous nor is it differentiable. So FMINCON must fail on your objective.
FMINCON has no capability to allow integer valued parameters in any way.
If you insist on the use of FMINCON, at best you could remove the mod from inside your function, then round the result afterwards. There is no assurance the result will be truly optimal then, but that is the best you can do using FMINCON.
You cannot use FMINCON to solve your problem. I believe some other tools, such as GA from the Global optimization toolbox, can do what you wish.
