Your unknown values are the upper (say) triangular elements in y.
Create f to handle the upper-triangular y, say by using the tril function along with transpose. I mean, if y is upper triangular, then
is a symmetric matrix with the same upper triangular part as y. Then you can define your objective and constraint for fmincon easily.
Or to make it even easier on you (but harder on fmincon), let y be a general matrix, and add a nonlinear equality constraint y' = y.
MATLAB mathematical toolbox documentation