If a solution exists to the problem, you finding it in a fully robust way using the transformation A*V == V*F may be problematic. At least so unless you are careful with the mathematics. Doing so admits the trivial solution V == 0. This happens because the transformed problem is now implicitly a homogeneous linear system of equations. As such, both the fsolve solution posed, as well as the sylvester solution posed are subtly wrong.
That does not mean the problem is impossible to solve however. How might we find a non-trivial solution to the transformed problem? We might try the good old kron trick. kron is a slick way of unrolling such matrix problems. Essentially, we create a new homogeneous linear system, where n is the number of rows or columns of the matrix A.
( kron(eye(n),A) - kron(F,eye(n)).' )*V(:) == 0
A problem when we do this is it does not force V to be invertable. It also recognizes that V is not unique. Just for kicks, lets try an example. A simple test case will show the problem.
By construction, we have created a test problem where a solution must exist. Now, can we recover that solution, if we knew only A and F? Construct the matrix M as:
M = kron(eye(n),A) - kron(F,eye(n)).';
(To understand how and why this works, you need to see what kron does with your matrix as I used it there twice. TRY IT!) M will be a 9x9 matrix here. If a non-trivial solution exists to the problem M*V(:) == 0 exists, then we MUST also have a solution to the problem A*V == V*F.
So M is as expected, a 9x9 matrix. It has rank 6 though. So there is a 3-dimensional subspace of solutions to the problem A*V==V*F. The solution will ne non-unique.
Does non-uniqueness make sense? Yes, perfect sense. For example, suppose you have found a valid solution V to the problem A==V*F*inv(V). Now consider the matrix k*V. Since linear algebra allows us to extract such a constant from matrix products, and since inv(k*V) = 1/k*inv(V), as long as k is non-zero, then we should see that
(k*V) * F* inv(k*V) == (k*1/k) * V*F*inv(V) == V*F*inv(V) == A
So clearly any solution V MUST be non-unique. (But scaling is not the only issue, as we will see later.) Anyway, that means we cannot so easily recover the original matrix V. Let try it. The way to solve the linear homogeneous problem is to employ null.
Any linear combination of the columns in Mnull will be a solution. Vor example, we might try this:
V1 = reshape(Mnull(:,1),3,3)
As you can see, it is not the same as V. But is it a valid solution? Perhaps.
Indeed, it is a solution. As long as V1 was invertable, then it MUST be A valid solution. But as easily, I could have chosen a completely different matrix for V, thus
V2 = reshape(Mnull*randn(3,1),3,3)
And again, we see that to within floating point trash, V2 is also a valid solution. Can I find the linear combination of columns that would have yielded the original matix V? Of course, but that is only because I know V itself.
So we have
Vrestored = reshape(Mnull*lincomb,3,3)
Now compare that to the original V, and we see they are the same, as always to within floating point trash.
So the solution we wanted to find was indeed hidden inside the columns of Mnull, though there is no valid reason to distinguish any of the infinitely many linear combinations we could have chosen.
We should recognize that as long as the solution chosen represents an invertable matrix V, then it MUST be a valid solution. But also that the solution is never unique, unless A and F were 2x2 matrices. In that case, my gut tells me that M should be a 4x4 matrix of rank 3.
As such, I've given you the complete solution. It takes only these four lines of code, with no iterative methods needed. Do with it as you wish.
M = kron(eye(n),A) - kron(F,eye(n)).';
V = reshape(Mnull(:,1),n,n);
I've chosen the first column of Mnull. This is completely arbitrary. In the event that V as chosen (incredibly rarely) results in a singular matrix, just choose a different column of Mnull, or choose some random linear combination of the columns. Can we insure the solution must result in a nonsingular matrix V? While the columns of Mnull must form a set of linearly independent columns, that does not force the resulting nxn matrix to be also non-singular that I see. Even if both A and F are full rank, we could still have a non-trivial solution for V that is less than full rank, when written in the form A*V-V*F == 0.