Camera parameters to extrapolate a point in world coordinates

I have an image that includes an arc which is part of a large circle that does not fit fully inside the image. I use arc points to fit a circle and find the circle center which truns out to be outside the image (at a distance 5 times larger than my image length). The problem is when I use pointsToWorld function to convert pixel locations to world dimensions (mm) I get an unreasonale number for raduis which is large than my actual object radius. I feel like this may be due to radial/tangential distortion coeficients in camerParams loosing accuracy to extrapolate worldPoint for a point outside image domain. Any idea how to resolve this?