It sounds like you're trying to estimate the inverse CDF function in kdensity.
This is basically done in two steps in Matlab:
- Using the kernel density estimation, compute an initial inverse CDF
- Use Newton's method to refine this estimation based on the change in the function value and grid size
It tries to make this roughly less than 1e-6 in terms of total change, and gives itself 100 iterations of Newton's method to do this.
The warning you are seeing indicates that kdensity tried to do this, but wasn't confident it was able to find a sufficiently good approximation within the 100 iterations. This is a non-fatal error. I would suggest two things:
- You could compute the empirical CDF, and the inverse CDF, then compare the two and see how good you think the inverse CDF is. If it's acceptable, you can ignore this warning.
- You can try excluding outliers, centering, or adding data to the estimation to help the estimation do a better job of computing the CDF.