Take one observation and compute prediction for that observation. Then replace some predictor in that observation with NaN and recompute prediction. Use the magnitude of the difference between the two predictions as a measure of importance for that predictor. If you know the observed (true) value of prediction, you can look at how close the predicted values are to the observed one.
This is just an idea off top of my head. I have not seen anything in the literature for one observation.
