error removing subplot listeners manager

After plotting some subplots I start to move around the axes (change position values). Normally this works fine but I'm running into an error with an invalid or deleted object in the matlab.graphics.internal.removeFromListeners function

The first part of the code fails at the delete command:

% Create subplot listeners to align plot boxes automatically 
if isappdata(ax, 'SubplotDeleteListenersManager')
     temp = getappdata(ax, 'SubplotDeleteListenersManager');
     delete(temp.SubplotDeleteListener); %<= this is failing ...
     rmappdata(ax, 'SubplotDeleteListenersManager');
end

What is this and why is it causing a problem?

Edit: I tried calling this code manually before the code that had been triggering it and it still fails. Seems like some sort of race condition or that rmappdata is not working as expected ...