pad the image with nan so that it becomes sqrt(2) times its original size. Do the random rotation on that. Find the bounding box of non-nan elements and crop. Fill the remaining nan with something appropriate.
Or... don't use the image augmenter, and use imrotate with a random angle, specifying an appropriate padding strategy.