max(matrix,[],3) means that you want to take the maximum along the third dimension. The result will have the same size as matrix except that the third dimension will become length 1.
For example if you had an RGB image then it would be 3 dimensional, with the third dimension reflecting the Red, Green, and Blue components. max(YourImage,[],3) would then be a pixel-by-pixel array of which color component was maximum for that pixel (a calculation useful as part of correcting for illumination.)
