How to fix the mapping between colors and values?

I have a few matrices containing different integers, and I know all the possible intergers which can be found within these matrices. But not every integer will appear in each matrices, like:

% possible integers 0 1 2 3
x = randi([0 3],10,10); % from 0 to 3
y = randi([0 2],10,10); % from 0 to 2
z = randi([0 1],10,10); % from 0 to 1

Then how can I fix the mapping between matrix integers and RGB colors so that specific integer is always associated with specific color( as commented ) ?

cmap= [0 0 1;... % blue for 0
          0 1 0;... % green for 1
          1 0 0;... % red for 2
          1 1 1]; % white for 3
      
figure;
imagesc(x);
colormap(cmap);

figure;
imagesc(y);
colormap(cmap);

figure;
imagesc(z);
colormap(cmap);

Answer 1

[zhehao.nkd]

I found out the solution by using

caxis manual
caxis([0,3]); 

But the prerequisite is that the values should be linearlly distributed with the same gap in between.

Answer 2

[KSSV]

x = randi([0 3],10,10); % from 0 to 3
y = randi([0 2],10,10); % from 0 to 2
z = randi([0 1],10,10); % from 0 to 1

cmap= [0 0 1;... % blue for 0
          0 1 0;... % green for 1
          1 0 0;... % red for 2
          1 1 1]; % white for 3
imagesc(x)
colormap(cmap)
colorbar