Ihope when you mentioned ‘visibility flag’, you meant theIcon Transparencytab inMask editor,which corresponds toIconOpaqueproperty inSimulink.Maskclass.
If by ‘Inside a Masked-Subsystem’, you meant about displaying an image on the Mask Icon then you can go to Mask->Edit Mask(Ctrl+M) and in the Icon Drawing pane, you can write the code for your Image.Forexample :-
Mobjt = Simulink.Mask.get('test/Sub');%Creating a Mask object for required Subsystem
if strcmp(Mobjt.IconOpaque,'on')%When the Icon Transparency is Opaque
A = imread('1.jpg');%Read the required image
image(A);%Displaying it on the Mask
else %When the Icon Transparency is Transparent
B = imread('2.jpg'); %Read the required image
image(B);%Displaying it on the Mask
But, if you want to display your Image inside the Subsystem then you can create another Subsystem(inside the Subsystem) and repeat the same steps as above.
You can also look in the following link below on How to Draw Mask Icon : -
Regrettably i did not indent to modify the MaskDisplay. I would like to display pictures inside a mask of a subsystem where the picture is changing occording to the mask settings (like displayed Swap_pictures.jpg attached).
To achieve this goal i would like to control the picture_visibility flag (like manually done in mask_dialog.jpg )