Not able to get accurate true positive (TP), true negative (TN), false positive (FP) and false negative(FN) values in image watermarking

I want to understand the basic idea of TP, TN, FP and FN, with respect to following points:

(1) Actually, I am performing tamper detection in image watermarking. At first, some watermark bits are embedded inside the original image to produce the watermarked image. Then, watermarked image is tampered to produce the tampered image.

(2) Now, the watermark bits are extracted from both the watermarked image and the tempred image.

(3) Then, these bits are converted into two binary images and assgined to 2 variables, such as (1) reference and (2) toTest.

(4) Then I am following the below given code to find the TP, TN, FP and FN. The problem is, for any amount of tampering the results for FP and FN comes as 0 only. AM I missing any logic or trick, plz clarify me.

function [TP,FP,TN,FN] = compareBinaryImages( reference, toTest )
 %   reference = ground truth binary image
 %   toTest = binary image to be compared to the reference image
 if(ndims(reference)~=2 && ndims(toTest)~=2) 
        error('Inputs must be two 2-dimensional matrices'); 
 end
 TP = nnz(reference==1 & toTest==1); % True positive
 FP = nnz(reference==0 & toTest==1); % False positive
 TN = nnz(reference==0 & toTest==0); % True negative
 FN = nnz(reference==1 & toTest==0); % False negative
end

Answer 1

[Image Analyst]

Of course. It all depends on how you do your watermark extraction on the tampered image. For example if you merely subtracted the images and looked for the presence of non-zeros, then even the slightest amound of tampering would show it tampered with. For example if you merely changed the value of one pixel, your algorithm shows that as a true tamper, which it should. Of course comparing some totally different image would also show a difference and you'd say it was a tamper but it might not be. For example if you compared a picture of a beach with a photo of the Grand Canyon it would show it as being tampered with but it's probably not an alteration of your original - it's just a completely different photo. So you need to make sure that you're really comparing something that is likely to be an alteration of your original photo, like by adding noise, rotating it, cropping it, scaling it, changing brightness or colors (overall or just in regions), etc.

Since your False positives and False negatives are always both zero, then whatever you're doing to detect tamper is working fantastic, as long as you're comparing photos that should sensibly be compared. Ideally you would not want any false positives or false negatives. I'm not sure if a true positive is an indication of an original image or a tampered one, but regardless your algorithm is good enough to not make any mistakes since all your predictions are correct (true).

[Aditya Shau]

Many thanks @ Image Analyst. I understood perefectly what you are suggesting.

But, I am getting FP and FN as 0 for any kind of attack,noise or cropping.

I feel some where I am missing it. Plz clarify my doubt. Am I am following the correct procedure or not for obatining FP and FN etc..

[Image Analyst]

I guess I didn't explain it enough. Think about it. Why SHOULD you get false results? If your algorithm is robust, then why should it produce incorrect results? WHy should it say that it was tampered when it was not tampered? Why should it say that it was original when it was not original? The answer is: it shouldn't. If your algorithm is so sophisticated and good, or your tampering was so blatantly obvious that any dumb algorithm could detect things properly, then you'll always produce true and accurate results. Can you show us an example (reference and test images) where you think your algorithm should report the wrong result?

[Aditya Shau]

Dear Image Analyst, I am thankful for your kind support and guidance.

Now, I have attached the two images. one is the watermarked image and another is the tempred image. Also I have attached the two binary images for the respective images. These binary images are formed by the extracted bits of the respective images, that are been found using my extraction algorithm. Following are the points that are been found:

1. Even I have tampered 50% of the image, the FP and FN are coming as zero. Further, the watermark bits that are extracted from the tampred image is exactly same as the original watermarked bits.
2. Also, I have attached the procedure for finding the false positive rate (FPR) and FNR in another way. Plz find whether it is correct or not.

[Image Analyst]

Your whole concept of true/false positive/negative is totally wrong/misguided. Those are metrics that you should compute for the whole image. You do not compute those on a pixel-by-pixel basis like you did. You do them on the whole image to gauge whether your algorithm worked or not, and once you have those you build up your ROC curve.

What you basically did was to compare two binary images and count how many pixels were

1. both white
2. actual=white, and detected=black
3. actual=black and detected=white
4. both black

but the sum of each of those cases is NOT the true/false positive/negative count. It's a completely different thing! To get the true/false positive/negative you have to make that determination on the whole image, and then you get one point on the ROC curve. Then you do it for another image, and plot it. Then as you do it for hundreds of images you'll be able to fill out that curve.

By the way, to do what you did, you can do this. I don't recommend it since it's not right, but anyway, this is the vectorized version of what you did:

% Initialization steps.
clc;    % Clear the command window.
fprintf('Beginning to run %s.m ...\n', mfilename);
close all;  % Close all figures (except those of imtool.)
clear;  % Erase all existing variables. Or clearvars if you want.
workspace;  % Make sure the workspace panel is showing.
format short g;
format compact;
fontSize = 15;

hFig = figure;
hFig.WindowState = 'maximized'
actual =  imread('bianry image of watermarked image.bmp');
actual = logical(actual); % Convert to binary image.
subplot(2, 2, 1);
imshow(actual);
title('Original', 'FontSize', fontSize);

detected = imread('binary image of tampered image.bmp');
detected = logical(detected);
subplot(2, 2, 2);
imshow(detected);
title('Detected', 'FontSize', fontSize);

% Get all 4 cases where pixels match or don't match.
bothWhite11 = actual & detected;
different10 = actual > detected;
different01 = actual < detected;
bothBlack00 = ~actual & ~detected;

% Display all 4 cases:
subplot(2, 4, 5);
imshow(bothWhite11);
title('Both White', 'FontSize', fontSize);
subplot(2, 4, 6);
imshow(different10);
title('actual=white and detected=black', 'FontSize', fontSize);
subplot(2, 4, 7);
imshow(different01);
title('actual=black and detected=white', 'FontSize', fontSize);
subplot(2, 4, 8);
imshow(bothBlack00);
title('Both Black', 'FontSize', fontSize);

% Find out the number of pixels that don't match
nonMatchingPixels = actual ~= detected;
fprintf('The number of non-matching pixels = %d.\n', nnz(nonMatchingPixels));

% Say how many occur for each of the 4 cases:
num11 = nnz(bothWhite11);
num10 = nnz(different10);
num01 = nnz(different01);
num00 = nnz(bothBlack00);
fprintf('The number of matching pixels where both are white = %d.\n', num11);
fprintf('The number of mismatched pixels where actual=white and detected=black = %d.\n', num10);
fprintf('The number of mismatched pixels where actual=black and detected=white = %d.\n', num01);
fprintf('The number of matching pixels where both are black = %d.\n', num00);

You'll see:

The number of non-matching pixels = 0.

The number of matching pixels where both are white = 133088.

The number of mismatched pixels where actual=white and detected=black = 0.

The number of mismatched pixels where actual=black and detected=white = 0.

The number of matching pixels where both are black = 129056.

As you can see, your two images are identical (did you upload the wrong ones?).

[Aditya Shau]

Excellent explanation. Thank you very much. I understood about FP, FN, TP,TN.

But, with these values how can I plot th ROC curve. I request you to kindly give some information about ROC curve, its use and everything for that it is required,

Thank you.

[Image Analyst]

Well first read the wikipedia article on ROC curves. Then what you do is to basically pick a set of parameters for your algorithm, like window width, order, whatever, then run a bunch of images through your algorithm and get the true positive and false positive rates. Then plot that as one point. Now that's not a curve yet, it's just one point on a curve, so to build up the curve you need to change the parameters and run another (or the same) batch of images and note the TP and FP rates. Now you have two points in the curve. Now do it a bunch more times and get more and more points along the curve and you'll find something like the shape in the ROI image I showed you. Then you can make a decision as to what's most important to you and pick that set of parameters. It's not true that the point on the upper left of the curve is always the best set of parameters to use. For example if you were doing something where people might die if you got it wrong (cancer diagnosis) you might want a really high TP rate, even if the FP rate is so high. You'd be willing to have a few more diagnoses of cancer even though that was not true rather than risk telling someone that they don't have cancer when they really do.

[Aditya Shau]

Dear sir, I read some articles on ROC. And understood the basic idea of ROC.

However, I have attached some sample results of my experiment. I request you to kindly hep to plot the ROC curve with these values. Once you suggest me the procedure for plotting, I will add some more experiment results on this and follow your procedure to build the plot. Actually, I have stored my results in excell sheet and facing some issues in bulding the plot in matlab. Thank you once again for your great help and support. Thank you.