how to train negative example in faster RCNN model

Hi,

I'm working on a faster RCNN model, and I'm asking how training this model with negative examples?

can anybody help me to know that!

Answer 1

[Raunak Gupta]

Hi,

From negative examples I assume you mean images in dataset which doesn’t have the objects in it. So, if you are training Faster RCNN detector using trainFasterRCNNObjectDetector I would suggest including some images as follow

1. Images which doesn’t have proper bounding boxes surrounding the object that is there is a mis match between object boundaries and actual bounding boxes.
2. Images where object doesn’t exist but there are bounding boxes.
3. Images where object is there but bounding boxes are not there.

I assume you may not have images satisfying above three conditions So, you can create your own dataset by labelling the images in Image Labeler App.

[Rummy18]

Hi Raunak,

Concerning 1: Should the bounding boxes have the same labels as the objects, which should be detected? E.g. "car"? Or something else, like e.g. "negative"?

Concerning 2: Where should the bounding boxes be placed (randomly?) and which label should be assigned to them (see 1)?

Concerning 3: If the object of interest is in the image, however not labeled, wouldn't this lead to a misclassification? Or how is such an image used during the training?

How to deal with objects that are quite similar? One is positive, but the other is negative?

--

Is it possbile to use seperate folders with positive and negative images as input data, as for example in the trainCascadeObjectDetector? In the case, the Image Labeler App was not used.

Thank you very much!

[Raunak Gupta]

Hi,

From the negative examples I see the examples that are misclassfied. So for example you are training for object of class 'car', If a bike is labeled as car it will be a negative example. Also if the object car is not present in the image and still it is detected that can also be treated as a negative example.

So for the first two concern I think the label should remain to 'car' and bounding boxes can be placed randomly or on any false object (as above example of bike when the trained class is 'car').

For the third concern I think the negative examples are introduced in first place to make the detector more robust to outliers or unseen data, so the missclassification may help.

For dealing with similar objects I think the classifier network should be trained with more example of both classes to effectiviely differentiate the object.

For seperate folders, I think it is not required to put them in seperate folder as long as you have labels. If the positive and negative samples are seperatly labeled, it defeats the purpose of putting negative examples as it won't produce missclassification.

Image Labeler App can always be used for creating the labeled data irrespective of object detector network.

Hope it helps!

[Dominique Chabot]

Hi Raunak,

I have a very similar question concerning training a Faster R-CNN object detector (as well as a YOLO v2 object detector to compare the performance of the two detectors), and I'm not sure the information you provided above satisfactorily answers my question.

I'm working with a very large image set (~60,000 aerial images collected for a polar bear survey) that is characterized by very few examples of the objects of interest (polar bears) relative to a massive abundance and diversity of 'background', including various background elements that could potentially be mistaken for polar bears by the object detector.

Creating training data representing the polar bears is very straightforward, using Image Labeler, etc. However, to prevent a huge number of false positives thoughout the large image set, I also want to train the network on a large number of 'background' examples, and am wondering what is the best way to do that.

So far, my idea to create background training samples is to create an imageDatastore of the entire image set minus the images known to contain polar bears, then use subset to randomly select a certain % of images from the full set, and finally use randomPatchExtractionDatastore to randomly extract patches of 'background' from the subset of images.

But I'm not sure what to do next. All the help files and tutorials for Faster R-CNN and YOLO v2 only refer to training samples that contain the objects of interest; but in my case, the small amount of background surrounding the polar bears in my labeled polar bear training images will not be nearly enough to teach the network all the different background variations in the full image set. And from the help files, it isn't clear to me how or if it would be possible to add my randomly extracted background patches (as described above) to the training data.

I understand that the trainingData input argument for trainFasterRCNNObjectDetector and trainYOLOv2ObjectDetector can ultimately be a simple 2-column table, with the 1st column specifying the training image filenames and the 2nd column specifying the position and size of the bounding box around the object of interest (in this case, 'polar bear') in each image. So I've been wondering if can simply include my randomly extracted background images (containing no polar bears) in this table and not have any bounding boxes associated to them in the 2nd column? Would that be considered valid input by the trainObjectDetector function (i.e. would the network actually learn background variations from these images that don't contain any bounding boxes and help reduce the false positive rate?). And if so, what would be the correct syntax for the table? For example, would this work:

Column 1: imageFilename Column 2: polarBear

'polarbearimage1.jpg' [980,393,31,56]

'polarbearimage2.jpg' [635,254,65,63]

'backgroundimage1.jpg' []

'backgroundimage2.jpg' []

Etc.

I would greatly appreciate any help or advice you can provide. Thank you!

[Raunak Gupta]

Hi Dominique,

I understand that you want to include the background portion seperately (without bounding boxes) to training the Object detector, but the main motto of object detection is to actually detect object that can be confined within a boundary and have a discrete label associated with it. Currently the trainFasterRCNNObjectDetector only accept the data with such methodology. This is due to the algorithm behind FasterRCNN Object detector which is suitable for getting classification and a regressed bounding box coordinate output.

Maybe the technique you are trying to implement is completely new, and can be think of as a research problem but I am not aware of any such methodology. Probably you can try to see how you can formulate this problem mathematically of not giving bounding boxes for particular images and see whether by some mathematical means the training can be benefitted with this approach.

Hope this provide some guiding light for your work :)

[Dominique Chabot]

Hi Raunak,

Thank you very much for your prompt reply and for the information.

I can assure you that the idea of training a CNN on samples of the target objects as well as separate samples of 'background' is not new, as I've been taking cues from several published papers that have done this for different kinds of animal surveys.

However, your helpful confirmation that bounding box-based object detectors can only be trained on images that contain at least one example of the target objects (no 'empty' background images) caused me to review the literature again and make some helpful observations:

1. Indeed, all the papers I have that used bounding box-based object detectors trained the networks only with images containing one or more animals (or at least one bounding box). The reason this straightforward approach is not suitable for my particular case is because the massive abundance and diversity of background in the full image set cannot be adequately learned by the network if I limit the training samples to only the small number of images containing polar bears.
2. I realized that all the studies that trained CNNs with separate background images (containing no animals) did not actually use bounding box-based detectors, but rather 'whole image' classifiers where the original full-resolution aerial images were subdivided into a grid of small patches (e.g. 224 x 224 pixels), and each individual patch was classified by the CNN as either "contains animal" or "does not contain animal". Thus, the network is trained on 2 classes of images: 'animal' and 'no animal', with latter being equivalent to what I've been referring to as 'background'. If I'm not mistaken, I believe I could likely adapt the approach I described in my initial question above to this technique using MATLAB's whole image-based classification workflows (e.g. like this) instead of the bounding box-based object detectors. Perhaps you can confirm?
3. There is one bounding box-based study, however, that employed an interesting strategy to incorporate separate training samples containing no animals, and I think it actually relates to Rahaf's original question above regarding "negative examples". The authors initially trained a CNN using only images containing animals from a first image set. Then they ran the trained CNN on a second image set, which generated a certain number of false positives (erroneous bounding boxes in places where no animal was present). Then they re-trained the CNN by additionally incorporating these false positives and inputting them as "negative examples" (thus, these images satisfy the criterion that there must be a bounding box in the image). I have attached the paper in case you're interested to see for yourself; the approach is very concisely described on page 3 in the "Algorithm Training" section. I believe this approach may also be viable for my case, but it brings me back to Rahaf's questions about how exactly to incorporate negative examples into the required input formats of the trainFasterRCNNObjectDetector function (i.e. either the 2-column ground truth table or a ground truth datastore). Specifically, how do you indicate (in terms of syntax) that a particular ground truth object in the training set is a negative example? Because it seems to me that if you don't explicitly tell the network that it's a negative example, then it will treat it as a positive example like all the others and erroneously train the network to detect non-animal background features, which would actually increase the false positive rate! Do you see what I mean?

Again, thank you very much for taking the time to comment on this.

Regards.

[Raunak Gupta]

Hi Dominique,

Glad, I could help.

The approach mentioned in second point looks the way to go. Separately doing the classification on patches looks okay because you mentioned mostly the data is background only.

For the third point, I still can reiterate that by negative example I mean the object that is present in the image is not the correct one. Surely having no object in the image makes it challenging to incorporate it into training data because inherently we assume there is some object in the image that we are using in training.

Perhaps you can train the network with two categories of bounding boxes instead of one where first one is for polar bear and second one can be for background so any negative example you have, can come in the second category. So, the groundTruth will become a 3-column table.

For the images in which a category is not present those can be represented in empty vector in one of the 2nd or 3rd column. For category, which is present, mx4 vector can be added again in 2nd or 3rd column where m is the number of objects of that category.

Then you can define custom training loop with help from here which can favour the training of polar bear category and penalize the background category of bounding boxes.

Unfortunately, I couldn't look into the research paper you shared (because of time constraints) but I am sure above information can help you implement the custom training scenario.

Example of the table

polar_bear       background
[39,35,37,49]	[88,23,13,28]
[43,49,33,38]	[]
[59,44,32,44]	[20,16,26,31]
[]	        []

[Dominique Chabot]

Thank you for your further reply, Raunak.

It's good to know that you think the approach in the 2nd point is achievable, and it indeed looks like perhaps the best/easiest way to go.

I had also thought of the possibility of training the object detector on a dedicated 'background' class, and I thank you for the example of how to format the table. I feel like this approach might end up being more of a hassle than the 'whole image' classification approach, but I'll definitely give it some more thought.

Thanks again, and take care!

[hammad younas]

Hi there,

Training a faster R-CNN network with own defined "background class" is a big mess. It is for the reason that you are treating the background class as "object of interest" which is not your intention. Thus i would suggest NOT to use this approach as you would end up with incorrect results.

Regards

Answer 2

[Rus Gabriela]

Hi Raunak,

I have a question too.

I am really new to Matlab and AI and I want to learn a little bit about object detection. I made Object Detection Using YOLO v2 Deep Learning from examples and works fine, but I want to test this detector on a new image (not from TestData). My question is: Can I use this Detector for unlabeled pictures (not from training data or TestData) from my PC?

[yanqi liu]

yes，sir，may be use

[bboxes,scores] = detect(detector,Ii,'Threshold',0.15);

to detect target on new image

[Rus Gabriela]

Thank you! I used this variable but not like this and wasn't work, but now it's fine.