Intersection over union (IoU) calculation for evaluating an image segmentation model

In the simplest case, segmentation is the process of dividing a digital image into several segments. The result of instance segmentation using Mask R-CNN is a mask applied to the desired object and a bounding box around this object.

In a practical task that I was solving, it was necessary to determine the buildings in the Google Earth photos. This task was successfully accomplished using Mask R-CNN for instance segmentation. After training the model, it was necessary to evaluate it quality for prediction the bounding boxes around buildings. I would like to acquaint readers with the approach (with code) I have used to evaluate the quality of predicting the coordinates of bounding boxes.

Intersection over union (IoU) is known to be a good metric for measuring overlap between two bounding boxes or masks.

👁 Image is created by Oleksii Sheremet with Microsoft Visio

If the prediction is completely correct, IoU = 1. The lower the IoU, the worse the prediction result.

👁 Image is created by Oleksii Sheremet with Microsoft Visio

All code was implemented in Google Colab. Let’s take a closer look. Import libraries and load source image:

import cv2

import numpy

import matplotlib.pyplot as plt

img = cv2.imread("./some_image.jpg")

plt.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))

👁 Image is created by Oleksii Sheremet with matplotlib module

Import libraries and load source image:

# An example of first bounding box

first_bb_points = [[250, 210], [440, 210], [440, 390], [250, 390]]

stencil = numpy.zeros(img.shape).astype(img.dtype)

contours = [numpy.array(first_bb_points)]

color = [255, 255, 255]

cv2.fillPoly(stencil, contours, color)

result1 = cv2.bitwise_and(img, stencil)

result1 = cv2.cvtColor(result1, cv2.COLOR_BGR2RGB)

plt.imshow(result1)

Let’s set the coordinates of the original bounding box and fill the entire content of the photo outside the bounding box with black:

👁 Image is created by Oleksii Sheremet with matplotlib module

The coordinates of the second bounding box are obtained as a result of prediction:

# An example of second bounding box

second_bb_points = [[280, 190], [438, 190], [438, 390], [280, 390]]

stencil = numpy.zeros(img.shape).astype(img.dtype)

contours = [numpy.array(second_bb_points)]

color = [255, 255, 255]

cv2.fillPoly(stencil, contours, color)

result2 = cv2.bitwise_and(img, stencil)

result2 = cv2.cvtColor(result2, cv2.COLOR_BGR2RGB)

plt.imshow(result2)

Like in the previous case, fill all the content outside the bounding box with black:

👁 Image is created by Oleksii Sheremet with matplotlib module

By filling images with black outside of the bounding boxes, pixels outside the area of interest do not affect the IoU calculation:

# IoU calculation

intersection = numpy.logical_and(result1, result2)

union = numpy.logical_or(result1, result2)

iou_score = numpy.sum(intersection) / numpy.sum(union)

print('IoU is %s' % iou_score)

Thus, we get the following result:

IoU is 0.7625239952938262

Conclusion

The proposed approach has worked well in practice. It is possible to define overlap between image areas, not necessarily rectangular. You can calculate the coordinates of the contour around the masked object, for example, using OpenCV. Then using these points and the approach described above, you can calculate IoU between the masks.

References

• Mask R-CNN for Object Detection and Segmentation
• Open Computer Vision Library
• Intersection over Union (IoU) for object detection

Share This Article

• Share on Facebook
• Share on LinkedIn
• Share on X