How to build a Face Mask Detector using RetinaNet Model!

Introduction Object detection is a tremendously important field in computer vision needed for autonomous driving, video surveillance, medical applications, and many other fields.

We are grappling with a pandemic that’s operating at a never-before-seen scale.

Researchers all over the globe are frantically trying to develop a vaccine or a cure for COVID-19 while doctors are just about keeping the pandemic from overwhelming the entire world.

On the other hand, many countries have found social distancing, using masks & gloves a way to curb the situation a little.

I recently had an idea to apply my deep learning knowledge to help the current situation a little.

In this article, I’ll introduce you to the implementation of RetinaNet with little background & working on it.

The cherry on top? We’ll build a “Face mask detector” using RetinaNet to help us in this ongoing pandemic.

You can extrapolate the same idea to build an AI-enabled solution for your smart home.

This AI-enabled solution would open the gate of your building to only people, who are wearing masks and gloves.

As the cost of drones is decreasing with time, we see a large spike in generation of aerial data.

So, you can use this RetinaNet model to detect different objects such as automobile vehicles (bikes, cars, etc.

) or pedestrians in aerial images or maybe even in satellite images to solve your different business problems.

So, you see applications of object detection model is endless.

  Table of Contents What is RetinaNet Need for RetinaNet The Architecture of RetinaNet Backbone Network Subnetwork for object Classification Subnetwork for object Regression Focal Loss Build Face mask detector using RetinaNet model Gather Data Create Dataset Model Training Model Testing Final Notes   What is RetinaNet: – RetinaNet is one of the best one-stage object detection models that has proven to work well with dense and small scale objects.

For this reason, it has become a popular object detection model that we use with aerial and satellite imagery.

   Need for RetinaNet: – RetinaNet was introduced by Facebook AI Research to tackle the dense detection problem.

It was needed to fill in for the imbalances and inconsistencies of the single-shot object detectors like YOLO and SSD while dealing with extreme foreground-background classes.

  The Architecture of RetinaNet: – In essence, RetinaNet is a composite network composed of: Backbone Network (i.


Bottom-up pathway + Top down a pathway with lateral connections eg.

ResNet + FPN) Subnetwork for object Classification Subnetwork for object Regression Image Source For better understanding, Let’s understand each component of architecture separately- The backbone Network Bottom up pathway: – Bottom up pathway (eg.

ResNet) is used for feature extraction.

So, It calculates the feature maps at different scales, irrespective of the input image size.

Top down pathway with lateral connections: – The top down pathway upsamples the spatially coarser feature maps from higher pyramid levels, and the lateral connections merge the top-down layers and the bottom-up layers with the same spatial size.

Higher-level feature maps tend to have small resolution though semantically stronger.

Therefore, more suitable for detecting larger objects; on the contrary, grid cells from lower-level feature maps have high resolution and hence are better at detecting smaller objects (see Fig.


So, with a combination of the top-down pathway and its lateral connections with bottom up the pathway, which do not require much extra computation, every level of the resulting feature maps can be both semantically and spatially strong.

Hence this architecture is scale-invariant and can provide better performance both in terms of speed and accuracy.

Subnetwork for object Classification A fully convolutional network (FCN) is attached to each FPN level for object classification.

  As it’s shown in the diagram above, This subnetwork incorporates 3*3 convolutional layers with 256 filters followed by another 3*3 convolutional layer with K*A filters.

Hence output feature map would be of size W*H*KA , where W & H are proportional to the width and height of the input feature map, and K & A are the numbers of object class and anchor boxes respectively.

At last Sigmoid layer (not softmax) is used for object classification.

And the reason for the last convolution layer to have KA filters is because, if there’re “A ” number of anchor box proposals for each position in feature map obtained from the last convolution layer then each anchor box has the possibility to be classified in K number of classes.

So the output feature map would be of size KA channels or filters.

Subnetwork for object Regression The regression subnetwork is attached to each feature map of the FPN in parallel to the classification subnetwork.

The design of the regression subnetwork is identical to that of the classification subnet, except that the last convolutional layer is of size 3*3 with 4 filters resulting in an output feature map with the size of W*H*4A .

Reason for last convolution layer to have 4 filters is because, in order to localize the class objects, regression subnetwork produces 4 numbers for each anchor box that predict the relative offset (in terms of center coordinates, width, and height) between the anchor box and the ground truth box.

Therefore, the output feature map of the regression subnet has 4A filters or channels.

  Focal Loss Focal Loss (FL) is an improved version of Cross-Entropy Loss (CE)  that tries to handle the class imbalance problem by assigning more weights to hard or easily misclassified examples  (i.


background with noisy texture or partial object or the object of our interest ) and to down-weight easy examples (i.


Background objects).

So Focal Loss reduces the loss contribution from easy examples and increases the importance of correcting misclassified examples.

Focal loss is just an extension of the cross-entropy loss function that would down-weight easy examples and focus training on hard negatives.

So to achieve these researchers have proposed- 1-  pt to the cross-entropy loss, with a tunable focusing parameter ≥0.

RetinaNet object detection method uses an α-balanced variant of the focal loss, where α=0.

25, γ=2 works the best.

Image Source    So focal loss can be defined as – The focal loss is visualized for several values of γ ∈[0,5], refer Figure 1.

We shall note the following properties of the focal loss- When an example is misclassified and pt is small, the modulating factor is near 1 and does not affect the loss.

As, pt→ 1, the factor goes to 0 and the loss for well-classified examples is down weighed.

The focusing parameter γ smoothly adjusts the rate at which easy examples are down-weighted.

As γ is increased, the effect of modulating factor increases likewise.

(After a lot of experiments and trials, researchers have found γ =2 to work best) Note:- when γ = 0 , FL is equivalent to CE.

Shown blue curve in Fig Intuitively, the modulating factor reduces the loss contribution from easy examples and extends the range in which an example receives the low loss.

You can read about Focal loss in detail in this article (link to my Focal loss article.

), Where I’ve talked about the evolution of cross-entropy into Focal loss, the need for focal loss, comparison of focal loss with Cross entropy.

And the cherry on top, I’ve used a couple of examples to explain why Focal loss is better than cross-entropy.

Now let’s see the implementation of RetinaNet to build Face mask Detector in Python –   Build Face mask detector using RetinaNet model Gather Data Any deep learning model would require a large volume of training data to give good results on the test data.

In this article (Link to my Web scrapping article), I’ve talked about the Web Scrapping method to gather a large volume of images for your deep learning project.

  Create Data Set: – We start by creating annotations for the training and validation dataset, using the tool LabelImg.

This excellent annotation tool lets you quickly annotate the bounding boxes of the objects to train the machine learning model.

You can install it using below command in anaconda command prompt pip install labelImg You can annotate each JPEG file using labelmg tool like below and it’ll generate XML files with coordinates of each bounding box.

And we’ll use these xml files to train our model.

  Model Training: – Step1.

Clone & install the keras-retinanet repository¶ import os print(os.

getcwd()) git clone https://github.


git %cd keras-retinanet/ !pip install .

!python setup.

py build_ext –inplace   Step2.

Import all required libraries import numpy as np import shutil import pandas as pd import os, sys, random import xml.


ElementTree as ET import pandas as pd from os import listdir from os.

path import isfile, join import matplotlib.

pyplot as plt from PIL import Image import requests import urllib from keras_retinanet.


visualization import draw_box, draw_caption , label_color from keras_retinanet.


image import preprocess_image, resize_image   Step3.

import JPEG & xml data pngPath=C:/Users/PraveenKumar/RetinaNet//maskDetectorJPEGImages/ annotPath=C:/Users/PraveenKumar/RetinaNet//maskDetectorXMLfiles/ data=pd.

DataFrame(columns=[fileName,xmin,ymin,xmax,ymax,class]) os.

getcwd() #read All files allfiles = [f for f in listdir(annotPath) if isfile(join(annotPath, f))] #Read all pdf files in images and then in text and store that in temp folder  #Read all pdf files in images and then in text and store that in temp folder for file in allfiles: #print(file) if (file.


“)[1]==xml): fileName=C:/Users/PraveenKumar/RetinaNet/maskDetectorJPEGImages/+file.



jpg)         tree = ET.

parse(annotPath+file)         root = tree.

getroot()         for obj in root.

iter(object):             cls_name = obj.


text            xml_box = obj.

find(bndbox)             xmin = xml_box.


text             ymin = xml_box.


text             xmax = xml_box.


text             ymax = xml_box.


text             # Append rows in Empty Dataframe by adding dictionaries             data = data.

append({fileName: fileName, xmin: xmin, ymin:ymin,xmax:xmax,ymax:ymax,class:cls_name}, ignore_index=True) data.

shape   Step3.

Write a function to show bounding boxes on the training dataset  # pick a random image   filepath = df.


values[0]   # get all rows for this image   df2 = df[df[fileName] == filepath]   im = np.


open(filepath))   # if theres a PNG it will have alpha channel   im = im[:,:,:3]   for idx, row in df2.

iterrows():     box = [       row[xmin],       row[ymin],       row[xmax],       row[ymax],     ]     print(box)     draw_box(im, box, color=(255, 0, 0))   plt.

axis(off)   plt.

imshow(im) plt.

show()                        show_image_with_boxes(data)   #Check few records of data data.

head()   #Define labels & write them in a file classes = [mask,noMask] with open(.


csv, w) as f:   for i, class_name in enumerate(classes):     f.

)         if not os.


exists(snapshots):   os.

mkdir(snapshots) Note: – It’s better to start with a pre-trained model in lieu of training a model from scratch.

We’ll use the ResNet50 model that’s already pre-trained on the Coco dataset.

URL_MODEL = https://github.






h5 urllib.


urlretrieve(URL_MODEL, PRETRAINED_MODEL)   Step4.

Train RetinaNet Model Note: – You can use below snippet of code to train your model if you’re using Google Colab.

#Put your training data path & file that has labels for your training data !keras_retinanet/bin/train.

py –freeze-backbone –random-transform –weights {PRETRAINED_MODEL} –batch-size 8 –steps 500 –epochs 15 csv maskDetectorData.

csv maskDetectorClasses.

csv But If you’re training on your local Jupyter notebook or different IDE then you can below command from your command prompt python keras_retinanet/bin/train.

py –freeze-backbone              –random-transform             –weights {PRETRAINED_MODEL}              –batch-size 8              –steps 500              –epochs 15               csv maskDetectorData.

csv maskDetectorClasses.

csv Let’s analyse each argument passed to the script train.


freeze-backbone: freeze the backbone layers, particularly useful when we use a small dataset, to avoid overfitting random-transform: randomly transform the dataset to get data augmentation weights: initialize the model with a pre-trained model (your own model or one released by Fizyr) batch-size: training batch size, the higher value gives a smoother learning curve steps: Number of steps for epochs epochs: number of epochs to train csv: annotations files generated by the script above   Step5.

Load Trained Model from glob import glob model_paths = glob(snapshots/resnet50_csv_0*.

h5) latest_path = sorted(model_paths)[-1] print(“path:”, latest_path) from keras_retinanet import models model = models.

load_model(latest_path, backbone_name=resnet50) model = models.

convert_model(model) label_map = {} for line in open(.


csv):   row = line.


split(,)   label_map[int(row[1])] = row[0]   Model Testing: – Step6.

Predict using trained model #Write a function to choose one image randomly from your dataset and predict using Trained model.

def show_image_with_predictions(df, threshold=0.

6):   # choose a random image   row = df.

sample()   filepath = row[fileName].

values[0]   print(“filepath:”, filepath)   # get all rows for this image   df2 = df[df[fileName] == filepath]   im = np.


open(filepath))   print(“im.

shape:”, im.

shape)   # if theres a PNG it will have alpha channel   im = im[:,:,:3]   # plot true boxes   for idx, row in df2.

iterrows():     box = [       row[xmin],       row[ymin],       row[xmax],       row[ymax],     ]     print(box)     draw_box(im, box, color=(255, 0, 0))   ### plot predictions ###   # get predictions   imp = preprocess_image(im)   imp, scale = resize_image(im)   boxes, scores, labels = model.

predict_on_batch(     np.

expand_dims(imp, axis=0)   )   # standardize box coordinates   boxes /= scale   # loop through each prediction for the input image   for box, score, label in zip(boxes[0], scores[0], labels[0]):     # scores are sorted so we can quit as soon     # as we see a score below threshold     if score < threshold:       break     box = box.


int32)     color = label_color(label)     draw_box(im, box, color=color)     class_name = label_map     caption = f”{class_name} {score:.

3f}”     draw_caption(im, box, caption)     score, label=score, label   plt.

axis(off)   plt.

imshow(im)   plt.

show()   return score, label plt.


figsize] = [20, 10]   #Feel free to change threshold as per your business requirement score, label=show_image_with_predictions(data, threshold=0.

6) #Feel free to change threshold as per your business requirement score, label=show_image_with_predictions(data, threshold=0.

6) #Feel free to change it as per your business requirement score, label=show_image_with_predictions(data, threshold=0.

6) #Feel free to change it as per your business requirement score, label=show_image_with_predictions(data, threshold=0.

6)   #Feel free to change it as per your business requirementscore, label=show_image_with_predictions(data, threshold=0.

6) #Feel free to change it as per your business requirement score, label=show_image_with_predictions(data, threshold=0.

6) #Feel free to change it as per your business requirement score, label=show_image_with_predictions(data, threshold=0.

6) #Feel free to change it as per your business requirement score, label=show_image_with_predictions(data, threshold=0.

6)   References: – http://arxiv.






pdf https://developers.


com/python/guide/how-retinanet-works/ https://analyticsindiamag.

com/what-is-retinanet-ssd-focal-loss/ https://github.

com/fizyr/keras-retinanet https://www.


org/news/object-detection-in-colab-with-fizyr-retinanet-efed36ac4af3/ https://deeplearningcourses.

com/ https://blog.


com/en/post/2018-12-05/retinanet-explained-and-demystified/   Final Notes: – To conclude, we went through the complete journey to make a face mask detector with the implementation of RetinaNet.

We created a dataset, trained a model, and ran inference (here is my Github repo for the notebook and dataset).

Retina Net is a powerful model that uses Feature Pyramid Networks & ResNet as its backbone.

I was able to get decent results for face mask detector with very limited dataset & very few epochs (6 epochs with 500 steps each) only.

You can change the threshold value.

Note:- Make sure you train your model at least for 20 epochs to get good results.

Idea is to submit the approach to build a face mask detector using the RetinaNet model.

One can always tweak the model, data & approach as per business requirement.

In general, RetinaNet is a good choice to start an object detection project, in particular, if you need to quickly get good results.

If you enjoyed this article, leave a few claps, it will encourage me to explore further machine learning opportunities ???? About the Author Praveen Kumar Anwla I’ve been working as a Data Scientist with product-based and Big 4 Audit firms for almost 5 years now.

I have been working on various NLP, Machine learning & cutting edge deep learning frameworks to solve business problems.

Please feel free to check out my personal blog, where I cover topics from Machine learning – AI, Chatbots to Visualization tools ( Tableau, QlikView, etc.

) & various cloud platforms like Azure, IBM & AWS cloud.

  You can also read this article on our Mobile APP Related Articles (adsbygoogle = window.

adsbygoogle || []).


Leave a Reply