Training a Neural Network to Detect Gestures with OpenCV in Python

My old model didn’t generalize well, and my ultimate goal was to build a model that could recognize my gestures in real time — so I decided to build my own dataset!I chose to focus on 5 gestures:I strategically chose 4 gestures that were also included in the Kaggle data set, so I could cross-validate my model against those images later..I also added the peace sign, although that gesture did not have an analogue in the Kaggle data set.From here, I built the dataset by setting up my webcam, and creating a click binding in OpenCV to capture and save images with unique filenames..I tried to vary the position and size of the gestures in the frame, so that my model would be more robust..In no time, I had built a dataset with 550 silhouette images each..Yes, you read that right — I captured over 2700 images.Training the new modelI then built a convolutional neural network using Keras & TensorFlow..I started with the excellent VGG-16 pre-trained model, and added 4 dense layers along with a dropout layer on top.I then took the unusual step of choosing to cross-validate my model on the original Kaggle dataset I had tried earlier..This was key — if my new model couldn’t generalize to images of other people’s hands that it hadn’t trained on before, than it was no better than my original model.In order to do this, I applied the same transformations to each Kaggle image that I had applied to my training data — background subtraction and binary thresholding..This gave them a similar “look” that my model was familiar with.L, Okay, and Palm gestures from Kaggle data set after transformation.ResultsThe model’s performance exceeded my expectations..It classified nearly every gesture in the test set correctly, ending up with a 98% F1 score, as well as 98% precision and accuracy scores..This was great news!As any well-seasoned researcher knows, though, a model that performs well in the lab but not in real life isn’t worth much..Having experienced the same failure with my initial model, I was cautiously optimistic that this model would perform well on gestures in real time.Smart Home IntegrationBefore testing my model, I wanted to add another twist..I’ve always been a bit of a smart home enthusiast, and my vision had always been to control my Sonos and Philips Hue lights using just my gestures.. More details

Leave a Reply