And what an exciting season it has been.
For the data scientist within you lets use this opportunity to do some analysis on basketball clips.
With the use of deep learning and opencv we can extract interesting insights from video clips.
See example gif below of the game b/w UCF and Duke where we can identify all the players + referees, label players into teams based on their jersey colour.
Later in the blog I show how we can also identify which player is attempting to shoot the basket .
And all of this can be done real time.
Detecting players and teamsYou can find the code on my Github repoSo let’s start.
Detecting PlayersI have used a pretrained detection model like the Faster RCNN to detect players.
It is easy to download a Faster RCNN trained on the COCO data set from the Tensorflow Object Detection API and test it.
The API takes as input each frame of image and detects among 80 different classes.
If you are new to Tensorflow Object Detection and want to learn more, please checkout this blog.
The model does quite well in detecting persons but there are many detections in this video due to the large number of people in the crowds.
See sample detection below.
I suppressed detections that were too big to more cleanly segment out players.
You can also play with the score threshold in the API to filter out low confidence detections.
Checkout the code on Github for tips on how to suppress boxes with low scores and multiple false detections.
Detection output from Pretrained Tensorflow modelDetecting TeamsNow comes the interesting part.
How do we detect which players are UCF vs Duke?.We can use OpenCV to do that.
If you are new to OpenCV please see the tutorial below:OpenCV TutorialOpenCV allows us to identify masks of specific colours and we can use that to identify white and black players.
The main steps are:Convert the image from BGR to HSV colour space.
In HSV space specify colour ranges for white and black.
This takes a bit of experimentation and you can visualize the impact of different thresholds in the notebook.
Use OpenCV to mask(colour) pixels that are in the threshold range.
OpenCV Bitwise_and to colour black any pixels not in the maskSee output below for white colour.
They are masked as “pink” with everything else in the background in blackDetecting white colour pixelsTo identify team for each individual player we extracted bounding box from tensorflow object detection and count the percent of pixels in that bounding box that are non black to decide the team for that player.
Overall code works quite well.
However this is a hard coded logic for identifying black and white jersey players.
It can be made more general by using clustering to find similar playersDetecting pose and who is shootingOpenPose is a real-time multi person pose detection library.
It detects persons in an image and outputs keypoints for the main joints for every person — can total up to 25 keypoints per person.
The code is open sourced.
You have to install as suggested in README here.
Once installed you can run images through it and get keypoints for all the players in a scene as shown below.
Open pose outputSo how do we identify players attempting to shoot a basket?We can look for players with their wrist keypoints above their head.
Implying hands raised.
This could indicate ready to shoot as in the scene above or could be defensive.
Further the coordinates of the ball along with those of the wrist keypoint can be used to identify which player with raised hands has ball close to them.
ConclusionDeep learning has made is possible to do really cool analysis by chaining different ideas.
There is a lot of open source code and pretrained models that you can use on your data to get started.
Above is just the starting point.
Other cool things that can be done include:Using OCR to read the game scores to allow your system to understand which team is winningTracking the ball to predict when a shot can score pointsTracking players to get stats for each individual playersDetecting events like a slam dunk, 3 point basketball etc.
Hope you pull the code and try it yourself.
I have my own deep learning consultancy and love to work on interesting problems.
I have helped many startups deploy innovative AI based solutions.
Check us out at — http://deeplearninganalytics.
org/.
You can also see my other writings at: https://medium.
com/@priya.
dwivediIf you have a project that we can collaborate on, then please contact me through my website or at info@deeplearninganalytics.
orgReferencesTensorflow Object Detection APIGood tutorial on Detecting colours using OpenCV.. More details