The ensemble achieved a validation accuracy of 0.821 which is a significant improvement from the baseline paper’s accuracy of 0.672.Background (The Problem)I was tasked to apply data fusion on UTD-MHAD to build a model to classify 27 different human actions, and like all master procrastinators, I left it to the last week to start doing it..The dataset contains 4 data modalities, namely:RGB videos (spatio-temporal)Depth videos (spatio-temporal)Skeleton joint positions (spatio-temporal)inertial sensor signals (temporal)All 4 modalities were time synchronized and stored in .avi and .mat format respectively.Task: Beat the baseline accuracy of 0.672The dataset came with a paper (C.Chen, 2015) which uses a Collaborative Representation Classifier (CRC) that had a validation accuracy of 0.672..This was calculated on a train-validation split where subjects 1, 3, 5, 7 were used for training, and subjects 2, 4, 6, 8 for validation and it was also the baseline accuracy I have to beat!All fired up, I immediately went on to the web to start looking for past codes and tutorials..Below are the plots of the Depth, Skeleton, and Inertial data of a subject performing a tennis swing..Since the data is a sequence of 6 channels (3-axis for accelerometer + 3-axis for gyroscope), the very first model we would be building is a Simple LSTM (S. Hochreiter et al., 1997) with a LSTM cell of 512 hidden units.Minimalist Keras Code to implement the Simple LSTM modelNetwork Diagram of Simple LSTM modelBack to overviewStep 3..As this a classification task with a well-balanced class distribution, the accuracy would suffice as the sole performance metric, without the need of calculating precision, recall or F1-score.To see if our model is over-fitting, we can also get Train-Validation Accuracy-Loss plot..A 27-Class confusion matrix can also be plotted to see which are the actions that are often misclassified as another.From the Accuracy-Loss plots, we can see that our model is over-fitting at very early epochs, with our validation accuracy plateauing after the 4th epoch..At epoch 15, we got a model with a validation accuracy of ~ 0.238, which is pretty far off from the baseline of 0.672 we have to beat.This suggests that we would have to either change strategy or apply more regularization techniques such as Dropout layers.Confusion matrix of the Simple LSTM model on Inertial dataOh gosh!.The other 25 actions had very poor performance.Before we go off stressing ourselves out, let us take a step back and look what we have done thus far.Data Science Pipeline (adapted from Practical Data Science Cook Book)We have just finished one full iteration of going from Step 1 -> 4 in the above flow chart and we got a first validation accuracy of 0.238..Ask any practitioner and they would all agree that Data Science is a highly iterative journey.With this foundation formed, we can now get creative and try different stuff to improve our model..This means that Keras would stop the training if it finds that the model’s validation accuracy is not increasing after 5 epochs.With these 3 callbacks set in place, we can safely leave our model training while we head out for lunch.Useful Keras CallbacksGoogle ColaboratoryAs we all know, training Deep Learning models is a very GPU intensive process..As the input data is a 1D Signal, this model uses a series of 1D Convolutional and 1D Maxpooling layers to extract higher dimensional, latent features before feeding them into 2 LSTM units which capture the temporal information..The output of the LSTM units is then flattened out and we attached a Dropout layer with a dropout rate of 0.5 before adding a Dense layer with a softmax activation to classify all 27 actions.This has got my validation accuracy to 0.700 just on the Inertial data which is the first time we beat the CRC model baseline of 0.. More details