We will analyze the average sales per day of a week.
Code to visualize Sales across weeks in a monthSales across weeks in a monthSales are on the higher side for days on first and last weeks during a month.
One random guess could be the credit of salary at month-end and more people shopping.
2.
Feature Creation and Average SalesEvery date can be uniquely identified by its year, month, week and day.
The main reason to represent the date this way is to find monthly, year-over-year, day-wise, and weekly trends.
Reason to keep week number is to identify generally whether sales increase as the month progresses, diminishes or it doesn’t affect much at all.
We are referring it as the weekly trends.
Also, we will remove data points where the store remains closed.
On Sundays, the store remains closed.
After performing the feature creation and cleaning step, the data looks likeTable2: Data OverviewNext, we will perform One-hot encoding of year, month, day and week fields.
One hot encoding is a binary representation of categorical data.
As a result of this step, we will have the following fields.
'Sales', 'weekday_0', 'weekday_1', 'weekday_2','weekday_3', 'weekday_4', 'weekday_5', 'year_2013', 'year_2014','year_2015', 'month_01', 'month_02', 'month_03', 'month_04', 'month_05','month_06', 'month_07', 'month_08', 'month_09', 'month_10', 'month_11','month_12', 'weeknbr_1', 'weeknbr_2', 'weeknbr_3', 'weeknbr_4','weeknbr_5'Average SalesWe will start by calculating the Average Sales.
We will denote it by avgSales.
We have 2.
5 years of data, i.
e.
2013, 2014 and mid-2015.
Since 2015 full-year data is not available, it is likely to add noise if used in average sales calculation.
We will only use the 2013 and 2014 year sales data to calculate the avgSales.
Code to calculate average salesavgSales comes out to be 4825.
983.
Deriving the model’s equationThe main idea is to find monthly, year-over-year, day-wise, and weekly trends.
We wish to attach a value to it, called index.
So, we wish to derive monthly, year-over-year, day-wise, and weekly trend index.
We assume that these trends are solely responsible for sales in a day, over the average sales(avgSales) we have calculated above.
Model’s equation (1)We have one-hot encoded input features.
At a time, only one month will be high or set to 1, along with one day, one week number and one year.
The input will take care of it.
So, our input equation gets modified to :Model’s equation (2)Let’s see the form of the model’s equation for 17th May 2014.
In this case, the year will be 2014, the month will be May, week number will be 3rd and the day will be Saturday.
All other inputs will be 0.
Coefficients raised to power 0 will become 1.
Model’s equation form for 17th May 2014We will take log on both sides, to simplify the equation and convert it to a form where it’s easy to apply linear regression.
We will apply an intercept-less regression and take the antilog of model coefficients to get the monthly, year-over-year, day-wise, and weekly trend index.
Model’s equation to apply regression4.
Training the ModelHere the dependent variable is ln(Sales)-ln(avgSales) and we will take log using base 2.
We have sales data for a store for a period of about 2.
5 years from January 2013 to June 2015.
We will not use last month data and keep it for validating model’s performance.
Training the modelNext, we will take the antilog of the model’s coefficient to get the respective index.
Taking antilog of model’s coefficientsYearly IndexMonthly IndexDay-wise IndexWeekly Index5.
Learning from the Regression Model to decompose Time seriesYearly TrendAn important component of Time series is year over year trends.
The yearly index that our model learned has clearly captured a higher trend in the year 2013 than in 2014 and 2015.
The captured yearly index will assist us to decompose the yearly trend from the time series data.
Let’s visualize the results.
Yearly IndexYearly trendWe can clearly see a decreasing trend over the years.
There can be many reasons for the decreasing trend like the drop in store quality or opening of a competitor store etc.
One needs to do a deeper analysis to find that.
It will be an interesting find.
Monthly TrendAnother important component of Time series is the monthly trend.
The monthly index that our model learned has clearly captured a higher trend in the month of December.
It will assist us to decompose the monthly trend from the time series data.
Let’s visualize the results.
Monthly IndexVisualizing monthly trendMonthly trendDay-wise TrendIn the EDA section, we have analyzed the sales over days during a week.
We have seen sales on a higher side for Saturday and Monday.
The day-index learned by the model is able to capture these patterns.
Let’s visualize the results.
Visualizing day-wise trendDay-wise trendWeekly TrendDuring the EDA section, we have seen a small peak over the first and last weeks during a month.
The weekly index learned by the model is able to capture this pattern.
Let’s visualize the results.
Weekly IndexVisualizing weekly trendsWeekly trends6.
A simpler and intuitive model using the decomposed componentsWe will devise a new, simple and intuitive model using the above learnings and let’s call it Multiplicative model.
The final model equation will be :Model’s equation (1)The yearly_index, monthly_index, day_index, and weekNbr_index will be used from the calculated tables depending on the date.
Coefficients tableUsing the above model to make a prediction for 21 Sept 2015.
It falls on September month, 3rd week and year 2015Sales = avgSales * Sep_index * 3rdWeek_index * 2015_index = 4825.
98 * 0.
880 * 0.
98 * 0.
9425 = 3922.
54Prediction for Dec 31st, 2015?.It falls on December month, 5th week and year 2015Sales = avgSales * Dec_coefficient * 5thWeek_coefficient * 2015_index= 4825.
98 * 1.
244 * 1.
069 * 0.
9425 = 6048.
747.
Model’s quality analysis and using it to get other useful informationIn parallel, a linear regression model without any constraints on intercept or logarithms on features was also trained.
Let’s call it Model-1 and our multiplicative model, Model-2.
We tested both the models on 3-months on unseen data for 50+ stores.
Mostly both of the models performed equally good in terms of Root Mean Square Error with similar RMSE values.
There were some cases were Model-2 was a clear winner with low RMSE compared to Model1.
One possible reason could be the feeding of domain-knowledge, avgSales, not allowing it to overfit and performs well on unseen data.
Code to compare the performance of two modelsBoth model performing similarly for most of the casesMultiplicative model (Model-2) is a clear winner for Stores 39,85 and 88Multiplicative Model is a winnerGetting Other useful information from the Multiplicative modelSome crucial information and fun-facts can be derived easily from the multiplicative model.
Like, if the business asks us what can we say or how can we compare the sales of May 2014 with November 2014.
We use the value of the monthly, yearly, day and weekly indexes learned to answer such questions.
May 2014 = may_index * 2014_index = 1.
02 * 0.
9879 = 1.
007Nov 2014 = nov_index* 2014_index = 1.
051 * 0.
9879 = 1.
038What extra percentage of sales is for Nov 2014 compared to May 2014(1.
038–1.
007)/1.
007 = 0.
0307So, November 2014 will be 3.
07% more seasonal than May 2014.
We can do a similar comparison for months, weeks, days across years and get a clearer picture from the decomposed forecasting model.
8.
ConclusionThe main idea of this blog post was to tweak traditional models to decompose time series and create a simpler and intuitive model.
Hope the purpose was served.
Let me know what you think and how I can improve this methodology further.
.