Well, in simplest terms, a Geo Experiment is like an A/B test but uses geographies to define your control and treatment group rather than individuals or web cookies.
In Geo Experiments, geographic regions are divided into treatment and control group.
The regions in the treatment group are exposed to a marketing intervention while regions in the control group remain status quo.
The marketing intervention happens for a duration of time and the response metric is observed.
The idea is to detect if there is an incremental lift in the response metric in the treatment regions.
Let’s use the Facebook question above as an example and assume we are running the ad campaign in the United States.
First, we randomize a set of cities in the United States putting them into treatment and control group.
Just like A/B test, randomization is a critical part of the experimental design.
Then in Facebook Business Manager, we set the campaign to only run in the cities in the treatment group.
We run the campaign for a duration of time and we measure the incremental customers acquired (agnostic of an attribution model) by comparing the difference between the control vs.
treatment group.
For example, we may see something like:This is a graph produced by the GeoexperimentsResearch R package developed at Google which implements the Geo Experiment methodology and provides an easy way to perform the analysis.
I’ll be demonstrating how to use this R package in Part 3!.But at a high level, the y-axis can be customer conversion for our example purposes, the x-axis is divided into three distinct periods:Pre-test period (Feb 05 — Mar 31): before the campaign startedTest period (Apr 01 — Apr 28): when the campaign is runningCooldown period (Apr 29 — May 05): campaign stops running but there might be a lingering effect of the ad that trickle inThe top graph illustrates the observed response metric overtime vs the predicted counterfactual.
We’ll talk more about this in detail in Part 2 but essentially the counterfactual is what we would have observed if the marketing campaign did not run.
The middle graph illustrates the difference between the observed and counterfactual by day which estimates the lift by day.
Finally, the bottom graph is the total lift we’ve observed in the experiment.
My explanation above is an oversimplification of the methodology.
To truly understand what’s going on, you need to go inside the hood and look at the engine.
The methodology and math behind Geo Experiment are different from a traditional A/B test but the overall idea is similar.
As mentioned, I will be leaving the detailed explanation of the mathematics to Part 2 of this series.
The methodology that I’ll be going through in Part 2 is research from Google, so if you are interested, take a look first.
A one-liner explanation: a regression model is used to learn the exchangeability factor between the control regions and the treatment regions and then used to predict the counterfactual during the intervention period.
Maybe this sentence doesn’t make any sense to you at all.
Don’t worry, I will be breaking this down in simple terms!.Stay tuned!I want to take a short moment to highlight that Geo Experiment is another use case of regression in marketing science.
If you haven’t read my article “One Thing Marketing Analysts Should Have In Their Analytics Toolkit”, take a look to see why regression is a tool every Marketing Analyst should have in their toolkit.
By no means is Geo Experiment designed to replace traditional A/B test.
They serve very different purposes and answers different questions but both are equally important.
Just as they say, “there is no one size fits all”, you shouldn’t just have butter with your bread.
Go ahead and add some variety, olive oil is great too!Challenges and LimitationsBy now, I might have sold you on how great this new tool is BUT, like any other tools, Geo Experiment comes with its own set of limitations and challenges:Overhead Cost: There may be a lot more work involved to set up a Geo Experiments depending on how you’ve set up your campaigns.
You may need to restructure your existing digital marketing accounts in a way that allows you to target campaigns at a city/region level to create your control and treatment group.
Platform: Not every marketing platform you advertise on allows you to target campaigns at a city/region level.
For example, you may be able to target Facebook paid ads at a city/region level but you don’t have that capability when you are running ads in podcasts.
Budget: Depending on your business, campaign, and many other factors, the budget you need for a campaign will greatly vary.
For companies that are just starting out with digital marketing, I wouldn’t be looking at Geo Experiments just yet.
If your business is mature and you’re looking to optimize, then I think its a good fit.
Controlling for variables: Can you control for marketing activities happening across different cities?.E.
g are there other marketing initiatives from your organization happening in specific cities that might overlap with your Geo Experiment?And finally, alway’s remember to choose the right tool for the right job.
There is no one size fits all!Originally published at artofmarketingscience.
github.
io on March 25, 2019.
.