It’s possible the business decides to go with a design or a sample size that wasn’t your recommendation..It’s ok to be a stickler, this is your foundation.Test DesignThe test design needs to be solid, so you’ll want to have an understanding of exactly what change is being made between test and control..If you’re approached by a stakeholder with a design that won’t allow you to accurately measure criteria, you’ll want to coach them on how they could design the test more effectively to read out on the results..I cover test design a bit in my article here.Sample SizeYou need to understand the sample size ahead of launch, and your expected effect size..If you run with a small sample and need an unreasonable effect size for it to be significant, it’s most likely not worth running..Customers will see a different price than what is advertised, and this has negative implications all around.It is so important in a large analytics organization to be collaborating across teams and have an understanding of the tests in flight and how they could impact your test.Population criteriaObviously you want to target the correct people.But often I’ve seen criteria so specific that the results of the test need to be caveated with “These results are not representative of our customer base, this effect is for people who [[lists criteria here]].” If your test targeted super performers, you know that it doesn’t apply to everyone in the base, but you want to make sure it is spelled out or doesn’t get miscommunicated to a more broad audience.Test durationThis is often directly related to sample size..(see Sarah’s article)You’ll want to estimate how long you’ll need to run the test to achieve the required sample size..If it’s going to take 6 months of running to get the required sample size, you probably want to rethink your population criteria or what you’re testing..And try not to make a mountain out of a molehill unless you’re testing something that is a dramatic change and has large implications for the business.Decisions!Getting agreement ahead of time on what decisions will be made based on the results of the test is imperative.Have you ever been in a situation where the business tests something, it’s not significant, and then they roll it out anyways?. More details