Types of Risk
We've learned a few ways to de-risk experiments and rollouts before they launch. Below is not an exhausive list for every situation - but a list of the some common situations to help you avoid known pitfalls.
You are the expert in your area and the final responsibility/decision regarding if your change is sufficiently prepared to go live to hundreds of thousands or millions of people.
Is this study partner related or shipping any type of encryption?
If a partner is involved in any way, it raises risk. If the answer to this question is "yes" - a Legal Review and possibly a VP Sign-off are required.
A partner could also be affected indirectly, for example if functionality or presentation is altered to display less of a partners content or search capabilities - this may violate an agreement with them.
We could engage a partner as a hosting or processing service or present a partner add-on. In this case Legal needs to be engaged to make sure we aren't misaligned with privacy policies, data handling, any other negotiations around revenue, licensing, contractual obligations, trademark usage, etc.
If your product involves encryption of any type - like a VPN - it is important to engage legal to make sure we comply with various countries regulations about code that includes encryption.
Please write legal-product@mozilla.com (if you are not already engaged with Legal) and explain what you want to do with the partner, to how many people (% of population and channel), and the potential outcomes. You can also file a legal bug
Does this change have high risk to the brand?
If the experiment fails, will folks notice? Would the press assail us? Does this experiment make you uneasy with regard to the Mozilla manifesto? Could it alienate our allies or community? Could it negatively impact perceptions of Mozilla or Firefox?
If the answer to this question is "yes" - talk to the Comms team to get thier thoughts. Depending on the risk, you may want to let your VP know, as they would likely handle any reaction.
Does this study have possible negative impact on revenue?
This could be anything directly changing the search bar, ads, pocket enagement, tiles, new tab page engagement OR anything that indirectly impacts these (ex: pushing any of these below the first viewable space).
If the answer to this question is "yes" - VP Sign-off is needed. Explain the risk and why you think that is worth taking. Try to estimate the expected impact (how many users impacted and how much of a change)