How can leadership interact with experiments?
Nimbus Console is the best place for exploring what is running, completed, and in draft. Currently there is basic filtering - which will be expanded this year to enable easier experiment discovery.
Each experiment should have an Experiment Brief following this template, as a link from Nimbus Console - which goes into the background, learning goals, and experiment design.
After an experiment is completed, there is a Results link in the right hand vertical gray bar.
For completed experiments, there should be information added to Nimbus Console in the "Takeaways" section for a high level tl;dr interpretations and actions taken for each experiment. If that is blank - ask the Product Manager for the takeaways and learnings and to fill that section.
Where do I find Experimentation Program metrics?
The Experiment Tracking spreadsheet is the current source of truth on experimentation program metrics.
- This is updated monthly before Product Days
- The summary view has the SPI metrics that are being reported out regularly.
- Tabs for "by feature" and "by area" information in graph format for experiments based on feature area and area (desktop, mobile, platform).
Until we improve discovery in the Console - the 2022 data sheet tab is a possible hack to discover experiments. There are comments at the top cell of each column explaining what is being captured.
- Please do not edit it.
- Example: you can find experiments that had a "CDOU Stat Sig" and "Leading Indicator Stat Sig" change. to easily find experiments that saw a stat sig change.
- Ask Shell, Jim Thomas, Devyani, or Daniel Berry for help interpreting anything in that spreadsheet.