Skip to main content

Leadership

How can leadership interact with experiments?​

  • Nimbus Console is the best place for exploring what is running, completed, and in draft. Currently there is basic filtering - which will be expanded this year to enable easier experiment discovery.

  • Each experiment should have an Experiment Brief following this template, as a link from Nimbus Console - which goes into the background, learning goals, and experiment design.

  • After an experiment is completed, there is a Results link in the right hand vertical gray bar. image

  • For completed experiments, there should be information added to Nimbus Console in the "Takeaways" section for a high level tl;dr interpretations and actions taken for each experiment. If that is blank - ask the Product Manager for the takeaways and learnings and to fill that section. image

Where do I find Experimentation Program metrics?​

The Experiment Tracking spreadsheet is the current source of truth on experimentation program metrics.

  • This is updated monthly before Product Days
  • The summary view has the SPI metrics that are being reported out regularly.
  • Tabs for "by feature" and "by area" information in graph format for experiments based on feature area and area (desktop, mobile, platform).

Until we improve discovery in the Console - the 2022 data sheet tab is a possible hack to discover experiments. There are comments at the top cell of each column explaining what is being captured.

  • Please do not edit it.
  • Example: you can find experiments that had a "CDOU Stat Sig" and "Leading Indicator Stat Sig" change. to easily find experiments that saw a stat sig change.
  • Ask Shell, Jim Thomas, Devyani, or Daniel Berry for help interpreting anything in that spreadsheet.