Skip to main content

Leadership

How can leadership interact with experiment effectively?#

  • Nimbus Console is the best place for exploring what is running, completed, and in draft. Currently there is basic filtering - which will be expanded this year to enable easier experiment discovery.
  • Each experiment should have an Experiment Brief (aka Experiment overview) as a link from Nimbus Console - which goes into the background and learning goals.
  • After an experiment is completed, there is a Results link.
  • There should be information added to Nimbus Console in the "Takeaways" section for a high level tl;dr interpretations and actions taken for each experiemnt. If that is blank - ask the Product Manager for the takeaways and learnings and to fill that section in for later discoverability.

Where do I find Experimentation Program metrics?#

The Experiment Tracking spreadsheet is the current source of truth on experimentation program metrics.

  • This is updated at least monthly before Product Days
  • The summary view has the SPI metrics that are being reported out regularly.
  • Tabs for "by feature" and "by area" show some of the information in graph format around what types of experiments are being run and where.

Until we improve discovery in the Console - the 2022 data sheet is a good hack to find some types of experiments.

  • Please do not edit it.
  • One example discovery case - you can use the "CDOU Stat Sig" and "Leading Indicator Stat Sig" to easily find experiments that saw a stat sig change. There are comments at the top of each column with what it is capturing.
  • Ask Shell, Jim, Devyani, or Daniel Berry for help interpreting anything in that spreadsheet.