Skip to main content

For Leadership

This guide provides leadership with an overview of how to monitor experiments, access results, and track the experimentation program's progress across teams.

How Can Leadership Interact with Experiments?​

  • Nimbus Console is the best place for exploring what is running, completed, and in draft. Currently there is basic filtering - which will be expanded this year to enable easier experiment discovery.

  • Each experiment should have an Experiment Brief following this template, as a link from Nimbus Console - which goes into the background, learning goals, and experiment design.

  • After an experiment is completed, there is a Results link in the right hand vertical gray bar. image

  • For completed experiments, there should be information added to Nimbus Console in the "Takeaways" section for a high level tl;dr interpretations and actions taken for each experiment. If that is blank - ask the Product Manager for the takeaways and learnings and to fill that section. image

Where Do I Find Experimentation Program Metrics?​

The Experiment Tracking spreadsheet is the current source of truth on experimentation program metrics.

  • This is updated monthly before Product Days
  • The summary view has the SPI metrics that are being reported out regularly.
  • Tabs for "by feature" and "by area" information in graph format for experiments based on feature area and area (desktop, mobile, platform).

Until we improve discovery in the Console - the 2022 data sheet tab is a possible hack to discover experiments. There are comments at the top cell of each column explaining what is being captured.

  • Please do not edit it.
  • Example: you can find experiments that had a "CDOU Stat Sig" and "Leading Indicator Stat Sig" change. to easily find experiments that saw a stat sig change.
  • Ask Shell, Jim Thomas, Devyani, or Daniel Berry for help interpreting anything in that spreadsheet.