Skip to main content

Experiment Reviewers

With Nimbus experiments, instead of a centralized committee responsible for launching and running experiments we want you, the experiment owner, to drive the process from ideation to completion.

If you are interested in learning more about your responsibilities, you can find our complete Access Control Policy here.

Onboarding for New Authors/Owners (L2)​

In order to use the Nimbus platform to create and launch experiments, you will need to go through the Experiment Owner onboarding process. Please ask in #ask-experimenter if you have any questions and for a shadow on the first experiments(s) you create.

Onboarding for New Reviewers (L3)​

Assuming you have been vouched for by a Nimbus administrator + a Product Owner (product manager or engineering manager if the team doesn’t have a dedicated PM), you should go through the following steps to get review access on Nimbus:

Basic access​

  • Read through rules and responsibilities for L3 users. Please note that you will have access to making changes in production for all experiments, and you should decline if you don't feel sufficiently qualified to review an experiment
  • Watch the half hour reviewer training video. The training slides are here for reference and links.
  • Follow the steps setup VPN
  • File a bug using this template to be added to either or all of the following collections: nimbus-experiments-desktop and/or nimbus-experiments-mobile and/or nimbus-web-experiments for your LDAP to be added on staging and production.
  • Note: on your first review(s) - please ask in #ask-experimenter for a shadow. It's very easy to partner with you sharing a screen to make sure you don't have any questions or uncertainties.

Testing Review Workflow on Staging​

  • Connect to the VPN
  • Go to Nimbus Staging (not Production!). Ask someone to create a dummy experiment and request review.
  • Click "Approve" on the dummy experiment, and then "Open Remote Settings": imageimage
  • Sign in. You should see a review request like this, if you don't refresh the page image
  • CHECK TO MAKE SURE THE SLUG OF THE DIFF MATCHES THE CHANGE YOU ARE APPROVING This is important!!
  • Press approve if everything looks good. If anything looks wrong, Reject and alert nimbus core team in #ask-experiments
  • Congrats, you have tested the workflow. You are now ready to review real experiments on production!

Reviews​

All changes to experiments and rollouts that impact production must be approved by a single L3 Nimbus reviewer, which you can request via the Nimbus console interface. You can find a list of recommended reviewers here.