Skip to main content

What's New in Nimbus - Q1 2024

Experimenter​

🛎️ Subscribe to notifications

Non-owners can now subscribe to notifications. Previously, these notifications were only being sent to experiment owners. This feature is available for all users and all experiments, and can be found in the Overview section on the Summary page of an experiment. (erichards) EXP-4229

🙌 New "My Experiments" tab when you go to https://experimenter.services.mozilla.com/

Find all experiments you own and all that you have subscribed to with one click! See the things that matter most to you quickly.

🙌 Recipe branches are now available from the include/exclude dropdowns.

Not only can include and exclude running and previously run expeirments - but now you can specify specific branches.

  • An example use is to rollout a winning message to everyone EXCEPT people that were in the treatments branch of the experiment (so people don't get related message variants again).

🛠️ Audience overlap

We now have warnings on Experimenter for three known cases of audience overlap. These warnings will prevent detectable cases of audience overlap, which generally result in under enrollment and can invalidate an experiment. (erichards) EXP-4180

🤖🍏💻 Helpful JSON Errors

As you are writing your JSON (the code that configures your feature in the Branches page), you may see Feature Manifest errors occured during validation followed by details. Experimenter is now comparing your feature manifest with the branch JSON you entered. It compares for all versions your experiments is targeting, as defined in your experiments Audience page with minimum version and maximum versions. This makes sure that the JSON you just typed will work for your feature on all the targeted firefox version. If there is an error message, it will include the property that didn't match the feature manifest, possible valid values, the line where the error is, and the version where that error happened. If you have an issue - to resolve first check the minumum versions your experiment targets (on the Audience page) and make sure it is as intended. If that is correct, next look at the line to see what aspect of the JSON is having an issue and how to resolve.

Mobile​

Coming soon...

Desktop​

Coming soon...

Jetstream & Results Analysis​

💻 Pairwise branch comparisons now enable you to change the "reference branch". By default, experiments compare the performance of treatments againt the control branch. But what if multiple treatments all performed better than control? Now in the Results Config section - you can select one of the treatment branches to be the "reference branch". In doing so you can see if there are stat sig differences between the branches and what those are.

💻 Analysis now defaults to EXPOSURE based instead of ENROLLMENTS This does not change the available results (i.e., results for Enrollments are still available), and you can still select any available Analysis Basis via the "Results Config" radio buttons on the left of your results page. Click here for more information about Exposure events, including how to set up experiments that have exposure events analysis results.