What's New in Nimbus - Q3 2023

Mobile

🤖🍏☁️ Pre-built nimbus-cli

We can now use pre-built nimbus-cli for FML transformations. This change installs the nimbus-cli as part of deployment and uses this to fetch and transform FML files from the Firefox Mobile, Focus, and Monitor apps. This means faster deploy times and we can remove the .experimenter.yaml files from the apps' repos and build scripts. (jhugman)

🍏 Support for testing on iOS devices to nimbus-cli

Previously, we had been restricted to tested on iOS simulators only, which meant a tester needed to have Xcode installed on a Mac, and certain OS integrations were not testable. This new capability opens up testing experiments and features to many more people, and on AppStore released apps and on real devices. The QA team have said that nimbus-cli has made "testing [on mobile] a cakewalk", and on device testing as "mind-blowing". (jhugman)

Desktop

💻 Rich JSON editing

The Branches page now has a new editor for entering branch value! The new editor supports auto-completion of most features on Desktop and Mobile, as well as syntax hilighting and JSON linting. It finds errors in your values live as you type them and highlight them for easy correction! (brennie)

Jetstream & Results Analysis

📈 Automated population sizing

Population sizing tool results are now available (for certain applications/locales/countries) in Experimenter during sizing configuration. (mwilliams)

📈 Support for downsampling

Previously, if a job failed due to memory issues, a Data Scientist would have to manually override the enrollments query to do downsampling. This is complex, time-consuming, and error-prone. Jetstream now has the sample_size (an integer 1-100 representing the sample rate) as a config option.

Cirrus

☁️ Glean integration for Cirrus

In order to get enrollment telemetry for Cirrus, we hooked up the Glean Python SDK. (ykhurana)

☁️ Record enrollment metrics

Add the ability to record enrollment metrics for Cirrus clients. (chumphreys)

☁️ Demo app

A simple test app with the node backend and react frontend that integrates cirrus that we can use for local testing and integration. (ykhurana)

☁️ Configurations for integrating Monitor

Adding new application options, configurations, and collections to allow Monitor to be hooked up with Cirrus. (ykhurana)

Experimenter

🙌 Inclusion and exclusion targeting

Experiments now have an extra dimension of targeting: participation in other experiments and rollouts! This allows experiment owners to easily launch follow-up studies to the same cohort of users. Additionally, exclusion can be used to prevent enrolling in one experiment if they're already in a conflicting experiment on a different feature. This capability enables experiment owners to launch more experiments -- and especially mobile messaging campaigns -- without worrying that users will have conflicting experiences. (brennie)

🙌 New fields in Takeaways section

Three new fields have been added to the Takeaways section for completed experiments: a checkbox for "QBR notable learnings", a checkbox for "Promising metric gains" (a.k.a. "Statistically Significant DaU Gain"), and a free-form text area for "Gain amount" that allows the user to dig deeper into what metrics changed and the significance. In addition to that, the two checkbox fields have been added as a filter on the homepage to allow easy access to impactful experiments. (erichards)

🙌 Outreachy History Tracker Project

This work was completed by our Outreachy intern. This introduces a new page which show the changelogs of a particualr experiment within the existing project. The link to this page is now available underneath "Summary" on the left sidebar of an experiment's detail view. (ykhurana, avi-88)