What's New in Nimbus - Q2 2023
This is a non-exhaustive list of the Nimbus team's accomplishments in Q2, including some deep dives into a few of the more exciting features that have landed.
Mobile
🤖 Android Messaging Surface Configurations
Adds native notifications as a messaging surface that can be targeted by the mobile-messaging system for the Retention/Growth Push Campaigns, and includes the pre-permission prompt shown to users before asking for Notification permission. This covers work in Fenix as well as in the Nimbus SDK. (jhugman)
The startup sequence currently lives across the app and Application Services, requiring coordination with multiple repos. This makes it very hard to test and difficult to reason about state close to startup performance. This work moves Nimbus Startup out of the iOS apps, making it possible to move it to the beginning of the app startup, and makes the initialization API simpler and more stable. (jhugman)
🍏🤖 Improve Mobile experiment QA
As we improve our feature set and support for mobile clients, testing the experiments created using these features will grow in complexity and length. Ensuring that these experiments can be tested in a reasonable time frame is key to allowing us to be able to get more experiments out. These improvements to the mobile clients will allow Softvision to test more experiments quicker, as well as lend a hand to future automation. (b4hand)
The Nimbus CLI (see below) was created to help with first-run testing. First-run is a frequently used, high value surface that requires the experiment configuration to ship with the mobile build, so it is crucial to have support for these experiments to test before shipping. (jhugman)
🍏🤖 Deprecate Feature Keys in v6 API
To prepare for the switch from single feature to multi-feature, the V6 serializer needed to switch over from using the
featurekey in the branch schema to using thefeatureskey. Desktop had already switched tofeatures, but mobile was still using singularfeature. This work switches the V6 serializer to always use thefeatureskey. (brennie)
Adds a Release Date field for mobile first run experiments to accurately calculate enrollment and durations to save multiple manual calculations. (erichards)
Ongoing support for mobile engineers using Nimbus. (jhugman)
🍏🤖 Core Active Behavioral Targeting
Applications are now able to share information with Nimbus necessary for the categorization of users as “core-active” and experiments/messaging are able to target based on this categorization. (chumphreys)
Desktop
Enables localized experiments on the Nimbus Desktop client. (brennie)
Jetstream & Results Analysis
Experimenter links to Results page are now disabled for experiments that have no results, and the Results page redirects to the Summary and shows appropriate messages to the user when there are no results. (mikew)
📈 Shared Schema for Jetstream Results
Define and create a new shared schema for consumption by Experimenter and Jetstream, ingestion of analysis results, and export of analysis results from Jetstream. (mikew)
Cirrus
Implements the core behaviour of the Cirrus API such that it receives a request from the client and can respond with the features and feature values. This requires fetching active experiments from Remote Setting, using the Nimbus SDK to compute enrollment, and using the FML to evaluate feature configurations. (ykhurana)
Creates FML client and Python bindings for Cirrus. (chumphreys)
Given the existing Nimbus SDK library, this work creates an extensible, shared Nimbus library in which the majority of the functions are static as opposed to the stateful functions within the existing library. (chumphreys)
Document/create ADRs for requirements, implementations, and system design diagrams. (ykhurana)
Experimenter
Tasks that were completed by Outreachy intern applicants to improve the Experimenter UI. (ykhurana)
🙌 Outreachy History Tracker Project [in progress]
This work is being completed by our Outreachy intern. This will introduce a new page which show the changelogs of a particualr experiment within the existing project. The primary goal is to introduce a new page that leverages Django, Tailwind CSS, and Alpine.js (new pattern!) to enhance the functionality and user experience of the application. (ykhurana, avi-88)
Enables users to to run experiments that target multiple features at once. We modified experiment creation pages to be able to select multiple features via a multiselect at the top of the page (before editing branches). Each feature value is validated against its respective schema as before. Modified client and API to support multiple features in each branch and ensure existing experiments that are already published continue to use the single feature branch type without impact. (brennie, jlockhart)
🔧 Additional Rollouts Improvements
Non-user facing work for rollouts, including improved backend logic to handle the Dirty state, integration tests, updated state diagrams and sequence diagrams. (erichards)
Deeper Dives
The nimbus-cli
🔥 The nimbus-cli was conceived at the team's Q2 workweek. It is designed to help QA engineers test mobile first run experiments, but has become a tool useful for interacting, inspecting and testing all experiments running on the supported mobile apps, and any feature that those apps configure via Nimbus.
Installation is:
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/mozilla/application-services/main/install-nimbus-cli.sh | bash
Further details are at https://experimenter.info/nimbus-cli.
Computing the date for expected results
🔧 We are now computing the date that the Jetstream results will be available for a given experiment. This new value is based on enrollment end date (or proposed enrollment end date if enrollment is not yet ended).
Release date for First Run experiments
Our approach for calculating dates and duration for experiments did not previously take the release trains into account. For example, a first run experiment targeting Firefox desktop v123 might be launched (status=Live) on 1/2/2023. However, v123 might not be released until 1/15/2023. The start_date for this experiment will therefore be 1/2/2023, but users will not start seeing it until 1/15 when the version is released.
We added another date, "proposed release date" or "first run release date", for first-run mobile experiments which will represent the release date of the browser version that is being targeted by the user.
This date gives us the ability to:
- 📈 Correctly compute enrollment and proposed end dates based on when users will actually start getting the release version
- ⏰ Send reminder emails to experiment owners at the right time so that they can end enrollment/end experiments
This new field shows up on the Audience page of the experiment, and is also viewable on the Summary page and on the timeline:
"First Run Release Date" field on the Audience page
"Release Date" field on the Summary page timeline
"Release Date" field in the Audience section of the Summary page
Desktop Rollout re-enrollment and resizing
- 📈 You can now re-enroll in rollouts that have increased in size (Bug 1833248)
- 💪 When force-enrolling you can add
&apply_targetingto the URL to check the targeting instead of skipping it (Bug 1736587)