Skip to main content

Feature Variables and Me

About this document#

This document is to illustrate the concepts of the Feature Variables feature of Nimbus, internally known as the Feature API.

The API used by software engineers is relatively small, so this document is not just for them. This document is also for designers, product managers and engineering managers who design, work on, and are responsible for features in the mobile apps we build. Successful experimentation requires multiple parts of the team to share an understanding of these concepts.

πŸ‘‹ Information

Neither Nimbus, nor all of the Feature Variables work are finished yet, but they are certainly still useful. This document will talk about features that aren't yet implemented, but will serve to illustrate the concepts. A sidebar like this will tell you when a feature isn't ready.

β›…οΈπŸ”¬πŸ”­ Nomenclature

Much of the literature around the methodology of experiments Nimbus implements has its roots in medical testing. The feature variables API does not require understanding of double blind experiments or data-science, but this document will occasionally use words like "treatment" or "exposure".

Introduction#

The "Feature" in the "Feature Variables API" refers to features of the application. It's pretty abstract, and how the application is divided up into features is up to the product teams. Over time, a feature may be involved in many experiments.

However, there is one rule:

β›…οΈπŸ”¬πŸ”­ Concept

For a given user each feature can only be involved in one experiment at a time.

If we see user change behavior after being exposed to an experimental treatment we need to be able to attribute it to that treatment, not another from a different experiment.

One easy way to start thinking about features, would be to identify user-visible surfaces of the app: the new-tab screen, the app-menu, the context-menu, the onboarding.

Imagine you're a designer, doing a re-design of the app's menu. It would be natural to call the app menu a "feature" of the app.

You've got some hypotheses around the icons, and whether they should be to right or left of the text. There is also some uncertainty around the copy for each menu item.

The uncertainties and hypotheses you have might translate into variations and variables you might configure the menu with. If these variables and variations are documented, they should travel as an adjunct to or part of the design itself. Later these will be turned into a more formal document that lives with the code, but it is at this stage when they should be thought about.

To narrow the scope for documentation purposes, we'll focus on a small number of variables. Within the app-menu feature, we'll consider the menu being made up of menu items, and we'll zoom in on the settings menu item.

As a team communication tool, it may help to consider a JSON object to enumerate the variables that are configurable for the settings icon, and their defaults.

{    "settings-menu-item-title": "Settings",    "settings-menu-item-icon": "ic_settings",    "settings-menu-item-enabled": true,    "settings-menu-item-action": "firefox://settings"}

This JSON object looks like what experimenters will be putting into branch configuration screens in Experimenter, under Feature Configuration.

Where did these keys come from? This is not up to Nimbus, but up to the app, i.e. the app team. In this hypothetical case, you have some theories about the title and the icon, and now the app needs to get those values from nimbus.

β›…οΈπŸ”¬πŸ”­ Naming Convention

Nimbus doesn't take a view on how you arrange the JSON, but by convention, like all other identifiers, it prefers kebab-case (i.e. lower-case-words-joined-with-dashes).

In the app code, the Variables object is a wrapper around this JSON object, and we have a number of getters to get values out. Notice that all getters return optional types, so it is up to the app developer to provide a default value.

let variables = nimbus.getVariables("app-menu")let action: String = variables.getString("settings-menu-item-action") ?? "firefox://settings"let title: String = variables.getText("settings-menu-item-title") ?? Strings.AppMenuSettingsTitlelet icon: UIImage = variables.getImage("settings-menu-item-icon") ?? UIImage(named: "icon-photon-gear")let isEnabled: Bool = variables.getBool("settings-menu-item-enabled") ?? true

It is a similar story in Kotlin:

val variables = nimbus.getVariables("app-menu")val action: String = variables.getString("settings-menu-item-action") ?: "firefox://settings"val title: String = variables.getText("settings-menu-item-title") ?: context.getString(R.string.app_menu_settings_title)val icon: Drawable = variables.getDrawable("settings-menu-item-icon") ?: context.getDrawable(R.drawable.ic_settings)val isEnabled: Bool = variables.getBool("settings-menu-item-enabled") ?? true

A few things to talk about here:

Fundamental types: Strings, Int, Bool#

getString(), getBool() and getInt() all return values as found in the JSON. If there is a disagreement about types, i.e. if the app is expecting a string, and the value in the JSON is an integer, the app gets nil or null.

Everything is optional#

If the app asks for a variable that is not specified in this particular experiment, then it gets back nil or null.

It is thus imperative that the app has a reasonable default. On the other hand, this allows us to have experiments which configure only small parts of a feature.

Text resources#

In the example above, the title uses getText(). This gets a string value with getString(). The value is then used as a key to look up the app resource string.

For example, on Android: getText("settings-menu-item-title") may get a string from the JSON "app_menu_settings_title", which is then resolves to R.string.app_menu_settings_title which is then used to look up the String in the Resources.

On iOS, getText uses a similar process via LocalizedString to look up the translated strings. You can specify the tableName as well as the key in the single value by joining it with a slash.

For example, getText("settings-menu-item-title") may get a string from the JSON "AppMenu/SettingsTitle" which uses bundle.localizedString("SettingsTitle", tableName: "AppMenu") to look up a localized string. If the app doesn't use tableName, then you can omit it: e.g. AppMenu_SettingsTitle would look for NSLocalizedString("AppMenu_SettingsTitle").

If getString() returned a string, and the resource lookup didn't succeed, getText() falls back to the string.

This means that you can use either pre-translated strings to try out experiments across locales, or target your experiment on a single language.

πŸŽ› Configuration

Resource lookup via Bundle and Context uses the objects passed to nimbus at construction time at app-startup. In Firefox for iOS and Fenix this is Bundle.main and context.applicationContext respectively.

πŸ“· Image resources#

In the example above, the icon uses getImage() and its Android analog getDrawable(). This gets a string value from the JSON with getString() and then uses that value to look up the pre-bundled resource.

For example on Android: getDrawable("settings-menu-item-icon") uses getString("settings-menu-item-icon") which might get the value "ic_settings" from JSON. This is then resolved to R.drawable.ic_settings, which is then resolved to context.resources.getDrawable(R.drawable.ic_settings).

On iOS: getImage("settings-menu-item-icon") uses getString("settings-menu-item-icon") which might get the value "icon_photon_gear", which is then used to get the named UIImage with UIImage(named:in:).

Making JSON more manageable#

We focused on the settings menu item in the above example, as a way of making a small enough example to reason about in this documentation, but it made for some very long variable names. The Variables object has itself a getVariables(key: String) method to make navigating the JSON more easily. This in turn allows the JSON to be organized in different ways.

Zooming out of our example above, which had just one menu item: we can re-arrange the JSON to accommodate multiple menu items, with a simpler nested structure:

{    "settings": {        "icon": "ic_settings",        "title": "Settings",        "action": "firefox://settings",        "enabled": true    },    "bookmarks": {        "icon": "ic_bookmarks",        "title": "View Bookmarks",        "action": "firefox://bookmark_list",        "enabled": true    },    "history": {        "icon": "ic_history",        "title": "View History",        "action": "firefox://history_list",        "enabled": true    }}

This might be accessed in Kotlin with:

val menuVariables = nimbus.getVariables("app-menu")var settingsItem = menuVariables.getVariables("settings").let { vars ->    val action: String = vars?.getString("action") ?: "firefox://settings"    val title: String = vars?.getText("title") ?: context.getString(R.string.app_menu_settings_title)    val icon: Drawable = vars?.getDrawable("icon") ?: context.getDrawable(R.drawable.ic_settings)
    MenuItem(icon, title, action)}

In Swift:

let menuVariables = nimbus.getVariables("app-menu")let settingsItem = menuVariables.getVariables("settings").let { vars ->    let action: String = vars?.getString("action") ?? "firefox://settings"    let title: String = vars?.getText("title") ?? Strings.AppMenuSettingsTitle    let icon: UIImage = vars?.getImage("icon") ?? UIImage(named: "icon-photon-gear")    let isEnabled: Bool = vars?.getBool("enabled") ?? true
    MZMenuItem(icon: icon, title: title, action: action)}

πŸ‘‹ Information

variables.getVariables() can be arbitrarily deep. variables.getVariables() returns an optional Variables object.

Recording exposure events#

β›…οΈπŸ”¬πŸ”­ Enrollment versus Exposure

When a client is selected to take part in an experiment, they are assigned a branch. This is enrollment.

However, the user may not be exposed to the branch until sometime later. The exposure is the earliest moment that the user could be affected by the experimental treatment.

Nimbus records the enrollments and exposure events using Glean.

Enrollments are recorded at each app start-up, and exposure events each time an exposure happens.

For experiments in Firefox for iOS and Android, enrollment happens shortly after app-startup.

In our example above, the app menu is constructed when a tab is open. The user is only exposed to the values of the JSON when they tap on the open-menu icon.

By default, exposure is recorded when nimbus.getVariables(featureId: String) is called. Whichever experiment the feature is enrolled inβ€” always exactly zero or oneβ€” has an exposure event recorded.

A second, optional argument is allowed for this method getVariables, to change this default behavior.

Here, the menu is constructed with variables from Nimbus, but the user doesn't see the menu until they open it.

val menu = createMenu(    nimbus.getVariables("app-menu", sendExposureEvents = false))val menuButton = Button(    icon = R.drawable.ic_menu,    onButtonPressed = {        nimbus.recordExposureEvent("app-menu")        show(menu)    })

This is a caricature of the same code in Swift.

let menuSheet = createMenuSheet(    nimbus.getVariables("app-menu", sendExposureEvents: false))let menuButton = UIButton()menuButton.addTarget(self, action: #selector(didOpenMenu), for: .touchUpInside)
func didOpenMenu() {    let nimbus = Nimbus.shared    nimbus.recordExposureEvent("app-menu")    viewController.present(menuSheet, animated: true, completion: nil)}

Nimbus will take care of finding out what experiment, if any, the user is enrolled in when using this feature.

Using configurable features to experiment with another#

The feature itself may be configurable, but we don't have to limit feature configuration to experiments about that feature.

We can imagine a world where we have multiple configurable features, say: an app-menu, onboarding and newtab. On each of these features we have a messaging surface, and we want to run an experiment to find which is the best surface to show the message about a behavior we wish to maximize: setting the browser to be the device default.

Can we configure an experiment to test each of the message on each of these messaging surfaces?

πŸ‘‹ Unimplemented

Coming soon While this is available in Experimenter, currently the Nimbus SDK does not support different features in different branches.

We might also imagine a world where we have multiple features as before. Two different product teams are experimenting with two new capabilities of the app: both require onboarding instructions, one has an entry point via a app menu item, and the other has an entry point in the new tab screen.

If it was one product team where communication is high, perhaps they might run one experiment, with two treatment branches: one branch with configuration for the onboarding and app-menu features, and one branch with configuration for the onboarding and new-tab features.

Both teams require the onboarding feature. This allows each team to run their own experiments, which do not interfere with one another.

β›…οΈπŸ”¬πŸ”­ Concept

While for any given user a feature may be involved in only one experiment, one experiment should be able to configure multiple features.

Because both product teams' experiments require the onboarding experiment, no user will be involved in both experiments.

πŸ‘‹ Unimplemented

Coming soon Configuring multiple features per branch isn't yet in Experimenter.

πŸ”§βš™οΈ Working with configurable features#

Throughout the process of designing and building these configurable features, the feature variables have needed to be documented. At first, when the feature is being envisioned and designed, the variables should travel with the designs themselves.

When the feature is being implemented, these variables will begin to acquire concrete names, types and organization, which will be used extract JSON from the Nimbus SDK and configure the application features themselves. This documentation will begin to take shape and textual organization that travel in the app's repository.

When the feature is being tested, QA testers are going to want to configure the features within bounds and tolerances set by the designs and the engineers.

Finally, when the feature is part of experiments, then the experiment owner, setting the branches in Experimenter needs to be able configure the branches with variables with spellings and organization that match the app implementation.

It's strongly recommended that you keep a track of these identifiers and descriptions of the configurability of each of the features in at least one place. After implementing a configurable feature, it should probably live with the code repository, so it can versioned and tracked with the code.

πŸ‘‹ Unimplemented

The Nimbus team has a few ideas about this, for documenting the features in an app in a machine readable format, tentatively called the Manifest. We're thinking:

  1. generating code for app developers to use in app code
    • managing defaults values
    • removing magic strings
  2. automating the registering of feature ids to Experimenter
    • currently this needs an Experimenter reviewer or admin.
  3. generating Experimenter UI for experimenters to use in experiment branch design
  4. generating on-device tools for feature testing, branch design and local development.

We're in the early stages of planning this, so would love to hear some feedback or ideas you may have.