Skip to main content

Nimbus Features

Nimbus is a full-featured experimentation platform that provides configuration, analysis and client libraries for both experiments and rollouts.

Experimenter Console#

  • Simple experiment configuration UI
  • Pre-launch testing support
  • Multifeature experiments
  • Rollouts
  • Simple targeting
  • Advanced customizable targeting
  • Remote settings integration
  • Experiment Review / Approval Workflow
  • Dashboards
  • Monitoring and Analysis

Experiment Types#

BranchesControlCo-existence / AvoidanceEnrollmentAnalysisGood for?Permanent Change
ExperimentA/B - equal sizes. Size fixed at launch - cannot grow.YOne per feature at a time. Avoids other live experiments on a specific feature. Keep as small as possible to learn.Fixed timeframe. 7 day typically. Analysis only AFTER enrollment endsAutomated for default metrics. Can select outcomes or create new customStat sig differences from control + absolute counts.No - all changes delivered through nimbus reverted.
Message experimentSame AND each branch is given a messageID that the messaging system uses.Same as experimentSame as experiment AND message system throttles max # per time frameSame as experimentSame as experimentSame as experimentN Same as experiment
Multi-feature experimentSame as experimentSame as experimentAvoids other live experiment with EITHER FEATURE - **restricted audienceSame as experimentSame as experimentSame as experimentN Same as experiment
Holdback ExperimentGo right to end stage of rollout - 90% treatmentY - 10%Same as experiment AND no other feature can be tested in this feature for the duration.Same as experimentSame as experimentDecision made to deploy quickly - but want data. A brief step before shipping as default in tree.No - all changes delivered through nimbus reverted. Fast follow shipping in tree.

Feature Delivery#

BranchesControlCo-existence / AvoidanceEnrollmentAnalysisGood for?Permanent Change
Rollout TodaySize increases 5%, 25%, 50%, 100%NUnrestricted co-existence Does not avoid other rollouts or experiments.N/ANone - teams can make and monitor their own dashboards - but no “stat sig analysis” offered.Risk mitigation for explosive failures (looking for crash, bugs, media issues)Yes. Changes do not revert on targeted versions. Teams ship as default in tree in the release at 100%.
Branch Promotion1 - 100%NOn an experiment with a winning branch - you can select “promote to rollout” which clones the ticket with the winning configuration filled inNNGet winning branch to users quicker than trainsSame as Rollout Today
Do No Harm Monitoring Platform ExperimentSame as experimentSame as experimentSame as experiment.No fixed timeframe means there is no automatic analysis trigger.Not automated. Requires OpMon configured or team using their own custom dashboards.Code changing/fixes landing during. Risk mitigation for quick major breaking stability / perf changes. Looking at adding search metrics as well.N Same as experiment

Multiple language Client integrations#

  • Android (Kotlin)
  • iOS (Swift)
  • Firefox Desktop Frontend (JS)
  • Firefox Platform (C++)

Client-side functionality#

  • Consumer opt-out
  • Integration with console preview testing
  • View experiments (about:studies)

Current client integrations#

  • Firefox Desktop
    • Frontend via JS SDK
    • Platform
    • Windows installer
  • Firefox Mobile via Nimbus-SDK Rust Component
    • Fenix
    • Firefox iOS
    • Focus Andriod
    • Focus iOS

Requesting feature support#

If you aren't sure we have what you need, pop into #ask-experimenter with your questions or file an issue