Skip to main content

Rollouts

Common questions about rollout management, sizing, and interaction with experiments.

Can a client be in a rollout and an experiment for the same feature at the same time?​

Yes, we have separate namespaces which means you can run rollouts and experiments simultaneously and each namespace does its accounting for users separately. You can run a 100% rollout and a 5% experiment and both will fill up, but everyone in the experiment is also in the rollout.

What happens when an experiment and rollout are on the same feature, and the experiment ends?​

Typically users will see the experiment applied, and then get the rollout treatment when that ends with a small delay to re-evaluate.

If both the experiment and the rollout are focused on the same feature, the experiment will take precedence and in this case the behavior will be seeing the experiment, when that ends the client will unenroll, then upon next recipe evaluation will join the rollout.

UNLESS the feature is mobile messaging, in that case while both are running, an experiment client with actually be enrolled in both and the treatment they'll receive will be the combination of both. This enables clients in subsequent experiments to easily get rolled out winners of prior experiments. If they're focused on different fields of the JSON, the behavior the clients gets will go from "experiment+rollout" to "rollout only".

How do I decide when to end a rollout?​

ex: I have a feature that is now on by default in v115, and I want to wait until it has saturated the user base before shutting off the rollout. Do we have a standard around how long we should keep a rollout going in a situation like this? We typically wait 1-3 releases for the upgrade tail to catch up. You'll probably have the vast majority, so it just depends on the ramifications if people have it enabled on 114 and then lose it.

If it's not a big deal if the feature goes away, turn off the rollout in the next release. People who haven't yet upgraded will lose the feature until they upgrade.

If it is a big deal for the feature to go away (think UX changes or bug fixes) - turn off in 117 or 118 and you'll have reached a few percent more with each release.

If it is a very big deal - we sometimes leave those running for 6 months or so to hit maximum reach. Please remember to set a reminder appointment to people on your team so someone remembers and doesn't leave it on forever.

How quickly can I get to 100% with a staged rollout?​

Your sizing steps are a balance of:

  • how many users do you need to gain confidence to go to the next step?
  • How big of a step are you comfortable with?

Some common patterns are:

  • Low risk: 5, 10, 50,100 (the steps may be bigger if only a fraction of the populatio will use the feature)
  • Med risk: 1, 5, 10, 25, 75, 100
  • High risk or scaling risk: .5, 1, 5, 10, 25, 50, 75, 100 (ex: you are worried about too many people using a back end server at once, and want to test small capacity loads)

The first step is usually the one where issues are discovered. It's easy to stop the rollout, land fixes, and start again with a higher minimum version (after the fixes have landed).

The speed you increase is based on:

  • What are you observing to decide you are OK moving forward?
  • How long will it take to see those changes?

Some common patterns are:

  • Fast: 2 days, 7 days, 10 days (ex: a frequently used feature, where the telemetry exists for your team to easily see issues)
  • Med: 1 week, 2 week, 3 week
  • Slow: 2 week, 4 weeks, 6 weeks (ex: it takes users a while to use the feature or looking at revenue or other indirect impacts)

How long should I wait for feature saturation before ending a rollout?​

I have a feature that is now on by default in v115, and I want to wait until it has saturated the user base before shutting off the rollout. Do we have a standard around how long we should keep a rollout going in a situation like this? Yes, we typically wait 1-3 releases for the upgrade tail to catch up. You'll probably have the vast majority, so it just depends on the ramifications if people have it enabled on 114 and then lose it.

If it's not a big deal, turn off the rollout in 116 and most people will have it and the stragglers will get it when they upgrade. If it is a big deal - turn off in 117 or 118 and you'll have reached a few percent more with each release.