Rollouts and Experiment Interaction
See the Rollout FAQ for general rollout information
Question
Can a client be in a rollout and an experiment for the same feature at the same time?
Answer
Yes, we have separate namespaces which means you can run rollouts and experiments simultaneously and each namespace does its accounting for users separately. You can run a 100% rollout and a 5% experiment and both will fill up, but everyone in the experiment is also in the rollout.
Question: What happens when an experiment and rollout are on the same feature, and the experiment ends?
Answer
Typically users will see the experiment applied, and then get the rollout treatment when that ends with a small delay to re-evaluate.
If both the experiment and the rollout are focused on the same feature, the experiment will take precedence and in this case the behavior will be seeing the experiment, when that ends the client will unenroll, then upon next recipe evaluation will join the rollout.
UNLESS the feature is mobile messaging, in that case while both are running, an experiment client with actually be enrolled in both and the treatment they’ll receive will be the combination of both. This enables clients in subsequent experiments to easily get rolled out winners of prior experiments. If they’re focused on different fields of the JSON, the behavior the clients gets will go from “experiment+rollout” to “rollout only”.
Question - How do I decide when to end a rollout?
ex: I have a feature that is now on by default in v115, and I want to wait until it has saturated the user base before shutting off the rollout. Do we have a standard around how long we should keep a rollout going in a situation like this?
Answer
We typically wait 1-3 releases for the upgrade tail to catch up. You'll probably have the vast majority, so it just depends on the ramifications if people have it enabled on 114 and then lose it.
If it's not a big deal if the feature goes away, turn off the rollout in the next release. People who haven't yet upgraded will lose the feature until they upgrade.
If it is a big deal for the feature to go away (think UX changes or bug fixes) - turn off in 117 or 118 and you'll have reached a few percent more with each release.
If it is a very big deal - we sometimes leave those running for 6 months or so to hit maximum reach. Please remember to set a reminder appointment to people on your team so someone remembers and doesn't leave it on forever.