An Outcome is a reusable configuration snippet that defines metrics and statistical summaries for Jetstream to compute, which will be displayed on the Experimenter results page.
Outcomes are defined in TOML snippets in the
outcomes/ path of the jetstream-config repository.
Watch a demo about how to use Outcomes:
A simple Outcome definition (excerpted from a live Outcome) looks like:
The important elements are:
descriptionfields, which define how the Outcome appears in Experimenter
- One or more
metricsblocks that describe the metrics to compute, and any supporting
data_sources as necessary.
Opening a pull request to
jetstream-config with an Outcome definition will cause it to be validated.
If it passes, a data scientist can merge the Outcome definition without additional review.
Experimenter will need to be re-deployed to pick up a new Outcome. Please ask in #nimbus-project if you're in a hurry!
As a data scientist, it's useful to define Outcomes whenever an endpoint is going to be used more than once. It reduces the amount of work you will need to do for each follow-up experiment and ensures that metrics are defined consistently.
If you're not certain about how you want to define a metric, it's okay to use a custom configuration first, and then copy-paste the metrics into an Outcome for later use.
Changing an Outcome does not re-run any experiments. A commit hash associated with the version of each Outcome is captured in the experiment metadata published to GCS, so it's possible to understand which version of an Outcome was associated with an experiment analysis.