Jetstream writes analysis results and enrollments information to BigQuery. Statistics data and metadata gets exported to GCS to make it accessible to the Experimenter console.
The datasets that back the Experimenter results dashboards are available in BigQuery in the
mozanalysis dataset in
moz-fx-data-experiments. Technical documentation is available in the Mozilla data docs.
Datasets used for monitoring the operation of Jetstream are part of the
monitoring dataset in
Jetstream logs errors and warning encountered during its analysis runs to
monitoring.logs. This datasets is used as basis for the Jetstream error dashboard and for setting up alerts.
logs table has the following schema:
|Timestamp of when the log event was recorded|
|Experiment slug for which event was recorded|
|Log level: ERROR, WARNING|
|Raised exception object|
|Name the Jetstream code file the exception was raised|
|Name the Jetstream function the exception was raised|
|Class name the exception raised|
monitoring.query_cost_v1 dataset contains the cost of each query run when analysing experiments. The dataset is updated daily and scrapes the cost information from the BigQuery logs. The query for determining the costs is part of bigquery-etl. The dataset is basis for the jetstream cost monitoring dashboard and alerts set up to send notifications when an analysis query exceeds a certain threshold.
query_cost_v1 table has the following schema:
|Timestamp of when the query was executed|
|Name of the table query was writing data to|
|SQL of the executed query|
|Number of bytes the query processed|
|Cost of the query in USD based on BigQuery pricing|
For monitoring Nimbus experiments, some common failure cases are exposed as part of the Experiments Enrollments dashboard. These monitoring rules will require access to collected experiments enrollment data which is available in
monitoring.experimenter_experiments_v1. This dataset is part of bigquery-etl and updated every 10 minutes by fetching data from the Experimenter API.
GCS Data Export
Jetstream exports statistics data and metadata of analysed experiments to the
mozanalysis GCS bucket.
After each analysis run has completed, Jetstream exports the statistics results of each experiments to the
statistics sub-directory as JSON. The JSON files follow the naming format:
Each file contains a JSON object for every row in the corresponding statistics table. The JSON files are pulled in by Experimenter and used for visualizing results on the Experimenter results page.
Metadata of analyzed experiments contains information about all metrics and outcomes that are computed during any analysis period. Metadata is written to JSON files into the
metadata sub-directory with the following naming schema:
Each JSON metadata file contains the following information:
"friendly_name": "Friendly metric name",
"description": "Metric description defined in mozanalysis or metric-hub",
"friendly_name": "Friendly outcome name",
"description": "Outcome description defined in metric-hub",
"metrics": [ // metrics computed as part of outcome
// commit hash of outcome version that was used in analysis
Metadata of metrics and outcomes is used to show names, descriptions and whether larger numbers are better in the Experimenter results.