|
In this article, you’ll learn the following: |
Overview
Your campaign setup is complete, and you execute the campaign. Now is the time to pass on the baton to VWO’s SmartStats engine, which constantly monitors your experiment outcomes and furnishes the results based on your statistical configuration for the experimented metric.
Your campaign’s Progress Indicator gives you a condensed overview of the entire campaign journey. It’s typically a progress bar on which you get to know the current status and the historical points of notable moments of the campaign, such as recommendations and retractions. When you’re on a fast track, or you just need to get an overall view of things, the Progress Indicator comes to your aid.
To access your test campaign’s Progress Indicator, from the main menu, go to the respective campaign for which you want to view the report and click the Report tab. The Campaign Progress Indicator is located immediately below your report filters.
A campaign typically goes through two phases:
| What phase? | What happens? | What do you get to do? |
| Initial Data Collection Phase |
Estimating the campaign duration Once the campaign is executed, VWO starts collecting data from visitors. After 500 visitors and 1 conversion each on the baseline and all the variations, VWO estimates the campaign duration based on the statistical configuration you’ve set for the primary metric.
INFO: To learn about how to estimate the duration of your campaign, click here. |
|
|
Testing Phase |
Identifying the better performer As your campaign progresses, VWO’s SmartStats engine carefully examines the outcomes to identify the probability of the uplift being actually true rather than by chance. It evaluates the performances of the individual variations and generates recommendations correspondingly. Based on the statistical configurations, the SmartStats engine issues recommendations and retractions, if any
NOTE: Data collection for your campaign will continue in this phase as well. |
|
Recommendations
When you configure the statistical parameters for the primary metric, you declare the minimum improvement you wish to test in the campaign. Based on this, in the course of the campaign, the SmartStats engine issues a recommendation on the following scenarios:
- When the Probability of Better hits a specified threshold (95%) for a variation, VWO declares it better than the baseline and recommends rolling it out.
- If a variation does not show potential to be better, the SmartStats engine recommends disabling it to keep the focus on the better-performing variations and achieve results faster.
However, to ensure statistical reliability, at least 10 conversions, each, on the baseline and the respective variation should be made before a recommendation is issued.
Retractions
While a recommendation has arrived, chances are there the visitor behavior changes, and the probability of Better retracts below a threshold. This threshold is called the Retraction Threshold, below which a previously declared winner is retracted. The retraction threshold is kept 5% below the decision threshold to avoid fluctuation in recommendations. In such cases, the SmartStats engine retracts the previously issued recommendation.
DID YOU KNOW? The chance of getting a retraction is equal to the false positive rate of the campaign.
REMEMBER: For the most reliable results, it's recommended that your campaign runs for at least 7 days. While you may see results sooner, waiting 7 days is strongly advised. Additionally, you would need to monitor a few other checks that can help prevent discrepancies in your campaign.
How Does the Campaign Progress Indicator Reveal Your Campaign Journey?
The Campaign Progress Indicator features a progress bar and states the overall status of your campaign. It reveals how many days remain until the final conclusion and any actionable items, such as disabling a variation or rolling out one.
The progress bar captures the recommendations and retractions and features them at the points of their occurrence. You can take a quick glimpse of these by simply hovering over them. Each recommendation and retraction carries the date and time of their occurrence. If two or more entries (recommendations and/or retractions) occur, then they are clubbed together, and their total number is indicated. In such scenarios, the entries are listed in the chronological order.
Notifications
To get a more detailed view of the entries, such as the reason for the retraction of a variation, along with the actionable links for disabling an under-performing variation and rolling out a better-performing one, you can access the Notifications icon at the end of the campaign progress indicator bar. The Notifications popup features two tabs:
- Recommendations - Features the recommendations and retractions that have occurred during the course of the campaign. If a recommendation suggests that you disable a variation, it also provides a link to the actionable for quick implementations.
- Experiment vitals - Features any breach in the vital parameters that are essential for your campaign to produce reliable results. You are warned of every breach and what caused it. It also displays who performed the action along with the timestamp.
With every vital, potential actions that can be taken to resolve the issue are also recommended.
Sometimes, a breached vital can be ignored if you’re aware of the consequences and if the cause for the breach is a conscious effort. Then, you can simply ignore them by clicking on the Ignore icon. By default, the ignored vitals are marked as “Hidden” and displayed in the Notifications popup. This actually resolves the breach alert inside Experiment Vitals. To hide the hidden warnings inside the Notifications popup, you can unselect Show hidden vitals.
When you’re managing too many tasks at a time and digging deeper into your campaigns demands more of your attention span, use the Campaign Progress Indicator for a quick remedy to gather insights and work on actionable items on the fly without fear of error.