Leverage Graphs to Analyze Your Campaign Report

About Graphs

Graphs in VWO are pictorial representations of your campaign data, capable of delivering the entire picture concisely. This enables you to grab information faster. Indeed, a glance is all it takes to understand the chances of your versions becoming the winner or the smart decision. However, reading values from graphs can be challenging at times. But don’t you worry yet, Just gear up - we’re gonna decipher them now.

Types of Graphs

Based on your campaign report data, VWO provides the following graphical formats:

Date Range

The Date Range graph gives you the real-time status of the conversions, revenue, conversion rate, and the number of visitors that have arrived at your campaign URLs. The following are the Key Performance Indicators (KPIs) represented in the graph:

Expected Value per Visitor Curve

This Probability Density (PD) curve in the shape of a bell depicts the prediction for your campaign, giving you the probable scope of the conversion rate (For binary metrics) or Metric value per visitor (For continuous metrics). It only features two parameters for this - the Expected value of the metric on the X-axis and the Probability Density on the Y-axis.

Let’s look at the following example of a PD curve of a binary metric representing our estimate of the conversion rate of the control version in a campaign.

The curve progresses upwards from around 39% and reaches the peak at 44.12%. It finally touches the ground at 49%. This means that the conversion rate range for your campaign is likely to fall between 39% and 49%, with 44.12% being the highest probability.

This range is generated by accumulating data points, which will keep the curve shrinking towards the centre with the incoming visitors. This means the nearness to certainty keeps increasing with more data.

In the below graph, the orange curve represents the control version, and the blue curve represents the variation. Here, the orange curve (control) is narrower than the blue curve (variation), indicating that the orange curve (control) has had more visitors as compared to the blue one (variation).

For a campaign that has several variations, you will be able to see the PD graph with a curve for the control and each of the variations. This brings the following scenarios based on the overlap of the curves.

No Overlap

When the curves appear so far apart from each other that there is no overlap between them, then the curve at the right would be the one with the higher conversion rate. This means that the one on the right side has a greater scope of becoming a winner.

The following graph shows the control (orange curve) and a variation (blue curve), which do not overlap each other, with both the control and the variation sharing almost the same level of probability density.

However, the control has a conversion rate of just 44.12%, while that of the variation is at 54.71%. In this case, the variation is a clear winner.

With Overlap

Quite often, you might encounter scenarios where these curves overlap. Overlap of the curves means that the probability of conversion rate is being shared between the variations. The area of overlap plays a significant role in deciding the winner.

Case 1:

If the majority of the curves fall within the overlapping area, the winning chances are equally spread between the two. This leaves no room for a conclusion. The following graph features two variations - blue and brown.

Here, the median conversion rates are much closer to each other, and so are the ranges of probabilities. In such cases, declaring a winner is almost impossible.

Case 2:

If the overlapping area is smaller, the chances of declaring a winner certainly fall to the party with the higher probability density and conversion rate. However, it doesn’t always become the case where both these parameters belong to the same candidate. Sometimes, the curve on the left has more certainty and a lesser range than the one on the right. This would more probably lead to a situation where declaring a winner could be complex, and you would rather settle down to declare a variant as the smart decision.

Case 3:

Chances are more likely that you will run your campaign with more variations than just one. Overlaps, here, can provide a good indication of which variants are superior to others. You can choose to select only those variations whose trends you want to read for a detailed comparison.

In the below graph, the orange curve indicates control, the pink curve indicates variation 1, and the blue curve indicates variation 2.

As you can see the blue curve has no overlap with the orange or pink curves, you can declare the blue curve (variation 2) as the winner, as it outperforms the others.

Expected improvement Curve

The expected improvement curve represents a specific variation's expected improvement over the baseline. It only features two parameters for this - the Expected improvement relative to the baseline on the X-axis and the Improvement Density on the Y-axis. The value marked on the curve represents the most probable improvement you can expect to achieve after deployment.

Let’s look at the following example of an EI curve of a variation relative to the baseline in a campaign.

The curve progresses upwards from around -60% and reaches the peak at 10%. It finally touches the ground at 80%. This means that the expected improvement for your campaign is likely to fall between -60% and 80%, with 10% being the most likely.

This range is generated by accumulating data points, which will keep the curve shrinking towards the centre with the incoming visitors. This means the nearness to certainty keeps increasing with more data.

The real strength of improvement distribution lies in its ability to provide a probabilistic understanding of potential improvements. Upon hovering over the curve the improvement distribution is divided into three distinct regions:

• Worse: (-infinity to -ROPE): The proportion of the Expected improvement curve in this region is the probability of that variation performing worse than the baseline.
• Equivalent: (-ROPE to +ROPE): The proportion of the Expected improvement curve in this region is the probability of that variation performing equivalent to the baseline.
• Better: (+ROPE to infinity): The proportion of the Expected improvement curve in this region is the probability of that variation performing better than the baseline.