Understanding Minimum Detectable Effect (MDE)

What is MDE?

Minimum Detectable Effect (MDE) is a critical concept in any type of testing. It represents the smallest improvement (uplift) you want to be able to confidently detect as statistically significant in a test metric. In essence, MDE defines the sensitivity of your experiment.

Why is MDE Important?

MDE directly impacts the sample size needed for your test. Smaller effects are harder to detect and require more data (samples) compared to larger effects. By defining the MDE upfront, you can ensure your test has the power to identify meaningful changes, even if they are subtle.

Visualization of MDE

Imagine comparing two objects. A significant difference in size is immediately noticeable, while a smaller difference might take longer to detect. Similarly, in testing, larger uplifts are easier to identify than smaller ones.

For instance, you want to test a new design for your website’s homepage to see if it will hike up your conversion rate.

Let's say your baseline conversion rate is 10%. This means that currently, 10% of visitors who come to your homepage convert. You set your MDE to ±5%. This means that you want to be able to detect changes in the conversion rate that are at least 5% larger or smaller than the baseline conversion rate.

In other words, if the new design results in a conversion rate between 9.5% and 10.5%, you would have a lower chance of detecting them as statistically significant.

Here's a table to summarize the concept:

By setting a smaller MDE, you can reduce the undetectable range and make your test more sensitive to smaller changes in the conversion rate. However, this will also require a larger sample size (more visitors to your website) to achieve statistical significance.

MDE and Experiment Design

While the exact uplift from a test is unknown beforehand (otherwise, the test wouldn't be needed), MDE is crucial for proper experimental design. Statistical principles require considering the minimum effect size you're willing to overlook to determine the necessary sample size.

Best Practices for MDE

Here are some key considerations for setting an MDE:

• Balance Sensitivity and Practicality: While it's tempting to set a very low MDE to detect even minor improvements, this can lead to impractically large sample sizes and extended test durations.  Choose an MDE value based on the duration you wish to run a test. Ideally a test should run in weekly cycles to ensure all visitor behaviours are accounted in a test.
• Metric Sensitivity: A smaller MDE might be appropriate for stable metrics where large significant uplifts are hard to achieve. Conversely, a larger MDE is preferred for sensitive metrics to detect even minor positive effects. 

For example, Imagine you're running an A/B test to improve your website's conversion rate. The size of the change you want to detect will influence how long you need to run the test.

• • Stable Metrics (e.g., Revenue per Visitor): These metrics exhibit less fluctuation and require a smaller MDE to detect significant changes.
  • Sensitive Metrics (e.g., Time Spent on Page): These metrics are more prone to change, making it easier to detect positive effects with a larger MDE.

• Sample Size and Traffic Volume: A larger sample size allows you to utilize lower MDE values, enabling the detection of smaller uplifts.
• Adaptive MDE Strategy: Starting Higher and Refining Over Time
  Begin your A/B test with a higher MDE to ensure the test is completed within a practical timeframe. As data accumulates and the observed visitors reach the required sample size, consider stepping down the MDE in subsequent phases of the test. This adaptive approach allows you to:
  
  • Rule out larger improvements: By starting with a higher MDE, you ensure that any improvements more than MDE are detected early in the test.
  • Optimize test duration: Adjusting the MDE as the test progresses can help balance the need for detecting smaller effects with the practical constraints of time and resources.
  • Maintain statistical power: Gradually lowering the MDE ensures that the test remains sensitive to smaller effects without compromising statistical guarantees.

By effectively utilizing MDE, you can design well-structured tests that deliver reliable results. MDE helps you strike the right balance between detecting meaningful changes and ensuring your tests are efficient and cost-effective.

Need more help?

For further assistance or more information, contact VWO Support.