How to Read SplitSignal Test Results?
Effect Size - Percentage of increase/decrease in clicks during the lifetime of the test.
Absolute Effect - Absolute number of increase/decrease in clicks during the lifetime of the test.
Confidence Level - This metric indicates the level of statistical significance of your test results.
How significant is the result:
- 95% or Higher: the change is large enough to be statistically significant. We are confident that the test change caused the increase/decrease in clicks.
- 94% or less: the change is not large enough to consider it statistically significant
Test Duration - shows how long the test ran.
- Positive - The test is positive and statistically significant.
- No Change - The test didn’t reach statistical significance.
- Negative - The test is negative and statistically significant.
“Good” Prediction Interval (Real Clicks and Predicted Clicks very closely match. This makes for the most conclusive tests)
“Fair” Prediction Interval (Real Clicks and Predicted Clicks mostly match. This makes for an often conclusive tests)
“Poor” Prediction Interval (Real Clicks and Predicted Clicks do not closely match. This makes for inconclusive tests)
NOTE On Predicted Clicks:
We don’t necessarily ‘predict’ clicks of the control group but instead build a counterfactual prediction based on the control clicks and its relationship with variant historical clicks. We understand the historical relationship between control and variant and then use it to understand if something has changed during the test. We’re not predicting the future. If the relationship remained stable through the test - the results are insignificant. If the relationship has changed through the test - the results become significant.
- SplitSignal FAQ
- Issues with Crawling a Domain
- What on-page changes are available for testing in SplitSignal?
- How to use the upload variables feature (import from file)?
- SplitSignal Pre-SEO Test Launch Checklist
- SEO Split-Testing: Is My Website a Good Fit?