All Collections
Statistical Significance With AB Testing
Statistical Significance With AB Testing
Updated over a week ago

Here are some quick tips on how to make better decisions on your AB tests with statistics


When you run an experiment or analyze data, you want to know if your findings are "statistically significant". When making decisions on effectiveness and engagement of marketing activities, you want to make sure your results aren't due to chance instead of the variable you're testing on (example: message copy). Running a chi square test is recommended and can be done without pulling out the good old pencil, paper or dusty TI-84 from high school. Here's a reliable handy chi-square calculator.


  • Determine the metric. In this example I'll pick CTR

  • Calculate your CTR (clicks / sends) for the two test variants you're looking to compare. (In this screenshot below I did this based on 20% vs 25% CTR)

  • Plug those numbers into the two top boxes

  • Click "Result"

    Screen Shot 2020-10-20 at 3.00.55 PM
  • In this example I need 1504 sends in each group of my test. Because I picked a power of 0.8, I can be 80% sure that group 2 will result in higher CTR than group 1

Did this answer your question?