![](https://i0.wp.com/jonathansblog.co.uk/wp-content/uploads/2019/08/img_20190828_143810_32215566120126554020.jpg?resize=1200%2C1600&ssl=1)
talk by split
experimentleaks.com
use data to drive the decisions. A/B testing, multivariant – controlled experiments reduce external influences. it can allow us to distinguish from noise and real signal.
measure ‘statistically significant’ detecting something meaningful.
errors: false positives + false negatives
![](https://i0.wp.com/jonathansblog.co.uk/wp-content/uploads/2019/08/img_20190828_144526_18628631910902753214.jpg?resize=1200%2C1600&ssl=1)
design like you are right, test like you are wrong.
![](https://i0.wp.com/jonathansblog.co.uk/wp-content/uploads/2019/08/img_20190828_144651_5903037854736030497.jpg?resize=1200%2C1600&ssl=1)
How to run an experiment:
![](https://i0.wp.com/jonathansblog.co.uk/wp-content/uploads/2019/08/img_20190828_144822_88496040020610898171.jpg?resize=1200%2C1600&ssl=1)
make a hypothesis, expected results, pick metrics – the standard scientific method.
lick the correct timeframe for the experiments.
try to measure more than one thing, but not too many things
![](https://i0.wp.com/jonathansblog.co.uk/wp-content/uploads/2019/08/img_20190828_145321_02439304126024234707.jpg?resize=1200%2C1600&ssl=1)
there are online calculators to show you the sensitivity of the experiment (eg how much of a change you can measure)
tools:
split
google optimise
a/b
etc
dont peek at the data while the experiment is running
analysis:
look for low p-values < 0.05