talk by split
experimentleaks.com
use data to drive the decisions. A/B testing, multivariant – controlled experiments reduce external influences. it can allow us to distinguish from noise and real signal.
measure ‘statistically significant’ detecting something meaningful.
errors: false positives + false negatives
design like you are right, test like you are wrong.
How to run an experiment:
make a hypothesis, expected results, pick metrics – the standard scientific method.
lick the correct timeframe for the experiments.
try to measure more than one thing, but not too many things
there are online calculators to show you the sensitivity of the experiment (eg how much of a change you can measure)
tools:
split
google optimise
a/b
etc
dont peek at the data while the experiment is running
analysis:
look for low p-values < 0.05