It’s frustrating as a data scientist to spend a substantial amount of time making sure confidence intervals are correct and then everyone ignores them. Not significance: no, people care about whether the number is colored red/green and not gray for insignificance. But once that hurdle is passed, then only the number remains. “What did that launch do? It increased goodthing 7.4%.” Never mind that the CI was 2.1% to 11.7%.
Really this is a limitation of our internal representations. We understand concrete numbers but it’s hard to intuitively grasp the idea of intervals. So instead folks anchor on the definite number and forget the rest. At best you might try to train folks to anchor on the concrete endpoints and then do some system 2 thinking. But mostly you just throw your hands up and go about your day.
But maybe this is a representation problem. What if we represented the numbers probabilistically? Terence Tao has an post using gifs to visualize random variables. With a little JavaScript we can make it a little nicer like so: 6.9%. It iterates between “plausible values” every 5 seconds: by the time someone is ready to anchor, it shifts.
Getting the “plausible values” is a little handwavy. It’s quite natural in the case of Bayesian estimates as draws from the posterior. That’s what I’m doing here, assuming a nice conjugate posterior to get a normal distribution. The frequentist framing is a little more gimmicky but you can imagine something like a uniform draw from the set of possibilities that haven’t been rejected. Bootstrapping gets you closer on the frequentist side but fails for non-symmetric or biased distributions.
Let’s go back to the example: when asked what the effect of the launch was, there’s no single number to point at. Instead it’s a gestalt: definitely positive but like hard to pin down. “Somewhere around seven or eight but could be a good bit higher or lower. You have to grapple with the uncertainty: you’d best tailor your conclusions to fit most of the possible iterations; otherwise you’re going to look pretty dumb when at that very moment the drawn number contradicts you.
Of course it’s rather hard to intuitively gauge digits, which makes this complicated (and argues for the 5 seconds between iterations). You can go much faster with more visual representations. Instead of the digit we can represent the interval on a graph and iterate much more quickly.
This is the hypothetical outcome plot as introduced by Hullman 2015. The standard advice for statistical visualization is that we have differing reliability for comparisons: length is easy to compare, area less so, and raw digits are not great (hence why most tables should be a figure). The same holds for visualizations of uncertainty, where error bars are relatively hard to grasp. They show that study participants using visualizations like this outperform standard static representations of uncertainty like error bars or violin plots.
I do hope one day I can unleash this idea on my non-statistical partners. I do think it would be a boon to principled decision making but I’ll need to invest in some bodyguards when PMs realize that from time to time a random draw implies they failed to meet their OKRs.