You Can't Learn A New Fact With A New Method

You find a fun stats problem: you need to account for some complication with new methodology. You find remarkable results! You sit back and congratulate yourself on being clever. But, uh, it never really works out. You’ve forgotten the fundamental truth of data science:

You can’t learn a new fact with a new method

Whenever you come to folks with a new fancy method they distrust it. So when you try to share your fascinating results with them they poke and prod the methodology and usually reject it because it’s new and hard to understands.

To avoid this you need to make the new methodology super boring. Statistical methods are magical in a way: scary and complicated when first introduced and painfully boring after folks get used to it.

So the solution is to use your method to replicate all the things you already know. Want to try a new goodness metric: verify it goes in the right direction for the last year of big launches¹. Want to try a new variance reduction technique: use it in parallel with the old one. If you don’t have anything you trust to replicate make up toy datasets and show it works on them.

By the time you finish folks will be lulled into acceptance and won’t bat an eye at your new approach: it’ll just fade into the background.

Now I’ve phrased this rather pragmatically: this is just the way to get things done. But others’ skepticism is well justified: we currently have method M implies fact F. But there’s two ways this can be consistent: either F is true or M is false (and then who knows about F). With a new method, we don’t exactly know yet if it’s trustworthy.

Hence replication. Of course, we might be able to recover some well-established facts with a wrong method by coincidence. But every successful replication makes that explanation less plausible. How many is enough? A handful usually suffice in my experience: certainly not enough to be epistemically satisfying² yet still sociologically satisfying.