There was a throwaway line in Boris Cherny’s interview at the Computer History Museum that caught my attention. It’s pretty well known that he’s not writing code anymore: Claude does all that for him. But then he mentions that he gets Claude to do the data analysis for him. I’m quite on board with the first bit; but I think this second is a bridge too far.

Of course, I would say that; my job is data science. Except I kind of hate data analysis. Not all analysis necesssarily, but I find routine data analysis incredibly boring and I too would love to automate it away. And I’m not particularly afraid of career obsolescence: formulating interesting problems and asking the right follow-up questions is a hard-fought skill that I suspect will still be valuable. Indeed, if I can automate the grunt work, I can perhaps command even higher compensation as I can get more leverage for myself and it’s more interesting work to boot.

But, unfortunately, it just doesn’t work quite yet. The problem is that statistics means never getting to say you’re right. You have to work with inherent uncertainty. There are no facts of the matter. All (reasonable) methods are valid in some situations and not others. Everything can be recast with a new understanding.

This does not lend itself well to the kinds of things LLMs are good at. And indeed it’s even harder because of Twyman’s law, “Any figure that looks interesting or different is usually wrong” and the LLM loves to find you something interesting. And not just interesting, but consistent with your hypothesis. Sometimes it p-hacks towards it, or even just makes it up. You don’t have the guardrails that regular code has: there are no unit tests you can run. You don’t have observability systems detecting issues in prod. You barely even know whether you can trust your input data, let alone whatever analysis you’re doing. This is a very hard problem.

I’ve yet to find a good one-shot analysis of anything that’s non-trivial. But it’s not obvious unless I look into the code and figure out exactly what’s going on. Nate Silver has similar observations, writing of his own work:

we’re probably still a ways away from when a one-shot prompt on Claude or ChatGPT will be able to build a complex Silver Bulletin-type model on its own. Indeed, “vibecoding” feels like a misnomer in this context. The LLMs very much rely on my expertise: I’ve probably literally given them a short novel’s worth of context for each model. … I need to give the LLMs a lot of direction — without it, they tend to fuck up.

But, of course, you really do need that expertise to be able to tell when an analysis goes wrong and you need to know what exactly the code is doing. And we’re putting these tools in the hands of non-technical folk who really want answers.

Will this be the end of the world? Probably not? I’m not worried about Boris using Claude to analyze Claude Code. He’s got his own skin in the game and the consequences are contained. Who I’m more concerned about are the scientists were already struggling to apply statistics. And there’s always a sense of cargo culting with statistics: whatever you need to publish is what’s done without necessarily understanding it. With a tempting one-stop shop I guarantee it’s going to be taken.

To avoid disaster, we’re going to need to make statistics more understandable such that folks don’t have to outsource it to Claude and even when they do they know enough and can observe enough to judge whether it’s acceptable. And I think we have a good shot. The thing to remember, though, is that analysis code is at the wrong level of abstraction. Because the interesting thing is that you can really specify an analysis in a sentence or two of natural language. All of the code complexity is incidental, not intrinsic. You just need a way to express these primitives. meterstick is a good example, ggplot is another. Are they less flexible than raw pandas or matplotlib code? Yes, sorta. But generally, if you can’t express it in those, you’re either doing it wrong, or you’re doing something very advanced, such that (1) you want to be very careful about it, and (2) you could always package it back into those libraries as another primitive.

This is actually something LLMs are decent at. Please take this description and summarize it with those abstractions. Then at the end we have a nicely reviewable code snippet that you might be able to train non-data-scientists to understand? Or at least it’s easier for me to dive in and review quickly!