Excellent talk by Rehan Khan from Snowflake.
The first half of the talk was interesting: Snowflake charges by consumption so any changes they make affect billing. This makes experimentation both very valuable and yet very precarious1. It’s also very difficult: he mentioned that until he had joined they had not run any successful experiments on the platform plagued as they are by a lack of scale and vast heterogeneity of units. So he talked a little bit about experimentation and some switchback designs.
But really we were all there for the second half of the talk where he discussed his views on what the future of data science will be now that we have LLMs “capable” of doing data science work. Khan’s perspective was quite valuable as it both confirms and challenges existing paradigms I’ve been exposed to.
He aptly describes this future as “McDonalds-ification”. The chief virtue of McDonalds is reliable mediocrity: it’s not “great” but you know that no matter where you are you will get a reasonable burger at a reasonable price. And it’s everywhere so it sets a floor for the culinary experience: you never have to suffer through something worse than McDonalds. The same, he argued, holds for data science now. With LLMs providing analysis it’s not going to be amazing: but it will be ubiquitous and cheap. He views the role of the data scientist to be evaluating and improving the reliability.
He got a little fuzzy at this point: how exactly do we know LLMs are good at data science and how do we improve. Most of the confidence in being able to reliably measure LLM performance comes from unit tests. As I mentioned in my earlier ephemera, this doesn’t work for data science: we have messy ambiguous problems that can’t be easily expressed by a single unit test. Stakeholders pine for a one-handed data scientist but the job really is to convey the appropriate nuance and uncertainty: not easy to judge out-of-context. He had a reasonable response when I pressed him on this point: basically trying to break down these more subjective evaluations into something more concrete. This can help a little. I do worry though that this is just a hopeless task: every interesting analysis ends up somewhere unexpected.
But maybe that’s ok: there can be a bifurcation. Not every analysis needs to be interesting: a lot of it is bog standard things like dashboards, monitoring pipelines, and recurring analyses. I basically ran a team which had a weekly experiment analysis: it was braindead work but someone had to do it. Instead of using it as a training opportunity for juniors I’d probably push for that to be automated now2. And that’s the sort of thing I can actually write a unit test for.
Indeed that’s where Khan recommends applying our expertise in the standardization and automation. Basically a four step process:
- Divide the work
- Define the skills
- Encode expertise
- Evaluate quality and improve
This brought up an association with Deming3 and quality control. Essentially quality is an outcome of the system not the result of individuals’ inherent skills. Thus our task becomes engineering the system such that it produces reliable quality results.
What I like about this approach though is that it is nicely robust to AI stagnation. Instead of waiting for the next model to solve everything we can make incremental progress and build a mindset of investigating new tech in a measured, deliberate, and coordinated manner.
This still leaves the problem of research: figuring out what to do when it’s not been done before. It’s here where I think the mainstream paradigms of individualization of tools and workflows makes the most sense. So long as the end product is understandable and portable who cares about the process. The prototype then serves as grist for the standardization mill. We constantly move from frontier to frontier until eventually there’s no need for pioneers anymore…