JSM 2026 - Monday

A lot more existential angst today: it’s a rough time to be a statistician. The discussant of the first session explicitly asked: “in this new era with AI are we still useful?”

What seems to be the consensus is that the technical side of things is being less emphasized: implementation and execution are things that LLMs are reasonable at (to a certain level of quality: I still find LLM code tediously verbose / opaque).

The discussant gave some hope of a higher purpose for statisticians as guardians of the scientific method, arbiters of epistemics, champions of rigor. Which is certainly a role that we can play though that’s not exactly the fun part of the job and earns us the reputation of killjoys. Still definitely can see the need and it’s not like anyone else is stepping up to the job.

One thing that too many presenters are delegating is their slides. LLM slides are awful. You may or may not lose your job to Claude, but you’re never going to be replaced by someone using Claude. I hope this goes the way of clip art and gets relegated to use solely by high school students and product managers.

A/B Testing Beyond Tech: Novel Methods for a New Era of Experimentation

The title for this session was misleading: it was just a bunch of tech companies presenting.

The first presentation used simulation to test the idea of using an LLM as an experiment analyst: upload your AB test dataset and have it tell you to launch or not (or keep iterating). The main criterion was whether the LLM can manage to make decisions early enough. This is rather unfair as the mathematics of early stopping are pretty well established and there wasn’t any side information that the LLM could take advantage of.

Still the LLM outcomes were pretty reasonable: it ended up close to a Bayesian approach in terms of # of days before decision and accuracy of choice.

It’s not entirely clear what the LLMs were doing: it was presented as a black box. Basically a big prompt and presumably some agentic coding. A bit of a red flag: apparently the prompt itself made a big difference so unclear if this is real performance or cherrypicked performance. Also it’s impossible to know whether the LLM actually peeked: would be nice to see a careful explication of this especially enforcing the sequential nature of the design.

As I was listening I was feeling this was easy to dismiss: there are optimal approaches, that’s the whole point of the inference. But on further reflection I think that’s wrong: the black box nature of an LLM-as-analyst is disturbing, but it’s good to try this out and see if we can. There’s a kernel of something here. This strikes me as a useful benchmark.

The context behind a launch decision is important. The wins for the LLM relative to the “correct” methods are in the inconclusive region where the chatbot was (correctly?) more hesitant. Ideally we would encode this in the loss function but that’s hard: letting non-technical stakeholders specify their goals in prompts could be helpful to elicit this. For instance, the company is in trouble and we really need a hail mary so only huge effect sizes are useful. Or they are about to IPO and just need to avoid catastrophic failures. These are the sorts of things that a statistician can incorporate for important analyses but being able to approximately automate them could make this a more standard part of analysis.

The next talk was on Optimizing Content Acquisition. The problem is to figure out what content to acquire for YouTube. The issue of course is that (1) it’s expensive and (2) you can’t collect usage data on the content until you’ve paid for it.

Apparently the traditional approach is just subject area experts¹. But that’s really hard to scale so you need something else. So you can do all sorts of machine learning models: then you experiment to check if it’s better.

There’s complications with this experiment though. Remember you have to buy the content before you can collect data. Thus a natural design is to look at two competing content approaches and buy both: then show one set to one set of users and the other to another set. Standard AB test: except you have a problem. It’s a bad user experience if your friend has access to a movie but you don’t just because you landed in different experiments. Thus you can only ablate the content on recommendations: search should still return the results. This is an interesting problem as you presumably want to figure out what the effect of total ablation is but you only get partial ablation. Unfortunately the talk didn’t go into the solution of this and just raised the question.

The next talk was a career talk from an econometrician on revenue finance. Teams want their ROI to be bigger and so they run all sorts of studies. One thing I hadn’t appreciated before is the role of bureaucracy as a defense: without institutional support and these councils and governance boards an individual analysis will get steamrolled by the political process when results don’t work. Though he described his role as giving teams the numbers they want while still being somewhat comfortable from a methodological approach. This strikes me as unhelpful but also I tend not to have success in these sorts of things so, you know, maybe he’s onto something. Better to nudge successfully in a better direction than being a statistical Cassandra.

That said most of the “compromises” were pretty reasonable. For instance switching to intent-to-treat to accommodate the fact that sales folks were uncomfortable abandoning clients (and of course they mostly cared about the good clients). Or most studies are underpowered so switching to a Bayesian approach and presenting results in terms of bets and risk (and introduce some shrinkage).

Is Spatial Statistics Dead?

Another misleading title: it’s very much not dead and really it’s traditional ML not LLM that’s threatening.

The central problem with spatial models is that covariances are very hard to fit and so most work deals with finding nice covariances.

Almost all of the spatial statistics examples I’ve seen involve air pollution and the first talk was no different. What was interesting was how they got their covariance function: instead of the real data which is sparse and hard to measure they instead had a physical model. It didn’t have great predictions but perhaps it could be used to inform the covariance model.

The way they got it was fascinating: they parameterized the covariance with a few parameter fields (since they can vary by location). Then to learn the parameters is just fitting a neural network to fit a regression function \hat{w} = F(X) where X is the empirical covariance map. The data is interesting as it’s all synthetic data: using a fixed model class just generate thousands of realized covariance structures for fixed parameters w.

This is just Approximate Bayesian Computation! This is the thing I did my thesis on! Of course they’re just getting a point estimate rather than a full posterior over the parameters. Quite fun!

The next talk covered an interesting study on a meta-analysis of >200 papers assessing the impact of Covid-19 on air quality. It would be disturbing if it wasn’t entirely predictable: the papers were pretty bad. There’s a variety of methods but they could be boiled down to simple differences, diff-in-diffs, linear models, etc. So they ran all of these specifications and found significant effects for most of them.

Then they did something clever and just replaced the 2020 data with 2019 data. They find a significant effect of lock-downs in a year without lockdowns! So something is bad. This is particularly troubling because these presumably passed peer review. I can see some of the approaches being somewhat reasonable but can we at least agree that a simple pre-post difference in means should fail review?

Of course this is a causal problem and when you discuss it in terms of potential outcomes it’s obvious that you need to do difference-in-difference. Of course you might want to do something a little more complicated: weather makes a big difference in these measurements: you’d be missing a confounder if you didn’t condition on this. Of course that’s rather hard so you need to introduce ML models to be able to handle something that complex.