Notes on JSM 2026 - Wednesday

Final day! This was a rewarding conference: definitely regret not getting out here more in the past years. There’s lots of interesting applied problems that statisticians are working on enabled by this new paradigm of large-scale semi-reliable data using LLMs. It’s exciting to see all this work and the importance of a statistical perspective on it. In some respects statistics missed the deep learning boat insisting on keeping parametric models and standard approaches. I don’t think we’re doing the same thing with generative AI: yes we’re skeptical but that’s helpful as we’re incorporating that power with the appropriate safeguards. Quite a fun time to be a statistician.

Generative AI in Modern Statistical Learning

Edgar Dobriban started us off with risk-controlled post-processing of decision policies. LLMs have some problems with expense. Thus you get all these routers: if a cheap model will do send it there and only send the expensive model the prompts that it benefits. You can cast this as a statistical decision problem. The setup is keep the chance of a bad decision below some epsilon at minimum cost. For each prompt you estimate how much better the expensive model’s decision is likely to be, and only route to it when that expected gain clears a threshold. The thresholds come from plugin estimates calibrated with conformal risk control, which is what buys you the finite-sample risk guarantee.

Xihong Lin discussed using synthetic genomic data for statistical inference. We have amazing generative models for phenotypic data and so it feels like we can use it to impute responses where we haven’t measured them. But of course they’re not perfect: can we make sure results still robust even when the model is wrong?

It was nice to see discussion of the full taxonomy of estimators: naive \(Y \sim X\) (biased), \(\tilde{Y} \sim \tilde{X}\) on synthetic (also biased), an “assisted” \(Y \sim X + \tilde{Y}\) that adds the synthetic prediction as a surrogate, and an “augmented” one that fits \(\tilde{Y} \sim \tilde{X}\) then corrects it with \((Y \sim X) - (\tilde{Y} \sim X)\) on the labeled data. Interestingly the augmented estimator takes an AIPW form, so it’s doubly robust: even if the generative model is wrong it won’t bias your estimate.

The final talk was Yaqi Duan on “Toward a Statistical Perspective on LLM Post-Training” which frames post-training as a statistical design problem under limited, structured feedback. The main problem is that your RLHF can be misaligned with your likelihood loss. Basically you’re hill climbing and the steepest ascent on your likelihood is pointing in one direction while your RLHF is informative about a different direction. Their method PILAF explicitly aligns the two.

A Constructively Skeptical View of AI

Giles Hooker had an excellent talk on interpretable ML: basically whether we’re actually learning anything from these methods. And it’s a little depressing: we don’t really know much and what we do know is fragile. Interpretable models vary greatly and can be misleading, counterfactual explanations¹ also vary greatly and break when they get Goodharted. Apparently Shapley values have their own problems with stability.

The only happy moment (only for me) was when he pointed out the same extrapolation problem with permutation importance as I had found in my own post. Indeed he goes a little further: you can actually exploit this problem to do nefarious things. For instance, we don’t want recidivism models to discriminate on race so you’d want the permutation importance or partial dependence plots to be innocuous. Turns out that you can mess around with the extrapolation region to mask discrimination on the actual data distribution.

The next talk was Frauke Kreuter discussing experiments in using LLMs to try to reproduce the empirical results in a paper. The idea being that it saves reviewers time: if the LLM can’t figure out your data and methods section it saves your human review time. This is a great idea and, assuming the false positive rate is low, a promising one: we’ll need to develop more of these automated checks to deal with the increased volume from LLMs. The results are a little sketchy though: the LLM recovered an effect size in 169 of 180 studies, and of those it landed within ±0.05 of the original Cohen’s d in about 24% of cases but reached the same qualitative conclusion in about 80%. I feel like reaching the same qualitative conclusion is too low of a bar: ideally you’d get exact replication. Though of course it’s not clear whether that’s an LLM execution problem or a human methods description problem.

Award Ceremony

The awards ceremony was lovely; great to see my two alma maters represented: Amy Herring from Duke winning the Elizabeth L. Scott Award and Larry Wasserman giving the COPSS lecture.

Larry’s talk, “Causal Inference Meets Optimal Transport”, was interesting, connecting optimal transport and causal inference. Optimal transport is the problem of finding a map which transforms one distribution into another. The optimality comes from minimizing a particular norm: the Wasserstein distance² (also known as the earthmover distance) and the optimal map just transforms the original distribution into its quantiles and then takes those quantiles and turns them into the target distribution’s values. Of course it gets much more complicated with higher dimensions and weirder structures: indeed computation seems to be a major challenge here.

The reason why you would want to do such a thing is that it helps you reason about distributions. The connection to causal inference comes from trying to estimate not just average treatment effects but rather the distribution of treatment effects. Like the parallel trends assumption behind difference-in-difference estimates you might consider a parallel transport assumption which transforms the distributions.

This is, as Larry admits, a bit sketchy. There’s a lot of assumptions here and it’s not clear anything is testable. In particular it wasn’t clear to me why the optimality was important: certainly it identifies a map but it’s not clear why we would choose Wasserstein over any other potential norm. Certainly the optimality seems to complicate the computation: otherwise we could do something like normalizing flows which are more convenient to fit.

Nevertheless it was a good finale to my week at JSM and reminded me of my first year grad classes from Larry. Sitting in the big hall surrounded by statisticians learning about some new statistics theory: definitely my happy place! Looking forward to going to Chicago for next year!