Notes on JSM 2026 - Sunday

It’s been a while since I’ve attended JSM and it’s great to be amongst statisticians. The discussions were fruitful and folks were cheery. Boston is a fun city to walk through and its restaurants have not disappointed yet.

Both sessions I attended today kept returning to the same idea: modern statistics isn’t primarily about deriving clever estimators anymore. It’s about understanding where data comes from and how it can mislead you. Whether the data comes from hospitals, smartwatches, surveys, or LLM evaluations, the statistical challenge is increasingly one of understanding what it is we’re measuring rather than computation.

Statistical Design and Inference in the Era of Ubiquitous Data

I was quite excited about this session as I view this as the main quest of our era in statistics. Our intellectual forebears had to develop methods to wring as much as they could out of the data they had. Now we’re inundated with massive amounts of data; most of it being pretty low quality. We’ll need to think hard about what is justifiable: our methods won’t tell us any more (as our p-values → 0).

The panel did not disappoint.

The problem is that this ubiquitous data is collected without intention¹: it’s just there. As statisticians we have the expertise and, perhaps more importantly, the mindset to contribute to making sure this data makes sense.

As always, survey statisticians got there first: they have a framework for Total Survey Error accounting for all possible flaws from data collection, coding, and analysis. Brady West has been writing papers on extending this into a Total Data Quality framework that applies much of the same thinking more broadly.

The useful approaches were to treat the data in a design-based mindset: “can we uncover a hypothetical experiment” in the data? Or maybe “what would have wrecked my experiment in the data”. The example of some treatment was given: certain hospitals have different resources so treatment is varied.

Missingness was another framework: with non-designed data what even does missingness mean? You need a design to even know there was data there to begin with! Jessilyn Dunn showed some interesting smartwatch data and the data issues are huge: there’s bias in who owns a watch, when they wear it, what algorithms get used (and updated), and why folks wear it. And of course imputation then plays a role both in what we can correct for and identifying when we cannot correct.

I don’t think we can do nothing here: there’s a ton of problems and most folks are unaware of them. Sitting and truly understanding the data is necessary to avoid making terrible decisions here. AI will only make it worse: you’ll get a result and it’ll even seem reasonable, but only because you aren’t intimately aware of the data and its limitations. And of course you have the LLM training itself: very rarely is the data described with regard to selection, quality, bias, etc. It’s sort of a big unclear kitchen sink.

Synthetic respondents seem to be getting popular: that’s going to be as ubiquitous as we have compute for. This is the next evolution for surveys: starting with high response rate probability samples to low response rate probability samples to nonprobability samples and now we’ve given up on asking humans anything so synthetic samples it is!

That said, on this long decline we’ve been able to get some really nice tools and it’s likely we can continue to do so. For instance PPI is a great fit here: you can use your ubiquitous synthetic survey responses as the estimator and get a small high quality probability sample to debias it.

To build on that, notably, we aren’t just stuck with observational data but we can also do our own data collection and experimentation. In particular the panelists pointed me towards Multitrait-multimethod matrices which let you decompose the total variance into validity, systemic error, and random error terms by measuring the same latent trait in multiple ways.

Finally sometimes we just have to insist that the data at hand can’t answer the question we pose of it. For that to work we need education: especially across disciplines. Non-experts are going to be doing ever more analysis. I’m editorializing here so burn me instead of the panelists²: but instead of teaching undergrads about t-tests we should be giving them a course in experiment design. I mean we can’t really get folks not to abuse the statistics we already teach them but at least we’re fighting the good fight.

I also liked the chair: to encourage questions from the usually shy crowd of statisticians she handed out medals to the first three questioners. I’ll need to borrow that for my next talk.

Measuring, Evaluating, and Driving Product Outcomes at Google

The first thing to say is that the session chair was brilliant. But yeah it was a great session.

Chris Quale led off with a talk “All Analyses Are Wrong, But Some Are Useful: In Pursuit of Being ‘Least’ Wrong” which deals with some of the softer sides of data science. Data science is changing and to stay relevant we need to double down on what we’re doing that’s valuable. And that is…

Data scientists should be the best equipped profession to reason about data with error and leverage that to make statements about the future.

Now I don’t agree with all that: we’re also quite interested in statements about the past and the current, but largely yes. I think our ability to think in terms of data generating processes is important. Coming right off the previous session this felt like a confirmation.

Of course explaining all this to non-statistical stakeholders is the tricky thing. Chris mentioned the importance of trust and relationship building which sitting in the audience I would have scoffed at but now find is really the secret to success in industry.

Also he recommended Veridical Data Science so I do need to read that.

Grace Deng followed up with “Evaluating Agents with Statistical and LLM-based Methods”. This is some fascinating data to work with: quite messy as it’s multimodal, sequential, and very unstructured. I’d love to hear more about the approach: Grace uses a great deal of LLMs to evaluate the data, turning it into more structured results (autoraters for satisfaction, topic models to understand trends, etc). This is my favorite thing about working in such a large company: we have so many things going on you get to find out about all the time.

The trickiest aspect seems to be that things change all the time. Like you get a new model release and now all of your ratings and topics might change. That’s annoying but also helpful. For instance suppose you find the model change now flips your conclusion: perhaps it wasn’t so strong to begin with? There’s an interesting connection with the garden of forking paths where you might have a couple different models or prompts try to answer the same question. If they agree that’s good and if they don’t you can probably not pay much attention to them. The one thing that bothered me was that I didn’t understand how any notion of ground truth came in. But that’s probably because I had to leave to deal with an audio spillover which also meant missing most of Anshul Bazaz’s following talk on “Meaningful Playtime Install: A Precision-Recall Framework for Predicting Long-Term Gamer Value”.

Back and no longer hearing the booming sounds of the next room over, I watched my colleague Henry Pascoe present on “Statistical Error Correction for Trustworthy LLM-Powered Metrics: Unifying Two Approaches to Slash Costs and Boost Quality”. This has been fascinating work which shows that two approaches: PPI (again!) and double sampling are essentially doing the same thing.

Both address the question of how to align human and LLM ratings of a video. It’s pretty appealing to use a LLM to rate videos but we might worry that, as Grace mentioned, there’s some model change which affects the performance. Thus we need some way to anchor and we can collect a small sample rated by both humans and LLMs and use that to extrapolate out.

Funnily enough it seems like double sampling was the more popular of the two amongst engineers. I personally like the GREG formulation especially since it makes more sense in more complicated settings where you want the full regression toolbox.

Finally Tim Huang talked about “Statistical Frameworks for Measuring AI’s Environmental Impact at Google Scale”. This was very messy data and very fiddly data. It’s more of a comprehensive accounting of all sorts of sources of energy: how much it takes to make the hardware, run and operate it, the effects of the power generation, and the eventual decommissioning. Lots of individual parameters to estimate and lots of measurement to pioneer. And of course things are changing all the time and in confounding ways (such as new hardware getting more intense workloads). Very detailed and impressive work: this is a good showing that a lot of statistical impact is not sophisticated modelling but rather very careful accounting of uncertainty across many disparate and messy inputs.