Jev and Calibration

I put my LLMs as Feature Engineering post on Hackernews and got great responses. Apparently it was fortuitous timing because unbeknownst to me a new model called Jev launched and claimed to be perfect for LLM classifications providing calibrated responses.

This is a huge claim if true. But, umm, there’s absolutely no way this is true: by construction Jev cannot be arbitrarily calibrated.

To explain calibration I’ll borrow from Nate Silver:

“One of the most important tests of a forecast - I would argue that it is the single most important one - is called calibration. Out of all the times you said there was a 40 percent chance of rain, how often did rain actually occur? If, over the long run, it really did rain about 40 percent of the time, that means your forecasts were well calibrated. If it wound up raining just 20 percent of the time instead, or 60 percent of the time, they weren’t.”

• Nate Silver, The Signal and the Noise

The easiest way why this can’t hold in general is to consider an omitted variable. Suppose we’re trying to detect scam messages and we’ve got the text “Your package was unable to be delivered. Please visit [link] to provide details”. Maybe it’s legit, maybe it’s a phishing attack, but either way any¹ calibrated classifier has to return a single number².

That’s a problem because calibration is not a property of a classifier but rather it’s a property of a classifier \(\times\) distribution pair. Consider two distributions: one where that message comes from a known source and another where it comes from an unknown source. The base rate in the first is probably >90% while the base rate in the second is probably <10%. And a single number obviously cannot be calibrated for both cases!

That said I wouldn’t judge Jev too harshly: overhyping a new model is table stakes. And I suspect³ this could be a great tool for LLM classification. Like the worst part about using LLMs as classifiers was the rigmarole of getting structured data out of the LLM which it just isn’t suited for. With Jev that’s a cinch: you get the structure without all the overhead of the LLM. And then of course you can throw those features into an actual classifier which can be calibrated!