I apologize for not giving him credit in my original post, but please follow @kirill_vish, the first author of this work.

@i000 It means that the loss will be very noisy. It's difficult to learn from noisy signal but not impossible.

@i000 I agree. But I think it's still possible to train good DNA language models, just more difficult.

@Throrf I don't think it's true. We have hundreds of thousands of species sequenced, and millions of genotyped people. I think we mostly just need to get better at training and evaluating these models.

@felixchin1 Good question! Evo was only trained on prokaryotes. ESM3 is a protein (not DNA) language model. Protein language models are still a much more powerful technology, but I hope it will eventually change.

The preprint:

Thank you Anthony for enriching the discussion! Counterexamples always exist, but from my experience it's rare to have a model that ranks samples well overall while providing no information at the tails of the distribution. I agree that your final metric should match the use case you truly care about, but ROC-AUC is usually the right place to start.

Bonus: a proof that the two definitions of ROC-AUC are equivalent.

Choosing a problem to work on: Level 1: Look around to see what's popular right now. Level 2: Try to foresee what will become trendy in a few years. Level 3: Just do what your best judgement tells you is important, trusting that everyone else will eventually come around.