When we began expanding FineWeb into multilingual space 🌎, we quickly noticed a lack of established benchmarks. Thus, we created FineTasks with two goals in mind: A. To encompass various LLM capabilities (knowledge, reasoning, etc.). B. To provide a reliable signal 📈.

@gui_penedo @Thom_Wolf 🤨 How much is that in FineWebs?? That's the only unit I can image.

Thanks for the AIME25 results

@lvwerra Le Onardo could secure even the future Italian funding tho

This is @karpathy 🐐 grade content on distributed training!

@420_gunna @vwxyzjn @natolambert @hamishivi EM == string Exact match, usually it's rather quasy exact match tho

@ryanmart3n @NeginRaoof_ We haven't uploaded re-extracted answers yet and yes we re-extracted the numina1.5

@vwxyzjn @natolambert @hamishivi 🤗 When I switched from EM to math-verify we hoped by about 15% on most models

There is pretty much nothing new in 0.5.2, compared to 0.5.1😅, so that won't help. I am not sure what you sourced your question/answers from, but at least for Numina I noticed that a lot of ground-truths are in very malformed latex, so one way to increased the recall is to re-extract the answers with LLM. We have done that but haven't run the final ablation yet.

@vwxyzjn @natolambert @hamishivi You should switch to math-verify especially because of em on gsm8k sucks

@natolambert @vwxyzjn @hamishivi What are you using now for evals?

RT @anton_lozhkov: LLM Reasoning labs will be eating good today🍔 We commandeered the HF cluster for a few days and generated 1.2M reasonin…

@e__honig Just my copy+paste issue. Math-verify will convert it to sympy.Number(41)

@Teknium1 I don't mind to review it, but won't do it myself. Should be literally two lines of code tho

RT @LoubnaBenAllal1: We just published the second OpenR1 update with OpenR1-220k-Math, our new large-scale dataset for mathematical reasoni…

@Noahpinion Not true at all; the biggest moat is the inherent byproduct that you need to obtain during the creation of LLM: infrastructure and data.

@zsytony 🫡 god-speed

@JFPuget Yeah I read that there non-contaminated* and my first comment

Why non-contaminated* ? See this absolutely awesome analysis 👇