Training LLMs can be much cheaper than previously thought. 0.1 million USD is sufficient for training LLaMA2-level LLMs🤯 While @OpenAI and @Meta use billions of dollars to train theirs, you can also train yours with much less money. Introducing our open-source project JetMoE: A thread 🧵

@krishnakaasyap source is from Huawei employees. BTW in terms of FLOPS they already catched up. But the communication is still a little behind NVIDIA

@simone_m_romeo yes. after all this is a research preview

@TsingYoga And the DPO version’s weight seems broken. It outputs random words

@TsingYoga I see. That makes sense

@JoJrobotics @luisbrasroque they can reason but not generalizable enough

@Caffeinix_alche The tasks won't be too long but are sufficient to give o1 a 0 score

@srivatsamath We will release and open-source a model that significantly outperforms o1 in computer-use agents and release the benchmark at the same time. Stay tuned

@gauranshsoni also almost 0% because their pre-training data does not contain sufficient long-horizon interactive computer-use decision making data

RT @tom_doerr: MeloTTS: A text-to-speech library supporting English, Spanish, French, Chinese,Japanese, and Korean, with various accents an…

@CongyueD @iclr_conf DDL TQL

@davidbau Consider this one, which democratizes Large model training and make it accessible to many research labs. Website: Paper:

Many people think @xai's 100K GPU cluster is no longer necessary given @deepseek_ai's success with only 2K GPUs. That is not true. The fact is that compute is always limited. If you have 100K GPUs then you can do a lot of LARGE experiments very QUICKLY, then iterate the model very fast.

@Alibaba_Qwen Will definitely grab a coffee with you when I’m back in Hangzhou!

BTW here is a comparison between DeepSeekMoE and JetMoE with similar parameter count. See Table 3 in this screenshot

@ZiqiPang Neither one. We should instead train an agentic one - it should do some bold/risky stuff that big companies like OpenAI won't release due to safety issues