Congratulations @jiawzhao on an excellent PhD defense! Jiawei has been a pioneer in hardware-efficient training. When he started his PhD, everyone was focusing on inference efficiency, and training runs were small, Jiawei took the bold step to pursue training efficiency. Slides: Jiawei's latest work Galore has led to huge reduction (>80%) of memory requirements for pretraining LLMs. It is based on a principled approach of projecting the gradient to a low-rank subspace while keeping the weights as full rank. Jiawei also worked on reduced precision training in logarithmic number system (LNS). The LNS is extreme quantization: instead of a standard floating-point system with exponent and mantissa, LNS only has the exponent part. Jiawei proposed a multiplicative update method to directly train in LNS. This will impact training on smaller edge devices: which is not yet mainstream today. Jiawei first started working on signSGD: surprisingly, just communicating the signs of gradients in distributed training is sufficient for accuracy while significantly reducing bandwidth requirements. A variant of it (Lion optimizer) is now used for LLMs. Jiawei has also done in-depth work on initializing neural networks with just zeros and ones, instead of random initialization. He has shown that this can help with better reproducibility while preserving accuracy. Congratulations @jiawzhao