Wei Ping
@_weiping
Followers
1,993
Following
261
Media
9
Statuses
222
Principal research scientist @NVIDIA . Working hard on building LLMs and multimodal LLMs. Views my own.
San Francisco, CA
Joined June 2020
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
FEMA
• 1498003 Tweets
نابولي
• 642248 Tweets
2NE1
• 331219 Tweets
Joker
• 172258 Tweets
天使の日
• 132117 Tweets
Grammys
• 100633 Tweets
Mancuso
• 99639 Tweets
Song of the Year
• 63697 Tweets
名探偵ピカチュウ
• 56792 Tweets
Record of the Year
• 55487 Tweets
Best Record
• 54059 Tweets
Ülke
• 52989 Tweets
Pogba
• 46942 Tweets
AFFAIR EP6
• 45005 Tweets
Ocho
• 37543 Tweets
Bielsa
• 37442 Tweets
Trump Bible
• 37003 Tweets
Uyuşturucu
• 31611 Tweets
HAPPY 2nd BIRTHDAY SMILEY
• 25314 Tweets
Celtics
• 16699 Tweets
Jamie Dimon
• 13478 Tweets
菅野メジャー
• 12502 Tweets
Semih Çelik
• 11979 Tweets
#اوقفوا_الاباده
• 10121 Tweets
Pinned Tweet
Introducing NVLM 1.0, a family of frontier-class multimodal LLMs that achieve state-of-the-art results on vision-language tasks, rivaling the leading proprietary models (e.g., GPT-4o) and open-access models (e.g., InternVL 2).
Remarkably, NVLM 1.0 shows improved text-only
11
112
439
Introducing ChatQA-1.5, a family of models that surpasses GPT-4-0613 and Command-R-Plus on RAG and conversational QA.
ChatQA-1.5 has two variants:
Llama3-ChatQA-1.5-8B,
Llama3-ChatQA-1.5-70B,
We also open source our instruction
14
137
628
🔥Our BigVGAN is a universal audio synthesizer!🔥
Trained only on speech data but shows extraordinary zero-shot generalization for non-speech vocalizations (laughter, applaud), singing voices, music, instrumental audio that are even recorded in varied noisy environment!
BigVGAN: A Universal Neural Vocoder with Large-Scale Training
abs:
project page:
sota zero-shot performance for various out-of-distribution scenarios (new speakers, novel languages, singing voices, music/instrumental audio)
2
43
168
7
39
209
Introducing ChatQA 2, a Llama3-based model with a 128K context window, designed to close the gap between open LLMs and leading proprietary models like GPT-4-Turbo in both long-context and RAG capabilities.
The long-context capability of LLMs is sometimes viewed as a rival to
3
42
156
Introducing RankRAG, a novel RAG framework that instruction-tunes a single LLM for the dual purposes of top-k context ranking and answer generation in RAG.
For context ranking, it performs exceptionally well by incorporating a small fraction of ranking data into the training
2
39
150
BigVGAN is accepted at ICLR 2023.
Listen audio samples:
A universal audio synthesis model, trained on speech only, works for out-of-distribution scenarios, e.g., unseen singing voices and music audio!
Code and models are released!
0
34
127
🔥Introducing IntructRetro 48B, which largely outperforms the instruction tuned GPT🔥
Surprisingly, we find that one can ablate the encoder from the Retro architecture and directly use it as a GPT decoder, while still obtaining comparable results on zero-shot QA tasks!
Instruction Tuning the Largest Pretrained Retrieval-Augmented LLM
This exciting new paper from NVIDIA introduces Retro 48B, the largest LLM pretrained with retrieval.
Continues pretraining a 43B parameter GPT model on additional 100B tokens by retrieving from 1.2T tokens (using
4
94
396
2
22
118
📢 The source code and models are now available at:
Long-Short Transformer: Efficient Transformers for Language and Vision
pdf:
abs:
On ImageNet, sota results (e.g., Top-1 accuracy 84.1% trained on 224 × 224 ImageNet-1K only), while being more scalable on high-resolution images
1
58
288
0
35
108
Our model also achieves SOTA results on NLP tasks, including long range arena and character-level language modeling. The models and code will be released soon!!
Long-Short Transformer: Efficient Transformers for Language and Vision
pdf:
abs:
On ImageNet, sota results (e.g., Top-1 accuracy 84.1% trained on 224 × 224 ImageNet-1K only), while being more scalable on high-resolution images
1
58
288
0
17
105
We release DiffWave, a versatile diffusion model for conditional & unconditional audio generation. It readily matches the SOTA neural vocoder in terms of quality. More interestingly, it can generate abundant realistic voices in time-domain without any conditional information!
``DiffWave: A Versatile Diffusion Model for Audio Synthesis. (arXiv:2009.09761v1 []),'' Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, Bryan Catanzaro,
0
3
17
2
39
102
ChatQA can outperform GPT-4 on a wide range of conversational QA tasks:
- ChatQA and GPT-4 take the same top-5 chunks from our best retriever, when long documents are involved.
- ChatQA performs very well on tabular data, arithmetic calculation, and “unanswerable” cases!
Nvidia presents ChatQA
Building GPT-4 Level Conversational QA Models
paper page:
introduce ChatQA, a family of conversational question answering (QA) models, that obtain GPT-4 level accuracies. Specifically, we propose a two-stage instruction tuning
6
98
485
1
21
88
Introducing NV-Embed, a generalist embedding model that ranks No. 1 on the MTEB Benchmark, which includes 56 diverse tasks, using only publicly available data. Notably, our model also achieves the highest score of 59.36 on 15 retrieval tasks within this benchmark.
NV-Embed
1
19
85
The model checkpoints and instruction-tuning data are now available!
Nvidia presents ChatQA 2
Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities
In this work, we introduce ChatQA 2, a Llama3-based model designed to bridge the gap between open-access LLMs and leading proprietary models (e.g., GPT-4-Turbo) in long-
3
62
259
0
9
46
Accepted as an oral presentation at
#ICLR2021
We release DiffWave, a versatile diffusion model for conditional & unconditional audio generation. It readily matches the SOTA neural vocoder in terms of quality. More interestingly, it can generate abundant realistic voices in time-domain without any conditional information!
2
39
102
0
5
42
The model checkpoint has been uploaded. Enjoy!
Introducing NV-Embed, a generalist embedding model that ranks No. 1 on the MTEB Benchmark, which includes 56 diverse tasks, using only publicly available data. Notably, our model also achieves the highest score of 59.36 on 15 retrieval tasks within this benchmark.
NV-Embed
1
19
85
1
5
40
Our NV-Embed-v2, has achieved a record-breaking score of 72.31 across 56 text embedding / retrieval tasks, reclaiming the top 1 on the Massive Text Embedding Benchmark (MTEB) leaderboard!
It also holds the No. 1 in the retrieval sub-category (15 tasks) in the leaderboard, which
0
7
40
📢 Language models simply generate way more factual text using our techniques:
🔥 i) Factual-nucleus sampling -- outperforms top-p by a large margin!
🔥 ii) Continued training with TopicPrefix & sentence completion loss -- much more effective than next-token-prediction loss!
Factuality Enhanced Language Models for Open-Ended Text Generation
abs:
factual-nucleus sampling improves generation factuality at inference, combination of sentence completion loss & TOPICPREFIX pre-processing improves factuality with continued training
0
0
31
0
7
40
Long Context and RAG : Better Together
Nvidia presents ChatQA 2
Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities
In this work, we introduce ChatQA 2, a Llama3-based model designed to bridge the gap between open-access LLMs and leading proprietary models (e.g., GPT-4-Turbo) in long-
3
62
259
1
7
39
🔥Retrieval meets Long Context LLMs🔥
We demonstrate that LLaMA2-70B with simple retrieval augmentation outperforms its 32K long context base model and GPT-3.5-turbo-16k in terms of average score on 7 long context tasks!
Retrieval meets Long Context LLMs
This is an important and timely paper investigating two important trends in LLMs: RAGs and long-context LLMs.
It compares retrieval augmentation and long-context windows for downstream tasks. Also investigates if the methods can be combined to
1
145
602
0
10
37
I will be giving a talk tomorrow at GTC 2023 on Retrieval meets Large Language Model !!
This talk covers the application for open-domain QA, and 🔥Re-ViLM🔥: our latest retrieval-augmented visual language model
Welcome to join:
1
10
34
🔥We should pretrain LLM with retrieval🔥
We find RETRO outperforms GPT on generation(less repetition, better factuality, lower toxicity) and knowledge-intensive task.
Our RETRO++, largely improves original RETRO(54.1 vs. 45.5 on NQ) and GPT w/ retrieval(50.9) on open-domain QA.
Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study
Presents RETRO++, which significantly outperforms retrieval-augmented GPT across different model sizes.
repo:
abs:
5
72
327
1
6
31
Have started our ICML poster session on Non-Autoregressive Neural Text-to-Speech. Please join if you are interested!
Session link:
Paper:
0
5
28
Have started our ICML poster session on "WaveFlow: A Compact Flow-based Model for Raw Audio". Please join us if you are interested!
Session link:
Paper:
Code:
0
2
25
📢Interesting findings:
1) Fine-tuning on self-generated data is more efficient to detoxify LM, as it mitigates the exposure bias
2) Large LM has the same toxicity level as smaller one
3) Training on adapter layers achieves a much better trade-off between toxicity & perplexity
Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models
abs:
0
19
59
0
9
25
📢📢📢 Our unconditional DiffWave can do zero-shot speech denoising now!! YES, unconditional generation of speech is useful as in language and vision! Check it out in Section V on our demo website:
We release DiffWave, a versatile diffusion model for conditional & unconditional audio generation. It readily matches the SOTA neural vocoder in terms of quality. More interestingly, it can generate abundant realistic voices in time-domain without any conditional information!
2
39
102
0
7
18
🔥BigVGAN rocks🔥
🔥BigVGAN demo with
@Gradio
is now live on
@huggingface
Spaces!🤗
BigVGAN is a universal waveform synthesizer with strong zero-shot robustness for every sound of the world, which is built towards generative AI for audio 🔊
👉
0
16
73
0
2
17
Want to share our ICLR oral paper
DiffWave:
A versatile diffusion model for audio synthesis
It introduced a fast sampling method for diffusion model (trained with 200 diffusion step, sampling with only 6 step)
Project:
Talk:
0
5
15
Clearly, researchers/engineers who are building LLMs, need such therapy first. A lot is going on & no work-life balance😂
In the future, once the robustness of our models will exceed some threshold, we will have *wildly effective* and dirt cheap AI therapy. Will lead to a radical improvement in people’s experience of life. One of the applications I’m most eagerly awaiting.
283
233
2K
0
0
13
Introducing Local Knowledge Powered Conversational Agents
Paper:
By
@sazoo_nlp
@wpingnet
@TheRealRPuri
@MohammadShoeybi
@MostofaPatwary
@ctnzr
0
4
13
Welcome to our ICLR poster session on Friday, May 10th, from 4:30 to 6:30 PM. Peng Xu (
@PengXu51108979
) will present our findings on "RAG meets Long Context LLMs!"
Our RAG-enhanced long-context LLM outperforms both the long-context baseline Llama2-70B-32k, and GPT-3.5-turbo-16k
1
1
13
🔥 BigVGAN-v2 offers top audio fidelity, fast CUDA inference, and commercial checkpoints! 🤯
🚀 BigVGAN-v2 is here!
It is our latest update of the universal vocoder to benefit all audio generative AI. It features:
🎵 State-of-the-art audio quality
⚡ Custom CUDA kernel with fast inference speed
🔊 New commercial-friendly checkpoints up to 44kHz
4
37
156
0
5
12
Thank you, Nando! We are really glad you enjoyed our paper, especially after the relentless hard work and dedication we put into it. 😃
The NVLM paper is outstanding. It is full of remarkable findings: (1) "dataset quality and task diversity are more important than scale", (2) positive transfer from multimodal datasets to text-only on math benchmarks, (3) model and data ablations, etc. Congrats to the authors on
3
19
220
0
0
14
We are excited to release ChatQA-2 (and its training data!), 128K long-context models that also have exceptional RAG capabilities for efficient inference or to handle inputs significantly longer than 128K tokens. The ChatQA-2 70B model outperforms GPT-4-Turbo-2024-04-09,
0
9
12
We release our code and model at:
Speech Denoising in the Waveform Domain with Self-Attention
abs:
project page:
0
13
76
0
2
11
Can't wait to test its RAG capability 👏
Announcing C4AI Command R+ open weights, a state-of-the-art 104B LLM with RAG, tooling and multilingual in 10 languages.
This release builds on our 35B and is a part of our commitment to make AI breakthroughs accessible to the research community. 🎉
2
61
231
0
1
11
We shared valuable insights on how to build cutting-edge multimodal LLMs, covering aspects such as architectural design, data curation, tagging high-resolution image tiles, and pushing toward state-of-the-art results in vision-language tasks, all while maintaining or even
Introducing NVLM 1.0, a family of frontier-class multimodal LLMs that achieve state-of-the-art results on vision-language tasks, rivaling the leading proprietary models (e.g., GPT-4o) and open-access models (e.g., InternVL 2).
Remarkably, NVLM 1.0 shows improved text-only
11
112
439
0
1
11
This is why it's important to value the publication of negative results.
The Megatron-LM paper investigates the scalability of the BERT model, as detailed in Tables 4 and 5. It empirically demonstrates that accuracies for downstream tasks tend to plateau.
Lazy twitter: A common question in NLP class is "if xBERT worked well, why didn't people make it bigger?" but I realize I just don't know the answer. I assume people tried but that a lot of that is unpublished. Is the theory that denoising gets too easy for big models?
46
41
481
1
0
11
We use hyperlinks from online conversations as a source of knowledge, augmenting dialog history to create higher quality text generation. The output from our 8.3B parameter model is difficult to distinguish from true conversation in human evaluations!
Introducing Local Knowledge Powered Conversational Agents
Paper:
By
@sazoo_nlp
@wpingnet
@TheRealRPuri
@MohammadShoeybi
@MostofaPatwary
@ctnzr
0
4
13
0
3
10
Knowing the right thing to do but not adhering to it makes us no different from those who are clueless, in the eyes of the world.
0
0
8
Skip pre-training and go directly to SFT.
NeurIPS 2024 will have a track for papers from high schoolers.
78
90
599
0
1
8
@YoavR7
I have already moved to Nvidia, so I could not make the decision. As far as I know, Baidu has a tentative plan to open source ParaNet in Parakeet toolkit. You can watch on this repo:
0
1
6
@JefferyTatsuya
Yes, the maximum context is 8k for now. If your documents are longer than 8k, you can use the uploaded retriever to obtain the top-5 or top-10 retrieved contexts, which have proven to be highly effective.
1
1
6
As an open model, DeepSeek-Coder-V2 by
@deepseek_ai
is truly amazing at coding and math. I really enjoyed reading their series of reports, which provide extensive technical detail 🫡
[Chatbot Arena Update]
We are excited to launch Math Arena and Instruction-Following (IF) Arena!
Math/IF are the two key domains testing models’ logical skills & real-world tasks. Key findings:
- Stats: 500K IF votes (35%), 180K Math votes (13%)
- Claude 3.5 Sonnet is now
#1
8
87
425
0
0
6
I tend to think both RL and Gumbel trick can work well if they are well executed w/ a reward function trained on massive human annotations. However, straight through/Gumbel trick give less noisy but biased gradient, so RL could win in the long run (my guess)
I don't understand why "RLHF" even needs RL? The reward function is a learned neural network and thus white-box. This means we could simply use straight through estimater (or Gumbel trick) to obtain a much better gradient.
(context: my understanding is from InstructGPT paper)
27
75
760
0
0
6
Flowers will eventually bloom as people work hard and maintain great patience. 静待花开终有时. 🫡
0
0
5
NO forms! Apache license!👍
New open source Flan-UL2 20B checkpoints :)
- Truly open source 😎 No forms! 🤭 Apache license 🔥
- Best OS model on MMLU/Big-Bench hard 🤩
- Better than Flan-T5 XXL & competitive to Flan-PaLM 62B.
- Size ceiling of Flan family just got higher!
Blog:
51
343
2K
0
0
5
Thanks!
@ravithejads
💡ChatQA: Building GPT-4 Level Conversational QA Models
Builds ChatQA conversational QA models, that obtain OpenAI GPT-4 level accuracies.
🌟 Key Contributions:
1️⃣ Two-Stage Instruction Tuning Method:
- Stage 1: Supervised Fine-Tuning on a mix of instruction-following and
3
15
63
0
0
4
8x22b, mixtral strikes again 🤯
magnet:?xt=urn:btih:9238b09245d0d8cd915be09927769d5f7584c1c9&dn=mixtral-8x22b&tr=udp%3A%2F%%3A1337%2Fannounce&tr=http%3A%2F%%3A1337%2Fannounce
272
821
6K
0
0
3
@lucidrains
1) Architecture. DiffWave uses compact WaveNet-like architecture to support both conditional & uncond generation. WaveGrad uses Upsample and Downsample blocks for mel-spectrogram inputs. 2) In addition to neural vocoding, we tackle the challenging unconditional generation task.
0
0
3
This can make us feel better:
Teaching LLM to say “I don’t know” when it want to hallucinate, is quite a bit of work too
Illusion of knowledge is more dangerous than ignorance:
It's Okay to say "I don't know" and admit that you don't know it.
It's shameful to pretend that you know everything.
90
1K
5K
0
1
3
@AkariAsai
Looking forward to it! We retrieved 1.2T tokens in InstructRetro and are eager to see the results of further scaling. Do you have a timeline :)
1
0
1
@Yuchenj_UW
@DrJimFan
We didn't find the checkpoint of Qwen2 VL 72B when we are tying to evaluate and compare last month. It seems it was just released now:
1
0
1
@srush_nlp
BERT is trained using a denoising auto-encoding objective. I vaguely recall there being some established connections between denoising auto-encoding and lossy compression e.g.,
1
0
1
@Francis_YAO_
Not to mention, some open weights models are released with benchmark numbers but without prompts😅
0
0
1
@heiga_zen
Same here! I really like the theme of the story: family. My son is only 2 yrs old. I guess I still need wait a couple of years before he could enjoy the movie :)
0
0
1