New study! We compared ChatGPT responses to people's medical questions with those of doctors. Healthcare professionals preferred ChatGPT 79% of the time; as more empathetic and higher quality.
I'm excited to figure out how to use LLMs to help doctors!
NLP research over the years:
1990s: specialized features for NER
2000s: non-parametric prior for unsupervised parsing
2010s: embeddings + a forward/backward, inside/outside RNN
Today: When you say “Super pretty please” to the LLM, performance goes up
I am thrilled to announce that I will be the Director of Research (Foundations of AI) for the
@JohnsHopkins
AI-X Foundry, the new university-wide org for Artificial Intelligence.
@HopkinsEngineer
#BloombergGPT
Chronicles are now live!
@TechAtBloomberg
released a detailed narrative of our experience training a 50B parameter
#LLM
. We hope this helps other groups building large
#AI
/
#NLProc
models. Let’s look at some of the highlights. 🧵
A few weeks ago, we published our
#BloombergGPT
paper on arXiv. It's now been updated to include the training logs (Chronicles) detailing our team's experience in training this 50B parameter
#LargeLanguageModel
. We hope it benefits others.
#AI
#NLProc
#LLM
Students often ask: How do you review PhD applications? What do you look for? Tell me the secrets behind the process!
I made a video last year explaining how it all works.
To all PhD applicants: good luck with your graduate school search this year!
@JohnsHopkins
Engineering
@HopkinsEngineer
is making a massive investment in AI. The plan is so ambitious and unique that it’s worth breaking down the announcement in detail.
This will transform the landscape of engineering research and education. 🧵
🚨Summer AI Research Internships🚨
The
@TechAtBloomberg
office of the CTO is looking for summer AI research interns! ML, NLP, Vision, LLMs
Spread the word.
Interested candidates should apply here:
We (
@jhuclsp
) are hiring faculty in Computer Science! Spread the word.
All areas/ranks, and especially candidates currently serving in faculty or industry research positions.
We also have an "early action" option. It's a unique ad, so it's worth a 🧵
1) you should not be working on research problems that will be solved by the next large language model release.
2) it is difficult to know what those problems are.
I’m delighted to have been part of this amazing team that spanned
@Bloomberg
’s
#AI
Engineering &
#ML
Product and Research group in CTO. This was a thrilling and exhausting experience, and we’re excited to share with the larger research community what we learned.
Author feedback: A weakness of this review was that it was generated by (what appears to be) chatGPT (gpt-3.5-turbo.) We suggest the reviewer use newer models, like GPT-4o or Calude 3 Opus, and evaluate the efficacy of multiple models for review generation.
In the spirit of First Day of School pictures.
I can't believe this is really semester
#30
at
@jhucomp
@HopkinsEngineer
. Despite having done this for a while, I'm really excited for this school year.
Really interesting. When we used RNNs (LSTMs), a persistent issue was that they would lose information in the middle of the sequence. Tokens at the start and end had a large effect, while tokens in the middle were lost. It's interesting to find Transformers having the same issue.
Lost in the Middle: How Language Models Use Long Contexts
Finds that performance of LMs is often highest when relevant info occurs at the beginning or end of the input context, and significantly degrades otherwise
Congratulations
@partha_p_t
! Well deserved.
We went to grad school together at
@CIS_Penn
, shared an adviser (
@earnmyturns
) and office. We've published (and failed to publish!) together. Partha has been a fantastic collaborator, colleague and friend.
Thank you to
@OpenAI
chatGPT for your valuable input and reviews regarding our submission. We will incorporate your randomly generated advice in the camera-ready version of our paper.
🤦♂️
Applying for a PhD in
#NLProc
(CS/ECE (STEM))?
What goes into my research statement?
Publications before I apply?
Email professors in advance?
Admissions are stressful! You have questions! For answers, watch my PhD Admissions AMA.
#admissions
Good news for AI in Baltimore!
"Baltimore region selected as federal tech hub for artificial intelligence and biotechnology"
Adding >1000 people to our Engineering school over the next few years should help 😄
I had a fantastic time returning to my PhD alma mater
@CIS_Penn
for a talk today. It’s been a while since I visited. I may have found my *old* office couch.
Thanks for hosting me!
Current state of AI: a friend was using a customer service chat and wasn't getting anywhere.
Friend: Can I speak to a live agent?
Chat: I am a live agent.
My takeaways:
1) General public believes AI is this advanced
2) LLMs will be integrated everywhere
Faculty (me) can be really bad at time estimates.
Me: Train the model on this data. Should take a few hours of work.
Student: 😐That will take 2 weeks.
Me: The budget documents are going to take me hours to complete!
Admin: 😑It's literally 5 minutes of work. Please do it.
This is a lovely and unexpected surprise! Our article on using AI Chatbots to answer patient questions was the most viewed paper of 2023 at
@JAMAInternalMed
!
Almost 250k views!
I am honored to receive an Optum Labs Research Award for the use of large language models for generating synthetic clinical records to reduce bias. We are very excited to start the project and thrilled about the recognition.
ARR needs your help! We received 5800+ submissions for the June cycle, but with our current capacity, we can handle only half of these submissions. We can't start the review process without significant help. (1/2)
JHU is
#1
in research spending for 44 years in a row!
I can only claim credit for the past 15 years. The 29 years before that were all achieved without my help.
🤣🤣🤣
TLDR:
@JohnsHopkins
is a fantastic place for research.
If you write a paper review and provide a list of weaknesses, and the author's response addresses those weaknesses, raise your score. Otherwise, you need to better explain your concerns.
Me in grad school: If we use the biggest machine, and are careful with our code, I think we can use a model with 1 million parameters.
Me now: let's keep it simple and use the 70b parameter model.
Great thread! Building training sets for LLMs is challenging and time consuming. It was a major part of BloombergGPT.
This will become increasingly important. The key differential between LLMs will increasingly become data.
Just decided to make a thread to show how many decisions and pre-processing steps you need to train large language models (LLMs) such as LLaMA. This is mainly based on CommonCrawl (CC) dataset and the pipeline used in LLaMA to generate its dataset. 1/n
Beautifully said.
A major focus of our PhD admissions process is to find those who have not just been successful so far, but who will be most successful in the future. We want those who have made the most of whatever opportunities they have had so far.
💎It’s PhD admission season!💎How do you make admission decisions?
Instead of the typical top pubs criterion, why not look for and lift up “diamonds in the rough”? The reward is so much bigger when you uncover diamonds who start shining *because of you*.
I'm honored to give an invited talk today about BloombergGPT from
@TechAtBloomberg
at the Symposium on Large Language Models for Financial Services (FinLLM )
@IJCAIconf
#ijcai2023
The talk is online, sorry I can't join in person!
An absolute privilege to speak today to the
@WHFellows
about AI risks and opportunities with
@munmun10
,
@lrvarshney
, and moderator Cara LaPointe.
@Surgeon_General
spoke about technology, mental health, and adolescence, a great intro to our topic!
JHU will hire ✨80✨ NEW tenure track faculty focused in Artificial Intelligence. An additional ✨30✨ new
@JHU_BDPs
will connect AI with areas throughout the University. These 🌟110🌟 new AI-focused faculty may be the largest investment in AI by a University. Ever. 😲
Congratulations to my PhD student Keith Harrigian on defending his
@JHUCompSci
thesis "Towards Robust Natural Language Processing to Promote Health Equity".
Thank you to committee members
@anqi_liu33
and
@2plus2make5
Two sides of being a CS professor:
At work: advise on the development and application of transformational technologies.
At home: can you help me fix my printer?
Congratulations Dr. Elliot Schumacher.
@elliotschu
Me: what have you been up to in the few months since you defended?
Elliot: I posted three new papers to arxiv.
I’ve got reading to do!!!
Hi
#neurips2023
folks 👋
@HopkinsEngineer
is growing dramatically in AI/ML/data science. 80 new faculty, 30 chairs, and many other opportunities. May be the largest engineering hiring initiative in history!
We're hiring in all areas and all ranks. Join us!
@JohnsHopkins
Engineering
@HopkinsEngineer
is making a massive investment in AI. The plan is so ambitious and unique that it’s worth breaking down the announcement in detail.
This will transform the landscape of engineering research and education. 🧵
CSStipendRankings: PhD Stipend Rankings
Happy to see that
@JHUCompSci
ranks
#4
overall on "After Fees & Living". Ensuring our graduate students are well supported is a priority in the department. Baltimore is also by far the biggest city in the top 5.
🚨 Postdocs at
@jhuclsp
!
We're interested in several topics, including:
- Explainable AI/NLP for medicine
- Clinical NLP
- Evaluation of LLMs
Apply:
Spread the word!
#ACL2024NLP
#ACL2024
@aclmeeting
Today I am working from the beautiful new
@JohnsHopkins
Washington DC building 555 Penn, home to
@JHUCarey
and
@SAISHopkins
.
It's amazing! Take a look at our neighbors down the block. Seems like a good place to be if you care about shaping US national policy about AI. 🙂🇺🇸
I read papers that build and test a dataset with an LLM. Even when it's justified and the paper does this well, there is still no grounding with a human eval.
We're creating a literature of benchmarks and papers with the same underlying bias.
My worry? House of cards.
Immensely satisfying today to see results from a baseline method beat chatGPT. Brings back some sanity to research.
"Wow, it's really amazing that chatGPT does so well! I mean, maybe it's trained on our test set, and it is worse than our baseline method. But still!"
Congratulations to my colleagues on this outstanding paper award. The best part? This paper doesn't mention GPT, LLMs, BERT, etc!
There are many fantastic and creative ideas out there. A diversity of research topics is great for the field.
I was sad to learn about the passing of Dr. Eugene Charniak. He was a giant in the field and recipient of the ACL lifetime achievement award, among many other honors. "Charniak parser" was a ubiquitous term when I was in graduate school. 🧵
Non-CS collaborator: can we just use BM25 for this project? We always try these fancy machine learning tools that sometimes help, but BM25 consistently works and is easy.
I heard this years ago from
@lintool
and Doug Oard. Still true.
@andriy_mulyar
@sleepinyourhat
@srush_nlp
@chrmanning
@ChrisGPotts
There are a lot of good comments already. I will try not to repeat anyone else.
In general, my advice is always to work on a topic that:
1) You don't think someone else will solve.
2) You are uniquely positioned to contribute.
Hot take for reviewers: if you write “repeat experiments with newer LLMs from the past few months” ask yourself: why?
“Newer” isn’t a good reason. What changes about the papers hypothesis?
Also: if your paper would be invalidated by a new model, write a different paper.
Good luck to everyone working on your ACL submission!
May your jobs finish on time, may the turkers produce the desired results, and may ChatGPT help you write a great paper (consistent with the ACL policy on AI writing aids).
To my students: I'll now go back to paper editing.
Johns Hopkins has a fantastic new PhD Fellowship: Vivien Thomas Scholars Initiative (VTSI). Fully funds a STEM PhD at JHU for the exceptional diverse talent that exists at HBCUs and minority serving institutions.
@JHUCompSci
@HopkinsEngineer