AI search engines use LLMs to summarize results for you, but can LLMs discern low-credibility sources from credible ones? Our paper measures the accuracy of LLMs' credibility ratings on news sources and political biases.
With
@OSoMe_IU
🔗
1/9
In other news, I married my best friend and the love of my life today. We have been dating for 7 years, most of which have been long-distance. Every reunion, including this one, was not easy. But we made it. And I'm the happiest person on earth.
Found this powerful scraping tool lately. It's a browser extension, so it works on all web pages you can visit and no coding is needed. Below is what it looks like when I try to scrape my Twitter timeline. It works especially well on tables. 1/2
🚨New dataset🚨
Studying online discussion about the 2022 US midterm elections? Check out our dataset MEIU22, a collection of social media posts from multiple platforms.
Paper:
Data+code:
1/5
Some personal updates: I just joined
@davidlazer
lab as a postdoc at
@Northeastern
. Exhausted due to moving & settling in, but quite excited about this new chapter.
If you are also in the Boston area, feel free to reach out, we can grab a coffee together!
TikTok announced that researchers can access more account-level data through its API. Although the decision-making process isn't transparent, it seems they are taking suggestions and requests from researchers, including us.
1/2
You probably have seen how powerful LLMs like ChatGPT are. But what if they are used to supercharge social bots?
Check out our new work: Anatomy of an AI-powered malicious social botnet, with
@OSoMe_IU
Link:
Code & data:
1/11
Fake accounts like those below use profile pictures generated by AI (GANs)! Our new study reveals their characteristics and estimates their prevalence on Twitter: about 10k daily active users!
With
@danishjeetsingh
and
@OSoMe_IU
1/7
Excited to share that our work "Characteristics and prevalence of fake social media profiles with AI-generated faces" has been accepted by
@journalsafetech
. Moreover,
@danishjeetsingh
will present our poster
@IC2S2
on Jul 19, be sure to check it out!
1/3
Some personal news: I passed my PhD defense yesterday. Still need to finalize my dissertation and some paperwork, but I guess you can call me Dr. now 😄😄😄.
#PhDone
Many thanks to my amazing advisors:
@OSoMe_IU
@yy
@RealBreaPerry
Alessandro Flammini (not on Twitter).
1/2
New work showing that ChatGPT can rate the credibility of news outlets (correlation > 0.5 w/ expert ratings). This capacity might help LLMs w/ Internet access ground their answers.
Link:
Code/Data:
W/
@OSoMe_IU
1/6 🧵
I've been using ChatGPT to help me filter new arXiv papers every day for two months. It saved me a lot of time and allowed me to stay on top of the field's development. In case you want to try:
Here is how it works: 1/5
Heading home from
@IC2S2
. One of the best conference experiences 😊. Met many old friends and made new ones. Very encouraged to see participants from more diverse backgrounds and younger generations joining the CSS community.
(Quite lucky that my flight was not affected)
People used to joke about
#ICWSM
being a Twitter conference, so I analyzed the papers to see if it's true. And yes, over 30% of the papers mentioned "twitter." I might have contributed a few times 😅. Wonder where the community is headed without free data 🤔. 1/2
Those doing computational social science should read this.
Three Gaps in Computational Text Analysis Methods for Social Sciences: A Research Agenda
Link:
Some late-night talking, things I usually wouldn't tell people. Although I consider myself doing ok in my career as a researcher, I still feel anxious almost all the time. I have panic attacks from time to time and trouble sleeping most nights. 1/3
After seeing more and more "verified" bot accounts on this platform, many wonder who would have the money to pay for the premium fee. Turns out they don't need to! If bots can create enough content impressions, they get paid! Pic below shows an operator bragging about this.
1/4
Twitter API updates:
1. Free tier -> $40k/month (yep, there is a k)
2. $100/month tier has very limited functionalities
3. Happends in 4-6 weeks
4. Decahose will prob be the only option for data collection
5. Based on conversation with Twitter enterprise rep
Source 👇
Just got news that the new Twitter API has an ENTRY price of $40k/month.
It'll probably kill the long tail of tools/services that were /are crucial for Twitter's network effect success.
Excited to share that our paper "Anatomy of an AI-powered malicious social botnet" with
@OSoMe_IU
has been published in
@journalqd
.
DOI:
I'll present it
@ICWSM
on June 4 at 10am and the poster session in that afternoon.
See the thread below for details:
You probably have seen how powerful LLMs like ChatGPT are. But what if they are used to supercharge social bots?
Check out our new work: Anatomy of an AI-powered malicious social botnet, with
@OSoMe_IU
Link:
Code & data:
1/11
Thrilled to be accepted to
@OpenAI
's researcher access program. The awarded credits will help me study the societal impact and misuse potential of the GPT series models.
Tomorrow at
@IC2S2
I have two presentations with fresh results.
Morning: I'll share analyses of the dissemination pattern of fact-checking articles on behalf of
@JehoonC
.
Afternoon: I'll present auditing of LLMs on their ability to judge news outlet credibility with
@OSoMe_IU
Looks like the results for
@IC2S2
are out. All 7 papers I was involved in got accepted either as talks or posters. Also got a tutorial there.
Can't wait to see all the old friends and make new ones in Philly this July!
The prevalence of social bots has been at the center of the dispute between Twitter and Elon Musk amid Musk's takeover. But how does the public perceive the issue of social bots? Our new working paper sheds some light.
🧵 1/7
Link:
FBI took down an AI-powered bot farm recently. Two examples provided are clearly using GAN-generated faces, similar to those documented in our recent paper . The 93-page affidavit describing the identification process is worth reading. 1/2
A social media model suggests that bad actors can make low-quality content go viral most efficiently by infiltrating a community and gaining followers rather than by just “flooding the zone” with junk. In PNAS Nexus:
In light of Meta's decision to discontinue CrowdTangle, we,
@transparenttech
and
@GWIDDP
, are conducting the DSA Data Access Audit. If you work with social media data in some way, please consider filling out the survey. Details:
Excited to announce our paper "Can GPT-4 Models Detect Misleading Visualizations?" to be presented at
@ieeevis
.
Link:
With
@ali_sarvghad
and talented undergrads Jason Alexander and Priyal Nanda.
1/4
Cool
#ICWSM24
paper "Generative AI in Crowdwork for Web and Social Media Research: A Survey of Workers at Three Platforms" shows that a fair number of crowdsource workers use AI chatbots in their tasks (look at the number of MTurk!)
Link:
Excited to announce that the 5th International Workshop on Cyber Social Threats (CySoc 2024) at
@icwsm
. Spotlight topic this year: the use and misuse of generative artificial intelligence.
CFP coming soon! Check out for updates!
1/2
Need to process large amounts of data with ChatGPT but struggle with the API calls? Check out my tutorial at
Currently covers the topics below w/ instructions and example code. If you find it helpful, plz share and star. Issues and PRs are welcome!
Checkout our new New media & Society paper “Asymmetrical perceptions of partisan political bots”
With
@YyjHarry
@OSoMe_IU
@JamesShanahan
DOI:
Free preprint:
1/3
I'm very thrilled to receive the 2021 Luddy Outstanding Research Award from
@IULuddy
. I want to thank the school for creating such a wonderful environment that fosters innovation and collaboration and to thank my mentors and collaborators. They made whatever I'm doing possible.
Found out that I'm featured in
@Mozilla
’s new AI Intersections Database, which maps where AI impacts meet social justice issues and catalogs the people and organizations working to make AI equitable for all. Link to the db:
Not to insinuate causality, but it appears the botnet was suspended by Twitter a couple of days after the release of our paper 😅. Before that, the botnet had been active for over 6 months undetected.
You probably have seen how powerful LLMs like ChatGPT are. But what if they are used to supercharge social bots?
Check out our new work: Anatomy of an AI-powered malicious social botnet, with
@OSoMe_IU
Link:
Code & data:
1/11
Can finally reflect on the post-API conference. Very nice gathering and lots of stimulating discussions. My takeaways:
1. All Twitter researchers are (unequally) screwed
2. Some emerging alternative solutions, but no silver bullet
3. Some good changes on the horizon
1/2
2023 was a special year. Finally got PhD and visited home for the first time after years of grinding. Finally got COVID and boy it hit me hard. Then I moved to Boston to start a new journey. Had the lowest point of my life, but also many highlights.
#HappyNewYear
Dealing with bipartite networks? You might want to check out our BiRank package that performs PageRank of different flavors on bipartite network directly. With the amazing Brian Aronson and
@yy
.
JOSS paper:
GitHub:
1/3
The team has been analyzing data we collected from Twitter, Meta, and Reddit in 2022. Just realized that the same kind of data collection and analysis will no longer be possible 😢
It's confirmed, the academic research API access for Twitter will be gone after Feb 13 🪦
It will be replaced by the $100/month "basic access." Details coming on Monday.
#NoResearchWithoutAPI
Wrote a unified interface for querying LLMs from different providers. Currently supports OpenAI, Google, and Together (various open-weight models).
Link:
BTW, I'll be serving as the publicity chair for ICWSM24. Please follow
@icwsm
for the updates, and feel free to DM if you have any questions regarding the conference!
The ICWSM24 organizational committee is working hard to prepare for the next conference at Buffalo. In the coming days, we will announce the logo, website, a new global south initiative, CPF, and the exact dates for the conference. Please stay tuned!
Friday at
@IC2S2
, my lovely coauthors will present 2 posters and 1 talk:
- 1:30PM: Effect of LLM-generated fact-checking
@mdeverna2
- 1:30PM: New metric for localness of news domains
@Pranav__Goel
- 2:30PM: Problems with social media polls
@przemyslslaw
1/2
Asked by some journalists if it's possible to estimate the prevalence of bots on Twitter the other day. I actually had some not perfect but reasonable ideas that all start with scanning millions of tweets/accounts.
Now you tell me Twitter did it by looking at 100 accounts?
Forgot to mention that these bots also leverage ChatGPT to produce content and replies. The paycheck from Twitter could easily cover the cost of API calls to OpenAI.
After seeing enough instances, I believe a part of this platform has already turned into a Westworld.
4/4
Have a Twitter project that was canceled or changed? We'd love to hear from you!
@transparenttech
is running a survey to understand how research is impacted by changes to Twitter. Here is the link:
#research
#socialmedia
#twitter
Friday at
@IC2S2
, we will present two papers on AI-powered bots/fake accounts on social media:
- Plenary talk at 9AM: Anatomy of an AI-powered malicious social botnet
- Poster at 1:30PM: Characteristics and prevalence of fake social media profiles with AI-generated faces
1/2
After listening to some friends' stories, I realized how lucky I was during my PhD. The journey was not easy, but there wasn't a single moment that I didn't feel supported.
My guess, this "basic access" is not enough for most research projects, the $100 monthly fee is too high for 95% of the researchers.
If we want to keep the whole Twitter research body and community alive, we will have to start considering alternatives.
#NoResearchWithoutAPI
Another nice thing is that you could upload a list of URLs and have the tool scrape the same elements from them automatically. Can't believe I was writing scrapers myself not long ago 😅
Link to the extension:
Works for all Chrome-based browsers.
2/2
The findings empirically show that AI-generated faces have been deployed at scale to create fake accounts. Earlier, we showed that ChatGPT has been used to enhance bots. Apparently, AI-powered fake accounts are here to stay. Are you ready for it?
6/7
You probably have seen how powerful LLMs like ChatGPT are. But what if they are used to supercharge social bots?
Check out our new work: Anatomy of an AI-powered malicious social botnet, with
@OSoMe_IU
Link:
Code & data:
1/11
📢CFP: 5th International Workshop on Cyber Social Threats (CySoc 2024) at
@icwsm
invites research, position, short, and demo paper submissions!
📅 Deadline: March 24
🔗 Instructions:
🚪Submission portal:
Twitter might lack the incentive to deal with these accounts because it's really a win-win situation: Bot operators make money while Twitter makes money and boosts DAU. Why bother?
But who loses?
3/4
After seeing more and more "verified" bot accounts on this platform, many wonder who would have the money to pay for the premium fee. Turns out they don't need to! If bots can create enough content impressions, they get paid! Pic below shows an operator bragging about this.
1/4
Happy to see that Musk's lawsuit against
@CCDHate
was dismissed by the court and that the ruling cited our survey with
@JosephineLukito
@m_dot_brown
. But reading the ruling gave me a lot to think about
1/3
@ChrisTorresLugo
and I will demo how to use OSoMe tools () to identify social media manipulation at 2022 Computation + Journalism Conference (). We have some cool news tools to show! You can join remotely and registration is free!
One lesson I learned the hard way: don't try to write your CV (or personal website) right before the job application. Create one RIGHT NOW and iterate it regularly. Add a new line when you have a new achievement to remind yourself you are progressing. 1/2
Something I have been doing increasingly is writing the manuscript, preparing the slides, and running the data analyses simultaneously. It helps me think about the project end-to-end and reminds me of the big picture while I'm dealing with details.
Keynote by
@rzafarani
: Fake news detection: Limited Ground Truth, Limited Text, No Understanding of Spreading Intent
#CySoc
Also on Zoom, DM me for link!
Checkout our new preprint "Prevalence of Low-Credibility Information on Twitter During the COVID-19 Outbreak". Collaboration with the amazing
@ChrisTorresLugo
,
@OSoMe_IU
Link:
1/5
2) Conservative personas are more confusing and Republican participants perform less well
3) Republican users are more likely to confuse conservative bots with humans, whereas Democratic users are more likely to confuse conservative human users with bots
3/3
Introducing Botometer X! The original Botometer website was disabled after Twitter (now X) suspended researcher access to their data. To fulfill the need for bot detection, we revived and rebranded it. Check it out at
1/3
The papers accepted by
#CySoc
2024 are now available online, congratulations to all the authors 👏!
We have a great program for everyone, details at 🚀. We look forward to seeing everyone soon!
If you can't make it in person, DM me for Zoom link!
A common strategy for increasing exposure is to reply to other accounts' posts. Sometimes they choose random posts, sometimes they target high-profile accounts and popular posts. Another strategy is, of course, to interact with other bots, which leads to anomalies.
2/4
Some collaboration tricks they don't teach in school: If you want something to happen, you have to be very clear about who is responsible, when you want to see the results, and what the expected outcome will be. The more specific, the better.
Quick analysis on bot activities of tweets mentioning 'Elon Musk' Apr 23-26. Two notable patterns
- Twitter&Musk deal news out -> higher volume -> lower bot activities
- Tweets expressing overall positive sentiment began to show higher bot amplification levels after Mon 4pm ET
🧶
Just ran Llama 3 8B on a virtual Linux machine with 8 CPU cores, 30G RAM, and no GPU. It was generating output at a speed slightly slower than GPT-4 🤯. Note that this is a ChatGPT-3.5 level model, which is pretty capable already. It just changed a lot of things.