Firecrawl
@firecrawl_dev
Followers
3,137
Following
12
Media
69
Statuses
358
Turn websites into LLM-ready data. Built by @mendableai team Open source:
San Francisco
Joined May 2024
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Telegram
• 1245361 Tweets
#خلصوا_صفقات_الهلال1
• 1227251 Tweets
Brazil
• 909926 Tweets
O Brasil
• 673758 Tweets
Elon Musk
• 664376 Tweets
Chelsea
• 514036 Tweets
Alexandre de Moraes
• 479925 Tweets
Osimhen
• 278244 Tweets
Sterling
• 200998 Tweets
Xandão
• 143250 Tweets
Napoli
• 134156 Tweets
Adeus Twitter
• 124643 Tweets
Ditadura
• 124505 Tweets
Threads
• 95139 Tweets
#مولد_ولي_العهد
• 86926 Tweets
Labor Day
• 65868 Tweets
Tchau
• 65537 Tweets
Anatel
• 63566 Tweets
Eddie
• 62083 Tweets
Nelson
• 53666 Tweets
Derrubaram o X
• 42526 Tweets
Bluesky
• 37982 Tweets
Tumblr
• 36343 Tweets
VPNs
• 35964 Tweets
Coreia do Norte
• 33429 Tweets
Benfica
• 32029 Tweets
CONGRATULATIONS RM
• 30461 Tweets
#الجناح_مطلب_اهلاوي
• 23809 Tweets
CONGRATULATIONS NAMJOON
• 22098 Tweets
Orkut
• 19059 Tweets
BARMY WITH BTS
• 15786 Tweets
Shelton
• 10363 Tweets
野菜の日
• 10349 Tweets
We're proud to power
@MeetGamma
's new Import from URL feature!
With this new integration, users can generate entire presentations from any website in under a minute ✨
2
18
95
Want to turn any website into an API with AI?
SmartCrawl is coming soon 🔥
Join the waitlist today, link below 👇
5
21
92
Introducing Fire Engine - The backend powering Firecrawl
It is the most scalable, reliable, and fast way to get web data.
Compared to leading competitors:
- It scrapes 40% more websites
- It is up to 33.17% faster
Read more in our official launch post 👇
5
21
80
We are excited to announce our SmartCrawl beta 🔥
This new feature allows you to turn any website into an API with AI Agents.
Check it out 👇
Introducing SmartCrawl by
@firecrawl_dev
- Turn any website into an API with AI
Watch the AI in action:
- Find the top 10 AC units on Amazon
- Extract and format JSON product data
- Generate a reusable automation accessible via API
Built with
@e2b_dev
@browserbasehq
37
80
619
2
12
75
We are very excited to launch our new integration with
@FlowiseAI
!
Flowise users can now easily add clean web data to their drag and drop LLM workflows 🔥
1
11
67
We are launching a new way to extract structured data from pages using raw HTML🚀
Instead of using markdown only, this mode can understand the structure of elements on the page.
Try it out by setting the extraction mode to 'llm-extraction-from-raw-html' 🔥
0
8
65
Did you know that we have a
@zapier
integration?
This allows you to use easily use our web scraping API with over 6000+ integrations.
Here's how you can get started:
1
7
50
Firecrawl now integrates with
@dify_ai
, making it easy to use web data in your projects 🔥
Our collaboration with this leading open-source LLM development platform is now live and ready for action!
Try it out using their open-source repo or on Dify cloud 👇 (1/2)
5
15
50
Firecrawl just crossed 7000 stars on Github ⭐️
Thanks to our community, contributors, and team!
Stay tuned for some exciting updates coming soon 👀
3
8
47
We are excited to announce that scraping is now up to 900ms faster on Firecrawl 🏎️💨
More speed and reliability improvements coming soon so stay tuned!
2
6
43
Firecrawl is now in crewAI🔥
Excited for people to start building crews with us!
Also, what a stacked update!
@crewAIInc
🤝
@AgentOpsAI
🤝
@firecrawl_dev
🔥🔥 crewAI v 0.36.0 is out 🔥🔥
RT please?
🔚 Force tool results as final agent result
🥷 Adding
@AgentOpsAI
native support
🔥 Adding
@firecrawl_dev
Tools
🧑💻 Improving coding Interpreter tool
➰ Option to create your own converter (docs pending)
📃 Updated Docs
🐛 Bug fixes
12
74
263
3
10
42
Want to see how startups are using Firecrawl to rapidly onboard customers?
Check out this awesome blog post by
@_Evan_Boyle
from
@cortexclick
👇
1
10
40
Want to do RAG on websites all with one DB call?
The team at
@postgresml
posted an amazing blog on how to use Firecrawl with their Korvus SDK which unifies the entire RAG pipeline in a single database query.
Check it out below 👇
2
11
39
Announcing Launch Week I 🔥
We're thrilled to introduce our first-ever Launch Week, starting next Monday.
Mark your calendars and stay tuned, we’ll be releasing a new feature every single day!
3
13
41
Some exciting updates today 🔥
- Stealth: Firecrawl now offers enhanced stealth when scraping at no extra cost
- Allow External Links to be crawled by
@jeffluipe
- Docker + dependabot improvements by
@sanixdarker
- Integration w/
@crewAIInc
coming next week 👀
2
7
33
Introducing Enhanced Dashboard Monitoring 📝
We are excited to give developers more transparency into every scrape and crawl request.
See processes, timing, failures and more on the improved Activity Logs page 🚀
1
7
31
We are excited to launch our brand new dashboard 🔥
Inside you can do things like:
- View and download data from jobs
- See breakdowns and analytics of usage
- Easily access playground for testing
Here's a closer look at some of the pages 👇
3
7
30
Did you know Firecrawl has crawl streaming built in?
New pages can stream immediately when scraped for instant use.
Here's how you can do it with our playground and API👇 (1/4)
1
8
30
Want to test out Firecrawl with real world use cases?
We have a wide variety of plug and play examples on our Github Repo!
Check them out today in the /examples folder 🔥
1
9
28
Firecrawl has reached 9,000
@github
stars!
Thank you everyone for the support! ⭐️
We have some major updates coming up next week - stay tuned! 👀
2
10
29
Want to get up to speed with Firecrawl quickly?
Check out this awesome video by Reda Marzouk explaining how to get started.
Link below👇
1
7
27
We are launching new parameters that enhance HTML content filtering 🔥
For scraping and crawling endpoints:
- onlyIncludeTags will only include content inside of the listed tags
- removeTags will not include content inside the listed tags
Try them out today 👇
2
8
27
Want to quickly start testing Firecrawl with real use cases?
We have an extensive example library right inside the main repo.
Check it out👇
2
9
26
In case you missed it, OpenAI's latest Structured Outputs update enables GPT-4o to consistently deliver structured JSON output 100% of the time.
Check out our tutorial on using it to extract structured data from web pages:
0
7
26
New Firecrawl tutorial out with
@weaviate_io
🔥
The web has a treasure trove of data to build RAG applications 🪙
Firecrawl (
@firecrawl_dev
🔥) is a powerful web scraping tool built by the
@mendableai
team!
In this recipe,
@ericciarla
and
@CShorten30
show you how to:
1. Scrape data from a web page
2. Index and store in
12
46
190
3
8
25
New video tutorial on Firecrawl 🔥
Learn how to scrape websites and extract data using GPT-4o. This tutorial walks you through the process step-by-step.
Check it out (link below) 👇
2
2
24
We're proud to announce our new integration with
@langflow_ai
!
Langflow by
@DataStax
users can now seamlessly incorporate clean web data into their flows with just a few clicks 🔥
2
5
23
Have you seen
@LangChainAI
's new Generative UI series by
@BraceSproul
?
It introduces building AI chat apps that call tools & render components based on user inputs.
We couldn’t help but integrate our own website data tool with Firecrawl 🔥
2
10
24
One of the most requested features was the ability to /scrape behind authenticated web pages 🔒
Today, we are releasing a headers option that allows you to forward authentication cookies to grab data behind auth walls 🔥
Check it out 👇
2
7
23
We're nearing 8,000 GitHub stars...
Can you help us get there? please 🥺
2
6
22
We are excited to announce our new integration with
@CamelAIOrg
!
With Camel AI and Firecrawl you can quickly build multi-agent systems that use data from the web.
Excited to have this integration live and stay tuned for a tutorial coming soon 👀
1
6
20
> "Are you a robot?"
> Nope.
Captchas? No problem ✅
SmartCrawl attempts to solve any captchas blocking its way.
Still WIP but we're constantly improving it
8
10
50
0
3
20
Easily connect web data to your AI apps 🔥🦜
🔥GenUI x FireCrawl
Great collab from our friends at
@firecrawl_dev
to combine our Generative UI app with their website data tool FireCrawl
Enable generative UI experiences over web data!
4
25
162
0
6
20
You can now ingest web data in
@dify_ai
🚀
Powered by Firecrawl🔥
💫 Dify v0.6.11 has landed!
The new update boosts Dify Workflow's teamwork features and adds a few cool new integrations.
🌟 New Features 🌟
🔥 We've added Firecrawl as a "Sync from website" option in Knowledge, so your RAG application can now tap directly into LLM-ready data,
1
14
58
1
4
18
Excited to share our official Launch YC post!
Check it out and give it an upvote 👇
2
6
19
Want to build workflows that use AI to answer questions about a website?
Check out this awesome example using
@langflow_ai
&
@cohere
🔥
2/ Scrape any website- Use
@firecrawl_dev
integration combined with Cohere LLMs to scrape text from any website and answer questions about it.
1
0
5
2
5
19
Introducing the Firecrawl Go SDK 🔥
This addition joins our Python and JavaScript SDKs, expanding support to the Go ecosystem.
Huge thanks to
@kentchiahaohsu
from our community who made this happen!
2
5
18
Our July update is live 🔥
We're excited to share this month's highlights with you including:
- Our YC launch
- Speed & Reliability improvements
- Amazing community projects from
@BraceSproul
and
@SouravMaji221
- We're Hiring!
1
6
17
Here are some exciting things we shipped last week 🔥
- Customize final output with pageOptions.removeTags
- Use our Python/Node SDKs with local instances
- Relative max depth
- Improved metadata for scraped pages
- Lower latency for US West
@flydotio
Take a peek 🧵
2
8
17
Awesome tutorial on building a better docs chatbot with and Firecrawl 🔥
Tired of sifting through endless documentation? Michael Ryaboy reveals how to build a smarter documentation chatbot using Firecrawl and . Learn to simplify web scraping, enhance search and optimize documentation process.🤖✨
0
0
4
0
7
17
We are excited to announce our integration with
@PraisonAI
🚀
Praison is a low code framework for orchestration of multi-agent systems built by
@MervinPraison
Docs:
0
8
17
@MeetGamma
enables its 20+ million users to create beautiful AI generated presentations, websites, and more.
Since June, they've used Firecrawl to easily allow users to import websites into their creations.
Read more on how
@thatsjonsense
and the team use us for onboarding 👇
1
5
17
Building knowledge graphs with web data
This new example from
@CamelAIOrg
uses Firecrawl to gather data for constructing KGs 🔥
🐫 Using CAMEL's Role-Playing Scraper for a Report & Knowledge Graph on the Turkish Shooter from the 2024 Paris Olympics! 🥇
Dive into web scraping, multi-agent role-playing, and knowledge graph construction in one comprehensive example. 🕵️♂️🤖
⭐ Highlights:
- 🛠️ Setup: Install
3
38
162
0
2
16
We are hiring for our marketing and growth team 🔥
Do you love hacking things together and have a knack for technical writing?
This role is for you 👇
1
6
16
Thank you everyone for the 8,000 stars⭐️
You all are awesome! 🔥
We're nearing 8,000 GitHub stars...
Can you help us get there? please 🥺
2
6
22
2
7
16
Want to cluster common topics from any web page?
New
@firecrawl_dev
example with
@e2b_dev
is live!
Check it out 👇
🔎 Visualize website topics with Sonnet 3.5 🧠
This open-source agent scrapes a website and plots the most common topics.
Techstack:
🔸
@firecrawl_dev
for web scraping
🔸 Code Interpreter SDK by
@e2b_dev
🔸 Claude-3.5 Sonnet by
@AnthropicAI
Credits:
@ericciarla
🔍 Code in
7
13
51
1
4
15
Awesome example of Agentic web scraping using our API 🔥
Agentic Website Scraping using Firecrawl: How to Setup Locally?
🔥 Integrate with
@LangChainAI
@llama_index
🤖 AI Agent Integration
@PraisonAI
🌐 Set up locally
@Docker
📊 Data Processing Automation Llama 3 LLM
@GroqInc
🔄 Search
@yousearchengine
@mendableai
@firecrawl_dev
54
56
284
1
1
15
We love building integrations to make Firecrawl easier to use 🔥
Over the last few weeks, we have launched 4 new
@firecrawl_dev
integrations with
@crewAIInc
,
@dify_ai
,
@FlowiseAI
, and
@langflow_ai
🚀
What integration should we launch next?
5
12
37
0
2
14
Want to see how you can build a smarter documentation chatbot with Firecrawl and KDB AI?
Check out this awesome article by Michael Ryaboy and
@kxsystems
🔥
3
6
14
Just like that Firecrawl has hit 6000 stars ⭐️
It's amazing to see the ways it is being used for all kinds of use cases that need easily accessible web data.
Thank you to our team, contributors, and community 🔥
2
6
14
Introducing our new Firecrawl X account with a spicy twist! 🔥
Get your website roasted on design and copy with Roast My Website (using our new screenshot feature) 🌶️
Ready to get roasted? 👇
1
8
13
Have you had a chance to try Firecrawl's LLM extraction?
Extracting structured data from a web page is 1 API call away
Here is a collection of examples with it 🔥🧵
1
7
13
Introducing the Firecrawl Rust SDK!
This addition joins our Python, JavaScript, Go SDKs, expanding Firecrawl's capabilities to the RUST ecosystem.
Check it out on our docs:
1
3
12
🔥🤝🖇️
Scraping web data for AI agents sucks.
@firecrawl_dev
is fixing that.
Live demo of Firecrawl turning entire websites into LLM-ready data in seconds w/
@CalebPeffer
19
127
880
0
2
12
SmartCrawl scraping behind authentication 👀
SmartCrawl is insane - we're seeing some really cool use cases 🤯
One of my favorites is extracting data behind auth walls 🔒
Check out this demo where it logs into
@firecrawl_dev
, gets my usage data in seconds, and turns it into an API!
14
40
337
1
2
12
AI Powered Website A/B Test Recommendations 🧪
Our friends at
@growth_book
created WebLens, which analyzes websites & suggests testable changes.
It might recommend testing things like moving a button or simplifying navigation.
See how they built it with Firecrawl below 👇
1
6
11
Just like that you can now use Firecrawl with
@LangChainAI
's Generative UI in Python as well 🔥
You see that
@LangChainAI
🦜 fork by
@firecrawl_dev
🔥 with the unstructured content extraction for GenUI?
That fork of
@BraceSproul
’s
@vercel
AI GenUI repo + their slick API for JS?
Hey
@firecrawl_dev
, here’s the Python LangGraph/LangServe Version 👇
Thanks for the inspo 🤍
5
10
34
0
5
11
Super excited launch our new integration with
@FlowiseAI
!
Easily scrape web pages and add them to your LLM flows 🚀
🔥Firecrawl Web Scraper
Crawl and convert any website into LLM-ready markdown or structured data in Flowise, thanks
@mendableai
team for the integration!
1
4
15
0
4
11
Prompt caching for Anthropic models is in beta! Check out this example 🔥
Prompt caching from
@AnthropicAI
is in beta 👀
You can chat with contexts up to 200k tokens faster and cheaper than ever. This is huge for RAG!
I threw together a quick test analyzing web data from
@firecrawl_dev
and it’s impressive!
2
10
90
0
1
11
Want to check out the repo? Here it is:
0
4
11
With Gemini's context caching you can chat with entire websites with no chunking or retrieval 👀
Context caching with Gemini is super underhyped!
It lets you chat with massive contexts quicker and more cost-effectively than ever.
I used
@firecrawl_dev
to crawl the entire
@dify_ai
website and had a conversation with it without any chunking or retrieval 👀
5
26
162
0
2
10
Link to blog post:
0
4
8
🔥🔥🔥
gm! more releases this beautiful friday (yes we release on fridays):
scraping URLs now in Lynq -- powered by our friends
@firecrawl_dev
use it to monitor competitors for changes or funds for new portfolio companies. it's super powerful!
1
4
13
0
2
8
Improve your blog's SEO with this latest Firecrawl example 🔥
Adding internal links to your blog has never been easier!
I built an OS example using
@firecrawl_dev
that:
- Scrapes your entire website
- Uses Claude to identify link opportunities
- Puts it all into a CSV file for easy execution
Inspired by
@MaxBrodeurUrbas
. Link below 👇
4
11
86
0
4
11
Sooo who's going to the most hyped open source meetup of the month? 🙋♂️
0
4
7
Want to run it yourself? Here is the repo:
0
0
7
The web has a treasure trove of data to build RAG applications 🪙
Firecrawl (
@firecrawl_dev
🔥) is a powerful web scraping tool built by the
@mendableai
team!
In this recipe,
@ericciarla
and
@CShorten30
show you how to:
1. Scrape data from a web page
2. Index and store in
12
46
190
0
2
7
Join the waitlist here:
0
1
6
👀👀
Scraping web data for your agents? Firecrawl is the easiest way to do it
Chat + demo sesh preview with
@CalebPeffer
@firecrawl_dev
👀
7
14
118
0
2
5
💙❤️🔥
Dify v.0.6.11のリリースです🎉！一番注目したいのはいよいよナレッジにSync from websiteが追加されたこと👏。
@firecrawl_dev
と連携すれば、
①URLを指定
②該当サイトをクローリング＋スクレイピング
③ナレッジに保存
といった流れが一瞬で作れる。あとから個別URLを指定できるのも便利✨。
6
62
411
0
4
5
Love this 🔥
We're open-sourcing our digital footprint exporter.
It automatically exports your data from Notion, ChatGPT, Twitter, etc and puts the data in LLM ready format.
Its like
@firecrawl_dev
but for your personal data.
Perfect for making an (actually useful) AI assistant.
(
@ollama
38
90
653
0
0
5
3) Here we use LLM extract to get and paginate through Airbnb listings w/ Firecrawl. Featuring
@e2b_dev
and
@AnthropicAI
🔎 Scrape and analyze website data with AI 🧠
Check out this open-source example of an LLM agent that crawls
@Airbnb
website to analyze the prices of houses.
Techstack:
🔸 Firecrawl by
@mendableai
🔸 Code Interpreter SDK by
@e2b_dev
🔸 Claude by
@AnthropicAI
7
67
442
1
2
5
This repo allows you to chat locally with websites 🔥
Excited to launch Local Website Chatbot 🚀
With this repo, you to chat with almost any website using local LLMs powered by
@ollama
,
@firecrawl_dev
, and
@LangChainAI
.
It's a fork of
@Hacubu
’s Local PDF Chatbot with website support 🔥
Repo link:
3
17
77
1
0
5
Follow our new account
@firecrawl_dev
!
Built with our /scrape endpoint and new screenshot param.
🔗 Try it out here:
0
1
5
Link to Repo:
0
0
4
👀👀👀
If you want to know what apps another app uses to build their tech stack, take a look at their list of subprocessors.
e.g. Zapier:
Screenshot shows what apps Zapier uses to build their new Products in Beta.
- BrightData (Central)
- Firecrawl (Chatbots)
0
2
14
0
0
4
Interesting in getting started? Here are the links:
Open source repo:
Dify Cloud: (2/2)
0
0
4
🔥⛓️
Did you know you can scrape a webpage in a few seconds in Langflow using Firecrawl
@firecrawl_dev
? And use Llama 3.1 for your flows using Groq
@GroqInc
? And call 100's of AI models using AI/ML API
@aimlapi
? And much more...
1
15
75
0
0
4
We are growing our team 🔥
Looking for a web automation engineer
Why you should work with Firecrawl?🔥
1. Our repo crossed 6k stars in two months
2. Paid customers from day 1
3. Building the future of web extraction for humans and AI. One API, one prompt => any data, format, from anywhere.
🔗👇
1
4
14
1
0
4
3. After entering our Firecrawl API key, we can get the clean markdown for any URL entered on the form and use this data with any one of
@zapier
's 6000+ other integrations 🔥
1
0
3
That's all for now! You can try it out at and stay tuned for more exciting updates🔥
Btw the dashboard is still in Alpha so feel free to share any feedback or requests you might have!
0
0
3
1) Video tutorial by
@nickscamara_
showcasing how to extract structured company data with a single API call
To extract structured data from the web you used to have to write complex web scrapers...
LLMs are changing that.
Here is Firecrawl 101: Learn how to extract structured company data with a single API call 👇
8
11
57
1
2
3
1
0
3
Check out the videos series here:
1
0
3
Another super exciting page is the Usage page. This is where you can see breakdowns and analytics of current usage:
1
0
3
Repo:
Graph made by:
0
0
3
One of the most exciting parts of the dashboard is the new Activity Logs page. This is where you can view and download data from your jobs:
1
0
3