We just launched SmartCrawl (by
@Firecrawl_dev
) which turns any website into an API with AI.
Agents can be hard but here's a tip - use tools like
@AgentOpsAI
or
@LangChainAI
's LangSmith to debug executions.
We do and they have proved invaluable.
Check out what we built:
After some user requests, BabyAGI JS now supports GPT 3.5 Turbo. This means costs to run are 90% cheaper 💸. Best of all, it only took 10 mins to implement thanks to
@LangChainAI
's model interoperability.
BabyAGI is now available in Javascript 🚀. Inspired by
@yoheinakajima
my implementation of this
#AutoGPT
utilizes
@LangchainAI
for execution, creation, and prioritization +
@Supabase
for memory. Now it is much easier to integrate with
@Mendableai
and other JS projects 🔥 (1/4)
With (built with
@LangchainAI
+
@Supabase
) you can now chat with any
@Github
repo in minutes 🚀.
I literally just entered the repo URL and started chatting. Here is a quick demo and a bit on how it works (1/4)
During yesterdays webinar hosted by
@LangChainAI
and
@hwchase17
there were a lot of questions about how implemented hybrid search. Here is a little explanation on how we did it and increased correct document matching by almost 20%. (1/6)
At
@mendableai
we found that hybrid search works better than semantic search alone. In fact, it boosted correct document matching by around 20%.
Based on this method, I implemented hybrid search for
@supabase
on
@Langchainai
JS. Now it is live 🚀
Thank you to
@hwchase17
,
Just used to search internal docs like bug reports and product requirements while coding.
For this demo all it took was ingesting a notion feature doc and the GitHub repo for BabyAGIJS.
Shout out to
@LangChainAI
for the awesome data loaders 🚀. (1/2)
Context caching with Gemini is super underhyped!
It lets you chat with massive contexts quicker and more cost-effectively than ever.
I used
@firecrawl_dev
to crawl the entire
@dify_ai
website and had a conversation with it without any chunking or retrieval 👀
Using the API, we created one of the fastest ways ingest + chat with your data, via a CLI.
Here we spun up a demo on
@Replit
, where we ingested
@LangChainAI
's latest JS docs and generated up to date code using via the shell. (1/3)
What if you could create next-gen managed chat apps in seconds?
Introducing the
@mendableai
Python SDK: Build cutting edge + production ready LLM chat apps without any hassle. 🚀
Get started now with pip install mendable-py
Powered by
@LangChainAI
,
@supabase
, and
@Railway
.
Ready for the
@Supabase
AI hackathon this week? Perfect time to try out
@LangChainAI
’s Supabase hybrid search based on what we use at
@mendableai
. Read more here:
Prompt caching from
@AnthropicAI
is in beta 👀
You can chat with contexts up to 200k tokens faster and cheaper than ever. This is huge for RAG!
I threw together a quick test analyzing web data from
@firecrawl_dev
and it’s impressive!
Adding internal links to your blog has never been easier!
I built an OS example using
@firecrawl_dev
that:
- Scrapes your entire website
- Uses Claude to identify link opportunities
- Puts it all into a CSV file for easy execution
Inspired by
@MaxBrodeurUrbas
. Link below 👇
An AI agent now edits & maintains our
@mendableai
documentation🤖
Based on chat data sent to Slack, the AI finds the right doc to edit and proposes changes via a PR for .
The best part, it is REALLY good 👀. Built with
@LangChainAI
@github
@docusaurus
Excited to launch Local Website Chatbot 🚀
With this repo, you to chat with almost any website using local LLMs powered by
@ollama
,
@firecrawl_dev
, and
@LangChainAI
.
It's a fork of
@Hacubu
’s Local PDF Chatbot with website support 🔥
Repo link:
We’re launching a new
@supabase
service:
It’s like if ChatGPT and Postgres had a love-child: launch as many databases as you want, build them with AI, create charts, create embeddings. 100% open source.
Testing Llama 2 with the new
@mendableai
Next.js Chatbot template 👀
Shout out to
@LangchainAI
and
@huggingface
for making these models so easy to spin up and test just hours after launch!
If you want to try the Next.js Chatbot:
With OpenAI's new Structured Outputs update, GPT-4o can now reliably return structured JSON output 100% of the time.
This is super exciting for all AI apps and especially
@firecrawl_dev
👀
Watch it extract structured data from a web page (tutorial below) 👇
Introducing Precise Mode for 🚀
Our solution to the common question: How can I get my model to not talk about X?
At
@mendableai
we hear this too often, so we decided to solve it with the help of
@LangChainAI
.
Here is how it works 👇 (1/5)
Good things come to those who launch
Our first launch week at
@firecrawl_dev
comes to a close today and I could not be more proud of the team.
We released 7 features/improvements for 7 days straight and blasted from 9k to 12k stars on Github!
yesterday we launched FireCrawl 🔥 - it's the best way to turn any website into llm ready markdown
less than 24 hours in, we are about to cross 300 stars + have integrations with
@LangChainAI
and
@llama_index
Claude 3.5 Sonnet paired with Artifacts is incredible.
Just got it to create a playable Flappy Bird-style game in one shot, incorporating
@firecrawl_dev
branding.
The progress is super exciting, congrats
@AnthropicAI
!
Focus is everything
A few months ago, we were building a product that was stretching us thin.
We tried selling just one piece of our stack and that experiment became
@firecrawl_dev
, which quickly surpassed our other projects revenue.
Now we're having our first launch week 🔥
So impressed by the power of
@JuliusAI_
, it's like having a dedicated junior analyst available around the clock.
I would have loved to use this during my data science days at Ford.
The game is changed.
Just tested out Gemini 1.5 Pro’s long context and reasoning capabilities with one of
@firecrawl_dev
's examples.
After ingesting an entire website, I asked Gemini to identify any contradictions.
It worked super well and actually gave us some very valuable suggestions!
With the launch of 's new dev focused AI chat search builder, we are burning through compute and generation credits.
Besides being big fans of
@rauchg
and
@vercel
, this is part of the reason
@mendableai
is applying to Vercel's new AI Accelerator. 🚀
Excited to launch the updated ingestion UI template for
@firecrawl_dev
🔥
Key improvements:
- New link gathering and filtering using /map
- Increased speed & simplified advanced options UI
The experience is so clean with the new /v1 endpoints and ready to install in any app!
Detecting data conflicts with FireCrawl and
@LangChainAI
🧵
While building
@mendableai
, we saw a common issue for companies doing RAG with lots of sources: data conflicts.
With LLMs, we can detect and report these conflicts. Our approach and a
@Replit
Repl below👇 (1/4)
Want to use natural language to interact with dataframes? ⚡️
At (built with
@LangChainAI
+
@supabase
) we are experimenting with ✨generating code from natural language✨ based on any docs. Here is a demo with some
@pandas_dev
docs and the
@mendableai
API:
🔥
@firecrawl_dev
(YC S22) is an all-in-one developer platform for crawling & scraping web data for AI applications.
It's an open source API that transforms any web data into a clean, LLM-ready format for RAG, agentic tasks, or training.
Introducing CLI Tokenizer - A tiny tool for building prompts
We constantly counted tokens when prompt engineering at
@mendableai
.
Now you can do it faster than ever right inside your terminal.
How it works 🔧
- Scrapes the repo with
@mendableai
bots
- Embeds all of the documents using text-embedding-ada-002
- Stores embeddings in
@Supabase
with PGvector
- Prompt + call LLM with
@LangchainAI
- Retrieves relevant documents with
@mendableai
’s hybrid retriever (2/4)
We enabled ’s dashboard copilot to take action using the latest tool calling models.
Teach your chat app, add data sources, or analyze data all from the copilot.
Here is what we learned about agents and how we built it with
@LangchainAI
(1/4)
🔎 Visualize website topics with Sonnet 3.5 🧠
This open-source agent scrapes a website and plots the most common topics.
Techstack:
🔸
@firecrawl_dev
for web scraping
🔸 Code Interpreter SDK by
@e2b_dev
🔸 Claude-3.5 Sonnet by
@AnthropicAI
Credits:
@ericciarla
🔍 Code in
The web has a treasure trove of data to build RAG applications 🪙
Firecrawl (
@firecrawl_dev
🔥) is a powerful web scraping tool built by the
@mendableai
team!
In this recipe,
@ericciarla
and
@CShorten30
show you how to:
1. Scrape data from a web page
2. Index and store in
Many enterprises want a single interface to access multiple projects (Unique Model + data combinations).
We just released our Router Alpha which uses
@LangChainAI
function calling to dynamically route queries.
Here is what we learned so far. (1/5)
To celebrate
#SupaLaunchWeek
7, I gave BabyAGIJS’s execution agent superior question answering ability by increasing the amount of document context with the help of
@Supabase
+
@LangchainAI
. Another experiment for
@mendableai
! (1/3)
An awesome new series on Generative UI is out by
@BraceSproul
🔥
In the early days of
@mendableai
this was pretty difficult to pull off
So cool to see how easy it is becoming with tools like
@LangChainAI
Have you seen
@LangChainAI
's new Generative UI series by
@BraceSproul
?
It introduces building AI chat apps that call tools & render components based on user inputs.
We couldn’t help but integrate our own website data tool with Firecrawl 🔥
Super excited to be cohosting the
@LangChainAI
meetup with
@GlasswingVC
in Boston on Aug 15th!
Come by and listen to our panel (from
@ChaosSearch
,
@mendableai
, and more) discuss Training, Deploying, and Evaluating Large Language Models in production!
With GPT 3.5 fine-tuning, you can get GPT 4 level responses at a much lower cost 💸
Here are some early insights from the team.
- Getting good training data is a challenge
- Transforming data is important
- Fine-tuning is pretty simple with good data
Announcing Launch Week I 🔥
We're thrilled to introduce our first-ever Launch Week, starting next Monday.
Mark your calendars and stay tuned, we’ll be releasing a new feature every single day!
At first we only used embedding search (text-embedding-ada-002) but we soon found its limitations. That's when we introduced hybrid search, which combines the strengths of both semantic and keyword-based search algorithms. Here are some tips on how to implement. (2/6)
🦜🛠️ Introducing LangSmith 🦜🔗
A unified platform to help developers debug, test, evaluate, and monitor their LLM applications.
Integrates seamlessly with LangChain, but doesn't require it.
There is still a lot of improvement that can be done here, such as making the agent more aware of things like the repo file structure, etc, but this is a really interesting start. Here is a link to the live demo for
@babyAGI_
(JS): (3/4)
My thoughts + takeaways from the
@LangChainAI
x
@Mendableai
meetup:
- Many enterprises are using AI for internal apps
- Unclear how will humans will interact with agents
- Agent errors can compound when looping
Thanks to
@hwchase17
and
@Kleida_Martiro
for a great panel!
Today we are starting our first ever Launch Week! 🔥
For this week, we’ll be releasing a new feature daily. We're saving the best for last so stay tuned 👀
Day 1 - Introducing Teams: Invite your co-workers to build with you in Firecrawl
@mendableai
has served a TON of messages using GPT 3.5. Through development + the custom prompts feature we learned about a couple concepts that can significantly improve 3.5 performance and consistency:
- Repetition
- Structured prompts (1/4)
Lastly, filtering the results is a very important. For both the semantic and keyword-based search we set relevance thresholds documents must hit before being included in the prompt context. For each corpus of data some experimentation is needed to find the right thresholds. (5/6)
One of the most exciting things about Mendable is the ability to pass in application context directly to the AI to create context-aware copilots 🧑✈️
Below, Mendable AI knows about the user's project and what page they're on
It then responds with the project's context in mind 🤯
We are very excited to launch our new web ingestion UI template!
Now you can easily create ingestion flows for your AI apps by using this UI.
Inspired by
@dify_ai
and built with
@shadcn
components 🔥
First, finding the right keyword search function for your use case is important. We opted to build our own search function using Postgres full text search, but would love to see hybrid search implemented with something like
@meilisearch
(3/6)
We just broke our daily gross volume record with
@firecrawl_dev
It's clear that web scraping is still a really hard problem and is only getting more important...
Introducing MocharyGPT🚀
Now you can get advice based on the legendary Mochary Method instantly. All powered by
@mendableai
's new google doc loader.
Before YC, I was introduced to
@mattmochary
's method on
@justinkan
’s pod and it was enlightening as a founder.
(1/3)
been using the
@mendableai
search on langchain very frequently while writing code lately.
it (almost) doesn't make sense to go back and manually read docs anymore.
all in-app search will slowly shift from serving an index of info to personalizing retrieval from these indexes.
Tired of search APIs giving you just a taste? 😒
Introducing /search: SERP + Firecrawl = FireSearch🔥
Say goodbye to search results with no actual page content! Think of the LLM use cases for this... 🧵👇
TypeScript AI devs 👋
I'm excited to open source Agentic, a standard library of AI tools that work across any LLM and TypeScript AI SDK.
All AI tools are usable both as normal TS classes as well as AI functions via decorator magic.
GitHub:
Details: 👇
Introducing SmartCrawl by
@firecrawl_dev
- Turn any website into an API with AI
Watch the AI in action:
- Find the top 10 AC units on Amazon
- Extract and format JSON product data
- Generate a reusable automation accessible via API
Built with
@e2b_dev
@browserbasehq
Launch Week Day 3: Introducing the Map Endpoint 🗺️
Our new /map feature helps you quickly gather URLs from a website.
In addition, you can also use the search parameter to find relevant URLs in a site.
While testing the
@mendableai
router Alpha, we were able improve classification performance by 20% with prompt engineering.
Here is what we found (1/3)
Once upon a time,
@kiwicopple
from
@supabase
asked me about my dream feature.
I asked for an AI chatbot in the supabase console that could write queries for me based on my database schema.
So I prototyped it with
@mendableai
and
@LangChainAI
We're proud to power
@MeetGamma
's new Import from URL feature!
With this new integration, users can generate entire presentations from any website in under a minute ✨
Introducing Enhanced Dashboard Monitoring 📝
We are excited to give developers more transparency into every scrape and crawl request.
See processes, timing, failures and more on the improved Activity Logs page 🚀
there are 246 companies in the latest
@ycombinator
batch (W24)
72 of them use
@supabase
: 29% of the batch
Build something people want, with Supabase ⚡️
Applying to
@ycombinator
has been one of the most impactful things we have done for .
Truly a transformational experience for both our startup and us as founders.
My advice? Just apply... you can even ask
@mendableai
Generative AI is already making a huge impact for many Customer Success teams in 2024.
Not sure which tools are the best? Here are a few of my favorites:
Next, getting the correct keywords is important. With natural language queries you will have a lot of unnecessary stopwords. Removing these will increase performance, and there are a bunch of packages like “keyword-extractor” that can help. (4/6)
Just pushed v0.0.3 of the
@mendableai
Python SDK where you can build managed, production-ready chat apps in 4 lines of code.
You can ingest
@Github
repos, websites, and more thanks to
@LangChainAI
's data connectors.
Here is a link to the repo:
Introducing Fire Engine - The backend powering Firecrawl
It is the most scalable, reliable, and fast way to get web data.
Compared to leading competitors:
- It scrapes 40% more websites
- It is up to 33.17% faster
Read more in our official launch post 👇
Whether you're an AI professional, researcher, or enthusiast, you won't want to miss the opportunity to network, gain insights from industry leaders, and be part of the evolving world of LLMs.
Space is limited so make sure to reserve your spot today:
Some of the agentic AI startups I plan on testing this weekend - any faves?
↓
Crew ai
MSFT AutoGen
Lutra
Otto grid
Gumloop
TextQL
Athena intelligence
Nexus GPT
AutoGPT
Multi-on
Firecrawl
Reworkd