BentoML - Build and Scale Compound AI Systems
@bentomlai
Followers
2,241
Following
197
Media
82
Statuses
476
Compute orchestration platform for rapid and reliable GenAI adoption, from model inference to advanced AI applications. 🍱
San Francisco, CA
Joined May 2017
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Vance
• 2396724 Tweets
Vance
• 2396724 Tweets
#VPDebate2024
• 332546 Tweets
Pence
• 136253 Tweets
Minnesota
• 134115 Tweets
Tampon Tim
• 96367 Tweets
Tiananmen Square
• 65895 Tweets
महात्मा गांधी
• 64225 Tweets
LINGLING MYCHERIE AMOUR EP4
• 62996 Tweets
からくりサーカス
• 61958 Tweets
गांधी जयंती
• 59758 Tweets
#原神KFCコラボ
• 57910 Tweets
लाल बहादुर शास्त्री
• 56557 Tweets
#GandhiJayanti
• 55539 Tweets
ケンタッキー
• 51354 Tweets
Obamacare
• 32481 Tweets
ニンテンドーミュージアム
• 28322 Tweets
リネット
• 26305 Tweets
Timmy
• 23926 Tweets
Vladimir
• 23703 Tweets
宮崎空港
• 17545 Tweets
ネットオーダー
• 17181 Tweets
原神コラボ
• 13907 Tweets
#LalBahadurShastri
• 11043 Tweets
Pinned Tweet
🚀Join the first LIVE! AGI Builders Meetup on 10/02, 9 AM, PT!
❓Self-hosting LLM vs Serverless API
🔍Uncover hidden costs & Optimize performance
💡Live Q&A with
@bentomlai
CEO
@chaoyu_
💤Can't attend? Register to get the recording!
#AI
#LLM
2
1
2
🎉 Big shoutout to
@Manjusaka_Lee
who helped us identify the root cause of a memory leak using his professional debugging skills and firm knowledge of Unix!
Your dedication and expertise is truly appreciated. Thank you for your invaluable contribution! 🙌
5
7
91
Dive into the synergy of OpenLLM and
@llama_index
in our latest blog post! Leverage the power of open-source LLMs and learn to build RAG systems that understand and respond accurately to your custom dataset. 🔗
3
18
53
🦄Yatai preview release is finally here! A new project from the BentoML team, helping ML teams to run model serving at scale on Kubernetes.
#BentoML
#Yatai
#MLOps
#MachineLearning
0
17
50
🚀 Just dropped Part 1 () of our blog series on Private
#RAG
Deployment with
#BentoML
! 🌟 Here’s what to expect:
🦙 Part 1: Serving A
@llama_index
RAG App as REST APIs
🤖️Part 2: Self-Hosting
#LLMs
and Embedding Models for RAG
☁️ Part 3: Multi-Model
0
12
46
🎉Integration alert:
@huggingface
, BentoML &
@DeepFloydIF
!🚀
🤔Think bringing your ML models to production is a daunting task?😮It doesn’t have to be! Explore the smooth deployment of state-of-the-art models like DeepFloyd IF on Hugging Face, all made possible by BentoML’s
1
7
43
This is going to be an exciting week! We've released a BentoML course as a part of MLZoomcamp! Check it out on github. Thanks
@Al_Grigor
for supporting us as we built this content.
1
4
30
🎉 Warm welcome to
@ShawnBasquiat
, our first Developer Advocate at
#BentoML
! 🌟👨💻 With his extensive experience in building dev communities, Shawn is a brilliant addition to our endeavor to strengthen relations with developers! Super excited to have him on the team and can’t wait
2
3
23
The most advanced ML model serving platform powered by our open source project BentoML
0
2
16
🚀 Elevate global communication with OpenAI Translator ()! This open-source project comes from
@yetone
from
#BentoML
, an all-stack engineer passionate about
#OpenSource
and
#AI
. It uses the
#ChatGPT
API for seamless translation, polishing, and summarization
0
2
15
🤔 Many organizations may choose proprietary models from providers like OpenAI for quick prototyping, but this can lead to limited control & customization. 🔒 In regulated industries, data sovereignty and security is crucial, pushing teams towards open-source models.
🛠️ These
1
4
15
💡 The integration between
@milvusio
Lite and
#BentoCloud
provides another way to build
#RAG
! In this blog post (), we use pre-built Bentos on
#BentoCloud
to deploy inference services for embeddings and querying, with Milvus Lite as the vector database. 👀
0
3
15
We conducted a benchmark study with the Llama 3 8B and 70B with 4-bit quantization on the LMDeploy inference backend.
LMDeploy consistently delivers low Time to First Token (TTFT) and the highest decoding speed across all user loads. Its ease of use is another significant
0
1
13
🚀 Just released a comprehensive benchmark on
#LLM
#inference
backends ()! Essential for every AI/ML engineer to know how these tools can enhance the performance & reliability of LLMs.
In our benchmark study, we evaluated the serving performance of
#Llama3
0
7
14
We conducted a benchmark study with the Llama 3 8B and 70B quantized on the vLLM inference backend.
vLLM consistently maintains a low Time to First Token (TTFT), even as user loads increase, making it suitable for scenarios where maintaining low latency is crucial.
vLLM offers
2
1
14
🚀
#Llama3
has arrived 🦙🦙🦙- the next generation of
@AIatMeta
state-of-the-art
#OpenSource
#LLM
! Want to deploy it? 🍱
#BentoML
has got you covered! Try this example project to self-host it! All the code is already there and all you need is starting the
1
3
13
🚀 PDM () is a
#Python
package manager created by a
#BentoML
software engineer
@frostming90
, driven by his passion for
#AI
,
#ML
, and
#OpenSource
. It tackles the headaches of dependency management, especially for large binary distributions. Check it out and
0
7
13
We are hiring multiple engineering roles to join our founding team, if you like building open-source software and passionate about
#MLOps
, come join us!
1
4
11
Awesome article from
@KhuyenTran16
about using
@bentomlai
to simplify model deployment!
That bento box graphic is making me hungry! 😄
0
5
10
Thank you to all the wonderful techies that came to the
#AGIBuildersMeetup
meetup yesterday at
@Cloudflare
HQ!
We appreciate you!
0
2
11
🚀Excited to see all of you tonight at the second
#AGIBuildersMeetup
! Thank all of you and see you next time⚡️!
@datastrato
@bentomlai
@MindsDB
0
1
12
🚀🌟 Dive into
#BentoML
in this blog post by
@ShawnBasquiat
! Explore everything from AI deployment to scaling with ease. 👉 Link
#AI
#MachineLearning
#BentoML
1
3
11
Excited to officially launch Bentoctl for deploying bentos into any cloud environment! Users want flexibility and choices when deploying which is why we include customizable terraform scripts.
Thanks for this great writeup by one of our founders
@bozhao
0
2
9
📷📷
@TomTom
x BentoML = Advanced AI solutions in navigation! Dive into our latest blog post to see how we're merging maps and AI for smarter mobility. 📷📷 Link:
#TomTom
#BentoML
#GenerativeAI
#MachineLearning
#AI
0
4
11
Checkout our latest article by
@bozhao
on how to deploy Stable Diffusion easily to AWS with BentoML!
Finally a way for everyone to see the power of AI harnessed alongside our own creativity
#stablediffusion
0
5
7
We're thrilled about the release of Llama 3.1 by
@AIatMeta
! Today marks a milestone for the future of open-source AI, possibly the most significant open-source AI announcement ever! Llama 3.1 introduces a collection of frontier models that rival proprietary models in performance
0
2
9
🙏 Huge thanks to everyone who came out to the
#AGIBuildersMeetup
SF last night! We had amazing talks on the latest
#AI
technologies! Special thanks to our speakers:
@zarazhangrui
@CozeHQ
@ShawnBasquiat
@bentomlai
Francisco Castillo
@arizeai
#OpenSource
#MachineLearning
0
0
10
Missed last week's session on scaling private
#RAG
? Here is the video: 🧐 Gain insights into production RAG challenges and learn how you can use
#BentoML
to package and scale all RAG components efficiently! 🙏Thank
@PhilipTannor
and
@llmopsSpace
for this
0
3
9
BentoML project has just got its 50th contributor! Thank you all for making it awesome!!
0
2
8
We have worked together with
@OutlinesOSS
to create a step-by-step tutorial on producing structured data outputs with
#LLMs
👉 ! Try it locally or on
#BentoCloud
!
#MachineLearning
#AI
#OpenSource
#StructuredData
🚀📚✨
0
3
9
It’s finally here! Our official 1.0 launch of BentoML! 🚀🚀🚀 We’ve spent years iterating on different architectures and features to finally arrive at what we believe is the standard model serving architecture of the future. 🎉🎉🎉
0
4
8
🌟 New blog drop! Learn how you can use
#vLLM
and
#BentoML
to build an
#LLM
server and deploy it on
#BentoCloud
for optimal efficiency & scalability in the cloud. 💡🚀 Thank all the vLLM maintainers for this wonderful project!
@woosuk_k
@zhuohan123
0
4
9
Stability AI has recently released Stable Diffusion 3 Medium, featuring significant improvements in image quality, typography, complex prompt understanding, and resource efficiency.
However, there are mixed opinions about its performance, especially in generating human figures.
0
0
9
🙌 Thanks to everyone who made the AGI Builders Meetup a success today! Special thanks to our speakers from
@twilio
@bentomlai
@arizeai
@Tabby_ML
@revia_tech
! Stay tuned for our next meetup!
0
2
9
💡Ever wondered how
#OpenLLM
synergizes with
@LangChainAI
&
#BentoML
? Dive into our blog post () and create a self-introduction generator with ease! 📖 Don't miss out!
#AI
#MachineLearning
0
1
9
BentoML 0.9.0 release is here!! Revamped micro-batching API design, adding
@huggingface
support, faster
@docker
image build, improved dependency management, and more!
0
3
8
Yatai 1.0 automates deployment to Kubernetes without requiring complex yaml. Simple software abstractions enable users to scale at their own pace.
📈 Automatically distribute Bento model components into separate auto scaling groups
0
2
7
🔔 Join us at the next
#AGIBuildersMeetup
in SF! Connect with
#AI
enthusiasts for tech talks & networking! 🗓️ July 30th, 5:30 PM - 8:00 PM PDT
And here is what you can expect:
✅ Rogue Agents — Stop AI from misusing APIs
@twilio
✅ Choosing Your Champion: LLM Inference Backend
3
5
8
🤔 Composing multiple
#AI
models can help us in a variety of use cases like sequential and parallel processing, inference graph, multimodal applications, and ensemble modeling. Read this
@thenewstack
article () to learn about these topics and how
#BentoML
0
3
8
🎉 Just dropped OpenLLM in Action Part 1: Understanding the Basics of OpenLLM - the first installment of our new blog series about OpenLLM! Embark on an AI odyssey with us and grasp the essence of OpenLLM. 🔥Read now:
#OpenLLM
#OpenSource
#BentoML
0
3
8
Excited to see our first article in
@TDataScience
! Our CEO
@chaoyu_
writes about design considerations for production model deployment systems.
1
2
6
Unlocking the secrets of how to productionize AI apps with
@fog_glutamine
's keynote speech at
#KubeCon
China this year! 🚀 Consider transforming open-source
#AI
models into cloud-ready apps in two steps - build and deploy. Then solve the challenges with
#BentoML
,
#OpenLLM
, and
1
3
8
Lookout! We’ve got a big launch coming up this month! More to come in the following weeks 😉
Have a great weekend!
And to our friends in the US, Happy 4th of July! 🎉
0
1
6
Are you looking to harness the power of
#AI
while safeguarding sensitive data and models? Register now to join
@anjuna_security
&
@bentomlai
's webinar on May 29th at 11am PDT. Learn how Model Serving can ensure secure & performant
#AI
initiatives for
0
0
8
🤔 Why do enterprise AI teams choose BentoCloud’s BYOC option? 🔒 Privacy, 💰 Cost efficiency, 🎯 Credits/Accounts, 💻 GPUs, & 🌐 No vendor lock-in. Read this blog post () to learn how you can deploy
#AI
applications in your own cloud with full control and
0
2
8
🚀 Dive into building private
#RAG
applications with our CEO
@chaoyu_
on a session with
@llmopsSpace
! The session will be hosted by
@PhilipTannor
, CEO and Co-Founder at
@deepchecks
.
🛠️ April 4, 8 AM PST. 👉 Register here:
#AI
#MachineLearning
#BentoML
0
3
8
Learn about an event-driven ML workflow using BentoML, Tekton, and KFServing - Cloud Event Horizon, a talk by
@ianmcoffey
from
@vmwopensource
0
3
6
The date drop is finally here! 🍱🍱🍱
We will be officially releasing BentoML 1.0 next week!
July 14th, 2022 - 🚀🚀Join us for our official launch party:
Blog post to follow about new features, architecture and opportunities that this opens up
0
0
6
🎉 Introducing
#BentoML
1.2! 🎉 Blog post:
New features await:
🛠️ Simplified SDK for more control
👀 Familiar input and output types integrating Pydantic
🚀 Direct deployments with `bentoml deploy .`
🖥️ New
#BentoCloud
Web UI for intuitive API interaction
0
0
6
Deploy ML models to production! Thanks for this awesome video
@musikalkemist
! Very cool to see you teach how to create a production level ML service. 🔥🔥🔥
0
2
5
0
5
6
“BentoML: Create an ML Powered Prediction Service in Minutes” by Nisar Ahmad
0
0
6
🎉 Really excited to see all of you last night! We had a wonderful meetup on
#AI
and
#MachineLearning
! 🙏 Big thanks to all the speakers
@echojuliett
@upstageai
,
@chaoyu_
@bentomlai
, and
@wenphan
@Voyage_AI_
! If you have an idea you'd like to present at future meetups, please
0
0
7
Don’t miss out on another fantastic event in San Francisco on June 3rd with
@TectonAI
&
@zilliz_universe
.
Learn all about
#RAG
:
❄️ RAG with self-deployed
@milvusio
and Snowpark Container Services
💯 Full RAG: A Modern Architecture for Hyperpersonalization with
@TectonAI
🌐 RAG
0
1
7
According to Gartner, 41% of orgs face
#AI
security breaches and 27% have faced intentional malicious attacks on their AI infrastructure 😮! It's time to secure your AI deployment and
#BentoCloud
is designed with that goal at the very beginning.
With BentoCloud, you are able to
1
0
5
Calling all
#AI
enthusiasts in
#SanFrancisco
! Join us for the
#AGIBuildersMeetup
on 5/21 () and learn the latest advancements in AI:
🤖️Uncovering use cases for
#LLM
-powered chatbots by
@zarazhangrui
, Head of GTM,
@CozeHQ
🍱 Multi-Modal Orchestration with
0
4
6
What is model monitoring? Why is it important? Our Lead Engineer
@Hrmthw
explains the different types of ML monitoring and shows how BentoML provides simple APIs for building a robust monitoring pipeline
#machinelearning
#monitoring
0
2
5
Check out this guided project on
@coursera
about Machine Learning model serving and deployment with BentoML
0
3
5
🚀 Thrilled to introduce OneDiffusion: Your go-to open-source platform for deploying diffusion models, such as SD v1.4, v1.5, v2.0 & SDXL v1.0!
Experience hassle-free deployment and serve your fined-tuned models with LoRA weights 😮! Check out our GitHub repo and harness the
0
2
5
Getting started with BentoML - serve and deploy your trained machine learning models in the cloud with just a few lines of code
#modelserving
#mlops
#machinelearning
0
6
6
🚀 In our latest blog post (), we draw from our experience working with
#BentoML
customers to discuss the secrets of building production-ready, scalable
#RAG
with
#OpenSource
and custom
#AI
models 🧐.
1
4
6
🔥
#StableDiffusion3
is here, but is it the ultimate winner in image generation? We put it to the test! 🖼️💡
Read our latest blog post () for a comparison of
#SD3
with
#SD2
and
#SDXL
, focusing on text rendering and human figures. Also, we explore different
0
2
6
Check out Axel Mendoza’s latest blog post about the “Best Model Serving Runtimes for Heavy ML Workloads”
Axel provides an in-depth analysis of the latest technologies in model serving, with practical tips and recommendations for AI and ML practitioners.
Read the full post and
0
1
6
Flask and FastAPI were designed for IO-intensive applications (web applications). ML workloads require a different architecture because they are often compute and memory intensive.
0
2
6
ML is not just for the most high tech companies anymore, we see it being used in all types of applications and growth is only accelerating. BentoML helps you ship more of your models to production:
0
1
4
Llama 3.1 is now supported in
#OpenLLM
!
openllm serve llama3.1:8b
openllm serve llama3.1:8b-4bit
openllm serve llama3.1:70b
openllm serve llama3.1:70b-4bit
openllm serve llama3.1:405b
Learn more:
#BentoML
#Llama3
#OpenSource
Starting today, open source is leading the way. Introducing Llama 3.1: Our most capable models yet.
Today we’re releasing a collection of new Llama 3.1 models including our long awaited 405B. These models deliver improved reasoning capabilities, a larger 128K token context
284
2K
6K
1
1
5
🔥 Exciting news! 🔥 BentoML has been listed in the
@CBinsights
LLMOps market map! 🌊 Featured in both AI Development Platforms AND Model Deployment & Serving. See this post () to learn more!
#AI
#BentoML
#OpenLLM
#LLMOps
#MachineLearning
0
1
6
🤖 Ready to unleash
#AI
potential with
#RAG
? Just 3 days to the
@llama_index
RAG
#Hackathon
! 🍱 Our CEO
@chaoyu_
will be one of the judges evaluating your AI projects! Sign up now and be part of this innovative AI journey! 🔗
1
1
6
Join us 9/29, 1pm PST for AMA with Aparna Dhinakaran. She is the Co-Founder and Chief Product Officer at
@arizeai
, an ML observability platform.
link to join👉:
1
2
6
🚀 Huge thanks to
@frostming90
for adding
#uv
to
#OpenLLM
v0.6! Now deploying
#LLM
APIs is much easier and lightning-fast with auto dependency management. No more Python or CUDA version conflicts! 🙌 Kudos to Frost for making AI hackers' life easier!
#OpenSource
#AI
0
0
6
After years of work, we’re proud to announce that next week, we’ll be launching Yatai 1.0!!! 🚀🚀🚀
Yatai is an open source, production-first platform that simplifies model serving and deployment.
Join us for our live launch event on November 10th!
1
2
5
Super excited to launch our new blog at ! We've got some great resources on model serving and more to come.
Practical guides as well as our thoughts on the future of ML 🤓
0
1
3
Learn how to unlock the full potential of LLMs in our next workshop with
@arizeai
on 10/12! Register 👉
From building a robust
#LLM
app with
#OpenLLM
to evaluating & troubleshooting it with
#ArizePhoenix
, we've got it covered. Perfect your LLM deployment
0
2
5
We're excited to share our experience and insights on scaling
#AI
models! 🌟 From
#OpenSource
models, cold start problem, and to using concurrency as a scaling metric on
#BentoCloud
, our blog post () has it all! Get the comprehensive info on making AI
0
0
5
#BentoML
can help you navigate deployment and scaling. 👋Join our virtual event () with
@CloudNativeFdn
on Thursday, May 23rd at 12 PM CST.
Our Head of Engineering
@sean_sheng
will cover the following topics:
✅ Discover best practices for handling cold
0
1
5
Love this video by
@MatthewBerman
showing how OpenLLM makes it easy for developers to create incredible apps on top of open-source LLMs while having first-class support for tools such as
@LangChainAI
,
@huggingface
Agents, and
@bentomlai
.
1
3
5
#AMA
with Gabe from
@openlayerco
,
@ycombinator
(S21) on debugging ML models
Q: Can you tell us what's the founding story?
A: Rishab, Vikas, and I were ML engineers at
#Apple
, mainly working with Siri. We noticed how hard and boring it was to debug and test models...
1
1
4
Continuous retraining is often used to avoid data drift and keep models fresh. Check out Jian Shen Yap writing about real world experience building services that retrain and redeploy with BentoML!
Awesome write up from the team at Shopback!
0
0
4
Yatai 1.0 was built with one of our favorite features of ALL TIME!
🎛️ Support for logging into containers in real-time and an audit log which allows you to REPLAY terminal sessions!
Join us tomorrow for our live launch event!
0
0
4
🚀 Last week, open-source AI saw a game-changer: Llama 3.1 with 405B parameters, rivaling top proprietary models! Right after, Mistral Large 2 dropped 🔥. With so many models emerging, it's crucial to understand what each brings to the table.
In this
0
1
5
For the first time ever, BentoML 1.0 is introducing a first class integration with MLflow!
Now you can build your models with MLFlow and then easily serve and deploy at scale with BentoML.
Check out the documentation for it now!
0
1
3
Date: 9/29 1pmPST
What: AMA with
@aparnadhinak
from
@arizeai
on
#MLOps
toolchain, Productionalizing your research model
Where:
@bentomlai
community slack. Join here👉:
0
3
4
Bento❤️gRPC! Read about why ML scientists want to use gRPC and how to use them ->
0
1
4
📢📢📢 We are excited to announce that Yatai project has entered the 1.0 alpha release phase. 🚀🚀🚀
Yatai helps run model serving at scale and simplifies model management and deployment in
#kubernetes
.
0
3
5
🎉 A big thank you to everyone who joined us at the
#AGIBuildersMeetup
in NYC today! It was inspiring to see so many AI enthusiasts and builders!
@humansignal_
@bentomlai
@TectonAI
👀 Stay tuned for more events like this. We look forward to seeing even more of you at our future
0
0
4
🚀
#BentoCloud
is now available on the
#GCP
Marketplace! 🌐 An inference platform that takes the infrastructure complexity out of production
#AI
workloads, BentoCloud empowers enterprise AI teams to build fast, secure, and scalable applications right in their own cloud
0
0
5
BentoML and
@arizeai
are thrilled to announce our partnership that streamlines the machine learning development lifecycle to supercharge production ML.
Learn more about from our blog:
0
1
5
OpenLLM now supports running Llama2 2X faster with new vLLM runtime.
vLLM is a framework optimized for fast inference serving. It implements efficient attention mechanisms such as PagedAttention and optimized CUDA kernels. Running Llama2 7B on OpenLLM with vLLM optimization
0
2
4
What an amazing post from
@ahmed_besbes_
on TDS! It shows how to build, ship and scale a production ML service in a concise tutorial format.
⚡ Train a model with PyTorch Lightning
🍱 Build an ML service with BentoML
💥 Deploy and scale with Bento Cloud
0
2
5
🎉 Big wins at the
#LlamaIndex
#RAG
#Hackathon
! 🚀
BentoML 1st 🥇:
#StreamLens
➡️
BentoML 2nd 🥈:
#PreciousPixels
➡️
Thanks to
@DataStax
,
@llama_index
and all the other partners and sponsors for a successful event!
#BentoML
0
0
5
An exciting blog post on how to unify
#realtime
and
#batchinference
with
#BentoML
and
#ApacheSpark
!
This approach improves performance, scalability, and cost savings. Check it out at
#mlops
#modelserving
#opensource
0
3
5