Portkey
@PortkeyAI
Followers
1,285
Following
619
Media
197
Statuses
659
LLM tokens processed through @PortkeyAI today: 12 BILLION Bring Your LLMs in Prod! Backed by @lightspeedvp
Leave a Star →
Joined March 2023
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
bambam
• 183492 Tweets
Pete Rose
• 143533 Tweets
Karina
• 100890 Tweets
#WWERaw
• 97679 Tweets
PlayStation
• 92507 Tweets
Lions
• 81270 Tweets
Dolphins
• 43656 Tweets
Hall of Fame
• 43544 Tweets
Titans
• 43045 Tweets
LOVE SENIOR SPECIAL Q1
• 40462 Tweets
Detroit
• 38836 Tweets
Seahawks
• 37138 Tweets
Seth
• 32933 Tweets
#BakeOffFamosos
• 28489 Tweets
Levis
• 28459 Tweets
Seattle
• 25934 Tweets
あと3ヶ月
• 20583 Tweets
コーヒーの日
• 20571 Tweets
Goff
• 19030 Tweets
#SEAvsDET
• 18267 Tweets
Braun
• 17128 Tweets
都民の日
• 16631 Tweets
Christ is King
• 16222 Tweets
Respecting Ancestors
• 13323 Tweets
Geno
• 12795 Tweets
Charlie Hustle
• 12363 Tweets
#OnePride
• 12167 Tweets
Huntley
• 11858 Tweets
McDaniel
• 11226 Tweets
Gunther
• 10067 Tweets
Tyreek
• 10064 Tweets
Pinned Tweet
In August,
@PortkeyAI
crossed 2 Billion total requests processed through our platform 💥
(this number was 0 last year!)
We're humbled to be production partners for thousands of leading AI companies around the world, like
@getpostman
@quizizz
@italic
@Pepper_Content
1
3
9
📢 Exciting News!!
Portkey is now natively integrated with
@llama_index
! 🎉
Building resilient and production-ready LLM apps has just become extremely easy 💆
Portkey adds 4 core production capabilities to any Llamaindex app: ⬇️
2
5
31
1
2
25
Now Presenting: AmulGPT - put in a news article and get one of those classical Amul topical ads!
Featuring
@swyx
in the house!
🚨 Bangalore! Emergency Hackathon. Tomorrow.
Elite builders. OpenAI's latest. (scary) Limited spots.
Food, credits (and sleep) - on us.
Go: 🏃
1
2
10
2
4
22
|￣￣￣￣￣￣￣￣￣￣￣￣￣|
| Guardrails on the Gateway |
|＿＿＿＿＿＿＿＿＿＿＿＿＿|
\ (•◡•) /
\ /
---
| |
1
4
21
Over *$8 Million USD* in committed investments & credits are available for early-stage AI startups today.
But where can you find them?
We scoured the depths of internet (and Twitter) to compile just that!
🥁 Introducing.. The AI Grants Finder
1
6
18
Is it ok to ship something new on a Saturday?
Portkey now supports models hosted on Groq's super fast LPUs!
16
2
16
🚀 Portkey now natively integrates with Tembo to conduct vector searches on Postgres
Use multiple LLM providers to create embeddings on
@tembo_io
- without changing your setups
Try it out:
1
3
17
🚀New Guide on the Portkey Blog: DSPy in Production
- by
@ganarajpr
, based on his talk at the LLMs in Prod BLR meetup.
Learn how to leverage DSPy by
@lateinteraction
to tackle real world challenges, optimize AI pipelines, and revolutionize e-commerce operations.
Read it now:
0
8
17
Portkey integrations ranked by awesomeness:
#1
@vercel
(
@jaredpalmer
et al)
#2
DSPy (
@lateinteraction
et al)
#3
Instructor (
@jxnlco
)
#4
@promptfoo
(
@iwebst
)
#5
@phidatahq
(
@ashpreetbedi
)
#6
@ControlFlowAI
(
@AAAzzam
)
#7
@tembo_io
(
@rywalker
)
..and 40 odd more.
Portkey is now a first class provider in the
@vercel
AI SDK.
When should you use Portkey's AI gateway with the AI SDK?
1. You're routing requests to multiple models & providers.
2. You're going to production and need metrics on cost, performance and accuracy
3. You want to
1
0
13
1
3
15
Live Now: LLMs in Production Event from SF! 🌟
We're kicking off an insightful evening exploring production-specific questions and issues on scaling LLM apps!
Link to join the livestream in the next tweet! ↓
feat.
@lightspeedvp
@databricks
@llama_index
@getpostman
@yi_ding
3
4
15
Happy to confirm this! We've observed millions of Gemini 1.5 Flash requests over the last 3 months, and the latency has steadily dropped 3x
We just shipped a series of changes which have significantly improved the Gemini 1.5 Flash latency (>3x reduction) and output tokens per second (>2x more)⚡️🚢
91
128
2K
0
0
13
New York, you can't miss this.
Catch
@jumbld
and
@ayushgarg_xyz
in the first ever LLMs in Prod meetup on the east coast.
We'll be sharing our hard won lessons in shipping AI apps to production.
Limited seats. Please register soon! Link ↓
1
5
13
Thrilled to talk about Portkey's
@F5
partnership at the
@RSAConference
!
There is not a year in the past 8 years that I have not used nginx
With the Portkey AI gateway, the best platform teams are at the best position for AI too :)
Pro tip: Lua unlocks the unlimited nginx power
1
2
7
1
1
13
🚨 Heads Up: Major Update from OpenAI! 🚨
Starting Jan 4, say goodbye to 33 models, including the iconic text-davinci-003 (aka GPT-3). This is the largest model sunset at OpenAI yet! 🌆
What's changing? Read on! ⬇️ 1/3
1
2
12
Billions of tokens. 200+ LLMs. 5,000 Github stars.
Behind these numbers lies a story waiting to be told.
Join Portkey's CEO
@jumbld
at the Bangalore Gen AI Meetup, as he unveils the lessons learnt building a massive-scale OSS AI gateway — a critical component in unlocking the
0
1
12
.
@jumbld
is live on India's national broadcaster, discussing India's efforts towards Gen AI and how Portkey is empowering AI innovation!
0
1
12
While
@AnthropicAI
's Claude Sonnet 3.5 has seen very quick adoption in production,
@GoogleDeepMind
just released the new Gemini 1.5 Pro experimental (0801) model which is now ranked
#1
on the LMSYS Chatbot Arena leaderboard, neck-and-neck with GPT-4 and Claude 3.5 Sonnet!
The
1
2
11
What’s cooking?
0
7
25
1
3
11
Streaming from cache is now live and is supported in these 3 portkey routes:
/v1/chat/completions,
/v1/completions and
/v1/prompts/:id/completions.
available across all providers.
We also cache function streaming calls now!
0
0
10
🔊 Sound on
BIG Update to Portkey — the URL slugs for all pages are now ✨clean✨
but there's more 👇
0
1
11
New Portkey +
@LangChainAI
Cookbooks ✨ thanks to
@yujian_tang
!
See how easy it is to start making your Langchain agents reliable, cost-efficient, and production-grade using Portkey:
Explore the Cookbooks:
- Portkey + OpenAI + The Milvus Project RAG Agent:
0
2
11
Over the last 8 months, Portkey has processed a total of 300 Billion+ tokens across 100+ LLMs
Every day, more and more of our customers are taking their LLM apps to production using Portkey
And this is just the start.
0
2
10
We are open sourcing our Guardrails on the Gateway framework today.
But why is this important?
We built the gateway to bring some uniformity to so many LLM APIs that are out there, and added fallbacks, load balancing to make the routing more reliable.
That was a good start!
We're launching our open-source AI guardrails framework on our AI gateway today.
Been building it with inputs from 600+ teams who use the gateway in production and have collectively made 1.4 billion API requests on our hosted platform itself!
trying our luck with an HN launch
17
33
174
1
2
10
Capturing a special moment from our community! 📸
Mohamed shared some fantastic feedback about our features at Portkey - from seamless integrations to our game-changing analytics, it's feedback like this that keeps us going 🚀
1
2
10
Our LLMs in Prod server is about to hit 200 members! 🎉
To celebrate, we are giving limited edition Portkey merch to the 200th new member - which includes the apocryphal Yud t-shirt, and more.
Who will be the lucky 200th? RT for karma!
➡️
1
4
10
🔥 Day 0 support for the new gpt-4o-2024-08-06 model on Portkey!
It's 50% cheaper on inputs and 33% cheaper on outputs.
Try it out now! 👇
0
1
9
🚨 Bangalore! Emergency Hackathon. Tomorrow.
Elite builders. OpenAI's latest. (scary) Limited spots.
Food, credits (and sleep) - on us.
Go: 🏃
1
2
10
.
@MistralAI
just released Codestral Mamba, a revolutionary 7B parameter coding model based on the Mamba architecture.
With linear time inference, 256k token context, and Apache 2.0 license, it's set to transform code generation.
Try out the Codestral Mamba model using Portkey!
1
2
9
Ever stumbled upon a buggy request in your logs and wished you could instantly rerun it to fix it?
Say Hello to Log Replays! 🔄
With Log Replays, you can open the full request in a fresh prompt playground straight from your logs — easily replay the buggy request and edit it
0
2
8
If you're looking for an easy way to switch to the latest OpenAI embedding models, configs are a great way to do that.
You can switch to the latest models without touching your codebase. Also lets you try multiple models in parallel as you experiment.
We call this
0
0
4
🚀 Last week, we had an absolute blast co-hosting the AI Agents: Inter-continental Hackathon with
@meetaugustai
at their BLR Chapter!
24 hours of pure innovation, collaboration, and building the future of AI agents.
A massive shoutout to all the participants, our co-hosts-
1
2
10
Today is a special day - we raised $3M
But something more important happened:
@isro
's Chandrayaan3 became the only mission in the world to land on moon’s south pole🌖
As we build a world-class LLMOps platform out of India, we take immense inspiration from what ISRO has achieved
3
1
8
Excited to share insights on the emerging AI Gateway pattern at AI Tinkerers Bangalore tomorrow!
RSVP:
Quality of demos at tomorrow's second AI Tinkerers Bangalore is 🤯 -- don't sleep on India for GenAI 🇮🇳 🚀
5
3
24
0
1
8
Portkey 🤝 OSS Friends
0
0
3
0
1
8
🚀 AI Devs! Don't miss this:
@jumbld
in conversation with
@yujian_tang
on "Enforcing real-time LLM behavior with Guardrails on the Gateway"
Learn how to
→ Build secure, robust AI apps
→ Prevent hallucinations
→ Implement effective guardrails
👇 Registration link below:
1
2
8
Next up, was
@RajaswaPatil
from
@getpostman
on building Postbot
From disjointed features to a smooth-talking AI assistant - this team's journey with Postbot is a masterclass in evolving AI architectures.
Multi-agent systems, here we come!
1
2
7
🔔Calling all Bengaluru AI builders🔔
Learn how to build production-grade & reliable AI apps this Sat at the
@huggingface
@Inferless_
party
Our CTO
@ayushgarg_xyz
will demo how you can add critical production functionalities on top of your existing LLM workflows with Portkey✨
2
3
7
A massive thank you to
@meronogbai
& Luke Vanagas for their recent contributions to the Gateway!
Meron's contribution adds support for multiple user messages with Google Gemini, and Luke's implementation of Anthropic's input_tokens and output_tokens response params allows
0
2
7
Attention Isn't All You Need
Mamba: A New Approach to LLMs. This State Space Model achieves Transformer-level performance with RNN-like efficiency.
@MistralAI
's Codestral Mamba, with just 7B parameters, outperforms many larger, widely-used LLMs.
We've done a deep dive to
1
1
8
.
@LangChainAI
& Portkey, better together.
Supercharge your Langchain apps & agents with Portkey's production-grade tools: logging, tracing, caching, tagging and retries.
All with 1 LoC change
🗃️ Caching, tracing, tagging, retries, and more with
@PortkeyAI
Add one line of headers:
llm = OpenAI(headers=Portkey.Config(api_key ="<PORTKEY_API_KEY>"))
and get caching, tracing, tagging, retries, and more with Portkey!
Docs:
1
1
9
0
1
7
Bringing AI agents to production just got a whole lot easier! 🚀
Over the past year, we've seen AI agents evolve from experimental tech to production-ready tools. But deploying them at scale? That's been a challenge.
With Portkey, we are addressing that challenge head-on.
2
0
7
Six Agent Frameworks, Twenty Six Cookbooks, AI Governance, and more - this was our month of June at Portkey 👇
1
2
7
🚨 PSA: Set the Fallback mode ON
Last Thursday, the
@AnthropicAI
API was unstable for 12 hours.
Due to Portkey Gateway's fallback feature, 99.86% of the requests made to Anthropic during this time succeeded despite Anthropic being down.
0
1
7
*The Docs Who Lived*
Team Portkey is in a 36-hour hackathon today. Expect significantly better documentation very soon!
0
2
6
80+ open-source models from
@FireworksAI_HQ
are now available on Portkey!
✅ Test your existing prompts on faster, cheaper open source models
✅ Switch to open source models without changing any underlying code
✅ Version, track, and deploy your prompts to production with one
0
2
7
OpenAI's new o1-preview and o1-mini models are supported on Portkey.
Compared to gpt-4o, these models reflect for a long time, and are able to answer questions like "how many r's in the word strawberry?", "how many words in your output?" exceptionally well.
o1 models work
1
1
7
Potterverse unite! 🪄
Thrilled to share the
@PatronusAI
's industry-leading evaluators for retrieval accuracy, hallucination detection, toxicity, and much more — are now available on Portkey Gateway.
1. Add your Patronus API key to Portkey
2. Define Guardrail checks & set
Introducing
@PatronusAI
+
@PortkeyAI
🚀
@PortkeyAI
is the leading open source AI gateway. It’s blazing fast and supports over 200+ LLMs. Today, you can use Patronus evaluators all within Portkey ✨
1
6
19
1
2
7
The Sorting Hat has spoken!
Open source → Gryffindor
OpenAI → Slytherin
Anthropic → Ravenclaw
Google → Hufflepuff
1
0
7
Last month, we brought
#LLMsinProd
to Bengaluru, and the AI community showed up in force!
500+ registrations in just 4 days for a practitioner-led deep dive into production AI
From building useful agents at scale
@getpostman
, to putting DSPy in production at
@zoro_uk
, here's a
2
0
6
🎉Portkey is live on the 'Our Open-source Friends' community, along with 40+ open-source companies!
Thanks to
@formbricks
and
@calcom
for welcoming us-
1
0
6
Ever feel like you're playing catch up with
@OpenAI
's model updates?
- Which models are being deprecated?
- Which models are being announced?
- Which models are currently available?
We've distilled the essentials into a quick, easy-to-understand snapshot with no jargon👇
1
3
6
.
@hey_ario
uses Portkey caching to run their Github workflows 24x times faster and save thousands of dollars.
"Portkey is a no brainer for people who use AI in their workflows - it really is the best caching solution"
- Kiran Prasad, Senior MLE
0
2
6
A crucial detail about the GPT-4o mini model pointed out by
#LLMsinProd
community member
@kleneway
-
The max output tokens for the model (that haven't moved much compared to all the increase in max input tokens) are 4x of all GPT-4 models and 2x of Gemini & Anthropic models!
.
@OpenAI
just dropped GPT-4o mini — a game-changer for AI accessibility! 🚀
✅ 60% cheaper than GPT-3.5 Turbo
✅ Outperforms competitors on key benchmarks
✅ Multimodal: supports both text and vision
Try out the new GPT-4o mini with Portkey! 👇
0
0
6
0
0
5
tired: i'll do a git diff between my prompt versions and track changes manually
wired: i'll just click "Update" on Portkey's prompt playground and get accurate diff summaries 💆
0
1
6
Here's a quick Llama 3 cookbook using Portkey Gateway & Together AI.
Get started! 🚀
We are thrilled to be a launch partner for Meta Llama 3.
Experience Llama 3 now at up to 350 tokens per second for Llama 3 8B and up to 150 tokens per second for Llama 3 70B, running in full FP16 precision on the Together API! 🤯
27
57
396
0
0
6
.
@OpenAI
just dropped GPT-4o mini — a game-changer for AI accessibility! 🚀
✅ 60% cheaper than GPT-3.5 Turbo
✅ Outperforms competitors on key benchmarks
✅ Multimodal: supports both text and vision
Try out the new GPT-4o mini with Portkey! 👇
0
0
6
🔥 Blazing Fast 1800 Tokens/Sec with
@CerebrasSystems
And you can route to the Cerebras API with Portkey on day 0:
Introducing Cerebras Inference
‣ Llama3.1-70B at 450 tokens/s – 20x faster than GPUs
‣ 60c per M tokens – a fifth the price of hyperscalers
‣ Full 16-bit precision for full model accuracy
‣ Generous rate limits for devs
Try now:
102
309
1K
2
1
6
Really excited about being a part of one the largest hackathons in the Bay Area!
Watch out for
@jumbld
in the OSS Alley!
@UCBerkeley
Welcome
@PortkeyAI
to OSS Alley!
A foundational model ops platform to help companies ship gen AI apps & features with confidence!
- Monitor usage, latency & costs
- Manage models with ease.
0
2
3
0
1
6
🎉Thrilled to see
@springroleinc
successfully implementing Semantic Caching for production use cases!
This single change will help reduce costs, eliminate latency for at least 20% of their requests, and make Albus more reliable.
Keep making waves, Team Albus! 🚀
Yesterday was a big day for us at Albus.🙌
We deployed:
1. Teams concept to add privacy over wiki data
2. Non-authed API to Slack Connect *anywhere*
3. Improved pre-processing for env. stability
4. Semantic caching thanks to
@PortkeyAI
🙏
5. Bug fixes, &
one more feature 🧵👇
1
1
7
1
0
6
🤖 You can now chat with our AI Gateway repo, powered by Llama 3.1 405B using
@huggingface
assistants!
Ask questions about the repo,
Get code snippets to use any LLM,
Streamline your AI integrations!
Check it out and start building!
👉🏼
0
0
6
.
@pyautogen
is one of the most popular frameworks for building multi-agent systems. The agents are customizable, conversable, and seamlessly allow human participation.
We created a simple cookbook on how to use AutoGen with Portkey to get full-stack observability on key metrics
1
2
5
🌟 Bangalore AI enthusiasts!
Swing by our booth at
#KCD2023
TODAY until 5 PM and snag your $50 in AI credits!
🎉 Let’s explore some live demos and talk all things AI. Can't wait to meet you all! 🤝
👉
@TFUGIndia
@TFUGBangalore
0
4
5
Portkey is designed from the ground up to be modular and adapt to your workflows, WHILE making them production-ready and reliable. This means, you can bring your existing services and workflows to Portkey seamlessly:
✅ Bring Your Own Guardrails
✅ Bring Your Own Agents
✅ Bring
1
1
5
Tim is building the future of game dev at Cohezion. Check it out! 👇
We’re glad that Portkey helps bring your LLM systems to production 🏭
👾 Calling all
#GenAI
folks out there! If you haven't given
@PortkeyAI
a whirl yet, you're in for a treat. No, they didn't send us a howler, but their service is legit awesome. Check 'em out – you won't regret it! 🕹️🔥
#AIAdventures
0
2
4
0
3
5
Using a single Portkey endpoint, you can now connect with *100+* open source models, switch from one to another, set up complex routing logics — all without breaking a sweat!
We're now integrated with Together AI, Perplexity, Mistral AI, and Anyscale 💥
Are we missing any?
0
1
5
Qwen-2-72B tops
@huggingface
’s new Open LLM Leaderboard!
Here’s the code to use Qwen-2-72B with OpenAI Client using Portkey -
0
1
4
New on Portkey: Take DBRX (
@databricks
) & Mixtral-8x22B(
@MistralAI
) to prod with full-stack monitoring and reliability baked in (w/o making any changes to your existing OpenAI code)
All thanks to
@togethercompute
inference!
0
1
5
Fresh from the LLMs in Prod community - all the important news this week, *just* for LLM practitioners
Find the links here:
0
0
5
The team at
@Walmarttech
have built one of the most impressive semantic caching stacks in the industry, that works at mega scale.
For the first time, we're bringing an exclusive peek inside how this was built, in a fireside chat with
@RohitChatter
, Walmart's Chief Arch.
Link ⬇️
1
1
5
huge unlock: Portkey's multimodal prompt templates now support unlimited context window
- Create complex, context-rich prompts without limitations
- Leverage multiple modalities, including text and images
- Integrate with your existing workflows and deploy with ease
0
1
5
We're honoured to contribute to the LLM Survey Report by the
@mlopscommunity
The survey underscores the pressing need to address production & reliability related challenges for LLM apps.
Read the in-depth report here:
1
0
5
🎉 Exciting news! Portkey now seamlessly integrates with
@pyautogen
!
Portkey helps you bring your Autogen agents to production fast!
Check out the details in the Autogen docs here:
2
2
5
..and this is not even the only thing we're doing this week!
1
0
5
Our
@llama_index
integration really is that simple →
We just make the OpenAI class in Llamaindex work with 200+ LLMs 🫡
1
0
5
literally a million dollar question
BUT WAIT.
Can I actually use AI Gateways like
@PortkeyAI
in the Gemini API Competition???
I'm getting worried here... 😬
I need an OFFICIAL reply.
#buildwithgemini
0
0
3
1
0
3
🚨 LLM Pricing Alert! 📉
@MistralAI
slashes prices across models:
- open-mistral-nemo-2407: 50% off (Now $0.15/0.15 per 1M tokens)
- mistral-large-2407: 33% off (Now $2/6 per 1M tokens)
- codestral-2405: 80% off (Now $0.2/0.6 per 1M tokens)
If you're using the AI Gateway, we're
0
2
5
🚨 Folks, looks like OpenAI is facing another downtime across all the APIs today
If you're looking to be proactively alerted about any downtimes across OpenAI, Anthropic, Anyscale, we post real-time updates at
1
1
4
*Monitoring
@crewAIInc
agents with Portkey*
Observability is a key problem while building Projects with CrewAI. With the help of Portkey, you can get insights into:
✅ Costs
✅ Performance
✅ Error Metrics
..and more.
We created a short notebook that shows how you can use
0
2
4
✅ Manage Groq Prompts
✅ Virtual Keys for Groq APIs
✅ Supports llama and mistral models
✅ Streaming support
✅ View logs and analytics for every request
Docs:
0
0
3
.
@llama_index
can now handle catastrophic LLM outages 🛡️
Seamlessly fallback to Llama 2 & Mistral models by
@anyscalecompute
in times of disruption with Portkey integration.
See how:
0
1
4
New Integration on the Gateway:
is a "wholesaler" of LLM inference tokens for open source models like Llama 3. It is 50-90% cheaper than leading inference providers in the market, and could be especially useful for batch inference
0
0
4
Portkey ❤️ Agents
Portkey now works with all major agent frameworks and gives you full-stack telemetry over everything - see granular logs, tabulate costs, monitor performance metrics, and more
1
0
4
Excellet (and exclusive) AI accelerator for early stage startups in India, Singapore, Indonesia, and UAE, by
@Accel
📅 Last date for applications: 10 Sep
1
1
4
Looking forward to host
@aravindputrevu
for APIs x Gen AI Builders Meetup at Portkey's Bengaluru office this Saturday!
Register here:
📢 Hello aspiring AI Engineers 📢
I'm doing a small networking meetup focussed on builders who are interested in building Gen AI apps on whatever stack they want.
We have limited spots for this one due to the short time.
Date: Upcoming Saturday - May 4th 2024
Venue:
@PortkeyAI
2
7
21
1
0
4
✨Build a secure customer support bot that protects user's PII data
Use
@LangChainAI
chatbot and
@PatronusAI
PII guardrails with Portkey. Detect and block PII. Keep user info safe.
Link to the cookbook-
0
0
4
We analysed latencies for GPT 3.5 & 4 over the last few months—there are some fascinating trends emerging:
The highlight? Latency gap between the two is narrowing, with GPT-4 getting faster!
⬇️
1
2
4
👇see, when you go from building prototypes to running llm apis at scale, the game changes.
and that's where something like Portkey can make life a bit easier🌺
just route through a proxy, add two headers, and you've got automatic retries and caching. small changes, big impact!
for something that sees quite popular usage OpenAI's embedding API seems to fail quite often, often enough to where I can't just let a batch job run for thousands of requests w/o a robust caching and retry mechanism in place. more a mild itch/annoyance than a big issue, but.
1
0
37
1
2
4