Braintrust
@braintrustdata
Followers
1,702
Following
51
Media
36
Statuses
143
Braintrust is the enterprise-grade stack for building AI products.
Joined August 2023
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
#TGS2024
• 203061 Tweets
Miami
• 104623 Tweets
Knicks
• 90240 Tweets
Towns
• 77712 Tweets
Minnesota
• 73516 Tweets
Randle
• 70639 Tweets
भगत सिंह
• 65399 Tweets
mingyu
• 59471 Tweets
FAYEYOKO MACAU FANMEET
• 58658 Tweets
#FayeYoko1stFMinMacau
• 57279 Tweets
Donte
• 43719 Tweets
Wolves
• 40469 Tweets
Paola
• 38775 Tweets
日本破壊クソメガネ
• 24550 Tweets
ババコンガ
• 21116 Tweets
シリウスS
• 21077 Tweets
Virginia Tech
• 20917 Tweets
Cam Ward
• 20313 Tweets
#bhagatsinghjayanti
• 14815 Tweets
#AFLGF
• 14139 Tweets
Rutgers
• 13798 Tweets
Pinned Tweet
Take off 🚀
I'm excited to announce a new product I've been working on called
@braintrustdata
.
Braintrust helps innovative companies and developers ship higher quality AI products by making it easy to run evals.
🔉 on
39
54
420
4
5
47
OpenAI announced: Reproducible outputs
💥 This is game changer for developers! It's now possible to actually evaluate and unit test your LLM apps! You can know that when a test passes locally, it'll pass for your team and in CICD.
1/3🧵
1
3
21
Our playground now supports all the new OpenAI models and some fun OS models 😎 thanks to
@perplexity_ai
3
1
18
Braintrust handles the boring work. Spend your time having fun building AI apps with your team 🥳
0
7
19
Over the past year, it's been an absolute joy getting to know
@mikeknoop
and build Braintrust with our friends at
@zapier
.
We worked together on a blog post that captures their workflow. If you want to build a world-class AI product, this is for you!
1
5
19
We evaluated Google's text-bison LLM against OpenAI's gpt-3.5-turbo on a SQL generation task in Braintrust.
Here's how they performed:
- finetuned-gpt3.5: 92.4%
- finetuned-bison: 84.2%
- gpt3.5: 78.7%
- bison: 74.8%
(We finetuned both models too!) Dig into the evals below:
2
2
14
.
@Retool
surveyed how companies are adopting AI. Some of the top challenges: model output accuracy, hallucinations, and prompt engineering.
Braintrust help's you solve these challenges: run evaluations, visualize and inspect your results, and experiment with prompts quickly.
4
3
11
Now, we can have much more fun evaluating our AI apps 🥳
Check out our docs on how to use Braintrust to evaluate your AI app. It's very easy to integrate Braintrust evals with your existing CICD workflow (see our docs below)
0
0
8
We have some exciting news today 😀!
Not too long ago, I announced a new company called
@braintrustdata
.
Today I'm super excited to share our $5m seed round led by
@saammotamedi
at
@GreylockVC
.
31
21
261
0
2
8
You can self host Braintrust within your own VPC! Learn how in our docs or just reach out to us 👋
0
2
10
We are super excited to partner with Liucija, Senior Data Scientist on the AI team
@Hostinger
, as they work towards leveraging AI for use cases like customer support, website building, and more.
If you'd like to learn how Braintrust helped Hostinger:
- 3x the number of AI
2
1
10
Super fun to host an AIUX Demo Night last week at
@eladgil
's office! We are super excited about the future of UIUX w/ AI and loved seeing what talented people are building. Thank you to everyone who came out and special shoutout to our demoers 🙂
If you’re interested in coming
1
0
10
LlamaIndex just released Llama Datasets so you can easily benchmark RAG pipelines.
We contributed a help desk dataset with Coda so you can easily benchmark chat qa & support use cases.
Check it out on Llamahub
Exciting to collaborate with
@llama_index
@braintrustdata
@siuheihk
on a great dataset for developing RAG apps!
1
0
3
0
1
8
The AI app development journey:
1. Start with a prototype and manually test
2. Get tired of manually testing
3. Evaluations enlightenment: add evaluations to your code
???
4. App is in production. Users rave about your app
Braintrust makes it easy to evaluate your AI code.
1
4
7
🤩 New feature: text blocks in the playground!
These blocks just return a constant or variable value without any LLM call.
This makes it easy to:
- debug your prompts
- mock API responses and vectorDB calls
0
1
7
Don't get stuck manually inputting test cases into your LLM app after every prompt change.
Braintrust makes it easy to automatically evaluate and test your LLM apps.
0
1
7
Braintrust also easily integrates with Pytest, Jest, etc. What other testing libraries do you like to use?
0
8
6
We are hiring for roles including:
- Software Engineer — Data Visualization
- Software Engineer — Systems
- Chief of Staff
Learn more here:
1
2
6
🎉We are hiring!
We're hiring engineers :)
Do you love:
* building visualizations on text, images, and numbers that (re-)render in <100ms?
* searching/grouping billions of rows of semistructured text-heavy data in <200ms?
* grinding away LLM latency by any means necessary?
If so, LMK
11
17
145
0
5
6
👎 Before:
- your app generates different outputs every test
- if you use LLMs to grade outputs, those grades would also be random every test
👍 Now w/ reproducible outputs:
- your app generates consistent outputs even if temperature !=0
- your model graded evals are consistent
1
0
6
Which LLM is the best at summarizing GitHub issues?
We informally tested to find:
GPT4>Mistral7b>Claude2.1>GPT3.5
It's easy to run evaluations with Braintrust using our eval libraries and AI proxy.
Check out the code below:
1
0
6
⏰ We added duration stats to experiments!
See which test cases were faster or took longer.
There's a tradeoff between speed <> quality. Use Braintrust to help you find the optimal balance 😇.
1
8
6
Spend your time building the fun parts of AI apps w/ Braintrust :)
0
1
6
🚨 Braintrust shoutout @ 24:45 - thank you
@eladgil
:)
🎧🍌New
@ThePeelPod
with
@EladGil
Stream the full episode here on X or links below
Timestamps:
03:46 Building cool monuments
09:12 Fixing education
16:38 Why AI is underhyped
19:02 Four trends to watch in AI
19:55 Why there aren’t large biotech companies
23:21 The current state
8
7
48
0
0
6
The LLM App Stack by a16z.
Validation is the most crucial step in building reliable and quality AI apps.
Braintrust helps you integrate evals to rapidly ship reliable AI.
1
0
5
Our prompt playground supports OpenAI function calling and tools now! Try it out on Braintrust.
0
0
4
😍 It's now so easy to use variables in our Playground.
We got tired of editing raw JSON so we upgraded our UI to support variable/object inputs better.
0
7
5
Simplify your evaluation scripts with Braintrust.
Just define 3 functions: data, task, and scores.
We do all the tedious optimizations like parallelizing requests for you.
0
9
5
@jerryjliu0
@llama_index
@FastAPI
Need an evaluations script for your AI app? We just opened a PR adding in Braintrust for create-llama to test and evaluate the LLM calls in the templates.
1
0
5
👋 Braintrust makes it easy to eval your AI app
0
0
4
We are excited to announce Braintrust is now SOC 2 Type II certified! We have supported enterprise customers from day 1, and achieving SOC 2 compliance is further validation of how seriously our team takes governance, risk, and compliance.
0
1
4
We are very excited Braintrust was featured in the inaugural Future 50! We are thankful for the recognition and can’t wait to continue supporting amazing AI teams.
I’m so excited to sareh the Future 50, a database of extraordinary, high-potential startups.
A few companies you'll learn about:
🚚 A trucking company doing $45M ARR
🧬 A biotech building "AWS for biology"
🇯🇵 Japan's answer to OpenAI
📈 A payments company that grew 20x in 18
3
19
103
0
1
4
@goodside
This is how we are thinking about deterministic outputs: it's a game changer for developers
OpenAI announced: Reproducible outputs
💥 This is game changer for developers! It's now possible to actually evaluate and unit test your LLM apps! You can know that when a test passes locally, it'll pass for your team and in CICD.
1/3🧵
1
3
21
0
0
4
Don't have an eval set already? Tired of writing scoring functions?
Our `autoevals` library makes it easy to grade your LLM outputs.
It includes prebuilt scoring functions:
• Model-based (using LLMs)
• Heuristic (e.g. Levenshtein distance)
• Statistical (e.g. BLEU)
0
1
4
Honored to be named as one of the most promising startups of 2024 by Business Insider! Thank you
@CorinneMRiley
& Shravan Narayen for nominating us
1
1
4
We have 5 tutorials on how to evaluate AI apps in our docs so far.
What other eval examples do you want us to make?
0
0
4
The Modern AI Stack by Menlo Ventures.
"Customers expect and deserve high-quality outputs, and enterprises are smart to be concerned that hallucinations could cause customers to lose trust."
Braintrust helps you integrate evals to rapidly ship AI without guesswork.
1
0
4
New cookbook on how to use the fantastic
@ragas_io
framework in Braintrust!
Among other things, the Braintrust implementation:
* Available in both TS and Python
* Uses function calling (which substantially boosts performance)
* Is fully debuggable
0
0
4
1
0
3
@rivet_ts
@AnthropicAI
@AssemblyAI
@GentraceAI
@huggingface
@MongoDB
@pinecone
We had a lot of fun working together on this plugin! 😄
0
1
3
.
@Retool
surveyed how companies are adopting AI. Some of the top challenges: model output accuracy, hallucinations, and prompt engineering.
Braintrust help's you solve these challenges: run evaluations, visualize and inspect your results, and experiment with prompts quickly.
4
3
11
0
0
3
🥳New feature: customize your experiment dashboards!
Choose only the charts you want to see for your experiment
0
0
3
😎 We just made our experiment sidebar resizable.
Now, you can quickly view what you need without having to change pages all the time.
0
0
3
It's so easy to manage test sets and datasets with Braintrust. We made a web UI for editing evals with your team so you don't need to make your own with Google Sheets/Retool. Our TS/Python library also...
1
0
3
No need to spend all your time testing changes manually after every prompt and pipeline change :)!
0
0
2
@mathemagic1an
@codegen
@ThriveCapital
🔥 we just signed up on the waitlist. we'd love to evaluate how it works on our codebase
1
0
2
@DescriptApp
These are awesome! We love using Descript and had a lot of fun making our launch video using it!
0
0
2
Read the full blog post here to save time on your AI app development journey:
0
0
2
Braintrust makes it fun to collaborate with your team on AI app development.
0
0
1
0
0
1
And it's easy to read from a dataset from your evaluation script or backend services.
1
0
1
Braintrust can be run on-prem. Keep your customer data secure when you evaluate and log data for your AI app w/ Braintrust.
0
0
1
