Andi Marafioti
@andi_marafioti
Followers
379
Following
118
Media
16
Statuses
147
🤖 Machine Learning R&D @huggingface | multimodal | 🎧 Sound Engineer | 🇦🇷 in 🇨🇭 | 🌍 ES, EN, FR, DE
Bern, Switzerland
Joined April 2022
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Yoro
• 325273 Tweets
Ranma
• 207224 Tweets
Olmedo
• 182535 Tweets
Emmy
• 145208 Tweets
Masi
• 111920 Tweets
The Bear
• 96067 Tweets
Schiff
• 89170 Tweets
Home Depot
• 68023 Tweets
Subway
• 54352 Tweets
STRAY KIDS DEADPOOL OST SLASH
• 33612 Tweets
Kalonzo
• 26571 Tweets
Shogun
• 25116 Tweets
Serbia
• 19059 Tweets
Doue
• 18838 Tweets
Jelert
• 17897 Tweets
RINOs
• 15971 Tweets
EmeğimeSahipÇık ÖSYM
• 15128 Tweets
The Weeknd
• 13664 Tweets
Pinned Tweet
I'm quite proud of my first 🤗hugging face blog!
6
33
173
Excited to share I've joined
@huggingface
🤗 as a ML Research Engineer part of the multimodal team! Excited to see what amazing new models and datasets we can create for the community!
13
13
220
🚀 Fine-tune Florence-2 on any task!
We are releasing fine tuning scripts for microsoft's Florence-2, alongside with a walkthrough blogspot, a space demo, and a Colab notebook.
@mervenoyann
@skalskip92
🧵
1
45
192
Almost there! Just one more star for my Florence2 fine-tuning GitHub repo to reach 128 and earn a new badge.
2
17
139
We added idefics2 and idefics2-chatty to the Unsolvable Problem Detection Leaderboard. 🚀 This benchmark was developed to measure the robustness of VLMs by asking them questions about images that cannot be answered.
#MachineLearning
#AI
#NLP
🧵
1
10
25
🚀
@argilla_io
is joining
@huggingface
🤗!
Argilla is the leading company in dataset creation with a ton of open-source contributions! Many of their efforts are to enable multilingual LLMs all over the world!
Oh, and they also are co-authors of Zephyr ORPO!
0
4
24
I've been working on a synthetic dataset and every day I optimize the script. So far I got:
Monday -> ETA 12 days
Tuesday -> ETA 6 days
Wednesday -> ETA 2 days
Today -> ETA 4 hours
My goal is to make it so efficient that I can 10x the original intended size 🚀🤗
1
1
16
@Ahmed_Masry97
@mervenoyann
@skalskip92
That's awesome! Let me know how it goes! you can start here:
You only need to do:
python distributed_train.py --dataset cauldron --epochs 5 --eval_steps 20000
Maybe playing a bit with the learning rate would also help :)
0
0
10
@Ahmed_Masry97
@mervenoyann
@skalskip92
You need to consider that we only fine-tuned on DocVQA, a pretty small dataset. The code to fine-tune on The Cauldron is up there and would probably achieve a better performance, but I'm GPU poor :(
1
0
5
It’s such a privilege to finish a hard work day and go into my garden and gather flowers. I’m so lucky 💞
0
0
4
Florence-2 fine-tuned on DocVQA excels at retrieving information from images! Just a few final tweaks, and I'll release the code for anyone to fine-tune for their own tasks. Stay tuned! 😊📚
0
2
4
Apple Intelligence is a huge tailwind for multimodal models. To do 'on-screen awareness' they need to understand image, text, audio, and video. I'm super excited to work on creating these models for the open-source community
@huggingface
0
0
4
@Laz4rz
@pixqc
@wateriscoding
the use cases are particular. Like you don’t want a proper database but you need sql speeds on complex queries. I almost dropped it completely but just a few days ago I thought about using it for the dataset I’m generating. In the end, I got more speed ups out of other things.
1
0
3
@Laz4rz
@pixqc
@wateriscoding
It allows for sql queries (pandas as well) and it’s pretty fast at it. I don’t think it drops everything into memory but I didnt profile it.
0
0
3
0
0
3
Overall, these results are fantastic for the open-source community, demonstrating that the gap between closed and open-source models is continuously shrinking! 🌐👏
#OpenSource
#AI
#Innovation
1
0
3
For the Absent Answer Detection task, idefics2-chatty is the best model of its class! It even beats GPT-4o and Gemini pro in the hardest setting! 🏆
#MachineLearning
#AI
#NLP
1
0
3
I'm creating question/answer pairs from documents using an LLM. It mostly works well, but the number of questions doesn't scale linearly if I feed the model several pages. I tend to split the pages, but the context becomes less rich. Any ideas?
0
0
2
We fine-tuned Florence-2 with a small learning rate and froze the vision encoder. Our experiments ranged from a single A100 GPU setup to a powerful 8 H100 GPU cluster. Showing the potential for small setups 🚀💻
1
0
3
@PriNova75
@huggingface
Right? It's crazy! I'm happy that at least we track it and show it on all jobs we are running to create some awareness among engineers. Also there is some incredible work by
@SashaMTL
that keeps us in check. But yes, the carbon emissions of ML training are nuts.
0
0
3
For the Incompatible Answer Set Detection, idefics2-chatty performs exceptionally well, outperforming Gemini Pro and LlaVA-1.6-13B for base type questions, though it falls behind for more complex queries. 📊
#AIResearch
#Tech
1
0
2
@xhinker
Hi Andrew! There’s a bug in the original implementation that doesn’t allow models to be fine tuned. We opened PRs on all the models to fix them, but you need to point the hub to our prs or use the model I uploaded. Did you change the model you were fine tuning?
2
0
1
@dingchilling
@huggingface
I’d love to work with music again sometime in the future :) this is a paper I wrote a few years ago:
1
0
1
Excited to share my journey in Machine Learning and Sound Engineering! 🌟 With 10 years of experience and a PhD, from Argentina to Switzerland, speaking four languages and always learning. Let's innovate together! 🚀🎧
#MachineLearning
#SoundEngineering
#TechForGood
0
0
2
Here’s the full blog:
0
0
2
@SanhEstPasMoi
@Grammarly
I've been using it for two years now, and it has improved my messages and taught me to write better. It's super helpful when it says, 'Remove this for confidence.'
My only two quarrels: 1) It constantly wants to delete my emojis :'( 2) Only English :(
0
0
2
@bigemptyboulder
@huggingface
I had six rounds of interviews but they were all pretty fair/on point and not very time consuming. Tips for a successful application are to make cool stuff you can showcase such that your profile stands out 🤗🚀
0
0
1
We fine-tuned the model on DocVQA, a hard benchmark, and it achieves 57% ANSL - outperforming Idefics2 (without fine-tuning) and DeepSeek-VL (with fine-tuning) 🤯
1
0
2
@mervenoyann
@pebkac_roll
And we’re open about putting them to good use! If you have an impactful open source project and need some compute power, definitely jump in my dms 🚀
0
0
2
Goal of the day: Set up a fine-tuning example for Florence-2. 🚀
1
1
2
I hate whenever I see a post that says “do x or you’ll be left behind”. No, you won’t. Relax, you can afford to lose a trend or two.
1
0
2
In the Incompatible Visual Question Detection, idefics2-chatty matches the best model so far in the <10B class, Qwen-VL-Chat, but lags behind larger, closed-source models. Still, an impressive feat for open-source! 🎨
#ComputerVision
#ML
1
0
2
@Laz4rz
@pixqc
@wateriscoding
I tried to use polars on production and it would be a bit buggy in unexpected ways. To me, polars shines when you need to process millions of rows, specially recurrently. It’s an use case that happens in many ml pipelines
1
0
2
@effemanfredi
Yes, the model has some custom code from Microsoft and at some point it sets the model_type to an empty string. It’s just a bug but hard to squash 😅
1
0
1
@Laz4rz
@Nigh8w0lf
@mervenoyann
@skalskip92
They have this in the paper. Basically they asked annotators to provide three levels of detail
2
0
2
I added Lora training to the florence2 fine tuning repo. I was expecting that with Lora I would be able to fit way larger batch size but reducing to 1% of trainable params I can fit 25% more samples per batch. Is that normal?
1
1
2
0
0
2
🚀 Exciting news! The MM-UPD benchmark is now live on
@huggingface
Hub as a leaderboard 🏆. It evaluates how vision language models handle unsolvable problems 🤓. Currently, top VLMs like GPT-4V and LLaVA-Next-34B struggle with it.
🔗 Link in the next tweet!
1
0
2
@LukaszBorchmann
@mervenoyann
@skalskip92
Hi 👋 I don't think a text-only model would achieve anything in this dataset. Where did you see a BERT base performing so well? I'm interested!
We didn't target SOTA performance here; performance would improve training on a larger dataset like The Cauldron.
1
0
2
big thanks to
@AtsuMiyaiAM
for putting it together!:
I'm currently working on submitting idefics2 to it!
0
1
1
I'm going to build my first space on
@huggingface
today :D
I want to visualize a dataset I'm generating and let other people explore it
1
0
2
Excellent summary on an incredible new multimodal model. The future is bright!
EPFL and Apple just released 4M-21: single any-to-any model that can do anything from text-to-image generation to generating depth masks! 🙀
Let's unpack 🧶
10
134
807
0
0
2
@mervenoyann
@huggingface
Awesome leaderboard! Interesting to see that there is only one model <10B
Seems like there are still some room for improvements in the field :)
0
0
1
1
0
1
For the tutorial on fine-tuning Florence-2. Would you be more interested in seeing how to do it with a collab notebook or how to do it with a multi-GPU setup to get better performance?
0
1
1
Great opportunity to use for free the HF 🤗 cluster for distillation projets!
Given the rise of big LLMs (100B+ params), it would be exciting to see more open-source experiments in distillation🥁
I'm happy to announce a 1000 A100 hours grant for open access work in the LLM distillation space 🤗
Examples:
- Distilling an 8B model into <1B model
- Writing
11
63
371
0
0
1
Here are a number of comparisons. For its size, it's great in captioning, but the larger models perform better.
It does best in visual question answering. Large models perform sometimes better, but not always.
It's SOTA on Referring Expression Comprehension.
1
0
1
@Laz4rz
@Nigh8w0lf
@mervenoyann
@skalskip92
Yeah, they are super different annotations and results. But if you need to fine tune for captioning I would still start from one of those. Try to get a feeling for which one is closest to what you want.
1
0
1
@mervenoyann
@huggingface
Merve could you save some from me? I’ll come to Paris office in July 🤗
0
0
1
I launched the first ablations on the new synthetic dataset! Wish me luck 🍀
0
0
1
Have you tried to answer MMMU questions? They are frigging hard!
At least the paper mentions that college experts achieve 76-82% accuracy, so it's not just me that's dumber than an LLM xD
0
0
2
Last one that really annoyed me was “if you’re not using these gmaps functions you’ll be left behind”… really?
0
0
1
That moment when you make your job so efficient that runs at lightspeed, but it crashes the node so you need to artificially throttle it :(
1
0
1
As a software engineer with 10 yoe, I use LLMs to code complex systems all the time. It’s like I have a small army of juniors that code whatever I tell them to and then I only need to review the code and put the pieces together.
1
0
1
2
0
1
@Laz4rz
Whenever you’re limited by I/O. Here it takes some time for me to load the images and submit them to the gpus, so instead of waiting for the images to be loaded, I load them in parallel while I wait for the gpus to finish their tasks
1
0
1
@xhinker
you're running into all the bugs we did 😅 This is pretty much the last one that actually breaks everything
1
0
1
@txhno
So far it's been on par with pip. Like pip-compile, uv generates a platform-specific requirements.txt file (unlike, e.g., poetry and pdm, which generate platform-agnostic poetry.lock and pdm.lock files).
1
0
1
@kazemi_sm
@WenhuChen
Do you have code for the implementation? I could evaluate idefics2 using our cluster :)
1
0
1
@realmrfakename
@huggingface
For now multimodal as in images+text, but I believe the field will naturally evolve to integrate audio and video to the mix 🌈
0
0
1
Where was uv all these years ❤️? An in-place replacement for pip that is 10-100x faster 🤯. Go ahead and try it, it's amazing:
2
0
1
@teodor_io
With async you get to control when each “thread” holds the processor, so race conditions are less likely.
1
0
1
Tomorrow I'm getting certified as a climbing guide with the Swiss alpine club! I'm going on a multi-pitch climb with a group and made this nice poster as marketing :)
0
0
1
@Laz4rz
Yes, if you use torch then this is already optimized in the DataLoader and trainer loops. I'm using HFs llm-swarm () to spawn "servers" with LLMs where I submit tasks. Creating the tasks takes time, and I do it while I wait for the servers to respond.
1
0
1
@m_chirculescu
PaliGemma is a larger model, and it is already pre-trained for VQA, so it does better out of the box. However, you need more computer power to fine-tune PaliGemma on your particular dataset, so I would choose Florence if you need to fine-tune it.
0
0
1