lovish
@louvishh
Followers
633
Following
693
Media
33
Statuses
180
phding @ucl and @aiatmeta (llama team). mostly random tweets here.
london
Joined July 2021
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
TEMU
• 446492 Tweets
最大5,000円報酬
• 236820 Tweets
Semarang
• 215422 Tweets
روما
• 203167 Tweets
領空侵犯
• 155196 Tweets
Arlington
• 124729 Tweets
Sven
• 123805 Tweets
Finale
• 34051 Tweets
Scout
• 31569 Tweets
#庭ラジ
• 28840 Tweets
Switchの後継機種
• 24769 Tweets
ソフトメーカーラインナップ
• 24381 Tweets
Rayo Vallecano
• 22269 Tweets
Medina
• 21958 Tweets
D-1 TO CHANYEOL SOLO
• 19199 Tweets
伊藤さん
• 17643 Tweets
瑞希誕生日
• 12657 Tweets
Pinned Tweet
405b is out! working on llama 3 has been a truly rewarding experience and i'm super grateful to all my teammates! i'm excited to see how the llama models will be used by the community!
p.s. - we wrote a paper and not just a tech report 😛
12
19
128
exactly 365 days ago, something snapped in my mind, and i finally bought a gym membership. happy then, happier now 🙂
43
32
1K
🚨 New Paper 🚨
Evaluations can have a lot of variance, throwing off model comparisons especially during pre-training. In our latest work, “Quantifying Variance in Evaluation Benchmarks”, we explore this phenomenon in depth.
A thread [1/n]
3
23
152
just remove sugar, cook in olive oil, bake/grill instead of fry. and of course, you can have cheat days too. and most importantly, don’t worry if there are off days. i had quite a few relapses where i was eating anything i wanted to. just pick yourself up and start again 😁
5
0
52
another one of my schoolmates is getting married, and here i am, still contemplating if i should apply for a phd.
5
1
35
and diet is also a very important part of this process. i tried a bunch of things - keto/intermittent/calorie deficit etc. i found that you don’t have to follow anything extreme and calorie deficit diets are just as good, and in fact more sustainable for long term.
1
0
36
since a lot of you’ve been asking, for the workouts, i focused on both strength training and cardio. cardio is important for burning calories and weights for building muscle. my average workout routine is 30 mins cardio + 35-45 mins strength training for 5/6 days a week.
1
0
34
working a full time job and still being handed 500 rupaye ka note forcefully by relatives whenever you meet them has to be the most desi thing ever ffs.
4
2
32
@giffmana
we use our internal evals repository to run all the evals. we did release the inputs/outputs/metrics using our repo for most evaluation tasks (including mmlu) here:
1
1
28
used to blast loud music in my room during wfh. have to wear earphones in the office like some civilised guy now.
2
0
21
after a lot of socializing, information overload, and llm discussions at iclr, it’s time for a solo trip in cape town!
2
0
21
fixed the fixed fix for llama3
0
3
17
saw an indian couple in barcelona eating rotis with fork and knife. i’ve seen everything now.
1
0
14
a preview of things to come from all things llama 😎 glad to be working with such an amazing team!
Excited to share a preview of Llama3, including the release of an 8B and 70B (82 MMLU, should be the best open weights model!), and preliminary results for a 405B model (still training, but already competitive with GPT4). Lots more still to come...
18
97
507
1
0
14
neurips deadline over, time for some yolo travel and pre-training runs 🙃
1
0
14
deleted instagram a month ago, and now i’ve an urge to shitpost/sadpost on the bird app. the thought of senior folks at work seeing those posts is holding me back 😬😂
2
1
12
i should save all the random papers, code, and memes in my bookmarks before this site blows up.
0
1
10
the funny thing is that sequoia india is going to give $10 million to ten teams of three engineers each, instead of just giving $100 million to a single company to train foundation models.
1
0
10
MMLU performance is at a chance level even after training for 210B tokens for the standard formulation (the model is presented with all the choices and asked to predict the most relevant choice). But MMLU-Cloze gives a better signal during the early stages of the training.
[5/n]
1
1
9
the amount of weird looks you get when you ask for a table for one in a restaurant is crazyyy. bro why you judging me, i’m just enjoying my solo trip.
2
0
9
palm, say-can, and dall-e 2. going through the ai research updates this week feels like …
1
0
9
anxiety hits on a different level when you see a senior author’s cursor in the paper section you are writing on overleaf.
0
0
8
my family is having bread pakoras with chai for their sunday breakfast and i’ve to be content with my egg white smoothie and oats!? .__. this healthy shit is hard fr 😭
3
0
8
what size 👀
GPT-4o mini is out! It's best in class for its size, especially at reasoning.
16
24
245
0
0
8
punjabis my age have only seen either badal or captain as the cm. it’s so refreshing to see bhagwant mann this time around. he used to do comedy in his past life and regularly visited my school in sangrur during annual events and performed standup comedy! fun times 😂
2
0
7
We prune samples with low item discrimination and while we find modest improvements in both standard error (a decrease) and monotonicity (an increase), the drift in the estimated accuracy is mildly concerning.
[7/n]
1
0
7
We track various metrics - seed mean/seed variance/95% CI/monotonicity for the 7B seed runs on both discrete and continuous metrics. We find that tracking continuous metrics is important as they have higher monotonicity and give higher signal compared to discrete metrics.
[4/n]
1
0
7
no offense to gemini, but what's this hokum with chain of thought MMLU, just report the 5-shot numbers lol
2
1
7
was planning to go for therapy today but the new mixtral is not gonna benchmark itself 🙃
magnet:?xt=urn:btih:9238b09245d0d8cd915be09927769d5f7584c1c9&dn=mixtral-8x22b&tr=udp%3A%2F%%3A1337%2Fannounce&tr=http%3A%2F%%3A1337%2Fannounce
274
826
6K
0
0
7
This work was done with my amazing collaborators -
@Aaditya6284
,
@RylanSchaeffer
, Andrew Poulton,
@sanmikoyejo
, Pontus Stenetorp,
@sharan0909
, and
@_dieuwke_
.
arXiv Link:
1
0
7
We try to reduce variance by taking inspiration from item analysis, where we define item difficulty (average score across models) and item discrimination (correlation b/w models’ score on a given point and models’ overall score) for each sample in the benchmark.
[6/n]
1
0
6
Moreover, using tinyBenchmarks as a cheap evaluation measure during early stages of pre-training does not give an informative signal on 3/3 of the benchmarks we examined due to increased variance.
[9/n]
1
0
6
quoting from : "we offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence." ... someday, i hope to have read all of noam shazeer's research.
1
1
5
We also analyse item response theory, as used by Polo et al. 2024 (tinyBenchmarks). We find that simply using the performance on the 100 samples selected by tinyBenchmarks can lead to large deviations in the mean (7% for ARC-C) and high variance.
[8/n]
1
0
5
i always use the same first word for every wordle. i guess this was bound to happen someday 😂🤷🏻♂️
wordle 206 1/6
🟩🟩🟩🟩🟩
2
0
5
just because ap dhillon dropped a new song doesn’t mean you have to put it in every one of your instagram stories.
1
0
4
Benchmark datasets are used for establishing progress with frontier AI models. Any major model release is accompanied by a slew of scores on these benchmarks. Yet, despite their importance, benchmark scores are often regarded as a one-dimensional number.
[2/n]
1
0
5
We present a deep dive into variance in benchmark scores across 13 popular benchmarks using over 280 models, including fully trained public models as well as a set of 7B models that we trained from scratch, differing only in their initialisation random seed.
[3/n]
1
0
4
@srush_nlp
@sharakelyan
probably because the same shot performance did not match gpt4. also the mmlu reporting is interesting (appendix 9.1): 5-shot mmlu did not match gpt4, and i guess that's why they developed the chain of thought based evaluation.
0
0
1
p.s. - i have met some incredible startup folks but none of them are working in ai. the current ai startup scene is dominated by people who all migrated from crypto/web3. complete chaos.
1
0
3
@billyuchenlin
@togethercompute
mmlu redux seems low, can you inspect/share the inputs/raw model outputs?
2
0
4
the uk doesn't know how to name its holidays. what the hell is a bank holiday?!
0
0
4
the only silver lining about living in bangalore is the good people living in this pathetic city. everything else, from traffic to water to infra, is abysmal.
No picnics, no photography, no playing sports in Cubbon Park? Bangaloreans are being policed in our most beloved green space.
Join the campaign on
@Jhatkaadotorg
to urge the Horticulture Department to roll back these bizarre rules!
RT & sign please! 🙇
19
163
395
0
0
3
aditya’s the best out there - great researcher, and the most helpful and kind person!!
📢📢At the last minute, I decided to go on the job market this year!!!
Grateful for RTs & promotion at your univ.😇
CV & Statements:
Will be at
#NeurIPS2023
! presenting AdANNS, Priming, Objaverse & MADLAD. DM if you are around, would love to catch up👋
2
49
181
0
0
3
the cherry on the top is the badal family and the captain losing their respective seats. total decimation.
0
0
3
Virtually nobody is pricing in what's coming in AI.
I wrote an essay series on the AGI strategic picture: from the trendlines in deep learning and counting the OOMs, to the international situation and The Project.
SITUATIONAL AWARENESS: The Decade Ahead
264
877
4K
0
0
3
i’m not crying. you are 🥺😭
0
0
3
@robdadashi
we use the mmlu number from the gemma report because we were getting a lower number using our internal evals. gemma was not following the instructions properly and was using a lot of ** ** to enclose text. and this is 5-shot mmlu, 0 shot was even lower.
1
0
3
@nsaphra
@andrewgwils
folks probably don’t know this but conference organizers have to submit a list of every single attendee to the ministry of external affairs for approval, which is a big pain. colt organizers in bangalore had to go through this.
0
0
3
1
0
3
there’s nothing worse than seeing someone with the same name as you doing idiotic stuff online.
0
0
3
looking to transfer volt gym membership in indiranagar. dm me if you want it.
2
0
2
@move_4_7
here you go:
since a lot of you’ve been asking, for the workouts, i focused on both strength training and cardio. cardio is important for burning calories and weights for building muscle. my average workout routine is 30 mins cardio + 35-45 mins strength training for 5/6 days a week.
1
0
34
0
0
2
@Mizzling_Gaze
the first 1/2 months are difficult, but wfh helped a lot in following a consistent routine during that period. and when i started seeing the progress after these initial couple of months, it motivated me to keep going.
1
0
2
@eugeneyan
not just llama 2, we use synthetic data from intermediate and expert llama 3 models as detailed here:
0
0
2
also, it's quite funny seeing all the rage against sam. i've lived in blr for over an year now, and i have yet to meet someone truly passionate about solving some problem. so, there's a mindset problem, and folks in startups need to worry less about retiring with lots of money.
1
0
2
best twitter bot 🤌🏻
In this organisation, women's median hourly pay is 45.1% lower than men's.
26
430
2K
1
0
2
@abhi_venigalla
@BlancheMinerva
can confirm this. the mixtral paper reports 59.7 for arc-c (which is using the normal eval setup) and 85.8 using 25-shot mmlu-style prompt. mistral-7b is 54.3 and 78.5 respectively.
0
0
1
I hereby challenge
Владимир Путин
to single combat
Stakes are Україна
35K
48K
362K
1
0
1