Bartłomiej Cupiał
@CupiaBart
Followers
1,038
Following
395
Media
5
Statuses
59
I sure do like machine learning
Warsaw, Poland
Joined May 2019
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Vance
• 1823448 Tweets
#WWERaw
• 128677 Tweets
連休明け
• 70595 Tweets
プライムデー
• 69512 Tweets
Lester Holt
• 63969 Tweets
National Anthem
• 51632 Tweets
Jana
• 51112 Tweets
Bobby
• 50032 Tweets
Teamsters
• 40313 Tweets
Rhea
• 40284 Tweets
Saint Dr MSG
• 38957 Tweets
Teoscar
• 37484 Tweets
Amber Rose
• 37070 Tweets
梅雨明け
• 35211 Tweets
#HomeRunDerby
• 25734 Tweets
Grammy
• 24474 Tweets
Bohm
• 21050 Tweets
Asaram Bapu Case
• 20914 Tweets
Anyay Ki Kahani
• 20718 Tweets
Sunane Ki Himmat
• 20387 Tweets
Gunther
• 19686 Tweets
ミュージアムカード
• 18028 Tweets
#SamsungGalaxyZFlip6diShopee
• 17995 Tweets
BOSSNOEUL TBNW WORKSHOP
• 16319 Tweets
Gunnar
• 15397 Tweets
ムーミンの肉
• 15203 Tweets
Pete Alonso
• 15027 Tweets
#リヴリー3周年
• 11004 Tweets
Fergie
• 10607 Tweets
LostFinder
• 10554 Tweets
Pinned Tweet
🚀Excited to share our latest work on fine-tuning RL models! By integrating fine-tuning with knowledge retention methods, we've achieved SOTA🔥in NetHack🎮, with scores surpassing 10K points, doubling the previous record. A detailed thread coming soon! ✨
7
24
109
So here's a story of, by far, the weirdest bug I've encountered in my CS career.
Along with
@maciejwolczyk
we've been training a neural network that learns how to play NetHack, an old roguelike game, that looks like in the screenshot. Recenlty, something unexpected happened.
149
2K
9K
The moral is, if you encounter an unexpected bug, be sure to consult lunar calendar. Big thanks to
@JensTuyls
for solving this for us!
53
137
4K
So apparently NetHack has a mechanic that slightly changes how the game plays every time it's full moon according to your system clock: The player character is luckier, werewolves appear in their animal form, and the dogs howl ominously.
6
47
2K
Next day in the morning I see a lot of messages on slack. Jens replied "Oh yes, it's probably a full moon today."
What.
2
47
2K
It doesn't make the game harder, but the model hasn't seen full moon data in its training set, so the score drops. In this particular case, it drops from 5k points to 3k points. We override the time so it's not a full moon, we evaluate the model - and it's 5k points again.
9
18
2K
I check a moon phase calendar, and yes, it's a full moon today. Hands shaking, I start a new NetHack game, and the message says "You are lucky! Full moon tonight."
What.
5
13
1K
By the point we've spent several hours on this, it's 7 PM. I am starting to feel like a madman. I can't even watch a TV show constantly thinking about the bug. Before going to sleep I decide to ask
@JensTuyls
, the author of the model, if he knows what might be broken.
1
6
734
Namely, the CUDA libraries that allow us to compute things quickly on GPU. So we suspect that maybe something about these libraries changed that degraded the model. Because what else could have? And yes, recently the version was changed from 11.8 to 12.4.
2
4
676
We use a model by
@JensTuyls
that clones expert behavior on NetHack, and we improve it using RL methods. That model gets 5000 points and we finetune it in the game so that the score improves. However, suddenly in a recent run, Jens' model only got 3000 points. Quite a drop.
3
5
667
Revert code a few weeks back? Still 3000 points. Luckily, the server we run our experiments on saves the files from the previous runs. We find the files corresponding to a run that previously got 5000 points, we re-run, and, well, it gets 3000. Nothing about the code changed.
2
5
635
The CUDA mismatch probably shouldn't impact the results in this particular way, but we see no other explanation. We override the version to 11.8 - we still get 3000 points. We build a new environment from scratch, for CUDA 12.4 - 3000 points. Welp.
1
3
633
We repeat the evaluation on a personal laptop. This is slow and expensive without the specialized hardware, but we make it work. Again, 3000 points. We disable multithreading, GPU, and some other things that have at least a conceivable chance of causing the problem - 3000 points.
1
2
632
We start suspecting our software stack. Thankfully, we use Singularity which means that our whole environment is in a single, self-contained file. That file hasn't changed for a few months, so that shouldn't be the problem. However, the container loads one thing from the server.
1
8
629
This problem is consistent between seeds so it's not just a fluke. Well, we probably screwed up something in the code for loading the model in the recent commit. Let's revert, no biggie. Except that after reversing to a version of the code from a few days back, we still get 3000.
1
4
605
@tomcocobrico
@maciejwolczyk
This is actually the case. I remember now that few months ago I had similar problem on a sub experiment. I just thought the code had buggs and abandoned that branch :D
0
0
355
@cglassey_author
@maciejwolczyk
My favorite interaction is using a cockatrice corpse as a weapon, very few monsters have stoning resistance :)
2
0
30
Hey, I wanted to tell you that I added NetHack to sample_factory. Just transferring the code boosted my results by over 30%! If you ever want to do experiments on NetHack, this is now definitely the best place, as other NetHack repositories are heavily research-oriented.
3
4
17
Huge thanks to my amazing co-authors:
@maciejwolczyk
@Michal_Zajac_
@rpascanu
@LukeKucinski
@PiotrRMilos
@MatOstasze
@m_bortkiewicz
0
0
7
@AviBenemanuel
@maciejwolczyk
Yes, but unfortunately singularity takes time from the host machine. Also changing the time on your machine breaks up other things :<
0
0
5
@sheepyk
@maciejwolczyk
Yes this is quite the issue. Maximizing the score means that you will just farm monsters. Finding items required for ascention or even Just doing a quest is too much for pure RL agent.
0
0
4
@RyanSullyvan
It turns out results in my paper accepted to ICML have lower score because of the full moon lol
1000 points or so
1
0
5
If you're interested
readme:
wandb report:
hugging face model card:
0
0
5
Model-free beating model based RL! Looks like we need harder environments.
Scaling has done wonders for deep learning, but for a long time it failed in on-policy RL... until now! We show that when done appropriately, scaling leads to state-of-the-art results in a variety of continuous control tasks🔥
Introducing BRO: Bigger, Regularized, Optimistic! 🧵
8
48
266
0
0
4
@aubekin
@maciejwolczyk
Unfortunately no. Currently best performing bot is AutoAscend but even it can only solve sokoban and reach mines end
0
0
3
@Abel_TorresM
In terms of luck, yes. But the rnn sees the message it never saw (full moon tonight) at the start of the game which can influence the hidden states.
1
0
3
@ulyanapiterbarg
Thank you for pointing this out. We wanted to focus on the simplest setting since this is a complex game, so we run experiments only in monk setting to decrease the number of experiments. I just ran experiments on @ and without tuning any of the hparams we get ~4.5k score.
1
0
3
@_rockt
@JensTuyls
@NetHack_LE
@HeinrichKuttler
I've already created PR to NLE repo , turning off this cool feature to improve reproducibility.
1
0
2
@dlowd
That's certainly true! I spend more time then I would like to getting my first ascention :)
1
0
2
@sheepyk
@maciejwolczyk
Thats why NetHack is such an interesting testbed for new RL approaches :)
0
0
1
@yasuoyamasaki
During a full moon:
Your base Luck is increased by one. Which is an advantage.
Werecreatures, especially at night, will usually be in their animal form. Which can cause a lycanthropy.
0
0
1