Maciej Wołczyk
@maciejwolczyk
Followers
526
Following
286
Media
26
Statuses
123
Post-Doc @IDEAS_NCBR , alumni of Jagiellonian University. Reinforcement learning / continual learning / transfer learning. Outside of work:🏃🏻🧗🎮📖🎸
Joined March 2011
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Oprah
• 368291 Tweets
Gus Walz
• 339577 Tweets
الرائد
• 284785 Tweets
Marçal
• 158519 Tweets
Macri
• 122914 Tweets
sabrina
• 112562 Tweets
Beyoncé
• 87451 Tweets
Barron
• 86314 Tweets
#النصر_الرايد
• 82276 Tweets
Ann Coulter
• 73779 Tweets
Happy Anniversary
• 69779 Tweets
Boulos
• 68923 Tweets
MUERO X DECÍRTELO OUT NOW
• 52422 Tweets
رونالدو
• 46204 Tweets
George Bush
• 34325 Tweets
Servette
• 30913 Tweets
Mudryk
• 28996 Tweets
Dotson
• 28333 Tweets
Mike Pence
• 24934 Tweets
CARAJO FURIOSO
• 24487 Tweets
Guiu
• 20089 Tweets
#LUGvBJK
• 19731 Tweets
Dinesh
• 18670 Tweets
Ken Salazar
• 18282 Tweets
Noni
• 16614 Tweets
Jimmy Carter
• 12336 Tweets
対象の3作品
• 12270 Tweets
Masuaku
• 12187 Tweets
エンティーム
• 11890 Tweets
#ابعدوا_رايد_اسماعيل_وغويدو
• 11756 Tweets
Mendy
• 10264 Tweets
Yesterday was the last day of MLSS^N, a summer school on ML and neuroscience I co-organised. If you missed it, you can catch up on the wonderful lectures of
@rpascanu
,
@SaxeLab
,
@vdbergrianne
and many other amazing speakers here:
4
48
200
Fine-tuning has been essential for LLMs, but it doesn’t work too well in reinforcement learning. 🤔 Why?
In our ICML 2024 spotlight, we highlight a crucial issue with this approach and show that if done correctly, it outperforms previous SOTA by 2x on a very challenging domain!🧵
2
40
190
I am proud to say I have joined
@ELLISforEurope
! Europe needs more continent-wide collaboration in AI research, and I'm here to help!
@maciejwolczyk
, a postdoc at IDEAS NCBR, has been accepted into the prestigious
@ELLISforEurope
! His research focuses on efficiently adapting deep learning models, especially in continual and reinforcement learning. 🙌 Congratulations!
#MachineLearning
#Research
#ELLIS
#AI
0
0
9
4
4
46
Transfer learning is an essential component of continual learning, but it's still not well understood. Although we have some insights into sharing knowledge between tasks in NLP and CV, what can be said about RL?
Find out in our
#NeurIPS2022
paper: (1/N)
1
5
36
I'm very proud to have received the FNP Start scholarship for talented young researchers!
💯
@maciejwolczyk
(IDEAS NCBR) i
@piotr_kicki
(
@PUT_Poznan
, IDEAS NCBR) wśród 100 najzdolniejszych młodych naukowców w Polsce według
@FNP_org_pl
. Gratulujemy!
🏆Każdy z laureatów otrzyma roczne stypendium – 30 tys. zł.
1
4
18
2
2
28
Thanks for having us, it's been great!
Great presentation by
@maciejwolczyk
and
@CupiaBart
of their paper "Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem" () as well as the story behind their
@NetHack_LE
full moon bug hunt at the
@UCL_DARK
Seminar Series.
2
9
60
1
1
18
Big thanks to my great co-authors,
@CupiaBart
,
@MatOstasze
,
@m_bortkiewicz
,
@Michal_Zajac_
, Razvan Pascanu,
@LukeKucinski
,
@PiotrRMilos
and the whole
@IDEAS_NCBR
team.
If you'd like to hear more, check out our UCL Dark talk:
1
3
17
Join me and
@CupiaBart
on a convoluted journey of troubleshooting a very very nasty bug.
So here's a story of, by far, the weirdest bug I've encountered in my CS career.
Along with
@maciejwolczyk
we've been training a neural network that learns how to play NetHack, an old roguelike game, that looks like in the screenshot. Recenlty, something unexpected happened.
148
2K
9K
0
0
16
Visit me today
@neuroAIworkshop
and check out our poster on Spatial Neural Networks. At GMUM we are really excited about combining ideas from AI and neuroscience!
3
3
14
I will be at
#NeurIPS2022
next week! Let me know if you want to grab a coffee and chat about transfer learning, RL, or something entirely different.
In other news, I'm on the job market for a research scientist position, so this would also make for a great conversation topic!
0
0
14
Check out this sneak peek into the geometry of SSM states! The hidden states in SSMs have some surprisingly nice properties -- you can just average them to boost performance at ICL tasks and easily retrieve states corresponding to similar tasks. Based on our latest pre-print.
Language modeling based on gated linear recurrence, such as Mamba, Hawk and RecurrentGemma is gaining traction! But can you use the statefulness of these models for more than efficiency? Super happy to introduce (snake 😉) state soups! 🧵
2
5
28
0
0
12
No papers to present at this year's
@iclr_conf
, but I'm very proud to be one of the highlighted reviewers!
0
0
11
If you want more details, check out our paper: , our website: and visit us at our poster session at ICML 2024 on Tuesday, 11:30 AM!
1
0
10
In fact, by simply combining fine-tuning with knowledge retention methods, we managed to outperform the previous SOTA in NetHack by 2x. This gives us hope that the current foundation models in RL can also be improved through careful online fine-tuning!
1
0
8
We'll be presenting our NeurIPS poster on Wednesday! Please come, say hi, and chat about the challenges of continual and reinforcement learning!
We are happy to present our Continual World benchmark: . To learn what works and what does not on the intersection of CL and RL, and why the forward transfer is important, meet us at
#NeurIPS2021
(Wed Dec 08, 08:30am - 10:00am GMT): .
1
0
18
1
2
9
Years of study in the continual learning field showed that neural networks catastrophically forget data that is not currently seen. Fortunately, this field also developed tools to stop neural nets from forgetting. We can use them to protect the old skills when learning new ones..
1
0
8
You can check out the full speaker lineup at
Big thanks to all of the speakers, co-organizers, and the participants who helped to make it happen!
1
1
8
If you simply fine-tune your NetHack model, it performs quite badly since it rapidly forgets the Far states it cannot reach at the beginning of the training. On the other hand, applying knowledge retention methods from continual learning again fixes this problem!
1
0
7
I'm happy to be a part of this initiative! Conditional computation is a fascinating and rather underexplored part of deep learning with practical applications (especially for efficiency) and intriguing ties to neuroscience. Check out our website and consider submiting a paper!
Announcing the 1st Dynamic Neural Networks (DyNN) workshop, a hybrid event
@icmlconf
2022! 👇
We hope DyNN can promote discussion on innovative approaches for dynamic neural networks from all perspectives.
Want to learn more?
1
27
91
1
1
5
@sarahookr
@tomasztrzcinsk1
Are there any openings in your IDEAS NCBR group? Seems like a good fit.
0
0
6
Check out our new paper introducing a generative autoencoder with a Gaussian mixture as the latent space prior! Since both class and style is contained in a single continuous space, we are able to smoothly change the label of a given image.
Finally out! Semi-supervised generative Cramer-Wold autoencoder with a GMM as target -- read early access at IEEE T-NNLS () or preprint on arXiv ()!
#DeepLearning
(Collaboration between
@JagiellonskiUni
and
@Know_Center
)
1
1
9
1
2
5
We can conceptualize this problem by partitioning the environment into the Close states, which are easily available at the beginning of the trajectory, and the Far states, which are only available after putting some work into learning.
1
0
5
Find more in our paper and visit us at the poster sesion on Thursday afternoon, Hall J
#110
!
Huge thanks to my amazing co-authors:
@Michal_Zajac_
@rpascanu
@LukeKucinski
@PiotrRMilos
(9/9)
0
0
5
We run experiments on NetHack, an extremely challenging env, using a massively scaled-up behavioral cloning model. If you compare the states the agent sees during the offline pre-training and states gathered by unrolling the cloned policy, the difference is massive!
1
0
5
There is still much to do, e.g. scaling up and finding tighter bounds on performance, but I'm really excited about this research direction! If you'd like to know more, check out our paper and visit me at the poster session on Thursday afternoon! (8/8)
0
0
5
Me: recursion jokes are dumb
Also me:
Me: recursion jokes are dumb
Also me:
Me: recursion jokes are dumb
Also me: ...
0
1
4
@dgrey0
Mildly interesting fact - in Polish there's a word "rogalik" which means "croissant" and sounds vaguely like "roguelike". So, since "rogalik" rolls off our tongues easier than "roguelike" (and since it's fun), we sometimes refer to roguelikes as croissants.
0
0
4
A similar problem will appear when your model is pre-trained offline through behavioral cloning. In complex environments, we can never perfectly clone the expert, so our policy will make small mistakes here and there.
1
0
4
In our case, the Close states represent opening the drawer, and the Far states represent picking up and moving objects. As we start fine-tuning, the model cannot solve Close, so it doesn’t see any examples from Far. This causes it to forget how to behave in Far states.
1
0
4
Surprisingly, the model disentangles the class and style variables without any explicit regularisation. Because of that we are able to smoothly change the class without affecting other features.
0
0
3
These mistakes compound as we roll out the trajectory, getting further and further from the pre-trained distribution. We will see Far states during fine-tuning much less often than during pre-training, which again leads to forgetting. How big of a problem is this in practice?
1
0
3
When trained, the model will figure out how to open the drawer but it will no longer be able to pick up the object. What happened?
1
0
3
Finding
#1
: Critic is more important for transfer than the other components. It's not enough to carry over skills and behaviors, you need the value function as well in order to maximize improvements. (4/N)
1
0
3
We focus on the SAC algorithm and perform an exhaustive study of transfer using the Continual World benchmark consisting of 10 robotic tasks. We explore the short-term transfer between pairs of tasks (10x10 = 100 pairs) as well as continual learning on the whole sequence. (2/N)
1
0
3
Imagine you have a robot steered by a neural network that can pick up objects. Now, the object it has to move is hidden inside a drawer.
1
0
3
@richinseattle
@CupiaBart
@JensTuyls
You might enjoy our latest paper then: it's about finetuning in RL in general, but we have a bunch of experiments on NetHack. There's also a growing community working on NetHack in ML, see e.g,
0
0
2
...however in practice, it doesn't happen! Even though CL methods remember A, they show much worse forward transfer on C than a simple fine-tuning approach. Solving catastrophic forgetting is not enough to have a good continual learner!
0
0
3
I'll be at
#ICML2024
for the rest of the week, let me know if you'd like to talk over coffee and/or lunch!
0
0
3
If you'd like to explore amazing ways how deep learning and neuroscience interact with each other, be on a lookout for our summer school! Co-organised by GMUM () and the wonderful folks at
@MLinPL
. More info coming shortly!
❗🔊 Attention, all machine learners – this summer you can learn from the very best in the fields of ML, CV, and computational neuroscience in the beautiful city of Kraków!
The MLSS^N summer school will be a joint effort between GMUM and ML in PL!
Details:
0
3
8
0
1
3
In continual learning terms, we can find a hyperrectangle of parameters that perform well on the current task and get a guarantee on the performance during the rest of the training, as long as we stay within this parameter region! (6/N)
1
0
2
Continual learning can be formalized as training on a sequence of tasks while maintaining performance on the previous tasks. Equivalently, we want to find the new solution within a region of parameters that perform well on previous tasks. (2/N)
1
0
2
Unfortunately, we had some sound issues during the first lecture, but after that it gets much better so don't get discouraged by that!
0
0
2
Finding
#4
: Reusing replay data from previous tasks can help a lot if done properly. Simply storing the whole past in SAC's buffer (Perfect Memory) is not a good idea, but distilling the knowledge through behavioral cloning works out great! (7/N)
1
0
2
You can also check out the full paper here:
0
0
2
To work with hyperrectangle parameter regions in practice, we represent each weight in a neural network as an interval [w_a, w_b] rather than a single point. Using interval arithmetic we implement the basic neural network components (layers, activations, etc). (4/N)
1
0
2
We take SAC apart into three components: the exploration policy, the paremeters of the actor, and the parameters of the critic. We study how carrying them over from previous tasks impacts the speed of learning a new task (forward transfer, FT).
Here are our main findings: (3/N)
1
0
2
Sneakpeek: It seems that even if CL methods can remember the past, they cannot efficiently reuse it.
Let's say we have a sequence of tasks A->B->C, where A->C exhibits high forward transfer, and B acts as a distractor. Then CL methods should remember A to perform better at C...
1
0
2
The empirical results are not groundbreaking, but we can match the performance of regularization methods (EWC/MAS/SI) while providing guarantees. Interval propagation is a bit tricky to work with due to stability issues, but CIFAR is still within reach (7/N)
1
0
2
These insights lead us to a simple, efficient algorithm we dubbed ClonEx-SAC. On the CW10 and CW20 benchmark sequences from Continual World, it considerably outperforms other much more sophisticated methods. (8/N)
1
0
2
Finding
#3
: Exploration becomes even more important when we get to longer sequences of tasks. Reusing policies from previous tasks to gather initial data in the new task improves the performance considerably. (6/N)
1
0
2
Finding
#2
: Contributions of SAC's components are additive, i.e. FT(actor, critic, exploration) ≈ FT(actor) + FT(critic) + FT(exploration).
Even though the critic makes the biggest difference, the actor and exploration still make considerable independent contributions! (5/N)
1
0
2
This way, given an interval neural network, we can compute the set of possible outputs for any particular input. Now, we can take the worst-case loss over this set, and as long as we stay within the parameter region, the performance will not get worse. (5/N)
1
0
1
However, the parameter regions can be highly irregular and the problem is NP-hard even if we assume they are polytopes as shown nicely by
@LauchLab
@syimplectic
@tommy_da_cat
We simplify the problem and assume all regions are hyperrectangles. (3/N)
1
0
1
@tim_zaman
@CupiaBart
At this point it was grasping at straws. But yes, we had cuda in our container, we just thought that maybe something about system cuda libraries interfered with our setup.
0
0
1
@LucasPCaccia
The regular FIM is the "correct" one for EWC, i.e. it was used in the original paper. I confirmed this with the authors at some point. Still, I haven't seen any experimental studies of the impact of using empirical Fisher in CL, it's an interesting question!
1
0
1
