Kagi
@KagiJournal
Followers
912
Following
65
Media
50
Statuses
330
Outcome independent. Dispelling the competency-locked fog of war. I study intelligence, specializing in RL.
Joined November 2023
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
ジャンプ
• 119625 Tweets
大社高校
• 76476 Tweets
#FOURTH_ALRIGHT
• 65965 Tweets
#CDTVライブライブ
• 64684 Tweets
FOURTH COMEBACK
• 62634 Tweets
神村学園
• 60038 Tweets
あと5話
• 53830 Tweets
呪術完結
• 46850 Tweets
カウントダウン
• 26412 Tweets
カミガタ
• 23639 Tweets
#音が出たら負け
• 21031 Tweets
ゲリラ豪雨
• 18915 Tweets
ACTRESS ENGFA FIRST MOVIE
• 17496 Tweets
ハンターハンター
• 17340 Tweets
キタニタツヤ
• 14084 Tweets
ドリアイ
• 11570 Tweets
1st MV OF POND
• 11457 Tweets
ファタール
• 11044 Tweets
I read it. 2h a day for 10 days. Worth it? Hell yes.
I'm not an academics guy, so it helped me catch up on what I'd get at university and what I was missing by being self taught.
Below, I'm sharing a ton of my thoughts regarding the whole experience.
550 pages of pure knowledge of fundamental reinforcement learning for free. I had too much practice recently and need to settle it all in with theory and struggle.
Authors claim it's a book for 2-semester course. Its amazing how beautiful life can be.
Goal: 2h of theory a day
13
93
1K
15
121
2K
550 pages of pure knowledge of fundamental reinforcement learning for free. I had too much practice recently and need to settle it all in with theory and struggle.
Authors claim it's a book for 2-semester course. Its amazing how beautiful life can be.
Goal: 2h of theory a day
13
93
1K
OKKAAAY I STARTED IT. At this point Im addicted to feeding my brain with quality content.
One paragraph summarizes their whole approach, that beats unreadable math equations by a laaarge margin.
Make it as easy as possible for people to gain competence and skill. We need it
Fantastic read, a must read, but I wouldn’t want to spend 2 semesters on this, hehe. Next up I might check out "Deep Reinforcement Learning in action". Lots of code there and at a first glance – a more practice-oriented teaching style with more visuals.
4
13
107
2
30
383
How do you like my AI lab colony? Virtual Los Alamos of Reinforcement Learning.
If I'll ever get a chance to do AI research, imma do it with style or not at all.
19
21
311
Discovered spiking neural networks recently, that more closely model biology, where neuron activation timing is crucial, and that are more energy efficient.
Sounds like a great side project in the future to get into it, but it's a complete paradigm shift, you dont use torch here
16
18
278
Fantastic read, a must read, but I wouldn’t want to spend 2 semesters on this, hehe. Next up I might check out "Deep Reinforcement Learning in action". Lots of code there and at a first glance – a more practice-oriented teaching style with more visuals.
4
13
107
OMG I CANT BELIEVE THIS COMMENT IS ALL I NEEDED TO MAKE THIS WORK. It's the first thing I tried this morning
I love you guys, thanks a lot, thanks
@jachiam0
- the main author of Spinning Up, which alongside
@vwxyzjn
's CleanRL made my project even possible. Heroes of RL🫡
1
3
59
ML grind challenge days 52-56 report:
On Tuesday I was almost out of hope, but I managed to solve the issues. I made a successful training pass and evaluation of hybrid action space agent. It's very close to being a usable prototype, here's a quick showcase:
2
1
58
This book is a cohesive view of RL, really put most pieces together for me after months of reading articles, blogs, papers and courses. It makes everything make so much sense and you'll understand things in both smart generalist and domain expert way.
1
0
51
Just read that Schmidhuber paper called Driven by Compression Progress, and oh boy, please read it. Just first 16 short pages.
It's a unified theory of what it means to understand the world. A concrete theory of beauty. The difference between external and intrinsic drive
3
3
49
I'm stunlocked, I need ML tech support. Been sitting on this for weeks now and Im probably in a cognitive local optimum and need fresh eyes on this.
Is there anyone with RL experience, especially with SAC, that would be willing to help me out? Twitter pls 🥺 Details on pic
4
4
43
It's a must read, but don't feel bad for skipping parts. Value your time. I often has these moments when I'm reading a subchapter and I feel like it's neither important nor interesting. If 2 pages in you think the subchapter is not for you, just skip it.
1
0
37
Pseudocode is top notch for concept learning, much clearer than half page math derivations. Math equations contain a lot of bloat for my straightforward mind. Helpful, but not always necessary. Dont feel bad for skipping it.
1
0
34
@AstleDsa
Spinning up is great is you want to understand the algorithms and how to code them. But Sutton Barto Book is when you want to understand background, ideas, notation, math and all other foundations to become really good at RL. There's no code here. Just practice vs theory
0
0
31
Best one so far, for me, was the first one. No neural networks here, just classic algorithms for solving simple environments, where mathematically every iteration guarantees improved of the policy, which is not the case for approximate methods.
Read part 1 most thoroughly.
1
1
28
Part 3.. I thought it’s gonna be my favorite because it deals with psychology, neuroscience and modern advances of RL, but it didn’t give me much overall. Books have their limitation and for understanding brain and ML analogy yt content is just better:
2
1
27
Remember that it's a classic written originally in 1990s and updated in 2012-2018. It's an influential work, so understanding their notation is worth the time as everyone follows it, but it has a reeeallly academic style.
1
0
27
Getting to specifics, the book consists of 3 parts:
1. Tabular methods
2. Approximate methods
3. Looking deeper
1
0
25
ML grind challenge days 57 and 58:
Yesterday I made successful speedup to not watch the training at 3fps.
Today I spent all day investigating why my models don't improve. I carefully looked at how I process observations and memories. It worked perfectly. And then I saw this:
2
0
23
It's been a month of my RL project. Can your tools do this?
I dream of an algorithm playground, a gallery where I will compare different algorithms and develop new ones. I dream of AI tournaments in complex, appealing environments.
Of course it's hard. It has to be. More below
3
0
23
Day 39: First steps with ML in Unity
I tried integrating Unity with Python, aaand.. it worked 😏 I mean not immediately, of course. But Im digging into the docs and playing with variables and source code to see how it works. There are no sources to learn this
2
0
19
ML grind challenge days 59 and 60: Mystery solved
Spent last days deep diving into every tensor, just to find out that everything works well and as intended.
The biggest clue as to what's up was my pioneering work with discovering new, unknown to humankind ways to mess up.
1
0
19
It also introduces stuff that I've never heard of in any course and that are rarely used in practice. It's good to at least glance over it to know what's up and where to search for information in case you need it. Just be aware of existence of things like tile coding.
1
0
19
Whenever I struggle with a project I think that this hardship is exactly the reason why very few people have made it to this point. This hardship is exactly the reason I must continue
Im burning day after day just to learn what does not work. Feels.. Not great. It will work soon
0
0
16
Part 2 is nice with introduction to neural networks, but it treats everything from scratch, while you can simply understand the difference between it and the tabular method version and that's completely sufficient.
1
0
15
Respect to all game developers, especially indie. The time needed to make good games is beyond impressive. I could make games, but I prefer to be an AI researcher with a game development hobby for showcasing the work.
1
0
14
One year ago I admired people "smart enough" to read scientific publications.
Now not only I read papers myself, but I'm entertaining myself with videos (below), that point out papers' lack of sense.
You guys would not believe how many papers are just noise not worth reading.
2
0
12
Feel burnt out with my main project and will soon need fresh air as a week-long fun exploratory study. What should I work on?
1. Intelligence is compression, Hutter Prize, Schmidhuber's paper
2. Spiking Neural Networks, modelling brains
3. Crypto basics, investigating 21e8
>>
Intelligence=Compression
11
SNN, modelling the brain
8
Understanding crypto
1
New programming languages
4
2
0
10
That was my vision every since I began: "Wouldn't it be cool If had an AI playground that showcases my work, shows progress and advancements of reinforcement learning and is visually pleasing at the same time?"
1
0
11
I'll read it during the summer for sure. After the summer it's gonna be time to get some experience in the industry. I feel like I'm getting really good at this stuff.
ONWARD 🫡
1
0
10
I'm reading this one mostly because it describes algorithms that weren't present in Barto Sutton RL book: Genetic algorithms, curiosity-driven learning and attention.
I immediately skipped part 1 of the book, about markov decision processes and actor critics. I know all of it.
0
0
10
Took more time than expected (as always), but I will soon be back to implementing more algorithms. I have all the skill I need, just need the time.
For now trying to see if people like it and If it all makes sense.
1
0
10
Modern advances though, are always inspiring. AlphaZero stuff was short and nice, I read a lot about it elsewhere so it’s not a discovery for me, but I can recommend reading it for the pipeline – what is needed currently to make something that good. It's an industry insight.
1
0
9
Day 40: Learning Unity particle system
I spent the day learning how to make great vfx with particle system, so it looks and feels nice. Im also combining continuous and discrete actions in one agent, so the complexity grows, hehe. Its satisfying to progress this, well done
2
0
9
Day 42: Figured out how to use example Unity envs
Not too satisfied with today, did less than 3h of real work and just wanted to go next. I planned to do marathon envs, but they werent maintained well and had so many errors, that I pivoted to official unity example envs
3
0
8
@MaxMynter
Damn, flex with the physical copy. Would be a shame if you didnt read it 😏 But cherry pick what to focus on, not every part is worth it if you know your stuff already
0
0
7
Day 81 of our ML grind challenge.
Today marks the day I got the multidiscrete branch of my hybrid hyper universal soft actor-critic working. Continuous part worked for a while now, so all the pieces came together.
Here is a discrete agent learning to go right in 15 seconds
1
0
6
Day 43: Super secret new RL repository for AI research
Monday. Today we course corrected. We're on a great track. I started a new secret repo that will hold my future RL algorithms interfacing with Unity. Lots of docs reading and playing with example envs to poke them around.
2
0
7
My head was steaming today, I felt like I was cooking. I was just crazy exploring everything imaginable. I looked for next thing to work on. Flexible Muscle-Based Locomotion for Bipedal Creatures? Learning to Brachiate via Simplified Model Imitation?
3
0
7
@jachiam0
Achiam, it's hard to express my gratefulness to you, I already fixed it because of you (forgot to check dms). Im writing a post rn
0
0
4
@noahgsolomon
That's my main project for months😏 Normal python RL, but with Unity engine instead of gym envs. So, limitless possibilities with custom environments for research.
When I saw your elegant PPO Bunny some months ago I was like: "Yeah thats exactly the direction I want to go in"
1
0
6
@0x_pix
Go grad if:
- You want phd
- Unsure what to focus on
- Want a safe and standard way
- It's free
But for the most brilliant, uni only slows em down. I got engineering degree as a great broad education when I wasnt experienced. Now I know what to study, so I can go faster myself
1
0
6
@noahgsolomon
Niceee, glad you like it. I started it and it will put so many pieces together, especially with the clearly explained standard notation. Love these influential works that inspired papers and blog posts that I already read
1
0
5
Day 41: Finished the basics of Unity ML
I finally figured out an issue that was on my mind since yesterday, when I had to leave it to go to sleep. It works now. I know pretty much all the basics. The only work left with this project is better visuals & models and then try it out
2
0
5
If you have any idea what might be going on, feel free to dm me. I can send the code or even sit on vc providing more details to make it as easy as possible to help me.
1
0
5
@0x_pix
I still consider doing my degrees, but only in alternative speedrunning ways of showing my portfolio and skipping classes based on skill. For now happy with my daily grind and how much I've learned
1
0
5
@notmoeezm
Kind of, yes. The book assumes you know nothing, but I spent last 4 months daily coding and reading stuff and doing courses, so the main friction of "wtf is going on" is far behind me. Just connecting the dots.
My superpower is also recognizing what is not worth reading, hehe.
1
0
5
@0x_pix
Uni was still a pain. Unimportant filler classes, 1h programming then 5h of writing the report, group projects that incentivize you to do as little work as possible, classes averaged up to help underperformers and slow down the best, ugh. Really resisted phd path, not my style
1
0
5
@vwxyzjn
Costa! I wouldn't be doing this if it wasn't for you, you're at least partially responsible 😏 I got to appreciate why you seed and graph everything
Thanks, If I'll have to, I'll do it. I could slowly morph your code into mine, but I work with Unity, not gym, so it takes a while
2
0
3
But I mean it, the graph is not useful. Im not even sure how to further approach this. Do I print grads after every operation and go step by step? Something breaks my backprop or the gradients vanish. Ugh. Machine learning is my passion if I had to be honest. AAAAAAAAAAAAAAAAAAAA
0
0
4
@trivo_121
@lelouchdaily
Yoo, welcome, good luck 🫡 I have never learned more over a period of time since we started. Hopefully you can too
0
0
4
@vwxyzjn
CASE CLOSED 😏 I managed to fix it thanks to the comments, so no need for more help. Love you guys
0
0
3
It seems like there was a tutorial back in the day on google colab, but with updates it got outdated and had to be removed. Anyway. Im just excited to work on this. I will spend however much time I need to make it great.
1
0
4
Day 50: Separation of Actor and Critic
Today I combined buffers and separated actor and critic for modularity, bcs I forgot soft q net uses 4 q nets total..
I'll have to work hard to understand how to optimize a model that could produce both continuous and discrete actions
2
0
4
As you see, it runs with different agents, each with distinct inputs and outputs. But that's been working for a while now. The problem is making it learn well in this extremely universal way.
Continuous action space agents learn well, currently struggling with multidiscrete ones
1
0
4
Spent the day with my beloved AlphaZero and MuZero. I saw example implementations, I watched paper analysis, I learned Monte Carlo (from casino name) Tree Search and the whole architecture. Not doing it yet, but my own implementation will be on potential to-do list
1
0
4
Day 44: Brainstorming and library foundation
Slow beginnings. I was brainstorming, even made notes on functionalities and how I want to piece the puzzles together. Brings me joy to sit on code, but in a creative way. No tutorials on this, just me and my brain. Also went gym, lfg
1
0
4
@SchmidhuberAI
Let me be an apprentice in your lab.
I’m a brilliant RL guy, a smart generalist that’s diligent, great with people, good at speaking and writing. I’ve got it all except mentorship and environment of people more experienced than me.
If you have no money, you should have no shame.
Knock.
Call.
Email.
Text.
DM.
Ask.
Life-changing doors don’t open themselves.
308
2K
12K
1
0
5
@CaxCaxCat
Oh officially it's just a research facility for improvement of AI in video games. Unofficially? Can't say it.
0
0
2
4. Random programming languages to get a feel of the syntax to understand the world better
I'm also thinking about
- Genetic algorithm
- MuZero implementation
- Curiosity-driven learning
But these are RL ideas too close to what I do right now, so I need something else
0
0
3
Day 49: Memory recall working completely ;)
Rewrote the main loop clean and Im really happy with how it looks for now. Memory made it tricky with my old mess, so I had to do it. I have everything needed for backprop now.
Went to the gym, Im just pumped up. Next week is crucial
0
0
3
Official docs just list main classes, attributes and methods and you brother figure out what to do with it. I'm not surprised nobody's working on this in public. If I learn it, I could literally just make a course about developing ML with Unity
1
0
3
Feels like Im doing some pioneering work, that will enable me to develop AI algorithms, but make it pretty and satisfying to interest an audience. I read Unity's paper that they released with ML Agents and I believe in this project.
1
0
3
It's so satisfying to get it to work. I set up basic integration, when I pick a random action for my agent and it executes the action in Unity, in Editor in real time. The potential is huge with side channels to enhance communication. And everything is open source so I can dig in
1
0
3
Anyway, tomorrow I attend a group meeting, so it will be a lighter day.
I love this project's idea and potential, but it might have been a little too ambitious to make it that universal. But now I gotta finish it, hehe. I'm learning so much, I'm not gonna lie
0
0
3
The upside of the struggle, though, is that I become pretty good at this. I've seen so many problems already. Tools become a natural extension of my body, and I struggle more with concepts and ideas rather than tools.
Time goes by, the plan remains. It will work one day
0
0
3
Quickly skimmed through their paper and it taught me about a branch of RL I didn't know before.
They have differentiable simulators, so you can backpropagate through the physics with respect to inputs, so you get closer to the global optimum.
Interesting, but not universal.
Excited for ICML next week! I'll be presenting Adaptive Horizon Actor Critic - a model-based RL method that learns high dim tasks in minutes using differentiable simulation.
Stop by Hall C 4-9 or get in touch if you want to grab a coffee some other time!
More on AHAC in the 🧵
7
40
315
1
0
3
@lelouchdaily
@trivo_121
Watch this vid , its quite entertaining. If you dont get the math, learn in this direction, then watch the video again. Then another course, then the video again, lol. Your brain will click with time. Then implement algorithms yourself, or do Kaggle
0
0
3
Day 46: First architecture design
It's getting harder to simply explain what I do. Today I spent my day figuring my new hybrid architecture PPO. How to make it accept visual and vector inputs, and output both continuous and discrete actions, huh? Had to relearn conv layers
1
0
3
Day 48: Lot of improvements and memory recall
Another big coding day. I made many improvements, so the model pass works in all cases. I started conceptualizing optimization, but I need to have a memory recall working first, so I spent time on it. It's not done yet
2
0
3
I aim at having intuition for the state of the art AI models and catch up to all modern knowledge, so I can bring the singularity a bit closer. Hotz catched up in 6 months with all the knowledge a few years back, I can do the same in the timeframe of this order of magnitude
1
0
3
@0xlemo
Oh is that really his point? I think he's saying that LLMs are quite useful, but they will never be agi bcs of their fundamental limitations.
So, generative ai is a cool tool and should be developed, but it's not a priority when it comes to advancing towards agi.
He's right
0
0
3
Today I did final little tweaks to the PPO, implemented pixel-input and continuous action space versions, as shown below, with half-cheetah with 6 float inputs. Didnt have more time to find good params for it, but I consider it solved. It flips later and smth breaks, but it works
2
0
3
@lelouchdaily
@trivo_121
Yup, exactly. If you don't know all the math you can literally just track what
@lelouchdaily
is doing 😏 Then its time for general ML courses like the famous ones from Andrew Ng on coursera, then you specialize. Karpathy has best possible tutorials on LLMs and backpropagation
2
0
3
@NeoGranicen
The keywords to search through the web is ML Agents and Unity. There are many videos and tutorials about constructing an environment and training the models.
The difference with me is that I trained them by hand in Python, which nobody does and you cant find it anywhere 😏
1
0
3
When I started it I didn't know what I'm signing up for. I would never think that there is no working multidiscrete SAC reference available for learning, I'm kinda just doing stuff to my best understanding. Month of everyday work and the amount of progress versus time spent hurts
1
0
3
@nordic_eacc
More like 20 hours, hehe. When I measured it, it was roughly 4min a page, it's not even that bad. When you don't start from nothing and know what is not worth reading, it's not a pain to go through
0
0
3
Zeroed gradients, hehe. Nice. At first I just printed out model weights and they barely changed since random initialization. So, we're getting somewhere.
Took me like an hour to make a computational graph to notice anything out of ordinary, but it ended with "ahh hell nah"
1
0
3
Day 47: First "Observation -> Agent -> Action" pass
I programmed so much today, I'm so happy. Not sure how, just past few days spent on brainstorming and conceptualizing all these elements in my head, and today I wrote a lot of code. Getting familiar with the tools, I suppose ;)
1
0
3
OH I almost forgot that Karpathy has a blog! I should just read the wholeeee thing, just like I read whole Hotz's blog and it was super nice. I should just stick to Karpathy. I will go wherever Karpathy wants me to go. Also saved a tutorial on cuda kernels from Howard for later
0
0
3
Detailed work report:
1. Day 52 was my lightest day since I started, just symbolic 30 minutes before bed. It was a social day
2. Day 53: Made working batched pass and multidiscrete action picking
3. Day 54: Optimization beginning, a lot of research
1
0
2
I accomplished the main goals. Finished off whole Karpathy, even read all his blog posts. But I ended up exploring saved resources that made me understand what is ahead of me. I did a lot of code-watching, but didnt write much though.
This week we want to understand and practice everything from Karpathy's tutorials. We already watched them all, so a bit of rewatching to do and a ton of writing code and playing around. Reshaping matrices and adding pytorch custom buffers has to be effortless
0
0
1
2
0
2
@lelouchdaily
I really felt out of energy yesterday, so I'm going to the cinema today for the last days of Dune 2 😏
1
0
2
Didn't do Hackerrank yet, Im making a slow and steady progress with lifestyle improvements.
I want to write early in the day what I plan on doing that day to set the vibe. If I start doing "whatever" I will end up with unaligned attention
0
0
2
And even if it starts learning, I have a long way to go with exporting to .onnx file for inference. I made initial attempt and I know how many operations arent supported in onnx, which is especially a problem with my universal and messy approach.
1
0
2
A lot of thoughts on what I read. About making a PhD, about his biohacking journey even, but the most important thing is that it got me thinking and I got to know his life better.
The plan for now (still got 3/4h before I go to sleep) is gym, then sit on code as much as I can
1
0
2
@lelouchdaily
When I feel like it, I just do 2 days in a day 😏 Nah, you just had days when you had to do client work and couldnt do anything with ML. I didnt miss a day since we started on 4th March
I will, but I gotta get insanely good, hehe. Its like I picked the perfect thing to work on
1
0
2
On a different note: It's been a month since we started. I have learned so incredibly much. But still I got some big parts missing. Max one more month of learning, studying, reading papers, and then nothing will stop me from making cool flashy projects and potential papers
0
0
2
@noahgsolomon
It's even better than I thought. Densely packed, simply explained and intuitive concepts 😏 Kinda more like your style. Every equation explained in detail with annotations. Barto Sutton has to be read, though, to build on their foundations
0
0
2
I understood the code, I progressed understanding of the math equations, cause the symbols are often confusing if they dont explain everything..What is left with SAC is to test
1
0
2
@lelouchdaily
I also started thinking about some time dedicated to playing games just so I can watch yt unrelated to ml, lol. Like I have no idea how to listen to mimetic theory lecture otherwise
1
0
2
I want to get the basics down with stable diffusion, language models, classifiers, but ultimate specialization is reinforcement learning and new architectures that will make learning as data efficient as it is for animals and humans.
1
0
2
@noahgsolomon
Absolutely stunning. Love the interactivity, love the chill intuitive language, the world needs this style of teaching
0
0
1
@miguelalonsojr
@vwxyzjn
Thanks Alonso, I already managed to fix it! Pairing it up with CleanRL would a great idea, I thought I'd have to port CleanRL code with LLAPI. When it comes to using mlagents SAC, Im doing it myself to do my own research on new algos in Unity, so dont want any wrappers for now ;)
1
0
2
Day 39 (for real this time, I miscounted it previously): Custom RL training, struggle with reward system
Dear journal, today I went haaard. But it was painful. Integrating my own RL code with Unity framework is not seamless. Old code depended on gym attributes, that I cant use
1
0
2
I finished SAC.. I finished value based and policy based RL algorithms. I implemented the discrete action version as a template for future use, but stumbled on technical problems along the way today
1
0
2