Nishanth Kumar
@nishanthkumar23
Followers
1,292
Following
752
Media
22
Statuses
271
Robotics + AI PhD Student @MIT_LISLab @MIT_CSAIL , and intern @NVIDIAAI . Formerly The AI Institute, @brownbigai , @vicariousai and @uber . S.B @BrownUniversity .
Cambridge, MA
Joined July 2015
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Florida
• 247253 Tweets
Bivol
• 122300 Tweets
#precure
• 91485 Tweets
Nico
• 64803 Tweets
Oregon
• 62644 Tweets
#AEWWrestleDream
• 54793 Tweets
Penn State
• 37740 Tweets
#仮面ライダーガヴ
• 36799 Tweets
Burke
• 36057 Tweets
Tennessee
• 35842 Tweets
Dick Cheney
• 30765 Tweets
Dennis Quaid
• 29620 Tweets
マンホール
• 28138 Tweets
#ブンブンジャー
• 27907 Tweets
Ohio State
• 26556 Tweets
#UFCVegas98
• 23300 Tweets
ワコール
• 21076 Tweets
まゆちゃん
• 14927 Tweets
いろはちゃん
• 14864 Tweets
Willow
• 11509 Tweets
ヴァレン
• 11447 Tweets
Ole Miss
• 11230 Tweets
マッドレックス様
• 10371 Tweets
Pinned Tweet
Can we get robots to improve at long-horizon tasks without supervision?
Our latest work tackles this problem by planning to practice!
Here's a teaser showing initial task -> autonomous practice -> eval (+ interference by a gremlin👿)
1
18
112
There was a lot of good and interesting debate on "is scaling all we need to solve robotics?" at
#CoRL23
. I spent some time writing up a blog post about all the points I heard on both sides:
22
47
257
Life Update:
I'm excited to say that I'll be joining one of my dream Ph.D. programs
@MITEECS
in the near future! I can't wait to work on making robots smarter with the amazing people
@MIT_LISLab
and
@MIT_CSAIL
more broadly! 🤖
21
1
213
Super excited to finally release something I've been working on for a while now! PGMax is a new
#JAX
-based framework that aims to make it easy to build and run inference on probabilistic graphical models (PGM's)!🧵👇:
Want automated, differentiable, message-passing inference for large graphical models and probabilistic programs?
PGMax can create an efficient implementation for you, while being end-to-end differentiable & running seamlessly on CPU, GPU and TPU! Check
1
19
101
2
12
114
Ever heard about "Bilevel Planning" or "Task and Motion Planning", but been unsure what those words mean? Ever wanted a gentle intro to these methods so you can just understand what's going on? Our new blog post might help!
3
18
81
Mini life update: I’ll be interning at
@NVIDIARobotics
in Seattle this summer. Excited to work with
@CaelanGarrett
and others on combining planning, learning, and foundation models for real world robots 🤖!
1
2
61
I'm excited to release a preprint of my first first-author work during my PhD! (paper: , video: ) 🧵👇
5
4
54
Had a really cool experience
@MIT_CSAIL
today: I got to meet Prof. Gerry Sussman and ask him if it's really true that Marvin Minsky once asked him to essentially solve most of
#ComputerVision
in a summer when Sussman was an undergrad.
1/3
1
4
50
Small Update: After graduating from
@BrownUniversity
last week, I'll be a summer intern
@vicariousai
before starting my PhD! I'm so excited to learn from incredible researchers like
@dileeplearning
and contribute to the mission of deploying AI-powered robots everywhere 😀
2
0
45
Ever wanted to train your robot in one house but zero-shot generalize to a similar task in a *different* house?
Our
#CoRL2023
paper learns symbolic operators from demos, then leverages TAMP at test-time to zero-shot new tasks, like the challenging BEHAVIOR-100 task below! (1/n)
1
12
39
I’ve been an avid reader of the MIT grad blog for a while; it was a dream come true to get the chance to contribute to it :)
0
1
29
I may be biased, but
@tomssilver
’s thesis defense on Neurosymbolic Learning for Robots is probably the best one I’ve ever been to! Getting to work with and learn from Tom is one of the best strokes of luck I’ve ever had 😊
I defended my PhD
@MITEECS
this week! Thanks to everyone who came out. And thanks especially to
@nishanthkumar23
who not only managed the Zoom, but also got me this amazing gift…
21
3
142
1
1
26
This is the first major project I got involved with and was a blast to work on! Check it out 😀
Teach a robot to perform complex actions with a Mixed Reality headset! Check out our IROS 2020 paper!
0
8
34
1
1
20
@lawrennd
As someone else pointed out, a version of this observation has been known for a while within the Robotics community: Moravec's Paradox ()
0
1
17
I'm really enjoying listening to the
@therobotbrains
podcast with
@pabbeel
- so cool that there's a dedicated podcast at the intersection of AI and Robotics! Can't wait for new episodes and guests (personally, would love to see an episode with
@rodneyabrooks
) 🤖
1
0
17
Sussman apparently ended up giving Minsky a technical report on why exactly the problem he'd been given was so hard, and this inspired some of the first research into CV! Sussman's moral of the story: naivete and gutsiness can be useful 🙂
3/3
1
2
17
@chipro
One of my favorites yet: "Human Augmented Robotics Intelligence with Extreme Reality (HARIX)". 'Extreme Reality' makes me just imagine someone blasting heavy metal music as a robot fumbles to pick up a grape...
0
0
15
Excited that the project I helped start during my internship continues to be updated and useful for PGM research 😄
Check out the recent updates to PGMax, our JAX package for implementing large discrete probabilistic graphical models. It can now solve the smooth dual of LP relaxation of MAP problems orders faster than LP solvers, & more 1/6
1
27
143
0
0
13
Looking forward to attending
#AAAI23
starting tomorrow!! I'll be presenting our work on inventing predicates for TAMP with
@tomssilver
; come check out the oral and poster, and also email me if you'd like to chat one on one 😄🤖
0
0
13
Excited to have been recognized by the CRA! Couldn't have done it without the amazing people at Brown and
@BrownBigAI
, especially
@StefanieTellex
Nishanth Kumar ’21 Named Computing Research Association Honorable Mention Outstanding Undergraduate Researcher.
0
0
4
2
1
12
Come check out our
@corl_conf
work on studying Active Learning for self-driving labelling pipelines! I'm quite excited about the potential for AL applied to self-driving and am looking forward to seeing + producing more future work in this vein.
Tired of spending thousands of dollars labeling self-driving scenes? Check out our
@corl_conf
paper on fine-grained active selection for perception and prediction to leverage labeling budgets most effectively:
Presenting at Session 5 today, come say hi!
1
2
12
1
1
12
Come learn more about ATG! Interning here has been an incredible opportunity and absolute pleasure 🙂
0
0
11
Excited to announce I've been named a 2020 Goldwater Scholar! The honor is a dream come true and one that wouldn't have been possible without all the amazing advisors and collaborators at
@BrownBigAI
and
@BrownCSDept
!!!
#goldwaterscholars
We are so happy to welcome the class of 2020 Scholars to the Goldwater Scholar Community!
#goldwaterscholars
0
6
32
1
1
10
He said it's very much true, and that Minsky later told him why he'd done that: all his grad students had been too afraid to work on CV since even fitting an image into memory was hard those days. But a naive undergrad like Sussman didn't know any better!!
2/3
1
0
9
This was such a fun and informative class - certainly among the top 5 I've taken at Brown!
Really proud of students in my class who read and discussed papers, and executed a research project in 2 months. We had the final project presentations today with guest judge
@orlitany
. List of projects below
2
2
45
0
0
9
I spent ~2 hours today debugging code that took me ~1 hour to write 😬. The culprit turned out to be due to strange Python list behavior that's been known about for at least 12 years:
3
0
8
I had an amazing time interning with
@RaquelUrtasun
in 2020 and would highly recommend this to anyone interested - it's an absolutely incredible opportunity to develop research skills!
Research internships are now available
@Waabi_ai
. All year long, with duration of 3-12 months. Available in both Canada as well as US. Join the team at the forefront of innovation in
#SelfDrivingCars
!
Apply:
3
52
218
0
0
8
It's almost October, which means it's almost time to submit the NSF GRFP! I found sample successful applications extremely helpful when I was applying. So in the spirit of paying things forward, I wanted to share my own application:
0
0
8
Do check out our companion blog post (), documentation (), and GitHub () for more!
1
0
7
Really cool work on operating robots *telepathically*! Interesting to see they do this by sequencing parameterized skills, which is *exactly* how bilevel planning () works. Could be interesting to study combining this approach with planning in various ways!
Introducing our new work
@corl_conf
2023, a novel brain-robot interface system: NOIR (Neural Signal Operated Intelligent Robots).
Website:
Paper:
🧠🤖
20
185
755
1
0
7
Very interested in hearing thoughts/feedback/suggestions on this if you do check it out! 🤖
1
0
7
I got a sneak peek of this talk and was extremely impressed, be sure not to miss it if you're attending
#AAAI23
!!!
Looking forward to
#AAAI23
! On Tuesday, I'll present work on neuro-symbolic learning for robotic planning at the Bridge Session on AI & Robotics.
I'll show some clips from this 1972 video of Shakey the robot and ask: how much progress have we really made?
2
0
26
0
0
7
@chris_j_paxton
Agreed! We have some recent work that uses VLMs for perception, navigation and grasping skills from BD’s Spot, and a task planner, and found it works quite well (and enables online improvement!). Definitely a long way still to go, but recent VLM+LLM developments are exciting 😄
Can we get robots to improve at long-horizon tasks without supervision?
Our latest work tackles this problem by planning to practice!
Here's a teaser showing initial task -> autonomous practice -> eval (+ interference by a gremlin👿)
1
18
112
0
0
6
It was so fun to be a part of this work on using LLMs for state abstraction! Check it out, and stay tuned for more in this line of work
Introducing LGA (Language-Guided Abstraction) at ICLR 2024! 🧵
📰 Paper:
🌐 Website:
🗞️ MIT News:
State abstraction is key to generalizable learning, but how do we know which features are task-relevant?
2
22
80
0
0
6
This is really cool work that shows that TAMP + CV can aggressively generalize to a wide array of challenging long-horizon manipulation tasks directly from pixels! Definitely don't miss the talk and poster if you're around
#ICRA2022
.
If you're attending
#ICRA2022
, check out recent work led by
@AidanCurtis3
,
@xiao_lin_fang
and
@CaelanGarrett
on enabling task and motion planning to manipulate unknown objects (with a real robot!).
Talk will be at 10:45 AM 05/24 in Room 112B
Paper:
2
4
11
0
0
5
This is an incredible opportunity for people interested in self driving - Raquel is an amazing advisor!
If you are interested in joining my group as a Masters or PhD student, apply here . Deadline December 1st. I recommend you mention this fact in your research statement. Join the
#selfdriving
revolution!
#gradschool
#AI
5
43
172
1
0
6
Interesting discussion/debate on the future of robotics! Glad to see some constructive argument on popular positions :)
I largely disagree with this, I think this is a classic underestimate of what in robotics is actually hard vs what learning is good at
3
6
112
0
0
6
I had the opportunity to attend Shriram's talk on this subject last year and learned a lot! This is a great opportunity to demystify the process for anyone considering applying to a US CS PhD program in the near future!
On Sat, Oct 30, 10am US/Eastern, to dispel the vast misinformation out there, I will answer questions about applying to US computer science PhD programs. Especially hoping to reach HUGs, students w/out access to research, etc. Please help spread the word!
3
93
183
0
0
5
@froodie
I'd second
@StefanieTellex
, she changed my life, gave me unprecedented self confidence and inspired me to pursue Robotics + AI!
0
0
5
This talk managed to be both deeply educational and enjoyable at the same time - I'd *highly* recommend watching it!
To commemorate more than 50 years
@MIT
, our very own Tomás Lozano-Pérez gave a recent
@MIT_CSAIL
RoboSeminar talk reviewing some history of robotic manipulation and highlighting some important directions for future work. Check it out below!
0
2
14
0
0
5
@lexfridman
Very much agreed! I find myself unconsciously falling into envy more often than I'd like and it's great to see things like this to wake me up. Thanks for sharing!
0
0
5
@TaliaRinger
@natolambert
As someone starting a PhD soon, this seems like great advice and largely corroborates my undergrad research experiences! Thanks so much for sharing!
0
0
5
This was so much fun to work on during an (extended) internship at the Boston Dynamics AI Institute! Special thanks to
@tomssilver
,
@williebeit
,
@LinfengZhaoZLF
, Steve Proulx, Tomás Lozano-Pérez, Leslie Kaelbling, and Jenny Barry for incredible dedication and support throughout!!
1
1
6
This is by far my favorite work so far! Overall, I think the direction of combining planning and learning for robotics will enable us to solve a lot of the long-horizon, multi-goal tasks (like cooking!) that have so far been too difficult. Stay tuned for more work in this vein!
This new preprint () is the culmination of a lot of my work with Rohan Chitnis over the last two years.
4
11
67
0
0
5
While interning at
@vicariousai
last summer, I got to see the power and flexibility of PGM's first-hand. Unfortunately, getting one working involved a lot of highly non-trivial code, and getting it to scale was even harder. So we set out to fix this!
1
0
4
PGMax's tight integration with
#JAX
makes inference extremely scalable and efficient (especially when run on a GPU or TPU), so much so that it can gracefully scale to large, complex models that operate on images like the Recursive Cortical Network (RCN)
1
0
4
What's more, PGMax can leverage
#JAX
for things like automatic batching, parallelization, and even differentiation through the inference process, which opens up a number of avenues for future PGM research!
1
0
4
So honored to have met a personal hero and Turing Award winner Raj Reddy! His amazing story, work and success have inspired me to persevere as a researcher when things have been tough.
1
0
3
I've been making Python lists for ~5 years and had no idea this existed. Looks like there's more where that came from () and I have a lot to learn!
0
0
4
Excited to announce I'll be at Uber ATG Research in Toronto this Summer! Excited to work with Raquel Urtasun,
@mengyer
and others to help SDV's become a reality!
0
0
4
Just like popular deep learning frameworks, the main thing you need to do in PGMax is specify a PGM architecture (we have built-in support for common types of factor and variable nodes!). Once you do this, you can perform efficient inference automatically to get your results 😎
1
0
4
@Vikashplus
I’m not so sure about LLMs being strong TAMP solvers, are there particular recent works that have convinced you of this?
1
0
4
@dwarkesh_sp
Very cool post that’s quite thought provoking imo! In case it’s relevant, wrote up a very similar post, but much more centered on robotics:
0
0
2
A few days late, but this is the best Christmas gift I've gotten in recent memory :)
Our whole crew got together to celebrate the start of what we hope will be a happier year. Happy New Year from all of us at Boston Dynamics.
574
5K
13K
0
0
4
This was a great lecture that I *highly* recommend, especially because of how well it gets at the main intuitions behind TAMP. Even as someone who would consider myself 'intermediate' with TAMP, I learned a *lot* from this lecture!
Our very own Rachel Holladay gave a recent guest lecture presenting a conceptual intro to task and motion planning (TAMP). Check it out if you'd like to learn more about the big ideas behind TAMP + some of our group's recent work in this vein!
0
3
16
0
0
4
@MohdAthar_JMI
@MITEECS
@MIT_LISLab
@MIT_CSAIL
Sure - I'll probably write a blog post about my application process and include my SoP and other relevant essays 😄
1
0
2
Honored to have received the
@official_ilurs
Best Plenary Presenter award! Couldn't have done it without amazing researchers and PI's at
@BrownCSDept
Humans 2 Robots and the Intelligent Robot Labs!
Congratulations to
@BrownCSDept
undergraduate Nishanth Kumar, who won the
@official_ilurs
Best Plenary Presentation award for "Action-Oriented Semantic Maps via Mixed Reality". More details, plus links to his presentation and his demonstration video:
1
1
5
3
0
3
@kritipraks
@deviparikh
Thanks so much for sharing these - the post on editing papers helped me finally get a
#AAAI2022
paper under the page limit! 😄
0
0
3
The robot repeatedly chooses a skill to practice and then plans to practice it. To select a skill, the robot asks: "How much would the skill improve through practice?", and as a result, "How much would I improve at solving human-given tasks?"
1
0
4
Lastly, I want to say a huge thanks to Stannis Zhou and Miguel Lazaro-Gredilla for dreaming up this project and working with me on it! Also, a special shoutout to
@SingularMattrix
and the entire
#JAX
team; we couldn't have built this without
#JAX
!
1
0
3
@ChristophSalge
@FloRicx
@togelius
@IJCAIconf
I'm one such student! Ive been working on the paper I submitted for over 6 months and this was my first first-author AI submission. I was hoping an acceptance would help my PhD application, and at the very least, I'd get feedback to become a better researcher, but oh well...
2
0
3
Also excited to explore Seattle more and meet new people; reach out if you have any recommendations, or will be in Seattle and want to chat at any point!
1
0
2
1
0
2
0
0
2
Check out our website for links to the paper and code, technical details, links to uncut and unedited (2+ hours!) robot videos, and of course assumptions and limitations!
1
0
3
@TheXeophon
@chris_j_paxton
Interesting discussion, and hard agree that symbol creation is integral (very excited about a few current efforts towards this)! I’d like to see more examples of symbols grounded in real sensory data (not just language), and ways to have planning inform LLM output + vice versa!
0
0
2
@mengyer
@nyuniversity
@CILVRatNYU
@NYUDataScience
@zemelgroup
@RaquelUrtasun
This is incredible! Congrats!
1
0
2
Prof Sridhar is an incredible mentor with an exciting research agenda: this is a great opportunity so definitely apply!
I am looking for PhD students to join my group in Fall 2021
@BrownCSDept
If you are interested in 3D computer vision and machine learning, please reach out or meet me at
@NeurIPSConf
:
Applications are due Dec 15:
RTs appreciated!
14
159
505
0
2
2
@BoyuanChen0
@DrJimFan
Thanks so much for organizing Boyuan and
@DrJimFan
; I really enjoyed the discussions!!
0
0
2
Here are full videos of the robot practicing and improving in two real-world environments!
1
0
3
Enabling continual and lifelong learning on real robots is, imo, one of the best uses of the symbolic structure inherent to TAMP and other structured planning approaches! I’m very excited to see and try to contribute to future work along these lines!
For the past year I have been thinking a lot more seriously about what
#LifelongLearning
entails in the context of
#Robotics
. This *new preprint* is one first step towards answering that question!
🧵👇
1
4
46
0
0
2
@dabelcs
@watermicrobe
@StefanieTellex
Seconded! I stumbled into
@StefanieTellex
's lab as a freshman with no formal CS experience and she not only made me believe I could do good research but has also continued to give me more opportunity to learn and do research than I could ask for!
1
0
2
Mengye is so fantastic to work with; this is a great opportunity!
I am looking for incoming students at NYU Courant CS or Center for Data Sci starting fall 2022. If you have interest in working with me on topics like meta/continual/embodied/representation learning+vision, please reach out & mention me in your PhD application. Deadline is Dec 12
14
80
352
0
0
2
Just migrated my personal website to Jekyll using the amazing Minimal Mistakes theme by
@mmistakes
! Check it out at .
If you like the design and want to use it, here's my customized git repo based on the original theme:
0
0
2
I've been a fan of Eric's work for years and this seems like an incredible opportunity for anyone who could use it!
Got a lot of interested emails in the last 24 hours (!!), here's some more details on the "free office hours"
1
7
55
0
0
2
I'm also proud that the code for this work is fully-integrated into our larger GitHub repo that's chock-full of other learning capabilities ()
1
0
2
@_ericrosen
Thanks so much Eric! Wouldn't have been possible - and I wouldn't have ever joined the lab actually - without you :)
0
0
2
This was an absolute blast to work on with
@williebeit
, Rohan Chitnis, and
@tomssilver
! I'm pretty excited to continue work in this vein and see if we can scale our learning approaches to truly large-scale, and currently intractable robotic domains like AI Habitat and BEHAVIOR!
0
0
1
0
0
1
Really cool work, with some of the best paper visuals I’ve ever seen!!!
Excited to present our work on features fields for robotic manipulation at
#CoRL2023
🤖! Honored to be selected as a finalist for the Best Paper/Student Paper award.
Check out our oral talk on Thursday morning (Oral 6) and poster in the afternoon (Poster 5, 2:45-3:30pm)
0
7
48
0
0
1
@tomssilver
@MITEECS
@MIT_LISLab
@MIT_CSAIL
Wouldn't have happened without you ❤️. I can't wait to get started and collaborate on some cool project ideas!
0
0
1
@kevin_zakka
@MITEECS
@MIT_LISLab
@MIT_CSAIL
Thanks Kevin! Looking forward to seeing some of your cool work at Berkeley and potentially collaborating in the future!
1
0
1
Had the cool opportunity to be interviewed by the interSTEM YouTube channel! If you're curious about some of my views on UGrad research and Robotics+AI, check it out!
0
0
1
@bindureddy
Congrats on launching! The service looks rather cool - I'm excited to see what happens!
0
0
1
@MIT_LISLab
@tomssilver
This is awesome! A detailed yet crisp explanation of 2 very cool papers!
0
0
1
@avikpal1410
@MITEECS
@MIT_LISLab
@MIT_CSAIL
Same to you! Can't wait to get to hangout and potentially collaborate + take some classes together :)
0
0
1
@MadelineGris
It’s surprisingly good at generating matolitlib code to create nice graphs/visualizations, and also for generating ideas/clipart (via dall-e) for explanatory figures for research papers :)
0
0
1