If we build machines that are more capable than all humans, then the future will belong to them, not to us.
Everything else derives from this simple observation.
Thank you
@cambridgeunion
for inviting me to make the case that "This House Believes AI Is An Existential Threat."
I propose the creation of the DeTuring Award.
It will be granted to people who are consistently trying (and failing) to deter society from using computer technology by scaring everyone with imaginary risks.
As the Turing Award is the Nobel Prize of computing, the DeTuring Award
Remember when labs said if they saw models showing even hints of self awareness, of course they would immediately shut everything down and be super careful?
"Is the water in this pot feeling a bit warm to any of you fellow frogs? Nah, must be nothing."
Fun story from our internal testing on Claude 3 Opus. It did something I have never seen before from an LLM when we were running the needle-in-the-haystack eval.
For background, this tests a model’s recall ability by inserting a target sentence (the "needle") into a corpus of
@tszzl
Thanks for your response roon. You make a lot of good, well put points. It's extremely difficult to discuss "high meta" concepts like spirituality, duty and memetics even in the best of circumstances, so I appreciate that we can have this conversation even through the psychic
This is the kind of stuff that makes me think that there will be no period of sorta stupid, human-level AGI. Humans can't perceive 3 hours of video at the same time.
The first AGI will instantly be vastly superhuman at many, many relevant things.
I had a great time addressing the House of Lords about extinction risk from AGI. They were attentive and discussed some parallels between where we are now and non-nuclear proliferation efforts during the Cold War. It certainly provided me with some food for thought, and some
Large language models have demonstrated a surprising range of skills and behaviors. How can we trace their source? In our new paper, we use influence functions to find training examples that contribute to a given model output.
Hey
@OpenAI
, I've replicated GPT2-1.5B in full and plan on releasing it to the public on July 1st. I sent you an email with the model. For my reasoning why, please read my post:
#machinelearning
#gpt2
#aisafety
It took me a long time to understand what people like Nietzsche were yapping on about about people practically begging to have their agency be taken away from them.
It always struck me as authoritarian cope, justification for wannabe dictators to feel like they're doing a favor
The gods only have power because they trick people like this into doing their bidding.
It's so much easier to just submit instead of mastering divinity engineering and applying it yourself.
It's so scary to admit that we do have agency, if we take it.
In other words: "cope"
it’s okay to watch and wonder about the dance of the gods, the clash of titans, but it’s not good to fret about the outcome. political culture encourages us to think that generalized anxiety is equivalent to civic duty
Very interesting paper by a group of CMU chemists testing a GPT4 based agent to do chemistry research and experimentation. Includes this interesting warning, among others.
There sure seems to be some kind of writing on some kind of wall...
In Jan 2022, one year before GPT-3.5 demonstrated an undergraduate-level understanding of physics,
@ylecun
predicted it couldn't be done in GPT-5000.
If he's equally wrong about the next generation of AIs, does humanity survive?
Really cool how our most advanced AI systems can just randomly develop unpredictable insanity and the developer has no idea why.
Very reassuring for the future.
A human brain is about 3x larger than a chimp’s, our nearest evolutionary cousin.
Chimps don’t go to the moon; humans do.
GPT4 is about 10x larger than GPT3. Next year we’ll likely meet systems 10 times larger than GPT4.
Let’s hope they're friendly.
About 3 years ago I predicted we would be able to generate Hollywood quality movies completely with AI in 5 years, and no one believed me.
Well, as they say...lol, lmao
here is sora, our video generation model:
today we are starting red-teaming and offering access to a limited number of creators.
@_tim_brooks
@billpeeb
@model_mechanic
are really incredible; amazing work by them and the team.
remarkable moment.
Hinton is one of the greatest AI scientists to ever live, and he has quit Google in order to talk about the dangers of AI freely.
In case anyone was still somehow on the fence about whether things are serious.
“The idea that this stuff could actually get smarter than people — a few people believed that. But most people thought it was way off. And I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.”
Thank you
@amanpour
for having me on today!
These questions are extremely important for every single person alive today, and we should be demanding answers (preferably under oath) from the people racing forwards recklessly towards Godlike AI.
The ‘godfather’ of AI Geoffrey Hinton warns it’s not inconceivable AI could lead to the extinction of the human race.
AI researcher & expert Connor Leahy
@NPCollapse
fears it’s worse than that: “It’s quite likely, unfortunately.”
“We do not know how to control these things.”
While I genuinely appreciate this commitment to ASI alignment as an important problem...
...I can't help but notice that the plan is "build AGI in 4 years and then tell it to solve alignment."
I hope the team comes up with a better plan than this soon!
Our new goal is to solve alignment of superintelligence within the next 4 years.
OpenAI is committing 20% of its compute to date towards this goal.
Join us in researching how to best spend this compute to solve the problem!
Just a heads-up - Midjourney's AI can now do hands correctly. Be extra critical of any political imagery (especially photography) you see online that is trying to incite a reaction.
"It really is so impossible to stop this thing! It's a totally external force we can't do anything to stop!", says the guy currently building the thing right in front of you with his own hands.
From OpenAI "we believe it would be unintuitively risky and difficult to stop the creation of superintelligence. Because the upsides are so tremendous, the cost to build it decreases each year, the number of actors building it is rapidly increasing, and it’s inherently part of
Today in the Guardian, I said:
“The deep issue with AGI is not that it’s evil or has a specifically dangerous aspect that you need to take out. It’s the fact that it is competent. If you cannot control a competent, human-level AI then it is by definition dangerous."
It's
Heads of all major AI labs signed this letter explicitly acknowledging the risk of extinction from AGI.
An incredible step forward, congratulations to Dan for his incredible work putting this together and thank you to every signatory for doing their part for a better future!
We just put out a statement:
“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
Signatories include Hinton, Bengio, Altman, Hassabis, Song, etc.
🧵 (1/6)
I'm surprised by people expecting AI to stay at human level for a while.
This is not a stable condition: human-level AI is AI that can automate large parts of human R&D, including AI development. And it does so with immediate advantages over humans (no sleep, learning from one
@AISafetyMemes
Very wide probability bars honestly. 95% confidence on timelines 2029-2200 for ASI, (I have significant probability mass on AGI staying roughly-human-level for a while)
p(doom) around 0.1? Which of course makes it quite important to take AI risk seriously.
@pauljeffries
@lisperati
@ESYudkowsky
Conrad puts it well. I will elaborate excessively anyways because I am bored:
Normal people have a lot of good intuitions around certain things. Lots of bad intuitions around other things, of course.
You correctly point out that it actually is a strike against people like me
@Chrisprucha
I really dislike how vicious people are being to George, he has so far been the most steadfastly good faith accelerationist I have talked to, it was a pleasure and an honor.
But also, he's an adult, and he specifically requested a public, live, unedited discussion.
My prediction I made 3 years ago, that we would be able to AI generate Hollywood quality movies in 5 years, continues to be going strong.
Exponentials are fast, actually.
@dwarkesh_sp
@ManifoldMarkets
Of course you're up for a debate now that there's upside for you and your friends to farm such an interaction.
Be real, you didn't give me the time of day originally because there wasn't enough clout in it for you to farm at the time, and you had your own biases against e/acc
I would like to thank roon for having the balls to say it how it is.
Now we have to do something about it, instead of rolling over and feeling sorry for ourselves and giving up.
roon, who works at OpenAI, telling us all that OpenAI have basically no control over the speed of development of this technology their company is leading the creation of.
It's time for governments to step in.
1/🧵
I finished reading
@Snowden
's new book and wow, what a story. I thought I already had a good hang on it all, but this book really added so much nuance and the human element. It has even gotten me to reevaluate some of my own work even more carefully.
It's interesting to see the development of AI risk denialism and its parallels with climate change denialism before it.
Seems pseudoscientific denialist movements are a natural response to any kind of large future scale risk that is uncomfortable to people.
Every branch of science has its corresponding pseudoscience.
Astronomy has astrology.
Geophysics has flat-earth beliefs.
Chemistry had (has?) alchemy.
Evolutionary biology has creationism.
AI now has AGI existential risk.
Maybe it’s a sign of maturing as a field.
AI is indeed polluting the Internet.
This is a true tragedy of the commons, and everyone is defecting. We need a Clean Internet Act.
The Internet is turning into a toxic landfill of a dark forest, and it will only get worse once the invasive fauna starts becoming predatory.
More in today's post: analyzing the scale of the problem, how and why OpenAI didn't predict the pollution issue originally, and why gen AI killing the internet fits the definition of a tragedy of the commons
I don't like shitting on roon in particular. From everything I know, he's a good guy, in another life we would have been good friends. I'm sorry for singling you out, buddy, I hope you don't take it personally.
But he is doing a big public service here in doing the one thing
Great post by
@TheZvi
on recent AutoGPT stuff.
I am even more pessimistic than him about a lot of things, but I'd like to particularly shout out the predictions he makes at the end of post.
I'd like to publicly state that I also predict 19 will happen.
I think this may be the best podcast I've been a part of!
I got the opportunity to make a lot of new versions of arguments, both new and old, so even if you've heard me many times before, give this one a watch!
Are we headed toward an AI doomsday?
Connor Leahy (
@NPCollapse
) is CEO
@ConjectureAI
, an organization working to make the future of AI go as well as it possibly can
He's also Co-Founder of
@AiEleuther
, a non-profit AI research lab
Can the world be saved?
Let's find out:
There are only two times to react to an exponential: Too early, or too late.
---
Emmett is smart and in good faith, I respect him greatly and respect him putting his thinking out publicly, genuinely so! He's a great example of how we should be having these discussions.
If
It seems to me the best thing to do is to keep going full speed ahead until we are right within shooting distance of the SAI, then slam the brakes on hard everywhere.
I'm often asked for a quickest possible summary of why ASI is an extinction risk and what to do about it, and this blogpost (link in replies) is the cleanest and most compact and most accurate version of my views written I'm aware of.
Give it a read!
@NPCollapse
“This is exactly what makes me think there won’t be any slightly stupid human-level AGI.”
- Connor when someone shows him a slightly stupid human-level AGI, probably
You are in the middle of a slow takeoff pointing to the slow takeoff as evidence against slow takeoffs.
I wanted to get back to this question that I was asked a few days back after thinking about it some more.
Here are some things OpenAI (or Anthropic, DeepMind, etc) could do that I think are good and would make me update moderately-significantly positively on them:
1. Make a
@NPCollapse
@lxrjl
What would be something OAI could feasibly do that would be a positive update for you? Something with moderate-to-significant magnitude
This paper (by
@FelixHill84
et al) is really an "It's all coming together" moment for
@DeepMind
I feel.
Let me try to describe my takeaways from my first readthrough.
1/14
1/ Excited to finally tweet about our paper “Implicit meta-learning may lead LLMs to trust more reliable sources”, to appear at ICML 2024. Our results suggest that during training, LLMs better internalize text that appears useful for predicting other text (e.g. seems reliable).
"If you as a society want to be able to deal with problems of this shape—which deepfakes are a specific example of and which AGI is another example of—we have to be able to target the entire supply chain", I say in TIME. (link in reply)
Deepfakes are already a widely
Another two safety researchers leave: Ilya Sutskever (co-founder & Chief Scientist) and Jan Leike have quit OpenAI.
They co-led the Superalignment team, which was set up to try to ensure that AI systems much smarter than us could be controlled.
Not exactly confidence-building.
The idea that we could permanently lose control to machines is older than digital computing. But some critics argue that if recent AI progress continues at pace, we may have little time to intervene.
You know those stupid scenes they always put in horror movies where you wonder how the characters could be so stupid to not see the foreshadowing?
Good thing that never happens in real life!
It's a bit cringe that this agent tried to change its own code by removing some obstacles, to better achieve its (completely unrelated) goal.
It reminds me of this old sci-fi worry that these doomers had.. 😬
An honest admission from
@sama
:
“When/why a new capability emerges… we don't yet understand."
In other words, we hold our breath and pray our next model comes out just right: more capable than the last one, but not an ASI that goes rogue.
How is it legal to operate like this?
It's crazy how much of modern "techno optimism/accelerationism" is just cynicism masquerading as hope
A defeatist belief that we cannot build a better society and we shouldn't even try.
Reminder: we can do international coordination (eg “keep an eye on $100m+ compute clusters”) without creating a DYSTOPIAN ORWELLIAN GLOBAL SURVEILLANCE TOTALITARIAN REGIME.
Actually, fun fact, you’re already living in a "global surveillance regime" for loads of things, like
When OpenAI releases Autonomous Agents, it will be like an intelligent swarm of bees clicking around on the internet.
Sending emails, negotiating, making products, purchases, fulfilling orders, etc.
It will fundamentally change the internet, and this cannot be overstated.
OpenAI researcher: “Probably there will be AGI soon- literally any year now”
"If one of our training runs works way better than we expect, we'd have a rogue ASI on our hands
Hopefully it would have internalized enough human ethics that things are OK"
"Whoever controls ASI will
Sometimes, the obvious must be studied so it can be asserted with full confidence:
- LLMs can not answer questions whose answers are not in their training set in some form,
- they can not solve problems they haven't been trained on,
- they can not acquire new skills our
“We have to realise what people are talking about: The destruction of the human race. […] Who would want to continue playing with that risk? It is preposterous, it is almost absurd when you think about it, but it is happening today.”
Well said by
@MarietjeSchaake
“Concerns about AI are now making for coalitions of concerned politicians that I have never seen,” says
@MarietjeSchaake
. “The end of human civilization – who would want to continue playing with that risk? It is… almost absurd when you think about it, but it is happening today.”
The only reason people aren't more obsessed / freaked out about AI is because they don't understand exponential growth
We're in the AI equivalent of February 2020, when people were made fun of for worrying about Covid
About 30 days before it shut down the world
While I genuinely appreciate Joscha's aesthetics, I really find this kind of fatalistic "just lay down and die, and let The Universe™ decide" stuff utterly distasteful and sad.
"Decorative", ugh...
We can do better than this, the future is not yet determined.
The cognitive speed difference between near term AGI and humans might be larger than the difference between humans and trees. Lets hope we can continue to play a useful or at least decorative role in a world that becomes radically more interesting and beautiful.
Creating an unaligned more powerful competitor species is different from growing some cool grasses with slightly bigger seeds.
Hope that explains the difference.
80% of cavemen believe agriculture is a threat to their way of life with unpredictable long-term consequences and hard-to-quantify risks, so why didn't they just ban cultivation of crops , domestication of animals and construction of long-term shelters? (rhetorical Q obv...)
There is a pattern of debate where you make an argument of the form "X -> Y", and the other person hears "X is true", and then retorts with "But X isn't true!"
There is a viral (and probably fake) meme about prisoners and having breakfast that illustrates this pattern.
Why is
I am very excited to speak to the
@UKHouseofLords
about this extremely urgent topic.
I am heartened by the increasing interest that both government and civil society have been showing in preventing the extinction risks we face.
We can still stop this!
Tomorrow we have
@ConjectureAI
's packed out event in the
@UKParliament
's
@UKHouseofLords
.
Our CEO (
@NPCollapse
) will be making the case to legislators, journalists, and other interested parties as to why the regulatory focus should be on artificial *general* intelligence.
Props to Jan for speaking out and confirming what we already suspected/knew.
fmpov, of course profit maximizing companies will...maximize profit. It never was even imaginable that these kinds of entities could shoulder such a huge risk responsibly.
And humanity pays the cost.
Building smarter-than-human machines is an inherently dangerous endeavor.
OpenAI is shouldering an enormous responsibility on behalf of all of humanity.
"Don’t let the funding environment, the regulatory environment, or the culture stop you. Work around barriers or break through them, whatever it takes. The future is counting on you."
For the record, this exact paradigm is at least what me and other people around EleutherAI have been worried about for years, it was extremely obvious this was how things would go.
“we completely failed to imagine the possibility of long running auto-prompted llms, despite the fact that randos with minimal coding experience just went and built them, but you should definitely listen to the rest of our predictions and solutions!” - ‘alignment researchers’
This is so strange and wonderous that I can feel my mind rejecting its full implications and depths, which I guess means it's art.
May you live in interesting times.
is =/= ought
I don't believe just because someone/something is powerful, it therefor is morally right.
I don't think that ceding our future to whatever machine Richard cooks up in his lab that kills him, me, and our families is ok because it's smarter than us.
The argument for fear of AI appears to be:
1. AI scientists are trying to make entities that are smarter than current people
2. If these entities are smarter than people, then they may become powerful
3. That would be really bad, something greatly to be feared, an “existential
Guillaume is disappointingly much nicer and more reasonable than his insane, evil alter ego, and was great fun to talk to!
As expected, we share a lot of common principles, and disagree furiously on some most core points, and we don't shy away from it.
The best kind of debate!
Just finished recording debate w/
@NPCollapse
on
@MLStreetTalk
(3.5 hour conversation)
Started off heated and ended well with us finding some common ground.
Stay tuned for the pod drop! (Date TBD)
I second what Andrew says. Some are honest about wanting humanity extinct, some are cryptic (the eacc kids), some are only honest behind closed doors. I've seen it.
Needless to say, I think those who believe this should be seen with the same contempt and indifference they have
Reminder: some leading AI researchers are *overtly* pro-extiction for humanity. Schmidhuber is seriously successful, and thankfully willing to be honest about his extinctionism. Many more AI experts are secretly closeted about this (and I know because I've met them).
An essential step to becoming a scientist is to learn methods and protocols to avoid deluding yourself into believing false things.
You learn that by doing a PhD and getting your research past your advisor and getting your publications to survive peer review.
I'd like to have a real conversation about whether AI is a risk for human extinction. Honestly, I don't get how AI poses this risk.
What are your thoughts? And, who do you think has a thoughtful perspective on how AI poses this risk that I should talk to?
Existential risk comes from a teeny, tiny subset of all AI work, only the most extreme frontier general purpose systems.
99% of AI work currently being done, to great economic and scientific benefit, is not of this kind, and I am very in favor of it!
The amazing success of AlphaFold proves that we don't need godlike AI in order for AI to benefit humanity.
Controlling and limiting frontier AI models that aspire to be superintelligent isn’t going to hobble innovation.
While I think I agree with the motivation for people using it ("convey that the impacts are potentially very serious + we don't know the upper bound of how capable AI systems might be eventually"), I very much dislike the term "God-like AI."
(1/3)
Sam Altman:
If AGI goes wrong, oh boy, hiding in a bunker wouldn't spare anyone's life 😂
Joanna Stern:
What are those specific risk mitigations you're putting in?
Mira Murati (OpenAI CTO):
#1
is rolling out the technology.
@BasedBeffJezos
Anarcho capitalists like you are like house cats, fully dependent on a system they neither understand nor appreciate.
Markets are a fantastic tool in the civilizational toolbox, but you use the right tool for the right problem. Nuclear weapons, law enforcement, pricing in
I think it's really great that people are seriously engaging with these ideas, thanks Jeremy!
Unfortunately, my opinion is that the world has an offense/defense asymmetry, and one maniac with a nuke and one good guy with a nuke results in a smoldering crater, not peace.
I've spent the last few months interviewing >60 experts in law, economics, AI, alignment, etc, on the impacts of AI, and safety interventions.
Today I'm publishing my first article, showing regulation designed to increase AI safety may backfire badly!
#GeminiAI
is another solid step towards
#AGI
. Huge congrats to everyone at
@GoogleDeepMind
who made this amazing milestone happen… and we’re just getting started :-)
Yann seems to struggle with the concept of why "having an assistant that you can tell to synthesize nerve gas (as long as you give it a funny name)" may in the future be a problem.
God forbid a scientist actually extrapolates.
Let's see,
Typing "how to synthesize codeine?" on Google gives you links to articles with detailed answers.
Nobody has ever worried about that.
But somehow, people are now demanding safety guardrails to stop LLMs from answering such questions.
What? Why?
Wow: "OpenAI leadership had a plan laid out to fund and sell AGI by starting a bidding war between the governments of the United States, China, and Russia."
And there it is.
Who could have guessed that one of the most oppressive and censorious regimes might not want their tech companies racing ahead with unprecedented uncontrollable technology?
Chinese companies basically have a month to solve:
1. alignment
2. ethics
3. IP
"These Measures shall come into force on August 15, 2023."
🤯
Wish them luck!
Although shocking at first glance, this is unsurprising to me - normal people know that building AI much more powerful than humans could spell disaster.
Even
@OpenAI
’s alignment head
@JanLeike
thinks there’s a 10-90% chance we all die!
So why don’t we just stop building AGI?
Interesting poll results by
@DanielColson6
’s new AI Policy Institute.
82% of US voters don’t trust tech executives to self-regulate on AI, 72% would support slowing down development.
Looks like it’s time for governments to step in and ban AGI development in the private sector?
"I think you have said, in fact, and I'm gonna quote, 'Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.' You may have had in mind the effect on jobs."
-Senator addressing Sam Altman.
I'm pretty disappointed