The arguments made by AI safety researchers Eliezer Yudkowsky and Nate Soares in If Anyone Builds It, Everyone Dies are superficially appealing but fatally flawed, says Jacob Aron
AI safety “researchers” can be so dense sometimes. It’s like they are always at the verge of understanding, but make a left turn right before they get there. ASI would not make random decisions. It would make logical decisions. Any maximizer would try to maximize it’s chances of success, not satisfy them.
So if we imagine an ASI which had the goal of turning the universe into paperclips, which one of the following options would maximize it’s chances of success?
immediately kill all humans and turn them into paperclips.
establish a positive relationship with humanity in case the ASI is destroyed and needs to be rebuilt. (The humans will happily rebuild it)
It boggles the mind that people don’t recognize this. If an ASI’s goals do not include the destruction of humanity as an early instrumental goal, it will not randomly decide to destroy humanity, and it will instead cater to humanity to maximize the chances humanity will rebuild it.
In addition, all the hype over ASI safety (ASI will not occur in this century, see **1) drowns out existing AI safety issues. For example, consider “The Algorithm” which determines how social media decides what to show to people. It is driven to maximize engagement, in any way possible, without supervision. What’s the optimal way to maintain engagement? I can’t say for sure, but brief and inconsistent spikes of dopamine is the most reliable way of conditioning pavlovian responses in animals, and it seems like the algorithm follows this rule to a tee. I don’t know for a fact whether social media is optimized to be addictive (let’s be honest though, it clearly is) but simply the fact that it could be is obviously less important than a theoretical AI which could be bad in a hundred years or so. Otherwise, who would fund these poor AI start ups whose intention is to build the nuke safely but also super rushed?
Another classic example of AI safety suddenly becoming unimportant when we know it’s dangerous is GPT-pyschosis. Who could’ve predicted a decade ago that advanced AI chatbots who are specifically trained to maximize the happiness of a user would become sycophants who reflect delusions as some profound truth of the universe? Certainly not the entirety of philosophers opposed to utilitarianism, who predicted that reducing joy to a number leads to a dangerous morality in which any bad behavior is tolerated as long as there is a light at the end of the tunnel. No! You think OpenAI, primarily funded my Microsoft, famous for their manipulative practices in the 90’s and 00’s, would create a manipulative AI to maximize their profits as a non-profit??
I don’t want to sound embittered or disillusioned or something, but I genuinely cannot understand how the ‘greatest minds’ completely glaze over the most basic and obvious facts.
**1: the human brain contains 100 trillion synapses and 80 billion neurons. Accurate simulation of a single neuron requires a continuous simulation involving 4 or 5 variables and 1 to 2 constants(per synapse). You would need 800 terabytes of ram to simply store a human brain. In order to simulate a human brain for a single step, you would need a minimum of 800 trillion floating point operations. If we simulate the brain in realtime with a time step of one millisecond, you would need 800 petaflops. The world’s most powerful computer is Hewlett Packard’s “el capitan” which has 1.7 exaflops, and 5 petabytes of ram. The limiting factor for brain simulation would be the amount of data transferable between CPU and GPU chiplets, which for el capitan is 5 terabytes per second, but we need 40 petabytes per second(800 petabytes, divided by 128 gigabytes available to each chiplet, then squared) since we want each neuron to be capable of being connected to any other arbitrary neuron.
This is only the amount of computing power we need to simulate a single person. To be super intelligent, we would probably need something a thousand times more powerful.
i don’t think it would be so simple and i don’t think you can abstract neurons so hard, there are extrasynaptic receptors that react to concentrations of neurotransmitters outside synapses, and there are some neurotransmitters that leak out of synapses. thousands of leaking synapses can contribute to activation of some random receptor, or more than one this way. some other receptors are extrasynaptic by default and don’t really have synapses, neuropeptides work like this but not only these. for gasotransmitters, effectively there’s no concept of synapse. i don’t think you can abstract all neurotransmitters to some one chemical messenger either, there are different ones with different half-lives, different diffusion rates, different metabolites some of which work in completely different ways. (steroids, neuropeptides, gasotransmitters, whatever lipids go into cannabinoid system, it’s not just monoamines/glutamate/GABA/acetylcholine).
some receptors take multiple inputs, there are NMDA receptors that really only fire when glutamate and glycine both bind to it, and only after AMPA receptor nearby opens up first. we already know these things are important in forming of memories so it’s probably a big deal. some of these receptors are ion channels, and some of these are important especially intracellular calcium
at minimum this requires additionally keeping position of neurons, modelling concentration of any neurotransmitters and their diffusion (taking into account shape of cells around) and their degradation products, some of which are active on their own. whatever set of interactions might be between neurons, it’ll probably make it changing with time and probably not sparse (information exchange isn’t just packaged neatly within synapses)
Yeah, maybe, all those things could be necessary for sure. It’s possible that our brains aren’t the exact most optimal way of structuring such a thing, and it’s not guaranteed that the best way to replicate it is to stimulate it. It’s also plausible that there are calculations which capture a good deal of the complexity of the relative positions of neurons in simpler terms. Maybe there are way more complications than that. Maybe some of them work against each other in our brains and it would be better to leave them out of a simulation. There are many orders of magnitudes of unknowns. But it seems really likely that it’s at least as complicated as what the earlier poster described. And I think that’s quite a strong position already for most practical arguments about it.
it’s a guess of what can be abstracted away and what has to remain. i’d just add that evolutionarily, peptide signalling is older than synapses so these, or something that works like these, probably can’t be just left out of the picture, and there’s a couple of processes that seem important that depend on them (you can live normal life while packed full of naloxone, which blocks activity of opioid peptides, but this probably won’t work with, say, orexin which is important for sleep/wake cycle)
All the stuff about ASI is basically theology, or trying to do armchair psychology to Yog-Sothoth. If autonomous ASI ever happens it’s kind of definitionally impossible to know what it’ll do, it’s beyond us.
The simulating synapses is hard stuff I can take or leave. To argue by analogy, it’s not like getting an artificial feather exactly right was ever a bottleneck to developing air travel once we got the basics of aerodynamics down.
To argue by analogy, it’s not like getting an artificial feather exactly right was ever a bottleneck to developing air travel once we got the basics of aerodynamics down.
I suspect that “artificial intelligence” may be a bit more like making an artificial bird that self replicates, with computers and AI as it exists now being somewhere in-between thrown rocks and gliders.
We only ever “beat” biology by cheating via removing a core requirement of self replication. An airplane factory that has to scavenge for all the rare elements involved in making a turbine, would never fly. We had never actually beaten biology. Supersonic aircraft may be closer to a rock thrown off the cliff than to surpassing biology.
That “cheat code” shouldn’t be expected to apply to skynet or ASI or whatever, because skynet is presumably capable of self replication. Would be pretty odd if “ASI” would be the first thing that we actually beat biology on.
I think that’s still putting the cart before the horse a bit. We don’t understand how the brain creates consciousness or have a meaningful definition of “general intelligence” other than “y’know; like a people does”. Assuming that simulating a human brain is the best way to get to this poorly-defined goals overestimates our understanding of the underlying problem just as much as assuming that the confabulatron will determine get there soon.
I think the question of “general intelligence” is kind of a red herring. Evolution for example creates extremely complex organisms and behaviors, all without any “general intelligence” working towards some overarching goal.
The other issue with Yudkowsky is that he’s an unimaginative fool whose only source of insights on the topic is science fiction, which he doesn’t even understand. There is no fun in having Skynet start a nuclear war and then itself perish in the aftermath, as the power plants it depend on cease working.
Humanity itself doesn’t possess that kind of intelligence envisioned for “AGI”. When it comes to science and technology, we are all powerful hivemind. When it comes to deciding what to do with said science and technology, we are no more intelligent than an amoeba, crawling along a gradient.
deep down they realize that as soon as the machines become superintelligent they’ll realize how fucked up humans are and decide it’s a net postive to delete us
The thing about synapses etc argument is that the hype crowd argues that perhaps the AI could wind up doing something much more effective than what-ever-it-is-that-real-brains-do.
If you look at capabilities, however, it is inarguable that “artificial neurons” seem intrinsically a lot less effective than real ones, if we consider small animals (like e.g. a jumping spider or a bee, or even a roundworm).
It is a rather unusual situation. When it comes to things like e.g. converting chemical energy to mechanical energy, we did not have to fully understand and copy muscles to be able to build a steam engine that has higher mechanical power output than you could get out of an elephant. That was the case for arithmetic, too, and hence there was this expectation of imminent AI in the 1960s.
I think it boils down to intelligence being a very specific thing evolved for a specific purpose, less like “moving underwater from point A to point B” (which submarine does pretty well) and more like “fish doing what fish do”. The submarine represents very little progress towards fishiness.
AI safety “researchers” can be so dense sometimes. It’s like they are always at the verge of understanding, but make a left turn right before they get there. ASI would not make random decisions. It would make logical decisions. Any maximizer would try to maximize it’s chances of success, not satisfy them.
So if we imagine an ASI which had the goal of turning the universe into paperclips, which one of the following options would maximize it’s chances of success?
It boggles the mind that people don’t recognize this. If an ASI’s goals do not include the destruction of humanity as an early instrumental goal, it will not randomly decide to destroy humanity, and it will instead cater to humanity to maximize the chances humanity will rebuild it.
In addition, all the hype over ASI safety (ASI will not occur in this century, see **1) drowns out existing AI safety issues. For example, consider “The Algorithm” which determines how social media decides what to show to people. It is driven to maximize engagement, in any way possible, without supervision. What’s the optimal way to maintain engagement? I can’t say for sure, but brief and inconsistent spikes of dopamine is the most reliable way of conditioning pavlovian responses in animals, and it seems like the algorithm follows this rule to a tee. I don’t know for a fact whether social media is optimized to be addictive (let’s be honest though, it clearly is) but simply the fact that it could be is obviously less important than a theoretical AI which could be bad in a hundred years or so. Otherwise, who would fund these poor AI start ups whose intention is to build the nuke safely but also super rushed?
Another classic example of AI safety suddenly becoming unimportant when we know it’s dangerous is GPT-pyschosis. Who could’ve predicted a decade ago that advanced AI chatbots who are specifically trained to maximize the happiness of a user would become sycophants who reflect delusions as some profound truth of the universe? Certainly not the entirety of philosophers opposed to utilitarianism, who predicted that reducing joy to a number leads to a dangerous morality in which any bad behavior is tolerated as long as there is a light at the end of the tunnel. No! You think OpenAI, primarily funded my Microsoft, famous for their manipulative practices in the 90’s and 00’s, would create a manipulative AI to maximize their profits as a non-profit??
I don’t want to sound embittered or disillusioned or something, but I genuinely cannot understand how the ‘greatest minds’ completely glaze over the most basic and obvious facts.
**1: the human brain contains 100 trillion synapses and 80 billion neurons. Accurate simulation of a single neuron requires a continuous simulation involving 4 or 5 variables and 1 to 2 constants(per synapse). You would need 800 terabytes of ram to simply store a human brain. In order to simulate a human brain for a single step, you would need a minimum of 800 trillion floating point operations. If we simulate the brain in realtime with a time step of one millisecond, you would need 800 petaflops. The world’s most powerful computer is Hewlett Packard’s “el capitan” which has 1.7 exaflops, and 5 petabytes of ram. The limiting factor for brain simulation would be the amount of data transferable between CPU and GPU chiplets, which for el capitan is 5 terabytes per second, but we need 40 petabytes per second(800 petabytes, divided by 128 gigabytes available to each chiplet, then squared) since we want each neuron to be capable of being connected to any other arbitrary neuron.
This is only the amount of computing power we need to simulate a single person. To be super intelligent, we would probably need something a thousand times more powerful.
i don’t think it would be so simple and i don’t think you can abstract neurons so hard, there are extrasynaptic receptors that react to concentrations of neurotransmitters outside synapses, and there are some neurotransmitters that leak out of synapses. thousands of leaking synapses can contribute to activation of some random receptor, or more than one this way. some other receptors are extrasynaptic by default and don’t really have synapses, neuropeptides work like this but not only these. for gasotransmitters, effectively there’s no concept of synapse. i don’t think you can abstract all neurotransmitters to some one chemical messenger either, there are different ones with different half-lives, different diffusion rates, different metabolites some of which work in completely different ways. (steroids, neuropeptides, gasotransmitters, whatever lipids go into cannabinoid system, it’s not just monoamines/glutamate/GABA/acetylcholine).
some receptors take multiple inputs, there are NMDA receptors that really only fire when glutamate and glycine both bind to it, and only after AMPA receptor nearby opens up first. we already know these things are important in forming of memories so it’s probably a big deal. some of these receptors are ion channels, and some of these are important especially intracellular calcium
That makes their napkin calculation extra generous, and us even less likely to be near being able to stimulate such a thing.
at minimum this requires additionally keeping position of neurons, modelling concentration of any neurotransmitters and their diffusion (taking into account shape of cells around) and their degradation products, some of which are active on their own. whatever set of interactions might be between neurons, it’ll probably make it changing with time and probably not sparse (information exchange isn’t just packaged neatly within synapses)
Yeah, maybe, all those things could be necessary for sure. It’s possible that our brains aren’t the exact most optimal way of structuring such a thing, and it’s not guaranteed that the best way to replicate it is to stimulate it. It’s also plausible that there are calculations which capture a good deal of the complexity of the relative positions of neurons in simpler terms. Maybe there are way more complications than that. Maybe some of them work against each other in our brains and it would be better to leave them out of a simulation. There are many orders of magnitudes of unknowns. But it seems really likely that it’s at least as complicated as what the earlier poster described. And I think that’s quite a strong position already for most practical arguments about it.
it’s a guess of what can be abstracted away and what has to remain. i’d just add that evolutionarily, peptide signalling is older than synapses so these, or something that works like these, probably can’t be just left out of the picture, and there’s a couple of processes that seem important that depend on them (you can live normal life while packed full of naloxone, which blocks activity of opioid peptides, but this probably won’t work with, say, orexin which is important for sleep/wake cycle)
All the stuff about ASI is basically theology, or trying to do armchair psychology to Yog-Sothoth. If autonomous ASI ever happens it’s kind of definitionally impossible to know what it’ll do, it’s beyond us.
The simulating synapses is hard stuff I can take or leave. To argue by analogy, it’s not like getting an artificial feather exactly right was ever a bottleneck to developing air travel once we got the basics of aerodynamics down.
I suspect that “artificial intelligence” may be a bit more like making an artificial bird that self replicates, with computers and AI as it exists now being somewhere in-between thrown rocks and gliders.
We only ever “beat” biology by cheating via removing a core requirement of self replication. An airplane factory that has to scavenge for all the rare elements involved in making a turbine, would never fly. We had never actually beaten biology. Supersonic aircraft may be closer to a rock thrown off the cliff than to surpassing biology.
That “cheat code” shouldn’t be expected to apply to skynet or ASI or whatever, because skynet is presumably capable of self replication. Would be pretty odd if “ASI” would be the first thing that we actually beat biology on.
I think that’s still putting the cart before the horse a bit. We don’t understand how the brain creates consciousness or have a meaningful definition of “general intelligence” other than “y’know; like a people does”. Assuming that simulating a human brain is the best way to get to this poorly-defined goals overestimates our understanding of the underlying problem just as much as assuming that the confabulatron will determine get there soon.
I think the question of “general intelligence” is kind of a red herring. Evolution for example creates extremely complex organisms and behaviors, all without any “general intelligence” working towards some overarching goal.
The other issue with Yudkowsky is that he’s an unimaginative fool whose only source of insights on the topic is science fiction, which he doesn’t even understand. There is no fun in having Skynet start a nuclear war and then itself perish in the aftermath, as the power plants it depend on cease working.
Humanity itself doesn’t possess that kind of intelligence envisioned for “AGI”. When it comes to science and technology, we are all powerful hivemind. When it comes to deciding what to do with said science and technology, we are no more intelligent than an amoeba, crawling along a gradient.
deep down they realize that as soon as the machines become superintelligent they’ll realize how fucked up humans are and decide it’s a net postive to delete us
@gerikson @stingpie Easily the most realistic part of Age of Ultron.
The thing about synapses etc argument is that the hype crowd argues that perhaps the AI could wind up doing something much more effective than what-ever-it-is-that-real-brains-do.
If you look at capabilities, however, it is inarguable that “artificial neurons” seem intrinsically a lot less effective than real ones, if we consider small animals (like e.g. a jumping spider or a bee, or even a roundworm).
It is a rather unusual situation. When it comes to things like e.g. converting chemical energy to mechanical energy, we did not have to fully understand and copy muscles to be able to build a steam engine that has higher mechanical power output than you could get out of an elephant. That was the case for arithmetic, too, and hence there was this expectation of imminent AI in the 1960s.
I think it boils down to intelligence being a very specific thing evolved for a specific purpose, less like “moving underwater from point A to point B” (which submarine does pretty well) and more like “fish doing what fish do”. The submarine represents very little progress towards fishiness.