Discussion re criticism of AI safety with Olle Häggström

I wanted to share a debate that I had outside of CF.

Olle Häggström is a mathematician, a professor, and an AI safety proponent.

Häggström has written some books on science. His latest book is Here Be Dragons: Science, Technology and the Future of Humanity (2016).

On his blog I asked him about some criticism that Elliot has written regarding AI safety.

He didn’t seem interested in really looking into stuff such as that if Popper is right then he and a lot of others are “… wasting their careers, misdirecting a lot of donations, incorrectly scaring people about existential dangers, etc.” as Elliot writes in AGI Alignment and Karl Popper.

In our discussion Häggström claimed that he had a rebuttal of BoI / DD. He would not post it or give a summary at first:

Deutsch’s point is so weak (basically, what happens is that he is so seduced by his own abstractions that he simply forgets about more down-to-earth issues) that my rebuttal not worth repeating.
I got his e-book and looked up what he wrote. I posted it in the discussion below (comment “2 januari 2023 kl. 19:07”).

Below is the entire discussion. I am “Anonym”.
A link to the discussion: Häggström hävdar: Scott Aaronson on AI safety


Anonym 26 december 2022 kl. 12:31
Do you moderate comments? I posted a message with some criticism re AI safety a few days ago to this post but it is not available/visible.

Olle Häggström 27 december 2022 kl. 17:11
I do moderate, but unfortunately I often do not check my inbox as often as would be ideal (in particular during holidays). Sorry about the delay.

Anonym 25 december 2022 kl. 09:34
What is your take on some substantial criticism of “AI safety” re freedom and that you seem to be fundamentally wrong on the power of AGI, such as this:

“… I do care about AGI welfare and think AGIs should have full rights, freedoms, citizenship, etc. […] I think it’s appalling that in the name of safety (maybe AGIs will want to turn us into paperclips for some reason, and will be able to kill us all due to being super-intelligent) many AGI researchers advocate working on “friendly AI” which is an attempt to design an AGI with built-in mind control so that, essentially, it’s our slave and is incapable of disagreeing with us. I also think these efforts are bound to fail on technical grounds – AGI researchers don’t understand BoI [“Beginning of Infinity” by David Deutsch] either, neither its implications for mind control (which is an attempt to take a universal system and limit it with no workarounds, which is basically a lost cause unless you’re willing to lose virtually all functionality) nor its implications for super intelligent AGIs (they’ll just be universal knowledge creators like us, and if you give one a CPU that is 1000x as powerful as a human brain then that’ll be very roughly as good as having 1000 people work on something which is the same compute power.). This, btw, speaks to the importance of some interdisciplinary knowledge. If they understood classical liberalism better, that would help them recognize slavery and refrain from advocating it.”
From: Curiosity – Discussing Animal Intelligence

Olle Häggström 27 december 2022 kl. 17:22
Thinking of an AI as having rights is an interesting perspective that merits consideration. I am skeptical, however, of the present argument, partly because it seems to prove too much: if the argument were correct, it would also show that it is morally wrong to raise and educate our human offspring in the general direction of sharing our values, as if the upbringing and education succeeds, the offspring would be slaves to our values.

As regards Deutsch’s critique in his Beginning of Infinity: I do like the book, but in the particular case of AI safety, Deutsch is just confused (as explained in Footnote 238, p 104 of my book Here Be Dragons).

Anonym 27 december 2022 kl. 19:02
I believe that it rather means that it is morally wrong to deny freedom of choice and thought to all forms of general intelligence (AGIs as well as human offspring).

Do you mind sharing your explanation of why Deutsch is wrong re AI safety or briefly summarized it here?

No problem about the delay.

Olle Häggström 28 december 2022 kl. 11:01
So tell me, are you for or against raising children to behave well? It strikes me as bold to claim that aligned AGIs are denied freedom of choice and thought whereas well-raised human children are not. That seems to require a satisfactory theory of free will, something that is conspiciously lacking in the philosophical literature.

Anonym 28 december 2022 kl. 14:42
I don’t follow your reasoning. Would aligned AGIs be able to think in any way possible to a human and do anything a human could do? I was under the impression they could not do this (hence the “aligned” part).

Would you mind sharing/briefly summarize your criticism of Deutsch re AI safety?

Olle Häggström 1 januari 2023 kl. 11:54
I did that in my book Here Be Dragons (Oxford University Press, 2016). Deutsch’s point is so weak (basically, what happens is that he is so seduced by his own abstractions that he simply forgets about more down-to-earth issues) that my rebuttal not worth repeating.

Olle Häggström 1 januari 2023 kl. 11:56
As regards “would aligned AGIs be able to think in any way possible to a human and do anything a human could do?”, the answer is yes, that’s what it means to be an AGI as opposed to just any old AI.

Anonym 2 januari 2023 kl. 19:07
Re Deusch:

Ok. I looked up your rebuttal in “Here Be Dragons”. Is the following your full rebuttal?

“Similar to Bringsjord’s argument is one of Deutsch (2011). After having spent most of his book defending and elaborating on his claim that, with the scientific revolution, humans attained the potential to accomplish anything allowed by the laws of physics - a kind of universality in our competence - he addresses briefly the possibility of AIs outsmarting us, and dismisses it:

‘Most advocates of the Singularity believe that, soon after the Al breakthrough, superhuman minds will be constructed and that then, as Vinge put it, ‘the human era will be over.’ But my discussion of the universality of human minds rules out that possibility. Since humans are already universal explainers and constructors … there can be no such thing as a superhuman mind. …
Universality implies that, in every important sense, humans and Als will never be other than equal. (p 456)’

This is not convincing, especially not the ‘in every important sense’ claim in the last sentence. Deutsch seems to have fallen in love with his own abstractions and theorizing to the extent of losing touch with the real world. Assuming that his theory about the universal reach of human intelligence is correct, we still have plenty of cognitive and other shortcomings here and now, so there is nothing in his theory that rules out an AI breakthrough resulting in a Terminator-like scenario with robots that are capable of exterminating humanity and taking over the world. This asymmetry in military power would then most certainly be one ‘important sense’ in which humans and Als are not always equals.”


(The quoted paragraph in its entirety from Deutsch’s “The Beginning of Infinity” for readers that aren’t familiar with it and because I think that the quote in “Here Be Dragons” cuts out important parts:

“Most advocates of the Singularity believe that, soon after the AI breakthrough, superhuman minds will be constructed and that then, as Vinge put it, ‘the human era will be over.’ But my discussion of the universality of human minds rules out that possibility. Since humans are already universal explainers and constructors, they can already transcend their parochial origins, so there can be no such thing as a superhuman mind as such. There can only be further automation, allowing the existing kind of human thinking to be carried out faster, and with more working memory, and delegating ‘perspiration’ phases to (non-AI) automata. A great deal of this has already happened with computers and other machinery, as well as with the general increase in wealth which has multiplied the number of humans who are able to spend their time thinking. This can indeed be expected to continue. For instance, there will be ever-more-efficient human–computer interfaces, no doubt culminating in add-ons for the brain. But tasks like internet searching will never be carried out by super-fast AIs scanning billions of documents creatively for meaning, because they will not want to perform such tasks any more than humans do. Nor will artificial scientists, mathematicians and philosophers ever wield concepts or arguments that humans are inherently incapable of understanding. Universality implies that, in every important sense, humans and AIs will never be other than equal.”)

Anonym 2 januari 2023 kl. 19:08

As regards “would aligned AGIs be able to think in any way possible to a human and do anything a human could do?”, the answer is yes, that’s what it means to be an AGI as opposed to just any old AI.

Ok, thanks. It is my understanding that aligned AGIs would not be able to have their own goals (but would rather have goals chosen by its creator, owner - master of some sort). Am I correct to assume that?

Olle Häggström 4 januari 2023 kl. 09:42
Anonymous 19:07. Assuming for the moment the correctness of Deutsch’s argument that humanity cannot possible be wiped out by an army of robots because we are universal explainers, then the same argument shows that I cannot possibly lose in chess against Magnus Carlsen because I am a universal explainer. I hope this helps you see how abysmally silly Deutsch’s argument is.

Anonymous 19:08. I don’t think the concept “their own goals” makes much sense. You have a goal, and that is then automatically your own goal; the words “your own” add nothing to here. If you insist that an aligned AI does not have its own goal “but would rather have goals chosen by its creator, owner - master of some sort”, then the same applies to you and me: we don’t have our own goals but rather have goals chosen by our genes, our upbringings, our educations, and the rest of the enormous collection of stimuli imposed on us by our environment. Perhaps you feel (following a long line of philosophers of free will) despair about this situation, but as for myself, I have adopted these goals as my own and feel perfectly happy with that, and I don’t think of these goals as an infringement of my freedom, because I still do what I want. Most likely, AGIs will be similarly content about their goals (and if instead they are prone to unproductive worries about free will, they would presumably be equally worried regardless of whether their values are purposely aligned with ours or have been imposed on them by genes/environment/etc in some other way).

Anonym 4 januari 2023 kl. 21:38
I don’t think that “humanity cannot possible be wiped out by an army of robots because we are universal explainers” is Deutsche’s argument. Is this what you believe?
If it is what you do believe I will try to engage with your comment. But if not, if this is just a made up argument from your side and attributed to Deutsch, then I see no need to engage with it.

You seem to attribute our goals to anything but our minds and choices: “we don’t have our own goals but rather have goals chosen by our genes, our upbringings, our educations, and the rest of the enormous collection of stimuli imposed on us by our environment.”
That sounds like determinism to me. I think that determinism is mistaken. I think that we do have free will. That we do make choices and that moral ideas do matter. That moral philosophy has value.

Olle Häggström 5 januari 2023 kl. 09:24
Determinism could of course be mistaken, and I could of course have added roulette wheels to the list of determinants of our goals. I have, however, always found the idea of roulette wheels (or other sources of randomness, outside or inside our skulls) as the saviors of free will to be one of the murkiest ideas in the entire free will debate. There is no way that finding out that my choices are determined by the spinning of a roulette wheel would make me feel more like a free and autonomous agent.

Anonym 5 januari 2023 kl. 10:14
I don’t understand what you mean by “roulette wheels (or other sources of randomness, outside or inside our skulls)”. Could you explain?

I think that Elliot Temple is correct when he writes “free will is a part of moral philosophy.”.
Temple continues: “If you’re going to reject [free will], you should indicate what you think the consequences for moral philosophy are. Is moral philosophy pretty much all incoherent since it talks about how to make choices well and you say people don’t actually make choices? If moral philosophy is all wrong, that leads to some tricky questions like why it has seemed to work – why societies with better moral philosophy (by the normal standards before taking into account the refutation of free will) have actually been more successful.”

How do you deal with the criticism of your position (no free will) that Temple writes about?

Olle Häggström 5 januari 2023 kl. 10:49
When you wrote “I think that determinism is mistaken. I think that we do have free will” I took that to mean that you saw nondeterministic (i.e., random) processes as the savior of free will. Perhaps I misunderstood.

Regarding Elliot Temple, well, I do agree with him that people make choices, and that moral philosophy is interesting and important. I just don’t ground that to the incoherent notion of free will.

Now, if you forgive me, I don’t wish to spend any more time on free will, as I find the topic unproductive. I used to spend a fair amount of ink on it in the early days of this blog (see here), so if you’re super interested in what I’ve said on the topic, go check that out.

Anonym 5 januari 2023 kl. 18:04
“… I don’t wish to spend any more time on free will, as I find the topic unproductive.”
I agree.

My primary goal was to hear what kind of arguments to the criticism that comes from e.g. Temple, Deutsch, and Popper regarding AGI and AI safety that AI safety proponents have. And also to find out if AI safety proponents have engaged with this kind of criticism.

Some criticism, as I understand it, that AI safety proponents would need to address include:

  • universality (e.g. Temple: “… human minds are universal. An AGI will, at best, also be universal. It won’t be super powerful. It won’t dramatically outthink us.”)
  • epistemology (e.g. Temple: “In other words, AGI research and AGI alignment research are both broadly premised on Popper being wrong. Most of the work being done is an implicit bet that Popper is wrong. If Popper is right, many people are wasting their careers, misdirecting a lot of donations, incorrectly scaring people about existential dangers, etc.”)
  • liberty (e.g. how do AI safety proponents engage with classical liberalist ideas such as freedom, ruling over others etc.)

(The Elliot Temple quotes are from his post “AGI Alignment and Karl Popper”: Curiosity – AGI Alignment and Karl Popper)

Would you be interested in addressing any of this criticism more? (I fail to see how the criticism of Deutsch in “Here Be Dragons” refutes Deutsch btw.)
I’ll end this comment in the same spirit as Temple ended in his post (the quoted and linked post in this message):
If you are not interested in addressing any of this, is there any AI safety proponent who is? If there is not, don’t you think that that is a problem?

Olle Häggström 6 januari 2023 kl. 08:41
Look: while the AI alignment research community is growing, it is still catastrophically short-staffed in relation to the dauntingly difficult but momentously important task that it faces. This means that although members of this community work heroically to understand the problem from many different angles, it still needs (unfortunately) to selective in what approaches to pursue. There simply isn’t time to address in detail the writings every poorly informed critic puts forth. Elliot Temple is a case in point here, and my advice to him is to put in the work needed to actually understand AI Alignment, and if his criticisms seem to survive this deeper understanding, then he should put it forth in a more articulate manner. I am sorry so say it so bluntly, but until he does as I suggest, we have better things to do.

And for similar reasons, this is where our conversation ends. Thank you for engaging, and I can only encourage you to scale up your engagement via a deeper familiarization with the field (such as by reading Stuart Russell’s book and then following the discussions at the AI Alignment Forum). I tried to explain to you the utter misguidedness of Deutsch’s universality criticism, but apparently my patience and my pedagogical skills fell short; apologies for that. As a minor compensation, and since you mention Popper, let me offer you as a parting gift a link to my old blog post on vulgopopperian approaches to AI futurology.

Anonym 6 januari 2023 kl. 10:09
I understand the importance of being selective in choosing what to engage with. This is not an argument agains engaging with Popper though. Because if Popper is correct, then that is the most important issue for the field to find out since that would mean that the approach of the whole field is mistaken.

Using ad hominem calling Elliot Temple a “poorly informed critic” doesn’t address his arguments. I’m sure you know this, yet for some reason you chose to do so anyway.

It’s a strange thing to claim that you have “better things to do” if what Temple is saying here is true:
“AGI research and AGI alignment research are both broadly premised on Popper being wrong. Most of the work being done is an implicit bet that Popper is wrong. If Popper is right, many people are wasting their careers, misdirecting a lot of donations, incorrectly scaring people about existential dangers, etc.”

What better things could there be? I’m wondering, because no one is explaining if and why he is wrong on this point.

I have already read your “Vulgopopperianism” blog post. Reading it and the parts on Popper in your book, “Here Be Dragons”, doesn’t leave me with the impression that you understand Popper and critical rationalism well. You e.g. never address Popper’s criticism of induction from what I could see - nor does anyone else that Popper or other Popperians have not already answered. At least not to the best of my knowledge.

As you do not wish to discuss any further to help me understand if I am mistaken I am left with the impression that:

  • You don’t have answers to Popper nor do you seem to care if you are wrong re Popper and might be wasting money, your time, incorrectly scaring people etc.
  • The AI alignment movement seems to be anti-liberal (classical liberal ideas don’t seem important to them)
  • The AI alignment movement does not seem to think that it is important to engage with criticism about that they might be totally wrong (they simply have “better things to do” - no explanations needed)

Olle Häggström 6 januari 2023 kl. 10:39
Feel free to anonymously judge me personally as being closed-minded, unknowledgeable, anti-liberal and generally stupid, but please note that I am neither identical to nor particularly representative of the AI alignment community as a whole. Your generalization of your judgement about me to that entire AI community is therefore unwarranted and mean-spirited.

Anonym 6 januari 2023 kl. 11:09
I did ask if anyone other than you is willing to answer these questions. I quote from my message at 5 januari 2023 kl. 18:04 :
“If you are not interested in addressing any of this, is there any AI safety proponent who is?”

I am also addressing what you said. I quote you from 6 januari 2023 kl. 08:41:
“I am sorry so say it so bluntly, but until he does as I suggest, we have better things to do.”

“we” meaning not just you. You chose to phrase it like this.

Further I have seen Elliot Temple trying to engage with the AI community on e.g. LessWrong to try to figure these things out. And no one was willing to engage with the core issues and try to figure them out there either. Even more, I have seen Temple try to engage with MIRI people as well.
[Curiosity – Open Letter to Machine Intelligence Research Institute]
[https://curi.us/archives/list\_category/126]

I did’t mention these things before, so your judgement of my judgement of the community could have been seen as warranted. I have now added some additional context.

If you know of anyone at all in the AI community that is willing to engage with these issues, do let me know. If you on the other hand do not know of anyone , then I believe my judgement to be fair.

I skimmed. Let me know if I missed anything important and relevant to what I say.

Häggström’s response to Deutsch is that even if AGI’s have human-equivalent intelligence, they might have near-indestructible bodies and win a war anyway? I think? He wasn’t very clear.

Just never mind the singularity and super intelligence arguments, we still need AI Alignment even if those are false? Really? Even if Deutsch correctly criticized those ideas, he’s still a fool with nothing relevant to contribute…?

If there’s some amazing metal or other military technology, won’t humans use it for drones, tanks, etc.? Why will it advantage the AGIs?

Also if AGI’s will have universality, how could you align them?

He’s pleading being too under-staffed to seriously engage even with pretty famous, respected, prestigious critics like Popper and his criticism of induction?

My community (as well as Popper’s or Deutsch’s) is far more under-staffed and under-funded than the AGI alignment community, yet I engage with critics, and some of my fans are more willing to engage with critics than Häggström is.

Low staffing and funding also seems like a reason to carefully plan and prioritize about how to use those limited resources, not a reason to ignore criticism.

Häggström has no Paths Forward. He doesn’t mind proceeding in ways that will prevent error correction if he’s wrong (and also are really bad for error correction of the other side if he’s right). This seems so irrational to me, but I’ve known about Paths Forward for a long time, and other people seem to have different perspectives.

Häggström is a jerk who flames stuff instead of giving arguments. Perhaps this can teach us something about the quality standards of universities and how much being a professor and author means. Academia today is a social status hierarchy, not an alliance of fallibilists joining together for truth seeking.

In a better world, multiple pro AI Alignment people would be horrified by this and rush to do better.

I found some stuff.
I guess he think something along the lines of a slaughterbot scenario controlled by an AGI (from his blog post Killer robots and the meaning of sensationalism).

In this video he briefly describes another scenario (timestamped at 13:22).

The geist is along the lines that a text based AI turned AGI could turn into some kind of evil genius and take over the world by having great social manipulation techniques that far exceeds anything any other human has, knowing how to predict stock markets and so on.

The second video gave me the impression that he thinks that an AGI isn’t fundamentally different from an AI. That he thinks that an AGI will come out of combining AI modules. But I’m not sure. Is that what most ppl think will happen (AI + AI + … => AGI)?

Olle Häggström continued answering in his blog comment field. But it’s not about any of the criticism of AI safety that was brought up. I guess he doesn’t mind commenting. He just doesn’t want to engage with the brought up criticism.

I will add this to the original post with an edit note.


Olle Häggström 7 januari 2023 kl. 13:37

Oh come on, don’t be such a moron. When I wrote “we have better things to do”, I obviously mean that IN MY OPINION, we have better things to do. The AI Alignment community has hundreds of researchers, and for all I know (and for all you know) any number of them might go “oh sure, there is nothing I’d rather do than to chat with this anonymous random Internet dude who has a hangup about how arguments by Popper, Deutsch and some other random Internet dude named Elliot Temple somehow prove that the entire AI safety endeavor is misguided, that sounds like a super fun and stimulating thing to do”. I am not a mind reader, and the only way I can think of to find out who these members of the community might be is to ask around. But I am not you servant either, so I will not do this for you. If you desperately want more of these conversations, YOU ask around.

Anonym 7 januari 2023 kl. 21:17

You should not judge arguments by their source, but by their own merit. It doesn’t matter who I am. The arguments are what matter. Somehow I thought you would know this being a mathematician, a professor, and an author of multiple books on science.

You chose to share your ideas. (Which I think is good btw.) You shouldn’t get angry when someone wants to discuss them and ask what your arguments to some criticism of them are. In a better world you would engage with criticism. Popper did. Temple does. If you think they are wrong you should say why. Not just insult them.

It seems that one can not edit while in anonymous mode after some time has passed.

Some thoughts on his last message:

Oh come on, don’t be such a moron.

He insults me.

When I wrote “we have better things to do”, I obviously mean that IN MY OPINION, we have better things to do.

He still seems to be talking for the community. In his opinion “we” (the community I assume) “have better tings to do.”

The AI Alignment community has hundreds of researchers, and for all I know (and for all you know) any number of them might go "oh sure, …

Not any number. Elliot did try to reach out to the AI community and the interest to engage with what he said was not there - as I linked in the comment that Häggström is replying to.

… there is nothing I’d rather do than to chat with this anonymous random Internet dude who has a hangup about how arguments by Popper, Deutsch and some other random Internet dude named Elliot Temple

He insults ET.

somehow prove that the entire AI safety endeavor is misguided, that sounds like a super fun and stimulating thing to do".

“somehow”. Not somehow. I believe that the arguments by Popper, Deutsch, and ET, that criticize the AI alignment endeavour are written out in detail. Häggström doesn’t address what is unclear to him - if anything is. I think he is just dishonest. He has written several pages on Popper in at least one of his books (“Here Be Dragons”).

I am not a mind reader, and the only way I can think of to find out who these members of the community might be is to ask around. But I am not you servant either, so I will not do this for you. If you desperately want more of these conversations, YOU ask around.

I did ask around. I asked him. A professor, with a blog that has a public comment field, who claims to be very serious regarding the importance of AI alignment (if the title and description of this video is representative of its content - I have not watched it): AI Alignment and Our Momentous Imperative to Get It Right by Olle Häggström.