12 – Value Alignment

Transcript

This transcript was generated automatically and may contain mistakes.

I felt like making an impromptu episode because I saw an interesting tweet today. Somebody tweeted at me, and it was in response to a thread, and the topic was value alignment, and the thread was about the dangers, or the alleged, I should say, the alleged dangers of AI and AGI. I’ve addressed those before in episode two, I think. But this particular instance was really interesting because, so I tweeted that narrow AI is just like all other computer programs. It’s just the execution of predefined tasks. And no new knowledge is being created to solve novel problems. But AGI, on the other hand, is a universal problem solver and it creates knowledge. So it is the opposite of not creating knowledge. There was a point of contention whether or not AI and AGI are really opposites.

And then I said, by the way, perfect value alignment is neither feasible nor desirable. Somebody had suggested that value alignment is important. And so my response to that, no, it’s not. And we shouldn’t. Not only is it unimportant, but it’s not even desirable. And so in response somebody tweeted at me and said, well, how do you know, this is a quote, “How do you know the AGI isn’t wrong, and why is perfect value alignment undesirable?” End quote. So I want to get a little deeper into this than Twitter might allow, and that’s why I felt like recording an impromptu episode on this.

This question, how do you know the AGI isn’t wrong, is it corresponds to a pattern that once you are familiar with it, you will see it all the time. And this pattern corresponds to the mistake that Popper identified as the “who should rule” mistake.

So if you’re thinking about political systems and you want to design a political system, let’s say that’s as fair as possible or as representative as possible or whatever your goal is, if you want to design a democracy, for example, if you ask the question “who should rule”, like if that’s your main focus, and you think the answer to that question, if you could find it, you think that the answer to that question would solve your problems in a democracy, that is, you’re in a really bad situation.

Now the reason that is, is because it sounds like a good question, right? It sounds like, well, who should rule? That seems like the first logical question to ask yourself and others, and it seems like the question that we’re asking ourselves, right? In the United States, who should rule seems like that is the question we ask every election. And yet that is not the thing that makes a democracy a democracy.

And I promise I’ll connect this to the question of AGI and value alignment, because these things are deeply connected, even though superficially they don’t seem to be. Now, when you ask the question, who should rule, what you’re doing is you’re looking for an authoritative answer. You’re looking for reasons why to do something or for reasons why to elect somebody. So rather than focusing on piecemeal improvement and error correction, you’re focusing on getting the end-all be-all answer. And it might only be for four years, but it’s, it’s supposedly the end-all be-all answer for four years. This is the guy, or woman one day, but this is the guy who’s going to turn it all around. This is going to solve all our problems. So this is your authoritative answer.

And now these questions are usually born out of good intentions. You want to find the right, you know, capital T, capital R, the right leader to solve political problems. But the problem with that is once you have found this supposedly right leader, you will then do everything in your power for him to remain in power. And that is exactly the wrong thing you want because you want a political system where it’s as easy as possible to remove bad leaders and bad policies without bloodshed. So anything that inhibits error correction in this way, anything that inhibits, for example, the removal of a bad leader or of a bad policy without bloodshed is immoral. Inhibiting the means of error correction is the very definition of immoral.

So when you ask how do you know, or how do you know this isn’t wrong, or how do you know this is right, or who should rule, you’re setting yourself up for an immoral outcome. So that is the problem with this, with this type of question. You could answer the question directly. You could try to answer the question directly. How do you know the AGI isn’t wrong? How do you know this leader is not the right leader? Well, you could try to come up with an answer for that, and you could answer it directly. But I think it’s more productive to point out kind of on a meta level on top of that.

Well, I think it’s better to point out that asking the question is misleading. Now, it’s not wrong to ask questions. Absolutely not. Of course, it’s important to ask questions. It’s just as you think about discussions and you think about rational inquiry, you want to think about what kinds of questions are productive and what kinds of questions, even though they’re born out of good intentions, might lead down an error-prone path.

So that is my conclusion here. This is the mistake that Popper identified, is that questions of this sort lead to bad answers. And they ask for an authoritative answer. And then once you have that answer, you’re determined not to change it. And especially if they’re based on good intentions, you’re especially determined not to change it. And anyone who dares to criticize them must be evil or must fail to see the allegedly manifest truth.

So that is the kind of issue we’re getting into here. And I think that is the same problem with the value alignment debate. This question, how can we make sure that AGI does as it is intended to do or does as it is told to do, or it doesn’t do anything that sways too far from what we would expect. Okay, so that is the same kind of “who should rule” mistake.

Now, I have addressed the difference between narrow AI and AGI before, just to recap it quickly. I basically don’t, it’s going to sound rude, but I basically don’t see anything special about narrow AI. I’ll admit that they’re impressive in their reach. They can solve problems. Those algorithms can solve problems that we weren’t able to solve before, and that’s great, and that’s really cool.

But other than the reach of narrow AI programs and maybe the level of sophistication that they contain, they’re not different from any other computer programs that we’ve written. They’re written to a specification, they’re written with a problem in mind, and they’re written to solve that problem. And all the knowledge that is required to solve the problem, even if it’s only implicit, is created by the programmer, in the programmer’s mind, before writing the program down.

Now, narrow AIs don’t differ in this regard from any other computer programs. If you can think of a super simple program like a temperature conversion program, or a slightly more complicated program like a navigation problem, like a program that will navigate you from point A to point B, they may update parameters at runtime given, you know, circumstances and inputs from the environment and whatever, and that’s all impressive and sophisticated, but that’s not what intelligence is. In fact, if there was a program that couldn’t do any of these things yet but was creative and then was able to teach itself how to do these things, that would be intelligence.

So when we ask the question, how do we know that a, a narrow AI would do the thing that it is intended to do, well, this follows the same reason that the, it follows the same logic as an evaluation of all the other programs we’ve written so far, because again, they’re not that different. So how do I know my temperature conversion algorithm does the thing it’s supposed to do? Well, I can test it. I can make sure that the outputs match the outputs that I expected for the given inputs. I can make sure that the side effects it performs, if it performs any, are the ones that I expected. I can criticize its structure and so forth. And then through an iterative process, I may come up with a pretty good solution to the problem of how to convert temperatures.

Now, there is nothing immoral about forcing, so to speak, a conventional, you know, classical computer program, such as a temperature conversion program or a narrow AI, down a specific path. That is what it was written to do. That thing cannot suffer. It’s not creative. It’s not conscious. There’s nothing immoral about doing that. So “who should rule” questions for traditional computer programs aren’t that dangerous in this regard.

However, if we were to apply the same question to an AGI, which is a person, we might get into trouble here and we might run into serious moral issues because once we find an answer to the question, how do we make sure AGI does exactly as it is told, and you realize that this is a person, then you also realize that the only way to ensure that it does as it is told is to force it.

So coercion is your only answer. And this is why people like Nick Bostrom in his book Superintelligence basically wrote guidelines as to what to do if you were to identify a program as not doing what it is told to do. In the case of AGI, he suggests, for example, placing it in a metal mesh so that it can’t communicate with anyone. He says that we shouldn’t let it interact with humans because it might scheme and persuade them to free it. He suggests crippling it by running it on hardware that is slow or short on memory. And he suggests setting up tripwires to monitor it and shut it off. So if you shut off an AGI, that’s murder. It’s not okay to do. And note that he says all this allegedly, quote, for the benefit of all humanity, end quote.

So this is why I said that these kinds of “who should rule” questions apply not only to political questions, but also to, or political matters, but also to matters of AGI. And indeed, they apply to many other fields as well. And because AGIs are people, “who should rule” questions, or like value alignment questions such as how do we make sure the AGI does as it’s told to do, lead to murderous, dangerous, and violent, like slaveholder-like manuals. That is why I think asking something like, how do we know the AGI isn’t wrong, and is perfect value alignment not desirable, those are dangerous questions to ask.

Now, so what do we do with this question of value alignment? I think this whole question, this problem of value alignment is a pseudo problem. I don’t think it’s really an issue. Now, why do I think that? I think that the question of how to align an AGI’s values is the same as how to align a child’s values.

A newborn AGI is a child. There’s no difference there. I mean, there’s a difference in terms of hardware, so it will not look like a human child, but people aren’t hardware. People are programs. So once you imagine forcing a child down a certain path, you realize how immoral it is to do. And unfortunately we still do this all the time, every day, by forcing kids to go to school. But if we ask that question of AGI, we’re committing the same crime of forcing somebody down a certain path against their will.

That is why it is important to remember that AGI is not like all the other computer programs. AGI, according to our best theories, will have consciousness, it will have the capacity to suffer, and it will choose its own problems that it wants to work on. It is not a computer program that people will write to convert temperatures or to navigate them from point A to point B. AGI is a computer program that will detect problems it wants to work on and solve those problems on its own. It will create knowledge, we’re in collaboration, but it will create knowledge to do that itself. That knowledge will not be fed or, you know, incorporated or leaked into the program by a programmer.

So in AGIs, there’s some underlying mechanism from which this problem-solving ability emerges. And this underlying mechanism can be specified, but the problems the AGI wants to solve can’t be specified because they can’t be predicted. The future growth of knowledge is unpredictable, and that includes the problems that people want to work on to create more knowledge.

So therefore, creating an AGI for the purpose of solving a specific problem is immoral. You would only want to create an AGI because you want to be a parent. So if you really want to be a parent and you think your primary obligation is to help the AGI learn, then you should have an AGI. You should create an AGI. Otherwise, you should not create an AGI. Because once you force it down a particular path, you become a slaveholder. That is not an exaggeration. So, therefore, I suggest that we should replace the whole value alignment debate with the question of how to help an AGI learn.

The problem is not how do we get our children to do exactly what we tell them to do. The problem is how can we encourage children to use this wonderful facility that they have, their creativity, to solve the problems that they want to solve in their own light? That is the primary question that we should be asking. How can we help AGIs detect and correct errors? And again, this focus on error correction is important, because if we don’t focus on error correction and we try to, quote, get things right, we will only come up with answers that we will entrench down the road and deem unchangeable because they’re supposed to be the right thing.

And it completely ignores this notion that we’re fallible. We may have made mistakes in trying to answer that question. And if we made the mistakes and that leads to new problems, we would then have to avoid those problems, because once we acknowledge that there are problems with the approach, with the answer that we came up with to the question of how to do, quote, the right thing, then we acknowledge that it’s not, after all, the right thing. So there must be something wrong, and we avoid error correction.

Instead, we should turn the whole thing around. We need to accept the fact that we’re fallible, that we make mistakes. That includes our moral judgment, and it includes our values. And morality has progressed, and we should expect people that live 200 years from now to look back in horror on the things that we do today.

That is not to say that I think we live in a horrible society. I think we live in a great society, especially in the West. But no doubt we will find errors with our moral knowledge that we want to then improve. And we will improve that, hopefully, over time. So if we were to simply align, if we somehow found a way to forcefully align an AGI with our values, well, we would then also entrench all the mistakes that we made and deem them unchangeable, because for now we think that they’re the right thing to do. And so therefore, we would deprive an AGI of the ability to correct those mistakes. And that is the very definition of immoral. So let’s replace the whole value alignment debate with the question of how can we help our AGI children detect and correct errors? Or in short, how can we help our AGIs learn?

Now, I assume that that replacement won’t satisfy those who are already convinced that value alignment is a real problem. Because value alignment, the problem of value alignment is born out of this question of, you know, if a superintelligence were to become dangerous, how do you then defend yourself? I’ve addressed these questions in episode two. And I think we can… I realize that the problem of value alignment is in response to that question. So it’s meant to solve a problem.

And my response to that is, I don’t think there is a problem that needs to be solved. First of all, because superintelligence can’t exist. And the second reason is that if you’re worried about danger and if you’re worried about harming people, then the very last thing you should do is forcing AI, or AGI, I should say, down a particular path, because then that is exactly the thing that you would be doing. You would be harming it.

So I think we can safely dispose of the value alignment problem without really running into any new problems. So in conclusion, I suggest avoiding questions of the sort “who should rule” and “how do we know that what the AGI does is right”, and replacing them with piecemeal error correction and a focus on how to help an AGI learn and correct its mistakes. Once the AGI creates knowledge about morals, then helping it detect errors in that area will also improve its moral knowledge, and therefore the value alignment problem just goes away.