Voices in AI – Episode 45: A Conversation with Stephen Wolfram

Wait. Why do you think the computer isn’t having a first-person experience?

It’s not a person. I am kidding. If you believe the computer experiences pain, I would love to have that conversation.

Let’s talk about the following situation. Let’s talk about a neural net. I mean, they’re not that sophisticated yet and they’re not that kind of recurrent, they tend to just feed the data through the network. But, you know, we’ve got a neural net and it’s being trained by experiences that it’s having. Then the neural net has some terrible experience. It’s terribly traumatic for the neural net. That trauma will have a consequence. If we were to look, sort of forensically, at how that had affected the weights in the neural net, we would find that there were all these weights that were affected in this or that way by the traumatic experience the neural net has had. In what sense do we then think–we then have to tease apart, what’s the difference between the effect of that experience the neural net had, and the experience the brain has.

That’s even more insidious than putting somehow people, and hurricanes, and iPhones in kind of the same level. That’s even worse because, in a way what you’re saying is, I had this car, and I’m lost in the woods and the car’s overheating, and the engine is out of oil, and the tires are torn up, and I’m tearing that car up. But, I’m being pursued or something, and I have to get out of the woods. I essentially just destroy this car making my way out. If your assumption is, “Well, that car experienced something” you know, you were afraid of getting eaten by lions but you killed the car in doing it. And to somehow put those two things on the same level, you can’t really make that choice.

Well, the morality of AI is a complicated matter. For example, if you consider…

I’m just asking about the basis of human rights. The basis of human rights are that humans feel pain. The reason we have laws against harming animals is because animals feel pain. What you’re suggesting is infinite loops. If you code an infinite loop, by golly, you should get fined, or go to jail.

Yeah. Well, the question, to put it in a different way, if I succeed in making a bot, autoresponder that’s like me and responds to e-mail independent of me. And for example, let’s say I’m no longer around, I’m dead, and all that’s left is the autoresponder. What are the obligations? How does one think about the autoresponder relative to thinking about the person that the autoresponder represents? What do you think? I think at that point, I haven’t actually thought this through properly, but I think if somebody says, “Let’s delete the autoresponder,” it’s interesting. What are the moral aspects of doing that?

If your argument is it’s the moral equivalent of killing a living person, I would love to hear that logic. You could say that’s a tragedy, that’s like burning the Mona Lisa, we would never want to do it. But to say that it’s the equivalent of killing Stephen Wolfram a second time, I mean, I would love to hear that argument.

I don’t know if that’s right. I have not thought that through. But, my reaction to you saying the computer can’t feel pain is, I don’t know why on Earth you’re saying that. So, let’s unpack that statement a little bit. I think it’s interesting to unpack. Let’s talk a little bit about how brains might work and what the world looks like at a time when we really know, you know, we’ve solved the problem of neurophysiology, we’ve solved, sort of, the problem of neuroscience, and we can readily make a simulation of a brain. We’ve got a simulated brain and it’s a simulated Byron, and it’s a simulated Stephen and those simulated brains can have a conversation just like we’re having conversation right now. But unlike our brains, it’s easy to go look at every neuron firing, basically, and see what’s happening. And then we start asking ourselves… The first question is, do you think then that the neuron level simulated brain is capable of feeling pain, and having feelings, and so on? One would assume so.

We would part company on that but I agree that many people would say that.

Well, I can’t see how you would not say that unless you believe that there is something about the brain that is not being simulated.

Well, let’s talk about that. I assume you’re familiar with the OpenWorm project.


The C. elegans is this nematode worm. Eighty percent of all animals on the planet are nematode worms. And they had their genome sequenced, and their brain has 302 neurons.

There is a difference between male and female worms actually. I think the female worm has 4 neurons.

Fair enough. I don’t know, that may be the case. Two of the 302, I understand, aren’t connected to the other ones. Just to set the problem up, so for 20 years people have said, “Let’s just model these 302 neurons in a computer, and let’s just build a digital nematode worm.” And of course, not only have they not done it, but there isn’t even a consensus in the group that it is possible. That what is occurring in those neurons may be happening at the Planck level. Your basic assumption in that is that physics is complete and that model you just took of my brain is the sum total of everything going on scientifically, and that is far from proven. In fact, there is more evidence against that proposition.

Let’s talk about this basic problem. Science – a lot of what goes on in science is an attempt to make models of things. Now, models are by their nature incomplete and controversial. That is, “What is a model?” A model is a way of representing and potentially predicting how a system will behave that captures certain essential features that you’re interested in, and elides other ones away. Because, if you don’t elide some features away, then you just have a copy of the system.

That’s what we’re trying to do. They’re trying to build an instantiation; it’s not a simulation.

No, but there is one case in which this doesn’t happen. If I’m right that it’s possible to make a complete, fundamental model of physics, then that is the one example in which there will be a complete model of something in the world. There’s no approximation, every bit works in the model exactly the way it does in real life. But, above that level, when you are saying, “Oh, I’m going to capture what’s going on in the brain with a model,” what you mean by that is, “I’m going to make some model which has a billion degrees of freedom” or something. And that model is going to capture everything essential about what’s happening in the brain, but it’s clearly not going to represent the motion of every electron in the brain. It’s merely going to capture the essential features of what’s happening in the brain, and that’s what 100 percent of models, other than this one case of modeling fundamental physics, that’s what they always do. They always say, “I’m capturing the part that I care about and I’m going to ignore the details that are somehow not important for me to care about.” When you make a model of anything, whether it’s a brain, whether it’s a snowflake, whether it’s a plant leaf or something, any of these kinds of things, it’s always controversial. Somebody will say, “This is a great model because it captures the overall shape of a snowflake” and somebody else will say, “No, no, no it’s a terrible model because look, it doesn’t capture this particular feature of the 3-D structure of ridges in the snowflake.” We’re going to have the same argument about brains. You can always say there’s some feature of brains, for example, you might have a simulation of a brain that does a really good job of representing how neuron firings work but, it doesn’t correctly simulate if you bash the brain on the side of its head, so to speak, and give it concussion, it doesn’t correctly represent a concussion because it isn’t something which is physically laid out in three-dimensional space the way that the natural brain is.

But wasn’t that your assumption of the problem you were setting up, that you have perfectly modeled Byron’s brain?

That’s a good point. The question is, for what purpose is the model adequate? Let’s say the model is adequate if listening to it talking over the phone it is indistinguishable in behavior from the actual Byron. But then, if you see it in person and you were to connect eyes to it, maybe the eye saccades will be different or it wouldn’t have those, whatever else. Models, by their nature, aren’t complete, but the idea of science, the idea of theoretical science is that you can make models which are useful. If you can’t make models, if the only way to figure out what the system does is just to have a copy of the system and watch it do its thing, then you can’t do theoretical science in the way that people have traditionally done theoretical science.

Let’s assume that we can make a model of a brain that is good enough that the brain can, for many purposes that we most care about, can emulate the real brain. So now the question is, “I’ve got this model brain, I can look at every feature of how it behaves when I ask it a question, or when it feels pain or whatever else.” But now the question is when I look at every detail, what can I say from that? What you would like to be able to say is to tell some overarching story. For example, “The brain is feeling pain.” But, that is a very complicated statement. What you would otherwise say is, there’s a billion neurons and they have this configuration of firings and synaptic weights, and God knows what else. Those billion neurons don’t allow you to come up with ‘a simple to describe story’, like, “The brain is feeling pain.” It’s just, here’s a gigabyte of data or something; it represents the state of the brain. That doesn’t give you the human level story of “the brain is feeling pain.” Now, the question is, will there be a human level story to be told about what’s happening inside brains? I think that’s a very open question. So, for example, take a field like linguistics. You might ask the question, how does a brain really understand language? Well, it might be the case that you can, sort of, see the language coming in, you can see all these little neuron firings going on and then, at the end of it, some particular consequence occurs. But then the question is, in the middle of that, can you tell the story of what happened?

Let me give you an analogy which I happen to have been looking at recently which might at first seem kind of far-fetched, but I think is actually very related. The analogy is mathematical theorems. For example, I’ve done lots of things where I’ve figured out mathematical truths using automated theorem proving. One, in particular, I did 20 years ago of finding the simplest axiom system for logic, for Boolean algebra. This particular proof generated automatically, it’s 130 steps, or so. It involves many intermediate stages, many lemmas. I’ve looked at this proof, off and on for 20 years, and the question is, can I tell what on Earth is going on? Can I tell any story about what’s happening? I can readily verify that, yes, the proof is correct, every step follows from every other step. The question is, can I tell somebody a humanly interesting story about the innards of this proof? The answer is, so far, I’ve completely failed. Now, what would it take for that to be such a story? Kind of interesting. If some of the lemmas that showed up in the intermediate stages of that proof were, in a sense, culturally famous, I would be in a much better position. That is when you look at a proof that people say, “Oh, yeah, this is a good proof of some mathematical theorem.” A lot of it is, “Oh, this is Gauss’” such and such theorem. “This is Euler’s” such and such theorem. That one’s using different stages in the proof. In other words, those intermediate stages are things about which there is a whole, kind of, culturally interwoven story that can be told, as opposed to just, “This is a lemma that was generated by an automatic theorem improving system. We can tell that it’s true but we have no idea what it’s really about, what it’s really saying, what its significance is, what its purpose is,” any of these kinds of words.

That’s also, by the way, the same thing that seems to be happening in the modern neural nets that we’re looking at. Let’s say we have an image identifier. The image identifier, inside itself, is making all kinds of distinction saying, “This image is of type A. This is not of type B.” Well, what is A and B? Well, it might be a human describable thing. “This image is very light. This image is very dark. This image has lots of vertical stripes. This image has lots of horizontal stripes.” They might be descriptors of images for which we have developed words in our languages, in our human languages. In fact, they’re probably not. In fact, they are, sort of, emergent concepts which are useful, kind of, symbolic concepts at an intermediate stage of the processing in this neural net but they’re not things for which we have in our, sort of, cultural development generated, produced, chosen to describe those concepts by words and things. We haven’t provided the cultural anchor for that concept. I think the same thing is true — so, the question is, when we look at brains and how they work and so on, and we look at the inner behavior and we’ve got a very good simulation, and we see all this complicated stuff going on, and we generate all this data, and we can see all these bits on the screen and so on. And then we say, “OK, well, what’s really going on?” Well, in a sense then we’re doing standard natural science. When we’re confronted with the world we see all sorts of complicated things going on and we say, “Well, what’s really going on?” And then we say, “Oh, well, actually there’s this general law” like the laws of thermodynamics, or some laws of motion, or something like this. There’s a general law that we can talk about, that describes some aspect of what’s happening in world.

So, a big question then is, when we look at brains, how much of what happens in brains can we expect to be capable of telling stories about? Now, obviously, when it comes to brains, there’s a long history in psychology, psychoanalysis etc. that people have tried to make up, essentially, stories about what’s happening in brains. But, we’re kind of going to know at some point. At the bit level we’re going to know what happens in brains and then the question is, how much of a story can be told? My guess is that that story is actually going to be somewhat limited. I mean, there’s this phenomenon I call ‘computational irreducibility’ that has to do with this question of whether you can effectively make, sort of, an overarching statement about what will happen in the system, or whether you do just have to follow every bit of its behavior to know what it’s going to do. One of the bad things that can happen is that, we have our brain, we have our simulated brain and it does what it does and we can verify that, based on every neuron firing, it’s going to do what we observe it to do but then, when we say, “Well, why did it do that?” We may be stuck having no very good description of it.

This phenomenon is deeply tied into all kinds of other fundamental science issues. It’s very much tied into Gödel’s theorem, for example. In Gödel’s theorem, the analogy is this: when you say, “OK, I’m going to describe arithmetic and I’m going to say arithmetic is that abstract system that satisfies the following axioms.” And then you start trying to work out the consequences of those axioms and you realize that, in addition to representing ordinary integers, those axioms allow all kinds of incredibly exotic integers, which, if you ask about certain kinds of questions, will give different answers from ordinary integers. And you might say, “OK, let’s try to add constraints. Let’s try to add a finite number of axioms that will lock down what’s going on.” Well, Gödel’s theorem shows that you can’t do that. It’s the same sort of mathematical structure, scientific structure as this whole issue of, you can’t expect to be able to find simple descriptions of what goes on in lots of these kinds of systems. I think one of the things that this leads to is the fact that, both in our own brains and in other intelligences, other computational intelligences, that there will be lots of kinds of inner behavior where we may not ever have an easy way to describe in large-scale symbolic terms, the kinds of things going on. And it’s a little bit shocking to us that we are now constructing systems that, we may never be able to say, in a sort of human understandable way, what’s going on inside these systems. You might say, “OK, the system has produced this output. Explain that output to me.” Just like, “The following mathematical theorem is true. Explain why it’s true.” Well, you know, if the “why it’s true” comes from an automated theorem prover, there may not be an explanation that humans can ever wrap their brains around about that. The main issue is, you might say, “Well, let’s just invent new words and a language to describe these new lumps of computation that we see happening in these different systems.” The problem, and that’s what one saw even from Gödel’s theorem, the problem is that the number of new concepts that one has to invent is not finite. That is, as you keep on going, you keep on looking at different kinds of things that brains, or other computational systems can do, that it’s an infinite diversity of possible things and there won’t be any time where you can say, “OK, there’s this fixed inventory of patterns that you have to know about and that you can maybe describe with words and that’s all you need, to be able to say what’s going to happen in these systems.”

So, as AIs get better and we personify them more and we give them more ability, and they do actually seem to experience the world, whether they do or they don’t, but they seem to, at what point, in your mind, can we no longer tell them what to do, can we no longer have them go plunge our toilet when it stops up. At what point are they afforded rights?

Leave a Reply

Your email address will not be published. Required fields are marked *