Cover art for ‘06 – Function Replication’

06 – Function Replication

Artificial Creativity

Listen on Apple Podcasts Listen on Google Podcasts Listen on Spotify

This episode relies heavily on video components, so you’ll want to watch it on YouTube: https://www.youtube.com/watch?v=xRaR3LOn1JQ

References

  • David Deutsch, “The Beginning of Infinity”, chapter 16, “The Evolution of Creativity”

Further reading

  • David Deutsch, “The Fabric of Reality”
  • David Deutsch, “The Beginning of Infinity”, chapter 7, “Artificial Creativity”
Subscribe to this podcast via RSS Back to all episodes

Transcript

This transcript was generated automatically and may contain mistakes.

Welcome back, folks. This is episode 6 of the podcast on Artificial Creativity. This is another video episode, so I highly recommend you pull this up on YouTube if you’re listening on some other channel. I’d like to start out with a brief summary of the previous episode. Since creativity is all about creating good explanations, and we’re trying to build a piece of software, we needed to know how to encode explanations in computer programs. So we looked at an example of a hard-to-vary explanation of multiplication and implemented it as a function in Clojure, only to realize that the function implementation we ended up with was a hard-to-vary explanation of how multiplication works. And it turns out that the laws of epistemology also apply to function implementations. We concluded that this was no accident and finished the last episode with the following conjecture: functions are explanations, and explanations are functions.

Now, functions offer themselves as computational representations of explanations because each part of a function maps onto how explanations work. The function’s body is the explanatory part. The parameters are the initial conditions or experimental setup. The return values are predictions and so forth. And therefore, explaining explanations in terms of functions is itself hard to vary. My guess is that the underlying reason for this is that the computability of the universe is the very reason our universe is explicable in the first place. There is a deep connection here between computation and explanation that I think deserves to be explored. For more on this, I recommend you read “The Fabric of Reality” by David Deutsch.

Now, before we get started, I should point out that, if it wasn’t obvious already, the material in this podcast is speculative. I expect to need to revisit much of what I say because it is most likely mistaken. This is very much in line with our epistemology. We only learn from our mistakes, and so I hope you bear with me as we work through these mistakes together to get a little closer to the truth.

In any case, what we know at this point is that there are a bunch of things that are equivalent. To understand something means to create an explanation of it, which means to replicate it. Explanations are functions, and those functions are replicas of whatever we’re trying to understand. By the way, if you’re not so sure about this, you may want to re-listen to episodes 4 and 5. I also recommend you read David’s chapter “The Evolution of Creativity” in “The Beginning of Infinity”. So that means the creative algorithm produces functions in response to problems, and that allows us to rephrase the problem of creativity as the problem of how to replicate functions.

This brings me to my next conjecture: a universal knowledge creator can replicate any function given only some clues about how that function works. This is the problem of knowledge creation phrased in terms of software. As I pointed out in the previous episode, this is what I and all the other students did when our teacher explained multiplication to us. So what that means is, we need to guess the structure or implementation of a function based only on a few clues here and there.

So I have a Clojure REPL on the left here, and a text editor on the right. A Clojure REPL, by the way, is basically a Clojure environment where we can run some Clojure code. So, for example, you know, this is going to evaluate to three, print line, hello. It’s just going to run the Clojure statement that we pass in.

So what I’ve prepared is I’ve prepared three different functions. And we’re going to try to replicate their implementation without looking it up. So only by trying the functions out with random parameters. Okay, so consider this first example, and I invite you to just observe what is going on in your mind as we are solving this together. So this first function is called anonymous1. And we’re just going to try and invoke it. And the first thing it does is it’s complaining that we’re not passing it enough arguments. Now in the real world, we might not get such a hint, but just bear with me here.

So what we’re going to do is we’re going to pass in one argument, and you just pick any random thing you want to pass in. I mean, we could pick a number, for example. And let’s try running this. And so if we pass in one, we get two. Now let’s try if we could run this function with more than one parameter. And it turns out we can’t. So let’s limit ourselves to one parameter for now.

So if we pass in one, we get two and almost immediately because it’s second nature, we kind of get, get a hunch that maybe if we pass in three, sorry, maybe if we pass in two, we should get the number three. And indeed we did. So maybe if we pass in three, we get the number four and that is true. So I would say it looks like an implementation of anonymous1 is it takes one parameter and then what it does is it simply adds one to the parameter. So you could write it like this or you could write it like this. This is more of the Clojure way to do it. I’m going to just set up this here.

Now, as programmers, we have the advantage of actually looking up the implementation. So this would not work in real life, obviously. But if I were to look up the actual implementation, we can see that indeed that’s what this function is doing. Of course, our understanding of this implementation is itself fallible. So, but this, I mean, this, this gives us, gives us all the hints we need to understand that this, this is indeed what this function is doing.

So what you just saw here is, I mean, we, we started out by, by guessing to pass in no arguments, then we passed in one arguments and we passed in more arguments and that didn’t work anymore, and so we, we found that it works with one, and then we seem to sort of immediately conjecture the implementation x plus one. But my guess is that our minds actually go through a ton of garbage implementations on the way there without us being consciously aware of it.

Okay, so that was the first function. Now let’s check out the second function, anonymous2. So this is interesting, we can indeed pass in zero arguments, and we get an empty string back. So maybe this function has to do something with strings, because this is also important to point out. This is supposed to work for any function, right? So this is not just about passing in numbers. So maybe let’s just pass in a string of hello. Okay, it’s just returning hello.

Let’s try a different string. It’s just giving back the same thing. So it almost looks like the function just returns the given argument, right? So let’s write that down for now. Defn anonymous2, it looks like it takes one argument, although it looks like it also allowed us to pass in zero arguments. So we can write it like that. That way all the arguments become optional, and we can pass in any number of them. So it almost seems as though if there are no arguments we’re passing in, if there are no arguments, we’re simply returning an empty string, and otherwise it looks like we’re just returning the first argument or something like that. Now, this is a first conjecture function implementation for this. This works for everything we’ve seen so far. But we should try to correct it.

So let’s try running this with maybe more than one parameter. See what it does now. And so now we get a different picture. I mean, it’s still sort of returning the things that we passed in, but it’s concatenating them. And so now we get a different idea, and this is actually a much more elegant implementation, too. So I think what’s going on here, of course, I wrote the function, so I know what’s going on here, but what this might suggest is going on is it’s basically just applying string concatenation to all the arguments. Now, if this is true, we need to make sure that, indeed, this works for any number of arguments. Now we could never test this exhaustively, as I said, but it looks like it’s really doing that. If this is also true, it should turn numbers, for example, into strings as well.

So this should return the string 1, 2, 3, and it does. So I’d say for now this is a pretty good and hard-to-vary implementation of this function based on the parameters and return values that we’ve seen. So let’s look up the actual implementation. And indeed, that’s exactly what happens here.

Okay. Now, what we saw here also is unless you keep trying this, you may never try to pass in more than one argument because the previous example may be biasing you here, or the recent experience of that previous example here may be biasing you. So you may end up thinking that all this function does is just return whatever you give it. But as we found out through further and further testing, it works for two parameters, and then turns out it works for zero parameters, and it also works for any number of parameters. So this is an example where the first implementation that we, that we had of the second function sort of lives on as an approximation in its successor, which is this. And that is just like explanations in science. For example, relativity contains Newtonian physics as an approximation, but its structure is entirely different.

Okay, let’s look at a third example, anonymous3. And now we’re getting a warning again that we’re not passing in enough arguments. So let’s try passing in, I don’t know, one argument, and this is still too few arguments. So let’s try passing in two arguments, and now it’s saying, okay, it’s the right number of arguments, but it can’t turn a string into a number. So this is again more help than we would get in real life probably, but this does mean that we’re probably going to have to pass in numbers instead of strings. So let’s try that instead.

Anonymous3, let’s try 1 and 2. This returns 2. So maybe it just returns the last argument. That could be a first conjecture. Let’s try passing in 1 and 3, and indeed it seems to just return the last argument. But let’s try to also change the first parameter. So now it returns six. So that means our first guess was false. So we’re going to have to try to come up with a different guess. Let’s say what happens for four and three. So maybe now you’re starting to get a picture of what’s happening here. If you pass in two and three, it’s six. If we pass in four and three, it’s 12. So if you now have a hunch that this is multiplication, let’s try passing in 5 and 5, and indeed that’s 25. So I’d say that’s a pretty good hunch.

So let’s try implementing this, and we saw this in the last episode. Defn anonymous3, this is just going to take two arguments, and what we implemented in the last episode was something that looked like this: repeat A, B times, and then simply add them all up. And let’s check what the actual implementation looks like. If I can type it. And indeed, that’s exactly what’s happening here.

So I think what’s happening in your minds as you’re watching this and trying to figure this out is the creative algorithm trying out all sorts of things to make sense of this. And to make sense of this, it tries to replicate this function. Now, can we be sure that our function implementations are correct? Well, no, for the same reason that we cannot be sure that our explanations are correct. Can we be sure that the target function, like anonymous3 is a target function, that, can we be sure that it will always return the same values for the same parameters? Also no. By the way, this is one of the criteria for what is known as a pure function. And my guess is that regularities in nature are indeed pure functions in this sense, or they wouldn’t really be irregularities.

But in any case, once we have a workable implementation, any further invocation of the target function only serves the purpose, not of confirmation, but error correction. Now, we could have come up with a bad explanation for any of these. For example, we could have come up with a multiplication table for the last example, but that would have been a bad implementation and it would have had inherently limited reach.

So what we’re after is two things. One, do both functions’ parameters and return values match? Or in other words, do we get the same set of return values for the same set of parameters for both functions? Now we cannot possibly try all of them, but at least the ones we have tried so far should match. And two, have we achieved a good, meaning hard-to-vary, implementation? The second criterion is crucial here, because there are always infinitely many bad implementations that will match the target function’s parameters and return values, for example, multiplication tables and any variations thereof. But we are after the harder-to-find good implementations.

Therefore, I propose the following challenge. Write a function f that takes a function g, and it can invoke g with arbitrary parameters and read the corresponding return values just like we did. Now at no point in that process may f actually read g’s implementation. That’s forbidden. And f’s goal is to implement a copy of g that is a good implementation.

Now I think this model will need some refinement. For example, I don’t think this is exactly analogous to how things work in real life. For example, in the case of multiplication in school, we did get clues about the structure of multiplication before we even talked about example calculations. In the sentence “three times five”, the word “times” is a clue that looping is involved. That’s not to say, however, that we couldn’t have learned from examples. So I guess that if the creative algorithm can learn only from examples, it will be even better at learning with clues about the structure as well.

Another difference is that we as programmers do have access to each target function’s implementation, as we saw earlier, and we even came up with it. So communicating with a sort of private function that lives on a server somewhere that we don’t have access to would be a better analogy. In any case, I think stating the problem in this way is part of the problem of how to build a universal explainer expressed in software terms, and it allows us to criticize any approaches in those terms. This is why I said in an earlier episode that the problem can be stated solely as an engineering problem so that software engineers without background knowledge in epistemology can work on it, though such knowledge of course would certainly help.

Now this all looks reminiscent of an old question that has been explored quite a bit in software engineering, the problem of how to write a program that can create another program based only on a few specifications. This has been explored in various fields. Genetic programming is a big one, and I also recently learned about something called program synthesis, and there are other fields. Now, I think these are all very promising and underrated areas of research, but as far as I can tell, they make two mistakes. First, they vary and select existing knowledge, but no new knowledge is being created. And second, they limit themselves to only work for very specific programs in particular target domains.

That means they actively avoid universality. For example, in genetic programming, tools have been built that can automatically design a circuit or guess the implementation of a mathematical function, and more even. But these are all separate algorithms that work specifically for target problems and nothing else. The way we phrased the problem here and the way we phrased the problem in the previous episode in terms of universal function replication actively encourages and insists on universality. We do not consider something a solution unless it can replicate, in principle, any function. So this function g that’s passed in here, this could be any function whatsoever. Now we may, at the beginning, demand that this be a pure function, as I said earlier, or a function that has some regularity. But in principle, this should work for anything because that’s exactly how we as people do it.

This also explains why there can’t really be several different target algorithms, or targeted algorithms rather, even genetic ones, because those put together could never make a universal explainer. So even if you’re in the field of genetic programming, which I think is the promising field here, because I think the way f would do this is just as we did. It would try a bunch of random things and make a bunch of random guesses and then eliminate those that didn’t work. And it’s going to just keep working its way forward that way. So we are in the domain of genetic or evolutionary algorithms here. It’s just that this has to be a unified algorithm that works for any function because just adding more and more things to a list, as we saw in an earlier episode, that doesn’t make that list inexhaustible. And due to universality, there is necessarily only one thing that can write any program, and that’s a universal explainer.

Now, I hasten to add, however, that I do not propose that solving this particular problem automatically leads to a fully-fledged artificial creativity because, as we will see in the next episode, there are big unsolved problems with evolutionary algorithms and with the very act of specifying an algorithm that would need to be solved as part of this endeavor. But I think that if you choose to work on it, you will encounter problems whose solutions will likely contribute to building artificial creativity. So my hope is that this presents an actionable item that we can work on specifically.

Again, the goal here is to write a function f. This could be written or evolved if we knew how to properly evolve programs. Write a function f that, given any function g, can replicate g’s implementation, create a hard-to-vary implementation of g without actually reading g’s implementation, only by what we did earlier, only by invoking g and trying random things out.

Thank you, as always, for watching. As I just hinted at, in the next episode, we will most likely talk about the problem of specification, which is related to the problem of sort of leaking knowledge into the explainer program that is supposed to create knowledge on its own. And we will explore some problems modern-day evolutionary algorithms face, and this part will be largely inspired by chapter 7 of “The Beginning of Infinity” called “Artificial Creativity”. A link to my Twitter profile, as always, is in the description. Tweet at me if you have any comments or questions, or leave a comment on whatever platform you tuned in. I’ll see you next time.


What people are saying

What are your thoughts?

You are responding to comment #. Clear
Markdown supported. cmd + enter to submit. You have free speech here. You’re responsible for what you write. Terms, privacy policy
Your real name is preferred.
This small puzzle helps protect the blog against automated spam.

Sign your comment with GPG to create or add to a public profile with all your comments.

Paste a detached signature of your comment.

              
Paste your public-key block if you haven’t before. You consent to your key’s contents, including your name, being displayed to the public.

              

Preview

Playing ~