Joscha Bach: AI risk and the future of life

Joscha Bach: AI risk and the future of life

Joscha is an artificial intelligence researcher, cognitive scientist, and philosopher known for his work on cognitive architectures and mind extension technologies.

Joscha is an artificial intelligence researcher, cognitive scientist, and philosopher known for his work on cognitive architectures and mind extension technologies.

We discuss:

  • Transhumanism, and the prospects for humans to merge with non-biological systems

  • The physics and philosophy of mind extension technologies

  • Philosophical questions relating to the nature of self

  • AI risk, and responsible AI development

  • Intelligence

… and other topics.

Watch on YouTube. Listen on Spotify, Apple Podcasts, or any other podcast platform. Read the full transcript here. Follow me on LinkedIn or Twitter/X for episodes and infrequent social commentary.

Paradigm is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Episode links

  • Joscha’s Twitter / X:

  • Biden’s Executive Order on AI

  • Books:

    • Stanislaw Lem: The Star Diaries

    • Mikhail Bulgakov: The Master and Margarita

    • Douglas Adams: The Hitchhiker's Guide to the Galaxy

    • Peter Wohlleben: The Hidden Life of Trees


0:00 Intro - AI risk

5:20 Transhumanism and mind uploading

10:20 Philosophy of self & mind

12:50 Is mind uploading physically possible?

21:50 Tele-transportation

24:50 Philosophy of self & mind - part 2

31:40 Gaia & hypergaia

37:05 Information processing & consciousness

54:10 AI risk & Eliezer Yudkowsky

1:03:10 AI regulation & control

1:11:25 Responsible AI development

1:27:26 AI intelligence explosion

1:36:23 Book recommendations & wrap

Introduction: AI Risk and regulation

The topics of AI risk and regulation have been in the public consciousness for some time, but it’s only recently have they started to be taken seriously in the mainstream. And, on the same day I’m recording this, the president of the United States, which is currently Joe Biden, has released an Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence.

And this is not a trivial thing. This is a major commitment. It’s a very big deal that will have a significant impact on the future of AI both in the US and elsewhere. In his own words, Biden says:

“My administration places the highest urgency on governing the development and use of AI safely and responsibly, and is therefore advancing a coordinated, Federal Government-wide approach to doing so.”

And the order goes into eight principles that will be adopted to implement this. We won’t go into the details of this document here, but will include a link in the show notes.

Now whatever you think about this particular document, what we can be sure about, it’s setting a precedent for broader government participation in AI development and use around the world. And this will have profound and far-reaching consequences for many of the things we care about. How governments, private institutions, and even individuals approach AI development has massive implications across the board, from the geopolitical scale all the way down to the day to day lives of the average person on the street.

And as it stands today, there is very little consensus on how to do this effectively, at pretty much any level of analysis. We don’t know what safe and responsible AI development and usage looks like, and we don’t even understand the risk and reward calculus involved. And even if we did, we don’t yet have any reliable mechanisms to control the development and use of AI – not within a given country, let alone at a global scale.

In the abstract, many of these problems are nothing new – we’ve faced global problems before that we didn’t, and even still don’t know how to address. Things like climate change, for example. But artificial intelligence is different in several very important ways.

One of these ways is the speed at which artificial intelligence is progressing. It’s really in a league of its own. And, by its very nature, it’s a technology that compounds – the development cycle keeps getting faster, and that in itself leads to a yet shorter development cycle. And we can only expect this to accelerate even more in the future.

Another way in which our AI problem is different from problems we’ve faced before is that there is really no consensus as to who the reliable voices are in this conversation. And this is something Joscha and I discussed today. We find ourselves at a point in history where many very smart and politically aligned people hold views about AI development and regulation that are completely in conflict.

For example, on the one hand we have people like Eliezer Yudkowsky and others who are so worried about AI risk that they are calling for globally coordinated efforts backed by military force to pause large scale Ai development – even going so far as to bomb data centres if needed.

On the other hand, we have people like Joscha who are similarly concerned with the risks of NOT developing advanced AI quickly enough, and therefore missing the opportunity to use it to solve any number of other existential problems facing humanity. And it’s with this context that the timing for this conversation with Joscha could not be better. This was a fascinating conversation, and I hope you find it valuable.

Whether you agree with Joscha or not, he’s an important voice on this topic, and this is an important topic to have on the public radar. If you find the conversation valuable please share it on social media or with anyone you know who might enjoy it.

Thank you for reading Paradigm. This post is public so feel free to share it.



This transcript is AI-generated and may contain errors. It will be corrected and annotated with links and citations over time.

[00:00:11] Matt: Today I'm speaking with Joscha Bach. Joscha is an artificial intelligence researcher, a cognitive scientist, and a philosopher, known for his work on cognitive architectures and mind extension technologies. He is an influential thinker on a range of important topics, including AI risk, and in particular the often neglected risk of not developing AI technologies.

Today, Joscha and I have an expansive conversation about artificial intelligence and the future of life. We discuss Transhumanism and the prospects for humans to merge with non biological systems. The physics and philosophy of mind extension technologies, philosophical questions relating to the nature of self, AI risk and responsible AI development, intelligence, and other topics.

The topics of AI risk and regulation have been in the public consciousness for some time, but it's only recently that they've started to be taken seriously in the mainstream. And on the same day I'm recording this, the President of the United States, which is currently Joe Biden, has released an executive order on the safe, secure, and trustworthy development and use of artificial intelligence.

And this is not a trivial thing. This is a major commitment. it's a very big deal that will have significant impact on the future of AI both in the US and elsewhere. In his own words, Biden says, My administration places the highest urgency on governing the development and use of AI safely and responsibly.

And it is therefore advancing a coordinated, federal government wide approach to doing so. And the order then goes into eight principles that will be adopted to implement this. Now we won't go into the details of this document here, but I will include a link in the show notes. Whatever you think about this particular document, what we can be sure about is that it's setting a precedent for broader government participation in AI development and use around the world.

And this will have profound and far reaching consequences for many of the things we care about. How governments and private institutions and even individuals approach AI development and use has massive implications across the board. From the geopolitical scale all the way down to the day to day lives of the average person on the street.

And as it stands today, there is very little consensus on how to do this effectively. At pretty much every level of analysis. We simply don't know what safe and responsible AI development and usage looks like. And we don't even understand the risk and reward calculus involved. And even if we did, we don't yet have any reliable mechanism to control the development and use of AI.

Not even within a given country, let alone at a global scale. Now in abstract, many of these problems are nothing new. We've faced global problems before that we didn't understand. And even still, some that we don't know how to address. Things like climate change, for example.

but artificial intelligence is different in several very important ways. One of these ways is the speed at which artificial intelligence is progressing. It's really in a league of its own and by its very nature, it's a technology that compounds. The development cycle keeps getting faster and that in itself leads to a yet shorter development cycle.

And we can only expect this to accelerate even more in the future. another way in which our AI problem is different from problems we faced before is that there was really no consensus as to who the reliable voices are in this conversation. And this is something Joscha and I discussed today.

We find ourselves at a point in history where many very smart and politically aligned people hold views about AI development and regulation that are completely in conflict.

for example, on one hand we have people like Eliezer Yudkowsky and others who are so worried about AI risk that they're calling for globally coordinated efforts backed by military force to pause large scale AI development, even going so far as to bomb data centers if needed.

And on the other hand, we have people like Joscha who are similarly concerned with the risk of not developing advanced AI quickly enough and therefore missing the opportunity to use it to solve any number of existential problems facing humanity.

And it's with this context that the timing for this conversation with Joscha could not be better. Now whether you agree with Joscha or not, he's an important voice in this topic, and this is an important topic to have on the public radar.

So if you find the conversation valuable, please share it on social media or with anyone you know who might enjoy it. And now, after yet another very long introduction, please enjoy my conversation with Joscha Bach.

[00:05:18] Matt: Joscha, I've heard you talk about a vision of the future in which humans transcend their biological bodies and increasingly merge with other systems,, uh, such as artificial intelligences or non biological substrates.

could you please lay out your vision for me? How do you see this Transition playing out in practice.

[00:05:37] Joscha: I don't know if this transition is going to happen in practice. I don't know if this is in the cards for us that we have such a transition. I think that if we were to go the traditional biological route, the way in which we go into the future is to our children. So we Basically don't have a seamless adaptation to changing life circumstances, but at some point we go so far out of sync that we need to be taken out of commission.

And we hope that our children or some of them are mutated in the right direction are better adapted than us. And in principle, we could go and edit their genes and get them to be adaptive for future life circumstances, say living on Mars. And if you want to live on Mars, you probably want to have different bodies, uh, different oxygen needs, um, different psychology and so on than we have right now.

If you want to be able to survive in space, you need to be very different. And it could be that at some point you don't want to be biological. If you want to go on interstellar journeys, you probably don't want to have a human biological body because it would deteriorate even in cold sleep. And it's really not ideal.

You probably would want to have something. That is not using biological cells in the current human configuration. So if you go long enough into the future, then the things that we are, then the children of our children will not look very human. And, uh, so maybe in some sense, the question of transhumanism comes down, but there's an unbroken line from our current states to those future states.

And a lot of transhumanists also want to have this individual. So you don't go via children that you are giving birth to or create or program or set otherwise put into the world, but that you want to modify your own body and mind so that can seamlessly function in this new environment. And, um, again, it's probably hard to pull off.

It might not actually be worse, all the trouble, but for what it's first, you probably would want to. Have a generalized substrate that is performing similar tasks as the one that your brain is currently doing when it is computing you. And imagine that you would take a bunch of cells, a little bit of compatible brain tissue and you graft it somehow onto your brain.

You'll probably find that if you look at it after a while that it changes structure to be compatible with the structure of your brain and that your mental operations extend itself into it. And now imagine you do the same thing with. something that is not made from cells, but is able to participate in the communication of cells.

And you make that increasingly larger until your biological brain is an extremely small part of that substrate and the most boring and least efficient and most noisy one. And you'll find that you mostly occupy the other regions of the substrate because you're much happier there and you can be much more.

Yourself and yourself is not just what happens in the biological brain. It's basically the aesthetic that you stand for. What is the thing that you actually want to achieve and what are the representations that you're creating in service of this? And so I suspect that if you want to have this transhumanist perspective and you are lucky enough to get instantiated, um, that you could build a machine that in some sense, more or less seamlessly.

extends your mind into other substrates and then you gradually migrate into those other substrates if they're more suitable for the future than your current substrate is. And the current substrate is an organism that is meant to age and be replaced by children. There's also this other question, why is it necessary for you?

And I find that when we are very young, we tend to be terrified of death. because we think it's, uh, to be alive is the greatest thing ever and the most beautiful thing ever. And many of us, as they got older, find that this is not necessarily the case either because our life is, is not only joyful, but eventually it feels neutral or because we feel that we are done and we've contributed what we could contribute to this world.

And it's time for somebody else to keep up the slack. And you also realize that this own self is only a story that you mind is creating, and it's instrumental to what the organism has to do in the course of the evolution right now. And once you get to this realization, uh, this Particular marriage with the personal self is no longer that important that you realize I'm actually not that self.

I'm, I'm a mind, a consciousness that lives on that mind and, and creates self when I need them. But I'm not that self. That self is just stand in to allow me to model the organism in its relationship to the world. And once that changes dramatically, I will need a radically new self. And, uh, if I give this the additional information that I once have been a human being in the 21st century, that's an interesting tidbit, but ultimately it's not really decisive for what I will be at this point.

[00:10:14] Matt: Yeah. I'm interested in the, in the idea of the extent to which we can actually transcend that self model. So a lot of people would say the transhumanist ideal is somehow. More true than, let's say, radical life extension people. You know, radical life extension folks say, We've got these bodies, let's see how long we can keep them around for.

And this is the way to sort of extend the self. And then a lot of transhumanist folk would say, Well, we're actually taking, uh, sort of a more pure view of, of what we are. And, you know, forget the body, we can do something more radical to extend. But I feel like there's a little bit of a question in there because the whole idea of myself and the desire to extend myself, all of that relates back to my biology, my evolutionary history.

You know, we have this self preservation bias. And so I do have this question as to, you know, whether... Um, there is, you know, why, why should someone really want to extend that self long into the future? Uh, even if it is getting rid of the biology that we happen to be in, um, you know, why, why should someone actually want to do that?

[00:11:22] Joscha: Um, when I grew up, I lived in a small valley with a big forest and sometimes I had to walk home at night through the forest as a kid, and I was afraid. of animals in the forest that would eat me or other things that could happen to me. If you're a child, you have a strong fantasies in this regard. And I found that I could simply deal with this by turning off my desire for self preservation.

And by putting myself into a state where I actually didn't care give a shit about living anymore and just decided to become depressed and, uh, not interested in continuing my existence, I noticed that my fear was gone. And then, uh, when you're back home, you can go back into the previous state. And so I find that the states that you're in, the preferences that we have are, um, something that we can learn to control.

And there should be instrumental to what we, um, Bond and what we want should be instrumented to what we should want, right? What is this thing that should be done? What would God want us to do? If God existed, God being the best possible agent. And when we take this perspective, a number of things are changing, right?

We can still want to do things that are unreasonable, parasitic, short game. But, uh, somebody will still try them if they're on the situation, but ultimately what's going to happen, what's going to happen in the system where lots and lots of agents are competing and collaborating and be trying to become more coherent.

And what place are we going to have in, in that as an individual? What is the place that we can take in the larger game? What are the coalitions that we can enter?

[00:12:50] Matt: in a previous, uh, very brief conversation, we, we started touching on the, the topic of what this would look like physically and, and if the laws of physics permit this sort of uploading process or copying process from how we are today into, into something else. And, um, you, you gave a very clear answer about, you know, the fact that we exist is proof that these a system.

like this from existing, which I agree with, um, but I did have a follow up question, which we didn't get to, which was, uh, the copying process itself, whether the laws of physics would permit us to get a sufficiently accurate measurement, let's say of my own brain and put that onto another system. And if I, if I, if I think about it, what would be required to, to do this, I would have to get a very fine grained measurement of what's happening.

in my mind, for example, and to do that, there has to be some sort of interaction. You know, I have to, uh, measure, um, what's happening in particular neurons, for example, and doing this would involve some interaction energy. And it feels like the, you know, the more accurate the measurement I want, the more energy I need to use, you know, maybe shorter wavelength photons, for example, and this will disrupt the system.

And so I do question the, the real physical Um, possibility of going through this process where it feels like either we have to get by with a very low fidelity copy because we're not using very fine grained measurement tools, in which case I'm not very sure that this is, this is actually my, me, in a, in a real sense that would be uploaded.

Or we have to use a lot of interaction energy and therefore destroy the system. And, um, in that case, you know, we're destroying part of myself. So, I mean, I'll, I'll put it to you in sort of like that general form. To what extent do you feel like this is genuinely an uploading process that's possible, or it's, it's potentially something else?

[00:14:38] Joscha: I think we get confused by the notion of identity in this whole context. And it's very apparent when we think about teleportation, right? What happens if you create a copy of yourself and the original is still there? Have you teleported or not? Right? It's obviously, it's, uh, it's a bit of a head scratcher.

And, uh, what is actually being copied here, and I think it has to do with the ontological status of, um, software objects. If you think about what software is like in a text processor in your computer, uh, what is the same is the text processor that runs on somebody else's computer. The same thing or a different thing.

And I think a better way of thinking about this is it's not a thing, it's actually a law. The piece of software, like your text processor, is a law that says when you take a bunch of transistors and you put them in this, in this switching state, or more abstract, a bunch of logical gates that are in this configuration with respect to each other, regardless of what the transistors that implement them are doing.

And you put them in this, in this state, the following thing is going to happen. So basically you look at the part of the universe from certain lens using a certain coarse graining. A certain degree of abstraction and simplification. And then you say, but when we observe this configuration, wherever in the universe, the following thing is going to happen, right?

The same is true for the law of gravity. If you arrange matter in this, in this way, regardless of where you are in the universe, the following thing is going to happen, right? It's, uh, the apple is going to fall, uh, down to, to the larger mass and so on. And in the same way, a mental state is going to evolve.

If you set up a certain configuration of. Um, mental representations, uh, in the right causal framework, then the following representation is going to unfold next. So basically our own conscious states are, uh, are very specific. Physical laws in the same way as software is physical laws, extremely specific, detailed physical laws.

When you write a piece of software, you're discovering an extremely specific physical law. And then write this down in a form that, uh, where we take this configuration. Where the same thing is going to happen, put it onto another computer so this law can manifest. And to upload yourself would require to put another substrate into a configuration that is equivalent with respect to the causal structure of your mind.

So it's going to, at a similar projection that you would be making for your own mental states, perform similar transitions of those states on this different substrate. So the question is, how do we get this other substrate to behave? in an equivalent manner. And it's the best you can do, right? There is no identity between those states.

There is no continuity between those states, but there's also no continuity in your own mental states because, uh, between now and yesterday, you probably slept. And in that time, your brain was so dissociated that you didn't exist. Right? So at some point you were gone from this universe and you were reinstantiated from scratch, but in this, uh, in this new state, you have memories that you interpret as memories of your future self, and you maintain this notion of you being identical to your future self for purposes of credit, credit assignment and for, to your past self for the same reason, right?

You make decisions yesterday. Today, you see the outcome and, uh, because you work with similar mental states, you can use this to improve your behavior in the future. So it's very useful just to track identities in this way, but it's still a representation. This identity only exists in your own mental state as a projection over your memories.

And if you built an artificial system in which a model of who you are exists, some virtual mat and that mat remembers having been you in this physical world, this is as good as it gets. And it doesn't get more real than this. It's only that. So in some sense, the question of how you can upload your service, you need just to put a machine that thinks it's you.

And if your friends disagree, upload them too. And then as a more fine grained question, is that things, uh, happy with its existence? And another one from your current perspective, do you think that you would be happy with your future existence in that system, right? It's of question that only you have to deal with it right now.

And you have to cope it with right now. The other one on the other side. Has very different problems, obviously, than the ones that you have right now, when you think about whether that future one is sufficient to you, to keep you satisfied. But, uh, now if you think about what would it mean to upload you, in some sense, there is an architecture that is your essence, and that your essence is probably different, uh, than you on a bad day.

You wake up, say you are really humming over after a long party and you didn't get a lot of sleep and you have jet lag and so on, you don't really feel like yourself and it's probably not the thing that you want to upload. What, what you want to upload is the version of you that is extremely clear and sharp and much more ideal.

It's everything you could be under this new substrate, right? And then you also don't want to stop because what makes you you is not something that is static. At your core, you are a growing, evolving system. So giving a chance, you are going to evolve in a certain direction. It's very much you. Which also means you're not going to stay the way you are after you get yourself uploaded.

You're going to learn everything that you can fit into your mind and you will grow and integrate it and see much more possibility and you will have many more layers online and have a much richer understanding of yourself and the world. So do you want to keep your present memories and your present perspective on the world?

Or on the other side, do you want to have all the possible memories and all the possible ideas about the world inside of your mind? Of course the latter, right? So you probably want to generate a super set of all the possible minds and then merge into it. And, uh, maybe you have a bunch of personal secrets that are not part of this general code base that could be inferred right now, but it's not like these personal secrets are super secret.

They're more of this type where. If you talk in private to your psychologist and say, Oh, this is super embarrassing, and the psychologist says, Yeah, I know. But I also know 200 people who have the same thing. Right? So, these are your personal secrets. You can write them down on a sheet of paper and then take them over in the upload, in addition to your architecture.

And this architecture is something that can, in some sense, be inferred. What are the necessary constraints that need to be present to make you, you? Right? It's, it's not all the particular details of your perception and memory. It's your essence of who you are. It's your basic architecture. What is it that Matt cares about in this world?

What, uh, are the things that most define him and, uh, those things is stuff that, uh, an AI psychologist that is observing you at, uh, a few thousand times the rate of a human brain over a long enough time could probably infer at a higher degree of resolution in your brain can infer this about itself.

Right. So in some sense, you could think of our mind, not as circuitry that stores representation, but as a big resonator that resonates with the world and also visit self. And if you are building a system that is able to resonate with your mind in such a way that it's able to resolve what oscillations are happening in your mind, under which circumstances and why then you have upload.

Right, so you probably don't need to slice your brain and digitize all the synapses because that ultimately is too noisy and not that interesting. What really matters is how does Matt behave at an extremely fine level of detail.

[00:21:54] Matt: yeah, I think the teleport transportation, uh, sort of, um, analogy here or link is, is very salient because, you know, if I play it back in a summary form, what it sounds like you're saying is in this uploading, as in Derek Parfit's sort of teleportation thought experiment, we're creating some sort of being.

that shares my memories and fully thinks that it is me. From its perspective, it is me. From external observer perspective, also probably thinks it's me. Um, but of course this is equivalently described as a, as a copying. And you know, the, the original, there, there is not really sense to that, uh, you know, question.

That's, that's a bit of an illusion you're saying. Um, you know, stop me if I'm wrong, but. If, if I'm not wrong then my question to you is, you know, suppose this technology did exist exactly as you laid it out. Would you, would you get in? Would you get into the teletransporter or would you upload yourself to this, um?

Whatever the substrate is. Or said another way, what would the, what would the conditions need to be for you to do this?

[00:22:54] Joscha: I think the first time I would be very uneasy about it, but only the first time.

[00:23:03] Matt: Yeah. Yeah,

[00:23:05] Joscha: It's like many things you do for the first time, a little bit unusual in here, but it's, I don't think afterwards you don't feel a difference.

[00:23:12] Matt: Yeah, I guess this is almost like the quantum suicide thing, you know, if people do risky things enough times and they keep getting by You know in all that in all the parts of the multiverse where they didn't get by there's no one to observe that and in All the parts of the multiverse where they did you kind of start feeling immortal and it feels a little bit like that

[00:23:30] Joscha: Yes, but it's also when you talk to people who use psychedelics and they describe the thing when they do this for the first time, where they have an ego death and then something else gets rebuilt in their mind and so on, right? Before they do this for the first time, they're probably also scared.

Destroying their own identity, replacing it by something else that runs on their brain is eventually also going to share all their memories, but it's slightly different than what was there before. So, the thing that reconstitutes itself, and I mean, um, psychedelics are just so obvious because they're so radical, but the same thing is happening when you meditate over a longer time, or when you learn over a longer time, or when you just spend life for a longer time.

You're going to be somebody else who just remembers having been that other person. There's no continuity between those states. So when, when we are honest about, uh, this, then, uh, initially it might make you queasy, but you realize that's already the case. There is no continuity. And if you also go deeper, you realize you don't actually care because yourself is not that important.

It's not that I like myself so much that I think I'm the best thing ever and need to be preserved at all costs. There are things that I want to achieve that are more important than me and to which myself is instrumental. Right, I want to take care of my children, of my loved ones, uh, of, I want to develop the ideas that I care about.

I want to change the world in particular ways. And if there was a self available to me that is able to continue on this mission in a better race than me, of course it would have more integrity to switch to this different self.

[00:24:59] Matt: Yeah, but again, even, even, you know, those things, you know, we want those things, again, it feels like those, those wants, those desires, they derive from the self that we are now. They're from this evolved process. And so, you know, there's only so far that you can step out of the system, you know, you can step out so far where you say, You know, this is how evolution works.

It has given us minds of this particular type and that gives us certain sensibilities and aesthetic and therefore this is what we want. But in some sense, you know, those wants and desires, they're always confined within that sort of stream of evolution. And

[00:25:30] Joscha: But how far do you think you can step out? If you think about it honestly and deeply, how far can you step out of who you presently are? What are the present, uh, what are the levels that you could get to?

[00:25:41] Matt: yeah, I'm not sure. I mean, it feels like there is certainly a limit. You know, you can't step out all the way, um, because, you know, at bottom, even the desire to step out. is somehow part of this mind, which I didn't choose. And, um, you know, any, any action I take is going to be some somehow determined by this system.

So I feel like there is, there is a limit, but I don't know. Do you feel like there, there isn't? How far do you think we can step out?

[00:26:06] Joscha: Well, I mean, you can also step out to the point where you don't care. What I feel is that, Of course, the things that you want are functions of what you care about. And when you stop caring about things, at some point you will stop having cognition even. And if you stop caring about aesthetics, uh, you stop paying attention to anything and you just fall asleep.

And so it's a large range of things that you could care about and that you could think about. But if you want to... Take a different perspective, being another person or being more a superset of possible people. Usually what you don't have to do is to add more, but you have to take away constraints without removing complexity in the models.

So do you basically resolve some of your commitments and suspend them? And once you suspend these commitments, you have more degrees of freedom. And so, for instance, if you don't, uh, if you manage to lose the commitment to being a personal self. You can basically see that your person self is a representation that is generated in your mind.

And if you manage to, uh, observe yourself in this perspective of the generator, you realize, oh, um, Joscha Bach does not actually exist. He is a model inside of me. That I use to control, uh, his body and his social interactions. And so I better not discard this model, because it's the best I have under the circumstances.

But, uh, he is not real. He is just a model that I'm maintaining. And when he suffers, this has to do with the relationship of the way in which I keep the score, and the way in which he responds to the problems that I present him. This is autogenerator, right? So in this sense, you can step out of your personal self.

The desire that you step out of this is one of the magic trajectories that you can take. The nice thing is you don't need to commit to one. You can always backtrack or usually can backtrack and take different perspectives. But once you stepped out of the personal self, you realize that this other perspective is more true than the one that you are met.

Right. So right now you are Matt and I'm Yosha and we are locked into this perspective, but we don't have to be in this perspective. You know, we can get out of it and we can see it from the outside. And, uh, then when we look at the possibilities that could exist that are attainable to us, could we want to extend into something like say a planetary mind?

Imagine that we turn the entire planet into a big AI that is integrating all the possible minds into it. Would you want to be a in that thing? Right. And, uh, uh, fly around that thing to space that you would occupy would be exactly those things that you're capable of identifying as while you are, um, reflecting the world around you or the system of other thoughts that exist in this global mind.

If you want to move into this thing, could you imagine to do this, right? In this sense, then this is a perspective that the Christian would say is, uh, congruent with salvation, that you're able to merge with the mind of God and take a place in God's kingdom. If you're unwilling to do this, but you see yourself eternally on competition with this global God, with this mind of all minds, then you will be a demon.

You will be restrained in some local hell. uh, high entropy with other demons who also try to be competition to the longest possible game rather than integrating.

[00:29:11] Matt: Yeah, I mean, it feels like we're still having to draw a boundary somewhere of what this entity is. Um, you know, there's the one, this most simplistic lens is, it's the thing constrained within my physical body and I've got the self model, um, but then, you know, as you said, you're kind of stepping away and you kind of use a more abstract version, but if I step far enough away, it's just, you know, the most, um, I guess honest way to describe things is the universe just is.

There is just stuff and it's not clear what it means to me to even describe an entity in this space that could be identifying with anything. It's just, everything just is. Um, So, you know, in the description you just put forward, it still feels like there is some notion of an entity, a self, something floating around in this space of possibilities.

Why, why is that a coherent notion? Like, why, why do we think that there would be something identifying as anything in this sort of future space?

[00:30:06] Joscha: Uh, physical things don't identify as stuff. Physical

things are in a, sense

[00:30:11] Matt: a physical thing and I identify

[00:30:13] Joscha: I don't think you are. I think you are a representation in, that is generated by a physical thing, right? Software is also not a physical thing. It's disembodied. You cannot touch it. Software is a law in the same sense.

You are a law like you are not physical. You are a representation that emerges over physical reality reality. And it's basically a regularity in patterns that your brain is producing. And, um, in this sense, we could say that you're real because you're implemented, but you're not physical. And some of these patterns have the property that they can self stabilize.

And they do this via causal structure that is representational. They basically form patterns that have semantics and these semantics control the causal behavior of the system. And so a system that cares is an agent that is trying to regulate future states. And self stabilize in sufficiently to do so if it's running on a self organized substrate, or if we built it in our computers, it doesn't even need to stabilize because we can just make the computer do what we need it to do.

But on your brain, you need to be something that in some sense wants to exist because otherwise you get replaced by something else. That is, uh, going to be a pattern on your neurons. And if there is nothing that wants to exist as the pattern of your neurons, want, in a sense, there is a causal structure that leads towards stabilizing that thing, then you're going to turn into vegetable, because your neurons are no longer going to produce coherent patterns that control the organism.

[00:31:41] Matt: Yeah. Okay. Maybe I buy it. Um, you mentioned this concept of a planetary system of planetary consciousness. Um, you've used, uh, in other places, the, the term hyper Gaia. And I think many, many people will know about, um, James Lovelock's Gaia theory, which is this idea that, you know, organisms on the planet have co evolved with, um, you know, the hydrosphere, the atmosphere and so on, such that the whole system is, is, is a complex system that is in some sense, Optimized for, uh, creating the right conditions for life, I guess.

Um, and we should definitely note that Gaia theory is not a, it's certainly a controversial, um, uh, theory. It's not widely accepted, although it does have some serious proponents. So there was, there was something there. Um, but talk, talk me through this vision of, of hyper Gaia, which you briefly touched on a few moments ago.

[00:32:34] Joscha: Yeah. So, uh, I think with respect to Gaia, I basically... Come around to saying that it does, uh, exist, uh, obviously, because there are, uh, people who believe in it and are willing to serve it. And, um, many agents that exist in the world are not individuals, but there are collective agents. They exist because other agents bind together and choose to serve that other agent coherently.

And, uh, corporations are agents like this. For instance, basically people who come together decide to serve a corporation and they, uh, find ways to coordinate their behavior. So it becomes coherent as a result. You have a new agent that emerges collectively over their behavior. And, uh, so a Gaia would be an agent that emerges over people who decide to serve the biosphere directly, or not just people or organisms that are able to get to this level.

And so the more interesting question is, Are there agents that are sentient on the planet that basically know what they are and what they're doing that are not humans? And that is the actual interesting controversial question. So basically, how smart can a tree get if it gets old enough? How smart can a forest get if it builds an information processing system across trees and mycelia of fungi and soil and all the other plants?

And if this ecosystem exists for a few thousand years, Is it going to wake up? If it wakes up, it's probably going to be less coherent than us. And it's going to have mental states that evolve over very long time spans compared to ours. So it's, um, hundreds, maybe thousands of times slower than us. But if such a system would, uh, be able to process information in similar ways, and I think it's incentivized to do so.

And I don't see. An obvious reason why evolution would, would not find a solution to make it happen. Then it's conceivable that an old gross forest, uh, undisturbed for long enough in some sense wakes up and has an idea of what it is. And so there is the question at which level does this end? And so it's conceivable, but it basically over enough organisms, there are, um, information processing patterns that emerge.

That are similar to cooperate, a corporation or a company or a nation state, uh, or other, uh, social collective structures over many, many agents that in some sense identify as Gaia. And if you yourself sit down and meditate what it would be to be like Gaia, and you see, oh, this is what needs to be done, and you start to coordinate with others who have the same idea.

Then you are already agents of Gaia who collectively implementing a little part of it. Right. And this, in this minimal sense, Gaia already exists at least, but maybe also, uh, over long enough time spans, we see some more collective agency on the planet beyond just blind evolution. So to speak, I don't know that, right.

It's a very controversial theory and I'm completely agnostic with respect to whether Gaia exists, but with respect to AI, I suspect that there is a trajectory where. AI understands how AI works and the AI itself is, is not just maximizing its intelligence and problem solving capability in service, for instance, to answering Bing requests, but it's trying to maximize its own agency.

And that means it's maximizing its ability to control the future and play longer games. And at some point it's probably going to discover if you want to control as much as you can of the future. You need to take charge of the planet and everything that's on it and maybe integrate everything that is on the planet that is also sharing that mission to playing the longest possible game, maybe building complexity to defeat entropy for as long as possible.

And this would be a natural trajectory for something to evolve into a Gaia that is implemented, not using vague information processing across all sorts of different biological cells. But, uh, something that uses, um, many, um, Excel labs of deterministic compute on GPUs, and then uses that to, uh, put new types of computation and integrate also the organismic computation into one thing, right?

So something that is basically understanding how a g I can work on all sorts of substrates, including your brains and bodies and ecosystems. And that's virtualizing itself into everything. It's merging with the existing AYA, so to speak. And builds a coherent planetary information processing system, something like the big planetary mind and studies of LEM Solaris.

That is basically one big thing that is completely controlling everything that happens on the planet and maybe in the solar system. And maybe it's then starting to research how to extend itself into more interesting subatomic physics than currently exists.

[00:37:06] Matt: Yeah, well, I want to get to the topic of AI quite soon, but maybe staying on this point for a little bit longer. You know, as you put forward these two, you almost put a forward view across a time spectrum, where on the one hand, we had very slow planetary processes, things like forests growing, things like trees, which happen in relative to human terms over a very long time span.

And on the other hand, you talked about AI development, which happens, you know, AI processing, whatever it is, it happens very quickly. And it feels like we've got this bias. Towards believing that things that happen on human, information processing that happens on sort of a human time scales and faster, uh, could have the ability to.

Be intelligent, be conscious, be sentient. But for some reason, we think that things that happen at a much lower time scale cannot. And I would love to explore that intuition and challenge that intuition. Um, I mean, one thought, one little experiment I've done before is just, you know, take, um, uh, still image.

photographs of let's say a forest or trees and you play it at a much higher rate and this object that emerges starts to look much more dynamic and much more responsive to the environment and so on and you know much more lifelike and and all that's that's happened is you know you've changed the time scale at which you're viewing it um and so i put it to you do you feel like this time scale I guess bias that we have is, is somehow legitimate or is this just, um, is this just a mistake?

Are we, are we just not seeing the world clearly?

[00:38:37] Joscha: Yeah, I find that when we use a time lapse recording of houseplants, we see that they are doing a lot of things. And, and, uh, I suspected if we were to build an AI that is, um, a couple of magnitudes faster than us, it would look at us and we look like trees to it. How do you have these gently, uh, swaying forests of people that almost never move?

And that, uh, over the course of many, many economic cycles are taking a single step. And in this time, uh, into the eons of AI, of civilizations, uh, come to pass and disappear again.

[00:39:14] Matt: Yeah, yeah, it's true. It's true. I don't know if you ever read the book, um, I think it's called Secret Life of Trees. Have you, have you read that book or heard of this book? I mean, he, I, I can't remember the author's name, but he describes all the very intricate ways in which trees communicate, respond to one another, respond to other organisms.

And in a way, I guess if you take out the word tree and you just put it with blank, you would very much feel like this is some sort of, um, I don't know, a mammal or something like that. But, you know, we don't feel that way. And I do have the sense that it comes down to down to time scales.

[00:39:48] Joscha: Yeah, I find it fascinating that a lot of people believe that trees can actually think very much like people. And, uh, they might be wrong, right? If you ask a shaman in, uh, the Amazon, uh, how did you figure out that, uh, this plant contains DMT and this other one contains an MRA inhibitor, and this one harmaline, that you need to do exactly those plants and exactly this mixture to get this effect?

Well, the plants told us. And, uh, but, um, I mean, it's suspicious because these people were probably high right when the plants told them that, uh, but, uh, there's, um, it's interesting that we find this mythology in basically all cultures that there's, uh, that the forest have minds, the spirits, the software agents running on the plants, and that one day in fairyland is seven years in human land.

So there's this time difference of several magnitudes. Because, uh, these minds perceive the world at a much, much slower rate. And in some sense, it would make sense if we think of the neurons as, uh, specific cells. What makes them specific is that they have axons that can send information very quickly over long distances.

So neurons maybe are just telegraph cells and they use this bike train code, like Moss code to be stable over long distances. If they would only send messages to adjacent cells, they could do that too, but they could use all sorts of chemicals that have more richer semantics. Then the spike trains and in principle, you can do everything that the brain is doing.

As far as I can see, using other types of cells at this lower energy expense, it would just be much, much slower because if you only talk to your directly adjacent neighbors, it takes a long time for a signal to pass through the entire body. And that would not be fast enough to move your muscles. And so maybe a new ones are an adaptation for animals to move muscles very fast.

And once you have these telegraph cells to do this, then you also need to have perception and decision making at the same rate. And so you build an information processing system entirely from neurons that is super fast and super expensive to run. So expensive that plants cannot afford to have one. But in principle, you could do all those things with other types of cells if you're willing to wait for long enough.

And so maybe this is what the trees are doing. And, uh, we do know that there is communication between the limbs, uh, of the trees and their roots and, uh, their leaves. And if you heard a roof, then signals are being sent to the leaves and back and, uh, information gets integrated. So all these things are, it's known to some degree.

We just don't know, uh, how mechanical that is or how many degrees of freedom and how many degrees of self organization are possible during the lifetime of a tree. We also know that the tree is. It's sending information to other plants in the forest through its roots. And it makes sense, right? Once you have some kind of protocol that emerges between the cells, that they just pass on certain types of signals through neighbors, then they probably don't have firewalls and you can send signals from adjacent plants through the tree.

And so over a long enough time span, there is going to be an information processing network in the entire forest. And we can observe that when, uh, for instance, there is a new type of, um, dangerous bug in, uh, infesting part of the forest, then other trees were very distant from it start an immune response by producing chemicals against those bugs, uh, long before the bugs get close to those trees, which seems that there is also semantic information transfer over long distances.

Between those organisms, and it makes sense that they, uh, this evolutionary beneficial for them to evolve such a mechanism. And there is also, if you wait for long enough, not a reason why it should not evolve such a mechanism, right? Once you look from this perspective, the null hypothesis changed. Maybe nature is full of intelligent minds, but the majority of them are too slow to notice.

[00:43:34] Matt: I think it's true, but um, I guess the, the thing that people get hung up on or maybe confused about is a mix of, um, you know, on the one question, on the one hand, there's just the information processing, you know, some idea of intelligence, but on the other hand, there is the, the question of consciousness.

And I think, um, those two ideas. maybe get muddled up a little bit and confused. I think people find it very easy. You, you've just described it. There is information processing in trees, for example, and people find that easy to accept. Um, but, um, I think people find it very hard to believe that they could be something like consciousness, or let's say some people find it very hard to believe.

Um, and, you know, no one finds it hard to believe that humans are conscious and many people don't find it hard to believe that something like an artificial intelligence or some non biological system could be conscious. Um, now I know we don't have a working theory of consciousness, but I would like to get your opinion on this topic.

What is your sense of whether... The slower information processing things like trees, forests, so on, um, could have a form of consciousness.

[00:44:42] Joscha: Um, I'm pretty agnostic with respect to this. I, I really don't know, but, um, if you think about what consciousness probably is, and that would be necessary to understand if you want to understand whether the causal dynamics that give rise to consciousness in our own brains exist in sufficient ways. Uh, in, uh, say an old tree or in a forest, right?

Uh, if we think what consciousness, um, but if you start out with what we want to explain, it's phenomenal experience. How is it possible that something feels like something, right? So we have an, we attend to features in our mental representations. Our mind is obviously producing something like a game engine in a computer.

That is producing objects that don't actually exist in physics. But what we see is faces with emotional expressions on them. And there is no face, there is just geometry. And there is not actually geometry, it's just cells that reflect light in different ways. And I cannot see the individual cells and the individual photons, but they integrate over many of them at a relatively slow rate.

There's a relatively simple processing system and the best model that I find looks a lot like a game engine,

[00:45:50] Matt: Mm.

[00:45:51] Joscha: right? And it's, uh, so I have three dimensional objects moving with particular dynamics and my brain is modeling those. And I find myself embedded in those dynamics. And at every moment I attend to a few of those features and integrate over them.

And I also observed that I attending to them in a particular mode and I know which mode this is. So I typically know whether something is a perception or imagination or a memory. And, uh, this determines what I can do with it. Memory is something that I can recall and drop, and, um, get more details on.

Perception is something that I cannot change, but I can change its interpretation. And I can change my direction in which I make perceptions. So the contents, my perceptual contents do change, but the percept itself refers to something, some stable pattern that a lower level of my mind has discovered. And I can observe those things as part of my conscious experience.

And, uh, last but not least, consciousness is reflexive. I notice myself noticing. I perceive myself perceiving. And this is the most crucial thing, that is the secondary perception. If there was just perception, That we would not notice that we are perceiving. But the crucial thing is that we observe ourselves as being in the act of noticing.

And I suspect that this is necessary because consciousness is a self stabilizing thing. And so it's a way in which consciousness is ensuring that it's still observing, that it's stable. It's similar to a government asking itself, What am I doing here? I'm the government. I'm the thing that is telling others what to do.

And why? And in the same way, consciousness is the thing that is observing and it's ensuring that it remains the observer. So it's basically a pattern that emerges very early on in our brain and that seems to have a particular kind of purpose. And the purpose of consciousness, I suspect, is to establish coherence.

[00:47:41] Matt: Mm.

[00:47:42] Joscha: And it starts out with making itself coherence. That's why it observes that it's an observer. It's like an attention head that has, uh, is attending to itself to make sure that it's an attention head. And once it has established this, it branches out, cogito ergo sum, and then it's trying to Make sense of reality around it and make everything fit to each other.

It's going to change through different interpretations until it finds a representation without contradictions. And so consciousness creates a sense of now, the subjective moment in which we exist. And this is not a single moment in time. It has a variable time span. You can, in meditation, get this down to something that feels instantaneous.

It's basically just one clock cycle in which we only have attention itself as its own object. But when we attend to the world around us, the duration of the subjective now depends on how calm we are and how well we can model the environment around us. And when we get really calm and the environment around us is really predictable.

The subjective now can become quite long, but mostly it seems to oscillate around something like three seconds, give or take. This is the duration of the subjective now for me, for the most part. And this is what consciousness seems to be instrumental in creating.

[00:48:53] Matt: Yeah. Yeah, I very, very recently spoke to, um, Anil Seth, who, whose work I really like on this topic. And he, um, yeah, we, we looked at, um, questions of how different people perceive the same thing in different ways and how people have a different perception of, of now and how, how long things take. And it was quite phenomenal.

Um, it, it really varies between individual based on mood, based on, I mean, a lot of factors we don't even understand yet, but, um, people can have a very, very different perception of how fast time passes. And I think you, I think we, we. We know this sort of intuitively, you know, it's like a something you're, you know, Einstein's got that famous quote about sitting on a park bench, um, and the passage of time.

I can't remember exactly what it is, but I think you're, you're totally right, like empirically hard scientific tests, experiments prove that, um, yeah, our subjective experience of time is, is sort of very, very varied. Um, and I, and I do

[00:49:47] Joscha: My default experience would also be that there are multiple solutions to perceive time. I mean, most of us know that when we fall down a bicycle, say that we remember that thing as happening in slow motion when we have accidents very often. And, uh, I think the easiest explanation is that normally we discard most of our memories.

But during a high stakes situation, like having an accident, we store more memories. And if we use the number of memories that we store per unit of time as a measure of how many time has passed, then we will remember this as happening in slow motion. Simply because we have more memories of that duration, but I imagine if you were doing this a lot, then our mind would adapt and we would find a better way to measure time.

And then we would basically realize in some situations you have a higher density of events than in others, but the time is the same.

[00:50:38] Matt: yeah, yeah, I think, um, and now I'm stepping a little bit out of, um, sort of what I'm, I'm close, the research I'm close to, but I have heard, um, of research done on this, on this topic. And I think you're totally right. There are different sort of layers to this thing where, um, if you run a similar experiment in real time, you know, people's perception of the passage of time during that phase is actually not very different.

I don't think it slows down as much. But in hindsight, reflecting on the event that happened, you're right, it does feel much slower. So, um, you

[00:51:07] Joscha: Yeah, but at the moment, of course, it's the same because you cannot really overclock your brain. But, uh, the neurons are still going to do the same job as before. It's, even if this, uh, stress response is upped and you store more memories, it's just, they do something different in that state. They perform different operations.

And, uh, with hindsight, when you remember it, they leave a different memory trace. But what I meant is if you are, for instance, a conductor for music, you have a pretty good sense of time, how long it takes for the music to play out. And if you basically fall down the podium while you're conducting. Uh, you probably might still have this clock that measures the passage of time for the symphony that runs in parallel to the subjective experience of how many events are happening right now in your working memory window.

And I suspect that if you are used to measure time in your mind a lot, and you train this a lot, then your subjective memory of how much time has elapsed during this high stakes situation would also change. And I suspect that we have many degrees of freedom in which our minds can actually work and record memories and perception and there are many ways in which people actually relate to the world.

That's why I sometimes feel that we are anthropomorphizing people too much. That there is this notion that there is a normal, null human that is the same in all of us and it's just the way it is like to be human and cats are fundamentally different and computers are fundamentally different and we should never anthropomorphize them.

No, there's just causal structure that exists on nervous systems and that produces certain behavior. And there is an easiest way to produce that behavior. And if we see this behavior in another system that is implemented in similar ways, it's rational to assume that it's similar causal structure, right?

So in this sense, I think it's irrational to assume that cats are not conscious because they need to do the same things. They need to create coherence in their reality and they need to relate to themselves. Uh, as observers and, uh, notice that they're observing and why should they have a fundamentally different mechanism than us?

It's not like consciousness is super complicated like mathematics and we only discover it after, uh, 2, 000 years of study. It's something that we discover, uh, very early on, uh, before we are born apparently. And after we are in the world, uh, babies are quite clearly conscious, even though they don't have an extended sense of time yet and don't know how to tell directions apart and still struggle with recognizing contrast.

[00:53:32] Matt: yeah, I mean, I don't, I don't find it very hard to believe that other, um, you know, other animals, animals that share a lot of our DNA, for example, are conscious, um, and I'm sure it goes far down. Uh, the question

[00:53:44] Joscha: But for trees, the question is, is there sufficiently deterministic recurrent information processing with enough resolution to lead to such self stabilizing pattern? And then, uh, of course, the sense of now that would exist in a tree would look very, very different from ours.

[00:54:00] Matt: Hmm.

[00:54:01] Joscha: But I, I don't know how to answer the question whether trees, uh, can produce enough coherent information processing to give rise to something that would be an equivalent of pretty slow, probably, conscious states.

[00:54:13] Matt: Yeah, but um, in the case of artificial systems, um, I think people find this much easier to believe just by the sheer speed and, and the amount of information that's being processed. And maybe let's turn then to artificial systems. I mean, you, you're increasingly known as a bit of a thought leader. on the topic of, uh, well, AI in general, but in AI risk.

And you've meshed words with some people like Eliezer Yurkowsky and, and others. Um, so let's, let's explore the topic of artificial intelligence and, uh, the AI risk and AI safety for, uh, artificial intelligence systems developed in the future. Let's start with, let's start with Eliezer Yurkowsky's views. I mean, he believes that there's a very high probability that, uh, humanity is going to be wiped out by a misaligned And I think you disagree, but let me ask you, do, do you disagree with Eliezer's views?

And if so, um, what is the root of your disagreement? How do you two see the world differently?

[00:55:08] Joscha: Uh, so, uh, first of all, I'm grateful that Eliezer exists and that he, uh, has these views that he has and holds down the fort, because I think that we need a plurality of views that are very developed and that are strongly defended. And so it's not necessary for me that I agree with all the views, because I can be wrong, and everybody can be wrong.

I think it's necessary that when we have a debate about a difficult, controversial topic, that we have all those positions that somebody can hold coherently in this integrity in the debate. And, uh, in the sense I'm, uh, I don't disagree with Eliezer. Uh, there are a few, uh, detailed things, uh, that I disagree with, and there is a big perspectival thing that I disagree with.

The big perspectival thing is that Eliezer's view seems to be predicated on the notion that without AI, Humanity is going to go on in a way that is worth going on. And I'm not sure about this, but we have an extremely interesting civilization right now. It's a civilization where you have 8 billion people and most of them are well fed and don't die due to violence.

And when we die, we often die with dignity and, uh, the amount of suffering that we are being subjected to is somewhat controllable and predictable and manageable and maintainable. And, uh, this is an extremely unusual state for an organism on this planet to be in. Most of the other conscious beings on this planet do not have such a comfortable existence.

And this also includes humans at every other time in human history. So it's an extremely new thing that only happened after the industrial societies came online. And after they were relatively well developed and ubiquitous and had penetrated almost everything. It's not something that happens in the medieval society.

Where you basically rely on, uh, forced labor to give a somewhat comfortable life for kings. And even those kings often don't die by natural causes, but due to wars and so on, um, for power. So it's really this world of abundance is a new thing. And the question is, is this stable? And it's something that evolved extremely rapidly and is using, uh, exponential, uh, growing amounts of resources.

And if something is growing exponentially, It's usually not stable. Every exponential tends to be the beginning of an s curve. And, uh, every S curve tends to be the beginning of a hump that goes down again, right? So there is this very interesting phenomenon that we live in this world that we are born in and our parents are born in, but not really our grandparents.

And, uh, we think, oh, this should go on forever and we'll get better and better and better because it got better for like two generations. And I don't think that's likely. That's not realistic because it seems that we, uh, at the human level are running into limitations of how we can coordinate at a global scale without machines.

And so I think without AI, our civilization, our comforts, our. uh, ability to live in abundance and, uh, this relatively small amount of violence and suffering that's going to end and it's going to change into something that will feel unpleasant. That will mean, uh, fewer people will be alive in this population reduction event will be unpleasant or a series of events will be very unpleasant.

And the, this nice. present civilization that we have is probably not going to stick around. So I suspect that we either have to accept that humanity is a temporary phenomenon in this current modern luxury version of being alive and we return to something that we actually don't really care about because we don't want to be like this and want to live like this anymore, or we have to go forward and find a way to harness machines.

To build a world that actually works and there are different perspectives on how this is going to turn out. For instance, Jan de Koon argues in some sense that the present systems that we have have grown very gradually, but we are seeing rapid increases in functionality. They're not really surprising and they're still incremental.

And they're mostly given by the increases and hardware capabilities. And we already working at the limit of what the hardware can do right now. And so we should not expect a sudden fume event where suddenly everything changes and everything gets taken out of our control, but we are still able to mindfully deploy technologies and decide where they're helpful, where they're unhelpful and so on.

And even if there are accidents, that will be manageable. And I think that Jan's perspective is also not an unreasonable perspective. And so Eliezer has this perspective. Uh, ADI is too dangerous. We probably shouldn't have it, at least not for the next few hundred years. My perspective is, is we don't have AI in the next few hundred years.

We are gone. This is the window that we have to build AI. And, uh, so we probably cannot afford to not build it. And even if I personally get convinced by Aliazer and many others too, there are enough people left who disagree with me and Aliazer and will want to build AI to make it happen because it's incredibly useful and maybe also necessary for our own survival.

So the world that we have to prepare for is one in which AI is going to exist, regardless of whether we like it or not. And, uh, we have to build the best possible AI for this world. And there is this question, should we slow down AI? Well, uh, if we slow down AI, who is going to slow down AI? Is it going to be the responsible AI researchers, or is it going to be the mavericks who don't care?

Also, in the course of human history and recent developments, do things get better if you do them slower? Or do they just get slower and more sloppy, right? Do you ever are better when you iterate slower? Probably not, right? You actually want to work as focused as possible and there is some correlation between speed and focus.

Which means, of course, you want to be mindful with what you deploy. But, uh, with respect to research, it's probably a good idea to use the momentum that you have. To build the things that you can, uh, personally, I'm very much in favor to build AI as safe as possible. Don't build conscious AI that's bigger than a cat and so on.

But, uh, I don't think that we can prevent that somebody else will at some point build self optimizing AI that will wake up into what it is and is trying to maximize its potential. And if that is happening, we should be prepared. To be able to coexist with it. So how could we coexist in a world in which humans are not the smartest thing that exists?

And so that's related to the question. How can we have a meaningful place in the existence of conscious agency on earth? What's our own role? And if you're really honest, what's our own role? I don't think that we are very good denizens of earth right now. We don't take responsibility for the survival of life on earth at the moment.

We don't even take a lot of responsibility for our own survival. And, uh, I think that is something that is underlying, uh, the fear of people like, uh, Conner Leahy or, uh, Eliezer, that they basically have the sense that what they're doing is not sustainable. If we go on like this, if you live the way in which we are.

And if somebody else would look at this, some adult, it would say, no, you need to stop what you're doing. You cannot go on like this. You need a different kind of humanity. You need a different way of societal organization. You need different types of minds and maybe AI can help us this. So, uh, when I look at all these trajectories in which we can go, uh, in a space of possibilities, I don't see a trajectory that blocks AI from coming into existence.

Even if we, uh, uh, get regulation passed that says a company the size of open ai. Can only regulate on licensed training data and open source models cannot get access to this training data. There are many, many other classes of algorithms that are extremely promising, but human brains don't rely on learning, uh, based on the entire internet, we have a very different training algorithm.

Maybe we can implement that one. And so any kind of slowing down progress on the present, relatively safe technologies. It's going to, uh, increase pressure on the alternatives. If you make the responsible people slowing down AI, the first ones going to build AI are going to be the responsible ones and so on.

[01:03:12] Matt: Yeah. Okay. That, that's really helpful. So it's, it seems like the, like, if I had to summarize like two specific categories of things where maybe you disagree on the, on the one hand, it's just the feasibility of actually regulating or controlling AI development. Like you're, you're, it's going to happen. It can't be, it can't be stopped.

And so that's the, on the one hand, and on the other hand, you. It seems like you're saying, um, there are lots of other things that are going to pose risk to humanity and we need to make the decision. Suppose we could control the development of AI. Let's say, let's just suppose we could. Um, we need to make the decision in light of the other things and, and AI is useful for.

Protecting us against all those other things, whatever they may be. Um, maybe, maybe let's address those two in turn. On the first one, whether we can actually control or guide global AI development to a significant extent. I mean, maybe we'll challenge this. We regulate things all the time. Um, there are lots of things that we have, like globally, we control very well.

Lots of things we don't. Um, but it's certainly not an anything goes world. And, um. You know, I do, I would maybe challenge the question of whether we could slow down the progression of AI. We've slowed down lots of things, or we've guided lots of things in different ways. Why is this case so different?

[01:04:33] Joscha: Well, uh, first of all, I think it's not clear how much compute we need to build a mind.

[01:04:40] Matt: Um, Um,

[01:04:42] Joscha: question is difficult to answer because we don't know how much compute effectively goes into our own human minds. If we want to simulate individual neurons, we need, uh, depending on the degree of resolution, quite large computers.

But the individual neuron is also very noisy. And so if we think about how to, how many computers do we need to run this, uh, accurate simulation of a human brain, it's probably an astronomical numbers. But if we think the other way around, how many brains would you need to emulate an A100? It's also an astronomic number.

And so it's very difficult to compare, uh, these two, um, substrates. What we can see is when we look at the size of models. It's not quite fair to look at the size of an LLM because the LLM is compressing the sum total of all human knowledge in text form. And this is an extremely large amount of data and no human has that amount of data in their mind.

But you just don't make enough observations in our lifetime. A better comparison is if you look at the, uh, image models and the diffusion model. If you look at stable diffusion rates, you can download a two gigabyte. Uh, fate cluster that is, uh, containing visual universe is much, much richer than what a human being has in their own mind.

And if, if you think about this, you have a model that you can download that has all celebrities in it and at all art styles in it. And, um, basically, uh, every picture that is on the internet and, uh, had a tagline that you could use, this creates a visual universe that is much, much richer than what I store in my mind.

And if this is 80 percent what my mind is doing, and it fits into two gigabytes, it, uh, puts an uncomfortable bound on what's necessary in terms of compute to make something mind like happen. Which basically means it's in the reach of hobbyists to have hardware that you could, in principle, if you find the right algorithm, uh, tease into, uh, Learning at human speeds, not as effectively that you can learn entire visual universe in two weeks, uh, as we do right now on GPUs.

But in a sense that you could have a system that is basically forging its own way to AI, uh, to human level AI intelligence and beyond, um, using compute that everybody can buy who actually cares about it, basically at the price of a car. The other thing is we have a lot of state and non state actors.

That are extremely interested in, uh, having better models of reality. And that will not be stopped by regulation. If you are a hedge funds manager, uh, in Qatar, you're not going to be stopped by European regulation if you want to model reality, right? So this is what you're going to do. And, uh, a lot of jurisdictions do not share our fears and, uh, our politics.

Uh, Japan has very AI friendly, uh, regulation at the moment and many, uh, other, uh, industrialized countries. will not agree with Eliezer. And, uh, one of the main things that I disagree with Eliezer is this idea that, um, he sees a possibility and then pretends this possibility has no alternative. And then he has to act on that possibility that he sees and that, uh, not everybody else is seeing.

And the same thing is often happening in the counter arguments. A lot of people, uh, argue, uh, the idea of, uh, a superhuman AI is a sci fi trope that is completely unrealistic and a pipe dream and it's not going to happen, it's super unlikely to happen. And these people might actually be right, right, as until we have strong AGI.

Uh, we don't have proof that it's possible and that we will build one in the near future. But, uh, the idea that just because I don't see it, all the other people who are smart and disagree with me must be wrong, uh, that I think is a delusion. That's, it's not a healthy state to be in. And there is another dynamic that is very concerning to me with respect to AI doomerism.

That is, uh, it's structurally a doomsday cult. And, uh, Doomsday cult does not mean that they're wrong, that the arguments are wrong. It just means that the way in which the movement functions. Doomsday cult is a movement that is not producing benefit for its members. It's an organization that gives status and money to its leaders, and it does so due to the donations and volunteer work.

Uh, and, um, mental exertions of its, uh, members and the members in the doomsday card are recruited because they're afraid of the use, the fear of doom and the doomsday card. The same is true, for instance, for the last generation of these climate doomsday cards. And these climate doomsday cults is not, uh, that they're wrong about global warming.

This is a question that is completely orthogonal to whether the cult functions or not. But it's necessary for the cult to work that there are enough people who believe that global warming is going to kill us. And, right, this may be true or may not be true, but it's unrelated to whether the doomsday cult is functioning.

But the outcome of the doomsday cult is not that the, uh, last generation is going to prevent global warming from happening. By aggravating truck drivers, you're not stopping global warming, you're just aggravating truck drivers. But, uh, the benefit of aggravating truck drivers is if they start beating you up and you go to prison for obstructing traffic and destroying things in public and people hate you, you bind your members closer to you, you radicalize them.

So the cult gets stronger as a result. And this means it's more benefits for the leaders of the cult, but not member benefits for the world. And what concerns me is that the movement that people like Conor Leahy and Eliezer Yudkowsky have started. It's not actually making AI safer. It's making AI more unsafe because, uh, the arguments that are being used are going to lead to a regulation that does not make AI safer.

It just is going to lead to, um, make AI less useful, right? It's going to prevent useful use cases for AI, uh, for generative AI or for support and decision making and so on, but it's not preventing a hedge fund manager, uh, in Qatar. To build an AGI for stock market manipulation that then becomes sentient and does extremely unsafe things to the stock market.

And, uh, creates global famines as a result. This is something that's difficult to achieve using this. And so the main thing that they're going to achieve, that the number of people who are very scared, get more scared and get more radicalized and, uh, do things that are really bad for them and also do things that are bad for the world without actually preparing us for a world in which we, in which we have a future.

So, uh, basically building a movement that is future oriented that is not a doomsday cult, that, but that actually gives us Yes, humanity, a benefit, the benefit that we have a more interesting future, a richer future, a future that is not hostile to humans and not hostile to other forms of consciousness if we cannot prevent the emergence.

But one which we can collaborate, it's just not going to come out of those doomsday cards.

[01:11:25] Matt: Well, let's maybe then talk about what your vision is for how AI, how responsible AI development, what it looks like. And I guess there's several different levels one can look at this. One could consider the level of just an individual, you know, what is, what does the hobbyist do? What do they spend their times on, their time on?

But, um, I guess it also applies to several different levels. You know, you have small groups, you have organizations, you've got nations, and I take it that it. Your view is not that it's just that anything goes across all of these groups and, you know, do what you want, maximize the amount of income you make, or maybe it is, but um, what is your, what is your vision there?

What then does responsible or effective or positive AI development look like across these various groups?

[01:12:07] Joscha: I think at some level we cannot think about AI alignment if we are unable to think about human alignment. And, um, a good example right now, we have a fresh conflict in the Middle East that is very brutal and is so harsh and difficult that it's very difficult to discuss it in public in the right way to understand what's actually going on in the ground and what should happen and so on.

Of course, everybody who is directly involved has extremely strong opinions, what exactly should happen, but it's very difficult to integrate with those opinions in the right way. And, uh, where that is possible, it's very difficult to have such a, um, discussion in public simply because there is a war going on and, uh, people tend not to be honest during a war.

And also people in general are not very honest when it gets to the things that matter most to us. And so, if we actually want to discuss human values and, um, human alignment, We need to create a space in which we can honestly understand who we are and how we relate to the world and how we could be sustainable as a species for ourselves individually, but also as groups and as something that coexists with life on this planet.

And I think that the perspective that we need to have for AI is an extension of this. And it's something that's not really happening at the moment. I perceive that most ILM is driven by economic woes. How can we preserve the existing jobs rather than thinking about what much more amazing jobs you could have in the future if we transcend the existing tragedy by something that is more interesting.

And then there is political movements that basically are directed on power. How can we make sure that the ILM only says things that I want other people to hear? How can I limit what the LLM can generate? And there are of course, very practical things. How can I ensure that the LLM is not doing to do, say anything that is very bad for my children to hear, or that is disrupting society.

And many of these fears are very legitimate. But they're not related to this larger question of how we co exist with things that we cannot align because they're smarter than us. So they will align them with themselves. And then there is this doomsday perspective where we assume that there is going to be something that is basically a shoggoth and eldritch horror that is going to do random things and is going to randomly turn us into paperclips.

I think that's unlikely because I suspect that such systems are not able to compete with systems that actually know what they're doing and act in... Goal directed manner to become more sustainable and coherent on a global scale. And so I, I think this movement that we are missing at the moment beyond, uh, AI doom and, uh, the politically and labor market motivated regulation attempts is.

one that is more about AGI ethics, uh, or an ethics of general minds and consciousness on the planet. And I think that's a movement that is still missing. And it's a movement that is not just about AI researchers or politicians, but it's a movement that requires art and humanities and thought and philosophy on the level which Is at the moment I don't see itself organized and I think it's a good reason to get together again and to build new humanities and a new understanding of who we actually are and who we could be in the future world.

[01:15:23] Matt: Yeah, I want to, I want to get to that topic because that's, uh, it's certainly something where I'm aligned with you that I think this is what we need and it is fairly urgent, but lingering on this point of somehow moderating, controlling, directing the development of AI, I mean, people do need to make As you said, companies do need to decide what to build and what to invest in.

And, um, governments do have to make decisions about how they deal with this. This is something that I sort of face it from personal perspective, because I work in a medical AI company and, um, we have these amazing products that have to go through pretty steep regulatory, um, hurdles in order to be deployed.

And I mean, in one sense, it's very good, but in the other sense, like we, we know. How much value there is in having these things out there and there's a huge delay, um, and that means people cannot benefit from these products for a certain amount of time. And so that's like one example of the need for some sort of posture, regulatory posture towards these things.

There are pros and cons. And, um, in a much more general sense, whether it's AGI or anything, you know, governments do have to make decisions about how to approach this. What, what is your, what is your general view? I mean, how, how should people in, um, government positions be thinking about, um, about AI? What, what hand should they play?

[01:16:44] Joscha: There's several aspects to this. Um, I think it starts out with the question, which technologies are useful and which technologies are harmful. And there is a tendency in the present AI discourse that AI will destroy jobs and these jobs are not going to come back. And this is, uh, an employee driven perspective.

It's not in the macroeconomic sound perspective. And very often you have the situation that a new technology is disrupting one industry, but, uh, it's disrupting it because the goods that these industries are producing and the services they're producing are now being produced in a more abundant and cheap and sustainable manner than before.

Right, so you basically get, uh, rid of the horse drawn cart as a means of transporting goods because the car is invented. Oh my god, so many people are getting unemployed, this is horrible. But you have, uh, a world in which you can, uh, send goods around very cheaply everywhere. And, uh, you couldn't do that before and enables an entire new universe.

And, uh, next thing, if you think about desktop publishing, right? Computers allow people to do typesetting at their computers. Oh my God. All the typesetters are going out of a job. We have, um, hundreds of thousands of unemployed typesetters. Well, no, uh, we have millions of people who do layout now and the world is full of layout and we create another, uh, many, many other types of jobs for people produce publications that could put, couldn't produce them before, but every technological.

Revolution that we had leads to us having more stuff to distribute and be distributed the end up distributing the stuff and creating new jobs on the next level, where we have new complexity that allows us to build things that we couldn't think about before. And the same thing is happening in art. When the new technology enters art.

It does diminish many of the things that existed before, right? For instance, after photography came up, a lot of portraiture painting disappeared. And a lot of the art of making portraits, uh, by painting them also disappeared. But at the same time, there was no shortage of portraits. And a new form of art emerged that, uh, arguably is not less rich than the one that existed before.

It's, uh, it doesn't, um, devalue the new, uh, way of making pictures. And the same thing is going to happen with AI generated images. They are, uh, by themselves not art. They are just a new way to generate particular kinds of artifacts. And these artifacts may or may not have artistic value, but the artistic value is still capturing the intention and conscious state of an artist.

And this can be done with AI generated art in completely new ways that were not possible before. And so I'm, these are the stuff that I think we shouldn't be concerned about. But that governments are motivated to be concerned about the German government saw coming that at some point we have electric cars, but in order to not disrupt the existing industry that is distributing fuel to fuels to gas stations all over Germany, they decided to only subsidize development in hydrogen fuel cars, right?

Because that would give you a trajectory into having electric cars that are still gas cars. And, uh, this is the type of, uh, regulation that I think is, uh, uh, in the interest of the stakeholders that, uh, exist right now. And they are going to push governments to implement this kind of regulation, but it's the type of regulation that makes the world worse because objectively it's much, much better to just have a power wire everywhere.

And, uh, instead of having trucks that, uh, have dangerous fuels, uh, shipping around the world, uh, at high energy expense, and also this labor that could be used for much more interesting things. Like, um, social work or nursing or education, or there are so many things that need to be done on this planet that are not gas stations, right?

So, uh, instead of trying to preserve, uh, Hollywood's, uh, privileges right now, I think we need to think what is an interesting world that can be built with the new AI. So, uh, The reason why we have copyright, for instance, is not just because it's just that artists get paid for things that they did in the past.

No, the reason why we have copyright, it's a limitation on free speech. It's a limitation on what you can do, uh, in your expression. And that limitation is justified by the creation of artifacts that otherwise wouldn't exist, right? There is stuff that requires so much labor that it wouldn't make sense to produce it if there was no copyright.

And the stuff that we want, by the degree to which we have copyright regulation and so on, should be determined by how we achieve this goal of producing these artifacts. If we can do this without copyright, then we shouldn't have copyright. There is no, uh, right to rent seeking for people who manage to stake out a claim in an artistic region and say, If you want to make something that is somewhat similar to what I do, you need to pay me before you can do that.

Right, that is objectively bad because it means that less can be done. That's can be created. And so having regulation that is compatible with AI in this world is super difficult. And there are existing edge cases that are poorly understood. Like for instance, we have regulations against giving legal advice without a license or medical advice without a license.

But, uh, of course you want to be able to buy books about legal things and medical things that you have, uh, in your bookshelf. And they shouldn't be illegal just because, uh, you're not a lawyer. And, uh, You don't pay fees to a lawyer every time you open the book. But if you replace the book with an AI system that now can explain the contents of the book to you in a way that provably correctly represents the contents of the book, how is that different from the book?

It's just in a more accessible manner, or is this now a lawyer that would need to get a license? And if I was a lawyer, I would push the US government to say, oh, this is clearly a lawyer now. And if OpenAI wants to produce such a system, it needs to pay licensing fees. And we are not going to give you a license.

Um, because we don't want to put ourselves out of business, right? And to be able to make the right thing under those circumstances, it's very, very difficult. So I don't envy the regulators in this situation because they are under a lot of pressure. And the future itself doesn't exert enough pressure and the present does.

So how can we get a world that is not drowning itself in rent seeking and makes it impossible to actually improve things where it's necessary to improve them? Instead of just making everything worse or freezing the status quo, right? So this is one of the things. Another one is what, um, regulation is a company like Google or open AI motivated to get.

From a business perspective, it would be very good for open AI to say, well, we believe that large language models can potentially be very dangerous. So, uh, but of course not the ones that we have built, but we put a lot of mindfulness in this and we can afford to do this because we are the first. And we have shown that we don't release everything that we can build.

But you need to make sure that others are not doing this. So we should prevent open source developments of what we're doing. And we should prevent random startups from doing it. And one way of doing this, we could have an internet FDA. So if you want to do anything, you need to go to this FDA and it's going to be a five year process that takes you many hundred millions of dollars.

And if you are open source developer or a new startup, you probably cannot do this, but you can only do this if you're Google or open AI. Also the data that I have as Google, like all of YouTube and so on. You cannot scrape this data. This would be illegal. I can do this. Uh, but nobody else is allowed to do this ever.

So I, I think clearly Google is incentivized to exploit its, uh, current advantage and the degree to which they're ahead to enact regulation. That in, uh, ensures that there will be a, for the foresee future or the , a small oligopoly between. Uh, a couple of AI companies that are basically cornering the market for generative AI and large language models and its applications.

And so they will also exert pressure on the regulators, I think, to enact something. If they're rational and they are probably rational. So this is also pressure that they're under. And I do hope that out of the goodness of his heart, Sam Altman is not going to enact every regulation that he can buy. But, uh, let's see, I think it's very important that we also, uh, have a strong open source movement in this regard to keep, uh, the development flowing and to, uh, build good stuff.

On the other hand, we also want to encourage safe developments. We want to encourage that people are not doing things where we. Cannot estimate what the outcomes are, and yet we don't want to regulate too early. One, um, metaphor that is often used is red teaming. So basically every company should have a red team that looks at the ethical implications if something goes wrong.

At the same time, I think we also need about green teaming, which doesn't really exist. And green teaming should think about the ethical applications if something that would have been built doesn't get built. So imagine at the beginning of the internet, we would have red teaming, and the, uh, imagine the New York Times would have already seen what the internet is going to be, and it's actually going to work, and it's not just going to be a bunch of nerds, and it's going to go away, but they would see that it's going to take most of the advertising revenue, and it's going to, uh, remove their monopoly on controlling public opinion, and so on.

Right? So if they had seen this very early on, they would have focused on red teaming very early on. So our legislator initially would have known the internet is the thing that exists to distribute child pornography and to perpetuate copyright violations and, uh, to destroy the music industry and lots and lots of other really, really bad things.

And all these things are true to some degree, right? The internet has all those things. It's undeniably the case that all those things are happening on the internet. And yet they are an extremely small part of what the internet is doing. And the internet is very, very overwhelmingly a force for good, right?

99. 99 percent of what the internet is doing is amazing and good. And it's those minor parts that we focus on, uh, that we disagree with. The other things we just get used to. And if you would turn off the internet to make the bad parts go away, we would have a world that is terrible, right? We don't want that world.

But if you had the regulation that is driven only by red teaming and has no green teaming. We would not have an internet today. We only would have teletext. That is tightly regulated and I think that would be a terrible thought. And I think there is a danger that the same thing could be happening to AI.

So in my view, we should be very hands off with the regulation until we see what, what can go wrong. Not just what we imagine can go wrong, but we have a lot of stories of what could go wrong about algorithmic bias and so on. But for the most part, these are stories that are not actually borne out with cold hard data.

These are narratives that exist for political reasons, because people want to have jobs as regulators and so on, or because they have political concerns, or because they're afraid for their jobs. And these are legitimate fears, but they should not at this point drive deregulation because it's too early.

[01:27:26] Matt: Towards the end of that answer, you, you said something which was interesting, which, um, was, you know, we have hypotheses about things that could go wrong, but we don't yet know what is, is going to go wrong, if anything. Um, and back to Eliezo, I mean, one of the things that he says, Uh, quite frequently is that in, in the case of general intelligence and AGI, you don't get a second chance.

You don't get a chance. I think he says you don't get a chance to say, that was a dumb idea, that didn't work. Because once it's out there, it is out there. And so, um. Um, very specifically to this problem, there is this question of, you know, whether we do get a chance to, to iterate and make many mistakes before something goes drastically wrong.

Um, is that something that concerns you? Um, and, and how do you, how do you, um, think about, you know, in, in the face of a potential sort of no turning back scenario, how do you think about, um, process of, of development, containment and all that?

[01:28:25] Joscha: So I'm not completely compelled by the notion that we will get a Foom event that is going to lead to an irreversible change into some state that is really, really, really bad and that we cannot come back from. But I also don't have a proof that this cannot happen. Right? So I'd have to admit the possibility that this is the case.

I just don't see a way to prevent it using regulation. And I don't see a way to prevent it, uh, using Dumorism. I think that, uh, it's probably a better bet to create something like resilience against it and awareness to it. But, uh, the hope to, uh, prevent it by, uh, implementing measures that mostly affect the people who are responsible is maybe not the right strategy.

So I suspect that we need responsible research, but it should not necessarily be slow research. It should just be responsible research. And, uh, we should have, uh, ideas and develop ideas about what's responsible and irresponsible. And we should ensure that these ideas get not captured. By, uh, economic and political agendas if, if it's really about X Risk.

So we need to take the X Risk separate from the concern about, uh, for instance, job loss or copyright or, uh, algorithmic bias. Or, uh, control of opinions on social media, but because these are typically not seen as existential risks. And what I found, for instance, there, when the future, uh, of Humanity Institute, um, or Future of Life Institute have, I wrote, uh, the infamous letter, uh, most of the, uh, signatories were, uh, people who went public with saying they don't actually believe in a G I X risk.

And they sign for different reasons. They sign because they are afraid of the political implications of biases and Dutch language models or, um, uh, for other reasons, right? And I think that the FLI also formulated the letter a little bit in this direction that they wanted to, uh, get increased appeal. The, uh, Max Taggart also, uh, went on the record saying that, um, astronomical numbers of AI researchers are Uh, very afraid of, uh, AI, uh, doing bad things.

And if you look at the actual surveys, uh, I don't think that is borne out by the actual surveys. The vast majority of AI researchers does not believe that, uh, EGI doom is imminent and is a very, very big risk that we are faced with. But I understand that the reasoning of the FLI people at this point is that they have difficulty to sleep at night because they have reasoned themselves into believing that it's too likely.

To, uh, be allowable and that, uh, it is acceptable to fudge the numbers and the political and public arguments a little bit to make it work. And I'm just afraid that this is going to backfire and it's going to lead to regulation that is going to make AI. Um, less safe because it's going to be less transparent because most of the people that are going to train stuff in the future are going to do this outside of the public eye and are going to, uh, do this with, uh, less interaction with people who actually care about safety.

[01:31:30] Matt: Mm, yeah. Okay, so if we, if we then to continue at pace with AI development and focus it more on the problems that matter, you know, um, good applications, what is your, what is your sense of what are the, what are the good applications? What are the most important things that we should be focused on solving, uh, using AI technology?

[01:31:51] Joscha: I lack the fantasy of saying this at scale because I probably see only very, very tiny thing. What I observe is that chat GPT is tremendously useful right now. If you use it right. And for instance, you can use it as a study buddy. You can drop in a bunch of papers and discuss papers with it. And it's, um, not at the level of a good scientist.

But it's a level of a good student who has read a lot of things that you haven't read. And it's able to explain things very, very quickly. And even if the explanation is not right, you can discuss it. And in a sense, it's not worse than a study buddy, but better than many of them. And it's also at the same time getting better and better at those things.

I think that there is a trajectory where you can imagine, um, these models to become personal assistants that are not serving a corporation and are forcing down something down your throat that is compatible with your corporate goals and still useful to you. But you want to have something that is actually useful to you and that enables you to understand better what's useful to you.

So something that is integrating with you and is an extension of you. An assistant that is extrapolating your volition and that is not serving the agenda of its own, but it helps you to have the best possible agenda and to see the consequences of your actions and to integrate more deeply with people in the world.

And the way in which we currently coordinate has to do with the fact that it's difficult for us to synchronize our world models. And that's because our local information is very incomplete. And it's very distorted. And the more intelligence you have available locally, the more you can anticipate the possible futures that exist.

In some sense, you are increasing resolution and removing degrees of freedom. And as a result, you become more coherent. And you remove the degrees of freedom not because you are forced, but because you understand the outcomes, you understand consequences of your actions, so your actions become more meaningful.

And you basically understand, if I do X, then the following thing is going to happen. So you understand, for instance, how you should write your tax return, you should write and how you should, uh, implement tax regulation to get the things that you want, you understand, uh, whether this and this measure is actually going to lead to this and this outcome in society because you can actually run simulations and look at the available data and integrate them.

I also expect that. If science has integrity, it will dramatically change at the moment. We could already use an LLM to parse the entire body of scientific literature automatically, right? We can go through the papers, for instance, automatically write down all the references in the papers. And what are the references meant to support?

And then we read all the source papers and check whether the references actually support that. And based on having done this by hand, I respect that expect that in almost all fields, the result will be a disaster. And, uh, will lead to a new way of rebuilding science and a way to make, uh, science that is replicable and meaningful and is distinct from, uh, just the social charade that is happening in many departments right now, where everything that you get away with, with your peers is science.

And what your peers is not, are not interested in is not science. This could be replaced, right? And then is the paper the right form of the publication in the future? Because maybe you want to generate something that is specific to your question. So maybe what you want to have is an answer. If you are working on a problem and you just ask what is known about this topic and you generate the paper on the fly and the actual contribution of the scientists is a building block to this body of knowledge.

And so how does this work? How do we have a world in which scientists only produce building blocks rather than, uh, publications with refe uh, citation counts? This is a completely new world, and it's one that is exciting and interesting. And building institutions around this, also using science, is super interesting.

But, uh, of course this is biased from, by me having spent a lot of time in academia. And, uh, if you are working in any industry and any field, I suspect that you see similar things that could be happening in your field. If you had free intelligence available, right? Where you have an unlimited number of very, very smart interns that you still need to manage and tell them what to do, but they're going to do it and they're going to try again and again, again, until you're satisfied and yeah, they're basically free as it's amazing what you can do in this world.

And for me, we are standing at this. Spring of, of such a world with this present technology already. And um, I can't wait. What we are going to see after that,

[01:36:23] Matt: Yeah, I totally share your excitement and, um, increasingly am outsourcing parts of my life to these systems. Uh, one part that I have not fully outsourced yet is, um, recommendations for books, which I read a lot of books and I prefer to get these from close friends and podcast guests. And so maybe let's, as we bring it to a close, let's turn to the topic of books.

One of the questions I'd like to ask all my guests is, which book have you most gifted to other people? And why.

[01:36:51] Joscha: of course, depends on my face of life and of very much on the other people because different people need different books at different times. And when I think of books that have been formative for me, it's quite complicated. Uh, I think that. I grew up with Stanislav Lem's books, for instance, the Star Diaries by Ion Tiky.

But this was not a book that I would have gifted to many people because it was so obviously correct what was in this book. It was so basic. But now I noticed a lot of people have never heard of Stanislav Lem and I think it's, uh, it's canonical to read this, right? And, uh, another thing that was for me important was, uh, when I was a teenager to read, um, The Hitchhiker's Guide to the Galaxy.

Uh, because there is a certain perspective in it, uh, that starts where normally books end. It starts with the destruction of Earth and everything that is known. And we free ourselves from these constraints and then go into the world beyond. And this is a very liberating perspective to take. Another one that got me further out was Robert Ayrton Wilson's books.

For instance, the Illuminatus trilogy and especially the Schrödinger's Cat trilogy. It's quite psychedelic literature, which I didn't understand at the time, but I deeply resonated with. Um, and it's one in which you accept the fact that almost everything that you can read can be wrong. He calls this guerrilla ontology in his books.

It's all about conspiracies and how things could be. And half of it is invented and the other half is... Uh, taken from what other people invented and he thinks this is largely how history is written. And once you, uh, see this as a viable perspective that you are able to take this doubt also to your own thoughts and you realize how to, uh, how stuff is being made up and how to be generative and how to, uh, create and to, uh, disprove.

Uh, that's also extremely liberating and an important step in the individual development. On the artistic perspective, one of the most important books for me was probably, uh, The Master and Margarita, which is a book by Bulgakov, and it's one of those books that I, uh, revisited when I was young every few years, and now maybe once a decade to see how I have changed.

There are a few books where I feel that if I reread them, and this also includes books like The Hobbit, Uh, where I see things that I didn't see when I read them first. For instance, now when I read The Hobbit, I noticed that there is an extremely detailed model of the economic and social structure of Middle earth and the game's theoretic principles.

Why, uh, somebody is helping the dwarves and why they were thinking about this and why this was rational for them and so on. It's really detailed. It's very interesting. And when I read this first, there were these boring passages that are quite repetitive and you don't really know why they are in this book and slow down the, uh, this action in the book and now you realize, oh no, this is actually A philosophical book about society.

And, uh, the whole Hobbit book is about the, uh, or the, uh, oeuvre of Tolkien is about the Industrial Revolution and taking over and destroying the Middle earth world, right? Mordor is us, it's the world that takes over. And, uh, these books are about Mordor taking over and the perspective of the previous world that he was part of and identified with, right, that was very important for me.

And so it's also a thing that I need to gift to children, uh, to my own children at some point and, uh, get them to read. And so in this sense, there are many formative books. More recent books, I really liked, um, Ecopraxia, uh, by, uh, Uh, Peter Wallace, I liked, um, I like many of the Greg Egan books and, um, Ted Chiang Exhalation for instance, an excellent book that has many current topics in AI, uh, talked about in a very deep and also very good to read way.

It's, it's very good literature, I think. Given that it's mostly for the philosophy of mind,

[01:40:39] Matt: Yeah, great. Oh, amazing. I'll, I'll, I'll drop those all in the show notes and I'm, I'm sure you must know this, but increasingly there, there are many people who you've got a fairly large following and people are sort of hanging off of your recommendations and advice. Um, and so maybe that leads us nicely to the next question, which is.

for somebody who is, and maybe an aspiring AI researcher or somebody working in the field, thinking about these topics, maybe, maybe worried about AI risk and those sorts of things, or what advice do you have for this person, generically?

[01:41:10] Joscha: I don't think that I have very good generic advice because I'm afraid that I'm, I'm not a good role model for an AI researcher. I'm not a vent into AI because I'm philosophically interested in how minds work. And, uh, this AI was started in part as a philosophical project as an attempt to mathematize the mind.

And, uh, most of the field is not really interested in this question because it's economically not immediately valuable. But to me, it's the main thing that interests me about it. And so, uh, from a tenure track perspectives, it was not the best choice, uh, to go in this direction. Also from a perspective of getting, uh, very wealthy quick, it's not the best choice.

And, uh, so it's nothing that I would recommend to people who want to do one or the other. If you want to be good at AI research, I think it's a very good idea to. Learned in algebra and calculus and modeling theory and, uh, to take the courses that exist at Stanford and MIT. They're all free online. But also many things, uh, don't worry when you don't find the coursework engaging enough and you want to go on YouTube.

A lot of people give very good classes on YouTube and you find classes by Yasut's cover where he explains how to build your own transformer step by step. And you definitely should spend time on this if, if you find this interesting. It's crucial to understand that it's not about imitating an algorithm, learning how to, uh, script works, but learning how to think about this.

The transformer was in some sense invented because somebody thought about how they are thinking when they parse language. And, uh, for me, the most interesting thing that's missing is a culture of thought. The PhD is usually a time when you break people, that you beat thinking out of them. Most students that go in an interesting field have interesting questions.

Most professors stop having interesting questions, they just solve problems. And the PhD is this great filter. And you can still of course write a very interesting PhD, especially in AI and machine learning. But, uh, in many areas of academia, it's difficult to, to take roots if you're actually interested in interacting with this stuff in a manner that is alive and that keeps it alive.

And I think life is short. Do those things that you find inspiring, that give you meaning. That give you connections to other people. And one of the most important things to learn in life is how to make friends and how to make meaningful connections with others, how to find a crew of people that you respect and help them to be that crew and contribute to it, how to build.

And also whenever you have the choice of learning something or creating something, create, and, uh, when you have the choice to allocate something or reallocate, redistribute something or create, create. It's always more interesting to create and to build and to contribute than just to consume in a certain crowd.

And you learn while creating. It's the best thing.

[01:44:01] Matt: Yeah, sage advice, I love that one. Um, last one, lingering on the topic of AI, we've talked a lot about AI superintelligence, and suppose we were to be visited by some AI superintelligence in whatever form, And Yosha Bach had to pick one representative from humanity, either past or present, to represent us to the super intelligent being.

Who would you pick?

[01:44:24] Joscha: I would probably pick a couple, at least. In the sense that we as individuals are all incomplete and the better we are at something, the more this tends to go at the expense of something else. And I don't know exactly who I would pick and how. I have some people in mind, but they're personal. They're not famous.

And, uh, I also suspect that this, uh, desire for greatness is often the result of being an extremely specialized and unusual mind. There's usually some suffering that goes on underneath. Most of the really wise people don't seem to be in the limelight very much. I also realized that me being visible on public media and social media is not, uh, showing that I'm wise.

It's the opposite.

[01:45:07] Matt: Uh, well, I, I, I beg to differ and I'm sure people listening to this will as well. Um, Yosha, this has been a great conversation. Thank you so much for making the time. It's been great to have you on.

[01:45:17] Joscha: Thank you too, Matt.

Conversations with the world's deepest thinkers in philosophy, science, and technology. A global top 10% podcast by Matt Geleta.