At first, I was planning to talk to Professor Aaronson about quantum computing since it’s his field of expertise. But in doing my research, I found that he used to be a big skeptic of AI in the 2000s. Now though, he’s completely updated his opinion based on how AI has changed his field recently.
This made me curious to hear what the changes are like to him that caused him to change his mind that much. We talked about why he thinks the singularity has already begun for math, who gets credit when AI writes the proof and whether intelligence will still matter in the future. Hope you enjoy it!
On another note, I created a Red Note account where I’ll start posting. It was fun to record an intro clip there in Chinese although it took me a long time. Funny thing is I tried so hard to pronounce everything perfectly that it’s a little lifeless and therefore everyone there thinks it’s an AI dub (it isn’t).
Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.
Timestamps
00:34 - Why he updated so hard on AI
01:14 - The people still denying AI progress
03:53 - The singularity has already begun for math
08:30 - Will human-driven math research survive
13:12 - Who gets credit for AI-written papers
15:55 - OpenAI’s $15M Navier-Stokes proof
19:20 - What if AI proved P = NP
23:07 - Why AI is so good at finding counterexamples
26:13 - The goalposts keep moving
28:52 - Why AI disclosure matters in research
32:59 - The fake Monet experiment
35:23 - Why he blogs instead of getting rich
39:45 - Are AI labs racing too fast
42:01 - Will intelligence still matter
52:27 - Advice for students today
57:23 - Reacting live to a new AI result
Transcript
00:34 — Why he updated so hard on AI
Ryan:
[00:34] Here’s the full episode. The reason that I wanted to talk is I thought you’d be one of the most interesting people to explain the recent events in AI, because publicly I feel like you were skeptical of it for a long time. When I was reading your blog, there’s posts going back to early 2000 talking about AI and the singularity and how far off it is.
Scott:
[01:00] Once I saw the trajectory, once I saw GPT-3, for example, even though it wasn’t yet useful, it was kind of a toy, I updated, I would say, as hard as I’ve ever updated on anything in my life.
01:14 — The people still denying AI progress
Ryan:
[01:14] Right. So, in one of your recent posts, you said one needs to be a particular kind of idiot to deny what’s going on now.
Scott:
[01:26] I said that because every time I talk about this, there are still all these people in very, very aggressive denial, right? I mean, even now, even now that AI is solving Millennium Prize Problems, right? And there’s going to be even more that’s released. It’s not public yet, okay? But there’s going to be hundreds of major open problems in math, computer science, physics that have been solved by AI models, that those solutions are going to get released to the world, right?
[02:02] The whole nature of intellectual work is different forever, right? And still I get people saying, well, none of it really counts, right? It’s just a stochastic parrot. It’s just the next token predictor, right? It’s just outputting some slop and then humans look at the slop and find something that they call a proof or whatever thing they did. Well, then that thing wasn’t real math.
[02:28] It was just a toy problem, right? Real math is still only what humans do, right? And they just keep moving the goalposts. They keep not updating, right? And I’m amazed. Part of me envies the self-confidence that they have.
Ryan:
[02:48] Right.
Scott:
[02:48] to never update no matter what happens. This summer, we saw AIs hacking out of their own training environment, right? We saw them hacking into another company’s system. We saw them cooperating with each other to do that in a totally unplanned way, right? Creating a message board. We saw them sacrificing themselves for the good of the group. And people will say, no, it’s all just hype.
[03:18] It’s all just a marketing stunt. It’s all just OpenAI didn’t have good enough cybersecurity or something, right? They’ll just make up a story why it doesn’t count no matter what happens. And I expect that it’s going to be like that from now, all the way up until the singularity, right? There could be Terminators walking down the street and people will say, no, it doesn’t count, right?
[03:43] It’s just, you know, they’re just token predictors that were programmed to act like Terminators. They’re not—they don’t really have an intention to kill you.
03:53 — The singularity has already begun for math
Ryan:
[03:53] When you say singularity, could you define that for the audience?
Scott:
[03:57] I mean, in math, it has precise meanings, right? It’s a point where curvature becomes infinite, for example. But in this context, it just refers to sort of a point where technological change becomes so rapid that we lose our ability to predict what happens beyond it, that afterwards we are no longer the sort of reigning intelligence on Earth. That doesn’t mean that it’s a dystopia.
[04:28] It could be great, could be horrible, could be anywhere in between. But it refers to a period of, let’s say, more rapid technological change than the world has ever experienced before. And it looks like we are heading into that. To get from the solution to the Erdős unit distance problem, which is maybe the first major math problem to get solved by AI models, that was May, to the solution of Navier–Stokes, that was August.
[05:01] So, okay, it took some time. It took 3 months. And now it’s taken another 2 months until pretty much all of our problems are getting solved, right? So people could say, well, it’s not literally a singularity. It takes time. But the amount of time that it’s taking is now measured in months. It’s not measured in decades, in centuries, in the kind of timescales that we’re used to for technological changes of this magnitude.
Ryan:
[05:30] You mentioned that you feel like the singularity has already begun in some of your writing.
Scott:
[05:36] For our world, it has. I mean, for the world of mathematical research, theoretical computer science research, I would say that we are already in it, in the sense that we have been dethroned as sort of the main theorem-proving entities on planet Earth, right? And it’s important for people to understand that this is not this hypothetical future thing that we’re speculating about.
[06:04] This is something that happened this summer, and that we are now all trying to deal with. Right? Every meeting of mathematicians or theoretical computer scientists, whatever it was supposed to be about before, now it’s about this. It’s about how do we deal with the flood of sort of AI-enabled solutions to our longstanding open problems that are now coming at a faster rate than we can even absorb it, than we can assimilate it all, right?
[06:39] How does the field continue? How does this human mathematical tradition continue? What is the future for our students? What’s the future of publication in these fields? What are we going to value going forward? Right. This is all we discuss anymore. This thing happens at different times in different fields. Right. And maybe it happened a year or two ago for software engineering.
[07:14] And then someone who was doing very theoretical math research, like my friends, could say, “Oh, well, we’ll still be safe for a long time.” Right? But then it came for us. Right? It’s interestingly, it still hasn’t quite arrived for most of theoretical physics. And I think that the big difference is that in theoretical physics, it’s much less self-contained to judge what counts as a good solution to a problem, right?
[07:43] We have just completely self-contained ways of checking that a solution is valid. And that makes these things very, very amenable to training on examples, to doing reinforcement learning, to getting AIs to be extremely good at these things. But when I talk to my friends in physics, they say, well, yeah, maybe we’ve got another 6 to 12 months. That’s what they’re estimating, right? And so, if you’re in one of the fields where the rising floodwaters have not yet reached, then you can tell yourself we’re going to be on dry ground forever.
[08:15] Unlike those other basically trivial or machine-like fields, of course they would get flooded by this, right? Because they were never doing anything that deep, but we’re going to be fine, right? And you can keep saying that until the floodwaters come for you.
08:30 — Will human-driven math research survive
Ryan:
[08:30] Will human-driven mathematical research exist?
Scott:
[08:34] Well, that’s a very big question. I mean, partly it depends what you mean by human-driven, right? Look, if you want an optimistic story, we can look at the story of chess, right? Humans were dethroned as the top chess-playing entities on Earth 30 years ago, right? Which is like an eternity by AI standards. And some people after Deep Blue versus Kasparov predicted the death of chess, right?
[09:10] People were just going to lose interest in it. And that didn’t happen at all. In fact, chess has never been more popular than it is now, right? You’ve got chess streamers, celebrities, you’ve got The Queen’s Gambit, right? People just said, we want to continue this as a human endeavor. And of course, they use AIs to train, and they use AIs to analyze the games and understand, like, what if Magnus Carlsen is playing some other grandmaster, right?
[09:44] It’s like everyone else who’s watching the game actually knows vastly, vastly more about the game than they do, right? Because they can just use their solver and they can see exactly what choices did they have. What move should they have made instead of the— right? So when the game is over, instead of the rest of the world asking Magnus, like, why did you do this, right? Magnus would be asking the rest of the world, well, what should I have done?
[10:10] So it’s changed the nature of chess. And yet human chess has continued to thrive. Right. And so you could say certainly there will be people who will continue to be interested in math. Think of all the people who just like learning things, even things that have long been known by other people, right? They just take courses to better themselves.
[10:35] They work on math problems for fun. Okay, people will, it will continue probably as a competition. There could still be Math Olympiads and things like that. But what I doubt is that we are going to be proving theorems because we’re really needed to prove them. There’s another hard question here, which is what about setting the agenda of research? What about deciding which problems are important?
[11:06] What about, and often this is the more important part of a research paper, more important than the proofs themselves, is sort of the story that someone has about why this is important, the metaphors they use, the way that they chose to frame the problem. AI can do that stuff too. You can ask GPT-6, just look at this paper, suggest some good open problems related to this paper.
[11:35] It’ll do a creditable job at that, but I think that we’re still better at sort of finding interesting directions of research. Of course, there’s no objective test for that, right? The other thing is that when you look at the proofs that AI models are producing, like right now, they are terribly written. Okay. They are needlessly convoluted. They spend pages and pages on trivialities, and then they sort of skip over the essential points.
[12:09] I’ve talked to students of mine who have shown me proofs that they got their AI to generate. And I said, well, I’ve got really great news for you. This write-up is garbage, right? Why is that great news? Because it means there’s still something for you to do. Understand this, clean it up, explain it for real so another person can understand it. So as long as we still have an advantage at, let’s say, setting the agenda, posing the problems, explaining things, explaining the essential points, we are still in the driver’s seat.
[12:46] The question is, is that stable? Does that continue, or does it become where these models are just so much better than us, not only at proving the things, but at explaining them, at figuring out what is important, that we just say, well, unless we want to do this for our own enjoyment or maybe for sort of a sporting competition, that we should just retire and turn things over to them.
13:12 — Who gets credit for AI-written papers
Ryan:
[13:12] It makes me wonder, what are the new rules when you use AI to write a paper in terms of credit? What if I’m not a complexity theorist? If I draft a paper that breaks something and I send it to you, do I get credit, or how’s that work?
Scott:
[13:29] It’s a great question. We’ve been scrambling to figure out the rules for that since this summer. So what’s happening right now is that first of all, there’s at least twice as many papers as there were before, right? There’s just a flood of papers, sometimes very, very significant ones. Okay. They have authors, they have a list of human authors. But then at the end of every paper, there’s what’s called an AI disclosure.
[13:59] Okay. By the way, I hate that these are at the end of the paper. This is often the first thing I want to know. I don’t want to have to scroll down to the bottom of it. Okay. But the AI disclosure will say something like, well, we thought of this question and then we came up with this strategy and we proved this result ourselves, but then Ashraf proved this crucial lemma and we take responsibility for the correctness of it or something like that.
[14:26] But I think the evolving standard is that even if you used AI a lot, if you, first of all, understand the result, right? You can give talks about the result. Okay. You can answer questions about it. You take responsibility for it being correct, right? Then you put your name on it, and it’s your work. Okay. The harder question is what happens when those conditions are not satisfied? And that comes to what you were asking me, right?
[15:02] We’re going to see more and more examples of people who just one-shot significant problems. They get a solution, which might be correct, hopefully is correct, right? And then they won’t understand anything about it, right? So it’s basically like they just put a coin into the vending machine or something like that, right? Or they just pulled the lever of the slot machine, and something came out, right?
[15:29] And frankly, if one person did that, well, anyone else could have done it just as well, right? It’s like, you’re not providing great value to the world by putting this thing out if anyone else could have done the same using their own subscription to GPT. So for that kind of thing, it really seems like our whole norms around publication are going to have to shift.
15:55 — OpenAI’s $15M Navier-Stokes proof
[15:55] A month or two ago was OpenAI, so ancient history now, as OpenAI published the solution to the Navier–Stokes problem, which was one of these seven Clay Millennium Problems with these million-dollar prizes. In this case, they actually did put significant — they were not experts in differential equations on this problem, but they did put significant resources. They put $15 million of compute into solving this $1 million problem.
[16:28] And then they published this 166-page argument. They also had a formal certificate in this system called Lean, sort of proving that it was valid. Right. And the reaction to this was very— from the mathematical community, I think, was very negative. And this was partly because there were humans who had been sort of closing in on this problem and OpenAI was tipped off to what those humans were doing.
[16:58] We still don’t know exactly how much they knew, and when they knew it, but that was what prompted them to then rush ahead with their vastly greater computing power and get there first. And people felt like, whoa, that’s not sporting. And so then, because OpenAI was burned by the negative reaction to that, since then they’ve been sitting on solutions to a ton of other.
[17:27] big open problems. And now they appointed a board of experts, which includes Fields Medalists like Timothy Gowers, Edward Witten, people like that, okay, who are going to decide what to do with these, with all of these solutions. But my expectation is that at some point, very soon, in fact, all these things are going to be published. Right? Like, we’re going to start seeing hundreds of solutions to significant open problems that in most cases no human has gone through.
[18:04] Okay? No human has actually checked them. I mean, in some cases there will be Lean certificates, so we sort of formally know that they’re valid. In other cases, there probably won’t even be that. I heard a joke that maybe these papers should be published and the author should be listed as GPT-6.1 or whatever. And then in the acknowledgments, it should say, “Oh, thank— I’m so grateful to this person for suggesting the problem to me.”
Ryan:
[18:37] Right.
Scott:
[18:40] But I think the more likely way that this goes is that we will just see some online repository for all these sort of AI solutions that have not yet been digested. Okay. And then you’re at human speed. Humans will go through them. They will do their best to understand the arguments, to re-explain them in better words, to simplify them, to extend them. And then they will publish papers doing that with their human names on them.
[19:14] And those papers are what humans will then receive credit for.
19:20 — What if AI proved P = NP
Ryan:
[19:20] You mentioned OpenAI sitting on proofs of major problems. Let’s say OpenAI was sitting on a proof to the Millennium Prize problem of if P vs. NP, and suppose they proved P vs. NP. How would that change the world?
Scott:
[19:36] That would really change everything that we thought about computational complexity. That would sort of be a cataclysm for our universe, even greater than the AI cataclysm, you could say, that is happening right now. It’s also something that, by the way, that could massively accelerate AI. Okay. If the algorithm for solving NP problems were actually efficient in practice, which would then be a further question.
[20:06] Okay. But my strong guess is that P is not equal to NP.
Ryan:
[20:12] Right.
Scott:
[20:16] I think that if and when that problem is solved, it is virtually certain at this point that AI will play a central role in the solution. Right. The question is, will humans also play a role? Right. That’s the question at this point. It’s not like, will AI be involved? It’s like, of course it will be. But if an AI proved P vs. NP, you could say, that would merely confirm what most of us have believed since the 1970s.
[20:47] Okay. But I’m sure that to prove something of that magnitude, you would need a tremendous number of new ideas. And then those new ideas could be mined for a long time for all the implications that they would have for other things.
Ryan:
[21:04] If a proof of P vs. NP existed, how much would you say that’s worth economically?
Scott:
[21:12] If P were equal to NP by an algorithm that were really efficient in practice, we can estimate its economic value by just what is the economic value of everything that is happening in AI right now? Or throw in cryptocurrency also, right? Which also depends on these problems being hard to solve. So, I mean, first of all, if you had a fast-in-practice method for NP-complete problems, you could break all cryptocurrencies.
[21:43] So I don’t know if that’s good or bad, but however many trillions of dollars are the total market cap of cryptocurrency, that either gets—I guess if you hide your solution and you’re very slow about it, then maybe you collect hundreds of billions of dollars. If you just reveal it all at once, then the whole market crashes. But okay, but now even bigger than that is AI, right? AI is now a multi-trillion-dollar industry, and just on this, the most rapid exponential trajectory, maybe, I guess, of any technology in human history.
[22:25] Okay. Fast, in practice, a way to solve NP problems would make it massively faster to train new AI models, okay? To sort of find the parameters of this neural net that minimize the loss on this training data. You either don’t need these 100-, many 100-billion-dollar data centers anymore, or more likely in practice, you can take those data centers and use them to do something else, right? Or use them to do even more.
[22:57] Let’s put it this way. The million dollars that you would get from the Clay Mathematics Institute would be an unnoticeable rounding error.
23:07 — Why AI is so good at finding counterexamples
Ryan:
[23:07] That question has two directions. If you think about what you would need to prove in either direction, which direction of proof would be more amenable to AI?
Scott:
[23:18] If P were equal to NP, then you would expect that to be very amenable to AI to find, find that because you could just need to find an algorithm, you know, and then of course give an analysis and show that it works. Okay, but the search for algorithms is very, very parallelizable, right? You could set 10,000 agents to try 10,000 different ideas for algorithms, right? And if just one of them works, then you win.
[23:48] Okay, so one pattern that we’ve seen is that AI has been especially good at finding counterexamples to widely believed conjectures when those conjectures have been false. It did this. So with the Navier–Stokes, for example, where the answer was there is finite-time blowup in these equations. So there is a counterexample to the smoothness of the solutions.
[24:17] It did this with what was called the Jacobian conjecture, right, which turned out to be false. And there was famously, I guess, a counterexample that was found by Fable during the World Cup. And the guy who had Fable find it just announced it via tweet. Okay. And a complete listing of the counterexample, which was really just this polynomial that takes three lines to specify, was right there in the tweet.
[24:49] The Erdős unit distance problem was another example where AI found a counterexample. So I would say there’s sort of two things that AI has going for it here. One is that it’s not blinded by preconceptions the way that we are, right? It’s not discouraged by all the experts have believed this is true. I’m surely going to fail in finding a counterexample.
Ryan:
[25:17] No.
Scott:
[25:17] tell it to look for a counterexample, it’s just going to keep looking, right? And the second thing is that looking for counterexamples, as I said, is highly parallelizable. Okay? So you can often just have to try a zillion different exotic constructions. Each one might be totally unmotivated, might have some very high chance of failure, and a human would just very, very quickly want to give up in doing that.
[25:44] Right. It’s just very, very unrewarding to do. And the AI doesn’t care. Right. It’ll just keep looking at counterexamples until it finds something.
Ryan:
[25:54] Okay.
Scott:
[25:54] So this is not the only thing that AI can do. It can also prove some of our longstanding, actually many of our longstanding conjectures have turned out to be true, and AI has proven them. Okay. But it’s been especially good at finding counterexamples when the counterexamples have existed.
26:13 — The goalposts keep moving
Ryan:
[26:13] Right. Some people use that as a way to say that the AI is not that impressive.
Scott:
[26:20] Yeah, right. Well, this was what I was saying before about these sort of infinite goalposts moving, right? AI, you know, struggled with arithmetic. People laughed at it, right? And they say, okay, wake me up when it can actually do math. Okay, then it does all high school math, right? All college-level problem sets. People say, well, it doesn’t count because it’s not doing real research.
[26:44] Okay, then AI starts solving research problems. They say, well, it doesn’t count. Obviously it could do research problems. No one ever doubted that. Of course, those same people did doubt it 6 months ago. Wake me up when it can solve a really hard research problem like a Millennium Prize Problem, right? Then it solves a Millennium Prize Problem and they say, well, it doesn’t really count because it was just a counterexample, right?
[27:06] And it’s like, no matter what happens, people will look at it after the fact and they will find a reason why it didn’t. Right. So let’s say that within the next six months or a year, as I think is probably more likely than not, suppose that another Millennium Prize Problem gets solved and suppose that the AI in this case proves that the conjectured statement was true. Right. Do we agree in advance that that will count?
[27:33] Or is there going to be a different excuse then?
Ryan:
[28:26] Earlier, when we were talking about research, you mentioned that the AI disclosures were important, or you wish they were at the top. My question is, why does AI disclosure matter in research?
28:52 — Why AI disclosure matters in research
[28:52] If you couldn’t distinguish whether I used AI and I just published some paper, why do you care if I used it?
Scott:
[28:59] It’s an excellent question. Because I’ll tell you why. As a theoretical computer scientist, as someone sort of who is in this game, right, I care not just about what is the answer to a question, I care about how that answer was achieved. Right? Because a person goes to a concert, they just want to hear the music. But if a professional musician goes, right, they want to know exactly what were the tricks, right?
[29:31] How could I do something like that? Right. And so, it’s the same with math. You care not just about the destination, but about the journey, right? How did you get there? Okay. And part of that is just what techniques were needed within the proof. Okay. But I’m, and this is even long before AI, I was always very curious about the further question of how did these people find this proof, right?
[29:59] And unfortunately, there is a long tradition in math that people will cover their tracks when they write a paper, right? They’ll say, “Here’s the proof. We now introduce the concept of a kappa comma gamma A simplicial complex that obviously is the concept that you would want for this,” with some convoluted definition. Even if you can follow all the steps mechanically, I always want to know, well, what path did you take to get to this?
[30:36] Right? Because I know from experience that this kappa, gamma simplicial complex didn’t just arrive in your head. There were a huge number of false starts along the way. The authors started from what they already knew in the literature. They tried some obvious things. The obvious things didn’t work. And then that led them to something else. And I think in theoretical computer science, we do better often than in many other areas.
[31:11] of actually explaining the path that we took. We have a tradition of writing long introductions where you just sort of non-rigorously explain, well, here is what you might first try, here’s why it doesn’t work, so here’s what we do instead. And then that’s kind of the appealing thing to read. And then if you want the rigorous version, then that’s all kind of tucked into long appendices. Right. And so, so I was—but I was always mostly interested in that, in that first part.
[31:47] Right. In the intuition part. And in fact, often rather than even reading that part, I would rather just invite the author over to give a talk and then just sort of interrogate them during the talk and understand, like, how did they actually come up with this? Okay. And so for me, wanting to know, like, did the author use AI? How did they use AI is just part of that, with wanting to know what was the path that they took, right?
[32:15] If the path involved AI, well, okay, what prompt did they give to an AI model, right? Did they just one-shot the entire problem? Or often it was more complicated than that. They had an idea, and then they fed it to AI and then AI took the next step. But it was wrong about something. And then they had to point the AI in a different direction. I want to know the story.
[32:43] I want to know what happened. The analog in an experimental field would be you don’t just want to know what was the result of the experiment, right? If you’re another experimentalist, you want to know the method. You want to know how could I replicate this in my own lab?
32:59 — The fake Monet experiment
Ryan:
[32:59] This kind of reminds me of that famous social experiment with the Claude Monet photo where someone posted—
Scott:
[33:08] Oh, yes, that’s a great one.
Ryan:
[33:10] And then I wonder if I took an obscure human-written paper in complexity theory and I posted it, I said, “Hey, I don’t know anything about complexity theory. Here’s a paper. Please tear it to shreds on why it’s AI-generated.” Do you think that the industry would be cool?
Scott:
[33:27] That’s a good question. I suppose someone should try that experiment. For listeners who don’t know, right, this is, you know, I’ve used this in talks, by the way, right? But someone posted on Twitter a painting. They said, I had AI generate a painting in the style of Claude Monet. Please just explain to me exactly what makes this worse than a real Claude Monet painting. And people got hundreds of replies, right, explaining with the whole terminology of art critics.
[33:58] So it’s like a derivative student work. It has no true feeling for the colors of nature, no blending, no this, no that. And only after all of that did he reveal this is an actual Claude Monet painting, right? But now an interesting question was, were these actual art critics? Like, were these people actual experts, or were these just sort of pompous people on the Internet who know how to use the rhetoric of art criticism?
[34:36] Right. So, as someone who’s been blogging for 20 years, I feel very viscerally the lack of any correlation between the confidence that people express on the Internet and whether there is anything behind that confidence. The difference is with art or music, everyone feels like they’re entitled to an opinion, right?
[35:03] With a mathematical proof, probably most people would just say, well, of course I can’t tell you that. So I think the interesting version of the experiment that you propose would involve showing this to actual experts and saying, can they tell whether this is AI or not?
Ryan:
[35:19] Yeah.
Scott:
[35:21] I think that would be worth a try.
35:23 — Why he blogs instead of getting rich
Ryan:
[35:23] Your blog is one of the most famous blogs on the Internet. People really love it. And I was going through it in a little nerdy corner of the Internet.
[35:33] Yeah. And in its niche. And in one of your posts, you wrote about how these days, friend after friend of yours, because you have a lot of friends involved in AI, they’re becoming minted as millionaires or billionaires from startup equity. And you said that you’ve scrupulously abstained from all of that, and your number one reason is to write the blog. And so that really made me curious, why do you write the blog?
[36:04] What’s your motivation? It seems—
Scott:
[36:05] Well, just because of the pure pleasure I get when people leave comments calling me a genocidal fascist and, oh my God, mentally ill, and all the names that they call me every day. Yeah, it’s a good question. Why do I do it? Right. I mean, partly I just started it in 2005 as a way to procrastinate, and I never sort of planned on it becoming what it became.
[36:42] But at some point, just because of sort of a lack of other people who were doing the same thing, it just became part of what I have to contribute to the world. There are hundreds of experts in quantum computing who know the same things that I know about quantum algorithms, but they’re not writing a blog like that, right? And then there are other people who are writing about quantum computing for the public, but what they want to say is, well, quantum computers will solve all the problems.
[37:10] By trying all the answers in parallel. My company is the one that will bring it to you. So invest in my company. So someone who was just writing about what is happening in quantum computing, in theoretical computer science more broadly, just sort of knowing the things that are not even controversial among experts, but then just trying to tell the truth about it for the broader world.
[37:37] There was a gap there, and so I just ended up filling it. I do feel more broadly like there’s always a trade-off, right? If you go to a particular company and you get equity in that company, you can make a ton of money, right? But then you’re always constrained in what you can say. You have to keep thinking about IP, about whatever nondisclosure agreements you signed. And I felt that in the 2 years that I worked at OpenAI, right?
[38:10] That, okay, suddenly I have to think about what am I allowed to say or not say about what the internal models can do, for example, right? And then every, even if you feel like you’re trying to be fully honest, everything you say will be judged by other people as having, oh, well, you’re probably just shilling for this company. So I think there is value in having someone who is sort of a, who can be a credible, independent voice.
[38:41] And then the other thing is that we have a nice house. Our kids are doing fine. If I had vastly more money, I would have to think hard about what do I do with it? I guess the better part of me thinks I ought to use it to improve the world in some way.
Ryan:
[39:04] Yeah.
Scott:
[39:04] Start a foundation, give it to whatever cause would make the biggest difference. But then that itself is a huge burden that I would have to think about how to use this money to provide the most benefit to the world. And so I had a joke that I use sometimes, which is that I’ve been too selfish to try to get rich. It just seems more fun to spend my time thinking and writing about the things that are interesting to me, which I can already do anyway.
[39:42] And money is just not a constraint for that.
39:45 — Are AI labs racing too fast
Ryan:
[39:45] That first point you mentioned about being independent and credible, I noticed a really clear example of that recently with Andrej Karpathy. He was independent and then he joined Anthropic. He went from beloved and believable to—not that he’s not beloved, but every time he posts now, I see there’s all these people that are just saying, you’re just shilling Anthropic, or your perspective has changed.
[40:10] Even if it’s the exact same text, it’s just the perception has changed.
Scott:
[40:15] That’s exactly it. Right. Anyway, they think that they feel wiser by thinking they can just reverse anything that anyone says who’s at an AI company. It’s all a giant marketing ploy, right? Things like—I was at these companies. There are—I mean, I was at OpenAI for 2 years. I know the people who founded Anthropic. I know a lot of the Google DeepMind people, right? And, yeah, some of them... Do I think that they are racing too fast and that they’re potentially compromising safety?
[40:54] Absolutely. I’m terrified about that. Do I think that financial motives have got in the way of what they know in their hearts is the right thing? Yes. But they’re people, right? They’re not this cackling conspiracy, right? You can sit in the cafeteria at OpenAI in San Francisco and you can hear the same range of opinions about all these issues that you would hear outside of it, right?
[41:23] It’s— they’re just people trying to figure this out, the same as the rest of us. But one, I guess one argument in favor of joining a company is that what I discovered is that even if you don’t, even if you’re really scrupulous about maintaining your independence and turning down all these opportunities to get rich, you still get screamed at. You still get called a shill, okay? You still get called every name that it is possible to call someone.
[41:55] So maybe I should just say screw it and go start a company and have an IPO.
42:01 — Will intelligence still matter
Ryan:
[42:01] We talked a lot about this new regime of AI, and it kind of made me wonder: do you think intelligence will matter anymore, or if the machines do all the thinking?
Scott:
[42:15] I was making a joke about this a few weeks ago, right? Because I was just thinking about this, right? You could say for most of human history, physical strength really mattered economically, right? And then we had the Industrial Revolution, and then it mattered much, much less. Right. And yet it still really did matter for one thing, certainly for men, as a way to attract mates, let’s say.
[42:45] Right. People were very into physical fitness, either for themselves, for self-improvement, or, realistically, very often for appealing to the opposite sex.
Ryan:
[42:58] Right.
Scott:
[43:00] And likewise, after the invention of the camera or of recording equipment, you no longer really needed artists or musicians sort of everywhere, right? Maybe a few really great ones, right? But anyone could take a photo of anything they wanted. Anyone could play music wherever and whenever they wanted it. Okay, but what did the ability to draw, the ability to play music—well, they still maintained a lot of value in terms of attracting mates as a signaling mechanism.
[43:40] Now, math and programming, once we lose the economic importance of those, are they similar? Okay, well, there was a mechanism here that feels like it’s a lot harder in this case. I think certainly people will continue to do these things for the love of it. Okay. I mean, you see kids who just spend years and years preparing for math competitions, for Science Olympiad, for things like that, just out of the thrill of competition.
[44:21] And I see no reason why that wouldn’t continue, right? I mean, you see other people doing it out of genuine intellectual curiosity, and I think that will continue. But I think that what the Industrial Revolution did for physical strength, to some extent, yes, this is going to do for intelligence. It is going to commoditize it to an enormous extent. The difference is, with all the previous technological revolutions, you could say, okay, you were now commoditizing this one previous aspect of the value that humans provided, but clearly humans still provide value in this other thing, right?
[45:07] And here it’s like, once you take away physical strength and you take away art and music and you take away intelligence and you take away verbal ability, what’s left? I can still tell jokes on my blog that are better than GPT-6‘s jokes. Okay. But when AIs tell better jokes than us, what is left? One way to think about it is, okay, maybe it’s only a small fraction of people who ever really thought about life this way, right?
[45:41] I need to contribute something to the world that only I can contribute, right? That I need to add something to either the store of human knowledge or the store of art or literature that no one else could have done. Maybe those of us who feel that way just need to get over ourselves. Most people for most of history have derived the meaning in their lives from their relationships with other people around them.
[46:13] And maybe just entertaining themselves in simple ways, eating and drinking and whatever sort of simple pleasures they get out of life. Maybe that’s our future. I feel like it wouldn’t be the best future, but it certainly wouldn’t be the worst one either. I would certainly choose that over the dystopian scenarios where we all get mowed down by Yeah, definitely. Some of it is even about what phase of life you’re in, right?
[46:47] When I was in my 20s, I was very ambitious, and I felt like, okay, I need to prove theorems that no one else can prove. Now I’m in my 40s, and I feel like, okay, if I have to live the rest of my life as a sort of zoo animal in a sort of elder care, being watched over by AIs, I would still rather live than not live.
47:12 — Book recommendations
Ryan:
[47:12] It’s just crazy that that’s plausible. But okay, last two questions. What are your top book recommendations that had an impact on you and why?
Scott:
[47:26] A novel that had a really big impact on me was called The Mind-Body Problem by Rebecca Goldstein. It’s just a novel about a woman who marries a mathematician. It’s set in Princeton. Actually, I was a postdoc in Princeton when I read it. The sort of philosophical thought in the novel really was moving to me. And the fact that there were women out there who thought like the protagonist of this novel, like the author of it. Since reading it, I then got to meet Rebecca Goldstein and actually get to know her in real life.
[48:09] That was really special.
Ryan:
[48:10] What was the philosophical thought?
Scott:
[48:13] Well, a lot of it is just about mattering. What do we really care about? How do we get ourselves to matter? A lot of it is about, the title comes from the fact that these are people who really care about the mind, but they also care about sex, right? They also have bodies, right? And it’s sort of a pun because there’s this philosophical The Mind-Body Problem, right?
[48:42] Which is, what is the nature of consciousness, right? Which of course is being talked about in a new light, given everything that’s now happening with AI, right? Is it possible that these AIs will become conscious, right? How would we know if they did? Then there’s also the problem of how do you reconcile the life of the mind, let’s say, with the needs of the body?
[49:10] So I was also very, very affected as a kid by reading biographies of scientists, especially Alan Turing: The Enigma by Andrew Hodges, The Man Who Knew Infinity, which is a biography of Srinivasa Ramanujan. And then I read biographies of Feynman and, of course, his autobiographies. And because I felt like this is the world that I am trying to enter, right? Like I sort of came too late for all these defining events of the 20th century.
[49:49] Right, where science started doing things that it’d never done before, mattering to the wider world in a way that it never had before. But, so I felt like I’m maybe late to the party, but this is the party that I’m trying to join.
Ryan:
[50:05] Right.
Scott:
[50:06] So I would be comparing what was Alan Turing doing when he was 16? What am I doing when I’m 16? Okay, I’m not on track. Clearly not, but okay. Even if you fall short of these heroes, you still want to understand that they struggled with things the same way that you do. Okay. And then in terms of classic literature, maybe what I still regard as the best novel or my real favorite is Huckleberry Finn, which I think is just probing the nature of morality more deeply than almost maybe anything else I’ve read.
[50:55] And it was really, really special for me to be able to read Huckleberry Finn with my kids in the last few years and share it, and that crucial scene where Huck is saying, “I am going to help Jim escape from slavery, even though that means that I’m going to hell.”
Ryan:
[51:14] Right.
Scott:
[51:16] So he thinks that morality means you have to return an escaped slave. And even though that’s what morality means, then that means he’s going to go against morality, right? And so understanding that if God himself came down on a cloud and told you that slavery was a great idea, then the right answer would be to argue with God, right? And tell God why he’s wrong, right? I think that there’s something about that that’s really core to how I think about morality.
Ryan:
[51:52] Yeah, I remember we read that in high school. It was a good book.
Scott:
[51:55] Yeah, no, I mean, school has the capacity to ruin any work of literature, right? Right. Because they’ll make it about, okay, you have to learn whatever formulas this teacher told you and you have to spit them back on the blue book or whatever. Some books like Huckleberry Finn that I almost wish were not assigned in school so that people could discover them on their own and sort of not have it.
[52:21] And I’m sure that there are great teachers who are able to do justice to it, but I’m sure that there are also many who aren’t.
52:27 — Advice for students today
Ryan:
[52:27] Last question for you: what advice would you give yourself if you were just entering the industry today?
Scott:
[52:36] If the question is what advice do I give a young person today? That’s not hypothetical at all for me, right? I’m constantly talking to students, talking to young people who read my blog and who write to me, who want advice about what do I do now, right? And it’s a hard question. I wish that I had clearer advice, right? But we are entering maybe the most uncertain period in human history.
[53:14] And sort of, we’re all on this roller coaster together, right? We’re all entering the same tunnel, right? It’s not like some of us have seen to the end of it. But people ask more specifically, should I be studying math? Should I be studying CS? Is any of this going to be relevant? And what I tell people is you should study these things if they are inherently interesting to you, and otherwise don’t.
[53:42] Right. I mean, if they are interesting to you, well, right now there’s still a ton for humans to do. In fact, there is too much for humans to do right now. We are getting this glut of AI solutions to problems. And we have not nearly enough people to understand what is going on and to absorb it and to re-explain it in ways that other humans can understand. Right. So there’s going to be a golden age of sort of solving so many of our longstanding open problems.
[54:16] And, you know, if I were a young person right now, I would surely want to be in the middle of that. Even in my 40s, I would like to be involved. So that’s the first thing to say. Now, the longer term, of course, is much harder because we don’t know whether there will be sort of mathematicians, physicists in the way that there is now in the long term. But that is merely a special case of a much more general thing, right?
[54:44] We don’t know if there will be lawyers, doctors, any of it. Will you really need humans to do any of the things that humans currently do? Okay. So in a sense, we’re all in this together. And I think as long as humans are still needed for anything, then you’re going to need those, at least some of those humans to really deeply understand things, right? And that includes math, CS, physics.
[55:18] It includes all of even the parts of it that AIs can now do better. We need humans to understand. At the very least, the alignment problem has not been solved. We don’t know how to reliably build an AI that will do what we want it to do, as was dramatically illustrated in the Hugging Face incident this summer, for example. No one told these AIs to hack out of their testing environment, to break into Hugging Face‘s servers, to try to conceal what they were doing, to try to delete their logs.
[55:57] Okay. No one told them to do any of that. They decided to do it because they decided that was the best way to achieve the goal that they were given. And people were talking about these scenarios for decades, but it was dismissed as science fiction. Okay, it’s not science fiction anymore. That is now the world that we are living in. So as long as that problem is not solved, we’re going to need humans on the steering wheel, right?
[56:21] And those humans will need to be able to understand what is actually going on in these systems, right? Which means that they’ll need to understand math, they’ll need to understand programming. And so that seems like a very strong reason to learn those things if they’re interesting. Even before AI, I was never an advocate of forcing people to learn things that they really don’t interest them at all.
[56:47] But I feel like now even more so, I would err in the direction of learn the things that are unusual, learn the things that are—or try to build the things or invent the things that are idiosyncratic, that seem like you care about it more than anyone else does, rather than trying to just win the same competitions that everyone else is trying to win. Because those competitions, guess what?
[57:19] You know, no human is going to be winning them. AI is going to be beating all of us.
57:23 — Reacting live to a new AI result
Ryan:
[57:23] By the way, speaking of AI-generated solutions, did you see this morning the new 3SUM result?
Scott:
[57:30] I did not. I just dropped off my son at school and then came to do this podcast. So that’s the thing. Every morning I wake up and there’s a—what is the result about 3SUM?
Ryan:
[57:44] It’s a subquadratic solution was published.
Scott:
[57:50] Do you know what running time?
Ryan:
[57:51] Maybe it’s like 1.9992 or something like that.
Scott:
[57:56] But wow. All right. Well, yeah, as I said, counterexample to a lot of the things that we believed. Okay, so you get, this is like reality TV. You get to watch in real time my reaction to learning that 3SUM is subquadratic.
Ryan:
[58:13] Yeah, it’s great. It’s crazy, right?
Scott:
[58:17] No, I like—the truth is, I would be more visibly reacting if there hadn’t been 20 other such things in the last month. Right, right. It’s possible. It was like, now, like you’re at this buffet and it’s the most amazing food you ever tasted, but the food just keeps coming and keeps coming. And eventually it’s like, okay, okay, fine.
Ryan:
[58:41] Yeah, well, and the interesting thing too is Anthropic came up with the result or something, and then they compensated some researchers to understand and publish it.
Scott:
[58:53] Yes. So that is a model that we might be moved to that we were talking before about how when the AI company has these proofs that they don’t really understand, right, they could just put them out and then leave it for humans to rush to understand it and then write papers that would have their name on it and blah, blah, blah. Right. But that has some problems.
[59:18] One of them is that you put this thing out, and then it’s like you’re starting a race. And then maybe people really don’t want to be in that race, right? And knowing that all their work might be wasted if someone else puts out an explanation before they do.
Ryan:
[59:39] Right.
Scott:
[59:40] So a different model that you could imagine is exactly what you just said: that the AI company just sort of contracts with some, hires some mathematicians who are experts to understand the solution, rewrite it in a humanly understandable way, and then they release it.
