This is Michael Bolin, the tech lead for the open source Codex repository and a former distinguished engineer at Meta. We talked about his career path, how OpenAI engineers use Codex and the difference between research-led vs engineering-led company cultures.
Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts.
Timestamps
00:06:34 - Overhauling Facebook’s build system
00:16:36 - Rewriting Facebook’s IDE
00:26:01 - Struggles after Principal Eng (E8) promo
00:28:39 - Building a virtual filesystem for Facebook
00:35:47 - Delayed Distinguished promo (E9) and learnings
00:43:05 - Research-led vs engineering-led cultures
00:44:53 - The story behind Codex
00:57:00 - Why Codex’s harness is open source
00:59:50 - Top technical book recommendations
01:05:02 - Why deep technical skills are still valuable (for now)
01:11:07 - How to start projects well
01:14:27 - Advice on writing better and career planning
01:17:06 - Advice for his younger self
Transcript
00:00:56 — Chickenfoot
Ryan:
[00:00:56] I was looking deep into your website and there was something, most everything I could find more information about. There’s this one thing that you seem to be pretty excited about at the time, but I couldn’t find any information about it because all the links were dead. What is chicken foot?
Michael:
[00:01:11] Oh man. That’s strangely relevant because it’s. It was my, my master’s thesis pro project. It was a Firefox extension project, probably one of the very few that was written, you know, in JavaScript for Firefox as a thesis project. And it was, it was actually a little coding tool in a sidebar Firefox. And it was like a little repl. It was called end user. The idea was end user programming for the web.
[00:01:35] And so there were functions like enter and click and things like that. And you’d say enter and you, you know, have pass the string arg and it would like find the box and you’d say click search or whatever. And a lot of the work was all these heuristics that we built under the hood that if you said enter first name, finding the words first in name and finding the text box that was closest and then using that, and then through JavaScript using that as the input.
[00:02:03] And I just think about that now because it’s really funny. We did a lot of work and that’s a lot of what these agents are doing right now is similar, but now actually truly in natural language, not in this little JavaScript replacement.
Ryan:
[00:02:17] Oh, interesting. So it parsed the front end and had this repl where you could say, I don’t know, find the first name field.
Michael:
[00:02:23] And, you know, yeah, we’d use like, you know, the accessibility tags and alt. Alt text for images and we’d, you know. And it worked well. It was really good at Craigslist, actually, because that’s one of the simplest websites you could. You could use. But I had, I had friends who like, made like, automated tasks and like, made real money off of things they automated with this tool, which is really fun.
00:02:45 — Working at Google
Ryan:
[00:02:45] When you entered the industry, you were really excited about going to Google and specifically working on Google Calendar. So what drew you to Google and what was that like?
Michael:
[00:02:55] Yeah, I mean, you know, so I got online in the 90s, right? I remember, you know, browsing the web and it was at a time you’d go to like, you try five different search engines to like, hopefully find the thing that you wanted. And I. And it’s funny because I distinctly remember my roommate in March of 2000 saying, like, hey, there’s this site, this search engine that looks a little bit better. I think it was still Stanford Google Edu.
[00:03:18] And I was like, oh, this is better. And like, you know, and then you started reading and they, they were just so different, right? Yahoo. Was all like, very cluttered and like, they tried to be very sparse and were more, I think, principled about it at that time. And then, you know, like a lot of things, you start seeing people who, you know, who gets a job there, and you’re like, oh, man, they’re taking really good people.
[00:03:38] Like, I want to work with, with those people. And I felt like they just, you know, really got the web. Like, especially at that time when, like, Microsoft didn’t. Around that time, Microsoft killed the Internet Explorer project altogether. And I was like, this is the portal into the web and you guys are, you know, dismantling it. Whereas, you know, Google was way more, you know, web forward. And so between that and like, the quality of engineering and the.
[00:03:59] And the impact they were having, you know, certainly a place that I was really excited to go to, coming out of the school.
Ryan:
[00:04:05] And what was the culture like there at the time? Like, I think in some of your writing you talked about, like, products versus infrastructure.
Michael:
[00:04:13] You know, a lot of companies, especially ones that get big like that, you know, it’s kind of like whatever they’re good at first is I feel like the founders always have a. Have a soft spot for, right? So certainly information retrieval and infrastructure, right, were like, key to, you know, growing that. That company. And whereas I was drawn there at the time because, like, you know, they put out things like Gmail, Right.
[00:04:38] Which, which were vague. Right. But it didn’t, it still I think didn’t have the, the same cachet inside the company. Right. As the search and that sort of thing. And so you know we, when I was working on it, Calendar, it was still like mostly consumer. It was like kind of, it was also sold to enterprises. Right. But like we were not the ones making the money. Like we were probably a call center. Right.
[00:05:02] At that point in time.
Ryan:
[00:05:03] I saw eventually you ended up leaving Google and I, from your post it looks like it was pretty bittersweet. So what left you to leaving Google?
Michael:
[00:05:12] I had been there for four years and you know, I mean realistically like investing was a thing, like finances definitely changed after that four year period ended. But you know, also it was just tough like I, I guess a bad habit at that point of working really hard on things that like really were important to me but maybe weren’t important to Google. Right. So like I worked on Calendar, that was pretty important.
[00:05:38] And then I worked on Google Tasks, which was like a very small feature within Calendar. Right. Like probably at least two to three orders of magnitude fewer users. Right. But I was really passionate about it. And then I was really passionate about the like the JavaScript infra and the, the closure, that suite of JavaScript tools. And you know, it’s great. I mean I enjoyed and I’m proud of all the effort that I put into the things, you know, that I went and wrote the book on Closure because I was so motivated but career wise, maybe not the best move, right?
[00:06:11] You’re like, but I’m doing all this high quality stuff and you see other people getting recognized. You’re like, but I’m working so hard, you know, like that’s, you know, kind of like the harder, not smarter type of mistake. And so, you know, it’s kind of like I should go somewhere or try something else and see if like the things that I’m excited about or the things that like the company I’m working at is most excited about.
00:06:34 — Overhauling Facebook’s build system
Ryan:
[00:06:34] Later you came back into big tech and you started working at Facebook. I understand that you were kind of like a JavaScript expert at the time. And then one of your first big projects at Meta was kind of build tooling in the Android code base. So I want to know the story behind how you got involved in that.
Michael:
[00:06:53] So at the time it was like, Facebook’s going to make a phone, right? Actually there were some failed projects, but it was like, this time it’s really going to happen. We’re going to partner with htc, we’re going to fork Android and do some stuff. So that seemed super exciting, right? As a person just coming into the company and I’d done quite a bit of Java. I was more JavaScript, but at this point, this is also where they called it Face Web, the version of.
[00:07:28] They put HTML5, Facebook on the phone, and that was clearly not working. And it was clear that mobile was going to be the future that was make or break for the company. And suddenly, you know, a friend was like, hey, like, I know you really like JavaScript, but, like, you should really pick Java or Objective C and, like, get good at this or you want to be a product person. And that was really, really great advice.
[00:07:56] And so I was like, well, I like Java, I don’t like Objective C, so, like, let’s go. And so that’s how I found myself on that project. And I, you know, we. We had a very short timeline because unlike almost every other project, there was a hard deadline. Usually, you know, you ship when it’s ready. This was like, nope, we gotta send a bill to htc because they’re gonna, like, burn it into phones on, like, March 1st or whatever it was.
[00:08:26] So, you know, it was really a scramble. But. And the original Android code base was. Or a bunch of. It was also, like, inherited from a contractor that Google paid. Like, it’s like Facebook didn’t want to make a native app and they paid some guy. And then it was like the app in the store and they’re like, here, the code’s yours now. And we should have thrown it away, but we didn’t. And so a lot of things that were frustrating, but also, like, iteration time, right?
[00:08:54] I think for everybody, if you’d been a web developer for such a long time, you’re used to this edit, refresh, and then, you know, like, the Android build system was. Was rough, right? It was this. Some build tools and Ant and it. Like, there were no. There was no way to modularize it out of the bot. Like, we had to hack it up just to even have, like, four modules or five modules or something like that.
[00:09:17] And it was so painful to get anything done that I was like, I need to. I need to fix the build system. Like, I was like, I know I’ve done a lot with Java. I know it’s not like, fundamentally this slow to, like, iteratively, you know, build this sort of thing. And so, you know, Facebook has this hackathon culture. So I was like, all right, hackathon, totally gonna, like, make a new build system. You Know, like unceremoniously in the style of Google’s build system.
[00:09:48] And there’s actually another build system called FB Build that was already a different mirror of, of the Google build system as well. That one’s written in Python, only did C, but yeah, and I was like, either I’m going to make this work or. And like, and then I’m going to be happy to be a developer or like, I don’t even know if I’m going to make it here because it’s just going to be like tearing my hair out, you know, working on this thing.
Ryan:
[00:10:17] If you didn’t fix it you would have quit the company?
Michael:
[00:10:18] Or I’ll at least switch projects or I need to find a way to find a happy place, right, and come into work every day. Like, I came, I want to write code, I want to do work, I just want to, you know, try to do my best work. And so it was funny, actually. I give people credit. A lot of people told me what I was doing was a very bad idea. Almost everybody, except one person.
[00:10:44] I was a senior Android engineer, but nobody said no, you know, whereas I felt like at Google, like, people said no more. And so, so I just rolled with it and, you know, it quickly, relatively quickly, had something that was, you know, dramatically better, like at least like twice as fast or something like that. And so that, you know, I turned a lot of heads and people were like, okay, you know, I guess we, yeah, we’ll go with that one, right?
Ryan:
[00:11:13] Like, I think the interesting thing to me is it seemed like all the odds were stacking. A lot of people who are an engineer notice the same problem, would not have chosen to start the project because there’s an existing one. And then also Google’s got some competing one. Maybe you’re not going to beat that one. What gave you the conviction that your project could beat the other ones and become the default?
Michael:
[00:11:39] Yeah, I mean, I guess a couple things. One, like I said, I’d done other Java stuff. I was like, this shouldn’t be that slow. Or just as a software engineer, this fundamentally should not be as slow as it is. And actually most of the pushback was of the nature. Like, well, if we deviate from the standard thing, we’re not going to be on the standard thing. And what if it gets like 100x better next week and now we’re stuck on your thing?
[00:12:07] Which is really funny when you think about it, because there’s so many things like making their own PHP virtual machine and language. They’ve certainly embraced doing their own thing many times. Why this one was different, I don’t know. I mean it was, it was just, I guess everything about mobile was like, how are we going to make this work? I guess there’s just like a lot of fear and you know, again, like timelines.
[00:12:29] It’s like we have the senior person, like, why it seems sounds like they’re going to go off and do a science project but we have a hard deadline. Like, is that the best use of our, of our time? But, you know, it worked out. It worked out. I will say another thing too was I also tried to couch it a little bit. I was like, hey, I’m just trying to make an Android build system. I’m trying to take over the company.
[00:12:52] I’m not trying to change anyone else’s project because I certainly felt that that was going to cause more, I was going to invite more friction. And so I knew that I tried to make it in a way that it could support more of the company, but I never pushed it. And so I was really heartened when the company, year or so later, the iOS people were like, hey, our build system sucks. You know, can we use Buck?
[00:13:19] I was like, yeah, sure. Like I don’t, you know, come on board.
Ryan:
[00:13:22] Yeah, that, that’s also another interesting because you, you came in and you didn’t have a whole lot of credibility. You were just hired.
[00:13:29] And then you’re trying to build something and get every. And also everyone’s saying, don’t do this. And then you have to convince them, hey, this is the right direction. How’d you influence that change without any credibility?
Michael:
[00:13:44] I mean, I borrowed some. There’s one senior Android engineer who was also ex Google, John Perlow. And he was like, I think he said, right. He’s like, you’re going to do it, just do it fast before anyone gets cancers you or something. He’s like, well, I’ll support you if you get this done. So he was one of the first people and he was a very prolific coder. So he was genuinely very happy to have a faster dev cycle.
[00:13:59] So he was one of the first people to give it kudos and I think that that helped a lot also. But I will say I made some big mistakes on my way in there. You mentioned about having no credibility is like, I came from Google and I was like, oh, this is where the X Bell Labs people are. All these great people. And then you go to Facebook and you’re like, oh, it’s a bunch of college kids. How could they possibly know what they’re doing?
[00:14:34] And a few times certainly I made the mistake. Well, at Google we did it this way and people were like, we really don’t care. And they were right in most cases. Right. That just because it worked there didn’t mean it was going to work here.
Ryan:
[00:14:51] One last thing on this topic is, I mean, what you built was way more performant than anything else by many multiples. What’s the intuition like the technical intuition behind what was done and what you did that made it so much more efficient?
Michael:
[00:15:08] I think the, the big thing was is that I just sat down and looked at the, the tool from Google and I was like, what is it actually doing right? And where. And I think the big thing is that the Google one would kind of just, if you changed anything, it would just start over from scratch. And so that’s why it was so slow, especially for, for incremental builds. And so then I started really understanding like, okay, but what depends on what, because there is these still Archer, like Android resources, which had a very bespoke thing and it was complicated.
[00:15:41] And because it was somewhat complicated, I think that’s probably why just by default I blew everything away and started over. But once I got in there and I was like, oh, okay, we can take advantage of it, if these things don’t change, then we can cache this result from this step and we don’t have to redo it. And suddenly that made things quite a bit faster. And then also just even supporting the idea that you could have more than the four modules that we had.
[00:16:07] I mean, I think every time someone added a module, they had to add like 200 lines of XML of ANT build script that nobody really understood. And so it was not. So no one wanted to modularize anything. Right. You didn’t want to be responsible for that. So a big thing with Buck was making it that it was like a lot simpler to add a new module. And so then that also meant we ended up with more modules and then builds are more incremental as a result.
[00:16:30] Right. So it’s really a change in the mindset.
Ryan:
[00:16:33] So less redundant work, basically.
Michael:
[00:16:35] Yeah.
00:16:36 — Rewriting Facebook’s IDE
Ryan:
[00:16:36] After you solve this build problems, I guess, in Android builds and obviously I went later to further parts of the company, I saw that you started to work more on like the ide. What was the problem that you saw that you saw in ides that made you want to get into that?
Michael:
[00:16:53] So I, yeah, I took, I did a brief stint after buck on messenger, iOS messenger. So I was like, okay, I’ve done Android. Maybe I should actually branch out. Even though I don’t like it and I still didn’t like it. People don’t even realize in Objective C there’s a thing that happens for a long time now called arp, like the automatic reference counting. Nowadays the compiler injects it, but you used to have to actually add code in Objective C to do every reference count.
[00:17:27] Like memory allocation. Yeah, or like every reference that you created. Like, you would have to do it. And, and the. And most people have never seen this type of code, but the iOS messenger code that came in through acquisition was so old that it was still written in this other way. And it’s, It’s. It’s incredibly painful. I guess now you’d have like, Codex cleaned it up or something like that. But at the time we just sucked it up and just Xcode just didn’t feel right to me.
[00:17:52] I didn’t like header files and implementation files. I still don’t. And also the. I would say for Both Android and iOS, you know, Facebook always had like, the biggest app, right? We had like one app with every. With every feature in it, and we, you know, ship it as opposed to Google. Like, they’d have like a drive app and a, you know, a sheets app or whatever, but they also owned one of the platforms, so they could put 20 apps by default on there.
[00:18:21] And so what that meant was, is that Facebook was always hitting the scaling limits of every mobile developer tool before everybody else, which was painful. But actually, as a DevTools person, it was kind of interesting because we got to solve problems that nobody had solved before, and not just as a science project, but because it was real business value. And so Xcode, similarly, we talked to Apple.
[00:18:46] We’re like, hey, Xcode’s not really scaling to our project. They’re like, your project’s too big. You should make it smaller. That was kind of the feedback that we got from them. And so it seemed justifiable to like, go and build an editor. It was similar thing. It was like, I was like, what is, you know, what is an IDE doing, right? It’s like it’s talking to Clang, the language server and that sort of stuff.
[00:19:06] I was like, we could build like a nicer shell on top of that, you know? And then we had started, by that point, deviating from Git and switching to Mercurial as a company. I was like, no one’s going to support Mercurial out of the box, right? And so like that should happen. And then like, oh, actually if Buck is going to be our build system, then like we really like, you know, like that’s Xcode’s never going to do all of these bespoke Facebook things.
[00:19:28] Right. So it seemed justifiable to like invest the time to, to try to like improve that experience. And I didn’t feel that way About Android because IntelliJ was actually like quite good and we had figured out how to make that work at scale. But, but Xcode was a little more difficult at that point.
Ryan:
[00:19:45] So there’s Xcode which was bad, didn’t fit the existing needs. And then there was another IDE that another team was building. I think it was web based. If I.
Michael:
[00:19:56] It was, it was web based. I’m not laughing at that part. That part’s fine. It was, but it was built off of an abandoned Google open source project written in gwt, which is Google Web Toolkit where you would write Java and it would code gen to JavaScript for everything. And I tried to build on what they had. I tried to build some credibility, even actually sped up their build a bit. But again, kind of like looking at the iteration times and also I was like, this is an abandoned open source project, it’s written in Google Web Toolkit.
[00:20:30] And I was like, we are the react company at that point, right? Like how. Why would we not empower people who want to build dev tools to use to build on a, you know, technologies that we are the leader in and actually really like, because, you know, we think we’re actually really good. So I was like, you guys are crazy. And so I started. And it’s similar to the thing where I, you know, I mentioned with the buck.
[00:20:58] I was like, I started it as the, the Java build tool, not the everything build tool. Right. I was like, it was a similar thing. I was like, hey, I’m going to go over here and start this other editor. But we’re just, we’re just going to focus on iOS like you. I’m not trying to take things over.
Ryan:
[00:21:12] Yeah, you didn’t want all the friction. And I mean that team, they had all the existing users though, right?
Michael:
[00:21:18] They had thousands of engineers, maybe a thousand.
Ryan:
[00:21:22] I think eventually leadership sided with what eventually became nucleide, which is what you, what you built. But why did they side with you when you didn’t have any users on your shared profit?
Michael:
[00:21:35] Yeah, I mean, I think it was a combination of two things. I mean I think the arguments that I made in favor of like the technology stack to build on another Was that new cloud was a desktop application. And part of it was, if this is actually going to be an iOS Xcode replacement, like, people are going to want to use it, needs to be able to talk to the simulator or plug in the phone or whatever. Like, in theory, we could go through the web and do all those things, but it just seems like a lot more.
[00:22:04] A lot of extra work. And. And I think the other part is that, you know, I had built up some credibility with the. The Buck thing that they’re willing to, like, take a gamble. Like, well, the last thing, you know, worked out, like, let me try this one.
Ryan:
[00:22:22] I saw that this, in combination with some of your other work, led to your promotion to E8, which is also known as principal in the industry. What was your reaction to that at the time?
Michael:
[00:22:33] Yeah, I mean, certainly I was. I was very excited. You know, I felt like, you know, in the ways that I had been out of step, I felt like, you know, when I was at Google, I was like, this was also. Not just. It was also validation that, like, oh, now I’m also growing, not just, like, technically, but, like, understanding, like, what it means to kind of, you know, do the things that, like, are in line with your employer.
[00:22:57] And so, like, that also just was equally valuable or satisfying.
Ryan:
[00:23:01] I know that Nucleide was open source, and I think Buck was as well, if I recall correctly. What’s the rationale for open sourcing?
Michael:
[00:23:14] Yeah, I mean, let’s see. Buck is the more interesting one. Nucleud kind of didn’t really get adopted externally, I think in both cases. I think, you know, all these companies have benefited so much from open source. Like, I think there’s just a feeling of, like, if it’s not really the secret sauce. Like, but let’s share this with other people, right? Like, I mean, like, Codex as well and a number other things in my career.
[00:23:39] So I do feel like that sharing of information is just good, even if no one uses your tool. Just as, like, seeing as a reference of, like, how a thing could be done could be great, you know, in the better scenarios, you actually get, you know, meaningful contributions back and things like that. And, you know, I remember, like, I think Uber adopted Buck. I think Airbnb did. Like, it was funny, right?
[00:24:05] Like I said, like, Facebook was the biggest app, so we would hit all these problems first, and then this next wave of companies would start hitting these things and be like, let’s check out what These guys did. I think, I mean honestly we also, you know, at Google it was, internally it’s Blaze, externally it’s Basil. So we were open source first and we always, we always did kind of wonder like if we put some pressure on.
[00:24:29] Ultimately they did. I think you’ve gotten a little bit of credit from that unofficially from some of the folks who worked on that. But I think also that it’s also a recruiting thing too or just showing like, hey, if you want to being at the leading edge in whatever area this technology is and you want to do this all the time, not just as a drive by contributor, this is what we do, this is who we are.
Ryan:
[00:24:54] And then this decision to open source, is this, was this a bottoms up decision where engineers said we’re just going to do this or is this kind of also leadership buy in? Hey, this is going to be valuable. And you got like a doc.
Michael:
[00:25:07] Yeah, yeah, I think, I think both cases it was certainly like bottoms up. I don’t think so. You know, things like React and Pytorch, like those are like the, the big success stories right? Where, where the value back to the company is like unquestionable. And then there’s this like longer tale of things where I think depending on the economy and other things, the managers get grumbly if they feel like their engineers are like doing too much open source or things like that.
[00:25:37] So it’s almost almost always I feel like, like bottoms up. I think. Yeah, I would say that’s, that’s often the case, but it wasn’t like met with a lot of resistance, you know, and you usually get like a good conference talk or two and a nice blog post. And those blog posts like, you know, do I would say pay dividends over time? Right. Like trying recruiting? Yeah, like they have like a longer shelf life than I think people realize.
00:26:01 — Struggles after Principal Eng (E8) promo
Ryan:
[00:26:01] Okay, so you got your E8 promo at this point and I imagine the expectations maybe in your mind are kind of going up a little bit and now you need to go and find an E8 problem, I guess. So what’d you do after you got promoted?
Michael:
[00:26:17] I think that’s where I did a. I got a little over my skis and I tried to help with web speed, I think is what I did because again it was a thing that was like a really big problem for the like, like the, just the load time of facebook. com was, was not in great shape. You know, the architecture was a bit stale and it was just, the problem was so big and I didn’t have, I think the A lot of people who had worked on web at Facebook had worked on it for a long time.
[00:26:52] And I had put myself in mobile and dev tools. I actually wasn’t in that world. And I remember I was like, what do we do? I mean, actually I sat down with another person. We started compiling V8 from Source and trying to see if we could change the way that we generated JavaScript, if it was going to be friendlier to V8. We were just trying to whack things. None of which panned out, by the way. And I just.
[00:27:16] It was, it was. There’s different. There’s different people who are, like, good for different projects, and I think that was just not the one. Like, I’m better in a project that involves writing a lot of code from the beginning. And I think a project like that was like, more about, like, looking at data and talking to a lot of people. And, like, I just. That’s not my strength.
Ryan:
[00:27:36] I think you mentioned that at this point in your career, the, the idea of a hero quest. Yeah, what’s. What’s that mean?
Michael:
[00:27:39] The idea that, you know, that it’s like, you know, it’s embarrassing because there’s like some. About ego. This idea that, you know, there’s this like, you know, Gordian not. And all these engineers are like, if only someone would come in and, like, solve this, you know, engineering problem. I was like, I know JavaScript, right, okay, just come in that way and.
[00:28:05] But I didn’t. I. I definitely did not. You know, and I think that it’s been an important learning and I’ve had to relearn it, like, at least one other time that, you know, I can do a lot of things, but there’s a smaller subset of things that I genuinely enjoy doing and that I’m going to be a lot more successful. Right. If I stay in the things that I genuinely enjoy doing. You know, I try to expand that over time, but, you know, we don’t all have to be the best at everything.
[00:28:35] Right. I think that’s accepted and, you know, embrace it.
00:28:39 — Building a virtual filesystem for Facebook
Ryan:
[00:28:39] So then how’d you find the E8 scope problem after that? Like, what was the next day?
Michael:
[00:28:44] That was, I think, a bit of luck as well, I guess. You know, we had. We had. Sometimes we’d have these little summits with, like, smaller groups of engineers about, like, brainstorming what’s going to bite us in the future. I mentioned, I was like, you know, the repo keeps growing, they’re going to have some scaling problem at some point, and a person or my manager Then he became my manager, Brian o’.
[00:29:09] Sullivan. He decided to get some people together to work on making a virtual file system to try to get ahead of that problem. And so we got myself and Adam Simpkins and Wes Furlong, both tremendous engineers, actually thought I was the worst engineer on the project for quite a bit.
Ryan:
[00:29:30] You mentioned a virtual file system on a high level. What’s the benefit to a company like Meta for if you had something like that?
Michael:
[00:29:39] If you have bought into the Monorepo philosophy, right? Put all the code in one repo. Most things that people do need only a subset of that repo. They have to look at the files at any point in time. The idea is that you design all your tooling around this virtual file system so that when you say clone the repo or then you update to a different commit or anything like that, you don’t actually have to like write out every single file in the repo on disk when you make that change, which is.
[00:30:12] Which is the default in your traditional file system, because that is going to grow proportionally to the time with the size of the repo, Right. So at some point you’re going to be very sad. And so there’s actually kind of two, at least two parts to it. One is building this virtual file system that’s like, hey, I know the users at this commit. If the operating system asks me for the contents of the file, I can go get it and it will appear like they had actually laid out all the files.
[00:30:43] The other part of it, which is the part that maybe I was actually better at, I think is then trying to anticipate, well, we have all these tools in our tool chain that are used to just reading all the files. You know, you just rip grep. It just reads everything, right? That sort of thing. And how do we start changing also our development flow and our tools such that they are designed with the virtual file system in mind, because if you just, you know, materialize all the files with your tool now, you’ve like actually just lost all your.
[00:31:13] The benefits that you’ve made.
Ryan:
[00:31:15] So on a high level, it’s basically just lazy loading a huge file system. So it’s more efficient because you don’t need to do everything at the beginning.
Michael:
[00:31:23] Yeah.
Ryan:
[00:31:24] And so that part of this project that you said you’re better at is. Is integrating everything into this on top of that primitive.
Michael:
[00:31:31] Yeah, yeah. So one thing I did actually with Hanson Wang, who’s here actually, I’m on the Codex team with me now, was I was like, oh, okay. You know, the traditional way you have, you know, you want really fast file search in your, in your ide and your editor. And you know, the way most of these things do is the first thing you do is they walk the entire file system to find out what the files are.
[00:31:52] And I was like, well, that’s going to be a serious problem, right? It’s going to undo all the benefits. And then it’s one of these things is like not, you know, first it was like, okay, how can we implement file search? That’s not going to undo all the benefits? And then it was, how can we do it even better than how it’s done today? And so we ended up building this file system called miles short for my files.
[00:32:18] And on a cron job, it would index, it would ingest all the new commits that had come in on trunk and keep track of which files had been added or removed. So just the names of the files, not the contents because all you need is the names. And then Hanson had some clever ideas about how we maintained that index such that we could support fuzzy file matching. So instead of just substring matching, you type, you know, like if you have like a camel case thing, you want to be able to type just maybe the uppercase letters or just your spelling’s bad or what have you.
[00:32:55] And so it came up with this really interesting way to represent, you know, kind of all the files that had been seen at some point and then some bits to represent, you know, if I’m at this commit, was this file present or not present at that point in time? And then also when you ask, I mean, you sent a query to this thing, so you’d send like what commit you’re at. And like if you’ve added any files locally or removed any locally.
[00:33:22] And I think we got this, you know, like, you know, over a million files in like, I don’t know, like 10 to 20 milliseconds or something like that, so that you would be in your editor, you know, typing. And so that was way faster than I think than, you know, like what xcode or like MDS code or something like would give you out of the box. And so that was, you know, so like it was solving a problem for Eden.
[00:33:46] That was the name of the file system originally, the virtual file system, you know, in anticipation, before it was even ready. But then it was so fast and it was available as a thrift service internally. Like people started using it for all sorts of other things. I think when I left, like, I don’t know, I think there were at least like 30 servers running miles like spread around the globe. So clearly that was more than just people like personally searching for files if they needed that much capacity.
Ryan:
[00:34:13] Yeah. I mean when you talked about, I guess the implementation details, most, most people, they, they don’t use leetcode on the job. But that sounds like. I was thinking about the. I don’t, I don’t even know how you input that. Is it like a tree, like a try thing?
Michael:
[00:34:30] So that’s what’s funny that try. Yeah, this was, this was pretty cool in that we had kind of like two parallel arrays. One was like the file contents and one was maybe an integer into that index. I forget. And then we had a 64 bit mask that was like, it was like. So you had like 26 lowercase, 26 capital 10 digits and like maybe dash and whatever. And every bit was set if that character was used at all in the file that you were searching.
[00:35:11] And so the first thing was we could blow through that list and exclude a bunch of things right off the top. But we also very designed. All these arrays were in parallel to each other so that for cache wise we knew it’d be very efficient for the CPU to read memory linearly and then it led itself to. You could just partition that array, it lend itself to parallelism.
00:35:47 — Delayed Distinguished promo (E9) and learnings
Ryan:
[00:35:47] I understand that you worked on Eden and then obviously Myles as well. And then these eventually led to another promotion but prior to that promotion there was some learnings you might have had about influence and conflict?
Michael:
[00:36:05] Yeah, I mean that’s one of the things I guess about being an E8 who primarily writes code. Right. There’s other people. I’d say actually the majority of people that level or higher are not writing code. Right. They’re, they’re actually kind of exclusively spent doing more like influence or working across teams, you know, writing the big Google Doc and getting everyone on board and all that sort of stuff.
[00:36:33] And so you know, as an E8 trying to have that level like, you know, that level of impact that you’re expected to have. It’s like, oh, it’s hard to do that just writing code. So it’s like I need to spend at least some time probably influencing other people. That would probably be good for me. That would probably make my manager Happy. And I think, you know, sometimes you’re just so confident in some insights you have.
[00:37:07] That or certainly I was, that I would just, I, I just came down like way too hard, I would say. And, and that did not go well for me. And yeah, so like, like promo got delayed because I was very excited or anxious about when Microsoft acquired GitHub. And oh yeah, because New Clyde was built on this, you know, primarily GitHub technology. And I was like, that’s going to go away because VS code is going to not make them be a project anymore.
[00:37:40] Which was true. Which did happen. And I was just so anxious that, that this was like now a risk. Right. And I was pushing people, but you know, people were, I didn’t really account for the fact that people were happy with what they were doing at the moment and like, didn’t want their cheese moved, you know, like right out from under them. And so, you know, I got like a little bit of a talking to.
[00:38:06] I, you know, I had to wait a little bit for promo and things like that. But I, you know, I actually even got some coaching. I think after that point, I can’t remember the exact chronology to, to work on that. Like, specifically.
Ryan:
[00:38:18] What’s the number one thing you learned about Coach?
Michael:
[00:38:21] I would say for me is like, I think I’m more aware of like things that trigger me, like technical decisions or what have you and recognize when that’s happening and be like, okay, let’s not act in that moment. Or you know, if I, if I don’t think I can have the conversation like, or I’m not the, in the best place to have it, like, maybe I go talk to the person’s manager instead rather than like, you know, like bull and china shop to the engineer and be like, so I have this thing, I’m thinking about it this way.
[00:38:52] I’m kind of riled up about it. Like, help me. Like, how can I, you know, work with your team or whatever? Like that’s happened to you.
Ryan:
[00:38:59] Well, it’s interesting to me in your career because the promo got postponed and you were, you were seeing that VS code was going to come up and the thing that New Clyde was built on was going to go down. And you were actually right in hindsight too, like, in the future that is what happened and you were battling for what was right, yet you were told, calm down a little bit and this. So I mean, yeah, what are your thoughts when you saw things play out and you go, actually, I was right the whole time.
Michael:
[00:39:33] I did have some conversations, like about A year later and I was like, hey, can we balance the ledger a little bit? Like it ding. Could be pretty hard for that thing. And it was kind of of good and we, we worked it out, I would say.
Ryan:
[00:39:46] Okay, so I guess the, the learning is just how you went about it, not what you were like.
Michael:
[00:39:52] Yeah, yeah, I would say that’s true.
00:39:56 — Joining OpenAI
Ryan:
[00:39:56] You seem to have a great time at Meta and eventually you left. So what was the thing that drew you to OpenAI?
Michael:
[00:40:03] Yeah, I mean a number of things. I mean one was so I, I interviewed at the end of 2023 with OpenAI I believe. And, and I had SPEN Meta trying to do LLM based developer tools, right with there’s a, you know, we had our own license version, even metamate. There’s like a little version of GitHub Copilot, right? That one did like a paper and a talk on that one. It was a code compose, right? That was code compose, yeah.
[00:40:31] And you know, we like now there’s a lot of enthusiasm around, you know, delivering quickly and pushing the boundaries of that sort of stuff. And, and you know, I mean truthfully, I mean, you know, you get feedback on some of these things. It’s like why is this not, you know, GPT4? And I was like, well we’re on like Lamba 2. It’s not the same thing. And you know, I was not, not a researcher, right?
[00:40:59] I was like, but I, and I, you know, just wanted to, to build the experiences, right? And that sort of thing. And, and so, so that was, that was one thing was that like I wanted to build stuff and I wanted to go to the place where I could actually build with the best model, you know. Two again was similar to like, you know, Google originally like seeing the people who were coming here to OpenAI. And I was like, well those are people I really respect.
[00:41:24] I was like those are. I could actually work with more senior people, you know, like Meta eights and nines at OpenAI felt like that I could where I was and you know, third was just like this also seems like. And it has been just a very special place at, at the point in time. So I told a lot of people, I was like, you know, I felt like this would be like the most similar to starting at Google in 2000.
[00:41:53] So not like, not to 1998 but like let’s say 2000, right. Like they kind of got some footing and got some you know, product market fit. And so like just as a personal level that was just a very exciting thing and you know the last one was that funny thing is that when I really enjoyed Calendar because it was consumer and I shipped to a lot of people, I went to Facebook because I thought huge consumer place.
[00:42:18] Ended up not doing consumer at all. Right. Ended up doing developer tools. But like, you know, my users were my friends at work. It was just like 20,000 people, not like a billion people, but it was good enough for me at the time. And then, you know, thinking about OpenAI and then the chance to actually, you know, come back to consumer or at least like have a large user base. Right. And so now working on Codex, you know, I wonder if over a million weeklies or I forget how much it is right now and just keeps growing like, like, you know, hockey stick.
[00:42:50] More like vertical line than hockey stick, even that, you know. So it’s like way more than the 20 to 40,000 developers, you know, it could affect it at Meta.
Ryan:
[00:43:00] Yeah, absolutely. And dev tooling for the industry almost.
Michael:
[00:43:04] Yeah.
00:43:05 — Research-led vs engineering-led cultures
Ryan:
[00:43:05] Meta, when I think of the company, it’s kind of engineering driven. Like engineers are king and queen, I guess. And it kind of like drive everything. It’s very bottoms up from that sense. And I feel like a lot of the lab companies are also like that, but on the research side. So rather than engineer is the first a class citizen, it’s like let’s make sure the research goes well and for good reason.
Michael:
[00:43:29] Right.
Ryan:
[00:43:30] Like that’s also part of why you came is the models are good. But as an engineer or you mentioned, you know, you weren’t doing research. What are your thoughts on that? Research led culture versus engineering led culture.
Michael:
[00:43:41] I mean it was certainly an adjustment, right? I’m not, I mean I think of anyone who comes here and says otherwise is a liar. But you know, if you’ve been at like the Fangs or whatever and. But you know, they are, you know, you talk about impact, right. Which I think is important. Like a real. Like you genuinely, you know, mean it. Like I, you know, I love the work that I do on Codex and the harness and I think we do a lot of very meaningful things.
[00:44:07] But you know, if the model weren’t very good, it wouldn’t really matter what we did, you know, on the harness. Right. And so, so I don’t. So you know, that that’s how it is, I would say. But I feel really great. We sit right next to the research team, we’re really close with them and so that relationship that we have that we get to co develop the thing, that was another reason leaving Meta in the LLM space was like I want to build the product with the people who are building the model so that we can do this thing together.
[00:44:46] Maybe you could do that there, sort of, but not, but anyway, the impact was not quite the same.
00:44:53 — The story behind Codex
Ryan:
[00:44:53] So you mentioned, I mean, when you got here, it sounds like you were working on Codex and starting that project up. So I understand with the initial launch of Codex cli, it was not exactly what you hoped for in terms of how it was received, but later it kind of really all came together. Can you tell that story?
Michael:
[00:45:13] Yeah, sure. It’s been a wild ride. So codec CLI, we launched it in April 2025. It was kind of like this, like one more thing moment at the end of the 04 mini live stream. And so we demoed it live, we open sourced it, you know, Core3Pro at that point, a lot of people tried it out. You know, everyone was excited to try a new coding agent, you know, and it was, and it was pretty good, but it was, it was pretty rushed to get it out the door right there was.
[00:45:44] Which in some ways was good for engagement because now we’re open source. We were getting pull requests, you know, coming in all over the place. And I forget it, you know, we were, I feel like we’re like 10 to 20,000 stars in like a week or two or something like that. So like that, that part was fun. And you know, there were people who did like it. You know, it’s not like nobody liked it or anything like that.
[00:46:03] But then, but that I, I think we didn’t, we weren’t maybe quite staffed to like really drive that the way that we needed to because, you know, we’re trying to cover multiple things as a company. And so, you know, just a month after that and you know, team of like seven engineers and I forget how many researchers launched Codex Web or cloud web where, you know, you’d use Codex in a container or you could just, you know, kick off a new thing from your phone, which is super cool.
[00:46:33] And that was like a more well staffed effort. And I think, you know, that one is still the, I think the long term vision is correct, but I think with a lot of this stuff that we’ve seen, you have to bring the users with you. And I think that was just a little bit ahead of its time and people were still actually more into the local coding agents. And so we saw the web product. There’s a big adoption.
[00:47:03] Initially it wasn’t as sticky, I would say, as we had hoped. And then through the summer, you know, both products still Kept, kept, you know, we’re working on both. But then somewhere that mid summer, again like I said, it’s like, you know, this moment like local agents are still the, I guess the stronger product market fit, but again I still think in the limit personally. Right. Like you’re going to need more machines than just your laptop as a place for agents to run.
[00:47:32] And so we shifted quite a bit over the summer so we brought more people onto the codec. Cli. Yeah, a few kid things bring more people on. You know, GPT5 was going to come out and that was looking really, really good. And I think I was personally excited about, because I prototyped it a couple times before was that in addition to the cli, then we also had enough staffing. We also started working on the VS code extension and I felt actually very strongly that the terminal is good for a lot of things, but it has a lot of limitations.
[00:48:10] Always you make a lot of compromises to make a nice UI in the terminal, whereas VS code we didn’t have to make quite as many compromises. August I think was a crazy month. I think GPT5 came out. I think we released our new refresh terminal ui also the GPT open source model came out. We supported that in the TUI as well. So that was really cool that you had an open weights model and an open harness.
[00:48:38] And then later that month the VS code extension came out and so we were just like shipping like crazy. And that’s really where that confluence of things I would say is where we started to see the inflection point that’s brought us to like, you know, the vertical growth that we’re, that we’re at today. And so that’s, you know, that’s been really exciting. I mean, you know, some of these things you can, you can go in the repo check for yourself.
[00:49:05] Right. And you know, gut check on some of these things in terms of number of people, number of commits and all this sort of thing. And yeah, so it’s, it’s, it’s been quite a ride.
Ryan:
[00:49:17] You mentioned the local versus the remote version of these coding agents and it sounds like you have a lot of conviction that the right long term direction is not the local versions, it’s remote in the cloud. Why is that?
Michael:
[00:49:31] Well, I think about, I think that the people who actually for whom it is sticky now and I think there’ll be more of it is that you imagine, if you imagine you want to just automatically, you know, set for the agent at every like GitHub issue or linear task thing that comes in or anything like that. I mean, obviously, you know, there’s, there’s costs with those things, but. Or you can, it could be abused, but like, you know, let’s say for an internal private repo, right, that you, yeah.
[00:49:56] Want to be able to just have it be a piece in any sort of automation pipeline. Right. And so you can’t. You have all that happening on your, on your laptop. And so I guess, I mean, you know, as a, as an individual ic, maybe we’ll still personally spend more time with the local agent. But in terms of, you know, compute time of agents doing work. Right. I think getting that set up in the cloud, I think, you know, a little bit, it’s like a little more to get set up right now, but once you have it, it’s, it’s quite nice.
Ryan:
[00:50:27] I see. Okay, so you’re not saying that product, the local one will change. You’re saying that like across the industry, the compute that goes towards agents majority will be in the cloud.
Michael:
[00:50:38] And yeah, I mean, even in. I think when the freeze code extension first came out, one of the things was the ability to take the conversation that you were having and hit a button, have it transfer to the cloud if you were set up to do it. Right. So I think that we’ll continue to see that where maybe you’re working on something, you want to throw it over here. You’re like, bring it back when it’s done.
00:51:00 — How he uses Codex
Ryan:
[00:51:00] Codex usage is up 5x since the beginning of the year and over a million people are using it now. I’m curious, has your AI workflow changed a lot since you’ve started to use this newer version of Codex?
Michael:
[00:51:14] Yeah, it has. I’m a much bigger user of the app now maybe than I thought I would be. For a while I was very strongly in our VS code extension. I want that sidebar. I want all the code there next to me. I feel like these things should be together. I’m not a person who doesn’t look at the code. I’m not like, I don’t for projects that are like true prototypes that are throwaway, I will actually not look at the code.
[00:51:45] It’s very freeing. I understand. I’m so excited about it. But for the code that goes into Codex itself, I’m like, no, I still need to look at this. This is pretty important. This is what’s going to affect everybody else. But you start to get a sense of, okay, I’m confident the model’s going to be able to do this change. And this Sort of thing I need to babys it. I’m going to just like write a lot up front.
[00:52:09] And so, you know, I have like four or five clones of the Codex repo of my machine and I, I have enjoyed in the Codex app like the multitasking I just is, is just a lot easier because now you’re just kind of hang out in one window and you know, I try and it’s off the game. Exactly. But it’s like, you know, it’s like how much throughput can I get right. In terms of how many balls can I juggle in the air?
[00:52:34] Sometimes it feels like, it can feel like a little hectic when you’re really trying to context but at the same time you’re like I’m getting a lot more done and that I almost feel a little bad writing code by hand sometimes because you’re like, if I had asked this in the right way. It’s like when you started it was like, oh, I’m just going to change these three lines. And then 30 minutes later you’re like, oh, I can’t kind of, you know, that, you know, we all, we all like to type still, I think, and that sort of thing.
Ryan:
[00:53:08] What percent of your code would you say is human written versus the model generates it these days?
Michael:
[00:53:14] Oh man, it’s model’s got to be like 80 to 90%. I mean, yeah, yeah. I mean like especially like debugging a test or like the CI thing is bad. Like, I’m like, I’m like, hey, you know, you know how to write, you know, print, debug, whatever, like. And that’s great, that’s really freeing.
Ryan:
[00:53:32] Digging into the problems that are suitable for LLMs and the ones that are not. Like what, what do you need to see where you think, okay, I need to go in and you know, write it myself. What’s that? 10% and also what’s in that? 80, 90%.
Michael:
[00:53:47] Yeah, I mean that’s a good one. I think about that, you know, because every time I sit down I’m like, like, should I write this? I mean the answer is almost always no. There’s things that are like lower level and someone, you know, so Codex, the, the harness. The part that I, you know, spend my time on is in rust. And that means that, you know, we can do and we do do like operating system specific things in that code base.
[00:54:15] And actually. And I spend a lot of time on, on the sandboxing. Right. So the thing that really upholds like, you know, the security integrity of what? Of what we’re doing and that the model can’t go outside, you know, the bounds that you set. You know, I, I do more of that by hand because I need to make sure, really sure that that’s all correct. Right. That our test coverage is, is good or sometimes I’ll, you know, seed it and then you know, once I’ve got like the groundwork and like, okay, I have the pieces that I, that I was a lot of feelings about and then could let it like fill in the rest and that sort of thing.
[00:54:50] But, but you know, a lot of like refactors and things like that. Actually, I think like a lot right now is like, you know, I’ll have a build up a big, you know, PR that does too many things. But I know it does too many things. I’ll be like, okay, please, you know, please split this up into reviewable sized commits, things like that. Like I think about how much time I probably spent on that sort of stuff before and now I can.
[00:55:12] Right. Alligate frees you up.
Ryan:
[00:55:15] What about like code review? Is there like what percent of lines of code do you and at OpenAI generally are people reviewing manually or are there agents that are like, you know, imagine you write a test or something. I don’t really need to review that.
Michael:
[00:55:31] I mean, yeah, I would say I liked the approach where you know, the, the agent should do, you know, multiple rounds of review or whatever until it’s confident and that it’s like, you know, worth a human’s time of looking at it. But we still do look at it ultimately before it goes in. I would say generally speaking, I think there’s, we have our agent’s empty file like everybody else does. There’s still sometimes you find a gap in knowledge that needs some context that needs to get added back into there or it’s just things that we haven’t memorialized the agent this but as a human I still happen to know and things like that.
[00:56:13] So we do catch things. But I’ll say actually now that people are also using AI to write their pull request summaries, our summaries across the team I would say are getting way better. So now also when I’m going into review, it’s been reviewed by Codex, there’s a summary that’s like that we actually make sure it has the why and the what of why, the reason behind the pr. So that is certainly, I think helping get through these PRs faster, which is good because there was a lot more review to do.
Ryan:
[00:56:45] That’s, I mean that’s Amazing. I feel like 50% of diffs have, like, you know, almost blank. The test plan says, you know, I don’t know, arc build or whatever.
Michael:
[00:56:56] The commander.
00:57:00 — Why Codex’s harness is open source
Ryan:
[00:57:00] I want to talk about the. The codec cli. That’s open source. Why is it open source
Michael:
[00:57:09] for something that’s that critical to what’s going on on your machine? That’s one of the aspects of open source. I’m not the most zealous about this sort of thing, but I sympathize with it. This idea that, hey, you’re going to put this thing on my machine. I care about what it’s doing, and I think in this domain in particular, it’s really important that people can look at it and have an idea of what it’s doing, because people have a lot of questions about AI agents and that sort of thing.
[00:57:43] And so I think this area, this domain, I think it’s actually really important. And also we have gotten a lot of, I think, great contributions and bug reports and things like that that we would have missed out on. And. And then also I think it’s, you know, just, you know, sharing with the world with, like, how this is done. So we do it through code. Right. I’ve. I’ve put out one blog post about how the Agent Loop works.
[00:58:14] Like there is a plan to do more of that. I’m actually excited to. It’s just, you know, just. It’s just time is this limiting factor. So it’s. The higher. Yeah, it was funny because I had two candidates who came through, and one’s like. He’s like, hey, I wrote it, right? I was like, no, no, I. I wrote it. And another person came in. He’s like, oh, you can tell that. That, you know, that you did not outsource the.
[00:58:35] Okay, good, right? I was like, oh, thank you.
Ryan:
[00:58:36] You know. So, yeah, you mentioned the blog post. How does Codex find what is available to it in its environment? When I’m running these things, I’ll see. It’s kind of amazing. It’s thinking to itself and, like, discovering all these things in my terminal. So. Yeah. How does that typically work?
Michael:
[00:58:55] Yeah, I mean, there’s, you know, there’s a few. I mean, there’s obviously what is, you know, in Codex’s base training, like, it loves to use RIP grep. It uses RIP Grep very well to find all sorts of things. And then there’s, you know, if you have your agent’s MD file and you’re saying, hey, in this repo, like, these tools are really important. Right? You should Use these. Right. Or the readmes or whatever.
[00:59:15] Or obviously if you use MCP and you associate these MCP servers with your, you know, where you’re working on. Right. That injects the set of tool definitions, like at the start of the conversation. Right. So then that kind of. Yeah, so that’s not even like discovery on Codex’s part. Right. It’s kind of just like put it front and center there. I see.
Ryan:
[00:59:38] So some of it’s the harness explicitly throwing that into the context. And then there is a big chunk though, where the model is just doing all the heavy lifting of finding things. Is that right?
Michael:
[00:59:49] Yeah.
00:59:50 — Top technical book recommendations
Ryan:
[00:59:50] Kind of like reflecting over your career the breadth and depth of your work. If we look across all of it, I mean, it’s insane. You were JavaScript front end. Then you have all these dev tools, build fuzzy file, search, virtual. Now you’re working on Codex. I’m sure you had to continue your engineering education to kind of get through all these projects. What are the top technical books that have helped you educate yourself?
Michael:
[01:00:20] Yeah, I mean, you know, so one was this, this book on operating systems. It’s like a thousand pages or something like this. It’s the Addison Wesley book. Trying to remember the author. But it was funny because I was, when I was working on the virtual file system, so I had actually gotten to that point in my career without ever writing C, like undergrad. Like, it was like more theoretical. Like we just never had to touch it.
[01:00:45] And then yet I’m like working on a virtual file system project and now it’s like very low operating system. That’s why, you know, joke. That was the kind of the worst engineer of the project. And you know, and someone said something and I realized I was like, I don’t know what they’re talking about. This is kind of embarrassing. And I was just like, you know, what book do I have to read? And I think my manager, Aaron Kushner at the time was like, he was like, well, there’s this thousand page book.
[01:01:07] And I was like, done. So Pipe bought it, read the thing cover to cover. It, like took it away with me. I took it everywhere until I finished it. It’s. It’s kind of amazing and sad or weird like how, how far many of us can get like in software engineering without really having a clue how computers work. Like, there’s just so many of those levels of abstraction. But, you know, one hand it’s very freeing and then on the other hand it’s a little bananas.
[01:01:33] And I think that what I would say to people right now is, is, you know, actively trying to go like deeper through the layers and understand these things. And like, because many times I saw other people do it and now I can do it, is that there are problems some other people could solve that I couldn’t solve because I just didn’t know that like there was this cruft between these two layers and if you got rid of it, right, you get like a 10x improvement or something like that.
[01:02:02] Right. If you, if you’re just operating so high up you don’t really know, you know, what you can, what you can break down, you know, so that. And so in terms of books, like I’ve enjoyed The, like the O’Reilly, the Rust books, you know, big fan of Rust, but they’re also like well written and thorough. And then honestly, another thing I’ve been telling people that’s not in the book category but probably more fun and I learned a lot actually are these CTFs are like capture the flag, like security type competitions and just, you know, it helps with like kind of like adversarial mindset.
[01:02:41] It helps with. I think they’re usually just like. It’s like a, it’s like a decathlon, like a computer decathlon, I feel like. Right. Because like, like there’ll be multiple challenges and maybe this one you need to like understand assembly and this one you need to understand like what someone’s like janky PHP admin page is doing and all that sort of thing. And it’s. And it just forces this breadth on you in a way that’s kind of hard to generate otherwise.
[01:03:09] And it’s also just more fun because it’s like a game and that sort of thing.
Ryan:
[01:03:12] Can you give some context on what CTF is?
Michael:
[01:03:15] Yeah. So it can be a number of things, but it’s usually a competition usually in like the infosec, the security domain. And there’ll be a set of. There’s like the Jeopardy style one which is like there’s a bunch of challenges and like, you know, designed like crafted ahead of time and they all have point values associated with them. And so, you know, there’s usually some fixed amount of time and it could be individual, it could be with a team and you’re trying to solve these things.
[01:03:42] And there’s a flag. There’s always like a secret, you know, piece of text in there that usually has some format. And the way is that you basically, if you can discover the secret piece of text that means that you have the, the challenge is set up that you would Only have gotten that if you had figured out, you know, reverse engineered or whatever you were supposed to do. And then you get submit that piece of text and that’s your, you know, token to demonstrate that you solved the challenge.
[01:04:07] And so it could be like a race to like solve everything first or get the most points in a certain amount of time or different things like that.
Ryan:
[01:04:14] So it’s like an escape room, but in your terminal and you have only computers to figure everything out.
Michael:
[01:04:19] Yes, I see.
Ryan:
[01:04:20] Okay. And your recommendation is people who want to become better engineers, they should invest in doing some of these CTF because they make you solve problems that make you a better engineer.
Michael:
[01:04:31] Yeah. And you know, and then you, you know, also develop skills and things that you just wouldn’t have. You know, if you’re just like a, you know, writing react every day, like, you’re probably not gonna like open up GDB and like reverse a, you know, a tic tac toe game. But like, I did that because of, you know, a CTF challenge was to do, you know, exactly that. Right. And, but then it’s like, but then I learned how to use GDP and like, and then, then you start, you know, when you’re faced with other problems, you’re like, oh my.
[01:04:58] Just toolkit of, of ways to solve things is just much broader now.
01:05:02 — Why deep technical skills are still valuable (for now)
Ryan:
[01:05:02] A lot of people, especially earlier in their career, they see the power of Codex doing everything for them in the terminal and I just imagine them saying, oh, well, I don’t need to learn gdb because Codex knows it.
[01:05:16] Wwhat would your advice be given the landscape of these AI tools for people thinking about their engineering education?
Michael:
[01:05:24] Yeah, no, that’s, I mean, I think everyone’s like struggling to answer that question right now. I do think about it, you know, a lot and I, I don’t have a great answer. I, I, I kind of personally come back to my thing about like, I still think like trying to forcefully kind of go through the levels of, of abstraction and just understand, you know, more like at a deeper level how things work is going to be important.
[01:05:51] I mean, I think this will change, I’m sure this will change over time, but right now it’s still kind of like, you know, the questions that you asked the agent is going to affect the quality of the thing that you get out. Right. So if you’re not asking the right questions, you’re not maybe going to get the best engineering solution out the other side. You know, as things go on, perhaps that will also be another Layer that’s removed.
[01:06:15] I mean, I think that’s certainly where we’re going. I don’t know what time frame we’ll get there. Things do seem to be happening faster than we expect. But yeah, I think in general learning how to ask what the right question is. And I haven’t for myself even totally pinned down what that means for someone who’s starting out new. I fortunately have experience to, to fall back on or like that’s where that, that, that taste or that intuition of what to ask has, has developed.
[01:06:45] But you know, if you’re, if you’re starting out, I’m still not sure yet. And it’s also hard to say because we, we don’t really know a hundred percent where everything’s going.
Ryan:
[01:06:54] Yeah. When you reflect over your career, the expectations at these really high levels is kind of crazy. I mean it’s like already for Most people this E7 or senior staff level is this unattainable level of impact. So someone who gets promoted to that level is thinking, I gotta really work super hard now because the bar is like up here. And then you went two levels past that. And so I’m curious in the day to day now, what are your thoughts on these crazy expectations?
[01:07:28] Is this stressful for you?
Michael:
[01:07:30] I mean it never, it was never not stressful, I think for me because I really, I think also because I sat in calibrations, right? And I, you know, so for a bunch of other people, you know, you talk about their level, you talk about their impact and you know, you want to be really fair and have this integrity around. Well, this level means this thing and this is the impact.
[01:07:51] Like that’s what it meets all is, that’s what it exceeds is. And then knowing that like then you’re, you know, playing it out in your mind like, well, someone’s sitting in my calibration and talking about this thing and they want to be fair, right? And it is a little scary. Like, you know, you get to E8 and you’re like, oh, you know, it’s a D1 or a director and you’re like, how do I have as much impact as a person who has maybe like over a hundred people in their org?
[01:08:13] And so that’s hard. A lot of people do it again, a lot of people who are an IC who do it, do it more by a different form of people management, right? Where they’re trying to, you know, like write the right doc and get the people aligned and do that sort of thing. And the reason that they do it and are Not a director is that oftentimes the people I’ve talked to, people who are in this, like, that they’re like, well, I have this technical credibility or because I built this thing, like when I go to the team and talk to them, it lands differently than if like a engineering director does or some version of that.
[01:08:46] Right. Happens a lot. And so then, you know, when you, when you say you’re influencing, you know, 50 to 100 people as a senior IC, then you’re like, oh, okay, that’s like, you know, D1 impact, you know, whereas as if you’re a coder, I see. Being really thoughtful about the projects that you pick. And it can’t just be like, this is fun for me type of project. I mean, if you actually care about not getting fired or you’re getting like Sam eats all or better.
[01:09:22] And even when I would start a project, certainly like the later I went on, I would think a lot about like, okay, like I maybe there’s this feature, I just want to write it because it’s fun. And I’d be like, no, I should let somebody else do that and think about, okay. I mean, it’s kind of like with Codex now, like, what code should I personally write to maximize my impact versus if someone else could do it about as well as me, let’s say 80% is good, I should probably let you know, have someone else do that.
[01:09:51] But even then, if you’re leading a five person project, is that still, you know, still getting to E8, E9 impact still hard. So really finding that project that’s a force multiplier. So, you know, like the virtual file system was a really great project because that was going to, you know, we knew that down the road like this was really going to unlock so many things, prevent us from being completely blocked.
[01:10:19] Right. Actually, another big part of it is, and you know, one of my managers talked to me about it is, you know, not actually recognizing senior managers enough who are the person who pairs the senior IC with the right project. And because some senior engineers are amazing, you know, fixers, coders, but they’re not the idea come uppers with, but they’re the person you want for very difficult technical projects.
[01:10:54] I don’t want to out anybody here, but I have people in mind. But a lot of times it’s the manager who realizes, oh, this project needs this person. And that person would have never realized it themselves.
01:11:07 — How to start projects well
Ryan:
[01:11:07] One of your old colleagues, Adam Ernst, I think. Hi, Astemia. What are some other engineers that you admire and why? And you know he mentioned your name, of course. And specifically he talked about your ability to start projects was really good. And obviously, I mean, look at all these projects, right? I mean, a lot of them, you created them out of nowhere. Like it was just, you had an idea and you went off and built a prototype.
[01:11:30] You came back and you were very convincing that it was a better solution. Do you have any advice for engineers who they have a problem and a solution and they want to like build a project from scratch?
Michael:
[01:11:41] I mean, I think, I know a lot of good projects, I think come from being a little bit dissatisfied about, about something. Right. And I think, you know, it’s funny, like sometimes I, for better or for worse, I was just so charging ahead and just like building the thing that like, not really thinking about like what the best way to do it was. So actually a really funny example is Google Calendar going back really far.
[01:12:06] I was like, I want weather. I want to have weather, little icons showing the weather in Google Calendar. And I was like, I’m going to do it. And I had mostly been JavaScript especially and not done any of the backend stuff. And I just charged through and I cobbled things together and made it happen. And then my tech lead was like, wait, how are you storing that information about the weather and stuff?
[01:12:30] And I was like, oh, I just threw a blob of XML and everything. He’s like, we should have talked about protocol buffers. I was like, binary formats to save bytes. That I was just so set on weather of all things that I never bothered to ask anybody if there was a better way to do what I was doing. I was like, it’s done.
Ryan:
[01:12:51] So I guess that’s a unique skill of yours, is the digging into the dissatisfaction and solving your own problems. Seems like almost every single one of these projects is I want my, I want something to happen. This, this shouldn’t be this way. And then you went and you solved it.
Michael:
[01:13:08] Yeah, yeah.
Ryan:
[01:13:10] Well, I think another unique thing that I see is a lot of people, they, they get to work and they, I don’t know, they’re, they’re in their dev environment. I’m sure there were many other people who saw with the buck story, for instance, they go, oh wow, this build is really slow. Oh well, like I guess I’m going to go to the micro kitchen and get it and call its builds. What gave you the confidence to know that you could make it so much better?
Michael:
[01:13:34] I mean, that one was, you know, I have to credit, like having been at Google, I never worked on the Blaze Team. I never knew how that like worked I ever touched the code or anything, but I was like, I know that there’s a thing out there that has this shape and is a lot better than this thing. So that’s an existence proof whether I could do it or not. Like tbd. Right. But I think that was helpful on that one.
[01:13:59] There’s a lot of, I guess, prior art that gives me confidence in things and I usually generally identify myself as a coding machine. I guess with Codex now everyone’s a coding machine, but that I felt that I always had confidence. Build a prototype quickly and at least answer that question or test my basic hypothesis spiritually. Should there be a way forward to this thing that I think should exist.
[01:14:23] And generally, you know, you’re, if you are determined, you find a way.
01:14:27 — Advice on writing better and career planning
Ryan:
[01:14:27] Yeah, I’ve noticed that pattern. A lot of people who go to big companies, they see this world class infrastructure and then they go to the other company, oh, you don’t have this, you don’t have this. And they build their, you know, those new versions of it. I’ve read a lot of your writing at this point and it is so clear. It’s some of the best examples of good technical writing. What advice would you have for engineers who want to write better?
Michael:
[01:14:52] I mean, I think a lot of it is, well, I think reading other good writing is certainly a good start. Right. Maybe, you know, consciously or subconsciously start to pick up patterns of like, what, what is out there? I think, you know, really high level thinking about like what what is, what is it that I’m trying to convey? What would someone actually really want to know? And, and outlining a lot up front right is I think a lot of people will give this feedback.
[01:15:17] But it’s, it’s impressive how really important that is. And just being like, does this set of things like linearly follow? I think is a big thing. And then asking yourself, oh, I went from this point to this point, was that too big of a jump? Is there something that somebody actually would have reasonably missed? And I think personally I feel like that’s a reasonable feel for where that gap would arise.
[01:15:44] And then if you can kind of anticipate that and magically put in the example that like someone would have that someone needed to make that jump, I think like that, at least for technical writing is like, is a big deal.
Ryan:
[01:15:59] You have that career note that I love and at the beginning of it you lay out this three step plan for impact. Can you explain that three step. I feel like that’s a good algorithm for people to use in their careers.
Michael:
[01:16:12] Step one is figure out what you really like to do. Right. And you know, as I mentioned, like, it’s good to broaden that, but it’s also good to, to be honest with yourself so you don’t know. And like the, the quote, the hero quest that I went on, that, that didn’t pan out because I was working on stuff I didn’t really, you know, truly love. And then two, step two was figuring out what your employer is, what’s really invaluable to them.
Ryan:
[01:16:37] Right.
Michael:
[01:16:37] And as I talked about at Google, I, I didn’t do a good job of that. I did stuff that I was really excited about, but it wasn’t, you know, it wasn’t, you know, AdWords for Google or anything like that. Um, yeah. And then step three is like, find that intersection and then just really lean into that. Um, and you know, the more that you can do that, I think the more successful, you know, you’re going to be.
[01:16:58] And, and, and the, I think the challenge is sometimes it’s not always there. Right. And maybe you have to go, you know, find somewhere else to make that happen.
01:17:06 — Advice for his younger self
Ryan:
[01:17:06] Okay, and then, yeah, last question for you is if you go back to yourself at the beginning of your career, knowing everything you know now, what advice would you give yourself?
Michael:
[01:17:14] I think I should have been open to learning more things sooner. And, and I, I, you know, to be a little gentle to myself and I think other people are in that situation is that, you know, there’s so much to learn when you’re starting and then whatever your first programming language is. I think it’s funny, I think everyone has a soft program spots for, they’ll like make excuses for it like ever. Like, oh, no, it’s a totally good language.
[01:17:37] And I think it’s because it’s like the first thing that enables you to do a thing, you know, do anything, right. And, and then it’s like, oh, okay, I can finally do something. It’s such a relief. And, and, but it’s also like a hazard because like then you kind of want to hold on to that thing because like you’re finally productive and now you’re like, ah, it took so long to get to this foothold. I don’t know how long it’s going to take to get to the next foothold.
[01:18:00] So I think like, in my particular case I probably did maybe went too deep with JavaScript. And like, like I said, it took a long time before I wrote nec. And I think, you know, if I had been a little bit more curious and a little more flexible in terms of what, like, types of projects I was willing to take on or things I was willing to learn. You know, it came eventually, but I think if that is probably the biggest thing that maybe could have made a shift for me earlier.
Ryan:
[01:18:26] I gather from your story there was a point with the xcode where you said you hated Objective C and then you were coming up with ways to compile the Objective C into Java into Objective C or something like that, maybe. And then also talking about the C for miles, it seemed like a very concerted, okay, I’m going to learn this as opposed to just kind of being open to it. Yeah, yeah, it makes sense. Well, maybe with Codex in the future it’ll be less of a hurdle for people.
[01:18:59] You could just kind of say, hey, I know JavaScript, write this in Rust or whatever.
Michael:
[01:19:04] No, it’s true. Opens a lot of doors. Sure.
Ryan:
[01:19:06] Awesome. Well, thank you so much for your time, I appreciate it.
Michael:
[01:19:08] All right, thank you, Ryan.
