S3: Today, I'm chatting with three of my AI researcher friends, from whom I learn a lot every time we talk, and who also happen to be at somewhat open-ish labs and companies, so you guys can actually say things on the record. I'm joined by Baron Millich, who is the CTO of Zyphra, which is
The first question I have, if we're in 2036, it's been 10 years, and we don't have like billions of crazy superintelligences that are running around that have like radically transformed the world, what is the most likely reason that that doesn't end up being the case? Other than sort of exogenous political shocks or like there's a war or they banned AI or something. But what is the most likely technical reason that we don't, like 2036 isn't like a crazy alien superintelligence world?
S4: I mean, like my reason would just be like, it's got to be that sort of there's been a classic thing, almost like Moravec's Paradox, where we think of the AI being like, if it can do this, it's going to be amazing. If it can solve these hard math problems, if it can win at chess, blah, blah. And then it solves these things, and then it's not that impactful. Obviously, it's somewhat impactful, but not everything. If somehow that continues and there's never the true spark of generalization that occurs, I think that could lead to the AI just being extremely good at everything that people put into a benchmark, put into an environment, but there are still some persistent Sim2Real, which is something I'm hard blocking everything. I think this is kind of unlikely. I think we do actually see this kind of generalization even from RL in practice already. But if it's just ridiculously hard to generalize meta-learning, plus we don't solve continual learning, it's just super hard and impossible. This would be my default scenario in that case.
S1: Yeah, I agree with that. Humans have a lot of advantages over models now, and each time a new model comes out, it'll sort of, it'll catch up in some of these areas, but like you end up getting. So there's this cycle that keeps repeating where people think, where a new model comes out and people are blown away and they're like, this is AGI, but then they use it a bit and then it starts to feel dumb after a month or so. So that cycle just might keep going and it's hard to predict how many times it's going to repeat. And like right now you don't get explosive. Yeah. Yeah. Yeah. Yeah. Yeah.
S2: For me, it's like a question of how far off like this global optimum of a learner you could have on a chip is like the transformer plus like RL, basically like the current recipe. So like, I think people imagine that once you have an agent which is better than all humans at AI research, even if it's 0.1% better than all humans, then the fact that you can run hundreds of thousands, if not millions of these in parallel, you can run them much faster, chips can speed up. That's going to outweigh every other bottleneck and you're eventually just going to hit this very fast takeoff with recursive self-improvement. I could imagine that if we continue along the trajectory that we're currently on with that paradigm where it's basically just self-attention, RL, scaling up RL. If you think about what happened with Moore's Law, we had this very nice straight line and that held for a really, really long time. But there were so many discrete discontinuities and innovations that had to happen to keep that scaling law going. And the same thing has kind of happened with LLMs. We had this pre-training scaling law, and then that was kind of hitting the diminishing returns. And then we came up with RL and solved that. And then we got this new diminishing returns curve to hit that made it keep looking like a straight line going up. And so if it requires another one of those discontinuities to solve, I'm not sure that the current method of training LLMs with the aural environments, even RSI-targeted aural environments, would be able to discover that discontinuity. And if not, we're probably going to hit this asymptotic curve.
S3: Sorry, but do you think discontinuity will be harder than anything that's come since 2012?
S2: If we had the answer to that, we'd kind of have the ability to implement it. But like maybe we should distinguish between a discontinuity that adds to the current paradigm. Again, it's like cumulative. Like there's something beyond the RL that we have to discover and maybe they're capable of like, you know, connecting the dots in that straight line. Or, again, how far off the global optimum are we? Do we have to go back and throw out gradient descent and neural nets in general? And I don't think if you continue to scale up the current paradigm, an LLM, no matter how many LLMs you're running, are capable of necessarily discovering that if it's too far away.
S3: Yeah, the only hope really is if deep learning just can't get us to an AI which is at least... Can dominate human research and human development, including the human ability to come up with new paradigms and so forth. Or like, I don't know, maybe humans would also never have discovered the next learning architecture, but to the extent humans could have discovered it eventually. But it just seems like, I don't know, if you just look at the progress that's happened from 2012 till now, and you just continue on that, I mean, I know that's been powered by huge amounts of compute scaling and so forth, but it would be weird if like. Yeah. Yeah. Yeah. Yeah. Simulations which incentivize getting better at not only AI R&D, but generally at science. So this is a thing that all the labs are targeting, many startups are targeting. Or another intuition pump is if you look at the ELO score of chess bots since the 80s, there's just like a very linear increase in ELO over time. But there's this huge discontinuity as they cross the human range of human experts always win against AI to like human experts never win against AI as this linear increase in ELO happened. And you could think, I agree with your point that so far, AI capabilities have not been that big of a deal in terms of their end economic impact in the world. But that's because they're slowly rising in ELO relative to humans. Yeah.
S4: Yeah, I mean, I agree it would be very, I mean, the only way for this to not happen is if, like, as you said, somehow asymptotes, like, just before, basically, because we're already pretty close, in my opinion, to, like, where we'll start crossing, like, the human ELO score. And so we'll need to asymptote before that. And, like, that's the only way, you know, in the scenario you pose where, like, somehow we're sitting here in 2035 and, like, everything is normal for this to happen, I think. I mean, the only other way is, like, there's, like, some dramatic, like, regulation on AI. It's like, this is kind of what I see as, like, the most likely way for this scenario to happen, actually, rather than the technical thing.
S2: Yeah, I think there are different kinds of research. There's like research where it's like the auto research style where the objective is already specified very cleanly and you're optimizing that objective. And I think everyone is picturing like if we continue along this path of like, you know, making pre-training loss go down, making our environment I was in primary school.
S3: Actually, John, I'm curious where you're like sort of wisdom of the ages or wisdom of like being in the trenches way back when. But presumably a big breakthrough was realizing that next token prediction is the, like you wouldn't have thought that nanoGPT speed run is the thing to be optimizing for in 2014. But now that we have come to this new paradigm, you wouldn't think to do a speed run on that and have you guys get really good at that. But maybe there's like an next... Inner loop to optimize that the AIs wouldn't anticipate. And there's an outer loop of revenue or something that eventually should be strong, but it's a very slow outer loop.
S1: Yeah, in fact, I remember in the early OpenAI days having the intuition that actually just minimizing log loss wasn't going to get you into intelligence because the important bits are accounting for such a small fraction of the loss that it was going to be overwhelmed by noise. So just training a language model and next token prediction just wasn't going to learn the interesting things you wanted to learn. And we needed to craft better objectives that would put more emphasis on the important things. And like, you can make all sorts of arguments for this and you could say, oh, humans probably don't learn how to like... We don't learn how to model everything in our environment. We can't, most, like people can't create a photorealistic reproduction of some kind of scene they've looked at. So there must be, we must need a better objective, but then it turned out that it just worked anyway.
S3: As you were pointing out, the inner loop, even in current AI research of like post-training benchmarks or whatever, doesn't necessarily translate into what users like.
S1: Oh yeah, I mean, the whole field relies a lot on generalization, and it's very hard to predict when you're going to get generalization or when you're going to get some kind of out-of-distribution generalization. So we know that if you train on the task you care about, you're going to do better. But the most important advances are often the types of generalization that we have no right to expect. So, for example, from just pre-training on this very naive... Next token prediction objective to various tasks of interest that require understanding of the input in some deep way or learning some skill from pre-training that's very rare and not very heavily represented. And then also generalization from the severifiable tasks to less severifiable ones. This is also a type of generalization that there's no reason a priori to expect it.
S3: Yeah, yeah. So this is an interesting question because one intuition pump that you could have for why you would see some sort of singularity very rapidly without even scaling up the inputs to your progress that are necessary. Yeah. Thinking about what is the optimal experiment to run, doing small-scale ablations, developing literally a century's worth of theory. So going back even before deep learning, before you decide what experiment to run, doing extremely optimal setting up of the experiment. Then you do a century of thinking after the experiment is over, where you're analyzing what happened and what the next experiment to run is.
S1: Well, I think if... If you think hard enough, you probably could have expected some of these things beforehand. There is probably some very clever way to do a small-scale experiment that'll let you build the theory that then will generalize to the large-scale experiment. So I would expect that we're nowhere near the ceiling of how well you can do research. And I would imagine a future where AI is doing a lot of... Analysis and theory building, like spending a comparable amount of compute to the amount that you're spending on the experiments themselves, doing various kinds of analysis and building a theory around what we've seen so far.
S2: I think there are really concrete examples of this when the objective is well specified. All thinking can do is update your posterior based on the bits that you've gotten since you formed your prior. You can't gain any new bits from just thinking. But when the objective is well specified and there is this data sitting around, I imagine there will be this big speedup in the current paradigm we're in. And a good example of this is if you've got an AI to think about the Kaplan scaling laws, an AI at this point would have noticed that they've just taken these intermediate checkpoints and didn't account for the annealing, and so this is wrong. That would have caught that. Years earlier, we would have made progress. Like would have cut off a year or two of progress just from that like observation from an AI. And like, again, once the objective is well specified, which is like lower pre-training loss or whatever, like there are many, many good examples where if you just thought about it a bit more, you would have been able to like cut down a significant amount of things that you've done. So like MUP and like how learning rates scale with like model size and like real realizing the model width is important in that as well. Like I feel like you can really back out a lot of these things and cut off like a lot of like hanging fruit. I would imagine like a 10x speedup if our thing is just like maximize the objective we're currently on. But I don't see how that generalizes at all to, you know, come up with the right objective in the first place. Like just thinking doesn't necessarily buy you the right objective in the first place.
S4: I mean, yeah, I think this is really the key question to any kind of very rapid RSI from current AIs. It's like, how well can AI generalize to learning its own objectives? Because to have any kind of self-propelling automated loop, you need the AI to propose objectives, optimize them, figure that out. Proposing your objective and like having this like not go off the rails at like any point for like a long, long time. Coming back to Moravec's Paradox, there might be like a case of Moravec's Paradox where like we think this kind of like autonomy and sort of like being like self-encapsulated so we can, you know, think of what we should do ourselves and then go do it and like have this loop is like super easy because we always do this. And like obviously evolution needs to create creatures that can like survive on their own for like long periods of time. And like this just might be something that for some reason is like really hard for the AI in the same way that like locomotion stuff is really hard. It's like math is super easy despite being super hard for us. I don't know.
S3: Doesn't the time horizon increasing suggest
S4: that that's- Yeah, exactly. I mean, this is another possibility, but I agree, there's no obvious evidence for this. In fact, the fact that RL is not super persistent and it's quite easy to do this is kind of evidence against this. But this would be potentially one of the reasons why we just don't get this immediate takeoff is if this is hard.
S3: If you look back from 2012 till now, or maybe from when you started doing your research till now, what part of all the innovations that have happened since that time, including purely engineering ones, including purely conceptual ones, what seems like the thing that is the thing that would be the last things humans would have to do before AI totally automated R&D?
S4: Probably just like iteratively asking the right questions. Like if you can get the AI to like do any experiment, but like you need to decide what experiments to do. And like right now, I think AI is not very good at this compared to coding experiments at all. Like whenever we talk about research, they propose like a bunch of like miscellaneous things, which are like very, very tiny steps.
S2: Or even going from DeepMind's approach of we're going to solve intelligence by learning to play games at a superhuman level. That's going to be the approach to another researcher like Radford being like, I'm going to try and just predict the next token off a very wide swath of data. And then even once Radford had discovered that, it took a while before people decided to scale it up because we had to come up with the idea of scaling laws and the fact that you could very reliably predict these things.
S1: I would say that the last job for humans or the role for humans that'll last the longest is like defining the objective and like deciding what we actually want. So like in that vein, something like deciding how the assistants should behave or what it means to be helpful or what the objective is when we're doing our All From Human Feedback is one such thing. And then later defining constitutions and model specs is another one. And I think even if the AIs can do all the technical work, we'll have to still do a lot of that and decide what we actually want.
S3: Alignment is the final job.
S1: Alignment is sort of the answer, but it's also alignment itself can be kind of decomposed into specification of the objective or figuring out what the right objectives should be. Actually achieving or optimizing the objective you've defined. And I think the first one is not going to go away anytime soon. And if I think about a post-training team and why you need a lot of people to be on the team, it's just because there are a lot of different areas where you have to figure out how the model should behave. And there's no way of... It would be very hard to automate the whole thing just because someone has to think about how should the model behave in this area.
S3: Jane Street started using Antithesis to test their software in early 2025, and they were so impressed by the product that they decided to invest in the company. I recently caught up with Ron Minsky, who co-leads Jane Street's tech group, to ask about how Antithesis actually plugs in.
S2: The thing that What I think is most impressive about Antithesis is we started using it in a team that was building high assurance software and being really careful. And nonetheless, it was able to shake out bugs that were otherwise going to be really hard to find. And that's important both because it helps it helps make those systems more reliable, but also because it helps
S3: the teams that build it to just move faster. This matters more and more as code production is increasingly automated. I think in general, as we've been using agents more and more, the key problem that you
S2: run into is the verification bottleneck. The time it takes from people to look at code and figure out, is that actually something you want to accept in your production software? And tools that make testing better are just incredibly helpful there. They just ease the verification bottleneck and make it possible for you to get more stuff done and move faster because you can have more confidence that
S3: the code generated by the agent is actually not introducing new problems. To see how Antithesis fits into your development process, go to antithesis.com/dwarkesh. What is the story for why there isn't huge consolidation in model providers? There's just so many things that point to centralization here. If you step back over the course of years, is there something that is going to prevent that?
S1: Yeah, I think distillation is the main thing that fights against the centralizing force because basically anything that can be learned through RL can be distilled very easily because it's a small number of bits. It's something that you can learn from a small amount of data. So if you can get trajectories from the model that show a behavior, you can easily distill it. So I think distillation is one of the... Things that fight centralization. There's also... I mean, there is a possibility that there will be company-specific models, that it'll be possible to learn from deployment and have a company continually improving its own model. And such a system could be provided by the current oligopoly of model providers or some other currently smaller company. But I think that'll change the game a bit.
S4: And I also want to point out that continual learning doesn't stop distillation. Even if your model is improving every day, people could be distilling it every day. So the loops could just operate at the same pace.
S3: Right, that makes sense. Okay, so copying model behavior, I guess you need to know yourself what the right distribution to prompt in order to get the relevant model behavior.
S1: Oh yeah, for just distilling with supervised learning, the prompt distribution is extremely important. So it's very non-trivial to distill a model, even if you have full access to it and have the cot, the chain of thought and everything. Yeah, it's non-trivial to distill all of the useful capabilities from it because you need to prompt the model with something. You need to prompt it with realistic prompts. You need to have a really wide distribution of realistic prompts. So yeah, one thing that's been coming out recently is some of the Chinese companies are probably using these router services which are designed to allow people. So I think this is like a very useful dataset for people in China. Distillation because it gives you the perfect prompt distribution. Yeah.
S4: I think this is one of those things where AI is helpful out there. Like if you actually look at like, you know, the frontier pipelines or say like the Chinese models that they actually put in their papers, it's a lot of like humans or like they get seed prompts from somewhere, which is some combination. And so it's like you can automate an awful lot of this prompt distribution gathering and environment creation. It's just like humans need to provide increasingly fewer amounts of bits as the models get better.
S3: It still seems you're bottlenecked by having a service which has users or users are going through.
S4: Not necessarily. I mean, yeah, that's obviously very helpful. But theoretically, you can just think about what users want.
S3: The whole point is that the user says, make me an application like this. Oh, that didn't work. I actually want you to make this new feature. But actually, let's step back and do this other thing. And capturing that whole trace is the... Or to the extent you could have done that anyway, then you just have like RSI.
S4: Yeah. I mean, like ultimately, like if you have this like fully automated loop, that is basically RSI, right? Like the AI is deciding, the data is deciding, the training, that is the loop. But yeah, I mean, like it depends how much human information you need. Like at some point, if you're just like, I want traces that look like this, you prompt that to the model, the model will be able to like come up with like a pretty good approximation.
S3: But what if you want to do like, make me a really good politician and then it has to be like anticipate de novo. How would a discussion in the Senate halls go or something? I just feel like there's going to be a lot of things.
S4: Ironically, this is actually, I think, easier for the distillers than the frontier labs, right? Because the distillers just like, I want a good politician. They go to the frontier model. The frontier model already knows how to be a good politician. So it just generates those trends. Whereas if you actually want to build the first model that does this, you have to actually somehow get data on what politicians do every day and build that. So it's actually much easier to say, I want something like this and then get the AI to produce a billion variations than to actually create the thing like this to begin with.
S2: I think you can actually make a really concrete prediction based off this observation that the Chinese labs have this router data. So I think the thing that Jess stated this originally was I was saying, wasn't it weird how Sonnet 5 and Opus 5 are like all almost objectively worse models than GLM 5.3, Kimi K3, even though they've had access to not only distillation, but logic distillation from Mythos. And so the counter here was that, okay, the prompt distribution really, really matters. You need to see what users are doing so that you can distill. I think the prediction from this is that the frontier labs don't necessarily have much of an advantage, if at all, in aural environments now. Because yes, user distribution matters for general behavior and so on, but like the best measure of a capability is the very, very hard RL environments you've made at the frontier. And so if you have access to those RL environments as Anthropic, and you have access to logic distillation, and you've still made a worse model, then maybe like...
S3: Then real-world deployment matters more than the environment. That's really interesting. But they had to incentivize those capabilities in the first place in Fable or the frontier model. And so it's weird that they can't incentivize them again... Or with a smaller model or something.
S2: Maybe we're just in this weird uncanny valley where actually trying to copy that frontier model too much, like the student-teacher gap or whatever it is, is just like too large. And like, I think people made this point with Opus, it's like the difference between Opus 4.6 and Opus 5 is that Opus 5 really feels like it's got this like AI as a judge checking every possible thing it's done. And it's like, that's why it uses so many tokens. It like tries to think about all these things, but it doesn't necessarily have the big model smell of Fable to know when to like stop doing that or like when's a good path to go down or whatever.
S3: The reaches exceed the grasp.
S2: Yeah.
S1: Yeah, I would offer a slightly different hypothesis. So I would say there are a couple of different axes for the environments you can create. And like one of them is difficulty and the other is realism. It's sort of easy to create or it's comparatively easy to create a lot of difficult environments. That are just like involve like doing a much more complicated task or doing something that requires a lot more cleverness. And you could say this is like the benchmarking distribution because a lot of the most prominent benchmarks just involve doing some... Very hard puzzle-like task that's easy to verify. And then there's sort of like the realism axis where you want the model to be good in the realistic coding agent setting where there's like multiple back and forth with the human and there's like multiple objectives. And like, I'd say... Like the people, like the labs who are crafting the model behavior for the first time need to push in both directions. And to get good model behavior, you need to really push on the realism axis and have like rubrics or some kind of human feedback that's informing the reward function you use there. But I think when, if you try to do distillation naively, you end up just sort of matching the teacher on the benchmarking distribution. And yeah, but if you don't have enough of the environments that really exercise the capabilities in these, like, trickier realistic settings, then you're not going to get those into your student model. And I think maybe one thing that's happening is the big models generalize better from the, like, the tricky, narrow tasks to these sort of models. More realistic tasks. So if you have a really good realistic prompt distribution for distillation, you can match the big model really well. But if you only have this distribution of easily verifiable tasks, then you can match the big model on all the benchmarks, but you do worse on this broader distribution. So that might even explain... Something about the smaller Anthropic models like Sonnet 5, though it's hard to predict exactly what they're doing to post-train those models. It could also be that they're always changing their post-training stack and they just got a few things wrong in some of these models. I don't know. They turned up like something too high and created some quirks that people really don't like. So it's like really easy to screw up post-training in some way that doesn't show up in benchmarks.
S4: I mean, just one other sort of very basic point is just like the frontier AI labs buy all their data from data companies. And like the Chinese can also just buy the same data from data companies. And they are, right? And like they are, exactly. There's a lot of people like, you know, being annoyed about this, but like, if they have exactly the same data and they can buy that, they can also distill, it means it's quite easy to keep up, really.
S3: Yeah, yeah. Okay, the other question I had is how the first models that are capable of automating AI R&D will actually be trained? Because there's a toy version, which is, this thing that Ryan was talking about, which is you just have GPT-8 try to build GPT-3 sized models that are really good at like inner loop type challenges of beating video games that require continual learning or just getting to a certain loss with like the least amount of compute, etc. But John, I think you had an interesting point that maybe that's not the way it actually will happen in practice. So I'd be curious about, by the point at which you have AIs that are actually capable of automating R&D, how are they probably trained?
S1: Yeah, I think we'll probably do some combination of learning from human feedback to absorb the researcher's taste and just creating a lot of practice environments, which involve doing multi-step research projects. People will in practice do some combination of those two things and just each iteration patch whatever seems to be most broken in the last iteration. So researchers will be using the AIs a lot and will notice that they have some consistent weaknesses and then those things will either be patched by collecting human feedback or creating environments.
S3: Yeah,
S2: makes sense. Maybe a useful way to think about this is how much of the lineage we roll back and then let's self-play from there. I think in the limit, you're picturing just giving them a GPU and maybe neural nets or something and saying, okay, figure out how to. Yeah. And so you obviously lock in all the previous history of the lineage, but you could imagine a world in which you roll back to before GRPO or something, and then you have environments which try to get it to discover the best form of RL models on, and then maybe you roll further. But I think we will still be so compute bottlenecked that like people will just keep like staying at the frontier and like diffing essentially the bugs and whatever improvements they found since the last model version, turning those into training environments.
S3: Which is also really good for having non-stale like new data between. And it is distilling, right? And that's maybe why
S2: Some of us feel like it's asymptotic. It's like you're always just trying to get the last three months of progress. And that progress is being contributed to by AI R&D, of course, but it also still has humans in the loop and it feels like you're just constantly inching closer and closer to what the human researchers are finding and capable of doing. Yeah.
S4: I mean, the one thing I will say, though, is like, obviously, if you're just distilling on, like, trajectories, you can never go above it. But environments can go quite far away above what a human can do. Like, it's very easy to design an environment that, like, no human can solve, but the AI can obviously still try and solve it. And so that would be the path to, like, go ahead of just, like, what the human AI research is.
S2: Do you have, like, an example of, like, in terms of RSI or, like, you know, working on a train set?
S3: But doing it even faster than a human speedrunner.
S4: Yeah. I mean, I feel like in AI research especially, it's very easy to define, like, goals, which, like, you know, you could say, like, the loss needs to be, like, 1.3 or something, and, like, no human can get there, you know, now. But, like, that's a very extremely measurable, verifiable task, and if the AI gets there, then great.
S3: Right, and building, like, a 100 million parameter model that beats Minecraft, that's maybe too easy, but, like, beats a much more complicated game or something.
S2: Isn't it crazy that 100 million parameter models to beat Minecraft we're calling that too easy? Like, imagine if you said that, like, five years ago.
S1: I would say a lot of research is not exactly like that though, where it's like hill climbing on a well-defined goal. It's sort of more like, here's an intuition we have about some way models should be better. And then we also have some idea for an algorithm that seems to go a little bit in this direction. So let's come up with a task that is sort of designed to show signs of life on this approach and see if we get those signs of life. And then if we do, we can make successively more realistic versions of the task.
S3: It's a lot more guided by intuition and then the inner loop is to elicit or make tests for that intuition rather than the test itself leading to the insight.
S1: Right, like you're not directly optimizing for the eventual objective you care about or the practical production objective. You're relaxing your objective a little bit. You're saying, yeah, let's relax on the realism axis a little bit and find some methods that actually work and then try to get back to realism later after the method matures a little bit. Yeah. And then there's research that's more oriented toward explaining things and developing a theory. Often we don't have mathematical theories in machine learning that are that predictive, but we have a lot of more informal theories for what's going on.
S4: Yeah. I mean, like presumably the models will be trained on like some combination of all of these tasks and like some will be very easily verifiable. Some will be like LLM judge or like just ask the human, like, does this look reasonable? And then the hope would be that like these would all generalize to like these much sort of hard, sort of more vague, fuzzy kind of tasks. And like it probably will to some extent, whether it generalizes enough that like the loop can become like self-sealing without humans being in the loop at all. It's like unclear.
S3: Yeah, yeah. Maybe taking a step back, here's what it seems to me that the plan for AI research going forward is. And you tell me if you think it's going to work or if you agree with this characterization. So the bet is that we will scale up our LVR training across millions of diverse environments, across hundreds of different kinds of domains. And what will emerge at the other end is an agent which has learned these basic skills. Or less than basic skills around being persistent, being able to triage information in context, eventually having end-to-end optimization of working with other agents and things like that. And such an agent will be very sample efficient. And within the context, you know, you've done research on how you actually scale up in-context learning to make it like arbitrarily long, but you keep scaling it up. And so what comes out the other end will be something that basically functions like a drop-in remote worker over the course of a week or a month. First of all, do you agree that that is a bet the labs are making? And second, is that enough? Like basically learning how to learn within the simulated environment within a data center and then getting deployed into the real world, but not actually like learning from real-world deployment, only learning these meta-task skills from the simulated environments in the data center.
S2: Yeah, I think it's now hard to separate out how much of the lab's effort is going towards direct RSI versus making generally intelligent models that they can continue to deploy to collect revenue to fund the next big training run. I think for the... Latter, yes, that's probably just the bet they're making. And it's very clear the pattern of where these environments are going over the last few years. Anthropic's lineage of environments is a very clear example of this. First, we just focus on coding and we're going We're going to get really, good at that. And then the task horizon that we've got from coding, which is probably the lowest hanging fruit in terms of data available on the internet to create environments, their own internal stuff that they can turn into environments. Then we're going to generalize. We're going to go after finance next. Just so much Excel data and all that sort of stuff in the RL training. And then, you know, it's PowerPoints. It's like this long tail of like the working economy. And like that seemed to work really well. And like a lot of the other labs and things, even the open source labs have now realized that that was the correct bet.
S3: But what is the implication from that? When I had Dario on the podcast, the thing I asked him was, if you truly expect models, which will be human-like in their ability to learn on the job, why would you try to bake in all these skills of working with PowerPoint or something? Wouldn't you just expect the model to be able to pick that up while it's deployed? And so, yeah, there's multiple different explanations. One is just that we expect models to get there soon, but they're not there. So why not amortize these skills into the model training? Another is that we're not concentrated on making it really good at widely deployed work. We just want it really good at RSI. And this is just like a way for us to like get revenue so that we can pour it back into a model that is actually like really good at doing RSI development. And then like once the singularity happens, the thing that comes out the other end will be really good at RSI. All the things which seem like bottlenecks to the current generation of models. Yeah, John, I don't know if you have a take on like what, how once you construe why there is so much task-specific knowledge in these models, if the path is like this kind of generalization.
S1: Yeah, I mean, if the models were good enough at learning in context, then in theory, you wouldn't need to train them on finance. They would just be able to figure out, read all the books on the fly and figure out how to do everything in the appropriate jurisdiction. Yeah, and you could argue that you need to do a lot of this domain-specific training just to make them more efficient. So even if they were smart enough to figure this out on the fly, you still might want to do a bunch of RL and bake all these intuitions into the weights so the model would be more efficient at runtime.
S3: Yeah.
S1: Yeah, I'd say in practice, it does seem like model providers are going domain by domain and trying to strengthen the models in the highest valued domain. And I'd say that's one of the answers to why the models have gotten so much better. It's just because the model providers have covered a lot of the high valued domains and the most common types of skills.
S4: I mean, I think another thing is just that like, it's not that expensive to do both at the same time, right? Because like the models are massive, they can easily afford in terms of their parameters to like learn everything. And like there is likely some transfer and sort of even just, even if like finance isn't specifically like the information is important for like RSI, just the general like That makes sense.
S1: I mean, there's one question about whether this current paradigm of doing like Sim2Real will be the dominant one forever. So basically you look at what the real-world tasks are like and then you try to create a bunch of environments. That can be simulated like in the data center and you can do RL on them. And I think obviously this has been very successful, but it also has a lot of weaknesses because a lot of things are just kind of hard to simulate, especially if they involve like interacting with a bunch of humans in real time. Yeah, so there's some question about whether Sim2Real will be the dominant framework forever.
S4: I think Sim2Real has to be the dominant framework where sample efficiency is kind of low. Because right now you need thousands and thousands of interactions with the humans and no human is going to sit there and deal with this, basically be in the loop of RL training. And so we kind of have to simulate that now to get the samples you need. But obviously if sample efficiency improves a lot, you'd expect learning from deployment to become a much bigger part of it.
S1: Though there are also other things you could do, like you can learn off-policy so you can take all the traces and even without re-simulating everything, you can potentially learn something from them.
S3: Jane Street just launched a new competition and it's their most ambitious one yet. Design a protocol emulator ASIC. Basically, if you have a chip that you want to test, you can connect it to this ASIC and then this ASIC will simulate realistic traffic. That way you can see how the chip responds without Jane Street is looking for flexible, general-purpose designs, not single-protocol emulators. When I was chatting with them, they suggested that I start off by trying to implement what are apparently three very common protocols, UART, SPI, and I2C. Jane Street also mentioned that they hoped that more ambitious designs will also tackle low-speed USB and Ethernet, and any other protocols that flex your chip-specific architecture. Importantly, your design should be programmable rather than smashing a bunch of specific protocols onto a chip. If a new protocol comes out after your ASIC is taped out, your chip still needs to be able to handle it. How exactly it does this is up to you, but there is one hard constraint. Your design must target an open source 130 nanometer process node. That's because Jane Street will pay to tape out the most novel submissions and send the physical copies to the winners. The competition is open until January 18th, 2027, and working in teams is highly encouraged. Go to janestreet.com/dwarkesh to download the template code and get started. I want to ask more about this because it's sort of weird that... You have 50% of compute that's spent on inference that is not directly helping the model become better. Like one of the key advantages you'd expect eventually digital minds to have is unlike a human who gets to have 50 years of like real-world experience, a model will get We'll get to, through all its instances, we'll get to experience, I don't know, millions of years of deployment across all kinds of economically relevant work in the economy. And right now, that data is just not, in a meaningful sense, helping the model get better. It's so obvious that eventually models should be able to learn from this data. And once they do, you would have something that almost feels like a widely deployed intelligence explosion because the model is assimilating so much information across all these deployed instances. But when do you expect this kind of hive mind kind of crazy
S4: I think broadly, like at a very basic level, this is already happening, right? Like just in the next generation of models. So like right now, you can always take your deployment data and put this into the pre-trained or the mid-trained of like future models, especially if you do like some kind of filtering or some kind of like judgment or annotation or like recent, you know, synthesis of that. How much do you think that explains the generation over generation improvement? I think it explains like quite a bit. I mean, especially like, I mean, this is, you know, I don't know whether the labs do this because theoretically they claim not to train on people's data, but the Chinese 100% do, and they definitely get this advantage, both obviously deploying. This is basically what distillation is. They take out the models, they get some of their deployment data, they get some fraction of that by pinging the model, and then they train their next generation of models on it, and they can suddenly do it on their own models as well. There's no reason not to whatsoever.
S2: I completely agree with this. I think if you zoom out far enough, this is definitely happening. You're picturing this, and we're all picturing this, this is what continual learning, the holy grail is, is this very, very organic live loop of an individual model getting an experience and live updating on the spot and learning from that. A lot of things break when you zoom into that level of granularity, but the big labs are doing this, the closed models are doing this. There's also early signs of life of people using open source models doing this at a much faster cadence. A good example is probably Composer. You have some sort of model and you are able to, or Harvey's doing the same thing with legal agents. It is getting very specific environments from the data that you have. For that particular task and things that users are complaining about and all the feedback that you're somehow extracting from your specific deployments. And a lot of these companies have the advantage over the big labs and that they can use this data really, really well. And then they will create environments. They will do a big post train of Kimi K3. They will go deploy it. They might do some online learning as well. Like Composer did online basically like Reinforce for a long time. So yeah, there's still a human in the loop. There's still a human saying, okay, these are the signals we care about. Here's how we're going to create environments from the data that we have. And there's still a longer cadence than maybe the one that you're thinking of. But it really is happening. And eventually that loop will become faster and faster.
S3: I mean, the Composer thing is interesting because... This is where the model, in Cursor, people press tab or they don't press tab on the next completion that the model suggests. And based on that, every single day, Composer gets better at predicting the next...
S2: So that was the old tab model. They actually did the same thing for... The actual, not just like the tab model, but the actual like generative model.
S3: Oh, that's interesting.
S2: And it's hard because when you do online reinforcement learning, you don't have groups, right? You just have one user saying one thing and then you get one rollout. And so like you have a big variance reduction problem. And like Cursor is kind of a fuzzy answer. Interesting.
S1: Yeah, I think your biggest problem is actually just not knowing what the reward function should be from natural data. And if you use some kind of superficial signal, like did they accept the code, the edit, that might get reward hacked in some way.
S3: But this seems like a bigger issue with the Sim2Real thing, where... The longer and longer horizon tasks get, the harder they are to simulate within a data center, right? It seems to me already, potentially, at least even in coding, we're getting to the point where there's like not some year-long coding task that doesn't. And eventually require you to like talk to a client or interact with the company or interact with users. And if you think about the gamut of things we would want AI to be capable of, you want eventually super intelligent people to like run a business or like start a new business and make it profitable. Or like have a profitable day trading in the markets or win a court case. And these are all things which are very hard to simulate in a data center. Like an inherent part of the learning there is interacting with the real world. And so maybe they may have to learn how to get better at these things from like the transfer between Sim to Real. But alternatively, maybe you do need weight updates from these kinds of interactions in order to get better at them. And then if that is the case, if transfer isn't strong enough and you do need weight updates, then the fact that the models are quite sample inefficient is like maybe a deeper problem. And the reason I'm curious about this is I feel like by default, I don't see how you don't get some kind of crazy recursive self-improvement within the next 10 years. But the one reason why that might not happen is in terms of like weight updates, the sample efficiency of weight updates, they just seem way far behind humans, right? Like plausibly million-fold behind humans in terms of how much data a human sees from birth to adulthood versus how much a model sees from, you know, like cold start to like finishing training. And so, yeah... This is all to say, first of all, is there going to be a good transfer between simulations and extremely long horizon, really complicated real shit that we want the AIs to do in the real world? And if not, does that really mean that the lack of sample efficiency in these models comes to bite us?
S2: I think maybe the way I'd break down the two types of tasks in which models get good and models will still continue to struggle is whether the task is cumulative or you have this non-stationary distribution you have to keep learning and relitigating a bunch of stuff. Maybe an example of a cumulative task might be RSI. It's theoretically possible to maybe less than a million token Python file, which from scratch trains a model that is capable of recursive self-improvement. And every discovery that you make is kind of a line. Yeah. I think the real world and the reason people are thinking so much about continual learning is it's not really a cumulative task. Imagine in a law firm. You have an agent acting as a legal associate. That's a very non-stationary distribution. You have to be able to fit in your context all the relationships between all the important people at that company, which are also changing all the time. You have all these implicit ways about how things are done, where to find. Yeah, I don't know if you guys have thoughts on this.
S1: Yeah, I would say today's models are weaker than humans in a lot of different ways and some of them might have to do with sample efficiency in a certain regime where I mean, in some regimes models are very sample efficient, like learning in context, but then there might be some like medium length regime where they're less sample efficient because humans can do some kind of weight update more efficiently than models. So I think like being less sample efficient in certain regimes might be one of the sources of weakness, but then I think there are other sources of weaknesses that are completely different than that. Having lower diversity of thought than humans or being bad at certain kinds of long horizon judgments. I mean, I think a lot of what people call taste is something about behavior that. So, yeah, I think the weaknesses of humans, which limit RSI along with other things, Yeah, there's a variety of them and some of them are related to sample efficiency and some of them aren't.
S2: Maybe an interesting thought experiment is like, if you were able to give a model like a context window of, I don't know, a trillion tokens or whatever you would have needed to fit in like your experience prior to like, let's say RLHF. And like it's got all that experience in the context window and it has the same sample efficiency and in-context learning ability as it does at a million tokens. Like do you think taste is then solved? Like would it be able to like make the same judgments that you did or is there like something fundamentally missing apart from just a longer context window with the same sample efficiency?
S1: Yeah, I mean it would have to be trained to learn from that context. So I'm not sure... Either it would have to be trained to learn the right update to make from that context.
S2: So you don't think you can just dump it all in? Your whole life research experience?
S4: In theory, I think, yes. I mean, this really just comes down to the question of like how meta-learnable is taste from like shorter horizon episodes. And like, I feel like there's no obvious reason it's super long because like humans somehow develop taste with not having many long episodes. Like we don't live to like 10,000. We develop pretty quickly, right? And so if you think about even in a PhD, the difference between a first year PhD student and a final postdoc or something, that's five years maybe. And they've only done maybe 10, 50, 30 research projects in total, but somehow they develop taste. And so theoretically, it's possible to develop it like that. The AI obviously will have vastly more experience in which to develop taste to meta-learn it. And then it's like, how well does that generalize to really long horizon things is, I think, the question, which I think is really unsolved at this point. We don't know.
S3: Going back to this question. Eventually there should be a regime where AIs are learning a ton from each individual instance of deployment that they have. Well, currently you could say there is a meta fuzzy process by which models do improve for deployment, but I feel like it's a very weak feedback loop. Do you see this around the horizon where there's this hive mind kind of learning that's very rapid? And if so, how exactly does it happen?
S1: Actually, I would say that around will we get a hive mind that learns from all of its deployment experience. I mean, a big part of that is actually about incentives rather than being a technical question. So companies aren't going to want to have the model provider learn from all
S2: I think that maybe the economics of this will pressure not necessarily weight updates to one big like common shared model, but like kind of like modules that get. So a very obvious example, this is Allura, but it might be something else like, you know, there's been a lot of work to try and fit like an arbitrary context length into a fixed size. Like this is all the linear tension stuff and all that sort of stuff and like cartridges, which are essentially KV caches trained to be very, very compressed KV caches to fit in a lot of information. That's another example of something that companies may be willing to sign up for if that gets subbed into the model. And it's not actually changing the base underlying model itself. So there's many different versions of learning from your data in real time. And the latter ones are not really helping the big labs because they are just these modules. But I think the economic pressure will force the labs to go down that path first before they can embark on this.
S4: So what economic pressure there? Because I feel like even if you have like a bunch of cartridges or laws or what not, you can still just like take all these traces and just like distill this, dump this to the pre-training of like your next generation of bloggers.
S2: Yes. So it may be a more indirect form of learning that the big labs are getting and that's obviously still that's really valuable to them. But I can't imagine a world in which we start off with like, you know, we're going to just like directly train this one big model on like all the exact data.
S4: No, I think it will definitely like go through stages because, I mean, this is assuming there's like one discontinuous event where it's like suddenly we fix like weight updates continuously and like in practice I think it's much more likely to be like, the cartridges and stuff allow you to specialize in deployment. Then you generate traces, you put that into your model. Like three months later, you come out with a model, which is better at this stuff. You specialize it again, you like consolidate it again. And then eventually we'll just like make this loop faster and faster. So instead of like every three months, we release a model. Now it's like every week and then every like day and then every hour, at which point we basically have obviously solved it.
S2: Yeah, and I think this is a good point as well because you asked how far off the current paradigm we are from being able to do this. We've done a bit of research to this and people have done a lot of research. At a really large scale, when you wash out enough noise and you have large enough patches, this outer loop process of putting data into mid-training, creating our own environments, it does work in some sort of continual learning regime. But the problem is when you zoom in close enough at a micro level, I've got one model and I'm trying to update it again for a law firm or something. And I'm trying to do that very continuously with a relatively small amount of data. All the methods kind of break down a bit. So if I SFT the model on just successful traces, off policy, on policy, eventually Yeah. You know, on-policy distillation seems like it would push this horizon out a little bit, but it still eventually succumbs to the same thing. And RL is not very good at like, it is good at like getting capabilities in, but it's not as good as getting like knowledge in and like just this. Very explicit knowledge of like, okay, like this person does this at this law firm and like this is a very specific process we find. And you have to pour in a lot of compute to create the right environments to get the knowledge.
S3: Do you think that the fundamental issue here, why you get worse at any of these other skills or there's forgetting and stuff, do you think it's fundamentally an issue of capacity or is it an issue of techniques?
S2: A little bit of both. I think SFT and even on-policy distillation can be way too destructive. The reason RL is so nice is because it changes a very, very small amount about the model and there's a lot of evidence for why this is the case. And so it kind of just tweaks it in this very, very, very small loss value to get it into the right point. But that also then limits what you can do with RL, like how much you can actually change the model.
S3: You're saying the reason this isn't the winner-take-all potentially is that it's just very hard to distill that much information into the base model? Without ruining something in it.
S4: It's easy to distill it into a different base model. This is why I think it's mostly technique. It's definitely not It's not just there isn't capacity. If you had some modeling with all this data and you taken nearly the same size model and pre-trained it from scratch with all of this stuff in mid-training, it will be better. And I think that's a lot of what's happening today. And so it's very much like there's a bottleneck that stops us from just keeping training the same model forever versus just getting all the data from the old model and training a new model from scratch. And this is exactly what I was saying, some combination of plasticity and catastrophic forgetting. If you just naively train on non-stationary data because you're adding new data as you go, basically this is messing with the data distributions so the old stuff is just forgotten and we don't really have good methods to stop that from happening.
S3: So maybe in the limit you're just bottlenecked by retraining the model from scratch with all this information?
S4: Yes, which of course is very expensive. Training a model from scratch is expensive. But you're going to do that anyway. Not necessarily. I mean, like maybe eventually if you have continued learning, you never train a new model. You just like, just have a model and it keeps learning and like expanding, right?
S3: But there might be like some deep technical reason why that's very difficult because of the like-I mean, that's the question. That's the question.
S2: I think we have pushed back like how much from scratch We need to do. Like it is definitely possible now to take like the pre-trained base and like do very good mid-training on top of that, like kind of continuously plus some RL from like different checkpoints that are later on in the training. And like that's looking more like continual learning, but certainly not the case of like, you know, take the most recent I'm a bit confused
S3: because isn't this literally what happens during training or during post-training or something? You have a model that's already gone through so much training and then you distill some fork it's been further RL'd or something. Isn't that literally what happens?
S2: It's still at a large enough scale, I think, that you're washing out a lot of the noise. And you're not just focused on one distribution, which, as Baron said, is like, you know, that is now a very, if you're just focusing on one task, right?
S3: I mean, in the eventual regime, you'd be doing, I don't know, there's billions of deployed instances. You're like doing, you're learning from all of them at once. And so hopefully there's some washing out of noise and stuff from that, right? Maybe that scale, yeah.
S4: Yeah, I mean, I think, like, definitely, as I was saying, like, you can do continual mid-training for, like, a long time, and you can, like, roll back to a checkpoint, give new mid-training data. But at the same time, like, you can't do this, like, indefinitely. Like, if you just keep continual mid-training the same base forever, it just, like, gets, it does, it's sort of asymptotes at some point. Like, you can't just learn new stuff in that base. And this is why people end up training new bases. Like, otherwise, you would just keep mid-training the same base forever.
S3: Whenever I finish recording an interview, I immediately brain dump all my thoughts into Slack. Things like what was most interesting and what should get cut. This ensures that my editors have all the context they need to start editing the episode. But it's not like these brain dumps have any clear timestamps and my unedited recordings are Now, whenever one of my editors posts a rough cut of the episode, GrokBot opens a transcript. And when it's done, it sends me its top. This has worked really well. Being able to send informal messages like I'm texting my editor and then having the transcript immediately reflect my preferences has just been so helpful. Try GrokBot yourself at x.ai/bot. Okay, let's talk a bit about data now. So I'm generally interested in this question of how much of AI progress is just explained by data progress. It doesn't mean it will be necessarily hard to automate, but that's a separate question. So is there some data distribution which if you trained current architectures on would result in a super intelligence that totally dominates human experts across every single field?
S2: Are we talking about like pre-training plus post-training data like environments as well? I think the existence of this is obvious. It's just like whether we can create the right environment.
S4: Yeah, and in the trivial case, we could just train it to output the Python file, which like trains the actual super intelligence. Like just have that memorized in the way it's.
S2: Yes, there's probably a ladder of RL environments that is possible to construct such that you would get an AI researcher which is at least as good as a human researcher. But the effort to climb each successive rung grows exponentially. And that's going to be the two things that you have to trade off against as to how fast we're going to hit that final rung where it's better. Yeah. I think that's fairly clear. And I think there's like, you know, we're still relatively early in like our environment creation. Like there's a lot of asymmetries that we exploit in order to create good environments. So one of those asymmetries, which we've talked about before, is like there's environments where it's It's easier to go backwards than forwards. And what I mean by that is it's very easy to define this complex data generating process, and this is this latent variable you keep hitting from the model. You can generate arbitrarily complex environments, and the model has to do a lot of irreducible token. Yeah, yeah, yeah, yeah. So there's all these asymmetries which we're cherry picking and we're counting on this task horizon generalization. But I think, yeah, again, there's just going to hit diminishing returns at some point. At some point, there's diminishing returns in how hard it is to create these environments in the first place, coming up with them because you can't necessarily just have these really, like these processes where it's easier to go backwards than forwards. Like you actually have to sit down and construct like something that looks like with humans, like, you know, a long enough time horizon, like it's going to be a really complex task to create. And then there's also going to be like the compute and time bottlenecks for the agent to actually do those tasks. So like, I think you're just going to start seeing this like curve flatten out.
S1: I saw something about how someone fine-tuned the Taka model, which is only trained on data up to 1930 on this modern coding agent data, and it did better than Claude III Opus on Sweep. So this model that has no knowledge of code whatsoever can be fine-tuned on a moderate amount of data and behave better as a coding agent than this much larger pre-trained model is pretty crazy. And it kind of shows you that once you have an example of the right expert behavior, it's actually surprisingly easy to copy that into a relatively weak model.
S2: But a counter-example to that model, there was a paper recently where they trained it up to fifth grade math. Also primary school, English and stuff, so it was a decent language model. And they tried to RL it to do late high school and college math, and the gap was just too large. They couldn't get it to climb at all. But if you did successive wrongs of year seven math and then year eight math and so on, you could obviously climb to year 12. So again, it's just like, what is the distance between the wrongs on those letters and how hard is it to create?
S4: And this just comes back to the RL signal problem. RL is not very good at exploring right now. And so if the model can't get in 128 rollouts, it's very unlikely to get signal to progress. And this is why in RL we need curricula, whereas in pre-training we don't, because that's not a problem for pre-training at all.
S2: Yeah. And again, pre-training data is different to post-training data. And I imagine as we continue on, like, yeah, humans will be involved less and less, but that doesn't change the fact that you're bottlenecked on, like, how much signal you can extract from the real world. So, like, there's a lot of signal in the world, and that's true. Like, you know, there's people doing, like, spreadsheet tasks. There's people doing, like, legal tasks and all this sort of stuff. But, you know, the capability frontier of where the models are at now, like, how many bits in the world are actually, like, really relevant to, like, improving the model's capabilities. How many new math problems are being solved that are just beyond the reach or grasp of the current models? How many new coding problems are being created or solved that are beyond the reach of the current models? I think that's why the diminishing returns kick in because even the world as a whole is not giving you the bits that are useful for tipping you into the next basin of capability.
S4: Yeah, I totally agree with this. It's like really a question of like where the signal is coming from. And so like the signal doesn't, you know, in pre-training, the signal is like already in Common Core, right? Like for the tasks that you care about in pre-training, the problem it's not just like getting signal at all. It's like filtering out all the noise that exists. And that's quite an automatable process. But as the models get better, as we enter mid-training and post-training, the signal just doesn't exist anywhere in the original data we have. No amount of filtering will get this. There's no hidden proof of the Millennium Prize problems sitting in Common Core. We can just filter. And so at that point, you have to get bits somewhere else, either from humans directly asking them to write out their reasoning or by creating environments where humans decide what environment should be created, what the objectives of these environments are, or some kind of training on the human data that exists in deployment. You have to get the bits from somewhere.
S3: Yeah, yeah. There's a question of how much of the progress in pre-training is being driven by data. I did this investigation with Jerry Han, who's a student at Princeton, where we basically trained all the recipes from 2019 till now, pairwise with all the datasets from 2019 to now. So you're saying like GPT-2 on the newest dataset, like UltraFineWeb, and you train Delphi, which is the newest training recipe or the open source training recipe on like the Pile or some old dataset. And you do like the whole grid and you see the... Getting to some level of capabilities, how much less compute does it take across this grid? And you see that the data seems to explain like 9x of a compute efficiency gain, but the architecture improvements explain like a 3x compute efficiency gain at a very small scale. And so to the extent that that is true at large scale, that most of the pre-training compute efficiency gains are coming from better data, how much can that continue? Can you keep just filtering data more and more and building more and more synthetic data until, yeah, do you have a sense of how much this kind of retraining progress can continue?
S2: I think my prior is that, again, the low-hanging fruit is somewhat exhausted. We got the internet as this big block, and it's not like the internet is necessarily growing at the same rate. All the useful stuff on the internet is growing at the same rate. We've probably got a bunch of point bumps but definitely not as many as have currently occurred. But that's also really interesting that
