So when people sort of say the protein folding problem has been solved, like at a conceptual level, yes, there might be sort of, yes, there has been, there have been some advances. We don't know what is the actual true ground state that proteins take and what is the actual distribution of structure that proteins take. What we are trying to do is basically someone got a structure, deposited it in the PDB, and we are trying to replicate and get the same structure. That's what we did, right? And it just so happens that it's useful. But that does not mean that we have understood all of protein dynamics.
Yeah. So, great to be here. What an exciting morning. So many cool announcements. I think the future of bioscience is being like announced right here. So this is the modeling session. We are all modelers. So of course it's natural for us to talk about data. So to start out with, I think one of the things that I like to think about when it comes to data is, is there a scaling, like how do you scale data properly? And this has brought me to this question about the bitter lesson, but recast in the frame of data. So the bitter lesson, for those of you who are not AI people, is the statement that methods that scale win eventually. If you can just scale enough, it wins. And so my question, I'll start with Sal here, is, is there a better lesson for data?
Yeah, I mean, for sure, I think that you obviously need the right data, right? Like in order for it to work. So I think like one misconception of scaling laws is that, you know, scaling laws are everywhere and they always exist. I think a lot of the work is actually finding that scaling law, right? Like so a lot of what we really do is trying to figure out what's a situation where if you put more compute into it, if you put more data into it, you actually get, you know, a better result out. And the reason that's a great situation is then, you know, once that happens, you can kind of just turn the crank, right? Like it becomes an engineering problem, which is something I like. But, you know, that it has to do with architecture, but it also very much has to do with data. So if you don't have the data with the right information statistics to solve the problem that you want, which is, you know, I think probably pretty obvious, like if you think about it for a second, like you're not going to get a model that has the capabilities and the understanding that you want. So you can only really pull the information from the data that you have and then use that to generate, like, to generalize beyond, right?
Yeah. So when it comes to data collection, how do you think about what types of data modalities are best? And maybe a kind of related question is, do you think that modelers have a tendency to work on problems which are, the data is available rather than maybe the data or the problem that solves your goal most, like maybe has the most impact for translational medicine?
Oh, for sure. So I think that, like, I won't speak for all modelers, but like, I'm lazy, so I'm going to work with what's available. And, you know, I think there is actually like a good side and a bad side to this. And like the good side is that, you know, I think like when you're doing like machine learning or like conventional machine learning, you're really looking for like the most pristine, high-quality like data examples that you can find. But when you actually, if you're like me, you're actually someone who goes and you're kind of like rooting around in the back room looking through people's like junk, like what's there. So like a concrete example of this is when we train, like when we train a protein language model, we train on metagenomic sequences, which are not the highest quality data. In fact, much of that data I can guarantee you isn't even like a real whole protein. And yet that makes the performance of the model go up for designing real proteins that work, that make for understanding proteins that like we know are things. And so, you know, that can, so that's like the positive side of it, but that can lead you to like a negative place where you say like, well, let's just scale up the data that we can generate easily. And I think that's not necessarily the way to do it. I think that's one of the things that's really exciting to me about like what we're talking about here today with the BBI. You know, it's really going out and saying like, what is the data that we need to solve the problem? It's the right resources, but also the right community. Like I think that, you know, one thing that we do at, or we try and do at Biohub anyway, is to work in the open, work with the community, and move the whole community forward. And I think that's a really critical piece for this because if you just have like people who are building models, you know, we're going to, we're going to lean towards the data that exists. If you just have people who generate the data, they're going to lean towards the things that can be generated. But like if you can work together as a community every step of the way in an open fashion, then you can actually figure out like what is the data that we need, and then you can find that scaling law. Cool. What do you think, is there a lesson for data?
Yeah. So I think the, so I was at DeepMind when Rich was with us and working and thinking about this idea of the bitter lesson. When Rich and I took, the thing that I took from Rich's original sort of lecture that he actually gave at DeepMind was not about the actual notion of whether data is useful or how should we think about machine learning. I think my sort of take was that he was talking about something more conceptual, and the conceptual thing that he was talking about is sometimes when we are looking at problems, we think about solutions in a very religious way. I'm a modeler or I'm a data generation person. And I think that is the bitter, bitter lesson, that if you approach the problem with that mindset, you might not succeed. The problem comes first, and you should be flexible in terms of your solution space. You should try both things. You should see what, you should try to understand what is the problem that you're trying to solve, and if it requires modeling effort, do the modeling effort, and if it requires the data, collecting more data, then collect data. But sort of the thing that was, that happened in machine learning in those, in those times is people said, we are machine learning researchers. The data set is there. Here's some training data. Here's some test data, and we will just optimize the model, and that is broken, right? In the sense that if your eventual sort of goal is to solve the problem, then you have to be able to look at both aspects of what goes into the process, and what goes into the process is not just data or modeling, it's also about sort of expertise. So the example that I have is like with AlphaFold, we looked at what was possible with current existing data sets because we did not have the core expertise of now sort of going to, or even the resources, by the way, to sort of say let's augment the PDB by a significant sort of order of magnitude. It was just not, I mean, the amount of sort of investment that organizations and scientists across the world had put in to construct that data was invaluable. So there you have to sort of focus on, well, getting the biggest bang out of your buck by investing it in modeling. But in other areas, when if you're sort of approaching say cell genomics, then we took the same approach and said, well, what can you do with cell by gene if you're looking at sort of cell genomics? And there it was very clear that we, after a bunch of work, that the data was not there yet to be able to build, to go after that grand ambition of building the virtual cell. And so it just brings people together to focus on what are the actual challenges if you want to advance science rather than being religiously following like advances on data generation or in modeling.
Cool. That's, I really like that answer. So what do you think like the actionable takeaway is from this? Like, I mean, always define the problem first and then, you know, figure out what solution space to search over, but more broadly, like how should the community be thinking about this as we go forward for the next, you know, generation of basically trying to solve translational medicine?
Yeah, I think the advice that I give to anyone who is working, who is starting in the area is that think of yourself as a multidisciplinary person. Understand the problem first. Why are you working on this problem? What are we trying to achieve? And then think about what is it that will be needed, whether it's modeling, whether it will be compute scaling, whether it will be sort of data generation. So understanding of that problem sort of is extremely important, and of course then you have to build sort of your expertise in the house. But, and there are also, it's not as if like you don't have constraints. Maybe there are constraints, like there is only a specific amount of data that you can sort of generate, or there is a specific model size that you can afford to sort of train. Understand those constraints and stop and try to fail fast and sort of look at what are the approaches that are going to be feasible in the long term in getting you to that intercept, the intercept level, the impact level that you're going after.
Yeah. Okay. Cool. So this actually leads me right into my next question, which was, when I look at sort of the evolution of modeling, let's say AlphaFold 2 was essentially a work of art, like a bunch of very carefully handcrafted features. There was a lot of thought, very intentional thought done to every part of the solution. And then I think, you know, some of Google or Alphabet's work has been, you know, kind of still stays in that space, but seems like the general consensus is to move to more scalable, more general strategies. Do you think that there's still a lot of like artisanal craft solutions or problems that need that, or is our resources better spent on generally speaking, you know, fixed resources, fixed money can go to compute, it can go to talent, it can go to data, you know, should we be focusing on scale first?
So my, again, sort of let's approach it from first principles and think about when, even so, like when you think about like the arts and crafts of how do you construct a model to be better. I think it was not an accidental thing, right? We did a lot of experimentation, but there was a vision behind it that all this scientific intuition that came from biophysics and biochemistry, that those interactions, that sort of amino acid sort of residues are not just doing their own thing, they are being influenced by other residues. So let's bake that in. Like why, like if you can, if you have learned something from the scientific community, use it, use that information and try to sort of encourage the model and sort of give it that unfair advantage that it has. And it basically makes the model much more data efficient because it doesn't have to replicate everything. I also have to sort of mention that curating good data is an art in itself, right? It's not as if like you will say, well, put more data. If you replicate the same amount of data, you're not going anywhere. So it's not just about big data, it's about good data. And actually understanding sort of the coverage of what data is necessary for making progress in the problem is, in fact, I would say, a much interesting and much more challenging problem in itself.
Yeah. Cool. Well, it looks like you have a thought. So, what do you think?
Yeah, I think, I mean, I very much agree. I think as you, it really depends on, you know, as you're saying, the problem to be solved. So if you have kind of a smaller amount of data, then certainly, you know, having more inductive bias in the model is going to help you, right? Like, and you actually need that to get the results on smaller scales of data. As you, you know, as you get more and more data, you know, you see that like sometimes the model can find things that like you didn't necessarily know about, and sometimes that inductive bias, if it wasn't exactly correct, can hold you back, right? Like so there is definitely that like tipping point as things get better. But you know, I guess like also I kind of object to your question too, because I think there actually is a lot of, I think there's a lot of craft actually to the scaling part of things as well. So you know, you need very, there's a lot of algorithmic work that actually goes into taking these models and training them on more data, putting more compute in, and making them bigger, right? Like, and we see that like not only is that from like an infrastructure perspective and you know, making the models like run inference faster, but you know, we're actually seeing that like things are going beyond transformers to more bespoke architectures that can, well, I mean, I don't think we're in a post-transformer world in any way, shape or form, but you know, we're modifying those architectures, right? Like in order to make them more fit for purpose and actually work better even at like internet scale data. So, you know, I think it's, that's kind of the project, right? Like as more and more data comes in, it's not only, you know, helping to curate that data and select what is the next like batch of information that we actually need. You know, it's more information than data. How do you get that next batch of information that the models need? But then also like at every step of the way, at every scale, what is the right architectures then to get the most out of that data?
Cool. So if you think about the state of protein structure prediction right now, you know, the news will say protein structure prediction has been solved. But then like if you talk to my friends, they will all say we have so much like left to do here. So many things like function, dynamics, and then design are still like just wide open problems. It's been five-ish years since AlphaFold 2 was announced. Progress has been made. But I'm curious, Pushmeet, what do you think, is there another big leap being made here? Is there, are there blockers in solving these problems? Do we not have the right data yet? Do we not have the right algorithms or ideas, or is this like something that we can see evolving in the future? And it's just kind of a matter of time.
Yeah, I think science, how it sort of operates, is basically by isolating something and then making progress step by step. So when people sort of say the protein folding problem has been solved, like at a conceptual level, yes, there might be sort of, yes, there has been, there have been some advances, but I think it's also like when you think about that narrative of proteins being the building blocks, proteins are not blocks and they're not sort of, they don't act as blocks, right? I say proteins are the building blocks all the time, but actually like I don't believe in it, right? Proteins are extremely complex, they are disordered, their shape might change depending on the context. And like what AlphaFold did, and in fact like John, John and I, John Jumper and I basically used to sort of discuss this, like what are we trying to solve? We don't know what is the actual true ground state that proteins take and what is the actual distribution of structure that proteins take. What we are trying to do is basically someone got a structure, deposited it in the PDB, and we are trying to replicate and get the same structure. That's what we did, right? And it just so happens that it's useful, but that does not mean that we have understood all of protein dynamics. So I think the, like at the top level, it's easier to communicate that we have made progress, but the scientists among us and in this crowd, we know, right? And I think that's the really important thing for us to understand and have that common ground that a lot of work needs to be done, and we have made progress and there's a lot to celebrate, but let's not stop the funding of protein structure prediction and protein dynamics because we are just getting started.
So what do you think the biggest blocker, if there's one thing you could wave a magic wand and say like we have more of blank that would accelerate one of those things, function, dynamics, or design, what would the magic wand be? What would you wave into existence?
Oh, I have basically, when, so I'm a machine learning, my background is like quite eclectic. I started as a security researcher, then went into computer vision, Bayesian theory, and finally got into discriminative machine learning and deep learning, and so on, and then AI for coding, and then finally science. So when you asked me that question, like my computer vision, the computer vision researcher in me was super excited about cryo-electron micrographs. I was like, what is this sort of PDB data? I should be working at the source, right? I should be looking at the cryo-electron micrographs. I don't want those structures, they must be sort of missing out all the data. I should just directly operate at cryo-electron micrographs. Now getting models that can scale at that level with the right amount of data and can extract all the dynamics and distributional information that is captured there would be an amazing thing. We tried, I did, but it requires more work, right?
