Hi everybody. It's nice to be back. This, I last year at this time I was pretty new on the job. So it's good to be back and have a chance to see so many people that I've, you know, read your work, some people I've written with and worked with. As you know, this is a very important topic for the health of the country. The number of people with dementia and Alzheimer's, the burden on caregivers, the burden on patients, and on the fiscal health of the country are closely entwined with our ability to make progress on Alzheimer's disease and related dementias. So it's delightful to be back here to see a group of people who are absolutely committed to helping fix and address this incredible problem.
I'm going to focus a little bit on some updates on how my thinking has evolved on how the NIH can support this research mission, specifically related to health economics and other parts of the mission — psychology, behavioral sciences — because I think fundamentally these are a tremendously important part of the portfolio for us to address this problem of Alzheimer's disease. The sort of history of this has been a little bit rocky over the last 15 years, going back, I think, to the debates over, you know, the ACA, and then resulting in a guide notice in 2015 essentially trying to clarify that the economics commitment by the NIH, which in my personal experience actually felt like a withdrawal of support from the NIH for a lot of the sort of social science agendas that I think potentially have a real role in improving the outcomes for Alzheimer's patients and for their caregivers. So I wanted to start with — I'm going to leave a lot of time for questions, but I want to start with the NIH mission and then we can talk about some pillars that I want — neutral pillars that I want to apply to all of — not just the social science — for all NIH work. Because just as I've been in this role in the last year and some, what I've learned sort of firsthand about is the kinds of pressures that the NIH faces to stay — in its support for basically almost any research. And my goal is to make to clear out a space where the NIH portfolio does not have political interference, that has a clear set of guidelines for researchers to stay focused.
I'm glad to see Jim Pura in the room because I'm frankly I'm trying to mimic the kind of ideas that the National Bureau of Economics Research has in protecting its portfolio. I've sent papers — that those of you who have had the privilege of being able to publish papers in NBER know that it has one golden rule in the working paper series, which is that you're not allowed to say anything about policy recommendations. Jim's nodding so I didn't get it wrong. And I'll tell you firsthand, for most of my career, I would send papers that stayed very, very clear of it. And I never realized the level of scrutiny until I sent a paper in 2020, I think it was, and it came back redlined with Jim's comments. And they were very, very constructive comments. I listened to every single one of them. And I didn't realize that NBER had this sort of extensive review system in order to keep NBER as completely nonpartisan, focused on its mission, and rather than allowing other agendas to enter into its mission by the side door. The discussion I'm going to have today has that same idea. But it has to be unlike the 2015 guide notice for health economics and for the way that economics is treated by the NIH as a special, distinct, almost stepchild discipline with a different set of rules that didn't apply to everybody. I want to put forward today — and I haven't formally issued this but I'm headed toward that, and this is why I'm really glad for your feedback about this — a set of guidelines or pillars or whatever you want to call them that will protect the NIH portfolio in the same way that NBER protects its portfolio from things that are outside of its mission.
So I want to start then with the NIH mission, which is up here. The NIH mission is to seek fundamental knowledge about the nature and behavior of living systems, and the application of knowledge to enhance health, lengthen life, and reduce illness and disability. To me, economics and psychology and other behavioral sciences are part of the living systems that we study or should study, right? They're a fundamental part of the knowledge that will enable us to improve the health and longevity of everybody, which is the center of the NIH mission. So the question: how do we do that while protecting the NIH mission? Now, I showed — did I show this slide last year? I don't know if I showed this slide last year. So I showed this — I've showed this a bunch of places and those of you who've seen it or see it for the first time often react in shock. It shocked me the first time I saw it. So this is life expectancy in the United States. I put Sweden by contrast just to put it up there. The eye automatically goes to the right side of the graph, the collapse during the pandemic, whereas it's relatively less collapse in Sweden. I don't want to focus on that. I want you to focus instead on the life expectancy in the United States, the blue line from 2010 to 2019, and then afterwards. And what you see is essentially a flatlining of life expectancy since 2010. That is a scandal.
What that means is that all of the amazing investments and discoveries that have happened — and there are so many things that look miraculous to me — since 2010 have not actually resulted in the achieving of the NIH mission of lengthening life. And in fact, we are in the middle of a chronic disease crisis. The Alzheimer's crisis — and I use the word advisedly — the tremendous number of people who are suffering from the condition, whose families have been completely — their lives have been completely overturned by this condition, are part of that crisis. We have to, at the NIH, become very, very serious about reversing this. And, you know, when I present this graph, especially to the social scientists, I get back — you know, referee number one, referee number two, referee number three — telling me, 'Oh, it's complicated, Jay, it's not just the NIH that's responsible for this.' You guys, I know that, I'm also a social scientist. Believe me, trust me, I've lived this. This is a tremendously complex outcome. But if the NIH's portfolio doesn't help address this outcome, then we are not achieving our mission. Right? So, and I just tell you another, as a matter of faith, I believe very strongly in the capacity for science, for scientific progress, to lengthen life, to make life better. And I suspect all of you do too. So even in your heart of hearts, put aside your instincts of being referee number three and say, 'Look, we really need to use our resources to solve this problem, not just for the American people, for the whole world.'
Okay. The longevity gap — if you look, every racial group has had flatlining of life expectancy. We spent the last 15 years focused on trying to address racial gaps while at the same time essentially we've made no progress in improving the health and longevity of minority populations. If we want to get serious about this, we have to be scrupulously apolitical and focus on the scientific facts that might help address and improve the situation that we find ourselves in. There's a lot of reasons for this. There's too much safe incremental science, research that doesn't replicate, funding concentrated in too few institutions. I'm going to talk about all of these as solutions — for all of these — that I've been working on. Too much safe incremental science. There's a policy of funding decision-making called the unified funding strategy that I'm going to talk about that the NIH has put forward. The research doesn't replicate. We have — I'll show you some data on the replication crisis and how at the NIH we're planning to — working to solve the problem, because it's not just — it's a fundamental problem in the sociology of science and how we evaluate scientific findings. It's an epistemological problem. And then finally, funding being concentrated in too few institutions. This again is something I'll describe how we're working on. About a third of our portfolio goes to about 20 institutions. I've been around the country, talked to people all over the place outside those 20 institutions. The United States is ready for an absolute renaissance in science. What we have to do is we have to supercharge our centers of excellence so that it's not just a few places that have the capacity to get NIA support and related funding, but everywhere where there's excellent scientists there needs to be an equal chance of getting access to NIH dollars.
All right. Let me just start with the first topic, which is the too much safe incremental science. So let me just make a very, very quick case about this. This is from a paper by my former colleague Nick Bloom at Stanford. What this makes in this graph is that by lots of broad measures, research productivity per dollar that we spend on research has become less and less productive. Now in aggregate we spend so much money on research, so the research productivity looks amazing and there are miracles, right? There are things like cures for sickle cell anemia, advances in Alzheimer's disease, in a whole host of topics that are worth celebrating. But the per dollar that we spend on research, the marginal benefit is smaller than it once was. And you can see on the green axis is the effective number of researchers. We've seen a tremendous increase in the number of researchers — 20, 30-fold increase in the number of researchers since the 1930s — and the marginal productivity has come down. Now one idea here is that we're just running out of ideas, right? There's just not enough low-hanging fruit. Now if that's true, there's not a ton we can do about it, right? But then the question is why are we spending so much money on research? But I don't believe that's true. I don't believe that we're running out of ideas. I think that the sociology of our professions have made it so that it's more difficult than it once was to have groundbreaking ideas overturn fields.
The unified funding strategy is a way to address this. I have been a member of study sections. Who in the room has been a member of NIH study sections? God bless each and every one of you. I appreciate every — it's sometimes grueling work. You all know, and if you've been on a study section, you know how these discussions often go. You get an application from somebody who you haven't really heard of before. You look at it and they have — it's a pretty cool idea they have, but they don't know exactly how to write grants. Maybe this is the first or second or third grant that they've written and they haven't sort of absorbed all the lessons of grantsmanship. And the methods are kind of a mess. But my god, if the idea works, if AIM1 actually comes through, then wow, AIMs two and three are going to overturn the world. You know exactly what happens. You give it a good innovation score, you give it a bad method score, and the overall score — you give it a bad score. And this pans out in the data. The data that we look at — the overall score correlates very strongly with the method score that the peer reviewers give, but much less well with the innovation score. And so many of our institutes have a policy of paylines where what you do is you rank all of the proposals that we get. We get 100,000 proposals a year. Our goal is to find 10,000 to fund, roughly. And you draw a line and say, okay, we have this much money and we'll fund everything that's above the payline that has a great overall score. Essentially almost all of the peer review is overlooked in favor of just that one number. And a lot of really highly innovative ideas, mainly from many early career researchers, just get dropped below the payline and they don't get opportunities. The unified funding strategy is an attempt to try to address this problem.
The NIH — I published a paper with Miko Pacelin a few years back where we found that the NIH was funding research where the papers written by NIH-funded researchers in the 2010s were about seven or eight years old. The ideas in the papers were about seven or eight years old. If you do the same measure in the 1980s, in actual funding, research that was zero, one, or two years old — the ideas that were zero, one, or two years old — we've become much more conservative in our willingness to fund research that is at the bleeding edge and that has the potential to overturn fields. The unified funding strategy gives institute directors the ability to pick projects that are highly innovative, maybe have a great innovation score, and taking into account peer review in a deeper way than we have in the past, rather than just looking at the overall score, which as I said correlates very strongly with methods, resulting in essentially a very, very conservative portfolio. Now the portfolio will take into consideration all of the peer review, and the portfolio will match the strategic aims of the institutes, which it doesn't always do — like sometimes you get fads where there's 10 great applications for the same idea and they all get funded because it's a good idea, but as a result parts of the strategic plan don't get funded. The portfolio will allow investments in early career researchers to a greater extent than was previously possible. And so it puts more responsibility in the hands of the institute directors to take all of the peer review into account and build a portfolio that has the capacity to take scientific risks.
On replication — I'm not going to go through all the numbers on this, but they are kind of shocking. If you look back at the literature on replication, I mean there are these audit studies looking at field after field — psychology, neurobiology, cancer biology — and they look at the top papers in those fields, a random sample of them, and they assign independent teams to look at those. And what they find over and over and over is that often the papers aren't — the methods aren't described in a way so that replication efforts or reproduction efforts are possible. When people do finally look at and try to replicate the findings, they routinely find effect sizes that are much smaller than are reported in the original paper. And I don't need to get into all the scandals. The problem really fundamentally is one of the sociology of our profession. How do we decide what's true or false in our scientific fields? And bluntly, it now falls essentially to an appeal to authority. Is the paper published in the New England Journal? Well, therefore it must be true. How often — and I'm not going to ask you to raise your hand — how often do you look at a paper, you look at where the paper's published, you read the abstract, and you decide whether it's true or false based on just that. If we're honest, we don't really read the papers deeply. And as a general rule — now I know there's people looking at me going, 'No, yes I do' — I mean, yeah, I know you do also, but it's hard because the volume of papers is so large. And it's really tempting then to appeal to authority to decide whether something's true or false. Is it published in Nature? Is it published in Science? Is it published in JAMA? Therefore it must be true. Instead of, well, what do the other scientists looking at the same question find? What do people looking at it from a different point of view — do they tend to find similar things? That sort of consilience is really how we should be deciding what's true or false in science. Science works because it has a bottom-up epistemology.
A bottom-up epistemology where the scientific community, carefully looking together at a scientific question, independently of each other, decide that they believe the same thing. That's how we determine truth in science, right? Rather than a top-down epistemology where scientific authority decides what's true and everyone falls in line. The replication crisis is a symptom of this epistemological crisis. This replication crisis happens because we have allowed authority to determine what is true or false in science rather than this bottom-up epistemology. The solution, then, is to elevate to the attention of everybody what do other scientists believe when they look at a paper, when they take it seriously. Right? So we're going to do a few things at the NIH. One is we're going to start funding replication work — like you'll be able to get R-level funding for — and when I say replication work I mean it very broadly: narrow replication of particular findings, meta-analysis of broader sets of findings, reproducibility where you look for different methods to look at the same problem, a whole range of sort of the science of replication will start to get funding. We'll have a platform where you can publish your replication work. If you're NIH-funded, you'll be able to publish — it's very difficult to get these papers published in scientific journals. Now, we're going to have a scientific journal at the NLM where you can publish your replication work. It's going to be a little bit of a different journal because the key thing is there's going to be very minimal gatekeeping. Minimal gatekeeping, right? So reviewer three won't be able to kill the publication of your paper, but there will be peer review. There'll be lots of rigorous peer review published alongside the paper.
Then there'll be enhanced visibility of replication work in PubMed. Imagine — and you don't have to imagine because you're going to see it relatively soon — a replication button. Every time you search for a paper in PubMed, a replication button comes up next to it. You click the replication button and then there's a knowledge graph of all related findings for each finding in the paper. You can go click and see what all the other papers in the field that are linked to it say. And there'll be an AI summary of what those replication findings are. And there'll be a little feedback button so you can tell us what we got wrong. So it will elevate what the scientific community believes about those ideas. Broadly, it'll reflect disagreement when there's disagreement as the primary thing that you interact with when you interact with PubMed, rather than just where is it published and what the abstract says. And finally, we have lots of metrics for scientific contribution. So rather than just looking at H-indices, which are just a god-awful measure — a mix-up of volume and influence — as the primary measure of scientific productivity, we'll have other measures of scientific productivity that we'll publish and disseminate that include things like: how often do you share your data when you publish your papers? How often do you share your code? How often do you share your tissue samples? How often is your work subject to replication? Replication now is a threat, right? The sociology of science is so sick that if someone comes to you and says, 'I want to replicate your paper,' you're going to view it as a threat to your career. Think about that. In fact, what it should be is that if someone comes and seeks to replicate your work, that's an honor for you, because your idea is so important that it's worth being replicated, worth the attention of other scientists.
Right. So, by putting metrics — How much time do I got? I got 15 minutes. I better hurry. Okay, let me — yeah, I got QA. Including QA? Oh god. Okay, let me just — let me go back to — let me talk about methods. Let me go back to health economics. I started the talk with about the pillars. Let me talk about why health economics is important. I mean there's lots and lots of reasons. My old colleagues Brian Tiessen and Julie Smopoulos wrote this interesting paper about economics being able to identify high-value treatments. There's lots and lots of great papers that have been funded by the NIA and other institutes on this. One was a really interesting paper — a seminar just organized last couple weeks on the zoster vaccine and dementia risk. This project was essentially launched by a paper that uses natural experiment methods that were pioneered in many ways by economists. The goal of health economics work that is fundable by the NIH is to achieve the NIH's mission. And as I said, I've spent my career on this. I fundamentally believe in the power of economics to help achieve this mission. Now let me describe three principles for NIH-funded research that I think can protect this portfolio. And so here are the three rules, the three pillars. Principles: the research has to be in scope, it has to be rigorous, and it has to be actionable. And let me describe each of these in turn. And actually I'm going to focus mostly on in scope.
The key thing about NIH research being in scope — remember what the mission is, right? It's research in living systems. The key thing about research being in scope has to do with whose decisions are being supported by the research. Whose decisions are being supported by the research. So most obviously, if your research helps people make better decisions about their own health, well, that's in scope. You're helping people make better decisions about their health, that's obviously in scope and it's within the NIH mission. If your research helps doctors figure out how to manage patients better, or helps patients decide to take their doctor's recommendation more seriously, that's obviously in scope. Nurses, other healthcare providers, psychiatrists, psychologists — if your research helps them make better decisions in their capacity to help their patients, that's all in scope. That's obvious. If your research helps scientists design their research better for the next experiment, the next experiment which is in scope of the NIH mission, then that's also in scope. So in scope is all of the basic science portfolio that we have. If your research helps hospitals in their public health capacity to make decisions that can help manage their patients better, well, that's in scope. Let me make a distinction there. A hospital also has a financial role, right? It has a capacity to try to — maybe it's a for-profit hospital that has to make money or it's a nonprofit hospital that has to not go out of business. If your research is helping hospitals in the financial capacity, well, that's not directly in scope. It's the same entity. Your research helps on the public health decision-making capacity of the hospitals — in scope. If it's helping on the financial capacity — out of scope.
Let me give you an example of a study which sounds really cool to me but probably be out of scope. Imagine a paper that you're looking at interest rates and some health outcome. That will help the Fed make decisions about what the interest rates ought to be, right? Cool paper, but not directly in scope because the decisions you're supporting are outside the biomedical research infrastructure system. The policy — and here's the rub — policymakers. Oh — actually before I get to that, I should say public health. I wear — hopefully not for too much longer — I wear the acting CDC director hat. It's really useful to have NIH research help you make decisions in that role. So if your research is used by public health people in public health, that's in scope. How about policymakers? That has to be out of scope. If the primary goal of your research is to inform policy, and in particular to inform policymakers or to advocate for policy, that's out of scope. Now health economics automatically, by its very nature, very often will help policymakers make decisions. So I want to be very clear about this. I'm not saying that if your research informs policymakers it's automatically out of scope. I'm saying that if your primary goal — and you start your paper or your proposal saying the most important part of this research is that we're going to inform policy, and it's not to inform patients or doctors or hospitals or public health people in public health. It's to inform policy. It's to inform legislators about what they should vote on or how they can construct legislation. That's out of scope. That can happen as a side effect, right? You're doing research, and very often researchers have no idea who's going to use their research, right? If you're doing your work, that's almost certainly going to be the case. I got eight minutes.
I better hurry. Okay. So, I'll take questions about scope. Rigorous — I can — this is like chum for health economists. We'll just sit there fighting about whether you really have a good instrument. The key thing here is, as far as health economics is concerned and social science is concerned, are you confusing correlation and causation. I'm not going to get into this. But it finally has to be actionable. You have to be looking at ideas that have a chance of actually changing the actions of the people that you're supporting the decisions of. Right? So non-falsifiable hypotheses, for instance, if you're testing those, you're likely not actionable. Okay. These principles I want to apply not just to economics but to all of the NIH portfolio. These are neutral principles that will protect the NIH portfolio from political interference, and it'll allow all fields to be treated in parallel rather than taking health economics aside and putting it in the woodshed. Actionable, rigorous, and in scope. A lot of this — we'll start to have discussions about this, and we're having listening sessions. John Phillips, who's in the room — where's John? There's John right there. You can go talk to John and other folks who were in the room to talk through how we're implementing this. And I'm looking for feedback from the research community about this. All right, I have seven — it says 7:08, Julie. So I got seven minutes left for questions. So let's open it up.
