FRED Con Live: The Now and Next of Economic Data

Okay. Good morning, everyone. On behalf of the St. Louis Fed and the FRED team, I want to welcome everyone joining us in the auditorium and those watching the live stream to FREDCon, where we'll be discussing the now and next of data. My name is Keith Taylor, and I'm the director of research information services here in St. Louis. I am also going to be your MC today, so I apologize — you'll be hearing from me a lot. A quick disclaimer: my views are mine alone and do not reflect those of the Federal Reserve Bank of St. Louis.

to educators through in-person outreach and our award-winning website federalreserveeducation.org. You can also get to it at fere.org, which is a little simpler. We tried to get like a frank.org or a freda.org going to go with Frasier and Fred and Alfred, but that didn't work out. And then finally, we have NEPHE, or Native Economic and Financial Education Empowerment, which provides financial literacy outreach and resources to Native communities. An interesting thing about all of our teams is that they function both as producers, creating data and content, and consumers, curating and aggregating information.

Over the last few years, the FRED team and really all of our teams have been wrestling with the challenges and opportunities posed by AI, trust, storytelling, and accessibility. These challenges and opportunities are heightened by how quickly change is occurring, creating really a sense of urgency in us. We've been asking ourselves, you know, how can we best position FRED to make the most of Gen AI while minimizing our exposure to risks? How can we preserve and enhance FRED's reputation as a trusted source of data since 1991? How can we best tell the stories behind the numbers in FRED? And how can we continue to provide easy access to the hundreds of thousands of data series found in FRED while reaching millions of people each year?

Talking to our users and partners, we found that they were grappling with many of the same opportunities, challenges, and questions. We also found a shared sense that we are sitting at an inflection point right now, and getting the next few years right, getting the answers to these questions right, is going to really impact the long run. We have ideas and aspirations, but we don't really have the answers. To help find those answers, we organized FREDCon to really start a conversation and learn from all of you, including the folks that will be on the stage today — Governor Waller, our moderators and panelists — having conversations with everyone in the audience, and engaging with folks who are watching the live stream who will undoubtedly be taking to social media. I'm so excited to learn from all of you today and really begin this conversation. So, thank you for attending, and I hope you enjoy the day.

All right. So, now I'd like to welcome Governor Chris Waller to provide some opening remarks. Christopher J. Waller took office as a member of the Board of Governors of the Federal Reserve System on December 18, 2020, to fill an unexpired term ending January 31, 2030. Prior to his appointment at the Board, Dr. Waller served as executive vice president and director of research at the Federal Reserve Bank of St. Louis since 2009. I will also point out that he was my boss's boss when he was here, and he's a huge champion of FRED. So, thank you for all your support over the years. In addition to his experience in the Federal Reserve System, Dr. Waller served as a professor and the Gilbert F. Schaefer Chair of Economics at the University of Notre Dame. He was also a research fellow with Notre Dame's Kellogg Institute for International Studies. From 1998 to 2003, Dr. Waller was a professor and the Carol Martin Gaton Chair of Macroeconomics and Monetary Economics at the University of Kentucky. During that time, he was also a research fellow at the Center for European Integration Studies at the University of Bonn. From 1992 to 1994, he served as the director of graduate studies at Indiana University's Department of Economics, where he also served as associate professor and an assistant professor. Please join me in welcoming Chris Waller.

Yeah. Yeah. No, we'll just — we're super casual here on the FRED team, as you can tell by all the t-shirts around. So, thanks, Chris. Those were really great opening remarks. I just have some follow-up questions to hone in on the themes of the conference. So I guess to start off, given your academic background, what's your perspective — how do you assess trust in the data that you're using to make decisions?

Thank you. So, like, listening to your speeches, reading your speeches, you really have a great way of reducing very complex topics into, you know, simple analogies, metaphors, things like that. Can you just talk about what's your process for that, to reach these broader lay audiences?

So the old 'consider your audience.'

Awesome. So, drawing on your long professional arc in economics, what are some of the significant moments that changed how you accessed and used data?

So, you know, speaking of graphics and visualizations, do you have any kind of go-to data visualizations? Do you have preferences — line graphs over bar charts, things like that? How do you feel about pie charts? Also a contentious issue.

And then, in your personal use of AI technology, what's been your most impactful experience?

Then finally, you've talked about this a little bit in your remarks, but why is it important to bring together experts from different sectors of the economy to discuss these kinds of issues?

Well, thank you. Thank you very much. And this was really great. And again, thank you on behalf of the St. Louis Fed and the FRED team for all your support over the years. And please join me in thanking Chris Waller.

So before moving on to our first panel, I just want to cover a few logistical items. For attendees in the auditorium, there is a microphone in front of you. And when we get to the Q&A during panels, if you just touch the gray space in front of the microphone, it'll turn green, and then you can start to talk. Sometimes there's a little lag when you push the button, so just give it a second. And then also, now would be a good time to remind people to silence their phones. We do have an online audience, so we don't need those disturbances. And if during the presentations you need to step out — if you're in the auditorium, obviously not if you're online — but if you need to step out, there's a door in the back there, and the bathroom will be on your right as you go down the steps.

All right. So now I'd like to introduce our first panel on Gen AI. While all of our panels will have some level of overlap, we'll see Gen AI addressed in the context of other panels as well. I mean, when we set up this conference and we picked these different categories, they were somewhat arbitrary. Storytelling, data accessibility, trust — they all kind of flow into one another, and AI really highlights, I think, those interconnectivities. So you'll be learning here from experts at the intersection of Gen AI and data, and how they think about the future of finding, interpreting, and communicating economic data. Our moderator is Brian Quistorf. Brian leads data science efforts at the U.S. Bureau of Economic Analysis as its chief data scientist, where he works at the intersection of economics and machine learning to modernize national accounts data. Before joining BEA, he was a research economist at Microsoft Research, where he worked on applied machine learning for economic questions. Please join me in welcoming our Gen AI panel.

Great, great. Good morning. Super happy to be here. I think the themes of the conference today are wonderful, and really happy to have a conversation with these wonderful panelists. As Keith said, I'm Brian Quistorf. I work at the Bureau of Economic Analysis. And my day job — standard disclaimer for someone working for a federal agency — these views are only my own. Trying to see if we can help bring AI tools to improve government functioning for efficiency and our users. I'll let each of the panelists introduce themselves and mention how AI touches on their role. And then we'll jump in with questions.

My name is Ara Krasen. I'm the lead economist at Ramp. It's an AI finance platform. With that, we see this great data set that covers business spend. And then a year and a half ago, we started tracking AI adoption and other metrics that track the intensity of AI adoption across businesses. It's called the Ramp AI Index, and it's now available in FRED as of yesterday.

Congratulations.

Thank you. My name is Arvind Rajagopalan. I run the data and AI practice at FedEx. I'm sure you've seen some trucks rolling around on the streets. So we make sure that the trucks actually do what they're supposed to and get things on time — essentially what we do. And I'm new to FedEx. Actually, one of the fun facts that I did not know is FedEx is one of the largest airlines. We have over 700 aircraft, interestingly enough. So the range of problems that we are looking to solve for go all the way from how do you optimize a route for a truck, and the driver can get things right on time, all the way to what is the best and the most cost-effective way of routing parcels from one part of the globe to another. So very interesting range of things that we're working on. Data and AI — we just started the practice a couple of years back. We're doing some amazing things both internally. So my focus is more so on the enterprise internal use cases. There's a team that does also something very similar to FRED in looking at how do we get the monetization of insights with the data that we have.

Thank you.

Hi, everybody. My name is Jacob Larivier. I lead the central econ and science team for AWS. So that's Amazon Web Services. What we do is we use data, economics, science, and engineering to build policies and technology that makes our customers' lives better. And it's pretty fun. I get to work with a lot of really smart people, and get to work with some terrific customers like people here at FRED. Background: I have a PhD in economics. I was a tenure-track professor at the University of Tennessee. I worked at Microsoft Research in the central econ team there in the office of the chief economist, MSR, and then I moved to Amazon about almost five years ago now, and I've been there ever since. Personal note: I think the first time I used the FRED data was as an undergraduate at Cal. So I've been a FRED user for on the order of 25 years.

Wow. So data is important for all users here. We can think about maybe starting at the beginning of how data is created and produced. How has Gen AI impacted the way that data is produced or created? And maybe we'll first start here with you.

I mean, it's obviously made it a lot faster, and it's allowed us to iterate much faster and very clearly produce better results. We can try different hypotheses and try different sensitivities of our work to see how different assumptions change the actual results. But one of the interesting things about the work that we do specifically is that we also produce work that is specifically trying to cover what it means to use AI. That is the main component. That is the main focus of our research. And so even — you know, we're simultaneously researching this topic of which we are also users that are learning about it for the first time, and that informs the kinds of metrics that we end up reporting and focusing on. The first metric that we started sharing publicly was just the share of firms that are using AI at all, because at the time that's really all that mattered. It was the metric that the government was tracking. It was the metric for which most of the readership and market was familiar with and understood. And it's also really all that mattered. There wasn't really much else to track given that AI's impact wasn't showing up in the economy more broadly. Over time, as the market became more advanced, as AI adoption became more advanced, we then started to think about how we were using AI and then the other metrics that would be helpful to capture the changes in the market. So how much were we spending on AI? That started to grow as a line item on our own bill on the team. Therefore, we figured that it would be helpful to track that across the market and start reporting that metric. So we started tracking the share of business spend that's AI-related. We started tracking per-employee-per-month spend. We started tracking the distribution of AI spend across different kinds of makers of the models — OpenAI versus Anthropic. And then increasingly even the GPU compute providers became more relevant. So you start to see how the increasing technical nature and advancement of AI adoption itself starts to inform the metrics that we care about and we publicly report.

That's awesome. Some very interesting stuff. Generative AI hasn't changed anything for us. Not at all. Like, you know, come on. I think it's — being the — I don't know if I can call myself the tech guy out of the economists that are sitting here and in the room. I'll say tech has been the lowest-hanging fruit, if you will, when it comes to generative AI. And my team — I tell my team, and I was actually having this conversation at dinner with a couple of folks, you know, we got to drink our own champagne first, right? If you don't drink our champagne, guess what? Nobody else is going to want to do it. So we are leveraging generative AI in many, many different ways across what we do at FedEx, especially from a technology perspective as well as enabling business outcomes using it. And I'll kind of go into a lot of different details as we go through the next set of questions. But the key things I just want to highlight here in terms of how it's changed the game is the volume and scale of data that has tremendously gone up, and the ability — as a lot of the economists here, you would probably know, our data scientists, a majority of the time is typically spent finding where the data is, cleaning it, making it ready for consumption, and then being able to use it. So as you think about all of those phases of the life cycle, generative AI is helping us through all of them. That's one key area, and I will double-click into each of those as we go through. The second thing that I want to talk about is also, AI in general is helping us bring physical data closer to digital data. This is something that used to typically sit in two different silos — what's happening, how do you start to bring things together so you can drive value for a corporation or an economy in general, and how do you do it much faster. I think it's been game-changing from both of those perspectives.

So from my perspective, if we think about how Gen AI is impacting the creation of data, there's at least two sides. One is on the input side, one is on the output side. And so on the input side, there's a bunch of measures of like Gen AI usage and things like that. And there's a bunch of different ways that we can think about that. People talk a lot about tokens as sort of like an overall measure of, say, inference and stuff like that. You can also think about training for models. And then on the output side, there's the stuff that we're sort of already looking at — the measures of sort of human welfare and stuff like that and its impact on those. Then the third sort of bucket would be kind of like the general equilibrium effects that we're seeing from Gen AI. So like you can think about like MCPs. We just heard Governor Waller talking about like, hey, we actually have to think about how agents are viewing FRED data and then architect the data so that it's useful for those use cases. And so that's the very exciting stuff, and I think where the most uncertainty is in terms of what the future looks like. And so even if you look at the economics literature — for the economists in the room, if you look at the economics literature, the first papers on, hey, what are the productivity effects on Gen AI, it's really looking at sort of like those output metrics and the narrow sort of impacts on a subset of output metrics like, you know, like software development and things like that. But we're just getting going. When did the MCP — when was MCP — when did that — we're doing a demo today, right? So this is brand new. Two weeks ago. Like we're in it, right? And so I'll be very excited to hear what people have to say about sort of where we're at. But it's really exciting. There's a lot of really exciting things that are happening relating to it, and I'm excited to hear folks' thoughts.

Speaking of, let's pick up on that a little bit more. As we have more and more data, and human time is limited, AI can help us find more data both internally and externally to the company. How are you all thinking about either data platforms or ways to access it? People often talk about AI-ready data. What does that mean for you all?

Well, I think with any technological innovation, there's going to be something good and there's going to be something bad about it. The democratization of data has been tremendous. And the democratization of the ability to do these kinds of analyses, too. The people who are good data scientists and researchers pre-AI are not the same as those who are good data scientists and researchers post-AI. You need to adjust to using these tools and skill sets. At the same time, there's a cost, and it informs how we think about data platforms and our research process in general. Because I went to a presentation by the economist Scott Cunningham recently, and he was talking about some work he'd done around Callaway-Sant'Anna, which is this economics methodology that's kind of popular for doing staggered difference-in-differences. And so he ran the same kind of analysis and question through three or four different AI models with the same exact prompt that produced completely different results for the same exact regression, because Callaway-Sant'Anna involves a lot of kind of fine-tuning and assumptions that you need to bake into the analysis, and different AIs defaulted to different kinds of assumptions and sensitivities. So if you're working particularly on a large team and you're approaching analyses this way and you're no longer writing code directly — you know, I don't think the suggestion is don't use AI, not at all. But the way that we need to do this work and produce organizational cultures around stress-testing that work and making sure that the data being pulled is actually correct, that it's coming from the right sources, that it's applying the assumptions that we would intentionally apply if we were working on it as a team by hand — it's going to come both from the documentation that you build around the data platform so that the AI can refer to something common, and around organizational cultures that are, you know, built over time.

Just to sort of break the sequence. Jacob, why don't you go?

Yeah, so thinking about Gen AI and sort of data products — and so I think that historically, you know, having run large sort of research teams, it is very important that the scientist or the economist knows the nature of the data that they're looking at. So one story not related to Gen AI is there was a project that I was working on once — this is back when I was at Microsoft — and I asked the project manager and the product manager of this data source, 'Hey, what is the definition of this single column in this single table? What's the definition? How's that data created?' And they told me, and I go, 'Are you sure?' Told me again, 'Are you sure?' Told them again, and I was like, 'Great, okay, we're going to undertake this investigation.' Took three really smart people like nine months. We get to the end of it, asked the exact same person about what the insight was that we got from the data, and he goes, 'No, that's not right.' I go, 'What do you mean?' He said, 'Well, that's not the actual definition of the data.' I'm like, 'What are you talking about?' I had, you know, like in-depth...

conversations with that person for like, you know, a month prior to undertaking the research. And I don't know if anybody in the audience has had a similar experience to this. It's not fun.

And so that problem is sort of orthogonal to Gen AI.

At the same time, the ability for, you know, a human to be able to inspect that is increasing in importance because the amount of stuff that Gen AI is allowing researchers to create, the amount of insights is increasing. The cost of generating insights has gone down. So what that means from my perspective is that investing more in like the context layers and the corpus of documents at a company and then making sure that your agentic systems have access to those and that you're using tooling that has really, really good connections to that to provide context is important. And at the same time, ultimately, it's still humans that are responsible for identifying what the data generating process is for everything that you're tracking. So ultimately, I think it's a lot of the same, but the importance of getting it right is going up. And the great thing about Gen AI is that it has broader access to the corpus of documents. And so making sure that we have a technological solution to access that corpus is really important and it's an exciting opportunity.

Yeah. Oh, awesome. Um, thank you. I think you stole a little bit of my thoughts as well. So that's perfect.

Mindbending system osmosis.

There you go. There you go. Um, I think I'm going to just talk briefly about a little bit more fundamentals. Um, how do you think about bringing data onto a common place? Um, historically, if you think about the last maybe 20, 25 years, the grand idea was, hey, let's put all the data together in a warehouse that becomes your source of truth or that becomes sort of your data platform of choice for all kind of analysis and whether it's simple business intelligence use cases creating a dashboard, um, or talk about, you know, data scientists wanting to run something against it, you know, run some hypothesis and test it, train it, etc. Now, I think the democratization, the federation model is working really well at large scale. So, I think one of the great examples is the MCP interoperability and the connectivity that was being talked about. While all of those things are making a lot more advancement, I think the fundamentals still are true. Ensure that you have a source of truth. Ensure that we have the right level of governance. We are protecting our data in terms of how do we classify each and every attribute. How do we tag assets and attributes and elements that becomes extremely critical. How do we make it available for search and discovery within a corporation, outside the corporation? Typically we are all used to doing keyword searches. Now that's kind of changed dramatically with generative AI. We're going into semantic search, the context piece that Jacob was talking about, because the context of what and how you're looking at certain attributes can really change the definition of what it truly means and the answers it gives. So, it's very important to generate context, validate it with data stewards and owners, ensure that it's available for search and discovery across the enterprise. These are very foundational and fundamental things that can very easily get overlooked. But then when you start asking questions and the answers start coming up like the example that we were talking about here, you get, you know, you ask the same question to four different models and you get four different answers, then you start freaking out what is right and what is not. So it's very important to sort of ground those models, give the right set of context. So those fundamentals do not go away. And then beyond that, things that nobody wants to talk about is the plumbing. How does data move from point A to point B? What kind of data contracts are established? Do we have the right level of quality of data, observability of data? Are we managing the right level of freshness? All of these become fundamentals, which actually I would say that Gen AI is starting to amplify the importance of why these things need to be done and there needs to be a good amount of investments in setting the foundation right and putting it in place so everything else that you build on top, like this 100-plus-year-old building, can stand the test of time.

I'll so iterating that or jumping on top of that, I guess, is that I've talked to a lot of people and they've said that the relative importance of data engineers has gone up.

Yeah, absolutely. Values moving from the programmers down to the data providers and the data engineers. Yeah.

Um, you know, selfishly, I think about a lot of the things that Arvind, you were just mentioning. And I think in a lot of organizations, they might have not had as much visibility across the organization in terms of data and processes. Simple things as ensuring that code that, you know, makes data is in a repository internally somewhere that's visible. That can take a lot of work to kind of break down those silos. Any lessons learned you might have from trying to do that work?

Um, code repos with what has evolved in the last eight, nine months with Mythos. Um, I think it's tons of lessons learned. I'm not sure how many of those lessons I want to share here. Um, scary thought, but you know, 100%. I think one of the things that the cloud opportunity opened up for a lot of corporations is everything became templatized. You run with blueprints and infrastructure became code, right? Infrastructure as a code became a norm. Before that, a lot of these activities would sit in between the ears of people, experts, the platform engineers, the database administrators, the systems administrators who would know exactly what to go and click, what kind of scripting to write. None of those things typically sat in repo. So a lot of that now sits in a code repo and you can learn from it and AI can learn from it and it can help you generate better, faster, cheaper code as well. So I think it's really helping scale operations in terms of how we are thinking about it for sure. There's no question about that. The other aspect of it, you know, that I briefly touched upon is we're also leveraging Gen AI to look at all of our hundreds of thousands of data assets and generate a definition of it, capture the semantic, capture the ontology. And that not just looks at, you know, the name of the table or the column to the example that Jacob was giving earlier, but it also looks at the data itself and understands what it needs to be and what the right context is, right, and how is it being used. So I think, you know, with a lot of the economists here, statistics still play a very important role in terms of how you look at it, how you look at data, how you're trying to capture more metadata about it and how you can put it to better use. And I think all of those are absolutely true and there are tons of lessons learned from that standpoint that probably it's going to require me to write a paper.

Wonderful. Um, Ara, picking up on the point you made just before about how when AI agents are used at the analysis stage and people are necessarily kind of distancing themselves a little bit from that work. How do you think about things like trust and reproducibility, maybe internally at first, internal within an organization, and potentially there's different constraints when going outside?

Well, anyone at the company can now put together an analysis to support a specific narrative that they're trying to push. Not that people are trying to, you know, be political about that, but it means that people who are not data scientists by training can now vibe code a dashboard that then ends up being used to make important company decisions without ever involving a data scientist who actually touches that data. And you know, I think historically the, you know, we would think about situations like that as a frustration for the data science team and almost something to lock down and prevent. And I don't think you can really do that in this modern age of working in an AI world. You kind of have to skate where the puck is going and recognize that you will have product managers and even executives at the company that will query your large data sets and therefore it is incumbent on you or incumbent on the data science team rather to figure out what those people need and what they're looking for. Either create those dashboards for them so that they don't have to vibe code their own thing that uses the wrong definitions or lock down the semantic layer of the data sets themselves so that when they're accessing these common data sets they are referring to a single source of truth. And it's much easier said than done. Like we have a Slack channel at Ramp because we're pretty tech forward on this and I think we're pretty, we were pretty far ahead at least on using AI to answer data questions. But instead of asking a data scientist a data question, now you can literally ask an AI agent in this like common Slack channel and it's not, you know, there are hundreds of questions in there daily so it's not like anyone's monitoring it that closely. But our CEO is asking questions in the channel about, funny enough, like AI adoption and like the share of the sort of dollar amount of AI spend that goes to the platform, which then kind of funnily overlaps with my work because it requires defining what AI spend really is, which is also becoming just increasingly complex. It's not just going through OpenAI or Anthropic, nor is it just going through a GPU vendor. It also might mean sort of AI features whether or not we can capture that. And you know, I found this like historical thread in which the CEO got an answer that I would have disagreed with if I did it myself because he, you know, got an answer from an AI that used assumptions that I wouldn't have used. And so in the pre-AI world or in a sort of less AI-forward world, you would look at that and think like, why is my, why is the CEO doing that? He needs to like focus on his job instead of asking these data science questions and then just ask me for the dashboard. And you realize that is a battle you will not win nor do you really want to fight. You should recognize that your stakeholders will use, will go to these tools when they need them. And every team, even outside of data science, is now better off thinking about how to better enable your stakeholders with AI rather than preventing them from using these tools at all or letting them get the wrong answer.

Yeah. Great. Jacob, you've got a team of economists trying to work on insights. How do you think about this problem?

Yeah. So, like with most new technologies, I think historically as a new technology comes out, the people that are best at using it tend to do better for themselves, right? And if you think about Gen AI, the amount of output has gone up. And so what that means is that the value of assessment has gone up relatively more and being able to parse the wheat from the chaff. And so to that end, the interrogation of output is extra important. And importantly, like Governor Waller was saying, being able to communicate the core intuition about why output A is better than output B is better than output C is really important. So in the example with the, you know, the four different models, chances are, you know, ultimately you're going to get to the empirical question of, well, on the basis of the data generating process, if you assume X, the correct demand in this case, if we're going to get technical, is from option A or model specification A, or at least it has one type of risk versus something else. And so I think that's really the scarce resource. And I think that the scarce, it's been a scarce resource for a while. I think that, you know, like Arvind, you've, you know, led large teams. I mean, talent is the scarce resource.

Absolutely.

And so I think that the relative importance of that goes up because the total amount of stuff out there goes up. And then in terms of like, I can give you an anecdote. So AWS has this product called Kira, which is a really nice coding, you know, I guess I don't, I'm not the product owner so I won't say exactly what it is, but it's a really, really good tool for being productive, I will say that, and doing some analysis. And so I wanted to see how this stuff worked. And so when a new version of it came out, I had a partner provide me with a data set and I was doing a type of causal analysis and I was able to do what would have taken historically probably something like two to four weeks to do. I did it in about 90 minutes. Interestingly, the first result when I did my first sort of prompt was the effect of X on Y is positive and significant. By the time I got to the right last model, it was negative and insignificant, right? And so the insight totally changed. But as any researcher knows, that's the life of any project is that it's going to evolve as you sort of learn more about what the right thing to do is. So the key thing from my perspective is that the scarce resource that has always been around is just going to become relatively more important because the amount of stuff goes up.

Shifting to post-analysis, communicating your results through visualization or writing. How are we seeing Gen AI affect those forms of communicating results within and outside an organization? And you can start again, jump in.

Internally, just even at Ramp, I mean, we're changing the vendors that we pay for, even just foundational vendors that a data team would typically invest in. We're moving off them entirely because they don't work effectively the way we expect them to, our stakeholders expect them to in this AI world of software. We're moving off of Looker. Who would have thought that we would move off like the most, one of Looker, Tableau being just foundational dashboarding, you know, software vendors, but have barely been updated to work well with agents or to create dashboards iteratively or add to them in the past couple years, whereas there are now actually vendors that are very AI-native and solve that data production problem. And that's just internally. But externally, I mean, you can see this in our work too. Just the first of all, you see those sort of micro trends that we observe inside the company are happening simultaneously slowly and also more quickly than you might expect in terms of the macro business spend data set. You know, Looker and Tableau still being foundationally the number one most paid for vendors in data visualization and just data communication as well and in BI tools in general. But month over month, you see AI-native competitors rise by single-digit percentage points in adoption rates. You know, you wouldn't normally expect that in a typical software development environment or any SaaS vendor environment. I mean, that's extremely rapid. Are people canceling their Looker subscriptions at scale? They're still the dominant players, but you're starting to see these shifts because of, you know, consumer and business demand for tools that work for them.

I'm going to add to the point that you were making, Ara. So, I think there are a couple of things that I see that are changing dramatically, right? One, companies have hundreds of thousands of dashboards typically, right? Over the years, there is a dashboard sprawl and you don't even know who created what and how long it's been sitting there and who's even using it, all of that stuff. Now, I think the trend really is that that is shifting dramatically where people are just able to ask a question and get an answer. At the end of the day, the fundamental reason why dashboards existed is to build a narrative and tell a story on what is exactly happening with data, right? So that goes back to the storytelling point. And now, what is more intuitive than you being able to ask a question, get an answer like that, right? Okay, well, maybe it's not like that. Maybe it takes like 30 seconds. Fine. But you still are able to do that. So that's one big thing that's changing in terms of the trend. The second piece that I see is this is a typical boardroom conversation, operations reviews conversations where I'm sure all of you have been. You go and present, you know, something and there is a question that's being asked on a specific data point and I said, what is that or how did you come up with that? And then that causes a whole exercise where like eight analysts go back and spend like two weeks trying to figure out exactly what happened and how that number came up, right? So the speed at which folks are able to answer those questions now has dramatically changed. Now you don't need to do that. You can ask the question right there, especially if the right context and the semantics are set up. You can ask the question right there, what is that data point and how did you arrive at that? You will get an 80% or 90% good answer. It may not be 100% to Ara's point unless a data scientist certifies it. But you get an 80, 90% good answer that you can respond to immediately. That is the other thing in general, the speed of analysis of data has dramatically changed. The third thing that I want to see is that is obviously aided by cloud providers having the right level of compute resources and what you're able to crunch now much faster on your phone versus what needed a supercomputer 20 years back or 30 years back. So I think those advancements have made a drastic difference. I'm sure a lot of you here, by show of hands, I think Governor Waller was talking about it. Show of hands, please. How many of you do time series models forecasting for your day-to-day job?

Yeah, right. Have the half the team, you know, you have the folks here. One of the providers just launched and open-sourced a time series foundation model, TimesFM, right? And you know, some of our revenue forecasting folks have been noodling on it. They're working and trying to like figure out what's happening. It has dramatically increased what they were able to do, the number of simulations that they were able to run and the forecasting, you know, Monte Carlo being the gold standard for many, many years. Like, you know, things are now accelerating the speed at which we're able to answer questions, simulate. This used to be again a two, three-day exercise. Somebody says, oh, what if this were to happen? What if, you know, something changed in the Middle East or, you know, fuel prices go up? You know, being able to do these simulations so rapidly is a game changer now. So I think speed, speed is the bottom line in terms of what's changing.

Yeah. Making the data available too. I was because when Governor Waller mentioned just during his remarks that FRED has seen 150% year-over-year growth in terms of users and also that now more than half of the users are AI agents and bots. I can understand that and it makes complete sense because FRED has always been designed in a way that makes data accessible and obviously is a credible source of data. But then you then think organizationally, both internally, externally, whether or not your tools or your data publishing practices are optimized for agents at all. Because when we started publishing Ramp AI Index, we were not necessarily thinking about whether or not agents would need to download and access data. Coding agents were not adopted in a broad way yet. And so the first way we started publishing it on our data set was we used like an embedded data visualization tool to help us create a pretty chart that had some like download the data button that people could access and click, but it was an embed. And so when we ran it through an AI, the AI couldn't see it. And so we realized that our data was not accessible to AI agents and then would probably get left behind in now this world that we're in where we want researchers to access our data as quickly as possible and they're going to do it through an AI agent that would run into a wall. So we had to literally update our website to post data in a way that was readable to AI agents in a way that is not necessarily intuitive unless you've, you know, you understand a little bit of like HTML and why something shows up and why something doesn't.

I think that's a great point that you make. I just want a quick 10 seconds on this. You know, a lot of the output artifacts that are being generated now as we build from engineers is both human and machine readable and that's very important to the point that you're making 100%.

So plus one to everything in particular, this federated data insights as sort of like the way that in a lot of institutions things are evolving. I remember about two, two and a half years ago, I remember talking to an executive at Amazon and I was like, do you, is there any additional sort of like metric that you could use to help achieve your goal? And she said, I am up to my eyeballs in dashboards right now. I don't want any more dashboards. And so ultimately, I think what people want are insights is like, hey, the data says we should take this action. Like that's ultimately where you're trying to get to to help your customers or to create a better product or to enable more productivity. Yeah. And then I don't really have a ton more to add than what's been said, except that I think articulating why one particular approach is better than another approach for the analysis is I think really important because just the amount of stuff that's coming out is increasing a lot. And so being able to articulate in simple language why one methodology or one research design is better than another or at least its comparative advantages, what it mitigates risk with respect to is really, really important. And so I think that's a big one. And then also this notion of simulations. So from my perspective, Gen AI from an analysis and a research perspective, it's free execution. You say exactly what you want your prompt to do and it will execute that. And it might execute it in a slightly different way, but that's just an imprecise prompt from my perspective. If you write the right prompt, it's going to execute that specific prompt. And so it's really an incompleteness. So if you know exactly what your prompt is and if it's well articulated, it's going to do that very quick. What that means from my perspective is like a time series analysis, like a prediction problem, it'll be good for that. But more like simulations and like structural modeling, like trying to figure out, okay, what's instead of just trying to, you know, if you see, to use an Alan Sorensen example, if I'm just seeing dots on a plot go up like that, well, you're going to say, okay, extend the line. But if that is like the height of a baseball after it's hit, well, then it's supposed to go like that, right? Because there's some structure on the problem. Figuring out what the right underlying structure that like your customers are exhibiting or like the economy is exhibiting or things like that, you can just do these robustness checks super quick. And so the relative value for like structural modeling I think is really large for Gen AI. But you got to make sure that you have very well complete articulated prompts because ultimately you have to interrogate that and put your name on it as a researcher and somebody who's providing insights. There's just because you use some Gen AI, some of your own stuff, you're still liable and you're still the expert.

Great. So AI is changing processes in a lot of workplaces. What have been some challenges that you all have faced internally either trying to advocate for more adoption or, you know, mitigating risks that comes from changing processes? Open up to anybody.

Yeah, I mean, I can go. One of the things that actually very refreshing for me as we're working through having conversation on how do we scale AI, the executive team at FedEx extremely supportive and they see the value. And the beauty is, you know, I'm not selling anymore. So, you know, it's really about focusing on what more can we do because there is massive demand. But one of the things that has really aided in that journey, even before I joined, was just making sure that everyone in the company is AI literate. And I think that is an extremely important thing. And we started doing some of that before we started opening up, you know, the AI toolkit to everyone. So people understand all the way from our CEO down, everyone understands what responsibility they are going to take as they start to leverage AI and different, you know, tools that are being opened up, whether it's Gemini, whether it's Claude, whether it's, you know, whatever those models are, it doesn't really make a difference. But I think the key is they need to understand the responsibility that you're going to give a prompt, it's going to give you an answer, it costs money at the end of the day, tokens are through, and you need to have the right guardrails. So all of those things, having the basic understanding of all of that has made life much easier in terms of driving adoption. So that is one critical aspect that I would talk about. The second piece is, you know, governance for both data and AI are becoming paramount. Having the right policies in place, having the right guardrails in place. And we're standing up our AI risk models where, you know, every AI model that is being built, you know, whatever the case may be and the use case is, it looks at, you know, close to 32 or so parameters to try and figure out a risk score. And based off of that, we try and figure out what do we do with that now, right? How much due diligence need to be paid, what kind of setup and changes need to be put in place before you actually can launch the model. So I think, you know, having that sort of a risk-based governance approach is working really well for us. And I think it also helps drive adoption in terms of what it's changing in terms of business processes. Now, having said those things, one of the front and foremost things as we think about release corporations is not try to apply AI as a band-aid solution for a broken process. It is very, very important to look at what is your business process. Is there an opportunity to reinvent or reimagine that business process with a new lens of looking at it through AI versus applying AI on a broken process? It's not really going to be a whole lot different than what we saw in the last 10 years with robotic process automation. AI will end up being that and you absolutely don't want to do that. So I think, you know, we are actually spending a ton of time as part of our transformation journey to really look at our business processes, reimagine them and see how do we do it better, faster, you know, from an AI standpoint. And as part of that, we have very aggressive target to put 50% plus of the workforce workflows and identify them. So, you know, bring a lot of capabilities and that takes away a lot of the white space in workflows in the corporation. So very aggressive targets, amazing things that we're doing, but the important part is not to forget business processes. So processes are key and it's very important to reimagine them with a new lens.

When we talk to businesses, I actually don't think there's many forces that are hampering AI adoption within a firm that wants to use it. There are plenty of, you know, people who encourage AI adoption, but when we talk to businesses, like the main thing that prevents greater AI adoption is generally ROI. And they want to see the ROI of this technology, which is so funny to me sometimes because I get it, but it's funny to me because if you were ever, there's no data set that informs anyone about the ROI of computers, you know, or any other technological innovation. There was perhaps, there was in the 90s some instinct to try to measure the value of computers, but like we never bothered to measure the ROI. We just all got computers over time. At the same time, I understand why there's this instinct within the business to try to measure the ROI of AI specifically because it is so volatile in terms of its spend. And also the purchaser of AI, the company that pays for it is not the controller of that AI spend. You essentially give it to your employees to use and they can run up bills as high as they want. And so because they can't control and because it's so volatile, there's this desire to have the one single metric to measure ROI to define how much we should spend on it. And then, you know, I think people forget, unless if they work with a lot of data in particular, if they're researchers, how hard it is typically to get that one perfect metric that says exactly the thing that you want. And unfortunately, what that's led to, and I'm going to say something negative about tech and AI for a moment. When you say that, like people like put their head up. It's like, oh, someone's saying something bad about AI finally. You know, we end up seeing these metrics, particularly from tech companies, by the way, are the biggest defenders of like the share of people at our company that use AI or the share of code that is now written by AI, number of lines of code that are written by AI. And those are so unsatisfying for everyone who's normal because you don't really see in our day-to-day life that software has gotten that much better. There's a really good chart in FRED. I think it was from a paper, from like a research paper, but it was like the number of apps on the app store has gone way up since the advent of AI because anyone, everyone can write apps, but the revenue per app, the revenue on the app store has actually gone down over the same period of time because all the apps suck and probably the rating of them for them has gone down over time too. And so, you know, those sort of metrics about AI adoption are not necessarily what we even want to drive towards because they don't deliver the, they don't communicate ROI the way that we, the end user of these tools wants it to be communicated, nor is it understood that way. And I will say like the companies that do it well when they try to measure the ROI of AI, you know, they will define one specific metric that correlates to an end improvement for the ultimate user. Like I talked to a company, like a CFO of a tech-ish company that had defined a share of customer service inquiries that they want to be automated and addressed entirely by AI by the end of the year. Okay, that's a great metric because you are getting at something that is ideally improving customer outcomes. I think Ramp actually does it well like internally as a team. You know, one of the metrics that we report monthly is the share of receipts on our platform that are addressed without human touch. Okay, it's like an expense management platform powered by AI. Ideally, all the features that we are producing make it easier to do your expenses such that no one has to think about doing them by hand. Now, the unfortunate thing about that though is that it means that every single organization is going to have to define its own specific metric for, you know, measuring the ROI of this technology. And that's just not satisfying. Nor is it the way that we've gotten used to how we access and interpret data. Because as much as I love FRED and I genuinely love being here, what FRED has done to our discourse is, fortunately or unfortunately, lead us to expect one perfect metric to capture a story. And unfortunately for the ROI of AI today, I don't necessarily think we will get one perfect metric that can be aggregated across the entire economy.

I have some thoughts. We are very early days and like there's this local question of like, hey, within your org, how do you get AI adopted more quickly or efficiently, say? And yes, to security should be P0, but we are so early days. And so a couple of examples. Anybody know when the iPhone came out? Anybody remember this off the top of their head?

2007. When is like the first like app that came out that people sort of that like kind of changed the way that people lived?

There you go. Well, we now know who to hang out with after the conference. So like, you know, Uber, I think Uber went like beta in like 2011.

Yeah, something.

We're very early days. Another example. So when I moved to Seattle, there were these two companies, Carigo and ReachNow. Anybody remember these companies?

These are companies that just... So Brian's smiling because he was in Seattle during this time. These were companies. This was like during the VC era of the mid-2010s.

Okay. Trying to leverage again the smartphone. How are we going to alter our transportation? So we're still figuring stuff out for like what these really big value props are. And like MCP, when was this? Two weeks ago. We're still figuring it out and so it's very early days. And by the way, what did we get with ReachNow and Carigo? What does that look like at the end of the capital cycle? Scooters.

Right. That's right. And that's, you know, that's the way this stuff works. It's taken time. This is a new technology and we're sort of adjusting the processes. So it's important to make sure everything is safe, make sure that people are supported, have trainings, and then I think experimentation is pretty good and we will figure it out. And the question is, what's the efficient number of apps in the app store and we can have that conversation with the beers.

Or a beer.

We'll do one more and then turn over to audience questions, so be thinking of something we can ask the panelists. Forecasting is always hard, you know, especially about the future, as they say. But what is something that, you know, paint us a vision where we might be going or something you're really looking forward to in the next couple years?

Oh, I think that we've seen the impact of AI and like if you put people on a spectrum of like tech people who think that it's going to have extremely dramatic effects on our economy, either we all have great wealth or we all die. And then you have economists on the other side of the spectrum kind of who I think generally assume that it will have much more modest impacts on our economy, positive but modest generally, but that there aren't many impacts that we can see today. I actually put myself somewhere in the middle but closer to the econ side where I think we will have modest economic gains ultimately but there are still very meaningful strong gains happening today in parts of the economy. They just don't show up in aggregated statistics because of the lack of proliferation throughout our economy and also very few companies use it in any intense way. I mean, coding agents are the most commercially advanced productive enhancing version of AI but less than 5% of American workers are software engineers. I mean, that's a generous number if you include like even the data scientists and other sort of technical roles. And so, you know, I think it's incumbent then on, there's a product issue there, right, where it's incumbent on the AI companies to build versions of AI that can address other parts of the economy more effectively that isn't so focused on technical workers and can expand potentially even beyond white collar work to, you know, ideally boost manufacturing or construction or those parts of our economy. Then there's a policy side there too where what is the role of the government in either regulating or enabling more AI proliferation through our economy. I don't think we figured that out yet because some of the regulations haven't really worked, at least sort of the safety regulations haven't really worked. And then the best ideas about proliferation are to potentially buy parts of the AI companies or subsidize it. I don't think that's a great solution either. I don't know about forecasting it. Therefore.

Well, if the economist has that answer, I don't know what I will say as a technical guy. As a tech guy, I'll just say this. I'm super excited, but can we just like please have one or two super apps? Can we just do that? Like make life easy so we don't have to scroll through the app store or, you know, play store, just, you know, figure out all these apps, read the reviews and come on, like let's just have that. But I think, you know, jokes apart, broadly I would say programs, large programs, transformation programs, migration programs, etc., that used to take four, five, six years, you know, the cycle time on those has shrunk significantly. So that's massive for private companies, you know, everyone actually, just the fact that you're able to get to those outcomes much faster. That is super exciting. And I think is a tremendous shift in terms of how we think about it. The other item I would also say is really around thinking about what's coming maybe a couple of years down the road. You're able to do things a lot more efficiently. I just hope that our, you know, we leverage the power of AI in various shapes and forms to help ourselves as humankind. I'm really excited about, you know, what it can do in the medical and healthcare field. How do we, you know, bring new treatments to treat things that we've not been able to in the past. Really excited about that. I think that to me is a very positive human outcome that, you know, with all this noise to signal ratio, we kind of tend to lose. We talk about all the other things. I think human outcomes, I think what we can do better for ourselves. I'm super excited about that.

Yeah, I agree with that. I think that there's a lot of good reason to be pretty excited and I still think it's pretty early days. The models aren't going to get worse than they are right now. And I think it's also clear that the allocative efficiency, there's two things going on. One is that we don't, we haven't, we're still in the early stages of figuring out the right way to deploy these technologies. And even within that, the allocative efficiency of their...

Deployment is still evolving. And so, you know, now you can choose: do you want to use like high effort, medium effort, low effort? Um, and then which sort of model and stuff like that. So, that just started. Like, for me that feels like it happened like a long time ago, but when did that happen? When did high, medium, and low efforts start happening? Like it feels like a while ago, but from t-shirt sizing to this.

Great. Uh, turn it over to the audience. Uh, if you would raise your hand, we'll call on you and then you can mic in. Amazing. You shut all the economists up and solved all the problems.

I mean, I do have a question, but I don't want to be the first one to jump, but I will do it. So I was at a conference on education last week and they were talking about the use of AI in education and one of the conclusions was that the emphasis should shift from assessment to instruction. So putting more emphasis on using AI to boost instruction to develop the skills versus putting AI to construct better tests or, you know, identify cheating. So I heard kind of a tension, and I love tension in panels, between, you know, Ara was making a point about educating the users of dashboards so they understand what they have with them versus Jacob was talking about evaluating the models and putting them to the test. So, is there a happy medium? Is there a better, you know, what should be prioritized? Educating people about the use of AI and data or testing the conclusions that they make?

I mean, I would even say you need to do — I wouldn't even call it educating so much as I would say like you need to figure out where your users are already working and go and put the right tool or right information in front of them or confirm that they are using the right tool, right? Because the democratization of this technology has made it such that they don't need to ever interface with a data scientist potentially to get an answer to their question. And so it is now incumbent on an often socially awkward data science team to physically go and talk to a salesperson or like an executive and figure out where they work. You know, that's a new skill set that we should probably start hiring for.

I think, you know, I don't know if your question was specific to education. Um, there's definitely a need for — I don't know if there is a silver bullet yet on this, but I think the critical aspect of it is not to lose our ability to think critically. Um, whether it's to write a prompt, whether it's to write code, whether it's to do analysis, whether it's to tell a story. I think the critical thinking aspect of it is very important. And I think that's one of the things — I was just reading a study a couple days back published by MIT talking about, you know, the risk of using AI in education, especially as students are coming up, is their ability to think should not get hampered. And I think that is in my mind most at risk as we need to protect that. How we do it, I think, you know, everyone can choose what works for them depending on what the scenarios are. But I think, you know, humans have evolved over tens of thousands, millions of years to get to a place where we can think critically, and I think that is something that we should not lose.

Great. Well, that'll be the final word for us. Uh, thanks for the question from the audience and can we give our panelists a round of applause? Thanks so much.

All right. Um, thank you Brian, Jacob, Arvind, and Ara. Uh, you know, that was great. Uh, I think ending on the critical thinking piece is really important because, you know, at least for FRED, we've got, you know, millions and millions of users. They are not all experts. Uh, and so that gave us a lot to think about. Um, so we now have a break before our next panel. Um, for those of you that are here in the house, we've got coffee and snacks outside. Um, we've really purposely scheduled longer breaks to have a good dialogue conversation, kind of take what started here on the stage and go out into the hallway. Um, for our live stream audience, you'll get the first demo of our newly launched FRED MCP connector, which allows people to seamlessly connect FRED to AI. And that demo should start at about 10:40 a.m. Central time. Uh, we'll reconvene here at 11:00 a.m. Central time for our next panel on trust and data. So, thank you very much.

Hi, I'm Brian Quistorf, a data scientist on the FRED team. We always like to meet users where they are, and today that's generative AI. I'd like to show you how we are unlocking the power of AI with our new MCP server and related tools. Let's check it out. Imagine that you are a student and you want to get a piece of economic information, something along the lines of what was the latest value for the unemployment rate. This is a standard question you might ask. This is a popular piece of economic information. So you might use your generative AI provider and ask a question. So here it responds with the latest unemployment rate for August 2026 was 4.1%. This is a seasonally adjusted U3 unemployment rate sourced from the BLS. What you may not have noticed is this information came directly from FRED. How does it work? Why don't we take a look? Introducing FredMCP. It's an add-in that gives large language models the ability to do some action. This action is getting economic information. I think most people are familiar with FRED. FRED is Federal Reserve Economic Data. It's a large repository of 800,000 economic time series hosted by the Federal Reserve Bank of St. Louis. We have data from 100 sources, something like 300 releases free of charge to the public that folks can come in and reference if they have some sort of economics data related questions. We've been on the web since 1991. It's awesome. fred.stlouisfed.org. Check it out. The MCP server is a way for you to get this in your generative AI provider. And what they do is they connect to our MCP server which is at mcp.stlouisfed.org and get economic data natively. And so the way that this works is there are five tools that we expose. One is search series. So imagine that you have a question. You'd like to find some piece of economic information. You should probably know what we have available. Search series does this via keywords. So if you were to search unemployment, it would return all the FRED series that have unemployment. The second tool is get series metadata. So metadata, if you don't know, is the information around our economic series. So the title, the notes, the source, all of that fun stuff. We expose a tool that allows an LLM to search over our metadata and get information about our specific series. Now, that's all great, but everybody typically wants the actual underlying observations, the value for unemployment, the value for CPI as of a given date. And that's why we expose tool three, which is get series observations. This tool basically allows an LLM to retrieve raw observations from FRED. We now have two other tools that are a little bit auxiliary but really help if you have data heavy loads or you really want to visualize a series. One is download series link. Basically this provides a download link so that if you have a question and you want to get the full series of unemployment but you don't want it running through the model context, you can actually just get a download link. So you can download a CSV with the same information you would have gotten in get series observations. And then finally we have graph series. This is just a tool that returns a FRED graph, a native FRED graph, the ones that you know and love. And that's it. Now what's interesting about MCP is you don't actually see these tools. It's great that you know them. I've now told you them. But what you ask is natural language questions to the actual underlying LLM. It decides which course of action. It decides which one of these tools to use and it responds accordingly with the information it gets back. So this is really cool. This is how these LLMs can now connect to FRED directly. So why don't we go through and see each one of these tools in action. Now imagine that you are a user and you would like to know what series are available over the employment situation. So maybe you just have a generic question. How's the labor market doing? Your large language model will look over the tool set that we expose and call a tool here. It's going to be the search series tool and it is going to return all the relevant series. I think it'll probably return about 20 series that are relevant to this search that you would like to do. So here this is, you know, a pretty standard fare when it comes to reviewing the employment situation. You have the unemployment rate, you know, payroll employment. I won't go through all of these, but it provides a smattering of what you might be interested in if you want to know, you know, what's going on in labor markets. There are a bunch of others, but this is great, but typically you want more than that, right? You want some information. And so why don't we ask, can you tell me more about the unemployment rate? We'll say, you know, what is the metadata? And it'll go through, it'll make another tool call and it'll respond with the information that is required to kind of understand what the series you're looking at is. And so it'll return the series ID in FRED. Every series has a unique ID here. It's UNRATE. Um, the title is the unemployment rate, where we're getting it. We're a data aggregator. The US Bureau of Labor Statistics, what release it's in. Um, you know, its aggregation, the units, and all that stuff. I won't go through this, but if you want to make sense of a particular series, it's a good place to start is the metadata. But this probably isn't what you're interested in. You probably want the latest observations. So, can you get me the last year of data? So, can you just return like what happened with the unemployment rate over the previous 12 months? So, it'll do the exact same thing. It'll call a tool. It'll go out directly to FRED, get the actual 12 observations that we're interested in, and return them to the LLM for presentation. So here you can see we have 12 months of unemployment data. Um, gives a little bit of context and you can actually see it on FRED if you're interested. We can actually go out to FRED, see the actual underlying series on FRED that you're interested in and do analysis from there. Now say you'd like to actually download this information. Then you provide a download link. And so this tool is actually usually meant for more computationally heavy workloads. You can imagine if you're using, you know, code assists to do economic analysis, you'd want to be downloading a lot of information from FRED. This just facilitates that. And so we can download — I won't go in and show you the actual CSV file, but we can download the entire history of unemployment natively using these tools. And then finally, most people like a visual view of economic information. FRED is very time series heavy. So we actually have the ability to make a native FRED graph. This is the graph that you know and love. So, can you graph the series? It'll think for a second. It'll spin. It'll call a tool and it will respond with a FRED graph. Here's the — it's still rendering. This is beautiful. You can actually click on the link, go directly to the page that created this. This tool is actually a little bit more comprehensive than just individual graphs. You can compare multiple series in one graph and do analysis that way. Now, the cool thing about the MCP server is if you notice, we never made any explicit reference to any of those underlying tools. You just ask it questions, natural language questions, and it responds using FRED as the backend information, this source of truth. You have data provenance. You can trace everything through. You can go back and see exactly where this information came from. And this MCP server actually works in any provider that exposes MCP as a layer. So you could use this, you could facilitate your own agents that do this. You can use ChatGPT or Claude or any of the other large providers, Gemini, to connect with FRED directly. This is an attempt for us to meet users where they're at. That's it. That's how we're bringing the power of FRED to your favorite LLM. Scan the QR code on your screen to stay up to date on how we're bringing FRED to your preferred platform. Thanks and enjoy FREDCon.

Let's see. All right. Welcome back everyone. Um, I'm pleased to introduce our next panel on trust in data. We'll learn from experts about the importance of trusting the accuracy, transparency, timeliness, and provenance of data. Our moderator is Erica Groshen. Dr. Groshen is a senior economics adviser at the Cornell University School of Industrial and Labor Relations and a research fellow at the Upjohn Institute for Employment Research. From 2013 to 2017, she served as the 14th Commissioner of the US Bureau of Labor Statistics. Prior to serving as commissioner, she was vice president in the research and statistics group of the Federal Reserve Bank of New York. Her research has centered on jobless recoveries, wage rigidity and dispersion, and the role of employers in the labor market. Please join me in welcoming our panel.

Well, it's a pleasure to be here. Thank you for having me and I've learned a lot from the two sessions already and expect to learn a lot more going forward. Um, let me introduce our panelists. I'm going to do that then I'm going to preface our conversation and we'll get into the questions. So, um, Katherine Elleroth is an assistant professor at Colby College. She's a macroeconomist who focuses on labor market dynamics. She co-creates and maintains the Elleroth and Misho quits and layoffs to non-employment release in FRED, which provides timely indicators on quits, layoffs, and subsequent labor force exits. As an expert passionate about building trust in labor market data, especially when information is incomplete, she's going to share practical ways to make data more credible. Uh, Dolun Batold is senior economist at Visa. He leads research on consumer spending and macroeconomic trends and is the architect of Visa's spending momentum index, which measures consumer activity. He brings experience from Visa, from the Minneapolis Fed, from the IMF, and the World Bank, where he combines real-time payments data with economic analysis to help businesses and policy makers better understand shifts in human consumer behavior. I'm going to have to slow down. So, he's going to talk a lot about how cleaning and analyzing data at the source boosts trust in what the numbers actually show. Aiden Smith has been with the Census Bureau since 2004 and currently serves as acting chief of the economic indicators division. In this role, he oversees 12 of the 13 principal federal economic indicators produced by the bureau. As an expert in data standards and governance, he's going to be talking about how strong policies and quality controls help federal agencies earn public trust. Chris Glenn is director of data science at Indeed Hiring Lab, where he leads a team of data scientists and engineers that transforms Indeed's first-party jobs data into a global labor market measurement platform. He'll discuss the challenges and the opportunities of building trust with data from technology platforms. So, we have some great people here to talk about trust. I'm going to start with some cool quotes that I found. It was only going to be one, but they each good in a different way, so I'll just read them. Um, the first one is quite short, but kind of cool. Trust starts with truth and ends with truth. Thought that was a good one. Uh, this one, this is the one your parents often said to you in some form or another. Trust is the easiest thing in the world to lose and the hardest thing in the world to get back. And then this is a new one I just heard today. Uh, this has a more techy flavor to it. Trust is a confident relationship with the unknown. So, let me start with a few things to help frame our discussion. Um, when I was at BLS, one of the many, many, many things I learned was the importance of trust in federal statistics in particular. And how that underlies almost every conversation that I had while I was there. It just came up over and over again. And I think the reason it's so fundamental is because it matters for use of the data. The whole purpose of data really — data nerds, as data nerds, we sometimes forget this because we just love the data. But why do we love it so much? It's because it allows people to make better decisions. And why would you base any kind of an important decision on data you didn't trust? So when trust goes down, data use goes down. The other part is also true, that every time we collect data, we are asking people to either reveal things about themselves or to do some work for us, right? They're taking on a burden. And why would they do that if it's not for the value of the data that's being created from it, from the data series, from the analytics that will go into it? When people lose trust in data, then the quality of the data goes down and its use goes down. So it dissolves the value of the data and there is evidence from that. It's not easy to measure, but I'll call your attention to a paper, an NBER paper that came out this April that I was lucky enough to help co-author, where we measured the decline in the cost to the country of the degradation in trust in BLS data after the firing of the BLS commissioner on August 1st, 2025. And the key thing there was that the accusations launched as a justification for that firing were that the data were not trustworthy, and it was groundless. No evidence has ever been presented to support that claim. But we saw an increase in the economic policy uncertainty index. That's associated with a loss to the US economy of $20 billion, which is more than 25 times the entire annual budget of the BLS. So is there an ROI on trust? Yeah, this looks like it's pretty significant. Okay. So as framing for the importance of trust there you have that. And the other thing I want to just mention is if you ever want to know how the federal statistical system is in some ways built around trying to ensure that that trust is maintained, you can go to the Principles and Practices for a Federal Statistical Agency, which is actually a wonderful short book put out by the Committee on National Statistics, which is part of the National Academy of Sciences, Engineering, and Medicine. Anyway, that's available. A new one just came out about a year and a half ago and you can see all of the different ways that the agencies operate in order to maintain that trust. So now let me turn things over to the panelists to start talking about this in more detail. Um, I'm going to start with a very basic question. Um, in your work, I'm going to start with asking Katherine first. How do you define trust in the data and why is it increasingly important? And then along with it, what kind of core qualities or practical steps do you think are important for maintaining trust in data? Katherine?

Yeah, thank you. So that is actually a really personal question to me. Um, you know, I've used FRED many years as a consumer, you know, for like personal curiosity, research, teaching, and I just trust it. Um, but turns out when you're on the other side and you're the provider, that's a question you face kind of daily is how do I ensure that people can trust my data? How can I make sure that the data we provide is credible, in particular coming from — so me and my co-author are essentially a two-person team that doesn't maybe have the same institutional reputation that maybe some of the other folks in the room enjoy, right? Um, and so some of the core qualities that we thought about relate to trust are, as Keith said in the introduction, provenance, transparency, um, timeliness, accuracy. Um, so how do I think they relate to trust? Um, so in my opinion, when we came up with the data, um, and then eventually provided them to FRED, I think provenance, transparency are really the non-negotiables. I think that is kind of the necessity. Um, provenance. You know, my analogy when I think about it is chain of custody. You need to have some process that you follow and can be trusted and is transparent. Um, but then let's be honest, if we look at FRED and we download data, how many of us are actually going to read the documentation or the fine print, right? I didn't. I just trusted it because it was the reputation that came with FRED, right? So, one of the questions we wrestled with is, okay, how do we even get to this point that we're reputational enough? Um, the way we approached it is, okay, let's just, you know, we can make a big documentation and then we're going to publish this for our data users in like bite-sized chunks where we talk about different things and try to keep it timely. Um, the other two qualities that relate to trust for me are the timeliness and the accuracy angle. And I talked with some of the panelists that there's kind of this obvious tension or tradeoff, right? If you ever want your data to be perfectly accurate — whether this is even possible — but you might have to really give up time. Um, when we thought about timeliness, I think we didn't think too much about it in the sense of speed, but in consistency and dependability, right? When I, as a data user or consumer, you know, rely on certain data series for policy or for business insights, it's the idea to trust that the data is going to be there when we expect it. Um, and the last quality I want to briefly talk about is the one that for me at the beginning was like, okay, that's the obvious, accuracy. How can I ever trust in data if it's inaccurate? Um, and then I'd be curious to hear maybe some of the other people here on the panel because then we wrestled with how do you measure accuracy, right? And there's clearly the technical component. You follow specific steps. We relied heavily on institutional knowledge about seasonal adjustment and in the end what we landed on is economic coherence. So is our data telling a similar story that we see with other data? Um, but yeah, so this is how we in the end decided these are kind of our core pillars to create trust. But again, as I said, I'd be curious, you know, we're still wrestling with some of those.

Oh yeah. So I apologize. I also have to start with a disclaimer that, you know, these are my views and don't necessarily reflect the views of Visa or its management. So any errors or omissions you hear are entirely the fault of my AI agent. All right. So on a more serious note, um, for Visa, of course, the same basic principles would apply to maintaining credibility in something like our spending momentum index as they would to any public agency. It's funny how it actually converts to that point even though we didn't start out trying to build credibility like a government agency would. And but for a global company like Visa, I would say three in particular are worth mentioning because they matter much more for a private company like ours. So one, how we handle the data, and then two, how we measure it, how consistently we measure it, and three, how adaptable it is, how flexible it is in reflecting evolving trends in our fast-moving industry. So let's address those one at a time. First and foremost, I would say responsible stewardship is really a critical foundation cornerstone of building trust because I think we all would agree here that the integrity of the organization that produces the data directly reflects on how much trust users of the data place in that output. And we have a leg up in that regard at Visa. I mean, I don't know if you know, but Visa consistently gets rated as one of the most ethical companies in the world. And I doubt that we set out to become that. It's just trust is core to our brand and our business. Without trust, our business doesn't exist. But it's not enough. I think for us to build credibility, we had to do actually two things. One of the unique challenges that we have is that we operate in over 190 markets around the world. And you can imagine the intersection of all the regulatory landscape that we have to abide by. So in practice what it means is that when it comes to responsible use of our transaction data we have to take the most conservative possible approach. And to give you a concrete example, internally we have a firewall and none of us in our economics team have access to any personally identifiable information. And you know, all our analysis including our spending momentum index starts from a data lake that's fully depersonalized and fully encrypted in a one-way hash. And that's critical. And the other part of it is actually independence. Funny enough, you know, we didn't start out to be that way, but it just becomes logistically really easy to get the legal approval and make sure that we can continue publishing on a monthly basis if our team is independent. So our team, the business economics group at Visa, operates like this independent think tank within the company. We don't have sales targets and we have this unique privilege of being able to publish our research and our spending momentum index without regard to what the current official position might be on any given issue by the company or our investor relations. And that goes a long way in building credibility with our clients and our partners when they know that what we're putting out there is our economy, we're putting our best foot forward to try to produce an accurate indicator as opposed to one that fits whatever the commercial interest of the company. The second part of building credibility, especially for a private company, is actually consistency. I think I would agree with Katherine here. Um, and consistency means trust means predictability. It actually costs money for our clients to take an indicator, build it into their models, build it into your data pipelines, and for them to justify that kind of investment to start trusting any indicator you're publishing. They have to trust that it will be there a year from now or two years from now, that it won't go behind a paywall or it won't, you know, change, that it will continue measuring the same thing in a transparent manner. So the transparency of the methodology is again very critical here. Um, and we've actually purposefully chosen when we were developing this nine or so years ago to avoid modeling because of this. We could have built a big AI model or machine learning model. We had all the skills and tools in place but we purposely made it very transparent and easy to understand. Why? Because, you know, it's important that when I'm showing the strength of consumer spending, our measure of it, it measures the same thing whether we're talking about households here in the St. Louis area or in Tokyo 14 time zones away across the world. So that I would say is the second most critical pillar. And the third one is actually adaptability. Um, in nearly 12 years that I've been at Visa, I don't feel like I was at the same company. I think we went through at least three or four distinct waves of innovation that really transformed the payments industry. What it means is that not only does it change consumer behavior but also how payments data correlates with external benchmarks like retail sales or personal consumption expenditures. And again the simplicity and transparency of the methodology was critical in being able to incorporate those changes and continue producing the same metric that, you know, what we were producing in 2015 or even 2008 is the same as in 2026. So I guess to boil it all down, it all comes down to three things. You know, how responsibly you handle the data, how much confidence you have that your methodology is transparent and consistent. And third, how flexible it is, how capable it is with adopting and changing with times and with the innovations that are constantly transforming the way we live.

Thank you. Want to add anything before we go on?

I'll just add to the emphasis on consistency. You know, when you say you're going to publish, publish, right? It's like the same thing as building trust in a relationship between humans. Do what you say you're going to do. Admit when you've made a mistake and be honest about what you're trying to do with the data and what you're trying to measure. And I think big plus one to everything Dolun just said. We have a similar experience with our data and letting it try to speak for itself. You know, we could get very fancy with modeling or machine learning, but really for us in a sense, the power of the data is giving it voice and letting it speak for itself and using some of the more sophisticated modeling on the back end for diagnostics and observability and trying to maintain quality in that way. So, I think there's a lot of overlap there with the Visa case.

I'll just add briefly. I was thinking about another angle on this is sort of the self-reinforcing aspect of trust. Um, and just very briefly sort of thinking through, you know, if you're going out with data that is not high quality or reliable and your data users get burned more than once, that, you know, there's going to be that consequence. There's going to be that degradation in trust. And on the flip side, if you are producing high quality data, obviously we're going to promote why it should be trusted and we'll talk more about that. Um, but at the end of the day, the folks using this to make decisions, are they getting the positive outcomes they expect from it? Are they getting that reliable end result? Um, so I just think that's another aspect that just sort of naturally comes with producing the data.

And there's one more, benchmarking, right? Right. So Aiden's team produces a lot of federal data. We take our private sector data and make sure that that's clearly benchmarked to publicly available statistics to understand how it's tracking, where it's diverging, if it's diverging, and making sure we understand the sources of divergence so that they're explainable to us, but also to our users.

The answer to accuracy. I think how you all do accuracy and I think how we do accuracy as well. I think just to echo, right, we do the same thing. We're like using what already has the reputation as well.

More data we have out there, the more validation opportunities we have.

Right. Well, Aiden, want to spend a little time talking about the ways that standards, governance, and institutional reputation contribute to establishing and maintaining trust.

Sure. So I'll start with the waiver as well. Any opinions expressed are my own and not those of the Census Bureau. However, um, yeah, I was thinking about this and as a bureaucrat, I love governance and standards and policy and that's part of my day-to-day. Uh, but I was trying to think about how to make it interesting, right? Reciting Statistical Policy Directive 1 through 4 is probably not what you guys are here for, but you should know that they exist. Oh, yeah. Yes. Um, so just trying to put a little flavor around it. I was thinking back, I actually had a friend recently visit me. Um, I live in DC and he lives in Ohio, but we go way back. And a good friend of mine, he came out, visited. We were up late one night, you know, had some drinks and we're having a great time talking about, you know, free will and all the big questions in the world. So at one point he knows what I do for a living. And so he did ask, he sort of explained he had concerns about a very high-profile economic indicator, not one of mine, thankfully. Um, so I sort of let him go and it became clear he had sort of fallen down some rabbit holes and he had some ideas about the way the data were produced. And at the end of that conversation, you know, I sort of asked what he was asking for and what he asked me was, why should I trust that estimate when it comes out? Uh, so I started, you know, thinking through how I engage with the work every day. So I asked him, I said, well, you know, have you gone to their site? Have you looked at their methodology documentation? Do you know how it's produced? Um, I said, have you looked at measures of error or other quality measures? What's the response rate like for this program? So we talked through a lot of that and the answer to all these questions was no. We're still friends, don't worry. Uh, and so I just, you know, I took the opportunity to sort of say, well, the reason I know these things exist for this indicator despite I don't work on it myself is because this framework exists. We have Office of Management and Budget which has established Statistical Policy Directives 1, 2, 3, and 4. They give us a framework to operate within for any of our federal statistical agencies to operate in. Um, and they guide agency execution of the delivery of that data. And along with that come all of those things I mentioned. So I knew these existed. Um, I gave him a quick rundown of each of those directives and kind of what they aim to address. Um, you know, fairly high level, right? Your first directive is really about sort of the affirmation of our responsibility as an agency to deliver high-quality data without undue influence from outside. Um, and it sets that groundwork for all of those ones that follow. Uh, we covered Stat Policy 2. Uh, he was still engaged. I got him another beer. It was great. Uh, so that really is where rubber hits the road. We've got quality standards and sort of guidelines for how to execute a statistical survey program from start to finish. Um, 3 is my bread and butter. Uh, covers our principal federal economic indicators. It's a lot of control around how we produce the data, how we evaluate it on an ongoing basis to make sure it remains relevant and of high quality. Uh, and then a lot of controls around release, right? We have to set those days a year in advance for every release that we're going to do that's a PFI. Um, and who has access to that data, right? Ensuring equitable access is a big part of that. And then a clear separation — as part of this is really getting into 3 and 4 — but a clear separation of the release of the data itself and political or policy interpretation. Um, those two things are very separate. Uh, our team, you know, these are our guiding principles as we put out our 12 indicators, you know, month after month. So, you know, my friend was still with me in the conversation. I didn't go too far into the weeds there. Um, but I thought what was interesting, and again you guys can do a quick AI summary of Statistical Policy 1 through 4. You'll get talking points. I tried this last night just to see if I was hitting the mark. Uh, it actually was a pretty good summary that came out. Um, I wasn't way off base at least. Um, so, um, but I think what's actually interesting is what does that look like in practice, right? What does it look like when an agency executes on this framework? So I actually, you know, again was chatting with my friend. I was like, let me tell you, I don't know the specifics of how that agency executes, uh, but I can tell you how I do for one of our indicators. So, I kind of walked them through, you know, a day in the life, a month in the life, if you will, of how we produce one of these things and how that framework informs all of the building blocks that we go through from collecting the data, protecting the data, how we prepare documentation, how we disseminate. So, one of the things I did want to highlight was that documentation piece, right? This goes back to I said, did you read their methodology? Um, and I said, I know they have it there because that is part of our guiding principle. You have to explain how you created that data. Um, I sort of shared with him some of the methodology documentation that we had, but even more importantly, the Census Bureau itself has quality standard guidelines. So these flow from OMB sort of saying you must create these standards. We'll let you do them how you will. Um, but I was revisiting those. It's been a while since I've looked at them and I think there's a lot of interesting pieces that pop up in there. One is, you know, obviously you have to convey uncertainty in your data. This is like critical. Um, when we put out an indicator and explain sort of the month-to-month trend to say, um, I might say such and such is up 5%, I immediately have to put that measure of error right after that. Um, that's critical to convey. If it's not statistically significant, then so be it, right? Um, it also controls a lot of the ways that we want to have comparative statements. Um, so this was something that came up where we actually wanted to add an additional few sentences to one of our press releases talking about — this was I think for new home sales. We wanted to add something on average price. Um, so they drafted something up about the current month average home price. This was great, but as a time series focused individual, I said, I don't want to — you know, yes, it's great to know the level, I want to know what it was last month too. So, our methodologist came in very quickly and said, uh, hold on. You can't put those statements back-to-back because they're not statistically significant or we haven't tested for that yet. Um, I said, well, I'm not saying month-to-month change. You're right. I, you know, can't I just — I'm going to say January was this and December was that? And they said, nope, can't do that. And they pointed me to the quality standards. So this was a good reminder for me about that core principle of yes, I could argue that well, I'm just stating one and then the other. But our standards are very specific about the implied comparison and if it's not been tested for significance, then we're not going to move forward with it. Um, I don't know how I'm doing on time, but I feel like I could go on forever about this. There's so many little individual pieces. Um, but we can maybe open it up a little bit more. Well, um, I'll...

Let me tell you my version of that story of this last one. So, when I got to BLS, I was still learning what the job was. The first time the jobs draft of the narrative around the jobs numbers came to me, a day before the release, I was asked to sign off on the language and I thought, 'Oh, you know, this is so cool, right?' But I said, 'Oh, I'm going to liven it up a little bit.' You know, so this is how I can because I had no access to the micro data or the programming or any of that. I only saw the numbers once they were final and the way it always is. So I looked at it and so I changed a few words there to liven it up. And then they came back to me after they saw it and they said, you know, 'Liveliness is not a goal here. We have words that we use to mean very specific things and it's not our job to interpret or be lively. It's our job to count the beans and report the counting of the beans.' And the longer I was there, the more I appreciated that those values, right? It's journalists and policy makers and financial markets' job to interpret the numbers, but the statistical agencies are not in that business. And so I think it helps a lot, it's part of having the credibility because it really clarifies the distinction between the two.

Anyway, sorry, just one more thing to really quickly add. I think one last point was just to convey, I think this infrastructure, the framework exists. Clearly my friend did not know this was out there and or just didn't have the initiative to go find it. I will give a shout out to Fred. One of the things I love is you guys bring a lot of this together and do make it visible. Methodology, all of the stuff behind the scenes. So I think the room for improvement is promotion, right? Is getting that story out in front. We've established, I think, a reputation, you know, at the Census Bureau of being trusted. We heard that from multiple folks here today, right? The gold standard. Go there first. That's great, but how do you maintain it? I think you need to get out in front of these things. You need to promote that. You need to lean in on those guiding principles and tell your friends as often as possible. Yes. You're all trusted voices. Use them.

This is also where I think public and private sector data can be complementary in a sense, and building trust across both is the co-movement together of series that are telling a similar story and reporting on the same markets. And we view, you know, federal data as a critical part of our trust agenda and building trust in the data that we have is how does it compare to federal data and I think it can flow backwards as well, and when there are interruptions. Yes.

And maybe to be the counter example to your friend, but our labor market indicators use micro Current Population Survey data, right? So I like completely accepted that the data is trusted. And so I feel like, you know, maybe I should have read a little bit more about those standards. But just as a counter example, I think there's a lot of data out there, right, that you might not publish or aggregate in that way but a lot of serious work on FRED will implicitly rely on that because they relate to, you know, the framework that you have but others will adopt it through, you know, using parts of the data.

I would double click on this interdependency between public data and the private data because the publicly available data is very critical in anchoring our methodology, making sure that we're producing something that's valuable. And there's a funny anecdote. I was talking about our spending momentum index in Canada and talking to folks from Statistics Canada about how they produce CPI statistics. And I'm giving them compliments, 'Oh, you know, we rely on your data to anchor our spending momentum index,' and he in turn turns to me and says, 'Oh, we in turn look at your data to calibrate our inflation estimates or something.' So it goes both ways.

Okay, well, Chris, why don't you tell us a little bit about how emerging technologies, including GenAI, are changing the way that we evaluate and build trust in data. What are the challenges and what are the opportunities?

I may add like kind of a second or third pillar. I think AI can also potentially help build trust in data. So me and my colleague are just a two-person team, you know, and we have very like we have day jobs. And I used, you know, I used AI to help me build a time series pipeline. And then I was like, hm, I wonder if I feed it my documentation, if I feed it my actual time series, if I feed it, you know, here's we're using underlying CPS micro data, how far would AI be able to kind of like back engineer what we do and see. And it, you know, it thought for a long time and we went back and forth, but it did a really good job. And to point out, it's like, oh, you know, I noticed you most likely used the X13-ARIMA seasonal adjustment and I noticed you don't have the noise so I assume you did something like this. So I think it can potentially be used also to generate trust. But I think we're still kind of in the early stages to figure out how.

I mean this is a slightly different angle on the conversation but not so much generative AI but I think it's interesting the cross-section of economic measurement with a new technology like AI or one of the cases that I came across recently was more computer vision but as an avenue to actually create higher quality data and again to go back and promote the use of that technology. And very quickly in this example, we had actually explored how is there a better way to measure housing starts on a monthly basis. We use field representatives currently as sort of the primary data collection method that comes along with cost, response burden, all sorts of things and data collection error as all of these programs will have. The idea was if you can get satellite imagery on the cadence and at the resolution that is required when you look at the excavation for a new home, it's very clear whether or not it's happening. However, when our field rep calls a builder and says, 'Hey, did you start construction?' who knows? I mean, hopefully they can get a hold of the builder. Hopefully, the builder replies. So anyway, I think that project was a very interesting sort of application of layering in a new technology like that to really just go straight to the higher quality data and again being able to communicate that to users. We haven't launched that yet, but prototypes.

Well, let me ask a somewhat related question. Do you think it would be helpful to trust and the quality of the data if we had more forums, institutions that were setting data standards for all kinds of data, particularly that companies collect? Right, so there are many companies that collect similar data. I'm a labor economist, that they all have data on their employees, right? What, right, their occupations, their pay, all of these things. And yet they keep them in different ways. But certainly medical records, transaction records, production records, all of those things. Is this something that would help to improve quality, reduce burden, improve trust because everybody would understand where the information came from?

Certainly, I think interoperability across multiple data sets could facilitate increased benchmarking and comparison. It's a really hard challenge to be able to incentivize individual entities to operate under the same standards. Can be anyways, but I think it could be helpful.

Yeah, I would say I think that you kind of stole kind of where I was thinking about this too. So it is a unique aspect of or comparative advantage that the federal statistical system has, again that overarching governance and I think that's where the private sector doesn't have that incentive, right, to build the taxonomy, you know, some sort of uniform taxonomy or these directives to control data quality. It would be great though. I think that the more that those two sides start to look similar, like you said, sort of that complementary relationship between the public and the private data. I mean we are increasingly relying on private data as an alternative source to traditional collection. So anywhere we make improvements on that is going to be great. I'll turn that question around actually and ask whether in the world of AI where agents are so good at reformatting and combining whether that's actually as big of an obstacle as it may have been 10 years ago.

But how much can we trust that? Well, I'll tell you about one thing I've been involved in. There is a group that is trying, that's actually supported by the US Chamber of Commerce Foundation, that is trying to devise standards for employer records and they've done a survey of employers, HR executives, asking them would they be interested in doing this knowing that their reporting burden would go way down, that lock-in with payroll agent companies would go down, and that the privacy protections would be built in from the beginning. And two-thirds of them said yes and their primary motivation was actually on the privacy, that this would give them a safe harbor. So personally part of the reason I asked that is that I think that this is part of the future of federal statistics is being more involved in the creation of the data sets because it helps you to get some of the advantages of surveys but using the framework of private sector collection.

Maybe if I can add to that as someone who deals, you know, with survey data is like, you know, I can convince all of you that the way I deal with the survey data is, you know, enough to trust it but in the end, and I know you probably know this better, Aiden, response rates have been declining drastically in particular in the pandemic, right? So can I trust the answers in the survey, right? I mean we all know we can reweight answers and we can make it representative but, you know, in the end this is still someone telling me whether they worked or not in the last week and there, you know, multiple incentives to answer this question. Or maybe from a different standpoint, right? Like having, you know, relying on like job postings from Indeed versus asking companies about it, you know, adds some credibility in that you remove that layer of what do they say, but it's kind of you can observe what they did.

Well we've got 10 minutes left. I want to open this up to questions from the audience about trust and all of the various elements that we've been hearing about. Questions? Well, trust the data. Oh, I see two in the back. Let's see.

I had a question just about disruptions in data collection. Famously, in October 2025, we had the government shutdown. So Erica in the BLS that resulted in interruption of the jobs report. Aiden I'm sure and data collection in the field that was affected and Chris at Indeed I'm sure there was a ripple with, as you say, people in journalism, academics, policy makers seeking other sources. Could you just share any observations about that?