360º Fireside Chat with PagerDuty, Lisbon’s Newest Tech Employer

Hi everyone, good afternoon or evening. I'm Tim Armandpour, the Head of Engineering here at PagerDuty. Pretty much anything that a customer of ours ends up interacting with, using, leveraging, purchasing — that sits with my organization. Been here roughly about seven years now.

Yeah, it's been a wild ride for sure. PagerDuty started by three ex-Amazon engineers that truly believed that for anyone working in technology, there has to be a better way to quickly react when things go a little sideways. They felt the pain of wearing a physical pager — and I don't know how many people in the audience remember what a physical pager looks like — but they truly believed they could elevate the experience of any developer or anybody working in operations. They really saw this view of DevOps catching a wave, seemed like earlier than a lot of other folks. The company started nearly 13 years ago. The founders were told no many times before they finally got their first round of funding. Many people thought PagerDuty was just a fancy alarm clock, and we always like to say, well, look at us now. The journey has been just one of great growth, challenging growth constantly. As I mentioned, we just celebrated our 13th birthday, so we're actually sitting here as a bunch of teenagers still trying to figure it out. It's not easy, we need help — we need help from probably many of you to come aboard and help us figure out the next great thing. As you mentioned, Ron, we're definitely not a startup — we're nearly a thousand people globally. We have major offices in San Francisco, Toronto, Atlanta, London, Sydney, and now Lisbon. We're still in that fun growth phase, trying to figure it out every single day.

Sure. One of the main underlying premises around PagerDuty is this notion of wanting to help you do your job better. That's really kind of where a lot of our motivation begins and ends. It's about anybody who interacts with our experiences, our platform, our APIs — we want to help you figure something out quickly and be able to do your job better. Because one thing we all know is that there's a lot of noise out there — whether it's in email, Slack, Microsoft Teams, or whatever chat tool of choice. And if we're sitting on the dev and ops side, systems that are constantly sending signals to us, posting to logs, it doesn't matter — there's just a lot of noise. Where we started was this thesis that if we can help the one that needs to do a thing do it better, then we give ourselves a better chance of working better and better. Where this has shown up over the last 13 years, without really a lot to do with anything we've done intentionally, it's just some of the beauty of the platform and the flexibility and ease of use we've created — we see folks outside of developer operations or IT starting to use PagerDuty. That's been going on for many years. There's a handful of examples. One top of mind — I probably can't name the customer — but a very large brand that many of you interact with, they use us to help facilitate their human resources practices. For example, maybe someone does a bad thing that breaks company policy and the decision is made to unfortunately move the person out of the company immediately. Someone in HR will actually create an incident that kicks off a workflow across a wide variety of teams that have to work very closely together, where the number one thing they're all measured by is the amount of time it takes to get it done. Maybe it's a high-risk situation, and then you see facilities or help desk, IT, the managers, other people in HR, legal — all get pages and alerts from PagerDuty to bring them into a live incident in order to make sure that the termination or the series of actions get done very quickly. We haven't changed a thing in our platform to facilitate that — we have countless customers that use us that way. We have companies that use us in aspects of even parts of IoT in industrials. For example, General Motors — they have these big factories that fabricate things, and one of their divisions actually has us hooked up to monitors that measure vibration on certain machines. When that vibration goes off, they can't manufacture at the same rate, alerts go off to alert people to assess and understand and resolve the issue. There's a number of things all over the map that have started getting used by folks leveraging PagerDuty over the years.

Yeah. So first question — why Lisbon, why not somewhere else. A few reasons. We take a step back here at PagerDuty — we are a group of people that are very motivated to make really well-informed decisions. We think we're more right than we're wrong in that capacity, and this isn't just at Paula's and Tim's and Ron's level — this is at all levels. We actually like to collect some data, both the quantitative and the qualitative sides, just how we approach things. From a quantitative side, we look at many of the studies and information that's publicly available, and we ask — whether it's local recruiters, local real estate firms, other companies, colleagues — about data. Then we get to the qualitative side, which is we have a lot of customers in the region across Europe — I think close to 2,000 distinct customers. Europe is not necessarily new for us. The wonderful country of Portugal became more new as we went through our discovery and our journey there. But we were able to ask a lot of companies, a lot of colleagues across our networks about the pros and cons and what should we consider. A few things stood out about Lisbon. One is just the thriving market landscape, especially the corridor from Porto to Lisbon. The number of universities that continue to graduate fantastic people. There's something about when you have other big companies nearby — it's not by accident, so there's a natural draw. The other thing is we wanted to build almost like a community, an extension of the PagerDuty community. We talk a lot about community here at PagerDuty. We believe deeply that part of why we want to exist is to help put a positive dent in the universe, we want to make the world a better place in our own special ways. And part of that is by creating a thriving and vibrant community — both with our customers, with our end users, but then also our people. We believe our people have a responsibility to keep carrying the torch on making community better. Lisbon and Portugal really stood out from that same sense of how important community is, how important not just the people you work with but the extension of your life. That really stood out compared to a lot of other locations we were looking at. Plus, from what I hear, Lisbon is a very desirable place to visit — we have no shortage of people that want to come back and forth. I'll be out there in June, a very quick stop on a family vacation through Europe. All those reasons put together is why Lisbon stood out for us. Now, second part of your question — remind me again what was the second question?

For many years we've had this notion, especially within PagerDuty but even before COVID, around being able to work well in a distributed manner — especially between product teams and engineering teams, we were very focused on making sure you didn't have to be physically co-located. Whether it's GitHub, Slack, name your video tool of choice — all of that was a norm within us before the pandemic. We were primarily split between East Coast and West Coast time zones in North America. Now that has grown, especially with Lisbon coming online. We have worked with other partners that help with some development and product work in other countries across the Atlantic, so it's not new. Part of what we're looking at is how we get more time zone savvy. We already have some teams that are more West Coast based, some teams that are more East Coast time zone based. Our main goal is — and Ron, for you to help with this — to ensure our teams are structured in a way where you don't have over-reliance or dependence on other teams in other parts of PagerDuty. This doesn't mean we don't want to work well together, but we want to create independent ownership, independent accountability, so the groups in Lisbon can absolutely thrive. The fun part about being in Lisbon is the very close proximity to our London office. In the UK alone, we've got something like six or seven hundred distinct customers. By virtue of that, we're big believers — I know I'm a big believer, and Paula as well — in getting close to the customers. It helps us make well-informed decisions as builders. Some of the biggest decisions we make are how we approach a problem, how much time we spend, do we shortcut it, do we invest for the long term — that input can very much come from customers. Being in Lisbon now, we have product managers, designers, and developers that are going to be very close to our customers to really understand what are the unique problems in the region that we don't readily see. We will not be doing things where everyone has to be on midnight calls or 6 a.m. calls — that's what we're intentionally looking to avoid. Many of us have been around long enough to know how poorly that works for the people and for the delivery that happens through our teams.

Yeah, and I think my background is software engineering, so as a software engineer, that's really a huge motivation — the fact that you need to handle these peaks, these loads, guaranteeing that everything keeps working. I think it's one of the best things an engineer could ask for, to try to solve this type of problems at a large scale. I want to just build on one aspect of what Paula was talking about because I think it's an important thing about PagerDuty. As she was talking about when part of the internet is broken, the expectation is that PagerDuty is still online. When you think about that from a software or system design perspective, we are constantly looking to architect for failure of some kind. The real secret sauce of PagerDuty comes not just in technology choices — I'm a big believer that technology is like part of the tool in the toolbox — it comes down to the people. How well-equipped are the people to be forward-looking on some of these things, but then also when things go sideways, how well are we equipped to react? That's some of the other things in terms of the engineering culture at PagerDuty that we believe is worth investing in. For example, if you want to go Google 'Failure Friday PagerDuty,' it's a blog series and some talks we put out years ago about our approach to chaos engineering, how we practice that, how we expect our teams to actually learn how to manage when something goes sideways — were you prepared, was the system prepared? There's a lot of practices to hone in on. We do believe that every individual here, especially in the world of engineering at PagerDuty, you're expected to approach things like a mechanic, not just a passenger in the car. We want you to understand how things work, and if you don't know, we'll help you. So really building that bridge between traditional development and traditional operations and removing all the fuzziness and the gray area — it's a big part of our culture and our ethos here at PagerDuty. I just want to add that the process and practice and how we approach things is as important as our technology stack at the end of the day.

Yeah. With hack week — two times a year, one week in the spring, one week in the fall — anybody in the company can take time off the clock and work on whatever they want. What we naturally start to see is many cross-functional teams across the company — not just engineering, not just product and engineering, but we have sales involved, solution consultants, sales operations, marketing, HR, etc. — come in and we build solutions to problems that are very near and dear to people's hearts. People then present these out and there's a whole panel of judges, like Paula mentioned, and we give out quirky prizes and all that fun stuff. But some of our most interesting and compelling contributions — not only to our customers but internally to ourselves — have come out of hack week. And that is very intentional and purposeful. We do approach many things with intent and purpose in mind, because we have this thing about us — given the way our customers depend on us and how we get integrated, PagerDuty's cost of being wrong is very high to someone on the other side, very high to the customer. It's an important distinction. We don't have the benefit of seasonality of traffic that you can predict like in e-commerce — I spent a long time in payments and PayPal where we knew in June we were starting to harden systems and buy hardware and get everything provisioned for the holiday rush, and then you get a break after January 7th. We don't have that benefit. We always have to be ready for anybody else's worst day, or for everybody to have a bad day. We approach technology and those decisions — I'm a big believer in pragmatic, practical solutions. The next shiny thing we can experiment with, but it probably won't be ready for our primary customer traffic anytime soon. But that's okay because we still need a healthy balance of learning new things and being forward-looking. When we say we're going to support 200 changes to the live environments a week, that'll vary depending on the team's needs. I'm a big believer in keep — I'm a fix-forward kind of guy, not necessarily roll back. I want to keep pushing on that front because I do believe if we can allow our developers and all of our teams to be able to do what they need to do as fast as they need to do it, or as careful as they need to do it — part of some of the underlying responsibilities, especially in Paula's teams, is to help these teams make sure they can do it safely and responsibly. So it's not by accident we can support 200 changes a day — it's actually very intentional. When we look at some of those classic metrics, like our change failure rate, it's like below 0.2 percent or something crazy like that. So we really honed in on some of these pieces. People ask me all the time, why are we still running around MySQL? Well, because it makes sense. There's some things you want to have control over, things that you actually do really well, versus taking the next shiny thing off the shelf. In some cases we want that next shiny thing because it's been tried and proven. But ultimately, a lot of things are driven from a strong sense of intent and purpose and what is that next outcome we're trying to make true.

Yeah, you know what Paula is saying — our three founders started this company in a unique mindset compared to how many other companies may have started in adjacent spaces. One of the big things was it was about the people. Our platform, our product architecture, almost the way we bring things to life, very much has the people in mind first versus the machines. It's a big distinction because we believed that we can elevate everyone's day to a new level. Imagine for developers what GitHub and even tools like Jira have done — made some of our jobs a little bit easier, a little bit better over time, really elevated the developer to this first-class citizen. We believe that that activity around support and resolving — and some terrible knowledge with firefighting so to speak — ought to be elevated in a similar manner because it's that important. And now we've evolved over time. There's a big people component to a lot of the things that we bring to life, how we approach things, how we orient. There's one value that is termed as 'bring yourself' and that one I think is super important because at the end of the day, from a community perspective, we believe every single person has to have a strong sense of belonging — every single day, every single interaction, every single moment. If you can't be your true self, that's a terrible way to spend the day. Secondly, from a company perspective, as we start continuing to build this company, expanding and coming into new cultures and new geographies, we know that we aren't going to be able to be successful as a company if it's all about the people — our customers, our end users, and our people, people, people. So that one stands out for me.

So I think it's a fair expectation to have hybrid teams across hubs, like what Paula was mentioning about working with each other. There's no better way to do it. For example, a part of Paula's organization, the SRE team, that helps us find better ways to keep running reliable software especially in the cloud — we have SRE team members across North America, we want SRE team members in Lisbon as well, and there's no doubt at some point those folks will work together. So there will be some aspect of that hybrid piece, with the empathy that we need to have the overlap. But the overlap shouldn't always be necessary. I think it's part of the key — we want people to be able to work fairly independently as best as possible as we continue to build up the teams locally.