Cofounder, Alation
High Velocity Development of Microservices • Venky Ganti & Rahul Lahiri • GOTO 2020
This presentation was recorded at GOTO Chicago 2020. #GOTOcon #GOTOchgo
http://gotochgo.com
Venky Ganti - Founder and CEO of Mesh Dynamics
Rahul Lahiri - Founder and Head of Product at Mesh Dynamics
ABSTRACT
Engineering teams spend significant effort on creating automated regression tests, and still cannot often go beyond “happy path” testing. Consumer technology companies therefore invested in testing in production. Unfortunately, these approaches do not apply for enterprise applications because a) their users cannot tolerate errors and b) they usually do not have as much user traffic.
In t...
About Venky Ganti
Venky Ganti, cofounder of Alation and founder and CEO of Mesh Dynamics, gave a presentation at GOTO Chicago 2020 alongside Rahul Lahiri on shifting testing earlier in the development lifecycle for microservices. Ganti described two trends driving engineering teams to take on additional testing responsibility: continuous testing driven by DevOps and the adoption of microservices architecture. He stated that the complexity of interactions among microservices increases the need for higher functional test coverage, and noted that engineering teams typically spend more than 25 percent of their effort on creating automated test suites. Ganti discussed limitations of existing approaches, saying that consumer tech companies' methods such as testing in production via canarying or traffic diff are "fairly late in the dev cycle" and may not converge quickly for enterprise applications with lower traffic. He presented Mesh Dynamics' approach, which he said leverages trace IDs injected at gateways and propagated downstream to simulate stateful services with high fidelity. Ganti stated that this enables teams to test in near-production environments early in development cycles, including within CI pipelines, allowing them to gain confidence and devote more effort to engineering new capabilities.
Transcript (27 segments)
V
Venky Ganti0:15
Hello, my name is Venky Ganti. Today, along with my co-founder Rahul Lahiri, I am going to discuss how we can shift testing and production left. In doing so, we enable engineering teams to prevent API regressions and detect them early in the development life cycle.
There are two big trends: continuous testing driven by DevOps and shift left, as well as the adoption of microservices architecture. These two trends are actually driving engineering teams to take on additional responsibility.
You take a typical engineering team. There are three test goals before they release a new piece of software or a new version of their software. The first goal is to prevent regressions to existing usage. The second goal is to test and debug new functionality and ideally also automate tests for that new functionality to prevent regressions in future. The third goal is infrastructure testing: performance, load testing, resilience of the infrastructure itself or the application, security testing, and so on. Because of the pressures of release deadlines, teams rarely get to spend a lot of time on this.
This adds up. The further they need to spend, it is hugely demanding on their time. Typically, more than 25% of all engineering effort in a team is spent on creating these automated test suites. Despite all this effort, the coverage of this regression test tends to be limited because of a variety of constraints, whether it's timeliness or the additional new capabilities they need to deliver for the business to function. All of this limits the amount of testing coverage they can get. The consequences: it slows engineering velocity, whether because of the effort they have to put in in the current release or because of bugs that arise in production.
Satisfaction or even churn. Because of the criticality of this regression testing issue and the amount of effort required, a common approach is that consumer tech companies such as Google, Facebook, and Twitter have come up with a variant which is often called testing in production. The key insight here is to leverage ongoing user activity to validate new versions of the software. In one variant, which is called canarying, a fraction of the traffic is diverted to the new version, whereas most of the traffic goes on to the production version. Then they observe top-line metrics.
Between a primary production version and the candidate build, the functional responses from the candidate build are then compared with what is coming out from the primary production version. This comparison has to be functionally possible. In that context, the candidate version has to have the exact snapshot of all the backend infrastructure that the primary is running on. Thus, they are able to compare for functional correctness. Both these approaches have the following problem: they are fairly late in the development cycle. The builds have to be released into production where the users can be exposed to that.
It takes a long time to converge. Because both the candidate and the primary versions, in both canarying and diff testing, have to run on exactly identical snapshots of the databases, the infrastructure tends to be pretty costly.
Now the question still stays: can we leverage user activity, especially past user activity, in the context of enterprise applications to shift this testing in production idea to the left? If you are able to do that, then there are significant benefits. The first one is that we are able to get 100% functional regression test coverage right in our CI pipeline.
State consistency between when we captured user activity and when we are testing. So what are the approaches for reconciling this state consistency? A common approach is to snapshot data just before capturing traffic. We snapshot the entire databases, all the databases that the application relies on, and then immediately start capturing the traffic for an uninterrupted period of time. When we have to test, we actually restore the data snapshots back to that particular snapshot and start replaying the traffic. Thus, we are able to compare what is happening with the test exactly with what used to happen in production. That's one way. The main problem here is that the infrastructure cost is still very high.
Fidelity, where the simulator is much lighter weight than all the production databases that you need to bring up. The question is: is this even possible, especially in our current scenarios where responses to identical requests can be non-equal? In the sense, if you take a stock ticker service, you ask for the same stock price, let's say Microsoft stock price, every few seconds you get a different response. In that context, how can we actually simulate the state with 100% fidelity? You have to really understand the context of the caller as well as the callee before you are able to simulate that.
Our insight here is that you can actually leverage tracing of the application to simulate.
In a production system, let's say, when we are capturing the usage, we inject a tracer ID at the gateway request, much like the Jaeger and Zipkin open tracing libraries, as well as APM products such as New Relic and AppDynamics. The same trace ID that is injected at the gateway request is propagated to the downstream services for that request, which spawns off requests to service B and service C and to the database. They all carry the same trace ID. Now, all of this usage data along with the tracing is stored in a usage database. Now when we have to test, we bring up the service A.
Simulators get this request instead of the actual services. They get the request parameters as well as the trace ID, and they can use both the request parameters and the trace ID to identify the exact response that they need to send back. So if this particular trace ID corresponds to the Microsoft stock price which receives 131.5 as a response, the trace ID will tell the simulators that for this particular trace ID, even though there are four different candidates that we can respond with, this is the precise category that we need to pick to simulate the context exactly. And that is what enables us to simulate the stateful components with 100% fidelity.
In your CI pipelines itself, so let me illustrate that with this once more. You have your application. We put in agents at various levels of the application, whether it is the gateway or between services or from the service to the database. We capture this API traffic along with the trace IDs in our usage database. And when you have a new set of versions of these services that you need to test, our usage drivers drive traffic to the gateway and re-inject the trace IDs that were introduced in production. And the simulators, all these agents have the ability to simulate the backend stateful services and databases. So consequently, your harness is super.
System under test, and compare with the responses that were observed in production. And you now have 100% regression test coverage with very, very little engineering effort in terms of test automation. Consequently, you get 100% regression coverage and you are able to prevent regressions or detect regressions fairly early in your development cycle. So the coverage is pretty high. The effort that you need to put in terms of test automation is much, much lower. Because of this, you would get high engineering velocity and high quality.
Now I am going to hand off to my co-founder Rahul Lahiri.
R
Rahul Lahiri12:11
Test data, you write assertions. And after you have tested them out, you will write the script to automate the test. With Mesh Dynamics, the test case creation, the test data creation, all of that is replaced by the usage capture, whether it is from a test setup or from the production setup. Assertions are automatically created and the test scripting is unnecessary because our replay driver takes care of replaying those requests. So for the demo, we will go through three steps. We will show how to create a test, how to run the test and review the results, and finally how to update the test via data updates.
For the demo, we use this application that we call Movie Book. Think of it as an online library system where you can look at all the items that you have.
Against the second version to identify what has changed. So first, we will start by creating some tests. This is our demo application. This shows all the items that we have. You can click on one of them to see the details, and you can click on this to rent an item. This is the Mesh Dynamics app. Here you are seeing the service graph of the Movie Book application. So when we capture the data, we will be capturing all of the interactions at each of these services, as well as all the interactions with the platforms like databases. So let's start the recording process.
Let's create a few test cases. Let's say, which is listing one of the items. So I will click on one of these items. Now let's create a third case and rent an item. Now I am going to click on one of the rent buttons. I am going to go through the rent process. Now let's go and stop that.
Now we can look at what we have captured. This is the capture we just did. So if I look at the details on that file, there are five different services from which we captured data. And once you pick one of the services, you can see all of the APIs for which we captured the data. So here we captured data for three of the APIs. We can look at one of the APIs. Here it shows you the structure of the API.
And we actually interacted with the app and those became our test cases. For the next step, let's run the test and review the results.
So now we will run the test using the data that we captured. The test is going against the second version of our application that we have for the demo. And here these three services are the system under test. We are mocking these neighboring services. So let's start the replay. During the test, our replay driver is picking up the requests from the golden and replaying them in the order in which they appear in the golden. The replay is now complete. Now let's review the results.
Version of the software. The left-hand side here shows what we got from the original version. The right-hand side shows what we got from the test version. This is just a standard GitHub-style diff. And for each of those diffs, you can have three things that you can do. You can either say there is a problem and you can go ahead and file a Jira bug with that issue, or you can change the assertion rules, or you can update the golden so that the next time you run the test, it uses the new data and it does not generate a diff. So let's say here the actor's last name area which appears in the original but does not appear in the new version of the software. Let's say that is a problem and you want to file a bug there.
Service and the API path, as well as the JSON path in which this problem was seen. And we also include the URL. If you click on the URL, you will see exactly what the reviewer saw when they filed the bug. So you can see the bug was filed against this missing item: actor's last name. Next, here the display actors data has changed from a first name last name format to a last name first name format, and that is the change you have made. So you want to update your data so that going forward you use this new format. So here I can go and say update the golden. What this will do is when we save the golden back, it will take all of the new occurrences of this display actors.
And maybe film counts is something where you don't really want to compare the values. So you can change the assertion to not look at the value and say just ignore the values. So the next time you run it, it won't generate an error. So now that we have made those changes, let's go ahead and save the golden. I'll give it a new name and I will save that. So that becomes a new golden that I can then run the next time. In a distributed services environment, it can get pretty hard to understand where did the differences occur. So we provide this tracing capability which shows you how that request executed across the different services.
And you can just click on the appropriate item and you can see what changed in the request or what changed in the response, which makes the debugging process very easy. So that is our demo of the Mesh Dynamics product that we are building to achieve this goal of being able to test in near production much earlier, not having to wait until the whole thing is ready to go into production, and also give you a very comprehensive test coverage without involving maximum amounts of engineering effort. Thank you.