Performance Regression Testing for React Native

Let's talk about developing React Native apps. Or, to be more precise, about developing performant React Native apps and the role of performance regression testing in this process.

React Native is an excellent framework for developing cross-platform mobile apps. It's fast; it's easy to use. However, certain things should be kept in mind while developing React Native apps. One of them is performance.

As React Native applications grow in size, they tend to slow down. React Native is a JavaScript framework, and JavaScript is a single-threaded language. While the UI is being rendered, the JavaScript thread is blocked, and no other JavaScript code can be executed.

Re-renders as a source of performance issues in React Native

Re-renders are one of the most common causes of performance problems in React Native apps. They can go unnoticed in React web apps; however, in React Native apps, an excessive amount of re-renders leads to serious performance issues and slow UI. On top of that, it's pretty hard to spot code changes that lead to re-renders just by reviewing the code.

When a component re-renders, a serialized message is sent through the bridge from the JavaScript thread to the native platform. The native platform unserializes the message and creates a new native view, which is then rendered on the screen. If the render result is the same as the previous result, it may not result in native view changes; however, the time it takes to re-render a portion of the element tree might block the JS thread.

If the JavaScript thread is unresponsive for a frame, it will be considered a dropped frame. Any animations controlled by JS would appear to freeze during that time; and if it lasts longer than 100ms, the user will surely feel it. Not only animations are impacted; response to touches can be affected as well. In fact, there are many performance issues that re-renders can cause. You can read about possible causes of such problems in the official docs guide on performance.

When does performance regression happen in development lifecycle?

Development lifecycle would make a good topic for a separate blog post, but let me be brief here. In a nutshell, a simplified version of what happens during the development process is as follows:

1. New feature is introduced
2. Tests are added
3. QA goes on
4. Release takes place

Looks good, right? Especially because in reality, tests are sometimes skipped, or the QA process is rushed for the sake of releasing to prod. Let's assume a best-case scenario where test coverage is maintained in the high percentile, coverage reports are monitored, integration tests are automated, regression tests are executed, etc.

Even in this perfect setting, there might be bugs that escape the QA engineer’s notice. But more importantly, performance problems that fail to be recognized by any test or QA process can occur.

Adding additional re-render can sound like no big deal, but performance regression is like a snowball. As more features are developed, it gets more excruciating to pinpoint or fix performance issues. And once they become apparent to end users, it usually requires a lot of refactoring and a dedicated team working for hours to find the problem and carry out improvements. These, in turn, can introduce breaking changes to the rest of the app and lead to months of further development work.

How can performance regression testing keep your app fast?

So, what can we do about this snowball of performance regression growing over time? How can we know that a new feature reduces performance? How can we catch the unnecessary re-renders and get actual performance metrics without the overhead of having a dedicated team working on performance improvement?

Of course, there is no silver bullet to finding ALL performance problems, especially the ones that originate on the native side. Still, we can address the ones that arise from incorrect usage of React or React Native, like long lists, not using memoization, unnecessary re-renders, etc.

It's not enough to only identify these performance issues, though; we also need to introduce some measuring tools into the development workflow and Continuous Integration to automate performance regression tests. While doing it manually on each PR review is possible, it takes precious development time, introduces complexity to the release process, and increases the chance of performance problems slipping through the PR review process and ending up in prod.

Performance regression testing made effortless with Reassure

Given that there hasn't been any tool for automating performance regression testing so far, we have decided to create one: meet Reassure.

Of course, introducing a new tool means introducing a new workflow. This can lead to lower adoption, as developers need to get familiar with it in the first place. To prevent that from happening, we've based Reassure on React Native Testing Library API as something React Native developers should be familiar with.

Reassure can be easily integrated with your current React Native Testing Library setup. Writing performance tests is pretty close to writing regular tests, as you’ll see in examples later in this blog post.

How to add Reassure to your project with yarn add --dev reassure or npm install --save-dev reassure

After you've added <rte-code>reassure<rte-code>, you can start writing performance regression tests by adding <rte-code>.perf-test.tsx<rte-code> extension.

Since there is a similarity between your regular tests and <rte-code>reassure <rte-code>performance regression tests, creating the latter will usually involve copy-pasting existing tests and altering them to match <rte-code>reassure<rte-code> API.

In a nutshell, instead of using <rte-code>render<rte-code> function from React Native Testing library, you would use <rte-code>measurePerfomance<rte-code> function and pass the <rte-code>scenario<rte-code> function to it, which will execute the scenario to measure.

To run <rte-code>reassure<rte-code>, you will run <rte-code>yarn reassure<rte-code> and get the results.

Let's take a look at a more detailed example, so we can understand how we can not only measure the performance of an existing component, but also refactor it while significantly improving its performance.

Refactoring long lists while measuring performance

Consider the following <rte-code>GameList<rte-code> component.

This component will get games data from <rte-code>useGames<rte-code> hook and render it on the screen. Before measuring component performance and optimizing it, let's first write a simple test for it.

We will create <rte-code>GameList.test.tsx<rte-code> file and write our test there. First, we need to mock <rte-code>useGames<rte-code> hook:

Initially, our component is pulling data from <rte-code>https://api.rawg.io/api<rte-code>, so we will need to match results to be in the same structure as returned from API. As you can see in the code above, we will generate <rte-code>1000<rte-code> game items with random ids, random names, and <rte-code>background_url<rte-code> pulled from <rte-code>imgur<rte-code>.

For the sake of simplicity in this particular example, we use <rte-code>jest.mock<rte-code> to mock our API results. Yet, it's advised to use msw to mock API call results,.

Now, let's write the test.

We are using <rte-code>NativeBase<rte-code> for our UI in this test, which means we also have to wrap our <rte-code>GameList<rte-code> with <rte-code>NativeBaseProvider<rte-code> and supply it with <rte-code>initialWindowMetrics<rte-code>, so it can successfully render our UI.

In this test, we have 1000 game items and list component is present. The test passes, but we can probably identify the problem here. There are 1000 rendered game items. Clearly, our component is not optimized and will render all 1000 items, even though they are not visible to the user initially.

Measuring app performance with Reassure

In the previous section, we've set up Reassure, so now, let's use it to measure the performance of our component. Let's copy our <rte-code>GameList.test.tsx<rte-code> to be <rte-code>GameList.perf-test.tsx<rte-code> and alter our performance test to match Reassure API.

Mocking

We will reuse the same mock we've created in <rte-code>GameList.test.tsx<rte-code>, but since the performance test will take way more time to run, we need to limit the test to only 100 items instead of 1000 and increase jest default timeout from 5000ms to a minute by using <rte-code>jest.setTimeout(60_000);<rte-code>.

Note that to get a more reliable performance measurement, Reassure executes each scenario ten times and averages the results (the number of runs can be easily configured using configure function).

Due to a slight API difference between Reassure and React Native Testing library, we also need to change our <rte-code>createWrapper<rte-code> function from the previous test in the following way:

Then we eliminate all our assertions and call <rte-code>measurePerformance<rte-code> function from reassure, passing it to our wrapper.

The final performance test looks like this:

At this point, we can run <rte-code>yarn reassure --baseline<rte-code> to gather baseline metrics to compare during all subsequent runs.

After running this command, <rte-code>.reassure/baseline.perf<rte-code> file is created with the following data:

Since there can be discrepancies between the measured performance metrics, Reassure is designed to execute multiple runs (10 by default) and provide the average for two main parameters:

• render count
• render duration

In the <rte-code>.perf<rte-code> file, we can see the exact rendering duration for each run and the number of renders for each run. We can increase the number of runs by providing <rte-code>runs: number<rte-code> option to <rte-code>measurePerfomance<rte-code> function.

Introducing scenarios

While measuring performance, we usually want to check whether the component is re-rendered due to UI interaction. In our example, when clicking on a game item, <rte-code>likes<rte-code> for this item are updated. The way it works at the moment is by setting the state on the component and, as a result, triggering <rte-code>GameList<rte-code> re-render.

Let's validate this scenario in our performance test and see how it will impact our metrics. We create a scenario async function where we can use any React Native Testing library methods.

Now we can pass this scenario to <rte-code>measurePerformance<rte-code> function:

Now, if we run <rte-code>yarn reassure<rte-code>, Reassure will compare <rte-code>.reassure/baseline.perf<rte-code> file <rte-code>with .reassure/current.perf<rte-code> file, and we will see the following results:

✅ Written Current performance measurements to .reassure/current.perf

🔗 /Users/vladimirnovick/dev/Callstack/Reassure/reassuredemo/.reassure/current.perf

❇️ Performance comparison results:

• Current: (unknown)
• Baseline: (unknown)

➡️ Signficant changes to render duration

• GameList perf test: 480.2 ms → 690.7 ms (+210.5 ms, +43.8%) 🔴🔴 | 1 → 2 >(+1, +100.0%) 🔴

➡️ Meaningless changes to render duration

➡️ Render count changes

• GameList perf test: 480.2 ms → 690.7 ms (+210.5 ms, +43.8%) 🔴🔴 | 1 → 2 >(+1, +100.0%) 🔴

➡️ Added scenarios

➡️ Removed scenarios

Here we can see an increase in the render count of the component and 43.8% increase in rendering time.

It's an excellent showcase of how new interactions during the development of new features can gradually increase rendering time and decrease the component's performance. Having a tool like Reassure gives us a detailed insight into how components' performance changes as the app grows.

Improving app performance with Reassure

Let's set this new metric as our new baseline and start improving our component to be more performant, measuring the number of re-renders and rendering time as we do our refactoring.

First of all, to set up a new baseline, let's run <rte-code>yarn reassure --baseline<rte-code>. As mentioned before, it will write new metrics in <rte-code>baseline.perf<rte-code> file and use it for comparison later on.

Refactoring to FlatList

You might be wondering, why not use FlatList? That’s a fair question. We shouldn't map over items, but rather use FlatList to render game items on the screen. So let's refactor our component accordingly:

Results will be outstanding:

Written Current performance measurements to .reassure/current.perf

🔗 /Users/vladimirnovick/dev/Callstack/Reassure/reassuredemo/.reassure/current.perf

❇️ Performance comparison results:

• Current: (unknown)
• Baseline: (unknown)

➡️ Signficant changes to render duration

• GameList perf test: 687.6 ms → 83.8 ms (-603.8 ms, -87.8%) 🟢🟢 | 2 → 2

➡️ Meaningless changes to render duration

➡️ Render count changes

➡️ Added scenarios

➡️ Removed scenarios

As you can see, there is an 87.8% improvement in rendering times.

If you run yarn test <rte-code>GameList.test.tsx<rte-code> to check if our unit test is working, it will fail due to the FlatList rendering only 10 game items out of 1000 items available. That’s because the remaining items are off-screen and do not need any rendering.

Let's also change our <rte-code>GameList.test.tsx<rte-code> test and switch from

to

For further improvement, let's move likes logic into <rte-code>GameItem<rte-code> component. This will yield the following results:

Performance comparison results:

• Current: (unknown)
• Baseline: (unknown)

➡️ Signficant changes to render duration

• GameList perf test: 687.6 ms → 47.3 ms (-640.3 ms, -93.1%) 🟢🟢 | 2 → 2

➡️ Meaningless changes to render duration

➡️ Render count changes

➡️ Added scenarios

➡️ Removed scenarios

✅ Written JSON output file .reassure/output.json

🔗 /Users/vladimirnovick/dev/Callstack/Reassure/reassuredemo/.reassure/output.> json

✅ Written output markdown output file .reassure/output.md

🔗 /Users/vladimirnovick/dev/Callstack/Reassure/reassuredemo/.reassure/output.md

As you can see from this example, we've successfully improved the performance of our component by 91.4% and reduced rendering times from 687.6ms to 59.3ms.

Using Reassure for performance regression testing in CI

Even though in our example, we've used Reassure as a tool to improve and measure performance, the area where the tool shines the brightest is Performance Regression testing.

As we develop new features, it's easy to gradually degrade performance until the problem becomes noticeable to the user. With Reassure, you can compile a report of performance changes by running Reassure in CI on your PRs.

It's important to note that the speed of the machine on which Reassure is running matters. To resolve any potential inconsistencies in the results, Reassure runs tests multiple times, averages the results, and provides a handy CLI tool to assess machine stability.

By running <rte-code>yarn reassure check-stability<rte-code>, it carries out performance measurements for the current code twice, so that baseline and current measurements refer to the same code. In such a case, the expected changes are 0% (no change). The degree of random performance changes will reflect the stability of your machine. This command can be run both on CI and local machines.

When integrating Reassure into CI, your CI agent stability is even more important than its speed. That’s because we want to see consistent results.

Tip: If your machine or CI agent stability is so-so, consider increasing the number of runs in the Reassure configure function.

You can create a custom script for CI to run to compare performance between your <rte-code>main<rte-code> branch and the current branch:

In the script above, we set the baseline branch to be the main branch and run <rte-code>yarn reassure --baseline<rte-code> to gather baseline metrics on it. Then, we compare them with the current branch by running <rte-code>yarn reassure<rte-code>. You can read more about setting up Reassure in CI in official docs.

Performance regression testing doesn’t have to be tiresome

Throughout this article, we’ve presented a few simple examples of things that can cause performance problems in the long run. As you've noticed, minor refactoring led to a 93.1% performance improvement. Even though performance issues presented in the article can be caught during a thorough PR review, spotting them can become inherently harder in complex codebases and as your React Native app grows. 

Making Reassure part of your test suite to address potential performance regressions and automate things in CI would greatly benefit you in the long run. Also, if you already write tests with react-native-testing-library (if you don't, you definitely should), creating Reassure perf tests is, to some degree, a copy-paste of existing tests with slight modification, and as such can be quickly introduced into the development workflow. 

If you have improvement suggestions, feel free to create issues in the Reassure repo and the team will address them. 

And if you're still hungry for knowledge, check out The Ultimate Guide to React Native, which discusses not only regression testing with Reassure, but also all things performance-related.