Which E2E testing framework to use for JS-based client applications?
Let us identify the right E2E testing framework for JavaScript-based client applications and propose an effective E2E testing process in the end.
What is End-to-END (E2E) testing?
End-to-End (E2E) testing is a technique used to test whether the flow of an application right from start to finish is behaving as expected.
In simple terms, a software testing methodology to test an application flow from start to end.
The purpose of performing end-to-end testing is,
- To identify system dependencies.
- To ensure that data integrity is maintained between various system components and systems.
- To simulate the real user scenario.
Why E2E testing?
- Test the complete flow of an application.
- Mimic the actual production scenario.
- Test application’s integrity with its interfaces.
The traditional way of E2E testing
- E2E testing is usually performed in an application using Selenium (Java).
- Covers the end-to-end of different application flows in a Staging environment — used for automated regression testing before release.
- E2E tests are written and executed by a Quality Analysis (QA) team and if there are any issues, they are reported as bugs.
Identifying the right framework
Let us try to narrow down an ideal JS-based E2E framework.
But why JS, why not Java?
Currently, the QA process involves E2E testing with Selenium (Java).
However, since frontend development is mostly based on JS frameworks like Angular, React, Vue or more, there is a need to identify a JS-based framework so that developers can write E2E tests along with development.
Developers can write E2E tests for an end-to-end feature implemented along with unit tests in the development phase. But note this is going to add more work on the developers. They can probably start writing E2E tests for happy flows initially until it becomes a practice.
Why E2E by Dev is important?
E2E tests written by developers help to identify feature issues before the QA phase, which is one of the obvious reasons.
However, another main reason would be to identify if E2E tests can help us reduce the frequently occurring bugs during development, mentioned below.
- Payload regression changes
Though this can be identified by unit tests at each layer,
E2E tests give us a bigger picture when multiple payloads are involved across the feature end-to-end.
(E.g. E2E testing for analytics instrumentation) - Visual regression changes
- Can E2E tests help out here?
- Can it report that the latest design changes fails in Safari but works in Chrome and Firefox?
- Can it report if we introduced a design regression which we were not supposed to?
Available JS frameworks for E2E testing
Some of the major JS frameworks available for E2E testing are listed below.
- WebdriverJS
- Protractor
- WebdriverIO
- NightwatchJS
- Cypress
- TestCafe
Note: The above-compared framework details are based at the time of writing this blog (June 2020).
Selenium-based frameworks Vs. Non-Selenium-based frameworks
Selenium-based frameworks use the web driver approach to interact with the browsers whereas the Non-Selenium-based frameworks can interact with the browser directly.
This is a major win for the Non-Selenium-based frameworks over the former as the overhead to install drivers is not there.
Among Selenium-based frameworks, WebdriverIO seems to be a suitable framework considering the minimal effort needed to set up the environment, run tests and output reports and WebdriverJS comes next with a little more effort for setup.
Among the Non-Selenium based frameworks, TestCafe wins over Cypress as it has support to run in different browsers along with mobile browser support.
Top 3 JS-based E2E testing frameworks (no particular order)
- TestCafe (Non-Selenium based)
- WebdriverIO (Selenium based)
- WebdriverJS (Selenium based)
All the above 3 are equally good and the decision to choose which one should be based on the tech stack and bandwidth of users who are going to be using this test framework in your project.
Selenium or Non-Selenium
From a developer perspective, TestCafe (Non-Selenium) wins for the below reasons,
- Open-source
- Easy to configure browsers
- Focus only on tests
- Good dashboard reports
Note: this is my personal opinion based on research and from a developer perspective, but if you feel Selenium works better for you from a process perspective in your company or other reasons — go ahead and use WebdriverIO or WebdriverJS, they are also equally good.
From a QA perspective, since QA already knows Selenium and WebdriverIO or WebdriverJS is based out of the same protocol that Selenium uses, writing tests in either of those can give a head-start to QA.
Although they may be familiar with Selenium concepts, a huge learning curve is there to learn JS tests and runners.
I would say learning will be almost same for QA even if they adopt Non-Selenium based frameworks — because usually, the E2E testing concepts are same behind all frameworks, only the ease of configuration and JS tests and runner syntax changes.
Maybe whether QA actually finds it easier to write in TestCafe than WebdriverIO, they should try it out.
Should QA even use JS to write E2E tests, why not continue with Java?
They can continue with Java. However, if Dev and QA combine to write tests in JS, they could be much more effective.
Some companies have already started doing this for creating exhaustive and thorough E2E test suites.
Assuming the best E2E test framework is chosen, can it solve visual regression problem?
Unfortunately, the answer is No. Though E2E test frameworks should ideally be able to solve the problem as per W3C standards, none of the frameworks supports it right now.
Mere screenshot comparison is not enough as each pixel is rendered by different screens differently and this would need an AI-based solution.
How to test visual changes, if not in E2E?
Visual regression testing
Visual testing is the act of verifying that an application’s graphical user interface (GUI) appears correctly.
The goal of the test is to find visual bugs so they can be fixed before the end-user sees them.
Additionally, visual testing can be used to verify content on a page.
This is ideal for sites that have graphical functionality (e.g., charts, dashboards, etc.) since verification with traditional automated functional testing tools can be very challenging.
Example
BackstopJS and Screener are popular visual testing tools with good customers.
Quick Summary
In sum, adopting E2E testing from the Dev side can help catch payload changes across a feature in the development phase rather than the QA phase but note, this will require more effort on developers.
Therefore, this decision needs to based on the bandwidth of the development team.
And is it worth QA to write tests along with Dev?
It could be effective or could be not — it ultimately depends on QA’s tech stack and differs for each company.
But note some companies have already started combining Dev and QA to create exhaustive E2E test suites.
Some key items to takeaway
1. Include unit tests to capture payload changes in the unit testing phase.
2. Let Developers start writing E2E tests using TestCafe or WebdriverIO or WebdriverJS — whichever works for your project.
Like said earlier, this will help to get the bigger picture which is missed out in unit tests.
3. QA can choose to try out JS-based frameworks like WebdriverIO (Selenium based) or TestCafe (Non-Selenium based and Dev preferred) and write tests along with Dev team for combined effective testing.
4. Explore visual regression testing solutions like BackstopJS or Screener for your project.
