Tackling Flaky Tests With Cypress and Playwright through Network Synchronization

Flaky tests are a common challenge in end-to-end testing. There are many types of flaky tests. In this blog we would cover the flakiness that comes when UI actions take place before the API response has arrived. We'll see how Cypress and Playwright, address these challenges.

Taming flakiness in Cypress

In Cypress, the cy.wait() command is used to pause the test execution. Let's explore how Cypress handles flakiness with the cy.intercept() and cy.wait() commands.

Let's consider an example of an online shopping application where a new order is created when we click the submit button.

cy.intercept("/orders/*").as("fetchOrder");
cy.get("[data-cy='submit']").click();
cy.wait("@fetchOrder");

Let's understand what the above example is trying to achieve line by line.

cy.intercept("/orders/\*").as("fetchOrder"): Sets up a network interception. It intercepts any network request that matches the pattern /orders/ and gives it a unique alias fetchOrder. This allows us to capture and control the network request for further testing.

cy.get("[data-cy='submit']").click(): Locates an HTML element with the attribute data-cy set to submit and simulates a click on it.

cy.wait("@fetchOrder"): Instructs Cypress to wait until the intercepted network request with the alias fetchOrder is completed before proceeding with the test.

The cy.wait() command involves two distinct waiting phases.

Phase 1: The command waits for a matching request to be sent from the browser. In the provided example, the wait command pauses execution until a request with the URL pattern /orders/ is initiated by the browser. This waiting period continues until a matching request is found. If the command fails to identify such a request within the configured request timeout, a timeout error message is triggered. Upon successfully detecting the matching request, the second phase kicks in.

Phase 2: In this phase, the command waits until the server responds. If the anticipated response fails to arrive within the configured response timeout, a timeout error is thrown. In the above example, the wait command in this phase will wait for the response of the request aliased as fetchOrders.

The dual-layered waiting mechanism, as explained above significantly contributes to the reliability of tests. It ensures a synchronized interaction between UI actions and server responses, facilitating more robust and dependable test scenarios.

Managing multiple responses

Consider a situation where a user adds a product to the cart thus initiating two concurrent requests. The first request adds the product to the cart, while the second request fetches the updated list of orders. To ensure the synchronization of these asynchronous actions, we must wait for both requests to be successfully completed before continuing with the test execution.

Cypress provides the times property in the cy.intercept() options, offering control over how many times a request with a particular pattern should be intercepted.

cy.intercept({ url: "/orders/*", times: 2 }).as("fetchOrders");
cy.get("[data-cy='submit']").click();
cy.wait(["@fetchOrders", "@fetchOrders"]);

Let's decode the above example line by line.

cy.intercept({ url: "/orders/\*", times: 2 }).as("fetchOrders"): Specifies that the interception should match requests with a pattern /orders/ and limit the interception to exactly two occurrences.

cy.get("[data-cy='submit']").click(): Locates an HTML element with the attribute data-cy set to submit and simulates a click on it.

cy.wait(["@fetchOrders", "@fetchOrders"]): Ensures that the test waits until the two intercepted requests with the alias fetchOrders are completed before moving on to the next steps.

Taming Flakiness in Playwright

Playwright offers page methods like waitForRequest and waitForResponse to address synchronization challenges between UI actions and API responses. Both these methods return a promise which is resolved when an API with a matching pattern is found and throws an error if it exceeds the configured timeout.

Let's consider the same example of an online shopping application where a new order is created when we click the submit button.

await page.getByRole("button", { name: "Submit" }).click();
await page.waitForResponse(response => response.url().includes("/orders/"));

In the above example, page.waitForResponse waits for a network response that matches with the URL pattern /orders/ after clicking the submit button.

Even though the above example seems simple, there is a chance for flakiness here. That is because the API might respond before Playwright starts waiting for it. It might happen for two reasons:

1. API is very fast.
2. External factors delay the test script.

Such situations could lead to timeouts and test failures.

To address the above issue, it's important to coordinate the promises so that the waitForResponse command runs at the same time as UI actions. The following example illustrates this approach.

const fetchOrder = page.waitForResponse(response =>
  response.url().includes("/orders/")
);

await page.getByRole("button", { name: "Submit" }).click();
await fetchOrder;

In the above example, the page starts watching for the responses matching the specific URL pattern, /orders/ before clicking the submit button. The waitForResponse command returns a promise, which we have saved into the variable fetchOrder. After performing the click action in the following line, we wait for the promise stored in fetchOrder to resolve. When it resolves, it signifies that the response has been received. This enables us to move on to the next assertion without facing any reliability issues.

Managing Multiple Responses

Let's consider a scenario similar to the one explained in Cypress, where we have to manage multiple responses, one to add a product and another to fetch the updated list of products.

To wait for the completion of 2 requests from the same URL pattern, consider the following approach.

const fetchOrders = Promise.all(
  [...new Array(2)].map(
    page.waitForResponse(response => response.url().includes("/orders/"))
  )
);
await page.getByRole("button", { name: "Submit" }).click();
await fetchOrders;

In the above example, we start waiting for two responses with the pattern /orders/ using Promise.all. The flaw in the above code is that when both the waitForResponse methods run in parallel, they end up tracking the exact same API request. In simpler terms, it's like waiting for just one request, as both of them wait for the completion of the same API.

To solve the above problem, it's important to improve the code by keeping track of the resolved APIs. Let's see how to achieve the same.

const trackedResponses = [];

const fetchOrders = Promise.all(
  [...new Array(2)].map(() =>
    page.waitForResponse(response => {
      const requestId = response.headers()?.["x-request-id"];

      if (
        response.url().includes("/orders/") &&
        !trackedResponses.includes(requestId)
      ) {
        trackedResponses.push(requestId);
        return true;
      }

      return false;
    })
  )
);

await page.getByRole("button", { name: "Submit" }).click();
await fetchOrders;

In the above example, we have initialized a new variable trackedResponses with an empty array, intended to store unique identifiers (request IDs) of resolved APIs. It checks if the URL includes the substring /orders/ and also whether the request ID has not already been tracked in trackedResponses array. If both conditions are satisfied, it adds the request ID to trackedResponses array and returns true, indicating that we should wait for the response. This approach prevents the monitoring of the same response more than once.

Conclusion

By understanding and implementing these synchronization techniques in Cypress and Playwright, we can significantly enhance the robustness and reliability of end-to-end tests, ultimately contributing to a more stable and trustworthy testing suite.

References

cy.intercept

cy.wait

page.waitForRequest

page.waitForResponse