Mastering UI Testing

Writing UI tests for the web is hard, a lot of times the tests are brittle and unmaintainable. You must run them multiple times and, sometime, you need to exclude some of them from your pipeline.

I'm going to explain to you the most important best practices that allow you to master the UI tests hell. I'm going to show you how much a tool like Cypress can help you writing and maintaining your UI tests.

I'm Stefano Magni (@NoriSte)

I'm a passionate front-end developer, a testing & automation lover.

I work for Conio, a bitcoin startup based in Milan.

Here you can find my recent open-source contributions.

Slides info

Don't take notes or make photos... These slides are publicly available 🎉

https://slides.com/noriste/working-software-2019-mastering-ui-testing

or

https://bit.ly/2FeVUzB

I developed a super simple companion repository for these slides. If you want a follow-up project to play with, check it out. It's a simple authentication.

https://github.com/NoriSte/working-software-mastering-ui-testing

Slides info

There are a lot of links into the slides 🔗.

Almost all the logos are links too 🔗.

Slides are organized both horizontally and vertically (see the bottom-right arrows).

Best practices are highlighted

My two UI testing principles:

• tests must not fail if the app is working (false negatives)

• if they fail, they must drive me directly to problem

Why is E2E testing hard?

Let's start:

Why is E2E testing hard?

Because we need to test four kind of contracts at the same time!

• HTML - 📄 semantic markup
• CSS - 👁 visual regressions
• JS - 🕹 interaction flows
• API - 🔁 client/server communication

Why is E2E testing hard?

Because it needs a real browser and a real context!

• ⌛️ everything is asynchronous
• 📦 E2E tests need a lot of reliable data
• 🐞 debugging E2E tests is hard
• 😭 the browser is slow, the UI is slow, the network is slow

Why is E2E testing hard?

Because generic browser automation tools s**k!

• ⌨️ no UI
• 🔒 no step-by-step test utilities
• 🚀 no integrated utilities
• 🤔 no clear reasons for failures

How could we make E2E testing easier?

First of all:

😱😱😱

Full E2E tests are not

so important!

UI integration tests (with a stubbed server) are way much more reliable!

Suffers from	Full E2E test	UI integration test
the network	Yes 😓	No 🎉
the server	Yes 😓	No 🎉
data resetting	Yes 😓	No 🎉
complex case data	Yes 😓	No 🎉

Full E2E tests are not so important!

	Full E2E test	UI integration test
confidence	100% 💪	90% 🤷‍♂️
performance*	~20"/test 😓	~2.5"/test 🚀

* benchmarking the Conio's back office test suite (env: staging)

UI integration tests (with a stubbed server) are way much faster!

Full E2E tests are not so important!

⬆️ E2E tests vs UI Integration tests ⬇️

Write a few E2E tests and a lot of UI integration tests

🤙

// Cypress example
cy.server();
cy.route({
  url: "/auth/token",
  method: "POST",
  response: "fixture:authentication-success.json"
})

cy.visit("/login")
cy.get(".username").type("stefano@conio.com")
cy.get(".password").type("mysupersecretpassword")
cy.get(".btn-primary").click()

// the front-end app is going to receive the content
// of the authentication-success.json fixture

Stubbing the back-end is easy with Cypress.

Here an authentication form example.

cy.route({
  url: "/auth/token",
  method: "POST",
  response: "fixture:unexisting-user.json"
})

cy.route({
  url: "/auth/token",
  method: "POST",
  response: "fixture:already-registered-user.json"
})

Stubbing the back-end allows us to check every user flow with ease.

cy.route({
  url: "/auth/token",
  method: "POST",
  response: {},
  status: 401
})

cy.route({
  url: "/auth/token",
  method: "POST",
  response: {},
  status: 500
})

And write an E2E test just for the happy path*.

cy.visit("/login")
cy.get(".username").type("stefano@conio.com")
cy.get(".password").type("mysupersecretpassword")
cy.get(".btn-primary").click()

* the Conio's back office test suite is composed by 80 UI integration tests and 12 E2E tests

🤔

What about the

client/server contract?

Test every request and response payload

🔁

How many times the front-end app doesn't work because of bad/misaligned payloads?

cy.server();
cy.route({
  url: "/auth/token",
  method: "POST",
  response: "fixture:authentication-success.json"
}).as("login-request")

cy.visit("/login")
cy.get(".username").type("stefano@conio.com")
cy.get(".password").type("mysupersecretpassword")
cy.get(".btn-primary").click()

cy.wait("@login-request").then(xhr => {
  expect(xhr.request.body).to.have.property("username", "stefano@conio.com")
  expect(xhr.request.body).to.have.property("password", "mysupersecretpassword")
})

Check the request payloads with the UI integration tests.

cy.server();
cy.route({
  url: "/auth/token",
  method: "POST"
}).as("login-request")

cy.visit("/login")
cy.get(".username").type("stefano@conio.com")
cy.get(".password").type("mysupersecretpassword")
cy.get(".btn-primary").click()

cy.wait("@login-request").then(xhr => {
  expect(xhr.status).to.equal(200)
  expect(xhr.response.body).to.have.property("access_token")
  expect(xhr.response.body).to.have.property("refresh_token")
})

Check the response payloads with the E2E tests.

Export/check every back-end schema/elasticsearch mapping/typescript definitions.

You can use Postman, GIT, my NPM package url-content-changes-checker, Jest snapshot testing.

My url-content-changes-checker in action

It highlights the changes in a text document

Do not slow down your tests with unnecessary sleeping

😴⌛️

Always wait for some determinist events, never make your tests sleeping.

How can I define the deterministic events?

Some examples are:

• every XHR requests

• UI element appearings

• everything valuable for the business

And how can I avoid test sleepings?

Cypress automatically waits!

// Cypress waits up to 60 seconds for a page load
cy.visit("/login")

// up to 4 seconds for an element appearing
cy.get(".username").type("stefano@conio.com")

// up to 5 seconds for an XHR request and
// up to 30 seconds for the XHR response
cy.wait("@login-request")

// and if cypress cannot wait for something, you can use my own plugin
cy.waitUntil(() => cy.getCookie("token")
  .then(cookie => Boolean(cookie && cookie.value)))

Cypress automatically waits!

Expose an authentication shortcut from the front-end

🏎💨

All the tests need authentication, do not authenticate through the UI all the times.

// front-end
if(window.Cypress) {
  window.conioApp.store = reduxStore
}

// Cypress
window.conioApp.store.dispatch({
  type: "LOGIN_REQUEST",
  "stafeno@conio.com",
  "mysupersecretpassword"
})

Consuming some shortcuts instead of the UI:

• is blazing fast

• avoids you to duplicate the authentication management

⬆️ through UI vs through shortcuts ⬇️

💪

Expose some constants from the front-end

A lot of times some tests fail for little to insignificant front-end changes.

// instead of...
cy.visit("/login")

// you can
import {LOGIN_PATH} from "@/navigation/paths";
cy.visit(LOGIN_PATH)

// and if "/login" becomes "/sign-in" you will not
// face an annoying test failure

We all hate false negative test failures. Work to reduce them as much as possible.

🙋‍♂️🙋‍♀️

Assert frequently

The more you assert, the more the failures drive you to the exact problem.

// Cypress example
cy.server();
cy.route("POST", "/auth/token").as("login-request");

cy.visit(LOGIN_PATH)
cy.get(".username").type("stefano@conio.com")
cy.get(".password").type("mysupersecretpassword")
cy.get(".btn-primary").click()

cy.wait("@route_login").then(xhr => {
  expect(xhr.request.body).to.have.property("username", "stefano@conio.com")
  expect(xhr.request.body).to.have.property("password", "mysupersecretpassword")
  expect(xhr.status).to.equal(200)
  expect(xhr.response.body).to.have.property("access_token")
  expect(xhr.response.body).to.have.property("refresh_token")
});

cy.get("#success-feedback").should("exist")
cy.url().should("not.include", LOGIN_PATH)

🏎 🚙 🚜

Split the different kind of tests

Different kind of tests have different purposes, usually you need to launch just some of them.

{
  "cy:test:integration": "cypress run --spec \"cypress/**/*.integration.*\"",
  "cy:test:e2e": "cypress run --spec \"cypress/**/*.e2e.*\""
}

My three types of Cypress tests

Type	Filename	When
integration tests	<name>.integration.test.js	• development • CI pipeline
e2e tests	<name>.e2e.test.js	• post CI pipeline • cron
monitoring tests	<name>.monitoring.test.js	• cron

👁

Test the front-end the same way the user consumes it

The user does not care about selector, he cares about contents.

// instead of testing the front-end through selectors...
cy.get(".username").type("stefano@conio.com")
cy.get(".password").type("mysupersecretpassword")
cy.get(".btn-primary").click()
cy.get("#success-feedback").should("exist")

// ... we can test it though contents, like the user does
import {
  USERNAME,
  PASSWORD,
  LOGIN,
  SUCCESS_FEEDBACK
} from "@/components/LoginForm/strings"

cy.getByText(USERNAME).type("stefano@conio.com")
cy.getByText(PASSWORD).type("mysupersecretpassword")
cy.getByText(LOGIN).click()
cy.getByText(SUCCESS_FEEDBACK).should("exist")

It's made easy by some plugins like cypress-testing-library or cypress-xpath.

More: if your tests consume contents (and not CSS classes selectors, IDs etc.)...

... You just need to take a look at a screenshot to understand why a test failed!

If you can not do that, use ARIA or data-testid attributes.

🤔

Why did I mention TDD?

Because Cypress is perfect to be used as your main development tool and browser.

You can write your acceptance test and apply an outside-in approach to your front-end app.

Cypress can be launched:

# headlessly
$ cypress run

# with its UI
$ cypress open

Cypress can be launched with your Chrome version of choice:

Cypress creates a dedicated, persistent, Chrome user. You can install your DevTools of choice:

The Test Runner is amazing:

The Test Runner allows time-travelling:

Error reporting is amazing:

skip-and-only-ui and cypress-watch-and-reload plugins are useful too:

Cypress Dashboard is perfect for CI pipelines:

• Tests are re-launched on every CTRL+S

• if a test fails, Cypress automatically save screenshots and videos

• clock (setTimeout) control

• they're working on crossbrowser support

Something more?

🎉

So you do not have to test your front-end manually anymore!

Some common questions:

😊

Does Cypress supports only Chrome?

They are working on cross-browser support but yes, only Chrome is supported until now!

Uhm 🤔

Cross-browser tests are more useful for visual regressions, integrate AppliTools with Cypress and you get it 😉

Is Cypress free?

Yes, you pay only if you want to have it upload a lot of test videos on its servers. 😉

Is it all about front-end testing?

🤔

Cypress solves the biggest testing problems but: no, there are other kind of tests.

• JS unit testing

• Component testing

• snapshot testing (HTML)

• regression testing (CSS)

• callback testing

• App visual testing (CSS)

For component development and tests: take a look at Storybook and its storyshots addon.

We are preparing a two-day workshop.

I and Jaga Santagostino are preparing a two-day workshop about everything related to JavaScript testing.

The first one will be about React testing.

Drop us a line if you want to know more about it 🗣

I'm working on a "UI Testing Best Practices" book on GitHub.

I've just begun writing it, all contributions are welcome 🙌

Recommended courses

• Kent's "Testing JavaScript" course

• Egghead "Test with Cypress" Course

Recommended sources

• Yoni's "JavaScript Testing Best Practices"

• Gleb Bahmutov's blog

• Pragmatic Front-End Testing Strategies

We're hiring a front-end developer!