Mastering UI Testing
Abstract: have you ever wrote a successful UI test? How many times did you end up discarding a UI test because it's brittle and unmaintainable? And even if you don't write tests, tell me: do you click/type/click/click/setcookie/click/click while developing your UI?
I'm going to explain you the most important best practices that allow you to master the UI tests hell, sleep at night (because you are sure your project works as expected), test the back-end responses too and completely revolutionize the way you develop your UI!
I'm going to talk about - UI testing best practices - live demos with Cypress - improve your workflow using a UI test framework as your everyday working tool (so it's not only about testing)
Mastering UI Testing
I'm Stefano Magni (@NoriSte)
I'm a passionate front-end developer, a testing lover, and an instructor.
I work for Conio, a bitcoin startup based in Milan.
My recent OS contributions: github.com/NoriSte/all-my-contributions
(or just run $ npx noriste)
Slides info
Don't take notes or make photos... These slides are publicly available 🎉
slides.com/noriste/voxxed-days-ticino-2019-mastering-ui-testing
or
bit.ly/2mcwNai
Slides info
There are a lot of links into the slides 🔗
Almost all the logos are links too 🔗
Slides are organized both horizontally and vertically (see the bottom-right arrows).
Best practices are highlighted
My two UI testing principles:
• tests must not fail if the app is working (false negatives)
• if they fail, they must drive me directly to problem
Why is E2E testing hard?
Let's start:
Why is E2E testing hard?
Because we need to test four kind of contracts at the same time!
• HTML - 📄 semantic markup
• CSS - 👁 visual regressions
• JS - 🕹 interaction flows
• API - 🔁 client/server communication
Why is E2E testing hard?
Because it needs a real browser and a real context!
• ⌛️ everything is asynchronous
• 📦 E2E tests need a lot of reliable data
• 🐞 debugging E2E tests is hard
• 😭 the browser is slow, the UI is slow, the network is slow
Why is E2E testing hard?
Because generic browser automation tools s**k!
• ⌨️ no UI
• 🔒 no step-by-step test utilities
• 🚀 no integrated utilities
• 🤔 no clear reasons for failures
How could we make E2E testing easier?
First of all:
😱😱😱
Full E2E tests are not
so important!
UI integration tests (with a stubbed server) are way much more reliable!
| Suffers from | Full E2E test | UI integration test |
|---|---|---|
| the network | Yes 😓 | No 🎉 |
| the server | Yes 😓 | No 🎉 |
| data resetting | Yes 😓 | No 🎉 |
| complex case data | Yes 😓 | No 🎉 |
Full E2E tests are not so important!
| Full E2E test | UI integration test | |
|---|---|---|
| confidence | 100% 💪 | 90% 🤷♂️ |
| performance* | ~20"/test 😓 | ~2.5"/test 🚀 |
* benchmarking the Conio's back office test suite (env: staging)
UI integration tests (with a stubbed server) are way much faster!
Full E2E tests are not so important!
⬆️ E2E tests vs UI Integration tests ⬇️
Write a few E2E tests and a lot of UI integration tests
🤙
// Cypress example
cy.server();
cy.route({
url: "/auth/token",
method: "POST",
response: "fixture:authentication-success.json"
})
cy.visit("/login")
cy.get(".username").type("stefano@conio.com")
cy.get(".password").type("mysupersecretpassword")
cy.get(".btn-primary").click()
// the front-end app is going to receive the content
// of the authentication-success.json fixtureStubbing the back-end is easy with Cypress.
Here an authentication form example.
cy.route({
url: "/auth/token",
method: "POST",
response: "fixture:unexisting-user.json"
})
cy.route({
url: "/auth/token",
method: "POST",
response: "fixture:already-registered-user.json"
})Stubbing the back-end allows us to check every user flow with ease.
cy.route({
url: "/auth/token",
method: "POST",
response: {},
status: 401
})
cy.route({
url: "/auth/token",
method: "POST",
response: {},
status: 500
})And write an E2E test just for the happy paths.
At the moment, the Conio's back office test suite is composed by 110 UI integration tests and 15 E2E tests
🤔
What about the
client/server contract?
Test every request and response payload
🔁
How many times the front-end app doesn't work because of bad/misaligned payloads?
cy.server();
cy.route({
url: "/auth/token",
method: "POST",
response: "fixture:authentication-success.json"
}).as("login-request")
cy.visit("/login")
cy.get(".username").type("stefano@conio.com")
cy.get(".password").type("mysupersecretpassword")
cy.get(".btn-primary").click()
cy.wait("@login-request").then(xhr => {
expect(xhr.request.body).to.have.property("username", "stefano@conio.com")
expect(xhr.request.body).to.have.property("password", "mysupersecretpassword")
})Check the request payloads with the UI integration tests.
cy.server();
cy.route({
url: "/auth/token",
method: "POST"
}).as("login-request")
cy.visit("/login")
cy.get(".username").type("stefano@conio.com")
cy.get(".password").type("mysupersecretpassword")
cy.get(".btn-primary").click()
cy.wait("@login-request").then(xhr => {
expect(xhr.status).to.equal(200)
expect(xhr.response.body).to.have.property("access_token")
expect(xhr.response.body).to.have.property("refresh_token")
})Check the response payloads with the E2E tests.
Export/check every back-end schema/elasticsearch mapping/typescript definitions.
You can use Postman, Pact, GIT, my NPM package url-content-changes-checker, Jest snapshot testing.
My url-content-changes-checker in action
It highlights the changes in a text document
Do not slow down your tests with unnecessary sleeping
😴⌛️
Always wait for some determinist events, never make your tests sleeping.
How can I define the deterministic events?
Some examples are:
• every XHR requests
• UI element appearings
• everything valuable for the business
And how can I avoid test sleepings?
Cypress automatically waits!
// Cypress waits up to 60 seconds for a page load
cy.visit("/login")
// up to 4 seconds for an element appearing
cy.get(".username").type("stefano@conio.com")
// up to 5 seconds for an XHR request and
// up to 30 seconds for the XHR response
cy.wait("@login-request")
// and if cypress cannot wait for something,
// you can use my own plugin
cy.waitUntil(() => cy.getCookie("token")
.then(cookie => Boolean(cookie && cookie.value)))Cypress automatically waits!
Expose an authentication shortcut from the front-end
🏎💨
All the tests need authentication, do not authenticate through the UI all the times.
// front-end
if(window.Cypress) {
window.conioApp.store = reduxStore
}
// Cypress
window.conioApp.store.dispatch({
type: "LOGIN_REQUEST",
"stafeno@conio.com",
"mysupersecretpassword"
})Consuming some shortcuts (app actions) instead of the UI:
• is blazing fast
• avoids you to duplicate the authentication management
⬆️ through UI vs through shortcuts ⬇️
💪
Expose some constants from the front-end
A lot of times some tests fail for little to insignificant
front-end changes.
// instead of...
cy.visit("/login")
// you can
import {LOGIN_PATH} from "@/navigation/paths";
cy.visit(LOGIN_PATH)
// and if "/login" becomes "/sign-in" you will not
// face an annoying test failureWe all hate false negative test failures. Work to reduce them as much as possible.
🙋♂️🙋♀️
Assert frequently
The more you assert, the more the failures drive you to the exact problem.
// Cypress example
cy.server();
cy.route("POST", "/auth/token").as("login-request");
cy.visit(LOGIN_PATH)
cy.get(".username").type("stefano@conio.com")
cy.get(".password").type("mysupersecretpassword")
cy.get(".btn-primary").click()
cy.wait("@route_login").then(xhr => {
expect(xhr.request.body).to.have.property("username", "stefano@conio.com")
expect(xhr.request.body).to.have.property("password", "mysupersecretpassword")
expect(xhr.status).to.equal(200)
expect(xhr.response.body).to.have.property("access_token")
expect(xhr.response.body).to.have.property("refresh_token")
});
cy.get("#success-feedback").should("exist")
cy.url().should("not.include", LOGIN_PATH)
🏎 🚙 🚜
Split the different kind of tests
Different kind of tests have different purposes, usually you need to launch just some of them.
{
"cy:test:integration": "cypress run --spec \"cypress/**/*.integration.*\"",
"cy:test:e2e": "cypress run --spec \"cypress/**/*.e2e.*\""
}My three types of Cypress tests
| Type | Filename | When |
|---|---|---|
| UI integration tests | <name>.integration.test.js | • development • CI pipeline |
| e2e tests | <name>.e2e.test.js | • post CI pipeline • cron |
| monitoring tests | <name>.monitoring.test.js | • cron |
👁
Test the front-end the same way the user consumes it
The user does not care about selectors, he cares about contents.
// instead of testing the front-end through selectors...
cy.get(".username").type("stefano@conio.com")
cy.get(".password").type("mysupersecretpassword")
cy.get(".btn-primary").click()
cy.get("#success-feedback").should("exist")
// ... we can test it though contents, like the user does
import {
USERNAME,
PASSWORD,
LOGIN,
SUCCESS_FEEDBACK
} from "@/components/LoginForm/strings"
cy.findByText(USERNAME).type("stefano@conio.com")
cy.findByText(PASSWORD).type("mysupersecretpassword")
cy.findByText(LOGIN).click()
cy.findByText(SUCCESS_FEEDBACK).should("be.visible")It's made easy by some plugins like cypress-testing-library or cypress-xpath.
More: if your tests consume contents (and not CSS selectors, IDs etc.)...
... You just need to take a look at a screenshot to understand why a test failed!
If you can not do that, use ARIA or data-testid attributes.
❤️
Are you a TDD lover?
Cypress is perfect to be used as your main development tool and browser.
You can write your acceptance test and apply an outside-in approach to your front-end app.
Cypress can be launched:
# headlessly
$ cypress run
# with its UI
$ cypress open
Cypress can be launched with your Chrome version of choice:
Cypress creates a dedicated, persistent, Chrome user. You can install your DevTools of choice:
The Test Runner is amazing:
The Test Runner allows time-travelling:
Error reporting is amazing:
skip-and-only-ui and cypress-watch-and-reload plugins are useful too:
Cypress Dashboard is perfect for CI pipelines:
• Tests are re-launched on every CTRL+S
• if a test fails, Cypress automatically save screenshots and videos
• clock (setTimeout) control
• code coverage from E2E tests unified with the standard one
... And more:
🎉
So you do not have to test your front-end manually anymore!
Some common questions:
😊
Does Cypress supports only Chrome?
They are working on cross-browser support but yes, only Chrome is supported until now!
Uhm 🤔
Cross-browser tests are more useful for visual regression tests, integrate AppliTools with Cypress and you get them 😉
Is Cypress free?
Yes, you pay only if you want to have it upload a lot of test videos on its servers. 😉
Is it all about front-end testing?
🤔
Cypress solves the biggest testing problems but: no, there are other kind of tests.
• JS unit testing
• Component testing
• snapshot testing (HTML)
• visual regression testing (CSS)
• API testing
• App visual testing (CSS)
For component development and tests: take a look at Storybook and its storyshots addon.
Just after the ReactJSDay course, we are preparing a two-day workshop.
I and Jaga Santagostino are preparing a two-day workshop about everything related to JavaScript testing.
The first one will be about React testing.
Drop us a line if you want to know more about it 🗣
I'm working on a "UI Testing Best Practices" book on GitHub.
I've just begun writing it, all contributions are welcome 🙌
Recommended courses
Recommended sources
Thank you!