E2E testing with Puppeteer

Organised by:

Hosted by:

Level: basic

Repository: https://github.com/NoriSte/cafe-meetup-e2e-testing-with-puppeteer

13/03/2019

Stefano Magni (@NoriSte)

Front-end Developer for

I'm Stefano Magni (@NoriSte)

I work for Conio, a bitcoin startup based in Milan.

 

You can find me on Twitter, GitHub or LinkedIn.

Long story short:

at the beginning there was the callback hell...

What is async/await?

function logAll(){
  logWithCallback("A", () => {
    logWithCallback("B", () => {
      logWithCallback("C", () => {
        logWithCallback("D", () => {});
      });
    });
  });
}
logAll();

... Then the promises come...

 

• better readability

• better control

• easier error catching

What is async/await?

function logAll(){
  logWithPromise("A")
  .then(() => logWithPromise("B"));
  .then(() => logWithPromise("C"));
  .then(() => logWithPromise("D"));
}
logAll();

... Then you can now await a promise...

 

• even better readability

• more concise code

What is async/await?

async function logAll(){
  await logWithPromise("A"));
  await logWithPromise("B"));
  await logWithPromise("C"));
  await logWithPromise("D"));
}
logAll();

from callbacks...

What is async/await?

async function logAll(){
  await logWithPromise("A");
  await logWithPromise("B");
  await logWithPromise("C");
  await logWithPromise("D");
}
logAll();
function logAll(){
  logWithCallback("A", () => {
    logWithCallback("B", () => {
      logWithCallback("C", () => {
        logWithCallback("D", () => {});
      });
    });
  });
}
logAll();

... to waiting promises

Jest is a zero-configuration testing framework.

What is Jest?

https://jestjs.io
beforeAll(async () => {
  // code to be run before the whole test suite
});
beforeEach(async () => {
  // code to be run before every test
});
afterAll(async () => {
  // code to be run after the whole test suite
});

describe('Description 1', () => {
  test('Test 1', () => {
      expect(1).toBe(1);
  });
  test('Test 2', () => {
      expect('hello').not.toBe('world');
  });

  describe('Inner description', () => {
    test('Inner test 1', () => {
        expect({foo: 'bar'}).toEqual({foo: 'bar'});
    });
    test('Inner test 2', () => {
        expect(0).toBeFalsy();
    });
  });
});

 PASS  ./test.js
  Description 1
    ✓ Test 1 (5ms)
    ✓ Test 2 (1ms)
    Inner description
      ✓ Inner test 1 (2ms)
      ✓ Inner test 2 (1ms)

Test Suites: 1 passed, 1 total
Tests:       4 passed, 4 total
Snapshots:   0 total
Time:        1.071s
Ran all test suites.

test.js

result of $ jest

What is E2E testing

(or UI testing)?

 

From Wikipedia:

End-to-end testing is a methodology used to test whether the flow of an application is performing as designed from start to finish.

It is useful to test your site/webapp/app in the same way your real users consume it.

The user interactions are automated.

 

From the Puppeteer.page.click API documentation:

This method fetches an element with selector, scrolls it into view if needed, and then uses page.mouse to click in the center of the element.

E2E tests are the only tool that can check if your website really works or not.

The most important thing:

E2E tests are hosted in a real browser.

What is Puppeteer?

Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol.

Puppeteer lets you control a Chrome instance and fire user-like interactions (click, typing etc.) within the page.

 

https://github.com/GoogleChrome/puppeteer

It introduces the concept of headless browser that is a full and standard browser that runs without a GUI.

1: Node running the script

2: Chromium launched with Puppeteer

3: Code executed in page

A typical script to launch Puppeteer

Remember some useful characteristics of Puppeteer...

You can install Chrome extensions on launch.

// welcome external devtools
return await puppeteer.launch({
  args: [
    // loading Chrome Vue devtools
    `--load-extension=/.../vue-devtools/shells/chrome/`,
    ]
});
https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#working-with-chrome-extensions

You can launch a single and shareable Chrome instance to improve performance.

// if you have previously saved the Chrome endpoint you can connect to it
// instead of creating a new one
browser = await puppeteer.connect({browserWSEndpoint: existingEndPoint});
e.g. https://github.com/smooth-code/jest-puppeteer

Chrome flags (hidden or newer settings) are customisable.

https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#puppeteerdefaultargsoptions
const browser = await puppeteer.launch({
    args: ['--no-sandbox']
});

You can go incognito.

https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#browsercreateincognitobrowsercontext
const browser = await puppeteer.launch();
// Create a new incognito browser context.
const context = await browser.createIncognitoBrowserContext();
// Create a new page in a pristine context.
const page = await context.newPage();
// Do stuff
await page.goto('https://example.com');

You could connect to a Chrome of yours (to leverage cookie, history etc.)

const browser = await puppeteer.launch({
    executablePath: '/path/to/Chrome'
});
https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#puppeteerexecutablepath

It allows you to easily understand if there are errors in page.

const browser = await puppeteer.launch({});
const page = await browser.newPage();

// when an error is thrown in Chrome
// it's logged by the node process
page.on('pageerror', e => console.log(e.text))
https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#event-pageerror

You can intercept requests.

await page.setRequestInterception(true);

page.on('request', request => {
  // it intercepts every call to register.php and
  // responds with a custom JSON
  if (request.url() === 'register.php') {
    request.respond({
      content: 'application/json',
      body: JSON.stringify({registered: true})
    });
  }
  else {
    // other requests work as usual
    request.continue();
  }
});
https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#event-request

You can set cookies.

await page.setCookie({
    name: 'loggedIn',
    value: '1'
});
https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagesetcookiecookies​

You can emulate devices.

const device = {
    'userAgent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1',
    'viewport': {
      'width': 375,
      'height': 667,
      'isMobile': true,
      'hasTouch': true,
      'isLandscape': false
    }
  }
await page.emulate(device);

// or you can use one of the presets

const devices = require('puppeteer/DeviceDescriptors');
const iPhone6 = devices['iPhone 6'];
await page.emulate(iPhone6);

https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pageemulateoptions

• $, $$, $eval, $$eval: DOM utilities

• addScriptTag, addStyleTag: to add resources in page

• page.evaluate: runs a function into the browser

• page.exposeFunction: adds a function

• page.click

• page.hover

And many others

https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md

How can we debug a Puppeteer script?

Turn off headless mode

So you can see what the browser is doing.

Directly from

https://github.com/GoogleChrome/puppeteer#debugging-tips

const browser = await puppeteer.launch({
    headless: false
});

Slow down the browser

Not ideal for very long tests but super useful though.

const browser = await puppeteer.launch({
  headless: false,
  slowMo: 250 // slow down every action by 250ms
});

Directly from

https://github.com/GoogleChrome/puppeteer#debugging-tips

Avoid closing the browser

afterAll(async () => {
  // await browser.close();
});

Directly from... me 😊

The button takes the user to another page.

Complete the test that checks it.

open ./dist/test-1.html

open ./test/test-1.e2e.test.js

• index.html has a big blue button

• test-1.e2e.test.js misses something (see the comments at line 22)

• launch the test suite with $ npm test

Demo 1

describe(`That's our first E2E test`, () => {
  test(`The button brings the user to the next page`, async () => {
    await page.goto(`file:${path.join(__dirname, './../dist/test-1.html')}`);

    // always add a 'data-test' attribute to the elements that will
    // participate to your tests
    await page.click('[data-test="button"]');

    // checking for a specific content is a good way to be 100% sure
    // that the page has been loaded
    await expect(page).toMatch('Hello from FETI');
  }, 5000);
});

Solution

It should be obvious but remember: the fact that your suite tests is running in a real and interactive browser... it doesn't mean that you should interact with it!

 

When you are running the browser in non-headless mode remember to not affect the test with your mouse/keyboard.

Never interact with the test

<a data-test="button">

<!--
find it with the following selector
[data-test="button"]
-->

If you can't rely on contents... Always add a data-test (or data-testid) attribute to the elements that will be referenced by the tests.

Best practice

Why?

Because every attribute has a “role”.

 

We use .classes for CSS (and they got replaced in case of CSS Modules), #ids for JS, and therefore they can change based on CSS and JS needs…

 

A dedicated attribute is more resilient to refactoring and leads every (diligent) developer to keep it or, at least, ask himself for the attribute use.

data-test attribute

Ok, so:

• E2E tests are very important...

• the first test seemed really easy to be written…

Amazing, I’ll write ~1000 E2E tests as soon as I back home!!!

Wow...

Let’s speak about the cons!

They’re the longest to be written, not in terms of lines of code but in terms of variables to be considered.

Cons

Why?

Because you aren’t in isolation (like in Unit tests), nor in a super-mocked environment...

 

You have a real browser on a real network, you’ll face network latencies, temporary downs of the server, service workers, unpredictable AD scripts and banners, (possible) browser drivers inconsistencies…

Cons

Simulating the (exact) user behaviour sometimes could be very tricky (blur, input typos etc.).

Cons

They’re not always parallelisable (in simple cases they are, but they can lead to errors and debugging hell).

Cons

The more complex they are, the more difficult they are to debug.

Cons

If you try to DRY your tests you’re probably adding another (hard to manage) complexity layer.

Cons

They can fail, get used with that, there are too many context variables to be considered.

Cons

They are slow, even with almost instant UI interactions.

Cons

Everyone defines them flaky and brittle...

Cons

Demo 2

It's the same of the first test but a the page has a cookie footer...

describe(`That's our second E2E test`, () => {
  beforeAll(async () => {
    await page.goto(`file:${path.join(__dirname, './../dist/test-2.html')}`);

    // don't let the test fail for a silly element like a cookie footer
    // It could be already accepted when you navigate to another page
    if(await page.$('[data-test="cookie-footer-acceptance"]')) {
      try {
        // what happens if it exists but isn't clickable (eg. it's hidden)?
        // A try/catch will manage the case
        await page.click('[data-test="cookie-footer-acceptance"]');
      } catch(e) {
        // the element exists but isn't clickable
      }
    }
  });
});

Solution

Never use some "sleep" code, you can’t determine how much a page/script could be waited for (render waitings, network conditions etc.).

 

Use waiters, promises, framework-specific render callbacks (like Vue.nextTick) but not sleep.

Never "sleep"

Again! Now the cookie footer disappears with a CSS animation!

Demo 3

if(await page.$('[data-test="cookie-footer-acceptance"]')) {
  try {
    await page.click('[data-test="cookie-footer-acceptance"]');
    // you can wait that an element is hidden
    // @see https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagewaitforselectorselector-options
    await page.waitForSelector('#cookie-footer', {
      hidden: true
    });
  } catch(e) {
    // the element exists but maybe it isn't clickable
  }
}

Solution

With page.evaluate you can execute code into the running browser (like you do when you write code into the DevTools console) and give back the result to your NodeJS script.

Page.evaluate

Page.evaluate example

const puppeteer = require('puppeteer');

(async (options) => {
  browser = await puppeteer.launch({
    headless: false,
    devtools: true
  });
  page = await browser.newPage();

  console.log('Node logging');
  await page.evaluate(
    () => console.log('Browser logging')
  );
})()

const seven = 7;
const result = await page.evaluate(aNumber => {
  // JQuery is in page? Redux?
  // Use them and give back results to the NodeJS script
  return aNumber * 10;
}, seven);
console.log(result); // 70

Take care of the scope 

const seven = 7;
const result = await page.evaluate(() => {
    // error! Because the scope is the Chrome's window!
    // Not the NodeJS script!
    return seven * 10;
});
console.log(result);

To better understand the scope problem: remember that the Node script and the launched Chrome instance are separated

Use a promise for every async code

const seven = 7;
const result = await page.evaluate(aNumber => {
  // you can use a promise for every async stuff
  return new Promise(resolve => {
    setTimeout(() => resolve(aNumber*10), 1000);
  });
}, seven);
console.log(result); // 70

About the test-4 solution

// code refactored without arrow functions
await page.evaluate(function() {
  // a Promise is returned to the page.evaluate
  // so it waits until the promise is fullfilled
  return new Promise(resolve => {
    // now we're in the Chromium instance, we can listen for
    // the event triggered on the window
    window.addEventListener('cookieFooterDidHide', function(){
      // when the event has been triggered we fullfill the promise
      resolve();
    });
  });
});
// the code after the page.evaluate will be run once the event
// in the browser will be triggered

Page.evaluate is often used to retrieve data from the DOM

const SELECTOR = '[href]:not([href=""])';
let link;

link = await page.evaluate((sel) => 
    document.querySelector(sel).getAttribute('href')
  , SELECTOR);

But it isn't the only way to do that!

const SELECTOR = '[href]:not([href=""])';
let link;

link = await page.evaluate((sel) => 
    document.querySelector(sel).getAttribute('href')
  , SELECTOR);

// compare the two following examples

link = await page.$eval(SELECTOR, el => el.getAttribute('href'));

// or

link = await page.$(SELECTOR).getProperty('href').jsonValue();

The last time with the button, trust me 😊

Now the cookie footer disappears (from a user perspective... not from a CSS one) dispatching an event!

Test 4

if(await page.$('[data-test="cookie-footer-acceptance"]')) {
  try {
    await page.click('[data-test="cookie-footer-acceptance"]');

    // @see https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pageevaluatepagefunction-args
    await page.evaluate(() => new Promise(resolve => {
      // the following code will run into the browser page
      window.addEventListener('cookieFooterDidHide', () => resolve());
    }));

  } catch(e) {
    // the element exists but maybe it isn't clickable
  }
}

Solution

What should I test?

I suggest you to test only the best cases or the ones that generate value ($$$) for your company.

Test something that broke in the past or that breaks often.

Let's see some other debugging tips

Capture console output

Directly from

https://github.com/GoogleChrome/puppeteer#debugging-tips
page.on('console', msg => console.log('PAGE LOG:', msg.text()));

Launch Puppeteer with the DevTools already opened 

const browser = await puppeteer.launch({
  headless: false,
  slowMo: 250, // slow down by 250ms
  devtools: true
});

Directly from

https://github.com/GoogleChrome/puppeteer#debugging-tips

Add an evaluate statement with debugger inside

Or add debugger to an existing evaluate statement (it works if the Chrome DevTools are opened).

await page.evaluate(() => {debugger;});
// The test will now stop executing in the above evaluate statement,
// and chromium will stop in debug mode.

Directly from

https://github.com/GoogleChrome/puppeteer#debugging-tips

Change default test timeout

jest.setTimeout(100000);

Directly from

https://github.com/GoogleChrome/puppeteer#debugging-tips

Run a single test/suite

describe.only('Temporary run only me please', () => {
    test('The test we want to isolate', () => { ... });
});

describe('A suite that won\'t be launched', () => {
    test('Test 2', () => { ... });
});

describe('Another suite that won\'t be launched', () => {
    test('Test 3', () => { ... });
});

Directly from

https://github.com/GoogleChrome/puppeteer#debugging-tips

Skip some tests/suites

describe('Suite 1', () => {
    test('The test we want to isolate', () => { ... });
});

describe.skip('A suite that will be skipped', () => {
    test('Test 2', () => { ... });
});

describe('Suite 3', () => {
    test.skip('A test that will be skipped', () => { ... });
});

Directly from

https://github.com/GoogleChrome/puppeteer#debugging-tips

console.log (into the Chrome instance) the name of the test, so you always know which test is running if you are watching them.

await page.evaluate(() => console.log('Test name'));

Directly from... me 😊

You can also add a fixed div in page to show the test name at the top of the page, it’s not ideal but it could be very useful in some situations.

Directly from... me 😊

Use the right assertion

https://jestjs.io/docs/en/expect
Expected: true
Received: false
    3 | describe(`Test`, () => {
    4 |   test(`Leverage assertions`, () => {
  > 5 |     expect(getObj().key === 5).toBe(true);
// Ok, what is the value of getObj().key?
Expected: 5
Received: 4
    3 | describe(`Test`, () => {
    4 |   test(`Leverage assertions`, () => {
  > 5 |     expect(getObj().key).toBe(5);
// ok I'll fix it without relaunching the suite

Directly from... me 😊

Run test serially with

$ npm test --runInBand

because a page executed in the background could have some unexpected behaviors...

Directly from... me 😊

Demo 5

 

• index.html is the usual Todo List app

• add all the todos in the array, remove the first two and then... read the store and check that the last two todos exist in the store itself.

P.s. I exposed a window.vueInstance variable so you can access the store with window.vueInstance.$store.state.todos

# TLDR
$ open test-5.e2e.test.js

Solution

This test leaves room for a series of cases that should make you think about how it’s difficult to manage all the cases consistently when E2E testing.

Other best practices

Puppeteer is Chrome/Firefox(at the moment) but if you use Selenium etc. remeber to avoid launching every test on every browser. Choose a reference browser and carefully select the test to be run on the other ones.

Choose where to test

Make a screenshot if a test fails. It can help you avoiding to relaunch the suite with the browser in non-headless mode.

Make screenshots

if (array.length !== 3)
    await page.screenshot({path: 'screenshot.png'})
expect(array.length).toBe(3)

In a page where everything changes soon… try to standardize your testing environment to avoid false negatives.

Attention to the pages that update frequently

Don't think to test every corner case with E2E testing, it's pure madness.

Leave perfectionism at home

An assertion is auto-explicative, an error isn't.

Assert frequently

await page.goto(AUTHENTICATED_ROOT);

expect(page.url()).toEqual(expect.not.stringContaining('/login'));

// you won't have a "element not found"
// if something went wrong with the authentication
await page.click('[data-test="create-new-post"]');

Puppeteer alternatives?

  • widely used

 

  • not so easy to install/setup

 

  • it has a WebDriver for every desktop browser (IE too)

 

  • Expected Conditions FTW (a sort of page.waitForSelector on steroids) 

Selenium WebDriver

  • amazing "DevTools" UI

 

  • clear errors

 

  • play/pause functionality

 

  • made only for E2E testing, it's not a generic automation browser

Cypress.io

  • works in real browsers (so you can even test on mobile ones)

 

  • supported by BrowserStack and CrossBrowserTesting

 

  • who uses it loves it 🙂 

     

TestCafe

 

  • Again, first of all: consider what and why you should test on mobile...

 

  • You can use the device emulation of Chrome DevTools with Puppeteer

 

  • If you really need to test on real devices choose TestCafe

What about mobile browsers?

If  you want to see the differences between Puppeteer/TestCafe/Cypress you can take a look at a repository I made to solve an issue on StackOverflow:

 

https://github.com/NoriSte/stackoverflow-52383438-cypress-issue

Are E2E tests framework-agnostic?

Yes but every framework has (obviously) a lot of libraries to help you write them.

What can I do with Puppeteer as well as test my web apps?

Automate annoying and repetitive browser tasks.

eg. We used it to configure a new website with ISPConfig.

Scrape whatever you want from the browser (eg. data fetch without APIs).

Make self-contained Node.js apps with a UI (thank to Carlo by Google.

Use it as a development tool

If you're developing a "change password" flow, for every change you make to the code you need to manually:

• login

• go to the profile page

• fill the form to change the password

• click logout

• login with the new password

Use it as a development tool

Just to realize that you have a bug... Then fix it and start again...

You could use Puppeteer to bring you directly at the end of the flow.

And once you finished... Add some assertions and your E2E test is ready 😊

Use it while studying

When you're studying a new framework you don't know if you're breaking something you developed just some hours ago... unless you check it manually! Use some tests instead.

Eg. I used them extensively during my first refactors while studying Vue.

Scrape the first 30 results from Google for the given query, have fun 😊

Demo 6

Solution

# TLDR
$ open test-6.e2e.test.js

Bonus track:

 

Why is testing so important?

It's useful to find bugs and corner cases while coding.

To avoid manually testing ourselves  the same things over and over.

To avoid any kind of regression while updating our (or others) code.

To confidently release our code without worries.

To constantly check that our products are up and running.

In the end: testing is fundamental to sleep at night...

Bonus track:

 

What kind of tests exist?

With UT you test functions, classes, in general everything you consider a "unit".

 

You check the behaviour of your units with every kind of input they can receive.

 

They say little about your code but they are extremely fast, both to be written and to be run.

Unit tests

With Integration tests you test a limited amount of units/components in a controlled environment.

 

You check how they work together and they allow you to start seeing the bigger picture.

 

Generally you mock (replace with fake but credible code) almost everything external to what you're focusing on.

 

Integration tests

They test the visual side of your web-app, they literally make a screenshot of your page and compare every pixel with the previous one.

 

They are extremely slow and they are almost dedicated to test CSS side of your project. 

Regression tests

Funny moment: a GIF I found while studying for this workshop 😁

The goal of the author was to tease the unit testers to introduce them about the need for E2E testing.

We are now experts 💪😁

End-to-end tests

The testing pyramid

Kent C. Dodds's return of investment testing "pyramid"

Always use the simplest kind of test

Never write an E2E test to test something you can test with an Integration one... And never write an Integration test if you can get the same result with an Unit one.

• E2E tests really matter, they don't know anything about your architecture, they test what the end user sees

 

• they’re quite easy to develop, with a flat learning curve

 

• write a few amount of tests, don't delegate to E2E testing something you can test with other testing methodologies

Conclusion

• remember to take the tests simple, they could be extremely time-consuming

 

• never "sleep" the browser

 

• use "data-test"/"data-testid" attributes

 

• remember that you can scrape/execute whatever you want on the web with an automated browser

Conclusion

• https://puppeteersandbox.com

Playground

• https://www.blazemeter.com/blog/top-15-ui-test-automation-best-practices-you-should-follow

• https://medium.freecodecamp.org/why-end-to-end-testing-is-important-for-your-team-cb7eb0ec1504

• https://blog.kentcdodds.com/write-tests-not-too-many-mostly-integration-5e8c7fff591c

• https://hackernoon.com/testing-your-frontend-code-part-iii-e2e-testing-e9261b56475

• https://gojko.net/2010/04/13/how-to-implement-ui-testing-without-shooting-yourself-in-the-foot-2/

• https://willowtreeapps.com/ideas/how-to-get-the-most-out-of-ui-testing

• http://www.softwaretestingmagazine.com/knowledge/graphical-user-interface-gui-testing-best-practices/

• https://www.slideshare.net/Codemotion/codemotion-webinar-protractor

• https://frontendmasters.com/courses/testing-javascript/introducing-end-to-end-testing/

• https://medium.com/welldone-software/an-overview-of-javascript-testing-in-2018-f68950900bc3

• https://medium.com/yld-engineering-blog/evaluating-cypress-and-testcafe-for-end-to-end-testing-fcd0303d2103 

Sources

• https://egghead.io/courses/end-to-end-testing-with-google-s-puppeteer-and-jest



• https://egghead.io/courses/end-to-end-testing-with-cypress

Egghead courses

• https://testingjavascript.com/

Kent's course (reccomended)

Any feedbacks??

• CaFE and Giacomo Zinetti

 

• GDS Communication for the amazing location

 

• Kent C. Dodds for the amazing “Solidifying what you learn” post

A special thank to:

Last but not least:

"Share to learn" really works!

Organised by:

Hosted by:

13/03/2019

Stefano Magni (@NoriSte)

Front-end Developer for

Thank you!

The repository with the code and the link to these slides

https://github.com/NoriSte/cafe-meetup-e2e-testing-with-puppeteer

E2E testing meetup for CaFE

By Stefano Magni

Made with Slides.com

E2E testing meetup for CaFE

In March 2019 I had a talk for the CaFE community in Como (https://www.meetup.com/it-IT/Como-and-Frontend-CaFE/events/259556783/). The talk aimed to introduce the attendees to the amazing world of browser automation, mostly for E2E testing but some showed examples were about web scraping too.

  • 4,272

More from Stefano Magni