Hasura's Console

E2E tests chronicles

Stefano Magni

Front-end tech leader

(Platform team)

I'm Stefano Magni

I'm a passionate Front-end engineer, and an instructor.

 

I'm in the Platform team of Hasura, a SAAS that gives you instant GraphQL & REST APIs on new & existing data sources.

You can find all my articles, courses, OSS contributions here

github.com/NoriSte/all-my-contributions

Including UI Testing Best Practices (1.4K ⭐ïļ)

github.com/NoriSte/ui-testing-best-practices

I'm a Front-end engineer

Not a QA engineer 😊

Hasura's Console

May 2022

Fix the Hasura Console's E2E tests

The main problems:

The E2E tests are slow

The E2E tests are flaky

The E2E tests are cryptic

Secondary problems:

Debugging the E2E tests is challenging

The server teams use the E2E tests also to test the server

The E2E tests are slow

The E2E tests are slow

☠ïļ cy.wait(10_000) ☠ïļ

Slow ðŸĒ

Nonexplanatory ðŸĪ”

Possibly flaky ðŸĪĶ‍♂ïļ

Let's fix it!... ?

Multiple requests ðŸ˜ģ

Unpredictable order ðŸĨķ

Different servers 😓

The E2E tests are slow

Approaches

Collect all the outgoing requests and

wait for their completion ðŸĪĶ‍♂ïļ

Intercept the same request to

different servers 😓

Wait for something that reflects the response ðŸĪ·â€â™‚ïļ

(commenting the whys)

The E2E tests are slow

The E2E tests are flaky

The E2E tests are flaky

No tests > Flaky tests

Undermine confidence and trust ðŸ˜Ī

Add friction ⚓ïļ

CI misconfiguration

The E2E tests are flaky

CI misconfiguration

Slow CI ðŸĒ

Tests went outdated ðŸ‘ī

We were paying for them ðŸ’ļ

The E2E tests are flaky

The E2E tests are cryptic

The E2E tests are cryptic

Test's code = 100x simpler than app's code

Tests' goals

To test features ðŸĪŠ

Ease the readers' job ðŸĪ“

Ease debugging themselves ðŸ’Ą

The E2E tests are flaky

Abstraction

Requires creating a mental model ðŸšĩ‍♀ïļ

Gets in the way of debugging ⚓ïļ

The E2E tests are cryptic

Abstraction

The E2E tests are cryptic

// spec.ts file
it('Create Query Action', createQueryAction);

// test.ts file (simplified version)
export const createMutationAction = () => {
  // ...
  clearActionDef();
  typeIntoActionDef(statements.createMutationActionText);
  clearActionTypes();
  // ...
};

// test.ts file contains the clearActionDef, typeIntoActionDef, etc.
const clearActionDef = () => {
  cy.get('textarea').first().type('{selectall}', { force: true });
  cy.get('textarea').first().trigger('keydown', {
    keyCode: 46,
    which: 46,
    force: true,
  });
};

// test.ts file contains also the statements
const statements = {
  createMutationActionText: `type Mutation {
    login (username: String!, password: String!): LoginResponse
  }`,
  createMutationCustomType: `type LoginResponse {
    accessToken: String!
  }
  `,
  createMutationHandler: 'https://hasura-actions-demo.glitch.me/login',
  // ...
}

Abstraction

The E2E tests are cryptic

it('Create Query Action', () => {
  cy.get('textarea').eq(0).as('actionDefinitionTextarea');
  cy.get('textarea').eq(1).as('typeConfigurationTextarea');

  cy.get('@actionDefinitionTextarea')
    .clearConsoleTextarea()
    .type(
    `type Mutation {
        login (username: String!, password: String!): LoginResponse
      }`,
    { force: true, delay: 0 }
  );

  cy.get('@typeConfigurationTextarea')
    .clearConsoleTextarea()
    .type(
    `type LoginResponse {
      accessToken: String!
    }
    `,
    { force: true, delay: 0 }
  );

  // ...
})

When abstraction is good?

When hides workarounds ðŸŦĢ

When they are soft 👍

The E2E tests are cryptic

Abstraction

The E2E tests are cryptic

/**
 * Clear a Console's textarea.
 * Work around cy.clear sometimes not
 * working in the Console's textareas.
 */
Cypress.Commands.add('clearConsoleTextarea',
  { prevSubject: 'element' },
  el => {
    cy.wrap(el).type('{selectall}',
      { force: true })
      .trigger('keydown', {
        keyCode: 46,
        which: 46,
        force: true,
      });
});

Abstraction

The E2E tests are cryptic

// ❌ bad abstraction
export const expectNotification = (
  {
    type,
    title,
    message,
  }: {
    type: 'success' | 'error';
    title: string;
    message?: string;
  },

  timeout = 10000
) => {
  const el = cy.get(
    type === 'success' ? '.notification-success' : '.notification-error',
    { timeout }
  );

  el.should('be.visible');
  el.should('contain', title);

  if (message) el.should('contain', message);
};



// ✅ good abstraction
function expectSuccessNotification(title: string) {
  cy.get('.notification-success')
    .should('be.visible')
    .should('contain', title)
}

Matching the test's code and test runner's commands

The E2E tests are cryptic

The E2E tests are cryptic

Use clear selectors

The E2E tests are cryptic

cy.get('textarea') // ðŸĪ”
  .eq(0)           // ðŸĪ”
  .type(`{enter}{uparrow}${statements.createMutationGQLQuery}`, {
    force: true,
  });
cy.get('textarea').eq(0).as('actionDefinitionTextarea');
cy.get('textarea').eq(1).as('typeConfigurationTextarea');

// âĪïļ
cy.get('@actionDefinitionTextarea').clearConsoleTextarea().type(/* ... */);

The E2E tests are cryptic

There's more in the article

Reduce data-testid attributes

Group related actions

Debugging the E2E tests is challenging

Debugging the E2E tests is challenging

What do you prefer?

Small and dependent E2E tests

Small and independent E2E tests

One long E2E test

Debugging the E2E tests is challenging

Small and dependent E2E tests

It's an antipattern 👎

Impossible to run tests in isolation 👎

A fake sense of the test's size 👎

Debugging the E2E tests is challenging

Small and dependent E2E tests

START (the application state is empty)

Test 1: create the entity

Test 2: modify the entity

Test 3: delete the entity

END (the application state is empty)

Debugging the E2E tests is challenging

Small and independent E2E tests

They can run in isolation 👍

It's a best practice 👍

Making them fast requires boilerplate 😓

Debugging the E2E tests is challenging

Making them fast is boilerplaty?

Every test should use the previous app state or create it ðŸ˜ą

Debugging the E2E tests is challenging

Small and independent E2E tests

1.  START (the application state is empty)

2.  Test 1: create the entity
    1.  BEFORE: Load the page (the application state is empty)
    2.  create the entity
    3.  AFTER: Delete the entity (the application state is empty)

3.  Test 2: modify the entity
    1.  BEFORE: create the entity (through APIs)
    2.  BEFORE: Load the page (the application state is empty)
    3.  modify the entity
    4.  AFTER: Delete the entity (through APIs, the application state is empty)

4.  Test 3: delete the entity
    1.  BEFORE: create the entity (through APIs)
    2.  BEFORE: Load the page (the application state is empty)
    3.  delete the entity
    4.  AFTER: Delete the action (the application state is empty)

5.  END (the application state is empty)

Debugging the E2E tests is challenging

Small and independent E2E tests

1.  START (the application state is empty)

2.  Test 1: create the entity
    1.  BEFORE: Does the entity exist?
        1.  NO: it's ok!
        2.  YES: delete the entity (through APIs)

    2.  BEFORE: Load the page (the application state is empty)
    3.  create the entity

3.  Test 2: modify the entity
    1.  BEFORE: Does the entity exist?
        1.  YES: it's ok!
        2.  NO: create the entity (through APIs)
    2.  BEFORE: Does the entity already includes the change the test is going to make?
        1.  YES: it's ok!
        2.  NO: modify the entity (through APIs)
    3.  BEFORE: Are we already on the correct page?
        1.  YES: it's ok!
        2.  NO: load the page
    4.  modify the entity

And so on...

Debugging the E2E tests is challenging

One long E2E test

It's long 👎

It's not modular 👎

Doesn't lie ðŸĪŠ

Debugging the E2E tests is challenging

One long E2E test

1.  START (the application state is empty)

2.  Test: CRUD
  1.  BEFORE: Delete the entity if it exists (the application state is empty)
  
  2.  BEFORE: Load the page
  
  3.  create the entity
  
  4.  modify the entity
  
  5.  delete the entity
  
  6.  AFTER: Delete the entity if it exists (the application state is empty)

3.  END (the application state is empty)

Debugging the E2E tests is challenging

The E2E nature is the problem...

Server-free tests are faster 👍

Application state problems do not exist 👍

Must be kept in sync with the server changes 😓 (TypeScript for the win!)

The server teams use the E2E tests also to test the server

The server teams use the E2E tests also to test the server

For every server change...

Does the Console's E2E work? Nice! 🎉

... Who said CI misconfiguration? 😓

Server and Console tests must be split!

Splitting server and Console tests

UI-free tests are faster 🚀

Server-free tests are faster 🚀

The server teams use the E2E tests also to test the server

A lot of work...

A lot of work...

Refactored some E2E tests...

Tracked and skipped all the flaky tests...

Asked the Feature teams to fix the tests...

A strong decision... 🙈

The future

More Storybook tests 🏎ïļ

TypeScript types generated by the server ðŸ˜ą

Playwright instead of Cypress 🏎ïļ

Stay tuned âĪïļ

Thank you âĪïļ

Stefano Magni

Front-end tech leader

(Platform team)

Hasura's Console E2E tests chronicles

By Stefano Magni

Hasura's Console E2E tests chronicles

I joined Hasura in May 2022, and one of my first tasks was to fix the E2E tests of the Hasura Console, main Hasura's front-end application. The main problems were: they were slow, and they were flaky. Then, by digging into the topic, there was more to say, more to decide, more to fix, and more to do. This presentation is about what we found out, the problems, the best practices, etc.

  • 2,040