Hello! My name is Vova, I’m a frontender in Tinkoff. Our team is responsible for two products for legal entities. I can say in figures about the size of the product: a complete regression of each product by two testers takes three days (without the influence of external factors).

The terms are significant and beg to fight them. Several ways to fight, the main ones:

  • Saw applications into smaller products with their release cycles.
  • Test coverage in accordance with with a test pyramid.

The last paragraph was the subject of my article.

image

Testing pyramid


As we know, there are three levels in the testing pyramid: unit tests, integration tests and e2e tests. I think many people are familiar with units, as well as with e2e, so I’ll dwell on integration tests in a bit more detail.

As part of integration testing, we test the operation of the entire application through interaction with the UI, however, the main difference from e2e tests is that we do not make real requests for backing. This is done in order to check only the interaction of all systems at the front in order to reduce the number of e2e tests in the future.

To write integration tests, we use Cypress. I won’t be in this article Compare it with other frameworks, I’ll just say why it turned out to be with us:

  1. Very detailed documentation.
  2. Easy debugging of tests (Cypress has a special GUI for this with time-travel steps in the test).

These points were important for our team, as we had no experience in writing integration tests and a very simple start was needed. In this article I want to talk about the path we have traveled, about which bumps we have filled, and to share recipes for implementation.

Beginning of the journey


At the very beginning, Angular Workspace with one application was used to organize the code. After installing the Cypress package, the cypress folder with the configuration and tests appeared in the root of the application, we stopped at this option. When trying to prepare the script in package.json necessary to run the application and run tests on top of it, we encountered the following problems:

  1. Some scripts that are not needed in integration tests were sewn up in index.html.
  2. To run the integration tests, you had to make sure that the server with the application was running.

The problem with index.html was solved through a separate assembly configuration — let's call it cypress — in which we specified custom index.html. How to implement this? We find your application configuration in angular.json, open the build section, add a separate configuration for Cypress there and do not forget to specify this configuration for serve-mode.

Example configuration for build:

"build": {  ...  "configurations": {    …//Другие конфигурации    "cypress": {      "aot": true,      "index": "projects/main-app-integrations/src/fixtures/index.html",      "fileReplacements": [        {          "replace": "projects/main-app/src/environments/environment.ts",          "with": "projects/main-app/src/environments/environment.prod.ts"        }      ]    }  } } 

Integrate with serve:

"serve": {  ...  "configurations": {    …//Другие конфигурации    "cypress": {      "browserTarget": "main-app:build:cypress"    }  } } 

From the main: for cypress configuration we specify aot assembly and replace files with environment - this is necessary to create prod-like assemblies during testing.

So, with index.html sorted out, it remains to raise the application, wait for the build to finish and run tests on top of it. To do this, use the start-server-and-test library and write scripts based on it:

 "main-app:cy:run": "cypress run",  "main-app:cy:open": "cypress open",  "main-app:integrations": "start-server-and-test main-app:serve:cypress http://localhost:8808/app/user/main-app:cy:run",  "main-app:integrations:open": "start-server-and-test main-app:serve:cypress http://localhost:8808/app/user/main-app:cy:open" 

As you can see, there are two types of scripts: open and run. Open mode opens the GUI of Cypress itself, where you can switch between tests and use time-travel. Run mode is just a test run and getting the final result of this run, great for running in CI.

Based on the results of the work done, we were able to get a starting frame for writing the first test and running it in CI.

Monorepository


The described approach has a very noticeable problem: if the repository has two or more applications, then the approach with one folder is not viable. And so it happened with us. But it happened in a rather interesting way. At the time of the introduction of Cypress, we moved to NX, and this handsome man out of the box makes it possible to work with Cypress.What is the principle of work in it:

  1. You have an application, for example main-app, the main-app-e2e application is created next to it.
  2. Rename main-app-e2e to main-app-integrations - you're amazing.

Now you can run integration tests with one command - ng e2e main-app-integrations. NX will automatically raise the main-app, wait for an answer, and run the tests.

Unfortunately, those who are currently using Angular Workspace, but it's okay, I have a recipe for you as well. We will use the file structure as in NX:

  1. Create the main-app-integrations folder next to your application.
  2. Create the src folder in it and put the contents of the cypress folder in it.
  3. Do not forget to transfer cypress.json (it will initially appear in the root) to the main-app-integrations folder.
  4. We correct cypress.json, specifying the paths to new folders with tests, plugins and auxiliary commands (integrationFolder, pluginsFile and supportFile parameters).
  5. Cypress can work with tests in any folders, the parameter is used to specify the folder
    project, therefore we change the command from cypress run/open to cypress run/open -–project./projects/main-app-integrations/src .

The solution for Angular Workspace is most similar to the solution for NX, except that we create the folder with our hands and it is not one of the projects in your mono-repository. Alternatively, you can directly use the NX builder for Cypress ( example of the NX repository with Cypress, you can see the final use of the nx-cypress builder there - focus on angular.json and the project
cart-e2e and products-e2e).

Visual Regressing


After the first five tests, we thought about screenshot testing, because, in fact, there are all the possibilities for this. I’ll say in advance that the word “screenshot-testing” causes great pain within the team, since the path to obtaining stable tests was not the easiest. Next, I will describe the main problems that we encountered and their solution.

The library cypress-image-snapshot was taken as a solution. The implementation did not take much time, and after 20 minutes we received the first screenshot of our application with a size of 1000 × 600 px. There was a lot of joy, because the integration and use were too simple, and the benefits could be enormous.

After generating five reference screenshots, we launched a test in CI, as a result, the build fell apart. It turned out that the screenshots created using the open and run commands, are different. The solution was quite simple: take screenshots only in CI mode, for this they removed screenshots in local mode, for example like this:

Cypress.Commands.overwrite(    'matchImageSnapshot',    (originalFn, subject, fileName, options) => {        if (Cypress.env('ci')) {            return originalFn(subject, fileName, options);        }        return subject;    }, ); 

In this solution, we look at the env parameter in Cypress, you can set it in different ways.

Fonts


Locally, the tests began to pass upon restarting, we are trying to run them again in CI. The result can be seen below:

ITKarma picture

It’s quite simple to notice the difference in the fonts in the diff screenshot. A reference screenshot was generated on macOS, and Linux was installed on the agents in CI.

Wrong decision


We picked up one of the standard fonts (like it was Ubuntu Font) that gave minimal pixel-by-pixel diff, and applied this font to text blocks (done in
index.html, which was intended only for cypress tests). Then increased the total diff to 0.05% and pixel by diff to 20%. With these parameters, we spent a week - until the first time when it was necessary to change the text in the component. As a result, the build remained green, although we did not update the screenshot. The current solution has proven futile.

The right decision


The original problem was in different environments, the solution in principle suggests itself - Docker. There are already ready-made docker images for Cypress. There are different variations of the images, we are interested in included, since Cypress is already included in the image and the Cypress binary will not be downloaded and unpacked every time (the Cypress GUI is launched via binary , and downloading and unpacking takes longer than downloading a docker image.
Based on the included docker image, we make our docker container, for this we made an integration-tests.Dockerfile file with similar contents:

FROM cypress:included:4.3.0 COPY package.json/app/COPY package-lock.json app/WORKDIR/app RUN npm ci COPY//app/ENTRYPOINT [] 

I would like to note the zeroing of ENTRYPOINT, this is due to the fact that it is set by default in the cypress/included image and points to the cypress run command, which prevents us from using other commands. We also split our dockerfile into layers so that you don't have to run npm ci every time you restart the tests.

Add the.dockerignore file (if it is not) to the root of the repository and we must specify node-modules/and */node-modules/in it.

To run our tests in Docker, we will write a bash script integration-tests.sh with the following contents:

docker build -t integrations -f integration-tests.Dockerfile. docker run --rm -v $PWD/projects/main-app-integrations/src:/app/projects/main-app-integrations/src integrations:latest npm run main-app:integrations 

Short description: we build our docker-container integration-tests.Dockerfile and point volume to the test folder so that we can get screenshots from Docker.

Again the fonts


After solving the problem described in the previous chapter, there was a lull in the builds, but about a day later we encountered the following problem (left and right screenshots of one component taken at different times):

ITKarma picture

I think the most attentive noticed that there is not enough heading in the popup. The reason is very simple - the font did not manage to load, because it was not connected through assets, but was on the CDN.

Wrong decision


Download fonts from CDN, drop them into assets for cypress configuration and in our custom
index.html for integration tests we connect them. With this decision, we lived a decent time until we changed the corporate font. There was no desire to crank the same story a second time.

The right decision


It was decided to start preloading all the necessary fonts for the test in
index.html for a cypress configuration, it looked something like this:

<link       rel="preload"       href="...."       as="font"       type="font/woff2"       crossorigin="anonymous"/> 

The number of test crashes due to fonts that did not have time to load decreased to a minimum, but not to zero: still, sometimes the font did not have time to load. The Cypress solution itself came to the rescue - waitForResource.
In our case, since the font preload was already connected, we simply redefined the visit command in Cypress, as a result, it not only navigates to the page, but also waits for the specified fonts to load. I would also like to add that waitForResource solves the problem of not only fonts, but also any loaded statics, such as images (because of them, screenshots also broke, and waitForResource helped a lot). After applying this solution, there were no problems with fonts and any downloadable statics.

Animations


It is with animations that our headache is connected, which remains to this day. At some point, screenshots will start appearing on which the element is animated, or a screenshot is taken before the animation begins. Such screenshots are unstable, and with each next comparison with the standard there will be differences. So which way did we go when solving the issue of animations?

First decision


The simplest thing that occurred to us at the initial stage: before creating a screenshot, stop the browser for a certain time so that the animations can complete. We went along the chain 100ms, 200ms, 500ms and as a result 1000ms. Looking back, I understand that this decision was initially terrible, but I just wanted to warn you against the same decision. Why terrible? The animation time is different, agents in CI can also play a bit sometimes, which is why any wait time for page stabilization from time to time was different.

Second solution


Even with a wait of 1 second, the page did not always manage to become stable. After a short review, we found a tool at Angular - Testability. The principle is based on monitoring the stability of ZoneJS:

Cypress.Commands.add('waitStableState', () => {    return cy.window().then(window => {        const [testability]: [Testability]=window.getAllAngularTestabilities();        return new Cypress.Promise(resolve => {            testability.whenStable(() => {                resolve();            }, 3000);        });    }); }); 

Thus, when creating screenshots, two commands were called: cy.wait (1000) and cy.waitStableState ().

Since then, there has not been a single randomly dropped screenshot, but let's calculate together how much time was spent idling the browser. Suppose you have 5 screenshots taken in the test, for each there is a stable waiting time of 1 second and some random time, suppose 1.5 seconds on average (I did not measure the average value in reality, so I took it from my head according to my own feelings). As a result, we spend an additional 12.5 seconds to create screenshots in the test. Imagine that you have already written 20 test scripts, where each test has at least 5 screenshots. We get that the overpayment for stability is ~ 4 minutes with the available 20 tests.

But this is not even the biggest problem. As discussed above, when running tests locally, screenshots are not chased, but in CI they are chased, and because of the expectations for each screenshot, the callbacks in the code worked, for example, at debounce Time, which already created randomization in the tests, because in CI and locally they passed in different ways.

Current Solution


Let's start with Angular animations. Our favorite framework during animation on the DOM element hangs the ng-animating class. This was the key to our decision, because now we need to make sure that the element does not have an animation class now. As a result, it resulted in such a function:

export function waitAnimation(element: Chainable<JQuery>): Chainable<JQuery> {    return element.should('be.visible').should('not.have.class', 'ng-animating'); } 

It seems nothing complicated, but it was this that formed the basis of our decisions. What you want to pay attention to in this approach: when taking a screenshot, you must understand the animation of which element can make your screenshot unstable, and add an assertion before creating the screenshot, which will verify that the element is not animating. But animations can also be in CSS. As Cypress himself says, any assertion on an element is waiting for the animation to finish on it - more here and here . That is, the essence of the approach is as follows: we have an animated element, add assertion to it - should ('be.visible')/should ('not.be.visible') - and Cypress will wait until the end animation on the element (perhaps, by the way, a solution with ng-animating is not necessary and only Cypress checks are enough, but for now we use the utility - waitAnimation).

As stated in the documentation itself, Cypress checks for a change in the position of an element on a page, but not all animations about changing a position, there are also fadeIn/fadeOut animations. In these cases, the solution principle is the same: check that the element is visible/not visible on the page.

When moving from the solution cy.wait (1000) + cy.waitStableState () to waitAnimation and Cypress Assertion, it took ~ 2 hours to stabilize the old screenshots, but as a result we got + 20-30 seconds instead of +4 minutes for the test execution time. At the moment, we carefully approach the review of screenshots: we check that they were not performed during the animation of the DOM elements and checks were added in the test for waiting for the animation. For example, we often add a “skeleton” display on the page until the data has loaded. Accordingly, the requirement immediately arrives at the review that when creating screenshots in the DOM, the skeleton should not be present, since there is an animation of smooth disappearance on it.

The problem with this approach is one: it is not always possible to foresee everything when creating a screenshot and it can still fall into CI. There is only one way to deal with this - to go and immediately edit the creation of such a screenshot, you cannot postpone it, otherwise it will accumulate like a snowball and ultimately you just turn off integration tests.

Screenshot size


You may have noticed an interesting feature: the default screenshot resolution is 1000 × 600 px. Unfortunately, there is a problem with the size of the browser window when starting up in Docker: even if you change the size of the viewport through Cypress, this will not help.We found a solution for the Chrome browser (for Electron, we could not quickly find a working solution, but the proposed one in this issue we did not start). First you need to change your browser to run tests on Chrome:

  1. Not for NX, we use the --browser chrome argument when running the cypress open/run command, and for the run command, specify the --headless parameter.
  2. For NX, in the project configuration in angular.json with tests, specify the browser: chrome parameter, and for the configuration that will run in CI, specify headless: true.

Now we make the changes in the plugins and get screenshots of 1440 × 900 px in size:

module.exports=(on, config) => {    on('before:browser:launch', (browser, launchOptions) => {        if (browser.name === 'chrome' && browser.isHeadless) {            launchOptions.args.push('--disable-dev-shm-usage');            launchOptions.args.push('--window-size=1440,1200');            return launchOptions;        }        return launchOptions;    }); }; 

Dates


Everything is simple here: if somewhere the date associated with the current is displayed, the screenshot taken today will fall tomorrow. Fixim just:

cy.clock(new Date(2025, 11, 22, 0).getTime(), ['Date']); 

Now the timers. We did not bother and use the blackout option when creating screenshots, for example:

cy.matchImageSnapshot('salary_signing-several-payments', {    blackout: ['.timer'], }); 

Flaky tests


Using the above recommendations, you can achieve maximum stability of the tests, but not 100%, because the tests are affected not only by your code, but also by the environment in which they run.

As a result, a certain percentage of tests will occasionally fall, for example, due to sagging agent performance in CI. First of all, we stabilize the test from our side: we add the necessary assertion before taking screenshots, but for the period of fixing such tests, you can use the retry of the fallen tests using cypress-plugin-retries.

We pump CI


In previous chapters, we learned how to run tests with one team and learned about working with screenshot testing. Now you can look in the direction of CI optimization. In our build it will be surely executed:

  1. The npm ci team.
  2. Raising the application in aot mode.
  3. Launch integration tests.

Let’s take a look at the first and second points and understand that similar steps are performed in your other build in the CI - build with the application assembly.
The main difference is not ng serve, but ng build. Thus, if we can get the already built application in the build with integration tests and raise the server with it, then we can reduce the execution time of the build with tests.

Why did we need this? We just have a large application and execution
npm ci + npm start in aot mode on the agent in CI took ~ 15 minutes, which in principle required a lot of effort from the agent, and integration tests were run on top of this. Suppose you have already written 20+ tests and on the 19th test your browser crashes, in which tests are run, due to the heavy load on the agent. As you know, restarting the build is again waiting for dependencies to be installed and the application to start.

Further I will talk only about scripts on the application side. You will need to solve the problem of transferring artifacts between tasks in CI yourself, so keep in mind that the new build with integration tests will have access to the assembled application from the task according to the build of your application.

Server with static


We need a ng serve replacement for raising the server with our application. There are many options, I'll start with our first - angular-http-server . There is nothing complicated in its configuration: we install the dependency, indicate in which folder our statics are located, indicate on which port to raise the application, and rejoice.

This solution was enough for us for the whole 20 minutes, and then we realized that we wanted to proxy some requests to the test circuit. Connect proxying for angular-http-server failed. The final solution was raising the server to Express . To solve the problem, express and express-http-proxy itself were used. We will distribute our statics using
express.static, the result is a script similar to this one:

const express=require('express'); const appStaticPathFolder='./dist'; const appBaseHref='./my/app'; const port=4200; const app=express(); app.use((req, res, next) => {    const accept=req        .accepts()        .join()        .replace('*/*', '');    if (accept.includes('text/html')) {        req.url=baseHref;    }    next(); }); app.use(appBaseHref, express.static(appStaticPathFolder)); app.listen(port); 

An interesting point here is that before listening to the baseHref route of the application, we also process all requests and look for the request for index.html. This is done for cases when the tests go to the application page, the path of which is different from baseHref. If you do not do this trick, then when you go to any page of your application, except the main one, a 404 error will arrive.Now add a pinch of proxy:

const proxy=require('express-http-proxy'); app.use(    '/common',    proxy('https://qa-stand.ru', {        proxyReqPathResolver: req => '/common' + req.url,    }), ); 

Let's take a closer look at what is happening. There are constants:

  1. appStaticForlderPath - the folder where the statics of your application is located.
  2. appBaseHref - your application may have baseHref, if not, you can specify ‘/’.

We proxy all requests starting with/common, and when proxing we keep the same path as the request, using the proxyReqPathResolver setting. If you do not use it, then all requests will simply go to https://qa-stand.ru.

Customization index.html


We needed to solve the problem with custom index.html, which we used when ng serve applications in Cypress mode. Let's write a simple script on node.js. The initial parameters were index.modern.html, we needed to turn it into index.html and remove unnecessary scripts from there:

const fs=require('fs'); const appStaticPathFolder='./dist'; fs.copyFileSync(appStaticPathFolder + '/index.modern.html', appStaticPathFolder + '/index.html'); fs.readFile(appStaticPathFolder + '/index.html', 'utf-8', (err, data) => {    const newValue=data        .replace(            '<script type="text/javascript" src="/auth.js"></script>',            '',        )        .replace(            '<script type="text/javascript" src="/analytics.js"></script>',            '',        );    fs.writeFileSync(appStaticPathFolder + '/index.html', newValue, 'utf-8'); }); 

Scripts


I really did not want to run npm ci of all the dependencies again to run the tests in CI (after all, this was already done in the task with the application build), so the idea came up to create a separate folder for all these scripts with my package.json. Let's name the folder, for example, integration-tests-scripts and drop three files there: server.js, create-index.js, package.json. The first two files were described above, now we will analyze the contents of package.json:

{  "name": "cypress-tests",  "version": "0.0.0",  "private": true,  "scripts": {    "create-index": "node./create-index.js",    "main-app:serve": "node./server.js",    "main-app:cy:run": "cypress run --project./projects/main-app-integrations ",    "main-app:integrations": "npm run create-index && start-server-and-test main-app:serve http://localhost:4200/my/app/main-app:cy:run"  },  "devDependencies": {    "@cypress/webpack-preprocessor": "4.1.0",    "@types/express": "4.17.2",    "@types/mocha": "5.2.7",    "@types/node": "8.9.5",    "cypress": "4.1.0",    "cypress-image-snapshot": "3.1.1",    "express": "4.17.1",    "express-http-proxy": "^1.6.0",    "start-server-and-test": "1.10.8",    "ts-loader": "6.2.1",    "typescript": "3.8.3",    "webpack": "4.41.6"  } } 

In package.json, there are only dependencies necessary for running integration tests ( with typescript support and screenshot testing) and scripts for starting the server, creating index.html and the well-known from the chapter on launching integration tests in Angular Workspace start-server-and-test.

Launch


We wrap the implementation of integration tests in the new Dockerfile - integration-tests-ci.Dockerfile :

FROM cypress/included:4.3.0 COPY integration-tests-scripts/app/WORKDIR/app RUN npm ci COPY projects/main-app-integrations/app/projects/main-app-integrations COPY dist/app/dist COPY tsconfig.json/app/ENTRYPOINT [] 

The bottom line is simple: copy and expand the integration-tests-scripts folder to the root of the application and copy everything that is needed to run the tests (this set may differ for you). The main differences from the previous file are that we do not copy the entire application inside the docker container, just the minimum optimization of the test execution time in CI.

Create the integration-tests-ci.sh file with the following contents:

docker build -t integrations -f integration-tests-ci.Dockerfile. docker run --rm -v $PWD/projects/main-app-integrations/src:/app/projects/main-app-integrations/src integrations:latest npm run main-app:integrations 

When the command with tests is run, package.json from the integration-tests-scripts folder will become the root and the main-app: integrations command will be launched in it. Accordingly, since this folder will expand to the root, the paths to the folder with the statics of your application should be indicated with the idea that everything will be launched from the root, and not from the integration-tests-scripts folder.

I also want to make a small remark: I called the final bash script for running integration tests as it evolved differently. You don’t need to do this, it was done only for the convenience of reading this article. You should always have one file left, for example integration-tests.sh, which you are already developing. If you have several applications in the repository and their preparation methods differ, you can either either variables in bash , or different files for each application - depends on your needs.

Summary


There was a lot of information - I think now it’s worth summing up based on the above.
Preparation of tools for local writing and running tests with a pinch of screenshot testing:

  1. Add dependency for Cypress.
  2. Preparing the test folder:
    1. Angular Single Application - leave everything in the cypress folder.
    2. Angular Workspace - create a folder the name of the integrations application next to the application that the tests will be chasing, and transfer everything from the cypress folder to it.
    3. NX - rename the project from the name of the application-e2e to the name of the application-integrations.

  3. Custom cypress-assembly for raising the application - we do the Cypress configuration in the build-section, specify aot there, a substitution for its own index.html, substitution of the environment file for the prod file and specify the Cypress assembly in the serve section (this item is necessary if you need any differences from the prod assembly, otherwise you can skip).
  4. Scripts for running and running tests:
    1. Angular Single Application - we prepare a script with serve-th of cypress-assembly and start of tests, we combine all this with start-server-and-test.
    2. Angular Workspace - similar to Angular Single Application, just specify the path to the tests when running cypress run/open.
    3. NX - run the tests using the ng e2e commands.

  5. We mix screenshot testing:
    1. Add a cypress-image-snapshot dependency.
    2. We redefine the standard command for comparing screenshots to run it only in CI.
    3. In tests, we don’t take screenshots on the random. If the screenshot is preceded by an animation, be sure to wait for it - for example, add Cypress Assertion to the animated element.
    4. We’ll mokate the date via cy.clock or use the blackout option when taking a screenshot.
    5. We expect any loaded statics in runtime through the custom cy.waitForResource command (pictures, fonts, etc.).

  6. Wrap it all up in Docker:
    1. Cooking Dockerfile.
    2. Create a bash file.


Run tests over the assembled application:

  1. In CI, we learn to throw artifacts of the assembled application between builds (it remains on you).
  2. Preparing the integration-tests-scripts folder:
    1. A script to raise the server for your application.
    2. A script to change your index.html (if you are satisfied with the original index.html, you can skip it).
    3. Add to the package.json folder with the necessary scripts and dependencies.
    4. Preparing a new Dockerfile.
    5. Create a bash file.

Useful links


  1. Angular Workspace + Cypress + CI - in this example, I created a base for the Angular Workspace application with wrapped CI based on the scripts described in article (without typescript support).
  2. Cypress - pay attention to the section trade-offs.
  3. Start-server-and-test - start the server, wait for a response and run tests.
  4. Cypress-image-snapshot - a library for screenshot testing.
  5. Cypress recipes - ready-made developments on Cypress, so as not to reinvent their bikes.
  6. Flaky Tests - article about unstable tests, at the end there is a link on an article from Google.
  7. Github Action - Cypress allows you to use predefined configs for GitHub Action, there are a lot of examples in README, from the interesting - built-in wait support -on. There is also a docker example.
.

Source