Hello, my name is Dmitry Karlovsky and I... love to drive all kinds of strange game. Caution, after this report you may have a strange but irresistible desire to remove all unit and e2e tests from your project, because they require a lot of resources, but give little profit.


ITKarma picture


This is a transcript of the speech at TechLead Conf 2020 . You can watch the video , read as an article or open in the interface presentations .


About Me


  • In programming for 25 years
  • The last 15 years are mostly front
  • Large and small projects
  • Developed a framework for the future

Path in testing


  • I didn’t write tests
  • Avoided tests
  • Left without testers
  • I had to figure it out myself
  • Stuffed a bunch of cones
  • Learned Zen
  • I want to share with you

Why tests? Quick feedback!


ITKarma picture


The sooner you run the tests, the shorter the debugging cycle and the less negative from those involved. And the smaller the cycle, the faster it is, and therefore the delivery of new features occurs earlier. Users are happy that nothing is falling. Superiors can more accurately plan dates. And colleagues are sure that you write code without bugs, although in fact they are full, but they simply do not live to see the review.


Why tests? Speeding up code writing!


  • The test will have to be written anyway
  • But restarting it faster than checking it with your hands

The sooner you write automated tests, the more time you save on manual testing.


Why tests? Localization of defects!


ITKarma picture


A good test points to a specific problem location.


Why tests? Actual documentation!


  • Separate documentation always lies
  • Even if it's a comment next to the code

In D, for example, there is a cool feature - the ability to write a test right next to the code . And then documentation is generated from this code, where the tests are given as code examples.


Ideal tests: capture only external behavior


Check everything important, but do not record the unimportant.


ITKarma picture


In watches, for example, it’s important for us to show the exact time, but we don’t care how the gears are connected in them or whether there are gears at all.


Ideal tests: balance between number and coverage


ITKarma picture


Their minimum possible number to cover all the necessary test cases.


Ideal tests: quick and easy


ITKarma picture


Running fast and easy to write.


Terms


  • Different things are often referred to by the same name
  • The same thing is often called by different names

Let's sync terminology to further speak the same language.


System (application)


System is the unit of delivery, your product. This is what you deploy to the server, run on the user’s machine, etc.


ITKarma picture


A system test is called a "system test" or "E2E test."


Module (unit)


A module is a unit of code. This is usually a file or even part of this file.


ITKarma picture


When you isolate a piece of code and write a test for it, such a test is called a "unit test" or "unit test."


Component (subsystem)


Component - a unit of functionality. It consists of a module, which is an entry point, and all its dependencies.


ITKarma picture


A component test is called a "component test." You can also come across the term “social unit test”, but this is some kind of oxymoron, since the component test is a special case of integration testing of a group of modules, rather than one module.


Testing horn


When talking about types of testing, they usually show a negative situation, where a little time is spent on unit tests, but a lot on manual ones.


ITKarma picture


This is not only slow. So also because of the human factor, it misses a lot of defects.


Testing pyramid


The horn is usually offered to be turned over to obtain a pyramid, where long and complex testing is minimized, and simple and fast is maximized.


ITKarma picture


Glass of testing


In real projects, it often turns out to be something in between, which has absorbed all the disadvantages of both approaches.


ITKarma picture


Developers write unit tests. There is already not enough of them for integration. Autotests write system tests with a strong delay. And you need to release it yesterday, so thorough manual testing of each release does not go anywhere.


Unit tests: Break abstractions


Take a simple function. It abstracts the consumer from its internal complexity and, in particular, from its dependencies.


sum( 1, 2 ) function sum( a, b ) { logger.trace( a, b ) return algebra.apply( '+', a, b ) } 

But in order to lock these dependencies, you have to put them into the public interface, turning the guts out.


new Sum( algebra, logger ).exec( 1, 2 ) class Sum { constructor( private algebra, private logger, ) {} exec( a, b ) { logger.trace( a, b ) return algebra.apply( '+', a, b ) } } 

Unit tests have a rather strong (and by no means always positive) effect on the architecture and the actual code.


Unit tests: Fragile


ITKarma picture


You have changed the interface of the CDMY0CDMY module. Now you need to not only conditionally change 10 of his tests, but also 10 mobs in the unit tests of dependent modules. And accordingly, 10 tests of these modules themselves, although their contract has not changed at all.


Test uselessness criteria


Therefore, we can formulate a criterion for the uselessness of the test. If:


  • Desired behavior has not changed
  • The test has fallen

That is a useless test, it captures unimportant behavior and will often break. And vice versa, if:


  • Desired behavior has changed
  • The test did not fail

Then this test does not check what it should, even worse.


Unit tests: Incomplete


The smaller and simpler the modules, the more important their interaction. Therefore, you need to write at least as many integration tests as possible to test the interaction.


ITKarma picture


System tests: Not scalable


As the application grows, not only the number of tests grows, but also the application launch time, which gives a quadratic dependence of the run time of all tests on the complexity of the application. That is, it is very easy to achieve a situation where the system test run takes several days on the server farm.


ITKarma picture


So, we tested this button, we land the plane, and on a new approach to test the next one.


System tests: Fragile


ITKarma picture


The fall of a trifle from the edge can easily bring you half of the tests. And go then look who is to blame.


Integration Tests: Slow


They pick up a few modules. And they do real work, which can be slow. In addition, the trouble may be in your test framework. For example, once we noticed that any, even the most trivial component test in Angular takes 100ms. They began to sort it out and found out that all this time was spent on initializing TestBed .


ITKarma picture


Cutting out this initialization accelerated our tests by 10 times.


Integration Tests: Fast


Integration tests can be quick, but you need to.


  1. Lazy application architecture
  2. Quick start of tests

Moreover, integration tests can be even faster than unit tests, since the tested code with real dependencies is better optimized by the JIT compiler. But disposable mokas are practically not optimized.


Component tests: Safety net


Component tests can hedge you when you under-test something somewhere.


ITKarma picture


Here the tests of the CDMY1CDMY module revealed a defect in the CDMY2CDMY module. Of course, you still have to look for it, but this is better than not knowing it at all (as in the case of unit tests) or finding out too late (in the case of system tests).


Component tests: Where is the error?


Reverse side - a defect in one module can make half of all tests redden and it can be difficult to figure out where to start looking for a defect.


ITKarma picture


Fractal testing


Here we come to the idea of ​​fractal testing, the idea of ​​which is to abandon unit and system tests, leaving only integration tests. At the same time, write them in the form of component tests that form a fractal structure, where higher-level tests rely on the fact that lower-level tests have already checked everything, which means there are probably no defects there.


ITKarma picture


Fractal testing: Here is the error!


If you run component tests not randomly, as they usually do, but in order from less dependent to more dependent, then the first test that fails will point to the module in which the defect is likely to creep in, and the rest of the tests may not be to fulfill.


ITKarma picture


For example, I wrote the MAM collector, which builds tests and styles in that in the same order as the code in the bundle. This allows me to constantly keep the debugger in stop mode on exceptions. And if any test crashes, the debugger immediately stops in the right context where the failed module is being tested.


Fractal Testing: The Shortest Way


You can go even further if running all the tests at all becomes a burden for you and restarting only those tests that depend on the changed code.


ITKarma picture


Here, the changed code is marked in red. Green - tests that do not make sense to restart. And orange - tests that need to be restarted. And again, in order from less dependent to more dependent.


Fractal testing: Permanent complexity


At each level, we have direct access to all the handles of the component, which is not available, for example, during system testing. Moreover, you only need to test the functionality added by him, relying on the fact that the level below has already been tested and works correctly.


ITKarma picture


The logic is exactly the same as with the standard library or external dependencies - we use them, but we don’t get wet and we don’t test, because we are confident in their performance. You must also write your code in such a way that there is such confidence.


Fractal testing: Easy to write


We just use components as in real code, without any moks. Here is an example of a high-level component test.


const app=new Todomvc({ context }) const title=guid() const rowsPrev=app.rows() app.NewTitle().value( title ) app.NewSubmit().click() assertEqual( app.rows()[0].title(), title ) assertEqual( app.rows().slice(1), rowsPrev ) assertEqual( app.NewTitle().value(), '' ) 

Here we create an instance of the application, providing it with the context of the environment that the test framework gave us. In this context, all external and non-deterministic dependencies are already locked, which allows you to run the application as if in a sandbox. Next, we add the task through the form and verify that it was correctly added, and that the form then cleans itself.


This is an extremely simple code that even June can write. And he checks exactly the behavior that is important to us, without going into the rendering details.


Fractal testing: isolation levels


The test given earlier is written once and can be run with different isolation levels by changing the context of the environment.


  1. No browser under NodeJS
  2. Without a server in different browsers
  3. With a test server
  4. Generally on prod

And we don’t need to write different types of tests for this: from unit to system.


Fractal testing: Quality


We got a pretty high degree of test coverage.


  • All modules are tested
  • All relationships are tested
  • Each level has all the necessary pens
  • Including the complete system
  • Including in different environments

Fractal testing: Simple support


And support, on the contrary, has been greatly simplified.


  • In quantity you need less than modular
  • Code is simpler than modular
  • The execution speed is comparable to modular
  • Fall only when changing meaningful behavior

Comparison of labor input


Comparing the traditional pyramid with the fractal approach, you can observe the following picture.


ITKarma picture


On the pyramid, you need to write a lot of different tests, and then rewrite a significant proportion of unit tests with any more or less serious refactoring.With fractal code testing, however, you need to write significantly less, and even with refactoring, you just need to fix the tests of the changed contracts and that’s all.


So stop rotating the testing horn already. You just need to throw it away. And it's better to write component tests in a fractal style.


Proximity to the ideal


In total, we came quite close to the ideal voiced at the beginning.


Criterion Reached?
Captures only external behavior +
Balance between number and reach +
Writing fast and easy + *

The last point is also achievable with the correct application architecture.


My project


I will give a few observations of their practice on the example of my working draft.


  • Saw a solo second year
  • A very complicated web widget
  • There are less than 5 competitors
  • Refactoring every month
  • 300 component tests

Even such a small number of tests is enough to ensure a decent level of quality.


Run tests at application launch


All the tests I have run in about half a second, which allows you to run them right at the start of the application and get the fastest possible feedback.


ITKarma picture


Other people's experience


Vasily Malykhin recently talked about the evolution of testing in their company. At first they wrote a lot of unit tests, and then they started to count and realized that they had little profit, and that they required a lot of expenses for their support.



Well, Kent Beck, who is considered the father of TDD, clearly says that he is trying to get as little as possible in his "unit" tests, that is, in fact, he writes component tests.


Restrictions: Monolithic architecture


ITKarma picture


To use fractal testing you need to break your monolith into components, those into smaller components and so on to indivisible components.


Restrictions: Heavy architecture


ITKarma picture


If you run a component test in time comparable to launching the entire application, then it may be easier for you to pick it up once and then run system tests on it. Only tests for this will need to be written in a special way under the assumption that the condition can be initially dirty. I gave an example of such a test earlier.


Limitations: No control inversion


ITKarma picture


Even in component tests, you have to get wet things like random or the outside world. Therefore, it is important to have a mechanism that allows you to lock any low-level thing at the upper level. That is, an inversion of control is needed. There are various patterns of its implementation: from singleton to the context of the environment.


Limitations: High cost of error


ITKarma picture


Finally, if you have a bunch of resources to ensure quality, then you can afford unit, integration, system, and mutation testing. And even triple duplication of code written in different languages ​​by different people in different paradigms.


Want more?



Reviews


  • 1 - Nothing new.
  • 2 - For some reason, it seemed that the author advertises himself, and does not talk about useful things to others. Also, the steepness of the application exceeded expectations. But there was nothing to “put out”.
  • 3 - I appreciate the fact that the speaker came up with his tried-and-tested idea, but, in my opinion, this does not apply.
  • 3 - Simple material, long introduction.
  • 3 - I do not understand why the new name?
  • 4 - An interesting approach, worthy of attention. I even recorded this report myself and reviewed it 2 times.
  • 4 - I liked the idea, but self-promotion is a bit overkill.
  • 5 - A controversial speaker, but the report is quite interesting, it is worthwhile to delve deeper into the topic.
  • 5 - The report was interesting and applicable.
.

Source