In one of the old projects, assertions from JUnit, kotlin.test, and AssertJ were piled in a heap. This was not his only problem: he was generally written as a letter from Uncle Fedor, and there was no time to stop and bring to a single look. And now this time has come.


The article will be a mini-study about what assertions are best by subjective criteria. At first I wanted to do something simple: to throw a set of tests in order to quickly rivet options with copy-paste. Then he selected the general test data, automated some checks, and how everything went... The result was a small Rosetta stone, and this article may be useful to you in order to choose an assert library that suits your realities.


I’ll make a reservation right away that the article will not compare testing frameworks, testing approaches and any tricky approaches to data verification. It will be about simple assertions.


ITKarma picture


If you are too lazy to read boring arguments, the history of my ordeals and other details, then you can go straight to the comparison results .


A bit of background


For a long time, my main PL was Scala, and the test framework was ScalaTest. Not that it was the best framework, but somehow I got used to it, so my opinion may be distorted because of this.


When at the old work they started writing in Kotlin , after a while they even made their own library, imitating the behavior of Skalov match-ups, but now it makes little sense after the appearance of Kotest (although it was better implemented to compare complex structures, with the recursive output of a more detailed message).


Requirements


I’ll make a reservation right away, the requirements are very subjective and biased, and part of it is generally taste. The requirements are as follows:


  1. Seamless integration with Kotlin and IntelliJ Idea. Scala libraries by this principle disappear - there is no desire to pervert with the setting of two runtimes. Despite this, ScalaTest will be present in the comparison as a starting point, simply because it worked a lot. By integration with IntelliJ I mean the ability to click on CDMY0CDMY to see a comparison of the real value and the expected one. This feature, generally speaking, works in the guts of IntelliJ Idea - but the developers of the Kotlin libraries probably heard about it and can solve this problem, right?
    ITKarma picture
  2. The ability to quickly understand the problem. That there was not CDMY1CDMY and stackrace, but a normal message, ideally containing the name of the variable and the division into "expected" and "actual". So that for the collections there is a message not just "two sets of 100 elements are not equal, here you are both in a string representation, look for the difference yourself", and details, for example, "... these elements should be, but they should not be, but these should not, but they are". Of course you can write descriptions everywhere yourself, but then why do I need a library? As it turned out a little later, the name of the variable was a wet dream, and with some thought, it will be obvious that this is not so easy to do.
  3. Adequacy of the record. CDMY2CDMY - Yoda style is difficult for me to read, but taste is great. In addition, I don’t want to think about the subtle nuances of the library - ideally there should be a limited set of keywords/designs, and so that you don’t have to recall the features from the series “this is an array, not a collection, so it needs a special method” or remember that the string is not CDMY3CDMY, but CDMY4CDMY. In other words, it is both readability and obviousness both when reading and when writing tests.
  4. Existence of a substring content check. Something like CDMY5CDMY.
  5. Exception checking. As a counter-example, I’ll give JUnit4, in which an exception is caught either in the annotation or in a variable of type CDMY6CDMY with annotation CDMY7CDMY.
  6. Comparison of collections and content of element (s) in them.
  7. Negation support for all of the above.
  8. Type checking. If the error is detected by the compiler, then this is much cooler than if it is detected when the test starts.At a minimum, typing should not interfere: if we know the type of the expected value, then the type of the real value returned by the generic function must be inferred. Counter example: CDMY8CDMY. CDMY9CDMY is superfluous here. The third option is to ignore types when calling an assert.
  9. Comparison of complex structures, for example, two dictionaries with nested containers. And even if an array of primitives is embedded in them. Still, they know about the inconsistency of their comparison ? So assertions are the last place where I want to think about it, even if it differs from behavior in runtime. For complex structures, there should be a recursive traversal of the object tree with useful information, rather than a dumb call to equals. By the way, in that under-library at the old work it was done.

We will not consider complex behavioral patterns or very complex checks where it is necessary to directly check at the CDMY10CDMY object that he is both a Swiss and a reaper, and plays the dude, and generally a good guy in the right place and time. I repeat that the goal is to choose assertions, and not a test framework. And a reference to the developer who writes unit tests, and not to a full-fledged QA engineer.


Contestants


Before writing this article, I thought that it would be enough to compare pieces of 5 libraries, but it turned out that these libraries are a dime a dozen.


I compared the following libraries:


  1. ScalaTest - as a reference point for me.
  2. JUnit 5 - as a reference point for a spherical Java developer.
  3. kotlin.test - for fans of multiplatform and official solutions. For our purposes, this is a wrapper over JUnit, but there are nuances.
  4. AssertJ is a fairly popular library with a rich assortment. Separated from FestAssert, on whose website Japanese sesame is traded for all old links to documentation.
  5. Kotest - aka KotlinTest, not to be confused with kotlin.test. The developers write that they were inspired by ScalaTest. In the guts there are even generated functions and classes for 1-22 arguments - in the best tradition of scala.
  6. Truth - a library from Google. According to the creators themselves, it is very similar to AssertJ.
  7. Hamсrest is a favorite of many auto-testers according to Yandex . On top of it is still working valid4j .
  8. Strikt - AssertJ owes a lot to it and also reminds him of its style.
  9. Kluent - the author writes that this is a wrapper over JUnit (although in fact - over kotlin.test), similar in style to Kotest. I liked the documentation - a bunch of examples by category, no walls of text.
  10. Atrium - according to the creators, they drew inspiration from AssertJ, but then got on their way. An original feature is the localization of assert messages (at the import level in maven/gradle).
  11. Expekt - drew inspiration from Chai.js. Project abandoned: last commit - 4 years ago.
  12. AssertK - like AssertJ, only AssertK (but there are nuances).
  13. HamKrest - like Hamсrest, only HamKrest (in fact, only name and style from Hamcrest).

If you want to violate the great number of these contestants, write in the comments what comparison library I missed or do pull request .


Evolution of the assessment methodology


When I already wrote 80 percent of the article and added the less well-known libraries, I came across the repository , where There is a comparison approximately in the form that I thought initially. Maybe someone there will be easier to read, but there are fewer contestants.


First, I created a set of tests that always fall, 1 test - 1 assert. Despite the fact that I like to automate any game , writing some kind of complicated crap to test the requirements was frankly lazy for me, so I planned to check everything almost manually and insert a joke" the very place where you can slap a minus "low technical level of material" ".


Then he decided that he still needed to protect himself from commonplace mistakes, and using JUnit did validator , which checks that all the tests that were supposed to fail, fail, and that there are no unknown tests. When I came across a bug in ScalaTest , I decided to make two variations: one, where all the tests pass, the second is where nothing passes and the validator added. An attentive reader may ask: who cuts the barber and what assertions are used there? Partly for objectivity, partly for portability assertions are not there at all :). At the same time there will be a demo/argument for those who believe that assertions are not needed at all.


Then I was at a crossroads: whether or not to carry out constants like CDMY11CDMY? If yes, then this is some kind of tenacity; if not, then I will definitely be mistaken when rewriting the test for other assertions. Having compiled a list of libraries that should be checked for a claim of completeness, I spat and decided to solve this problem by inheritance: I wrote a common skeleton for all tests and made interface for assets , which must be redefined for each library. It looks a bit scary, but it can be used as a Rosetta stone.


However, there is a problem with parameterized checks and type erasure. Reified parameters can only be in inline-functions, and they cannot be redefined. Therefore, it’s good to use constructions like


assertThrows<T>{...} 

in the code does not work, I had to use their additions without the reified parameter:


assertThrows(expectedClass){...} 

I honestly got a little deeper into this problem and decided to score on it. In the end, kotlin.test has a similar issue with the Asserter : assertion for checking exceptions is not included in it, and is an external function . Why should I show off if the creators of the language have the same problem? :)


All the resulting code can be found in the GitHub repositories . After wrapped up abstractions, I left the option with ScalaTest as is, and put it in a separate folder as a separate project.


Results


Further, we stupidly summarize the points according to the requirements: 0 - if the requirement is not fulfilled, 0.5 - if partially fulfilled, 1 - if everything as a whole is ok. Maximum - 9 points.


In a good way, you need to arrange the coefficients so that more important features have more weight. And on some points so generally give fractional estimates. But I think that for everyone these weights will be different. At the same time, at least somehow I had to sort the plate so that the “rather good” libraries were at the top, and the “rather not so” libraries were at the bottom. So I left a dumb amount.


Library Integration Description of the error Readability Substring Exceptions Collections Denials Type inference Complex Structures Total
Kotest ± ± + + + + + no - 6.0
Kluent ± ± + + + + + no - 6.0
AssertJ ± + ± + ± + + no ± 6.0
Truth ± + + + - + + no - 5.5
Strikt ± ± ± + + + + no - 5.5
ScalaTest ± ± ± + + + + no - 5.5
HamKrest ± - ± + + ± + yes - 5.5
AssertK ± ± ± + ± + + no - 5.0
Atrium ± ± ± + + ± + no - 5.0
Hamcrest ± ± ± + - ± + yes - 5.0
JUnit + + - ± + - ± ignore - 4.5
kotlin.test + ± - - + - - yes - 3.5
Expekt ± ± - + - ± + no - 3.5

Notes for each library:


Kotest
  • To connect only assertions, you need to dig deeper a bit. In my opinion, it’s hard to understand right away.
  • It doesn’t have the option of catching an exception with an explicit parameter, not reified, but this is not really what you need: few will deal with such perversions.
  • Complex structures: the test with nested arrays failed. I got this ticket .
  • Integration: CDMY12CDMY is only for simple assertions.
  • Typing: sometimes when using generics, you need to write them an explicit type.
  • Description of the errors: almost perfect, lacking only details of the difference between the two sets.

Kluent
  • You can write both CDMY13CDMY and CDMY14CDMY. DSL lovers will love it.
  • Interesting entry for catching exceptions:
    invoking { block() } shouldThrow expectedClass.kotlin withMessage expectedMessage 
  • The error descriptions are generally excellent, there are no details of the difference between the two collections, which is not so. I was also upset by the error in the CDMY15CDMY format - it is not clear what the verifiable Iterable actually contains.
  • Integration: CDMY16CDMY is only for simple assertions.
  • Complex structures: the test with nested arrays failed.

AssertJ
  • An impressive number of methods for comparison - still remember them... You need to know that lists need to be compared through CDMY17CDMY, sets through CDMY18CDMY, and dictionaries through CDMY19CDMY.


  • Integration: CDMY20CDMY is only for simple assertions.


  • Exceptions: just some kind of exception is caught, not a specific one. The error message, accordingly, does not contain the class name.


  • Complex structures: there is CDMY21CDMY, which compares almost well. However, I would like to have a more detailed error: assert determines that the values ​​for one key are not equal, but does not say which one. And despite the fact that he correctly determined that the two dictionaries with arrays are equal, the denial of this assertion


    assertThat(actual) .usingRecursiveComparison() .isNotEqualTo(unexpected) 

    didn’t work correctly: for the same structures, the test for their inequality did not fail.


  • Typing: sometimes when using generics, you need to write them an explicit type. Apparently, this is a limitation of DSL.



Truth
  • Detailed error messages, sometimes even verbose.
  • Exceptions: not supported, write that CDMY22CDMY from JUnit5. Interestingly, if assers do not run through JUnit, then what?
  • Readability: except for a joke with an exception, a strange name for a method that checks for the presence of all elements in a collection: CDMY23CDMY. But I think this is insignificant against the general background, since here you can compare collections without thinking to write CDMY24CDMY.
  • Integration: CDMY25CDMY for primitive assert only.
  • Complex structures: the test with nested arrays failed.
  • Typing: the type is not inferred from the expected value, I had to write explicitly.
  • Fun Stektreys with shorteners links "to improve readability":
    expected: 1 but was : 2 at asserts.truth.TruthAsserts.simpleAssert(TruthAsserts.kt:10) at common.FailedAssertsTestBase.simple assert should have descriptive message(FailedAssertsTestBase.kt:20) at [[Reflective call: 4 frames collapsed (https://goo.gl/aH3UyP)]].(:0) at [[Testing framework: 27 frames collapsed (https://goo.gl/aH3UyP)]].(:0) at java.base/java.util.ArrayList.forEach(ArrayList.java:1540) at [[Testing framework: 9 frames collapsed (https://goo.gl/aH3UyP)]].(:0) at java.base/java.util.ArrayList.forEach(ArrayList.java:1540) at [[Testing framework: 9 frames collapsed (https://goo.gl/aH3UyP)]].(:0) at java.base/java.util.ArrayList.forEach(ArrayList.java:1540) at [[Testing framework: 17 frames collapsed (https://goo.gl/aH3UyP)]].(:0) at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)... 

Strikt
  • It doesn’t have the option of catching an exception with an explicit parameter, not reified, but this is actually not really necessary: ​​few will deal with such perversions.


  • Negation for the contents of a substring in a string does not look consistent: CDMY26CDMY, although there is a normal CDMY27CDMY for collections.


  • Readability: you need to use CDMY28CDMY for arrays. Same problem with denial: CDMY29CDMY. Moreover, we still have to think about the type, because for CDMY30CDMY Strikt for some reason could not determine the desired assert itself. For lists - CDMY31CDMY, for sets - CDMY32CDMY.


  • Typing: sometimes when using generics, you need to write them an explicit type. Moreover, for arrays you still need to choose the right type with variation. Take a look at this piece of code:


    val actual: Array<String>=arrayOf("1") val expected: Array<String>=arrayOf("2") expectThat(actual).contentEquals(expected) 

    It will not compile because the compiler cannot determine the overload for CDMY33CDMY. This is because the desired CDMY34CDMY is defined with a covariant type:


    infix fun <T> Assertion.Builder<Array<out T>>.contentEquals(other: Array<out T>) 

    Because of this, you have to write


    val actual: Array<out String>=arrayOf("1") val expected: Array<String>=arrayOf("2") expectThat(actual).contentEquals(expected) 

  • Integration: no CDMY35CDMY.


  • Description of the error: there are no details for the dictionary and arrays, but in general it’s pretty detailed.


  • Complex structures: the test with nested arrays failed.



ScalaTest
  • Integration: when comparing collections, you cannot open the comparison.
  • Description of the error: it is written in the collections that they are simply not equal. There are no details for the dictionary either.
  • Readability: keep in mind the features of DSL for negation and CDMY36CDMY, the difference between CDMY37CDMY and CDMY38CDMY, as well as the need for CDMY39CDMY.
  • Complex structures: the test with nested arrays did not pass, there is a ticket .
  • Typing: the type is not inferred from the expected value, I had to write explicitly.

HamKrest
  • The project, judging by the tickets, is in a half-abandoned state.In addition, the documentation is very thin - I had to read the source code of the library to guess the name of the right mater.
  • I expected that it was enough to change Hamcrest imports, but it wasn’t there: quite a lot is different here.
  • Exception catch record - furious:
    assertThat( { block() }, throws(has(RuntimeException::message, equalTo(expectedMessage)))) 
  • Collections: No check for multiple items. Pull request hangs for 3.5 years. I wrote this: CDMY40CDMY.
  • No support for arrays.
  • Complex structures: the test with nested arrays failed.
  • Integration: no CDMY41CDMY.
  • Description of the error - somehow about nothing:
    expected: a value that not contains 1 or contains 3 but contains 1 or contains 3 

AssertK
  • As you might guess from the name - almost everything coincides with AssertJ. However, the syntax is sometimes a little different (there are no some methods, some methods are called differently).


  • Readability: If CDMY42CDMY is written in AssertJ, then in AssertK the same construction will not work correctly, because CDMY43CDMY accepts CDMY44CDMY in it. It is clear that the goal is on CDMY45CDMY, but it would be worth considering an alternative. Some other libraries have a similar problem, but they cause a compilation error in them, but not here. Moreover, the developers know the problem - this is one of the first tickets . In addition, you need to distinguish between CDMY46CDMY and CDMY47CDMY.


  • Integration: no CDMY48CDMY.


  • Exceptions: just some kind of exception is caught, not a specific one, then its type must be checked separately.


  • Complex structures: there is no analogue of CDMY49CDMY.


  • Descriptions of errors - there are details (although not everywhere), but in some places strange:


    expected to contain exactly:<[3, 4, 5]> but was:<[1, 2, 3]> at index:0 unexpected:<1> at index:1 unexpected:<2> at index:1 expected:<4> at index:2 expected:<5> 

    That's why there are two posts on the first index?


  • Typing: sometimes when using generics, you need to write them an explicit type.



Atrium
  • Available in two styles: fluent and infix. I was expecting differences like CDMY50CDMY versus CDMY51CDMY, but no, it's CDMY52CDMY versus CDMY53CDMY. In my opinion, a very dubious difference, given the fact that the infix method can be called without "infixness". However, for infix recording, sometimes you need to use the placeholder CDMY54CDMY: CDMY55CDMY. It seems to be explained why so , but It looks weird. Although on average in the hospital I like infix assertions (helicopters because of the rocky past), for Atrium I wrote in fluent-style.
  • Readability: strange negatives: CDMY56CDMY, but CDMY57CDMY. But this is not critical. I had to think about how to check for the presence of several elements in the collection: CDMY58CDMY accepts CDMY59CDMY, and CDMY60CDMY cannot deduce the type, made a dumb cast. It is clear that the goal is on CDMY61CDMY, but it would be worth considering an alternative. The denial of multiple items is written as CDMY62CDMY.
  • Array support
  • It doesn’t have the option of catching an exception with an explicit parameter, not reified, but this is not really what you need: few will deal with such perversions.
  • Complex structures: the test with nested arrays failed.
  • Integration: no CDMY63CDMY.
  • Description of the error: in places there are no details (when comparing dictionaries, for example), in places the description is rather confusing:
    expected that subject: [4, 2, 1] (java.util.Arrays.ArrayList <938196491>) ◆ does not contain: ⚬ an entry which is: 1 (kotlin.Int <211381230>) ✘  number of such entries: 1  is: 0 (kotlin.Int <1934798916>)   has at least one element: true  is: true 
  • Typing: sometimes when using generics, you need to write them an explicit type.

Hamcrest
  • Readability: strange syntax for negatives (either


    assertThat(actual, `is`(not(unexpected))) 

    either


    assertThat(actual, not(unexpected)) 

    You need to know the nuance of CDMY64CDMY vs CDMY65CDMY vs CDMY66CDMY vs CDMY67CDMY. I had to think about how to check for the presence of several elements in the collection: CDMY68CDMY accepts CDMY69CDMY, and CDMY70CDMY without knowledge CDMY71CDMY just can not be converted into an array. It is clear that the goal is on CDMY72CDMY, but it would be worth considering an alternative. It turned out as a result


    assertThat(collection, allOf(items.map { hasItem(it) })) 

    With denial, even more fun:


    assertThat(collection, not(anyOf(items.map { hasItem(it) }))) 

  • In continuation of this bacchanalia with CDMY73CDMY, I put ± in the "collection" column, because it would be better if there were no assertions than such.


  • Exceptions: no separate verification.


  • Integration: no CDMY74CDMY.


  • Error description: no details for collections.


  • Complex structures: the test with nested arrays failed.



JUnit
  • Readability: Yoda-style CDMY75CDMY, you need to remember the nuances and differences of the methods: that arrays must be compared through CDMY76CDMY, collections through CDMY77CDMY, etc.
  • Description of errors: for those cases when JUnit did have methods, it was quite normal.
  • Substring: of course it is possible through CDMY78CDMY, but it doesn’t look very much.
  • Denials: there is no negation for CDMY79CDMY, which is logical, there is no negation for CDMY80CDMY.
  • Collections: there is no verification of the content of the element, CDMY81CDMY for CDMY82CDMY and CDMY83CDMY does not fit at all, because order is important to him.
  • Complex structures: stupidly not.

kotlin.test
  • Very poor. It seems like it should be a wrapper over JUnit, but there are even fewer methods. Obviously, this is a retribution for cross-platform.
  • The problems are the same as with JUnit, and plus to this:
  • No substring validation.
  • There is not even a hint of comparison of collections represented by CDMY84CDMY, there is no comparison of arrays.
  • Typing: JUnit doesn't care about types in CDMY85CDMY, and kotlin.test swears that it cannot deduce a type.
  • Description of the errors: nothing to evaluate.

Expekt
  • You can write in two styles CDMY86CDMY and CDMY87CDMY, and the second option is not infix. The difference is insignificant, I chose the second.
  • Readability: CDMY88CDMY vs CDMY89CDMY and CDMY90CDMY. And there is a private method CDMY91CDMY. I had to think about how to check for the presence of several elements in the collection: CDMY92CDMY accepts CDMY93CDMY. It is clear that the goal is on CDMY94CDMY, but it would be worth considering an alternative. For negation, you still need to remember about CDMY95CDMY: CDMY96CDMY.
  • Typing: the type is not inferred from the expected value, I had to write explicitly.
  • Exception Support no .
  • Array support no .
  • Complex structures: the test with nested arrays failed.
  • Description of the error: just different text for different assertions without details.
  • Integration: no CDMY97CDMY.

Conclusion


Of all this diversity, I personally liked Kotest, Kluent, and AssertJ. In general, I was once again saddened at how sucks work with arrays in Kotlin and was quite surprised that nowhere except AssertJ is there a normal recursive comparison of dictionaries and collections (and even there, negation does not work). Before writing the article, I thought that in the libraries of assert these points should be thought out.


What to choose in the end - the team has yet to decide, but as far as I know, most are still leaning towards AssertJ. I hope that this article has come in handy for you and will help you choose the library for assertions that suits your needs.

.

Source