How we fixed our flaky unit test suite

Remember every test has a cost, including CI pipeline time, development time, developer experience.

This cost must come with value.

Spend the best efforts by testing use casesValuable tests ensures the application works with minimal false positives and little effort.

Using mocks makes the test easier to write, but increase the chances of false positives.

Testing implementation details, i.


auxiliary classes used inside you use cases orchestration classes (or functions) don’t ensure the application works.

That’s why you should focus your efforts on testing the use case public API surface rather than testing that a mock was called, or even if a data repository returns the right set of data.

Thankfully, there is an agile testing technique that helps us in writing self-documenting and valuable test cases.

Let the (B)ehaviour (D)rive the test cases (D)evelopmentBehavior Driven Development is a technique that helps us testing components guided by the user’s behavior.

Even though BDD is better suited to end to end acceptance tests, you can base your unit tests on the behavior of the use cases nevertheless.

Use an ubiquous language to describe your tests, avoiding to describe implementation details such as “when file payload is null”.

Prefer “when user deleted the file”.

Use BDD scenariosWe also stopped using the method name as the test scenario description and started making heavy use of BDD scenarios:GIVEN a preconditionAND another preconditionWHEN i do somethingTHEN a post condition happensAND another post condition happensIt doesn’t work only for use cases scenarios.

For example, you could use a BDD scenario for testing the behavior of a list:GIVEN that a list has an itemWHEN i remove an itemAND i try to get another itemTHEN the list should return emptyResultThis is a test scenario using these patterns.

We know there’s a lot of room for improvement, and at Qualyteam we are constantly doing so.

You can noticeThe BDD style scenario description tells us exactly what we are testingThe absence of the new keywordFluent assertions explaining an odd but necessary assertion (the last one)The lack of mocks being set upWrapping upYou might be wondering if use cases tests without mocking collaborator classes are actually integration tests.

We usually don’t care about labels, but we consider these tests sociable unit tests.

The point is: do your unit test suite allow you to refactor implementation details without failing while giving you the confidence that your application still works?I’m a big fan of Kent Dodds, a prominent Javascript and React developer.

In the frontend context, he advocates for unit tests that behaves more like functional tests, testing the user’s behavior.

He even wrote a testing library for that.

So maybe you shouldn’t spend so much effort testing implementation details, like a Redux saga, and focus more on what that saga does for the user behavior.

We used his mindset to refactor our tests, but adapting it to the backend.

Also, I want to thank my fellow developers Pedro Ramos and Leonardo Prange for helping with the R&D job.

.. More details

Leave a Reply