Having any kind of automated tests is enough
In this article, we are going to go through the typical scenarios that the dev teams are facing when it comes to choosing how their software should be tested. This comes down to questions like:
· Where do we start?
· Where should we put most of our testing effort to?
· How can we proceed having put too much effort in ETE’s?
· How to handle management pressure?
We will try to answer these questions using the concepts of sad and happy testing pyramids.
Sad automated testing pyramid version
Now what we have here is the unfortunate reality for a lot of projects out there. In fact, a lot of them aren’t even on that stage as there is no automation whatsoever. Throughout the years only manual, functional tests have been done. Back in the past, that is how the test software testing looked like in most of the software projects. I still remember one of the QA team members showing me a (literally..) sea of spreadsheets containing intricate details of all the manual test. A summary of them was meant to be shown once and made obsolete a few weeks later. Now, even with automation at the place, we could still dive into a situation where tons of precious time is wasted.
When we move up from unit to end-to-end tests the time needed to run them increases considerably along with the cost of writing and maintaining them. And remember that software improves and matures basically by day, and changes to an existing feature are just a matter of weeks if not days.
The end-to-end testing seems a natural choice as, in the end, we are testing our software exactly like our users would use it. Let's go through an example:
Suppose an end-to-end test goes through 10 conditional statements somewhere in our codebase (and that is not that much) until a response comes back with the result. To illustrate, this can be a button click triggering a complex POST operation with a variety of conditions and sub-systems to take under consideration.
To cover all combinations of those paths and make sure that the application works 100% fine you would need to write a set of 100 super slow, super hard to maintain ETE tests which probably will just be a waste of time in the end.
Even if that suite would be kept, it is guaranteed that it would not be triggered
during the development stage because of the amount of time it takes to process. In fact, such a suite would be too much even for a continuous integration job triggered after each repository push. A separate, overnight job would need to be created, in order not to clog the server during the business hours. Now the developer would need to wait an entire day to make sure that his changes did not break any adjacent functionalities.
This is definitely not your ideal scenario for a fast-paced, by-hour commits focused agile project.
Let’s take a look at the standard sad version of the automated testing pyramid that I have stumbled upon in my career:
This is the most typical version where most of the automation is put in the ETE test suites. There is still a chance to get out of this and flip the coin:
· Filter out the ETE suite just leaving the most critical parts of the application.
· Start building from the ground up by writing tons of unit tests.
· Back them up (do not try to equal them) by some integration tests, keeping the numbers similar to the current percentage.
The following two are a bit less common but still having the vast majority held under the ETE suite.
The other 20% are just done by, most likely, a minority of the developer in a team. These guys still care for their solution to be failure-proof in a matter of seconds by anyone and anytime.
A way to go here? Get together with other devs contributing to these 20% portions and start pushing your team lead to make them mandatory:
· As part of your JIRA tickets “done” checklist
· Rejecting reviews if a more complex solution was submitted without any UT’s / IT’s.
· Teach the “unbelievers” whenever you can
The next version (a lot of happiness but still a bit of sadness to it) which I have stumbled upon once has still a shed of light in the tunnel:
Tons of IT’s, and a small fraction of ETE which have started to be put in place to replace the IT’s shortly.
So, you have become a team leader in a project like that, now what? It’s relatively simple to increase quality. Here most likely the dev team is proficient in writing automated testing and gets the most important ideas behind them. Just get them to switch from IT-oriented to UT-oriented, the rest will follow.
Finally we got a seeming basket case:
What can you do here?
No, fight. Not till the end but try to fight to start introducing UT / IT level automation. I have been there and done it. So can you.
Happy automated testing pyramid version
Now, this is how automation should look like in a perfect world. Personally, I have never reached this ideal in a real-world project, but I have definitely come close once or twice those figures.
The vast majority of work put into validating the correctness of the code/features should be present in the UT’s.
Thanks to that we have covered all the meaningful logical paths contained in our functionality, and we know they work as units.
We have got an oiled machine that can be run in a matter of seconds during any stage of development and tell us if we are on the green or not.
Now, all we need to do is check how higher level components cooperate with each other and supply them with real, not-mocked data (database, web service). If we have a high UT coverage.. then writing an IT shouldn't cause many problems. If so it most likely is the database, transaction or other framework problem which is actually the aim of those tests to fail-proof. In an IT we don't need to cover all the logical paths as in the UT’s.
We just need to choose one or two crucial ones and just confirm on a higher level what already has been done on a lower, unit level.
Keeping your ETE’s to an absolute minimum makes sure the critical end-user paths are covered, and you do not clog up the build server. It is also likely that your devs would be running them locally as, even though the time to run each of them is significant, there is only a bunch affected by their changes.
You definitely would not want to you see your pharaohs angry face like the one above. That is when he comes to the construction site once it reaches its deadline. If you're going to make your pharaoh happy and content with the quality of his future burial then try to follow these simple rules:
· Start from the ground up. Build a solid foundation of UT’s so that your team is familiar with them and it has become their second nature.
· At the beginning forget about higher level automation.
· Do not try to match the number of IT’s with the number of UT’s. In most cases do not exceed the amount of IT’s by 25% of the number of UT’s.
· Keep your ETE’s to an absolute minimum. Just a handful per major feature.
· Do not let the pressure of management dictate your actions. Though UT’s are transparent to them, this is from where the quality comes from. They will definitely change their mind once that client demo happens and strangely nothing breaks or crashes.
Now your pharaoh can stand tall and proud.