Help, our automation is taking over!

I have always been a fan of testing and, in particular, automated testing. Recently, as my team moved to being “by the book” Agile, we made automated testing not only the way of QA life, but a part of the whole process – starting with how the initial requirements are defined, code is developed and ending with, yes, testing (deployment to come).

As testing now permeates the team’s existence, it creates an impression that a lot of what we do is furthering automation rather than building the product, possibly sinking a lot of time into it (hence the title). A question was asked: “Isn’t this approach actually more costly than what we used to do?” This article is the answer, with as many concrete numbers as I can dig up.

As far as the QA process, what is the main difference between what we used to do and what we do now?

Previously, QA was a standalone part of the process. QA would get the spec and the application increments and would do their best to write test cases and match the spec against the application. They would attempt to automate as much as time permits, experience showed that time did not permit much.

Now QA is the responsibility of everyone. The process starts by defining requirements in terms of user stories and then immediately translating them into BDD scenarios 1. The spec, as a document, no longer exists!

The complete process proceeds as follows:

  1. During iteration planning each story that deserves to be automated is identified and a separate task for automation is created
  2. Immediately after the sprint planning the Product Owner (PO), either with the help of QA or on her own, creates BDD scenarios that describe how each feature should operate. This is often done even for the stories that are not going to be automated
  3. QA takes the scenarios from the PO, augments them with its own scenarios and commits the whole thing to the repository
  4. As the developers complete the stories, they either ask QA to implement the automated tests based on the created scenarios or do so themselves
  5. By the end of the sprint QA is able to run the tests both from this and past sprints to validate the functionality of the application

In the appendix to this post I included the broken down estimate of the amount of time it previously took (to create the spec, answer questions and tests) vs. the time it takes now. Assuming I did not fudge the numbers the conclusion appears to be that the amount of time is about the same (actually less with the new process, but I am willing to attribute this to a roundoff error).

This seems to address the concern of the cost of adding automated testing into the process. It, however, does not nearly show all the other benefits we gained as a group. The greatest one, by far, is the supplementation of the spec by BDD scenarios. Before the QA group was often confused when dealing with (many) versions of the spec — not any more! In addition, writing out BDDs causes the PO to think as QA and specify a lot more “hidden” requirements from the start. It also allows QA to “see” the product even before something is written and ask PO necessary questions very early on.

Developers similarly benefit from seeing BDD scenarios as they also no longer deal with the spec. They are also able to discuss the product not only with the PO but also with QA (although I am still waiting for this to become a more frequent occurrence).

Finally, and almost for free, we get an automated test suite to help us with CI and regression. As we develop more and more tests, our regression time goes down, while it always used to increase before. QA are now much less stressed and tired than they used to be.

Do I recommend automation and BDD as a means to implement it? Yes, very much so. Will it be easy? Not at all, putting it on an existing project can be quite painful both technically and culturally. Implementing it successfully, though, will benefit your team for years to come.


Below is the time estimate of our process, before and after.


  • The figures are derived for the same set of requirements, enough to keep the team busy for about two weeks 2
  • The figures are provided for the parts of the process that have changed
  • In “Before” QA is mostly manual with some automation
  • In “After” QA works on writing automated tests, manual testing is done only when automation is not possible
  • Estimates for the actual Dev and QA are not included as this part of the process did not change (we are still coding and testing)
  • All the numbers are derived by multiplying by the number of people involved
  • Time differences are not always translated directly. Even if the total amount of time between “Before” and “After” might be going down, it is possible that the saving go mostly to the PO, for example, while an individual developer might be spending more time on some part of the process.

Before – QA is done manually after Dev is finished

Process step Duration (min) Duration (max)
PO creates the spec 2 days 3 days
Dev managers does the initial review and provides comments to PO 0.25 days 0.5 days
The review cycle is repeated at least 3 times, 1-2 hours per person per cycle 0.5 days 1 day
Dev manager takes the final result and breaks it up into tasks 1 day 1 day
Tasks and the spec are delivered to the dev/qa team, each person spends at least 2-3 hours reading the spec and understanding the tasks 1.5 days 2.5 days
Dev manager and PO spend at least 1-2 hours with each developer answering questions 1 day 2 days
QA spend a day trying to create the test plan based on the spec 2 days 2 days
QA perform mostly manual regression at the end of the sprint, the regression time goes up each sprint 3 days 3 days
Total 11.25 days 18 days

Now – QA is a part of the process, everything that makes sense is automated

Process step Duration (min) Duration (max)
PO defines high level stories 1 day 3 days
The whole team spends 4 hours estimating and breaking up the stories 4.5 days 4.5 days
PO spends 1 day writing BDD scenarios 1 day 1 day
QA spend 2 hours analyzing BDDs developed by PO, this becomes the test spec 0.5 day 0.5 day
Dev manager and PO spend at least 1-2 hours with each developer answering questions 1 day 2 days
QA performs regression at the end of the sprint, currently a day and decreasing 2 day 2 day
Total 10 days 13 days


1 BDD stands for Behavior-Driven Development. More information can be found here.

2 This is the most dangerous assumption in the whole post. Are we really doing as much now as we did before? If we are actually doing less, then the whole thing breaks! It is not possible to really test this without redoing one of our older projects, so I am going on my intuitive feel. It seems, that in terms of real output we are doing at least as much, if not more than before. The only “real” proof I have is that so far no one had asked “And this is all you did?” after seeing the results of a sprint.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s