In an Agile development environment, test automation becomes a crucial part of daily activity. Agile teams tend to be small, with a lot of features to test and rarely enough time to do it all. A well written automated test suite is a tremendous help. Especially if you have many new builds each day.

However, a rather large portion of automation projects will fail. After seeing several of them fail, sometimes under my care and sometimes under care of someone else, I’ve compiled a list of reason which caused those projects to fail.

QA people are not programmers
It’s easy to write a script that will go from one page to another, and check that certain words appear. It is easy to write several scripts that complete from start to finish in one big function. It gets harder when you have to write a test that is smart enough to test every possible input for a field, and then has to repeat itself many times for the rest of the fields on the form. Having many “dumb” tests is easy, but completely pointless. Making the tests smart is tricky, and will require many common programing principles, such as DRY (Don’t Repeat Yourself)

Solution:

• Recruit actual developers to guide you in making your test code nice and reusable.
• Test developer should have at least some programing experience
• Read up on good programing practices.

Not enough time and resources allocated for development of tests
A test suite which tests the application and gives good feedback will take a lot of time and resources to create. Giving someone an hour a week to work on automation is not enough to even get started. It takes time to get your thoughts together, think each problem through and write a test. Even after the test is written and running, you have to constantly maintain and fix them.

Solution:

• You will need a dedicated person to write and maintain the tests.
• If you cannot afford a dedicated test writer, at least allocated 8 hours a week for writing test.

Tests are fragile
Maintenance of the test is a very time consuming job simply because automated tools are fragile and dumb. Any slight change in the interface, or work flow of the application will probably make your test break. This is simply because computers are not really smart, they only think they are. So it will be your job to hold its hand while it tries to fall over and die.

Solution:

• Fix your test often, get in the habit. It’s easy to fix one test per day instead of waiting for weeks and having to fix dozens of them at the time
• Have an agreement with developers to not change the interface on daily basis, or at least without notifying you.

Tests are too complicated and too interdependent
It’s easy to get carried away and try to make a single test check many features. If you cannot describe what the test does in 5 words or less, your test is probably too complicated. Giant tests are hard to maintain, it becomes much harder to add new tests, and when the test does fail it is difficult to figure out why they failed. To make matters even worse, you can have each test depend on the previous one to leave the application in desired state for the current test. Like a house of cards all your tests will come crashing down when one fails.

Solution:

• Write each test to complete 1 and only 1 specific task, no more.
• Write each test as a stand alone test, that way you can run any test on its own and it will run and pass on its own
• Break up complicated feature into small sub-features, and have fixture data available for each test.

Expensive
Most commercial automation tools will cost you an arm and a leg. We are talking about thousands and thousands of dollars for a single license. Management will buy these ridiculously expensive products, and will expect everything to be up and running in a week or two. But you will soon discover that there is no such thing as “Record and playback,” and everything that the sales person told you is probably not true. There is no tool that will do your job for you, you will have to spend a lot of time and effort to make them work. Soon after you will have the management breathing down your back expecting to justify the enormous purchase with some results.

Solution:

• More then likely you will be able to find some open source tool that will do the job just fine, sometimes even better. Look around, free is a great price to pay!

Expectations

“But the sales person told me that with the tool’s advanced “Record and playback” features, and object recognition, we will have an effective test suite in hours! How come the whole application is not being tested automatically? We gave you whole two weeks before the release of this product to finish the job!”

Success is measured by the expectations, and no one will spend $17,000 on a product without having “raised” expectations.

Solution:

• Tell everyone that there is no such thing as “Record/Play.” Test writers rarely use the record button, and even then only for reference.
• Manage the expectations, a good test suite is a tremendous commitment
• Start writing tests as soon as physically possible, don’t wait for the 2 weeks before release to start writing. Actually, 2 months will probably not be enough

Lack of usefulness

Any regularly ran automated test is better then nothing at all. But having a test that covers the wrong functionality, or does not fail when it is supposed to fail, is close to having nothing at all.

Solution:

• Test the tests, make sure they pass when they should and fail when the should
• Have a clear map of the test coverage
• Write tests for the critical functionality first, everything else later

Finding bugs is easy, reporting them is where a lot of up and coming testers get in trouble. A defect report that is too long can be just as bad as one that is too short. Developers are instant gratification junkies, and do not want to read 5 paragraphs of information that may or many not relate to the issue at hand. After reading a dozen bug essays, anyone will say “just get to the point!”

On the other extreme, I’ve seen a bug report that read “The application is broken.”

After attending a recent Quality Assurance conference, I realized how few companies have jumped on the “Agile Development” bandwagon. As Agile development practices become more and more mainstream, some companies choose to leave their old comfortable development cycles; sometimes with fear and disdain. But nevertheless, there exists a thirst for knowledge about how things actually work in Agile environment; is it just another buzz word?

Even though you can find several books on the topic of testing and running a QA department, this is not what I want to talk about right now. Over time, people have asked me questions about the day-to-day practices and overall spirit of an Agile organization. I would like to try to answer these instead of giving another lecture of DO’s and DON’Ts.

If you are about to enter the world of Agile QA, you need to be prepared for several changes. The biggest change that you will experience is “Trust”, the major selling points of Agile development methodology; power to developers! In a truly Agile team, any member of the team feels ownership of the whole product, not just parts of it. This feeling of ownership is reenforced by allowing the individual to make decisions independently. Team of developers can say “NO” to unreasonable features, push back deadlines, and express opinions on how to make things better. This trust and power in-turn is propagated to the QA department.

Product development places the trust in testers to find the defects and help develop the highest quality software possible. Management trusts that you are competent in your profession, and adopts a “laissez-affair” attitude towards how the department is run. You are given a chance to experiment, make mistakes, and figure out exactly what works for your team and what does not.

Short iteration cycles (development cycles lasting no more then 4 weeks, at the end of which a feature is delivered) re-enforce the low risk atmosphere. These short development cycles can be as dependent or independent of each other as the team desires. This means that the team is allowed to make drastic changes to their practices from iteration to iteration. Anything from the format of the meetings to size of teams can be changed, without resistance from management.

This means a tester can be as integrated or separated from the development team as needed. Inefficient test plans can be scrapped and replaced by better ones, or dropped all together. Some projects might require you to write detailed test cases, others will let you go with no test cases at all. The choice is in your hands; and if something is not working out, you are allowed to play around until you hit the optimal level of performance AND job happiness.

But this power extends beyond the QA department. As a QA representative, you can make adjustments in the world of developers. You are allowed to attend any and all of the meetings with developers. This means you can make suggestions about broken processes, help in the estimation of the project, make adjustments to how the new builds are given to you, and sit in with any developer as they are coding! You have the ability to find bugs while the code is being written, isn’t that exciting?!

Furthermore, you are placed in a unique position of being the middleman between the Product Developers and programers. This gives you the power to align product developer’s expectations with what is possible in the real world. And you are able to make suggestions on the functionality of the product to both the programers and product developers… best part of all is that they listen and implement good suggestions!

When you have this much involvement in the development of the product, you no longer need large volumes of documentation. You are close enough to the product to know the ins and outs of every feature because you helped to develop and implement it. This is much more reliable than pages of acceptance criteria which haven’t been updated in years (let’s be honest, it happens more often then we want to admit to).

Instead of relying on the extensive documentation to write test cases, testers use “tasks” to test the product. A task is a short description of the new feature, it can be as brief or as detailed as the team chooses. Detailed task will spell out every single condition, input and output of each module in the feature thus providing you with pre-made test cases. However, some of the tasks are high level (i.e. “user should be able to register”); thus giving the developer the freedom to work through every aspect of the feature without being constrained by tiny details. Even though the tester does not have a “pre-made” test plan, in this case, he is able to influence the developer in every aspect of the feature. This not only allows the tester to find bugs based on intimate knowledge of the product, but also leaves room to fix minor things that drive everyone crazy. For those perfectionists like me, it is a dream come true!

There are many more advantages for a tester in Agile world that I would like to dive into more detail in my future posts.

-Dima