In an Agile development environment, test automation becomes a crucial part of daily activity. Agile teams tend to be small, with a lot of features to test and rarely enough time to do it all. A well written automated test suite is a tremendous help. Especially if you have many new builds each day.

However, a rather large portion of automation projects will fail. After seeing several of them fail, sometimes under my care and sometimes under care of someone else, I’ve compiled a list of reason which caused those projects to fail.

QA people are not programmers
It’s easy to write a script that will go from one page to another, and check that certain words appear. It is easy to write several scripts that complete from start to finish in one big function. It gets harder when you have to write a test that is smart enough to test every possible input for a field, and then has to repeat itself many times for the rest of the fields on the form. Having many “dumb” tests is easy, but completely pointless. Making the tests smart is tricky, and will require many common programing principles, such as DRY (Don’t Repeat Yourself)

Solution:

• Recruit actual developers to guide you in making your test code nice and reusable.
• Test developer should have at least some programing experience
• Read up on good programing practices.

Not enough time and resources allocated for development of tests
A test suite which tests the application and gives good feedback will take a lot of time and resources to create. Giving someone an hour a week to work on automation is not enough to even get started. It takes time to get your thoughts together, think each problem through and write a test. Even after the test is written and running, you have to constantly maintain and fix them.

Solution:

• You will need a dedicated person to write and maintain the tests.
• If you cannot afford a dedicated test writer, at least allocated 8 hours a week for writing test.

Tests are fragile
Maintenance of the test is a very time consuming job simply because automated tools are fragile and dumb. Any slight change in the interface, or work flow of the application will probably make your test break. This is simply because computers are not really smart, they only think they are. So it will be your job to hold its hand while it tries to fall over and die.

Solution:

• Fix your test often, get in the habit. It’s easy to fix one test per day instead of waiting for weeks and having to fix dozens of them at the time
• Have an agreement with developers to not change the interface on daily basis, or at least without notifying you.

Tests are too complicated and too interdependent
It’s easy to get carried away and try to make a single test check many features. If you cannot describe what the test does in 5 words or less, your test is probably too complicated. Giant tests are hard to maintain, it becomes much harder to add new tests, and when the test does fail it is difficult to figure out why they failed. To make matters even worse, you can have each test depend on the previous one to leave the application in desired state for the current test. Like a house of cards all your tests will come crashing down when one fails.

Solution:

• Write each test to complete 1 and only 1 specific task, no more.
• Write each test as a stand alone test, that way you can run any test on its own and it will run and pass on its own
• Break up complicated feature into small sub-features, and have fixture data available for each test.

Expensive
Most commercial automation tools will cost you an arm and a leg. We are talking about thousands and thousands of dollars for a single license. Management will buy these ridiculously expensive products, and will expect everything to be up and running in a week or two. But you will soon discover that there is no such thing as “Record and playback,” and everything that the sales person told you is probably not true. There is no tool that will do your job for you, you will have to spend a lot of time and effort to make them work. Soon after you will have the management breathing down your back expecting to justify the enormous purchase with some results.

Solution:

• More then likely you will be able to find some open source tool that will do the job just fine, sometimes even better. Look around, free is a great price to pay!

Expectations

“But the sales person told me that with the tool’s advanced “Record and playback” features, and object recognition, we will have an effective test suite in hours! How come the whole application is not being tested automatically? We gave you whole two weeks before the release of this product to finish the job!”

Success is measured by the expectations, and no one will spend $17,000 on a product without having “raised” expectations.

Solution:

• Tell everyone that there is no such thing as “Record/Play.” Test writers rarely use the record button, and even then only for reference.
• Manage the expectations, a good test suite is a tremendous commitment
• Start writing tests as soon as physically possible, don’t wait for the 2 weeks before release to start writing. Actually, 2 months will probably not be enough

Lack of usefulness

Any regularly ran automated test is better then nothing at all. But having a test that covers the wrong functionality, or does not fail when it is supposed to fail, is close to having nothing at all.

Solution:

• Test the tests, make sure they pass when they should and fail when the should
• Have a clear map of the test coverage
• Write tests for the critical functionality first, everything else later