Brief About Automation!!!!

Introduction

Today software test automation is becoming more and more popular in both C/S and web environment. As the requirements keep changing (mostly new requirements are getting introduced on daily basis) constantly and the testing window is getting smaller and smaller every day, the managers are realizing a greater need for test automation. This is good news for us (people who do test automation). 

Benefits of Automation

Automated testing tools are capable of running continuously without any productivity loss or fatigue, with minimal or no manual intervention. This implies that organizations can plan testing activities beyond the traditional eight-hour work shift. This translates to a testing program that reduces the elapsed duration for testing by as much as two-thirds of the time required for manual testing.

Drivers	Direct Benefits	Indirect Benefits
People	Savings in staffing costs due to efficient redeployment of workforce	Motivated workforce, increased customer satisfaction
Process	Savings in testing lifecycle costs due to reduced execution time	Enhanced process efficiency, innovations
Technology	Improved productivity due to additional test cycles within a given schedule	Lower application lifecycle costs resulting from improved product quality

When should test Automation?

1)The test must be repeated

2)The test's workflow and its validation evolve and change slowly over time

3)The test validates a business process or workflow ,rather than validating the look and feel,color,table,layout .etc

4) When there are frequent regression testing iterations

5)The test produces results for a regulatory body that demands that those results be electronically recorded and archived as formal evidence of compliance

6)The test's pass/fail results are reasonably easy to determine and capture with selected automation tool

When should Test Automation Be Avoided?

1)In the case of Adhoc testing where subject matter expert randomly prowls through a variety of combination work flows

2)In the case of onetime testing or when is repeated only few times

3)In the case of testing which requires covering multiple functional areas so that the test travels through a small amount of virtually all of the product  functionality

4)Testing where look and feel ,color table lay out et..are validated

5)Test cases for which test data cannot be determined before hand

selecting any testing tool:

Anyone who has contemplated the implementation of an automated test tool has quickly realized the wide variety of options on the market in terms of both the kinds of test tools being offered and the number of vendors. The best tool for any particular situation depends on the system engineering environment that applies and the testing methodology that will be used, which in turn will dictate how automation will be invoked to support the process.

1) Do you have necessary skilled resource to allocate for automation tasks?

2) What is your budget?

3) Easy to maintain automated tests with a central repository whereby users can separate GUI object definitions from the script

4) Does the tool satisfy your testing needs? Is it suitable for the project environment and technology you are using? Does it support all tools and objects used in the code? Sometime you may get stuck for small tests due to inabilities of the tool to identify the objects used in the application.

5) Does the tool provide you the free trial version so that you can evaluate it before making a decision? Also does the tool have all features available in trial version?

6) Does it provide simple interface yet powerful features to accomplish complex tasks?

7)  Does it integrate well with your other testing tools like project planning and Test management tools

8) How is the tool learning curve? Is the learning time acceptable for your goals?

Costs of Automation

The cost elements for automation can be classified as fixed and recurring costs. Fixed costs are one time investments that are needed initially to establish the automation environment. Recurring costs are incurred during the testing lifecycle. The cost matrix depicts the distribution of fixed and recurring costs across people, process and technology drivers.

Drivers	One Time Costs	Recurring Costs
People	Cost of training staff on automation tools  Staffing costs for automation script development	Staffing costs for automation script maintenance
Process	Costs for establishing new processes (workflow, configuration management, process management and so on)	Not Applicable
Technology	Cost of hardware and software Licenses for automation	Cost of maintaining hardware and automation software

What is Framework?

Automation Frameworks can provide reusable code bases which support the deployment of the testing tool into the engagement.

Types of Frameworks?

            1.Test Script Modularity

            2.Test Library Architecture

            3.Data-Driven Testing

            4.Keyword-Driven or Table-Driven Testing

            5.Hybrid Test Automation

Benefits Of Framework?

                 1. Reduce testing time.
                 2. Improve testing productivity.
                 3. Improve product quality
                 4. Reduce QA costs.
                 5. Consistent test results.
                 6. Schedule test run  7.Re-use

Observations

I have met a number of QA and Test managers who are frustrated with their automation. According to them the tool is not doing what it is supposed to do. Here is a true story, the client (I had the opportunity to work with them for some time) found out that the tool they have just bought does not support the application they are testing (I am not making it up). How can this happen! – It does happen more often than one would think. I will get back on this when I discuss possible solutions. A manager of one of the major telecom companies that I had a recent interview with told me that after three years and more than a million dollar he is still struggling with automation. This is pretty sad and I get the feeling that he is not alone.

Solutions/Suggestions

Let’s discuss some of the reasons for this frustration and some of the solutions to this problem.

Unrealistic expectations: Most managers have their first encounter with any automation tool when they look at the demo and everything looks nice and simple.But everything is not so nice and simple when you try to use the tool with your application. The vendors will only tell you the things you want to hear (how easy to use, how simple to set up, how it will save time and money, how it will help you find more bugs etc.). This builds a false set of hopes and expectations.

Lack of planning: A great deal of planning is required from selection to implementation of the tool. “Evaluating Tools” by Elisabeth Hendrickson is a very good article on step by step process of selecting a tool. She talks about “Tool Audience” as one of the steps. This would be an ideal way to select a tool. It may not happen in every place because of the everyday workload of the people involved. But the participation of the users in the process is very important, because they are the ones who will use the tool day in and day out. I am almost certain that what happened to one of my clients (the tool they have bought did not support the application they were testing) would not have happened if the users were involved in the selection process. 

Lack of a process: Lack of a process may also contribute to failure of automation. Most places do have some kind of process in place. In most cases (although it differs from place to place) developers write code against a set of requirements. If the requirement does not call for a change in GUI then, there should not be any change in GUI. But if the GUI keep changing constantly from one release to another without any requirement for that change then, there is a problem in the process. You may have the best tool and the best (for your environment) architecture is in place and you will still have problems with your automation because of a faulty process.

Conclusion

I think there is a need to educate  managers about the benefits and limitations of automation. There is a need to separate the facts from the fictions. But here is the problem, in most cases consultants are brought in to fix problems of prior attempts instead of initial setup. At this point the managers have already learned (painfully) the pitfalls of automation. In order to avoid this painful experience I would recommend (most automation engineers will agree with me) to spend more time up front doing research about the styles and techniques of automation and find an architecture that fits the environment. There is no doubt that automation adds a great value to overall QA process but, short of knowledge and understanding about automation and lack of planning can also cause a nightmare.

Sudhakar.Mangi