Evolution of test automation frameworks!!!!!!!!!!!!

Implementing automated tests is something that everybody wants to do. If you ask any tester - test automation is her/his aim. It is the golden target that every tester aims for, but only a few of them take pains to assess the required knowledge, being under the illusion that a programming language or expensive tool will suffice to cope with all the problems that are likely to arise. This is not true. Writing good automated tests is much harder than that. 
1. Dimensions
The first thing that you must consider when setting out test automation efforts is framework or architecture (sometimes called a test harness) in which tests will be placed. By these terms, I mean all code that will support and facilitate test cases development and execution, but also the project structure and any management that will take place (e.g. test data management or documentation). Let's run through the different framework concepts. Perhaps you will recognise the path that you took during your test automation adventure (if you are an experienced tester). Perhaps, if you have just entered the path, it will help you make some decisions. This trip will also consider tester development skills and their evolution that accompanies the evolution of framework. 

Thinking about automation framework, one can think about it in many dimensions. I borrowed these dimensions from Brian Marick’s “Testing quadrant” concept (business facing, technology facing, supporting the team, critique the product) and used it to create frames, a starting point for our weigh ups. We will consider automation frameworks in the following dimensions:
Supporting the code - we will see what coding means to automated testing and how code complexity grows with our needs
Supporting the users - shows different styles and approaches of implementing tests that focus on other individuals then a tester (business, customers)
In Supporting the tests we will talk about what else is needed for framework to be useful and see in what way the context is important

Diagram 1 - Test automation dimensions

2. Supporting the code

When reading testing press, blog posts comments, various forum threads on testing, one special question tops all the others in the context of test automation. This question is usually posed by manual testers lacking technical skills or testing beginners and it reads - how much development skill do I need to proceed with test automation? or do I need to be a developer to start out with automating tests? This is an important question and the supporting the code dimension tries to clarify these doubts.

Linear tests

We usually start with the simplest approach - record and play. A tool records our actions and then makes it possible to have them played back. Tests usually look like linear code - one line after another and so on. Easy to create but leads us to a maintenance nightmare. Often these linear scripts need to be re-factored to run them more than once. Also, the recording feature can be used to speed up test creation in much more complicated frameworks instead of typing it from scratch each time. This approach is easy to use for a tester who does not possess any development skills.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

@Test public void linearTest() {   selenium.open(url);   selenium.type(text);   selenium.click(button);   selenium.verifyTitle(title);   selenium.click(link);   selenium.wait(time);   selenium.open(url2);   selenium.type(text2);   selenium.click(button2);   selenium.verifyTitle(title2);   selenium.click(link2);   selenium.wait(time); }

Example of linear test code.

Structured tests

With higher complexity of test cases that we want to implement, the need for more sophisticated test scripts emerges. We start to familiarize with a programming or script language that the tool supports and start to use its features - conditional statements, variables and loops. We start to call one script from another creating a chains of scripts. With this, our scripts can do more, more extensively and more reliably. On the other hand, lack of development skills can easily lead to creating spaghetti code with the complex and tangled control structure. We still suffer from maintenance issues and going into raptures about new possibilities we do not realise that we are coming closer to a dead end.

Reusable tests

Finally, when maintenance issues hit us it might be too late. But we see that some portions of code repeat and understand that we can reduce the maintenance impact by creating reusable blocks of code. We start using functions or procedures to extract code and create libraries to gather it in one place. We share it between tests and projects. Tests get more clear and readable. At this point, the tester has made a big progress with her/his development skills. She/He knows the basics of the language that she/he uses and starts to explore more advanced features.

1 2 3 4 5 6 7 8 9 10 11

@Test public void reusableTest() {   CQ55 cq = new CQ55();   cq.openWcm();   cq.open(INTEGRATION_AUTHOR_INSTANCE);   PresentationTemplatePage presentation = new PresentationTemplatePage();   presentation.runExportPdfWorkflow();   presentation.confirmWorkflowInfoDialog();   presentation.waitUntillWorkflowNotFinish(); }

Example of reusable test code in Java.

Design Patterns

Sometimes we can go much further. We might bravely step into the development world and take little bites from anything that can help us. Design patterns are that thing. A design pattern is a general reusable solution to a problem occurring commonly. With design patterns we can improve some tests capabilities for example Selenium’s Page Object pattern reduces the maintenance impact and fosters order in source code. In Selenium alsoSingleton can be used to provide a driver instance to tests instead of inheritance. If you need to verify a database, the Entity can be useful to reduce the quantity of database connections. One connection fills up the entity and from now, actual values for assertions can be fetched from that object. Using the design patterns in automated tests is not necessary but recommended. It requires some dedication from a tester as it can be a giant leap for her/him in terms of her/his development skills. I think it is worth trying at least.

3. Supporting the user

The aforementioned architectures fits best when technical engineers are the individuals that implement tests. But what if we want a customer to be involved in the process of writing tests or we have more non-technical testers? Is such participation possible and what can be derived from it? What if we want to separate a test design from implementation? A code from tests? 

Keyword-driven tests

In the Keyword-Driven framework tests are created in an external file - spreadsheet, text or xml outside of code. Tests consist of keywords that correspond to the actions that user can take upon the application under a test e.g.: click, type, drag and drop. These actions can take arguments like object to be clicked on or typed in. To improve maintainability one can move to the higher level of actions that can be undertaken and define more complex keywords like “login”, “add document”, “create an invoice” - it only depends on the application under the test. The keywords are defined in code and adding a new one or changing the old one involves fixing a code base. Also, since the test cases are on a different level, one can change the execution tool beneath without changing test cases itself. Playing this game makes sense only if a non-technical person will be involved in test automation because this approach has some serious shortcomings. The fact that tests are defined in a text file, outside code and “outside” a programming language brings about problems with debugging. We lose a test case abstraction staying over only with the keyword abstraction. Since tests that we want to implement can be really complicated and GUI testing tends to be flaky, the loss of debugging on the test level can be painful. 

The concept of the keyword-driven framework may seem to be complicated especially to implement. Fortunately there are tools/frameworks already implemented that can be integrated with yours. For example Google’s Robot Framework can be easily integrated with Selenium/Webdriver. Also commercial tools have such capabilities. If not, they provide sets of helper objects that can facilitate this task. But still the tester needs to demonstrate development skills in order to get all of these working.

keyword	argument 1	argument 2
open	www.mypage.com
click	addButton
type	editName	John
type	editAge	30
click	submitButton
verifyText	header	Thank John!

Table 1 - an example of keyword driven test in a spreadsheet manner.

Data-Driven tests

When Keyword-Driven tests focus on the logical flow of actions, Data-driven tests focus on data and inputs/outputs. Usually, we have a test case defined and set of test data that will be passed into it as arguments. This way we can execute one scenario with different inputs or outputs for example when testing some sort of calculations or algorithms. Test data can be defined in external file - in this way we separate test data from test code. But this is not necessary. TheTestNG framework, allowing implementation of unit tests  (and not only) for Java applications, comes with the @DataProvider feature letting us to create data sets that can be used and shared among the test method. Using this framework let us remove redundancy from tests when we have many similar tests differing with data only. Moving the data to external source facilitates users, customer, business to create tests and a better collaboration.

BDD tests

But if one need to emphasis a business point of view and requirements then one could go BDD - Behaviour Driven Development. Originating from TDD and eXtreme programming,  Given-When-Then style tests focus on the system behaviour understandable to people and allows non technical individuals to participate in test creation. This style focuses on collaboration between business, QA and developer and this collaboration is facilitated with a ubiquitous language, understandable to everyone. BDD is about examples and there are tools that help us express them in a different manner. Tools like FIT, Fitness, Cucumber allows to create examples in spreadsheets or wiki tables or text files. No development skills are required for creating tests (defining feature or scenario) but a mediation layer called fixture or step definitions must be implemented in a programming language to make them all work.

Model Based tests

On the other hand, sometimes we don’t need to support a non-technical user but we might need to conduct a more technical analysis of the system and therefore test in a more pragmatic way. In such case, the Model Based testing might be a good choice. In this framework, test cases are derived from prepared model automatically so changes in the model are reflected in changes in test cases. With such an approach, one can manage requirements via managing models. Like in aforementioned frameworks, a test design is separated from implementation. Models in this case are an abstraction layer. These models can be created and maintained by testers but also business analysts or whoever who can handle a modelling tool. Since the most often used model is a finite state machine reflecting system behaviour, the modeller must understand fundamentals of software engineering at least. Besides the model, code that executes tests must be implemented and this is usually done by devs. When looking for MBT (Model Based Testing) tools, one can find Spec Explorer or MISTA - a Model Based Testing tool ready for Selenium Integration. Model Based Testing evangelists claim that testing is always based on models. If not as a tangible, genuine form then as a mental one. I think there is something in this statement. Thinking about flows of actions, the structure or behaviour is much easier when visualised. And even if we do not transcribe it, we have always such a model in mind in one or another form.

4. Hybrids

In most cases our needs go beyond one solution. Usually we need to combine nearly all of them or parts of them to accomplish our work. This is what we call aHybrid and implementing such a hybrid on your own may require serious development skills from the tester. But tools like Fit/Fitnesse, Robot Framework or Test Complete has all that features available ready to use so there is no need to implement it from scratch. A hybrid may consist of a mix of BDD with Data-driven tests and Page Objects so in the real world we will rather meet conjunctions then the exact implementation of one framework. These framework does not exclude using them both.

What is the best approach and the golden solution? There are no golden solution, one might say, but this is determined by one’s needs. The most important questions that determines that can be as follows:

• who will use the system and who will develop the tests?
• does the user possess the right technical skills?
• do we need to separate tests from code?
• do we need to emphasise collaboration?
• do we have the time and budget to implement it on our own? or do we emphasise learning  by implementing it from scratch?

 
Diagram 2 - the Hybrid

5. Supporting the tests

The architecture construction is not done yet. While most of the commercial tools provide many helper objects to support tests, using tools like Selenium, one must develop many of the themselves. Or use already created ones like Robot Framework. Nevertheless, this may denote that the code supporting tests will be the main effort in the implementation.

 What do I mean by “Supporting the tests”? I mean that in tests, we need to conduct comparisons, analysis, set up, tear down and all of this code must be implemented to help us achieve that. Features supporting the tests generally vary as our needs do in terms of the application under a test. Depending on the context, Web applications might need, for example, verification of JavaScript errors. Using the Oracle database causes that we might need to implement the mechanism to import a database dump file before every test runs or develops a set of SQL queries to clear up database tables. When testing analytics, we must have means to check that proper requests are sent by the application and extract desired properties from it. So as you see, this effort might require, not only development skills, but also decent domain knowledge (databases, telecommunication etc). Let’s try to name some issues supporting tests:

• logging - to console, to a text file, to an HTML file with a timestamp allowing us to analyse and then precisely conduct a root cause analysis of “red” or flaky tests,
• comparisons and checkpoints - comparing images, XML files, text file, database tables, application under test control’s values, strings, numbers,
• statistics - created at the end of tests, the number of successful, failed tests, execution time, brief of failed tests,
• abstraction layers - at the test run level, test suite level, test level, action level (“add document”, “edit user”), operation level (click, type). It may be needed to inject probes for a better analysis of some issues that happen during tests,
• email support - creating an email to be sent after tests, connect with an email box to verify the email content,
• working with files,
• working with databases,
• test evidence - integrate with a video or screen-shot capture tool to gather the evidence of conducted tests,
• API’s - use tools/libraries (like DefaultHttpClient) to be able to submit HTTP requests in order to communicate with API’s,
• test execution parallelization - this can be developed as a solution in code but also can be brought out of code e.g.: by splitting tests to different test suites and run with different jobs in some CI tool (like Jenkins),

...and many more according to one’s needs.

 Beside all the things that were described here, implementing a test case is the most important issue anyway. You can have sophisticated code with many design patterns, BDD, keywords, data, models but pointless and imprecise tests ruins all the effort. Hence, implementing well balanced tests itself is the most important thing to achieve. That is a subject for another post, though.