Before launching automation works, it is recommended to measure the feasibility and the opportunity of the project in order to assess the relevance of the tools in the company’s context, the tests repository to be industrialized and the potential profit of the project. Such as study is characterized by a major step called the automation proof of concept or 'POC'.
This article will deal with the different steps required for this preliminary study and answers more specifically to the problematics the POC for tests automation deals with: when does the POC takes place in the automation project, what are the main challenges behind the concept, the process and the good practices to have.
1. Definition of an automation POC
2. Challenges of an automation POC
3. Report the results to the decision-maker
4. Good practices to execute a POC Squash TA
What is a POC for test automation?
A POC for test automation is one of the prior steps of the implementation of a test automation project.
In the buying process, the POC enables the company to focus better on important subjects, create the conditions for a better adoption from the users community, initiate change, implement good practices and lay the foundations for a successful deployment.
A POC focuses, in a more practical way, on the issues linked to the customer context. Thus, it allows to measure efficiently the opportunity for an automation project, depending on the organizational, technical and applicative context.
What are the main challenges for a POC?
The automated test development process is often described by the QA teams as one of the most difficult exercises that an organization can have to deal with.
Developing test automation is a long and difficult road, and it requires a lot of commitment and rigor from the development teams. This is why it is important to study the context and implement the strategy to set up.
The POC may address all or part of the main issues related to the implementation of an automation project. These issues can be organized under three axis:
1) The strategy
Objective - To study the test repository for automation, the team organizational changes, the need for skills improvement, anticipate third party expert interventions…
First, it is primordial to take note of the context and analyze the existing test repository, to check if it is suitable to the automation needs (test cases granularity, test case classification (non-regression testing, integration)).
Elaborate an automation strategy will enable one to identify the prerequisites and the impacts related to the implementation of automation (remedial plan on the test repository, description of the expertise required for automation, target organization, development environment, etc).
2) Technical feasibility study
Objective – To make the right choices for tooling.
This step is required to check if the tests are technically eligible to automation: choosing the compatible automated program to interact with the SUT (system under test), the use of proxy, etc.
It consists in taking a representative sample of the test cases identified beforehand and create the associated automatic scripts. This study enables one to assess the technical complexity of the implementation (installation of suitable environment, scripting, preparation of the data sets, configuration, variability, orchestration, security issues) and calculate a return on investment (ROI) on the reference sample.
3) Opportunity study
Objective - To measure the return on investment, the profits in terms of quality and time, the investment in equipment and Human Resources…
The opportunity study allows one to calculate the different gains provided by test automation on the chosen scope, in terms of:
Time - Period before production
Quality - test coverage, test execution volume, types of tests to be industrialized (depending on the functional and technical stability, manual vs automatic execution time, etc.)
Finance - ROI
Report the results to the decision-maker
The results reporting of an automation POC is useful to raise awareness among the decision makers and highlight the subjects to pay attention to before taking any decision. The experts in charge of the automation POC have also the role of a consultant and shall advise the customer and guide him through the decision-making process.
The results reporting is made, most of the time, through a face-to-face meeting. It also includes the delivery of several additional documents, such as the opportunity study, the selection matrix for the tests to be automated, the automated scripts and the feasibility study report. The report also enables one to suggest an automation plan.
What are the good practices to execute a POC?
Provide the customer with the necessary prerequisites for a good POC (environment preparation, SUT dedicated to automation (different from the testing environment),
Define the POC scope
Find a solution to the issues noticed when initiating the project.
If the automation POC is validated, the different steps of the automation project can be implemented:
1. Definition of the automation platform’s architecture
2. Implementation of the automation tooling within a dedicated platform
3. Definition of the automation cycle
4. Adaptation of the team’s organization to new processes
5. Training in use of tools and new processes for the users and all the operators
6. Project launching
Time allocated to all the different steps of an automated POC depends on the organization of the in-house teams, the tools used and the existing repository.
An estimation for the automation POC is available when the execution architecture is based on the functional test automation tools produced by Henix, for their customer’s automation works.
A Squash TA POC is composed of the 3 phases described above. It is addressed by a Henix consultant and lasts approximately 20 days.
Understand the context, the challenges and the needs: approximatively 2 days.
Feasibility study: approximatively 6 days. On average, 5 automated tests are made during the feasibility study. There may be Batch tests, Web services tests, and/or IHL Web.
Opportunity study: approximatively 7 days.
Why Squash TA?
Squash TA (Test Automation) is the only Framework on the market suitable with most of the common automated programs (SoapUi, Selenium, Sahi, Dbunit,…). It enables one to run the data supply and the data cleaning before and after the tests or manage aging factors of the data. Moreover, it is fully interfaced with the Squash TM test repository manager or it can be controlled from a continuous integration platform (Jenkins).
The technical process for a POC with the Squash toolkit:
Comments