Software Test Automation Challenges/Problems And Possible Remedies

Software test automation is a process where the software is tested by another software program with less or no human intervention. As we all know, software testing happens in various stages of software testing. Unit tests are where the software is tested at unit (function/method) level. Unit tests are automated easily using Unit test frameworks such as JUnit Test in case of Java Programming Language. In unit testing user operations are not tested. Next, the developer Integration Tests can also be automated but again this testing is to verify if the interfaces(contracts) are working as expected. Even certain regression test suites/Performance and capacity test cases can be automated. Apart from Unit tests automation, there are different challenges automating Developer Integration, Regression,System, PCT tests. Based on my experience, I've listed down some of the common challenges/problems faced during test automation below.

1. Automated Tests can be extended beyond unit tests, but this is limited to software components that are of non UI type. When it comes to UI testing, we fall back to manual testing procedures. The remedy for this challenge is to design the software in such a way that the back end code is separated from its user interface. If you are dealing with legacy/existing code and if the code is not well written to separate out business logic with UI logic, the code refactoring should be considered to be able to automated testing of the business logic. By this leaving apart the UI code, rest of the code is tested by automated tests.

2. Problem in automating regression tests are that the software code gets

changed very often due to changing requirements. This becomes a challenge when agile methodology of software development is being followed. There will be a shippable software release in every 2 months (on an average). Before getting into solving the problem, lets see what's done in regression tests. Regression test is all about testing the core functionality of the software to see if they are impacted (broken) due to the new changes being done. User critical operations of the application are listed down and tested. In a product based company, the code gets into maintenance after the release. New features/product enhancements, bug fixes continue. However, the core product functionalities remain. And regression tests are to make sure these core functions work even after new code is added.

Coming to the automation part, as said above, the main functions of the software remain the same, but the code to achieve would've been changed. The code would've been refactored to incorporate some other changes and automated regression test code is out of sync with code and may lead to incorrect test results. The solution (as how looked at it) is that the interfaces must not change. This means that the method/function signatures' must remain the same though the method code might have been changed. If this agreement is in place, it becomes easier for automating regression tests. However this may not be the case, the developers may be forced to change the contract to achieve other application functions or to fix certain bugs. In this cases, the regression tests automation author must be in sync with the developer and make necessary change in the tests code.

3. How are we testing the user scenarios that involve the UI and require user interventions? Well, UI testing can also be automated. There are tools to automate the black box testing. Some examples for UI(black box) testing are salenium(Web Pages Automation), TestingWhiz, EggPlant(for almost any kind of application test automation including UI). The salenium records the user actions on browsers and play that. EggPlant works based on image matching technique. When we learn more about these tools, it seems that we can automation every aspects of the application, but there is a challenge too.

Using these tools we can automate the user scenario if we know exactly how the software should behave on all user actions upfront especially for automation tools that work image matching. We need to define the expected UI screens for every user actions. This works for ideal conditions, but there are several factors that can cause the software to stop functioning. For example, lets say the application is communicating to a server and the communication gets lost. This can happen at any time we do not this upfront. These kind of failure scenarios become difficult to automate. The remedy for this problem is that the automated test code/script can define fixed wait time before deciding  there is no response by the application. 

4. Every time UI is changed, there is effort required to update test scripts, increased maintenance cost. For certain automation tools like EggPlant the if the screen UI is even slightly changed. (The text is capitalized), the test script requires changes. The remedy for this problem is to make developers and other team members involved in the software creation about the automation scripts. The software test engineer has to in constant touch with developers to understand what changes are happening, are there any UI changes, etc and make sure the automation code/scripts are updated as per the changes.

5. The Performance and Capacity Testing is a good idea to automate since the manual testing takes lot of time. In this type of testing (PCT), the performance of each user operation or critical user operations are measured under different application usage patterns. For Ex: The user operation 'Upload File' takes 35 seconds at the load (CPU usage) 60% etc. The most difficult part in this type of automation testing is that, the application behaviour can not be defined upfront at different load conditions, and hence the automation scripts can not be written to pass or fail. However, the performance measurements aspects can be carried out and may be for base-lining when the application is first being tested for performance and capacity. For mission critical applications that do have defined speed requirements, can easily be automated, the test case is failed if a particular mission critical operation did not complete in defined/stipulated time period.

If the TDD (Test Driven Development) is not in place, the automation is a real challenge. Most of the times, the automated tests go out of sync may be because software development planning effort estimation did not consider the efforts required for automation or automation is not enforced strictly by the organization.