QASource, in conjunction with Sauce Labs, recently presented a webinar titled: “Reducing False Positives in Automated Testing.” Below are answers to some of the questions that were asked before and after the webinar.
- What is the best way to test for existence/non-existence of web elements?
The best way to check for existence/non-existence of web elements is by implementing dynamic object synchronization in your framework. This marginally increases execution time, but it dramatically increases the reliability of tests and saves time that will be spent later for analysis of false positives. Thus, dynamic object synchronization is a key strategy worth implementing in your framework. - How do you handle known bugs? (Known bugs are bugs that exist in the product, but are not immediately fixed) Do you write soft errors into tests to prevent them from failing and instead mark build as unstable OR do you let them fail always OR do you remove them from the test suite completely?
You can handle this situation by grouping suites with different names or priorities. Cases with known bugs can be grouped in a separate suite and then prioritized for fixing. This can also be left as unstable, while all of the other builds pass. However, it is not a good idea to remove tests from the suite, as the underlying scenario will be missed with this approach. - Will putting dependent tests in the same script help in reducing execution time of the suite rather than creating individual scripts for each test, as this will consume more time for login, setup, running the test and logout?
Dependent tests are more prone to leaving footprints in case of failures, which can lead to false positives. With independent tests, you can see a marginal increase in execution time, but it saves a lot of time for identifying and fixing false positives, giving reliable test runs and better ROI on automation. - What are commonly used automation tools?
There are many commercial and open source tools available. However, be sure to do a feasibility analysis as per your requirements before settling on a specific automation tool. Some of the most common tools are Selenium, Telerik test studio, QTP (UFT), Test Complete, Watir, Squish, Appium, CodedUI etc. You can learn more about picking the appropriate automation tool here. - Can you clarify on dynamic object synchronization? A number of our tests fail/timeout due to inadequate wait times that are manually entered into a central test configuration document that is then used by all tests.
Dynamic object synchronization allows you to get the presence of an element up to a defined time limit. If the element is found within the time limit, then the underlying statement is executed. This allows successful implementation by a centralized configuration. - Is the delete functionality tested separately? What happens if “delete” fails as a part of the AfterClass?
Yes, delete or truncation of records should be tested independently before the actual tests. The tear down handles similar scenarios in tests. - How are re-execution capabilities implemented? How is this reported to the CI server?
There are many methods to accomplish implementing re-execution capabilities. Writing a test executor that listens to the status of tests and re-executes them if they fail is the most common method. Another approach is to configure different CI builds, where one is triggered on the status of others and then groups the failed tests for execution. - Does an unmaintained test suite lead to false positives?
If tests are dependent on each other than flow needs to be maintained in test suites. If there are any functional changes the corresponding tests will fail and the underlying functionality will change leading to possible false positives.
Interested in viewing any of QASource's other webinars?
Browse our collection of webinars here.
