In software test automation, we often find test scenarios where verifying the presence of web elements or its mouse events is not sufficient to ensure a defect free implementation of the application’s feature.
When testing test scenarios like verification of a video being played in a video player, verification of audio messages, zoom functionality of an image viewer, brightness of the image, robotic simulators, etc., visual or audio confirmation is indispensable to ensure that the app feature is working as desired.
Oftentimes, such scenarios are marked as “cannot be automated” and are left for human or manual verification in every release cycle. This adds to the overall costs and longer regression cycles due to the required manual efforts.
AI-enabled technologies like computer vision or speech analytics provide viable solutions to automate such test scenarios where we need human-like verification.
Example of AI-Based Verification
Let’s say we want to automate the verification of Zoom functionality of a video player. With the presence of ‘Zoom’ buttons, its mouse events and attributes can be verified using traditional web drivers like Selenium and Puppeteer.
How can we verify that the image is getting zoomed in on clicking the‘ Zoom In’ button and if the right section of the image is being zoomed in?
These test scenarios can be verified by utilizing Computer vision technologies. We can provide the zoomed image as an input to a computer vision engine that will perform image analysis and verify the image as per the test requirements provided.
In order to do so, image featurization techniques can be used to generate image vectors and run through a trained AI model to classify the image. We can integrate this computer vision system with our UI automation framework and run the system as a whole in an automation suite.
Best Methods for Verifying Objects or Image
- Contours analysis method should be used to identify the shape of an object and automate the verification of bounding boxes.
- Contour approximation, which uses the Ramer–Douglas–Peucker (RDP) algorithm, works well for approximating a contour shape to another shape with fewer vertices.
- In order to verify if two objects are overlapping, use union/intersection operations along with Gaussian Blur operators.
- ‘RETR_TREE’ method is an easy-to-use method for extracting both internal and external contours of an object and organizing them into a tree graph for analysis.
Key Takeaways
- Application features/functionality that requires Human-like visual and audio verification of test scenarios can be achieved by integrating UI automation frameworks with AI systems such as Computer vision/Speech-to-text.
- Various Deep learning techniques can be used for image featurization and classification.
QASource’s AI Automation Expertise
At QASource, we have extensive experience in designing and building AI test automation solutions for software applications/robotics simulators requiring Human-like verification.
We have expertise in computer vision technologies and Deep neural networks using various CNN architectures like RESNet, AlexNet, and OneNet as well as custom Deep neural networks.
Have Suggestions?
We would love to hear your feedback, questions, comments and suggestions. This will help us to make us better and more useful next time.
Share your thoughts and ideas at knowledgecenter@qasource.com
