QASource recently presented the webinar, "How to Get Faster, More Reliable Automated Testing Results Using Continuous Integration" with its partner, froglogic.
In this webinar, Vivek Kumar, Automation Architect and Anand Ramakrishnan, QA Director at QASource, illustrate how to incorporate the Squish GUI tester into your application lifecycle using built-in CI integrations, how to pair Squish with the Jenkins Continuous Integration server to run your test suites automatically and how to enhance your automated testing workflow with a CI-based protocol.
During the live presentation, attendees were given the opportunity to ask questions, but we were unable to get to them all. Here, our expert engineers answered them. We've placed their responses below.
1. Is there an option to re-run failed test cases in a Jenkins configuration?
Yes, you can execute failed test cases from Jenkins. In order to do so, you will need to install the 'Naginator plugin' first. Once you’ve installed the plugin, go to 'Jenkins job' and add the 'Retry Build after failure' section from the post build action. Naginator offers various parameters, so you can select any of those based on your needs. You will also have the flexibility to execute failed test cases more than once.
2. Can you please share a few comments on how you are using SQUISH for distributed testing?
We orchestrated automated tests based on the components, and we are using multiple virtual machines to execute automated tests in parallel. Squish needs to be running on a specific port on each machine, and the same information is provided to the scripts when launching the application. Once we have completed executing scripts on all machines, we collect the HTML results and share with stakeholders as a single report.
3. Could we configure a pipeline script to skip the rest of test suites if the first test suite failed?
Yes, its possible to skip test suites if one particular test suite or the first test suite fails during execution. You will need to configure your pipeline jobs to automatically skip the rest of the test suite if the first fails.
4. We have an enterprise-level Jenkins server, do we need have the Squish CI plugin installed?
Yes, the Squish CI plugin needs to be installed on the Jenkins server even though you have already set up an enterprise-level Jenkins server. The Squish CI plugin is not installed by default, so it will need to be set up before configuring Squish automation tests with Jenkins.
5. Is the CI pipeline within the Dev team or test team?
The CI pipeline is generally set up by a DevOps team or a release engineer since they have the appropriate skill set to handle such tasks. First, the Dev team needs to commit their code into the source control repository. After the Dev team has committed their code, the DevOps team will build the pipeline job so it can download code changes from the version control system and generate builds. The pipeline job will be able to execute unit, integration and acceptance tests on either the Dev or integration environment, stage and then the production environment.
6. What GUI test can be used with an application written on MFC, an image comparison or any else?
There are various factors that can help you finalize the test approach for your application developed with MFC. We've listed a couple of these factors that can help you analyze whether you should use image-based or object-based testing below:
- The image-based testing approach is recommended if you are concerned about providing elements on the UI
- The object-based testing approach is recommended if you are looking for a solution that can easily adapt to changes or is maintainable
7. Can we make Jenkins job run Squish tests with a desktop application?
Yes, you can configure Jenkins job to run a Squish test with a desktop application. The steps to configure Jenkins job are similar for any application, whether it's a web or desktop application. Jenkins' role is to trigger a test on the specified machine where AUT is available and the Squish toolkit will execute the desktop application test.
8. What test automation tool would you suggest to GUI test our application on Linux (Ubuntu)?
There are various GUI automation testing tools available in the market that can run on multiple platforms, including Windows, Mac or Linux. Some of these tools are froglogic's Squish, Ranorex and IBM's Rational Functional Tester.
Selecting the right test automation tool completely depends on the technologies used in your application and your testing requirements. QASource is a tool agnostic company and we can provide you with more details or help you find the best tool for your requirements. Reach out to our team, we'd love to connect with you.
9. We've recently started using Squish for Windows (6.3), and have issues with test stability. Our object naming will vary with new builds, and tests are failing on CI Jenkins that are passing when ran manually. How can we improve this?
If we could have access to your environment then we could provide you with the exact reason. There are multiple reasons such as application dynamic objects, CI configuration or test scripts dependency that could cause your tests to fail. For strategies on how to reduce these issues or false positives, please watch Reducing Fales Positives, one of our previous webinars.
10. How do we build multiple branches (Sprint Branch, Dev Branch, Master Branch) using a pipeline?
In order to build multiple branches for different purposes, like developing tickets or features, you will need to create separate pipeline jobs with each pointing to a different branch. Each pipeline will have separate a Jenkins file where the pipeline script is written to download code from version control system, generate the builds and then execute either unit, integration or acceptance tests depending upon the branch type.
11. Can we distribute the test executions across various nodes through JENKINS?
Yes. In order to achieve this, you will need to set up different Jenkins nodes and specify the remote host IPs and Ports in each job's pipeline script. Once this is done, you would be able to execute the tests in parallel. Also, you can include or skip any test cases for any specific node by configuring the node while setting up the pipeline script.
12. Can you elaborate more on Squish's capability for non-qt embedded testing? An example is a WPF application with embedded winforms which plots graphics (e.g. line trace, markers).
Yes, Squish has the capability to automate both qt and non-qt embedded applications. Squish provides dedicated support for WinForms and WPF controls, which is highly beneficial when automating a Windows Presentation Foundation (WPF) application.
Squish supports both standard and complex .NET WPF (Windows Presentation Foundation) controls (Item views, menus, tabs, etc.). Since you have a special requirement for automating graph plotting, we can provide you with more details only after the feasibility analysis of your application. Connect with our team for more information.
13. Is there a benefit of using Squish over TestComplete?
It all depends upon your testing requirements. Below are some of the features that are only available in Squish:
- Squish's IDE can run on multiple operating systems
- Squish supports the automation of native macOS applications.
- Squish also supports Ruby, Perl and Tcl scripting languages, as well as other scripting languages that are also supported by TestComplete.
Interested in other webinars by QASource? Browse our collection here.
