Comparing automated visual GUI testing tools: an industrial case study

V. Garousi, Wasif Afzal, Adem Caglar, Ihsan Berk Isik, Berker Baydan , Seckin Caylak , Ahmet Zeki Boyraz, Burak Yolacan , Kadir Herkiloglu

Research output: Chapter in Book/Report/Conference proceedingConference paper

8 Citations (Scopus)

Abstract

Visual GUI testing (VGT) is a tool-driven technique, which uses image recognition for interaction and assertion of the behaviour of system under test. Motivated by a real industrial need, in the context of a large Turkish software and systems company providing solutions in the areas of defense and IT sector, we systematically planned and applied a VGT project in this industrial context. The goal of the initial phase of the project was to empirically evaluate two well-known VGT tools (Sikuli and JAutomate) to help the company select the best tool for a given testing project. Our results show that both two tools suffer from similar test ‘Replay’ problems such as the inability to find smaller-sized images. The repeatability of test executions was better for JAutomate in case of one of the two software under test (SUT) while it was comparable for the other. In terms of test development effort, for both tools, there were high correlations with number of steps in test suites, however the effort is reduced if test code is reused. The study has already provided benefits to the test engineers and managers in the company by increasing the know-how in the company w.r.t. VGT, and by identifying the challenges and their workarounds in using the tools. The industrial case study in this paper intends to add to the body of evidence in VGT and help other researchers and practitioners.
Original languageEnglish
Title of host publicationA-TEST’17: Proceedings of the 8th ACM SIGSOFT International Workshop on Automated Software Testing
EditorsTanja Vos, Sigrid Eldh, Wishnu Prasetya
Place of PublicationNews York
PublisherAssociation for Computing Machinery
Pages21-28
ISBN (Print)9781450351553
DOIs
Publication statusPublished - 2017
EventESEC/FSE'17 - Paderborn
Duration: 4 Sep 20178 Sep 2017

Workshop

WorkshopESEC/FSE'17
CityPaderborn
Period4/09/178/09/17

Fingerprint Dive into the research topics of 'Comparing automated visual GUI testing tools: an industrial case study'. Together they form a unique fingerprint.

  • Cite this

    Garousi, V., Afzal, W., Caglar, A., Berk Isik, I., Baydan , B., Caylak , S., Zeki Boyraz, A., Yolacan , B., & Herkiloglu, K. (2017). Comparing automated visual GUI testing tools: an industrial case study. In T. Vos, S. Eldh, & W. Prasetya (Eds.), A-TEST’17: Proceedings of the 8th ACM SIGSOFT International Workshop on Automated Software Testing (pp. 21-28). Association for Computing Machinery. https://doi.org/10.1145/3121245.3121250