A Mapping Study of Scientific Merit of Papers, which Subject are Web Applications Test Techniques, Considering their Validity Threats

Nascimento, Arnaldo Gil Sardella; Schmitz, Eber; Silva, Monica; Lima, Priscila

doi:10.4301/S1807-1775201815006

Acessibilidade / Reportar erro

Brasil

JISTEM - Journal of Information Systems and Technology Management

Español English

Brasil

Español English

sumário « anterior atual seguinte »

Sumário

Original Article • JISTEM J.Inf.Syst. Technol. Manag. 15 • 2018 • https://doi.org/10.4301/S1807-1775201815006 copy

A Mapping Study of Scientific Merit of Papers, which Subject are Web Applications Test Techniques, Considering their Validity Threats

Authorship SCIMAGO INSTITUTIONS RANKINGS

Abstract

Progress in software engineering requires (1) more empirical studies of quality, (2) increased focus on synthesizing evidence, (3) more theories to be built and tested, and (4) the validity of the experiment is directly related with the level of confidence in the process of experimental investigation. This paper presents the results of a qualitative and quantitative classification of the threats to the validity of software engineering experiments comprising a total of 92 articles published in the period 2001-2015, dealing with software testing of Web applications. Our results show that 29.4% of the analyzed articles do not mention any threats to validity, 44.2% do it briefly, and 14% do it judiciously; that leaves a question: these studies have scientific value?

Keywords
Applications; Conclusion; Construction; External; Internal; Literature; Mapping; Qualitative; Quantitative; Reviews; Study; Systematic; Tests; Threats; Validity; Web

Threat	External	Internal	Construction	Conclusion
Frequency	45 (48.9%)	36 (39.1%)	24 (26.1%)	10 (10.9%)
Papers	E1, E2, E4, E5, E6, E7, E8, E9, E10, E11, E12, E13, E14, E15, E16, E18, E19, E20, E21, E23, E24, E25, E26, E27, E29, E61, E63, E65, E67, E68, E72, E75, E76, E80, E81, E82, E83, E84, E86, E88, E90, E91, E92, E94, E95.	E1, E2, E4, E5, E6, E7, E8, E9, E10, E11, E12, E13, E14, E15, E16, E18, E19, E21, E23, E24, E25, E27, E29, E65, E67, E68, E72, E75, E76, E80, E81, E86, E88, E91, E94, E95.	E1, E2, E4, E7, E8, E10, E11, E12, E14, E15, E16, E24, E27, E29, E61, E63, E64, E65, E68, E72, E75, E76, E90, E91.	E6, E7, E11, E12, E14, E24, E27, E65, E75, E90.

Threat	External	Internal	Construction	Conclusion
Summarized threats	39 (60%)	31 (47.7%)	17 (26.2%)	9 (13.8%)
Detailed threats	7 (10.8%)	7 (10.8%)	7 (10.8%)	1 (1.5%)
Papers with summarized threats	E2, E4, E5, E6, E7, E8, E9, E10, E11, E12, E13, E14, E15, E18, E21, E23, E24, E25, E27, E29, E63, E67, E68, E72, E74, E75, E76, E80, E81, E82, E83, E84, E86, E88, E90, E91, E92, E94, E95.	E1, E2, E4, E5, E6, E7, E8, E9, E10, E11, E12, E13, E14, E15, E18, E19, E21, E23, E24, E29, E67, E68, E73, E74, E76, E80, E81, E88, E91, E94, E95.	E2, E4, E7, E8, E10, E11, E12, E14, E24, E29, E63, E68, E72, E75, E76, E90, E91.	E6, E7, E11, E12, E14, E24, E27, E75, E90.
Papers with detailed threats	E1, E16, E19, E20, E26, E61, E65.	E16, E19, E27, E65, E72, E75, E86.	E1, E15, E16, E27, E61, E64, E65.	E65.

Year
2001	2002	2004	2005	2006	2007	2008	2009	2010	2011	2012	2013	2014	2015
Frequency of articles that reference threats to validity
1 (1.5%)	0 (0.0%)	0 (0.0%)	2 (3.0%)	1 (1.5%)	2 (3.0%)	2 (3.0%)	1 (1.5%)	7 (10.8%)	4 (6.1%)	3 (4.6%)	4 (6.1%)	29 (44.6%)	9 (13.8%)
Papers
E1.	--	--	E6, E19.	E27.	E14, E17.	E15, E22.	E24.	E2, E4, E7, E8, E20, E21, E23.	E3, E9, E12, E16.	E5, E10, E18.	E11, E13, E28, E29	E25, E26, E60, E61, E64, E65, E66, E67, E68, E69, E70, E72, E74, E76, E78, E79, E81, E82, E84, E85, E86, E87, E89, E90, E91, E92, E93, E94, E95.	E62, E63, E71, E73, E75, E77, E80, E83, E88.
Frequency of articles that do not reference threats to validity
0 (0.0%)	1 (3.7%)	2 (7.4%)	2 (7.4%)	3 (11.1%)	0 (0.0%)	3 (11.1%)	3 (11.1%)	5 (18.5%)	4 (14.8%)	2 (7.4%)	1 (3.7%)	1 (3.7%)	0 (0.0%)
Papers
--	E40.	E30, E39.	E35, E43.	E32, E36, E50.	--	E37, E44, E59.	E38, E49, E51.	E33, E34, E42, E53, E58.	E52, E54, E55, E57.	E41, E46.	E48.	E56.	--

Type of Research	Empirical Study	Sistematic Literature Review	Mapping Study	Survey Study	Qualitative Survey
Frequency	83	4	3	1	1
Papers	E1, E3, E5, E6, E7, E8, E9, E10, E11, E12, E13, E14, E15, E16, E17, E18, E19, E20, E21, E22, E23, E25, E26, E27, E28, E29, E30, E32, E33, E34, E35, E36, E37, E38, E39, E40, E41, E42, E43, E44, E46, E48, E49, E50, E51, E53, E54, E57, E58, E59, E60, E61, E63, E64, E65, E66, E67, E68, E69, E70, E71, E72, E73, E74, E76, E77, E78, E79, E80, E81, E82, E83, E84, E85, E86, E87, E88, E90, E91, E92, E93, E94, E95.	E24, E4, E52, E62.	E55, E75, E89.	E56.	E2.

Empirical Study
Threat	External	Internal	Construction	Conclusion
Frequency	42	26	21	10
Papers	E1, E5, E6, E7, E8, E9, E10, E11, E12, E13, E14, E15, E16, E18, E19, E20, E21, E23, E25, E26, E27, E29, E61, E63, E65, E67, E68, E72, E75, E76, E80, E81, E82, E83, E84, E86, E88, E90, E91, E92, E94, E95.	E1, E5, E6, E7, E8, E9, E10, E11, E12, E14, E15, E16, E27, E29, E65, E67, E68, E72, E75, E76 E80, E81, E86, E88, E94, E95.	E1, E7, E8, E10, E11, E12, E14, E15, E16, E27, E29, E61, E63, E64, E65, E68, E72, E75, E76, E90, E91.	E6, E7, E11, E12, E14, E27, E65, E75, E90, E91.
Systematic Literature Review
Threat	External	Internal	Construction	Conclusion
Frequency	2	2	2	1
Papers	E4, E24.	E4, E24.	E4, E24.	E24.
Mapping Study
Threat	External	Internal	Construction	Conclusion
Frequency	1	1	1	1
Papers	E75.	E75.	E75.	E75.
Survey Study
Threat	External	Internal	Construction	Conclusion
Frequency	0	0	0	0
Papers	--	--	--	--
Qualitative Survey
Threat	External	Internal	Construction	Conclusion
Frequency	1	1	1	0
Papers	E2.	E2.	E2.	--

TECSI Laboratório de Tecnologia e Sistemas de Informação - FEA/USP Av. Prof. Luciano Gualberto, 908 FEA 3, 05508-900 - São Paulo/SP Brasil, Tel.: +55 11 2648 6389, +55 11 2648 6364 - São Paulo - SP - Brazil
E-mail: jistemusp@gmail.com

Acompanhe os números deste periódico no seu leitor de RSS

[1] Address for correspondence: Arnaldo Gil Sardella Nascimento. Universidade Federal do Rio de janeiro, Rio de Janeiro, Brazil. Email: agsardella@gmail.com

[E1]	BIBLE, JOHN, GREGG ROTHERMEL, and DAVID S. ROSENBLUM (2001). A Comparative Study of Coarse-and Fine-Grained Safe Regression Test-Selection Techniques. ACM Transactions on Software Engineering and Methodology 10.2: 149-183.
[E2]	ENGSTRÖM, EMELIE, and PER RUNESON (2010). A qualitative survey of regression testing practices. Product-Focused Software Process Improvement. 3-16. Berlin, Springer Berlin Heidelberg.
[E3]	DA MOTA SILVEIRA NETO, PAULO ANSELMO, et al. (2011). A systematic mapping study of software product lines testing. Information and Software Technology 53.5: 407-423.
[E4]	ENGSTRÖM, EMELIE, PER RUNESON, and MATS SKOGLUND (2010). A systematic review on regression test selection techniques. Information and Software Technology 52.1: 14-30.
[E5]	MARBACK, AARON, HYUNSOOK DO, and NATHAN EHRESMANN (2012). An Effective Regression Testing Approach for PHP Web Applications. Software Testing, Verification and Validation (ICST), IEEE Fifth International Conference. 221-230.
[E6]	SPRENKLE, SARA, et al. (2005). An empirical comparison of test suite reduction techniques for user-session-based testing of web applications. Software Maintenance, ICSM’05. Proceedings of the 21st IEEE International Conference.
[E7]	ENGSTRÖM, E., PER RUNESON, and GREGER WIKSTRAND (2010). An empirical evaluation of regression testing based on fix-cache recommendations. Software Testing, Verification and Validation (ICST), Third International Conference.
[E8]	PRAPHAMONTRIPONG, UPSORN, and JEFF OFFUTT (2010). Applying mutation testing to web applications. Software Testing, Verification, and Validation Workshops (ICSTW), Third International Conference.
[E9]	MESBAH, ALI, and MUKUL R. PRASAD (2011). Automated cross-browser compatibility testing. Proceedings of the 33rd International Conference on Software Engineering. ACM.
[E10]	DALLMEIER, VALENTIN, et al. (2012). Automatically generating test cases for specification mining. Software Engineering, IEEE Transactions 38.2: 243-257.
[E11]	LEOTTA, MAURIZIO, et al. (2013). Capture-replay vs. programmable web testing: An empirical assessment during test case evolution. Reverse Engineering (WCRE), 20th Working Conference. IEEE.
[E12]	MARCHETTO, ALESSANDRO, et al. (2011). Crawlability metrics for automated web testing. International journal on software tools for technology transfer 13.2: 131-149.
[E13]	MIRSHOKRAIE, SHABNAM, ALI MESBAH, and KARTHIK PATTABIRAMAN (2013). Efficient JavaScript mutation testing. Software Testing, Verification and Validation (ICST), IEEE Sixth International Conference. IEEE.
[E14]	MARCHETTO, ALESSANDRO, FILIPPO RICCA, and PAOLO TONELLA (2007). Empirical validation of a web fault taxonomy and its usage for fault seeding. Web Site Evolution. WSE 2007. 9th IEEE International Workshop.
[E15]	ARTZI, SHAY, et al. (2008). Finding bugs in dynamic web applications. Proceedings of the 2008 international symposium on Software testing and analysis. ACM.
[E16]	ENGSTROM, EMELIE, PER RUNESON, and ANDREAS LJUNG (2011). Improving Regression Testing Transparency and Efficiency with History-Based Prioritization--An Industrial Case Study. Software Testing, Verification and Validation (ICST), 2011 IEEE Fourth International Conference. IEEE.
[E17]	HALFOND, WILLIAM GJ, and ALESSANDRO ORSO (2007). Improving test case generation for web applications using automated interface discovery. Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering. ACM.
[E18]	MESBAH, ALI, ARIE VAN DEURSEN, and DANNY ROEST (2012). Invariant-based automatic testing of modern web applications. Software Engineering, IEEE Transactions 38.1: 35-53.
[E19]	ELBAUM, SEBASTIAN, et al. (2005). Leveraging user-session data to support web application testing. Software Engineering, IEEE Transactions 31.3: 187-202.
[E20]	OFFUTT, JEFF, and YE WU (2010). Modeling presentation layers of web applications for testing. Software & Systems Modeling 9.2: 257-280.
[E21]	LI, NUO, et al. (2010). Perturbation-based user-input-validation testing of web applications. Journal of Systems and Software 83.11: 2263-2274.
[E22]	SAMPATH, SREEDEVI, et al. (2008). Prioritizing user-session-based test cases for web applications testing. Software Testing, Verification, and Validation, 2008 1st International Conference. IEEE.
[E23]	ROEST, DANNY, ALI MESBAH, and ARIE VAN DEURSEN (2010). Regression testing ajax applications: Coping with dynamism. Software Testing, Verification and Validation (ICST), 2010 Third International Conference. IEEE.
[E24]	AFZAL, WASIF, RICHARD TORKAR, and ROBERT FELDT (2009). A systematic review of search-based testing for non-functional system properties. Information and Software Technology 51.6: 957-976.
[E25]	MAEZAWA, YUTA, et al. (2014). Validating ajax applications using a delay-based mutation technique. Proceedings of the 29th ACM/IEEE international conference on Automated software engineering. ACM.
[E26]	COLOMBO, CHRISTIAN, MARK MICALLEF, and MARK SCERRI (2014). Verifying Web Applications: From Business Level Specifications to Automated Model-Based Testing. arXiv preprint arXiv1403.7258.
[E27]	SAMPATH, SREEDEVI, et al. (2006). Web application testing with customized test requirements-An experimental comparison study. Software Reliability Engineering, 2006. ISSRE’06. 17th International Symposium. IEEE.
[E28]	DOĞAN, SERDAR, AYSU BETIN-CAN, and VAHID GAROUSI (2014). Web application testing: A systematic literature review. Journal of Systems and Software 91: 174-201.
[E29]	DALLMEIER, VALENTIN, et al. (2013). WebMate: Generating test cases for web 2.0. SWQD 133: 55-69.
[E30]	TONELLA, PAOLO, and FILIPPO RICCA (2004). A 2-layer model for the white-box testing of web applications. Telecommunications Energy Conference, 2004. INTELEC 2004. 26th Annual International. IEEE.
[E31]	BARADHI, GHINWA, and NASHAT MANSOUR (1997). A comparative study of five regression testing algorithms. Software Engineering Conference, 1997. Proceedings. Australian. IEEE.
[E32]	ZHENG, JIANG, et al. (2006). A lightweight process for change identification and regression test selection in using cots components. Commercial-off-the-Shelf (COTS)-Based Software Systems, 2006. Fifth International Conference. IEEE.
[E33]	SAXENA, PRATEEK, et al. (2010). A symbolic execution framework for javascript. Security and Privacy (SP), 2010 IEEE Symposium. IEEE.
[E34]	DOBOLYI, KINGA (2010). An Exploration of User-Visible Errors in Web-based Applications to Improve Web-based Applications. Dissertation. University of Virginia.
[E35]	ZHENG, JIANG, et al. (2005). An initial study of a lightweight process for change identification and regression test selection when source code is not available. Software Reliability Engineering, 2005. ISSRE 2005. 16th IEEE International Symposium. IEEE.
[E36]	ZHENG, JIANG, et al. (2006). Applying regression test selection for COTS-based applications. Proceedings of the 28th international conference on Software engineering. ACM.
[E37]	HALFOND, WILLIAM GJ, and ALESSANDRO ORSO (2008). Automated identification of parameter mismatches in web applications. Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering. ACM.
[E38]	RAN, LIHUA, et al. (2009). Building test cases and oracles to automate the testing of web database applications. Information and Software Technology 51.2: 460-477.
[E39]	OFFUTT, JEFF, et al. (2004). Bypass testing of web applications. Software Reliability Engineering, 2004. ISSRE 2004. 15th International Symposium. IEEE.
[E40]	YANG, JI-TZAY, et al. (2002). Constructing and Object-Oriented Architecture for Web Application Testing. Journal of Computing and Information Science in Engineering 18.1: 59-84.
[E41]	CHOUDHARY, SHAUVIK ROY, MUKUL R. PRASAD, and ALESSANDRO ORSO (2012). Crosscheck: Combining crawling and differencing to better detect cross-browser incompatibilities in web applications. Software Testing, Verification and Validation (ICST), 2012 IEEE Fifth International Conference. IEEE.
[E42]	HALLÉ, SYLVAIN, et al. (2010). Eliminating navigation errors in web applications via model checking and runtime enforcement of navigation state machines. Proceedings of the IEEE/ACM international conference on Automated software engineering. ACM.
[E43]	BAJWA, HARPREET, WENLIANG XIONG, and FRANK MAURER (2005). Evaluating current testing processes of web-portal applications. Web Engineering. 603-605. Springer Berlin Heidelberg.
[E44]	RAFFELT, HARALD, et al. (2008). Hybrid test of web applications with webtest. Proceedings of the 2008 workshop on Testing, analysis, and verification of web services and applications. ACM.
[E45]	AGRAWAL, H., HORGAN, J. R., KRAUSER, E. W. and LONDON, S. (1993). Incremental Regression Testing. ICSM 93: 348-357.
[E46]	MIRSHOKRAIE, SHABNAM, and ALI MESBAH (2012). JSART: JavaScript assertion-based regression testing. Web Engineering. 238-252. Springer Berlin Heidelberg.
[E47]	KUNG, DAVID C., et a.l (1996). On regression testing of object-oriented programs. Journal of Systems and Software 32.1: 21-40.
[E48]	SAKAMOTO, KAZUNORI, et al. (2013). POGen: a test code generator based on template variable coverage in gray-box integration testing for web applications. Fundamental Approaches to Software Engineering. 343-358. Berlin, Springer Berlin Heidelberg.
[E49]	HALFOND, WILLIAM GJ, SASWAT ANAND, and ALESSANDRO ORSO (2009). Precise interface identification to improve testing and analysis of web applications. Proceedings of the eighteenth international symposium on Software testing and analysis. ACM.
[E50]	BRYCE, RENÉE C., and CHARLES J. COLBOURN (2006). Prioritized interaction testing for pair-wise coverage with seeding and constraints. Information and Software Technology 48.10: 960-970.
[E51]	CHITTIMALLI, PAVAN KUMAR, and MARY JEAN HARROLD (2009). Recomputing coverage information to assist regression testing. Software Engineering, IEEE Transactions 35.4: 452-469.
[E52]	SOUZA, SIMONE RS, et al. (2011). Research in concurrent software testing: A systematic review. Proceedings of the Workshop on Parallel and Distributed Systems: Testing, Analysis, and Debugging. ACM.
[E53]	ANDREWS, ANNELIESE A., et al. (2010). Scalability issues with using FSMWeb to test web applications. Information and Software Technology 52.1: 52-66.
[E54]	ZHENG, YUNHUI, TAO BAO, and XIANGYU ZHANG (2011). Statically locating web application bugs caused by asynchronous calls. Proceedings of the 20th international conference on World wide web. ACM.
[E55]	PALACIOS, MARCOS, JOSÉ GARCÍA-FANJUL, and JAVIER TUYA (2011). Testing in Service Oriented Architectures with dynamic binding: A mapping study. Information and Software Technology 53.3: 171-189.
[E56]	LI, YUAN-FANG, PARAMJIT K. DAS, and DAVID L. DOWE (2014). Two decades of Web application testing - A survey of recent advances. Information Systems 43: 20-54.
[E57]	PENG, XUAN, AND LU LU (2011). User-session-based automatic test case generation using GA. International Journal of Physical Sciences 6.13: 3232-3245.
[E58]	CHOUDHARY, SHAUVIK ROY, HUSAYN VERSEE, and ALESSANDRO ORSO (2010). WEBDIFF: Automated identification of cross-browser issues in web applications. Software Maintenance (ICSM), 2010 IEEE International Conference. IEEE.
[E59]	SPRENKLE, SARA, et al. (2008). WebVizOr: a visualization tool for applying automated oracles and analyzing test results of web applications. Practice and Research Techniques, 2008. TAIC PART’08. Testing: Academic & Industrial Conference. IEEE.
[E60]	JAVED, ABU (2014). A Model-Driven Framework for Context-Dependent Component Testing. PhD Thesis. School of Information Technology and Electrical Engineering. The University of Queensland.
[E61]	AMANNEJAD, YASAMAN, et al. (2014). A search-based approach for cost-effective software test automation decision support and an industrial case study. Software Testing, Verification and Validation Workshops (ICSTW), 2014 IEEE Seventh International Conference. IEEE.
[E62]	JAIN, RITU, and UGRASEN SUMAN (2015). A Systematic Literature Review on Global Software Development Life Cycle. ACM SIGSOFT Software Engineering Notes 40.2: 1-14.
[E63]	ORTAC, ALPER, MARTIN MONPERRUS, and MIRA MEZINI (2015). Abmash: mashing up legacy Web applications by automated imitation of human actions. Software: Practice and Experience 45.5: 581-612.
[E64]	DEVROEY, XAVIER, GILLES PERROUIN, and PIERRE-YVES SCHOBBENS (2014). Abstract test case generation for behavioural testing of software product lines. Proceedings of the 18th International Software Product Line Conference: Companion Volume for Workshops, Demonstrations and Tools-Volume 2. ACM.
[E65]	RODRIGUES, E. M., DE OLIVEIRA, F. M., COSTA, L. T., BERNARDINO, M., ZORZO, A. F., SOUZA, S. D. R. S., and SAAD, R. (2014). An empirical comparison of model-based and capture and replay approaches for performance testing. Empirical Software Engineering 20.6: 1831-1860.
[E66]	HOQUE, MOHAMMAD IFTEKHARUL, et al. (2014). An Empirical Study on Refactoring Activity. arXiv preprint arXiv:1412.6359.
[E67]	ABURAS, ALI, and ALEX GROCE (2014). An improved memetic algorithm with method dependence relations (MAMDR). Quality Software (QSIC), 2014 14th International Conference. IEEE.
[E68]	NGUYEN, BAO N., and ATIF M. MEMON (2014). An observe-model-exercise* paradigm to test event-driven systems with undetermined input spaces. Software Engineering, IEEE Transactions 40.3: 216-234.
[E69]	TAPPENDEN, ANDREW F., and JAMES MILLER (2014). Automated cookie collection testing. ACM Transactions on Software Engineering and Methodology (TOSEM) 23.1: 3.
[E70]	GARG, DEEPAK (2014). Automated Techniques to Detect Faults Early in Large Software Applications. PhD thesis. The University of Western Australia.
[E71]	SHAHBAZ, MUZAMMIL, PHIL MCMINN, and MARK STEVENSON (2015). Automatic generation of valid and invalid test data for string validation routines using web searches and regular expressions. Science of Computer Programming 97: 405-425.
[E72]	MARIANI, LEONARDO, et al. (2014). Automatic testing of GUI based applications. Software Testing, Verification and Reliability 24.5: 341-366.
[E73]	SHEN, DU, et al. (2015). Automating performance bottleneck detection using search-based application profiling. Proceedings of the 2015 International Symposium on Software Testing and Analysis. ACM.
[E74]	MARAS, JOSIP (2014). Automating Reuse in Web Application Development. PhD thesis. School of Innovation, Design and Engineering. Mälardalen University. Sweden.
[E75]	ZHI, JUNJI, et al. (2015). Cost, benefits and quality of software development documentation: A systematic mapping. Journal of Systems and Software 99: 175-198.
[E76]	ALSHAHWAN, NADIA, and MARK HARMAN (2014). Coverage and fault detection of the output-uniqueness test selection criteria. Proceedings of the 2014 International Symposium on Software Testing and Analysis. ACM.
[E77]	MAHAJAN, SONAL, and WILLIAM GJ HALFOND (2015). Detection and localization of html presentation failures using computer vision-based techniques. Software Testing, Verification and Validation (ICST), 2015 IEEE 8th International Conference. IEEE.
[E78]	MIRZAAGHAEI, MEHDI, AND ALI MESBAH (2014). DOM-based test adequacy criteria for web applications. Proceedings of the 2014 International Symposium on Software Testing and Analysis. ACM.
[E79]	CHAUDHARY, NEHA, O. P. SANGWAN, and RICHA ARORA (2014). Event-Coverage and Weight based Method for Test Suite Prioritization. International Journal of Information Technology and Computer Science (IJITCS) 6.12: 61.
[E80]	MIRSHOKRAIE, SHABNAM, ALI MESBAH, and KARTHIK PATTABIRAMAN (2015). Guided Mutation Testing for JavaScript Web Applications. Software Engineering, IEEE Transactions 41.5 429-444.
[E81]	JU, XIAOLIN, et al. (2014). HSFal: Effective fault localization using hybrid spectrum of full slices and execution slices. Journal of Systems and Software 90: 3-17.
[E82]	TONELLA, PAOLO, ROBERTO TIELLA, and CU DUY NGUYEN (2014). Interpolated n-grams for model based testing. Proceedings of the 36th International Conference on Software Engineering. ACM.
[E83]	MIRSHOKRAIE, SHABNAM, ALI MESBAH, and KARTHIK PATTABIRAMAN (2015). JSeft: automated JavaScript unit test generation. Software Testing, Verification and Validation (ICST), 2015 IEEE 8th International Conference. IEEE.
[E84]	MILANI FARD, AMIN, MEHDI MIRZAAGHAEI, and ALI MESBAH (2014). Leveraging existing tests in automated test generation for web applications. Proceedings of the 29th ACM/IEEE international conference on Automated software engineering. ACM.
[E85]	FLOSS, BRIANNA (2014). Overcoming Hard Problems in Software Testing with Testing as a Service. Dissertation. Florida Institute of Technology.
[E86]	ZAREI, ALIREZA (2014). Performance improvements in crawling modern Web applications. Dissertation. The Faculty of Graduate and Postdoctoral Studies. The University of British Columbia.
[E87]	CHRISTOPHE, LAURENT, et al. (2014). Prevalence and maintenance of automated functional tests for web applications. Software Maintenance and Evolution (ICSME), 2014 IEEE International Conference. IEEE.
[E88]	SHAMSHIRI, SINA, et al. (2015). Random or Genetic Algorithm Search for Object-Oriented Test Suite Generation? Proceedings of the 2015 on Genetic and Evolutionary Computation Conference. ACM.
[E89]	DENG, DONGCHENG (2014). Reliable Embedded Systems Development. Dissertation. University of Calgary.
[E90]	MADHAVAN, ARVIND (2014). Semi Automated User Acceptance Testing using Natural Language Techniques. Dissertation. Iowa State University.
[E91]	KRUSE, PETER M., JIRKA NASAREK, and N. CONDORI FERNANDEZ (2014). Systematic Testing of Web Applications with the Classification Tree Method. XVII Iberoamerican Conference on Software Engineering (CIbSE 2014).
[E92]	MAYO, QUENTIN, RYAN MICHAELS, and RENEE BRYCE (2014). Test Suite Reduction by Combinatorial-Based Coverage of Event Sequences. Software Testing, Verification and Validation Workshops (ICSTW), 2014 IEEE Seventh International Conference. IEEE.
[E93]	SEQUEIRA, SHELDON (2014). Understanding web application test assertion failures. Dissertation. The Faculty of Graduate and Postdoctoral Studies. The University of British Columbia.
[E94]	GHIDUK, AHMED S (2014). Using Evolutionary Algorithms for Higher-Order Mutation Testing. International Journal of Computer Science Issues (IJCSI) 11.2: 93.
[E95]	OCARIZA JR, FROLIN S., KARTHIK PATTABIRAMAN, and ALI MESBAH (2014). Vejovis: Suggesting fixes for JavaScript faults. Proceedings of the 36th International Conference on Software Engineering. ACM.