Cloud Testing- Issues, Challenges, Needs and

Software engineering : an international Journal (SeiJ), Vol. 1, no. 1, SePteMBer 2011 9 Cloud Testing- Issues, Challenges, Needs and Practice Jerry Gao 1,2, Xiaoying Bai2, and Wei-Tek Tsai2,3 1San Jose State University, USA, 2Tsinghua University, China, and 3Arizona State University, USA jerry.gao@sjsu.edu Abstract- Cloud computing not only changes the way of obtaining computing resources (such as computers, infrastructures, data storage, and application services), but also changes the way of managing and delivering computing services, technologies, and solutions. Cloud computing leads an opportunity in offering testing as a service (TaaS) for SaaS and clouds. Meanwhile, it causes new issues, challenges and needs in software testing, particular in testing clouds and cloud-based applications. This paper provides a comprehensive tutorial on cloud testing and cloud-based application testing. It answers the common questions raised by engineers and managers, and it provides clear concepts, discusses the special objectives, features, requirements, and needs in cloud testing. It offers a clear comparative view between web-based software testing and cloud-based application testing. In addition, it examines the major issues, challenges, and needs in testing cloud-based software applications. Furthermore, it also summarizes and compares different commercial products and solutions supporting cloud testing as services. Keywords - cloud testing, cloud-based software testing, testing cloud services, performance testing and evaluation, and scalability testing. 1. INTRODUCTION Cloud computing received significant attention recently as it changes the way computation and services to customers, For example, it changes the way of providing and managing computing resources, such as CPUs, databases, and storage systems. Today, leading players, such as Amazon, Google, IBM, Microsoft, and Salesforce.com offer their cloud infrastructure for services. In 2010, Garner estimated that “the cloud service market will reach $150.1 billion in 2013”. Similarly, Merrill Lynch also predicted that “cloud computing market will reach $160 billion in 2011”. A recent study of Market Research Media forecasts that U.S. government spending on cloud computing is entering an explosive growth phase at about 40% CAGR over the next six years. Expenditure will pass $7 billion by 2015. Merrill Lynch estimates that within the next five years, the annual global market for cloud computing will surge to $95 billion. Cloud computing not only brings new business opportunities, but also causes some major impacts on software testing and maintenance. A major impact is known as Testing as a Service (TaaS) in Clouds. TaaS cloud infrastructures is considered as a new business and service model, in which a provider undertakes software testing activities of a given application system in a cloud infrastructure for customers as a service based on their demands. Although there are many published papers discussing cloud architectures, technologies, and models, design, and management, cloud testing and TaaS are still new subjects in software testing community. Hence, test engineers and quality assurance managers encountered many issues and challenges in testing modern clouds and cloud-based applications. Typical questions are listed below. • What is cloud testing? And what are its special test process and scope, requirements and features? • What types of cloud testing, environments, and forms do we need to perform for SaaSs/clouds and cloud-based applications? • What are the major differences between conventional software testing and cloud-based software testing? • What are the special requirements and distinct features of cloud-based software testing? • What are the special issues, and challenges, and needs in cloud testing? • What are the current practice, tools, and major players? This paper is written to attempt to answer these questions. Its primary contribution is to provide a comprehensive tutorial and discussion about cloud testing. The paper introduces basic concepts about cloud testing, including its scope, objectives, distinct requirements, features, and benefits. In addition, it covers software testing issues and challenges for clouds and cloud-based applications in different perspectives from end-users, vendors, and application providers. Moreover, it presents a comparative view between cloud testing and conventional software Gao et al. : Cloud testinG-issues, ChallenGes, needs and PraCtiCe10 testing. Furthermore, the paper also covers several major players, and provides a comparative view on emergent technologies and tools in cloud testing as a service. The rest of this paper is structured as follows. The next section discusses cloud testing concepts, including definitions, objectives, motivation and benefits, testing forms, types, and environments as well as activities. Section 3 is devoted to the discussion about what is new in cloud testing. It focuses on the new requirements, features, and differences between cloud testing and conventional software testing. Section 4 examines the primary issues, challenges, and needs in cloud testing. Section 5 reviews the existing emergent technologies, solutions, and tools in software cloud testing. Section 6 reviews the published papers in cloud testing and cloud-based software testing. Finally, the conclusion remarks are given in Section 7. 2. UNDERSTANDING CLOUD TESTING According to [1], cloud computing provides a cost- effective and flexible means through which scalable computing power and diverse services (computer hardware and software resources, networks and computing infrastructures), diverse application services, business processes to personal intelligence and collaboration are delivered as services to large-scale global users whenever and wherever they need. Cloud computing is the next stage of the Internet evolution. A typical cloud must have several distinct properties: elasticity and scalability, multi-tenancy, self-managed function capabilities, service billing and metering functions, connectivity interfaces and technologies. In addition, a cloud supports large scale user accesses at distributed locations over the Internet, offers on-demand application services at anytime, and provides both virtual and/or physical appliances for customers. There are three types of clouds: a) private clouds, which are internal clouds based on a private network behind a firewall; b) public clouds, which are the clouds with public accessible services over the Internet; and c) hybrid clouds, which are made of different types of clouds, including public and private clouds. 2.1 What is Cloud Testing? According to Wikipedia, "cloud testing is a form of software testing in which Web applications that leverage Cloud computing environments ("cloud") seek to simulate real-world user traffic as a means of load testing and stress testing web sites. The ability and costs to simulate Web traffic for software testing purposes has been an inhibitor to overall Web reliability." Based on our recent literature survey, there is a few of published papers addressing cloud-testing concepts, issues, and challenges. The authors in [3] collected some other views about cloud software testing from practitioners in the real world. They are listed below. “Cloud testing basically aligns with the concept of cloud and SaaS. It provides the ability to test by leveraging the cloud, thereby bringing the same benefits that the cloud brings to customers….” (by Vinita Ananth, Director - APJ Region, HP Software- as-a-Service). “Testing in the cloud leverages cloud computing environments and seeks to simulate real-world user traffic as a means of load or stress testing Web sites. (By Nivedan Prakash) “Cloud testing is the answer to the less-than- realistic performance test that originates within the infrastructure of one of our clients. When we use cloud testing, we take advantage of hardware and bandwidth that more closely mimics our observed, real world conditions. Essentially, we execute the test in cloud-based infrastructure and bandwidth.” (R V Ramanan, President – Global Delivery and Chief Software Architect, Hexaware Technologies) In short, cloud-based software testing refers to testing and measurement activities on a cloud-based environment and infrastructure by leveraging cloud technologies and solutions. It has four major objectives. • To assure the quality of cloud-based applications deployed in a cloud, including their functional services, business processes, and system performance as well as scalability based on a set of application-based system requirements in a cloud. • To validate software as a service (SaaS) in a cloud environment, including software performance, scalability, security and measurement based on certain economic scales and pre-defined SLAs. • To check the provided automatic cloud-based functional services, for example auto-provisioned functions. • To test cloud compatibility and inter-operation capability between SaaS and applications in a cloud infrastructure, for example, checking the APIs of SaaS and their cloud connectivity to others. Software engineering : an international Journal (SeiJ), Vol. 1, no. 1, SePteMBer 2011 11 2.2 Why is Cloud Testing Important? Comparing with current software testing, cloud-based testing has several unique advantages listed below. • Reduce costs by leveraging with computing resources in clouds – This refers to effectively using virtualized resources and shared cloud infrastructure to eliminate required computer resources and licensed software costs in a test laboratory. • Take the advantage of on-demand test services (by a third-party) to conduct large-scale and effective real-time online validation for internet- based software in clouds. • Easily leverage scalable cloud system infrastructure to test and evaluate system (SaaS/ Cloud/Application) performance and scalability. IBM reported the experience on cloud testing in small business division, where a flexible and cost-efficient cloud-based development and testing environment is implemented, and cloud testing has demonstrated the following major benefits in [19]. • Reduce its capital and licensing expenses as much as 50% to 75% using virtualized resources. • Reduce operating and labor costs as much as 30% to 50% by automating development and testing resource provisioning and configuration. • Shorten its development and testing setup time from weeks to minutes. • Improve product quality and reduce the detected defects by as much as 15% to 30%. • Help to accelerate cloud computing initiatives with IBMCloudBurst™ implemented through QuickStart services. 2.3. Forms of Cloud-Based Software Testing There are four different forms of cloud-based software testing. Each of them has different focuses and objectives. • Testing a SaaS in a cloud – It assures the quality of a SaaS in a cloud based on its functional and non-functional service requirements. • Testing of a cloud – It validates the quality of a cloud from an external view based on the provided cloud specified capabilities and service features. Cloud and SaaS vendors as well as end users are interested in carrying on this type of testing. • Testing inside a cloud - It checks the quality of a cloud from an internal view based on the internal infrastructures of a cloud and specified cloud capabilities. Only cloud vendors can perform this type of testing since they have accesses to internal infrastructures and connections between its internal SaaS(s) and automatic capabilities, security, management and monitor. • Testing over clouds – It tests cloud-based service applications over clouds, including private, public, and hybrid clouds based on system- level application service requirements and specifications. This usually is performed by the cloud-based application system providers. FIGURE 1 DIFFERENT VIEWS FOR CLOUD-BASED SOFTWARE TESTING Figure 1 shows three different views of software testing in a cloud environment. The first is the vendor view, which presents the testing view from the engineers of a cloud vendor. They perform vendor-oriented software testing tasks. The next is the user view, which presents the consumer-oriented testing view from cloud-based application users through web-based user interfaces. They conduct testing and QA jobs to assure the quality Gao et al. : Cloud testinG-issues, ChallenGes, needs and PraCtiCe12 of provided application services in a system-oriented test view in a given cloud infrastructure where different cloud- based applications may interact with each other. They need to perform different testing tasks to assure the quality of the cloud-based application systems over clouds, such as cloud-based application integration, end-to-end system function testing, system performance and scalability over different clouds. The detailed testing tasks are discussed in the rest part of this section. As shown in Figure 2, there are three types of cloud test environments: • A cloud-based enterprise test environment, in which application vendors deploy web-based applications in a cloud to validate their quality in a cloud infrastrutcure. • A private/public cloud test environment, in which vendors deploy SaaS applications SaaS in a private (or public) cloud to validate their quality. • A hybrid cloud test environment, in which vendors deploy cloud-based applications on a hybrid cloud infrastructure to check their quality. FIGURE 2 DIFFERENT TEST ENVIRONMENTS IN CLOUD Table 1 identifies and presents the detailed tasks in three types of cloud testing below. • Cloud/SaaS-oriented testing - This type of testing activities usually is performed inside a cloud by engineers of cloud/SaaS vendors. The primary objective is to assure the quality of the provided service functions offered in a cloud (or a SaaS program). These engineers must go through unit testing, integration, system function validation and regression testing, as well as performance and scalability evaluation. Since clouds and SaaS usually provide certain service APIs and connectivity interfaces to their customers, it is required task for engineers to validate these APIs and connectivity in a cloud environment. In addition, testing cloud-based or SaaS-based security services and functional features must be tested. Furthermore, performance testing and scalability evaluation in a cloud is very important and critical to cloud/SaaS vendors because this assures the quality of cloud elasticity to support SaaS and cloud services inside a cloud. Table 1 shows the detailed testing tasks. • Online-based application testing on a cloud - This type of testing activities usually is performed to check online application systems on a cloud by using with cloud-based large-scale traffic and user accesses. This is a common usage of cloud technology to help current online application vendors to conduct online-based system function testing and performance evaluation on a cloud by taking the advantage of cloud environment so that diverse and scalable computing resources in a cloud can be used without using any in- house test laboratory. When applications are connected with legacy systems, the quality of the connectivity between the legacy systems and the under-test application deployed on a cloud must be validated. Table 1 shows the detailed tasks. • Cloud-based application testing over clouds - This type of testing refers to the engineering activities performed to assure the quality of a cloud-based application crossing different clouds. When applications are developed to be deployed and executed over different clouds, new testing tasks are needed to assure its quality. Unlike the previous two types, the primary testing objective here is to assure the quality of the end-to-end application over clouds. This suggests that the system-level integration, function validation, performance evaluation, and scalability measurement must cope with different cloud technologies. This definitely complicates the tasks for checking system compatibility, interoperability, and connectivity between different clouds. Table 1 shows the detailed tasks. Software engineering : an international Journal (SeiJ), Vol. 1, no. 1, SePteMBer 2011 13 table 1 COMPARATIVE VIEW AMONG DIFFERENT PARTIES Test Type Testing focuses Cloud/SaaS-Oriented Testing inside a Cloud Online Application-Based Testing on a Cloud Cloud-Based Application Testing over Clouds Service Function Testing GUI-based and API- based service functions Testing SaaS/Cloud- based service functions inside a cloud Testing online-based application service functions on a cloud Testing cloud-based application service functions over a cloud infrastructure Integration Testing SaaS interactions and Cloud connections Vendor-specific component and service integration inside a private/public cloud Integration between online clients and back-end servers on a cloud - End-to-end application integration over clouds - Integration with legacy systems over clouds API and Connectivity Testing API interfaces and connectivity protocols (HTTPS, REST, SOAP, RMI) SaaS/Cloud API & connectivity testing in a cloud Testing user-centered service APIs and connectivity on a cloud Testing application service APIs and connectivity over clouds Performance & Scalability Testing Performance and scalability based on a SLA SaaS/Cloud performance and scalability testing in a cloud based on the given SLA User-oriented application performance and scalability testing on a cloud End-to-end system-level performance and scalability inside/on/over cloud based on a given SLA Security Testing SaaS/Application data, processes, functions, and user privacy SaaS/Cloud security features and user privacy in a cloud User-oriented security and privacy on a cloud System-level end-to-end security over clouds Interoperability & Compatibility Testing Validate different client interfaces and technologies and diverse compatibilities on different platforms and browsers Testing Cloud/ SaaS compatibility, connectivity protocols and UI/client technologies inside a cloud Testing user-centered interoperability, compatibility of platforms/ OS/browsers, and client technologies on a cloud Testing application compatibility, end-to- end interoperability and application connectivity to legacy systems over clouds Regression Testing Changed & impacted SaaS/Cloud service features and related APIs/ connectivity Cloud/SaaS-oriented regression testing inside a cloud User-centered re-validation on a cloud End-to-end application system regression over clouds FIGURE 3. THE SCOPE OF CLOUD-BASED SOFTWARE TESTING Test/QoS REQs & SLAs Test Modeling and Adequacy Price Modeling & Billing Test Simulation & Generation Frameworks and Tools On-Demand TaaS Service Test Process & Management Gao et al. : Cloud testinG-issues, ChallenGes, needs and PraCtiCe14 3. WHAT ARE THE NEW FEATURES IN CLOUD TESTING? Unlike testing conventional web-based software, testing clouds and cloud-based software has several unique testing quality assurance objectives, requirements, and distinct features. This section is dedicated to these subjects. Figure 3 shows the primary contents in the scope of cloud testing. 3.1. New Requirements and Features in Cloud Testing There are four new requirements and features in cloud testing. • Cloud-based testing environment – This refers to use a selected cloud infrastru