Cloud - Test strategies & Different cloud models

Cloud as a technology is no longer new. Many organizations are openly adopting cloud based services and many others are migrating existing services to cloud. This change or adoption of cloud does raise many questions – “what are the important factors impacting cloud testing?”, “how should we test the success of migrating to cloud?”, etc.

Important factors impacting cloud testing
Cloud does not change the traditional testing process that is adopted by different organizations. The fundamental test process consisting of test planning and control, test analysis and design, test implementation and execution, evaluating existing criteria, and reporting and test closure activities is equally applicable in the cloud context. Before we start, it is important to look at cloud definition and its characteristics.

The National Institute of Standards and Technology (NIST), USA clearly defines the essential characteristics of cloud computing. Cloud testing derives its test strategy based on this definition. According to NIST, “cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management of effort or service provider interaction.”

The cloud computing model promotes availability and is composed of five essential characteristics (on-demand self-service, broad network access, resource pooling, rapid elasticity and measured service); three service models [Cloud Software as a Service (SaaS), Cloud Platform as a Service (PaaS), and Cloud Infrastructure as a Service (IaaS)]; and four deployment models (private cloud, community cloud, public cloud, and hybrid cloud). Key enabling technologies of this model include: (1) fast wide-area networks, (2) powerful, inexpensive server computers, and (3) high-performance virtualization for commodity hardware.

Test strategy is significantly impacted as a result of one or more of these essential characteristics, service models and deployment models. This essentially means that while the business process testing (or functional test strategy) will more or less remain the same, irrespective of cloud or web based hosting model, non-functional aspects of testing like application performance testing, security testing, interface testing, compatibility testing and test environment management (environment provisioning, environment availability and environment release) are the key areas that will get impacted (will differ from web) while switching over to cloud.

Another important factor determining test approach is the software development model. Generally one of the following three software development models is seen. Each model is unique and depending on the model, test strategy will undergo a change.

• Cloud is used only to host production application: In this case the quantum of testing is very much restricted since any test will directly impact production. Any test that requires interaction with system’s DB (login validation, data entry, data modification, etc.) will not be allowed (though risky and not advisable, in certain situations, test logins are created on the production and shared with a restricted audience to run E2E functional tests).
• Development happens outside cloud environment, while testing and deployment happen on the cloud: In this scenario, the code is built in-house and deployed on a test server hosted on the cloud. Here, it is essential to understand whether all interfaces to the application are deployed on the cloud or whether the test server has to connect to certain applications outside the cloud environment. Interface testing is a critical focus area as defects can reside at the interfacing points.
• Development, testing and deployment are all done on cloud: While this model is yet to be widely embraced by software organizations (data security, privacy and complete dependence on cloud provider are causes of concern), this model works seamlessly. From a testing perspective, this is the best model as the team need not validate in different environments. Full-fledged “Continuous Integration Testing (CIT)” runs through automated tests, and performance testing can be seamlessly executed.

ConclusionFunctionally, there will be no change in the test approach between web based and cloud environment. However, test teams should focus on certain aspects like performance, security, environment availability, integration with interfaces, customer experience and adherence to SLAs when testing in a cloud based environment.

Test strategy undergoes changes depending on the service models and implementation models selected. Choice of tools and techniques will be similar as in the case of web based applications, however, during tool selection the fact that they are available on cloud and they support the pay-as-you-use model should be considered.

If you have already migrated solutions to the cloud, we would like to hear your thoughts and experiences around some of the factors and challenges you faced while testing cloud solutions. Please feel free to share your comments. Come back to this space for my forthcoming blogs on the various test strategies, test environments and types of testing that organizations need to consider for successful cloud adoption.

Test strategies for different cloud models – An overview

Here, I’ve elaborated the different aspects of testing that needs to be considered while migrating to the cloud. However, I would like to point out that in a way what is mentioned below is a generic test strategy and does not cover every possible test dimension. Depending on situations and the environment, this strategy should be modified.

• Infrastructure as a Service (IaaS): From the testing perspective, if the sole purpose of cloud migration is to utilize IaaS capabilities, then focus should be more on testing application performance (availability, scalability, elasticity and similar features). This is similar to a web based testing team having a good quality test environment that is always available and can be virtualized with ease.
• Platform as a Service (PaaS): Test strategy or approach will not change significantly between IaaS and PaaS. Since PaaS makes available a development environment (platform for application development on the cloud along with the ability to virtualize the environment and the supported hardware), the only difference between IaaS and PaaS is the need for interface testing which is more pronounced in PaaS compared to IaaS. Since applications using PaaS inherit cloud characters, tests to validate scalability, high-availability, and multitenancy should be run.
• Software as a Service (SaaS): SaaS offers ready to use applications as a service. From an end user perspective, in most cases, no download of software is needed to access SaaS applications (sometimes users, however, may need to download a plugin). Access to the software is generally governed through an authentication and authorization mechanism. “Pay-as-you-use” model is the default standard for customers utilizing SaaS applications. From a testing perspective, SaaS environment is where the full spectrum of testing is needed. This includes, testing for upgrades, installation, configurability, multitenancy, backward compatibility, metered usage (testing different subscription models), security, compliance, legacy integration, reliability, data integrity, privacy, scalability, elasticity, SLA’s and user experience. It should be noted that, irrespective of SaaS being a delivery model, customers will not compromise the functional accuracy because of SaaS. In most cases, traditional testing tools will be sufficient while testing SaaS based application.

Functionally, there will be not much of a change in the test approach between a web based and cloud environment. However, the test teams should focus on certain aspects like performance, security, environment availability, integration with interfaces, customer experience and adherence to SLAs while testing in a cloud based environment.

Depending on the service models and implementation models selected, the test strategy will undergo changes. The choice of tools and techniques will be similar as in the case of web based applications, however during tool selection one should consider the fact that these are available on the cloud and support the pay-as-you-use model.

Implementation models

• Private cloud: Since the infrastructure is owned by one customer, concerns on shared resources resulting in security violations and performance impact due to excessive load on shared resources are eliminated. From a testing perspective, performance, scalability and security tests provide better results (reliable test results) in a private cloud environment compared to other models. Access to the private cloud is generally restricted.
• Community cloud: This is more like a shared cloud infrastructure for community or closed group of individuals. Access is restricted to specific groups or individuals who are members of the community. From a testing perspective, the focus would be on testing authentication and authorization.
• Public cloud: Cloud infrastructure is publicly accessible but controlled through authentication and authorization. In this model, the focus of Testing would be on performance, scalability, elasticity, security, broad network access, and user experience among others.
• Hybrid cloud: This consists of two or more independently operating clouds (private, community or public) that are integrated. Depending on the combination, and considering the service model, there may be changes to the focus of testing. However, integration and interface testing, performance, scalability, security, user experience and broad network access tests are common in this environment.

Based on the choice of cloud platform services and the implementation model, organizations need to design their test strategy, taking into consideration the possible test environments as specified above.