TESTING IN BIG DATA APPLICATION

As a greater number of organizations are adopting “Big Data” as their Data Analytics solution, they are finding it difficult to define a robust testing strategy and setting up an optimal test environment for Big Data. This is mostly due to the lack of knowledge and understanding on Big Data testing as the technology is still gaining popularity in the industry. Big Data involves processing of huge volume of structured/unstructured data across different nodes using languages such as “Map-reduce”, “Hive” and “Pig”. A robust testing strategy needs to be defined well in advance in order to ensure that the functional and non-functional requirements are met and that the data conforms to acceptable quality. In this blog we intend to define recommended test approaches in order to test “Hadoop” based applications.
Traditional testing approaches on Hadoop are based upon sample data record sets, which is fine for unit testing activities. However, the challenge comes in determining how to validate an entire data set consisting of millions, and even billions, of records.
The below diagram describes a typical Big Data implementation design.
In order to successfully test a Big Data Analytics application, the test strategy should include the following testing considerations at a minimum.

Data Staging Validation:

Data from various sourcing systems like RDBMS, social media, web logs etc. should be validated to ensure that proper data is pulled into the Hadoop system. Some of the high validations, which have to be performed, are:
  • Comparing source data with the data landed on Hadoop system to ensure they match
  • Verify the right data is extracted and loaded into the correct HDFS location
Some teams will always verify sample sets of data by using sampling algorithm because verification of full dataset are difficult to achieve. However this approach may not uncover all the data inconsistencies and may result in data quality issues within HDFS. Hence it is very important to include full dataset validation in your test strategy. Tools such as Datameer, Talend or Informatica can be used for validating the staged data. Import jobs should be created to pull the data into these cial media, web logs etc. should be validated to ensure that proper data is pulled into the Hadoop system. Some of the high validations, which have to be performed, are:
  • Comparing source data with the data landed on Hadoop system to ensure they match
  • Verify the right data is extracted and loaded into the correct HDFS location
Some teams will always verify sample sets of data by using sampling algorithm because verification of full dataset are difficult to achieve. However this approach may not uncover all the data inconsistencies and may result in data quality issues within HDFS. Hence it is very important to include full dataset validation in your test strategy. Tools such as Datameer, Talend or Informatica can be used for validating the staged data. Import jobs should be created to pull the data into these tools from source and staging systems. The data should be compared using the Data Analytics capability of these tools. The below diagram describes the overall approach for staging validations.
Fig 2: Data Staging Validation

Transformation or “Map-reduce” validation

This type of validation is similar to Data Warehousing Testing wherein a tester verifies that the business rules are applied on the data. However, in this case there is a slight difference in the test approach as Hadoop data should be tested for volume, variety and velocity.
Typically DWH testing involves testing of Gigabytes of data, whereas Hadoop testing in comparison involves testing of Petabytes of data. There is a definitive approach to test the DWH by using “sampling” techniques, although this cannot be achieved with a Hadoop application because even sampling testing will be challenging in a Hadoop Framework. There are numerous probabilities and combinations in large volumes of data which render sampling techniques ineffective as a validation approach.
DWH systems can only process structured data, however Hadoop systems may handle both structured and unstructured data with limited additional efforts. This Testing wherein a tester verifies that the business rules are applied on the data. However, in this case there is a slight difference in the test approach as Hadoop data should be tested for volume, variety and velocity.
Typically DWH testing involves testing of Gigabytes of data, whereas Hadoop testing in comparison involves testing of Petabytes of data. There is a definitive approach to test the DWH by using “sampling” techniques, although this cannot be achieved with a Hadoop application because even sampling testing will be challenging in a Hadoop Framework. There are numerous probabilities and combinations in large volumes of data which render sampling techniques ineffective as a validation approach.
DWH systems can only process structured data, however Hadoop systems may handle both structured and unstructured data with limited additional efforts. This potential is already leading to new ways of data exploration which in turn will result in an increase of scenarios for Hadoop Testing.
The key validations to be performed are:
  • Verification of the ETL: implemented on the data
  • Verification of data aggregations/segregation rules implemented on the data
  • Verification of output data. Validate that processed data remains the same even when executed on a distributed environment.
  • Verify the batch processes designed for data transformation.
Hive is the most reliable language to perform Data Warehouse Validation. Testers should write HQL queries that replicate the data requirements and compare it with the output produced by the MR jobs (Development team). If there are no discrepancies found in the report, then the test script is considered pass.

Data Warehouse Validation

Fig 3: Transformation Validation

Data Warehouse Validation

This testing is performed after the data processed in Hadoop environment is loaded to the Enterprise Data warehouse. The high level scenarios to be tested include:
  • Verify the processed data from HDFS file system is moved correctly to the EDW tables
  • Verify the EDW data requirements are met
  • Verify the data is aggregated as per specified requirement
Data warehouse validation is similar to data staging validation. DataMeer or Talend or Informatica tool can be used for validating the data loads from Hadoop to traditional Data warehouse.

Architecture Testing

As Hadoop involves processing large volumes of data, architecture testing is very critical for the application to be successful. Poorly designed systems may cause performance degradation and the system could fail to meet the SLAs contracted with the Business. At the minimum, Performance and Failover test services should be performed in a Hadoop environment.
Performance testing should be conducted by setting up large volumes of data with an environment similar to production. Performance metrics like job completion time, data throughput, memory utilization and similar system level metrics should be verified as part of the Performance testing services.
Failover test should be performed as Hadoop consists of name node and several data nodes hosted on different machines. The objective of Failover testing is to verify that data processing happens seamlessly in case of failure of data nodes.

Comments

Post a Comment

Popular posts from this blog

How to calculate maximum number of concurrent users using littile law formula

When load testing an application, the first set of tests should focus on measuring the maximum throughput. This is especially true of multi-user, interactive applications like web applications. The maximum throughput is best measured by running a few emulated users with zero think time. This means that each emulated user sends a request, receives a response and immediately loops back to send the next request. Although this is artificial, it is the best way to quickly determine the maximum performance of the server infrastructure. Littile Law: Once you have that throughput (say X), we can use  Little’s Law  to estimate the number of real simultaneous users that the application can support. In simple terms, Little’s Law states that : N = X  / λ where N is the number of concurrent users,  λ is the average arrival rate and X is the throughput. Note that the arrival rate is the inverse of the inter-arrival time i.e. the time between requests. To under...
Read more

How to develop a security test strategy

What do you need to write a security test plan? The folowing documents comprise the list of what I would expect as inputs to the creation of the individual security test plans. This is a good point to go and review your overall security delivery plan. Does it include these documents as deliverables? Does the supplier have any of these as standard off-the-shelf products? : System security requirements System threat model System risk model Secure coding standards Technical threat modelling Infrastructure secure configuration guides Security defect metrics What is in a security test plan? The security test strategy should as a minimum specify the following points as the core content for each of the security test plans: What are the inputs to the test plan? When in the process does this test occur? Who specifies the test plan contents? Who performs the test? How many instances of the test are expected? What are the required outputs of the test? What are the success...
Read more

Singleton Design Pattern using Page Factory

Image
n this tutorial, We will learn about the  Framework Design Pattern using singleton class  in Selenium. In the Previous tutorial we have seen, How can we run the Test using Page Factory. Now question is, Will we repeat the Browser invoke and Logger code again and again in each and every Test. No, To resolve this issue we will use the Singleton Pattern. We will design the Single  Test Base  class.  First, Let's see what is  Singleton Pattern .  With the Singleton design pattern you can: Ensure that only one instance of a class is created Provide a global point of access to the object In  TestBase  class, we will Design the Singleton Pattern, We will write the command code here and all Test Classes will inherit this  TestBase  class.  n previous Example, we have seen we were invoking  WebDriver   instance  in each @Test. Here, We will Implement the WebDriver initializat...
Read more