In this blog post
By Manoranjitham Vetriveeran
Big Data Testing is a trending topic in the Software Industry, its various properties like volume, velocity, variety, variability, value, complexity and performance puts forth many challenges. On click of a button we generate megabytes of data.
Testing such large collections of various data types ranging from tables to texts and images is a challenge.
With this article, we would study about what is Big Data, its characteristics, its importance, processes involved, and different aspects of testing, the challenges, how to secure the data and the best practices.
Big Data – What is it?
Big Data is a collection of data sets which are large and complex and are difficult to process and does not fit well into tables and that responds poorly to manipulation by SQL.
Suppose we have a 100 MB doc which is difficult to send, or a 100 MB image which is difficult to view, or a 100 TB video which is difficult to edit. In any of these instances, we have a Big Data problem. Thus, Big data can be system specific.
Big Data not only deals with the size of the data and is related to the 4 V’s.
- Volume (scale of data)
- Velocity (different forms of data)
- Variety (analysis of Streaming data in microseconds)
- Veracity (Certainty of data)
Big Data comes in different sizes and formats. Hence, the three different categories:
- Structured
- Unstructured
- Semi-structured data
What are the aspects of Big Data Testing?
A very strong test data and QA environment are required to ensure error-free processing of data. Some of the key aspects of Big Data testing are as follows;
Big Data testing can be performed in two ways.
Functional Testing
Functional testing is performed to identify data issues because of coding errors or node configuration errors, while non-functional testing focuses on performance bottlenecks and validates the non-functional requirements. Testing should include the below four phases;
Validation of pre-Hadoop processing
Data is extracted from various sources such as web logs, social media, RDBMS, etc., and uploaded into HDFS (Hadoop Distributed File System – This article considers Hadoop ecosystem). This can be appropriately used on other Big Data ecosystems as well). Here we need to ensure it is extracted properly and uploaded into correct HDFS location. Also, validation of the file partition and replication into different data nodes.
Validation of MapReduce Process
Testing of business logic on a single node than on a set of nodes or multiple nodes to ensure valid generation of the “key-value” pair. Validation of aggregation and consolidation of data after reduced operation. Now compare the output generated data with the input files to ensure the generated output file meets all the requirements.
Validation of Extract-Transform-Load Process
Last stage of testing where data generated by the previous stage is first unloaded and then loaded into the repository system. Inspection of data aggregation to ensure there is no data corruption and it is loaded into the target system.
Reports Validation
Validation of reports deals with required data and all indicators are displayed correctly.
- Non-Functional Testing
Performance Testing
This is performed to obtain the metrics of response time, data processing capacity, and speed of data consumption. It is conducted to access the Performance limiting conditions which causes performance problems.
Verification of data storage at different nodes and testing the JVM parameters must be involved. Test the values for connection timeout and query timeout.
Failover testing
Verification of seamless processing of data in case of data nodes failure and validation of the recovery process on switching to other data nodes.
Big Data Testing – Challenges
Understanding the data and its impact on the business is the real challenge. Also, dealing with unstructured data drawn from sources such as tweets, text documents and social media posts is one of the biggest challenges.
Testers need to understand business rules and the statistical correlation between different subsets of data. Major attention in big data testing include:
- Data security and Scalability of the data storage media.
- Performance issues and the workload on the system due to huge data volumes.
Big Data Testing – Best Practices / quick to-do list
To overcome this challenge, Quality assurance and testing professionals must move ahead to understand and analyze challenges in real time. Testers must be capable of handling data structure layouts, processes, and data loads.
- Tester must avoid sampling approach. It may look easy and scientific, but is risky. It’s better to plan load coverage at the outset, and consider deployment of automation tools to access data across various layers.
- Testers need to derive patterns and learning mechanisms from drill-down charts and aggregate data.
- If the tester has good experience on programming languages, it would definitely help on map-reduce process validation.
- Ensure the right-time incorporation of changes in requirements. This calls for continuous collaboration and discussions with stakeholders.
How to Secure the Big Data
Big data applications work on data from different sources and data travels extraordinarily fast across the globe, so security testing is another important aspect.
Once we’ve got the data, the difficulty exists not only in analyzing the massive data lake to get key insights but the security of this large volume of data must be ensured while developing the application.
From a security point of view, there are more number of risks such as unauthorized access, privilege escalation, lack of visibility and many more.
- Unauthorized access can put sensitive and highly confidential data at risk of theft and loss. We must have a centralized control over big data.
- Over-privileged accounts raise insider threats. Admin shouldn’t have complete access to Hadoop clusters. Instead of giving full access, it should be restricted to the specific commands and actions required.
- Organizations must establish trust by using Kerberos Authentication while ensuring conformity to predefined security policies.
- To overcome the NoSQL injection, must start by encrypting or hashing passwords, and ensure end-to-end encryption by encrypting data at rest using algorithms.
- To detect unauthorized file modifications by malicious server agents, a technique called secure untrusted data repository is to be used.
- In addition to an antivirus software, organizations should start using trusted certificates and connect only to trusted devices on their network by using a mobile device management solution
- The incorporation of Hadoop nodes, clusters, applications, services, and users into Active Directory permits IT to give users granular privileges based on their job function.
However, there is a solution to every possible problem that emerges. All we need to do is identify the effective and suitable solution.
Conclusion
Big data is making a rapid move and is going to transform how we live, how we work, and how we think. To be successful, testers must learn the components of Big data ecosystem from scratch. Applying right test strategies and following best practices would help ensure qualitative software testing. The idea is to improve the big data testing quality which will help to identify defects in early stages and reduce overall cost.
