BIG DATA TESTING
-
Upload
qainfotech123 -
Category
Documents
-
view
39 -
download
0
description
Transcript of BIG DATA TESTING
![Page 1: BIG DATA TESTING](https://reader035.fdocuments.us/reader035/viewer/2022081603/568129fe550346895d8cdd9d/html5/thumbnails/1.jpg)
BIG DATA TESTING
By QA InfoTech
![Page 2: BIG DATA TESTING](https://reader035.fdocuments.us/reader035/viewer/2022081603/568129fe550346895d8cdd9d/html5/thumbnails/2.jpg)
Scenario
![Page 3: BIG DATA TESTING](https://reader035.fdocuments.us/reader035/viewer/2022081603/568129fe550346895d8cdd9d/html5/thumbnails/3.jpg)
OMG!! Did he just asked me to catch rats in a place full of snakes
3
![Page 4: BIG DATA TESTING](https://reader035.fdocuments.us/reader035/viewer/2022081603/568129fe550346895d8cdd9d/html5/thumbnails/4.jpg)
Agenda
1. What is Big Data2. Characteristic of Big Data3. Meaning of BIG DATA to “US”4. Hadoop6. Submitting a Map Reduce Job
![Page 5: BIG DATA TESTING](https://reader035.fdocuments.us/reader035/viewer/2022081603/568129fe550346895d8cdd9d/html5/thumbnails/5.jpg)
What is BIG DATA?
• ‘Big Data’ is similar to ‘small data’, but bigger in size
• Big Data generates value from the storage and processing of very large quantities of digital information that cannot be analyzed with traditional computing techniques.
• Walmart handles more than 1 million customer transactions every hour.
• Facebook handles 40 billion photos from its user base.
• Decoding the human genome originally took 10years to process; now it can be achieved in one week.
![Page 6: BIG DATA TESTING](https://reader035.fdocuments.us/reader035/viewer/2022081603/568129fe550346895d8cdd9d/html5/thumbnails/6.jpg)
Three Characteristics of Big Data V3s
Volume
•Data
quantity
Velocity
•Data Speed
Variety
•Data Types
![Page 7: BIG DATA TESTING](https://reader035.fdocuments.us/reader035/viewer/2022081603/568129fe550346895d8cdd9d/html5/thumbnails/7.jpg)
What BIG DATA TESTING mean to Testers?
Take into consideration these 3 perspectives:• Data • Infrastructure• Validation Tools
![Page 8: BIG DATA TESTING](https://reader035.fdocuments.us/reader035/viewer/2022081603/568129fe550346895d8cdd9d/html5/thumbnails/8.jpg)
Now the questions comes what technology is needed for handling BIG DATA ?
1.HADOOP
![Page 9: BIG DATA TESTING](https://reader035.fdocuments.us/reader035/viewer/2022081603/568129fe550346895d8cdd9d/html5/thumbnails/9.jpg)
Hadoop & Its Components
• Hadoop is an open-source software framework for storing and processing big data in a distributed fashion on large clusters of commodity hardware. Essentially, it accomplishes two tasks: massive data storage and faster processing.
Source: http://www.trieuvan.com/apache/hadoop/common/
![Page 10: BIG DATA TESTING](https://reader035.fdocuments.us/reader035/viewer/2022081603/568129fe550346895d8cdd9d/html5/thumbnails/10.jpg)
How is Hadoop Helping?
• HDFS: Java based distributed FS that can run and store all kinds of data• Map Reduce: A software programming model for processing large set of
data in parallel• YARN: A resource management framework for scheduling and handling
resource requests from distributed applications
![Page 11: BIG DATA TESTING](https://reader035.fdocuments.us/reader035/viewer/2022081603/568129fe550346895d8cdd9d/html5/thumbnails/11.jpg)
11
This is our Input File : Input Sampleset.txt
![Page 12: BIG DATA TESTING](https://reader035.fdocuments.us/reader035/viewer/2022081603/568129fe550346895d8cdd9d/html5/thumbnails/12.jpg)
12
Map Reduce Program For Max Temperature :Driver Class
Job job = new Job();job.setJarByClass(MaxTemperatureDriver.class);job.setJobName("Max Temperature");
FileInputFormat.addInputPath(job, new Path(args[0]));FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setMapperClass(MaxTemperatureMapper.class);job.setReducerClass(MaxTemperatureReducer.class);
![Page 13: BIG DATA TESTING](https://reader035.fdocuments.us/reader035/viewer/2022081603/568129fe550346895d8cdd9d/html5/thumbnails/13.jpg)
13
Mapper Class
@Overridepublic void map(LongWritable key, Text value, Context context)throws IOException, InterruptedException {String line = value.toString();String year = line.substring(15, 19);int airTemperature;if (line.charAt(87) == '+') { // parseInt doesn't like leading plus// signsairTemperature = Integer.parseInt(line.substring(88, 92));} else {airTemperature = Integer.parseInt(line.substring(87, 92));}
![Page 14: BIG DATA TESTING](https://reader035.fdocuments.us/reader035/viewer/2022081603/568129fe550346895d8cdd9d/html5/thumbnails/14.jpg)
14
Reducer Class
@Overridepublic void reduce(Text key, Iterable<IntWritable> values,Context context)throws IOException, InterruptedException { int maxValue = Integer.MIN_VALUE;for (IntWritable value : values) {maxValue = Math.max(maxValue, value.get());}context.write(key, new IntWritable(maxValue));}}
![Page 15: BIG DATA TESTING](https://reader035.fdocuments.us/reader035/viewer/2022081603/568129fe550346895d8cdd9d/html5/thumbnails/15.jpg)
Thank You
For more information, please:
• Contact us at [email protected]
• Visit us at www.qainfotech.com
• Read our blog at www.qainfotech.com/blog
• Follow us on Twitter at www.twitter.com/qainfotech
USA
Office
International
Headquarters
Noida
Uttar Pradesh, India
Farmington Hills
Michigan, U.S.A.