ANN-inspired Straggler MapReduce Detection in Big Data Processing

Abstract

One of the most challenging aspects of using MapReduce is detect straggler nodes while processing large-scale data in parallel manner. Identifying ongoing tasks on weak nodes is how it’s described. The overall calculation time is the amount of time taken by the system which is categorized into two phases; Map Phase (duplicate, join) and the Reduce Phase (mix, sort, and lessen). The primary objevtive of this study is to estimate the accurate execution time in each location. The proposed approach uses an Artificial Neural Network (ANN) with Genetic Algorithm (GA) on Hadoop to detect straggler tasks and calculate the remaining task execution time, which is crucial in straggler task identification. The comparative analysis is done with some eÿcient models in this domain, like LATE and ESAMR. The actual execution time for Word-Count is evaluated and benchmarking is done. It was found that the proposed model is capable of detecting straggler tasks in accurately estimating execution time. It also helps in reducing the execution time that it takes to complete a task.