Passer à la navigation principale Passer à la recherche Passer au contenu principal

A platform for scalable one-pass analytics using MapReduce

  • Boduo Li
  • , Edward Mazur
  • , Yanlei Diao
  • , Andrew McGregor
  • , Prashant Shenoy
  • University of Massachusetts

Résultats de recherche: Le chapitre dans un livre, un rapport, une anthologie ou une collection › Contribution à une conférence › Revue par des pairs

129 Citations (Scopus)

Résumé

Today's one-pass analytics applications tend to be data-intensive in nature and require the ability to process high volumes of data efficiently. MapReduce is a popular programming model for processing large datasets using a cluster of machines. However, the traditional MapReduce model is not well-suited for one-pass analytics, since it is geared towards batch processing and requires the data set to be fully loaded into the cluster before running analytical queries. This paper examines, from a systems standpoint, what architectural design changes are necessary to bring the benefits of the MapReduce model to incremental one-pass analytics. Our empirical and theoretical analyses of Hadoop-based MapReduce systems show that the widely-used sort-merge implementation for partitioning and parallel processing poses a fundamental barrier to incremental one-pass analytics, despite various optimizations. To address these limitations, we propose a new data analysis platform that employs hash techniques to enable fast in-memory processing, and a new frequent key based technique to extend such processing to workloads that require a large key-state space. Evaluation of our Hadoop-based prototype using real-world workloads shows that our new platform significantly improves the progress of map tasks, allows the reduce progress to keep up with the map progress, with up to 3 orders of magnitude reduction of internal data spills, and enables results to be returned continuously during the job.

langue originaleAnglais
titreProceedings of SIGMOD 2011 and PODS 2011
EditeurAssociation for Computing Machinery
Pages985-996
Nombre de pages12
ISBN (imprimé)9781450306614
Les DOIs
étatPublié - 1 janv. 2011
Modification externeOui
Evénement2011 ACM SIGMOD and 30th PODS 2011 Conference - Athens, Grcce
Durée: 12 juin 2011 → 16 juin 2011

Série de publications

NomProceedings of the ACM SIGMOD International Conference on Management of Data
ISSN (imprimé)0730-8078

Une conférence

Une conférence2011 ACM SIGMOD and 30th PODS 2011 Conference
Pays/TerritoireGrcce
La villeAthens
période12/06/11 → 16/06/11

Empreinte digitale

Examiner les sujets de recherche de « A platform for scalable one-pass analytics using MapReduce ». Ensemble, ils forment une empreinte digitale unique.

Contient cette citation