Equi-join optimization on spark
-
摘要: 等值连接作为数据分析中最常用、代价最高的操作之一,在Spark上的实现和优化与传统并行数据库有很大的差别,传统并行数据仓库中基于数据预划分的连接算法在Spark上难以实现,而目前被广泛采用的Broadcast Join和Repartition Join性能较差,如何提高连接性能成为基于Spark的海量数据分析的关键.本研究将SimiJoin与Partition Join的优势相结合,并基于Spark上的特性提出了一种优化的等值连接算法.代价分析和实验表明本算法比现有基于Spark的数据分析系统中的连接算法性能提升1~2倍.Abstract: Equi-join is one of the most common and costly operations in data analysis. The implementations of equijoin on Spark are different from those in parallel databases. Equi-join algorithms based on data prepartitioning which are commonly used in parallel databases can hardly be implemented on Spark. Currently common used equijoin algorithms on Spark, such as broadcast join and repartition join, are not efficient. How to improve equijoin performance on Spark becomes the key of big data analysis on Spark. This work combines the advantages of SimiJoin and Partition Join and provides an optimized equijoin algorithm based on the features of Spark. It is indicated by cost analysis and experiment that this algorithm is 12 times faster than algorithms used in stateoftheart data analysis systems on Spark.
-
Key words:
- Spark /
- SQL /
- big data analysis /
- equi-join /
- in-memory computation
点击查看大图
计量
- 文章访问数: 2617
- HTML全文浏览量: 36
- PDF下载量: 1308
- 被引次数: 0