面向新硬件的数据处理软件技术

涂云山; 储佳佳; 张耀; 翁楚良

doi:10.3969/j.issn.1000-5641.2018.05.003

面向新硬件的数据处理软件技术

doi: 10.3969/j.issn.1000-5641.2018.05.003

华东师范大学数据科学与工程学院, 上海 200062

基金项目:

国家自然科学基金 61732014

国家自然科学基金 61772204

详细信息

作者简介:
涂云山, 男, 硕士研究生, 研究方向为并行与分布式系统.E-mail:yunshantu@stu.ecnu.edu.cn

通讯作者:
翁楚良, 男, 教授, 博士生导师, 研究方向为并行与分布式系统、新型存储与内存计算、系统虚拟化与系统安全.E-mail:clweng@dase.ecnu.edu.cn

中图分类号: TP392
计量
- 文章访问数: 248
- HTML全文浏览量: 85
- PDF下载量: 1122
- 被引次数: 0
出版历程
- 收稿日期: 2018-07-04
- 刊出日期: 2018-09-25

Data processing software technology for new hardware

School of Data Science and Engineering, East China Normal University, Shanghai 200062, China

摘要

摘要: 近年来，计算机硬件技术飞速发展，取得了显著的进步，一些高性能、低时延的新型硬件技术不断涌现，如：异构的处理器、可编程的高速网卡/交换机、易失/非易失的存储器等，给传统的计算机体系结构和系统带来新的机遇和挑战.然而，在大数据处理中，直接将传统的软件技术应用到新型硬件上很难发挥出硬件技术突破所带来的全部潜在性能.因此，这就促使我们重新思考传统的软件技术，以便可以释放硬件进步带来的全部红利.本文从计算、传输、存储三个方面讨论了面向新型硬件的数据处理软件技术，梳理和分析了该领域中的相关工作，总结概述已取得的进展，分析存在的新问题和挑战，从而为未来探索数据处理性能"天花板"的研究提供有价值的参考.
- 新硬件 /
- 数据处理 /
- 软件栈
Abstract: The rapid development of computer hardware in recent years has brought profound technological progress. New hardware for high-performance and low-latency applications (e.g., heterogeneous processors, programmable high-speed NICs/switches, and volatile/non-volatile memory) have been emerging, which bring both new opportunities and challenges to traditional computer architectures and systems. In the case of big data processing, it is difficult to adapt traditional software technology directly onto new hardware, this makes realizing the full potential brought by breakthroughs in hardware technology challenging. Hence, a rethink of traditional software technology can help unlock the benefits brought by advancements in hardware technology. This paper reviews:data processing technology for new hardware from the perspective of computing, transmission, and storage; an analysis of related work in the field; a summary of progress made to date; and new problems and challenges that exist. The study also provides a valuable reference point for future research on exploring the performance ceiling of systems for data processing.
- new hardware /
- data processing /
- software stack

HTML全文

图 1 异构的处理器

Fig. 1 Heterogeneous processors

下载: 全尺寸图片幻灯片

表 1 网络软件栈开销

Tab. 1 Network software stack overhead

		接收器运行/μs	CPU空闲/μs
报文处理	进	1.26(37.6%)	1.24(20.0%)
报文处理	出	1.05(31.3%)	1.42(22.9%)
进程调度		0.17(5.0%)	2.40(38.8%)
拷贝	进	0.24(7.1%)	0.25(4.0%)
拷贝	出	0.44(13.2%)	0.55(8.9%)
内核交换	返回	0.10(2.9%)	0.20(3.3%)
内核交换	系统调用	0.10(2.9%)	0.13(2.1%)
总计		3.36	6.19

下载: 导出CSV

表 2 存储软件栈开销

Tab. 2 Storage software stack overhead

	SATA HDDs/μs	SAS SSDs/μs	NVMe SSDs/μs
文件系统	72(1.86%)	71(45.51%)	60(63.16%)
块设备层	11(0.28%)	10(6.41%)	11(11.58%)
驱动 & 设备	3 791(97.86%)	75(48.08%)	24(25.26%)
总计	3 874	156	95

下载: 导出CSV

表 3 不同存储器的性能对比

Tab. 3 Performance comparison of different memory/storage technologies

	PCM	STT-RAM	RRAM	SRAM	DRAM	Flash	HDD
读时延/ns	48	10	116	10	55	25 000	3 000 000
写时延/ns	150	10	145	10	55	200 000	3 000 000
能耗	中	中/低	低	非常低	中	高	非常高
使用寿命/次	10$^{8}$	10$^{12}$	10$^{6}$	10$^{15}$	$>10^{15}$	10$^{5}$	$>10^{15}$
非易失性	是	是	是	否	否	是	是

下载: 导出CSV

参考文献(79)

[1]	陆游游, 舒继武.闪存存储系统综述[J].计算机研究与发展, 2013, 50(1):49-59. http://d.old.wanfangdata.com.cn/Periodical/jsjyjyfz201301005
[2]	程学旗, 靳小龙, 王元卓, 等.大数据系统和分析技术综述[J].软件学报, 2014(9):1889-1908. http://d.old.wanfangdata.com.cn/Periodical/rjxb201409002
[3]	孟小峰, 慈祥.大数据管理:概念、技术与挑战[J].计算机研究与发展, 2013, 50(1):146-169. http://d.old.wanfangdata.com.cn/Periodical/jsjyjyfz201301014
[4]	李星, 吕方, 刘颖, 等.关于多核/众核系统可扩展性趋势的探讨[C]//2014全国高性能计算学术年会. 2014.
[5]	周旭.面向多核/众核体系结构的确定性并行关键技术研究[D].长沙: 国防科学技术大学, 2013. https://max.book118.com/html/2018/0505/164922643.shtm
[6]	HARRIS M, HARRIS M, PURCELL T, et al. GPGPU: General purpose computation on graphics hardware[C]//ACM SIGGRAPH 2004 Course Notes. ACM, 2004: 33.
[7]	ISTVÁN Z, SIDLER D, ALONSO G. Caribou:Intelligent distributed storage[J]. Proceedings of the Vldb Endowment, 2017, 10(11):1202-1213. doi: 10.14778/3137628
[8]	SIDLER D, ISTVÁN Z, OWAIDA M, et al. Accelerating pattern matching queries in hybrid CPU-FPGA architectures[C]//ACM International Conference. ACM, 2017: 403-415.
[9]	KARA K, GICEVA J, ALONSO G. FPGA-based data partitioning[C]//ACM International Conference on Management of Data. ACM, 2017: 433-445.
[10]	中国计算机学会. CCF 2016-2017中国计算机科学技术发展报告[M].北京:机械工业出版社, 2017.
[11]	张铁赢, 黄贵, 章颖强, 等. X-DB:软硬一体的新型数据库系统[J].计算机研究与发展, 2018, 55(2):319-326. http://d.old.wanfangdata.com.cn/NSTLQK/NSTL_QKJJ0235373512/
[12]	FIRESTONE D, PUTNAM A, MUNDKUR S, et al. Azure accelerated networking: SmartNICs in the public cloud[C]//15th USENIX Symposium on Networked Systems Design and Implementation. USENIX Association, 2018.
[13]	HONDA M, LETTIERI G, EGGERT L, et al. PASTE: A network programming interface for non-volatile main memory[C]//15th USENIX Symposium on Networked Systems Design and Implementation. USENIX Association, 2018.
[14]	JIN X, LI X, ZHANG H, et al. NetCache: Balancing key-value stores with fast in-network caching[C]//Proceedings of the 26th Symposium on Operating Systems Principles. ACM, 2017: 121-136.
[15]	LI B, RUAN Z, XIAO W, et al. KV-Direct: High-performance in-memory key-value store with programmable NIC[C]//Proceedings of the 26th Symposium on Operating Systems Principles. ACM, 2017: 137-152.
[16]	AVNI H, 王鹏.面向数据库的持久化事务内存[J].计算机研究与发展, 2018, 55(2):305-318. http://d.old.wanfangdata.com.cn/Periodical/jsjyjyfz201802007
[17]	王健.存储新篇章, 详解英特尔傲腾内存[J].电脑爱好者, 2017(11):88-92. http://d.wanfangdata.com.cn/Periodical/jsjywl201709025
[18]	PETER S, LI J, ZHANG I, et al. Arrakis:The operating system is the control plane[J]. ACM Transactions on Computer Systems, 2015, 33(4):1-30. doi: 10.1145/2841315
[19]	Fio.[EB/OL].[2018-07-02]. http://git.kernel.dk/?p=fio.git;a=summary.
[20]	Block I/O Layer Tracing (blktrace).[EB/OL].[2018-07-02]. https://git.kernel.org/pub/scm/linux/kernel/git/axboe/blktrace.git.
[21]	Netronome Agilio SmartNICs.[EB/OL].[2018-07-02]. https://www.netronome.com/products/smartnic/overview.
[22]	Barefoot Tofino.[EB/OL].[2018-07-02]. https://www.barefootnetworks.com/products/brief-tofino/.
[23]	Cavium Xpliant Family.[EB/OL].[2018-07-02]. https://www.cavium.com/xpliant-ethernet-switch-productfamily.html.
[24]	XIE Y. Modeling, architecture, and applications for emerging memory technologies[J]. IEEE Design & Test of Computers, 2011, 28(1):44-51. http://ieeexplore.ieee.org/document/5708260/
[25]	XIE Y. Future memory and interconnect technologies[C]//Design, Automation & Test in Europe Conference & Exhibition (DATE), 2013. IEEE, 2013: 964-969.
[26]	BEZ R, PIROVANO A. Non-volatile memory technologies:Emerging concepts and new materials[J]. Materials Science in Semiconductor Processing, 2004, 7(4/6):349-355. http://d.old.wanfangdata.com.cn/NSTLQK/NSTL_QKJJ0219057528/
[27]	MEENA J S, SZE S M, CHAND U, et al. Overview of emerging nonvolatile memory technologies[J]. Nanoscale Research Letters, 2014, 9(1):526. doi: 10.1186/1556-276X-9-526
[28]	YU S, CHEN P Y. Emerging memory technologies:Recent trends and prospects[J]. IEEE Solid-State Circuits Magazine, 2016, 8(2):43-56. doi: 10.1109/MSSC.2016.2546199
[29]	Intel Optane Technology.[EB/OL].[2018-07-02]. https://www.intel.cn/content/www/cn/zh/architecture-andtechnology/intel-optane-technology.html.
[30]	KASHYAP S, MIN C, KIM K, et al. A scalable ordering primitive for multicore machines[C]//The Thirteenth EuroSys Conference. 2018: 1-15.
[31]	BHAT S S, EQBAL R, CLEMENTS A T, et al. Scaling a file system to many cores using an operation log[C]//Symposium on Operating Systems Principles. ACM, 2017: 69-86.
[32]	DREBES A, POP A, HEYDEMANN K, et al. NUMA-aware scheduling and memory allocation for data-flow task-parallel applications[J]. ACM Sigplan Notices, 2016, 51(8):1-2. http://dl.acm.org/authorize?N08329
[33]	LEIS V, BONCZ P, KEMPER A, et al. Morsel-driven parallelism: A NUMA-aware query evaluation framework for the many-core age[C]//Proceedings of SIGMOD'14. ACM, 2014: 743-754.
[34]	NVIDIA NVLink.[EB/OL].[2018-07-02]. https://www.nvidia.com/en-us/data-center/nvlink/.
[35]	HE B, LU M, YANG K, et al. Relational query coprocessing on graphics processors[J]. ACM Transactions on Database Systems, 2009, 34(4):1-39. http://dl.acm.org/citation.cfm?id=1620588
[36]	BRESS S, SAAKE G. Why it is time for a HyPE:A hybrid query processing engine for efficient GPU coprocessing in DBMS[J]. Proceedings of the Vldb Endowment, 2013, 6(12):1398-1403. doi: 10.14778/2536274
[37]	MapD.[EB/OL].[2018-07-02]. https://www.mapd.com/.
[38]	SALMI M F. Processing Big Data in Main Memory and on GPU[D]. Columbus, USA: The Ohio State University, 2016.
[39]	MILETIĆ V, KOVAČIĆ B, LENKOVIĆ K. PG-Strom: Application of parallel programming technology NVIDIA CUDA on PostgreSQL database management system[C]//Razvoj Poslovnih I Informatičkih Sustava Case. 2013.
[40]	HE B, YU J X. High-throughput transaction executions on graphics processors[J]. Proceedings of the Vldb Endowment, 2011, 4(5):314-325. doi: 10.14778/1952376
[41]	SIDLER D, ISTVÁN Z, ALONSO G. Low-latency TCP/IP stack for data center applications[C]//International Conference on Field Programmable Logic and Applications. IEEE, 2016: 1-4.
[42]	FRANCISCO P. IBM Puredata System for Analytics Architecture[R]. IBM Redbooks, 2014.
[43]	Netezza.[EB/OL].[2018-06-25]. https://www.ibm.com/analytics/netezza/.
[44]	BELAY A, PREKAS G, KLIMOVIC A, et al. Ⅸ: A protected dataplane operating system for high throughput and low latency[C]//Usenix Conference on Operating Systems Design and Implementation. USENIX Association, 2014: 49-65.
[45]	Data Plane Development Kit.[EB/OL].[2018-07-02]. https://dpdk.org/.
[46]	JEONG E Y, WOO S, JAMSHED M, et al. mTCP: A highly scalable user-level TCP stack for multicore systems[C]//Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation. USENIX Association, 2014.
[47]	OUSTERHOUT J, GOPALAN A, GUPTA A, et al. The RAMCloud Storage System[J]. ACM Transactions on Computer Systems, 2015, 33(3):7. http://d.old.wanfangdata.com.cn/OAPaper/oai_arXiv.org_1207.0140
[48]	RUMBLE S M, KEJRIWAL A, OUSTERHOUT J. Log-structured memory for DRAM-based storage[C]//Usenix Conference on File and Storage Technologies. USENIX Association, 2014: 1-16.
[49]	LEE C, PARK S J, KEJRIWAL A, et al. Implementing linearizability at large scale and low latency[C]//Proceedings of SOSP'15. ACM, 2015: 71-86.
[50]	KEJRIWAL A, GOPALAN A, GUPTA A, et al. SLIK: Scalable low-latency indexes for a key-value store[C]//Usenix Conference on Usenix Technical Conference. USENIX Association, 2016: 57-70.
[51]	ONGARO D, RUMBLE S M, STUTSMAN R, et al. Fast crash recovery in RAMCloud[C]//ACM Symposium on Operating Systems Principles. ACM, 2011: 29-41.
[52]	NARAYANAN D, HODSON O, CASTRO M. FaRM: Fast remote memory[C]//Usenix Conference on Networked Systems Design and Implementation. USENIX Association, 2014: 401-414.
[53]	LIM H, HAN D, ANDERSEN D G, et al. MICA: A holistic approach to fast in-memory key-value storage[C]//Usenix Conference on Networked Systems Design and Implementation. USENIX Association, 2014: 429-444.
[54]	PolarDB.[EB/OL].[2018-07-02]. https://help.aliyun.com/product/58609.html.
[55]	SHI J, YAO Y, CHEN R, et al. Fast and concurrent RDF queries with RDMA-based distributed graph exploration[C]//Usenix Conference on Operating Systems Design and Implementation. USENIX Association, 2016: 317-332.
[56]	WEI X, SHI J, CHEN Y, et al. Fast in-memory transaction processing using RDMA and HTM[C]//Symposium on Operating Systems Principles. ACM, 2015: 87-104.
[57]	COBURN J, CAULFIELD A M, AKEL A, et al. NV-Heaps:Making persistent objects fast and safe with next-generation, non-volatile memories[J]. ACM Sigplan Notices, 2011, 46(3):105-118. doi: 10.1145/1961296
[58]	VOLOS H, TACK A J, Swift M M. Mnemosyne:Lightweight persistent memory[J]. ACM SIGARCH Computer Architecture News, 2011, 39(1):91-104. http://d.old.wanfangdata.com.cn/NSTLQK/NSTL_QKJJ0223080469/
[59]	HWANG T, JUNG J, WON Y. Heapo:Heap-based persistent object store[J]. ACM Transactions on Storage (TOS), 2015, 11(1):3. http://d.old.wanfangdata.com.cn/NSTLQK/NSTL_QKJJ0234405612/
[60]	Persistent Memory Development Kit.[EB/OL].[2018-07-02]. http://pmem.io/pmdk/.
[61]	ANDREI M, LEMKE C, RADESTOCK G, et al. SAP HANA adoption of non-volatile memory[J]. Proceedings of the VLDB Endowment, 2017, 10(12):1754-1765. doi: 10.14778/3137765
[62]	OGLEARI M A, MILLER E L, ZHAO J. Steal but no force: Efficient hardware Undo+Redo Logging for persistent memory systems[C]//2018 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2018: 336-349.
[63]	ARULRAJ J, PAVLO A, DULLOOR S R. Let's talk about storage & recovery methods for non-volatile memory database systems[C]//Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. ACM, 2015: 707-722.
[64]	AGRAWAL R, JAGADISH H V. Recovery algorithms for database machines with non-volatile main memory[C]//International Workshop on Database Machines. Berlin: Springer, 1989: 269-285.
[65]	GAO S, XU J, HE B, et al. PCMLogging: Reducing transaction logging overhead with PCM[C]//Proceedings of the 20th ACM International Conference on Information and Knowledge Management. ACM, 2011: 2401-2404.
[66]	OUKID I, BOOSS D, LEHNER W, et al. SOFORT: A hybrid SCM-DRAM storage engine for fast data recovery[C]//Proceedings of the Tenth International Workshop on Data Management on New Hardware. ACM, 2014: 8.
[67]	ARULRAJ J, PERRON M, PAVLO A. Write-behind logging[J]. Proceedings of the VLDB Endowment, 2016, 10(4):337-348. doi: 10.14778/3025111
[68]	ZHANG Y Y, YANG J, MEMARIPOUR A, et al. Mojim:A reliable and highly-available non-volatile memory system[J]. ACM SIGARCH Computer Architecture News, 2015, 43(1):3-18. http://d.old.wanfangdata.com.cn/NSTLQK/NSTL_QKJJ0234962514/
[69]	Storage Performance Development Kit.[EB/OL].[2018-07-02]. http://www.spdk.io.
[70]	KIM H J, LEE Y S, KIM J S. NVMeDirect: A User-space I/O framework for application-specific optimization on NVMe SSDs[C]//Proceedings of HotStorage. 2016.
[71]	BJØRLING M, GONZÁALEZ J, BONNET P. LightNVM: The linux open-channel SSD subsystem[C]//Proceedings of FAST'17. USENIX Association. 2017: 359-374.
[72]	CAULFIELD A M, MOLLOV T I, EISNER L A, et al. Providing safe, user space access to fast, solid state disks[J]. ACM SIGARCH Computer Architecture News, 2012, 40(1):387-400. http://dl.acm.org/citation.cfm?id=2151017
[73]	YANG J, MINTURN D B, HADY F. When poll is better than interrupt[C]//Proceedings of FAST'12. USENIX Association. 2012.
[74]	LI C, DING C, SHEN K. Quantifying the cost of context switch[C]//Proceedings of the 2007 Workshop on Experimental Computer Science. ACM, 2007: 2.
[75]	SHIN W, CHEN Q, OH M, et al. OS I/O path optimizations for flash solid-state drives[C]//USENIX Annual Technical Conference. 2014: 483-488.
[76]	YU Y J, SHIN D I, SHIN W, et al. Optimizing the block I/O subsystem for fast storage devices[J]. ACM Transactions on Computer Systems (TOCS), 2014, 32(2):6. http://dl.acm.org/citation.cfm?id=2619092
[77]	XU J, SWANSON S. NOVA: A log-structured file system for hybrid volatile/non-volatile main memories[C]//Proceedings of FAST'16. USENIX Association. 2016: 323-338.
[78]	VOLOS H, NALLI S, PANNEERSELVAM S, et al. Aerie: Flexible file-system interfaces to storage-class memory[C]//Proceedings of the Ninth European Conference on Computer Systems. ACM, 2014: 14.
[79]	KANNAN S, ARPACI-DUSSEAU A C, ARPACI-DUSSEAU R H, et al. Designing a true direct-access file system with DevFS[C]//Proceedings of the 16th USENIX Conference on File and Storage Technologies. 2018: 241-256.