Data calculation and performance optimization of dairy traceability based on Hadoop/Hive

ZHU Shu-xin; LI Yue; YUAN Pei-sen; XU Huan-liang; WANG Kang; XIE Zhong-hong

doi:10.3969/j.issn.1000-5641.2018.04.010

Issue 4

Jul. 2018

Turn off MathJax

Article Contents

Article Navigation > Journal of East China Normal University (Natural Sciences) > 2018 > (4): 99-108

ZHU Shu-xin, LI Yue, YUAN Pei-sen, XU Huan-liang, WANG Kang, XIE Zhong-hong. Data calculation and performance optimization of dairy traceability based on Hadoop/Hive[J]. Journal of East China Normal University (Natural Sciences), 2018, (4): 99-108. doi: 10.3969/j.issn.1000-5641.2018.04.010

Citation:

ZHU Shu-xin, LI Yue, YUAN Pei-sen, XU Huan-liang, WANG Kang, XIE Zhong-hong. Data calculation and performance optimization of dairy traceability based on Hadoop/Hive[J]. Journal of East China Normal University (Natural Sciences), 2018, (4): 99-108. doi: 10.3969/j.issn.1000-5641.2018.04.010

Citation:

PDF( 1550 KB)

Data calculation and performance optimization of dairy traceability based on Hadoop/Hive

doi: 10.3969/j.issn.1000-5641.2018.04.010

1.
College of Information Science and Technology, Nanjing Agricultural University, Nanjing 210095, China
2.
Jiangsu Collaborative Innovation Center of Meat Production and Processing, Quality and Safety Control, Nanjing 210095, China

Received Date: 2017-06-19
Publish Date: 2018-07-25

Abstract

Abstract

In order to enhance the performance of traditional dairy traceability systems for the production data of large-scale enterprise, this paper analyzed the supply chain process of dairy enterprises, key traceability units and traceability information; combining Hadoop/Hive big data technology and distributed database technology, the paper designed and constructed a dairy products traceability framework based on Hadoop/Hive. We built a simulated large-scale data environment and used actual production data to test the system performance. The experimental results showed that after the introduction of the Hadoop/Hive technology system, the average data storage speed, the average data access speed, and the average data exchange rate increased by 87.43%, 27.10% and 58.16%, respectively. The improved traceability system for dairy products is superior to the traditional dairy traceability system in storing and processing large-scale data.
- Hadoop/Hive,
- dairy products traceability,
- data calculation,
- performance optimization

FullText(HTML)

References(17)

References

[1]	ABOUZIED A, BAJDA-PAWLIKOWSKI K, HUANG J, et al. HadoopDB in action: Building real world applications[C]//ACM SIGMOD International Conference on Management of Data. ACM, 2010: 1111-1114.
[2]	ABOUZEID A, BAJDA-PAWLIKOWSKI K, ABADI D, et al. HadoopDB:An architectural hybrid of MapReduce and DBMS technologies for analytical workloads[J]. Proceedings of the VLDB Endowment, 2009, 2(1):922-933. doi: 10.14778/1687627
[3]	ISMAIL A S, AL-FEEL H, MOKHTAR H M O. Introducing a new arabic endpoint for DBpedia internationalization project[C]//International Database Engineering & Applications Symposium. ACM, 2016: 284-289.
[4]	TORRES D, SKAF-MOLLI H, MOLLI P. et al. BlueFinder: Recommending wikipedia links using DBpedia properties[C]//Proceedings of the 5th Annual ACM Web Science Conference (WebSci'13). New York: ACM, 2013: 413-422. DOI: https://doi.org/10.1145/2464464.2464515.
[5]	叶育鑫, 欧阳丹彤.混合语义约简和选择估值优化SPARQL[J].电子学报, 2010, 38(5):1205-1210. https://www.wenkuxiazai.com/word/9256b54d2b160b4e767fcf0c-1.doc
[6]	王德文, 肖凯, 肖磊.基于Hive的电力设备状态信息数据仓库[J].电力系统保护与控制, 2013(9):125-130. doi: 10.7667/j.issn.1674-3415.2013.09.021
[7]	曲朝阳, 朱莉, 张士林.基于Hadoop的广域测量系统数据处理[J].电力系统自动化, 2013, 37(4):92-97. doi: 10.7500/AEPS201111169
[8]	刘越, 李锦涛, 虎嵩林.基于代价估计的Hive多维索引分割策略选择算法[J].计算机研究与发展, 2016, 53(4):798-810. doi: 10.7544/issn1000-1239.2016.20151163
[9]	董新华, 李瑞轩, 周湾湾, 等. Hadoop系统性能优化与功能增强综述[J].计算机研究与发展, 2013, 50(s2):1-15. http://www.wanfangdata.com.cn/details/detail.do?_type=perio&id=jsjyjyfz2013z2002
[10]	THUSOO A, SARMA J S, JAIN N, et al. Hive:A warehousing solution over a map-reduce framework[J]. Proceedings of the VLDB Endowment, 2009, 2(2):1626-1629. doi: 10.14778/1687553
[11]	OLSEN P, BORIT M. How to define traceability[J]. Trends in Food Science & Technology, 2013, 29(2):142-150. https://www.sciencedirect.com/science/article/pii/S0924224412002117
[12]	钱建平, 刘学馨, 杨信廷, 等.可追溯系统的追溯粒度评价指标体系构建[J].农业工程学报, 2014, 30(1):98-104. http://www.oalib.com/paper/4922974
[13]	SHVACHKO K, KUANG H, RADIA S, et al. The Hadoop distributed file system[C]//Proceedings of the IEEE 26th Symposium on Mass Storage Systems and Technologies. Washington: IEEE Computer Society, 2010: 1-10. DOI: 10.1109/MSST.2010.5496972.
[14]	张良均. Hadoop大数据分析与挖掘实战[M].北京:机械工业出版社, 2016.
[15]	荀亚玲, 张继福, 秦啸. MapReduce集群环境下的数据放置策略[J].软件学报, 2015(8):2056-2073. http://www.oalib.com/paper/5071219
[16]	叶晓江, 刘鹏.实战Hadoop2.0从云计算到大数据[M].北京:电子工业出版社, 2016.
[17]	张佳兰, 昝林森, 刘永峰, 等.我国DHI测定现状及存在的问题[J].中国牛业科学, 2007, 33(5):56-59. http://epub.cqvip.com/articledetail.aspx?id=1000000427570