Woodpecker: Fine-grained contention simulation database testing framework
-
摘要: 随着应用的丰富以及应用对性能的要求提升，各种形态的数据库开发业务应运而生，如行存、列存以及内存型等.因而数据库测试也变得越发重要.现有的数据库自动化测试框架在测试的灵活性、功能的全面性以及测试工作的细致性上都存在问题.首先，测试功能支持不够全面，无法同时支持功能测试、性能测试和数据库系统功能测试；其次，广泛使用的TPC-C基准测试在粗粒度仓库级别上模拟负载冲突，无法量化表示冲突，因此冲突模拟与实际预期误差较大；最后，测试案例的使用一般是静态的，无法动态地进行测试案例的组织和运行.对此，设计并实现了一个灵活的数据库自动化测试框架Woodpecker：提供一套语义丰富、易用、高效的测试任务描述语言，第一次实现上述3种测试任务混合运行的测试框架，并且首次支持量化的负载冲突模拟，包括读写冲突和写写冲突.实验部分以主键自增测试为例，展示了Woodpecker测试描述语言的优势，通过实验说明了Woodpecker对混合测试任务的支持，实验还展示了Woodpecker负载冲突细粒度模拟的正确性和可用性.Abstract: With the rich and the growing peformance requirement of applications, various database developments appear, such as row-based, column-based, main memory database and so on. Hence database testing becomes more and more important. Existing automated testing frameworks of database have some problems in flexibility of test, integrality of function and the granularity of test. Firstly they can't support functional test, performance test and DBMS functional test at the same time. Secondly popular benchmark TPC-C simulates workload contention by warehouse which can't quantify contention and maybe execution is different with expectation. Finally the use of test case is static, they can't organize and run test case dynamically. As a consequence this paper presents a flexible database testing framework which is called Woodpecker, its test task descriptive language has abundant primitives, easy-to-use and could write test case efficiently. Woodpecker is the first test framework that can mix three above-mentioned tests in one test case, and simulates workload contention in fine granularity which include read-write contention and write-write contention. We use a simple case in auto increment testing to show the descriptive language's advantage, last we verify the capacity of mixing three tests in Woodpecker, and besides, we show the results of workload contention simulation in performance test.
-
Key words:
- database /
- testing framework /
- contention simulation
-
图 9 不同冲突强度和冲突比例下的负载测试结果
Fig. 9 Workload result in different contention intensity and ratio
表 1 数据类型
Tab. 1 Data type
数据类型 说明 int 整型 long 长整型 boolean 布尔型 char 字符 float 单精度浮点 double 双精度浮点 String 字符串 Decimal 小数 List$<$var_category$>$ 链表 Connection JDBC链接 Statement SQL执行器 PStatement 预编译SQL执行器 CStatement 存储过程SQL执行器 ResultSet 执行结果集 IdealResultSet$<$var_category$>$ 理想结果集 PerformanceResult 性能测试结果 
下载: 导出CSV
表 2 关键字
Tab. 2 Keywords
关键字 说明 GET_CONN[ms_category] 获取JDBC连接 GET_STAT[connection_name] 获取执行器 GET_PSTAT[connection_name; "sql"] 获取预编译执行器 GET_CSTAT[connection_name; "sql"] 获取存储过程执行器 SQL["sql"; stat_name; type] SQL语句 PSQL[pstat_name; type; variables] 预编译语句 CSQL[cstat_name; type; variable param_type] 存储过程语句 TX[connection_name; tx_id ] 事务 VERIFY[variable1; variable2; relation_operator] 验证结果集 SLEEP[number; time_unit] 暂停时间 关键字 说明 ST[number1; number2] 性能测试 IMPORT_DBI[number] 导入测试场景 CLEAR_DBI[number] 清空测试场景 IMPORT_IRS[source; variables] 结果集导入 SYS [sys_statement; ip_address] 系统功能 DEF_PROC[str_variable; stat_name] 定义存储过程 IF(condition){} ELSE{} 分支 WHILE(number ){} 循环 ## 注释 ERROR 此案例错误
下载: 导出CSV
表 3 系统功能命令
Tab. 3 System function keywords
sys_statement 说明 is_cluster_available 当前集群是否可服务 wait_available n $n$ s内等待所有集群可服务 merge 对主集群发起合并后返回 is_merge_done 集群合并是否完成 wait_merge_done n $n$ s内等所有集群合并完成 shell_command shell命令 reelect 对主集群选举换主 exist_master 集群是否有主 kill_server -option 下线server start_server variable 启动server add_server-option ip 添加server set_master variable 所有集群中设定主集群
下载: 导出CSV
-
[1] GEORGE L. HBase-The Definitive Guide: Random Access to Your Planet-Size Data[M]. [S. l. ]: O'Reilly, 2011. [2] STONEBRAKER M, WEISBERG A. The VoltDB Main Memory DBMS[J]. IEEE Data Eng Bull, 2013, 36(2):21-27. http://www.researchgate.net/publication/297362705_The_voltdb_main_memory_dbms [3] FARBER F, CHA S K, PRIMSCH J, et al. SAP HANA database:Data manä gement for modern business applications[J]. ACM Sigmod Record, 2012, 40(4):45-51. doi: 10.1145/2094114 [4] MARICK B. When should a test be automated?[C/OL]//Proceedings of the International Software/Internet Quality Week. [2017-04-21]. http://www.uml.org.cn/Test/12/automate.pdf. [5] ECNU. CEDAR: A distributed, shared-nothing relational database[EB/OL]. (2016-09-26)[2017-04-21]. https://github.com/daseECNU/Cedar. [6] ORACLE. The MySQL Test Framework[EB/OL]. (2016-02-11)[2017-07-12]. https://dev.mysql.com/doc/mysqltest/2.0/en/. [7] LEUTENEGGER S T, DIAS D. A modeling study of the TPC-C benchmark[J]. ACM Sigmod Record, 1993, 22(2):22-31. doi: 10.1145/170036 [8] YAN C, CHEUNG A. Leveraging lock contention to improve OLTP application performance[J]. Proceedings of the VLDB Endowment, 2016, 9(5):444-455. doi: 10.14778/2876473 [9] PARRISH A S, ZWEBEN S H. Clarifying some fundamental concepts in software testing[J]. IEEE Transactions on Software Engineering, 1993, 19(7):742-746. doi: 10.1109/32.238573 [10] KOREL B. Automated software test data generation[J]. Software Engineering IEEE Transactions on, 1990, 16(8):870-879. doi: 10.1109/32.57624 [11] DEMILLO R A. Progress toward automated software testing[C]//International Conference on Software Engineering. IEEE Computer Society Press, 1991: 180-183. [12] AHMAD S. XML APIs testing using advance data driven techniques (ADDT)[EB/OL]. (2003-11-17)[2017-04-21]. https://www.stickyminds.com/article/xml-apis-testing-using-advance-data-driven-techniques-addt. [13] 曹娜. 大型关系数据库自动化测试工具的设计与实现[D]. 武汉: 华中科技大学, 2009. http://cdmd.cnki.com.cn/Article/CDMD-10487-2010212391.htm [14] FRANKL P G, WEYUKER E J. Testing software to detect and reduce risk[J]. Journal of Systems & Software, 2000, 53(3):275-286. http://www.sciencedirect.com/science/article/pii/S0164121200000182 [15] COOPER B F, SILBERSTEIN A, TAM E, et al. Benchmarking cloud serving systems with YCSB[C]//Proceedings of the 1st ACM Symposium on Cloud Computing. ACM, 2010: 143-154. [16] IBM. TATP: Telecom application transaction processing benchmark[EB/OL]. (2011-07-06)[2017-04-21]. http://tatpbenchmark.sourceforge.net. [17] KODAGANALLUR V. Incorporating language processing into Java applications:A JavaCC tutorial[J]. IEEE Software, 2004, 21(4):70-77. doi: 10.1109/MS.2004.16 [18] SILBERSCHATZ A, KORTH H F, SUDARSHAN S. Database System Concepts[M], 4th ed. Columbus:McGraw-Hill Education, 2001. [19] BERNSTEIN P A, GOODMAN N. Multiversion concurrency control-theory and algorithms[J]. ACM Transactions on Database Systems, 1983, 8(4):465-483. doi: 10.1145/319996.319998 -
点击查看大图
图(9) / 表(3)
计量
- 文章访问数: 169
- HTML全文浏览量: 124
- PDF下载量: 326
- 被引次数: 0