Electric energy abnormal data detection based on Isolation Forests
-
摘要: 随着电力系统信息化建设的深入，用户对于电能量数据的质量要求逐渐提高，因此保证海量电能量数据的准确性、可靠性以及完整性具有重要意义.本文采用一种基于孤立森林的异常检测算法，实现大规模电能量数据的异常检测.孤立森林算法通过划分大规模电能量数据集，生成随机二叉树和孤立森林构建模型，通过计算测试电能量数据样本到每棵树的根结点的距离检测异常数据点.该算法不仅能够快速处理海量数据，而且结果准确、可靠性高.本文在大规模电能量数据的正向有功总电量PAP和反向有功总电量RAP字段上进行检测，实验结果表明，该算法检测效率较高，并具有较高的检测正确率.Abstract: With the development of power information systems, users' requirements for the quality of power data has gradually increased. Hence, it is important to ensure the accuracy, reliability, and integrity of massive power data. In this paper, an anomaly detection algorithm based on Isolation Forests is used to realize anomaly detection of large-scale electric energy data. Isolation Forest algorithms generate random binary trees and isolated forest models by dividing training samples and detecting abnormal data points. The algorithm can not only process massive data quickly, but it also offers accurate results and a high degree of reliability. In this paper, the positive active total power (PAP) and reverse active total power (RAP) fields of large-scale electric energy data are determined. The experimental results show that the algorithm has high detection efficiency and accuracy.
-
Key words:
- Isolation Forest /
- anomaly detection /
- electric energy data
-
图 2 基于孤立森林的电能量异常检测流程架构
Fig. 2 Processing architecture for the abnormal detection of electric energy based on iForest
图 3 iTree构成和电能量异常预测示意图
Fig. 3 Schematic of iTree composition and electric energy anomaly prediction diagram
图 4 字段PAP电能量数据异常检测结果
Fig. 4 Abnormal detection result of electric energy data in field PAP
图 5 字段PAP电能量数据异常检测结果(纵坐标用对数刻度表示)
Fig. 5 Abnormal detection result of electric energy data in field PAP (logarithmic scale for ordinates)
图 6 字段RAP电能量数据异常检测结果
Fig. 6 Abnormal detection result of electric energy data in field RAP
算法1 $iForest(D, t, \psi)$ 输入: $D-$大规模电能量数据集, $t$-iTree的数量, $\psi $-每棵iTree中电能量样本数 输出: $t$棵iTrees构成的孤立森林 1: 对电能量数据集$D$进行预处理 2: 设置iTree的最大高度$l=ceiling(\log _2 \psi)$ 3: for i=1 to t do 4: $D'\leftarrow sample(D, \psi)$ 5: $iForest\leftarrow iForest\cup iTree(D', 0, l)$ 6: end for 7: return iForest 
下载: 导出CSV
算法2 $iTree({D', e, l})$ 输入: $D'-$输入电能量数据子样本集, $e$-当前iTree高度, $l$-限定iTree的最大高度 输出: 一棵iTree 1: if $e\ge l{\rm or} | {D'}|\le 1$ 2: return ex Node {Size$\leftarrow \vert D'\vert \}$ 3: end if 4: else 5: $Q$是数据集$D'$的属性集; 随机取某一属性$q\in Q$ 6: 随机选择分割点$p$, $p$的取值在$q$的取值范围内 7: $D'_l \leftarrow filter(D', q < p)$ 8: $D'_r \leftarrow filter(D', q\ge p)$ 9: end else 10: return $inNode\{Left\leftarrow iTree(D'_l, e+1, l), $ $ Right\leftarrow iTree(D'_r, e+1, l), $ $SplitAtt\leftarrow q, $ $SplitValue\leftarrow p\}$
下载: 导出CSV
算法3 $Outlier\_\det ection\left({T, x} \right)$ 输入: $T-$电能量数据集构建的孤立森林, $x$-电能量数据集$D$中某一待检测的电能量数据 输出: 是否为异常值 1: 得到$T$中iTree的数量$t$ 2: for i=1 to $t$ do 3: 得到$iTree_t $包含的电能量数据量$\psi $和高度$h$ 4: 计算$x$到$iTree_t $根结点的距离$h(x)$ 5: end for 6: $E(h(x))=\sum\limits_{i=0}^t {h(x)} $ 7: $S(x, n)=2^{-\frac{E(h(x))}{c(n)}}$ 8: if $S(x, n)\approx 1$ 9: return 1 10: end if 11: else if $S(x, n)\approx 0$ 12: return-1 13: end else if 14: else 15: return 0 16: end else
下载: 导出CSV
表 1 电能量数据异常检测算法对比实验
Tab. 1 Experimental comparison of electrical energy data anomaly detection with different algorithms
算法 异常点个数 精确率 召回率 本算法 6 0.83 1 One-class SVM 18 0.28 1 标准差算法 14 0.14 0.4
下载: 导出CSV
-
[1] 罗志仕, 张晋明.对国内电能量计量系统现状的调查研究[J].大科技, 2013(12):66-67. http://d.old.wanfangdata.com.cn/Periodical/mkxdh200906003 [2] CHANDOLA V, BANERJEE A, KUMAR V. Anomaly detection[J]. ACM Computing Surveys, 2009, 41(3):1-58. http://d.old.wanfangdata.com.cn/Periodical/xajtdxxb201104007 [3] 简富俊, 曹敏, 王磊, 等.基于SVM的AMI环境下用电异常检测研究[J].电测与仪表, 2014, 51(6):64-69. doi: 10.3969/j.issn.1001-1390.2014.06.014 [4] 王增平, 张晋芳, 钱诚.基于同步测量信息的电网拓扑错误辨识方法[J].电力自动化设备, 2012, 32(1):1-8. doi: 10.3969/j.issn.1006-6047.2012.01.001 [5] 王兴志, 严正, 沈沉, 等.基于在线核学习的电网不良数据检测与辨识方法[J].电力系统保护与控制, 2012(1):50-55. doi: 10.3969/j.issn.1674-3415.2012.01.009 [6] ESKIN E, ARNOLD A, PRERAU M, et al. A geometric framework for unsupervised anomaly detection:Detecting intrusions in unlabeled data[M]//Applications of Data Mining in Computer Security. Amsterdam:Kluwer Academic Publisher, 2002:77-101. [7] MONEDERO I, BISCARRI F, LEÓN C, et al. Detection of frauds and other non-technical losses in a power utility using Pearson coefficient, Bayesian networks and decision trees[J]. International Journal of Electrical Power & Energy Systems, 2012, 34(1):90-98. http://www.wanfangdata.com.cn/details/detail.do?_type=perio&id=1168ae75f4620198aa1421e79f4ae7ee [8] PLATT J, SCHÖKOPF B, SHAWE-TAYLOR J, et al. Estimating the support of a high-dimensional distribution[J]. Neural computation, 2001, 13(7):1443-1471. doi: 10.1162/089976601750264965 [9] STEINWART I, HUSH D, SCOVEL C. A classification framework for anomaly detection[J]. Journal of Machine Learning Research, 2005(6):211-232. http://d.old.wanfangdata.com.cn/Periodical/jsjxb201901015 [10] 陈阳, 王勇, 孙伟.基于YARN规范的智能电网大数据异常检测[J].信息网络安全, 2017(7):11-17. doi: 10.3969/j.issn.1671-1122.2017.07.002 [11] 严英杰, 盛戈皞, 陈玉峰, 等.基于大数据分析的输变电设备状态数据异常检测方法[J].中国电机工程学报, 2015, 35(1):52-59. http://d.old.wanfangdata.com.cn/Periodical/zgdjgcxb201501008 [12] 肖坚红, 严小文, 周永真, 等.基于数据挖掘的计量装置在线监测与智能诊断系统的设计与实现[J].电测与仪表, 2014, 51(14):1-5. doi: 10.3969/j.issn.1001-1390.2014.14.001 [13] 魏瑶, 朱伟义, 龚桃荣, 等.基于数据挖掘技术的用电异常分析系统设计[J].电力信息与通信技术, 2014, 12(5):70-73. http://www.cnki.com.cn/Article/CJFDTotal-DXXH201405019.htm [14] 田野, 张程, 毛昕儒, 等.运用PCA改进BP神经网络的用电异常行为检测[J].重庆理工大学学报(自然科学版), 2017, 31(8):125-133. doi: 10.3969/j.issn.1674-8425(z).2017.08.021 [15] LIU F T, TING K M, ZHOU Z H. Isolation forest[C]//2008 Eighth IEEE International Conference on Data Mining. IEEE, 2008: 413-422. [16] 倪永峰, 闫连山, 崔允贺, 等.面向软件定义网络的隐蔽通信检测机制[J].计算机系统应用, 2018, 27(9):143-150. http://d.old.wanfangdata.com.cn/Periodical/jsjxtyy201809023 [17] 朱佳俊, 陈功, 施勇, 等.基于用户画像的异常行为检测[J].通信技术, 2017, 50(10):2310-2315. doi: 10.3969/j.issn.1002-0802.2017.10.032 [18] 李新鹏, 高欣, 阎博, 等.基于孤立森林算法的电力调度流数据异常检测方法[J].电网技术, 2019, 43(4):1447-1456. http://www.wanfangdata.com.cn/details/detail.do?_type=perio&id=CN201711016716.0 [19] 朱炜玉, 史斌, 姜继平, 等.基于水质时间序列异常检测的动态预警方法[J].环境科学与技术, 2018, 41(12):131-137. http://www.wanfangdata.com.cn/details/detail.do?_type=perio&id=hjkxyjs201812019 [20] 韩明涛.时间序列模式挖掘的算法研究[J].山东大学学报(工学版), 2004, 34(3):88-91. doi: 10.3969/j.issn.1672-3961.2004.03.021 [21] 余宇峰, 朱跃龙, 万定生, 等.基于滑动窗口预测的水文时间序列异常检测[J].计算机应用, 2014, 34(8):2217-2220. http://d.old.wanfangdata.com.cn/Periodical/jsjyy201408016 [22] 孙梅玉.基于距离和密度的时间序列异常检测方法研究[J].计算机工程与应用, 2012(20):11-17. doi: 10.3778/j.issn.1002-8331.2012.20.003 [23] 曹旭, 曹瑞彤.基于大数据分析的网络异常检测方法[J].电信科学, 2014, 30(6):152-156. doi: 10.3969/j.issn.1000-0801.2014.06.025 [24] LEYS C, LEY C, KLEIN O, et al. Detecting outliers:Do not use standard deviation around the mean, use absolute deviation around the median[J]. Journal of Experimental Social Psychology, 2013, 49(4):764-766. doi: 10.1016/j.jesp.2013.03.013 -
点击查看大图
图(6) / 表(4)
计量
- 文章访问数: 78
- HTML全文浏览量: 160
- PDF下载量: 1
- 被引次数: 0