{"id":789,"date":"2021-12-29T14:03:43","date_gmt":"2021-12-29T14:03:43","guid":{"rendered":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/thomas-newman\/?p=789"},"modified":"2022-02-18T12:03:12","modified_gmt":"2022-02-18T12:03:12","slug":"report-anomaly-detection-for-functional-data","status":"publish","type":"post","link":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/thomas-newman\/2021\/12\/29\/report-anomaly-detection-for-functional-data\/","title":{"rendered":"Anomaly detection for functional data"},"content":{"rendered":"\t\t
We call an anomaly, an observation that differs significantly from the rest of the data. These are important to detect as they can indicate the existence of a problem. Early identification could even prevent the problem recurring and avoid subsequently worse issues. There exist different types of anomalies; it may be a single observation that deviates from the rest, which is called a point anomaly or a group of points called a collective anomaly. In this blog however, we will be looking at anomalies in functional data. Figure 1 shows the data used, it consists of measurements made minute by minute across 50 days but we do not know what it measures. Hence, the two main types of anomaly will be, shape anomalies, which are curves that display a different behaviour than the other curves and magnitude anomalies, curves that at some point have a significant spike away from the rest of the curves. To detect these anomalies, we will look at two different methods. The total variation depth and modified shape similarity index (TVDMSS) and the functional outlyingness map using directional outlyingness (FOM(DO)), which methodology is discussed in Section 2. Then, Section 3 contains the results obtained from applying these methods to the data described and compares them. Finally, Section 4 contains the limitations of both methods and a general conclusion. Throughout this blog, the terms anomaly and outliers will be used interchangeably.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t Since we are working with functional data, standard outliers detection techniques (for a single time series) will not be applicable. Here, outliers will have to be detected with respect to each curve.\u00a0The following TVDMSS method is based on the work of\u00a0<\/span>Huang & Sun (2019).<\/a>\u00a0<\/span>For each curves in our functional, data both the total variation depth (TVD) and modified shape similarity (MSS) are computed. The TVD can be interpreted as the total variability in the depth (along an interval of interest) of a curve compared to all other curves. At any given point in time, the deepest curve would be the one closest to the median of all curves at that point in time. The MSS is simply a measure of how similar the shape of a curve is compared to the shape of the all other curves. The lower the similarity the more likely it is to be considered an outlier.<\/span><\/p> Using these measures we can then start detecting anomalies. First, a classical boxplot is constructed using the MSS of each curve. An outlier corresponds to a curve for which the MSS is less than the P \u00d7 50% central region. Were the 50% central region corresponds to the 50% deepest curves. Outliers detected using this approach are considered shape outliers. Next, shape outliers are removed and a functional boxplot is constructed using the TVD. This functional boxplot is based on all the original curves prior to removing the shape outliers. Once again, the outlier detection region is Q \u00d7 50% central region. The RStudio function (tvdmss<\/em><\/a> from the package fdaoutlier<\/em><\/a>) used for this project allowed to independently modify the values of P, Q and the size of the corresponding central regions. Therefore, allowing the sensitivity detection level to be tuned. Outliers detected using the functional boxplot are considered magnitude outliers. In addition, for this blog, anomalies that characterise as both shape and magnitude outliers will also be highlighted, which the standard TVDMSS does not include. This was done by detecting magnitude anomalies while not removing shape anomalies.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t A large number of outlier detection methods rely on using functional depths; as we just saw, this is true for TVDMSS. Dai & Genton (2019)<\/a> argue that these might fall short of properly describing the centrality of functions and not detect anomalies characterised by their directional outlyingness. Therefore, using the work of Rousseeuw et al. (2018)<\/a>, Dai & Genton (2019)<\/a> came up with the functional directional outlyingness which accounts for both the pointwise depth and the directional anomalies.<\/p> Outliers are detected using a functional outlier map (FOM). This is visually represented by a scatterplot displaying points and a cut-off line. If the points are out of the cut-off region they are considered outliers. The points are plotted using a functional directional outlyingness value on the x-axis; this reflects the total outlyingness of a given curve and the variation of the directional outlyingness on the y-axis; which represents the shape outlyingness of that curve. To fit this method,\u00a0 we used the function fOult <\/em><\/a>from the R package mrfDepth<\/a>. <\/em>This function does not allow to change the level of anomaly detection, therefore, no tuning is possible for how strict we want to be with our anomalies.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t This section presents the results from applying the TVDMSS and FOM(DO) to the data presented in Figure 1.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t After manually tuning the TVDMSS algorithm to a level of detection which empirically seemed reasonable (P = 0.5 and Q = 1.1) we obtain the following results. As shown in Figure 2, three shape anomalies were detected using the classical boxplot of MSS. These are represented by the red circle below the boxplot.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t Next, the magnitude outliers were identified using the functional boxplot as seen in Figure 3. As we can see, two magnitude anomalies were detected, represented by the blue lines. The pale area corresponds to the 50% central region and the slightly darker lines are the anomaly thresholds.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t Finally, Figure 4 displays all the anomalies detected using the TVDMSS. We can see that one anomaly was classified as both a shape and magnitude anomaly. Furthermore, the shape anomaly detected by the boxplot of MSS corresponding to the lowest red line at t = 0 seems questionable. Empirically, it does not look very different from the other curves.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t2. Methods<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
2.1 Total variation depth and modified shape similarity index<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
2.2 Functional outlyingness map using directional outlyingness <\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
3. Results<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
3.1 Detecting anomalies using Total variation depth and modified shape similarity index
<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t