You are here

WP4 - mPlane Supervisor: Iterative and Adaptive Analysis

Work package title: mPlane Supervisor: Iterative and Adaptive Analysis
Start date or starting event: M4
Activity Type: RTD
Leader: FTW - Pedro Casas casas@ftw.at
Participants: POLITO TI ALBLF EURECOM ENST NEC TID FW NETVISOR FHA ULG A-LBELL

Objectives

The Supervisor represents the core of mPlane. Based on the intelligent reasoner and a set of analysis modules, it coordinates the measurements and the analysis performed by probes and analyses aggregate data stored in the repository. Its main component, the reasoner, provides the intelligence and adaptability required by the specific monitoring applications supported by mPlane (performance evaluation, root cause analysis, etc.). The objectives of this work package are the following:

  • Design and implement a set of analysis modules capable of automatically processing the historical and/or the real-time data coming from both the data processing and storage layer (WP3) and the measurement layer (WP2) to provide useful pictures of the network and the traffic required by the specific monitoring applications supported by mPlane.
  • Design and implement a reasoner responsible for the orchestration of the iterative analysis and the correlation of the results exposed by the previously mentioned analysis modules. Such reasoning system is capable of generating conclusions and triggering further measurements to provide more accurate and detailed insights regarding the particular monitoring applications. As such, the reasoner offers the necessary adaptability and smartness of the mPlane to find the proper high-level yet accurate explanations to the problems under analysis.
  • Design and implement a knowledge structure where the reasoner accesses and updates the generated knowledge from past analysis experiences related to the particular monitoring applications supported by mPlane. This knowledge structure defines a set of basic domain-knowledge-based rules that allow the reasoner to take decisions based on the particular monitoring application he is orchestrating, which can eventually be expanded by learning from past experiences.

Even if desirable, the Supervisor is not intended to fully automate the monitoring applications supported by mPlane, but rather to better guide and support the mPlane user in the search for accurate explanations and solutions. As such, the knowledge structure will provide certain functionalities in the form of APIs that will allow the user to define and execute particular tasks, based on his knowledge domain.

As already mentioned, this WP represents the core of the measurement plane, where the necessary intelligence for data correlation and analysis will be designed and implemented. As depicted in Figure A, WP4 interacts both with WP2 and/or WP3, depending on the particular monitoring application being targeted (root cause analysis, performance evaluation, “quality of the Internet”, etc.), the particular temporal requirements of the analysis (real-time analysis, historical-data analysis), and the particular inputs needed for the analysis (i.e., new data not previously monitored by the probes at WP2, or not available at WP3).

 

 

Interaction between WP2, WP3, and WP4 

Figure A. Interaction between WP2, WP3, and WP4

 The reasoner in WP4.2 decides which data to use for the analysis, either by asking for pre-processed data from WP3.3 (Access to Analytic Data), or by directly getting data from the probes designed by WP2. WP4.1 provides a set of analysis modules capable of processing and merging both sources of data to get useful information for the specific monitoring applications supported by mPlane. As such, these analysis modules provide different “analysis services” to the reasoner, which correlates and synthesizes results to retrieve high-level answers.

The analysis performed at WP4 and its interaction with WP2 and WP3 are iterative processes, in which intermediate analysis results obtained by WP4.1 trigger additional monitoring and analysis tasks, potentially at components designed by all three WPs. This iterative process is depicted by the closed data workflow - control workflow loops in Figure A.

Description of work

The workpackage contains two tasks:

Partners contribution

  • FTW (WP leader) will work on the definition and implementation of data mining and machine learning techniques for data analysis, correlation and knowledge discovery. FTW will work on the design and conception of mPlane intelligent reasoner, studying knowledge-representation techniques, rule-based reasoning systems, and more advanced learning-based reasoning systems. The interplays between adaptive traffic monitoring and analysis will also be addressed by FTW.
  • POLITO will work on T4.1 (on QoE provided to End-Users specifically focusing on video services) and T4.2 (on troubleshooting of CDN and Cloud based services). It will also work on the correlation and mining of temporal and spatial analysis of data in the context of T4.1
  • TI will work on T4.1 and T4.2, especially on use cases where the role of ISPs is more relevant (e.g., QoE for End-Users, SLA monitoring). TI will also contribute external data (e.g., routing/topology information, etc.) needed to feed the troubleshooting algorithms.
  • ALBLF will work on T4.2 to define methodologies to exploit measurements collected in mPlane to dimension, control, and manage content centric networks and data centres in general.
  • EURECOM will work on T4.2 to the definition of the troubleshooting mechanisms to be sued when facing the analysis of QoE provided to End-Users.
  • ENST will work on T4.1  (by defining advanced algorithms specialized for applications such as BitTorrent/LEDBAT, or cloud applications) and T4.2  (leveraging on the comparative study of passive/active measurement in T2.2 to decide which, when and how to exploit either of the two).
  • NEC will work on T4.1 on the analysis modules related to the use case “troubleshooting the cloud” and on T4.2 on the design of a reasoner that correlates off-line and on-line data with the purpose of troubleshooting network congestion in the cloud as well as monitoring the activities of thin client applications and CDNs for QoE improvements.
  • TID will work on T4.1 (on the smartphone cloud application diagnosis aiming to understand the features that need to be monitored and causality between components in the service delivery). TID will also work on T4.2. The goal of that engagement is to identify whether specific metrics could be obtained actively in order to minimize the energy consumption overhead of continuous measurements on a smartphone.
  • FW will support the definition and analysis of the troubleshooting cases in the context of WP4.
  • NETVISOR will support the integration of WP2 to the requirements of WP4.
  • FHA will work mainly on automated troubleshooting and analysis from an End-User perspective. FHA is interested in root cause analysis and circumvention of problems that need fixing when key players of the Internet are not willing to share mPlane data.
  • ULG will cooperate with NEC to analyse network data collected within the mPlane reasoner, using compressive sensing and discriminative subspace learning in order to improve the accuracy of network event detection.  Further, the mPlane reasoner will be used to derive, from traceroute data, IGP weights of links. 
  • ALBELL effort will focus on the application of on-line and sequential learning in order to perform traffic prediction at multiple timescales, the design of adaptive learning models, and means to perform cooperative monitoring by interconnecting distributed mPlane components running on individual routers.

Deliverables

  • D4.1 – (M12; editor ULG): Design of Analysis Modules. This deliverable is meant to be Public.  This deliverable describes the design and specification of a first set of basic analysis modules for addressing the use cases identified in WP1. These analysis modules include both stream and batch processing algorithms.
  • D4.2 – (M20; editor FTW): Design of the Reasoner. This deliverable is meant to be Public. This deliverable describes the design and specification of the reasoner system with a limited set of iteratively analysis rules as knowledge structure, and also evaluates the possible extensions to be included into the knowledge structure regarding learning of new rules.
  • D4.3 – (M24; editor ULG): Cross-check of Analysis Modules and Reasoner Interactions. This deliverable is meant to be Public. This deliverable presents an extended set of Analysis Modules, including both the improvements done to those presented in deliverable D4.1 as well as the new analysis algorithms designed and developed to address use-cases. The deliverable also describes a complete workflow description for the different use-cases, including both stream processing for real-time monitoring applications as well as batch processing for “off-line” analysis. This workflow description specifies the iterative interaction loop between WP2, WP3, T4.1, and T4.2. Based on this description, the deliverable also presents a cross-check of the analysis modules and the reasoner interactions and additional requirements that must be fulfilled for the final release of the complete Supervisor. Finally, this deliverable presents a detailed intermediate evaluation of the Supervisor (analysis modules and reasoner interaction), relative to some selected use-cases.
  • D4.4 – (M34; editor FTW): Final Implementation of mPlane Supervisor. This software deliverable is meant to be Public.  Final release of the complete mPlane Supervisor, including a complete description of the analysis modules, the reasoner, and its knowledge structure. This deliverable also specifies the final-release workflow tasks for each of the use-cases addressed by mPlane.