Description:

The mobile probe stores data in a mongoDB database in json format.

For this purpose a mongoDB to mPlane reference implemenation proxy has been created.
The proxy is not specific to the mobile proxy and, therefore, it can be configureed to extract data from any mongoDB schema. 

Repository Description:

The repository consists of a MongoDB database and a node.js interface. The Mobile Probe sends HTTP requests from the device to the server with the aggregated measurements. The node.js script is in turn responsible for receiving, parsing and storing the incoming data to the database. Each measurement is inserted with fields such as timestamp, the device's unique ID and IP address. In this way, the repository can handle concurrent reports from multiple devices. As can be seen in the provided sample below, a report contains the required performance metrics from the different layers that the probe is monitoring.

{ 
  "report":{ 
    "measurementsArray":[ 
      { 
        "timestamp":"2015-03-31 13:19:57.608",
        "LOCATION":{ 
          "timeStamp":1427800797607,
          "provider":"NO_LOCATION",
          "measurementType":"location",
          "contentType":"Measurement"
        },
        "extras":"mplane",
        "deviceID":"352605059221028",
        "ip":"83.53.30.1",
        "version":"2"
      },
      { 
        "CELL_INFO":{ 
          "measurementReason":"onSignalStrengthsChanged",
          "networkOperatorName":"",
          "currentCellLocationCID":125067479,
          "timeStamp":1427800798013,
          "measurementType":"cellInfo",
          "contentType":"Measurement",
          "neighborCellInfo":[

          ],
          "currentCellLocationLAC":864,
          "dataActivity":,
          "networkOperatorCountryCode":"es",
          "latestGSMBitError":99,
          "currentCellLocation":"[864,125067479,-1]",
          "getNetworkType":3,
          "currentCellLocationPSC":-1,
          "dataState":,
          "latestGSMSignalStrength":30,
          "networkOperatorCode":"21407"
        },
        "timestamp":"2015-03-31 13:19:58.050",
        "extras":"mplane",
        "deviceID":"352605059221028",
        "ip":"83.53.30.1",
        "version":"2"
      },
      { 
        "timestamp":"2015-03-31 13:19:58.851",
        "HW":{ 
          "network_type":"WiFi",
          "wifi_RSSI":-37,
          "Cpu Tot: ":1,
          "MemFree":"20908"
        },
        "extras":"mplane",
        "deviceID":"352605059221028",
        "ip":"83.53.30.1",
        "version":"2"
      },
      { 
        "timestamp":"2015-03-31 14:54:55.716",
        "LogCat":"I/AwesomePlayer( 1854): setDataSource_l(URL suppressed)",
        "version":"2",
        "deviceID":"358848043406974",
        "ip":"83.53.30.1",
        "extras":"mplane"
      },
      { 
        "timestamp":"2015-03-31 14:54:56.546",
        "LogCat":"I/AwesomePlayer( 1854): Could not offload audio decode, try pcm offload",
        "version":"2",
        "deviceID":"358848043406974",
        "ip":"83.53.30.1",
        "extras":"mplane"
      },
      { 
        "tcp_complete":{ 
          "c_sack_opt":"1",
          "s_rtt_avg":"0.914270",
          "s_cwin_max":"59856",
          "c_win_min":"14600",
          "s_last":"11838.654000",
          "s_pkts_dup":"0",
          "s_rtt_cnt":"965",
          "s_rtt_max":"36.791000",
          "c_pkts_unk":"0",
          "c_bytes_all":"16916",
          "s_pkts_data":"2522",
          "s_pkts_unrto":"0",
          "c_pkts_reor":"0",
          "s_syn_retx":"0",
          "c_win_max":"522880",
          "p2p_st":"0",
          "ed2k_c2s":"0",
          "c_pkts_push":"17",
          "c_ssl":"r3---sn-h5q7ened.googlevideo.com",
          "c_pkts_rto":"0",
          "c_rtt_cnt":"18",
          "s_sack_opt":"1",
          "ed2k_c2c":"0",
          "c_bytes_retx":"0",
          "c_mss":"1460",
          "c_bytes_uniq":"16916",
          "ed2k_sig":"0",
          "s_bytes_all":"3481036",
          "c_pkts_data":"17",
          "c_cwin_min":"235",
          "s_pkts_ooo":"19",
          "c_pkts_retx":"0",
          "c_rtt_std":"60.187121",
          "durat":"41905.741000",
          "c_win_scl":"6",
          "c_rtt_min":"21.642000",
          "s_pkts_fc":"0",
          "http_t":"0",
          "c_f1323_opt":"1",
          "c_rtt_max":"187.524000",
          "s_ssl":"*.c.docs.google.com",
          "s_first_ack":"218.297000",
          "ed2k_chat":"0",
          "s_rst_cnt":"0",
          "c_ack_cnt_p":"1195",
          "first_abs":"1427836180809.872070",
          "s_win_0":"0",
          "s_pkts_fs":"17",
          "s_rtt_min":"0.051000",
          "c_cwin_max":"1093",
          "s_cwin_ini":"1440",
          "c_rtt_avg":"73.632758",
          "s_bytes_retx":"0",
          "s_mss":"1460",
          "c_ip":"192.168.1.243",
          "c_pkts_unfs":"0",
          "s_win_scl":"7",
          "last":"249797.323000",
          "c_pkts_dup":"0",
          "c_pkts_unrto":"0",
          "s_fin_cnt":"1",
          "s_pkts_unfs":"0",
          "c_sack_cnt":"235",
          "s_rtt_std":"2.552807",
          "s_win_max":"63872",
          "s_ack_cnt":"2529",
          "s_pkts_retx":"0",
          "s_win_min":"28960",
          "c_mss_min":"235",
          "s_port":"443",
          "s_pkts_unk":"0",
          "s_ttl_max":"57",
          "s_pkts_rto":"2",
          "s_ack_cnt_p":"5",
          "c_port":"40886",
          "c_isint":"1",
          "s_sack_cnt":"0",
          "c_ttl_max":"64",
          "c_tm_opt":"1",
          "s_pkts_all":"2529",
          "s_pkts_push":"44",
          "c_ttl_min":"64",
          "c_pkts_all":"1213",
          "c_mss_max":"1093",
          "s_f1323_opt":"1",
          "c_syn_retx":"0",
          "c_last":"11217.761000",
          "s_isint":"0",
          "c_pkts_fs":"0",
          "s_mss_max":"1440",
          "s_mss_min":"51",
          "c_first_ack":"22.055000",
          "s_pkts_reor":"0",
          "ed2k_data":"0",
          "c_pkts_fc":"0",
          "c_syn_cnt":"1",
          "c_fin_cnt":"0",
          "c_cwin_ini":"235",
          "s_syn_cnt":"1",
          "c_pkts_ooo":"0",
          "c_first":"30.773000",
          "s_first":"218.529000",
          "con_t":"8192",
          "s_ip":"74.125.168.8",
          "p2p_t":"0",
          "c_rst_cnt":"0",
          "c_win_0":"0",
          "s_tm_opt":"1",
          "s_cwin_min":"51",
          "s_ttl_min":"53",
          "first":"207891.582000",
          "c_ack_cnt":"1212",
          "s_bytes_uniq":"3481036"
        },
        "timestamp":"2015-03-31 22:09:40.809",
        "version":"2",
        "deviceID":"358848047597851",
        "ip":"83.53.30.1",
        "extras":"mplane"
      }
    ]
  }
}

mPlane proxy interface

The proxy can be found here: https://github.com/fp7mplane/components/blob/master/mongoDB-mobileProbe/(link is external)

The proxy Interface is written in python and it sits on top of MongoDB. It exports the collections of mongoDB as capabilities using the Reference Implementation. Afterwards, the mPlane clients can pull the requested data. 

The mongoDB proxy can be used by any component that has data stored in a mongoDB database. For details on how to install and run the probe visit the github help file:

https://github.com/fp7mplane/components/blob/master/mongoDB-mobileProbe/(link is external)

Installation

Copy files from the mPlane interface (from the components GitHub) into protocol-ri/: - mobile_probe_settings.json The registry.json file, copy it into protocol-ri/mplane/components/. - mongo.py The Python interface, copy it into protocol-ri/mplane/components/. - supervisor.conf , component.conf and client.conf The configuration files, copy them into protocol-ri/conf/.

Configuration

The mongoDB proxy can be used by any component that has data stored in a mongoDB database. The configuration is given at a json file of the following format.

{  
   "measurements":{  
      "mobile_probe_hardware":{  
         "collection":"mobileMeasurements",
         "search":"HW",
         "return":{  
            "observer.link":"HW.network_type",
            "snr":"HW.wifi_RSSI",
            "cpuload":"HW.Cpu Tot: ",
            "memload":"HW.MemFree"
         }
      },
      "mobile_probe_cellInfo":{  
         "collection":"mobileMeasurements",
         "search":"CELL_INFO",
         "return":{  
            "snr":"lCELL_INFO.atestGSMSignalStrength",
            "observer.link":"CELL_INFO.currentCellLocation"
         }
      },
      "mobile_probe_videoLog":{  
         "collection":"mobileMeasurements",
         "capability_name":"mobile_probe_videoLog",
         "search":"LogCat",
         "return":{  
            "measurement.identifier":"LogCat"
         }
      }
   },
   "collection":"mobileMeasurements",
   "db":"mPlane",
   "timestamp_field":"date",
   "source_filed":"deviceID"
}

The settings above will pull data from db mplane and collection mobileMeasurements. The field "devicID" will be used as the source field when making a query for a specific device. Furthermore, the timestamp field is "date"

The measurements dictionary contains information about the offered capabilities and the association of the return values between the mongoDB database and the mPlane reference implemenation. For instance, the mongoDB entryHW.wifi_RSSI will be returned as the mPlane's RI snr field.

The settings above will generate three capabilities.

query (mobile_probe_hardware) 
query (mobile_probe_videoLog) 
query (mobile_probe_cellInfo)

Launching the data extractor

After installing the RI, in a terminal window start supervisor:

cd ~/protocol-ri
python3 -m scripts/mpsup --config ./conf/supervisor.conf

alternatively:

cd ~/protocol-ri 
python3 -m mplane.supervisor --config ./conf/supervisor.conf

In another terminal start the mobile probe as a component:

cd ~/protocol-ri 
python3 -m scripts/mpcom --config ./conf/component.conf 

The expected output for this example should be:

...
Capability registration outcome:
mobile_probe_hardware: Ok
mobile_probe_videoLog: Ok
mobile_probe_cellInfo: Ok

Checking for Specifications...

Retreiving data

You need to have mongoDB setup and running on your machine. You should also have the settings.json configured in order to access and translate the mongoDB field into the mPlanes RI entries (see above).

Start a client to test the component:

mpcli --config ./conf/client.conf

The expected output should be:

ok
mPlane client shell (rev 20.1.2015, sdk branch)
Type help or ? to list commands. ^D to exit.

Now check that the mongoDB capabilities is registered:

|mplane| listcap
Capability mobile_probe_cellInfo (token cd6b5fa2dc4ddfd3d7c4f16898bf1b60)
Capability mobile_probe_hardware (token f3a12891623783d20d4f63f01e86104d)
Capability mobile_probe_videoLog (token 57a0a9484788520090bf3e1c97aa6df8)

And run a test capability. In this example we ask for any mongoDB hardware entries for mobile
device with IMEI 352605059221028 between 2013-09-20 and 2015-5-5:

|mplane| runcap mobile_probe_hardware
|when| = 2013-09-20 ... 2015-5-5
source.device = 352605059221028
ok

If everything works ok the data should be pulled from the mongoDB database and transformed into an mPlane result:

|mplane| listmeas
Result  mobile_probe_hardware-0 (token 6ed23ada2c8921f7cf9de917402a4283): 

2015-03-31 11:19:58.851000 ... 2015-03-31 11:20:04.377000

You can now access the measurments:

|mplane| showmeas 6ed23ada2c8921f7cf9de917402a4283
result: query
    label       : mobile_probe_hardware-0
    token       : 6ed23ada2c8921f7cf9de917402a4283
    when        : 2015-03-31 11:19:58.851000 ... 2015-03-31 11:20:04.377000
    registry    : http://ict-mplane.eu/registry/demo
    parameters  ( 1): 
                           source.device: 352605059221028
    metadata    ( 1): 
                          System_version: 1.0
    resultvalues( 2):
          result 0:
                                     cpuload: 1.0
                                     memload: 20908.0
                               observer.link: WiFi
                                         snr: -37.0
          result 1:
                                     cpuload: 0.6805555820465088
                                     memload: 44084.0
                               observer.link: WiFi
                                         snr: -39.0