Repository mPlane interfaces for Tstat

Repository mPlane interfaces for Tstat is a set of mPlane protocol-based tools to import logs and RRDs generated by Tstat (developed by POLITO and FTW) into mPlane compliant repositories. The code is located in github page of mPlane.

Currently the repository interfaces supports three different indirect importing approaches:

1. 1. Log bulk importer
   2. Log streaming importer
   3. RRD importer 

The repository interface plays the role of a proxy which imports data from Tstat probes using the mPlane protocol. The repository proxy register itself to the supervisor and enables its indirect export capabilities at the registration phase. Simple ad-hoc protocols have been developed to setup the asynchronous and indirect export between the Tstat proxy, and the repository proxies.

To activate the indirect import capabilities on Tstat logs and RRDs, it is recommended to activate the tstatrepository.py as well as the tstat.py, as described in the Tstat page.

You can run the mPlane repository interface by entering into protocol-ri folder and execute:

>>>  export PYTHONPATH=.
>>> ./scripts/mpcom --config ./mplane/components/tstat/conf/tstatrepository.conf

The tstatrepository component in the protocol-ri supports all the aforementioned indirect export techniques. We detail their some use example in the following.

• Log bulk importer
  
  

  The log bulk importer uses MATH (Mplane Authorized Transfer via HTTP) developed by FTW to import data from Tstat and store them in DBStream. It can be customized to dump logs to different databases, or to simply copy and store them on the repository filesystem.

  In mPlane, client can run the specification from now to unlimited future. For instance, in the example below, we instruct Tstat proxy to start exporting data from the probe to a repository proxy, which is running on the same host (localhost).

  |mplane| runcap tstat-exporter_log
  

  |when| = now + inf

  repository.url = localhost:3000 
  ok
  

  NOTE: The repository.url contains the IP address of repository and the port value associated to repository_log_port.

• Log streaming importer
  

  The repository interface enables the streaming of logs collected in real-time by Tstat. The code contains in repository_streaming_importer.py acts as a simple endpoint server which receives the streamed log and redirect them to the stdout. It is possible to customize the code of the file repository_streaming_importer.py to redirect the data to other consumers such as, e.g., a WeBrowse module.

  For instance, to activate the streaming indirect export of log_tcp_complete for 1 day, execute (as above, repository and probe running both on localhost):

  |mplane| runcap tstat-exporter_streaming
  |when| = now + 1d
  log.folder = /path/to/log/folder/     # insert here the path where the logs are
  log.time = 60
  log.type = log_tcp_complete
  repository.url = localhost:9001
  ok
  

  NOTE: The repository.url contains the IP address of the repository and the port value associated to repository_streaming_port in tstatrepository.py.

• RRD importer
  

  The repository proxy imports the RRD files collected by the Tstat. The RRD files are then sent to Graphite for storage and graphical presentation.

  For instance, to activate the RRD indirect export form now to 1 hour, run at the client (as above, repository and probe running both on localhost):

  |mplane| runcap tstat-exporter_rrd
  |when| = now + 1h
  repository.url = localhost:9000
  ok
  

  NOTE: The repository.url contains the IP address of the repository and the port value associated to repository_rrd_port in tstatrepository.py.

Official version

• July 10th, 2015 - frozen release for D3.4
  • Tstatrepository-proxy [tar.gz] -- NEW -- this archive includes tstatrepository.py, all repository_importers (bulk log, streaming and rrd), and the corresponding configuration files. However, we suggest to use the GitHub version. See above.