Tstat

Description:

Tstat is an open source passive traffic analyzer developed at Telecommunication Network Group (TNG) of Politecnico di Torino. It started as a branch of TCPtrace with a focus on TCP statiscs only, but over the years it evolved in a full fledge monitoring solution offering an extensive set of measurements for IP, TCP and UDP, as well as traffic classification capabilities through a combination of Finite State Machine Deep Packet Inspection (FSM DPI) and Behavioural engines.

As shown in the figure, Tstat can produce real time analysis processing live traffic acquired from both standard PC NICs, using the libpcap standard interface, or dedicated solutions like DPDK for 10Gbps+ live analysis using off the shelf hardware, or dedicated solutions like DAG cards.
Tstat can also process previously recorded packet-level traces of various formats (libpcap, snoop, etherpeek, etc.).

Tstat supports different output formats:

• log files: text files collecting per-flow stats. Each file is related to a different protocol or application (e.g., TCP/UDP traffic, video streaming, etc.); For each flow, Tstat will produce a detailed set of statistics like bytes/packets sent, received, retransmitted, Round-Trip-Time, application protocol, etc.
  A sample trace with some post processing examples is available at Tstat website.
• histograms: a set of histogram, each one related to a specific traffic features (e.g., incoming/outgoing IP bitrate, number of TCP flows, etc.). Statistics are collected over a time window (by default 5 minutes) and saved in separated text files for easy postprocessing.
• RRD: a set of histograms saved in RRD format, which can be easily browsed to produce historical plots using a simple web interface -- or exported to be used by a Graphite GUI.
  An example of measurements is available at http://tstat.tlc.polito.it:8002/dashboard/
• pcap: dump packets in pcap files, using advanced filtering, e.g., dump only BitTorrent or Youtube traffic. This is useful to capture packets trace of selected traffic, limiting disk usage.

Quick start:

We recommend the user to refer to the last version available on SVN:

svn checkout http://tstat.polito.it/svn/software/tstat/trunk tstat

The software works for Linux, BSD, Mac OS, and Android. To compile the software, from the Tstat main software folder

./autogen.sh
./configure
make
sudo make install

Last command is optional and it will install tstat into /usr/local/bin.

Note: to compile on Android, please refer to http://tstat.polito.it/svn/software/tstat/trunk/doc/android.txt

To run Tstat in live mode from a network interface DEVNAME (e.g., eth0) and create logs in the directory OUTDIR, run:

sudo ./tstat/tstat -l -i DEVNAME -s OUTPUTDIR

For more information, please refer to the official website http://tstat.polito.it where complete and detailed HOWTO can be found.

New features supported by the mPlane project

Thanks to the support of the mPlane project we extended Tstat functionalities with the following features

• HTTP module: it allows to save text log files reporting information about HTTP queries/responses.
• IP address anonymization: it allows one to mask local IP address monitored using hashing functions using the CryptoPan to preserve prefixes while guaranteeing strong anonymization of addresses.
• Blockmon integration: we collaborated with NEC to integrate Tstat analysis modules in Blockmon. It means that Tstat can run as a Blockmon module, and output can be then processeed in a streaming fashion.
• log_sync: a client/server application which allows to continuously export from Tstat logs from probes to repositories, offering both a bulk synchronization, and a streaming exporter.
• improved log configurability: rather than collecting a monolitich set of stats, Tstat now offers more fine-grained control on which set of features a saved in the logs. Per-flow stats are now grouped in macro classes which can be added or removed at runtime.
• Improved internal configuration: all parameters can now be configured easily in a separate file, without the need to recompile Tstat.
• Improved performance and optimized memory usage. In particular, now garbage collection operations cannot interfere with packet processing, and create possible packet loss.
• Integration with high speed DPDK packet capture: ENST and POLITO cooperated to build a DPDK-based engine based on Tstat capable of  live processing of traffic at 40Gbps with suitable NICs. Check the instructions on the Tstat-DPDK branch.
• Android integration: thanks to the effort of TID, Tstat works also on rooted Android devices.
• OpenWRT integration: thanks to the effort of NETVISOR, starting from release 37196, the OpenWRT Linux distribution contains Tstat as a "Network Utilities" package.

mPlane proxy interface

Component-initiated (with supervisor)

Tstat mPlane proxy interface is available on github. It allows to expose Tstat as a probe component to the Supervisor, which can then later be used to turn on and off features by setting capabilities, to start exporting results to repositories, etc.

Reminder: the whole mPlane Reference Implementation is built using Python3. All libraries and software below thus refer to Python3 supported tools.

To test the software:

• STEP1: running Tstat

From the root folder of the Tstat software obtained doing a svn checkout

>>>sudo tstat -l -i eth0 -s tstat_output_logs -T tstat-conf/runtime.conf -R tstat-conf/rrd.conf -r /rrdfiles/

-l: enables live capture using libpcap

-i interface: specifies the interface to be used to capture traffic

-s dir: puts the results into directory "dir"

-T runtime.conf: configuration file to enable/disable logs at runtime

-R conf: specifies the configuration file for integration with RRDtool.

-r path: path where to create/update the RRDtool database

For more information, please refer to the official Tstat website.

The example above runs Tstat in live capture mode, from the eth0 interface, using the default runtime and RRD configuration, and saving the logs in tstat_output_logs folder.

• STEP2: download and configure the Tstat mPlane proxy

>>> git clone https://github.com/fp7mplane/protocol-ri.git
>>> cd protocol-ri/mplane/
>>> mv components components.orig (or rm -rf components)
>>> git clone https://github.com/fp7mplane/components/
>>> cd ../

Add to the [Authorizations] section of the 'conf/supervisor.conf' file the new supported capabilities listed below:

tstat-log_http_complete = guest,admin
tstat-exporter_streaming = guest,admin
tstat-log_rrds = guest,admin
tstat-exporter_rrd = guest,admin
tstat-exporter_log = guest,admin
repository-collect_rrd = guest,admin
repository-collect_streaming = guest,admin
repository-collect_log = guest,admin

• STEP3: running the standard mPlane supervisor

Open a new terminal, enter 'protocol-ri' folder and execute:

>>> export PYTHONPATH=.
>>> ./scripts/mpsup --config ./conf/supervisor.conf

• STEP4: running the mPlane Tstat proxy

Open a new terminal, and enter 'protocol-ri' folder. 

Change the paths inside Tstat's configuration files in ./mplane/components/tstat/conf/tstat.conf . If you run the Tstat as presented in STEP1, change the paths to:

runtimeconf = PATH_TO_TSTAT/tstat-conf/runtime.conf
tstat_rrd_path = /rrdfiles/

>>>  export PYTHONPATH=.

>>> ./scripts/mpcom --config ./mplane/components/tstat/conf/tstat.conf

Added <Service for <capability: measure (tstat-log_tcp_complete-core) when now ... future token 7ee0a281 schema 39952155 p/m/r 0/3/42>>
Added <Service for <capability: measure (tstat-log_tcp_complete-end_to_end) when now ... future token 1ab5668d schema ce6233ed p/m/r 0/3/7>>
Added <Service for <capability: measure (tstat-log_tcp_complete-tcp_options) when now ... future token 68cc4936 schema 62657c35 p/m/r 0/3/46>>
Added <Service for <capability: measure (tstat-log_tcp_complete-p2p_stats) when now ... future token 9963ddd3 schema 348428a8 p/m/r 0/3/6>>
Added <Service for <capability: measure (tstat-log_tcp_complete-layer7) when now ... future token c03c028d schema c445bac9 p/m/r 0/3/4>>
Added <Service for <capability: measure (tstat-log_rrds) when now ... future token f0bab5b4 schema 3231d66d p/m/r 0/3/9>>
Added <Service for <capability: measure (tstat-exporter_rrd) when past ... future token 58d1109f schema 1216bc3b p/m/r 1/3/3>>
Added <Service for <capability: measure (tstat-log_http_complete) when now ... future token d8c8fa10 schema d2deb3c1 p/m/r 0/3/17>>
Added <Service for <capability: measure (tstat-exporter_streaming) when now ... future token 2ad7da68 schema ffb9654b p/m/r 4/3/0>>
Added <Service for <capability: measure (tstat-exporter_log) when past ... future token eb1e0c4f schema 9baaae2e p/m/r 1/3/0>>

 NOTE: For dependecies, you need to install the psutil, unidecode and the rrdtool python3 modules first. The easiest way is to use pip3.

In the Supervisor screen you should see the Tstat proxy registering and exposing capabilities.

• STEP5: run the mPlane repository proxy

Open a new terminal, create a certificate using the 'create_component_cert.sh' script for repository proxy. Please refer to HOWTO.txt for more information.

NOTE: it is recommended to use Repository-Polito as name for the certificate in order to be compatible with tstatrepository.conf by default.

Enter protocol-ri folder and execute:

>>>  export PYTHONPATH=.
>>> ./scripts/mpcom --config ./mplane/components/tstat/conf/tstatrepository.conf

Added <Service for <capability: measure (repository-collect_rrd) when past ... future token 5628ceb3 schema 1216bc3b p/m/r 1/3/3>>
Added <Service for <capability: measure (repository-collect_streaming) when now ... future token 39de2de6 schema 9baaae2e p/m/r 1/3/0>>
Added <Service for <capability: measure (repository-collect_log) when past ... future token 0ccb3dc4 schema 9baaae2e p/m/r 1/3/0>>

NOTE: The repository proxy expected that the Graphite and DBStream are running on default setting.

As above, on the Supervisor window the capabilities of the proxies should be visible.

• STEP6: run the standard mPlane client, and connect to the Supervisor.

Open a new terminal, enter protocol-ri folder and execute:

>>>  export PYTHONPATH=.
>>> ./scripts/mpcli --config ./conf/client.conf

ok
mPlane client shell (rev 20.1.2015, sdk branch)
Type help or ? to list commands. ^D to exit.
|mplane| listcap    # list the available capabilities
Capability repository-collect_log (token 0ccb3dc4c3290bbbb63caeb1a9f44a6d)
Capability repository-collect_rrd (token 5628ceb366cf23076ab131f701b6dbd0)
Capability repository-collect_streaming (token 39de2de6bf9e6b1423cb42f258a40683)
Capability tstat-exporter_log (token eb1e0c4f3b91673a52733948ef0a9c98)
Capability tstat-exporter_rrd (token 58d1109f591c69dba93193bc88688131)
Capability tstat-exporter_streaming (token 2ad7da68b76a91e9ed96c8d90c0df4b3)
Capability tstat-log_http_complete (token d8c8fa10aa833948024fc18477f1d69b)
Capability tstat-log_rrds (token f0bab5b4e6305b9faf5e4bdbdc4f71a5)
Capability tstat-log_tcp_complete-core (token 7ee0a2812d9decc8085f2b204df0af7d)
Capability tstat-log_tcp_complete-end_to_end (token 1ab5668dd6da4000cee08007cee23a73)
Capability tstat-log_tcp_complete-layer7 (token c03c028db352b9a41852cad68b37df4b)
Capability tstat-log_tcp_complete-p2p_stats (token 9963ddd359a679e6b48c0c9d0b282782)
Capability tstat-log_tcp_complete-tcp_options (token 68cc4936ffec4c0aab829796d8a07c74)
|mplane|

•  Activating passive measurements

The Tstat mPlane proxy allows to activate and control all passive measurements offered by Tstat. Check here for a complete documentation.
For instance, to activate the collection of Core TCP set in log_tcp_complete for 30 minutes, enter in the client window:

|mplane| runcap tstat-log_tcp_complete-core
|when| = now + 30m

To activate the RRD collection forever, run:

|mplane| runcap tstat-log_rrds
|when| = now + inf

NOTE: To reset the scheduling option when you need to run:

|mplane| unset when

• Activating indirect log and RRD exporting

  Now that Tstat started logging TCP flows, and filling in RRD data, we can start the Tstat repository proxy to export this data into mPlane compliant repositories. Currently the proxy offers three different indirect exporting approaches:

  1. 1. Log bulk exporter
     2. Log streaming exporter
     3. RRD exporter

1. 1. 1. Activating Log bulk exporter
         The Tstat proxy sends log files collected by Tstat to the repository proxy. The log files are then stored in DBStream, using an asynchronous indirect export, supported by a custom protocol.
         To activate the log indirect export form now to forever, run in the client:

         |mplane| runcap tstat-exporter_log
         

         |when| = now + inf

         repository.url = localhost:3000 
         ok
         
         

         NOTE: The repository.url contains the IP address of repository and the port value associated to repository_log_port.

2. Activating log streaming exporter
   The exporter enable the streaming of logs collected in real-time by Tstat. It will create a direct and asynchronous channel with the repository, where rows in the selected logs will be piped into. The code contains in tstatrepository.py acts as a simple endpoint server which receives the streamed log and redirect them to stdout (or to a file -- just redicted stdout).

   For instance, to activate the streaming indirect export of log_tcp_complete for 1 day, run at the client:
   

   |mplane| runcap tstat-exporter_streaming
   |when| = now + 1d
   log.folder = /path/to/log/folder/   # this is where the logs are stored
   log.time = 60
   log.type = log_tcp_complete
   repository.url = localhost:9001
   ok

   NOTE: The repository.url contains the IP address of the repository and the port value associated to repository_streaming_port in tstatrepository.py.

   
   

3. Activating RRD exporter
   The Tstat proxy sends the RRD files collected by the Tstat to the repository proxy. The RRD files are then sent to Graphite for storage and graphical presentation using the Graphite (and possibly Grafana) GUI.

   For instance, to activate the RRD indirect export from now, and for 1 hour, at the client interface just run:

   |mplane| runcap tstat-exporter_rrd
   |when| = now + 1h
   repository.url = localhost:9000
   ok
   
   

   If the specification execute properly, the data will be available on Graphite web interface.

   NOTE: The repository.url contains the IP address of repository and the port value associated 
   to repository_rrd_port.

 Official version

• July 10th, 2015 - frozen release for D2.3
  • Tstat v3.0 [tar.gz] -- NEW -- we suggest to use the SVN version. See above.
  • Tstat-DPDK v0.9 [tar.gz] -- NEW -- we suggest to use the SVN version. See this page.
  • Tstat-proxy [tar.gz] -- NEW -- this archive includes tstat.py, all tstat_exporters (bulk log, streaming and rrd), and the corresponding configuration files. However, we suggest to use the GitHub version. See above.
• May 15th, 2014 - frozen release for D2.2
  • Tstat v2.4 [tar.gz] -- OLD -- kept here for historical reasons.