Wireshark · Wireshark-users: Re: [Wireshark-users] Wireshark & monitoring in the enterprise environment

[mv652@xxxxxxxxxxxx]

Hi Martin, 

To answer your questions (and again, note this only applies to 
high-frequency algo-type trading and microsecond/millisecond analysis) 

"Sure, latency monitoring using Wireshark wouldn't do, however, AFAIU,
monitoring of the throughput - say with a window 1 sec long - may work, no?"
1 sec these days is the equivalent of 10 seconds polling a few years ago.  
If your polling rate is every 10seconds or 30seconds, what happens to peaks 
of traffics within those seconds?  They are not reported.
Similarly, with 1sec polling, you don't see a spike that lasts 
100milliseconds or 100microseconds.  On a particular service we use, spikes 
at this time scale are told to go from ~6mbps to +110mbps. 


"As for the cards, I recall Endace claiming microsecond (or even lower)
granularities. Does it mean that when using Wireshark for the analysis
of the capured data, microseconds are simply stripped off?"
In fact, I recall some partnership between CACE and Endace recently been 
announced (I'm sure some people on this list know more about this).  I 
wonder if this relationship has advantages for CACE's capture card which 
seems very good for the asking price.
I'm assuming that with a capture card (such as endace's) providing a very 
low time granularity, results are more accurate.  However, I am not sure at 
which point within the OS wireshark makes a record of the packets.
Kernel, server load, disk access etc. all play a part when trying to capture 
details at this level. 

The solutions that promise to provide 100% accuracy at this level are 
unfortunately very expensive at this stage.  For those of us with a budget 
to consider we try and come up with a compromise or alternative solution. 

Considering it's free, wireshark has been, and still is, a fantastic tool. 

Again, I think your article serves a good purpose, just keep in mind the 
outcome you are looking for.  The limitation is not within your explanation, 
but rather within the tools available. 


Cheers, 


------------------------------ 

Message: 8
Date: Mon, 09 Feb 2009 11:57:06 +0100
From: Martin Sustrik <sustrik@xxxxxxxxxx>
Subject: Re: [Wireshark-users] Wireshark & monitoring in the
     enterprise      environment
To: Community support list for Wireshark
     <wireshark-users@xxxxxxxxxxxxx>
Message-ID: <49900C02.1040604@xxxxxxxxxx>
Content-Type: text/plain; charset=us-ascii; format=flowed 

Mario, 

Thanks for comments, find my replies inlined. 

mv652@xxxxxxxxxxxx wrote:
> In my opinion, I think you've written a well documented introduction for
> anyone looking to monitor Financial Trading (or similar) Data.  It gives a
> very nice and simple methodology to get a 'general feel' for the traffic
> passing over the network. 
>
> One caveat I'd add with regards to monitoring Financial Trading activity
> (and I'm happy if anyone can explain any different), is that this is good
> for a general network data analysis.  For high-frequency algo-type trading
> activity, wireshark seems to have found it's limitation (at least for the
> moment). 
>
> The granularity of the wireshark I/O is at a minimum of 0.001sec (1
> millisecond).  In high-frequency trading we are now monitoring at the depths
> of microseconds (0.000001 sec) as opposed to milliseconds.  There are two
> problems as a consequence:
> 1) The NIC used for 'monitoring' is unable to poll at these levels (CACE's
> TurboCap only provides 3-5 microsecond granularity, and this is a
> 'specialized' monitoring card)

Sure, latency monitoring using Wireshark wouldn't do, however, AFAIU,
monitoring of the throughput - say with a window 1 sec long - may work, no? 

As for the cards, I recall Endace claiming microsecond (or even lower)
granularities. Does it mean that when using Wireshark for the analysis
of the capured data, microseconds are simply stripped off? 

> 2) Wireshark does provide microsecond timsetamps for the data, but the
> accuracy of the data is questionable, dependable on the hardware used and
> other processes occuring at the time on the monitoring system. 
>
> Please don't get me wrong.  I think wireshark is a fantastic network
> analysis tool and a lot of good work has been done by the authors and the
> community to make it the great piece of software it is today.  Just to point
> out that I think you should indicate there is a limitation to the analysis
> of data that can be captured. 
>
> I only say this purely due to the fact the document been written analysing
> Financial Trading traffic.

Yup, understood. 

Martin