Wireshark-users: Re: [Wireshark-users] Experiencing Packet Loss in High Volume Packet Capture App
Date Index · Thread Index · Other Months · All Mailing Lists
Date Prev · Date Next · Thread Prev · Thread Next
From: John Powell <jrp999@xxxxxxxxx>
Date: Mon, 26 Nov 2012 15:03:18 -0600
Hi,

Thanks for your suggestions.

Nothing seems too out of the ordinary with Netstat -s:

# netstat -s
Ip:
    510795 total packets received
    0 forwarded
    0 incoming packets discarded
    509784 incoming packets delivered
    393560 requests sent out
    38236 dropped because of missing route
Icmp:
    656 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 10
        timeout in transit: 3
        echo requests: 643
    653 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 10
        echo replies: 643
IcmpMsg:
        InType3: 10
        InType8: 643
        InType11: 3
        OutType0: 643
        OutType3: 10
Tcp:
    2012 active connections openings
    36 passive connection openings
    16 failed connection attempts
    3 connection resets received
    7 connections established
    504715 segments received
    377170 segments send out
    5428 segments retransmited
    0 bad segments received.
    16 resets sent
Udp:
    4413 packets received
    10 packets to unknown port received.
    0 packet receive errors
    10288 packets sent
UdpLite:
TcpExt:
    2 invalid SYN cookies received
    19 TCP sockets finished time wait in fast timer
    8754 delayed acks sent
    53 delayed acks further delayed because of locked socket
    Quick ack mode was activated 15 times
    220 packets directly queued to recvmsg prequeue.
    126 packets directly received from prequeue
    166272 packets header predicted
    72932 acknowledgments not containing data received
    204520 predicted acknowledgments
    0 TCP data loss events
    78 retransmits in slow start
    1996 other TCP timeouts
    15 DSACKs sent for old packets
    2 DSACKs received
    9 connections aborted due to timeout
    TCPDSACKIgnoredNoUndo: 2
    TCPSpuriousRTOs: 9
    TCPSackShiftFallback: 1
IpExt:
    InMcastPkts: 95
    OutMcastPkts: 126
    InBcastPkts: 102
    InOctets: 2077269099
    OutOctets: 2408075398
    InMcastOctets: 28155
    OutMcastOctets: 29395
    InBcastOctets: 7446

The NIC driver looks adequate to me??

# ethtool -i eth1
driver: tg3
version: 3.122
firmware-version: 5761-v3.80
bus-info: 0000:30:00.0

I think it is a disk contention issue:

LVM | -LogVol_Data | busy    113% | read       0  | write  16384 | KiB/r      0 | KiB/w      4 | MBr/s   0.00 | MBw/s  64.00  | avq 18308.86 | avio 0.06 ms |

DSK |          sdb | busy    113% | read       0  | write    134 | KiB/r      0 |
KiB/w    495 | MBr/s   0.00 | MBw/s  64.81  | avq   143.40 | avio 7.46 ms |

Any thoughts as to if this might be a disk contention issue and if so how to mitigate the problem?

Thanks.

John

On Sun, Nov 25, 2012 at 4:31 AM, Banyan He <banyan@xxxxxxxxxxx> wrote:
check out netstat -s seeing if you can find where it is being dropped. Also remember ethtool -s <int> for the NIC driver level. You probably can try out tcpdump for the capture as well seeing if you can find the difference. Just in case, it is the problem with wireshark.
------------
Banyan He
Blog: http://www.rootong.com
Email: banyan@xxxxxxxxxxx
On 2012-11-24 6:31 AM, John Powell wrote:
Hi Everyone,

I am running CentOS 6.3 on a HP 8200 using 3TB WD Green drives using a EXT4 file system.

I am using Wireshark 1.8.2 compiled from source.

I am using DUMPCAP to rotate and store historical Packet Captures.

Whether I capture the packets with Wireshark or view the DUMPCAP created file, I see dropouts in the packets being captured.

I tried to turning off journalling but this did not seem to help much:

umount /dev/mapper/VolGroup00-LogVol_Data

/sbin/tune2fs -o journal_data_writeback /dev/mapper/VolGroup00-LogVol_Data

/sbin/tune2fs -O ^has_journal /dev/mapper/VolGroup00-LogVol_Data

/sbin/e2fsck -f /dev/mapper/VolGroup00-LogVol_Data


I have a attached a couple of IOGraphs from Wireshark showing the packet drops.

Thanks alot!

-John