Wireshark-dev: [Wireshark-dev] PCAP indexing & compression specifications
From: Thomas Baudelet <thomas.baudelet@xxxxxxxxx>
Date: Wed, 10 May 2017 14:52:46 +0200
This is a copy from the bug, on the advice of Graham
https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=13694

Today most professional capture files can't be opened directly in Wireshark due to their size and in front of a 10GB pcap file, the open source user has:
    * sequential packet readers (tcpdump/dumpcap) + capture filters
    * custom index tools

At the opposite, network softwares are building indexes that are pretty efficient (Riverbed Packet Analyzer index and micro-index, Extrahop, etc ...)

Several open source projects exist, but afaik none are really linked/approved by Wireshark community yet:

#####
ntop n2disk & pcapIndex:
http://luca.ntop.org/pcapIndex.pdf
Luca seemed interested to share: https://twitter.com/lucaderi/status/839490394924670976

cppip:
https://blogs.cisco.com/security/tools-of-the-trade-the-compressed-pcap-packet-indexing-program

sancp:
http://blog.vorant.com/2008/04/pcap-indexing.html

moloch:
https://github.com/aol/moloch

...
######

Discussion is wide and not straightforward: indexing will be a trade off between size of index / speed / evolutivity. The idea is to throw ideas / experience / suggestions and find some bases for some specifications/code integration/development to start and maybe integrate Wireshark one day.
(File -> Index PCAP !)