Wireshark · Ethereal-dev: Re: [Ethereal-dev] some questiones about checksum and memery

[Guy Harris <guy@xxxxxxxxxx>]

On Tue, Jun 04, 2002 at 10:32:52AM +0800, Buding Chen wrote:
>     1) I want to debug ethereal smoothly, then how to remove "-O2" argument
> of gcc in all Makefile files including all subdirectory? Are there some
> convenient ways such as add some argument when excuting ./configure?

There might be a convenient way, but I don't know of one offhand; I just
remove the "-O2" by hand if the optimization gets in the way of
debugging.

>     2) If a IP packet based TCP fails in TCP checksum in function
> dissect_tcp(...), why is it allowed to pass to subdissector without dropping
> it? So does IP header checksum. I think this packet is damaged and not worth
> dissecting.

The packet is not necessarily damaged:

	1) the checksum might have been computed incorrectly (unlikely,
	   but it *has* happened during development);

	2) the packet might have been a packet transmitted by the
	   machine on which Ethereal was running, and it might have been
	   transmitted on an interface with TCP checksum offloading, in
	   which case the copy of the packet handed to the OS packet
	   capture mechanism, and thus to libpcap and then Ethereal,
	   will not necessarily have a valid TCP checksum (this has
	   *definitely* happened with Sun gigabit Ethernet adapters, and
	   is the reason why there's an option in the TCP dissector not
	   to check the checksum).

If we don't pass on the payload of packets with bad checksums, the TCP
checksum checking option, if turned off, should allow those packets to
be passed to the dissector.  I'd suggest having similar options for
other checksums as well.

>     3) Ethereal eats a lot of memery when and after loading a big file. I
> find the size of memery is bigger than file's. I search the source code and
> find some useless memery is freeed in function wtap_sequential_close(...)
> after loading the file.

The memory freed in "wtap_sequential_close()" is *NOT* useless; if it
were useless, we shouldn't have bothered allocating it in the first
place!

The memory freed by "wtap_sequential_close()" itself is a buffer into
which packet data is read when the file is initially read in; that
buffer will not be much larger than the largest link-layer frame, so
it's not going to be bigger than 128KB or so.

If the particular capture file type being read has a "sequential_close"
routine, that routine is also called.  The only capture file type that
has such a routine is the DOS-based Sniffer capture file type; the
memory that frees is a buffer used to decompress compressed sniffer
files - without that buffer, you can't read compressed files....  It's
64KB.

So "wtap_sequential_close()" isn't the place to look if you want to
significantly reduce Ethereal's memory consumption; the stuff it frees
doesn't consume very much memory.

> Who eats these large memery?

A lot of the memory used by Ethereal is used by the GTK+ GUI widget for
displaying the packet list.  For every packet being displayed, a string
is allocated for every single column being displayed, containing the
text of that column.

We also allocate a data structure per frame - a "frame_data" structure,
as declared in "epan/frame_data.h", which is about 64 bytes long on
machines with 32-bit "long"s and pointers, so that's about 64 bytes per
frame.

If the capture file is full of very small frames - the minimum-sized
Ethernet frame of 60 bytes, for example - the amount of memory required
by those data structures could be about the same amount of data as are
in the file.

If you have larger packets, however, it's still 64 bytes, so the amount
of data wouldn't be as close to the amount of data in the file.

*Some* frames might have per-protocol data associated with them by the
dissector in order to make it possible to dissect packets in random
order when the user clicks on them.  How much data that requires depends
on the capture file.

If reassembly of frames into larger packets is done, we currently
allocate memory for the reassembled frame data, so that might consume
more memory.

> And are there some ways to reduce the size of the memery?

Well, replacing the GUI widget for the packet list with a widget that
doesn't store the contents of all the columns, but instead calls back to
a routine to get the contents of the columns, would help.  There would
be no need to store text for the frame number, the time stamps, and
possibly the addresses (although the addresses themselves would have to
be associated with the frame - however, we could have only one copy in
memory of every address that shows up in a column).  I have worked on
some code to implement that, but it's not done yet, and I haven't had
time to work on it very much recently.

Unfortunately, as random access to frames in gzipped capture files is
*very* expensive, it would not be practical to have the routine that
returns the columns just dissect the frame and generate the Protocol and
Info columns.  We could, however, save pointers to those columns with
the frame, *and*, if the columns are constant strings (as most Protocol
columns are), have the pointer just point to the constant string, and
store flags indicating whether the Protocol and Info column values are
dynamically-allocated or not.

It might be possible to shrink the "frame_data" structure, e.g. by
keeping only the absolute time stamps, and computing the delta and
relative time stamps.

It *might* also be possible to get rid of the "data_src" field; I'll
have to look at that.  The "num" field is just the ordinal number of the
"frame_data" structure in the list of frames, so it could perhaps also
be eliminated.

If random access to frames in gzipped captures weren't expensive, or if
we didn't support reading gzipped captures, we could perhaps not store
the time stamps at all, and just read them from the capture file,
although that's a bit of a pain for the delta from the previous packet. 

In addition, that would also mean that we might not have to store
reassembled data for frames - we could just store information about
which parts of which frames need to be reassembled, and read the data
for those frames and reassemble on demand.

Unfortunately, making random access to gzipped captures inexpensive is
not easy - gzip format, unlike compressed Sniffer format, is not
designed for efficient random access, as the "history" the decompressor
needs when moving sequentially through the capture could include
information that goes all the way back to the beginning of the capture
file.