Wireshark · Wireshark-dev: Re: [Wireshark-dev] New packet list - out of memory?

["Anders Broman" <anders.broman@xxxxxxxxxxxx>]

-----Original Message-----
From: wireshark-dev-bounces@xxxxxxxxxxxxx
[mailto:wireshark-dev-bounces@xxxxxxxxxxxxx] On Behalf Of Bill Meier
Sent: den 6 oktober 2009 00:49
To: Developer support list for Wireshark
Subject: Re: [Wireshark-dev] New packet list - out of memory?

Bill Meier wrote:
>> A data point: A few days ago while working on bug #2375 I noticed
that 
>> on Windows that if I defined the env logicals to not use se_alloc & 
>> etc that *much* less memory was required to load a capture. (20% of 
>> that used with se_alloc & etc ? I don't quite remember).
>> 
>
>Details:
>
>Env: Windows Vista
>
>Actions:
>
>1. Load Wireshark
>    Determine "Private Working Set", "Working Set", "Commit Size"
>
>2. Load a 40 Meg capture file.
>    Determine ....  again
>
>The above was done for:
>
>a. Wireshark with new_packet_list:
>
>b. Wireshark w/new_packet_list:
>     set WIRESHARK_DEBUG_EP_NO_CHUNKS=T
>     set WIRESHARK_DEBUG_SE_NO_CHUNKS=T
>
>c. Wireshark without new packet list
>
>d. Wireshark without new packet list
>     set WIRESHARK_DEBUG_EP_NO_CHUNKS=T
>     set WIRESHARK_DEBUG_SE_NO_CHUNKS=T
>
>Measurements
>
>Note: The Working Set was always about 20 Meg larger than the
>    Private Working Set(PWS) so only the PWS is shown.
>
>(All values in Megs)
>                                        EP/SE Chunks        No Chunks
>                                        PWS  Commit        PWS  Commit
>                                        ---- -------     ----- -------
>Wireshark: current_packet_list
>      After startup:                     42      62         42     52
>      After 40M CF load:                180     399        195    205
>
>
>Wireshark: new_packet_list:
>      After startup:                     42      62         42     52
>      After 40M CF load:                136     564        165    177
>
>
>A conclusion:
>
>Commit Size (which I believe is the amount of VM address space used
>    by the process and which is presumably the limiting factor for
>    memory usage).
>       - Best for new_packet_list/no_chunks;
>       - worst (quite) for new_packet_list/chunks;
>
>Comments welcome...
>
>(I did these tests somewhat quickly so I could have missed
something...)
>
>Bill

Hi,
I expanded on Bills measurements using the file that caused out of
memory for the new packet list. (Thanks Bill I didn't know about the
commit size option in the task manager).
 
146 MB 661417 pkts (TCP reassembly)

Version 1.2.2 (SVN Rev 29910)
PWS	  Commit		
 42 016	     61 876	
704 656	  1 639 256	         
Loading time ~57s
 
Old packet list SVN 30353
PWS	  Commit		
 43 060	     63 384	
492 456	  1 427 480	         
Loading time 47.752 - 48.485 (2 Loading time measures)

Old packet list SVN 30353
set WIRESHARK_DEBUG_SE_NO_CHUNKS=T
PWS	  Commit		
 43 060	     63 384	
559 744	    579 968	         
Loading time 48.797

Old packet list SVN 30353
set WIRESHARK_DEBUG_EP_NO_CHUNKS=T
set WIRESHARK_DEBUG_SE_NO_CHUNKS=T
 43 208	     53 312
559 612	    569 632
Loading time 55.162

Old packet list SVN 30353
#define EMEM_ALLOCS_PER_CHUNK (EMEM_PACKET_CHUNK_SIZE / 64)
PWS	  Commit		
 43 288	     64 188	
492 104	    567 060	         
Loading time 47.736 - 48.391 (2 Loading time measures)

 

(This file can't be loaded using 512)
New packet list SVN 30353
#define EMEM_ALLOCS_PER_CHUNK (EMEM_PACKET_CHUNK_SIZE / 64)
PWS	  Commit		
 43 492	     64 512	
332 400	    471 264	         
Loading time 55.895 - 57.018 (4 Loading time measures)

New packet list SVN 30353
#define EMEM_ALLOCS_PER_CHUNK (EMEM_PACKET_CHUNK_SIZE / 64)
set WIRESHARK_DEBUG_SE_NO_CHUNKS=T
PWS	  Commit		
 43 660	     64 736	
468 496	    493 660	         
Loading time 53.666

New packet list SVN 30353
#define EMEM_ALLOCS_PER_CHUNK (EMEM_PACKET_CHUNK_SIZE / 64)
set WIRESHARK_DEBUG_EP_NO_CHUNKS=T
set WIRESHARK_DEBUG_SE_NO_CHUNKS=T
PWS	  Commit		
 43 660	     64 736	
468 428	    481 516	         
Loading time 1:01.215 - 1:02.682 (2 Loading time measures) 

  Compared with 1.2.2 memory usage has improved quite a bit. Loading
time
  has improved with the old packet list but not so much with the new
packet list!?

Conclusions:
- se alloced memory is a problem as the allocation "waists" memory.
  This is not a big problem for ep memory as it's given back to the pool
after each packet.
- Using the old packet list the file loads faster !?
  This was not the case some way along the line...
- Surprisingly enough the file loads faster not using chunks for
se_alloc ???
  Whereas it is faster using ep memory.

Increasing EMEM_ALLOCS_PER_CHUNK looks like a good thing - drawbacks?
Should the use of se_alloc be discouraged/removed?

Regards
Anders

________________________________________________________________________
___
Sent via:    Wireshark-dev mailing list <wireshark-dev@xxxxxxxxxxxxx>
Archives:    http://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
 
mailto:wireshark-dev-request@xxxxxxxxxxxxx?subject=unsubscribe