Wireshark · Ethereal-dev: Re: [Ethereal-dev] What to do, if Ethereal is running out of memory?

["Jim Young" <sysjhy@xxxxxxxxxxxxxxx>]

Hello Ulf,

The news that a call to g_malloc() can (and will) result in the 
program aborting if g_malloc() can NOT if fact allocate the 
requested amount of memory is extremely disappointing; 
but this news is not all together that surprising. :-(

Unless we write everything from scratch, we'll always be at 
the mercy of the coders of the third party APIs and libraries.   
Obviously it's a tradeoff between what's the desired thing to 
do versus what's the practical thing to do versus what's the 
responsible thing to do versus what's the actual thing one can 
do.  

I would propose that Ethereal could be augmented with some 
sort of memory watching instrumentation/enumeration code.
At one time I had such instrumentation code in an MS-DOS 
based application.  It kept up with core free (unallocated 
system memory), heap free, heap allocated etc.  I also had 
to be aware of the largest chunk(s) of usable memory I could 
allocate.  Just because the heap check routines reported that 
I had 100k of memory available did NOT mean that I could 
actually allocate a 100K chunk (or even a 5k chunk)!  
Sometimes an application's memory allocation pattern will 
result in significant memory fragmentation leaving one starved 
for usable memory it spite of the fact that the system believes 
their to be plenty of free memory available!
  
The proposed (and optional) Memory Stats window (similar to 
the Capture Stats window) would be available to keep track of 
how much memory is free, available and allocated in real time.  
A button on this proposed window (and an associated function 
for programmatic use) would allow one to enumerate (and log) 
the state of the memory subsystem.  One can crudely simulate 
some of this proposed functionality on a MS Window XP system 
by opening the Task Manager and watching Page File and 
System Memory usage on the "Performance" tab, or by finding 
the Ethereal instance on the Task Manager's "Processes" tab 
and watching its "Mem Usage" value.

<rant>
A very long time ago I was responsible for an MS-DOS based 
application that had to run 24/7 for months at a time.  In many
environments this application simply could take NO downtime.  
Running as an MS-DOS based application, I was keenly aware 
of memory leakage and memory allocation issues.  It was heavily 
dependant on third party APIs and libraries for the underlying 
communications layers.   Although I never had access to their 
source-code I became aware of just how these third party 
libraries behaved in a memory constrained environment.  Like the 
current g_malloc() situation, I determined that some of the third 
party API functions did NOT handle memory allocations failures 
gracefully.  Some libraries would result in the calling program 
to abruptly terminate (apparently like g_malloc()), but worse
still others libraries would simply fail to check for the NULL 
pointer and blissfully attempt to write into arbitrary memory!

Through experimentation I eventually determined what were 
the "brittle" functions and how much memory they generally 
needed to succeed.  I ended up writing wrappers for these brittle 
functions.  These wrappers would query the heap to determine 
if there was enough memory available for the third party function 
to succeed or not.  Obviously these wrappers were not as efficient 
as the third party function itself reporting back the allocation 
failure, but at least the wrapper kept my application in control of 
how to deal with the issue.  I could then attempt to handle the 
allocation problem more gracefully.

I'm still amazed at how much source code I find that doesn't 
even bother to check to see if malloc() returns a NULL pointer 
or not.  When asked, these programmers usually respond with 
some lame answer that they really don't have to worry about 
running out of memory because their application runs on a system 
with virtual memory (i.e. generally a Unix variant) which has much 
more swap space than real memory.  They generally don't seem 
at all concerned that swap space is also a finite resource.
</rant>

I hope you find some of this information useful. ;-)

Best regards,

Jim Young

>>> ulf.lamping@xxxxxx 10/10/04 04:22AM >>>
> One solution to this problem might be to have a trigger for something

> like a memory low water mark (how to create that event?). So if there

> are less than let's say 10MB free RAM (user preference), this could 
> create an event to stop capturing or loading data. I know that this 
> isn't a 100% solution, but a lot better than today IMHO.