Wireshark · Wireshark-dev: Re: [Wireshark-dev] Reordering capture files

[Martin Mathieson <martin.r.mathieson@xxxxxxxxxxxxxx>]

I have something working for me now.  If will probably be a couple of weeks before I have time to do much more with it, so posting it now in case anyone finds it useful or wants to improve it.

It has only been tested with IxCatapult log files so far, but hopefully it will work with any well-behaved file type supported by wiretap.

It is pretty inefficient (linear search for place to insert, inserting from wrong end, long queue length to suit the file I've been testing with, ....), but for now I don't mind waiting for a few seconds to get a nicely ordered log.

Regards,

Martin

On Mon, Sep 17, 2012 at 11:02 AM, Martin Mathieson <martin.r.mathieson@xxxxxxxxxxxxxx> wrote:

I think I will write a separate wiretap console app (tonight, if I get carried away).

My log files tend to be long (100s MB), so I don't think a script such as yours would work well for me.

Ultimately it would be good to have this functionality in (probably) editcap, and if there is demand, make it another option when exporting frames from wireshark.

Regards,

Martin

On Mon, Sep 17, 2012 at 3:15 PM, James Howard Young <jyoung@xxxxxxx> wrote:

Hello Martin,

I've used mergecap to do this.

It's a bit of a hacky solution but I used to split the
original out-or-order packets out to separate 1 packet
trace files.  I then used mergecap's default chronological
merging behavior to piece the sections back into a single
file.   The problem with this approach is that mergecap can
ONLY work with about 512 trace files at a time.   Depending
on how many out of order sections the trace file contains
this can get very tedious to do manually.

I eventually cooked up a shell script that helped automate
the process.  But the script was very slow, but it did
(eventually) get the work done.

The script's main loop would fix-up one negative delta
section at a time.  It used capinfos -o report to determine
if the trace file had any out-of-order packets.  If so it
ran a tshark -td report grepping for any negative delta
time records and piping it to head -1.  It would extract
the frame number of the 1st negative delta time packet
and use that particular frame number to split the trace
file into two parts. The first part would contain all the
frames up to the out-of-order frame.   The second part
would contain all frame from the out-of-order frame onward.
The two parts would be merged back together with mergecap.
Then the newly reconstituted trace file would be processed
again and again until the capinfos -o report indicated
that the was in strict chronological order.

Unfortunately I lost that particular script but it shouldn't
be too hard to re-create.

But I think augmenting editcap or mergecap or even adding
a new wiretap based console app to do this would be useful.
I've thought about this before but I simply have NOT had the
time to invest in following up.  I suspect the ability of
wireshark with pcapng to support multiple concurrent inputs
might make it more likely for to have trace files with
"out-of-order" packets.

I hope this helps,

Jim Y.


On 9/17/12 9:21 AM, "Martin Mathieson" <martin.r.mathieson@xxxxxxxxxxxxxx>
wrote:

>Hi,
>
>
>I work with log files that are created from multiple sources, and
>although the timestamps are in good synchronisation, they are often
>written to the log file slightly out of order (up to a few milliseconds).
>
>
>editcap -S supports rewriting the timestamps to be in strict
>chronological order (assuming the frame order will be correct), but this
>is the opposite of what I need.
>
>
>So, I'm considering adding a new flag to editcap, or maybe creating a
>separate console program using wiretap (I haven't thought through how
>difficult it might be to support other options at the same time).  It
>would work something like this:
>- specify a number N (say 200), that would serve as the number of frames
>in the ordering buffer
>- when a new frame is read from the input file, insert it into its
>correct time order within the (up-to) N frames in the buffer.  Hopefully
>not a linear search :)
>- write the earliest frame to the output file
>- when we reach the end of the input file, just flush the sorted buffer
>to the output file
>
>
>Not sure if I'd keep the buffered frames in memory, or go back and reread
>them when it was time to write them out.
>
>
>
>
>Am I missing an existing way to get this functionality?  Am I the only
>person who needs this functionality?
>
>
>Thanks,
>Martin
>

___________________________________________________________________________
Sent via:    Wireshark-dev mailing list <wireshark-dev@xxxxxxxxxxxxx>
Archives:    http://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
             mailto:wireshark-dev-request@xxxxxxxxxxxxx?subject=unsubscribe

/* Reorder the frames from an input dump file, and write to output dump file.
 *
 * $Id: reorder.c 42553 2012-05-10 16:10:44Z martinm $
 *
 * Wireshark - Network traffic analyzer
 * By Gerald Combs <gerald@xxxxxxxxxxxxx>
 * Copyright 1998 Gerald Combs
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License along
 * with this program; if not, write to the Free Software Foundation, Inc.,
 * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 *
 */


#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdarg.h>

#include "wtap.h"

/* #define REORDER_DEBUG */


/* Show command-line usage */
/* TODO: add reoder list length as an optional param? */
static void usage(void)
{
    printf("usage:  reorder <infile> <outfile>\n");
}

/* Remember where this frame was in the file */
typedef struct FrameRecord_t {
    gint64               offset;
    guint32              length;

    struct wtap_nstime   time;

    /* List item pointers */
    struct FrameRecord_t *prev;
    struct FrameRecord_t *next;
} FrameRecord_t;

/* This is pretty big, but I don't mind waiting a few seconds */
#define MAX_REORDER_LIST_LENGTH 3000
static int g_FrameRecordCount;
static FrameRecord_t *g_FrameListHead;

/* Is time1 earlier than time2? */
static gboolean isEarlierTime(struct wtap_nstime time1,
                              struct wtap_nstime time2)
{
    if (time1.secs < time2.secs) {
        return TRUE;
    }
    if (time1.secs == time2.secs) {
        return (time1.nsecs < time2.nsecs);
    }
    else {
        return FALSE;
    }
}

/* Is the reorder list empty? */
static gboolean ReorderListEmpty(void)
{
    return (g_FrameRecordCount == 0);
}

/* Is the reorder list full? */
static gboolean ReorderListFull(void)
{
    return (g_FrameRecordCount >= MAX_REORDER_LIST_LENGTH);
}

/* Add a new frame to the reorder list */
/* TODO: would presumably be much faster to add it from the end!!! */
static void ReorderListAdd(gint64 offset, guint32 length,
                           struct wtap_nstime time)
{
    FrameRecord_t *tmp;
    FrameRecord_t *newFrameRecord = g_malloc(sizeof(FrameRecord_t));

    /* Populate fields */
#ifdef REORDER_DEBUG
    printf("\nAdded with offset=%06llu, length=%05u, secs=%lu, nsecs=%d\n",
           offset, length, time.secs, time.nsecs);
#endif
    newFrameRecord->offset = offset;
    newFrameRecord->length = length;
    newFrameRecord->time = time;

    /* We will definitely add it below, so inc counter */
    g_FrameRecordCount++;

    /* First time, this will be the head */
    if (g_FrameListHead == NULL) {
#ifdef REORDER_DEBUG
        printf("this item will be head - only item\n");
#endif
        g_FrameListHead = newFrameRecord;
        newFrameRecord->prev = NULL;
        newFrameRecord->next = NULL;
        return;
    }

    /* Look for the place in the list where this item fits */
    tmp = g_FrameListHead;
    while (tmp != NULL) {
        if (isEarlierTime(time, tmp->time)) {
#ifdef REORDER_DEBUG
            printf("Time was less, writing before element\n");
#endif

            /* Insert newFrameRecord before tmp */

            /* Fix up previous item */
            if (tmp != g_FrameListHead) {
                newFrameRecord->prev = tmp->prev;
                newFrameRecord->prev->next = newFrameRecord;
            }
            else {
                g_FrameListHead = newFrameRecord;
            }

            /* Fix up next item */
            newFrameRecord->next = tmp;
            tmp->prev = newFrameRecord;

            return;
        }

        /* Didn't find an item to insert in front of */
        if (tmp->next == NULL) {
#ifdef REORDER_DEBUG
            printf("Reached the end of the list, so insert here\n");
#endif
            /* We are the new last item */
            tmp->next = newFrameRecord;
            newFrameRecord->prev = tmp;
            newFrameRecord->next = NULL;

            return;
        }
        else {
            /* Move onto the next item */
            tmp = tmp->next;
        }
    }
}

/* Dump the first item in the reorder list to the output file, and pop it */
static void ReorderListDumpFirst(wtap *wth, wtap_dumper *pdh)
{
    union wtap_pseudo_header pseudo_header;
    int    err;
    gchar  *errinfo;
    const struct wtap_pkthdr *phdr;
    guint8 buf[8000];
    struct wtap_pkthdr new_phdr;

    FrameRecord_t *prev_head = g_FrameListHead;

#ifdef REORDER_DEBUG
    printf("\nDumping frame (offset=%llu, length=%u) (%u items in list)\n", 
           g_FrameListHead->offset, g_FrameListHead->length,
           g_FrameRecordCount);
#endif

    /* Re-read the first frame from the stored location */
    wtap_seek_read(wth,
                   g_FrameListHead->offset,
                   &pseudo_header,
                   buf,
                   g_FrameListHead->length,
                   &err,
                   &errinfo);
#ifdef REORDER_DEBUG
    printf("re-read: err is %u, buf is (%s)\n", err, buf);
#endif

    /* Get packet header */
    phdr = wtap_phdr(wth);

#ifdef REORDER_DEBUG
    printf("re-read: secs are %lu, nsecs are %u\n", phdr->ts.secs, phdr->ts.nsecs);
#endif

    /* Copy, and set length. TODO: why does length need to be set? */
    memcpy((void*)&new_phdr, phdr, sizeof(struct wtap_pkthdr));
    new_phdr.len = g_FrameListHead->length;

    /* Dump frame to outfile */
    if (!wtap_dump(pdh, &new_phdr, &pseudo_header, buf, &err)) {
#ifdef REORDER_DEBUG
        printf("Error (%s) writing frame to outfile\n", wtap_strerror(err));
#endif
        exit(1);
    }


    /* Now remove the first item from the list */
    if (g_FrameListHead->next == NULL) {
        g_FrameListHead = NULL;
    }
    else {
        g_FrameListHead->next->prev = NULL;
        g_FrameListHead = g_FrameListHead->next;
    }

    /* And free the struct */
    g_free(prev_head);
    g_FrameRecordCount--;

#ifdef REORDER_DEBUG
    printf("Frame written, %u remaining\n", g_FrameRecordCount);
#endif
}


/********************************************************************/
/* Main function.                                                   */
/********************************************************************/
int main(int argc, char *argv[])
{
    wtap *wth = NULL;
    wtap_dumper *pdh = NULL;
    int err;
    gchar *err_info;
    gint64 data_offset;
    const struct wtap_pkthdr *phdr;
    guint32 read_count = 0;

    /* 1st arg is infile, 2nd arg is outfile */
    char *infile;
    char *outfile;
    if (argc == 3) {
        infile = argv[1];
        outfile = argv[2];
    }
    else {
        usage();
        exit(1);
    }

    /* Open infile */
    wth = wtap_open_offline(infile, &err, &err_info, TRUE);
    if (wth == NULL) {
        printf("reorder: Can't open %s: %s\n", infile, wtap_strerror(err));
        exit(1);
    }

#ifdef REORDER_DEBUG
    printf("file_type is %u\n", wtap_file_type(wth));
#endif

    /* Open outfile (same filetype/encap as input file) */
    pdh = wtap_dump_open(outfile, wtap_file_type(wth), wtap_file_encap(wth), 65535, FALSE, &err);
    if (pdh == NULL) {
        printf("Failed to open output file: (%s) - error %s\n", outfile, wtap_strerror(err));
        exit(1);
    }


    /* Read each frame from infile */
    while (wtap_read(wth, &err, &err_info, &data_offset)) {
        read_count++;
        phdr = wtap_phdr(wth);

        /* Add it to the reordering list */
        ReorderListAdd(data_offset, phdr->len, phdr->ts);

        /* If/when the list gets full, dump the first item out */
        if (ReorderListFull()) {
#ifdef REORDER_DEBUG
            printf("List is full, dumping first!\n");
#endif
            /* Write out the first one */
            ReorderListDumpFirst(wth, pdh);
        }
    }

    /* Flush out the remaining (ordered) frames */
    while (!ReorderListEmpty()) {
        ReorderListDumpFirst(wth, pdh);
    }

    /* Close outfile */
    if (!wtap_dump_close(pdh, &err)) {
        printf("reorder: Error closing %s: %s\n", outfile, wtap_strerror(err));
        exit(1);
    }

    /* TODO: write how many frames, and how many were out of order? */

    /* Finally, close infile */
    wtap_fdclose(wth);

    return 0;
}