On Tue, Dec 20, 2016 at 09:12:01AM +0100, Jirka Novak wrote:
> Hi Peter,
>
> first: happy-shark is very good idea. All notes above I would like to
> discuss and then I will try to add my samples with respect to conclusions...
>
> > For a basic testing framework, have a look at
> > https://github.com/wireshark/happy-shark
> >
> > To-do: populate the test/ directory with lots of specific samples :-)
>
> This approach looks good to me, I will try to add samples.
> What I'm missing there is idea how to add description for a sample - I
> mean "description" - why sample is there and "specification" - what
> wireshark should do with it.
> Testing with PDML is good idea, but it tests dissectors only. Time to
> time you want to store sample because GUI should do something with it
> (e.g. show it in red).
The GUI shows it in red because it has "expert info". This is also
visible in the PDML output and can be matched against.
As for Alexis' suggestion of the ExtensiveTesting program, that seems
more appropriate for core behavior (expansions of trees) and statistics
options. For dissectors I think it is overkill since we have tshark.
> 1) Won't be good idea to ask for e.g. <sample>.description.txt and
> <sample>.specification.txt for each sample?
Description is currently lacking I think (the folder name is a short
hint, but that is about it). The specification is basically the XSL +
post-processed PDML file.
> 2) Won't be good idea to allow skip a sample from automatic testing
> (because it is for GUI demonstration)?
You can invoke individual tests (which is most likely what you want when
you are testing changes to a single dissector). GUI versus tshark
single-pass and two-pass (-2) should produce the same results.
> > I think that happy-shark (which uses PDML output with a XSL
> > post-processor) does what you want.
>
> I tried to run make in top directory and it failed for first try :-) I
> checked the output and found that something changed in DNS dissector and
> dns.flags.recdesired .. value="1" was in past and now value="FFFFFFFF".
> It is probably about signed/unsigned value - not important now.
> But this issue pointed to me more things:
> 3) There should be easy way how to describe which wireshark version
> processed stored PDML file.
> Probably different outputs should be stored for main branches (2.0, 2.2,
> ...). It is obvious that each version of wireshark will change (improve)
> some outputs. Therefore older release will not be able to process it
> correctly (PDML diff will fail).
> I think that without output version you can't run regression tests
> automatically.
Branches can work, but given that there is only little testdata for now,
it seems better to track the master branch directly.
> 4) There should be easy way how to run custom wireshark version without
> modifying Makefile. I have multiple versions for testing. I think
> environment variable solve it.
Set environment varialbe TSHARK_EXECUTABLE to your custom build or
adjust $PATH to include the executables directory in the build
directory.
> 5) There should be easy way how to describe which filter should be used
> for preprocessing PDML. Now only filter.xsl is used.
XSL is currently the only available format because it is powerful given
the XML nature. Are there any particular things that are lacking in XSL?
(In theory it should be possible to add support for other filter types,
or create a specification that translates to XSL.)
> Maintaining such sample repository would be hard work. But I believe
> that when there will be rules how to add sample and which information
> must be added to it, it will be much easier.
>
> 6) Rules must be written.
> 6) There should be a procedure which check whether repository/new sample
> contains expected information. Something like git review procedure for
> wireshark source/gerrit.
For now you can file pull requests on the happy-shark project on Github.
It is still in experimental phase, nothing is set in stone and things
can change as you see them fit :-)
As you can see in the README, there are very basic guidelines, but
nothing very specific about how the process of developing the XSL works
(due to lack of experience).
--
Kind regards,
Peter Wu
https://lekensteyn.nl
