qt5base-lts/tests/arthur
Jason McDonald 66498850f2 Allow arthur tests to use private headers.
The tests include qapplication_p.h, therefore the .pro files need to
specify
    QT += core-private gui-private

Change-Id: Icd6d7ba31f79b8c7b15e0327dd5ce44c7f486221
Reviewed-by: Rohan McGovern
2011-05-20 12:04:07 +10:00
..
baselineserver Updated version references in autotests 2011-05-13 15:38:50 +03:00
common Don't use Qt3Support in arthur test suite. 2011-05-19 14:34:08 +10:00
data Don't use Qt3Support in arthur test suite. 2011-05-19 14:34:08 +10:00
datagenerator Allow arthur tests to use private headers. 2011-05-20 12:04:07 +10:00
htmlgenerator Allow arthur tests to use private headers. 2011-05-20 12:04:07 +10:00
lance Allow arthur tests to use private headers. 2011-05-20 12:04:07 +10:00
performancediff Allow arthur tests to use private headers. 2011-05-20 12:04:07 +10:00
shower Allow arthur tests to use private headers. 2011-05-20 12:04:07 +10:00
.gitattributes Initial import from the monolithic Qt. 2011-04-27 12:05:43 +02:00
.gitignore Initial import from the monolithic Qt. 2011-04-27 12:05:43 +02:00
arthurtester.pri Initial import from the monolithic Qt. 2011-04-27 12:05:43 +02:00
arthurtester.pro Initial import from the monolithic Qt. 2011-04-27 12:05:43 +02:00
README Initial import from the monolithic Qt. 2011-04-27 12:05:43 +02:00

This is a simple regression testing framework for Arthur.

There are three parts to it.

1) datagenerator . which is the main part. it's used to 
   run all the tests, create output png files,
   measure rendering performance and output it in
   data.xml file. datagenerator reads in
   framework.ini to figure out where to look for tests.
   if the framework.ini file isn't in the current directory
   you have to specify its location with the -framework
   parameter. by default datagenerator generates output in the
   current directory - to specify output directory pass in
   -output parameter followed by a directory name. if the given
   directory doesn't exist it will be created. 

   it's recommended to always specify -iterations argument (followed
   by the number). if the argument has been specified datagenerator
   will try to render all testcases the given amount of times yielding 
   a lot more precise results.

   if one wants to test just a specified engine -engine argument
   should be specified (followed by the desired engine name).

   individual testsuites can be run with -suite paramenter.
   individual testcases from testcases can also be reran with
   -testcase paramenter.

   given files can also be tested individually. if the option -file
   is specified with a SVG file engines will be tested against just
   that file and output will be written directly to stdout (not files).
   it's useful when optimizing one engine with particular case in
   mind.

   Example command line:
   ./bin/datagenerator -framework data/framework.ini -output sampleout -iterations 5 -engine [ENGINE]

2) htmlgenerator. which is used to generate html pages out of
   generated output (generated by datagenerator). htmlgenerator also
   reads in the framework.ini file. so the option -framework also
   applies. but htmlgenerator takes a mandatory directory name as the
   second argument. the mandatory directory name is the directory to
   which you output the files generated by datagenerator (either
   whatever followed the -output or the current directory)

   htmlgenerator will generate html pages that can be viewed to
   manually inspect the results.

   Example command line:
   ./bin/htmlgenerator -framework data/framework.ini sampleout

3) performancediff. diffs the performance data between two output
   directories. so assuming you already have data for all the engines,
   you change something in an engine. to see the performance
   difference run
   ./bin/datagenerator -framework data/framework.ini -output newoutput -iterations 5
   and then
   ./bin/performancediff sampleout newoutput
   
   The performance results will be printed out.

   The two first columns are in ms. The last one is a 
   percentage of difference. 

   After the percentage a '+' or a '-' sign followed by a 
   number will be sometimes printed.

   '+' signifies the new results were mathematically better. '-' is of
   course just the opposite.
   the number following '+' is the minimum rendering time the engine
   spent rendering the given file up till this moment.
   the number following '-' is the maximum rendering time the engine
   spent rendering the given file up till this moment.

   (if the current results are out of the scope from within maximum and
   minimum number then the respective sign is printed out.)
-------------------------------------------------------------------

Note that the generated output directories can be copied from one
machine to the other. htmlgenerator will generate webpages for all
engines with valid data.xml files. So if you want add NativeWin32 and
NativeMac engines you'll have to have someone with a windows and mac
run datagenerator with "-engine NativeWin32" or "-engine NativeMac"
options and then sending you the respective output directories.