38be0d1383
This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12
85 lines
3.7 KiB
Plaintext
85 lines
3.7 KiB
Plaintext
This is a simple regression testing framework for Arthur.
|
|
|
|
There are three parts to it.
|
|
|
|
1) datagenerator . which is the main part. it's used to
|
|
run all the tests, create output png files,
|
|
measure rendering performance and output it in
|
|
data.xml file. datagenerator reads in
|
|
framework.ini to figure out where to look for tests.
|
|
if the framework.ini file isn't in the current directory
|
|
you have to specify its location with the -framework
|
|
parameter. by default datagenerator generates output in the
|
|
current directory - to specify output directory pass in
|
|
-output parameter followed by a directory name. if the given
|
|
directory doesn't exist it will be created.
|
|
|
|
it's recommended to always specify -iterations argument (followed
|
|
by the number). if the argument has been specified datagenerator
|
|
will try to render all testcases the given amount of times yielding
|
|
a lot more precise results.
|
|
|
|
if one wants to test just a specified engine -engine argument
|
|
should be specified (followed by the desired engine name).
|
|
|
|
individual testsuites can be run with -suite paramenter.
|
|
individual testcases from testcases can also be reran with
|
|
-testcase paramenter.
|
|
|
|
given files can also be tested individually. if the option -file
|
|
is specified with a SVG file engines will be tested against just
|
|
that file and output will be written directly to stdout (not files).
|
|
it's useful when optimizing one engine with particular case in
|
|
mind.
|
|
|
|
Example command line:
|
|
./bin/datagenerator -framework data/framework.ini -output sampleout -iterations 5 -engine [ENGINE]
|
|
|
|
2) htmlgenerator. which is used to generate html pages out of
|
|
generated output (generated by datagenerator). htmlgenerator also
|
|
reads in the framework.ini file. so the option -framework also
|
|
applies. but htmlgenerator takes a mandatory directory name as the
|
|
second argument. the mandatory directory name is the directory to
|
|
which you output the files generated by datagenerator (either
|
|
whatever followed the -output or the current directory)
|
|
|
|
htmlgenerator will generate html pages that can be viewed to
|
|
manually inspect the results.
|
|
|
|
Example command line:
|
|
./bin/htmlgenerator -framework data/framework.ini sampleout
|
|
|
|
3) performancediff. diffs the performance data between two output
|
|
directories. so assuming you already have data for all the engines,
|
|
you change something in an engine. to see the performance
|
|
difference run
|
|
./bin/datagenerator -framework data/framework.ini -output newoutput -iterations 5
|
|
and then
|
|
./bin/performancediff sampleout newoutput
|
|
|
|
The performance results will be printed out.
|
|
|
|
The two first columns are in ms. The last one is a
|
|
percentage of difference.
|
|
|
|
After the percentage a '+' or a '-' sign followed by a
|
|
number will be sometimes printed.
|
|
|
|
'+' signifies the new results were mathematically better. '-' is of
|
|
course just the opposite.
|
|
the number following '+' is the minimum rendering time the engine
|
|
spent rendering the given file up till this moment.
|
|
the number following '-' is the maximum rendering time the engine
|
|
spent rendering the given file up till this moment.
|
|
|
|
(if the current results are out of the scope from within maximum and
|
|
minimum number then the respective sign is printed out.)
|
|
-------------------------------------------------------------------
|
|
|
|
Note that the generated output directories can be copied from one
|
|
machine to the other. htmlgenerator will generate webpages for all
|
|
engines with valid data.xml files. So if you want add NativeWin32 and
|
|
NativeMac engines you'll have to have someone with a windows and mac
|
|
run datagenerator with "-engine NativeWin32" or "-engine NativeMac"
|
|
options and then sending you the respective output directories.
|