85 lines
3.7 KiB
Plaintext
85 lines
3.7 KiB
Plaintext
|
This is a simple regression testing framework for Arthur.
|
||
|
|
|
||
|
|
There are three parts to it.
|
||
|
|
|
||
|
|
1) datagenerator . which is the main part. it's used to
|
||
|
|
run all the tests, create output png files,
|
||
|
|
measure rendering performance and output it in
|
||
|
|
data.xml file. datagenerator reads in
|
||
|
|
framework.ini to figure out where to look for tests.
|
||
|
|
if the framework.ini file isn't in the current directory
|
||
|
|
you have to specify its location with the -framework
|
||
|
|
parameter. by default datagenerator generates output in the
|
||
|
|
current directory - to specify output directory pass in
|
||
|
|
-output parameter followed by a directory name. if the given
|
||
|
|
directory doesn't exist it will be created.
|
||
|
|
|
||
|
|
it's recommended to always specify -iterations argument (followed
|
||
|
|
by the number). if the argument has been specified datagenerator
|
||
|
|
will try to render all testcases the given amount of times yielding
|
||
|
|
a lot more precise results.
|
||
|
|
|
||
|
|
if one wants to test just a specified engine -engine argument
|
||
|
|
should be specified (followed by the desired engine name).
|
||
|
|
|
||
|
|
individual testsuites can be run with -suite paramenter.
|
||
|
|
individual testcases from testcases can also be reran with
|
||
|
|
-testcase paramenter.
|
||
|
|
|
||
|
|
given files can also be tested individually. if the option -file
|
||
|
|
is specified with a SVG file engines will be tested against just
|
||
|
|
that file and output will be written directly to stdout (not files).
|
||
|
|
it's useful when optimizing one engine with particular case in
|
||
|
|
mind.
|
||
|
|
|
||
|
|
Example command line:
|
||
|
|
./bin/datagenerator -framework data/framework.ini -output sampleout -iterations 5 -engine [ENGINE]
|
||
|
|
|
||
|
|
2) htmlgenerator. which is used to generate html pages out of
|
||
|
|
generated output (generated by datagenerator). htmlgenerator also
|
||
|
|
reads in the framework.ini file. so the option -framework also
|
||
|
|
applies. but htmlgenerator takes a mandatory directory name as the
|
||
|
|
second argument. the mandatory directory name is the directory to
|
||
|
|
which you output the files generated by datagenerator (either
|
||
|
|
whatever followed the -output or the current directory)
|
||
|
|
|
||
|
|
htmlgenerator will generate html pages that can be viewed to
|
||
|
|
manually inspect the results.
|
||
|
|
|
||
|
|
Example command line:
|
||
|
|
./bin/htmlgenerator -framework data/framework.ini sampleout
|
||
|
|
|
||
|
|
3) performancediff. diffs the performance data between two output
|
||
|
|
directories. so assuming you already have data for all the engines,
|
||
|
|
you change something in an engine. to see the performance
|
||
|
|
difference run
|
||
|
|
./bin/datagenerator -framework data/framework.ini -output newoutput -iterations 5
|
||
|
|
and then
|
||
|
|
./bin/performancediff sampleout newoutput
|
||
|
|
|
||
|
|
The performance results will be printed out.
|
||
|
|
|
||
|
|
The two first columns are in ms. The last one is a
|
||
|
|
percentage of difference.
|
||
|
|
|
||
|
|
After the percentage a '+' or a '-' sign followed by a
|
||
|
|
number will be sometimes printed.
|
||
|
|
|
||
|
|
'+' signifies the new results were mathematically better. '-' is of
|
||
|
|
course just the opposite.
|
||
|
|
the number following '+' is the minimum rendering time the engine
|
||
|
|
spent rendering the given file up till this moment.
|
||
|
|
the number following '-' is the maximum rendering time the engine
|
||
|
|
spent rendering the given file up till this moment.
|
||
|
|
|
||
|
|
(if the current results are out of the scope from within maximum and
|
||
|
|
minimum number then the respective sign is printed out.)
|
||
|
|
-------------------------------------------------------------------
|
||
|
|
|
||
|
|
Note that the generated output directories can be copied from one
|
||
|
|
machine to the other. htmlgenerator will generate webpages for all
|
||
|
|
engines with valid data.xml files. So if you want add NativeWin32 and
|
||
|
|
NativeMac engines you'll have to have someone with a windows and mac
|
||
|
|
run datagenerator with "-engine NativeWin32" or "-engine NativeMac"
|
||
|
|
options and then sending you the respective output directories.
|