516 lines
20 KiB
HTML
516 lines
20 KiB
HTML
|
<html>
|
||
|
|
<head>
|
||
|
|
<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||
|
|
<title>SimpleTest for PHP test runner and display documentation</title>
|
||
|
|
<link rel="stylesheet" type="text/css" href="docs.css" title="Styles">
|
||
|
|
</head>
|
||
|
|
<body>
|
||
|
|
<div class="menu_back">
|
||
|
|
<div class="menu">
|
||
|
|
<h2>
|
||
|
|
<a href="index.html">SimpleTest</a>
|
||
|
|
</h2>
|
||
|
|
<ul>
|
||
|
|
<li>
|
||
|
|
<a href="overview.html">Overview</a>
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<a href="unit_test_documentation.html">Unit tester</a>
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<a href="group_test_documentation.html">Group tests</a>
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<a href="server_stubs_documentation.html">Server stubs</a>
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<a href="mock_objects_documentation.html">Mock objects</a>
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<a href="partial_mocks_documentation.html">Partial mocks</a>
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<span class="chosen">Reporting</span>
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<a href="expectation_documentation.html">Expectations</a>
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<a href="web_tester_documentation.html">Web tester</a>
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<a href="form_testing_documentation.html">Testing forms</a>
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<a href="authentication_documentation.html">Authentication</a>
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<a href="browser_documentation.html">Scriptable browser</a>
|
||
|
|
</li>
|
||
|
|
</ul>
|
||
|
|
</div>
|
||
|
|
</div>
|
||
|
|
<h1>Test reporter documentation</h1>
|
||
|
|
<div class="content">
|
||
|
|
|
||
|
|
<p>
|
||
|
|
SimpleTest pretty much follows the MVC pattern
|
||
|
|
(Model-View-Controller).
|
||
|
|
The reporter classes are the view and the model is your
|
||
|
|
test cases and their hiearchy.
|
||
|
|
The controller is mostly hidden from the user of
|
||
|
|
SimpleTest unless you want to change how the test cases
|
||
|
|
are actually run, in which case it is possible to
|
||
|
|
override the runner objects from within the test case.
|
||
|
|
As usual with MVC, the controller is mostly undefined
|
||
|
|
and there are other places to control the test run.
|
||
|
|
</p>
|
||
|
|
|
||
|
|
<p>
|
||
|
|
<a class="target" name="html">
|
||
|
|
<h2>Reporting results in HTML</h2>
|
||
|
|
</a>
|
||
|
|
</p>
|
||
|
|
<p>
|
||
|
|
The default test display is minimal in the extreme.
|
||
|
|
It reports success and failure with the conventional red and
|
||
|
|
green bars and shows a breadcrumb trail of test groups
|
||
|
|
for every failed assertion.
|
||
|
|
Here's a fail...
|
||
|
|
<div class="demo">
|
||
|
|
<h1>File test</h1>
|
||
|
|
<span class="fail">Fail</span>: createnewfile->True assertion failed.<br>
|
||
|
|
<div style="padding: 8px; margin-top: 1em; background-color: red; color: white;">1/1 test cases complete.
|
||
|
|
<strong>0</strong> passes, <strong>1</strong> fails and <strong>0</strong> exceptions.</div>
|
||
|
|
</div>
|
||
|
|
And here all tests passed...
|
||
|
|
<div class="demo">
|
||
|
|
<h1>File test</h1>
|
||
|
|
<div style="padding: 8px; margin-top: 1em; background-color: green; color: white;">1/1 test cases complete.
|
||
|
|
<strong>1</strong> passes, <strong>0</strong> fails and <strong>0</strong> exceptions.</div>
|
||
|
|
</div>
|
||
|
|
The good news is that there are several points in the display
|
||
|
|
hiearchy for subclassing.
|
||
|
|
</p>
|
||
|
|
<p>
|
||
|
|
For web page based displays there is the
|
||
|
|
<span class="new_code">HtmlReporter</span> class with the following
|
||
|
|
signature...
|
||
|
|
<pre>
|
||
|
|
class HtmlReporter extends SimpleReporter {
|
||
|
|
public HtmlReporter($encoding) { ... }
|
||
|
|
public makeDry(boolean $is_dry) { ... }
|
||
|
|
public void paintHeader(string $test_name) { ... }
|
||
|
|
public void sendNoCacheHeaders() { ... }
|
||
|
|
public void paintFooter(string $test_name) { ... }
|
||
|
|
public void paintGroupStart(string $test_name, integer $size) { ... }
|
||
|
|
public void paintGroupEnd(string $test_name) { ... }
|
||
|
|
public void paintCaseStart(string $test_name) { ... }
|
||
|
|
public void paintCaseEnd(string $test_name) { ... }
|
||
|
|
public void paintMethodStart(string $test_name) { ... }
|
||
|
|
public void paintMethodEnd(string $test_name) { ... }
|
||
|
|
public void paintFail(string $message) { ... }
|
||
|
|
public void paintPass(string $message) { ... }
|
||
|
|
public void paintError(string $message) { ... }
|
||
|
|
public void paintException(string $message) { ... }
|
||
|
|
public void paintMessage(string $message) { ... }
|
||
|
|
public void paintFormattedMessage(string $message) { ... }
|
||
|
|
protected string _getCss() { ... }
|
||
|
|
public array getTestList() { ... }
|
||
|
|
public integer getPassCount() { ... }
|
||
|
|
public integer getFailCount() { ... }
|
||
|
|
public integer getExceptionCount() { ... }
|
||
|
|
public integer getTestCaseCount() { ... }
|
||
|
|
public integer getTestCaseProgress() { ... }
|
||
|
|
}
|
||
|
|
</pre>
|
||
|
|
Here is what some of these methods mean. First the display methods
|
||
|
|
that you will probably want to override...
|
||
|
|
<ul class="api">
|
||
|
|
<li>
|
||
|
|
<span class="new_code">HtmlReporter(string $encoding)</span>
|
||
|
|
<br>
|
||
|
|
is the constructor.
|
||
|
|
Note that the unit test sets up the link to the display
|
||
|
|
rather than the other way around.
|
||
|
|
The display is a mostly passive receiver of test events.
|
||
|
|
This allows easy adaption of the display for other test
|
||
|
|
systems beside unit tests, such as monitoring servers.
|
||
|
|
The encoding is the character encoding you wish to
|
||
|
|
display the test output in.
|
||
|
|
In order to correctly render debug output when
|
||
|
|
using the web tester, this should match the encoding
|
||
|
|
of the site you are trying to test.
|
||
|
|
The available character set strings are described in
|
||
|
|
the PHP <a href="http://www.php.net/manual/en/function.htmlentities.php">html_entities()</a>
|
||
|
|
function.
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<span class="new_code">void paintHeader(string $test_name)</span>
|
||
|
|
<br>
|
||
|
|
is called once at the very start of the test when the first
|
||
|
|
start event arrives.
|
||
|
|
The first start event is usually delivered by the top level group
|
||
|
|
test and so this is where <span class="new_code">$test_name</span>
|
||
|
|
comes from.
|
||
|
|
It paints the page titles, CSS, body tag, etc.
|
||
|
|
It returns nothing (<span class="new_code">void</span>).
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<span class="new_code">void paintFooter(string $test_name)</span>
|
||
|
|
<br>
|
||
|
|
Called at the very end of the test to close any tags opened
|
||
|
|
by the page header.
|
||
|
|
By default it also displays the red/green bar and the final
|
||
|
|
count of results.
|
||
|
|
Actually the end of the test happens when a test end event
|
||
|
|
comes in with the same name as the one that started it all
|
||
|
|
at the same level.
|
||
|
|
The tests nest you see.
|
||
|
|
Closing the last test finishes the display.
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<span class="new_code">void paintMethodStart(string $test_name)</span>
|
||
|
|
<br>
|
||
|
|
is called at the start of each test method.
|
||
|
|
The name normally comes from method name.
|
||
|
|
The other test start events behave the same way except
|
||
|
|
that the group test one tells the reporter how large
|
||
|
|
it is in number of held test cases.
|
||
|
|
This is so that the reporter can display a progress bar
|
||
|
|
as the runner churns through the test cases.
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<span class="new_code">void paintMethodEnd(string $test_name)</span>
|
||
|
|
<br>
|
||
|
|
backs out of the test started with the same name.
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<span class="new_code">void paintFail(string $message)</span>
|
||
|
|
<br>
|
||
|
|
paints a failure.
|
||
|
|
By default it just displays the word fail, a breadcrumbs trail
|
||
|
|
showing the current test nesting and the message issued by
|
||
|
|
the assertion.
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<span class="new_code">void paintPass(string $message)</span>
|
||
|
|
<br>
|
||
|
|
by default does nothing.
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<span class="new_code">string _getCss()</span>
|
||
|
|
<br>
|
||
|
|
Returns the CSS styles as a string for the page header
|
||
|
|
method.
|
||
|
|
Additional styles have to be appended here if you are
|
||
|
|
not overriding the page header.
|
||
|
|
You will want to use this method in an overriden page header
|
||
|
|
if you want to include the original CSS.
|
||
|
|
</li>
|
||
|
|
</ul>
|
||
|
|
There are also some accessors to get information on the current
|
||
|
|
state of the test suite.
|
||
|
|
Use these to enrich the display...
|
||
|
|
<ul class="api">
|
||
|
|
<li>
|
||
|
|
<span class="new_code">array getTestList()</span>
|
||
|
|
<br>
|
||
|
|
is the first convenience method for subclasses.
|
||
|
|
Lists the current nesting of the tests as a list
|
||
|
|
of test names.
|
||
|
|
The first, most deeply nested test, is first in the
|
||
|
|
list and the current test method will be last.
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<span class="new_code">integer getPassCount()</span>
|
||
|
|
<br>
|
||
|
|
returns the number of passes chalked up so far.
|
||
|
|
Needed for the display at the end.
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<span class="new_code">integer getFailCount()</span>
|
||
|
|
<br>
|
||
|
|
is likewise the number of fails so far.
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<span class="new_code">integer getExceptionCount()</span>
|
||
|
|
<br>
|
||
|
|
is likewise the number of errors so far.
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<span class="new_code">integer getTestCaseCount()</span>
|
||
|
|
<br>
|
||
|
|
is the total number of test cases in the test run.
|
||
|
|
This includes the grouping tests themselves.
|
||
|
|
</li>
|
||
|
|
<li>
|
||
|
|
<span class="new_code">integer getTestCaseProgress()</span>
|
||
|
|
<br>
|
||
|
|
is the number of test cases completed so far.
|
||
|
|
</li>
|
||
|
|
</ul>
|
||
|
|
One simple modification is to get the HtmlReporter to display
|
||
|
|
the passes as well as the failures and errors...
|
||
|
|
<pre>
|
||
|
|
<strong>class ShowPasses extends HtmlReporter {
|
||
|
|
|
||
|
|
function paintPass($message) {
|
||
|
|
parent::paintPass($message);
|
||
|
|
print "&<span class=\"pass\">Pass</span>: ";
|
||
|
|
$breadcrumb = $this->getTestList();
|
||
|
|
array_shift($breadcrumb);
|
||
|
|
print implode("-&gt;", $breadcrumb);
|
||
|
|
print "-&gt;$message<br />\n";
|
||
|
|
}
|
||
|
|
|
||
|
|
function _getCss() {
|
||
|
|
return parent::_getCss() . ' .pass { color: green; }';
|
||
|
|
}
|
||
|
|
}</strong>
|
||
|
|
</pre>
|
||
|
|
</p>
|
||
|
|
<p>
|
||
|
|
One method that was glossed over was the <span class="new_code">makeDry()</span>
|
||
|
|
method.
|
||
|
|
If you run this method, with no parameters, on the reporter
|
||
|
|
before the test suite is run no actual test methods
|
||
|
|
will be called.
|
||
|
|
You will still get the events of entering and leaving the
|
||
|
|
test methods and test cases, but no passes or failures etc,
|
||
|
|
because the test code will not actually be executed.
|
||
|
|
</p>
|
||
|
|
<p>
|
||
|
|
The reason for this is to allow for more sophistcated
|
||
|
|
GUI displays that allow the selection of individual test
|
||
|
|
cases.
|
||
|
|
In order to build a list of possible tests they need a
|
||
|
|
report on the test structure for drawing, say a tree view
|
||
|
|
of the test suite.
|
||
|
|
With a reporter set to dry run that just sends drawing events
|
||
|
|
this is easily accomplished.
|
||
|
|
</p>
|
||
|
|
|
||
|
|
<p>
|
||
|
|
<a class="target" name="other">
|
||
|
|
<h2>Extending the reporter</h2>
|
||
|
|
</a>
|
||
|
|
</p>
|
||
|
|
<p>
|
||
|
|
Rather than simply modifying the existing display, you might want to
|
||
|
|
produce a whole new HTML look, or even generate text or XML.
|
||
|
|
Rather than override every method in
|
||
|
|
<span class="new_code">HtmlReporter</span> we can take one
|
||
|
|
step up the class hiearchy to <span class="new_code">SimpleReporter</span>
|
||
|
|
in the <em>simple_test.php</em> source file.
|
||
|
|
</p>
|
||
|
|
<p>
|
||
|
|
A do nothing display, a blank canvas for your own creation, would
|
||
|
|
be...
|
||
|
|
<pre>
|
||
|
|
<strong>require_once('simpletest/simple_test.php');</strong>
|
||
|
|
|
||
|
|
class MyDisplay extends SimpleReporter {<strong>
|
||
|
|
</strong>
|
||
|
|
function paintHeader($test_name) {
|
||
|
|
}
|
||
|
|
|
||
|
|
function paintFooter($test_name) {
|
||
|
|
}
|
||
|
|
|
||
|
|
function paintStart($test_name, $size) {<strong>
|
||
|
|
parent::paintStart($test_name, $size);</strong>
|
||
|
|
}
|
||
|
|
|
||
|
|
function paintEnd($test_name, $size) {<strong>
|
||
|
|
parent::paintEnd($test_name, $size);</strong>
|
||
|
|
}
|
||
|
|
|
||
|
|
function paintPass($message) {<strong>
|
||
|
|
parent::paintPass($message);</strong>
|
||
|
|
}
|
||
|
|
|
||
|
|
function paintFail($message) {<strong>
|
||
|
|
parent::paintFail($message);</strong>
|
||
|
|
}
|
||
|
|
}
|
||
|
|
</pre>
|
||
|
|
No output would come from this class until you add it.
|
||
|
|
</p>
|
||
|
|
|
||
|
|
<p>
|
||
|
|
<a class="target" name="cli">
|
||
|
|
<h2>The command line reporter</h2>
|
||
|
|
</a>
|
||
|
|
</p>
|
||
|
|
<p>
|
||
|
|
SimpleTest also ships with a minimal command line reporter.
|
||
|
|
The interface mimics JUnit to some extent, but paints the
|
||
|
|
failure messages as they arrive.
|
||
|
|
To use the command line reporter simply substitute it
|
||
|
|
for the HTML version...
|
||
|
|
<pre>
|
||
|
|
<?php
|
||
|
|
require_once('simpletest/unit_tester.php');
|
||
|
|
require_once('simpletest/reporter.php');
|
||
|
|
|
||
|
|
$test = &new GroupTest('File test');
|
||
|
|
$test->addTestFile('tests/file_test.php');
|
||
|
|
$test->run(<strong>new TextReporter()</strong>);
|
||
|
|
?>
|
||
|
|
</pre>
|
||
|
|
Then invoke the test suite from the command line...
|
||
|
|
<pre class="shell">
|
||
|
|
php file_test.php
|
||
|
|
</pre>
|
||
|
|
You will need the command line version of PHP installed
|
||
|
|
of course.
|
||
|
|
A passing test suite looks like this...
|
||
|
|
<pre class="shell">
|
||
|
|
File test
|
||
|
|
OK
|
||
|
|
Test cases run: 1/1, Failures: 0, Exceptions: 0
|
||
|
|
</pre>
|
||
|
|
A failure triggers a display like this...
|
||
|
|
<pre class="shell">
|
||
|
|
File test
|
||
|
|
1) True assertion failed.
|
||
|
|
in createnewfile
|
||
|
|
FAILURES!!!
|
||
|
|
Test cases run: 1/1, Failures: 1, Exceptions: 0
|
||
|
|
</pre>
|
||
|
|
</p>
|
||
|
|
<p>
|
||
|
|
One of the main reasons for using a command line driven
|
||
|
|
test suite is of using the tester as part of some automated
|
||
|
|
process.
|
||
|
|
To function properly in shell scripts the test script should
|
||
|
|
return a non-zero exit code on failure.
|
||
|
|
If a test suite fails the value <span class="new_code">false</span>
|
||
|
|
is returned from the <span class="new_code">SimpleTest::run()</span>
|
||
|
|
method.
|
||
|
|
We can use that result to exit the script with the desired return
|
||
|
|
code...
|
||
|
|
<pre>
|
||
|
|
<?php
|
||
|
|
require_once('simpletest/unit_tester.php');
|
||
|
|
require_once('simpletest/reporter.php');
|
||
|
|
|
||
|
|
$test = &new GroupTest('File test');
|
||
|
|
$test->addTestFile('tests/file_test.php');
|
||
|
|
<strong>exit ($test->run(new TextReporter()) ? 0 : 1);</strong>
|
||
|
|
?>
|
||
|
|
</pre>
|
||
|
|
Of course we don't really want to create two test scripts,
|
||
|
|
a command line one and a web browser one, for each test suite.
|
||
|
|
The command line reporter includes a method to sniff out the
|
||
|
|
run time environment...
|
||
|
|
<pre>
|
||
|
|
<?php
|
||
|
|
require_once('simpletest/unit_tester.php');
|
||
|
|
require_once('simpletest/reporter.php');
|
||
|
|
|
||
|
|
$test = &new GroupTest('File test');
|
||
|
|
$test->addTestFile('tests/file_test.php');
|
||
|
|
<strong>if (TextReporter::inCli()) {</strong>
|
||
|
|
exit ($test->run(new TextReporter()) ? 0 : 1);
|
||
|
|
<strong>}</strong>
|
||
|
|
$test->run(new HtmlReporter());
|
||
|
|
?>
|
||
|
|
</pre>
|
||
|
|
This is the form used within SimpleTest itself.
|
||
|
|
</p>
|
||
|
|
|
||
|
|
<p>
|
||
|
|
<a class="target" name="xml">
|
||
|
|
<h2>Remote testing</h2>
|
||
|
|
</a>
|
||
|
|
</p>
|
||
|
|
<p>
|
||
|
|
SimpleTest ships with an <span class="new_code">XmlReporter</span> class
|
||
|
|
used for internal communication.
|
||
|
|
When run the output looks like...
|
||
|
|
<pre class="shell">
|
||
|
|
<?xml version="1.0"?>
|
||
|
|
<run>
|
||
|
|
<group size="4">
|
||
|
|
<name>Remote tests</name>
|
||
|
|
<group size="4">
|
||
|
|
<name>Visual test with 48 passes, 48 fails and 4 exceptions</name>
|
||
|
|
<case>
|
||
|
|
<name>testofunittestcaseoutput</name>
|
||
|
|
<test>
|
||
|
|
<name>testofresults</name>
|
||
|
|
<pass>This assertion passed</pass>
|
||
|
|
<fail>This assertion failed</fail>
|
||
|
|
</test>
|
||
|
|
<test>
|
||
|
|
...
|
||
|
|
</test>
|
||
|
|
</case>
|
||
|
|
</group>
|
||
|
|
</group>
|
||
|
|
</run>
|
||
|
|
</pre>
|
||
|
|
You can make use of this format with the parser
|
||
|
|
supplied as part of SimpleTest itself.
|
||
|
|
This is called <span class="new_code">SimpleTestXmlParser</span> and
|
||
|
|
resides in <em>xml.php</em> within the SimpleTest package...
|
||
|
|
<pre>
|
||
|
|
<?php
|
||
|
|
require_once('simpletest/xml.php');
|
||
|
|
|
||
|
|
...
|
||
|
|
$parser = &new SimpleTestXmlParser(new HtmlReporter());
|
||
|
|
$parser->parse($test_output);
|
||
|
|
?>
|
||
|
|
</pre>
|
||
|
|
The <span class="new_code">$test_output</span> should be the XML format
|
||
|
|
from the XML reporter, and could come from say a command
|
||
|
|
line run of a test case.
|
||
|
|
The parser sends events to the reporter just like any
|
||
|
|
other test run.
|
||
|
|
There are some odd occasions where this is actually useful.
|
||
|
|
</p>
|
||
|
|
<p>
|
||
|
|
A problem with large test suites is thet they can exhaust
|
||
|
|
the default 8Mb memory limit on a PHP process.
|
||
|
|
By having the test groups output in XML and run in
|
||
|
|
separate processes, the output can be reparsed to
|
||
|
|
aggregate the results into a much smaller footprint top level
|
||
|
|
test.
|
||
|
|
</p>
|
||
|
|
<p>
|
||
|
|
Because the XML output can come from anywhere, this opens
|
||
|
|
up the possibility of aggregating test runs from remote
|
||
|
|
servers.
|
||
|
|
A test case already exists to do this within the SimpleTest
|
||
|
|
framework, but it is currently experimental...
|
||
|
|
<pre>
|
||
|
|
<?php
|
||
|
|
<strong>require_once('../remote.php');</strong>
|
||
|
|
require_once('../reporter.php');
|
||
|
|
|
||
|
|
$test_url = ...;
|
||
|
|
$dry_url = ...;
|
||
|
|
|
||
|
|
$test = &new GroupTest('Remote tests');
|
||
|
|
$test->addTestCase(<strong>new RemoteTestCase($test_url, $dry_url)</strong>);
|
||
|
|
$test->run(new HtmlReporter());
|
||
|
|
?>
|
||
|
|
</pre>
|
||
|
|
The <span class="new_code">RemoteTestCase</span> takes the actual location
|
||
|
|
of the test runner, basically a web page in XML format.
|
||
|
|
It also takes the URL of a reporter set to do a dry run.
|
||
|
|
This is so that progress can be reported upward correctly.
|
||
|
|
The <span class="new_code">RemoteTestCase</span> can be added to test suites
|
||
|
|
just like any other group test.
|
||
|
|
</p>
|
||
|
|
|
||
|
|
</div>
|
||
|
|
<div class="copyright">
|
||
|
|
Copyright<br>Marcus Baker, Jason Sweat, Perrick Penet 2004
|
||
|
|
</div>
|
||
|
|
</body>
|
||
|
|
</html>
|