| 1 |
|
| 2 |
Testing Apache with the Perl Test Harness |
| 3 |
Prerequisites |
| 4 |
------------- |
| 5 |
These two modules must first be installed; |
| 6 |
|
| 7 |
- perl-ExtUtils-MakeMaker |
| 8 |
- perl-Test |
| 9 |
|
| 10 |
You'll need to install the CPAN modules listed in: |
| 11 |
Apache-Test/lib/Bundle/ApacheTest.pm |
| 12 |
All you have to do to install them all in one shot is: |
| 13 |
perl -MCPAN -e 'install Bundle::ApacheTest' |
| 14 |
|
| 15 |
Which are also available in one tarball here: |
| 16 |
http://perl.apache.org/~dougm/httpd-test-bundle-0.02.tar.gz |
| 17 |
|
| 18 |
Note: Crypt::SSLeay requires OpenSSL to be installed (only required |
| 19 |
for t/TEST -ssl): http://www.openssl.org/ |
| 20 |
More accurate results may be obtained by using the same openssl command |
| 21 |
line and libraries as consumed by APR-util and mod_ssl, due to X509 |
| 22 |
formatting behavior differences. |
| 23 |
|
| 24 |
For an extensive documentation see |
| 25 |
http://perl.apache.org/docs/general/testing/testing.html |
| 26 |
or |
| 27 |
http://svn.apache.org/viewvc/perl/modperl/docs/trunk/src/docs/general/testing/testing.pod |
| 28 |
|
| 29 |
To run the tests for all Apache web server modules, some additional |
| 30 |
CPAN modules will be required. If the tests don't work, make sure |
| 31 |
that you have up to date versions of each of these perl modules: |
| 32 |
|
| 33 |
``` |
| 34 |
cpan App::cpanminus |
| 35 |
cpanm Bundle::ApacheTest \ |
| 36 |
HTTP::DAV DateTime Time::HiRes \ |
| 37 |
Test::Harness Crypt::SSLeay Net::SSLeay IO::Socket::SSL \ |
| 38 |
IO::Socket::IP IO::Select LWP::Protocol::https AnyEvent \ |
| 39 |
AnyEvent::WebSocket::Client LWP::Protocol::AnyEvent::http FCGI |
| 40 |
``` |
| 41 |
|
| 42 |
|
| 43 |
Quick Start |
| 44 |
----------- |
| 45 |
|
| 46 |
If you don't care how it works and just want to run the tests, here's |
| 47 |
how you go about doing that. |
| 48 |
|
| 49 |
1. You need an installation of Apache. (1.3.x thru trunk) |
| 50 |
2. Any DSOs you wish to use should be configured in that Apache's |
| 51 |
httpd.conf (the test harness will pick this configuration up) |
| 52 |
3. Setup: |
| 53 |
perl Makefile.PL -apxs /path/to/apache/bin/apxs |
| 54 |
4. Run the tests: |
| 55 |
t/TEST |
| 56 |
5. Evaluate test output. |
| 57 |
|
| 58 |
Getting a little deeper |
| 59 |
----------------------- |
| 60 |
|
| 61 |
The test harness will run every .t file under the t/ directory. So |
| 62 |
let's say you only want to run the tests for PHP. Do this: |
| 63 |
t/TEST -httpd /path/to/apache/bin/httpd t/php |
| 64 |
|
| 65 |
That will start the test server, run the .t tests under t/php and shut |
| 66 |
down the test server. You can also control each of these steps. |
| 67 |
|
| 68 |
This will start the test server: |
| 69 |
t/TEST -httpd /path/to/apache/bin/httpd -start |
| 70 |
|
| 71 |
This will run the PHP tests in the test environment: |
| 72 |
t/TEST t/php |
| 73 |
|
| 74 |
This will stop the test server: |
| 75 |
t/TEST -stop |
| 76 |
|
| 77 |
This will run the server under gdb (using -X): |
| 78 |
t/TEST -d gdb |
| 79 |
|
| 80 |
Note: At this point, you have a working test environment. You can |
| 81 |
look in t/conf for the test server configuration files. These are |
| 82 |
generated by the test harness. Once you have a working test |
| 83 |
environment, you do not need to specify 'httpd' on the t/TEST command |
| 84 |
line. For instance, to start the server up again, the command |
| 85 |
t/TEST -start |
| 86 |
would be sufficient. |
| 87 |
|
| 88 |
Running Regression Tests |
| 89 |
------------------------ |
| 90 |
For a full regression test, you should have all modules loaded. Build the server |
| 91 |
with |
| 92 |
configure --enable-modules=reallyall --enable-load-all-modules ... |
| 93 |
among other things. Edit the generated httpd.conf and comment all mpm modules |
| 94 |
that you do not want. Run "t/TEST -clean" again. |
| 95 |
|
| 96 |
You will see some |
| 97 |
skipped: cannot find module 'XXX' |
| 98 |
as not all modules are in every apache release (but the tests run for all). |
| 99 |
|
| 100 |
All in all, some >4k tests will run and the result needs to be: PASS |
| 101 |
|
| 102 |
Trouble Shooting |
| 103 |
---------------- |
| 104 |
If you have a "PASS" at the end of "t/TEST", congratulations! If not, this |
| 105 |
sections gives some advise in order to find the cause. Feel free to expand |
| 106 |
this to make life easier for others. |
| 107 |
|
| 108 |
0. If your test startup hangs forever in "waiting for server to warm up", but |
| 109 |
the test server is reachable under port 8529, you might be the victim of |
| 110 |
ipv4/6 confusion. The default servername configured is "localhost" and |
| 111 |
some operating systems define 127.0.0.1 *as well as* ::1 in /etc/hosts. |
| 112 |
If the test server listens only on 0.0.0.0 it might not answer requests to |
| 113 |
::1 and that causes the test startup to hang. |
| 114 |
Solution: comment the ::1 line in /etc/hosts and see if that improves matters. |
| 115 |
1. Run "t/TEST -clean" every time you change something in your Apache |
| 116 |
configuration. The test suite picks up certain things from your installed |
| 117 |
httpd.conf (such as LoadModule statements) and will not see your changes |
| 118 |
unless you clean it. |
| 119 |
2. Failures in proxy.t may originate from the fact that the test script cannot |
| 120 |
open the specified port. This happens on some machines if you abort a test |
| 121 |
run and the socket is not properly shut down. Check if the error goes |
| 122 |
away after a reboot. (proxy.t tests are slow, so chances you interrupt tests |
| 123 |
at that point are good.) |
| 124 |
3. Failures in access.t may result from reverse lookups not working or giving |
| 125 |
other answers than expected. In the cause 0 above, if the test client |
| 126 |
connects via 127.0.0.1, a "Grant for localhost" might resolve to "::1" |
| 127 |
and therefore will not match the access rules of the tests. |
| 128 |
Solution: check that your servername is 'localhost' (which is |
| 129 |
the default) and that it *always* resolves to 127.0.0.1. |
| 130 |
4. If some ssl test cases fail, especially when t/ssl/proxy.t fails, the |
| 131 |
reason can be mismatches between your installed SSL library and the one |
| 132 |
httpd uses. The "openssl" binary found in your $PATH is used to create |
| 133 |
binary setup files by t/TEST. If another version of openssl then tries |
| 134 |
to read these from your Apache server process, it might fail. |
| 135 |
Try the following: |
| 136 |
> t/TEST -clean |
| 137 |
> PATH=<bin dir of correct openssl>:$PATH t/TEST |
| 138 |
If a lot of ssl tests fail, check in the error log for the presence of |
| 139 |
a certificate validation error. If you find it, check the expiration date |
| 140 |
of the TLS/SSL certificates used by the tests, they might be expired. |
| 141 |
Running TEST -clean should delete the old ssl certificates, so they'll be |
| 142 |
regenerated during the next run. |
| 143 |
5. If you see failures in the modules/h2.t test cases, please notify the dev |
| 144 |
mailing list with a description of your setup. These tests are quite young, |
| 145 |
currently only valid in 2.4.x and later and interact with quite some other |
| 146 |
modules as well as Openssl versions installed. Some tests require mod_ssl |
| 147 |
so make sure to load it in the httpd conf. |
| 148 |
6. Segmentation faults and core dumps occurring while executing the test suite |
| 149 |
might indicate a real problem but always run again the tests after |
| 150 |
a clean make install to avoid inconsistencies from old objects. |
| 151 |
7. If you see error messages like "Parse errors: Bad plan. |
| 152 |
You planned X tests but ran Y." it usually means that you are missing |
| 153 |
a perl module or the tested httpd module depends on another one |
| 154 |
not loaded in the httpd config. |
| 155 |
8. If you see SSL certificate errors, remove t/conf/ssl/ca prior to |
| 156 |
t/TEST -clean |
| 157 |
9. perl 5.28 in MacOS homebrew seems to hang the test suite. Invoking |
| 158 |
/usr/bin/perl Makefile.PL -apxs ... will cause an older perl to be used. |
| 159 |
|
| 160 |
Smoking Tests |
| 161 |
------------- |
| 162 |
|
| 163 |
Sometimes it's possible that the test is passing properly for the |
| 164 |
first time, when it's run for the first time in the thread. But when |
| 165 |
you run it again, the test might fail. It's important to run the |
| 166 |
repetition smoke testing. For example to repeat the tests 5 times you |
| 167 |
can run: |
| 168 |
|
| 169 |
t/SMOKE -times=5 |
| 170 |
|
| 171 |
It's also possible that a test will pass when it's run after a |
| 172 |
particular test, but if moved to run after a different state it may |
| 173 |
fail. For this reason by default the tests run in random order. |
| 174 |
|
| 175 |
Since it's important to be able to reproduce the problem with the |
| 176 |
random testing, whenever -order=random is used, the used seed is |
| 177 |
printed to STDERR. Which can be then fed into the future tests with: |
| 178 |
via APACHE_TEST_SEED environment variable. |
| 179 |
|
| 180 |
By adding the option -order=repeat, the tests will be run in |
| 181 |
alphabetical order. |
| 182 |
|
| 183 |
Combining these two important smoke testing techniques, one can run |
| 184 |
tests with: |
| 185 |
|
| 186 |
t/SMOKE -times=N -order=(repeat|random) |
| 187 |
|
| 188 |
For example, to run the mod_rewrite tests 5 times, one would: |
| 189 |
|
| 190 |
t/SMOKE -times=5 -verbose t/modules/rewrite.t |
| 191 |
|
| 192 |
So the tests can be repeated N times, and run in the following three |
| 193 |
modes: |
| 194 |
|
| 195 |
- randomize all tests |
| 196 |
- repeat the whole tests suite N times |
| 197 |
|
| 198 |
For configuration options and default settings run: |
| 199 |
|
| 200 |
t/SMOKE -help |
| 201 |
|
| 202 |
For more information refer to the Apache::TestSmoke manpage. |
| 203 |
|
| 204 |
|
| 205 |
Test Environment Configuration |
| 206 |
------------------------------ |
| 207 |
|
| 208 |
The test server is configured with conf files like any normal Apache |
| 209 |
server. The tricky part is those conf files are generated by the |
| 210 |
harness just prior to starting the server. t/conf/httpd.conf is |
| 211 |
generated by t/conf/httpd.conf.in. If that does not exist, the |
| 212 |
harness will generate a working configuration and will include |
| 213 |
LoadModule (and AddModule for Apache 1.3) directives from the |
| 214 |
httpd.conf associated with the httpd binary you are using for testing. |
| 215 |
If t/conf/extra.conf.in exists, t/conf/extra.conf will be generated |
| 216 |
from that, and an Include directive for that file will be put in the |
| 217 |
generated t/conf/httpd.conf. t/conf/apache_test_config.pm is |
| 218 |
generated from the test configuration. It contains all the |
| 219 |
information about the configuration of your test server. You can |
| 220 |
access this information in test scripts by: |
| 221 |
my $env = Apache::TestConfig->thaw; |
| 222 |
Apache::TestConfig access apache_test_config.pm and returns a hash |
| 223 |
reference with all the information. Look through |
| 224 |
apache_test_config.pm, it's a lot of stuff. Once these conf files are |
| 225 |
generated, you have a working test environment, and they must be |
| 226 |
'cleaned' if you wish to make changes to them. To clean the |
| 227 |
environment: |
| 228 |
t/TEST -clean |
| 229 |
(Now you will have to specify your httpd binary when starting back up |
| 230 |
again.) |
| 231 |
|
| 232 |
|
| 233 |
More Information |
| 234 |
---------------- |
| 235 |
|
| 236 |
For more information on using the test harness and writing tests, see |
| 237 |
the README in Apache-Test and the examples in Apache-Test/t. |
| 238 |
|
| 239 |
The test harness was originally written by Doug MacEachern and is |
| 240 |
discussed on the httpd dev mailing list (dev@httpd.apache.org). |
| 241 |
|
| 242 |
It is also included in modperl-2.0 source along with tests for |
| 243 |
modperl-2.0. |
Parent Directory
| 
Revision Log