We write a lot of PHP unit tests at Wayfair, and we want to be able to run them as fast as possible, which seems like a good use case for parallelization. Running tests in parallel is not built in to PHPUnit, but there are ways to do it. When we looked we found three: parallel-phpunit, ParaTest, and GNU Parallel. All met some of our needs, but none was exactly what we wanted, so we got to work.
After hacking for a bit, we settled on these requirements:
- Easy to set up
- Fast to run
- Minimalist configuration and resource usage
- Not dependent on PHP, because chicken-before-egg
and these specifications for input, operation and output:
- Use suites--work with existing test suites, or make suites as needed out of individual test files
- Run suites in parallel
- Preserve exit codes and errors
We looked at GNU Parallel. It worked, but it was an additional dependency, and it is not conveniently packaged for a broad set of platforms. It also ended up running more slowly than backgrounding in the shell, and since we didn't need any of the fancier/nicer features of it, we cut it out of our scripts.
ParaTest is awesome, but it uses PHP, which complicates things when we're testing new versions or features of PHP.
Parallel-phpunit was the closest existing tool to what we wanted, but we didn't like the overhead of invoking a separate PHPUnit process for each file. The logical design of our new 'Sweet Parallel PHPUnit' is a suite-enabled bash test-runner script, similar to parallel-phpunit, with output and error codes handled to our liking. The Linux PIPESTATUS array variable was the key to doing this last part in bash.
So we finally got everything working, as you can see https://github.com/wayfair/sp-phpunit on github, and it was time for the moment of truth. Did it actually work any faster than our other options, on our own largest battery of tests? YES! We cut our run time down by 36% relative to the fastest alternative, while maintaining a small memory footprint! Before opensourcing it, we also generated some generic tests, to convince ourselves that our success wasn't a coincidental artifact of our own test suite.
We wrote scripts to handle a few different scenarios. Here is what they generate:
- One massive file with 2500 unit tests.
- 25 folders each with 100 files, each containing exactly 1 unit test.
- 10 folders each containing 10 files that have 1 unit test that sleeps for between 0 and 2 seconds
- Same thing, but with one anomalous file, hand-edited, that sleeps for 30 seconds instead of 0-2.
- 25 files each containing 100 unit tests.
Below you can see the results of each suite run against the other parallel options, as well as PHPUnit directly for comparison.
| Running with 6 Parallel threads, average over 5 runs(minutes:seconds) | ||||
| Test Case | sp-phpunit | paratest | Parallel-phpunit | phpunit(original) |
| 2500 files with 1 | 00:08.93 | 02:00.00 | 03:05.67 | 00:34.87 |
| 25 files with 100 tests | 00:01.35 | 00:02.12 | 00:02.65 | 00:01.47 |
| One file with 2500 tests | 00:01.83 | 00:01.99 | 00:01.73 | 00:01.49 |
| 100 files with sleeps | 00:18.51 | 00:19.45 | 00:22.85 | 01:40.36 |
| 100 files with sleeps(one file sleeps for 30 seconds) | 00:45.47 | 00:30.47 | 00:32.55 | 02:10.55 |
You can see that the more files you have, the more sp-phpunit really shines. We happen to have many small files, with many quick tests spread out among them, so our real test suite is most like the first line in the table, and the improvements are dramatic.
The TODO list for this project is not empty. The way that sp-phpunit generates its temporary suites has no knowledge of how long each sub test/file will take. This can lead to some bad luck where, for example, if you do 6 parallel runs, 5 might finish in 3 seconds, but one that happens to contain the slow tests will take say another minute to finish. This is clearly shown in the last row of our table. The 'sleep 30' is added to a bunch of other tests, and the cumulative effect, because of the grouping that we do, pushes the cumulative time for sp-phpunit higher than the other frameworks.
In upcoming versions I'd like to implement something so that you can pass in a folder to create a suite from. Also since this was created for our system only, I'm sure there are some options that other people will want or need that we have not implemented simply because the default behavior works for us. I hope this has given some insight into how and why we built sp-phpunit. We hope others find it as useful as we have. If you do happen to check it out and have some results you would like to share, please reach out. We'd be very excited to hear about it!