Regression Testing

Sometimes you need to ensure that modifications to existing code do not cause performance regressions.

PHPBench allows you to store results and use them as a baseline for subsequent runs.

Creating a baseline

The baseline is a tagged benchmark result. When creating a baseline we should try and ensure it is as accurate as possible, it is therefore recommended to use a low value of Retry Threshold and use a appropriate amount of iterations (e.g. 10):

Run your benchmark on the code to which you want to compare:

$phpbenchruntests/Benchmark/MyBenchmark.php--tag=original--retry-threshold=5--iterations=10

Compare against the baseline

Switch to the new version of your code and run PHPBench again:

$phpbenchruntests/Benchmark/MyBenchmark.php--report=aggregate--ref=original--retry-threshold=5--iterations=10

Note that:

  • --ref is used to reference the previous, tagged, run, which will merge the previous benchmarks into the results. - --report=aggregate shows the aggregate report, and if a baseline is present it will show differences.

The final report should look something like:

👁 ../_images/baseline.png

The mean, mode and rstdev columns show the percentage difference from the previous run.

Assertions

You can compare against baselines in assertions, e.g.

$phpbenchruntests/Benchmark/MyBenchmark.php--ref=original--retry-threshold=5--iterations=10--assert="variant.mode <= baseline.mode +/- 5%"

We assert the current run’s mode is less than or equal to the baseline mode and tolerate a variance within 5% of the baseline mode.

See Assertions for more information.