Behat Code Coverage

Behat Code Coverage is an extension for Behat that can generate code coverage reports when testing a PHP application. The extension wraps the same php-code-coverage library that is used by PHPUnit thereby producing reports that are familiar to developers and interoperable with other tooling.

Note

The authors of Behat pedantically, but correctly, point out that .feature files are not strictly speaking tests even though when constructed properly the scenarios described in them should cover both the happy and sad paths in an application.

Technically, Behat is a scenario runner, not a test runner. The scenarios might be run by hand. Or the application under scrutiny might not be a local PHP application, it might be running on a remote server and/or the software might not even be written in PHP. Additionally by the very nature of needing to invoke the entire application to perform each scenario, it would be very hard to construct a set of scenarios that cover all possible codepaths in an application. Something like PHPUnit is much better to use here if your goal is comprehensive code coverage as you can unit test each component in isolation.

However, out in the real world we don’t normally draw a distinction between the .feature files as a standalone concept and the Contexts that implement them - we simply refer to Behat testing. We also tend to use Behat when the application being tested is written in PHP. And as with any test suite, it’s nice to know how much of your application code is covered by a test suite. What you do with that information is up to you :)

License

Behat Code Coverage is licensed under the BSD-2-Clause License. See license.txt for full details.

Installation

The recommended way to install Behat Code Coverage is to use Composer. From the command line simply execute the following to add dvdoug/behat-code-coverage to the require-dev section of your project’s composer.json file. Composer will automatically take care of downloading the source and configuring an autoloader:

composer require --dev dvdoug/behat-code-coverage

The code is also available to download from GitHub

After installation, in your project’s behat.yml or behat.yml.dist, add DVDoug\Behat\CodeCoverage\Extension into the extensions key under default to enable it.

The below example represents a typical configuration you may wish to use as a starting point. Each option is more fully explained in the next section.

default:
    extensions:
        DVDoug\Behat\CodeCoverage\Extension:
            filter:
                include:
                    directories:
                        'src': ~
            reports:
                html:
                    target: build/coverage-behat
                text:
                    showColors: true

Compatibility

Since both PHPUnit and Behat Code Coverage both depend upon php-code-coverage, it is not always possible to run an older version of PHPUnit alongside the newest version of this library or vice versa since they both need to be compatible with the same underlying version of the code coverage component. Where reasonably possible Behat Code Coverage maintains support for multiple versions of the coverage library to ease the upgrade path, however where a new version is significantly incompatible with older ones or where a compelling new feature exists the minimum supported version may be raised. Fixes for any subsequently discovered bugs will be backported as appropriate.

Behat Code Coverage	Behat	php-code-coverage	PHPUnit	Notes
5.3	3.10+	9.2+, 10, 11	9.5.5+, 10, 11
5.2	3.5+	9.2+	9.4+	Cobertura format support
5.1	3.5+	9.1+	9.3.4+	Caching between runs
5.0	3.5+	6, 7, 8, 9	7, 8, 9

Configuration

To do anything useful, once installed, you should configure Behat Code Coverage according to your project needs.

Each of the settings described below has a direct equivalent in PHPUnit, if your codebase uses both testing frameworks you may wish to ensure the settings are aligned between the two tools.

Settings are configured in your project’s behat.yml or behat.yml.dist.

Driver

In order to generate code coverage data, you must have a code coverage driver installed - you may use either Xdebug or the PCOV extension. Drivers are detected at runtime, they do not need to be configured.

Filter

A filter is the mechanism to include (or exclude) directories or files from the generated coverage reports. Technically this is an optional configuration setting, but practically speaking most projects will want to configure this to report only on their own source files rather than e.g. test files or vendor files.

For directories, list them by key. Optionally, you can specify a prefix or suffix:

default:
    extensions:
        DVDoug\Behat\CodeCoverage\Extension:
            filter:
                include:
                    directories:
                        'src':
                            suffix: 'Controller.php'
                        'src/Repository':
                            prefix: 'API'
                exclude:
                    directories:
                        'tests/data': ~

For files, list them by name:

default:
    extensions:
        DVDoug\Behat\CodeCoverage\Extension:
            filter:
                include:
                    files:
                        - 'bootstrap.php'
                        - 'index.php'
                exclude:
                    files:
                        - 'src/ExcludeMe.php'
                        - 'src/ExcludeMeToo.php'

Filters have two additional settings, to control how files that have not been covered should be handled in the report. By default uncovered files are included but not processed.

default:
    extensions:
        DVDoug\Behat\CodeCoverage\Extension:
            filter:
                includeUncoveredFiles: true # defaults to true

• Included means that the file is incorporated into the reporting, showing up as 0% covered. This allows you to see files that you haven’t written any tests for yet. If you choose not to include uncovered files, then only files that have been loaded by your test suite (whether directly or indirectly) will be shown on the report.

• Uncovered files, by definition, will not been loaded into the PHP runtime environment during the execution of the test suite. This means that they have not been analysed by the coverage driver for detection of things like executable vs non-executable lines, dead code detection or calculation of branches and paths. By opting uncovered files into processing, the files will be loaded via an include() call and passed through the driver in exactly the same way as covered files are. However, not all files can be safely loaded in this way, for instance include()ing a script may have runtime side-effects. Because Behat Code Coverage cannot know the structure of your codebase and which files are safe to include() and which are not, by default uncovered file processing is disabled for safety reasons.

Reports

Behat Code Coverage allows you to generate reports in any/all of the report formats provided by php-code-coverage. Reports are configured under the reports key.

Text

The text report is the easiest way to get started, it simply outputs the report results to the screen after each test run. It is configured by setting the text key. The default values are outlined below:

default:
    extensions:
        DVDoug\Behat\CodeCoverage\Extension:
            reports:
                text:
                    showColors: false
                    showOnlySummary: false
                    showUncoveredFiles: false
                    lowUpperBound: 50
                    highLowerBound: 90

• If showColors is true, the results will be output in a colour-coded format, red for low coverage, amber/yellow for a medium amount of coverage and green for high coverage. If desired, the thresholds for each colour can be configured via lowUpperBound and highLowerBound.

• If you have a large codebase, outputting the coverage data for each and every individual file to the CLI may be too be noisy to be helpful. If so, you can set showOnlySummary to true which will output only a project-level overview.

• By default, when showing data for individual files the text report does not show data for uncovered files (even if data collection enabled under filter). This can be changed if desired by setting showUncoveredFiles to true. Note that this is a report-specific display option only, choosing not to show uncovered files in the text report has no impact on whether they are included in other types of report.

HTML

The HTML report is the most common format of report. As well as a summary report for the project providing high-level data, it also includes a detailed overview of each file showing the coverage on a function by function, line by line basis. It is configured by setting the html key. The default values are outlined below:

default:
    extensions:
        DVDoug\Behat\CodeCoverage\Extension:
            reports:
                html:
                    target: <directory>  # no default value, you must specify
                    lowUpperBound: 50
                    highLowerBound: 90
                    colors:
                        successLow: '#dff0d8'
                        successMedium: '#c3e3b5'
                        successHigh: '#99cb84'
                        warning: '#fcf8e3'
                        danger: '#f2dede'
                    customCSSFile: ~ # defaults to null, i.e. no custom CSS file

• The mandatory target key specifies the target directory to place the report files.

• The HTML report is heavily colour-coded format using red for low coverage, amber/yellow for a medium amount of coverage and green for high coverage. If desired, the thresholds for each colour can be configured via lowUpperBound and highLowerBound, and the precise colours used via the colors key. Further customisation may be done via specifying a custom CSS file

  Note

  The colors and customCSSFile settings are only available from php-code-coverage v10

Clover

Originating from the Java world, Clover-format reports are a standard way of exchanging coverage data between tools. It is configured by setting the clover key. The default values are outlined below:

default:
    extensions:
        DVDoug\Behat\CodeCoverage\Extension:
            reports:
                clover:
                    target: <file>  # no default value, you must specify
                    name: ''

• The mandatory target key specifies the destination filename to use for the report. The reports are in XML format, an .xml file extension is suggested.

• Optionally, you can configure the name of your project via the name key.

Cobertura

Also originating from the Java world, Cobertura-format reports are becoming a standard way of exchanging coverage data between tools. It is configured by setting the cobertura key. The default values are outlined below:

default:
    extensions:
        DVDoug\Behat\CodeCoverage\Extension:
            reports:
                cobertura:
                    target: <file>  # no default value, you must specify
                    name: ''

• The mandatory target key specifies the destination filename to use for the report. The reports are in XML format, an .xml file extension is suggested.

• Optionally, you can configure the name of your project via the name key.

Crap4j

An older, discontinued tool from the Java world. You can generate Crap4j-compatible reports by setting the crap4j key. The default values are outlined below:

default:
    extensions:
        DVDoug\Behat\CodeCoverage\Extension:
            reports:
                crap4j:
                    target: <file>  # no default value, you must specify
                    name: ''

• The mandatory target key specifies the destination filename to use for the report. The reports are in XML format, an .xml file extension is suggested.

• Optionally, you can configure the name of your project via the name key.

PHP “.cov”

A PHP or “.cov” report is a raw serialisation of internal php-code-coverage state, allowing for full fidelity of data to be preserved. They can be manipulated by the phpcov tool, for instance to combine reports from multiple testing tools. You can generate PHP “.cov” reports by setting the php key.

default:
    extensions:
        DVDoug\Behat\CodeCoverage\Extension:
            reports:
                php:
                    target: <file>  # no default value, you must specify

• The mandatory target key specifies the destination filename to use for the report. The reports are actually PHP, but a .cov file extension is customary.

PHPUnit XML

You can generate PHPUnit XML reports by setting the xml key.

default:
    extensions:
        DVDoug\Behat\CodeCoverage\Extension:
            reports:
                xml:
                    target: <directory>  # no default value, you must specify

• The mandatory target key specifies the target directory to use for the report.

Branch and path coverage

When using Xdebug as a coverage driver, it has the ability to generate branch and path coverage data as well as the traditional line-based data. More information on this topic is available at https://doug.codes/php-code-coverage.

default:
    extensions:
        DVDoug\Behat\CodeCoverage\Extension:
            branchAndPathCoverage: true

By default branchAndPathCoverage is true when running under Xdebug, false otherwise.

Caching

Since analysing source code files to generate coverage reports is computationally expensive, Behat Code Coverage makes use of a cache to ameliorate this.

default:
    extensions:
        DVDoug\Behat\CodeCoverage\Extension:
            cache: <directory>

The default cache directory is sys_get_temp_dir() . '/behat-code-coverage-cache'. You may wish to relocate this this to be inside your project workspace.

Changelog

Unreleased - 2024-xx-xx

5.3.4 - 2024-04-03

Fixed

• Better handling of relative paths

5.3.3 - 2024-02-03

Added

• Compatibility with phpunit/php-code-coverage v11

Changed

• Minimum phpunit/php-code-coverage version bumped to 10.1

5.3.2 - 2023-12-09

Added

• Support for Symfony 7

Removed

• Support for Symfony 4

5.3.1 - 2023-05-08

Changed

• Stopped using some deprecated methods inside phpunit/php-code-coverage v10.1

Fixed

• Better handling of PCOV and Xdebug running side by side

5.3.0 - 2023-02-04

Added

• Compatibility with phpunit/php-code-coverage v10

• Added colors and customCSSFile options to HTML report configuration

5.2.2 - 2021-11-30

Added

• Support for Symfony 6

Changed

• Improved error messages when a coverage driver cannot be found

• Deprecated processUncoveredFiles setting, it has been removed as of php-code-coverage v10.

5.2.1 - 2021-01-10

Fixed

• When Xdebug was enabled, but its coverage feature was disabled an exception was thrown. This scenario is now treated the same as when no coverage driver is loaded at all (a warning is printed but Behat is allowed to run to completion)

5.2.0 - 2020-10-11

Added

• Added support for the Cobertura report format

Changed

• Minimum phpunit/php-code-coverage version bumped to 9.2

5.1.1 - 2020-08-14

Fixed

• Make the --no-coverage option work again

5.1.0 - 2020-08-10

Added

• Support for phpunit/php-code-coverage‘s static analysis cache introduced in v9.1. This can be configured via the cache key in behat.yml, otherwise defaults to sys_get_temp_dir() . '/behat-code-coverage-cache'

Changed

• Minimum phpunit/php-code-coverage version bumped to 9.1

Removed

• Support for Symfony 3.4, in alignment with https://github.com/Behat/Behat/issues/1296

5.0.1 - 2021-09-13

Fixed

• Make the --no-coverage option work again

5.0.0 - 2020-08-07

Added

• Compatibility with phpunit/php-code-coverage v9. Branch and path coverage is automatically enabled when running under Xdebug. For more information on this feature, see https://doug.codes/php-code-coverage

• branchAndPathCoverage configuration key to enable/disable path and branch coverage. Setting this to true explicitly will warn when the feature cannot be used.

• Support for PCOV

Removed

• The old report configuration key, use reports instead

• Removed forceCoversAnnotation and mapTestClassNameToCoveredClassName configuration keys, these options are not supported by php-code-coverage anymore

• Removed the whitelist configuration key to align with php-code-coverage v9 terminology. All former subkeys of whitelist are now subkeys of filter

• Renamed addUncoveredFilesFromWhitelist and processUncoveredFilesFromWhitelist to includeUncoveredFiles and processUncoveredFiles to align with php-code-coverage v9 terminology

• Removed the custom driver selection logic and replaced it with built-in logic from php-code-coverage

• The RemoteXDebug driver, it was a companion to an old Symfony bundle, not a generally-usable feature

• Removed legacy LeanPHP\Behat\CodeCoverage alias

4.1.1 - 2020-02-15

Added

• Compatibility with phpunit/php-code-coverage v8

4.1.0 - 2019-11-04

Added

• Added reports configuration key to enable generation of multiple coverage output formats, with schema validation of the available format-specific options

Deprecated

• The report configuration key as it only allowed for a single report type

4.0.1 - 2019-08-04

Added

• Added back support for LeanPHP\Behat\CodeCoverage in behat.yml for seamless drop-in of the fork

• Support for phpdbg

Fixed

• Issue with directories containing dashes in the name

• Issue with xdebug and delete calls

4.0.0 - 2019-08-04

Added

• Support for version 7.0 of phpunit/php-code-coverage

Changed

• Changed namespace of all code to LeanPHP\Behat\CodeCoverage from DVDoug\Behat\CodeCoverage

• Minimum version of PHP supported is 7.1

Removed

• Support for Symfony components older than <3.4

• Support for HHVM