Code coverage of test cases
In a perfect world, every line of every Biskit module should be touched by some test case. In practise, we are still a bit away from this ideal situation. It's very easy to see whether or not a certain module has any tests associated with it. But how do we know how much of the code is actually covered by these tests? There are several python tools to answer this question. Among the most popular ones is the coverage.py package.
Installation of coverage.py
coverage.py is available from the PythonPackageIndex:
The easiest option is to install coverage directly from the PythonPackageIndex using the new PIP python installer. [Ubuntu's Debian package (python-coverage) is very outdated]. If you haven't done yet, first install PIP. For example (Ubuntu/Debian):
sudo apt-get install python-setuptools python-pip
Now fetch and install coverage:
sudo pip install coverage
Note: The coverage program only works with the latest Biskit version (SVN release 977 or higher). Any older version (including release 2.3.1) will give you an error from test.py which had to be adapted for use with external programs.
Record and analyze test coverage
Now run one or all Biskit test cases while letting coverage watch and record which lines have been touched:
cd biskit/Biskit coverage run python test.py -e pvm old
This creates a raw output folder .coverage. There are different ways of accessing the coverage information. The easiest is:
coverage report *py
... will list all python files and their test coverage.
More informative is the color-coded html version of your code that coverage can create:
coverage html
This will create a htmlcov folder within the current directory with a nicely color-coded HTML version of each modules code coverage. For example, after running the Executor.py test case, coverage shows that we are only covering one set of possible input methods and the red portion of the code is not tested: