Code coverage of test cases

In a perfect world, every line of every Biskit module should be touched by some test case. In practise, we are still a bit away from this ideal situation. It's very easy to see whether or not a certain module has any tests associated with it. But how do we know how much of the code is actually covered by these tests? There are several python tools to answer this question. Among the most popular ones is the coverage.py package.

Installation of coverage.py

coverage.py is available from the PythonPackageIndex:

http://pypi.python.org/pypi/coverage

The easiest option is to install coverage directly from the PythonPackageIndex using the new PIP python installer. [Ubuntu's Debian package (python-coverage) is very outdated]. If you haven't done yet, first install PIP. For example (Ubuntu/Debian):

sudo apt-get install python-setuptools python-pip

Now fetch and install coverage:

sudo pip install coverage 

Note: The coverage program only works with the latest Biskit version (SVN release 977 or higher). Any older version (including release 2.3.1) will give you an error from test.py which had to be adapted for use with external programs.

Record and analyze test coverage

Now run one or all Biskit test cases while letting coverage watch and record which lines have been touched:

cd biskit/Biskit
coverage run python test.py -e pvm old

This creates a raw output folder .coverage. There are different ways of accessing the coverage information. The easiest is:

coverage report *py

... will list all python files and their test coverage.

More informative is the color-coded html version of your code that coverage can create:

coverage html

This will create a htmlcov folder within the current directory with a nicely color-coded HTML version of each modules code coverage. For example, after running the Executor.py test case, coverage shows that we are only covering one set of possible input methods and the red portion of the code is not tested: