Development:Running Tests

From OpenLP
Jump to: navigation, search

Running the Tests Locally

To run the tests, navigate to the root directory of the OpenLP project, and then run the following command:

$ pytest

On some platforms (like Ubuntu), you will need to run pytest-3.

$ pytest-3

On Windows you may need to run it using pytest's file path e.g:

[OpenLP path]> C:\Python36\Scripts\pytest.exe

To run only a particular set of tests within a file, run the following command:

$ pytest tests/openlp_core/common/

Finally, to only run a particular test, run the following command:

$ pytest tests/openlp_core/common/

Or, for a test method on a class:

$ pytest tests/openlp_core/common/

Code Coverage

How can you be sure you're writing the right tests? This is where code coverage comes in. pytest supports the coverage through a plugin, which can be installed on most systems.

On Ubuntu or Debian:

$ sudo apt install python3-pytest-cov

On Fedora:

$ sudo dnf install python3-pytest-cov

Once you have coverage installed, just run the tests like this:

$ pytest --cov=openlp --cov-report=html

Then open up index.html in the htmlcov subdirectory in your current directory. The easiest way is to run a local webserver, like this:

$ (cd htmlcov && python3 -m http.server)

Then open your browser and navigate to http://localhost:8000 and you'll see something like this:


Logging into Jenkins

We use Jenkins for our Continuous Integration server. Jenkins checks out the code, runs all our tests automatically, and runs other code analysis tools like flake8 on the code base. This gives us a set sane environment as a baseline for our tests.

  1. Go to our OpenLP Jenkins instance and create an account. Skip to step 3 if you've done this already.
  2. Once you've registered, contact Raoul (superfly in IRC) and give him your username. He will let you know once your account has the correct permissions set up.
  3. Log into Jenkins
  4. Click on the "Branch" tab
  5. Click on the play/time icon in the far right column of the first job in the list.
  6. Type in your branch on Launchpad's URL (lp:~username/openlp/branchname) and click "Build"
  7. Watch your branch being built
  8. If anything fails, you need to fix your code :-)

Running Jenkins Locally

Please note: This step is no longer needed. When you propose a merge, our Jenkins server will automatically run all the tests in your branch and comment on the your merge proposal with the results.

Running the tests as described above is a little bit tedious, so we've written a script to make your life a little bit easier.

The script requires the python-jenkins module installed. This is already packaged in Debian and Fedora, but Windows and macOS developers will need to install it manually.


$ sudo apt install python3-jenkins


$ sudo dnf install python3-jenkins

Other platforms:

$ pip install python-jenkins

Once you've pushed your code up to Launchpad, simply run the script, and it will detect your remote branch, kick off the tests, and present you with the results on the command line. The script uses your username and either your password or a personal access token.

$ python3 scripts/ -u <username> -p <password or token>

To make things slightly easier for yourself, below is a bash script which you can use to run the tests:


if [[ "$USER" == "" || "$PASSWORD" == "" ]]; then
        echo "No username or password set."
        exit 2
python3 scripts/ -u $USER -p $PASSWORD $@

Save this script to somewhere in your $PATH (I usually add $HOME/bin to my $PATH anyway, so I stuck it in there), and call it ci. Then, when you want to run the tests all you have to do is navigate to your branch and run ci:

$ ci

Copy the result output into the description box of your merge proposal to show the team that you've run the tests on Jenkins.