Development:Unit Tests

From OpenLP
Jump to: navigation, search

Introduction

If you haven't read through Test-Driven Development yet, go and read that first. That will explain some of the reasoning behind writing tests.

Here are two basic tests:

from openlp.core.lib import str_to_bool

 
def test_str_to_bool_with_true_bool():
    """
    Test the str_to_bool function with True input
    """
    # GIVEN: A boolean value set to true
    true_boolean = True

    # WHEN: We "convert" it to a bool
    true_result = str_to_bool(true_boolean)

    # THEN: We should get back a True bool
    assert isinstance(true_result, bool), 'The result should be a boolean'
    assert true_result is True, 'The result should be True'


def test_str_to_bool_with_false_bool():
    """
    Test the str_to_bool function with False input
    """
    # GIVEN: A boolean value set to false
    false_boolean = False

    # WHEN: We "convert" it to a bool
    false_result = str_to_bool(false_boolean)

    # THEN: We should get back a False bool
    assert isinstance(false_result, bool), 'The result should be a boolean'
    assert false_result is False, 'The result should be False'
  1. Each function is a test
  2. In order for a function to be recognised as a test, it needs to start with "test_"
  3. Each test is divided into 3 sections: GIVEN, WHEN, THEN:
    • GIVEN: These are the things we start with
    • WHEN: This is what happens
    • THEN: These are the results
  4. For simplicity's sake I have two tests which each have 1 GIVEN-WHEN-THEN combination. As a general rule, if you have more than one WHEN, then you probably need to split your test into smaller tests. In some cases this is not possible, but this is a good rule of thumb.

To actually run the tests we use the pytest test runner and framework. The test actually match the unittest style due to our history, rather than pytest style tests, but pytest happily runs that style of tests.

You'll see that in the tests I use the assert keyword to determine that my results are what I expect. pytest will catch the AssertionError exceptions that these assert statements raise if the statement is not as expected, and will record that test as having failed. The failure message is an optional argument of the assert methods.

You'll see this test is passing a boolean into a function that looks like it should take a string. Tests should check to see how a piece of code behaves depending on what input it receives.

In this function's case, it is able to take a boolean so I test that it can take a boolean value and will return the correct result. Here is another test for the same function:

def test_str_to_bool_with_int():
    """
    Test the str_to_bool function with an integer input
    """
    # GIVEN: An integer value
    int_string = 1

    # WHEN: we convert it to a bool
    int_result = str_to_bool(int_string)

    # THEN: we should get back a false
    assert int_result is False, 'The result should be False'


def test_str_to_bool_with_invalid_string():
    """
    Test the str_to_bool function with an invalid string
    """
    # GIVEN: An string value with completely invalid input
    invalid_string = 'my feet are wet'

    # WHEN: we convert it to a bool
    str_result = str_to_bool(invalid_string)

    # THEN: we should get back a false
    assert str_result is False, 'The result should be False'

This one tests what happens if I send in invalid input.

Test Naming

In order for pytest to pick up the tests, they need to be named correctly. In the above examples I've named them according to the convention you need to use. Briefly, it is this:

  1. Test functions should be named test_<function_name>
  2. Test case classes should be named Test<ClassName> (you'll see some of these around, though they're not required)

Mocked Functions

Right, so now we have this one function, and it calls some other function - how do I know that I'm testing just this function and not the other function I'm calling?

Quite simply, we use a fantastic library called "mock" which is able to replace functions, methods and classes with our own magic versions that don't do anything. We can also make these magic versions act a particular way or return certain results. Mock is part of the unittest module.

Using MagicMock

One of the classes that can be used to replace other classes and functions is the MagicMock class.

from unittest.mock import MagicMock, patch

from openlp.core.lib import translate, check_directory_exists


def test_translate():
    """
    Test the translate() function
    """
    # GIVEN: A string to translate and a mocked Qt translate function
    context = 'OpenLP.Tests'
    text = 'Untranslated string'
    comment = 'A comment'
    encoding = 1
    n = 1
    mocked_translate = MagicMock(return_value='Translated string')

    # WHEN: we call the translate function
    result = translate(context, text, comment, encoding, n, mocked_translate)

    # THEN: the translated string should be returned, and the mocked function should have been called
    mocked_translate.assert_called_with(context, text, comment, encoding, n)
    assert result == 'Translated string', 'The translated string should have been returned'

The mocked function pretends it is the function we are calling, returns a fixed value and keeps track of what it was called with so it can be checked later.

The translate() function above is unusual in that the function called is passed in. In most cases this doesn't have. This is where the "patch()" method comes in.

Using The patch Function

The patch() method patches an object and inserts itself into the place of the object you want to mock out.

Here is another test, this one uses the patch() function

def test_check_directory_exists():
    """
    Test the check_directory_exists() function
    """
    # GIVEN: A directory to check and a mocked out os.makedirs and os.path.exists
    with patch('openlp.core.lib.os.path.exists') as mocked_exists, \
            patch('openlp.core.lib.os.makedirs') as mocked_makedirs:
        directory_to_check = 'existing/directory'

        # WHEN: os.path.exists returns True and we check to see if the directory exists
        mocked_exists.return_value = True
        check_directory_exists(directory_to_check)

        # THEN: Only os.path.exists should have been called
        mocked_exists.assert_called_with(directory_to_check)
        assert mocked_makedirs.called is False, 'os.makedirs should not have been called'

So what I'm doing is I'm actually replacing os.path.exists() and os.makedirs() with my own mock methods, and then when I call the function it calls my mocks instead of the real functions.

This way we can completely control all the data around a function and check that it reacts accordingly. These are proper unit tests, they test the smallest possible unit (or block) of code.

We won't always be able to do this level of testing, but this is a good way to get used to writing tests

Please note: When you patch a method, you need to patch it where it is imported. So you don't just patch os.path.exists, you need to patch openlp.core.lib.os.path.exists.

Writing Your Tests

When writing a set of tests, the easiest way to do it is to start with the shortest path that the code takes.

For instance, let's look at this function:

def validate_thumb(file_path, thumb_path):
    """
    Validates whether an file's thumb still exists and if is up to date.
    **Note**, you must **not** call this function, before checking the
    existence of the file.

    ``file_path``
        The path to the file. The file **must** exist!

    ``thumb_path``
        The path to the thumb.
    """
    if not os.path.exists(thumb_path):
        return False
    image_date = os.stat(file_path).st_mtime
    thumb_date = os.stat(thumb_path).st_mtime
    return image_date <= thumb_date

The shortest path through this function would happen if os.path.exists() returned False. So let's write a test which mocks out os.path.exists() and makes it return False.

def test_validate_thumb_file_does_not_exist():
    """
    Test the validate_thumb() function when the thumbnail does not exist
    """
    # GIVEN: A mocked out os module, with path.exists returning False, and fake paths to a file and a thumb
    with patch('openlp.core.lib.os') as mocked_os:
        file_path = 'path/to/file'
        thumb_path = 'path/to/thumb'
        mocked_os.path.exists.return_value = False

        # WHEN: we run the validate_thumb() function
        result = validate_thumb(file_path, thumb_path)

        # THEN: we should have called a few functions, and the result should be False
        mocked_os.path.exists.assert_called_with(thumb_path)
        assert result is False, 'The result should be False'

Triggering and Handling Exceptions

Sometimes you need to check that a function handles exceptions properly. To do this you need to both raise an exception and then check that the exception was handled correctly in the code. Once again, the mock library comes to our rescue.

To raise an exception, simply mock out the method that should raise the exception, and then add a *side effect* to the method. If the function you're testing is supposed to raise an exception itself, or allow one to bubble up, then you can assert that the exception is raised using the pytest.raises() context manager.

import pytest


def test_check_directory_exists():
    """
    Test the check_directory_exists() function
    """
    # GIVEN: A directory to check and a mocked out os.makedirs and os.path.exists
    with patch('openlp.core.lib.os.path.exists') as mocked_exists, \
            patch('openlp.core.lib.os.makedirs') as mocked_makedirs:
        directory_to_check = 'existing/directory'

        # WHEN: Some unhandled exception is raised
        mocked_exists.side_effect = ValueError()

        # THEN: check_directory_exists raises an exception
        mocked_exists.assert_called_with(directory_to_check)
        with pytest.raises(ValueError):
            check_directory_exists(directory_to_check)

Write Thorough Tests

  1. Check what you got back from the function and that it was what you expected.
  2. Test each code path, start with the shortest.
  3. Check the mocked functions that you expected to be called were called.
  4. Check they were called with the expected parameters.

An example of writing a test for the get_filesystem_encoding() function in openlp/core/utils/__init__.py

from mock import MagicMock, patch
 
from openlp.core.utils import get_filesystem_encoding


def test_get_filesystem_encoding_cp1252():
    """
    Test the get_filesystem_encoding() function with an encoding of cp1252
    """
    # GIVEN: sys.getfilesystemencoding returns "cp1252"
    with patch('openlp.core.utils.sys.getfilesystemencoding') as mocked_getfilesystemencoding, \
            patch('openlp.core.utils.sys.getdefaultencoding') as mocked_getdefaultencoding:
        mocked_getfilesystemencoding.return_value = 'cp1252'

        # WHEN: get_filesystem_encoding() is called
        result = get_filesystem_encoding()

        # THEN: getdefaultencoding should have been called
        mocked_getfilesystemencoding.assert_called_with()
        assert mocked_getdefaultencoding.called is False
        assert result == 'cp1252', 'The result should be "cp1252"'


def test_get_filesystem_encoding_cp1252():
    """
    Test the get_filesystem_encoding() function with an encoding of cp1252
    """
    # GIVEN: sys.getfilesystemencoding returns None and sys.getdefaultencoding returns "utf-8"
    with patch('openlp.core.utils.sys.getfilesystemencoding') as mocked_getfilesystemencoding, \
            patch('openlp.core.utils.sys.getdefaultencoding') as mocked_getdefaultencoding:
        mocked_getfilesystemencoding.return_value = None
        mocked_getdefaultencoding.return_value = 'utf-8'

        # WHEN: get_filesystem_encoding() is called
        result = get_filesystem_encoding()

        # THEN: getdefaultencoding should have been called
        mocked_getfilesystemencoding.assert_called_with()
        mocked_getdefaultencoding.assert_called_with()
        assert result == 'utf-8', 'The result should be "utf-8"'

Location of tests

Tests are located under the tests directory. Historically tests were divided into functional and interfaces tests, but we no longer make those distinctions.

Software required

In order to run the unit tests, you will need to install pytest. On most Linux distributions you need to install the python3-pytest package. On Windows and macOS you should be able to just use pip to install pytest.

Running the tests

See the Running Tests page on how to run the tests you've written.