Development:Unit Tests

From OpenLP
Jump to: navigation, search

Introduction

If you haven't read through Test-Driven Development yet, go and read that first. That will explain some of the reasoning behind writing tests.

A very basic test case (or test suite as it is sometimes called), with two tests:

from unittest import TestCase
 
from openlp.core.lib import str_to_bool
 
class TestLibModule(TestCase):
 
    def str_to_bool_with_true_bool_test(self):
        """
        Test the str_to_bool function with True input
        """
        # GIVEN: A boolean value set to true
        true_boolean = True

        # WHEN: We "convert" it to a bool
        true_result = str_to_bool(true_boolean)

        # THEN: We should get back a True bool
        self.assertIsInstance(true_result, bool, 'The result should be a boolean')
        self.assertTrue(true_result, 'The result should be True')
		
    def str_to_bool_with_false_bool_test(self):
        """
        Test the str_to_bool function with False input
        """
        # GIVEN: A boolean value set to false
        false_boolean = False

        # WHEN: We "convert" it to a bool
        false_result = str_to_bool(false_boolean)

        # THEN: We should get back a True bool
        self.assertIsInstance(false_result, bool, 'The result should be a boolean')
        self.assertFalse(false_result, 'The result should be True')
  1. We have a class which holds a set of tests related to each other.
  2. Each method in the class is a test (you can see, it ends with "_test").
  3. In order for a method to be recognised as a test, it needs to end with "_test".
  4. Each test is divided into 3 sections: GIVEN, WHEN, THEN:
    • GIVEN: These are the things we start with
    • WHEN: This is what happens
    • THEN: These are the results
  5. For simplicity's sake I have two tests which each have 1 GIVEN-WHEN-THEN combination. As a general rule, if you have more than one WHEN, then you probably need to split your test into smaller tests. In some cases this is not possible, but this is a good rule of thumb.

I've called this class TestLibModule, however it will usually have a name related to the module being tested.

To actually run the tests I am using a python test running framework called nosetests (or just nose). You'll see that in the tests I use the assert-methods of the TestCase-class to determine that my results are what I expect.

nose will catch the AssertionError exceptions that these assert-methods raise if the statement is not as expected, and will record that test as having failed. The failure-message is an optional argument of the assert-methods, if none is provided a default is used.

You'll see this test is passing a boolean into a function that looks like it should take a string. Tests should check to see how a piece of code behaves depending on what input it receives.

In this function's case, it is able to take a boolean so I test that it can take a boolean value and will return the correct result. Here is another test for the same function:

    def str_to_bool_with_int_test(self):
        """
        Test the str_to_bool function with an integer input
        """
        # GIVEN: An integer value
        int_string = 1

        # WHEN: we convert it to a bool
        int_result = str_to_bool(int_string)

        # THEN: we should get back a false
        self.assertFalse(int_result, 'The result should be False')
 
    def str_to_bool_with_invalid_string_test(self):
        """
        Test the str_to_bool function with an invalid string
        """
        # GIVEN: An string value with completely invalid input
        invalid_string = u'my feet are wet'

        # WHEN: we convert it to a bool
        str_result = str_to_bool(invalid_string)

        # THEN: we should get back a false
        self.assertFalse(str_result, 'The result should be False')

This one tests what happens if I send in invalid input.

Mocked Functions

Right, so now we have this other function, and it calls some other function - how do I know that I'm testing just this function and not the other function I'm calling?

Quite simply, we use a fantastic library called "mock" which is able to replace functions, methods and classes with our own magic versions that don't do anything.

Using MagicMock

One of the classes that can be used to replace other classes and functions is the MagicMock class.

from unittest import TestCase

from tests.functional import MagicMock, patch

from openlp.core.lib import translate, check_directory_exists

class TestLibModule(TestCase):

    def translate_test(self):
        """
        Test the translate() function
        """
        # GIVEN: A string to translate and a mocked Qt translate function
        context = u'OpenLP.Tests'
        text = u'Untranslated string'
        comment = u'A comment'
        encoding = 1
        n = 1
        mocked_translate = MagicMock(return_value=u'Translated string')

        # WHEN: we call the translate function
        result = translate(context, text, comment, encoding, n, mocked_translate)

        # THEN: the translated string should be returned, and the mocked function should have been called
        mocked_translate.assert_called_with(context, text, comment, encoding, n)
        self.assertEqual(result, 'Translated string', 'The translated string should have been returned')

The mocked function pretends it is the function we are calling, returns a fixed value and keeps track of what it was called with so it can be checked later.

The translate() function above is unusual in that the function called is passed in. In most cases this doesn't have. This is where the "patch()" method comes in.

Using The patch Function

The patch() method patches an object and inserts itself into the place of the object you want to mock out.

Here is another test, this one uses the patch() function

    def check_directory_exists_test(self):
        """
        Test the check_directory_exists() function
        """
        # GIVEN: A directory to check and a mocked out os.makedirs and os.path.exists
        with patch(u'openlp.core.lib.os.path.exists') as mocked_exists, \
                patch(u'openlp.core.lib.os.makedirs') as mocked_makedirs:
            directory_to_check = u'existing/directory'

            # WHEN: os.path.exists returns True and we check to see if the directory exists
            mocked_exists.return_value = True
            check_directory_exists(directory_to_check)

            # THEN: Only os.path.exists should have been called
            mocked_exists.assert_called_with(directory_to_check)
            self.assertFalse(mocked_makedirs.called, 'os.makedirs should not have been called')

So what I'm doing is I'm actually substituting os.path.exists() and os.makedirs() for my own mock methods, and then when I call the function it calls my mocks instead of the real functions.

This way we can completely control all the data around a function and check that it reacts accordingly. These are proper unit tests, they test the smallest possible unit (or block) of code.

We won't always be able to do this level of testing, but this is a good way to get used to writing tests

Notice when you patch a method, you need to patch it where it is imported. So you don't just patch "os.path.exists", you need to patch "openlp.core.lib.os.path.exists".

Writing Your Tests

When writing a set of tests, the easiest way to do it is to start with the shortest path that the code takes.

For instance, let's look at this function:

def validate_thumb(file_path, thumb_path):
    """
    Validates whether an file's thumb still exists and if is up to date.
    **Note**, you must **not** call this function, before checking the
    existence of the file.

    ``file_path``
        The path to the file. The file **must** exist!

    ``thumb_path``
        The path to the thumb.
    """
    if not os.path.exists(thumb_path):
        return False
    image_date = os.stat(file_path).st_mtime
    thumb_date = os.stat(thumb_path).st_mtime
    return image_date <= thumb_date

The shortest path through this function would happen if os.path.exists() returned False. So let's write a test which mocks out os.path.exists and makes it return False.

    def validate_thumb_file_does_not_exist_test(self):
        """
        Test the validate_thumb() function when the thumbnail does not exist
        """
        # GIVEN: A mocked out os module, with path.exists returning False, and fake paths to a file and a thumb
        with patch(u'openlp.core.lib.os') as mocked_os:
            file_path = u'path/to/file'
            thumb_path = u'path/to/thumb'
            mocked_os.path.exists.return_value = False
            
            # WHEN: we run the validate_thumb() function
            result = validate_thumb(file_path, thumb_path)
            
            # THEN: we should have called a few functions, and the result should be False
            mocked_os.path.exists.assert_called_with(thumb_path)
            self.assertFalse(result, 'The result should be False')

Triggering and Handling Exceptions

Sometimes you need to check that a function handles exceptions properly. To do this you need to both raise an exception and then check that the exception was handled correctly in the code. Once again, the mock library comes to our rescue.

To raise an exception, simply mock out the method that should raise the exception, and then add a *side effect* to the method. If the function you're testing is supposed to raise an exception itself, or allow one to bubble up, then you can assert that the exception is raised using the self.assertRaises() method from the TestCase class.

    def check_directory_exists_test(self):
        """
        Test the check_directory_exists() function
        """
        # GIVEN: A directory to check and a mocked out os.makedirs and os.path.exists
        with patch(u'openlp.core.lib.os.path.exists') as mocked_exists, \
                patch(u'openlp.core.lib.os.makedirs') as mocked_makedirs:
            directory_to_check = u'existing/directory'

            # WHEN: Some unhandled exception is raised
            mocked_exists.side_effect = ValueError()

            # THEN: check_directory_exists raises an exception
            mocked_exists.assert_called_with(directory_to_check)
            self.assertRaises(ValueError, check_directory_exists, directory_to_check)

Write Thorough Tests

  1. Check what you got back from the function and that it was what you expected.
  2. Test each code path, start with the shortest.
  3. Check the mocked functions that you expected to be called were called.
  4. Check they were called with the expected parameters.

An example of writing a test for the get_filesystem_encoding() function in openlp/core/utils/__init__.py

from unittest import TestCase
 
from mock import MagicMock, patch
 
from openlp.core.utils import get_filesystem_encoding
 
class TestLib(TestCase):
 
    def get_filesystem_encoding_cp1252_test(self):
        """
        Test the get_filesystem_encoding() function with an encoding of cp1252
        """
        # GIVEN: sys.getfilesystemencoding returns "cp1252"
        with patch(u'openlp.core.utils.sys.getfilesystemencoding') as mocked_getfilesystemencoding, \
                patch(u'openlp.core.utils.sys.getdefaultencoding') as mocked_getdefaultencoding:
            mocked_getfilesystemencoding.return_value = u'cp1252' 

            # WHEN: get_filesystem_encoding() is called
            result = get_filesystem_encoding()

            # THEN: getdefaultencoding should have been called
            mocked_getfilesystemencoding.assert_called_with()
            self.assertFalse(mocked_getdefaultencoding.called)
            self.assertEqual(result, 'cp1252', 'The result should be "cp1252"')
 
    def get_filesystem_encoding_cp1252_test(self):
        """
        Test the get_filesystem_encoding() function with an encoding of cp1252
        """
        # GIVEN: sys.getfilesystemencoding returns None and sys.getdefaultencoding returns "utf-8"
        with patch(u'openlp.core.utils.sys.getfilesystemencoding') as mocked_getfilesystemencoding, \
                patch(u'openlp.core.utils.sys.getdefaultencoding') as mocked_getdefaultencoding:
            mocked_getfilesystemencoding.return_value = None
            mocked_getdefaultencoding.return_value = u'utf-8' 

            # WHEN: get_filesystem_encoding() is called
            result = get_filesystem_encoding()

            # THEN: getdefaultencoding should have been called
            mocked_getfilesystemencoding.assert_called_with()
            mocked_getdefaultencoding.assert_called_with()
            self.assertEqual(result, 'utf-8', 'The result should be "utf-8"')

Location of tests

Tests are located under the tests directory. Inside this is a functional directory for the unit tests, which contain additional directories for each namespace.

Software required

In order to run the unit tests, you will need the following Python packages/libraries installed:

  1. Mock
  2. Nose

On Ubuntu/Debian you can simple install the python3-mock and python3-nose packages. Most other distributions will also have these packages. On Windows and Mac OS X you will need to use pip or easy_install to install these packages.

Running the tests

See the Running Tests page on how to run the tests you've written.

See Also

Examples of GUI tests: http://www.voom.net/pyqt-qtest-example

Complete list of assert-methods (TestCase class): https://docs.python.org/3.3/library/unittest.html#assert-methods