If you haven't read through Test-Driven Development yet, go and read that first. That will explain some of the reasoning behind writing tests.
A very basic test case (or test suite as it is sometimes called), with two tests:
from unittest import TestCase from openlp.core.lib import str_to_bool class TestLibModule(TestCase): def str_to_bool_with_true_bool_test(self): """ Test the str_to_bool function with True input """ # GIVEN: A boolean value set to true true_boolean = True # WHEN: We "convert" it to a bool true_result = str_to_bool(true_boolean) # THEN: We should get back a True bool self.assertIsInstance(true_result, bool, 'The result should be a boolean') self.assertTrue(true_result, 'The result should be True') def str_to_bool_with_false_bool_test(self): """ Test the str_to_bool function with False input """ # GIVEN: A boolean value set to false false_boolean = False # WHEN: We "convert" it to a bool false_result = str_to_bool(false_boolean) # THEN: We should get back a True bool self.assertIsInstance(false_result, bool, 'The result should be a boolean') self.assertFalse(false_result, 'The result should be True')
- We have a class which holds a set of tests related to each other.
- Each method in the class is a test (you can see, it ends with "_test").
- In order for a method to be recognised as a test, it needs to end with "_test".
- Each test is divided into 3 sections: GIVEN, WHEN, THEN:
- GIVEN: These are the things we start with
- WHEN: This is what happens
- THEN: These are the results
- For simplicity's sake I have two tests which each have 1 GIVEN-WHEN-THEN combination. As a general rule, if you have more than one WHEN, then you probably need to split your test into smaller tests. In some cases this is not possible, but this is a good rule of thumb.
I've called this class
TestLibModule, however it will usually have a name related to the module being tested.
To actually run the tests I am using a python test running framework called nosetests (or just nose). You'll see that in the tests I use the assert-methods of the TestCase-class to determine that my results are what I expect.
nose will catch the AssertionError exceptions that these assert-methods raise if the statement is not as expected, and will record that test as having failed. The failure-message is an optional argument of the assert-methods, if none is provided a default is used.
You'll see this test is passing a boolean into a function that looks like it should take a string. Tests should check to see how a piece of code behaves depending on what input it receives.
In this function's case, it is able to take a boolean so I test that it can take a boolean value and will return the correct result. Here is another test for the same function:
def str_to_bool_with_int_test(self): """ Test the str_to_bool function with an integer input """ # GIVEN: An integer value int_string = 1 # WHEN: we convert it to a bool int_result = str_to_bool(int_string) # THEN: we should get back a false self.assertFalse(int_result, 'The result should be False') def str_to_bool_with_invalid_string_test(self): """ Test the str_to_bool function with an invalid string """ # GIVEN: An string value with completely invalid input invalid_string = u'my feet are wet' # WHEN: we convert it to a bool str_result = str_to_bool(invalid_string) # THEN: we should get back a false self.assertFalse(str_result, 'The result should be False')
This one tests what happens if I send in invalid input.
Right, so now we have this other function, and it calls some other function - how do I know that I'm testing just this function and not the other function I'm calling?
Quite simply, we use a fantastic library called "mock" which is able to replace functions, methods and classes with our own magic versions that don't do anything.
One of the classes that can be used to replace other classes and functions is the MagicMock class.
from unittest import TestCase from tests.functional import MagicMock, patch from openlp.core.lib import translate, check_directory_exists class TestLibModule(TestCase): def translate_test(self): """ Test the translate() function """ # GIVEN: A string to translate and a mocked Qt translate function context = u'OpenLP.Tests' text = u'Untranslated string' comment = u'A comment' encoding = 1 n = 1 mocked_translate = MagicMock(return_value=u'Translated string') # WHEN: we call the translate function result = translate(context, text, comment, encoding, n, mocked_translate) # THEN: the translated string should be returned, and the mocked function should have been called mocked_translate.assert_called_with(context, text, comment, encoding, n) self.assertEqual(result, 'Translated string', 'The translated string should have been returned')
The mocked function pretends it is the function we are calling, returns a fixed value and keeps track of what it was called with so it can be checked later.
The translate() function above is unusual in that the function called is passed in. In most cases this doesn't have. This is where the "patch()" method comes in.
Using The patch Function
The patch() method patches an object and inserts itself into the place of the object you want to mock out.
Here is another test, this one uses the patch() function
def check_directory_exists_test(self): """ Test the check_directory_exists() function """ # GIVEN: A directory to check and a mocked out os.makedirs and os.path.exists with patch(u'openlp.core.lib.os.path.exists') as mocked_exists, \ patch(u'openlp.core.lib.os.makedirs') as mocked_makedirs: directory_to_check = u'existing/directory' # WHEN: os.path.exists returns True and we check to see if the directory exists mocked_exists.return_value = True check_directory_exists(directory_to_check) # THEN: Only os.path.exists should have been called mocked_exists.assert_called_with(directory_to_check) self.assertFalse(mocked_makedirs.called, 'os.makedirs should not have been called')
So what I'm doing is I'm actually substituting os.path.exists() and os.makedirs() for my own mock methods, and then when I call the function it calls my mocks instead of the real functions.
This way we can completely control all the data around a function and check that it reacts accordingly. These are proper unit tests, they test the smallest possible unit (or block) of code.
We won't always be able to do this level of testing, but this is a good way to get used to writing tests
Notice when you patch a method, you need to patch it where it is imported. So you don't just patch "os.path.exists", you need to patch "openlp.core.lib.os.path.exists".
Writing Your Tests
When writing a set of tests, the easiest way to do it is to start with the shortest path that the code takes.
For instance, let's look at this function:
def validate_thumb(file_path, thumb_path): """ Validates whether an file's thumb still exists and if is up to date. **Note**, you must **not** call this function, before checking the existence of the file. ``file_path`` The path to the file. The file **must** exist! ``thumb_path`` The path to the thumb. """ if not os.path.exists(thumb_path): return False image_date = os.stat(file_path).st_mtime thumb_date = os.stat(thumb_path).st_mtime return image_date <= thumb_date
The shortest path through this function would happen if os.path.exists() returned False. So let's write a test which mocks out os.path.exists and makes it return False.
def validate_thumb_file_does_not_exist_test(self): """ Test the validate_thumb() function when the thumbnail does not exist """ # GIVEN: A mocked out os module, with path.exists returning False, and fake paths to a file and a thumb with patch(u'openlp.core.lib.os') as mocked_os: file_path = u'path/to/file' thumb_path = u'path/to/thumb' mocked_os.path.exists.return_value = False # WHEN: we run the validate_thumb() function result = validate_thumb(file_path, thumb_path) # THEN: we should have called a few functions, and the result should be False mocked_os.path.exists.assert_called_with(thumb_path) self.assertFalse(result, 'The result should be False')
Triggering and Handling Exceptions
Sometimes you need to check that a function handles exceptions properly. To do this you need to both raise an exception and then check that the exception was handled correctly in the code. Once again, the mock library comes to our rescue.
To raise an exception, simply mock out the method that should raise the exception, and then add a *side effect* to the method. If the function you're testing is supposed to raise an exception itself, or allow one to bubble up, then you can assert that the exception is raised using the self.assertRaises() method from the TestCase class.
def check_directory_exists_test(self): """ Test the check_directory_exists() function """ # GIVEN: A directory to check and a mocked out os.makedirs and os.path.exists with patch(u'openlp.core.lib.os.path.exists') as mocked_exists, \ patch(u'openlp.core.lib.os.makedirs') as mocked_makedirs: directory_to_check = u'existing/directory' # WHEN: Some unhandled exception is raised mocked_exists.side_effect = ValueError() # THEN: check_directory_exists raises an exception mocked_exists.assert_called_with(directory_to_check) self.assertRaises(ValueError, check_directory_exists, directory_to_check)
Write Thorough Tests
- Check what you got back from the function and that it was what you expected.
- Test each code path, start with the shortest.
- Check the mocked functions that you expected to be called were called.
- Check they were called with the expected parameters.
An example of writing a test for the get_filesystem_encoding() function in openlp/core/utils/__init__.py
from unittest import TestCase from mock import MagicMock, patch from openlp.core.utils import get_filesystem_encoding class TestLib(TestCase): def get_filesystem_encoding_cp1252_test(self): """ Test the get_filesystem_encoding() function with an encoding of cp1252 """ # GIVEN: sys.getfilesystemencoding returns "cp1252" with patch(u'openlp.core.utils.sys.getfilesystemencoding') as mocked_getfilesystemencoding, \ patch(u'openlp.core.utils.sys.getdefaultencoding') as mocked_getdefaultencoding: mocked_getfilesystemencoding.return_value = u'cp1252' # WHEN: get_filesystem_encoding() is called result = get_filesystem_encoding() # THEN: getdefaultencoding should have been called mocked_getfilesystemencoding.assert_called_with() self.assertFalse(mocked_getdefaultencoding.called) self.assertEqual(result, 'cp1252', 'The result should be "cp1252"') def get_filesystem_encoding_cp1252_test(self): """ Test the get_filesystem_encoding() function with an encoding of cp1252 """ # GIVEN: sys.getfilesystemencoding returns None and sys.getdefaultencoding returns "utf-8" with patch(u'openlp.core.utils.sys.getfilesystemencoding') as mocked_getfilesystemencoding, \ patch(u'openlp.core.utils.sys.getdefaultencoding') as mocked_getdefaultencoding: mocked_getfilesystemencoding.return_value = None mocked_getdefaultencoding.return_value = u'utf-8' # WHEN: get_filesystem_encoding() is called result = get_filesystem_encoding() # THEN: getdefaultencoding should have been called mocked_getfilesystemencoding.assert_called_with() mocked_getdefaultencoding.assert_called_with() self.assertEqual(result, 'utf-8', 'The result should be "utf-8"')
Location of tests
Tests are located under the
tests directory. Inside this is a
functional directory for the unit tests, which contain additional directories for each namespace.
In order to run the unit tests, you will need the following Python packages/libraries installed:
On Ubuntu/Debian you can simple install the python3-mock and python3-nose packages. Most other distributions will also have these packages. On Windows and Mac OS X you will need to use pip or easy_install to install these packages.
Running the tests
See the Running Tests page on how to run the tests you've written.
Examples of GUI tests: http://www.voom.net/pyqt-qtest-example
Complete list of assert-methods (TestCase class): https://docs.python.org/3.3/library/unittest.html#assert-methods