If you haven't read through Test-Driven Development yet, go and read that first. That will explain some of the reasoning behind writing tests.
Here are two basic tests:
from openlp.core.lib import str_to_bool def test_str_to_bool_with_true_bool(): """ Test the str_to_bool function with True input """ # GIVEN: A boolean value set to true true_boolean = True # WHEN: We "convert" it to a bool true_result = str_to_bool(true_boolean) # THEN: We should get back a True bool assert isinstance(true_result, bool), 'The result should be a boolean' assert true_result is True, 'The result should be True' def test_str_to_bool_with_false_bool(): """ Test the str_to_bool function with False input """ # GIVEN: A boolean value set to false false_boolean = False # WHEN: We "convert" it to a bool false_result = str_to_bool(false_boolean) # THEN: We should get back a False bool assert isinstance(false_result, bool), 'The result should be a boolean' assert false_result is False, 'The result should be False'
- Each function is a test
- In order for a function to be recognised as a test, it needs to start with "test_"
- Each test is divided into 3 sections: GIVEN, WHEN, THEN:
- GIVEN: These are the things we start with
- WHEN: This is what happens
- THEN: These are the results
- For simplicity's sake I have two tests which each have 1 GIVEN-WHEN-THEN combination. As a general rule, if you have more than one WHEN, then you probably need to split your test into smaller tests. In some cases this is not possible, but this is a good rule of thumb.
To actually run the tests we use the
pytest test runner and framework. The test actually match the
unittest style due to our history, rather than
pytest style tests, but
pytest happily runs that style of tests.
You'll see that in the tests I use the assert keyword to determine that my results are what I expect.
pytest will catch the AssertionError exceptions that these assert statements raise if the statement is not as expected, and will record that test as having failed. The failure message is an optional argument of the assert methods.
You'll see this test is passing a boolean into a function that looks like it should take a string. Tests should check to see how a piece of code behaves depending on what input it receives.
In this function's case, it is able to take a boolean so I test that it can take a boolean value and will return the correct result. Here is another test for the same function:
def test_str_to_bool_with_int(): """ Test the str_to_bool function with an integer input """ # GIVEN: An integer value int_string = 1 # WHEN: we convert it to a bool int_result = str_to_bool(int_string) # THEN: we should get back a false assert int_result is False, 'The result should be False' def test_str_to_bool_with_invalid_string(): """ Test the str_to_bool function with an invalid string """ # GIVEN: An string value with completely invalid input invalid_string = 'my feet are wet' # WHEN: we convert it to a bool str_result = str_to_bool(invalid_string) # THEN: we should get back a false assert str_result is False, 'The result should be False'
This one tests what happens if I send in invalid input.
In order for
pytest to pick up the tests, they need to be named correctly. In the above examples I've named them according to the convention you need to use. Briefly, it is this:
- Test functions should be named
- Test case classes should be named
Test<ClassName>(you'll see some of these around, though they're not required)
Right, so now we have this one function, and it calls some other function - how do I know that I'm testing just this function and not the other function I'm calling?
Quite simply, we use a fantastic library called "mock" which is able to replace functions, methods and classes with our own magic versions that don't do anything. We can also make these magic versions act a particular way or return certain results. Mock is part of the unittest module.
One of the classes that can be used to replace other classes and functions is the MagicMock class.
from unittest.mock import MagicMock, patch from openlp.core.lib import translate, check_directory_exists def test_translate(): """ Test the translate() function """ # GIVEN: A string to translate and a mocked Qt translate function context = 'OpenLP.Tests' text = 'Untranslated string' comment = 'A comment' encoding = 1 n = 1 mocked_translate = MagicMock(return_value='Translated string') # WHEN: we call the translate function result = translate(context, text, comment, encoding, n, mocked_translate) # THEN: the translated string should be returned, and the mocked function should have been called mocked_translate.assert_called_with(context, text, comment, encoding, n) assert result == 'Translated string', 'The translated string should have been returned'
The mocked function pretends it is the function we are calling, returns a fixed value and keeps track of what it was called with so it can be checked later.
The translate() function above is unusual in that the function called is passed in. In most cases this doesn't have. This is where the "patch()" method comes in.
Using The patch Function
The patch() method patches an object and inserts itself into the place of the object you want to mock out.
Here is another test, this one uses the patch() function
def test_check_directory_exists(): """ Test the check_directory_exists() function """ # GIVEN: A directory to check and a mocked out os.makedirs and os.path.exists with patch('openlp.core.lib.os.path.exists') as mocked_exists, \ patch('openlp.core.lib.os.makedirs') as mocked_makedirs: directory_to_check = 'existing/directory' # WHEN: os.path.exists returns True and we check to see if the directory exists mocked_exists.return_value = True check_directory_exists(directory_to_check) # THEN: Only os.path.exists should have been called mocked_exists.assert_called_with(directory_to_check) assert mocked_makedirs.called is False, 'os.makedirs should not have been called'
So what I'm doing is I'm actually replacing
os.makedirs() with my own mock methods, and then when I call the function it calls my mocks instead of the real functions.
This way we can completely control all the data around a function and check that it reacts accordingly. These are proper unit tests, they test the smallest possible unit (or block) of code.
We won't always be able to do this level of testing, but this is a good way to get used to writing tests
Writing Your Tests
When writing a set of tests, the easiest way to do it is to start with the shortest path that the code takes.
For instance, let's look at this function:
def validate_thumb(file_path, thumb_path): """ Validates whether an file's thumb still exists and if is up to date. **Note**, you must **not** call this function, before checking the existence of the file. ``file_path`` The path to the file. The file **must** exist! ``thumb_path`` The path to the thumb. """ if not os.path.exists(thumb_path): return False image_date = os.stat(file_path).st_mtime thumb_date = os.stat(thumb_path).st_mtime return image_date <= thumb_date
The shortest path through this function would happen if
os.path.exists() returned False. So let's write a test which mocks out
os.path.exists() and makes it return
def test_validate_thumb_file_does_not_exist(): """ Test the validate_thumb() function when the thumbnail does not exist """ # GIVEN: A mocked out os module, with path.exists returning False, and fake paths to a file and a thumb with patch('openlp.core.lib.os') as mocked_os: file_path = 'path/to/file' thumb_path = 'path/to/thumb' mocked_os.path.exists.return_value = False # WHEN: we run the validate_thumb() function result = validate_thumb(file_path, thumb_path) # THEN: we should have called a few functions, and the result should be False mocked_os.path.exists.assert_called_with(thumb_path) assert result is False, 'The result should be False'
Triggering and Handling Exceptions
Sometimes you need to check that a function handles exceptions properly. To do this you need to both raise an exception and then check that the exception was handled correctly in the code. Once again, the mock library comes to our rescue.
To raise an exception, simply mock out the method that should raise the exception, and then add a *side effect* to the method. If the function you're testing is supposed to raise an exception itself, or allow one to bubble up, then you can assert that the exception is raised using the
pytest.raises() context manager.
import pytest def test_check_directory_exists(): """ Test the check_directory_exists() function """ # GIVEN: A directory to check and a mocked out os.makedirs and os.path.exists with patch('openlp.core.lib.os.path.exists') as mocked_exists, \ patch('openlp.core.lib.os.makedirs') as mocked_makedirs: directory_to_check = 'existing/directory' # WHEN: Some unhandled exception is raised mocked_exists.side_effect = ValueError() # THEN: check_directory_exists raises an exception mocked_exists.assert_called_with(directory_to_check) with pytest.raises(ValueError): check_directory_exists(directory_to_check)
Write Thorough Tests
- Check what you got back from the function and that it was what you expected.
- Test each code path, start with the shortest.
- Check the mocked functions that you expected to be called were called.
- Check they were called with the expected parameters.
An example of writing a test for the
get_filesystem_encoding() function in
from mock import MagicMock, patch from openlp.core.utils import get_filesystem_encoding def test_get_filesystem_encoding_cp1252(): """ Test the get_filesystem_encoding() function with an encoding of cp1252 """ # GIVEN: sys.getfilesystemencoding returns "cp1252" with patch('openlp.core.utils.sys.getfilesystemencoding') as mocked_getfilesystemencoding, \ patch('openlp.core.utils.sys.getdefaultencoding') as mocked_getdefaultencoding: mocked_getfilesystemencoding.return_value = 'cp1252' # WHEN: get_filesystem_encoding() is called result = get_filesystem_encoding() # THEN: getdefaultencoding should have been called mocked_getfilesystemencoding.assert_called_with() assert mocked_getdefaultencoding.called is False assert result == 'cp1252', 'The result should be "cp1252"' def test_get_filesystem_encoding_cp1252(): """ Test the get_filesystem_encoding() function with an encoding of cp1252 """ # GIVEN: sys.getfilesystemencoding returns None and sys.getdefaultencoding returns "utf-8" with patch('openlp.core.utils.sys.getfilesystemencoding') as mocked_getfilesystemencoding, \ patch('openlp.core.utils.sys.getdefaultencoding') as mocked_getdefaultencoding: mocked_getfilesystemencoding.return_value = None mocked_getdefaultencoding.return_value = 'utf-8' # WHEN: get_filesystem_encoding() is called result = get_filesystem_encoding() # THEN: getdefaultencoding should have been called mocked_getfilesystemencoding.assert_called_with() mocked_getdefaultencoding.assert_called_with() assert result == 'utf-8', 'The result should be "utf-8"'
Location of tests
Tests are located under the
tests directory. Historically tests were divided into
interfaces tests, but we no longer make those distinctions.
In order to run the unit tests, you will need to install
pytest. On most Linux distributions you need to install the
python3-pytest package. On Windows and macOS you should be able to just use
pip to install
Running the tests
See the Running Tests page on how to run the tests you've written.