2. Writing tests

We are using pytest in order to write our tests.

2.1. How to run

Before running the tests, make sure the dependencies in requirements-dev.txt are installed. Running the tests will clean your mongodb and remove every extensions, so make sure you made a backup if needed. Tests must go inside tests/ folder.

You might need to install chromedriver (chrome driver) or geckodriver (Firefox driver).

To run the tests, the simplest way is to execute pytest tests/ --driver [Firefox|Chrome] to launch the whole test suite. If you want to only execute the tests included in tests/test_foo.py, then you can use the command pytest tests/test_foo.py --driver [Firefox|Chrome].

If you have any problems with the tests, it might be because they are launched with python2 instead of python3. In this case, instead of running pytest [args], run python3 -m pytest [args]

2.2. Test architecture

We can divide our testing architecture into three parts.

2.2.1. Travis

We are using travis for continuous integrations. Each time a commit is pushed on master, travis will install everything tozti needs to run and also execute our test suite. This enables us to see in a glance if the modifications we made are breaking something.

2.2.2. Unit testing

Unit testing is one of the two types of tests we have. It consists in testing some functions independently from all others. As they are very long to write, only the topological sort and the mechanism to find extensions have unit tests.

2.2.3. Integration testing

Integration tests do not target a single functions (like unit tests), but tests the behaviour of the whole of Tozti. We use intensive integration testing in order to test the storage, the router and the js-router mechanisms.

Here, a test consists of:

  • loading an extension
  • launching tozti
  • testing one functionality
  • closing tozti

2.3. Writing tests for Tozti with pytest

Writing a test with pytest is easy. First, you need to create a python file prefixed by tests_ which will contain your test function. A test file can contain several test functions and should import pytest. A test function must have a name starting with test_. Its name must be explicit as it will be displayed upon test failure. Finally, a test fails if it raises an exception. Assertions are convenient when writing tests. assert expr will do nothing if expr evaluates to False, and will fail otherwhise.

A test should:

  • not rely on data not defined inside of the test. If you have 10 tests, then the result of executing the test must be independent of the order in which they are executed
  • be precise and test only one thing
  • change rarely. Each time you edit a test it looses part of its purpose.

2.3.1. Passing parameters to a test

Most of the time your tests will not take any arguments and will be self contained. But sometimes, you will want to write a generic test and use it on different inputs and outputs. For exemple, to test a function foo that takes two arguments and computes their sum, you could do this:

def test_foo_1():
    assert(foo(3, 4) == 7)
def test_foo_2():
    assert(foo(0, 4) == 4)

This is correct, but it is more convenient to write:

@pytest.mark.parametrize("a, b, expected", [(3, 4, 7), (0, 4, 4)])
def test_foo(a, b, expected):
    assert(foo(a, b) == expected)

Here, we are parameterizing the test over the arguments a, b and expected.

2.3.2. The notion of Fixture

You may want to execute something before and after your test. For instance launch a background process, initialize the connection to a database and make sure it is correctly closed. That is what Pytest’s fixture is for. I will not dwell on it too much, as there are several online ressources.

You can find one particularly useful fixture in the file tests/conftest.py. This fixture, called tozti, will:

  • install a series of extension if needed
  • launch tozti
  • execute Tozti
  • completly close Tozti

The following is a simple example of how to use it:

@pytest.mark.extensions(["extension1", "extension2", ...., "extensionn"])
def test_ultra_super_genial(tozti):
    test_something

The line @pytest.mark.extensions(...) is used to specify the names of the extensions which should be installed. The extensions themselves must be put in the folder tests/extensions. You can then use the object tozti (which is a subprocess.Popen object if tozti could be launched, None otherwise) in order to perform some operations. For exemple, the function tozti_still_running(tozti) returns True if tozti is still running.

Other fixture are also present in tests/conftest.py. The fixture db will load a mongodb database and empty it before the test for exemple. Notice that importing tests/conftest.py is not needed in order to use the fixtures as this file is automatically loaded by pytest. As such you should beware with defining variables named db or tozti. You can find other usefull fonctions inside tests/commons.py. To include a function defined there (for exemple tozti_still_running), please add the following line:

from tests.commons import tozti_still_running