Magic Tricks on Testing (Sandy Metz)

Based on https://www.youtube.com/watch?v=URSWYvyc42M

The talk gives a clear recommendation on which methods of an object are subject to unit tests, and which should be omitted.

An object, the Object under Test (OUT), is a regular class, containing methods (functions within objects are called methods).

The ruleset implies that a message (a.k.a method call) is either of type command or query, but never both.

Hint

There is a principle for separating commands and queries, called the Command–query separation (CQS) principle. It is generally not always advisable to strictly follow this rule, though.

A query is a call on a method which

• returns data to the caller

• does not alter the state of any object (no side-effects)

A command is a call on a method which

• returns no data

• alters the state of one or many objects (side-effects)

Furthermore, there are three origin types for messages, which define from where a method is called from the perspective of the OUT:

• methods, which are called by an outside object -> Incoming

• methods, which call methods from outside objects -> Outgoing

• methods, which call internal (private) methods -> Send-to-Self

In total then, there are six types of methods:

Type of messages types

Example module

This module feature three object, Wheel, Gear and Observer.

class Wheel:

    def __init__(self, rim, tire):
        self.rim = rim  # diameter of wheel rim
        self.tire = tire  # diameter of tire

    def diameter(self):
        return self.rim + (self.tire * 2)


class Observer:
    # e.g. writes data into the database

    def changed(self, key, value):
        # update database
        return True


class Gear:

    def __init__(self, chainring, cog, wheel, observer=Observer()):
        self.chainring = chainring  # length of chain ring
        self.cog = cog  # amount of cogs on wheel
        self.wheel = wheel  # Wheel instance
        self.observer = observer  # Observer instance

    def gear_inches(self):
        return self._ratio() * self.wheel.diameter()

    def _ratio(self):
        return self.chainring / float(self.cog)

    def set_cog(self, cog):
        self.cog = cog
        self._changed()

    def _changed(self):
        self.observer.changed(self.chainring, self.cog)

Incoming Query

Rule

✅ Test incoming query messages by making assertions about what they send back.

Incoming queries should always be tested, as it is our object, and we are responsible for its methods to return the correct value. So, we write one or multiple tests for it, asserting towards the returned value.

The diameter(self) method in Wheel is a subject to incoming queries, so we write a test for it, asserting to the return value:

class WheelTest(unittest.TestCase):

# GOOD: Test incoming query
def test_calculates_diameter(self):
    wheel = Wheel(26, 1.5)
    # assert the return value
    assert math.isclose(wheel.diameter(), 29, rel_tol=0.02)

The same applies to the gear_inches(self) method in Gear, so we write another test:

class GearTest(unittest.TestCase):

# GOOD: Test incoming query
def test_calculates_gear_inches(self):
    gear = Gear(52, 11, Wheel(26, 1.5))
    # assert the return value
    assert math.isclose(gear.gear_inches(), 137.1, rel_tol=0.01)

Incoming Commands

Rule

✅ Test incoming command messages by making assertions about direct public side effects.

Incoming commands should always be tested, as it is our object, and we are responsible for the direct public side-effects our methods have. So, we write one or multiple tests for it, asserting towards these side-effects.

The set_cog(self) method of the Gear class is a subject to incoming commands, as it changes the objects state (side-effect), but does not return a value. Consequentially, we write a unit test for it:

class GearTest(unittest.TestCase):

# ...

# GOOD: Test incoming command
def test_set_cog(self):
    gear = Gear(52, 11, Wheel(26, 1.5))
    gear.set_cog(27)
    # assert the direct public side effect
    assert 27 == gear.cog

Messages To Self (Query/Command)

Rule

❌ Do not test private methods.

Testing internal queries (assert return values) or commands (assert if certain set-method was called or value was changed) explicitly is redundant since these methods are already covered by other tests. Therefore, defining such tests do not improve the security of your code, but fail, if the code is improved, since it is bound to it.

The _ratio(self) method of class Gear is a Send-to-Self query, since it is a private method (Python marks private method with a preceding underscore). A test is redundant, since the method is already covered by the test_calculates_gear_inches(self) method, since gear_inches(self) is calling _ratio(self). No additional test is necessary.

This test is redundant:

class GearTest(unittest.TestCase):

# ...

# BAD: redundant test
def test_calculates_ratio(self):
gear = Gear(52, 11, Wheel(26, 1.5))
assert math.isclose(gear._ratio(), 4.7, rel_tol=0.01)

Additionally, if the internal implementation of _ratio(self) changes (or its name), this test breaks, so there is a maintenance effort for a test of no value.

Outgoing Query

Rule

❌ Do not test outgoing query messages.

An outgoing query of one object is always an incoming query of another object. Therefore asserting the return value of an outgoing query is already covered by the tests of that other object.

Don’t write tests for it, since they are redundant:

class GearTest(unittest.TestCase):

# ...

# BAD: test outgoing queries -> redundant test
def test_calculate_gear_inches_outgoing(self):
gear = Gear(52, 11, Wheel(26, 1.5))
diameter = gear.wheel.diameter()
assert math.isclose(diameter, 29, rel_tol=0.01)

Outgoing Commands

Rule

✅ Expect to send outgoing command messages (assert on method calls of outside object).

A unit test is supposed to test the object under test, not the effects it has on other distant objects -> this is then an integration test. The only thing to test is that outgoing command messages are sent, on other words, that these OUT methods have been called.

Testing an outgoing command against a real interface, which relies on a chain of events is not a good idea, since it easily breaks for reasons out of the unit tests scope.

class GearTest(unittest.TestCase):

    # ...

    # BAD: test against real interface (here: set_cog(27) calls an Observer instance)
    def test_saves_changed_cog_in_db(self):
        gear = Gear(chainring=52, cog=11, wheel=Wheel(26, 1.5),
                    observer=Observer())
        gear.set_cog(27)

Therefore instead of calling the real interface, use a mock object, which the internal outgoing command method calls instead, to have a stable edge. Remember: A mock is a test double which promises to have the same API as the real object (must stay in-sync).

from unittest import mock

class GearTest(unittest.TestCase):

    # GOOD: test against a mock, instead of a real out-of-scope interface
    def test_notifies_observers_when_cogs_changes(self):

        with mock.patch("unittest_magic_tricks.Observer") as MockObserver:
            wheel = Wheel(rim=26, tire=1.5)
            gear = Gear(chainring=52, cog=11, wheel=wheel,
                        observer=MockObserver)
            gear._changed()
            # assert that method was called once
            MockObserver.changed.assert_called_once()
            gear.set_cog(27)

Summary

Determine, which methods in an object are subject to which type of message.

Write tests according to this recommendation:

Unit Test Minimalist Overview

Refrain from writing additional tests, if those don’t add any value or are redundant.

Don’t test objects other than the object under test.

Refrain from calling outside objects, as errors in these objects will lead to the failure of your test, even though the error lies outside of the object under test.

Use mocks to check if outside methods are called to avoid unwanted side-effects, which breaks subsequent tests from failing.