Advice to avoid with-redefs often focuses on the wrong things. It is true that with-redefs can cause race conditions in your test suite. And that functions can be captured in closures, leading to them not being redefined as one might expect. And that with-redefs doesn’t redefine inlined functions. And that with-redefs can cause problems with type-hinted functions. And that they cause problems when used with macros. And that mutating the global environment is something functional programmers should know is a bad practice. Etc., etc.
This does not touch on the heart of the problem. Good software is loosely coupled. Good tests do not break when the implementation changes while the behavior does not.
Let’s look at an example. In an application with an imperative architecture, we are likely to have a handler, a “business logic” layer, some pure helper functions, and an infrastructure layer.
Side-effecting functions are red. “Business logic” is used in the common, unfortunate sense of “whatever happens between the handler and the database”. It does not signify a model of a business process.
If we are going to unit test the “business logic” function, we run into a problem: the business logic function is directly coupled to the infrastructure layer. This leads developers who are starting to learn how to write tests to do one of two things:
- Rely on integration tests.
- Use
with-redefs
Experienced developers have certainly discovered what is at the end of the first road. It leads to test failures that do not point immediately to the problem. Instead, time is needed to debug tests! It introduces a combinatorial explosion of scenarios to test. And the suite test suite gets slower and slower.
with-redefs seems like a way to speed things up. Instead of hitting a database, we can redefine the database function for our tests. We might even mistake it for a unit test. But now the tests run faster, so we call it a win.
The problem is that our tests don’t just depend on the behavior of our “business” layer. The tests break encapsulation and depend directly on implementation details.
Suppose we refactor our business layer to call a different function in the infrastructure layer. Our tests would either hit the database or break. Or suppose we change the function signature of the database function. Our test might pass, even though we introduced a bug.
Nor does with-redefs mitigate the fundamental flaws of integration testing, with the sole exception of test suite speed. The combinatorial explosion of scenarios persists. It is unlikely that anything but a small sliver of the state space could be tested. Tests don’t give feedback on design. They fail to make complected code painful to write and test.
It’s often said that tests introduce technical debt. It seems likely to me that this judgment arises from writing bad tests, and this in turn arises from writing software that lacks encapsulation. The brittleness with-redefs introduces to a test suite is a perfect example. Using with-redefs will give you higher maintenance costs and lower confidence in your tests. It is a sign not only of a problem that arises from writing tests, but of fundamental design mistakes.
This is not to say there is no role for with-redefs. with-redefs can be a useful technique for getting legacy code under test. It is similar to the “link substitution” pattern Michael Feathers describes in Working Effectively with Legacy Code. with-redefs also works well in the context of mutation testing.
Think carefully before you use with-redefs: are you writing tomorrow’s legacy code today?