The ocelot game works like this: a developer has written a feature, and presents it to an ocelot.
Not a real ocelot. Ocelots are playful. But they’re wild animals. Their claws easily poke holes in things. We want to poke holes in code.
The ocelot is a role played by a developer who tries to poke holes in another developer’s code. Specifically, they try to change the code in such a way that the feature breaks while still passing the unit tests.
If the ocelot can find a way to do this that isn’t a purely IO related detail covered by integration tests, we’ve found a problem. The ocelot makes the breaking change and pushes it.
You might think the problem is about improving testing. But testing – as we know – isn’t mostly about testing. It’s about design!
The Prey
What the ocelot really wants to sink her claws into is an untestable design. The ocelot’s real prey isn’t a missing unit test. That’s a great find! But it is not the point.
No – the ocelot wants to corner the developer: “there’s no way to test this”.
The admission of untestability is the victory of the ocelot.
If software is untestable, it is broken by design. Michael Feathers has a great talk on this. I’ve never seen a well designed feature that could not be unit tested.
The Hunting Grounds
The ocelot is particularly attuned to certain patterns. The developer should present the interfaces and boundaries for their feature.
The ocelot knows where the prey tend to hide: in the code that sneaks around the interfaces.
Why not mutation testing?
Mutation testing is great, and certainly belongs in the ocelot’s bag of tricks.
But some systems are so lacking in cohesion that they are difficult to mutation test.
Think of a distributed monolith, where microservices are coupled at the database level. A developer has to make changes by running multiple services at a time, and testing that they work – these tests are often manual, and only verify the system works (maybe, for now).
Or think of a system where we have a SPA and an RPC backend. (Maybe we try to dignify it by incorrectly calling it a REST API.) The feature may well involve both systems. But without running them, it’s difficult to show how they break.
The point is that systems that exhibit tight coupling may be hard to mutation test in the first place, since the behavior is strewn across the system.
Clojure does have a mutation testing library, but I have not found that it fits my needs.
What About IO?
What about
- Inserting a record into the database
- Inserting a record into Kafka
Can the ocelot win by showing the unit tests don’t detect changes at the database level?
No – assuming the IO is relegated to a distinct data layer. Putting data concerns behind an interface and having a single integration test will protect you from the Ocelot.
What About PR Reviews?
In my experience, PR reviews have a genuine but limited effect.
But if they work for you, by all means continue. If:
- You are pushing code to production at least daily
- Most months you don’t have user-facing defects
- Your system is easy to change
Then keep doing what you’re doing!
In general, though, feature branching is an anti-pattern predictive of lower performance. Trunk based development has far superior empirical support.
What about Pair Programming?
Pair programming suffers from the fact that it is not adversarial. Or worse – it really is adversarial!
Much better to encourage a critical approach that uses a sense of playfulness to spot problems without conflict. We want to create the mindset that looks for problems and avoids the herd mentality.
More to the point, as Rich Hickey says in Simple Made Easy: “Your sensibilities about simplicity being equal to ease of use are wrong. They’re just simply wrong. Right?”
Two people with the wrong sensibilities might be worse than one. The ocelot game is for teams that do not have a solid grasp on the lessons in Simple Made Easy. What we want is some way to start prove that we have complected systems.
Why play the ocelot game?
Not just because it is fun. But because it is usually ineffective to preach simplicity to the unconvinced. Developers can look at a textbook example of a tightly coupled system and say: “doesn’t look coupled to me.”
But tightly coupled systems cause businesses to be overtaken by competitors. We want to work on new features, but the weight of an unmaintainable system strangles our productivity. The best of intentions do not translate to good results.
We want to identify design defects immediately and in an actionable way.
So – ocelots – break the feature and push it.
Prior Work
I have tried to determine if this idea has already been proposed. It contains elements of red team/blue team practices, as well as mutation testing. After searching, I couldn’t find anything quite like this. I’m sure someone has thought it up before – and maybe the idea resurfaced in my brain without its original source. If you recognize this – message me.