Behold! Working software!

This is a common dialog in my house:

Me: Please brush your hair.

8yo: I did!

Me: Your hair is super tangled though. Please brush again.

8yo: I already brushed it!

Me: When I say "please brush your hair", I don't mean "run a brush through your hair", I mean "use a brush to ensure that your hair is disentangled to a point that further activity today will not cause knotting."

8yo: I am only 8 years old, and thus am ill equipped to parse that level of implicature from your utterances.

And on it goes, for "brush your teeth", "wash your hair", "clean your room", "eat your dinner".

This dialog is about software testing.

I was never taught the fundamentals of software testing, or at least not in the way that I understand them now. I was given platitudes and philosophies, and based on those I developed an intuitive sense about how testing actually fits within software product development, but I didn't examine that intuition. And, in hindsight, I also never verified that I could trust my own tests; the testing framework; the compiler; my local development machine.

That examination I never did is software testing. Without it, I wasn't brushing my software. I was running a brush through it.

Epistemology: keeping philosophers employed for millennia

I think it is paramount that any discussion of testing fundamentals avoid using domain terms like "unit" or "integration" or "functional", et al. Depending on your history and current context, the definitions for those words might vary from mine, and I think part of "fundamentals" is "making sure everyone is on the same page, ontologically speaking". So I'm just going to avoid ambiguous terms and instead lean on something much stronger.

That's right, I'm going to use epistemology.

You might push back with "but isn't that a heavy hammer"? I argue that the fundamental question of "why do we test software" is answered with "so we can know that our software does what we believe it does". That back-and-forth vibrates in my head very close to "how do we know what we know", which epistemology tries to answer.

I'm also leaning on terminology from epistemology so we can kind of trick ourselves into not thinking about how we tend to approach software testing, focusing instead on why. I don't want our thoughts to slip on a domain term-shaped banana peel and fall into a deeply-worn neural groove, unable to see the totality of the landscape from Way Down Here, so that we can remain free to examine testing objectively, holistically, and without bias.

When using terms from epistemology, I'll capitalize them, which might make the text feel awkward to read, but I think imparts a Germanic gravitas to the ideas presented.

Belief, Justification, and Knowledge

What is Knowledge? It's a question that has plagued philosophers for at least 35 years, and will likely continue to do so for at least as many years in the future.

An easy place to start is to say that Knowledge is a Belief that is: 1. true, and 2. well Justified. Those with more exposure to epistemology than myself (i.e. almost everyone) know that there's more to Knowledge than that, but for the purposes of definition and this particular article I think it's a fine starting point.

Consider the following:

1. Kelly sits at the front desk
2. Kelly always wears a bucket hat
3. Elizabeth is watching the front desk while Kelly is putting away office supplies
4. Elizabeth is wearing a bucket hat today

If I glance at the front desk in the morning and notice a bucket-hatted head at the desk, I might feel my Belief is Justified and that I Know that Kelly is in the office today. However, even though the Belief is true in this case (Kelly is in the office, after all), it's still not Knowledge because the Justification is coincidental.

I really like this example, as it is representative of a lot of times I've tested a piece of software and Justified a Belief by sheer coincidence. My Knowledge about the software isn't real, and is only true by accident.

Epistemological Lens

The fundamental utility of software testing is to provide an epistemic backdrop for asserting Knowledge on the software under test. Which is a pretentious way of saying "testing provides you confidence that your software does what you think it does."

It seems condescending to write what testing is, as if the idea is a novel one. Of course "testing is making sure software does what you think it does" -- anyone who has done any testing knows that maxim. But, let me describe some situations where I adhered to the letter of the law (i.e. "best practices" in a testing context) without actually creating any Knowledge about the software.

I have worked on many projects where I have answered the question "have you written tests" with one (or more) of the following:

• "Yes, for each class I implemented I also added a _test class that had assertions on each method exposed by the class."
• "Yes, the code coverage is 90%."
• "Yes, all of the code was test-driven."

To some extent, each of these examples probably provides a small amount of Knowledge about a specific piece of the system under test. If you've ever heard the phrase "it works on my machine" uttered, or uttered it yourself (throughout, say, your 25 years of experience in software product development, cough), you know that just because a test passes doesn't mean that the software is functioning as intended within the constraints it's supposed to.

So, what might a more epistemic approach to testing look like? I'm glad you asked. Or didn't. I'm going to talk about it anyway.

Building a chain of inference

My first software product development experience in a regulated environment was in 2011, and the product was a desktop application that configured an insulin pump that would deliver insulin into a user's body. What is interesting about insulin delivery from a testing perspective is that there is only One Right Answer, and the rest are Wrong Answers that range in impact from "the user doesn't feel quite right" to "the user no longer needs an insulin pump".

This project was my introduction to Installation Qualification, Operational Qualification, and Performance Qualification (IQ/OQ/PQ) specifications^[1]. These are documents that outline:

1. How to install a specific software unit for a specific purpose
2. What the specific software unit will be used for, and proof that it functions as expected
3. That the specific software unit will continue to operate correctly without degrading

For example, we used some very specific version of NUnit for this particular project, so we had an IQ/OQ document that described how to install it (this was before NuGet became the de facto dependency management utility for .NET projects); which operations NUnit would perform in the context of the project; and a suite of examples that proved that NUnit functioned as we expected to use it.

Probably this seems like overkill. I mean, why wouldn't you just assume that a library does what it says? If you're asking this question then I know you haven't spent any time developing for mobile platforms. Through the lens of developing Knowledge, this practice establishes that this version of this library, installed in this manner results in a Trusted Use of this functionality, which is very, very powerful.

This project had similar procedures for the operating system, the development framework, any software dependency, the build system, and more. The actual "software tests" were just the top rung of a ladder of epistemic inference, where each rung depended on the ones below it, and each were validated and Known.

Through this system of epistemic rigor, the team could answer the following questions confidently:

• Do we Know that our software will build correctly?
• Do we Know that our software will run?
• Do we Know that our software will maintain an accurate measurement of insulin, glucose, etc. despite what units are used to display those amounts?
• Do we Know that the insulin pump will always administer the correct dosage?

You can revisit earlier paragraphs in this section and see that the assertions that were made about the insulin pump configuration software held up because of a proven chain of inference:

• The measurements and units of insulin were correct because we wrote tests that exercised the conversion logic;
• The test executions were trusted because we validated the installation and operation of the testing framework on a specific version of an operating system;
• The specific version of the operating system was trusted because it was validated against a known set of hardware

So not only do you have to have the Right Test, i.e. the test that Justifiably reflects Beliefs about the software, but you also have to establish Knowledge about the context that the tests (and software) execute in. Otherwise you miss the implicit clause in the statement "a passing test [on my machine] implies a correct implementation; the test passes [on my machine], therefore the implementation is correct" -- a statement which, hopefully, is obviously suboptimal, given that a correct implementation is usually intended to run on more than just your machine.

Testing ain't free

Of course, if scaling an epistemic ladder were all there was to software testing, there wouldn't be thousands of schools of thought on how to test software.

The major challenge of software testing isn't necessarily validation or verification of the software itself, it's how to intelligently minimize the cost of validation and verification efforts. Given an amount of time (a schedule) and an amount of functionality (a scope), how do you allocate your time to testing activities?

At one extreme, you can do no testing whatsoever. This is bad^[2]. Similarly, spending 100% of your schedule doing nothing but testing should obviously be equally bad. See, one small thought experiment and we've already ruled out 0% and 100%, narrowing the number of choices from "uncountably infinite" to ... oh, it's still "uncountably infinite". Moving on, then.

Beyond the question of allocating time to testing activities, there's managing the interactions between test code and application code. Here are some examples from my own experience:

• Better architectural practices make tests easier to write, and vice versa
• Testing state (rather than behavior) makes it difficult to change or simplify architecture
• To maximize "percent of software covered by test, per time spent writing a test", write an end-to-end test
• End-to-end tests tend to exercise the software from the points where it would interact with a human, and the frameworks that can automate these tests tend to be garbage
• End-to-end tests tend to take a comparatively large amount of time to execute
• All you have to do to hit a target code coverage metric is to exercise the target amount of the software, but eschew assertions^[3]
• Taking care to minimize the runtime of test execution increases the likelihood that developers will actually run the tests and catch regression errors earlier in development

For example, you can certainly minimize the amount of time allocated to writing tests if you focus on end-to-end tests, but that will increase the amount of time spent waiting for tests to execute on the developer machines and in the CI/CD pipeline. The amount of time (and frustration) spent agonizing over flaky tests^[4] will also increase.

From the ladder, to the allocation of time, to the attention paid to interactions, we arrive at a definition:

Fundamentally, Software Testing is the practice of pragmatically navigating the tension between:

• The need for Knowledge: a rigorous set of exercises that demonstrate correct functioning of a software product, per a set of known requirements
• The need for Delivery: actually developing and delivering software and not just spending all day writing tests
• The need for Habitability: constructing your tests in a way that doesn't impede your ability to add, alter, or maintain production code

But arriving at a definition is the easy part^[5]. The actual practice, for a specific project or product, is much more difficult, and the constraints underlying that project can shift during execution. With a solid foundation now underneath us, however, we can turn to all the books and articles on Software Testing and feel confident in our ability to apply them to our particular context.

Building up tools/perspectives

If you have spent any time whatsoever reading about software testing, you have encountered two specific kinds of content:

1. A bunch of theoretical constructs you can use to frame your overall testing strategy, such as "the Agile Testing Quadrants", "the Test Pyramid", or "the Funky Rhombus of Software Solidity"^[6]
2. Documentation for writing tests using a specific library (xUnit, etc.) or within a specific framework (.NET MAUI, React Native, etc.)

Yes, these are abstractions, and we know that abstractions obscure what we're trying to do: build Knowledge via Justification. But they are absolutely worth learning, and worth learning well. The philosophy gets you a foundation, but the rest you have to build yourself, and if you don't add tools to your toolbox you'll find building quite difficult. To state it another way: time is money, and we can use tools in our toolbox to construct a cost-effective approach that ensures we're building tests that provide epistemic Knowledge about the software under test. But the tools have to be there first.

Don't, however, forget that frameworks and libraries are means to an end: Knowledge. Taken in isolation, it's so easy to look at a theoretical framing and believe you're using it correctly. But, without mindfully using tests to build up Knowledge about your application, you're doing yourself a disservice. In my first attempt at the previous sentence, I wrote "you're just spinning your wheels", but that's inaccurate, because "spinning your wheels" implies expending effort while not moving. If you're misapplying a testing theory you're also misinformed about the operation of your software, and likely impacting the habitability of your application code. You're expending effort while moving backwards.^[7]

Testing tension: tended to

Every single testing tool, guideline, philosophy, or vocab term that has emerged to the level of Best Practice is a high-level attempt to manage the tension between Knowledge, Delivery, and Habitability. These Best Practices also have the unintentional side-effect of abstracting away the fundamentals and putting us at a remove from the First Principles of software testing.

Let's look at a quick before-and-after to get a visceral sense of applying Best Practices with and without this tension in mind. I was working on a server-side Java application back when that was a thing people did. We had oodles of unit tests, per "Best Practices", but they were just covered in mocks. Every dependency in a class was mocked in the test. The approach seemed reasonable, but for some reason it just wasn't very comforting, seeing all those tests pass.

Now we have a vocabulary to really reach in and see what's going on. The Justification we were creating was circular. Your function is supposed to call RebateFactory.getRebate^[8] and deal with the result, and you Believe it does that because you wrote the function and you wrote the test and you wrote the mock. But, since it's a mock, it doesn't know that the actual RebateFactory returns cents; the mock behavior you set up assumed dollars. Oops. That's an expensive lesson.

Further, all those mocks are locking down implementation, not behavior, which severely affects Habitability (and, eventually, Delivery). Maybe you decide that RebateFactory should actually be two different classes, as it's been getting larger and taking on too much responsibility. You split it, creating AbstractCouponManagerSingleton (teehee), and ... eight thousand failing tests? You know what, maybe RebateFactory is fine the way it is, just growing and growing.

Back to my Java project. I didn't have this epistemic view then, but I did have a vague feeling that the mocks were not really leading anywhere good. So we got rid of them. Instead, we used the Spring integration tests, injecting actual implementations as dependencies. Spring invoked our actual code with real HTTP request objects (with fake data) and the only thing we mocked was whatever was happening at the data layer, capturing how our app would interact with it. Known inputs, known outputs, and real interactions in the middle: Knowledge.

Even now, looking back on it, I feel the comfort that brought me. But I also see that our Knowledge at the data layer may not have been perfectly well Justified. It's enough to tear one's hair out, if I had any left.

But, just because I'm bald doesn't mean I don't know how to brush my hair.