Interlude: Test data randomization

In a recent blog post I said some unkind things about randomized test data. Or I guess more specifically about some people who had used randomized data to drive the design of their application through misapplied TDD.

Don't get me wrong: I have nothing against the practice of randomizing test data. It's a practice like any other and I'm sure some day I might find a valid use case for it. But, as mentioned in the blog post, there are risks that come with test data randomization that shouldn't be ignored – risks that can (and do) result in actual money being spent chasing down failures that otherwise wouldn't have happened.

So in this interlude I want to discuss two things: "datasets" and a practice called Test Data Management.

Datasets

Sometimes you have a test method and you want to run it against several different inputs, such as:

A QA engineer walks into a bar. Orders a beer. Orders 0 beers. Orders 99999999999 beers. Orders a lizard. Orders -1 beers. Orders a ueicbksjdhd.

First real customer walks in and asks where the bathroom is. The bar bursts into flames, killing everyone.

— Brenan Keller (@brenankeller) November 30, 2018

Here you might be executing your method against each individual test input, such as:

orderBeers(1);
orderBeers(0);
orderBeers(99999999999);
orderBeers('lizard');
orderBeers(-1);
orderBeers('ueicbksjdhd');

Some testing libraries have special constructs for executing tests multiple times over a collection of specified inputs. I've seen these called "datasets", "data points", or "parameterized tests".

For example, in JUnit you could use a parameterized test:

@RunWith(Parameterized.class)
public class OrderTests {
    private int numberOfBeers;
    private boolean shouldPass;

    @Parameterized
    public static Iterable<Object[]> data() {
        return Arrays.asList(new Object[][]{
                        {1, true},
                        {0, true},
                        {99999999999, true},
                        // {"lizard", false}, will probably throw a runtime exception
                        {-1, false}
                }
        );
    }
       
    public OrderTests(int numberOfBeers, boolean shouldPass) {
        this.numberOfBeers = numberOfBeers;
        this.shouldPass = shouldPass;
    }
    
    @Test
    public void shouldDiscernValidOrder() throws Exception {
        boolean result = new BeerOrder().orderCount(this.numberOfBeers);
        assertThat(result, is(equalTo(shouldPass)));
    }
}

When this example is executed the test runner will instantiate a new test object for each of the values presented in the parameterized data, essentially looping through the data provided. I haven't seen a JavaScript library that will do the same: you'll have to provide your own looping structure.

JUnit also has the concept of theories, which can be used like parameterized tests but also give you control over the assumptions that can be applied to your test data prior to test execution. I'm not going to provide an example of a theory because I'm sleepy. You should definitely give that link a read tho.

Test Data Management

What if we were intentional about grooming and applying the test data we use to validate our system? When I say "intentional", I mean that our team has agreed-upon methods, goals, and tools for:

• Acquiring test data
• Aggregating test data
• Masking (sanitizing) test data
• Applying test data
• Refreshing test data

This practice is called, as the header suggests, "Test Data Management". Typically you would identify a source of test data (even if it's "write it yourself based on documentation or source code of the external dependency") and then figure out how to make sure it's devoid of Personally Identifiable Information (PII), usually by injecting Presidents' Names or Famous Mathematicians or Tolkien Characters or something. Then you would set up a schedule for how often you would get more (or updated) test data.

But what do you do with that data?

If you've worked with applications that use databases, you might have come across a common testing approach: spin up an in-memory database, populate it with known data, and then use that data to test your application's repository layer (or DAO for us older folks). For example, you might have a UserRepository with a method loadUser(int userId). Your setup would INSERT INTO user (id, name) VALUES (1, 'billy bob'), and your test would:

User user = userRepository.loadUser(1);
assertThat(user.getName(), is(equalTo('billy bob')));

You can use the exact same approach when your application consumes a web service (or some other external entity). Instead of using SQL to inject data into an in-memory database, you may (for example) register a mock endpoint on a fake server and have it return a known JSON packet under specific circumstances.

In React, testing with Cypress, assuming a file in <root>/fixtures/user.json, your test would look like:

describe('User component', () => {
    it('should get user information', () => {
        cy.fixture('user.json').as('userData')
        cy.server()
        cy.route({
            method: 'GET',
            url: '/api/users/*',
            response: '@userData'
        })
        // exercise the component that requests data from /api/users/1

When your component accesses /api/users/1 to get the user information from the external dependency, Cypress would serve up the data in user.json and your application would go on its merry way.

Sinon has something similar for the unit level if you're in JavaScript, and most testing frameworks I've used have something similar if you look around.

Wrap up

Okay, so, there are a couple of points I hope you take away:

1. Test data should not be randomized. Use Theories or Datasets if you want to run a test against a bunch of different inputs
2. Test data should not only "not be randomized", it should be curated and pampered. Take a look at Test Data Management practices and give your tests the data they deserve

Remember, your tests are only as good as your data, and they're only as trustworthy as your data is up-to-date.

... and I'm sorry for saying unkind things about people who use randomized test data to drive code design. Mostly.