Property-based testing in JavaScript

As developers we are always looking for new ways to ensure that our code works as we would expected. One way to achieve this, which is not as commonly used as for example unit tests, is property-based testing. Property-based testing is a testing method based on (pseudo-)randomly generated input.

To create a test case you need to define the following parts:

• Unit under test: E.g. a function, endpoint in your API
• Input generator function: A function that will generate pseudo-random input for your test case (based on a seed)
• A property predicate: A predicate that has to hold true for all inputs generated by the generator function. It should include the execution of your unit under test

The test runner will then roughly do the following when executing your test:

1. Ideally it will seed your generator function with a random seed. This is necessary to make the results reproducible
2. It will feed the next output of your generator function into the property predicate
3. It will check whether the output of your unit-under-test passes the property predicate
   • If false: fail test and print the input that lead to non-compliance with the property description
   • If true: continue with 2. (until some number of iterations is reached)

A simple example for a property based test would be the following property based test for the plus function (for simplicity’s sake I left out the seed part of the generator function). This property-based test tests whether the sum of two positive numbers is always positive:

• Unit under test: const add = (x, y) => x + y
• Input generator function: const generatePositiveInt = () => Math.round(Math.random() * 100000)
• Property predicate: const prop = (in1, in2) => add(in1, in2) >= 0

The advantage of this type of testing is that your function will be tested with way more input values compared to testing with regular unit tests. This allows you to find edge cases where your unit-under-test does not behave as you would expect it to. This is quite similar to Fuzzing in the IT-Security community, where the goal is to find security issues or memory leaks by providing (invalid) random data.

Depending on the unit-under-test, the properties can be quite hard to define. For the simple plus example from before it was quite easy, but different algorithms have different properties and they are often very unique to the unit-under-test. Another thing to note is that property-based tests don’t have any guarantees that all of your code is covered with each test-run. The goal is rather to find issues eventually by generating enough random data. They also don’t describe the actual behavior of the code with explicit inputs and outputs, as you would do in unit tests. This is why property based test should be employed additionally to unit tests. The unit tests ensure that all code branches are covered and the behavior is documented, while the property based tests are there to find cases where the code does not work as expected (with respect to the properties that were defined).

Property-Based Testing in JavaScript

Now let us take a look at property-based testing in JavaScript. In other languages like Haskell (QuickCheck) and Clojure (test.check) there are well established libraries for property-based testing, while in the JavaScript world it has not yet gained a lot of traction. Few frameworks exist that are actively developed and have a substantial amount of users (> 1000 downloads per month on npm). I picked some of them for a comparison:

• jsverify: A library highly inspired by Haskells QuickCheck
• quick_check: An implementation of Haskells QuickCheck in JavaScript
• testcheck: A library that is a thin wrapper around ClojureScripts test.check

To be able to compare those frameworks, we will test the implementation of Array.prototype.filter. You can find the full example, where you can actually run the tests, here.

function filter(fn, array) {
    return array.filter(fn);
}

As the first step we have to define properties for this function. I picked some, but there might be additional properties not listed here. Think about it.

• After any filter invocation the length of the returned array will be equal or less the length before
• After any filter invocation the elements of the resulting array are contained in the original one
• filter (with the same fn argument) is idempotent, which means multiple invocations will not change the result
• filter should be independent of the order of elements regarding the length of the resulting array

Those properties translate to the following javascript:

const R = require('ramda');
const filter = require('../index');

module.exports = [
    {
        description: 'Length after filter <= Length before filter',
        test: (fn, arr) => filter(fn, arr).length <= arr.length
    },
    {
        description: 'Elements are contained in array before filter',
        test: (fn, arr) => filter(fn, arr).every((e) => arr.includes(e))
    },
    {
        description: 'Idempotence',
        test: (fn, arr) => {
            const once = filter(fn, arr);
            const twice = filter(fn, filter(fn, arr));
            return R.equals(once, twice);
        }
    },
    {
        description: 'Length is independent from sorting of the array',
        test: (fn, arr) => {
            const sortedThenFiltered = filter(fn, arr.sort());
            return filter(fn, arr).length === sortedThenFiltered.length;
        }
    }
];

Implementing the tests

After implementing those test for all of the frameworks mentioned above, I quickly realized that the differences between all those frameworks are minimal. All of them basically have the same syntax and a very similar feature set (which is not surprising as they all are based on Haskells Quickcheck). To emphasize this here are the definitions for a property in jsverify and quick_check (where property refers to a single element of the array above):

// jsverify
jv.property(
    property.description,
    // First input: A function that returns a boolean for any input
    jv.fn(jv.bool),
    // Second input: An array of any values
    jv.array(any),
    // The property
    property.test
);

// quick_check
it(property.description, function () {
    qc.forAll(
        // First input: A function that returns a boolean for any input
        qc.function(qc.bool),
        // Second input: An array of any values
        qc.array,
        // The property
        (...args) => assert(property.test(...args))
    );
});

Basically they look the same except for the property description (because jsverify has built-in mocha integration) and quick_check actually requiring an assert. All frameworks fulfill the most important points for property based testing:

• Integrates well with test frameworks, such as mocha, jasmine, ava, etc.
• Shrinking: Reduces a failing testcase to the minimum failing test case (e.g. if your test fails for an input array, the test runner will reduce this array to the minimum number of elements to get the failing test)
• Miscellanious Generators: Numbers, Lists, Objects, Custom and more

I had some issues with the testcheck library though. It was horribly slow compared to the other libraries, especially when increasing the number of examples that are tested in a single test run. E.g. when testing with 100 examples, the tests took almost 20 seconds to finish. Plus, it provides the least amount of pre-defined generators. For our test case I especially missed any generator that generates functions, so I had to write my own.

On the other hand, writing a custom generator is not that complicated (for all frameworks). For our function, we need a deterministic function that gets a single argument of any type and returns a boolean:

const objectHash = require('object-hash');
const genFunc = gen.intWithin(0, 9).then((i) => {
    return function hashEq(val) {
        return objectHash(val).startsWith(i.toString());
    }
});

When a value for this generator is built, we basically get a function that checks whether the hash of the value starts with i, where i is a value drawn from the range from 0 to 9. This function is deterministic, so we can use it as the input for our property test. This concept of generator transformation exists in all of the mentioned frameworks (although it is called map in the others).

Conclusion

Property-based testing is a cool and useful addition to your test suite. We saw how to build your own property-based tests and which frameworks exist to support you doing that. My current personal favorite is quick_check, due to the good documentation and the variety of available generators. In order to show you how generators can work, we built an example using a generator transformation. This should serve as a base to define your own properties and implement your own tests for your product. I hope you have fun with it!