Skip to content

Randomness

Randomness can be useful to generate arbitrary test data (e.g. for fuzzing, property testing, generating default values…). However, tests that exhibit randomness are quickly hard to debug: re-running them may give a different result!

Prepared brings a very simple solution to this problem:

  • When a test uses random values, the initial seed is printed to the standard output,
  • Users can force a specific seed value at the start of the test.

Together, this allows users to easily replicate failures in CI on their local machines. Let's walk through an example.

Example walkthrough

Let's imagine you are sending data to another system (e.g. a REST DTO, or storing something in a database…). You want to ensure the data you send and the data you receive are always exactly the same. You also want to ensure that injection attacks are not possible.

Proving that something is impossible is difficult. However, using a probabilistic approach, we can reduce the risks.

Let's write a simple probabilistic test: 

test("Round trip serialization check") {
    val payload = Array(random.nextInt(0, 64)) { random.nextInt().toChar() }
        .joinToString("")

    check(deserialize(serialize(payload)) == payload)
}

This test generates a random string of data, makes it go through the round trip of our serialize and deserialize functions, and checks that the result is identical to the original data.

The difference between random and Random

Note that the test uses random, and not Random.

Random is the Kotlin standard library's default random source. If you use it directly, the behavior described on this page will not activate. Prepared will display a compile-time warning on usage of Random in a context where random is available.

Writing this kind of tests in a real situation

This example was somewhat simplified for the purposes of simplicity. If you want to write a test like this in a real project, we recommend making a few modifications to it.

First, we recommend generating real data, instead of a string representation. For example, generating instances of your model classes, and using those.

Then, we recommend increasing the number of executions. Probabilistic tests are useful when rare cases happen relatively often, which requires a large amount of executions. For example, you may want to use repeat to increase the number of test executions: 

repeat(1000) {
    test("Round trip serialization check $it/1000") {
        // The test itself is unchanged
    }
}

When running this test, Prepared prints the following line: 

» Prepared ‘randomSource’: Random generator with seed 3286522734459043202. 
To reproduce this execution, add 'random.setSeed(3286522734459043202L)' at 
the start of the test, before any random generation.

If the test fails in your CI environment, or another developer's machine, you can explicitly set the seed at the start of the test to reproduce the same execution: 

test("Round trip serialization check") {
    random.setSeed(3286522734459043202L)

    val payload = Array(random.nextInt(0, 64)) { random.nextInt().toChar() }
        .joinToString("")

    check(deserialize(serialize(payload)) == payload)
}

If random.setSeed() is called when after a seed has already been generated, Prepared throws an exception.

Random generators are order-dependent

Tests can only be reproduced if all calls to the random generator are executed in the exact same order each time.

If you have to use random values in contexts where the order is unknown, instantiate them at the start of the test, and pass the variables to your different systems.

Alternative syntax

As we have seen, we can use the random.nextInt() syntax to generate a controllable random value. If using prepared values, Prepared offers a shorthand: randomInt(), which behaves exactly in the same way as other prepared values: 

val adminId by randomInt(0, 1000)

val admin by prepared {
    User(id = adminId(), …)
}

test("Test name") {
    check(service.create(admin()).id == adminId())
}
The adminId is a randomized value. Like any other prepared value, each test will receive a different instance, and referring to it multiple times within a single test returns the same value each time.

A benefit of this approach is that prepared values print their actual value to the standard output, so it becomes easier to see which values were generated by the test.

Interactions with fixtures

Random control works in exactly the same way within prepared values as they do within a test, because prepared values are executed in the context of a specific test.

Specifically, setting the seed explicitly must be done before referring to any prepared values that use random values.

However, randomness control is not available within shared values, because their result must be shared between multiple tests, and therefore a single test couldn't set the seed for all of them. We advise against using shared values that contain randomness, as they lack any way to control it.