Bee on flowers

Let your test automation choose the input data

What if you do not need to figure out what test data you have to put into your tests? That sounds like something impossible?

The problematic context

We test today a login class. The login class just has a password field and a validator to check if the password is a valid password. The class can look something like the following class.

class Login():
    def __init__(self):
        self.__pwd = None
    
    @property
    def password(self):
        return self.__pwd

    @password.setter
    def password(self, value):
        self.__pwd = value

    @property
    def is_valid(self):
        return True

The test to check this Login class can look like this. We just check if a random password is valid.

from login import Login

def test_valid_password(driver):
    login = Login()
    login.password  = "abcd"
    assert login.is_valid 

Our business now wants that all passwords are between 8 and 12 characters. Our programmers change the program. They add a validator class to validate if the input is valid.

class Validator(object):
    def check(self, input):
        if len(input) < 8:
            return False
        elif len(input) > 12:
            return False
        return True

class Login():
    def __init__(self):
        self.__pwd = None
    
    @property
    def password(self):
        return self.__pwd

    @password.setter
    def password(self, value):
        self.__pwd = value

    @property
    def is_valid(self):
        return Validator().check(self.__pwd)

According the new business rules, the previous test will still succeed. Do we as testers add more tests now ? Of course we add some more tests.

from login import Login 


def test_check_min_minus_1():
    login = Login()
    login.password = "1234567"
    assert login.is_valid == False

def test_check_min():
    login = Login()
    login.password = "12345678"
    assert login.is_valid

def test_check_min_plus_one():
    login = Login()
    login.password = "12345679"
    assert login.is_valid

def test_check_max_minus_1():
    login = Login()
    login.password = "12345678901"
    assert login.is_valid

def test_check_max():
    login = Login()
    login.password = "123456789012"
    assert login.is_valid

def test_check_max_plus_one():
    login = Login()
    login.password = "1234567890123"
    assert login.is_valid == False

We added a lot more tests now with different lengths of passwords. We worked very hard to test our login class. Time for a holiday. When we return from holiday, the first thing we notice is that are tests are green. Yes, everything just works fine.

The problem

That day, we receive a telephone call that a user can not log in. His password is not accepted anymore. There is something wrong. A developer then tells us that on our holiday the requirements changed a little bit.

The requirements changed? Yes, business wants now that the characters “_” and “-” are not valid inputs.

Because we did not took these values, the tests did not fail on the new requirements. That day we added some more tests in our test suite.

def test_weird_chars():
    login = Login()
    login.password = "123456-89012"
    assert login.is_valid == False

Can we generate our test inputs?

There is a library called hypothesis for python that can generate some data for us.

Hypothesis is a Python library for creating unit tests which are simpler to write and more powerful when run, finding edge cases in your code you wouldn’t have thought to look for. It is stable, powerful and easy to add to any existing test suite.

Hypothesis documentation

Why do we not try that library? The tests for the first situation can be written as follows with hypothesis.

from hypothesis import given
from hypothesis.strategies import text
from login import Login 


@given(text())
def test_check_password(pwd):
    login = Login()
    login.password  = pwd
    assert login.is_valid

Hypothesis has some decorators to inject test data for us. In the first situation, add some random text in the tests. The framework will now do some magic and generates data for our test. It will also run the test more than once.

In case we do not alter anything in our test and we are in the situation that the login length is checked, the test will fail with a value that has not the correct length. So we need to alter the tests because they fail now.

@given(text(min_size=8, max_size=12))
def test_check_password(pwd):
    login = Login()
    login.password  = pwd
    assert login.is_valid

The text strategy can have a minimum and maximum size. Now all our tests will succeed again. We can also add a new test to check if the other values are not valid. Just alter the min_size and max_size to do that.

The current tests will fail now when the new requirements where implemented. It will find a value with an _ or a – sign in it and will fail.

If we want to change that, we need to exclude some characters that are valid. We can do that with regular expressions. Hypothesis has a strategy for that. Let’s try that one:

from hypothesis import given
from hypothesis.strategies import from_regex
from login import Login 


@given(from_regex("^[a-zA-Z0-9_]{8,12}$",fullmatch=True))
def test_check_password(pwd):
    login = Login()
    login.password  = pwd
    assert login.is_valid

Now the tests are succeeding. Because we have now a lot more input values, we are more confident that the tests are better now.

With hypothesis we can write tests with input data that is described by a specification. We can perform operations on that and assert about the result.

This is called property-based testing. It was first used by the Haskell library Quickcheck.

Hypothesis can do a lot more for us. That is for a next post.

About the author

I currently work as a Test Automation Consultant at b.ignited. Here I work for different clients in different industries to help them start and speed up their testing cycles

I’ve been testing software since 2000 when I became involved in testing telephone applications and hardware. Since then, I’ve been able to expand my experience by testing a variety of embedded, web, mobile and desktop applications. I have used various development methodologies from waterfall to agile.

I consider myself to be a lifelong learner.

Comments

Comments are closed.