Negative Testing with Fit

Someone asked me how you do negative testing in Fit.  People generally refer to tests that are supposed to cause errors as negative tests.  The first thing is to get the positive test cases working, those that should not produce any errors.  If the positive cases are not passing, you cannot depend on any results.  Though some negative tests cases may be producing correct results, if positive test cases are not passing, you cannot depend on the results of negative test cases (those that should cause errors).  If the developers have to change code to get the positive tests working, they are likely to make a change that will affect the negative test cases.  All specified functions should have positive tests.  Get the positive test cases working before concentrating on getting the negative test cases working.

The developers in our group came up with the following simple fixture to deal with errors that generate error messages.  Note this fixture is designed around the idea that when there is an error, an error message is generated.  You include the fixture and put the following table in your test if you are not expecting an error message.  This is something you would put in your positive tests.

ErrorMessage
Messages

When you run your test, if there are no errors, you get the table with the two gray rows.  If on the other hand you get an error you are not expecting, it adds a new row to the table with the error message and the indication that this is surplus, i.e. something you were not expecting.  Since this is an error you are not expecting, it colors the row red.

ErrorMessage
Messages
Something went wrong. surplus

Now lets say that you were creating a negative test (one in which you were expecting an error) where you expected the error message “Something went wrong.”  You would create the following table and put it in your test.

ErrorMessage
Messages
Something went wrong.

When you run your test, if that is the only error you get, the row with the error message turns green.  You were expecting the error message and you got what you expected.

ErrorMessage
Messages
Something went wrong.

Now lets suppose that you ran the test where you were expecting the error message “Something went wrong,” and for whatever reason that error message was not produced.  In that case the fixture would indicate that the error message was missing and turned the row red.

ErrorMessage
Messages
Something went wrong. missing

Here is what happens if you are expecting the error message “Something went wrong,” got that message, but also got the additional error message “Something else went wrong.”  The message you were expecting would be shown in green and the additional message would be shown in red as surplus.

ErrorMessage
Messages
Something went wrong.
Something else went wrong. surplus

When we ask someone about the second error message, we learn that it was due to a change in requirements (of course this would never happen in the real world) and that it really should be there.  Now we can add that row to our fixture as an expected error message as shown below.

ErrorMessage
Messages
Something went wrong.
Something else went wrong.

The next time we run the tests, we are expecting both error messages and getting both error messages so everybody is happy.

ErrorMessage
Messages
Something went wrong.
Something else went wrong.

The book Fit for Developing Software discusses error reporting in a ColumnFixture and in an ActionFixture.  We built a number of DoFixtures to execute actions in our application.  When the action caused an error, the Error Message fixture displayed the error message.  We found this method very effective for testing error handling in the application. 

Advertisements

Tags: , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: