Posts Tagged ‘automated software testing’

What to do in a Watir test if you don’t have a unique id

January 21, 2008

OK, you don’t have ids on the controls you need to access.  Now what do you do.  I was faced with this situation.  I needed to verify that one menu item followed another.  Neither of the menu items had an id.  I could simply use indicies on the links.  That would have made for a very brittle test (one that breaks easily).  I happened to notice that the parent cell (TD) had a unique class (sideNav).  If I could associate my test with that cell, it would be a bit more stable.

Within the cell was a list structure (UL), with each list item (LI) followed by a link that contained text.  I was supposed to make sure that the link with text of “Item-of-Interest” was followed by the link with the text “Following-Item”.  The structure of this list was very regular: list item followed by a link, followed by text, followed by the next list item.
<TD class=sideNav>
    <UL>
       <LI>
         <A>
            #text
       <LI>
         <A>
            #text
       <LI>
         <A>
            #text
       <LI>
         <A>
            #text
       <LI>
         <A>
            #text
       <LI>
         <A>
            #text     <– Item-of-Interest
       <LI>
         <A>
            #text     <– Following-Item
       <LI>
         <A>
            #text
If I just used indicies to locate the links, an additional link to Item-of-Interest in front of the list could break the test.  That is not an unusual situation on our website.  In addition, using the “sideNav” class provided some additional insurance that I was testing what I thought I was testing.  I decided I could locate the cell with class=sideNav, loop through the list until I found the first text I was looking for and test the next link to see that it had the “Following-Item” text.  The code below is a sample of how to implement the test.

i = 1 
while $br.cell(:class, "sideNav").ul(:index,1).li(:index, i).link(:index, 1).exists? 
   if $br.cell(:class, "sideNav").ul(:index,1).li(:index, i).link(:index, 1).text =~ /Item-of-Interest/i 
      if $br.cell(:class, "sideNav").ul(:index,1).li(:index, i + 1).link(:index, 1).exists? 
         assert($br.cell(:class, "sideNav").ul(:index,1).li(:index, i + 1).link(:index, 1).text =~ /Following-Item/i) 
      end 
   end 
   i += 1 
end

The statement that begins with “while” and ends with “.exists?” is there to prevent the test from failing with an exception if the combination does not exist.  I loop through the list items with an index so that once I find the first text, I will look for the “Following-Item” text at index+1.  Again, I do an “exists?” test before I do the assert to keep the code from throwing an exception.

It turns out that this worked out OK for testing this condition.  Although it is not the best situation, using the cell class to identify the area was better than just using indicies.

Do yourself a favor (use id’s for Watir testing)

January 9, 2008

When you use Watir to test web pages, the best way to identify buttons and text fields is with ids.  The problem may be that the controls you want to test do not have ids.  The solution to this may be political rather than technical.  I know that developers and testers usually want to stay out of politics.  I have some ideas that may help you deal with the situation.

The first soultion involves finding a developer or developers who are willing to help you make it easier to test code.  There are some out there.  Usually they have worked in an environment where there is an emphasis on testing like Test Driven Development (TDD).  It could be that it is a forward thinking developer that realizes the advantages of having a good set of test suites for code they are developing.  If you can find developers who share this interest with you, it is the best way to procede.  Getting things done informally is usually the easiest and fastest way to getting things done.

If that does not work, the next thing to try is to sell your manager that it is to their benefit to get the developers to help you out.  Now I know you don’t like to “sell” anything.  The truth is that managers like things that save them money, make them look more productive, or make them look smarter to their managers.  Conviencing your manager to get the developers to add ids to the web pages will do this in the following three ways.

First, it will make the tests cheaper to develop.  You can change “cheaper” around to whatever your manager’s hot buttons are.  Basically it means that you can create more tests in the same amount of time or that you will not have to hire more testers. You will probably be able to start testing earlier because tests are easier to write.  You get the picture. You will have more tests earlier.

Second, tests will be less brittle.  If you are new to the testing world, a brittle test is one that breaks easily.  In web testing, this usually happens because you have to rely on tag structure based on position rather than an id that stays with a tag regardless of where it moves on the page.  If you end up testing while developers are still coding and you can use ids, your tests will not break as much.

Third, it will be cheaper to maintain your tests.  Undoubtedly there will be bug fixes or new features added to the original application.  Your tests will have to be modified. It will be easier for someone else to understand and maintain the tests if you use ids.
           
I really cannot stress enough how much time you will save and how much easier your life will be as a tester if you can get ids on the controls.  It would certainly be worth buying a developer a lunch or two if you can convence them to add ids for you.

If you look back at my blog post How to start writing web test scripts with Ruby and Watir, https://jimhmatthews.wordpress.com/2007/10/28/how-to-start-writing-web-test-scripts-with-ruby-and-watir/, you will notice that I had text_fields, ie.text_field(:id, “billing_name”) and a button, ie.button(:id, “checkout_button”) that were identified with a unique id.  Now if these move around on the page, my tests will not break because I can still find them with the id.
     def enter_billing_information
       assert(ie.text_field(:id, “billing_name).value == ”)
       assert(ie.text_field(:id, “billing_address”).value == ”)
       assert(ie.text_field(:id, “billing_city”).value == ”)
       assert(ie.text_field(:id, “billing_state”).value == ”)
       assert(ie.text_field(:id, “billing_zipcode”.value == ”)
       ie.text_field(:id, “billing_name).set’Joe Tester’
       ie.text_field(:id, “billing_address”).set’123 Address’
       ie.text_field(:id, “billing_city”).set’Austin”
       ie.text_field(:id, “billing_state”).set’TX’
       ie.text_field(:id, “billing_zipcode”).set’12345′
       assert(ie.text_field(:id, “billing_name).value == ‘Joe Tester’)
       assert(ie.text_field(:id, “billing_address”).value == ‘123 Address’)
       assert(ie.text_field(:id, “billing_city”).value == ‘Austin’)
       assert(ie.text_field(:id, “billing_state”).value == ‘TX’)
       assert(ie.text_field(:id, “billing_zipcode”).value == ‘12345’)
     end
and
     def checkout_button
       ie.button(:id, “checkout_button”)
     end
If you are going to the trouble to add unique ids, you might as well make them descriptive about what they reference.  Hence I chose “billing_name” and “checkout_button.”

Next time I will talk about what to do if you have to test a page that does not have ids and you cannot get them on there anytime soon if at all.

Negative Testing with Fit

December 13, 2007

Someone asked me how you do negative testing in Fit.  People generally refer to tests that are supposed to cause errors as negative tests.  The first thing is to get the positive test cases working, those that should not produce any errors.  If the positive cases are not passing, you cannot depend on any results.  Though some negative tests cases may be producing correct results, if positive test cases are not passing, you cannot depend on the results of negative test cases (those that should cause errors).  If the developers have to change code to get the positive tests working, they are likely to make a change that will affect the negative test cases.  All specified functions should have positive tests.  Get the positive test cases working before concentrating on getting the negative test cases working.

The developers in our group came up with the following simple fixture to deal with errors that generate error messages.  Note this fixture is designed around the idea that when there is an error, an error message is generated.  You include the fixture and put the following table in your test if you are not expecting an error message.  This is something you would put in your positive tests.

ErrorMessage
Messages

When you run your test, if there are no errors, you get the table with the two gray rows.  If on the other hand you get an error you are not expecting, it adds a new row to the table with the error message and the indication that this is surplus, i.e. something you were not expecting.  Since this is an error you are not expecting, it colors the row red.

ErrorMessage
Messages
Something went wrong. surplus

Now lets say that you were creating a negative test (one in which you were expecting an error) where you expected the error message “Something went wrong.”  You would create the following table and put it in your test.

ErrorMessage
Messages
Something went wrong.

When you run your test, if that is the only error you get, the row with the error message turns green.  You were expecting the error message and you got what you expected.

ErrorMessage
Messages
Something went wrong.

Now lets suppose that you ran the test where you were expecting the error message “Something went wrong,” and for whatever reason that error message was not produced.  In that case the fixture would indicate that the error message was missing and turned the row red.

ErrorMessage
Messages
Something went wrong. missing

Here is what happens if you are expecting the error message “Something went wrong,” got that message, but also got the additional error message “Something else went wrong.”  The message you were expecting would be shown in green and the additional message would be shown in red as surplus.

ErrorMessage
Messages
Something went wrong.
Something else went wrong. surplus

When we ask someone about the second error message, we learn that it was due to a change in requirements (of course this would never happen in the real world) and that it really should be there.  Now we can add that row to our fixture as an expected error message as shown below.

ErrorMessage
Messages
Something went wrong.
Something else went wrong.

The next time we run the tests, we are expecting both error messages and getting both error messages so everybody is happy.

ErrorMessage
Messages
Something went wrong.
Something else went wrong.

The book Fit for Developing Software discusses error reporting in a ColumnFixture and in an ActionFixture.  We built a number of DoFixtures to execute actions in our application.  When the action caused an error, the Error Message fixture displayed the error message.  We found this method very effective for testing error handling in the application. 

Get Fit ( but not necessarily FitNesse )

December 4, 2007

As I said in “About Me”, I believe in using whatever tool makes sense for what you are trying to test.  In a recent situation, we needed to test a “rules engine” for modifying URLs.  The test cases were pretty straight forward.   They were the current URLs and what we expected them to be after being modified by the rules engine.

The test tool we chose for this situation is “Fit”.  You may have heard of Fit and FitNesse before.  FitNess is the wiki that was originally used to drive Fit. For background information on Fit and FitNesse, go to http://fit.c2.com/

There are a number of drawbacks of using FitNesse to drive the tests.  The two most significant ones are that it is difficult to do version control of the test cases and that only one set of tests can be run from the wicki at a time.

The solution was to run the tests with Fit but not use FitNesse.  This solved the two main problems of using FitNesse.  First, the tests could be put under version control.  Second, anyone can checkout the tests and the Fit driver to their system and run the tests without worrying if someone else is running the tests.

We chose Fit because the test cases are pairs of inputs and expected outputs.  Fit allows you to specify a table with a column of inputs and a column of expected outputs.  Each row is a test case with input and the expected output.  We knew from the beginning that we were going to have a large number of test cases.  Using a table structure made it easy to manage data for the tests.  The table below shows an example of what inputs and expected results might look like.

inputurl1 expectedoutputurl1
inputurl2 expectedoutputurl2
inputurl3 expectedoutputurl3

Anyone who is familiar with the application should be able to understand the inputs and expected results.  Developers as well as testers are able to create test case pairs.  Developers can run the tests on their system while they are writing new code.

The test cases we developed fell into four categories.  The first was the existing URLs that we expected to be modified in a certain way.  The second was the existing URLs that we expected to be unmodified by the rules engine.  The third set of cases were ones where there were currently not any existing URLs but test URLs that should be modified in a certain way because of the rules.  The fourth set of cases was currently non-existent URLS that should not be modified by the rules engine.

We have implemented this testing system for the rules engine and it has proven effective.  There are about 300 test case pairs.  The test cases were developed by two testers and three developers.

For a further discussion of where Fit can be used as an effective testing tool, see the following link:

http://awta.wikispaces.com/Fit+-+Fitness+Implementation+Strategy