Stop, Look & Listen.

I look up, red light.... red light... GREEN. I'm off!
Whoosh. A cement truck shoots past at 40mph, inches from my nose.
[Cold sweat, dry throat]
I look around, remember my green cross code, and cross the road. I make it to the other side safely. After I get over my anger (ok ... fear) I realize the mistake was as much mine as his. I relied on a proxy, rather than examining the real and readily apparent risk.

The red man/green man lights are not going to play truck versus pedestrian, I am - I need to use my head (and eyes!). The lights are based on a dumb set of rules, and clocks. They don't really know if a PSYCHOTIC CEMENT TRUCK DRIVER is playing the game with me, or not. I should use the lights as an indication of when to use my own head, eyes and common sense to check for trucks. I'm a fool if I depend on a set of [installed by the lowest bidder] lights and timers to tell me if a [speeding, badly driven] truck is heading straight for me.

Your automated tests are just like the traffic lights, they display red/green based on a set of dumb non-sapient rules. They check for pre-determined responses, a particular string, number or response code. What that actually means is that a given string etc was supplied, not that all the hidden action [i.e.: trucks speeding just round the corner] is behaving itself correctly.

The automated tests are a start, an indicator, and maybe if they are all showing Green, you need to smarten them up a little. Get them to check for some indicators that might help you see the bugs. (Maybe they could check for errors in the logs, or other less visible symptoms)

Giving Selenium RC, some of the ease of use of WATIR.

Iterators and the ability to run your code easily against a whole array, are one of the cool features of Ruby. This 'ruby way' is one of the benefits of the browser test tool/driver WATIR. Your tests can easily grab a list of elements and process them easily. Making simple things like checking all the links on a page, relatively trivial to code - and to read the code.
e.g.
In WATIR:

ie.links do
  # Do something with the link
end

By default, this sort of functionality is not present in Selenium RC. Selenium's heavily imperative API leads the coder towards the use of For loops, etc. Rather than Ruby's more elegant functional approach - designed like Ruby - to reduce [tester] developer cycles.

But with a little coding, Ruby can extend the Selenium Client to work more in tune with Ruby...
e.g.

browser.links.each do |a_link|
  puts a_link.href
  if a_link.text=="Maps"
    a_link.click
    break
  end # if
end # link

The steps are, firstly create a Element class, that we can use to hold the browser object and locator etc:

class Element

  def initialize(browser, locator)
    @browser=browser
    @locator=locator
  end # init

  def href
    return @browser.get_attribute("#{@locator}@href")
  end # href

  def text
    return @browser.get_text(@locator)
  end # text

  def click
    @browser.click(@locator)
  end # click

  def to_s
    return "Text: #{@browser.get_text(@locator)} \nHREF: #{@browser.get_attribute(@locator + '@href')}"
  end # to_s

end # class

Then add the #links method to the Selenium Client Driver:

class SeleniumClientDriver < Selenium::Client::Driver
  def links
    elements = Array.new
    number_of_links=self.get_eval("window.document.links.length").to_i
    number_of_links.times do |link_num|
      elements.push Element.new(self,"dom=window.document.links[#{link_num}]")
    end # times
    return elements
  end # links
  
end # selenium

Then you can create your browser object using SeleniumClientDriver and call the .links method, with a code block as above.

# Now we can iterate through the links
browser.links.each do |a_link|
  puts a_link.href
  if a_link.text=="Maps"
    a_link.click
    break
  end # if
end # link

A complete code example can be downloaded.

Bug in Selenium RC, affecting XPath locators in Internet Explorer

I've recently noticed a bug in Selenium RC, affecting XPath locators in Internet Explorer. The issue is actually a integration bug between ill-formatted HTML and Selenium. If the HTML element you are trying to 'locate', has a class attribute beginning with a space - then selenium won't be able to locate the element using that class in the XPath. e.g.:
Your HTML:

 <div class="outer blue">
  <div class=" inner green">
  </div>
 </div>

The Selenium RC command, is shown below.
Firefox seems to correctly handle the leading space:

# Will return 1 in FireFox, and 0 in Internet Explorer
browser.get_xpath_count("//*[@class='inner green']")

...while IE doesn't.

# Will return 1 in FireFox, and 1 in Internet Explorer
browser.get_xpath_count("//*[@class='outer blue']")

You may argue (as I have) that the class attribute shouldn't have a leading space. In the Spec, spaces are described as being separators between class attribute values. So the creators/maintainers of the page should 'fix' the HTML, allowing the selenium code/tests to correctly locate the element.

TAP formatted results for RSpec?

In centuries past, When traders traveled from port to port, across the Mediterranean, they encountered a host of different countries and city states. Each with their own cultures, religions and languages. The traders needed way communicate with other traders, customers and suppliers, no matter which port they landed in. To fill this need, came Sabir, or Lingua Franca. This common pidgin language facilitated simple communication and more importantly commerce.

Test tools have a similar need for communication. For example a typical web application will have several layers of tests. Unit tests, system tests, acceptance tests and UI- tests to mention a few. These types of tests may well utilise a range of different tools.

E.g.:
Your Java back-end might use JUnit for unit tests.
Ruby & Selenium tests at the front end may use RSpec
Your JavaScript code may use js-test-driver
Your Perl system admin code may use PerlUnit.
...
(You get the picture)

While these tools all provide their own result formatting, they don't all 'speak the same language'. As a developer or tester, how can you be sure that all is good without looking at/understanding a range of result types. If I want an automated system to summarise the results, you can see this might get messy.

This is where TAP (Test Anything Protocol) comes in. This standard for command line test results came out of the Perl community and provides a simple uncluttered way to represent your test results.

As a frequent user of Ruby's RSpec tool, I've created a custom formatter for RSpec results, allowing this Specification-oriented test runner to output TAP formatted results. It uses RSpec's built-in ability to utilise external formatters.

You can download my first draft implementation of the formatter.

To use it just include it, like this:

spec --require tap_format --format TAPFormat:my_results.tap my_spec.rb

Your RSpec file is called: my_spec.rb (in this example)
Your results will appear in the file: my_results.tap

Example?

Is an oft-chanted mantra in software testing. Whether it's explaining discovered bugs, or explaining why a particular feature needs testing, by providing a possible bug, examples are useful. In fact, bug reports are just 'examples of failures'. We know, as testers, that the application has bugs - we just need to document specific instances.

But I've found when mentoring and teaching testing techniques, that examples can be elusive. It's not that bug-examples are rare, but finding the right bug at the right time to illustrate a particular concept, can be tricky.

There's usually plenty of verbiage about the concepts, but actual working (or more correctly: appropriately broken) examples of applications, are few. So to help remedy this, I'm going to create some examples that can be used for demonstrating black box testing techniques. The first is Boundary Value Analysis. It's a simple example of an application that has equivalence partitions and therefore boundaries - and a related boundary bug. (There are probably in fact many 'bugs', but one is deliberate).

What do YOU do to stay up-to-date?

That's a question you might get in an interview. In fact its probably a question that should get asked a lot more. Some times it's re-phrased: "What was the last book/blog/etc you read on XYZ?".

Why is it a good question? Well it helps the interviewer discern whether they are dealing with a professional [or not]. Many professions require their practitioners to update their skills or read/be trained in their field throught their career. The NHS even have an acronym for it, CPD, or Continuing Professional Development.

My answer, as of yesterday, is: "How to Break Web Software?" by Mike Andrews and James A. Whittaker. (of 'JW on Test' fame). The Harry Robinson quote on the cover states why you need to read this book: "The techniques in this book are not an option for testers- they are mandatory and these are the guys to tell you how to apply them!". He's right. The book works both as a refresher and a practical guide. Many of the tips are tried and trusted, but the wide range and detailed descriptions mean you will undoubtedly expand your knowledge either in breadth or detail.

I've now added this book to my list of books I'd recommend to testers, on the client sites I consult with. It's a practical book, with real world examples and some good background information.

We don't need testers...

Think of an industry where people all around the world sit in front of their computers, collaborating to develop new products. These people spend all day typing into their computers, making these new products and selling them online all round the world.

They use the latest technology and tools, and these people are smart and educated. They're lucky, their business is global, its connected, if they're quick and ambitious they can earn a good living. The economy needs these people, and their businesses, as they help generate and distribute goods and wealth.

Now these people aren't perfect, they make mistakes, I'm going to have to say it: "They are only human.". Deadlines, short term business strategies etc mean these people take shortcuts. But salaries are rising, their skills are in demand and if they don't make that quick-[errh?]-fix then their bonus will go to that guy down the corridor. But the consequences are that they can make mistakes, that affect them and their customers.

But these people aren't bad people. They are just under the same pressures we all are. They check their work for mistakes, they do the best given the time and tools they've been given.

Now do you think some-one else should be there to check the work was at least ok?

Someone to play devil's-advocate with the product and see if anything could go wrong?

The same person could check the bigger picture, what else could go wrong? This product is used all round the world, and in conjunction with lots of other products - surely some mix of those products might have a problem...

As a customer, I'd definitely want someone there to check the product was safe and reliable.
As a manager I'd want someone there to catch the problems, and keep everyone on the straight and narrow, and let me know what the problems were [before my boss found out].

This is where I start to get to confused. The above description fits more than one industry.
I could be describing:
- Investment Banking.
- Software development.
etc

Now would you trust the bankers to do all the right checks? ensure it was all good?
Sure, they are responsible for their own work, and checking its quality.
But wouldn't you want at least a bit of auditing and regulation?

What about software?
Why do some teams just decide that they can do without testers?
Unlike the banks, their currency is their product. Its value is the customer's experience of the product and its quality. Shouldn't those teams invest in the software equivalent of auditing and regulation?