All our testing should be Exploratory

Exploratory testing, that’s a manual activity, right? Wrong.

Exploratory testing is an approach to testing that allows you the freedom to learn about an application, improve your skills and enhance your tests in an intelligent way based on feedback from execution. If you want to benefit from the full value of exploratory testing you should be taking a conscious exploratory approach across your entire test strategy and not just some manual stuff you do around the edges.

I find that in strong teams, most testing, including automated regression testing, is exploratory. Being mindful of this will make your testing better and help avoid the pitfalls of scripted testing (pitfalls? that’s a whole other post that’s already been covered by people much smarter than me).

For years I have been promoting the use of exploratory testing on Agile projects. Unfortunately, I don’t think I really understood what that meant. I forget how many times I have presented slides or written proposals stating that automating acceptance tests and regression tests will free up testers to do the valuable work of “manual exploratory testing”. Hey, you can even put some structure around it, maybe session based or story based. Lots of others seem to agree, so it must be the right approach, right?

However, something never sat quite comfortably with me. I have always used a cobbled together set of tools to help me in my testing.  A batch file here, a perl script there, the odd bit of sql that I’d tweak each time I ran it. But this is lost when talking about exploratory testing in a way that it implies it is manual. I have heard the term “assisted exploratory testing” but that sounds wrong to me; somehow against the exploratory spirit.

Having moved back into the testing space after a couple of years embroiled in the world of project management, I’ve begun thinking about testing with a fresh perspective . A few things have caused me to re-evaluate my understanding of exploratory testing and what it means within the context of an Agile project.

In particular, Michael Bolton’s excellent post http://www.developsense.com/blog/2010/09/exploratory-testing-and-review and James Bach’s much more blunt and arguably more impactful post http://www.satisfice.com/blog/archives/496 lead me to re-evaluate Cem Kaner’s synthesis/definition of exploratory testing:

Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.

At no point has anyone trying to define Exploratory testing inferred that it had to be manual, but this seems to be an assumption many people make as it is the easiest form to understand.

Within the agile community I hear a common debate about who should automate acceptance tests, when they should be automated and what the benefits are.  Usually someone makes the argument that the investment in automation frees up skilled testers to do “manual exploratory testing”.  I now feel this is missing the point. Why can’t all our testing be exploratory? There is no reason to think of the automated test suite, often the cornerstone of an agile project, as being outside the realm of exploratory testing.

Dan North made a great point in a recent talk about his incubating ideas around deliberate discovery http://skillsmatter.com/podcast/agile-scrum/keynote-deliberate-discovery-code-like-you-mean-it. He is right to state that exploratory testers are some of the few people currently following this model of deliberate discovery, but wrong, to describe these individuals as “skilled manual exploratory testers”.

I am not proposing automating your exploratory testing. I am proposing that if you value exploratory testing you should ensure that you take an exploratory approach to all your tests including those you have automated.

So, how does exploratory testing compare with a more linear scripted approach to testing on an Agile project?

Exploratory testing on an Agile project might look like this:

  • The assumption in the testers mind is always “there is something we don’t know”.
  • Testers and the rest of the team come up with ideas to test the specified functionality and the proposed implementation.
  • Some tests are written down as acceptance tests, this is how we think the system should and will behave.
  • When valuable these will be automated to allow developers to quickly validate if what they have built is what the team expects. If it isn’t, either the code changes to match the tests or the tests change to match the code.
  • Once the requirement is a real thing that can be manipulated, the testers will explore the new code and how it interacts with the rest of the code, feeding back what they learn to the team.
  • Automated tests are updated to represent what is currently known about the application. Kept as lean as testers need it to be without any explicit links with the original stories they were developed from.
  • A continuous integration server executes these automated tests, providing a feedback loop that informs exploratory testing.
  • Testers also utilise all or parts of this suite to help them explore the system alongside manual execution of tests.

Scripted testing on an Agile project can look like this:

  • The assumption in the tester’s mind is always “we will identify what we need to test and automate as much of it as we can”.
  • Testers and the rest of the team come up with ideas to test the specified functionality and the proposed implementation.
  • Some tests are written down as acceptance tests, this is how we think the system should and will behave.
  • Requirements are not completed until all acceptance tests pass.
  • Acceptance tests are automated and added to the CI build as per the story and must continue to pass for the life of the project.
  • Every time an automated test fails, huge amounts of energy are exerted to make the application pass a possibly outdated test scenario or seek permission to change.
  • Your automated regression suite can end up driving what the team does instead of informing what a team does.

Why all our testing should be exploratory.

One of the most valuable aspects of exploratory testing is that tests change based on feedback from the system under test. Some of these tests may be written down, some may be coded and some may only exist in the head of the person exploring the software. The feedback loop might be the time it takes to manually execute a test or the time between a code change and a CI server running a suite of tests.  Automated tests are one model of how a system works or should work at any point in time and not a concrete set of criteria that have to be met.

It’s the learning from test execution and adapting to new information that can make testing truly powerful.  If you limit that learning to manual testing, you are limiting your testing.

About these ads
This entry was posted in Agile, Automation, Exploratory, Testing. Bookmark the permalink.

2 Responses to All our testing should be Exploratory

  1. Chris Morris says:

    I have had a lot of value from random testing. It is true that intelligent exploratory testing can find errors that random testing does not – but the converse is also true.

    This is easy to do with a web app – I have a spider, that gets a page, and chooses at random a link to follow or a form to fill in, then starts again with the resulting page. In order to get deep into the application, it needs some hints to fill in appropriate form values (or randomly use an inappropriate value). It is a really good way to find code injection defects, and finds other defects too.

    The academic literature frowns on random testing – but testing became a research topic back in the days when programmer time was cheaper than CPU time.

  2. I believe automated testing is the best solution in all cases!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s