Testing & Software Blog

If you have an adversarial relationship between testers and developers, you’re doing it wrong. Once got asked in a job interview “How do you avoid going native?”, and thought that was a rather ridiculous question. The common enemy is the bugs.

“Exploratory Testing” isn’t some revolutionary new technique, but describes what I’ve actually been doing for years. Ticking boxes on a test script is not only dull work, but you’re far less likely to discover any significant bugs.

If you’re supposed to be testing an interface with a third party product, and the third party product starts returning messages like “Object reference not set to an instance of this object”, then you’re also performing acceptance testing on the third party product regardless of whether that was supposed to have been within the scope of your testing.

Just because the functional specification says it’s a common function does not necessarily mean that it’s been implemented as a common piece of code.

Test automation is an awful lot harder than either management thinks it is, or the tool vendors claim it is. Scripting high-volume repetitive tasks on a small part of the system adds a lot more value than attempting to automate an entire end-to-end process.

Using metrics related to bugs logged to measure performance of developers is just asking for trouble. It just politicises the process of logging bugs and does nothing for software quality.

Sanitising your test data because somebody needs to use one of the test databases for a pre-sales demo is occasionally part of the job. So is verifying that the customer data that’s supposed to have been scrambled and anonymised actually has been. Good way to test this is to copy and past a name and address into Google, and see if it finds a match. So there is no such address as “Apocalypstick Avenue, Elektra City”. Turns out there really is a Platypus Road, but not in Scunthorpe. Their local authority’s pest control department is unlikely ever to have to deal with an infestation of tribbles…

Thinking of putting a few of these on my LinkedIn profile to make it less boring.

One of the Twitter testing community, @JariLaakso, posted this question:

Two trains are going to crash. Brakes don’t work. What would you do?

Of course, you can’t answer this without asking a lot of questions to establish context.

• Are you on board one of the trains? If so, are you a passenger or a member of the train crew? If train crew, are you in the driving cab, or elsewhere on the train?
• Is your train actually moving, or is it stationary and about to be hit by the other train?
• If you’re not on board the train, are you a bystander, or someone like a signaller?
• How long before the collision? Just seconds, or longer than that?
• Is there anyone on board the trains at all? Perhaps it’s a staged crash for a film?
• Is the “crash” even an impending collision? Or is it a case of software gone blue-screen-of-death meaning the brakes can’t be released and the train needs rebooting before it can go anywhere? This may sound silly, but I’ve been on a Virgin Trains Pendolino that had to do precisely that.

The actual answer turned out to be none of those things, but that’s not really the point. It’s about asking the right questions to get the information you need to be able to answer the original question “What do you do?”.

As an aside, real-life rail (or air) accident reports can often be worthwhile reading for a tester. I’m not talking about sensationalist reports of death and destruction, but the technical stories behind the accidents and how they occurred.

I remember reading L.T.C.Rolt’s classic “Red For Danger” at a formative age. It’s a very well-written and readable account of the evolution of railway safety throughout the steam age. It starts with the development of early primitive signalling systems from the 1840s onwards, and tells of the lessons learned from each successive serious accident. As the story moves into the 20th century, increasingly sophisticated systems from signal interlocking to better and stronger rolling stock meant far fewer disastrous accidents. But even the best systems can fail, with sometimes fatal consequences, and the book explains how.

It’s essentially the story of bugs.

Great blog post about exploratory testing by James Marcus Bach, and why some people Just Don’t Get It.

It’s difficult for them because Factory School people, by the force of their creed, seek to minimize the role of humanness in any technical activity. They are radical mechanizers. They are looking for algorithms instead of heuristics. They want to focus on artifacts, not thoughts or feelings or activities. They need to deny the role and value of tacit knowledge and skill. Their theory of learning was state of the art in the 18th century: memorization and mimicry. Then, when they encounter ET, they look from something to memorize or mimic, and find nothing.

Those of us who study ET, when we try to share it, talk a lot about cognitive science, epistemology, and modern learning theory. We talk about the importance of practice. This sounds to the Factory Schoolers like incomprehensible new agey incantations in High Elvish. They suspect we are being deliberately obscure just to keep our clients confused and intimidated.

As I’ve explained in previous blog posts, I’ve always taken an exploratory approach to testing, even if what I did wasn’t formally identified as such. Trying to force testing into a purely mechanical script-based approach not only sucks all the fun out of testing, risking disillusionment and burnout, but makes the actual testing less effective.

And while we’re on the subject of old-school techniques, are these guys for real? “Unlike a traditional development process, ours establishes all the system’s requirements before a line of code is written“. Seriously, folks, does anyone still try to develop software that way in 2011? Sounds like a perfect way to implement what the client thought they wanted eighteen months ago.

Remember that old cartoon of the swing hanging from the branch of the tree?

Another testing story of mine.

The two related projects were interfaces with external systems handling rent deductions and water billing respectively, both of them for a large overseas customer.

A major problem was that neither developers nor testers had any access to the third-party systems other than specifications for the file formats to be used in the interfaces. This made it impossible to perform complete end-to-end testing within the internal test environment. My biggest challenge as the tester on the project was to try to simulate the external system by creating input files based on those file specifications. The physical sending and receiving of files was beyond the scope of my own testing.

One warning flag was the way one of the sample input files didn’t quite match the file specification. This really ought to have been taken as an omen of how the project would unfold.

It was all flat text files with fixed-length fields, so I put together a suite of SQL scripts which I ran in TOAD (A third-party tool to access Oracle databases) to generate the input files containing the data our system would expect in response to the output files it had produced. These scripts covered various “Happy Path” scenarios, and I’d hack the files with a text editor to test various error conditions. This meant I could simulate end-to-end business scenarios from the perspective of our own system.

I laced my test reports with caveats making it clear that we hadn’t been able to test the full processes with an actual instance of the third-party system. So as soon as the system went into acceptance testing with the client, the response was a flood of issues and defects, almost all of them relating to flawed assumptions and understandings during business analysis. The fixing and retesting went on for more than a year, far longer than the initial development phase of the project.

I can’t honestly describe this as a happy and successful project, but it’s the nature of experience that you frequently learn more from something that’s gone badly from something that went well. While I feel I did as professional a job of testing as was possible under the circumstances, I still wonder how things could have been done better. I did ask my line manager whether the operators of the third-party systems had test instances of their systems available for developers of third-party interfaces, and never got any answers.

This is a good example of risk of using the Waterfall method of development for a project as full of assumptions and unknown unknowns as this one.

Another of my occasional professional-related blog posts. I have considered spinning these off into a separate blog, but for the time being, it’s another category on my personal blog. As I’ve explained before, what I do in my day job isn’t completely unrelated from some of what I do outside of work.

So, the all-time favourite bug?

This is a question I’ve been asked recently. While it’s difficult to single out one bug from all my years of testing experience, this one does stand out, and comes from one of the most challenging projects from my testing career so far.

The application was a work scheduling system, part of a very comprehensive housing management product. A call centre would take calls from tenants reporting broken windows or blocked drains, then the system would schedule appointments for operatives to carry out repairs. The operatives themselves could update progress on these appointments in real time using a mobile app, allowing the system to maintain the daily work plan dynamically.

It was a complex application, involving two third-party products, one of them the mobile app, the other, which formed the focus of my own testing, being the scheduling engine itself. This was a standalone system which took XML messages containing the call information, and sent back XML containing either the appointment, or the reason why it couldn’t create one.

One thing that became apparent early on was the scope of my testing didn’t encompass just our own product and the XML interfaces. End-to-end testing soon revealed a lot of bugs in the scheduling engine itself. We had some complex business rules for dealing with things like multi-person jobs, multiple jobs which had to be performed in sequence, or jobs where the call centre wanted to assign a particular operatives rather than let the scheduling engine choose. With a near-infinite number of potential combinations of conditions I performed extensive exploratory testing on increasingly arcane permutations. A straightforward example might be “Remove wasps nest” followed by “Carry out repairs to guttering“, which required two different trades but had to be performed in the right order. Some of the less straightforward scenarios resulted in the scheduler returning gnomic utterances such as “The optimal schedule does not include this call“, or, if you were really unlucky, “Object reference is not set to an instance of this object“. Unfortunately I could never find a test case that would repeat that last one consistently.

The most significant bug occurred when the schedule filled up. The dynamic nature of the schedule meant that sometimes appointments did get missed. For example, if a previous appointment ended up taking far longer than anticipated, and the system couldn’t find another operative available in that slot because everyone else was busy, the appointment would fall out of the schedule and need to be re-scheduled. The same would happen if an operative went sick or crashed his van during the course of the day.

Unfortunately it did the same thing when I filled up the schedule and tried to schedule further appointments. Rather than reject them with “No operatives available with correct skill” as it should have done, it decided the plan would be more ‘efficient’ if it appointed the new jobs and displaced existing ones it had already committed to. Not a situation which would lead to happy tenants, who would not be impressed if they’d taken an afternoon off work for a plumber who never turned up.

Given the severity, that bug was reported to the third-party supplier and fixed before the system went live on any customer site. But that particular scenario ended up in the suite of regression tests run each time we received a new release of the third-party scheduling engine.

Regular readers of this blog don’t really need to be told that I’m a very keen music fan and amateur rock critic. Writing about a small club-based scene I’ve come to know quite a few band members over the years. I’ve even had people suggest I should quit working in the IT industry and become a full-time music writer. But while being on the fringes of the music scene is can be a great experience, I’m not convinced I want to jump ship and join the circus.

But I can see a lot of parallels between music criticism and my professional career as a software tester.

Not that I’m suggesting that testing and reviewing are exactly the same. To start with music is inherently more subjective than software. But there just as it can be a judgement call as to whether or not a piece of software is fit for purpose, it’s never completely subjective as to whether a record or performance is good, bad or indifferent. There are those that claim all opinions are equally valid when it comes to reviews, and there is no such thing as an objectively good or bad record. If you believe that, you clearly haven’t heard Lou Reed’s appalling collaboration with Metallica. It seems to me that both testing and reviewing are something many people can attempt, and just about anyone can do badly, but take skill and experience to do well. You only have to look at the reviews on websites to which anyone can post without moderation to realise there are bad reviewers out there just like there are bad testers.

To review a record or concert requires both an understanding of what the artist is trying to achieve, and an honest assessment of how well they’ve succeed in achieving it. That in turn requires the equivalent of domain knowledge. Just like a lot of indie-pop reviewers come horribly unstuck attempting to review progressive rock or metal releases, ask me to review a dubstep or free-jazz record and I wouldn’t know where to start. But just as testers from different backgrounds will approach things from different angles and uncover different bugs, a reviewer with deep specialist knowledge of a specific genre will have a quite different perspective from one whose taste is far broader. Something that’s meant to have crossover appeal benefits from both viewpoints.

Then there is the issue of speaking truth to power, which can require both courage and diplomacy. Egos even bigger than those of developers go with the territory. When an artist has poured their heart and soul into making a record, they don’t always appreciate being told how their work could have been better. Much like the way developers don’t always appreciate being told the code they’ve slaved over is riddled with bugs they they really ought to have picked up in their own unit testing. And if you’ve ever had the misfortune to work in a dysfunctionally political environment where project managers surround themselves with yes-men and tend to shoot the messenger whenever those messengers are bearers of bad news, then you’ll recognise those over-zealous fans who sometimes try to vilify anyone that attempts constructive criticism.

It’s true that there are a lot of rock critics out there who exhibit exactly the same sort of adversarial behaviour that gives some testers a bad name. Yes, writing and reading excoriating reviews of mediocre records can occasionally be cathartic, but informed and honest constructive criticism is far more valuable in the long run. Just as software testing is a vital part of making sure software is fit for purpose, constructive criticism has a role in making music better.

Perhaps it’s my tester’s ability to see patterns, but what I hope the above goes to show is that sometimes what you do in your “day job” and an apparently unrelated activity you do in your spare time can have more in common than you think. Certainly there are transferable skills, especially those softer ones which are much in demand.