Another testing story of mine.
The two related projects were interfaces with external systems handling rent deductions and water billing respectively, both of them for a large overseas customer.
A major problem was that neither developers nor testers had any access to the third-party systems other than specifications for the file formats to be used in the interfaces. This made it impossible to perform complete end-to-end testing within the internal test environment. My biggest challenge as the tester on the project was to try to simulate the external system by creating input files based on those file specifications. The physical sending and receiving of files was beyond the scope of my own testing.
One warning flag was the way one of the sample input files didn’t quite match the file specification. This really ought to have been taken as an omen of how the project would unfold.
It was all flat text files with fixed-length fields, so I put together a suite of SQL scripts which I ran in TOAD (A third-party tool to access Oracle databases) to generate the input files containing the data our system would expect in response to the output files it had produced. These scripts covered various “Happy Path” scenarios, and I’d hack the files with a text editor to test various error conditions. This meant I could simulate end-to-end business scenarios from the perspective of our own system.
I laced my test reports with caveats making it clear that we hadn’t been able to test the full processes with an actual instance of the third-party system. So as soon as the system went into acceptance testing with the client, the response was a flood of issues and defects, almost all of them relating to flawed assumptions and understandings during business analysis. The fixing and retesting went on for more than a year, far longer than the initial development phase of the project.
I can’t honestly describe this as a happy and successful project, but it’s the nature of experience that you frequently learn more from something that’s gone badly from something that went well. While I feel I did as professional a job of testing as was possible under the circumstances, I still wonder how things could have been done better. I did ask my line manager whether the operators of the third-party systems had test instances of their systems available for developers of third-party interfaces, and never got any answers.
This is a good example of risk of using the Waterfall method of development for a project as full of assumptions and unknown unknowns as this one.
