How do we minimize variance in our tests? How do we ensure that if two of our team mates were to test an application, they would find the same holes? That the results would be independent of whether it was tested by a senior or a relative junior?
That seems impossible - a senior would of course find more holes -, and I admit we haven't solved the problem yet. But, this is important to our clients. Clients rightly demand that the testing methodology and the test engineer's skills should be good enough to minimize variances in the tests.
Today I want to discuss the ideas we have been experimenting with and hear your thoughts.
On to the options:
- Training: Ensure all engineers are equally well-trained, that everyone knows the methodology and tests thoroughly. We do a lot of training here, but we know that variances still occur. Two engineers might understand the same technique a bit differently, and this could lead to variance. Training alone, we feel, is inadequate to minimize variance. It's an obvious first step.
- Quizzes: This is the simplest method, and something we do regularly. Conduct periodic quizzes to see variances between engineers. Analyze the questions where there's maximum variance. That tells us where to focus our attention on to reduce variance. Quizzes have limitations: two engineers might just interpret the question differently and give seemingly incorrect answers. And a written quiz is no guarantee that careless errors will not occur in a live test.
- Test the same application twice: Let two engineers work independently on a live project. Do they arrive at the same findings, or is there a difference? If our 50 engineers were split into 25 pairs and tested different apps, we would understand our variances better. The approach sounds attractive, but it's very difficult to do in practice. We are perennially short of people, and staffing 25 projects with additional team members is a luxury we can't afford yet.
- Have all the engineers test the same sample app for a day: Same as above, but simpler to implement. Set up a sample app (an open source one like Wordpress for example), and have all 50 security testers go after it for one day. Check the variance. We love the sound of it, especially the whiff of "Capture the Flag". We might do this one of these days, but we are just worried that a 1-day simulation is a far-cry from a 10-day test.
For now, quizzes and training are the techniques we use most to measure and reduce variance. We have done a bit of option 3, but not across the entire team. And we haven't tried 4 yet. We'd love to hear from you the methods you follow to reduce variance in your team's work.