Once developers fix the holes discovered in a pen test, the app is thrown back at us for a re-test. "Confirm all holes have now been fixed".
How much time should we estimate for the re-test? We have tried out several formulae. I think we’ve got it right, finally.
First, the obvious approach: a confirmatory re-test just checks if the holes discovered in the first test have been closed, rt.? So, that should take us a LOT less time than the first test. After all, we are just checking a subset of the tests.
Thus a 30 person-day test that discovered 6 holes probably merits a 3-day re-test. Clients love that kind of speed.
But, wait... "What about regression testing?" you ask. Won’t fixing one thing probably break something else? Shouldn’t we run the app through an entire series of regression tests? Huh.
Well, turns out that’s really the least of our problems. And we learnt that the hard way 6 months ago.
We were testing a new app for an eCommerce company in Boston. The 15 person-day test revealed 10 holes. Two months later, the app came back for a re-test after the developers had fixed the holes. We took 3 days, and spotted that 4 of the holes were still open. We signed off on the other 6 holes.
First test: 15 days
Holes discovered: 10
Second test: 3 days
Holes still open: 4
A month later, the app came back for a re-re-test. The customer wanted to check that those 4 holes were finally closed. To cut a long story short, we were lucky to get 10 person-days for this third test. We found 5 holes. Two holes we reported after the second test were still open. But, worse, 3 holes from the first test were open too! And we had signed off on them a month earlier!
Third test: 10 days
Holes still open: 5 [ including 3 from the first test :( ]
What happened? No, the holes weren’t magically re-introduced between the 2nd and the 3rd test. They were there all along. Our 2nd test just missed it. If that isn’t embarrassing, tell us what is!
Handling the embarrassment was the simpler part – we came out straight with the customer, there’s nothing you want hide when it comes to securing an app. The tougher part was the questions that it raised about our testing process. We have invested 4 years refining our testing methodology and are quite proud about it here. And it looked right out silly to miss a hole we had discovered ourselves two months earlier.
But the post-mortem analysis was the best I’ve seen in a while.
Traditionally, the re-test engineer studies the previous report and prepares the test plan. Look at the holes from the last test, define tests to see if that’s been closed. There’s little time for a re-test, so the engineer focuses on the findings from the last test. Experience shows that’s fine. We (no the customer) have discovered fresh new holes after the first test – that test is very thorough.
The re-test engineer who spends just 2-3 days with the app cannot define test cases as well as the first pen-tester who spent 15 days on it. And herein lies the rub.
The re-test defined test cases very narrowly. For example, the first test reported a SQL Injection on the username field in the “Change Password” page. The developers fixed that. The re-test confirmed that the username field is safe against SQL Injection. But there were 3 more fields on that page: one for the old password, and two for the new password. The last of those was still vulnerable to a SQL Injection.
The test cases for the 2nd test were prepared studying the report, and the engineer didn’t think it necessary to check the remaining fields. The reported vulnerability was on the username, after all.
The test plan for the first test, and the 3rd test both included SQL Injection tests for the Change Password page. When the first test discovered SQL Injection in the username field, it stopped further SQL Injection tests on the same page and reported the finding. The report recommended that all inputs be protected against SQL Injection. The developers hadn’t done that – they just fixed the username field. But the 2nd test missed that.
The 3rd test, in contrast, used the test plan from the first test. And so discovered the SQL Injection in the obscure variable.
So where does all this take us to? What’s the right time estimate for a confirmatory re-test?
The confirmatory re-test requires as much time as the first test, save the time it takes for understanding the app, preparing the threat profile and preparing the test plan. The re-test can re-use those from the first test. But the test cases itself, it has to follow that from the first test.
If T1 = Time for studying the app, T2 = Time for creating Threat profile, T3 = time to create test plan, T4 = time to execute the tests, and T5 = time to create report
Time taken for first test = T1 + T2 + T3 + T4 + T5
Time estimate for the second test = T4 + T5
We have been following this for several months now, and it’s worked so far. Customers understand why it's better to verify against the original test plan and be sure even when it takes more time, than save a few days by testing narrowly against the test report, but still be uncertain.