For application security testers, there is a ton of great material on the Internet and elsewhere about what to do during a security test. If you want to test for SQL injection, there are a million guides that will walk you through the steps. What I’ve found is that there is a lot less discussion of what you shouldn’t do during a test in order to avoid mistakes or to stay out of trouble.
This is an important discussion because there are so many things that can go wrong during an application security test. A request to test a web application is basically permission being granted to break a piece of software in unexpected ways. If you are not careful about how you go about a test, a lot more can break including the supporting platform and infrastructure. You can also make a lot of people unhappy and create a lot of extra work if you don’t plan carefully. You don’t want to be that guy who is called into the CISO’s office to explain why the company is losing money because a production system went down or why thousands of their most important customers are getting unwanted emails sent to them.
Although there is almost an unlimited number of things that can go wrong during a test, I’ve put together my top 10 list. A lot of these lessons have been learned the hard way over the years either through my own actions or having watched my colleagues at work. If only someone had shared these kinds of concerns with me when I had started my career, I might have been saved myself some headaches.
1. Failure to scope a test
Before any security testing begins, you need to be on the same page with the client in regards to what needs to be tested. This is what scoping is for. Just because a client gives you a URL doesn’t mean you are ready to start testing. Although there are many questions that need to be answered during scoping, one of the big questions is what exactly are you testing. Failure to answer this question can lead to you missing something during a test. For instance, there may be an API that is called by the web application that the client is expecting you to test but that wasn’t included among the URLs provided. At the same time, there may be components that the client doesn’t want you to test such as a configuration page that controls settings for many environments including production. Scoping an application before testing it addresses these and other questions and makes sure that there won’t be any surprises when you deliver the report.
2. Failure to do a walkthrough
For all but the simplest of applications, it is highly recommended to get a walkthrough of the application before you start testing. You should never go into a security test thinking you can just figure it out during testing. For one, you may waste precious time trying to understand some type of functionality in the application. Secondly, you may miss a vulnerability if you don’t understand how the application works. This is commonly seen with complex workflows. If you can’t trigger all the steps in a workflow, how can you expect to fully test it? A walkthrough also helps make sure you understood the scoping (#1 above) and allows you to ask additional questions in that regard.
3. Starting a test without notifying the client
The trap you may fall into here is once the client gives you scoping information, a walkthrough, and a URL, you think you can immediately begin testing. There may be any number of things that a client needs to happen before a test such as taking a backup of the database, notifying the incident response team, preparing test data, setting up IP address whitelisting, and notifying UAT or other testers using the same environment. If you fire off Burp’s spider before these things are in place, you may be creating a big headache for your client and a potential delay for their release.
4. Not updating the client during testing
There should be continual contact between you and the client during an application security test. Communication ensures that you address any issues that come up during testing and that there are no surprises once the test is complete. Many clients will want to be notified of high risk issues almost immediately, so that they can be start working on the remediation. The last thing you want is to delay a mission critical app from going live because of a high risk finding that you surprised the client with in a report issued a week after the test finished.
5. Changing administrative settings during a test
This is something that needs to be carefully clarified during scoping of the application. Many applications have administrative functionality with settings that if modified by you or an automated scanner, will impact the configuration of the application or its infrastructure. Sometimes these settings may even impact other environments including production. Unless you want to risk bringing down a production environment, discuss the scope of admin functionality with the client before the test.
6. Testing out of scope functionality
In today’s mashable web, it is not unusual to see a web application make calls to many third party domains. Even though you may see a lot of traffic going back and forth between the web application and other domains, don’t be tempted to shoot off a Burp Active scan at the domain. Unless the client has given you explicit permission to the test a certain domain, don’t touch it. This includes subdomains of the domain you are testing. Testing related domains may require prior notice and agreements and any unexpected testing can wreak havoc for your client and their partners. Out of scope functionality may also include other backend components that support the application. For instance, don’t upload that web shell to a server unless you have explored this situation with the client.
7. Testing in production
There may be times when a client wants you to test in production. Unless they are holding a gun to your head, push back against this poor idea. There are so many things that can go wrong and there are many good alternatives to a production test. You are dealing with live data in production and no matter how careful you are, it can easily be corrupted. Or if you accidentally bring down a server or impact other systems, the fallout is so much greater than if it had happened in a staging environment. If you must test in production, make sure you get all the signoffs and then pray that the gods have favor on you.
8. Not taking screenshots
Part of your job as an application security tester is to carefully document your work. Environments tend to be in a constant state of flux and if you find a vulnerability and forgot to document it and take screenshots, you may have missed your opportunity. Perhaps a code release causes the vulnerability to change in some way. Now you may have to go back and retest the issue which will cut into your valuable time. Even if the developer somehow fixed the finding while you were testing, you still want to document it so that the organization is aware that there was an issue. For a tester, getting caught without screenshots is like getting caught with your pants down.
9. Triggering side effect
Automated tools like Burp are pretty amazing in how they can outperform human testers in many ways. However one area where the humans still win is in understanding the potential side effects of our actions during testing. There may be actions in the application that when triggered a single time have no real affect, but if triggered a 1000 times during a Burp scan, the consequences could be significant. An example is a form that triggers emails. If through Burp you unintentionally send off emails to a thousand clients of your client, you are going to be that guy again in the CISO’s office. Or if you lock out a thousand user accounts in production and shut down business and affect a company’s bottom line, you might be that guy in the CEO’s office.
10. Disclosing a report or findings
During and after a test when you are exchanging information about findings with members of the client’s team, you may forget just how sensitive this information is. Simply put, information about vulnerabilities or the vulnerability report itself are a roadmap that an attacker can use to attack your client’s organization. This information needs to be treated with the utmost care. You should not leave reports lying around and you should clean up your laptop when a test is over. You don’t want to be reading about your client the next day in a major security blog after you lost a vulnerability report on the way to work.
Obviously there are many other things that can go wrong during a security test besides these ten items. These are the ones that stand out most to me from my years of testing. If you have other items that you have seen come back to bite you during testing, drop us a line and I’ll update the list at a future date. Good luck, stay out of trouble, and happy testing.