How To Create An Awesome Application Security Report


If you want to excel as a pentester, it is not enough to be a highly technical security expert. You have to be able to produce high quality reports that effectively communicate an application’s security state to a client. In this post, I go over some of the key components of an application security report and give you some writing tips to improve your main deliverable as a pentester.

The application security report is the primary way that you share your work from a pentest with a client. You may spend days or weeks testing the security of an application, but at the end of the test, all the client will see is what the report says. If your report lacks attention to detail or is short on facts, it is going to reflect poorly on all the hard work you put into the test. In addition, if your report fails to effectively communicate security issues, the client made not fully understand existing vulnerabilities, the risk to the application, and how to properly remediate findings. If they can’t make proper security decisions based on your report, you may end up leaving the organization in a worse position than when you started the test.

In general, the purpose of the report is to communicate the overall security posture of an application. It is a point-in-time snapshot of the application’s security that describes how the application is vulnerable, the risk of it being exploited, and a plan for improving its security. After reading a report, the client should understand exactly how the application measures up to industry best practices and be able to use that information to develop an approach for shoring up any security weaknesses going forward.

Audience
Before we get into the specifics of the application security report, it is important to keep in mind who the audience will be. One of the first rules of communicating is to know your audience and writing the application security report is no different. Each audience for a report will typically bring different interests and goals when it comes to reading and it is your job to help tailor the report accordingly. Typically, there may be any number of audiences for a report including:

  • Developers
  • IT staff
  • Auditors
  • Project managers
  • Managers
  • Executives

Let’s talk about a few of these groups and some of the concerns they bring while reading the application security report.

One of the primary consumers of the report will be developers. They are primarily concerned with fixing any vulnerabilities identified in the report. When they read the report, they are looking for clear remediation steps for issues. The more specific the remediation the better. Many solutions are platform dependent, so if they can receive the right fix for their technology, all the better. They also are interested in further researching the issue since they may not have a security background. Any URLs that point to additional references on the topic can be helpful in this regard. Finally, since they will want to be sure they fixed the issue properly, they will want to know how to reproduce it. Step-by-step information including screenshots on how the vulnerability can be exploited is important in this regard.

A second primary audience for the report is managers and IT staff. These are the people that help bring the application live and keep it running. Security is not likely their primary concern and in some cases, security may be seen more as a roadblock than something beneficial. For this group, they are concerned with understanding each finding at a high level, so findings that are written too technically or don’t contain summary information may not resonant. Since they are concerned with keeping their application up and running, impact and risk levels are of primary importance. A high risk finding may be a show stopper when it comes to going live so don’t be surprised if this group tries to negotiate over risk. They are also concerned with the impact to other systems so you may be asked to provide feedback on whether the vulnerability has reach beyond the current application.

A third important audience for your report is upper management and executives. These may be the CISO’s, VPs, or CIOs and CTOs of the organization. They are paid to be concerned with the big picture including the overall risk the application poses to the entire organization. They are likely to be less concerned with any one point-in-time test than the overall trend of application security over a period of time. For that reason they are interested in metrics over time. Anything you can do to provide numbers, charts, and graphs will go a long way. Since this group also controls a budget, they are likely to be interested in the cost of mitigations. With their high level focus, this group may only read the executive summary section of the report.

 

 

 

 

Report Sections
Although there are many ways to organize an application security report, most have several standard sections that you will want to include:

  • Executive Summary
  • Contact Information
  • Technical Report
  • Methodology
  • Appendix

Contact Information

Don’t assume that the client has your contact information in their rolodex. Clients should be able to pick up the report and look up your number or email if they need to get a hold of you. It is usually good to provide at least a couple of numbers in case one person is not available. It is common to see both the tester and their manager’s contact information in the report. This is also a good place to track draft and version numbers. This can include internal drafts such as peer reviews, and changes made after the client has reviewed the document.

Executive Summary

The executive summary is a high-level, non-technical overview of the test. Anyone without a technical or security background should be able to read this section and have a good understanding of the report and its conclusions. The section should start with a description of what was tested and the approach that was used. If there were any limitations during testing, this a good place to point them out. Next provide a summary of the findings, highlighting the most severe vulnerabilities. There is no need to discuss all the findings or focus on lower-risk issues. A very essential part of the executive summary is an overall assessment of risk. Consider this the answer you would provide the CISO if he entered an elevator with you and asked you if he should be concerned about the application you just tested. The executive summary is also a good place to put tables and graphs as well. This is a great way to visually summarize information while including metrics that upper management is accustomed to viewing.

Technical Section

The technical section of the report is where you do a deep dive into each of the vulnerabilities you discovered. This section is organized by vulnerabilities, usually ranked from high to low risk. Each vulnerability will include a description of the issue, supporting evidence for the finding, the impact and risk level, the recommended remediation, and additional references. This is a chance to show off your expertise while doing a thorough and detailed job of explaining the issue to the client. Although you can read this post for how to write up findings, one thing to keep in mind is to avoid using standard boilerplate text that you copy and paste. The client is paying a lot of money for this test so take the time to add specific information to the finding such as how it applies in their environment, how the risk is related to factors such as the sensitivity of the data and the application risk level, and how the remediation will work for the specific application platform and infrastructure. It can be very helpful to have a summary table at the beginning of this section that lists all the findings and the risk levels.

 

 

Methodology

Although each pentester may take a slightly different approach when it comes to testing, you should be following a standard methodology. Your methodology ensures that you provide thorough coverage and consistent results while testing and it gives you credibility with the client. Describe your methodology to the client in this section so that they will be clear on what was and was not tested. If you follow a third-party approach such as OWASP, indicate that here. Provide some detail on the kinds of tests that were performed. At a minimum you can categorize what was tested. In addition, provide information on what tools you used during the test.

Although the methodology should be discussed prior to testing usually during scoping, it is good to have this documented for future reference and for readers who may not have been privy to earlier conversations. Ensuring that the client is clear on what tests were in scope and the approach used can help a tester avoid misunderstandings.

Appendix

The appendix is the place to include any additional information that doesn’t fit well in other sections. It might include a disclaimer about confidentiality, sensitivity, and how the report is allowed to be distributed. It may include a proviso that talks about the point-in-time applicability of the report. The client may want you to include an independent assessment letter in case they are required to provide it to a third party. You can also include details such as how you calculated risk or impact levels. This also is a place for more detailed information about what was tested such as URLs, accounts, and roles.

Additional Thoughts
One thing to keep in mind is that since the application security report is often a required deliverable according to the contract, it should be produced with the highest of standards. The report should be:

  • Detailed
  • Comprehensive
  • Informative
  • Easy to read
  • Error free

You should always have at least one other person peer-review the document. After you work on a report for a while, it will be easy to miss errors that someone with a fresh set of eyes will catch. It is also good to get another perspective on issues, for example on risk ratings. What may appear to be a high risk finding to you may actually be more of a medium risk when compensating controls are taken into account. A peer review may remind of these factors. In addition, make sure you perform a spelling and grammar check on the document. With technology, there are no excuses for spelling and grammar errors. These communicate a sense of sloppiness to the client and they detract from the excellent work you did during the testing.

In terms of report preparation, I find that it is best to prepare the report while you test. If you wait until testing is finished, you may be pressed for time and have to rush your writing. Or in other cases you may forget some items if you haven’t taken fastidious notes while testing. At a minimum, try to write up each finding as you go and then you’ll have the technical section of the report pretty much finished by the time testing is over.

Report writing is not everyone’s favorite part of pentesting, but is a necessary component and when done properly it can positively reflect on you and the work you do. Put in the extra time and energy to give the client a high-quality, value-add report and they will be very appreciative and likely more secure in the long run.

Good luck and happy reporting!