How to Create Vulnerability Management Reports for Executives

Vulnerability Management programs produce a lot of data about the vulnerabilities and remediation efforts. Due to this, it is very easy and common for technical security individuals to drown their executives in all of the minute details of what’s going on. It’s important to understand the reporting needs of executives, before you start cobbling together spreadsheets and dashboards. The reports produced by the security team are the communication mechanism that often means the difference between additional resources being allocated versus not.

What do Executives Care About

To begin with, let’s discuss a few matters of fact regarding Executives. First of all, once you start reporting to C-Suites beyond the CISO level, technical comprehension drops dramatically. Talking about specific kinds of vulnerabilities that affect specific kinds of assets is wasted effort. Secondly, beyond the CISO level, C-Suites speak in terms of the business at an aggregate level. Conversations focus predominantly on metrics of revenue, efficiency, cost, and ROI. If your reports haven’t distilled your takeaways into this language, what you’re trying to communicate will fall on deaf ears. 

When reporting vulnerability management progress to Executives, that data should be focused on a few key metrics and answer more questions than they create. To put it simply, Executives want to know if security is getting better or worse and if their investment into such a program is worth the money. One hard, but true, reality to keep in mind is that Executives outside of the CIO and CISO realm tend to view security as a cost center for something that just has to be done. AKA, Executives entertain security because they must, not because they want to. Keep this in mind as you’re preparing reports for an audience that only cares because they have to.

Overall Risk

Let’s get into the specifics. The primary metric you’ll want to communicate up the chain should capture and convey the risk to the entire Enterprise. This allows Executives to get a high level picture of the current state of the Vulnerability Management program and how exposed the company might be to a breach. We recommend viewing the overall risk over time. This allows Executives to track progress which can be more enlightening than a snapshot view. For a nascent vulnerability management program, the trend should be an improving score as remediation efforts are formalized and the backlog of vulnerabilities worked through. For a mature program, the trend should be a consistent, low risk score. Variations from the expected trends can point to issues in the Vulnerability Management program that need to be understood and addressed.

At Nopsec, we provide our clients with an Overall Score for this purpose (see image for example below).  It is a weighted average of the business risk scores for every asset in the Enterprise. Each asset’s business risk score is a combination of the inherent risk of the vulnerabilities found on the asset, the asset’s business criticality, and any environment configurations that secure the network and assets. Therefore, our Overall Score is a measure of the overall risk from vulnerabilities across the Enterprise. Most importantly though, it’s a simple and to-the-point chart. Any Executive could look at this and only require a sentence or two of explanation to get the takeaway.

It should be noted that when looking at vulnerability metrics over time, the frequency of the measurement should be chosen appropriately to view trends in the data but not overwhelm those trends by short-term variability. Vulnerability detection and remediation efforts are expected to have cyclical behavior due to software development cycles. For example, Microsoft releases patches on a monthly basis (Patch Tuesday) so it is unrealistic to expect a Vulnerability Management team to address Microsoft issues more frequently.  The specific choice of measurement frequency will be dependent on the tech stacks and remediation processes used within the Enterprise.  But for executive level reporting, monthly reporting is reasonable.  

Track Ability to Meet SLA Goals 

Another useful metric is to quantify the ability of the Security team are vulnerability management SLAs. SLAs define the remediation timeframe expected for a given vulnerability instance based on properties such as the vulnerability risk level and business criticality of the system on which it resides. As such, the SLA objectives should be specific to an organization and set based on an organization’s risk appetite and any regulatory requirements. SLAs represent clear goals for the Vulnerability Management team to meet. 

SLAs provide a great opportunity to tell the story of how things are going, operationally speaking. Recall that efficiency is language spoken by Executives. By tracking how well a Vulnerability Management team is meeting the SLAs, Executives can infer if the team has the resources it needs and/or how efficiently the team is operating. Showing month in and month out that your team is unable to hit SLA targets provides the chances to make a case for additional investment and then justify it.

Here is how we display this at NopSec:

Notice again, we put emphasis on keeping these metrics simple and digestible. 

Metrics per Business Unit or VM Ownership team

For a larger organization, there may be multiple units that have vulnerability management responsibilities. In that case, each business unit should be tracked with the same metrics so that Executives can understand if there are any units performing more poorly than their counterparts and are in particular need of process, technology, or resource changes. It should not be assumed that each Vulnerability Management team will have the same workload. For example, a team that only needs to track that automated patches on windows laptops have been applied will have much less to do than a team dealing with a more diverse tech stack of servers and the software running on them. Therefore, metrics that track concrete goals related to vulnerability risk are more enlightening than metrics that merely track effort such as number of vulnerabilities addressed or tickets closed.

As mentioned above, this specific view is best reserved for larger, more complex organizations that have more mature security programs. In cases like this, usually Executives are more educated and in tune with security comings and goings. In subsidiary situations, it’s not uncommon for various business units to have their own dedicated Executives that come together to discuss the business as a whole. In such hierarchical scenarios, viewing security metrics by business line is a justified level of additional granularity.

Here is how we display this information in such cases:

When to Show Deeper Data

When all metrics are looking good, a few key metrics should be all that is needed.  But when metrics aren’t trending the right way or goals aren’t being met, Executives will want more information. At this point, one needs to dig a little deeper into the data to understand the why and provide courses of action. That may include: 

• Looking at specific remediation efforts to make sure process steps aren’t being skipped, 
• Determining if specific vulnerabilities are being repeatedly addressed, which could point to failure to address the vulnerability at its initial introduction in the software development or deployment cycle, 
• Determining if workloads match resourcing,  
• Considering if automating certain types of remediation efforts may be useful, and
• Identifying systems that have reached obsolescence and should be replaced.

While Executives don’t need to see the raw data that are used in these explorations, they do need to understand what the data says and the courses of action they point to. In cases like this, the individuals responsible for the reporting need to be prepared to articulate the layers of the onion. Should Executives begin to ask deeper questions, avoid treating the situation like the floodgates have opened and immediately drown them in technical information. Expect Executives to drive at the specific answer they are looking for. Add supporting technical context where appropriate, but tow that line. We’ve found that this leads to the best outcomes for both parties.

Conclusion

Reporting is a tricky thing. What you show and how you show it is completely contingent on the audience that will be receiving it. Executive audiences are probably the trickiest group of individuals to articulate usually very technical information to. That is the challenge Vulnerability Management teams are up against when they showcase what’s going on in their world. Remember to keep things simple and to the point, but most importantly tell the vulnerability management story in a language that the Executive audience will understand.