uses cookies to make interactions with the Company’s Websites easy and meaningful. When you visit one of the Company’s Websites,’s servers send a cookie to your computer. Standing alone, cookies do not personally identify you; they merely recognize your Web browser. Unless you choose to identify yourself to, either by responding to a promotional offer, opening an account, or filling out a Web form (such as a “Contact Us” or a “Free Trial” Web form), you remain anonymous to the Company. Please go to our privacy statement for details.


Creating a Vulnerability Management Program – Penetration Testing: Valuable and Complicated

Code on Monitors - Penetration Testing Feat Img

Once you’ve started a vulnerability scanning system, you may want to take the next step in identifying vulnerabilities: penetration testing, commonly referred to as pentesting. Gartner’s definition[1] of pentesting says it “goes beyond vulnerability scanning to use multi-step and multi-vector attack scenarios that first find vulnerabilities and then attempt to exploit them to move deeper into the enterprise infrastructure.”

Pentesting is a long-established method of independently verifying an organization’s ability to detect and defend against attacks. A company might do so to:

  • Assess where its security program stands in relation to potential threats
  • Validate the efficiency of its security controls 
  • Continue its vulnerability management as new platforms and applications are added to its infrastructure

In addition, companies may need to demonstrate their compliance with regulations, such as the PCI Data Security Standard, to avoid potential legal liability in case of a breach. Or they may simply wish to meet security standards established by such bodies as the National Institute of Standards and Technology (NIST) and the Center for Internet Security (CIS).

Because pentesting has been around for decades as a manual service carried out by vendors hired directly by companies, the bulk of this discussion will focus on how to select such a vendor. However, there has been a movement of late to include automated pentesting applications as an alternative or a sort of hybrid of the two, Penetration-Testing-as-a-Service (PTaaS). Many of the following considerations apply to all options.

The Basics of Penetration Testing

Pentesting can be as broad or narrow as the client wishes. It can focus on external or internal networks or both, external and/or internal web-based applications, and other points of potential exposure that are not relevant to all organizations, such as wireless networks or point-of-sale (POS) hardware.

  1. White Box Penetration Testing

Also known as internal or clear box testing, white box penetration testing is the method in which the tester has complete information on the IT systems they are testing, including: infrastructure, source code, and environment. The tester often also has internal access to the organization’s applications and uses this to test the vulnerabilities any employees can accidentally or purposefully exploit. This is an extensive method that also includes the quality of code and basic design. 

  1. Black Box Penetration Testing

In the case of black box pentesting, the tester has little, if any, prior knowledge of the organization’s underlying infrastructure, applications, or source code. The tester uses their own skills to identify and exploit vulnerabilities using trial-and-error. This more closely simulates an actual cyber attack. 

  1. Gray Box Penetration Testing

Gray box pentesting, like the name suggests, is a hybrid method between white and black box testing. Using this method, the tester has some knowledge and limited access to the test organization’s environments. Like black box pentesting, gray box pentesting simulates external attacks to identify real vulnerabilities and how they can be exploited.

Pentesting organizations vary significantly, from global firms offering services in many countries to the regional firms that confine themselves to limited territories. Size may or may not reflect an organization’s ability to run effective tests. The key is to know the experience of the testers who would be assigned to your project – something to ask about in RFPs, as we’ll note in a minute. Be aware as well that total years of experience of a tester is not as important as the experience the tester has in the area you want tested. A 30-year veteran of testing networks may not be as valuable to you as a 5-year veteran of cloud testing if you’re mainly concerned about your exposure in the cloud.

Reaching Agreement on What is to be Tested 

Before contacting pentesting firms, you first need to establish within your organization what should be tested. Pentesting can be disruptive. You don’t want to surprise business executives in another part of your company when they learn a key operational aspect under their management has lost functionality due to testing. Additionally, there are certain potential risks in running a pentest that must be understood and accepted throughout the organization before agreement is signed.

There is a long list of issues to consider. Here is a starting point: 

  • What types of assets should be tested? What assets are off-limits? Are there assets that can be tested only during certain hours of the day?
  • Which environments should be included — test or development environments or all environments, including production?
  • Should cloud services be tested? (Note that the cloud providers likely have limits on what they allow their users to test. Some may not allow any testing.)
  • Do you want to test for vulnerabilities for remote workers? Will that cause disruptions or conflicts with the worker’s ISP? (Again, you’ll want to clear this with the remote workers beforehand, too!)
  • Are there agreed-upon times to test or not test across the organization?
  • What is your organization’s risk tolerance regarding the availability and integrity of assets?
  • Where should the testers be located – remote or onsite?
  • If a significant vulnerability is discovered, how should the test team contact the project lead? (Emails may not be sufficient.)
  • How much autonomy do you want to give the vendor? Can the testers try phishing emails or phone calls with your employees, for example? This means a higher level of risk, but can result in a more thorough test.

Questions to Ask in Hiring a Pentest Vendor

 Your organization likely has a well-established method of producing requests for proposals, and you’ll want to be as thorough with your pentest RFP as you would with any other. A few salient points to keep in mind: 

  • Be as concrete, specific, and detailed in the questions you ask them to answer as possible. You will need to compare them on similar features and services so you know exactly what you are purchasing ahead of time.
  • Ask for clear evidence of their qualifications for your project specifically, not just any pentesting case. You’ll want to know if they’ve tested for organizations similar to yours with the types of sensitivities and idiosyncrasies particular to your industry or type of operation.
  • Particularly important in light of the tester experience we mentioned earlier, is who will be assigned your project? What is their familiarity with the areas you want tested? The name of the vendor is less important than the background of the people doing the testing!
  • Ask for sample final reports and additional materials to see if the reports seem like they’re prepackaged or truly show customized results.
  • Once you get a report, the work has just begun. How will they provide guidance on remediating any issues they find?

Exercise caution when considering a low bid rather than a high one. The low bid may show fewer days assigned to the project, which could suggest the vendor will rely heavily on automated tools to vet many vulnerabilities, but those results could be of low value. Vendors who spend more days on a project may be using the time to dig deeper into your infrastructure and provide more valuable information as a result.

The Downside of Traditional Pentesting and the Rise of PTaaS/Automated Pentests

 As you may have gathered, pentesting as described can be expensive. What’s more, it is a static test, showing where your organization was at a point in time. And the results may not be easily incorporated into remedial actions.

This is why PTaaS has emerged as a competitive offering. PTaaS provides a cloud-based platform for running and sharing pentest results, with a team of testers on the PTaaS side doing the testing. Many, but not all, of the tests they run are automated. Some PTaaS firms claim that their staff testers are well-versed in specific industries, just as the traditional vendors claim. PTaaS clients can see the results in real time and, in many cases, automatically remediate the issue through the platform.

PTaaS vendors also claim to have a larger base of testers to call upon, with the advantage of being able to hire anyone from anywhere to run the tests. There are also crowd-sourced pentesting firms that throw open the doors to a much larger group of testers.

PTaaS can also perform ongoing testing more affordably than the traditional pentest firms. An automated pentest system, purchased by a company, is the cheapest option. But keep in mind that anything with an automated function may have limits that human-driven testing doesn’t. Most automated tools, for example, are not able to run tests on wireless networks and web apps, and they aren’t equipped to do the social engineering testing – phishing and the like – that the traditional firms can perform.

As we’ve noted, the road to peak cybersecurity protection is long. See how other leading cybersecurity teams are succeeding by reading the 2022 State of Vulnerability Management Report.



Question #1: What is pentesting?

Answer: Penetration testing, or pentesting, goes beyond the basic vulnerability scanner. Pentesting is a manual or automated exercise that provides visibility into misconfigurations or other vulnerabilities that could lead to a successful cybersecurity attack. 

Question #2: What is Penetration-Testing-as-a-Service (PTaaS)?

Answer: Penetration-Testing-as-a-Service is a managed service offering that provides pentesting ad hoc or on a scheduled basis. 


If you haven’t read the previous installments of this series you can do so here:

Next: Creating a Vulnerability Management Program – Cybersecurity Risk: Why You Need Both Vulnerability and Threat Assessments


[1] “How to Select a Penetration Testing Provider,” By Toby Bussa, Claudio Neiva, and Mitchell Schneider, Sept. 11, 2020, Gartner.

Schedule a Product Demo Today!

See how NopSec's security insights and cyber threat exposure management platform can organize your security chaos.