Five Things Security Teams Wish They Could Test Between Annual Pen Tests

Every security team has a list. Things they would test if the time, the budget, or the tooling allowed. They do not. Four of the five items below are drawn from NopSec customer conversations. The fifth is the meta-question every CISO has asked.

1. The shadow IT and developer-stood-up assets nobody scoped

The most common findings in NopSec’s pen testing practice are the assets — and more often, the services on the assets — nobody knew existed. From a recent NopSec Attack Path Analysis webinar, Shawn Evans, NopSec Head of Security Research, described the pattern:

“Developers oftentimes have SMB shared resources… a dev box with read-write access enabled to a web application that’s in staging… If I just have simple basic domain access, all I need to do is upload a JSP to this web server and suddenly I have remote command execution on a very critical system.”^[1]

The cost is exactly what Shawn described: a path to RCE on a critical system, originating from an asset that did not appear in any inventory. Annual pen tests run against the asset list the engagement was scoped to — not the dev box that was stood up six weeks ago, not the staging environment that was never in inventory, not the SMB share someone enabled for a quick proof-of-concept and never disabled.

What you want is a test that maps your external attack surface against what is actually reachable, not against what is on the asset list.

2. The legacy systems and misconfigured trust nobody owns

From the NopSec customer pain point document: “We have no visibility into our public cloud configurations.”^[2] From the same Attack Path Analysis webinar: “misconfigured trust… poor GPO policies that make certain resources overly trusting… default passwords, default example scripts…”^[1]

Legacy systems persist year-over-year while ownership decays. The application that “should be deprecated” is still running. The GPO set up for a project that wrapped up three years ago is still applying. The default credentials on the appliance nobody updates are still default. Industry breach research has consistently identified misconfigurations and human error among the leading drivers of security breaches.

Annual pen tests catch these shadow assets on the year somebody specifically negotiates them into the engagement Statement of Work. They do not catch the slow drift between engagements.

What you want is external validation that surfaces what is reachable in your environment, on a cadence that matches how ownership of legacy systems decays.

3. Whether the patch actually closed the path

From the NopSec customer pain point document, item #10 was: “We spend hours validating if a vulnerability is patched correctly… We still have to manually re-scan every time a fix is applied.”^[2]

A scanner re-scan tells you the CVE signature is gone. It does not tell you the exploit path is closed — whether the service restarted cleanly, whether a dependency update introduced a different path, whether residual artifacts from the pre-patch exploitation wave are still resident on disk.

Re-validation is not on the original engagement Statement of Work. A new engagement requires another scope negotiation, another budget request, another quarter on the calendar.

What you want is the ability to re-run the same scoped test after remediation and confirm the path is closed. Not in three months. Now.

4. Which vulnerabilities are actually exploitable in your environment

This is item #6 in the NopSec customer pain point document, and it may be the most common pain across the customer base: “We have no idea which vulnerabilities are actually exploitable. Half the time, we patch something that was totally internal anyway.”^[2]

Most teams work with hundreds of thousands of CVE findings from their scanners. A smaller fraction matters in any given environment. A smaller fraction still are actively exploited in the wild. NopSec’s research has consistently shown that the high-CVSS celebrity vulnerabilities that dominate news cycles are a small percentage of the vulnerabilities actually being exploited.^[3]

The annual pen test surfaces exploitable paths against the assets in scope. The rest of the CVEs sit unvalidated, with no signal about which would matter to an attacker in your environment.

What you want is validation that maps CVE findings to actual attack paths, not abstract severity ratings.

5. Did the pen test miss anything?

This is the meta-question every CISO has asked, sometimes silently. Your testers had a week. Your environment is bigger than that. Compliance signed off. Risk did not.

Your testers did the work in the time available, prioritizing the assets most likely to yield findings. The report describes what they found, not what they did not have time to look at.

The cost of waiting twelve months is not theoretical. The IBM Security Cost of a Data Breach Report 2024 puts the average global breach at $4.88 million, with a mean time to identify and contain of more than 250 days.^[4] NopSec’s own research, citing the SANS Incident Response Survey, documents that 14 percent of firms take 30 to 180 days from compromise to detection.^[5]

Adding coverage in next year’s engagement does not retire the question for this year. The hosts that did not get tested will not get tested by extending the scope in the engagement Statement of Work.

What you want is the same methodology, applied to the rest of the asset list, on a cadence that does not require a procurement cycle.

What changes when validation does not require a procurement cycle

The list is not the problem. The list is honest. The problem has been the cadence and cost of running the test each item warrants.

NopSec Continuous Adversarial Emulation is built for this list. Same five-phase methodology our pen testing team has run for eighteen years, executed by purpose-built agents under human oversight. Run in hours, not weeks. The output is a real pen test report with documented exploitation paths, remediation guidance, and a complete attack log.

Pricing and early access

$2,999 per test through the Pentest Starter tier. Early access is now open.

What to do next

Stop adding items to that list. Start running the tests. Get early access to NopSec Continuous Adversarial Emulation at nopsec.com/resources/adversarial-emulation/continuous-adversarial-emulation/.

Sources

1. NopSec Attack Path Analysis webinar with Shawn Evans, NopSec Head of Security Research. ↩ ↩
2. NopSec Customer Painpoint document. Items #6, #9, and #10. ↩ ↩ ↩
3. NopSec, “Don’t Let Celebrity Vulnerabilities Steal Your Focus,” white paper, November 2021. ↩
4. IBM Security, “Cost of a Data Breach Report 2024,” published in collaboration with the Ponemon Institute. ↩
5. NopSec, “Cyber Exposure Management Whitepaper,” October 2021. Citing the SANS Incident Response Survey. ↩