In an age where cyber threats advance at an unprecedented pace, safeguarding digital assets has never been more important. Penetration testing, or pen testing, has traditionally been a stronghold in this defense strategy, a reliable method used to uncover potential vulnerabilities in a network or system.

This task, historically carried out by human experts with an in-depth understanding of cyber-attacks, is witnessing a fundamental shift. The cybersecurity landscape is rapidly evolving, bringing with it a new era where automation takes the lead in vulnerability detection and management.

Click the links below to jump to the section you would like to learn more about:

The Rise of Automated Pen Testing

As we stand at this crossroad, it is essential for organizations to understand the emerging trends and implications of automated pen testing, also known as vulnerability scanning. In recent years, there has been a significant shift towards employing sophisticated tools and software capable of simulating cyber-attacks with minimal human intervention to help identify vulnerabilities before they can be exploited.

The rise of automated testing offers the key advantages of speed and efficiency, allowing organizations to evaluate a wide range of networks at a frequency that was previously unattainable. This ensures potential vulnerabilities can be identified and addressed more swiftly, enhancing an organization’s defenses against cyber threats.

Unmasking the Dangers

However, the transition towards vulnerability scanning is not without its pitfalls. A few potential dangers associated with this shift include the fact that automated pen tests:

  • Only cover a fraction of the comprehensive six-step penetration testing process
  • Tend to miss complex vulnerabilities identifiable by human expertise
  • Are prone to generating false positives and negatives
  • Focus primarily on known threats, missing zero-day vulnerabilities and unique system weaknesses
  • Lack the ability to analyze an organization’s unique setup

A Fraction of the Full Picture

A standard, comprehensive penetration test is made up of six stages:

  • Intelligence Gathering
  • Threat Modeling
  • Vulnerability Analysis
  • Exploitation
  • Post Exploitation
  • Reporting

An automated pen test, which, as previously established, is another term for vulnerability scanning, falls under the third stage in this multi-step process. It is important to note that scanning is but one of many types of assessments that are often included during this particular phase. Other types of assessments include checking for default passwords and performing a web application analysis, among others.

In essence, relying solely on automated pen tests for your security evaluation means engaging in a fragment of a fragment of a full-scale penetration test.

Lack of Human Insight

Automated systems work on predefined algorithms and patterns. Despite their speed and efficiency, they might miss vulnerabilities that are not defined within their databases. Human experts, on the other hand, can approach a problem with a fresh perspective, intuition, and creative thinking, potentially identifying complex vulnerabilities that an automated tool might overlook.

Let's Get Technical

One example of this would be abusing default passwords to discover firmware version information that has additional high-risk vulnerabilities associated with it.

Take the Polycom VVX 400/410 devices running software version 5.3.1. After logging in with default user credentials of 123, you might enumerate the software version on the home page. Exploit research would lead you to discover CVE-2021-41322, a high risk privilege escalation vulnerability that allows escalation to administrative account access on the device.

A common network level vulnerability scanner would not detect this issue as the software version isn’t disclosed in HTTP headers or publicly accessible files. However, a human individually researching and attempting default user credentials, followed by firmware version enumeration and subsequent exploit research, would discover this issue.

False Positives and Negatives

Automated tools can sometimes misinterpret data or patterns, leading to false positives (flagging non-issues) or false negatives (failing to identify actual vulnerabilities). This can be a significant drawback as false positives can waste valuable time and resources, and false negatives can leave systems exposed to potential attacks.

To illustrate the issues with false positives, I’d like to borrow the words of a close friend and cloud architect:

If you want me to fix things, I need to know what things might actually be broken. Not what might be broken ‘if’ we use ‘this’ in conjunction with ‘that.’ …Most of these [findings] take me literally hours per item to discover that we don’t actually have ‘that.’

This is a real problem, especially when deadlines need to be met.

On the other side, the potential of encountering false negatives is even more concerning. Many vulnerabilities go undetected by network vulnerability scanners due to a number of reasons, ranging from network traffic to the inability to accurately identify version information.

If a scanner cannot pinpoint the version or service in question, it may not run the applicable plugins, creating a gap in the security assessment. Furthermore, network scanners might interpret timeouts caused by network traffic or other disruptions as either false positives or negatives, undermining the reliability of the scan.

Limited Scope

Automated tools generally focus on known threats, documented in various databases, leaving a blind spot for zero-day vulnerabilities and unique or custom system weaknesses. Furthermore, these scans are typically not tailored to an organization’s specific network or system configuration.

This depersonalized approach may overlook specific use cases, workflows, and configurations that could be potentially vulnerable, which a more tailored, human-led testing process might spot.

Essentially, the scope of automated testing is limited compared to a manual test, which can be designed to be more exploratory and personal to an organization’s setup, thereby potentially uncovering a broader range of vulnerabilities.

Identifying Genuine Penetration Testing

So, how can you ensure that you are getting a comprehensive penetration test? Here are a few indicators:

Tailored Strategies

Real pen tests are designed based on the unique characteristics and needs of an organization. Experienced testers develop strategies that adapt throughout the engagement, allowing for a more comprehensive and insightful process that considers the organization’s specific risk profile and threat landscape.

Collaborating with pen testers grants organizations a significant advantage. They bring to the table deep expertise and up-to-date knowledge on the latest tactics, techniques, and procedures (TTPs) used by hackers. Leveraging their insights can guide organizations in crafting more robust and resilient security strategies, fostering a fortified defense against potential cyber threats.

Comprehensive Reporting

At the conclusion of a genuine penetration test, you should expect to receive a detailed report that not only lists identified vulnerabilities but also provides a deep analysis of the potential impact and risks associated with each threat. It should also include recommendations for remediation and future prevention. This depth of analysis is often missing in automated tests.

Open Line of Communication

An effective pen test should involve an open, collaborative communication process between the testing team and the organization. This allows for a real-time discussion of findings, an exchange of insights, and the development of more effective remediation strategies. It creates a cooperative environment where the focus is on enhancing the organization’s security posture through shared knowledge and expertise.

Conclusion

As the industry navigates this shift towards automation, it is essential to strike a balance between leveraging vulnerability scanning tools for efficiency and retaining the expertise and insight that human testers provide.

Vulnerability scans can be a valuable component of a comprehensive cybersecurity strategy, but it should not replace real pen tests entirely. Combining the strengths of both approaches will create a robust, adaptable, and resilient cybersecurity infrastructure that stands ready to face the evolving threats of the digital age.