{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreihqlb2x2kwbjxl7sqjann55xtc7tbpm5zor525fxo5xpllzmvjugy",
    "uri": "at://did:plc:rrwxywdlrz5fkwj5g4u4jnrk/app.bsky.feed.post/3mgpmjwbr4xw2"
  },
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreie6a5dovmwq5bxlsw5phr2t6sekhsgfbcwqtd3g73trjs4fgnt2l4"
    },
    "mimeType": "image/jpeg",
    "size": 9610384
  },
  "path": "/article/4141544/i-replaced-manual-pen-tests-with-automation-heres-what-i-learned.html",
  "publishedAt": "2026-03-10T07:30:00.000Z",
  "site": "https://www.csoonline.com",
  "tags": [
    "Penetration Testing, Security, Security Practices"
  ],
  "textContent": "More accreditation and compliance requirements have been added in response to cyber incidents. While these frameworks play an important role in establishing security baselines, true security is more than just achieving a perfect compliance score. As I often say, “policies and procedures won’t stop an attacker, they’ll just have more documents to exfiltrate when they breach us.”\n\nTesting how our environments withstand a determined threat actor is the real validation of security posture. That’s where the annual manual penetration test comes in, with boards now demanding to see positive results.\n\nThere are, however, significant issues with manual penetration testing I have experienced, particularly when conducted only annually.\n\n## Speed, scope, and the human bottleneck\n\nThe constraints of manual testing became increasingly apparent as our environment grew more complex. Every engagement was bound by time and budget, forcing difficult trade-offs about what to test and how deeply. The quality and comprehensiveness of results varied significantly depending on which consultant we engaged, their individual expertise, their familiarity with emerging techniques, and how much they could accomplish within the contracted hours.\n\nTraditional penetration testing delivered what I came to see as a fundamentally flawed value proposition. We’d invest significant budget to receive a snapshot of our security posture weeks after the test concluded and from that moment it began aging like milk. There was no ongoing feedback loop, no continuous validation of our security controls. We were essentially flying blind between annual tests, hoping our defenses remained effective even as the threat landscape evolved daily around us.\n\n## The remediation black hole\n\nPerhaps most frustrating was what happened after we received findings. Our teams would work diligently to implement fixes, but we rarely had the budget or opportunity to bring testers back to validate remediation. We were left with uncertainty. This gap between identification and verification created a dangerous blind spot in our security program.\n\nTraditional vulnerability assessments leaned heavily on CVSS severity scores that did not tell us how exploitable a vulnerability was in our specific environment or where it sat within a realistic attack path. We needed to understand what an attacker could actually accomplish by chaining vulnerabilities together.\n\n## A better way forward\n\nFrustrated with these limitations, I explored automated penetration testing, a category that includes breach and attack simulation (BAS) and continuous automated red teaming (CART). Platforms like Pentera and Horizon3.ai’s NodeZero conduct continuous, on-demand simulations using real-world attacker tactics, techniques, and procedures.\n\nThey offer black box testing (simulating external attackers), grey box testing (simulating insider threats), and custom scenarios targeting specific risks like ransomware or zero-day exploits.\n\nMost importantly, they deliver results instantly, no waiting weeks for reports, and enable immediate retesting to validate fixes.\n\n## The implementation and investment\n\nWe moved from $35,000 for an annual manual test to $90,000 annually for an automated platform, delivering over $1.3 million worth of equivalent testing. Our cadence jumped from one test per year to a minimum of 38, with unlimited flexibility for additional simulations.\n\nWe established a fortnightly rhythm of black box and grey box tests, supplemented by monthly custom scenarios targeting specific concerns like ransomware attacks. This gave our team two weeks to remediate before retesting confirmed fixes worked. These tools test more in a day than human testers accomplish in a week, rapidly adjusting to findings and leveraging gaps to probe deeper.\n\n## Unexpected lessons and team transformation\n\nThe platform delivered insights that fundamentally changed our understanding. Take password security: we’d adopted longer passphrases, confident that fourteen-character phrases would increase breach time from eight months to twelve billion years. The tool shattered that confidence, cracking a 23-character passphrase containing upper- and lower-case letters, numbers, and special characters in under half an hour. The lesson was humbling, humans are predictable. Attackers maintain wordlists and precomputed hash lists in rainbow tables specifically targeting common phrases. Passphrase length matters, but quality matters more.\n\nThe retesting capabilities proved game changing. Security teams could identify problems, remediate them, and immediately retest to verify fixes were effective. The platform generated both executive-level reports for board presentations and detailed technical reports for security teams to action instantly, not weeks later.\n\nPerhaps most importantly, the platform elevated our team’s capability. Until your team experiences an automated penetration testing tool exploiting their environment, they won’t fully comprehend how to apply defensive concepts to their specific systems. Each simulated attack was fully documented, providing real-time learning opportunities. The teams began treating the platform as a game they were determined to win.\n\n## Rethinking prioritization: attack paths over severity scores\n\nOne of the most significant revelations was how automated penetration testing transformed our vulnerability management. We discovered that the critical-rated vulnerability receiving immediate attention might be buried five layers deep in an attack path, while a low-rated vulnerability we’d deprioritized could be the initial entry point attackers would exploit. More revealing still, the platform showed how seemingly low-risk vulnerabilities could be chained together to access critical systems.\n\nThis changed our patching strategy. Instead of reflexively addressing vulnerabilities by CVSS severity ratings, we focused on what attackers could actually use to establish a foothold. Given the overwhelming number of vulnerabilities requiring constant attention, this intelligence about actual attack pathways proved invaluable allowing us to focus limited resources where they’d produce the greatest security outcome rather than chasing severity scores that didn’t reflect real-world risk.\n\nThe gap between configuration and reality\n\nWe place enormous faith in our security tooling when we enable a feature, we assume it’s working. The automated penetration testing platform delivered a sobering lesson: test your controls, don’t just trust the GUI.\n\nI experienced this firsthand when we enabled a functionality to mitigate a specific risk. It looked perfect on screen, but it wasn’t working. The platform methodically tested different attack types, including the scenario we thought we’d protected against. The attack succeeded, the security tool’s features weren’t functioning due to a bug. We didn’t have the protection we thought we did.\n\nIt reminds me of the defender’s dilemma: “Defenders have to be right 100% of the time; attackers only have to get it right once.” I’d much prefer our own testing tools highlight these gaps than have attackers discover them.\n\n## The ultimate validation: Testing your detection and response\n\nAnother powerful application is validating your detection tools and SOC. The first time I ran a proof of concept, I deliberately didn’t inform our third-party SOC. Our internal SIEM immediately generated numerous alerts. It took four hours for the external SOC to contact us — a lifetime in cybersecurity.\n\nWhen you’re paying for a third-party service, validating their response is invaluable and I strongly recommend running at least one unannounced test. The results may surprise you, and it’s far better to discover gaps during your own testing than during an actual incident.\n\nOne final lesson: as your security resilience improves and you achieve consistently high scores, you reach a plateau. Moving to a new automated penetration testing platform can yield fresh findings, as each tool takes different approaches, providing opportunities to continue improving rather than becoming complacent.\n\n## The verdict: Evolution, not elimination\n\nShould you replace manual penetration testing with automated platforms? The answer is nuanced. For ongoing security validation, continuous improvement, and operational resilience, automated testing should become your primary validation method. The ROI, learning opportunities, and continuous feedback loop far exceed what annual manual testing delivers.\n\nHowever, I wouldn’t completely eliminate manual testing. There’s still value in bringing in specialized human testers for complex custom applications, critical infrastructure changes, or when you need creative thinking that only experienced security researchers provide. Think of automated platforms as your daily training regimen, with manual tests as occasional specialized assessments.\n\nThe real question is whether you can afford _not_ to adopt continuous automated validation. The gap between annual manual tests leaves you vulnerable for 364 days a year. Automated penetration testing fills that gap, transforms your team’s capabilities, and validates your security posture continuously, not just once a year when auditors ask.",
  "title": "I replaced manual pen tests with automation. Here’s what I learned."
}