Changing the Landscape of Pentesting

Tuesday, October 11, 2011

Andrew Weidenhamer


Though I believe penetration assessments to be important in assessing an organization’s overall security posture, I think they are 1) being performed poorly and 2) the results from them are being disseminated in a wrong way.

The goal of any security assessment is to help an organization become MORE secure than they were before the assessment was performed, thus reducing their overall risk. Many penetration assessments are performed by identifying vulnerabilities and breaking into as many systems as possible by exploiting these vulnerabilities.

The report is then issued which has the list of these vulnerabilities, a perceived risk rating, and finally recommendations on how to remediate the vulnerabilities. What many pentesters lose sight of is the objective for performing the penetration assessment to begin with: to help the client become MORE secure.

This type of penetration assessment provides absolutely no value to the client and certainly does not make them any more secure.

Many pentesters don’t see an unencrypted service enabled on a firewall which protects an organization’s PCI zone and wonder “why” this service is allowed but rather how I can use this service to break into this system.

The recommendation for such vulnerability would be to use a more secure service; however, what is lost is “why” the vulnerability occurred in the first place and the impact to the business if such vulnerability was exploited especially with regard to the environment in which it was discovered.

A penetration assessment needs to be just as much interview based (if not more) as it does technical. Without understanding the underlying reasons as to why such vulnerabilities occurred in the first place, it is impossible to provide any other recommendations other than tactical to the client.

The client will then tactically remediate the vulnerability maybe by updating a system with a specific patch or shutting down a specific service and then a year later vulnerabilities of a similar nature will resurface.

Why? Because the underlying reasons as to why such vulnerabilities occurred in the first place are unknown. Is it a patch management problem? Is it a change management problem? Are there no policies and procedures or minimum security baselines preventing such vulnerabilities? Is it a management problem? Is it a line-of-business problem? Is it a combination of the above?

The list goes on and on, but without trying to understand the “why” it is impossible to truly help the client. It is no longer acceptable to report that the entire compromise of an organization’s Windows domain was obtained without at least attempting to understand “why” it was possible and how to protect against future occurrences.

Today’s market has become so diluted with companies and individuals claiming they can perform penetration assessments (if you don’t believe me attend Defcon one year). Organizations need to have a better understanding as to how these hired service providers are actually performing these assessments.

If a company performs security assessments with little or no interaction with their client, be very skeptical of using this company. As the old cliche goes, you get what you pay for. Bottom line is, penetration testing is no longer for the geeky technical guy who only cares about breaking into systems or for someone who knows how to run a vulnerability scanner.

It’s for professionals who truly understand security and are interested in really helping an organization reduce their overall risk.

Possibly Related Articles:
Service Provider
Penetration Testing Consulting Security Audits Network Security Ethical Hacking Remediation Pentesting
Post Rating I Like this!
Javvad Malik Valid points Andrew and well made. In my opinion, it extends beyond pen testing, but into other aspects of information security. Too many times we'll focus on a particular control, e.g. screen lock after 15 mins of inactivity, without understanding 'why'. e.g. it's a trading screen displaying live share prices.

Security can never be applied without understanding the business and like you say interviewing the users / business before making recommendations. We tend to have too many companies who adopt a 'methodology' to walk into any company and say, "this is your solution and we can implement it" without taking the time to understand the business.
nathan ouellette I have a few observations on this topic. I think the reason you see so many pen testing shops (and the reports you're complaining about) is because the service is nearly as commoditized as vulnerability scanning. There are a lot of security firms that provide this service. Additionally, you also have to separate out what a tactical 'test' could mean to some people against truly strategic and program level recommendations. If you're talking service terms, those are very different costs and very different services in many cases (not all). A penetration test is generally not sold as or touted as a strategic management consulting service. You aren't getting any maturity models, roadmaps or multi-year recommendations at the testing level. Perhaps at an 'assessment' or 'management consulting' level you will. Testing is quite often performed by a technical resource, at technical resource rates. These are very different types of engagements than strategic consulting services. (usually)

I would even counter that unless the service is a hybrid engagement, interviews could potentially be counter-productive to the buyer's overall goal. Maybe they are engaging a firm to view their posture as a malicious external user might view it. That's a very real possiblity and where the 'test' conditions come into play. An attacker would perform reconnaisance. An attacker would find holes and exploit them using public information...or maybe social engineering if that's in scope. Too much information can actually taint the testing approach. Even if one of their dirty little secrets is that they haven't patched client-side software in three years, I don't want to know...I think my job is to be paid to find it. This is why scoping discussions and success factors are so important. We can't simply cast stones using a wide net at the pen testing industry as a whole. It's relative. They are all within whatever confines and scope that the engagement calls for. If I know too much about my target, I could very well be doing them a disservice unless I find out the information on my own, just like a malicious user might do. But that's my devil's advocate example.

Additionally, I am unsure why a vulnerability, an exploit and a result is of no value. That's a very wide stone to cast and an awfully large generic accusation. I'll oversimplify this: if I sit down to start an internal test...and I happen to open network neighborhood on a Windows host because I'm curious of what I see...and I start RDP and guess "domain\administrator:password" and that happens to work for the domain administrator...that's going to be a result worth knowing. Even if I don't have any consultant recommendations behind it. The result itself, will and should stand on its own legs by sheer virtue of the fact that it's a devastating, positive result within a pen test. Maybe their admins are apathetic, undervalued and spiteful. Maybe no one else knew that that password existed for five years. There can be lots of reasons but overall the end result is the same, shame on everyone who didn't pay attention to basic policy/process maturity/enforcement. Even if my exploit and my vector was completely basic, ridiculously effortless and seemingly bottom level stuff to security folks...I am hoping the customer is able to understand where the breakdown happened without the tester having to interview 50 people to deduce it. I think the point you're trying to make is that root cause and program level/process maturity problems are really what the customer needs to hear. And I would agree. But I wouldn't place blame at 'pen testers'. Customers aren't always interested or even buying that type of subject matter expertise when they hire an agency at the rates that attack and pen services might be attracting at the time. But regardless of that, I would challenge any organization that cannot infer root cause process problems given ANY result within a penetration test. If there is a result, there is most likely a problem. The margin of 'acceptable risks' are pretty low when we're talking exploits and real world exposures. But I can appreciate the spirit of what you're saying. Let's not let buyers of pen testing services off the hook though. Their job is to understand what happened, and that takes introspection and pointing their own eyes inward to review the "why" that you're referring to in most cases. The tester's job may not be to translate what happened into business impact (it might be, depending on the scope). But 100% of the time it's the buyers job to do just that.

Could some pen testers do their clients better by providing meaningful results? Sure. But 'results' are relative...and so is the approach you use to obtain them.
nathan ouellette Woops, had some copy paste issues in there. Last two sentences were meant for the middle of the response :)
Andrew Weidenhamer Thanks for the feedback. I would like to comment on a couple things you mentioned in your above statements.
First and foremost, I agree with what you stated in your second paragraph with regards to pentesting should be or take on a blind approach. However, after the assessment is completed, some discussion needs to be held with the client as to why some of the vulnerabilities in which were discovered, especially the most critical, occurred in the first place in order to provide strategic recommendations. Keep in mind, that as an information security management company, we have a relationship with our client. Our focus is to help our client improve their overall security posture and help reduce risk to what that company deems acceptable. In the case of the company I work for, this is true whether or not the client purchased a penetration assessment or advisory/consulting services. Obviously, this is a much different focus than an actual attacker. Attackers don't care whether or not the organization actually remediates the issues and/or reduces risk. An attacker is not looking out for the best interest of the subject organization. An attacker has no reason or even cares to understand why the vulnerability exists. Other than the methodology used to assess the organization, you can't really draw the comparison between a consulting company and an attacker.

With regards to your second point , I may have misspoke when I stated "no" value to the client. However, I do stand by the fact that it provides "little" value to the client. More times than none, even when you consider your above scenario, the client remediates the one vulnerability that was discovered without actually trying to determine why the vulnerability occurred in the first place and how to ensure future instances don't surface. Then, you come back and perform a penetration assessment the following year, and you discover the same exact vulnerabilities. It's like putting Anbesol on a chipped tooth. It may help alleviate the problem for the time being but certainly doesn't do anything to fix the actual problem. Even if you argue that there is nothing a consulting company can do if the client chooses to be ignorant or is simply complacent with regards to root problem analysis, that doesn't change the fact that it is still our client. It's like I having an unruly child. No matter how bad the child is, you can't help but love him/her and do everything in your power to ensure that it takes the right path in life. Our relationship with our client means everything. This is especially true when you are competing against other consulting companies that over simplify the assessment process by using the "let's get in and out" approach to perform their engagements.
The views expressed in this post are the opinions of the Infosec Island member that posted this content. Infosec Island is not responsible for the content or messaging of this post.

Unauthorized reproduction of this article (in part or in whole) is prohibited without the express written permission of Infosec Island and the Infosec Island member that posted this content--this includes using our RSS feed for any purpose other than personal use.