Looking Beyond "Black Box Testing"

Tuesday, June 21, 2011

Rafal Los


Is black box application testing - the idea that we're going to test an application with next to no knowledge about it - a thing of the past?

Evolution of "Black Box Testing"

Black box testing originated in the penetration testing world. A talented information security professional (or white-hat hacker) would be paid to 'hack into' an application given only absolutely minimal information about the site or application such as the URL.

As the need to scale the efforts of these highly skilled testers grew companies sprang up which took their core skill set and transformed it into a piece of technology ...a "black box testing tool".  

The idea sounds really cool - treat the application or site as if it were a black box with zero knowledge and see what [vulnerabilities] you can find.

As the boys on Top Gear say "... but there's a problem".

The Problem

In this model where you're blindly hacking away at something you don't understand - you can't reasonably expect great results... can you? Yet people do... and vendors have tried to compensate for some of those incredibly ambitious expectations by building better and better parsers and black box testing tools.  

This whole arms race between developers building more complex applications, and security black box testing tools trying to understand them without human interaction is about to end badly - unless something is done really, really soon.

Why is this such a problem you ask?  If you think back with me to 1997 when a web site was just that - a web site - one could reasonably expect a web spider to traverse a site successfully by following links and even clicking simple buttons or relatively simple forms.

Today, this simply isn't true... or possible in many cases.  If you don't believe me, open up Google's GMail and ask yourself how you'd crawl that site... err... application. It would be nearly impossible with the technology we used back in 1998, or even 3 years ago!  Yet, crawler technology is at the heart of black box testing tools.  So you can see we have a problem.

It gets worse too.

Many modern applications are data driven which means that you have to have a valid data set to drive the application. The crazy thing is that crawlers and have pre-defined data sets, or worse generate their own on the fly and often don't really care much for how an application is meant to be driven.  

Of course... if you're expecting a black-box 'scan' (I hate that word) you don't want to give over the real data which will drive the application correctly anyway. By now the light bulb is undoubtedly lit brightly as you are realizing the major issues here. Yet we've pressed on.

We as vendors have built some absolutely incredible technology into these platforms.  I have to tip my hat to the product managers, architects and developers who've slaved for hours and years building an ever-more complex platform of black-box testing.  

Today's capabilities include intelligent crawling, redundant page detection capabilities, some absolutely insane AJAX and JavaScript parsing engines and more that just absolutely blow my mind every time I crush another site or application.  

The problem is, boys and girls - even the greatest black-box testing tool is relatively tame without a good driver. Much like a Formula 1 car is just a very expensive piece of eye candy without the right driver - black-box testing tools need someone trained, and qualified to drive them.

Here's where I run into a brick wall.

The Myth of the "Hackers Eye View"

If you've ever used the argument of the "Hackers eye view" you should be ashamed of yourself.

The real "Hackers eye view" requires a near-infinite amount of time, resources, and capability. I have news for you - a black-box testing tool isn't going to have that.  Ever.

When you're performing an assessment of a site or application from a pure black-box perspective; that is, with zero-knowledge of the target, you're only handicapping yourself.

You're kidding yourself if you think an attacker isn't going to try and socially engineer your source code, real or test data, or anything else that may be of use to penetrate the application of site. Why in the world wouldn't you want to use all that same knowledge to your advantage to race to the big, nasty, security defects before those "bad guys" do?

Can We Be Realistic For a Moment?

Since we're not very familiar with the problem, let's talk about what we're doing about it, and what I think the appropriate steps are. Obviously we're working feverishly over here with some insanely intelligent people to get us over the limitations of black-box testing technology, and move us forward.  

But move us forward to what, you ask?

The next step forward is what you've heard me refer to as 'hybrid'... or what some have called gray-box testing ...or more appropriately "real-time hybrid analysis". Let's talk about what this new 'real-time hybrid' technology will accomplish, and what it will do... I won't go into details in this post, except leaving it at a few bullet-points for you to think about.

  • provide real-time insight into the application being attacked from inside the application
  • give the attacking engine real-time feedback on application coverage
  • virtually eliminate false positives & false negatives (yea ...)
  • correlate multiple 'exploits' to specific line-of-code defects
  • determine whether data-sets being used for testing are working properly

...and there's more but I don't want to spill all the beans in just this post.

Before you start to think this is just another marketing pitch, and you've been suckers into reading it - think though this - if all the things I've just listed above in the bullets are true... give me 1 reason you wouldn't dump your current black-box scanner and jump head-long into this technology quantum leap?

This isn't theoretical, or a pipe-dream.  This is stuff we're working on right now... and I can't wait to show you guys this stuff.  If you're coming to HP Discover - find me and we'll talk about it more, or maybe I can show you a demo in the very, very near future...

Cross-posted from Following the White Rabbit

Possibly Related Articles:
Information Security
Application Security Tools Penetration Testing Development Black Box Testing Crawlers
Post Rating I Like this!
The views expressed in this post are the opinions of the Infosec Island member that posted this content. Infosec Island is not responsible for the content or messaging of this post.

Unauthorized reproduction of this article (in part or in whole) is prohibited without the express written permission of Infosec Island and the Infosec Island member that posted this content--this includes using our RSS feed for any purpose other than personal use.