As the attacks on infrastructure become more complicated, the true nature of deep penetration attacks prove food for thought for all developers and operators. Consider this case - where the apache open source infratructure itself became significantly exposed by a simple XSS attack that utilized some social engineering techniques (i.e. getting folks to click on things), to load others up with credentials. After that, its off to the races!
In this case, a simple redirect hosted by a url shortening site exposed the clickers to a xss redirect, which then took the credentials of the clickee - in this case - administrators of some of the apache foundation infrastructure. From there, the path takes a meandering journey through key infrastructure - up to and including source code repositories and support infrastructure.
To note as well, the captured initial credentials exposed other systems with cached credentials, cookies, etc. Much like pulling on a thread, the intruders just had to keep pulling and following. Of course, these guys knew what they were doing (turning off notifications for source code changes, which servers to go after, where to look, etc.)
Consider that they had several hours to monkey around within the infrastructure - before teams noticed the breach. I recall an exposure several years ago where intruders had access for several months to key components of the ssh-key infrastructure. As far as is documented, no major damage (modified file payloads, etc.) has been identified. But this is a good example of why regular monitoring and scanning is important, especially in a multiple component distributed architecture.
As a side note - kudos to the apache team for a full, quick and detailed documentation of their exposure. We all learn from this - and we're all richer for it.