Continuing on with the series on logging titled "Opening Pandora's Box" we move into the third stage of realization - paralysis.
The third stage, paralysis, comes at the point where you're suddenly stricken with the feeling that so much potential is at your fingertips... and you have no idea what to do next. It's like giving a kid a giant box of building blocks... where all of a sudden dozens of ideas flood the mind and you're stuck at step 1.
If you've ever had a fantastic idea for a blog post, an article or a research project and you have so many good ideas you have no idea which one to pursue first - that's exactly the feeling.
Paralysis can come from over-dependence on analysis... I'm sure you've heard the term "analysis paralysis", where someone spends so much time trying to make the perfect decision while factors change that ultimately no decision is made before the deadline passes or some ending event happens.
In the logging world you can have so many awesome ideas, and put yourself in such a dangerous place collecting billions of log items per day that it can be overwhelming to think of what to do with all this wonderful data.
Some of that paralysis can be mitigated by working with your product vendor (or in the case of open source, the community) with pre-built sets of analysis filters and rules. This type of approach - not re-inventing the proverbial wheel every time - can at least lend a slight push and get you out of neutral gear.
Unfortunately, for reasons that are probably best not discussed here, too often analysts seem to throw away what has been pre-built and start from scratch... or worse the pre-built things they start with are crap.
Having spent some time with our customer base talking about this very thing - here's what I've learned. These are a few solid ideas which may help if you're stuck in neutral ("paralysis"):
- Make yourself a plan of increasingly deeper insight, in stages. Start with a series of questions that begin with "We'd like to know ... " and start high level and slowly dig into layers of intelligence requiring more in-depth analysis (or more logging input?)
- Feed the log analysis engine slowly until you figure out its capacity for analysis in near-real-time - every system, or tool has a limit... if you pile in 'everything' and the system breaks you never know what caused the breakage
- Review the questions you're asking, to make sure they provide technical and business value-add
- Ask lots of "What if...?" questions based on your existing knowledge.
- Start with simple answers such as "We'd like to know if there was any suspicious traffic during our maintenance window" ...and allow for room to 'dig deeper'
Whatever you do, get out of neutral... otherwise all of your great ideas, and all those great log bits, go nowhere. Worse yet, you'll be no smarter and you'll have wasted potentially a lot of time.
Cross-posted from Following the White Rabbit