LISA 2010: Sysadmins Discuss Virtual Mentorship

Tuesday, November 23, 2010

Jamie Adams


In November of this year, I attended the 24th Large Installation System Administration (LISA) Conference in San Jose, CA. LISA is an annual technical conference sponsored by USENIX: the Advanced Computing Systems Association.

On Friday, November 12th, I led a "Guru Is In" session on "Security" where we had a lively exchange of views and information.

In attendance were junior and senior system administrators, seasoned administrators, and salty old-schoolers like myself. At one point, we discussed the idea of mentorship and the operational environment many younger system administrators find themselves.

For me, this was a subject of poignant interest and later that evening I reflected upon my early work environments compared to what system administrators face today.

First and foremost, the ratio of systems to system administrators is much different. Today, there seems to be fewer system administrators but each is responsible for far more systems.

In the early days, system administrators were required to have a diverse set of skills and knowledge. For example, we were required to have a deep knowledge of how file systems worked, networking concepts, and access control mechanisms.

Today, there is a greater dependency on automated tools in order to improve productivity. Our field also seems to have as many specialists as the medical community.

There are system administrators focused on different operating systems, cluster specialists, network administrators, database administrators, system virtualization managers, and even security administrators.

When I started, I was originally a system programmer but eventually assumed responsibilities as a system administrator. I had a mentor who assigned me specific jobs such as performing system backups, account management, and analyzing daily logs.

I continued to perform those jobs until my mentor determined I was competent in that role and then I moved on to the next role. This could last anywhere from weeks to months. I rotated through all of the on-site roles until I was capable of performing any task in the data center.

In those early days, vendor manuals and the few technical books available on the market were treasured. More importantly, the strong relationship with my mentors helped me gain the most out of my experiences.

A good mentor would not tell you "exactly" how to do something but rather encourage you to learn how and why things are done.

Of course, they would instruct you on the site's best practices but most importantly they taught you "how to learn."

In other words, how to solve problems without jeopardizing an entire system. Of course, if I made a serious mistake or had a lapse in judgment... let's just say that some of us were motivated through public humiliation. But it was all in good fun.

Several years later when I became the mentor, I would immediately set precedence by asking them if they did any research on the problem before coming to me with the question.

If they came to me with a question they've asked before, I would ask them why they didn't write down the answer the first time. It is so important that system administrators become self-sufficient.

I believe having a mentor is extremely beneficial. These days, however, several factors hinder this type of relationship:

  • Fewer on-site administrators.
  • Higher turnover rate. Many administrators don't remain on a project as long as they did many years ago.
  • The Internet. The massive amount of online technical resources allows one to be self-taught.
  • Volumes of printed publications (e.g., O'Reilly books)

The unfortunate fact is that many junior system administrators can quickly get in over their heads.

One of the administrators at the LISA 2010 conference told me he started out being responsible for only a few machines but the number of machines quickly grew out of control.

Many of his generic configuration decisions did not scale well and it became very difficult to manage. This is a classic problem.

Many of the operational resources, such as software repositories for updates, a good back-up system, and sound configuration management, hadn't been considered during initial set up either.

These are the kinds of things experienced administrators can pass on to junior administrators, including documentation skills! Yes, this skill is one that must be mastered but is often brushed aside and considered trivial early in one's career.

Of course, there are plenty of training programs and certifications available. Personally, I've never been a fan of vendor specific certifications because it focuses on proprietary technology. It certainly benefits the immediate company which has hired the individual or has paid for the training.

However, I think it is far more critical for a system administrator to have a vendor neutral understanding of the fundamentals. This gives them a strong foundation which supports easily learning other technologies.

Finally, I strongly believe a mentor helps harvest those soft skills. After all, knowledge is far more powerful and useful when coupled with wisdom.

Because there are fewer on-site administrators and there seems to be higher turnover rate these days, it is critical that a virtual mentorship be in place. This may not be a one-on-one relationship but having a means to ask questions in almost real-time is critical.

Strong organizations and communities such as USENIX and LOPSA are invaluable. Membership dues must be reasonable because many junior administrators aren't willing to shell out the cash early on if they don't see great value.

In an effort to protect an organization's assets, many restrict or deny access to valuable on line tools and resources such as instant messaging, sites with "blog" in the URL, or user group websites.

This is unfortunate because these resources can help administrators quickly resolve problems. Additionally, most system administrators maintain a long term professional relationship with fellow system administrators and having the ability to ask questions via instant messaging is critical.

I think an organization that already entrusts their information technology assets to a system administrator should seriously consider trusting them not to abuse such Internet access.

Providing the newer system administrators with access to the tools that can help them become successful, faster, will ultimately be beneficial to the organizations that they support.

Cross-posted from Security Blanket Technical Blog.

Possibly Related Articles:
Security Training
Enterprise Security Training Networks Administration LISA
Post Rating I Like this!
The views expressed in this post are the opinions of the Infosec Island member that posted this content. Infosec Island is not responsible for the content or messaging of this post.

Unauthorized reproduction of this article (in part or in whole) is prohibited without the express written permission of Infosec Island and the Infosec Island member that posted this content--this includes using our RSS feed for any purpose other than personal use.