My colleague Andre Harricharran, security consultant with Security Compass, offers a brief overview of a NIST recommendation for evaluating the effectiveness of a security training program:
An information security training program is crucial for ensuring and maintaining a good security posture; in order to effectively manage this program you have to be able to measure it. This article introduces a concept recommended by NIST in their Special Publication 800-16, for evaluating training effectiveness.
Measuring the overall success of a training program is broken down by NIST into the following four distinct but interrelated measurements. As such, success or failure on one objective influences the outcome of other objectives.
- Learner satisfaction
- Learning/ Teaching effectiveness
- Job performance effectiveness
- Program effectiveness
Determining the overall effectiveness of a training program takes into account employee and employer objectives, the delivery vehicle used for training delivery, the usefulness of the training and the ROI for the organization among other options. To obtain these measurements require planning in advance as measurements must be taken before and after, and integrated into the training program.
The following tools are typically used for assessing each training performance objective:
i. Learner satisfaction: This is typically extracted from the use of a post training questionnaire. It collects the learner’s subjective assessment of the training material and delivery vehicle used.
ii. Learning and teaching effectiveness: Observed by measuring participant’s knowledge before and after the training, and by the evaluator being mindful of the overall performance of participants.
iii. Job performance effectiveness: Supervisors should assess and observe employee’s performance before and after the training.
iv. Program effectiveness: The organization should link training outcomes to strategic goals and calculate ROI for training initiatives.
The NIST SP800-16 guideline outlines training requirements for information security within an organization and provides insight into the evaluation process. For evaluation, it highlights measurement tools and provides samples that can be used to get you started with evaluating your training program and its outcomes.
The following list are typical goals for training, each should be measured using all four measurements discussed above in order to ascertain the overall success of a training program:
i. Influencing staff behavior, for example training on selecting a memorable secure password.
ii. Increase technical knowledge, for example training on mobile in-security.
iii. Facilitating the adoption of new processes and procedures, for example adapting an SDLC or a risk based approach for security.
iv. Raising awareness and getting buy-in for specific changes.
v. Increasing performance.
vi. To reduce the occurrence of defects in a process.
For more information please see: NIST Special Publication 800-16, Information Technology Security Training Requirements: Information Technology Security Training Requirements.
Cross-posted from the SC Labs blog