Posts Tagged ‘Levels of Evaluation’

Did it stick? Easy level 3 evaluation.

August 10, 2010
Four Evaluation Levels

Donald Kirkpatrick's 4 Evaluation Levels: Reaction, Learning, Transfer (Behavior), & Results. Jack Phillips added a 5th level: Impact (also known as ROI)

Our last two newsletters have looked at how CEOs view learning and development and the metrics they would like to have available to them. There are some very interesting results of a CEO survey so if you missed either of these newsletter issues, take a look in my archives

I have conducted several workshops recently on Making Training Stick for experienced and seasoned trainers. When I talked about Level 3 evaluation the energy level in the room sank noticeably. The trainers began to whine: 

  “We don’t have time for evaluation” 

 ”Nobody cares about the results anyway” 

“Leadership doesn’t care about evaluation results, so why do them?” 

  “It’s too hard to isolate the impact of training” 

Over 90% of trainers conduct Level 1 evaluations, although only 36% agree this type of evaluation has value, according to a study on evaluation conducted by the American Society for Training and Development.  54% of trainers conduct Level 3 evaluations to some extent yet 75% believe this type of evaluation is valuable! 

Why should you evaluate on-the-job behavior changes that are the result of learning and development? 

  • The purpose of learning and development is to impact on-the-job performance.
  • A recent survey of CEOs indicated this level is important to them.
  • The information from a Level 3 evaluation can provide important feedback for revising training activities – before, during, and after the learning event – based on what is and is not producing changes in job performance. 
  • This type of evaluation must be done in order to conduct meaningful and accurate Level 4 or 5 evaluations.
  • It’s possible to do these types of evaluations with a minimum of time and effort. 

I agree it’s not always easy to measure on-the-job performance changes that are linked to participation in learning and development. Supervisors may not cooperate with observing and reporting on-the-job trainee behavior. Complicated metrics and/or case study write-ups can take up a lot of time, and few trainers have this kind of time. 

However, research shows that trainee reports – what they are using on the job, or what they plan to use once they return to work – provide useful information as to what has stuck. Electronic surveying technologies provide quick and easy ways to measure this feedback. Most training departments are using electronic surveys to conduct their Level 1 reaction surveys these days anyway, whether a part of their Learning Management System or a stand-alone, web-based system such as Survey Monkey. These survey systems are low-cost or no-cost. 

I’m not suggesting a 20-item survey here. Look back at the learning objectives for the training. Identify on-the-job behaviors linked to the learning objectives. (This should be quite easy, and if it’s not, revise the learning objectives for the next class.) Write 3-5 multiple choice questions that are variations of these: 

  • Are you using ___________ (what you learned)?

A. I’m not using it    

B. I haven’t had time or opportunity to use it, although I’d like to or plan to  

C. I don’t need to use this skill in my job   

D. I am using it 

  • If you have a need ________(to do something that would use the skill in question), how would you handle it, what would you do? (Be honest. Remember, this is anonymous.) 

A. Do it the way I’ve always done it   

B. Ask my supervisor   

C. Use _____(new skill) 

  • If you aren’t using _____(new skill), what would it take for you to start using it?

A. More practice     

B. Job aids that recap the steps    

C. Another class 

If the training has many learning objectives and/or includes many skills, develop and send more than one 3-5 question survey. 

Survey results in almost all survey systems can be downloaded into spreadsheet format and the responses translated into percentages or other simple metrics. You may be surprised at the results! 

 Until next time…. 

Barbara