The Official ASTD Blog
Learning Industry News and Opinion

Rod and Staff – Do They Still Apply?

May 19, 2011 14:43 by jllorens

The concept of support and accountability that comprise Level 3 required drivers is not new. In fact, it is a very old idea, earlier termed “rod and staff”.
Support and accountability (or rod and staff) are among the strongest predictors of on-the-job performance and accomplishment of goals. Here is a seven-minute recording discussing the principles.

Click here to access the recording. (Link will load as a WMA audio file.)


Tags: , , , , , , , ,

Categories: Kirkpatrick | News

Categories: Kirkpatrick | News
Actions: E-mail | Permalink | Comments (0) | Comment RSSRSS comment feed

Should I Always Conduct a Level 1 Evaluation?

May 11, 2011 11:49 by jllorens

Questions related to whether or not a Level 1 evaluation should always be conducted are some of the most frequently asked of Kirkpatrick Partners.

Wendy and Jim Kirkpatrick have recorded a short podcast of less than five minutes to answer this question.

Click here to access the recording. (Link will load as a WMA audio file.)


Tags: , , , , , , ,

Categories: Kirkpatrick

Categories: Kirkpatrick
Actions: E-mail | Permalink | Comments (0) | Comment RSSRSS comment feed

Resetting Your “Levels 3 and 4 Are Too Difficult” Default

April 26, 2011 10:43 by jllorens

Kirkpatrick Partners hears this all the time: Levels 3 and 4 are too hard! Many learning professionals believe Level 4 is out of their reach, and are even intimidated by Level 3. Both of these are unnecessary apprehensions that will prevent maximization of program impact, and could even jeopardize careers.

There are many simple and effective ways to cross the bridge from the comfortable, familiar territory of Levels 1 and 2 to the strategic realm of Levels 3 and 4. Here is one to try:

Query past training participants three months after the conclusion of training and ask:
•    To what degree have you applied what you learned from xx event?
•    If you have, what factors have led to your successful application?
•    If you have, what are any early indications that your efforts are creating positive impact?
•    If you have not applied what you learned, what are the reasons?
•    What additional training, education or support do you need to move ahead?

You can administer these through a questionnaire (as is), survey (with drop down boxes with possible answers), one-on-one interviews or group interviews. It is best that you do this in situations where there has been structured support and accountability.

You can also ask similar questions of supervisors to capture their observations.

Read the original article on training “defaults.”


Tags: , , , , , , , ,

Categories: Kirkpatrick

Categories: Kirkpatrick
Actions: E-mail | Permalink | Comments (0) | Comment RSSRSS comment feed

Resetting Your “Always Start with Level 1” Default

April 19, 2011 17:45 by jllorens

It is a commonly held belief that the Kirkpatrick Model starts with Level 1 Reaction, and then moves sequentially through levels 2, 3, and 4. This tradition has been developed and solidified over five decades. The truth is that when used properly, the four levels begin with Level 4. This is the first of the Kirkpatrick Foundational Principles, entitled “The End is the Beginning.”

Here is why to start with Level 4. When an external client or internal business leader asks you to develop and deliver a training program, it is most unwise to do a cursory pulse check of their expectations, and then jump to the DDD process (Design, Develop, Deliver), followed by some evaluation. This could lead to great waste of resources if the desired end result is not accomplished.

Instead, particularly for mission critical programs, interview the client and sponsor to find out as much as you can about what prompted the request, what is the overall intent, and specifically what will success look like from a strategic level.

Once you uncover or (more commonly) negotiate the answers to those questions, you have the structure for your Level 4 Results. Then, work backwards through determining Level 3 Behavior, and, if necessary, Level 2 Learning and Level 1 Reaction. In this way, you will know exactly where you are going. You will then be able to direct training, reinforcement, and evaluation efforts towards accomplishing mission-critical results with a minimum of wasted time, money and resources.


Tags: , , , , , , ,

Categories: Kirkpatrick | News

Categories: Kirkpatrick | News
Actions: E-mail | Permalink | Comments (0) | Comment RSSRSS comment feed

Resetting Your Survey Default

April 6, 2011 11:00 by Ann Pace

At the beginning of the month we shared with you an article by Jim Kirkpatrick about your “training defaults”. He defines these as hard-wired behaviors that have become habits, but may not lead to the highest level of results. These are things we do automatically without considering if they are outdated or less effective than other options.

This week we ask you to reset your “survey default”, which we see often as a reliance on surveys as the only means of measurement and data collection.

Employee surveys have been the standard evaluation method for decades. It may be due to their ease of implementation, and ability to gather large amounts of quantitative data quickly. Many organizations even refer to their Level 1 reaction sheets as “evals.”

Surveys are a critical component in your evaluation toolkit, but they should be combined with other methods that yield richer qualitative information to complement the quantitative data.

Here are some reasons to consider a mixed approach. Research shows that targeted observations with accompanying checklists are statistically more valid and reliable than surveys. In addition, surveys only gather certain type of quantitative data, and little genuinely useful qualitative information.

When Kirkpatrick Partners conducts impact studies, surveys are augmented with a relevant combination of focus groups, direct observation, work reviews, interviews, or other data collection methods. This is a best practice you can duplicate for your own initiatives.
When you do use surveys, here are some recommendations: 

• For mission critical initiatives, survey not only actual training participants, but also managers, supervisors, co-workers, peers and customers.
• Ensure that your surveys contain learner-centered questions.
• Use hybrid evaluations; that is, surveys that include questions from two or more of the Kirkpatrick levels.
• Avoid questions that attempt to empirically quantify and isolate the impact of the training event. It is mathematically impossible, contrary to business partnership, and produces measurements that get questioned and doubted by stakeholders. Here are examples of questions to avoid:

o  “What percentage of your time on the job relates to the content covered?”
o “What percentage of what you learned do you plan to apply?”
o “What percentage of what you learned in training do you currently apply on the job?”
o “What percent increase in productivity have you seen in your work since the course?”
o “What percentage of that improvement can you directly attribute to the course?”
o “How confident are you in these percentages?”

Read the original article.


Tags: , , , , , , ,

Categories: Kirkpatrick | News

Categories: Kirkpatrick | News
Actions: E-mail | Permalink | Comments (0) | Comment RSSRSS comment feed

Resetting your Level 2 Pre- and Posttest Default

March 28, 2011 18:40 by jllorens

At the beginning of the month we shared with you an article by Jim Kirkpatrick about your “training defaults”. He defines these as hard-wired behaviors that have become habits, but may not lead to the highest level of results. These are things we do automatically without considering if they are outdated or less effective than other options.

This week we encourage you to think about a common Level 2 “default” many training professionals possess: automatic use of pre and posttests.

There is certainly a time and a place for pre and posttests, but it is not before and after every learning event. Please “uncheck” automatic pre and posttests as part of your Level 2 assessment methods.

Testing is often time-consuming, expensive, and traumatic for training participants. Be purposeful about the use of pre and posttests. Situations where it is recommended include:

•    Prepositioning the learner to the material to be covered in training
•    Determining which elements of a course to emphasize and/or deemphasize
•    Allowing employees to earn the right to test out of a program
•    Demonstrating the amount of knowledge acquired during a formal learning event (if value of the event itself is being questioned)
•    Meeting compliance issues
•    Ensuring that participants are ready for more advanced training

If you decide to use pre and posttests, be realistic about the value of information they can provide. This form of assessment generally taps into the lowest levels of Bloom’s learning taxonomy in the cognitive domain. This kind of data (e.g., “Our participants increased in knowledge an average of 37%”) typically elicits a response of “Who cares?” from stakeholders.

To maximize training evaluation resources, and ultimately training effectiveness, focus your energies on assessing Level 2 skill demonstrations. Use the resulting data as part of your chain of evidence to show how learners were prepared to perform the required behaviors on the job. Redeploy the resources you saved by skipping pre and posttests to on-the-job reinforcement of the critical behaviors. This will drive the results the training was likely designed to deliver in the first place.

Click to read the original article.


Tags: , , , , , , , , ,

Categories: Kirkpatrick | News

Categories: Kirkpatrick | News
Actions: E-mail | Permalink | Comments (0) | Comment RSSRSS comment feed

Modifying Your Level 1 Default

March 15, 2011 14:30 by Ann Pace

Two weeks ago we shared with you an article by Jim Kirkpatrick about your “training defaults”. He defines these as hard-wired behaviors that have become habits, but may not lead to the highest level of results. These are things we do automatically without considering if they are outdated or less effective than other options.

During workshops, we frequently ask participants, “What is the most common way to conduct Level 1 evaluation?”  Invariably the response is, “Give the participants a smile sheet immediately after the program.”

Indeed, a written survey is the most common, nearly exclusive Level 1 evaluation practice. Unfortunately it is also often inadequate to obtain the most useful Level 1 information.

Consider these other options to measure Level 1:

• Have your “fingers on the pulse” (formative evaluation) during the course, in addition to afterwards. Be aware of participant body language and level of involvement to see if people are with you. If they are not, you can do something about it right away and hopefully correct the problem. For online programs, ask polling questions within or at the end of each module to maintain participant attention.

• Use a trained observer in class to note how participants are reacting to various modules and activities. This is a costly option best reserved for pilots and major initiatives.

• Have a colleague conduct a focus group or one-on-one interviews after the program to gather important information that will likely not surface with even the best survey questions. Strongly consider this with pilot programs and when survey data indicates potential problems.

Read the original article.


Tags: , , , , , , ,

Categories: Kirkpatrick

Categories: Kirkpatrick
Actions: E-mail | Permalink | Comments (0) | Comment RSSRSS comment feed

Modifying your TNA Default

March 8, 2011 09:04 by jllorens

Last week the article entitled “Checking Your Training Evaluation Defaults” discussed a number of common practices that hinder training effectiveness.  This week we continue the discussion by providing alternatives to the first potentially harmful default: training needs assessments (TNAs).

The Kirkpatrick Model starts with Level 4, not Level 1, as is the popular practice. Training needs assessments are typically conducted with business leaders in an attempt to determine overall business needs under the false belief that you are starting with the end in mind.

Please uncheck this default for the simple reason that if you are conducting training needs assessments, you are pre-determining that the solution to business needs is training. Instead, approach business leaders and tell them you would like to work with them from a new perspective.

Develop a methodology that you call a “performance needs assessment” or a “business needs assessment”. Ask questions about business pain points, needs, challenges and/or opportunities. Then work to determine solutions, one of which might be formal training. In this way you deliver learning and performance solutions that target results rather than just adding more training activity.

Click to read the original article.


Tags: , , , , , , ,

Categories: Kirkpatrick | News

Categories: Kirkpatrick | News
Actions: E-mail | Permalink | Comments (0) | Comment RSSRSS comment feed

Checking Your Training Evaluation Defaults

March 2, 2011 10:00 by Ann Pace

Computers have default settings designed to make common functions easier to perform. We become so accustomed to them we may never consider other (or better) ways to do the things we do regularly.

There are often similar “default settings” for training evaluation. Read this blog posting to find out if you are guilty of following a routine that may be limiting your training effectiveness.

Read the rest of the blog posting.


Tags: , , , , , , ,

Categories: Kirkpatrick | News

Categories: Kirkpatrick | News
Actions: E-mail | Permalink | Comments (0) | Comment RSSRSS comment feed

Don’t Forget to Gather Qualitative Evidence

February 22, 2011 15:11 by jllorens

Most training professionals realize the importance of reporting numeric data related to training events. But how about testimonials, stories and examples of how the training has helped participants to achieve success on the job?

This type of qualitative evidence is just as important as numeric data in helping stakeholders to see the complete story of value created by training and subsequent reinforcement.

When Jim Kirkpatrick was a training director at a bank in Indianapolis, one of the programs he conducted was a book club. It was open to all bank employees during lunch hour. It took no time away from work and had a modest budget to purchase copies of the books being discussed.

Click to read the rest of the blog posting.


Tags: , , , , , , ,

Categories: Kirkpatrick | News

Categories: Kirkpatrick | News
Actions: E-mail | Permalink | Comments (0) | Comment RSSRSS comment feed