ight:250px;width:100%;">

As you prepare for your trip to Chicago, think about how you can get the highest return on your ASTD ICE investment.  Consider the following questions and corresponding tips:

1. What can you do before your trip to ensure you get the content you need back on the job?
   
   
   • Plan your sessions.
   • Work with you manager to target key learning outcomes.
   • Clear the deck so you don't worry about what you are missing at work.
2. What can you do during the conference to ensure you get the content you need back on the job?
   
   
   • Show up! (That's half the battle)
   • Turn off the Blackberry, iPhone, during sessions.
   • Ask relevant questions.
   • Engage in exercises.
   • Attend sessions with an open mind.
3. What can you do after the conference to transfer learning into action?
   
   
   • Compile all of your notes into one system (do this on the plane ride home if possible!)
   • Meet with you manager to discuss key learning outcomes.
   • Present content to your team.
   • Apply at least one new skill, knowledge area, or piece of information within three days of your return.
   • Follow-up with at least one new contact within three days of your return.

Tags:

Categories: Books | Celebrity Bloggers | Chapters | Conferences | Evaluation and ROI | International

Business alignment is an imperative if you are interested in driving a positive return on your organization's training investment. As described in an earlier blog,  Positioning Your Programs for Success in 2010, the process of achieving alignment includes four steps:

1. Clarify stakeholder needs
2. Develop SMART objectives
3. Communicate program objectives
4. Evaluate success, including the ROI when appropriate

Clarying the ultimate need, or the payoff need, sets the stage for identifying a correct solution. But it is the business need that defines specific measures that must improve in order to take advantage of the payoff opportunity. These business needs represents the business measures that will ultimately be converted to monetary value and compared to the program costs, used in the ROI calculation. But what happens when business measures are not so clear? How do you align a program to the business when participants come to the program armed with their own specific business needs?  When these situations occur, try using action plans.

What is an action plan?

In Chapter 8 of the upcoming ASTD Handbook of Measuring and Evaluating Training (coming soon!), Holly Burkett describes action plans in detail. But in a nutshell, an action plan is a tool by which participants identify specific actions they will take using content presented in your program to achieve some end. That 'end' is often the improvement in key business measures. Below is an example of an action plan intended to align planned actions with improvement in a business measure and calculate the ROI.

How does action planning work?

To ensure a successful action planning process, the following steps should be followed at their designated time frames.

Before the Program

Prior to the program, set the stage for action planning.

• Communicate the action plan requirement.
• As part of pre-work, have participants identify the business measure(s) that need to improve (and that can improve) through the successful application of knowledge, skill, and/or information presented in the program.

During the Program

During the program participants learn more about action planning.

• Describe the action planning process at the beginning of the program.
• Teach the action planning process, explaining how the document should be completed.
• Allow time to develop the action plan.
• Have the facilitator approve the action plan.
• Require participants to assign monetary value to their business measures and provide the basis for this value (see Items A, B, C on the right side of the action plan).
• If time allows, have participants present their action plans to the group.
• Explain the follow-up process.

After the Program

Post-program activities serve as the follow-up evaluation.

• Require participants to provide improvement data (Item D on the right side of the action plan).
• Ask participants to isolate the effects of the program (Item E on the right side of the action plan).
• Ask participants to provide their confidence level in their estimates (Item F on the right side of the action plan).
• Collect action plans at a pre-determined follow-up time.
• Summarize the data and calculate the ROI.

* * * *

The action planning process takes effort, but can be a powerful tool for collecting business impact data. In addition, data collected through the action planning process are used to calculate the ROI.

For an example of action planning in action, watch the webcast, Measuring the ROI in Performance Improvement Training The webcast was conducted on behalf of Learn.com. The actual case study (which serves as the handout referenced in the webcast) is attached below.

02.19.2010_Measuring ROI in Performance Improvement Training_Using Action Plans to Collect Level 4 Data.pdf (582.08 kb)

Tags: evaluation, ROI, action plans

Categories: Books | Celebrity Bloggers | Evaluation and ROI

On Wednesday, January 27, President Obama gave his State of the Union Address. Upon completion, the pollsters and pundants were in usual form. If you happened to watch the post-address discussion on CNN, you probably saw John King provide the latest Twitter results. That's right. Polling using Twitter. John King provided us another example of how technology is aiding us in collecting survey data.

Twitter, Facebook, LinkedIn along with SurveyMonkey, SurveyPro, Metrics-that-Matter and many other technologies provide an array of opportunity to collect data from colleagues, customers, and the like. While the program evaluation community has embraced technology to make data collection more convenient, less expensive, and more interactive, we often have such a reliance on it, that we fail to realize the potential error in the results that can surface from depending solely on technology. Types of error most immediately at risk are coverage error and non-response error.

Coverage Error
Coverage error occurs when we collect data and report results only from a group of respondents who have access to the delivery mode we employ. While admittedly, John King's results were not representative of the country at large, consider some of the people he missed:

• People who follow CNN on Twitter, but choose not to tweet.
• People who don't follow CNN on Twitter.
• People who don't know about Twitter.
• People who don't have computers.

Non-Response Error
Non-response error occurs when people do not respond to a survey. With a low response rate it becomes difficult to draw conclusions with the survey results. People fail to respond to surveys for a variety of reasons, including (but not limited to):

• Lack of time
• Lack of interest
• No incentive
• No access
  
• Too many surveys
• Too many emails 
• Technology challenged
• Technology resistant
  

In order to take advantage of what technology has to offer in terms of data collection and mitigate coverage and non-response error, consider the following steps taken from the work of Don Dillman (2009) and other experts of survey research.

1. Identify your primary mode of data collection for a given survey project.
You may choose technology as your primary mode. If so, then steps 2-5 below will use technology. If you choose paper-based or telephone surveys as your primary mode of data collection, you will use whichever one of those to complete the following steps.

2. Provide pre-notice prior to administering the survey.
This communication will come in the form of email, if you plan to email your survey; a letter or memorandum if you plan to use paper-based survey; or a brief telephone call if you plan to use telephone as your primary method of data collection. The purpose of the pre-notice is to advise potential respondents of the importance of the survey. In addition, the pre-notice will explain to them when they will receive the survey, what they can expect in terms of time commitment, completion timeline, planned use of the data, and any incentives you are willing to offer for survey completion.

3. Administer the survey.
Three days after the pre-notice has been distributed, send the survey. As part of the survey instructions, explain again the importance of survey, time commitment, completion timeline, planned use of the data, and incentives.

4. Administer the survey a second time.
After a week or two, administer the survey a second time using, again, your primary mode of delivery. This second distribution serves as a reminder and makes it convenient for the audience by providing the entire survey with instructions.

5. Send a follow-up reminder.
By now, you should have received a large number of surveys. But there are still a few people who need another reminder. So, using your primary mode of delivery, send a reminder to those who have not yet responded.

6. Administer the survey a third time -- using a different delivery method.
This last contact with potential respondents is your opportunity to influence people to respond by attacking the issue from another position. This time, change your delivery method. If your previous contacts have been electronic, send potential respondents a paper-based survey or place a call to them. By changing the delivery method, you give people who have not had access to (or who chose not to access) your survey opportunity to respond.

* * * *

Reference
Dillman, D. A., Smyth, J. D., and Christian, L. M. (2009). Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method, 3rd edition. Hoboken, New Jersey: John Wiley & Sons.

Additional Resources
Alreck, P. L. and Settle, R. B. (1995). The Survey Research Handbook, 2nd edition. New York: McGraw-Hill

Fink, A. (2002) Series Editor. The Survey Kit 2nd edition. Thousand Oaks: Sage Publications.

Trochim, W. M. The Research Methods Knowledge Base, 2nd Edition. Internet WWW page, at URL: <http://www.socialresearchmethods.net/kb/> (version current as of October 20, 2006).

Tags: measurement, evaluation, roi, surveys

Categories: Books | Evaluation and ROI

It's the middle of January and conference season is upon us. ASTD kicks it off with TK 2010 in Las Vegas. Then, its off to San Diego with Training 2010. And of course the big one - ASTD 2010 ICE takes place in May in Chicago. Some of you will attend one or more of the 2010 conferences with the intent of finding an evaluation process that fits your needs. To ensure you are clear on those needs, ask yourself the following questions.

1. What purpose will evaluation serve?

The first step toward achieving a goal is purpose clarification. Get clear on the purpose you are trying to serve through adopting an evaluation approach. Do you want to justify your spending? Do you want to increase your budget? Are you looking for an approach to help you ensure your team implements the right programs for the right people to meet the right needs? Whatever your purpose for pursuing an evaluation approach, get clear. Write a simple purpose statement to stay focused.

2. Who are our stakeholders?

While it may sound simple, it is sometimes surprising to see who gets left off this list.  Think about all of the stakeholders who have a vested interest in your training programs. Among the many stakeholders are the participants, supervisors, senior executives, and the suppliers from whom you purchase programs. There is also your team including designers, developers, performance consultants, and evaluators. Identify them all.

3. What types of decisions do these stakeholders make about our programs?

Your many stakeholders make decisions about your programs routinely. Participants decide whether or not they are going to engage. Supervisors decide whether or not they are going to support participants as they apply what they learn in a program. Executives decide whether or not they are going to continue funding programs. Think about all of the possible decisions being made about your programs.

4. What type of data do our stakeholders need to make these decisions?

Given the type of decisions being made, think about what type of information would help them make those decisions. For example, what type of information would supervisors need in order to fully support a program. Would they need to know how a program will help change the work habits of their staff? Or maybe they would need to know how a program is improving the quality of work. Maybe they would want to know what is preventing their staff from being successful with the application of knowledge and skill acquired during a program. 

5. What type of data are we providing our stakeholders?

What information do you currently provide your stakeholders about programs? Are you only sharing learning objectives or do you actually provide the success with those objectives? Are you collecting data post-program to describe to supervisors the barriers and enablers to application of knowledge and proving them insights as to how they can better support their team? Do you describe to them how improvement in quality is directly linked to your program?

6. What are the gaps?

Given your stakeholders, the types of decisions they make about your programs, the types of data they need, and the data you are currently providing, what are you missing? These gaps in your data are one of the primary needs to be filled with your new evaluation approach.

With a clear view of what you need in terms of data, look at other criteria important to your selection. Maybe you want a process that is:

• Credible 
• Simple
• Appropriate for a variety of programs
• Economical
• Theoretically sound
• Accounts for all program costs
• Accounts for other factors
• Applicable on a pre-program basis
  

Make your list and pack your bags. Upon arrival at the conference, read your purpose statement, review your list, and attend as many sessions on training evaluation as you can. 

Now you are ready to make your selection.

Tags: measurement, evaluation, roi

Categories: Books | Conferences | Evaluation and ROI

As 2009 comes to a close and 2010 rolls in, people around the world will reflect on their accomplishments and set goals for the new year. To ensure your training programs support the 2010 goals of your organization, resolve to invest more time than in the past in positioning your programs for success.   Four simple steps can help: 

1.  Clarify Needs
Before deciding to offer a program, be clear on the highest level of need.  This is a need that ultimately leads to making money, saving money, and/or avoiding costs. Examples of these opportunities are customer satisfaction, employee engagement, morale, market share, image, productivity, and operating costs. Once you are clear there, identify lower levels of need that are more specific and that lead you to a program aligned with the ultimate goal. The following series of questions can help you with this process.

• What areas need improvement that will help your organization ultimately make money, save money, or avoid cost? (Highest Level of Need)
• What specific business measures would tell you that improvement has been made?  (Business Needs)
• What needs to happen (or stop happening) in order to improve the above defined measures? (Performance Needs)
• What is it that people need to know in order to do what you want them to do to address your business needs? (Learning Needs)
• How best can you deliver knowledge, skill, and/or information people need to know to do what you want them to do? (Preference Needs)

2.  Develop SMART Objectives
Based on the identified needs, develop objectives that reflect each level of need. Be sure your objectives are SMART. Specific, measurable, achievable, realistic, and time-bound objectives representative of stakeholder needs are critical to program success. 

3.  Communicate Program Objectives
This step, while on the surface is obvious, is often overlooked. This is particularly true when it comes to participants.  Objectives are your positioning power. By communicating specific, measurable objectives reflective of all levels of need, designers, developers, and facilitators know what they need to do to make the program successful. Evaluators know what questions to ask during the evaluation. Supervisors, managers and senior leaders recognize that the program is on track with their goals. But the group to whom objectives are often not communicated so clearly are participants. This is particularly true when it comes to objectives beyond those targeting knowledge acquisition. Participants need to know not only what they are going to learn in a program, but what they are expected to do with what they learn and why they are expected to do it.  

4. Evaluate Success
If  you want to know whether or not your program is successful, evaluation is must. Programs for which needs are clear, objectives are SMART, and all stakeholders are in the 'loop' are more likely to drive results. But you will never know how well the program achieved those results or how to improve the program without evaluation The good news is that if you are clear as to why a program is being offered and you have set and communicated SMART objectives, evaluation is relatively simple!

Your Assignment                                                                                                                    When you return to work, identify a program and work with your team to answer the following questions:

1. Are the needs for this program clear?
2. Are the objectives SMART?
3. Have we communicated the objectives to everyone who needs to know? 

Further Reading

Annulis, H. and Gaudet, C. (in press) Developing Powerful Objectives. In Phillips, P. P. (editor) Handbook of Measuring and Evaluating Training. Alexandria: ASTD.                                

Phillips, J. J. and Phillips, P. P. (2008) Beyond Learning Objectives: Develop Measurable Objectives That Link to the Bottom Line. Alexandria: ASTD. 

Tags: measurement, evaluation, roi, objectives

Categories: Books | Evaluation and ROI

Tags: measurement, evaluation, roi, research

Categories: Books | Evaluation and ROI