CH-5-6 Program Design – Employee Training and Development : Book Guide

This Document Contains Chapters 5 to 6 Chapter Five Program Design Program design is the heart of effective training because it directly influences knowledge and skill acquisition. Accordingly, it is imperative that training programs be carefully designed to ensure that maximum learning is achieved. A host of factors should be considered when designing training, and the goal of this chapter is to articulate these considerations. For example, this chapter reviews practical issues in training program design, such as preparing a training site, choosing a consultant or vendor, and creating a work environment that maximizes learning and transfer of training, including manager and peer support. The role of knowledge management in transfer of training is also discussed. OBJECTIVES 1. Be able to choose and prepare a training site based on how trainees will be involved and interact with the content and each other in the course. 2. Prepare for instruction using a curriculum road map, lesson plan, design document, and concept map. 3. Explain how trainees’ age, generational differences, and personality might influence how programs are designed. 4. Prepare a request for proposal (RFP) and a list of questions to evaluate training consultants and suppliers. 5. Explain the program design elements that should be included to ensure near and far transfer of training. 6. Develop a self-management module for a training program. 7. Design application assignments and action plans to enhance learning and transfer of training. 8. Make recommendations about what managers can do before, during, and after training to facilitate learning and transfer. 9. Identify different ways to manage knowledge and the conditions necessary for employees to share knowledge. INTRODUCTION Program design refers to the organization and coordination of the training program. Program design is the heart of effective training because it directly influences knowledge and skill acquisition. Accordingly, it is imperative that training programs be carefully designed to ensure that maximum learning is achieved. It is important to take a broad perspective when designing training, regardless of whether it is an online or a face-to-face program. Employees have to be motivated to attend training, transfer trained skills, and share their knowledge with others. As such, program design should not only consider what happens during the focal training program, but also in the wider organizational context. The three phases of the program design process include pre-training, the learning event, and post-training. It is important to consider what happens before training, during training, and after training. Training should not be considered an isolated event. Phase 1—Pre-Training Involves preparing, motivating, and energizing trainees to attend the learning event Phase 2—The Learning Event Involves preparing instruction and the physical environment to facilitate learning Phase 3—Post-Training Refers to transfer of training, getting learners to apply what they have learned to their work CONSIDERATIONS IN DESIGNING EFFECTIVE PROGRAMS Selecting and Preparing the Training Site The training site refers to the room where training will be conducted. A good training site offers the following features: • It is comfortable and accessible. • It is quiet, private, and free from interruptions. • It has sufficient space for trainees to move around easily, offers enough room for trainees to have adequate work space, and has good visibility for trainees to see each other, the trainer, and any visual displays or examples Training sites can be on-site at the company or off-site such as at a resort or conference center. There is no right answer as to whether training should be held on- or off-site. Each has advantages and potential limitations. Details to be Considered in the Training Room Recognizing that learning can be facilitated through both physical and mental involvement, it is important to select a training venue that is conducive to learning. While trainers may not always have the luxury of a “perfect” training location, they should strive for a venue that helps facilitate learning. Important considerations include: • noise • colors • room structure • lighting • wall and floor covering • chairs • glare • ceiling • electrical outlets • acoustics • technology Seating Arrangements Seating arrangements should be based on an understanding of the desired type of trainee interaction and trainee-trainer interaction. • fan-type seating—trainees can easily switch from listening to practicing in groups • classroom-type seating—appropriate when lecture and audiovisual presentations are the primary training methods • conference-type seating—appropriate for total groups discussions (with no small-group interaction) • horseshoe type-seating—appropriate for both presentation and total-group instruction Choosing Trainers Trainers, whether from inside or outside the company, should have expertise in the topic and experience in training. Trainers need to be both skilled in the subject matter at hand and in program facilitation. How Trainers Can Make the Training Site and Instruction Conducive to Learning Create a Learning Setting When choosing a training room, determine the extent to which trainees decide when, where, and how they will learn, and whether learning will occur by interaction with others. Preparation of Materials You need to know your content very well. Use mental and physical rehearsal to build your confidence and assess your strengths and areas for improvement. Observe master trainers to get new ideas. Design the training from the audience’s perspective—ask “So what?” about everything you do. Know the Audience Trainers need to be aware of the shared values of the learners that may be based around age, personality, or other characteristics. Doing so will help you use language, stories, illustrations, and references that are relevant for your trainees. An awareness of learners’ ages and generations is important because it can help trainers create an appropriate learning environment. • Traditionalists prefer a standard training room with a stable, orderly learning environment. They do not like to be put on the spot in front of others. • Baby Boomers prefer classroom learning. Baby Boomers respond well to interactive training activities. They like group activities and well-organized training materials that provide an overview of the information and an easy way to access more detailed information. • Members of Generation X prefer a self-directed learning environment that includes technology-delivered methods. They respond best to training methods that allow them to work at their own pace. • Although techno-savvy, Millennials like to learn by working alone and helping others to learn. They prefer blended learning that includes self-paced online learning for learning basic concepts followed by group activities and hands-on practice for further knowledge and skill acquisition. • Nexters (like Millennials) prefer entertaining training activities. Training needs to be interactive and use music, art, and games. • Because most training groups include a mix of generations, it is important to provide a learning environment that can benefit all learners. Instructors should consider personality styles when delivering training. Many companies use the Myers-Briggs Type Indicator (MBT), which focuses on how people gather information and make decisions. Instructors can have trainees complete assessments such as this prior to training and then use this information to design training that users will find interesting and meaningful. Pre-Training: Enhance Motivation to Learn through Communications, Pre-work, and Management Involvement To enhance trainee motivation to learn, it is important to communicate to trainees the purpose of the course, learning objectives, course prerequisites, and who else will be attending. Pre-work, such as readings or application assignments, can motivate trainees and ensure trainees come to the program with a sense of purpose and focus. The training program can be more efficient when trainees complete pre-work because “the basics” have already been covered. Managers can enhance motivation by encouraging employees to attend training and setting expectations for learning. Provide an Overview of the Course When beginning a program, it is important to give learners an idea of the big picture, including the objectives, timeline, activities, and assignments. A concept map can be used for organizing and presenting knowledge. A concept map includes concepts shown in boxes, with the relationships between the concepts indicated by connecting lines. Help Trainees Retain and Recall Training Content Consider if you can chunk learning topics into short sessions of no longer than 20 minutes. After 20 minutes, you either need to take a break or use different learning techniques, such as small group discussion. Doing so helps to maintain attention and help commit learning to memory. Mnemonics and metaphors can be useful for recalling important ideas. They help relate concepts, behaviors, and knowledge to concepts that the learner already knows. Mnemonics are acronyms in which the first letter of the word represents a step in the process. ADDIE, in the context of trainer, refers to analysis, design, delivery, implementation, and evaluation. Another way to help trainees recall what they have learned is to have them consider how they can use what they are learning. Application assignments increase the likelihood that trainees will recall the training content and apply it to their work setting when they encounter the appropriate cues in the environment. Classroom Management Monitor the room for extra chairs, overflowing trash cans, and piles of materials left over from previous training sessions. Interacting with Trainees A trainer must communicate the topics that will be covered, the learning approach that will be used, and their expectations for trainees. Trainers also need to be dramatic and enthusiastic to draw attention to important points. Trainers should recognize that their expectations of trainees can result in trainees confirming those expectations, resulting in a self-fulfilling prophecy. If trainers believe trainees will be successful, trainees are more apt to learn. If trainers believe trainees will fail, they likely will. Trainers’ mindsets impact how they interact with trainees. The quality of this interaction, in turn, impacts the learning of others. How trainers engage trainees is based on the size of the room and the number of trainees. The larger the room, the larger gestures need to be. To create intimacy, trainers need to move toward trainees. One of the best ways to get trainees’ attention is to facilitate discussions from different parts of the training room. Examples of how to get trainees involved: • prepare questions to be discussed in breakout groups • use creative activities and games • use assessments for trainees to learn about themselves and others • incorporate roleplaying Leading a Discussion Discussions can be an effective means to engage trainees. They should be based on clearly defined goals and rules for participation—discussion should be planned. Discussions can be used to brainstorm ideas and solutions, debrief activities, apply knowledge, and evaluate experiences. Open ended questions should be used as opposed to questions that elicit no more than “yes” or “no” responses. It is also important to have different opening questions, follow-up questions, and discussion prompts. Dealing with Disruptive Trainees For trainees who disrupt, sleep through, or constantly interrupt the training session, consider using activities that get them moving, engaged, and energized. Ask disruptive trainees to leave only as a last resort when all other options discussed here have failed. Managing Group Dynamics To ensure an even distribution of knowledge or expertise in groups, ask trainees to indicate whether they consider themselves novice, experienced, or expert in terms of knowledge about a topic. Arrange groups so they contain individuals of different expertise. Group dynamics can be altered by changing learners’ positions in the room. Pay attention to group dynamics by wandering through the room. Curriculum, Course, and Lesson Design A curriculum refers to an organized program of study designed to meet a complex learning objective, such as preparing a learner to become a salesperson, certified computer network technician, licensed nurse, or manager. In comparison to a curriculum, a course or program usually covers more specific learning objectives and addresses a more limited number of competencies. Curriculum Road Map A curriculum road map refers to a figure showing all of the courses in a curriculum, the paths that learners can take through it, and the sequences in which courses have to be completed. A design document can be used to guide the development of training and explain the training to managers, SMEs, reviewers, or other trainers. A detailed lesson plan translates the content and sequence of training activities into a guide that is used to help deliver the training. Lesson plans include the sequence of activities that will be conducted in the training session and identify the administrative details. The lesson plan overview matches major activities of the training program and specific times or time intervals. How to Choose a Vendor or Consultant for Training Services If a company decides to purchase training, it is important to choose a high quality provider. Many companies identify potential vendors by using requests for proposals. A request for proposal (RFP) is a document that outlines for potential vendors and consultants the type of service the company is seeking, the type and number of references needed, the number of employees who need to be trained, funding for the project, the follow-up process used to determine level of satisfaction and service, and the expected date of completion of the project. Program Design Implications of Transfer of Training Determine if Focus Is on Near or Far Transfer One key consideration is whether the focus of training is on near or far transfer. Near transfer means the need to apply learned capabilities exactly in a work situation. Far transfer refers to a trainee’s ability to apply learned capabilities to the work environment when it is not identical to that of the training session. Near transfer is appropriate for closed skills, while far transfer is appropriate for open skills. Performing a closed skill involves responding to predictable situations with standardized responses. In contrast, performing an open skill involves responding to variable situations with adaptive, tailored responses. There is a one best way to perform closed skills; whereas there are multiple ways to perform open skills that are contingent upon the situation at hand. Programs that emphasizes near transfer should include the following: • using standardized procedures, processes, and checklists • providing explanations as to any differences between training and work tasks • encouraging trainees to focus only on important differences between training tasks and work tasks vs. unimportant ones • providing an explanation why the procedure should be performed to help trainees understand underlying concepts • focusing on behaviors and skills that contribute to effective performance Programs that emphasize far transfer should include the following: • teaching general concepts, broad principles, or key behaviors • focusing on general principles that might apply to a greater set of contexts beyond those presented in the training session • providing a list of prompts or questions to help trigger thoughts and questions from trainees One way to prepare trainees to deal with the obstacles that they may face in work environments is to provide instruction in self-management techniques at the end of the training program. Self-management training includes discussing relapses, setting transfer and performance goals, identifying obstacles that might inhibit transfer, generating strategies to overcome obstacles and resources, and administering rewards for successful transfer. Encourage Manager Support for Training Manager support is the degree to which trainees’ managers emphasize the importance of attending training programs and stress the application of what is learned back on the job. Increasing levels of manager support, from low to high, include: • acceptance of training • encouragement of training • participating in training • reinforcement—discussing progress with trainees • practice—allowing trainees to practice their skills • teaching—serving as a trainer Manager support can involve developing an action plan with trainees. An action plan is a written document that details steps that the trainee and his/her manager will take to ensure transfer of the training content to the job. It includes: • a goal identifying what training content will be used and how it will be used • strategies for reaching the goal and for receiving feedback • expected results Given the importance of manager support, efforts are needed to garner their commitment to training. Manager support can be gained by: • briefing managers on the purpose of the training and its relationship to the business strategy • encouraging trainees to bring work related problems to the training • sharing information regarding the benefits collected from past participants • asking trainees to complete action plans with their managers • using managers as trainers, if possible and when appropriate • rewarding managers for their support—linking support to performance appraisal and compensation systems Peer Support Transfer of training can be enhanced by a support network among trainees. A support network is a group of two or more trainees who meet and discuss their progress in using new capabilities back on the job, whether it is via face-to-face or through online communication. Peers can share successful experiences, discuss how they coped with challenges and obtained any resources, and provide encouragement and feedback. Opportunity to Use Learned Capabilities The opportunity to use learned capabilities is the extent to which the trainee is provided with, or seeks out, experiences to use new knowledge, skills, or behaviors. The opportunity to use learned capabilities is central to successful transfer. Opportunity to perform is influenced by the work environment and trainee motivation to actively pursue assignments that will utilize their new capabilities. Breadth of opportunity refers to the number of trained tasks that are performed on the job. Activity level refers to the frequency with which these tasks are performed. Task type is the difficulty or criticality of the trained tasks performed on the job. Opportunity to perform can be measured by asking former trainees to indicate: • whether they perform a task • how many times they perform the task • the extent to which they perform difficult and challenging tasks Low levels of opportunity may indicate that refresher training is needed, the work environment is unsupportive, or the training content is not relevant. Technological Support A performance support system is computer application that can provide skills training, information access, and expert advice. A performance support system can be used to enhance transfer by acting as an as-needed reference for trainees attempting to apply new knowledge, skills, and behaviors on the job. Trainers can monitor trainees’ use of these systems, which provides the trainer with valuable information about the transfer of training problems that trainees are encountering. Using Knowledge Management for Learning and Transfer of Training Knowledge management refers to the process of enhancing company performance by designing and implementing tools, processes, systems, structures, and cultures to improve the creation, sharing, and use of knowledge. Ways to create, share, and use knowledge include: • Use technology, e-mail, and social networking sites to allow people to store and share information • Publish directories of what employees do, how they can be contacted, and what knowledge they have. • Develop informational maps that identify where specific knowledge is stored in the company. • Create top management positions for cataloging and facilitating the exchange of information in the company. • Require employees to give presentations to other employees about what they have learned from training programs. • Allow employees to take time off from work to acquire knowledge, study problems, attend training, and use technology. • Create an online library of learning resources • Design office space to facilitate interaction between employees. • Create communities of practice using face-to-face meetings, wikis, or blogs for employees who share a common interest in a subject Use “after-action reviews” at the end of each project to review what happened and what can be learned from it. Keys for effective knowledge management: Training and IT collaboration. The training department and the IT department must collaborate. Training can help develop the culture, content, and learning strategies. IT can develop the systems for accessing, sharing, and storing knowledge and delivering training. Creating knowledge management leadership positions. Chief learning, or knowledge, officers (CLOs) are leaders of a company’s knowledge management efforts. Their job is to: • develop, implement, and link a knowledge/learning culture with the company’s technology infrastructure • locate knowledge and find ways to create, capture, and distribute it • ensure that trainers, information technologists, and business units support and contribute to the development of knowledge management practices • provide management direction and support for learning and knowledge management Easy-to-use technology. Building the correct technology infrastructure and making it easy for the employees to access and share information are necessary for successful knowledge management efforts. Employee trust and willingness to share. Trust and a willingness to share information are key factors that relate to knowledge sharing. To encourage knowledge sharing, companies must recognize and promote employees to learn, teach, and share. DISCUSSION QUESTIONS 1. What is a design document? What is included in a design document? How is it useful for training? A design document can be used to guide the development of training and to explain the training to managers, SMEs, reviewers, or other trainers. Information for the design document is based on the information obtained from the needs assessment. The level of detail in the design document can vary. • Scope of project includes the goals, outcomes, or achievement expectations for trainees; a description of the trainees; a description of how long it will take to develop the course and the checkpoints or tasks that need to be completed as the course is developed; and the length of the course. • Delivery includes what the course will cover, how it will be delivered, an estimate of the training time, and the identification of any special conditions or issues that may affect the course. • Objectives refer to the course or program objectives. Those are broader summary statements of the purpose of the program. • Resources refer to the materials that need to be purchased or developed for the course. • Who is involved includes trainers, program designers, and individuals who will be involved in the design, delivery, and evaluation. • The topical outline includes a brief outline of the topics that will be covered in the program. • Administration and evaluation refers to who will be in charge of course scheduling, how trainees will enroll, how the course will be evaluated, and who will review and update the course. • Links to other programs refer to any other needs, such as a train-the-trainer program or manager introduction or kickoff for the program. 2. How might course design differ for Baby Boomers compared to Gen Xers? Baby Boomers prefer classroom learning. Baby Boomers respond well to interactive training activities—they like group activities and well-organized training materials that provide an overview of the information and an easy way to access more detailed information. Compared to the other groups, they are especially motivated to learn if they believe that training content will benefit them personally. Baby Boomers need to work on translating the knowledge they have into skills. Gen Xers prefer a self-directed learning environment that includes technology-delivered methods. They respond best to training methods that allow them to work at their own pace, such as through online learning. Gen Xers are highly motivated to view training as a way to increase their employability. They like to learn by doing, experimentation, and feedback. They respond best to training materials that provide visual stimulation with relatively few words. 3. How does a concept map help learners? A concept map can be used for organizing and presenting knowledge. Such a map includes concepts shown in boxes, with the relationships between the concepts indicated by connecting lines. A concept map helps to frame key concepts and organize content into memory. 4. Explain the three phases of the instructional process, which phase do you think is most important? Why? The three phases of the instructional process include pre-training, the learning event, and post-training. Pre-training involves preparing, motivating, and energizing trainees to attend the learning event. The learning event involves preparing and providing instruction. Post-training refers to transfer of training. Student answers would vary as to which phase is most important. Typically, most effort, attention, and financial resources tend to be devoted to designing and choosing the learning event itself. However, what happens before and after the learning event may be equally, if not more, important in determining if learners are motivated to learn and transfer. 5. What could be done to increase the likelihood of transfer of training if the work environment conditions are unfavorable and cannot be changed? One way to prepare trainees to deal with the obstacles that they may face in work environments is to provide instruction in self-management techniques at the end of the training program. Self-management training includes discussing relapses, setting transfer and performance goals, identifying obstacles that might inhibit transfer, generating strategies to overcome obstacles and resources, and administering rewards for successful transfer. 6. Customer service training involves far transfer. What design features would you include in a customer service training program to ensure that transfer of training occurred? What is a curriculum road map? Why is it important? Programs that emphasize far transfer should include the following: • teaching general concepts, broad principles, or key behaviors • focusing on general principles that might apply to a greater set of contexts beyond those presented in the training session • providing a list of prompts or questions to help trigger thoughts and questions from trainees A curriculum road map refers to a figure showing all of the courses in a curriculum, the paths that learners can take through it, and the sequences in which courses have to be completed (e.g., identify prerequisite courses). It is important because for each course, information can be used to help develop, design, and evaluate the program. 7. What is an application assignment? Why should it be considered in designing a training program or course? Application assignments refer to assignments in which trainees are asked to identify work problems or situations and to apply training content to solve them. The use of application assignments in training helps trainees understand the link between the learned capabilities and real-world application, which makes it easier to recall the capability when needed. Application assignments increase the likelihood that trainees will recall the training content and apply it to their work setting when they encounter the appropriate cues (problems and work situations) in the environment. 8. How might you motivate managers to play a more active role in ensuring transfer of training? Manager support can be gained by: • briefing managers on the purpose of the training and its relationship to the business strategy • encouraging trainees to bring work related problems to the training • sharing information regarding the benefits collected from past participants • asking trainees to complete action plans with their managers • using managers as trainers, if possible and when appropriate • rewarding managers for their support—linking support to performance appraisal and compensation systems 9. If you were asked to implement a knowledge management system, what would you recommend to ensure that employees shared and accessed knowledge? Explain your recommendations. There are several strategies to help ensure employees share and access knowledge: • use technology, e-mail, and social networking sites to allow people to store and share information • publish directories of what employees do, how they can be contacted, and what knowledge they have • develop informational maps that identify where specific knowledge is stored in the company • allow employees to take time off from work for knowledge management • create an online library of learning resources • design office space to facilitate interaction among employees • create communities of practice where employees can collaborate and share ideas, solutions, and innovations • ensure technology is user friendly 10. What type of seating arrangements would you choose for a training course that involved small-group case discussions? For a lecture including PowerPoint and use of YouTube videos? Explain your choices. For small-group case discussions, a fan-type arrangement may be most effective because trainees can easily switch from listening to groups. For a lecture including PowerPoint and use of YouTube videos, classroom seating would be most appropriate because trainees can easily view these support materials and because training is more passive. 11. List the steps in project management. Discuss how each step helps in effective design. Project management refers to the skills needed to manage a team of people and resources to create a learning solution. SPADES includes project management concepts that incorporate principles of instructional systems design (ISD). • Start—involves understanding the stakeholders for the learning situation and conducting a needs assessment • Plan—involves identifying what you need to do and any resources that may be needed; identifying tasks, how long it will take to complete them, and any expertise that is needed • Administer—involves overseeing tasks and communicating with the project team, stakeholders, and SMEs; includes ensuring that tasks are completed and deadlines and budgets met • Development—involves identifying training objectives, methods, materials, and how the program will be evaluated • Engage—involves implementing training and subsequent evaluation • Stop—involves finishing the project, paying vendors and suppliers, and meeting with stakeholders Chapter Six Training Evaluation This chapter provides an overview of how to evaluate training programs, including the types of outcomes that need to be measured and the types of evaluation designs available. The chapter highlights the importance of evaluating whether the training has accomplished its objectives and, particularly, whether job performance and organizational results have improved as a result. Formative and summative evaluation are discussed and compared, and the process of evaluating training is outlined and outcomes used to evaluate training are described in detail. In an environment of accountability, expertise in assessing the effective of training from multiple perspectives is invaluable. OBJECTIVES 1. Explain why evaluation is important. 2. Identify and choose outcomes to evaluate a training program. 3. Discuss the process used to plan and implement a good training evaluation. 4. Discuss the strengths and weaknesses of different evaluation designs. 5. Choose the appropriate evaluation design based on the characteristics of the company and the importance and purpose of the training. 6. Conduct a cost-benefit analysis for a training program. 7. Explain the role of big data, workforce analytics, and dashboards in determining the value of training practices. INTRODUCTION Training effectiveness refers to the benefits that the company and trainees experience as a result of training. Benefits for the trainees include learning new knowledge, skills, and behaviors. Potential benefits for the company include increased sales, improved quality, and more satisfied customers. Training evaluation refers to the process of collecting data regarding outcomes needed to determine whether training is effective. Training outcomes, or criteria, refer to measures that the trainer and the company use to evaluate training programs. Evaluation design refers to the collection of information that will be used to determine the effectiveness of the training program. REASONS FOR EVALUATING TRAINING Companies have made large dollar investments in training and education and view training as a strategy to be successful. Training evaluation provides a way to understand the investments that training produces and provides information needed to improve training. If a company received an inadequate return on investment, the company will likely reduce its investment or look for training providers outside the company who can provide training that yields the desired results. Formative Evaluation Formative evaluation refers to the evaluation of training that takes place during program design. Formative evaluation helps to ensure that the training program is well organized and runs smoothly and that trainees learn and are satisfied. Formative evaluation provides information about how to make a program better. Formative evaluations ask employees, managers, and SMEs about their opinions about training content, methods, and the like. Training content may be changed to be more accurate, easier to understand, or more appealing, and training methods can be adjusted to improve learning. Formative evaluation involves pilot testing. Pilot testing is the process of previewing a training program with potential trainees and their managers. The pilot testing group is then asked to provide feedback about the content and the methods of delivery. This feedback enables the trainer to make any needed improvements. Summative Evaluation Summative evaluation refers to evaluation conducted to determine the extent to which trainees have changed as a result of training. That is, summative evaluation examines whether trainees have improved or acquired knowledge, skills attitudes, behaviors, or other outcomes. Summative evaluation may also include examining the business impact of training. The Importance of Evaluation There are multiple reasons to evaluate training effectiveness: • To identify the program’s strengths and weaknesses, including whether the program is meeting the learning objectives, the quality of the learning environment, and if transfer is occurring. • To assess whether the various features of the training context and content contribute to learning and transfer. • To identify which trainees benefited most or leased from the program and why. • To gather information, such as testimonials, to use for marketing training. • To determine financial benefits and costs of the program. • To compare the costs and benefits of training versus other HRM investments. • To compare the costs and benefits of various training programs in order to choose the most effective programs. OVERVIEW OF THE EVALUATION PROCESS The evaluation process involves five key components. Needs assessment The evaluation process should begin with determining training needs. Needs assessment helps identify what knowledge, skills, behavior, or other learned capabilities are needed. Develop measurable learning objectives and analyze transfer of training Identify specific, measurable training objectives to guide the program. Besides considering the learning program objectives, it is important to consider the expectations of those individuals who support the program and have an interest in it. Analysis of the work environment can be useful for determining how well content will be transferred. Development outcome measures Based on the preceding steps, outcome measures are designed to assess the extent to which learning and transfer have occurred. Choose an evaluation strategy Once the outcomes have been identified, determine an evaluation strategy. Factors such as expertise, how quickly the information is needed, change potential, and the organizational culture should be considered. Plan and execute the evaluation Planning and executing the evaluation involves previewing the program, as well as collecting training outcomes according to the evaluation design. The results of the evaluation are used to modify, market, or gain additional support for the program. OUTCOMES USED IN THE EVALUATION OF TRAINING PROGRAMS There are six primary outcomes that can be used to evaluate training effectiveness. Reaction Outcomes Reaction outcomes refer to the trainees’ perceptions of the training experience relating to the content, facilities, trainer, and methods of delivery. An accurate evaluation should include all the factors related to a successful learning environment. Reactions are often referred to as a measure of “creature comfort.” Key questions to consider include: • Did the trainees like the program? • Did the environment help learning? • Was material meaningful? This information is typically collected at the program’s conclusion via a questionnaire. Reactions are often assessed by asking trainees to respond to a series of strongly agree-strongly disagree statements about the learning experience. Reaction measures can also include open-ended questions about the experience, such as “What did you learn that you are most likely to try on the job?” and “What topics covered in this program seem confusing?” It is often believed that reactions are related to learning and transfer. Research suggests that reactions have the strongest relationship with affective outcomes. Reactions are also significantly related to changes in declarative and procedural knowledge. Learning or Cognitive Outcomes Learning or cognitive outcomes relate to familiarity with information, including principles, facts, techniques, procedures, and processes. Typically paper and pencil tests or self-assessments are used to assess cognitive outcomes. Self-assessments refer to learners’ estimates of how much they learned from training. Tests and quizzes are often preferred over self-assessments because self-assessments are only moderately related to learning. Self-assessments are more strongly related to learners’ reactions and their motivation to learn. Behavior and Skill-Based Outcomes Behavior and skill-based outcomes relate to proficiency with technical or motor skills and behavior. These outcomes include both the learning of skills and their transfer. Skill learning is often assessed by observing performance in work samples. Skill transfer is typically assessed by observing trainees on the job or via managerial and peer ratings. Affective Outcomes Affective outcomes include attitudes and motivation. Affective outcomes that might be collected include self-efficacy, employee engagement, motivation to learn, tolerance for diversity, safety attitudes, and customer service orientation. The attitude of interest depends on training objectives. Affective outcomes can be measured using surveys. Results Results are those outcomes used to determine the benefits of the training program to the company. Examples include reduced costs, increased employee retention, increased sales, increased production, and improved quality or customer service. Return on Investment Return on Investment involves comparing the training program’s benefits in monetary terms to the program’s costs. • Benefits are the value the company receives from the training. • Direct costs include salaries and benefits of trainees, trainers, consultants, and any others involved in the training; program materials and supplies; equipment and facilities; and travel costs. • Indirect costs include office supplies, facilities, equipment and related expenses not directly related to the training program; travel and expenses not billed to one particular program; and training department management and staff support salaries. DETERMINING WHETHER OUTCOMES ARE APPROPRIATE An important issue in choosing outcomes is to determine whether they are appropriate. Relevance Criteria relevance refers to the extent to which training outcomes are related to the learned capabilities emphasized in the program. One way to ensure the relevance is to choose outcomes based on the learning objectives for the program. There are two ways outcomes may lack relevance—criterion contamination and criterion deficiency. • Criterion contamination refers to the inclusion of inappropriate or irrelevant outcomes. • Criterion deficiency refers to the omission of important information. Reliability Reliability is whether outcomes can be measured consistently over time. Predominantly, we are concerned with consistency over time, such that a reliable test contains items that do not change in meaning or interpretation over time. Discrimination Discrimination refers to whether performance on the outcome reflects true differences in performance. For example, we want tests that can discriminate between high and low performance. A test that may not discriminate is one that is too easy. In this instance, both high and low performers would do well on the test. Both groups would appear “good” even though they are not. Practicality Practicality is the ease with which the outcome measures can be collected. One reason companies give for not including learning, performance, and behavior outcomes is that collecting them is too burdensome. EVALUATION PRACTICES Below are percentage estimates of organizations examining different training outcomes. • Reactions 92% • Cognitive 81% • Behavior 55% • Results 37% • ROI 18% • None 4% Reactions and cognitive outcomes are the most frequently used outcomes in training evaluation. Despite the less frequent use of cognitive, behavioral, and results outcomes, research suggests that training can have a positive effect on these outcomes. There are a number of reasons why companies fail to evaluate training. Learning professionals report that access to data and tools needed to obtain them are the most significant barriers. Which Training Outcomes Should Be Collected? It is not always necessary to collect data on all of the training outcomes. While collecting data on all outcomes is ideal, doing so may not always be necessary depending on the scope of the training, its strategic value, and practical considerations. As much as possible, evaluation should include behavior or skill-based, affective, and results outcomes to determine the extent to which transfer occurred. (Reactions and cognitive measures do not help to measure transfer.) It is important to recognize the limitations of only choosing to measure reactions and cognitive outcomes, the two most commonly measured outcomes. The various training outcomes measures are largely independent of each other. It cannot be assumed that positive reactions lead to greater transfer. Research suggests that the relationships among the outcomes are small. There are three types of transfer: • Positive transfer is demonstrated when learning occurs with positive changes in on-the-job behavior. • No transfer is demonstrated when learning occurs without changes in on-the-job behavior, • Negative transfer is evident when learning occurs, but on-the-job behavior is lower than pre-training levels. Learning, behavior, and results should be measured after sufficient time has elapsed to determine whether training has had an influence on these outcomes. EVALUATION DESIGNS The design of the training evaluation determines the confidence that can be placed in the results. No training evaluator can be absolutely certain that the results of the evaluation are completely true. The evaluator should strive for the most rigorous design possible. Threats to Validity: Alternative Explanations for Evaluation Results Threats to validity refer to factors that will lead an evaluator to question either the believability of the study results or the extent to which the evaluation results are generalizable to other groups of trainees and situations. Internal validity is the believability of the study. An evaluation needs internal validity to provide confidence that the findings are due to training and not another factor. External validity refers to the generalizability of the evaluation results to other groups and other situations. Methods to control for threats to validity: • Use pre-tests and post-tests to determine the extent to which trainees have changed from pre-training to post-training measures. The pre-training measure essentially establishes a baseline. • Use a control group (i.e., a group that participates in the evaluation study, but does not receive the training) to rule out factors other than training as the cause of changes in the trainees. • Random assignment of employees to control and training groups. Randomization helps to ensure that members of the control group and training group are of similar makeup prior to the training. Types of Evaluation Designs Posttest only The post-test only design involves collecting only post-training outcome measures. This design would be strengthened by the use of a control group to rule out alternative explanations. This design is appropriate when trainees can be expected to have similar levels of knowledge, behavior, etc. prior to training. Pretest/Posttest This design involves collecting both pre-training and post-training outcome measures to determine whether a change has occurred after training. Pretest/Posttest with Comparison Group This design includes pre-training and post-training outcome measurements and a control group. If the post-training improvement is greater for the group that receives training, there is evidence that training was responsible for the change. Time Series The time series design involves collecting outcome measurements at periodic intervals pre- and post-training. A comparison group may also be used. The strength of this design can be improved by using reversal, which refers to a time period when participants no longer receive training. This design allows for an analysis of the stability of training outcomes over time. Solomon Four-Group The Solomon Four-Group design combines the pretest/posttest comparison group design and the posttest-only control group design. It involves the use of four groups: • Pre-test, treatment, post-test • Pre-test, no treatment, post-test • No pre-test, treatment, post-test • No pre-test, no treatment, post-test This design provides the most controls for internal and external validity. Considerations in Choosing an Evaluation Design There are several reasons why no evaluation or a less rigorous design may be appropriate: • Managers and trainers may be unwilling to devote the time • Managers and trainees may lack the expertise to evaluate • The company may view training as an investment from which it expects little or no return A more rigorous evaluation design should be considered when: • The evaluation results can be used to change the program. • The training is ongoing and has the potential to affect many employees. • The training program involves multiple classes and a large number of trainees. • Cost justification for training is based on numerical indicators. • Trainers or others in the company have the expertise to evaluate. • The cost of training creates a need to show that it works. • There is sufficient time for conducting an evaluation. • There is interest in measuring change from pre-training levels or in comparing two or more different programs. Evaluation designs without pretesting or comparison groups are most appropriate when you are interested only in whether a specific level of performance has been achieved, and not how much change has occurred. Sometimes naturally occurring comparison groups are available. This can occur because of the realities of scheduling employees to attend training, when not all employees can attend. When this occurs, for example, a pre-test/post-test design could be employed with a control group. DETERMINING RETURN ON INVESTMENT ROI is an important training outcome. ROI can be assessed by conducting a cost-benefit analysis. A cost-benefit analysis determines the net economic benefits of training using accounting methods. There is an increased interest in measuring the ROI of training because of the need to show results. However because ROI analysis can be costly, it should be limited only to training programs where there was a significant investment. Training cost information is important for several reasons: • to understand total expenditures for training, including direct and indirect costs • to compare the costs of alternative training programs • to evaluate the proportion of the training budget spent on the development of training, administrative costs, and evaluation as well as how much is spent on various types of employees • to control costs The process of determining ROI includes: • understanding the objectives of the training program • isolating the effects of training from other factors that might influence the data • converting data to a monetary value and calculating ROI Determining Costs One method to determine costs is the resource requirements model. The resource requirements model compares equipment, facilities, personnel, and materials costs across different stages of the training process. This model can help determine overall differences in costs among training programs. Cost incurred at different stages of the training process can be compared across groups. Accounting can also be used to calculate costs. There are seven categories of cost sources: • program development or purchase • instructional materials • equipment and hardware • facilities • travel and lodging • salary of the trainer and support staff • cost of either lost productivity or replacement workers while trainees are away from their jobs Determining Benefits To identify benefits, one must review the original reasons training was conducted. A number of methods may be helpful to identify training benefits, including: • technical, practitioner and academic literature that summarizes benefits of training programs • pilot training programs that assess the benefits from a small group of trainees before a company commits more resources • observing successful job performers to determine what successful job performers do differently from unsuccessful ones • asking trainees and managers to provide estimates of benefits Calculating ROI To calculate return on investment, follow these steps: • identify outcomes • place a monetary value on the outcomes • determine the annual change in outcomes • obtain an annual amount of benefits by multiplying the change in outcomes by the monetary value • determine training costs • calculate net benefit by subtracting the training costs from benefits • calculate ROI by dividing net benefits by costs—the ROI gives an estimate of the dollar return expected from each dollar invested in training Other Methods of Cost-Benefit Analysis Utility analysis assesses the dollar value of training based on estimates of the difference in job performance between trained and untrained employees, the number of employees trained, the length of time the program is expected to influence performance, and the variability in job performance in the untrained group of employees. Utility analysis employs a highly sophisticated formula that requires the use of pretest and posttest with a comparison group. Other types of economic analyses evaluate training as it benefits the firm or government using direct and indirect costs, incentives paid by the government for training, wage increases received by trainees as a result of the training, tax rates, and discount rates. Practical Considerations in Determining Return on Investment Training programs best suited for ROI analysis have clearly identified outcomes, are not one-time events, are highly visible in the company, are strategically focused, and have effects that can be isolated. Success Cases and Return on Expectations Success cases refer to concrete examples of the impact of training that show how learning has led to results that the company finds worthwhile and the managers find credible. Success cases do not attempt to isolate the influence of training, but rather to provide evidence that training was useful. Return on expectations (ROE) demonstrates to key business stakeholders, such as top-level managers, that their expectations about training have been satisfied. ROE depends on establishing a business partnership with key stakeholders from the start of a training program through its evaluation. MEASURING HUMAN CAPITAL AND TRAINING ACTIVITY It is important to remember that evaluation can involve determining the extent to which training contributes to business strategy and helps achieve business goals. Metrics are valuable for benchmarking purposes, for understanding the current amount of training activity in a company, and for tracking historical trends in training activity. Another way to understand the value of training is through comparisons with other companies; companies could review the ATD’s report that summarizes company provided data in the U.S. Big Data and Workforce Analytics Big data refer to complex datasets developed by compiling data across different organizational systems, including marketing and sales, HR, finance, accounting, customer service, and operations. Three dimensions characterize big data: volume, variety, and velocity. • Volume refers to the amount of available data. • Variety includes the large number of sources and types of data. • Velocity refers to the huge amount of data that is being generated and the speed with which it must be evaluated, captured, and made useful. In the present context, big data allow for the making of decisions about human capital based on data, rather than on intuition and conventional wisdom. In a training context, big data can be used to: • evaluate the effectiveness of programs • determine their impact on business results • develop predictive models for forecasting training needs, course enrollments, and outcomes Workforce analytics refers to the practice of using quantitative and scientific methods to analyze data from HR databases, corporate financial statements, employee surveys, and other data sources. DISCUSSION QUESTIONS 1. What can be done to motivate companies to evaluate training programs? Management should be made aware of the multiple reasons to evaluate training effectiveness: • To identify the program’s strengths and weaknesses, including whether the program is meeting the learning objectives, the quality of the learning environment, and if transfer of training back to the job is occurring. • To assess whether the various features of the training context and content contribute to learning and the transfer of learning back to the job. • To identify which trainees benefited most or leased from the program and why. • To gather information, such as trainees’ testimonials, to use for marketing training programs. • To determine financial benefits and costs of the program. • To compare the costs and benefits of training versus other human resource investments. • To compare the costs and benefits of various training programs in order to choose the most effective programs. Many companies may not evaluate training due to a lack of expertise, time, and resources. Companies may incorrectly assume that training automatically results in value, which may not always be the case. Efforts should be made to overcome these obstacles. In doing so, companies may be more motivated evaluate training. 2. What do threats to validity have to do with training evaluation? Identify internal and external threats to validity. Are internal and external threats similar? Explain. If threats to validity exist, the evaluator may question whether a training program was really effective or if possible benefits were the result of other factors. Internal validity is the believability of the study. An evaluation needs internal validity to provide confidence that the findings are due to training and not another factor. External validity refers to the generalizability of the evaluation results to other groups and other situations. Characteristics of the company, the outcome measures, and the persons in the evaluation study are the internal threats to validity. Reaction to pretest, reaction to evaluation, interaction of selection and training, and interaction of methods are the external threats to validity. Internal and external threats are not similar. Threats to external validity relate to how study participants react to being included in the study and the effects of multiple types of training, whereas internal validity provides confidence that the results of the evaluation are due to the training program and not to another factor. 3. What are the strengths and weaknesses of each of the following designs: posttest-only, pretest/posttest with comparison group, and pretest/posttest only? A post-test-only design requires less time and effort than the others, but fails to recognize initial differences in trainees or the possibility that the outcomes are due to another business. Pretest/post-test only acknowledges the performance of the trained group before the training, but ignores other business factors that may be occurring between the two evaluations. Pretest/post-test with comparison group is the most thorough, but involves the most time and effort to collect data. 4. What are results outcomes? Why do you think that most organizations don’t use results outcomes for evaluating their training programs? Results outcomes are level 4 criteria in Kirkpatrick’s framework. They are the benefits of the training program for the company, such as lower employee turnover, increased sales, better customer service, and the like. These data may be difficult to collect, and companies may not have expertise to statistically link training to these outcomes. 5. This chapter discussed several factors that influence the choice of evaluation design. Which of these factors would have the greatest influence on your choice of and evaluation design? Which would have the least influence? Explain your choices. The factors that influence the choice of evaluation design include: • change potential—which determines whether the program can be modified • importance—which determines whether ineffective training affects customer service, product development, or relationships among employees • scale—which determines the number of trainees involved • purpose of training—which determines whether training is conducted for learning, results, or both • organization culture—which determines whether demonstrating results is a part of company norms and expectations • expertise—which determines the possibility of analyzing a complex study • cost—which determines whether the evaluation is expensive • time frame—which determines when the information is needed 6. How might you estimate the benefits of a training program designed to teach employees how to use the Internet to monitor stock prices? A test could be performed to see if the employees are capable of monitoring the stock prices. Surveys could be used to see how frequently they monitor stock in a given day to determine how well they have integrated that knowledge into their behavior. Then employees could estimate how much behavior benefits the company. To determine the economic benefits of the training program, ROI can be calculated through a cost benefit analysis. Also, the following methods may help to identify the benefits of the training program: • Technical, academic, and practitioner literature that summarizes the benefits that have been shown to relate to the specific training program. • Observance of successful job performers helps a company determine what successful job performers do differently than unsuccessful job performers. • Trainees and their managers provide estimates of training benefits. 7. A group of managers (N=25) participated in the problem-solving module of a leadership development program two weeks ago. The module consisted of two days in which the group focused on the correct process to use in problem solving. Each manager supervises fifteen to twenty employees. The company is willing to change the program, and there is an increasing emphasis in the company to show that training expenses are justifiable. You are asked to evaluate this program. Your boss would like the results of the evaluation no later than six weeks from now. Discuss the outcomes you would collect and the design you would use. How might your answer change if the managers have not yet attended the program? If the evaluation is done after the program is completed, a posttest only design could be used, comparing the group of managers to a control group. You could assess trainees reactions at this point, to have them reflect on their perceived valued of the program. More importantly, you could have the managers provide self-assessments of their performance and motivation to transfer. Specifically, you could examine various decisions made by the managers over several weeks and determine their effectiveness in solving various problems. Further, the managers’ employees could provide data on the managers’ ability to successfully transfer. If the managers had not yet attended the program, they can be used as a comparison group or pre-test skills and productivity can be recorded for later comparison. Hence, a pretest/post-test with comparison group method could be applied. 8. What practical considerations need to be taken into account when calculating a training program’s ROI? Training programs best suited for ROI analysis have clearly identified outcomes, are not one-time events, are highly visible in the company, are strategically focused, and have effects that can be isolated. 9. What is return on expectations (ROE)? How can it be used to show the costs and benefits of training without collecting statistics and conducting analyses? Explain its strengths and weaknesses compared to a cost-benefit analysis. Return on expectations (ROE) refers to the process through which evaluation demonstrates to key business stakeholders, such as top-level managers, that their expectations about training have been satisfied. ROE depends on establishing a business partnership with business stakeholders from the start of a training program through its evaluation. The ROE is used as an estimate in an ROI analysis. 10. What are the characteristics of big data? Explain how big data could be used to show that learning influences business outcomes. Big data refers to complex datasets developed by compiling data across different organizational systems, including marketing and sales, HR, finance, accounting, accounting customer service, and operations. Three dimensions characterize big data: volume, variety, and velocity. • Volume refers to the amount of available data. • Variety includes the large number of sources and types of data. • Velocity refers to the huge amount of data that is being generated and the speed with which it must be evaluated, captured, and made useful. Data can be used from different sources to demonstrate relationships between learning and business outcomes, such as data from employee records, learning and performance management, systems and customer relationship management systems. The goal is the merge the data from these sources to show that employees who attended training had positive influence of different metrics. Solution Manual for Employee Training and Development Raymond Andrew Noe 9780078112850, 9781259539367, 9780071267786