(May 6) This was my first CSTD symposium but not my last. Not only was it a fun experience, it was also a great learning and networking experience. I was able to exchange with people I had not met before and I was given the opportunity to share my thoughts and ideas with them. This symposium also caused me to pause and reflect on what I was doing as a training professional and on how I can change things for the better. I also came away with some tools that I anticipate trying in the next few weeks. This was one of the best two days I spent this year and I hope to be able to attend the next symposium in Vancouver.
Submitted by: Ofelia Ribeiro
Wednesday, May 9, 2007
Angela's Final Thoughts and Observations
(May 7)
Thought #1: What a Symposium! Several interesting sessions that fostered hallway discussions and exchanges (dare I call that informal learning). Interesting to note that most presenters who were internal to the organization never referred to what they did as ‘training’, but seemed to selectively choose the words ‘learning’ and ‘development’. The word ‘training’ was used by an external consultant. I ascribe no meaning to this publicly, but only mention it as an observation.
Thought #2: What I learned from all this…there are some terrific people doing terrific things for their organizations. I learned a little about informal learning (and obviously have a lot more to figure out in order to make sense of evaluating this elusive creature). One common theme that seemed to recur was the need to establish good and consistent communication with regard to strategy, change management, etc., and this communication and engagement needs to be at all levels of the organization. Everyone needs to be on the same page, talking the same language, moving in the same direction.
Another common theme was the need to link to corporate/business objectives and goals. The subtitle of the Symposium is ‘How to know if your learning is meeting business needs.’ Well, the message was clear that we should be linking to business needs, and we should be mapping/measuring the impact of training to corporate goals – this was the WHAT. What I need to do is link to corporate goals and objectives, and measure. But where’s the HOW? How do I show learning’s distinct and specific contribution to corporate outcomes? Am I talking just about formal learning? How about informal learning’s impact on performance and, ultimately corporate goals?
Ahh – it was a good Symposium. Thank you for letting me share my thoughts, my learnings, my questions. And a respectful expression of gratitude to all the presenters who shared experiences and wisdom with me. Now, where do I get a ticket to Le Cirque du Soleil?
Submitted by: Angela van Barneveld
Thought #1: What a Symposium! Several interesting sessions that fostered hallway discussions and exchanges (dare I call that informal learning). Interesting to note that most presenters who were internal to the organization never referred to what they did as ‘training’, but seemed to selectively choose the words ‘learning’ and ‘development’. The word ‘training’ was used by an external consultant. I ascribe no meaning to this publicly, but only mention it as an observation.
Thought #2: What I learned from all this…there are some terrific people doing terrific things for their organizations. I learned a little about informal learning (and obviously have a lot more to figure out in order to make sense of evaluating this elusive creature). One common theme that seemed to recur was the need to establish good and consistent communication with regard to strategy, change management, etc., and this communication and engagement needs to be at all levels of the organization. Everyone needs to be on the same page, talking the same language, moving in the same direction.
Another common theme was the need to link to corporate/business objectives and goals. The subtitle of the Symposium is ‘How to know if your learning is meeting business needs.’ Well, the message was clear that we should be linking to business needs, and we should be mapping/measuring the impact of training to corporate goals – this was the WHAT. What I need to do is link to corporate goals and objectives, and measure. But where’s the HOW? How do I show learning’s distinct and specific contribution to corporate outcomes? Am I talking just about formal learning? How about informal learning’s impact on performance and, ultimately corporate goals?
Ahh – it was a good Symposium. Thank you for letting me share my thoughts, my learnings, my questions. And a respectful expression of gratitude to all the presenters who shared experiences and wisdom with me. Now, where do I get a ticket to Le Cirque du Soleil?
Submitted by: Angela van Barneveld
Session Review: Evaluation of Learning
(May 4) Hafsa El Khettab and François Ste-Marie or ARINSO Canada facilitated a workshop of learning evaluation. Unfortunately trainers do not always put much energy into developing evaluation tools for the training programs they develop. But evaluating the results of training is crucial if we are to attain the objectives we have been set. The presenters led an animated discussion with delegates sharing their past experiences as well as some of the challenges they have been given to develop evaluation tools.
The highlight of the workshop was the presentation of numerous tools developed by the workshop facilitators. Participants were able to try some of these out through small group exercises. Many of these tools grew from collaborations with instructional designers and other training and development professionals. Essentially, evaluation tools should evolve from training goals and objectives. This is the only way to ensure that there is a link between training and organizational needs.
Submitted by: Ofelia Ribeiro
The highlight of the workshop was the presentation of numerous tools developed by the workshop facilitators. Participants were able to try some of these out through small group exercises. Many of these tools grew from collaborations with instructional designers and other training and development professionals. Essentially, evaluation tools should evolve from training goals and objectives. This is the only way to ensure that there is a link between training and organizational needs.
Submitted by: Ofelia Ribeiro
Friday Keynote: Connecting the Dots on Training ROI: Skills Development to Business Outcomes…
(May 4)
Summary: In this session, Allan presented some attention-getting statistics on Canada’s failing productivity, and showed the link between economy, productivity, and skills. Canadian business invests less money per employee on training when compared to American and European countries. Productivity has declined in small and medium business enterprises over the last 25 years. The major culprit is basic skills, literacy – both requirements to participate in the knowledge society (so impact is economic and social). The statistics presented to evidence that training adds value were mostly from small and medium business data. Understandably, the larger ROI numbers were documented for situations where low-skilled workers are provided with training. Corporate metrics such as shareholders returns were noted to be higher in organizations that invested in training. However, with al this data, Allan still acknowledged that ROI was still a nebulous concept for most, and understanding and interpreting the ROI numbers that come out of a study are still a challenge for most of us.
Ang’s thoughts: A fascinating presentation – moreso again from the point of view from all the questions it stimulated. Yesterday, it was mentioned that the evaluation models put forth by Kirkpatrick and Phillips were applicable to formal training situations which, as we also now know, are about 20-30% of the learning that goes on in an organization. This is the data that is used to calculate ROI (return on investment). I had a chance to speak with Allan after his presentation and asked him for his perspective on a few things. For instance, if up to 80% of learning in an organization is informal, what impact does that have on ROI? If ROI is calculated based only on formal learning, where is the consideration given for the impact or modifying effect of informal learning on ROI? I cannot see how you can control for informal learning effects so that ROI is cleanly and solely the result of formal learning, but Allan is much more the expert in this area and indicated it was possible. However, he acknowledged that he, too, had an interest in the relationship between ROI and informal learning and would do more work in that area in the future. FANTASTIC!
Submitted by: Angela van Barneveld
Summary: In this session, Allan presented some attention-getting statistics on Canada’s failing productivity, and showed the link between economy, productivity, and skills. Canadian business invests less money per employee on training when compared to American and European countries. Productivity has declined in small and medium business enterprises over the last 25 years. The major culprit is basic skills, literacy – both requirements to participate in the knowledge society (so impact is economic and social). The statistics presented to evidence that training adds value were mostly from small and medium business data. Understandably, the larger ROI numbers were documented for situations where low-skilled workers are provided with training. Corporate metrics such as shareholders returns were noted to be higher in organizations that invested in training. However, with al this data, Allan still acknowledged that ROI was still a nebulous concept for most, and understanding and interpreting the ROI numbers that come out of a study are still a challenge for most of us.
Ang’s thoughts: A fascinating presentation – moreso again from the point of view from all the questions it stimulated. Yesterday, it was mentioned that the evaluation models put forth by Kirkpatrick and Phillips were applicable to formal training situations which, as we also now know, are about 20-30% of the learning that goes on in an organization. This is the data that is used to calculate ROI (return on investment). I had a chance to speak with Allan after his presentation and asked him for his perspective on a few things. For instance, if up to 80% of learning in an organization is informal, what impact does that have on ROI? If ROI is calculated based only on formal learning, where is the consideration given for the impact or modifying effect of informal learning on ROI? I cannot see how you can control for informal learning effects so that ROI is cleanly and solely the result of formal learning, but Allan is much more the expert in this area and indicated it was possible. However, he acknowledged that he, too, had an interest in the relationship between ROI and informal learning and would do more work in that area in the future. FANTASTIC!
Submitted by: Angela van Barneveld
Supper and Networking at Mount Stephen Club
(May 3) The magnificent home of wealthy businessman Mount Stephen now transformed into a club was the backdrop to a fun evening of networking. Plentiful wine encouraged conversation and exchanges of phone numbers and email addresses. I have many pictures with old friends and new acquaintances in sumptuous surroundings. We were allowed to visit all the rooms in the club and many of us imagine what it must have been like to live in this home at the turn of the century. The supper was excellent and people mingled throughout until very very late. This was possibly the best ending to a stimulating day.
Submitted by: Ofelia Ribeiro
Submitted by: Ofelia Ribeiro
Session Review: Bridging the Gap Between Educational and Business Evaluation
(May 3) Claude Martel of Educonsillium proposes that evaluation methods must consider not just what the individual learned but how this impacts on the organization. It is a given that all organizations must engage in training evaluation. However, the reasons why trainers engage in evaluation do not always link with what organizational leaders are looking for. One reason is that we doe not use the same language – or vocabulary – to examine the impact of training. While the training department looks at learner satisfaction or learning, administrators are more commonly concerned with how the training supports strategic processes and impacts on critical competences.
Martel led a lively discussion and facilitated an examination of how trainers can better understand managers’ needs and how they see training efforts. Much of the disparity is related to our not seeing things in the same way. As trainers we often have to prove the value of what we do, how it links with organizational goals and why it should be supported. As training departments are challenged to align their activities with strategic needs, indicators of success should be remodeled to take into account our new role. This presentation encouraged fruitful reflection on how we all can do our jobs better. I know there are a number of ideas that I will carry with me as I build my career.
Submitted by: Ofelia Ribeiro
Martel led a lively discussion and facilitated an examination of how trainers can better understand managers’ needs and how they see training efforts. Much of the disparity is related to our not seeing things in the same way. As trainers we often have to prove the value of what we do, how it links with organizational goals and why it should be supported. As training departments are challenged to align their activities with strategic needs, indicators of success should be remodeled to take into account our new role. This presentation encouraged fruitful reflection on how we all can do our jobs better. I know there are a number of ideas that I will carry with me as I build my career.
Submitted by: Ofelia Ribeiro
Session Review: Assessment of knowledge: Test Items that Measure Learning Effectively
(May 3)
Summary: In this session, Atena presented the process and procedure that the TD Bank Financial Group uses to develop their test items. The strategy was to develop performance-based items that measure on-the-job transferable knowledge and skills. Through the analysis process, they determine if testing is required and, if yes, ensure that the test items map to the learning objectives, which map to the performance objectives, which map to the business objectives. Performance-based items focus more on testing conceptual, process, procedure, and principle-based content which usually requires an element of judgment or decision-making. This is in contrast to test items that focus on facts and access recall rather than higher-level cognitive skills. Also covered are item writing strategies, establishing validity and reliability, and testing the test.
Ang’s thoughts: A new approach that I came away with is the value of asking some L3 questions on the L1 evaluation to set a baseline. This session was more a confirmation of my existing knowledge of test item development and processes – always a nice thing. One thing that Atena reinforced was that the writing of test items is not an easy task. Perhaps the development of the stem is the easiest piece, but coming up with feasible distractors is a real tough one. It was mentioned in an audience discussion that the expected number of good test items that someone would write on a day is somewhere between 8 and 10. This may surprise most people, but perhaps not those who have actually been involved in creating good test items.
Submitted by: Angela van Barneveld
Summary: In this session, Atena presented the process and procedure that the TD Bank Financial Group uses to develop their test items. The strategy was to develop performance-based items that measure on-the-job transferable knowledge and skills. Through the analysis process, they determine if testing is required and, if yes, ensure that the test items map to the learning objectives, which map to the performance objectives, which map to the business objectives. Performance-based items focus more on testing conceptual, process, procedure, and principle-based content which usually requires an element of judgment or decision-making. This is in contrast to test items that focus on facts and access recall rather than higher-level cognitive skills. Also covered are item writing strategies, establishing validity and reliability, and testing the test.
Ang’s thoughts: A new approach that I came away with is the value of asking some L3 questions on the L1 evaluation to set a baseline. This session was more a confirmation of my existing knowledge of test item development and processes – always a nice thing. One thing that Atena reinforced was that the writing of test items is not an easy task. Perhaps the development of the stem is the easiest piece, but coming up with feasible distractors is a real tough one. It was mentioned in an audience discussion that the expected number of good test items that someone would write on a day is somewhere between 8 and 10. This may surprise most people, but perhaps not those who have actually been involved in creating good test items.
Submitted by: Angela van Barneveld
Subscribe to:
Posts (Atom)
