(May 6) This was my first CSTD symposium but not my last. Not only was it a fun experience, it was also a great learning and networking experience. I was able to exchange with people I had not met before and I was given the opportunity to share my thoughts and ideas with them. This symposium also caused me to pause and reflect on what I was doing as a training professional and on how I can change things for the better. I also came away with some tools that I anticipate trying in the next few weeks. This was one of the best two days I spent this year and I hope to be able to attend the next symposium in Vancouver.
Submitted by: Ofelia Ribeiro
Wednesday, May 9, 2007
Angela's Final Thoughts and Observations
(May 7)
Thought #1: What a Symposium! Several interesting sessions that fostered hallway discussions and exchanges (dare I call that informal learning). Interesting to note that most presenters who were internal to the organization never referred to what they did as ‘training’, but seemed to selectively choose the words ‘learning’ and ‘development’. The word ‘training’ was used by an external consultant. I ascribe no meaning to this publicly, but only mention it as an observation.
Thought #2: What I learned from all this…there are some terrific people doing terrific things for their organizations. I learned a little about informal learning (and obviously have a lot more to figure out in order to make sense of evaluating this elusive creature). One common theme that seemed to recur was the need to establish good and consistent communication with regard to strategy, change management, etc., and this communication and engagement needs to be at all levels of the organization. Everyone needs to be on the same page, talking the same language, moving in the same direction.
Another common theme was the need to link to corporate/business objectives and goals. The subtitle of the Symposium is ‘How to know if your learning is meeting business needs.’ Well, the message was clear that we should be linking to business needs, and we should be mapping/measuring the impact of training to corporate goals – this was the WHAT. What I need to do is link to corporate goals and objectives, and measure. But where’s the HOW? How do I show learning’s distinct and specific contribution to corporate outcomes? Am I talking just about formal learning? How about informal learning’s impact on performance and, ultimately corporate goals?
Ahh – it was a good Symposium. Thank you for letting me share my thoughts, my learnings, my questions. And a respectful expression of gratitude to all the presenters who shared experiences and wisdom with me. Now, where do I get a ticket to Le Cirque du Soleil?
Submitted by: Angela van Barneveld
Thought #1: What a Symposium! Several interesting sessions that fostered hallway discussions and exchanges (dare I call that informal learning). Interesting to note that most presenters who were internal to the organization never referred to what they did as ‘training’, but seemed to selectively choose the words ‘learning’ and ‘development’. The word ‘training’ was used by an external consultant. I ascribe no meaning to this publicly, but only mention it as an observation.
Thought #2: What I learned from all this…there are some terrific people doing terrific things for their organizations. I learned a little about informal learning (and obviously have a lot more to figure out in order to make sense of evaluating this elusive creature). One common theme that seemed to recur was the need to establish good and consistent communication with regard to strategy, change management, etc., and this communication and engagement needs to be at all levels of the organization. Everyone needs to be on the same page, talking the same language, moving in the same direction.
Another common theme was the need to link to corporate/business objectives and goals. The subtitle of the Symposium is ‘How to know if your learning is meeting business needs.’ Well, the message was clear that we should be linking to business needs, and we should be mapping/measuring the impact of training to corporate goals – this was the WHAT. What I need to do is link to corporate goals and objectives, and measure. But where’s the HOW? How do I show learning’s distinct and specific contribution to corporate outcomes? Am I talking just about formal learning? How about informal learning’s impact on performance and, ultimately corporate goals?
Ahh – it was a good Symposium. Thank you for letting me share my thoughts, my learnings, my questions. And a respectful expression of gratitude to all the presenters who shared experiences and wisdom with me. Now, where do I get a ticket to Le Cirque du Soleil?
Submitted by: Angela van Barneveld
Session Review: Evaluation of Learning
(May 4) Hafsa El Khettab and François Ste-Marie or ARINSO Canada facilitated a workshop of learning evaluation. Unfortunately trainers do not always put much energy into developing evaluation tools for the training programs they develop. But evaluating the results of training is crucial if we are to attain the objectives we have been set. The presenters led an animated discussion with delegates sharing their past experiences as well as some of the challenges they have been given to develop evaluation tools.
The highlight of the workshop was the presentation of numerous tools developed by the workshop facilitators. Participants were able to try some of these out through small group exercises. Many of these tools grew from collaborations with instructional designers and other training and development professionals. Essentially, evaluation tools should evolve from training goals and objectives. This is the only way to ensure that there is a link between training and organizational needs.
Submitted by: Ofelia Ribeiro
The highlight of the workshop was the presentation of numerous tools developed by the workshop facilitators. Participants were able to try some of these out through small group exercises. Many of these tools grew from collaborations with instructional designers and other training and development professionals. Essentially, evaluation tools should evolve from training goals and objectives. This is the only way to ensure that there is a link between training and organizational needs.
Submitted by: Ofelia Ribeiro
Friday Keynote: Connecting the Dots on Training ROI: Skills Development to Business Outcomes…
(May 4)
Summary: In this session, Allan presented some attention-getting statistics on Canada’s failing productivity, and showed the link between economy, productivity, and skills. Canadian business invests less money per employee on training when compared to American and European countries. Productivity has declined in small and medium business enterprises over the last 25 years. The major culprit is basic skills, literacy – both requirements to participate in the knowledge society (so impact is economic and social). The statistics presented to evidence that training adds value were mostly from small and medium business data. Understandably, the larger ROI numbers were documented for situations where low-skilled workers are provided with training. Corporate metrics such as shareholders returns were noted to be higher in organizations that invested in training. However, with al this data, Allan still acknowledged that ROI was still a nebulous concept for most, and understanding and interpreting the ROI numbers that come out of a study are still a challenge for most of us.
Ang’s thoughts: A fascinating presentation – moreso again from the point of view from all the questions it stimulated. Yesterday, it was mentioned that the evaluation models put forth by Kirkpatrick and Phillips were applicable to formal training situations which, as we also now know, are about 20-30% of the learning that goes on in an organization. This is the data that is used to calculate ROI (return on investment). I had a chance to speak with Allan after his presentation and asked him for his perspective on a few things. For instance, if up to 80% of learning in an organization is informal, what impact does that have on ROI? If ROI is calculated based only on formal learning, where is the consideration given for the impact or modifying effect of informal learning on ROI? I cannot see how you can control for informal learning effects so that ROI is cleanly and solely the result of formal learning, but Allan is much more the expert in this area and indicated it was possible. However, he acknowledged that he, too, had an interest in the relationship between ROI and informal learning and would do more work in that area in the future. FANTASTIC!
Submitted by: Angela van Barneveld
Summary: In this session, Allan presented some attention-getting statistics on Canada’s failing productivity, and showed the link between economy, productivity, and skills. Canadian business invests less money per employee on training when compared to American and European countries. Productivity has declined in small and medium business enterprises over the last 25 years. The major culprit is basic skills, literacy – both requirements to participate in the knowledge society (so impact is economic and social). The statistics presented to evidence that training adds value were mostly from small and medium business data. Understandably, the larger ROI numbers were documented for situations where low-skilled workers are provided with training. Corporate metrics such as shareholders returns were noted to be higher in organizations that invested in training. However, with al this data, Allan still acknowledged that ROI was still a nebulous concept for most, and understanding and interpreting the ROI numbers that come out of a study are still a challenge for most of us.
Ang’s thoughts: A fascinating presentation – moreso again from the point of view from all the questions it stimulated. Yesterday, it was mentioned that the evaluation models put forth by Kirkpatrick and Phillips were applicable to formal training situations which, as we also now know, are about 20-30% of the learning that goes on in an organization. This is the data that is used to calculate ROI (return on investment). I had a chance to speak with Allan after his presentation and asked him for his perspective on a few things. For instance, if up to 80% of learning in an organization is informal, what impact does that have on ROI? If ROI is calculated based only on formal learning, where is the consideration given for the impact or modifying effect of informal learning on ROI? I cannot see how you can control for informal learning effects so that ROI is cleanly and solely the result of formal learning, but Allan is much more the expert in this area and indicated it was possible. However, he acknowledged that he, too, had an interest in the relationship between ROI and informal learning and would do more work in that area in the future. FANTASTIC!
Submitted by: Angela van Barneveld
Supper and Networking at Mount Stephen Club
(May 3) The magnificent home of wealthy businessman Mount Stephen now transformed into a club was the backdrop to a fun evening of networking. Plentiful wine encouraged conversation and exchanges of phone numbers and email addresses. I have many pictures with old friends and new acquaintances in sumptuous surroundings. We were allowed to visit all the rooms in the club and many of us imagine what it must have been like to live in this home at the turn of the century. The supper was excellent and people mingled throughout until very very late. This was possibly the best ending to a stimulating day.
Submitted by: Ofelia Ribeiro
Submitted by: Ofelia Ribeiro
Session Review: Bridging the Gap Between Educational and Business Evaluation
(May 3) Claude Martel of Educonsillium proposes that evaluation methods must consider not just what the individual learned but how this impacts on the organization. It is a given that all organizations must engage in training evaluation. However, the reasons why trainers engage in evaluation do not always link with what organizational leaders are looking for. One reason is that we doe not use the same language – or vocabulary – to examine the impact of training. While the training department looks at learner satisfaction or learning, administrators are more commonly concerned with how the training supports strategic processes and impacts on critical competences.
Martel led a lively discussion and facilitated an examination of how trainers can better understand managers’ needs and how they see training efforts. Much of the disparity is related to our not seeing things in the same way. As trainers we often have to prove the value of what we do, how it links with organizational goals and why it should be supported. As training departments are challenged to align their activities with strategic needs, indicators of success should be remodeled to take into account our new role. This presentation encouraged fruitful reflection on how we all can do our jobs better. I know there are a number of ideas that I will carry with me as I build my career.
Submitted by: Ofelia Ribeiro
Martel led a lively discussion and facilitated an examination of how trainers can better understand managers’ needs and how they see training efforts. Much of the disparity is related to our not seeing things in the same way. As trainers we often have to prove the value of what we do, how it links with organizational goals and why it should be supported. As training departments are challenged to align their activities with strategic needs, indicators of success should be remodeled to take into account our new role. This presentation encouraged fruitful reflection on how we all can do our jobs better. I know there are a number of ideas that I will carry with me as I build my career.
Submitted by: Ofelia Ribeiro
Session Review: Assessment of knowledge: Test Items that Measure Learning Effectively
(May 3)
Summary: In this session, Atena presented the process and procedure that the TD Bank Financial Group uses to develop their test items. The strategy was to develop performance-based items that measure on-the-job transferable knowledge and skills. Through the analysis process, they determine if testing is required and, if yes, ensure that the test items map to the learning objectives, which map to the performance objectives, which map to the business objectives. Performance-based items focus more on testing conceptual, process, procedure, and principle-based content which usually requires an element of judgment or decision-making. This is in contrast to test items that focus on facts and access recall rather than higher-level cognitive skills. Also covered are item writing strategies, establishing validity and reliability, and testing the test.
Ang’s thoughts: A new approach that I came away with is the value of asking some L3 questions on the L1 evaluation to set a baseline. This session was more a confirmation of my existing knowledge of test item development and processes – always a nice thing. One thing that Atena reinforced was that the writing of test items is not an easy task. Perhaps the development of the stem is the easiest piece, but coming up with feasible distractors is a real tough one. It was mentioned in an audience discussion that the expected number of good test items that someone would write on a day is somewhere between 8 and 10. This may surprise most people, but perhaps not those who have actually been involved in creating good test items.
Submitted by: Angela van Barneveld
Summary: In this session, Atena presented the process and procedure that the TD Bank Financial Group uses to develop their test items. The strategy was to develop performance-based items that measure on-the-job transferable knowledge and skills. Through the analysis process, they determine if testing is required and, if yes, ensure that the test items map to the learning objectives, which map to the performance objectives, which map to the business objectives. Performance-based items focus more on testing conceptual, process, procedure, and principle-based content which usually requires an element of judgment or decision-making. This is in contrast to test items that focus on facts and access recall rather than higher-level cognitive skills. Also covered are item writing strategies, establishing validity and reliability, and testing the test.
Ang’s thoughts: A new approach that I came away with is the value of asking some L3 questions on the L1 evaluation to set a baseline. This session was more a confirmation of my existing knowledge of test item development and processes – always a nice thing. One thing that Atena reinforced was that the writing of test items is not an easy task. Perhaps the development of the stem is the easiest piece, but coming up with feasible distractors is a real tough one. It was mentioned in an audience discussion that the expected number of good test items that someone would write on a day is somewhere between 8 and 10. This may surprise most people, but perhaps not those who have actually been involved in creating good test items.
Submitted by: Angela van Barneveld
Session Review (Angela's): Evaluating Informal Learning
(May 3)
Summary: This was an energetic session that discussed informal learning and evaluation of said. They contrasted informal learning with formal, non-formal, and self-directed learning, stressed the importance of evaluating informal learning (which apparently is up to 80% of the learning that goes on in the workplace), and fundamentals for evaluating informal learning. They also looked at how other domains are evaluating informal learning (museums, marketing, web design).
The two presenters, Saul and Adnan, offered an interesting dichotomy of perspectives, one able to speak from only the academic research perspective, while the other could take that and merge it into practical experience and situations.
Ang’s thoughts: I must admit that this session left me more confused at the end than I was at the beginning – but I’ll take comfort in the fact that cognitive dissonance is one of those things that helps to move me forward.
The presenters defined informal learning as having the locus of control within the individual, with objectives and evaluation set by the individual, and having exploration and experiences as the means of learning. I cannot seem to get my head around evaluating this. I may not even know it’s going on! And by its very definition, doesn’t that seem true – does one advertise their informal learning? I absolutely agree that informal learning is important in the workplace – how can it not be when it covers off the majority of the learning that takes place.
From the audience, I sensed a real desire to formalize informal learning, to capture it somehow. I think the desire was more to facilitate the informal learning. That is, if I knew what people chose to learn, how they wanted to learn it, etc., I could create resources for self-service that could facilitate their learning – makes sense.
The presenters tried to debunk the myth that if you can’t measure it, you can’t manage it. I remain unconvinced. Roger Kauffman has the same perspective, that he can measure anything – still not happening for me.
Other statements that confused me – “realities about measuring informal learning is that it encourages people to do more learning.” What? Isn’t informal learning people doing the learning they need? What does more mean? Okay – I’m hanging my ignorance out there and perhaps I am not the only one with these sorts of questions in my head.
One example of a domain of informal learning evaluation came from marketing communications and the impact of ads and press releases – direct marketing measures and brand recognition studies. Could you argue that the bombardment of advertising/marketing propaganda that sticks in my head is a form of informal learning? I suppose. Did I seek this out this information? Did I set the objectives and evaluation? Am I not understanding informal learning?
How about the web designer example which seems to me to be more of a usability assessment than learning. Are we equating learn-ability of an interface, or site navigability with informal learning? I don’t know. Can one extrapolate learning from usage statistics? OY! Well, the easier an interface is to learn and use, the more likely it will be used. But what about what sits behind the interface – content? Reference materials?
Perhaps having a structure for informal learning (which seems counterintuitive to me), like development plans, coaches and mentors moves us toward facilitating informal learning.
Seriously, if anyone has any comments on this – either to clarify things for me or to commiserate, I am all ears!
Submitted by: Angela van Barneveld
Summary: This was an energetic session that discussed informal learning and evaluation of said. They contrasted informal learning with formal, non-formal, and self-directed learning, stressed the importance of evaluating informal learning (which apparently is up to 80% of the learning that goes on in the workplace), and fundamentals for evaluating informal learning. They also looked at how other domains are evaluating informal learning (museums, marketing, web design).
The two presenters, Saul and Adnan, offered an interesting dichotomy of perspectives, one able to speak from only the academic research perspective, while the other could take that and merge it into practical experience and situations.
Ang’s thoughts: I must admit that this session left me more confused at the end than I was at the beginning – but I’ll take comfort in the fact that cognitive dissonance is one of those things that helps to move me forward.
The presenters defined informal learning as having the locus of control within the individual, with objectives and evaluation set by the individual, and having exploration and experiences as the means of learning. I cannot seem to get my head around evaluating this. I may not even know it’s going on! And by its very definition, doesn’t that seem true – does one advertise their informal learning? I absolutely agree that informal learning is important in the workplace – how can it not be when it covers off the majority of the learning that takes place.
From the audience, I sensed a real desire to formalize informal learning, to capture it somehow. I think the desire was more to facilitate the informal learning. That is, if I knew what people chose to learn, how they wanted to learn it, etc., I could create resources for self-service that could facilitate their learning – makes sense.
The presenters tried to debunk the myth that if you can’t measure it, you can’t manage it. I remain unconvinced. Roger Kauffman has the same perspective, that he can measure anything – still not happening for me.
Other statements that confused me – “realities about measuring informal learning is that it encourages people to do more learning.” What? Isn’t informal learning people doing the learning they need? What does more mean? Okay – I’m hanging my ignorance out there and perhaps I am not the only one with these sorts of questions in my head.
One example of a domain of informal learning evaluation came from marketing communications and the impact of ads and press releases – direct marketing measures and brand recognition studies. Could you argue that the bombardment of advertising/marketing propaganda that sticks in my head is a form of informal learning? I suppose. Did I seek this out this information? Did I set the objectives and evaluation? Am I not understanding informal learning?
How about the web designer example which seems to me to be more of a usability assessment than learning. Are we equating learn-ability of an interface, or site navigability with informal learning? I don’t know. Can one extrapolate learning from usage statistics? OY! Well, the easier an interface is to learn and use, the more likely it will be used. But what about what sits behind the interface – content? Reference materials?
Perhaps having a structure for informal learning (which seems counterintuitive to me), like development plans, coaches and mentors moves us toward facilitating informal learning.
Seriously, if anyone has any comments on this – either to clarify things for me or to commiserate, I am all ears!
Submitted by: Angela van Barneveld
Session Review (Ofelia's): Evaluating Informal Learning
(May 3) Evaluating any form of learning is always a challenge but particularly so for informal learning, or learning initiated by an individual to particular ends. The dynamic duo of Saul Carliner and Adnan Qayyum put on a lively presentation and facilitated a discussion on the nature of informal learning, what it is, when and how to evaluate it. This was the only presentation that I attended where the delegates were encouraged to participate in a group activity that examined a case study. The most important thing learned as a result of this presentation is that it is possible to evaluate informal learning. However, it is crucial that there be some consistency between the approach used and the reasons behind conducting an evaluation. This is not easy when we consider the nature of informal learning. Most of the delegates opted for methods that formalize informal learning. But is this really possible?
Carliner and Qayyum presented various aspects of informal learning and why it is important for the workplace. It is estimated that close to 70% of all workplace learning is conducted in an informal manner. Furthermore, it is often the source of knowledge that is immediately utilized by most workers. Measuring any form of learning is important but evaluating learning proves a challenge because it is difficult to capture. Is the traditional Kirkpatrick model useful here, given that it is the learner who decides on the objectives and goals of learning? Whatever we decide to track and assess should depend on the nature of the learning and the agreed-upon goals. Sometimes the learning that is done needs to be adapted into a form that facilitates sharing with others – via electronic or other means – in order to better enable some form of evaluation. This suggests that all levels of leadership promote and find creative ways to evaluate informal learning.
Submitted by: Ofelia Ribeiro
Carliner and Qayyum presented various aspects of informal learning and why it is important for the workplace. It is estimated that close to 70% of all workplace learning is conducted in an informal manner. Furthermore, it is often the source of knowledge that is immediately utilized by most workers. Measuring any form of learning is important but evaluating learning proves a challenge because it is difficult to capture. Is the traditional Kirkpatrick model useful here, given that it is the learner who decides on the objectives and goals of learning? Whatever we decide to track and assess should depend on the nature of the learning and the agreed-upon goals. Sometimes the learning that is done needs to be adapted into a form that facilitates sharing with others – via electronic or other means – in order to better enable some form of evaluation. This suggests that all levels of leadership promote and find creative ways to evaluate informal learning.
Submitted by: Ofelia Ribeiro
Session Review: Creating a Learning and Developmental Strategy Map – Dofasco Case Study
(May 3)
Summary: In this session, Tracy and Theresa presented their first attempt at a Strategy Map in order to align the Dofasco Learning and Development organization with the corporate goals and objectives. This map serves as a communication tool to inform employees about their place and contribution to the larger corporate picture. It also serves as a feedback mechanism for process improvement. The key objective was to demonstrate the value of the Learning and Development organization, whose customers are an internal audience. They suggest a Balanced Scorecard in order to track metrics, and recommend that we measure what matters. They indicated that the 4 themes that require alignment and measurement in a Balanced Scorecard include: guiding principles, customer perspective, internal process perspective, and learning and growth perspective
The presenters mentioned that Dofasco, as company of about 6,800 employees, was now in a merger situation and would be absorbed into an organization of over 300,000 employees. The strategy map was developed for Dofasco, and the presenters didn’t know how it would play out in the new organization. Because of the current state of flux, the Learning and Development Strategy Map was not linked to corporate objectives or goals, as none were available. So, the current focus is on defining and measuring the value proposition provided by the learning department.
Ang’s thoughts:
I had the chance to chat with Theresa at the end of the session and she was very kind in addressing my questions. Kudos to Theresa and Tracy for proactively developing the map, as it will be easier to have something to react to once the merger is finalized and skills, tools, processes, etc. need to be consolidated – better than starting from scratch in the new organization. We discussed the challenge for Learning/HR departments to align with corporate objectives and establish the measurements of impacts – well document in the industry research and literature. They mentioned that a lack of data impedes credit being given to the learning/HR departments for their contribution to corporate successes and, again, the literature indicates that there is uncertainty as to what to measure or how to measure it to show value.
From this, I am able to take away their process for developing their strategy map and scorecard. Communication is key, as is alignment not only with corporate goals and objectives, but also with various functional groups to ensure that they are all pulling in the same direction.
One thing I need to mention, as I noticed it also in the Cirque du Soleil keynote and discussed it with Theresa. Learning departments seem to be intentionally moving away from the word ‘Training’ as an identifier of what they do – and have adopted the departmental names of ‘Learning’ and ‘Development’. This may seem inconsequential to some, but it really strikes a cord with me, as I think of focus is placed in very difference places when one refers to training and one refers to learning.
Submitted by: Angela van Barneveld
Summary: In this session, Tracy and Theresa presented their first attempt at a Strategy Map in order to align the Dofasco Learning and Development organization with the corporate goals and objectives. This map serves as a communication tool to inform employees about their place and contribution to the larger corporate picture. It also serves as a feedback mechanism for process improvement. The key objective was to demonstrate the value of the Learning and Development organization, whose customers are an internal audience. They suggest a Balanced Scorecard in order to track metrics, and recommend that we measure what matters. They indicated that the 4 themes that require alignment and measurement in a Balanced Scorecard include: guiding principles, customer perspective, internal process perspective, and learning and growth perspective
The presenters mentioned that Dofasco, as company of about 6,800 employees, was now in a merger situation and would be absorbed into an organization of over 300,000 employees. The strategy map was developed for Dofasco, and the presenters didn’t know how it would play out in the new organization. Because of the current state of flux, the Learning and Development Strategy Map was not linked to corporate objectives or goals, as none were available. So, the current focus is on defining and measuring the value proposition provided by the learning department.
Ang’s thoughts:
I had the chance to chat with Theresa at the end of the session and she was very kind in addressing my questions. Kudos to Theresa and Tracy for proactively developing the map, as it will be easier to have something to react to once the merger is finalized and skills, tools, processes, etc. need to be consolidated – better than starting from scratch in the new organization. We discussed the challenge for Learning/HR departments to align with corporate objectives and establish the measurements of impacts – well document in the industry research and literature. They mentioned that a lack of data impedes credit being given to the learning/HR departments for their contribution to corporate successes and, again, the literature indicates that there is uncertainty as to what to measure or how to measure it to show value.
From this, I am able to take away their process for developing their strategy map and scorecard. Communication is key, as is alignment not only with corporate goals and objectives, but also with various functional groups to ensure that they are all pulling in the same direction.
One thing I need to mention, as I noticed it also in the Cirque du Soleil keynote and discussed it with Theresa. Learning departments seem to be intentionally moving away from the word ‘Training’ as an identifier of what they do – and have adopted the departmental names of ‘Learning’ and ‘Development’. This may seem inconsequential to some, but it really strikes a cord with me, as I think of focus is placed in very difference places when one refers to training and one refers to learning.
Submitted by: Angela van Barneveld
Session Review: Using Teams To Assess Learning
(May 3) Micro-teaching is one way that is becoming increasingly used to develop new instructors. Whether the teaching is done face-to-face or online, many of the problems are similar. There is never enough time to do everything one needs to do. Instructors are rarely able to take on the new challenge of teaching others, yet they must be assessed effectively. Most were selected because they were subject experts and not because they had previous teaching experience. Nevertheless, it is important that these individuals be familiar with the teaching process before they begin their new tasks. This presentation sets out to examine some of the challenges related to micro-teaching and provides tools to develop evaluation methods centered on principles of improvement.
Jean-Marc Guillemette, of LearnPlex, presented the main issues related to micro-teaching: motivation to encourage new instructors, methods that optimize group teaching efforts, and tools such as evaluation grids that assist assessment of new instructors. The presentation focuses on team learning as a process that both engages and informs new instructors. When, in the process, is the optimal time for evaluation? Not at the end of instruction but when training is near completion and after instructors have had a chance to apply what they have learned. In order to effectively evaluate performance, team members should be given the opportunity to assess one another. Self-assessment is another important method. The issues that should be addressed in evaluation are: how was the micro-teaching episode, what were the positive points, and what could be improved.
Guillemette proved to be an excellent speaker who shared many of his experiences with group teaching methods. A discussion revolving around some of the problems surrounding the approach led to fruitful exchange between delegates and added to the session. In recent years I have also been involved in micro-teaching efforts and the approach used here helped to focus on some of the issues that regularly arise. It seems evident that evaluation is not the endpoint. Regardless of the context, individuals involved in micro-teaching must constantly assess their abilities and share their experiences with other cohorts in an ongoing assessment of their development as instructors. This is what makes this approach popular and effective.
Submitted by: Ofelia Ribeiro
Jean-Marc Guillemette, of LearnPlex, presented the main issues related to micro-teaching: motivation to encourage new instructors, methods that optimize group teaching efforts, and tools such as evaluation grids that assist assessment of new instructors. The presentation focuses on team learning as a process that both engages and informs new instructors. When, in the process, is the optimal time for evaluation? Not at the end of instruction but when training is near completion and after instructors have had a chance to apply what they have learned. In order to effectively evaluate performance, team members should be given the opportunity to assess one another. Self-assessment is another important method. The issues that should be addressed in evaluation are: how was the micro-teaching episode, what were the positive points, and what could be improved.
Guillemette proved to be an excellent speaker who shared many of his experiences with group teaching methods. A discussion revolving around some of the problems surrounding the approach led to fruitful exchange between delegates and added to the session. In recent years I have also been involved in micro-teaching efforts and the approach used here helped to focus on some of the issues that regularly arise. It seems evident that evaluation is not the endpoint. Regardless of the context, individuals involved in micro-teaching must constantly assess their abilities and share their experiences with other cohorts in an ongoing assessment of their development as instructors. This is what makes this approach popular and effective.
Submitted by: Ofelia Ribeiro
Ofelia's Comments on the Opening Keynote
(May 3) France Dufresne and Sylvie Geneau of Cirque du Soleil gave a fascinating presentation on their experience of building a leadership culture in a creative environment. The challenges they addressed are distinct from most of those that we are familiar with. The implementation of a coherent administrative structure was difficult in an organization that was rapidly growing and unused to working in a centralized model. Nevertheless, the speakers were able to balance the creative nature of the Cirque while at the same time maintain some control over the organizational challenges they experienced. In this way, the Cirque is no different from any other organization.
Although the original mandate was to develop leadership training, it was evident that this approach would not be successful. A different approach – one that aligned the management process with the creative nature of the Cirque’s talent – was required. Visioning through storytelling and discussion enabled leadership to be expressed through a common language. Coaches, who engaged in appreciative inquiry methods, facilitated the process. Managers had to learn to be leaders by simultaneously maintaining control over the organization and encouraging creativity of the talent. At the root of the success of this project was their ability to celebrate their stories and successes and to share them.
Throughout the presentation, images and music of Cirque shows enthralled the CSTD delegates. Those of us who had attended Cirque shows sat back and allowed our memories of these events to take over. The incredible abilities of the entertaining performers combined with the exquisite music combined together to project a moving experience. The speakers alternated and their dynamism was apparent. Their ease with one another and with alternating between English and French proved stimulating and extremely instructive. It may have been a challenge for some of the delegates to understand how one can alternate between seemingly disparate mind-sets but it is part of the Quebecois culture – especially in Montreal – to alternate between the two languages. This presentation technique added to the experience.
It is unfortunate that the discussion following the presentation could not have been extended. The leadership model presented can be adapted to other organizations interested in developing their managers as leaders. Essentially, all leaders must be able to engage and support the teams in their area and be able to understand and communicate with staff as expectations change. This becomes increasingly important as contexts change, as organizations grow and go global. The image that stays with me is that the process of leadership is a journey: We are constantly moving, learning and changing.
Submitted by: Ofelia Ribeiro
Although the original mandate was to develop leadership training, it was evident that this approach would not be successful. A different approach – one that aligned the management process with the creative nature of the Cirque’s talent – was required. Visioning through storytelling and discussion enabled leadership to be expressed through a common language. Coaches, who engaged in appreciative inquiry methods, facilitated the process. Managers had to learn to be leaders by simultaneously maintaining control over the organization and encouraging creativity of the talent. At the root of the success of this project was their ability to celebrate their stories and successes and to share them.
Throughout the presentation, images and music of Cirque shows enthralled the CSTD delegates. Those of us who had attended Cirque shows sat back and allowed our memories of these events to take over. The incredible abilities of the entertaining performers combined with the exquisite music combined together to project a moving experience. The speakers alternated and their dynamism was apparent. Their ease with one another and with alternating between English and French proved stimulating and extremely instructive. It may have been a challenge for some of the delegates to understand how one can alternate between seemingly disparate mind-sets but it is part of the Quebecois culture – especially in Montreal – to alternate between the two languages. This presentation technique added to the experience.
It is unfortunate that the discussion following the presentation could not have been extended. The leadership model presented can be adapted to other organizations interested in developing their managers as leaders. Essentially, all leaders must be able to engage and support the teams in their area and be able to understand and communicate with staff as expectations change. This becomes increasingly important as contexts change, as organizations grow and go global. The image that stays with me is that the process of leadership is a journey: We are constantly moving, learning and changing.
Submitted by: Ofelia Ribeiro
Angela's Thoughts about the Opening Keynote
(May 3)
Session Title: Measurement at Cirque du Soleil – Leadership from engagement to deployment to outcomes
Summary: In this session, France and Sylvie presented the challenges, creative approaches, and promising future of the leadership program implemented at Le Cirque du Soleil. Imagine managing an ever-expanding global decentralized organization with a highly mobile workforce – young, creative, outside-the-box people - where 40 nationalities and over 25 languages are present at any one time. An additional challenge with the implementation of a leadership program into this very creative setting was the resistance to anything that even remotely resembled corporate structure. It was perceived as taboo in this creative culture.
The need was to blend art and business. After comparing themselves to high-performing organizations, Le Cirque fell short on characteristics related to leadership and decision-making. Operating expenses were increasing at a higher rate than revenue. Turn-over was high, and accountability was low.
Three major components were proposed for leadership development – establishing the vision, management process alignment, and behavioural alignment. The preservation of the Le Cirque culture was the primary concern and any threat to that would be a huge barrier to program acceptance. So, after identifying and acknowledging the strengths of the organization through appreciative inquiry rather than focusing on what need to change, a blended learning program was established to develop and sustain leadership development. Where they thought that engagement of resources would take 6 months, they found that it actually took 2 years. Measurement of the program was mostly to capture satisfaction and was performed on a regular and frequent basis through interviews, observations, and feedback from all levels of the organization. Outcomes included a reduced perception of the taboo of management and leadership, more manager focus on talent development and promotion, common management language, and an understanding of expectations. Their continual mantra throughout the session was “Evoke, Invoke, and Provoke.”
Ang’s thoughts: Here are the two big things I came away with – (1) that culture and group identity are forces to contend with and cannot be ignored in times of organizational change, and (2) in some cases (probably more cases), it’s better to bring the mountain to Mohammed.
Culture: Okay – let me just say it out loud – Le Cirque is COOL! I think the real genius of the management/leadership program developers was to fully acknowledge the desire for uniqueness and the unrestrictive creativity that was of prime importance to the Le Cirque employees and managers. Employees did not want to work from a business/management model that represented someone else – they wanted a model that reflected them uniquely. For the launch, program coordinators moved away from working in a typical meeting room setting, and instead staged the environment to reflect the familiar, the uniqueness, and the creativity. The president kicked off the session and there were no desks or tables or chairs, no documents and no PowerPoint. Instead, the setting was theatrical, with projections on the walls of key words, scented air, pillows on the floor. All this was used to evoke the senses. What followed was much story-telling and writing and sharing of leadership stories. BRILLIANT! Walls started to come down and the group moved beyond their initial taboos to focus on content and discussions.
The mountain: I think the other stroke of genius was not to overlay a business model on the organization, but to evolve a model that suited the needs of managers and the organization. Le Cirque did not bring their leaders/managers into an established leadership program (aka the mountain) – of which there are many, but brought the program to the leaders/managers and where they were at. Yes, it took time – it took four times longer than anticipated, but it seems to me that Le Cirque needed this time, as the culture/identity need of the organization was very, very strong. Le Cirque is starting to see the benefits with tremendous growth in show production, expansion to Asia, and a more solid management of operations during all this expansion.
Submitted by: Angela van Barneveld
Session Title: Measurement at Cirque du Soleil – Leadership from engagement to deployment to outcomes
Summary: In this session, France and Sylvie presented the challenges, creative approaches, and promising future of the leadership program implemented at Le Cirque du Soleil. Imagine managing an ever-expanding global decentralized organization with a highly mobile workforce – young, creative, outside-the-box people - where 40 nationalities and over 25 languages are present at any one time. An additional challenge with the implementation of a leadership program into this very creative setting was the resistance to anything that even remotely resembled corporate structure. It was perceived as taboo in this creative culture.
The need was to blend art and business. After comparing themselves to high-performing organizations, Le Cirque fell short on characteristics related to leadership and decision-making. Operating expenses were increasing at a higher rate than revenue. Turn-over was high, and accountability was low.
Three major components were proposed for leadership development – establishing the vision, management process alignment, and behavioural alignment. The preservation of the Le Cirque culture was the primary concern and any threat to that would be a huge barrier to program acceptance. So, after identifying and acknowledging the strengths of the organization through appreciative inquiry rather than focusing on what need to change, a blended learning program was established to develop and sustain leadership development. Where they thought that engagement of resources would take 6 months, they found that it actually took 2 years. Measurement of the program was mostly to capture satisfaction and was performed on a regular and frequent basis through interviews, observations, and feedback from all levels of the organization. Outcomes included a reduced perception of the taboo of management and leadership, more manager focus on talent development and promotion, common management language, and an understanding of expectations. Their continual mantra throughout the session was “Evoke, Invoke, and Provoke.”
Ang’s thoughts: Here are the two big things I came away with – (1) that culture and group identity are forces to contend with and cannot be ignored in times of organizational change, and (2) in some cases (probably more cases), it’s better to bring the mountain to Mohammed.
Culture: Okay – let me just say it out loud – Le Cirque is COOL! I think the real genius of the management/leadership program developers was to fully acknowledge the desire for uniqueness and the unrestrictive creativity that was of prime importance to the Le Cirque employees and managers. Employees did not want to work from a business/management model that represented someone else – they wanted a model that reflected them uniquely. For the launch, program coordinators moved away from working in a typical meeting room setting, and instead staged the environment to reflect the familiar, the uniqueness, and the creativity. The president kicked off the session and there were no desks or tables or chairs, no documents and no PowerPoint. Instead, the setting was theatrical, with projections on the walls of key words, scented air, pillows on the floor. All this was used to evoke the senses. What followed was much story-telling and writing and sharing of leadership stories. BRILLIANT! Walls started to come down and the group moved beyond their initial taboos to focus on content and discussions.
The mountain: I think the other stroke of genius was not to overlay a business model on the organization, but to evolve a model that suited the needs of managers and the organization. Le Cirque did not bring their leaders/managers into an established leadership program (aka the mountain) – of which there are many, but brought the program to the leaders/managers and where they were at. Yes, it took time – it took four times longer than anticipated, but it seems to me that Le Cirque needed this time, as the culture/identity need of the organization was very, very strong. Le Cirque is starting to see the benefits with tremendous growth in show production, expansion to Asia, and a more solid management of operations during all this expansion.
Submitted by: Angela van Barneveld
Angela's First Thoughts
(May 3) The first order of business is to express my gratitude to Bell Canada, who sponsored me as a student to attend the Symposium. Further, I get to share my thoughts and observations through this blog, which I find to be an exciting opportunity. Any response or comments to my blog would be fantastic!
I’ll blog about the sessions I attending, summarizing the intent of the session and then sharing my thoughts.
Submitted by: Angela van Barneveld
I’ll blog about the sessions I attending, summarizing the intent of the session and then sharing my thoughts.
Submitted by: Angela van Barneveld
Ofelia's First Thoughts
(May 3) It is a beautiful spring day in Montreal and a short walk from the Metro to the symposium reminds me that it was not so long ago that the streets were covered in snow. Even though a few traces can still be seen here and there, especially on Mount Royal, it is a good thing it has virtually melted and the tree buds look like they will explode any minute now. All in all, a great week for out-of-town visitors set to explore this beautiful city!
I am on my way to the CSTD symposium located at McGill University in the heart of downtown. I have been anticipating this symposium for several weeks. As a trainer, teacher, designer of instruction and lifelong learner, I am eager to network with other professionals working in assessment and evaluation efforts and to learn from their experiences.
As I walk to McGill, I quickly look at the schedule that I have been anticipating for several weeks. I am in need of a coffee and look for the nearest outlet. This is Montreal, after all. We can get real coffee here! After that first sip, I look around for some familiar faces and am surprised by the number of people already present. It seems that close to 200 people from across Canada have registered for the symposium. I see some old friends and arrange for a chat at the coffee break and lunch. I encourage a friend to go to the supper at the Mount Stephen Club because it is a beautiful building and worth the trip.
Submitted by: Ofelia Ribeiro
I am on my way to the CSTD symposium located at McGill University in the heart of downtown. I have been anticipating this symposium for several weeks. As a trainer, teacher, designer of instruction and lifelong learner, I am eager to network with other professionals working in assessment and evaluation efforts and to learn from their experiences.
As I walk to McGill, I quickly look at the schedule that I have been anticipating for several weeks. I am in need of a coffee and look for the nearest outlet. This is Montreal, after all. We can get real coffee here! After that first sip, I look around for some familiar faces and am surprised by the number of people already present. It seems that close to 200 people from across Canada have registered for the symposium. I see some old friends and arrange for a chat at the coffee break and lunch. I encourage a friend to go to the supper at the Mount Stephen Club because it is a beautiful building and worth the trip.
Submitted by: Ofelia Ribeiro
About the Bloggers
In addition to Saul Carliner, the bloggers of this event include two other representatives of Concordia University.
Ofelia Ribeiro is a PhD student in Educational Technology at Concordia University and has extensive experience training professionals in the workplace, teaching college students and developing professional activities. Her interests lie in professional development and in developing training programs for professionals. She has presented at the 2006 ASTD conference (evaluation of a training program for trainers) and will present at the Association for Media and Technology in Education in Canada (AMTEC)/Canadian Association for Distance Education (CADE) conference later this month (professional development and teacher technology use).
Angela van Barneveld is a PhD student in Educational Technology at Concordia University and also has extensive experience in workplace learning and performance. She is currently Program Manager for the Cognos Academy in Ottawa, Ontario. She has also worked in the training groups at Nortel and the Canada Revenue Agency. Her interests include e-learning, mobile learning, and the "business" of learning. She has presented at numerous conferences, including those of the Association for the Advancement of Computers in Education (AACE) and the e-Learning Guild.
Submitted by:
Saul Carliner
Ofelia Ribeiro is a PhD student in Educational Technology at Concordia University and has extensive experience training professionals in the workplace, teaching college students and developing professional activities. Her interests lie in professional development and in developing training programs for professionals. She has presented at the 2006 ASTD conference (evaluation of a training program for trainers) and will present at the Association for Media and Technology in Education in Canada (AMTEC)/Canadian Association for Distance Education (CADE) conference later this month (professional development and teacher technology use).
Angela van Barneveld is a PhD student in Educational Technology at Concordia University and also has extensive experience in workplace learning and performance. She is currently Program Manager for the Cognos Academy in Ottawa, Ontario. She has also worked in the training groups at Nortel and the Canada Revenue Agency. Her interests include e-learning, mobile learning, and the "business" of learning. She has presented at numerous conferences, including those of the Association for the Advancement of Computers in Education (AACE) and the e-Learning Guild.
Submitted by:
Saul Carliner
About this Blog
Greetings, fellow CSTD members,
This is a blog which summarizes highlights of, and impressions about, the 2007 CSTD National Symposium in Montreal, held May 3 and 4, 2007 in Montreal. The theme of the Symposium was Assessment, Measurement and Evaluation....How to Know if Your Learning is Meeting Business Needs.
For those who attended, we hope that this blog provides a helpful resource for recalling some of the key moments of this terrific event.
For those who were not able to attend, we hope that this blog not only summarizes what was covered, but also gives you a first-hand impression of the experience of attending.
Saul Carliner
Ofelia Ribeiro
Angela van Barneveld,
May 8, 2007
This is a blog which summarizes highlights of, and impressions about, the 2007 CSTD National Symposium in Montreal, held May 3 and 4, 2007 in Montreal. The theme of the Symposium was Assessment, Measurement and Evaluation....How to Know if Your Learning is Meeting Business Needs.
For those who attended, we hope that this blog provides a helpful resource for recalling some of the key moments of this terrific event.
For those who were not able to attend, we hope that this blog not only summarizes what was covered, but also gives you a first-hand impression of the experience of attending.
Saul Carliner
Ofelia Ribeiro
Angela van Barneveld,
May 8, 2007
Subscribe to:
Posts (Atom)
