To support fair and consistent evaluations, CDE is developing tools to promote common interpretations of teacher quality and help evaluators provide useful and actionable feedback to educators. One such tool is Elevate Colorado, an online inter-rater agreement training system, being developed in partnership with My Learning Plan. This online system helps evaluators develop a deeper understanding of the professional practices embodied the Colorado State Model Evaluation System's teacher rubric. Using the system, evaluators can view short videos of practicing teachers, rate the videos according to the Colorado State Model Evaluation System rubric and then receive feedback showing how closely they rated the videos relative to ratings from master scorers.
Features of Elevate Colorado
Note: Elevate Colorado will not be fully developed until August 2014. These initial videos are being released on Aug. 15 to allow educators to begin coming to a common understanding of what the Colorado State Model Evaluation System professional practices look like in a variety of classrooms. This initial release is not intended to be used as the sole requirement for certification of evaluators. Districts and schools may use these resources within their own systems to develop calibration and certification requirements. Additional videos will be released this school year and in August 2014 modules will be ready for trainers, calibration/certification of teacher evaluators and professional development.
- For the 2013-14 school year, six 10-15 minute videos are being released that are designed to give educators the opportunity to quickly gauge their understanding of the professional practices in the model system rubric
- Each video has been aligned to professional practices in the Colorado State Model Evaluation System rubric by a group of teachers and administrators from around the state who have been trained as Master Scorers
- Two video clips were created for each level (elementary, middle and high school) so that educators have access to examples they need
- Immediate feedback is provided
Process: Administrators and teachers from around the state applied to be Master Scorers. The selected Master Scorers were trained on the Elevate Colorado system and best practices in observation. The Master Scorers first watched six video clips individually and aligned them to the professional practices. CDE then held a two-day session where the Master Scorers calibrated their thinking by reviewing the collated evidence and coming to agreement on the final evidence and scores. Through this process, the Master Scorers determined that they would only align practices that were explicit and high quality examples of the professional practices.
There may be multiple examples of the same professional practice within a video, but in most cases we limited the alignment to one-two examples of each. For example, students may be engaged and working in groups throughout a lesson segment, but we tried to align it only once or twice during the video clip. In addition, Standard II, Classroom Environment provided a special challenge for Master Scorers. Many of these classroom examples did not have explicit examples of a teacher addressing classroom culture or student behaviors because students were on task and performing as expected. We decided that because the goal was to show explicit examples of the teacher’s practice that we would not align professional practices unless it was evident that the teacher was addressing it in that moment. In actual practice, this would be a conversation with the teacher later about how the teacher established the practices and routines that encourage positive culture and appropriate student behavior.
Uses of Elevate Colorado
For Individual Professional Development
Any Colorado educator can use this system to view video clips of lessons. Select an elementary, middle school or high school video and practice aligning the professional practices and evidence. When finished, click on the Evidence Comparison tab to compare your results to that of the Master Scorers. For more practice two videos are provided at each level.
For Staff Development
Because this is an online system, staff can view videos and align evidence prior to coming together. Once together, review the results, refer back to evidence in the video by clicking on the film icon, and calibrate based on the master scores. This practice allows for rich discussion that can uncover inconsistencies and/or bias in observation as well as support staff in coming to consensus on what is high quality implementation of the Teacher Quality Standards.
In addition, these video clips can be used to generate discussion about the type of coaching conversations and support that could be offered to the teacher.
For Evaluator Training
Training for evaluators can incorporate each of the above recommendations. However, we know that this is the very first step in “norming” or “calibrating,” so during the 2013-14 school year, CDE will continue to build “teaching cases” that will include a full lesson observation, two-three video clips of 10-15 minute duration and artifacts that capture a broader body of evidence to support the calculation of an overall rating for professional practices. We know that with more videos and practice, evaluators will refine their skills in aligning evidence to the rubric for a more accurate and fair assessment of teacher performance.
How do you know if you are meeting a level of proficiency in identifying professional practices?
Current research* indicates that a minimum of 75 percent alignment is recommended. As you align evidence at the professional practices level to each video clip, you will be able to compare your response to that of the Master Scorers through the Evidence Comparison tab. Remember that Master Scorers sought to include only explicit, quality evidence for each practice. In other words, try not to infer what is happening, but what is visible in teacher and student actions.
Keep in mind that we are not yet using this system to determine an overall professional practice rating. These 10-15 minute video clips would not provide sufficient evidence because of the ongoing nature of evaluation process over a full school year.
* Measuring and Promoting Inter-rater Agreement of Teacher and Principal Performance Ratings, Center for Educator Compensation Reform. Graham, Milanowski, and Miller, Feb. 2012, p. 9-12.
Ready to begin?
Visit the Elevate Colorado website to get started. For detailed instructions on how to create an account and use the system, make sure to download this reader-friendly technical user's guide.
Share your skills and expertise with educators across Colorado to support inter-rater agreement
The Educator Effectiveness Unit is partnering with Colorado educators to create a library of professional practice that will be made available in Elevate Colorado. The library will deepen Colorado educators’ understanding of the Teacher Quality Standards and will support coaches and evaluators in identifying quality teaching practices. To make this a robust system with the goals and values of Colorado educators, CDE would like to film interested teachers’ lessons and reflections, and collect supporting artifacts that will then be aligned to the professional practices by experienced Master Scorers. If you are interested in being filmed or learning more, please contact Dawn Pare at email@example.com.