Site Index | CDE Offices | Staff Directory

Elevate Colorado (Inter-rater Agreement)

To support fair and consistent evaluations, CDE is developing tools to promote common interpretations of teacher quality and help evaluators provide useful and actionable feedback to educators. One such tool is Elevate Colorado, an online inter-rater agreement training system, being developed in partnership with My Learning Plan. This online system helps evaluators develop a deeper understanding of the professional practices embodied the Colorado State Model Evaluation System's teacher rubric. Using the system, evaluators can view short videos of practicing teachers, rate the videos according to the Colorado State Model Evaluation System rubric and then receive feedback showing how closely they rated the videos relative to ratings from master scorers.

Another tool to support fair and consistent evaluations is The Resource Guide for Deepening the Understanding of Teacher's Professional Practices. The resource guide is meant to support Colorado school district employees, administrators and teachers in developing an understanding of the Colorado State Model Evaluation System Teacher Rubric. As individuals apply the rubric to their own teaching and to that of others, the resource guide can support observers and coaches in accurately identifying evidence for the professional practices. The guide also supports classroom teachers to accurately reflect on their teaching and plan for implementation of specific practices in their instruction. The research, examples provided and the glossary can be used to support the development of a common language for Colorado school district employees to analyze, reflect and plan instruction.

*Please note that the resource guide reflects upcoming changes to the professional practices in the 2014-15 school year rubric which will be available in June 2014 on the CDE website.

 *This is a living document and because it will be updated periodically and eventually be an interactive online format, we want to encourage your feedback and submission of examples and resources for the continued development of this resource.  Please contact Dawn Paré at pare_d@cde.state.co.us with your feedback, questions or additional resources.

Features of Elevate Colorado

Note: Elevate Colorado will not be fully developed until August 2014. These initial videos are being released to allow educators to begin coming to a common understanding of what the Colorado State Model Evaluation System professional practices look like in a variety of classrooms. This initial release is not intended to be used as the sole requirement for certification of evaluators. Districts and schools may use these resources within their own systems to develop calibration and certification requirements. Additional videos will be in August 2014 modules will be ready for trainers, calibration/certification of teacher evaluators and professional development.

  • For the 2013-14 school year, six 10-15 minute videos are being released that are designed to give educators the opportunity to quickly gauge their understanding of the professional practices in the model system rubricEach video has been aligned to professional practices in the Colorado State Model Evaluation System rubric by a group of teachers and administrators from around the state who have been trained as Master Scorers
  • Two video clips were created for each level (elementary, middle and high school) so that educators have access to examples they need
  • Immediate feedback is provided

Process: Administrators and teachers from around the state applied to be Master Scorers. The selected Master Scorers were trained on the Elevate Colorado system and best practices in observation. The Master Scorers first watched six video clips individually and aligned them to the professional practices. CDE then held a two-day session where the Master Scorers calibrated their thinking by reviewing the collated evidence and coming to agreement on the final evidence and scores. Through this process, the Master Scorers determined that they would only align practices that were explicit and high quality examples of the professional practices.

There may be multiple examples of the same professional practice within a video, but in most cases we limited the alignment to one-two examples of each. For example, students may be engaged and working in groups throughout a lesson segment, but we tried to align it only once or twice during the video clip. In addition, Standard II, Classroom Environment provided a special challenge for Master Scorers. Many of these classroom examples did not have explicit examples of a teacher addressing classroom culture or student behaviors because students were on task and performing as expected. We decided that because the goal was to show explicit examples of the teacher’s practice that we would not align professional practices unless it was evident that the teacher was addressing it in that moment. In actual practice, this would be a conversation with the teacher later about how the teacher established the practices and routines that encourage positive culture and appropriate student behavior.

Uses of Elevate Colorado

For Individual Professional Development

Any Colorado educator can use this system to view video clips of lessons. Select an elementary, middle school or high school video and practice aligning the professional practices and evidence. When finished, click on the Evidence Comparison tab to compare your results to that of the Master Scorers. For more practice two videos are provided at each level.

For Staff Development

Because this is an online system, staff can view videos and align evidence prior to coming together. Once together, review the results, refer back to evidence in the video by clicking on the film icon, and calibrate based on the master scores. This practice allows for rich discussion that can uncover inconsistencies and/or bias in observation as well as support staff in coming to consensus on what is high quality implementation of the Teacher Quality Standards. 

In addition, these video clips can be used to generate discussion about the type of coaching conversations and support that could be offered to the teacher. 

For Evaluator Training

Training for evaluators can incorporate each of the above recommendations. However, we know that this is the very first step in “norming” or “calibrating,” so during the 2013-14 school year, CDE will continue to build “teaching cases” that will include a full lesson observation, two-three video clips of 10-15 minute duration and artifacts that capture a broader body of evidence to support the calculation of an overall rating for professional practices. We know that with more videos and practice, evaluators will refine their skills in aligning evidence to the rubric for a more accurate and fair assessment of teacher performance.

How do you know if you are meeting a level of proficiency in identifying professional practices?

Current research* indicates that a minimum of 75 percent alignment is recommended. As you align evidence at the professional practices level to each video clip, you will be able to compare your response to that of the Master Scorers through the Evidence Comparison tab. Remember that Master Scorers sought to include only explicit, quality evidence for each practice. In other words, try not to infer what is happening, but what is visible in teacher and student actions.

Keep in mind that we are not yet using this system to determine an overall professional practice rating. These 10-15 minute video clips would not provide sufficient evidence because of the ongoing nature of evaluation process over a full school year.

* Measuring and Promoting Inter-rater Agreement of Teacher and Principal Performance Ratings, Center for Educator Compensation Reform. Graham, Milanowski, and Miller, Feb. 2012, p. 9-12.

Ready to Begin?

Visit the Elevate Colorado website to get started. For detailed instructions on how to create an account and use the system, make sure to download this reader-friendly technical user's guide.  

 

Share your skills and expertise with educators across Colorado to support inter-rater agreement

The Educator Effectiveness Unit is partnering with Colorado educators to create a library of professional practice that will be made available in Elevate Colorado. The library will deepen Colorado educators’ understanding of the Teacher Quality Standards and will support coaches and evaluators in identifying quality teaching practices.  To make this a robust system with the goals and values of Colorado educators, CDE would like to film interested teachers’ lessons and reflections, and collect supporting artifacts that will then be aligned to the professional practices by experienced Master Scorers.  If you are interested in being filmed or learning more, please contact Dawn Pare at pare_d@cde.state.co.us

Quick Links

 

Contact Us

Colorado Dept. of Education
201 East Colfax Ave.
Denver, CO 80203
Phone: 303-866-6600
Fax: 303-830-0793
Contact CDE

CDE Hours
Mon - Fri 8 a.m. to 5 p.m.
See also: Licensing Hours

Educator Licensing