Title II-D Dissemination Report (PDF)
The Elementary and Secondary Education Act, reauthorized as the No Child Left Behind Act (2001), provided eligible local educational agencies (LEAs) with Title II, Part D funds for the purchase of technology and technology‐related professional development prior to the termination of the funding in 2010‐2011. Although only 25% of the IID funds had to be earmarked for PD, Colorado LEAs allocated a greater percentage to PD than was statutorily required. Therefore, the evaluation report focused on the PD activities funded with IID funds. There is evidence, though not statistically significant, that there is a relationship between personnel technological proficiency and student technology literacy and that LEAs with lower rates of personnel proficiency were more likely to spend a greater percentage of funds on PD for teachers and administrators.
Special Education Teachers & Attitudes Toward Mathematics (PDF)
The purpose of the study was to explore the beliefs and attitudes of special education teachers have about the discipline of mathematics, teaching mathematics, and learning mathematics. The study utilized a mixed method design that was conducted in two phases. Forty-eight in-service special education teachers participated in Phase One of the study, which consisted of quantitative data collection through surveys related to mathematics anxiety level and alignment of beliefs with the National Council of Teachers of Mathematics (NCTM) Standards. A sub-sample of seven teachers was purposefully selected to participate in Phase Two of the study, which consisted of a qualitative data collection through a semi-structured interview. Quantitative results indicated that the study sample had relatively low levels of mathematics anxiety and a relatively high degree of alignment with reform-based mathematics beliefs promoted by the NCTM. Qualitative results expanded upon the quantitative results of Phase One of the study and indicated that the beliefs of the sub-sample participants could be categorized according to beliefs common to general education mathematics teachers.
Characteristics of Online Students (PDF)
Over 16,000 Colorado students attend an online public school. However, as the number of students attending online schools has grown and changed over the years, interest and questions about online schools from policymakers, media, and the general public has piqued. This study sets out to answer some of these questions. It utilizes Colorado Department of Education (CDE) collected data to analyze demographics, trends, and performance in online schools over time. It includes assessment, pupil enrollment, demographic and socioeconomic data collected from 2003 through 2011, as this is both the earliest and most recent data available from CDE at the student level.
As part of a statewide effort to help Colorado district and state leaders understand the scope and range of student assessments currently in use, the Colorado Legacy Foundation (CLF) contracted with Augenblick, Palaich and Associates (APA) in November 2011 to conduct a first of its kind study intended to provide an inventory of the nature, type, and frequency of student assessments that are given in school districts in the state. This work is particularly important in light of upcoming changes to the state’s assessment system and requirements for districts as a result of recent legislation including Colorado’s Achievement Plan for Kids (CAP4K) and the Educator Effectiveness bill, S.B. 10‐191. The information generated through this study will inform state efforts to implement new assessment systems, as well as inform local leaders on how to best leverage existing assessments.
Measuring Preschool Teachers' Perceived Competency (PDF)
Research has demonstrated the impact of early oral language development on a child’s later reading comprehension. Additionally, research has suggested that teachers’ knowledge of effective practices in literacy plays an important role in students’ ability to learn to read. The problem is that preschool teachers’ knowledge of strategies for developing language is unknown because there is no known instrument for assessing preschool teachers’ knowledge of these strategies. The research questions for this study examined the development of an instrument to measure preschool teachers’ perceived competency and knowledge of strategies for language development. Challs reading stage theory was used as the theoretical foundation. This quantitative, non-experimental study was conducted using a descriptive, cross-sectional design. After a pre-pilot review of the instrument by literacy experts, a pilot study was completed using a convenience sample of 250 teachers who volunteered to answer the questions on the instrument. Reliability statistics demonstrated a high level of internal consistency for Section 2, promoting extended discourse (α = .86) and low levels of internal consistency for the other two sections of the instrument. Further analysis of Section 2 revealed a positive moderate effect size of 0.53, indicating significant variability in test scores between high and low performing teachers. Use of the instrument developed through this study supports social change by providing early childhood professionals information to understand teachers’ instructional decisions, determine professional development to increase teachers knowledge, and inform preschool teachers pre-service preparation.
Fidelity of Implementation and Achievement (PDF)
The purpose of this report was to increase our understanding of the scope and impact of the Positive Behavioral Intervention Supports (PBIS) framework in Colorado. Also, this study serves to initiate a research agenda that supports best-practices and facilitates successful PBIS implementation. The key findings include: (1) PBIS schools tend to support higher percentages of minority and free lunch students compared to the state; (2) more than half of PBIS sites are implementing with fidelity; and (3) for elementary and middle schools roughly a three to five point gain on the Benchmarks of Quality predicts a 1% increase in math and reading proficiency scores on the CSAP.
Title I-A SES Provider Evaluation (PDF)
The Elementary and Secondary Education Act requires that schools on second year of school improvement and beyond offer tutoring outside of school time to students using providers that have been approved by the State Educational Agency (SEA). The SEA is also responsible for evaluating the effectiveness of the approved providers in increasing the academic performance of the students served as measured by student performance on the state assessment(s). The Center for Research Strategies conducted the evaluation of the providers approved to implement SES tutoring for Colorado students. The performance of students served by each provider is compared to a randomly selected group of students who were eligible for services but did not receive any such services. The provider effectiveness report summarizes the demographics of the students served by each company and how they performed in relation to other providers and the comparison group.
Tell Survey 2011 Research Brief (PDF)
The Teaching, Empowering, Leading and Learning (TELL) Colorado survey is an anonymous statewide survey of licensed, school‐based educators to assess teaching conditions at the school, district and state level. (Note: teachers and principals in charter schools are included and do not need to be licensed to participate.) The survey results are intended to support school and district improvement planning and to inform policy decisions. In February 2011, the second iteration of the TELL Colorado Survey was conducted. Nearly 30,000 educators (47 percent) from across the state shared their perceptions of the teaching and learning conditions in the schools in which they work, indicating whether they have the kind of supportive teaching and learning conditions necessary for enabling teachers and students to be successful. Both the 2009 and now the 2011 TELL Colorado Survey indicate that students perform at higher levels in schools with more positive teaching conditions. In particular, there is stronger student achievement at schools where the parents/guardians and the community at large know what is going in the school, are influential decision makers and support teachers and the school as a whole.
Effects of Online Math Intervention Program on Achievement (PDF)
The purpose of this report was to examine the impact of the Compass Learning Odyssey program on math achievement in low‐performing middle school math students in twelve southern Colorado school districts. Our findings indicate that intervention participants fail to statistically differ on overall CSAP growth compared to the matched control. However, the Compass Learning students performing at the unsatisfactory level on the 2009 CSAP exhibited growth percentiles that were greater than those exhibited by the control. In addition, a larger percentage of intervention students that were at the unsatisfactory level in 2009 moved to a higher proficiency level in the 2010 school year compared to the control group (i.e. 25% to 19% respectively). Finally, the math measures of academic progress assessment revealed Rasch Unit (RIT) growth rates for program participants that exceeded expectations based on national norms.
Sixty-seven Colorado school districts operate all their schools on a four-day week rather than a five-day week. Colorado law requires that all districts provide a specified amount of ‘contact time’ for students. Consequently, the shorter week includes longer days so the actual ‘contact time’ is the same as the schools with longer weeks. This report compares the academic achievement and student growth of the four-day districts to the academic achievement of five-day districts of similar size. Overall, the results indicate that both groups of districts perform similarly on the state assessments and that their students show very similar amounts of academic growth as reflected by the Colorado Growth Model.
Title I-A Dissemination Report (PDF)
The Tile I, Part A Dissemination Report summarizes the findings from the program evaluation conducted by OMNI Institute, addressing two of our evaluation questions: (1) Do students served in Title I schoolwide schools have a better growth trajectory than students served in Title I Targeted Assistance schools and not-served students? (2) Is there a relationship between the school’s median growth percentile and the amount of title I per pupil allocation to the school? The evaluation results indicate that although students in SW schools started with a lower reading and math scale scores, their growth trajectories across 3 years was the same as students served in targeted assistance schools and students not served with Title I funds. In the PPA study, there was some evidence that a higher per pupil allocation (over $900 per student) was correlated with higher median growth percentile, though not all trends were statistically significant.
Title II-A Dissemination Report (PDF)
The Elementary and Secondary Education Act, reauthorized as the No Child Left Behind Act (2001) allows local educational agencies (LEAs) to use Title II, Part A funds to improve the quality of teaching and administration within schools and districts. In the earlier years of NCLB, LEAs were permitted to use funds for such activities as classroom size reduction (CSR), in addition to other activities such as professional development, retention and recruitment. However, when research did not yield significant evidence of the effectiveness of classroom size reduction practices, CDE implemented policies to reduce the amount of funding used to support CSR out of Title II, Part A funds. CDE was interested in comparing the longitudinal trends in activities and strategies supported by Title II, Part A funds, paying particular attention to changes in CSR from before to after the policy change. The exploratory analyses of the funded activities revealed a reduction of II-A funds used to support CSR and increases in other more effective strategies.
Shining a Light on Remediation (PDF)
The purpose of this study was to study postsecondary readiness for 17,499 Colorado students by examining the congruence between middle school and high school state assessment results (Colorado State Assessment Program) from 2007, ACT results from 2008 and the need for remediation for Colorado students who graduated from high school in the spring of 2009 and entered a Colorado postsecondary institution in fall 2009. By examining the assessment results for these students from as early as the sixth grade, it was clear that if students were not proficient on the state assessment in sixth grade, they were likely to require remediation in their first year of college. If middle school teachers would analyze the state assessment data for this purpose they would be better able to identify which students are very likely be postsecondary ready and which students are not. Also teachers could use the assessment results to target the academic skills of struggling students early in middle school to focus on preparing them to be postsecondary ready. The eighth grade results could be used to gauge how successful the middle and K-8 schools have been in moving students toward Postsecondary and Workforce Readiness (PWR). High schools could use the data from the middle school years to target incoming ninth graders who are not yet proficient on the state assessment. More precise use of state assessment data could focus educators on the ultimate goal of developing postsecondary- and workforce-ready students in all grades, not just those for which graduation is rapidly approaching.
Predicting ACT Scores Using CSAP (PDF)
Research indicates 9th and 10th grade CSAP can reliably predict a range of Colorado ACT (CO ACT) scores for each student. Using the Colorado Growth Model, predictive linkages can be made from the 9th and 10th grade CSAP to the 11th grade CO ACT in each content area tested. This means that every possible 9th and 10th grade CSAP score has a predicted range of CO ACT scores for students making typical growth. Additionally, 9th grade scores can be used to directly predict 11th grade CO ACT performance without waiting for the 10th grade performance data. Including CO ACT scores in the Colorado Growth Model as an 11th grade state assessment also enhances the consistency of interpretations of growth data into upper high school grades.
A key intent of the NCLB growth pilot is to reward low-status schools who are closing the gap to proficiency. In this article, we demonstrate that the capability of proposed models to identify those schools depends on how the growth model is incorporated into accountability decisions. Six pilot-approved growth models were applied to vertically scaled mathematics assessment data from a single state collected over 2 years. Student and school classifications were compared across models. Accountability classifications using status and growth to proficiency as defined by each model were considered from two perspectives. The first involved adding the number of students moving toward proficiency to the count of proficient students, while the second involved a multitier accountability system where each school was first held accountable for status and then held accountable for the growth of their nonproficient students. Our findings emphasize the importance of evaluating status and growth independently when attempting to identify low-status schools with insufficient growth among non-proficient students.
Glossary and reading aloud test items are often listed as allowed in many states’ accommodation policies for ELL students, when taking states’ large-scale mathematics assessments. However, little empirical research has been conducted on the effects of these two accommodations on ELL students’ test performance. Further, no research is available to examine how students use the provided accommodations. The present study employed a randomized experimental design and a think-aloud procedure to delve into the effects of the two accommodations. A total of 605 ELL and non-ELL students from two states participated in the experimental component and a subset of 68 ELL students participated in the think-aloud component of the study. Results showed no significant effect of glossary, and mixed effects of read aloud on ELL students’ performance. Read aloud was found to have a significant effect for the ELL sample in one state, but not the other. Significant interaction effects between students’ prior content knowledge and accommodations were found, suggesting the given accommodation was effective for the students who had acquired content knowledge. During the think-aloud analysis, students did not actively utilize the provided glossary, indicating lack of familiarity with the accommodation. Implications for the effective use of accommodations and future research agendas are discussed.
Accommodations have been widely utilized as a way of increasing the validity of content assessments for ELL students. However, concerns have also arisen regarding the validity of accommodation use, as well as accessibility and fairness. While many states have developed ELL‐specific accommodation policies and guidelines, little research has been available on how the accommodation policies are carried out in practice. The present study investigated two states’ accommodation policies, specifically for the states’ respective large-scale Grade 8 math assessments, and conducted a case study to examine teachers’ understanding of the policies and uses of accommodations in their respective schools. Results indicated a wide variation in applying the policies in practice, which raises a validity concern for providing accommodations and interpreting accommodated test results. Based on the findings, implications and recommendations for an appropriate use of accommodations are offered.
Is Growth in Student Achievement Scale Dependent? (PDF)
The impetus for this paper can be traced back to two simple questions that most parents likely ask at some point in time during their child s education: 1) How much has my child learned? 2) Is the amount my child has learned good enough? These are straightforward and intuitively important questions about growth. The first asks a question about the magnitude of growth. The second asks a question about criteria for judging the amount of growth. While the questions may be intuitive, the psychometric and statistical gymnastics involved in coming up with a defensible answer are not. Indeed, the deceptively simple nature of the questions masks important conceptual and philosophical undercurrents. Learning of what? How should learning be measured? Can a single number capture this phenomenon? Who decides how much learning is good enough? Once decisions about what constitutes good enough are decided, does this eliminate questions of magnitude? Or are both questions compatible, such that posing one will tend to beg an answer to the other?
In what follows we pose a research question that is also deceptively straightforward: Do interpretations of student growth depend on the way longitudinal test scores have been scaled? This paper provides a theoretical and empirical context where one can give a provisional answer of no to this question. We show that when growth interpretations are made normatively, they appear insensitive to most admissible transformation of the underlying score scale. In particular we focus attention on student growth estimates aggregated at the school-level that derive from a growth model currently being used for state accountability purposes in Colorado and Massachusetts and the value-added estimates produced by multivariate mixed-effects models. The former model relies upon quantile regression, an approach not requiring an underlying score scale with interval properties. The latter value-added model assumes interval scale properties. The two models present a contrast allowing us to examine both the within methodology impact of non-interval scales as well as the impact across methodologies.
Growth, Standards and Accountability (PDF)
Over the last decade, large scale annual testing has provided states with unprecedented access to longitudinal student data. Current use of this data focuses primarily upon analyses of growth most often directed toward accountability decisions. Analyses using this longitudinal data source for other purposes have gone largely untapped. This paper introduces analysis techniques and results showing how student growth percentiles, a normative growth analysis technique, can be used to examine the illuminate the relationship between standards based accountability systems and the performance standards on which they are based.
A Primer on Student Growth Percentiles (PDF)
Following the 2001 reauthorization of the Elementary and Secondary Education Act of 1965, commonly known as the No Child Left Behind Act (NCLB), states were required to implement large scale testing of all students to an extent never before seen in the United States (No Child Left Behind Act of 2001 (NCLB), 2002). Starting with the 2005-2006 school year, NCLB required states to test students in reading and mathematics from grades 3 through 8 and at least once in grades 10 through 12.
Accountability systems constructed according to federal adequate yearly progress (AYP) requirements currently rely upon annual measurement of student achievement to make judgments about school quality. Since their adoption, such status measures have been the focus of persistent criticism (Linn, 2003; Linn, Baker, & Betebenner, 2002). Status measures, though appropriate for making judgments about the achievement level of students at a school for a given year, are inappropriate for judgments about educational effectiveness. In this regard, status measures are blind to the possibility of low achieving students attending effective schools. It is this possibility that has led some critics of NCLB to label its achievement mandates as unfair and misguided and to demand the use of growth analyses as a better means of auditing the quality of schools.
A fundamental premise associated with the use of student growth for school accountability is that “good” schools bring about student growth in excess of that found at “bad” schools. Despite the utility of using growth to assess the effectiveness of schools, such analyses fail to address one of the fundamental questions concerning the growth of students: How much growth did a student make? This paper describes the advantages of student growth percentiles to quantify change in student achievement.
Early Reading Development (PDF)
In a 3-year longitudinal study, middle- to upper-middle-class preschool children at high family risk (HR group, N=67) and low family risk (LR group, N=57) for dyslexia (or reading disability, RD), were evaluated yearly from before kindergarten to the end of second grade. Both phonological processing and literacy skills were tested at each of four time points. Consistent with the well-known familiarity of RD, 34% of the HR group compared with 6% of the LR group became RD. Participants who became RD showed deficits in both implicit and explicit phonological processing skills at all four time points, clearly indicating a broader phonological deficit than is often found at older ages. The predictors of literacy skill did not vary by risk group. Both risk groups underwent a similar developmental shift from letter-name knowledge to phoneme awareness as the main predictor of later literacy skill. This shift, however, occurred 2 years later in the HR group. Familial risk was continuous rather than discrete because HR children who did not become RD performed worse than LR non-RD children on some phonological and literacy measures. Finally, later RD could be predicted with moderate accuracy at age 5 years, with the strongest predictor being letter-name knowledge.