Group items tagged

There is strong evidence that the choice of instructional materials has large effects on student learning-effects that rival in size those that are associated with differences in teacher effectiveness.  But whereas improving teacher quality through changes in the preparation and professional development of teachers and the human resources policies surrounding their employment is challenging, expensive, and time-consuming, making better choices among available instructional materials should be relatively easy, inexpensive, and quick.

Teacher pension systems target retirements within a narrow range of the career cycle by penalizing individuals who separate too soon or remain employed too long. The penalties result in the retention of some teachers who would otherwise choose to leave, and the premature exit of some teachers who would otherwise choose to stay. We examine how the effects of teachers' pension incentives on workforce composition influence teacher quality. Teachers who are held in by the "pull" incentives in the pension systems are not more effective, on average, than the typical teacher. Teachers who are encouraged to exit by the "push" incentives are more effective on average. We conclude that the net effect of teachers' pension incentives on workforce quality is small, but negative. Given the substantial and growing costs of current systems, and the lack of evidence regarding their efficacy, experimentation by traditional and charter schools with alternative retirement benefit structures would be useful.

woke up yesterday morning to read Norm Scott's post on Education Notes Online about a new study of the effects of charter schools on achievement in New York City.  The study, by economists Caroline Hoxby and Sonali Murarka, finds a charter school effect of .09 standard deviations per year of treatment in math and .04 standard deviations per year in reading.  I haven't read the study closely yet, but I was struck by Norm's headline:  "Study Shows NO Improvement in NYC Charters Over Public Schools."  The effects that Hoxby and Murarka report are statistically significant, which means that we can reject the claim that they are zero.  But are they big?  That's a surprisingly complicated question. I'm going to argue that the answer hinges on "compared to what?"

Charter schools were developed, in part, to serve as an R&D engine for traditional public schools, resulting in a wide variety of school strategies and outcomes. In this paper, we collect unparalleled data on the inner-workings of 35 charter schools and correlate these data with credible estimates of each school's effectiveness. We find that traditionally collected input measures -- class size, per pupil expenditure, the fraction of teachers with no certification, and the fraction of teachers with an advanced degree -- are not correlated with school effectiveness. In stark contrast, we show that an index of five policies suggested by over forty years of qualitative research -- frequent teacher feedback, the use of data to guide instruction, high-dosage tutoring, increased instructional time, and high expectations -- explains approximately 50 percent of the variation in school effectiveness. Our results are robust to controls for three alternative theories of schooling: a model emphasizing the provision of wrap-around services, a model focused on teacher selection and retention, and the "No Excuses'' model of education. We conclude by showing that our index provides similar results in a separate sample of charter schools.

The Boston Teacher Residency is an innovative practice-based preparation program in which candidates work alongside a mentor teacher for a year before becoming a teacher of record in Boston Public Schools. We find that BTR graduates are more racially diverse than other BPS novices, more likely to teach math and science, and more likely to remain teaching in the district through year five. Initially, BTR graduates for whom value-added performance data are available are no more effective at raising student test scores than other novice teachers in English language arts and less effective in math. The effectiveness of BTR graduates in math improves rapidly over time, however, such that by their fourth and fifth years they out-perform veteran teachers. Simulations of the program's overall impact through retention and effectiveness suggest that it is likely to improve student achievement in the district only modestly over the long run.

In a provocative and influential paper, Jesse Rothstein (2010) finds that standard value-added models (VAMs) suggest implausible future teacher effects on past student achievement, a finding that obviously cannot be viewed as causal. This is the basis of a falsification test (the Rothstein falsification test) that appears to indicate bias in VAM estimates of current teacher contributions to student learning. More precisely, the falsification test is designed to identify whether or not students are effectively randomly assigned conditional on the covariates included in the model. Rothstein's finding is significant because there is considerable interest in using VAM teacher effect estimates for high-stakes teacher personnel policies, and the results of the Rothstein test cast considerable doubt on the notion that VAMs can be used fairly for this purpose. However, in this paper, we illustrate-theoretically and through simulations-plausible conditions under which the Rothstein falsification test rejects VAMs even when students are randomly assigned, conditional on the covariates in the model, and even when there is no bias in estimated teacher effects.

In our fruitless, deadlocked debate over whether charter schools "work," charter opponents frequently cite the so-called CREDO study (discussed here), a 2009 analysis of charter school performance in 16 states. The results indicated that overall charter effects on student achievement were negative and statistically significant in both math and reading, but both effects sizes were tiny. Given the scope of the study, it's perhaps more appropriate to say that it found wide variation in charter performance within and between states - some charters did better, others did worse and most were no different. On the whole, the size of the aggregate effects, both positive and negative, tended to be rather small. Recently, charter opponents' tendency to cite this paper has been called "cherrypicking." Steve Brill sometimes levels this accusation, as do others. It is supposed to imply that CREDO is an exception - that most of the evidence out there finds positive effects of charter schools relative to comparable regular public schools.

Using a unique identification strategy, and two classes of fixed-effects regression models, this study presents the cleanest estimates to date of a direct effect of teacher turnover on student achievement. Three research questions guide the investigation: What is the average effect of teacher turnover on student achievement? Are the effects different for different kinds of schools? What explains the relationship between teacher turnover and student achievement

The Bill & Melinda Gates Foundation's "Measures of Effective Teaching" (MET) Project seeks to validate the use of a teacher's estimated "value-added"-computed from the year-on-year test score gains of her students-as a measure of teaching effectiveness. Using data from six school districts, the initial report examines correlations between student survey responses and value-added scores computed both from state tests and from higher-order tests of conceptual understanding. The study finds that the measures are related, but only modestly. The report interprets this as support for the use of value-added as the basis for teacher evaluations. This conclusion is unsupported, as the data in fact indicate that a teachers' value-added for the state test is not strongly related to her effectiveness in a broader sense. Most notably, value-added for state assessments is correlated 0.5 or less with that for the alternative assessments, meaning that many teachers whose value-added for one test is low are in fact quite effective when judged by the other. As there is every reason to think that the problems with value-added measures apparent in the MET data would be worse in a high-stakes environment, the MET results are sobering about the value of student achievement data as a significant component of teacher evaluations.

"In numerous previous posts I have summarized New Jersey charter school enrollment data, frequently pointing out that the highest performing charter schools in New Jersey tend to be demographically very different from schools in their surrounding neighborhoods and similar grade level schools throughout their host districts or cities. I have tried to explain over and over that the reason these differences are important is because they constrain the scalability of charter schooling as a replicable model of "success." Again, to the extent that charter successes are built on serving vastly different student populations, we can simply never know (even with the best statistical analyses attempting to sort out peer factors, control for attrition, etc.) whether the charter schools themselves, their instructional strategies/models are effective and/or would be effective with larger numbers of more representative students. Here, I take a quick look at the other side of the picture, again focusing on the city of Newark. Specifically, I thought it would be interesting to evaluate the effect on Newark schools enrollment of the shift in students to charter schools, now that charters have taken on a substantial portion of students in the city. If charter enrollments are - as they seem to be - substantively different from district schools enrollments, then as those charter populations grow and remain different from district schools, we can expect the district schools population to change.  In particular, given the demography of charter schools in Newark, we would expect those schools to be leaving behind a district of escalating disadvantage - but still a district serving the vast majority of kids in the city."

In a provocative and influential paper, Jesse Rothstein (2010) finds that standard value-added models (VAMs) suggest implausible future teacher effects on past student achievement, a finding that obviously cannot be viewed as causal. This is the basis of a falsification test (the Rothstein falsification test) that appears to indicate bias in VAM estimates of current teacher contributions to student learning. Rothstein's finding is significant because there is considerable interest in using VAM teacher effect estimates for high-stakes teacher personnel policies, and the results of the Rothstein test cast considerable doubt on the notion that VAMs can be used fairly for this purpose. However, in this paper, we illustrate-theoretically and through simulations-plausible conditions under which the Rothstein falsification test rejects VAMs even when there is no bias in estimated teacher effects, and even when students are randomly assigned conditional on the covariates in the model. On the whole, our findings show that the "Rothstein falsification test" is not definitive in showing bias, which suggests a much more encouraging picture for those wishing to use VAM teacher effect estimates for policy purposes.

"Class Size: What Research Says and What It Means for State Policy argues that increasing average class size by one student will save about 2% of total education spending with negligible impact on academic achievement. It justifies this conclusion on the basis that Class-Size Reduction (CSR) is not particularly effective and is not as cost-effective as other reforms. However, this conclusion is based on a misleading review of the CSR research literature. The report puts too much emphasis on studies that are of poor quality or that do not focus on settings that are particularly relevant to the debate on class-size policy in the United States. It argues that class-size reduction is less cost-effective than other reform policies, but it bases this contention on an incomplete accounting of the benefits of smaller classes and an uncritical, unexamined list of alternative policies. The report's estimates of the potential cost savings are flawed as, in reality, schools cannot structurally reduce class size by only one student. Well-documented and long-term non-academic gains from CSR are not addressed. Likewise, the recommendation for releasing the ―least effective‖ teachers assumes a valid way of making such determinations is available. "

This book aims to inform the policy debate by examining four primary research questions in several geographic locations: (1) What are the characteristics of students transferring to charter schools? (2) What effect do charter schools have on test-score gains for students who transfer between TPSs and charter schools? (3) What is the effect of attending a charter high school on the probability of graduating and of entering college? (4) What effect does the introduction of charter schools have on test scores of students in nearby TPSs? We  examine similarities and diferences in the answers to these questions across locations, seeking insights about the policy levers that might be available to improve the outcomes associated with charter schools.

The National Study of CMO Effectiveness is a four-year study designed to assess the impact of CMOs on student achievement and to identify CMO structures and practices that are most effective in raising achievement. Earlier reports from the study documented substantial variation in CMOs' student achievement impacts and in CMOs' use of particular educational strategies and practices. The last report from the study found that the most effective CMOs tend to emphasize two practices in particular: high expectations for student behavior and intensive teacher coaching and monitoring. This report provides a more in-depth description of these two promising CMO practices, drawing on surveys and interviews with staff in high-performing CMOs that emphasize one or both practices.

In this paper we report on work estimating the stability of value-added estimates of teacher effects, an important area of investigation given public interest in workforce policies that implicitly assume effectiveness is a stable attribute within teachers. The results strongly reject the hypothesis that teacher performance is completely stable within teachers over long periods of time, but estimates suggest that a component of performance appears to persist within teachers, even over a ten-year panel. We also find that little of the changes in teacher effectiveness estimates within teachers can be explained by observable characteristics.

Teachers who embodied the teaching skills outlined in certain popular teaching frameworks tended to help their students learn more, concludes a new study released by the Bill & Melinda Gates Foundation. The report is the second major release from the foundation's Measures of Effective Teaching project, which seeks to identify the best gauges of effective teaching. Ultimately, the Seattle-based foundation plans to devise a prototype teacher-effectiveness measure based on the findings.

With teacher quality repeatedly cited as the most important schooling factor influencing student achievement, there has been increased interest in examining the efficacy of teacher training programs. This paper presents research examining the variation between and impact that individual teacher training institutions in Washington state have on the effectiveness of teachers they train. Using administrative data linking teachers' initial endorsements to student achievement on state reading and math tests, we find the majority of teacher training programs produce teachers who are no more or less effective than teachers who trained out-of-state. However, we do find a number of cases where there are statistically significant differences between estimates of training program effects for teachers who were credentialed at various in-state programs. These findings are robust to a variety of different model specifications.

"Given the magnitude of the recent recession, and the high-stakes testing the U.S. has implemented under the No Child Left Behind Act (NCLB), it is important to understand the effects of large-scale job losses on student achievement. We examine the effects of state-level job losses on fourth- and eighth-grade test scores, using federal Mass Layoff Statistics and 1996-2009 National Assessment of Educational Progress data. Results indicate that job losses decrease scores. Effects are larger for eighth than fourth graders and for math than reading assessments, and are robust to specification checks. Job losses to 1% of a state's working-age population lead to a .076 standard deviation decrease in the state's eighth-grade math scores. This result is an order of magnitude larger than those found in previous studies that have compared students whose parents lose employment to otherwise similar students, suggesting that downturns affect all students, not just students who experience parental job loss. Our findings have important implications for accountability schemes: we calculate that a state experiencing one-year job losses to 2% of its workers (a magnitude observed in seven states) likely sees a 16% increase in the share of its schools failing to make Adequate Yearly Progress under NCLB. "

"It's becoming a familiar story: Great teachers get low scores from "value-added" teacher evaluation models. Newspapers across the country have published accounts of extraordinary teachers whose evaluations, based on their students' state test scores, seem completely out of sync with the reality of their practice. Los Angeles teachers have figured prominently in these reports. Researchers are not surprised by these stories, because dozens of studies have documented the serious flaws in these ratings, which are increasingly used to evaluate teachers' effectiveness. The ratings are based on value-added models such as the L.A. school district's Academic Growth over Time system, which uses complex statistical metrics to try to sort out the effects of student characteristics (such as socioeconomic status) from the effects of teachers on test scores. A study we conducted at Stanford University showed what these teachers are experiencing."

The SAS Educational Value-Added Assessment System (SAS® EVAAS®) is the most widely used value-added system in the country. It is also self-proclaimed as "the most robust and reliable" system available, with its greatest benefit to help educators improve their teaching practices. This study critically examined the effects of SAS® EVAAS® as experienced by teachers, in one of the largest, high-needs urban school districts in the nation - the Houston Independent School District (HISD). Using a multiple methods approach, this study critically analyzed retrospective quantitative and qualitative data to better comprehend and understand the evidence collected from four teachers whose contracts were not renewed in the summer of 2011, in part given their low SAS® EVAAS® scores. This study also suggests some intended and unintended effects that seem to be occurring as a result of SAS® EVAAS® implementation in HISD. In addition to issues with reliability, bias, teacher attribution, and validity, high-stakes use of SAS® EVAAS® in this district seems to be exacerbating unintended effects.