Group items tagged

Evidence (and evidence-based reform) are entirely neutral on the nature of teaching. Whatever works is what is valued. The distinction between teaching driven by accountability and teaching informed by evidence is crucial. Using test scores to evaluate teachers and schools, at least as defined by NCLB, runs the risk of focusing teachers on a narrow band of reading and math skills, and school and district leaders often try to improve performance by "alignment," trying to get teachers to spend more time on the skills and knowledge likely to be assessed. In contrast, evidence-based policies have no such limitations.

"The so-called Vergara trial in California, in which the state's tenure and layoff statutes were deemed unconstitutional, already has its first "spin-off," this time in New York, where a newly-formed organization, the Partnership for Educational Justice (PEJ), is among the organizations and entities spearheading the effort. Upon first visiting PEJ's new website, I was immediately (and predictably) drawn to the "Research" tab. It contains five statements (which, I guess, PEJ would characterize as "facts"). Each argument is presented in the most accessible form possible, typically accompanied by one citation (or two at most). I assume that the presentation of evidence in the actual trial will be a lot more thorough than that offered on this webpage, which seems geared toward the public rather than the more extensive evidentiary requirements of the courtroom (also see Bruce Baker's comments on many of these same issues surrounding the New York situation). That said, I thought it might be useful to review the basic arguments and evidence PEJ presents, not really in the context of whether they will "work" in the lawsuit (a judgment I am unqualified to make), but rather because they're very common, and also because it's been my observation that advocates, on both "sides" of the education debate, tend to be fairly good at using data and research to describe problems and/or situations, yet sometimes fall a bit short when it comes to evidence-based discussions of what to do about them (including the essential task of acknowledging when the evidence is still undeveloped). PEJ's five bullet points, discussed below, are pretty good examples of what I mean."

The use of rate changes is still proliferating rapidly at all levels of our education system. These measures, which play an important role in the provisions of No Child Left Behind, are already prominent components of many states' core accountability systems (e..g, California), while several others will be using some version of them in their new, high-stakes school/district "grading systems." New York State is awarding millions in competitive grants, with almost half the criteria based on rate changes. District consultants issue reports recommending widespread school closures and reconstitutions based on these measures. And, most recently, U.S. Secretary of Education Arne Duncan used proficiency rate increases as "preliminary evidence" supporting the School Improvement Grants program. Meanwhile, on the public discourse front, district officials and other national leaders use rate changes to "prove" that their preferred reforms are working (or are needed), while their critics argue the opposite. Similarly, entire charter school sectors are judged, up or down, by whether their raw, unadjusted rates increase or decrease. So, what's the problem? In short, it's that year-to-year changes in proficiency rates are not valid evidence of school or policy effects. These measures cannot do the job we're having them do, even on a limited basis. This really has to stop.

The nation's lawmakers have frequently voiced the basic principle that important policy decisions should be evidence based. In this commentary, the authors describe the approach the U.S. Department of Education has taken in its Increasing Educational Productivity project. The authors argue that the department's actual practice in this instance has fallen short of the rhetorical embrace of evidence-based decision making, and they explain the potential harm done when leaders do not heed the importance of grounding policy in high-quality research.

So, I've seen on more than a few occasions these last few weeks references to the recent Dobbie and Fryer article on NYC charter schools as the latest evidence that money doesn't matter in schools. That costly stuff like class size, or  overall measures of total per pupil expenditures are simply unimportant, and can easily be replaced/substituted with no-cost alternatives like those employed in no excuses charter schools (like high expectations, tutoring, additional time, and wrap-around services). I'll set aside the issue that many of these supposedly more effective alternatives do, in fact, have cost implications. Instead, I'll focus my critique on whether this Dobbie/Fryer study provides any substantive evidence that money doesn't matter - either broadly, or in the narrower context of looking specifically at NYC charter schools.

The No Child Left Behind (NCLB) Act compelled states to design school accountability systems based on annual student assessments. The effect of this federal legislation on the distribution of student achievement is a highly controversial but centrally important question. This study presents evidence on whether NCLB has influenced student achievement based on an analysis of state-level panel data on student test scores from the National Assessment of Educational Progress (NAEP). The impact of NCLB is identified using a comparative interrupted time series analysis that relies on comparisons of the test-score changes across states that already had school accountability policies in place prior to NCLB and those that did not. Our results indicate that NCLB generated statistically significant increases in the average math performance of fourth graders (effect size 5 0.23 by 2007) as well as improvements at the lower and top percentiles. There is also evidence of improvements in eighth-grade math achievement, particularly among traditionally low-achieving groups and at the lower percentiles. However, we find no evidence that NCLB increased fourth-grade reading achievement.

Former Florida Governor Jeb Bush and the Foundation for Excellence in Education have embarked on a well-funded campaign to spread selected Florida education reforms to other states. These reforms include assigning letter grades to schools, high-stakes testing, promotion and graduation requirements, bonus pay, a wide variety of alternative teacher credentialing policies, and various types of school choice mechanisms. This policy potpourri was recently presented by Gov. Bush in Michigan, and the documents used allow for a concrete consideration and review. Regrettably, Bush's Michigan speech relies on a selective misrepresentation of test score data. Further, he offers no evidence that the purported test score gains were caused by the recommended reforms. Other viable explanations, such as a major investment in class-size reduction and a statewide reading program, receive no or little attention. Moreover, the presentation ignores less favorable findings, while evidence showing limited or negative effects of the proposed strategies is omitted. Considering the overwhelming evidence that retention is ineffective (if not harmful), it is troubling to see Mr. Bush endorse such an approach. Finally, Florida's real problems of inequitable and inadequate education remain unaddressed.

"In the debate over charter schools, KIPP (Knowledge is Power Program) schools are hailed by charter advocates as illustrative of what these alternatives to public schools can produce. With KIPP, poverty need not impede academic success. Enroll students from economically impoverished backgrounds in a "no excuses" school like KIPP and their chances of attaining academic success would soar markedly. There, neither hunger, poor health, relentless stress, lack of access to the material sustenance and cultural experiences available to students from more affluent homes, nor other adverse effects of poverty are impediments to learning and the attainment of good test scores. If only poor youngsters were not in the nothing-but-excuses public schools where they are taught by nothing-but-excuses teachers. So the story goes and so it was conveyed to me by a KIPP schools manager who, in an oped exchange, presented what the chain considers its best supporting evidence. Whether this evidence actually makes the case for KIPP I will discuss below"

There's a time and place for "show me the data"! "What's the 'evidence'?" is one of our five hallowed habits of mind at Mission Hill and Central Park East. But "evidence" comes in many forms, and the trade-offs involved are part of the data, too-if we pay equal attention to them. Maybe "At what price?" should be the 6th "habit of mind." As I've said before, we're entering an era reminiscent of bad science fiction where everyone is wondering "Who's following me? Who's collecting the data on me? Is there no place to hide?"

In 1981, Chile introduced nationwide school choice by providing vouchers to any student wishing to attend private school. As a result, more than 1000 private schools entered the market, and the private enrollment rate increased by 20 percentage points, with greater impacts in larger, more urban, and wealthier communities. We use this differential impact to measure the effects of unrestricted choice on educational outcomes. Using panel data for about 150 municipalities, we find no evidence that choice improved average educational outcomes as measured by test scores, repetition rates, and years of schooling. However, we find evidence that the voucher program led to increased sorting, as the "best" public school students left for the private sector.

In The Louisiana Recovery School District: Lessons for the Buckeye State, the Thomas B. Fordham Institute criticizes local urban governance structures and presents the decentralized, charter-school-driven Recovery School District (RSD) in New Orleans as a successful model for fiscal and academic performance. Absent from the review is any consideration of the chronic under-funding and racial history of New Orleans public schools before Hurricane Katrina, and no evidence is provided that a conversion to charter schools would remedy these problems. The report also misreads the achievement data to assert the success of the RSD, when the claimed gains may be simply a function of shifting test standards. The report also touts the replacement of senior teachers with new and non-traditionally prepared teachers, but provides no evidence of the efficacy of this practice. Additionally, the report claims public support for the reforms, but other indicators-never addressed in the report-reveal serious concerns over access, equity, performance, and accountability. Ultimately, the report is a polemic advocating the removal of public governance and the replacement of public schools with privately operated charter networks. It is thin on data and thick on claims, and should be read with great caution by policymakers in Ohio and elsewhere.

Randomized trials are a common way to provide rigorous evidence on the impacts of education programs. This article discusses the trade-offs associated with study designs that involve random assignment of students within schools and describes the experience from one such study of Teach for America (TFA). The TFA experiment faced challenges with recruitment, randomization of students, and analysis. The solutions to those challenges may be instructive for experimenters who wish to study future interventions at the student or classroom level. The article concludes that within-school random assignment studies such as the TFA evaluation are challenging but, under the right conditions, are also feasible and potentially very rewarding in terms of generating useful evidence for policy.

Over the past two decades, state and federal education policies have tried to hold schools more accountable for educating their students. A common criticism of these policies is that they may induce schools to "game the system" with strategies such as excluding certain types of students from computation of school average test scores. In this post, based on our recent New York Fed staff report, "Vouchers, Responses, and the Test Taking Population: Regression Discontinuity Evidence from Florida," we investigate whether Florida schools resorted to such strategic behavior in response to a voucher program. We find some evidence that Florida's schools strategically reclassified weak students into exempt categories, and we draw some lessons that are applicable to New York City's education policies.

While numerous recent authors have studied the effects of school accountability systems on student test performance and school "gaming" of accountability incentives, there has been little attention paid to substantive changes in instructional policies and practices resulting from school accountability. The lack of research is primarily due to the unavailability of appropriate data to carry out such an analysis. This paper brings to bear new evidence from a remarkable five-year survey conducted of a census of public schools in Florida, coupled with detailed administrative data on student performance. We show that schools facing accountability pressure changed their instructional practices in meaningful ways. In addition, we present medium-run evidence of the effects of school accountability on student test scores, and find that a significant portion of these test score gains can likely be attributed to the changes in school policies and practices that we uncover in our surveys.

The demand seems so reasonable: Evaluate teachers on the basis of how much their students have learned. After all, if schools exist to educate, then what's wrong with asking for evidence that they are successful? It's a fair question. The problem is agreeing on what kind of evidence to accept.

A proponent of evidence-based policymaking, Ladd shows that our current school reform policies are not based on evidence. She cites research demonstrating that the gaps between the most affluent and the least affluent children more than doubled in the past half-century, and that the income-based achievement gap is now much larger than the racial achievement gap. She points out that the racial achievement gap narrowed from the 1970s to the 1980s and has remained stagnant since then.

Faced with mounting policy pressures from federal and state accountability programs, school leaders are reallocating curricula, time, even diet in an attempt to boost student achievement. To explore whether they are using test score data to reallocate their teacher resources as well, I designed a cross-case, cross-sectional study and explored principals' reported staffing practices in one higher performing and one lower performing elementary school in each of five Florida school districts. Findings show that school leaders are "staffing to the test" by hiring, moving, and developing teachers in an effort to increase their schools' overall performance. The paper discusses the implications of evidence-based staffing for policy, practice and future research.

Unlike in elementary schools, high school teacher effects may be confounded with unobserved track-level treatments (such as the AVID program) that are correlated with individual teachers. I present a strategy that exploits detailed course-taking information to credibly estimate the effects of 9th grade Algebra and English teachers on test scores. I document substantial bias due to track-specific treatments and I show that traditional tests for the existence of teacher effects are flawed. After accounting for bias, I find sizable algebra teacher effects and little evidence of English teacher effects. I find little evidence of teacher spillovers across subjects.

How did the largest and most aggressive federal education initiative, No Child Left Behind (NCLB), garner bipartisan support under George W. Bush, and how have the central tenets of that legislation-accountability, standards, and testing-maintained and increased their value as approaches to education reform despite the tremendous evidence that they do not work?

There is a widespread consensus among practitioners, researchers, and policy makers that current teacher evaluation systems in most school districts do little to help teachers improve or to support personnel decision making. For this reason, new approaches to teacher evaluation are being developed and tested.  There is also a growing consensus that evidence of teachers' contributions to student learning should be a component of teacher evaluation systems, along with evidence about the quality of teachers' practice. Value-added models (VAMs) for examining gains in student test scores from one year to the next are promoted as tools to accomplish this goal. Policy makers can benefit from research about what these models can and cannot do, as well as from research about the effects of other approaches to teacher evaluation. This background paper addresses both of these important concerns.