Group items tagged

Isolating the effect of a given teacher on student achievement (value-added modeling) is complicated when the student is taught the same subject by more than one teacher. We consider three methods, which we call the Partial Credit Method, Teacher Team Method, and Full Roster Method, for estimating teacher effects in the presence of co-teaching. The Partial Credit Method apportions responsibility between teachers according to the fraction of the year a student spent with each. This method, however, has practical problems limiting its usefulness. As alternatives, we propose two methods that can be more stably estimated based on the premise that co-teachers share joint responsibility for the achievement gains of their shared students. The Teacher Team Method uses a single record for each student and a set of variables for each teacher or group of teachers with shared students, whereas the Full Roster Method contains a single variable for each teacher, but multiple records for shared students. We explore the properties of these two alternative methods and then compare the estimates generated using student achievement and teacher roster data from a large urban school district. We find that both methods produce very similar point estimates of teacher value added. However, the Full Roster Method better maintains the links between teachers and students and can be more robustly implemented in practice. 

...a large section of our report summarizes the relevant methods - those rigorous and appropriate designs that should be applied to the questions at hand, but are noticeably absent even at the most cursory level in Roza and Hill's materials. To save you all the trouble of actually reading our entire brief, I've copied and pasted below the section of our brief where we address relevant methods

Children in New York City who learned to read using an experimental curriculum that emphasized nonfiction texts outperformed those at other schools that used methods that have been encouraged since the Bloomberg administration's early days, according to a new study to be released Monday. For three years, a pilot program tracked the reading ability of approximately 1,000 students at 20 New York City schools, following them from kindergarten through second grade. Half of the schools adopted a curriculum designed by the education theorist E. D. Hirsch Jr.'s Core Knowledge Foundation. The other 10 used a variety of methods, but most fell under the definition of "balanced literacy," an approach that was spread citywide by former Schools Chancellor Joel I. Klein, beginning in 2003.

In a series of recent blog posts and in a forthcoming article I have discussed the potential problems with using bad, versus entirely inappropriate measures for determining teacher effectiveness.  I have pointed out, for example, that using value-added measures to estimate teacher effectiveness and then determine whether a teacher should be denied tenure, or have their tenure removed might raise due process concerns which arise from the imprecision and potential outright inaccuracy of teacher effectiveness estimates derived from such methods. I have also explained that in some states like New Jersey, which have adopted Student Growth Percentile measures as an evaluation tool, that where those measures are used as a basis for dismissing teachers, teachers (or their attorney's) might simply rely on the language of the authors of those methods to point out that they are not designed to, nor were they intended to attribute responsibility for the measured student growth to the teacher. Where attribution of responsibility is off the table the dismissing a teacher on an assumption of ineffectiveness based on these measures is entirely inappropriate, and a potential violation of the teacher's due process rights. But, the problem is that state legislatures are increasingly mandating that these measures absolutely be used when making high stakes personnel decisions.

A keystone of this reform movement is the replacement of subjective evaluation with quantifiable measures of each teacher's effectiveness. The quantitative method is known as value-added modeling (VAM), a statistical analysis of student scores that seeks to identify how much an individual teacher contributes to a pupil's progress over the years. The use of VAM in teacher evaluations is growing, but the method remains extremely controversial. Critics often claim that it does not and cannot measure actual teacher quality. This paper addresses that claim. Part I analyzes data from Florida public schools to show that a VAM score in a teacher's third year is a good predictor of that teacher's success in his or her fifth year. Having established that VAM is a useful predictive tool, Part II of the paper addresses the most effective ways that VAM can be used in tenure reform."

There is a method to his madness. Bloomberg and his Chancellor Joel Klein have initiated shut down or initiated the closing of more than 100 public schools, many of which have deep roots in their communities. No two situations are exactly alike. Nonetheless, here is a handy template to go by if you are a mayor who is eager to break up large public schools and hand over their buildings to privately run charter school operations, but don't want to leave your fingerprints at the scene of the crime

Most readers of the NJDOE report on reforming the state's school finance formula likely glided right past the seemingly innocuous recommendation to shift the enrollment count method for funding from a fall enrollment count to an average daily attendance figure. After all, on its face, the argument provided seems to make sense. Let's fund on this basis so that we can incentivize increased attendance in our most impoverished and low performing districts. (Another argument I've heard in other states is "why would we fund kids who aren't there?"). The data were even presented to validate that attendance rates are lower in these districts (Figure 3.1). I, however, could not let this pass, because Average Daily Attendance as a basis for funding is actually a well understood trick of the trade for reducing aid to districts and schools with higher poverty and minority concentrations.  I have both blogged about this topic in the past, and written published research directly and indirectly related to the topic.[1] The intent of this blog post is to provide a (very limited, oversimplified) primer on the common methods of counting general student populations for purposes of determining state aid to schools (charter and district) and to provide some commentary on the pros and cons of each.

The Shock Doctrine, as articulated by journalist Naomi Klein, describes the process by which corporate interests use catastrophes as instruments to maximize their profit. Sometimes the events they use are natural (earthquakes), sometimes they are human-created (the 9/11 attacks) and sometimes they are a bit of both (hurricanes made stronger by human-intensified global climate change). Regardless of the particular cataclysm, though, the Shock Doctrine suggests that in the aftermath of a calamity, there is always corporate method in the smoldering madness - a method based in Disaster Capitalism. Though Klein's book provides much evidence of the Shock Doctrine, the Disaster Capitalists rarely come out and acknowledge their strategy. That's why Watkins' outburst of candor, buried in this front-page New York Times article yesterday, is so important: It shows that the recession and its corresponding shock to school budgets is being  used by corporations to maximize revenues, all under the gauzy banner of "reform."

Interest in Student Growth Modeling (SGM) and Value Added Modeling (VAM) arises from educators concerned with measuring the effectiveness of teaching and other school activities through changes in student performance as a companion and perhaps even an alternative to status. Several formal statistical models have been proposed for year-to-year growth and these fall into at least three clusters: simple change (e.g., differences on a vertical scale), residualized change (e.g., simple linear or quantile regression techniques), and value tables  (varying salience of different achievement level outcomes across two years). Several of these methods have been implemented by states and districts.  This paper reviews relevant literature and reports results of a data-based comparison of six basic SGM models that may permit aggregating across teachers or schools to provide evaluative information.  Our investigation raises some issues that may compromise current efforts to implement VAM in teacher and school evaluations and makes suggestions for both practice and research based on the results.

"Most of the attention to the methodological shortcomings of the law focuses on "adequate yearly progress" (AYP) - the crude requirement that all schools must make "adequate progress" toward the goal of 100 percent proficiency by 2014. And AYP is indeed an inept measure. But the problems are actually much deeper than AYP. Rather, it's the underlying methods and assumptions of NCLB (including AYP) that have had a persistent, negative impact on the way we interpret testing data. I'm not going to get into all of the details here, since I discuss them so frequently (follow the links below), but the most important issues include:"

"Academic tracking in secondary education appears to confound an increasingly common method for gauging differences in teacher quality, according to two recently released studies. Failing to account for how students are sorted into more- or less-rigorous classes-as well as the effect different tracks have on student learning-can lead to biased "value added" estimates of middle and high school teachers' ability to boost their students' standardized-test scores, the papers conclude."

"Top researchers studying new "value added" or "growth index" models for measuring a teacher's contribution to student achievement completely agree on only one thing: These methods should be used in staff-evaluation systems with more care than they have been so far."

There is a dearth of research that supports paying teachers beyond their base salaries to improve student achievement, but there is a broad body of research that indicates that pay for performance might actually do damage as teachers feel a threat to their livelihoods because of this narrow method of measuring their efficacy. Pay for performance has been documented as compromising the good will and cooperation among teachers since it creates competition for a small amount of money, which can result in an "I'm out for myself only" attitude. Such a tone can hurt the necessary collaboration and communication found to nurture student achievement and success.

Three recent reports on the Milwaukee Parental Choice Program (MPCP), produced by the School Choice Demonstration Project at the University of Arkansas use largely sound methods, but the data they assemble provide little to support the 22-year-old school voucher program. Those are the conclusions of three separate reviews released today of the reports.

The SAS Educational Value-Added Assessment System (SAS® EVAAS®) is the most widely used value-added system in the country. It is also self-proclaimed as "the most robust and reliable" system available, with its greatest benefit to help educators improve their teaching practices. This study critically examined the effects of SAS® EVAAS® as experienced by teachers, in one of the largest, high-needs urban school districts in the nation - the Houston Independent School District (HISD). Using a multiple methods approach, this study critically analyzed retrospective quantitative and qualitative data to better comprehend and understand the evidence collected from four teachers whose contracts were not renewed in the summer of 2011, in part given their low SAS® EVAAS® scores. This study also suggests some intended and unintended effects that seem to be occurring as a result of SAS® EVAAS® implementation in HISD. In addition to issues with reliability, bias, teacher attribution, and validity, high-stakes use of SAS® EVAAS® in this district seems to be exacerbating unintended effects.

As the closing of "failing" schools becomes an annual ritual, along with the opening of brand-new schools (some of which will eventually join the ranks of "failing" schools), it is time to ask about where accountability truly lies. I wonder if  it ever occurs to anyone in the New York City Department of Education that their own policies of closing schools and shuffling low-performing students around like checker pieces on a checker board have actually created "failing" schools. Every time they close a large high school with large numbers of low-performing students, those students are then pushed off into another large high school (like Dewey) that is doomed to "fail." Why doesn't the leadership of the DOE ever take responsibility for helping schools that have disproportionate numbers of students who enter ninth grade with low test scores, including students with disabilities, homeless students, and students who are English language learners? Their methods of "reform" look like 52-pickup: Just throw the cards in the air and hope that somehow you come up with a winning hand.

A new report from Stanford University researcher Linda Darling-Hammond details what the components of a comprehensive teacher evaluation system should look like at a time when such assessments have become one of the most contentious debates in education today. Much of the controversy swirls around the growing trend of using students' standardized test scores over time to help assess teacher effectiveness. This "value-added" method of assessment - which involves the use of complicated formulas that supposedly evaluate how much "value" a teacher adds to a student's achievement - is considered unreliable and not valid by many experts, though school reformers have glommed onto it with great zeal.

Virtually everyone agrees that teacher evaluation in the United States needs an overhaul. Existing systems rarely help teachers improve or clearly distinguish those who are succeeding from those who are struggling. The tools that are used do not always represent the important features of good teaching. Criteria and methods for evaluating teachers vary substantially across districts and at key career milestones-when teachers complete pre-service teacher education, become initially licensed, are considered for tenure, and receive a professional license.  A comprehensive system should address these purposes in a coherent way and provide support for supervision and professional learning, identify teachers who need additional assistance and-in some cases-a change of career, and recognize expert teachers who can contribute to the learning of their peers. This report outlines an integrated approach that connects these goals to a teaching-career continuum and a professional development system that supports effectiveness for all teachers at every stage of their careers.

Sixteen years ago, Charlotte Danielson, an Oxford-trained economist, developed a description of good teaching that became the foundation for attempts by federal and state officials and school districts to quantify teacher performance. The Danielson method - articulated in her book, "Enhancing Professional Practice: A Framework for Teaching" (ASCD, 1996) - describes good teaching using numerous criteria within four broad areas of performance: the quality of questions and discussion techniques; a knowledge of students' special needs; the expectations set for learning and achievement; and the teacher's involvement in professional development activities. "If all you do is judge teachers by test results," Ms. Danielson told Ginia Bellafante in an interview for a Big City column in the Metropolitan section of The New York Times last month, "it doesn't tell you what you should do differently."

On occasion, students and reporters ask me what makes me trust or distrust folks who claim to be education researchers, and it's a harder question to answer than one might think. As an historian with some quantitative training, I am eclectic on methods-I have no purity test other than "the evidence and reasoning have to fit the conclusion." It's not the existence of error: even great researchers make occasional errors, and it's a good thing in the long run for researchers to take intellectual risks (which imply likely error/failure). Further, we all have the various myside biases cognitive psychologists write about. But when I come across something like the following produced by the Cato Institute's Andrew Coulson and displayed by Matthew Ladner twice on Jay Greene's blog (including on Thursday), I start to wonder.