Group items tagged

This second report from the Measures of Effective Teaching (MET) project offers ground-breaking descriptive information regarding the use of classroom observation instruments to measure teacher performance. It finds that observation scores have somewhat low reliabilities and are weakly though positively related to value-added measures. Combining multiple observations can enhance reliabilities, and combining observation scores with student evaluations and test-score information can increase their ability to predict future teacher value-added. By highlighting the variability of classroom observation measures, the report makes an important contribution to research and provides a basis for the further development of observation rubrics as evaluation tools. Although the report raises concerns regarding the validity of classroom observation measures, we question the emphasis on validating observations with test-score gains. Observation scores may pick up different aspects of teacher quality than test-based measures, and it is possible that neither type of measure used in isolation captures a teacher's contribution to all the useful skills students learn. From this standpoint, the authors' conclusion that multiple measures of teacher effectiveness are needed appears justifiable. Unfortunately, however, the design calls for random assignment of students to teachers in the final year of data collection, but the classroom observations were apparently conducted prior to randomization, missing a valuable opportunity to assess correlations across measures under relatively bias-free conditions.

In recent years, interest has grown in using classroom observation as a means to several ends, including teacher development, teacher evaluation, and impact evaluation of classroom-based interventions. Although education practitioners and researchers have developed numerous observational instruments for these purposes, many developers fail to specify important criteria regarding instrument use. In this article, the authors argue that for classroom observation to succeed in its aims, improved observational systems must be developed. These systems should include not only observational instruments but also scoring designs capable of producing reliable and cost-efficient scores and processes for rater recruitment, training, and certification. To illustrate how such a system might be developed and improved, the authors provide an empirical example that applies generalizability theory to data from a mathematics observational instrument.

Although most new teacher evaluations are still in various phases of pre-implementation, it's safe to say that classroom observations and/or value-added (VA) scores will be the most heavily-weighted components toward teachers' final scores, depending on whether teachers are in tested grades and subjects. One gets the general sense that many - perhaps most - teachers strongly prefer the former (observations, especially peer observations) over the latter (VA). One of the most common arguments against VA is that the scores are error-prone and unstable over time - i.e., that they are unreliable. And it's true that the scores fluctuate between years (also see here), with much of this instability due to measurement error, rather than "real" performance changes. On a related note, different model specifications and different tests can yield very different results for the same teacher/class. These findings are very important, and often too casually dismissed by VA supporters, but the issue of reliability is, to varying degrees, endemic to all performance measurement. Actually, many of the standard reliability-based criticisms of value-added could also be leveled against observations. Since we cannot observe "true" teacher performance, it's tough to say which is "better" or "worse," despite the certainty with which both "sides" often present their respective cases. And, the fact that both entail some level of measurement error doesn't by itself speak to whether they should be part of evaluations.*

In a previous post, I compared value-added (VA) and classroom observations in terms of reliability - the degree to which they are free of error and stable over repeated measurements. But even the most reliable measures aren't useful unless they are valid - that is, unless they're measuring what we want them to measure. Arguments over the validity of teacher performance measures, especially value-added, dominate our discourse on evaluations. There are, in my view, three interrelated issues to keep in mind when discussing the validity of VA and observations. The first is definitional - in a research context, validity is less about a measure itself than the inferences one draws from it. The second point might follow from the first: The validity of VA and observations should be assessed in the context of how they're being used. Third and finally, given the difficulties in determining whether either measure is valid in and of itself, as well as the fact that so many states and districts are already moving ahead with new systems, the best approach at this point may be to judge validity in terms of whether the evaluations are improving outcomes. And, unfortunately, there is little indication that this is happening in most places.

"The modernization of teacher evaluation systems, an increasingly common component of school reform efforts, promises to reveal new, systematic information about the performance of individual classroom teachers. Yet while states and districts race to design new systems, most discussion of how the information might be used has focused on traditional human resource-management tasks, namely, hiring, firing, and compensation. By contrast, very little is known about how the availability of new information, or the experience of being evaluated, might change teacher effort and effectiveness. In the research reported here, we study one approach to teacher evaluation: practice-based assessment that relies on multiple, highly structured classroom observations conducted by experienced peer teachers and administrators. While this approach contrasts starkly with status quo "principal walk-through" styles of class observation, its use is on the rise in new and proposed evaluation systems in which rigorous classroom observation is often combined with other measures, such as teacher value-added based on student test scores."

A while back, I noted that states and districts should exercise caution in assigning weights (importance) to the components of their teacher evaluation systems before they know what the other components will be. For example, most states that have mandated new evaluation systems have specified that growth model estimates count for a certain proportion (usually 40-50 percent) of teachers' final scores (at least those in tested grades/subjects), but it's critical to note that the actual importance of these components will depend in no small part on what else is included in the total evaluation, and how it's incorporated into the system. In slightly technical terms, this distinction is between nominal weights (the percentage assigned) and effective weights (the percentage that actually ends up being the case). Consider an extreme hypothetical example - let's say a district implements an evaluation system in which half the final score is value-added and half is observations. But let's also say that every teacher gets the same observation score. In this case, even though the assigned (nominal) weight for value-added is 50 percent, the actual importance (effective weight) will be 100 percent, since every teacher receives the same observation score, and so all the variation between teachers' final scores will be determined by the value-added component.

As teacher evaluations are becoming more prevalent in schools across the country amid a growing debate on how best to grade teachers, a new report out today adds to the growing number of studies concluding that student test scores should be one of several determinants in measuring student effectiveness. Instead, teachers should be assessed based on a combination of classroom observations, student feedback and value-added student achievement gains, according to the Measures of Effective Teaching project's paper, "Gathering Feedback for Teaching." Educators should be observed by certified raters through multiple, high-quality observations with clear standards.

The key element in the agreement reached between the New York City teachers' union and the city's Education Department last Thursday was given a closer look, as The New York Times and Gotham Schools examined the role to be played by so-called "independent validators" in rating teacher effectiveness. As part of the deal reached between the city Education Department and the United Federation of Teachers, independent observers will be brought into New York City's classrooms "to monitor the weakest teachers and provide a second opinion to supplement observations by the school principal," writes Winnie Hu in The Times.

This study examined five instruments used to assess the effectiveness of teacher practices based on classroom observations. The study first examined whether observers could reliably assess teachers with each instrument, and then examined how well each instrument, along with other information, predicted student achievement.

"In this study, we test the idea that new charters hit their mark early in their operations and do not vary much after that. The notion originated from time spent in young charter schools studying their experience as new organizations and as agents of education reform. Interviews with school staff along with our own observations of school activities and operations have formed the impression that the "rules" of a school get set early on in the life of the school. By rules, we mean the adult and student cultures, the formal and informal procedures for identifying and addressing problems, and the school community's commitment to student learning as the primary focus of the school. These are obviously richly nuances facets of a school with myriad potential interactions among them, just as with any social organization. Yet, however they come about, we have observed that they are shaped quickly in new schools. Moreover, these norms and behaviors are sturdy and difficult (though not impossible) to change later on."

Reviewers Guarino and Stacy question the emphasis placed on validating classroom observations with test score gains. Observation scores may pick up different aspects of teacher quality than do test-based measures. It is possible that neither type of measure used in isolation captures a teacher's contribution to all the useful skills that students learn in schools. From this standpoint, the authors' conclusion that multiple measures of teacher effectiveness are needed is justifiable. The omission of relevant information is a shortcoming of the report. Key details regarding the study design and methodological approach are lacking.

Teachers can expected unannounced observations to factor into their annual ratings under the terms of the evaluations agreement that Gov. Andrew Cuomo announced today. The unannounced observations are one of several ways that the State Education Department and state teachers union, NYSUT, agreed to flesh out the state's 2010 evaluation law, seen as so open-ended as to stymie implementation. The agreement, which Cuomo is set to turn into law through the state budget amendment process, resolves some major points of contention while continuing to leave many elements of districts' evaluation system subject to local collective bargaining. Districts and their unions have until the end of 2012 to turn the framework into a local evaluation system, or risk losing state aid.

The New York City teachers' union has long called the process used by the city's Education Department for reviewing and dismissing struggling teachers partisan and unfair. But now, as part of an agreement reached Thursday, the Education Department and the United Federation of Teachers will put into effect an evaluation system that will bring independent observers into the city's classrooms to monitor the weakest teachers.

In recent years, the casual observer of debates over public education policy might be led to believe that improving teacher quality and ensuring that low income and minority school children have access to high quality teachers has little or nothing to do with the equity or adequacy of financing of schools. The casual observer might be led to believe that there actually exists a sizable body of empirical research that confirms a) that high quality teaches matter, b) that money doesn't matter and c) by extension money has nothing to do with recruiting, retaining or redistributing teacher quality. These arguments, while politically convenient for those hoping to avoid thorny questions of tax policy and state aid formulas, are not actually grounded in any body of decisive, empirical research. Rather, to the contrary, it is reasonably well understood that while teacher quality does indeed matter, teacher wages also matter and teacher working conditions matter, both in terms of the level of quality of the overall teacher workforce and in the distribution of quality teachers.

One thing that the DOE and the UFT seem to have agreed upon is that the instructional framework developed by consultant Charlotte Danielson is potentially useful and constructive, though they disagree about how these rubrics are being used to evaluate teachers currently in NYC schools. Below are the observations of one teacher about how the Danielson rubric is being applied in his school.

This report is intended for policymakers and practitioners wanting to understand the implications of the Measures of Effective Teaching (MET) project's interim analysis of classroom observations. Those wanting to explore all the technical aspects of the study and analysis also should read the companion research report, available at www.metproject.org.

Both a value-added method and principal observations tied to a teaching framework identified the same teachers as particularly high or low-performing under Chicago's teacher-evaluation pilot, a new study concludes. But principals struggled to provide high-quality "coaching" and support to teachers based on the results, the report says-a finding indicates just how difficult it will be to use the systems to improve teaching and learning.

Few people define themselves as being a member of a special interest. That term applies to the folks on the other side -- the people you disagree with. New Yorkers got more evidence of that this month with the formation of StudentsFirstNY. In a nutshell, the group wants to preserve and extend the education policies of New York City Mayor Michael Bloomberg and battle the teachers union, which has had an increasingly rancorous relationship with Bloomberg. In its mission statement, the group declares, StudentsFirstNY will be New York's leading voice for students who depend on public education for the skills they need to succeed, but who are too often failed by a system that puts special interests, rather than the interests of children, first. Nice sentiments. But the people behind this statement hardly qualify as disinterested observers anymore than the United Federation of Teachers does. The New York StudentsFirst group is an offshoot of the national organization StudentsFirst, created by former Washington, D.C. schools superintendent Michelle Rhee. It includes many who have backed the Bloomberg administration's education policies over the years -- people who even their foes have come to call reformers. The name persists after 10 years of "reformers" running the city's schools and racking up a decidedly mixed record. Whatever they have or have not done for students in New York City and beyond, though, these policies have helped make some people rich and successful.

The massive Measures of Effective Teaching Project is finding that teacher effectiveness assessments similar to those used in some district value-added systems aren't good at showing which differences are important between the most and least effective educators, and often totally misunderstand the "messy middle" that most teachers occupy. Yet the project's latest findings suggest more nuanced teacher tests, multiple classroom observations and even student feedback can all create a better picture of what effective teaching looks like. Researchers dug into the latest wave of findings from the study of more than 3,000 classes for a standing-room-only ballroom at the American Educational Research Association's annual conference here on Saturday.

What effective CMOs do rely on heavily is trust, relationships, and clear communication. In well-run charters, there is a common belief about teaching and learning, and teachers are hired and retained based on whether they share that belief. Teachers know they are getting ongoing feedback and no surprises. They know that the principal doing their observations and evaluations is a master teacher operating on the same definition of good instruction as they are. They know that every other teacher in the building is a potential collaborator. In other words, they trust their coworkers and operate in a culture of common understanding and mutual respect. Evaluation is understood to be more about organizational improvement than about passing judgment on an individual. In fact, some CMOs have tried and dumped merit pay because they felt it disrupted this collaborative culture.