Group items tagged

In a previous post, I compared value-added (VA) and classroom observations in terms of reliability - the degree to which they are free of error and stable over repeated measurements. But even the most reliable measures aren't useful unless they are valid - that is, unless they're measuring what we want them to measure. Arguments over the validity of teacher performance measures, especially value-added, dominate our discourse on evaluations. There are, in my view, three interrelated issues to keep in mind when discussing the validity of VA and observations. The first is definitional - in a research context, validity is less about a measure itself than the inferences one draws from it. The second point might follow from the first: The validity of VA and observations should be assessed in the context of how they're being used. Third and finally, given the difficulties in determining whether either measure is valid in and of itself, as well as the fact that so many states and districts are already moving ahead with new systems, the best approach at this point may be to judge validity in terms of whether the evaluations are improving outcomes. And, unfortunately, there is little indication that this is happening in most places.

Chicago mayor, Rahm Emanuel has taken a page from the Koch Bros. book of tricks by using the "research says" tactic to push his longer-school-day campaign on resistant city schools. As I have shown numerous times on this blog and elsewhere, there is no reliable or valid research to support Rahm's claim that more seat time in school produces better learning outcomes. But buoyed by support from corporate reform groups like Stand For Children, the mayor's publicists at CPS, like Becky Carroll and his hand-picked CEO J.C. Brizard, continue to claim that there are studies to validate this obvious political and anti-union agenda.

This second report from the Measures of Effective Teaching (MET) project offers ground-breaking descriptive information regarding the use of classroom observation instruments to measure teacher performance. It finds that observation scores have somewhat low reliabilities and are weakly though positively related to value-added measures. Combining multiple observations can enhance reliabilities, and combining observation scores with student evaluations and test-score information can increase their ability to predict future teacher value-added. By highlighting the variability of classroom observation measures, the report makes an important contribution to research and provides a basis for the further development of observation rubrics as evaluation tools. Although the report raises concerns regarding the validity of classroom observation measures, we question the emphasis on validating observations with test-score gains. Observation scores may pick up different aspects of teacher quality than test-based measures, and it is possible that neither type of measure used in isolation captures a teacher's contribution to all the useful skills students learn. From this standpoint, the authors' conclusion that multiple measures of teacher effectiveness are needed appears justifiable. Unfortunately, however, the design calls for random assignment of students to teachers in the final year of data collection, but the classroom observations were apparently conducted prior to randomization, missing a valuable opportunity to assess correlations across measures under relatively bias-free conditions.

A federal district court in Florida has ruled that parents of a student with autism have stated a valid claim that charter school employees' use of excessive force constituted a violation of the student's substantive due process rights. The court concluded that the parents had alleged facts sufficient to show: (1) use of excessive force that "shocks the conscience;" and (2) the charter school employees acted under color of state law. It found, however, that the parents' allegations of emotional injuries and psychological harm were insufficient to rise to the level of conduct that shocks the conscience.

The resolution commends Scott for "his concern about the overemphasis on high stakes testing" and says the board believes "our state's future prosperity relies on a high-quality education system that prepares students for college and careers," not just their ability to jump through the state's hoops. And it asks the state Legislature to "reexamine the public school accountability system in Texas and to develop a system that encompasses multiple assessments, reflects greater validity, uses more cost efficient sampling techniques and other external evaluation arrangements."

Reviewers Guarino and Stacy question the emphasis placed on validating classroom observations with test score gains. Observation scores may pick up different aspects of teacher quality than do test-based measures. It is possible that neither type of measure used in isolation captures a teacher's contribution to all the useful skills that students learn in schools. From this standpoint, the authors' conclusion that multiple measures of teacher effectiveness are needed is justifiable. The omission of relevant information is a shortcoming of the report. Key details regarding the study design and methodological approach are lacking.

This post includes a few more preliminary musings regarding the use of value-added measures and student growth percentiles for teacher evaluation, specifically for making high-stakes decisions, and especially in those cases where new statutes and regulations mandate rigid use/heavy emphasis on these measures, as I discussed in the previous post.

The SAS Educational Value-Added Assessment System (SAS® EVAAS®) is the most widely used value-added system in the country. It is also self-proclaimed as "the most robust and reliable" system available, with its greatest benefit to help educators improve their teaching practices. This study critically examined the effects of SAS® EVAAS® as experienced by teachers, in one of the largest, high-needs urban school districts in the nation - the Houston Independent School District (HISD). Using a multiple methods approach, this study critically analyzed retrospective quantitative and qualitative data to better comprehend and understand the evidence collected from four teachers whose contracts were not renewed in the summer of 2011, in part given their low SAS® EVAAS® scores. This study also suggests some intended and unintended effects that seem to be occurring as a result of SAS® EVAAS® implementation in HISD. In addition to issues with reliability, bias, teacher attribution, and validity, high-stakes use of SAS® EVAAS® in this district seems to be exacerbating unintended effects.

A new report from Stanford University researcher Linda Darling-Hammond details what the components of a comprehensive teacher evaluation system should look like at a time when such assessments have become one of the most contentious debates in education today. Much of the controversy swirls around the growing trend of using students' standardized test scores over time to help assess teacher effectiveness. This "value-added" method of assessment - which involves the use of complicated formulas that supposedly evaluate how much "value" a teacher adds to a student's achievement - is considered unreliable and not valid by many experts, though school reformers have glommed onto it with great zeal.

Unlike the Texas miracle, the Harvard-Columbia revelations are not based on fraudulent numbers. But what is deeply problematic is the spin that the authors give to their findings. The study examined the incomes of adults who, as children in the 4th through the 8th grades, had teachers of different "Value Added" scores, with Value Added defined as improvement in the scores of students on standardized tests. The study claims that the individuals who had excellent teachers as children have higher incomes as adults; we will examine the validity of this claim below. But first we must ask what these higher incomes mean. When they were children, these individuals were poor. What the H-C authors fail to mention is that even when they had excellent teachers as children and therefore have higher incomes as adults, these individuals, despite their higher incomes, remain poor.

Ms. Moloney has been testing a new framework for evaluating teachers this year at the school, which is actually in Brighton Beach, after receiving training over the summer. It was designed by Charlotte Danielson who wrote a common-sense framework to help both teachers and administrators identify good teaching. It's similar to a tool kit, with 22 strategies every teacher should master. The city is trying out the Danielson framework at 107 schools to learn how much training principals need so they can become certified evaluators once the state's evaluation system goes into effect, said Kirsten Busch, executive director of the Office of Teacher Effectiveness. The city has until next January to negotiate an evaluation system with its teachers' union. At P.S. 100, Ms. Moloney and her teachers believe classroom observations are much more valid than a controversial rating system the city used that was based solely on student progress on state exams.

A new study by two French researchers published in the Journal of Psychology: General shows how telling students that failure is a natural element of learning -- instead of pressuring them to succeed -- may increase their academic performance. "Teachers should not hesitate to tell children that what they're going to do is very difficult," said author Jean-Claude Croizet, a University of Poitiers professor. He conducted the study with Poitiers postdoctoral student Frederique Autin. The study's findings, publicized by the American Psychological Association, come amid mounting cries against high-stakes standardized tests in the U.S. As more and more states seek to tie students' standardized test scores to teacher evaluations, statisticians often question the validity of those exams. According to Croizet and Autin, high-stakes test trigger a psychological mechanism and lack of confidence that makes it harder to assess aptitude.

The use of rate changes is still proliferating rapidly at all levels of our education system. These measures, which play an important role in the provisions of No Child Left Behind, are already prominent components of many states' core accountability systems (e..g, California), while several others will be using some version of them in their new, high-stakes school/district "grading systems." New York State is awarding millions in competitive grants, with almost half the criteria based on rate changes. District consultants issue reports recommending widespread school closures and reconstitutions based on these measures. And, most recently, U.S. Secretary of Education Arne Duncan used proficiency rate increases as "preliminary evidence" supporting the School Improvement Grants program. Meanwhile, on the public discourse front, district officials and other national leaders use rate changes to "prove" that their preferred reforms are working (or are needed), while their critics argue the opposite. Similarly, entire charter school sectors are judged, up or down, by whether their raw, unadjusted rates increase or decrease. So, what's the problem? In short, it's that year-to-year changes in proficiency rates are not valid evidence of school or policy effects. These measures cannot do the job we're having them do, even on a limited basis. This really has to stop.

You know what one of my great pet peeves is? When prominent people, who are granted a prominent place in our society's discourse, cite "research" without telling us what that research is. Case in point: Newark Superintendent of Schools Cami Anderson: Research shows that effective teachers put students on an entirely different life trajectory - toward college, a higher salary, even a more stable family life. I am committed to ensuring that we have a strong teacher in every classroom and great leader in every school. Based on my 20-plus years in education, I know we must significantly change how we recruit, select, develop and retain our educators. [...] Some research shows that we lose our best teachers to charter schools and other professions because they feel they are not growing and they become disheartened seeing students in ineffective classrooms. After multiple poor ratings validated by several people, we should presume that these few teachers are ineffective and partner with the union to manage them out - efficiently. [emphasis mine] I would dearly love to see this "research." I would love to evaluate it for myself and decide whether it's think-tanky nonsense or serious work done by serious people. But I can't, can I? Because Anderson won't tell me what it is, and the Star-Ledger thinks it's enough for her to cite it without checking it for themselves.

I have written much on this blog about problems with the use of Value-added Estimates of teacher effect (used loosely) on student test score gains on this blog. I have addressed problems with both the reliability and validity of VAM estimates, and I have pointed out how SGP based estimates of student growth are invalid on their face for determining teacher effectiveness. But, I keep hearing two common refrains from the uber-reformy (those completely oblivious to the statistics and research of VAM while also lacking any depth of understanding of the complexities of the social systems [schools] into which they propose to implement VAM as a de-selection tool) crowd. Sadly, these are the people who seem to be drafting policies these days.

A memo has recently surfaced in which the New York State Department of Education appears to encourage educators to mislead students about upcoming standardized field tests meant to "provide the data necessary to ensure the validity and reliability of the New York State Testing program." "Students should not be informed of the connection between these field tests and State assessments," the memo reads. "The field tests should be described as brief tests of achievement in the subject."

Redistributing highly effective teachers from low- to high-need schools is an education policy tool that is at the center of several major current policy initiatives. The underlying assumption is that  teacher productivity is portable across different schools settings. Using elementary and secondary school data from North Carolina and Florida, this paper investigates the validity of this assumption. Among teachers who switched between schools with substantially different poverty levels or academic performance levels, we find no change in those teachers' measured effectiveness before and after a school change. This pattern holds regardless of the direction of the school change. We also find that high-performing teachers' value-added dropped and low-performing teachers' value-added gained in the post-move years, primarily as a result of regression to the within-teacher mean and unrelated to school setting changes. Despite such shrinkages, high-performing teachers in the pre-move years still outperformed low-performing teachers after moving to schools with different settings.

"Bruce Baker has another brilliant analysis, this time gauging the validity of school ratings just released by the state of New York. A thumbnail sketch: New York is stiffing its neediest schools and districts. Here are the takeaways"

The Center For American Progress has published another report justifying the firing of teachers today, based on statistical models that may some day become valid. "Designing High Quality Evaluation Systems," by John Tyler, recounts the standard reasons why of educators do not trust high-stakes test-driven algorithms, and even contributes a couple of new insights into problems that are unique to high school test scores. An urban teacher reading Tyler's evidence would likely conclude that he has written an ironclad indictment of value-added models for high-stakes purposes. But, as is usually true of CAP's researchers, he concludes that the work of economists in improving value-added models is so impressive that education will benefit from their experiments if educators don't blow it.

The recent spate of cheating scandals in cities like Atlanta, Chicago, Los Angeles, and Washington presents an interesting conundrum. Those opposed to education reform schemes tied to the evaluation of student test scores and teacher compensation, or "value added" evaluation, claim that the teachers and administrators who were caught cheating were the victims, compelled to cheat out of fear for their livelihoods. On the other hand, value-added advocates solemnly pronounce that there is no excuse for cheating and that, moreover, cheating teachers and administrators provide the very evidence that reform is necessary. Both positions are valid. Can we work our way out?