Group items tagged

Education Week reports on the growth of websites that attempt to provide parents with help in choosing schools, including rating schools according to testing results. The most prominent of these sites is GreatSchools.org. Its test-based school ratings could not be more simplistic - they are essentially just percentile rankings of schools' proficiency rates as compared to all other schools in their states (the site also provides warnings about the data, along with a bunch of non-testing information). This is the kind of indicator that I have criticized when reviewing states' school/district "grading systems." And it is indeed a poor measure, albeit one that is widely available and easy to understand. But it's worth quickly discussing the fact that such criticism is conditional on how the ratings are employed - there is a difference between the use of testing data to rate schools for parents versus for high-stakes accountability purposes. In other words, the utility and proper interpretation of data vary by context, and there's no one "correct way" to rate schools. The optimal design might differ depending on the purpose for which the ratings will be used. In fact, the reasons why a measure is problematic in one context might very well be a source of strength in another.

The use of rate changes is still proliferating rapidly at all levels of our education system. These measures, which play an important role in the provisions of No Child Left Behind, are already prominent components of many states' core accountability systems (e..g, California), while several others will be using some version of them in their new, high-stakes school/district "grading systems." New York State is awarding millions in competitive grants, with almost half the criteria based on rate changes. District consultants issue reports recommending widespread school closures and reconstitutions based on these measures. And, most recently, U.S. Secretary of Education Arne Duncan used proficiency rate increases as "preliminary evidence" supporting the School Improvement Grants program. Meanwhile, on the public discourse front, district officials and other national leaders use rate changes to "prove" that their preferred reforms are working (or are needed), while their critics argue the opposite. Similarly, entire charter school sectors are judged, up or down, by whether their raw, unadjusted rates increase or decrease. So, what's the problem? In short, it's that year-to-year changes in proficiency rates are not valid evidence of school or policy effects. These measures cannot do the job we're having them do, even on a limited basis. This really has to stop.

The standard reformy template is that teachers should only be able to get tenure after 3 years of good ratings in a row and that teachers should be subject to losing tenure if they get 2 bad years in a row.  Further, it is possible that the evaluations might actually stipulate that you can only get a good rating if you achieve a certain rating on the quantitative portion of the evaluation - or the VAM score. Likewise for bad ratings (that is, the quantitative measure overrides all else in the system). The premise of the dice rolling activity from my previous post was that it is necessarily much less likely to roll the same number (or subset of numbers) three times in a row than twice (exponentially in fact). That is, it is much harder to overcome the odds based on error rates to achieve tenure, and much easier to lose it. Again, this is much due to the noisiness of the data, and less due to the difficulty of actually being "good" year after year. The ratings simply jump around a lot. See my previous post.

In part 1 I demonstrated there was little correlation between how a teacher was rated in 2009 to how that same teacher was rated in 2010.  So what can be more crazy than a teacher being rated highly effective one year and then highly ineffective the next?  How about a teacher being rated highly effective and highly ineffective IN THE SAME YEAR. I will show in this post how exactly that happened for hundreds of teachers in 2010.  By looking at the data I noticed that of the 18,000 entries in 2010, about 6,000 were repeated names.  This is because there are two ways that one teacher can get multiple value-added ratings for the same year.

Readers of my blog know I'm both a data geek and a skeptic of the usefulness of Value-added data specifically as a human resource management tool for schools and districts. There's been much talk this week about the release of the New York City teacher ratings to the media, and subsequent publication of those data by various news outlets. Most of the talk about the ratings has focused on the error rates in the ratings, and reporters from each news outlet have spent a great deal of time hiding behind their supposed ultra-responsibleness of being sure to inform the public that these ratings are not absolute, that they have significant error ranges, etc.  Matt Di Carlo over at Shanker Blog has already provided a very solid explanatory piece on the error ranges and how those ranges affect classification of teachers as either good or bad. But, the imprecision - as represented by error ranges - of each teacher's effectiveness estimate is but one small piece of this puzzle. And in my view, the various other issues involved go much further in undermining the usefulness of the value added measures which have been presented by the media as necessarily accurate albeit lacking in precision.

"A few weeks ago, Students First NY (SFNY) released a report, in which they presented a very simple analysis of the distribution of "unsatisfactory" teacher evaluation ratings ("U-ratings") across New York City schools in the 2011-12 school year. The report finds that U-ratings are distributed unequally. In particular, they are more common in schools with higher poverty, more minorities, and lower proficiency rates. Thus, the authors conclude, the students who are most in need of help are getting the worst teachers. There is good reason to believe that schools serving larger proportions of disadvantaged students have a tougher time attracting, developing and retaining good teachers, and there is evidence of this, even based on value-added estimates, which adjust for these characteristics (also see here). However, the assumptions upon which this Students First analysis is based are better seen as empirical questions, and, perhaps more importantly, the recommendations they offer are a rather crude, narrow manifestation of market-based reform principles."

A majority of New York City voters approve of the public release of ratings for thousands of public school teachers, even though a plurality of voters believe that the ratings are flawed, a new poll has found. The poll, released early Wednesday by the Quinnipiac University Polling Institute, found that 58 percent of voters approved of the release of the ratings, known as teacher data reports, while 38 percent disapproved and 5 percent were undecided or did not answer.

Education officials lacked confidence in the controversial ratings of teachers who oversaw the city's highest- and lowest-performing students, cautioning schools about considering them in tenure decisions, the Daily News has learned. The city Education Department called superintendents last spring about the shakiness of ratings for teachers at the very top and bottom of the spectrum, which 33% of all ratings, including those of teachers not up for tenure, fell into. And all of the teachers at more than 30 schools fell into this category, a News analysis finds.

America does not have a general education crisis; we have a poverty crisis. Results of an international student assessment indicate that U.S. schools with fewer than 25 percent of their students living in poverty rank first in the world among advanced industrial countries. But when you add in the scores of students from schools with high poverty rates, the United States sinks to the middle of the pack. At nearly 22 percent and rising, the child-poverty rate in the United States is the highest among wealthy nations in the world. (Poverty rates in Denmark and in Finland, which is justifiably celebrated as a top global performer on the Program for International Student Assessment exams, are below 5 percent). In New York City, the child-poverty rate climbed to 30 percent in 2010.

The new evaluation system pretends to be balanced, but it is not. Teachers will be ranked on a scale of 1-100. Teachers will be rated as "ineffective, developing, effective, or highly effective." Forty percent of their grade will be based on the rise or fall of student test scores; the other sixty percent will be based on other measures, such as classroom observations by principals, independent evaluators, and peers, plus feedback from students and parents. But one sentence in the agreement shows what matters most: "Teachers rated ineffective on student performance based on objective assessments must be rated ineffective overall." What this means is that a teacher who does not raise test scores will be found ineffective overall, no matter how well he or she does with the remaining sixty percent. In other words, the 40 percent allocated to student performance actually counts for 100 percent. Two years of ineffective ratings and the teacher is fired.

"Much has been made of late regarding the erroneous classification of 44 teachers in Washington DC as ineffective, thus facing job consequences. This particular erroneous rating was based on an "error" in the calculation of the teachers' total ratings, as acknowledged by the consulting firm applying the ratings. That is, in this case, the consultants simply did not carry out their calculations as intended. This is not to suggest by any stretch that the intended calculations are necessarily more accurate or precise than the unintended error. That is, there certainly may be far more - are likely far more than these 44 teachers whose ratings fall arbitrarily and capriciously in the zone whereby those teachers would face employment consequences. So, how can we tell… how can we identify such teachers."

"Evidently, the CEO of Harmony Charter Schools will testify today that Harmony Science Cademy Schools in Texas have a 100% graduation rate. We often hear charter school operators make this claim. Are charters schools really that great? Should we open more charter schools to increase the graduation rate? And what, exactly, are these schools doing to have such extraordinary graduation rates."

"New York City has just released the new round of results from its school rating system (they're called "progress reports"). It relies considerably more on student growth (60 out of 100 points) than absolute performance (25 points), and there are efforts to partially adjust most of the measures via peer group comparisons.* All of this indicates that the city's system is more focused on school rather than student test-based performance, compared with many other systems around the U.S. The ratings are high-stakes. Schools receiving low grades - a D or F in any given year, or a C for three consecutive years - enter a review process by which they might be closed. The number of schools meeting these criteria jumped considerably this year. There is plenty of controversy to go around about the NYC ratings, much of it pertaining to two important features of the system. They're worth discussing briefly, as they are also applicable to systems in other states."

New York State is set to release its annual testing data tomorrow. Throughout the state, and especially in New York City, we will hear a lot about changes in school and district proficiency rates. The rates themselves have advantages - they are easy to understand, comparable across grades and reflect a standards-based goal. But they also suffer severe weaknesses, such as their sensitivity to where the bar is set and the fact that proficiency rates and the actual scores upon which they're based can paint very different pictures of student performance, both in a given year as well as over time. I've discussed this latter issue before in the NYC context (and elsewhere), but I'd like to revisit it quickly.

"Here's a fact that may not surprise you: the children of the rich perform better in school, on average, than children from middle-class or poor families. Students growing up in richer families have better grades and higher standardized test scores, on average, than poorer students; they also have higher rates of participation in extracurricular activities and school leadership positions, higher graduation rates and higher rates of college enrollment and completion. Whether you think it deeply unjust, lamentable but inevitable, or obvious and unproblematic, this is hardly news. It is true in most societies and has been true in the United States for at least as long as we have thought to ask the question and had sufficient data to verify the answer. What is news is that in the United States over the last few decades these differences in educational success between high- and lower-income students have grown substantially."

So why do we assume, when it comes to evaluating schools, that we must look at numbers alone? Sure, there have been calls to build additional indicators, beyond test scores, into school grading systems. These might include graduation rates, student or teacher attendance rates, results from student surveys, AP course-taking or exam-passing rates, etc. Our own recent paper on model state accountability systems offers quite a few ideas along these lines. This is all well and good. But it's not enough. It still assumes that we can take discrete bits of data and spit out a credible assessment of organizations as complex as schools. That's not the way it works in businesses, famous for their "bottom lines." Fund managers don't just look at the profit and loss statements for the companies in which they invest. They send analysts to go visit with the team, hear about their strategy, kick the tires, talk to insiders, find out what's really going on. Their assessment starts with the numbers, but it doesn't end there. So it should be with school accountability systems.

What many Americans do not realize is that the program that was dismantled had been extraordinarily successful in graduating Latino students and sending them to college. Nationally, Latino students drop out of high school at a much higher rate than any other racial or ethnic group-about 18 percent, according to the National Center for Education Statistics. Among Latinos, aged 18 to 24, 27 percent have not graduated from high school, according to the Pew Hispanic Center. Nationally, the college-enrollment rate of Latinos, while at an all-time high, is only 32 percent-lower than that of other racial and ethnic groups, according to the same study. In contrast, students completing Tucson's Mexican-American studies program graduate high school and enter college at a higher rate in a district that is 60 percent Latino.

A recent report by the New York Civil Liberties Union exposed the escalating number of students who have been suspended since Mayor Michael Bloomberg took control of the city's schools more than a decade ago. Some believe one contributing factor may lie in the growing number of the public charter schools created during his tenure that develop their own discipline codes and have higher than average suspension rates. Advocates for Children, a nonprofit that represents the legal rights of public school children, believe that the rise in charters (77 in 2008 and 135 in 2012) has gone hand in hand with the fact that a number of them exclude children-particularly those with special needs-at higher than average rates.

About a week ago, Florida officials went into crisis mode after revealing that the proficiency rate on the state's writing test (FCAT) dropped from 81 percent to 27 percent among fourth graders, with similarly large drops in the other two grades in which the test is administered (eighth and tenth). The panic was almost immediate. For one thing, performance on the writing FCAT is counted in the state's school and district ratings. Many schools would end up with lower grades and could therefore face punitive measures. Understandably, a huge uproar was also heard from parents and community members. How could student performance decrease so dramatically? There was so much blame going around that it was difficult to keep track - the targets included the test itself, the phase-in of the state's new writing standards, and test-based accountability in general. Despite all this heated back-and-forth, many people seem to have overlooked one very important, widely-applicable lesson here: That proficiency rates, which are not "scores," are often extremely sensitive to where you set the bar.

"I have reviewed, albeit superficially, the test-based components of several states' school rating systems (e.g., OH, FL, NYC, LA, CO), with a particular focus on the degree to which they are actually measuring student performance (how highly students score), rather than school effectiveness per se (whether students are making progress). Both types of measures have a role to play in accountability systems, even if they are often confused or conflated, resulting in widespread misinterpretation of what the final ratings actually mean, and many state systems' failure to tailor interventions to the indicators being used. One aspect of these systems that I rarely discuss is the possibility that the ratings systems are an end in themselves."