Group items tagged

I've posted a few blogs recently on the topic of Student Growth Percentile Scores, or SGPs and how many state policymakers have moved to adopt these measures and integrate them into new evaluation systems for teachers. In my first post, I argued that SGPs are simply not designed to make inferences about teacher effectiveness.

This new post from Ed News Colorado is in response to my critique of Student Growth Percentiles here: http://schoolfinance101.wordpress.com/2011/09/02/take-your-sgp-and-vamit-damn-it/

The specifics of how growth models should be constructed and used to evaluate schools and teachers is a topic of lively policy debate in states and school districts nationwide. In this paper we take up the question of model choice and examine three competing approaches. The first approach, reflected in the popular student growth percentiles (SGPs) framework, eschews all controls for student covariates and schooling environments. The second approach, typically associated with value-added models (VAMs), controls for student background characteristics and aims to identify the causal effects of schools and teachers. The third approach, also VAM-based, fully levels the playing field so that the correlation between school- and teacher-level growth measures and student demographics is essentially zero. We argue that the third approach is the most desirable for use in educational evaluation systems. Our case rests on personnel economics, incentive-design theory, and the potential role that growth measures can play in improving instruction in K-12 schools

This post includes a few more preliminary musings regarding the use of value-added measures and student growth percentiles for teacher evaluation, specifically for making high-stakes decisions, and especially in those cases where new statutes and regulations mandate rigid use/heavy emphasis on these measures, as I discussed in the previous post.

In the near future my article with Preston Green and Joseph Oluwole on legal concerns regarding the use of Value-added modeling for making high stakes decisions will come out in the BYU Education and Law Journal. In that article, we expand on various arguments I first laid out in this blog post about how use of these noisy and potentially biased metrics is likely to lead to a flood of litigation challenging teacher dismissals. In short, as I have discussed on numerous occasions on this blog, value-added models attempt to estimate the effect of the individual teacher on growth in measured student outcomes. But, these models tend to produce very imprecise estimates with very large error ranges, jumping around a lot from year to year.  Further, individual teacher effectiveness estimates are highly susceptible to even subtle changes to model variables. And failure to address key omitted variables can lead to systemic model biases which may even lead to racially disparate teacher dismissals (see here & for follow up , here) .

I spent some time the other day, while out running, pondering the usefulness of student growth percentile estimates and value added estimates of teacher effectiveness for the average school or district level practitioner. How would they use them? What would they see in them? How might these performance snapshots inform practice?

I have written much on this blog about problems with the use of Value-added Estimates of teacher effect (used loosely) on student test score gains on this blog. I have addressed problems with both the reliability and validity of VAM estimates, and I have pointed out how SGP based estimates of student growth are invalid on their face for determining teacher effectiveness. But, I keep hearing two common refrains from the uber-reformy (those completely oblivious to the statistics and research of VAM while also lacking any depth of understanding of the complexities of the social systems [schools] into which they propose to implement VAM as a de-selection tool) crowd. Sadly, these are the people who seem to be drafting policies these days.

A series of blog posts from School Finance 101.

In a series of recent blog posts and in a forthcoming article I have discussed the potential problems with using bad, versus entirely inappropriate measures for determining teacher effectiveness.  I have pointed out, for example, that using value-added measures to estimate teacher effectiveness and then determine whether a teacher should be denied tenure, or have their tenure removed might raise due process concerns which arise from the imprecision and potential outright inaccuracy of teacher effectiveness estimates derived from such methods. I have also explained that in some states like New Jersey, which have adopted Student Growth Percentile measures as an evaluation tool, that where those measures are used as a basis for dismissing teachers, teachers (or their attorney's) might simply rely on the language of the authors of those methods to point out that they are not designed to, nor were they intended to attribute responsibility for the measured student growth to the teacher. Where attribution of responsibility is off the table the dismissing a teacher on an assumption of ineffectiveness based on these measures is entirely inappropriate, and a potential violation of the teacher's due process rights. But, the problem is that state legislatures are increasingly mandating that these measures absolutely be used when making high stakes personnel decisions.

This technical report contains four main sections:  1) Data. Description of the data used to implement the student growth model, including data processing rules and relevant issues that arose during processing.  2) Model. Statistical description of the model.  3) Reporting. Description of reporting metrics and computation of effectiveness scores.  4) Results. Overview of key model results aimed at providing information on model quality and characteristics. It is important to note that results presented in this report are based on 2010-11 and prior school years' data. The model will be re-estimated with 2011-12 data when they are available"

"Our findings are consistent with research that finds models including student background and classroom characteristics are highly correlated with simpler specifications that only include a single-subject lagged test score, while value-added models estimated with school or student fixed effects have a lower correlation. Interestingly, teacher effectiveness estimates based on median student growth percentiles are highly correlated with estimates from VAMs that include only a lagged test score and those that also include lagged scores and student background characteristics, despite the fact that the two methods for estimating teacher effectiveness are, at least conceptually, quite different. However, even when the correlations between job performance estimates generated by different models are quite high, differences in the composition of students in teachers' classrooms can have sizable effects on the differences in their effectiveness estimates."

"I received my 'growth score' today from the New York State Education Department. I know,  I really shouldn't care what my score is. I know 100% of my students tested at or above grade level in Math and English Language Arts.  I know my class' scores were near or at the very top of my district's scores. I know my district is also at or nearly at the top of the region's and states' scores. I know I work my heart out and push my students to excel. My students always, ALWAYS  succeed. Yet according to the NYSED my growth score is so so. I'm rated effective with a growth score of 14 out of 20. Keep in mind, my student's mean scale in math  is 708.4 and ELA it is 678.  I'm confident both scores are well above that state mean. So why did I get a mediocre growth score? The state's explanation of it's calculation should be a eye opener for all  of us."

The U.S. Education Department is offering the waivers to states that adopt an "index" system of multiple measures that go beyond annual test results in determining school performance. These include test score growth over time, graduation rates and other evidence that schools have produced students who are college- or career-ready. States also must show plans for evaluating teachers and principals by multiple measures.

The fact that so many states have mandated 4-5 category schemes right out of the gate, seemingly based on little more than speculation among advocacy groups, is yet another instance of the rushed, ill-considered drive to overhaul teacher evaluations. It's amazing how certain some people seem about what these new systems are supposed to look like, given the fact that there is barely a shred of evidence as to their optimal form. In these situations, it's often wise to encourage experimentation, see how different configurations turn out and learn from this variation.