Group items tagged

State assessments are increasingly used as outcome measures for education evaluations and pretest scores are generally used as control variables in these evaluations. The correlation between the pretest and outcome (posttest) measures is a factor in determining, among other things, the statistical power of a study. This report examines the variability in pretest-posttest correlation coefficients for state assessment data on samples of low-performing, average-performing, and proficient students to determine how sample characteristics (e.g., achievement level) affect pretest-posttest correlation coefficients. As an application, this report illustrates how statistical power is affected by variations in pretest-posttest correlation coefficients across groups with different sample characteristics. Achievement data from four states and two large districts are examined. The results confirm that pretest-posttest correlation coefficients are smaller for samples of low performers than for samples representing the full range of performers, thus, resulting in lower statistical power for impact studies than would be the case if the study sample included a more representative group of students.

"Our findings are consistent with research that finds models including student background and classroom characteristics are highly correlated with simpler specifications that only include a single-subject lagged test score, while value-added models estimated with school or student fixed effects have a lower correlation. Interestingly, teacher effectiveness estimates based on median student growth percentiles are highly correlated with estimates from VAMs that include only a lagged test score and those that also include lagged scores and student background characteristics, despite the fact that the two methods for estimating teacher effectiveness are, at least conceptually, quite different. However, even when the correlations between job performance estimates generated by different models are quite high, differences in the composition of students in teachers' classrooms can have sizable effects on the differences in their effectiveness estimates."

In today's New York Times there was a story about a research study which supposedly proved that students who had teachers with good value-added scores were more successful in life. This inspired me to complete something I have been working on for several months, off-and-on, a detailed analysis of the raw data supplied in the most quoted value-added study there is, a paper written in Dallas in 1997. This is the paper which 'proved' that students who had three effective teachers in a row got dramatically higher test scores than their unlucky peers who had three ineffective teachers in a row.  I've written about it previously much less formally here and here. The New York Times story frustrated me since I know that value-added does not correlate with future student income. Value-added does not correlate with teacher quality. Value-added doesn't correlate with principal evaluations. It doesn't correlate with anything including, as I'll demonstrate in this post, with itself.

So far, after two full years under IMPACT, the correlation is modest at best. An analysis of scores by DCPS consultant Mathematica shows that it was 0.34 (+1 indicating the highest possible positive correlation, and 0 indicating no correlation) in 2009-10. Last year it was 0.35.

Charter schools were developed, in part, to serve as an R&D engine for traditional public schools, resulting in a wide variety of school strategies and outcomes. In this paper, we collect unparalleled data on the inner-workings of 35 charter schools and correlate these data with credible estimates of each school's effectiveness. We find that traditionally collected input measures -- class size, per pupil expenditure, the fraction of teachers with no certification, and the fraction of teachers with an advanced degree -- are not correlated with school effectiveness. In stark contrast, we show that an index of five policies suggested by over forty years of qualitative research -- frequent teacher feedback, the use of data to guide instruction, high-dosage tutoring, increased instructional time, and high expectations -- explains approximately 50 percent of the variation in school effectiveness. Our results are robust to controls for three alternative theories of schooling: a model emphasizing the provision of wrap-around services, a model focused on teacher selection and retention, and the "No Excuses'' model of education. We conclude by showing that our index provides similar results in a separate sample of charter schools.

The Bill & Melinda Gates Foundation's "Measures of Effective Teaching" (MET) Project seeks to validate the use of a teacher's estimated "value-added"-computed from the year-on-year test score gains of her students-as a measure of teaching effectiveness. Using data from six school districts, the initial report examines correlations between student survey responses and value-added scores computed both from state tests and from higher-order tests of conceptual understanding. The study finds that the measures are related, but only modestly. The report interprets this as support for the use of value-added as the basis for teacher evaluations. This conclusion is unsupported, as the data in fact indicate that a teachers' value-added for the state test is not strongly related to her effectiveness in a broader sense. Most notably, value-added for state assessments is correlated 0.5 or less with that for the alternative assessments, meaning that many teachers whose value-added for one test is low are in fact quite effective when judged by the other. As there is every reason to think that the problems with value-added measures apparent in the MET data would be worse in a high-stakes environment, the MET results are sobering about the value of student achievement data as a significant component of teacher evaluations.

Recent analyses challenge common wisdom regarding the superiority of private schools relative to public schools, raising questions about the role of school processes and climate in shaping achievement in different types of schools. While holding demographic factors constant, this multilevel analysis of National Assessment of Educational Progress (NAEP) mathematics data on over 270,000 fourth and eighth graders in over 10,000 schools examines differences among schools on five critical factors: (1) school size, (2) class size, (3) school climate/parental involvement, (4) teacher certification, and (5) instructional practices. This study provides nationally representative evidence that both teacher certification and some reform-oriented mathematics teaching practices correlate positively with achievement and are more prevalent in public schools than in demographically similar private schools. Additionally, smaller class size, more prevalent in private schools, is significantly correlated with achievement.

The specifics of how growth models should be constructed and used to evaluate schools and teachers is a topic of lively policy debate in states and school districts nationwide. In this paper we take up the question of model choice and examine three competing approaches. The first approach, reflected in the popular student growth percentiles (SGPs) framework, eschews all controls for student covariates and schooling environments. The second approach, typically associated with value-added models (VAMs), controls for student background characteristics and aims to identify the causal effects of schools and teachers. The third approach, also VAM-based, fully levels the playing field so that the correlation between school- and teacher-level growth measures and student demographics is essentially zero. We argue that the third approach is the most desirable for use in educational evaluation systems. Our case rests on personnel economics, incentive-design theory, and the potential role that growth measures can play in improving instruction in K-12 schools

"This paper arises out of frustration with the results of school reforms carried out over the past few decades. These efforts have failed. They need to be abandoned. In their place must come recognition that income inequality causes many social problems, including problems associated with education. Sadly, compared to all other wealthy nations, the USA has the largest income gap between its wealthy and its poor citizens. Correlates associated with the size of the income gap in various nations are well described in Wilkinson & Pickett (2010), whose work is cited throughout this article. They make it clear that the bigger the income gap in a nation or a state, the greater the social problems a nation or a state will encounter. Thus it is argued that the design of better economic and social policies can do more to improve our schools than continued work on educational policy independent of such concerns."

Theories are like maps, the authors argue. They are "useful in helping us get somewhere." "Rather than attempt to imitate the hard sciences, social scientists would be ... doing what they do best: thinking deeply about what prompts human beings to behave the way they do." This includes gathering data-anecdotes, myths, and other such "soft" insights along with the so-called "hard" ones. It means including direct data, not just indirect test data which we hope "correlates" with reality." It means acknowledging tradeoffs: Do life, liberty, and happiness sometimes clash? Of course, this kind of "soft science" leads to less certainty. But less certainty where certainty doesn't exist is a good thing. One reason we need to stick with even flawed forms of democracy is that there isn't any flawless form! Every form of voting, for example, rests on a bias about whom and what is more important. Anyone studying the gerrymandering of districts in New York state notes that the latest plan makes it likely that a majority of voters will be "out-voted" by a minority when it comes to our state's legislative bodies. Our Constitution rests on similar "gerrymandering"-some voters count more than others.

I may not be able to prove that my literature class makes a difference in my students' test results, but there is a positive correlation between how much time students spend reading and higher scores. The problem is that low-income students, who begin school with a less-developed vocabulary and are less able to comprehend complex sentences than their more privileged peers, are also less likely to read at home. Many will read only during class time, with a teacher supporting their effort. But those are the same students who are more likely to lose out on literary reading in class in favor of extra test prep. By "using data to inform instruction," as the Department of Education insists we do, we are sorting lower-achieving students into classes that provide less cultural capital than their already more successful peers receive in their more literary classes and depriving students who viscerally understand the violence and despair in Steinbeck's novels of the opportunity to read them. It is ironic, then, that English Language Arts exams are designed for "cultural neutrality." This is supposed to give students a level playing field on the exams, but what it does is bleed our English classes dry. We are trying to teach students to read increasingly complex texts, but they are complex only on the sentence level - not because the ideas they present are complex, not because they are symbolic, allusive or ambiguous. These are literary qualities, and they are more or less absent from testing materials.

Educational interventions are often evaluated and compared on the basis of their impacts on test scores. Decades of research have produced two empirical regularities: interventions in later grades tend to have smaller effects than the same interventions in earlier grades, and the test score impacts of early educational interventions almost universally "fade out" over time. This paper explores whether these empirical regularities are an artifact of the common practice of rescaling test scores in terms of a student's position in a widening distribution of knowledge. If a standard deviation in test scores in later grades translates into a larger difference in knowledge, an intervention's effect on normalized test scores may fall even as its effect on knowledge does not. We evaluate this hypothesis by fitting a model of education production to correlations in test scores across grades and with college-going using both administrative and survey data. Our results imply that the variance in knowledge does indeed rise as children progress through school, but not enough for test score normalization to fully explain these empirical regularities.

This second report from the Measures of Effective Teaching (MET) project offers ground-breaking descriptive information regarding the use of classroom observation instruments to measure teacher performance. It finds that observation scores have somewhat low reliabilities and are weakly though positively related to value-added measures. Combining multiple observations can enhance reliabilities, and combining observation scores with student evaluations and test-score information can increase their ability to predict future teacher value-added. By highlighting the variability of classroom observation measures, the report makes an important contribution to research and provides a basis for the further development of observation rubrics as evaluation tools. Although the report raises concerns regarding the validity of classroom observation measures, we question the emphasis on validating observations with test-score gains. Observation scores may pick up different aspects of teacher quality than test-based measures, and it is possible that neither type of measure used in isolation captures a teacher's contribution to all the useful skills students learn. From this standpoint, the authors' conclusion that multiple measures of teacher effectiveness are needed appears justifiable. Unfortunately, however, the design calls for random assignment of students to teachers in the final year of data collection, but the classroom observations were apparently conducted prior to randomization, missing a valuable opportunity to assess correlations across measures under relatively bias-free conditions.

Eric Hanushek testifies in school finance cases. Again, and again, and again. Thirty-some years ago in the Maryland (Hornbeck) case and most recently in the Colorado (Lobato) case. And each time, Hanushek, an economist at the Hoover Institution, testifies to the same position: increased funding for K-12 schools will not improve their effectiveness; court-ordered remedies that cost money will not improve the lot of poor students or English Language Learners or anyone else for that matter. Hanushek is nothing if not a believer in the unconditional truth emanating from his regression equations. But of course, those equations have not always been as clear cut in their implications as some might believe. In 1997, Hanushek published an article in which he argued that a summary of dozens and dozens of correlation studies proved that teacher experience is unrelated to their students' achievement-the financial implications being obvious.

"The education reform debate is fueled by a seemingly endless and even fruitless point-counterpoint among the corporate reformers-typically advocates for and from the Gates Foundation (GF), Teach for America (TFA), and charter chains such as Knowledge Is Power Program (KIPP)-and educators/scholars of education. Since the political and public machines have embraced the corporate reformers, GF, TFA, and KIPP have acquired the bully pulpit of the debate and thus are afforded most often the ability to frame the point, leaving educators and scholars to be in a constant state of generating counter-points. This pattern disproportionately benefits corporate reformers, but it also exposes how those corporate reformers manage to maintain the focus of the debate on data. The statistical thread running through most of the point-counterpoint is not only misleading (the claims coming from the corporate reformers are invariably distorted, while the counter-points of educators and scholars remain ignored among politicians, advocates, the public, and the media), but also a distraction. Since the metrics debate (test scores, graduation rates, attrition, populations of students served, causation/correlation) appears both enduring and stagnant, I want to make a clear statement with some elaboration that I reject the "ends-justify-the-means" assumptions and practices-the broader "no excuses" ideology-underneath the numbers, and thus, we must stop focusing on the outcomes of programs endorsed by the GF or TFA and KIPP. Instead, we must unmask the racist and classist policies and practices hiding beneath the metrics debate surrounding GF, TFA, and KIPP (as prominent examples of practices all across the country and types of schools)."

The role of poverty in shaping educational outcomes is one of the most common debates going on today. It can also be one of the most shallow. The debate tends to focus on income. For example (and I'm generalizing a bit here), one "side" argues that income and test scores are strongly correlated; the other "side" points to the fact that many low-income students do very well and cautions against making excuses for schools' failure to help poor kids. Both arguments have merit, but it bears quickly mentioning that the focus on the relationship between income and achievement is a rather crude conceptualization of the importance of family background (and non-schooling factors in general) for education outcomes.

Unlike in elementary schools, high school teacher effects may be confounded with unobserved track-level treatments (such as the AVID program) that are correlated with individual teachers. I present a strategy that exploits detailed course-taking information to credibly estimate the effects of 9th grade Algebra and English teachers on test scores. I document substantial bias due to track-specific treatments and I show that traditional tests for the existence of teacher effects are flawed. After accounting for bias, I find sizable algebra teacher effects and little evidence of English teacher effects. I find little evidence of teacher spillovers across subjects.

Will Dobbie and Roland Fryer's new study of 35 New York City charter schools attempts to find a preliminary answer to the question of how different practices within charters are correlated with student progress on math and ELA tests. In general, this study's premise and methods represent a promising shift away from just looking at test scores to measure school quality; it acknowledges variations between charters and gets to the issue of what policies and practices are actually happening inside these schools.

In a state struggling to get better at teaching all students, every reform has a glaring omission. Some reforms want to pay special teachers more. As if there are Blackwater mercenary teachers out there, ready to storm the castle and kick down the achievement gap.  They even tried this to the extreme in New York, luring in teachers with $100,000 salaries to get the cream of the crop. It didn't really work. A University of Vanderbilt study, the most extensive on performance pay, concluded there was no correlation between closing the gap and performance pay. They value purpose more than profit. Teachers do not sit around all day brooding upon what their colleagues may or may not make. They deserve professional pay, but teachers are neither mercenaries nor missionaries some folks want to reform how we pay teachers. They pay no attention to reforming how they teach.

Perhaps the most controversial issue in Ed Reform is whether or not it is fair to tie teacher evaluation to their 'performance' as defined by reformers as how their students do on standardized exams.  Since even reformers acknowledge that teachers aren't able to take students from a low starting score to any absolute target of high performance, they have devised something that is intended to be fair.  It is known as 'value-added.' The idea, which has been around for about 30 years, is that there could be a way to compare how a teacher's students do on some test with how those same students would have done in a parallel universe where they had an 'average' teacher instead.  If it is possible to make such a measurement, it would determine that teacher's individual contribution to his student's 'learning.' To someone who is not a teacher, this sounds reasonable enough.  When you've spent time in schools, though, you know some of the basic problems with standardized tests.