Group items tagged

The second element of the new stupid is Translating Research Simplistically. For two decades, advocates of class-size reduction have referenced the findings from the Student Teacher Achievement Ratio (STAR) project, a class-size experiment conducted in Tennessee in the late 1980s. Researchers found significant achievement gains for students in small kindergarten classes and additional gains in 1st grade, especially for black students. The results seemed to validate a crowd-pleasing reform and were famously embraced in California, where in 1996 legislators adopted a program to reduce class sizes that cost nearly $800 million in its first year and billions in its first decade. The dollars ultimately yielded disappointing results, however, with the only major evaluation (a joint American Institutes for Research and RAND study) finding no effect on student achievement.

...This criticism of the abandonment of scholarly norms is of particular salience with respect to the release of CFR [Chetty, Friedman, and Rockoff], because the authors are using the authority of their study to advocate public policy prescriptions which are simply not supported by their underlying analysis, even if one were to grant its validity.

The Bill & Melinda Gates Foundation's "Measures of Effective Teaching" (MET) Project seeks to validate the use of a teacher's estimated "value-added"-computed from the year-on-year test score gains of her students-as a measure of teaching effectiveness. Using data from six school districts, the initial report examines correlations between student survey responses and value-added scores computed both from state tests and from higher-order tests of conceptual understanding. The study finds that the measures are related, but only modestly. The report interprets this as support for the use of value-added as the basis for teacher evaluations. This conclusion is unsupported, as the data in fact indicate that a teachers' value-added for the state test is not strongly related to her effectiveness in a broader sense. Most notably, value-added for state assessments is correlated 0.5 or less with that for the alternative assessments, meaning that many teachers whose value-added for one test is low are in fact quite effective when judged by the other. As there is every reason to think that the problems with value-added measures apparent in the MET data would be worse in a high-stakes environment, the MET results are sobering about the value of student achievement data as a significant component of teacher evaluations.

My new book, When Can You Trust the Experts: How to Tell Good Science from Bad in Education is now available. (There's a link for a free download of Chapter 1 on this page.) I wrote the book out of frustration with a particular problem: the word "research" has become meaningless in education. Every product is claimed to be research-based. But we all know that can't be the case. How are teachers and administrators supposed to know which claims are valid?

An important new study  by Professors Adam Maltese of Indiana University and Craig Hochbein of the University of Louisville sheds new light on the validity of state scores. This study found that rising scores on the state tests did not correlate with improved performance on the ACT. In fact, students at "declining" schools did just as well and sometimes better than students where the scores were going up. The study was published in the Journal of Research in Science Teaching. Its title is ""The Consequences of 'School Improvement': Examining the Association Between Two Standardized Assessments Measuring School Improvement and Student Science Achievement."

"Education is a not a scientific enterprise. The purpose is not to describe the world, but to change it, to make it more similar to some ideal that we envision. (I wrote about this distinction at some length in my new book. I also discussed on this brief video.) Thus science is ideally value-neutral. Yes, scientists seldom live up to that ideal; they have a point of view that shapes how they interpret data, generate theories, etc., but neutrality is an agreed-upon goal, and lack of neutrality is a valid criticism of how someone does science. Education, in contrast, must entail values, because it entails selecting goals. We want to change the world - we want kids to learn things --facts, skills, values. Well, which ones? There's no better or worse answer to this question from a scientific point of view."

Most readers of the NJDOE report on reforming the state's school finance formula likely glided right past the seemingly innocuous recommendation to shift the enrollment count method for funding from a fall enrollment count to an average daily attendance figure. After all, on its face, the argument provided seems to make sense. Let's fund on this basis so that we can incentivize increased attendance in our most impoverished and low performing districts. (Another argument I've heard in other states is "why would we fund kids who aren't there?"). The data were even presented to validate that attendance rates are lower in these districts (Figure 3.1). I, however, could not let this pass, because Average Daily Attendance as a basis for funding is actually a well understood trick of the trade for reducing aid to districts and schools with higher poverty and minority concentrations.  I have both blogged about this topic in the past, and written published research directly and indirectly related to the topic.[1] The intent of this blog post is to provide a (very limited, oversimplified) primer on the common methods of counting general student populations for purposes of determining state aid to schools (charter and district) and to provide some commentary on the pros and cons of each.

Although much has been written about the importance of leadership in the determination of organizational success, there is little quantitative evidence due to the difficulty of separating the impact of leaders from other organizational components - particularly in the public sector. Schools provide an especially rich environment for studying the impact of public sector management, not only because of the hypothesized importance of leadership but also because of the plentiful achievement data that provide information on institutional outcomes. Outcome-based estimates of principal value-added to student achievement reveal significant variation in principal quality that appears to be larger for high-poverty schools. Alternate lower-bound estimates based on direct estimation of the variance yield smaller estimates of the variation in principal productivity but ones that are still important, particularly for high poverty schools. Patterns of teacher exits by principal quality validate the notion that a primary channel for principal influence is the management of the teacher force. Finally, looking at principal transitions by quality reveals little systematic evidence that more effective leaders have a higher probability of exiting high poverty schools.

The key element in the agreement reached between the New York City teachers' union and the city's Education Department last Thursday was given a closer look, as The New York Times and Gotham Schools examined the role to be played by so-called "independent validators" in rating teacher effectiveness. As part of the deal reached between the city Education Department and the United Federation of Teachers, independent observers will be brought into New York City's classrooms "to monitor the weakest teachers and provide a second opinion to supplement observations by the school principal," writes Winnie Hu in The Times.

"In previous posts, I've conveyed my distaste for the oft obsessively narrow thinking of the traditional labor economist when engaged in education policy research. I've picked on the assumption that greed and personal interest are necessarily the sole driving force of all human rational decision making. And I've picked on the obsession with narrow and circular validity tests. Yet still, sometimes, I see quotes from researchers I otherwise generally respect, that completely blow my mind. I gotta say, this quote from Tom Kane of Harvard regarding compensation for teachers holding masters degrees is right up there with the worst of them - most notably because it conveys such an obscenely narrow perspective of compensation policies (public or private sector) and broader complexities of labor market dynamics."

"The current debate regarding interpretation of recently released results of the Program for International Student Assessment (PISA) implies that we need to choose between two interpretations: 1. The educational sky isn't falling. Although US students have never done well on international tests, as a nation we have made remarkable economic progress and remain the strongest most innovative economy; or 2. PISA results should be a wake up call. PISA assesses important 21st century skills. Our students' abilities remain stagnant while other countries are racing ahead. Could these both be valid claims?"

"While any teacher evaluation system must include a measure of growth of student learning over time, a snapshot of a student's skills, understanding and content knowledge doesn't give a true picture of a teacher's performance. Our current state tests are not designed to measure growth from year to year, and we are years away from having valid state tests that are. "

"Class Size: What Research Says and What It Means for State Policy argues that increasing average class size by one student will save about 2% of total education spending with negligible impact on academic achievement. It justifies this conclusion on the basis that Class-Size Reduction (CSR) is not particularly effective and is not as cost-effective as other reforms. However, this conclusion is based on a misleading review of the CSR research literature. The report puts too much emphasis on studies that are of poor quality or that do not focus on settings that are particularly relevant to the debate on class-size policy in the United States. It argues that class-size reduction is less cost-effective than other reform policies, but it bases this contention on an incomplete accounting of the benefits of smaller classes and an uncritical, unexamined list of alternative policies. The report's estimates of the potential cost savings are flawed as, in reality, schools cannot structurally reduce class size by only one student. Well-documented and long-term non-academic gains from CSR are not addressed. Likewise, the recommendation for releasing the ―least effective‖ teachers assumes a valid way of making such determinations is available. "

In a forthcoming book, Teaching and Its Predicaments (Harvard University Press, August 2011), I argue that fragmented school governance in the U.S. coupled with the lack of coherent educational infrastructure make it difficult either to broadly improve teaching and learning or to have valid knowledge of the extent of improvement.

Minnesota parents who teach their children at home feel like they have received a promotion. "The state is recognizing the validity of home education," Lorna Cook of Willmar said of a new law that frees homeschoolers from most of the bureaucracy of old laws. "To those of us who are home educators, the statistics show that, overall for home education, parents are doing a pretty good job."

It is a dangerous and potentially failed thing to manipulate an audience by evoking our tendency to be compassionate for children. I have often used the documentary The Corridor of Shame, concerning the inequity in school funding in my home state of South Carolina, in my education courses both to highlight the corrosive power of poverty in the lives and learning of children and to confront the dishonesty of emotional appeals that do more to harm a valid message than reinforce it.

Some Twitter followers may have noticed the ongoing back and forth regarding the validity of the recent Monmouth University Poll on education reform.I'd certainly rather spend my time on more substantive discussion. As I've noted on many occasions, polls are what they are. They ask what they ask. And the responses to the questions must always be evaluated only with respect to what was asked. Questions about specific policies in particular require that the policies in question be described correctly. This is a point raised the other day by Matt Di   Carlo about the Monmouth Poll here. Yesterday, Patrick Murray, director of the polling institute posted a response to some of the criticisms levied against the recent Monmouth poll. Unfortunately, I found his response to be much less fulfilling and in many ways far more disturbing than the poll itself. Quite honestly, I'd have left this issue alone if not for some particularly troublesome assertions made by the polling institute director Patrick Murray.

The Think Tank Review Project provides the public, policy makers, and the press with timely, academically sound reviews of selected think-tank publications. Reviewers for the Think Tank Review Project apply academic peer review standards to reports from think tanks and write brief reviews for the project web site. They are asked to examine the reports for the validity of assumptions, methodology, results, and strength of links between results and policy recommendations. The reviews, written in non-academic language, are intended to help policy makers, reporters, and others assess the merits of the reviewed reports.