Group items matching

in title, tags, annotations or url

About a week ago, Florida officials went into crisis mode after revealing that the proficiency rate on the state's writing test (FCAT) dropped from 81 percent to 27 percent among fourth graders, with similarly large drops in the other two grades in which the test is administered (eighth and tenth). The panic was almost immediate. For one thing, performance on the writing FCAT is counted in the state's school and district ratings. Many schools would end up with lower grades and could therefore face punitive measures. Understandably, a huge uproar was also heard from parents and community members. How could student performance decrease so dramatically? There was so much blame going around that it was difficult to keep track - the targets included the test itself, the phase-in of the state's new writing standards, and test-based accountability in general. Despite all this heated back-and-forth, many people seem to have overlooked one very important, widely-applicable lesson here: That proficiency rates, which are not "scores," are often extremely sensitive to where you set the bar.

I have been writing critically about states' school rating systems (e.g., Ohio, Florida, Louisiana), and I thought I would find one that is, at least in my (admittedly value-laden) opinion, more defensibly designed. It didn't quite turn out as I had hoped. One big starting point in my assessment is how heavily the systems weight absolute performance (how highly students score) versus growth (how quickly students improve). As I've argued many times, the former (absolute level) is a poor measure of school performance in a high-stakes accountability system. It does not address the fact that some schools, particularly those in more affluent areas, serve  students who, on average, enter the system at a higher-performing level. This amounts to holding schools accountable for outcomes they largely cannot control (see Doug Harris' excellent book for more on this in the teacher context). Thus, to whatever degree testing results can be used to judge actual school effectiveness, growth measures, while themselves highly imperfect, are to be preferred in a high-stakes context. There are a few states that assign more weight to growth than absolute performance (see this prior post on New York City's system). One of them is Colorado's system, which uses the well-known "Colorado Growth Model" (CGM).

Mayor Michael R. Bloomberg and the schools chancellor, Dennis M. Walcott, often boast that student performance is improving in New York City, as evidenced by the percentage of students passing state exams and graduating from high school. But a new analysis finds that most city students are holding steady, getting very similar test scores between third and sixth grades.

"Most research on educational interventions, including school vouchers, focuses on impacts on short-term outcomes such as students' scores on standardized tests. Few studies are able to track longer-term outcomes, and even fewer are able to do so in the context of a randomized experiment. In the first study using a randomized experiment to measure the impact of school vouchers on college enrollment, we examine the college-going behavior through 2011 of students who participated in a voucher experiment as elementary school students in the late 1990s. We find no overall impacts on college enrollments but we do find large, statistically significant positive impacts on the college going of African American students who participated in the study. Our estimates indicate that using a voucher to attend private school increased the overall college enrollment rate among African Americans by 24 percent."

"This post is about rolling out some of the left over data I have from my various endeavors this summer.  These data include data from New York State personnel master files (PMFs) linked to New York City public schools and charter schools, NYC teacher value-added scores, and various bits of data on New York City charter and district schools including school site budget/annual financial report information. Here, I use these data combined with some of my previous stuff, to take a first, cursory shot at characterizing the teaching workforce of charter school teachers in New York City. All findings use data from 2008 to 2010."

"The organization that was launched to serve public schools so poor or dysfunctional they couldn't attract qualified teachers now sends fully a third of its recruits to privately run charter schools, many with stellar academic reputations, flush budgets and wealthy donors. TFA also sends its rookies, who typically have just 15 to 20 hours of teaching experience, to districts that have recently laid off scores of more seasoned teachers."

This technical report contains four main sections:  1) Data. Description of the data used to implement the student growth model, including data processing rules and relevant issues that arose during processing.  2) Model. Statistical description of the model.  3) Reporting. Description of reporting metrics and computation of effectiveness scores.  4) Results. Overview of key model results aimed at providing information on model quality and characteristics. It is important to note that results presented in this report are based on 2010-11 and prior school years' data. The model will be re-estimated with 2011-12 data when they are available"

"I have reviewed, albeit superficially, the test-based components of several states' school rating systems (e.g., OH, FL, NYC, LA, CO), with a particular focus on the degree to which they are actually measuring student performance (how highly students score), rather than school effectiveness per se (whether students are making progress). Both types of measures have a role to play in accountability systems, even if they are often confused or conflated, resulting in widespread misinterpretation of what the final ratings actually mean, and many state systems' failure to tailor interventions to the indicators being used. One aspect of these systems that I rarely discuss is the possibility that the ratings systems are an end in themselves."

"The education reform debate is fueled by a seemingly endless and even fruitless point-counterpoint among the corporate reformers-typically advocates for and from the Gates Foundation (GF), Teach for America (TFA), and charter chains such as Knowledge Is Power Program (KIPP)-and educators/scholars of education. Since the political and public machines have embraced the corporate reformers, GF, TFA, and KIPP have acquired the bully pulpit of the debate and thus are afforded most often the ability to frame the point, leaving educators and scholars to be in a constant state of generating counter-points. This pattern disproportionately benefits corporate reformers, but it also exposes how those corporate reformers manage to maintain the focus of the debate on data. The statistical thread running through most of the point-counterpoint is not only misleading (the claims coming from the corporate reformers are invariably distorted, while the counter-points of educators and scholars remain ignored among politicians, advocates, the public, and the media), but also a distraction. Since the metrics debate (test scores, graduation rates, attrition, populations of students served, causation/correlation) appears both enduring and stagnant, I want to make a clear statement with some elaboration that I reject the "ends-justify-the-means" assumptions and practices-the broader "no excuses" ideology-underneath the numbers, and thus, we must stop focusing on the outcomes of programs endorsed by the GF or TFA and KIPP. Instead, we must unmask the racist and classist policies and practices hiding beneath the metrics debate surrounding GF, TFA, and KIPP (as prominent examples of practices all across the country and types of schools)."

"New York state published a list of schools based on measures like test scores and graduation rates. At the top are "reward" schools. At the bottom are "priority" schools. This is the amazing discovery. The schools that enroll mostly white and Asian students in affluent neighborhoods are doing a great job; they get a reward. The schools that enroll mostly black and Hispanic students in poor neighborhoods are doing a bad job; they are in line to get sanctions, interventions."

"A rising tide of protest is sweeping across the nation as growing numbers of parents, teachers, administrators and academics take action against high-stakes testing. Instead of test-and-punish policies, which have failed to improve academic performance or equity, the movement is pressing for broader forms of assessment. From Texas to New York and Florida to Washington, reform activists are pressing to reduce the number of standardized exams. They also seek to scale back the consequences attached to test scores and use multiple measures to evaluate students, educators, schools and districts."

"Earlier this summer, the New York City Independent Budget Office (IBO) presented findings from a longitudinal analysis of NYC student performance. That is, they followed a cohort of over 45,000 students from third grade in 2005-06 through 2009-10 (though most results are 2005-06 to 2008-09, since the state changed its definition of proficiency in 2009-10). The IBO then simply calculated the proportion of these students who improved, declined or stayed the same in terms of the state's cutpoint-based categories (e.g., Level 1 ["below basic" in NCLB parlance], Level 2 [basic], Level 3 [proficient], Level 4 [advanced]), with additional breakdowns by subgroup and other variables. The short version of the results is that almost two-thirds of these students remained constant in their performance level over this time period - for instance, students who scored at Level 2 (basic) in third grade in 2006 tended to stay at that level through 2009; students at the "proficient" level remained there, and so on. About 30 percent increased a category over that time (e.g., going from Level 1 to Level 2). The response from the NYC Department of Education (NYCDOE) was somewhat remarkable. It takes a minute to explain why, so bear with me."

"It has been difficult to discern what specific details are left on the table in the Chicago teachers' negotiations. Broadly, we know the union leadership resents Mayor Rahm Emanuel's enthusiasm for non-unionized charter schools and neighborhood school closings. It is also clear that professional evaluation is a big issue, as it is in states and cities across the country. To what extent should teachers be judged by their students' test scores, as opposed to by more holistic measures? Job security, especially for teachers in schools that will be shut down, has been eroding, which the CTU sees as a calamity, yet many reformers applaud. And of course, there is pay. Is it fair for teachers to demand regular raises when unemployment is so high, and budgets at every level of government are strapped? I'm not going to pronounce on these questions today, but I do want to offer a quick history of teacher unionism to keep things in perspective. The modern teachers' union movement began in Chicago in 1897, and many of the problems back then -- from low school budgets to testing to debates over classroom autonomy -- remain more than salient today."

For this podcast, we spoke with Jack Buckley, commissioner at the National Center for Education Statistics, the center which administers the NAEP test. He took Early Ed Watch on a tour of the data from the most recent NAEP scores. Among the highlights is a trend that Buckley says suggests that students who are both at the top and bottom of their grade level may be improving-a finding contrary to the notion that No Child Left Behind has led teachers to focus on low-performing students at the expense of high-performing ones.

This report carefully reviews high-quality empirical evidence from the last several years on the test score effects of three approaches to modifying the organization of schools: (1) starting schools later in the morning, (2) favoring K-8 grade configuration instead of junior high or middle school configurations, and (3) increasing teacher specialization by grade and subject. It estimates the earnings benefits of each intervention and, for interventions (1) and (2), compares monetary benefits to costs. The report concludes that benefits outweigh costs, although the rough cost estimates suggest that better data are required to draw definite conclusions. The report's main conclusion is that organizational reforms deserve a more prominent place in education debates, and that individual school districts should carefully consider them alongside more popular reform options. The review points to a few shortcomings but concludes that the report's analyses are solid and helpful and that the results are presented carefully and cautiously.

Over the past two decades, state and federal education policies have tried to hold schools more accountable for educating their students. A common criticism of these policies is that they may induce schools to "game the system" with strategies such as excluding certain types of students from computation of school average test scores. In this post, based on our recent New York Fed staff report, "Vouchers, Responses, and the Test Taking Population: Regression Discontinuity Evidence from Florida," we investigate whether Florida schools resorted to such strategic behavior in response to a voucher program. We find some evidence that Florida's schools strategically reclassified weak students into exempt categories, and we draw some lessons that are applicable to New York City's education policies.

The Center For American Progress has published another report justifying the firing of teachers today, based on statistical models that may some day become valid. "Designing High Quality Evaluation Systems," by John Tyler, recounts the standard reasons why of educators do not trust high-stakes test-driven algorithms, and even contributes a couple of new insights into problems that are unique to high school test scores. An urban teacher reading Tyler's evidence would likely conclude that he has written an ironclad indictment of value-added models for high-stakes purposes. But, as is usually true of CAP's researchers, he concludes that the work of economists in improving value-added models is so impressive that education will benefit from their experiments if educators don't blow it.

Sunday's newspaper featured a story about full-time public virtual schools, a new model of education that's growing fast even though critics say there's scant evidence that it is an effective way to teach kids. The story focused on Herndon-based K12 Inc., the nation's largest operator of virtual schools. Its schools (which educate about 95,000 students in 29 states and the District) tend to have lower state test scores and graduation rates than brick and mortar schools.

Now that six months have gone by, it's time to check in on our three miracle schools and see if their improvements have continued in a steady rate. Only someone who knows nothing about education and schools would expect these improvements to continue. Why would they? If you get new students each year and often a new crop of new teachers, why would scores continue to rise until they eventually reach 100%?

Value-added and other types of growth models are probably the most controversial issue in education today. These methods, which use sophisticated statistical techniques to attempt to isolate a teacher's effect on student test score growth, are rapidly assuming a central role in policy, particularly in the new teacher evaluation systems currently being designed and implemented.