Group items tagged

The 2013-2014 school year may seem like a long way off, but state school officials are already fretting over a perfect storm of education reforms that could make today's extensive state testing regimen seem like a snap. That's the year when students could take as many as five state-mandated tests, on top of their teachers' occasional pop quizzes and the tests given several times each year by the local school systems. While the Maryland School Assessment will be phased out, those tests will still overlap with a new battery of four new assessments to be field tested here and in 23 states.

Though most of the critiques so far focus on the inherently volatile nature and large margins of error in any such calculation, here in NY State we have a special problem: the state tests themselves have been fatally flawed for many years.  There has been rampant test score inflation over the past decade; many of the test questions themselves are amazingly dumb and ambiguous; and there are other severe problems with the scaling and the design of these exams that only testing experts fully understand.  Though the State Education Department claims to have now solved these problems, few actually believe this to be the case. As further evidence, see Fred Smith's analysis below.  Fred is a  retired assessment expert for the NYC Board of Education, who has written widely on the fundamental flaws in the state tests.  Here, he shows how deep problems remain in their design and execution -- making their results, and the new teacher evaluation system and  teacher data reports based upon them, essentially worthless.  He goes on to urge parents to boycott the state exams this spring.

As student learning is the primary goal of teaching, it seems like common sense to evaluate teachers based on how much their students gain on state standardized tests. Indeed, many states have adopted this idea in response to federal incentives tied to much-needed funding. However, previous experience is not promising. Recently evaluated experiments in Tennessee and New York did not improve achievement when teachers were evaluated and rewarded based on student test scores. In the District of Columbia, contrary to expectations, reading scores on national tests dropped and achievement gaps grew after a new test-based teacher-evaluation system was installed. In Portugal, a study of test-based merit pay attributed score declines to the negative effects of teacher competition, leading to less collaboration and sharing of knowledge. I was once bullish on the idea of using "value-added methods" for assessing teacher effectiveness. I have since realized that these measures, while valuable for large-scale studies, are seriously flawed for evaluating individual teachers, and that rigorous, ongoing assessment by teaching experts serves everyone better. Indeed, reviews by the National Research Council, the RAND Corp., and the Educational Testing Service have all concluded that value-added estimates of teacher effectiveness should not be used to make high-stakes decisions about teachers. Why?

"We opted out, not for us, but for the other students who are subjected to this testing regimen. This is our way of protesting for the students with disabilities, students whose primary language is not English, and students who were administered a test they were not prepared for. The tests are not developmentally appropriate, have not been properly field-tested and take away from other critical areas of education that are quickly dwindling - like art and music. We also are opposed to student test results being tied to teacher performance."

"This was written by sophomore Jack Eiselt, who attends Myers Park High School in the Charlotte-Mecklenburg School District in North Carolina. This year the district launched a new testing regime so that students take standardized tests in every subject so that all teachers can be evaluated based in part on the test scores of their students. Students in every grade helped field-test a total of 52 new tests this spring, kindergarteners included. This piece appeared in the Charlotte Observer."

"Leonie Haimson is fed up with the line that the mainstream media has taken about education controversies. Reporters usually think that every protest is organized by the unions, defending their self-interest, and they are warring with high-minded reformers. She says this is balderdash! (Sorry, Leonie, my word, not yours.)   If parents hold a protest against high-stakes testing and against test-based teacher evaluations (which causes more time to be devoted to testing), most reporters will say the union made them do it, the union doesn't want to be held accountable.   Well, guess what? The unions are not leading the Opt Out movement. Many teachers support it, because they know how pointless the new tests are, but the great majority of people leading the movement are parents. They don't want their children to be pressured by fear of the Big Standardized Test, they don't want them to be ranked and labeled, they don't want them to hate school because of the endless test prep."

Under the existing high-school testing system, students have to pass a single test that covers math and English. The new tests would also cover math and English, and the administration is reviewing recommendations that students pass tests in social studies and science, for a total of 12 tests.

...In light of current controversies around testing and teacher evaluation, let's do a little thought experiment. How would Miss Snug have handled this lesson if it were occurring just before a round of standardized testing? Would she not have had to interrupt the children's speculations and instructed them that actual circumstances in word problems must be completely disregarded, because the point is to arrive at the answer the test designers have in mind? After all, how could test designers anticipate the lines of thought that spontaneously erupted in her classroom? Real life, and real thought, are too complicated to be foreseen - and so need to be put aside at testing time.

A memo has recently surfaced in which the New York State Department of Education appears to encourage educators to mislead students about upcoming standardized field tests meant to "provide the data necessary to ensure the validity and reliability of the New York State Testing program." "Students should not be informed of the connection between these field tests and State assessments," the memo reads. "The field tests should be described as brief tests of achievement in the subject."

My four children have all attended public schools in our middle-class suburban district. When my oldest was in fourth grade, in 1998, he took the state tests, and I was not even aware of it. Later, he said the tests were kind of fun; he got to miss his regular classes. Six years later, in 2004, our daughter was in fourth grade. Long before the state tests, a letter came home. Prep classes were being offered before and after school. While the sessions were not mandatory, students were strongly urged to attend. Eventually the results were printed in our local newspaper. The news was grim; the nearby districts, in wealthier towns, had creamed us. The following year, our middle school added a mandatory course to prep for the state English test. That 1998/2004 divide - what happened in the interim was the 2002 No Child Left Behind law - should be kept in mind when analyzing a new, widely publicized study that closely tracked 2.5 million students over 20 years to determine whether teachers who helped raise children's test scores have a lasting effect on their lives. The researchers conclude that having such a teacher improved students' odds of going to a good college, the quality of the neighborhoods where they lived and their lifetime earnings.

"In all my many posts about the interpretation of state testing data, it seems that I may have failed to articulate one major implication, which is almost always ignored in the news coverage of the release of annual testing data. That is: raw, unadjusted changes in student test scores are not by themselves very good measures of schools' test-based effectiveness. In other words, schools can have a substantial impact on performance, but student test scores also increase, decrease or remain flat for reasons that have little or nothing to do with schools."

This report finds that the exclusion of thousands of students with disabilities from reported Connecticut Mastery Test results has distorted reported trends in test scores. Following test scores from year to year in the same grade, the study finds that statewide improvements in standard Connecticut Mastery Test (CMT) scores reported by the Connecticut State Department of Education (SDE) between 2008 and 2009 -- the period of the largest reported gains -- were largely the result of the exclusion of students with disabilities from these standard test results, rather than overall improvements in performance. For example, 84% of the reported improvement in 4th grade math proficiency between 2008 and 2009 and 69% of the improvement in 8th grade reading proficiency could be attributed to the exclusion of these students. Much of the reported improvements in later years could also be attributed to this exclusion, though there were some modest overall gains as well.

Coherence is the core principle underlying standards-based educational reforms. Assessments aligned with content standards are designed to guide instruction and raise achievement. The authors investigate the coherence of standards-based reform's key instruments using the Surveys of Enacted Curriculum. Analyzing 138 standards-assessment pairs spread across grades and the three No Child Left Behind tested subjects, the authors find that roughly half of standards content is tested on the corresponding test and roughly half of test content corresponds to the standards. A moderate proportion of test content is at the wrong level of cognitive demand as compared to the corresponding standards, and vice versa. Between 17% and 27% of content on a typical test covers topics not mentioned in the corresponding standards. Policy and research implications are discussed.

Uneducated Guesses challenges everything our policymakers thought they knew about education and education reform, from how to close the achievement gap in public schools to admission standards for top universities. In this explosive book, Howard Wainer uses statistical evidence to show why some of the most widely held beliefs in education today--and the policies that have resulted--are wrong. He shows why colleges that make the SAT optional for applicants end up with underperforming students and inflated national rankings, and why the push to substitute achievement tests for aptitude tests makes no sense. Wainer challenges the thinking behind the enormous rise of advanced placement courses in high schools, and demonstrates why assessing teachers based on how well their students perform on tests--a central pillar of recent education reforms--is woefully misguided. He explains why college rankings are often lacking in hard evidence, why essay questions on tests disadvantage women, why the most grievous errors in education testing are not made by testing organizations--and much more.

Test-based accountability rests on the assumption that accountability for scores on tests will provide needed incentives for teachers to improve student performance. Evidence shows, however, that simple test-based accountability can generate perverse incentives and seriously inflated scores. This paper discusses the logic of achievement tests, issues that arise in using them as proxy indicators of educational quality, and the mechanism underlying the inflation of scores. It ends with suggestions, some speculative, for improving the incentives faced by teachers by modifying systems of student assessment and combining them with numerous other measures, many of which are more subjective than are test scores.

It has been a challenging week for many third- through eighth-grade public school students in New York City, as they have started their days on Tuesday, Wednesday and Thursday with the federally mandated English Language Arts exams. But as Gotham Schools reported on Wednesday, the week has been especially challenging for some students with special needs. This year, test-taking time has doubled for all students. For those students with disabilities who are given more time to complete the tests, "testing can stretch as long as three hours on each day of testing. That means the students could spend more than half of the school day - and more than 18 hours total - on state exams this week and next," Jessica Campbell reports for Gotham.

The recent Pineapple and the Hare fiasco does more than identify a daft reading passage on New York State's 8th grade English Language Arts test. Education Commissioner John King scrapped the selection and its six multiple-choice items, admitting they were "ambiguous," when the questions became public last week. The episode opens the door to discussing how the 2012 exams were put together. The State Education Department signed a five-year, $32 million agreement with NCS Pearson to develop English Language Arts and math assessments in grades three to eight. In fact, math testing was administered over three days this week for 1.2 million students. Pearson has grown immensely over the last decade, securing contracts with many states required to test students under the No Child Left Behind Act. This year it succeeded CTB/McGraw-Hill as New York's test vendor.

A recent article in The Chronicle of Higher Education suggests that students cheat more in online than in face-to-face classes. The article tells the story of Bob Smith (not his real name, obviously), who was a student in an online science course.  Bob logged in once a week for half an hour in order to take a quiz. He didn't read a word of his textbook, didn't participate in discussions, and still he got an A. Bob pulled this off, he explained, with the help of a collaborative cheating effort. Interestingly, Bob is enrolled at a public university in the U.S., and claims to work diligently in all his other (classroom) courses. He doesn't cheat in those courses, he explains, but with a busy work and school schedule, the easy A is too tempting to pass up. Bob's online cheating methods deserve some attention. He is representative of a population of students that have striven to keep up with their instructor's efforts to prevent cheating online. The tests were designed in a way that made cheating more difficult, including limited time to take the test, and randomized questions from a large test bank (so that no two students took the exact same test). But the design of the test had two potential flaws

""Odds-beating charter school." Those words are like an impenetrable shield for those who operate such places. They are also the holy grail of the education reform movement, which is constantly seeking shortcuts to radically increase measures of educational achievement, which these days is pretty much defined by increased math and language test scores. One problem with radical test score gains, as many researchers have noted, is that miraculous improvements in test scores over short periods of time are more often the result of cheating, student skimming, or other test manipulation. We've seen this pattern repeated all over the nation, starting with the so-called Texas Miracle under former US education secretary Rod Paige's oversight."

In a provocative and influential paper, Jesse Rothstein (2010) finds that standard value-added models (VAMs) suggest implausible future teacher effects on past student achievement, a finding that obviously cannot be viewed as causal. This is the basis of a falsification test (the Rothstein falsification test) that appears to indicate bias in VAM estimates of current teacher contributions to student learning. More precisely, the falsification test is designed to identify whether or not students are effectively randomly assigned conditional on the covariates included in the model. Rothstein's finding is significant because there is considerable interest in using VAM teacher effect estimates for high-stakes teacher personnel policies, and the results of the Rothstein test cast considerable doubt on the notion that VAMs can be used fairly for this purpose. However, in this paper, we illustrate-theoretically and through simulations-plausible conditions under which the Rothstein falsification test rejects VAMs even when students are randomly assigned, conditional on the covariates in the model, and even when there is no bias in estimated teacher effects.