Group items tagged

"February 13, 2011 Scholars Question New Book's Gloom on Education Doubts are raised about study behind 'Academically Adrift' Scholars of Education Question the Limits of 'Academically Adrift' By David Glenn It has been a busy month for Richard Arum and Josipa Roksa. In mid-January, the University of Chicago Press published their gloomy account of the quality of undergraduate education, Academi­cally Adrift: Limited Learning on College Campuses. Since then the two sociologists have been through a torrent of radio interviews and public lectures. In the first days after the book's release, they had to handle a certain amount of breathless reaction, both pro and con, from people who hadn't actually read it. But now that more people in higher education have had time to digest their arguments, sophisticated conversations are developing about the study's lessons and about its limitations. Many college leaders are praising the ambition of Mr. Arum and Ms. Roksa's project, and some say they hope the book will focus new attention on the quality of undergraduate instruction. When the authors spoke last month at the annual meeting of the Association of American Colleges and Universities, in San Francisco, the ballroom far overfilled its capacity, and they were introduced as "rock stars." But three lines of skepticism have also emerged. Fiirst, some scholars say that Academically Adrift's heavy reliance on the Collegiate Learning Assessment, a widely used essay test that measures reasoning and writing skills, limits the value of the study. Second, some people believe the authors have not paid enough attention to the deprofessionalization of faculty work and the economic strains on colleges, factors that the critics say have played significant roles in the ero­sion of instructional quality. Third, some readers challenge the authors' position that the federal government should provide far more money to study the quality of college learning, but should not otherwise do mu

"December 12, 2010 Student Learning: Measure or Perish By Kevin Carey For the past three months, The Chronicle's reporters have been writing a series of articles collectively titled Measuring Stick, describing the consequences of a higher-education system that refuses to consistently measure how much students learn. From maddening credit-transfer policies and barely regulated for-profit colleges to a widespread neglect of teaching, the articles show that without information about learning, many of the most intractable problems facing higher education today will go unsolved. Failing to fill the learning-information deficit will have many consequences: * The currency of exchange in higher education will continue to suffer from abrupt and unpredictable devaluation. Students trying to assemble course credits from multiple institutions into a single degree-that is, most students-frequently have their credits discounted for no good reason. That occurs not only when students transfer between the two- and four-year sectors, or when the institutions involved have divergent educational philosophies. A student trying to transfer credits from an introductory technical-math course at Bronx Community College to other colleges within the City University of New York system, for example, would be flatly denied by five institutions and given only elective credit by three others. John Jay College of Criminal Justice, by contrast, would award the student credit for an introductory modern-math course acceptable for transfer by every CUNY campus, including Bronx Community College-except that BCC would translate that course into trigonometry and college algebra, not technical math. Students who emerge from this bureaucratic labyrinth should be awarded credit in Kafka studies for their trouble. Credit devaluation, which wastes enormous amounts of time, money, and credentialed learning every year, is rooted in mistrust. Because colleges don't know what students in

"Measuring Student Learning: Many Tools November 23, 2010, 2:44 pm By David Glenn Suppose that you've served on a faculty committee that has devised a list of collegewide learning objectives for your undergraduates. You don't want that list to just sit there on a Web site as a testimony to your college's good intentions. (Right?) You want to take reasonable steps to measure whether your students are actually meeting the goals you've defined. How best to do that is, of course, a highly contested question. Some scholars urge colleges to use nationally normed tests, like the Collegiate Learning Assessment, that attempt to capture students' critical-thinking and analytic-writing skills. Others say it is better to use student portfolios that allow students to demonstrate their skills in the context of their course work. (For a taste of that debate, see this post and the comments it engendered.) Charles Blaich, director of Wabash College's Center of Inquiry in the Liberal Arts, advocates an all-of-the-above approach. Colleges should use as many reasonable kinds of data as they can get their hands on, he says. The CLA and other national tests can be powerful tools, but they can't possibly capture a college's full range of learning objectives."

"January 22, 2012 States Push Even Further to Cut Spending on Colleges By Eric Kelderman For nearly four years, governors and state legislators have focused on little else in higher education but cutting budgets to deal with historic gaps in revenue. Now, with higher-education support at a 25-year low, lawmakers are considering some policy changes that have been off-limits in the past, such as consolidating campuses and eliminating governing boards. Such proposals reflect the reality that, in most states, money for higher education will be constrained for the foreseeable future. Systems in Georgia and New York have already taken the unusual step of combining campuses under a single president. Other states, such as Ohio, are talking about giving institutions more freedom from state regulations, although for college administrators there's a trade-off: They would get more flexibility but even less state money. On the agenda in many statehouses this year will be bills that would tie higher-education appropriations to the completion rates of students at public colleges. Such performance-based models, which have had a mixed record in recent decades, are again popular with lawmakers trying to squeeze the most out of every tax dollar and to reward colleges that are more efficient at producing graduates. Related Content State Support For Higher Education Falls 7.6% in 2012 Fiscal Year Calif. Governor Goes After For-Profits With Limits on Cal Grants Legislators aren't demanding that colleges be more cost-efficient just to reduce spending on higher education, says Travis J. Reindl, a higher-education researcher for the bipartisan National Governors Association. They also want to keep colleges affordable for students. "We'll still be talking about money, money, money," Mr. Reindl says of the legislative sessions ahead. "Governors are increasingly interested in how the money is being spent by higher education ... and how much of that money is going to come out of

"January 24, 2012 Colleges Can Take 4 Steps to Assure Quality, Group Says By Dan Berrett Increasing the percentage of college graduates in the United States has become a collective aspiration of policy makers, advocates for higher education, and President Obama. But this push for quantity will mean little if colleges cannot demonstrate the quality of the degrees they confer, says an advocacy group. The group, the New Leadership Alliance for Student Learning and Accountability, released today a set of guidelines it says will help colleges assess and improve student achievement and, in the process, better demonstrate the quality of their offerings. The guidelines are being presented at the Council for Higher Education Accreditation's annual meeting in Washington, with endorsements from 27 organizations, chiefly accreditors and associations. The guidelines stake out four broad principles of assessment and accountability for a college to follow: setting ambitious goals for the outcomes of undergraduate education; gathering evidence about how the institution is faring in pursuit of those outcomes; using that evidence to improve learning; and sharing the results. The essential idea is to clearly articulate and make intentional the objectives that guide student learning, said David C. Paris, executive director of the alliance. "That's our goal," he said, "an evidence-based profession." The alliance was started in 2009 by several higher-education leaders and foundations to respond to growing calls for accountability in the sector. The assumption was that colleges needed to define how they would substantiate student learning-or lawmakers would do it for them. The new guidelines expand on the alliance's previous efforts, including a statement of principles to guide student learning, which were released in 2008, and a pledge by more than 100 college presidents to take steps at their institutions that are largely identical to the ones set out in the new guidelines. O

"Ready for Change.edu? January 11, 2012 - 3:00am By Paul Fain Andrew S. Rosen takes the long view when talking about higher education. As CEO of Kaplan, Inc., he often defends the role of for-profit colleges in an evolving marketplace, peppering versions of his stump speech with tales about the creation of public universities and community colleges. His point is that some skepticism about for-profits is similar to the snobbery those older sectors faced from elite private higher education. Rosen goes further in his debut book, Change.edu: Rebooting for the New Talent Economy, which attempts to paint a picture of higher education's future as well as its history. He also takes a turn as a journalist of sorts - an interesting twist for the former general counsel of the Washington Post Co. - writing about his campus visits to other institutions, a couple of which are Kaplan competitors. The book is ambitious in its scope, particularly for an author with obvious vested interests. But most reviewers have given Rosen high marks. Kirkus Reviews writes: "Incredibly, his argument never comes off as self-serving; the author's thorough exploration of 'Harvard Envy' and the rise of 'resort' campuses is both fascinating and enlightening." Rosen recently answered questions over e-mail about his book, which was released by Kaplan Publishing. Q: The book arrives amid a series of challenges for your industry. What did you hope to accomplish by writing it? A: I've spent most of my life studying or working in education, with students of all ages and preparation levels: top students from America's most elite institutions and working adults and low-income students who have few quality choices to change their lives. I've come to see how the American higher education system (as with K-12) is profoundly tilted in favor of those who already have advantages. Our society keeps investing more and more in the relatively small and unchanging number of students who have the privil

"The Extraordinary Value of Great Universities Richard Florida Dec 15, 2011 2 Comments The Extraordinary Value of Great Universities Reuters Share Print Email The United States is home to more than a third of the world's top 400 research universities. But how exactly do universities factor into the wealth, innovation, and economic competitiveness of their host nations? To get at this, my colleague Charlotta Mellander and I looked into the statistical associations between a nation's concentration of leading universities and broader measures of economic competitiveness, innovation, human capital and social well-being. We based our analysis on a statistical technique that enables us to control for the effects of population size. While correlation is not causation (none of these findings prove that anything more than an association exists) the results are nonetheless striking. In fact, they number among the very strongest I have ever seen in this type of analysis. The concentration of great universities in a nation is extraordinarily closely related to its economic competitiveness. It is closely associated with economic output per capita (.74), total factor productivity (.77) and overall competitiveness (.71) based on the Global Competitiveness Index developed by Harvard's Michael Porter. Universities are also a key force in technology. A nation's concentration of leading universities is closely associated with its level of innovation, measured as patents (.78) and its research and development expenditures (.74). While Stanford's role in Silicon Valley-style high-technology entrepreneurship is the stuff of legend, universities are closely associated with the entrepreneurial level of nations. The concentration of world-class universities is closely associated with a nation's level of entrepreneurship as measured on the Global Entrepreneurship Monitor (.69). Technology is one key factor in economic competitiveness, but a nation'

"December 12, 2010 Is Your Psychology 102 Course Any Good? Here are 22 ways to measure quality - but some of these measures have quality issues of their own. By David Glenn In The Chronicle's "Measuring Stick" series this year, we have looked at debates about how to gauge the quality of departments or entire universities. In this final week, we are looking at the individual course, higher education's basic component. We have sketched 22 potentially useful ways to assess a course's quality. Some of them are commonplace, and some are just emerging. We focus on one section of Psychology 102 at an imaginary university. For each of the 22 measures, the table below explains why it might matter; how easy it typically is for the public to find this kind of information about a course; and the potential limits and pitfalls of using the method."

"December 12, 2010 Is Your Psychology 102 Course Any Good? Here are 22 ways to measure quality - but some of these measures have quality issues of their own. By David Glenn In The Chronicle's "Measuring Stick" series this year, we have looked at debates about how to gauge the quality of departments or entire universities. In this final week, we are looking at the individual course, higher education's basic component. We have sketched 22 potentially useful ways to assess a course's quality. Some of them are commonplace, and some are just emerging. We focus on one section of Psychology 102 at an imaginary university. For each of the 22 measures, the table below explains why it might matter; how easy it typically is for the public to find this kind of information about a course; and the potential limits and pitfalls of using the method."

"The Measuring Stick Finale: A Hawk and a Skeptic Walk Into a Bar … December 21, 2010, 12:38 pm By David Glenn As promised, this blog will expire with the calendar year (cue violins). In this last post, I've tried to distill some fundamental arguments about assessment and accountability in higher education. I've borrowed liberally from comments that readers have left here and elsewhere on The Chronicle's site. Many thanks to all of you for reading and arguing. Of course there are more than two sides to these debates. In that respect, what follows is pathetically reductive. But I've tried not to put my thumb on the scale on behalf of either of these characters. I've tried to convey the strongest cases on each side of an admittedly-crudely-drawn line. (If I've failed to do that, you should of course call me out in the comments.) The scene: Friday, 7:45 p.m. A bar on the outskirts of a moderately selective public university. The décor and the jukebox appeal to disillusioned 36-year-old faculty members and a few graduate students. The fraternities leave this place alone. Accountability Skeptic: Did you see that memo from the dean today? He's hired some consultant to teach us how to "design learning outcomes" for our students. I can't imagine a bigger waste of time and money. And I don't think the dean even believes in this stuff himself. I think he's just trying to keep the accreditors off his back. Accountability Hawk: Don't be so cynical. Tuition and fees here have gone up by more than 50 percent since 2000. Students are taking on miserable levels of debt to be in our classrooms. They deserve to have faculty members who are focused on their learning-and that means that we need some kind of common understandings in our departments about the knowledge and skills students are supposed to be picking up. Skeptic: Listen. I do focus on my students. I assess their learning every week. It's called grading. I get the feeling that the dea

"Online Course Provider, StraighterLine, to Offer Critical-Thinking Tests to Students January 19, 2012, 12:29 pm By Jeff Selingo As alternatives to the college diploma have been bandied about recently, one question always seems to emerge: How do you validate badges or individual classes as a credential in the absence of a degree? One company that has been hailed by some as revolutionizing introductory courses might have an answer. The company, StraighterLine, announced on Thursday that beginning this fall it will offer students access to three leading critical-thinking tests, allowing them to take their results to employers or colleges to demonstrate their proficiency in certain academic areas. The tests-the Collegiate Learning Assessment, sponsored by the Council for Aid to Education, and the Proficiency Profile, from the Educational Testing Service-each measure critical thinking and writing, among other academic areas. The iSkills test, also from ETS, measures the ability of a student to navigate and critically evaluate information from digital technology. Until now, the tests were largely used by colleges to measure student learning, but students did not receive their scores. That's one reason that critics of the tests have questioned their effectiveness since students have little incentive to do well. Burck Smith, the founder and chief executive of StraighterLine, which offers online, self-paced introductory courses, said on Thursday that students would not need to take classes with StraighterLine in order to sit for the tests. But he hopes that, for students who do take both classes and tests, the scores on the test will help validate StraighterLine courses. StraighterLine doesn't grant degrees and so can't be accredited. It depends on accredited institutions to accept its credits, which has not always been an easy task for the company. "For students looking to get a leg up in the job market or getting into college," Mr. Smith said, "t

"September 19, 2010 A Measure of Education Is Put to the Test Results of national exam will go public in 2012 By David Glenn You have 90 minutes to complete this test. Here is your scenario: You are the assistant to a provost who wants to measure the quality of your university's general-education program. Your boss is considering adopting the Collegiate Learning Assessment, or CLA, a national test that asks students to demonstrate their ability to synthesize evidence and write persuasively. The CLA is used at more than 400 colleges. Since its debut a decade ago, it has been widely praised as a sophisticated alternative to multiple-choice tests. At some colleges, its use has helped spark sweeping changes in instruction and curriculum. And soon, many more of the scores will be made public. But skeptics say the test is too detached from the substantive knowledge that students are actually expected to acquire. Others say those who take the test have little motivation to do well, which makes it tough to draw conclusions from their performance."

"A Final Word on the Presidents' Student-Learning Alliance November 22, 2010, 1:36 pm By David Glenn Last week we published a series of comments (one, two, three) on the Presidents' Alliance for Excellence in Student Learning and Accountability. Today we're pleased to present a reply from David C. Paris, executive director of the presidential alliance's parent organization, the New Leadership Alliance for Student Learning and Accountability: I was very pleased to see the responses to the announcement of the Presidents' Alliance as generally welcoming ("commendable," "laudatory initiative," "applaud") the shared commitment of these 71 founding institutions to do more-and do it publicly and cooperatively-with regard to gathering, reporting, and using evidence of student learning. The set of comments is a fairly representative sample of positions on the issues of evidence, assessment, and accountability. We all agree that higher education needs to do more to develop evidence of student learning, to use it to measure and improve our work, and to be far more transparent and accountable in reporting the results. The comments suggest different approaches-and these differences are more complementary than contradictory-to where we should focus our efforts and how change will occur. I'd suggest that none of us has the answer, and while each of these approaches faces obstacles, each can contribute to progress in this work. For William Chace, Cliff Adelman, and Michael Poliakoff, the focus should be on some overarching measures or concepts that will clearly tell us, our students, and the public how well we are doing. Obtaining agreement on a "scale and index," or the appropriate "active verbs" describing competence, or dashboards and other common reporting mechanisms will drive change by establishing a common framework for evaluation. Josipa Roksa, on the other hand, suggests that real change will only happen from the ground up. Facu

"If and When Money Matters: The Relationships among Educational Expenditures, Student Engagement, and Students' Learning Outcomes Issue/Topic: Finance--Does Money Matter?; Postsecondary Finance--Efficiency/Performance-Based Funding Author(s): Pike, Gary; Kuh, George; Smart, John; Ethington, Corinna; McCormick, Alexander Organization(s): University of Memphis; Indiana University Purdue University at Indianapolis; University of Indiana Publication: Research in Higher Education Published On: 9/18/2010 Background: Past research on expenditures and college outcomes has been characterized by weak and contradictory findings. Surprising little is known about whether and how "money matters" to desired outcomes of college. It seems reasonable to expect that combined expenditures for instruction, academic support, student services, and institution support would be positively and directly related to student engagement, but indirectly related to student learning. Purpose: To examine the relationships among educational expenditures, student engagement and learning outcomes for first-year students and seniors. Findings/Results: * Expenditures were significantly and positively related to...first-year students' self-reported cognitive outcomes (in areas such as general education, writing and speaking effectively, quantitative analysis, and critical thinking). * Expenditures were not significantly related to first-year students' non-cognitive development (as measured by responses to questions concerning self-understanding, working with others, developing ethical standards, and civic/community engagement). * For a wider range of learning objectives (e.g., academic challenge, collaborative learning, educational enrichment), the relationship between expenditures and outcomes was indirect and mediated by student engagement variables. * Between-institution differences were very small compared to the differences among students within institutions. * All of the enga

"The Measuring Stick Finale: a Hawk and a Skeptic Walk Into a Bar … December 21, 2010, 12:38 pm By David Glenn As promised, this blog will expire with the calendar year (cue violins). In this last post, I've tried to distill some fundamental arguments about assessment and accountability in higher education. I've borrowed liberally from comments that readers have left here and elsewhere on The Chronicle's site. Many thanks to all of you for reading and arguing. Of course there are more than two sides to these debates. In that respect, what follows is pathetically reductive. But I've tried not to put my thumb on the scale on behalf of either of these characters. I've tried to convey the strongest cases on each side of an admittedly-crudely-drawn line. (If I've failed to do that, you should of course call me out in the comments.) The scene: Friday, 7:45 p.m. A bar on the outskirts of a moderately selective public university. The décor and the jukebox appeal to disillusioned 36-year-old faculty members and a few graduate students. The fraternities leave this place alone. Accountability Skeptic: Did you see that memo from the dean today? He's hired some consultant to teach us how to "design learning outcomes" for our students. I can't imagine a bigger waste of time and money. And I don't think the dean even believes in this stuff himself. I think he's just trying to keep the accreditors off his back. Accountability Hawk: Don't be so cynical. Tuition and fees here have gone up by more than 50 percent since 2000. Students are taking on miserable levels of debt to be in our classrooms. They deserve to have faculty members who are focused on their learning-and that means that we need some kind of common understandings in our departments about the knowledge and skills students are supposed to be picking up."

"December 19, 2010 2 Studies Shed New Light on the Meaning of Course Evaluations By David Glenn Under the mandate of a recently enacted state law, the Web sites of public colleges and universities in Texas will soon include student-evaluation ratings for each and every undergraduate course. Bored and curious people around the planet-steelworkers in Ukraine, lawyers in Peru, clerical workers in India-will be able, if they're so inclined, to learn how students feel about Geology 3430 at Texas State University at San Marcos. But how should the public interpret those ratings? Are student-course evaluations a reasonable gauge of quality? Are they correlated with genuine measures of learning? And what about students who choose not to fill out the forms-does their absence skew the data? Two recent studies shed new light on those old questions. In one, three economists at the University of California at Riverside looked at a pool of more than 1100 students who took a remedial-mathematics course at a large university in the West (presumably Riverside) between 2007 and 2009. According to a working paper describing the study, the course was taught by 33 different instructors to 97 different sections during that period. The instructors had a good deal of freedom in their teaching and grading practices-but every student in every section had to pass a common high-stakes final exam, which they took after filling out their course evaluations. That high-stakes end-of-the-semester test allowed the Riverside economists to directly measure student learning. The researchers also had access to the students' pretest scores from the beginning of the semester, so they were able to track each student's gains. Most studies of course evaluations have lacked such clean measures of learning. Grades are an imperfect tool, as students' course ratings are usually strongly correlated with their grades in the course. Because of that powerful correlation, some studies have suggested that

"February 12, 2012 'Academically Adrift': The News Gets Worse and Worse 'Academically Adrift': The News Gets Worse and Worse 1 Michael Morgenstern for The Chronicle Enlarge Image By Kevin Carey In the last few months of 2010, rumors began circulating among higher-education policy geeks that the University of Chicago Press was about to publish a new book written by a pair of very smart sociologists who were trying to answer a question to which most people thought they already knew the answer: How much do students learn while they're in college? Their findings, one heard, were ... interesting. The book, Academically Adrift, by Richard Arum and Josipa Roksa, fulfilled that promise-and then some. It was no surprise that The Chronicle gave prominent coverage to the conclusion that "American higher education is characterized by limited or no learning for a large proportion of students," but few people anticipated that the book would become the rare piece of serious academic scholarship that jumps the fence and roams free into the larger culture. Vanity Fair used space normally allotted to Kennedy hagiography to call it a "crushing exposé of the heretofore secret society known as 'college.'" The gossip mavens at Gawker ran the book through their patented Internet cynicism machine and wrote that "To get a college degree, you must go into a soul-crushing amount of debt. And what do you get for all that money? Not learning." The New Yorker featured Academically Adrift in a typically brilliant essay by Louis Menand. In one of her nationally syndicated columns, Kathleen Parker called the book a "dense tome" while opining that the failure of higher education constituted a "dot-connecting exercise for Uncle Shoulda, who someday will say-in Chinese-'How could we have let this happen?'" Her response proved that Kathleen Parker has a gift for phrasing and did not actually read the book, whose main text runs to only 144 concise and well-argued pages. But the definitive

"Big Data's Arrival February 1, 2012 - 3:00am By Paul Fain New students are more likely to drop out of online colleges if they take full courseloads than if they enroll part time, according to findings from a research project that is challenging conventional wisdom about student success. But perhaps more important than that potentially game-changing nugget, researchers said, is how the project has chipped away at skepticism in higher education about the power of "big data." Researchers have created a database that measures 33 variables for the online coursework of 640,000 students - a whopping 3 million course-level records. While the work is far from complete, the variables help track student performance and retention across a broad range of demographic factors. The data can show what works at a specific type of institution, and what doesn't. That sort of predictive analytics has long been embraced by corporations, but not so much by the academy. The ongoing data-mining effort, which was kicked off last year with a $1 million grant from the Bill and Melinda Gates Foundation, is being led by WCET, the WICHE Cooperative for Educational Technologies. Project Participants American Public University System Community College System of Colorado Rio Salado College University of Hawaii System University of Illinois-Springfield University of Phoenix A broad range of institutions (see factbox) are participating. Six major for-profits, research universities and community colleges -- the sort of group that doesn't always play nice -- are sharing the vault of information and tips on how to put the data to work. "Having the University of Phoenix and American Public University, it's huge," said Dan Huston, coordinator of strategic systems at Rio Salado College, a participant. According to early findings from the research, at-risk students do better if they ease into online education with a small number of courses, which flies in the face of widely-he

Charles Kolb, President, Committee for Economic Development January 11, 2011 03:35 PM Reforming American Postsecondary Education Are we about to enter an era of postsecondary education reform comparable to what we've seen in the K-12 arena for almost 30 years? In 1983, the U.S. Department of Education released perhaps its most famous and widely read report, "A Nation At Risk." Referring to "a rising tide of mediocrity" in America's elementary and secondary school system, "A Nation At Risk" described the stark challenges faced by American elementary and secondary education. The report became an immediate catalyst for the school reform movement of the last 27 years. That reform movement included initiatives such as education secretary William Bennett's "Wall Chart of State Performance Indicators," the 1989 Charlottesville education summit between President George H.W. Bush and the nations' governors, the subsequent bipartisan national education goals effort that spanned the first Bush and Clinton Administrations, George W. Bush's "No Child Left Behind Act," and now President Obama's "Race to the Top" challenge for state structural reform. As with many K-12 education reform efforts, change has been hesitant, often rancorous, and has achieved mixed results. Nonetheless, there has been steady progress on standards, accountability, measurements and assessment, and a growing consensus about what our children need to know and how we should measure their achievements as they progress toward high-school graduation. What is strikingly absent is that throughout this period of K-12 activity, American postsecondary education has received a "pass." Not a passing grade -- just a pass. There has been precious little discussion about what our young people should be learning in their postsecondary education experience. The typical postsecondary-education debate in Washington and around the country has concerned access and funding. These topics are certainly important, but they h

"Measuring College-Teacher Quality January 13, 2011, 10:40 am By Kevin Carey David Glenn's Chronicle article on using course sequence grades to estimate teacher quality in higher education illustrates a crucial flaw in the way education researchers often think about the role of evidence in education practice. The article cites a recent study of Calculus grades in the Air Force Academy. All students there are required to take Calculus I and II. They're randomly assigned to instructors who use the same syllabus. Students all take the same final, which is collectively graded by a pool of instructors. These unusual circumstances control for many external factors that might otherwise complicate an analysis of teacher quality. The researchers found that students taught by permanent faculty got worse grades in Calculus I than students taught by short-term faculty. But the pattern reversed when those students went on to Calculus II-those taught by full-time faculty earned better grades in the more advanced course, suggesting that short-term faculty might have been "teaching to the test" at the expense of deeper conceptual understanding. Students taught by full-time faculty were also more likely to enroll in upper-level math in their junior and senior years. In addition, the study found that student course evaluations were positively correlated with grades in Calculus I but negatively correlated with grades in Calculus II."