Group items tagged

Obama administration will be tough on colleges because its officials value higher education and believe it needs to perform much better, and successfully educate many more students, to drive the American economy.

In her own speech to the Higher Learning Commission’s members on Sunday, Sylvia Manning, the group’s president, cited several signs that the new administration seemed willing to delve into territory that not long ago would have been viewed as off-limits to federal intrusion. Among them: A recently published “draft” of a guide to accreditation that many accrediting officials believe is overly prescriptive. A just-completed round of negotiations over proposed rules that deal with the definition of a “credit hour” and other issues that touch on academic quality -- areas that have historically been the province of colleges and their faculties. And, of special relevance for the Higher Learning Commission, a trio of critical letters from the Education Department’s inspector general challenging the association’s policies and those of two other regional accreditors on key matters -- and in North Central’s case, questioning its continued viability. With that stroke, Manning noted, the department’s newfound activism “has come to the doorstep, or into the living room, of HLC.”

Pressure to measure student learning -- to find out which tactics and approaches are effective, which create efficiency without lowering results -- is increasingly coming from what Broad called the Obama administration's "kitchen cabinet," foundations like the Lumina Foundation for Education (which she singled out) to which the White House and Education Department are increasingly looking for education policy help.

She cited an October speech in which the foundation's president, Jamie P. Merisotis, said that student learning should be recognized as the "primary measure of quality in higher education," and heralded the European Union's Bologna process as a potential path for making that so

we cannot lay low and hope that the glare of the spotlight will eventually fall on others," Broad told the Higher Learning Commission audience.

While higher ed groups have been warned repeatedly that they must act before Congress next renews the Higher Education Act -- a process that will begin in earnest in two or three years -- the reality is that politicians in Washington no longer feel obliged to hold off on major changes to higher education policy until that main law is reviewed. Congress has passed "seven major pieces of legislation" related to higher education in recent years, and "I wish I could tell you that the window is open" until the next reauthorization, Broad said. "But we cannot presume that we have the luxury of years within which to get our collective house in order. We must act quickly."

But where will such large-scale change come from? The regional accreditors acting together to align their standards? Groups of colleges working together to agree on a common set of learning outcomes for general education, building on the work of the American Association of Colleges and Universities? No answers here, yet.

Besides the CLA, which is sponsored by the Council for Aid to Education, other tests that participants in the voluntary system may use are the Collegiate Assessment of Academic Proficiency, from ACT Inc., and the Measure of Academic Proficiency and Progress, offered by the Educational Testing Service.

The test has sometimes been criticized for relying on a cross-sectional system rather than a longitudinal model, in which the same students would be tested in their first and fourth years of college.

there have long been concerns about just how motivated students are to perform well on the CLA.

Mr. Hosch suggested that small groups of similar colleges should create consortia for measuring student learning. For example, five liberal-arts colleges might create a common pool of faculty members that would evaluate senior theses from all five colleges. "That wouldn't be a national measure," Mr. Hosch said, "but it would be much more authentic."

Mr. Shavelson said. "The challenge confronting higher education is for institutions to address the recruitment and motivation issues if they are to get useful data. From my perspective, we need to integrate assessment into teaching and learning as part of students' programs of study, thereby raising the stakes a bit while enhancing motivation of both students and faculty

"I do agree with his central point that it would not be prudent to move to an accountability system based on cross-sectional assessments of freshmen and seniors at an institution," said Mr. Arum, who is an author, with Josipa Roksa, of Academically Adrift: Limited Learning on College Campuses, forthcoming from the University of Chicago Press

"The Board of Regents here saw that it would be an important test because it measures analytical ability, problem-solving ability, critical thinking, and communication. Those are the skills that you want every undergraduate to walk away with." (Other large systems that have embraced the CLA include California State University and the West Virginia system.)

value added

We began by administering a retired CLA question, a task that had to do with analyzing crime-reduction strategies,

performance task that mirrors the CLA

Mr. Ernsting and Ms. McConnell are perfectly sincere about using CLA-style tasks to improve instruction on their campuses. But at the same time, colleges have a less high-minded motive for familiarizing students with the CLA style: It just might improve their scores when it comes time to take the actual test.

by 2012, the CLA scores of more than 100 colleges will be posted, for all the world to see, on the "College Portrait" Web site of the Voluntary System of Accountability, an effort by more than 300 public colleges and universities to provide information about life and learning on their campuses.

If familiarizing students with CLA-style tasks does raise their scores, then the CLA might not be a pure, unmediated reflection of the full range of liberal-arts skills. How exactly should the public interpret the scores of colleges that do not use such training exercises?

Trudy W. Banta, a professor of higher education and senior adviser to the chancellor for academic planning and evaluation at Indiana University-Purdue University at Indianapolis, believes it is a serious mistake to publicly release and compare scores on the test. There is too much risk, she says, that policy makers and the public will misinterpret the numbers.

most colleges do not use a true longitudinal model: That is, the students who take the CLA in their first year do not take it again in their senior year. The test's value-added model is therefore based on a potentially apples-and-oranges comparison.

freshman test-takers' scores are assessed relative to their SAT and ACT scores, and so are senior test-takers' scores. For that reason, colleges cannot game the test by recruiting an academically weak pool of freshmen and a strong pool of seniors.

students do not always have much motivation to take the test seriously

seniors, who are typically recruited to take the CLA toward the end of their final semester, when they can already taste the graduation champagne.

Of the few dozen universities that had already chosen to publish CLA data on that site, roughly a quarter of the reports appeared to include erroneous descriptions of the year-to-year value-added scores.

It is clear that CLA scores do reflect some broad properties of a college education.

Students' CLA scores improved if they took courses that required a substantial amount of reading and writing. Many students didn't take such courses, and their CLA scores tended to stay flat.

Colleges that make demands on students can actually develop their skills on the kinds of things measured by the CLA.

Mr. Shavelson believes the CLA's essays and "performance tasks" offer an unusually sophisticated way of measuring what colleges do, without relying too heavily on factual knowledge from any one academic field.

Politicians and consumers want easily interpretable scores, while colleges need subtler and more detailed data to make internal improvements.

The CLA is used at more than 400 colleges

Since its debut a decade ago, it has been widely praised as a sophisticated alternative to multiple-choice tests

users given completely random praise by a computer program liked it more than the same program without praise, even though they knew in advance the praise was meaningless.

Nass demonstrates, however, that people internalize praise and criticism differently—while we welcome the former, we really dwell on and obsess over the latter. In the criticism sandwich, then, "the criticism blasts the first list of positive achievements out of listeners' memory. They then think hard about the criticism (which will make them remember it better) and are on the alert to think even harder about what happens next. What do they then get? Positive remarks that are too general to be remembered"

And because we focus so much on the negative, having a similar number of positive and negative comments "feels negative overall"

The best strategy, he suggests, is "to briefly present a few negative remarks and then provide a long list of positive remarks...You should also provide as much detail as possible within the positive comments, even more than feels natural, because positive feedback is less memorable" (33).

The Change Process as Problem Solving and Problem Finding

What is not useful about this framework is that it does not allow for change efforts that begin with the organization in extremis

this framework is that it gives rise to thinking about a staged approach to changing things.

Change as a “How” Problem

Change as a “What” Problem

Change as a “Why” Problem

The Approach taken to Change Management Mirrors Management's Mindset

People in core units, buffered as they are from environmental turbulence and with a history of relying on adherence to standardized procedures, typically focus on “how” questions.

To summarize: Problems may be formulated in terms of “how,” “what” and “why” questions. Which formulation is used depends on where in the organization the person posing the question or formulating the problem is situated, and where the organization is situated in its own life cycle. “How” questions tend to cluster in core units. “What” questions tend to cluster in buffer units. People in perimeter units tend to ask “what” and “how” questions. “Why” questions are typically the responsibility of top management.

One More Time: How do you manage change? The honest answer is that you manage it pretty much the same way you’d manage anything else of a turbulent, messy, chaotic nature, that is, you don’t really manage it, you grapple with it. It’s more a matter of leadership ability than management skill. The first thing to do is jump in. You can’t do anything about it from the outside. A clear sense of mission or purpose is essential. The simpler the mission statement the better. “Kick ass in the marketplace” is a whole lot more meaningful than “Respond to market needs with a range of products and services that have been carefully designed and developed to compare so favorably in our customers’ eyes with the products and services offered by our competitors that the majority of buying decisions will be made in our favor.” Build a team. “Lone wolves” have their uses, but managing change isn’t one of them. On the other hand, the right kind of lone wolf makes an excellent temporary team leader. Maintain a flat organizational team structure and rely on minimal and informal reporting requirements. Pick people with relevant skills and high energy levels. You’ll need both. Toss out the rulebook. Change, by definition, calls for a configured response, not adherence to prefigured routines. Shift to an action-feedback model. Plan and act in short intervals. Do your analysis on the fly. No lengthy up-front studies, please. Remember the hare and the tortoise. Set flexible priorities. You must have the ability to drop what you’re doing and tend to something more important. Treat everything as a temporary measure. Don’t “lock in” until the last minute, and then insist on the right to change your mind. Ask for volunteers. You’ll be surprised at who shows up. You’ll be pleasantly surprised by what they can do. Find a good “straw boss” or team leader and stay out of his or her way. Give the team members whatever they ask for — except authority. They’ll generally ask only for what they really need in the way of resources. If they start asking for authority, that’s a signal they’re headed toward some kind of power-based confrontation and that spells trouble. Nip it in the bud! Concentrate dispersed knowledge. Start and maintain an issues logbook. Let anyone go anywhere and talk to anyone about anything. Keep the communications barriers low, widely spaced, and easily hurdled. Initially, if things look chaotic, relax — they are. Remember, the task of change management is to bring order to a messy situation, not pretend that it’s already well organized and disciplined.

The old encyclopedia of the Middle Ages and the Renaissance based its prestige on its claim to comprehensiveness. But by the middle of the sixteenth century, these claims had become very difficult for any single author or work to support. Ironically, as the plausibility of the old claims weakened, demand for the genre intensified. This is attested to by the great commercial success of the Cyclopaedia and by the still greater success of the renowned Encyclopédie of Diderot and d'Alembert. For the latter, just as for Chambers, the indexical format of the encyclopedic dictionary speaks to an epistemological urgency. In a world of rapid change, quick access to knowledge becomes as important as knowledge itself.

Taken together, these papers suggest that during the sixteenth and seventeenth centuries factors such as an increasing production and dissemination of books, developing networks of scientific communication, discoveries and innovations in the sciences, and new economic relationships all conspired to produce such quantities of new information that a substantial reorganization of the intellectual world was required.

cluster analysis,

nearly two-thirds of those surveyed were involved in the full range of scholarly activity they examined

Just over a third, however, stood out as focused almost solely on one of two types of scholarship: on teaching practices, or on using knowledge from their discipline to identify or solve societal problems.

pedagogy-focused scholars were found mainly at liberal-arts colleges and, compared with the general population surveyed, tended to be younger, heavily represented in history departments, and more likely to be female and untenured

Those focused on problem-solving were located mainly at research and doctoral institutions, and were evenly dispersed across disciplines and more likely than others surveyed to be male and tenured.

how faculty members rate those priorities are fairly consistent across academic disciplines,

The study was conducted by B. Jan Middendorf, acting director of Kansas State University's office of educational innovation and evaluation; Russell J. Webster, a doctoral student in psychology at Kansas State; and Steve Benton, a senior research officer at the IDEA Center

The IDEA Center, an education research group based at Kansas State University, has been spreading its particular course-evaluation gospel since 1975. The central innovation of the IDEA system is that departments can tailor their evaluation forms to emphasize whichever learning objectives are most important in their discipline.

(Roughly 350 colleges use the IDEA Center's system, though in some cases only a single department or academic unit participates.)

The new North Texas instrument that came from these efforts tries to correct for biases that are beyond an instructor's control. The questionnaire asks students, for example, whether the classroom had an appropriate size and layout for the course. If students were unhappy with the classroom, and if it appears that their unhappiness inappropriately colored their evaluations of the instructor, the system can adjust the instructor's scores accordingly.

The survey instrument, known as SALG, for Student Assessment of their Learning Gains, is now used by instructors across the country. The project's Web site contains more than 900 templates, mostly for courses in the sciences.

"So the ability to do some quantitative analysis of these comments really allows you to take a more nuanced and effective look at what these students are really saying."

Mr. Frick and his colleagues found that his new course-evaluation form was strongly correlated with both students' and instructors' own measures of how well the students had mastered each course's learning goals.

Elaine Seymour, who was then director of ethnography and evaluation research at the University of Colorado at Boulder, was assisting with a National Science Foundation project to improve the quality of science instruction at the college level. She found that many instructors were reluctant to try new teaching techniques because they feared their course-evaluation ratings might decline.

"Students are the inventory," Mr. Crumbley says. "The real stakeholders in higher education are employers, society, the people who hire our graduates. But what we do is ask the inventory if a professor is good or bad. At General Motors," he says, "you don't ask the cars which factory workers are good at their jobs. You check the cars for defects, you ask the drivers, and that's how you know how the workers are doing."

William H. Pallett, president of the IDEA Center, says that when course rating surveys are well-designed and instructors make clear that they care about them, students will answer honestly and thoughtfully.

In Mr. Bain's view, student evaluations should be just one of several tools colleges use to assess teaching. Peers should regularly visit one another's classrooms, he argues. And professors should develop "teaching portfolios" that demonstrate their ability to do the kinds of instruction that are most important in their particular disciplines. "It's kind of ironic that we grab onto something that seems fixed and fast and absolute, rather than something that seems a little bit messy," he says. "Making decisions about the ability of someone to cultivate someone else's learning is inherently a messy process. It can't be reduced to a formula."

She stressed that NSSE averages may no longer reflect any single reality of one type of faculty member. She challenged Paris’s description of powerful faculty members by noting that many adjuncts have relatively little control over their pedagogy, and must follow syllabuses and rules set by others. So the power to execute NSSE ideas, she said, may not rest with those doing most of the teaching.

Research presented here, however, by the Wabash College National Study of Liberal Arts Education offered concrete evidence of direct correlations between NSSE attributes and specific skills, such as critical thinking skills. The Wabash study, which involves 49 colleges of all types, features cohorts of students being analyzed on various NSSE benchmarks (for academic challenge, for instance, or supportive campus environment or faculty-student interaction) and various measures of learning, such as tests to show critical thinking skills or cognitive skills or the development of leadership skills.

The irony of the Wabash work with NSSE data and other data, Blaich said, was that it demonstrates the failure of colleges to act on information they get -- unless someone (in this case Wabash) drives home the ideas.“In every case, after collecting loads of information, we have yet to find a single thing that institutions didn’t already know. Everyone at the institution didn’t know -- it may have been filed away,” he said, but someone had the data. “It just wasn’t followed. There wasn’t sufficient organizational energy to use that data to improve student learning.”

“I want to try to make the point that there is a distinction between participating in NSSE and using NSSE," he said. "In the end, what good is it if all you get is a report?"