Group items tagged

The problem with the test is that it does not directly align with our program's learning outcomes and it does not yield useful information for closing the loop. So why do we use it? Because it is accepted by accreditors as a direct measure and it is less expensive and time-consuming than more useful tools.

Without exception, the most useful information for improving the program and student learning comes from the anecdotal and indirect information.

We don't have the time and the resources to do what we really want to do to continuously improve the quality of our programs and instruction. We don't have a culture of continuous improvement. We don't make changes on a regular basis, because we are trapped by the catalog publishing cycle, accreditation visits, and the entrenched misunderstanding of the purposes of assessment.

The institutions that use it are ones that have adequate resources to do so. The time necessary for training, whole-system involvement, and developing the programs for improvement is daunting. And it is only being used by one regional accrediting body, as far as I know.

Until higher education as a whole is willing to look at changing its approach to assessment, I don't think it will happen

The NRC project's directors say that those small samples are not a problem, because the reputational scores were not converted directly into program assessments. Instead, the scores were used to develop a profile of the kinds of traits that faculty members value in doctoral programs in their field.

For one thing, Mr. Stigler says, the relationships between programs' reputations and the various program traits are probably not simple and linear.

if these correlations between reputation and citations were plotted on a graph, the most accurate representation would be a curved line, not a straight line. (The curve would occur at the tipping point where high citation levels make reputations go sky-high.)

Mr. Stigler says that it was a mistake for the NRC to so thoroughly abandon the reputational measures it used in its previous doctoral studies, in 1982 and 1995. Reputational surveys are widely criticized, he says, but they do provide a check on certain kinds of qualitative measures.

The committee's largest sponsors include GE, Merrill Lynch and Company, IBM, McKinsey and Company, General Motors, and Pfizer.

our award criteria: 1) articulation and evidence of outcomes; 2) success with regard to outcomes; 3) information to the public about outcomes; and 4) use of outcomes for educational improvement.

In addition to Capella University, Portland State University (OR), St. Olaf College (MN) and the University of Arkansas - Fort Smith (AR) also will receive the 2010 CHEA Award. The award will be presented at the 2010 CHEA Annual Conference, which will be held January 25-28 in Washington, D.C

We do not have any contact with your students. Instructors retain full control of the process, from designing the assignments in the first place, to specifying learning outcomes and attaching weights to each outcome. Instructors also review the work of our TAs through a step called the Interim Check, which happens after 10% of the assignments have been completed. Faculty provide feedback, offer any further instructions and eventually sign-off on the work done, before our TAs continue with the remainder of the assignments

Finally, upon the request of the instructor, the weights he/she specified to the learning outcomes will be rubric-based scores which are used to generate a composite score for each student assignment

As an added bonus, our Virtual-TAs provide a detailed, summative report for the instructor on the overall class performance on the given assignment, which includes a look at how the class fared on each outcome, where the students did well, where they stumbled and what concepts, if any, need reinforcing in class the following week.

We can also, upon request, generate reports by Student Learning Outcomes (SLOs). This report can be used by the instructor to immediately address gaps in learning at the individual or classroom level.

Think of this as a micro-closing-of-the-loop that happens each week.  Contrast this with the broader, closing-the-loop that accompanies program-level assessment of learning, which might happen at the end of a whole academic year or later!

The IDEA Center, an education research group based at Kansas State University, has been spreading its particular course-evaluation gospel since 1975. The central innovation of the IDEA system is that departments can tailor their evaluation forms to emphasize whichever learning objectives are most important in their discipline.

(Roughly 350 colleges use the IDEA Center's system, though in some cases only a single department or academic unit participates.)

The new North Texas instrument that came from these efforts tries to correct for biases that are beyond an instructor's control. The questionnaire asks students, for example, whether the classroom had an appropriate size and layout for the course. If students were unhappy with the classroom, and if it appears that their unhappiness inappropriately colored their evaluations of the instructor, the system can adjust the instructor's scores accordingly.

Elaine Seymour, who was then director of ethnography and evaluation research at the University of Colorado at Boulder, was assisting with a National Science Foundation project to improve the quality of science instruction at the college level. She found that many instructors were reluctant to try new teaching techniques because they feared their course-evaluation ratings might decline.

"So the ability to do some quantitative analysis of these comments really allows you to take a more nuanced and effective look at what these students are really saying."

Mr. Frick and his colleagues found that his new course-evaluation form was strongly correlated with both students' and instructors' own measures of how well the students had mastered each course's learning goals.

The survey instrument, known as SALG, for Student Assessment of their Learning Gains, is now used by instructors across the country. The project's Web site contains more than 900 templates, mostly for courses in the sciences.

"Students are the inventory," Mr. Crumbley says. "The real stakeholders in higher education are employers, society, the people who hire our graduates. But what we do is ask the inventory if a professor is good or bad. At General Motors," he says, "you don't ask the cars which factory workers are good at their jobs. You check the cars for defects, you ask the drivers, and that's how you know how the workers are doing."

William H. Pallett, president of the IDEA Center, says that when course rating surveys are well-designed and instructors make clear that they care about them, students will answer honestly and thoughtfully.

In Mr. Bain's view, student evaluations should be just one of several tools colleges use to assess teaching. Peers should regularly visit one another's classrooms, he argues. And professors should develop "teaching portfolios" that demonstrate their ability to do the kinds of instruction that are most important in their particular disciplines. "It's kind of ironic that we grab onto something that seems fixed and fast and absolute, rather than something that seems a little bit messy," he says. "Making decisions about the ability of someone to cultivate someone else's learning is inherently a messy process. It can't be reduced to a formula."

"The evaluation part is critical. No matter which process we used, whether it was the [team] or hybrid model, they all did significantly worse than we hoped [in the evaluation stage]," Terwiesch says. "It's no good generating a great idea if you don't recognize the idea as great. It's like me sitting here and saying I had the idea for Amazon. If I had the idea but didn't do anything about it, then it really doesn't matter that I had the idea."

He says an online system that creates a virtual "suggestion box" can accomplish the same goal as long as it is established to achieve a particular purpose.

Imposing structure doesn't replace or stifle the creativity of employees, Ulrich adds. In fact, the goal is to establish an idea generation process that helps to bring out the best in people. "We have found that, in the early phases of idea generation, providing very specific process guideposts for individuals [such as] 'Generate at least 10 ideas and submit them by Wednesday,' ensures that all members of a team contribute and that they devote sufficient creative energy to the problem."

The results of the experiment with the students showed that average quality of the ideas generated by the hybrid process were better than those that came from the team process by the equivalent of roughly 30 percentage points.

in about three times more ideas than the traditional method.

"We find huge differences in people's levels of creativity, and we just have to face it. We're not all good singers and we're not all good runners, so why should we expect that we all are good idea generators?

They found that ideas built around other ideas are not statistically better than any random suggestion.

"In innovation, variance is your friend. You want wacky stuff because you can afford to reject it if you don't like it. If you build on group norms, the group kills variance."

"The Board of Regents here saw that it would be an important test because it measures analytical ability, problem-solving ability, critical thinking, and communication. Those are the skills that you want every undergraduate to walk away with." (Other large systems that have embraced the CLA include California State University and the West Virginia system.)

value added

We began by administering a retired CLA question, a task that had to do with analyzing crime-reduction strategies,

performance task that mirrors the CLA

Mr. Ernsting and Ms. McConnell are perfectly sincere about using CLA-style tasks to improve instruction on their campuses. But at the same time, colleges have a less high-minded motive for familiarizing students with the CLA style: It just might improve their scores when it comes time to take the actual test.

by 2012, the CLA scores of more than 100 colleges will be posted, for all the world to see, on the "College Portrait" Web site of the Voluntary System of Accountability, an effort by more than 300 public colleges and universities to provide information about life and learning on their campuses.

If familiarizing students with CLA-style tasks does raise their scores, then the CLA might not be a pure, unmediated reflection of the full range of liberal-arts skills. How exactly should the public interpret the scores of colleges that do not use such training exercises?

Trudy W. Banta, a professor of higher education and senior adviser to the chancellor for academic planning and evaluation at Indiana University-Purdue University at Indianapolis, believes it is a serious mistake to publicly release and compare scores on the test. There is too much risk, she says, that policy makers and the public will misinterpret the numbers.

most colleges do not use a true longitudinal model: That is, the students who take the CLA in their first year do not take it again in their senior year. The test's value-added model is therefore based on a potentially apples-and-oranges comparison.

freshman test-takers' scores are assessed relative to their SAT and ACT scores, and so are senior test-takers' scores. For that reason, colleges cannot game the test by recruiting an academically weak pool of freshmen and a strong pool of seniors.

students do not always have much motivation to take the test seriously

seniors, who are typically recruited to take the CLA toward the end of their final semester, when they can already taste the graduation champagne.

Of the few dozen universities that had already chosen to publish CLA data on that site, roughly a quarter of the reports appeared to include erroneous descriptions of the year-to-year value-added scores.

It is clear that CLA scores do reflect some broad properties of a college education.

Students' CLA scores improved if they took courses that required a substantial amount of reading and writing. Many students didn't take such courses, and their CLA scores tended to stay flat.

Colleges that make demands on students can actually develop their skills on the kinds of things measured by the CLA.

Mr. Shavelson believes the CLA's essays and "performance tasks" offer an unusually sophisticated way of measuring what colleges do, without relying too heavily on factual knowledge from any one academic field.

Politicians and consumers want easily interpretable scores, while colleges need subtler and more detailed data to make internal improvements.

The CLA is used at more than 400 colleges

Since its debut a decade ago, it has been widely praised as a sophisticated alternative to multiple-choice tests

the feasibility study is adapting the Collegiate Learning Assessment, an instrument developed by the Council for Aid to Education in the United States, to an international context.

At least six nations are participating in the feasibility study.

14 countries are expected to participate in the full project, with an average of 10 institutions per country and about 200 students per institution,

The project's target population will be students nearing the end of three-year or four-year degrees, and will eventually measure student knowledge in economics and engineering.

While the goal of the project is not to produce another global ranking of universities, the growing preoccupation with such lists has crystallized what Mr. Yelland described as the urgency of pinning down what exactly it is that most of the world's universities are teaching and how well they are doing

Judith S. Eaton, president of the Council for Higher Education Accreditation, said she was also skeptical about whether the project would eventually yield common international assessment mechanisms.

Ms. Eaton noted, the same sets of issues recur across borders and systems, about how best to enhance student learning and strengthen economic development and international competitiveness.

only if the information they seek to collect is intelligently framed, which the most widely used current measure -- graduation rates -- is not

"work force ready"

Accountability is not quite as straightforward as we think," said Rhoades, who described himself as "not a 'just say no' guy" about accountability. "It's not a question of whether [colleges and faculty should be held accountable], but how, and by whom," he said. "It's about who's developing the measures, and what behaviors do they encourage?"

federal government needs to be the objective protector of taxpayers' dollars,"

Judith Eaton, president of the Council for Higher Education Accreditation, said that government regulation would be a major mistake, but said that accreditors needed to come to agreement on "community-driven, outcomes-based standards" to which colleges should be held.

But while they complain when policy makers seek to develop measures that compare one institution against another, colleges "keep lists of peers with which they compare themselves" on many fronts, Miller said.

The gathering drew about 500 government officials, institutional leaders, and researchers

both education and research must become more relevant and responsive to society.

many faculty members may be "uncomfortable" with having deeper links to industry because they don't understand that world. Students, however, are highly practical, Mr. Fadel said, and are specifically seeking education that will get them a job or give them an advantage in the workplace.

"I'm sorry, as a student, you do not go to university to learn. You go to get a credential," he said.

That does not mean colleges simply ought to turn out more graduates for in-demand professions like science and engineering, Mr. Fadel added. Colleges need to infuse other disciplines with science and engineering skills.

The steps of empowerment evaluation

designating a “critical friend” to communicate areas of potential improvement,

collecting evaluation data,

encouraging a cycle of reflection and action

establishing a culture of evidence

developing reflective educational practitioners.

cultivating a community of learners

yearly cycles of improvement at the Stanford University School of Medicine

Political science cultivates skills that are substantially different from those in classics, and in each case those skills can't be measured with a general-education test.

he wants to use tests of reasoning that are appropriate for each discipline

I believe Richard Paul has spent a lifetime articulating the characteristics of discipline based critical thinking. But anyway, I think it is interesting that an attempt is being made to develop (perhaps) a "national standard" for critical thinking in classics. In order to assess anything effectively we need a standard. Without a standard there are no criteria and therefore no basis from which to assess. But standards do not necessarily have to be established at the national level. This raises the issue of scale. What is the appropriate scale from which to measure the quality and effectiveness of an educational experience? Any valid approach to quality assurance has to be multi-scaled and requires multiple measures over time. But to be honest the issues of standards and scale are really just the tip of the outcomes iceberg.

the UW Study of Undergraduate Learning (UW SOUL). Results from the UW SOUL show that learning in college is disciplinary; therefore, real assessment of learning must occur (with central support and resources)in the academic departments. Generic approaches to assessing thinking, writing, research, quantitative reasoning, and other areas of learning may be measuring something, but they cannot measure learning in college.

It turns out there is a six week, or 210+ hour serious reading exposure to two or more domains outside ones own, that "turns on" cross domain mapping as a robust capability. Some people just happen to have accumulated, usually by unseen and unsensed happenstance involvements (rooming with an engineer, son of a dad changing domains/careers, etc.) this minimum level of basics that allows robust metaphor based mapping.

Going PublicDouglas C. Bennett, President of Earlham College, suggests each institution having a public learning audit document and gives the example of what this means for Earlham College as a way for public accountability.

More TransparencyMartha Kanter, from the US Education Department, calls for more transparency in the way higher education does accreditation.

Despite the uptick in activity, "I still feel like there's no there there" when it comes to colleges' efforts to measure student learning, Kevin Carey, policy director at Education Sector, said in a speech at the Council for Higher Education Accreditation meeting Tuesday.

Most of the assessment activity on campuses can be found in nooks and crannies of the institutions - by individual professors, or in one department - and it is often not tied to goals set broadly at the institutional level.

Nine Principles of Good Practice for Assessing Student Learning