Principles of Genuine Evaluation When we set out to explore the notion of ‘Genuine Evaluation’, we identified 5 important aspects of it: VALUE-BASED -transparent and defensible values (criteria of merit and worth and standards of performance) EMPIRICAL – credible evidence about what has happened and what has caused this, USABLE – reported in such a way that it can be understood and used by those who can and should use it (which doesn’t necessarily mean it’s used or used well, of course) SINCERE – a commitment by those commissioning evaluation to respond to information about both success and failure (those doing evaluation can influence this but not control it) HUMBLE – acknowledges its limitations From now until the end of the year, we’re looking at each of these principles and collecting some of the highlights and lowlights  from 2010 (and previously).

The issue that needs to be addressed and spectacularly has been avoided is whether controlled studies (one group does the articulation of and then measurement of outcomes, and a control group does what we have been doing before this mania took hold) can demonstrate or falsify the claim that outcomes assessment results in better-educated students. So far as I can tell, we instead gather data on whether we have in fact been doing outcomes assessment. Not the issue, people. jwp

So far, the most useful tool to me, in physics, has been the rubric, which is used widely in grading open-ended assessments in the humanities.

This method has revolutionized the way I grade. No longer do I have to keep track of how many points are deducted from which type of misstep on what problem for how many students. In the past, I often would get through several tests before I realized that I wasn’t being consistent with the deduction of points, and then I’d have to go through and re-grade all the previous tests. Additionally, the rubric method encourages students to refer to a solution, which I post after the test is administered, and they are motivated to meet with me in person to discuss why they got a 2 versus a 3 on a given problem, for example.

his opens up the opportunity to talk with them personally about their problem-solving skills and how they can better them. The emphasis is moved away from point-by-point deductions and is redirected to a more holistic view of problem solving.

I was very pleased to see the responses to the announcement of the Presidents’ Alliance as generally welcoming (“commendable,” “laudatory initiative,” “applaud”) the shared commitment of these 71 founding institutions to do more—and do it publicly and cooperatively—with regard to gathering, reporting, and using evidence of student learning.

establishing institutional indicators of educational progress that could be valuable in increasing transparency may not suggest what needs changing to improve results

As Adelman’s implied critique of the CLA indicates, we may end up with an indicator without connections to practice.

Arne Duncan, said state and local governments should rethink their policies of giving pay raises to teachers who have master’s degrees because evidence suggests that the degree alone does not improve student achievement.

"If each of the five benchmarks does not measure a distinct dimension of engagement and includes substantial error among its items, it is difficult to inform intervention strategies to improve undergraduates' educational experiences,"

nly one benchmark, enriching educational experiences, had a significant effect on the seniors' cumulative GPA.

Other critics have asserted that the survey's mountains of data remain largely ignored.

Senators want to know why accreditors haven’t protected the public interest.

Congress shouldn’t blame accreditors: it should blame itself. The existing accreditation system has neither ensured quality nor ferreted out fraud. Why? Because Congress didn’t want it to. If Congress truly wants to protect the public interest, it needs to create a system that ensures real accountability.

But turning accreditors into gatekeepers changed the picture. In effect, accreditors now held a gun to the heads of colleges and universities since federal financial aid wouldn’t flow unless the institution received “accredited” status.

Increasing college completion is meaningless unless certificates and degrees represent real learning, which community colleges must work harder to ensure, says a report released on Thursday by the Center for Community College Student Engagement.

This year's report centers on "deep learning," or "broadly applicable thinking, reasoning, and judgment skills—abilities that allow individuals to apply information, develop a coherent world view, and interact in more meaningful ways."

67 percent of community-college students said their coursework often involved analyzing the basic elements of an idea, experience, or theory; 59 percent said they frequently synthesized ideas, information, and experiences in new ways. Other averages were lower: 56 percent of students, for example, reported being regularly asked to examine the strengths or weaknesses of their own views on a topic. And just 52 percent of students said they often had to make judgments about the value or soundness of information as part of their academic work.

evidence of widespread cheating

business course on strategic management,

I don’t condone cheating. But I think it is equally pathetic that faculty are put in situations where they feel the only option for an examination is an easy to grade multiple choice or true/false test

In a venture known as the Presidents' Alliance for Excellence in Student Learning and Accountability, they have promised to take specific steps to gather more evidence about student learning, to use that evidence to improve instruction, and to give the public more information about the quality of learning on their campuses.

The 71 pledges, officially announced on Friday, are essentially a dare to accreditors, parents, and the news media: Come visit in two years, and if we haven't done these things, you can zing us.

deepen an ethic of professional stewardship and self-regulation among college leaders

Neil Gershenfeld: The beckoning promise of personal fabrication

But when it comes to the fundamentals of measuring and improving student learning, engineering professors naturally have more to talk about with their counterparts at, say, Georgia Tech than with the humanities professors at Villanova

But there is no nationally normed way to measure the particular kind of critical thinking that students of classics acquire

er colleagues have created discipline-specific critical-reasoning tests for classics and political science

The National Institute for Learning Outcomes Assessment has just released a white paper about the regional accreditors’ role in prodding colleges to assess their students’ learning

All four presidents suggested that their campuses’ learning-assessment projects are fueled by Fear of Accreditors. One said that a regional accreditor “came down on us hard over assessment.” Another said, “Accreditation visit coming up. This drives what we need to do for assessment.”

Western Association of Schools and Colleges, Ms. Provezis reports, “almost every action letter to institutions over the last five years has required additional attention to assessment, with reasons ranging from insufficient faculty involvement to too little evidence of a plan to sustain assessment.”

Does testing for statistical significance encourage or discourage thoughtful data analysis? Posted by Patricia Rogers on October 20th, 2010

Epidemiology, 9(3):333–337). which argues not only for thoughtful interpretation of findings, but for not reporting statistical significance at all.

We also would like to see the interpretation of a study based not on statistical significance, or lack of it, for one or more study variables, but rather on careful quantitative consideration of the data in light of competing explanations for the findings.

The Association for Assessment of Learning in Higher Education, Inc. (AALHE) is an organization of practitioners interested in using effective assessment practice to document and improve student learning.

it is designed to be a resource by all who are interested in the improvement of learning,

Over 317,000 waiters and waitresses have college degrees (over 8,000 of them have doctoral or professional degrees), along with over 80,000 bartenders, and over 18,000 parking lot attendants.  All told, some 17,000,000 Americans with college degrees are doing jobs that the BLS says require less than the skill levels associated with a bachelor’s degree.

I have long been a proponent of Charles Murray’s thesis that an increasing number of persons attending college do not have the cognitive abilities or other attributes usually necessary for success at higher levels of learning. 

As more and more try to attend colleges, either college degrees will be watered down (something already happening I suspect) or drop-out rates will rise.  

Empowerment Evaluation in Stanford University's School of Medicine

Empowerment evaluation provides a method for gathering, analyzing, and sharing data about a program and its outcomes and encourages faculty, students, and support personnel to actively participate in system changes.

It assumes that the more closely stakeholders are involved in reflecting on evaluation findings, the more likely they are to take ownership of the results and to guide curricular decision making and reform.