Group items tagged

First in a series from the Urban Institute on outcome management for non-profits, for an audience of non-evaluation-savvy leadership and staff. Lots to steal here if we ever create an Assessment Handbook for WSU.

A summary of the current state of "thinking" with regard to CLA. Many fallacies and contradictions are (unintentionally) exposed. At least CLA appears to be more about skills than content (though the question of how it is graded isn't even raised), but the "performance task" approach is the smallest possible step in that direction.

"Recently, in working with a board of directors of a grassroots organization, I was reminded of how important it is to "let go" of rigid adherence to typologies and other traditional notions of what an evaluation should look like. For example, I completed an evaluation that incorporated elements of all of the stages of program development - a needs assessment (e.g., how much do board members know about their programs and budget), a process evaluation (e.g., how well do the board members communicate with each other when they meet), and an outcome evaluation (e.g., how effective is their marketing plan for recruiting children and families for its programs)."

Needs evaluation, process evaluation, outcomes evaluation -- all useful for improvement.

Our membership begins November 1

So here it is: by definition, the value-added component of the D.C. IMPACT evaluation system defines 50 percent of all teachers in grades four through eight as ineffective or minimally effective in influencing their students' learning. And given the imprecision of the value-added scores, just by chance some teachers will be categorized as ineffective or minimally effective two years in a row. The system is rigged to label teachers as ineffective or minimally effective as a precursor to firing them.

How assessment of value-added actually works in one setting: the Washington, D.C. public schools. This article actually works the numbers to show that the system is set up to put teachers in the firing zone. Note the tyranny of numerical ratings (some of them subjective) converted into meanings like "minimally effective".

"We conclude that value-added data has an important role to play in teacher evaluation systems, but that there is much to be learned about how best to use value-added information in human resource decisions." No mention of the role of assessment in improvement.

Google Gadget Ventures is a new Google pilot program dedicated to helping developers create richer, more useful gadgets. Inspired by the success of iGoogle, which has been driven by the creation by 3rd-party developers of a broad range of gadgets, Gadget Ventures provides two types of funding: Grants of $5,000 to those who've built gadgets we'd like to see developed further. You're eligible to apply for a grant if you've developed a gadget that's in our gadgets directory and gets at least 250,000 weekly page views. To apply, you must submit a one-page proposal detailing how you'd use the grant to improve your gadget. Seed investments of $100,000 to developers who'd like to build a business around the gadgets platform. Only Google Gadget Venture grant recipients are eligible for this type of funding. Submitting a business plan detailing how you plan to build a viable business around the gadgets platform is a required part of the seed investment application process. It's our hope that Google Gadget Ventures will give developers the opportunity to create a new generation of gadgets to benefit users. ---------- Consider this form of authentic assessment and the metrics they apply.

The homeopathic fallacy again, and as goes law school, so goes law....

At the Middle States meeting, the Temple officials offered strategies for winning faculty involvement and follow through: Recognize differences among departments; publicize success stories; start with the basics; reward -- don't punish -- flaws that are revealed. I think it would be worth talking about these approaches and the extent to which OAI is / isn't or should / shouldn't build them in.

a small but clear stress made for independent review

Abstract: This study explores the effect on discussion quality of adding a feedback mechanism that presents users with an aggregate peer rating of the usefulness of the participant's contributions in online, asynchronous discussion. Participants in the study groups were able to specify the degree to which they thought any posted comment was useful to the discussion. Individuals were regularly presented with feedback (aggregated and anonymous) summarizing peers' assessment of the usefulness of their contribution, along with a summary of how the individuals rated their peers. Results indicate that continuous, anonymous, aggregated feedback had no effect on either the students' or the instructors' perception of discussion quality. This is kind of a show-stopper. It's just one study but when you look at the results there appears to be no effect whatsoever from peers giving feedback about the usefulness of discussion posts, nor any perceived improvement in the quality of the discussions as evaluated by faculty. It looks like we'll need to begin looking carefully at just what kinds of feedback will really make a difference. Following up on Corinna's earlier post http://blogs.hbr.org/cs/2010/03/twitters_potential_as_microfee.html about the effectiveness of short immediate feedback being more effective than lengthier feedback that actually hinders performance. The trick will be to figure out just what kinds of feedback will actually work in embedded situations. It's interesting that an assessment of utility wasn't useful...?

the outsourcing loop becomes a diatribe against rubrics...

It's hard to see how either outsourced assessment or harvested assessment can be accomplished convincingly without rubrics. How else can the standards of the teacher be enacted by the grader? From there we are driven to consider how, in the absence of a rubric, the standards of the teacher can be enacted by the student. Is it "ethical" to use the Potter Stewart standard: "I'll know it when I see it"?

Yes, who is the "priest" in the preceding rendering--one who shares principles of quality (rubrics), or one who divines a grade a proclaims who is a "real, viable, educated adult"?

a long blog, but underlying principles align with the Guide to Effective Assessment on many levels.

High school principal George Wood eloquently contrasts standardized NCLB-style testing with his school's performance assessments.

"So far, the most useful tool to me, in physics, has been the rubric, which is used widely in grading open-ended assessments in the humanities. "

A focus on improving the grading experience, rather than the learning experience, but still a big step forward for (some) hard scientists.

feldstein summarizes the findings from North Carolina.

Washington Monthly's College Guide includes an introduction by the editor which frames the issue of assessment in terms of the unsustainability of the current paradigm and its rising costs. He asserts that "the higher education market is missing a measure of value, quality divided by price". "Value measures would allow colleges to do what they can't do now: lower prices without being punished by the market." The introduction points to several other useful articles in the College Guide.