Group items tagged

"Assessing student learning outcomes relative to a valid and reliable standard that is academically-sound and employer-relevant presents a challenge to the scholarship of teaching and learning. In this paper, readers are guided through a method for collaboratively developing and validating a rubric that integrates baseline data collected from academics and professionals. The method addresses two additional goals: (1) to formulate and test a rubric as a teaching and learning protocol for a multi-section course taught by various instructors; and (2) to assure that students' learning outcomes are consistently assessed against the rubric regardless of teacher or section. Steps in the process include formulating the rubric, collecting data, and sequentially analyzing the techniques used to validate the rubric and to insure precision in grading papers in multiple sections of a course."

We should take this opportunity seriously and write up our work. Let me know if you want to join me.

"One purpose of this article is to provide clear definitions of the terms "validity" and "reliability" and illustrate these definitions through examples. A second purpose is to clarify how these issues may be addressed in the development of scoring rubrics."

ongoing discussion of these tools. Note Longanecker's comment and ask me why.

"What is AHELO? The OECD Assessment of Higher Education Learning Outcomes (AHELO) is a ground-breaking initiative to assess learning outcomes on an international scale by creating measures that would be valid for all cultures and languages. Between ten and thirty-thousand higher education students in over ten different countries will take part in a feasibility study to determine the bounds of this ambitious project, with an eye to the possible creation of a full-scale AHELO upon its completion."

What is not challenged is the validity and utility of the construct itself--reputation rankings.

he VALUE project comes into the middle of this tension, as it proposes to create frameworks (or metarubrics) that provide flexible criteria for making valid judgments about student work that might result from a wide range of assessments and learning opportunities, over time. In this interview, Terrel Rhodes, Director of the VALUE project and Vice President of the Association for American Colleges and Universities (AAC&U) describes the assumptions and goals behind the Project. He especially addresses how electronic portfolios serve those goals as the locus of evaluation by educators, providing frameworks for judgments tailored to local contexts but calibrated to "Essential Learning Outcomes," with broad significance for student achievement. The aims and ambitions of the VALUE Project have the potential to move us further down the road toward a more systematic engagement with the expansion of learning. -Randy Bass

This paragraph is the one with the most interesting set of assumptions. There are implications about "validity" Bass notes earlier and the role of numbers as "less robust" rather than, say, an interesting and important ingredient in that conversation. Mostly though I see the designation that the rubrics are "too broad to be useful" as a flag that these are not really rubrics, but, well, flags...

Social Software and Social Networking. Social software includes a broad range of technologies, such as social networking, social collaboration, social media and social validation. Organizations should consider adding a social dimension to a conventional Web site or application and should adopt a social platform sooner, rather than later, because the greatest risk lies in failure to engage and thereby, being left mute in a dialogue where your voice must be heard.

our work, our friends (Ken and Jim), and more context that shapes our strategy.

How about using examples of highly motivational lecture and teaching techniques like the Richard Dawkins video I presented on this forum, recently. Even if teacher's do not consciously try to adopt good working techniques, there is at least a strong subconscious human tendency to mimic behaviors. I think that if teachers see more effective techniques, they will automatically begin to adopt adopt them.

"The three general categories for computing interrater reliability introduced and described in this paper are: 1) consensus estimates, 2) consistency estimates, and 3) measurement estimates. The assumptions, interpretation, advantages, and disadvantages of estimates from each of these three categories are discussed, along with several popular methods of computing interrater reliability coefficients that fall under the umbrella of consensus, consistency, and measurement estimates. Researchers and practitioners should be aware that different approaches to estimating interrater reliability carry with them different implications for how ratings across multiple judges should be summarized, which may impact the validity of subsequent study results."

Since it is not clear if the incentive to participate in this study influenced the decision to participate, it remains similarly unclear if incentives to participate correlate with performance.

at last a decent article on rubric development--a good place to jump off.

Complex inter-relationship between: space, time, locality, practice, boundary crossings between different practices. For example trainee doctor in the hospital in one practice, translation of this experience into 'evidence for assessment purposes' needs to then be 'validated' by auditors in another community of practice.

perhaps these sum to utility, but number 5---generativity- would benefit from some unpacking.-

"Those who are involved with assessment are unfortunately not immune to unethical practices. Abuses in preparing students to take tests as well as in the use and interpretation of test results have been widely publicized. Misuses of test data in high-stakes decisions, such as scholarship awards, retention/promotion decisions, and accountability decisions, have been reported all too frequently. Even claims made in advertisements about the success rates of test coaching courses have raised questions about truth in advertising. Given these and other occurrences of unethical behavior associated with assessment, the purpose of this digest is to examine the available standards of ethical practice in assessment and the issues associated with implementation of these standards. "

FYI, Toulmin was the primary influence on the first WSU Critical Thinking Rubric. (Carella was the other philosopher.)

The lack of correlation might be rephrased--there appears to be no corrlation between what is useful for faculty who teach and what is useful for the VSA. A corollary question: Of what use is the VSA?

an important question.