Skip to main content

Home/ CTLT and Friends/ Group items tagged validating rubrics

Rss Feed Group items tagged

Filter: All | Bookmarks | Topics Simple Middle
Corinna Lo

IJ-SoTL - A Method for Collaboratively Developing and Validating a Rubric - 1 views

academics.georgiasouthern.edu/...index.htm

assessment rubric

shared by Corinna Lo on 16 Apr 10 - Cached
  • Corinna Lo on 16 Apr 10
     
    "Assessing student learning outcomes relative to a valid and reliable standard that is academically-sound and employer-relevant presents a challenge to the scholarship of teaching and learning. In this paper, readers are guided through a method for collaboratively developing and validating a rubric that integrates baseline data collected from academics and professionals. The method addresses two additional goals: (1) to formulate and test a rubric as a teaching and learning protocol for a multi-section course taught by various instructors; and (2) to assure that students' learning outcomes are consistently assessed against the rubric regardless of teacher or section. Steps in the process include formulating the rubric, collecting data, and sequentially analyzing the techniques used to validate the rubric and to insure precision in grading papers in multiple sections of a course."
Nils Peterson

AAC&U News | April 2010 | Feature - 1 views

www.aacu.org/...feature.cfm

shared by Nils Peterson on 01 Apr 10 - Cached
  • Comparing Rubric Assessments to Standardized Tests
  • First, the university, a public institution of about 40,000 students in Ohio, needed to comply with the Voluntary System of Accountability (VSA), which requires that state institutions provide data about graduation rates, tuition, student characteristics, and student learning outcomes, among other measures, in the consistent format developed by its two sponsoring organizations, the Association of Public and Land-grant Universities (APLU), and the Association of State Colleges and Universities (AASCU).
  • And finally, UC was accepted in 2008 as a member of the fifth cohort of the Inter/National Coalition for Electronic Portfolio Research, a collaborative body with the goal of advancing knowledge about the effect of electronic portfolio use on student learning outcomes.  
  • ...13 more annotations...
  • outcomes required of all UC students—including critical thinking, knowledge integration, social responsibility, and effective communication
  • “The wonderful thing about this approach is that full-time faculty across the university  are gathering data about how their  students are doing, and since they’ll be teaching their courses in the future, they’re really invested in rubric assessment—they really care,” Escoe says. In one case, the capstone survey data revealed that students weren’t doing as well as expected in writing, and faculty from that program adjusted their pedagogy to include more writing assignments and writing assessments throughout the program, not just at the capstone level. As the university prepares to switch from a quarter system to semester system in two years, faculty members are using the capstone survey data to assist their course redesigns, Escoe says.
  • the university planned a “dual pilot” study examining the applicability of electronic portfolio assessment of writing and critical thinking alongside the Collegiate Learning Assessment,
  • The rubrics the UC team used were slightly modified versions of those developed by AAC&U’s Valid Assessment of Learning in Undergraduate Education (VALUE) project. 
  • In the critical thinking rubric assessment, for example, faculty evaluated student proposals for experiential honors projects that they could potentially complete in upcoming years.  The faculty assessors were trained and their rubric assessments “normed” to ensure that interrater reliability was suitably high.
  • “It’s not some nitpicky, onerous administrative add-on. It’s what we do as we teach our courses, and it really helps close that assessment loop.”
  • There were many factors that may have contributed to the lack of correlation, she says, including the fact that the CLA is timed, while the rubric assignments are not; and that the rubric scores were diagnostic and included specific feedback, while the CLA awarded points “in a black box”:
  • faculty members may have had exceptionally high expectations of their honors students and assessed the e-portfolios with those high expectations in mind—leading to results that would not correlate to a computer-scored test. 
  • “The CLA provides scores at the institutional level. It doesn’t give me a picture of how I can affect those specific students’ learning. So that’s where rubric assessment comes in—you can use it to look at data that’s compiled over time.”
  • Their portfolios are now more like real learning portfolios, not just a few artifacts, and we want to look at them as they go into their third and fourth years to see what they can tell us about students’ whole program of study.”  Hall and Robles are also looking into the possibility of forming relationships with other schools from NCEPR to exchange student e-portfolios and do a larger study on the value of rubric assessment of student learning.
  • “We’re really trying to stress that assessment is pedagogy,”
  • “We found no statistically significant correlation between the CLA scores and the portfolio scores,”
  • In the end, Escoe says, the two assessments are both useful, but for different things. The CLA can provide broad institutional data that satisfies VSA requirements, while rubric-based assessment provides better information to facilitate continuous program improvement.
    • Nils Peterson
      Nils Peterson on 01 Apr 10
       
      CLA did not provide information for continuous program improvement -- we've heard this argument before
  • Gary Brown on 01 Apr 10
     
    The lack of correlation might be rephrased--there appears to be no corrlation between what is useful for faculty who teach and what is useful for the VSA. A corollary question: Of what use is the VSA?
Corinna Lo

Scoring rubric development: validity and reliability. Moskal, Barbara M. & Jon A. Leydens - 1 views

pareonline.net/getvn.asp

reliability assessment validity

shared by Corinna Lo on 16 Apr 10 - Cached
  • Corinna Lo on 16 Apr 10
     
    "One purpose of this article is to provide clear definitions of the terms "validity" and "reliability" and illustrate these definitions through examples. A second purpose is to clarify how these issues may be addressed in the development of scoring rubrics."
Gary Brown

IJ-SoTL: Current Issue: Volumn 3, Number 2 - July 2009 - 0 views

academics.georgiasouthern.edu/...v3n2.html

validating rubrics

shared by Gary Brown on 09 Jul 09 - Cached
  • A Method for Collaboratively Developing and Validating a Rubric Sandra Allen (Columbia College Chicago) & John Knight (University of Tennessee at Martin)
  • Gary Brown on 09 Jul 09
     
    at last a decent article on rubric development--a good place to jump off.
Gary Brown

Can We Promote Experimentation and Innovation in Learning as well as Accountability? In... - 0 views

www.academiccommons.org/...value-project-rhodes-interview

shared by Gary Brown on 18 Mar 09 - Cached
  • Theron DesRosier on 18 Mar 09
     
    he VALUE project comes into the middle of this tension, as it proposes to create frameworks (or metarubrics) that provide flexible criteria for making valid judgments about student work that might result from a wide range of assessments and learning opportunities, over time. In this interview, Terrel Rhodes, Director of the VALUE project and Vice President of the Association for American Colleges and Universities (AAC&U) describes the assumptions and goals behind the Project. He especially addresses how electronic portfolios serve those goals as the locus of evaluation by educators, providing frameworks for judgments tailored to local contexts but calibrated to "Essential Learning Outcomes," with broad significance for student achievement. The aims and ambitions of the VALUE Project have the potential to move us further down the road toward a more systematic engagement with the expansion of learning. -Randy Bass
  • Gary Brown on 18 Mar 09
     
    This paragraph is the one with the most interesting set of assumptions. There are implications about "validity" Bass notes earlier and the role of numbers as "less robust" rather than, say, an interesting and important ingredient in that conversation. Mostly though I see the designation that the rubrics are "too broad to be useful" as a flag that these are not really rubrics, but, well, flags...
Gary Brown

Wise Men Gone: Stephen Toulmin and John E. Smith - The Chronicle Review - The Chronicle... - 0 views

chronicle.com/...63649

critical thinking

shared by Gary Brown on 26 Jan 10 - Cached
  • Toulmin, born in London in 1922, earned his undergraduate degree in 1942 from King's College, Cambridge, in mathematics and physics. After participating in radar research and intelligence work during World War II in England and at Allied headquarters in Germany, he returned to Cambridge, where he studied with Ludwig Wittgenstein, the greatest influence on his thought, earning his Ph.D. in moral philosophy in 1948.
  • Toulmin moved to the United States, where he taught at Brandeis, Michigan State, and Northwestern Universities and the University of Chicago before landing in 1993 at the University of Southern California.
  • Toulmin's first, most enduring contribution to keeping philosophy sensible came in his 1958 book, The Uses of Argument (Cambridge University Press). Deceptively formalistic on its surface because it posited a general model of argument, Toulmin's view, in fact, was better described as taxonomic, yet flexible. He believed that formal systems of logic misrepresent the complex way that humans reason in most fields requiring what philosophers call "practical reason," and he offered, accordingly, a theory of knowledge as warranted belief.
  • ...4 more annotations...
  • Toulmin rejected the abstract syllogistic logic, meant to produce absolute standards for proving propositions true, that had become fashionable in analytic philosophy. Instead he argued (in the spirit of Wittgenstein) that philosophers must monitor how people actually argue if the philosophers' observations about persuasion are to make any sense. Toulmin took jurisprudential reasoning as his chief example in The Uses of Argument, but he believed that some aspects of a good argument depend on the field in which they're presented, while others are "field invariant."
  • Toulmin's "central thesis is that every sort of argumentation can in principle claim rationality and that the criteria to be applied when determining the soundness of the argumentation depend on the nature of the problems to which the argumentation relates."
  • But Toulmin, trained in the hard sciences and mathematics himself, saw through the science worship of less-credentialed sorts. He didn't relent, announcing "our need to reappropriate the wisdom of the 16th-century humanists, and develop a point of view that combines the abstract rigor and exactitude of the 17th-century 'new philosophy' with a practical concern for human life in its concrete detail."
  • Toulmin declared its upshot: "From now on, permanent validity must be set aside as illusory, and our idea of rationality related to specific functions of ... human reason. ... For me personally, the outcome of 40 years of philosophical critique was thus a new vision of—so to speak—the rhetoric of philosophy."
  • Gary Brown on 26 Jan 10
     
    FYI, Toulmin was the primary influence on the first WSU Critical Thinking Rubric. (Carella was the other philosopher.)
1 - 6 of 6
Showing 20▼ items per page